mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description This PR removes private, unexported helpers inside live eval modules that no product code, CI job, CLI command or doc calls. None of them is in an `__all__`, a package re-export, `docs/content/docs/` or a consumer repo. Two of them were reached only by tests; those test lines go with them. Refs #4136 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - `headroom/evals/comprehensive_benchmark.py`: remove `TIER1_TASKS`, `TIER1_CODE_TASKS`, `EXTENDED_TASKS` and `TIER3_TASKS`. `suite_runner.py` defines its own task lists and only imports `DEFAULT_TASKS`/`run_lm_eval`. - `headroom/evals/memory/judge.py`: remove `create_batch_judge`. It is not exported from `headroom.evals.memory`. - `headroom/evals/metrics.py`: remove `compute_bleu`, which was test-only and not in `headroom.evals.__all__`. The "BLEU" mention in `headroom/evals/README.md` is left for the follow-up docs PR. - `headroom/evals/datasets.py`: remove `generate_retrieval_probes`, which was test-only and not exported. - Tests: drop the 4 `compute_bleu` asserts and the `generate_retrieval_probes` block, and rename the two test functions to match what they still cover. - Evidence: a word search (`rg -c -w <name>`) across the repo (code, tests, docs, wiki, CI) and the private consumer repos finds only each definition, plus the removed test lines. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ pytest -q tests/test_evals_metrics.py tests/test_evals_datasets.py tests/test_evals_cjk_tokenization.py tests/test_memory_eval.py tests/test_evals/test_comprehensive_benchmark_log_redaction.py tests/test_dataset_recall_runner.py tests/test_compression_fidelity_regression.py 53 passed in 4.70s $ ruff check (touched files) All checks passed! $ mypy headroom/evals/metrics.py headroom/evals/datasets.py headroom/evals/memory/judge.py headroom/evals/comprehensive_benchmark.py --ignore-missing-imports Success: no issues found in 4 source files ``` ## Real Behavior Proof - Environment: macOS, repo venv, branch checkout ata6148739, temp HOME / HEADROOM_WORKSPACE_DIR, offline. - Exact command / steps: (1) shared smoke harness: start a real proxy from this checkout against a local fake upstream and send one Anthropic and one OpenAI request with a compressible tool result; (2) run the eval paths that sit next to the removed helpers: the exported metrics, the lm-eval default task list, the memory judge, and the zero-cost CCR round-trip eval that CI runs (command below). - Observed result: smoke requests return HTTP 200 with savings; the remaining metrics, the default task list, `simple_judge` and the CCR round-trip eval all run, and the CCR round-trip passes 10/10. - Not tested: the paid LLM evals (they need API keys); none of them calls a removed helper. ```text $ bash smoke.sh <worktree> branch: cleanup/providers-evals-dead-eval-helpers @a6148739health: 200 POST /v1/messages -> HTTP200 smoke-ok POST /v1/chat/completions -> HTTP200 smoke-ok stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82 proxy.log WARNING/ERROR lines: 2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes. 4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base 4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection. $ HOME=$(mktemp -d) python -c "<compute_f1 / compute_rouge_l / compute_exact_match; DEFAULT_TASKS; simple_judge; CompressionOnlyRunner CCR round-trip n=10>" f1 0.8 rougeL 0.8 em True default lm-eval tasks ['gsm8k'] simple_judge (3.0, 'Moderate token overlap (F1=0.50)') CCR round-trip: 10/10 passed ``` ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: N/A, dead code only. - Stable/default behavior changed: No. - Kill switch / disable path: N/A. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Part of maintainer-requested cleanup #4136 (dead code, duplication, debuggability), split into small PRs for review. The unused methods on the exported `EvalSuite` and `CompressionEvaluator` classes (`add_case`, `to_jsonl`, `evaluate_suite`) are public API, so they are deprecated in a separate PR (`cleanup/providers-evals-deprecate-eval-suite-methods`), not removed here. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
149 lines
5.6 KiB
Python
149 lines
5.6 KiB
Python
from __future__ import annotations
|
|
|
|
import math
|
|
import sys
|
|
from types import SimpleNamespace
|
|
|
|
import pytest
|
|
|
|
from headroom.evals import metrics
|
|
|
|
|
|
def test_normalize_tokenize_and_exact_match() -> None:
|
|
assert metrics.normalize_text(" Hello,\nWORLD ") == "hello, world"
|
|
assert metrics.tokenize("Hello, world! API_v2") == ["hello", "world", "api_v2"]
|
|
assert metrics.compute_exact_match(" Hello World ", "hello\nworld") is True
|
|
assert metrics.compute_exact_match("hello", "world") is False
|
|
|
|
|
|
def test_f1_and_rouge_cover_edge_cases() -> None:
|
|
assert metrics.compute_f1("", "value") == 0.0
|
|
assert metrics.compute_f1("alpha beta", "gamma delta") == 0.0
|
|
assert metrics.compute_f1("alpha beta gamma", "alpha gamma") == pytest.approx(0.8)
|
|
|
|
assert metrics.compute_rouge_l("", "value") == 0.0
|
|
assert metrics.compute_rouge_l("alpha beta", "gamma delta") == 0.0
|
|
assert metrics.compute_rouge_l("alpha beta gamma", "alpha gamma") == pytest.approx(0.8)
|
|
|
|
|
|
def test_compute_semantic_similarity_and_zero_norm(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
fake_numpy = SimpleNamespace(
|
|
dot=lambda a, b: sum(x * y for x, y in zip(a, b)),
|
|
linalg=SimpleNamespace(norm=lambda a: math.sqrt(sum(x * x for x in a))),
|
|
)
|
|
monkeypatch.setitem(sys.modules, "numpy", fake_numpy)
|
|
|
|
class FakeModel:
|
|
def __init__(self, embeddings: list[list[float]]) -> None:
|
|
self.embeddings = embeddings
|
|
|
|
def encode(self, values: list[str]) -> list[list[float]]:
|
|
assert values == ["first", "second"]
|
|
return self.embeddings
|
|
|
|
monkeypatch.setattr(
|
|
"headroom.models.ml_models.MLModelRegistry.get_sentence_transformer",
|
|
lambda model_name=None: FakeModel([[1.0, 0.0], [1.0, 0.0]]),
|
|
)
|
|
assert metrics.compute_semantic_similarity("first", "second") == 1.0
|
|
|
|
monkeypatch.setattr(
|
|
"headroom.models.ml_models.MLModelRegistry.get_sentence_transformer",
|
|
lambda model_name=None: FakeModel([[0.0, 0.0], [1.0, 0.0]]),
|
|
)
|
|
assert metrics.compute_semantic_similarity("first", "second") == 0.0
|
|
|
|
|
|
def test_compute_answer_equivalence_uses_multiple_paths(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
monkeypatch.setattr(metrics, "compute_semantic_similarity", lambda a, b: 0.2)
|
|
exact = metrics.compute_answer_equivalence("Answer", "answer", ground_truth="missing")
|
|
assert exact["exact_match"] is True
|
|
assert exact["equivalent"] is True
|
|
assert exact["ground_truth_in_a"] is False
|
|
assert exact["ground_truth_in_b"] is False
|
|
|
|
monkeypatch.setattr(metrics, "compute_semantic_similarity", lambda a, b: 0.1)
|
|
high_f1 = metrics.compute_answer_equivalence(
|
|
"alpha beta gamma",
|
|
"alpha gamma",
|
|
semantic_threshold=0.95,
|
|
f1_threshold=0.75,
|
|
)
|
|
assert high_f1["equivalent"] is True
|
|
assert high_f1["semantic_similarity"] == 0.1
|
|
|
|
monkeypatch.setattr(metrics, "compute_semantic_similarity", lambda a, b: 0.95)
|
|
semantic = metrics.compute_answer_equivalence(
|
|
"completely different",
|
|
"nothing in common",
|
|
semantic_threshold=0.9,
|
|
f1_threshold=0.99,
|
|
)
|
|
assert semantic["equivalent"] is True
|
|
assert semantic["semantic_similarity"] == 0.95
|
|
|
|
def raise_import_error(a: str, b: str) -> float:
|
|
raise ImportError("missing dependency")
|
|
|
|
monkeypatch.setattr(metrics, "compute_semantic_similarity", raise_import_error)
|
|
ground_truth = metrics.compute_answer_equivalence(
|
|
"The capital is Paris.",
|
|
"Paris is definitely the capital city.",
|
|
ground_truth="paris",
|
|
semantic_threshold=0.99,
|
|
f1_threshold=0.99,
|
|
)
|
|
assert ground_truth["semantic_similarity"] is None
|
|
assert ground_truth["ground_truth_in_a"] is True
|
|
assert ground_truth["ground_truth_in_b"] is True
|
|
assert ground_truth["equivalent"] is True
|
|
|
|
not_equivalent = metrics.compute_answer_equivalence(
|
|
"alpha beta",
|
|
"gamma delta",
|
|
ground_truth="omega",
|
|
semantic_threshold=0.99,
|
|
f1_threshold=0.99,
|
|
)
|
|
assert not_equivalent["equivalent"] is False
|
|
|
|
|
|
def test_information_recall_reports_preserved_and_missing_facts() -> None:
|
|
result = metrics.compute_information_recall(
|
|
"Alice likes pizza and Bob likes ramen.",
|
|
"Alice likes pizza.",
|
|
["Alice", "Bob", "ramen", "Carol"],
|
|
)
|
|
assert result == {
|
|
"total_probes": 4,
|
|
"facts_in_original": 3,
|
|
"facts_preserved": 1,
|
|
"facts_lost": ["Bob", "ramen"],
|
|
"recall": pytest.approx(1 / 3),
|
|
}
|
|
|
|
empty_original = metrics.compute_information_recall("No facts here", "Still none", ["Alice"])
|
|
assert empty_original["facts_in_original"] == 0
|
|
assert empty_original["recall"] == 1.0
|
|
|
|
|
|
def test_tool_schema_compaction_integrity() -> None:
|
|
"""Property names that collide with DROP_KEYS must survive schema compaction.
|
|
|
|
Runs the full CompressionOnlyRunner.evaluate_tool_schema_compaction() path
|
|
against the built-in cases and asserts zero failures. This is zero-cost
|
|
(no API calls) and safe for CI smoke runs.
|
|
"""
|
|
from headroom.evals.runners.compression_only import CompressionOnlyRunner
|
|
|
|
runner = CompressionOnlyRunner()
|
|
result = runner.evaluate_tool_schema_compaction()
|
|
|
|
assert result.passed, (
|
|
f"Tool schema compaction integrity failures "
|
|
f"({result.failed_cases}/{result.total_cases}):\n" + "\n".join(result.errors)
|
|
)
|
|
assert result.total_cases == 4, f"Expected 4 built-in cases, got {result.total_cases}"
|
|
# Annotations were stripped so byte count must have shrunk.
|
|
assert result.total_tokens_saved > 0, "Expected at least some annotation tokens to be stripped"
|