Files
headroom/tests/test_evals_metrics.py
Tejas ChopraandClaude Opus 5.5 de8969267c chore(evals): remove dead and test-only eval helpers (#4079)
## Description

This PR removes private, unexported helpers inside live eval modules
that no product code, CI job, CLI command or doc calls. None of them is
in an `__all__`, a package re-export, `docs/content/docs/` or a consumer
repo. Two of them were reached only by tests; those test lines go with
them.

Refs #4136
## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- `headroom/evals/comprehensive_benchmark.py`: remove `TIER1_TASKS`,
`TIER1_CODE_TASKS`, `EXTENDED_TASKS` and `TIER3_TASKS`.
`suite_runner.py` defines its own task lists and only imports
`DEFAULT_TASKS`/`run_lm_eval`.
- `headroom/evals/memory/judge.py`: remove `create_batch_judge`. It is
not exported from `headroom.evals.memory`.
- `headroom/evals/metrics.py`: remove `compute_bleu`, which was
test-only and not in `headroom.evals.__all__`. The "BLEU" mention in
`headroom/evals/README.md` is left for the follow-up docs PR.
- `headroom/evals/datasets.py`: remove `generate_retrieval_probes`,
which was test-only and not exported.
- Tests: drop the 4 `compute_bleu` asserts and the
`generate_retrieval_probes` block, and rename the two test functions to
match what they still cover.
- Evidence: a word search (`rg -c -w <name>`) across the repo (code,
tests, docs, wiki, CI) and the private consumer repos finds only each
definition, plus the removed test lines.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest -q tests/test_evals_metrics.py tests/test_evals_datasets.py tests/test_evals_cjk_tokenization.py tests/test_memory_eval.py tests/test_evals/test_comprehensive_benchmark_log_redaction.py tests/test_dataset_recall_runner.py tests/test_compression_fidelity_regression.py
53 passed in 4.70s
$ ruff check (touched files)
All checks passed!
$ mypy headroom/evals/metrics.py headroom/evals/datasets.py headroom/evals/memory/judge.py headroom/evals/comprehensive_benchmark.py --ignore-missing-imports
Success: no issues found in 4 source files
```

## Real Behavior Proof

- Environment: macOS, repo venv, branch checkout at a6148739, temp HOME
/ HEADROOM_WORKSPACE_DIR, offline.
- Exact command / steps: (1) shared smoke harness: start a real proxy
from this checkout against a local fake upstream and send one Anthropic
and one OpenAI request with a compressible tool result; (2) run the eval
paths that sit next to the removed helpers: the exported metrics, the
lm-eval default task list, the memory judge, and the zero-cost CCR
round-trip eval that CI runs (command below).
- Observed result: smoke requests return HTTP 200 with savings; the
remaining metrics, the default task list, `simple_judge` and the CCR
round-trip eval all run, and the CCR round-trip passes 10/10.
- Not tested: the paid LLM evals (they need API keys); none of them
calls a removed helper.

```text
$ bash smoke.sh <worktree>
branch: cleanup/providers-evals-dead-eval-helpers @ a6148739
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines:
   2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.

$ HOME=$(mktemp -d) python -c "<compute_f1 / compute_rouge_l / compute_exact_match; DEFAULT_TASKS; simple_judge; CompressionOnlyRunner CCR round-trip n=10>"
f1 0.8 rougeL 0.8 em True
default lm-eval tasks ['gsm8k']
simple_judge (3.0, 'Moderate token overlap (F1=0.50)')
CCR round-trip: 10/10 passed
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A, dead code only.
- Stable/default behavior changed: No.
- Kill switch / disable path: N/A.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.

The unused methods on the exported `EvalSuite` and
`CompressionEvaluator` classes (`add_case`, `to_jsonl`,
`evaluate_suite`) are public API, so they are deprecated in a separate
PR (`cleanup/providers-evals-deprecate-eval-suite-methods`), not removed
here.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 08:57:19 -07:00

149 lines
5.6 KiB
Python

from __future__ import annotations
import math
import sys
from types import SimpleNamespace
import pytest
from headroom.evals import metrics
def test_normalize_tokenize_and_exact_match() -> None:
assert metrics.normalize_text(" Hello,\nWORLD ") == "hello, world"
assert metrics.tokenize("Hello, world! API_v2") == ["hello", "world", "api_v2"]
assert metrics.compute_exact_match(" Hello World ", "hello\nworld") is True
assert metrics.compute_exact_match("hello", "world") is False
def test_f1_and_rouge_cover_edge_cases() -> None:
assert metrics.compute_f1("", "value") == 0.0
assert metrics.compute_f1("alpha beta", "gamma delta") == 0.0
assert metrics.compute_f1("alpha beta gamma", "alpha gamma") == pytest.approx(0.8)
assert metrics.compute_rouge_l("", "value") == 0.0
assert metrics.compute_rouge_l("alpha beta", "gamma delta") == 0.0
assert metrics.compute_rouge_l("alpha beta gamma", "alpha gamma") == pytest.approx(0.8)
def test_compute_semantic_similarity_and_zero_norm(monkeypatch: pytest.MonkeyPatch) -> None:
fake_numpy = SimpleNamespace(
dot=lambda a, b: sum(x * y for x, y in zip(a, b)),
linalg=SimpleNamespace(norm=lambda a: math.sqrt(sum(x * x for x in a))),
)
monkeypatch.setitem(sys.modules, "numpy", fake_numpy)
class FakeModel:
def __init__(self, embeddings: list[list[float]]) -> None:
self.embeddings = embeddings
def encode(self, values: list[str]) -> list[list[float]]:
assert values == ["first", "second"]
return self.embeddings
monkeypatch.setattr(
"headroom.models.ml_models.MLModelRegistry.get_sentence_transformer",
lambda model_name=None: FakeModel([[1.0, 0.0], [1.0, 0.0]]),
)
assert metrics.compute_semantic_similarity("first", "second") == 1.0
monkeypatch.setattr(
"headroom.models.ml_models.MLModelRegistry.get_sentence_transformer",
lambda model_name=None: FakeModel([[0.0, 0.0], [1.0, 0.0]]),
)
assert metrics.compute_semantic_similarity("first", "second") == 0.0
def test_compute_answer_equivalence_uses_multiple_paths(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(metrics, "compute_semantic_similarity", lambda a, b: 0.2)
exact = metrics.compute_answer_equivalence("Answer", "answer", ground_truth="missing")
assert exact["exact_match"] is True
assert exact["equivalent"] is True
assert exact["ground_truth_in_a"] is False
assert exact["ground_truth_in_b"] is False
monkeypatch.setattr(metrics, "compute_semantic_similarity", lambda a, b: 0.1)
high_f1 = metrics.compute_answer_equivalence(
"alpha beta gamma",
"alpha gamma",
semantic_threshold=0.95,
f1_threshold=0.75,
)
assert high_f1["equivalent"] is True
assert high_f1["semantic_similarity"] == 0.1
monkeypatch.setattr(metrics, "compute_semantic_similarity", lambda a, b: 0.95)
semantic = metrics.compute_answer_equivalence(
"completely different",
"nothing in common",
semantic_threshold=0.9,
f1_threshold=0.99,
)
assert semantic["equivalent"] is True
assert semantic["semantic_similarity"] == 0.95
def raise_import_error(a: str, b: str) -> float:
raise ImportError("missing dependency")
monkeypatch.setattr(metrics, "compute_semantic_similarity", raise_import_error)
ground_truth = metrics.compute_answer_equivalence(
"The capital is Paris.",
"Paris is definitely the capital city.",
ground_truth="paris",
semantic_threshold=0.99,
f1_threshold=0.99,
)
assert ground_truth["semantic_similarity"] is None
assert ground_truth["ground_truth_in_a"] is True
assert ground_truth["ground_truth_in_b"] is True
assert ground_truth["equivalent"] is True
not_equivalent = metrics.compute_answer_equivalence(
"alpha beta",
"gamma delta",
ground_truth="omega",
semantic_threshold=0.99,
f1_threshold=0.99,
)
assert not_equivalent["equivalent"] is False
def test_information_recall_reports_preserved_and_missing_facts() -> None:
result = metrics.compute_information_recall(
"Alice likes pizza and Bob likes ramen.",
"Alice likes pizza.",
["Alice", "Bob", "ramen", "Carol"],
)
assert result == {
"total_probes": 4,
"facts_in_original": 3,
"facts_preserved": 1,
"facts_lost": ["Bob", "ramen"],
"recall": pytest.approx(1 / 3),
}
empty_original = metrics.compute_information_recall("No facts here", "Still none", ["Alice"])
assert empty_original["facts_in_original"] == 0
assert empty_original["recall"] == 1.0
def test_tool_schema_compaction_integrity() -> None:
"""Property names that collide with DROP_KEYS must survive schema compaction.
Runs the full CompressionOnlyRunner.evaluate_tool_schema_compaction() path
against the built-in cases and asserts zero failures. This is zero-cost
(no API calls) and safe for CI smoke runs.
"""
from headroom.evals.runners.compression_only import CompressionOnlyRunner
runner = CompressionOnlyRunner()
result = runner.evaluate_tool_schema_compaction()
assert result.passed, (
f"Tool schema compaction integrity failures "
f"({result.failed_cases}/{result.total_cases}):\n" + "\n".join(result.errors)
)
assert result.total_cases == 4, f"Expected 4 built-in cases, got {result.total_cases}"
# Annotations were stripped so byte count must have shrunk.
assert result.total_tokens_saved > 0, "Expected at least some annotation tokens to be stripped"