Commit Graph
3 Commits
Author SHA1 Message Date
Tejas ChopraandClaude Opus 5.5 de8969267c chore(evals): remove dead and test-only eval helpers (#4079)
## Description

This PR removes private, unexported helpers inside live eval modules
that no product code, CI job, CLI command or doc calls. None of them is
in an `__all__`, a package re-export, `docs/content/docs/` or a consumer
repo. Two of them were reached only by tests; those test lines go with
them.

Refs #4136
## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- `headroom/evals/comprehensive_benchmark.py`: remove `TIER1_TASKS`,
`TIER1_CODE_TASKS`, `EXTENDED_TASKS` and `TIER3_TASKS`.
`suite_runner.py` defines its own task lists and only imports
`DEFAULT_TASKS`/`run_lm_eval`.
- `headroom/evals/memory/judge.py`: remove `create_batch_judge`. It is
not exported from `headroom.evals.memory`.
- `headroom/evals/metrics.py`: remove `compute_bleu`, which was
test-only and not in `headroom.evals.__all__`. The "BLEU" mention in
`headroom/evals/README.md` is left for the follow-up docs PR.
- `headroom/evals/datasets.py`: remove `generate_retrieval_probes`,
which was test-only and not exported.
- Tests: drop the 4 `compute_bleu` asserts and the
`generate_retrieval_probes` block, and rename the two test functions to
match what they still cover.
- Evidence: a word search (`rg -c -w <name>`) across the repo (code,
tests, docs, wiki, CI) and the private consumer repos finds only each
definition, plus the removed test lines.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest -q tests/test_evals_metrics.py tests/test_evals_datasets.py tests/test_evals_cjk_tokenization.py tests/test_memory_eval.py tests/test_evals/test_comprehensive_benchmark_log_redaction.py tests/test_dataset_recall_runner.py tests/test_compression_fidelity_regression.py
53 passed in 4.70s
$ ruff check (touched files)
All checks passed!
$ mypy headroom/evals/metrics.py headroom/evals/datasets.py headroom/evals/memory/judge.py headroom/evals/comprehensive_benchmark.py --ignore-missing-imports
Success: no issues found in 4 source files
```

## Real Behavior Proof

- Environment: macOS, repo venv, branch checkout at a6148739, temp HOME
/ HEADROOM_WORKSPACE_DIR, offline.
- Exact command / steps: (1) shared smoke harness: start a real proxy
from this checkout against a local fake upstream and send one Anthropic
and one OpenAI request with a compressible tool result; (2) run the eval
paths that sit next to the removed helpers: the exported metrics, the
lm-eval default task list, the memory judge, and the zero-cost CCR
round-trip eval that CI runs (command below).
- Observed result: smoke requests return HTTP 200 with savings; the
remaining metrics, the default task list, `simple_judge` and the CCR
round-trip eval all run, and the CCR round-trip passes 10/10.
- Not tested: the paid LLM evals (they need API keys); none of them
calls a removed helper.

```text
$ bash smoke.sh <worktree>
branch: cleanup/providers-evals-dead-eval-helpers @ a6148739
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines:
   2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.

$ HOME=$(mktemp -d) python -c "<compute_f1 / compute_rouge_l / compute_exact_match; DEFAULT_TASKS; simple_judge; CompressionOnlyRunner CCR round-trip n=10>"
f1 0.8 rougeL 0.8 em True
default lm-eval tasks ['gsm8k']
simple_judge (3.0, 'Moderate token overlap (F1=0.50)')
CCR round-trip: 10/10 passed
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A, dead code only.
- Stable/default behavior changed: No.
- Kill switch / disable path: N/A.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.

The unused methods on the exported `EvalSuite` and
`CompressionEvaluator` classes (`add_case`, `to_jsonl`,
`evaluate_suite`) are public API, so they are deprecated in a separate
PR (`cleanup/providers-evals-deprecate-eval-suite-methods`), not removed
here.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 08:57:19 -07:00
GarmandClaude Opus 4.7 efd2ac1ca4 chore: renormalize line endings to LF
`.gitattributes` declares `*.py text eol=lf` and `*.sh text eol=lf`, but
74 files (73 .py, 1 .sh) are stored in the index with CRLF line endings,
violating that contract. Every macOS/Linux clone reports these files as
"modified" on fresh checkout because git's diff engine sees the stored
bytes don't match the attribute contract, even though the working tree
and index match byte-for-byte.

Running `git add --renormalize .` rewrites each affected blob so the
stored form matches the attribute declaration. No semantic changes —
every affected file's diff is "N insertions, N deletions" with inserts
and deletes being the same lines modulo line endings.

Follow-up commit adds `.git-blame-ignore-revs` so `git blame` / GitHub
blame skip this mechanical commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 15:33:30 +02:00
JerrettDavisandCopilot 38bf3e639c test: expand coverage across helper slices
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-23 07:39:52 -05:00