mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description Do not count provider failures as matching benchmark answers. The current CLI `BeforeAfterRunner`, `CompressionEvaluator` and `BatchCompressionEvaluator` turn caught exceptions into `ERROR: ...` response strings, then score those strings normally. Identical outages receive perfect exact/F1 scores; an error mentioning the expected answer can also receive ground-truth or keyword credit. Their result types already have an `error` field, but these failures never populated it. The CLI ground-truth mode can also send a failed provider diagnostic to a judge or award it fallback credit. In the core and batch runtime reproductions, one failed case and one valid control incorrectly produce **2/2 passed (100%)** in each evaluator. After the fix they produce the expected **1/2 passed (50%)**. The successful control deliberately returns literal `ERROR:` text, so the fix follows exception provenance rather than guessing from answer text. No issue linked: found by source inspection and verified against main `17d38e5699b7d7c69ac6addcd762f39e42dd7f00`. Open/closed duplicate screening found CJK metric and suite-deprecation work, but no equivalent provider-failure fix. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Track actual provider exceptions separately from diagnostic strings in all three evaluation paths, including direct and configured-proxy dispatch in the current CLI runner. - In CLI ground-truth mode, a failed provider call returns a failed result without creating a judge or applying answer-matching fallback. Successful judge and fallback behavior stays unchanged. - If either required call fails, assign no pairwise exact/F1/ROUGE/semantic credit and do not claim preserved accuracy. - If the compressed call fails, diagnostic text cannot receive ground-truth or keyword credit. A genuinely successful compressed answer retains that independent credit if only the original call failed. - Populate the existing `error` field with side-labelled exception details. Retain response diagnostics, timings and token counts. - Keep successful-case thresholds, result schemas, suite denominators and compression behavior unchanged. - Add 23 regressions covering original-only/compressed-only/both-call failures, distinct exceptions, diagnostic text containing the expected answer, literal `ERROR:` successful answers, serialization and aggregation. Batch and CLI regressions use real Anthropic/OpenAI SDKs over local HTTP, including successful judging and ordinary/BFCL judge-failure fallback controls. ## Testing - [ ] Unit tests pass (`pytest`) — focused suites passed; full repository suite not claimed. - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) — normal repository hook. - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text Exact main, unchanged native extension, new regression tests: 17 failed, 6 passed Patched focused + adjacent suites: 29 passed Normal commit hooks: plugin sync / Ruff alignment / merge-conflict guard / Ruff lint+format / mypy / commitlint: Passed ``` Exact head: `6a16fe1ffb389d69e9dfe425d677bb7c589e8333`. The final-head independent review and additional native selection are recorded below. Focused command: `PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python -m pytest -p pytest_asyncio.plugin tests/test_evals_provider_failures.py tests/test_evals_core_deprecations.py tests/test_evals_cjk_tokenization.py -q -o addopts=''`. This explicitly loads pytest-asyncio and excludes unrelated globally installed pytest plugins. ### Broader check limits and upload exception The actual Python `_core` extension was built from unchanged native source at the base revision using Rust 1.95 and the matching Xcode 26.4 SDK. Its SHA-256 is `5c6d3df4a4c7cd620c06f02e14ef1d4f3922eeac483dec7d1693006ba8e494b9`. This patch changes no Rust sources, Cargo configuration or dependencies. Baseline Rust formatting and clippy completed, but the complete workspace test build repeatedly hit the local 1.25 GiB disk-reserve guard and was stopped. No full Rust-test, full pytest, complete pre-push or hosted-CI pass is claimed. The [installed pre-push hook](https://github.com/headroomlabs-ai/headroom/blob/17d38e5699b7d7c69ac6addcd762f39e42dd7f00/scripts/install-git-hooks.sh) documents a discretionary upload exception. Publication uses that documented exception after normal commit hooks, real runtime proof and independent review; required hosted CI and maintainer review remain pending. This does not turn the incomplete local pre-push gate into a pass. ## Real Behavior Proof - Environment: macOS arm64, Python 3.13.5, actual source-built native compressor; no model download or paid provider calls. - Core path: public `CompressionEvaluator` constructor and suite API with its documented `llm_fn` injection. One callback raises a controlled timeout; the control returns a valid answer containing literal `ERROR:` text. Real native compression runs in both cases. - Batch path: public `BatchCompressionEvaluator` constructor and `run`; actual Anthropic SDK calls a loopback HTTP server returning controlled 403/200 responses. No mocked SDK or compressor. The synthetic model uses Headroom's documented production estimating tokenizer registration, so external BPE-cache warmth cannot change the before/after counts. These are estimates, not actual Anthropic token measurements. - Exact command / steps: save the script below as `runtime_proof.py`, then run in the source checkout with the built extension using `env -i PATH="$PATH" PYTHONPATH=. HEADROOM_OFFLINE=1 HEADROOM_TELEMETRY=off HEADROOM_BEACON=off DO_NOT_TRACK=1 HEADROOM_CCR_BACKEND=memory HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 LITELLM_LOCAL_MODEL_COST_MAP=True HEADROOM_WORKSPACE_DIR=<owned-temp-workspace> HEADROOM_CONFIG_DIR=<owned-temp-config> XDG_CACHE_HOME=<owned-temp-cache> python runtime_proof.py`. - Isolation: no ambient credentials are inherited. Only the isolated batch section sets `HEADROOM_OFFLINE=0`, because that benchmark constructor rejects even loopback in offline mode; its endpoint and key are explicitly synthetic. A process audit hook refuses external socket connections and DNS. Telemetry and HuggingFace/Transformers downloads remain off. - Observed result: baseline **2/8** independent failure/success checks pass; patched **8/8** pass. Both mixed suites change from false 100% to correct 50%, failed-case F1 changes from 1 to 0, errors become visible, and the valid literal-error control retains F1=1. Both execute real compression with identical before/after input/output estimates: core **2,817 → 2,168**, batch **3,220 → 2,478**. Four actual local SDK requests are made per run. - Not tested: live provider endpoints, model answer quality, other evaluators, other platforms, production throughput, full repository tests or hosted CI. No compression-quality or savings improvement is claimed. <details><summary>Standalone public-API runtime reproduction (no mocks)</summary> ```python """Actual evaluators, native compression and loopback SDK; no mocks or models.""" import json import os import sys import tempfile import threading import warnings from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from pathlib import Path # Fail closed before any unexpected outbound connection, including model lookup. def local_only(event, args): if event == "socket.connect": address = args[1] if isinstance(address, tuple) and address[0] not in ("127.0.0.1", "::1"): raise RuntimeError(f"external connection refused: {address[0]}") if event == "socket.getaddrinfo" and args[0] not in ("127.0.0.1", "::1", "localhost"): raise RuntimeError(f"external DNS refused: {args[0]}") sys.addaudithook(local_only) from headroom.evals.core import CompressionEvaluator, EvalCase, EvalSuite from headroom.evals.batch_compression_eval import ( BatchCompressionEvaluator, BatchTestCase, BatchRequest, TestCategory, ) from headroom.transforms.content_router import ContentRouterConfig from headroom.tokenizers import EstimatingTokenCounter, register_tokenizer # Pin the documented production tokenizer for this synthetic model; otherwise # a newly warmed shared BPE cache changes reported token counts across runs. register_tokenizer("claude-fixture", EstimatingTokenCounter(chars_per_token=3.5)) context = json.dumps([{"id": i, "answer": 42, "payload": "synthetic " * 10} for i in range(80)]) def provider(context, query): if query == "outage": raise TimeoutError("42 synthetic provider outage") return "ERROR: 42 is the expected fixture" core = CompressionEvaluator(llm_fn=provider) with warnings.catch_warnings(): warnings.simplefilter("ignore", DeprecationWarning) core_result = core.evaluate_suite(EvalSuite("synthetic", [ EvalCase(id="failed", context=context, query="outage", ground_truth="42"), EvalCase(id="literal", context=context, query="quote diagnostic", ground_truth="42"), ])) assert all(r.compressed_tokens < r.original_tokens for r in core_result.results), "real native compression required" print(json.dumps({"evaluator":"core", "passed":core_result.passed_cases, "accuracy":core_result.accuracy_preservation_rate, "f1":[r.f1_score for r in core_result.results], "errors":[r.error for r in core_result.results], "tokens":[[r.original_tokens,r.compressed_tokens] for r in core_result.results]})) responses = [False, False, True, True] requests = [] class Handler(BaseHTTPRequestHandler): def log_message(self, *args): pass def do_POST(self): requests.append(json.loads(self.rfile.read(int(self.headers["Content-Length"])))) success = responses.pop(0) body = ({"id":"msg_fixture", "type":"message", "role":"assistant", "model":"claude-fixture", "content":[{"type":"text", "text":"ERROR: 42 is the expected fixture"}], "stop_reason":"end_turn", "stop_sequence":None, "usage":{"input_tokens":1,"output_tokens":1}} if success else {"type":"error", "error":{"type":"permission_error", "message":"42 synthetic outage"}}) payload = json.dumps(body).encode() self.send_response(200 if success else 403) self.send_header("Content-Type", "application/json") self.send_header("Content-Length",str(len(payload))) self.end_headers() self.wfile.write(payload) with tempfile.TemporaryDirectory(prefix="headroom-eval-public-api-") as root: for name in ("HEADROOM_WORKSPACE_DIR","HEADROOM_CONFIG_DIR","XDG_CACHE_HOME"): os.environ[name] = str(Path(root)/name.lower()) server = ThreadingHTTPServer(("127.0.0.1",0),Handler) thread = threading.Thread(target=server.serve_forever,daemon=True) thread.start() # Only this isolated process enables benchmark client construction; endpoints # and credentials are explicitly synthetic, with an external-socket deny rule. os.environ.update(HEADROOM_OFFLINE="0",ANTHROPIC_API_KEY="synthetic-fixture", ANTHROPIC_BASE_URL=f"http://127.0.0.1:{server.server_port}") os.environ.pop("ANTHROPIC_AUTH_TOKEN",None) batch = BatchCompressionEvaluator(model="claude-fixture", router_config=ContentRouterConfig(enable_kompress=False),use_semantic_similarity=False) case = BatchTestCase(id="fixture",category=TestCategory.JSON_EXTRACTION, request=BatchRequest(custom_id="fixture",messages=[{"role":"user","content":context}]), ground_truth="42",ground_truth_keywords=["42"]) try: batch_result = batch.run([case,case]) assert all(r.compressed_tokens < r.original_tokens for r in batch_result.results), "real native compression required" print(json.dumps({"evaluator":"batch", "passed":batch_result.passed_cases, "accuracy":batch_result.accuracy_preservation_rate, "f1":[r.f1_score for r in batch_result.results], "errors":[r.error for r in batch_result.results], "keywords":[r.keywords_found for r in batch_result.results], "requests":len(requests), "tokens":[[r.original_tokens,r.compressed_tokens] for r in batch_result.results]})) finally: batch._llm_client.close() server.shutdown(); server.server_close(); thread.join(timeout=2) checks = [core_result.passed_cases==1,core_result.avg_f1_score==0.5, core_result.results[0].error is not None,core_result.results[1].error is None, batch_result.passed_cases==1,batch_result.avg_f1_score==0.5, batch_result.results[0].keywords_found==[],batch_result.results[1].error is None] print(f"{sum(checks)}/{len(checks)} independent failure/success oracle checks passed") assert all(checks) ``` </details> ## Runtime Rollout Safety - Rollout-managed feature(s): none; existing evaluation helpers. - Minimum rollout channel: unchanged. - Stable/default behavior changed: failed provider calls no longer produce successful accuracy claims or matching-answer credit. - Kill switch / disable path: none added; callers still receive existing result types and preserved diagnostics. - Unsafe override required: none in product code. The isolated local proof's benchmark-client offline setting is described above. - Qualification impact: provider outages become failed benchmark cases; suite denominators stay unchanged. Historical reports are not rewritten. - Rollback path: revert this Python-only commit; no data migration. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation — no new API/configuration or schema. - [x] My changes generate no new warnings in the executed focused suites - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes — focused suites passed, full suite unclaimed. - [x] I did **not** edit `CHANGELOG.md` — generated by release-please. ## Additional Notes AI-assisted contribution. One logical bug is fixed across all three existing evaluation paths; this is not a new evaluator, scoring policy for successful answers, status schema or provider feature. ### Current CLI runner runtime proof The public `BeforeAfterRunner.evaluate_case` API executes real native compression and actual Anthropic/OpenAI SDK calls against synthetic loopback HTTP endpoints. Eight scenarios cover direct outages, either side failing with configured proxy dispatch, literal `ERROR:` successful answers, failed ground-truth calls, successful judging, and ordinary/BFCL judge-failure fallback. The configured proxy endpoint is a protocol fixture; this does not launch or qualify a full Headroom proxy. Before the fix, 4/8 scenario oracles pass. After the fix, 8/8 pass. In ground-truth outage mode, HTTP requests fall from two to one and the judge is never created. Success and fallback controls retain their existing verdicts. Native compression reports the same fixture estimates, 2,817 → 2,168, before and after. <details><summary>Standalone current CLI API reproduction</summary> Save the following as `runner_runtime_proof.py` and use the same isolated command/environment as the first runtime reproduction, changing only the script filename. It rejects external DNS and socket connections and supplies only explicit synthetic loopback credentials. ```python """Actual CLI-runner API + native compression + local SDK protocols only.""" import json import os import sys import tempfile import threading from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from pathlib import Path def local_only(event, args): if event == "socket.connect": address = args[1] if isinstance(address, tuple) and address[0] not in ("127.0.0.1", "::1"): raise RuntimeError("external socket refused") if event == "socket.getaddrinfo" and args[0] not in ("127.0.0.1", "::1", "localhost"): raise RuntimeError("external DNS refused") sys.addaudithook(local_only) from headroom.evals.core import EvalCase, EvalMode from headroom.evals.runners.before_after import BeforeAfterRunner, LLMConfig from headroom.transforms.content_router import ContentRouterConfig responses = [] requests = [] class Handler(BaseHTTPRequestHandler): def log_message(self, *args): pass def do_POST(self): requests.append((self.path, json.loads(self.rfile.read(int(self.headers["Content-Length"]))))) outcome = responses.pop(0) failed = isinstance(outcome, Exception) if failed: body = {"error": {"type": "permission_error", "message": str(outcome)}} elif "chat/completions" in self.path: body = {"id": "chat_fixture", "object": "chat.completion", "created": 0, "model": "fixture", "choices": [{"index": 0, "message": {"role": "assistant", "content": outcome}, "finish_reason": "stop"}]} else: body = {"id": "msg_fixture", "type": "message", "role": "assistant", "model": "claude-fixture", "content": [{"type": "text", "text": outcome}], "stop_reason": "end_turn", "stop_sequence": None, "usage": {"input_tokens": 1, "output_tokens": 1}} payload = json.dumps(body).encode() self.send_response(403 if failed else 200) self.send_header("Content-Type", "application/json") self.send_header("Content-Length", str(len(payload))) self.end_headers() self.wfile.write(payload) checks = [] context = json.dumps([{"id": i, "answer": 42, "payload": "synthetic " * 10} for i in range(80)]) with tempfile.TemporaryDirectory(prefix="headroom-cli-eval-proof-") as root: for name in ("HEADROOM_WORKSPACE_DIR", "HEADROOM_CONFIG_DIR", "XDG_CACHE_HOME"): os.environ[name] = str(Path(root) / name.lower()) server = ThreadingHTTPServer(("127.0.0.1", 0), Handler) thread = threading.Thread(target=server.serve_forever, daemon=True) thread.start() endpoint = f"http://127.0.0.1:{server.server_port}" os.environ.update(HEADROOM_OFFLINE="0", ANTHROPIC_API_KEY="synthetic-fixture", ANTHROPIC_BASE_URL=endpoint, OPENAI_API_KEY="synthetic-fixture", OPENAI_BASE_URL=endpoint + "/v1") os.environ.pop("ANTHROPIC_AUTH_TOKEN", None) # The configured proxy endpoint is a protocol fixture, not a Headroom proxy. scenarios = [ ("direct_outage", False, EvalMode.BEFORE_AFTER, [TimeoutError("42"), TimeoutError("42")], False, False, 2, False), ("proxy_original_failed", True, EvalMode.BEFORE_AFTER, [TimeoutError("42"), "42"], False, True, 2, False), ("proxy_compressed_failed", True, EvalMode.BEFORE_AFTER, ["42", TimeoutError("42")], False, False, 2, False), ("literal_ERROR_success", True, EvalMode.BEFORE_AFTER, ["ERROR: 42", "ERROR: 42"], True, True, 2, False), ("ground_truth_outage", False, EvalMode.GROUND_TRUTH, [TimeoutError("42"), TimeoutError("judge unavailable")], False, False, 1, True), ("ground_truth_success", False, EvalMode.GROUND_TRUTH, ["42", "Reasoning: fixture\nScore: 5"], True, True, 2, False), ("ground_truth_fallback_success", False, EvalMode.GROUND_TRUTH, ["42", TimeoutError("judge unavailable")], True, True, 2, False), ("ground_truth_bfcl_fallback_success", False, EvalMode.GROUND_TRUTH, ["42", TimeoutError("judge unavailable")], True, True, 2, True), ] try: for name, proxy, mode, outcomes, expected_pass, expected_gt, expected_calls, bfcl in scenarios: responses[:] = outcomes before = len(requests) runner = BeforeAfterRunner( llm_config=LLMConfig(provider="anthropic", model="claude-fixture", headroom_proxy_url=endpoint if proxy else None), router_config=ContentRouterConfig(enable_kompress=False), use_semantic_similarity=False, ) case = EvalCase(id=name, context=context, query="What is the answer?", ground_truth="42") if bfcl: case.ground_truth = '[{"answer": {"value": [42]}}]' case.metadata["source"] = "BFCL" try: result = runner.evaluate_case(case, mode=mode) finally: runner._llm_client.close() if runner._proxy_client is not None: runner._proxy_client.close() calls = len(requests) - before assert result.compressed_tokens < result.original_tokens, "real native compression required" passed = (result.accuracy_preserved is expected_pass and result.contains_ground_truth is expected_gt and (result.error is None) is expected_pass and calls == expected_calls) checks.append(passed) print(json.dumps({"scenario": name, "oracle_pass": passed, "accuracy_preserved": result.accuracy_preserved, "f1": result.f1_score, "contains_ground_truth": result.contains_ground_truth, "error": result.error, "requests": calls, "judge_created": runner._judge_fn is not None, "tokens": [result.original_tokens, result.compressed_tokens]})) finally: server.shutdown(); server.server_close(); thread.join(timeout=2) print(f"{sum(checks)}/{len(checks)} actual CLI-runner API oracle scenarios passed") assert all(checks) ``` </details> ### Final independent validation At final head `6a16fe1ffb389d69e9dfe425d677bb7c589e8333`, independent review passes **35 unique focused tests** (23 provider regressions, 6 metrics, 5 CJK and 1 deprecation test). The documented focused command above selects 29 of those tests. A separately written public-runner oracle using the actual Ollama SDK passes **8/8 scenarios**, including preservation of the existing offline refusal for a genuinely successful call that would require an external judge. The Anthropic/OpenAI local runtime proof passes **8/8 CLI scenarios**, and the earlier core/batch proof passes **8/8 checks**. No paid or remote model endpoint is contacted. The final-head Makefile Python test-file selection passes **177 tests with 4 optional embedding skips**, with one existing Anthropic approximation warning. Whole-tree pinned Ruff0.17.0, normal commit hooks and diff checks pass. Complete Rust workspace/full pytest/pre-push success and required hosted CI remain unclaimed. ### Maintainer current-main requalification Exact head `3089d374067aac942b60ab88f6b510cc8a208947`, upstream main `e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`. Mechanically merged upstream/maine4a60ed7and re-reviewed provider-exception provenance across core, batch and CLI before/after plus ground-truth evaluation, including existing discussion. Failed calls retain diagnostics without pairwise answer credit or judging; successful literal ERROR text retains normal scoring, and a successful compressed answer retains independent ground-truth credit when only the original call failed. No contributor production logic was rewritten. All 23 provider-failure regressions and 12 adjacent metric/CJK/deprecation tests pass locally with the existing development native extension. Ruff lint/format and whitespace checks pass. The shared test-only Gemini streamed-request fixture repair was applied for independently established main failures; all ten cases pass. No fresh native-build or external model-provider proof is claimed. The previously independently established main Anthropic 2.x temperature-argument incompatibility remains separate from this fix; inherited baseline failures are not attributed to the PR. Formal exact-head hosted qualification is being handled by maintainers. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. Maintainer commit-message qualification repair: current head29d67b902d. Corrected only the maintainer main-sync commit message to Conventional Commit format. The complete file tree is byte-identical to the previously tested head; exact-head hosted checks are being refreshed. Nothing merged. ### Maintainer current-main requalification Exact head `34587a4ed1704d45bd22f7930461b981f7b54c42`, upstream main `c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`. PR4503 current-main independent source qualification Author rudycelekli; live head verified29d67b902d. Live authoritative upstream/main verifiedc01cf07d94. Clean mechanical merge only in G:/git/headroom-agent4503-c01 branch agent4503-c01-review. Prepared head34587a4ed1Prepared tree6106931b4cNo conflicts, contributor rewrites, shared-ledger edits or publication. Main-relative delta: 3 evaluator production files and 368-line provider regression file. Inherited Gemini fixture repair is now already on main and absent from delta. Read actual current PR body, author discussion, all review bodies, fresh inline Devin CLI finding, prior shared ledger4503 qualification at e4a. Devin CLI gap is addressed by BeforeAfterRunner direct/proxy and ground-truth paths with regression controls. Exceptions tracked by provenance; diagnostic strings retained; pairwise exact/F1/ROUGE become false/zero and semantic callback skipped if either call fails. Failed compressed response cannot gain ground-truth or keyword credit. Successful compressed response retains independent credit if only original failed, but accuracy_preserved stays false. Ground-truth failed provider skips judge and BFCL fallback. Successful literal ERROR answers stay valid. Judge failure successful-answer fallback unchanged. Denominators deliberately unchanged: failed calls are denied answer credit but remain suite cases for pass fraction and F1 averaging. Mixed valid/failed suites score50%, not100%. Semantic averages omit None as previously. No new failure policy was invented. Fresh verification:35 passed in16.97s: tests/test_evals_provider_failures.py tests/test_evals_core_deprecations.py tests/test_evals_cjk_tokenization.py tests/test_evals_metrics.py. Used G:/git/headroom/.venv/Scripts/python.exe and .codex/run-review-pytest.py, explicit pytest_asyncio plugin, plugin autoload off, basetemp .codex/pytest-agent4503, controlled synthetic loopback SDK fixtures; no live model/provider calls. Installed Anthropic0.76.0. HEADROOM_REVIEW_NATIVE_PATH points to root freshly built main .codex/bot4657-target/debug/_core.dll; this is NOT a native build of the branch. Ruff check and Ruff format --check all4 delta files pass; main-relative git diff --check passes; owned tree clean. No new source defect found. Source approval justified after maintainer-managed exact-head hosted CI qualification. Previous Anthropic2.5 temperature incompatibility was established on old main and remains a separate qualification concern if repeated; this review makes no current SDK2.5 compatibility claim and does not attribute inherited provider errors to contributor answer quality. Evidence: agent4503-c01-discussion.json, agent4503-c01-inline.json, agent4503-c01-tests.log. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. ### Maintainer current-main requalification Exact head `519504a14b2646175de172948c06322cd145eea2`, upstream main `2b0315b2d4b0d54947050bbdad2de9e89df06378`. Requalified against upstream main2b0315b2d4. All 71 integration patches match the main delta after diff-location normalization; provider-failure provenance, unchanged suite denominators and CLI scoring fixes remain intact. The 166-case focused evaluator/adjacent/doctor suite passes with the locked local Anthropic SDK. Exceptions receive no answer credit, successful literal ERROR text remains an answer, and independent compressed-answer ground-truth credit is preserved. Fresh SDK 2.6.0 inspection confirms Messages.create rejects temperature before HTTP dispatch. Both affected batch and BeforeAfterRunner messages.create call ASTs are identical to current main and reproduce that TypeError. This remains inherited SDK incompatibility, not a new evaluator regression or grounds for a contributor change request. Current author and Devin discussion checked; the CLI path is covered by this fix. Ruff lint/format passes for all 33 changed Python files; offline lock and whitespace checks pass. The existing native artifact matches its complete recorded input hashes and checksum; no fresh build or cloud answer-quality qualification is claimed. No new source blocker found. Maintainers will qualify the exact published head and distinguish any shared-main SDK failure from PR regressions. Nothing merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. ### Maintainer current-main requalification Exact head `2aa600e292437435f18d47c945c277a120edf992`, upstream main `0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`. Clean integration with upstream/main0ddaa56c21. All 17 integration patches exactly match the prior-reviewed-main delta after diff-location normalization; provider-failure scoring/diagnostic policy unchanged. Fresh evaluator provider-failure/core-deprecation/doctor and incoming Vertex/backend/CLI tests: 313 passed, 11 skipped, 1 warning with the existing locked SDK environment. Twelve changed main Python files pass Ruff lint/format; offline lock and whitespace pass. Fresh Anthropic2.6.0 probe executes the reviewed current-main BatchCompressionEvaluator and BeforeAfterRunner SDK methods with fake credentials and HTTP/socket calls forbidden. Both fail with unexpected keyword temperature before any HTTP attempt. Full method ASTs are identical between current main and this candidate; hashes and zero-network results are in pr4503-0dd-sdk26-proof.json. This is demonstrated inherited SDK incompatibility, not a new provider-failure regression or grounds for contributor changes. No temperature-policy rewrite, SDK pin or compatibility skip was added. ERROR strings are not scored as successful answers; CLI BeforeAfterRunner and ground-truth paths are covered. Author/Devin discussion read. Static source/test review found no unrelated execution/workflow/dependency changes; tests use isolated state and fake/loopback providers. Flagged inherited workflow-validation job is excluded by unchanged workflow paths. Native DLL reused only after complete tracked input-set/hash and artifact checksum verification, no fresh build or cloud answer-quality claim. Formal approval awaits exact-head hosted qualification, accurately distinguishing shared SDK failures from regressions. Maintainers handle checks; nothing merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. --------- Signed-off-by: Rudy Celekli <rudy@gradiahq.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com>
369 lines
13 KiB
Python
369 lines
13 KiB
Python
"""Provider failures must not become matching benchmark answers."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from headroom.evals.core import CompressionEvaluator, EvalCase, EvalMode, EvalSuite
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def isolated_evaluation_runtime(monkeypatch, tmp_path):
|
|
monkeypatch.setenv("HEADROOM_OFFLINE", "1")
|
|
monkeypatch.setenv("HEADROOM_TELEMETRY", "off")
|
|
monkeypatch.setenv("DO_NOT_TRACK", "1")
|
|
monkeypatch.setenv("HF_HUB_OFFLINE", "1")
|
|
monkeypatch.setenv("TRANSFORMERS_OFFLINE", "1")
|
|
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path / "workspace"))
|
|
monkeypatch.setenv("HEADROOM_CONFIG_DIR", str(tmp_path / "config"))
|
|
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "memory")
|
|
|
|
|
|
def _case(case_id="failure"):
|
|
return EvalCase(
|
|
id=case_id,
|
|
context='{"answer": 42, "detail": "public synthetic fixture"}',
|
|
query="What is the answer?",
|
|
ground_truth="42",
|
|
)
|
|
|
|
|
|
def _responses(*outcomes):
|
|
values = iter(outcomes)
|
|
|
|
def llm(context, query):
|
|
outcome = next(values)
|
|
if isinstance(outcome, Exception):
|
|
raise outcome
|
|
return outcome
|
|
|
|
return llm
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"outcomes,failed_sides",
|
|
[
|
|
((TimeoutError("42 timeout"), TimeoutError("42 timeout")), ["original", "compressed"]),
|
|
((TimeoutError("42 timeout"), "42"), ["original"]),
|
|
(("42", TimeoutError("42 timeout")), ["compressed"]),
|
|
((ValueError("first"), RuntimeError("second")), ["original", "compressed"]),
|
|
],
|
|
)
|
|
def test_failures_are_not_scored_as_matching_answers(outcomes, failed_sides):
|
|
semantic_calls = []
|
|
|
|
def similarity(original, compressed):
|
|
semantic_calls.append((original, compressed))
|
|
return 1.0
|
|
|
|
evaluator = CompressionEvaluator(
|
|
llm_fn=_responses(*outcomes), semantic_similarity_fn=similarity
|
|
)
|
|
result = evaluator.evaluate_case(_case())
|
|
assert not result.accuracy_preserved
|
|
assert result.exact_match is False
|
|
assert result.f1_score == 0.0
|
|
assert result.semantic_similarity is None
|
|
assert result.contains_ground_truth is ("compressed" not in failed_sides)
|
|
assert semantic_calls == []
|
|
assert result.error is not None
|
|
for side in failed_sides:
|
|
assert f"{side}: " in result.error
|
|
assert result.to_dict()["error"] == result.error
|
|
|
|
|
|
def test_literal_error_text_is_a_successful_answer_when_no_call_raised():
|
|
answer = "ERROR: 42 is the expected diagnostic"
|
|
evaluator = CompressionEvaluator(llm_fn=_responses(answer, answer))
|
|
result = evaluator.evaluate_case(_case())
|
|
assert result.accuracy_preserved
|
|
assert result.f1_score == 1.0
|
|
assert result.exact_match is True
|
|
assert result.contains_ground_truth is True
|
|
assert result.error is None
|
|
|
|
|
|
def test_failed_calls_do_not_inflate_suite_accuracy():
|
|
evaluator = CompressionEvaluator(
|
|
llm_fn=_responses("42", "42", TimeoutError("same"), TimeoutError("same"))
|
|
)
|
|
with pytest.warns(DeprecationWarning, match="evaluate_suite"):
|
|
result = evaluator.evaluate_suite(EvalSuite("mixed", [_case("ok"), _case("failed")]))
|
|
assert result.total_cases == 2
|
|
assert result.passed_cases == 1
|
|
assert result.failed_cases == 1
|
|
assert result.accuracy_preservation_rate == 0.5
|
|
assert result.avg_f1_score == 0.5
|
|
|
|
|
|
@pytest.fixture
|
|
def batch_endpoint(monkeypatch, tmp_path):
|
|
"""Actual SDK endpoint with controlled errors; no external model traffic."""
|
|
import json
|
|
import threading
|
|
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
|
|
|
pytest.importorskip("anthropic")
|
|
responses = []
|
|
requests = []
|
|
|
|
class Handler(BaseHTTPRequestHandler):
|
|
def log_message(self, *args):
|
|
pass
|
|
|
|
def do_POST(self):
|
|
requests.append(json.loads(self.rfile.read(int(self.headers["Content-Length"]))))
|
|
outcome = responses.pop(0)
|
|
failed = isinstance(outcome, Exception)
|
|
body = (
|
|
{"type": "error", "error": {"type": "permission_error", "message": str(outcome)}}
|
|
if failed
|
|
else {
|
|
"id": "msg_fixture",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"model": "claude-fixture",
|
|
"content": [{"type": "text", "text": outcome}],
|
|
"stop_reason": "end_turn",
|
|
"stop_sequence": None,
|
|
"usage": {"input_tokens": 1, "output_tokens": 1},
|
|
}
|
|
)
|
|
if not failed and "chat/completions" in self.path:
|
|
body = {
|
|
"id": "chat_fixture",
|
|
"object": "chat.completion",
|
|
"created": 0,
|
|
"model": "fixture",
|
|
"choices": [
|
|
{
|
|
"index": 0,
|
|
"message": {"role": "assistant", "content": outcome},
|
|
"finish_reason": "stop",
|
|
}
|
|
],
|
|
}
|
|
payload = json.dumps(body).encode()
|
|
self.send_response(403 if failed else 200)
|
|
self.send_header("Content-Type", "application/json")
|
|
self.send_header("Content-Length", str(len(payload)))
|
|
self.end_headers()
|
|
self.wfile.write(payload)
|
|
|
|
server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
|
|
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
|
thread.start()
|
|
monkeypatch.setenv("HEADROOM_OFFLINE", "0") # Batch constructor guards even loopback.
|
|
monkeypatch.setenv("HEADROOM_TELEMETRY", "off")
|
|
monkeypatch.setenv("DO_NOT_TRACK", "1")
|
|
monkeypatch.setenv("HF_HUB_OFFLINE", "1")
|
|
monkeypatch.setenv("TRANSFORMERS_OFFLINE", "1")
|
|
monkeypatch.setenv("ANTHROPIC_API_KEY", "synthetic-fixture")
|
|
monkeypatch.setenv("ANTHROPIC_BASE_URL", f"http://127.0.0.1:{server.server_port}")
|
|
monkeypatch.delenv("ANTHROPIC_AUTH_TOKEN", raising=False)
|
|
monkeypatch.setenv("OPENAI_API_KEY", "synthetic-fixture")
|
|
monkeypatch.setenv("OPENAI_BASE_URL", f"http://127.0.0.1:{server.server_port}/v1")
|
|
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path / "workspace"))
|
|
monkeypatch.setenv("HEADROOM_CONFIG_DIR", str(tmp_path / "config"))
|
|
try:
|
|
yield responses, requests
|
|
finally:
|
|
server.shutdown()
|
|
server.server_close()
|
|
thread.join(timeout=2)
|
|
|
|
|
|
def _batch_case():
|
|
import json
|
|
|
|
from headroom.evals.batch_compression_eval import BatchRequest, BatchTestCase, TestCategory
|
|
|
|
content = json.dumps([{"id": i, "answer": 42, "payload": "synthetic " * 10} for i in range(80)])
|
|
return BatchTestCase(
|
|
id="batch-failure",
|
|
category=TestCategory.JSON_EXTRACTION,
|
|
request=BatchRequest(
|
|
custom_id="batch-failure", messages=[{"role": "user", "content": content}]
|
|
),
|
|
ground_truth="42",
|
|
ground_truth_keywords=["42"],
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"outcomes,compressed_failed",
|
|
[
|
|
([TimeoutError("42 timeout"), TimeoutError("42 timeout")], True),
|
|
([TimeoutError("42 timeout"), "42"], False),
|
|
(["42", TimeoutError("42 timeout")], True),
|
|
([ValueError("different first"), RuntimeError("distinct second")], True),
|
|
],
|
|
)
|
|
def test_batch_sdk_failures_are_not_answers(batch_endpoint, outcomes, compressed_failed):
|
|
from headroom.evals.batch_compression_eval import BatchCompressionEvaluator
|
|
from headroom.transforms.content_router import ContentRouterConfig
|
|
|
|
responses, requests = batch_endpoint
|
|
responses.extend(outcomes)
|
|
evaluator = BatchCompressionEvaluator(
|
|
model="claude-fixture",
|
|
router_config=ContentRouterConfig(enable_kompress=False),
|
|
use_semantic_similarity=False,
|
|
)
|
|
try:
|
|
result = evaluator.evaluate_case(_batch_case())
|
|
finally:
|
|
evaluator._llm_client.close()
|
|
assert len(requests) == 2
|
|
assert result.compressed_tokens < result.original_tokens # Real native compression ran.
|
|
assert not result.accuracy_preserved
|
|
assert result.exact_match is False
|
|
assert result.f1_score == 0.0
|
|
assert result.rouge_l_score == 0.0
|
|
assert result.semantic_similarity is None
|
|
assert result.ground_truth_match is (not compressed_failed)
|
|
assert result.keywords_found == ([] if compressed_failed else ["42"])
|
|
assert result.keywords_missing == (["42"] if compressed_failed else [])
|
|
assert result.error is not None
|
|
assert result.to_dict()["error"] == result.error
|
|
|
|
|
|
def test_batch_literal_error_answer_and_aggregate_are_preserved(batch_endpoint):
|
|
from headroom.evals.batch_compression_eval import BatchCompressionEvaluator
|
|
from headroom.transforms.content_router import ContentRouterConfig
|
|
|
|
responses, requests = batch_endpoint
|
|
responses.extend(["ERROR: 42", "ERROR: 42", TimeoutError("42"), TimeoutError("42")])
|
|
evaluator = BatchCompressionEvaluator(
|
|
model="claude-fixture",
|
|
router_config=ContentRouterConfig(enable_kompress=False),
|
|
use_semantic_similarity=False,
|
|
)
|
|
try:
|
|
result = evaluator.run([_batch_case(), _batch_case()])
|
|
finally:
|
|
evaluator._llm_client.close()
|
|
assert len(requests) == 4
|
|
assert result.passed_cases == 1
|
|
assert result.failed_cases == 1
|
|
assert result.accuracy_preservation_rate == 0.5
|
|
assert result.avg_f1_score == 0.5
|
|
assert result.results[0].error is None
|
|
assert result.results[0].f1_score == 1.0
|
|
|
|
|
|
def _runner_case():
|
|
return EvalCase(
|
|
id="runner-fixture",
|
|
context=_batch_case().request.messages[0]["content"],
|
|
query="What is the answer?",
|
|
ground_truth="42",
|
|
)
|
|
|
|
|
|
def _runner(proxy=False):
|
|
import os
|
|
|
|
from headroom.evals.runners.before_after import BeforeAfterRunner, LLMConfig
|
|
from headroom.transforms.content_router import ContentRouterConfig
|
|
|
|
return BeforeAfterRunner(
|
|
llm_config=LLMConfig(
|
|
provider="anthropic",
|
|
model="claude-fixture",
|
|
headroom_proxy_url=os.environ["ANTHROPIC_BASE_URL"] if proxy else None,
|
|
),
|
|
router_config=ContentRouterConfig(enable_kompress=False),
|
|
use_semantic_similarity=False,
|
|
)
|
|
|
|
|
|
def _close_runner(runner):
|
|
runner._llm_client.close()
|
|
if runner._proxy_client is not None:
|
|
runner._proxy_client.close()
|
|
|
|
|
|
@pytest.mark.parametrize("proxy", [False, True])
|
|
@pytest.mark.parametrize(
|
|
"outcomes,compressed_failed",
|
|
[
|
|
([TimeoutError("42"), TimeoutError("42")], True),
|
|
([TimeoutError("42"), "42"], False),
|
|
(["42", TimeoutError("42")], True),
|
|
],
|
|
)
|
|
def test_cli_runner_failures_are_not_answers(batch_endpoint, proxy, outcomes, compressed_failed):
|
|
responses, requests = batch_endpoint
|
|
responses.extend(outcomes)
|
|
runner = _runner(proxy)
|
|
try:
|
|
result = runner.evaluate_case(_runner_case())
|
|
finally:
|
|
_close_runner(runner)
|
|
assert len(requests) == 2
|
|
assert result.compressed_tokens < result.original_tokens
|
|
assert not result.accuracy_preserved
|
|
assert result.exact_match is False
|
|
assert result.f1_score == 0.0
|
|
assert result.semantic_similarity is None
|
|
assert result.contains_ground_truth is (not compressed_failed)
|
|
assert result.error is not None
|
|
|
|
|
|
@pytest.mark.parametrize("proxy", [False, True])
|
|
def test_cli_runner_literal_error_success_is_preserved(batch_endpoint, proxy):
|
|
responses, requests = batch_endpoint
|
|
responses.extend(["ERROR: 42", "ERROR: 42"])
|
|
runner = _runner(proxy)
|
|
try:
|
|
result = runner.run(EvalSuite("literal", [_runner_case()]))
|
|
finally:
|
|
_close_runner(runner)
|
|
assert len(requests) == 2
|
|
assert result.passed_cases == 1
|
|
assert result.accuracy_preservation_rate == 1.0
|
|
assert result.results[0].error is None
|
|
|
|
|
|
def test_cli_ground_truth_failure_skips_judge_and_bfcl_matching(batch_endpoint):
|
|
responses, requests = batch_endpoint
|
|
# The spare high-scoring judge reply must not be consumed after provider failure.
|
|
responses.extend([TimeoutError("42"), "Reasoning: fixture\nScore: 5"])
|
|
runner = _runner()
|
|
case = _runner_case()
|
|
case.ground_truth = '[{"answer": {"value": [42]}}]'
|
|
case.metadata["source"] = "BFCL"
|
|
try:
|
|
result = runner.evaluate_case(case, mode=EvalMode.GROUND_TRUTH)
|
|
finally:
|
|
_close_runner(runner)
|
|
assert len(requests) == 1
|
|
assert runner._judge_fn is None
|
|
assert not result.accuracy_preserved
|
|
assert result.contains_ground_truth is False
|
|
assert result.f1_score == 0.0
|
|
assert result.error.startswith("compressed: ")
|
|
|
|
|
|
@pytest.mark.parametrize("judge_fails,bfcl", [(False, False), (True, False), (True, True)])
|
|
def test_cli_ground_truth_success_and_judge_fallback_unchanged(batch_endpoint, judge_fails, bfcl):
|
|
responses, requests = batch_endpoint
|
|
responses.extend(
|
|
["42", TimeoutError("judge unavailable") if judge_fails else "Reasoning: fixture\nScore: 5"]
|
|
)
|
|
runner = _runner()
|
|
case = _runner_case()
|
|
if bfcl:
|
|
case.ground_truth = '[{"answer": {"value": [42]}}]'
|
|
case.metadata["source"] = "BFCL"
|
|
try:
|
|
result = runner.evaluate_case(case, mode=EvalMode.GROUND_TRUTH)
|
|
finally:
|
|
_close_runner(runner)
|
|
assert len(requests) == 2
|
|
assert result.accuracy_preserved
|
|
assert result.contains_ground_truth is True
|
|
assert result.f1_score == (0.0 if judge_fails else 1.0)
|
|
assert result.error is None
|