Files
headroom/tests/test_evals_provider_failures.py
364c59544c fix(evals): exclude provider failures from answer scores (#4503)
## Description

Do not count provider failures as matching benchmark answers. The
current CLI `BeforeAfterRunner`, `CompressionEvaluator` and
`BatchCompressionEvaluator` turn caught exceptions into `ERROR: ...`
response strings, then score those strings normally. Identical outages
receive perfect exact/F1 scores; an error mentioning the expected answer
can also receive ground-truth or keyword credit. Their result types
already have an `error` field, but these failures never populated it.
The CLI ground-truth mode can also send a failed provider diagnostic to
a judge or award it fallback credit.

In the core and batch runtime reproductions, one failed case and one
valid control incorrectly produce **2/2 passed (100%)** in each
evaluator. After the fix they produce the expected **1/2 passed (50%)**.
The successful control deliberately returns literal `ERROR:` text, so
the fix follows exception provenance rather than guessing from answer
text.

No issue linked: found by source inspection and verified against main
`17d38e5699b7d7c69ac6addcd762f39e42dd7f00`. Open/closed duplicate
screening found CJK metric and suite-deprecation work, but no equivalent
provider-failure fix.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Track actual provider exceptions separately from diagnostic strings in
all three evaluation paths, including direct and configured-proxy
dispatch in the current CLI runner.
- In CLI ground-truth mode, a failed provider call returns a failed
result without creating a judge or applying answer-matching fallback.
Successful judge and fallback behavior stays unchanged.
- If either required call fails, assign no pairwise
exact/F1/ROUGE/semantic credit and do not claim preserved accuracy.
- If the compressed call fails, diagnostic text cannot receive
ground-truth or keyword credit. A genuinely successful compressed answer
retains that independent credit if only the original call failed.
- Populate the existing `error` field with side-labelled exception
details. Retain response diagnostics, timings and token counts.
- Keep successful-case thresholds, result schemas, suite denominators
and compression behavior unchanged.
- Add 23 regressions covering original-only/compressed-only/both-call
failures, distinct exceptions, diagnostic text containing the expected
answer, literal `ERROR:` successful answers, serialization and
aggregation. Batch and CLI regressions use real Anthropic/OpenAI SDKs
over local HTTP, including successful judging and ordinary/BFCL
judge-failure fallback controls.

## Testing

- [ ] Unit tests pass (`pytest`) — focused suites passed; full
repository suite not claimed.
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`) — normal repository hook.
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
Exact main, unchanged native extension, new regression tests:
17 failed, 6 passed

Patched focused + adjacent suites:
29 passed

Normal commit hooks:
plugin sync / Ruff alignment / merge-conflict guard / Ruff lint+format / mypy / commitlint: Passed
```

Exact head: `6a16fe1ffb389d69e9dfe425d677bb7c589e8333`. The final-head
independent review and additional native selection are recorded below.

Focused command: `PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python -m pytest -p
pytest_asyncio.plugin tests/test_evals_provider_failures.py
tests/test_evals_core_deprecations.py
tests/test_evals_cjk_tokenization.py -q -o addopts=''`. This explicitly
loads pytest-asyncio and excludes unrelated globally installed pytest
plugins.

### Broader check limits and upload exception

The actual Python `_core` extension was built from unchanged native
source at the base revision using Rust 1.95 and the matching Xcode 26.4
SDK. Its SHA-256 is
`5c6d3df4a4c7cd620c06f02e14ef1d4f3922eeac483dec7d1693006ba8e494b9`. This
patch changes no Rust sources, Cargo configuration or dependencies.
Baseline Rust formatting and clippy completed, but the complete
workspace test build repeatedly hit the local 1.25 GiB disk-reserve
guard and was stopped. No full Rust-test, full pytest, complete pre-push
or hosted-CI pass is claimed.

The [installed pre-push
hook](https://github.com/headroomlabs-ai/headroom/blob/17d38e5699b7d7c69ac6addcd762f39e42dd7f00/scripts/install-git-hooks.sh)
documents a discretionary upload exception. Publication uses that
documented exception after normal commit hooks, real runtime proof and
independent review; required hosted CI and maintainer review remain
pending. This does not turn the incomplete local pre-push gate into a
pass.

## Real Behavior Proof

- Environment: macOS arm64, Python 3.13.5, actual source-built native
compressor; no model download or paid provider calls.
- Core path: public `CompressionEvaluator` constructor and suite API
with its documented `llm_fn` injection. One callback raises a controlled
timeout; the control returns a valid answer containing literal `ERROR:`
text. Real native compression runs in both cases.
- Batch path: public `BatchCompressionEvaluator` constructor and `run`;
actual Anthropic SDK calls a loopback HTTP server returning controlled
403/200 responses. No mocked SDK or compressor. The synthetic model uses
Headroom's documented production estimating tokenizer registration, so
external BPE-cache warmth cannot change the before/after counts. These
are estimates, not actual Anthropic token measurements.
- Exact command / steps: save the script below as `runtime_proof.py`,
then run in the source checkout with the built extension using `env -i
PATH="$PATH" PYTHONPATH=. HEADROOM_OFFLINE=1 HEADROOM_TELEMETRY=off
HEADROOM_BEACON=off DO_NOT_TRACK=1 HEADROOM_CCR_BACKEND=memory
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1
LITELLM_LOCAL_MODEL_COST_MAP=True
HEADROOM_WORKSPACE_DIR=<owned-temp-workspace>
HEADROOM_CONFIG_DIR=<owned-temp-config>
XDG_CACHE_HOME=<owned-temp-cache> python runtime_proof.py`.
- Isolation: no ambient credentials are inherited. Only the isolated
batch section sets `HEADROOM_OFFLINE=0`, because that benchmark
constructor rejects even loopback in offline mode; its endpoint and key
are explicitly synthetic. A process audit hook refuses external socket
connections and DNS. Telemetry and HuggingFace/Transformers downloads
remain off.
- Observed result: baseline **2/8** independent failure/success checks
pass; patched **8/8** pass. Both mixed suites change from false 100% to
correct 50%, failed-case F1 changes from 1 to 0, errors become visible,
and the valid literal-error control retains F1=1. Both execute real
compression with identical before/after input/output estimates: core
**2,817 → 2,168**, batch **3,220 → 2,478**. Four actual local SDK
requests are made per run.
- Not tested: live provider endpoints, model answer quality, other
evaluators, other platforms, production throughput, full repository
tests or hosted CI. No compression-quality or savings improvement is
claimed.

<details><summary>Standalone public-API runtime reproduction (no
mocks)</summary>

```python
"""Actual evaluators, native compression and loopback SDK; no mocks or models."""
import json
import os
import sys
import tempfile
import threading
import warnings
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path

# Fail closed before any unexpected outbound connection, including model lookup.
def local_only(event, args):
    if event == "socket.connect":
        address = args[1]
        if isinstance(address, tuple) and address[0] not in ("127.0.0.1", "::1"):
            raise RuntimeError(f"external connection refused: {address[0]}")
    if event == "socket.getaddrinfo" and args[0] not in ("127.0.0.1", "::1", "localhost"):
        raise RuntimeError(f"external DNS refused: {args[0]}")

sys.addaudithook(local_only)
from headroom.evals.core import CompressionEvaluator, EvalCase, EvalSuite
from headroom.evals.batch_compression_eval import (
    BatchCompressionEvaluator, BatchTestCase, BatchRequest, TestCategory,
)
from headroom.transforms.content_router import ContentRouterConfig
from headroom.tokenizers import EstimatingTokenCounter, register_tokenizer

# Pin the documented production tokenizer for this synthetic model; otherwise
# a newly warmed shared BPE cache changes reported token counts across runs.
register_tokenizer("claude-fixture", EstimatingTokenCounter(chars_per_token=3.5))

context = json.dumps([{"id": i, "answer": 42, "payload": "synthetic " * 10} for i in range(80)])

def provider(context, query):
    if query == "outage":
        raise TimeoutError("42 synthetic provider outage")
    return "ERROR: 42 is the expected fixture"

core = CompressionEvaluator(llm_fn=provider)
with warnings.catch_warnings():
    warnings.simplefilter("ignore", DeprecationWarning)
    core_result = core.evaluate_suite(EvalSuite("synthetic", [
        EvalCase(id="failed", context=context, query="outage", ground_truth="42"),
        EvalCase(id="literal", context=context, query="quote diagnostic", ground_truth="42"),
    ]))
assert all(r.compressed_tokens < r.original_tokens for r in core_result.results), "real native compression required"
print(json.dumps({"evaluator":"core", "passed":core_result.passed_cases,
                  "accuracy":core_result.accuracy_preservation_rate,
                  "f1":[r.f1_score for r in core_result.results],
                  "errors":[r.error for r in core_result.results],
                  "tokens":[[r.original_tokens,r.compressed_tokens] for r in core_result.results]}))

responses = [False, False, True, True]
requests = []
class Handler(BaseHTTPRequestHandler):
    def log_message(self, *args): pass
    def do_POST(self):
        requests.append(json.loads(self.rfile.read(int(self.headers["Content-Length"]))))
        success = responses.pop(0)
        body = ({"id":"msg_fixture", "type":"message", "role":"assistant", "model":"claude-fixture",
                 "content":[{"type":"text", "text":"ERROR: 42 is the expected fixture"}],
                 "stop_reason":"end_turn", "stop_sequence":None,
                 "usage":{"input_tokens":1,"output_tokens":1}}
                if success else {"type":"error", "error":{"type":"permission_error", "message":"42 synthetic outage"}})
        payload = json.dumps(body).encode()
        self.send_response(200 if success else 403)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length",str(len(payload)))
        self.end_headers()
        self.wfile.write(payload)

with tempfile.TemporaryDirectory(prefix="headroom-eval-public-api-") as root:
    for name in ("HEADROOM_WORKSPACE_DIR","HEADROOM_CONFIG_DIR","XDG_CACHE_HOME"):
        os.environ[name] = str(Path(root)/name.lower())
    server = ThreadingHTTPServer(("127.0.0.1",0),Handler)
    thread = threading.Thread(target=server.serve_forever,daemon=True)
    thread.start()
    # Only this isolated process enables benchmark client construction; endpoints
    # and credentials are explicitly synthetic, with an external-socket deny rule.
    os.environ.update(HEADROOM_OFFLINE="0",ANTHROPIC_API_KEY="synthetic-fixture",
                      ANTHROPIC_BASE_URL=f"http://127.0.0.1:{server.server_port}")
    os.environ.pop("ANTHROPIC_AUTH_TOKEN",None)
    batch = BatchCompressionEvaluator(model="claude-fixture",
                router_config=ContentRouterConfig(enable_kompress=False),use_semantic_similarity=False)
    case = BatchTestCase(id="fixture",category=TestCategory.JSON_EXTRACTION,
             request=BatchRequest(custom_id="fixture",messages=[{"role":"user","content":context}]),
             ground_truth="42",ground_truth_keywords=["42"])
    try:
        batch_result = batch.run([case,case])
        assert all(r.compressed_tokens < r.original_tokens for r in batch_result.results), "real native compression required"
        print(json.dumps({"evaluator":"batch", "passed":batch_result.passed_cases,
                          "accuracy":batch_result.accuracy_preservation_rate,
                          "f1":[r.f1_score for r in batch_result.results],
                          "errors":[r.error for r in batch_result.results],
                          "keywords":[r.keywords_found for r in batch_result.results],
                          "requests":len(requests),
                          "tokens":[[r.original_tokens,r.compressed_tokens] for r in batch_result.results]}))
    finally:
        batch._llm_client.close()
        server.shutdown(); server.server_close(); thread.join(timeout=2)

checks = [core_result.passed_cases==1,core_result.avg_f1_score==0.5,
          core_result.results[0].error is not None,core_result.results[1].error is None,
          batch_result.passed_cases==1,batch_result.avg_f1_score==0.5,
          batch_result.results[0].keywords_found==[],batch_result.results[1].error is None]
print(f"{sum(checks)}/{len(checks)} independent failure/success oracle checks passed")
assert all(checks)
```

</details>

## Runtime Rollout Safety

- Rollout-managed feature(s): none; existing evaluation helpers.
- Minimum rollout channel: unchanged.
- Stable/default behavior changed: failed provider calls no longer
produce successful accuracy claims or matching-answer credit.
- Kill switch / disable path: none added; callers still receive existing
result types and preserved diagnostics.
- Unsafe override required: none in product code. The isolated local
proof's benchmark-client offline setting is described above.
- Qualification impact: provider outages become failed benchmark cases;
suite denominators stay unchanged. Historical reports are not rewritten.
- Rollback path: revert this Python-only commit; no data migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — no new
API/configuration or schema.
- [x] My changes generate no new warnings in the executed focused suites
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — focused
suites passed, full suite unclaimed.
- [x] I did **not** edit `CHANGELOG.md` — generated by release-please.

## Additional Notes

AI-assisted contribution. One logical bug is fixed across all three
existing evaluation paths; this is not a new evaluator, scoring policy
for successful answers, status schema or provider feature.




### Current CLI runner runtime proof

The public `BeforeAfterRunner.evaluate_case` API executes real native
compression and actual Anthropic/OpenAI SDK calls against synthetic
loopback HTTP endpoints. Eight scenarios cover direct outages, either
side failing with configured proxy dispatch, literal `ERROR:` successful
answers, failed ground-truth calls, successful judging, and
ordinary/BFCL judge-failure fallback. The configured proxy endpoint is a
protocol fixture; this does not launch or qualify a full Headroom proxy.

Before the fix, 4/8 scenario oracles pass. After the fix, 8/8 pass. In
ground-truth outage mode, HTTP requests fall from two to one and the
judge is never created. Success and fallback controls retain their
existing verdicts. Native compression reports the same fixture
estimates, 2,817 → 2,168, before and after.

<details><summary>Standalone current CLI API reproduction</summary>

Save the following as `runner_runtime_proof.py` and use the same
isolated command/environment as the first runtime reproduction, changing
only the script filename. It rejects external DNS and socket connections
and supplies only explicit synthetic loopback credentials.

```python
"""Actual CLI-runner API + native compression + local SDK protocols only."""
import json
import os
import sys
import tempfile
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path


def local_only(event, args):
    if event == "socket.connect":
        address = args[1]
        if isinstance(address, tuple) and address[0] not in ("127.0.0.1", "::1"):
            raise RuntimeError("external socket refused")
    if event == "socket.getaddrinfo" and args[0] not in ("127.0.0.1", "::1", "localhost"):
        raise RuntimeError("external DNS refused")


sys.addaudithook(local_only)
from headroom.evals.core import EvalCase, EvalMode
from headroom.evals.runners.before_after import BeforeAfterRunner, LLMConfig
from headroom.transforms.content_router import ContentRouterConfig

responses = []
requests = []


class Handler(BaseHTTPRequestHandler):
    def log_message(self, *args):
        pass

    def do_POST(self):
        requests.append((self.path, json.loads(self.rfile.read(int(self.headers["Content-Length"])))))
        outcome = responses.pop(0)
        failed = isinstance(outcome, Exception)
        if failed:
            body = {"error": {"type": "permission_error", "message": str(outcome)}}
        elif "chat/completions" in self.path:
            body = {"id": "chat_fixture", "object": "chat.completion", "created": 0,
                    "model": "fixture", "choices": [{"index": 0,
                    "message": {"role": "assistant", "content": outcome}, "finish_reason": "stop"}]}
        else:
            body = {"id": "msg_fixture", "type": "message", "role": "assistant", "model": "claude-fixture",
                    "content": [{"type": "text", "text": outcome}], "stop_reason": "end_turn",
                    "stop_sequence": None, "usage": {"input_tokens": 1, "output_tokens": 1}}
        payload = json.dumps(body).encode()
        self.send_response(403 if failed else 200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(payload)))
        self.end_headers()
        self.wfile.write(payload)


checks = []
context = json.dumps([{"id": i, "answer": 42, "payload": "synthetic " * 10} for i in range(80)])
with tempfile.TemporaryDirectory(prefix="headroom-cli-eval-proof-") as root:
    for name in ("HEADROOM_WORKSPACE_DIR", "HEADROOM_CONFIG_DIR", "XDG_CACHE_HOME"):
        os.environ[name] = str(Path(root) / name.lower())
    server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
    thread = threading.Thread(target=server.serve_forever, daemon=True)
    thread.start()
    endpoint = f"http://127.0.0.1:{server.server_port}"
    os.environ.update(HEADROOM_OFFLINE="0", ANTHROPIC_API_KEY="synthetic-fixture",
                      ANTHROPIC_BASE_URL=endpoint, OPENAI_API_KEY="synthetic-fixture",
                      OPENAI_BASE_URL=endpoint + "/v1")
    os.environ.pop("ANTHROPIC_AUTH_TOKEN", None)
    # The configured proxy endpoint is a protocol fixture, not a Headroom proxy.
    scenarios = [
        ("direct_outage", False, EvalMode.BEFORE_AFTER, [TimeoutError("42"), TimeoutError("42")], False, False, 2, False),
        ("proxy_original_failed", True, EvalMode.BEFORE_AFTER, [TimeoutError("42"), "42"], False, True, 2, False),
        ("proxy_compressed_failed", True, EvalMode.BEFORE_AFTER, ["42", TimeoutError("42")], False, False, 2, False),
        ("literal_ERROR_success", True, EvalMode.BEFORE_AFTER, ["ERROR: 42", "ERROR: 42"], True, True, 2, False),
        ("ground_truth_outage", False, EvalMode.GROUND_TRUTH, [TimeoutError("42"), TimeoutError("judge unavailable")], False, False, 1, True),
        ("ground_truth_success", False, EvalMode.GROUND_TRUTH, ["42", "Reasoning: fixture\nScore: 5"], True, True, 2, False),
        ("ground_truth_fallback_success", False, EvalMode.GROUND_TRUTH, ["42", TimeoutError("judge unavailable")], True, True, 2, False),
        ("ground_truth_bfcl_fallback_success", False, EvalMode.GROUND_TRUTH, ["42", TimeoutError("judge unavailable")], True, True, 2, True),
    ]
    try:
        for name, proxy, mode, outcomes, expected_pass, expected_gt, expected_calls, bfcl in scenarios:
            responses[:] = outcomes
            before = len(requests)
            runner = BeforeAfterRunner(
                llm_config=LLMConfig(provider="anthropic", model="claude-fixture",
                    headroom_proxy_url=endpoint if proxy else None),
                router_config=ContentRouterConfig(enable_kompress=False),
                use_semantic_similarity=False,
            )
            case = EvalCase(id=name, context=context, query="What is the answer?", ground_truth="42")
            if bfcl:
                case.ground_truth = '[{"answer": {"value": [42]}}]'
                case.metadata["source"] = "BFCL"
            try:
                result = runner.evaluate_case(case, mode=mode)
            finally:
                runner._llm_client.close()
                if runner._proxy_client is not None:
                    runner._proxy_client.close()
            calls = len(requests) - before
            assert result.compressed_tokens < result.original_tokens, "real native compression required"
            passed = (result.accuracy_preserved is expected_pass
                      and result.contains_ground_truth is expected_gt
                      and (result.error is None) is expected_pass
                      and calls == expected_calls)
            checks.append(passed)
            print(json.dumps({"scenario": name, "oracle_pass": passed,
                "accuracy_preserved": result.accuracy_preserved, "f1": result.f1_score,
                "contains_ground_truth": result.contains_ground_truth, "error": result.error,
                "requests": calls, "judge_created": runner._judge_fn is not None,
                "tokens": [result.original_tokens, result.compressed_tokens]}))
    finally:
        server.shutdown(); server.server_close(); thread.join(timeout=2)
print(f"{sum(checks)}/{len(checks)} actual CLI-runner API oracle scenarios passed")
assert all(checks)

```

</details>


### Final independent validation

At final head `6a16fe1ffb389d69e9dfe425d677bb7c589e8333`, independent
review passes **35 unique focused tests** (23 provider regressions, 6
metrics, 5 CJK and 1 deprecation test). The documented focused command
above selects 29 of those tests. A separately written public-runner
oracle using the actual Ollama SDK passes **8/8 scenarios**, including
preservation of the existing offline refusal for a genuinely successful
call that would require an external judge. The Anthropic/OpenAI local
runtime proof passes **8/8 CLI scenarios**, and the earlier core/batch
proof passes **8/8 checks**. No paid or remote model endpoint is
contacted.

The final-head Makefile Python test-file selection passes **177 tests
with 4 optional embedding skips**, with one existing Anthropic
approximation warning. Whole-tree pinned Ruff0.17.0, normal commit hooks
and diff checks pass. Complete Rust workspace/full pytest/pre-push
success and required hosted CI remain unclaimed.


### Maintainer current-main requalification

Exact head `3089d374067aac942b60ab88f6b510cc8a208947`, upstream main
`e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`.

Mechanically merged upstream/main e4a60ed7 and re-reviewed
provider-exception provenance across core, batch and CLI before/after
plus ground-truth evaluation, including existing discussion. Failed
calls retain diagnostics without pairwise answer credit or judging;
successful literal ERROR text retains normal scoring, and a successful
compressed answer retains independent ground-truth credit when only the
original call failed. No contributor production logic was rewritten.

All 23 provider-failure regressions and 12 adjacent
metric/CJK/deprecation tests pass locally with the existing development
native extension. Ruff lint/format and whitespace checks pass. The
shared test-only Gemini streamed-request fixture repair was applied for
independently established main failures; all ten cases pass. No fresh
native-build or external model-provider proof is claimed. The previously
independently established main Anthropic 2.x temperature-argument
incompatibility remains separate from this fix; inherited baseline
failures are not attributed to the PR. Formal exact-head hosted
qualification is being handled by maintainers.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


Maintainer commit-message qualification repair: current head
29d67b902d. Corrected only the maintainer
main-sync commit message to Conventional Commit format. The complete
file tree is byte-identical to the previously tested head; exact-head
hosted checks are being refreshed. Nothing merged.


### Maintainer current-main requalification

Exact head `34587a4ed1704d45bd22f7930461b981f7b54c42`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

PR4503 current-main independent source qualification
Author rudycelekli; live head verified
29d67b902d.
Live authoritative upstream/main verified
c01cf07d94.
Clean mechanical merge only in G:/git/headroom-agent4503-c01 branch
agent4503-c01-review.
Prepared head 34587a4ed1
Prepared tree 6106931b4c
No conflicts, contributor rewrites, shared-ledger edits or publication.
Main-relative delta: 3 evaluator production files and 368-line provider
regression file. Inherited Gemini fixture repair is now already on main
and absent from delta.
Read actual current PR body, author discussion, all review bodies, fresh
inline Devin CLI finding, prior shared ledger4503 qualification at e4a.
Devin CLI gap is addressed by BeforeAfterRunner direct/proxy and
ground-truth paths with regression controls.
Exceptions tracked by provenance; diagnostic strings retained; pairwise
exact/F1/ROUGE become false/zero and semantic callback skipped if either
call fails. Failed compressed response cannot gain ground-truth or
keyword credit. Successful compressed response retains independent
credit if only original failed, but accuracy_preserved stays false.
Ground-truth failed provider skips judge and BFCL fallback. Successful
literal ERROR answers stay valid. Judge failure successful-answer
fallback unchanged.
Denominators deliberately unchanged: failed calls are denied answer
credit but remain suite cases for pass fraction and F1 averaging. Mixed
valid/failed suites score50%, not100%. Semantic averages omit None as
previously. No new failure policy was invented.
Fresh verification:35 passed in16.97s:
tests/test_evals_provider_failures.py
tests/test_evals_core_deprecations.py
tests/test_evals_cjk_tokenization.py tests/test_evals_metrics.py.
Used G:/git/headroom/.venv/Scripts/python.exe and
.codex/run-review-pytest.py, explicit pytest_asyncio plugin, plugin
autoload off, basetemp .codex/pytest-agent4503, controlled synthetic
loopback SDK fixtures; no live model/provider calls. Installed
Anthropic0.76.0. HEADROOM_REVIEW_NATIVE_PATH points to root freshly
built main .codex/bot4657-target/debug/_core.dll; this is NOT a native
build of the branch.
Ruff check and Ruff format --check all4 delta files pass; main-relative
git diff --check passes; owned tree clean.
No new source defect found. Source approval justified after
maintainer-managed exact-head hosted CI qualification. Previous
Anthropic2.5 temperature incompatibility was established on old main and
remains a separate qualification concern if repeated; this review makes
no current SDK2.5 compatibility claim and does not attribute inherited
provider errors to contributor answer quality.
Evidence: agent4503-c01-discussion.json, agent4503-c01-inline.json,
agent4503-c01-tests.log.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `519504a14b2646175de172948c06322cd145eea2`, upstream main
`2b0315b2d4b0d54947050bbdad2de9e89df06378`.

Requalified against upstream main
2b0315b2d4. All 71 integration patches
match the main delta after diff-location normalization; provider-failure
provenance, unchanged suite denominators and CLI scoring fixes remain
intact. The 166-case focused evaluator/adjacent/doctor suite passes with
the locked local Anthropic SDK. Exceptions receive no answer credit,
successful literal ERROR text remains an answer, and independent
compressed-answer ground-truth credit is preserved.

Fresh SDK 2.6.0 inspection confirms Messages.create rejects temperature
before HTTP dispatch. Both affected batch and BeforeAfterRunner
messages.create call ASTs are identical to current main and reproduce
that TypeError. This remains inherited SDK incompatibility, not a new
evaluator regression or grounds for a contributor change request.
Current author and Devin discussion checked; the CLI path is covered by
this fix.

Ruff lint/format passes for all 33 changed Python files; offline lock
and whitespace checks pass. The existing native artifact matches its
complete recorded input hashes and checksum; no fresh build or cloud
answer-quality qualification is claimed. No new source blocker found.
Maintainers will qualify the exact published head and distinguish any
shared-main SDK failure from PR regressions. Nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `2aa600e292437435f18d47c945c277a120edf992`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.

Clean integration with upstream/main
0ddaa56c21. All 17 integration patches
exactly match the prior-reviewed-main delta after diff-location
normalization; provider-failure scoring/diagnostic policy unchanged.
Fresh evaluator provider-failure/core-deprecation/doctor and incoming
Vertex/backend/CLI tests: 313 passed, 11 skipped, 1 warning with the
existing locked SDK environment. Twelve changed main Python files pass
Ruff lint/format; offline lock and whitespace pass.

Fresh Anthropic2.6.0 probe executes the reviewed current-main
BatchCompressionEvaluator and BeforeAfterRunner SDK methods with fake
credentials and HTTP/socket calls forbidden. Both fail with unexpected
keyword temperature before any HTTP attempt. Full method ASTs are
identical between current main and this candidate; hashes and
zero-network results are in pr4503-0dd-sdk26-proof.json. This is
demonstrated inherited SDK incompatibility, not a new provider-failure
regression or grounds for contributor changes. No temperature-policy
rewrite, SDK pin or compatibility skip was added. ERROR strings are not
scored as successful answers; CLI BeforeAfterRunner and ground-truth
paths are covered.

Author/Devin discussion read. Static source/test review found no
unrelated execution/workflow/dependency changes; tests use isolated
state and fake/loopback providers. Flagged inherited workflow-validation
job is excluded by unchanged workflow paths. Native DLL reused only
after complete tracked input-set/hash and artifact checksum
verification, no fresh build or cloud answer-quality claim. Formal
approval awaits exact-head hosted qualification, accurately
distinguishing shared SDK failures from regressions. Maintainers handle
checks; nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Signed-off-by: Rudy Celekli <rudy@gradiahq.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-09 12:21:15 -06:00

369 lines
13 KiB
Python

"""Provider failures must not become matching benchmark answers."""
from __future__ import annotations
import pytest
from headroom.evals.core import CompressionEvaluator, EvalCase, EvalMode, EvalSuite
@pytest.fixture(autouse=True)
def isolated_evaluation_runtime(monkeypatch, tmp_path):
monkeypatch.setenv("HEADROOM_OFFLINE", "1")
monkeypatch.setenv("HEADROOM_TELEMETRY", "off")
monkeypatch.setenv("DO_NOT_TRACK", "1")
monkeypatch.setenv("HF_HUB_OFFLINE", "1")
monkeypatch.setenv("TRANSFORMERS_OFFLINE", "1")
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path / "workspace"))
monkeypatch.setenv("HEADROOM_CONFIG_DIR", str(tmp_path / "config"))
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "memory")
def _case(case_id="failure"):
return EvalCase(
id=case_id,
context='{"answer": 42, "detail": "public synthetic fixture"}',
query="What is the answer?",
ground_truth="42",
)
def _responses(*outcomes):
values = iter(outcomes)
def llm(context, query):
outcome = next(values)
if isinstance(outcome, Exception):
raise outcome
return outcome
return llm
@pytest.mark.parametrize(
"outcomes,failed_sides",
[
((TimeoutError("42 timeout"), TimeoutError("42 timeout")), ["original", "compressed"]),
((TimeoutError("42 timeout"), "42"), ["original"]),
(("42", TimeoutError("42 timeout")), ["compressed"]),
((ValueError("first"), RuntimeError("second")), ["original", "compressed"]),
],
)
def test_failures_are_not_scored_as_matching_answers(outcomes, failed_sides):
semantic_calls = []
def similarity(original, compressed):
semantic_calls.append((original, compressed))
return 1.0
evaluator = CompressionEvaluator(
llm_fn=_responses(*outcomes), semantic_similarity_fn=similarity
)
result = evaluator.evaluate_case(_case())
assert not result.accuracy_preserved
assert result.exact_match is False
assert result.f1_score == 0.0
assert result.semantic_similarity is None
assert result.contains_ground_truth is ("compressed" not in failed_sides)
assert semantic_calls == []
assert result.error is not None
for side in failed_sides:
assert f"{side}: " in result.error
assert result.to_dict()["error"] == result.error
def test_literal_error_text_is_a_successful_answer_when_no_call_raised():
answer = "ERROR: 42 is the expected diagnostic"
evaluator = CompressionEvaluator(llm_fn=_responses(answer, answer))
result = evaluator.evaluate_case(_case())
assert result.accuracy_preserved
assert result.f1_score == 1.0
assert result.exact_match is True
assert result.contains_ground_truth is True
assert result.error is None
def test_failed_calls_do_not_inflate_suite_accuracy():
evaluator = CompressionEvaluator(
llm_fn=_responses("42", "42", TimeoutError("same"), TimeoutError("same"))
)
with pytest.warns(DeprecationWarning, match="evaluate_suite"):
result = evaluator.evaluate_suite(EvalSuite("mixed", [_case("ok"), _case("failed")]))
assert result.total_cases == 2
assert result.passed_cases == 1
assert result.failed_cases == 1
assert result.accuracy_preservation_rate == 0.5
assert result.avg_f1_score == 0.5
@pytest.fixture
def batch_endpoint(monkeypatch, tmp_path):
"""Actual SDK endpoint with controlled errors; no external model traffic."""
import json
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
pytest.importorskip("anthropic")
responses = []
requests = []
class Handler(BaseHTTPRequestHandler):
def log_message(self, *args):
pass
def do_POST(self):
requests.append(json.loads(self.rfile.read(int(self.headers["Content-Length"]))))
outcome = responses.pop(0)
failed = isinstance(outcome, Exception)
body = (
{"type": "error", "error": {"type": "permission_error", "message": str(outcome)}}
if failed
else {
"id": "msg_fixture",
"type": "message",
"role": "assistant",
"model": "claude-fixture",
"content": [{"type": "text", "text": outcome}],
"stop_reason": "end_turn",
"stop_sequence": None,
"usage": {"input_tokens": 1, "output_tokens": 1},
}
)
if not failed and "chat/completions" in self.path:
body = {
"id": "chat_fixture",
"object": "chat.completion",
"created": 0,
"model": "fixture",
"choices": [
{
"index": 0,
"message": {"role": "assistant", "content": outcome},
"finish_reason": "stop",
}
],
}
payload = json.dumps(body).encode()
self.send_response(403 if failed else 200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(payload)))
self.end_headers()
self.wfile.write(payload)
server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
monkeypatch.setenv("HEADROOM_OFFLINE", "0") # Batch constructor guards even loopback.
monkeypatch.setenv("HEADROOM_TELEMETRY", "off")
monkeypatch.setenv("DO_NOT_TRACK", "1")
monkeypatch.setenv("HF_HUB_OFFLINE", "1")
monkeypatch.setenv("TRANSFORMERS_OFFLINE", "1")
monkeypatch.setenv("ANTHROPIC_API_KEY", "synthetic-fixture")
monkeypatch.setenv("ANTHROPIC_BASE_URL", f"http://127.0.0.1:{server.server_port}")
monkeypatch.delenv("ANTHROPIC_AUTH_TOKEN", raising=False)
monkeypatch.setenv("OPENAI_API_KEY", "synthetic-fixture")
monkeypatch.setenv("OPENAI_BASE_URL", f"http://127.0.0.1:{server.server_port}/v1")
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path / "workspace"))
monkeypatch.setenv("HEADROOM_CONFIG_DIR", str(tmp_path / "config"))
try:
yield responses, requests
finally:
server.shutdown()
server.server_close()
thread.join(timeout=2)
def _batch_case():
import json
from headroom.evals.batch_compression_eval import BatchRequest, BatchTestCase, TestCategory
content = json.dumps([{"id": i, "answer": 42, "payload": "synthetic " * 10} for i in range(80)])
return BatchTestCase(
id="batch-failure",
category=TestCategory.JSON_EXTRACTION,
request=BatchRequest(
custom_id="batch-failure", messages=[{"role": "user", "content": content}]
),
ground_truth="42",
ground_truth_keywords=["42"],
)
@pytest.mark.parametrize(
"outcomes,compressed_failed",
[
([TimeoutError("42 timeout"), TimeoutError("42 timeout")], True),
([TimeoutError("42 timeout"), "42"], False),
(["42", TimeoutError("42 timeout")], True),
([ValueError("different first"), RuntimeError("distinct second")], True),
],
)
def test_batch_sdk_failures_are_not_answers(batch_endpoint, outcomes, compressed_failed):
from headroom.evals.batch_compression_eval import BatchCompressionEvaluator
from headroom.transforms.content_router import ContentRouterConfig
responses, requests = batch_endpoint
responses.extend(outcomes)
evaluator = BatchCompressionEvaluator(
model="claude-fixture",
router_config=ContentRouterConfig(enable_kompress=False),
use_semantic_similarity=False,
)
try:
result = evaluator.evaluate_case(_batch_case())
finally:
evaluator._llm_client.close()
assert len(requests) == 2
assert result.compressed_tokens < result.original_tokens # Real native compression ran.
assert not result.accuracy_preserved
assert result.exact_match is False
assert result.f1_score == 0.0
assert result.rouge_l_score == 0.0
assert result.semantic_similarity is None
assert result.ground_truth_match is (not compressed_failed)
assert result.keywords_found == ([] if compressed_failed else ["42"])
assert result.keywords_missing == (["42"] if compressed_failed else [])
assert result.error is not None
assert result.to_dict()["error"] == result.error
def test_batch_literal_error_answer_and_aggregate_are_preserved(batch_endpoint):
from headroom.evals.batch_compression_eval import BatchCompressionEvaluator
from headroom.transforms.content_router import ContentRouterConfig
responses, requests = batch_endpoint
responses.extend(["ERROR: 42", "ERROR: 42", TimeoutError("42"), TimeoutError("42")])
evaluator = BatchCompressionEvaluator(
model="claude-fixture",
router_config=ContentRouterConfig(enable_kompress=False),
use_semantic_similarity=False,
)
try:
result = evaluator.run([_batch_case(), _batch_case()])
finally:
evaluator._llm_client.close()
assert len(requests) == 4
assert result.passed_cases == 1
assert result.failed_cases == 1
assert result.accuracy_preservation_rate == 0.5
assert result.avg_f1_score == 0.5
assert result.results[0].error is None
assert result.results[0].f1_score == 1.0
def _runner_case():
return EvalCase(
id="runner-fixture",
context=_batch_case().request.messages[0]["content"],
query="What is the answer?",
ground_truth="42",
)
def _runner(proxy=False):
import os
from headroom.evals.runners.before_after import BeforeAfterRunner, LLMConfig
from headroom.transforms.content_router import ContentRouterConfig
return BeforeAfterRunner(
llm_config=LLMConfig(
provider="anthropic",
model="claude-fixture",
headroom_proxy_url=os.environ["ANTHROPIC_BASE_URL"] if proxy else None,
),
router_config=ContentRouterConfig(enable_kompress=False),
use_semantic_similarity=False,
)
def _close_runner(runner):
runner._llm_client.close()
if runner._proxy_client is not None:
runner._proxy_client.close()
@pytest.mark.parametrize("proxy", [False, True])
@pytest.mark.parametrize(
"outcomes,compressed_failed",
[
([TimeoutError("42"), TimeoutError("42")], True),
([TimeoutError("42"), "42"], False),
(["42", TimeoutError("42")], True),
],
)
def test_cli_runner_failures_are_not_answers(batch_endpoint, proxy, outcomes, compressed_failed):
responses, requests = batch_endpoint
responses.extend(outcomes)
runner = _runner(proxy)
try:
result = runner.evaluate_case(_runner_case())
finally:
_close_runner(runner)
assert len(requests) == 2
assert result.compressed_tokens < result.original_tokens
assert not result.accuracy_preserved
assert result.exact_match is False
assert result.f1_score == 0.0
assert result.semantic_similarity is None
assert result.contains_ground_truth is (not compressed_failed)
assert result.error is not None
@pytest.mark.parametrize("proxy", [False, True])
def test_cli_runner_literal_error_success_is_preserved(batch_endpoint, proxy):
responses, requests = batch_endpoint
responses.extend(["ERROR: 42", "ERROR: 42"])
runner = _runner(proxy)
try:
result = runner.run(EvalSuite("literal", [_runner_case()]))
finally:
_close_runner(runner)
assert len(requests) == 2
assert result.passed_cases == 1
assert result.accuracy_preservation_rate == 1.0
assert result.results[0].error is None
def test_cli_ground_truth_failure_skips_judge_and_bfcl_matching(batch_endpoint):
responses, requests = batch_endpoint
# The spare high-scoring judge reply must not be consumed after provider failure.
responses.extend([TimeoutError("42"), "Reasoning: fixture\nScore: 5"])
runner = _runner()
case = _runner_case()
case.ground_truth = '[{"answer": {"value": [42]}}]'
case.metadata["source"] = "BFCL"
try:
result = runner.evaluate_case(case, mode=EvalMode.GROUND_TRUTH)
finally:
_close_runner(runner)
assert len(requests) == 1
assert runner._judge_fn is None
assert not result.accuracy_preserved
assert result.contains_ground_truth is False
assert result.f1_score == 0.0
assert result.error.startswith("compressed: ")
@pytest.mark.parametrize("judge_fails,bfcl", [(False, False), (True, False), (True, True)])
def test_cli_ground_truth_success_and_judge_fallback_unchanged(batch_endpoint, judge_fails, bfcl):
responses, requests = batch_endpoint
responses.extend(
["42", TimeoutError("judge unavailable") if judge_fails else "Reasoning: fixture\nScore: 5"]
)
runner = _runner()
case = _runner_case()
if bfcl:
case.ground_truth = '[{"answer": {"value": [42]}}]'
case.metadata["source"] = "BFCL"
try:
result = runner.evaluate_case(case, mode=EvalMode.GROUND_TRUTH)
finally:
_close_runner(runner)
assert len(requests) == 2
assert result.accuracy_preserved
assert result.contains_ground_truth is True
assert result.f1_score == (0.0 if judge_fails else 1.0)
assert result.error is None