Files
headroom/tests/test_cold_start_fast_pass.py
trs-80andClaude Opus 5 6e712672cb fix(anthropic): stop the spurious compaction warning and drop dead handler code (#3678)
## Description

Three small cleanups in `headroom/proxy/handlers/anthropic.py`, found
while reading the handler end to end. One is a real bug, two are dead
code.

**Spurious "system prompt compaction FAILED" on every request.** With
`HEADROOM_SYSTEM_COMPACT` enabled and no `ContentRouter` in the
pipeline, the handler read `_sys_modified`, a local that is only
assigned on the router branch. The resulting `UnboundLocalError` was
caught by the surrounding `except Exception` and logged as a compaction
failure on every request, even though nothing had failed. The flag is
now only read on the branch that sets it.

**No-op OpenAI fallback stub.** `ProxyConfig.fallback_enabled` /
`fallback_provider` gated a block in the Anthropic error path whose body
was two comments and a log line saying "Attempting fallback to OpenAI".
No conversion or retry ever existed, and nothing else in `headroom/`,
`tests/` or `docs/` references either field. Both fields and the block
are gone.

**Redundant pre-upstream semaphore releases.**
`handle_anthropic_messages` wraps its body in a `try/finally` that calls
the idempotent `_finalize_pre_upstream()` on every exit. Six explicit
calls placed directly before early `return`s (413, invalid JSON,
message-count 400, cache hit, security 403, Bedrock error) therefore did
nothing. They are removed. The calls that deliberately release *before*
a long await (streaming and backend handoffs) are kept, since those
change timing rather than correctness.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/handlers/anthropic.py`: move the `if _sys_modified:`
bookkeeping inside the router branch that assigns it, so the compaction
success/skip path no longer raises `UnboundLocalError` into the failure
log.
- `headroom/proxy/handlers/anthropic.py`: delete the
`fallback_enabled`/`fallback_provider` block in the upstream-error path;
it contained no executable fallback.
- `headroom/proxy/models.py`: delete the two unused `ProxyConfig`
fields. No loader, CLI flag, env var or doc maps to them.
- `headroom/proxy/handlers/anthropic.py`: drop six `await
_finalize_pre_upstream()` calls that sit immediately before an early
`return` already covered by the handler's `finally`.
- `tests/test_system_compaction_handler.py`: new
`test_system_compaction_without_content_router_is_not_a_failure`, which
fails on `main` with `cannot access local variable '_sys_modified'` and
passes here.
- `tests/test_system_compaction_handler.py`: also
`test_system_compaction_with_content_router_records_the_transform`,
which stubs the router lookup and `compact_system_prompt` so the success
branch (transform tag + debug log) is exercised; closes the codecov
partial on the moved `if _sys_modified:` line.
- `tests/test_anthropic_pre_upstream_backpressure.py`: extend the
early-exit semaphore test to the three validation paths it did not cover
(413, invalid JSON, message-count 400), so removing the explicit
releases is proven safe rather than assumed.
- `tests/test_anthropic_stage_timings.py`,
`tests/test_cache_mode_cold_start.py`,
`tests/test_cold_start_fast_pass.py`: stop setting the removed config
fields.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q tests/test_anthropic_pre_upstream_backpressure.py tests/test_system_compaction_handler.py \
    tests/test_anthropic_stage_timings.py tests/test_cache_mode_cold_start.py tests/test_cold_start_fast_pass.py
41 passed, 2 warnings in 7.6s

# same files with the new test against upstream/main's handler:
FAILED tests/test_system_compaction_handler.py::test_system_compaction_without_content_router_is_not_a_failure
E   'system prompt compaction FAILED: cannot access local variable '_sys_modified' where it is not associated with a value'

$ ruff check headroom/proxy/handlers/anthropic.py headroom/proxy/models.py tests/test_anthropic_pre_upstream_backpressure.py tests/test_system_compaction_handler.py
All checks passed!

$ mypy headroom/proxy/handlers/anthropic.py headroom/proxy/models.py
Success: no issues found in 2 source files

$ grep -rn 'fallback_enabled\|fallback_provider' headroom/ tests/ docs/
(no output)
```

## Real Behavior Proof

- Environment: macOS 25.6, Python 3.12 venv with the repo's compiled
`headroom._core`, fresh worktree of `upstream/main` @ `67eb910e`, the
three commits cherry-picked on top.
- Exact command / steps: read `handle_anthropic_messages` from the
semaphore acquire through the outer `finally` and confirmed
`_finalize_pre_upstream()` is idempotent and runs in `finally` on every
exit; grepped every reference to `fallback_enabled` /
`fallback_provider`; ran the five touched test files on the branch and
again with the new compaction test against `main`'s handler; ran `ruff
check` and `mypy` on the touched source files.
- Observed result: 40 tests pass on the branch. The new compaction test
reproduces the spurious failure log on `main` and passes here. No
remaining references to the deleted config fields. Lint and type check
clean on the touched files.
- Not tested: no live proxy traffic. The compaction fix only changes
which branch reads a local, the fallback removal deletes code with no
behavior, and the semaphore change is covered by the extended early-exit
test rather than by observing a real backlog. `ruff format --check`
flags `anthropic.py` on `main` already; this PR does not reformat
unrelated lines.

## Runtime Rollout Safety

- Rollout-managed feature(s): `HEADROOM_SYSTEM_COMPACT` (env-gated, off
by default). No new gate is introduced.
- Minimum rollout channel: n/a. The fix only affects the log line and
`transforms_applied` bookkeeping when compaction is enabled and no
router is present; upstream requests were already unaffected.
- Stable/default behavior changed: no. With the env var unset, none of
the changed lines execute. `ProxyConfig(fallback_enabled=...)` would now
raise `TypeError`, but no code path, doc or config loader ever passed
it.
- Kill switch / disable path: unset `HEADROOM_SYSTEM_COMPACT` restores
the pre-existing behavior for the compaction path; the other two changes
remove dead code and have no runtime switch.
- Unsafe override required: no.
- Qualification impact: none. No compression, cache or provider behavior
changes.
- Rollback path: `git revert` of the three commits; they touch seven
files and no persisted state or schema.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

A fourth commit that unified the budget and rate-limit preflight across
all provider handlers was deliberately left out. Open PR #3579 already
enforces the spend budget on the OpenAI and Gemini routes with two
approvals, and #3375 targets the same issue #3374. The remaining
rate-limit-key unification will be proposed separately once #3579 lands.

Documentation checkbox is unticked because `fallback_enabled` /
`fallback_provider` appear in no documentation and no user-facing
behavior changed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-24 10:59:20 -05:00

295 lines
9.8 KiB
Python

"""Cold-start fast pass: when background compression defers a cold-start-large
request, the handler still runs the pipeline synchronously with
skip_kompress=True so the FORWARDED (and therefore provider-cached,
byte-identically frozen) form carries the cheap savings. Only the Kompress ML
stage stays deferred to the background job."""
from __future__ import annotations
import json
from types import SimpleNamespace
from unittest.mock import MagicMock
import anyio
from fastapi import Request
from headroom.config import TransformResult
from headroom.proxy.handlers.anthropic import AnthropicHandlerMixin
from headroom.proxy.models import ProxyConfig
_COMPRESSED_TEXT = "compressed tool output"
class _DummyTokenizer:
def count_messages(self, messages) -> int:
return json.dumps(messages).count(" ") + 1
def count_text(self, text: str) -> int:
return max(1, text.count(" ") + 1)
class _DummyMetrics:
async def record_request(self, **kwargs):
return None
async def record_stage_timings(self, path, timings):
return None
async def record_failed(self, **kwargs):
return None
def record_compression_failed(self, reason: str) -> None:
return None
async def record_rate_limited(self, **kwargs):
return None
class _ResponseStub:
status_code = 200
headers: dict[str, str] = {}
content = b'{"id":"msg_1","type":"message","role":"assistant","content":[],"usage":{"input_tokens":1,"output_tokens":1}}'
def json(self):
return {
"id": "msg_1",
"type": "message",
"role": "assistant",
"content": [],
"usage": {"input_tokens": 1, "output_tokens": 1},
}
class _RecordingBackgroundCompressor:
def __init__(self) -> None:
self.enqueued: list[tuple[str, object, object]] = []
def enqueue(self, key, compress, store) -> bool:
self.enqueued.append((key, compress, store))
return True
def _fake_pipeline_apply(messages, model, **kwargs):
compressed = []
for msg in messages:
new = dict(msg)
if msg.get("role") == "user" and isinstance(msg.get("content"), list):
new["content"] = [
{**part, "content": _COMPRESSED_TEXT}
if isinstance(part, dict) and part.get("type") == "tool_result"
else part
for part in msg["content"]
]
compressed.append(new)
return TransformResult(
messages=compressed,
tokens_before=1000,
tokens_after=100,
transforms_applied=["read_lifecycle:stale:test.py"],
)
class _DummyAnthropicHandler(AnthropicHandlerMixin):
ANTHROPIC_API_URL = "https://api.anthropic.com"
def __init__(self) -> None:
self.rate_limiter = None
self.metrics = _DummyMetrics()
self.config = ProxyConfig(
optimize=True,
image_optimize=False,
retry_max_attempts=1,
retry_base_delay_ms=1,
retry_max_delay_ms=1,
connect_timeout_seconds=10,
mode="token",
cache_enabled=False,
rate_limit_enabled=False,
prefix_freeze_enabled=False,
memory_enabled=False,
)
self.usage_reporter = None
self.anthropic_provider = SimpleNamespace(get_context_limit=lambda model: 200_000)
self.anthropic_pipeline = SimpleNamespace(apply=MagicMock(side_effect=_fake_pipeline_apply))
self.anthropic_backend = None
self.cost_tracker = None
self.memory_handler = None
self.cache = None
self.security = None
self.ccr_context_tracker = None
self.ccr_injector = None
self.ccr_response_handler = None
self.ccr_feedback = None
self.ccr_batch_processor = None
self.ccr_mcp_server = None
self.traffic_learner = None
self.tool_injector = None
self.read_lifecycle_manager = None
self.logger = SimpleNamespace(log=lambda *a, **k: None)
self.request_logger = self.logger
self.usage_observer = None
self.image_compressor = None
self.session_tracker_store = SimpleNamespace(
compute_session_id=lambda *a, **k: "sess-1",
get_or_create=lambda *a, **k: SimpleNamespace(
get_frozen_message_count=lambda: 0,
get_last_original_messages=lambda: [],
get_last_forwarded_messages=lambda: [],
record_request=lambda *a, **k: None,
),
resolve_tracker=lambda *a, **k: SimpleNamespace(
get_frozen_message_count=lambda: 0,
get_last_original_messages=lambda: [],
get_last_forwarded_messages=lambda: [],
record_request=lambda *a, **k: None,
),
)
# Cold-start deferral wiring under test.
self._background_compression_enabled = True
self._background_compression_min_tokens = 1
self._background_compressor = _RecordingBackgroundCompressor()
self.executor_calls: list[float] = []
async def _run_compression_in_executor(self, fn, timeout):
self.executor_calls.append(timeout)
return fn()
async def _next_request_id(self) -> str:
return "req-fastpass-test"
def _extract_tags(self, headers):
return {}
async def _retry_request(self, method, url, headers, body, **_kwargs):
self.captured_body = body
return _ResponseStub()
def _get_compression_cache(self, session_id):
self.comp_cache_updates: list[tuple] = getattr(self, "comp_cache_updates", [])
return SimpleNamespace(
apply_cached=lambda m: m,
compute_frozen_count=lambda m: 0,
mark_stable_from_messages=lambda *a, **k: None,
should_defer_compression=lambda h: False,
mark_stable=lambda h: None,
content_hash=lambda c: "h",
update_from_result=lambda *a: self.comp_cache_updates.append(a),
_cache={},
_stable_hashes=set(),
)
def _build_request(body: dict) -> Request:
payload = json.dumps(body).encode("utf-8")
async def receive():
return {"type": "http.request", "body": payload, "more_body": False}
scope = {
"type": "http",
"asgi": {"version": "3.0"},
"http_version": "1.1",
"method": "POST",
"scheme": "https",
"path": "/v1/messages",
"raw_path": b"/v1/messages",
"query_string": b"",
"headers": [(b"authorization", b"Bearer sk-ant-api-test")],
"client": ("127.0.0.1", 12345),
"server": ("testserver", 443),
}
return Request(scope, receive)
def test_cold_start_runs_fast_pass_and_defers_only_kompress(monkeypatch):
import headroom.tokenizers as _tk
monkeypatch.setattr(_tk, "get_tokenizer", lambda model: _DummyTokenizer())
handler = _DummyAnthropicHandler()
request = _build_request(
{
"model": "claude-3-5-sonnet-latest",
"messages": [
{
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": "toolu_1",
"content": "verbose stale tool output " * 200,
}
],
},
],
}
)
anyio.run(handler.handle_anthropic_messages, request)
# The fast pass ran synchronously with the ML stage disabled.
assert handler.executor_calls, "fast pass never ran through the executor"
sync_calls = [
c for c in handler.anthropic_pipeline.apply.call_args_list if c.kwargs.get("skip_kompress")
]
assert len(sync_calls) == 1, "expected exactly one synchronous skip_kompress pass"
# The full pipeline (kompress included) went to the background queue,
# keyed against the ORIGINAL messages for content-hash reuse.
assert len(handler._background_compressor.enqueued) == 1
_key, bg_compress, _store = handler._background_compressor.enqueued[0]
bg_compress()
bg_calls = [
c
for c in handler.anthropic_pipeline.apply.call_args_list
if not c.kwargs.get("skip_kompress")
]
assert len(bg_calls) == 1, "background job must run the full pipeline"
# The FORWARDED body carries the fast-pass form — that is what the
# provider caches and the byte-identical freeze locks in.
forwarded = handler.captured_body["messages"]
assert forwarded[0]["content"][0]["content"] == _COMPRESSED_TEXT
# Fast-pass results were stored in the compression cache.
assert handler.comp_cache_updates
def test_fast_pass_failure_falls_back_to_full_deferral(monkeypatch):
import headroom.tokenizers as _tk
monkeypatch.setattr(_tk, "get_tokenizer", lambda model: _DummyTokenizer())
handler = _DummyAnthropicHandler()
async def _boom(fn, timeout):
raise TimeoutError("fast pass exceeded budget")
handler._run_compression_in_executor = _boom # type: ignore[method-assign]
original_text = "verbose stale tool output " * 200
request = _build_request(
{
"model": "claude-3-5-sonnet-latest",
"messages": [
{
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": "toolu_1",
"content": original_text,
}
],
},
],
}
)
anyio.run(handler.handle_anthropic_messages, request)
# Fail-open: original messages forwarded, background job still queued.
forwarded = handler.captured_body["messages"]
assert forwarded[0]["content"][0]["content"] == original_text
assert len(handler._background_compressor.enqueued) == 1