mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description
Three small cleanups in `headroom/proxy/handlers/anthropic.py`, found
while reading the handler end to end. One is a real bug, two are dead
code.
**Spurious "system prompt compaction FAILED" on every request.** With
`HEADROOM_SYSTEM_COMPACT` enabled and no `ContentRouter` in the
pipeline, the handler read `_sys_modified`, a local that is only
assigned on the router branch. The resulting `UnboundLocalError` was
caught by the surrounding `except Exception` and logged as a compaction
failure on every request, even though nothing had failed. The flag is
now only read on the branch that sets it.
**No-op OpenAI fallback stub.** `ProxyConfig.fallback_enabled` /
`fallback_provider` gated a block in the Anthropic error path whose body
was two comments and a log line saying "Attempting fallback to OpenAI".
No conversion or retry ever existed, and nothing else in `headroom/`,
`tests/` or `docs/` references either field. Both fields and the block
are gone.
**Redundant pre-upstream semaphore releases.**
`handle_anthropic_messages` wraps its body in a `try/finally` that calls
the idempotent `_finalize_pre_upstream()` on every exit. Six explicit
calls placed directly before early `return`s (413, invalid JSON,
message-count 400, cache hit, security 403, Bedrock error) therefore did
nothing. They are removed. The calls that deliberately release *before*
a long await (streaming and backend handoffs) are kept, since those
change timing rather than correctness.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/anthropic.py`: move the `if _sys_modified:`
bookkeeping inside the router branch that assigns it, so the compaction
success/skip path no longer raises `UnboundLocalError` into the failure
log.
- `headroom/proxy/handlers/anthropic.py`: delete the
`fallback_enabled`/`fallback_provider` block in the upstream-error path;
it contained no executable fallback.
- `headroom/proxy/models.py`: delete the two unused `ProxyConfig`
fields. No loader, CLI flag, env var or doc maps to them.
- `headroom/proxy/handlers/anthropic.py`: drop six `await
_finalize_pre_upstream()` calls that sit immediately before an early
`return` already covered by the handler's `finally`.
- `tests/test_system_compaction_handler.py`: new
`test_system_compaction_without_content_router_is_not_a_failure`, which
fails on `main` with `cannot access local variable '_sys_modified'` and
passes here.
- `tests/test_system_compaction_handler.py`: also
`test_system_compaction_with_content_router_records_the_transform`,
which stubs the router lookup and `compact_system_prompt` so the success
branch (transform tag + debug log) is exercised; closes the codecov
partial on the moved `if _sys_modified:` line.
- `tests/test_anthropic_pre_upstream_backpressure.py`: extend the
early-exit semaphore test to the three validation paths it did not cover
(413, invalid JSON, message-count 400), so removing the explicit
releases is proven safe rather than assumed.
- `tests/test_anthropic_stage_timings.py`,
`tests/test_cache_mode_cold_start.py`,
`tests/test_cold_start_fast_pass.py`: stop setting the removed config
fields.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest -q tests/test_anthropic_pre_upstream_backpressure.py tests/test_system_compaction_handler.py \
tests/test_anthropic_stage_timings.py tests/test_cache_mode_cold_start.py tests/test_cold_start_fast_pass.py
41 passed, 2 warnings in 7.6s
# same files with the new test against upstream/main's handler:
FAILED tests/test_system_compaction_handler.py::test_system_compaction_without_content_router_is_not_a_failure
E 'system prompt compaction FAILED: cannot access local variable '_sys_modified' where it is not associated with a value'
$ ruff check headroom/proxy/handlers/anthropic.py headroom/proxy/models.py tests/test_anthropic_pre_upstream_backpressure.py tests/test_system_compaction_handler.py
All checks passed!
$ mypy headroom/proxy/handlers/anthropic.py headroom/proxy/models.py
Success: no issues found in 2 source files
$ grep -rn 'fallback_enabled\|fallback_provider' headroom/ tests/ docs/
(no output)
```
## Real Behavior Proof
- Environment: macOS 25.6, Python 3.12 venv with the repo's compiled
`headroom._core`, fresh worktree of `upstream/main` @ `67eb910e`, the
three commits cherry-picked on top.
- Exact command / steps: read `handle_anthropic_messages` from the
semaphore acquire through the outer `finally` and confirmed
`_finalize_pre_upstream()` is idempotent and runs in `finally` on every
exit; grepped every reference to `fallback_enabled` /
`fallback_provider`; ran the five touched test files on the branch and
again with the new compaction test against `main`'s handler; ran `ruff
check` and `mypy` on the touched source files.
- Observed result: 40 tests pass on the branch. The new compaction test
reproduces the spurious failure log on `main` and passes here. No
remaining references to the deleted config fields. Lint and type check
clean on the touched files.
- Not tested: no live proxy traffic. The compaction fix only changes
which branch reads a local, the fallback removal deletes code with no
behavior, and the semaphore change is covered by the extended early-exit
test rather than by observing a real backlog. `ruff format --check`
flags `anthropic.py` on `main` already; this PR does not reformat
unrelated lines.
## Runtime Rollout Safety
- Rollout-managed feature(s): `HEADROOM_SYSTEM_COMPACT` (env-gated, off
by default). No new gate is introduced.
- Minimum rollout channel: n/a. The fix only affects the log line and
`transforms_applied` bookkeeping when compaction is enabled and no
router is present; upstream requests were already unaffected.
- Stable/default behavior changed: no. With the env var unset, none of
the changed lines execute. `ProxyConfig(fallback_enabled=...)` would now
raise `TypeError`, but no code path, doc or config loader ever passed
it.
- Kill switch / disable path: unset `HEADROOM_SYSTEM_COMPACT` restores
the pre-existing behavior for the compaction path; the other two changes
remove dead code and have no runtime switch.
- Unsafe override required: no.
- Qualification impact: none. No compression, cache or provider behavior
changes.
- Rollback path: `git revert` of the three commits; they touch seven
files and no persisted state or schema.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
A fourth commit that unified the budget and rate-limit preflight across
all provider handlers was deliberately left out. Open PR #3579 already
enforces the spend budget on the OpenAI and Gemini routes with two
approvals, and #3375 targets the same issue #3374. The remaining
rate-limit-key unification will be proposed separately once #3579 lands.
Documentation checkbox is unticked because `fallback_enabled` /
`fallback_provider` appear in no documentation and no user-facing
behavior changed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
295 lines
9.8 KiB
Python
295 lines
9.8 KiB
Python
"""Cold-start fast pass: when background compression defers a cold-start-large
|
|
request, the handler still runs the pipeline synchronously with
|
|
skip_kompress=True so the FORWARDED (and therefore provider-cached,
|
|
byte-identically frozen) form carries the cheap savings. Only the Kompress ML
|
|
stage stays deferred to the background job."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
from types import SimpleNamespace
|
|
from unittest.mock import MagicMock
|
|
|
|
import anyio
|
|
from fastapi import Request
|
|
|
|
from headroom.config import TransformResult
|
|
from headroom.proxy.handlers.anthropic import AnthropicHandlerMixin
|
|
from headroom.proxy.models import ProxyConfig
|
|
|
|
_COMPRESSED_TEXT = "compressed tool output"
|
|
|
|
|
|
class _DummyTokenizer:
|
|
def count_messages(self, messages) -> int:
|
|
return json.dumps(messages).count(" ") + 1
|
|
|
|
def count_text(self, text: str) -> int:
|
|
return max(1, text.count(" ") + 1)
|
|
|
|
|
|
class _DummyMetrics:
|
|
async def record_request(self, **kwargs):
|
|
return None
|
|
|
|
async def record_stage_timings(self, path, timings):
|
|
return None
|
|
|
|
async def record_failed(self, **kwargs):
|
|
return None
|
|
|
|
def record_compression_failed(self, reason: str) -> None:
|
|
return None
|
|
|
|
async def record_rate_limited(self, **kwargs):
|
|
return None
|
|
|
|
|
|
class _ResponseStub:
|
|
status_code = 200
|
|
headers: dict[str, str] = {}
|
|
content = b'{"id":"msg_1","type":"message","role":"assistant","content":[],"usage":{"input_tokens":1,"output_tokens":1}}'
|
|
|
|
def json(self):
|
|
return {
|
|
"id": "msg_1",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"content": [],
|
|
"usage": {"input_tokens": 1, "output_tokens": 1},
|
|
}
|
|
|
|
|
|
class _RecordingBackgroundCompressor:
|
|
def __init__(self) -> None:
|
|
self.enqueued: list[tuple[str, object, object]] = []
|
|
|
|
def enqueue(self, key, compress, store) -> bool:
|
|
self.enqueued.append((key, compress, store))
|
|
return True
|
|
|
|
|
|
def _fake_pipeline_apply(messages, model, **kwargs):
|
|
compressed = []
|
|
for msg in messages:
|
|
new = dict(msg)
|
|
if msg.get("role") == "user" and isinstance(msg.get("content"), list):
|
|
new["content"] = [
|
|
{**part, "content": _COMPRESSED_TEXT}
|
|
if isinstance(part, dict) and part.get("type") == "tool_result"
|
|
else part
|
|
for part in msg["content"]
|
|
]
|
|
compressed.append(new)
|
|
return TransformResult(
|
|
messages=compressed,
|
|
tokens_before=1000,
|
|
tokens_after=100,
|
|
transforms_applied=["read_lifecycle:stale:test.py"],
|
|
)
|
|
|
|
|
|
class _DummyAnthropicHandler(AnthropicHandlerMixin):
|
|
ANTHROPIC_API_URL = "https://api.anthropic.com"
|
|
|
|
def __init__(self) -> None:
|
|
self.rate_limiter = None
|
|
self.metrics = _DummyMetrics()
|
|
self.config = ProxyConfig(
|
|
optimize=True,
|
|
image_optimize=False,
|
|
retry_max_attempts=1,
|
|
retry_base_delay_ms=1,
|
|
retry_max_delay_ms=1,
|
|
connect_timeout_seconds=10,
|
|
mode="token",
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
prefix_freeze_enabled=False,
|
|
memory_enabled=False,
|
|
)
|
|
self.usage_reporter = None
|
|
self.anthropic_provider = SimpleNamespace(get_context_limit=lambda model: 200_000)
|
|
self.anthropic_pipeline = SimpleNamespace(apply=MagicMock(side_effect=_fake_pipeline_apply))
|
|
self.anthropic_backend = None
|
|
self.cost_tracker = None
|
|
self.memory_handler = None
|
|
self.cache = None
|
|
self.security = None
|
|
self.ccr_context_tracker = None
|
|
self.ccr_injector = None
|
|
self.ccr_response_handler = None
|
|
self.ccr_feedback = None
|
|
self.ccr_batch_processor = None
|
|
self.ccr_mcp_server = None
|
|
self.traffic_learner = None
|
|
self.tool_injector = None
|
|
self.read_lifecycle_manager = None
|
|
self.logger = SimpleNamespace(log=lambda *a, **k: None)
|
|
self.request_logger = self.logger
|
|
self.usage_observer = None
|
|
self.image_compressor = None
|
|
self.session_tracker_store = SimpleNamespace(
|
|
compute_session_id=lambda *a, **k: "sess-1",
|
|
get_or_create=lambda *a, **k: SimpleNamespace(
|
|
get_frozen_message_count=lambda: 0,
|
|
get_last_original_messages=lambda: [],
|
|
get_last_forwarded_messages=lambda: [],
|
|
record_request=lambda *a, **k: None,
|
|
),
|
|
resolve_tracker=lambda *a, **k: SimpleNamespace(
|
|
get_frozen_message_count=lambda: 0,
|
|
get_last_original_messages=lambda: [],
|
|
get_last_forwarded_messages=lambda: [],
|
|
record_request=lambda *a, **k: None,
|
|
),
|
|
)
|
|
# Cold-start deferral wiring under test.
|
|
self._background_compression_enabled = True
|
|
self._background_compression_min_tokens = 1
|
|
self._background_compressor = _RecordingBackgroundCompressor()
|
|
self.executor_calls: list[float] = []
|
|
|
|
async def _run_compression_in_executor(self, fn, timeout):
|
|
self.executor_calls.append(timeout)
|
|
return fn()
|
|
|
|
async def _next_request_id(self) -> str:
|
|
return "req-fastpass-test"
|
|
|
|
def _extract_tags(self, headers):
|
|
return {}
|
|
|
|
async def _retry_request(self, method, url, headers, body, **_kwargs):
|
|
self.captured_body = body
|
|
return _ResponseStub()
|
|
|
|
def _get_compression_cache(self, session_id):
|
|
self.comp_cache_updates: list[tuple] = getattr(self, "comp_cache_updates", [])
|
|
return SimpleNamespace(
|
|
apply_cached=lambda m: m,
|
|
compute_frozen_count=lambda m: 0,
|
|
mark_stable_from_messages=lambda *a, **k: None,
|
|
should_defer_compression=lambda h: False,
|
|
mark_stable=lambda h: None,
|
|
content_hash=lambda c: "h",
|
|
update_from_result=lambda *a: self.comp_cache_updates.append(a),
|
|
_cache={},
|
|
_stable_hashes=set(),
|
|
)
|
|
|
|
|
|
def _build_request(body: dict) -> Request:
|
|
payload = json.dumps(body).encode("utf-8")
|
|
|
|
async def receive():
|
|
return {"type": "http.request", "body": payload, "more_body": False}
|
|
|
|
scope = {
|
|
"type": "http",
|
|
"asgi": {"version": "3.0"},
|
|
"http_version": "1.1",
|
|
"method": "POST",
|
|
"scheme": "https",
|
|
"path": "/v1/messages",
|
|
"raw_path": b"/v1/messages",
|
|
"query_string": b"",
|
|
"headers": [(b"authorization", b"Bearer sk-ant-api-test")],
|
|
"client": ("127.0.0.1", 12345),
|
|
"server": ("testserver", 443),
|
|
}
|
|
return Request(scope, receive)
|
|
|
|
|
|
def test_cold_start_runs_fast_pass_and_defers_only_kompress(monkeypatch):
|
|
import headroom.tokenizers as _tk
|
|
|
|
monkeypatch.setattr(_tk, "get_tokenizer", lambda model: _DummyTokenizer())
|
|
|
|
handler = _DummyAnthropicHandler()
|
|
request = _build_request(
|
|
{
|
|
"model": "claude-3-5-sonnet-latest",
|
|
"messages": [
|
|
{
|
|
"role": "user",
|
|
"content": [
|
|
{
|
|
"type": "tool_result",
|
|
"tool_use_id": "toolu_1",
|
|
"content": "verbose stale tool output " * 200,
|
|
}
|
|
],
|
|
},
|
|
],
|
|
}
|
|
)
|
|
|
|
anyio.run(handler.handle_anthropic_messages, request)
|
|
|
|
# The fast pass ran synchronously with the ML stage disabled.
|
|
assert handler.executor_calls, "fast pass never ran through the executor"
|
|
sync_calls = [
|
|
c for c in handler.anthropic_pipeline.apply.call_args_list if c.kwargs.get("skip_kompress")
|
|
]
|
|
assert len(sync_calls) == 1, "expected exactly one synchronous skip_kompress pass"
|
|
|
|
# The full pipeline (kompress included) went to the background queue,
|
|
# keyed against the ORIGINAL messages for content-hash reuse.
|
|
assert len(handler._background_compressor.enqueued) == 1
|
|
_key, bg_compress, _store = handler._background_compressor.enqueued[0]
|
|
bg_compress()
|
|
bg_calls = [
|
|
c
|
|
for c in handler.anthropic_pipeline.apply.call_args_list
|
|
if not c.kwargs.get("skip_kompress")
|
|
]
|
|
assert len(bg_calls) == 1, "background job must run the full pipeline"
|
|
|
|
# The FORWARDED body carries the fast-pass form — that is what the
|
|
# provider caches and the byte-identical freeze locks in.
|
|
forwarded = handler.captured_body["messages"]
|
|
assert forwarded[0]["content"][0]["content"] == _COMPRESSED_TEXT
|
|
|
|
# Fast-pass results were stored in the compression cache.
|
|
assert handler.comp_cache_updates
|
|
|
|
|
|
def test_fast_pass_failure_falls_back_to_full_deferral(monkeypatch):
|
|
import headroom.tokenizers as _tk
|
|
|
|
monkeypatch.setattr(_tk, "get_tokenizer", lambda model: _DummyTokenizer())
|
|
|
|
handler = _DummyAnthropicHandler()
|
|
|
|
async def _boom(fn, timeout):
|
|
raise TimeoutError("fast pass exceeded budget")
|
|
|
|
handler._run_compression_in_executor = _boom # type: ignore[method-assign]
|
|
|
|
original_text = "verbose stale tool output " * 200
|
|
request = _build_request(
|
|
{
|
|
"model": "claude-3-5-sonnet-latest",
|
|
"messages": [
|
|
{
|
|
"role": "user",
|
|
"content": [
|
|
{
|
|
"type": "tool_result",
|
|
"tool_use_id": "toolu_1",
|
|
"content": original_text,
|
|
}
|
|
],
|
|
},
|
|
],
|
|
}
|
|
)
|
|
|
|
anyio.run(handler.handle_anthropic_messages, request)
|
|
|
|
# Fail-open: original messages forwarded, background job still queued.
|
|
forwarded = handler.captured_body["messages"]
|
|
assert forwarded[0]["content"][0]["content"] == original_text
|
|
assert len(handler._background_compressor.enqueued) == 1
|