mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(anthropic): stop the spurious compaction warning and drop dead handler code (#3678)
## Description
Three small cleanups in `headroom/proxy/handlers/anthropic.py`, found
while reading the handler end to end. One is a real bug, two are dead
code.
**Spurious "system prompt compaction FAILED" on every request.** With
`HEADROOM_SYSTEM_COMPACT` enabled and no `ContentRouter` in the
pipeline, the handler read `_sys_modified`, a local that is only
assigned on the router branch. The resulting `UnboundLocalError` was
caught by the surrounding `except Exception` and logged as a compaction
failure on every request, even though nothing had failed. The flag is
now only read on the branch that sets it.
**No-op OpenAI fallback stub.** `ProxyConfig.fallback_enabled` /
`fallback_provider` gated a block in the Anthropic error path whose body
was two comments and a log line saying "Attempting fallback to OpenAI".
No conversion or retry ever existed, and nothing else in `headroom/`,
`tests/` or `docs/` references either field. Both fields and the block
are gone.
**Redundant pre-upstream semaphore releases.**
`handle_anthropic_messages` wraps its body in a `try/finally` that calls
the idempotent `_finalize_pre_upstream()` on every exit. Six explicit
calls placed directly before early `return`s (413, invalid JSON,
message-count 400, cache hit, security 403, Bedrock error) therefore did
nothing. They are removed. The calls that deliberately release *before*
a long await (streaming and backend handoffs) are kept, since those
change timing rather than correctness.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/anthropic.py`: move the `if _sys_modified:`
bookkeeping inside the router branch that assigns it, so the compaction
success/skip path no longer raises `UnboundLocalError` into the failure
log.
- `headroom/proxy/handlers/anthropic.py`: delete the
`fallback_enabled`/`fallback_provider` block in the upstream-error path;
it contained no executable fallback.
- `headroom/proxy/models.py`: delete the two unused `ProxyConfig`
fields. No loader, CLI flag, env var or doc maps to them.
- `headroom/proxy/handlers/anthropic.py`: drop six `await
_finalize_pre_upstream()` calls that sit immediately before an early
`return` already covered by the handler's `finally`.
- `tests/test_system_compaction_handler.py`: new
`test_system_compaction_without_content_router_is_not_a_failure`, which
fails on `main` with `cannot access local variable '_sys_modified'` and
passes here.
- `tests/test_system_compaction_handler.py`: also
`test_system_compaction_with_content_router_records_the_transform`,
which stubs the router lookup and `compact_system_prompt` so the success
branch (transform tag + debug log) is exercised; closes the codecov
partial on the moved `if _sys_modified:` line.
- `tests/test_anthropic_pre_upstream_backpressure.py`: extend the
early-exit semaphore test to the three validation paths it did not cover
(413, invalid JSON, message-count 400), so removing the explicit
releases is proven safe rather than assumed.
- `tests/test_anthropic_stage_timings.py`,
`tests/test_cache_mode_cold_start.py`,
`tests/test_cold_start_fast_pass.py`: stop setting the removed config
fields.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest -q tests/test_anthropic_pre_upstream_backpressure.py tests/test_system_compaction_handler.py \
tests/test_anthropic_stage_timings.py tests/test_cache_mode_cold_start.py tests/test_cold_start_fast_pass.py
41 passed, 2 warnings in 7.6s
# same files with the new test against upstream/main's handler:
FAILED tests/test_system_compaction_handler.py::test_system_compaction_without_content_router_is_not_a_failure
E 'system prompt compaction FAILED: cannot access local variable '_sys_modified' where it is not associated with a value'
$ ruff check headroom/proxy/handlers/anthropic.py headroom/proxy/models.py tests/test_anthropic_pre_upstream_backpressure.py tests/test_system_compaction_handler.py
All checks passed!
$ mypy headroom/proxy/handlers/anthropic.py headroom/proxy/models.py
Success: no issues found in 2 source files
$ grep -rn 'fallback_enabled\|fallback_provider' headroom/ tests/ docs/
(no output)
```
## Real Behavior Proof
- Environment: macOS 25.6, Python 3.12 venv with the repo's compiled
`headroom._core`, fresh worktree of `upstream/main` @ `67eb910e`, the
three commits cherry-picked on top.
- Exact command / steps: read `handle_anthropic_messages` from the
semaphore acquire through the outer `finally` and confirmed
`_finalize_pre_upstream()` is idempotent and runs in `finally` on every
exit; grepped every reference to `fallback_enabled` /
`fallback_provider`; ran the five touched test files on the branch and
again with the new compaction test against `main`'s handler; ran `ruff
check` and `mypy` on the touched source files.
- Observed result: 40 tests pass on the branch. The new compaction test
reproduces the spurious failure log on `main` and passes here. No
remaining references to the deleted config fields. Lint and type check
clean on the touched files.
- Not tested: no live proxy traffic. The compaction fix only changes
which branch reads a local, the fallback removal deletes code with no
behavior, and the semaphore change is covered by the extended early-exit
test rather than by observing a real backlog. `ruff format --check`
flags `anthropic.py` on `main` already; this PR does not reformat
unrelated lines.
## Runtime Rollout Safety
- Rollout-managed feature(s): `HEADROOM_SYSTEM_COMPACT` (env-gated, off
by default). No new gate is introduced.
- Minimum rollout channel: n/a. The fix only affects the log line and
`transforms_applied` bookkeeping when compaction is enabled and no
router is present; upstream requests were already unaffected.
- Stable/default behavior changed: no. With the env var unset, none of
the changed lines execute. `ProxyConfig(fallback_enabled=...)` would now
raise `TypeError`, but no code path, doc or config loader ever passed
it.
- Kill switch / disable path: unset `HEADROOM_SYSTEM_COMPACT` restores
the pre-existing behavior for the compaction path; the other two changes
remove dead code and have no runtime switch.
- Unsafe override required: no.
- Qualification impact: none. No compression, cache or provider behavior
changes.
- Rollback path: `git revert` of the three commits; they touch seven
files and no persisted state or schema.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
A fourth commit that unified the budget and rate-limit preflight across
all provider handlers was deliberately left out. Open PR #3579 already
enforces the spend budget on the OpenAI and Gemini routes with two
approvals, and #3375 targets the same issue #3374. The remaining
rate-limit-key unification will be proposed separately once #3579 lands.
Documentation checkbox is unticked because `fallback_enabled` /
`fallback_provider` appear in no documentation and no user-facing
behavior changed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1032,7 +1032,6 @@ class AnthropicHandlerMixin:
|
||||
# Check request body size
|
||||
content_length = request.headers.get("content-length")
|
||||
if content_length and int(content_length) > MAX_REQUEST_BODY_SIZE:
|
||||
await _finalize_pre_upstream()
|
||||
return JSONResponse(
|
||||
status_code=413,
|
||||
content={
|
||||
@@ -1055,7 +1054,6 @@ class AnthropicHandlerMixin:
|
||||
async with stage_timer.measure("read_request_json"):
|
||||
body, original_body_bytes = await read_request_json_with_bytes(request)
|
||||
except (json.JSONDecodeError, ValueError) as e:
|
||||
await _finalize_pre_upstream()
|
||||
return JSONResponse(
|
||||
status_code=400,
|
||||
content={
|
||||
@@ -1132,7 +1130,6 @@ class AnthropicHandlerMixin:
|
||||
|
||||
# Validate message array size
|
||||
if len(messages) > MAX_MESSAGE_ARRAY_LENGTH:
|
||||
await _finalize_pre_upstream()
|
||||
return JSONResponse(
|
||||
status_code=400,
|
||||
content={
|
||||
@@ -1424,9 +1421,6 @@ class AnthropicHandlerMixin:
|
||||
f"hits={cached.hit_count}"
|
||||
)
|
||||
|
||||
# Unit 4: release the pre-upstream semaphore on cache
|
||||
# hit — no upstream call will happen.
|
||||
await _finalize_pre_upstream()
|
||||
return Response(
|
||||
content=cached.response_body,
|
||||
headers=response_headers,
|
||||
@@ -1455,9 +1449,6 @@ class AnthropicHandlerMixin:
|
||||
if hasattr(e, "reason"):
|
||||
from fastapi.responses import JSONResponse as _JSONResp
|
||||
|
||||
# Unit 4: release the pre-upstream semaphore on
|
||||
# security block — no upstream call will happen.
|
||||
await _finalize_pre_upstream()
|
||||
return _JSONResp(
|
||||
status_code=403,
|
||||
content={
|
||||
@@ -2969,15 +2960,15 @@ class AnthropicHandlerMixin:
|
||||
model=model,
|
||||
request_id=request_id,
|
||||
)
|
||||
if _sys_modified:
|
||||
transforms_applied.append("anthropic:system_prompt_compaction")
|
||||
logger.debug(
|
||||
"[%s] system prompt compaction: %d -> %d bytes (%.0f%% saved)",
|
||||
request_id,
|
||||
_sys_before,
|
||||
_sys_after,
|
||||
(1 - _sys_after / max(_sys_before, 1)) * 100,
|
||||
)
|
||||
if _sys_modified:
|
||||
transforms_applied.append("anthropic:system_prompt_compaction")
|
||||
logger.debug(
|
||||
"[%s] system prompt compaction: %d -> %d bytes (%.0f%% saved)",
|
||||
request_id,
|
||||
_sys_before,
|
||||
_sys_after,
|
||||
(1 - _sys_after / max(_sys_before, 1)) * 100,
|
||||
)
|
||||
except Exception as _sys_compaction_exc:
|
||||
_sys_modified = False
|
||||
logger.warning(
|
||||
@@ -3683,8 +3674,6 @@ class AnthropicHandlerMixin:
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"[{request_id}] Bedrock backend error: {e}")
|
||||
# Unit 4: release the pre-upstream semaphore on error.
|
||||
await _finalize_pre_upstream()
|
||||
return JSONResponse(
|
||||
status_code=500,
|
||||
content={
|
||||
@@ -5036,12 +5025,6 @@ class AnthropicHandlerMixin:
|
||||
# Log full error details internally for debugging
|
||||
logger.error(f"[{request_id}] Request failed: {type(e).__name__}: {e}")
|
||||
|
||||
# Try fallback if enabled
|
||||
if self.config.fallback_enabled and self.config.fallback_provider == "openai":
|
||||
logger.info(f"[{request_id}] Attempting fallback to OpenAI")
|
||||
# Convert to OpenAI format and retry
|
||||
# (simplified - would need message format conversion)
|
||||
|
||||
# Return sanitized error message to client (don't expose internal details)
|
||||
return JSONResponse(
|
||||
status_code=502,
|
||||
|
||||
@@ -371,10 +371,6 @@ class ProxyConfig:
|
||||
# (repeatable); env: HEADROOM_COMPRESSORS.
|
||||
compressors: set[str] | None = None
|
||||
|
||||
# Fallback
|
||||
fallback_enabled: bool = False
|
||||
fallback_provider: str | None = None
|
||||
|
||||
# Timeouts
|
||||
request_timeout_seconds: int = 300
|
||||
connect_timeout_seconds: int = 10
|
||||
|
||||
@@ -39,6 +39,7 @@ from fastapi.testclient import TestClient
|
||||
|
||||
from headroom.cli.proxy import proxy as proxy_cli
|
||||
from headroom.proxy.handlers.anthropic import AnthropicHandlerMixin
|
||||
from headroom.proxy.helpers import MAX_MESSAGE_ARRAY_LENGTH, MAX_REQUEST_BODY_SIZE
|
||||
from headroom.proxy.models import CacheEntry, ProxyConfig
|
||||
from headroom.proxy.server import HeadroomProxy, create_app
|
||||
|
||||
@@ -136,8 +137,6 @@ class _DummyAnthropicHandler(AnthropicHandlerMixin):
|
||||
mode="token",
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
fallback_enabled=False,
|
||||
fallback_provider=None,
|
||||
prefix_freeze_enabled=False,
|
||||
memory_enabled=False,
|
||||
)
|
||||
@@ -291,8 +290,8 @@ class _DummyAnthropicHandler(AnthropicHandlerMixin):
|
||||
)
|
||||
|
||||
|
||||
def _build_request(body: dict, headers: dict[str, str]) -> Request:
|
||||
payload = json.dumps(body).encode("utf-8")
|
||||
def _build_request(body: dict | bytes, headers: dict[str, str]) -> Request:
|
||||
payload = body if isinstance(body, bytes) else json.dumps(body).encode("utf-8")
|
||||
|
||||
async def receive():
|
||||
return {"type": "http.request", "body": payload, "more_body": False}
|
||||
@@ -723,8 +722,6 @@ def test_compression_is_not_bypassed_when_gated(stage_log_capture):
|
||||
mode="token",
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
fallback_enabled=False,
|
||||
fallback_provider=None,
|
||||
prefix_freeze_enabled=False,
|
||||
memory_enabled=False,
|
||||
anthropic_pre_upstream_concurrency=2,
|
||||
@@ -848,11 +845,12 @@ def test_auto_computed_default_on_this_machine():
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Semaphore released on HTTPException / early-exit paths even with an #
|
||||
# already-held permit. Explicitly covers the 4 pre-upstream early exits: #
|
||||
# already-held permit. Explicitly covers the pre-upstream early exits: #
|
||||
# - rate_limiter deny (429) #
|
||||
# - cost_tracker block (429) #
|
||||
# - security scan block (403) #
|
||||
# - cache hit (200) #
|
||||
# - oversized body (413), invalid JSON (400), too many messages (400) #
|
||||
# Each test holds 1 permit of a Semaphore(2) with a concurrent request, #
|
||||
# then verifies the handler restores ``_value`` to the original after #
|
||||
# the early return. #
|
||||
@@ -910,7 +908,15 @@ class _CacheHit:
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"scenario",
|
||||
["rate_limiter", "cost_tracker", "security", "cache"],
|
||||
[
|
||||
"rate_limiter",
|
||||
"cost_tracker",
|
||||
"security",
|
||||
"cache",
|
||||
"oversized_body",
|
||||
"invalid_json",
|
||||
"too_many_messages",
|
||||
],
|
||||
)
|
||||
def test_early_exit_paths_release_semaphore_under_contention(scenario):
|
||||
"""Hold one permit of a Semaphore(1) with a concurrent request, trigger
|
||||
@@ -931,13 +937,19 @@ def test_early_exit_paths_release_semaphore_under_contention(scenario):
|
||||
elif scenario == "cache":
|
||||
handler.cache = _CacheHit()
|
||||
|
||||
req = _build_request(
|
||||
{
|
||||
"model": "claude-3-5-sonnet-latest",
|
||||
"messages": [{"role": "user", "content": "hello"}],
|
||||
},
|
||||
{"authorization": "Bearer sk-ant-api-test"},
|
||||
)
|
||||
body: dict | bytes = {
|
||||
"model": "claude-3-5-sonnet-latest",
|
||||
"messages": [{"role": "user", "content": "hello"}],
|
||||
}
|
||||
headers = {"authorization": "Bearer sk-ant-api-test"}
|
||||
if scenario == "oversized_body":
|
||||
headers["content-length"] = str(MAX_REQUEST_BODY_SIZE + 1)
|
||||
elif scenario == "invalid_json":
|
||||
body = b"{not json"
|
||||
elif scenario == "too_many_messages":
|
||||
body["messages"] = [{"role": "user", "content": "hi"}] * (MAX_MESSAGE_ARRAY_LENGTH + 1)
|
||||
req = _build_request(body, headers)
|
||||
expected_status = {"oversized_body": 413, "invalid_json": 400, "too_many_messages": 400}
|
||||
|
||||
# Drive several iterations to confirm each early-exit call fully
|
||||
# releases the semaphore rather than leaking a permit AND that the
|
||||
@@ -968,6 +980,8 @@ def test_early_exit_paths_release_semaphore_under_contention(scenario):
|
||||
# security returns a JSONResponse; cache returns a Response.
|
||||
assert raised is None, f"{scenario}: unexpected exception {raised!r}"
|
||||
assert result is not None
|
||||
if scenario in expected_status:
|
||||
assert result.status_code == expected_status[scenario], result.body
|
||||
assert sem._value == original_value, (
|
||||
f"{scenario}: semaphore leak got={sem._value}, want={original_value}"
|
||||
)
|
||||
|
||||
@@ -68,8 +68,6 @@ class _DummyAnthropicHandler(AnthropicHandlerMixin):
|
||||
mode="token",
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
fallback_enabled=False,
|
||||
fallback_provider=None,
|
||||
prefix_freeze_enabled=False,
|
||||
memory_enabled=False,
|
||||
)
|
||||
|
||||
@@ -107,8 +107,6 @@ class _DummyAnthropicHandler(AnthropicHandlerMixin):
|
||||
mode="cache",
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
fallback_enabled=False,
|
||||
fallback_provider=None,
|
||||
prefix_freeze_enabled=False,
|
||||
memory_enabled=False,
|
||||
)
|
||||
|
||||
@@ -105,8 +105,6 @@ class _DummyAnthropicHandler(AnthropicHandlerMixin):
|
||||
mode="token",
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
fallback_enabled=False,
|
||||
fallback_provider=None,
|
||||
prefix_freeze_enabled=False,
|
||||
memory_enabled=False,
|
||||
)
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
"""The Anthropic handler's system-prompt compaction must tolerate a pipeline
|
||||
with no ContentRouter instead of logging a spurious failure, and must record
|
||||
the transform when a router does rewrite the prompt."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
pytest.importorskip("fastapi")
|
||||
|
||||
from fastapi.testclient import TestClient # noqa: E402
|
||||
|
||||
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
|
||||
|
||||
|
||||
def _post_with_system_compaction(
|
||||
monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
|
||||
) -> str:
|
||||
"""POST one /v1/messages request with HEADROOM_SYSTEM_COMPACT on; return the log."""
|
||||
monkeypatch.setenv("HEADROOM_SYSTEM_COMPACT", "1")
|
||||
app = create_app(
|
||||
ProxyConfig(
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
cost_tracking_enabled=False,
|
||||
anthropic_api_url="http://127.0.0.1:9",
|
||||
)
|
||||
)
|
||||
# The headroom logger does not propagate to root, so capture it directly.
|
||||
proxy_logger = logging.getLogger("headroom.proxy")
|
||||
proxy_logger.addHandler(caplog.handler)
|
||||
caplog.set_level(logging.DEBUG, logger="headroom.proxy")
|
||||
try:
|
||||
with TestClient(app) as client:
|
||||
client.post(
|
||||
"/v1/messages",
|
||||
headers={"x-api-key": "sk-ant-test"},
|
||||
json={
|
||||
"model": "claude-sonnet-4-5",
|
||||
"max_tokens": 8,
|
||||
"system": "You are terse.",
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
},
|
||||
)
|
||||
finally:
|
||||
proxy_logger.removeHandler(caplog.handler)
|
||||
assert "Request failed" in caplog.text # reached the (dead) upstream, so compaction ran
|
||||
return caplog.text
|
||||
|
||||
|
||||
def test_system_compaction_without_content_router_is_not_a_failure(
|
||||
monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
monkeypatch.setattr(
|
||||
"headroom.transforms.compression_units.find_content_router", lambda _pipeline: None
|
||||
)
|
||||
log = _post_with_system_compaction(monkeypatch, caplog)
|
||||
assert "system prompt compaction FAILED" not in log
|
||||
|
||||
|
||||
def test_system_compaction_with_content_router_records_the_transform(
|
||||
monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
router = object()
|
||||
monkeypatch.setattr(
|
||||
"headroom.transforms.compression_units.find_content_router", lambda _pipeline: router
|
||||
)
|
||||
|
||||
def fake_compact(
|
||||
payload: dict[str, Any], **kwargs: Any
|
||||
) -> tuple[dict[str, Any], bool, int, int]:
|
||||
assert kwargs["router"] is router
|
||||
return payload, True, 100, 40
|
||||
|
||||
monkeypatch.setattr("headroom.proxy.system_compaction.compact_system_prompt", fake_compact)
|
||||
log = _post_with_system_compaction(monkeypatch, caplog)
|
||||
assert "system prompt compaction: 100 -> 40 bytes (60% saved)" in log
|
||||
assert "system prompt compaction FAILED" not in log
|
||||
Reference in New Issue
Block a user