mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): keep read maturation out of the provider-confirmed prefix (#4522)
## Description In token mode with read maturation on, a Claude Code session loses Anthropic cache reads on the turn a Read matures. The proxy swaps a Read the provider has already cached for a marker, so the provider re-writes its cache from that Read onward at the cache-write rate. Extra Headroom compares each request's forwarded bytes inside the provider-confirmed prefix with the previous turn's. Its integrity probe records this as a change a fixed 9 messages back from the end of the cached prefix: Claude Code appends two messages per turn, and `quiesce_turns=5` matures a Read five assistant turns after its file was last touched. Root cause: `ReadMaturationManager.apply` leaves messages below `frozen_message_count` alone (`headroom/transforms/read_maturation.py:207`), and its documented invariant is that the frozen prefix is never mutated (`read_maturation.py:42`, `headroom/config.py:665`). The Anthropic handler, though, passes it the pipeline's freeze count (`headroom/proxy/handlers/anthropic.py:2626`), which `prepare_turn` clamps to `min(tracker_frozen, cache_count)` (`headroom/proxy/session_engine.py:134`, assigned at `anthropic.py:2077`). Maturation also runs after `finalize_turn` (`anthropic.py:2475`) has replayed the provider-confirmed floor `_confirmed_frozen` (`anthropic.py:2070`). `cache_count` stops at the first tool_result the compression cache cannot vouch for. That is a tool_result that never went through a pipeline run that changed something, or one with no text at all (#4523). Either can hold the count low for a whole session, so every Read that quiesces is already inside the provider's cache, and it matures. #4299 found this for the first request after a restart (`anthropic.py:2038`) and seeded the cache there; the steady state has the same cause. The fix passes maturation `max(frozen_message_count, _confirmed_frozen)`. A Read now matures only while the provider has not confirmed it: a cold cache, a lapsed TTL, or a restart without the state file. In cache mode `_confirmed_frozen` stays 0, so nothing changes there (#4198 already reports maturation as inert in that mode). Evidence from one Extra Headroom user's macOS proxy (wheel 0.41.0, token mode, read maturation on): - The rewriting request logged `read_maturation:1` and `Pipeline: freezing first 50/123 messages`. The provider-confirmed floor was 121 messages, and the forwarded change sat at message 112. - Over that machine's logs, 32 warm-cache requests matured a Read, and each re-wrote 2.3k to 30k tokens of cache. - Across Extra Headroom installs, 10 of 13 sampled events of this kind changed the forwarded bytes exactly 9 messages back from the cached floor. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/handlers/anthropic.py`: maturation gets the larger of the pipeline's freeze and the provider-confirmed floor that `finalize_turn` just replayed. - `tests/test_read_maturation_handler_nobust.py`: new `test_read_inside_the_cached_prefix_stays_verbatim` through the real handler, with honest cache usage and two session shapes. In `quiet`, nothing compresses. In `tool-reference`, a ToolSearch result precedes the Read and the later tool output compresses. Each asserts that the pipeline's freeze stayed below the Read, that the Read is never matured, and that every forwarded request starts with the previous one's messages. Both fail on main. - The three tests that expected an already cached Read to mature (`test_held_read_keeps_client_breakpoint_then_matures`, `test_quiesce_turns_config_is_honored`, `test_replayed_marker_is_not_rebooked_in_the_emitted_savings`) passed only because their fixture compresses nothing, so the freeze never reached the Read. They now drive an upstream that reports no cache, where maturation still applies. `_drive_session` gains `cache_hits` and `tools`, and the first test uses it instead of its own copy of the mock. - `tests/test_prefix_tracker_restart.py`: the matured-reference restart test matures each Read on arrival (`read_maturation_quiesce_turns=0`), before the provider caches it. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev --extra proxy pytest tests/test_read_maturation_handler_nobust.py tests/test_read_maturation.py tests/test_prefix_tracker_restart.py -q 55 passed, 1 warning in 16.03s $ uv run --frozen --extra dev --extra proxy pytest tests scripts/tests -q 22 failed, 17129 passed, 581 skipped, 111 warnings in 2840.29s (0:47:20) # 16 fail the same way on main (f10725e6) on this machine: # tests/test_proxy_uds.py (11), tests/test_cli/test_wrap_persistent.py (4), # tests/test_image_compression.py::TestOnnxRouter::test_full_classify_with_image # 6 pass when run on their own, on this branch and on main (order-dependent): # tests/test_proxy_health.py (4 readyz kompress tests), # tests/test_offline_egress_chokepoint.py::TestUpdateCheckOffline::test_explicit_update_check_never_constructs_request, # tests/test_stateless_no_disk.py::test_cli_stateless_flag_exports_env_for_children $ ruff check . && ruff format --check . All checks passed! 1834 files already formatted $ mypy headroom --ignore-missing-imports Found 2 errors in 2 files (checked 578 source files) # the same 2 as main, in files this PR does not touch: # integrations/langchain/chat_model.py:380, memory/adapters/embedders.py:389 ``` ## Real Behavior Proof - Environment: macOS 27 on an M2; this branch and upstream main (f10725e6), each from its own worktree; `headroom proxy` started with Extra Headroom's flags and env (token mode, read maturation on via the beta channel, `--no-http2 --no-rate-limit --log-messages`, isolated HOME, telemetry off), in front of a local mock Anthropic upstream that reports cache_read for the longest message prefix shared with an earlier request and cache_write for the rest. - Exact command / steps: one scripted Claude Code session per build: a ToolSearch answered with a `tool_reference` block, a Read of a 7.7 KB file, then 8 Bash turns. It runs once with short Bash output (quiet) and once with 300-entry JSON listings (busy). Each forwarded request is then diffed against the previous one with `cache_control` stripped. - Observed result: on main, both sessions rewrite message 4 on the 15-message request (9 back from the 13-message floor); this branch forwards every cached prefix unchanged. | Session | Build | Read matured | Cached prefix changed | cache_read/forwarded | Cache writes | tok_saved/tok_before | |---|---|---|---|---|---|---| | quiet | main | yes | request 8, message 4 | 86.2% | 4,105 | 10.9% | | quiet | this PR | no | never | 90.4% | 3,559 | 0.0% | | busy | main | yes | request 8, message 4 | 67.0% | 39,360 | 43.2% | | busy | this PR | no | never | 79.5% | 25,994 | 42.2% | - Not tested: Linux and Windows; real Anthropic traffic on this branch (the rewrite itself was seen on real traffic, the fix only against the mock upstream); cache mode, which this change leaves alone by construction. ## Runtime Rollout Safety - Rollout-managed feature(s): read_maturation - Minimum rollout channel: beta (unchanged) - Stable/default behavior changed: no; read maturation is beta-only and off by default. With it on, a Read the provider has already cached now stays verbatim instead of maturing. - Kill switch / disable path: leave read_maturation off (`HEADROOM_READ_MATURATION=0`); this change has no switch of its own, revert the commit - Unsafe override required: no - Qualification impact: none expected; it removes forwarded-prefix changes on maturing turns - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes The trade: maturation savings are given up only for Reads the provider has already cached. Maturing such a Read re-writes everything after it at the cache-write rate (1.25x or 2x input) to save 0.1x on the Read for the remaining turns. `test_read_cached_with_its_turn_stays_verbatim` already gives this reason; this PR makes it hold whatever the compression cache knows. Related: #4523 keeps a tool_result with no text from stopping the compression cache's count, which is one of the two reasons the pipeline freeze sticks. It is independent of this PR: alone, it stops the rewrite only in sessions where tool output compresses. Extra Headroom carries this change as a temporary patch against 0.41.0 until a release includes it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JerrettDavis
parent
fba09f674c
commit
526e06b7e0
@@ -2645,9 +2645,14 @@ class AnthropicHandlerMixin:
|
||||
prefix_tracker.restored_maturation_state = None
|
||||
maturation_mgr.restore_state(state)
|
||||
_maturation_mgr = maturation_mgr
|
||||
# Never below the floor finalize_turn just replayed: the
|
||||
# pipeline's freeze is clamped to what the local compression
|
||||
# cache can byte-replay, which stops at the first tool_result
|
||||
# it never stored, so on its own it lets a cached Read mature
|
||||
# and re-write the provider cache from that Read onward.
|
||||
maturation = maturation_mgr.apply(
|
||||
optimized_messages,
|
||||
frozen_message_count=frozen_message_count,
|
||||
frozen_message_count=max(frozen_message_count, _confirmed_frozen),
|
||||
)
|
||||
if maturation.replacements_applied or maturation.holding_msg_indices:
|
||||
optimized_messages = maturation.messages
|
||||
|
||||
@@ -137,13 +137,14 @@ _SCENARIOS = {
|
||||
|
||||
|
||||
def _restart_between_turns(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, scenario: str
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, scenario: str, **overrides: Any
|
||||
) -> tuple[list[dict[str, Any]], list[dict[str, Any]], int, dict[str, str]]:
|
||||
"""Some turns in one proxy, the next turn in a fresh one, one warm
|
||||
provider cache. Returns (last forwarded before, first forwarded after,
|
||||
its read hit, the original behind each retrieval reference forwarded
|
||||
before the restart)."""
|
||||
build, turns, config, kompress = _SCENARIOS[scenario]
|
||||
config = {**config, **overrides}
|
||||
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "memory")
|
||||
# As the desktop app runs it: a cold start forwards the fast pass.
|
||||
monkeypatch.setenv("HEADROOM_BACKGROUND_COMPRESSION", "1")
|
||||
@@ -182,8 +183,11 @@ def test_restored_retrieval_references_resolve_to_their_originals(
|
||||
tracker kept replaying pointed at a hash that no longer resolved."""
|
||||
monkeypatch.setenv(PREFIX_STATE_FILE_ENV, str(tmp_path / "prefix-state.json"))
|
||||
|
||||
# Each Read matures as it arrives, before the provider caches it, so
|
||||
# references exist before the restart. A Read that quiesces later is
|
||||
# already cached by then and stays verbatim.
|
||||
previous, current, _, originals = _restart_between_turns(
|
||||
monkeypatch, tmp_path, "read-maturation"
|
||||
monkeypatch, tmp_path, "read-maturation", read_maturation_quiesce_turns=0
|
||||
)
|
||||
|
||||
emitted = _ccr_hashes(current)
|
||||
|
||||
@@ -183,64 +183,10 @@ def test_held_read_keeps_client_breakpoint_then_matures(monkeypatch):
|
||||
cost_tracking_enabled=False,
|
||||
log_requests=False,
|
||||
)
|
||||
app = create_app(config)
|
||||
forwarded: list[list[dict]] = []
|
||||
|
||||
with TestClient(app) as client:
|
||||
proxy = client.app.state.proxy
|
||||
original_retry = proxy._retry_request
|
||||
|
||||
async def _mock_upstream(method, url, headers, body, stream=False, **kwargs):
|
||||
msgs = body.get("messages", []) or []
|
||||
forwarded.append(copy.deepcopy(msgs))
|
||||
# Simulate Anthropic honestly caching up to the handler's breakpoint
|
||||
# (system blocks are cached too), so the prefix tracker advances
|
||||
# frozen_message_count as in prod.
|
||||
bp = _breakpoint_index(msgs)
|
||||
sys_tokens = sum(
|
||||
max(1, len(str(b.get("text", ""))) // 4)
|
||||
for b in (body.get("system") or [])
|
||||
if isinstance(b, dict)
|
||||
)
|
||||
cached = sys_tokens + (sum(_est_tokens(m) for m in msgs[: bp + 1]) if bp >= 0 else 0)
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"id": "msg_x",
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"content": [{"type": "text", "text": "ok"}],
|
||||
"usage": {
|
||||
"input_tokens": 20,
|
||||
"output_tokens": 2,
|
||||
"cache_read_input_tokens": cached,
|
||||
"cache_creation_input_tokens": 0,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
proxy._retry_request = _mock_upstream
|
||||
try:
|
||||
for n in range(0, 7):
|
||||
r = client.post(
|
||||
"/v1/messages",
|
||||
headers={
|
||||
"x-api-key": "test-key",
|
||||
"anthropic-version": "2023-06-01",
|
||||
"x-headroom-session-id": "nobust-1",
|
||||
"content-type": "application/json",
|
||||
},
|
||||
json={
|
||||
"model": MODEL,
|
||||
"max_tokens": 20,
|
||||
"system": SYSTEM,
|
||||
"tools": [READ_TOOL],
|
||||
"messages": _convo(n),
|
||||
},
|
||||
)
|
||||
assert r.status_code == 200, f"turn {n}: {r.text[:300]}"
|
||||
finally:
|
||||
proxy._retry_request = original_retry
|
||||
# The provider confirms no cache (as after a lapse), so nothing is frozen
|
||||
# and a quiet Read may mature. Once the provider has cached it, it stays
|
||||
# verbatim instead: test_read_inside_the_cached_prefix_stays_verbatim.
|
||||
forwarded = _drive_session(config, n_turns=7, session_id="nobust-1", cache_hits=False)
|
||||
|
||||
assert forwarded, "no requests were forwarded"
|
||||
|
||||
@@ -263,7 +209,7 @@ def test_held_read_keeps_client_breakpoint_then_matures(monkeypatch):
|
||||
# Read, which dropped Claude Code's tail breakpoint for the whole hold and
|
||||
# re-sent the Read and everything after it uncached on every held turn, to
|
||||
# save one cache write of the Read. The held Read is now cached with its
|
||||
# turn, and maturing it re-writes the cache from the Read onward once.
|
||||
# turn.
|
||||
moved = [
|
||||
turn
|
||||
for turn, msgs in enumerate(forwarded)
|
||||
@@ -288,11 +234,14 @@ def _drive_session(
|
||||
session_id: str,
|
||||
saved_out: list[int] | None = None,
|
||||
convo=_convo,
|
||||
cache_hits: bool = True,
|
||||
tools: list[dict] | None = None,
|
||||
) -> list[list[dict]]:
|
||||
"""Drive ``n_turns`` cumulative turns of ``convo`` through the real handler
|
||||
with a mocked upstream; return the forwarded message arrays per turn. When
|
||||
``saved_out`` is given, each turn's emitted ``x-headroom-tokens-saved`` is
|
||||
appended to it."""
|
||||
appended to it. The upstream caches up to the handler's breakpoint, or
|
||||
reports no cache at all when ``cache_hits`` is False."""
|
||||
app = create_app(config)
|
||||
forwarded: list[list[dict]] = []
|
||||
with TestClient(app) as client:
|
||||
@@ -309,6 +258,8 @@ def _drive_session(
|
||||
if isinstance(b, dict)
|
||||
)
|
||||
cached = sys_tokens + (sum(_est_tokens(m) for m in msgs[: bp + 1]) if bp >= 0 else 0)
|
||||
if not cache_hits:
|
||||
cached = 0
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
@@ -340,7 +291,7 @@ def _drive_session(
|
||||
"model": MODEL,
|
||||
"max_tokens": 20,
|
||||
"system": SYSTEM,
|
||||
"tools": [READ_TOOL],
|
||||
"tools": tools or [READ_TOOL],
|
||||
"messages": convo(n),
|
||||
},
|
||||
)
|
||||
@@ -406,6 +357,121 @@ def test_read_cached_with_its_turn_stays_verbatim(monkeypatch):
|
||||
assert _first_matured_turn(forwarded) is None, "a cached Read was matured"
|
||||
|
||||
|
||||
DEFERRED_TOOL = {
|
||||
"name": "mcp__notes__search",
|
||||
"description": "Search notes",
|
||||
"defer_loading": True,
|
||||
"input_schema": {"type": "object", "properties": {"q": {"type": "string"}}},
|
||||
}
|
||||
|
||||
|
||||
def _tool_search_convo(nquiet: int) -> list[dict]:
|
||||
"""``_busy_convo`` after a ToolSearch whose result holds only tool_reference
|
||||
blocks, as Claude Code loads deferred tools. The compression cache reads no
|
||||
text from that result, so its count never passes it, however much of the
|
||||
later tool output compresses."""
|
||||
msgs = _busy_convo(nquiet)
|
||||
search = [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_use",
|
||||
"id": "toolu_ts",
|
||||
"name": "ToolSearch",
|
||||
"input": {"query": "notes"},
|
||||
}
|
||||
],
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_result",
|
||||
"tool_use_id": "toolu_ts",
|
||||
"content": [{"type": "tool_reference", "tool_name": DEFERRED_TOOL["name"]}],
|
||||
}
|
||||
],
|
||||
},
|
||||
]
|
||||
return msgs[:1] + search + msgs[1:]
|
||||
|
||||
|
||||
def _without_cache_control(messages: list[dict]) -> list[dict]:
|
||||
return json.loads(
|
||||
json.dumps(messages),
|
||||
object_hook=lambda d: {k: v for k, v in d.items() if k != "cache_control"},
|
||||
)
|
||||
|
||||
|
||||
def _prefix_rewrites(forwarded: list[list[dict]]) -> list[int]:
|
||||
"""Turns whose forwarded messages do not start with the previous turn's.
|
||||
The mock caches each turn through its last message, so any such turn
|
||||
re-writes the provider cache. Breakpoints move every turn and are ignored."""
|
||||
rewrites = []
|
||||
for turn in range(1, len(forwarded)):
|
||||
prev = _without_cache_control(forwarded[turn - 1])
|
||||
if _without_cache_control(forwarded[turn])[: len(prev)] != prev:
|
||||
rewrites.append(turn)
|
||||
return rewrites
|
||||
|
||||
|
||||
def _record_pipeline_freeze(monkeypatch) -> list[int]:
|
||||
from headroom.proxy import session_engine
|
||||
|
||||
seen: list[int] = []
|
||||
prepare = session_engine.prepare_turn
|
||||
|
||||
def _recording_prepare(*args, **kwargs):
|
||||
prep = prepare(*args, **kwargs)
|
||||
seen.append(prep.frozen_message_count)
|
||||
return prep
|
||||
|
||||
monkeypatch.setattr(session_engine, "prepare_turn", _recording_prepare)
|
||||
return seen
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("convo", "tools", "pipeline_stop"),
|
||||
[
|
||||
# Nothing compresses, so the compression cache never stores the Read.
|
||||
(_convo, [READ_TOOL], 2),
|
||||
# Tool output compresses, but the count stops at the ToolSearch result.
|
||||
(_tool_search_convo, [READ_TOOL, DEFERRED_TOOL], 2),
|
||||
],
|
||||
ids=["quiet", "tool-reference"],
|
||||
)
|
||||
def test_read_inside_the_cached_prefix_stays_verbatim(monkeypatch, convo, tools, pipeline_stop):
|
||||
"""By the time the Read's file quiesces the provider has cached it, so
|
||||
maturing it would re-write the cache from the Read onward. The pipeline's
|
||||
own freeze stops below the Read here; maturation must follow the
|
||||
provider-confirmed floor that finalize_turn replays instead."""
|
||||
from headroom.cache.compression_store import reset_compression_store
|
||||
|
||||
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "memory")
|
||||
reset_compression_store()
|
||||
pipeline_freeze = _record_pipeline_freeze(monkeypatch)
|
||||
config = ProxyConfig(
|
||||
optimize=True,
|
||||
read_maturation=True,
|
||||
mode="token",
|
||||
cache_enabled=True,
|
||||
rate_limit_enabled=False,
|
||||
cost_tracking_enabled=False,
|
||||
log_requests=False,
|
||||
)
|
||||
|
||||
forwarded = _drive_session(
|
||||
config, n_turns=8, session_id="cached-read-2", convo=convo, tools=tools
|
||||
)
|
||||
|
||||
assert pipeline_freeze and max(pipeline_freeze) <= pipeline_stop, (
|
||||
f"fixture must keep the pipeline freeze below the Read: {pipeline_freeze}"
|
||||
)
|
||||
assert _first_matured_turn(forwarded) is None, "a cached Read was matured"
|
||||
assert not _prefix_rewrites(forwarded), "the forwarded cached prefix changed"
|
||||
|
||||
|
||||
def test_quiesce_turns_config_is_honored(monkeypatch):
|
||||
"""`quiesce_turns` must be runtime-configurable end-to-end: a fresh Read of
|
||||
/x/foo.py matures `quiesce_turns` quiet turns after it appears (the convo
|
||||
@@ -428,7 +494,7 @@ def test_quiesce_turns_config_is_honored(monkeypatch):
|
||||
cost_tracking_enabled=False,
|
||||
log_requests=False,
|
||||
)
|
||||
forwarded = _drive_session(config, n_turns=4, session_id="quiesce-cfg-1")
|
||||
forwarded = _drive_session(config, n_turns=4, session_id="quiesce-cfg-1", cache_hits=False)
|
||||
first = _first_matured_turn(forwarded)
|
||||
assert first == 2, (
|
||||
f"expected the Read to mature at turn 2 with quiesce_turns=2, "
|
||||
@@ -503,7 +569,9 @@ def test_replayed_marker_is_not_rebooked_in_the_emitted_savings(monkeypatch):
|
||||
log_requests=False,
|
||||
)
|
||||
saved: list[int] = []
|
||||
forwarded = _drive_session(config, n_turns=5, session_id="first-appearance-1", saved_out=saved)
|
||||
forwarded = _drive_session(
|
||||
config, n_turns=5, session_id="first-appearance-1", saved_out=saved, cache_hits=False
|
||||
)
|
||||
|
||||
first = _first_matured_turn(forwarded)
|
||||
assert first is not None, "the Read never matured, so there is nothing to replay"
|
||||
|
||||
Reference in New Issue
Block a user