fix(proxy): keep read maturation out of the provider-confirmed prefix (#4522)

## Description

In token mode with read maturation on, a Claude Code session loses
Anthropic cache reads on the turn a Read matures. The proxy swaps a Read
the provider has already cached for a marker, so the provider re-writes
its cache from that Read onward at the cache-write rate. Extra Headroom
compares each request's forwarded bytes inside the provider-confirmed
prefix with the previous turn's. Its integrity probe records this as a
change a fixed 9 messages back from the end of the cached prefix: Claude
Code appends two messages per turn, and `quiesce_turns=5` matures a Read
five assistant turns after its file was last touched.

Root cause: `ReadMaturationManager.apply` leaves messages below
`frozen_message_count` alone
(`headroom/transforms/read_maturation.py:207`), and its documented
invariant is that the frozen prefix is never mutated
(`read_maturation.py:42`, `headroom/config.py:665`). The Anthropic
handler, though, passes it the pipeline's freeze count
(`headroom/proxy/handlers/anthropic.py:2626`), which `prepare_turn`
clamps to `min(tracker_frozen, cache_count)`
(`headroom/proxy/session_engine.py:134`, assigned at
`anthropic.py:2077`). Maturation also runs after `finalize_turn`
(`anthropic.py:2475`) has replayed the provider-confirmed floor
`_confirmed_frozen` (`anthropic.py:2070`). `cache_count` stops at the
first tool_result the compression cache cannot vouch for. That is a
tool_result that never went through a pipeline run that changed
something, or one with no text at all (#4523). Either can hold the count
low for a whole session, so every Read that quiesces is already inside
the provider's cache, and it matures. #4299 found this for the first
request after a restart (`anthropic.py:2038`) and seeded the cache
there; the steady state has the same cause.

The fix passes maturation `max(frozen_message_count,
_confirmed_frozen)`. A Read now matures only while the provider has not
confirmed it: a cold cache, a lapsed TTL, or a restart without the state
file. In cache mode `_confirmed_frozen` stays 0, so nothing changes
there (#4198 already reports maturation as inert in that mode).

Evidence from one Extra Headroom user's macOS proxy (wheel 0.41.0, token
mode, read maturation on):
- The rewriting request logged `read_maturation:1` and `Pipeline:
freezing first 50/123 messages`. The provider-confirmed floor was 121
messages, and the forwarded change sat at message 112.
- Over that machine's logs, 32 warm-cache requests matured a Read, and
each re-wrote 2.3k to 30k tokens of cache.
- Across Extra Headroom installs, 10 of 13 sampled events of this kind
changed the forwarded bytes exactly 9 messages back from the cached
floor.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/handlers/anthropic.py`: maturation gets the larger of
the pipeline's freeze and the provider-confirmed floor that
`finalize_turn` just replayed.
- `tests/test_read_maturation_handler_nobust.py`: new
`test_read_inside_the_cached_prefix_stays_verbatim` through the real
handler, with honest cache usage and two session shapes. In `quiet`,
nothing compresses. In `tool-reference`, a ToolSearch result precedes
the Read and the later tool output compresses. Each asserts that the
pipeline's freeze stayed below the Read, that the Read is never matured,
and that every forwarded request starts with the previous one's
messages. Both fail on main.
- The three tests that expected an already cached Read to mature
(`test_held_read_keeps_client_breakpoint_then_matures`,
`test_quiesce_turns_config_is_honored`,
`test_replayed_marker_is_not_rebooked_in_the_emitted_savings`) passed
only because their fixture compresses nothing, so the freeze never
reached the Read. They now drive an upstream that reports no cache,
where maturation still applies. `_drive_session` gains `cache_hits` and
`tools`, and the first test uses it instead of its own copy of the mock.
- `tests/test_prefix_tracker_restart.py`: the matured-reference restart
test matures each Read on arrival (`read_maturation_quiesce_turns=0`),
before the provider caches it.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev --extra proxy pytest tests/test_read_maturation_handler_nobust.py tests/test_read_maturation.py tests/test_prefix_tracker_restart.py -q
55 passed, 1 warning in 16.03s

$ uv run --frozen --extra dev --extra proxy pytest tests scripts/tests -q
22 failed, 17129 passed, 581 skipped, 111 warnings in 2840.29s (0:47:20)
# 16 fail the same way on main (f10725e6) on this machine:
#   tests/test_proxy_uds.py (11), tests/test_cli/test_wrap_persistent.py (4),
#   tests/test_image_compression.py::TestOnnxRouter::test_full_classify_with_image
# 6 pass when run on their own, on this branch and on main (order-dependent):
#   tests/test_proxy_health.py (4 readyz kompress tests),
#   tests/test_offline_egress_chokepoint.py::TestUpdateCheckOffline::test_explicit_update_check_never_constructs_request,
#   tests/test_stateless_no_disk.py::test_cli_stateless_flag_exports_env_for_children

$ ruff check . && ruff format --check .
All checks passed!
1834 files already formatted

$ mypy headroom --ignore-missing-imports
Found 2 errors in 2 files (checked 578 source files)
# the same 2 as main, in files this PR does not touch:
# integrations/langchain/chat_model.py:380, memory/adapters/embedders.py:389
```

## Real Behavior Proof

- Environment: macOS 27 on an M2; this branch and upstream main
(f10725e6), each from its own worktree; `headroom proxy` started with
Extra Headroom's flags and env (token mode, read maturation on via the
beta channel, `--no-http2 --no-rate-limit --log-messages`, isolated
HOME, telemetry off), in front of a local mock Anthropic upstream that
reports cache_read for the longest message prefix shared with an earlier
request and cache_write for the rest.
- Exact command / steps: one scripted Claude Code session per build: a
ToolSearch answered with a `tool_reference` block, a Read of a 7.7 KB
file, then 8 Bash turns. It runs once with short Bash output (quiet) and
once with 300-entry JSON listings (busy). Each forwarded request is then
diffed against the previous one with `cache_control` stripped.
- Observed result: on main, both sessions rewrite message 4 on the
15-message request (9 back from the 13-message floor); this branch
forwards every cached prefix unchanged.

| Session | Build | Read matured | Cached prefix changed |
cache_read/forwarded | Cache writes | tok_saved/tok_before |
  |---|---|---|---|---|---|---|
  | quiet | main | yes | request 8, message 4 | 86.2% | 4,105 | 10.9% |
  | quiet | this PR | no | never | 90.4% | 3,559 | 0.0% |
  | busy | main | yes | request 8, message 4 | 67.0% | 39,360 | 43.2% |
  | busy | this PR | no | never | 79.5% | 25,994 | 42.2% |
- Not tested: Linux and Windows; real Anthropic traffic on this branch
(the rewrite itself was seen on real traffic, the fix only against the
mock upstream); cache mode, which this change leaves alone by
construction.

## Runtime Rollout Safety

- Rollout-managed feature(s): read_maturation
- Minimum rollout channel: beta (unchanged)
- Stable/default behavior changed: no; read maturation is beta-only and
off by default. With it on, a Read the provider has already cached now
stays verbatim instead of maturing.
- Kill switch / disable path: leave read_maturation off
(`HEADROOM_READ_MATURATION=0`); this change has no switch of its own,
revert the commit
- Unsafe override required: no
- Qualification impact: none expected; it removes forwarded-prefix
changes on maturing turns
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

The trade: maturation savings are given up only for Reads the provider
has already cached. Maturing such a Read re-writes everything after it
at the cache-write rate (1.25x or 2x input) to save 0.1x on the Read for
the remaining turns. `test_read_cached_with_its_turn_stays_verbatim`
already gives this reason; this PR makes it hold whatever the
compression cache knows.

Related: #4523 keeps a tool_result with no text from stopping the
compression cache's count, which is one of the two reasons the pipeline
freeze sticks. It is independent of this PR: alone, it stops the rewrite
only in sessions where tool output compresses. Extra Headroom carries
this change as a temporary patch against 0.41.0 until a release includes
it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
gglucass
2026-10-28 23:45:17 -05:00
committed by GitHub
co-authored by Claude Opus 5.5 JerrettDavis
parent fba09f674c
commit 526e06b7e0
3 changed files with 143 additions and 66 deletions
+6 -1
View File
@@ -2645,9 +2645,14 @@ class AnthropicHandlerMixin:
prefix_tracker.restored_maturation_state = None
maturation_mgr.restore_state(state)
_maturation_mgr = maturation_mgr
# Never below the floor finalize_turn just replayed: the
# pipeline's freeze is clamped to what the local compression
# cache can byte-replay, which stops at the first tool_result
# it never stored, so on its own it lets a cached Read mature
# and re-write the provider cache from that Read onward.
maturation = maturation_mgr.apply(
optimized_messages,
frozen_message_count=frozen_message_count,
frozen_message_count=max(frozen_message_count, _confirmed_frozen),
)
if maturation.replacements_applied or maturation.holding_msg_indices:
optimized_messages = maturation.messages
+6 -2
View File
@@ -137,13 +137,14 @@ _SCENARIOS = {
def _restart_between_turns(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, scenario: str
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, scenario: str, **overrides: Any
) -> tuple[list[dict[str, Any]], list[dict[str, Any]], int, dict[str, str]]:
"""Some turns in one proxy, the next turn in a fresh one, one warm
provider cache. Returns (last forwarded before, first forwarded after,
its read hit, the original behind each retrieval reference forwarded
before the restart)."""
build, turns, config, kompress = _SCENARIOS[scenario]
config = {**config, **overrides}
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "memory")
# As the desktop app runs it: a cold start forwards the fast pass.
monkeypatch.setenv("HEADROOM_BACKGROUND_COMPRESSION", "1")
@@ -182,8 +183,11 @@ def test_restored_retrieval_references_resolve_to_their_originals(
tracker kept replaying pointed at a hash that no longer resolved."""
monkeypatch.setenv(PREFIX_STATE_FILE_ENV, str(tmp_path / "prefix-state.json"))
# Each Read matures as it arrives, before the provider caches it, so
# references exist before the restart. A Read that quiesces later is
# already cached by then and stays verbatim.
previous, current, _, originals = _restart_between_turns(
monkeypatch, tmp_path, "read-maturation"
monkeypatch, tmp_path, "read-maturation", read_maturation_quiesce_turns=0
)
emitted = _ccr_hashes(current)
+131 -63
View File
@@ -183,64 +183,10 @@ def test_held_read_keeps_client_breakpoint_then_matures(monkeypatch):
cost_tracking_enabled=False,
log_requests=False,
)
app = create_app(config)
forwarded: list[list[dict]] = []
with TestClient(app) as client:
proxy = client.app.state.proxy
original_retry = proxy._retry_request
async def _mock_upstream(method, url, headers, body, stream=False, **kwargs):
msgs = body.get("messages", []) or []
forwarded.append(copy.deepcopy(msgs))
# Simulate Anthropic honestly caching up to the handler's breakpoint
# (system blocks are cached too), so the prefix tracker advances
# frozen_message_count as in prod.
bp = _breakpoint_index(msgs)
sys_tokens = sum(
max(1, len(str(b.get("text", ""))) // 4)
for b in (body.get("system") or [])
if isinstance(b, dict)
)
cached = sys_tokens + (sum(_est_tokens(m) for m in msgs[: bp + 1]) if bp >= 0 else 0)
return httpx.Response(
200,
json={
"id": "msg_x",
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": "ok"}],
"usage": {
"input_tokens": 20,
"output_tokens": 2,
"cache_read_input_tokens": cached,
"cache_creation_input_tokens": 0,
},
},
)
proxy._retry_request = _mock_upstream
try:
for n in range(0, 7):
r = client.post(
"/v1/messages",
headers={
"x-api-key": "test-key",
"anthropic-version": "2023-06-01",
"x-headroom-session-id": "nobust-1",
"content-type": "application/json",
},
json={
"model": MODEL,
"max_tokens": 20,
"system": SYSTEM,
"tools": [READ_TOOL],
"messages": _convo(n),
},
)
assert r.status_code == 200, f"turn {n}: {r.text[:300]}"
finally:
proxy._retry_request = original_retry
# The provider confirms no cache (as after a lapse), so nothing is frozen
# and a quiet Read may mature. Once the provider has cached it, it stays
# verbatim instead: test_read_inside_the_cached_prefix_stays_verbatim.
forwarded = _drive_session(config, n_turns=7, session_id="nobust-1", cache_hits=False)
assert forwarded, "no requests were forwarded"
@@ -263,7 +209,7 @@ def test_held_read_keeps_client_breakpoint_then_matures(monkeypatch):
# Read, which dropped Claude Code's tail breakpoint for the whole hold and
# re-sent the Read and everything after it uncached on every held turn, to
# save one cache write of the Read. The held Read is now cached with its
# turn, and maturing it re-writes the cache from the Read onward once.
# turn.
moved = [
turn
for turn, msgs in enumerate(forwarded)
@@ -288,11 +234,14 @@ def _drive_session(
session_id: str,
saved_out: list[int] | None = None,
convo=_convo,
cache_hits: bool = True,
tools: list[dict] | None = None,
) -> list[list[dict]]:
"""Drive ``n_turns`` cumulative turns of ``convo`` through the real handler
with a mocked upstream; return the forwarded message arrays per turn. When
``saved_out`` is given, each turn's emitted ``x-headroom-tokens-saved`` is
appended to it."""
appended to it. The upstream caches up to the handler's breakpoint, or
reports no cache at all when ``cache_hits`` is False."""
app = create_app(config)
forwarded: list[list[dict]] = []
with TestClient(app) as client:
@@ -309,6 +258,8 @@ def _drive_session(
if isinstance(b, dict)
)
cached = sys_tokens + (sum(_est_tokens(m) for m in msgs[: bp + 1]) if bp >= 0 else 0)
if not cache_hits:
cached = 0
return httpx.Response(
200,
json={
@@ -340,7 +291,7 @@ def _drive_session(
"model": MODEL,
"max_tokens": 20,
"system": SYSTEM,
"tools": [READ_TOOL],
"tools": tools or [READ_TOOL],
"messages": convo(n),
},
)
@@ -406,6 +357,121 @@ def test_read_cached_with_its_turn_stays_verbatim(monkeypatch):
assert _first_matured_turn(forwarded) is None, "a cached Read was matured"
DEFERRED_TOOL = {
"name": "mcp__notes__search",
"description": "Search notes",
"defer_loading": True,
"input_schema": {"type": "object", "properties": {"q": {"type": "string"}}},
}
def _tool_search_convo(nquiet: int) -> list[dict]:
"""``_busy_convo`` after a ToolSearch whose result holds only tool_reference
blocks, as Claude Code loads deferred tools. The compression cache reads no
text from that result, so its count never passes it, however much of the
later tool output compresses."""
msgs = _busy_convo(nquiet)
search = [
{
"role": "assistant",
"content": [
{
"type": "tool_use",
"id": "toolu_ts",
"name": "ToolSearch",
"input": {"query": "notes"},
}
],
},
{
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": "toolu_ts",
"content": [{"type": "tool_reference", "tool_name": DEFERRED_TOOL["name"]}],
}
],
},
]
return msgs[:1] + search + msgs[1:]
def _without_cache_control(messages: list[dict]) -> list[dict]:
return json.loads(
json.dumps(messages),
object_hook=lambda d: {k: v for k, v in d.items() if k != "cache_control"},
)
def _prefix_rewrites(forwarded: list[list[dict]]) -> list[int]:
"""Turns whose forwarded messages do not start with the previous turn's.
The mock caches each turn through its last message, so any such turn
re-writes the provider cache. Breakpoints move every turn and are ignored."""
rewrites = []
for turn in range(1, len(forwarded)):
prev = _without_cache_control(forwarded[turn - 1])
if _without_cache_control(forwarded[turn])[: len(prev)] != prev:
rewrites.append(turn)
return rewrites
def _record_pipeline_freeze(monkeypatch) -> list[int]:
from headroom.proxy import session_engine
seen: list[int] = []
prepare = session_engine.prepare_turn
def _recording_prepare(*args, **kwargs):
prep = prepare(*args, **kwargs)
seen.append(prep.frozen_message_count)
return prep
monkeypatch.setattr(session_engine, "prepare_turn", _recording_prepare)
return seen
@pytest.mark.parametrize(
("convo", "tools", "pipeline_stop"),
[
# Nothing compresses, so the compression cache never stores the Read.
(_convo, [READ_TOOL], 2),
# Tool output compresses, but the count stops at the ToolSearch result.
(_tool_search_convo, [READ_TOOL, DEFERRED_TOOL], 2),
],
ids=["quiet", "tool-reference"],
)
def test_read_inside_the_cached_prefix_stays_verbatim(monkeypatch, convo, tools, pipeline_stop):
"""By the time the Read's file quiesces the provider has cached it, so
maturing it would re-write the cache from the Read onward. The pipeline's
own freeze stops below the Read here; maturation must follow the
provider-confirmed floor that finalize_turn replays instead."""
from headroom.cache.compression_store import reset_compression_store
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "memory")
reset_compression_store()
pipeline_freeze = _record_pipeline_freeze(monkeypatch)
config = ProxyConfig(
optimize=True,
read_maturation=True,
mode="token",
cache_enabled=True,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
)
forwarded = _drive_session(
config, n_turns=8, session_id="cached-read-2", convo=convo, tools=tools
)
assert pipeline_freeze and max(pipeline_freeze) <= pipeline_stop, (
f"fixture must keep the pipeline freeze below the Read: {pipeline_freeze}"
)
assert _first_matured_turn(forwarded) is None, "a cached Read was matured"
assert not _prefix_rewrites(forwarded), "the forwarded cached prefix changed"
def test_quiesce_turns_config_is_honored(monkeypatch):
"""`quiesce_turns` must be runtime-configurable end-to-end: a fresh Read of
/x/foo.py matures `quiesce_turns` quiet turns after it appears (the convo
@@ -428,7 +494,7 @@ def test_quiesce_turns_config_is_honored(monkeypatch):
cost_tracking_enabled=False,
log_requests=False,
)
forwarded = _drive_session(config, n_turns=4, session_id="quiesce-cfg-1")
forwarded = _drive_session(config, n_turns=4, session_id="quiesce-cfg-1", cache_hits=False)
first = _first_matured_turn(forwarded)
assert first == 2, (
f"expected the Read to mature at turn 2 with quiesce_turns=2, "
@@ -503,7 +569,9 @@ def test_replayed_marker_is_not_rebooked_in_the_emitted_savings(monkeypatch):
log_requests=False,
)
saved: list[int] = []
forwarded = _drive_session(config, n_turns=5, session_id="first-appearance-1", saved_out=saved)
forwarded = _drive_session(
config, n_turns=5, session_id="first-appearance-1", saved_out=saved, cache_hits=False
)
first = _first_matured_turn(forwarded)
assert first is not None, "the Read never matured, so there is nothing to replay"