mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description Closes #3997 With `read_maturation` on, every held Read drops Claude Code's tail cache breakpoint, so the rest of the conversation is re-sent uncached on every held turn. On one machine on 2026-10-05 (wheel 0.39.0), 37 of 1,353 requests were in 8 holds. One 13-request hold kept `cache_read` stuck at 176,693 with `cache_write` 0, while the uncached tail grew from 3,879 to 14,639 tokens per request. Over its last 10 requests that hold sent 83.8k tokens uncached. With the tail cached, the same requests cost about 36k token-equivalents. Cause: when `ReadMaturationManager.apply` returns `holding_msg_indices`, the handler calls `relocate_cache_breakpoint`. That strips every `cache_control` from the earliest held Read to the end of the conversation and puts one marker on the message before the Read. Claude Code marks the last two messages, so its tail marker is always dropped. The proxy already logs this on every held turn as `event=cache_breakpoints ... dropped=true tail_grew=true`. Nothing after the held Read is cache-written until the Read matures, which takes at least `quiesce_turns` (5) turns and up to `max_hold_turns` (25). The hold saves one cache write of the Read (1.25x for a 5m TTL, 2x for 1h, on a few thousand tokens). It pays for that with a full-price resend of the Read and everything after it on every held turn. This PR removes the relocation and forwards the client's breakpoints unchanged, so the held Read is cached with its turn. On the next request, the provider-confirmed prefix (`frozen_message_count`) covers the Read, and the existing `i < frozen_message_count` gate in `ReadMaturationManager.apply` leaves it verbatim. That is intended. Maturing a Read that is already cached rewrites everything after it at the cache-write rate, and a week of this machine's traffic, replayed, shows that costs far more than it saves: - Median rewrite: 19.2k tokens at the 1h write price (2x). - What maturing saves: a Read of about 2.1k tokens, read at 0.1x over a median of 36 remaining turns. - Net: maturing every Read that is already cached comes to about -23.7M base-input-token equivalents per week. Not maturing them comes to 0. The result: maturing a Read that is already cached becomes rare, and tokens saved from `read_maturation` drop by design. A Read still matures when it is outside the confirmed prefix once its file goes quiet, for example on a turn where the frozen count is reset to 0 or where the client put no breakpoint after the Read. Trade-offs: - A session that ends right after the Read pays the cache-write premium on the tail once, where the old code sent it uncached. Break-even is about 2-3 held requests, and holds last at least 5 turns. - The frozen count the gate uses is the handler's clamped one: `prepare_turn` takes min(provider-confirmed, locally replayable). If no tool output in the session has been compressed since the Read, the clamp can hold the count at the Read even though the provider has cached it. A Read can then still mature inside the provider-cached prefix, which is a rewrite. Sessions with compressible tool output release the clamp on the next request (see the probe below). Using the unclamped count for this gate would be a one-line follow-up. I left it out to keep this PR to the breakpoint fix. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/read_maturation.py`: delete `relocate_cache_breakpoint`. The handler was its only production caller. Update the module docstring: the timeline, invariant 1 (now "the frozen prefix is never mutated"), and why a cached Read stays verbatim. The manager logic is unchanged. - `headroom/proxy/handlers/anthropic.py`: forward `maturation.messages` as returned, with no relocation. - `headroom/config.py`, `headroom/proxy/models.py`, `headroom/cli/proxy.py`, `headroom/rollout.py`, `docs/content/docs/proxy.mdx`: these descriptions said held Reads are kept out of the prefix cache, which is no longer true. They now say Reads stay verbatim and compress once quiet unless the provider cache already holds them. - `tests/test_read_maturation_handler_nobust.py`: `test_verbatim_read_never_cache_written_before_maturation` asserted the old invariant: the verbatim Read is never forwarded inside the cached prefix. That invariant is exactly the bug, so the test is now `test_held_read_keeps_client_breakpoint_then_matures`. It asserts that the client's breakpoint is forwarded where the client put it on every turn. It still checks that the Read is held and then matures in order: in this fixture nothing compresses, so the frozen count stays at the Read. A comment in the test explains the change. - `tests/test_read_maturation_handler_nobust.py`: new `test_read_cached_with_its_turn_stays_verbatim`. It drives the real handler for 7 turns with compressible Bash output around the Read, and asserts three things: the breakpoints are kept on every turn, the frozen count passes the held Read from the second request on, and the Read is never rewritten. - `tests/test_read_maturation.py`: new `test_held_read_matures_only_outside_frozen_prefix`. A held Read matures at quiesce when it is in the live zone and stays verbatim when the frozen prefix covers it. The test for a client breakpoint on a fresh Read now asserts the breakpoint stays put and survives maturation. `TestBreakpointRelocation` is removed along with the helper. - `tests/test_cache_ttl_preserved.py`: drop the two relocation TTL tests. The tool-search deferral TTL tests stay. - `tests/test_live/test_live_maturation.py`: request A asserts the held Read is cache-written with the client's tail marker. Request B, with the Read inside the confirmed prefix, asserts it is not matured and that A's cached prefix is read back. ## Testing - [x] Unit tests pass (`pytest`) - the read-maturation, breakpoint and rollout files below, not the whole suite - [x] Linting passes (`ruff check .`) - on the changed files, plus `ruff format --check` - [x] Type checking passes (`mypy headroom`) - on the six changed modules - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output New and updated tests with `headroom/` at `main` (befdb52f): ```text $ uv run --frozen --extra dev pytest tests/test_read_maturation.py tests/test_read_maturation_handler_nobust.py tests/test_cache_ttl_preserved.py -q E AssertionError: client cache breakpoint moved or dropped on turn(s) [0, 1, 2, 3, 4] E AssertionError: client cache breakpoint moved or dropped on turn(s) [0, 1, 2, 3, 4] FAILED tests/test_read_maturation_handler_nobust.py::test_held_read_keeps_client_breakpoint_then_matures FAILED tests/test_read_maturation_handler_nobust.py::test_read_cached_with_its_turn_stays_verbatim ========================= 2 failed, 33 passed in 9.68s ========================= ``` With this change: ```text $ uv run --frozen --extra dev pytest tests/gateway/test_cache_breakpoint_budget.py tests/test_cache_breakpoint_diagnostics.py \ tests/test_cache_control_move_bust.py tests/test_issue_2671_block_growth_cache.py tests/test_rollout.py \ tests/test_testing_harness.py tests/test_read_maturation.py tests/test_read_maturation_handler_nobust.py \ tests/test_cache_ttl_preserved.py tests/test_live/test_live_maturation.py -q ======================== 170 passed, 2 skipped in 9.11s ======================== $ uv run --frozen ruff check <10 changed .py files> All checks passed! $ uv run --frozen ruff format --check <10 changed .py files> 10 files already formatted $ uv run --frozen --extra dev mypy headroom/transforms/read_maturation.py headroom/proxy/handlers/anthropic.py \ headroom/config.py headroom/cli/proxy.py headroom/proxy/models.py headroom/rollout.py --ignore-missing-imports Success: no issues found in 6 source files ``` ## Real Behavior Proof - Environment: macOS 27 (arm64), Python 3.11.13, `uv run --frozen --extra dev`, this branch and `main` (befdb52f). Live evidence comes from a Claude Code session behind a 0.39.0 proxy with `HEADROOM_READ_MATURATION=1` on the beta channel. - Exact command / steps: a probe script drives `create_app(ProxyConfig(read_maturation=True, mode="token", cache_enabled=True))` through `TestClient` for 8 turns of one Claude Code-shaped conversation. The conversation has two 1h system breakpoints, client breakpoints on the last block of the last two messages, compressible Bash JSON output, a 3 KB line-numbered Read at message 10, and then quiet turns. The upstream mock reports cache reads up to the last forwarded breakpoint, using the prefix tracker's own estimator. The probe prints `count_cache_breakpoints` for each request in and out, the frozen count maturation saw, and the Read's forwarded form. - Observed result: on `main`, every held turn forwards 3 of the client's 4 breakpoints, with the tail breakpoint 1, 3, 5, 7, 9 messages from the end. The frozen count never passes the Read (10), and the Read matures on turn 5. On this branch, all 4 breakpoints reach the tail on every turn, the frozen count passes the Read from the second request on (11, 13, 15, ...), and the Read stays verbatim. In production, the same hold logged `event=cache_breakpoints ... dropped=true tail_grew=true` on every held request, with the cost figures in the Description. ```text main (befdb52f): INFO turn 0: in_total=4 out_total=3 in_last_tail=0 out_last_tail=1 maturation_frozen=0 read=verbatim INFO turn 1: in_total=4 out_total=3 in_last_tail=0 out_last_tail=3 maturation_frozen=10 read=verbatim INFO turn 2: in_total=4 out_total=3 in_last_tail=0 out_last_tail=5 maturation_frozen=10 read=verbatim INFO turn 3: in_total=4 out_total=3 in_last_tail=0 out_last_tail=7 maturation_frozen=10 read=verbatim INFO turn 4: in_total=4 out_total=3 in_last_tail=0 out_last_tail=9 maturation_frozen=10 read=verbatim INFO turn 5: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=10 read=matured INFO turn 6: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=21 read=matured INFO turn 7: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=23 read=matured this branch: INFO turn 0: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=0 read=verbatim INFO turn 1: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=11 read=verbatim INFO turn 2: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=13 read=verbatim INFO turn 3: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=15 read=verbatim INFO turn 4: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=17 read=verbatim INFO turn 5: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=19 read=verbatim INFO turn 6: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=21 read=verbatim INFO turn 7: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=23 read=verbatim ``` - Not tested: `tests/test_live/test_live_maturation.py` against the real API (no `ANTHROPIC_API_KEY` here; it was updated and skips). This exact branch has not run against live traffic. The live figures above come from the 0.39.0 wheel. The full `pytest` suite was not run, only the files listed under Testing. ## Runtime Rollout Safety - Rollout-managed feature(s): read_maturation (behavior change inside the existing feature; no new feature). - Minimum rollout channel: Beta (unchanged - inherits the feature's existing gate). - Stable/default behavior changed: No. read_maturation is beta-gated and off by default, and none of this code runs when it is off. With it on, held Reads now keep the client's cache breakpoints and are cached with their turn, so a Read the provider cache already covers is no longer matured. - Kill switch / disable path: disabling read_maturation (`HEADROOM_READ_MATURATION=0`, or drop `--read-maturation`) disables this with it; the old breakpoint relocation has no separate switch because it only added cost. - Unsafe override required: No. - Qualification impact: None. - Rollback path: revert the commit; no persisted format or config surface changes, and matured-marker session state is unchanged. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` - it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - `relocate_cache_breakpoint` was public in `headroom.transforms.read_maturation` but not re-exported. A repo-wide grep found no caller outside the handler and tests. - `count_cache_breakpoints` and the `event=cache_breakpoints` log line already flagged every held turn as `dropped=true`. With this change those turns log `dropped=false`. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3977" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>