Files
headroom/tests/test_cache_ttl_preserved.py
205668d953 fix(read-maturation): keep the client's tail cache breakpoint on held Reads (#3977)
## Description

Closes #3997

With `read_maturation` on, every held Read drops Claude Code's tail
cache breakpoint, so the rest of the conversation is re-sent uncached on
every held turn. On one machine on 2026-10-05 (wheel 0.39.0), 37 of
1,353 requests were in 8 holds. One 13-request hold kept `cache_read`
stuck at 176,693 with `cache_write` 0, while the uncached tail grew from
3,879 to 14,639 tokens per request. Over its last 10 requests that hold
sent 83.8k tokens uncached. With the tail cached, the same requests cost
about 36k token-equivalents.

Cause: when `ReadMaturationManager.apply` returns `holding_msg_indices`,
the handler calls `relocate_cache_breakpoint`. That strips every
`cache_control` from the earliest held Read to the end of the
conversation and puts one marker on the message before the Read. Claude
Code marks the last two messages, so its tail marker is always dropped.
The proxy already logs this on every held turn as
`event=cache_breakpoints ... dropped=true tail_grew=true`. Nothing after
the held Read is cache-written until the Read matures, which takes at
least `quiesce_turns` (5) turns and up to `max_hold_turns` (25). The
hold saves one cache write of the Read (1.25x for a 5m TTL, 2x for 1h,
on a few thousand tokens). It pays for that with a full-price resend of
the Read and everything after it on every held turn.

This PR removes the relocation and forwards the client's breakpoints
unchanged, so the held Read is cached with its turn. On the next
request, the provider-confirmed prefix (`frozen_message_count`) covers
the Read, and the existing `i < frozen_message_count` gate in
`ReadMaturationManager.apply` leaves it verbatim. That is intended.
Maturing a Read that is already cached rewrites everything after it at
the cache-write rate, and a week of this machine's traffic, replayed,
shows that costs far more than it saves:

- Median rewrite: 19.2k tokens at the 1h write price (2x).
- What maturing saves: a Read of about 2.1k tokens, read at 0.1x over a
median of 36 remaining turns.
- Net: maturing every Read that is already cached comes to about -23.7M
base-input-token equivalents per week. Not maturing them comes to 0.

The result: maturing a Read that is already cached becomes rare, and
tokens saved from `read_maturation` drop by design. A Read still matures
when it is outside the confirmed prefix once its file goes quiet, for
example on a turn where the frozen count is reset to 0 or where the
client put no breakpoint after the Read.

Trade-offs:

- A session that ends right after the Read pays the cache-write premium
on the tail once, where the old code sent it uncached. Break-even is
about 2-3 held requests, and holds last at least 5 turns.
- The frozen count the gate uses is the handler's clamped one:
`prepare_turn` takes min(provider-confirmed, locally replayable). If no
tool output in the session has been compressed since the Read, the clamp
can hold the count at the Read even though the provider has cached it. A
Read can then still mature inside the provider-cached prefix, which is a
rewrite. Sessions with compressible tool output release the clamp on the
next request (see the probe below). Using the unclamped count for this
gate would be a one-line follow-up. I left it out to keep this PR to the
breakpoint fix.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/read_maturation.py`: delete
`relocate_cache_breakpoint`. The handler was its only production caller.
Update the module docstring: the timeline, invariant 1 (now "the frozen
prefix is never mutated"), and why a cached Read stays verbatim. The
manager logic is unchanged.
- `headroom/proxy/handlers/anthropic.py`: forward `maturation.messages`
as returned, with no relocation.
- `headroom/config.py`, `headroom/proxy/models.py`,
`headroom/cli/proxy.py`, `headroom/rollout.py`,
`docs/content/docs/proxy.mdx`: these descriptions said held Reads are
kept out of the prefix cache, which is no longer true. They now say
Reads stay verbatim and compress once quiet unless the provider cache
already holds them.
- `tests/test_read_maturation_handler_nobust.py`:
`test_verbatim_read_never_cache_written_before_maturation` asserted the
old invariant: the verbatim Read is never forwarded inside the cached
prefix. That invariant is exactly the bug, so the test is now
`test_held_read_keeps_client_breakpoint_then_matures`. It asserts that
the client's breakpoint is forwarded where the client put it on every
turn. It still checks that the Read is held and then matures in order:
in this fixture nothing compresses, so the frozen count stays at the
Read. A comment in the test explains the change.
- `tests/test_read_maturation_handler_nobust.py`: new
`test_read_cached_with_its_turn_stays_verbatim`. It drives the real
handler for 7 turns with compressible Bash output around the Read, and
asserts three things: the breakpoints are kept on every turn, the frozen
count passes the held Read from the second request on, and the Read is
never rewritten.
- `tests/test_read_maturation.py`: new
`test_held_read_matures_only_outside_frozen_prefix`. A held Read matures
at quiesce when it is in the live zone and stays verbatim when the
frozen prefix covers it. The test for a client breakpoint on a fresh
Read now asserts the breakpoint stays put and survives maturation.
`TestBreakpointRelocation` is removed along with the helper.
- `tests/test_cache_ttl_preserved.py`: drop the two relocation TTL
tests. The tool-search deferral TTL tests stay.
- `tests/test_live/test_live_maturation.py`: request A asserts the held
Read is cache-written with the client's tail marker. Request B, with the
Read inside the confirmed prefix, asserts it is not matured and that A's
cached prefix is read back.

## Testing

- [x] Unit tests pass (`pytest`) - the read-maturation, breakpoint and
rollout files below, not the whole suite
- [x] Linting passes (`ruff check .`) - on the changed files, plus `ruff
format --check`
- [x] Type checking passes (`mypy headroom`) - on the six changed
modules
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

New and updated tests with `headroom/` at `main` (befdb52f):

```text
$ uv run --frozen --extra dev pytest tests/test_read_maturation.py tests/test_read_maturation_handler_nobust.py tests/test_cache_ttl_preserved.py -q
E   AssertionError: client cache breakpoint moved or dropped on turn(s) [0, 1, 2, 3, 4]
E   AssertionError: client cache breakpoint moved or dropped on turn(s) [0, 1, 2, 3, 4]
FAILED tests/test_read_maturation_handler_nobust.py::test_held_read_keeps_client_breakpoint_then_matures
FAILED tests/test_read_maturation_handler_nobust.py::test_read_cached_with_its_turn_stays_verbatim
========================= 2 failed, 33 passed in 9.68s =========================
```

With this change:

```text
$ uv run --frozen --extra dev pytest tests/gateway/test_cache_breakpoint_budget.py tests/test_cache_breakpoint_diagnostics.py \
    tests/test_cache_control_move_bust.py tests/test_issue_2671_block_growth_cache.py tests/test_rollout.py \
    tests/test_testing_harness.py tests/test_read_maturation.py tests/test_read_maturation_handler_nobust.py \
    tests/test_cache_ttl_preserved.py tests/test_live/test_live_maturation.py -q
======================== 170 passed, 2 skipped in 9.11s ========================

$ uv run --frozen ruff check <10 changed .py files>
All checks passed!
$ uv run --frozen ruff format --check <10 changed .py files>
10 files already formatted
$ uv run --frozen --extra dev mypy headroom/transforms/read_maturation.py headroom/proxy/handlers/anthropic.py \
    headroom/config.py headroom/cli/proxy.py headroom/proxy/models.py headroom/rollout.py --ignore-missing-imports
Success: no issues found in 6 source files
```

## Real Behavior Proof

- Environment: macOS 27 (arm64), Python 3.11.13, `uv run --frozen
--extra dev`, this branch and `main` (befdb52f). Live evidence comes
from a Claude Code session behind a 0.39.0 proxy with
`HEADROOM_READ_MATURATION=1` on the beta channel.
- Exact command / steps: a probe script drives
`create_app(ProxyConfig(read_maturation=True, mode="token",
cache_enabled=True))` through `TestClient` for 8 turns of one Claude
Code-shaped conversation. The conversation has two 1h system
breakpoints, client breakpoints on the last block of the last two
messages, compressible Bash JSON output, a 3 KB line-numbered Read at
message 10, and then quiet turns. The upstream mock reports cache reads
up to the last forwarded breakpoint, using the prefix tracker's own
estimator. The probe prints `count_cache_breakpoints` for each request
in and out, the frozen count maturation saw, and the Read's forwarded
form.
- Observed result: on `main`, every held turn forwards 3 of the client's
4 breakpoints, with the tail breakpoint 1, 3, 5, 7, 9 messages from the
end. The frozen count never passes the Read (10), and the Read matures
on turn 5. On this branch, all 4 breakpoints reach the tail on every
turn, the frozen count passes the Read from the second request on (11,
13, 15, ...), and the Read stays verbatim. In production, the same hold
logged `event=cache_breakpoints ... dropped=true tail_grew=true` on
every held request, with the cost figures in the Description.

```text
main (befdb52f):
INFO turn 0: in_total=4 out_total=3 in_last_tail=0 out_last_tail=1 maturation_frozen=0 read=verbatim
INFO turn 1: in_total=4 out_total=3 in_last_tail=0 out_last_tail=3 maturation_frozen=10 read=verbatim
INFO turn 2: in_total=4 out_total=3 in_last_tail=0 out_last_tail=5 maturation_frozen=10 read=verbatim
INFO turn 3: in_total=4 out_total=3 in_last_tail=0 out_last_tail=7 maturation_frozen=10 read=verbatim
INFO turn 4: in_total=4 out_total=3 in_last_tail=0 out_last_tail=9 maturation_frozen=10 read=verbatim
INFO turn 5: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=10 read=matured
INFO turn 6: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=21 read=matured
INFO turn 7: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=23 read=matured

this branch:
INFO turn 0: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=0 read=verbatim
INFO turn 1: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=11 read=verbatim
INFO turn 2: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=13 read=verbatim
INFO turn 3: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=15 read=verbatim
INFO turn 4: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=17 read=verbatim
INFO turn 5: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=19 read=verbatim
INFO turn 6: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=21 read=verbatim
INFO turn 7: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=23 read=verbatim
```

- Not tested: `tests/test_live/test_live_maturation.py` against the real
API (no `ANTHROPIC_API_KEY` here; it was updated and skips). This exact
branch has not run against live traffic. The live figures above come
from the 0.39.0 wheel. The full `pytest` suite was not run, only the
files listed under Testing.

## Runtime Rollout Safety

- Rollout-managed feature(s): read_maturation (behavior change inside
the existing feature; no new feature).
- Minimum rollout channel: Beta (unchanged - inherits the feature's
existing gate).
- Stable/default behavior changed: No. read_maturation is beta-gated and
off by default, and none of this code runs when it is off. With it on,
held Reads now keep the client's cache breakpoints and are cached with
their turn, so a Read the provider cache already covers is no longer
matured.
- Kill switch / disable path: disabling read_maturation
(`HEADROOM_READ_MATURATION=0`, or drop `--read-maturation`) disables
this with it; the old breakpoint relocation has no separate switch
because it only added cost.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: revert the commit; no persisted format or config
surface changes, and matured-marker session state is unchanged.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- `relocate_cache_breakpoint` was public in
`headroom.transforms.read_maturation` but not re-exported. A repo-wide
grep found no caller outside the handler and tests.
- `count_cache_breakpoints` and the `event=cache_breakpoints` log line
already flagged every held turn as `dropped=true`. With this change
those turns log `dropped=false`.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3977"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-07 20:57:44 -05:00

49 lines
2.0 KiB
Python

"""Re-anchored cache breakpoints must keep the client's TTL.
``normalize_message_cache_control`` deliberately preserves an explicit
``cache_control.ttl`` so a client on Anthropic's 1h cache isn't silently
downgraded to the 5-minute default (#2375). Tool-search deferral also strips a
breakpoint and re-places it, and used to hardcode a bare ephemeral marker —
undoing that guarantee. A downgrade is invisible (the request still succeeds)
and costs a full prefix re-write on every gap past 5 minutes, so it needs a test
rather than a comment.
"""
from typing import Any
from headroom.proxy.helpers import inject_tool_search_deferral
TTL_1H = {"type": "ephemeral", "ttl": "1h"}
def _tools(marker: dict[str, Any] | None) -> list[dict[str, Any]]:
# Needs >= _TOOL_SEARCH_MIN_TOOLS (12) to trigger, with one core tool resident
# and the tools-array breakpoint riding on a tool that will be deferred.
tools: list[dict[str, Any]] = [{"name": "read", "description": "core", "input_schema": {}}]
for i in range(12):
t: dict[str, Any] = {"name": f"rare_{i}", "description": "rare", "input_schema": {}}
if marker is not None and i == 11:
t["cache_control"] = marker
tools.append(t)
return tools
def _tool_markers(tools: Any) -> list[dict[str, Any]]:
return [t["cache_control"] for t in tools if isinstance(t, dict) and "cache_control" in t]
def test_tool_search_deferral_keeps_ttl() -> None:
out = inject_tool_search_deferral(_tools(TTL_1H))
assert out is not _tools(TTL_1H), "deferral did not apply — fixture no longer triggers it"
assert _tool_markers(out) == [TTL_1H], "tools breakpoint lost the 1h ttl"
def test_tool_search_deferral_defaults_to_5m() -> None:
out = inject_tool_search_deferral(_tools({"type": "ephemeral"}))
assert _tool_markers(out) == [{"type": "ephemeral"}]
def test_tool_search_deferral_no_breakpoint_adds_none() -> None:
# Nothing was stripped, so nothing should be invented.
assert _tool_markers(inject_tool_search_deferral(_tools(None))) == []