Files
headroom/tests/test_cross_turn_cache_safety.py
gglucassandClaude Fable 5 aebe98954e fix(cache): keep cached-prefix replay after background recompression (#3380)
## Description

Since the non-inflation size bound added in #3052 (shipped 0.36.x), the
Anthropic proxy token path repeatedly busts the provider prompt cache:
the bound declines the cached-prefix replay exactly when background
compression lands a SMALLER form of an already-forwarded message, so the
forwarded bytes change mid-history and the provider cache misses from
the first changed byte. Steady state on a busy session is a
160-210k-token cache re-write every 1-2 requests (measured across a
0.35.0 -> 0.37.0 swap on identical workload: dollar cache-hit rate 90%
-> 52%, billable input $/M up 2.2x; full data in #3379).

Reworked per review (thanks @chopratejas - the direction is yours):
rather than disabling the size bound wholesale, `overlay_cached_prefix`
now takes `confirmed_frozen_count` and splits at that floor.

- Inside the provider-confirmed prefix: replay is unconditional. The
replay source is exactly what the provider hashed, so changing those
bytes can only bust the cache; an "inflating" replay there bills at the
~0.1x cache-read rate and is still the cheaper request.
- Beyond the floor: the size bound arbitrates each turn exactly as on
main - a shrinking replay repairs freeze drift (#1850), an inflating one
is declined so a fresh compression improvement reaches the wire.
- When the provider count collapses (cold cache, TTL lapse) the floor
collapses with it and every accumulated improvement lands at once: the
natural re-baselining that bounds long-session growth (the #3026
direction of travel) by the TTL-lapse cadence instead of pinning
forever.

The #3052 alignment/append-only guards stay enforced unconditionally.

The OpenAI chat path now carries the same floor (reviewer question: is
the root cause live there too?). It is: the tracker is provider-generic
and is fed OpenAI's own `prompt_tokens_details.cached_tokens`, the
forwarded-bytes feedback loop is identical, both handlers compress
through the same ContentRouter, and OpenAI's automatic prompt caching
busts on the first changed byte exactly like Anthropic's. Exposure is
narrower - the token path's `FREEZE_POLICY_REPLAYABLE` already stops
most recompression of already-forwarded messages - but not zero: an
entry evicted from the local byte-replay cache drops out of that freeze,
and cache mode freezes from the tracker only. Remaining callers with no
provider-confirmed count in scope (sidecar, `openai.py:9940`) pass
nothing and keep today's fully size-bounded behavior; `bedrock.py` /
`gemini.py` never call `finalize_turn` at all, tracked in #3391.

Closes #3379

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `overlay_cached_prefix` gains keyword-only `confirmed_frozen_count:
int | None = None`: an unconditional-replay floor for the
provider-confirmed prefix; the size bound still governs everything
beyond it, and alignment guards are untouched.
- `finalize_turn` forwards the floor (default `None`, so all existing
callers are unchanged).
- The OpenAI chat call site passes the same floor, snapshotted before
`prepare_turn`'s replayable clamp (that count runs over the LOCAL
byte-replay cache and can exceed what the provider confirmed; a cold
prefix zeroes it, so cold re-baselining matches Anthropic's).
- The Anthropic confirmed-clamp call site stashes the pre-clamp tracker
count (the post-`prepare_turn` min under-floors when the local
byte-replay cache evicted entries) and passes it; the stash defaults to
0 on paths that never compute a tracker count (backpressure).
- Tests: floor replays a recompressed confirmed prefix; improvement
beyond the floor lands while the confirmed region replays;
originals-drift beyond the floor is still repaired when the replay
shrinks; alignment guards unaffected by the floor; block-append within
the floor; `finalize_turn` pass-through; the reviewer-requested
multi-turn harness test - a background improvement to an
already-forwarded message never busts the warm cache across 8 turns and
lands on the first cold turn; the reviewer-requested growth bound -
forwarded size grows by only its own delta each warm turn and the first
cold turn re-baselines to exactly the fresh pipeline's output; and an
end-to-end OpenAI chat test asserting the provider-confirmed prefix is
replayed even when the replay inflates (fails without the new argument).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_cache_prefix_overlay.py tests/test_session_engine.py tests/test_cross_turn_cache_safety.py tests/test_cache_control_move_bust.py -q
============================== 58 passed in 0.44s ==============================
$ uvx ruff check headroom/cache/prefix_tracker.py headroom/proxy/session_engine.py headroom/proxy/handlers/anthropic.py tests/test_cache_prefix_overlay.py tests/test_session_engine.py tests/test_cross_turn_cache_safety.py
All checks passed!
$ uv run --frozen --extra dev mypy headroom/cache/prefix_tracker.py headroom/proxy/session_engine.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: macOS 15 (arm64), Python 3.12; regression measured on the
released headroom-ai 0.37.0 under real Claude Code traffic through
`headroom proxy` (token mode); this branch validated with the repo test
suite.
- Exact command / steps: diffed per-request PERF logs across a 0.35.0 ->
0.37.0 swap on the same machine and workload (sequences and aggregates
in #3379); then ran the pytest/ruff/mypy commands above on this branch.
A deployment carrying the unconditional-replay behavior on the confirmed
prefix restored the healthy pattern live: the broken wheel capped cached
reads at 157k across 32 large requests (prefix perpetually re-written)
while the patched deployment accumulated to 305k cached reads within 15
minutes.
- Observed result: 58 passed including the new multi-turn harness test;
on stock 0.37.0 every background-compression completion produced a
cache_read collapse to the ~7.7k tools+system head plus a 160-210k
cache_write.
- Not tested: a live OpenAI `cached_tokens` comparison for the OpenAI
half (the Anthropic measurement is what #3379 carries; the OpenAI change
is argued from shared mechanism plus an end-to-end handler test, not
from its own dollar data). A live provider run of this exact branch (the
wheel comparison plus the live patched-deployment measurement isolate
the bound as the trigger; the unit and multi-turn regressions reproduce
decline and fix deterministically). `mypy headroom` was run only on the
two touched modules, not the whole package.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: On the Anthropic proxy
confirmed-clamp path and the OpenAI chat path, both of which now replay
the provider-confirmed prefix unconditionally (the pre-0.36 posture)
while keeping the size bound beyond the confirmed floor. The overlay's
default and every other caller keep the fully size-bounded behavior.
- Kill switch / disable path: remove the `confirmed_frozen_count=`
argument at the two call sites (anthropic.py, openai.py chat); the
parameter defaults to `None`.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert the commits; the parameter is additive and
default-preserving.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

Lineage for review: #1850 made freeze forward the cached prefix
byte-identical, #3052 added alignment guards plus the size bound after
#3026's misalignment blowup, #3271 unified both modes on the session
engine. The first cut of this PR disabled the bound on the
confirmed-clamp path wholesale; the reviewed rework scopes unconditional
replay to the provider-confirmed prefix, which additionally gives
cold-lapse re-baselining (improvements land whenever the floor
collapses), addressing the growth analysis raised in review. Economics
discussion in the review thread.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-02 16:50:58 +05:30

283 lines
12 KiB
Python

"""Cross-turn cache-safety invariant — the test class that catches cache busts.
Why the +150%-cache_create / +41%-cost bug slipped through: every prior cache
test was SINGLE-turn and used a fake tracker, so nobody exercised the real
multi-turn invariant that actually governs prompt-cache cost:
Across append-only turns, the forwarded prefix must stay BYTE-IDENTICAL to
what was forwarded (and cached) last turn — otherwise the provider re-creates
the whole suffix (a cache bust) instead of reading it.
This simulates the provider's prefix cache (longest byte-identical leading run of
messages = cache_read; the rest = cache_create) and drives the REAL
``PrefixCacheTracker`` + the freeze model + ``overlay_cached_prefix`` over several
turns. It asserts the invariant directly, and proves the guard is load-bearing:
WITHOUT the overlay the freeze forwards the agent's original bytes and busts every
turn; WITH it the prefix stays stable.
"""
from headroom.cache.prefix_tracker import (
PrefixCacheTracker,
PrefixFreezeConfig,
overlay_cached_prefix,
)
def _toklen(m) -> int:
return max(1, len(str(m.get("content", ""))))
def _compress(m):
"""Deterministic stand-in for a real compressor (kompress is deterministic
per content via the result cache): shrink the content by half."""
c = str(m.get("content", ""))
return {**m, "content": c[: max(1, len(c) // 2)]}
def _apply_freeze(original, frozen_count):
"""Faithful model of pipeline.apply()'s freeze: the frozen prefix is
forwarded as the agent's ORIGINAL bytes; everything else is compressed.
(Mirrors content_router.py: `result_slots[i] = message` for i < frozen.)"""
return [
(original[i] if i < frozen_count else _compress(original[i])) for i in range(len(original))
]
def _provider_cache_read(forwarded, prev_forwarded):
"""Longest byte-identical leading run of messages the provider can serve from
cache, in tokens. A single differing message breaks the prefix (bust)."""
if not prev_forwarded:
return 0
matched = 0
for a, b in zip(forwarded, prev_forwarded):
if a == b:
matched += _toklen(a)
else:
break
return matched
def _drive_turns(*, use_overlay: bool, turns: int = 5):
"""Return per-turn (expected_cache_read, actual_cache_read). A bust is any
turn where actual < expected (the previously-cached prefix wasn't reused)."""
# min_cached_tokens=0 so freeze activates from turn 2 regardless of size.
tracker = PrefixCacheTracker("anthropic", PrefixFreezeConfig(min_cached_tokens=0))
convo: list[dict] = []
prev_forwarded: list[dict] | None = None
out = []
for t in range(1, turns + 1):
# Append-only growth: one new large tool output per turn.
convo = convo + [{"role": "user", "content": f"tool-output-turn-{t}:" + "X" * 400}]
frozen = tracker.get_frozen_message_count()
forwarded = _apply_freeze(convo, frozen)
if use_overlay:
forwarded = overlay_cached_prefix(
forwarded,
convo,
tracker.get_last_original_messages(),
tracker.get_last_forwarded_messages(),
)
expected_read = sum(_toklen(m) for m in prev_forwarded) if prev_forwarded else 0
actual_read = _provider_cache_read(forwarded, prev_forwarded)
out.append((expected_read, actual_read))
counts = [_toklen(m) for m in forwarded]
write = sum(counts) - actual_read
tracker.update_from_response(
actual_read, write, forwarded, message_token_counts=counts, original_messages=convo
)
prev_forwarded = forwarded
return out
def test_freeze_busts_cache_every_turn_without_overlay():
"""Proves the test is load-bearing: the raw freeze path busts the cache."""
results = _drive_turns(use_overlay=False)
# From turn 2 on, a hit was expected but the prefix broke (actual < expected).
busts = [exp > act for (exp, act) in results[1:]]
assert any(busts), "expected the un-fixed freeze path to bust the prefix cache"
def test_overlay_keeps_prefix_byte_identical_no_bust():
"""The fix: every turn reuses the full previously-cached prefix — no bust."""
results = _drive_turns(use_overlay=True)
for exp, act in results[1:]:
assert act >= exp, (
f"cache bust: expected to read {exp} cached tokens but only read {act} "
"— forwarded prefix diverged from last turn"
)
def test_cache_create_stays_bounded_to_the_delta_with_overlay():
"""Cost proxy: with the fix, per-turn cache_create ≈ the new delta only, not
the whole re-created prefix (which is what drove +150% cache_create)."""
tracker = PrefixCacheTracker("anthropic", PrefixFreezeConfig(min_cached_tokens=0))
convo: list[dict] = []
prev_forwarded: list[dict] | None = None
creates = []
for t in range(1, 6):
convo = convo + [{"role": "user", "content": f"turn-{t}:" + "X" * 400}]
frozen = tracker.get_frozen_message_count()
forwarded = overlay_cached_prefix(
_apply_freeze(convo, frozen),
convo,
tracker.get_last_original_messages(),
tracker.get_last_forwarded_messages(),
)
read = _provider_cache_read(forwarded, prev_forwarded)
counts = [_toklen(m) for m in forwarded]
create = sum(counts) - read
creates.append(create)
tracker.update_from_response(
read, create, forwarded, message_token_counts=counts, original_messages=convo
)
prev_forwarded = forwarded
# Steady-state cache_create per turn should be ~one delta message, NOT growing
# with conversation length. Assert the last turn creates no more than the
# first (which had no cache to reuse).
assert creates[-1] <= creates[0] + 1
def test_background_improvement_never_busts_warm_cache_and_lands_on_cold():
"""The #3379 regression and its fix, end to end: a background improvement
to an already-forwarded message must NOT change warm-cache bytes (the
confirmed floor overrides it), and must land the moment the provider
count collapses (cold cache re-baseline) - so long-session growth is
bounded by the TTL-lapse cadence instead of unbounded pinning."""
tracker = PrefixCacheTracker("anthropic", PrefixFreezeConfig(min_cached_tokens=0))
convo: list[dict] = []
prev_forwarded: list[dict] | None = None
improved_from_turn = 4
def pipeline(original, frozen, turn):
out = _apply_freeze(original, frozen)
if turn >= improved_from_turn:
# Background compression later found a much better form for the
# very first message (this PR's stated trigger).
out = [{**out[0], "content": str(out[0]["content"])[:4]}] + out[1:]
return out
for t in range(1, 9):
convo = convo + [{"role": "user", "content": f"tool-output-turn-{t}:" + "X" * 400}]
frozen = tracker.get_frozen_message_count()
forwarded = overlay_cached_prefix(
pipeline(convo, frozen, t),
convo,
tracker.get_last_original_messages(),
tracker.get_last_forwarded_messages(),
confirmed_frozen_count=frozen,
)
expected = sum(_toklen(m) for m in prev_forwarded) if prev_forwarded else 0
actual = _provider_cache_read(forwarded, prev_forwarded)
assert actual >= expected, f"turn {t}: warm-cache bust ({actual} < {expected})"
counts = [_toklen(m) for m in forwarded]
tracker.update_from_response(
actual,
sum(counts) - actual,
forwarded,
message_token_counts=counts,
original_messages=convo,
)
prev_forwarded = forwarded
# Cold cache: the provider count collapses, the floor collapses with it,
# and the accumulated improvement finally reaches the wire.
convo = convo + [{"role": "user", "content": "tool-output-turn-9:" + "X" * 400}]
cold = overlay_cached_prefix(
pipeline(convo, 0, 9),
convo,
tracker.get_last_original_messages(),
tracker.get_last_forwarded_messages(),
confirmed_frozen_count=0,
)
assert len(str(cold[0]["content"])) < len(str(prev_forwarded[0]["content"])), (
"cold-cache turn must re-baseline: the background improvement lands"
)
def test_prefix_growth_bounded_and_rebaselines_on_the_cold_turn():
"""The pinned-prefix cost of the confirmed floor, measured.
While the cache stays warm the confirmed prefix is pinned to its
first-forwarded form, so later compression improvements to it do not land -
a real context-window cost, not just a cache benefit. This asserts the two
bounds that keep it from becoming #3026-style growth:
1. Warm turns cannot compound: per-turn growth of the forwarded request is
exactly the new message, never the re-inflation of the pinned prefix.
2. The first cold turn re-baselines to precisely what the fresh pipeline
would forward, so the pinned overhead is bounded by the TTL-lapse
cadence rather than accumulating for the life of the session.
Worst case for the floor: background compression keeps finding a better
form for EVERY message it has already seen, improving each turn.
"""
tracker = PrefixCacheTracker("anthropic", PrefixFreezeConfig(min_cached_tokens=0))
convo: list[dict] = []
prev_forwarded: list[dict] | None = None
def pipeline(original, frozen, turn):
out = _apply_freeze(original, frozen)
# Later turns find better forms for everything already forwarded.
keep = max(4, 200 // turn)
return [{**m, "content": str(m["content"])[:keep]} for m in out[:-1]] + out[-1:]
warm_ratios = []
for t in range(1, 9):
convo = convo + [{"role": "user", "content": f"tool-output-turn-{t}:" + "X" * 400}]
frozen = tracker.get_frozen_message_count()
fresh = pipeline(convo, frozen, t)
forwarded = overlay_cached_prefix(
fresh,
convo,
tracker.get_last_original_messages(),
tracker.get_last_forwarded_messages(),
confirmed_frozen_count=frozen,
)
total = sum(_toklen(m) for m in forwarded)
ideal = sum(_toklen(m) for m in fresh)
if prev_forwarded is not None:
prev_total = sum(_toklen(m) for m in prev_forwarded)
assert total - prev_total <= _toklen(forwarded[-1]), (
f"turn {t}: forwarded grew by {total - prev_total} tokens, more than the "
f"{_toklen(forwarded[-1])}-token delta message - the prefix compounded"
)
warm_ratios.append(total / ideal)
actual = _provider_cache_read(forwarded, prev_forwarded)
counts = [_toklen(m) for m in forwarded]
tracker.update_from_response(
actual,
sum(counts) - actual,
forwarded,
message_token_counts=counts,
original_messages=convo,
)
prev_forwarded = forwarded
# Measured on this worst-case model, forwarded/fresh over turns 2-8:
# 1.35, 1.84, 2.33, 2.83, 3.35, 3.88, 4.35 - the pinned prefix does widen
# against an ideal pipeline while the cache stays warm (the cost the floor
# buys the cache hit with). What the two assertions bound is the shape:
# each warm turn adds only its own delta, and the cold turn below zeroes
# the gap outright.
# Cold turn: the provider count collapses, the floor collapses with it, and
# the forwarded request is exactly the fresh pipeline's output again.
convo = convo + [{"role": "user", "content": "tool-output-turn-9:" + "X" * 400}]
fresh_cold = pipeline(convo, 0, 9)
cold = overlay_cached_prefix(
fresh_cold,
convo,
tracker.get_last_original_messages(),
tracker.get_last_forwarded_messages(),
confirmed_frozen_count=0,
)
assert sum(_toklen(m) for m in cold) == sum(_toklen(m) for m in fresh_cold), (
"cold turn must re-baseline to the fresh pipeline's output"
)
assert warm_ratios[-1] > warm_ratios[0], (
"harness sanity: the warm-turn model must actually be improving the "
"prefix, otherwise the re-baseline assertion proves nothing"
)