mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-23 02:49:17 +08:00
## Description Since the non-inflation size bound added in #3052 (shipped 0.36.x), the Anthropic proxy token path repeatedly busts the provider prompt cache: the bound declines the cached-prefix replay exactly when background compression lands a SMALLER form of an already-forwarded message, so the forwarded bytes change mid-history and the provider cache misses from the first changed byte. Steady state on a busy session is a 160-210k-token cache re-write every 1-2 requests (measured across a 0.35.0 -> 0.37.0 swap on identical workload: dollar cache-hit rate 90% -> 52%, billable input $/M up 2.2x; full data in #3379). Reworked per review (thanks @chopratejas - the direction is yours): rather than disabling the size bound wholesale, `overlay_cached_prefix` now takes `confirmed_frozen_count` and splits at that floor. - Inside the provider-confirmed prefix: replay is unconditional. The replay source is exactly what the provider hashed, so changing those bytes can only bust the cache; an "inflating" replay there bills at the ~0.1x cache-read rate and is still the cheaper request. - Beyond the floor: the size bound arbitrates each turn exactly as on main - a shrinking replay repairs freeze drift (#1850), an inflating one is declined so a fresh compression improvement reaches the wire. - When the provider count collapses (cold cache, TTL lapse) the floor collapses with it and every accumulated improvement lands at once: the natural re-baselining that bounds long-session growth (the #3026 direction of travel) by the TTL-lapse cadence instead of pinning forever. The #3052 alignment/append-only guards stay enforced unconditionally. The OpenAI chat path now carries the same floor (reviewer question: is the root cause live there too?). It is: the tracker is provider-generic and is fed OpenAI's own `prompt_tokens_details.cached_tokens`, the forwarded-bytes feedback loop is identical, both handlers compress through the same ContentRouter, and OpenAI's automatic prompt caching busts on the first changed byte exactly like Anthropic's. Exposure is narrower - the token path's `FREEZE_POLICY_REPLAYABLE` already stops most recompression of already-forwarded messages - but not zero: an entry evicted from the local byte-replay cache drops out of that freeze, and cache mode freezes from the tracker only. Remaining callers with no provider-confirmed count in scope (sidecar, `openai.py:9940`) pass nothing and keep today's fully size-bounded behavior; `bedrock.py` / `gemini.py` never call `finalize_turn` at all, tracked in #3391. Closes #3379 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `overlay_cached_prefix` gains keyword-only `confirmed_frozen_count: int | None = None`: an unconditional-replay floor for the provider-confirmed prefix; the size bound still governs everything beyond it, and alignment guards are untouched. - `finalize_turn` forwards the floor (default `None`, so all existing callers are unchanged). - The OpenAI chat call site passes the same floor, snapshotted before `prepare_turn`'s replayable clamp (that count runs over the LOCAL byte-replay cache and can exceed what the provider confirmed; a cold prefix zeroes it, so cold re-baselining matches Anthropic's). - The Anthropic confirmed-clamp call site stashes the pre-clamp tracker count (the post-`prepare_turn` min under-floors when the local byte-replay cache evicted entries) and passes it; the stash defaults to 0 on paths that never compute a tracker count (backpressure). - Tests: floor replays a recompressed confirmed prefix; improvement beyond the floor lands while the confirmed region replays; originals-drift beyond the floor is still repaired when the replay shrinks; alignment guards unaffected by the floor; block-append within the floor; `finalize_turn` pass-through; the reviewer-requested multi-turn harness test - a background improvement to an already-forwarded message never busts the warm cache across 8 turns and lands on the first cold turn; the reviewer-requested growth bound - forwarded size grows by only its own delta each warm turn and the first cold turn re-baselines to exactly the fresh pipeline's output; and an end-to-end OpenAI chat test asserting the provider-confirmed prefix is replayed even when the replay inflates (fails without the new argument). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest tests/test_cache_prefix_overlay.py tests/test_session_engine.py tests/test_cross_turn_cache_safety.py tests/test_cache_control_move_bust.py -q ============================== 58 passed in 0.44s ============================== $ uvx ruff check headroom/cache/prefix_tracker.py headroom/proxy/session_engine.py headroom/proxy/handlers/anthropic.py tests/test_cache_prefix_overlay.py tests/test_session_engine.py tests/test_cross_turn_cache_safety.py All checks passed! $ uv run --frozen --extra dev mypy headroom/cache/prefix_tracker.py headroom/proxy/session_engine.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: macOS 15 (arm64), Python 3.12; regression measured on the released headroom-ai 0.37.0 under real Claude Code traffic through `headroom proxy` (token mode); this branch validated with the repo test suite. - Exact command / steps: diffed per-request PERF logs across a 0.35.0 -> 0.37.0 swap on the same machine and workload (sequences and aggregates in #3379); then ran the pytest/ruff/mypy commands above on this branch. A deployment carrying the unconditional-replay behavior on the confirmed prefix restored the healthy pattern live: the broken wheel capped cached reads at 157k across 32 large requests (prefix perpetually re-written) while the patched deployment accumulated to 305k cached reads within 15 minutes. - Observed result: 58 passed including the new multi-turn harness test; on stock 0.37.0 every background-compression completion produced a cache_read collapse to the ~7.7k tools+system head plus a 160-210k cache_write. - Not tested: a live OpenAI `cached_tokens` comparison for the OpenAI half (the Anthropic measurement is what #3379 carries; the OpenAI change is argued from shared mechanism plus an end-to-end handler test, not from its own dollar data). A live provider run of this exact branch (the wheel comparison plus the live patched-deployment measurement isolate the bound as the trigger; the unit and multi-turn regressions reproduce decline and fix deterministically). `mypy headroom` was run only on the two touched modules, not the whole package. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: Stable. - Stable/default behavior changed: On the Anthropic proxy confirmed-clamp path and the OpenAI chat path, both of which now replay the provider-confirmed prefix unconditionally (the pre-0.36 posture) while keeping the size bound beyond the confirmed floor. The overlay's default and every other caller keep the fully size-bounded behavior. - Kill switch / disable path: remove the `confirmed_frozen_count=` argument at the two call sites (anthropic.py, openai.py chat); the parameter defaults to `None`. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert the commits; the parameter is additive and default-preserving. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable. ## Additional Notes Lineage for review: #1850 made freeze forward the cached prefix byte-identical, #3052 added alignment guards plus the size bound after #3026's misalignment blowup, #3271 unified both modes on the session engine. The first cut of this PR disabled the bound on the confirmed-clamp path wholesale; the reviewed rework scopes unconditional replay to the provider-confirmed prefix, which additionally gives cold-lapse re-baselining (improvements land whenever the floor collapses), addressing the growth analysis raised in review. Economics discussion in the review thread. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
283 lines
12 KiB
Python
283 lines
12 KiB
Python
"""Cross-turn cache-safety invariant — the test class that catches cache busts.
|
|
|
|
Why the +150%-cache_create / +41%-cost bug slipped through: every prior cache
|
|
test was SINGLE-turn and used a fake tracker, so nobody exercised the real
|
|
multi-turn invariant that actually governs prompt-cache cost:
|
|
|
|
Across append-only turns, the forwarded prefix must stay BYTE-IDENTICAL to
|
|
what was forwarded (and cached) last turn — otherwise the provider re-creates
|
|
the whole suffix (a cache bust) instead of reading it.
|
|
|
|
This simulates the provider's prefix cache (longest byte-identical leading run of
|
|
messages = cache_read; the rest = cache_create) and drives the REAL
|
|
``PrefixCacheTracker`` + the freeze model + ``overlay_cached_prefix`` over several
|
|
turns. It asserts the invariant directly, and proves the guard is load-bearing:
|
|
WITHOUT the overlay the freeze forwards the agent's original bytes and busts every
|
|
turn; WITH it the prefix stays stable.
|
|
"""
|
|
|
|
from headroom.cache.prefix_tracker import (
|
|
PrefixCacheTracker,
|
|
PrefixFreezeConfig,
|
|
overlay_cached_prefix,
|
|
)
|
|
|
|
|
|
def _toklen(m) -> int:
|
|
return max(1, len(str(m.get("content", ""))))
|
|
|
|
|
|
def _compress(m):
|
|
"""Deterministic stand-in for a real compressor (kompress is deterministic
|
|
per content via the result cache): shrink the content by half."""
|
|
c = str(m.get("content", ""))
|
|
return {**m, "content": c[: max(1, len(c) // 2)]}
|
|
|
|
|
|
def _apply_freeze(original, frozen_count):
|
|
"""Faithful model of pipeline.apply()'s freeze: the frozen prefix is
|
|
forwarded as the agent's ORIGINAL bytes; everything else is compressed.
|
|
(Mirrors content_router.py: `result_slots[i] = message` for i < frozen.)"""
|
|
return [
|
|
(original[i] if i < frozen_count else _compress(original[i])) for i in range(len(original))
|
|
]
|
|
|
|
|
|
def _provider_cache_read(forwarded, prev_forwarded):
|
|
"""Longest byte-identical leading run of messages the provider can serve from
|
|
cache, in tokens. A single differing message breaks the prefix (bust)."""
|
|
if not prev_forwarded:
|
|
return 0
|
|
matched = 0
|
|
for a, b in zip(forwarded, prev_forwarded):
|
|
if a == b:
|
|
matched += _toklen(a)
|
|
else:
|
|
break
|
|
return matched
|
|
|
|
|
|
def _drive_turns(*, use_overlay: bool, turns: int = 5):
|
|
"""Return per-turn (expected_cache_read, actual_cache_read). A bust is any
|
|
turn where actual < expected (the previously-cached prefix wasn't reused)."""
|
|
# min_cached_tokens=0 so freeze activates from turn 2 regardless of size.
|
|
tracker = PrefixCacheTracker("anthropic", PrefixFreezeConfig(min_cached_tokens=0))
|
|
convo: list[dict] = []
|
|
prev_forwarded: list[dict] | None = None
|
|
out = []
|
|
for t in range(1, turns + 1):
|
|
# Append-only growth: one new large tool output per turn.
|
|
convo = convo + [{"role": "user", "content": f"tool-output-turn-{t}:" + "X" * 400}]
|
|
|
|
frozen = tracker.get_frozen_message_count()
|
|
forwarded = _apply_freeze(convo, frozen)
|
|
if use_overlay:
|
|
forwarded = overlay_cached_prefix(
|
|
forwarded,
|
|
convo,
|
|
tracker.get_last_original_messages(),
|
|
tracker.get_last_forwarded_messages(),
|
|
)
|
|
|
|
expected_read = sum(_toklen(m) for m in prev_forwarded) if prev_forwarded else 0
|
|
actual_read = _provider_cache_read(forwarded, prev_forwarded)
|
|
out.append((expected_read, actual_read))
|
|
|
|
counts = [_toklen(m) for m in forwarded]
|
|
write = sum(counts) - actual_read
|
|
tracker.update_from_response(
|
|
actual_read, write, forwarded, message_token_counts=counts, original_messages=convo
|
|
)
|
|
prev_forwarded = forwarded
|
|
return out
|
|
|
|
|
|
def test_freeze_busts_cache_every_turn_without_overlay():
|
|
"""Proves the test is load-bearing: the raw freeze path busts the cache."""
|
|
results = _drive_turns(use_overlay=False)
|
|
# From turn 2 on, a hit was expected but the prefix broke (actual < expected).
|
|
busts = [exp > act for (exp, act) in results[1:]]
|
|
assert any(busts), "expected the un-fixed freeze path to bust the prefix cache"
|
|
|
|
|
|
def test_overlay_keeps_prefix_byte_identical_no_bust():
|
|
"""The fix: every turn reuses the full previously-cached prefix — no bust."""
|
|
results = _drive_turns(use_overlay=True)
|
|
for exp, act in results[1:]:
|
|
assert act >= exp, (
|
|
f"cache bust: expected to read {exp} cached tokens but only read {act} "
|
|
"— forwarded prefix diverged from last turn"
|
|
)
|
|
|
|
|
|
def test_cache_create_stays_bounded_to_the_delta_with_overlay():
|
|
"""Cost proxy: with the fix, per-turn cache_create ≈ the new delta only, not
|
|
the whole re-created prefix (which is what drove +150% cache_create)."""
|
|
tracker = PrefixCacheTracker("anthropic", PrefixFreezeConfig(min_cached_tokens=0))
|
|
convo: list[dict] = []
|
|
prev_forwarded: list[dict] | None = None
|
|
creates = []
|
|
for t in range(1, 6):
|
|
convo = convo + [{"role": "user", "content": f"turn-{t}:" + "X" * 400}]
|
|
frozen = tracker.get_frozen_message_count()
|
|
forwarded = overlay_cached_prefix(
|
|
_apply_freeze(convo, frozen),
|
|
convo,
|
|
tracker.get_last_original_messages(),
|
|
tracker.get_last_forwarded_messages(),
|
|
)
|
|
read = _provider_cache_read(forwarded, prev_forwarded)
|
|
counts = [_toklen(m) for m in forwarded]
|
|
create = sum(counts) - read
|
|
creates.append(create)
|
|
tracker.update_from_response(
|
|
read, create, forwarded, message_token_counts=counts, original_messages=convo
|
|
)
|
|
prev_forwarded = forwarded
|
|
# Steady-state cache_create per turn should be ~one delta message, NOT growing
|
|
# with conversation length. Assert the last turn creates no more than the
|
|
# first (which had no cache to reuse).
|
|
assert creates[-1] <= creates[0] + 1
|
|
|
|
|
|
def test_background_improvement_never_busts_warm_cache_and_lands_on_cold():
|
|
"""The #3379 regression and its fix, end to end: a background improvement
|
|
to an already-forwarded message must NOT change warm-cache bytes (the
|
|
confirmed floor overrides it), and must land the moment the provider
|
|
count collapses (cold cache re-baseline) - so long-session growth is
|
|
bounded by the TTL-lapse cadence instead of unbounded pinning."""
|
|
tracker = PrefixCacheTracker("anthropic", PrefixFreezeConfig(min_cached_tokens=0))
|
|
convo: list[dict] = []
|
|
prev_forwarded: list[dict] | None = None
|
|
improved_from_turn = 4
|
|
|
|
def pipeline(original, frozen, turn):
|
|
out = _apply_freeze(original, frozen)
|
|
if turn >= improved_from_turn:
|
|
# Background compression later found a much better form for the
|
|
# very first message (this PR's stated trigger).
|
|
out = [{**out[0], "content": str(out[0]["content"])[:4]}] + out[1:]
|
|
return out
|
|
|
|
for t in range(1, 9):
|
|
convo = convo + [{"role": "user", "content": f"tool-output-turn-{t}:" + "X" * 400}]
|
|
frozen = tracker.get_frozen_message_count()
|
|
forwarded = overlay_cached_prefix(
|
|
pipeline(convo, frozen, t),
|
|
convo,
|
|
tracker.get_last_original_messages(),
|
|
tracker.get_last_forwarded_messages(),
|
|
confirmed_frozen_count=frozen,
|
|
)
|
|
expected = sum(_toklen(m) for m in prev_forwarded) if prev_forwarded else 0
|
|
actual = _provider_cache_read(forwarded, prev_forwarded)
|
|
assert actual >= expected, f"turn {t}: warm-cache bust ({actual} < {expected})"
|
|
counts = [_toklen(m) for m in forwarded]
|
|
tracker.update_from_response(
|
|
actual,
|
|
sum(counts) - actual,
|
|
forwarded,
|
|
message_token_counts=counts,
|
|
original_messages=convo,
|
|
)
|
|
prev_forwarded = forwarded
|
|
|
|
# Cold cache: the provider count collapses, the floor collapses with it,
|
|
# and the accumulated improvement finally reaches the wire.
|
|
convo = convo + [{"role": "user", "content": "tool-output-turn-9:" + "X" * 400}]
|
|
cold = overlay_cached_prefix(
|
|
pipeline(convo, 0, 9),
|
|
convo,
|
|
tracker.get_last_original_messages(),
|
|
tracker.get_last_forwarded_messages(),
|
|
confirmed_frozen_count=0,
|
|
)
|
|
assert len(str(cold[0]["content"])) < len(str(prev_forwarded[0]["content"])), (
|
|
"cold-cache turn must re-baseline: the background improvement lands"
|
|
)
|
|
|
|
|
|
def test_prefix_growth_bounded_and_rebaselines_on_the_cold_turn():
|
|
"""The pinned-prefix cost of the confirmed floor, measured.
|
|
|
|
While the cache stays warm the confirmed prefix is pinned to its
|
|
first-forwarded form, so later compression improvements to it do not land -
|
|
a real context-window cost, not just a cache benefit. This asserts the two
|
|
bounds that keep it from becoming #3026-style growth:
|
|
|
|
1. Warm turns cannot compound: per-turn growth of the forwarded request is
|
|
exactly the new message, never the re-inflation of the pinned prefix.
|
|
2. The first cold turn re-baselines to precisely what the fresh pipeline
|
|
would forward, so the pinned overhead is bounded by the TTL-lapse
|
|
cadence rather than accumulating for the life of the session.
|
|
|
|
Worst case for the floor: background compression keeps finding a better
|
|
form for EVERY message it has already seen, improving each turn.
|
|
"""
|
|
tracker = PrefixCacheTracker("anthropic", PrefixFreezeConfig(min_cached_tokens=0))
|
|
convo: list[dict] = []
|
|
prev_forwarded: list[dict] | None = None
|
|
|
|
def pipeline(original, frozen, turn):
|
|
out = _apply_freeze(original, frozen)
|
|
# Later turns find better forms for everything already forwarded.
|
|
keep = max(4, 200 // turn)
|
|
return [{**m, "content": str(m["content"])[:keep]} for m in out[:-1]] + out[-1:]
|
|
|
|
warm_ratios = []
|
|
for t in range(1, 9):
|
|
convo = convo + [{"role": "user", "content": f"tool-output-turn-{t}:" + "X" * 400}]
|
|
frozen = tracker.get_frozen_message_count()
|
|
fresh = pipeline(convo, frozen, t)
|
|
forwarded = overlay_cached_prefix(
|
|
fresh,
|
|
convo,
|
|
tracker.get_last_original_messages(),
|
|
tracker.get_last_forwarded_messages(),
|
|
confirmed_frozen_count=frozen,
|
|
)
|
|
total = sum(_toklen(m) for m in forwarded)
|
|
ideal = sum(_toklen(m) for m in fresh)
|
|
if prev_forwarded is not None:
|
|
prev_total = sum(_toklen(m) for m in prev_forwarded)
|
|
assert total - prev_total <= _toklen(forwarded[-1]), (
|
|
f"turn {t}: forwarded grew by {total - prev_total} tokens, more than the "
|
|
f"{_toklen(forwarded[-1])}-token delta message - the prefix compounded"
|
|
)
|
|
warm_ratios.append(total / ideal)
|
|
actual = _provider_cache_read(forwarded, prev_forwarded)
|
|
counts = [_toklen(m) for m in forwarded]
|
|
tracker.update_from_response(
|
|
actual,
|
|
sum(counts) - actual,
|
|
forwarded,
|
|
message_token_counts=counts,
|
|
original_messages=convo,
|
|
)
|
|
prev_forwarded = forwarded
|
|
|
|
# Measured on this worst-case model, forwarded/fresh over turns 2-8:
|
|
# 1.35, 1.84, 2.33, 2.83, 3.35, 3.88, 4.35 - the pinned prefix does widen
|
|
# against an ideal pipeline while the cache stays warm (the cost the floor
|
|
# buys the cache hit with). What the two assertions bound is the shape:
|
|
# each warm turn adds only its own delta, and the cold turn below zeroes
|
|
# the gap outright.
|
|
# Cold turn: the provider count collapses, the floor collapses with it, and
|
|
# the forwarded request is exactly the fresh pipeline's output again.
|
|
convo = convo + [{"role": "user", "content": "tool-output-turn-9:" + "X" * 400}]
|
|
fresh_cold = pipeline(convo, 0, 9)
|
|
cold = overlay_cached_prefix(
|
|
fresh_cold,
|
|
convo,
|
|
tracker.get_last_original_messages(),
|
|
tracker.get_last_forwarded_messages(),
|
|
confirmed_frozen_count=0,
|
|
)
|
|
assert sum(_toklen(m) for m in cold) == sum(_toklen(m) for m in fresh_cold), (
|
|
"cold turn must re-baseline to the fresh pipeline's output"
|
|
)
|
|
assert warm_ratios[-1] > warm_ratios[0], (
|
|
"harness sanity: the warm-turn model must actually be improving the "
|
|
"prefix, otherwise the re-baseline assertion proves nothing"
|
|
)
|