mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(cache): stop Claude Code forks from cold-starting the conversation's lineage (#4043)
### Maintainer qualification refresh (2026-10-09) Current head: `a91a1ec24ab20b6dcea0f6e7f8d41feb7d0beea0`. The newly supplied live Claude Code/Anthropic before-after evidence resolves the previous runtime-proof hold. Clean synchronization with current main `01d9194` preserves contributor logic. Independently reran all 88 fork-lineage/prefix tests with the existing native extension; current-head hosted checks are green. Approved for human merge review. The live capture below remains correctly attributed to the preceding implementation head, not represented as a new capture of this synchronization commit. No merge performed. ## Description Claude Code sends side requests alongside an agent's turns. The subagent progress summary and the parent session's side query both re-send the history plus one prompt, with the breakpoint one message earlier, so they read the prompt cache without writing it. resolve_tracker matched such a fork as a continuation and replaced the lineage snapshot with it. The next real turn extends the history before the fork, not the fork, so it matched no lineage. It started a cold tracker (frozen=0), and the token-mode pipeline rewrote history the provider had already cached. Production, 2026-10-08: a review subagent busted Anthropic's cache on 35 of 81 turns. cache_read_input_tokens fell to the 6,065-token system prompt while 100-150k tokens were re-written. Its 600k-token parent did the same after its own side queries. In the proxy log, every bust is the first main request after a fork and shows no frozen prefix. Fork-free sessions on the same proxy were 96/96 and 130/130 full hits. A lineage now keeps its last three request snapshots. A request that extends an earlier one resumes that turn instead of starting cold. Each response is also recorded against the turn that sent it, through a request-scoped ContextVar (the same pattern as the tenant key and CCR store). A fork's response can no longer replace the main line's frozen prefix or overlay source, whether it lands before or after the in-flight main turn's response. Templated fan-out siblings that extend a shared earlier snapshot now stay on one tracker and each resumes its own turn's state. Splitting them would mint a new lineage per fork and exhaust the per-session cap. The fan-out test asserts the per-line state instead of object identity. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cache/prefix_tracker.py`: each lineage keeps a `deque(maxlen=_LINEAGE_HISTORY)` (3) of request snapshots instead of one. `resolve_tracker` resumes the turn a request extends, and the turn it resolved is held in a request-scoped `ContextVar` (`_resolved_turn`) so the response is recorded against that turn. - `tests/test_anthropic_fork_lineage_cache.py` (new): handler-level regression. A main turn after a fork (progress summary or side query) must replay its cached prefix byte for byte, and a fork must not replace the lineage it branched from, whichever response lands first. - `tests/test_cache/test_prefix_tracker.py`: the fan-out test asserts per-line state instead of tracker object identity. ## Testing - [x] Unit tests pass (`pytest tests/test_anthropic_fork_lineage_cache.py tests/test_cache/ tests/test_proxy_anthropic_cache_stability.py`: 272 passed, with `test_cache/test_dynamic_detector.py` run on its own: 57 passed, 2 skipped) - [x] Linting passes (`ruff check` on the changed files) - [x] Type checking passes for the changed module (`mypy headroom/cache/prefix_tracker.py`: no issues) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # new test file, without the fix (parent commit) FAILED tests/test_anthropic_fork_lineage_cache.py::test_main_turn_after_a_fork_replays_its_cached_prefix[summary] FAILED tests/test_anthropic_fork_lineage_cache.py::test_main_turn_after_a_fork_replays_its_cached_prefix[side-query] FAILED tests/test_anthropic_fork_lineage_cache.py::test_fork_does_not_replace_the_lineage_it_branched_from[fork-first] FAILED tests/test_anthropic_fork_lineage_cache.py::test_fork_does_not_replace_the_lineage_it_branched_from[main-first] AssertionError: main turn 2 re-sent turn 1's history with different bytes ============================== 4 failed ============================== # with the fix ============================== 4 passed ============================== ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13.16, real Claude Code 2.1.295 sessions against the Anthropic API through `headroom proxy --mode token` (details below). - Exact command / steps: the scripted Claude Code session described below, run once on `17abee27` (before) and once on `9b27664a` (after). - Observed result: main turns right after a fork arrived with a cold tracker 5/5 before, 0/5 after; cached tokens lost 3,363 before, 0 after. - Not tested: the full collapse to the system prompt seen in production fork-heavy sessions (did not reproduce in this short session); a deployed proxy running this fix. Live before/after capture on 2026-10-09. Real Claude Code 2.1.295 sessions against the Anthropic API (subscription login, `claude-sonnet-5-5`), through `headroom proxy --mode token` on Windows 11 / Python 3.13.16, with the native extension built in each tree. - **Before:** `17abee27` (merge base with `main`) - **After:** `9b27664a` (this PR) Both ran the same scripted session: a stream-json SDK client with `agentProgressSummaries` and `promptSuggestions` on, one general-purpose subagent that `cat`s ten ~40 KB JSON files one tool call at a time, then three parent follow-up turns. Each request's kind comes from Claude Code's own debug log (`[API REQUEST] source=…`). `sdk` and `agent:*` are main turns; `agent_summary` and `prompt_suggestion` are forks. Token counts are the upstream `usage` values from the proxy's PERF lines. | Main turns right after a fork | Before | After | |---|---|---| | Count | 5 | 5 | | Arrived with a cold tracker (no `Pipeline: freezing` line) | **5/5** | **0/5** | | Read the full prior prefix | 4/5 | **5/5** | | Cached tokens lost | 3,363 | 0 | - **Before**, subagent turn #15 (after its 23-message `agent_summary` fork): cold, `cache_read` 22,137 against the 25,500 cached by the previous turn, `cache_creation` 4,902. - **After**, the matching subagent turn #17 (after a 21-message `agent_summary` fork): frozen 20/23, `cache_read` 25,311 = 24,192 + 1,119, exactly the previous turn's read + write. This short session did not reproduce the full collapse to the system prompt (~6k read) seen in production fork-heavy sessions. In most cold turns the pipeline happened to regenerate identical bytes. The capture does show the mechanism deterministically: before, every fork leaves the next main turn of that conversation starting cold, and one of those cold turns rewrote 3.4k cached tokens. After, none start cold and every post-fork turn reads its whole prior prefix. <details><summary>Per-request tables (before / after)</summary> **Before (`17abee27`)** | # | kind | msgs | frozen | cache_read_input_tokens | cache_creation_input_tokens | |---|---|---|---|---|---| | 1 | main (parent) | 2 | - | 0 | 15,072 | | 2 | main (subagent) | 2 | - | 5,406 | 7,859 | | 3 | main (subagent) | 5 | 3 | 13,265 | 660 | | 4 | main (subagent) | 7 | 3 | 13,925 | 1,176 | | 5 | main (subagent) | 9 | 3 | 15,101 | 1,319 | | 6 | main (subagent) | 11 | 3 | 16,420 | 1,138 | | 7 | fork (agent_summary) | 11 | 3 | 16,476 | 0 | | 8 | main (subagent) <- after fork | 13 | - | 17,558 | 1,127 | | 9 | main (subagent) | 15 | 3 | 18,685 | 1,134 | | 10 | main (subagent) | 17 | 3 | 19,819 | 1,126 | | 11 | main (subagent) | 19 | 3 | 20,945 | 1,136 | | 12 | main (subagent) | 21 | 3 | 22,081 | 1,133 | | 13 | main (subagent) | 23 | 3 | 23,214 | 2,286 | | 14 | fork (agent_summary) | 23 | 22 | 23,321 | 0 | | 15 | main (subagent) <- after fork | 25 | - | 22,137 | 4,902 | | 16 | main (subagent) | 27 | 24 | 27,039 | 1,260 | | 17 | main (subagent) | 29 | 24 | 28,299 | 7,523 | | 18 | main (parent) | 4 | 3 | 15,072 | 2,204 | | 19 | fork (prompt_suggestion) | 6 | 5 | 17,276 | 335 | | 20 | main (parent) <- after fork | 6 | - | 17,611 | 31 | | 21 | fork (prompt_suggestion) | 8 | 7 | 17,642 | 362 | | 22 | main (parent) <- after fork | 8 | - | 18,004 | 29 | | 23 | main (parent) | 10 | 9 | 18,033 | 242 | | 24 | fork (prompt_suggestion) | 12 | 9 | 18,275 | 128 | | 25 | main (parent) <- after fork | 12 | - | 18,403 | 18 | | 26 | fork (prompt_suggestion) | 14 | 9 | 18,421 | 142 | **After (`9b27664a`)** | # | kind | msgs | frozen | cache_read_input_tokens | cache_creation_input_tokens | |---|---|---|---|---|---| | 1 | main (parent) | 2 | - | 5,406 | 9,666 | | 2 | main (subagent) | 2 | - | 10,523 | 2,742 | | 3 | main (parent) | 4 | 3 | 15,072 | 692 | | 4 | main (subagent) | 5 | 3 | 13,265 | 1,681 | | 5 | fork (prompt_suggestion) | 6 | 3 | 15,764 | 25 | | 6 | main (subagent) | 7 | 3 | 14,946 | 1,246 | | 7 | main (subagent) | 9 | 3 | 16,192 | 1,138 | | 8 | main (subagent) | 11 | 3 | 17,330 | 2,303 | | 9 | main (subagent) | 13 | 3 | 19,633 | 1,167 | | 10 | main (subagent) | 15 | 3 | 20,800 | 1,137 | | 11 | main (subagent) | 17 | 3 | 21,937 | 1,129 | | 12 | main (subagent) | 19 | 3 | 23,066 | 1,126 | | 13 | main (subagent) | 21 | 20 | 24,192 | 1,119 | | 14 | main (parent) <- after fork | 6 | 3 | 15,789 | 31 | | 15 | fork (prompt_suggestion) | 8 | 3 | 15,820 | 108 | | 16 | fork (agent_summary) | 21 | 20 | 24,248 | 0 | | 17 | main (subagent) <- after fork | 23 | 20 | 25,311 | 710 | | 18 | main (parent) <- after fork | 8 | 3 | 15,928 | 851 | | 19 | fork (prompt_suggestion) | 10 | 3 | 16,779 | 688 | | 20 | main (parent) <- after fork | 10 | 3 | 17,467 | 29 | | 21 | main (parent) | 12 | 3 | 17,496 | 130 | | 22 | main (parent) | 14 | 3 | 17,626 | 249 | | 23 | fork (prompt_suggestion) | 16 | 3 | 17,875 | 131 | | 24 | main (parent) <- after fork | 16 | 3 | 18,006 | 18 | | 25 | fork (prompt_suggestion) | 18 | 3 | 18,024 | 136 | </details> ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: yes, prefix-tracker lineage matching for requests that extend an earlier snapshot of the same lineage - Kill switch / disable path: none - Unsafe override required: no - Qualification impact: none expected - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Not covered here: a few small partial misses in production that forks don't explain, and a synthetic-only risk that an older tool result gets recompressed once it falls past a lagging frozen boundary. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4043" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JerrettDavis
parent
a9eb2a7b6f
commit
1cf4d053d2
Vendored
+142
-22
@@ -23,7 +23,8 @@ import json
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from collections import OrderedDict
|
||||
from collections import OrderedDict, deque
|
||||
from contextvars import ContextVar
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
@@ -63,6 +64,22 @@ _PROVIDER_CACHE_TTL_SECONDS = {
|
||||
# cold-prefix detection, but never below the configured session cleanup floor.
|
||||
_CACHE_TTL_CLEANUP_MARGIN_SECONDS = 60
|
||||
|
||||
# Recent request snapshots kept per lineage. A Claude Code fork (the subagent
|
||||
# progress summary, the parent's side query) re-sends the history plus one
|
||||
# prompt, so the next real turn extends the snapshot BEFORE the fork. A
|
||||
# request that resumes an older snapshot keeps it next-newest, so any number
|
||||
# of forks off one turn leaves that turn resumable.
|
||||
_LINEAGE_HISTORY = 3
|
||||
|
||||
# (tracker, turn, resumed state) the current request resolved to. Each proxied
|
||||
# request runs in its own asyncio task, so it reads the state it resumed and
|
||||
# ``update_from_response`` records its response against its own turn, even
|
||||
# while a fork of the same conversation is in flight on the same tracker. Same
|
||||
# pattern as the request-scoped tenant key and CCR store.
|
||||
_resolved_turn: ContextVar[tuple[Any, int, tuple[Any, ...] | None] | None] = ContextVar(
|
||||
"headroom_resolved_lineage_turn", default=None
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class PrefixFreezeConfig:
|
||||
@@ -1153,6 +1170,11 @@ class PrefixCacheTracker:
|
||||
# net-cost/TTL P_alive gate could never see idle time. The handler reads
|
||||
# this and forwards it to the pipeline as `idle_seconds`.
|
||||
self._idle_seconds_at_fetch: float = 0.0
|
||||
# Turn state after each recent request's response (None while it is in
|
||||
# flight), keyed by the turn SessionTrackerStore issued it. A request
|
||||
# resumes the state of the turn it extends, so a fork's response never
|
||||
# becomes the main line's frozen prefix or replay source.
|
||||
self._turn_states: dict[int, tuple[Any, ...] | None] = {}
|
||||
|
||||
# Session-scoped ReadMaturationManager (Mechanism B), created
|
||||
# lazily by the handler when read maturation is enabled. Rides
|
||||
@@ -1169,13 +1191,14 @@ class PrefixCacheTracker:
|
||||
|
||||
Returns 0 on turn 0 (cold start) or if caching is disabled/below threshold.
|
||||
"""
|
||||
turn_number, cached_tokens, cached_messages, _, _ = self._request_turn_state()
|
||||
if not self.config.enabled:
|
||||
return 0
|
||||
if self._turn_number == 0:
|
||||
if turn_number == 0:
|
||||
return 0
|
||||
if self._cached_token_count < self.config.min_cached_tokens:
|
||||
if cached_tokens < self.config.min_cached_tokens:
|
||||
return 0
|
||||
return self._cached_message_count
|
||||
return int(cached_messages)
|
||||
|
||||
def update_from_response(
|
||||
self,
|
||||
@@ -1208,6 +1231,7 @@ class PrefixCacheTracker:
|
||||
if total_cached == 0:
|
||||
self._cached_token_count = 0
|
||||
self._cached_message_count = 0
|
||||
self._record_turn_state()
|
||||
return
|
||||
|
||||
# Estimate per-message token counts if not provided
|
||||
@@ -1227,6 +1251,7 @@ class PrefixCacheTracker:
|
||||
|
||||
self._cached_token_count = total_cached
|
||||
self._cached_message_count = frozen_count
|
||||
self._record_turn_state()
|
||||
|
||||
logger.debug(
|
||||
"PrefixCacheTracker[%s]: turn=%d, cached=%d tokens, "
|
||||
@@ -1241,10 +1266,56 @@ class PrefixCacheTracker:
|
||||
)
|
||||
|
||||
def get_last_original_messages(self) -> list[dict[str, Any]]:
|
||||
return copy.deepcopy(self._last_original_messages)
|
||||
return copy.deepcopy(self._request_turn_state()[3])
|
||||
|
||||
def get_last_forwarded_messages(self) -> list[dict[str, Any]]:
|
||||
return copy.deepcopy(self._last_forwarded_messages)
|
||||
return copy.deepcopy(self._request_turn_state()[4])
|
||||
|
||||
def _turn_state(self) -> tuple[Any, ...]:
|
||||
return (
|
||||
self._turn_number,
|
||||
self._cached_token_count,
|
||||
self._cached_message_count,
|
||||
self._last_original_messages,
|
||||
self._last_forwarded_messages,
|
||||
)
|
||||
|
||||
def _request_turn_state(self) -> tuple[Any, ...]:
|
||||
"""The state this request resumed at resolve time, else the current one.
|
||||
|
||||
A handler may read after an ``await`` (OpenAI's overlay runs after
|
||||
compression), by which time a sibling request on this lineage may have
|
||||
resumed a different turn.
|
||||
"""
|
||||
resolved = _resolved_turn.get()
|
||||
if resolved is not None and resolved[0] is self and resolved[2] is not None:
|
||||
return resolved[2]
|
||||
return self._turn_state()
|
||||
|
||||
def _record_turn_state(self) -> None:
|
||||
"""Remember this response's state under the turn that sent it."""
|
||||
resolved = _resolved_turn.get()
|
||||
if resolved is not None and resolved[0] is self:
|
||||
if resolved[1] in self._turn_states:
|
||||
self._turn_states[resolved[1]] = self._turn_state()
|
||||
# The request has its response; later reads see the current state.
|
||||
_resolved_turn.set((self, resolved[1], None))
|
||||
|
||||
def _resume_turn(self, turn: int | None) -> None:
|
||||
"""Continue from ``turn``'s response, not whichever response came last.
|
||||
|
||||
A turn still in flight (or one recorded before turn tracking) has no
|
||||
state yet; the current state stands, as it did before.
|
||||
"""
|
||||
state = self._turn_states.get(turn) if turn is not None else None
|
||||
if state is not None:
|
||||
(
|
||||
self._turn_number,
|
||||
self._cached_token_count,
|
||||
self._cached_message_count,
|
||||
self._last_original_messages,
|
||||
self._last_forwarded_messages,
|
||||
) = state
|
||||
|
||||
def record_returned(
|
||||
self,
|
||||
@@ -1311,7 +1382,8 @@ class PrefixCacheTracker:
|
||||
if idle_seconds is None:
|
||||
idle_seconds = self.seconds_since_activity()
|
||||
ttl = self.resolved_cache_ttl_seconds()
|
||||
expected = self._cached_token_count
|
||||
# The turn this request resumed, not whichever sibling answered last.
|
||||
expected = self._request_turn_state()[1]
|
||||
|
||||
# Nothing was cached last turn → cold start, not a miss.
|
||||
if expected <= 0:
|
||||
@@ -1366,7 +1438,7 @@ class PrefixCacheTracker:
|
||||
content. Anything else (a frozen message rewritten, the prefix
|
||||
reordered, the list now shorter) counts as a prefix change.
|
||||
"""
|
||||
prev = self._last_forwarded_messages
|
||||
prev: list[dict[str, Any]] = self._request_turn_state()[4]
|
||||
if not prev:
|
||||
# No recorded prefix to compare — can't claim it changed.
|
||||
return True
|
||||
@@ -1565,6 +1637,10 @@ class SessionTrackerStore:
|
||||
# but different tool profiles must never share frozen-prefix state.
|
||||
self._lineage_affinities: dict[str, str | None] = {}
|
||||
self._lineage_counter = itertools.count(1)
|
||||
# Recent (snapshot, turn) per tracker key, newest first; the newest
|
||||
# snapshot is the one in ``_lineages``.
|
||||
self._lineage_history: dict[str, deque[tuple[list[Any], int]]] = {}
|
||||
self._turn_counter = itertools.count(1)
|
||||
|
||||
def peek(self, session_id: str) -> PrefixCacheTracker | None:
|
||||
"""Return the live tracker for ``session_id``, else None.
|
||||
@@ -1614,6 +1690,7 @@ class SessionTrackerStore:
|
||||
def _discard_tracker(self, tracker_key: str) -> None:
|
||||
"""Remove a tracker and any lineage indexes that point to it."""
|
||||
self._trackers.pop(tracker_key, None)
|
||||
self._lineage_history.pop(tracker_key, None)
|
||||
for session_id, family in list(self._lineages.items()):
|
||||
family.pop(tracker_key, None)
|
||||
self._lineage_affinities.pop(tracker_key, None)
|
||||
@@ -1707,22 +1784,48 @@ class SessionTrackerStore:
|
||||
for key in expired_keys:
|
||||
family.pop(key, None)
|
||||
self._lineage_affinities.pop(key, None)
|
||||
self._lineage_history.pop(key, None)
|
||||
self._trackers.pop(key, None)
|
||||
by_length = sorted(family.items(), key=lambda item: len(item[1]), reverse=True)
|
||||
best_key: str | None = None
|
||||
for accepted in (
|
||||
(RELATION_EXACT, RELATION_MESSAGE_APPEND),
|
||||
(RELATION_BLOCK_APPEND,),
|
||||
):
|
||||
for key, chain in by_length:
|
||||
if self._lineage_affinities.get(key) != cache_affinity:
|
||||
continue
|
||||
relation = _classify_history_canonical(snap, chain)
|
||||
if relation.kind in accepted:
|
||||
best_key = key
|
||||
break
|
||||
if best_key is not None:
|
||||
break
|
||||
|
||||
def continuation(
|
||||
candidates: list[tuple[str, list[Any], int | None]],
|
||||
) -> tuple[str | None, int | None]:
|
||||
for accepted in (
|
||||
(RELATION_EXACT, RELATION_MESSAGE_APPEND),
|
||||
(RELATION_BLOCK_APPEND,),
|
||||
):
|
||||
for key, chain, turn in candidates:
|
||||
if self._lineage_affinities.get(key) != cache_affinity:
|
||||
continue
|
||||
if _classify_history_canonical(snap, chain).kind in accepted:
|
||||
return key, turn
|
||||
return None, None
|
||||
|
||||
def latest_turn(key: str) -> int | None:
|
||||
history = self._lineage_history.get(key)
|
||||
return history[0][1] if history else None
|
||||
|
||||
best_key, best_turn = continuation(
|
||||
[(key, chain, latest_turn(key)) for key, chain in by_length]
|
||||
)
|
||||
if best_key is None:
|
||||
# A fork (Claude Code's subagent summary or side query: the
|
||||
# history plus one prompt) became a lineage's latest snapshot, so
|
||||
# this turn extends the snapshot before it. Resume that turn
|
||||
# instead of starting a cold lineage that recompresses history the
|
||||
# provider already cached.
|
||||
best_key, best_turn = continuation(
|
||||
sorted(
|
||||
(
|
||||
(key, chain, turn)
|
||||
for key in family
|
||||
for chain, turn in list(self._lineage_history.get(key, ()))[1:]
|
||||
),
|
||||
key=lambda item: len(item[1]),
|
||||
reverse=True,
|
||||
)
|
||||
)
|
||||
|
||||
if best_key is None:
|
||||
rewrite_candidates: list[tuple[tuple[int, int, int], str]] = []
|
||||
@@ -1808,6 +1911,22 @@ class SessionTrackerStore:
|
||||
family = self._lineages.setdefault(session_id, family)
|
||||
family[best_key] = snap
|
||||
self._lineage_affinities[best_key] = cache_affinity
|
||||
if isinstance(tracker, PrefixCacheTracker): # test stubs keep no turns
|
||||
tracker._resume_turn(best_turn if best_turn is not None else latest_turn(best_key))
|
||||
turn = next(self._turn_counter)
|
||||
history = self._lineage_history.setdefault(best_key, deque(maxlen=_LINEAGE_HISTORY))
|
||||
branch = next((entry for entry in history if entry[1] == best_turn), None)
|
||||
if branch is not None and branch is not history[0]:
|
||||
# A fork off an older turn: that turn stays the branch point the
|
||||
# next main turn extends, so keep it behind this request rather
|
||||
# than let successive forks push it out.
|
||||
history.remove(branch)
|
||||
history.appendleft(branch)
|
||||
history.appendleft((snap, turn))
|
||||
live = {t for _, t in history}
|
||||
tracker._turn_states = {t: s for t, s in tracker._turn_states.items() if t in live}
|
||||
tracker._turn_states[turn] = None
|
||||
_resolved_turn.set((tracker, turn, tracker._turn_state()))
|
||||
return tracker
|
||||
|
||||
def compute_session_id(
|
||||
@@ -1877,6 +1996,7 @@ class SessionTrackerStore:
|
||||
for key in [k for k in family if k not in self._trackers]:
|
||||
del family[key]
|
||||
self._lineage_affinities.pop(key, None)
|
||||
self._lineage_history.pop(key, None)
|
||||
if not family:
|
||||
del self._lineages[base]
|
||||
logger.debug("SessionTrackerStore: cleaned up %d expired sessions", len(expired))
|
||||
|
||||
@@ -0,0 +1,343 @@
|
||||
"""A Claude Code side request must not cost the main conversation its prompt cache.
|
||||
|
||||
Production, 2026-10-08: a W.A.L.T.E.R. review subagent busted Anthropic's
|
||||
prompt cache on 35 of its 81 turns (``cache_read_input_tokens`` fell to the
|
||||
6,065-token system prompt while ~100-150k tokens were re-written), and its
|
||||
600k-token parent session did the same after its own side requests. Every
|
||||
bust followed a *fork*: Claude Code re-sends the agent's history plus one
|
||||
extra prompt (the subagent progress summary merges it into the last user
|
||||
message; the parent's side query appends a reply and a new user message) with
|
||||
its breakpoint one message earlier, so it reads the cache without writing.
|
||||
|
||||
The fork extended the conversation's prefix-tracker lineage and replaced its
|
||||
snapshot. The next main turn extends the history the fork branched from, not
|
||||
the fork, so it matched no lineage, started a cold tracker (frozen=0) and the
|
||||
token-mode pipeline recompressed tool results it had forwarded verbatim the
|
||||
turn before. Different bytes, so the provider's prefix cache missed from the
|
||||
first changed message onward.
|
||||
|
||||
The upstream below models Anthropic's prompt cache (a prefix is readable once
|
||||
a breakpoint wrote it), so the assertions are on what the provider would do
|
||||
with the bytes the handler forwards.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextvars
|
||||
import copy
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
pytest.importorskip("fastapi")
|
||||
|
||||
import httpx
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from headroom.cache.prefix_tracker import PrefixCacheTracker, SessionTrackerStore
|
||||
from headroom.proxy.server import ProxyConfig, create_app
|
||||
|
||||
MODEL = "claude-sonnet-4-5-20250929"
|
||||
SYSTEM = [
|
||||
{
|
||||
"type": "text",
|
||||
"text": "You review pull requests. Report only real defects. " * 40,
|
||||
"cache_control": {"type": "ephemeral"},
|
||||
}
|
||||
]
|
||||
BASH_TOOL = {
|
||||
"name": "Bash",
|
||||
"description": "Run a shell command",
|
||||
"input_schema": {
|
||||
"type": "object",
|
||||
"properties": {"command": {"type": "string"}},
|
||||
"required": ["command"],
|
||||
},
|
||||
}
|
||||
MARKER = {"type": "ephemeral"}
|
||||
|
||||
|
||||
def _bash_turn(i: int) -> list[dict[str, Any]]:
|
||||
tid = f"toolu_{i:02d}"
|
||||
rows = [
|
||||
{"path": f"src/module_{i}/file_{j}.py", "size": 1000 + j, "status": "ok"}
|
||||
for j in range(400)
|
||||
]
|
||||
return [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": [
|
||||
{"type": "tool_use", "id": tid, "name": "Bash", "input": {"command": f"ls {i}"}}
|
||||
],
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [{"type": "tool_result", "tool_use_id": tid, "content": json.dumps(rows)}],
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def _main(turns: int) -> list[dict[str, Any]]:
|
||||
"""The agent's own request: Claude Code's breakpoint rides the newest block."""
|
||||
messages: list[dict[str, Any]] = [
|
||||
{"role": "user", "content": [{"type": "text", "text": "Review the change."}]}
|
||||
]
|
||||
for i in range(turns):
|
||||
messages += _bash_turn(i)
|
||||
messages[-1]["content"][-1]["cache_control"] = dict(MARKER)
|
||||
return messages
|
||||
|
||||
|
||||
def _summary_fork(turns: int) -> list[dict[str, Any]]:
|
||||
"""Subagent progress summary: same history, the prompt merged into the last
|
||||
user message, breakpoint one message earlier (reads, never writes)."""
|
||||
messages = _main(turns)
|
||||
del messages[-1]["content"][-1]["cache_control"]
|
||||
messages[-1]["content"].append({"type": "text", "text": "Summarize your progress in 5 words."})
|
||||
messages[-2]["content"][-1]["cache_control"] = dict(MARKER)
|
||||
return messages
|
||||
|
||||
|
||||
def _side_query_fork(turns: int) -> list[dict[str, Any]]:
|
||||
"""Parent-session side query: history + the agent's reply + a new prompt."""
|
||||
messages = _main(turns)
|
||||
messages.append({"role": "assistant", "content": [{"type": "text", "text": "Done."}]})
|
||||
messages.append({"role": "user", "content": [{"type": "text", "text": "Name this session."}]})
|
||||
del messages[-3]["content"][-1]["cache_control"]
|
||||
messages[-2]["content"][-1]["cache_control"] = dict(MARKER)
|
||||
return messages
|
||||
|
||||
|
||||
def _strip(obj: Any) -> Any:
|
||||
if isinstance(obj, dict):
|
||||
return {k: _strip(v) for k, v in obj.items() if k != "cache_control"}
|
||||
if isinstance(obj, list):
|
||||
return [_strip(v) for v in obj]
|
||||
return obj
|
||||
|
||||
|
||||
def _has_marker(message: dict[str, Any]) -> bool:
|
||||
content = message.get("content")
|
||||
return isinstance(content, list) and any(
|
||||
isinstance(block, dict) and "cache_control" in block for block in content
|
||||
)
|
||||
|
||||
|
||||
class _PromptCache:
|
||||
"""Anthropic's prompt cache at message granularity.
|
||||
|
||||
A request reads the longest prefix an earlier breakpoint wrote, then
|
||||
writes a cache entry at each of its own breakpoints. Tools and system are
|
||||
part of every key because Anthropic renders them before the messages.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._written: set[str] = set()
|
||||
|
||||
def request(self, body: dict[str, Any]) -> tuple[int, int, int]:
|
||||
messages = body.get("messages") or []
|
||||
head = json.dumps(_strip([body.get("tools"), body.get("system")]), sort_keys=True)
|
||||
keys = [
|
||||
head + json.dumps(_strip(messages[: i + 1]), sort_keys=True)
|
||||
for i in range(len(messages))
|
||||
]
|
||||
marks = [i for i, m in enumerate(messages) if _has_marker(m)]
|
||||
if not marks:
|
||||
return 0, 0, -1
|
||||
hit = max((i for i in range(marks[-1] + 1) if keys[i] in self._written), default=-1)
|
||||
self._written.update(keys[i] for i in marks)
|
||||
# The tracker's own estimate: usage it can map back onto exactly the
|
||||
# messages the provider cached, so only the fork can move the floor.
|
||||
tokens = PrefixCacheTracker._estimate_message_tokens(messages)
|
||||
system_tokens = len(json.dumps(body.get("system"))) // 4
|
||||
read = system_tokens + sum(tokens[: hit + 1])
|
||||
write = sum(tokens[hit + 1 : marks[-1] + 1])
|
||||
return read, write, hit
|
||||
|
||||
|
||||
def _drive(requests: list[list[dict[str, Any]]]) -> list[tuple[list[dict[str, Any]], int]]:
|
||||
"""Send each history through the real handler; return (forwarded, read_hit)."""
|
||||
app = create_app(
|
||||
ProxyConfig(
|
||||
optimize=True,
|
||||
mode="token",
|
||||
# Deterministic structural compression only: the Kompress model
|
||||
# is not available everywhere the suite runs.
|
||||
disable_kompress=True,
|
||||
rate_limit_enabled=False,
|
||||
cost_tracking_enabled=False,
|
||||
log_requests=False,
|
||||
)
|
||||
)
|
||||
cache = _PromptCache()
|
||||
seen: list[tuple[list[dict[str, Any]], int]] = []
|
||||
with TestClient(app) as client:
|
||||
proxy = client.app.state.proxy
|
||||
|
||||
async def _upstream(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001, ANN003
|
||||
read, write, hit = cache.request(body)
|
||||
seen.append((copy.deepcopy(body.get("messages") or []), hit))
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"id": "msg_x",
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"model": MODEL,
|
||||
"content": [{"type": "text", "text": "ok"}],
|
||||
"stop_reason": "end_turn",
|
||||
"usage": {
|
||||
"input_tokens": 10,
|
||||
"output_tokens": 2,
|
||||
"cache_read_input_tokens": read,
|
||||
"cache_creation_input_tokens": write,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
proxy._retry_request = _upstream
|
||||
for messages in requests:
|
||||
response = client.post(
|
||||
"/v1/messages",
|
||||
headers={
|
||||
"x-api-key": "test-key",
|
||||
"anthropic-version": "2023-06-01",
|
||||
"content-type": "application/json",
|
||||
},
|
||||
json={
|
||||
"model": MODEL,
|
||||
"max_tokens": 64,
|
||||
"system": SYSTEM,
|
||||
"tools": [BASH_TOOL],
|
||||
"messages": messages,
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200, response.text[:300]
|
||||
assert len(seen) == len(requests)
|
||||
return seen
|
||||
|
||||
|
||||
@pytest.mark.parametrize("fork", [_summary_fork, _side_query_fork], ids=["summary", "side-query"])
|
||||
def test_main_turn_after_a_fork_replays_its_cached_prefix(monkeypatch, fork) -> None: # noqa: ANN001
|
||||
from headroom.cache.compression_store import reset_compression_store
|
||||
|
||||
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "memory")
|
||||
reset_compression_store()
|
||||
|
||||
turns = range(1, 6)
|
||||
requests: list[list[dict[str, Any]]] = []
|
||||
for n in turns:
|
||||
requests += [_main(n), fork(n)]
|
||||
seen = _drive(requests)
|
||||
mains = seen[0::2]
|
||||
|
||||
# Not vacuous: the pipeline did rewrite tool results somewhere.
|
||||
assert any(
|
||||
_strip(forwarded) != _strip(original) for (forwarded, _), original in zip(seen, requests)
|
||||
), "nothing was compressed; the test cannot observe a prefix change"
|
||||
|
||||
for (previous, _), (current, hit), n in zip(mains, mains[1:], turns):
|
||||
assert _strip(current[: len(previous)]) == _strip(previous), (
|
||||
f"main turn {n + 1} re-sent turn {n}'s history with different bytes"
|
||||
)
|
||||
assert hit >= len(previous) - 1, (
|
||||
f"main turn {n + 1} read only {hit + 1}/{len(previous)} cached messages"
|
||||
)
|
||||
|
||||
|
||||
def _record(tracker, messages, *, cached: int) -> None: # noqa: ANN001
|
||||
tracker.update_from_response(
|
||||
cache_read_tokens=cached,
|
||||
cache_write_tokens=0,
|
||||
messages=messages,
|
||||
original_messages=messages,
|
||||
message_token_counts=[1000] * len(messages),
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("fork_responds_first", [True, False], ids=["fork-first", "main-first"])
|
||||
def test_fork_does_not_replace_the_lineage_it_branched_from(fork_responds_first) -> None: # noqa: ANN001
|
||||
"""Store level, both orderings seen in production: the summary fork is sent
|
||||
while a slow main turn is still in flight (fork answers first), or after a
|
||||
fast one answered (main answers first). Either way the next main turn must
|
||||
resume the main turn's state, not the fork's and not a cold tracker.
|
||||
|
||||
Each request runs in its own context, as each proxied request runs in its
|
||||
own asyncio task."""
|
||||
store = SessionTrackerStore()
|
||||
main_k, fork_k, main_next = _strip(_main(3)), _strip(_summary_fork(3)), _strip(_main(4))
|
||||
main_ctx, fork_ctx = contextvars.Context(), contextvars.Context()
|
||||
main_tracker = main_ctx.run(store.resolve_tracker, "sid", "anthropic", messages=main_k)
|
||||
if not fork_responds_first:
|
||||
main_ctx.run(_record, main_tracker, main_k, cached=1000 * len(main_k))
|
||||
fork_tracker = fork_ctx.run(store.resolve_tracker, "sid", "anthropic", messages=fork_k)
|
||||
fork_ctx.run(_record, fork_tracker, fork_k, cached=1000 * (len(fork_k) - 1))
|
||||
if fork_responds_first:
|
||||
main_ctx.run(_record, main_tracker, main_k, cached=1000 * len(main_k))
|
||||
|
||||
resumed = contextvars.Context().run(
|
||||
store.resolve_tracker, "sid", "anthropic", messages=main_next
|
||||
)
|
||||
|
||||
assert resumed is main_tracker
|
||||
assert resumed.get_frozen_message_count() == len(main_k)
|
||||
assert resumed.get_last_original_messages() == main_k
|
||||
|
||||
|
||||
@pytest.mark.parametrize("forks", [1, 3, 5])
|
||||
def test_successive_forks_keep_the_turn_they_branch_from(forks) -> None: # noqa: ANN001
|
||||
"""Several forks off one main turn (progress summaries, side queries) must
|
||||
not push that turn out of the lineage history: the next main turn still
|
||||
extends it, so it must resume its frozen prefix rather than start cold."""
|
||||
store = SessionTrackerStore()
|
||||
base = _strip(_main(3))
|
||||
main_tracker = contextvars.Context().run(_send, store, base)
|
||||
for i in range(forks):
|
||||
side = base + [
|
||||
{"role": "assistant", "content": [{"type": "text", "text": f"Done {i}."}]},
|
||||
{"role": "user", "content": [{"type": "text", "text": f"Side query {i}."}]},
|
||||
]
|
||||
contextvars.Context().run(_send, store, side)
|
||||
|
||||
main_next = _strip(_main(4))
|
||||
resumed = contextvars.Context().run(
|
||||
store.resolve_tracker, "sid", "anthropic", messages=main_next
|
||||
)
|
||||
|
||||
assert resumed is main_tracker
|
||||
assert resumed.get_frozen_message_count() == len(base)
|
||||
assert resumed.get_last_original_messages() == base
|
||||
|
||||
|
||||
def _send(store: SessionTrackerStore, messages: list[dict[str, Any]]) -> PrefixCacheTracker:
|
||||
tracker = store.resolve_tracker("sid", "anthropic", messages=messages)
|
||||
_record(tracker, messages, cached=1000 * len(messages))
|
||||
return tracker
|
||||
|
||||
|
||||
@pytest.mark.parametrize("fork_responds_first", [True, False], ids=["fork-first", "main-first"])
|
||||
def test_cache_miss_is_classified_against_the_resumed_turn(fork_responds_first) -> None: # noqa: ANN001
|
||||
"""A sibling request answering while this one is in flight must not change
|
||||
what this request's cache outcome is measured against."""
|
||||
store = SessionTrackerStore()
|
||||
base = _strip(_main(3))
|
||||
contextvars.Context().run(_send, store, base)
|
||||
|
||||
main_next, fork = _strip(_main(4)), _strip(_summary_fork(3))
|
||||
main_ctx, fork_ctx = contextvars.Context(), contextvars.Context()
|
||||
main_tracker = main_ctx.run(store.resolve_tracker, "sid", "anthropic", messages=main_next)
|
||||
fork_tracker = fork_ctx.run(store.resolve_tracker, "sid", "anthropic", messages=fork)
|
||||
assert fork_tracker is main_tracker
|
||||
if fork_responds_first:
|
||||
# The fork read nothing: the shared tracker now expects no cache.
|
||||
fork_ctx.run(_record, fork_tracker, fork, cached=0)
|
||||
|
||||
# The main turn misses on a prefix its parent cached.
|
||||
attribution = main_ctx.run(main_tracker.classify_cache_miss, 0, main_next, 0.0)
|
||||
|
||||
assert attribution.is_miss
|
||||
assert attribution.expected_cached_tokens == 1000 * len(base)
|
||||
assert not attribution.prefix_changed
|
||||
if not fork_responds_first:
|
||||
fork_ctx.run(_record, fork_tracker, fork, cached=0)
|
||||
@@ -1,5 +1,6 @@
|
||||
"""Tests for PrefixCacheTracker — cache-aware compression."""
|
||||
|
||||
import contextvars
|
||||
import time
|
||||
|
||||
import pytest
|
||||
@@ -744,33 +745,50 @@ class TestConversationLineageResolution:
|
||||
assert trackers["A"]._turn_number == 4
|
||||
assert trackers["B"]._turn_number == 4
|
||||
|
||||
def test_identical_first_turns_share_until_divergence_then_split(self, store):
|
||||
"""Templated fan-outs send byte-identical first turns. While histories
|
||||
are identical, sharing a tracker is harmless (the provider cache line
|
||||
is identical too); they must split as soon as the histories diverge."""
|
||||
def test_identical_first_turns_keep_per_line_state_after_divergence(self, store):
|
||||
"""Templated fan-outs send byte-identical first turns, then diverge.
|
||||
Each line extends its own earlier snapshot and must resume that turn's
|
||||
frozen prefix and replay source, never its sibling's. (A sibling that
|
||||
extends an earlier snapshot is indistinguishable from a main turn after
|
||||
a Claude Code fork, so they share the lineage rather than split.)
|
||||
|
||||
Each request runs in its own context, as each proxied request runs in
|
||||
its own asyncio task."""
|
||||
sid = "shared"
|
||||
first = [{"role": "user", "content": "verify the fix " + "p" * 300}]
|
||||
t_a1 = store.resolve_tracker(sid, "anthropic", messages=first)
|
||||
t_b1 = store.resolve_tracker(sid, "anthropic", messages=first)
|
||||
assert t_b1 is t_a1
|
||||
|
||||
a2 = first + [
|
||||
{"role": "assistant", "content": "answer A"},
|
||||
{"role": "user", "content": "next A"},
|
||||
]
|
||||
b2 = first + [
|
||||
{"role": "assistant", "content": "answer B"},
|
||||
{"role": "user", "content": "next B"},
|
||||
]
|
||||
t_a2 = store.resolve_tracker(sid, "anthropic", messages=a2)
|
||||
t_b2 = store.resolve_tracker(sid, "anthropic", messages=b2)
|
||||
assert t_a2 is not t_b2
|
||||
def turn(history: list[dict], frozen: int) -> PrefixCacheTracker:
|
||||
def send() -> PrefixCacheTracker:
|
||||
tracker = store.resolve_tracker(sid, "anthropic", messages=history)
|
||||
tracker.update_from_response(
|
||||
cache_read_tokens=2000 * frozen,
|
||||
cache_write_tokens=0,
|
||||
messages=history,
|
||||
message_token_counts=[2000] * len(history),
|
||||
)
|
||||
return tracker
|
||||
|
||||
a3 = a2 + [
|
||||
{"role": "assistant", "content": "answer A2"},
|
||||
{"role": "user", "content": "next A2"},
|
||||
]
|
||||
assert store.resolve_tracker(sid, "anthropic", messages=a3) is t_a2
|
||||
return contextvars.Context().run(send)
|
||||
|
||||
def reply(history: list[dict], text: str) -> list[dict]:
|
||||
return history + [
|
||||
{"role": "assistant", "content": f"answer {text}"},
|
||||
{"role": "user", "content": f"next {text}"},
|
||||
]
|
||||
|
||||
a2, b2 = reply(first, "A"), reply(first, "B")
|
||||
turn(a2, frozen=3)
|
||||
turn(b2, frozen=2)
|
||||
|
||||
a3 = store.resolve_tracker(sid, "anthropic", messages=reply(a2, "A2"))
|
||||
assert a3.get_last_original_messages() == a2
|
||||
assert a3.get_frozen_message_count() == 3
|
||||
b3 = store.resolve_tracker(sid, "anthropic", messages=reply(b2, "B2"))
|
||||
assert b3.get_last_original_messages() == b2
|
||||
assert b3.get_frozen_message_count() == 2
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"cc_turn2",
|
||||
|
||||
Reference in New Issue
Block a user