fix(cache): stop Claude Code forks from cold-starting the conversation's lineage (#4043)

### Maintainer qualification refresh (2026-10-09)

Current head: `a91a1ec24ab20b6dcea0f6e7f8d41feb7d0beea0`. The newly
supplied live Claude Code/Anthropic before-after evidence resolves the
previous runtime-proof hold. Clean synchronization with current main
`01d9194` preserves contributor logic. Independently reran all 88
fork-lineage/prefix tests with the existing native extension;
current-head hosted checks are green. Approved for human merge review.
The live capture below remains correctly attributed to the preceding
implementation head, not represented as a new capture of this
synchronization commit. No merge performed.

## Description

Claude Code sends side requests alongside an agent's turns. The subagent
progress summary and the parent session's side query both re-send the
history plus one prompt, with the breakpoint one message earlier, so
they read the prompt cache without writing it. resolve_tracker matched
such a fork as a continuation and replaced the lineage snapshot with it.
The next real turn extends the history before the fork, not the fork,
so it matched no lineage. It started a cold tracker (frozen=0), and the
token-mode pipeline rewrote history the provider had already cached.

Production, 2026-10-08: a review subagent busted Anthropic's cache on
35 of 81 turns. cache_read_input_tokens fell to the 6,065-token system
prompt while 100-150k tokens were re-written. Its 600k-token parent did
the same after its own side queries. In the proxy log, every bust is
the first main request after a fork and shows no frozen prefix.
Fork-free sessions on the same proxy were 96/96 and 130/130 full hits.

A lineage now keeps its last three request snapshots. A request that
extends an earlier one resumes that turn instead of starting cold. Each
response is also recorded against the turn that sent it, through a
request-scoped ContextVar (the same pattern as the tenant key and CCR
store). A fork's response can no longer replace the main line's frozen
prefix or overlay source, whether it lands before or after the
in-flight main turn's response.

Templated fan-out siblings that extend a shared earlier snapshot now
stay on one tracker and each resumes its own turn's state. Splitting
them would mint a new lineage per fork and exhaust the per-session cap.
The fan-out test asserts the per-line state instead of object identity.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cache/prefix_tracker.py`: each lineage keeps a
`deque(maxlen=_LINEAGE_HISTORY)` (3) of request snapshots instead of
one. `resolve_tracker` resumes the turn a request extends, and the turn
it resolved is held in a request-scoped `ContextVar` (`_resolved_turn`)
so the response is recorded against that turn.
- `tests/test_anthropic_fork_lineage_cache.py` (new): handler-level
regression. A main turn after a fork (progress summary or side query)
must replay its cached prefix byte for byte, and a fork must not replace
the lineage it branched from, whichever response lands first.
- `tests/test_cache/test_prefix_tracker.py`: the fan-out test asserts
per-line state instead of tracker object identity.

## Testing

- [x] Unit tests pass (`pytest
tests/test_anthropic_fork_lineage_cache.py tests/test_cache/
tests/test_proxy_anthropic_cache_stability.py`: 272 passed, with
`test_cache/test_dynamic_detector.py` run on its own: 57 passed, 2
skipped)
- [x] Linting passes (`ruff check` on the changed files)
- [x] Type checking passes for the changed module (`mypy
headroom/cache/prefix_tracker.py`: no issues)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
# new test file, without the fix (parent commit)
FAILED tests/test_anthropic_fork_lineage_cache.py::test_main_turn_after_a_fork_replays_its_cached_prefix[summary]
FAILED tests/test_anthropic_fork_lineage_cache.py::test_main_turn_after_a_fork_replays_its_cached_prefix[side-query]
FAILED tests/test_anthropic_fork_lineage_cache.py::test_fork_does_not_replace_the_lineage_it_branched_from[fork-first]
FAILED tests/test_anthropic_fork_lineage_cache.py::test_fork_does_not_replace_the_lineage_it_branched_from[main-first]
  AssertionError: main turn 2 re-sent turn 1's history with different bytes
============================== 4 failed ==============================

# with the fix
============================== 4 passed ==============================
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13.16, real Claude Code 2.1.295
sessions against the Anthropic API through `headroom proxy --mode token`
(details below).
- Exact command / steps: the scripted Claude Code session described
below, run once on `17abee27` (before) and once on `9b27664a` (after).
- Observed result: main turns right after a fork arrived with a cold
tracker 5/5 before, 0/5 after; cached tokens lost 3,363 before, 0 after.
- Not tested: the full collapse to the system prompt seen in production
fork-heavy sessions (did not reproduce in this short session); a
deployed proxy running this fix.

Live before/after capture on 2026-10-09. Real Claude Code 2.1.295
sessions against the Anthropic API (subscription login,
`claude-sonnet-5-5`), through `headroom proxy --mode token` on Windows
11 / Python 3.13.16, with the native extension built in each tree.

- **Before:** `17abee27` (merge base with `main`)
- **After:** `9b27664a` (this PR)

Both ran the same scripted session: a stream-json SDK client with
`agentProgressSummaries` and `promptSuggestions` on, one general-purpose
subagent that `cat`s ten ~40 KB JSON files one tool call at a time, then
three parent follow-up turns. Each request's kind comes from Claude
Code's own debug log (`[API REQUEST] source=…`). `sdk` and `agent:*` are
main turns; `agent_summary` and `prompt_suggestion` are forks. Token
counts are the upstream `usage` values from the proxy's PERF lines.

| Main turns right after a fork | Before | After |
|---|---|---|
| Count | 5 | 5 |
| Arrived with a cold tracker (no `Pipeline: freezing` line) | **5/5** |
**0/5** |
| Read the full prior prefix | 4/5 | **5/5** |
| Cached tokens lost | 3,363 | 0 |

- **Before**, subagent turn #15 (after its 23-message `agent_summary`
fork): cold, `cache_read` 22,137 against the 25,500 cached by the
previous turn, `cache_creation` 4,902.
- **After**, the matching subagent turn #17 (after a 21-message
`agent_summary` fork): frozen 20/23, `cache_read` 25,311 = 24,192 +
1,119, exactly the previous turn's read + write.

This short session did not reproduce the full collapse to the system
prompt (~6k read) seen in production fork-heavy sessions. In most cold
turns the pipeline happened to regenerate identical bytes. The capture
does show the mechanism deterministically: before, every fork leaves the
next main turn of that conversation starting cold, and one of those cold
turns rewrote 3.4k cached tokens. After, none start cold and every
post-fork turn reads its whole prior prefix.

<details><summary>Per-request tables (before / after)</summary>

**Before (`17abee27`)**

| # | kind | msgs | frozen | cache_read_input_tokens |
cache_creation_input_tokens |
|---|---|---|---|---|---|
| 1 | main (parent) | 2 | - | 0 | 15,072 |
| 2 | main (subagent) | 2 | - | 5,406 | 7,859 |
| 3 | main (subagent) | 5 | 3 | 13,265 | 660 |
| 4 | main (subagent) | 7 | 3 | 13,925 | 1,176 |
| 5 | main (subagent) | 9 | 3 | 15,101 | 1,319 |
| 6 | main (subagent) | 11 | 3 | 16,420 | 1,138 |
| 7 | fork (agent_summary) | 11 | 3 | 16,476 | 0 |
| 8 | main (subagent) <- after fork | 13 | - | 17,558 | 1,127 |
| 9 | main (subagent) | 15 | 3 | 18,685 | 1,134 |
| 10 | main (subagent) | 17 | 3 | 19,819 | 1,126 |
| 11 | main (subagent) | 19 | 3 | 20,945 | 1,136 |
| 12 | main (subagent) | 21 | 3 | 22,081 | 1,133 |
| 13 | main (subagent) | 23 | 3 | 23,214 | 2,286 |
| 14 | fork (agent_summary) | 23 | 22 | 23,321 | 0 |
| 15 | main (subagent) <- after fork | 25 | - | 22,137 | 4,902 |
| 16 | main (subagent) | 27 | 24 | 27,039 | 1,260 |
| 17 | main (subagent) | 29 | 24 | 28,299 | 7,523 |
| 18 | main (parent) | 4 | 3 | 15,072 | 2,204 |
| 19 | fork (prompt_suggestion) | 6 | 5 | 17,276 | 335 |
| 20 | main (parent) <- after fork | 6 | - | 17,611 | 31 |
| 21 | fork (prompt_suggestion) | 8 | 7 | 17,642 | 362 |
| 22 | main (parent) <- after fork | 8 | - | 18,004 | 29 |
| 23 | main (parent) | 10 | 9 | 18,033 | 242 |
| 24 | fork (prompt_suggestion) | 12 | 9 | 18,275 | 128 |
| 25 | main (parent) <- after fork | 12 | - | 18,403 | 18 |
| 26 | fork (prompt_suggestion) | 14 | 9 | 18,421 | 142 |

**After (`9b27664a`)**

| # | kind | msgs | frozen | cache_read_input_tokens |
cache_creation_input_tokens |
|---|---|---|---|---|---|
| 1 | main (parent) | 2 | - | 5,406 | 9,666 |
| 2 | main (subagent) | 2 | - | 10,523 | 2,742 |
| 3 | main (parent) | 4 | 3 | 15,072 | 692 |
| 4 | main (subagent) | 5 | 3 | 13,265 | 1,681 |
| 5 | fork (prompt_suggestion) | 6 | 3 | 15,764 | 25 |
| 6 | main (subagent) | 7 | 3 | 14,946 | 1,246 |
| 7 | main (subagent) | 9 | 3 | 16,192 | 1,138 |
| 8 | main (subagent) | 11 | 3 | 17,330 | 2,303 |
| 9 | main (subagent) | 13 | 3 | 19,633 | 1,167 |
| 10 | main (subagent) | 15 | 3 | 20,800 | 1,137 |
| 11 | main (subagent) | 17 | 3 | 21,937 | 1,129 |
| 12 | main (subagent) | 19 | 3 | 23,066 | 1,126 |
| 13 | main (subagent) | 21 | 20 | 24,192 | 1,119 |
| 14 | main (parent) <- after fork | 6 | 3 | 15,789 | 31 |
| 15 | fork (prompt_suggestion) | 8 | 3 | 15,820 | 108 |
| 16 | fork (agent_summary) | 21 | 20 | 24,248 | 0 |
| 17 | main (subagent) <- after fork | 23 | 20 | 25,311 | 710 |
| 18 | main (parent) <- after fork | 8 | 3 | 15,928 | 851 |
| 19 | fork (prompt_suggestion) | 10 | 3 | 16,779 | 688 |
| 20 | main (parent) <- after fork | 10 | 3 | 17,467 | 29 |
| 21 | main (parent) | 12 | 3 | 17,496 | 130 |
| 22 | main (parent) | 14 | 3 | 17,626 | 249 |
| 23 | fork (prompt_suggestion) | 16 | 3 | 17,875 | 131 |
| 24 | main (parent) <- after fork | 16 | 3 | 18,006 | 18 |
| 25 | fork (prompt_suggestion) | 18 | 3 | 18,024 | 136 |

</details>

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, prefix-tracker lineage matching
for requests that extend an earlier snapshot of the same lineage
- Kill switch / disable path: none
- Unsafe override required: no
- Qualification impact: none expected
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Not covered here: a few small partial misses in production that forks
don't explain, and a synthetic-only risk that an older tool result gets
recompressed once it falls past a lagging frozen boundary.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4043"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Serge ARADJ
2026-10-13 16:21:38 -07:00
committed by GitHub
co-authored by Claude Opus 5.5 JerrettDavis
parent a9eb2a7b6f
commit 1cf4d053d2
3 changed files with 523 additions and 42 deletions
+142 -22
View File
@@ -23,7 +23,8 @@ import json
import logging
import os
import time
from collections import OrderedDict
from collections import OrderedDict, deque
from contextvars import ContextVar
from dataclasses import dataclass
from typing import Any
@@ -63,6 +64,22 @@ _PROVIDER_CACHE_TTL_SECONDS = {
# cold-prefix detection, but never below the configured session cleanup floor.
_CACHE_TTL_CLEANUP_MARGIN_SECONDS = 60
# Recent request snapshots kept per lineage. A Claude Code fork (the subagent
# progress summary, the parent's side query) re-sends the history plus one
# prompt, so the next real turn extends the snapshot BEFORE the fork. A
# request that resumes an older snapshot keeps it next-newest, so any number
# of forks off one turn leaves that turn resumable.
_LINEAGE_HISTORY = 3
# (tracker, turn, resumed state) the current request resolved to. Each proxied
# request runs in its own asyncio task, so it reads the state it resumed and
# ``update_from_response`` records its response against its own turn, even
# while a fork of the same conversation is in flight on the same tracker. Same
# pattern as the request-scoped tenant key and CCR store.
_resolved_turn: ContextVar[tuple[Any, int, tuple[Any, ...] | None] | None] = ContextVar(
"headroom_resolved_lineage_turn", default=None
)
@dataclass
class PrefixFreezeConfig:
@@ -1153,6 +1170,11 @@ class PrefixCacheTracker:
# net-cost/TTL P_alive gate could never see idle time. The handler reads
# this and forwards it to the pipeline as `idle_seconds`.
self._idle_seconds_at_fetch: float = 0.0
# Turn state after each recent request's response (None while it is in
# flight), keyed by the turn SessionTrackerStore issued it. A request
# resumes the state of the turn it extends, so a fork's response never
# becomes the main line's frozen prefix or replay source.
self._turn_states: dict[int, tuple[Any, ...] | None] = {}
# Session-scoped ReadMaturationManager (Mechanism B), created
# lazily by the handler when read maturation is enabled. Rides
@@ -1169,13 +1191,14 @@ class PrefixCacheTracker:
Returns 0 on turn 0 (cold start) or if caching is disabled/below threshold.
"""
turn_number, cached_tokens, cached_messages, _, _ = self._request_turn_state()
if not self.config.enabled:
return 0
if self._turn_number == 0:
if turn_number == 0:
return 0
if self._cached_token_count < self.config.min_cached_tokens:
if cached_tokens < self.config.min_cached_tokens:
return 0
return self._cached_message_count
return int(cached_messages)
def update_from_response(
self,
@@ -1208,6 +1231,7 @@ class PrefixCacheTracker:
if total_cached == 0:
self._cached_token_count = 0
self._cached_message_count = 0
self._record_turn_state()
return
# Estimate per-message token counts if not provided
@@ -1227,6 +1251,7 @@ class PrefixCacheTracker:
self._cached_token_count = total_cached
self._cached_message_count = frozen_count
self._record_turn_state()
logger.debug(
"PrefixCacheTracker[%s]: turn=%d, cached=%d tokens, "
@@ -1241,10 +1266,56 @@ class PrefixCacheTracker:
)
def get_last_original_messages(self) -> list[dict[str, Any]]:
return copy.deepcopy(self._last_original_messages)
return copy.deepcopy(self._request_turn_state()[3])
def get_last_forwarded_messages(self) -> list[dict[str, Any]]:
return copy.deepcopy(self._last_forwarded_messages)
return copy.deepcopy(self._request_turn_state()[4])
def _turn_state(self) -> tuple[Any, ...]:
return (
self._turn_number,
self._cached_token_count,
self._cached_message_count,
self._last_original_messages,
self._last_forwarded_messages,
)
def _request_turn_state(self) -> tuple[Any, ...]:
"""The state this request resumed at resolve time, else the current one.
A handler may read after an ``await`` (OpenAI's overlay runs after
compression), by which time a sibling request on this lineage may have
resumed a different turn.
"""
resolved = _resolved_turn.get()
if resolved is not None and resolved[0] is self and resolved[2] is not None:
return resolved[2]
return self._turn_state()
def _record_turn_state(self) -> None:
"""Remember this response's state under the turn that sent it."""
resolved = _resolved_turn.get()
if resolved is not None and resolved[0] is self:
if resolved[1] in self._turn_states:
self._turn_states[resolved[1]] = self._turn_state()
# The request has its response; later reads see the current state.
_resolved_turn.set((self, resolved[1], None))
def _resume_turn(self, turn: int | None) -> None:
"""Continue from ``turn``'s response, not whichever response came last.
A turn still in flight (or one recorded before turn tracking) has no
state yet; the current state stands, as it did before.
"""
state = self._turn_states.get(turn) if turn is not None else None
if state is not None:
(
self._turn_number,
self._cached_token_count,
self._cached_message_count,
self._last_original_messages,
self._last_forwarded_messages,
) = state
def record_returned(
self,
@@ -1311,7 +1382,8 @@ class PrefixCacheTracker:
if idle_seconds is None:
idle_seconds = self.seconds_since_activity()
ttl = self.resolved_cache_ttl_seconds()
expected = self._cached_token_count
# The turn this request resumed, not whichever sibling answered last.
expected = self._request_turn_state()[1]
# Nothing was cached last turn → cold start, not a miss.
if expected <= 0:
@@ -1366,7 +1438,7 @@ class PrefixCacheTracker:
content. Anything else (a frozen message rewritten, the prefix
reordered, the list now shorter) counts as a prefix change.
"""
prev = self._last_forwarded_messages
prev: list[dict[str, Any]] = self._request_turn_state()[4]
if not prev:
# No recorded prefix to compare — can't claim it changed.
return True
@@ -1565,6 +1637,10 @@ class SessionTrackerStore:
# but different tool profiles must never share frozen-prefix state.
self._lineage_affinities: dict[str, str | None] = {}
self._lineage_counter = itertools.count(1)
# Recent (snapshot, turn) per tracker key, newest first; the newest
# snapshot is the one in ``_lineages``.
self._lineage_history: dict[str, deque[tuple[list[Any], int]]] = {}
self._turn_counter = itertools.count(1)
def peek(self, session_id: str) -> PrefixCacheTracker | None:
"""Return the live tracker for ``session_id``, else None.
@@ -1614,6 +1690,7 @@ class SessionTrackerStore:
def _discard_tracker(self, tracker_key: str) -> None:
"""Remove a tracker and any lineage indexes that point to it."""
self._trackers.pop(tracker_key, None)
self._lineage_history.pop(tracker_key, None)
for session_id, family in list(self._lineages.items()):
family.pop(tracker_key, None)
self._lineage_affinities.pop(tracker_key, None)
@@ -1707,22 +1784,48 @@ class SessionTrackerStore:
for key in expired_keys:
family.pop(key, None)
self._lineage_affinities.pop(key, None)
self._lineage_history.pop(key, None)
self._trackers.pop(key, None)
by_length = sorted(family.items(), key=lambda item: len(item[1]), reverse=True)
best_key: str | None = None
for accepted in (
(RELATION_EXACT, RELATION_MESSAGE_APPEND),
(RELATION_BLOCK_APPEND,),
):
for key, chain in by_length:
if self._lineage_affinities.get(key) != cache_affinity:
continue
relation = _classify_history_canonical(snap, chain)
if relation.kind in accepted:
best_key = key
break
if best_key is not None:
break
def continuation(
candidates: list[tuple[str, list[Any], int | None]],
) -> tuple[str | None, int | None]:
for accepted in (
(RELATION_EXACT, RELATION_MESSAGE_APPEND),
(RELATION_BLOCK_APPEND,),
):
for key, chain, turn in candidates:
if self._lineage_affinities.get(key) != cache_affinity:
continue
if _classify_history_canonical(snap, chain).kind in accepted:
return key, turn
return None, None
def latest_turn(key: str) -> int | None:
history = self._lineage_history.get(key)
return history[0][1] if history else None
best_key, best_turn = continuation(
[(key, chain, latest_turn(key)) for key, chain in by_length]
)
if best_key is None:
# A fork (Claude Code's subagent summary or side query: the
# history plus one prompt) became a lineage's latest snapshot, so
# this turn extends the snapshot before it. Resume that turn
# instead of starting a cold lineage that recompresses history the
# provider already cached.
best_key, best_turn = continuation(
sorted(
(
(key, chain, turn)
for key in family
for chain, turn in list(self._lineage_history.get(key, ()))[1:]
),
key=lambda item: len(item[1]),
reverse=True,
)
)
if best_key is None:
rewrite_candidates: list[tuple[tuple[int, int, int], str]] = []
@@ -1808,6 +1911,22 @@ class SessionTrackerStore:
family = self._lineages.setdefault(session_id, family)
family[best_key] = snap
self._lineage_affinities[best_key] = cache_affinity
if isinstance(tracker, PrefixCacheTracker): # test stubs keep no turns
tracker._resume_turn(best_turn if best_turn is not None else latest_turn(best_key))
turn = next(self._turn_counter)
history = self._lineage_history.setdefault(best_key, deque(maxlen=_LINEAGE_HISTORY))
branch = next((entry for entry in history if entry[1] == best_turn), None)
if branch is not None and branch is not history[0]:
# A fork off an older turn: that turn stays the branch point the
# next main turn extends, so keep it behind this request rather
# than let successive forks push it out.
history.remove(branch)
history.appendleft(branch)
history.appendleft((snap, turn))
live = {t for _, t in history}
tracker._turn_states = {t: s for t, s in tracker._turn_states.items() if t in live}
tracker._turn_states[turn] = None
_resolved_turn.set((tracker, turn, tracker._turn_state()))
return tracker
def compute_session_id(
@@ -1877,6 +1996,7 @@ class SessionTrackerStore:
for key in [k for k in family if k not in self._trackers]:
del family[key]
self._lineage_affinities.pop(key, None)
self._lineage_history.pop(key, None)
if not family:
del self._lineages[base]
logger.debug("SessionTrackerStore: cleaned up %d expired sessions", len(expired))
+343
View File
@@ -0,0 +1,343 @@
"""A Claude Code side request must not cost the main conversation its prompt cache.
Production, 2026-10-08: a W.A.L.T.E.R. review subagent busted Anthropic's
prompt cache on 35 of its 81 turns (``cache_read_input_tokens`` fell to the
6,065-token system prompt while ~100-150k tokens were re-written), and its
600k-token parent session did the same after its own side requests. Every
bust followed a *fork*: Claude Code re-sends the agent's history plus one
extra prompt (the subagent progress summary merges it into the last user
message; the parent's side query appends a reply and a new user message) with
its breakpoint one message earlier, so it reads the cache without writing.
The fork extended the conversation's prefix-tracker lineage and replaced its
snapshot. The next main turn extends the history the fork branched from, not
the fork, so it matched no lineage, started a cold tracker (frozen=0) and the
token-mode pipeline recompressed tool results it had forwarded verbatim the
turn before. Different bytes, so the provider's prefix cache missed from the
first changed message onward.
The upstream below models Anthropic's prompt cache (a prefix is readable once
a breakpoint wrote it), so the assertions are on what the provider would do
with the bytes the handler forwards.
"""
from __future__ import annotations
import contextvars
import copy
import json
from typing import Any
import pytest
pytest.importorskip("fastapi")
import httpx
from fastapi.testclient import TestClient
from headroom.cache.prefix_tracker import PrefixCacheTracker, SessionTrackerStore
from headroom.proxy.server import ProxyConfig, create_app
MODEL = "claude-sonnet-4-5-20250929"
SYSTEM = [
{
"type": "text",
"text": "You review pull requests. Report only real defects. " * 40,
"cache_control": {"type": "ephemeral"},
}
]
BASH_TOOL = {
"name": "Bash",
"description": "Run a shell command",
"input_schema": {
"type": "object",
"properties": {"command": {"type": "string"}},
"required": ["command"],
},
}
MARKER = {"type": "ephemeral"}
def _bash_turn(i: int) -> list[dict[str, Any]]:
tid = f"toolu_{i:02d}"
rows = [
{"path": f"src/module_{i}/file_{j}.py", "size": 1000 + j, "status": "ok"}
for j in range(400)
]
return [
{
"role": "assistant",
"content": [
{"type": "tool_use", "id": tid, "name": "Bash", "input": {"command": f"ls {i}"}}
],
},
{
"role": "user",
"content": [{"type": "tool_result", "tool_use_id": tid, "content": json.dumps(rows)}],
},
]
def _main(turns: int) -> list[dict[str, Any]]:
"""The agent's own request: Claude Code's breakpoint rides the newest block."""
messages: list[dict[str, Any]] = [
{"role": "user", "content": [{"type": "text", "text": "Review the change."}]}
]
for i in range(turns):
messages += _bash_turn(i)
messages[-1]["content"][-1]["cache_control"] = dict(MARKER)
return messages
def _summary_fork(turns: int) -> list[dict[str, Any]]:
"""Subagent progress summary: same history, the prompt merged into the last
user message, breakpoint one message earlier (reads, never writes)."""
messages = _main(turns)
del messages[-1]["content"][-1]["cache_control"]
messages[-1]["content"].append({"type": "text", "text": "Summarize your progress in 5 words."})
messages[-2]["content"][-1]["cache_control"] = dict(MARKER)
return messages
def _side_query_fork(turns: int) -> list[dict[str, Any]]:
"""Parent-session side query: history + the agent's reply + a new prompt."""
messages = _main(turns)
messages.append({"role": "assistant", "content": [{"type": "text", "text": "Done."}]})
messages.append({"role": "user", "content": [{"type": "text", "text": "Name this session."}]})
del messages[-3]["content"][-1]["cache_control"]
messages[-2]["content"][-1]["cache_control"] = dict(MARKER)
return messages
def _strip(obj: Any) -> Any:
if isinstance(obj, dict):
return {k: _strip(v) for k, v in obj.items() if k != "cache_control"}
if isinstance(obj, list):
return [_strip(v) for v in obj]
return obj
def _has_marker(message: dict[str, Any]) -> bool:
content = message.get("content")
return isinstance(content, list) and any(
isinstance(block, dict) and "cache_control" in block for block in content
)
class _PromptCache:
"""Anthropic's prompt cache at message granularity.
A request reads the longest prefix an earlier breakpoint wrote, then
writes a cache entry at each of its own breakpoints. Tools and system are
part of every key because Anthropic renders them before the messages.
"""
def __init__(self) -> None:
self._written: set[str] = set()
def request(self, body: dict[str, Any]) -> tuple[int, int, int]:
messages = body.get("messages") or []
head = json.dumps(_strip([body.get("tools"), body.get("system")]), sort_keys=True)
keys = [
head + json.dumps(_strip(messages[: i + 1]), sort_keys=True)
for i in range(len(messages))
]
marks = [i for i, m in enumerate(messages) if _has_marker(m)]
if not marks:
return 0, 0, -1
hit = max((i for i in range(marks[-1] + 1) if keys[i] in self._written), default=-1)
self._written.update(keys[i] for i in marks)
# The tracker's own estimate: usage it can map back onto exactly the
# messages the provider cached, so only the fork can move the floor.
tokens = PrefixCacheTracker._estimate_message_tokens(messages)
system_tokens = len(json.dumps(body.get("system"))) // 4
read = system_tokens + sum(tokens[: hit + 1])
write = sum(tokens[hit + 1 : marks[-1] + 1])
return read, write, hit
def _drive(requests: list[list[dict[str, Any]]]) -> list[tuple[list[dict[str, Any]], int]]:
"""Send each history through the real handler; return (forwarded, read_hit)."""
app = create_app(
ProxyConfig(
optimize=True,
mode="token",
# Deterministic structural compression only: the Kompress model
# is not available everywhere the suite runs.
disable_kompress=True,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
)
)
cache = _PromptCache()
seen: list[tuple[list[dict[str, Any]], int]] = []
with TestClient(app) as client:
proxy = client.app.state.proxy
async def _upstream(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001, ANN003
read, write, hit = cache.request(body)
seen.append((copy.deepcopy(body.get("messages") or []), hit))
return httpx.Response(
200,
json={
"id": "msg_x",
"type": "message",
"role": "assistant",
"model": MODEL,
"content": [{"type": "text", "text": "ok"}],
"stop_reason": "end_turn",
"usage": {
"input_tokens": 10,
"output_tokens": 2,
"cache_read_input_tokens": read,
"cache_creation_input_tokens": write,
},
},
)
proxy._retry_request = _upstream
for messages in requests:
response = client.post(
"/v1/messages",
headers={
"x-api-key": "test-key",
"anthropic-version": "2023-06-01",
"content-type": "application/json",
},
json={
"model": MODEL,
"max_tokens": 64,
"system": SYSTEM,
"tools": [BASH_TOOL],
"messages": messages,
},
)
assert response.status_code == 200, response.text[:300]
assert len(seen) == len(requests)
return seen
@pytest.mark.parametrize("fork", [_summary_fork, _side_query_fork], ids=["summary", "side-query"])
def test_main_turn_after_a_fork_replays_its_cached_prefix(monkeypatch, fork) -> None: # noqa: ANN001
from headroom.cache.compression_store import reset_compression_store
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "memory")
reset_compression_store()
turns = range(1, 6)
requests: list[list[dict[str, Any]]] = []
for n in turns:
requests += [_main(n), fork(n)]
seen = _drive(requests)
mains = seen[0::2]
# Not vacuous: the pipeline did rewrite tool results somewhere.
assert any(
_strip(forwarded) != _strip(original) for (forwarded, _), original in zip(seen, requests)
), "nothing was compressed; the test cannot observe a prefix change"
for (previous, _), (current, hit), n in zip(mains, mains[1:], turns):
assert _strip(current[: len(previous)]) == _strip(previous), (
f"main turn {n + 1} re-sent turn {n}'s history with different bytes"
)
assert hit >= len(previous) - 1, (
f"main turn {n + 1} read only {hit + 1}/{len(previous)} cached messages"
)
def _record(tracker, messages, *, cached: int) -> None: # noqa: ANN001
tracker.update_from_response(
cache_read_tokens=cached,
cache_write_tokens=0,
messages=messages,
original_messages=messages,
message_token_counts=[1000] * len(messages),
)
@pytest.mark.parametrize("fork_responds_first", [True, False], ids=["fork-first", "main-first"])
def test_fork_does_not_replace_the_lineage_it_branched_from(fork_responds_first) -> None: # noqa: ANN001
"""Store level, both orderings seen in production: the summary fork is sent
while a slow main turn is still in flight (fork answers first), or after a
fast one answered (main answers first). Either way the next main turn must
resume the main turn's state, not the fork's and not a cold tracker.
Each request runs in its own context, as each proxied request runs in its
own asyncio task."""
store = SessionTrackerStore()
main_k, fork_k, main_next = _strip(_main(3)), _strip(_summary_fork(3)), _strip(_main(4))
main_ctx, fork_ctx = contextvars.Context(), contextvars.Context()
main_tracker = main_ctx.run(store.resolve_tracker, "sid", "anthropic", messages=main_k)
if not fork_responds_first:
main_ctx.run(_record, main_tracker, main_k, cached=1000 * len(main_k))
fork_tracker = fork_ctx.run(store.resolve_tracker, "sid", "anthropic", messages=fork_k)
fork_ctx.run(_record, fork_tracker, fork_k, cached=1000 * (len(fork_k) - 1))
if fork_responds_first:
main_ctx.run(_record, main_tracker, main_k, cached=1000 * len(main_k))
resumed = contextvars.Context().run(
store.resolve_tracker, "sid", "anthropic", messages=main_next
)
assert resumed is main_tracker
assert resumed.get_frozen_message_count() == len(main_k)
assert resumed.get_last_original_messages() == main_k
@pytest.mark.parametrize("forks", [1, 3, 5])
def test_successive_forks_keep_the_turn_they_branch_from(forks) -> None: # noqa: ANN001
"""Several forks off one main turn (progress summaries, side queries) must
not push that turn out of the lineage history: the next main turn still
extends it, so it must resume its frozen prefix rather than start cold."""
store = SessionTrackerStore()
base = _strip(_main(3))
main_tracker = contextvars.Context().run(_send, store, base)
for i in range(forks):
side = base + [
{"role": "assistant", "content": [{"type": "text", "text": f"Done {i}."}]},
{"role": "user", "content": [{"type": "text", "text": f"Side query {i}."}]},
]
contextvars.Context().run(_send, store, side)
main_next = _strip(_main(4))
resumed = contextvars.Context().run(
store.resolve_tracker, "sid", "anthropic", messages=main_next
)
assert resumed is main_tracker
assert resumed.get_frozen_message_count() == len(base)
assert resumed.get_last_original_messages() == base
def _send(store: SessionTrackerStore, messages: list[dict[str, Any]]) -> PrefixCacheTracker:
tracker = store.resolve_tracker("sid", "anthropic", messages=messages)
_record(tracker, messages, cached=1000 * len(messages))
return tracker
@pytest.mark.parametrize("fork_responds_first", [True, False], ids=["fork-first", "main-first"])
def test_cache_miss_is_classified_against_the_resumed_turn(fork_responds_first) -> None: # noqa: ANN001
"""A sibling request answering while this one is in flight must not change
what this request's cache outcome is measured against."""
store = SessionTrackerStore()
base = _strip(_main(3))
contextvars.Context().run(_send, store, base)
main_next, fork = _strip(_main(4)), _strip(_summary_fork(3))
main_ctx, fork_ctx = contextvars.Context(), contextvars.Context()
main_tracker = main_ctx.run(store.resolve_tracker, "sid", "anthropic", messages=main_next)
fork_tracker = fork_ctx.run(store.resolve_tracker, "sid", "anthropic", messages=fork)
assert fork_tracker is main_tracker
if fork_responds_first:
# The fork read nothing: the shared tracker now expects no cache.
fork_ctx.run(_record, fork_tracker, fork, cached=0)
# The main turn misses on a prefix its parent cached.
attribution = main_ctx.run(main_tracker.classify_cache_miss, 0, main_next, 0.0)
assert attribution.is_miss
assert attribution.expected_cached_tokens == 1000 * len(base)
assert not attribution.prefix_changed
if not fork_responds_first:
fork_ctx.run(_record, fork_tracker, fork, cached=0)
+38 -20
View File
@@ -1,5 +1,6 @@
"""Tests for PrefixCacheTracker — cache-aware compression."""
import contextvars
import time
import pytest
@@ -744,33 +745,50 @@ class TestConversationLineageResolution:
assert trackers["A"]._turn_number == 4
assert trackers["B"]._turn_number == 4
def test_identical_first_turns_share_until_divergence_then_split(self, store):
"""Templated fan-outs send byte-identical first turns. While histories
are identical, sharing a tracker is harmless (the provider cache line
is identical too); they must split as soon as the histories diverge."""
def test_identical_first_turns_keep_per_line_state_after_divergence(self, store):
"""Templated fan-outs send byte-identical first turns, then diverge.
Each line extends its own earlier snapshot and must resume that turn's
frozen prefix and replay source, never its sibling's. (A sibling that
extends an earlier snapshot is indistinguishable from a main turn after
a Claude Code fork, so they share the lineage rather than split.)
Each request runs in its own context, as each proxied request runs in
its own asyncio task."""
sid = "shared"
first = [{"role": "user", "content": "verify the fix " + "p" * 300}]
t_a1 = store.resolve_tracker(sid, "anthropic", messages=first)
t_b1 = store.resolve_tracker(sid, "anthropic", messages=first)
assert t_b1 is t_a1
a2 = first + [
{"role": "assistant", "content": "answer A"},
{"role": "user", "content": "next A"},
]
b2 = first + [
{"role": "assistant", "content": "answer B"},
{"role": "user", "content": "next B"},
]
t_a2 = store.resolve_tracker(sid, "anthropic", messages=a2)
t_b2 = store.resolve_tracker(sid, "anthropic", messages=b2)
assert t_a2 is not t_b2
def turn(history: list[dict], frozen: int) -> PrefixCacheTracker:
def send() -> PrefixCacheTracker:
tracker = store.resolve_tracker(sid, "anthropic", messages=history)
tracker.update_from_response(
cache_read_tokens=2000 * frozen,
cache_write_tokens=0,
messages=history,
message_token_counts=[2000] * len(history),
)
return tracker
a3 = a2 + [
{"role": "assistant", "content": "answer A2"},
{"role": "user", "content": "next A2"},
]
assert store.resolve_tracker(sid, "anthropic", messages=a3) is t_a2
return contextvars.Context().run(send)
def reply(history: list[dict], text: str) -> list[dict]:
return history + [
{"role": "assistant", "content": f"answer {text}"},
{"role": "user", "content": f"next {text}"},
]
a2, b2 = reply(first, "A"), reply(first, "B")
turn(a2, frozen=3)
turn(b2, frozen=2)
a3 = store.resolve_tracker(sid, "anthropic", messages=reply(a2, "A2"))
assert a3.get_last_original_messages() == a2
assert a3.get_frozen_message_count() == 3
b3 = store.resolve_tracker(sid, "anthropic", messages=reply(b2, "B2"))
assert b3.get_last_original_messages() == b2
assert b3.get_frozen_message_count() == 2
@pytest.mark.parametrize(
"cc_turn2",