Files
headroom/tests/test_anthropic_fork_lineage_cache.py
gglucassandClaude Opus 5.5 62b3bedaba fix(cache): keep prefix-tracker state across a graceful proxy restart (#4299)
## Description

Every proxy restart rewrites the prompt cache of every warm
conversation. After a restart the proxy has no `PrefixCacheTracker` for
the conversation, so it compresses the whole history afresh, the early
messages come out as different bytes than the prefix it forwarded before
the restart, and the provider re-writes the cache from the first changed
message.

Measured on 2026-10-19 on one Mac running Claude Code through the proxy:
after each graceful restart (app relaunches for updates: proxy SIGTERM'd
and respawned), the next request of every active session read only ~25k
cached tokens (tools + system) and re-wrote the rest, up to 247k, 310k,
361k and 432k tokens at a time. The restart-aligned rewrites came to
about 4.6M tokens of 1h cache writes that day (2x the input price),
about a quarter of all cache writes. A client-side integrity probe
reported `kind=tracker_reset, stable_messages=861,
first_changed_message=3`.

This adds an opt-in `HEADROOM_PREFIX_STATE_FILE=<path>`. When set, the
proxy saves the `SessionTrackerStore` on graceful shutdown and restores
it at startup, before serving, so the first request after a restart
replays byte-identical frozen-prefix bytes. Unset, nothing changes. The
file holds conversation content, so it is never a default.

Related but different: #4156 (away recap replaces the lineage), #4287
(CCR proactive expansion rewrites), #4204 (deferred tool loads in the
lineage key), #4284 (mid-history divergence starts a cold lineage).

No existing issue; found while auditing one machine's token usage.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `SessionTrackerStore.save_state` / `load_state`
(`headroom/cache/prefix_tracker.py`). Per tracker: turn number, cached
token and message counts, last original and last forwarded messages, the
per-turn states that lineage resolution resumes, provider, last activity
and observed cache TTL; per lineage: session id, cache affinity and the
recent snapshot history. So `resolve_tracker` matches the next request
to the restored tracker, forks included. Each distinct message is stored
once per tracker (successive turns, snapshots and the original/forwarded
pairs mostly repeat), and new lineage keys and turn ids continue past
the restored ones.
- CCR entries ride along (review: retrieval integrity). Per tracker,
`save_state` collects every CCR reference in its saved messages and
read-maturation state (`hash=<hex>` markers and SmartCrusher's
`<<ccr:<hex>`) and writes the live `CompressionStore` entries behind
them (new `CompressionStore.export_entries`, which reads without
counting a retrieval). A tracker and its entries count against the size
cap together, so a tracker is never saved without them. `load_state`
re-stores them (`import_entries`) with their original expiry, before any
tracker is installed, so a replayed marker resolves from the first
request even with `HEADROOM_CCR_BACKEND=memory`. An entry a persistent
backend still holds is left as it is; a malformed one rejects the file.
- Restore keeps only trackers still inside the store's own retention
(last activity plus the observed or configured cache TTL and its cleanup
margin, at least the session TTL), the same rule the live store uses.
NaN or future timestamps are dropped.
- Versioned schema (`version: 1`). Any read, parse or validation error
logs one warning and leaves the store empty; the whole file is validated
before anything is installed. The file is written atomically
(`headroom.fsutil.write_text`: temp file, fsync, rename) at mode 0600:
an existing file is tightened to 0600 in place, never unlinked first, so
a failed write (ENOSPC) keeps the previous snapshot. Deleted after a
successful load. Stateless mode (`persistence_allowed`) neither writes
nor reads it (a stateless start used to load, and so delete, a
snapshot). Ignored with multiple workers (one file cannot hold N
per-process stores).
- Size cap: the 256 most recently active trackers and at most 64 MB,
newest first; whatever does not fit starts cold, as everything did
before.
- `HeadroomProxy.startup()` loads, `HeadroomProxy.shutdown()` saves as
its first step, before any await, so the shutdown step timeout cannot
cut it short. No periodic writes.
- Token-mode `CompressionCache` seed: a restored tracker has no
`CompressionCache` behind it in the new process, so the freeze computed
from that cache froze nothing: the pipeline recompressed messages the
provider cached, read maturation rewrote Reads inside the cached prefix,
and beyond the provider-confirmed floor the overlay declined a larger
previous form. One shared `seed_restored_compression_cache` seeds the
session's cache from the restored original/forwarded pair
(`update_from_result`) on the first request, on all three token-mode
paths: Anthropic `/v1/messages`, OpenAI `/v1/chat/completions`, and the
session-aware `/v1/compress` (sidecar). It gives the same freeze count
as before the restart.
- `ReadMaturationManager.export_state` / `validate_state` /
`restore_state`: matured markers, booked ids and replay token deltas
ride along with the tracker, so a matured Read's removal is not booked
as savings again on every request after a restart. The saved state is
validated at load with the rest of the file, so a bad entry rejects the
file instead of failing a request.
- `tests/test_anthropic_fork_lineage_cache.py`: `_drive` accepts a
shared provider-cache model and config overrides, so a second call is a
proxy restart in front of a warm provider cache.
- Documented the variable in `docs/content/docs/configuration.mdx`.
- `pyproject.toml` / `uv.lock` (review: Dependency audit): the `aiohttp`
constraint floor is now `>=3.14.4` (CVE-2026-111063, CVE-2026-111064 and
GHSA-2g87-pp6c-29x5, all fixed in 3.14.4), relocked with `uv lock
--upgrade-package aiohttp==3.14.4`. Only aiohttp (3.14.3 -> 3.14.4) and
yarl (1.22.0 -> 1.25.1, which 3.14.4 requires) move. Main still resolves
3.14.3, so the same commit is on #4299, #4286, #4285 and #4269.

Remaining gaps (unchanged by this PR, none of them changes forwarded
message bytes):
- Session-sticky `anthropic-beta` union (`HEADROOM_BETA_HEADER_STICKY`):
in memory, so after a restart a session's sticky set starts from the
client's own header again. Headers only.
- Output shaper: stateless per request (byte-stable steering text,
effort from the current request's structure), so nothing to persist.
- Gemini and Bedrock paths resume the restored tracker and its overlay
replay; they keep no token-mode `CompressionCache`, so there is nothing
to seed. A crash or SIGKILL still loses everything, by design (no
periodic writes).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# New regression tests for the review findings, run against the c141e2ec source
# (headroom/ stashed, new tests kept): all six fail, the 15 earlier tests pass.
$ uv run --frozen --extra dev pytest tests/test_prefix_tracker_restart.py -q
FAILED tests/test_prefix_tracker_restart.py::test_restored_retrieval_references_resolve_to_their_originals
FAILED tests/test_prefix_tracker_restart.py::test_a_stateless_proxy_leaves_a_saved_snapshot_alone
FAILED tests/test_prefix_tracker_restart.py::test_a_failed_save_keeps_the_previous_snapshot
FAILED tests/test_prefix_tracker_restart.py::test_saved_ccr_entries_resolve_after_a_restart
FAILED tests/test_prefix_tracker_restart.py::test_openai_token_mode_replays_its_prefix_after_a_restart
FAILED tests/test_prefix_tracker_restart.py::test_compress_sidecar_session_replays_its_prefix_after_a_restart
6 failed, 15 passed in 8.10s
# Why, in order: hash=cc146b21626392f0dee9890c no longer resolves; the stateless start
# deleted the snapshot; ENOSPC left no snapshot; retrieve() returns None; the OpenAI chat
# and /v1/compress paths re-compressed message 2 under the new query.

# Head c379245e:
$ uv run --frozen --extra dev pytest tests/test_prefix_tracker_restart.py -q
21 passed in 9.50s

$ uv run --frozen --extra dev pytest tests/test_prefix_tracker_restart.py tests/test_anthropic_fork_lineage_cache.py \
    tests/test_cache/test_prefix_tracker.py tests/test_compression_store.py tests/test_ccr_sqlite_backend.py \
    tests/test_issue_2671_block_growth_cache.py tests/test_cache_mode_cold_start.py tests/test_cache_prefix_overlay.py \
    tests/test_cross_turn_cache_safety.py tests/test_session_engine.py tests/test_read_maturation.py \
    tests/test_read_maturation_handler_nobust.py tests/test_graceful_shutdown.py tests/test_proxy_handlers_batch.py \
    tests/test_cache_control_move_bust.py tests/test_startup_log_noise.py tests/test_proxy_openai_cache_stability.py \
    tests/test_compress_session_mode.py tests/test_proxy_anthropic_cache_stability.py tests/test_ccr_row_drop_store_bridge.py \
    tests/test_bedrock_prefix_tracker_wiring.py -q
486 passed, 1 skipped in 46.92s

$ uv run --frozen --extra dev pytest tests/test_anthropic_*.py tests/test_proxy_anthropic_*.py tests/test_proxy_openai*.py \
    tests/test_openai_chat_*.py tests/test_cache_aligner_prefix_stability.py tests/test_cache_prefix_safety.py tests/test_cold_prefix.py \
    tests/test_gemini_prefix_freeze.py tests/test_prefix_tracker_client_rewrote_reply.py tests/test_tool_injection*.py tests/test_ccr_*.py -q
737 passed, 16 skipped in 51.45s

$ uvx ruff@0.17.0 check headroom tests
All checks passed!
$ uvx ruff@0.17.0 format --check headroom tests
1606 files already formatted
$ uv run --frozen --extra dev mypy headroom/cache/prefix_tracker.py headroom/cache/compression_store.py \
    headroom/proxy/handlers/anthropic.py headroom/proxy/handlers/openai.py headroom/proxy/server.py
Success: no issues found in 5 source files
```

## Real Behavior Proof

- Environment: macOS 27 (arm64), this branch at c379245e, the real proxy
process (`python -m headroom.cli proxy --port 6811 --no-http2
--no-rate-limit --mode token --read-maturation
--read-maturation-quiesce-turns 1 --read-maturation-min-size-bytes 200
--disable-kompress --no-subscription-tracking`,
`HEADROOM_ROLLOUT_CHANNEL=beta`, clean `HOME` and workspace,
`HEADROOM_PREFIX_STATE_FILE` set, `HEADROOM_CCR_BACKEND=memory`,
`HEADROOM_BACKGROUND_COMPRESSION=1`) in front of the live Anthropic API,
driven by Claude Code 2.1.303 (`claude -p --model haiku`,
`ANTHROPIC_BASE_URL=http://127.0.0.1:6811`). Each restart is a SIGTERM
(graceful shutdown) and a fresh process.
- Exact command / steps: one Claude Code session through the proxy,
three graceful restarts, the last one without the state file as the
control:
1. Turn 1: `claude -p` reads three files (4.8, 17 and 8.9 KB) one per
turn, 5 requests through the proxy. SIGTERM: `SessionTrackerStore: saved
2 of 2 prefix trackers (181266 bytes)`; file mode `-rw-------`; it holds
2 CCR entries, for the two Reads that read maturation had replaced with
`Retrieve original: hash=` markers.
2. Restart: `restored 2 of 2 prefix trackers`, file deleted. `GET
/v1/retrieve/3a739d6ec5febcd49d0cfc0e` and
`/v1/retrieve/2ee4d4d435432b4cca9b72af` on the new process: HTTP 200,
`original_content` equal to the pre-restart entries (5,129 and 18,525
chars, the Read outputs of the two files). With the in-process CCR
store, c141e2ec loses such entries at a restart (the reviewer's
reproduction;
`test_restored_retrieval_references_resolve_to_their_originals` fails on
it).
3. `claude -p --resume`, then SIGTERM + restart with the state file
again, then `claude -p --resume` again (one request each, no tools).
4. Control: SIGTERM, snapshot moved aside, restart without
`HEADROOM_PREFIX_STATE_FILE`, `claude -p --resume`.
- Observed result: after a restart with the state file the first request
read 99% of its prefix from the provider cache and both restored
retrieval hashes resolved; the control re-wrote everything after the
system prompt (proxy `PERF` lines, which match Claude Code's reported
usage):

  | First request after a restart | cache_read | cache_write | hit |
  |---|---|---|---|
  | With the state file, resume after a resume | 40,406 | 514 | 99% |
| Without the state file (control), same history | 26,053 | 11,250 | 70%
|

With the state file the proxy forwarded messages 0-10 byte-identical to
what it forwarded before the restart (compared from the saved snapshots,
`cache_control` positions excluded since Claude Code moves its
breakpoint every turn), and froze 14 of 15 messages. In the control, the
forwarded bytes changed from message 1, and the third Read, verbatim
inside the cached prefix, matured into a new marker (10,205 -> 368
chars), so everything after the system prompt was re-written.
- Not tested: Kompress itself (`--disable-kompress`; the unit suite uses
a deterministic stand-in). Windows file modes. The OpenAI chat and
`/v1/compress` seeds against a live OpenAI endpoint (covered by the
handler-level restart tests above with a query-dependent compressor).
The very first `--resume` of a `claude -p` session (step 3's first
request) read only 26,053 tokens and wrote 14,845, although the client's
messages and the proxy's forwarded messages 0-10 were byte-identical to
the pre-restart request; the read stopped at the system-prompt
breakpoint, so the change is most likely in the system/tools part of the
request, which I did not capture. The resume after it, across another
restart, read 99%.

## Runtime Rollout Safety

- Rollout-managed feature(s): None; opt-in environment variable
`HEADROOM_PREFIX_STATE_FILE`, unset by default.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: No; with the variable unset nothing
is read or written and no code path changes (the new handler branches
only fire on a tracker restored from the file, and the CCR export/import
only runs inside save/load).
- Kill switch / disable path: Unset `HEADROOM_PREFIX_STATE_FILE`.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Unset the variable or revert this PR; a leftover state
file is inert without the variable. The file format is still `version:
1` (now with a `ccr` list); a file written by c141e2ec has no `ccr` list
and is rejected at load, which starts cold.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

- Review follow-up (c379245e): CCR entries are saved with their trackers
(P1), the seed covers the OpenAI chat and `/v1/compress` token paths, a
failed save keeps the previous snapshot, and stateless mode no longer
loads (and deletes) a snapshot. Each has a regression test that fails on
c141e2ec.
- Caller isolation: the restored store has exactly the keys, lineages
and retention the in-memory store had at shutdown, and replay still
requires the caller to send the same original messages; see the reply on
that thread for the full reasoning.
- The full `pytest tests` run was not repeated for this head (shared
machine); the focused runs above cover every module the change touches.
- Size cap choice: 256 trackers covers many concurrent Claude Code
sessions plus their subagents; 64 MB bounds shutdown time (encoding is
synchronous) and disk use.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4299"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 19:44:00 -05:00

350 lines
14 KiB
Python

"""A Claude Code side request must not cost the main conversation its prompt cache.
Production, 2026-10-08: a W.A.L.T.E.R. review subagent busted Anthropic's
prompt cache on 35 of its 81 turns (``cache_read_input_tokens`` fell to the
6,065-token system prompt while ~100-150k tokens were re-written), and its
600k-token parent session did the same after its own side requests. Every
bust followed a *fork*: Claude Code re-sends the agent's history plus one
extra prompt (the subagent progress summary merges it into the last user
message; the parent's side query appends a reply and a new user message) with
its breakpoint one message earlier, so it reads the cache without writing.
The fork extended the conversation's prefix-tracker lineage and replaced its
snapshot. The next main turn extends the history the fork branched from, not
the fork, so it matched no lineage, started a cold tracker (frozen=0) and the
token-mode pipeline recompressed tool results it had forwarded verbatim the
turn before. Different bytes, so the provider's prefix cache missed from the
first changed message onward.
The upstream below models Anthropic's prompt cache (a prefix is readable once
a breakpoint wrote it), so the assertions are on what the provider would do
with the bytes the handler forwards.
"""
from __future__ import annotations
import contextvars
import copy
import json
from typing import Any
import pytest
pytest.importorskip("fastapi")
import httpx
from fastapi.testclient import TestClient
from headroom.cache.prefix_tracker import PrefixCacheTracker, SessionTrackerStore
from headroom.proxy.server import ProxyConfig, create_app
MODEL = "claude-sonnet-4-5-20250929"
SYSTEM = [
{
"type": "text",
"text": "You review pull requests. Report only real defects. " * 40,
"cache_control": {"type": "ephemeral"},
}
]
BASH_TOOL = {
"name": "Bash",
"description": "Run a shell command",
"input_schema": {
"type": "object",
"properties": {"command": {"type": "string"}},
"required": ["command"],
},
}
MARKER = {"type": "ephemeral"}
def _bash_turn(i: int) -> list[dict[str, Any]]:
tid = f"toolu_{i:02d}"
rows = [
{"path": f"src/module_{i}/file_{j}.py", "size": 1000 + j, "status": "ok"}
for j in range(400)
]
return [
{
"role": "assistant",
"content": [
{"type": "tool_use", "id": tid, "name": "Bash", "input": {"command": f"ls {i}"}}
],
},
{
"role": "user",
"content": [{"type": "tool_result", "tool_use_id": tid, "content": json.dumps(rows)}],
},
]
def _main(turns: int) -> list[dict[str, Any]]:
"""The agent's own request: Claude Code's breakpoint rides the newest block."""
messages: list[dict[str, Any]] = [
{"role": "user", "content": [{"type": "text", "text": "Review the change."}]}
]
for i in range(turns):
messages += _bash_turn(i)
messages[-1]["content"][-1]["cache_control"] = dict(MARKER)
return messages
def _summary_fork(turns: int) -> list[dict[str, Any]]:
"""Subagent progress summary: same history, the prompt merged into the last
user message, breakpoint one message earlier (reads, never writes)."""
messages = _main(turns)
del messages[-1]["content"][-1]["cache_control"]
messages[-1]["content"].append({"type": "text", "text": "Summarize your progress in 5 words."})
messages[-2]["content"][-1]["cache_control"] = dict(MARKER)
return messages
def _side_query_fork(turns: int) -> list[dict[str, Any]]:
"""Parent-session side query: history + the agent's reply + a new prompt."""
messages = _main(turns)
messages.append({"role": "assistant", "content": [{"type": "text", "text": "Done."}]})
messages.append({"role": "user", "content": [{"type": "text", "text": "Name this session."}]})
del messages[-3]["content"][-1]["cache_control"]
messages[-2]["content"][-1]["cache_control"] = dict(MARKER)
return messages
def _strip(obj: Any) -> Any:
if isinstance(obj, dict):
return {k: _strip(v) for k, v in obj.items() if k != "cache_control"}
if isinstance(obj, list):
return [_strip(v) for v in obj]
return obj
def _has_marker(message: dict[str, Any]) -> bool:
content = message.get("content")
return isinstance(content, list) and any(
isinstance(block, dict) and "cache_control" in block for block in content
)
class _PromptCache:
"""Anthropic's prompt cache at message granularity.
A request reads the longest prefix an earlier breakpoint wrote, then
writes a cache entry at each of its own breakpoints. Tools and system are
part of every key because Anthropic renders them before the messages.
"""
def __init__(self) -> None:
self._written: set[str] = set()
def request(self, body: dict[str, Any]) -> tuple[int, int, int]:
messages = body.get("messages") or []
head = json.dumps(_strip([body.get("tools"), body.get("system")]), sort_keys=True)
keys = [
head + json.dumps(_strip(messages[: i + 1]), sort_keys=True)
for i in range(len(messages))
]
marks = [i for i, m in enumerate(messages) if _has_marker(m)]
if not marks:
return 0, 0, -1
hit = max((i for i in range(marks[-1] + 1) if keys[i] in self._written), default=-1)
self._written.update(keys[i] for i in marks)
# The tracker's own estimate: usage it can map back onto exactly the
# messages the provider cached, so only the fork can move the floor.
tokens = PrefixCacheTracker._estimate_message_tokens(messages)
system_tokens = len(json.dumps(body.get("system"))) // 4
read = system_tokens + sum(tokens[: hit + 1])
write = sum(tokens[hit + 1 : marks[-1] + 1])
return read, write, hit
def _drive(
requests: list[list[dict[str, Any]]], cache: _PromptCache | None = None, **config: Any
) -> list[tuple[list[dict[str, Any]], int]]:
"""Send each history through the real handler; return (forwarded, read_hit).
One app per call: a second call with the same ``cache`` is a proxy restart
in front of a provider whose prompt cache stayed warm."""
app = create_app(
ProxyConfig(
optimize=True,
mode="token",
# Deterministic structural compression only: the Kompress model
# is not available everywhere the suite runs.
disable_kompress=True,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
**config,
)
)
cache = cache or _PromptCache()
seen: list[tuple[list[dict[str, Any]], int]] = []
with TestClient(app) as client:
proxy = client.app.state.proxy
async def _upstream(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001, ANN003
read, write, hit = cache.request(body)
seen.append((copy.deepcopy(body.get("messages") or []), hit))
return httpx.Response(
200,
json={
"id": "msg_x",
"type": "message",
"role": "assistant",
"model": MODEL,
"content": [{"type": "text", "text": "ok"}],
"stop_reason": "end_turn",
"usage": {
"input_tokens": 10,
"output_tokens": 2,
"cache_read_input_tokens": read,
"cache_creation_input_tokens": write,
},
},
)
proxy._retry_request = _upstream
for messages in requests:
response = client.post(
"/v1/messages",
headers={
"x-api-key": "test-key",
"anthropic-version": "2023-06-01",
"content-type": "application/json",
},
json={
"model": MODEL,
"max_tokens": 64,
"system": SYSTEM,
"tools": [BASH_TOOL],
"messages": messages,
},
)
assert response.status_code == 200, response.text[:300]
assert len(seen) == len(requests)
return seen
@pytest.mark.parametrize("fork", [_summary_fork, _side_query_fork], ids=["summary", "side-query"])
def test_main_turn_after_a_fork_replays_its_cached_prefix(monkeypatch, fork) -> None: # noqa: ANN001
from headroom.cache.compression_store import reset_compression_store
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "memory")
reset_compression_store()
turns = range(1, 6)
requests: list[list[dict[str, Any]]] = []
for n in turns:
requests += [_main(n), fork(n)]
seen = _drive(requests)
mains = seen[0::2]
# Not vacuous: the pipeline did rewrite tool results somewhere.
assert any(
_strip(forwarded) != _strip(original) for (forwarded, _), original in zip(seen, requests)
), "nothing was compressed; the test cannot observe a prefix change"
for (previous, _), (current, hit), n in zip(mains, mains[1:], turns):
assert _strip(current[: len(previous)]) == _strip(previous), (
f"main turn {n + 1} re-sent turn {n}'s history with different bytes"
)
assert hit >= len(previous) - 1, (
f"main turn {n + 1} read only {hit + 1}/{len(previous)} cached messages"
)
def _record(tracker, messages, *, cached: int) -> None: # noqa: ANN001
tracker.update_from_response(
cache_read_tokens=cached,
cache_write_tokens=0,
messages=messages,
original_messages=messages,
message_token_counts=[1000] * len(messages),
)
@pytest.mark.parametrize("fork_responds_first", [True, False], ids=["fork-first", "main-first"])
def test_fork_does_not_replace_the_lineage_it_branched_from(fork_responds_first) -> None: # noqa: ANN001
"""Store level, both orderings seen in production: the summary fork is sent
while a slow main turn is still in flight (fork answers first), or after a
fast one answered (main answers first). Either way the next main turn must
resume the main turn's state, not the fork's and not a cold tracker.
Each request runs in its own context, as each proxied request runs in its
own asyncio task."""
store = SessionTrackerStore()
main_k, fork_k, main_next = _strip(_main(3)), _strip(_summary_fork(3)), _strip(_main(4))
main_ctx, fork_ctx = contextvars.Context(), contextvars.Context()
main_tracker = main_ctx.run(store.resolve_tracker, "sid", "anthropic", messages=main_k)
if not fork_responds_first:
main_ctx.run(_record, main_tracker, main_k, cached=1000 * len(main_k))
fork_tracker = fork_ctx.run(store.resolve_tracker, "sid", "anthropic", messages=fork_k)
fork_ctx.run(_record, fork_tracker, fork_k, cached=1000 * (len(fork_k) - 1))
if fork_responds_first:
main_ctx.run(_record, main_tracker, main_k, cached=1000 * len(main_k))
resumed = contextvars.Context().run(
store.resolve_tracker, "sid", "anthropic", messages=main_next
)
assert resumed is main_tracker
assert resumed.get_frozen_message_count() == len(main_k)
assert resumed.get_last_original_messages() == main_k
@pytest.mark.parametrize("forks", [1, 3, 5])
def test_successive_forks_keep_the_turn_they_branch_from(forks) -> None: # noqa: ANN001
"""Several forks off one main turn (progress summaries, side queries) must
not push that turn out of the lineage history: the next main turn still
extends it, so it must resume its frozen prefix rather than start cold."""
store = SessionTrackerStore()
base = _strip(_main(3))
main_tracker = contextvars.Context().run(_send, store, base)
for i in range(forks):
side = base + [
{"role": "assistant", "content": [{"type": "text", "text": f"Done {i}."}]},
{"role": "user", "content": [{"type": "text", "text": f"Side query {i}."}]},
]
contextvars.Context().run(_send, store, side)
main_next = _strip(_main(4))
resumed = contextvars.Context().run(
store.resolve_tracker, "sid", "anthropic", messages=main_next
)
assert resumed is main_tracker
assert resumed.get_frozen_message_count() == len(base)
assert resumed.get_last_original_messages() == base
def _send(store: SessionTrackerStore, messages: list[dict[str, Any]]) -> PrefixCacheTracker:
tracker = store.resolve_tracker("sid", "anthropic", messages=messages)
_record(tracker, messages, cached=1000 * len(messages))
return tracker
@pytest.mark.parametrize("fork_responds_first", [True, False], ids=["fork-first", "main-first"])
def test_cache_miss_is_classified_against_the_resumed_turn(fork_responds_first) -> None: # noqa: ANN001
"""A sibling request answering while this one is in flight must not change
what this request's cache outcome is measured against."""
store = SessionTrackerStore()
base = _strip(_main(3))
contextvars.Context().run(_send, store, base)
main_next, fork = _strip(_main(4)), _strip(_summary_fork(3))
main_ctx, fork_ctx = contextvars.Context(), contextvars.Context()
main_tracker = main_ctx.run(store.resolve_tracker, "sid", "anthropic", messages=main_next)
fork_tracker = fork_ctx.run(store.resolve_tracker, "sid", "anthropic", messages=fork)
assert fork_tracker is main_tracker
if fork_responds_first:
# The fork read nothing: the shared tracker now expects no cache.
fork_ctx.run(_record, fork_tracker, fork, cached=0)
# The main turn misses on a prefix its parent cached.
attribution = main_ctx.run(main_tracker.classify_cache_miss, 0, main_next, 0.0)
assert attribution.is_miss
assert attribution.expected_cached_tokens == 1000 * len(base)
assert not attribution.prefix_changed
if not fork_responds_first:
fork_ctx.run(_record, fork_tracker, fork, cached=0)