mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description Every proxy restart rewrites the prompt cache of every warm conversation. After a restart the proxy has no `PrefixCacheTracker` for the conversation, so it compresses the whole history afresh, the early messages come out as different bytes than the prefix it forwarded before the restart, and the provider re-writes the cache from the first changed message. Measured on 2026-10-19 on one Mac running Claude Code through the proxy: after each graceful restart (app relaunches for updates: proxy SIGTERM'd and respawned), the next request of every active session read only ~25k cached tokens (tools + system) and re-wrote the rest, up to 247k, 310k, 361k and 432k tokens at a time. The restart-aligned rewrites came to about 4.6M tokens of 1h cache writes that day (2x the input price), about a quarter of all cache writes. A client-side integrity probe reported `kind=tracker_reset, stable_messages=861, first_changed_message=3`. This adds an opt-in `HEADROOM_PREFIX_STATE_FILE=<path>`. When set, the proxy saves the `SessionTrackerStore` on graceful shutdown and restores it at startup, before serving, so the first request after a restart replays byte-identical frozen-prefix bytes. Unset, nothing changes. The file holds conversation content, so it is never a default. Related but different: #4156 (away recap replaces the lineage), #4287 (CCR proactive expansion rewrites), #4204 (deferred tool loads in the lineage key), #4284 (mid-history divergence starts a cold lineage). No existing issue; found while auditing one machine's token usage. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `SessionTrackerStore.save_state` / `load_state` (`headroom/cache/prefix_tracker.py`). Per tracker: turn number, cached token and message counts, last original and last forwarded messages, the per-turn states that lineage resolution resumes, provider, last activity and observed cache TTL; per lineage: session id, cache affinity and the recent snapshot history. So `resolve_tracker` matches the next request to the restored tracker, forks included. Each distinct message is stored once per tracker (successive turns, snapshots and the original/forwarded pairs mostly repeat), and new lineage keys and turn ids continue past the restored ones. - CCR entries ride along (review: retrieval integrity). Per tracker, `save_state` collects every CCR reference in its saved messages and read-maturation state (`hash=<hex>` markers and SmartCrusher's `<<ccr:<hex>`) and writes the live `CompressionStore` entries behind them (new `CompressionStore.export_entries`, which reads without counting a retrieval). A tracker and its entries count against the size cap together, so a tracker is never saved without them. `load_state` re-stores them (`import_entries`) with their original expiry, before any tracker is installed, so a replayed marker resolves from the first request even with `HEADROOM_CCR_BACKEND=memory`. An entry a persistent backend still holds is left as it is; a malformed one rejects the file. - Restore keeps only trackers still inside the store's own retention (last activity plus the observed or configured cache TTL and its cleanup margin, at least the session TTL), the same rule the live store uses. NaN or future timestamps are dropped. - Versioned schema (`version: 1`). Any read, parse or validation error logs one warning and leaves the store empty; the whole file is validated before anything is installed. The file is written atomically (`headroom.fsutil.write_text`: temp file, fsync, rename) at mode 0600: an existing file is tightened to 0600 in place, never unlinked first, so a failed write (ENOSPC) keeps the previous snapshot. Deleted after a successful load. Stateless mode (`persistence_allowed`) neither writes nor reads it (a stateless start used to load, and so delete, a snapshot). Ignored with multiple workers (one file cannot hold N per-process stores). - Size cap: the 256 most recently active trackers and at most 64 MB, newest first; whatever does not fit starts cold, as everything did before. - `HeadroomProxy.startup()` loads, `HeadroomProxy.shutdown()` saves as its first step, before any await, so the shutdown step timeout cannot cut it short. No periodic writes. - Token-mode `CompressionCache` seed: a restored tracker has no `CompressionCache` behind it in the new process, so the freeze computed from that cache froze nothing: the pipeline recompressed messages the provider cached, read maturation rewrote Reads inside the cached prefix, and beyond the provider-confirmed floor the overlay declined a larger previous form. One shared `seed_restored_compression_cache` seeds the session's cache from the restored original/forwarded pair (`update_from_result`) on the first request, on all three token-mode paths: Anthropic `/v1/messages`, OpenAI `/v1/chat/completions`, and the session-aware `/v1/compress` (sidecar). It gives the same freeze count as before the restart. - `ReadMaturationManager.export_state` / `validate_state` / `restore_state`: matured markers, booked ids and replay token deltas ride along with the tracker, so a matured Read's removal is not booked as savings again on every request after a restart. The saved state is validated at load with the rest of the file, so a bad entry rejects the file instead of failing a request. - `tests/test_anthropic_fork_lineage_cache.py`: `_drive` accepts a shared provider-cache model and config overrides, so a second call is a proxy restart in front of a warm provider cache. - Documented the variable in `docs/content/docs/configuration.mdx`. - `pyproject.toml` / `uv.lock` (review: Dependency audit): the `aiohttp` constraint floor is now `>=3.14.4` (CVE-2026-111063, CVE-2026-111064 and GHSA-2g87-pp6c-29x5, all fixed in 3.14.4), relocked with `uv lock --upgrade-package aiohttp==3.14.4`. Only aiohttp (3.14.3 -> 3.14.4) and yarl (1.22.0 -> 1.25.1, which 3.14.4 requires) move. Main still resolves 3.14.3, so the same commit is on #4299, #4286, #4285 and #4269. Remaining gaps (unchanged by this PR, none of them changes forwarded message bytes): - Session-sticky `anthropic-beta` union (`HEADROOM_BETA_HEADER_STICKY`): in memory, so after a restart a session's sticky set starts from the client's own header again. Headers only. - Output shaper: stateless per request (byte-stable steering text, effort from the current request's structure), so nothing to persist. - Gemini and Bedrock paths resume the restored tracker and its overlay replay; they keep no token-mode `CompressionCache`, so there is nothing to seed. A crash or SIGKILL still loses everything, by design (no periodic writes). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # New regression tests for the review findings, run against thec141e2ecsource # (headroom/ stashed, new tests kept): all six fail, the 15 earlier tests pass. $ uv run --frozen --extra dev pytest tests/test_prefix_tracker_restart.py -q FAILED tests/test_prefix_tracker_restart.py::test_restored_retrieval_references_resolve_to_their_originals FAILED tests/test_prefix_tracker_restart.py::test_a_stateless_proxy_leaves_a_saved_snapshot_alone FAILED tests/test_prefix_tracker_restart.py::test_a_failed_save_keeps_the_previous_snapshot FAILED tests/test_prefix_tracker_restart.py::test_saved_ccr_entries_resolve_after_a_restart FAILED tests/test_prefix_tracker_restart.py::test_openai_token_mode_replays_its_prefix_after_a_restart FAILED tests/test_prefix_tracker_restart.py::test_compress_sidecar_session_replays_its_prefix_after_a_restart 6 failed, 15 passed in 8.10s # Why, in order: hash=cc146b21626392f0dee9890c no longer resolves; the stateless start # deleted the snapshot; ENOSPC left no snapshot; retrieve() returns None; the OpenAI chat # and /v1/compress paths re-compressed message 2 under the new query. # Headc379245e: $ uv run --frozen --extra dev pytest tests/test_prefix_tracker_restart.py -q 21 passed in 9.50s $ uv run --frozen --extra dev pytest tests/test_prefix_tracker_restart.py tests/test_anthropic_fork_lineage_cache.py \ tests/test_cache/test_prefix_tracker.py tests/test_compression_store.py tests/test_ccr_sqlite_backend.py \ tests/test_issue_2671_block_growth_cache.py tests/test_cache_mode_cold_start.py tests/test_cache_prefix_overlay.py \ tests/test_cross_turn_cache_safety.py tests/test_session_engine.py tests/test_read_maturation.py \ tests/test_read_maturation_handler_nobust.py tests/test_graceful_shutdown.py tests/test_proxy_handlers_batch.py \ tests/test_cache_control_move_bust.py tests/test_startup_log_noise.py tests/test_proxy_openai_cache_stability.py \ tests/test_compress_session_mode.py tests/test_proxy_anthropic_cache_stability.py tests/test_ccr_row_drop_store_bridge.py \ tests/test_bedrock_prefix_tracker_wiring.py -q 486 passed, 1 skipped in 46.92s $ uv run --frozen --extra dev pytest tests/test_anthropic_*.py tests/test_proxy_anthropic_*.py tests/test_proxy_openai*.py \ tests/test_openai_chat_*.py tests/test_cache_aligner_prefix_stability.py tests/test_cache_prefix_safety.py tests/test_cold_prefix.py \ tests/test_gemini_prefix_freeze.py tests/test_prefix_tracker_client_rewrote_reply.py tests/test_tool_injection*.py tests/test_ccr_*.py -q 737 passed, 16 skipped in 51.45s $ uvx ruff@0.17.0 check headroom tests All checks passed! $ uvx ruff@0.17.0 format --check headroom tests 1606 files already formatted $ uv run --frozen --extra dev mypy headroom/cache/prefix_tracker.py headroom/cache/compression_store.py \ headroom/proxy/handlers/anthropic.py headroom/proxy/handlers/openai.py headroom/proxy/server.py Success: no issues found in 5 source files ``` ## Real Behavior Proof - Environment: macOS 27 (arm64), this branch atc379245e, the real proxy process (`python -m headroom.cli proxy --port 6811 --no-http2 --no-rate-limit --mode token --read-maturation --read-maturation-quiesce-turns 1 --read-maturation-min-size-bytes 200 --disable-kompress --no-subscription-tracking`, `HEADROOM_ROLLOUT_CHANNEL=beta`, clean `HOME` and workspace, `HEADROOM_PREFIX_STATE_FILE` set, `HEADROOM_CCR_BACKEND=memory`, `HEADROOM_BACKGROUND_COMPRESSION=1`) in front of the live Anthropic API, driven by Claude Code 2.1.303 (`claude -p --model haiku`, `ANTHROPIC_BASE_URL=http://127.0.0.1:6811`). Each restart is a SIGTERM (graceful shutdown) and a fresh process. - Exact command / steps: one Claude Code session through the proxy, three graceful restarts, the last one without the state file as the control: 1. Turn 1: `claude -p` reads three files (4.8, 17 and 8.9 KB) one per turn, 5 requests through the proxy. SIGTERM: `SessionTrackerStore: saved 2 of 2 prefix trackers (181266 bytes)`; file mode `-rw-------`; it holds 2 CCR entries, for the two Reads that read maturation had replaced with `Retrieve original: hash=` markers. 2. Restart: `restored 2 of 2 prefix trackers`, file deleted. `GET /v1/retrieve/3a739d6ec5febcd49d0cfc0e` and `/v1/retrieve/2ee4d4d435432b4cca9b72af` on the new process: HTTP 200, `original_content` equal to the pre-restart entries (5,129 and 18,525 chars, the Read outputs of the two files). With the in-process CCR store,c141e2ecloses such entries at a restart (the reviewer's reproduction; `test_restored_retrieval_references_resolve_to_their_originals` fails on it). 3. `claude -p --resume`, then SIGTERM + restart with the state file again, then `claude -p --resume` again (one request each, no tools). 4. Control: SIGTERM, snapshot moved aside, restart without `HEADROOM_PREFIX_STATE_FILE`, `claude -p --resume`. - Observed result: after a restart with the state file the first request read 99% of its prefix from the provider cache and both restored retrieval hashes resolved; the control re-wrote everything after the system prompt (proxy `PERF` lines, which match Claude Code's reported usage): | First request after a restart | cache_read | cache_write | hit | |---|---|---|---| | With the state file, resume after a resume | 40,406 | 514 | 99% | | Without the state file (control), same history | 26,053 | 11,250 | 70% | With the state file the proxy forwarded messages 0-10 byte-identical to what it forwarded before the restart (compared from the saved snapshots, `cache_control` positions excluded since Claude Code moves its breakpoint every turn), and froze 14 of 15 messages. In the control, the forwarded bytes changed from message 1, and the third Read, verbatim inside the cached prefix, matured into a new marker (10,205 -> 368 chars), so everything after the system prompt was re-written. - Not tested: Kompress itself (`--disable-kompress`; the unit suite uses a deterministic stand-in). Windows file modes. The OpenAI chat and `/v1/compress` seeds against a live OpenAI endpoint (covered by the handler-level restart tests above with a query-dependent compressor). The very first `--resume` of a `claude -p` session (step 3's first request) read only 26,053 tokens and wrote 14,845, although the client's messages and the proxy's forwarded messages 0-10 were byte-identical to the pre-restart request; the read stopped at the system-prompt breakpoint, so the change is most likely in the system/tools part of the request, which I did not capture. The resume after it, across another restart, read 99%. ## Runtime Rollout Safety - Rollout-managed feature(s): None; opt-in environment variable `HEADROOM_PREFIX_STATE_FILE`, unset by default. - Minimum rollout channel: Stable. - Stable/default behavior changed: No; with the variable unset nothing is read or written and no code path changes (the new handler branches only fire on a tracker restored from the file, and the CCR export/import only runs inside save/load). - Kill switch / disable path: Unset `HEADROOM_PREFIX_STATE_FILE`. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Unset the variable or revert this PR; a leftover state file is inert without the variable. The file format is still `version: 1` (now with a `ccr` list); a file written byc141e2echas no `ccr` list and is rejected at load, which starts cold. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable. ## Additional Notes - Review follow-up (c379245e): CCR entries are saved with their trackers (P1), the seed covers the OpenAI chat and `/v1/compress` token paths, a failed save keeps the previous snapshot, and stateless mode no longer loads (and deletes) a snapshot. Each has a regression test that fails onc141e2ec. - Caller isolation: the restored store has exactly the keys, lineages and retention the in-memory store had at shutdown, and replay still requires the caller to send the same original messages; see the reply on that thread for the full reasoning. - The full `pytest tests` run was not repeated for this head (shared machine); the focused runs above cover every module the change touches. - Size cap choice: 256 trackers covers many concurrent Claude Code sessions plus their subagents; 64 MB bounds shutdown time (encoding is synchronous) and disk use. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4299" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
350 lines
14 KiB
Python
350 lines
14 KiB
Python
"""A Claude Code side request must not cost the main conversation its prompt cache.
|
|
|
|
Production, 2026-10-08: a W.A.L.T.E.R. review subagent busted Anthropic's
|
|
prompt cache on 35 of its 81 turns (``cache_read_input_tokens`` fell to the
|
|
6,065-token system prompt while ~100-150k tokens were re-written), and its
|
|
600k-token parent session did the same after its own side requests. Every
|
|
bust followed a *fork*: Claude Code re-sends the agent's history plus one
|
|
extra prompt (the subagent progress summary merges it into the last user
|
|
message; the parent's side query appends a reply and a new user message) with
|
|
its breakpoint one message earlier, so it reads the cache without writing.
|
|
|
|
The fork extended the conversation's prefix-tracker lineage and replaced its
|
|
snapshot. The next main turn extends the history the fork branched from, not
|
|
the fork, so it matched no lineage, started a cold tracker (frozen=0) and the
|
|
token-mode pipeline recompressed tool results it had forwarded verbatim the
|
|
turn before. Different bytes, so the provider's prefix cache missed from the
|
|
first changed message onward.
|
|
|
|
The upstream below models Anthropic's prompt cache (a prefix is readable once
|
|
a breakpoint wrote it), so the assertions are on what the provider would do
|
|
with the bytes the handler forwards.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import contextvars
|
|
import copy
|
|
import json
|
|
from typing import Any
|
|
|
|
import pytest
|
|
|
|
pytest.importorskip("fastapi")
|
|
|
|
import httpx
|
|
from fastapi.testclient import TestClient
|
|
|
|
from headroom.cache.prefix_tracker import PrefixCacheTracker, SessionTrackerStore
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
|
|
MODEL = "claude-sonnet-4-5-20250929"
|
|
SYSTEM = [
|
|
{
|
|
"type": "text",
|
|
"text": "You review pull requests. Report only real defects. " * 40,
|
|
"cache_control": {"type": "ephemeral"},
|
|
}
|
|
]
|
|
BASH_TOOL = {
|
|
"name": "Bash",
|
|
"description": "Run a shell command",
|
|
"input_schema": {
|
|
"type": "object",
|
|
"properties": {"command": {"type": "string"}},
|
|
"required": ["command"],
|
|
},
|
|
}
|
|
MARKER = {"type": "ephemeral"}
|
|
|
|
|
|
def _bash_turn(i: int) -> list[dict[str, Any]]:
|
|
tid = f"toolu_{i:02d}"
|
|
rows = [
|
|
{"path": f"src/module_{i}/file_{j}.py", "size": 1000 + j, "status": "ok"}
|
|
for j in range(400)
|
|
]
|
|
return [
|
|
{
|
|
"role": "assistant",
|
|
"content": [
|
|
{"type": "tool_use", "id": tid, "name": "Bash", "input": {"command": f"ls {i}"}}
|
|
],
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": [{"type": "tool_result", "tool_use_id": tid, "content": json.dumps(rows)}],
|
|
},
|
|
]
|
|
|
|
|
|
def _main(turns: int) -> list[dict[str, Any]]:
|
|
"""The agent's own request: Claude Code's breakpoint rides the newest block."""
|
|
messages: list[dict[str, Any]] = [
|
|
{"role": "user", "content": [{"type": "text", "text": "Review the change."}]}
|
|
]
|
|
for i in range(turns):
|
|
messages += _bash_turn(i)
|
|
messages[-1]["content"][-1]["cache_control"] = dict(MARKER)
|
|
return messages
|
|
|
|
|
|
def _summary_fork(turns: int) -> list[dict[str, Any]]:
|
|
"""Subagent progress summary: same history, the prompt merged into the last
|
|
user message, breakpoint one message earlier (reads, never writes)."""
|
|
messages = _main(turns)
|
|
del messages[-1]["content"][-1]["cache_control"]
|
|
messages[-1]["content"].append({"type": "text", "text": "Summarize your progress in 5 words."})
|
|
messages[-2]["content"][-1]["cache_control"] = dict(MARKER)
|
|
return messages
|
|
|
|
|
|
def _side_query_fork(turns: int) -> list[dict[str, Any]]:
|
|
"""Parent-session side query: history + the agent's reply + a new prompt."""
|
|
messages = _main(turns)
|
|
messages.append({"role": "assistant", "content": [{"type": "text", "text": "Done."}]})
|
|
messages.append({"role": "user", "content": [{"type": "text", "text": "Name this session."}]})
|
|
del messages[-3]["content"][-1]["cache_control"]
|
|
messages[-2]["content"][-1]["cache_control"] = dict(MARKER)
|
|
return messages
|
|
|
|
|
|
def _strip(obj: Any) -> Any:
|
|
if isinstance(obj, dict):
|
|
return {k: _strip(v) for k, v in obj.items() if k != "cache_control"}
|
|
if isinstance(obj, list):
|
|
return [_strip(v) for v in obj]
|
|
return obj
|
|
|
|
|
|
def _has_marker(message: dict[str, Any]) -> bool:
|
|
content = message.get("content")
|
|
return isinstance(content, list) and any(
|
|
isinstance(block, dict) and "cache_control" in block for block in content
|
|
)
|
|
|
|
|
|
class _PromptCache:
|
|
"""Anthropic's prompt cache at message granularity.
|
|
|
|
A request reads the longest prefix an earlier breakpoint wrote, then
|
|
writes a cache entry at each of its own breakpoints. Tools and system are
|
|
part of every key because Anthropic renders them before the messages.
|
|
"""
|
|
|
|
def __init__(self) -> None:
|
|
self._written: set[str] = set()
|
|
|
|
def request(self, body: dict[str, Any]) -> tuple[int, int, int]:
|
|
messages = body.get("messages") or []
|
|
head = json.dumps(_strip([body.get("tools"), body.get("system")]), sort_keys=True)
|
|
keys = [
|
|
head + json.dumps(_strip(messages[: i + 1]), sort_keys=True)
|
|
for i in range(len(messages))
|
|
]
|
|
marks = [i for i, m in enumerate(messages) if _has_marker(m)]
|
|
if not marks:
|
|
return 0, 0, -1
|
|
hit = max((i for i in range(marks[-1] + 1) if keys[i] in self._written), default=-1)
|
|
self._written.update(keys[i] for i in marks)
|
|
# The tracker's own estimate: usage it can map back onto exactly the
|
|
# messages the provider cached, so only the fork can move the floor.
|
|
tokens = PrefixCacheTracker._estimate_message_tokens(messages)
|
|
system_tokens = len(json.dumps(body.get("system"))) // 4
|
|
read = system_tokens + sum(tokens[: hit + 1])
|
|
write = sum(tokens[hit + 1 : marks[-1] + 1])
|
|
return read, write, hit
|
|
|
|
|
|
def _drive(
|
|
requests: list[list[dict[str, Any]]], cache: _PromptCache | None = None, **config: Any
|
|
) -> list[tuple[list[dict[str, Any]], int]]:
|
|
"""Send each history through the real handler; return (forwarded, read_hit).
|
|
|
|
One app per call: a second call with the same ``cache`` is a proxy restart
|
|
in front of a provider whose prompt cache stayed warm."""
|
|
app = create_app(
|
|
ProxyConfig(
|
|
optimize=True,
|
|
mode="token",
|
|
# Deterministic structural compression only: the Kompress model
|
|
# is not available everywhere the suite runs.
|
|
disable_kompress=True,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
**config,
|
|
)
|
|
)
|
|
cache = cache or _PromptCache()
|
|
seen: list[tuple[list[dict[str, Any]], int]] = []
|
|
with TestClient(app) as client:
|
|
proxy = client.app.state.proxy
|
|
|
|
async def _upstream(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001, ANN003
|
|
read, write, hit = cache.request(body)
|
|
seen.append((copy.deepcopy(body.get("messages") or []), hit))
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "msg_x",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"model": MODEL,
|
|
"content": [{"type": "text", "text": "ok"}],
|
|
"stop_reason": "end_turn",
|
|
"usage": {
|
|
"input_tokens": 10,
|
|
"output_tokens": 2,
|
|
"cache_read_input_tokens": read,
|
|
"cache_creation_input_tokens": write,
|
|
},
|
|
},
|
|
)
|
|
|
|
proxy._retry_request = _upstream
|
|
for messages in requests:
|
|
response = client.post(
|
|
"/v1/messages",
|
|
headers={
|
|
"x-api-key": "test-key",
|
|
"anthropic-version": "2023-06-01",
|
|
"content-type": "application/json",
|
|
},
|
|
json={
|
|
"model": MODEL,
|
|
"max_tokens": 64,
|
|
"system": SYSTEM,
|
|
"tools": [BASH_TOOL],
|
|
"messages": messages,
|
|
},
|
|
)
|
|
assert response.status_code == 200, response.text[:300]
|
|
assert len(seen) == len(requests)
|
|
return seen
|
|
|
|
|
|
@pytest.mark.parametrize("fork", [_summary_fork, _side_query_fork], ids=["summary", "side-query"])
|
|
def test_main_turn_after_a_fork_replays_its_cached_prefix(monkeypatch, fork) -> None: # noqa: ANN001
|
|
from headroom.cache.compression_store import reset_compression_store
|
|
|
|
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "memory")
|
|
reset_compression_store()
|
|
|
|
turns = range(1, 6)
|
|
requests: list[list[dict[str, Any]]] = []
|
|
for n in turns:
|
|
requests += [_main(n), fork(n)]
|
|
seen = _drive(requests)
|
|
mains = seen[0::2]
|
|
|
|
# Not vacuous: the pipeline did rewrite tool results somewhere.
|
|
assert any(
|
|
_strip(forwarded) != _strip(original) for (forwarded, _), original in zip(seen, requests)
|
|
), "nothing was compressed; the test cannot observe a prefix change"
|
|
|
|
for (previous, _), (current, hit), n in zip(mains, mains[1:], turns):
|
|
assert _strip(current[: len(previous)]) == _strip(previous), (
|
|
f"main turn {n + 1} re-sent turn {n}'s history with different bytes"
|
|
)
|
|
assert hit >= len(previous) - 1, (
|
|
f"main turn {n + 1} read only {hit + 1}/{len(previous)} cached messages"
|
|
)
|
|
|
|
|
|
def _record(tracker, messages, *, cached: int) -> None: # noqa: ANN001
|
|
tracker.update_from_response(
|
|
cache_read_tokens=cached,
|
|
cache_write_tokens=0,
|
|
messages=messages,
|
|
original_messages=messages,
|
|
message_token_counts=[1000] * len(messages),
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize("fork_responds_first", [True, False], ids=["fork-first", "main-first"])
|
|
def test_fork_does_not_replace_the_lineage_it_branched_from(fork_responds_first) -> None: # noqa: ANN001
|
|
"""Store level, both orderings seen in production: the summary fork is sent
|
|
while a slow main turn is still in flight (fork answers first), or after a
|
|
fast one answered (main answers first). Either way the next main turn must
|
|
resume the main turn's state, not the fork's and not a cold tracker.
|
|
|
|
Each request runs in its own context, as each proxied request runs in its
|
|
own asyncio task."""
|
|
store = SessionTrackerStore()
|
|
main_k, fork_k, main_next = _strip(_main(3)), _strip(_summary_fork(3)), _strip(_main(4))
|
|
main_ctx, fork_ctx = contextvars.Context(), contextvars.Context()
|
|
main_tracker = main_ctx.run(store.resolve_tracker, "sid", "anthropic", messages=main_k)
|
|
if not fork_responds_first:
|
|
main_ctx.run(_record, main_tracker, main_k, cached=1000 * len(main_k))
|
|
fork_tracker = fork_ctx.run(store.resolve_tracker, "sid", "anthropic", messages=fork_k)
|
|
fork_ctx.run(_record, fork_tracker, fork_k, cached=1000 * (len(fork_k) - 1))
|
|
if fork_responds_first:
|
|
main_ctx.run(_record, main_tracker, main_k, cached=1000 * len(main_k))
|
|
|
|
resumed = contextvars.Context().run(
|
|
store.resolve_tracker, "sid", "anthropic", messages=main_next
|
|
)
|
|
|
|
assert resumed is main_tracker
|
|
assert resumed.get_frozen_message_count() == len(main_k)
|
|
assert resumed.get_last_original_messages() == main_k
|
|
|
|
|
|
@pytest.mark.parametrize("forks", [1, 3, 5])
|
|
def test_successive_forks_keep_the_turn_they_branch_from(forks) -> None: # noqa: ANN001
|
|
"""Several forks off one main turn (progress summaries, side queries) must
|
|
not push that turn out of the lineage history: the next main turn still
|
|
extends it, so it must resume its frozen prefix rather than start cold."""
|
|
store = SessionTrackerStore()
|
|
base = _strip(_main(3))
|
|
main_tracker = contextvars.Context().run(_send, store, base)
|
|
for i in range(forks):
|
|
side = base + [
|
|
{"role": "assistant", "content": [{"type": "text", "text": f"Done {i}."}]},
|
|
{"role": "user", "content": [{"type": "text", "text": f"Side query {i}."}]},
|
|
]
|
|
contextvars.Context().run(_send, store, side)
|
|
|
|
main_next = _strip(_main(4))
|
|
resumed = contextvars.Context().run(
|
|
store.resolve_tracker, "sid", "anthropic", messages=main_next
|
|
)
|
|
|
|
assert resumed is main_tracker
|
|
assert resumed.get_frozen_message_count() == len(base)
|
|
assert resumed.get_last_original_messages() == base
|
|
|
|
|
|
def _send(store: SessionTrackerStore, messages: list[dict[str, Any]]) -> PrefixCacheTracker:
|
|
tracker = store.resolve_tracker("sid", "anthropic", messages=messages)
|
|
_record(tracker, messages, cached=1000 * len(messages))
|
|
return tracker
|
|
|
|
|
|
@pytest.mark.parametrize("fork_responds_first", [True, False], ids=["fork-first", "main-first"])
|
|
def test_cache_miss_is_classified_against_the_resumed_turn(fork_responds_first) -> None: # noqa: ANN001
|
|
"""A sibling request answering while this one is in flight must not change
|
|
what this request's cache outcome is measured against."""
|
|
store = SessionTrackerStore()
|
|
base = _strip(_main(3))
|
|
contextvars.Context().run(_send, store, base)
|
|
|
|
main_next, fork = _strip(_main(4)), _strip(_summary_fork(3))
|
|
main_ctx, fork_ctx = contextvars.Context(), contextvars.Context()
|
|
main_tracker = main_ctx.run(store.resolve_tracker, "sid", "anthropic", messages=main_next)
|
|
fork_tracker = fork_ctx.run(store.resolve_tracker, "sid", "anthropic", messages=fork)
|
|
assert fork_tracker is main_tracker
|
|
if fork_responds_first:
|
|
# The fork read nothing: the shared tracker now expects no cache.
|
|
fork_ctx.run(_record, fork_tracker, fork, cached=0)
|
|
|
|
# The main turn misses on a prefix its parent cached.
|
|
attribution = main_ctx.run(main_tracker.classify_cache_miss, 0, main_next, 0.0)
|
|
|
|
assert attribution.is_miss
|
|
assert attribution.expected_cached_tokens == 1000 * len(base)
|
|
assert not attribution.prefix_changed
|
|
if not fork_responds_first:
|
|
fork_ctx.run(_record, fork_tracker, fork, cached=0)
|