mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(cache): keep prefix lineage across a replaced system tail (#3934)
## Description Claude Code can end a `/v1/messages` request with a temporary `role:"system"` reminder and replace it on the next turn. `SessionTrackerStore.resolve_tracker` records the full history, so the stored chain ends in the old reminder and is never a prefix of the next request: `_classify_history_canonical` returns `diverged` and the request gets a fresh, empty tracker. The confirmed frozen prefix and the previously forwarded messages are gone, the #3380 replay has nothing to replay, and any recompressed history re-writes the provider prompt cache. The reminder moves every turn, so this repeats for the whole session. This change lets lineage resolution set aside that one replaced trailing system message, only when nothing matched as recorded. It changes tracker selection only: forwarded bytes, the replay body, and breakpoint placement are untouched, and the existing replay already stops at the first changed message, so the new reminder goes out as the client sent it. Closes #3933 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cache/prefix_tracker.py`: new `_extends_past_system_tail(current, previous)`. It is true when `previous` ends in a `role:"system"` message, everything before that message includes user/assistant history and is unchanged in `current`, and `current` is longer than `previous`. Because `current` must be longer, siblings that only swap their final instruction stay separate. - `SessionTrackerStore.resolve_tracker`: after the existing exact/append, block-append and block-rewrite passes find nothing (and there are no rewrite candidates, so an ambiguous rewrite tie is not overridden), Anthropic requests consider chains with the same cache affinity that pass that check. Only a unique longest candidate is taken; two equal-length candidates start a fresh lineage, as before. The full current snapshot is stored as usual. - `extract_cache_stable_delta` (cache mode's prefix/delta split): a `diverged` request that replaced the previous trailing system reminder is split too. Every recorded message before the reminder, plus the model reply the handlers record after it, must match the current request position for position (canonical compare, same rule via `_extends_past_system_tail`). The previously forwarded bytes are replayed without the old reminder, and everything after it is the delta to compress. Before this, cache mode tagged these turns `cache_mode_prefix_mismatch` and forwarded the new messages raw. - `HEADROOM_TRANSIENT_SYSTEM_LINEAGE=0` (also `false`/`no`/`off`) turns off both the lineage fallback and the delta split. It uses the same parsing as `HEADROOM_STABLE_BOUNDARY_BREAKPOINT`. - `tests/test_cache/test_prefix_tracker.py`: - Tracker retention: the tracker and its confirmed frozen count are kept, and the existing overlay restores the unchanged prefix. - A reminder that moves and changes over several turns: full snapshots are stored. - Isolation: different cache affinity, OpenAI provider, changed earlier history, changed leading system message, changed historical system message, same-length siblings `[user: task, system: A]` vs `[user: task, system: B]`, ambiguous equal-length siblings, exact-match precedence, and the kill switch. - The delta split: with and without the recorded reply, a system message followed by a client turn is not treated as a reminder, a mismatched reply is rejected, and the kill switch disables it. - `tests/test_cache_mode_cold_start.py`: drives `handle_anthropic_messages` in `mode="cache"` with the real `SessionTrackerStore` over three turns. The first turn compresses a tool result and later turns replace the reminder. The test asserts that the third request replays the second request's compressed prefix byte-for-byte (ignoring `cache_control`), compresses the new tool result, and ends in the current reminder. ## Testing - [x] Unit tests pass (`pytest`) - targeted files below, not the whole suite - [x] Linting passes (`ruff check .`) - on the changed files, plus `ruff format --check` - [x] Type checking passes (`mypy headroom`) - on `headroom/cache/prefix_tracker.py` - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output New tests with the source at `main` (d5318ac2): ```text $ python -m pytest tests/test_cache_mode_cold_start.py -q -p no:cacheprovider FAILED tests/test_cache_mode_cold_start.py::test_cache_mode_replaced_system_tail_keeps_compressed_prefix ========================= 1 failed, 2 passed in 1.70s ========================== ``` With the lineage commit only (2c5af818plus the length fix, before the delta split), the cache-mode tests fail on the raw new tool result: ```text E AssertionError: assert 'verbose tool...ool output 3 ' == 'compressed tool output' E TypeError: cannot unpack non-iterable NoneType object FAILED tests/test_cache_mode_cold_start.py::test_cache_mode_replaced_system_tail_keeps_compressed_prefix FAILED tests/test_cache/test_prefix_tracker.py::TestConversationLineageResolution::test_cache_delta_replays_prefix_past_replaced_system_tail ========================= 2 failed, 73 passed in 2.83s ========================= ``` The same-length sibling test fails at2c5af818: ```text FAILED tests/test_cache/test_prefix_tracker.py::TestConversationLineageResolution::test_swapped_final_instruction_is_not_a_continuation ========================= 1 failed, 70 passed in 0.75s ========================= ``` With all three commits: ```text $ python -m pytest tests/test_cache/test_prefix_tracker.py tests/test_cache_mode_cold_start.py \ tests/test_cache_mode_delta_marker.py tests/test_observed_wire_shapes.py tests/test_proxy_handler_helpers.py \ tests/test_issue_2671_block_growth_cache.py tests/test_proxy_anthropic_cache_stability.py \ tests/test_proxy_openai_cache_stability.py -q -p no:cacheprovider ======================= 225 passed, 1 warning in 46.30s ======================== $ ruff check headroom/cache/prefix_tracker.py tests/test_cache/test_prefix_tracker.py tests/test_cache_mode_cold_start.py All checks passed! $ ruff format --check headroom/cache/prefix_tracker.py tests/test_cache/test_prefix_tracker.py tests/test_cache_mode_cold_start.py 3 files already formatted $ mypy headroom/cache/prefix_tracker.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS 27.0.1, Python 3.11.13, this branch (5b0c20f2) vs `main`d5318ac2. The proxy was started from each checkout with `python -m headroom.proxy.server --port <free> --anthropic-api-url http://127.0.0.1:<stub> --no-rate-limit`, once with `HEADROOM_MODE=cache` (the default) and once with `HEADROOM_MODE=token`. The `headroom proxy` CLI refuses to start in this dev venv because the optional `mcp`, `magika` and `onnxruntime` extras are missing; the server module itself runs. The upstream is a local stdlib HTTP stub that records each forwarded body and answers with the next `tool_use`. It returns Anthropic-shaped `usage` from a simulated prompt cache: the whole forwarded prompt is cached; the next request reads the leading run of messages that are byte-identical to the previous forwarded request (ignoring `cache_control`) and writes the rest. Tokens are approximated as serialized chars / 3. No real provider was contacted. - Exact command / steps: `PROOF_MODE=cache|token proof.py <checkout> <python> <label>` (script below) sends four `claude-fable-5` turns of a paged tool loop. Each turn adds an assistant `tool_use` and a ~31 KB JSON `tool_result`, and each request ends in a `role:"system"` reminder whose text changes every turn. Controls: `-noreminder` sends the same turns without reminders, and `-mixed` drops only the first turn's reminder, which is how Claude Code interleaves them. - Observed result: in cache mode every reminder turn on the branch replays the previously forwarded history byte-identical and compresses only the new tool result (simulated read/write per turn: main 10,598/10,581, 21,162/10,581, 31,726/10,580 with nothing compressed; branch 10,598/3,106, 13,687/3,106, 16,776/3,105); in the mixed run main re-writes the cache on the first reminder turn (76 read / 31,667 write) and the branch reads 6,212 / writes 3,106; in token mode main re-writes the history every reminder turn (read 76/281/486) and the branch reads 3,123/6,212/9,301, matching the no-reminder control. Simulated cache read / write per turn, and the proxy's `PERF tok_after` (tokens forwarded): | Mode, run | Turn 1 | Turn 2 | Turn 3 | Turn 4 | | --- | --- | --- | --- | --- | | cache, main | 0 / 10,615 (10,886) | 10,598 / 10,581 (21,747) | 21,162 / 10,581 (32,608) | 31,726 / 10,580 (43,469) | | cache, branch before delta split (2c5af818) | 0 / 10,615 (10,886) | 10,598 / 10,581 (21,747) | 21,162 / 10,581 (32,608) | 31,726 / 10,580 (43,469) | | cache, branch | 0 / 10,615 (10,886) | 10,598 / 3,106 (15,122) | 13,687 / 3,106 (19,358) | 16,776 / 3,105 (23,594) | | cache, branch, `HEADROOM_TRANSIENT_SYSTEM_LINEAGE=0` | 0 / 10,615 (10,886) | 10,598 / 10,581 (21,747) | 21,162 / 10,581 (32,608) | 31,726 / 10,580 (43,469) | | cache, main, no reminders (control) | 0 / 3,123 (4,253) | 3,123 / 3,089 (8,489) | 6,212 / 3,089 (12,725) | 9,301 / 3,088 (16,961) | | cache mixed, main | 0 / 3,123 (4,253) | 3,123 / 3,106 (8,498) | 76 / 31,667 (32,608) | 31,726 / 10,580 (43,469) | | cache mixed, branch before delta split (2c5af818) | 0 / 3,123 (4,253) | 3,123 / 3,106 (8,498) | 6,212 / 10,581 (19,359) | 16,776 / 10,580 (30,220) | | cache mixed, branch | 0 / 3,123 (4,253) | 3,123 / 3,106 (8,498) | 6,212 / 3,106 (12,734) | 9,301 / 3,105 (16,970) | | token, main | 0 / 3,140 (4,262) | 76 / 3,271 (4,429) | 281 / 3,271 (4,595) | 486 / 3,270 (4,762) | | token, branch | 0 / 3,140 (4,262) | 3,123 / 3,106 (8,498) | 6,212 / 3,106 (12,734) | 9,301 / 3,105 (16,970) | | token, main, no reminders (control) | 0 / 3,123 (4,253) | 3,123 / 3,089 (8,489) | 6,212 / 3,089 (12,725) | 9,301 / 3,088 (16,961) | How to read it: - **Cache mode, main:** for a session that always ends in a reminder, the cache is stable but nothing is ever compressed. Each turn gets a fresh tracker, and the cold-start branch freezes the whole request because its last message is not a user turn. - **Cache mode, mixed run on main:** the first reminder turn after a compressed turn re-sends the compressed history raw and re-writes the cache. - **Lineage commit alone (2c5af818):** kept the history stable, because the cached-prefix replay in `finalize_turn` restores it. New tool results still went out raw (`cache_mode_prefix_mismatch`). - **With the delta split:** they are compressed (31,572 -> 9,187 bytes each), and the mixed run matches the no-reminder control. - **Every forwarded request on the branch** ends in that turn's own reminder text, and the old reminder is not replayed. - **Turn 1 of the always-reminder cache run** is forwarded raw on both main and the branch. A cold request whose last message is a system message is frozen whole by `_strict_previous_turn_frozen_count`. This PR does not change that. - Downstream production evidence (Headroom Desktop on the 0.37.0 wheel, `claude-fable-5`, token mode, provider-reported `usage`, no request content): one long Claude Code session went 0 read / 490,996 write (cold), then 490,996 read / 12,509 write on an 862-message request ending in the reminder, then 7,259 read / 491,260 write on the next 864-message request, although its first 861 messages were unchanged. 27 such continuations in that window wrote 15,183,730 cache tokens and read 195,993. After the lineage fix was applied downstream, a live run of 13 requests (9 replaced the reminder) read 664,386 and wrote 49,646 cache tokens (92.99% reuse). - Not tested: no real Anthropic request or live provider cache billing for this branch; details below. - No real Anthropic request was made from this branch, so live provider cache billing was not measured for this PR. The cache numbers above come from the stub's simulation. - The downstream numbers come from a vendored lineage patch on the 0.37.0 wheel in token mode. That patch has the same selection rules but not this exact code, and it has no cache-mode delta split, which is new here and has not run against a live provider. - Not exercised end to end: - streaming responses (they record the reply the same way, but were not driven here) - the Bedrock backend path - the `headroom proxy` CLI entry point with all extras installed - models that relocate the reminder into the top-level `system` field. There the reminder is part of the system segment and this change has no effect. - OpenAI paths are unchanged by design and are covered only by their existing tests. ## Runtime Rollout Safety - Rollout-managed feature(s): none. This is a fix in lineage selection and in cache mode's prefix/delta split, not a rollout feature in `headroom/rollout.py`. - Minimum rollout channel: n/a (applies on every channel). - Stable/default behavior changed: yes, for Anthropic requests whose previous request ended in a `role:"system"` message that the client replaced: - **Token mode:** they keep their existing prefix tracker, so the confirmed prefix is replayed byte-stable instead of being recompressed. That history is therefore no longer recompressed on those turns. - **Cache mode (default):** the same request is split into the previously forwarded prefix plus a compressed delta instead of being forwarded with its new messages raw. Compression on those turns goes up, cache reads stay at the replayed prefix, and a session that alternates reminder and non-reminder turns no longer re-writes its cache on the first reminder turn. - **Unchanged:** OpenAI, conversations without a trailing system message, siblings that only swap their final instruction, and every case where an existing relation already matches. - Kill switch / disable path: `HEADROOM_TRANSIENT_SYSTEM_LINEAGE=0` (or `false`/`no`/`off`), read per request. It disables both the lineage fallback and the cache-mode split. In the proof, the cache-mode run with it set matches `main` exactly. - Unsafe override required: no. - Qualification impact: expect higher provider cache reads and lower cache writes on Claude Code sessions that send trailing system reminders in token mode, and more compression of new messages on those turns in cache mode. This was measured here only against a stub; live provider billing for this branch was not measured. - Rollback path: set `HEADROOM_TRANSIENT_SYSTEM_LINEAGE=0` and restart the proxy, or revert the three commits. No persisted state or schema changes. <details> <summary>proof.py</summary> ```python """Boot the proxy (`python -m headroom.proxy.server`) from a given checkout against a local stub Anthropic upstream and replay a Claude Code shaped conversation (paged tool calls, ~31 KB JSON tool results) whose trailing role:"system" reminder is replaced every turn. Usage: PROOF_MODE=cache|token proof.py <checkout-dir> <python> <label> Label suffix "-noreminder" drops the reminders (control); "-mixed" drops only the first one. Reports, per turn, how many previously forwarded history messages were forwarded again byte-identical (cache_control ignored), the newest tool result's size before/after the proxy, and the stub's simulated prompt-cache read/write. No real provider is contacted: the upstream is 127.0.0.1. """ import json import os import socket import subprocess import sys import threading import time import urllib.request from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer checkout, python, label = sys.argv[1], sys.argv[2], sys.argv[3] received: list[dict] = [] usage: list[tuple[int, int]] = [] def strip(x): """Drop cache_control markers: the proxy re-places them every turn and the provider does not hash them into the cached prefix.""" if isinstance(x, dict): return {k: strip(v) for k, v in x.items() if k != "cache_control"} if isinstance(x, list): return [strip(v) for v in x] return x class Stub(BaseHTTPRequestHandler): def do_POST(self): body = json.loads(self.rfile.read(int(self.headers["content-length"]))) received.append(body) # Minimal provider prompt-cache simulation: the whole forwarded prompt # is cached (client-style breakpoint on the newest message); this turn # reads the leading run of messages byte-identical to the previous # forwarded request, and writes the rest. Tokens ~= serialized chars / 3. def toks(x): return len(json.dumps(strip(x), sort_keys=True)) // 3 msgs = body["messages"] total = toks(body.get("system")) + sum(toks(m) for m in msgs) read = 0 if len(received) > 1 and strip(received[-2].get("system")) == strip(body.get("system")): prev = received[-2]["messages"] k = 0 while k < min(len(prev), len(msgs)) and strip(prev[k]) == strip(msgs[k]): k += 1 if k: read = toks(body.get("system")) + sum(toks(m) for m in msgs[:k]) usage.append((read, total - read)) payload = json.dumps( { "id": f"msg_{len(received)}", "type": "message", "role": "assistant", "model": body["model"], "content": assistant(len(received) + 1)["content"], "stop_reason": "tool_use", "stop_sequence": None, "usage": { "input_tokens": 10, "output_tokens": 2, "cache_creation_input_tokens": total - read, "cache_read_input_tokens": read, }, } ).encode() self.send_response(200) self.send_header("content-type", "application/json") self.send_header("content-length", str(len(payload))) self.end_headers() self.wfile.write(payload) def log_message(self, *args): pass def assistant(k: int) -> dict: return {"role": "assistant", "content": [ {"type": "tool_use", "id": f"toolu_{k}", "name": "list_services", "input": {"page": k}}]} def tool_result(k: int) -> dict: rows = [{"id": i, "name": f"service-{k}-{i}", "status": "ok" if (i + k) % 7 else "degraded", "latency_ms": 20 + (i * 37 + k) % 400, "region": ["eu", "us", "ap"][i % 3]} for i in range(300)] return {"role": "user", "content": [ {"type": "tool_result", "tool_use_id": f"toolu_{k}", "content": json.dumps(rows)}]} def free_port() -> int: with socket.socket() as s: s.bind(("127.0.0.1", 0)) return s.getsockname()[1] stub = ThreadingHTTPServer(("127.0.0.1", free_port()), Stub) threading.Thread(target=stub.serve_forever, daemon=True).start() port = free_port() env = { **os.environ, "PYTHONPATH": checkout, "HEADROOM_BEACON": "off", "HEADROOM_MODE": os.environ.get("PROOF_MODE", "token"), "LITELLM_LOCAL_MODEL_COST_MAP": "True", "HEADROOM_DATA_DIR": os.path.join(os.path.dirname(__file__), f"data-{label}"), } env.pop("GROK_CLI_CHAT_PROXY_BASE_URL", None) log = open(os.path.join(os.path.dirname(__file__), f"proxy-{label}.log"), "w") proxy = subprocess.Popen( [ python, "-m", "headroom.proxy.server", "--port", str(port), "--anthropic-api-url", f"http://127.0.0.1:{stub.server_address[1]}", "--no-rate-limit", ], env=env, stdout=log, stderr=subprocess.STDOUT, ) try: for _ in range(240): try: urllib.request.urlopen(f"http://127.0.0.1:{port}/health", timeout=1) break except Exception: time.sleep(0.5) else: raise SystemExit("proxy did not come up") turns = [] convo = [{"role": "user", "content": "Page through the services and find the degraded ones."}] for n in range(1, 5): convo = convo + [assistant(n), tool_result(n)] turns.append(convo + [{"role": "system", "content": f"Reminder variant {n}."}]) if label.endswith("noreminder"): # Control: the same conversation without the trailing system reminder. turns = [t[:-1] for t in turns] elif label.endswith("mixed"): # Claude Code sends reminders on some turns only: the first request # has none (so it is compressed and that form is cached), later ones do. turns[0] = turns[0][:-1] previous = None for n, messages in enumerate(turns, 1): req = urllib.request.Request( f"http://127.0.0.1:{port}/v1/messages", data=json.dumps({"model": "claude-fable-5", "max_tokens": 64, "system": "You are a coding agent.", "messages": messages}).encode(), headers={"content-type": "application/json", "x-api-key": "stub-key", "anthropic-version": "2023-06-01"}, ) urllib.request.urlopen(req, timeout=120).read() forwarded = received[-1]["messages"] newest = max(i for i, m in enumerate(messages) if m["role"] == "user") fresh = (f"newest tool_result {len(json.dumps(messages[newest]))}->" f"{len(json.dumps(forwarded[newest]))} bytes") cache = f"simulated cache read={usage[-1][0]} write={usage[-1][1]}" if previous is None: print(f"[{label}] turn {n}: cold; {fresh}; {cache}") else: # The previous turn's trailing system reminder is replaced this turn. limit = len(previous) - (previous[-1]["role"] == "system") same = 0 while same < limit and strip(forwarded[same]) == strip(previous[same]): same += 1 print(f"[{label}] turn {n}: {same}/{limit} previously forwarded history messages " f"byte-identical; {fresh}; {cache}") previous = forwarded with open(os.path.join(os.path.dirname(__file__), f"forwarded-{label}.json"), "w") as f: json.dump(received, f, indent=1) finally: proxy.terminate() proxy.wait(timeout=30) stub.shutdown() ``` </details> ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` - it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - Headroom Desktop has shipped this as a downstream patch since 2026-09-07 (a wrapper around `resolve_tracker` with the same guards). This PR expresses it inside `resolve_tracker` instead, without mutating the lineage family. - Why Anthropic only: the shape was observed from Claude Code on `/v1/messages`; the OpenAI path keeps its current behaviour until there is evidence of the same pattern there. - Why the current history must extend past the old reminder (not just equal the history before it): a request that drops the reminder and adds nothing is not a continuation we have seen, and keeping the condition strict keeps sibling retries apart. - Documentation: the new env var is described in the `resolve_tracker` docstring; I did not find a docs page listing the prefix-tracker env vars (`HEADROOM_STABLE_BOUNDARY_BREAKPOINT` is not listed either). Happy to add one if there is a preferred place. - `prefix_tracker.py` changed in #3895 after the version the downstream patch targeted; the bug reproduces unchanged on `main`d5318ac2. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3934" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JD Davis
parent
9b9a883e96
commit
16eaec1e95
Vendored
+109
-1
@@ -367,6 +367,81 @@ def _classify_history_canonical(
|
||||
)
|
||||
|
||||
|
||||
_TRANSIENT_SYSTEM_LINEAGE_ENV = "HEADROOM_TRANSIENT_SYSTEM_LINEAGE"
|
||||
|
||||
|
||||
def _transient_system_lineage_enabled() -> bool:
|
||||
return os.environ.get(_TRANSIENT_SYSTEM_LINEAGE_ENV, "").strip().lower() not in (
|
||||
"0",
|
||||
"false",
|
||||
"no",
|
||||
"off",
|
||||
)
|
||||
|
||||
|
||||
def _extends_past_system_tail(current_messages: list[Any], previous_messages: list[Any]) -> bool:
|
||||
"""Whether ``current`` continues ``previous`` once its trailing system turn is set aside.
|
||||
|
||||
Claude Code ends some requests with a ``role:"system"`` reminder and
|
||||
replaces it on the next turn, so the recorded history is never a prefix of
|
||||
the new one. Only that last message is excluded: everything before it
|
||||
must survive unchanged and include real user/assistant history, so a
|
||||
changed leading or historical system instruction still diverges, and the
|
||||
new history must be longer, so a sibling that only swaps the final
|
||||
instruction is not a continuation.
|
||||
"""
|
||||
if len(previous_messages) < 2 or len(current_messages) <= len(previous_messages):
|
||||
return False
|
||||
tail = previous_messages[-1]
|
||||
if not isinstance(tail, dict) or tail.get("role") != "system":
|
||||
return False
|
||||
stable = previous_messages[:-1]
|
||||
return current_messages[: len(stable)] == stable and any(
|
||||
isinstance(message, dict) and message.get("role") in ("user", "assistant")
|
||||
for message in stable
|
||||
)
|
||||
|
||||
|
||||
def _replaced_system_tail_delta(
|
||||
current_messages: list[dict[str, Any]],
|
||||
previous_original_messages: list[dict[str, Any]],
|
||||
previous_forwarded_messages: list[dict[str, Any]],
|
||||
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]] | None:
|
||||
"""Cache-stable split for a request that replaced the previous trailing system turn.
|
||||
|
||||
The handlers record the model's reply after the request it answered, so the
|
||||
replaced reminder is the last recorded message or sits right before that
|
||||
reply. It is left out of the replayed prefix (it is not in this request);
|
||||
every other recorded message must match ``current`` position for position.
|
||||
"""
|
||||
if not _transient_system_lineage_enabled() or len(previous_forwarded_messages) != len(
|
||||
previous_original_messages
|
||||
):
|
||||
return None
|
||||
current = _lineage_snapshot(_canonicalize_for_prefix_compare(current_messages))
|
||||
previous = _lineage_snapshot(_canonicalize_for_prefix_compare(previous_original_messages))
|
||||
if len(current) != len(current_messages) or len(previous) != len(previous_original_messages):
|
||||
# The projection dropped a directive-only message; raw indices would shift.
|
||||
return None
|
||||
for request_len in (len(previous), len(previous) - 1):
|
||||
reply = previous[request_len:]
|
||||
if reply and not (isinstance(reply[0], dict) and reply[0].get("role") == "assistant"):
|
||||
continue
|
||||
reminder = request_len - 1
|
||||
if (
|
||||
_extends_past_system_tail(current, previous[:request_len])
|
||||
and current[reminder : reminder + len(reply)] == reply
|
||||
):
|
||||
return (
|
||||
copy.deepcopy(
|
||||
previous_forwarded_messages[:reminder]
|
||||
+ previous_forwarded_messages[request_len:]
|
||||
),
|
||||
copy.deepcopy(current_messages[reminder + len(reply) :]),
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
def classify_history_relation(
|
||||
current_messages: list[dict[str, Any]],
|
||||
previous_messages: list[dict[str, Any]],
|
||||
@@ -427,10 +502,18 @@ def extract_cache_stable_delta(
|
||||
This is a COMPARISON + slice only: the returned prefix is the previously-forwarded
|
||||
bytes verbatim and the delta is the raw appended messages — never a rebuild from the
|
||||
canonical projection — so the projection dropping non-semantic fields is safe.
|
||||
|
||||
A previous request that ended in a ``role:"system"`` reminder the client has since
|
||||
replaced also qualifies: its forwarded bytes are replayed without that reminder
|
||||
(``HEADROOM_TRANSIENT_SYSTEM_LINEAGE=0`` disables this).
|
||||
"""
|
||||
if not previous_original_messages or previous_forwarded_messages is None:
|
||||
return None
|
||||
relation = classify_history_relation(current_messages, previous_original_messages)
|
||||
if relation.kind == RELATION_DIVERGED:
|
||||
return _replaced_system_tail_delta(
|
||||
current_messages, previous_original_messages, previous_forwarded_messages
|
||||
)
|
||||
if relation.kind not in (RELATION_EXACT, RELATION_MESSAGE_APPEND):
|
||||
# A same-message block append needs a block-level splice in
|
||||
# ``overlay_cached_prefix``; slicing only whole messages would silently
|
||||
@@ -1433,7 +1516,10 @@ class SessionTrackerStore:
|
||||
where a large leading run and two-block identity suffix survive while
|
||||
the middle tail is regenerated; all other rewrites start a fresh
|
||||
lineage. This keeps #2671's stable cache boundary attached without
|
||||
merging unrelated parallel sub-calls.
|
||||
merging unrelated parallel sub-calls. When nothing else matches, an
|
||||
Anthropic chain whose trailing ``role:"system"`` reminder the client
|
||||
replaced is matched on the history before that reminder
|
||||
(``HEADROOM_TRANSIENT_SYSTEM_LINEAGE=0`` disables this).
|
||||
Byte-identical histories (templated fan-outs before they diverge)
|
||||
intentionally share a tracker: their provider cache line is identical
|
||||
too, so sharing is harmless.
|
||||
@@ -1527,6 +1613,28 @@ class SessionTrackerStore:
|
||||
len(rewrite_candidates) == 1 or rewrite_candidates[0][0] != rewrite_candidates[1][0]
|
||||
):
|
||||
best_key = rewrite_candidates[0][1]
|
||||
elif (
|
||||
not rewrite_candidates
|
||||
and provider == "anthropic"
|
||||
and _transient_system_lineage_enabled()
|
||||
):
|
||||
# Nothing matched as recorded. A chain whose only change is
|
||||
# its replaced trailing system reminder is still this
|
||||
# conversation; without it every such turn starts a cold
|
||||
# tracker, the confirmed prefix is not replayed, and any
|
||||
# recompressed history re-writes the provider cache. Only a
|
||||
# unique longest chain qualifies, so equal-length siblings
|
||||
# that differ in their reminder stay separate.
|
||||
tail_candidates = [
|
||||
(len(chain), key)
|
||||
for key, chain in by_length
|
||||
if self._lineage_affinities.get(key) == cache_affinity
|
||||
and _extends_past_system_tail(snap, chain)
|
||||
]
|
||||
if tail_candidates and (
|
||||
len(tail_candidates) == 1 or tail_candidates[0][0] != tail_candidates[1][0]
|
||||
):
|
||||
best_key = tail_candidates[0][1]
|
||||
|
||||
if best_key is None:
|
||||
cap = self._default_config.max_lineages_per_session
|
||||
|
||||
@@ -13,6 +13,8 @@ from headroom.cache.prefix_tracker import (
|
||||
PrefixCacheTracker,
|
||||
PrefixFreezeConfig,
|
||||
SessionTrackerStore,
|
||||
extract_cache_stable_delta,
|
||||
overlay_cached_prefix,
|
||||
)
|
||||
|
||||
|
||||
@@ -775,6 +777,149 @@ class TestConversationLineageResolution:
|
||||
fresh = store.resolve_tracker("sid", "anthropic", messages=self._history("A", 2))
|
||||
assert fresh._turn_number == 0
|
||||
|
||||
@staticmethod
|
||||
def _reminder_turns() -> tuple[list[dict], list[dict]]:
|
||||
"""Claude Code shape: a trailing system reminder replaced by the next turn."""
|
||||
history = [
|
||||
{"role": "user", "content": "Inspect example.py."},
|
||||
{"role": "assistant", "content": "Reading the file."},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_result",
|
||||
"tool_use_id": "read-1",
|
||||
"content": "original source line\n" * 400,
|
||||
}
|
||||
],
|
||||
},
|
||||
]
|
||||
reminder = {"role": "system", "content": "Batch independent tools this turn."}
|
||||
previous = [*history, reminder]
|
||||
current = [
|
||||
*history,
|
||||
{"role": "assistant", "content": "Now run the checks."},
|
||||
{"role": "user", "content": "The checks passed."},
|
||||
dict(reminder),
|
||||
]
|
||||
return previous, current
|
||||
|
||||
def test_replaced_system_tail_keeps_confirmed_tracker(self, store):
|
||||
"""The previous request ended in a system reminder the client swapped
|
||||
for new turns. Its unchanged history must keep the tracker and its
|
||||
confirmed frozen prefix instead of starting a cold lineage."""
|
||||
previous, current = self._reminder_turns()
|
||||
forwarded = [dict(m) for m in previous]
|
||||
forwarded[2] = {**previous[2], "content": "previously cached compressed result"}
|
||||
tracker = store.resolve_tracker("sid", "anthropic", previous, cache_affinity="a")
|
||||
tracker.update_from_response(
|
||||
cache_read_tokens=0,
|
||||
cache_write_tokens=1_000_000,
|
||||
messages=forwarded,
|
||||
original_messages=previous,
|
||||
message_token_counts=[100] * len(previous),
|
||||
)
|
||||
frozen = tracker.get_frozen_message_count()
|
||||
assert frozen > 0
|
||||
|
||||
assert store.resolve_tracker("sid", "anthropic", current, cache_affinity="a") is tracker
|
||||
assert tracker.get_frozen_message_count() == frozen
|
||||
# The existing replay restores the unchanged prefix and stops at the
|
||||
# replaced reminder; this turn's tail is forwarded as processed.
|
||||
processed = [dict(m) for m in current]
|
||||
processed[2] = {**current[2], "content": "a different compressed result"}
|
||||
replayed = overlay_cached_prefix(
|
||||
processed, current, previous, forwarded, confirmed_frozen_count=frozen
|
||||
)
|
||||
assert replayed[:3] == forwarded[:3]
|
||||
assert replayed[3:] == processed[3:]
|
||||
|
||||
def test_moving_system_tail_records_full_snapshots(self, store):
|
||||
"""The reminder moves and changes every turn; each turn stays on one
|
||||
tracker and the store records the full current history."""
|
||||
previous, current = self._reminder_turns()
|
||||
tracker = store.resolve_tracker("sid", "anthropic", previous)
|
||||
for i in range(4):
|
||||
current[-1] = {"role": "system", "content": f"reminder {i}"}
|
||||
assert store.resolve_tracker("sid", "anthropic", current) is tracker
|
||||
assert store._lineages["sid"]["sid"][-1]["content"][0]["text"] == f"reminder {i}"
|
||||
assert len(store._lineages["sid"]["sid"]) == len(current)
|
||||
current = [
|
||||
*current[:-1],
|
||||
{"role": "assistant", "content": f"reply {i}"},
|
||||
{"role": "user", "content": f"next {i}"},
|
||||
current[-1],
|
||||
]
|
||||
|
||||
def test_system_tail_fallback_respects_affinity_and_provider(self):
|
||||
previous, current = self._reminder_turns()
|
||||
for provider, affinity in (("anthropic", "other-tools"), ("openai", "a")):
|
||||
store = SessionTrackerStore()
|
||||
tracker = store.resolve_tracker("sid", provider, previous, cache_affinity="a")
|
||||
resolved = store.resolve_tracker("sid", provider, current, cache_affinity=affinity)
|
||||
assert resolved is not tracker
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("index", "role"),
|
||||
[(0, "user"), (0, "system"), (2, "system")],
|
||||
ids=["changed-history", "changed-leading-system", "changed-historical-system"],
|
||||
)
|
||||
def test_system_tail_fallback_requires_unchanged_history(self, store, index, role):
|
||||
"""Only the trailing reminder is set aside: any other change, including
|
||||
a changed leading or historical system instruction, still diverges."""
|
||||
previous, current = self._reminder_turns()
|
||||
previous.insert(index, {"role": role, "content": "original"})
|
||||
current.insert(index, {"role": role, "content": "changed"})
|
||||
tracker = store.resolve_tracker("sid", "anthropic", previous)
|
||||
assert store.resolve_tracker("sid", "anthropic", current) is not tracker
|
||||
|
||||
def test_swapped_final_instruction_is_not_a_continuation(self, store):
|
||||
"""Same-length siblings that differ only in their final system message
|
||||
stay separate: the new history must extend past the old reminder."""
|
||||
first = [{"role": "user", "content": "task"}, {"role": "system", "content": "A"}]
|
||||
second = [{"role": "user", "content": "task"}, {"role": "system", "content": "B"}]
|
||||
tracker = store.resolve_tracker("sid", "anthropic", first)
|
||||
assert store.resolve_tracker("sid", "anthropic", second) is not tracker
|
||||
assert store.resolve_tracker("sid", "anthropic", first) is tracker
|
||||
|
||||
def test_ambiguous_system_tail_siblings_are_not_merged(self, store, monkeypatch):
|
||||
"""Two lineages that differ only in their reminder (recorded here with
|
||||
the fallback disabled) are an ambiguous match: start a fresh lineage
|
||||
instead of guessing, and an exact match still wins."""
|
||||
previous, current = self._reminder_turns()
|
||||
sibling = [*previous[:-1], {"role": "system", "content": "other reminder"}]
|
||||
monkeypatch.setenv("HEADROOM_TRANSIENT_SYSTEM_LINEAGE", "0")
|
||||
tracker = store.resolve_tracker("sid", "anthropic", previous)
|
||||
other = store.resolve_tracker("sid", "anthropic", sibling)
|
||||
assert other is not tracker
|
||||
monkeypatch.delenv("HEADROOM_TRANSIENT_SYSTEM_LINEAGE")
|
||||
assert store.resolve_tracker("sid", "anthropic", current) not in (tracker, other)
|
||||
assert store.resolve_tracker("sid", "anthropic", sibling) is other
|
||||
|
||||
def test_cache_delta_replays_prefix_past_replaced_system_tail(self, monkeypatch):
|
||||
"""Cache mode's delta split leaves the replaced reminder out of the
|
||||
replayed prefix, with or without the model reply the handlers record
|
||||
after the request, and compresses everything after it as new."""
|
||||
previous, current = self._reminder_turns()
|
||||
reply = current[3]
|
||||
forwarded = [*previous[:2], {**previous[2], "content": "compressed"}, previous[3], reply]
|
||||
for recorded in ([*previous, reply], previous):
|
||||
replay = forwarded[: len(recorded)]
|
||||
prefix, delta = extract_cache_stable_delta(current, recorded, replay)
|
||||
assert prefix == replay[:3] + replay[4:]
|
||||
assert delta == current[len(prefix) :]
|
||||
|
||||
# A system message followed by a client turn is history, not a reminder.
|
||||
changed = [*previous, {"role": "user", "content": "Continue."}]
|
||||
dropped = [*previous[:3], changed[4], *current[3:]]
|
||||
assert extract_cache_stable_delta(dropped, changed, changed) is None
|
||||
# The client's copy of the reply must match the recorded one.
|
||||
other = [*previous, {"role": "assistant", "content": "Something else."}]
|
||||
assert extract_cache_stable_delta(current, other, other) is None
|
||||
|
||||
monkeypatch.setenv("HEADROOM_TRANSIENT_SYSTEM_LINEAGE", "0")
|
||||
assert extract_cache_stable_delta(current, [*previous, reply], forwarded) is None
|
||||
|
||||
def test_shared_session_id_is_not_rotated(self, store):
|
||||
"""Composition guard: lineage resolution must not leak into session-id
|
||||
derivation — beta stickiness and the compression cache key on it."""
|
||||
|
||||
@@ -235,3 +235,101 @@ def test_cache_mode_prefix_mismatch_passes_through_with_tag(monkeypatch):
|
||||
assert forwarded[0]["content"][0]["content"] == original_text
|
||||
# ...but now visibly tagged instead of silent.
|
||||
assert handler.recorded_tags["passthrough_reason"] == "cache_mode_prefix_mismatch"
|
||||
|
||||
|
||||
def _assistant(turn: int) -> dict:
|
||||
return {
|
||||
"role": "assistant",
|
||||
"content": [{"type": "tool_use", "id": f"toolu_{turn}", "name": "read", "input": {}}],
|
||||
}
|
||||
|
||||
|
||||
def _tool_result(turn: int) -> dict:
|
||||
return {
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_result",
|
||||
"tool_use_id": f"toolu_{turn}",
|
||||
"content": f"verbose tool output {turn} " * 200,
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _compress_unfrozen(messages, model, frozen_message_count=0, **kwargs):
|
||||
"""Like ``_fake_pipeline_apply`` but leaves frozen messages alone, as the
|
||||
real pipeline does."""
|
||||
result = _fake_pipeline_apply(messages[frozen_message_count:], model)
|
||||
return TransformResult(
|
||||
messages=list(messages[:frozen_message_count]) + result.messages,
|
||||
tokens_before=result.tokens_before,
|
||||
tokens_after=result.tokens_after,
|
||||
transforms_applied=result.transforms_applied,
|
||||
)
|
||||
|
||||
|
||||
class _ConversationHandler(_DummyAnthropicHandler):
|
||||
"""Real session tracker store; the upstream answers with the next tool call."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
from headroom.cache.prefix_tracker import SessionTrackerStore
|
||||
|
||||
super().__init__()
|
||||
self.session_tracker_store = SessionTrackerStore()
|
||||
self.anthropic_pipeline = SimpleNamespace(apply=MagicMock(side_effect=_compress_unfrozen))
|
||||
self.forwarded: list[list[dict]] = []
|
||||
|
||||
async def _retry_request(self, method, url, headers, body, **_kwargs):
|
||||
self.forwarded.append(json.loads(json.dumps(body["messages"])))
|
||||
reply = {
|
||||
"id": f"msg_{len(self.forwarded)}",
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"content": _assistant(len(self.forwarded) + 1)["content"],
|
||||
"usage": {"input_tokens": 1, "output_tokens": 1},
|
||||
}
|
||||
return SimpleNamespace(
|
||||
status_code=200, headers={}, content=json.dumps(reply).encode(), json=lambda: reply
|
||||
)
|
||||
|
||||
|
||||
def test_cache_mode_replaced_system_tail_keeps_compressed_prefix(monkeypatch):
|
||||
"""Claude Code replaces a trailing system reminder every turn. Cache mode
|
||||
must replay last turn's compressed prefix without the old reminder and
|
||||
compress only the new messages, not fall back to raw forwarding."""
|
||||
import headroom.tokenizers as _tk
|
||||
|
||||
monkeypatch.setattr(_tk, "get_tokenizer", lambda model: _DummyTokenizer())
|
||||
handler = _ConversationHandler()
|
||||
model = "claude-fable-5"
|
||||
history = [{"role": "user", "content": "Read the files."}, _assistant(1), _tool_result(1)]
|
||||
turns = [
|
||||
history,
|
||||
[*history, _assistant(2), _tool_result(2), {"role": "system", "content": "Reminder A"}],
|
||||
[
|
||||
*history,
|
||||
_assistant(2),
|
||||
_tool_result(2),
|
||||
_assistant(3),
|
||||
_tool_result(3),
|
||||
{"role": "system", "content": "Reminder B"},
|
||||
],
|
||||
]
|
||||
for messages in turns:
|
||||
request = _build_request({"model": model, "max_tokens": 16, "messages": messages})
|
||||
anyio.run(handler.handle_anthropic_messages, request)
|
||||
|
||||
from headroom.cache.prefix_tracker import _strip_cache_control
|
||||
|
||||
# The proxy re-places cache_control markers every turn; compare the bytes under them.
|
||||
second, third = (_strip_cache_control(m) for m in handler.forwarded[1:])
|
||||
assert [m["role"] for m in third] == [m["role"] for m in turns[2]]
|
||||
# Everything before the replaced reminder is last turn's forwarded bytes...
|
||||
assert third[:5] == second[:5]
|
||||
assert second[2]["content"][0]["content"] == _COMPRESSED_TEXT
|
||||
assert second[4]["content"][0]["content"] == _COMPRESSED_TEXT
|
||||
# ...the new tool result is compressed, and the current reminder is sent as is.
|
||||
assert third[6]["content"][0]["content"] == _COMPRESSED_TEXT
|
||||
assert third[-1] == {"role": "system", "content": "Reminder B"}
|
||||
assert "passthrough_reason" not in handler.recorded_tags
|
||||
|
||||
Reference in New Issue
Block a user