fix(cache): keep prefix lineage across a replaced system tail (#3934)

## Description


Claude Code can end a `/v1/messages` request with a temporary
`role:"system"` reminder and replace it on the next turn.
`SessionTrackerStore.resolve_tracker` records the full history, so the
stored chain ends in the old reminder and is never a prefix of the next
request: `_classify_history_canonical` returns `diverged` and the
request gets a fresh, empty tracker. The confirmed frozen prefix and the
previously forwarded messages are gone, the #3380 replay has nothing to
replay, and any recompressed history re-writes the provider prompt
cache. The reminder moves every turn, so this repeats for the whole
session.

This change lets lineage resolution set aside that one replaced trailing
system message, only when nothing matched as recorded. It changes
tracker selection only: forwarded bytes, the replay body, and breakpoint
placement are untouched, and the existing replay already stops at the
first changed message, so the new reminder goes out as the client sent
it.

Closes #3933

## Type of Change


- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cache/prefix_tracker.py`: new
`_extends_past_system_tail(current, previous)`. It is true when
`previous` ends in a `role:"system"` message, everything before that
message includes user/assistant history and is unchanged in `current`,
and `current` is longer than `previous`. Because `current` must be
longer, siblings that only swap their final instruction stay separate.
- `SessionTrackerStore.resolve_tracker`: after the existing
exact/append, block-append and block-rewrite passes find nothing (and
there are no rewrite candidates, so an ambiguous rewrite tie is not
overridden), Anthropic requests consider chains with the same cache
affinity that pass that check. Only a unique longest candidate is taken;
two equal-length candidates start a fresh lineage, as before. The full
current snapshot is stored as usual.
- `extract_cache_stable_delta` (cache mode's prefix/delta split): a
`diverged` request that replaced the previous trailing system reminder
is split too. Every recorded message before the reminder, plus the model
reply the handlers record after it, must match the current request
position for position (canonical compare, same rule via
`_extends_past_system_tail`). The previously forwarded bytes are
replayed without the old reminder, and everything after it is the delta
to compress. Before this, cache mode tagged these turns
`cache_mode_prefix_mismatch` and forwarded the new messages raw.
- `HEADROOM_TRANSIENT_SYSTEM_LINEAGE=0` (also `false`/`no`/`off`) turns
off both the lineage fallback and the delta split. It uses the same
parsing as `HEADROOM_STABLE_BOUNDARY_BREAKPOINT`.
- `tests/test_cache/test_prefix_tracker.py`:
- Tracker retention: the tracker and its confirmed frozen count are
kept, and the existing overlay restores the unchanged prefix.
- A reminder that moves and changes over several turns: full snapshots
are stored.
- Isolation: different cache affinity, OpenAI provider, changed earlier
history, changed leading system message, changed historical system
message, same-length siblings `[user: task, system: A]` vs `[user: task,
system: B]`, ambiguous equal-length siblings, exact-match precedence,
and the kill switch.
- The delta split: with and without the recorded reply, a system message
followed by a client turn is not treated as a reminder, a mismatched
reply is rejected, and the kill switch disables it.
- `tests/test_cache_mode_cold_start.py`: drives
`handle_anthropic_messages` in `mode="cache"` with the real
`SessionTrackerStore` over three turns. The first turn compresses a tool
result and later turns replace the reminder. The test asserts that the
third request replays the second request's compressed prefix
byte-for-byte (ignoring `cache_control`), compresses the new tool
result, and ends in the current reminder.

## Testing

- [x] Unit tests pass (`pytest`) - targeted files below, not the whole
suite
- [x] Linting passes (`ruff check .`) - on the changed files, plus `ruff
format --check`
- [x] Type checking passes (`mypy headroom`) - on
`headroom/cache/prefix_tracker.py`
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

New tests with the source at `main` (d5318ac2):

```text
$ python -m pytest tests/test_cache_mode_cold_start.py -q -p no:cacheprovider
FAILED tests/test_cache_mode_cold_start.py::test_cache_mode_replaced_system_tail_keeps_compressed_prefix
========================= 1 failed, 2 passed in 1.70s ==========================
```

With the lineage commit only (2c5af818 plus the length fix, before the
delta split), the cache-mode tests fail on the raw new tool result:

```text
E   AssertionError: assert 'verbose tool...ool output 3 ' == 'compressed tool output'
E   TypeError: cannot unpack non-iterable NoneType object
FAILED tests/test_cache_mode_cold_start.py::test_cache_mode_replaced_system_tail_keeps_compressed_prefix
FAILED tests/test_cache/test_prefix_tracker.py::TestConversationLineageResolution::test_cache_delta_replays_prefix_past_replaced_system_tail
========================= 2 failed, 73 passed in 2.83s =========================
```

The same-length sibling test fails at 2c5af818:

```text
FAILED tests/test_cache/test_prefix_tracker.py::TestConversationLineageResolution::test_swapped_final_instruction_is_not_a_continuation
========================= 1 failed, 70 passed in 0.75s =========================
```

With all three commits:

```text
$ python -m pytest tests/test_cache/test_prefix_tracker.py tests/test_cache_mode_cold_start.py \
    tests/test_cache_mode_delta_marker.py tests/test_observed_wire_shapes.py tests/test_proxy_handler_helpers.py \
    tests/test_issue_2671_block_growth_cache.py tests/test_proxy_anthropic_cache_stability.py \
    tests/test_proxy_openai_cache_stability.py -q -p no:cacheprovider
======================= 225 passed, 1 warning in 46.30s ========================

$ ruff check headroom/cache/prefix_tracker.py tests/test_cache/test_prefix_tracker.py tests/test_cache_mode_cold_start.py
All checks passed!
$ ruff format --check headroom/cache/prefix_tracker.py tests/test_cache/test_prefix_tracker.py tests/test_cache_mode_cold_start.py
3 files already formatted
$ mypy headroom/cache/prefix_tracker.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS 27.0.1, Python 3.11.13, this branch (5b0c20f2) vs
`main` d5318ac2. The proxy was started from each checkout with `python
-m headroom.proxy.server --port <free> --anthropic-api-url
http://127.0.0.1:<stub> --no-rate-limit`, once with
`HEADROOM_MODE=cache` (the default) and once with `HEADROOM_MODE=token`.
The `headroom proxy` CLI refuses to start in this dev venv because the
optional `mcp`, `magika` and `onnxruntime` extras are missing; the
server module itself runs. The upstream is a local stdlib HTTP stub that
records each forwarded body and answers with the next `tool_use`. It
returns Anthropic-shaped `usage` from a simulated prompt cache: the
whole forwarded prompt is cached; the next request reads the leading run
of messages that are byte-identical to the previous forwarded request
(ignoring `cache_control`) and writes the rest. Tokens are approximated
as serialized chars / 3. No real provider was contacted.
- Exact command / steps: `PROOF_MODE=cache|token proof.py <checkout>
<python> <label>` (script below) sends four `claude-fable-5` turns of a
paged tool loop. Each turn adds an assistant `tool_use` and a ~31 KB
JSON `tool_result`, and each request ends in a `role:"system"` reminder
whose text changes every turn. Controls: `-noreminder` sends the same
turns without reminders, and `-mixed` drops only the first turn's
reminder, which is how Claude Code interleaves them.
- Observed result: in cache mode every reminder turn on the branch
replays the previously forwarded history byte-identical and compresses
only the new tool result (simulated read/write per turn: main
10,598/10,581, 21,162/10,581, 31,726/10,580 with nothing compressed;
branch 10,598/3,106, 13,687/3,106, 16,776/3,105); in the mixed run main
re-writes the cache on the first reminder turn (76 read / 31,667 write)
and the branch reads 6,212 / writes 3,106; in token mode main re-writes
the history every reminder turn (read 76/281/486) and the branch reads
3,123/6,212/9,301, matching the no-reminder control.

Simulated cache read / write per turn, and the proxy's `PERF tok_after`
(tokens forwarded):

  | Mode, run | Turn 1 | Turn 2 | Turn 3 | Turn 4 |
  | --- | --- | --- | --- | --- |
| cache, main | 0 / 10,615 (10,886) | 10,598 / 10,581 (21,747) | 21,162
/ 10,581 (32,608) | 31,726 / 10,580 (43,469) |
| cache, branch before delta split (2c5af818) | 0 / 10,615 (10,886) |
10,598 / 10,581 (21,747) | 21,162 / 10,581 (32,608) | 31,726 / 10,580
(43,469) |
| cache, branch | 0 / 10,615 (10,886) | 10,598 / 3,106 (15,122) | 13,687
/ 3,106 (19,358) | 16,776 / 3,105 (23,594) |
| cache, branch, `HEADROOM_TRANSIENT_SYSTEM_LINEAGE=0` | 0 / 10,615
(10,886) | 10,598 / 10,581 (21,747) | 21,162 / 10,581 (32,608) | 31,726
/ 10,580 (43,469) |
| cache, main, no reminders (control) | 0 / 3,123 (4,253) | 3,123 /
3,089 (8,489) | 6,212 / 3,089 (12,725) | 9,301 / 3,088 (16,961) |
| cache mixed, main | 0 / 3,123 (4,253) | 3,123 / 3,106 (8,498) | 76 /
31,667 (32,608) | 31,726 / 10,580 (43,469) |
| cache mixed, branch before delta split (2c5af818) | 0 / 3,123 (4,253)
| 3,123 / 3,106 (8,498) | 6,212 / 10,581 (19,359) | 16,776 / 10,580
(30,220) |
| cache mixed, branch | 0 / 3,123 (4,253) | 3,123 / 3,106 (8,498) |
6,212 / 3,106 (12,734) | 9,301 / 3,105 (16,970) |
| token, main | 0 / 3,140 (4,262) | 76 / 3,271 (4,429) | 281 / 3,271
(4,595) | 486 / 3,270 (4,762) |
| token, branch | 0 / 3,140 (4,262) | 3,123 / 3,106 (8,498) | 6,212 /
3,106 (12,734) | 9,301 / 3,105 (16,970) |
| token, main, no reminders (control) | 0 / 3,123 (4,253) | 3,123 /
3,089 (8,489) | 6,212 / 3,089 (12,725) | 9,301 / 3,088 (16,961) |

  How to read it:
- **Cache mode, main:** for a session that always ends in a reminder,
the cache is stable but nothing is ever compressed. Each turn gets a
fresh tracker, and the cold-start branch freezes the whole request
because its last message is not a user turn.
- **Cache mode, mixed run on main:** the first reminder turn after a
compressed turn re-sends the compressed history raw and re-writes the
cache.
- **Lineage commit alone (2c5af818):** kept the history stable, because
the cached-prefix replay in `finalize_turn` restores it. New tool
results still went out raw (`cache_mode_prefix_mismatch`).
- **With the delta split:** they are compressed (31,572 -> 9,187 bytes
each), and the mixed run matches the no-reminder control.
- **Every forwarded request on the branch** ends in that turn's own
reminder text, and the old reminder is not replayed.
- **Turn 1 of the always-reminder cache run** is forwarded raw on both
main and the branch. A cold request whose last message is a system
message is frozen whole by `_strict_previous_turn_frozen_count`. This PR
does not change that.
- Downstream production evidence (Headroom Desktop on the 0.37.0 wheel,
`claude-fable-5`, token mode, provider-reported `usage`, no request
content): one long Claude Code session went 0 read / 490,996 write
(cold), then 490,996 read / 12,509 write on an 862-message request
ending in the reminder, then 7,259 read / 491,260 write on the next
864-message request, although its first 861 messages were unchanged. 27
such continuations in that window wrote 15,183,730 cache tokens and read
195,993. After the lineage fix was applied downstream, a live run of 13
requests (9 replaced the reminder) read 664,386 and wrote 49,646 cache
tokens (92.99% reuse).
- Not tested: no real Anthropic request or live provider cache billing
for this branch; details below.
- No real Anthropic request was made from this branch, so live provider
cache billing was not measured for this PR. The cache numbers above come
from the stub's simulation.
- The downstream numbers come from a vendored lineage patch on the
0.37.0 wheel in token mode. That patch has the same selection rules but
not this exact code, and it has no cache-mode delta split, which is new
here and has not run against a live provider.
  - Not exercised end to end:
- streaming responses (they record the reply the same way, but were not
driven here)
    - the Bedrock backend path
    - the `headroom proxy` CLI entry point with all extras installed
- models that relocate the reminder into the top-level `system` field.
There the reminder is part of the system segment and this change has no
effect.
- OpenAI paths are unchanged by design and are covered only by their
existing tests.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is a fix in lineage selection
and in cache mode's prefix/delta split, not a rollout feature in
`headroom/rollout.py`.
- Minimum rollout channel: n/a (applies on every channel).
- Stable/default behavior changed: yes, for Anthropic requests whose
previous request ended in a `role:"system"` message that the client
replaced:
- **Token mode:** they keep their existing prefix tracker, so the
confirmed prefix is replayed byte-stable instead of being recompressed.
That history is therefore no longer recompressed on those turns.
- **Cache mode (default):** the same request is split into the
previously forwarded prefix plus a compressed delta instead of being
forwarded with its new messages raw. Compression on those turns goes up,
cache reads stay at the replayed prefix, and a session that alternates
reminder and non-reminder turns no longer re-writes its cache on the
first reminder turn.
- **Unchanged:** OpenAI, conversations without a trailing system
message, siblings that only swap their final instruction, and every case
where an existing relation already matches.
- Kill switch / disable path: `HEADROOM_TRANSIENT_SYSTEM_LINEAGE=0` (or
`false`/`no`/`off`), read per request. It disables both the lineage
fallback and the cache-mode split. In the proof, the cache-mode run with
it set matches `main` exactly.
- Unsafe override required: no.
- Qualification impact: expect higher provider cache reads and lower
cache writes on Claude Code sessions that send trailing system reminders
in token mode, and more compression of new messages on those turns in
cache mode. This was measured here only against a stub; live provider
billing for this branch was not measured.
- Rollback path: set `HEADROOM_TRANSIENT_SYSTEM_LINEAGE=0` and restart
the proxy, or revert the three commits. No persisted state or schema
changes.

<details>
<summary>proof.py</summary>

```python
"""Boot the proxy (`python -m headroom.proxy.server`) from a given checkout
against a local stub Anthropic upstream and replay a Claude Code shaped
conversation (paged tool calls, ~31 KB JSON tool results) whose trailing
role:"system" reminder is replaced every turn.

Usage: PROOF_MODE=cache|token proof.py <checkout-dir> <python> <label>
Label suffix "-noreminder" drops the reminders (control); "-mixed" drops only
the first one. Reports, per turn, how many previously forwarded history
messages were forwarded again byte-identical (cache_control ignored), the
newest tool result's size before/after the proxy, and the stub's simulated
prompt-cache read/write. No real provider is contacted: the upstream is
127.0.0.1.
"""

import json
import os
import socket
import subprocess
import sys
import threading
import time
import urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

checkout, python, label = sys.argv[1], sys.argv[2], sys.argv[3]
received: list[dict] = []
usage: list[tuple[int, int]] = []


def strip(x):
    """Drop cache_control markers: the proxy re-places them every turn and the
    provider does not hash them into the cached prefix."""
    if isinstance(x, dict):
        return {k: strip(v) for k, v in x.items() if k != "cache_control"}
    if isinstance(x, list):
        return [strip(v) for v in x]
    return x


class Stub(BaseHTTPRequestHandler):
    def do_POST(self):
        body = json.loads(self.rfile.read(int(self.headers["content-length"])))
        received.append(body)
        # Minimal provider prompt-cache simulation: the whole forwarded prompt
        # is cached (client-style breakpoint on the newest message); this turn
        # reads the leading run of messages byte-identical to the previous
        # forwarded request, and writes the rest. Tokens ~= serialized chars / 3.
        def toks(x):
            return len(json.dumps(strip(x), sort_keys=True)) // 3

        msgs = body["messages"]
        total = toks(body.get("system")) + sum(toks(m) for m in msgs)
        read = 0
        if len(received) > 1 and strip(received[-2].get("system")) == strip(body.get("system")):
            prev = received[-2]["messages"]
            k = 0
            while k < min(len(prev), len(msgs)) and strip(prev[k]) == strip(msgs[k]):
                k += 1
            if k:
                read = toks(body.get("system")) + sum(toks(m) for m in msgs[:k])
        usage.append((read, total - read))
        payload = json.dumps(
            {
                "id": f"msg_{len(received)}",
                "type": "message",
                "role": "assistant",
                "model": body["model"],
                "content": assistant(len(received) + 1)["content"],
                "stop_reason": "tool_use",
                "stop_sequence": None,
                "usage": {
                    "input_tokens": 10,
                    "output_tokens": 2,
                    "cache_creation_input_tokens": total - read,
                    "cache_read_input_tokens": read,
                },
            }
        ).encode()
        self.send_response(200)
        self.send_header("content-type", "application/json")
        self.send_header("content-length", str(len(payload)))
        self.end_headers()
        self.wfile.write(payload)

    def log_message(self, *args):
        pass


def assistant(k: int) -> dict:
    return {"role": "assistant", "content": [
        {"type": "tool_use", "id": f"toolu_{k}", "name": "list_services", "input": {"page": k}}]}


def tool_result(k: int) -> dict:
    rows = [{"id": i, "name": f"service-{k}-{i}", "status": "ok" if (i + k) % 7 else "degraded",
             "latency_ms": 20 + (i * 37 + k) % 400, "region": ["eu", "us", "ap"][i % 3]}
            for i in range(300)]
    return {"role": "user", "content": [
        {"type": "tool_result", "tool_use_id": f"toolu_{k}", "content": json.dumps(rows)}]}


def free_port() -> int:
    with socket.socket() as s:
        s.bind(("127.0.0.1", 0))
        return s.getsockname()[1]


stub = ThreadingHTTPServer(("127.0.0.1", free_port()), Stub)
threading.Thread(target=stub.serve_forever, daemon=True).start()
port = free_port()
env = {
    **os.environ,
    "PYTHONPATH": checkout,
    "HEADROOM_BEACON": "off",
    "HEADROOM_MODE": os.environ.get("PROOF_MODE", "token"),
    "LITELLM_LOCAL_MODEL_COST_MAP": "True",
    "HEADROOM_DATA_DIR": os.path.join(os.path.dirname(__file__), f"data-{label}"),
}
env.pop("GROK_CLI_CHAT_PROXY_BASE_URL", None)
log = open(os.path.join(os.path.dirname(__file__), f"proxy-{label}.log"), "w")
proxy = subprocess.Popen(
    [
        python, "-m", "headroom.proxy.server", "--port", str(port),
        "--anthropic-api-url", f"http://127.0.0.1:{stub.server_address[1]}",
        "--no-rate-limit",
    ],
    env=env, stdout=log, stderr=subprocess.STDOUT,
)
try:
    for _ in range(240):
        try:
            urllib.request.urlopen(f"http://127.0.0.1:{port}/health", timeout=1)
            break
        except Exception:
            time.sleep(0.5)
    else:
        raise SystemExit("proxy did not come up")

    turns = []
    convo = [{"role": "user", "content": "Page through the services and find the degraded ones."}]
    for n in range(1, 5):
        convo = convo + [assistant(n), tool_result(n)]
        turns.append(convo + [{"role": "system", "content": f"Reminder variant {n}."}])

    if label.endswith("noreminder"):
        # Control: the same conversation without the trailing system reminder.
        turns = [t[:-1] for t in turns]
    elif label.endswith("mixed"):
        # Claude Code sends reminders on some turns only: the first request
        # has none (so it is compressed and that form is cached), later ones do.
        turns[0] = turns[0][:-1]
    previous = None
    for n, messages in enumerate(turns, 1):
        req = urllib.request.Request(
            f"http://127.0.0.1:{port}/v1/messages",
            data=json.dumps({"model": "claude-fable-5", "max_tokens": 64,
                             "system": "You are a coding agent.",
                             "messages": messages}).encode(),
            headers={"content-type": "application/json", "x-api-key": "stub-key",
                     "anthropic-version": "2023-06-01"},
        )
        urllib.request.urlopen(req, timeout=120).read()
        forwarded = received[-1]["messages"]
        newest = max(i for i, m in enumerate(messages) if m["role"] == "user")
        fresh = (f"newest tool_result {len(json.dumps(messages[newest]))}->"
                 f"{len(json.dumps(forwarded[newest]))} bytes")
        cache = f"simulated cache read={usage[-1][0]} write={usage[-1][1]}"
        if previous is None:
            print(f"[{label}] turn {n}: cold; {fresh}; {cache}")
        else:
            # The previous turn's trailing system reminder is replaced this turn.
            limit = len(previous) - (previous[-1]["role"] == "system")
            same = 0
            while same < limit and strip(forwarded[same]) == strip(previous[same]):
                same += 1
            print(f"[{label}] turn {n}: {same}/{limit} previously forwarded history messages "
                  f"byte-identical; {fresh}; {cache}")
        previous = forwarded
    with open(os.path.join(os.path.dirname(__file__), f"forwarded-{label}.json"), "w") as f:
        json.dump(received, f, indent=1)
finally:
    proxy.terminate()
    proxy.wait(timeout=30)
    stub.shutdown()
```

</details>

## Review Readiness


- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist


- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)


N/A

## Additional Notes


- Headroom Desktop has shipped this as a downstream patch since
2026-09-07 (a wrapper around `resolve_tracker` with the same guards).
This PR expresses it inside `resolve_tracker` instead, without mutating
the lineage family.
- Why Anthropic only: the shape was observed from Claude Code on
`/v1/messages`; the OpenAI path keeps its current behaviour until there
is evidence of the same pattern there.
- Why the current history must extend past the old reminder (not just
equal the history before it): a request that drops the reminder and adds
nothing is not a continuation we have seen, and keeping the condition
strict keeps sibling retries apart.
- Documentation: the new env var is described in the `resolve_tracker`
docstring; I did not find a docs page listing the prefix-tracker env
vars (`HEADROOM_STABLE_BOUNDARY_BREAKPOINT` is not listed either). Happy
to add one if there is a preferred place.
- `prefix_tracker.py` changed in #3895 after the version the downstream
patch targeted; the bug reproduces unchanged on `main` d5318ac2.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3934"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
gglucass
2026-10-04 13:58:14 -07:00
committed by GitHub
co-authored by Claude Opus 5.5 JD Davis
parent 9b9a883e96
commit 16eaec1e95
3 changed files with 352 additions and 1 deletions
+109 -1
View File
@@ -367,6 +367,81 @@ def _classify_history_canonical(
)
_TRANSIENT_SYSTEM_LINEAGE_ENV = "HEADROOM_TRANSIENT_SYSTEM_LINEAGE"
def _transient_system_lineage_enabled() -> bool:
return os.environ.get(_TRANSIENT_SYSTEM_LINEAGE_ENV, "").strip().lower() not in (
"0",
"false",
"no",
"off",
)
def _extends_past_system_tail(current_messages: list[Any], previous_messages: list[Any]) -> bool:
"""Whether ``current`` continues ``previous`` once its trailing system turn is set aside.
Claude Code ends some requests with a ``role:"system"`` reminder and
replaces it on the next turn, so the recorded history is never a prefix of
the new one. Only that last message is excluded: everything before it
must survive unchanged and include real user/assistant history, so a
changed leading or historical system instruction still diverges, and the
new history must be longer, so a sibling that only swaps the final
instruction is not a continuation.
"""
if len(previous_messages) < 2 or len(current_messages) <= len(previous_messages):
return False
tail = previous_messages[-1]
if not isinstance(tail, dict) or tail.get("role") != "system":
return False
stable = previous_messages[:-1]
return current_messages[: len(stable)] == stable and any(
isinstance(message, dict) and message.get("role") in ("user", "assistant")
for message in stable
)
def _replaced_system_tail_delta(
current_messages: list[dict[str, Any]],
previous_original_messages: list[dict[str, Any]],
previous_forwarded_messages: list[dict[str, Any]],
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]] | None:
"""Cache-stable split for a request that replaced the previous trailing system turn.
The handlers record the model's reply after the request it answered, so the
replaced reminder is the last recorded message or sits right before that
reply. It is left out of the replayed prefix (it is not in this request);
every other recorded message must match ``current`` position for position.
"""
if not _transient_system_lineage_enabled() or len(previous_forwarded_messages) != len(
previous_original_messages
):
return None
current = _lineage_snapshot(_canonicalize_for_prefix_compare(current_messages))
previous = _lineage_snapshot(_canonicalize_for_prefix_compare(previous_original_messages))
if len(current) != len(current_messages) or len(previous) != len(previous_original_messages):
# The projection dropped a directive-only message; raw indices would shift.
return None
for request_len in (len(previous), len(previous) - 1):
reply = previous[request_len:]
if reply and not (isinstance(reply[0], dict) and reply[0].get("role") == "assistant"):
continue
reminder = request_len - 1
if (
_extends_past_system_tail(current, previous[:request_len])
and current[reminder : reminder + len(reply)] == reply
):
return (
copy.deepcopy(
previous_forwarded_messages[:reminder]
+ previous_forwarded_messages[request_len:]
),
copy.deepcopy(current_messages[reminder + len(reply) :]),
)
return None
def classify_history_relation(
current_messages: list[dict[str, Any]],
previous_messages: list[dict[str, Any]],
@@ -427,10 +502,18 @@ def extract_cache_stable_delta(
This is a COMPARISON + slice only: the returned prefix is the previously-forwarded
bytes verbatim and the delta is the raw appended messages — never a rebuild from the
canonical projection — so the projection dropping non-semantic fields is safe.
A previous request that ended in a ``role:"system"`` reminder the client has since
replaced also qualifies: its forwarded bytes are replayed without that reminder
(``HEADROOM_TRANSIENT_SYSTEM_LINEAGE=0`` disables this).
"""
if not previous_original_messages or previous_forwarded_messages is None:
return None
relation = classify_history_relation(current_messages, previous_original_messages)
if relation.kind == RELATION_DIVERGED:
return _replaced_system_tail_delta(
current_messages, previous_original_messages, previous_forwarded_messages
)
if relation.kind not in (RELATION_EXACT, RELATION_MESSAGE_APPEND):
# A same-message block append needs a block-level splice in
# ``overlay_cached_prefix``; slicing only whole messages would silently
@@ -1433,7 +1516,10 @@ class SessionTrackerStore:
where a large leading run and two-block identity suffix survive while
the middle tail is regenerated; all other rewrites start a fresh
lineage. This keeps #2671's stable cache boundary attached without
merging unrelated parallel sub-calls.
merging unrelated parallel sub-calls. When nothing else matches, an
Anthropic chain whose trailing ``role:"system"`` reminder the client
replaced is matched on the history before that reminder
(``HEADROOM_TRANSIENT_SYSTEM_LINEAGE=0`` disables this).
Byte-identical histories (templated fan-outs before they diverge)
intentionally share a tracker: their provider cache line is identical
too, so sharing is harmless.
@@ -1527,6 +1613,28 @@ class SessionTrackerStore:
len(rewrite_candidates) == 1 or rewrite_candidates[0][0] != rewrite_candidates[1][0]
):
best_key = rewrite_candidates[0][1]
elif (
not rewrite_candidates
and provider == "anthropic"
and _transient_system_lineage_enabled()
):
# Nothing matched as recorded. A chain whose only change is
# its replaced trailing system reminder is still this
# conversation; without it every such turn starts a cold
# tracker, the confirmed prefix is not replayed, and any
# recompressed history re-writes the provider cache. Only a
# unique longest chain qualifies, so equal-length siblings
# that differ in their reminder stay separate.
tail_candidates = [
(len(chain), key)
for key, chain in by_length
if self._lineage_affinities.get(key) == cache_affinity
and _extends_past_system_tail(snap, chain)
]
if tail_candidates and (
len(tail_candidates) == 1 or tail_candidates[0][0] != tail_candidates[1][0]
):
best_key = tail_candidates[0][1]
if best_key is None:
cap = self._default_config.max_lineages_per_session
+145
View File
@@ -13,6 +13,8 @@ from headroom.cache.prefix_tracker import (
PrefixCacheTracker,
PrefixFreezeConfig,
SessionTrackerStore,
extract_cache_stable_delta,
overlay_cached_prefix,
)
@@ -775,6 +777,149 @@ class TestConversationLineageResolution:
fresh = store.resolve_tracker("sid", "anthropic", messages=self._history("A", 2))
assert fresh._turn_number == 0
@staticmethod
def _reminder_turns() -> tuple[list[dict], list[dict]]:
"""Claude Code shape: a trailing system reminder replaced by the next turn."""
history = [
{"role": "user", "content": "Inspect example.py."},
{"role": "assistant", "content": "Reading the file."},
{
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": "read-1",
"content": "original source line\n" * 400,
}
],
},
]
reminder = {"role": "system", "content": "Batch independent tools this turn."}
previous = [*history, reminder]
current = [
*history,
{"role": "assistant", "content": "Now run the checks."},
{"role": "user", "content": "The checks passed."},
dict(reminder),
]
return previous, current
def test_replaced_system_tail_keeps_confirmed_tracker(self, store):
"""The previous request ended in a system reminder the client swapped
for new turns. Its unchanged history must keep the tracker and its
confirmed frozen prefix instead of starting a cold lineage."""
previous, current = self._reminder_turns()
forwarded = [dict(m) for m in previous]
forwarded[2] = {**previous[2], "content": "previously cached compressed result"}
tracker = store.resolve_tracker("sid", "anthropic", previous, cache_affinity="a")
tracker.update_from_response(
cache_read_tokens=0,
cache_write_tokens=1_000_000,
messages=forwarded,
original_messages=previous,
message_token_counts=[100] * len(previous),
)
frozen = tracker.get_frozen_message_count()
assert frozen > 0
assert store.resolve_tracker("sid", "anthropic", current, cache_affinity="a") is tracker
assert tracker.get_frozen_message_count() == frozen
# The existing replay restores the unchanged prefix and stops at the
# replaced reminder; this turn's tail is forwarded as processed.
processed = [dict(m) for m in current]
processed[2] = {**current[2], "content": "a different compressed result"}
replayed = overlay_cached_prefix(
processed, current, previous, forwarded, confirmed_frozen_count=frozen
)
assert replayed[:3] == forwarded[:3]
assert replayed[3:] == processed[3:]
def test_moving_system_tail_records_full_snapshots(self, store):
"""The reminder moves and changes every turn; each turn stays on one
tracker and the store records the full current history."""
previous, current = self._reminder_turns()
tracker = store.resolve_tracker("sid", "anthropic", previous)
for i in range(4):
current[-1] = {"role": "system", "content": f"reminder {i}"}
assert store.resolve_tracker("sid", "anthropic", current) is tracker
assert store._lineages["sid"]["sid"][-1]["content"][0]["text"] == f"reminder {i}"
assert len(store._lineages["sid"]["sid"]) == len(current)
current = [
*current[:-1],
{"role": "assistant", "content": f"reply {i}"},
{"role": "user", "content": f"next {i}"},
current[-1],
]
def test_system_tail_fallback_respects_affinity_and_provider(self):
previous, current = self._reminder_turns()
for provider, affinity in (("anthropic", "other-tools"), ("openai", "a")):
store = SessionTrackerStore()
tracker = store.resolve_tracker("sid", provider, previous, cache_affinity="a")
resolved = store.resolve_tracker("sid", provider, current, cache_affinity=affinity)
assert resolved is not tracker
@pytest.mark.parametrize(
("index", "role"),
[(0, "user"), (0, "system"), (2, "system")],
ids=["changed-history", "changed-leading-system", "changed-historical-system"],
)
def test_system_tail_fallback_requires_unchanged_history(self, store, index, role):
"""Only the trailing reminder is set aside: any other change, including
a changed leading or historical system instruction, still diverges."""
previous, current = self._reminder_turns()
previous.insert(index, {"role": role, "content": "original"})
current.insert(index, {"role": role, "content": "changed"})
tracker = store.resolve_tracker("sid", "anthropic", previous)
assert store.resolve_tracker("sid", "anthropic", current) is not tracker
def test_swapped_final_instruction_is_not_a_continuation(self, store):
"""Same-length siblings that differ only in their final system message
stay separate: the new history must extend past the old reminder."""
first = [{"role": "user", "content": "task"}, {"role": "system", "content": "A"}]
second = [{"role": "user", "content": "task"}, {"role": "system", "content": "B"}]
tracker = store.resolve_tracker("sid", "anthropic", first)
assert store.resolve_tracker("sid", "anthropic", second) is not tracker
assert store.resolve_tracker("sid", "anthropic", first) is tracker
def test_ambiguous_system_tail_siblings_are_not_merged(self, store, monkeypatch):
"""Two lineages that differ only in their reminder (recorded here with
the fallback disabled) are an ambiguous match: start a fresh lineage
instead of guessing, and an exact match still wins."""
previous, current = self._reminder_turns()
sibling = [*previous[:-1], {"role": "system", "content": "other reminder"}]
monkeypatch.setenv("HEADROOM_TRANSIENT_SYSTEM_LINEAGE", "0")
tracker = store.resolve_tracker("sid", "anthropic", previous)
other = store.resolve_tracker("sid", "anthropic", sibling)
assert other is not tracker
monkeypatch.delenv("HEADROOM_TRANSIENT_SYSTEM_LINEAGE")
assert store.resolve_tracker("sid", "anthropic", current) not in (tracker, other)
assert store.resolve_tracker("sid", "anthropic", sibling) is other
def test_cache_delta_replays_prefix_past_replaced_system_tail(self, monkeypatch):
"""Cache mode's delta split leaves the replaced reminder out of the
replayed prefix, with or without the model reply the handlers record
after the request, and compresses everything after it as new."""
previous, current = self._reminder_turns()
reply = current[3]
forwarded = [*previous[:2], {**previous[2], "content": "compressed"}, previous[3], reply]
for recorded in ([*previous, reply], previous):
replay = forwarded[: len(recorded)]
prefix, delta = extract_cache_stable_delta(current, recorded, replay)
assert prefix == replay[:3] + replay[4:]
assert delta == current[len(prefix) :]
# A system message followed by a client turn is history, not a reminder.
changed = [*previous, {"role": "user", "content": "Continue."}]
dropped = [*previous[:3], changed[4], *current[3:]]
assert extract_cache_stable_delta(dropped, changed, changed) is None
# The client's copy of the reply must match the recorded one.
other = [*previous, {"role": "assistant", "content": "Something else."}]
assert extract_cache_stable_delta(current, other, other) is None
monkeypatch.setenv("HEADROOM_TRANSIENT_SYSTEM_LINEAGE", "0")
assert extract_cache_stable_delta(current, [*previous, reply], forwarded) is None
def test_shared_session_id_is_not_rotated(self, store):
"""Composition guard: lineage resolution must not leak into session-id
derivation — beta stickiness and the compression cache key on it."""
+98
View File
@@ -235,3 +235,101 @@ def test_cache_mode_prefix_mismatch_passes_through_with_tag(monkeypatch):
assert forwarded[0]["content"][0]["content"] == original_text
# ...but now visibly tagged instead of silent.
assert handler.recorded_tags["passthrough_reason"] == "cache_mode_prefix_mismatch"
def _assistant(turn: int) -> dict:
return {
"role": "assistant",
"content": [{"type": "tool_use", "id": f"toolu_{turn}", "name": "read", "input": {}}],
}
def _tool_result(turn: int) -> dict:
return {
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": f"toolu_{turn}",
"content": f"verbose tool output {turn} " * 200,
}
],
}
def _compress_unfrozen(messages, model, frozen_message_count=0, **kwargs):
"""Like ``_fake_pipeline_apply`` but leaves frozen messages alone, as the
real pipeline does."""
result = _fake_pipeline_apply(messages[frozen_message_count:], model)
return TransformResult(
messages=list(messages[:frozen_message_count]) + result.messages,
tokens_before=result.tokens_before,
tokens_after=result.tokens_after,
transforms_applied=result.transforms_applied,
)
class _ConversationHandler(_DummyAnthropicHandler):
"""Real session tracker store; the upstream answers with the next tool call."""
def __init__(self) -> None:
from headroom.cache.prefix_tracker import SessionTrackerStore
super().__init__()
self.session_tracker_store = SessionTrackerStore()
self.anthropic_pipeline = SimpleNamespace(apply=MagicMock(side_effect=_compress_unfrozen))
self.forwarded: list[list[dict]] = []
async def _retry_request(self, method, url, headers, body, **_kwargs):
self.forwarded.append(json.loads(json.dumps(body["messages"])))
reply = {
"id": f"msg_{len(self.forwarded)}",
"type": "message",
"role": "assistant",
"content": _assistant(len(self.forwarded) + 1)["content"],
"usage": {"input_tokens": 1, "output_tokens": 1},
}
return SimpleNamespace(
status_code=200, headers={}, content=json.dumps(reply).encode(), json=lambda: reply
)
def test_cache_mode_replaced_system_tail_keeps_compressed_prefix(monkeypatch):
"""Claude Code replaces a trailing system reminder every turn. Cache mode
must replay last turn's compressed prefix without the old reminder and
compress only the new messages, not fall back to raw forwarding."""
import headroom.tokenizers as _tk
monkeypatch.setattr(_tk, "get_tokenizer", lambda model: _DummyTokenizer())
handler = _ConversationHandler()
model = "claude-fable-5"
history = [{"role": "user", "content": "Read the files."}, _assistant(1), _tool_result(1)]
turns = [
history,
[*history, _assistant(2), _tool_result(2), {"role": "system", "content": "Reminder A"}],
[
*history,
_assistant(2),
_tool_result(2),
_assistant(3),
_tool_result(3),
{"role": "system", "content": "Reminder B"},
],
]
for messages in turns:
request = _build_request({"model": model, "max_tokens": 16, "messages": messages})
anyio.run(handler.handle_anthropic_messages, request)
from headroom.cache.prefix_tracker import _strip_cache_control
# The proxy re-places cache_control markers every turn; compare the bytes under them.
second, third = (_strip_cache_control(m) for m in handler.forwarded[1:])
assert [m["role"] for m in third] == [m["role"] for m in turns[2]]
# Everything before the replaced reminder is last turn's forwarded bytes...
assert third[:5] == second[:5]
assert second[2]["content"][0]["content"] == _COMPRESSED_TEXT
assert second[4]["content"][0]["content"] == _COMPRESSED_TEXT
# ...the new tool result is compressed, and the current reminder is sent as is.
assert third[6]["content"][0]["content"] == _COMPRESSED_TEXT
assert third[-1] == {"role": "system", "content": "Reminder B"}
assert "passthrough_reason" not in handler.recorded_tags