Files
headroom/tests/test_cache_mode_cold_start.py
16eaec1e95 fix(cache): keep prefix lineage across a replaced system tail (#3934)
## Description


Claude Code can end a `/v1/messages` request with a temporary
`role:"system"` reminder and replace it on the next turn.
`SessionTrackerStore.resolve_tracker` records the full history, so the
stored chain ends in the old reminder and is never a prefix of the next
request: `_classify_history_canonical` returns `diverged` and the
request gets a fresh, empty tracker. The confirmed frozen prefix and the
previously forwarded messages are gone, the #3380 replay has nothing to
replay, and any recompressed history re-writes the provider prompt
cache. The reminder moves every turn, so this repeats for the whole
session.

This change lets lineage resolution set aside that one replaced trailing
system message, only when nothing matched as recorded. It changes
tracker selection only: forwarded bytes, the replay body, and breakpoint
placement are untouched, and the existing replay already stops at the
first changed message, so the new reminder goes out as the client sent
it.

Closes #3933

## Type of Change


- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cache/prefix_tracker.py`: new
`_extends_past_system_tail(current, previous)`. It is true when
`previous` ends in a `role:"system"` message, everything before that
message includes user/assistant history and is unchanged in `current`,
and `current` is longer than `previous`. Because `current` must be
longer, siblings that only swap their final instruction stay separate.
- `SessionTrackerStore.resolve_tracker`: after the existing
exact/append, block-append and block-rewrite passes find nothing (and
there are no rewrite candidates, so an ambiguous rewrite tie is not
overridden), Anthropic requests consider chains with the same cache
affinity that pass that check. Only a unique longest candidate is taken;
two equal-length candidates start a fresh lineage, as before. The full
current snapshot is stored as usual.
- `extract_cache_stable_delta` (cache mode's prefix/delta split): a
`diverged` request that replaced the previous trailing system reminder
is split too. Every recorded message before the reminder, plus the model
reply the handlers record after it, must match the current request
position for position (canonical compare, same rule via
`_extends_past_system_tail`). The previously forwarded bytes are
replayed without the old reminder, and everything after it is the delta
to compress. Before this, cache mode tagged these turns
`cache_mode_prefix_mismatch` and forwarded the new messages raw.
- `HEADROOM_TRANSIENT_SYSTEM_LINEAGE=0` (also `false`/`no`/`off`) turns
off both the lineage fallback and the delta split. It uses the same
parsing as `HEADROOM_STABLE_BOUNDARY_BREAKPOINT`.
- `tests/test_cache/test_prefix_tracker.py`:
- Tracker retention: the tracker and its confirmed frozen count are
kept, and the existing overlay restores the unchanged prefix.
- A reminder that moves and changes over several turns: full snapshots
are stored.
- Isolation: different cache affinity, OpenAI provider, changed earlier
history, changed leading system message, changed historical system
message, same-length siblings `[user: task, system: A]` vs `[user: task,
system: B]`, ambiguous equal-length siblings, exact-match precedence,
and the kill switch.
- The delta split: with and without the recorded reply, a system message
followed by a client turn is not treated as a reminder, a mismatched
reply is rejected, and the kill switch disables it.
- `tests/test_cache_mode_cold_start.py`: drives
`handle_anthropic_messages` in `mode="cache"` with the real
`SessionTrackerStore` over three turns. The first turn compresses a tool
result and later turns replace the reminder. The test asserts that the
third request replays the second request's compressed prefix
byte-for-byte (ignoring `cache_control`), compresses the new tool
result, and ends in the current reminder.

## Testing

- [x] Unit tests pass (`pytest`) - targeted files below, not the whole
suite
- [x] Linting passes (`ruff check .`) - on the changed files, plus `ruff
format --check`
- [x] Type checking passes (`mypy headroom`) - on
`headroom/cache/prefix_tracker.py`
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

New tests with the source at `main` (d5318ac2):

```text
$ python -m pytest tests/test_cache_mode_cold_start.py -q -p no:cacheprovider
FAILED tests/test_cache_mode_cold_start.py::test_cache_mode_replaced_system_tail_keeps_compressed_prefix
========================= 1 failed, 2 passed in 1.70s ==========================
```

With the lineage commit only (2c5af818 plus the length fix, before the
delta split), the cache-mode tests fail on the raw new tool result:

```text
E   AssertionError: assert 'verbose tool...ool output 3 ' == 'compressed tool output'
E   TypeError: cannot unpack non-iterable NoneType object
FAILED tests/test_cache_mode_cold_start.py::test_cache_mode_replaced_system_tail_keeps_compressed_prefix
FAILED tests/test_cache/test_prefix_tracker.py::TestConversationLineageResolution::test_cache_delta_replays_prefix_past_replaced_system_tail
========================= 2 failed, 73 passed in 2.83s =========================
```

The same-length sibling test fails at 2c5af818:

```text
FAILED tests/test_cache/test_prefix_tracker.py::TestConversationLineageResolution::test_swapped_final_instruction_is_not_a_continuation
========================= 1 failed, 70 passed in 0.75s =========================
```

With all three commits:

```text
$ python -m pytest tests/test_cache/test_prefix_tracker.py tests/test_cache_mode_cold_start.py \
    tests/test_cache_mode_delta_marker.py tests/test_observed_wire_shapes.py tests/test_proxy_handler_helpers.py \
    tests/test_issue_2671_block_growth_cache.py tests/test_proxy_anthropic_cache_stability.py \
    tests/test_proxy_openai_cache_stability.py -q -p no:cacheprovider
======================= 225 passed, 1 warning in 46.30s ========================

$ ruff check headroom/cache/prefix_tracker.py tests/test_cache/test_prefix_tracker.py tests/test_cache_mode_cold_start.py
All checks passed!
$ ruff format --check headroom/cache/prefix_tracker.py tests/test_cache/test_prefix_tracker.py tests/test_cache_mode_cold_start.py
3 files already formatted
$ mypy headroom/cache/prefix_tracker.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS 27.0.1, Python 3.11.13, this branch (5b0c20f2) vs
`main` d5318ac2. The proxy was started from each checkout with `python
-m headroom.proxy.server --port <free> --anthropic-api-url
http://127.0.0.1:<stub> --no-rate-limit`, once with
`HEADROOM_MODE=cache` (the default) and once with `HEADROOM_MODE=token`.
The `headroom proxy` CLI refuses to start in this dev venv because the
optional `mcp`, `magika` and `onnxruntime` extras are missing; the
server module itself runs. The upstream is a local stdlib HTTP stub that
records each forwarded body and answers with the next `tool_use`. It
returns Anthropic-shaped `usage` from a simulated prompt cache: the
whole forwarded prompt is cached; the next request reads the leading run
of messages that are byte-identical to the previous forwarded request
(ignoring `cache_control`) and writes the rest. Tokens are approximated
as serialized chars / 3. No real provider was contacted.
- Exact command / steps: `PROOF_MODE=cache|token proof.py <checkout>
<python> <label>` (script below) sends four `claude-fable-5` turns of a
paged tool loop. Each turn adds an assistant `tool_use` and a ~31 KB
JSON `tool_result`, and each request ends in a `role:"system"` reminder
whose text changes every turn. Controls: `-noreminder` sends the same
turns without reminders, and `-mixed` drops only the first turn's
reminder, which is how Claude Code interleaves them.
- Observed result: in cache mode every reminder turn on the branch
replays the previously forwarded history byte-identical and compresses
only the new tool result (simulated read/write per turn: main
10,598/10,581, 21,162/10,581, 31,726/10,580 with nothing compressed;
branch 10,598/3,106, 13,687/3,106, 16,776/3,105); in the mixed run main
re-writes the cache on the first reminder turn (76 read / 31,667 write)
and the branch reads 6,212 / writes 3,106; in token mode main re-writes
the history every reminder turn (read 76/281/486) and the branch reads
3,123/6,212/9,301, matching the no-reminder control.

Simulated cache read / write per turn, and the proxy's `PERF tok_after`
(tokens forwarded):

  | Mode, run | Turn 1 | Turn 2 | Turn 3 | Turn 4 |
  | --- | --- | --- | --- | --- |
| cache, main | 0 / 10,615 (10,886) | 10,598 / 10,581 (21,747) | 21,162
/ 10,581 (32,608) | 31,726 / 10,580 (43,469) |
| cache, branch before delta split (2c5af818) | 0 / 10,615 (10,886) |
10,598 / 10,581 (21,747) | 21,162 / 10,581 (32,608) | 31,726 / 10,580
(43,469) |
| cache, branch | 0 / 10,615 (10,886) | 10,598 / 3,106 (15,122) | 13,687
/ 3,106 (19,358) | 16,776 / 3,105 (23,594) |
| cache, branch, `HEADROOM_TRANSIENT_SYSTEM_LINEAGE=0` | 0 / 10,615
(10,886) | 10,598 / 10,581 (21,747) | 21,162 / 10,581 (32,608) | 31,726
/ 10,580 (43,469) |
| cache, main, no reminders (control) | 0 / 3,123 (4,253) | 3,123 /
3,089 (8,489) | 6,212 / 3,089 (12,725) | 9,301 / 3,088 (16,961) |
| cache mixed, main | 0 / 3,123 (4,253) | 3,123 / 3,106 (8,498) | 76 /
31,667 (32,608) | 31,726 / 10,580 (43,469) |
| cache mixed, branch before delta split (2c5af818) | 0 / 3,123 (4,253)
| 3,123 / 3,106 (8,498) | 6,212 / 10,581 (19,359) | 16,776 / 10,580
(30,220) |
| cache mixed, branch | 0 / 3,123 (4,253) | 3,123 / 3,106 (8,498) |
6,212 / 3,106 (12,734) | 9,301 / 3,105 (16,970) |
| token, main | 0 / 3,140 (4,262) | 76 / 3,271 (4,429) | 281 / 3,271
(4,595) | 486 / 3,270 (4,762) |
| token, branch | 0 / 3,140 (4,262) | 3,123 / 3,106 (8,498) | 6,212 /
3,106 (12,734) | 9,301 / 3,105 (16,970) |
| token, main, no reminders (control) | 0 / 3,123 (4,253) | 3,123 /
3,089 (8,489) | 6,212 / 3,089 (12,725) | 9,301 / 3,088 (16,961) |

  How to read it:
- **Cache mode, main:** for a session that always ends in a reminder,
the cache is stable but nothing is ever compressed. Each turn gets a
fresh tracker, and the cold-start branch freezes the whole request
because its last message is not a user turn.
- **Cache mode, mixed run on main:** the first reminder turn after a
compressed turn re-sends the compressed history raw and re-writes the
cache.
- **Lineage commit alone (2c5af818):** kept the history stable, because
the cached-prefix replay in `finalize_turn` restores it. New tool
results still went out raw (`cache_mode_prefix_mismatch`).
- **With the delta split:** they are compressed (31,572 -> 9,187 bytes
each), and the mixed run matches the no-reminder control.
- **Every forwarded request on the branch** ends in that turn's own
reminder text, and the old reminder is not replayed.
- **Turn 1 of the always-reminder cache run** is forwarded raw on both
main and the branch. A cold request whose last message is a system
message is frozen whole by `_strict_previous_turn_frozen_count`. This PR
does not change that.
- Downstream production evidence (Headroom Desktop on the 0.37.0 wheel,
`claude-fable-5`, token mode, provider-reported `usage`, no request
content): one long Claude Code session went 0 read / 490,996 write
(cold), then 490,996 read / 12,509 write on an 862-message request
ending in the reminder, then 7,259 read / 491,260 write on the next
864-message request, although its first 861 messages were unchanged. 27
such continuations in that window wrote 15,183,730 cache tokens and read
195,993. After the lineage fix was applied downstream, a live run of 13
requests (9 replaced the reminder) read 664,386 and wrote 49,646 cache
tokens (92.99% reuse).
- Not tested: no real Anthropic request or live provider cache billing
for this branch; details below.
- No real Anthropic request was made from this branch, so live provider
cache billing was not measured for this PR. The cache numbers above come
from the stub's simulation.
- The downstream numbers come from a vendored lineage patch on the
0.37.0 wheel in token mode. That patch has the same selection rules but
not this exact code, and it has no cache-mode delta split, which is new
here and has not run against a live provider.
  - Not exercised end to end:
- streaming responses (they record the reply the same way, but were not
driven here)
    - the Bedrock backend path
    - the `headroom proxy` CLI entry point with all extras installed
- models that relocate the reminder into the top-level `system` field.
There the reminder is part of the system segment and this change has no
effect.
- OpenAI paths are unchanged by design and are covered only by their
existing tests.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is a fix in lineage selection
and in cache mode's prefix/delta split, not a rollout feature in
`headroom/rollout.py`.
- Minimum rollout channel: n/a (applies on every channel).
- Stable/default behavior changed: yes, for Anthropic requests whose
previous request ended in a `role:"system"` message that the client
replaced:
- **Token mode:** they keep their existing prefix tracker, so the
confirmed prefix is replayed byte-stable instead of being recompressed.
That history is therefore no longer recompressed on those turns.
- **Cache mode (default):** the same request is split into the
previously forwarded prefix plus a compressed delta instead of being
forwarded with its new messages raw. Compression on those turns goes up,
cache reads stay at the replayed prefix, and a session that alternates
reminder and non-reminder turns no longer re-writes its cache on the
first reminder turn.
- **Unchanged:** OpenAI, conversations without a trailing system
message, siblings that only swap their final instruction, and every case
where an existing relation already matches.
- Kill switch / disable path: `HEADROOM_TRANSIENT_SYSTEM_LINEAGE=0` (or
`false`/`no`/`off`), read per request. It disables both the lineage
fallback and the cache-mode split. In the proof, the cache-mode run with
it set matches `main` exactly.
- Unsafe override required: no.
- Qualification impact: expect higher provider cache reads and lower
cache writes on Claude Code sessions that send trailing system reminders
in token mode, and more compression of new messages on those turns in
cache mode. This was measured here only against a stub; live provider
billing for this branch was not measured.
- Rollback path: set `HEADROOM_TRANSIENT_SYSTEM_LINEAGE=0` and restart
the proxy, or revert the three commits. No persisted state or schema
changes.

<details>
<summary>proof.py</summary>

```python
"""Boot the proxy (`python -m headroom.proxy.server`) from a given checkout
against a local stub Anthropic upstream and replay a Claude Code shaped
conversation (paged tool calls, ~31 KB JSON tool results) whose trailing
role:"system" reminder is replaced every turn.

Usage: PROOF_MODE=cache|token proof.py <checkout-dir> <python> <label>
Label suffix "-noreminder" drops the reminders (control); "-mixed" drops only
the first one. Reports, per turn, how many previously forwarded history
messages were forwarded again byte-identical (cache_control ignored), the
newest tool result's size before/after the proxy, and the stub's simulated
prompt-cache read/write. No real provider is contacted: the upstream is
127.0.0.1.
"""

import json
import os
import socket
import subprocess
import sys
import threading
import time
import urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

checkout, python, label = sys.argv[1], sys.argv[2], sys.argv[3]
received: list[dict] = []
usage: list[tuple[int, int]] = []


def strip(x):
    """Drop cache_control markers: the proxy re-places them every turn and the
    provider does not hash them into the cached prefix."""
    if isinstance(x, dict):
        return {k: strip(v) for k, v in x.items() if k != "cache_control"}
    if isinstance(x, list):
        return [strip(v) for v in x]
    return x


class Stub(BaseHTTPRequestHandler):
    def do_POST(self):
        body = json.loads(self.rfile.read(int(self.headers["content-length"])))
        received.append(body)
        # Minimal provider prompt-cache simulation: the whole forwarded prompt
        # is cached (client-style breakpoint on the newest message); this turn
        # reads the leading run of messages byte-identical to the previous
        # forwarded request, and writes the rest. Tokens ~= serialized chars / 3.
        def toks(x):
            return len(json.dumps(strip(x), sort_keys=True)) // 3

        msgs = body["messages"]
        total = toks(body.get("system")) + sum(toks(m) for m in msgs)
        read = 0
        if len(received) > 1 and strip(received[-2].get("system")) == strip(body.get("system")):
            prev = received[-2]["messages"]
            k = 0
            while k < min(len(prev), len(msgs)) and strip(prev[k]) == strip(msgs[k]):
                k += 1
            if k:
                read = toks(body.get("system")) + sum(toks(m) for m in msgs[:k])
        usage.append((read, total - read))
        payload = json.dumps(
            {
                "id": f"msg_{len(received)}",
                "type": "message",
                "role": "assistant",
                "model": body["model"],
                "content": assistant(len(received) + 1)["content"],
                "stop_reason": "tool_use",
                "stop_sequence": None,
                "usage": {
                    "input_tokens": 10,
                    "output_tokens": 2,
                    "cache_creation_input_tokens": total - read,
                    "cache_read_input_tokens": read,
                },
            }
        ).encode()
        self.send_response(200)
        self.send_header("content-type", "application/json")
        self.send_header("content-length", str(len(payload)))
        self.end_headers()
        self.wfile.write(payload)

    def log_message(self, *args):
        pass


def assistant(k: int) -> dict:
    return {"role": "assistant", "content": [
        {"type": "tool_use", "id": f"toolu_{k}", "name": "list_services", "input": {"page": k}}]}


def tool_result(k: int) -> dict:
    rows = [{"id": i, "name": f"service-{k}-{i}", "status": "ok" if (i + k) % 7 else "degraded",
             "latency_ms": 20 + (i * 37 + k) % 400, "region": ["eu", "us", "ap"][i % 3]}
            for i in range(300)]
    return {"role": "user", "content": [
        {"type": "tool_result", "tool_use_id": f"toolu_{k}", "content": json.dumps(rows)}]}


def free_port() -> int:
    with socket.socket() as s:
        s.bind(("127.0.0.1", 0))
        return s.getsockname()[1]


stub = ThreadingHTTPServer(("127.0.0.1", free_port()), Stub)
threading.Thread(target=stub.serve_forever, daemon=True).start()
port = free_port()
env = {
    **os.environ,
    "PYTHONPATH": checkout,
    "HEADROOM_BEACON": "off",
    "HEADROOM_MODE": os.environ.get("PROOF_MODE", "token"),
    "LITELLM_LOCAL_MODEL_COST_MAP": "True",
    "HEADROOM_DATA_DIR": os.path.join(os.path.dirname(__file__), f"data-{label}"),
}
env.pop("GROK_CLI_CHAT_PROXY_BASE_URL", None)
log = open(os.path.join(os.path.dirname(__file__), f"proxy-{label}.log"), "w")
proxy = subprocess.Popen(
    [
        python, "-m", "headroom.proxy.server", "--port", str(port),
        "--anthropic-api-url", f"http://127.0.0.1:{stub.server_address[1]}",
        "--no-rate-limit",
    ],
    env=env, stdout=log, stderr=subprocess.STDOUT,
)
try:
    for _ in range(240):
        try:
            urllib.request.urlopen(f"http://127.0.0.1:{port}/health", timeout=1)
            break
        except Exception:
            time.sleep(0.5)
    else:
        raise SystemExit("proxy did not come up")

    turns = []
    convo = [{"role": "user", "content": "Page through the services and find the degraded ones."}]
    for n in range(1, 5):
        convo = convo + [assistant(n), tool_result(n)]
        turns.append(convo + [{"role": "system", "content": f"Reminder variant {n}."}])

    if label.endswith("noreminder"):
        # Control: the same conversation without the trailing system reminder.
        turns = [t[:-1] for t in turns]
    elif label.endswith("mixed"):
        # Claude Code sends reminders on some turns only: the first request
        # has none (so it is compressed and that form is cached), later ones do.
        turns[0] = turns[0][:-1]
    previous = None
    for n, messages in enumerate(turns, 1):
        req = urllib.request.Request(
            f"http://127.0.0.1:{port}/v1/messages",
            data=json.dumps({"model": "claude-fable-5", "max_tokens": 64,
                             "system": "You are a coding agent.",
                             "messages": messages}).encode(),
            headers={"content-type": "application/json", "x-api-key": "stub-key",
                     "anthropic-version": "2023-06-01"},
        )
        urllib.request.urlopen(req, timeout=120).read()
        forwarded = received[-1]["messages"]
        newest = max(i for i, m in enumerate(messages) if m["role"] == "user")
        fresh = (f"newest tool_result {len(json.dumps(messages[newest]))}->"
                 f"{len(json.dumps(forwarded[newest]))} bytes")
        cache = f"simulated cache read={usage[-1][0]} write={usage[-1][1]}"
        if previous is None:
            print(f"[{label}] turn {n}: cold; {fresh}; {cache}")
        else:
            # The previous turn's trailing system reminder is replaced this turn.
            limit = len(previous) - (previous[-1]["role"] == "system")
            same = 0
            while same < limit and strip(forwarded[same]) == strip(previous[same]):
                same += 1
            print(f"[{label}] turn {n}: {same}/{limit} previously forwarded history messages "
                  f"byte-identical; {fresh}; {cache}")
        previous = forwarded
    with open(os.path.join(os.path.dirname(__file__), f"forwarded-{label}.json"), "w") as f:
        json.dump(received, f, indent=1)
finally:
    proxy.terminate()
    proxy.wait(timeout=30)
    stub.shutdown()
```

</details>

## Review Readiness


- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist


- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)


N/A

## Additional Notes


- Headroom Desktop has shipped this as a downstream patch since
2026-09-07 (a wrapper around `resolve_tracker` with the same guards).
This PR expresses it inside `resolve_tracker` instead, without mutating
the lineage family.
- Why Anthropic only: the shape was observed from Claude Code on
`/v1/messages`; the OpenAI path keeps its current behaviour until there
is evidence of the same pattern there.
- Why the current history must extend past the old reminder (not just
equal the history before it): a request that drops the reminder and adds
nothing is not a continuation we have seen, and keeping the condition
strict keeps sibling retries apart.
- Documentation: the new env var is described in the `resolve_tracker`
docstring; I did not find a docs page listing the prefix-tracker env
vars (`HEADROOM_STABLE_BOUNDARY_BREAKPOINT` is not listed either). Happy
to add one if there is a preferred place.
- `prefix_tracker.py` changed in #3895 after the version the downstream
patch targeted; the bug reproduces unchanged on `main` d5318ac2.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3934"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-04 13:58:14 -07:00

336 lines
12 KiB
Python

"""Issue #2357: cache mode must not silently forward everything uncompressed.
Cache mode compresses only the inter-turn delta against the previously
forwarded prefix. Before this fix, both delta-miss cases fell through to a
silent passthrough:
- session cold start (no previous turn recorded) — including a resumed
100k-token transcript, so a fresh proxy never compressed anything while the
dashboard kept reporting "optimization enabled";
- mid-session prefix mismatch (client rewrote history) — an intentional
passthrough, but with no tag or log explaining the 0 savings.
Now a cold start runs the same full-message compression as non-cache modes
(there is no provider cache prefix to protect yet), and the mismatch
passthrough is tagged with ``passthrough_reason=cache_mode_prefix_mismatch``.
"""
from __future__ import annotations
import json
from types import SimpleNamespace
from unittest.mock import MagicMock
import anyio
from fastapi import Request
from headroom.config import TransformResult
from headroom.proxy.handlers.anthropic import AnthropicHandlerMixin
from headroom.proxy.models import ProxyConfig
_COMPRESSED_TEXT = "compressed tool output"
class _DummyTokenizer:
def count_messages(self, messages) -> int:
return json.dumps(messages).count(" ") + 1
def count_text(self, text: str) -> int:
return max(1, text.count(" ") + 1)
class _DummyMetrics:
async def record_request(self, **kwargs):
return None
async def record_stage_timings(self, path, timings):
return None
async def record_failed(self, **kwargs):
return None
def record_compression_failed(self, reason: str) -> None:
return None
async def record_rate_limited(self, **kwargs):
return None
class _ResponseStub:
status_code = 200
headers: dict[str, str] = {}
content = b'{"id":"msg_1","type":"message","role":"assistant","content":[],"usage":{"input_tokens":1,"output_tokens":1}}'
def json(self):
return {
"id": "msg_1",
"type": "message",
"role": "assistant",
"content": [],
"usage": {"input_tokens": 1, "output_tokens": 1},
}
def _fake_pipeline_apply(messages, model, **kwargs):
compressed = []
for msg in messages:
new = dict(msg)
if msg.get("role") == "user" and isinstance(msg.get("content"), list):
new["content"] = [
{**part, "content": _COMPRESSED_TEXT}
if isinstance(part, dict) and part.get("type") == "tool_result"
else part
for part in msg["content"]
]
compressed.append(new)
return TransformResult(
messages=compressed,
tokens_before=1000,
tokens_after=100,
transforms_applied=["read_lifecycle:stale:test.py"],
)
class _DummyAnthropicHandler(AnthropicHandlerMixin):
ANTHROPIC_API_URL = "https://api.anthropic.com"
def __init__(self, previous_messages: list | None = None) -> None:
self.rate_limiter = None
self.metrics = _DummyMetrics()
self.config = ProxyConfig(
optimize=True,
image_optimize=False,
retry_max_attempts=1,
retry_base_delay_ms=1,
retry_max_delay_ms=1,
connect_timeout_seconds=10,
mode="cache",
cache_enabled=False,
rate_limit_enabled=False,
prefix_freeze_enabled=False,
memory_enabled=False,
)
self.usage_reporter = None
self.anthropic_provider = SimpleNamespace(get_context_limit=lambda model: 200_000)
self.anthropic_pipeline = SimpleNamespace(apply=MagicMock(side_effect=_fake_pipeline_apply))
self.anthropic_backend = None
self.cost_tracker = None
self.memory_handler = None
self.cache = None
self.security = None
self.ccr_context_tracker = None
self.ccr_injector = None
self.ccr_response_handler = None
self.ccr_feedback = None
self.ccr_batch_processor = None
self.ccr_mcp_server = None
self.traffic_learner = None
self.tool_injector = None
self.read_lifecycle_manager = None
self.logger = SimpleNamespace(log=lambda *a, **k: None)
self.request_logger = self.logger
self.usage_observer = None
self.image_compressor = None
prev = previous_messages or []
tracker = MagicMock()
tracker.get_frozen_message_count.return_value = 0
tracker.get_last_original_messages.return_value = prev
tracker.get_last_forwarded_messages.return_value = prev
tracker._cached_token_count = 0
tracker.classify_cache_miss.return_value = SimpleNamespace(is_miss=False)
self.session_tracker_store = SimpleNamespace(
compute_session_id=lambda *a, **k: "sess-1",
get_or_create=lambda *a, **k: tracker,
resolve_tracker=lambda *a, **k: tracker,
)
self._background_compression_enabled = False
self.recorded_tags: dict = {}
async def _record_request_outcome(self, outcome) -> None:
self.recorded_tags = dict(outcome.tags or {})
async def _run_compression_in_executor(self, fn, timeout):
return fn()
async def _next_request_id(self) -> str:
return "req-cache-cold-start-test"
async def _retry_request(self, method, url, headers, body, **_kwargs):
self.captured_body = body
return _ResponseStub()
def _build_request(body: dict) -> Request:
payload = json.dumps(body).encode("utf-8")
async def receive():
return {"type": "http.request", "body": payload, "more_body": False}
scope = {
"type": "http",
"asgi": {"version": "3.0"},
"http_version": "1.1",
"method": "POST",
"scheme": "https",
"path": "/v1/messages",
"raw_path": b"/v1/messages",
"query_string": b"",
"headers": [(b"authorization", b"Bearer sk-ant-api-test")],
"client": ("127.0.0.1", 12345),
"server": ("testserver", 443),
}
return Request(scope, receive)
_TOOL_RESULT_MESSAGE = {
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": "toolu_1",
"content": "verbose stale tool output " * 200,
}
],
}
def test_cache_mode_cold_start_compresses_full_request(monkeypatch):
import headroom.tokenizers as _tk
monkeypatch.setattr(_tk, "get_tokenizer", lambda model: _DummyTokenizer())
handler = _DummyAnthropicHandler(previous_messages=[])
request = _build_request(
{"model": "claude-3-5-sonnet-latest", "messages": [_TOOL_RESULT_MESSAGE]}
)
anyio.run(handler.handle_anthropic_messages, request)
# The full pipeline ran and its output was forwarded upstream.
assert handler.anthropic_pipeline.apply.call_count == 1
forwarded = handler.captured_body["messages"]
assert forwarded[0]["content"][0]["content"] == _COMPRESSED_TEXT
# No passthrough tag: compression genuinely ran.
assert "passthrough_reason" not in handler.recorded_tags
def test_cache_mode_prefix_mismatch_passes_through_with_tag(monkeypatch):
import headroom.tokenizers as _tk
monkeypatch.setattr(_tk, "get_tokenizer", lambda model: _DummyTokenizer())
# A previous turn exists but is NOT a prefix of the current request.
previous = [{"role": "user", "content": [{"type": "text", "text": "totally different"}]}]
handler = _DummyAnthropicHandler(previous_messages=previous)
original_text = "verbose stale tool output " * 200
request = _build_request(
{"model": "claude-3-5-sonnet-latest", "messages": [_TOOL_RESULT_MESSAGE]}
)
anyio.run(handler.handle_anthropic_messages, request)
# Conservative passthrough preserved: bytes forwarded unmodified...
assert handler.anthropic_pipeline.apply.call_count == 0
forwarded = handler.captured_body["messages"]
assert forwarded[0]["content"][0]["content"] == original_text
# ...but now visibly tagged instead of silent.
assert handler.recorded_tags["passthrough_reason"] == "cache_mode_prefix_mismatch"
def _assistant(turn: int) -> dict:
return {
"role": "assistant",
"content": [{"type": "tool_use", "id": f"toolu_{turn}", "name": "read", "input": {}}],
}
def _tool_result(turn: int) -> dict:
return {
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": f"toolu_{turn}",
"content": f"verbose tool output {turn} " * 200,
}
],
}
def _compress_unfrozen(messages, model, frozen_message_count=0, **kwargs):
"""Like ``_fake_pipeline_apply`` but leaves frozen messages alone, as the
real pipeline does."""
result = _fake_pipeline_apply(messages[frozen_message_count:], model)
return TransformResult(
messages=list(messages[:frozen_message_count]) + result.messages,
tokens_before=result.tokens_before,
tokens_after=result.tokens_after,
transforms_applied=result.transforms_applied,
)
class _ConversationHandler(_DummyAnthropicHandler):
"""Real session tracker store; the upstream answers with the next tool call."""
def __init__(self) -> None:
from headroom.cache.prefix_tracker import SessionTrackerStore
super().__init__()
self.session_tracker_store = SessionTrackerStore()
self.anthropic_pipeline = SimpleNamespace(apply=MagicMock(side_effect=_compress_unfrozen))
self.forwarded: list[list[dict]] = []
async def _retry_request(self, method, url, headers, body, **_kwargs):
self.forwarded.append(json.loads(json.dumps(body["messages"])))
reply = {
"id": f"msg_{len(self.forwarded)}",
"type": "message",
"role": "assistant",
"content": _assistant(len(self.forwarded) + 1)["content"],
"usage": {"input_tokens": 1, "output_tokens": 1},
}
return SimpleNamespace(
status_code=200, headers={}, content=json.dumps(reply).encode(), json=lambda: reply
)
def test_cache_mode_replaced_system_tail_keeps_compressed_prefix(monkeypatch):
"""Claude Code replaces a trailing system reminder every turn. Cache mode
must replay last turn's compressed prefix without the old reminder and
compress only the new messages, not fall back to raw forwarding."""
import headroom.tokenizers as _tk
monkeypatch.setattr(_tk, "get_tokenizer", lambda model: _DummyTokenizer())
handler = _ConversationHandler()
model = "claude-fable-5"
history = [{"role": "user", "content": "Read the files."}, _assistant(1), _tool_result(1)]
turns = [
history,
[*history, _assistant(2), _tool_result(2), {"role": "system", "content": "Reminder A"}],
[
*history,
_assistant(2),
_tool_result(2),
_assistant(3),
_tool_result(3),
{"role": "system", "content": "Reminder B"},
],
]
for messages in turns:
request = _build_request({"model": model, "max_tokens": 16, "messages": messages})
anyio.run(handler.handle_anthropic_messages, request)
from headroom.cache.prefix_tracker import _strip_cache_control
# The proxy re-places cache_control markers every turn; compare the bytes under them.
second, third = (_strip_cache_control(m) for m in handler.forwarded[1:])
assert [m["role"] for m in third] == [m["role"] for m in turns[2]]
# Everything before the replaced reminder is last turn's forwarded bytes...
assert third[:5] == second[:5]
assert second[2]["content"][0]["content"] == _COMPRESSED_TEXT
assert second[4]["content"][0]["content"] == _COMPRESSED_TEXT
# ...the new tool result is compressed, and the current reminder is sent as is.
assert third[6]["content"][0]["content"] == _COMPRESSED_TEXT
assert third[-1] == {"role": "system", "content": "Reminder B"}
assert "passthrough_reason" not in handler.recorded_tags