Commit Graph
3 Commits
Author SHA1 Message Date
16eaec1e95 fix(cache): keep prefix lineage across a replaced system tail (#3934)
## Description


Claude Code can end a `/v1/messages` request with a temporary
`role:"system"` reminder and replace it on the next turn.
`SessionTrackerStore.resolve_tracker` records the full history, so the
stored chain ends in the old reminder and is never a prefix of the next
request: `_classify_history_canonical` returns `diverged` and the
request gets a fresh, empty tracker. The confirmed frozen prefix and the
previously forwarded messages are gone, the #3380 replay has nothing to
replay, and any recompressed history re-writes the provider prompt
cache. The reminder moves every turn, so this repeats for the whole
session.

This change lets lineage resolution set aside that one replaced trailing
system message, only when nothing matched as recorded. It changes
tracker selection only: forwarded bytes, the replay body, and breakpoint
placement are untouched, and the existing replay already stops at the
first changed message, so the new reminder goes out as the client sent
it.

Closes #3933

## Type of Change


- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cache/prefix_tracker.py`: new
`_extends_past_system_tail(current, previous)`. It is true when
`previous` ends in a `role:"system"` message, everything before that
message includes user/assistant history and is unchanged in `current`,
and `current` is longer than `previous`. Because `current` must be
longer, siblings that only swap their final instruction stay separate.
- `SessionTrackerStore.resolve_tracker`: after the existing
exact/append, block-append and block-rewrite passes find nothing (and
there are no rewrite candidates, so an ambiguous rewrite tie is not
overridden), Anthropic requests consider chains with the same cache
affinity that pass that check. Only a unique longest candidate is taken;
two equal-length candidates start a fresh lineage, as before. The full
current snapshot is stored as usual.
- `extract_cache_stable_delta` (cache mode's prefix/delta split): a
`diverged` request that replaced the previous trailing system reminder
is split too. Every recorded message before the reminder, plus the model
reply the handlers record after it, must match the current request
position for position (canonical compare, same rule via
`_extends_past_system_tail`). The previously forwarded bytes are
replayed without the old reminder, and everything after it is the delta
to compress. Before this, cache mode tagged these turns
`cache_mode_prefix_mismatch` and forwarded the new messages raw.
- `HEADROOM_TRANSIENT_SYSTEM_LINEAGE=0` (also `false`/`no`/`off`) turns
off both the lineage fallback and the delta split. It uses the same
parsing as `HEADROOM_STABLE_BOUNDARY_BREAKPOINT`.
- `tests/test_cache/test_prefix_tracker.py`:
- Tracker retention: the tracker and its confirmed frozen count are
kept, and the existing overlay restores the unchanged prefix.
- A reminder that moves and changes over several turns: full snapshots
are stored.
- Isolation: different cache affinity, OpenAI provider, changed earlier
history, changed leading system message, changed historical system
message, same-length siblings `[user: task, system: A]` vs `[user: task,
system: B]`, ambiguous equal-length siblings, exact-match precedence,
and the kill switch.
- The delta split: with and without the recorded reply, a system message
followed by a client turn is not treated as a reminder, a mismatched
reply is rejected, and the kill switch disables it.
- `tests/test_cache_mode_cold_start.py`: drives
`handle_anthropic_messages` in `mode="cache"` with the real
`SessionTrackerStore` over three turns. The first turn compresses a tool
result and later turns replace the reminder. The test asserts that the
third request replays the second request's compressed prefix
byte-for-byte (ignoring `cache_control`), compresses the new tool
result, and ends in the current reminder.

## Testing

- [x] Unit tests pass (`pytest`) - targeted files below, not the whole
suite
- [x] Linting passes (`ruff check .`) - on the changed files, plus `ruff
format --check`
- [x] Type checking passes (`mypy headroom`) - on
`headroom/cache/prefix_tracker.py`
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

New tests with the source at `main` (d5318ac2):

```text
$ python -m pytest tests/test_cache_mode_cold_start.py -q -p no:cacheprovider
FAILED tests/test_cache_mode_cold_start.py::test_cache_mode_replaced_system_tail_keeps_compressed_prefix
========================= 1 failed, 2 passed in 1.70s ==========================
```

With the lineage commit only (2c5af818 plus the length fix, before the
delta split), the cache-mode tests fail on the raw new tool result:

```text
E   AssertionError: assert 'verbose tool...ool output 3 ' == 'compressed tool output'
E   TypeError: cannot unpack non-iterable NoneType object
FAILED tests/test_cache_mode_cold_start.py::test_cache_mode_replaced_system_tail_keeps_compressed_prefix
FAILED tests/test_cache/test_prefix_tracker.py::TestConversationLineageResolution::test_cache_delta_replays_prefix_past_replaced_system_tail
========================= 2 failed, 73 passed in 2.83s =========================
```

The same-length sibling test fails at 2c5af818:

```text
FAILED tests/test_cache/test_prefix_tracker.py::TestConversationLineageResolution::test_swapped_final_instruction_is_not_a_continuation
========================= 1 failed, 70 passed in 0.75s =========================
```

With all three commits:

```text
$ python -m pytest tests/test_cache/test_prefix_tracker.py tests/test_cache_mode_cold_start.py \
    tests/test_cache_mode_delta_marker.py tests/test_observed_wire_shapes.py tests/test_proxy_handler_helpers.py \
    tests/test_issue_2671_block_growth_cache.py tests/test_proxy_anthropic_cache_stability.py \
    tests/test_proxy_openai_cache_stability.py -q -p no:cacheprovider
======================= 225 passed, 1 warning in 46.30s ========================

$ ruff check headroom/cache/prefix_tracker.py tests/test_cache/test_prefix_tracker.py tests/test_cache_mode_cold_start.py
All checks passed!
$ ruff format --check headroom/cache/prefix_tracker.py tests/test_cache/test_prefix_tracker.py tests/test_cache_mode_cold_start.py
3 files already formatted
$ mypy headroom/cache/prefix_tracker.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS 27.0.1, Python 3.11.13, this branch (5b0c20f2) vs
`main` d5318ac2. The proxy was started from each checkout with `python
-m headroom.proxy.server --port <free> --anthropic-api-url
http://127.0.0.1:<stub> --no-rate-limit`, once with
`HEADROOM_MODE=cache` (the default) and once with `HEADROOM_MODE=token`.
The `headroom proxy` CLI refuses to start in this dev venv because the
optional `mcp`, `magika` and `onnxruntime` extras are missing; the
server module itself runs. The upstream is a local stdlib HTTP stub that
records each forwarded body and answers with the next `tool_use`. It
returns Anthropic-shaped `usage` from a simulated prompt cache: the
whole forwarded prompt is cached; the next request reads the leading run
of messages that are byte-identical to the previous forwarded request
(ignoring `cache_control`) and writes the rest. Tokens are approximated
as serialized chars / 3. No real provider was contacted.
- Exact command / steps: `PROOF_MODE=cache|token proof.py <checkout>
<python> <label>` (script below) sends four `claude-fable-5` turns of a
paged tool loop. Each turn adds an assistant `tool_use` and a ~31 KB
JSON `tool_result`, and each request ends in a `role:"system"` reminder
whose text changes every turn. Controls: `-noreminder` sends the same
turns without reminders, and `-mixed` drops only the first turn's
reminder, which is how Claude Code interleaves them.
- Observed result: in cache mode every reminder turn on the branch
replays the previously forwarded history byte-identical and compresses
only the new tool result (simulated read/write per turn: main
10,598/10,581, 21,162/10,581, 31,726/10,580 with nothing compressed;
branch 10,598/3,106, 13,687/3,106, 16,776/3,105); in the mixed run main
re-writes the cache on the first reminder turn (76 read / 31,667 write)
and the branch reads 6,212 / writes 3,106; in token mode main re-writes
the history every reminder turn (read 76/281/486) and the branch reads
3,123/6,212/9,301, matching the no-reminder control.

Simulated cache read / write per turn, and the proxy's `PERF tok_after`
(tokens forwarded):

  | Mode, run | Turn 1 | Turn 2 | Turn 3 | Turn 4 |
  | --- | --- | --- | --- | --- |
| cache, main | 0 / 10,615 (10,886) | 10,598 / 10,581 (21,747) | 21,162
/ 10,581 (32,608) | 31,726 / 10,580 (43,469) |
| cache, branch before delta split (2c5af818) | 0 / 10,615 (10,886) |
10,598 / 10,581 (21,747) | 21,162 / 10,581 (32,608) | 31,726 / 10,580
(43,469) |
| cache, branch | 0 / 10,615 (10,886) | 10,598 / 3,106 (15,122) | 13,687
/ 3,106 (19,358) | 16,776 / 3,105 (23,594) |
| cache, branch, `HEADROOM_TRANSIENT_SYSTEM_LINEAGE=0` | 0 / 10,615
(10,886) | 10,598 / 10,581 (21,747) | 21,162 / 10,581 (32,608) | 31,726
/ 10,580 (43,469) |
| cache, main, no reminders (control) | 0 / 3,123 (4,253) | 3,123 /
3,089 (8,489) | 6,212 / 3,089 (12,725) | 9,301 / 3,088 (16,961) |
| cache mixed, main | 0 / 3,123 (4,253) | 3,123 / 3,106 (8,498) | 76 /
31,667 (32,608) | 31,726 / 10,580 (43,469) |
| cache mixed, branch before delta split (2c5af818) | 0 / 3,123 (4,253)
| 3,123 / 3,106 (8,498) | 6,212 / 10,581 (19,359) | 16,776 / 10,580
(30,220) |
| cache mixed, branch | 0 / 3,123 (4,253) | 3,123 / 3,106 (8,498) |
6,212 / 3,106 (12,734) | 9,301 / 3,105 (16,970) |
| token, main | 0 / 3,140 (4,262) | 76 / 3,271 (4,429) | 281 / 3,271
(4,595) | 486 / 3,270 (4,762) |
| token, branch | 0 / 3,140 (4,262) | 3,123 / 3,106 (8,498) | 6,212 /
3,106 (12,734) | 9,301 / 3,105 (16,970) |
| token, main, no reminders (control) | 0 / 3,123 (4,253) | 3,123 /
3,089 (8,489) | 6,212 / 3,089 (12,725) | 9,301 / 3,088 (16,961) |

  How to read it:
- **Cache mode, main:** for a session that always ends in a reminder,
the cache is stable but nothing is ever compressed. Each turn gets a
fresh tracker, and the cold-start branch freezes the whole request
because its last message is not a user turn.
- **Cache mode, mixed run on main:** the first reminder turn after a
compressed turn re-sends the compressed history raw and re-writes the
cache.
- **Lineage commit alone (2c5af818):** kept the history stable, because
the cached-prefix replay in `finalize_turn` restores it. New tool
results still went out raw (`cache_mode_prefix_mismatch`).
- **With the delta split:** they are compressed (31,572 -> 9,187 bytes
each), and the mixed run matches the no-reminder control.
- **Every forwarded request on the branch** ends in that turn's own
reminder text, and the old reminder is not replayed.
- **Turn 1 of the always-reminder cache run** is forwarded raw on both
main and the branch. A cold request whose last message is a system
message is frozen whole by `_strict_previous_turn_frozen_count`. This PR
does not change that.
- Downstream production evidence (Headroom Desktop on the 0.37.0 wheel,
`claude-fable-5`, token mode, provider-reported `usage`, no request
content): one long Claude Code session went 0 read / 490,996 write
(cold), then 490,996 read / 12,509 write on an 862-message request
ending in the reminder, then 7,259 read / 491,260 write on the next
864-message request, although its first 861 messages were unchanged. 27
such continuations in that window wrote 15,183,730 cache tokens and read
195,993. After the lineage fix was applied downstream, a live run of 13
requests (9 replaced the reminder) read 664,386 and wrote 49,646 cache
tokens (92.99% reuse).
- Not tested: no real Anthropic request or live provider cache billing
for this branch; details below.
- No real Anthropic request was made from this branch, so live provider
cache billing was not measured for this PR. The cache numbers above come
from the stub's simulation.
- The downstream numbers come from a vendored lineage patch on the
0.37.0 wheel in token mode. That patch has the same selection rules but
not this exact code, and it has no cache-mode delta split, which is new
here and has not run against a live provider.
  - Not exercised end to end:
- streaming responses (they record the reply the same way, but were not
driven here)
    - the Bedrock backend path
    - the `headroom proxy` CLI entry point with all extras installed
- models that relocate the reminder into the top-level `system` field.
There the reminder is part of the system segment and this change has no
effect.
- OpenAI paths are unchanged by design and are covered only by their
existing tests.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is a fix in lineage selection
and in cache mode's prefix/delta split, not a rollout feature in
`headroom/rollout.py`.
- Minimum rollout channel: n/a (applies on every channel).
- Stable/default behavior changed: yes, for Anthropic requests whose
previous request ended in a `role:"system"` message that the client
replaced:
- **Token mode:** they keep their existing prefix tracker, so the
confirmed prefix is replayed byte-stable instead of being recompressed.
That history is therefore no longer recompressed on those turns.
- **Cache mode (default):** the same request is split into the
previously forwarded prefix plus a compressed delta instead of being
forwarded with its new messages raw. Compression on those turns goes up,
cache reads stay at the replayed prefix, and a session that alternates
reminder and non-reminder turns no longer re-writes its cache on the
first reminder turn.
- **Unchanged:** OpenAI, conversations without a trailing system
message, siblings that only swap their final instruction, and every case
where an existing relation already matches.
- Kill switch / disable path: `HEADROOM_TRANSIENT_SYSTEM_LINEAGE=0` (or
`false`/`no`/`off`), read per request. It disables both the lineage
fallback and the cache-mode split. In the proof, the cache-mode run with
it set matches `main` exactly.
- Unsafe override required: no.
- Qualification impact: expect higher provider cache reads and lower
cache writes on Claude Code sessions that send trailing system reminders
in token mode, and more compression of new messages on those turns in
cache mode. This was measured here only against a stub; live provider
billing for this branch was not measured.
- Rollback path: set `HEADROOM_TRANSIENT_SYSTEM_LINEAGE=0` and restart
the proxy, or revert the three commits. No persisted state or schema
changes.

<details>
<summary>proof.py</summary>

```python
"""Boot the proxy (`python -m headroom.proxy.server`) from a given checkout
against a local stub Anthropic upstream and replay a Claude Code shaped
conversation (paged tool calls, ~31 KB JSON tool results) whose trailing
role:"system" reminder is replaced every turn.

Usage: PROOF_MODE=cache|token proof.py <checkout-dir> <python> <label>
Label suffix "-noreminder" drops the reminders (control); "-mixed" drops only
the first one. Reports, per turn, how many previously forwarded history
messages were forwarded again byte-identical (cache_control ignored), the
newest tool result's size before/after the proxy, and the stub's simulated
prompt-cache read/write. No real provider is contacted: the upstream is
127.0.0.1.
"""

import json
import os
import socket
import subprocess
import sys
import threading
import time
import urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

checkout, python, label = sys.argv[1], sys.argv[2], sys.argv[3]
received: list[dict] = []
usage: list[tuple[int, int]] = []


def strip(x):
    """Drop cache_control markers: the proxy re-places them every turn and the
    provider does not hash them into the cached prefix."""
    if isinstance(x, dict):
        return {k: strip(v) for k, v in x.items() if k != "cache_control"}
    if isinstance(x, list):
        return [strip(v) for v in x]
    return x


class Stub(BaseHTTPRequestHandler):
    def do_POST(self):
        body = json.loads(self.rfile.read(int(self.headers["content-length"])))
        received.append(body)
        # Minimal provider prompt-cache simulation: the whole forwarded prompt
        # is cached (client-style breakpoint on the newest message); this turn
        # reads the leading run of messages byte-identical to the previous
        # forwarded request, and writes the rest. Tokens ~= serialized chars / 3.
        def toks(x):
            return len(json.dumps(strip(x), sort_keys=True)) // 3

        msgs = body["messages"]
        total = toks(body.get("system")) + sum(toks(m) for m in msgs)
        read = 0
        if len(received) > 1 and strip(received[-2].get("system")) == strip(body.get("system")):
            prev = received[-2]["messages"]
            k = 0
            while k < min(len(prev), len(msgs)) and strip(prev[k]) == strip(msgs[k]):
                k += 1
            if k:
                read = toks(body.get("system")) + sum(toks(m) for m in msgs[:k])
        usage.append((read, total - read))
        payload = json.dumps(
            {
                "id": f"msg_{len(received)}",
                "type": "message",
                "role": "assistant",
                "model": body["model"],
                "content": assistant(len(received) + 1)["content"],
                "stop_reason": "tool_use",
                "stop_sequence": None,
                "usage": {
                    "input_tokens": 10,
                    "output_tokens": 2,
                    "cache_creation_input_tokens": total - read,
                    "cache_read_input_tokens": read,
                },
            }
        ).encode()
        self.send_response(200)
        self.send_header("content-type", "application/json")
        self.send_header("content-length", str(len(payload)))
        self.end_headers()
        self.wfile.write(payload)

    def log_message(self, *args):
        pass


def assistant(k: int) -> dict:
    return {"role": "assistant", "content": [
        {"type": "tool_use", "id": f"toolu_{k}", "name": "list_services", "input": {"page": k}}]}


def tool_result(k: int) -> dict:
    rows = [{"id": i, "name": f"service-{k}-{i}", "status": "ok" if (i + k) % 7 else "degraded",
             "latency_ms": 20 + (i * 37 + k) % 400, "region": ["eu", "us", "ap"][i % 3]}
            for i in range(300)]
    return {"role": "user", "content": [
        {"type": "tool_result", "tool_use_id": f"toolu_{k}", "content": json.dumps(rows)}]}


def free_port() -> int:
    with socket.socket() as s:
        s.bind(("127.0.0.1", 0))
        return s.getsockname()[1]


stub = ThreadingHTTPServer(("127.0.0.1", free_port()), Stub)
threading.Thread(target=stub.serve_forever, daemon=True).start()
port = free_port()
env = {
    **os.environ,
    "PYTHONPATH": checkout,
    "HEADROOM_BEACON": "off",
    "HEADROOM_MODE": os.environ.get("PROOF_MODE", "token"),
    "LITELLM_LOCAL_MODEL_COST_MAP": "True",
    "HEADROOM_DATA_DIR": os.path.join(os.path.dirname(__file__), f"data-{label}"),
}
env.pop("GROK_CLI_CHAT_PROXY_BASE_URL", None)
log = open(os.path.join(os.path.dirname(__file__), f"proxy-{label}.log"), "w")
proxy = subprocess.Popen(
    [
        python, "-m", "headroom.proxy.server", "--port", str(port),
        "--anthropic-api-url", f"http://127.0.0.1:{stub.server_address[1]}",
        "--no-rate-limit",
    ],
    env=env, stdout=log, stderr=subprocess.STDOUT,
)
try:
    for _ in range(240):
        try:
            urllib.request.urlopen(f"http://127.0.0.1:{port}/health", timeout=1)
            break
        except Exception:
            time.sleep(0.5)
    else:
        raise SystemExit("proxy did not come up")

    turns = []
    convo = [{"role": "user", "content": "Page through the services and find the degraded ones."}]
    for n in range(1, 5):
        convo = convo + [assistant(n), tool_result(n)]
        turns.append(convo + [{"role": "system", "content": f"Reminder variant {n}."}])

    if label.endswith("noreminder"):
        # Control: the same conversation without the trailing system reminder.
        turns = [t[:-1] for t in turns]
    elif label.endswith("mixed"):
        # Claude Code sends reminders on some turns only: the first request
        # has none (so it is compressed and that form is cached), later ones do.
        turns[0] = turns[0][:-1]
    previous = None
    for n, messages in enumerate(turns, 1):
        req = urllib.request.Request(
            f"http://127.0.0.1:{port}/v1/messages",
            data=json.dumps({"model": "claude-fable-5", "max_tokens": 64,
                             "system": "You are a coding agent.",
                             "messages": messages}).encode(),
            headers={"content-type": "application/json", "x-api-key": "stub-key",
                     "anthropic-version": "2023-06-01"},
        )
        urllib.request.urlopen(req, timeout=120).read()
        forwarded = received[-1]["messages"]
        newest = max(i for i, m in enumerate(messages) if m["role"] == "user")
        fresh = (f"newest tool_result {len(json.dumps(messages[newest]))}->"
                 f"{len(json.dumps(forwarded[newest]))} bytes")
        cache = f"simulated cache read={usage[-1][0]} write={usage[-1][1]}"
        if previous is None:
            print(f"[{label}] turn {n}: cold; {fresh}; {cache}")
        else:
            # The previous turn's trailing system reminder is replaced this turn.
            limit = len(previous) - (previous[-1]["role"] == "system")
            same = 0
            while same < limit and strip(forwarded[same]) == strip(previous[same]):
                same += 1
            print(f"[{label}] turn {n}: {same}/{limit} previously forwarded history messages "
                  f"byte-identical; {fresh}; {cache}")
        previous = forwarded
    with open(os.path.join(os.path.dirname(__file__), f"forwarded-{label}.json"), "w") as f:
        json.dump(received, f, indent=1)
finally:
    proxy.terminate()
    proxy.wait(timeout=30)
    stub.shutdown()
```

</details>

## Review Readiness


- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist


- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)


N/A

## Additional Notes


- Headroom Desktop has shipped this as a downstream patch since
2026-09-07 (a wrapper around `resolve_tracker` with the same guards).
This PR expresses it inside `resolve_tracker` instead, without mutating
the lineage family.
- Why Anthropic only: the shape was observed from Claude Code on
`/v1/messages`; the OpenAI path keeps its current behaviour until there
is evidence of the same pattern there.
- Why the current history must extend past the old reminder (not just
equal the history before it): a request that drops the reminder and adds
nothing is not a continuation we have seen, and keeping the condition
strict keeps sibling retries apart.
- Documentation: the new env var is described in the `resolve_tracker`
docstring; I did not find a docs page listing the prefix-tracker env
vars (`HEADROOM_STABLE_BOUNDARY_BREAKPOINT` is not listed either). Happy
to add one if there is a preferred place.
- `prefix_tracker.py` changed in #3895 after the version the downstream
patch targeted; the bug reproduces unchanged on `main` d5318ac2.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3934"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-04 13:58:14 -07:00
trs-80andClaude Opus 5 6e712672cb fix(anthropic): stop the spurious compaction warning and drop dead handler code (#3678)
## Description

Three small cleanups in `headroom/proxy/handlers/anthropic.py`, found
while reading the handler end to end. One is a real bug, two are dead
code.

**Spurious "system prompt compaction FAILED" on every request.** With
`HEADROOM_SYSTEM_COMPACT` enabled and no `ContentRouter` in the
pipeline, the handler read `_sys_modified`, a local that is only
assigned on the router branch. The resulting `UnboundLocalError` was
caught by the surrounding `except Exception` and logged as a compaction
failure on every request, even though nothing had failed. The flag is
now only read on the branch that sets it.

**No-op OpenAI fallback stub.** `ProxyConfig.fallback_enabled` /
`fallback_provider` gated a block in the Anthropic error path whose body
was two comments and a log line saying "Attempting fallback to OpenAI".
No conversion or retry ever existed, and nothing else in `headroom/`,
`tests/` or `docs/` references either field. Both fields and the block
are gone.

**Redundant pre-upstream semaphore releases.**
`handle_anthropic_messages` wraps its body in a `try/finally` that calls
the idempotent `_finalize_pre_upstream()` on every exit. Six explicit
calls placed directly before early `return`s (413, invalid JSON,
message-count 400, cache hit, security 403, Bedrock error) therefore did
nothing. They are removed. The calls that deliberately release *before*
a long await (streaming and backend handoffs) are kept, since those
change timing rather than correctness.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/handlers/anthropic.py`: move the `if _sys_modified:`
bookkeeping inside the router branch that assigns it, so the compaction
success/skip path no longer raises `UnboundLocalError` into the failure
log.
- `headroom/proxy/handlers/anthropic.py`: delete the
`fallback_enabled`/`fallback_provider` block in the upstream-error path;
it contained no executable fallback.
- `headroom/proxy/models.py`: delete the two unused `ProxyConfig`
fields. No loader, CLI flag, env var or doc maps to them.
- `headroom/proxy/handlers/anthropic.py`: drop six `await
_finalize_pre_upstream()` calls that sit immediately before an early
`return` already covered by the handler's `finally`.
- `tests/test_system_compaction_handler.py`: new
`test_system_compaction_without_content_router_is_not_a_failure`, which
fails on `main` with `cannot access local variable '_sys_modified'` and
passes here.
- `tests/test_system_compaction_handler.py`: also
`test_system_compaction_with_content_router_records_the_transform`,
which stubs the router lookup and `compact_system_prompt` so the success
branch (transform tag + debug log) is exercised; closes the codecov
partial on the moved `if _sys_modified:` line.
- `tests/test_anthropic_pre_upstream_backpressure.py`: extend the
early-exit semaphore test to the three validation paths it did not cover
(413, invalid JSON, message-count 400), so removing the explicit
releases is proven safe rather than assumed.
- `tests/test_anthropic_stage_timings.py`,
`tests/test_cache_mode_cold_start.py`,
`tests/test_cold_start_fast_pass.py`: stop setting the removed config
fields.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q tests/test_anthropic_pre_upstream_backpressure.py tests/test_system_compaction_handler.py \
    tests/test_anthropic_stage_timings.py tests/test_cache_mode_cold_start.py tests/test_cold_start_fast_pass.py
41 passed, 2 warnings in 7.6s

# same files with the new test against upstream/main's handler:
FAILED tests/test_system_compaction_handler.py::test_system_compaction_without_content_router_is_not_a_failure
E   'system prompt compaction FAILED: cannot access local variable '_sys_modified' where it is not associated with a value'

$ ruff check headroom/proxy/handlers/anthropic.py headroom/proxy/models.py tests/test_anthropic_pre_upstream_backpressure.py tests/test_system_compaction_handler.py
All checks passed!

$ mypy headroom/proxy/handlers/anthropic.py headroom/proxy/models.py
Success: no issues found in 2 source files

$ grep -rn 'fallback_enabled\|fallback_provider' headroom/ tests/ docs/
(no output)
```

## Real Behavior Proof

- Environment: macOS 25.6, Python 3.12 venv with the repo's compiled
`headroom._core`, fresh worktree of `upstream/main` @ `67eb910e`, the
three commits cherry-picked on top.
- Exact command / steps: read `handle_anthropic_messages` from the
semaphore acquire through the outer `finally` and confirmed
`_finalize_pre_upstream()` is idempotent and runs in `finally` on every
exit; grepped every reference to `fallback_enabled` /
`fallback_provider`; ran the five touched test files on the branch and
again with the new compaction test against `main`'s handler; ran `ruff
check` and `mypy` on the touched source files.
- Observed result: 40 tests pass on the branch. The new compaction test
reproduces the spurious failure log on `main` and passes here. No
remaining references to the deleted config fields. Lint and type check
clean on the touched files.
- Not tested: no live proxy traffic. The compaction fix only changes
which branch reads a local, the fallback removal deletes code with no
behavior, and the semaphore change is covered by the extended early-exit
test rather than by observing a real backlog. `ruff format --check`
flags `anthropic.py` on `main` already; this PR does not reformat
unrelated lines.

## Runtime Rollout Safety

- Rollout-managed feature(s): `HEADROOM_SYSTEM_COMPACT` (env-gated, off
by default). No new gate is introduced.
- Minimum rollout channel: n/a. The fix only affects the log line and
`transforms_applied` bookkeeping when compaction is enabled and no
router is present; upstream requests were already unaffected.
- Stable/default behavior changed: no. With the env var unset, none of
the changed lines execute. `ProxyConfig(fallback_enabled=...)` would now
raise `TypeError`, but no code path, doc or config loader ever passed
it.
- Kill switch / disable path: unset `HEADROOM_SYSTEM_COMPACT` restores
the pre-existing behavior for the compaction path; the other two changes
remove dead code and have no runtime switch.
- Unsafe override required: no.
- Qualification impact: none. No compression, cache or provider behavior
changes.
- Rollback path: `git revert` of the three commits; they touch seven
files and no persisted state or schema.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

A fourth commit that unified the budget and rate-limit preflight across
all provider handlers was deliberately left out. Open PR #3579 already
enforces the spend budget on the OpenAI and Gemini routes with two
approvals, and #3375 targets the same issue #3374. The remaining
rate-limit-key unification will be proposed separately once #3579 lands.

Documentation checkbox is unticked because `fallback_enabled` /
`fallback_provider` appear in no documentation and no user-facing
behavior changed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-24 10:59:20 -05:00
aaeba0a319 fix(proxy): compress cache-mode cold starts and tag prefix-mismatch passthrough (#2365)
## Description

Since v0.31.0 shipped cache mode as the default (68676daa), users report
the dashboard showing "Optimization ENABLED" while every request
forwards with 0 tokens saved — Before Compression == After Compression
even on 100k-token requests (#2357).

Root cause: in the cache-mode branch of `handle_anthropic_messages`,
when `_extract_cache_stable_delta` returns `None` the handler silently
sets `optimized_messages = messages`. That single fall-through covers
two very different cases:

1. **Session cold start** — no previous turn recorded for the session.
Every fresh proxy session, including a resumed 100k-token Claude Code
transcript, was forwarded raw. Until session identity stabilized (#2193
helped), this could be *every* turn, i.e. compression literally never
ran.
2. **Mid-session prefix mismatch** — the client rewrote history. This
passthrough is intentional (replaying a rewritten transcript risks
per-turn cache busts) but was invisible: no tag, no log, so
`optimize:true` + 0 savings looked like a broken product.

This PR: (1) cold starts now run the same full-message compression as
non-cache modes — there is no provider cache prefix to protect yet, and
the compressed output is recorded as the forwarded messages so later
turns replay it byte-identically through the existing stable-delta path
(append-only cache safety preserved); (2) the mismatch passthrough is
kept but tagged `passthrough_reason=cache_mode_prefix_mismatch` and
logged, mirroring the existing `pre_upstream_backpressure` inline-tag
pattern.

Fixes #2357

## Type of Change

- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Documentation update
- [ ] Refactoring (no functional changes)

## Changes Made

- `headroom/proxy/handlers/anthropic.py` (cache-mode branch only): split
the `delta is None` fall-through. When
`prefix_tracker.get_last_original_messages()` is empty (cold start), run
the full pipeline via `_run_compression_in_executor` (same call shape as
the non-cache branch) and append a `cache_mode:cold_start_full`
transform marker. When a previous turn exists but the delta is `None`
(prefix mismatch), keep the conservative passthrough but set
`tags["passthrough_reason"] = "cache_mode_prefix_mismatch"` and log it.
`CompressionDecision` untouched — it is the frozen pre-pipeline gate;
inline tags are the established mechanism for mid-pipeline passthrough
reasons.
- `tests/test_cache_mode_cold_start.py` (new): handler-level tests using
the same dummy-handler harness as `tests/test_cold_start_fast_pass.py`,
with `mode="cache"`. Cold start → pipeline invoked once, compressed form
forwarded upstream, no passthrough tag. Prefix mismatch → pipeline not
invoked, original bytes forwarded unmodified, outcome tags carry
`passthrough_reason=cache_mode_prefix_mismatch`.

## Testing

- [x] Unit tests pass locally
- [x] Lint/format/type checks pass locally

```
$ python -m pytest tests/test_cache_mode_cold_start.py tests/test_cache_mode_delta_marker.py tests/test_cache_prefix_overlay.py tests/test_cache/test_prefix_tracker.py tests/test_token_headroom_mode.py tests/test_cold_start_fast_pass.py tests/test_anthropic_pre_upstream_backpressure.py tests/test_handler_outcome_tag_invariant.py -q
146 passed

$ ruff check headroom/proxy/handlers/anthropic.py tests/test_cache_mode_cold_start.py
All checks passed!
$ ruff format --check headroom/proxy/handlers/anthropic.py tests/test_cache_mode_cold_start.py
2 files already formatted
$ mypy headroom --ignore-missing-imports
(no error lines)
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13.11, this branch; real
`AnthropicHandlerMixin.handle_anthropic_messages` driven end-to-end
through a FastAPI `Request` with upstream stubbed at `_retry_request`
(harness identical to the existing
`tests/test_cold_start_fast_pass.py`), `mode="cache"`.
- Exact command / steps: `python -m pytest
tests/test_cache_mode_cold_start.py -q`, plus a manual run of the
mismatch scenario with `logging.basicConfig(level=INFO)`.
- Observed result: Cold start: `anthropic_pipeline.apply` invoked once
and the forwarded upstream body contains the compressed tool_result
content (previously: forwarded raw with zero pipeline invocations).
Mismatch: bytes forwarded unmodified, and the proxy log now emits
`[req-...] Compression skipped: reason=cache_mode_prefix_mismatch` with
the same reason present in `RequestOutcome.tags["passthrough_reason"]`
(previously: nothing).
- Not tested: a live multi-turn session against the real Anthropic API
measuring `cache_read_input_tokens` across turns (the byte-identical
replay contract the cold-start path relies on is the same one exercised
by the existing stable-delta tests in
`tests/test_cache_mode_delta_marker.py` and
`tests/test_cache_prefix_overlay.py`, all green).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
2026-08-11 23:55:11 -05:00