Files
headroom/tests/test_openai_message_level_cache_control.py
35ca9b8356 fix(proxy): reconcile Chat content-block cache markers after prefix replay (#4333)
## Description

OpenCode V2 talking to Claude through an OpenAI-compatible gateway
(LiteLLM → Bedrock) marks
Chat **content blocks**: two system text blocks, the newest user or tool
text block, plus one tool
definition. After the Chat prefix replay (`_restore_frozen_prefix` /
`finalize_turn`), last turn's
block marker is forwarded again next to the client's moved one. The
client sends four markers, the
proxy forwards five, and the provider rejects the turn:

```text
A maximum of 4 blocks with cache_control may be provided. Found 5.
```

#3895 fixed the same accumulation for message-dict and `tool_calls`
markers, and its helper
explicitly left block markers alone. This PR extends
`mirror_client_message_cache_control` to
content blocks, so all three Chat marker holders follow the client's
current request.

Related: #3894 (same symptom, message-level variant), #4332 (separate
token-mode lost breakpoint).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `mirror_client_message_cache_control` now also reconciles
`content[].cache_control`:
- The client marks a block → copy the client's own value (`ttl` kept per
position).
- The client marks no block in that message → drop replayed block
markers.
This also covers OpenCode resending an older turn as a **string** while
the replay restores
    the block form it forwarded when that turn was newest.
- The client marks blocks but the block count no longer aligns → leave
the message unchanged.
    No index guessing.
- Only `cache_control` keys change; never creates a marker and never
moves one to another block.
- Why not reuse `normalize_message_cache_control`: in Chat the system
prompt lives in `messages`.
Its legacy fallback strips every block marker and re-places one, which
would also drop the
  client's system breakpoints.
- Why not rely on `enforce_cache_breakpoint_budget`: it only runs for
`provider == "anthropic"`.
  This traffic is OpenAI Chat to a gateway, so it never runs here.
- The insertion point is unchanged. The existing mirror call already
runs after both Chat replays
and before forwarding and the tracker snapshot, so cache and token mode
and streaming and
  buffered forwarding are all covered.
- Docstring and comment no longer claim OpenAI-compatible OpenCode never
marks blocks.

## Testing

- [x] Unit tests pass (`pytest`): see below
- [x] Linting passes (`ruff check .`, `ruff format --check`)
- [x] Type checking passes (`mypy headroom/cache/prefix_tracker.py`)
- [x] New tests added for new functionality
- [x] Manual testing performed

New: `tests/test_openai_block_cache_control_replay.py` (20 tests).
Without the fix, 17 fail.

- Helper cases:
  - moved user → tool breakpoint
  - byte-identical replayed content
  - mid-message positions with `ttl: 1h` and `5m`
  - client with no block markers (none created)
  - client with no markers at all
  - mixed block, message and `tool_calls` markers
  - block-count mismatch (left unchanged)
  - string client content vs replayed blocks
- literal `cache_control` inside tool text and arguments stays user data
  - idempotence
- Handler cases capture the real forwarded body through
`/v1/chat/completions`:
  - cache and token mode (token with frozen floor 2)
  - streaming and buffered
  - block-form and string-form older user turn
- the tracker snapshot records the reconciled layout, and turn 3 stays
bounded

### Test Output

```text
$ pytest tests/test_openai_block_cache_control_replay.py tests/test_openai_message_level_cache_control.py
29 passed in 5.19s

# same new file with the fix reverted
17 failed, 3 passed

# related cache / prefix / OpenAI / proxy / LiteLLM / compression-cache suites
1217 passed, 33 skipped

# full suite, before vs after: identical failure set except three tests that
# fail intermittently under -n 8 on both builds and pass in isolation on both.
# All remaining failures are local-environment only (no Rust toolchain,
# sandboxed network, missing optional extras).
41 failed, 16305 passed, 860 skipped, 14 errors

$ ruff check . && ruff format --check headroom tests     # clean
$ mypy headroom/cache/prefix_tracker.py                  # Success: no issues found
```

## Real Behavior Proof

- Environment: macOS arm64, Python 3.13, OpenCode 2.1.2 with the bundled
Headroom plugin → `headroom proxy --mode cache|token` → LiteLLM gateway
→ Bedrock Claude Haiku; synthetic data only.
- Exact command / steps: Ran the same OpenCode agent task (one tool
returning 120 synthetic JSON records, called 3 times in sequence)
through the proxy, once on installed 0.41.0 and once importing this
branch's Python source via `PYTHONPATH` (Rust crates unchanged since
0.41.0); an `httpx` hook in the proxy recorded only the `cache_control`
paths of each outbound body.
- Observed result: 0.41.0 forwards 5 markers on the second request and
the gateway returns 400 "Found 5"; with this branch, cache and token
mode forward at most 4 markers on every request and the agent returns
the exact expected answer after 3 tool calls.
- Not tested: Windows; Anthropic `/v1/messages` (unchanged path);
Responses API; clients other than OpenCode V2; live sessions longer than
4 requests; compression of cache-protected Chat text blocks.

### Details

| Build | Mode | Inbound markers per request | Outbound markers per
request | Result |
| --- | --- | --- | --- | --- |
| 0.41.0 | cache | 4, 4 | 4, **5** | 400 "Found 5", turn aborted |
| this PR | cache | 4, 4, 4, 4 | 4, 4, 4, 4 | exact answer after 3 tool
calls |
| this PR | token | 4, 4, 4, 4 | 4, 4, 4, 3 | exact answer after 3 tool
calls (see note) |

Loopback HTTP replay of the same shapes (4 steps, cache and token, with
and without a
provider-confirmed frozen floor):

- 0.41.0 forwards 4/5/6/7 markers.
- This branch forwards 4/4/4/4 in all four cases.

Paths and values match the client, and the non-marker message bytes are
unchanged.

The token-mode `3` also reproduces on unpatched 0.41.0 and is a separate
issue.
`CompressionCache._swap_tool_result_content` writes a cached compression
back as a string for
`role: "tool"`, which drops the client's block marker on the newest tool
result. That is a lost
breakpoint, not a 400. Not changed here; tracked in #4332.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: Chat Completions forwarding no longer
re-sends replayed
content-block `cache_control` markers the client did not send this turn.
- Kill switch / disable path: none (bug fix; output is the client's own
marker layout)
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines

Co-authored-by: Thomas Bartik <22518863+tbartik@users.noreply.github.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-03 22:57:17 -06:00

269 lines
11 KiB
Python

"""``cache_control`` outside content blocks must not accumulate across replayed turns.
Chat Completions clients that reach Anthropic models through a gateway (OpenCode
via ``@ai-sdk/openai-compatible``, then LiteLLM) put breakpoints on the message
dict and on an assistant's ``tool_calls`` entries, not on a content block, and
move them to the newest messages on every call.
The prefix replay (``finalize_turn`` / ``overlay_cached_prefix``) forwards earlier
turns' messages byte-identical, so the markers they carried back then came along
and piled up behind the client's current ones. Two markers from the client became
four and more on the wire, the gateway translated them into Anthropic blocks, and the
request failed with ``A maximum of 4 blocks with cache_control may be provided``.
"""
from __future__ import annotations
import copy
import json
from types import SimpleNamespace
import httpx
import pytest
pytest.importorskip("fastapi")
from fastapi.testclient import TestClient
from headroom.cache.prefix_tracker import mirror_client_message_cache_control
from headroom.proxy.server import ProxyConfig, create_app
CC = {"type": "ephemeral"}
def _marked(msg: dict, marker: dict | None = CC) -> dict:
return {**msg, "cache_control": dict(marker)} if marker else dict(msg)
def _message_markers(messages: list[dict]) -> list[int]:
return [i for i, m in enumerate(messages) if isinstance(m, dict) and "cache_control" in m]
def _all_markers(messages: list[dict]) -> list[str]:
"""Every marker outside content blocks: ``"i"`` on a message, ``"i.j"`` on a tool call."""
found = []
for i, m in enumerate(messages):
if "cache_control" in m:
found.append(str(i))
for j, call in enumerate(m.get("tool_calls") or []):
if "cache_control" in call:
found.append(f"{i}.{j}")
return found
# ── the helper ──────────────────────────────────────────────────────────────
def test_replayed_markers_are_dropped_and_the_clients_are_kept() -> None:
client = [
_marked({"role": "system", "content": "sys"}),
{"role": "user", "content": "q"},
{"role": "tool", "tool_call_id": "t3", "content": "old output"},
_marked({"role": "tool", "tool_call_id": "t5", "content": "new output"}),
]
# What a replay produced: last turn's markers on user and tool 3 came along.
replayed = [
_marked({"role": "system", "content": "sys"}),
_marked({"role": "user", "content": "q"}),
_marked({"role": "tool", "tool_call_id": "t3", "content": "old <compressed>"}),
{"role": "tool", "tool_call_id": "t5", "content": "new output"},
]
out = mirror_client_message_cache_control(replayed, client)
assert _message_markers(out) == [0, 3]
# Only the marker moves: the replayed (compressed) content is what the
# provider cached, and it must stay byte-identical.
assert [m["content"] for m in out] == [m["content"] for m in replayed]
def test_replayed_tool_call_markers_are_dropped_and_the_clients_are_kept() -> None:
def call(cid: str, marker: bool) -> dict:
c = {"id": cid, "type": "function", "function": {"name": "bash", "arguments": "{}"}}
return _marked(c) if marker else c
client = [
{"role": "assistant", "content": "", "tool_calls": [call("t3", False)]},
{"role": "assistant", "content": "", "tool_calls": [call("t5", True)]},
]
# Last step's newest call (t3) carried the marker; the replay brought it back.
replayed = [
{"role": "assistant", "content": "", "tool_calls": [call("t3", True)]},
{"role": "assistant", "content": "", "tool_calls": [call("t5", True)]},
]
out = mirror_client_message_cache_control(replayed, client)
assert _all_markers(out) == ["1.0"]
assert out[1] is replayed[1] # already right: left as-is
def test_tool_calls_of_different_length_are_left_alone() -> None:
c = {"id": "t", "type": "function", "function": {"name": "bash", "arguments": "{}"}}
client = [{"role": "assistant", "content": "", "tool_calls": [c]}]
messages = [{"role": "assistant", "content": "", "tool_calls": [_marked(c), _marked(c)]}]
assert mirror_client_message_cache_control(messages, client) is messages
def test_the_clients_marker_value_is_copied_verbatim() -> None:
ttl = {"type": "ephemeral", "ttl": "1h"}
client = [_marked({"role": "user", "content": "q"}, ttl)]
out = mirror_client_message_cache_control([_marked({"role": "user", "content": "q"})], client)
assert out[0]["cache_control"] == ttl
def test_nothing_to_do_returns_the_same_list() -> None:
client = [_marked({"role": "system", "content": "sys"}), {"role": "user", "content": "q"}]
messages = copy.deepcopy(client)
assert mirror_client_message_cache_control(messages, client) is messages
def test_block_markers_already_where_the_client_put_them_are_left_alone() -> None:
block = {"type": "text", "text": "x", "cache_control": CC}
client = [{"role": "user", "content": [dict(block)]}]
messages = [{"role": "user", "content": [dict(block)]}]
out = mirror_client_message_cache_control(messages, client)
assert out is messages
assert out[0]["content"][0]["cache_control"] == CC
def test_misaligned_lists_are_returned_unchanged() -> None:
messages = [
_marked({"role": "user", "content": "a"}),
_marked({"role": "user", "content": "b"}),
]
assert mirror_client_message_cache_control(messages, messages[:1]) is messages
assert mirror_client_message_cache_control(messages, None) is messages
# ── through the Chat Completions handler ────────────────────────────────────
BIG = "row," * 4000
class _Tracker:
"""Stands in for the session tracker with last turn already recorded."""
def __init__(self, previous_original: list[dict], previous_forwarded: list[dict]):
self._original = previous_original
self._forwarded = previous_forwarded
def get_frozen_message_count(self) -> int:
return 0
def get_last_original_messages(self): # noqa: ANN201
return copy.deepcopy(self._original)
def get_last_forwarded_messages(self): # noqa: ANN201
return copy.deepcopy(self._forwarded)
def update_from_response(self, **kwargs): # noqa: ANN003
return None
def _turns() -> tuple[list[dict], list[dict], list[dict]]:
"""The shape OpenCode sends to a Claude model, three steps into one turn.
Last step the client marked the system prompt, the newest tool call (t3) and
its result (tool 3), and headroom forwarded tool 3 compressed. This step the
client moved both markers to t5 and tool 5 and left t3 and tool 3 unmarked.
"""
system = {"role": "system", "content": "You are an agent. " * 50}
user = {"role": "user", "content": "run three commands"}
call3 = {
"role": "assistant",
"content": "",
"tool_calls": [
{"id": "t3", "type": "function", "function": {"name": "bash", "arguments": "{}"}}
],
}
tool3 = {"role": "tool", "tool_call_id": "t3", "content": BIG}
call5 = {
"role": "assistant",
"content": "",
"tool_calls": [
{"id": "t5", "type": "function", "function": {"name": "bash", "arguments": "{}"}}
],
}
tool5 = {"role": "tool", "tool_call_id": "t5", "content": "30"}
def with_marked_call(assistant: dict) -> dict:
return {**assistant, "tool_calls": [_marked(assistant["tool_calls"][0])]}
previous_original = [_marked(system), user, with_marked_call(call3), _marked(tool3)]
previous_forwarded = [
_marked(system),
user,
with_marked_call(call3),
_marked({**tool3, "content": "row,<compressed: 4000 rows>"}),
]
current = [_marked(system), user, call3, tool3, with_marked_call(call5), _marked(tool5)]
return previous_original, previous_forwarded, current
@pytest.mark.parametrize("mode", ["cache", "token"])
def test_forwarded_markers_follow_the_client_not_the_replay(mode: str) -> None:
previous_original, previous_forwarded, current = _turns()
captured: dict = {}
config = ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
ccr_inject_tool=False,
ccr_handle_responses=False,
ccr_context_tracking=False,
image_optimize=False,
)
with TestClient(create_app(config)) as client:
proxy = client.app.state.proxy
proxy.config.optimize = True
proxy.config.mode = mode
tracker = _Tracker(previous_original, previous_forwarded)
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: "s"
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: tracker
)
proxy.openai_pipeline.apply = lambda **kwargs: SimpleNamespace(
messages=kwargs["messages"],
transforms_applied=[],
timing={},
tokens_before=100,
tokens_after=100,
waste_signals=None,
)
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
captured["body"] = json.loads(body) if isinstance(body, (bytes, str)) else body
return httpx.Response(
200,
json={
"id": "c",
"choices": [
{
"index": 0,
"message": {"role": "assistant", "content": "ok"},
"finish_reason": "stop",
}
],
"usage": {"prompt_tokens": 100, "completion_tokens": 1, "total_tokens": 101},
},
)
proxy._retry_request = _fake_retry
response = client.post(
"/v1/chat/completions",
headers={"authorization": "Bearer test-key"},
json={"model": "claude-sonnet-4-6", "messages": current},
)
assert response.status_code == 200, response.text
forwarded = captured["body"]["messages"]
# Precondition: the replay was live — tool 3 went out compressed.
assert forwarded[3]["content"] != BIG
# The client marked the system prompt, t5 and tool 5; last step's markers on
# t3 and tool 3 must not ride along with the replayed bytes.
assert _all_markers(forwarded) == _all_markers(current) == ["0", "4.0", "5"]