fix(proxy): keep cache_control outside content blocks where the Chat Completions client put it (#3895)

## Description

Chat Completions clients that reach Anthropic models through a gateway
(OpenCode via `@ai-sdk/openai-compatible`, then LiteLLM) do not mark
content blocks. They mark the message dict and the entries of an
assistant's `tool_calls`, and move those markers to the newest messages
on every call. `handle_openai_chat` replays earlier turns' forwarded
messages byte-identical: `finalize_turn` in every mode, plus
`_restore_frozen_prefix` in cache mode. The markers those messages
carried back then come along with them.

`normalize_message_cache_control` exists for exactly this accumulation,
but it only handles block-level markers, and this handler does not call
it after its replays. A client sending three markers per step (system,
newest tool call, newest tool result) gets four forwarded, then six, and
the gateway fails with `A maximum of 4 blocks with cache_control may be
provided. Found 6`, which breaks every tool turn of three or more steps.
`enforce_cache_breakpoint_budget` does not catch it either:
`count_cache_breakpoints` does not count `tool_calls` markers, so it
sees four.

This adds `mirror_client_message_cache_control`. It applies the rule the
block-level normalizer already applies (the client's current request
decides where markers go) to the two places it does not see:

- on each message, and on each entry of its `tool_calls`, drop the
marker unless the client put one there in this request;
- where the client did, copy the client's own value;
- leave content untouched, so the replayed bytes the provider cached
stay identical.

The handler applies it once, after both replays.

Closes #3894

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cache/prefix_tracker.py`: new
`mirror_client_message_cache_control(messages, client_messages)` (and a
small `_mirror_marker` helper).
- It touches only markers on the message dict and on `tool_calls`
entries; block-level markers stay with
`normalize_message_cache_control`.
- It returns the input list unchanged when nothing moved. When the
message lists, or one message's `tool_calls`, are not index-aligned,
that part is left as-is: it does not guess.
- `headroom/proxy/handlers/openai.py`: call it in `handle_openai_chat`
after the cache-mode restore/replay block, before memory injection, with
`original_client_messages`.
- `tests/test_openai_message_level_cache_control.py`:
- 7 unit tests for the helper: stale message and `tool_calls` markers
dropped, the client's value (including `ttl`) copied verbatim, identity
when there is nothing to do, block markers ignored, misaligned messages
or `tool_calls` left unchanged;
- 1 handler test, parametrized over `cache` and `token`, that drives
`/v1/chat/completions` with the exact OpenCode shape (system, newest
tool call and newest tool result marked) and a tracker holding last
step's compressed forward.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q tests/test_openai_message_level_cache_control.py
9 passed

# the handler test on main, without the fix (both modes):
E   AssertionError: assert ['0', '2.0', '3', '4.0', '5'] == ['0', '4.0', '5']

$ pytest -q tests/test_proxy/*.py tests/test_*openai*.py tests/test_*prefix*.py tests/test_*cache*.py \
    --deselect tests/test_proxy/test_issue_3650_router_emits_visible_retrievable_sentinel.py
2 failed, 1455 passed, 5 skipped
# FAILED tests/test_proxy/test_cc_switch_reconciler.py::test_switching_provider_recaptures
# FAILED tests/test_proxy/test_cc_switch_reconciler.py::test_official_resets_a_captured_third_party_upstream
# Both fail identically on main without this change:
# `pytest tests/test_proxy/test_cc_switch_reconciler.py` → 2 failed, 14 passed.

$ ruff check <changed files> && ruff format --check <changed files>
All checks passed!
3 files already formatted
```

## Real Behavior Proof

- Environment: OpenCode 1.18.30 → `headroom proxy` 0.39.1 → LiteLLM
proxy → AWS Bedrock `bedrock-claude-sonnet-4.6`, in a Kubernetes pod,
with pass-through taps before and after the proxy counting every
`cache_control` (message dict, content blocks, `tool_calls`). This
change was applied to 0.39.1 (the two changed files dropped over the
installed package) and run as a second proxy in the same pod. Plus the
four requests captured from that session, replayed against a mock
upstream.
- Exact command / steps: a turn asking for `date`, `uname -r` and
`nproc` through the bash tool, one after the other: five requests, the
last three after a tool result.
- Observed result: stock 0.39.1 forwards 6 markers on the fourth request
where the client sent 3, and Bedrock answers 400; with this change it
forwards exactly the client's 3 and the whole turn completes, in both
cache and token mode. Details:

  | Build (live, Bedrock) | Request 4 forwarded (client sent 3) | Turn |
  |---|---|---|
  | stock 0.39.1, `cache` | 6 → 400 `Found 6` | fails |
| 0.39.1 + this change, `cache` | 3 (system, 4.tc0, tool#5) | all 5
requests 200, answer returned |
| 0.39.1 + this change, `token` | 3 | all 5 requests 200, answer
returned |

Replay of the captured requests, markers forwarded for requests 1 to 4:

  | Build | Forwarded (client sent 3 / 2 / 3 / 3) |
  |---|---|
  | 0.37.0 | 3 / 2 / 3 / 3 |
  | `main` (`0a2c80d5`) | 3 / 2 / **4** / **6** |
  | this branch | 3 / 2 / 3 / 3 |

- Not tested: this branch installed as a package (live runs used 0.39.1
with these two files patched in, because that is the version the
deployment ships); non-OpenCode Chat Completions clients.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a (stable)
- Stable/default behavior changed: yes, narrowly. On
`/v1/chat/completions`, `cache_control` on the message dict and on
`tool_calls` entries now matches the client's current request instead of
accumulating from replayed turns. Content and block-level markers are
unchanged.
- Kill switch / disable path: none new; revert the PR
- Unsafe override required: no
- Qualification impact: none expected. The provider keys its cache on
content, not on marker placement (the premise
`normalize_message_cache_control` already relies on), and the client's
own positions are the ones it would have sent without the proxy.
- Rollback path: revert this PR; no state, schema or config migration
involved

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- `mypy headroom` could not run in my environment: it stops in numpy's
stubs (`Type statement is only supported in Python 3.12`) before it
reaches this code.
- No documentation change: this restores the behaviour the existing
docstrings already describe for block-level markers.
- Follow-up, not in this PR: `count_cache_breakpoints` does not count
`tool_calls` markers, so the last-stop guard under-counts for these
clients (it saw four where six went out). Teaching it to count them also
means teaching the repair step to remove them, so I kept it separate.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
afernandez-stratio
2026-10-01 12:16:26 -05:00
committed by GitHub
co-authored by Claude Opus 5.5 JerrettDavis
parent 1982b3a773
commit a6d6c14cf7
3 changed files with 350 additions and 0 deletions
+71
View File
@@ -870,6 +870,77 @@ def normalize_message_cache_control(
return out if changed else messages
def mirror_client_message_cache_control(
messages: list[dict[str, Any]],
client_messages: list[dict[str, Any]] | None,
) -> list[dict[str, Any]]:
"""Keep ``cache_control`` outside content blocks exactly where the client put it.
OpenAI-compatible clients that talk to Anthropic models through a gateway
(OpenCode via ``@ai-sdk/openai-compatible``, then LiteLLM) do not mark
content blocks. They mark the MESSAGE dict — ``{"role": "tool", "content":
"...", "cache_control": {...}}`` — and, on an assistant turn, the entries of
its ``tool_calls`` list, which the gateway turns into ``tool_use`` blocks.
``normalize_message_cache_control`` only sees block markers
(``_client_marker_positions`` reads list content), so after
``overlay_cached_prefix`` / ``finalize_turn`` replay an earlier turn's
forwarded messages, these markers ride along untouched. A client that marks
three places per request (system, the newest tool call, the newest tool
result) ends up forwarding four, then five and more, and the gateway's
translation to Anthropic blocks crosses the limit of four (``A maximum of 4
blocks with cache_control may be provided. Found 6``).
The client's current request is the authority for these markers, the same
rule the block-level normalizer applies: on each message and on each of its
``tool_calls``, drop the marker unless the client put one there in this
request, in which case copy the client's own value. Content — and any
block-level marker in it — is left alone, so the replayed bytes the provider
cached stay byte-identical.
Index alignment is the invariant (the pipeline preserves message count and
each message's tool calls); where it does not hold, that message or list is
returned unchanged rather than guessing, and the breakpoint budget guard
still bounds the total.
"""
if not isinstance(client_messages, list) or len(client_messages) != len(messages):
return messages
changed = False
out: list[dict[str, Any]] = []
for msg, client in zip(messages, client_messages):
if not isinstance(msg, dict):
out.append(msg)
continue
new_msg, msg_changed = _mirror_marker(msg, client)
calls = new_msg.get("tool_calls")
client_calls = client.get("tool_calls") if isinstance(client, dict) else None
if (
isinstance(calls, list)
and isinstance(client_calls, list)
and len(calls) == len(client_calls)
):
mirrored = [_mirror_marker(c, cc) for c, cc in zip(calls, client_calls)]
if any(c for _, c in mirrored):
new_msg = {**new_msg, "tool_calls": [c for c, _ in mirrored]}
msg_changed = True
out.append(new_msg)
changed = changed or msg_changed
return out if changed else messages
def _mirror_marker(holder: Any, client_holder: Any) -> tuple[Any, bool]:
"""Give ``holder`` the client's own ``cache_control`` (or none); report change."""
if not isinstance(holder, dict):
return holder, False
client_marker = client_holder.get("cache_control") if isinstance(client_holder, dict) else None
if isinstance(client_marker, dict):
if holder.get("cache_control") == client_marker:
return holder, False
return {**holder, "cache_control": dict(client_marker)}, True
if "cache_control" in holder:
return {k: v for k, v in holder.items() if k != "cache_control"}, True
return holder, False
class PrefixCacheTracker:
"""Tracks provider prefix cache state across turns in a session.
+13
View File
@@ -4350,6 +4350,19 @@ class OpenAIHandlerMixin:
confirmed_frozen_count=_openai_confirmed_frozen,
).messages
# Both replays above forward earlier turns' messages byte-identical,
# including the cache_control each carried back then. Chat Completions
# clients mark the message dict and an assistant's tool_calls, which the
# block-level normalizer does not see, so without this the markers
# accumulate turn over turn and an Anthropic-backed gateway rejects the
# request (more than four breakpoints). Content is untouched: only the
# markers move.
from headroom.cache.prefix_tracker import mirror_client_message_cache_control
optimized_messages = mirror_client_message_cache_control(
optimized_messages, original_client_messages
)
# Memory: inject context and tools for OpenAI requests.
#
# PR-A3 follow-up to A2: memory context now routes exclusively to
@@ -0,0 +1,266 @@
"""``cache_control`` outside content blocks must not accumulate across replayed turns.
Chat Completions clients that reach Anthropic models through a gateway (OpenCode
via ``@ai-sdk/openai-compatible``, then LiteLLM) put breakpoints on the message
dict and on an assistant's ``tool_calls`` entries, not on a content block, and
move them to the newest messages on every call.
The prefix replay (``finalize_turn`` / ``overlay_cached_prefix``) forwards earlier
turns' messages byte-identical, so the markers they carried back then came along
and piled up behind the client's current ones. Two markers from the client became
four and more on the wire, the gateway translated them into Anthropic blocks, and the
request failed with ``A maximum of 4 blocks with cache_control may be provided``.
"""
from __future__ import annotations
import copy
import json
from types import SimpleNamespace
import httpx
import pytest
pytest.importorskip("fastapi")
from fastapi.testclient import TestClient
from headroom.cache.prefix_tracker import mirror_client_message_cache_control
from headroom.proxy.server import ProxyConfig, create_app
CC = {"type": "ephemeral"}
def _marked(msg: dict, marker: dict | None = CC) -> dict:
return {**msg, "cache_control": dict(marker)} if marker else dict(msg)
def _message_markers(messages: list[dict]) -> list[int]:
return [i for i, m in enumerate(messages) if isinstance(m, dict) and "cache_control" in m]
def _all_markers(messages: list[dict]) -> list[str]:
"""Every marker outside content blocks: ``"i"`` on a message, ``"i.j"`` on a tool call."""
found = []
for i, m in enumerate(messages):
if "cache_control" in m:
found.append(str(i))
for j, call in enumerate(m.get("tool_calls") or []):
if "cache_control" in call:
found.append(f"{i}.{j}")
return found
# ── the helper ──────────────────────────────────────────────────────────────
def test_replayed_markers_are_dropped_and_the_clients_are_kept() -> None:
client = [
_marked({"role": "system", "content": "sys"}),
{"role": "user", "content": "q"},
{"role": "tool", "tool_call_id": "t3", "content": "old output"},
_marked({"role": "tool", "tool_call_id": "t5", "content": "new output"}),
]
# What a replay produced: last turn's markers on user and tool 3 came along.
replayed = [
_marked({"role": "system", "content": "sys"}),
_marked({"role": "user", "content": "q"}),
_marked({"role": "tool", "tool_call_id": "t3", "content": "old <compressed>"}),
{"role": "tool", "tool_call_id": "t5", "content": "new output"},
]
out = mirror_client_message_cache_control(replayed, client)
assert _message_markers(out) == [0, 3]
# Only the marker moves: the replayed (compressed) content is what the
# provider cached, and it must stay byte-identical.
assert [m["content"] for m in out] == [m["content"] for m in replayed]
def test_replayed_tool_call_markers_are_dropped_and_the_clients_are_kept() -> None:
def call(cid: str, marker: bool) -> dict:
c = {"id": cid, "type": "function", "function": {"name": "bash", "arguments": "{}"}}
return _marked(c) if marker else c
client = [
{"role": "assistant", "content": "", "tool_calls": [call("t3", False)]},
{"role": "assistant", "content": "", "tool_calls": [call("t5", True)]},
]
# Last step's newest call (t3) carried the marker; the replay brought it back.
replayed = [
{"role": "assistant", "content": "", "tool_calls": [call("t3", True)]},
{"role": "assistant", "content": "", "tool_calls": [call("t5", True)]},
]
out = mirror_client_message_cache_control(replayed, client)
assert _all_markers(out) == ["1.0"]
assert out[1] is replayed[1] # already right: left as-is
def test_tool_calls_of_different_length_are_left_alone() -> None:
c = {"id": "t", "type": "function", "function": {"name": "bash", "arguments": "{}"}}
client = [{"role": "assistant", "content": "", "tool_calls": [c]}]
messages = [{"role": "assistant", "content": "", "tool_calls": [_marked(c), _marked(c)]}]
assert mirror_client_message_cache_control(messages, client) is messages
def test_the_clients_marker_value_is_copied_verbatim() -> None:
ttl = {"type": "ephemeral", "ttl": "1h"}
client = [_marked({"role": "user", "content": "q"}, ttl)]
out = mirror_client_message_cache_control([_marked({"role": "user", "content": "q"})], client)
assert out[0]["cache_control"] == ttl
def test_nothing_to_do_returns_the_same_list() -> None:
client = [_marked({"role": "system", "content": "sys"}), {"role": "user", "content": "q"}]
messages = copy.deepcopy(client)
assert mirror_client_message_cache_control(messages, client) is messages
def test_block_level_markers_are_left_to_the_block_normalizer() -> None:
block = {"type": "text", "text": "x", "cache_control": CC}
client = [{"role": "user", "content": [dict(block)]}]
messages = [{"role": "user", "content": [dict(block)]}]
out = mirror_client_message_cache_control(messages, client)
assert out is messages
assert out[0]["content"][0]["cache_control"] == CC
def test_misaligned_lists_are_returned_unchanged() -> None:
messages = [
_marked({"role": "user", "content": "a"}),
_marked({"role": "user", "content": "b"}),
]
assert mirror_client_message_cache_control(messages, messages[:1]) is messages
assert mirror_client_message_cache_control(messages, None) is messages
# ── through the Chat Completions handler ────────────────────────────────────
BIG = "row," * 4000
class _Tracker:
"""Stands in for the session tracker with last turn already recorded."""
def __init__(self, previous_original: list[dict], previous_forwarded: list[dict]):
self._original = previous_original
self._forwarded = previous_forwarded
def get_frozen_message_count(self) -> int:
return 0
def get_last_original_messages(self): # noqa: ANN201
return copy.deepcopy(self._original)
def get_last_forwarded_messages(self): # noqa: ANN201
return copy.deepcopy(self._forwarded)
def update_from_response(self, **kwargs): # noqa: ANN003
return None
def _turns() -> tuple[list[dict], list[dict], list[dict]]:
"""The shape OpenCode sends to a Claude model, three steps into one turn.
Last step the client marked the system prompt, the newest tool call (t3) and
its result (tool 3), and headroom forwarded tool 3 compressed. This step the
client moved both markers to t5 and tool 5 and left t3 and tool 3 unmarked.
"""
system = {"role": "system", "content": "You are an agent. " * 50}
user = {"role": "user", "content": "run three commands"}
call3 = {
"role": "assistant",
"content": "",
"tool_calls": [
{"id": "t3", "type": "function", "function": {"name": "bash", "arguments": "{}"}}
],
}
tool3 = {"role": "tool", "tool_call_id": "t3", "content": BIG}
call5 = {
"role": "assistant",
"content": "",
"tool_calls": [
{"id": "t5", "type": "function", "function": {"name": "bash", "arguments": "{}"}}
],
}
tool5 = {"role": "tool", "tool_call_id": "t5", "content": "30"}
def with_marked_call(assistant: dict) -> dict:
return {**assistant, "tool_calls": [_marked(assistant["tool_calls"][0])]}
previous_original = [_marked(system), user, with_marked_call(call3), _marked(tool3)]
previous_forwarded = [
_marked(system),
user,
with_marked_call(call3),
_marked({**tool3, "content": "row,<compressed: 4000 rows>"}),
]
current = [_marked(system), user, call3, tool3, with_marked_call(call5), _marked(tool5)]
return previous_original, previous_forwarded, current
@pytest.mark.parametrize("mode", ["cache", "token"])
def test_forwarded_markers_follow_the_client_not_the_replay(mode: str) -> None:
previous_original, previous_forwarded, current = _turns()
captured: dict = {}
config = ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
ccr_inject_tool=False,
ccr_handle_responses=False,
ccr_context_tracking=False,
image_optimize=False,
)
with TestClient(create_app(config)) as client:
proxy = client.app.state.proxy
proxy.config.optimize = True
proxy.config.mode = mode
tracker = _Tracker(previous_original, previous_forwarded)
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: "s"
proxy.session_tracker_store.get_or_create = lambda session_id, provider: tracker
proxy.openai_pipeline.apply = lambda **kwargs: SimpleNamespace(
messages=kwargs["messages"],
transforms_applied=[],
timing={},
tokens_before=100,
tokens_after=100,
waste_signals=None,
)
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
captured["body"] = json.loads(body) if isinstance(body, (bytes, str)) else body
return httpx.Response(
200,
json={
"id": "c",
"choices": [
{
"index": 0,
"message": {"role": "assistant", "content": "ok"},
"finish_reason": "stop",
}
],
"usage": {"prompt_tokens": 100, "completion_tokens": 1, "total_tokens": 101},
},
)
proxy._retry_request = _fake_retry
response = client.post(
"/v1/chat/completions",
headers={"authorization": "Bearer test-key"},
json={"model": "claude-sonnet-4-6", "messages": current},
)
assert response.status_code == 200, response.text
forwarded = captured["body"]["messages"]
# Precondition: the replay was live — tool 3 went out compressed.
assert forwarded[3]["content"] != BIG
# The client marked the system prompt, t5 and tool 5; last step's markers on
# t3 and tool 3 must not ride along with the replayed bytes.
assert _all_markers(forwarded) == _all_markers(current) == ["0", "4.0", "5"]