mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description OpenCode V2 talking to Claude through an OpenAI-compatible gateway (LiteLLM → Bedrock) marks Chat **content blocks**: two system text blocks, the newest user or tool text block, plus one tool definition. After the Chat prefix replay (`_restore_frozen_prefix` / `finalize_turn`), last turn's block marker is forwarded again next to the client's moved one. The client sends four markers, the proxy forwards five, and the provider rejects the turn: ```text A maximum of 4 blocks with cache_control may be provided. Found 5. ``` #3895 fixed the same accumulation for message-dict and `tool_calls` markers, and its helper explicitly left block markers alone. This PR extends `mirror_client_message_cache_control` to content blocks, so all three Chat marker holders follow the client's current request. Related: #3894 (same symptom, message-level variant), #4332 (separate token-mode lost breakpoint). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `mirror_client_message_cache_control` now also reconciles `content[].cache_control`: - The client marks a block → copy the client's own value (`ttl` kept per position). - The client marks no block in that message → drop replayed block markers. This also covers OpenCode resending an older turn as a **string** while the replay restores the block form it forwarded when that turn was newest. - The client marks blocks but the block count no longer aligns → leave the message unchanged. No index guessing. - Only `cache_control` keys change; never creates a marker and never moves one to another block. - Why not reuse `normalize_message_cache_control`: in Chat the system prompt lives in `messages`. Its legacy fallback strips every block marker and re-places one, which would also drop the client's system breakpoints. - Why not rely on `enforce_cache_breakpoint_budget`: it only runs for `provider == "anthropic"`. This traffic is OpenAI Chat to a gateway, so it never runs here. - The insertion point is unchanged. The existing mirror call already runs after both Chat replays and before forwarding and the tracker snapshot, so cache and token mode and streaming and buffered forwarding are all covered. - Docstring and comment no longer claim OpenAI-compatible OpenCode never marks blocks. ## Testing - [x] Unit tests pass (`pytest`): see below - [x] Linting passes (`ruff check .`, `ruff format --check`) - [x] Type checking passes (`mypy headroom/cache/prefix_tracker.py`) - [x] New tests added for new functionality - [x] Manual testing performed New: `tests/test_openai_block_cache_control_replay.py` (20 tests). Without the fix, 17 fail. - Helper cases: - moved user → tool breakpoint - byte-identical replayed content - mid-message positions with `ttl: 1h` and `5m` - client with no block markers (none created) - client with no markers at all - mixed block, message and `tool_calls` markers - block-count mismatch (left unchanged) - string client content vs replayed blocks - literal `cache_control` inside tool text and arguments stays user data - idempotence - Handler cases capture the real forwarded body through `/v1/chat/completions`: - cache and token mode (token with frozen floor 2) - streaming and buffered - block-form and string-form older user turn - the tracker snapshot records the reconciled layout, and turn 3 stays bounded ### Test Output ```text $ pytest tests/test_openai_block_cache_control_replay.py tests/test_openai_message_level_cache_control.py 29 passed in 5.19s # same new file with the fix reverted 17 failed, 3 passed # related cache / prefix / OpenAI / proxy / LiteLLM / compression-cache suites 1217 passed, 33 skipped # full suite, before vs after: identical failure set except three tests that # fail intermittently under -n 8 on both builds and pass in isolation on both. # All remaining failures are local-environment only (no Rust toolchain, # sandboxed network, missing optional extras). 41 failed, 16305 passed, 860 skipped, 14 errors $ ruff check . && ruff format --check headroom tests # clean $ mypy headroom/cache/prefix_tracker.py # Success: no issues found ``` ## Real Behavior Proof - Environment: macOS arm64, Python 3.13, OpenCode 2.1.2 with the bundled Headroom plugin → `headroom proxy --mode cache|token` → LiteLLM gateway → Bedrock Claude Haiku; synthetic data only. - Exact command / steps: Ran the same OpenCode agent task (one tool returning 120 synthetic JSON records, called 3 times in sequence) through the proxy, once on installed 0.41.0 and once importing this branch's Python source via `PYTHONPATH` (Rust crates unchanged since 0.41.0); an `httpx` hook in the proxy recorded only the `cache_control` paths of each outbound body. - Observed result: 0.41.0 forwards 5 markers on the second request and the gateway returns 400 "Found 5"; with this branch, cache and token mode forward at most 4 markers on every request and the agent returns the exact expected answer after 3 tool calls. - Not tested: Windows; Anthropic `/v1/messages` (unchanged path); Responses API; clients other than OpenCode V2; live sessions longer than 4 requests; compression of cache-protected Chat text blocks. ### Details | Build | Mode | Inbound markers per request | Outbound markers per request | Result | | --- | --- | --- | --- | --- | | 0.41.0 | cache | 4, 4 | 4, **5** | 400 "Found 5", turn aborted | | this PR | cache | 4, 4, 4, 4 | 4, 4, 4, 4 | exact answer after 3 tool calls | | this PR | token | 4, 4, 4, 4 | 4, 4, 4, 3 | exact answer after 3 tool calls (see note) | Loopback HTTP replay of the same shapes (4 steps, cache and token, with and without a provider-confirmed frozen floor): - 0.41.0 forwards 4/5/6/7 markers. - This branch forwards 4/4/4/4 in all four cases. Paths and values match the client, and the non-marker message bytes are unchanged. The token-mode `3` also reproduces on unpatched 0.41.0 and is a separate issue. `CompressionCache._swap_tool_result_content` writes a cached compression back as a string for `role: "tool"`, which drops the client's block marker on the newest tool result. That is a lost breakpoint, not a 400. Not changed here; tracked in #4332. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: Chat Completions forwarding no longer re-sends replayed content-block `cache_control` markers the client did not send this turn. - Kill switch / disable path: none (bug fix; output is the client's own marker layout) - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines Co-authored-by: Thomas Bartik <22518863+tbartik@users.noreply.github.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
269 lines
11 KiB
Python
269 lines
11 KiB
Python
"""``cache_control`` outside content blocks must not accumulate across replayed turns.
|
|
|
|
Chat Completions clients that reach Anthropic models through a gateway (OpenCode
|
|
via ``@ai-sdk/openai-compatible``, then LiteLLM) put breakpoints on the message
|
|
dict and on an assistant's ``tool_calls`` entries, not on a content block, and
|
|
move them to the newest messages on every call.
|
|
The prefix replay (``finalize_turn`` / ``overlay_cached_prefix``) forwards earlier
|
|
turns' messages byte-identical, so the markers they carried back then came along
|
|
and piled up behind the client's current ones. Two markers from the client became
|
|
four and more on the wire, the gateway translated them into Anthropic blocks, and the
|
|
request failed with ``A maximum of 4 blocks with cache_control may be provided``.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import copy
|
|
import json
|
|
from types import SimpleNamespace
|
|
|
|
import httpx
|
|
import pytest
|
|
|
|
pytest.importorskip("fastapi")
|
|
|
|
from fastapi.testclient import TestClient
|
|
|
|
from headroom.cache.prefix_tracker import mirror_client_message_cache_control
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
|
|
CC = {"type": "ephemeral"}
|
|
|
|
|
|
def _marked(msg: dict, marker: dict | None = CC) -> dict:
|
|
return {**msg, "cache_control": dict(marker)} if marker else dict(msg)
|
|
|
|
|
|
def _message_markers(messages: list[dict]) -> list[int]:
|
|
return [i for i, m in enumerate(messages) if isinstance(m, dict) and "cache_control" in m]
|
|
|
|
|
|
def _all_markers(messages: list[dict]) -> list[str]:
|
|
"""Every marker outside content blocks: ``"i"`` on a message, ``"i.j"`` on a tool call."""
|
|
found = []
|
|
for i, m in enumerate(messages):
|
|
if "cache_control" in m:
|
|
found.append(str(i))
|
|
for j, call in enumerate(m.get("tool_calls") or []):
|
|
if "cache_control" in call:
|
|
found.append(f"{i}.{j}")
|
|
return found
|
|
|
|
|
|
# ── the helper ──────────────────────────────────────────────────────────────
|
|
|
|
|
|
def test_replayed_markers_are_dropped_and_the_clients_are_kept() -> None:
|
|
client = [
|
|
_marked({"role": "system", "content": "sys"}),
|
|
{"role": "user", "content": "q"},
|
|
{"role": "tool", "tool_call_id": "t3", "content": "old output"},
|
|
_marked({"role": "tool", "tool_call_id": "t5", "content": "new output"}),
|
|
]
|
|
# What a replay produced: last turn's markers on user and tool 3 came along.
|
|
replayed = [
|
|
_marked({"role": "system", "content": "sys"}),
|
|
_marked({"role": "user", "content": "q"}),
|
|
_marked({"role": "tool", "tool_call_id": "t3", "content": "old <compressed>"}),
|
|
{"role": "tool", "tool_call_id": "t5", "content": "new output"},
|
|
]
|
|
|
|
out = mirror_client_message_cache_control(replayed, client)
|
|
|
|
assert _message_markers(out) == [0, 3]
|
|
# Only the marker moves: the replayed (compressed) content is what the
|
|
# provider cached, and it must stay byte-identical.
|
|
assert [m["content"] for m in out] == [m["content"] for m in replayed]
|
|
|
|
|
|
def test_replayed_tool_call_markers_are_dropped_and_the_clients_are_kept() -> None:
|
|
def call(cid: str, marker: bool) -> dict:
|
|
c = {"id": cid, "type": "function", "function": {"name": "bash", "arguments": "{}"}}
|
|
return _marked(c) if marker else c
|
|
|
|
client = [
|
|
{"role": "assistant", "content": "", "tool_calls": [call("t3", False)]},
|
|
{"role": "assistant", "content": "", "tool_calls": [call("t5", True)]},
|
|
]
|
|
# Last step's newest call (t3) carried the marker; the replay brought it back.
|
|
replayed = [
|
|
{"role": "assistant", "content": "", "tool_calls": [call("t3", True)]},
|
|
{"role": "assistant", "content": "", "tool_calls": [call("t5", True)]},
|
|
]
|
|
|
|
out = mirror_client_message_cache_control(replayed, client)
|
|
|
|
assert _all_markers(out) == ["1.0"]
|
|
assert out[1] is replayed[1] # already right: left as-is
|
|
|
|
|
|
def test_tool_calls_of_different_length_are_left_alone() -> None:
|
|
c = {"id": "t", "type": "function", "function": {"name": "bash", "arguments": "{}"}}
|
|
client = [{"role": "assistant", "content": "", "tool_calls": [c]}]
|
|
messages = [{"role": "assistant", "content": "", "tool_calls": [_marked(c), _marked(c)]}]
|
|
assert mirror_client_message_cache_control(messages, client) is messages
|
|
|
|
|
|
def test_the_clients_marker_value_is_copied_verbatim() -> None:
|
|
ttl = {"type": "ephemeral", "ttl": "1h"}
|
|
client = [_marked({"role": "user", "content": "q"}, ttl)]
|
|
out = mirror_client_message_cache_control([_marked({"role": "user", "content": "q"})], client)
|
|
assert out[0]["cache_control"] == ttl
|
|
|
|
|
|
def test_nothing_to_do_returns_the_same_list() -> None:
|
|
client = [_marked({"role": "system", "content": "sys"}), {"role": "user", "content": "q"}]
|
|
messages = copy.deepcopy(client)
|
|
assert mirror_client_message_cache_control(messages, client) is messages
|
|
|
|
|
|
def test_block_markers_already_where_the_client_put_them_are_left_alone() -> None:
|
|
block = {"type": "text", "text": "x", "cache_control": CC}
|
|
client = [{"role": "user", "content": [dict(block)]}]
|
|
messages = [{"role": "user", "content": [dict(block)]}]
|
|
out = mirror_client_message_cache_control(messages, client)
|
|
assert out is messages
|
|
assert out[0]["content"][0]["cache_control"] == CC
|
|
|
|
|
|
def test_misaligned_lists_are_returned_unchanged() -> None:
|
|
messages = [
|
|
_marked({"role": "user", "content": "a"}),
|
|
_marked({"role": "user", "content": "b"}),
|
|
]
|
|
assert mirror_client_message_cache_control(messages, messages[:1]) is messages
|
|
assert mirror_client_message_cache_control(messages, None) is messages
|
|
|
|
|
|
# ── through the Chat Completions handler ────────────────────────────────────
|
|
|
|
BIG = "row," * 4000
|
|
|
|
|
|
class _Tracker:
|
|
"""Stands in for the session tracker with last turn already recorded."""
|
|
|
|
def __init__(self, previous_original: list[dict], previous_forwarded: list[dict]):
|
|
self._original = previous_original
|
|
self._forwarded = previous_forwarded
|
|
|
|
def get_frozen_message_count(self) -> int:
|
|
return 0
|
|
|
|
def get_last_original_messages(self): # noqa: ANN201
|
|
return copy.deepcopy(self._original)
|
|
|
|
def get_last_forwarded_messages(self): # noqa: ANN201
|
|
return copy.deepcopy(self._forwarded)
|
|
|
|
def update_from_response(self, **kwargs): # noqa: ANN003
|
|
return None
|
|
|
|
|
|
def _turns() -> tuple[list[dict], list[dict], list[dict]]:
|
|
"""The shape OpenCode sends to a Claude model, three steps into one turn.
|
|
|
|
Last step the client marked the system prompt, the newest tool call (t3) and
|
|
its result (tool 3), and headroom forwarded tool 3 compressed. This step the
|
|
client moved both markers to t5 and tool 5 and left t3 and tool 3 unmarked.
|
|
"""
|
|
system = {"role": "system", "content": "You are an agent. " * 50}
|
|
user = {"role": "user", "content": "run three commands"}
|
|
call3 = {
|
|
"role": "assistant",
|
|
"content": "",
|
|
"tool_calls": [
|
|
{"id": "t3", "type": "function", "function": {"name": "bash", "arguments": "{}"}}
|
|
],
|
|
}
|
|
tool3 = {"role": "tool", "tool_call_id": "t3", "content": BIG}
|
|
call5 = {
|
|
"role": "assistant",
|
|
"content": "",
|
|
"tool_calls": [
|
|
{"id": "t5", "type": "function", "function": {"name": "bash", "arguments": "{}"}}
|
|
],
|
|
}
|
|
tool5 = {"role": "tool", "tool_call_id": "t5", "content": "30"}
|
|
|
|
def with_marked_call(assistant: dict) -> dict:
|
|
return {**assistant, "tool_calls": [_marked(assistant["tool_calls"][0])]}
|
|
|
|
previous_original = [_marked(system), user, with_marked_call(call3), _marked(tool3)]
|
|
previous_forwarded = [
|
|
_marked(system),
|
|
user,
|
|
with_marked_call(call3),
|
|
_marked({**tool3, "content": "row,<compressed: 4000 rows>"}),
|
|
]
|
|
current = [_marked(system), user, call3, tool3, with_marked_call(call5), _marked(tool5)]
|
|
return previous_original, previous_forwarded, current
|
|
|
|
|
|
@pytest.mark.parametrize("mode", ["cache", "token"])
|
|
def test_forwarded_markers_follow_the_client_not_the_replay(mode: str) -> None:
|
|
previous_original, previous_forwarded, current = _turns()
|
|
captured: dict = {}
|
|
|
|
config = ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
image_optimize=False,
|
|
)
|
|
with TestClient(create_app(config)) as client:
|
|
proxy = client.app.state.proxy
|
|
proxy.config.optimize = True
|
|
proxy.config.mode = mode
|
|
tracker = _Tracker(previous_original, previous_forwarded)
|
|
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: "s"
|
|
proxy.session_tracker_store.get_or_create = (
|
|
lambda session_id, provider, cache_ttl_seconds=None: tracker
|
|
)
|
|
proxy.openai_pipeline.apply = lambda **kwargs: SimpleNamespace(
|
|
messages=kwargs["messages"],
|
|
transforms_applied=[],
|
|
timing={},
|
|
tokens_before=100,
|
|
tokens_after=100,
|
|
waste_signals=None,
|
|
)
|
|
|
|
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
captured["body"] = json.loads(body) if isinstance(body, (bytes, str)) else body
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "c",
|
|
"choices": [
|
|
{
|
|
"index": 0,
|
|
"message": {"role": "assistant", "content": "ok"},
|
|
"finish_reason": "stop",
|
|
}
|
|
],
|
|
"usage": {"prompt_tokens": 100, "completion_tokens": 1, "total_tokens": 101},
|
|
},
|
|
)
|
|
|
|
proxy._retry_request = _fake_retry
|
|
|
|
response = client.post(
|
|
"/v1/chat/completions",
|
|
headers={"authorization": "Bearer test-key"},
|
|
json={"model": "claude-sonnet-4-6", "messages": current},
|
|
)
|
|
|
|
assert response.status_code == 200, response.text
|
|
forwarded = captured["body"]["messages"]
|
|
# Precondition: the replay was live — tool 3 went out compressed.
|
|
assert forwarded[3]["content"] != BIG
|
|
# The client marked the system prompt, t5 and tool 5; last step's markers on
|
|
# t3 and tool 3 must not ride along with the replayed bytes.
|
|
assert _all_markers(forwarded) == _all_markers(current) == ["0", "4.0", "5"]
|