mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): keep cache_control outside content blocks where the Chat Completions client put it (#3895)
## Description Chat Completions clients that reach Anthropic models through a gateway (OpenCode via `@ai-sdk/openai-compatible`, then LiteLLM) do not mark content blocks. They mark the message dict and the entries of an assistant's `tool_calls`, and move those markers to the newest messages on every call. `handle_openai_chat` replays earlier turns' forwarded messages byte-identical: `finalize_turn` in every mode, plus `_restore_frozen_prefix` in cache mode. The markers those messages carried back then come along with them. `normalize_message_cache_control` exists for exactly this accumulation, but it only handles block-level markers, and this handler does not call it after its replays. A client sending three markers per step (system, newest tool call, newest tool result) gets four forwarded, then six, and the gateway fails with `A maximum of 4 blocks with cache_control may be provided. Found 6`, which breaks every tool turn of three or more steps. `enforce_cache_breakpoint_budget` does not catch it either: `count_cache_breakpoints` does not count `tool_calls` markers, so it sees four. This adds `mirror_client_message_cache_control`. It applies the rule the block-level normalizer already applies (the client's current request decides where markers go) to the two places it does not see: - on each message, and on each entry of its `tool_calls`, drop the marker unless the client put one there in this request; - where the client did, copy the client's own value; - leave content untouched, so the replayed bytes the provider cached stay identical. The handler applies it once, after both replays. Closes #3894 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cache/prefix_tracker.py`: new `mirror_client_message_cache_control(messages, client_messages)` (and a small `_mirror_marker` helper). - It touches only markers on the message dict and on `tool_calls` entries; block-level markers stay with `normalize_message_cache_control`. - It returns the input list unchanged when nothing moved. When the message lists, or one message's `tool_calls`, are not index-aligned, that part is left as-is: it does not guess. - `headroom/proxy/handlers/openai.py`: call it in `handle_openai_chat` after the cache-mode restore/replay block, before memory injection, with `original_client_messages`. - `tests/test_openai_message_level_cache_control.py`: - 7 unit tests for the helper: stale message and `tool_calls` markers dropped, the client's value (including `ttl`) copied verbatim, identity when there is nothing to do, block markers ignored, misaligned messages or `tool_calls` left unchanged; - 1 handler test, parametrized over `cache` and `token`, that drives `/v1/chat/completions` with the exact OpenCode shape (system, newest tool call and newest tool result marked) and a tracker holding last step's compressed forward. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest -q tests/test_openai_message_level_cache_control.py 9 passed # the handler test on main, without the fix (both modes): E AssertionError: assert ['0', '2.0', '3', '4.0', '5'] == ['0', '4.0', '5'] $ pytest -q tests/test_proxy/*.py tests/test_*openai*.py tests/test_*prefix*.py tests/test_*cache*.py \ --deselect tests/test_proxy/test_issue_3650_router_emits_visible_retrievable_sentinel.py 2 failed, 1455 passed, 5 skipped # FAILED tests/test_proxy/test_cc_switch_reconciler.py::test_switching_provider_recaptures # FAILED tests/test_proxy/test_cc_switch_reconciler.py::test_official_resets_a_captured_third_party_upstream # Both fail identically on main without this change: # `pytest tests/test_proxy/test_cc_switch_reconciler.py` → 2 failed, 14 passed. $ ruff check <changed files> && ruff format --check <changed files> All checks passed! 3 files already formatted ``` ## Real Behavior Proof - Environment: OpenCode 1.18.30 → `headroom proxy` 0.39.1 → LiteLLM proxy → AWS Bedrock `bedrock-claude-sonnet-4.6`, in a Kubernetes pod, with pass-through taps before and after the proxy counting every `cache_control` (message dict, content blocks, `tool_calls`). This change was applied to 0.39.1 (the two changed files dropped over the installed package) and run as a second proxy in the same pod. Plus the four requests captured from that session, replayed against a mock upstream. - Exact command / steps: a turn asking for `date`, `uname -r` and `nproc` through the bash tool, one after the other: five requests, the last three after a tool result. - Observed result: stock 0.39.1 forwards 6 markers on the fourth request where the client sent 3, and Bedrock answers 400; with this change it forwards exactly the client's 3 and the whole turn completes, in both cache and token mode. Details: | Build (live, Bedrock) | Request 4 forwarded (client sent 3) | Turn | |---|---|---| | stock 0.39.1, `cache` | 6 → 400 `Found 6` | fails | | 0.39.1 + this change, `cache` | 3 (system, 4.tc0, tool#5) | all 5 requests 200, answer returned | | 0.39.1 + this change, `token` | 3 | all 5 requests 200, answer returned | Replay of the captured requests, markers forwarded for requests 1 to 4: | Build | Forwarded (client sent 3 / 2 / 3 / 3) | |---|---| | 0.37.0 | 3 / 2 / 3 / 3 | | `main` (`0a2c80d5`) | 3 / 2 / **4** / **6** | | this branch | 3 / 2 / 3 / 3 | - Not tested: this branch installed as a package (live runs used 0.39.1 with these two files patched in, because that is the version the deployment ships); non-OpenCode Chat Completions clients. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a (stable) - Stable/default behavior changed: yes, narrowly. On `/v1/chat/completions`, `cache_control` on the message dict and on `tool_calls` entries now matches the client's current request instead of accumulating from replayed turns. Content and block-level markers are unchanged. - Kill switch / disable path: none new; revert the PR - Unsafe override required: no - Qualification impact: none expected. The provider keys its cache on content, not on marker placement (the premise `normalize_message_cache_control` already relies on), and the client's own positions are the ones it would have sent without the proxy. - Rollback path: revert this PR; no state, schema or config migration involved ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - `mypy headroom` could not run in my environment: it stops in numpy's stubs (`Type statement is only supported in Python 3.12`) before it reaches this code. - No documentation change: this restores the behaviour the existing docstrings already describe for block-level markers. - Follow-up, not in this PR: `count_cache_breakpoints` does not count `tool_calls` markers, so the last-stop guard under-counts for these clients (it saw four where six went out). Teaching it to count them also means teaching the repair step to remove them, so I kept it separate. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JerrettDavis
parent
1982b3a773
commit
a6d6c14cf7
Vendored
+71
@@ -870,6 +870,77 @@ def normalize_message_cache_control(
|
||||
return out if changed else messages
|
||||
|
||||
|
||||
def mirror_client_message_cache_control(
|
||||
messages: list[dict[str, Any]],
|
||||
client_messages: list[dict[str, Any]] | None,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Keep ``cache_control`` outside content blocks exactly where the client put it.
|
||||
|
||||
OpenAI-compatible clients that talk to Anthropic models through a gateway
|
||||
(OpenCode via ``@ai-sdk/openai-compatible``, then LiteLLM) do not mark
|
||||
content blocks. They mark the MESSAGE dict — ``{"role": "tool", "content":
|
||||
"...", "cache_control": {...}}`` — and, on an assistant turn, the entries of
|
||||
its ``tool_calls`` list, which the gateway turns into ``tool_use`` blocks.
|
||||
``normalize_message_cache_control`` only sees block markers
|
||||
(``_client_marker_positions`` reads list content), so after
|
||||
``overlay_cached_prefix`` / ``finalize_turn`` replay an earlier turn's
|
||||
forwarded messages, these markers ride along untouched. A client that marks
|
||||
three places per request (system, the newest tool call, the newest tool
|
||||
result) ends up forwarding four, then five and more, and the gateway's
|
||||
translation to Anthropic blocks crosses the limit of four (``A maximum of 4
|
||||
blocks with cache_control may be provided. Found 6``).
|
||||
|
||||
The client's current request is the authority for these markers, the same
|
||||
rule the block-level normalizer applies: on each message and on each of its
|
||||
``tool_calls``, drop the marker unless the client put one there in this
|
||||
request, in which case copy the client's own value. Content — and any
|
||||
block-level marker in it — is left alone, so the replayed bytes the provider
|
||||
cached stay byte-identical.
|
||||
|
||||
Index alignment is the invariant (the pipeline preserves message count and
|
||||
each message's tool calls); where it does not hold, that message or list is
|
||||
returned unchanged rather than guessing, and the breakpoint budget guard
|
||||
still bounds the total.
|
||||
"""
|
||||
if not isinstance(client_messages, list) or len(client_messages) != len(messages):
|
||||
return messages
|
||||
changed = False
|
||||
out: list[dict[str, Any]] = []
|
||||
for msg, client in zip(messages, client_messages):
|
||||
if not isinstance(msg, dict):
|
||||
out.append(msg)
|
||||
continue
|
||||
new_msg, msg_changed = _mirror_marker(msg, client)
|
||||
calls = new_msg.get("tool_calls")
|
||||
client_calls = client.get("tool_calls") if isinstance(client, dict) else None
|
||||
if (
|
||||
isinstance(calls, list)
|
||||
and isinstance(client_calls, list)
|
||||
and len(calls) == len(client_calls)
|
||||
):
|
||||
mirrored = [_mirror_marker(c, cc) for c, cc in zip(calls, client_calls)]
|
||||
if any(c for _, c in mirrored):
|
||||
new_msg = {**new_msg, "tool_calls": [c for c, _ in mirrored]}
|
||||
msg_changed = True
|
||||
out.append(new_msg)
|
||||
changed = changed or msg_changed
|
||||
return out if changed else messages
|
||||
|
||||
|
||||
def _mirror_marker(holder: Any, client_holder: Any) -> tuple[Any, bool]:
|
||||
"""Give ``holder`` the client's own ``cache_control`` (or none); report change."""
|
||||
if not isinstance(holder, dict):
|
||||
return holder, False
|
||||
client_marker = client_holder.get("cache_control") if isinstance(client_holder, dict) else None
|
||||
if isinstance(client_marker, dict):
|
||||
if holder.get("cache_control") == client_marker:
|
||||
return holder, False
|
||||
return {**holder, "cache_control": dict(client_marker)}, True
|
||||
if "cache_control" in holder:
|
||||
return {k: v for k, v in holder.items() if k != "cache_control"}, True
|
||||
return holder, False
|
||||
|
||||
|
||||
class PrefixCacheTracker:
|
||||
"""Tracks provider prefix cache state across turns in a session.
|
||||
|
||||
|
||||
@@ -4350,6 +4350,19 @@ class OpenAIHandlerMixin:
|
||||
confirmed_frozen_count=_openai_confirmed_frozen,
|
||||
).messages
|
||||
|
||||
# Both replays above forward earlier turns' messages byte-identical,
|
||||
# including the cache_control each carried back then. Chat Completions
|
||||
# clients mark the message dict and an assistant's tool_calls, which the
|
||||
# block-level normalizer does not see, so without this the markers
|
||||
# accumulate turn over turn and an Anthropic-backed gateway rejects the
|
||||
# request (more than four breakpoints). Content is untouched: only the
|
||||
# markers move.
|
||||
from headroom.cache.prefix_tracker import mirror_client_message_cache_control
|
||||
|
||||
optimized_messages = mirror_client_message_cache_control(
|
||||
optimized_messages, original_client_messages
|
||||
)
|
||||
|
||||
# Memory: inject context and tools for OpenAI requests.
|
||||
#
|
||||
# PR-A3 follow-up to A2: memory context now routes exclusively to
|
||||
|
||||
@@ -0,0 +1,266 @@
|
||||
"""``cache_control`` outside content blocks must not accumulate across replayed turns.
|
||||
|
||||
Chat Completions clients that reach Anthropic models through a gateway (OpenCode
|
||||
via ``@ai-sdk/openai-compatible``, then LiteLLM) put breakpoints on the message
|
||||
dict and on an assistant's ``tool_calls`` entries, not on a content block, and
|
||||
move them to the newest messages on every call.
|
||||
The prefix replay (``finalize_turn`` / ``overlay_cached_prefix``) forwards earlier
|
||||
turns' messages byte-identical, so the markers they carried back then came along
|
||||
and piled up behind the client's current ones. Two markers from the client became
|
||||
four and more on the wire, the gateway translated them into Anthropic blocks, and the
|
||||
request failed with ``A maximum of 4 blocks with cache_control may be provided``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import json
|
||||
from types import SimpleNamespace
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
pytest.importorskip("fastapi")
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from headroom.cache.prefix_tracker import mirror_client_message_cache_control
|
||||
from headroom.proxy.server import ProxyConfig, create_app
|
||||
|
||||
CC = {"type": "ephemeral"}
|
||||
|
||||
|
||||
def _marked(msg: dict, marker: dict | None = CC) -> dict:
|
||||
return {**msg, "cache_control": dict(marker)} if marker else dict(msg)
|
||||
|
||||
|
||||
def _message_markers(messages: list[dict]) -> list[int]:
|
||||
return [i for i, m in enumerate(messages) if isinstance(m, dict) and "cache_control" in m]
|
||||
|
||||
|
||||
def _all_markers(messages: list[dict]) -> list[str]:
|
||||
"""Every marker outside content blocks: ``"i"`` on a message, ``"i.j"`` on a tool call."""
|
||||
found = []
|
||||
for i, m in enumerate(messages):
|
||||
if "cache_control" in m:
|
||||
found.append(str(i))
|
||||
for j, call in enumerate(m.get("tool_calls") or []):
|
||||
if "cache_control" in call:
|
||||
found.append(f"{i}.{j}")
|
||||
return found
|
||||
|
||||
|
||||
# ── the helper ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_replayed_markers_are_dropped_and_the_clients_are_kept() -> None:
|
||||
client = [
|
||||
_marked({"role": "system", "content": "sys"}),
|
||||
{"role": "user", "content": "q"},
|
||||
{"role": "tool", "tool_call_id": "t3", "content": "old output"},
|
||||
_marked({"role": "tool", "tool_call_id": "t5", "content": "new output"}),
|
||||
]
|
||||
# What a replay produced: last turn's markers on user and tool 3 came along.
|
||||
replayed = [
|
||||
_marked({"role": "system", "content": "sys"}),
|
||||
_marked({"role": "user", "content": "q"}),
|
||||
_marked({"role": "tool", "tool_call_id": "t3", "content": "old <compressed>"}),
|
||||
{"role": "tool", "tool_call_id": "t5", "content": "new output"},
|
||||
]
|
||||
|
||||
out = mirror_client_message_cache_control(replayed, client)
|
||||
|
||||
assert _message_markers(out) == [0, 3]
|
||||
# Only the marker moves: the replayed (compressed) content is what the
|
||||
# provider cached, and it must stay byte-identical.
|
||||
assert [m["content"] for m in out] == [m["content"] for m in replayed]
|
||||
|
||||
|
||||
def test_replayed_tool_call_markers_are_dropped_and_the_clients_are_kept() -> None:
|
||||
def call(cid: str, marker: bool) -> dict:
|
||||
c = {"id": cid, "type": "function", "function": {"name": "bash", "arguments": "{}"}}
|
||||
return _marked(c) if marker else c
|
||||
|
||||
client = [
|
||||
{"role": "assistant", "content": "", "tool_calls": [call("t3", False)]},
|
||||
{"role": "assistant", "content": "", "tool_calls": [call("t5", True)]},
|
||||
]
|
||||
# Last step's newest call (t3) carried the marker; the replay brought it back.
|
||||
replayed = [
|
||||
{"role": "assistant", "content": "", "tool_calls": [call("t3", True)]},
|
||||
{"role": "assistant", "content": "", "tool_calls": [call("t5", True)]},
|
||||
]
|
||||
|
||||
out = mirror_client_message_cache_control(replayed, client)
|
||||
|
||||
assert _all_markers(out) == ["1.0"]
|
||||
assert out[1] is replayed[1] # already right: left as-is
|
||||
|
||||
|
||||
def test_tool_calls_of_different_length_are_left_alone() -> None:
|
||||
c = {"id": "t", "type": "function", "function": {"name": "bash", "arguments": "{}"}}
|
||||
client = [{"role": "assistant", "content": "", "tool_calls": [c]}]
|
||||
messages = [{"role": "assistant", "content": "", "tool_calls": [_marked(c), _marked(c)]}]
|
||||
assert mirror_client_message_cache_control(messages, client) is messages
|
||||
|
||||
|
||||
def test_the_clients_marker_value_is_copied_verbatim() -> None:
|
||||
ttl = {"type": "ephemeral", "ttl": "1h"}
|
||||
client = [_marked({"role": "user", "content": "q"}, ttl)]
|
||||
out = mirror_client_message_cache_control([_marked({"role": "user", "content": "q"})], client)
|
||||
assert out[0]["cache_control"] == ttl
|
||||
|
||||
|
||||
def test_nothing_to_do_returns_the_same_list() -> None:
|
||||
client = [_marked({"role": "system", "content": "sys"}), {"role": "user", "content": "q"}]
|
||||
messages = copy.deepcopy(client)
|
||||
assert mirror_client_message_cache_control(messages, client) is messages
|
||||
|
||||
|
||||
def test_block_level_markers_are_left_to_the_block_normalizer() -> None:
|
||||
block = {"type": "text", "text": "x", "cache_control": CC}
|
||||
client = [{"role": "user", "content": [dict(block)]}]
|
||||
messages = [{"role": "user", "content": [dict(block)]}]
|
||||
out = mirror_client_message_cache_control(messages, client)
|
||||
assert out is messages
|
||||
assert out[0]["content"][0]["cache_control"] == CC
|
||||
|
||||
|
||||
def test_misaligned_lists_are_returned_unchanged() -> None:
|
||||
messages = [
|
||||
_marked({"role": "user", "content": "a"}),
|
||||
_marked({"role": "user", "content": "b"}),
|
||||
]
|
||||
assert mirror_client_message_cache_control(messages, messages[:1]) is messages
|
||||
assert mirror_client_message_cache_control(messages, None) is messages
|
||||
|
||||
|
||||
# ── through the Chat Completions handler ────────────────────────────────────
|
||||
|
||||
BIG = "row," * 4000
|
||||
|
||||
|
||||
class _Tracker:
|
||||
"""Stands in for the session tracker with last turn already recorded."""
|
||||
|
||||
def __init__(self, previous_original: list[dict], previous_forwarded: list[dict]):
|
||||
self._original = previous_original
|
||||
self._forwarded = previous_forwarded
|
||||
|
||||
def get_frozen_message_count(self) -> int:
|
||||
return 0
|
||||
|
||||
def get_last_original_messages(self): # noqa: ANN201
|
||||
return copy.deepcopy(self._original)
|
||||
|
||||
def get_last_forwarded_messages(self): # noqa: ANN201
|
||||
return copy.deepcopy(self._forwarded)
|
||||
|
||||
def update_from_response(self, **kwargs): # noqa: ANN003
|
||||
return None
|
||||
|
||||
|
||||
def _turns() -> tuple[list[dict], list[dict], list[dict]]:
|
||||
"""The shape OpenCode sends to a Claude model, three steps into one turn.
|
||||
|
||||
Last step the client marked the system prompt, the newest tool call (t3) and
|
||||
its result (tool 3), and headroom forwarded tool 3 compressed. This step the
|
||||
client moved both markers to t5 and tool 5 and left t3 and tool 3 unmarked.
|
||||
"""
|
||||
system = {"role": "system", "content": "You are an agent. " * 50}
|
||||
user = {"role": "user", "content": "run three commands"}
|
||||
call3 = {
|
||||
"role": "assistant",
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{"id": "t3", "type": "function", "function": {"name": "bash", "arguments": "{}"}}
|
||||
],
|
||||
}
|
||||
tool3 = {"role": "tool", "tool_call_id": "t3", "content": BIG}
|
||||
call5 = {
|
||||
"role": "assistant",
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{"id": "t5", "type": "function", "function": {"name": "bash", "arguments": "{}"}}
|
||||
],
|
||||
}
|
||||
tool5 = {"role": "tool", "tool_call_id": "t5", "content": "30"}
|
||||
|
||||
def with_marked_call(assistant: dict) -> dict:
|
||||
return {**assistant, "tool_calls": [_marked(assistant["tool_calls"][0])]}
|
||||
|
||||
previous_original = [_marked(system), user, with_marked_call(call3), _marked(tool3)]
|
||||
previous_forwarded = [
|
||||
_marked(system),
|
||||
user,
|
||||
with_marked_call(call3),
|
||||
_marked({**tool3, "content": "row,<compressed: 4000 rows>"}),
|
||||
]
|
||||
current = [_marked(system), user, call3, tool3, with_marked_call(call5), _marked(tool5)]
|
||||
return previous_original, previous_forwarded, current
|
||||
|
||||
|
||||
@pytest.mark.parametrize("mode", ["cache", "token"])
|
||||
def test_forwarded_markers_follow_the_client_not_the_replay(mode: str) -> None:
|
||||
previous_original, previous_forwarded, current = _turns()
|
||||
captured: dict = {}
|
||||
|
||||
config = ProxyConfig(
|
||||
optimize=False,
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
cost_tracking_enabled=False,
|
||||
log_requests=False,
|
||||
ccr_inject_tool=False,
|
||||
ccr_handle_responses=False,
|
||||
ccr_context_tracking=False,
|
||||
image_optimize=False,
|
||||
)
|
||||
with TestClient(create_app(config)) as client:
|
||||
proxy = client.app.state.proxy
|
||||
proxy.config.optimize = True
|
||||
proxy.config.mode = mode
|
||||
tracker = _Tracker(previous_original, previous_forwarded)
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: "s"
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: tracker
|
||||
proxy.openai_pipeline.apply = lambda **kwargs: SimpleNamespace(
|
||||
messages=kwargs["messages"],
|
||||
transforms_applied=[],
|
||||
timing={},
|
||||
tokens_before=100,
|
||||
tokens_after=100,
|
||||
waste_signals=None,
|
||||
)
|
||||
|
||||
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
||||
captured["body"] = json.loads(body) if isinstance(body, (bytes, str)) else body
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"id": "c",
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"message": {"role": "assistant", "content": "ok"},
|
||||
"finish_reason": "stop",
|
||||
}
|
||||
],
|
||||
"usage": {"prompt_tokens": 100, "completion_tokens": 1, "total_tokens": 101},
|
||||
},
|
||||
)
|
||||
|
||||
proxy._retry_request = _fake_retry
|
||||
|
||||
response = client.post(
|
||||
"/v1/chat/completions",
|
||||
headers={"authorization": "Bearer test-key"},
|
||||
json={"model": "claude-sonnet-4-6", "messages": current},
|
||||
)
|
||||
|
||||
assert response.status_code == 200, response.text
|
||||
forwarded = captured["body"]["messages"]
|
||||
# Precondition: the replay was live — tool 3 went out compressed.
|
||||
assert forwarded[3]["content"] != BIG
|
||||
# The client marked the system prompt, t5 and tool 5; last step's markers on
|
||||
# t3 and tool 3 must not ride along with the replayed bytes.
|
||||
assert _all_markers(forwarded) == _all_markers(current) == ["0", "4.0", "5"]
|
||||
Reference in New Issue
Block a user