mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 18:41:07 +08:00
## Description The Anthropic handler flips a streaming request to the buffered `stream:false` path when the outgoing body carries a redeemable CCR marker (#3092). A marker stays in the conversation history for the rest of the session, so once one tool result was compressed, every later turn carried an owned marker and was buffered. On a buffered turn the client receives nothing until the whole generation finishes. The report shows the effect: 71% of Opus turns buffered (51% in a second report on Linux), zero retrievals, the buffered share climbing from 0% to 90% with proxy uptime, and time to first byte rising from about 1 s to as much as 37 s. This PR keeps buffering only for turns that show the model a marker it has not seen yet, meaning one absent from what this session forwarded last turn. Turns whose markers were all forwarded earlier stream. The model can still ask for one of those originals, and the client never declared `headroom_retrieve`, so on those turns the stream withholds the call, the proxy serves it from the compression store, and the continuation round streams into the same client message. This reuses the filter and continuation-round machinery the proxy already uses for its memory tools (#2195). Closes #3587 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Performance improvement ## Changes Made - `headroom/proxy/handlers/anthropic.py`: - `_outgoing_redeemable_markers` returns `(any_owned, new_owned)`. "New" is decided against the session's last forwarded messages (via the existing `has_new_ccr_markers`), not by message position, so a marker that token mode adds to an earlier tool result on this request still counts as new. With no forwarded history, every owned marker counts as new. `_outgoing_body_has_redeemable_marker` returns the second value. - A new owned marker buffers, as on `main`. An owned marker that was already forwarded streams with `serve_ccr_retrieval=True`. No owned marker streams as before (#3071). - `headroom/proxy/memory_tool_stream.py`: `MemoryToolStreamFilter(hold_tool_use=True)` holds every block from the round's first `tool_use` until the caller decides (`release_held(drop_tool_use=...)`; `closing_frames` releases them). Text before the first tool call still streams live. Off by default, so the memory-tool behavior is unchanged. - `headroom/proxy/handlers/streaming.py`: `_stream_response(serve_ccr_retrieval=...)` adds `headroom_retrieve` to the withheld tool names and holds client tool calls. `_continue_memory_tool_stream(serve_retrieval=...)` serves withheld retrievals and continues the round. A round that retrieved alongside a client tool is continued without the unrun client call, for the model to re-issue with the content in hand. That is what `handle_response` already does on the buffered path. - `headroom/ccr/response_handler.py`: `serve_withheld_retrievals` runs withheld calls through the existing `_execute_retrieval` and counts them in `ccr_retrievals`. - Tests: - New `tests/test_ccr_stream_retrieval.py`. Two turns of one session through the real proxy app: turn 1 introduces an owned marker and buffers; on turn 2 the upstream asks for that original. - The original is served server-side, the answer reaches the client, and no `headroom_retrieve` appears in the client's SSE. - The same with a retrieval mixed with a client-owned tool, in both orders: only the model's re-issued call reaches the client. - An old-marker turn without a retrieval streams byte-identical to the upstream. - Filter unit tests for hold mode. - `tests/test_ccr_buffered_stream_signed_thinking.py`: two-turn tests for the gate (a marker already forwarded does not buffer; a marker first shown in an earlier message does). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`): see Additional Notes - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output The new streaming-retrieval tests on the previous head of this PR (64164fc2), which reproduce the review finding: ```text E assert 'headroom_retrieve' not in ['headroom_retrieve'] FAILED tests/test_ccr_stream_retrieval.py::test_later_turn_retrieval_is_served_by_the_proxy FAILED tests/test_ccr_stream_retrieval.py::test_retrieval_mixed_with_a_client_tool[True] FAILED tests/test_ccr_stream_retrieval.py::test_retrieval_mixed_with_a_client_tool[False] 3 failed, 1 passed ``` At this head (d10a7446): ```text $ pytest tests/test_ccr_stream_retrieval.py tests/test_ccr_buffered_stream_signed_thinking.py tests/test_memory_tool_stream.py -q 79 passed $ pytest <every test file with ccr, anthropic, stream, buffer, memory or proxy in its name> tests/test_proxy -m "not live" -q 17 failed, 4504 passed, 129 skipped $ ruff check headroom tests/test_ccr_stream_retrieval.py && ruff format --check headroom tests/test_ccr_stream_retrieval.py All checks passed! ``` The 17 failures are in `tests/test_proxy_uds.py`, `tests/test_upstream_guard.py` and `tests/test_ccr_batch_processor.py`. The identical set fails on `main` (fc1ba891) in my environment, so they are environment-related and not from this change. ## Real Behavior Proof - Environment: macOS (Darwin 24.6.0, arm64), Python 3.12.3. The real proxy app served by uvicorn in-process, cache mode, CCR on, `headroom_retrieve` resident. Only the upstream model is faked, at the HTTP client the proxy uses, so the proxy's own buffering, streaming filter and continuation rounds all run. A streamed answer is 100 text tokens at 15 ms each; a buffered answer arrives after 1.5 s; on a retrieval turn the model writes 5 tokens, calls `headroom_retrieve` for an earlier marker, then writes the 100-token answer once the original comes back. - Exact command / steps: replay a 30-turn agent session through a streaming HTTP client. New owned markers appear on turns 1, 8, 15 and 22, every turn carries the earlier ones, and on turns 4, 11, 18 and 26 the model retrieves an earlier original. Per turn, record buffered or streamed, time to the first text token, time to the first answer token, whether `headroom_retrieve` reached the client, and whether the answer arrived. Two alternating rounds each of `main` (fc1ba891), the previous head of this PR (64164fc2) and this head (d10a7446). - Observed result: `main` buffered 30 of 30 turns (time-to-first-token p50 1,516 ms) and served all 8 retrievals; the previous head buffered 4 turns (p50 24 ms) but leaked all 8 retrievals to the client, which never got an answer on those turns; this head buffers 4 turns (p50 24 ms), serves all 8 retrievals server-side, leaks none, and the answer on a retrieval turn starts at 105 ms. Full numbers below. | Build | Turns buffered | TTFT p50, all turns | Retrievals served | Leaked to client | Retrieval turns answered | Answer starts (retrieval turns, p50) | |---|---|---|---|---|---|---| | `main` (fc1ba891) | 30 of 30 | 1,515.8 ms | 8 of 8 | 0 | 8 of 8 | 1,591.0 ms | | previous head (64164fc2) | 4 of 30 | 24.3 ms | 0 of 8 | 8 | 0 of 8 | — | | this head (d10a7446) | 4 of 30 | 23.9 ms | 8 of 8 | 0 | 8 of 8 | 105.3 ms | The two rounds of each build matched on every count. In this setup a buffered turn's first token arrives when the whole generation is done (1.5 s); with real replies of 3–37 s, as in the report, each avoided buffered turn saves correspondingly more. - Not tested: a live Anthropic upstream. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: yes. Turns whose redeemable markers were all forwarded on an earlier turn now stream instead of buffering, with `headroom_retrieve` withheld from the client and served by the proxy. On those turns, client tool-call blocks reach the client when the round ends rather than as they stream; text is unchanged. Turns that show the model a new marker buffer as before; turns with no owned marker stream as before. - Kill switch / disable path: none added. Existing CCR controls are unchanged. - Unsafe override required: no - Qualification impact: none - Rollback path: revert the two commits ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation: n/a, no user-facing docs describe this gate - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Screenshots (if applicable) n/a ## Additional Notes - Review follow-ups: - The first revision decided "new" by position;64164fc2decides it from what the session forwarded. -64164fc2let a later-turn retrieval reach the client;d10a7446serves it in the stream. The earlier "tradeoff" note no longer applies. - On a round that retrieved alongside a client tool, the client call is dropped and the model decides again with the content in hand, matching the buffered path. If the continuation round itself fails, the client receives the upstream error frame, as with the memory-tool continuation. - The issue also asks for a config switch to disable the buffered CCR path entirely. That is left out to keep this change focused. - `mypy headroom/proxy/handlers/anthropic.py` stops in my local environment on a `numpy` stub (`Type statement is only supported in Python 3.12 and greater`) before reaching project code. CI runs mypy. --------- Co-authored-by: JD Davis <mxjerrett@gmail.com>
458 lines
17 KiB
Python
458 lines
17 KiB
Python
"""Buffered-CCR streaming vs. byte-faithful passthrough (issue #2952).
|
|
|
|
The buffered-CCR path is the one place the Anthropic handler changes the
|
|
request *for its own benefit*: it flips ``stream`` to False so the reply comes
|
|
back as one JSON document it can inspect for ``headroom_retrieve`` calls, then
|
|
resynthesizes SSE for the client.
|
|
|
|
That only works if the flip reaches the wire. When conversation history carries
|
|
a signed ``thinking`` block, ``select_outbound_body`` forwards the client's
|
|
original bytes instead — ``"stream": true`` and all — so upstream streams, the
|
|
JSON parse fails, resynthesis is skipped, and the client is left with a 200 and
|
|
nothing it can read. These tests pin the three defenses: don't take the path,
|
|
survive the reply if we somehow do, and never cache a body in the wrong format.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
import json
|
|
from datetime import datetime
|
|
|
|
import pytest
|
|
|
|
fastapi = pytest.importorskip("fastapi")
|
|
httpx = pytest.importorskip("httpx")
|
|
|
|
from fastapi.testclient import TestClient # noqa: E402
|
|
|
|
from headroom.proxy.models import CacheEntry # noqa: E402
|
|
from headroom.proxy.semantic_cache_key_policy import compute_cache_partition # noqa: E402
|
|
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
|
|
|
|
RETRIEVE_TOOL = {
|
|
"name": "headroom_retrieve",
|
|
"description": "Retrieve original content",
|
|
"input_schema": {"type": "object", "properties": {}},
|
|
}
|
|
|
|
SIGNED_THINKING_TURN = {
|
|
"role": "assistant",
|
|
"content": [
|
|
{
|
|
"type": "thinking",
|
|
"thinking": "private reasoning",
|
|
"signature": "sig-abc123",
|
|
},
|
|
{"type": "text", "text": "Answered."},
|
|
],
|
|
}
|
|
|
|
SSE_BODY = (
|
|
b'event: message_start\ndata: {"type":"message_start","message":{"id":"msg_1"}}\n\n'
|
|
b'event: message_stop\ndata: {"type":"message_stop"}\n\n'
|
|
)
|
|
|
|
|
|
def _config() -> ProxyConfig:
|
|
return ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=True,
|
|
rate_limit_enabled=False,
|
|
memory_enabled=False,
|
|
)
|
|
|
|
|
|
@pytest.fixture
|
|
def ccr_marker() -> str:
|
|
"""A marker this proxy actually owns, so retrieval could really fire.
|
|
|
|
The buffered path is only taken when the outgoing body carries a redeemable
|
|
marker (#3071) — ``headroom_retrieve`` has nothing to expand otherwise. These
|
|
tests are about what happens *on* that path, so they have to earn it.
|
|
"""
|
|
from headroom.cache.backends import InMemoryBackend
|
|
from headroom.cache.compression_store import get_compression_store, reset_compression_store
|
|
|
|
reset_compression_store()
|
|
store = get_compression_store(backend=InMemoryBackend())
|
|
hash_key = store.store(
|
|
"the original, uncompressed tool output",
|
|
"<<ccr:placeholder>>",
|
|
original_tokens=100,
|
|
compressed_tokens=5,
|
|
tool_name="Read",
|
|
)
|
|
try:
|
|
yield hash_key
|
|
finally:
|
|
reset_compression_store()
|
|
|
|
|
|
def _body(*, with_thinking: bool, marker: str | None = None) -> dict:
|
|
# The marker rides in the latest user turn: only content new this turn
|
|
# gates buffering (#3587).
|
|
latest = "hi" if marker is None else f"hi — earlier output is at <<ccr:{marker}>>"
|
|
messages: list[dict] = [{"role": "user", "content": "hi" if with_thinking else latest}]
|
|
if with_thinking:
|
|
messages.append(SIGNED_THINKING_TURN)
|
|
messages.append({"role": "user", "content": latest})
|
|
return {
|
|
"model": "claude-sonnet-4-20250514",
|
|
"max_tokens": 64,
|
|
"stream": True,
|
|
"tools": [RETRIEVE_TOOL],
|
|
"messages": messages,
|
|
}
|
|
|
|
|
|
def _headers() -> dict[str, str]:
|
|
return {"Authorization": "Bearer test-key", "x-api-key": "test-key"}
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("with_thinking", "relaxation_enabled", "expect_plain_streaming"),
|
|
[
|
|
# Locked (kill switch engaged): the flip cannot reach upstream, so
|
|
# buffering would ask for a stream and then parse it as JSON, stranding
|
|
# the client with an unreadable 200. This is #2952 exactly.
|
|
(True, False, True),
|
|
# Relaxed (default): no transform touched the thinking block, so the
|
|
# flip DOES land and buffered retrieval becomes the coherent choice --
|
|
# the outcome #2952 wanted before the blanket lock made it unreachable.
|
|
(True, True, False),
|
|
# No thinking block: unaffected in either regime.
|
|
(False, True, False),
|
|
(False, False, False),
|
|
],
|
|
)
|
|
def test_signed_thinking_history_skips_the_buffered_ccr_path(
|
|
with_thinking: bool,
|
|
relaxation_enabled: bool,
|
|
expect_plain_streaming: bool,
|
|
ccr_marker: str,
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
) -> None:
|
|
"""The buffered path is only chosen when the stream:false flip can land."""
|
|
monkeypatch.setenv("HEADROOM_THINKING_PRESERVING_MUTATIONS", "1" if relaxation_enabled else "0")
|
|
calls: dict[str, object] = {}
|
|
|
|
async def fake_stream_response(url, headers, body, *args, **kwargs): # noqa: ANN001
|
|
calls["stream_body"] = body
|
|
return fastapi.responses.StreamingResponse(iter([SSE_BODY]), media_type="text/event-stream")
|
|
|
|
async def fake_retry(method, url, headers, req_body, *args, **kwargs): # noqa: ANN001
|
|
calls["buffered_body"] = json.loads(json.dumps(req_body))
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "msg_1",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"model": "claude-sonnet-4-20250514",
|
|
"content": [{"type": "text", "text": "ok"}],
|
|
"stop_reason": "end_turn",
|
|
"usage": {"input_tokens": 10, "output_tokens": 5},
|
|
},
|
|
headers={"content-type": "application/json"},
|
|
)
|
|
|
|
app = create_app(_config())
|
|
with TestClient(app) as client:
|
|
client.app.state.proxy._stream_response = fake_stream_response
|
|
client.app.state.proxy._retry_request = fake_retry
|
|
resp = client.post(
|
|
"/v1/messages",
|
|
json=_body(with_thinking=with_thinking, marker=ccr_marker),
|
|
headers=_headers(),
|
|
)
|
|
|
|
assert resp.status_code == 200, resp.text
|
|
if expect_plain_streaming:
|
|
# Passthrough is locked in, so we must not pretend we can buffer.
|
|
assert "stream_body" in calls, "expected the plain streaming path"
|
|
assert "buffered_body" not in calls
|
|
# The turn still leaves as a streaming request, matching the bytes
|
|
# that passthrough will actually forward.
|
|
assert calls["stream_body"]["stream"] is True
|
|
else:
|
|
assert "buffered_body" in calls, "expected the buffered CCR path"
|
|
# The buffered request itself streams; the proxy folds the reply.
|
|
assert calls["buffered_body"]["stream"] is True
|
|
|
|
|
|
@pytest.mark.parametrize("upstream_delay", [0.0, 1.2], ids=["prompt", "past-keepalive"])
|
|
def test_buffered_ccr_relays_an_unexpected_sse_reply_and_does_not_cache_it(
|
|
upstream_delay: float, ccr_marker: str
|
|
) -> None:
|
|
"""A 200 SSE reply on the buffered path reaches the client as a stream.
|
|
|
|
The delay matters: ``_BufferedCCRResponse`` commits SSE response headers
|
|
after a 1 s keepalive, and past that point it can only forward a result
|
|
that exposes a ``body_iterator``. A plain ``Response`` there degrades to a
|
|
bare ``event: error`` — which is what a real (multi-second) Anthropic turn
|
|
hit in #2952.
|
|
"""
|
|
|
|
async def fake_retry(method, url, headers, req_body, *args, **kwargs): # noqa: ANN001
|
|
if upstream_delay:
|
|
await asyncio.sleep(upstream_delay)
|
|
return httpx.Response(200, content=SSE_BODY, headers={"content-type": "text/event-stream"})
|
|
|
|
app = create_app(_config())
|
|
with TestClient(app) as client:
|
|
proxy = client.app.state.proxy
|
|
proxy._retry_request = fake_retry
|
|
resp = client.post(
|
|
"/v1/messages",
|
|
json=_body(with_thinking=False, marker=ccr_marker),
|
|
headers=_headers(),
|
|
)
|
|
|
|
assert resp.status_code == 200, resp.text
|
|
assert resp.headers["content-type"].startswith("text/event-stream")
|
|
assert b"message_start" in resp.content
|
|
# Caching SSE bytes under a key with no `stream` component is what
|
|
# served a stream to a buffered caller in the first place.
|
|
assert proxy.cache is not None
|
|
assert len(proxy.cache._cache) == 0, "an unparseable body must never be cached"
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("marker_kind", "expect_buffered"),
|
|
[
|
|
("owned", True),
|
|
("none", False),
|
|
("foreign", False),
|
|
],
|
|
)
|
|
def test_buffering_is_gated_on_a_redeemable_marker(
|
|
marker_kind: str, expect_buffered: bool, ccr_marker: str
|
|
) -> None:
|
|
"""A resident ``headroom_retrieve`` is not on its own a reason to buffer (#3071).
|
|
|
|
The tool is injected once and kept resident so the tools array stays
|
|
byte-stable for the prompt cache. Buffering on its presence alone meant
|
|
every later streaming turn of a sticky session lost incremental delivery —
|
|
time-to-first-byte became the whole generation. Retrieval can only expand a
|
|
marker that is in the outgoing body *and* redeemable now, so that is what
|
|
the wire-format decision keys on.
|
|
"""
|
|
marker = {
|
|
"owned": ccr_marker,
|
|
"none": None,
|
|
# Correct shape, not ours: adopting it would send the model to a
|
|
# retrieval that is guaranteed to miss (#2836).
|
|
"foreign": "deadbeefcafe",
|
|
}[marker_kind]
|
|
|
|
calls: dict[str, object] = {}
|
|
|
|
async def fake_stream_response(url, headers, body, *args, **kwargs): # noqa: ANN001
|
|
calls["stream_body"] = body
|
|
return fastapi.responses.StreamingResponse(iter([SSE_BODY]), media_type="text/event-stream")
|
|
|
|
async def fake_retry(method, url, headers, req_body, *args, **kwargs): # noqa: ANN001
|
|
calls["buffered_body"] = json.loads(json.dumps(req_body))
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "msg_1",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"model": "claude-sonnet-4-20250514",
|
|
"content": [{"type": "text", "text": "ok"}],
|
|
"stop_reason": "end_turn",
|
|
"usage": {"input_tokens": 10, "output_tokens": 5},
|
|
},
|
|
headers={"content-type": "application/json"},
|
|
)
|
|
|
|
app = create_app(_config())
|
|
with TestClient(app) as client:
|
|
client.app.state.proxy._stream_response = fake_stream_response
|
|
client.app.state.proxy._retry_request = fake_retry
|
|
resp = client.post(
|
|
"/v1/messages",
|
|
json=_body(with_thinking=False, marker=marker),
|
|
headers=_headers(),
|
|
)
|
|
|
|
assert resp.status_code == 200, resp.text
|
|
if expect_buffered:
|
|
assert "buffered_body" in calls, "a redeemable marker must still buffer"
|
|
# The buffered request itself streams; the proxy folds the reply.
|
|
assert calls["buffered_body"]["stream"] is True
|
|
else:
|
|
assert "stream_body" in calls, "nothing to retrieve — the client must keep streaming"
|
|
assert "buffered_body" not in calls
|
|
assert calls["stream_body"]["stream"] is True
|
|
|
|
|
|
def _owned_marker(content: str) -> str:
|
|
"""Store one more original in the (fixture-reset) store; return its hash."""
|
|
from headroom.cache.compression_store import get_compression_store
|
|
|
|
return get_compression_store().store(
|
|
content,
|
|
"<<ccr:placeholder>>",
|
|
original_tokens=100,
|
|
compressed_tokens=5,
|
|
tool_name="Read",
|
|
)
|
|
|
|
|
|
def _tool_turn(tool_use_id: str, output: str) -> list[dict]:
|
|
return [
|
|
{
|
|
"role": "assistant",
|
|
"content": [{"type": "tool_use", "id": tool_use_id, "name": "Read", "input": {}}],
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": [{"type": "tool_result", "tool_use_id": tool_use_id, "content": output}],
|
|
},
|
|
]
|
|
|
|
|
|
def _agent_body(messages: list[dict]) -> dict:
|
|
return {
|
|
"model": "claude-sonnet-4-20250514",
|
|
"max_tokens": 64,
|
|
"stream": True,
|
|
"tools": [RETRIEVE_TOOL],
|
|
"messages": messages,
|
|
}
|
|
|
|
|
|
def _replay_session(turns: list[list[dict]]) -> list[str]:
|
|
"""Send each message list as one turn of the same session; report the path taken."""
|
|
paths: list[str] = []
|
|
|
|
async def fake_stream_response(url, headers, body, *args, **kwargs): # noqa: ANN001
|
|
paths.append("streamed")
|
|
return fastapi.responses.StreamingResponse(iter([SSE_BODY]), media_type="text/event-stream")
|
|
|
|
async def fake_retry(method, url, headers, req_body, *args, **kwargs): # noqa: ANN001
|
|
paths.append("buffered")
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "msg_1",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"model": "claude-sonnet-4-20250514",
|
|
"content": [{"type": "text", "text": "ok"}],
|
|
"stop_reason": "end_turn",
|
|
"usage": {"input_tokens": 10, "output_tokens": 5},
|
|
},
|
|
headers={"content-type": "application/json"},
|
|
)
|
|
|
|
app = create_app(_config())
|
|
with TestClient(app) as client:
|
|
client.app.state.proxy._stream_response = fake_stream_response
|
|
client.app.state.proxy._retry_request = fake_retry
|
|
for messages in turns:
|
|
resp = client.post("/v1/messages", json=_agent_body(messages), headers=_headers())
|
|
assert resp.status_code == 200, resp.text
|
|
return paths
|
|
|
|
|
|
def test_a_marker_already_forwarded_does_not_buffer_later_turns(ccr_marker: str) -> None:
|
|
"""A marker carried forward in history does not make every later turn buffer.
|
|
|
|
The marker was in front of the model on the turn that introduced it, and
|
|
that turn buffered. Buffering every later turn for it is #3587: the
|
|
buffered share climbed toward 90% of turns with zero retrievals.
|
|
"""
|
|
first = [
|
|
{"role": "user", "content": "read the logs"},
|
|
*_tool_turn("tu_1", f"earlier output at <<ccr:{ccr_marker}>>"),
|
|
]
|
|
second = [*first, *_tool_turn("tu_2", "short")]
|
|
|
|
assert _replay_session([first, second]) == ["buffered", "streamed"]
|
|
|
|
|
|
def test_a_marker_first_shown_in_an_earlier_message_still_buffers(ccr_marker: str) -> None:
|
|
"""New to the model is what counts, not where the marker sits.
|
|
|
|
Token mode can compress an earlier tool result on a later request, so the
|
|
model first sees that marker in the middle of the history. Retrieval has
|
|
to be handled server-side there too, or the call reaches the client.
|
|
"""
|
|
newly_compressed = _owned_marker("a different, uncompressed tool output")
|
|
first = [
|
|
{"role": "user", "content": "read the logs"},
|
|
*_tool_turn("tu_1", f"earlier output at <<ccr:{ccr_marker}>>"),
|
|
]
|
|
second = [
|
|
{"role": "user", "content": "read the logs"},
|
|
*_tool_turn(
|
|
"tu_1",
|
|
f"earlier output at <<ccr:{ccr_marker}>> and <<ccr:{newly_compressed}>>",
|
|
),
|
|
*_tool_turn("tu_2", "short"),
|
|
]
|
|
|
|
assert _replay_session([first, second]) == ["buffered", "buffered"]
|
|
|
|
|
|
def test_cache_hit_never_replays_a_foreign_content_type() -> None:
|
|
"""A cache entry cannot hand a caller a wire format it did not ask for."""
|
|
body = {
|
|
"model": "claude-sonnet-4-20250514",
|
|
"max_tokens": 64,
|
|
"stream": False,
|
|
"messages": [{"role": "user", "content": "hi"}],
|
|
}
|
|
payload = json.dumps(
|
|
{
|
|
"id": "msg_1",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"model": "claude-sonnet-4-20250514",
|
|
"content": [{"type": "text", "text": "cached"}],
|
|
"stop_reason": "end_turn",
|
|
"usage": {"input_tokens": 1, "output_tokens": 1},
|
|
}
|
|
).encode()
|
|
|
|
async def fail_retry(*args, **kwargs): # noqa: ANN001, ANN002, ANN003
|
|
raise AssertionError("upstream must not be called on a cache hit")
|
|
|
|
app = create_app(_config())
|
|
with TestClient(app) as client:
|
|
proxy = client.app.state.proxy
|
|
proxy._retry_request = fail_retry
|
|
key = proxy.cache._compute_key(
|
|
body["messages"],
|
|
body["model"],
|
|
# Same partition the handler derives for this caller (01-F15).
|
|
partition=compute_cache_partition(_headers()),
|
|
upstream_base_url=None,
|
|
system=None,
|
|
tools=None,
|
|
tool_choice=None,
|
|
temperature=None,
|
|
top_p=None,
|
|
top_k=None,
|
|
max_tokens=64,
|
|
stop=None,
|
|
thinking=None,
|
|
output_config=None,
|
|
)
|
|
proxy.cache._cache[key] = CacheEntry(
|
|
response_body=payload,
|
|
response_headers={"content-type": "text/event-stream"},
|
|
created_at=datetime.now(),
|
|
ttl_seconds=3600,
|
|
)
|
|
|
|
resp = client.post("/v1/messages", json=body, headers=_headers())
|
|
|
|
assert resp.status_code == 200, resp.text
|
|
assert resp.headers["content-type"].startswith("application/json")
|
|
assert resp.json()["content"][0]["text"] == "cached"
|