Files
headroom/tests/test_ccr_buffered_stream_signed_thinking.py
SDandJD Davis 946a70cfcb fix(ccr): buffer a stream only for a marker new this turn (#4607)
## Description

The Anthropic handler flips a streaming request to the buffered
`stream:false` path when the outgoing body carries a redeemable CCR
marker (#3092). A marker stays in the conversation history for the rest
of the session, so once one tool result was compressed, every later turn
carried an owned marker and was buffered. On a buffered turn the client
receives nothing until the whole generation finishes.

The report shows the effect: 71% of Opus turns buffered (51% in a second
report on Linux), zero retrievals, the buffered share climbing from 0%
to 90% with proxy uptime, and time to first byte rising from about 1 s
to as much as 37 s.

This PR keeps buffering only for turns that show the model a marker it
has not seen yet, meaning one absent from what this session forwarded
last turn. Turns whose markers were all forwarded earlier stream. The
model can still ask for one of those originals, and the client never
declared `headroom_retrieve`, so on those turns the stream withholds the
call, the proxy serves it from the compression store, and the
continuation round streams into the same client message. This reuses the
filter and continuation-round machinery the proxy already uses for its
memory tools (#2195).

Closes #3587

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Performance improvement

## Changes Made

- `headroom/proxy/handlers/anthropic.py`:
- `_outgoing_redeemable_markers` returns `(any_owned, new_owned)`. "New"
is decided against the session's last forwarded messages (via the
existing `has_new_ccr_markers`), not by message position, so a marker
that token mode adds to an earlier tool result on this request still
counts as new. With no forwarded history, every owned marker counts as
new. `_outgoing_body_has_redeemable_marker` returns the second value.
- A new owned marker buffers, as on `main`. An owned marker that was
already forwarded streams with `serve_ccr_retrieval=True`. No owned
marker streams as before (#3071).
- `headroom/proxy/memory_tool_stream.py`:
`MemoryToolStreamFilter(hold_tool_use=True)` holds every block from the
round's first `tool_use` until the caller decides
(`release_held(drop_tool_use=...)`; `closing_frames` releases them).
Text before the first tool call still streams live. Off by default, so
the memory-tool behavior is unchanged.
- `headroom/proxy/handlers/streaming.py`:
`_stream_response(serve_ccr_retrieval=...)` adds `headroom_retrieve` to
the withheld tool names and holds client tool calls.
`_continue_memory_tool_stream(serve_retrieval=...)` serves withheld
retrievals and continues the round. A round that retrieved alongside a
client tool is continued without the unrun client call, for the model to
re-issue with the content in hand. That is what `handle_response`
already does on the buffered path.
- `headroom/ccr/response_handler.py`: `serve_withheld_retrievals` runs
withheld calls through the existing `_execute_retrieval` and counts them
in `ccr_retrievals`.
- Tests:
- New `tests/test_ccr_stream_retrieval.py`. Two turns of one session
through the real proxy app: turn 1 introduces an owned marker and
buffers; on turn 2 the upstream asks for that original.
- The original is served server-side, the answer reaches the client, and
no `headroom_retrieve` appears in the client's SSE.
- The same with a retrieval mixed with a client-owned tool, in both
orders: only the model's re-issued call reaches the client.
- An old-marker turn without a retrieval streams byte-identical to the
upstream.
    - Filter unit tests for hold mode.
- `tests/test_ccr_buffered_stream_signed_thinking.py`: two-turn tests
for the gate (a marker already forwarded does not buffer; a marker first
shown in an earlier message does).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`): see Additional Notes
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

The new streaming-retrieval tests on the previous head of this PR
(64164fc2), which reproduce the review finding:

```text
E   assert 'headroom_retrieve' not in ['headroom_retrieve']
FAILED tests/test_ccr_stream_retrieval.py::test_later_turn_retrieval_is_served_by_the_proxy
FAILED tests/test_ccr_stream_retrieval.py::test_retrieval_mixed_with_a_client_tool[True]
FAILED tests/test_ccr_stream_retrieval.py::test_retrieval_mixed_with_a_client_tool[False]
3 failed, 1 passed
```

At this head (d10a7446):

```text
$ pytest tests/test_ccr_stream_retrieval.py tests/test_ccr_buffered_stream_signed_thinking.py tests/test_memory_tool_stream.py -q
79 passed

$ pytest <every test file with ccr, anthropic, stream, buffer, memory or proxy in its name> tests/test_proxy -m "not live" -q
17 failed, 4504 passed, 129 skipped

$ ruff check headroom tests/test_ccr_stream_retrieval.py && ruff format --check headroom tests/test_ccr_stream_retrieval.py
All checks passed!
```

The 17 failures are in `tests/test_proxy_uds.py`,
`tests/test_upstream_guard.py` and `tests/test_ccr_batch_processor.py`.
The identical set fails on `main` (fc1ba891) in my environment, so they
are environment-related and not from this change.

## Real Behavior Proof

- Environment: macOS (Darwin 24.6.0, arm64), Python 3.12.3. The real
proxy app served by uvicorn in-process, cache mode, CCR on,
`headroom_retrieve` resident. Only the upstream model is faked, at the
HTTP client the proxy uses, so the proxy's own buffering, streaming
filter and continuation rounds all run. A streamed answer is 100 text
tokens at 15 ms each; a buffered answer arrives after 1.5 s; on a
retrieval turn the model writes 5 tokens, calls `headroom_retrieve` for
an earlier marker, then writes the 100-token answer once the original
comes back.
- Exact command / steps: replay a 30-turn agent session through a
streaming HTTP client. New owned markers appear on turns 1, 8, 15 and
22, every turn carries the earlier ones, and on turns 4, 11, 18 and 26
the model retrieves an earlier original. Per turn, record buffered or
streamed, time to the first text token, time to the first answer token,
whether `headroom_retrieve` reached the client, and whether the answer
arrived. Two alternating rounds each of `main` (fc1ba891), the previous
head of this PR (64164fc2) and this head (d10a7446).
- Observed result: `main` buffered 30 of 30 turns (time-to-first-token
p50 1,516 ms) and served all 8 retrievals; the previous head buffered 4
turns (p50 24 ms) but leaked all 8 retrievals to the client, which never
got an answer on those turns; this head buffers 4 turns (p50 24 ms),
serves all 8 retrievals server-side, leaks none, and the answer on a
retrieval turn starts at 105 ms. Full numbers below.

| Build | Turns buffered | TTFT p50, all turns | Retrievals served |
Leaked to client | Retrieval turns answered | Answer starts (retrieval
turns, p50) |
|---|---|---|---|---|---|---|
| `main` (fc1ba891) | 30 of 30 | 1,515.8 ms | 8 of 8 | 0 | 8 of 8 |
1,591.0 ms |
| previous head (64164fc2) | 4 of 30 | 24.3 ms | 0 of 8 | 8 | 0 of 8 | —
|
| this head (d10a7446) | 4 of 30 | 23.9 ms | 8 of 8 | 0 | 8 of 8 | 105.3
ms |

The two rounds of each build matched on every count. In this setup a
buffered turn's first token arrives when the whole generation is done
(1.5 s); with real replies of 3–37 s, as in the report, each avoided
buffered turn saves correspondingly more.
- Not tested: a live Anthropic upstream.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes. Turns whose redeemable markers
were all forwarded on an earlier turn now stream instead of buffering,
with `headroom_retrieve` withheld from the client and served by the
proxy. On those turns, client tool-call blocks reach the client when the
round ends rather than as they stream; text is unchanged. Turns that
show the model a new marker buffer as before; turns with no owned marker
stream as before.
- Kill switch / disable path: none added. Existing CCR controls are
unchanged.
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the two commits

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation: n/a, no
user-facing docs describe this gate
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Screenshots (if applicable)

n/a

## Additional Notes

- Review follow-ups:
- The first revision decided "new" by position; 64164fc2 decides it from
what the session forwarded.
- 64164fc2 let a later-turn retrieval reach the client; d10a7446 serves
it in the stream. The earlier "tradeoff" note no longer applies.
- On a round that retrieved alongside a client tool, the client call is
dropped and the model decides again with the content in hand, matching
the buffered path. If the continuation round itself fails, the client
receives the upstream error frame, as with the memory-tool continuation.
- The issue also asks for a config switch to disable the buffered CCR
path entirely. That is left out to keep this change focused.
- `mypy headroom/proxy/handlers/anthropic.py` stops in my local
environment on a `numpy` stub (`Type statement is only supported in
Python 3.12 and greater`) before reaching project code. CI runs mypy.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 23:08:37 -06:00

458 lines
17 KiB
Python

"""Buffered-CCR streaming vs. byte-faithful passthrough (issue #2952).
The buffered-CCR path is the one place the Anthropic handler changes the
request *for its own benefit*: it flips ``stream`` to False so the reply comes
back as one JSON document it can inspect for ``headroom_retrieve`` calls, then
resynthesizes SSE for the client.
That only works if the flip reaches the wire. When conversation history carries
a signed ``thinking`` block, ``select_outbound_body`` forwards the client's
original bytes instead — ``"stream": true`` and all — so upstream streams, the
JSON parse fails, resynthesis is skipped, and the client is left with a 200 and
nothing it can read. These tests pin the three defenses: don't take the path,
survive the reply if we somehow do, and never cache a body in the wrong format.
"""
from __future__ import annotations
import asyncio
import json
from datetime import datetime
import pytest
fastapi = pytest.importorskip("fastapi")
httpx = pytest.importorskip("httpx")
from fastapi.testclient import TestClient # noqa: E402
from headroom.proxy.models import CacheEntry # noqa: E402
from headroom.proxy.semantic_cache_key_policy import compute_cache_partition # noqa: E402
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
RETRIEVE_TOOL = {
"name": "headroom_retrieve",
"description": "Retrieve original content",
"input_schema": {"type": "object", "properties": {}},
}
SIGNED_THINKING_TURN = {
"role": "assistant",
"content": [
{
"type": "thinking",
"thinking": "private reasoning",
"signature": "sig-abc123",
},
{"type": "text", "text": "Answered."},
],
}
SSE_BODY = (
b'event: message_start\ndata: {"type":"message_start","message":{"id":"msg_1"}}\n\n'
b'event: message_stop\ndata: {"type":"message_stop"}\n\n'
)
def _config() -> ProxyConfig:
return ProxyConfig(
optimize=False,
cache_enabled=True,
rate_limit_enabled=False,
memory_enabled=False,
)
@pytest.fixture
def ccr_marker() -> str:
"""A marker this proxy actually owns, so retrieval could really fire.
The buffered path is only taken when the outgoing body carries a redeemable
marker (#3071) — ``headroom_retrieve`` has nothing to expand otherwise. These
tests are about what happens *on* that path, so they have to earn it.
"""
from headroom.cache.backends import InMemoryBackend
from headroom.cache.compression_store import get_compression_store, reset_compression_store
reset_compression_store()
store = get_compression_store(backend=InMemoryBackend())
hash_key = store.store(
"the original, uncompressed tool output",
"<<ccr:placeholder>>",
original_tokens=100,
compressed_tokens=5,
tool_name="Read",
)
try:
yield hash_key
finally:
reset_compression_store()
def _body(*, with_thinking: bool, marker: str | None = None) -> dict:
# The marker rides in the latest user turn: only content new this turn
# gates buffering (#3587).
latest = "hi" if marker is None else f"hi — earlier output is at <<ccr:{marker}>>"
messages: list[dict] = [{"role": "user", "content": "hi" if with_thinking else latest}]
if with_thinking:
messages.append(SIGNED_THINKING_TURN)
messages.append({"role": "user", "content": latest})
return {
"model": "claude-sonnet-4-20250514",
"max_tokens": 64,
"stream": True,
"tools": [RETRIEVE_TOOL],
"messages": messages,
}
def _headers() -> dict[str, str]:
return {"Authorization": "Bearer test-key", "x-api-key": "test-key"}
@pytest.mark.parametrize(
("with_thinking", "relaxation_enabled", "expect_plain_streaming"),
[
# Locked (kill switch engaged): the flip cannot reach upstream, so
# buffering would ask for a stream and then parse it as JSON, stranding
# the client with an unreadable 200. This is #2952 exactly.
(True, False, True),
# Relaxed (default): no transform touched the thinking block, so the
# flip DOES land and buffered retrieval becomes the coherent choice --
# the outcome #2952 wanted before the blanket lock made it unreachable.
(True, True, False),
# No thinking block: unaffected in either regime.
(False, True, False),
(False, False, False),
],
)
def test_signed_thinking_history_skips_the_buffered_ccr_path(
with_thinking: bool,
relaxation_enabled: bool,
expect_plain_streaming: bool,
ccr_marker: str,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""The buffered path is only chosen when the stream:false flip can land."""
monkeypatch.setenv("HEADROOM_THINKING_PRESERVING_MUTATIONS", "1" if relaxation_enabled else "0")
calls: dict[str, object] = {}
async def fake_stream_response(url, headers, body, *args, **kwargs): # noqa: ANN001
calls["stream_body"] = body
return fastapi.responses.StreamingResponse(iter([SSE_BODY]), media_type="text/event-stream")
async def fake_retry(method, url, headers, req_body, *args, **kwargs): # noqa: ANN001
calls["buffered_body"] = json.loads(json.dumps(req_body))
return httpx.Response(
200,
json={
"id": "msg_1",
"type": "message",
"role": "assistant",
"model": "claude-sonnet-4-20250514",
"content": [{"type": "text", "text": "ok"}],
"stop_reason": "end_turn",
"usage": {"input_tokens": 10, "output_tokens": 5},
},
headers={"content-type": "application/json"},
)
app = create_app(_config())
with TestClient(app) as client:
client.app.state.proxy._stream_response = fake_stream_response
client.app.state.proxy._retry_request = fake_retry
resp = client.post(
"/v1/messages",
json=_body(with_thinking=with_thinking, marker=ccr_marker),
headers=_headers(),
)
assert resp.status_code == 200, resp.text
if expect_plain_streaming:
# Passthrough is locked in, so we must not pretend we can buffer.
assert "stream_body" in calls, "expected the plain streaming path"
assert "buffered_body" not in calls
# The turn still leaves as a streaming request, matching the bytes
# that passthrough will actually forward.
assert calls["stream_body"]["stream"] is True
else:
assert "buffered_body" in calls, "expected the buffered CCR path"
# The buffered request itself streams; the proxy folds the reply.
assert calls["buffered_body"]["stream"] is True
@pytest.mark.parametrize("upstream_delay", [0.0, 1.2], ids=["prompt", "past-keepalive"])
def test_buffered_ccr_relays_an_unexpected_sse_reply_and_does_not_cache_it(
upstream_delay: float, ccr_marker: str
) -> None:
"""A 200 SSE reply on the buffered path reaches the client as a stream.
The delay matters: ``_BufferedCCRResponse`` commits SSE response headers
after a 1 s keepalive, and past that point it can only forward a result
that exposes a ``body_iterator``. A plain ``Response`` there degrades to a
bare ``event: error`` — which is what a real (multi-second) Anthropic turn
hit in #2952.
"""
async def fake_retry(method, url, headers, req_body, *args, **kwargs): # noqa: ANN001
if upstream_delay:
await asyncio.sleep(upstream_delay)
return httpx.Response(200, content=SSE_BODY, headers={"content-type": "text/event-stream"})
app = create_app(_config())
with TestClient(app) as client:
proxy = client.app.state.proxy
proxy._retry_request = fake_retry
resp = client.post(
"/v1/messages",
json=_body(with_thinking=False, marker=ccr_marker),
headers=_headers(),
)
assert resp.status_code == 200, resp.text
assert resp.headers["content-type"].startswith("text/event-stream")
assert b"message_start" in resp.content
# Caching SSE bytes under a key with no `stream` component is what
# served a stream to a buffered caller in the first place.
assert proxy.cache is not None
assert len(proxy.cache._cache) == 0, "an unparseable body must never be cached"
@pytest.mark.parametrize(
("marker_kind", "expect_buffered"),
[
("owned", True),
("none", False),
("foreign", False),
],
)
def test_buffering_is_gated_on_a_redeemable_marker(
marker_kind: str, expect_buffered: bool, ccr_marker: str
) -> None:
"""A resident ``headroom_retrieve`` is not on its own a reason to buffer (#3071).
The tool is injected once and kept resident so the tools array stays
byte-stable for the prompt cache. Buffering on its presence alone meant
every later streaming turn of a sticky session lost incremental delivery —
time-to-first-byte became the whole generation. Retrieval can only expand a
marker that is in the outgoing body *and* redeemable now, so that is what
the wire-format decision keys on.
"""
marker = {
"owned": ccr_marker,
"none": None,
# Correct shape, not ours: adopting it would send the model to a
# retrieval that is guaranteed to miss (#2836).
"foreign": "deadbeefcafe",
}[marker_kind]
calls: dict[str, object] = {}
async def fake_stream_response(url, headers, body, *args, **kwargs): # noqa: ANN001
calls["stream_body"] = body
return fastapi.responses.StreamingResponse(iter([SSE_BODY]), media_type="text/event-stream")
async def fake_retry(method, url, headers, req_body, *args, **kwargs): # noqa: ANN001
calls["buffered_body"] = json.loads(json.dumps(req_body))
return httpx.Response(
200,
json={
"id": "msg_1",
"type": "message",
"role": "assistant",
"model": "claude-sonnet-4-20250514",
"content": [{"type": "text", "text": "ok"}],
"stop_reason": "end_turn",
"usage": {"input_tokens": 10, "output_tokens": 5},
},
headers={"content-type": "application/json"},
)
app = create_app(_config())
with TestClient(app) as client:
client.app.state.proxy._stream_response = fake_stream_response
client.app.state.proxy._retry_request = fake_retry
resp = client.post(
"/v1/messages",
json=_body(with_thinking=False, marker=marker),
headers=_headers(),
)
assert resp.status_code == 200, resp.text
if expect_buffered:
assert "buffered_body" in calls, "a redeemable marker must still buffer"
# The buffered request itself streams; the proxy folds the reply.
assert calls["buffered_body"]["stream"] is True
else:
assert "stream_body" in calls, "nothing to retrieve — the client must keep streaming"
assert "buffered_body" not in calls
assert calls["stream_body"]["stream"] is True
def _owned_marker(content: str) -> str:
"""Store one more original in the (fixture-reset) store; return its hash."""
from headroom.cache.compression_store import get_compression_store
return get_compression_store().store(
content,
"<<ccr:placeholder>>",
original_tokens=100,
compressed_tokens=5,
tool_name="Read",
)
def _tool_turn(tool_use_id: str, output: str) -> list[dict]:
return [
{
"role": "assistant",
"content": [{"type": "tool_use", "id": tool_use_id, "name": "Read", "input": {}}],
},
{
"role": "user",
"content": [{"type": "tool_result", "tool_use_id": tool_use_id, "content": output}],
},
]
def _agent_body(messages: list[dict]) -> dict:
return {
"model": "claude-sonnet-4-20250514",
"max_tokens": 64,
"stream": True,
"tools": [RETRIEVE_TOOL],
"messages": messages,
}
def _replay_session(turns: list[list[dict]]) -> list[str]:
"""Send each message list as one turn of the same session; report the path taken."""
paths: list[str] = []
async def fake_stream_response(url, headers, body, *args, **kwargs): # noqa: ANN001
paths.append("streamed")
return fastapi.responses.StreamingResponse(iter([SSE_BODY]), media_type="text/event-stream")
async def fake_retry(method, url, headers, req_body, *args, **kwargs): # noqa: ANN001
paths.append("buffered")
return httpx.Response(
200,
json={
"id": "msg_1",
"type": "message",
"role": "assistant",
"model": "claude-sonnet-4-20250514",
"content": [{"type": "text", "text": "ok"}],
"stop_reason": "end_turn",
"usage": {"input_tokens": 10, "output_tokens": 5},
},
headers={"content-type": "application/json"},
)
app = create_app(_config())
with TestClient(app) as client:
client.app.state.proxy._stream_response = fake_stream_response
client.app.state.proxy._retry_request = fake_retry
for messages in turns:
resp = client.post("/v1/messages", json=_agent_body(messages), headers=_headers())
assert resp.status_code == 200, resp.text
return paths
def test_a_marker_already_forwarded_does_not_buffer_later_turns(ccr_marker: str) -> None:
"""A marker carried forward in history does not make every later turn buffer.
The marker was in front of the model on the turn that introduced it, and
that turn buffered. Buffering every later turn for it is #3587: the
buffered share climbed toward 90% of turns with zero retrievals.
"""
first = [
{"role": "user", "content": "read the logs"},
*_tool_turn("tu_1", f"earlier output at <<ccr:{ccr_marker}>>"),
]
second = [*first, *_tool_turn("tu_2", "short")]
assert _replay_session([first, second]) == ["buffered", "streamed"]
def test_a_marker_first_shown_in_an_earlier_message_still_buffers(ccr_marker: str) -> None:
"""New to the model is what counts, not where the marker sits.
Token mode can compress an earlier tool result on a later request, so the
model first sees that marker in the middle of the history. Retrieval has
to be handled server-side there too, or the call reaches the client.
"""
newly_compressed = _owned_marker("a different, uncompressed tool output")
first = [
{"role": "user", "content": "read the logs"},
*_tool_turn("tu_1", f"earlier output at <<ccr:{ccr_marker}>>"),
]
second = [
{"role": "user", "content": "read the logs"},
*_tool_turn(
"tu_1",
f"earlier output at <<ccr:{ccr_marker}>> and <<ccr:{newly_compressed}>>",
),
*_tool_turn("tu_2", "short"),
]
assert _replay_session([first, second]) == ["buffered", "buffered"]
def test_cache_hit_never_replays_a_foreign_content_type() -> None:
"""A cache entry cannot hand a caller a wire format it did not ask for."""
body = {
"model": "claude-sonnet-4-20250514",
"max_tokens": 64,
"stream": False,
"messages": [{"role": "user", "content": "hi"}],
}
payload = json.dumps(
{
"id": "msg_1",
"type": "message",
"role": "assistant",
"model": "claude-sonnet-4-20250514",
"content": [{"type": "text", "text": "cached"}],
"stop_reason": "end_turn",
"usage": {"input_tokens": 1, "output_tokens": 1},
}
).encode()
async def fail_retry(*args, **kwargs): # noqa: ANN001, ANN002, ANN003
raise AssertionError("upstream must not be called on a cache hit")
app = create_app(_config())
with TestClient(app) as client:
proxy = client.app.state.proxy
proxy._retry_request = fail_retry
key = proxy.cache._compute_key(
body["messages"],
body["model"],
# Same partition the handler derives for this caller (01-F15).
partition=compute_cache_partition(_headers()),
upstream_base_url=None,
system=None,
tools=None,
tool_choice=None,
temperature=None,
top_p=None,
top_k=None,
max_tokens=64,
stop=None,
thinking=None,
output_config=None,
)
proxy.cache._cache[key] = CacheEntry(
response_body=payload,
response_headers={"content-type": "text/event-stream"},
created_at=datetime.now(),
ttl_seconds=3600,
)
resp = client.post("/v1/messages", json=body, headers=_headers())
assert resp.status_code == 200, resp.text
assert resp.headers["content-type"].startswith("application/json")
assert resp.json()["content"][0]["text"] == "cached"