feat(ccr): answer headroom_retrieve on streaming chat completions (#4631)

## Description

Streaming `/v1/chat/completions` compresses tool output into retrieval
markers but cannot answer `headroom_retrieve`, so the content is lost to
the model on this path. Sessions that mix streaming and non-streaming
turns also flip the tool in and out of `tools` and bust the prefix each
way.

Behind a new default-off flag, a streaming turn that carries a marker
now runs as `stream: false` upstream through the same code as a
non-streaming turn, then goes back as `chat.completion.chunk` frames.
Builds on the groundwork in `feat/buffered-ccr-chat-streaming` and the
#3816 continuation loop.

Part of #3815. Flipping the default stays a separate decision, as the
issue asks.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `--ccr-buffered-chat-streaming` /
`HEADROOM_CCR_BUFFERED_CHAT_STREAMING`, on the CLI too because `headroom
proxy` builds its own config.
- The non-streaming branch became `_direct_chat_turn()` at the same
indent, so usage, the prefix tracker and savings are shared rather than
copied. `git diff` shows almost no moved lines there.
- Two gates. Eligibility is decided before injection and lets the tool
stay resident. Buffering is decided at dispatch from the final body, so
a turn with no marker keeps its real stream (the #3071 lesson).
- `_buffered_chat_reply_as_sse` wraps the JSON reply: non-200 passes
through with its status, a lone leftover retrieval fails closed with the
continuation's status. A retrieval next to a client tool follows
`handle_response`: a successful continuation replaces the turn, and only
a failed one goes back to the client whole.
- `stream_options` is dropped upstream since OpenAI answers 400 to it
without a stream. The usage chunk follows the same rule as a plain
stream.
- Frames and the OpenAI continuation now carry other non-empty message
fields, and frames keep opaque tool call fields, so `reasoning_content`,
`reasoning_details` and Gemini thought signatures survive.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`): see Test Output, no errors
in changed files
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest <every file touching the chat route, handle_response, ccr, buffered or settings_store>
3067 passed, 73 skipped
$ ruff check . && ruff format --check .
All checks passed! 1881 files already formatted
$ mypy headroom
2 errors, both in files this PR does not touch (memory/adapters/embedders.py, integrations/langchain/chat_model.py)
```

The full suite did not finish locally (stopped at a time limit about 40%
through, no failures to that point), so CI is the full run. `cargo`
checks were not run since no Rust changed.
`tests/test_openai_chat_buffered_ccr.py` covers the list in the issue:
frames ending in `[DONE]`, `finish_reason: tool_calls`, status fidelity
inside the grace window, heartbeat past it, identical accounting against
a non-streaming turn, byte-stable tools across a mixed session, flag off
stays on the plain stream.

## Real Behavior Proof

- Environment: macOS 27 arm64, Python 3.13, real `headroom proxy --mode
token` in front of a local stub upstream, real `openai` 2.41 SDK client
with `stream=True`. No provider keys.
- Exact command / steps: `headroom proxy --port 18932 --mode token
--openai-api-url http://127.0.0.1:18931
[--ccr-buffered-chat-streaming]`, then one turn with a 1,500 line log as
the tool result. It compresses to a `Retrieve more: hash=` marker. The
stub calls `headroom_retrieve` when it is offered and a marker is
present.
- Observed result: flag off, the model only gets the summary. Flag on,
the retrieval is answered inside the proxy and the SDK reads a normal
stream with usage.

```text
flag off
  upstream: stream=true  tools=[list_rows]
  client:   "I can only see a compressed summary."

flag on
  upstream: stream=false tools=[list_rows, headroom_retrieve] reply=tool_call
  upstream: (continuation)                                    reply=text
  log:      CCR: buffering streaming chat turn (stream:false upstream, resynthesized as chat.completion.chunk)
            CCR: retrieval tool call on the direct chat path, resolving before replying
  client:   4 chunks, "Row row-0007 is in the original output.", finish_reason=stop, usage present
```

- Not tested: a real provider, Copilot, xAI, or a gateway with its own
strict validation. Windows.

## Runtime Rollout Safety

- Rollout-managed feature(s): none, plain opt-in flag
- Minimum rollout channel: n/a
- Stable/default behavior changed: no
- Kill switch / disable path: flag is off by default
- Unsafe override required: no
- Qualification impact: none while off
- Rollback path: unset the flag

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Memory tools still only go in on non-streaming turns, so with memory
on a mixed session keeps that part of the toggle. A turn with no marker
cannot run the memory loop, so I left it.
- Turn hooks treat a buffered turn like a stream: stream-safe
`on_request` only, no `on_response`. Buffering is decided after hooks
run, so anything else would split a hook's two halves.
- The issue mentions a ChatGPT-auth exclusion. The chat handler has no
such routing, so there was nothing to exclude.
- Past the grace window `x-headroom-*` headers are lost, same as the
shared wrapper on the other paths.
- #3600 picks a marker-free pipeline for streaming chat. If it lands, an
eligible turn needs to keep the marker pipeline.

---------

Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
SK Arin
2026-11-04 14:41:30 -06:00
committed by GitHub
co-authored by Tejas Chopra JerrettDavis
parent 36540379e0
commit 31dcaf41f7
9 changed files with 1085 additions and 23 deletions
+1
View File
@@ -349,6 +349,7 @@ By default Headroom stores originals so the model can recover them via `headroom
|---|---|---|
| `--no-ccr` / `HEADROOM_NO_CCR` | CCR on | Disable retrieval markers **and** the injected `headroom_retrieve` tool. |
| `--lossless` / `HEADROOM_LOSSLESS` | `false` | Format-native lossless compaction only — no CCR marker, no retrieval tool. |
| `--ccr-buffered-chat-streaming` / `HEADROOM_CCR_BUFFERED_CHAT_STREAMING` | `false` | Answer `headroom_retrieve` on streaming `/v1/chat/completions`. A turn carrying a redeemable marker runs as `stream: false` upstream and is re-emitted as `chat.completion.chunk` frames, so its text arrives all at once; other turns keep their real stream |
| `--no-ccr-proactive-expansion` | expansion on | Stop proactively re-expanding compressed content when the model appears to need it. |
### File-read handling
+52 -5
View File
@@ -54,6 +54,20 @@ RESIDUAL_CCR_SKIPPED_MIXED = "skipped_mixed_tools"
RESIDUAL_CCR_ERROR = "error"
def _openai_message_extras(message: dict[str, Any]) -> dict[str, Any]:
"""The fields a provider put on a chat message besides the standard three.
Whoever replays the turn may have to send these back (DeepSeek's
``reasoning_content``, OpenRouter's ``reasoning_details``) and gets rejected
if they went missing. Empty ones (``refusal: null``) stay behind.
"""
return {
key: value
for key, value in message.items()
if key not in ("role", "content", "tool_calls") and value
}
@dataclass
class CCRToolResult:
"""Result of handling a CCR tool call."""
@@ -397,6 +411,7 @@ class CCRResponseHandler:
first = choices[0] if isinstance(choices, list) and choices else {}
message = first.get("message", {}) if isinstance(first, dict) else {}
return {
**_openai_message_extras(message),
"role": "assistant",
"content": message.get("content"),
"tool_calls": message.get("tool_calls"),
@@ -951,7 +966,8 @@ class StreamingCCRHandler:
response["usage"] = usage
return response
def _openai_response_to_chunks(self, response: dict[str, Any]) -> list[bytes]:
@staticmethod
def _openai_response_to_chunks(response: dict[str, Any]) -> list[bytes]:
"""Split a non-streaming ``chat.completion`` body into SSE chunk frames.
A streaming client reads ``choices[].delta``, not ``choices[].message``.
@@ -968,10 +984,7 @@ class StreamingCCRHandler:
message = {}
finish_reason = choice.get("finish_reason") or "stop"
base: dict[str, Any] = {"object": "chat.completion.chunk"}
for key in ("id", "created", "model", "system_fingerprint"):
if response.get(key) is not None:
base[key] = response[key]
base = StreamingCCRHandler._openai_chunk_envelope(response)
def frame(delta: dict[str, Any], reason: str | None) -> bytes:
payload = {
@@ -982,6 +995,10 @@ class StreamingCCRHandler:
frames = [frame({"role": message.get("role") or "assistant"}, None)]
extras = _openai_message_extras(message)
if extras:
frames.append(frame(extras, None))
content = message.get("content")
if content:
frames.append(frame({"content": content}, None))
@@ -999,10 +1016,15 @@ class StreamingCCRHandler:
{
"tool_calls": [
{
# Spread first: gateways hang opaque fields
# off a tool call that must survive the
# round trip (Gemini's thought signatures).
**tool_call,
"index": index,
"id": tool_call.get("id", ""),
"type": tool_call.get("type", "function"),
"function": {
**function,
"name": function.get("name", ""),
"arguments": function.get("arguments", ""),
},
@@ -1016,6 +1038,31 @@ class StreamingCCRHandler:
frames.append(frame({}, finish_reason))
return frames
@staticmethod
def _openai_chunk_envelope(response: dict[str, Any]) -> dict[str, Any]:
"""The fields every ``chat.completion.chunk`` frame of a reply repeats."""
envelope: dict[str, Any] = {"object": "chat.completion.chunk"}
for key in ("id", "created", "model", "system_fingerprint"):
if response.get(key) is not None:
envelope[key] = response[key]
return envelope
@staticmethod
def openai_chat_sse_frames(response: dict[str, Any], *, include_usage: bool) -> list[bytes]:
"""Render a whole ``chat.completion`` body as the SSE a stream would be.
``include_usage`` adds the trailing usage-only chunk (``choices: []``)
that ``stream_options.include_usage`` asks for.
"""
frames = StreamingCCRHandler._openai_response_to_chunks(response)
usage = response.get("usage")
if include_usage and usage is not None:
envelope = StreamingCCRHandler._openai_chunk_envelope(response)
payload = {**envelope, "choices": [], "usage": usage}
frames.append(f"data: {json.dumps(payload)}\n\n".encode())
frames.append(b"data: [DONE]\n\n")
return frames
async def _response_to_sse(
self,
response: dict[str, Any],
+13
View File
@@ -528,6 +528,17 @@ def dashboard(port: int, no_open: bool) -> None:
"Env: HEADROOM_CCR_INLINE_RESOLVE."
),
)
@click.option(
"--ccr-buffered-chat-streaming",
is_flag=True,
envvar="HEADROOM_CCR_BUFFERED_CHAT_STREAMING",
help=(
"Answer headroom_retrieve on streaming /v1/chat/completions. A turn that "
"carries a redeemable marker runs as stream:false upstream and is "
"re-emitted as chat.completion.chunk frames, so that turn's text arrives "
"all at once. Off by default. Env: HEADROOM_CCR_BUFFERED_CHAT_STREAMING."
),
)
@click.option(
"--no-ccr-proactive-expansion",
is_flag=True,
@@ -1178,6 +1189,7 @@ def proxy(
no_ccr: bool,
lossless: bool,
ccr_inline_resolve: bool,
ccr_buffered_chat_streaming: bool,
no_ccr_proactive_expansion: bool,
proxy_extension: tuple[str, ...],
compressor: tuple[str, ...],
@@ -1518,6 +1530,7 @@ def proxy(
ccr_inject_marker=not no_ccr,
ccr_handle_responses=not no_ccr,
ccr_resolve_markers_inline=ccr_inline_resolve,
ccr_buffered_chat_streaming=ccr_buffered_chat_streaming,
lossless=lossless,
ccr_proactive_expansion=not no_ccr_proactive_expansion,
# Flatten repeat-flag tuple AND any comma-separated values inside it.
+256 -18
View File
@@ -1399,16 +1399,171 @@ def _should_buffer_openai_responses_stream_ccr(
)
def _should_inject_openai_chat_ccr_tool(*, ccr_inject_tool: bool, stream: bool) -> bool:
def _should_inject_openai_chat_ccr_tool(
*, ccr_inject_tool: bool, stream: bool, buffered_eligible: bool = False
) -> bool:
"""Return whether chat-completions can redeem an injected CCR tool.
The chat streaming path forwards SSE events immediately and deliberately
does not run the response continuation loop. Injecting ``headroom_retrieve``
there makes OpenAI-compatible clients attempt an unknown tool call. The
A plain chat streaming turn forwards SSE events immediately and does not run
the response continuation loop, so injecting ``headroom_retrieve`` there
makes OpenAI-compatible clients attempt an unknown tool call. The
non-streaming path can intercept and resolve it; Responses streaming has a
separate buffered-CCR path and is unaffected by this predicate.
``buffered_eligible`` is the third case: if this turn carries a marker it
will run the continuation loop before replying, so the tool is redeemable.
If it carries none, the model has no hash to call it with and the tool stays
resident only to keep the tools array byte-stable.
"""
return bool(ccr_inject_tool and not stream)
return bool(ccr_inject_tool and (buffered_eligible or not stream))
def _is_openai_chat_stream_ccr_eligible(
*,
body: dict[str, Any],
enabled: bool,
ccr_response_handler_enabled: bool,
ccr_inject_tool: bool,
routed_to_backend: bool,
upstream_base_url: str | None = None,
) -> bool:
"""Return whether a streaming chat turn may answer retrievals by buffering.
Unlike :func:`_should_buffer_openai_responses_stream_ccr` this cannot key on
``headroom_retrieve`` already being in ``tools``: on this path that test is
what decides injection, so it would never pass.
Requiring a non-empty client ``tools`` array keeps the rule from #3810: a
request with no tools is not an agent harness and must not be handed a tools
array it never sent. The rest are turns buffering cannot serve:
- A backend route streams through its own path, with no continuation loop.
- ``n > 1`` and ``logprobs`` ask for output that resynthesis does not carry.
- OpenCode Zen rejects requests Headroom has reshaped (``stream:false`` plus
``accept: application/json``) with ``403 FreeTierError`` (#3656).
"""
n = body.get("n")
return bool(
body.get("stream")
and enabled
and ccr_response_handler_enabled
and ccr_inject_tool
and body.get("tools")
and not routed_to_backend
and not (isinstance(n, int) and n > 1)
and not body.get("logprobs")
and not is_opencode_zen_base(upstream_base_url)
)
def _chat_body_has_ccr_marker(body: dict[str, Any]) -> bool:
"""Is there anything in the outgoing messages a resident tool could be called for?
Goes by marker shape alone, without the ownership check that decides
injection (#2836). An expired hash can still draw a retrieval call once the
tool is resident, and only the continuation loop can answer that call.
"""
from headroom.ccr.tool_injection import CCRToolInjector
messages = body.get("messages")
if not isinstance(messages, list):
return False
probe = CCRToolInjector(provider="openai", inject_tool=False, inject_system_instructions=False)
try:
return bool(probe.scan_for_markers(messages))
except Exception: # pragma: no cover - defensive
logger.debug("CCR: marker probe failed; keeping the buffered path", exc_info=True)
return True
def _take_stream_options_for_buffered_chat(body: dict[str, Any]) -> bool:
"""Drop ``stream_options`` from a body going out as ``stream: false``.
OpenAI answers 400 to ``stream_options`` on a non-streaming request. Returns
whether the client should still get the usage chunk, decided by the same
rule a plain streamed turn uses so the two cannot drift apart.
"""
_apply_stream_usage_option(body)
stream_options = body.pop("stream_options", None)
return isinstance(stream_options, dict) and bool(stream_options.get("include_usage"))
def _buffered_chat_reply_as_sse(
result: Response,
*,
ccr_response_handler: Any,
include_usage: bool,
continuation_status: int | None,
request_id: str,
) -> Response:
"""Turn the direct chat path's JSON reply into what a streaming client reads.
Anything but a 200 goes back untouched. Inside the grace window that hands
the client the real status, and past it :func:`buffered_ccr_asgi_call` turns
it into a typed stream error (#2997, #3079).
"""
from fastapi.responses import JSONResponse, StreamingResponse
from headroom.ccr.response_handler import RESIDUAL_CCR_ERROR, StreamingCCRHandler
if result.status_code != 200:
return result
try:
reply = json.loads(result.body)
except (TypeError, ValueError):
reply = None
choices = reply.get("choices") if isinstance(reply, dict) else None
first_choice = choices[0] if isinstance(choices, list) and choices else None
if not isinstance(first_choice, dict) or not isinstance(first_choice.get("message"), dict):
logger.warning(f"[{request_id}] CCR: rejecting malformed buffered chat 200 reply")
return JSONResponse(
status_code=502,
content={
"error": {
"type": "upstream_protocol_error",
"message": "Upstream returned an invalid buffered response.",
}
},
)
sse_headers = {
k: v
for k, v in result.headers.items()
if k.lower() not in ("content-length", "content-type")
}
# A retrieval call standing alone means the continuation failed, and the
# client cannot act on a tool it never declared. Next to a client tool it is
# the turn handle_response gives back when a continuation fails, because the
# model made no newer decision, so that goes to the client as is. A plain JSON
# error works on both sides of the commit: the wrapper turns it into a typed
# stream error if SSE is already on the wire, so a rate limit on the
# continuation still reads as a rate limit.
if ccr_response_handler.residual_ccr_status(reply, "openai") == RESIDUAL_CCR_ERROR:
status = continuation_status or 502
logger.warning(
f"[{request_id}] CCR: buffered chat reply still contains headroom_retrieve "
f"after handling; failing closed with {status}"
)
return JSONResponse(
status_code=status,
headers=sse_headers,
content={
"error": {
"message": "Unable to safely complete streamed CCR retrieval.",
"type": OPENAI_ERROR_FORMAT.error_type_for_status(status),
}
},
)
frames = StreamingCCRHandler.openai_chat_sse_frames(reply, include_usage=include_usage)
async def _frames():
for frame in frames:
yield frame
return StreamingResponse(_frames(), media_type="text/event-stream", headers=sse_headers)
def _responses_input_to_items(input_data: Any) -> list[dict[str, Any]]:
@@ -4237,14 +4392,34 @@ class OpenAIHandlerMixin:
# anchored on the previous turn's tool list never busts.
tools = body.get("tools")
_original_tools = tools # Preserve for diagnostic / future retry
# Decided before injection: a turn that can buffer can answer the tool,
# so it may carry it even though the client asked for a stream. Whether
# it actually buffers is settled at dispatch, from the final body.
from headroom.proxy.route_advice import advice_from
_ccr_handler = getattr(self, "ccr_response_handler", None)
chat_stream_ccr_eligible = _is_openai_chat_stream_ccr_eligible(
body=body,
enabled=self.config.ccr_buffered_chat_streaming,
ccr_response_handler_enabled=bool(
_ccr_handler and getattr(_ccr_handler.config, "enabled", True)
),
ccr_inject_tool=self.config.ccr_inject_tool,
routed_to_backend=(
getattr(self, "anthropic_backend", None) is not None
or advice_from(request) is not None
),
upstream_base_url=upstream_base_url,
)
can_inject_ccr_tool = _should_inject_openai_chat_ccr_tool(
ccr_inject_tool=self.config.ccr_inject_tool,
stream=stream,
buffered_eligible=chat_stream_ccr_eligible,
)
if (
self.config.ccr_inject_tool or self.config.ccr_inject_system_instructions
) and not _bypass:
if self.config.ccr_inject_tool and stream:
if self.config.ccr_inject_tool and stream and not chat_stream_ccr_eligible:
logger.info(
f"[{request_id}] CCR: skipping retrieval-tool injection for "
"OpenAI chat streaming; this path cannot intercept tool calls"
@@ -4253,7 +4428,8 @@ class OpenAIHandlerMixin:
provider="openai",
inject_tool=False, # routed through sticky helper below
inject_system_instructions=(
self.config.ccr_inject_system_instructions and not stream
self.config.ccr_inject_system_instructions
and (not stream or chat_stream_ccr_eligible)
),
)
injector.scan_for_markers(optimized_messages)
@@ -4263,7 +4439,7 @@ class OpenAIHandlerMixin:
injector.verify_ownership()
if (
self.config.ccr_inject_system_instructions
and not stream
and (not stream or chat_stream_ccr_eligible)
and injector.has_compressed_content
):
optimized_messages = injector.inject_into_system_message(
@@ -5175,8 +5351,31 @@ class OpenAIHandlerMixin:
)
url = _append_request_query(url, request.url.query)
# The tool stays resident for the session so the tools array is
# byte-stable, but the model has nothing to call it with unless this body
# carries a marker. A turn with none keeps its real stream: buffering it
# would make time-to-first-byte the whole generation for nothing (#3071).
# Checking for the tool too keeps a bypassed turn, which got none, streaming.
buffered_chat_ccr = (
chat_stream_ccr_eligible
and has_ccr_retrieve_tool(body.get("tools"))
and _chat_body_has_ccr_marker(body)
)
buffered_chat_include_usage = False
buffered_chat_continuation_status: int | None = None
if buffered_chat_ccr:
buffered_chat_include_usage = _take_stream_options_for_buffered_chat(body)
body["stream"] = False
# A strict gateway rejects `stream: false` with `Accept: text/event-stream` (#3078).
_accept_key = next((k for k in headers if k.lower() == "accept"), "accept")
headers[_accept_key] = "application/json"
logger.info(
f"[{request_id}] CCR: buffering streaming chat turn "
"(stream:false upstream, resynthesized as chat.completion.chunk)"
)
try:
if stream:
if stream and not buffered_chat_ccr:
# Request usage stats for accurate token counting instead of
# byte-based estimation — without overriding an explicit client
# choice (see _apply_stream_usage_option).
@@ -5210,7 +5409,12 @@ class OpenAIHandlerMixin:
original_messages=original_client_messages,
outcome_provider=openai_chat_outcome_provider,
)
else:
async def _direct_chat_turn() -> Response:
# Usage, the prefix tracker, savings and CCR continuations all
# live in here, so a buffered turn is accounted for exactly like
# a non-streaming one instead of through a second copy.
nonlocal headers, buffered_chat_continuation_status
headers = await apply_copilot_api_auth(headers, url=url)
response = await self._retry_request("POST", url, headers, body)
@@ -5232,7 +5436,13 @@ class OpenAIHandlerMixin:
# hook actually calls the model again.
_hook_usage = TurnHookUsage()
if _registered_turn_hooks() and response.status_code == 200:
# A buffered turn only ran stream-safe on_request hooks (that is
# decided before dispatch), so it skips their response half too.
if (
_registered_turn_hooks()
and response.status_code == 200
and not buffered_chat_ccr
):
try:
_hook_resp_json = response.json()
except (ValueError, json.JSONDecodeError):
@@ -5512,6 +5722,7 @@ class OpenAIHandlerMixin:
# means on a path that changed nothing.
if _ccr_cont_error is not None or _ccr_final_json is _pre_ccr_json:
if _ccr_cont_error is not None:
buffered_chat_continuation_status = _ccr_cont_error
logger.warning(
f"[{request_id}] CCR: continuation returned HTTP "
f"{_ccr_cont_error}; forwarding the upstream reply "
@@ -5668,13 +5879,9 @@ class OpenAIHandlerMixin:
# the live (hooked) `messages` (#327).
#
# ``not stream`` mirrors the read gate at the cache lookup
# above. It is currently redundant here — a streaming chat
# request returns via ``_stream_response`` well before this
# point — but the Anthropic handler had the same shape until a
# buffered-CCR branch started falling through to its store
# site, which let a response built for a stream:true request
# answer a later non-streaming caller (#3019). Stating the
# invariant keeps that from being reintroduced silently.
# above. A buffered turn reaches this point with ``stream`` still
# true, and a response built for a stream:true request must not
# answer a later non-streaming caller (#3019).
if (
self.cache
and not stream
@@ -5795,6 +6002,37 @@ class OpenAIHandlerMixin:
status_code=response.status_code,
headers=response_headers,
)
if not buffered_chat_ccr:
return await _direct_chat_turn()
async def _buffered_chat_turn() -> Response:
return _buffered_chat_reply_as_sse(
await _direct_chat_turn(),
ccr_response_handler=_ccr_handler,
include_usage=buffered_chat_include_usage,
continuation_status=buffered_chat_continuation_status,
request_id=request_id,
)
async def _record_buffered_failure(**_: Any) -> None:
# The shared wrapper only knows "openai"; keep the label this
# handler uses for xAI and custom upstreams.
await self.metrics.record_failed(provider=openai_chat_outcome_provider)
_buffered_call = buffered_ccr_asgi_call(
operation=asyncio.create_task(_buffered_chat_turn()),
fmt=OPENAI_ERROR_FORMAT,
grace_seconds=self.config.buffered_ccr_grace_seconds,
record_failed=_record_buffered_failure,
request_id=request_id,
)
class _BufferedCCRResponse(Response):
async def __call__(self, scope, receive, send): # noqa: ANN001
await _buffered_call(scope, receive, send)
return _BufferedCCRResponse(media_type="text/event-stream")
except Exception as e:
await self.metrics.record_failed(provider=openai_chat_outcome_provider)
# Error names the failure; text and traceback go to debug.
+10
View File
@@ -422,6 +422,16 @@ class ProxyConfig:
# stream-idle watchdog fires. 0 or less disables the keepalive entirely.
# See headroom/proxy/buffered_ccr_response.py (#3079).
buffered_ccr_grace_seconds: float = DEFAULT_BUFFERED_CCR_GRACE_SECONDS
# Buffered CCR for /v1/chat/completions streaming. Without it this is the
# one streaming path with no way to answer a retrieval: content carries
# markers but `headroom_retrieve` is withheld, and the tool toggles in and
# out for sessions that mix streaming with non-streaming turns, busting the
# tools cache segment each way. Enabling it flips turns that carry a
# redeemable marker to `stream: false` upstream and resynthesizes
# chat.completion.chunk frames, as Responses streaming already does. Off by
# default: this is the most incident-prone path in the proxy (#2465, #2997,
# #3079), so the default flips separately, with measurements (#3815).
ccr_buffered_chat_streaming: bool = False
# Connection pool
max_connections: int = 500
+1
View File
@@ -6497,6 +6497,7 @@ def _proxy_config_from_env() -> ProxyConfig:
"HEADROOM_BUFFERED_CCR_GRACE_SECONDS",
DEFAULT_BUFFERED_CCR_GRACE_SECONDS,
),
ccr_buffered_chat_streaming=_get_env_bool("HEADROOM_CCR_BUFFERED_CHAT_STREAMING", False),
vertex_api_url=os.environ.get("VERTEX_TARGET_API_URL"),
backend=_get_env_str("HEADROOM_BACKEND", "anthropic"),
bedrock_region=_get_env_str("HEADROOM_BEDROCK_REGION", "us-west-2"),
+13
View File
@@ -165,6 +165,19 @@ SETTINGS: tuple[SettingField, ...] = (
),
tier="advanced",
),
SettingField(
"HEADROOM_CCR_BUFFERED_CHAT_STREAMING",
"ccr_buffered_chat_streaming",
"Buffered CCR for streaming chat",
"Compression",
"bool",
default=False,
help=(
"Answer retrieval calls on streaming Chat Completions by buffering "
"turns that carry CCR markers. Buffered text arrives all at once."
),
tier="advanced",
),
# --- Limits ---
SettingField(
"HEADROOM_RPM",
+21
View File
@@ -797,6 +797,27 @@ class TestCLIProxyEnvVars:
assert captured_config["config"].openai_api_url == "http://my-vllm:4000"
assert captured_config["config"].gemini_api_url == "http://my-gemini:5000"
@pytest.mark.parametrize(
("args", "env", "expected"),
[
([], {}, False),
(["--ccr-buffered-chat-streaming"], {}, True),
([], {"HEADROOM_CCR_BUFFERED_CHAT_STREAMING": "1"}, True),
],
)
def test_ccr_buffered_chat_streaming_reaches_config(self, runner, args, env, expected):
"""`headroom proxy` builds its own ProxyConfig, so the flag has to be wired here too."""
captured_config = {}
def mock_run_server(config, **kwargs):
captured_config["config"] = config
with patch("headroom.proxy.server.run_server", mock_run_server):
result = runner.invoke(main, ["proxy", *args], env=env, catch_exceptions=False)
assert result.exit_code == 0, result.output
assert captured_config["config"].ccr_buffered_chat_streaming is expected
@pytest.mark.parametrize("timeout", [-1, 0, 1, 10000])
def test_request_timeout_cli_flags(self, runner, timeout):
"""Fast-fail CLI flags should map into ProxyConfig."""
+718
View File
@@ -0,0 +1,718 @@
"""Buffered CCR on streaming /v1/chat/completions (#3815).
Compression still runs on a streaming chat turn, so content carries retrieval
markers, but this path had no way to answer ``headroom_retrieve``. With
``ccr_buffered_chat_streaming`` on, a turn whose outgoing body carries a
redeemable marker runs as ``stream: false`` upstream through the same code as a
non-streaming turn, then goes back to the client as ``chat.completion.chunk``
frames. Everything else keeps its real stream.
"""
from __future__ import annotations
import json
from typing import Any
import pytest
fastapi = pytest.importorskip("fastapi")
httpx = pytest.importorskip("httpx")
from fastapi.responses import Response # noqa: E402
from fastapi.testclient import TestClient # noqa: E402
from headroom.cache.compression_store import ( # noqa: E402
get_compression_store,
reset_compression_store,
)
from headroom.ccr.response_handler import CCRResponseHandler, StreamingCCRHandler # noqa: E402
from headroom.ccr.tool_injection import CCR_TOOL_NAME # noqa: E402
from headroom.proxy.handlers.openai import ( # noqa: E402
_buffered_chat_reply_as_sse,
_is_openai_chat_stream_ccr_eligible,
_take_stream_options_for_buffered_chat,
)
from headroom.proxy.helpers import _reset_session_ccr_tracker_for_test # noqa: E402
from headroom.proxy.server import ProxyConfig, _proxy_config_from_env, create_app # noqa: E402
ORIGINAL = "row 1: the original uncompressed rows\nrow 2: more of them"
CLIENT_TOOL = {"type": "function", "function": {"name": "Read", "description": "read a file"}}
USAGE_FIRST = {"prompt_tokens": 100, "completion_tokens": 20, "total_tokens": 120}
USAGE_FINAL = {"prompt_tokens": 400, "completion_tokens": 10, "total_tokens": 410}
@pytest.fixture(autouse=True)
def reset_store():
reset_compression_store()
_reset_session_ccr_tracker_for_test()
yield
reset_compression_store()
_reset_session_ccr_tracker_for_test()
def _stored_hash() -> str:
return get_compression_store().store(
original=ORIGINAL,
compressed="[2 items compressed to 0]",
original_item_count=2,
compressed_item_count=0,
)
def _completion(message: dict[str, Any], finish_reason: str, usage: dict[str, int]) -> dict:
return {
"id": "chatcmpl-1",
"object": "chat.completion",
"created": 1,
"model": "gpt-4o",
"choices": [{"index": 0, "message": message, "finish_reason": finish_reason}],
"usage": usage,
}
def _tool_call(name: str, arguments: dict[str, Any], call_id: str = "call_1") -> dict:
return {
"id": call_id,
"type": "function",
"function": {"name": name, "arguments": json.dumps(arguments)},
}
def _retrieve_reply(hash_key: str) -> dict:
message = {
"role": "assistant",
"content": None,
"tool_calls": [_tool_call(CCR_TOOL_NAME, {"hash": hash_key})],
}
return _completion(message, "tool_calls", USAGE_FIRST)
def _final_reply() -> dict:
return _completion(
{"role": "assistant", "content": "there are 2 rows"},
"stop",
USAGE_FINAL,
)
def _messages(hash_key: str | None) -> list[dict[str, Any]]:
content = "how many rows?"
if hash_key is not None:
content += f" [2 items compressed to 0. Retrieve more: hash={hash_key}]"
return [{"role": "user", "content": content}]
def _request(hash_key: str | None, **overrides: Any) -> dict[str, Any]:
return {
"model": "gpt-4o",
"messages": _messages(hash_key),
"stream": True,
"tools": [CLIENT_TOOL],
**overrides,
}
def _events(text: str) -> list[Any]:
"""Parse an SSE body into its JSON payloads, keeping the [DONE] sentinel."""
events: list[Any] = []
for line in text.splitlines():
if line.startswith("data: "):
data = line[len("data: ") :]
events.append(data if data == "[DONE]" else json.loads(data))
return events
def _chunks(text: str) -> list[dict[str, Any]]:
return [event for event in _events(text) if event != "[DONE]"]
class _Proxy:
"""A live app whose upstream calls are served from a script."""
def __init__(self, upstream: list[Any], **config: Any) -> None:
self._remaining = list(upstream)
self.sent: list[dict[str, Any]] = []
self.sent_headers: list[dict[str, str]] = []
self.outcomes: list[Any] = []
self.plain_streams = 0
settings = {
"optimize": False,
"cache_enabled": False,
"rate_limit_enabled": False,
"ccr_handle_responses": True,
"ccr_inject_tool": True,
"ccr_buffered_chat_streaming": True,
**config,
}
self._app = create_app(ProxyConfig(**settings))
async def _retry(self, method, url, headers, body, *args, **kwargs): # noqa: ANN001
self.sent.append(json.loads(json.dumps(body)))
self.sent_headers.append(dict(headers))
entry = self._remaining.pop(0)
if isinstance(entry, Exception):
raise entry
status, payload = entry if isinstance(entry, tuple) else (200, entry)
return httpx.Response(status, json=payload, headers={"content-type": "application/json"})
def __enter__(self) -> _Proxy:
self._client = TestClient(self._app)
self._client.__enter__()
proxy = self._client.app.state.proxy
proxy._retry_request = self._retry
self.proxy = proxy
async def _plain_stream(*args: Any, **kwargs: Any) -> Response:
self.plain_streams += 1
return Response(b"data: plain\n\n", media_type="text/event-stream")
proxy._stream_response = _plain_stream
real_record = proxy._record_request_outcome
async def _capture(outcome, *args, **kwargs): # noqa: ANN001
self.outcomes.append(outcome)
return await real_record(outcome, *args, **kwargs)
proxy._record_request_outcome = _capture
return self
def __exit__(self, *exc: Any) -> None:
self._client.__exit__(*exc)
def post(self, body: dict[str, Any], **extra_headers: str) -> httpx.Response:
return self._client.post(
"/v1/chat/completions",
json=body,
headers={
"Authorization": "Bearer test-key",
"x-api-key": "test-key",
"Accept": "text/event-stream",
**{name.replace("_", "-"): value for name, value in extra_headers.items()},
},
)
def test_flag_off_keeps_the_plain_stream_and_withholds_the_tool():
hash_key = _stored_hash()
with _Proxy([], ccr_buffered_chat_streaming=False) as proxy:
resp = proxy.post(_request(hash_key))
assert resp.text == "data: plain\n\n"
assert proxy.plain_streams == 1
assert proxy.sent == [], "flag off must not take the buffered path"
def test_flag_defaults_off():
assert ProxyConfig().ccr_buffered_chat_streaming is False
def test_env_var_turns_the_flag_on(monkeypatch):
monkeypatch.setenv("HEADROOM_CCR_BUFFERED_CHAT_STREAMING", "1")
assert _proxy_config_from_env().ccr_buffered_chat_streaming is True
def test_retrieval_is_answered_and_the_client_reads_a_real_stream():
hash_key = _stored_hash()
with _Proxy([_retrieve_reply(hash_key), _final_reply()]) as proxy:
resp = proxy.post(_request(hash_key))
assert resp.status_code == 200, resp.text
assert resp.headers["content-type"].startswith("text/event-stream")
assert "content-length" not in resp.headers or int(resp.headers["content-length"]) == len(
resp.content
)
events = _events(resp.text)
assert events[-1] == "[DONE]"
chunks = _chunks(resp.text)
assert {chunk["object"] for chunk in chunks} == {"chat.completion.chunk"}
choices = [chunk for chunk in chunks if chunk["choices"]]
assert choices[0]["choices"][0]["delta"]["role"] == "assistant"
text = "".join(chunk["choices"][0]["delta"].get("content") or "" for chunk in choices)
assert text == "there are 2 rows"
assert choices[-1]["choices"][0]["finish_reason"] == "stop"
assert CCR_TOOL_NAME not in resp.text
first, continuation = proxy.sent
assert first["stream"] is False
assert "stream_options" not in first, "OpenAI answers 400 to stream_options without a stream"
assert CCR_TOOL_NAME in {tool["function"]["name"] for tool in first["tools"]}
assert ORIGINAL.splitlines()[0] in json.dumps(continuation)
assert {k.lower(): v for k, v in proxy.sent_headers[0].items()}["accept"] == "application/json"
def test_a_turn_with_no_marker_keeps_its_real_stream():
"""Buffering a turn retrieval cannot help only costs time to first byte (#3071)."""
with _Proxy([]) as proxy:
resp = proxy.post(_request(None))
assert resp.text == "data: plain\n\n"
assert proxy.plain_streams == 1
assert proxy.sent == []
def test_a_foreign_marker_does_not_buffer():
"""Another context tool can use the same marker shape; only owned hashes count (#2836)."""
with _Proxy([]) as proxy:
resp = proxy.post(_request("0123456789abcdef0123456789abcdef"))
assert resp.text == "data: plain\n\n"
assert proxy.sent == []
def test_a_client_tool_call_still_ends_in_tool_calls():
hash_key = _stored_hash()
client_call = _tool_call("Read", {"file_path": "a.py"})
reply = _completion(
{"role": "assistant", "content": None, "tool_calls": [client_call]},
"tool_calls",
USAGE_FIRST,
)
with _Proxy([reply]) as proxy:
resp = proxy.post(_request(hash_key))
choices = [chunk["choices"][0] for chunk in _chunks(resp.text) if chunk["choices"]]
assert choices[-1]["finish_reason"] == "tool_calls"
tool_deltas = [
choice["delta"]["tool_calls"][0] for choice in choices if "tool_calls" in choice["delta"]
]
assert [delta["function"]["name"] for delta in tool_deltas] == ["Read"]
assert tool_deltas[0]["index"] == 0
@pytest.mark.parametrize(
("stream_options", "expect_usage"),
[
(None, True),
({"include_usage": True}, True),
({"include_usage": False}, False),
({}, True),
],
)
def test_usage_chunk_follows_the_plain_stream_rule(stream_options, expect_usage):
hash_key = _stored_hash()
overrides = {} if stream_options is None else {"stream_options": stream_options}
with _Proxy([_final_reply()]) as proxy:
resp = proxy.post(_request(hash_key, **overrides))
usage_chunks = [chunk for chunk in _chunks(resp.text) if chunk.get("usage")]
if expect_usage:
assert len(usage_chunks) == 1
assert usage_chunks[0]["choices"] == []
assert usage_chunks[0]["usage"] == USAGE_FINAL
assert _chunks(resp.text)[-1] is not None and _chunks(resp.text)[-1]["usage"]
else:
assert usage_chunks == []
assert "stream_options" not in proxy.sent[0]
def test_upstream_error_inside_the_grace_window_keeps_its_status():
"""A 429 must reach the client as a 429, not a 200 with no frames (#2997)."""
hash_key = _stored_hash()
error = {"error": {"message": "slow down", "type": "rate_limit_error"}}
with _Proxy([(429, error)]) as proxy:
resp = proxy.post(_request(hash_key))
assert resp.status_code == 429
assert resp.json()["error"]["message"] == "slow down"
class _Slow(_Proxy):
async def _retry(self, method, url, headers, body, *args, **kwargs): # noqa: ANN001
import asyncio
await asyncio.sleep(0.05)
return await super()._retry(method, url, headers, body, *args, **kwargs)
def test_a_slow_turn_gets_a_first_byte_before_the_answer():
"""Past the grace window the stream commits and a heartbeat runs (#2465, #3079)."""
hash_key = _stored_hash()
with _Slow([_final_reply()], buffered_ccr_grace_seconds=0.01) as proxy:
resp = proxy.post(_request(hash_key))
assert resp.status_code == 200
assert resp.text.startswith(": ping\n\n")
assert _events(resp.text)[-1] == "[DONE]"
def test_an_error_after_the_commit_is_a_typed_stream_error():
hash_key = _stored_hash()
error = {"error": {"message": "slow down", "type": "rate_limit_error"}}
with _Slow([(429, error)], buffered_ccr_grace_seconds=0.01) as proxy:
resp = proxy.post(_request(hash_key))
assert resp.status_code == 200, "the status was already committed"
events = _events(resp.text)
assert events[-1] == {"error": {"message": "slow down", "type": "rate_limit_error"}}
def test_a_transport_failure_inside_the_grace_window_is_a_clean_502():
hash_key = _stored_hash()
with _Proxy([httpx.ConnectError("boom")], ccr_buffered_chat_streaming=True) as proxy:
before = proxy.proxy.metrics.requests_failed
resp = proxy.post(_request(hash_key))
after = proxy.proxy.metrics.requests_failed
assert resp.status_code == 502
assert resp.json()["error"]["type"] == "server_error"
assert after == before + 1
def test_buffered_and_non_streaming_turns_account_identically():
"""Same turn both ways: the recorded numbers have to match (#3615, #3699)."""
fields = (
"status_code",
"provider_input_tokens",
"output_tokens",
"tokens_saved",
"cache_read_tokens",
"cache_write_tokens",
"uncached_input_tokens",
)
def run(stream: bool) -> dict[str, Any]:
reset_compression_store()
hash_key = _stored_hash()
with _Proxy([_retrieve_reply(hash_key), _final_reply()]) as proxy:
resp = proxy.post(_request(hash_key, stream=stream))
assert resp.status_code == 200
assert len(proxy.outcomes) == 1
return {field: getattr(proxy.outcomes[0], field) for field in fields}
buffered = run(stream=True)
assert buffered == run(stream=False)
assert buffered["provider_input_tokens"] == 500, "both upstream calls must be counted"
def test_tools_stay_byte_stable_across_a_session_that_mixes_modes():
"""The toggle from #3815: stream and non-stream turns must send the same tools."""
hash_key = _stored_hash()
upstream = [_final_reply(), _final_reply(), _final_reply()]
messages = _messages(hash_key)
with _Proxy(upstream) as proxy:
for stream in (False, True, False):
resp = proxy.post(_request(hash_key, messages=messages, stream=stream))
assert resp.status_code == 200, resp.text
assert len(proxy.sent) == 3
tools = [json.dumps(body["tools"]) for body in proxy.sent]
assert tools[0] == tools[1] == tools[2]
assert CCR_TOOL_NAME in tools[0]
@pytest.mark.parametrize(
("status", "error_type"),
[(429, "rate_limit_error"), (400, "invalid_request_error"), (500, "server_error")],
)
def test_failed_continuation_fails_closed_with_its_status(status, error_type):
"""The client must not get a tool it never declared, and backoff still has to fire."""
hash_key = _stored_hash()
upstream = [_retrieve_reply(hash_key), (status, {"error": {"message": "nope"}})]
with _Proxy(upstream) as proxy:
resp = proxy.post(_request(hash_key))
assert resp.status_code == status
assert CCR_TOOL_NAME not in resp.text
assert resp.json()["error"]["type"] == error_type
assert "nope" not in resp.text, "the upstream error body is untrusted and stays in the proxy"
def test_failed_continuation_after_the_commit_is_a_typed_stream_error():
hash_key = _stored_hash()
upstream = [_retrieve_reply(hash_key), (429, {"error": {"message": "nope"}})]
with _Slow(upstream, buffered_ccr_grace_seconds=0.01) as proxy:
resp = proxy.post(_request(hash_key))
assert resp.status_code == 200, "the status was already committed"
assert CCR_TOOL_NAME not in resp.text
assert _events(resp.text)[-1]["error"]["type"] == "rate_limit_error"
def test_an_expired_marker_is_still_answered_while_the_tool_is_resident():
"""The hash is gone but the model can still ask for it, and the client cannot answer."""
hash_key = _stored_hash()
messages = _messages(hash_key)
upstream = [_final_reply(), _retrieve_reply(hash_key), _final_reply()]
with _Proxy(upstream) as proxy:
first = proxy.post(_request(hash_key, messages=messages, stream=False))
assert first.status_code == 200, first.text
reset_compression_store()
resp = proxy.post(_request(hash_key, messages=messages))
assert resp.status_code == 200, resp.text
assert proxy.plain_streams == 0
assert CCR_TOOL_NAME not in resp.text
assert "there are 2 rows" in resp.text
def test_a_bypassed_turn_keeps_its_stream():
"""Bypass skips injection, so there is nothing to answer and no reason to buffer."""
hash_key = _stored_hash()
with _Proxy([]) as proxy:
resp = proxy.post(_request(hash_key), x_headroom_bypass="true")
assert resp.text == "data: plain\n\n"
assert proxy.sent == []
def test_response_hooks_do_not_run_on_a_buffered_turn(monkeypatch):
"""Their on_request half never ran; it is decided before dispatch, like a stream."""
import headroom.proxy.turn_hooks as turn_hooks
calls: list[str] = []
class _Hook:
name = "buffered-only"
def on_request(self, ctx): # noqa: ANN001
calls.append("on_request")
async def on_response(self, ctx, response, call_model): # noqa: ANN001
calls.append("on_response")
return None
hash_key = _stored_hash()
with _Proxy([_final_reply()]) as proxy:
monkeypatch.setattr(turn_hooks, "_hooks", [_Hook()], raising=False)
monkeypatch.setattr(turn_hooks, "registered_turn_hooks", lambda: [_Hook()])
resp = proxy.post(_request(hash_key))
assert resp.status_code == 200, resp.text
assert calls == []
def _mixed_reply(hash_key: str) -> dict:
return _completion(
{
"role": "assistant",
"content": None,
"tool_calls": [
_tool_call(CCR_TOOL_NAME, {"hash": hash_key}),
_tool_call("Read", {"file_path": "a.py"}, call_id="call_2"),
],
},
"tool_calls",
USAGE_FIRST,
)
def test_retrieval_next_to_a_client_tool_lets_the_model_decide_again():
"""The unrun client call is dropped, and the continuation replaces the turn."""
hash_key = _stored_hash()
with _Proxy([_mixed_reply(hash_key), _final_reply()]) as proxy:
resp = proxy.post(_request(hash_key))
assert resp.status_code == 200, resp.text
assert CCR_TOOL_NAME not in resp.text
assert "there are 2 rows" in resp.text
replayed = proxy.sent[1]["messages"][-2]
assert [call["function"]["name"] for call in replayed["tool_calls"]] == [CCR_TOOL_NAME]
def test_mixed_turn_goes_back_whole_when_its_continuation_fails():
"""The model made no newer decision, so the client gets the turn it did make."""
hash_key = _stored_hash()
upstream = [_mixed_reply(hash_key), (500, {"error": {"message": "nope"}})]
with _Proxy(upstream) as proxy:
resp = proxy.post(_request(hash_key))
assert resp.status_code == 200, resp.text
choices = [chunk["choices"][0] for chunk in _chunks(resp.text) if chunk["choices"]]
names = [
choice["delta"]["tool_calls"][0]["function"]["name"]
for choice in choices
if "tool_calls" in choice["delta"]
]
assert names == [CCR_TOOL_NAME, "Read"]
assert choices[-1]["finish_reason"] == "tool_calls"
def test_continuation_replays_what_the_provider_put_on_the_assistant_message():
"""DeepSeek rejects a tool-call turn replayed without its reasoning_content."""
hash_key = _stored_hash()
reply = _retrieve_reply(hash_key)
message = reply["choices"][0]["message"]
message["reasoning_content"] = "thinking it over"
message["reasoning_details"] = [{"text": "t"}]
message["refusal"] = None
message["annotations"] = []
with _Proxy([reply, _final_reply()]) as proxy:
resp = proxy.post(_request(hash_key))
assert resp.status_code == 200, resp.text
replayed = proxy.sent[1]["messages"][-2]
assert replayed["role"] == "assistant"
assert replayed["reasoning_content"] == "thinking it over"
assert replayed["reasoning_details"] == [{"text": "t"}]
assert "refusal" not in replayed, "empty fields stay out of the replayed message"
assert "annotations" not in replayed
class _RequiresReasoning(_Proxy):
"""An upstream that rejects a tool-call turn replayed without its reasoning."""
async def _retry(self, method, url, headers, body, *args, **kwargs): # noqa: ANN001
for message in body["messages"]:
if message.get("tool_calls") and "reasoning_content" not in message:
self.sent.append(json.loads(json.dumps(body)))
return httpx.Response(400, json={"error": {"message": "reasoning_content missing"}})
return await super()._retry(method, url, headers, body, *args, **kwargs)
def test_a_provider_that_requires_reasoning_on_replay_gets_it():
hash_key = _stored_hash()
reply = _retrieve_reply(hash_key)
reply["choices"][0]["message"]["reasoning_content"] = "required-replay"
with _RequiresReasoning([reply, _final_reply()]) as proxy:
resp = proxy.post(_request(hash_key))
assert resp.status_code == 200, resp.text
assert "there are 2 rows" in resp.text
@pytest.mark.parametrize(
"payload",
[
{"error": {"message": "gateway said no"}},
{"choices": []},
{"choices": None},
{"choices": "nope"},
{"choices": [None]},
{"choices": [{}]},
{"choices": [{"finish_reason": "stop"}]},
{"choices": [{"message": None, "finish_reason": "stop"}]},
],
)
def test_a_200_with_no_usable_choice_is_not_replayed_as_an_empty_stream(payload):
hash_key = _stored_hash()
with _Proxy([payload]) as proxy:
resp = proxy.post(_request(hash_key))
assert resp.status_code == 502
assert resp.json()["error"]["type"] == "upstream_protocol_error"
def _as_sse(payload: Any) -> Response:
return _buffered_chat_reply_as_sse(
Response(json.dumps(payload).encode(), status_code=200),
ccr_response_handler=CCRResponseHandler(),
include_usage=False,
continuation_status=None,
request_id="test",
)
@pytest.mark.parametrize(
"payload",
[
None,
[],
{"choices": [None]},
{"choices": [{}]},
{"choices": [{"message": "text"}]},
],
)
def test_the_helper_rejects_a_reply_with_no_usable_choice(payload):
result = _as_sse(payload)
assert result.status_code == 502
assert json.loads(result.body)["error"]["type"] == "upstream_protocol_error"
def test_a_refusal_is_a_valid_reply():
hash_key = _stored_hash()
reply = _completion(
{"role": "assistant", "content": None, "refusal": "no"}, "stop", USAGE_FIRST
)
with _Proxy([reply]) as proxy:
resp = proxy.post(_request(hash_key))
assert resp.status_code == 200, resp.text
deltas = [chunk["choices"][0]["delta"] for chunk in _chunks(resp.text) if chunk["choices"]]
assert {"refusal": "no"} in deltas
def test_a_buffered_reply_never_lands_in_the_response_cache():
"""It was built for a stream:true request; a later caller must not get it (#3019)."""
hash_key = _stored_hash()
with _Proxy([_final_reply()], cache_enabled=True) as proxy:
calls: list[Any] = []
async def _set(*args: Any, **kwargs: Any) -> None:
calls.append(args)
proxy.proxy.cache.set = _set
resp = proxy.post(_request(hash_key))
assert resp.status_code == 200
assert calls == []
@pytest.mark.parametrize(
("overrides", "expected"),
[
({}, True),
({"body": {"stream": False, "tools": [CLIENT_TOOL]}}, False),
({"body": {"stream": True, "tools": []}}, False),
({"body": {"stream": True}}, False),
({"body": {"stream": True, "tools": [CLIENT_TOOL], "n": 2}}, False),
({"body": {"stream": True, "tools": [CLIENT_TOOL], "n": 1}}, True),
({"body": {"stream": True, "tools": [CLIENT_TOOL], "logprobs": True}}, False),
({"enabled": False}, False),
({"ccr_response_handler_enabled": False}, False),
({"ccr_inject_tool": False}, False),
({"routed_to_backend": True}, False),
({"upstream_base_url": "https://opencode.ai/zen/v1"}, False),
],
)
def test_eligibility(overrides, expected):
args: dict[str, Any] = {
"body": {"stream": True, "tools": [CLIENT_TOOL]},
"enabled": True,
"ccr_response_handler_enabled": True,
"ccr_inject_tool": True,
"routed_to_backend": False,
"upstream_base_url": "https://api.openai.com",
**overrides,
}
assert _is_openai_chat_stream_ccr_eligible(**args) is expected
def test_taking_stream_options_leaves_other_fields_alone():
body = {"stream": True, "stream_options": {"include_usage": True}, "temperature": 0}
assert _take_stream_options_for_buffered_chat(body) is True
assert body == {"stream": True, "temperature": 0}
def test_frames_keep_the_fields_a_client_replays_on_its_next_turn():
call = {
**_tool_call("Read", {"file_path": "a.py"}),
"extra_content": {"google": {"thought_signature": "sig"}},
}
reply = _completion(
{
"role": "assistant",
"reasoning_content": "thinking it over",
"reasoning_details": [{"text": "t"}],
"refusal": None,
"content": "reading",
"tool_calls": [call],
},
"tool_calls",
USAGE_FIRST,
)
frames = StreamingCCRHandler.openai_chat_sse_frames(reply, include_usage=False)
deltas = [
chunk["choices"][0]["delta"]
for chunk in _chunks(b"".join(frames).decode())
if chunk["choices"]
]
assert {"reasoning_content": "thinking it over", "reasoning_details": [{"text": "t"}]} in deltas
tool_delta = next(delta["tool_calls"][0] for delta in deltas if "tool_calls" in delta)
assert tool_delta["extra_content"] == {"google": {"thought_signature": "sig"}}
assert tool_delta["index"] == 0
assert frames[-1] == b"data: [DONE]\n\n"