mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): record a stream the client closes after its terminal event as completed (#4286)
## Description Codex CLI (0.156.1) closes the HTTP connection as soon as it reads `response.completed`. It does not wait for the upstream EOF. In `StreamingMixin._stream_response`, `completed_normally` is set only after `async for chunk in response.aiter_bytes()` ends. The client's hang-up arrives as `asyncio.CancelledError` (the generator is waiting for the next upstream read) or `GeneratorExit` (the generator is suspended at a `yield`). Neither is an `Exception`, so both skip that assignment and reach `finally` with `completed_normally=False`. Since #3747 that records the turn as a 502 failure. `emit_request_outcome` short-circuits on `>= 400`, so the turn's usage, cache metrics and prefix-tracker update are dropped, even though the client got a complete 200 response. A second ordering problem hits the `GeneratorExit` case. The read loop yields each chunk to the client before `_parse_sse_usage_from_buffer` parses it. If the client closes at the yield of the terminal chunk, the terminal event and its usage are never parsed. Reproduced on wheel 0.41.0 and on current main. `codex exec` through a local proxy prints `ok`, while `/stats` shows `requests.total 0`, `failed_by_provider.openai 1`, `tokens.input 0`. A `HEADROOM_CODEX_WIRE_DEBUG=1` capture has every upstream chunk, including the complete `response.completed` event, and no `http_stream_upstream_complete` frame (written only after the loop ends normally). `_sse_event_outcome` classifies the captured events as terminal and not failed, so the classification is right. Only the timing of the hang-up loses the turn. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `_stream_response` (direct HTTP SSE): - Each chunk is parsed (usage, `stream_terminal`, `stream_failed`) before it is yielded. The memory filter and context guard still run on exactly the bytes the client receives, in the same order. Nothing awaits between the parse and the yield, so streaming latency is unchanged. - If that parse raises (a complete event with invalid UTF-8), the error is held until the chunk has been forwarded, then raised. The client gets the upstream bytes and then the error event, as on main. - A new `except (GeneratorExit, asyncio.CancelledError)` records success only when a terminal event was handed to the client, then re-raises. The outcome status uses `completed_normally or client_left_after_terminal`. - That is tracked by `_client_frame`, which wraps `_guard_client_bytes` at every client-facing yield: the main loop and the memory filter's closing and continuation frames. Unfiltered, each yield carries the whole parsed chunk, so it reuses the parser's `stream_terminal`. With the memory tool filter on, the parser can see `message_stop` before the client gets its first frame, and the filter holds `message_delta` / `message_stop` until after the stream, so each forwarded frame (a whole event) is classified as it goes out, until the terminal one. - `completed_normally` itself still means "the loop finished". It also gates draining mid-turn pending messages, which must not be yielded into a closed stream. - `_stream_response_bedrock`: same pattern. Each event is yielded right after its own usage and terminal bookkeeping, so the terminal flag always matches what the client got, and a hang-up after the terminal event sets `completed_normally`. That `finally` never yields. - `_stream_openai_via_backend`: it already parsed each chunk before yielding all of it, with no filter. It now gets the same hang-up handling. - A hang-up before the terminal event reaches the client is still a 502, as #3747 intended. - Tests: - `tests/test_stream_client_hangup_after_terminal.py` drives `_stream_response` with a fake upstream that sends `response.created`, then `response.completed` with usage, then never reaches EOF. The client either `aclose()`s at the terminal chunk or is cancelled while the proxy waits for EOF. Both must record status 200 with input 1,000 / cached 600 / output 5. A hang-up before the terminal event must still record 502. - Same file, memory tool filter on, one upstream chunk holding a whole Anthropic message (`message_start` through `message_stop`): `test_memory_filtered_hangup_before_the_client_gets_message_stop_is_a_failure[1-5]` reads 1 to 5 frames (frames 1-4 come from that one chunk, frame 5 is the held `message_delta`), closes, and must record 502. `test_memory_filtered_hangup_after_message_stop_is_forwarded_records_success` reads until `message_stop` is forwarded, closes, and must record 200 with its usage. - `test_malformed_event_after_a_valid_one_still_forwards_the_chunk`: a chunk with a valid event and then one with invalid UTF-8 reaches the client verbatim, followed by the error event, and records 502. - `tests/test_route_advice.py::test_backend_client_hangup_after_terminal_event_is_completed` covers both backend wrappers. - `pyproject.toml` / `uv.lock` (review: Dependency audit): the `aiohttp` constraint floor is now `>=3.14.4` (CVE-2026-111063, CVE-2026-111064 and GHSA-2g87-pp6c-29x5, all fixed in 3.14.4), relocked with `uv lock --upgrade-package aiohttp==3.14.4`. Only aiohttp (3.14.3 -> 3.14.4) and yarl (1.22.0 -> 1.25.1, which 3.14.4 requires) move. Main still resolves 3.14.3, so the same commit is on #4299, #4286, #4285 and #4269. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed mypy was run on the touched module only (output below), not on all of `headroom`. ### Test Output ```text $ uv run --frozen --extra dev pytest <the 43 test files that reference _stream_response, handlers.streaming, _stream_openai_via_backend, context_guard, StreamingMixin or sse_buffer> tests/test_proxy_openai_responses_integration.py -q -p no:randomly 1111 passed, 14 skipped, 1 warning in 58.03s # The new tests against 4eeb4619's streaming.py (this PR's previous head): FAILED test_stream_client_hangup_after_terminal.py::test_memory_filtered_hangup_before_the_client_gets_message_stop_is_a_failure[1] FAILED ...[2] FAILED ...[3] FAILED ...[4] FAILED ...[5] (each: assert 200 == 502) FAILED test_stream_client_hangup_after_terminal.py::test_malformed_event_after_a_valid_one_still_forwards_the_chunk 6 failed, 4 passed (the four passing: both Codex hang-up cases, the hang-up-before-terminal guard, the success after the filtered terminal) # The same two test files against main's streaming.py: FAILED test_stream_client_hangup_after_terminal.py::...[closed_at_terminal_yield] FAILED test_stream_client_hangup_after_terminal.py::...[cancelled_awaiting_eof] FAILED test_stream_client_hangup_after_terminal.py::test_memory_filtered_hangup_after_message_stop_is_forwarded_records_success FAILED test_route_advice.py::test_backend_client_hangup_after_terminal_event_is_completed[anthropic] FAILED test_route_advice.py::test_backend_client_hangup_after_terminal_event_is_completed[openai] 5 failed, 61 passed $ uvx ruff@0.17.0 check headroom/proxy/handlers/streaming.py tests/test_stream_client_hangup_after_terminal.py tests/test_route_advice.py All checks passed! $ uvx ruff@0.17.0 format --check (same files) 3 files already formatted $ uv run --frozen --extra dev mypy headroom/proxy/handlers/streaming.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS, Python 3.11, Codex CLI 0.156.1 signed in to ChatGPT. A local proxy on this branch's head (ef0773a7) started with `python -m headroom.cli proxy --port 6821 --no-http2 --no-rate-limit` and a clean HOME. The first revision of this PR also ran the Codex steps against main's `streaming.py`: main reported `requests.total 0`, `failed_by_provider.openai 1`, `tokens.input 0`. - Exact command / steps: (1) Codex: `codex exec --skip-git-repo-check -c 'model_providers.headroom.base_url="http://127.0.0.1:6821/v1"' "Reply with the single word: ok"` twice, then `GET /stats`. (2) Controlled upstream (a local stub, not Anthropic): a server on 127.0.0.1:6831 answers `POST /v1/messages` with one write holding a whole message (`message_start` through `message_stop`) and holds its EOF back 3 s. The proxy runs with `--memory --memory-storage global --no-memory-context --anthropic-api-url http://127.0.0.1:6831`, so the memory tool filter is on. A raw-socket client POSTs a streaming request with one client tool (memory tools are injected only beside client-declared tools) and closes right after reading `message_start`. A second request is read through `message_stop`. The early hang-up is repeated on a proxy running 4eeb4619's `streaming.py`. - Observed result: (1) Codex printed `ok` both times. `/stats`: `requests.total 2`, `failed 0`, `tokens.input 30198`, `tokens.output 10`, one PERF line per turn, no error or traceback in the log. (2) This branch: the early hang-up client got only `message_start`, and `/stats` went to `failed 1` (`failed_by_provider.anthropic 1`) with no PERF line for that turn. The full read added 1 to `requests.total`, with a PERF line. On4eeb4619the same early hang-up was recorded as a success (`requests.total 1`, `failed 0`, PERF line). Without the client tool (no memory tools, so no filter) the client got all six events from the one chunk and hung up before the stub's EOF; that turn was recorded as completed, `total_ms=3`. - Not tested: the Bedrock and OpenAI-via-backend wrappers against a live backend (unit tests only), a proxy-executed memory continuation round through a real proxy (unit tests only), and Codex over WebSocket, which uses a different handler. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: Stable. - Stable/default behavior changed: Yes, accounting only. A stream whose terminal event arrived without a failure before the client hung up is now recorded as completed: its usage, cache metrics and prefix-tracker update are kept instead of being counted as a 502. Bytes sent to the client are unchanged. - Kill switch / disable path: Not applicable; no new behavior to switch off. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable. ## Additional Notes - Malformed upstream content: a parse failure on a chunk (a complete event with invalid UTF-8) is raised after that chunk is forwarded, so the client gets the upstream bytes and then the error event, as on main. `_stream_openai_via_backend` already parsed before yielding on main; its order is unchanged here. - "Terminal" means the terminal event was handed to the client, i.e. the frame carrying it was yielded to the ASGI server. If a client disconnects in the same instant that frame goes out, the turn counts as completed. The upstream did bill it, so its usage belongs in the stats. - Cost: the unfiltered path (Codex, plain Anthropic) adds no parsing. With the memory filter on, each forwarded frame is classified once, and only until the terminal one. - Related: #3747, which introduced the failed-stream classification this builds on. - No documentation change needed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4286" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
f380692b46
commit
762a337bbe
@@ -20,6 +20,7 @@ from headroom.proxy.helpers import (
|
||||
RETRYABLE_OVERLOAD_STATUSES,
|
||||
jitter_delay_ms,
|
||||
overload_retry_is_futile,
|
||||
parse_sse_events_from_byte_buffer,
|
||||
retry_after_ms,
|
||||
)
|
||||
from headroom.proxy.provider_usage import (
|
||||
@@ -1652,6 +1653,13 @@ class StreamingMixin:
|
||||
full_sse_bytes = bytearray()
|
||||
parsed_response = None # Set by memory block; used by CCR + prefix tracker
|
||||
completed_normally = False
|
||||
# Set when the client hangs up after the terminal event; it marks
|
||||
# the outcome only, so pending messages are not drained into a
|
||||
# stream nobody reads.
|
||||
client_left_after_terminal = False
|
||||
# A terminal event the client was handed, not just one parsed from
|
||||
# upstream: the memory filter holds message_stop until the end.
|
||||
client_got_terminal = False
|
||||
pending_messages: list[dict] = []
|
||||
# Proxy-injected memory tools are invisible to the client, so their
|
||||
# tool_use blocks must never reach it (GH #2195).
|
||||
@@ -1661,6 +1669,23 @@ class StreamingMixin:
|
||||
else None
|
||||
)
|
||||
|
||||
def _client_frame(frame: bytes) -> bytes:
|
||||
"""``_guard_client_bytes``, noting when the terminal event goes out."""
|
||||
nonlocal client_got_terminal
|
||||
guarded = _guard_client_bytes(frame)
|
||||
if guarded and not client_got_terminal:
|
||||
if memory_filter is None:
|
||||
# Unfiltered, the client gets each whole chunk once it
|
||||
# is parsed, so every event the parser has seen.
|
||||
client_got_terminal = stream_state["stream_terminal"]
|
||||
else:
|
||||
# Filtered frames are complete events the filter decoded.
|
||||
client_got_terminal = any(
|
||||
_sse_event_outcome(name, data, stream_state["sse_dialect"])[1]
|
||||
for name, data in parse_sse_events_from_byte_buffer(bytearray(frame))
|
||||
)
|
||||
return guarded
|
||||
|
||||
try:
|
||||
async with contextlib.aclosing(upstream_response) as response:
|
||||
sse_chunk_index = 0
|
||||
@@ -1690,19 +1715,6 @@ class StreamingMixin:
|
||||
tail = bytes(stream_state["sse_buffer"][-MAX_SSE_BUFFER_SIZE // 2 :])
|
||||
stream_state["sse_buffer"] = bytearray(tail)
|
||||
|
||||
# Always stream immediately — buffering breaks
|
||||
# real-time clients (LangGraph, LangChain, etc.)
|
||||
# The context guard runs last, on exactly what the
|
||||
# client receives (after the memory filter), and may
|
||||
# hold back bytes only until the first complete event.
|
||||
client_frames = (
|
||||
[chunk] if memory_filter is None else memory_filter.feed(chunk)
|
||||
)
|
||||
for frame in client_frames:
|
||||
guarded_frame = _guard_client_bytes(frame)
|
||||
if guarded_frame:
|
||||
yield guarded_frame
|
||||
|
||||
if _codex_wire_debug:
|
||||
capture_codex_wire_debug(
|
||||
"http_stream_upstream_chunk",
|
||||
@@ -1735,8 +1747,14 @@ class StreamingMixin:
|
||||
)
|
||||
memory_enabled = False
|
||||
|
||||
# Parse complete SSE events from buffer
|
||||
usage = self._parse_sse_usage_from_buffer(stream_state, provider)
|
||||
# Parse complete SSE events from buffer. A failure (e.g.
|
||||
# invalid UTF-8) is raised once the chunk is forwarded:
|
||||
# malformed upstream content still reaches the client.
|
||||
parse_error: Exception | None = None
|
||||
try:
|
||||
usage = self._parse_sse_usage_from_buffer(stream_state, provider)
|
||||
except Exception as e:
|
||||
usage, parse_error = None, e
|
||||
if usage:
|
||||
if "input_tokens" in usage:
|
||||
stream_state["input_tokens"] = usage["input_tokens"]
|
||||
@@ -1759,6 +1777,24 @@ class StreamingMixin:
|
||||
"cache_creation_ephemeral_1h_input_tokens"
|
||||
]
|
||||
|
||||
# Always stream immediately — buffering breaks
|
||||
# real-time clients (LangGraph, LangChain, etc.)
|
||||
# The context guard runs last, on exactly what the
|
||||
# client receives (after the memory filter), and may
|
||||
# hold back bytes only until the first complete event.
|
||||
# Yielded only after the chunk is parsed: a client may
|
||||
# hang up on the terminal event (Codex does), closing
|
||||
# this generator at the yield.
|
||||
client_frames = (
|
||||
[chunk] if memory_filter is None else memory_filter.feed(chunk)
|
||||
)
|
||||
for frame in client_frames:
|
||||
guarded_frame = _client_frame(frame)
|
||||
if guarded_frame:
|
||||
yield guarded_frame
|
||||
if parse_error is not None:
|
||||
raise parse_error
|
||||
|
||||
# Memory tool handling after stream completes
|
||||
# Chunks were already yielded in real-time above, so we only
|
||||
# do silent background processing here — no yielding.
|
||||
@@ -1780,7 +1816,7 @@ class StreamingMixin:
|
||||
)
|
||||
if memory_filter is not None:
|
||||
for frame in memory_filter.closing_frames():
|
||||
guarded_frame = _guard_client_bytes(frame)
|
||||
guarded_frame = _client_frame(frame)
|
||||
if guarded_frame:
|
||||
yield guarded_frame
|
||||
guarded_tail = _drain_context_guard()
|
||||
@@ -1804,7 +1840,7 @@ class StreamingMixin:
|
||||
stream_state=stream_state,
|
||||
request_id=request_id,
|
||||
):
|
||||
guarded_frame = _guard_client_bytes(frame)
|
||||
guarded_frame = _client_frame(frame)
|
||||
if guarded_frame:
|
||||
yield guarded_frame
|
||||
memory_filter = None
|
||||
@@ -1849,7 +1885,7 @@ class StreamingMixin:
|
||||
stream_state=stream_state,
|
||||
request_id=request_id,
|
||||
):
|
||||
guarded_frame = _guard_client_bytes(frame)
|
||||
guarded_frame = _client_frame(frame)
|
||||
if guarded_frame:
|
||||
yield guarded_frame
|
||||
|
||||
@@ -1900,6 +1936,13 @@ class StreamingMixin:
|
||||
stream_state["stream_terminal"] and not stream_state["stream_failed"]
|
||||
)
|
||||
|
||||
except (GeneratorExit, asyncio.CancelledError):
|
||||
# Codex closes the connection as soon as it reads
|
||||
# response.completed, before the upstream EOF ends the loop.
|
||||
client_left_after_terminal = (
|
||||
client_got_terminal and not stream_state["stream_failed"]
|
||||
)
|
||||
raise
|
||||
except (httpx.ConnectError, httpx.ConnectTimeout, httpx.PoolTimeout) as e:
|
||||
logger.error(f"[{request_id}] Connection error to upstream API: {e}")
|
||||
guarded_tail = _drain_context_guard()
|
||||
@@ -1985,7 +2028,8 @@ class StreamingMixin:
|
||||
client=client,
|
||||
waste_signals=waste_signals,
|
||||
status_code=_stream_outcome_status(
|
||||
upstream_response.status_code, completed_normally
|
||||
upstream_response.status_code,
|
||||
completed_normally or client_left_after_terminal,
|
||||
),
|
||||
conversation_key=conversation_key,
|
||||
conversation_tokens_saved=conversation_tokens_saved,
|
||||
@@ -2143,7 +2187,6 @@ class StreamingMixin:
|
||||
chunk_bytes = sse_line.encode()
|
||||
if prefix_tracker is not None:
|
||||
full_sse_bytes.extend(chunk_bytes)
|
||||
yield chunk_bytes
|
||||
|
||||
# Track usage from message_start event
|
||||
if event.event_type == "message_start":
|
||||
@@ -2188,9 +2231,15 @@ class StreamingMixin:
|
||||
stream_terminal = stream_terminal or terminal
|
||||
if failed:
|
||||
logger.error(f"[{request_id}] Bedrock stream error: {event.data}")
|
||||
# After the bookkeeping: the client may hang up on the terminal event.
|
||||
yield chunk_bytes
|
||||
|
||||
completed_normally = stream_terminal and not stream_failed
|
||||
|
||||
except (GeneratorExit, asyncio.CancelledError):
|
||||
# A client that hung up after the terminal event got the whole stream.
|
||||
completed_normally = stream_terminal and not stream_failed
|
||||
raise
|
||||
except Exception as e:
|
||||
error_message = format_exception_message(e)
|
||||
logger.error(f"[{request_id}] Bedrock streaming error: {error_message}")
|
||||
@@ -2410,6 +2459,13 @@ class StreamingMixin:
|
||||
completed_normally = (
|
||||
stream_state.get("stream_terminal", False) and not stream_failed
|
||||
)
|
||||
except (GeneratorExit, asyncio.CancelledError):
|
||||
# A client that hung up after the terminal event got the whole stream.
|
||||
stream_failed = stream_state.get("stream_failed", False)
|
||||
completed_normally = (
|
||||
stream_state.get("stream_terminal", False) and not stream_failed
|
||||
)
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"[{request_id}] Backend streaming error: {e}")
|
||||
error_data = public_errors.openai_error_body(
|
||||
|
||||
@@ -405,6 +405,70 @@ def test_backend_clean_eof_requires_protocol_completion(provider, complete):
|
||||
assert outcomes[0].status_code == (200 if complete else 502)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("provider", ["anthropic", "openai"])
|
||||
def test_backend_client_hangup_after_terminal_event_is_completed(provider):
|
||||
"""A client may close on the terminal event, before the backend's EOF."""
|
||||
from types import MethodType
|
||||
|
||||
from headroom.proxy.handlers.streaming import StreamingMixin
|
||||
|
||||
class Backend(_Backend):
|
||||
async def stream_message(self, body, headers):
|
||||
yield SimpleNamespace(
|
||||
event_type="message_stop",
|
||||
raw_sse='event: message_stop\ndata: {"type":"message_stop"}\n\n',
|
||||
data={"type": "message_stop"},
|
||||
)
|
||||
await asyncio.Event().wait()
|
||||
|
||||
async def stream_openai_message(self, body, headers):
|
||||
yield "data: [DONE]\n\n"
|
||||
await asyncio.Event().wait()
|
||||
|
||||
outcomes = []
|
||||
|
||||
async def record(outcome):
|
||||
outcomes.append(outcome)
|
||||
|
||||
handler = _handler(Backend(provider))
|
||||
handler._record_request_outcome = record
|
||||
handler._extract_anthropic_cache_ttl_metrics = lambda usage: (0, 0)
|
||||
handler._parse_sse_usage_from_buffer = MethodType(
|
||||
StreamingMixin._parse_sse_usage_from_buffer, handler
|
||||
)
|
||||
|
||||
async def drive():
|
||||
arguments = {
|
||||
"body": {"messages": []},
|
||||
"headers": {},
|
||||
"model": "m",
|
||||
"request_id": "rid",
|
||||
"original_tokens": 0,
|
||||
"optimized_tokens": 0,
|
||||
"tokens_saved": 0,
|
||||
"transforms_applied": [],
|
||||
"tags": {},
|
||||
"optimization_latency": 0.0,
|
||||
"backend": handler.anthropic_backend,
|
||||
}
|
||||
if provider == "anthropic":
|
||||
response = await StreamingMixin._stream_response_bedrock(
|
||||
handler, provider=provider, **arguments
|
||||
)
|
||||
else:
|
||||
response = await StreamingMixin._stream_openai_via_backend(
|
||||
handler, start_time=0.0, **arguments
|
||||
)
|
||||
body = response.body_iterator
|
||||
while b"message_stop" not in (chunk := await body.__anext__()) and b"[DONE]" not in chunk:
|
||||
pass
|
||||
await body.aclose()
|
||||
|
||||
asyncio.run(drive())
|
||||
assert len(outcomes) == 1
|
||||
assert outcomes[0].status_code == 200
|
||||
|
||||
|
||||
@pytest.mark.parametrize("provider", ["anthropic", "openai"])
|
||||
@pytest.mark.parametrize("complete", [False, True])
|
||||
def test_backend_prefix_tracker_advances_only_after_completion(provider, complete):
|
||||
|
||||
@@ -0,0 +1,227 @@
|
||||
"""A client that hangs up after the terminal event still got a complete stream.
|
||||
|
||||
Codex CLI closes the connection as soon as it reads ``response.completed``,
|
||||
before the upstream EOF ends the proxy's read loop. The outcome was decided
|
||||
only after that loop, so every such turn recorded as a 502 failure with no
|
||||
usage and no cache metrics, although the client got a complete 200 response.
|
||||
Success needs the terminal event the client was handed, not one only parsed
|
||||
from upstream: the memory filter holds ``message_stop`` until after the stream.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from typing import Any
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from headroom.proxy.server import HeadroomProxy
|
||||
|
||||
|
||||
def _event(name: str, payload: dict[str, Any]) -> bytes:
|
||||
return f"event: {name}\ndata: {json.dumps(payload)}\n\n".encode()
|
||||
|
||||
|
||||
START = _event("response.created", {"type": "response.created", "response": {"id": "r1"}})
|
||||
COMPLETED = _event(
|
||||
"response.completed",
|
||||
{
|
||||
"type": "response.completed",
|
||||
"response": {
|
||||
"id": "r1",
|
||||
"status": "completed",
|
||||
"usage": {
|
||||
"input_tokens": 1_000,
|
||||
"input_tokens_details": {"cached_tokens": 600, "cache_write_tokens": 0},
|
||||
"output_tokens": 5,
|
||||
},
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
# A whole Anthropic message in one upstream chunk: the parser sees message_stop
|
||||
# before the client gets the first frame the memory filter splits it into.
|
||||
MESSAGE = b"".join(
|
||||
[
|
||||
_event(
|
||||
"message_start",
|
||||
{
|
||||
"type": "message_start",
|
||||
"message": {
|
||||
"id": "m1",
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"model": "claude-test",
|
||||
"content": [],
|
||||
"usage": {"input_tokens": 1_000, "output_tokens": 1},
|
||||
},
|
||||
},
|
||||
),
|
||||
_event(
|
||||
"content_block_start",
|
||||
{"type": "content_block_start", "index": 0, "content_block": {"type": "text"}},
|
||||
),
|
||||
_event(
|
||||
"content_block_delta",
|
||||
{
|
||||
"type": "content_block_delta",
|
||||
"index": 0,
|
||||
"delta": {"type": "text_delta", "text": "ok"},
|
||||
},
|
||||
),
|
||||
_event("content_block_stop", {"type": "content_block_stop", "index": 0}),
|
||||
_event(
|
||||
"message_delta",
|
||||
{
|
||||
"type": "message_delta",
|
||||
"delta": {"stop_reason": "end_turn"},
|
||||
"usage": {"output_tokens": 5},
|
||||
},
|
||||
),
|
||||
_event("message_stop", {"type": "message_stop"}),
|
||||
]
|
||||
)
|
||||
MEMORY_STREAM = {
|
||||
"url": "https://api.anthropic.com/v1/messages",
|
||||
"headers": {"x-api-key": "sk-test"},
|
||||
"body": {"model": "claude-test", "max_tokens": 100, "stream": True, "messages": []},
|
||||
"provider": "anthropic",
|
||||
"model": "claude-test",
|
||||
"memory_user_id": "user-1",
|
||||
"server_memory_tool_names": frozenset({"memory_save"}),
|
||||
}
|
||||
|
||||
|
||||
def _proxy(*chunks: bytes, upstream_eof: asyncio.Event | None = None) -> HeadroomProxy:
|
||||
proxy = object.__new__(HeadroomProxy)
|
||||
proxy.http_client = MagicMock(spec=httpx.AsyncClient)
|
||||
proxy._config = MagicMock()
|
||||
proxy._config.ccr_inject_tool = False
|
||||
proxy._config.retry_max_attempts = 1
|
||||
proxy.config = proxy._config
|
||||
proxy._record_request_outcome = AsyncMock()
|
||||
|
||||
response = MagicMock()
|
||||
response.headers = httpx.Headers({"content-type": "text/event-stream"})
|
||||
response.status_code = 200
|
||||
|
||||
async def aiter_bytes():
|
||||
for chunk in chunks or (START, COMPLETED):
|
||||
yield chunk
|
||||
if upstream_eof is not None:
|
||||
await upstream_eof.wait() # the upstream EOF the client never waits for
|
||||
|
||||
response.aiter_bytes = aiter_bytes
|
||||
response.aclose = AsyncMock()
|
||||
proxy.http_client.build_request = MagicMock(return_value=MagicMock())
|
||||
proxy.http_client.send = AsyncMock(return_value=response)
|
||||
return proxy
|
||||
|
||||
|
||||
async def _stream(proxy: HeadroomProxy, **overrides: Any):
|
||||
arguments: dict[str, Any] = {
|
||||
"url": "https://chatgpt.com/backend-api/codex/responses",
|
||||
"headers": {"authorization": "Bearer test"},
|
||||
"body": {"model": "gpt-5.4", "stream": True, "input": "Reply with: ok"},
|
||||
"provider": "openai",
|
||||
"model": "gpt-5.4",
|
||||
"request_id": "test-hangup",
|
||||
"original_tokens": 10,
|
||||
"optimized_tokens": 10,
|
||||
"tokens_saved": 0,
|
||||
"transforms_applied": [],
|
||||
"tags": {},
|
||||
"optimization_latency": 0.0,
|
||||
}
|
||||
if overrides.get("memory_user_id"):
|
||||
proxy.memory_handler = MagicMock()
|
||||
response = await proxy._stream_response(**{**arguments, **overrides})
|
||||
return response.body_iterator
|
||||
|
||||
|
||||
@pytest.mark.parametrize("hangup", ["closed_at_terminal_yield", "cancelled_awaiting_eof"])
|
||||
async def test_client_hangup_after_terminal_event_records_success_with_usage(hangup):
|
||||
proxy = _proxy(upstream_eof=asyncio.Event())
|
||||
body = await _stream(proxy)
|
||||
seen = b""
|
||||
if hangup == "closed_at_terminal_yield":
|
||||
# The generator is suspended at the yield of the terminal chunk.
|
||||
while b"response.completed" not in seen:
|
||||
seen += await body.__anext__()
|
||||
await body.aclose()
|
||||
else:
|
||||
# The generator read past the terminal chunk and waits for EOF.
|
||||
async def read_all() -> None:
|
||||
nonlocal seen
|
||||
async for chunk in body:
|
||||
seen += chunk
|
||||
|
||||
reader = asyncio.create_task(read_all())
|
||||
while b"response.completed" not in seen:
|
||||
await asyncio.sleep(0)
|
||||
reader.cancel()
|
||||
with pytest.raises(asyncio.CancelledError):
|
||||
await reader
|
||||
|
||||
proxy._record_request_outcome.assert_awaited_once()
|
||||
outcome = proxy._record_request_outcome.await_args.args[0]
|
||||
assert outcome.status_code == 200
|
||||
assert outcome.provider_input_tokens == 1_000
|
||||
assert outcome.cache_read_tokens == 600
|
||||
assert outcome.output_tokens == 5
|
||||
|
||||
|
||||
async def test_client_hangup_before_terminal_event_is_still_a_failure():
|
||||
proxy = _proxy(upstream_eof=asyncio.Event())
|
||||
body = await _stream(proxy)
|
||||
assert b"response.created" in await body.__anext__()
|
||||
await body.aclose()
|
||||
|
||||
outcome = proxy._record_request_outcome.await_args.args[0]
|
||||
assert outcome.status_code == 502
|
||||
|
||||
|
||||
@pytest.mark.parametrize("frames_read", [1, 2, 3, 4, 5])
|
||||
async def test_memory_filtered_hangup_before_the_client_gets_message_stop_is_a_failure(
|
||||
frames_read,
|
||||
):
|
||||
"""Frames 1-4 come from the one upstream chunk; frame 5 is the held message_delta."""
|
||||
proxy = _proxy(MESSAGE)
|
||||
body = await _stream(proxy, **MEMORY_STREAM)
|
||||
frames = [await body.__anext__() for _ in range(frames_read)]
|
||||
assert b"message_start" in frames[0]
|
||||
assert not any(b"message_stop" in frame for frame in frames)
|
||||
await body.aclose()
|
||||
|
||||
outcome = proxy._record_request_outcome.await_args.args[0]
|
||||
assert outcome.status_code == 502
|
||||
|
||||
|
||||
async def test_memory_filtered_hangup_after_message_stop_is_forwarded_records_success():
|
||||
proxy = _proxy(MESSAGE)
|
||||
body = await _stream(proxy, **MEMORY_STREAM)
|
||||
while b"message_stop" not in await body.__anext__():
|
||||
pass
|
||||
await body.aclose()
|
||||
|
||||
outcome = proxy._record_request_outcome.await_args.args[0]
|
||||
assert outcome.status_code == 200
|
||||
assert outcome.provider_input_tokens == 1_000
|
||||
assert outcome.output_tokens == 5
|
||||
|
||||
|
||||
async def test_malformed_event_after_a_valid_one_still_forwards_the_chunk():
|
||||
"""Invalid UTF-8 ends the stream with an error, after the upstream bytes."""
|
||||
valid = _event("response.output_text.delta", {"type": "response.output_text.delta"})
|
||||
malformed = b'event: response.output_text.delta\ndata: {"delta": "\xff"}\n\n'
|
||||
proxy = _proxy(START, valid + malformed)
|
||||
body = await _stream(proxy)
|
||||
frames = [frame async for frame in body]
|
||||
|
||||
assert frames[:2] == [START, valid + malformed]
|
||||
assert b"error" in frames[2]
|
||||
outcome = proxy._record_request_outcome.await_args.args[0]
|
||||
assert outcome.status_code == 502
|
||||
Reference in New Issue
Block a user