fix(proxy): record a stream the client closes after its terminal event as completed (#4286)

## Description

Codex CLI (0.156.1) closes the HTTP connection as soon as it reads
`response.completed`. It does not wait for the upstream EOF. In
`StreamingMixin._stream_response`, `completed_normally` is set only
after `async for chunk in response.aiter_bytes()` ends. The client's
hang-up arrives as `asyncio.CancelledError` (the generator is waiting
for the next upstream read) or `GeneratorExit` (the generator is
suspended at a `yield`). Neither is an `Exception`, so both skip that
assignment and reach `finally` with `completed_normally=False`. Since
#3747 that records the turn as a 502 failure. `emit_request_outcome`
short-circuits on `>= 400`, so the turn's usage, cache metrics and
prefix-tracker update are dropped, even though the client got a complete
200 response.

A second ordering problem hits the `GeneratorExit` case. The read loop
yields each chunk to the client before `_parse_sse_usage_from_buffer`
parses it. If the client closes at the yield of the terminal chunk, the
terminal event and its usage are never parsed.

Reproduced on wheel 0.41.0 and on current main. `codex exec` through a
local proxy prints `ok`, while `/stats` shows `requests.total 0`,
`failed_by_provider.openai 1`, `tokens.input 0`. A
`HEADROOM_CODEX_WIRE_DEBUG=1` capture has every upstream chunk,
including the complete `response.completed` event, and no
`http_stream_upstream_complete` frame (written only after the loop ends
normally). `_sse_event_outcome` classifies the captured events as
terminal and not failed, so the classification is right. Only the timing
of the hang-up loses the turn.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `_stream_response` (direct HTTP SSE):
- Each chunk is parsed (usage, `stream_terminal`, `stream_failed`)
before it is yielded. The memory filter and context guard still run on
exactly the bytes the client receives, in the same order. Nothing awaits
between the parse and the yield, so streaming latency is unchanged.
- If that parse raises (a complete event with invalid UTF-8), the error
is held until the chunk has been forwarded, then raised. The client gets
the upstream bytes and then the error event, as on main.
- A new `except (GeneratorExit, asyncio.CancelledError)` records success
only when a terminal event was handed to the client, then re-raises. The
outcome status uses `completed_normally or client_left_after_terminal`.
- That is tracked by `_client_frame`, which wraps `_guard_client_bytes`
at every client-facing yield: the main loop and the memory filter's
closing and continuation frames. Unfiltered, each yield carries the
whole parsed chunk, so it reuses the parser's `stream_terminal`. With
the memory tool filter on, the parser can see `message_stop` before the
client gets its first frame, and the filter holds `message_delta` /
`message_stop` until after the stream, so each forwarded frame (a whole
event) is classified as it goes out, until the terminal one.
- `completed_normally` itself still means "the loop finished". It also
gates draining mid-turn pending messages, which must not be yielded into
a closed stream.
- `_stream_response_bedrock`: same pattern. Each event is yielded right
after its own usage and terminal bookkeeping, so the terminal flag
always matches what the client got, and a hang-up after the terminal
event sets `completed_normally`. That `finally` never yields.
- `_stream_openai_via_backend`: it already parsed each chunk before
yielding all of it, with no filter. It now gets the same hang-up
handling.
- A hang-up before the terminal event reaches the client is still a 502,
as #3747 intended.
- Tests:
- `tests/test_stream_client_hangup_after_terminal.py` drives
`_stream_response` with a fake upstream that sends `response.created`,
then `response.completed` with usage, then never reaches EOF. The client
either `aclose()`s at the terminal chunk or is cancelled while the proxy
waits for EOF. Both must record status 200 with input 1,000 / cached 600
/ output 5. A hang-up before the terminal event must still record 502.
- Same file, memory tool filter on, one upstream chunk holding a whole
Anthropic message (`message_start` through `message_stop`):
`test_memory_filtered_hangup_before_the_client_gets_message_stop_is_a_failure[1-5]`
reads 1 to 5 frames (frames 1-4 come from that one chunk, frame 5 is the
held `message_delta`), closes, and must record 502.
`test_memory_filtered_hangup_after_message_stop_is_forwarded_records_success`
reads until `message_stop` is forwarded, closes, and must record 200
with its usage.
- `test_malformed_event_after_a_valid_one_still_forwards_the_chunk`: a
chunk with a valid event and then one with invalid UTF-8 reaches the
client verbatim, followed by the error event, and records 502.
-
`tests/test_route_advice.py::test_backend_client_hangup_after_terminal_event_is_completed`
covers both backend wrappers.
- `pyproject.toml` / `uv.lock` (review: Dependency audit): the `aiohttp`
constraint floor is now `>=3.14.4` (CVE-2026-111063, CVE-2026-111064 and
GHSA-2g87-pp6c-29x5, all fixed in 3.14.4), relocked with `uv lock
--upgrade-package aiohttp==3.14.4`. Only aiohttp (3.14.3 -> 3.14.4) and
yarl (1.22.0 -> 1.25.1, which 3.14.4 requires) move. Main still resolves
3.14.3, so the same commit is on #4299, #4286, #4285 and #4269.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

mypy was run on the touched module only (output below), not on all of
`headroom`.

### Test Output

```text
$ uv run --frozen --extra dev pytest <the 43 test files that reference _stream_response, handlers.streaming, _stream_openai_via_backend, context_guard, StreamingMixin or sse_buffer> tests/test_proxy_openai_responses_integration.py -q -p no:randomly
1111 passed, 14 skipped, 1 warning in 58.03s

# The new tests against 4eeb4619's streaming.py (this PR's previous head):
FAILED test_stream_client_hangup_after_terminal.py::test_memory_filtered_hangup_before_the_client_gets_message_stop_is_a_failure[1]
FAILED ...[2]  FAILED ...[3]  FAILED ...[4]  FAILED ...[5]      (each: assert 200 == 502)
FAILED test_stream_client_hangup_after_terminal.py::test_malformed_event_after_a_valid_one_still_forwards_the_chunk
6 failed, 4 passed   (the four passing: both Codex hang-up cases, the hang-up-before-terminal guard, the success after the filtered terminal)

# The same two test files against main's streaming.py:
FAILED test_stream_client_hangup_after_terminal.py::...[closed_at_terminal_yield]
FAILED test_stream_client_hangup_after_terminal.py::...[cancelled_awaiting_eof]
FAILED test_stream_client_hangup_after_terminal.py::test_memory_filtered_hangup_after_message_stop_is_forwarded_records_success
FAILED test_route_advice.py::test_backend_client_hangup_after_terminal_event_is_completed[anthropic]
FAILED test_route_advice.py::test_backend_client_hangup_after_terminal_event_is_completed[openai]
5 failed, 61 passed

$ uvx ruff@0.17.0 check headroom/proxy/handlers/streaming.py tests/test_stream_client_hangup_after_terminal.py tests/test_route_advice.py
All checks passed!
$ uvx ruff@0.17.0 format --check (same files)
3 files already formatted
$ uv run --frozen --extra dev mypy headroom/proxy/handlers/streaming.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS, Python 3.11, Codex CLI 0.156.1 signed in to
ChatGPT. A local proxy on this branch's head (ef0773a7) started with
`python -m headroom.cli proxy --port 6821 --no-http2 --no-rate-limit`
and a clean HOME. The first revision of this PR also ran the Codex steps
against main's `streaming.py`: main reported `requests.total 0`,
`failed_by_provider.openai 1`, `tokens.input 0`.
- Exact command / steps: (1) Codex: `codex exec --skip-git-repo-check -c
'model_providers.headroom.base_url="http://127.0.0.1:6821/v1"' "Reply
with the single word: ok"` twice, then `GET /stats`. (2) Controlled
upstream (a local stub, not Anthropic): a server on 127.0.0.1:6831
answers `POST /v1/messages` with one write holding a whole message
(`message_start` through `message_stop`) and holds its EOF back 3 s. The
proxy runs with `--memory --memory-storage global --no-memory-context
--anthropic-api-url http://127.0.0.1:6831`, so the memory tool filter is
on. A raw-socket client POSTs a streaming request with one client tool
(memory tools are injected only beside client-declared tools) and closes
right after reading `message_start`. A second request is read through
`message_stop`. The early hang-up is repeated on a proxy running
4eeb4619's `streaming.py`.
- Observed result: (1) Codex printed `ok` both times. `/stats`:
`requests.total 2`, `failed 0`, `tokens.input 30198`, `tokens.output
10`, one PERF line per turn, no error or traceback in the log. (2) This
branch: the early hang-up client got only `message_start`, and `/stats`
went to `failed 1` (`failed_by_provider.anthropic 1`) with no PERF line
for that turn. The full read added 1 to `requests.total`, with a PERF
line. On 4eeb4619 the same early hang-up was recorded as a success
(`requests.total 1`, `failed 0`, PERF line). Without the client tool (no
memory tools, so no filter) the client got all six events from the one
chunk and hung up before the stub's EOF; that turn was recorded as
completed, `total_ms=3`.
- Not tested: the Bedrock and OpenAI-via-backend wrappers against a live
backend (unit tests only), a proxy-executed memory continuation round
through a real proxy (unit tests only), and Codex over WebSocket, which
uses a different handler.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes, accounting only. A stream whose
terminal event arrived without a failure before the client hung up is
now recorded as completed: its usage, cache metrics and prefix-tracker
update are kept instead of being counted as a 502. Bytes sent to the
client are unchanged.
- Kill switch / disable path: Not applicable; no new behavior to switch
off.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

- Malformed upstream content: a parse failure on a chunk (a complete
event with invalid UTF-8) is raised after that chunk is forwarded, so
the client gets the upstream bytes and then the error event, as on main.
`_stream_openai_via_backend` already parsed before yielding on main; its
order is unchanged here.
- "Terminal" means the terminal event was handed to the client, i.e. the
frame carrying it was yielded to the ASGI server. If a client
disconnects in the same instant that frame goes out, the turn counts as
completed. The upstream did bill it, so its usage belongs in the stats.
- Cost: the unfiltered path (Codex, plain Anthropic) adds no parsing.
With the memory filter on, each forwarded frame is classified once, and
only until the terminal one.
- Related: #3747, which introduced the failed-stream classification this
builds on.
- No documentation change needed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4286"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
gglucass
2026-10-20 21:46:43 -05:00
committed by GitHub
co-authored by Claude Opus 5.5
parent f380692b46
commit 762a337bbe
3 changed files with 367 additions and 20 deletions
+76 -20
View File
@@ -20,6 +20,7 @@ from headroom.proxy.helpers import (
RETRYABLE_OVERLOAD_STATUSES,
jitter_delay_ms,
overload_retry_is_futile,
parse_sse_events_from_byte_buffer,
retry_after_ms,
)
from headroom.proxy.provider_usage import (
@@ -1652,6 +1653,13 @@ class StreamingMixin:
full_sse_bytes = bytearray()
parsed_response = None # Set by memory block; used by CCR + prefix tracker
completed_normally = False
# Set when the client hangs up after the terminal event; it marks
# the outcome only, so pending messages are not drained into a
# stream nobody reads.
client_left_after_terminal = False
# A terminal event the client was handed, not just one parsed from
# upstream: the memory filter holds message_stop until the end.
client_got_terminal = False
pending_messages: list[dict] = []
# Proxy-injected memory tools are invisible to the client, so their
# tool_use blocks must never reach it (GH #2195).
@@ -1661,6 +1669,23 @@ class StreamingMixin:
else None
)
def _client_frame(frame: bytes) -> bytes:
"""``_guard_client_bytes``, noting when the terminal event goes out."""
nonlocal client_got_terminal
guarded = _guard_client_bytes(frame)
if guarded and not client_got_terminal:
if memory_filter is None:
# Unfiltered, the client gets each whole chunk once it
# is parsed, so every event the parser has seen.
client_got_terminal = stream_state["stream_terminal"]
else:
# Filtered frames are complete events the filter decoded.
client_got_terminal = any(
_sse_event_outcome(name, data, stream_state["sse_dialect"])[1]
for name, data in parse_sse_events_from_byte_buffer(bytearray(frame))
)
return guarded
try:
async with contextlib.aclosing(upstream_response) as response:
sse_chunk_index = 0
@@ -1690,19 +1715,6 @@ class StreamingMixin:
tail = bytes(stream_state["sse_buffer"][-MAX_SSE_BUFFER_SIZE // 2 :])
stream_state["sse_buffer"] = bytearray(tail)
# Always stream immediately — buffering breaks
# real-time clients (LangGraph, LangChain, etc.)
# The context guard runs last, on exactly what the
# client receives (after the memory filter), and may
# hold back bytes only until the first complete event.
client_frames = (
[chunk] if memory_filter is None else memory_filter.feed(chunk)
)
for frame in client_frames:
guarded_frame = _guard_client_bytes(frame)
if guarded_frame:
yield guarded_frame
if _codex_wire_debug:
capture_codex_wire_debug(
"http_stream_upstream_chunk",
@@ -1735,8 +1747,14 @@ class StreamingMixin:
)
memory_enabled = False
# Parse complete SSE events from buffer
usage = self._parse_sse_usage_from_buffer(stream_state, provider)
# Parse complete SSE events from buffer. A failure (e.g.
# invalid UTF-8) is raised once the chunk is forwarded:
# malformed upstream content still reaches the client.
parse_error: Exception | None = None
try:
usage = self._parse_sse_usage_from_buffer(stream_state, provider)
except Exception as e:
usage, parse_error = None, e
if usage:
if "input_tokens" in usage:
stream_state["input_tokens"] = usage["input_tokens"]
@@ -1759,6 +1777,24 @@ class StreamingMixin:
"cache_creation_ephemeral_1h_input_tokens"
]
# Always stream immediately — buffering breaks
# real-time clients (LangGraph, LangChain, etc.)
# The context guard runs last, on exactly what the
# client receives (after the memory filter), and may
# hold back bytes only until the first complete event.
# Yielded only after the chunk is parsed: a client may
# hang up on the terminal event (Codex does), closing
# this generator at the yield.
client_frames = (
[chunk] if memory_filter is None else memory_filter.feed(chunk)
)
for frame in client_frames:
guarded_frame = _client_frame(frame)
if guarded_frame:
yield guarded_frame
if parse_error is not None:
raise parse_error
# Memory tool handling after stream completes
# Chunks were already yielded in real-time above, so we only
# do silent background processing here — no yielding.
@@ -1780,7 +1816,7 @@ class StreamingMixin:
)
if memory_filter is not None:
for frame in memory_filter.closing_frames():
guarded_frame = _guard_client_bytes(frame)
guarded_frame = _client_frame(frame)
if guarded_frame:
yield guarded_frame
guarded_tail = _drain_context_guard()
@@ -1804,7 +1840,7 @@ class StreamingMixin:
stream_state=stream_state,
request_id=request_id,
):
guarded_frame = _guard_client_bytes(frame)
guarded_frame = _client_frame(frame)
if guarded_frame:
yield guarded_frame
memory_filter = None
@@ -1849,7 +1885,7 @@ class StreamingMixin:
stream_state=stream_state,
request_id=request_id,
):
guarded_frame = _guard_client_bytes(frame)
guarded_frame = _client_frame(frame)
if guarded_frame:
yield guarded_frame
@@ -1900,6 +1936,13 @@ class StreamingMixin:
stream_state["stream_terminal"] and not stream_state["stream_failed"]
)
except (GeneratorExit, asyncio.CancelledError):
# Codex closes the connection as soon as it reads
# response.completed, before the upstream EOF ends the loop.
client_left_after_terminal = (
client_got_terminal and not stream_state["stream_failed"]
)
raise
except (httpx.ConnectError, httpx.ConnectTimeout, httpx.PoolTimeout) as e:
logger.error(f"[{request_id}] Connection error to upstream API: {e}")
guarded_tail = _drain_context_guard()
@@ -1985,7 +2028,8 @@ class StreamingMixin:
client=client,
waste_signals=waste_signals,
status_code=_stream_outcome_status(
upstream_response.status_code, completed_normally
upstream_response.status_code,
completed_normally or client_left_after_terminal,
),
conversation_key=conversation_key,
conversation_tokens_saved=conversation_tokens_saved,
@@ -2143,7 +2187,6 @@ class StreamingMixin:
chunk_bytes = sse_line.encode()
if prefix_tracker is not None:
full_sse_bytes.extend(chunk_bytes)
yield chunk_bytes
# Track usage from message_start event
if event.event_type == "message_start":
@@ -2188,9 +2231,15 @@ class StreamingMixin:
stream_terminal = stream_terminal or terminal
if failed:
logger.error(f"[{request_id}] Bedrock stream error: {event.data}")
# After the bookkeeping: the client may hang up on the terminal event.
yield chunk_bytes
completed_normally = stream_terminal and not stream_failed
except (GeneratorExit, asyncio.CancelledError):
# A client that hung up after the terminal event got the whole stream.
completed_normally = stream_terminal and not stream_failed
raise
except Exception as e:
error_message = format_exception_message(e)
logger.error(f"[{request_id}] Bedrock streaming error: {error_message}")
@@ -2410,6 +2459,13 @@ class StreamingMixin:
completed_normally = (
stream_state.get("stream_terminal", False) and not stream_failed
)
except (GeneratorExit, asyncio.CancelledError):
# A client that hung up after the terminal event got the whole stream.
stream_failed = stream_state.get("stream_failed", False)
completed_normally = (
stream_state.get("stream_terminal", False) and not stream_failed
)
raise
except Exception as e:
logger.error(f"[{request_id}] Backend streaming error: {e}")
error_data = public_errors.openai_error_body(
+64
View File
@@ -405,6 +405,70 @@ def test_backend_clean_eof_requires_protocol_completion(provider, complete):
assert outcomes[0].status_code == (200 if complete else 502)
@pytest.mark.parametrize("provider", ["anthropic", "openai"])
def test_backend_client_hangup_after_terminal_event_is_completed(provider):
"""A client may close on the terminal event, before the backend's EOF."""
from types import MethodType
from headroom.proxy.handlers.streaming import StreamingMixin
class Backend(_Backend):
async def stream_message(self, body, headers):
yield SimpleNamespace(
event_type="message_stop",
raw_sse='event: message_stop\ndata: {"type":"message_stop"}\n\n',
data={"type": "message_stop"},
)
await asyncio.Event().wait()
async def stream_openai_message(self, body, headers):
yield "data: [DONE]\n\n"
await asyncio.Event().wait()
outcomes = []
async def record(outcome):
outcomes.append(outcome)
handler = _handler(Backend(provider))
handler._record_request_outcome = record
handler._extract_anthropic_cache_ttl_metrics = lambda usage: (0, 0)
handler._parse_sse_usage_from_buffer = MethodType(
StreamingMixin._parse_sse_usage_from_buffer, handler
)
async def drive():
arguments = {
"body": {"messages": []},
"headers": {},
"model": "m",
"request_id": "rid",
"original_tokens": 0,
"optimized_tokens": 0,
"tokens_saved": 0,
"transforms_applied": [],
"tags": {},
"optimization_latency": 0.0,
"backend": handler.anthropic_backend,
}
if provider == "anthropic":
response = await StreamingMixin._stream_response_bedrock(
handler, provider=provider, **arguments
)
else:
response = await StreamingMixin._stream_openai_via_backend(
handler, start_time=0.0, **arguments
)
body = response.body_iterator
while b"message_stop" not in (chunk := await body.__anext__()) and b"[DONE]" not in chunk:
pass
await body.aclose()
asyncio.run(drive())
assert len(outcomes) == 1
assert outcomes[0].status_code == 200
@pytest.mark.parametrize("provider", ["anthropic", "openai"])
@pytest.mark.parametrize("complete", [False, True])
def test_backend_prefix_tracker_advances_only_after_completion(provider, complete):
@@ -0,0 +1,227 @@
"""A client that hangs up after the terminal event still got a complete stream.
Codex CLI closes the connection as soon as it reads ``response.completed``,
before the upstream EOF ends the proxy's read loop. The outcome was decided
only after that loop, so every such turn recorded as a 502 failure with no
usage and no cache metrics, although the client got a complete 200 response.
Success needs the terminal event the client was handed, not one only parsed
from upstream: the memory filter holds ``message_stop`` until after the stream.
"""
from __future__ import annotations
import asyncio
import json
from typing import Any
from unittest.mock import AsyncMock, MagicMock
import httpx
import pytest
from headroom.proxy.server import HeadroomProxy
def _event(name: str, payload: dict[str, Any]) -> bytes:
return f"event: {name}\ndata: {json.dumps(payload)}\n\n".encode()
START = _event("response.created", {"type": "response.created", "response": {"id": "r1"}})
COMPLETED = _event(
"response.completed",
{
"type": "response.completed",
"response": {
"id": "r1",
"status": "completed",
"usage": {
"input_tokens": 1_000,
"input_tokens_details": {"cached_tokens": 600, "cache_write_tokens": 0},
"output_tokens": 5,
},
},
},
)
# A whole Anthropic message in one upstream chunk: the parser sees message_stop
# before the client gets the first frame the memory filter splits it into.
MESSAGE = b"".join(
[
_event(
"message_start",
{
"type": "message_start",
"message": {
"id": "m1",
"type": "message",
"role": "assistant",
"model": "claude-test",
"content": [],
"usage": {"input_tokens": 1_000, "output_tokens": 1},
},
},
),
_event(
"content_block_start",
{"type": "content_block_start", "index": 0, "content_block": {"type": "text"}},
),
_event(
"content_block_delta",
{
"type": "content_block_delta",
"index": 0,
"delta": {"type": "text_delta", "text": "ok"},
},
),
_event("content_block_stop", {"type": "content_block_stop", "index": 0}),
_event(
"message_delta",
{
"type": "message_delta",
"delta": {"stop_reason": "end_turn"},
"usage": {"output_tokens": 5},
},
),
_event("message_stop", {"type": "message_stop"}),
]
)
MEMORY_STREAM = {
"url": "https://api.anthropic.com/v1/messages",
"headers": {"x-api-key": "sk-test"},
"body": {"model": "claude-test", "max_tokens": 100, "stream": True, "messages": []},
"provider": "anthropic",
"model": "claude-test",
"memory_user_id": "user-1",
"server_memory_tool_names": frozenset({"memory_save"}),
}
def _proxy(*chunks: bytes, upstream_eof: asyncio.Event | None = None) -> HeadroomProxy:
proxy = object.__new__(HeadroomProxy)
proxy.http_client = MagicMock(spec=httpx.AsyncClient)
proxy._config = MagicMock()
proxy._config.ccr_inject_tool = False
proxy._config.retry_max_attempts = 1
proxy.config = proxy._config
proxy._record_request_outcome = AsyncMock()
response = MagicMock()
response.headers = httpx.Headers({"content-type": "text/event-stream"})
response.status_code = 200
async def aiter_bytes():
for chunk in chunks or (START, COMPLETED):
yield chunk
if upstream_eof is not None:
await upstream_eof.wait() # the upstream EOF the client never waits for
response.aiter_bytes = aiter_bytes
response.aclose = AsyncMock()
proxy.http_client.build_request = MagicMock(return_value=MagicMock())
proxy.http_client.send = AsyncMock(return_value=response)
return proxy
async def _stream(proxy: HeadroomProxy, **overrides: Any):
arguments: dict[str, Any] = {
"url": "https://chatgpt.com/backend-api/codex/responses",
"headers": {"authorization": "Bearer test"},
"body": {"model": "gpt-5.4", "stream": True, "input": "Reply with: ok"},
"provider": "openai",
"model": "gpt-5.4",
"request_id": "test-hangup",
"original_tokens": 10,
"optimized_tokens": 10,
"tokens_saved": 0,
"transforms_applied": [],
"tags": {},
"optimization_latency": 0.0,
}
if overrides.get("memory_user_id"):
proxy.memory_handler = MagicMock()
response = await proxy._stream_response(**{**arguments, **overrides})
return response.body_iterator
@pytest.mark.parametrize("hangup", ["closed_at_terminal_yield", "cancelled_awaiting_eof"])
async def test_client_hangup_after_terminal_event_records_success_with_usage(hangup):
proxy = _proxy(upstream_eof=asyncio.Event())
body = await _stream(proxy)
seen = b""
if hangup == "closed_at_terminal_yield":
# The generator is suspended at the yield of the terminal chunk.
while b"response.completed" not in seen:
seen += await body.__anext__()
await body.aclose()
else:
# The generator read past the terminal chunk and waits for EOF.
async def read_all() -> None:
nonlocal seen
async for chunk in body:
seen += chunk
reader = asyncio.create_task(read_all())
while b"response.completed" not in seen:
await asyncio.sleep(0)
reader.cancel()
with pytest.raises(asyncio.CancelledError):
await reader
proxy._record_request_outcome.assert_awaited_once()
outcome = proxy._record_request_outcome.await_args.args[0]
assert outcome.status_code == 200
assert outcome.provider_input_tokens == 1_000
assert outcome.cache_read_tokens == 600
assert outcome.output_tokens == 5
async def test_client_hangup_before_terminal_event_is_still_a_failure():
proxy = _proxy(upstream_eof=asyncio.Event())
body = await _stream(proxy)
assert b"response.created" in await body.__anext__()
await body.aclose()
outcome = proxy._record_request_outcome.await_args.args[0]
assert outcome.status_code == 502
@pytest.mark.parametrize("frames_read", [1, 2, 3, 4, 5])
async def test_memory_filtered_hangup_before_the_client_gets_message_stop_is_a_failure(
frames_read,
):
"""Frames 1-4 come from the one upstream chunk; frame 5 is the held message_delta."""
proxy = _proxy(MESSAGE)
body = await _stream(proxy, **MEMORY_STREAM)
frames = [await body.__anext__() for _ in range(frames_read)]
assert b"message_start" in frames[0]
assert not any(b"message_stop" in frame for frame in frames)
await body.aclose()
outcome = proxy._record_request_outcome.await_args.args[0]
assert outcome.status_code == 502
async def test_memory_filtered_hangup_after_message_stop_is_forwarded_records_success():
proxy = _proxy(MESSAGE)
body = await _stream(proxy, **MEMORY_STREAM)
while b"message_stop" not in await body.__anext__():
pass
await body.aclose()
outcome = proxy._record_request_outcome.await_args.args[0]
assert outcome.status_code == 200
assert outcome.provider_input_tokens == 1_000
assert outcome.output_tokens == 5
async def test_malformed_event_after_a_valid_one_still_forwards_the_chunk():
"""Invalid UTF-8 ends the stream with an error, after the upstream bytes."""
valid = _event("response.output_text.delta", {"type": "response.output_text.delta"})
malformed = b'event: response.output_text.delta\ndata: {"delta": "\xff"}\n\n'
proxy = _proxy(START, valid + malformed)
body = await _stream(proxy)
frames = [frame async for frame in body]
assert frames[:2] == [START, valid + malformed]
assert b"error" in frames[2]
outcome = proxy._record_request_outcome.await_args.args[0]
assert outcome.status_code == 502