Files
headroom/tests/test_mid_turn_steering.py
lorenzozaneandJD Davis 42d4f58b43 fix(proxy): classify failed streams as failed requests (#3747)
## Description

Streams that fail after HTTP headers are sent were counted as completed,
hiding failures from the dashboard. This change records failed outcomes
for exceptions, protocol error events, and clean transport EOF without
the expected SSE terminal event. Client HTTP status remains 200 once
streaming has begun; the recorded outcome becomes 502.

Closes #3406

## Type of Change

- [x] Bug fix
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update

## Changes Made

- Classify actual SSE event fields and JSON envelopes instead of
searching generated text for `event: error`.
- Recognize canonical OpenAI Responses `response.failed` events.
- Require Anthropic `message_stop`, Chat Completions `[DONE]`, or
Responses `response.completed`/`response.incomplete` before counting
successful completion. Native Gemini and Cloud Code require meaningful
candidate `finishReason` values for every requested candidate;
prompt-blocking and wrapped error envelopes are failures.
`response.incomplete` is a normal terminal result such as reaching the
output-token limit; `response.failed` is a failure.
- Reuse already-parsed complete SSE events for outcome tracking,
avoiding the added full partial-buffer copy/scan on every chunk.
- Apply completion checks to direct HTTP, Bedrock/Anthropic-format
backends, and OpenAI-format backends. Failed streams do not advance
prefix-cache state.
- Preserve main's safe incomplete UTF-8 tail handling, allowing failure
outcomes to be recorded after cleanup.
- Pass through list/object/non-string `type` fields without treating
them as protocol markers or crashing the stream.

## Testing

Current head: `7a2ec1202ee9f0976dbdd4317bcfb77d26668cbb`.

- Four new pre-fix regressions fail for content containing error text,
canonical Responses failure, and missing backend completion markers;
after correction they pass.
- 221 focused streaming, parser, backend cache-metric, Gemini
prefix-replay, and resilience tests pass; three platform/dependency
skips and one existing Starlette warning. Forty-four
routing/classification controls also pass after final formatting.
Non-string type regressions reproduce the review finding before the
guard and pass afterward.
- Positive completion and failed-prefix-state controls cover both
backend dialects. Responses completed/incomplete/failed semantics and
one classification per complete chunked event are covered.
- Ruff lint/format, whitespace, and Linux-target mypy over 566 source
files pass.

## Real Behavior Proof

- Setup: Windows Python 3.12, real uvicorn proxy, and a separate real
loopback SSE HTTP upstream; optimization, semantic caching, subscription
polling, and cost tracking disabled. Only network destination is
redirected; no live provider traffic.
- Steps: run `pr3747-resume18-runtime.py`, sending 27 real TCP requests
across Chat Completions, Responses, Anthropic Messages, native Gemini,
and Cloud Code, then query the running proxy's `/stats`.
- Observed: 12 valid terminal streams record successful outcomes; 15
error/truncated streams record 502 outcomes despite wire HTTP 200.
`/stats` reports 12 completed and 15 failed requests. Coverage includes
normal completion, generated error text, a malformed non-string type
passed byte-for-byte, provider error events, Responses
failure/incomplete, clean EOF, and an incomplete UTF-8 tail.
- Before correction, the same runtime replay misclassifies six cases:
generated error text, Chat EOF/UTF-8 tail, Responses failure/EOF, and
Anthropic EOF. Evidence: `pr3747-resume16-runtime-red.log`; corrected
evidence: `pr3747-resume18-runtime.log`; tests:
`pr3747-resume18-full-tests.log`.
- Gemini regression: hosted CI exposed native/Cloud Code prefix replay
failures. The previous published head `7a2ec1202` misclassifies six
valid STOP/MAX_TOKENS/multi-candidate completions in this same TCP
replay; `8e69df7d6` correctly records them. Clean EOF, blocked prompts,
error envelopes, and incomplete multi-candidate streams remain failures.
Evidence: `pr3747-resume18-runtime-published-red.log`.
- Not tested: live provider billing or account access. Backend-specific
paths are covered by integration/regression tests; the TCP replay
exercises direct HTTP paths.

## Runtime Rollout Safety

- Rollout-managed features: none; classification uses existing streaming
and outcome paths.
- Default behavior changed: completed counters require protocol
completion, failure counters include post-start failures, and failed
streams skip prefix-cache updates.
- Client-visible HTTP status cannot be rewritten after headers are sent.
Existing stream forwarding and error-frame behavior remain in place.
- No new unsafe override. Rollback: revert the failure-classification
commits.

## Review Readiness

- [x] Self-review completed
- [x] Regression tests and local checks pass
- [x] Current runtime behavior supplied
- [x] Did not edit CHANGELOG.md

Hosted checks and review must qualify this newly published head before
approval. No PR merge performed.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-08 10:07:30 -05:00

317 lines
13 KiB
Python

import asyncio
from unittest.mock import AsyncMock, MagicMock
import httpx
import pytest
from headroom.proxy.server import HeadroomProxy
class TestMidTurnSteering:
def test_mid_turn_queue_exists_on_streaming_mixin(self):
"""StreamingMixin has _mid_turn_queues class attribute after the fix."""
from headroom.proxy.handlers.streaming import StreamingMixin
assert hasattr(StreamingMixin, "_mid_turn_queues")
assert hasattr(StreamingMixin, "_active_streams")
def test_mid_turn_message_queued_when_stream_active(self):
"""When a session has an active stream, mid-turn messages are queued."""
from headroom.proxy.handlers.streaming import StreamingMixin
mixin = StreamingMixin()
session_key = "test-session-123"
mixin._active_streams.add(session_key)
body = {"messages": [{"role": "user", "content": "follow-up"}]}
result = mixin._queue_mid_turn_message(session_key, body)
assert result["status"] == 202
assert result["event"] == "headroom_queued"
assert not mixin._mid_turn_queues[session_key].empty()
queued = mixin._mid_turn_queues[session_key].get_nowait()
assert queued == body
# Cleanup
mixin._active_streams.discard(session_key)
del mixin._mid_turn_queues[session_key]
def test_no_queue_when_no_prior_stream(self):
"""When no stream is active, _mid_turn_queues stays empty for the session."""
from headroom.proxy.handlers.streaming import StreamingMixin
mixin = StreamingMixin()
session_key = "inactive-session"
assert session_key not in mixin._active_streams
assert session_key not in mixin._mid_turn_queues
def test_should_queue_only_with_explicit_session_header(self):
"""Regression: mid-turn queuing must require an explicit session header.
Without ``x-headroom-session-id`` the session key is a coarse
``md5(model + system[:500])`` shared by concurrent independent streams
(e.g. a main conversation plus background/parallel requests). Queuing
those wrongly returns a 202 to a streaming caller, whose SDK stream
parser then fails on an empty (non-SSE) stream. Only opt-in callers
that send the header may be queued.
"""
from headroom.proxy.handlers.streaming import StreamingMixin
mixin = StreamingMixin()
session_key = "shared-md5-key"
# An earlier stream on this (fallback) session key is in flight.
mixin._active_streams.add(session_key)
try:
# No explicit header (header-less concurrent stream): must NOT queue,
# even though the key collides in _active_streams.
assert mixin._should_queue_mid_turn(session_key, None) is False
assert mixin._should_queue_mid_turn(session_key, "") is False
# Explicit header present: opt-in client, queuing is allowed.
assert mixin._should_queue_mid_turn(session_key, session_key) is True
# Explicit header but no active stream: nothing to queue behind.
assert mixin._should_queue_mid_turn("other-key", "other-key") is False
finally:
mixin._active_streams.discard(session_key)
def _create_mock_proxy(self):
proxy = object.__new__(HeadroomProxy)
proxy.http_client = MagicMock(spec=httpx.AsyncClient)
proxy._config = MagicMock()
proxy._config.memory_enabled = False
proxy._config.ccr_inject_tool = False
proxy._config.retry_max_attempts = 1
proxy._config.retry_base_delay_ms = 0
proxy._config.retry_max_delay_ms = 0
proxy.config = proxy._config
proxy.memory_handler = None
# Exercise real event parsing: successful completion requires the
# message_stop marker before pending messages can be drained.
proxy._finalize_stream_response = AsyncMock(return_value=None)
return proxy
@staticmethod
def _create_mock_upstream_response(
chunks: list[bytes], *, terminal_exception: BaseException | None = None
):
mock_response = AsyncMock()
mock_response.headers = httpx.Headers({"content-type": "text/event-stream"})
mock_response.status_code = 200
async def aiter_bytes():
for chunk in chunks:
yield chunk
if terminal_exception is not None:
raise terminal_exception
mock_response.aiter_bytes = aiter_bytes
mock_response.aclose = AsyncMock()
return mock_response
@pytest.mark.asyncio
async def test_mid_turn_stream_cancellation_clears_active_session_and_queue(self):
proxy = self._create_mock_proxy()
session_key = "cancelled-session"
mock_response = self._create_mock_upstream_response(
[
b'event: message_start\ndata: {"type":"message_start"}\n\n',
],
terminal_exception=asyncio.CancelledError(),
)
proxy.http_client.build_request = MagicMock(return_value=MagicMock())
proxy.http_client.send = AsyncMock(return_value=mock_response)
result = await proxy._stream_response(
url="https://api.anthropic.com/v1/messages",
headers={"x-api-key": "sk-test", "x-headroom-session-id": session_key},
body={
"model": "claude-sonnet-4-20250514",
"max_tokens": 100,
"stream": True,
"messages": [{"role": "user", "content": "hi"}],
},
provider="anthropic",
model="claude-sonnet-4-20250514",
request_id="test-cancelled",
original_tokens=10,
optimized_tokens=10,
tokens_saved=0,
transforms_applied=[],
tags={},
optimization_latency=0.0,
session_key=session_key,
)
proxy._queue_mid_turn_message(
session_key,
{"messages": [{"role": "user", "content": "follow-up"}]},
)
try:
with pytest.raises(asyncio.CancelledError):
async for _chunk in result.body_iterator:
pass
assert session_key not in proxy._active_streams
assert session_key not in proxy._mid_turn_queues
mock_response.aclose.assert_awaited_once()
finally:
proxy._active_streams.discard(session_key)
proxy._mid_turn_queues.pop(session_key, None)
@pytest.mark.asyncio
async def test_mid_turn_stream_exception_clears_active_session_and_queue(self):
proxy = self._create_mock_proxy()
session_key = "errored-session"
mock_response = self._create_mock_upstream_response(
[
b'event: message_start\ndata: {"type":"message_start"}\n\n',
],
terminal_exception=RuntimeError("stream exploded"),
)
proxy.http_client.build_request = MagicMock(return_value=MagicMock())
proxy.http_client.send = AsyncMock(return_value=mock_response)
result = await proxy._stream_response(
url="https://api.anthropic.com/v1/messages",
headers={"x-api-key": "sk-test", "x-headroom-session-id": session_key},
body={
"model": "claude-sonnet-4-20250514",
"max_tokens": 100,
"stream": True,
"messages": [{"role": "user", "content": "hi"}],
},
provider="anthropic",
model="claude-sonnet-4-20250514",
request_id="test-errored",
original_tokens=10,
optimized_tokens=10,
tokens_saved=0,
transforms_applied=[],
tags={},
optimization_latency=0.0,
session_key=session_key,
)
proxy._queue_mid_turn_message(
session_key,
{"messages": [{"role": "user", "content": "follow-up"}]},
)
try:
chunks = [chunk async for chunk in result.body_iterator]
assert any(b"event: error" in chunk for chunk in chunks)
assert session_key not in proxy._active_streams
assert session_key not in proxy._mid_turn_queues
mock_response.aclose.assert_awaited_once()
finally:
proxy._active_streams.discard(session_key)
proxy._mid_turn_queues.pop(session_key, None)
# --- #1608: mid-turn coalescing must be gated to Claude Code clients ---
def _normal_stream(self):
return self._create_mock_upstream_response(
[
b'event: message_start\ndata: {"type":"message_start"}\n\n',
b'event: message_stop\ndata: {"type":"message_stop"}\n\n',
]
)
async def _run_stream(self, proxy, session_key, user_agent):
mock_response = self._normal_stream()
proxy.http_client.build_request = MagicMock(return_value=MagicMock())
proxy.http_client.send = AsyncMock(return_value=mock_response)
return await proxy._stream_response(
url="https://api.anthropic.com/v1/messages",
headers={"x-api-key": "sk-test", "user-agent": user_agent},
body={
"model": "claude-sonnet-4-20250514",
"max_tokens": 100,
"stream": True,
"messages": [{"role": "user", "content": "hi"}],
},
provider="anthropic",
model="claude-sonnet-4-20250514",
request_id="test-1608",
original_tokens=10,
optimized_tokens=10,
tokens_saved=0,
transforms_applied=[],
tags={},
optimization_latency=0.0,
session_key=session_key,
)
@pytest.mark.asyncio
async def test_non_claude_client_not_registered_active(self):
# An OpenCode subagent shares the main agent's body-derived session
# key; if it registered as active, the concurrent request would be
# swallowed. Non-Claude-Code clients must never be registered.
proxy = self._create_mock_proxy()
session_key = "opencode-session"
result = await self._run_stream(proxy, session_key, "opencode/1.0")
try:
assert session_key not in proxy._active_streams
finally:
async for _chunk in result.body_iterator:
pass
proxy._active_streams.discard(session_key)
proxy._mid_turn_queues.pop(session_key, None)
@pytest.mark.asyncio
async def test_claude_code_client_registered_active(self):
proxy = self._create_mock_proxy()
session_key = "claude-session"
result = await self._run_stream(proxy, session_key, "claude-code/1.2.3")
try:
assert session_key in proxy._active_streams
finally:
async for _chunk in result.body_iterator:
pass
proxy._active_streams.discard(session_key)
proxy._mid_turn_queues.pop(session_key, None)
@pytest.mark.asyncio
async def test_pending_event_not_emitted_for_non_claude(self):
# Even with a queued message, a non-Claude-Code stream must not emit the
# custom `headroom_pending_messages` SSE event — @ai-sdk/anthropic can't
# parse it and throws "invalid_union / No matching discriminator".
proxy = self._create_mock_proxy()
session_key = "opencode-pending"
proxy._queue_mid_turn_message(
session_key, {"messages": [{"role": "user", "content": "queued"}]}
)
result = await self._run_stream(proxy, session_key, "opencode/1.0")
try:
body = b"".join([chunk async for chunk in result.body_iterator])
assert b"headroom_pending_messages" not in body
finally:
proxy._active_streams.discard(session_key)
proxy._mid_turn_queues.pop(session_key, None)
@pytest.mark.asyncio
async def test_pending_event_emitted_for_claude_code(self):
proxy = self._create_mock_proxy()
session_key = "claude-pending"
proxy._queue_mid_turn_message(
session_key, {"messages": [{"role": "user", "content": "queued"}]}
)
result = await self._run_stream(proxy, session_key, "claude-code/1.2.3")
try:
body = b"".join([chunk async for chunk in result.body_iterator])
assert b"headroom_pending_messages" in body
finally:
proxy._active_streams.discard(session_key)
proxy._mid_turn_queues.pop(session_key, None)
class TestCoalescingCapability:
"""The capability predicate that gates the mid-turn coalescing protocol."""
def test_claude_code_supports_coalescing(self):
from headroom.proxy.auth_mode import supports_mid_turn_coalescing
assert supports_mid_turn_coalescing("claude-code") is True
def test_other_clients_do_not_support_coalescing(self):
from headroom.proxy.auth_mode import supports_mid_turn_coalescing
for client in ("opencode", "codex", "cursor", "aider", "", None):
assert supports_mid_turn_coalescing(client) is False