mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): classify failed streams as failed requests (#3747)
## Description Streams that fail after HTTP headers are sent were counted as completed, hiding failures from the dashboard. This change records failed outcomes for exceptions, protocol error events, and clean transport EOF without the expected SSE terminal event. Client HTTP status remains 200 once streaming has begun; the recorded outcome becomes 502. Closes #3406 ## Type of Change - [x] Bug fix - [ ] New feature - [ ] Breaking change - [ ] Documentation update ## Changes Made - Classify actual SSE event fields and JSON envelopes instead of searching generated text for `event: error`. - Recognize canonical OpenAI Responses `response.failed` events. - Require Anthropic `message_stop`, Chat Completions `[DONE]`, or Responses `response.completed`/`response.incomplete` before counting successful completion. Native Gemini and Cloud Code require meaningful candidate `finishReason` values for every requested candidate; prompt-blocking and wrapped error envelopes are failures. `response.incomplete` is a normal terminal result such as reaching the output-token limit; `response.failed` is a failure. - Reuse already-parsed complete SSE events for outcome tracking, avoiding the added full partial-buffer copy/scan on every chunk. - Apply completion checks to direct HTTP, Bedrock/Anthropic-format backends, and OpenAI-format backends. Failed streams do not advance prefix-cache state. - Preserve main's safe incomplete UTF-8 tail handling, allowing failure outcomes to be recorded after cleanup. - Pass through list/object/non-string `type` fields without treating them as protocol markers or crashing the stream. ## Testing Current head: `7a2ec1202ee9f0976dbdd4317bcfb77d26668cbb`. - Four new pre-fix regressions fail for content containing error text, canonical Responses failure, and missing backend completion markers; after correction they pass. - 221 focused streaming, parser, backend cache-metric, Gemini prefix-replay, and resilience tests pass; three platform/dependency skips and one existing Starlette warning. Forty-four routing/classification controls also pass after final formatting. Non-string type regressions reproduce the review finding before the guard and pass afterward. - Positive completion and failed-prefix-state controls cover both backend dialects. Responses completed/incomplete/failed semantics and one classification per complete chunked event are covered. - Ruff lint/format, whitespace, and Linux-target mypy over 566 source files pass. ## Real Behavior Proof - Setup: Windows Python 3.12, real uvicorn proxy, and a separate real loopback SSE HTTP upstream; optimization, semantic caching, subscription polling, and cost tracking disabled. Only network destination is redirected; no live provider traffic. - Steps: run `pr3747-resume18-runtime.py`, sending 27 real TCP requests across Chat Completions, Responses, Anthropic Messages, native Gemini, and Cloud Code, then query the running proxy's `/stats`. - Observed: 12 valid terminal streams record successful outcomes; 15 error/truncated streams record 502 outcomes despite wire HTTP 200. `/stats` reports 12 completed and 15 failed requests. Coverage includes normal completion, generated error text, a malformed non-string type passed byte-for-byte, provider error events, Responses failure/incomplete, clean EOF, and an incomplete UTF-8 tail. - Before correction, the same runtime replay misclassifies six cases: generated error text, Chat EOF/UTF-8 tail, Responses failure/EOF, and Anthropic EOF. Evidence: `pr3747-resume16-runtime-red.log`; corrected evidence: `pr3747-resume18-runtime.log`; tests: `pr3747-resume18-full-tests.log`. - Gemini regression: hosted CI exposed native/Cloud Code prefix replay failures. The previous published head `7a2ec1202` misclassifies six valid STOP/MAX_TOKENS/multi-candidate completions in this same TCP replay; `8e69df7d6` correctly records them. Clean EOF, blocked prompts, error envelopes, and incomplete multi-candidate streams remain failures. Evidence: `pr3747-resume18-runtime-published-red.log`. - Not tested: live provider billing or account access. Backend-specific paths are covered by integration/regression tests; the TCP replay exercises direct HTTP paths. ## Runtime Rollout Safety - Rollout-managed features: none; classification uses existing streaming and outcome paths. - Default behavior changed: completed counters require protocol completion, failure counters include post-start failures, and failed streams skip prefix-cache updates. - Client-visible HTTP status cannot be rewritten after headers are sent. Existing stream forwarding and error-frame behavior remain in place. - No new unsafe override. Rollback: revert the failure-classification commits. ## Review Readiness - [x] Self-review completed - [x] Regression tests and local checks pass - [x] Current runtime behavior supplied - [x] Did not edit CHANGELOG.md Hosted checks and review must qualify this newly published head before approval. No PR merge performed. --------- Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
@@ -12,6 +12,7 @@ import logging
|
||||
import time
|
||||
from collections.abc import AsyncIterator
|
||||
from typing import TYPE_CHECKING, Any
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from headroom.proxy.auth_mode import classify_client, supports_mid_turn_coalescing
|
||||
from headroom.proxy.handlers._debug_dump import write_upstream_error_dump
|
||||
@@ -40,6 +41,104 @@ from headroom.proxy.thinking_tokens import ThinkingTokens, extract_thinking_toke
|
||||
from headroom.utils import format_exception_message
|
||||
|
||||
logger = logging.getLogger("headroom.proxy")
|
||||
STREAM_FAILURE_STATUS = 502
|
||||
|
||||
|
||||
def _stream_outcome_status(status_code: int, completed_normally: bool) -> int:
|
||||
"""Keep post-start stream failures out of the success counter."""
|
||||
return status_code if completed_normally else STREAM_FAILURE_STATUS
|
||||
|
||||
|
||||
def _gemini_response_data(data: Any) -> dict[str, Any]:
|
||||
if not isinstance(data, dict):
|
||||
return {}
|
||||
response = data.get("response")
|
||||
return response if isinstance(response, dict) else data
|
||||
|
||||
|
||||
def _gemini_finished_candidates(data: Any) -> set[int]:
|
||||
candidates = _gemini_response_data(data).get("candidates")
|
||||
finished: set[int] = set()
|
||||
if not isinstance(candidates, list):
|
||||
return finished
|
||||
for position, candidate in enumerate(candidates):
|
||||
if not isinstance(candidate, dict):
|
||||
continue
|
||||
reason = candidate.get("finishReason")
|
||||
index = candidate.get("index", position)
|
||||
if (
|
||||
isinstance(reason, str)
|
||||
and reason.strip()
|
||||
and reason != "FINISH_REASON_UNSPECIFIED"
|
||||
and isinstance(index, int)
|
||||
and not isinstance(index, bool)
|
||||
and index >= 0
|
||||
):
|
||||
finished.add(index)
|
||||
return finished
|
||||
|
||||
|
||||
def _gemini_requested_candidates(body: dict[str, Any]) -> int:
|
||||
request = body.get("request")
|
||||
request = request if isinstance(request, dict) else body
|
||||
config = request.get("generationConfig") or request.get("generation_config")
|
||||
if not isinstance(config, dict):
|
||||
return 1
|
||||
count = config.get("candidateCount", config.get("candidate_count", 1))
|
||||
return count if isinstance(count, int) and not isinstance(count, bool) and count > 0 else 1
|
||||
|
||||
|
||||
def _sse_event_outcome(event_name: str | None, data_str: str, dialect: str) -> tuple[bool, bool]:
|
||||
"""Classify a complete protocol event, never text inside a model delta."""
|
||||
try:
|
||||
data = json.loads(data_str)
|
||||
except (TypeError, ValueError):
|
||||
data = None
|
||||
event_type = data.get("type") if isinstance(data, dict) else None
|
||||
if not isinstance(event_type, str):
|
||||
event_type = None
|
||||
failed = event_name in {"error", "response.failed"} or event_type in {
|
||||
"error",
|
||||
"response.failed",
|
||||
}
|
||||
if isinstance(data, dict) and data.get("error") is not None:
|
||||
failed = True
|
||||
if dialect == "gemini":
|
||||
response = _gemini_response_data(data)
|
||||
feedback = response.get("promptFeedback")
|
||||
block_reason = feedback.get("blockReason") if isinstance(feedback, dict) else None
|
||||
failed = (
|
||||
failed
|
||||
or response.get("error") is not None
|
||||
or (
|
||||
isinstance(block_reason, str)
|
||||
and bool(block_reason.strip())
|
||||
and block_reason != "BLOCK_REASON_UNSPECIFIED"
|
||||
)
|
||||
)
|
||||
terminal = bool(_gemini_finished_candidates(data))
|
||||
elif dialect == "anthropic":
|
||||
terminal = event_name == "message_stop" or event_type == "message_stop"
|
||||
elif dialect == "responses":
|
||||
# Incomplete is a normal terminal result (e.g. the output-token limit),
|
||||
# unlike response.failed. It still incurred a successful model request.
|
||||
terminal = event_name in {"response.completed", "response.incomplete"} or event_type in {
|
||||
"response.completed",
|
||||
"response.incomplete",
|
||||
}
|
||||
else:
|
||||
terminal = data_str.strip() == "[DONE]"
|
||||
return failed, terminal
|
||||
|
||||
|
||||
def _sse_contains_error_event(payload: bytes) -> bool:
|
||||
from headroom.proxy.helpers import parse_sse_events_from_byte_buffer
|
||||
|
||||
return any(
|
||||
_sse_event_outcome(event_name, data, "chat")[0]
|
||||
for event_name, data in parse_sse_events_from_byte_buffer(bytearray(payload))
|
||||
)
|
||||
|
||||
|
||||
_ROUND_USAGE_KEYS = (
|
||||
"input_tokens",
|
||||
@@ -335,6 +434,16 @@ class StreamingMixin:
|
||||
# ``stream_state``, no reassignment is needed.
|
||||
events = parse_sse_events_from_byte_buffer(buffer)
|
||||
for _event_name, data_str in events:
|
||||
failed, terminal = _sse_event_outcome(
|
||||
_event_name,
|
||||
data_str,
|
||||
stream_state.get(
|
||||
"sse_dialect",
|
||||
provider if provider in {"anthropic", "gemini"} else "chat",
|
||||
),
|
||||
)
|
||||
stream_state["stream_failed"] = stream_state.get("stream_failed", False) or failed
|
||||
stream_state["stream_terminal"] = stream_state.get("stream_terminal", False) or terminal
|
||||
if not data_str or data_str == "[DONE]":
|
||||
continue
|
||||
|
||||
@@ -343,6 +452,12 @@ class StreamingMixin:
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
|
||||
if provider == "gemini":
|
||||
finished = stream_state.setdefault("gemini_finished_candidates", set())
|
||||
finished.update(_gemini_finished_candidates(data))
|
||||
expected = stream_state.get("gemini_candidate_count", 1)
|
||||
stream_state["stream_terminal"] = set(range(expected)).issubset(finished)
|
||||
|
||||
if provider == "anthropic":
|
||||
event_type = data.get("type", "")
|
||||
if event_type == "message_start":
|
||||
@@ -1266,6 +1381,18 @@ class StreamingMixin:
|
||||
# not corrupt downstream parsing.
|
||||
"sse_buffer": bytearray(),
|
||||
"ttfb_ms": None, # Time to first byte from upstream
|
||||
"stream_failed": False,
|
||||
"stream_terminal": False,
|
||||
"gemini_candidate_count": _gemini_requested_candidates(body),
|
||||
"sse_dialect": (
|
||||
"anthropic"
|
||||
if provider == "anthropic"
|
||||
else "gemini"
|
||||
if provider == "gemini"
|
||||
else "responses"
|
||||
if urlsplit(url).path.rstrip("/").endswith("/responses")
|
||||
else "chat"
|
||||
),
|
||||
}
|
||||
|
||||
# Track if we need to handle memory tools
|
||||
@@ -1829,7 +1956,9 @@ class StreamingMixin:
|
||||
status_code=upstream_response.status_code,
|
||||
metadata={"total_bytes": stream_state["total_bytes"]},
|
||||
)
|
||||
completed_normally = True
|
||||
completed_normally = (
|
||||
stream_state["stream_terminal"] and not stream_state["stream_failed"]
|
||||
)
|
||||
|
||||
except (httpx.ConnectError, httpx.ConnectTimeout, httpx.PoolTimeout) as e:
|
||||
logger.error(f"[{request_id}] Connection error to upstream API: {e}")
|
||||
@@ -1915,7 +2044,9 @@ class StreamingMixin:
|
||||
parsed_response=parsed_response,
|
||||
client=client,
|
||||
waste_signals=waste_signals,
|
||||
status_code=upstream_response.status_code,
|
||||
status_code=_stream_outcome_status(
|
||||
upstream_response.status_code, completed_normally
|
||||
),
|
||||
conversation_key=conversation_key,
|
||||
conversation_tokens_saved=conversation_tokens_saved,
|
||||
)
|
||||
@@ -2012,8 +2143,12 @@ class StreamingMixin:
|
||||
# stream closes (see finally: block below). Not on the hot path
|
||||
# for anything the client sees.
|
||||
full_sse_bytes = bytearray()
|
||||
completed_normally = False
|
||||
stream_failed = False
|
||||
stream_terminal = False
|
||||
|
||||
async def generate():
|
||||
nonlocal completed_normally, stream_failed, stream_terminal
|
||||
try:
|
||||
assert backend is not None
|
||||
|
||||
@@ -2088,9 +2223,16 @@ class StreamingMixin:
|
||||
]
|
||||
|
||||
# Handle errors
|
||||
if event.event_type == "error":
|
||||
failed, terminal = _sse_event_outcome(
|
||||
event.event_type, json.dumps(event.data), "anthropic"
|
||||
)
|
||||
stream_failed = stream_failed or failed
|
||||
stream_terminal = stream_terminal or terminal
|
||||
if failed:
|
||||
logger.error(f"[{request_id}] Bedrock stream error: {event.data}")
|
||||
|
||||
completed_normally = stream_terminal and not stream_failed
|
||||
|
||||
except Exception as e:
|
||||
error_message = format_exception_message(e)
|
||||
logger.error(f"[{request_id}] Bedrock streaming error: {error_message}")
|
||||
@@ -2110,7 +2252,7 @@ class StreamingMixin:
|
||||
# and _stream_openai_via_backend (OpenAI-via-backend
|
||||
# sibling). Run before the outcome funnel so prefix state
|
||||
# is consistent regardless of metric path.
|
||||
if prefix_tracker is not None:
|
||||
if completed_normally and prefix_tracker is not None:
|
||||
import copy as _copy
|
||||
|
||||
tracker_messages = (
|
||||
@@ -2169,6 +2311,7 @@ class StreamingMixin:
|
||||
overhead_ms=optimization_latency,
|
||||
tags=tags,
|
||||
client=client,
|
||||
status_code=_stream_outcome_status(200, completed_normally),
|
||||
log_full_messages=getattr(self.config, "log_full_messages", False),
|
||||
cache_read_tokens=stream_state["cache_read_input_tokens"],
|
||||
cache_write_tokens=stream_state["cache_creation_input_tokens"],
|
||||
@@ -2263,6 +2406,8 @@ class StreamingMixin:
|
||||
# closes (cheap, no buffering of in-flight chunks back to
|
||||
# the client).
|
||||
full_sse_bytes = bytearray()
|
||||
completed_normally = False
|
||||
stream_failed = False
|
||||
|
||||
def _absorb(usage: dict[str, int] | None) -> None:
|
||||
if not usage:
|
||||
@@ -2288,6 +2433,10 @@ class StreamingMixin:
|
||||
if parsed is not None and not stream_state["output_tokens"]:
|
||||
stream_state["output_tokens"] = parsed
|
||||
yield chunk_bytes
|
||||
stream_failed = stream_state.get("stream_failed", False)
|
||||
completed_normally = (
|
||||
stream_state.get("stream_terminal", False) and not stream_failed
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"[{request_id}] Backend streaming error: {e}")
|
||||
error_data = public_errors.openai_error_body(
|
||||
@@ -2341,7 +2490,7 @@ class StreamingMixin:
|
||||
# the non-streaming sibling. Done before outcome funnel
|
||||
# so prefix state is consistent regardless of metric
|
||||
# path.
|
||||
if prefix_tracker is not None:
|
||||
if completed_normally and prefix_tracker is not None:
|
||||
tracker_messages = (
|
||||
optimized_messages
|
||||
if optimized_messages is not None
|
||||
@@ -2389,6 +2538,7 @@ class StreamingMixin:
|
||||
overhead_ms=optimization_latency,
|
||||
tags=tags,
|
||||
client=client,
|
||||
status_code=_stream_outcome_status(200, completed_normally),
|
||||
log_full_messages=getattr(self.config, "log_full_messages", False),
|
||||
cache_read_tokens=cache_read_tokens,
|
||||
cache_write_tokens=cache_write_tokens,
|
||||
|
||||
@@ -112,7 +112,7 @@ def _gemini_provider(sent: list[dict[str, Any]]):
|
||||
if ":streamGenerateContent" in str(request.url):
|
||||
chunks = (
|
||||
'data: {"candidates": [{"content": {"role": "model", '
|
||||
'"parts": [{"text": "ack"}]}}]}\n\n'
|
||||
'"parts": [{"text": "ack"}]}, "finishReason": "STOP"}]}\n\n'
|
||||
+ f"data: {json.dumps({'usageMetadata': usage})}\n\n"
|
||||
)
|
||||
return httpx.Response(
|
||||
|
||||
@@ -81,7 +81,8 @@ class TestMidTurnSteering:
|
||||
proxy._config.retry_max_delay_ms = 0
|
||||
proxy.config = proxy._config
|
||||
proxy.memory_handler = None
|
||||
proxy._parse_sse_usage_from_buffer = MagicMock(return_value=None)
|
||||
# Exercise real event parsing: successful completion requires the
|
||||
# message_stop marker before pending messages can be drained.
|
||||
proxy._finalize_stream_response = AsyncMock(return_value=None)
|
||||
return proxy
|
||||
|
||||
|
||||
@@ -183,6 +183,77 @@ class _Backend:
|
||||
yield # pragma: no cover
|
||||
|
||||
|
||||
class _FailingBackend(_Backend):
|
||||
async def stream_message(self, body, headers):
|
||||
self.served = True
|
||||
yield SimpleNamespace(
|
||||
event_type="message_start",
|
||||
raw_sse="event: message_start\ndata: {}\n\n",
|
||||
data={},
|
||||
)
|
||||
raise RuntimeError("stream finalization failed")
|
||||
|
||||
|
||||
class _ErrorEventBackend(_Backend):
|
||||
async def stream_message(self, body, headers):
|
||||
self.served = True
|
||||
yield SimpleNamespace(
|
||||
event_type="error",
|
||||
raw_sse='event: error\ndata: {"error": {"type": "api_error"}}\n\n',
|
||||
data={"error": {"type": "api_error"}},
|
||||
)
|
||||
|
||||
async def stream_openai_message(self, body, headers):
|
||||
self.served = True
|
||||
yield 'data: {"type": "er'
|
||||
yield 'ror", "message": "provider failed"}\n\n'
|
||||
yield "data: [DONE]\n\n"
|
||||
|
||||
|
||||
async def _drive_with_outcome(handler, backend):
|
||||
from headroom.proxy.handlers.streaming import StreamingMixin
|
||||
|
||||
response = await StreamingMixin._stream_response_bedrock(
|
||||
handler,
|
||||
{"messages": []},
|
||||
{},
|
||||
"anthropic",
|
||||
"m",
|
||||
"rid",
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
[],
|
||||
{},
|
||||
0.0,
|
||||
backend=backend,
|
||||
)
|
||||
async for _ in response.body_iterator:
|
||||
pass
|
||||
|
||||
|
||||
async def _drive_openai_with_outcome(handler, backend):
|
||||
from headroom.proxy.handlers.streaming import StreamingMixin
|
||||
|
||||
response = await StreamingMixin._stream_openai_via_backend(
|
||||
handler,
|
||||
{"messages": []},
|
||||
{},
|
||||
"m",
|
||||
"rid",
|
||||
0.0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
[],
|
||||
{},
|
||||
0.0,
|
||||
backend=backend,
|
||||
)
|
||||
async for _ in response.body_iterator:
|
||||
pass
|
||||
|
||||
|
||||
async def _drive(handler, **kw):
|
||||
from headroom.proxy.handlers.streaming import StreamingMixin
|
||||
|
||||
@@ -240,6 +311,271 @@ def test_streaming_without_a_route_uses_the_configured_backend():
|
||||
assert configured.served
|
||||
|
||||
|
||||
def test_failed_stream_finalization_is_not_counted_as_completed():
|
||||
backend = _FailingBackend("anthropic")
|
||||
outcomes = []
|
||||
|
||||
async def record_outcome(outcome):
|
||||
outcomes.append(outcome)
|
||||
|
||||
handler = _handler(backend)
|
||||
handler._extract_anthropic_cache_ttl_metrics = lambda usage: (0, 0)
|
||||
handler._record_request_outcome = record_outcome
|
||||
asyncio.run(_drive_with_outcome(handler, backend))
|
||||
|
||||
assert len(outcomes) == 1
|
||||
assert outcomes[0].status_code == 502
|
||||
|
||||
|
||||
def test_stream_outcome_status_preserves_success_and_reclassifies_failure():
|
||||
from headroom.proxy.handlers.streaming import _stream_outcome_status
|
||||
|
||||
assert _stream_outcome_status(200, True) == 200
|
||||
assert _stream_outcome_status(200, False) == 502
|
||||
assert _stream_outcome_status(529, False) == 502
|
||||
|
||||
|
||||
def test_sse_error_detection_handles_native_and_split_events():
|
||||
from headroom.proxy.handlers.streaming import _sse_contains_error_event
|
||||
|
||||
assert _sse_contains_error_event(b'event: error\ndata: {"type":"error"}\n\n')
|
||||
assert _sse_contains_error_event(b'data: {"type":"er' + b'ror"}\n\n')
|
||||
assert not _sse_contains_error_event(b'data: {"type":"message"}\n\n')
|
||||
|
||||
|
||||
def test_sse_error_text_in_response_content_is_not_an_error():
|
||||
from headroom.proxy.handlers.streaming import _sse_contains_error_event
|
||||
|
||||
assert not _sse_contains_error_event(b'data: {"text":"event: error"}\n\n')
|
||||
|
||||
|
||||
def test_responses_failed_envelope_is_an_error():
|
||||
from headroom.proxy.handlers.streaming import _sse_contains_error_event
|
||||
|
||||
assert _sse_contains_error_event(
|
||||
b'data: {"type":"response.failed","response":{"status":"failed",'
|
||||
b'"error":{"code":"server_error"}}}\n\n'
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("provider", ["anthropic", "openai"])
|
||||
@pytest.mark.parametrize("complete", [False, True])
|
||||
def test_backend_clean_eof_requires_protocol_completion(provider, complete):
|
||||
from types import MethodType
|
||||
|
||||
from headroom.proxy.handlers.streaming import StreamingMixin
|
||||
|
||||
class Backend(_Backend):
|
||||
async def stream_message(self, body, headers):
|
||||
yield SimpleNamespace(
|
||||
event_type="message_start", raw_sse="event: message_start\ndata: {}\n\n", data={}
|
||||
)
|
||||
if complete:
|
||||
yield SimpleNamespace(
|
||||
event_type="message_stop",
|
||||
raw_sse='event: message_stop\ndata: {"type":"message_stop"}\n\n',
|
||||
data={"type": "message_stop"},
|
||||
)
|
||||
|
||||
async def stream_openai_message(self, body, headers):
|
||||
yield 'data: {"choices":[{"delta":{"content":"partial"}}]}\n\n'
|
||||
if complete:
|
||||
yield "data: [DONE]\n\n"
|
||||
|
||||
outcomes = []
|
||||
|
||||
async def record(outcome):
|
||||
outcomes.append(outcome)
|
||||
|
||||
handler = _handler(Backend(provider))
|
||||
handler._record_request_outcome = record
|
||||
handler._extract_anthropic_cache_ttl_metrics = lambda usage: (0, 0)
|
||||
handler._parse_sse_usage_from_buffer = MethodType(
|
||||
StreamingMixin._parse_sse_usage_from_buffer, handler
|
||||
)
|
||||
drive = _drive_with_outcome if provider == "anthropic" else _drive_openai_with_outcome
|
||||
asyncio.run(drive(handler, handler.anthropic_backend))
|
||||
assert len(outcomes) == 1
|
||||
assert outcomes[0].status_code == (200 if complete else 502)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("provider", ["anthropic", "openai"])
|
||||
@pytest.mark.parametrize("complete", [False, True])
|
||||
def test_backend_prefix_tracker_advances_only_after_completion(provider, complete):
|
||||
from types import MethodType
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from headroom.proxy.handlers.streaming import StreamingMixin
|
||||
|
||||
class Backend(_Backend):
|
||||
async def stream_message(self, body, headers):
|
||||
yield SimpleNamespace(event_type="message_start", raw_sse="", data={})
|
||||
if complete:
|
||||
yield SimpleNamespace(event_type="message_stop", raw_sse="", data={})
|
||||
|
||||
async def stream_openai_message(self, body, headers):
|
||||
yield 'data: {"choices":[{"delta":{"content":"partial"}}]}\n\n'
|
||||
if complete:
|
||||
yield "data: [DONE]\n\n"
|
||||
|
||||
backend = Backend(provider)
|
||||
handler = _handler(backend)
|
||||
handler._extract_anthropic_cache_ttl_metrics = lambda usage: (0, 0)
|
||||
handler._parse_sse_usage_from_buffer = MethodType(
|
||||
StreamingMixin._parse_sse_usage_from_buffer, handler
|
||||
)
|
||||
handler._parse_sse_to_response = lambda *args: None
|
||||
handler._assistant_message_from_response_json = lambda response: None
|
||||
tracker = MagicMock()
|
||||
tracker.classify_cache_miss.return_value = SimpleNamespace(is_miss=False)
|
||||
|
||||
async def drive():
|
||||
arguments = {
|
||||
"body": {"messages": []},
|
||||
"headers": {},
|
||||
"model": "m",
|
||||
"request_id": "rid",
|
||||
"original_tokens": 0,
|
||||
"optimized_tokens": 0,
|
||||
"tokens_saved": 0,
|
||||
"transforms_applied": [],
|
||||
"tags": {},
|
||||
"optimization_latency": 0.0,
|
||||
"backend": backend,
|
||||
"prefix_tracker": tracker,
|
||||
}
|
||||
if provider == "anthropic":
|
||||
response = await StreamingMixin._stream_response_bedrock(
|
||||
handler, provider=provider, **arguments
|
||||
)
|
||||
else:
|
||||
response = await StreamingMixin._stream_openai_via_backend(
|
||||
handler, start_time=0.0, **arguments
|
||||
)
|
||||
async for _ in response.body_iterator:
|
||||
pass
|
||||
|
||||
asyncio.run(drive())
|
||||
assert tracker.update_from_response.call_count == int(complete)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"event, data, expected_failed, expected_terminal",
|
||||
[
|
||||
("response.completed", '{"type":"response.completed"}', False, True),
|
||||
("response.incomplete", '{"type":"response.incomplete"}', False, True),
|
||||
("response.failed", '{"type":"response.failed"}', True, False),
|
||||
(None, "[DONE]", False, False),
|
||||
],
|
||||
)
|
||||
def test_responses_terminal_semantics(event, data, expected_failed, expected_terminal):
|
||||
from headroom.proxy.handlers.streaming import _sse_event_outcome
|
||||
|
||||
assert _sse_event_outcome(event, data, "responses") == (expected_failed, expected_terminal)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("event_type", ['["error"]', '{"unexpected":true}', "7", "true"])
|
||||
def test_malformed_event_type_is_not_interpreted_as_a_protocol_marker(event_type):
|
||||
from headroom.proxy.handlers.streaming import _sse_contains_error_event
|
||||
|
||||
payload = ('data: {"type":' + event_type + "}\n\ndata: [DONE]\n\n").encode()
|
||||
assert not _sse_contains_error_event(payload)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("wrapped", [False, True])
|
||||
@pytest.mark.parametrize("finish_reason", ["STOP", "MAX_TOKENS"])
|
||||
def test_gemini_finish_reason_is_a_terminal_event(wrapped, finish_reason):
|
||||
import json
|
||||
|
||||
from headroom.proxy.handlers.streaming import _sse_event_outcome
|
||||
|
||||
payload = {"candidates": [{"index": 0, "finishReason": finish_reason}]}
|
||||
if wrapped:
|
||||
payload = {"response": payload}
|
||||
assert _sse_event_outcome(None, json.dumps(payload), "gemini") == (False, True)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("wrapped", [False, True])
|
||||
def test_gemini_prompt_block_is_a_failed_outcome(wrapped):
|
||||
import json
|
||||
|
||||
from headroom.proxy.handlers.streaming import _sse_event_outcome
|
||||
|
||||
payload = {"promptFeedback": {"blockReason": "SAFETY"}}
|
||||
if wrapped:
|
||||
payload = {"response": payload}
|
||||
failed, _ = _sse_event_outcome(None, json.dumps(payload), "gemini")
|
||||
assert failed
|
||||
|
||||
|
||||
def test_gemini_completion_requires_all_requested_candidates():
|
||||
from headroom.proxy.handlers.streaming import StreamingMixin
|
||||
|
||||
parser = StreamingMixin()
|
||||
state = {"sse_buffer": bytearray(), "gemini_candidate_count": 2}
|
||||
state["sse_buffer"].extend(b'data: {"candidates":[{"index":0,"finishReason":"STOP"}]}\n\n')
|
||||
parser._parse_sse_usage_from_buffer(state, "gemini")
|
||||
assert not state.get("stream_terminal", False)
|
||||
state["sse_buffer"].extend(b'data: {"candidates":[{"index":1,"finishReason":"STOP"}]}\n\n')
|
||||
parser._parse_sse_usage_from_buffer(state, "gemini")
|
||||
assert state["stream_terminal"]
|
||||
|
||||
|
||||
def test_outcome_classification_runs_once_for_a_chunked_complete_event(monkeypatch):
|
||||
from headroom.proxy.handlers import streaming
|
||||
|
||||
seen = []
|
||||
classify = streaming._sse_event_outcome
|
||||
|
||||
def record(*args):
|
||||
seen.append(args)
|
||||
return classify(*args)
|
||||
|
||||
monkeypatch.setattr(streaming, "_sse_event_outcome", record)
|
||||
state = {"sse_buffer": bytearray()}
|
||||
parser = streaming.StreamingMixin()
|
||||
payload = b'data: {"text":"' + b"x" * 10_000 + b'"}\n\n'
|
||||
for offset in range(0, len(payload), 101):
|
||||
state["sse_buffer"].extend(payload[offset : offset + 101])
|
||||
parser._parse_sse_usage_from_buffer(state, "openai")
|
||||
assert len(seen) == 1
|
||||
assert not state["sse_buffer"]
|
||||
|
||||
|
||||
def test_backend_error_event_is_not_counted_as_completed():
|
||||
backend = _ErrorEventBackend("anthropic")
|
||||
outcomes = []
|
||||
|
||||
async def record_outcome(outcome):
|
||||
outcomes.append(outcome)
|
||||
|
||||
handler = _handler(backend)
|
||||
handler._extract_anthropic_cache_ttl_metrics = lambda usage: (0, 0)
|
||||
handler._record_request_outcome = record_outcome
|
||||
asyncio.run(_drive_with_outcome(handler, backend))
|
||||
|
||||
assert backend.served
|
||||
assert len(outcomes) == 1
|
||||
assert outcomes[0].status_code == 502
|
||||
|
||||
|
||||
def test_openai_backend_error_event_is_not_counted_as_completed():
|
||||
backend = _ErrorEventBackend("openai")
|
||||
outcomes = []
|
||||
|
||||
async def record_outcome(outcome):
|
||||
outcomes.append(outcome)
|
||||
|
||||
handler = _handler(backend)
|
||||
handler._parse_sse_usage_from_buffer = lambda state, provider: None
|
||||
handler._record_request_outcome = record_outcome
|
||||
asyncio.run(_drive_openai_with_outcome(handler, backend))
|
||||
|
||||
assert backend.served
|
||||
assert len(outcomes) == 1
|
||||
assert outcomes[0].status_code == 502
|
||||
|
||||
|
||||
async def _drive_openai(handler, **kw):
|
||||
from headroom.proxy.handlers.streaming import StreamingMixin
|
||||
|
||||
|
||||
Reference in New Issue
Block a user