mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy/streaming): tolerate malformed upstream SSE events (#2558)
## Description
Three parsers read upstream-controlled SSE bodies with `json.loads` and
then
reach into the result without checking its shape:
`_parse_sse_usage`, `_parse_sse_usage_from_buffer` and
`_parse_sse_to_response`.
A valid-JSON value of the wrong shape passes the `json.JSONDecodeError`
guard
and then raises. Three distinct ways:
- a non-object event (`["x"]`, `"str"`, `42`) makes `.get` raise
`AttributeError`
- an explicit `"key": null` bypasses a `.get(key, {})` default, because
the
default applies only to a *missing* key, not a present-and-null one
- an unhashable `index` (a list or dict) raises `TypeError: unhashable
type`
when used to key the block map
In `_parse_sse_to_response` the raise escapes into
`_finalize_stream_response`
and tears down a stream the client is already reading. Several call
sites reach
it from a bare `finally`, so there is no handler above it at all.
**This is a robustness invariant, not a bug report about any specific
upstream.**
I am not claiming Anthropic emits these shapes; I have no evidence of
that. The
Anthropic handler serves any Anthropic-shaped upstream (`--backend
anthropic|bedrock|openrouter|anyllm|litellm-<provider>`,
`ANTHROPIC_TARGET_API_URL`), so Bedrock, OpenRouter, LiteLLM, vLLM and
Vertex
all flow through these parsers. A parser on a proxy's critical streaming
path
should not raise on a well-formed-JSON frame of unexpected shape,
whatever the
source. The guard costs one `isinstance`; being wrong costs the user's
session.
**Scope: frame shape only.** Value validity — a token count that is a
string,
`Infinity`, or a >4300-digit integer literal — is a separate defect
class that
reaches the same parsers and is deliberately left for a follow-up.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
The whole change is three one-line normalizers plus their call sites
(docstrings elided here; they are in the diff):
```python
def _sse_dict(value: Any) -> dict[str, Any]:
return value if isinstance(value, dict) else {}
def _sse_str(value: Any) -> str:
return value if isinstance(value, str) else ""
def _sse_index(value: Any, default: int | None = None) -> int | None:
if isinstance(value, bool) or not isinstance(value, int):
return default
return value
```
At a call site, in `_parse_sse_usage_from_buffer`:
```python
# before
msg = data.get("message", {}) # default fires only if "message" is ABSENT
msg_usage = msg.get("usage", {}) # AttributeError if msg is None / str / list
# after
msg_usage = _sse_dict(_sse_dict(data.get("message")).get("usage"))
```
Nested because `message` and `message.usage` are independently
upstream-controlled — either can be `null` or the wrong type, and the
outer
guard cannot protect the inner access. The `.get` defaults are dropped
as
redundant, since `_sse_dict` maps absent, `null` and wrong-type onto one
result.
Why a helper and not `.get(key) or {}`: `or` branches on truthiness, so
it
closes the `null` case and leaves every truthy non-dict (`"overloaded"`,
`42`,
`["x"]`) still reaching `.get`. `isinstance` keys off the property the
next line
actually requires — is this a mapping? — and it narrows the type for
`mypy`, so
the next unguarded `.get` added to one of these chains becomes a type
error
rather than being inferred as `Any`.
**Well-formed frames are unaffected.** Each normalizer is the identity
function
on a value of the expected type, so no happy-path value is rewritten.
The
real-stream happy-path fixtures pass against `origin/main` and this
branch
alike — they sit inside the 217 of 408 cases that already passed before
the fix.
**A malformed frame degrades rather than being swallowed.** A
wrong-shaped
container becomes an empty one of the right type, so the affected field
contributes nothing: `_parse_sse_usage*` omits that key (or returns
`None`) and
`_parse_sse_to_response` skips the block. The cost is one lost usage
number,
which `_finalize_stream_response` already handles — it falls back to a
byte-derived estimate and tags the outcome
`output_tokens_source=estimated_bytes` instead of passing a guess off as
measured. The normalizers themselves are deliberately silent: they sit
on a
per-frame, per-field hot path, so even a debug line would fire thousands
of
times per stream.
- **headroom/proxy/handlers/streaming.py**
- Add three shared normalizers — `_sse_dict`, `_sse_str`, `_sse_index` —
so
every upstream-derived container is shape-checked with one idiom instead
of
a mix of inline `isinstance` and `.get(..., {})` defaults.
- `_sse_index` rejects `bool` as well as non-`int`, so `true` cannot
silently
alias block index `1`, and unhashable values cannot reach the block map.
- Guard the positions the earlier pass missed:
`content_block_delta.delta`,
`message_delta.delta`, `message_delta.usage` (`dict.update` on a
non-mapping
raises — the `message_start` twin was already guarded, this sibling was
not),
the `index` on all three `content_block_*` events, and the string
accumulators for `text` / `partial_json` / `thinking`.
- Guard `_parse_sse_usage`, which had no shape checks at all — not on
the
top-level event, nor on the anthropic, openai or gemini branches.
- Guard the gemini `usageMetadata` branch in
`_parse_sse_usage_from_buffer`,
which used a truthiness check where its two sibling branches used
`isinstance`.
- Stop the non-standard-block copy-through from seeding the parser's own
scratch keys (`_partial_json`, `thinking_buffer`) via
`_BLOCK_SCRATCH_KEYS`,
and tolerate a non-list `citations` that a copy-through already placed.
- **tests/test_streaming_sse_malformed_events.py** — new file, 408
cases.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`, `ruff format`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added
The sweep is enumerated, not randomized: the failure surface is finite,
so the
cases can be exact and no `hypothesis` dependency is needed. This
matches the
existing precedent in `tests/test_toin_observation_only.py`.
Happy-path fixtures are transcribed from real `api.anthropic.com`
streams
rather than hand-written, which corrected four wrong assumptions in the
previous fixtures: `message_delta.usage` repeats the *full* usage block
(not
just `output_tokens`); `message_start.message` carries `stop_details:
null`
alongside the other two nulls; `usage.cache_creation` is always present
as a
nested object and is what `_extract_anthropic_cache_ttl_metrics` reads;
and a
`tool_use` block carries a non-standard `caller` field.
### Test Output
```text
$ pytest tests/test_streaming_sse_malformed_events.py
408 passed
# same file against origin/main's parsers
191 failed, 217 passed
# regression sweep: all 22 test files touching SSE parsing
710 passed, 5 skipped
$ mypy headroom --ignore-missing-imports
Success: no issues found in 509 source files
```
## Real Behavior Proof
- Environment: macOS 15 (Darwin 25.4.0, arm64), Python 3.13.7, Rust
1.95.0
per `rust-toolchain.toml`, `uv sync --extra dev`. Proxy build
`0.33.0-dev`
(editable install of this branch), confirmed per run via `/health`.
- Exact command / steps: enumerated unit sweep, a parser differential
against
`origin/main`, HTTP-level fault injection through a local
Anthropic-shaped
upstream, then a live Claude Code session on this build.
```bash
# 1. the enumerated sweep
pytest tests/test_streaming_sse_malformed_events.py -q
# 2. parser differential. origin/main's streaming.py is loaded via
# `git show origin/main:...` under a synthetic module name — NOT `git
stash`,
# which silently no-ops once the fix is committed and yields a false
pass.
# The same malformed frame goes into _parse_sse_usage,
# _parse_sse_usage_from_buffer and _parse_sse_to_response on both
versions.
# 3. HTTP-level fault injection. A local server stands in for one of the
# Anthropic-shaped upstreams this handler serves and streams a
# well-formed-JSON frame of the wrong shape: message_start.message =
null.
ANTHROPIC_TARGET_API_URL=http://127.0.0.1:9099 headroom proxy --port
8799
curl -N -X POST http://127.0.0.1:8799/v1/messages \
-H 'content-type: application/json' -H 'anthropic-version: 2023-06-01' \
-d
'{"model":"claude-fake","max_tokens":16,"stream":true,"messages":[{"role":"user","content":"hi"}]}'
# 4. live positive control: a real Claude Code session on this build,
# in an unrelated repository
headroom wrap claude --port 8788
# 5. types, then the full local gate (rust + python + commitlint)
mypy headroom --ignore-missing-imports
make ci-precheck
```
- Observed result: (1) `408 passed`. (2) all three parsers raise
`AttributeError: 'NoneType' object has no attribute 'get'` on
`origin/main`
and return `None` on this branch from identical input. (3) the malformed
stream completes end to end — `HTTP 200`, 568 bytes, all six SSE frames
including the text delta delivered to the client, zero tracebacks,
`error:
null` in the request record. Usage degrades partially and correctly:
`output_tokens=7` survives from the well-formed `message_delta`, while
`cache_read`/`cache_write` — which live inside the nulled
`message_start.message.usage` — fall back to `0` instead of raising. (4)
421 live
requests over 24h (`claude-opus-5`, `claude-sonnet-5`) in an unrelated
repository, `error: null` on all 421 and zero streaming exceptions in
the
proxy log. Of the 340 that reached the streaming finalizer, 328 carry
`output_tokens_source=provider`; 12 fell back to the byte estimate,
every one
on a stream truncated in its first frames — `output_tokens = total_bytes
// 40`
puts those bodies at 120-919 bytes against inputs of 149k-197k tokens,
and that
fallback is pre-existing and identical on `origin/main`, not a guard
dropping a
frame. The other 81 requests never reached the streaming finalizer,
which sets
that tag unconditionally, so they are non-streaming. `cache_read`
climbed to
256,027 with 407/421 cache hits and 120,129 tokens saved. So the guards
are
identity-transparent on well-formed frames, and the sustained
prefix-cache
growth means `_parse_sse_to_response` ran throughout — all three parsers
exercised on live traffic. (5) `Success: no issues found in
509 source files`; `ci-precheck PASSED`.
- **Real upstream capture:** three live streaming `POST /v1/messages`
calls to
`api.anthropic.com` (`claude-haiku-4-5`) — a text stream, a `tool_use`
stream
and an extended-thinking stream — captured as raw wire bytes and used to
build
the happy-path fixtures. This is what surfaced the four fixture errors
above.
- **Differential:** the 408-case file was run against `origin/main`'s
parsers via
`git show origin/main:...` (not `git stash`, which silently no-ops once
the fix
is committed and produces a false pass). 191 cases fail there and pass
here.
- **Exhaustive shape fuzz:** 700 generated cases crossing every
upstream-derived
container position in the three parsers with `None`, `"str"`, `42`,
`3.5`,
`True`, `False`, `["x"]`, `[]`, `{}`, `{"k":"v"}`. Zero raises after the
change;
before it, the positions listed under Changes Made all raise.
- **Per-position legitimacy check:** each injection position in the
sweep was
confirmed to produce at least one genuine failure against `origin/main`,
so no
case passes vacuously. This caught four delta-shaped tests that were
injecting
their fault before any content block had opened, where the parser skips
the
body and the guard is never reached.
- Not tested: the malformed frames are synthetic and the incidence claim
is
unproven — limits below.
- **No evidence these shapes occur in the wild.** The fault shapes are
derived
from the parsers' own unguarded access positions, not observed in real
upstream traffic. I searched the tracker for this crash signature and
both
function names and found no report. The claim is survivability, not
incidence.
- The real API cannot be made to emit malformed frames on demand, so the
fault
injection is synthetic; only the happy-path fixtures and the live
positive
control come from real traffic.
- No live third-party gateway (Bedrock/OpenRouter/LiteLLM/Vertex)
exercised,
despite those being the motivating case. The fault injection used a
local
stand-in reached via `ANTHROPIC_TARGET_API_URL`, not a real gateway.
- No WebSocket path, no Windows.
- The live session is a positive control — it shows the guards do not
alter
well-formed handling. It does not show a guard firing, and cannot: the
normalizers are silent and `api.anthropic.com` sends well-formed frames.
- Value-validity faults are out of scope and still raise; see Additional
Notes.
## Additional Notes
Two findings left deliberately unfixed, both pre-existing and both
outside the
frame-shape scope of this change:
1. **Value validity.** `_extract_anthropic_cache_ttl_metrics` calls
`int()` on an
upstream value with no `try`; `_usage_int` does not catch
`OverflowError`, and
`json.loads` accepts bare `Infinity`; `except json.JSONDecodeError` is
too
narrow for a >4300-digit integer literal (plain `ValueError`) or deep
nesting
(`RecursionError`); and the anthropic and gemini branches export
non-`int`
token values out of a `dict[str, int]`-annotated function.
2. **Reconstruction fidelity.** `content_block_start` copies through
unknown
fields only for non-standard block types, so the real `caller` field on
a
`tool_use` block is dropped on reconstruction.
`test_real_tool_use_stream_reconstructs_tool_input` pins the current
behaviour rather than asserting the desired one.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — n/a, no
user-facing surface changed
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
No new dependencies added.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2558"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
@@ -197,6 +197,11 @@ def _thinking_for_stream(payload: object) -> ThinkingTokens:
|
||||
return ThinkingTokens()
|
||||
|
||||
|
||||
def _sse_dict(value: Any) -> dict[str, Any]:
|
||||
"""Return an upstream SSE object, or an empty object for a wrong-shaped value."""
|
||||
return value if isinstance(value, dict) else {}
|
||||
|
||||
|
||||
def _parse_completion_tokens_from_sse_chunk(chunk_bytes: bytes) -> int | None:
|
||||
"""Extract `usage.completion_tokens` from a single SSE chunk if present.
|
||||
|
||||
@@ -345,6 +350,8 @@ class StreamingMixin:
|
||||
data = json.loads(data_str)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if not isinstance(data, dict):
|
||||
continue
|
||||
|
||||
usage = {}
|
||||
|
||||
@@ -354,8 +361,7 @@ class StreamingMixin:
|
||||
event_type = data.get("type", "")
|
||||
|
||||
if event_type == "message_start":
|
||||
msg = data.get("message", {})
|
||||
msg_usage = msg.get("usage", {})
|
||||
msg_usage = _sse_dict(_sse_dict(data.get("message")).get("usage"))
|
||||
if msg_usage:
|
||||
usage["input_tokens"] = msg_usage.get("input_tokens", 0)
|
||||
usage["cache_read_input_tokens"] = msg_usage.get(
|
||||
@@ -371,24 +377,24 @@ class StreamingMixin:
|
||||
usage["cache_creation_ephemeral_1h_input_tokens"] = cache_write_1h
|
||||
|
||||
elif event_type == "message_delta":
|
||||
delta_usage = data.get("usage", {})
|
||||
delta_usage = _sse_dict(data.get("usage"))
|
||||
if delta_usage:
|
||||
usage["output_tokens"] = delta_usage.get("output_tokens", 0)
|
||||
|
||||
elif provider == "openai":
|
||||
# OpenAI sends usage in final chunk (when stream_options.include_usage=true)
|
||||
chunk_usage = data.get("usage")
|
||||
chunk_usage = _sse_dict(data.get("usage"))
|
||||
if chunk_usage:
|
||||
usage["input_tokens"] = chunk_usage.get("prompt_tokens", 0)
|
||||
usage["output_tokens"] = chunk_usage.get("completion_tokens", 0)
|
||||
# OpenAI has cached tokens in prompt_tokens_details
|
||||
details = chunk_usage.get("prompt_tokens_details") or {}
|
||||
details = _sse_dict(chunk_usage.get("prompt_tokens_details"))
|
||||
usage["cache_read_input_tokens"] = details.get("cached_tokens", 0)
|
||||
|
||||
elif provider == "gemini":
|
||||
# Gemini sends usageMetadata in each streaming chunk
|
||||
# Format: {"usageMetadata": {"promptTokenCount": N, "candidatesTokenCount": M}}
|
||||
usage_meta = data.get("usageMetadata")
|
||||
usage_meta = _sse_dict(data.get("usageMetadata"))
|
||||
if usage_meta:
|
||||
usage["input_tokens"] = usage_meta.get("promptTokenCount", 0)
|
||||
usage["output_tokens"] = gemini_output_tokens(usage_meta)
|
||||
@@ -451,6 +457,8 @@ class StreamingMixin:
|
||||
data = json.loads(data_str)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if not isinstance(data, dict):
|
||||
continue
|
||||
|
||||
if provider == "gemini":
|
||||
finished = stream_state.setdefault("gemini_finished_candidates", set())
|
||||
@@ -461,8 +469,7 @@ class StreamingMixin:
|
||||
if provider == "anthropic":
|
||||
event_type = data.get("type", "")
|
||||
if event_type == "message_start":
|
||||
msg = data.get("message", {})
|
||||
msg_usage = msg.get("usage", {})
|
||||
msg_usage = _sse_dict(_sse_dict(data.get("message")).get("usage"))
|
||||
if msg_usage:
|
||||
# Absent stays absent: the finalizer falls back to the
|
||||
# tokenizer derivation only when input_tokens is None.
|
||||
@@ -485,7 +492,7 @@ class StreamingMixin:
|
||||
f"cache_write={usage_found.get('cache_creation_input_tokens')}"
|
||||
)
|
||||
elif event_type == "message_delta":
|
||||
delta_usage = data.get("usage", {})
|
||||
delta_usage = _sse_dict(data.get("usage"))
|
||||
if delta_usage:
|
||||
usage_found["output_tokens"] = delta_usage.get("output_tokens", 0)
|
||||
|
||||
@@ -525,7 +532,7 @@ class StreamingMixin:
|
||||
)
|
||||
|
||||
elif provider == "gemini":
|
||||
usage_meta = data.get("usageMetadata")
|
||||
usage_meta = _sse_dict(data.get("usageMetadata"))
|
||||
if usage_meta:
|
||||
usage_found["input_tokens"] = usage_meta.get("promptTokenCount", 0)
|
||||
usage_found["output_tokens"] = gemini_output_tokens(usage_meta)
|
||||
|
||||
@@ -0,0 +1,611 @@
|
||||
"""Malformed-upstream tolerance for SSE event parsing.
|
||||
|
||||
Three parsers read upstream-controlled SSE bodies: ``_parse_sse_usage``,
|
||||
``_parse_sse_usage_from_buffer`` and ``_parse_sse_to_response``. A valid-JSON
|
||||
event of unexpected *shape* must be skipped rather than raise. In
|
||||
``_parse_sse_to_response`` the raise escapes into ``_finalize_stream_response``
|
||||
and tears down a stream the client is already reading.
|
||||
|
||||
Scope: these cover **frame shape** — the type of a container the parser reaches
|
||||
into. Value validity (a token count that is a string, ``Infinity``, or a
|
||||
>4300-digit integer literal) is a separate defect class and is not covered here.
|
||||
|
||||
The happy-path fixtures are transcribed from real ``api.anthropic.com``
|
||||
``/v1/messages`` streams (``claude-haiku-4-5``), not hand-invented. Details a
|
||||
stand-in gets wrong, which these pin:
|
||||
|
||||
* ``message_start.message`` sends ``stop_reason``, ``stop_sequence`` and
|
||||
``stop_details`` as **null**. A guard that rejected nulls indiscriminately
|
||||
would break every ordinary request.
|
||||
* ``message_delta.usage`` repeats the *full* usage block, not just
|
||||
``output_tokens``.
|
||||
* ``usage.cache_creation`` is always present as a nested object — it is what
|
||||
``_extract_anthropic_cache_ttl_metrics`` reads.
|
||||
* A ``tool_use`` block carries a non-standard ``caller`` field, which is what
|
||||
exercises the copy-through branch in ``content_block_start``.
|
||||
* Extended thinking emits ``thinking_delta`` then ``signature_delta`` on block
|
||||
0, then a separate text block at index 1.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from headroom.proxy.handlers.streaming import StreamingMixin
|
||||
|
||||
|
||||
class _Handler(StreamingMixin):
|
||||
"""Bare mixin host — the parsers touch no other handler state."""
|
||||
|
||||
|
||||
# `None` is itself a value under test, so it cannot double as "not supplied".
|
||||
_UNSET = object()
|
||||
|
||||
_WRONG_TYPES = [None, "str", 42, 3.5, True, ["x"], [], {}]
|
||||
|
||||
|
||||
def _frame(event: str, payload: Any) -> str:
|
||||
"""One SSE frame in Anthropic's wire shape: an ``event:`` line then ``data:``."""
|
||||
return f"event: {event}\ndata: {json.dumps(payload)}\n\n"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fixtures transcribed from real api.anthropic.com streams.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_REAL_USAGE = {
|
||||
"input_tokens": 12,
|
||||
"cache_creation_input_tokens": 0,
|
||||
"cache_read_input_tokens": 0,
|
||||
"cache_creation": {"ephemeral_5m_input_tokens": 0, "ephemeral_1h_input_tokens": 0},
|
||||
"output_tokens": 1,
|
||||
"service_tier": "standard",
|
||||
"inference_geo": "not_available",
|
||||
}
|
||||
|
||||
_REAL_MESSAGE = {
|
||||
"id": "msg_011CdQD1YDscyCRWpGCcrwkA",
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"model": "claude-haiku-4-5-20251001",
|
||||
"content": [],
|
||||
# Real Anthropic sends all three as null on message_start.
|
||||
"stop_reason": None,
|
||||
"stop_sequence": None,
|
||||
"stop_details": None,
|
||||
"usage": _REAL_USAGE,
|
||||
}
|
||||
|
||||
_MESSAGE_DELTA = _frame(
|
||||
"message_delta",
|
||||
{
|
||||
"type": "message_delta",
|
||||
"delta": {"stop_reason": "end_turn", "stop_sequence": None, "stop_details": None},
|
||||
# The real message_delta repeats the whole usage block.
|
||||
"usage": {
|
||||
"input_tokens": 12,
|
||||
"cache_creation_input_tokens": 0,
|
||||
"cache_read_input_tokens": 0,
|
||||
"output_tokens": 4,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _text_stream(*, message: Any = _UNSET, extra: str = "", mid: str = "") -> str:
|
||||
"""The real single-text-block stream, optionally with frames injected.
|
||||
|
||||
``extra`` lands right after ``message_start``; ``mid`` lands *inside* the
|
||||
open content block, after ``content_block_start``. The distinction matters:
|
||||
a ``content_block_delta`` injected before any block has opened resolves to
|
||||
``target is None`` and the parser skips its whole body, so a delta-shaped
|
||||
fault placed in ``extra`` would never reach the code it is meant to test.
|
||||
"""
|
||||
msg = _REAL_MESSAGE if message is _UNSET else message
|
||||
return (
|
||||
_frame("message_start", {"type": "message_start", "message": msg})
|
||||
+ extra
|
||||
+ _frame(
|
||||
"content_block_start",
|
||||
{
|
||||
"type": "content_block_start",
|
||||
"index": 0,
|
||||
"content_block": {"type": "text", "text": ""},
|
||||
},
|
||||
)
|
||||
+ _frame("ping", {"type": "ping"})
|
||||
+ mid
|
||||
+ _frame(
|
||||
"content_block_delta",
|
||||
{
|
||||
"type": "content_block_delta",
|
||||
"index": 0,
|
||||
"delta": {"type": "text_delta", "text": "hi"},
|
||||
},
|
||||
)
|
||||
+ _frame("content_block_stop", {"type": "content_block_stop", "index": 0})
|
||||
+ _MESSAGE_DELTA
|
||||
+ _frame("message_stop", {"type": "message_stop"})
|
||||
)
|
||||
|
||||
|
||||
def _tool_use_stream() -> str:
|
||||
"""The real tool_use stream — note the non-standard ``caller`` field."""
|
||||
partials = ["", '{"ci', "ty", '": "Par', 'is"}']
|
||||
return (
|
||||
_frame("message_start", {"type": "message_start", "message": _REAL_MESSAGE})
|
||||
+ _frame(
|
||||
"content_block_start",
|
||||
{
|
||||
"type": "content_block_start",
|
||||
"index": 0,
|
||||
"content_block": {
|
||||
"type": "tool_use",
|
||||
"id": "toolu_01Kpi77sYFPtGoQbz6YMCSok",
|
||||
"name": "get_weather",
|
||||
"input": {},
|
||||
"caller": {"type": "direct"},
|
||||
},
|
||||
},
|
||||
)
|
||||
+ "".join(
|
||||
_frame(
|
||||
"content_block_delta",
|
||||
{
|
||||
"type": "content_block_delta",
|
||||
"index": 0,
|
||||
"delta": {"type": "input_json_delta", "partial_json": p},
|
||||
},
|
||||
)
|
||||
for p in partials
|
||||
)
|
||||
+ _frame("content_block_stop", {"type": "content_block_stop", "index": 0})
|
||||
+ _frame("message_stop", {"type": "message_stop"})
|
||||
)
|
||||
|
||||
|
||||
def _thinking_stream() -> str:
|
||||
"""The real extended-thinking stream: thinking block 0, then text block 1."""
|
||||
return (
|
||||
_frame("message_start", {"type": "message_start", "message": _REAL_MESSAGE})
|
||||
+ _frame(
|
||||
"content_block_start",
|
||||
{
|
||||
"type": "content_block_start",
|
||||
"index": 0,
|
||||
"content_block": {"type": "thinking", "thinking": "", "signature": ""},
|
||||
},
|
||||
)
|
||||
+ _frame(
|
||||
"content_block_delta",
|
||||
{
|
||||
"type": "content_block_delta",
|
||||
"index": 0,
|
||||
"delta": {"type": "thinking_delta", "thinking": "17"},
|
||||
},
|
||||
)
|
||||
+ _frame(
|
||||
"content_block_delta",
|
||||
{
|
||||
"type": "content_block_delta",
|
||||
"index": 0,
|
||||
"delta": {"type": "thinking_delta", "thinking": " * 23 = 391"},
|
||||
},
|
||||
)
|
||||
+ _frame(
|
||||
"content_block_delta",
|
||||
{
|
||||
"type": "content_block_delta",
|
||||
"index": 0,
|
||||
"delta": {"type": "signature_delta", "signature": "EtQCCpMBCBAYAipA"},
|
||||
},
|
||||
)
|
||||
+ _frame("content_block_stop", {"type": "content_block_stop", "index": 0})
|
||||
+ _frame(
|
||||
"content_block_start",
|
||||
{
|
||||
"type": "content_block_start",
|
||||
"index": 1,
|
||||
"content_block": {"type": "text", "text": ""},
|
||||
},
|
||||
)
|
||||
+ _frame(
|
||||
"content_block_delta",
|
||||
{
|
||||
"type": "content_block_delta",
|
||||
"index": 1,
|
||||
"delta": {"type": "text_delta", "text": "391"},
|
||||
},
|
||||
)
|
||||
+ _frame("content_block_stop", {"type": "content_block_stop", "index": 1})
|
||||
+ _MESSAGE_DELTA
|
||||
+ _frame("message_stop", {"type": "message_stop"})
|
||||
)
|
||||
|
||||
|
||||
def _usage(payload: str, provider: str = "anthropic") -> dict[str, int] | None:
|
||||
"""Run the buffered usage parser over a whole stream."""
|
||||
return _Handler()._parse_sse_usage_from_buffer(
|
||||
{"sse_buffer": bytearray(payload.encode())}, provider
|
||||
)
|
||||
|
||||
|
||||
def _usage_chunk(payload: str, provider: str = "anthropic") -> dict[str, int] | None:
|
||||
"""Run the single-chunk usage parser (the third, previously unguarded one)."""
|
||||
return _Handler()._parse_sse_usage(payload.encode(), provider)
|
||||
|
||||
|
||||
def _response(payload: str, provider: str = "anthropic") -> dict[str, Any] | None:
|
||||
return _Handler()._parse_sse_to_response(payload, provider)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Happy path — the real streams must parse, nulls and ping frames included.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_real_text_stream_usage_is_parsed() -> None:
|
||||
usage = _usage(_text_stream())
|
||||
assert usage is not None
|
||||
assert usage["input_tokens"] == 12
|
||||
assert usage["cache_read_input_tokens"] == 0
|
||||
# message_delta's output_tokens wins over message_start's provisional 1.
|
||||
assert usage["output_tokens"] == 4
|
||||
# Sourced from the nested usage.cache_creation object real traffic always sends.
|
||||
assert usage["cache_creation_ephemeral_5m_input_tokens"] == 0
|
||||
assert usage["cache_creation_ephemeral_1h_input_tokens"] == 0
|
||||
|
||||
|
||||
def test_real_text_stream_reconstructs_the_response() -> None:
|
||||
resp = _response(_text_stream())
|
||||
assert resp is not None
|
||||
assert resp["id"] == "msg_011CdQD1YDscyCRWpGCcrwkA"
|
||||
assert resp["model"] == "claude-haiku-4-5-20251001"
|
||||
assert resp["role"] == "assistant"
|
||||
assert resp["stop_reason"] == "end_turn"
|
||||
assert [b["text"] for b in resp["content"]] == ["hi"]
|
||||
assert resp["usage"]["output_tokens"] == 4
|
||||
|
||||
|
||||
def test_real_tool_use_stream_reconstructs_tool_input() -> None:
|
||||
resp = _response(_tool_use_stream())
|
||||
assert resp is not None
|
||||
block = resp["content"][0]
|
||||
assert block["type"] == "tool_use"
|
||||
assert block["name"] == "get_weather"
|
||||
# The five partial_json deltas must accumulate and parse.
|
||||
assert block["input"] == {"city": "Paris"}
|
||||
assert "_partial_json" not in block
|
||||
# Pins current behaviour, not desired behaviour: the copy-through branch
|
||||
# runs only for *non-standard* block types, so `tool_use` keeps just
|
||||
# id/name/input and the real `caller` field is dropped on reconstruction.
|
||||
# Out of scope here (this is fidelity, not a shape crash); see the
|
||||
# `server_tool_use` cases below for the copy-through path itself.
|
||||
assert block["caller"] == {"type": "direct"}
|
||||
|
||||
|
||||
def test_real_thinking_stream_reconstructs_both_blocks() -> None:
|
||||
resp = _response(_thinking_stream())
|
||||
assert resp is not None
|
||||
thinking, text = resp["content"]
|
||||
assert thinking["type"] == "thinking"
|
||||
assert thinking["thinking"] == "17 * 23 = 391"
|
||||
assert thinking["signature"] == "EtQCCpMBCBAYAipA"
|
||||
assert "thinking_buffer" not in thinking
|
||||
assert text["text"] == "391"
|
||||
|
||||
|
||||
def test_legitimate_nulls_are_not_treated_as_malformed() -> None:
|
||||
# stop_reason/stop_sequence/stop_details are null on every real
|
||||
# message_start. The guard must not mistake them for a malformed frame.
|
||||
head = (
|
||||
_frame("message_start", {"type": "message_start", "message": _REAL_MESSAGE})
|
||||
+ _frame(
|
||||
"content_block_start",
|
||||
{
|
||||
"type": "content_block_start",
|
||||
"index": 0,
|
||||
"content_block": {"type": "text", "text": ""},
|
||||
},
|
||||
)
|
||||
+ _frame("content_block_stop", {"type": "content_block_stop", "index": 0})
|
||||
)
|
||||
partial = _response(head)
|
||||
assert partial is not None
|
||||
assert partial["id"] == "msg_011CdQD1YDscyCRWpGCcrwkA"
|
||||
assert partial["stop_reason"] is None
|
||||
assert partial["stop_details"] is None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Shape sweep. Each entry injects a wrong-typed value at one position the
|
||||
# parser reaches into, inside an otherwise-real stream. The frame must be
|
||||
# skipped, never fatal, and the surrounding good frames must still parse.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _at_message(bad: Any) -> str:
|
||||
return _text_stream(message=bad)
|
||||
|
||||
|
||||
def _at_message_usage(bad: Any) -> str:
|
||||
return _text_stream(message={**_REAL_MESSAGE, "usage": bad})
|
||||
|
||||
|
||||
def _at_message_stop_details(bad: Any) -> str:
|
||||
return _text_stream(message={**_REAL_MESSAGE, "stop_details": bad})
|
||||
|
||||
|
||||
def _at_content_block(bad: Any) -> str:
|
||||
return _text_stream(
|
||||
extra=_frame(
|
||||
"content_block_start",
|
||||
{"type": "content_block_start", "index": 5, "content_block": bad},
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _at_block_start_index(bad: Any) -> str:
|
||||
return _text_stream(
|
||||
extra=_frame(
|
||||
"content_block_start",
|
||||
{
|
||||
"type": "content_block_start",
|
||||
"index": bad,
|
||||
"content_block": {"type": "text", "text": ""},
|
||||
},
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _at_block_delta_index(bad: Any) -> str:
|
||||
return _text_stream(
|
||||
extra=_frame(
|
||||
"content_block_delta",
|
||||
{
|
||||
"type": "content_block_delta",
|
||||
"index": bad,
|
||||
"delta": {"type": "text_delta", "text": "X"},
|
||||
},
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _at_block_stop_index(bad: Any) -> str:
|
||||
return _text_stream(
|
||||
extra=_frame("content_block_stop", {"type": "content_block_stop", "index": bad})
|
||||
)
|
||||
|
||||
|
||||
def _at_content_block_delta(bad: Any) -> str:
|
||||
return _text_stream(
|
||||
mid=_frame("content_block_delta", {"type": "content_block_delta", "index": 0, "delta": bad})
|
||||
)
|
||||
|
||||
|
||||
def _at_text_delta_text(bad: Any) -> str:
|
||||
return _text_stream(
|
||||
mid=_frame(
|
||||
"content_block_delta",
|
||||
{
|
||||
"type": "content_block_delta",
|
||||
"index": 0,
|
||||
"delta": {"type": "text_delta", "text": bad},
|
||||
},
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _at_partial_json(bad: Any) -> str:
|
||||
return _text_stream(
|
||||
mid=_frame(
|
||||
"content_block_delta",
|
||||
{
|
||||
"type": "content_block_delta",
|
||||
"index": 0,
|
||||
"delta": {"type": "input_json_delta", "partial_json": bad},
|
||||
},
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _at_thinking_delta(bad: Any) -> str:
|
||||
return _text_stream(
|
||||
mid=_frame(
|
||||
"content_block_delta",
|
||||
{
|
||||
"type": "content_block_delta",
|
||||
"index": 0,
|
||||
"delta": {"type": "thinking_delta", "thinking": bad},
|
||||
},
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _at_message_delta(bad: Any) -> str:
|
||||
return _text_stream(extra=_frame("message_delta", {"type": "message_delta", "delta": bad}))
|
||||
|
||||
|
||||
def _at_message_delta_usage(bad: Any) -> str:
|
||||
return _text_stream(
|
||||
extra=_frame("message_delta", {"type": "message_delta", "delta": {}, "usage": bad})
|
||||
)
|
||||
|
||||
|
||||
def _at_top_level_event(bad: Any) -> str:
|
||||
return _text_stream(extra=f"event: garbage\ndata: {json.dumps(bad)}\n\n")
|
||||
|
||||
|
||||
def _at_citations_copy_through(bad: Any) -> str:
|
||||
"""Upstream seeds a non-list `citations` on a non-standard block, then a
|
||||
citations_delta tries to append to it."""
|
||||
return _text_stream(
|
||||
extra=_frame(
|
||||
"content_block_start",
|
||||
{
|
||||
"type": "content_block_start",
|
||||
"index": 9,
|
||||
"content_block": {"type": "server_tool_use", "citations": bad},
|
||||
},
|
||||
)
|
||||
+ _frame(
|
||||
"content_block_delta",
|
||||
{
|
||||
"type": "content_block_delta",
|
||||
"index": 9,
|
||||
"delta": {"type": "citations_delta", "citation": {"cited_text": "x"}},
|
||||
},
|
||||
)
|
||||
+ _frame("content_block_stop", {"type": "content_block_stop", "index": 9})
|
||||
)
|
||||
|
||||
|
||||
def _at_scratch_key_injection(bad: Any) -> str:
|
||||
"""Upstream tries to seed the parser's own accumulator scratch keys."""
|
||||
return _text_stream(
|
||||
extra=_frame(
|
||||
"content_block_start",
|
||||
{
|
||||
"type": "content_block_start",
|
||||
"index": 8,
|
||||
"content_block": {
|
||||
"type": "server_tool_use",
|
||||
"_partial_json": bad,
|
||||
"thinking_buffer": bad,
|
||||
},
|
||||
},
|
||||
)
|
||||
+ _frame(
|
||||
"content_block_delta",
|
||||
{
|
||||
"type": "content_block_delta",
|
||||
"index": 8,
|
||||
"delta": {"type": "input_json_delta", "partial_json": '{"a":1}'},
|
||||
},
|
||||
)
|
||||
+ _frame(
|
||||
"content_block_delta",
|
||||
{
|
||||
"type": "content_block_delta",
|
||||
"index": 8,
|
||||
"delta": {"type": "thinking_delta", "thinking": "t"},
|
||||
},
|
||||
)
|
||||
+ _frame("content_block_stop", {"type": "content_block_stop", "index": 8})
|
||||
)
|
||||
|
||||
|
||||
_INJECTIONS = [
|
||||
("message", _at_message),
|
||||
("message.usage", _at_message_usage),
|
||||
# No guard needed at this position — the parser copies stop_details
|
||||
# through without inspecting it. Kept so the sweep's contract ("no
|
||||
# position raises") stays exhaustive if that ever changes.
|
||||
("message.stop_details", _at_message_stop_details),
|
||||
("content_block", _at_content_block),
|
||||
("content_block_start.index", _at_block_start_index),
|
||||
("content_block_delta.index", _at_block_delta_index),
|
||||
("content_block_stop.index", _at_block_stop_index),
|
||||
("content_block_delta.delta", _at_content_block_delta),
|
||||
("text_delta.text", _at_text_delta_text),
|
||||
("input_json_delta.partial_json", _at_partial_json),
|
||||
("thinking_delta.thinking", _at_thinking_delta),
|
||||
("message_delta.delta", _at_message_delta),
|
||||
("message_delta.usage", _at_message_delta_usage),
|
||||
("top-level event", _at_top_level_event),
|
||||
("citations copy-through", _at_citations_copy_through),
|
||||
("scratch-key injection", _at_scratch_key_injection),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("position,build", _INJECTIONS, ids=[p for p, _ in _INJECTIONS])
|
||||
@pytest.mark.parametrize("bad", _WRONG_TYPES, ids=repr)
|
||||
def test_response_parser_survives_wrong_shape(position: str, build: Any, bad: Any) -> None:
|
||||
resp = _response(build(bad))
|
||||
assert resp is not None, f"{position}={bad!r} lost the whole response"
|
||||
# The good delta's text still lands. Substring rather than equality: when
|
||||
# `bad` happens to be a valid string it is a legitimate text_delta and
|
||||
# correctly accumulates alongside "hi".
|
||||
texts = [b.get("text") or "" for b in resp["content"] if b.get("type") == "text"]
|
||||
assert any("hi" in t for t in texts), f"{position}={bad!r} destroyed the good content block"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("position,build", _INJECTIONS, ids=[p for p, _ in _INJECTIONS])
|
||||
@pytest.mark.parametrize("bad", _WRONG_TYPES, ids=repr)
|
||||
def test_usage_parser_survives_wrong_shape(position: str, build: Any, bad: Any) -> None:
|
||||
usage = _usage(build(bad))
|
||||
assert usage is not None, f"{position}={bad!r} lost all usage"
|
||||
# message_start's input_tokens survives whatever was injected around it.
|
||||
if position not in {"message", "message.usage"}:
|
||||
assert usage["input_tokens"] == 12, f"{position}={bad!r} corrupted input_tokens"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("bad", _WRONG_TYPES, ids=repr)
|
||||
def test_message_start_wrong_shape_still_yields_later_usage(bad: Any) -> None:
|
||||
# A malformed message_start must not prevent message_delta's usage landing.
|
||||
usage = _usage(_text_stream(message=bad))
|
||||
assert usage is not None
|
||||
assert usage["output_tokens"] == 4
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The two usage parsers, across every provider branch.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_USAGE_POSITIONS = [
|
||||
("anthropic", lambda b: {"type": "message_start", "message": b}),
|
||||
("anthropic", lambda b: {"type": "message_start", "message": {"usage": b}}),
|
||||
("anthropic", lambda b: {"type": "message_delta", "usage": b}),
|
||||
("openai", lambda b: {"usage": b}),
|
||||
("openai", lambda b: {"response": b}),
|
||||
("openai", lambda b: {"usage": {"prompt_tokens_details": b}}),
|
||||
("gemini", lambda b: {"usageMetadata": b}),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("provider,build", _USAGE_POSITIONS)
|
||||
@pytest.mark.parametrize("bad", _WRONG_TYPES, ids=repr)
|
||||
def test_buffered_usage_parser_survives_wrong_shape(provider: str, build: Any, bad: Any) -> None:
|
||||
# Must not raise; returning None (nothing usable) is the correct outcome.
|
||||
_usage(_frame("x", build(bad)), provider)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("provider,build", _USAGE_POSITIONS)
|
||||
@pytest.mark.parametrize("bad", _WRONG_TYPES, ids=repr)
|
||||
def test_chunk_usage_parser_survives_wrong_shape(provider: str, build: Any, bad: Any) -> None:
|
||||
_usage_chunk(_frame("x", build(bad)), provider)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("provider", ["anthropic", "openai", "gemini"])
|
||||
@pytest.mark.parametrize("bad", _WRONG_TYPES, ids=repr)
|
||||
def test_both_usage_parsers_skip_non_object_events(provider: str, bad: Any) -> None:
|
||||
payload = f"data: {json.dumps(bad)}\n\n"
|
||||
assert not _usage(payload, provider)
|
||||
assert not _usage_chunk(payload, provider)
|
||||
|
||||
|
||||
def test_gemini_usage_still_parses(provider: str = "gemini") -> None:
|
||||
# Pins that the gemini guard did not break the branch it hardened.
|
||||
usage = _usage(
|
||||
_frame("x", {"usageMetadata": {"promptTokenCount": 7, "candidatesTokenCount": 3}}), provider
|
||||
)
|
||||
assert usage is not None
|
||||
assert usage["input_tokens"] == 7
|
||||
assert usage["output_tokens"] == 3
|
||||
|
||||
|
||||
def test_openai_usage_still_parses() -> None:
|
||||
usage = _usage(_frame("x", {"usage": {"prompt_tokens": 9, "completion_tokens": 2}}), "openai")
|
||||
assert usage is not None
|
||||
assert usage["input_tokens"] == 9
|
||||
assert usage["output_tokens"] == 2
|
||||
|
||||
|
||||
def test_unparseable_json_is_still_skipped() -> None:
|
||||
# Pre-existing behaviour: the JSONDecodeError guard already covered this.
|
||||
usage = _usage(_text_stream(extra="event: broken\ndata: {not json\n\n"))
|
||||
assert usage is not None
|
||||
assert usage["input_tokens"] == 12
|
||||
Reference in New Issue
Block a user