mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): discard partial SSE tails during finalization (#3411)
## Description Closes #3405. When an SSE stream is interrupted in the middle of an event or UTF-8 character, the streaming finalizer appended `\n\n` to the residual byte buffer. That synthetic terminator made the incomplete event look complete and sent it through the strict UTF-8 decoder, causing a secondary `UnicodeDecodeError` during cleanup. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (breaking change that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Stop synthesizing an SSE event terminator during Anthropic streaming finalization; discard the unterminated residual tail instead. - Apply the same rule to the OpenAI-via-backend streaming cleanup path. - Keep complete events parsed during normal chunk processing, while preventing partial SSE/UTF-8 data from being decoded as a complete event. - Replace the old truncated-buffer recovery test with a regression covering a truncated UTF-8 tail and asserting finalization completes without raising. ## Testing - [x] Focused unit tests pass - [x] Linting passes - [x] Formatting passes - [x] New regression coverage added - [ ] Full test suite - [ ] Live upstream streaming run ### Test Output ```text python -m pytest tests/test_proxy_streaming_request_logger.py -q 9 passed in 71.37s python -m pytest tests/test_sse_byte_buffer_policy.py tests/test_sse_utf8_split.py -q 8 passed in 10.78s python -m ruff check headroom/proxy/handlers/streaming.py tests/test_proxy_streaming_request_logger.py All checks passed! python -m ruff format --check headroom/proxy/handlers/streaming.py tests/test_proxy_streaming_request_logger.py 2 files already formatted ``` The focused proxy test used the published `headroom-ai==0.37.0` wheel only for its compatible Rust extension; the source changes and tests ran from this checkout. ## Real Behavior Proof - Environment: Windows 10, Python 3.11.9, pytest 9.1.1. - Exact command / steps: ran the new regression with the production finalizer reverted; it failed with `UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 ... unexpected end of data`. Restored the fix and ran the regression plus the existing SSE parser tests. - Observed result: the old implementation fails when finalization appends a terminator to an unterminated event ending in a partial UTF-8 byte; the fixed implementation clears the incomplete tail without raising, and the focused proxy suite passes 9 tests. - Not tested: a live upstream provider disconnect; the regression exercises the exact residual-buffer/finalizer path with a partial UTF-8 sequence. ## Runtime Rollout Safety - Rollout-managed feature(s): None; this is an internal parser/finalization bug fix with no feature flag or staged rollout. - Minimum rollout channel: Standard release containing this bug fix; no separate rollout channel is required. - Stable/default behavior changed: No; complete SSE events remain parsed during normal streaming, while only unterminated residual tails are discarded during finalization. - Kill switch / disable path: Not applicable; revert this PR or the release containing it. - Unsafe override required: No. - Qualification impact: None; there are no API or configuration changes, and focused regression coverage was added. - Rollback path: Revert this change; the prior behavior may reintroduce a secondary UnicodeDecodeError on interrupted partial UTF-8 SSE tails. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] Focused unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Complete SSE events continue to be parsed during streaming. Only an unterminated residual tail is discarded at finalization, so interrupted cleanup cannot turn incomplete wire data into a parser exception. --------- Signed-off-by: UniversePeak <UniversePeak@users.noreply.github.com> Co-authored-by: UniversePeak <UniversePeak@users.noreply.github.com>
This commit is contained in:
@@ -947,34 +947,15 @@ class StreamingMixin:
|
||||
|
||||
# Per-chunk SSE parsing only flushes events terminated by ``\n\n``.
|
||||
# When upstream truncates mid-event (client disconnect, network
|
||||
# drop, connection reset), the message_start (cache_read /
|
||||
# cache_creation) or message_delta (output_tokens) usage events
|
||||
# can sit in the residual buffer and never be parsed — surfacing
|
||||
# as cache_read=cache_write=0 in PERF logs and poisoning the
|
||||
# downstream freeze heuristic for the next request. Append the
|
||||
# terminator so the buffer parser drains whatever's there. The
|
||||
# per-event try/except in the parser swallows incomplete JSON,
|
||||
# so this is safe even when the truncation cut mid-payload.
|
||||
# drop, connection reset), the residual bytes are not a complete
|
||||
# SSE event. Do not append a synthetic terminator here: doing so
|
||||
# turns a partial UTF-8 sequence or partial event into input for
|
||||
# the strict decoder and can raise while finalizing the stream.
|
||||
# Complete usage events have already been parsed in the read loop;
|
||||
# discard the incomplete tail instead of treating it as authoritative.
|
||||
sse_buffer = stream_state.get("sse_buffer")
|
||||
if isinstance(sse_buffer, bytearray) and len(sse_buffer) > 0:
|
||||
sse_buffer.extend(b"\n\n")
|
||||
late_usage = self._parse_sse_usage_from_buffer(stream_state, provider) or {}
|
||||
for key in (
|
||||
"input_tokens",
|
||||
"output_tokens",
|
||||
"cache_read_input_tokens",
|
||||
"cache_creation_input_tokens",
|
||||
"cache_creation_ephemeral_5m_input_tokens",
|
||||
"cache_creation_ephemeral_1h_input_tokens",
|
||||
):
|
||||
if key not in late_usage:
|
||||
continue
|
||||
current = stream_state.get(key)
|
||||
# Only fill in unset (None) or default-zero slots so a
|
||||
# real cache_read=0 from earlier in the stream isn't
|
||||
# clobbered by a later partial event.
|
||||
if current is None or current == 0:
|
||||
stream_state[key] = late_usage[key]
|
||||
sse_buffer.clear()
|
||||
|
||||
output_tokens = stream_state["output_tokens"]
|
||||
output_tokens_source = "provider"
|
||||
@@ -2215,14 +2196,13 @@ class StreamingMixin:
|
||||
yield f"data: {json.dumps(error_data)}\n\n".encode()
|
||||
yield b"data: [DONE]\n\n"
|
||||
finally:
|
||||
# Late-flush: if upstream truncated the stream mid-event,
|
||||
# the buffer parser hasn't seen the closing ``\n\n`` yet.
|
||||
# Mirror _finalize_stream_response: append the terminator
|
||||
# and drain anything still parseable.
|
||||
# A residual buffer without an SSE terminator is an
|
||||
# incomplete event. Complete usage frames were parsed in
|
||||
# the read loop; never synthesize ``\n\n`` here, because
|
||||
# that can make partial UTF-8 look like a complete event.
|
||||
buf = stream_state["sse_buffer"]
|
||||
if len(buf) > 0:
|
||||
buf.extend(b"\n\n")
|
||||
_absorb(self._parse_sse_usage_from_buffer(stream_state, "openai"))
|
||||
buf.clear()
|
||||
|
||||
# Mirror the non-streaming sibling (``_extract_responses_usage``
|
||||
# in handlers/openai.py): only infer cache metrics when
|
||||
|
||||
@@ -274,14 +274,8 @@ async def test_finalize_openai_responses_stream_uses_provider_usage_for_dashboar
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_finalize_stream_response_recovers_usage_from_truncated_buffer() -> None:
|
||||
"""When upstream truncates mid-event (no trailing \\n\\n), the per-chunk
|
||||
parser leaves the message_start usage event sitting in sse_buffer and
|
||||
PERF logs cache_read=cache_write=0 — which then poisons the freeze
|
||||
heuristic on the next request. The finalizer must flush the residual
|
||||
buffer so the real cache_read / cache_creation tokens still land in
|
||||
the log even on aborted streams.
|
||||
"""
|
||||
async def test_finalize_stream_response_discards_truncated_utf8_tail() -> None:
|
||||
"""An interrupted SSE event must not be made complete by finalization."""
|
||||
proxy = _build_proxy_with_real_logger(log_full_messages=False)
|
||||
|
||||
partial_message_start = (
|
||||
@@ -291,6 +285,7 @@ async def test_finalize_stream_response_recovers_usage_from_truncated_buffer() -
|
||||
b'"content":[],"stop_reason":null,"usage":{'
|
||||
b'"input_tokens":1234,"cache_read_input_tokens":50000,'
|
||||
b'"cache_creation_input_tokens":2500,"output_tokens":1}}}'
|
||||
b"\xe5"
|
||||
)
|
||||
|
||||
state = {
|
||||
@@ -319,9 +314,10 @@ async def test_finalize_stream_response_recovers_usage_from_truncated_buffer() -
|
||||
start_time=0.0,
|
||||
)
|
||||
|
||||
assert state["input_tokens"] == 1234
|
||||
assert state["cache_read_input_tokens"] == 50000
|
||||
assert state["cache_creation_input_tokens"] == 2500
|
||||
assert state["input_tokens"] is None
|
||||
assert state["cache_read_input_tokens"] == 0
|
||||
assert state["cache_creation_input_tokens"] == 0
|
||||
assert state["sse_buffer"] == bytearray()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
Reference in New Issue
Block a user