fix(proxy): discard partial SSE tails during finalization (#3411)

## Description

Closes #3405.

When an SSE stream is interrupted in the middle of an event or UTF-8
character, the streaming finalizer appended `\n\n` to the residual byte
buffer. That synthetic terminator made the incomplete event look
complete and sent it through the strict UTF-8 decoder, causing a
secondary `UnicodeDecodeError` during cleanup.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (breaking change that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Stop synthesizing an SSE event terminator during Anthropic streaming
finalization; discard the unterminated residual tail instead.
- Apply the same rule to the OpenAI-via-backend streaming cleanup path.
- Keep complete events parsed during normal chunk processing, while
preventing partial SSE/UTF-8 data from being decoded as a complete
event.
- Replace the old truncated-buffer recovery test with a regression
covering a truncated UTF-8 tail and asserting finalization completes
without raising.

## Testing

- [x] Focused unit tests pass
- [x] Linting passes
- [x] Formatting passes
- [x] New regression coverage added
- [ ] Full test suite
- [ ] Live upstream streaming run

### Test Output

```text
python -m pytest tests/test_proxy_streaming_request_logger.py -q
9 passed in 71.37s

python -m pytest tests/test_sse_byte_buffer_policy.py tests/test_sse_utf8_split.py -q
8 passed in 10.78s

python -m ruff check headroom/proxy/handlers/streaming.py tests/test_proxy_streaming_request_logger.py
All checks passed!

python -m ruff format --check headroom/proxy/handlers/streaming.py tests/test_proxy_streaming_request_logger.py
2 files already formatted
```

The focused proxy test used the published `headroom-ai==0.37.0` wheel
only for its compatible Rust extension; the source changes and tests ran
from this checkout.

## Real Behavior Proof

- Environment: Windows 10, Python 3.11.9, pytest 9.1.1.
- Exact command / steps: ran the new regression with the production
finalizer reverted; it failed with `UnicodeDecodeError: 'utf-8' codec
can't decode byte 0xe5 ... unexpected end of data`. Restored the fix and
ran the regression plus the existing SSE parser tests.
- Observed result: the old implementation fails when finalization
appends a terminator to an unterminated event ending in a partial UTF-8
byte; the fixed implementation clears the incomplete tail without
raising, and the focused proxy suite passes 9 tests.
- Not tested: a live upstream provider disconnect; the regression
exercises the exact residual-buffer/finalizer path with a partial UTF-8
sequence.

## Runtime Rollout Safety

- Rollout-managed feature(s): None; this is an internal
parser/finalization bug fix with no feature flag or staged rollout.
- Minimum rollout channel: Standard release containing this bug fix; no
separate rollout channel is required.
- Stable/default behavior changed: No; complete SSE events remain parsed
during normal streaming, while only unterminated residual tails are
discarded during finalization.
- Kill switch / disable path: Not applicable; revert this PR or the
release containing it.
- Unsafe override required: No.
- Qualification impact: None; there are no API or configuration changes,
and focused regression coverage was added.
- Rollback path: Revert this change; the prior behavior may reintroduce
a secondary UnicodeDecodeError on interrupted partial UTF-8 SSE tails.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] Focused unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Complete SSE events continue to be parsed during streaming. Only an
unterminated residual tail is discarded at finalization, so interrupted
cleanup cannot turn incomplete wire data into a parser exception.

---------

Signed-off-by: UniversePeak <UniversePeak@users.noreply.github.com>
Co-authored-by: UniversePeak <UniversePeak@users.noreply.github.com>
This commit is contained in:
宇宙的山峰
2026-09-23 21:16:01 -05:00
committed by GitHub
co-authored by UniversePeak
parent 9a1110997a
commit 6c0e817bcc
2 changed files with 19 additions and 43 deletions
+12 -32
View File
@@ -947,34 +947,15 @@ class StreamingMixin:
# Per-chunk SSE parsing only flushes events terminated by ``\n\n``.
# When upstream truncates mid-event (client disconnect, network
# drop, connection reset), the message_start (cache_read /
# cache_creation) or message_delta (output_tokens) usage events
# can sit in the residual buffer and never be parsed — surfacing
# as cache_read=cache_write=0 in PERF logs and poisoning the
# downstream freeze heuristic for the next request. Append the
# terminator so the buffer parser drains whatever's there. The
# per-event try/except in the parser swallows incomplete JSON,
# so this is safe even when the truncation cut mid-payload.
# drop, connection reset), the residual bytes are not a complete
# SSE event. Do not append a synthetic terminator here: doing so
# turns a partial UTF-8 sequence or partial event into input for
# the strict decoder and can raise while finalizing the stream.
# Complete usage events have already been parsed in the read loop;
# discard the incomplete tail instead of treating it as authoritative.
sse_buffer = stream_state.get("sse_buffer")
if isinstance(sse_buffer, bytearray) and len(sse_buffer) > 0:
sse_buffer.extend(b"\n\n")
late_usage = self._parse_sse_usage_from_buffer(stream_state, provider) or {}
for key in (
"input_tokens",
"output_tokens",
"cache_read_input_tokens",
"cache_creation_input_tokens",
"cache_creation_ephemeral_5m_input_tokens",
"cache_creation_ephemeral_1h_input_tokens",
):
if key not in late_usage:
continue
current = stream_state.get(key)
# Only fill in unset (None) or default-zero slots so a
# real cache_read=0 from earlier in the stream isn't
# clobbered by a later partial event.
if current is None or current == 0:
stream_state[key] = late_usage[key]
sse_buffer.clear()
output_tokens = stream_state["output_tokens"]
output_tokens_source = "provider"
@@ -2215,14 +2196,13 @@ class StreamingMixin:
yield f"data: {json.dumps(error_data)}\n\n".encode()
yield b"data: [DONE]\n\n"
finally:
# Late-flush: if upstream truncated the stream mid-event,
# the buffer parser hasn't seen the closing ``\n\n`` yet.
# Mirror _finalize_stream_response: append the terminator
# and drain anything still parseable.
# A residual buffer without an SSE terminator is an
# incomplete event. Complete usage frames were parsed in
# the read loop; never synthesize ``\n\n`` here, because
# that can make partial UTF-8 look like a complete event.
buf = stream_state["sse_buffer"]
if len(buf) > 0:
buf.extend(b"\n\n")
_absorb(self._parse_sse_usage_from_buffer(stream_state, "openai"))
buf.clear()
# Mirror the non-streaming sibling (``_extract_responses_usage``
# in handlers/openai.py): only infer cache metrics when
+7 -11
View File
@@ -274,14 +274,8 @@ async def test_finalize_openai_responses_stream_uses_provider_usage_for_dashboar
@pytest.mark.asyncio
async def test_finalize_stream_response_recovers_usage_from_truncated_buffer() -> None:
"""When upstream truncates mid-event (no trailing \\n\\n), the per-chunk
parser leaves the message_start usage event sitting in sse_buffer and
PERF logs cache_read=cache_write=0 — which then poisons the freeze
heuristic on the next request. The finalizer must flush the residual
buffer so the real cache_read / cache_creation tokens still land in
the log even on aborted streams.
"""
async def test_finalize_stream_response_discards_truncated_utf8_tail() -> None:
"""An interrupted SSE event must not be made complete by finalization."""
proxy = _build_proxy_with_real_logger(log_full_messages=False)
partial_message_start = (
@@ -291,6 +285,7 @@ async def test_finalize_stream_response_recovers_usage_from_truncated_buffer() -
b'"content":[],"stop_reason":null,"usage":{'
b'"input_tokens":1234,"cache_read_input_tokens":50000,'
b'"cache_creation_input_tokens":2500,"output_tokens":1}}}'
b"\xe5"
)
state = {
@@ -319,9 +314,10 @@ async def test_finalize_stream_response_recovers_usage_from_truncated_buffer() -
start_time=0.0,
)
assert state["input_tokens"] == 1234
assert state["cache_read_input_tokens"] == 50000
assert state["cache_creation_input_tokens"] == 2500
assert state["input_tokens"] is None
assert state["cache_read_input_tokens"] == 0
assert state["cache_creation_input_tokens"] == 0
assert state["sse_buffer"] == bytearray()
@pytest.mark.asyncio