fix(proxy): record outcome when a passthrough stream ends mid-character (#4429)

## Description

When a streaming passthrough response ends inside a multi-byte UTF-8
character, the `finally` block of `_handle_streaming_passthrough`
appends a `\n\n` terminator to the leftover bytes and the strict SSE
decoder raises `UnicodeDecodeError`. The request outcome is then never
recorded and the ASGI app logs a traceback. This PR drops the unusable
tail, keeps the usage already received, and records the outcome with the
shared stream-failure status (502, `_stream_outcome_status`) because a
stream cut mid-character did not complete.

Fixes #4427

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/handlers/openai.py`: in
`_handle_streaming_passthrough`'s `finally`, the final
`_parse_sse_usage_from_buffer` call is wrapped in
`contextlib.suppress(UnicodeDecodeError)`. A final
complete-but-unterminated event is still parsed as before; only an
undecodable tail is discarded. When that happens the outcome is built
with `status_code=_stream_outcome_status(upstream_response.status_code,
completed_normally)`, the same helper the other streaming paths use in
`streaming.py`, so `emit_request_outcome` books it as failed instead of
as a 200 success. The observed token fields are unchanged.
- `tests/test_proxy_handler_helpers.py`: new test
`test_vertex_stream_passthrough_records_outcome_when_stream_ends_mid_character`
asserts one outcome, `status_code == 502`, 13 input / 5 output tokens
kept, the upstream response closed exactly once, and that
`emit_request_outcome` calls `record_failed` once and `record_request`
never.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

With only the round-2 source change reverted (new assertions kept),
`tests/test_proxy_handler_helpers.py -k mid_character`:

```text
tests/test_proxy_handler_helpers.py:908: in test_vertex_stream_passthrough_records_outcome_when_stream_ends_mid_character
    assert outcomes[0].status_code == 502
E   AssertionError: assert 200 == 502
1 failed, 74 deselected
```

With the fix, plus `tests/test_route_advice.py
tests/test_5xx_accounting_all_providers.py
tests/test_outcome_records_5xx_as_failed.py`:

```text
============================= 152 passed in 6.00s ==============================
```

```text
$ ruff check headroom/proxy/handlers/openai.py tests/test_proxy_handler_helpers.py
All checks passed!
$ ruff format --check headroom/proxy/handlers/openai.py tests/test_proxy_handler_helpers.py
2 files already formatted
```

`mypy` was not run.

## Real Behavior Proof

- Environment: WSL2 Ubuntu on Windows 11, Python 3.12.15, branch based
on upstream/main `e995eb9f`; `headroom/_core.abi3.so` taken from the
official headroom-ai 0.41.0 wheel (untracked, not committed). Real proxy
on 127.0.0.1:18940, local stub upstream on 127.0.0.1:18941
(`VERTEX_TARGET_API_URL`); the stub log shows the proxied request, so no
provider API was contacted.
- Exact command / steps: start the stub (chunked `text/event-stream`:
one complete usage event `data:
{"type":"message_delta","usage":{"output_tokens":3}}`, then `data:
{"t":"\xe2\x82`, then a clean end), start `python -m
headroom.proxy.server --host 127.0.0.1 --port 18940 --no-cache
--no-rate-limit`, read `GET /stats`, `POST
/v1/projects/p/locations/l/publishers/mistral/models/m:streamRawPredict`
with `{"x":1}`, read `GET /stats` again. Same steps on the base and on
this branch.
- Observed result: base: proxy log shows `Exception in ASGI application`
ending in `UnicodeDecodeError: 'utf-8' codec can't decode bytes in
position 12-13: unexpected end of data`, and `requests.total
before/after: 0 0`. This branch: no traceback in the proxy log, and
`requests.total before/after: 0 1` (this was the first version of the
fix, before the status change). Re-run for the failure status on this
head (a76fd811): no traceback, `requests.total before/after: 0 0`, and
`/stats` shows `requests.failed: 1` (`failed_by_provider:
{vertex:mistral: 1}`); the stub log shows the request. The client
received the same 388 bytes in every run.
- Not tested: other ASGI servers and HTTP/2 upstreams; real
Vertex/Gemini upstreams; the `MAX_SSE_BUFFER_SIZE` tail-truncation path
(which can also start a buffer mid-character); a client disconnect (as
opposed to a truncated upstream) as the trigger.

The failure-status re-run reuses the same stub and client with one extra
line printing `/stats` `requests`. Details: scripts and logs are in
`/var/tmp/cb-pub/proof/fix-openai-passthrough-stream-release/` (stub.py,
client.py, run.sh, partial_base.proxy.log, partial_fix.proxy.log,
partial_v2.proxy.log, client_v2.py).

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only the failure case: an undecodable
stream tail no longer raises
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Documentation: N/A, no user-facing behavior beyond the fix.
- A second suspected problem in the same function (no background release
of the upstream stream on client disconnect) did not reproduce against
the real proxy (780 client resets at several timings; the stub saw the
proxy close its upstream connection every time), so it is not changed
here.

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
reproduction was run on my machine on a clone of the repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4429"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
HetCreep
2026-10-24 08:38:14 -07:00
committed by GitHub
co-authored by Claude Opus 5.5 JerrettDavis
parent 4bf1bf55d1
commit 75cfbba91d
2 changed files with 88 additions and 1 deletions
+15 -1
View File
@@ -113,6 +113,7 @@ from headroom.proxy.buffered_ccr_response import (
from headroom.proxy.compression_decision import CompressionDecision
from headroom.proxy.cost import header_safe_transforms
from headroom.proxy.handlers._debug_dump import _debug_dump_mode, _redact_debug_value
from headroom.proxy.handlers.streaming import _stream_outcome_status
from headroom.proxy.image_isolation import run_image_compression_isolated
from headroom.proxy.outcome import RequestOutcome
from headroom.proxy.output_shaper import shaper_enabled_for, steering_allowed_for
@@ -11759,10 +11760,20 @@ class OpenAIHandlerMixin:
)
yield chunk
finally:
completed_normally = True
buf = stream_state["sse_buffer"]
if len(buf) > 0:
buf.extend(b"\n\n")
_absorb_usage(self._parse_sse_usage_from_buffer(stream_state, stream_provider))
# A truncated stream can end inside a multi-byte character;
# the strict decoder then raises and would skip the outcome
# below. Keep the usage already parsed, drop the unusable
# tail, and report the stream as failed.
try:
_absorb_usage(
self._parse_sse_usage_from_buffer(stream_state, stream_provider)
)
except UnicodeDecodeError:
completed_normally = False
output_tokens = stream_state["output_tokens"] or 0
cache_read_tokens = stream_state["cache_read_input_tokens"] or 0
@@ -11803,6 +11814,9 @@ class OpenAIHandlerMixin:
total_latency_ms=(time.time() - start_time) * 1000,
ttfb_ms=stream_state["ttfb_ms"] or 0,
tags=tags,
status_code=_stream_outcome_status(
upstream_response.status_code, completed_normally
),
client=client,
)
)
+73
View File
@@ -22,6 +22,7 @@ from headroom.proxy.helpers import (
_headroom_bypass_enabled,
relocate_system_messages_to_top_level,
)
from headroom.proxy.outcome import emit_request_outcome
from headroom.proxy.server import HeadroomProxy
@@ -849,6 +850,78 @@ def test_vertex_stream_passthrough_preserves_chunks_and_records_usage() -> None:
assert outcome.cache_read_tokens == 2
def test_vertex_stream_passthrough_records_outcome_when_stream_ends_mid_character() -> None:
closed: list[bool] = []
class _CountingChunks(_AsyncChunks):
async def aclose(self) -> None:
closed.append(True)
class _TruncatedClient(_VertexStreamClient):
async def send(self, request, stream=False): # noqa: ANN001, ANN201
return httpx.Response(
200,
request=request,
headers={"content-type": "text/event-stream"},
stream=_CountingChunks(
[
b'data: {"usageMetadata":{"promptTokenCount":13,'
b'"candidatesTokenCount":5}}\n\n',
b'data: {"t":"\xe2\x82', # first 2 bytes of a 3-byte character
]
),
)
handler = object.__new__(HeadroomProxy)
handler.http_client = _TruncatedClient()
outcomes = []
async def next_request_id(): # noqa: ANN202
return "req_vertex_truncated"
async def record(outcome): # noqa: ANN001, ANN202
outcomes.append(outcome)
handler._next_request_id = next_request_id
handler._record_request_outcome = record
response = asyncio.run(
handler.handle_passthrough(
_VertexStreamPassthroughRequest(),
"https://vertex.test",
"streamGenerateContent",
"vertex:google",
)
)
async def collect(): # noqa: ANN202
return [chunk async for chunk in response.body_iterator]
assert len(asyncio.run(collect())) == 2
assert len(outcomes) == 1
assert closed == [True]
# Usage already received is kept, but a stream cut mid-character did not
# complete: it takes the same failure status as other post-start stream
# failures, so the outcome funnel books it as failed, not as a success.
assert outcomes[0].optimized_tokens == 13
assert outcomes[0].output_tokens == 5
assert outcomes[0].status_code == 502
failed: list[str] = []
requested: list[dict] = []
class _Metrics:
async def record_failed(self, provider): # noqa: ANN001, ANN202
failed.append(provider)
async def record_request(self, **kwargs): # noqa: ANN003, ANN202
requested.append(kwargs)
asyncio.run(emit_request_outcome(SimpleNamespace(metrics=_Metrics()), outcomes[0]))
assert failed == [outcomes[0].provider]
assert requested == []
def test_stream_finalizer_records_vertex_provider_for_dashboard() -> None:
handler = object.__new__(HeadroomProxy)
handler.config = SimpleNamespace(log_full_messages=False)