mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): record outcome when a passthrough stream ends mid-character (#4429)
## Description
When a streaming passthrough response ends inside a multi-byte UTF-8
character, the `finally` block of `_handle_streaming_passthrough`
appends a `\n\n` terminator to the leftover bytes and the strict SSE
decoder raises `UnicodeDecodeError`. The request outcome is then never
recorded and the ASGI app logs a traceback. This PR drops the unusable
tail, keeps the usage already received, and records the outcome with the
shared stream-failure status (502, `_stream_outcome_status`) because a
stream cut mid-character did not complete.
Fixes #4427
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/openai.py`: in
`_handle_streaming_passthrough`'s `finally`, the final
`_parse_sse_usage_from_buffer` call is wrapped in
`contextlib.suppress(UnicodeDecodeError)`. A final
complete-but-unterminated event is still parsed as before; only an
undecodable tail is discarded. When that happens the outcome is built
with `status_code=_stream_outcome_status(upstream_response.status_code,
completed_normally)`, the same helper the other streaming paths use in
`streaming.py`, so `emit_request_outcome` books it as failed instead of
as a 200 success. The observed token fields are unchanged.
- `tests/test_proxy_handler_helpers.py`: new test
`test_vertex_stream_passthrough_records_outcome_when_stream_ends_mid_character`
asserts one outcome, `status_code == 502`, 13 input / 5 output tokens
kept, the upstream response closed exactly once, and that
`emit_request_outcome` calls `record_failed` once and `record_request`
never.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
With only the round-2 source change reverted (new assertions kept),
`tests/test_proxy_handler_helpers.py -k mid_character`:
```text
tests/test_proxy_handler_helpers.py:908: in test_vertex_stream_passthrough_records_outcome_when_stream_ends_mid_character
assert outcomes[0].status_code == 502
E AssertionError: assert 200 == 502
1 failed, 74 deselected
```
With the fix, plus `tests/test_route_advice.py
tests/test_5xx_accounting_all_providers.py
tests/test_outcome_records_5xx_as_failed.py`:
```text
============================= 152 passed in 6.00s ==============================
```
```text
$ ruff check headroom/proxy/handlers/openai.py tests/test_proxy_handler_helpers.py
All checks passed!
$ ruff format --check headroom/proxy/handlers/openai.py tests/test_proxy_handler_helpers.py
2 files already formatted
```
`mypy` was not run.
## Real Behavior Proof
- Environment: WSL2 Ubuntu on Windows 11, Python 3.12.15, branch based
on upstream/main `e995eb9f`; `headroom/_core.abi3.so` taken from the
official headroom-ai 0.41.0 wheel (untracked, not committed). Real proxy
on 127.0.0.1:18940, local stub upstream on 127.0.0.1:18941
(`VERTEX_TARGET_API_URL`); the stub log shows the proxied request, so no
provider API was contacted.
- Exact command / steps: start the stub (chunked `text/event-stream`:
one complete usage event `data:
{"type":"message_delta","usage":{"output_tokens":3}}`, then `data:
{"t":"\xe2\x82`, then a clean end), start `python -m
headroom.proxy.server --host 127.0.0.1 --port 18940 --no-cache
--no-rate-limit`, read `GET /stats`, `POST
/v1/projects/p/locations/l/publishers/mistral/models/m:streamRawPredict`
with `{"x":1}`, read `GET /stats` again. Same steps on the base and on
this branch.
- Observed result: base: proxy log shows `Exception in ASGI application`
ending in `UnicodeDecodeError: 'utf-8' codec can't decode bytes in
position 12-13: unexpected end of data`, and `requests.total
before/after: 0 0`. This branch: no traceback in the proxy log, and
`requests.total before/after: 0 1` (this was the first version of the
fix, before the status change). Re-run for the failure status on this
head (a76fd811): no traceback, `requests.total before/after: 0 0`, and
`/stats` shows `requests.failed: 1` (`failed_by_provider:
{vertex:mistral: 1}`); the stub log shows the request. The client
received the same 388 bytes in every run.
- Not tested: other ASGI servers and HTTP/2 upstreams; real
Vertex/Gemini upstreams; the `MAX_SSE_BUFFER_SIZE` tail-truncation path
(which can also start a buffer mid-character); a client disconnect (as
opposed to a truncated upstream) as the trigger.
The failure-status re-run reuses the same stub and client with one extra
line printing `/stats` `requests`. Details: scripts and logs are in
`/var/tmp/cb-pub/proof/fix-openai-passthrough-stream-release/` (stub.py,
client.py, run.sh, partial_base.proxy.log, partial_fix.proxy.log,
partial_v2.proxy.log, client_v2.py).
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only the failure case: an undecodable
stream tail no longer raises
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the commit
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A
## Additional Notes
- Documentation: N/A, no user-facing behavior beyond the fix.
- A second suspected problem in the same function (no background release
of the upstream stream on client disconnect) did not reproduce against
the real proxy (780 client resets at several timings; the stub saw the
proxy close its upstream connection every time), so it is not changed
here.
Produced with AI assistance (Claude Code) operated by me (HetCreep); the
reproduction was run on my machine on a clone of the repository.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4429"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JerrettDavis
parent
4bf1bf55d1
commit
75cfbba91d
@@ -113,6 +113,7 @@ from headroom.proxy.buffered_ccr_response import (
|
||||
from headroom.proxy.compression_decision import CompressionDecision
|
||||
from headroom.proxy.cost import header_safe_transforms
|
||||
from headroom.proxy.handlers._debug_dump import _debug_dump_mode, _redact_debug_value
|
||||
from headroom.proxy.handlers.streaming import _stream_outcome_status
|
||||
from headroom.proxy.image_isolation import run_image_compression_isolated
|
||||
from headroom.proxy.outcome import RequestOutcome
|
||||
from headroom.proxy.output_shaper import shaper_enabled_for, steering_allowed_for
|
||||
@@ -11759,10 +11760,20 @@ class OpenAIHandlerMixin:
|
||||
)
|
||||
yield chunk
|
||||
finally:
|
||||
completed_normally = True
|
||||
buf = stream_state["sse_buffer"]
|
||||
if len(buf) > 0:
|
||||
buf.extend(b"\n\n")
|
||||
_absorb_usage(self._parse_sse_usage_from_buffer(stream_state, stream_provider))
|
||||
# A truncated stream can end inside a multi-byte character;
|
||||
# the strict decoder then raises and would skip the outcome
|
||||
# below. Keep the usage already parsed, drop the unusable
|
||||
# tail, and report the stream as failed.
|
||||
try:
|
||||
_absorb_usage(
|
||||
self._parse_sse_usage_from_buffer(stream_state, stream_provider)
|
||||
)
|
||||
except UnicodeDecodeError:
|
||||
completed_normally = False
|
||||
|
||||
output_tokens = stream_state["output_tokens"] or 0
|
||||
cache_read_tokens = stream_state["cache_read_input_tokens"] or 0
|
||||
@@ -11803,6 +11814,9 @@ class OpenAIHandlerMixin:
|
||||
total_latency_ms=(time.time() - start_time) * 1000,
|
||||
ttfb_ms=stream_state["ttfb_ms"] or 0,
|
||||
tags=tags,
|
||||
status_code=_stream_outcome_status(
|
||||
upstream_response.status_code, completed_normally
|
||||
),
|
||||
client=client,
|
||||
)
|
||||
)
|
||||
|
||||
@@ -22,6 +22,7 @@ from headroom.proxy.helpers import (
|
||||
_headroom_bypass_enabled,
|
||||
relocate_system_messages_to_top_level,
|
||||
)
|
||||
from headroom.proxy.outcome import emit_request_outcome
|
||||
from headroom.proxy.server import HeadroomProxy
|
||||
|
||||
|
||||
@@ -849,6 +850,78 @@ def test_vertex_stream_passthrough_preserves_chunks_and_records_usage() -> None:
|
||||
assert outcome.cache_read_tokens == 2
|
||||
|
||||
|
||||
def test_vertex_stream_passthrough_records_outcome_when_stream_ends_mid_character() -> None:
|
||||
closed: list[bool] = []
|
||||
|
||||
class _CountingChunks(_AsyncChunks):
|
||||
async def aclose(self) -> None:
|
||||
closed.append(True)
|
||||
|
||||
class _TruncatedClient(_VertexStreamClient):
|
||||
async def send(self, request, stream=False): # noqa: ANN001, ANN201
|
||||
return httpx.Response(
|
||||
200,
|
||||
request=request,
|
||||
headers={"content-type": "text/event-stream"},
|
||||
stream=_CountingChunks(
|
||||
[
|
||||
b'data: {"usageMetadata":{"promptTokenCount":13,'
|
||||
b'"candidatesTokenCount":5}}\n\n',
|
||||
b'data: {"t":"\xe2\x82', # first 2 bytes of a 3-byte character
|
||||
]
|
||||
),
|
||||
)
|
||||
|
||||
handler = object.__new__(HeadroomProxy)
|
||||
handler.http_client = _TruncatedClient()
|
||||
outcomes = []
|
||||
|
||||
async def next_request_id(): # noqa: ANN202
|
||||
return "req_vertex_truncated"
|
||||
|
||||
async def record(outcome): # noqa: ANN001, ANN202
|
||||
outcomes.append(outcome)
|
||||
|
||||
handler._next_request_id = next_request_id
|
||||
handler._record_request_outcome = record
|
||||
|
||||
response = asyncio.run(
|
||||
handler.handle_passthrough(
|
||||
_VertexStreamPassthroughRequest(),
|
||||
"https://vertex.test",
|
||||
"streamGenerateContent",
|
||||
"vertex:google",
|
||||
)
|
||||
)
|
||||
|
||||
async def collect(): # noqa: ANN202
|
||||
return [chunk async for chunk in response.body_iterator]
|
||||
|
||||
assert len(asyncio.run(collect())) == 2
|
||||
assert len(outcomes) == 1
|
||||
assert closed == [True]
|
||||
# Usage already received is kept, but a stream cut mid-character did not
|
||||
# complete: it takes the same failure status as other post-start stream
|
||||
# failures, so the outcome funnel books it as failed, not as a success.
|
||||
assert outcomes[0].optimized_tokens == 13
|
||||
assert outcomes[0].output_tokens == 5
|
||||
assert outcomes[0].status_code == 502
|
||||
|
||||
failed: list[str] = []
|
||||
requested: list[dict] = []
|
||||
|
||||
class _Metrics:
|
||||
async def record_failed(self, provider): # noqa: ANN001, ANN202
|
||||
failed.append(provider)
|
||||
|
||||
async def record_request(self, **kwargs): # noqa: ANN003, ANN202
|
||||
requested.append(kwargs)
|
||||
|
||||
asyncio.run(emit_request_outcome(SimpleNamespace(metrics=_Metrics()), outcomes[0]))
|
||||
assert failed == [outcomes[0].provider]
|
||||
assert requested == []
|
||||
|
||||
|
||||
def test_stream_finalizer_records_vertex_provider_for_dashboard() -> None:
|
||||
handler = object.__new__(HeadroomProxy)
|
||||
handler.config = SimpleNamespace(log_full_messages=False)
|
||||
|
||||
Reference in New Issue
Block a user