mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): count output tokens from CRLF-framed SSE streams too (#4298)
## Description When an upstream streams a response with no usage chunk, `estimate_output_tokens()` (`headroom/proxy/stream_output_tokens.py`) recovers the output-token count from the stream itself. It uses the request's output ceiling when the turn stopped on it, otherwise the generated text, and only as a last resort `total_bytes // 40`. All of that goes through `_iter_sse_payloads()`, which splits events on `"\n\n"` and lines on `"\n"` only. SSE allows CRLF and bare-CR line endings, and the rest of the proxy handles them (#649). A CRLF-framed stream has no `"\n\n"` in it, though: it becomes one block, every `data:` line keeps a trailing `\r`, the concatenated payloads aren't valid JSON, and nothing is recovered. The stop reason isn't found either, so even the exact-ceiling rung is skipped and the estimate falls through to the raw-bytes heuristic. For the stream in the proof below that's 281 instead of 130 tokens, a 2.2× over-count on the number that feeds the cost model and the output-savings estimate. Closes: N/A (no matching issue found) ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/stream_output_tokens.py`: `_iter_sse_payloads()` normalises `\r\n` and bare `\r` to `\n` before splitting. JSON payloads escape their own carriage returns (`\r` inside a string is the two characters `\` `r`), so this can't alter the data. LF streams are processed exactly as before. The function is shared by `extract_stream_text()` and the ceiling detector, so both rungs are fixed. - `tests/test_stream_output_tokens.py`: `test_crlf_and_cr_framed_streams_are_parsed[crlf, cr]` takes an LF stream that ends on `finish_reason: "length"`, re-frames it with CRLF and with bare CR, and checks that the extracted text matches the LF stream's, that the ceiling rung returns `(64, "exact_ceiling")`, and that the text rung returns `estimated_text`. ## Testing - [x] Unit tests pass (`pytest`): `tests/test_stream_output_tokens.py`, `tests/test_streaming_usage_parser.py`, `tests/test_streaming_sse_malformed_events.py`, `tests/test_sse_byte_buffer_policy.py`, `tests/test_proxy_streaming_resilience.py`, `tests/test_openai_streaming_backend.py`, `tests/test_sse_utf8_split.py`, not the full suite - [x] Linting passes (`ruff check .`): on the changed files - [ ] Type checking passes (`mypy headroom`): not run - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output New test on `main` (fix reverted, test kept): ```text $ python -m pytest tests/test_stream_output_tokens.py -q E AssertionError: assert '' == 'Hello world,...Hello world, ' E AssertionError: assert '' == 'Hello world,...Hello world, ' FAILED tests/test_stream_output_tokens.py::test_crlf_and_cr_framed_streams_are_parsed[crlf] FAILED tests/test_stream_output_tokens.py::test_crlf_and_cr_framed_streams_are_parsed[cr] ========================= 2 failed, 27 passed in 1.31s ========================= ``` With the fix: ```text $ python -m pytest tests/test_stream_output_tokens.py tests/test_streaming_usage_parser.py tests/test_streaming_sse_malformed_events.py \ tests/test_sse_byte_buffer_policy.py tests/test_proxy_streaming_resilience.py tests/test_openai_streaming_backend.py tests/test_sse_utf8_split.py -q ======================== 425 passed, 3 skipped in 8.34s ======================== $ ruff check headroom/proxy/stream_output_tokens.py tests/test_stream_output_tokens.py All checks passed! $ ruff format --check headroom/proxy/stream_output_tokens.py tests/test_stream_output_tokens.py 2 files already formatted ``` ## Real Behavior Proof - Environment: macOS 27.0.1 (arm64), Python 3.12.11, source tree on this branch (native `headroom._core` from the 0.41.0 wheel), real `headroom proxy` in front of a local stub Anthropic upstream that streams `/v1/messages` with CRLF framing (`event:`/`data:` lines ending `\r\n`) and no `usage` fields, `HEADROOM_OFFLINE=1` - Exact command / steps: started the stub on :18951 and `HEADROOM_OFFLINE=1 python -m headroom.cli proxy --port 18952 --mode token --anthropic-api-url http://127.0.0.1:18951 --no-rate-limit`, then sent a streaming `POST /v1/messages` (`max_tokens: 1024`). The stub streams 81 `text_delta` events (521 chars of text, 11,275 bytes on the wire) and ends with `stop_reason: end_turn`. I read the proxy's `No usage chunk in SSE` log line, on `main` and on this branch. - Observed result: the client gets the same 11,275 bytes in both runs. On `main` the proxy can't parse the stream and records 281 output tokens from raw bytes. On this branch it records 130, counted from the stream text. ```text == main proxy status: 200 | streamed bytes to client: 11275 No usage chunk in SSE; output_tokens=281 (estimated from 11275 raw SSE bytes) == this branch proxy status: 200 | streamed bytes to client: 11275 No usage chunk in SSE; output_tokens=130 (counted from stream text) ``` - Not tested: a real upstream that frames with CRLF (stub only); the OpenAI chat path. There, without memory mode, the proxy doesn't buffer the stream text at all, so the estimate is bytes-based on both branches. That's a separate gap; I checked it with a CRLF OpenAI stub and got the same number on `main` and on this branch. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: N/A - Stable/default behavior changed: only for CRLF- or CR-framed streams without a usage chunk: their output tokens are now counted from the text or the ceiling instead of the raw-bytes fallback. LF streams are unchanged. - Kill switch / disable path: N/A - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Related but separate: #4252 covers CR-only frames in the streaming byte-buffer parser (`sse_byte_buffer_policy`), a different code path. This PR touches only the post-stream estimate. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JerrettDavis
parent
47288d67e9
commit
755293dd68
@@ -40,9 +40,12 @@ TEXT_CHARS_PER_TOKEN = 4
|
||||
def _iter_sse_payloads(sse: str) -> Iterator[Any]:
|
||||
"""Yield each ``data:`` payload in an SSE stream as a parsed object.
|
||||
|
||||
Tolerates the two things real streams do that a naive split does not:
|
||||
an event whose ``data:`` is spread over multiple lines, and ``[DONE]``.
|
||||
Tolerates the things real streams do that a naive split does not: an
|
||||
event whose ``data:`` is spread over multiple lines, ``[DONE]``, and CRLF
|
||||
or bare-CR line endings (both valid SSE; JSON payloads escape their own
|
||||
CRs, so normalising them cannot touch the data).
|
||||
"""
|
||||
sse = sse.replace("\r\n", "\n").replace("\r", "\n")
|
||||
for block in sse.split("\n\n"):
|
||||
lines = [ln for ln in block.split("\n") if ln.startswith("data:")]
|
||||
if not lines:
|
||||
|
||||
@@ -121,6 +121,25 @@ def test_multi_line_data_fields_concatenate() -> None:
|
||||
assert extract_stream_text(sse) == "joined"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("newline", ["\r\n", "\r"], ids=["crlf", "cr"])
|
||||
def test_crlf_and_cr_framed_streams_are_parsed(newline: str) -> None:
|
||||
"""SSE allows CRLF and bare CR line endings; the proxy forwards both (#649)."""
|
||||
lf = _sse(
|
||||
_chat_delta("Hello world, " * 20),
|
||||
{"choices": [{"index": 0, "delta": {}, "finish_reason": "length"}]},
|
||||
)
|
||||
sse = lf.replace("\n", newline)
|
||||
|
||||
assert extract_stream_text(sse) == extract_stream_text(lf) == "Hello world, " * 20
|
||||
assert estimate_output_tokens(
|
||||
sse_text=sse, total_bytes=len(sse.encode()), body={"max_tokens": 64}
|
||||
) == (64, "exact_ceiling")
|
||||
assert estimate_output_tokens(sse_text=sse, total_bytes=len(sse.encode())) == (
|
||||
len("Hello world, " * 20) // TEXT_CHARS_PER_TOKEN,
|
||||
"estimated_text",
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# The estimate itself
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
Reference in New Issue
Block a user