fix(proxy): count output tokens from CRLF-framed SSE streams too (#4298)

## Description

When an upstream streams a response with no usage chunk,
`estimate_output_tokens()` (`headroom/proxy/stream_output_tokens.py`)
recovers the output-token count from the stream itself. It uses the
request's output ceiling when the turn stopped on it, otherwise the
generated text, and only as a last resort `total_bytes // 40`. All of
that goes through `_iter_sse_payloads()`, which splits events on
`"\n\n"` and lines on `"\n"` only.

SSE allows CRLF and bare-CR line endings, and the rest of the proxy
handles them (#649). A CRLF-framed stream has no `"\n\n"` in it, though:
it becomes one block, every `data:` line keeps a trailing `\r`, the
concatenated payloads aren't valid JSON, and nothing is recovered. The
stop reason isn't found either, so even the exact-ceiling rung is
skipped and the estimate falls through to the raw-bytes heuristic. For
the stream in the proof below that's 281 instead of 130 tokens, a 2.2×
over-count on the number that feeds the cost model and the
output-savings estimate.

Closes: N/A (no matching issue found)

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/stream_output_tokens.py`: `_iter_sse_payloads()`
normalises `\r\n` and bare `\r` to `\n` before splitting. JSON payloads
escape their own carriage returns (`\r` inside a string is the two
characters `\` `r`), so this can't alter the data. LF streams are
processed exactly as before. The function is shared by
`extract_stream_text()` and the ceiling detector, so both rungs are
fixed.
- `tests/test_stream_output_tokens.py`:
`test_crlf_and_cr_framed_streams_are_parsed[crlf, cr]` takes an LF
stream that ends on `finish_reason: "length"`, re-frames it with CRLF
and with bare CR, and checks that the extracted text matches the LF
stream's, that the ceiling rung returns `(64, "exact_ceiling")`, and
that the text rung returns `estimated_text`.

## Testing

- [x] Unit tests pass (`pytest`): `tests/test_stream_output_tokens.py`,
`tests/test_streaming_usage_parser.py`,
`tests/test_streaming_sse_malformed_events.py`,
`tests/test_sse_byte_buffer_policy.py`,
`tests/test_proxy_streaming_resilience.py`,
`tests/test_openai_streaming_backend.py`,
`tests/test_sse_utf8_split.py`, not the full suite
- [x] Linting passes (`ruff check .`): on the changed files
- [ ] Type checking passes (`mypy headroom`): not run
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

New test on `main` (fix reverted, test kept):

```text
$ python -m pytest tests/test_stream_output_tokens.py -q
E   AssertionError: assert '' == 'Hello world,...Hello world, '
E   AssertionError: assert '' == 'Hello world,...Hello world, '
FAILED tests/test_stream_output_tokens.py::test_crlf_and_cr_framed_streams_are_parsed[crlf]
FAILED tests/test_stream_output_tokens.py::test_crlf_and_cr_framed_streams_are_parsed[cr]
========================= 2 failed, 27 passed in 1.31s =========================
```

With the fix:

```text
$ python -m pytest tests/test_stream_output_tokens.py tests/test_streaming_usage_parser.py tests/test_streaming_sse_malformed_events.py \
    tests/test_sse_byte_buffer_policy.py tests/test_proxy_streaming_resilience.py tests/test_openai_streaming_backend.py tests/test_sse_utf8_split.py -q
======================== 425 passed, 3 skipped in 8.34s ========================

$ ruff check headroom/proxy/stream_output_tokens.py tests/test_stream_output_tokens.py
All checks passed!
$ ruff format --check headroom/proxy/stream_output_tokens.py tests/test_stream_output_tokens.py
2 files already formatted
```

## Real Behavior Proof

- Environment: macOS 27.0.1 (arm64), Python 3.12.11, source tree on this
branch (native `headroom._core` from the 0.41.0 wheel), real `headroom
proxy` in front of a local stub Anthropic upstream that streams
`/v1/messages` with CRLF framing (`event:`/`data:` lines ending `\r\n`)
and no `usage` fields, `HEADROOM_OFFLINE=1`
- Exact command / steps: started the stub on :18951 and
`HEADROOM_OFFLINE=1 python -m headroom.cli proxy --port 18952 --mode
token --anthropic-api-url http://127.0.0.1:18951 --no-rate-limit`, then
sent a streaming `POST /v1/messages` (`max_tokens: 1024`). The stub
streams 81 `text_delta` events (521 chars of text, 11,275 bytes on the
wire) and ends with `stop_reason: end_turn`. I read the proxy's `No
usage chunk in SSE` log line, on `main` and on this branch.
- Observed result: the client gets the same 11,275 bytes in both runs.
On `main` the proxy can't parse the stream and records 281 output tokens
from raw bytes. On this branch it records 130, counted from the stream
text.

  ```text
  == main
  proxy status: 200 | streamed bytes to client: 11275
No usage chunk in SSE; output_tokens=281 (estimated from 11275 raw SSE
bytes)
  == this branch
  proxy status: 200 | streamed bytes to client: 11275
  No usage chunk in SSE; output_tokens=130 (counted from stream text)
  ```
- Not tested: a real upstream that frames with CRLF (stub only); the
OpenAI chat path. There, without memory mode, the proxy doesn't buffer
the stream text at all, so the estimate is bytes-based on both branches.
That's a separate gap; I checked it with a CRLF OpenAI stub and got the
same number on `main` and on this branch.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: only for CRLF- or CR-framed streams
without a usage chunk: their output tokens are now counted from the text
or the ceiling instead of the raw-bytes fallback. LF streams are
unchanged.
- Kill switch / disable path: N/A
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Related but separate: #4252 covers CR-only frames in the streaming
byte-buffer parser (`sse_byte_buffer_policy`), a different code path.
This PR touches only the post-stream estimate.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Rudy Mizrahi Celekli
2026-10-24 08:43:49 -07:00
committed by GitHub
co-authored by Claude Opus 5.5 JerrettDavis
parent 47288d67e9
commit 755293dd68
2 changed files with 24 additions and 2 deletions
+5 -2
View File
@@ -40,9 +40,12 @@ TEXT_CHARS_PER_TOKEN = 4
def _iter_sse_payloads(sse: str) -> Iterator[Any]:
"""Yield each ``data:`` payload in an SSE stream as a parsed object.
Tolerates the two things real streams do that a naive split does not:
an event whose ``data:`` is spread over multiple lines, and ``[DONE]``.
Tolerates the things real streams do that a naive split does not: an
event whose ``data:`` is spread over multiple lines, ``[DONE]``, and CRLF
or bare-CR line endings (both valid SSE; JSON payloads escape their own
CRs, so normalising them cannot touch the data).
"""
sse = sse.replace("\r\n", "\n").replace("\r", "\n")
for block in sse.split("\n\n"):
lines = [ln for ln in block.split("\n") if ln.startswith("data:")]
if not lines:
+19
View File
@@ -121,6 +121,25 @@ def test_multi_line_data_fields_concatenate() -> None:
assert extract_stream_text(sse) == "joined"
@pytest.mark.parametrize("newline", ["\r\n", "\r"], ids=["crlf", "cr"])
def test_crlf_and_cr_framed_streams_are_parsed(newline: str) -> None:
"""SSE allows CRLF and bare CR line endings; the proxy forwards both (#649)."""
lf = _sse(
_chat_delta("Hello world, " * 20),
{"choices": [{"index": 0, "delta": {}, "finish_reason": "length"}]},
)
sse = lf.replace("\n", newline)
assert extract_stream_text(sse) == extract_stream_text(lf) == "Hello world, " * 20
assert estimate_output_tokens(
sse_text=sse, total_bytes=len(sse.encode()), body={"max_tokens": 64}
) == (64, "exact_ceiling")
assert estimate_output_tokens(sse_text=sse, total_bytes=len(sse.encode())) == (
len("Hello world, " * 20) // TEXT_CHARS_PER_TOKEN,
"estimated_text",
)
# --------------------------------------------------------------------------- #
# The estimate itself
# --------------------------------------------------------------------------- #