mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): count a ceiling-stopped stream exactly instead of estimating its text (#3771)
## What's wrong When an upstream omits its usage chunk, `estimate_output_tokens` counts the stream's own generated text. That's right for a turn that finished on its own, and catastrophically wrong for one shape of turn: **a tool call cut off by `max_tokens`.** Tool arguments stream as `input_json_delta` fragments. An upstream that stops mid-object drops the incomplete remainder rather than emit unparseable JSON — while still billing every token it generated. Measured against Anthropic (`claude-sonnet-4-5`, a `Write` call over a ~14k-token context): ``` stop_reason: max_tokens usage.output_tokens: 1500 recoverable streamed text: ~45 characters (~11 tokens by the text rung) ``` **A 100x under-count**, landing on the most expensive turns there are, and biasing output-token accounting downward exactly where it hurts. Output tokens feed the cost model and the output-shaping savings estimate. ## The fix There is nothing to estimate. The ceiling is denominated in output tokens, so a turn that stopped *because* it hit the ceiling produced exactly that many. Confirmed at two ceilings, both exact: ``` max_tokens=1500 -> usage.output_tokens=1500 max_tokens=4096 -> usage.output_tokens=4096 ``` So `estimate_output_tokens` gains a rung above the text estimate, sourced `exact_ceiling`. It covers all three surfaces the proxy forwards: | surface | terminal signal | ceiling field | |---|---|---| | Anthropic messages | `message_delta.delta.stop_reason` | `max_tokens` | | OpenAI chat | `choices[].finish_reason: length` | `max_completion_tokens` | | OpenAI responses | `response.incomplete_details.reason` | `max_output_tokens` | All three fields are denominated in output tokens with reasoning included, which is what makes the ceiling exact rather than a lower bound. The body passed at the call site is the **provider request as sent**, so the ceiling read is the one the upstream actually enforced — not what the client asked for before the proxy's own levers touched it. Two deliberate refusals to guess: - A truncated turn whose body carries no usable ceiling falls through to the existing rungs. - `bool` is rejected explicitly. It's an `int` subclass, and `max_tokens: True` is not a one-token answer. The `PERF` log line names the new basis (`exact: turn hit its output-token ceiling`) rather than implying an estimate. ## Behaviour change None for turns that finish normally, and none for any turn whose provider sent a usage chunk — the overwhelming majority. This rung is reachable only on the path that was already estimating, and it replaces a wrong number with an exact one. ## Tests Six new tests in `tests/test_stream_output_tokens.py`: **6 fail before this change, 27 pass after.** They cover the truncated Anthropic tool call (including an explicit assertion of what the text rung would have said), both OpenAI surfaces, the normal-completion case that must still count text, fall-through when no usable ceiling exists, and the `bool` guard. Also run: `ruff check` / `ruff format --check` (0.16.7, per `scripts/verify-ruff-version.py`) / `mypy headroom --ignore-missing-imports` — all clean; plus 79 streaming/usage tests across `tests/test_proxy`, all passing. ## Provenance Found while prototyping an output echo-ratio measurement against live Anthropic traffic — the probe reported a tool-call turn as ~2 words of output against 4096 billed tokens, which turned out to be this rather than a bug in the probe. Unrelated to that work otherwise; filed on its own. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This commit is contained in:
@@ -969,13 +969,20 @@ class StreamingMixin:
|
||||
output_tokens, output_tokens_source = estimate_output_tokens(
|
||||
sse_text=full_sse_data,
|
||||
total_bytes=stream_state["total_bytes"],
|
||||
# The provider body as sent, so a turn stopped by its output
|
||||
# ceiling can be counted exactly instead of estimated. Matters
|
||||
# most for a tool call truncated mid-arguments, whose dropped
|
||||
# JSON leaves almost no text to count.
|
||||
body=body,
|
||||
)
|
||||
# Name the actual basis. The old message always said "from N bytes"
|
||||
# even though that is now only true for the fallback rung, and an
|
||||
# operator reading it needs to know which estimate they are looking
|
||||
# at before trusting the number.
|
||||
basis = (
|
||||
"counted from stream text"
|
||||
"exact: turn hit its output-token ceiling"
|
||||
if output_tokens_source == "exact_ceiling"
|
||||
else "counted from stream text"
|
||||
if output_tokens_source == "estimated_text"
|
||||
else f"estimated from {stream_state['total_bytes']} raw SSE bytes"
|
||||
)
|
||||
|
||||
@@ -123,15 +123,88 @@ def extract_stream_text(sse: str) -> str:
|
||||
return "".join(parts)
|
||||
|
||||
|
||||
def estimate_output_tokens(*, sse_text: str, total_bytes: int) -> tuple[int, str]:
|
||||
# Terminal stop reasons that mean "the output-token ceiling cut this turn off".
|
||||
# Anthropic messages say ``max_tokens``; OpenAI chat completions say ``length``.
|
||||
_CEILING_STOP_REASONS = frozenset({"max_tokens", "length", "max_output_tokens"})
|
||||
|
||||
# Request fields that carry that ceiling, per surface. All three are denominated
|
||||
# in OUTPUT tokens (reasoning included, where a model emits it), which is what
|
||||
# makes the ceiling an exact count rather than a lower bound.
|
||||
_CEILING_FIELDS = ("max_tokens", "max_completion_tokens", "max_output_tokens")
|
||||
|
||||
|
||||
def _hit_output_ceiling(sse_text: str) -> bool:
|
||||
"""Whether the stream ended because it hit the request's output ceiling."""
|
||||
for obj in _iter_sse_payloads(sse_text):
|
||||
if not isinstance(obj, dict):
|
||||
continue
|
||||
|
||||
# OpenAI chat completions — choices[].finish_reason
|
||||
choices = obj.get("choices")
|
||||
if isinstance(choices, list):
|
||||
for choice in choices:
|
||||
if isinstance(choice, dict) and choice.get("finish_reason") in (
|
||||
_CEILING_STOP_REASONS
|
||||
):
|
||||
return True
|
||||
continue
|
||||
|
||||
# Anthropic messages — message_delta.delta.stop_reason
|
||||
delta = obj.get("delta")
|
||||
if isinstance(delta, dict) and delta.get("stop_reason") in _CEILING_STOP_REASONS:
|
||||
return True
|
||||
|
||||
# OpenAI responses — response.incomplete_details.reason
|
||||
response = obj.get("response")
|
||||
if isinstance(response, dict):
|
||||
details = response.get("incomplete_details")
|
||||
if isinstance(details, dict) and details.get("reason") in _CEILING_STOP_REASONS:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _requested_ceiling(body: Any) -> int | None:
|
||||
"""The output-token ceiling the request asked for, or ``None``."""
|
||||
if not isinstance(body, dict):
|
||||
return None
|
||||
for field in _CEILING_FIELDS:
|
||||
value = body.get(field)
|
||||
# bool is an int subclass and would sail through as 0/1.
|
||||
if isinstance(value, int) and not isinstance(value, bool) and value > 0:
|
||||
return value
|
||||
return None
|
||||
|
||||
|
||||
def estimate_output_tokens(*, sse_text: str, total_bytes: int, body: Any = None) -> tuple[int, str]:
|
||||
"""Return ``(tokens, source)`` for a stream with no provider usage chunk.
|
||||
|
||||
``source`` names which rung of the ladder produced the number so the caller
|
||||
can log it honestly rather than implying the provider reported it:
|
||||
|
||||
* ``exact_ceiling`` — the turn hit its output-token ceiling (exact)
|
||||
* ``estimated_text`` — counted from the generated text (good)
|
||||
* ``estimated_bytes`` — the raw-wire fallback (poor, last resort)
|
||||
|
||||
The top rung exists because counting the stream's text is catastrophically
|
||||
wrong for exactly one shape of turn: a TOOL CALL cut off by ``max_tokens``.
|
||||
The arguments stream as ``input_json_delta`` fragments, and an upstream that
|
||||
stops mid-object drops the incomplete remainder rather than emit unparseable
|
||||
JSON — while still billing every token it generated. Measured against
|
||||
Anthropic: a turn billed 1500 output tokens left ~45 characters of
|
||||
recoverable text, which the text rung reads as ~11 tokens. A 100x
|
||||
under-count, landing on the most expensive turns there are.
|
||||
|
||||
Nothing needs estimating there. The ceiling is denominated in output tokens,
|
||||
so a turn that stopped *because* it hit the ceiling produced exactly that
|
||||
many. ``body`` is the provider request as sent, so the value read here is
|
||||
the ceiling the upstream actually enforced — not whatever the client
|
||||
originally asked for before the proxy's own levers touched it.
|
||||
"""
|
||||
if _hit_output_ceiling(sse_text):
|
||||
ceiling = _requested_ceiling(body)
|
||||
if ceiling is not None:
|
||||
return ceiling, "exact_ceiling"
|
||||
|
||||
text = extract_stream_text(sse_text)
|
||||
if text:
|
||||
# At least one token for any non-empty answer: integer division would
|
||||
|
||||
@@ -169,3 +169,116 @@ def test_falls_back_to_bytes_when_no_text_is_recoverable() -> None:
|
||||
def test_negative_or_zero_bytes_are_safe() -> None:
|
||||
assert estimate_output_tokens(sse_text="", total_bytes=0) == (0, "estimated_bytes")
|
||||
assert estimate_output_tokens(sse_text="", total_bytes=-5) == (0, "estimated_bytes")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# A turn stopped by its output ceiling is counted exactly, never estimated.
|
||||
#
|
||||
# Measured against Anthropic while probing echo ratios: a Write tool call cut
|
||||
# off by ``max_tokens`` billed 1500 output tokens and left ~45 characters of
|
||||
# recoverable stream text. Tool arguments stream as ``input_json_delta``
|
||||
# fragments, and an upstream that stops mid-object drops the incomplete
|
||||
# remainder rather than emit unparseable JSON. The text rung reads that as ~11
|
||||
# tokens: a 100x under-count, on the most expensive turns there are. Repeated
|
||||
# at two ceilings, both exact:
|
||||
#
|
||||
# max_tokens=1500 -> usage.output_tokens=1500
|
||||
# max_tokens=4096 -> usage.output_tokens=4096
|
||||
#
|
||||
# Which is the fix: the ceiling is denominated in output tokens, so a turn that
|
||||
# stopped because it hit the ceiling produced exactly that many.
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def _truncated_anthropic_tool_call() -> str:
|
||||
"""An Anthropic tool call whose arguments were cut off mid-JSON."""
|
||||
return _sse(
|
||||
{
|
||||
"type": "content_block_start",
|
||||
"index": 0,
|
||||
"content_block": {"type": "tool_use", "id": "toolu_1", "name": "Write", "input": {}},
|
||||
},
|
||||
{
|
||||
"type": "content_block_delta",
|
||||
"index": 0,
|
||||
"delta": {"type": "input_json_delta", "partial_json": '{"path"'},
|
||||
},
|
||||
{
|
||||
"type": "content_block_delta",
|
||||
"index": 0,
|
||||
"delta": {"type": "input_json_delta", "partial_json": ': "a/b.py"'},
|
||||
},
|
||||
{"type": "message_delta", "delta": {"stop_reason": "max_tokens"}},
|
||||
{"type": "message_stop"},
|
||||
done=False,
|
||||
)
|
||||
|
||||
|
||||
def test_truncated_tool_call_is_counted_from_the_ceiling_not_the_text() -> None:
|
||||
sse = _truncated_anthropic_tool_call()
|
||||
tokens, source = estimate_output_tokens(
|
||||
sse_text=sse, total_bytes=len(sse.encode()), body={"max_tokens": 1500}
|
||||
)
|
||||
assert source == "exact_ceiling"
|
||||
assert tokens == 1500
|
||||
|
||||
# What the text rung would have said, and why it cannot be trusted here.
|
||||
text_tokens, text_source = estimate_output_tokens(sse_text=sse, total_bytes=len(sse.encode()))
|
||||
assert text_source == "estimated_text"
|
||||
assert text_tokens < 50
|
||||
|
||||
|
||||
def test_openai_chat_length_finish_reason_uses_max_completion_tokens() -> None:
|
||||
sse = _sse(
|
||||
_chat_delta("partial answer that ran out of room"),
|
||||
{"choices": [{"index": 0, "delta": {}, "finish_reason": "length"}]},
|
||||
)
|
||||
tokens, source = estimate_output_tokens(
|
||||
sse_text=sse, total_bytes=len(sse.encode()), body={"max_completion_tokens": 900}
|
||||
)
|
||||
assert (tokens, source) == (900, "exact_ceiling")
|
||||
|
||||
|
||||
def test_openai_responses_incomplete_uses_max_output_tokens() -> None:
|
||||
sse = _sse(
|
||||
{"type": "response.output_text.delta", "delta": "cut off"},
|
||||
{
|
||||
"type": "response.incomplete",
|
||||
"response": {"incomplete_details": {"reason": "max_output_tokens"}},
|
||||
},
|
||||
)
|
||||
tokens, source = estimate_output_tokens(
|
||||
sse_text=sse, total_bytes=len(sse.encode()), body={"max_output_tokens": 256}
|
||||
)
|
||||
assert (tokens, source) == (256, "exact_ceiling")
|
||||
|
||||
|
||||
def test_a_turn_that_finished_normally_still_counts_its_text() -> None:
|
||||
"""The new rung must not swallow the common case."""
|
||||
sse = _sse(
|
||||
_chat_delta("a complete answer "),
|
||||
_chat_delta("that stopped on its own"),
|
||||
{"choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}]},
|
||||
)
|
||||
tokens, source = estimate_output_tokens(
|
||||
sse_text=sse, total_bytes=len(sse.encode()), body={"max_tokens": 4096}
|
||||
)
|
||||
assert source == "estimated_text"
|
||||
assert tokens == len(extract_stream_text(sse)) // TEXT_CHARS_PER_TOKEN
|
||||
|
||||
|
||||
def test_ceiling_rung_falls_through_when_the_body_does_not_carry_one() -> None:
|
||||
"""A truncated turn with no usable ceiling must degrade, not guess."""
|
||||
sse = _truncated_anthropic_tool_call()
|
||||
for body in (None, {}, {"max_tokens": 0}, {"max_tokens": "1500"}):
|
||||
_, source = estimate_output_tokens(sse_text=sse, total_bytes=len(sse.encode()), body=body)
|
||||
assert source == "estimated_text", body
|
||||
|
||||
|
||||
def test_boolean_ceiling_is_not_mistaken_for_a_token_count() -> None:
|
||||
"""``bool`` is an ``int`` subclass; ``max_tokens: True`` is not 1 token."""
|
||||
sse = _truncated_anthropic_tool_call()
|
||||
_, source = estimate_output_tokens(
|
||||
sse_text=sse, total_bytes=len(sse.encode()), body={"max_tokens": True}
|
||||
)
|
||||
assert source == "estimated_text"
|
||||
|
||||
Reference in New Issue
Block a user