fix(backends/litellm): align streaming usage tokens with the non-streaming path (#2688)

## Description

On the LiteLLM/Bedrock streaming path, the terminal `message_delta`
usage does not match what the non-streaming path reports for the same
response. Two token buckets are wrong, both because the streaming code
does not reuse the logic the non-streaming path already has in
`_anthropic_usage_from_litellm`. This PR fixes both so the streaming
usage object is a conformant Anthropic usage object.

**1. `input_tokens` double-counts the cached prefix.** LiteLLM's
`prompt_tokens` is the total prompt size, inclusive of cache-read and
cache-write tokens (`AmazonConverseConfig._transform_usage` takes
Bedrock's raw `inputTokens` and adds `cacheReadInputTokens +
cacheWriteInputTokens`, on both the streaming and non-streaming paths).
Anthropic's `input_tokens` must exclude both, since the response also
carries `cache_read_input_tokens` and `cache_creation_input_tokens` and
clients treat the three buckets as disjoint (they sum them for the total
prompt and price each differently: cache-read 0.1x, cache-write 1.25x).
The non-streaming path subtracts (added in #1848 for #1345); the
streaming path added in #2144 emitted the inclusive `prompt_tokens`
straight through. So on every cached streaming turn the client counts
the cached prefix twice, once at the discounted cache rate and again
inside `input_tokens` at the full input rate.

**2. `output_tokens` reports an SSE chunk count, not real tokens.** The
streaming loop increments `output_tokens` once per
`content_block_delta`. That is a chunk count, and it undercounts the
true output token total several-fold because each delta carries multiple
tokens. The trailing usage chunk (requested via
`stream_options.include_usage`) already carries the provider's real
`completion_tokens`, but `stream_message` only read prompt and cache
tokens from it and discarded `completion_tokens`. The non-streaming path
uses `completion_tokens` directly. This undercount reaches both the
client and Headroom's own accounting: the Bedrock streamer copies
`output_tokens` into the `RequestOutcome`, which feeds the `RequestLog`,
the `PERF tok_out` field, and the verbosity savings estimate.

Both fixes mirror `_anthropic_usage_from_litellm`: subtract the cache
tokens from `input_tokens`, and prefer the real `completion_tokens` for
`output_tokens` (falling back to the delta count only when no usage
chunk arrives).

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/backends/litellm.py` (`stream_message`): terminal
`message_delta` `input_tokens` is now `max(prompt_tokens - cache_read -
cache_write, 0)` instead of the inclusive `prompt_tokens`;
`output_tokens` now uses the trailing usage chunk's `completion_tokens`
when present, falling back to the per-delta count only when no usage
chunk arrives. Both match `_anthropic_usage_from_litellm` on the
non-streaming path.
- `tests/test_bedrock_tool_result_cache_and_streaming_stats.py`:
`test_terminal_message_delta_carries_real_cache_stats` updated (its
`input_tokens == 1500` assertion encoded the double-count: 1500
inclusive with 1200 read + 300 write left 0 uncached) to a 1600-token
prompt with 100 uncached tokens and a real-`completion_tokens` output
assertion; added
`test_terminal_message_delta_input_tokens_excludes_cache_tokens`
(disjoint-bucket invariant),
`test_output_tokens_uses_real_completion_tokens_not_delta_count`, and
`test_output_tokens_falls_back_to_delta_count_without_usage_chunk`. The
`_mock_stream_with_final_usage` helper now carries `completion_tokens`
to mirror the real trailing chunk.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
# Fail-before (source fix stashed, updated tests kept):
  assert final_usage["input_tokens"] == 100   -> assert 1600 == 100
  assert final_usage["input_tokens"] == 500   -> assert 5000 == 500
  assert final_usage["output_tokens"] == 50    -> assert 1 == 50
  assert final_usage["output_tokens"] == 137   -> assert 1 == 137

# Pass-after (fix applied):
tests/test_bedrock_tool_result_cache_and_streaming_stats.py ................
16 passed in 6.21s

# Related suites (streaming cache metrics, bedrock streaming input tokens,
# nonstream cache usage, litellm cache stats, backend bugs, streaming resilience):
all passed

# uvx ruff@0.15.17 check  -> All checks passed!
# uvx mypy@1.20.2 headroom/backends/litellm.py -> Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.11, litellm installed in the
project venv, pytest 9.1.1 with pytest-asyncio 1.4.0 (asyncio_mode=auto
per pyproject).
- Exact command / steps: verified litellm inclusivity and the
completion_tokens source by reading
`litellm/llms/bedrock/chat/converse_transformation.py`
(`_transform_usage` does `input_tokens += cacheReadInputTokens;
input_tokens += cacheWriteInputTokens; prompt_tokens=input_tokens;
completion_tokens=output_tokens`) and `invoke_handler.py` (the streaming
trailing chunk calls the same `_transform_usage`); then fail-before with
`git stash push headroom/backends/litellm.py` and `python -m pytest
tests/test_bedrock_tool_result_cache_and_streaming_stats.py` (the four
value assertions fail, emitting inclusive 1600 / 5000 for input and the
delta count 1 for output), and pass-after with `git stash pop` and
rerunning the file (16 passed).
- Observed result: on the fixed streaming path the terminal
`message_delta.usage` reports `input_tokens` as the genuinely-uncached
count with cache tokens only in `cache_read_input_tokens` and
`cache_creation_input_tokens` (prompt_tokens=1600, read=1200, write=300
gives input_tokens=100 and 100+1200+300=1600, versus the old inclusive
1600 that a client sums with the cache fields to 3100), and reports
`output_tokens` as the provider's `completion_tokens` (50 or 137 in the
tests) rather than the single content-delta count. Both now match the
non-streaming response shape.
- Not tested: a live Bedrock Converse streaming call against real AWS
credentials (no Bedrock access in this environment). The litellm usage
shape is exercised through a mock that mirrors
`stream_options.include_usage`'s trailing chunk, and the litellm
inclusivity and completion_tokens fields were verified against the
installed litellm source rather than assumed.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`: it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Docs and manual testing are N/A: this is an internal usage-mapping
correctness fix with no API surface or documented behavior change. The
streaming path already read the top-level `cache_read_input_tokens` /
`cache_creation_input_tokens` attrs, so the input subtraction uses the
same values the response emits; the OpenAI nested
`prompt_tokens_details` dialect on streaming is a separate gap and out
of scope here.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Abhay Singh
2026-10-02 11:07:56 -05:00
committed by GitHub
co-authored by JD Davis
parent 57a57088de
commit 90a68e0291
2 changed files with 143 additions and 5 deletions
+23 -2
View File
@@ -1431,6 +1431,14 @@ class LiteLLMBackend(Backend):
final_input_tokens = 0
final_cache_read_tokens = 0
final_cache_write_tokens = 0
# Real output-token count from the trailing usage chunk. The
# ``output_tokens`` counter incremented per content_block_delta below
# is only a delta *count* (one per SSE chunk), which undercounts the
# true token total several-fold. Prefer the provider's
# completion_tokens when the usage chunk carries it, exactly like the
# non-streaming path (_anthropic_usage_from_litellm), and fall back to
# the delta count only when no usage chunk arrives.
final_output_tokens = 0
# Extended thinking is BUFFERED, not streamed live. A thinking block
# is only legal to replay if it leads the turn and carries a real
@@ -1533,6 +1541,7 @@ class LiteLLMBackend(Backend):
final_cache_write_tokens = int(
getattr(cu, "cache_creation_input_tokens", 0) or 0
)
final_output_tokens = int(getattr(cu, "completion_tokens", 0) or 0)
if not hasattr(chunk, "choices") or not chunk.choices:
continue
@@ -1669,9 +1678,21 @@ class LiteLLMBackend(Backend):
data={"type": "content_block_stop", "index": current_block_index},
)
delta_usage: dict[str, Any] = {"output_tokens": output_tokens}
delta_usage: dict[str, Any] = {"output_tokens": final_output_tokens or output_tokens}
if final_input_tokens or final_cache_read_tokens or final_cache_write_tokens:
delta_usage["input_tokens"] = final_input_tokens
# LiteLLM's prompt_tokens is the *total* prompt size, inclusive
# of the cache-read and cache-write tokens (Bedrock reports raw
# inputTokens and LiteLLM's AmazonConverseConfig._transform_usage
# adds cacheReadInputTokens + cacheWriteInputTokens onto it).
# Anthropic's input_tokens must exclude both, since the cache
# fields below report them separately and clients treat the three
# buckets as disjoint. Without the subtraction a cached streaming
# turn double-counts the cached prefix in input_tokens at the full
# input rate. Mirrors the non-streaming path in
# _anthropic_usage_from_litellm (#1345 / #1848).
delta_usage["input_tokens"] = max(
final_input_tokens - final_cache_read_tokens - final_cache_write_tokens, 0
)
if final_cache_read_tokens:
delta_usage["cache_read_input_tokens"] = final_cache_read_tokens
if final_cache_write_tokens:
@@ -118,7 +118,11 @@ class TestStreamingCacheStatsCompletion:
"""stream_message must request usage and re-surface real cache stats."""
def _mock_stream_with_final_usage(
self, cache_read: int = 0, cache_write: int = 0, prompt_tokens: int = 0
self,
cache_read: int = 0,
cache_write: int = 0,
prompt_tokens: int = 0,
completion_tokens: int = 50,
):
async def mock_stream():
chunk1 = MagicMock()
@@ -136,6 +140,7 @@ class TestStreamingCacheStatsCompletion:
]
chunk2.usage = MagicMock(
prompt_tokens=prompt_tokens,
completion_tokens=completion_tokens,
cache_read_input_tokens=cache_read,
cache_creation_input_tokens=cache_write,
)
@@ -169,8 +174,12 @@ class TestStreamingCacheStatsCompletion:
patch("headroom.backends.litellm.acompletion", new_callable=AsyncMock) as mock_acomp,
patch("headroom.backends.litellm._fetch_bedrock_inference_profiles", return_value={}),
):
# LiteLLM's prompt_tokens is inclusive of the cache tokens, so this
# 1600-token prompt is 100 uncached + 1200 cache-read + 300
# cache-write. Anthropic's input_tokens must report only the 100
# uncached tokens; the cache fields carry the rest.
mock_acomp.return_value = self._mock_stream_with_final_usage(
cache_read=1200, cache_write=300, prompt_tokens=1500
cache_read=1200, cache_write=300, prompt_tokens=1600
)()
backend = LiteLLMBackend(provider="bedrock", region="us-east-1")
@@ -191,9 +200,117 @@ class TestStreamingCacheStatsCompletion:
message_deltas = [e for e in events if e.event_type == "message_delta"]
assert len(message_deltas) == 1
final_usage = message_deltas[0].data["usage"]
assert final_usage["input_tokens"] == 1500
assert final_usage["input_tokens"] == 100
assert final_usage["cache_read_input_tokens"] == 1200
assert final_usage["cache_creation_input_tokens"] == 300
# Real completion_tokens from the usage chunk, not the 1 content delta.
assert final_usage["output_tokens"] == 50
@pytest.mark.asyncio
async def test_terminal_message_delta_input_tokens_excludes_cache_tokens(self):
"""input_tokens must be disjoint from the cache buckets (#1345/#1848).
LiteLLM/Bedrock reports prompt_tokens inclusive of cache-read and
cache-write tokens, but Anthropic's three input buckets are disjoint:
a client sums input_tokens + cache_read_input_tokens +
cache_creation_input_tokens to get the total prompt, and bills each at
a different rate (cache-read 0.1x, cache-write 1.25x). Emitting the
inclusive prompt_tokens as input_tokens double-counts the whole cached
prefix at the full input rate on every cached streaming turn. The
non-streaming path already subtracts; the streaming path must match.
"""
with (
patch("headroom.backends.litellm.acompletion", new_callable=AsyncMock) as mock_acomp,
patch("headroom.backends.litellm._fetch_bedrock_inference_profiles", return_value={}),
):
mock_acomp.return_value = self._mock_stream_with_final_usage(
cache_read=4000, cache_write=500, prompt_tokens=5000
)()
backend = LiteLLMBackend(provider="bedrock", region="us-east-1")
events = [
e
async for e in backend.stream_message(
{"model": "test", "messages": [{"role": "user", "content": "hi"}]}, {}
)
]
final_usage = next(e.data["usage"] for e in events if e.event_type == "message_delta")
# 5000 inclusive - 4000 read - 500 write = 500 genuinely-uncached.
assert final_usage["input_tokens"] == 500
assert final_usage["cache_read_input_tokens"] == 4000
assert final_usage["cache_creation_input_tokens"] == 500
# The disjoint-bucket invariant: the three sum back to prompt_tokens.
assert (
final_usage["input_tokens"]
+ final_usage["cache_read_input_tokens"]
+ final_usage["cache_creation_input_tokens"]
== 5000
)
@pytest.mark.asyncio
async def test_output_tokens_uses_real_completion_tokens_not_delta_count(self):
"""output_tokens must be the provider's completion_tokens, not the SSE
delta count.
The streaming loop increments output_tokens once per
content_block_delta, which is a chunk count and undercounts the true
token total several-fold. When the trailing usage chunk carries a real
completion_tokens (stream_options.include_usage), report that instead,
matching the non-streaming path. Here the stream emits a single content
delta but the usage chunk reports 137 output tokens.
"""
with (
patch("headroom.backends.litellm.acompletion", new_callable=AsyncMock) as mock_acomp,
patch("headroom.backends.litellm._fetch_bedrock_inference_profiles", return_value={}),
):
mock_acomp.return_value = self._mock_stream_with_final_usage(
prompt_tokens=200, completion_tokens=137
)()
backend = LiteLLMBackend(provider="bedrock", region="us-east-1")
events = [
e
async for e in backend.stream_message(
{"model": "test", "messages": [{"role": "user", "content": "hi"}]}, {}
)
]
final_usage = next(e.data["usage"] for e in events if e.event_type == "message_delta")
# One content delta streamed, but the real count is 137.
assert final_usage["output_tokens"] == 137
@pytest.mark.asyncio
async def test_output_tokens_falls_back_to_delta_count_without_usage_chunk(self):
"""When no usage chunk arrives, output_tokens falls back to the delta
count so a value is still reported."""
async def mock_stream():
for text in ("a", "b", "c"):
chunk = MagicMock()
chunk.usage = None
chunk.choices = [
MagicMock(delta=MagicMock(content=text, tool_calls=None), finish_reason=None)
]
yield chunk
with (
patch("headroom.backends.litellm.acompletion", new_callable=AsyncMock) as mock_acomp,
patch("headroom.backends.litellm._fetch_bedrock_inference_profiles", return_value={}),
):
mock_acomp.return_value = mock_stream()
backend = LiteLLMBackend(provider="openrouter")
events = [
e
async for e in backend.stream_message(
{"model": "test", "messages": [{"role": "user", "content": "hi"}]}, {}
)
]
final_usage = next(e.data["usage"] for e in events if e.event_type == "message_delta")
# Three content deltas, no usage chunk -> fall back to the count of 3.
assert final_usage["output_tokens"] == 3
@pytest.mark.asyncio
async def test_no_extra_message_start_when_usage_never_reported(self):