fix(proxy): record the provider's billed input count on streamed turns (#4052)

## Description

Every input-volume figure Headroom reports for streamed Claude traffic
was Headroom's own tokenizer estimate, not Anthropic's billed count.
That covers "input after Headroom", `/stats` `tokens.input`, the
dashboard's per-model table and the licence usage report's
`tokens_after`. Since Claude Code always streams, this was effectively
all Claude Code traffic, so customers comparing our numbers with the
Anthropic Console could not reconcile them.

**Root cause.** `emit_request_outcome` records `provider_input_tokens or
optimized_tokens`. The streaming finalizer never set
`provider_input_tokens` for Anthropic (#3946 fixed only the uncached
sub-count), so every streamed Claude turn fell back to
`optimized_tokens`. That value is:
- an `o200k_base` count standing in for Claude's private tokenizer,
measured against `count_tokens` on 6 real Claude Code transcripts at
15–25% under Claude 4.x and 35–38% under Claude 5.x;
- a count of the forwarded **messages only**, which never sees the
system prompt or tool definitions.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **`headroom/proxy/provider_usage.py` (new).** One place that knows
each provider's usage semantics:
- Anthropic, Bedrock and Vertex: `input_tokens` is the uncached tail,
and the billed total is `input + cache_read + cache_write`.
- OpenAI `prompt_tokens`/`input_tokens` and Gemini `promptTokenCount`
are already inclusive of cached tokens.
- **Streaming finalizer.** Sets `provider_input_tokens` from the
provider's usage on the Anthropic/OpenAI/Gemini SSE path, the Bedrock
stream path and the OpenAI-via-backend stream path.
`RequestOutcome.from_stream` gains the parameter.
- **OpenAI pass-through.** On both streaming and non-streaming paths,
Anthropic-shape usage was read as if `input_tokens` were the total. It
now uses the billed total and is marked provider-reported.
- **Gemini non-streaming and OpenAI Responses WebSocket.** These already
used the provider's count as `optimized_tokens` but never declared it,
so they now also set `provider_input_tokens`, but only when the provider
actually sent a count.
- **No estimate is ever labelled as billed.** The Bedrock #1132
`message_start` backfill and Gemini's fallback for a missing
`promptTokenCount` both stay estimated. The pass-through usage dialect
is chosen by provider, not inferred from which keys are present.
- **Provenance counters.** Every input total now carries how much of it
is billed versus estimated:
- `/stats` `tokens.input_provider_reported`, `input_estimated`,
`input_provider_reported_percent`, `requests_input_*`.
- Licence usage report: `tokens_after_provider_reported`,
`tokens_after_estimated`, `requests_provider_reported`, plus the billed
`cache_read_tokens`, `cache_write_tokens` and `uncached_input_tokens`,
so the cloud side can reconcile with the console (input + cache read +
cache write). The report also states `tokens_saved_basis:
"headroom_tokenizer"`.
- A one-time `event=input_tokens_estimated` warning per provider/model
when a successful request carried no provider usage.
- **No delta math changes.** `optimized_tokens` stays on the local
scale, and `tokens_saved`, ratios and the ledger delta are untouched.
The provider count is never subtracted from a local count.

## Testing

- [ ] Unit tests pass (`pytest`): targeted suites only, listed below.
The full suite is left to CI (see Additional Notes).
- [x] Linting passes (`ruff check .`): run on all changed files.
- [x] Type checking passes (`mypy headroom`): run on the changed
modules.
- [x] New tests added for new functionality
- [x] Manual testing performed

`tests/test_provider_billed_input.py` adds 15 tests, plus regression
tests for the review fixes in
`tests/test_bedrock_streaming_input_tokens.py` (the #1132 backfill is
reported as estimated) and `tests/test_gemini_ccr_continuation_usage.py`
(a missing `promptTokenCount` is not provider-reported):
- **Recorded fixture.**
`tests/fixtures/anthropic/stream_cached_turn.sse` is a **real recorded
Anthropic stream**: a second call on a cached ~8.9K-token system prompt,
billed as 19 uncached + 8,857 cache-read. It is replayed in 97-byte
chunks through the proxy's own SSE usage parser and
`_finalize_stream_response`, then through `emit_request_outcome` into a
real `PrometheusMetrics`, `CostTracker` and `UsageReporter`. The test
asserts that `/stats` input and the licence `tokens_after` equal
Anthropic's 8,876.
- Other cases:
  - an OpenAI stream records the inclusive `prompt_tokens`;
  - a stream without usage stays estimated;
  - each provider shape is covered;
  - the pass-through normaliser returns the total.

Against the unpatched handlers, the 5 behaviour tests fail. The 8 that
pass exercise only the new helper.

### Test Output

```text
$ pytest tests/test_provider_billed_input.py tests/test_proxy_handler_helpers.py tests/test_request_outcome.py \
    tests/test_proxy_streaming_request_logger.py tests/test_proxy_cache_ttl_metrics.py \
    tests/test_usage_reporter_snapshot.py tests/test_5xx_accounting_all_providers.py tests/test_copilot_provider_label.py \
    tests/test_bedrock_streaming_input_tokens.py tests/test_gemini_ccr_continuation_usage.py -q
181 passed in 6.56s

# same new tests against the unpatched handlers
FAILED tests/test_provider_billed_input.py::test_passthrough_normaliser_returns_anthropic_total_not_uncached_tail
FAILED tests/test_provider_billed_input.py::test_recorded_anthropic_stream_records_anthropics_billed_input
FAILED tests/test_provider_billed_input.py::test_openai_stream_records_inclusive_prompt_tokens
FAILED tests/test_provider_billed_input.py::test_streamed_claude_turn_reaches_dashboard_and_licence_report_as_billed
FAILED tests/test_provider_billed_input.py::test_estimated_input_is_reported_as_estimated
5 failed, 8 passed

$ ruff check <changed files>        -> All checks passed!
$ mypy <changed proxy/telemetry modules> -> Success: no issues found in 6 source files
```

## Real Behavior Proof

- Environment: local proxy from this branch (`headroom proxy --port 8931
--stateless --no-telemetry --no-cache`) against the real Anthropic API.
For the baseline, the same command ran from a `main` worktree on port
8932.
- Exact command / steps: 3 streamed `claude-sonnet-4-6` turns through
the proxy, with a cached ~20K-token system prompt, one tool and a
300-item JSON tool result. Anthropic's `message_start` usage (input +
cache read + cache write) was summed from each stream and compared with
`GET /stats` → `tokens`.
- Observed result: the author-provided live Anthropic comparison below
reports exact billed-input agreement at 61,999 tokens; the baseline
undercounts by 37%. This external run was not independently repeated
during the current-head review.

  | | Anthropic billed input | Headroom `/stats` `tokens.input` |
  |---|---|---|
  | `main` | 62,001 | **38,835** (−37%) |
| this PR | 61,999 | **61,999** (exact),
`input_provider_reported_percent: 100.0` |

The two runs differ by 2 tokens because they are separate API calls.
`proxy_compression_saved` was 29,313 on both, so savings behaviour is
unchanged.
- Not tested: live Bedrock, Vertex and Gemini streams (unit-tested on
provider shapes only), and the licence cloud endpoint receiving the new
fields.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is accounting only, and request
and response bytes are unchanged.
- Minimum rollout channel: N/A
- Stable/default behavior changed: reported input volume now uses the
provider's billed count where the provider reports one. New fields are
added to `/stats` and the licence usage payload; no existing field is
removed or renamed.
- Kill switch / disable path: none needed. With no usage frame, the
fallback is the previous estimate, now labelled `input_estimated`.
- Unsafe override required: no
- Qualification impact: dashboards and licence reports will show larger,
correct Claude input totals once installs upgrade, so historical and new
figures are not directly comparable.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes: targeted
suites only; full suite on CI
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- **Full suite.** Locally, the full run on the pre-rebase base reached
2,147 passed before stopping on
`tests/test_cli/test_mcp_reconcile.py::test_ordinary_install_does_not_adopt_serena`.
That test fails identically on unmodified `main` (unrelated MCP install
args). The full local run was then abandoned because several
wrap/install tests launch a real `serena project index` (rust-analyzer,
tsserver, rustc), which should be stubbed in a separate PR.
- **Docs.** The new `/stats` and licence fields are not yet documented.
- Follow-ups not in this PR:
- **Raw Bedrock invoke pass-through (`handlers/bedrock.py`)** forwards
bytes without parsing them, so it still records an estimate. It is now
counted as `input_estimated`, so it's visible rather than silent.
- **Batch create** has no usage until results come back, so it stays
estimated as above.
- **`tokens_saved` is still on the `o200k_base` scale**, so it
under-reports Claude savings. Calibrating it to provider tokens is a
separate PR.
  - **The licence cloud endpoint** needs to read the new fields.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


## Current maintainer verification (October 8)

Exact head: `81527148ed9777e1b9ca921bbf6bb457d49354db`, containing main
`01d9194854233ef6ed04005cf0a5b8c32bf65d6d` through a clean merge. No
contributor logic was modified. 63 focused tests pass with the existing
compatible native extension on Windows/Python 3.12. Three
backend-boundary probes preserve zero client input and exact cache-read,
cache-write, and combined provider input totals. This is local
mocked-backend/TestClient evidence; the author-provided
external-provider evidence above was not independently rerun.
Current-head hosted snapshot: 31 successful checks, 12 skips, no failed
or live checks.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Tejas Chopra
2026-10-09 14:16:49 -07:00
committed by GitHub
co-authored by Claude Opus 5.5 JerrettDavis
parent 01d9194854
commit a9815bbb5f
14 changed files with 805 additions and 8 deletions
+20 -1
View File
@@ -900,6 +900,12 @@ class CostTracker:
self._tool_saved_write_1h_by_model: dict[str, float] = {}
self._tool_saved_list_by_model: dict[str, float] = {}
self._tokens_sent_by_model: dict[str, int] = {}
# Of _tokens_sent_by_model: the part that is the provider's own billed
# count (the rest is Headroom's tokenizer estimate), and how many
# requests it came from. The licence usage report sends both so the
# cloud side can tell a billed figure from an estimate.
self._provider_tokens_sent_by_model: dict[str, int] = {}
self._provider_requests_by_model: dict[str, int] = {}
# Completion tokens keyed by ``(model, long_context)`` — same reason as
# the savings buckets: the >200k completion rate is a different number.
self._output_tokens_by_tier: dict[tuple[str, bool], int] = {}
@@ -930,6 +936,8 @@ class CostTracker:
self._tool_saved_write_1h_by_model.clear()
self._tool_saved_list_by_model.clear()
self._tokens_sent_by_model.clear()
self._provider_tokens_sent_by_model.clear()
self._provider_requests_by_model.clear()
self._output_tokens_by_tier.clear()
self._requests_by_model.clear()
self._api_cache_read_by_model.clear()
@@ -1023,13 +1031,15 @@ class CostTracker:
output_tokens: int = 0,
cache_inferred: bool = False,
tool_schema_saved: int = 0,
provider_reported: bool = False,
):
"""Record token counts per model and accumulate request cost for budget enforcement.
Args:
model: Model name.
tokens_saved: Tokens removed by compression (Headroom's count).
tokens_sent: Compressed message tokens sent (Headroom's count).
tokens_sent: Input tokens sent: the provider's billed count when
``provider_reported``, else Headroom's local estimate.
cache_read_tokens: Cache read tokens from API response usage.
cache_write_tokens: Cache write tokens from API response usage.
uncached_tokens: Non-cached input tokens from API response usage.
@@ -1047,6 +1057,8 @@ class CostTracker:
attributed. The dashboard's per-model "Tokens Saved" column
therefore showed compression only, while the headline above it
counted both.
provider_reported: True when ``tokens_sent`` is the provider's own
billed input (uncached + cache read + cache write).
"""
# Post-guard invariant (all providers): Headroom never forwards a request
# larger than the original (handlers revert any inflation before sending),
@@ -1134,6 +1146,13 @@ class CostTracker:
self._tool_saved_list_by_model.get(model, 0.0) + list_part
)
self._tokens_sent_by_model[model] = self._tokens_sent_by_model.get(model, 0) + tokens_sent
if provider_reported:
self._provider_tokens_sent_by_model[model] = (
self._provider_tokens_sent_by_model.get(model, 0) + tokens_sent
)
self._provider_requests_by_model[model] = (
self._provider_requests_by_model.get(model, 0) + 1
)
okey = (model, long_context)
self._output_tokens_by_tier[okey] = self._output_tokens_by_tier.get(okey, 0) + max(
0, output_tokens
+11
View File
@@ -565,6 +565,8 @@ class GeminiHandlerMixin:
status_code=response.status_code,
original_tokens=total_input_tokens,
optimized_tokens=total_input_tokens,
# Gemini's own promptTokenCount (inclusive of cached content).
provider_input_tokens=total_input_tokens,
output_tokens=output_tokens,
tokens_saved=0,
attempted_input_tokens=total_input_tokens,
@@ -872,6 +874,9 @@ class GeminiHandlerMixin:
total_latency = (time.time() - start_time) * 1000
total_input_tokens = optimized_tokens # fallback
# Gemini's own promptTokenCount, only when it reported one; the
# fallback above is Headroom's estimate and must not pass as billed.
provider_prompt_tokens = 0
output_tokens = 0
cache_read_tokens = 0
resp_json = None
@@ -890,6 +895,8 @@ class GeminiHandlerMixin:
if usage.get("promptTokenCount") is None
else usage["promptTokenCount"]
)
if usage.get("promptTokenCount") is not None:
provider_prompt_tokens = total_input_tokens
output_tokens = gemini_output_tokens(
usage
) # includes thinking tokens (2.5-family)
@@ -988,6 +995,8 @@ class GeminiHandlerMixin:
total_input_tokens = _usage_int(
usage.get("promptTokenCount"), total_input_tokens
)
if usage.get("promptTokenCount") is not None:
provider_prompt_tokens = total_input_tokens
output_tokens = _usage_int(usage.get("candidatesTokenCount"), output_tokens)
cache_read_tokens = _usage_int(
usage.get("cachedContentTokenCount"), cache_read_tokens
@@ -1048,6 +1057,8 @@ class GeminiHandlerMixin:
status_code=response.status_code,
original_tokens=effective_original_tokens,
optimized_tokens=total_input_tokens,
# Gemini's own promptTokenCount (inclusive of cached content).
provider_input_tokens=provider_prompt_tokens,
output_tokens=output_tokens,
tokens_saved=tokens_saved,
attempted_input_tokens=total_input_tokens + tokens_saved,
+23 -4
View File
@@ -37,6 +37,7 @@ from headroom.proxy.helpers import (
from headroom.proxy.identity import resolve_memory_identity
from headroom.proxy.loopback_guard import is_loopback_host
from headroom.proxy.modes import is_cache_mode
from headroom.proxy.provider_usage import billed_input_for_provider, billed_input_from_usage
from headroom.proxy.rate_limit_identity import rate_limit_identity
from headroom.proxy.semantic_cache_key_policy import compute_request_cache_partition
from headroom.proxy.stage_timer import StageTimer, emit_stage_timings_log
@@ -584,7 +585,7 @@ def _usage_int(value: Any) -> int:
return 0
def _passthrough_usage_from_json(payload: Any) -> dict[str, int]:
def _passthrough_usage_from_json(payload: Any, provider: str | None = None) -> dict[str, int]:
"""Normalize usage from pass-through provider response shapes."""
if not isinstance(payload, dict):
return {}
@@ -608,7 +609,11 @@ def _passthrough_usage_from_json(payload: Any) -> dict[str, int]:
details = usage.get("prompt_tokens_details") or usage.get("input_tokens_details") or {}
cache_read = details.get("cached_tokens") if isinstance(details, dict) else None
return {
"input_tokens": _usage_int(input_tokens),
# The billed total. Anthropic-shape usage reports input_tokens as
# the uncached tail only (cache buckets are disjoint); OpenAI's
# headline figure is already inclusive. Callers derive uncached as
# total - read - write, which is only right on the total.
"input_tokens": billed_input_from_usage(usage, provider) or _usage_int(input_tokens),
"output_tokens": _usage_int(output_tokens),
"cache_read_input_tokens": _usage_int(usage.get("cache_read_input_tokens", cache_read)),
"cache_creation_input_tokens": _usage_int(usage.get("cache_creation_input_tokens")),
@@ -9453,6 +9458,8 @@ class OpenAIHandlerMixin:
model=model_for_metrics,
original_tokens=max(0, input_delta) + max(0, saved_delta),
optimized_tokens=max(0, input_delta),
# Summed from OpenAI's own response.completed usage frames.
provider_input_tokens=max(0, input_delta),
output_tokens=max(0, output_delta),
tokens_saved=max(0, saved_delta),
attempted_input_tokens=max(0, attempted_delta),
@@ -10121,6 +10128,8 @@ class OpenAIHandlerMixin:
model=model_name,
original_tokens=residual_input_tokens + residual_tokens_saved,
optimized_tokens=residual_input_tokens,
# Summed from OpenAI's own response.completed usage frames.
provider_input_tokens=residual_input_tokens,
output_tokens=residual_output_tokens,
tokens_saved=residual_tokens_saved,
attempted_input_tokens=residual_attempted_input_tokens,
@@ -11757,7 +11766,7 @@ class OpenAIHandlerMixin:
usage: dict[str, int] = {}
if response.headers.get("content-type", "").lower().startswith("application/json"):
try:
usage = _passthrough_usage_from_json(response.json())
usage = _passthrough_usage_from_json(response.json(), provider)
except (json.JSONDecodeError, ValueError, TypeError):
usage = {}
input_tokens = usage.get("input_tokens", 0)
@@ -11773,6 +11782,7 @@ class OpenAIHandlerMixin:
status_code=response.status_code,
original_tokens=input_tokens,
optimized_tokens=input_tokens,
provider_input_tokens=input_tokens,
output_tokens=output_tokens,
tokens_saved=0,
attempted_input_tokens=input_tokens,
@@ -11938,10 +11948,18 @@ class OpenAIHandlerMixin:
buf.extend(b"\n\n")
_absorb_usage(self._parse_sse_usage_from_buffer(stream_state, stream_provider))
input_tokens = stream_state["input_tokens"] or 0
output_tokens = stream_state["output_tokens"] or 0
cache_read_tokens = stream_state["cache_read_input_tokens"] or 0
cache_write_tokens = stream_state["cache_creation_input_tokens"] or 0
# Anthropic-shape SSE usage reports only the uncached tail as
# input_tokens; the billed total adds the disjoint cache buckets.
# Gemini's promptTokenCount is already inclusive.
input_tokens = billed_input_for_provider(
stream_provider,
stream_state["input_tokens"],
cache_read=cache_read_tokens,
cache_write=cache_write_tokens,
)
uncached_input_tokens = max(
0,
input_tokens - cache_read_tokens - cache_write_tokens,
@@ -11953,6 +11971,7 @@ class OpenAIHandlerMixin:
model=_passthrough_model_from_path(path, endpoint_name),
original_tokens=input_tokens,
optimized_tokens=input_tokens,
provider_input_tokens=input_tokens,
output_tokens=output_tokens,
tokens_saved=0,
attempted_input_tokens=input_tokens,
+59 -1
View File
@@ -22,6 +22,7 @@ from headroom.proxy.helpers import (
overload_retry_is_futile,
retry_after_ms,
)
from headroom.proxy.provider_usage import billed_input_for_provider
from headroom.proxy.token_counting import gemini_output_tokens
if TYPE_CHECKING:
@@ -1001,6 +1002,24 @@ class StreamingMixin:
effective_optimized_tokens - cache_read_tokens - cache_write_tokens, 0
)
# The provider's own billed input for this turn. Without it the outcome
# falls back to ``optimized_tokens`` — Headroom's tokenizer estimate of
# the forwarded messages only (no system prompt, no tool definitions) —
# and every volume/cost total downstream (dashboard, licence usage
# report) silently reports the estimate as if it were the bill. Every
# Claude Code turn is a streamed Anthropic turn, so this was the bulk of
# Claude traffic.
billed_provider_input = (
billed_input_for_provider(
provider,
provider_input_tokens,
cache_read=cache_read_tokens,
cache_write=cache_write_tokens,
)
if isinstance(provider_input_tokens, int)
else 0
)
# Prefix-tracker mutation is provider-specific state that lives
# outside the metric funnel. Run it before the funnel so the next
# request inherits correct prefix state regardless of metric path.
@@ -1100,6 +1119,7 @@ class StreamingMixin:
request_id=request_id,
original_tokens=effective_original_tokens,
optimized_tokens=effective_optimized_tokens,
provider_input_tokens=billed_provider_input,
output_tokens=output_tokens,
tokens_saved=tokens_saved,
transforms_applied=transforms_applied,
@@ -2189,8 +2209,22 @@ class StreamingMixin:
# already reported by the backend is preserved untouched.
if event.event_type == "message_start" and not event.raw_sse:
msg_usage = event.data.setdefault("message", {}).setdefault("usage", {})
if not msg_usage.get("input_tokens") and optimized_tokens > 0:
# A zero with cache usage beside it is a real count: a
# fully cached prompt bills only the cache buckets. Only
# a bare zero is LiteLLM's unknown placeholder.
_cached = any(
isinstance(msg_usage.get(k), int) and msg_usage.get(k) > 0
for k in ("cache_read_input_tokens", "cache_creation_input_tokens")
)
if (
not msg_usage.get("input_tokens")
and not _cached
and optimized_tokens > 0
):
msg_usage["input_tokens"] = optimized_tokens
# The client-bound backfill is Headroom's estimate,
# not backend usage: never report it as billed.
stream_state["input_tokens_backfilled"] = True
# Format as SSE
if event.raw_sse:
@@ -2208,6 +2242,8 @@ class StreamingMixin:
usage = msg.get("usage", {})
if "input_tokens" in usage:
stream_state["input_tokens"] = usage["input_tokens"]
if not stream_state.get("input_tokens_backfilled"):
stream_state["backend_input_tokens"] = usage["input_tokens"]
stream_state["cache_read_input_tokens"] = usage.get(
"cache_read_input_tokens", 0
)
@@ -2225,6 +2261,7 @@ class StreamingMixin:
stream_state["output_tokens"] = usage["output_tokens"]
if "input_tokens" in usage:
stream_state["input_tokens"] = usage["input_tokens"]
stream_state["backend_input_tokens"] = usage["input_tokens"]
if "cache_read_input_tokens" in usage:
stream_state["cache_read_input_tokens"] = usage[
"cache_read_input_tokens"
@@ -2309,6 +2346,11 @@ class StreamingMixin:
# ``from_stream`` as ``optimized + saved``. Bedrock
# doesn't propagate frozen_message_count either — same
# fallback as the SSE finalizer (#455).
# Bedrock forwards Anthropic-shape usage: input_tokens is the
# uncached tail only, cache buckets are disjoint. Only a count
# the backend itself reported qualifies; the message_start
# backfill above is Headroom's estimate.
_bedrock_input = stream_state.get("backend_input_tokens")
outcome = RequestOutcome.from_stream(
body=body,
provider=_backend_name,
@@ -2316,6 +2358,16 @@ class StreamingMixin:
request_id=request_id,
original_tokens=original_tokens,
optimized_tokens=optimized_tokens,
provider_input_tokens=(
billed_input_for_provider(
"anthropic",
_bedrock_input,
cache_read=stream_state["cache_read_input_tokens"],
cache_write=stream_state["cache_creation_input_tokens"],
)
if isinstance(_bedrock_input, int)
else 0
),
output_tokens=stream_state["output_tokens"],
tokens_saved=tokens_saved,
transforms_applied=transforms_applied,
@@ -2543,6 +2595,12 @@ class StreamingMixin:
request_id=request_id,
original_tokens=original_tokens,
optimized_tokens=optimized_tokens,
# OpenAI-shape usage from the backend: the headline figure
# already includes cached tokens (cache_write above is
# subtracted from it, not added).
provider_input_tokens=(
upstream_input if isinstance(upstream_input, int) else 0
),
output_tokens=output_tokens,
tokens_saved=tokens_saved,
transforms_applied=transforms_applied,
+30
View File
@@ -337,6 +337,7 @@ class RequestOutcome:
original_messages: list[dict] | None = None,
conversation_key: str | None = None,
conversation_tokens_saved: int | None = None,
provider_input_tokens: int = 0,
) -> RequestOutcome:
"""Construct an outcome from the locals available at streaming
finalize. Three streaming finalizers
@@ -407,6 +408,7 @@ class RequestOutcome:
model=model,
original_tokens=original_tokens,
optimized_tokens=optimized_tokens,
provider_input_tokens=max(int(provider_input_tokens or 0), 0),
output_tokens=output_tokens,
tokens_saved=tokens_saved,
conversation_key=conversation_key,
@@ -441,6 +443,29 @@ class RequestOutcome:
# ── The funnel ───────────────────────────────────────────────────────
_estimated_input_warned: set[tuple[str, str]] = set()
def _warn_estimated_input(provider: str, model: str) -> None:
"""Log once per (provider, model) when a request's input volume is an estimate.
Every successful response from Anthropic, OpenAI and Gemini carries usage,
so landing here on those providers means a code path dropped it. The
request still records (fail open), but the volume it contributes is
Headroom's tokenizer estimate, not the bill, and the coverage counters say so.
"""
key = (str(provider), str(model))
if key in _estimated_input_warned or len(_estimated_input_warned) >= 256:
return
_estimated_input_warned.add(key)
logger.warning(
"event=input_tokens_estimated provider=%s model=%s: no provider usage on this "
"request; input volume recorded from Headroom's local tokenizer estimate",
provider,
model,
)
async def emit_request_outcome(handler: Any, outcome: RequestOutcome) -> None:
"""Single funnel for per-request bookkeeping. The contract.
@@ -652,6 +677,9 @@ async def emit_request_outcome(handler: Any, outcome: RequestOutcome) -> None:
# Deliberately NOT used for any delta: differencing this against
# ``original_tokens`` mixes tokenizer scales. See the field docs.
billed_input_tokens = outcome.provider_input_tokens or outcome.optimized_tokens
input_provider_reported = outcome.provider_input_tokens > 0
if not input_provider_reported and billed_input_tokens > 0:
_warn_estimated_input(outcome.provider, outcome.model)
# 1. Prometheus / SavingsTracker.
await handler.metrics.record_request(
@@ -684,6 +712,7 @@ async def emit_request_outcome(handler: Any, outcome: RequestOutcome) -> None:
# so counting it as a write would both double it and apply a premium
# OpenAI never charges.
cache_inferred=outcome.cache_inferred,
input_provider_reported=input_provider_reported,
)
# 2. Cost tracker (optional).
@@ -704,6 +733,7 @@ async def emit_request_outcome(handler: Any, outcome: RequestOutcome) -> None:
# cost tracker feeds the dashboard's per-model table, which read
# compression only while its own headline counted both layers.
tool_schema_saved=tool_search_saved,
provider_reported=input_provider_reported,
)
# 3. Per-request log (optional). The ``client`` outcome field is
+22
View File
@@ -165,6 +165,14 @@ class PrometheusMetrics:
self.inbound_responses_by_status: dict[str, int] = defaultdict(int)
self.tokens_input_total = 0
# Of tokens_input_total: how much is the provider's own billed count
# versus Headroom's local tokenizer estimate (provider reported no
# usage, or the path never parsed it). Reported beside every input
# total so nobody reads an estimate as the bill.
self.tokens_input_provider_reported_total = 0
self.tokens_input_estimated_total = 0
self.requests_input_provider_reported = 0
self.requests_input_estimated = 0
self.tokens_output_total = 0
self.tokens_saved_total = 0
# Tool-schema savings (deferral + turn-hook tool shrink), aggregated from
@@ -427,6 +435,10 @@ class PrometheusMetrics:
self.inbound_responses_by_status.clear()
self.tokens_input_total = 0
self.tokens_input_provider_reported_total = 0
self.tokens_input_estimated_total = 0
self.requests_input_provider_reported = 0
self.requests_input_estimated = 0
self.tokens_output_total = 0
self.tokens_saved_total = 0
self.tool_search_saved_total = 0
@@ -854,6 +866,10 @@ class PrometheusMetrics:
# ``CacheMix.normalized``. Defaults False, preserving behaviour for the
# providers that report disjoint buckets.
cache_inferred: bool = False,
# True when ``input_tokens`` is the provider's own billed count, False
# when it is Headroom's local estimate. None (callers predating the
# split) records nothing in either bucket.
input_provider_reported: bool | None = None,
):
"""Record metrics for a request.
@@ -931,6 +947,12 @@ class PrometheusMetrics:
self.requests_cached += 1
self.tokens_input_total += input_tokens
if input_provider_reported is True:
self.tokens_input_provider_reported_total += input_tokens
self.requests_input_provider_reported += 1
elif input_provider_reported is False and input_tokens > 0:
self.tokens_input_estimated_total += input_tokens
self.requests_input_estimated += 1
self.tokens_output_total += output_tokens
self.tokens_saved_total += tokens_saved
self.tool_search_saved_total += max(0, int(tool_search_saved))
+130
View File
@@ -0,0 +1,130 @@
"""The provider's own billed input-token count, from its own usage block.
Every volume and cost figure Headroom reports ("input after Headroom", the
licence usage report, the dashboard's per-model table) should be the number
the provider billed, not Headroom's local tokenizer estimate. Headroom counts
Claude with ``o200k_base`` as a stand-in for a private tokenizer; on real
Claude Code transcripts that runs 15-25% under Claude 4.x and 35-38% under
Claude 5.x, and it never sees the system prompt or tool definitions at all.
The providers disagree about what their headline input field means, which is
why this lives in one place:
* Anthropic (and Bedrock/Vertex, which forward Anthropic-shape usage):
``usage.input_tokens`` is only the tokens AFTER the last cache breakpoint.
Cache reads and writes are separate, disjoint buckets, so the billed total
is ``input_tokens + cache_read_input_tokens + cache_creation_input_tokens``.
The Anthropic Console reports the same three columns.
* OpenAI (chat ``prompt_tokens``, Responses ``input_tokens``): inclusive.
``*_tokens_details.cached_tokens`` is a SUBSET of the headline figure.
* Gemini: ``promptTokenCount`` is inclusive; ``cachedContentTokenCount`` is a
subset of it.
A return of 0 means "the provider reported nothing", never "zero tokens":
callers fall back to the local estimate and the request is counted as
estimated rather than provider-reported (see ``RequestOutcome``).
"""
from __future__ import annotations
from collections.abc import Mapping
from typing import Any
__all__ = [
"anthropic_billed_input",
"is_anthropic_dialect",
"billed_input_from_usage",
"billed_input_for_provider",
]
# Providers whose usage blocks are Anthropic-shaped (disjoint cache buckets).
_ANTHROPIC_SHAPE = frozenset({"anthropic", "bedrock", "vertex", "vertex_ai", "claude"})
def _int(value: Any) -> int:
if isinstance(value, bool):
return 0
try:
return max(int(value), 0)
except (TypeError, ValueError):
return 0
def is_anthropic_dialect(provider: str | None) -> bool:
"""True for providers whose usage blocks keep cache buckets disjoint.
Decided by the provider, never by which keys a usage block happens to
carry: OpenAI-compatible gateways (LiteLLM among them) mirror top-level
``cache_read_input_tokens`` beside an INCLUSIVE ``input_tokens``, so the
keys alone cannot tell the two dialects apart.
"""
name = (provider or "").lower()
return name in _ANTHROPIC_SHAPE or "anthropic" in name
def anthropic_billed_input(input_tokens: Any, cache_read: Any = 0, cache_write: Any = 0) -> int:
"""Billed input for an Anthropic-shape usage block: the three disjoint buckets."""
return _int(input_tokens) + _int(cache_read) + _int(cache_write)
def billed_input_for_provider(
provider: str | None,
input_tokens: Any,
*,
cache_read: Any = 0,
cache_write: Any = 0,
) -> int:
"""Billed input from already-extracted usage fields, by provider family.
``input_tokens`` is the provider's headline input field exactly as it
reported it (Anthropic's uncached-only ``input_tokens``, OpenAI's inclusive
``prompt_tokens``/``input_tokens``, Gemini's inclusive ``promptTokenCount``).
Returns 0 when the provider reported no input count at all.
"""
if input_tokens is None:
return 0
if is_anthropic_dialect(provider):
total = anthropic_billed_input(input_tokens, cache_read, cache_write)
# message_start can carry input_tokens=0 alongside real cache buckets
# on a fully cached turn; that is still a provider-reported count.
return total
return _int(input_tokens)
def billed_input_from_usage(payload: Mapping[str, Any] | None, provider: str | None) -> int:
"""Billed input from a raw provider response (or its ``usage`` block).
``provider`` selects the dialect (see ``is_anthropic_dialect``); it is
required because the same keys mean different things across dialects.
Gemini's ``usageMetadata`` is unambiguous and recognised by shape. Returns
0 when no input count is present.
"""
if not isinstance(payload, Mapping):
return 0
meta = payload.get("usageMetadata")
if isinstance(meta, Mapping):
return _int(meta.get("promptTokenCount"))
usage = payload.get("usage") if isinstance(payload.get("usage"), Mapping) else payload
if not isinstance(usage, Mapping):
return 0
if "promptTokenCount" in usage:
return _int(usage.get("promptTokenCount"))
if is_anthropic_dialect(provider):
if "input_tokens" not in usage:
return 0
return anthropic_billed_input(
usage.get("input_tokens"),
usage.get("cache_read_input_tokens"),
usage.get("cache_creation_input_tokens"),
)
# OpenAI and OpenAI-compatible dialects: the headline figure is inclusive.
if "prompt_tokens" in usage:
return _int(usage.get("prompt_tokens"))
if "input_tokens" in usage:
return _int(usage.get("input_tokens"))
return 0
+13
View File
@@ -5150,6 +5150,19 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
},
"tokens": {
"input": m.tokens_input_total,
# Provenance of "input": the provider's own billed count versus
# Headroom's tokenizer estimate. Only the first reconciles with
# the provider console (input + cache read + cache write).
"input_provider_reported": m.tokens_input_provider_reported_total,
"input_estimated": m.tokens_input_estimated_total,
"input_provider_reported_percent": round(
(m.tokens_input_provider_reported_total / m.tokens_input_total * 100)
if m.tokens_input_total > 0
else 0,
2,
),
"requests_input_provider_reported": m.requests_input_provider_reported,
"requests_input_estimated": m.requests_input_estimated,
"output": m.tokens_output_total,
"output_saved": output_reduction.get("tokens_saved", 0),
"output_reduction_percent": output_reduction.get("reduction_percent", 0),
+46
View File
@@ -118,6 +118,7 @@ class UsageReporter:
self._last_tokens_saved_by_model: dict[str, int] = {}
self._last_tokens_sent_by_model: dict[str, int] = {}
self._last_requests_by_model: dict[str, int] = {}
self._last_provider_counters: dict[str, dict[str, int]] = {}
async def validate_license(self) -> LicenseInfo:
"""Validate the license key against the cloud API.
@@ -301,6 +302,13 @@ class UsageReporter:
self._last_report_time = now
return
# Provenance of tokens_after: how much of it is the provider's own
# billed count versus Headroom's tokenizer estimate, plus the billed
# cache split. Lets the cloud side show a number that reconciles with
# the provider's console (input + cache read + cache write) and flag
# any remainder as an estimate instead of passing it off as the bill.
provider_deltas = self._provider_counter_deltas(cost_tracker)
payload = {
"license_key": self._license_key,
"period_start": period_start.isoformat(),
@@ -310,6 +318,18 @@ class UsageReporter:
"tokens_after": total_tokens_after,
"tokens_saved": total_tokens_saved,
"models": delta_reqs_by_model,
"tokens_after_provider_reported": provider_deltas["tokens_after_provider_reported"],
"tokens_after_estimated": max(
0, total_tokens_after - provider_deltas["tokens_after_provider_reported"]
),
"requests_provider_reported": provider_deltas["requests_provider_reported"],
"cache_read_tokens": provider_deltas["cache_read_tokens"],
"cache_write_tokens": provider_deltas["cache_write_tokens"],
"uncached_input_tokens": provider_deltas["uncached_input_tokens"],
# Units, stated so no consumer has to guess: tokens_after is billed
# input where provider-reported; tokens_saved is Headroom's local
# tokenizer count of what compression removed.
"tokens_saved_basis": "headroom_tokenizer",
}
try:
@@ -348,6 +368,31 @@ class UsageReporter:
except Exception:
logger.warning("Failed to send usage report", exc_info=True)
_PROVIDER_COUNTERS = {
"tokens_after_provider_reported": "_provider_tokens_sent_by_model",
"requests_provider_reported": "_provider_requests_by_model",
"cache_read_tokens": "_api_cache_read_by_model",
"cache_write_tokens": "_api_cache_write_by_model",
"uncached_input_tokens": "_api_uncached_by_model",
}
def _current_provider_counters(self, cost_tracker: Any) -> dict[str, dict[str, int]]:
return {
name: dict(getattr(cost_tracker, attr, {}) or {})
for name, attr in self._PROVIDER_COUNTERS.items()
}
def _provider_counter_deltas(self, cost_tracker: Any) -> dict[str, int]:
"""Totals of the provenance counters since the last successful report."""
current = self._current_provider_counters(cost_tracker)
deltas: dict[str, int] = {}
for name, by_model in current.items():
last = getattr(self, "_last_provider_counters", {}).get(name, {})
deltas[name] = sum(
max(0, int(value) - int(last.get(model, 0))) for model, value in by_model.items()
)
return deltas
def _snapshot_metrics(self) -> None:
"""Take a snapshot of current proxy metrics for delta computation."""
if self._proxy is None or self._proxy.cost_tracker is None:
@@ -356,6 +401,7 @@ class UsageReporter:
self._last_tokens_saved_by_model = dict(ct._tokens_saved_by_model)
self._last_tokens_sent_by_model = dict(ct._tokens_sent_by_model)
self._last_requests_by_model = dict(ct._requests_by_model)
self._last_provider_counters = self._current_provider_counters(ct)
self._last_report_time = datetime.now(timezone.utc)
def _save_cache(self) -> None:
+27
View File
@@ -0,0 +1,27 @@
event: message_start
data: {"type":"message_start","message":{"model":"claude-sonnet-4-6","id":"msg_011CfqCxQgQFZwvbJZt3ZtS6","type":"message","role":"assistant","content":[],"container":null,"stop_reason":null,"stop_sequence":null,"stop_details":null,"usage":{"input_tokens":19,"cache_creation_input_tokens":0,"cache_read_input_tokens":8857,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":0},"output_tokens":5,"service_tier":"standard","inference_geo":"global"},"diagnostics":null} }
event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""} }
event: ping
data: {"type": "ping"}
event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Rerun job j"} }
event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"17 with"} }
event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" --retry."} }
event: content_block_stop
data: {"type":"content_block_stop","index":0 }
event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null,"stop_details":null,"container":null},"usage":{"input_tokens":19,"cache_creation_input_tokens":0,"cache_read_input_tokens":8857,"output_tokens":14} }
event: message_stop
data: {"type":"message_stop" }
+63 -2
View File
@@ -52,7 +52,7 @@ def _make_bedrock_backend(events: list[StreamEvent]) -> MagicMock:
return mock
def _bedrock_events(input_tokens: int) -> list[StreamEvent]:
def _bedrock_events(input_tokens: int, **cache: int) -> list[StreamEvent]:
"""Build a minimal Anthropic streaming sequence as LiteLLM emits it."""
message_start = {
"type": "message_start",
@@ -63,7 +63,7 @@ def _bedrock_events(input_tokens: int) -> list[StreamEvent]:
"type": "message",
"content": [],
# LiteLLM hardcodes this to 0 — the bug under test.
"usage": {"input_tokens": input_tokens, "output_tokens": 0},
"usage": {"input_tokens": input_tokens, "output_tokens": 0, **cache},
},
}
block_start = {
@@ -150,3 +150,64 @@ def test_bedrock_streaming_preserves_nonzero_upstream_input_tokens() -> None:
upstream_input_tokens = 777
body = _post_stream(_make_bedrock_backend(_bedrock_events(input_tokens=upstream_input_tokens)))
assert _message_start_input_tokens(body) == upstream_input_tokens
def _post_stream_stats(backend: MagicMock) -> dict:
"""Run one streamed turn and return /stats tokens (provenance counters)."""
config = ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
backend="anyllm",
anyllm_provider="anthropic",
)
with patch("headroom.proxy.server.AnyLLMBackend", return_value=backend):
app = create_app(config)
with TestClient(app) as client:
resp = client.post(
"/v1/messages",
json={
"model": "claude-3-5-sonnet-20241022",
"messages": [{"role": "user", "content": "hello there general"}],
"max_tokens": 64,
"stream": True,
},
headers={"x-api-key": "sk-ant-test", "anthropic-version": "2023-06-01"},
)
assert resp.status_code == 200, resp.text[:200]
return client.get("/stats").json()["tokens"]
def test_bedrock_backfilled_input_is_reported_as_estimated() -> None:
"""The #1132 backfill is Headroom's estimate: it must not count as billed input."""
tokens = _post_stream_stats(_make_bedrock_backend(_bedrock_events(input_tokens=0)))
assert tokens["input"] > 0
assert tokens["input_provider_reported"] == 0
assert tokens["input_estimated"] == tokens["input"]
def test_bedrock_reported_input_is_provider_reported() -> None:
tokens = _post_stream_stats(_make_bedrock_backend(_bedrock_events(input_tokens=777)))
assert tokens["input_provider_reported"] == 777
assert tokens["input_estimated"] == 0
def test_bedrock_fully_cached_zero_is_billed_input_not_a_placeholder() -> None:
"""input_tokens=0 beside a cache bucket is a real count (a fully cached
prompt), not LiteLLM's placeholder: it is not backfilled, and the cache
buckets count as provider-reported billed input."""
events = _bedrock_events(input_tokens=0, cache_read_input_tokens=8857)
assert _message_start_input_tokens(_post_stream(_make_bedrock_backend(events))) == 0
tokens = _post_stream_stats(
_make_bedrock_backend(_bedrock_events(input_tokens=0, cache_read_input_tokens=8857))
)
assert tokens["input_provider_reported"] == 8857
assert tokens["input_estimated"] == 0
def test_bedrock_cache_write_alone_also_counts_as_reported() -> None:
tokens = _post_stream_stats(
_make_bedrock_backend(_bedrock_events(input_tokens=0, cache_creation_input_tokens=1200))
)
assert tokens["input_provider_reported"] == 1200
assert tokens["input_estimated"] == 0
@@ -136,3 +136,64 @@ async def test_ccr_continuation_present_null_usage_does_not_502(
# The pre-continuation count (100) survives as the fallback.
assert handler.outcomes[0].optimized_tokens == 100
assert handler.outcomes[0].cache_read_tokens == 0
class _NoUsageResponse:
"""A successful 200 whose usageMetadata carries no promptTokenCount."""
status_code = 200
content = json.dumps(
{"candidates": [{"content": {"parts": [{"text": "hi"}]}}], "usageMetadata": {}}
).encode()
headers = {"content-type": "application/json"}
def json(self) -> object:
return json.loads(self.content)
class _NoCcrHandler(_CcrHandler):
def has_ccr_tool_calls(self, resp_json, provider) -> bool: # noqa: ANN001
return False
@pytest.mark.asyncio
async def test_missing_prompt_token_count_is_not_reported_as_provider_input(
monkeypatch: pytest.MonkeyPatch,
) -> None:
async def payload(request): # noqa: ANN001, ANN201
return {"contents": [{"role": "user", "parts": [{"text": "hello"}]}]}
monkeypatch.setattr("headroom.proxy.helpers._read_request_json", payload)
handler = _Handler()
handler.ccr_response_handler = _NoCcrHandler()
async def no_usage(method, url, headers, body): # noqa: ANN001, ANN202
return _NoUsageResponse()
handler._retry_request = no_usage
response = await handler.handle_gemini_generate_content(_FakeRequest(), "gemini-pro")
assert response.status_code == 200
(outcome,) = handler.outcomes
# Volume still falls back to the local estimate...
assert outcome.optimized_tokens == 100
# ...but it must not be labelled as Gemini's own billed count.
assert outcome.provider_input_tokens == 0
@pytest.mark.asyncio
async def test_reported_prompt_token_count_is_provider_input(
monkeypatch: pytest.MonkeyPatch,
) -> None:
async def payload(request): # noqa: ANN001, ANN201
return {"contents": [{"role": "user", "parts": [{"text": "hello"}]}]}
monkeypatch.setattr("headroom.proxy.helpers._read_request_json", payload)
handler = _Handler()
handler.ccr_response_handler = _NoCcrHandler()
response = await handler.handle_gemini_generate_content(_FakeRequest(), "gemini-pro")
assert response.status_code == 200
assert handler.outcomes[0].provider_input_tokens == 100
+297
View File
@@ -0,0 +1,297 @@
"""Input volume must be the provider's own billed count, not Headroom's estimate.
Headroom's local tokenizer is a stand-in for Claude's private one (15-25% under
Claude 4.x, 35-38% under Claude 5.x on real transcripts) and only ever sees the
forwarded messages, never the system prompt or tool definitions. Every streamed
Anthropic turn (all Claude Code traffic) used to fall back to that estimate for
"input after Headroom", the dashboard input total and the licence usage report,
so none of them reconciled with the Anthropic Console.
The Anthropic fixture is a real recorded stream: a second call on a cached
~8.9K-token system prompt. Anthropic billed 19 uncached + 8,857 cache-read
tokens; the messages Headroom counts locally are a single short user turn.
"""
from __future__ import annotations
import asyncio
import json
from pathlib import Path
from types import SimpleNamespace
import anyio
import pytest
from headroom.proxy.cost import CostTracker
from headroom.proxy.handlers.openai import _passthrough_usage_from_json
from headroom.proxy.outcome import RequestOutcome, emit_request_outcome
from headroom.proxy.prometheus_metrics import PrometheusMetrics
from headroom.proxy.provider_usage import (
billed_input_for_provider,
billed_input_from_usage,
)
from headroom.proxy.server import HeadroomProxy
from headroom.telemetry.reporter import UsageReporter
FIXTURE = Path(__file__).parent / "fixtures" / "anthropic" / "stream_cached_turn.sse"
# From the fixture's message_start usage: input 19 + cache_read 8857 + cache_write 0.
ANTHROPIC_BILLED = 19 + 8_857
# What Headroom's tokenizer would see: one short user message.
LOCAL_ESTIMATE = 17
# ── provider shape semantics ──────────────────────────────────────────────
@pytest.mark.parametrize(
"provider,payload,expected",
[
# Anthropic: input_tokens is the uncached tail; cache buckets are disjoint.
(
"anthropic",
{
"usage": {
"input_tokens": 19,
"cache_read_input_tokens": 8_857,
"cache_creation_input_tokens": 100,
}
},
8_976,
),
# OpenAI chat: prompt_tokens already includes cached_tokens.
(
"openai",
{"usage": {"prompt_tokens": 5_000, "prompt_tokens_details": {"cached_tokens": 4_000}}},
5_000,
),
# OpenAI Responses: input_tokens already includes cached_tokens.
(
"openai",
{"usage": {"input_tokens": 5_000, "input_tokens_details": {"cached_tokens": 4_000}}},
5_000,
),
# Gemini: promptTokenCount includes cachedContentTokenCount.
(
"gemini",
{"usageMetadata": {"promptTokenCount": 7_000, "cachedContentTokenCount": 6_000}},
7_000,
),
# OpenAI-compatible gateways (LiteLLM) mirror top-level Anthropic-named
# cache keys beside an INCLUSIVE input_tokens. Keys alone must not flip
# the dialect, or the cached part is counted twice.
("openai", {"usage": {"input_tokens": 5_000, "cache_read_input_tokens": 4_000}}, 5_000),
(
"vertex:anthropic",
{"usage": {"input_tokens": 50, "cache_read_input_tokens": 4_000}},
4_050,
),
# Nothing reported: 0 means "unknown", never "zero tokens billed".
("anthropic", {"id": "x"}, 0),
("openai", None, 0),
],
)
def test_billed_input_from_usage_per_provider_shape(provider, payload, expected) -> None: # noqa: ANN001
assert billed_input_from_usage(payload, provider) == expected
def test_billed_input_for_provider_adds_anthropic_cache_buckets_only() -> None:
assert billed_input_for_provider("anthropic", 19, cache_read=8_857, cache_write=0) == 8_876
assert billed_input_for_provider("bedrock", 2, cache_read=360_949, cache_write=840) == 361_791
# OpenAI/Gemini headline figures are inclusive: never add the cache split.
assert billed_input_for_provider("openai", 5_000, cache_read=4_000) == 5_000
assert billed_input_for_provider("gemini", 7_000, cache_read=6_000) == 7_000
assert billed_input_for_provider("anthropic", None) == 0
def test_passthrough_normaliser_returns_anthropic_total_not_uncached_tail() -> None:
usage = _passthrough_usage_from_json(
{"usage": {"input_tokens": 19, "output_tokens": 14, "cache_read_input_tokens": 8_857}},
"anthropic",
)
# Callers derive uncached as input - read - write, which is only right on the total.
assert usage["input_tokens"] == ANTHROPIC_BILLED
assert usage["input_tokens"] - usage["cache_read_input_tokens"] == 19
# ── recorded Anthropic stream through the real SSE parser and finalizer ──
def _replay_recorded_stream(provider: str, raw: bytes, chunk_size: int = 97) -> dict:
"""Feed recorded SSE bytes through the proxy's own usage parser, in odd-sized
chunks so events split across reads the way they do on the wire."""
proxy = object.__new__(HeadroomProxy)
state = {
"input_tokens": None,
"output_tokens": None,
"cache_read_input_tokens": None,
"cache_creation_input_tokens": None,
"cache_creation_ephemeral_5m_input_tokens": None,
"cache_creation_ephemeral_1h_input_tokens": None,
"total_bytes": 0,
"sse_buffer": bytearray(),
"ttfb_ms": 3.0,
}
for i in range(0, len(raw), chunk_size):
chunk = raw[i : i + chunk_size]
state["total_bytes"] += len(chunk)
state["sse_buffer"].extend(chunk)
usage = proxy._parse_sse_usage_from_buffer(state, provider)
for key, value in (usage or {}).items():
if key in state:
state[key] = value
return state
def _finalize(provider: str, stream_state: dict, optimized_tokens: int) -> RequestOutcome:
handler = object.__new__(HeadroomProxy)
handler.config = SimpleNamespace(log_full_messages=False)
outcomes: list[RequestOutcome] = []
async def record(outcome): # noqa: ANN001, ANN202
outcomes.append(outcome)
handler._record_request_outcome = record
asyncio.run(
handler._finalize_stream_response(
body={"messages": [{"role": "user", "content": "Which job reruns target t17?"}]},
provider=provider,
model="claude-sonnet-4-6" if provider == "anthropic" else "gpt-5",
request_id=f"req_{provider}_billed",
original_tokens=optimized_tokens,
optimized_tokens=optimized_tokens,
tokens_saved=0,
transforms_applied=[],
optimization_latency=1.0,
stream_state=stream_state,
start_time=0.0,
)
)
(outcome,) = outcomes
return outcome
def test_recorded_anthropic_stream_records_anthropics_billed_input() -> None:
state = _replay_recorded_stream("anthropic", FIXTURE.read_bytes())
assert state["input_tokens"] == 19
assert state["cache_read_input_tokens"] == 8_857
outcome = _finalize("anthropic", state, optimized_tokens=LOCAL_ESTIMATE)
assert outcome.provider_input_tokens == ANTHROPIC_BILLED
assert outcome.uncached_input_tokens == 19
assert outcome.cache_read_tokens == 8_857
# The local estimate stays the local estimate: savings math must never mix
# it with the provider's scale.
assert outcome.optimized_tokens == LOCAL_ESTIMATE
def test_anthropic_stream_without_usage_stays_estimated() -> None:
# An error before message_start: no provider count exists, so the outcome
# must not invent one (provider_input_tokens 0 => counted as estimated).
state = _replay_recorded_stream("anthropic", b"")
outcome = _finalize("anthropic", state, optimized_tokens=LOCAL_ESTIMATE)
assert outcome.provider_input_tokens == 0
def test_openai_stream_records_inclusive_prompt_tokens() -> None:
frame = {
"id": "c1",
"choices": [],
"usage": {
"prompt_tokens": 12_000,
"completion_tokens": 30,
"prompt_tokens_details": {"cached_tokens": 11_000},
},
}
raw = f"data: {json.dumps(frame)}\n\ndata: [DONE]\n\n".encode()
state = _replay_recorded_stream("openai", raw)
outcome = _finalize("openai", state, optimized_tokens=9_000)
assert outcome.provider_input_tokens == 12_000
# ── the full chain: outcome -> metrics/cost tracker -> licence usage report ──
def _handler() -> SimpleNamespace:
cost = CostTracker()
return SimpleNamespace(
metrics=PrometheusMetrics(cost_tracker=cost, stateless=True),
cost_tracker=cost,
logger=None,
)
def _reporter_payload(cost_tracker: CostTracker) -> dict:
reporter = object.__new__(UsageReporter)
reporter._proxy = SimpleNamespace(cost_tracker=cost_tracker)
reporter._last_report_time = None
reporter._last_tokens_saved_by_model = {}
reporter._last_tokens_sent_by_model = {}
reporter._last_requests_by_model = {}
reporter._last_provider_counters = {}
reporter._license_key = "k"
reporter._cloud_url = "https://cloud.example"
reporter._license_info = None
sent: list[dict] = []
class _Client:
async def post(self, url, json, **kwargs): # noqa: A002, ANN001, ANN003, ANN202
sent.append(json)
return SimpleNamespace(status_code=200, json=lambda: {})
async def _get_client(): # noqa: ANN202
return _Client()
reporter._get_client = _get_client
anyio.run(reporter._report_usage)
(payload,) = sent
return payload
def test_streamed_claude_turn_reaches_dashboard_and_licence_report_as_billed() -> None:
state = _replay_recorded_stream("anthropic", FIXTURE.read_bytes())
outcome = _finalize("anthropic", state, optimized_tokens=LOCAL_ESTIMATE)
handler = _handler()
asyncio.run(emit_request_outcome(handler, outcome))
m = handler.metrics
assert m.tokens_input_total == ANTHROPIC_BILLED
assert m.tokens_input_provider_reported_total == ANTHROPIC_BILLED
assert m.tokens_input_estimated_total == 0
assert m.requests_input_provider_reported == 1
payload = _reporter_payload(handler.cost_tracker)
# "Input after Headroom" now reconciles with the Anthropic Console
# (input + cache read + cache write) for this turn.
assert payload["tokens_after"] == ANTHROPIC_BILLED
assert payload["tokens_after_provider_reported"] == ANTHROPIC_BILLED
assert payload["tokens_after_estimated"] == 0
assert payload["requests_provider_reported"] == 1
assert payload["cache_read_tokens"] == 8_857
assert payload["uncached_input_tokens"] == 19
def test_estimated_input_is_reported_as_estimated() -> None:
handler = _handler()
outcome = RequestOutcome(
request_id="req_no_usage",
provider="bedrock",
model="anthropic.claude-sonnet-4-6",
original_tokens=500,
optimized_tokens=400,
output_tokens=0,
tokens_saved=100,
attempted_input_tokens=500,
)
asyncio.run(emit_request_outcome(handler, outcome))
assert handler.metrics.tokens_input_estimated_total == 400
assert handler.metrics.requests_input_estimated == 1
payload = _reporter_payload(handler.cost_tracker)
assert payload["tokens_after"] == 400
assert payload["tokens_after_provider_reported"] == 0
assert payload["tokens_after_estimated"] == 400
+3
View File
@@ -307,6 +307,9 @@ async def test_funnel_passes_canonical_record_tokens_shape() -> None:
# it here is what lets the dashboard's per-model table count the layer
# its own headline counts. Zero for this outcome (no deferral tags).
"tool_schema_saved": 0,
# Whether the positional tokens_sent is the provider's billed count.
# This outcome carries no provider_input_tokens, so it is an estimate.
"provider_reported": False,
}