mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): record the provider's billed input count on streamed turns (#4052)
## Description Every input-volume figure Headroom reports for streamed Claude traffic was Headroom's own tokenizer estimate, not Anthropic's billed count. That covers "input after Headroom", `/stats` `tokens.input`, the dashboard's per-model table and the licence usage report's `tokens_after`. Since Claude Code always streams, this was effectively all Claude Code traffic, so customers comparing our numbers with the Anthropic Console could not reconcile them. **Root cause.** `emit_request_outcome` records `provider_input_tokens or optimized_tokens`. The streaming finalizer never set `provider_input_tokens` for Anthropic (#3946 fixed only the uncached sub-count), so every streamed Claude turn fell back to `optimized_tokens`. That value is: - an `o200k_base` count standing in for Claude's private tokenizer, measured against `count_tokens` on 6 real Claude Code transcripts at 15–25% under Claude 4.x and 35–38% under Claude 5.x; - a count of the forwarded **messages only**, which never sees the system prompt or tool definitions. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **`headroom/proxy/provider_usage.py` (new).** One place that knows each provider's usage semantics: - Anthropic, Bedrock and Vertex: `input_tokens` is the uncached tail, and the billed total is `input + cache_read + cache_write`. - OpenAI `prompt_tokens`/`input_tokens` and Gemini `promptTokenCount` are already inclusive of cached tokens. - **Streaming finalizer.** Sets `provider_input_tokens` from the provider's usage on the Anthropic/OpenAI/Gemini SSE path, the Bedrock stream path and the OpenAI-via-backend stream path. `RequestOutcome.from_stream` gains the parameter. - **OpenAI pass-through.** On both streaming and non-streaming paths, Anthropic-shape usage was read as if `input_tokens` were the total. It now uses the billed total and is marked provider-reported. - **Gemini non-streaming and OpenAI Responses WebSocket.** These already used the provider's count as `optimized_tokens` but never declared it, so they now also set `provider_input_tokens`, but only when the provider actually sent a count. - **No estimate is ever labelled as billed.** The Bedrock #1132 `message_start` backfill and Gemini's fallback for a missing `promptTokenCount` both stay estimated. The pass-through usage dialect is chosen by provider, not inferred from which keys are present. - **Provenance counters.** Every input total now carries how much of it is billed versus estimated: - `/stats` `tokens.input_provider_reported`, `input_estimated`, `input_provider_reported_percent`, `requests_input_*`. - Licence usage report: `tokens_after_provider_reported`, `tokens_after_estimated`, `requests_provider_reported`, plus the billed `cache_read_tokens`, `cache_write_tokens` and `uncached_input_tokens`, so the cloud side can reconcile with the console (input + cache read + cache write). The report also states `tokens_saved_basis: "headroom_tokenizer"`. - A one-time `event=input_tokens_estimated` warning per provider/model when a successful request carried no provider usage. - **No delta math changes.** `optimized_tokens` stays on the local scale, and `tokens_saved`, ratios and the ledger delta are untouched. The provider count is never subtracted from a local count. ## Testing - [ ] Unit tests pass (`pytest`): targeted suites only, listed below. The full suite is left to CI (see Additional Notes). - [x] Linting passes (`ruff check .`): run on all changed files. - [x] Type checking passes (`mypy headroom`): run on the changed modules. - [x] New tests added for new functionality - [x] Manual testing performed `tests/test_provider_billed_input.py` adds 15 tests, plus regression tests for the review fixes in `tests/test_bedrock_streaming_input_tokens.py` (the #1132 backfill is reported as estimated) and `tests/test_gemini_ccr_continuation_usage.py` (a missing `promptTokenCount` is not provider-reported): - **Recorded fixture.** `tests/fixtures/anthropic/stream_cached_turn.sse` is a **real recorded Anthropic stream**: a second call on a cached ~8.9K-token system prompt, billed as 19 uncached + 8,857 cache-read. It is replayed in 97-byte chunks through the proxy's own SSE usage parser and `_finalize_stream_response`, then through `emit_request_outcome` into a real `PrometheusMetrics`, `CostTracker` and `UsageReporter`. The test asserts that `/stats` input and the licence `tokens_after` equal Anthropic's 8,876. - Other cases: - an OpenAI stream records the inclusive `prompt_tokens`; - a stream without usage stays estimated; - each provider shape is covered; - the pass-through normaliser returns the total. Against the unpatched handlers, the 5 behaviour tests fail. The 8 that pass exercise only the new helper. ### Test Output ```text $ pytest tests/test_provider_billed_input.py tests/test_proxy_handler_helpers.py tests/test_request_outcome.py \ tests/test_proxy_streaming_request_logger.py tests/test_proxy_cache_ttl_metrics.py \ tests/test_usage_reporter_snapshot.py tests/test_5xx_accounting_all_providers.py tests/test_copilot_provider_label.py \ tests/test_bedrock_streaming_input_tokens.py tests/test_gemini_ccr_continuation_usage.py -q 181 passed in 6.56s # same new tests against the unpatched handlers FAILED tests/test_provider_billed_input.py::test_passthrough_normaliser_returns_anthropic_total_not_uncached_tail FAILED tests/test_provider_billed_input.py::test_recorded_anthropic_stream_records_anthropics_billed_input FAILED tests/test_provider_billed_input.py::test_openai_stream_records_inclusive_prompt_tokens FAILED tests/test_provider_billed_input.py::test_streamed_claude_turn_reaches_dashboard_and_licence_report_as_billed FAILED tests/test_provider_billed_input.py::test_estimated_input_is_reported_as_estimated 5 failed, 8 passed $ ruff check <changed files> -> All checks passed! $ mypy <changed proxy/telemetry modules> -> Success: no issues found in 6 source files ``` ## Real Behavior Proof - Environment: local proxy from this branch (`headroom proxy --port 8931 --stateless --no-telemetry --no-cache`) against the real Anthropic API. For the baseline, the same command ran from a `main` worktree on port 8932. - Exact command / steps: 3 streamed `claude-sonnet-4-6` turns through the proxy, with a cached ~20K-token system prompt, one tool and a 300-item JSON tool result. Anthropic's `message_start` usage (input + cache read + cache write) was summed from each stream and compared with `GET /stats` → `tokens`. - Observed result: the author-provided live Anthropic comparison below reports exact billed-input agreement at 61,999 tokens; the baseline undercounts by 37%. This external run was not independently repeated during the current-head review. | | Anthropic billed input | Headroom `/stats` `tokens.input` | |---|---|---| | `main` | 62,001 | **38,835** (−37%) | | this PR | 61,999 | **61,999** (exact), `input_provider_reported_percent: 100.0` | The two runs differ by 2 tokens because they are separate API calls. `proxy_compression_saved` was 29,313 on both, so savings behaviour is unchanged. - Not tested: live Bedrock, Vertex and Gemini streams (unit-tested on provider shapes only), and the licence cloud endpoint receiving the new fields. ## Runtime Rollout Safety - Rollout-managed feature(s): none. This is accounting only, and request and response bytes are unchanged. - Minimum rollout channel: N/A - Stable/default behavior changed: reported input volume now uses the provider's billed count where the provider reports one. New fields are added to `/stats` and the licence usage payload; no existing field is removed or renamed. - Kill switch / disable path: none needed. With no usage frame, the fallback is the previous estimate, now labelled `input_estimated`. - Unsafe override required: no - Qualification impact: dashboards and licence reports will show larger, correct Claude input totals once installs upgrade, so historical and new figures are not directly comparable. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes: targeted suites only; full suite on CI - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - **Full suite.** Locally, the full run on the pre-rebase base reached 2,147 passed before stopping on `tests/test_cli/test_mcp_reconcile.py::test_ordinary_install_does_not_adopt_serena`. That test fails identically on unmodified `main` (unrelated MCP install args). The full local run was then abandoned because several wrap/install tests launch a real `serena project index` (rust-analyzer, tsserver, rustc), which should be stubbed in a separate PR. - **Docs.** The new `/stats` and licence fields are not yet documented. - Follow-ups not in this PR: - **Raw Bedrock invoke pass-through (`handlers/bedrock.py`)** forwards bytes without parsing them, so it still records an estimate. It is now counted as `input_estimated`, so it's visible rather than silent. - **Batch create** has no usage until results come back, so it stays estimated as above. - **`tokens_saved` is still on the `o200k_base` scale**, so it under-reports Claude savings. Calibrating it to provider tokens is a separate PR. - **The licence cloud endpoint** needs to read the new fields. 🤖 Generated with [Claude Code](https://claude.com/claude-code) ## Current maintainer verification (October 8) Exact head: `81527148ed9777e1b9ca921bbf6bb457d49354db`, containing main `01d9194854233ef6ed04005cf0a5b8c32bf65d6d` through a clean merge. No contributor logic was modified. 63 focused tests pass with the existing compatible native extension on Windows/Python 3.12. Three backend-boundary probes preserve zero client input and exact cache-read, cache-write, and combined provider input totals. This is local mocked-backend/TestClient evidence; the author-provided external-provider evidence above was not independently rerun. Current-head hosted snapshot: 31 successful checks, 12 skips, no failed or live checks. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JerrettDavis
parent
01d9194854
commit
a9815bbb5f
+20
-1
@@ -900,6 +900,12 @@ class CostTracker:
|
||||
self._tool_saved_write_1h_by_model: dict[str, float] = {}
|
||||
self._tool_saved_list_by_model: dict[str, float] = {}
|
||||
self._tokens_sent_by_model: dict[str, int] = {}
|
||||
# Of _tokens_sent_by_model: the part that is the provider's own billed
|
||||
# count (the rest is Headroom's tokenizer estimate), and how many
|
||||
# requests it came from. The licence usage report sends both so the
|
||||
# cloud side can tell a billed figure from an estimate.
|
||||
self._provider_tokens_sent_by_model: dict[str, int] = {}
|
||||
self._provider_requests_by_model: dict[str, int] = {}
|
||||
# Completion tokens keyed by ``(model, long_context)`` — same reason as
|
||||
# the savings buckets: the >200k completion rate is a different number.
|
||||
self._output_tokens_by_tier: dict[tuple[str, bool], int] = {}
|
||||
@@ -930,6 +936,8 @@ class CostTracker:
|
||||
self._tool_saved_write_1h_by_model.clear()
|
||||
self._tool_saved_list_by_model.clear()
|
||||
self._tokens_sent_by_model.clear()
|
||||
self._provider_tokens_sent_by_model.clear()
|
||||
self._provider_requests_by_model.clear()
|
||||
self._output_tokens_by_tier.clear()
|
||||
self._requests_by_model.clear()
|
||||
self._api_cache_read_by_model.clear()
|
||||
@@ -1023,13 +1031,15 @@ class CostTracker:
|
||||
output_tokens: int = 0,
|
||||
cache_inferred: bool = False,
|
||||
tool_schema_saved: int = 0,
|
||||
provider_reported: bool = False,
|
||||
):
|
||||
"""Record token counts per model and accumulate request cost for budget enforcement.
|
||||
|
||||
Args:
|
||||
model: Model name.
|
||||
tokens_saved: Tokens removed by compression (Headroom's count).
|
||||
tokens_sent: Compressed message tokens sent (Headroom's count).
|
||||
tokens_sent: Input tokens sent: the provider's billed count when
|
||||
``provider_reported``, else Headroom's local estimate.
|
||||
cache_read_tokens: Cache read tokens from API response usage.
|
||||
cache_write_tokens: Cache write tokens from API response usage.
|
||||
uncached_tokens: Non-cached input tokens from API response usage.
|
||||
@@ -1047,6 +1057,8 @@ class CostTracker:
|
||||
attributed. The dashboard's per-model "Tokens Saved" column
|
||||
therefore showed compression only, while the headline above it
|
||||
counted both.
|
||||
provider_reported: True when ``tokens_sent`` is the provider's own
|
||||
billed input (uncached + cache read + cache write).
|
||||
"""
|
||||
# Post-guard invariant (all providers): Headroom never forwards a request
|
||||
# larger than the original (handlers revert any inflation before sending),
|
||||
@@ -1134,6 +1146,13 @@ class CostTracker:
|
||||
self._tool_saved_list_by_model.get(model, 0.0) + list_part
|
||||
)
|
||||
self._tokens_sent_by_model[model] = self._tokens_sent_by_model.get(model, 0) + tokens_sent
|
||||
if provider_reported:
|
||||
self._provider_tokens_sent_by_model[model] = (
|
||||
self._provider_tokens_sent_by_model.get(model, 0) + tokens_sent
|
||||
)
|
||||
self._provider_requests_by_model[model] = (
|
||||
self._provider_requests_by_model.get(model, 0) + 1
|
||||
)
|
||||
okey = (model, long_context)
|
||||
self._output_tokens_by_tier[okey] = self._output_tokens_by_tier.get(okey, 0) + max(
|
||||
0, output_tokens
|
||||
|
||||
@@ -565,6 +565,8 @@ class GeminiHandlerMixin:
|
||||
status_code=response.status_code,
|
||||
original_tokens=total_input_tokens,
|
||||
optimized_tokens=total_input_tokens,
|
||||
# Gemini's own promptTokenCount (inclusive of cached content).
|
||||
provider_input_tokens=total_input_tokens,
|
||||
output_tokens=output_tokens,
|
||||
tokens_saved=0,
|
||||
attempted_input_tokens=total_input_tokens,
|
||||
@@ -872,6 +874,9 @@ class GeminiHandlerMixin:
|
||||
total_latency = (time.time() - start_time) * 1000
|
||||
|
||||
total_input_tokens = optimized_tokens # fallback
|
||||
# Gemini's own promptTokenCount, only when it reported one; the
|
||||
# fallback above is Headroom's estimate and must not pass as billed.
|
||||
provider_prompt_tokens = 0
|
||||
output_tokens = 0
|
||||
cache_read_tokens = 0
|
||||
resp_json = None
|
||||
@@ -890,6 +895,8 @@ class GeminiHandlerMixin:
|
||||
if usage.get("promptTokenCount") is None
|
||||
else usage["promptTokenCount"]
|
||||
)
|
||||
if usage.get("promptTokenCount") is not None:
|
||||
provider_prompt_tokens = total_input_tokens
|
||||
output_tokens = gemini_output_tokens(
|
||||
usage
|
||||
) # includes thinking tokens (2.5-family)
|
||||
@@ -988,6 +995,8 @@ class GeminiHandlerMixin:
|
||||
total_input_tokens = _usage_int(
|
||||
usage.get("promptTokenCount"), total_input_tokens
|
||||
)
|
||||
if usage.get("promptTokenCount") is not None:
|
||||
provider_prompt_tokens = total_input_tokens
|
||||
output_tokens = _usage_int(usage.get("candidatesTokenCount"), output_tokens)
|
||||
cache_read_tokens = _usage_int(
|
||||
usage.get("cachedContentTokenCount"), cache_read_tokens
|
||||
@@ -1048,6 +1057,8 @@ class GeminiHandlerMixin:
|
||||
status_code=response.status_code,
|
||||
original_tokens=effective_original_tokens,
|
||||
optimized_tokens=total_input_tokens,
|
||||
# Gemini's own promptTokenCount (inclusive of cached content).
|
||||
provider_input_tokens=provider_prompt_tokens,
|
||||
output_tokens=output_tokens,
|
||||
tokens_saved=tokens_saved,
|
||||
attempted_input_tokens=total_input_tokens + tokens_saved,
|
||||
|
||||
@@ -37,6 +37,7 @@ from headroom.proxy.helpers import (
|
||||
from headroom.proxy.identity import resolve_memory_identity
|
||||
from headroom.proxy.loopback_guard import is_loopback_host
|
||||
from headroom.proxy.modes import is_cache_mode
|
||||
from headroom.proxy.provider_usage import billed_input_for_provider, billed_input_from_usage
|
||||
from headroom.proxy.rate_limit_identity import rate_limit_identity
|
||||
from headroom.proxy.semantic_cache_key_policy import compute_request_cache_partition
|
||||
from headroom.proxy.stage_timer import StageTimer, emit_stage_timings_log
|
||||
@@ -584,7 +585,7 @@ def _usage_int(value: Any) -> int:
|
||||
return 0
|
||||
|
||||
|
||||
def _passthrough_usage_from_json(payload: Any) -> dict[str, int]:
|
||||
def _passthrough_usage_from_json(payload: Any, provider: str | None = None) -> dict[str, int]:
|
||||
"""Normalize usage from pass-through provider response shapes."""
|
||||
if not isinstance(payload, dict):
|
||||
return {}
|
||||
@@ -608,7 +609,11 @@ def _passthrough_usage_from_json(payload: Any) -> dict[str, int]:
|
||||
details = usage.get("prompt_tokens_details") or usage.get("input_tokens_details") or {}
|
||||
cache_read = details.get("cached_tokens") if isinstance(details, dict) else None
|
||||
return {
|
||||
"input_tokens": _usage_int(input_tokens),
|
||||
# The billed total. Anthropic-shape usage reports input_tokens as
|
||||
# the uncached tail only (cache buckets are disjoint); OpenAI's
|
||||
# headline figure is already inclusive. Callers derive uncached as
|
||||
# total - read - write, which is only right on the total.
|
||||
"input_tokens": billed_input_from_usage(usage, provider) or _usage_int(input_tokens),
|
||||
"output_tokens": _usage_int(output_tokens),
|
||||
"cache_read_input_tokens": _usage_int(usage.get("cache_read_input_tokens", cache_read)),
|
||||
"cache_creation_input_tokens": _usage_int(usage.get("cache_creation_input_tokens")),
|
||||
@@ -9453,6 +9458,8 @@ class OpenAIHandlerMixin:
|
||||
model=model_for_metrics,
|
||||
original_tokens=max(0, input_delta) + max(0, saved_delta),
|
||||
optimized_tokens=max(0, input_delta),
|
||||
# Summed from OpenAI's own response.completed usage frames.
|
||||
provider_input_tokens=max(0, input_delta),
|
||||
output_tokens=max(0, output_delta),
|
||||
tokens_saved=max(0, saved_delta),
|
||||
attempted_input_tokens=max(0, attempted_delta),
|
||||
@@ -10121,6 +10128,8 @@ class OpenAIHandlerMixin:
|
||||
model=model_name,
|
||||
original_tokens=residual_input_tokens + residual_tokens_saved,
|
||||
optimized_tokens=residual_input_tokens,
|
||||
# Summed from OpenAI's own response.completed usage frames.
|
||||
provider_input_tokens=residual_input_tokens,
|
||||
output_tokens=residual_output_tokens,
|
||||
tokens_saved=residual_tokens_saved,
|
||||
attempted_input_tokens=residual_attempted_input_tokens,
|
||||
@@ -11757,7 +11766,7 @@ class OpenAIHandlerMixin:
|
||||
usage: dict[str, int] = {}
|
||||
if response.headers.get("content-type", "").lower().startswith("application/json"):
|
||||
try:
|
||||
usage = _passthrough_usage_from_json(response.json())
|
||||
usage = _passthrough_usage_from_json(response.json(), provider)
|
||||
except (json.JSONDecodeError, ValueError, TypeError):
|
||||
usage = {}
|
||||
input_tokens = usage.get("input_tokens", 0)
|
||||
@@ -11773,6 +11782,7 @@ class OpenAIHandlerMixin:
|
||||
status_code=response.status_code,
|
||||
original_tokens=input_tokens,
|
||||
optimized_tokens=input_tokens,
|
||||
provider_input_tokens=input_tokens,
|
||||
output_tokens=output_tokens,
|
||||
tokens_saved=0,
|
||||
attempted_input_tokens=input_tokens,
|
||||
@@ -11938,10 +11948,18 @@ class OpenAIHandlerMixin:
|
||||
buf.extend(b"\n\n")
|
||||
_absorb_usage(self._parse_sse_usage_from_buffer(stream_state, stream_provider))
|
||||
|
||||
input_tokens = stream_state["input_tokens"] or 0
|
||||
output_tokens = stream_state["output_tokens"] or 0
|
||||
cache_read_tokens = stream_state["cache_read_input_tokens"] or 0
|
||||
cache_write_tokens = stream_state["cache_creation_input_tokens"] or 0
|
||||
# Anthropic-shape SSE usage reports only the uncached tail as
|
||||
# input_tokens; the billed total adds the disjoint cache buckets.
|
||||
# Gemini's promptTokenCount is already inclusive.
|
||||
input_tokens = billed_input_for_provider(
|
||||
stream_provider,
|
||||
stream_state["input_tokens"],
|
||||
cache_read=cache_read_tokens,
|
||||
cache_write=cache_write_tokens,
|
||||
)
|
||||
uncached_input_tokens = max(
|
||||
0,
|
||||
input_tokens - cache_read_tokens - cache_write_tokens,
|
||||
@@ -11953,6 +11971,7 @@ class OpenAIHandlerMixin:
|
||||
model=_passthrough_model_from_path(path, endpoint_name),
|
||||
original_tokens=input_tokens,
|
||||
optimized_tokens=input_tokens,
|
||||
provider_input_tokens=input_tokens,
|
||||
output_tokens=output_tokens,
|
||||
tokens_saved=0,
|
||||
attempted_input_tokens=input_tokens,
|
||||
|
||||
@@ -22,6 +22,7 @@ from headroom.proxy.helpers import (
|
||||
overload_retry_is_futile,
|
||||
retry_after_ms,
|
||||
)
|
||||
from headroom.proxy.provider_usage import billed_input_for_provider
|
||||
from headroom.proxy.token_counting import gemini_output_tokens
|
||||
|
||||
if TYPE_CHECKING:
|
||||
@@ -1001,6 +1002,24 @@ class StreamingMixin:
|
||||
effective_optimized_tokens - cache_read_tokens - cache_write_tokens, 0
|
||||
)
|
||||
|
||||
# The provider's own billed input for this turn. Without it the outcome
|
||||
# falls back to ``optimized_tokens`` — Headroom's tokenizer estimate of
|
||||
# the forwarded messages only (no system prompt, no tool definitions) —
|
||||
# and every volume/cost total downstream (dashboard, licence usage
|
||||
# report) silently reports the estimate as if it were the bill. Every
|
||||
# Claude Code turn is a streamed Anthropic turn, so this was the bulk of
|
||||
# Claude traffic.
|
||||
billed_provider_input = (
|
||||
billed_input_for_provider(
|
||||
provider,
|
||||
provider_input_tokens,
|
||||
cache_read=cache_read_tokens,
|
||||
cache_write=cache_write_tokens,
|
||||
)
|
||||
if isinstance(provider_input_tokens, int)
|
||||
else 0
|
||||
)
|
||||
|
||||
# Prefix-tracker mutation is provider-specific state that lives
|
||||
# outside the metric funnel. Run it before the funnel so the next
|
||||
# request inherits correct prefix state regardless of metric path.
|
||||
@@ -1100,6 +1119,7 @@ class StreamingMixin:
|
||||
request_id=request_id,
|
||||
original_tokens=effective_original_tokens,
|
||||
optimized_tokens=effective_optimized_tokens,
|
||||
provider_input_tokens=billed_provider_input,
|
||||
output_tokens=output_tokens,
|
||||
tokens_saved=tokens_saved,
|
||||
transforms_applied=transforms_applied,
|
||||
@@ -2189,8 +2209,22 @@ class StreamingMixin:
|
||||
# already reported by the backend is preserved untouched.
|
||||
if event.event_type == "message_start" and not event.raw_sse:
|
||||
msg_usage = event.data.setdefault("message", {}).setdefault("usage", {})
|
||||
if not msg_usage.get("input_tokens") and optimized_tokens > 0:
|
||||
# A zero with cache usage beside it is a real count: a
|
||||
# fully cached prompt bills only the cache buckets. Only
|
||||
# a bare zero is LiteLLM's unknown placeholder.
|
||||
_cached = any(
|
||||
isinstance(msg_usage.get(k), int) and msg_usage.get(k) > 0
|
||||
for k in ("cache_read_input_tokens", "cache_creation_input_tokens")
|
||||
)
|
||||
if (
|
||||
not msg_usage.get("input_tokens")
|
||||
and not _cached
|
||||
and optimized_tokens > 0
|
||||
):
|
||||
msg_usage["input_tokens"] = optimized_tokens
|
||||
# The client-bound backfill is Headroom's estimate,
|
||||
# not backend usage: never report it as billed.
|
||||
stream_state["input_tokens_backfilled"] = True
|
||||
|
||||
# Format as SSE
|
||||
if event.raw_sse:
|
||||
@@ -2208,6 +2242,8 @@ class StreamingMixin:
|
||||
usage = msg.get("usage", {})
|
||||
if "input_tokens" in usage:
|
||||
stream_state["input_tokens"] = usage["input_tokens"]
|
||||
if not stream_state.get("input_tokens_backfilled"):
|
||||
stream_state["backend_input_tokens"] = usage["input_tokens"]
|
||||
stream_state["cache_read_input_tokens"] = usage.get(
|
||||
"cache_read_input_tokens", 0
|
||||
)
|
||||
@@ -2225,6 +2261,7 @@ class StreamingMixin:
|
||||
stream_state["output_tokens"] = usage["output_tokens"]
|
||||
if "input_tokens" in usage:
|
||||
stream_state["input_tokens"] = usage["input_tokens"]
|
||||
stream_state["backend_input_tokens"] = usage["input_tokens"]
|
||||
if "cache_read_input_tokens" in usage:
|
||||
stream_state["cache_read_input_tokens"] = usage[
|
||||
"cache_read_input_tokens"
|
||||
@@ -2309,6 +2346,11 @@ class StreamingMixin:
|
||||
# ``from_stream`` as ``optimized + saved``. Bedrock
|
||||
# doesn't propagate frozen_message_count either — same
|
||||
# fallback as the SSE finalizer (#455).
|
||||
# Bedrock forwards Anthropic-shape usage: input_tokens is the
|
||||
# uncached tail only, cache buckets are disjoint. Only a count
|
||||
# the backend itself reported qualifies; the message_start
|
||||
# backfill above is Headroom's estimate.
|
||||
_bedrock_input = stream_state.get("backend_input_tokens")
|
||||
outcome = RequestOutcome.from_stream(
|
||||
body=body,
|
||||
provider=_backend_name,
|
||||
@@ -2316,6 +2358,16 @@ class StreamingMixin:
|
||||
request_id=request_id,
|
||||
original_tokens=original_tokens,
|
||||
optimized_tokens=optimized_tokens,
|
||||
provider_input_tokens=(
|
||||
billed_input_for_provider(
|
||||
"anthropic",
|
||||
_bedrock_input,
|
||||
cache_read=stream_state["cache_read_input_tokens"],
|
||||
cache_write=stream_state["cache_creation_input_tokens"],
|
||||
)
|
||||
if isinstance(_bedrock_input, int)
|
||||
else 0
|
||||
),
|
||||
output_tokens=stream_state["output_tokens"],
|
||||
tokens_saved=tokens_saved,
|
||||
transforms_applied=transforms_applied,
|
||||
@@ -2543,6 +2595,12 @@ class StreamingMixin:
|
||||
request_id=request_id,
|
||||
original_tokens=original_tokens,
|
||||
optimized_tokens=optimized_tokens,
|
||||
# OpenAI-shape usage from the backend: the headline figure
|
||||
# already includes cached tokens (cache_write above is
|
||||
# subtracted from it, not added).
|
||||
provider_input_tokens=(
|
||||
upstream_input if isinstance(upstream_input, int) else 0
|
||||
),
|
||||
output_tokens=output_tokens,
|
||||
tokens_saved=tokens_saved,
|
||||
transforms_applied=transforms_applied,
|
||||
|
||||
@@ -337,6 +337,7 @@ class RequestOutcome:
|
||||
original_messages: list[dict] | None = None,
|
||||
conversation_key: str | None = None,
|
||||
conversation_tokens_saved: int | None = None,
|
||||
provider_input_tokens: int = 0,
|
||||
) -> RequestOutcome:
|
||||
"""Construct an outcome from the locals available at streaming
|
||||
finalize. Three streaming finalizers
|
||||
@@ -407,6 +408,7 @@ class RequestOutcome:
|
||||
model=model,
|
||||
original_tokens=original_tokens,
|
||||
optimized_tokens=optimized_tokens,
|
||||
provider_input_tokens=max(int(provider_input_tokens or 0), 0),
|
||||
output_tokens=output_tokens,
|
||||
tokens_saved=tokens_saved,
|
||||
conversation_key=conversation_key,
|
||||
@@ -441,6 +443,29 @@ class RequestOutcome:
|
||||
# ── The funnel ───────────────────────────────────────────────────────
|
||||
|
||||
|
||||
_estimated_input_warned: set[tuple[str, str]] = set()
|
||||
|
||||
|
||||
def _warn_estimated_input(provider: str, model: str) -> None:
|
||||
"""Log once per (provider, model) when a request's input volume is an estimate.
|
||||
|
||||
Every successful response from Anthropic, OpenAI and Gemini carries usage,
|
||||
so landing here on those providers means a code path dropped it. The
|
||||
request still records (fail open), but the volume it contributes is
|
||||
Headroom's tokenizer estimate, not the bill, and the coverage counters say so.
|
||||
"""
|
||||
key = (str(provider), str(model))
|
||||
if key in _estimated_input_warned or len(_estimated_input_warned) >= 256:
|
||||
return
|
||||
_estimated_input_warned.add(key)
|
||||
logger.warning(
|
||||
"event=input_tokens_estimated provider=%s model=%s: no provider usage on this "
|
||||
"request; input volume recorded from Headroom's local tokenizer estimate",
|
||||
provider,
|
||||
model,
|
||||
)
|
||||
|
||||
|
||||
async def emit_request_outcome(handler: Any, outcome: RequestOutcome) -> None:
|
||||
"""Single funnel for per-request bookkeeping. The contract.
|
||||
|
||||
@@ -652,6 +677,9 @@ async def emit_request_outcome(handler: Any, outcome: RequestOutcome) -> None:
|
||||
# Deliberately NOT used for any delta: differencing this against
|
||||
# ``original_tokens`` mixes tokenizer scales. See the field docs.
|
||||
billed_input_tokens = outcome.provider_input_tokens or outcome.optimized_tokens
|
||||
input_provider_reported = outcome.provider_input_tokens > 0
|
||||
if not input_provider_reported and billed_input_tokens > 0:
|
||||
_warn_estimated_input(outcome.provider, outcome.model)
|
||||
|
||||
# 1. Prometheus / SavingsTracker.
|
||||
await handler.metrics.record_request(
|
||||
@@ -684,6 +712,7 @@ async def emit_request_outcome(handler: Any, outcome: RequestOutcome) -> None:
|
||||
# so counting it as a write would both double it and apply a premium
|
||||
# OpenAI never charges.
|
||||
cache_inferred=outcome.cache_inferred,
|
||||
input_provider_reported=input_provider_reported,
|
||||
)
|
||||
|
||||
# 2. Cost tracker (optional).
|
||||
@@ -704,6 +733,7 @@ async def emit_request_outcome(handler: Any, outcome: RequestOutcome) -> None:
|
||||
# cost tracker feeds the dashboard's per-model table, which read
|
||||
# compression only while its own headline counted both layers.
|
||||
tool_schema_saved=tool_search_saved,
|
||||
provider_reported=input_provider_reported,
|
||||
)
|
||||
|
||||
# 3. Per-request log (optional). The ``client`` outcome field is
|
||||
|
||||
@@ -165,6 +165,14 @@ class PrometheusMetrics:
|
||||
self.inbound_responses_by_status: dict[str, int] = defaultdict(int)
|
||||
|
||||
self.tokens_input_total = 0
|
||||
# Of tokens_input_total: how much is the provider's own billed count
|
||||
# versus Headroom's local tokenizer estimate (provider reported no
|
||||
# usage, or the path never parsed it). Reported beside every input
|
||||
# total so nobody reads an estimate as the bill.
|
||||
self.tokens_input_provider_reported_total = 0
|
||||
self.tokens_input_estimated_total = 0
|
||||
self.requests_input_provider_reported = 0
|
||||
self.requests_input_estimated = 0
|
||||
self.tokens_output_total = 0
|
||||
self.tokens_saved_total = 0
|
||||
# Tool-schema savings (deferral + turn-hook tool shrink), aggregated from
|
||||
@@ -427,6 +435,10 @@ class PrometheusMetrics:
|
||||
self.inbound_responses_by_status.clear()
|
||||
|
||||
self.tokens_input_total = 0
|
||||
self.tokens_input_provider_reported_total = 0
|
||||
self.tokens_input_estimated_total = 0
|
||||
self.requests_input_provider_reported = 0
|
||||
self.requests_input_estimated = 0
|
||||
self.tokens_output_total = 0
|
||||
self.tokens_saved_total = 0
|
||||
self.tool_search_saved_total = 0
|
||||
@@ -854,6 +866,10 @@ class PrometheusMetrics:
|
||||
# ``CacheMix.normalized``. Defaults False, preserving behaviour for the
|
||||
# providers that report disjoint buckets.
|
||||
cache_inferred: bool = False,
|
||||
# True when ``input_tokens`` is the provider's own billed count, False
|
||||
# when it is Headroom's local estimate. None (callers predating the
|
||||
# split) records nothing in either bucket.
|
||||
input_provider_reported: bool | None = None,
|
||||
):
|
||||
"""Record metrics for a request.
|
||||
|
||||
@@ -931,6 +947,12 @@ class PrometheusMetrics:
|
||||
self.requests_cached += 1
|
||||
|
||||
self.tokens_input_total += input_tokens
|
||||
if input_provider_reported is True:
|
||||
self.tokens_input_provider_reported_total += input_tokens
|
||||
self.requests_input_provider_reported += 1
|
||||
elif input_provider_reported is False and input_tokens > 0:
|
||||
self.tokens_input_estimated_total += input_tokens
|
||||
self.requests_input_estimated += 1
|
||||
self.tokens_output_total += output_tokens
|
||||
self.tokens_saved_total += tokens_saved
|
||||
self.tool_search_saved_total += max(0, int(tool_search_saved))
|
||||
|
||||
@@ -0,0 +1,130 @@
|
||||
"""The provider's own billed input-token count, from its own usage block.
|
||||
|
||||
Every volume and cost figure Headroom reports ("input after Headroom", the
|
||||
licence usage report, the dashboard's per-model table) should be the number
|
||||
the provider billed, not Headroom's local tokenizer estimate. Headroom counts
|
||||
Claude with ``o200k_base`` as a stand-in for a private tokenizer; on real
|
||||
Claude Code transcripts that runs 15-25% under Claude 4.x and 35-38% under
|
||||
Claude 5.x, and it never sees the system prompt or tool definitions at all.
|
||||
|
||||
The providers disagree about what their headline input field means, which is
|
||||
why this lives in one place:
|
||||
|
||||
* Anthropic (and Bedrock/Vertex, which forward Anthropic-shape usage):
|
||||
``usage.input_tokens`` is only the tokens AFTER the last cache breakpoint.
|
||||
Cache reads and writes are separate, disjoint buckets, so the billed total
|
||||
is ``input_tokens + cache_read_input_tokens + cache_creation_input_tokens``.
|
||||
The Anthropic Console reports the same three columns.
|
||||
* OpenAI (chat ``prompt_tokens``, Responses ``input_tokens``): inclusive.
|
||||
``*_tokens_details.cached_tokens`` is a SUBSET of the headline figure.
|
||||
* Gemini: ``promptTokenCount`` is inclusive; ``cachedContentTokenCount`` is a
|
||||
subset of it.
|
||||
|
||||
A return of 0 means "the provider reported nothing", never "zero tokens":
|
||||
callers fall back to the local estimate and the request is counted as
|
||||
estimated rather than provider-reported (see ``RequestOutcome``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
from typing import Any
|
||||
|
||||
__all__ = [
|
||||
"anthropic_billed_input",
|
||||
"is_anthropic_dialect",
|
||||
"billed_input_from_usage",
|
||||
"billed_input_for_provider",
|
||||
]
|
||||
|
||||
# Providers whose usage blocks are Anthropic-shaped (disjoint cache buckets).
|
||||
_ANTHROPIC_SHAPE = frozenset({"anthropic", "bedrock", "vertex", "vertex_ai", "claude"})
|
||||
|
||||
|
||||
def _int(value: Any) -> int:
|
||||
if isinstance(value, bool):
|
||||
return 0
|
||||
try:
|
||||
return max(int(value), 0)
|
||||
except (TypeError, ValueError):
|
||||
return 0
|
||||
|
||||
|
||||
def is_anthropic_dialect(provider: str | None) -> bool:
|
||||
"""True for providers whose usage blocks keep cache buckets disjoint.
|
||||
|
||||
Decided by the provider, never by which keys a usage block happens to
|
||||
carry: OpenAI-compatible gateways (LiteLLM among them) mirror top-level
|
||||
``cache_read_input_tokens`` beside an INCLUSIVE ``input_tokens``, so the
|
||||
keys alone cannot tell the two dialects apart.
|
||||
"""
|
||||
name = (provider or "").lower()
|
||||
return name in _ANTHROPIC_SHAPE or "anthropic" in name
|
||||
|
||||
|
||||
def anthropic_billed_input(input_tokens: Any, cache_read: Any = 0, cache_write: Any = 0) -> int:
|
||||
"""Billed input for an Anthropic-shape usage block: the three disjoint buckets."""
|
||||
return _int(input_tokens) + _int(cache_read) + _int(cache_write)
|
||||
|
||||
|
||||
def billed_input_for_provider(
|
||||
provider: str | None,
|
||||
input_tokens: Any,
|
||||
*,
|
||||
cache_read: Any = 0,
|
||||
cache_write: Any = 0,
|
||||
) -> int:
|
||||
"""Billed input from already-extracted usage fields, by provider family.
|
||||
|
||||
``input_tokens`` is the provider's headline input field exactly as it
|
||||
reported it (Anthropic's uncached-only ``input_tokens``, OpenAI's inclusive
|
||||
``prompt_tokens``/``input_tokens``, Gemini's inclusive ``promptTokenCount``).
|
||||
Returns 0 when the provider reported no input count at all.
|
||||
"""
|
||||
if input_tokens is None:
|
||||
return 0
|
||||
if is_anthropic_dialect(provider):
|
||||
total = anthropic_billed_input(input_tokens, cache_read, cache_write)
|
||||
# message_start can carry input_tokens=0 alongside real cache buckets
|
||||
# on a fully cached turn; that is still a provider-reported count.
|
||||
return total
|
||||
return _int(input_tokens)
|
||||
|
||||
|
||||
def billed_input_from_usage(payload: Mapping[str, Any] | None, provider: str | None) -> int:
|
||||
"""Billed input from a raw provider response (or its ``usage`` block).
|
||||
|
||||
``provider`` selects the dialect (see ``is_anthropic_dialect``); it is
|
||||
required because the same keys mean different things across dialects.
|
||||
Gemini's ``usageMetadata`` is unambiguous and recognised by shape. Returns
|
||||
0 when no input count is present.
|
||||
"""
|
||||
if not isinstance(payload, Mapping):
|
||||
return 0
|
||||
|
||||
meta = payload.get("usageMetadata")
|
||||
if isinstance(meta, Mapping):
|
||||
return _int(meta.get("promptTokenCount"))
|
||||
|
||||
usage = payload.get("usage") if isinstance(payload.get("usage"), Mapping) else payload
|
||||
if not isinstance(usage, Mapping):
|
||||
return 0
|
||||
|
||||
if "promptTokenCount" in usage:
|
||||
return _int(usage.get("promptTokenCount"))
|
||||
|
||||
if is_anthropic_dialect(provider):
|
||||
if "input_tokens" not in usage:
|
||||
return 0
|
||||
return anthropic_billed_input(
|
||||
usage.get("input_tokens"),
|
||||
usage.get("cache_read_input_tokens"),
|
||||
usage.get("cache_creation_input_tokens"),
|
||||
)
|
||||
|
||||
# OpenAI and OpenAI-compatible dialects: the headline figure is inclusive.
|
||||
if "prompt_tokens" in usage:
|
||||
return _int(usage.get("prompt_tokens"))
|
||||
if "input_tokens" in usage:
|
||||
return _int(usage.get("input_tokens"))
|
||||
return 0
|
||||
@@ -5150,6 +5150,19 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
},
|
||||
"tokens": {
|
||||
"input": m.tokens_input_total,
|
||||
# Provenance of "input": the provider's own billed count versus
|
||||
# Headroom's tokenizer estimate. Only the first reconciles with
|
||||
# the provider console (input + cache read + cache write).
|
||||
"input_provider_reported": m.tokens_input_provider_reported_total,
|
||||
"input_estimated": m.tokens_input_estimated_total,
|
||||
"input_provider_reported_percent": round(
|
||||
(m.tokens_input_provider_reported_total / m.tokens_input_total * 100)
|
||||
if m.tokens_input_total > 0
|
||||
else 0,
|
||||
2,
|
||||
),
|
||||
"requests_input_provider_reported": m.requests_input_provider_reported,
|
||||
"requests_input_estimated": m.requests_input_estimated,
|
||||
"output": m.tokens_output_total,
|
||||
"output_saved": output_reduction.get("tokens_saved", 0),
|
||||
"output_reduction_percent": output_reduction.get("reduction_percent", 0),
|
||||
|
||||
@@ -118,6 +118,7 @@ class UsageReporter:
|
||||
self._last_tokens_saved_by_model: dict[str, int] = {}
|
||||
self._last_tokens_sent_by_model: dict[str, int] = {}
|
||||
self._last_requests_by_model: dict[str, int] = {}
|
||||
self._last_provider_counters: dict[str, dict[str, int]] = {}
|
||||
|
||||
async def validate_license(self) -> LicenseInfo:
|
||||
"""Validate the license key against the cloud API.
|
||||
@@ -301,6 +302,13 @@ class UsageReporter:
|
||||
self._last_report_time = now
|
||||
return
|
||||
|
||||
# Provenance of tokens_after: how much of it is the provider's own
|
||||
# billed count versus Headroom's tokenizer estimate, plus the billed
|
||||
# cache split. Lets the cloud side show a number that reconciles with
|
||||
# the provider's console (input + cache read + cache write) and flag
|
||||
# any remainder as an estimate instead of passing it off as the bill.
|
||||
provider_deltas = self._provider_counter_deltas(cost_tracker)
|
||||
|
||||
payload = {
|
||||
"license_key": self._license_key,
|
||||
"period_start": period_start.isoformat(),
|
||||
@@ -310,6 +318,18 @@ class UsageReporter:
|
||||
"tokens_after": total_tokens_after,
|
||||
"tokens_saved": total_tokens_saved,
|
||||
"models": delta_reqs_by_model,
|
||||
"tokens_after_provider_reported": provider_deltas["tokens_after_provider_reported"],
|
||||
"tokens_after_estimated": max(
|
||||
0, total_tokens_after - provider_deltas["tokens_after_provider_reported"]
|
||||
),
|
||||
"requests_provider_reported": provider_deltas["requests_provider_reported"],
|
||||
"cache_read_tokens": provider_deltas["cache_read_tokens"],
|
||||
"cache_write_tokens": provider_deltas["cache_write_tokens"],
|
||||
"uncached_input_tokens": provider_deltas["uncached_input_tokens"],
|
||||
# Units, stated so no consumer has to guess: tokens_after is billed
|
||||
# input where provider-reported; tokens_saved is Headroom's local
|
||||
# tokenizer count of what compression removed.
|
||||
"tokens_saved_basis": "headroom_tokenizer",
|
||||
}
|
||||
|
||||
try:
|
||||
@@ -348,6 +368,31 @@ class UsageReporter:
|
||||
except Exception:
|
||||
logger.warning("Failed to send usage report", exc_info=True)
|
||||
|
||||
_PROVIDER_COUNTERS = {
|
||||
"tokens_after_provider_reported": "_provider_tokens_sent_by_model",
|
||||
"requests_provider_reported": "_provider_requests_by_model",
|
||||
"cache_read_tokens": "_api_cache_read_by_model",
|
||||
"cache_write_tokens": "_api_cache_write_by_model",
|
||||
"uncached_input_tokens": "_api_uncached_by_model",
|
||||
}
|
||||
|
||||
def _current_provider_counters(self, cost_tracker: Any) -> dict[str, dict[str, int]]:
|
||||
return {
|
||||
name: dict(getattr(cost_tracker, attr, {}) or {})
|
||||
for name, attr in self._PROVIDER_COUNTERS.items()
|
||||
}
|
||||
|
||||
def _provider_counter_deltas(self, cost_tracker: Any) -> dict[str, int]:
|
||||
"""Totals of the provenance counters since the last successful report."""
|
||||
current = self._current_provider_counters(cost_tracker)
|
||||
deltas: dict[str, int] = {}
|
||||
for name, by_model in current.items():
|
||||
last = getattr(self, "_last_provider_counters", {}).get(name, {})
|
||||
deltas[name] = sum(
|
||||
max(0, int(value) - int(last.get(model, 0))) for model, value in by_model.items()
|
||||
)
|
||||
return deltas
|
||||
|
||||
def _snapshot_metrics(self) -> None:
|
||||
"""Take a snapshot of current proxy metrics for delta computation."""
|
||||
if self._proxy is None or self._proxy.cost_tracker is None:
|
||||
@@ -356,6 +401,7 @@ class UsageReporter:
|
||||
self._last_tokens_saved_by_model = dict(ct._tokens_saved_by_model)
|
||||
self._last_tokens_sent_by_model = dict(ct._tokens_sent_by_model)
|
||||
self._last_requests_by_model = dict(ct._requests_by_model)
|
||||
self._last_provider_counters = self._current_provider_counters(ct)
|
||||
self._last_report_time = datetime.now(timezone.utc)
|
||||
|
||||
def _save_cache(self) -> None:
|
||||
|
||||
+27
@@ -0,0 +1,27 @@
|
||||
event: message_start
|
||||
data: {"type":"message_start","message":{"model":"claude-sonnet-4-6","id":"msg_011CfqCxQgQFZwvbJZt3ZtS6","type":"message","role":"assistant","content":[],"container":null,"stop_reason":null,"stop_sequence":null,"stop_details":null,"usage":{"input_tokens":19,"cache_creation_input_tokens":0,"cache_read_input_tokens":8857,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":0},"output_tokens":5,"service_tier":"standard","inference_geo":"global"},"diagnostics":null} }
|
||||
|
||||
event: content_block_start
|
||||
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""} }
|
||||
|
||||
event: ping
|
||||
data: {"type": "ping"}
|
||||
|
||||
event: content_block_delta
|
||||
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Rerun job j"} }
|
||||
|
||||
event: content_block_delta
|
||||
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"17 with"} }
|
||||
|
||||
event: content_block_delta
|
||||
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" --retry."} }
|
||||
|
||||
event: content_block_stop
|
||||
data: {"type":"content_block_stop","index":0 }
|
||||
|
||||
event: message_delta
|
||||
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null,"stop_details":null,"container":null},"usage":{"input_tokens":19,"cache_creation_input_tokens":0,"cache_read_input_tokens":8857,"output_tokens":14} }
|
||||
|
||||
event: message_stop
|
||||
data: {"type":"message_stop" }
|
||||
|
||||
@@ -52,7 +52,7 @@ def _make_bedrock_backend(events: list[StreamEvent]) -> MagicMock:
|
||||
return mock
|
||||
|
||||
|
||||
def _bedrock_events(input_tokens: int) -> list[StreamEvent]:
|
||||
def _bedrock_events(input_tokens: int, **cache: int) -> list[StreamEvent]:
|
||||
"""Build a minimal Anthropic streaming sequence as LiteLLM emits it."""
|
||||
message_start = {
|
||||
"type": "message_start",
|
||||
@@ -63,7 +63,7 @@ def _bedrock_events(input_tokens: int) -> list[StreamEvent]:
|
||||
"type": "message",
|
||||
"content": [],
|
||||
# LiteLLM hardcodes this to 0 — the bug under test.
|
||||
"usage": {"input_tokens": input_tokens, "output_tokens": 0},
|
||||
"usage": {"input_tokens": input_tokens, "output_tokens": 0, **cache},
|
||||
},
|
||||
}
|
||||
block_start = {
|
||||
@@ -150,3 +150,64 @@ def test_bedrock_streaming_preserves_nonzero_upstream_input_tokens() -> None:
|
||||
upstream_input_tokens = 777
|
||||
body = _post_stream(_make_bedrock_backend(_bedrock_events(input_tokens=upstream_input_tokens)))
|
||||
assert _message_start_input_tokens(body) == upstream_input_tokens
|
||||
|
||||
|
||||
def _post_stream_stats(backend: MagicMock) -> dict:
|
||||
"""Run one streamed turn and return /stats tokens (provenance counters)."""
|
||||
config = ProxyConfig(
|
||||
optimize=False,
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
backend="anyllm",
|
||||
anyllm_provider="anthropic",
|
||||
)
|
||||
with patch("headroom.proxy.server.AnyLLMBackend", return_value=backend):
|
||||
app = create_app(config)
|
||||
with TestClient(app) as client:
|
||||
resp = client.post(
|
||||
"/v1/messages",
|
||||
json={
|
||||
"model": "claude-3-5-sonnet-20241022",
|
||||
"messages": [{"role": "user", "content": "hello there general"}],
|
||||
"max_tokens": 64,
|
||||
"stream": True,
|
||||
},
|
||||
headers={"x-api-key": "sk-ant-test", "anthropic-version": "2023-06-01"},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text[:200]
|
||||
return client.get("/stats").json()["tokens"]
|
||||
|
||||
|
||||
def test_bedrock_backfilled_input_is_reported_as_estimated() -> None:
|
||||
"""The #1132 backfill is Headroom's estimate: it must not count as billed input."""
|
||||
tokens = _post_stream_stats(_make_bedrock_backend(_bedrock_events(input_tokens=0)))
|
||||
assert tokens["input"] > 0
|
||||
assert tokens["input_provider_reported"] == 0
|
||||
assert tokens["input_estimated"] == tokens["input"]
|
||||
|
||||
|
||||
def test_bedrock_reported_input_is_provider_reported() -> None:
|
||||
tokens = _post_stream_stats(_make_bedrock_backend(_bedrock_events(input_tokens=777)))
|
||||
assert tokens["input_provider_reported"] == 777
|
||||
assert tokens["input_estimated"] == 0
|
||||
|
||||
|
||||
def test_bedrock_fully_cached_zero_is_billed_input_not_a_placeholder() -> None:
|
||||
"""input_tokens=0 beside a cache bucket is a real count (a fully cached
|
||||
prompt), not LiteLLM's placeholder: it is not backfilled, and the cache
|
||||
buckets count as provider-reported billed input."""
|
||||
events = _bedrock_events(input_tokens=0, cache_read_input_tokens=8857)
|
||||
assert _message_start_input_tokens(_post_stream(_make_bedrock_backend(events))) == 0
|
||||
tokens = _post_stream_stats(
|
||||
_make_bedrock_backend(_bedrock_events(input_tokens=0, cache_read_input_tokens=8857))
|
||||
)
|
||||
assert tokens["input_provider_reported"] == 8857
|
||||
assert tokens["input_estimated"] == 0
|
||||
|
||||
|
||||
def test_bedrock_cache_write_alone_also_counts_as_reported() -> None:
|
||||
tokens = _post_stream_stats(
|
||||
_make_bedrock_backend(_bedrock_events(input_tokens=0, cache_creation_input_tokens=1200))
|
||||
)
|
||||
assert tokens["input_provider_reported"] == 1200
|
||||
assert tokens["input_estimated"] == 0
|
||||
|
||||
@@ -136,3 +136,64 @@ async def test_ccr_continuation_present_null_usage_does_not_502(
|
||||
# The pre-continuation count (100) survives as the fallback.
|
||||
assert handler.outcomes[0].optimized_tokens == 100
|
||||
assert handler.outcomes[0].cache_read_tokens == 0
|
||||
|
||||
|
||||
class _NoUsageResponse:
|
||||
"""A successful 200 whose usageMetadata carries no promptTokenCount."""
|
||||
|
||||
status_code = 200
|
||||
content = json.dumps(
|
||||
{"candidates": [{"content": {"parts": [{"text": "hi"}]}}], "usageMetadata": {}}
|
||||
).encode()
|
||||
headers = {"content-type": "application/json"}
|
||||
|
||||
def json(self) -> object:
|
||||
return json.loads(self.content)
|
||||
|
||||
|
||||
class _NoCcrHandler(_CcrHandler):
|
||||
def has_ccr_tool_calls(self, resp_json, provider) -> bool: # noqa: ANN001
|
||||
return False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_missing_prompt_token_count_is_not_reported_as_provider_input(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
async def payload(request): # noqa: ANN001, ANN201
|
||||
return {"contents": [{"role": "user", "parts": [{"text": "hello"}]}]}
|
||||
|
||||
monkeypatch.setattr("headroom.proxy.helpers._read_request_json", payload)
|
||||
|
||||
handler = _Handler()
|
||||
handler.ccr_response_handler = _NoCcrHandler()
|
||||
|
||||
async def no_usage(method, url, headers, body): # noqa: ANN001, ANN202
|
||||
return _NoUsageResponse()
|
||||
|
||||
handler._retry_request = no_usage
|
||||
response = await handler.handle_gemini_generate_content(_FakeRequest(), "gemini-pro")
|
||||
|
||||
assert response.status_code == 200
|
||||
(outcome,) = handler.outcomes
|
||||
# Volume still falls back to the local estimate...
|
||||
assert outcome.optimized_tokens == 100
|
||||
# ...but it must not be labelled as Gemini's own billed count.
|
||||
assert outcome.provider_input_tokens == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reported_prompt_token_count_is_provider_input(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
async def payload(request): # noqa: ANN001, ANN201
|
||||
return {"contents": [{"role": "user", "parts": [{"text": "hello"}]}]}
|
||||
|
||||
monkeypatch.setattr("headroom.proxy.helpers._read_request_json", payload)
|
||||
|
||||
handler = _Handler()
|
||||
handler.ccr_response_handler = _NoCcrHandler()
|
||||
response = await handler.handle_gemini_generate_content(_FakeRequest(), "gemini-pro")
|
||||
|
||||
assert response.status_code == 200
|
||||
assert handler.outcomes[0].provider_input_tokens == 100
|
||||
|
||||
@@ -0,0 +1,297 @@
|
||||
"""Input volume must be the provider's own billed count, not Headroom's estimate.
|
||||
|
||||
Headroom's local tokenizer is a stand-in for Claude's private one (15-25% under
|
||||
Claude 4.x, 35-38% under Claude 5.x on real transcripts) and only ever sees the
|
||||
forwarded messages, never the system prompt or tool definitions. Every streamed
|
||||
Anthropic turn (all Claude Code traffic) used to fall back to that estimate for
|
||||
"input after Headroom", the dashboard input total and the licence usage report,
|
||||
so none of them reconciled with the Anthropic Console.
|
||||
|
||||
The Anthropic fixture is a real recorded stream: a second call on a cached
|
||||
~8.9K-token system prompt. Anthropic billed 19 uncached + 8,857 cache-read
|
||||
tokens; the messages Headroom counts locally are a single short user turn.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
import anyio
|
||||
import pytest
|
||||
|
||||
from headroom.proxy.cost import CostTracker
|
||||
from headroom.proxy.handlers.openai import _passthrough_usage_from_json
|
||||
from headroom.proxy.outcome import RequestOutcome, emit_request_outcome
|
||||
from headroom.proxy.prometheus_metrics import PrometheusMetrics
|
||||
from headroom.proxy.provider_usage import (
|
||||
billed_input_for_provider,
|
||||
billed_input_from_usage,
|
||||
)
|
||||
from headroom.proxy.server import HeadroomProxy
|
||||
from headroom.telemetry.reporter import UsageReporter
|
||||
|
||||
FIXTURE = Path(__file__).parent / "fixtures" / "anthropic" / "stream_cached_turn.sse"
|
||||
# From the fixture's message_start usage: input 19 + cache_read 8857 + cache_write 0.
|
||||
ANTHROPIC_BILLED = 19 + 8_857
|
||||
# What Headroom's tokenizer would see: one short user message.
|
||||
LOCAL_ESTIMATE = 17
|
||||
|
||||
|
||||
# ── provider shape semantics ──────────────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"provider,payload,expected",
|
||||
[
|
||||
# Anthropic: input_tokens is the uncached tail; cache buckets are disjoint.
|
||||
(
|
||||
"anthropic",
|
||||
{
|
||||
"usage": {
|
||||
"input_tokens": 19,
|
||||
"cache_read_input_tokens": 8_857,
|
||||
"cache_creation_input_tokens": 100,
|
||||
}
|
||||
},
|
||||
8_976,
|
||||
),
|
||||
# OpenAI chat: prompt_tokens already includes cached_tokens.
|
||||
(
|
||||
"openai",
|
||||
{"usage": {"prompt_tokens": 5_000, "prompt_tokens_details": {"cached_tokens": 4_000}}},
|
||||
5_000,
|
||||
),
|
||||
# OpenAI Responses: input_tokens already includes cached_tokens.
|
||||
(
|
||||
"openai",
|
||||
{"usage": {"input_tokens": 5_000, "input_tokens_details": {"cached_tokens": 4_000}}},
|
||||
5_000,
|
||||
),
|
||||
# Gemini: promptTokenCount includes cachedContentTokenCount.
|
||||
(
|
||||
"gemini",
|
||||
{"usageMetadata": {"promptTokenCount": 7_000, "cachedContentTokenCount": 6_000}},
|
||||
7_000,
|
||||
),
|
||||
# OpenAI-compatible gateways (LiteLLM) mirror top-level Anthropic-named
|
||||
# cache keys beside an INCLUSIVE input_tokens. Keys alone must not flip
|
||||
# the dialect, or the cached part is counted twice.
|
||||
("openai", {"usage": {"input_tokens": 5_000, "cache_read_input_tokens": 4_000}}, 5_000),
|
||||
(
|
||||
"vertex:anthropic",
|
||||
{"usage": {"input_tokens": 50, "cache_read_input_tokens": 4_000}},
|
||||
4_050,
|
||||
),
|
||||
# Nothing reported: 0 means "unknown", never "zero tokens billed".
|
||||
("anthropic", {"id": "x"}, 0),
|
||||
("openai", None, 0),
|
||||
],
|
||||
)
|
||||
def test_billed_input_from_usage_per_provider_shape(provider, payload, expected) -> None: # noqa: ANN001
|
||||
assert billed_input_from_usage(payload, provider) == expected
|
||||
|
||||
|
||||
def test_billed_input_for_provider_adds_anthropic_cache_buckets_only() -> None:
|
||||
assert billed_input_for_provider("anthropic", 19, cache_read=8_857, cache_write=0) == 8_876
|
||||
assert billed_input_for_provider("bedrock", 2, cache_read=360_949, cache_write=840) == 361_791
|
||||
# OpenAI/Gemini headline figures are inclusive: never add the cache split.
|
||||
assert billed_input_for_provider("openai", 5_000, cache_read=4_000) == 5_000
|
||||
assert billed_input_for_provider("gemini", 7_000, cache_read=6_000) == 7_000
|
||||
assert billed_input_for_provider("anthropic", None) == 0
|
||||
|
||||
|
||||
def test_passthrough_normaliser_returns_anthropic_total_not_uncached_tail() -> None:
|
||||
usage = _passthrough_usage_from_json(
|
||||
{"usage": {"input_tokens": 19, "output_tokens": 14, "cache_read_input_tokens": 8_857}},
|
||||
"anthropic",
|
||||
)
|
||||
# Callers derive uncached as input - read - write, which is only right on the total.
|
||||
assert usage["input_tokens"] == ANTHROPIC_BILLED
|
||||
assert usage["input_tokens"] - usage["cache_read_input_tokens"] == 19
|
||||
|
||||
|
||||
# ── recorded Anthropic stream through the real SSE parser and finalizer ──
|
||||
|
||||
|
||||
def _replay_recorded_stream(provider: str, raw: bytes, chunk_size: int = 97) -> dict:
|
||||
"""Feed recorded SSE bytes through the proxy's own usage parser, in odd-sized
|
||||
chunks so events split across reads the way they do on the wire."""
|
||||
proxy = object.__new__(HeadroomProxy)
|
||||
state = {
|
||||
"input_tokens": None,
|
||||
"output_tokens": None,
|
||||
"cache_read_input_tokens": None,
|
||||
"cache_creation_input_tokens": None,
|
||||
"cache_creation_ephemeral_5m_input_tokens": None,
|
||||
"cache_creation_ephemeral_1h_input_tokens": None,
|
||||
"total_bytes": 0,
|
||||
"sse_buffer": bytearray(),
|
||||
"ttfb_ms": 3.0,
|
||||
}
|
||||
for i in range(0, len(raw), chunk_size):
|
||||
chunk = raw[i : i + chunk_size]
|
||||
state["total_bytes"] += len(chunk)
|
||||
state["sse_buffer"].extend(chunk)
|
||||
usage = proxy._parse_sse_usage_from_buffer(state, provider)
|
||||
for key, value in (usage or {}).items():
|
||||
if key in state:
|
||||
state[key] = value
|
||||
return state
|
||||
|
||||
|
||||
def _finalize(provider: str, stream_state: dict, optimized_tokens: int) -> RequestOutcome:
|
||||
handler = object.__new__(HeadroomProxy)
|
||||
handler.config = SimpleNamespace(log_full_messages=False)
|
||||
outcomes: list[RequestOutcome] = []
|
||||
|
||||
async def record(outcome): # noqa: ANN001, ANN202
|
||||
outcomes.append(outcome)
|
||||
|
||||
handler._record_request_outcome = record
|
||||
asyncio.run(
|
||||
handler._finalize_stream_response(
|
||||
body={"messages": [{"role": "user", "content": "Which job reruns target t17?"}]},
|
||||
provider=provider,
|
||||
model="claude-sonnet-4-6" if provider == "anthropic" else "gpt-5",
|
||||
request_id=f"req_{provider}_billed",
|
||||
original_tokens=optimized_tokens,
|
||||
optimized_tokens=optimized_tokens,
|
||||
tokens_saved=0,
|
||||
transforms_applied=[],
|
||||
optimization_latency=1.0,
|
||||
stream_state=stream_state,
|
||||
start_time=0.0,
|
||||
)
|
||||
)
|
||||
(outcome,) = outcomes
|
||||
return outcome
|
||||
|
||||
|
||||
def test_recorded_anthropic_stream_records_anthropics_billed_input() -> None:
|
||||
state = _replay_recorded_stream("anthropic", FIXTURE.read_bytes())
|
||||
assert state["input_tokens"] == 19
|
||||
assert state["cache_read_input_tokens"] == 8_857
|
||||
|
||||
outcome = _finalize("anthropic", state, optimized_tokens=LOCAL_ESTIMATE)
|
||||
|
||||
assert outcome.provider_input_tokens == ANTHROPIC_BILLED
|
||||
assert outcome.uncached_input_tokens == 19
|
||||
assert outcome.cache_read_tokens == 8_857
|
||||
# The local estimate stays the local estimate: savings math must never mix
|
||||
# it with the provider's scale.
|
||||
assert outcome.optimized_tokens == LOCAL_ESTIMATE
|
||||
|
||||
|
||||
def test_anthropic_stream_without_usage_stays_estimated() -> None:
|
||||
# An error before message_start: no provider count exists, so the outcome
|
||||
# must not invent one (provider_input_tokens 0 => counted as estimated).
|
||||
state = _replay_recorded_stream("anthropic", b"")
|
||||
outcome = _finalize("anthropic", state, optimized_tokens=LOCAL_ESTIMATE)
|
||||
assert outcome.provider_input_tokens == 0
|
||||
|
||||
|
||||
def test_openai_stream_records_inclusive_prompt_tokens() -> None:
|
||||
frame = {
|
||||
"id": "c1",
|
||||
"choices": [],
|
||||
"usage": {
|
||||
"prompt_tokens": 12_000,
|
||||
"completion_tokens": 30,
|
||||
"prompt_tokens_details": {"cached_tokens": 11_000},
|
||||
},
|
||||
}
|
||||
raw = f"data: {json.dumps(frame)}\n\ndata: [DONE]\n\n".encode()
|
||||
state = _replay_recorded_stream("openai", raw)
|
||||
|
||||
outcome = _finalize("openai", state, optimized_tokens=9_000)
|
||||
|
||||
assert outcome.provider_input_tokens == 12_000
|
||||
|
||||
|
||||
# ── the full chain: outcome -> metrics/cost tracker -> licence usage report ──
|
||||
|
||||
|
||||
def _handler() -> SimpleNamespace:
|
||||
cost = CostTracker()
|
||||
return SimpleNamespace(
|
||||
metrics=PrometheusMetrics(cost_tracker=cost, stateless=True),
|
||||
cost_tracker=cost,
|
||||
logger=None,
|
||||
)
|
||||
|
||||
|
||||
def _reporter_payload(cost_tracker: CostTracker) -> dict:
|
||||
reporter = object.__new__(UsageReporter)
|
||||
reporter._proxy = SimpleNamespace(cost_tracker=cost_tracker)
|
||||
reporter._last_report_time = None
|
||||
reporter._last_tokens_saved_by_model = {}
|
||||
reporter._last_tokens_sent_by_model = {}
|
||||
reporter._last_requests_by_model = {}
|
||||
reporter._last_provider_counters = {}
|
||||
reporter._license_key = "k"
|
||||
reporter._cloud_url = "https://cloud.example"
|
||||
reporter._license_info = None
|
||||
sent: list[dict] = []
|
||||
|
||||
class _Client:
|
||||
async def post(self, url, json, **kwargs): # noqa: A002, ANN001, ANN003, ANN202
|
||||
sent.append(json)
|
||||
return SimpleNamespace(status_code=200, json=lambda: {})
|
||||
|
||||
async def _get_client(): # noqa: ANN202
|
||||
return _Client()
|
||||
|
||||
reporter._get_client = _get_client
|
||||
anyio.run(reporter._report_usage)
|
||||
(payload,) = sent
|
||||
return payload
|
||||
|
||||
|
||||
def test_streamed_claude_turn_reaches_dashboard_and_licence_report_as_billed() -> None:
|
||||
state = _replay_recorded_stream("anthropic", FIXTURE.read_bytes())
|
||||
outcome = _finalize("anthropic", state, optimized_tokens=LOCAL_ESTIMATE)
|
||||
handler = _handler()
|
||||
|
||||
asyncio.run(emit_request_outcome(handler, outcome))
|
||||
|
||||
m = handler.metrics
|
||||
assert m.tokens_input_total == ANTHROPIC_BILLED
|
||||
assert m.tokens_input_provider_reported_total == ANTHROPIC_BILLED
|
||||
assert m.tokens_input_estimated_total == 0
|
||||
assert m.requests_input_provider_reported == 1
|
||||
|
||||
payload = _reporter_payload(handler.cost_tracker)
|
||||
# "Input after Headroom" now reconciles with the Anthropic Console
|
||||
# (input + cache read + cache write) for this turn.
|
||||
assert payload["tokens_after"] == ANTHROPIC_BILLED
|
||||
assert payload["tokens_after_provider_reported"] == ANTHROPIC_BILLED
|
||||
assert payload["tokens_after_estimated"] == 0
|
||||
assert payload["requests_provider_reported"] == 1
|
||||
assert payload["cache_read_tokens"] == 8_857
|
||||
assert payload["uncached_input_tokens"] == 19
|
||||
|
||||
|
||||
def test_estimated_input_is_reported_as_estimated() -> None:
|
||||
handler = _handler()
|
||||
outcome = RequestOutcome(
|
||||
request_id="req_no_usage",
|
||||
provider="bedrock",
|
||||
model="anthropic.claude-sonnet-4-6",
|
||||
original_tokens=500,
|
||||
optimized_tokens=400,
|
||||
output_tokens=0,
|
||||
tokens_saved=100,
|
||||
attempted_input_tokens=500,
|
||||
)
|
||||
|
||||
asyncio.run(emit_request_outcome(handler, outcome))
|
||||
|
||||
assert handler.metrics.tokens_input_estimated_total == 400
|
||||
assert handler.metrics.requests_input_estimated == 1
|
||||
payload = _reporter_payload(handler.cost_tracker)
|
||||
assert payload["tokens_after"] == 400
|
||||
assert payload["tokens_after_provider_reported"] == 0
|
||||
assert payload["tokens_after_estimated"] == 400
|
||||
@@ -307,6 +307,9 @@ async def test_funnel_passes_canonical_record_tokens_shape() -> None:
|
||||
# it here is what lets the dashboard's per-model table count the layer
|
||||
# its own headline counts. Zero for this outcome (no deferral tags).
|
||||
"tool_schema_saved": 0,
|
||||
# Whether the positional tokens_sent is the provider's billed count.
|
||||
# This outcome carries no provider_input_tokens, so it is an estimate.
|
||||
"provider_reported": False,
|
||||
}
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user