mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description Every input-volume figure Headroom reports for streamed Claude traffic was Headroom's own tokenizer estimate, not Anthropic's billed count. That covers "input after Headroom", `/stats` `tokens.input`, the dashboard's per-model table and the licence usage report's `tokens_after`. Since Claude Code always streams, this was effectively all Claude Code traffic, so customers comparing our numbers with the Anthropic Console could not reconcile them. **Root cause.** `emit_request_outcome` records `provider_input_tokens or optimized_tokens`. The streaming finalizer never set `provider_input_tokens` for Anthropic (#3946 fixed only the uncached sub-count), so every streamed Claude turn fell back to `optimized_tokens`. That value is: - an `o200k_base` count standing in for Claude's private tokenizer, measured against `count_tokens` on 6 real Claude Code transcripts at 15–25% under Claude 4.x and 35–38% under Claude 5.x; - a count of the forwarded **messages only**, which never sees the system prompt or tool definitions. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **`headroom/proxy/provider_usage.py` (new).** One place that knows each provider's usage semantics: - Anthropic, Bedrock and Vertex: `input_tokens` is the uncached tail, and the billed total is `input + cache_read + cache_write`. - OpenAI `prompt_tokens`/`input_tokens` and Gemini `promptTokenCount` are already inclusive of cached tokens. - **Streaming finalizer.** Sets `provider_input_tokens` from the provider's usage on the Anthropic/OpenAI/Gemini SSE path, the Bedrock stream path and the OpenAI-via-backend stream path. `RequestOutcome.from_stream` gains the parameter. - **OpenAI pass-through.** On both streaming and non-streaming paths, Anthropic-shape usage was read as if `input_tokens` were the total. It now uses the billed total and is marked provider-reported. - **Gemini non-streaming and OpenAI Responses WebSocket.** These already used the provider's count as `optimized_tokens` but never declared it, so they now also set `provider_input_tokens`, but only when the provider actually sent a count. - **No estimate is ever labelled as billed.** The Bedrock #1132 `message_start` backfill and Gemini's fallback for a missing `promptTokenCount` both stay estimated. The pass-through usage dialect is chosen by provider, not inferred from which keys are present. - **Provenance counters.** Every input total now carries how much of it is billed versus estimated: - `/stats` `tokens.input_provider_reported`, `input_estimated`, `input_provider_reported_percent`, `requests_input_*`. - Licence usage report: `tokens_after_provider_reported`, `tokens_after_estimated`, `requests_provider_reported`, plus the billed `cache_read_tokens`, `cache_write_tokens` and `uncached_input_tokens`, so the cloud side can reconcile with the console (input + cache read + cache write). The report also states `tokens_saved_basis: "headroom_tokenizer"`. - A one-time `event=input_tokens_estimated` warning per provider/model when a successful request carried no provider usage. - **No delta math changes.** `optimized_tokens` stays on the local scale, and `tokens_saved`, ratios and the ledger delta are untouched. The provider count is never subtracted from a local count. ## Testing - [ ] Unit tests pass (`pytest`): targeted suites only, listed below. The full suite is left to CI (see Additional Notes). - [x] Linting passes (`ruff check .`): run on all changed files. - [x] Type checking passes (`mypy headroom`): run on the changed modules. - [x] New tests added for new functionality - [x] Manual testing performed `tests/test_provider_billed_input.py` adds 15 tests, plus regression tests for the review fixes in `tests/test_bedrock_streaming_input_tokens.py` (the #1132 backfill is reported as estimated) and `tests/test_gemini_ccr_continuation_usage.py` (a missing `promptTokenCount` is not provider-reported): - **Recorded fixture.** `tests/fixtures/anthropic/stream_cached_turn.sse` is a **real recorded Anthropic stream**: a second call on a cached ~8.9K-token system prompt, billed as 19 uncached + 8,857 cache-read. It is replayed in 97-byte chunks through the proxy's own SSE usage parser and `_finalize_stream_response`, then through `emit_request_outcome` into a real `PrometheusMetrics`, `CostTracker` and `UsageReporter`. The test asserts that `/stats` input and the licence `tokens_after` equal Anthropic's 8,876. - Other cases: - an OpenAI stream records the inclusive `prompt_tokens`; - a stream without usage stays estimated; - each provider shape is covered; - the pass-through normaliser returns the total. Against the unpatched handlers, the 5 behaviour tests fail. The 8 that pass exercise only the new helper. ### Test Output ```text $ pytest tests/test_provider_billed_input.py tests/test_proxy_handler_helpers.py tests/test_request_outcome.py \ tests/test_proxy_streaming_request_logger.py tests/test_proxy_cache_ttl_metrics.py \ tests/test_usage_reporter_snapshot.py tests/test_5xx_accounting_all_providers.py tests/test_copilot_provider_label.py \ tests/test_bedrock_streaming_input_tokens.py tests/test_gemini_ccr_continuation_usage.py -q 181 passed in 6.56s # same new tests against the unpatched handlers FAILED tests/test_provider_billed_input.py::test_passthrough_normaliser_returns_anthropic_total_not_uncached_tail FAILED tests/test_provider_billed_input.py::test_recorded_anthropic_stream_records_anthropics_billed_input FAILED tests/test_provider_billed_input.py::test_openai_stream_records_inclusive_prompt_tokens FAILED tests/test_provider_billed_input.py::test_streamed_claude_turn_reaches_dashboard_and_licence_report_as_billed FAILED tests/test_provider_billed_input.py::test_estimated_input_is_reported_as_estimated 5 failed, 8 passed $ ruff check <changed files> -> All checks passed! $ mypy <changed proxy/telemetry modules> -> Success: no issues found in 6 source files ``` ## Real Behavior Proof - Environment: local proxy from this branch (`headroom proxy --port 8931 --stateless --no-telemetry --no-cache`) against the real Anthropic API. For the baseline, the same command ran from a `main` worktree on port 8932. - Exact command / steps: 3 streamed `claude-sonnet-4-6` turns through the proxy, with a cached ~20K-token system prompt, one tool and a 300-item JSON tool result. Anthropic's `message_start` usage (input + cache read + cache write) was summed from each stream and compared with `GET /stats` → `tokens`. - Observed result: the author-provided live Anthropic comparison below reports exact billed-input agreement at 61,999 tokens; the baseline undercounts by 37%. This external run was not independently repeated during the current-head review. | | Anthropic billed input | Headroom `/stats` `tokens.input` | |---|---|---| | `main` | 62,001 | **38,835** (−37%) | | this PR | 61,999 | **61,999** (exact), `input_provider_reported_percent: 100.0` | The two runs differ by 2 tokens because they are separate API calls. `proxy_compression_saved` was 29,313 on both, so savings behaviour is unchanged. - Not tested: live Bedrock, Vertex and Gemini streams (unit-tested on provider shapes only), and the licence cloud endpoint receiving the new fields. ## Runtime Rollout Safety - Rollout-managed feature(s): none. This is accounting only, and request and response bytes are unchanged. - Minimum rollout channel: N/A - Stable/default behavior changed: reported input volume now uses the provider's billed count where the provider reports one. New fields are added to `/stats` and the licence usage payload; no existing field is removed or renamed. - Kill switch / disable path: none needed. With no usage frame, the fallback is the previous estimate, now labelled `input_estimated`. - Unsafe override required: no - Qualification impact: dashboards and licence reports will show larger, correct Claude input totals once installs upgrade, so historical and new figures are not directly comparable. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes: targeted suites only; full suite on CI - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - **Full suite.** Locally, the full run on the pre-rebase base reached 2,147 passed before stopping on `tests/test_cli/test_mcp_reconcile.py::test_ordinary_install_does_not_adopt_serena`. That test fails identically on unmodified `main` (unrelated MCP install args). The full local run was then abandoned because several wrap/install tests launch a real `serena project index` (rust-analyzer, tsserver, rustc), which should be stubbed in a separate PR. - **Docs.** The new `/stats` and licence fields are not yet documented. - Follow-ups not in this PR: - **Raw Bedrock invoke pass-through (`handlers/bedrock.py`)** forwards bytes without parsing them, so it still records an estimate. It is now counted as `input_estimated`, so it's visible rather than silent. - **Batch create** has no usage until results come back, so it stays estimated as above. - **`tokens_saved` is still on the `o200k_base` scale**, so it under-reports Claude savings. Calibrating it to provider tokens is a separate PR. - **The licence cloud endpoint** needs to read the new fields. 🤖 Generated with [Claude Code](https://claude.com/claude-code) ## Current maintainer verification (October 8) Exact head: `81527148ed9777e1b9ca921bbf6bb457d49354db`, containing main `01d9194854233ef6ed04005cf0a5b8c32bf65d6d` through a clean merge. No contributor logic was modified. 63 focused tests pass with the existing compatible native extension on Windows/Python 3.12. Three backend-boundary probes preserve zero client input and exact cache-read, cache-write, and combined provider input totals. This is local mocked-backend/TestClient evidence; the author-provided external-provider evidence above was not independently rerun. Current-head hosted snapshot: 31 successful checks, 12 skips, no failed or live checks. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>