Files
a9815bbb5f fix(proxy): record the provider's billed input count on streamed turns (#4052)
## Description

Every input-volume figure Headroom reports for streamed Claude traffic
was Headroom's own tokenizer estimate, not Anthropic's billed count.
That covers "input after Headroom", `/stats` `tokens.input`, the
dashboard's per-model table and the licence usage report's
`tokens_after`. Since Claude Code always streams, this was effectively
all Claude Code traffic, so customers comparing our numbers with the
Anthropic Console could not reconcile them.

**Root cause.** `emit_request_outcome` records `provider_input_tokens or
optimized_tokens`. The streaming finalizer never set
`provider_input_tokens` for Anthropic (#3946 fixed only the uncached
sub-count), so every streamed Claude turn fell back to
`optimized_tokens`. That value is:
- an `o200k_base` count standing in for Claude's private tokenizer,
measured against `count_tokens` on 6 real Claude Code transcripts at
15–25% under Claude 4.x and 35–38% under Claude 5.x;
- a count of the forwarded **messages only**, which never sees the
system prompt or tool definitions.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **`headroom/proxy/provider_usage.py` (new).** One place that knows
each provider's usage semantics:
- Anthropic, Bedrock and Vertex: `input_tokens` is the uncached tail,
and the billed total is `input + cache_read + cache_write`.
- OpenAI `prompt_tokens`/`input_tokens` and Gemini `promptTokenCount`
are already inclusive of cached tokens.
- **Streaming finalizer.** Sets `provider_input_tokens` from the
provider's usage on the Anthropic/OpenAI/Gemini SSE path, the Bedrock
stream path and the OpenAI-via-backend stream path.
`RequestOutcome.from_stream` gains the parameter.
- **OpenAI pass-through.** On both streaming and non-streaming paths,
Anthropic-shape usage was read as if `input_tokens` were the total. It
now uses the billed total and is marked provider-reported.
- **Gemini non-streaming and OpenAI Responses WebSocket.** These already
used the provider's count as `optimized_tokens` but never declared it,
so they now also set `provider_input_tokens`, but only when the provider
actually sent a count.
- **No estimate is ever labelled as billed.** The Bedrock #1132
`message_start` backfill and Gemini's fallback for a missing
`promptTokenCount` both stay estimated. The pass-through usage dialect
is chosen by provider, not inferred from which keys are present.
- **Provenance counters.** Every input total now carries how much of it
is billed versus estimated:
- `/stats` `tokens.input_provider_reported`, `input_estimated`,
`input_provider_reported_percent`, `requests_input_*`.
- Licence usage report: `tokens_after_provider_reported`,
`tokens_after_estimated`, `requests_provider_reported`, plus the billed
`cache_read_tokens`, `cache_write_tokens` and `uncached_input_tokens`,
so the cloud side can reconcile with the console (input + cache read +
cache write). The report also states `tokens_saved_basis:
"headroom_tokenizer"`.
- A one-time `event=input_tokens_estimated` warning per provider/model
when a successful request carried no provider usage.
- **No delta math changes.** `optimized_tokens` stays on the local
scale, and `tokens_saved`, ratios and the ledger delta are untouched.
The provider count is never subtracted from a local count.

## Testing

- [ ] Unit tests pass (`pytest`): targeted suites only, listed below.
The full suite is left to CI (see Additional Notes).
- [x] Linting passes (`ruff check .`): run on all changed files.
- [x] Type checking passes (`mypy headroom`): run on the changed
modules.
- [x] New tests added for new functionality
- [x] Manual testing performed

`tests/test_provider_billed_input.py` adds 15 tests, plus regression
tests for the review fixes in
`tests/test_bedrock_streaming_input_tokens.py` (the #1132 backfill is
reported as estimated) and `tests/test_gemini_ccr_continuation_usage.py`
(a missing `promptTokenCount` is not provider-reported):
- **Recorded fixture.**
`tests/fixtures/anthropic/stream_cached_turn.sse` is a **real recorded
Anthropic stream**: a second call on a cached ~8.9K-token system prompt,
billed as 19 uncached + 8,857 cache-read. It is replayed in 97-byte
chunks through the proxy's own SSE usage parser and
`_finalize_stream_response`, then through `emit_request_outcome` into a
real `PrometheusMetrics`, `CostTracker` and `UsageReporter`. The test
asserts that `/stats` input and the licence `tokens_after` equal
Anthropic's 8,876.
- Other cases:
  - an OpenAI stream records the inclusive `prompt_tokens`;
  - a stream without usage stays estimated;
  - each provider shape is covered;
  - the pass-through normaliser returns the total.

Against the unpatched handlers, the 5 behaviour tests fail. The 8 that
pass exercise only the new helper.

### Test Output

```text
$ pytest tests/test_provider_billed_input.py tests/test_proxy_handler_helpers.py tests/test_request_outcome.py \
    tests/test_proxy_streaming_request_logger.py tests/test_proxy_cache_ttl_metrics.py \
    tests/test_usage_reporter_snapshot.py tests/test_5xx_accounting_all_providers.py tests/test_copilot_provider_label.py \
    tests/test_bedrock_streaming_input_tokens.py tests/test_gemini_ccr_continuation_usage.py -q
181 passed in 6.56s

# same new tests against the unpatched handlers
FAILED tests/test_provider_billed_input.py::test_passthrough_normaliser_returns_anthropic_total_not_uncached_tail
FAILED tests/test_provider_billed_input.py::test_recorded_anthropic_stream_records_anthropics_billed_input
FAILED tests/test_provider_billed_input.py::test_openai_stream_records_inclusive_prompt_tokens
FAILED tests/test_provider_billed_input.py::test_streamed_claude_turn_reaches_dashboard_and_licence_report_as_billed
FAILED tests/test_provider_billed_input.py::test_estimated_input_is_reported_as_estimated
5 failed, 8 passed

$ ruff check <changed files>        -> All checks passed!
$ mypy <changed proxy/telemetry modules> -> Success: no issues found in 6 source files
```

## Real Behavior Proof

- Environment: local proxy from this branch (`headroom proxy --port 8931
--stateless --no-telemetry --no-cache`) against the real Anthropic API.
For the baseline, the same command ran from a `main` worktree on port
8932.
- Exact command / steps: 3 streamed `claude-sonnet-4-6` turns through
the proxy, with a cached ~20K-token system prompt, one tool and a
300-item JSON tool result. Anthropic's `message_start` usage (input +
cache read + cache write) was summed from each stream and compared with
`GET /stats` → `tokens`.
- Observed result: the author-provided live Anthropic comparison below
reports exact billed-input agreement at 61,999 tokens; the baseline
undercounts by 37%. This external run was not independently repeated
during the current-head review.

  | | Anthropic billed input | Headroom `/stats` `tokens.input` |
  |---|---|---|
  | `main` | 62,001 | **38,835** (−37%) |
| this PR | 61,999 | **61,999** (exact),
`input_provider_reported_percent: 100.0` |

The two runs differ by 2 tokens because they are separate API calls.
`proxy_compression_saved` was 29,313 on both, so savings behaviour is
unchanged.
- Not tested: live Bedrock, Vertex and Gemini streams (unit-tested on
provider shapes only), and the licence cloud endpoint receiving the new
fields.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is accounting only, and request
and response bytes are unchanged.
- Minimum rollout channel: N/A
- Stable/default behavior changed: reported input volume now uses the
provider's billed count where the provider reports one. New fields are
added to `/stats` and the licence usage payload; no existing field is
removed or renamed.
- Kill switch / disable path: none needed. With no usage frame, the
fallback is the previous estimate, now labelled `input_estimated`.
- Unsafe override required: no
- Qualification impact: dashboards and licence reports will show larger,
correct Claude input totals once installs upgrade, so historical and new
figures are not directly comparable.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes: targeted
suites only; full suite on CI
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- **Full suite.** Locally, the full run on the pre-rebase base reached
2,147 passed before stopping on
`tests/test_cli/test_mcp_reconcile.py::test_ordinary_install_does_not_adopt_serena`.
That test fails identically on unmodified `main` (unrelated MCP install
args). The full local run was then abandoned because several
wrap/install tests launch a real `serena project index` (rust-analyzer,
tsserver, rustc), which should be stubbed in a separate PR.
- **Docs.** The new `/stats` and licence fields are not yet documented.
- Follow-ups not in this PR:
- **Raw Bedrock invoke pass-through (`handlers/bedrock.py`)** forwards
bytes without parsing them, so it still records an estimate. It is now
counted as `input_estimated`, so it's visible rather than silent.
- **Batch create** has no usage until results come back, so it stays
estimated as above.
- **`tokens_saved` is still on the `o200k_base` scale**, so it
under-reports Claude savings. Calibrating it to provider tokens is a
separate PR.
  - **The licence cloud endpoint** needs to read the new fields.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


## Current maintainer verification (October 8)

Exact head: `81527148ed9777e1b9ca921bbf6bb457d49354db`, containing main
`01d9194854233ef6ed04005cf0a5b8c32bf65d6d` through a clean merge. No
contributor logic was modified. 63 focused tests pass with the existing
compatible native extension on Windows/Python 3.12. Three
backend-boundary probes preserve zero client input and exact cache-read,
cache-write, and combined provider input totals. This is local
mocked-backend/TestClient evidence; the author-provided
external-provider evidence above was not independently rerun.
Current-head hosted snapshot: 31 successful checks, 12 skips, no failed
or live checks.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-09 14:16:49 -07:00
..