fix(proxy): bill the whole prompt on the gateway path so savings read true (#3632)

## Description

`/stats` reported **97% of tokens saved** on a real Claude Code session
through Kong. The compression is real, the percentage was not.

Anthropic reports a cached turn as `input_tokens` = the *uncached
remainder*, with the cached prefix in `cache_read_input_tokens` /
`cache_creation_input_tokens`. The direct Anthropic handler already sums
all three into `provider_input_tokens`
([handlers/anthropic.py:3587](https://github.com/headroomlabs-ai/headroom/blob/main/headroom/proxy/handlers/anthropic.py#L3587)).
The gateway path recorded `usage.input_tokens` alone, so `tokens.input`
held the uncached sliver and every ratio over it was inflated. On a
fresh 5-call run the old formula reports **100.0%**, because the
denominator was 10 tokens against an 113,490-token prompt.

Affects anything speaking the `/v1/compress` contract: the Kong plugin,
the LiteLLM package, custom gateways. The direct proxy path was already
correct.

Second, smaller problem: tokens alone mislead here even when counted
right. Deferred tool schemas sit at the front of the prompt, so after
turn 1 they would have been cache reads at roughly a tenth of list
input. "90% of tokens" is nowhere near "90% of the bill" (measured on
this laptop's demo traffic: 604 calls, ~$3.12 saved of ~$23.34, about
13%).

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `gateway_turn.complete_outcome`: records the whole billed prompt in
`provider_input_tokens` and the remainder in `uncached_input_tokens`,
matching the direct handler
- `gateway_turn.normalize_usage`: new `input_includes_cache`, decided by
the presence of Anthropic's cache fields rather than the key name.
OpenAI's Responses API also calls its field `input_tokens` but already
counts the cached prefix in it, so keying off the name would
double-count every Responses prompt
- `/stats` `savings.by_layer.tool_search.usd`: the cache-aware dollar
value of deferral, the same figure the cost card already uses
- `/stats` `tokens.cost_weighted_savings_percent`: the saving in money
beside the saving in tokens
- dashboard: a "Cost-weighted: X%" line under Tokens Saved, and the "Of
total wire" tooltip now says the denominator includes the cached prefix
- tests: gateway shapes (Anthropic split, OpenAI chat, OpenAI
Responses), and `/stats` honesty on cache-heavy traffic

No request is shaped differently and no bill changes. Reporting only.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/gateway
250 passed, 4 skipped in 122.95s

$ pytest tests/test_gateway_turn_unit.py tests/test_dashboard_token_savings.py \
         tests/test_dashboard_accessibility.py tests/test_dashboard_static_assets.py
64 passed in 10.58s

$ pytest tests/gateway tests/test_gateway_turn_unit.py tests/test_stats_new_input_savings_rate.py \
         tests/test_stats_tool_schema_savings_are_honest.py tests/test_proxy_savings_history.py \
         tests/test_proxy_stats_recent_requests.py tests/test_proxy_dashboard_stats_cache.py \
         tests/test_cost_card_attribution.py tests/test_cost_tracker_totals.py \
         tests/test_cost_budget_total_prompt.py tests/test_cost_tracker_counterfactual.py
376 passed, 5 skipped   (one pre-existing failure, test_outcome_anthropic_cache_fields, asserted
                         provider_input_tokens == 300 for a 2,800-token prompt: updated here)

# the new tests, without the source change
FAILED tests/test_gateway_turn_unit.py::test_complete_outcome_bills_the_whole_prompt_on_a_cached_anthropic_turn
FAILED tests/test_gateway_turn_unit.py::test_complete_outcome_does_not_double_count_openai_responses_cached_tokens
FAILED tests/test_gateway_turn_unit.py::test_complete_outcome_does_not_double_count_openai_cached_tokens
FAILED tests/test_stats_tool_schema_savings_are_honest.py::test_stats_publishes_the_cost_weighted_saving_next_to_the_token_saving
FAILED tests/test_stats_tool_schema_savings_are_honest.py::test_tool_schema_layer_carries_its_dollar_value

$ ruff check / ruff format --check (0.16.7) on the changed files
All checks passed! / 5 files already formatted
$ mypy --ignore-missing-imports headroom/proxy/gateway_turn.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, Headroom built from this branch,
licensed extensions installed, Kong 3.9 (DB-less,
`kong-plugin-headroom`) in Docker in front of `api.anthropic.com`,
Claude Code 2.1.273 as the client
- Exact command / steps: same stack before and after the change; Claude
Code fixes three planted bugs in a small Python project through Kong (5
model calls), then `GET /stats`
- Observed result, one run on the fixed build:

```text
billed prompt 113,490 = cache_read 86,430 + cache_write 27,050 + uncached 10
old formula would have reported: 100.0%   (denominator was just the 10 uncached tokens)
fixed formula reports:            32.52%
tokens.cost_weighted_savings_percent: 27.7%
savings.by_layer.tool_search: 54,590 tokens, $0.043
cost card: paid $0.10, saved $0.04 (27.7%)
```

- On a longer real session through the same stack (110 calls, Opus +
Sonnet), the old figure was 97%; recomputing that session's rows at
provider cache rates gives ~15% of input tokens and ~13% of spend
- Not tested: non-Anthropic gateways end to end (OpenAI and Responses
shapes are covered by unit tests over `normalize_usage` /
`complete_outcome`, not a live gateway run)

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: reported numbers only. Gateway users'
"tokens saved %" drops sharply (97% -> ~15% on a long session) because
it was wrong in our favour. Worth a release note so nobody reads it as a
regression
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: `tokens.input` on the gateway path now counts
the cached prefix, so any external dashboard differencing it against
`original_tokens` sees a step change
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Found while building a Kong + Headroom demo: the dashboard claimed 90%+
tokens saved while the invoice moved about 13%. The dollar figures were
already cache-aware; only the token percentage was wrong, plus the
missing per-layer dollars.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Tejas Chopra
2026-10-04 15:43:11 -07:00
committed by GitHub
co-authored by Claude Opus 5 JerrettDavis
parent 0b7dab47c6
commit befdb52fd5
6 changed files with 216 additions and 11 deletions
@@ -248,6 +248,9 @@
<div class="mt-1 text-xs text-gray-600 leading-relaxed" x-show="(stats.tokens?.new_input_tokens || 0) > 0">
<span x-text="'Of new input: ' + (stats.tokens?.new_input_savings_percent || 0).toFixed(2) + '%'" title="Savings as a fraction of the tokens that newly entered context (uncached + cache write), which is what compression can touch. The headline above recounts the session's cached history on every turn, so it reads low on a long session even when compression is working."></span>
</div>
<div class="mt-1 text-xs text-gray-600 leading-relaxed">
<span x-text="'Cost-weighted: ' + (stats.tokens?.cost_weighted_savings_percent || 0).toFixed(1) + '%'" title="The same saving in money. Lower than the token share because most removed tokens would have been cheap prefix-cache reads."></span>
</div>
<div class="mt-2 h-8">
<svg class="w-full h-full" viewBox="0 0 100 32" preserveAspectRatio="none" role="img"
:aria-label="'Savings trend sparkline, ' + (stats.savings_history || []).length + ' recent samples; latest ' + (stats.tokens?.savings_percent || 0).toFixed(1) + ' percent'">
+40 -3
View File
@@ -231,6 +231,13 @@ class NormalizedUsage:
output_tokens: int | None = None
cache_read: int | None = None
cache_write: int | None = None
# Whether ``input_tokens`` already counts the cached prompt. Anthropic's
# ``input_tokens`` is the uncached remainder (reads and writes are reported
# beside it); OpenAI's ``prompt_tokens`` is the whole prompt with
# ``cached_tokens`` as a subset of it. Billing the prompt correctly needs to
# know which of the two arrived, so normalize_usage records it rather than
# letting each consumer guess.
input_includes_cache: bool = False
@property
def has_cache_signal(self) -> bool:
@@ -327,6 +334,16 @@ def normalize_usage(usage: dict[str, Any]) -> NormalizedUsage:
input_tokens = _usage_int(usage, "input_tokens")
if input_tokens is None:
input_tokens = _usage_int(usage, "prompt_tokens")
# Anthropic alone reports the prompt split in two: ``input_tokens`` is the
# uncached remainder and the cached part sits in its own two counters.
# Everywhere else the reported prompt total already contains the cached
# prefix — including OpenAI's Responses API, which names its field
# ``input_tokens`` too, so the key name cannot be the discriminator. Keying
# off Anthropic's cache fields can misread only a relay that mixes the two
# vocabularies, and then it errs towards not inflating the prompt.
input_includes_cache = not (
"cache_read_input_tokens" in usage or "cache_creation_input_tokens" in usage
)
output_tokens = _usage_int(usage, "output_tokens")
if output_tokens is None:
output_tokens = _usage_int(usage, "completion_tokens")
@@ -351,6 +368,7 @@ def normalize_usage(usage: dict[str, Any]) -> NormalizedUsage:
output_tokens=output_tokens,
cache_read=cache_read,
cache_write=cache_write,
input_includes_cache=input_includes_cache,
)
@@ -1358,14 +1376,33 @@ def complete_outcome(
tokenizer scale), output and the two cache counters as reported, provider
status so a 5xx is funnelled as a failure, and the gateway's provider
latency added to the compress-side wall clock.
``provider_input_tokens`` is the WHOLE billed prompt, cached part included,
matching what the direct Anthropic handler records. Anthropic's
``input_tokens`` counts only the uncached remainder, so on that shape the
cache counters are added back; every other shape already contains them, so
the cached part is subtracted to get ``uncached_input_tokens``.
Recording only the uncached sliver here left every ratio built on the input
counter (notably ``/stats`` ``tokens.savings_percent``) dividing savings by
a denominator tens of times too small on cache-heavy agent traffic.
"""
usage = usage or NormalizedUsage()
cache_read = usage.cache_read or 0
cache_write = usage.cache_write or 0
reported_input = usage.input_tokens or 0
if usage.input_includes_cache:
billed_input = reported_input
uncached_input = max(reported_input - cache_read - cache_write, 0)
else:
billed_input = reported_input + cache_read + cache_write
uncached_input = reported_input
return dataclasses.replace(
draft,
output_tokens=usage.output_tokens or 0,
provider_input_tokens=usage.input_tokens or 0,
cache_read_tokens=usage.cache_read or 0,
cache_write_tokens=usage.cache_write or 0,
provider_input_tokens=billed_input,
uncached_input_tokens=uncached_input,
cache_read_tokens=cache_read,
cache_write_tokens=cache_write,
status_code=status,
total_latency_ms=draft.total_latency_ms + float(latency_ms or 0.0),
)
+26 -2
View File
@@ -4736,6 +4736,14 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
# Build unified savings summary (all layers)
cache_net_usd = prefix_cache_stats.get("totals", {}).get("net_savings_usd", 0.0)
# Cache-aware dollar value of tool-schema deferral, the same figure the
# cost card uses. Exposed beside the token count because tokens alone
# mislead: deferred schemas would mostly have been cache reads, an order
# of magnitude cheaper than list input, so a big token number is a small
# dollar number. The token count without it invited "we removed 90% of
# the tokens, so we saved 90% of the cost".
_cost_tracker_stats = proxy.cost_tracker.stats() if proxy.cost_tracker else {}
tool_schema_usd = float(_cost_tracker_stats.get("tool_savings_usd", 0.0) or 0.0)
total_tokens_all_layers = all_layers_tokens_saved
persistent_savings = m.savings_tracker.stats_preview()
display_session = persistent_savings.get("display_session", {})
@@ -4864,13 +4872,18 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
"tokens_saved": tool_schema_tokens,
"requests": tool_schema_requests,
"window": len(recent_request_logs),
# Priced at what those tokens would actually have been
# billed at (provider cache-read rate for the warm turns,
# cache-write for the cold one), not at list input.
"lifetime_usd": round(tool_schema_usd, 4),
"description": (
"Tool-definition tokens kept out of the model's context "
"by deferring heavy tool schemas until they're searched "
"for. Counted only when Headroom performed the deferral — "
"not when the client (e.g. Claude Code / Codex) already "
"had tool search enabled. Aggregated over the recent "
"request window."
"had tool search enabled. Token and request counts use "
"the recent request window; lifetime_usd covers the "
"full process lifetime."
),
},
},
@@ -4972,6 +4985,17 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
else 0,
2,
),
# The same saving expressed in money rather than tokens, so a
# reader is never left to assume the two match. They don't, and
# not by a little: most tokens Headroom removes would have been
# prefix-cache reads, billed at roughly a tenth of list input
# (and less on some providers), so a large token share is a much
# smaller cost share. Mirrors summary.cost.savings_pct, which
# prices every layer at the rate its tokens would really have
# cost.
"cost_weighted_savings_percent": float(
(summary.get("cost") or {}).get("savings_pct", 0.0) or 0.0
),
},
"latency": {
"average_ms": avg_latency_ms,
+4 -1
View File
@@ -375,7 +375,10 @@ def test_outcome_anthropic_cache_fields(headroom_client, outcome_spy) -> None:
assert o.output_tokens == 12
assert o.cache_read_tokens == 2000
assert o.cache_write_tokens == 500
assert o.provider_input_tokens == 300
# Anthropic's input_tokens is the uncached remainder; the billed prompt is
# that plus both cache buckets, as the direct handler has always recorded.
assert o.provider_input_tokens == 2800
assert o.uncached_input_tokens == 300
def test_outcome_recorded_immediately_without_relay(headroom_client, outcome_spy) -> None:
+57 -5
View File
@@ -434,7 +434,8 @@ def test_normalize_openai_chat_shape() -> None:
"prompt_tokens_details": {"cached_tokens": 64},
}
)
assert u == NormalizedUsage(100, 20, 64, None)
# OpenAI-shaped: prompt_tokens already counts the cached prefix.
assert u == NormalizedUsage(100, 20, 64, None, input_includes_cache=True)
assert u.has_cache_signal and u.should_apply
@@ -442,17 +443,19 @@ def test_normalize_openai_responses_shape() -> None:
u = normalize_usage(
{"input_tokens": 10, "output_tokens": 2, "input_tokens_details": {"cached_tokens": 8}}
)
assert u == NormalizedUsage(10, 2, 8, None)
# Responses names the field ``input_tokens`` like Anthropic but, like the
# rest of OpenAI, already counts the cached prefix inside it.
assert u == NormalizedUsage(10, 2, 8, None, input_includes_cache=True)
def test_normalize_kong_flat_shape() -> None:
u = normalize_usage({"prompt_tokens": 50, "completion_tokens": 5, "cached_tokens": 40})
assert u == NormalizedUsage(50, 5, 40, None)
assert u == NormalizedUsage(50, 5, 40, None, input_includes_cache=True)
def test_normalize_nested_usage_descends_once() -> None:
u = normalize_usage({"id": "x", "usage": {"prompt_tokens": 7, "completion_tokens": 3}})
assert u == NormalizedUsage(7, 3, None, None)
assert u == NormalizedUsage(7, 3, None, None, input_includes_cache=True)
assert not u.has_cache_signal and not u.should_apply
@@ -546,7 +549,10 @@ def test_complete_outcome_fills_only_existing_fields() -> None:
draft = dataclasses.replace(_outcome(), total_latency_ms=5.0)
done = gt.complete_outcome(draft, NormalizedUsage(120, 30, 80, 4), status=200, latency_ms=250)
assert done.output_tokens == 30
assert done.provider_input_tokens == 120
# Anthropic shape: input_tokens is the uncached remainder, so the billed
# prompt is that plus the two cache counters.
assert done.provider_input_tokens == 204
assert done.uncached_input_tokens == 120
assert done.optimized_tokens == 50 # never overwritten by the provider count
assert (done.cache_read_tokens, done.cache_write_tokens) == (80, 4)
assert done.total_latency_ms == 255.0
@@ -555,6 +561,52 @@ def test_complete_outcome_fills_only_existing_fields() -> None:
assert failed.status_code == 529 and failed.output_tokens == 0
def test_complete_outcome_bills_the_whole_prompt_on_a_cached_anthropic_turn() -> None:
"""A warm agent turn is ~all cache read; billing only the uncached sliver
made ``/stats`` divide savings by a denominator tens of times too small."""
usage = normalize_usage(
{
"input_tokens": 3,
"output_tokens": 210,
"cache_read_input_tokens": 41_000,
"cache_creation_input_tokens": 1_200,
}
)
done = gt.complete_outcome(_outcome(), usage, status=200, latency_ms=100)
assert done.provider_input_tokens == 42_203
assert done.uncached_input_tokens == 3
def test_complete_outcome_does_not_double_count_openai_responses_cached_tokens() -> None:
"""The Responses shape uses Anthropic's field name for an OpenAI-style total."""
usage = normalize_usage(
{
"input_tokens": 12_000,
"output_tokens": 40,
"input_tokens_details": {"cached_tokens": 9_000},
}
)
done = gt.complete_outcome(_outcome(), usage, status=200, latency_ms=100)
assert done.provider_input_tokens == 12_000
assert done.uncached_input_tokens == 3_000
def test_complete_outcome_does_not_double_count_openai_cached_tokens() -> None:
"""OpenAI's prompt_tokens already contains cached_tokens."""
usage = normalize_usage(
{
"prompt_tokens": 12_000,
"completion_tokens": 40,
"prompt_tokens_details": {"cached_tokens": 9_000},
}
)
assert usage.input_includes_cache is True
done = gt.complete_outcome(_outcome(), usage, status=200, latency_ms=100)
assert done.provider_input_tokens == 12_000
assert done.uncached_input_tokens == 3_000
assert done.cache_read_tokens == 9_000
def test_compact_gateway_tools_is_deterministic_and_isolated() -> None:
tools = [
{
@@ -0,0 +1,86 @@
"""Tool-schema deferral must not read as a near-total saving in /stats.
Deferral withholds the same schemas on every turn, so its token count grows
with the session while the tokens it removed would, after the first turn, have
been prefix-cache reads. Two things therefore have to hold:
* the token percentage divides by every input token the provider billed, cached
prefix included — a gateway turn that reports only its uncached remainder as
input made this read ~97% on real Claude Code traffic;
* the same saving is published in money as well, priced at cache rates, because
the token share and the cost share are not the same number.
"""
from __future__ import annotations
import asyncio
from fastapi.testclient import TestClient
from headroom.proxy.server import ProxyConfig, create_app
def _make_client(tmp_path, monkeypatch) -> TestClient:
monkeypatch.setenv("HEADROOM_SAVINGS_PATH", str(tmp_path / "proxy_savings.json"))
config = ProxyConfig(cache_enabled=False, rate_limit_enabled=False, log_requests=False)
return TestClient(create_app(config))
def _record_warm_agent_turns(proxy, turns: int = 20) -> None:
"""One cold turn then warm ones, each withholding the same tool schemas."""
for turn in range(turns):
cache_read = 0 if turn == 0 else 40_000
cache_write = 40_000 if turn == 0 else 500
asyncio.run(
proxy.metrics.record_request(
provider="anthropic",
model="claude-sonnet-4-5",
# The whole billed prompt, as every handler now reports it.
input_tokens=cache_read + cache_write + 20,
output_tokens=150,
tokens_saved=0,
latency_ms=10.0,
cache_read_tokens=cache_read,
cache_write_tokens=cache_write,
uncached_input_tokens=20,
tool_search_saved=10_000,
)
)
def test_token_percentage_counts_the_cached_prefix_in_the_denominator(tmp_path, monkeypatch):
with _make_client(tmp_path, monkeypatch) as client:
proxy = client.app.state.proxy
_record_warm_agent_turns(proxy)
tokens = client.get("/stats").json()["tokens"]
# 200k withheld against ~800k billed: a fifth, not the ~97% that came out
# when the denominator held only the uncached remainder of each turn.
assert tokens["saved"] == 200_000
assert tokens["input"] > 700_000
assert 15.0 < tokens["savings_percent"] < 30.0
assert tokens["savings_percent"] == tokens["all_layers_savings_percent"]
def test_stats_publishes_the_cost_weighted_saving_next_to_the_token_saving(tmp_path, monkeypatch):
with _make_client(tmp_path, monkeypatch) as client:
proxy = client.app.state.proxy
_record_warm_agent_turns(proxy)
payload = client.get("/stats").json()
tokens = payload["tokens"]
assert "cost_weighted_savings_percent" in tokens
# Removed tokens are mostly cache reads, so the money share is the smaller
# of the two. Equal values would mean the cost side priced them at list.
assert tokens["cost_weighted_savings_percent"] < tokens["savings_percent"]
def test_tool_schema_layer_carries_its_dollar_value(tmp_path, monkeypatch):
with _make_client(tmp_path, monkeypatch) as client:
proxy = client.app.state.proxy
_record_warm_agent_turns(proxy)
layer = client.get("/stats").json()["savings"]["by_layer"]["tool_search"]
assert layer["tokens"] >= 0 # window-scoped; the dollars are lifetime
assert "usd" not in layer
assert layer["lifetime_usd"] >= 0.0