feat(savings): show the new-input savings basis beside the whole-wire figure (#3485)

## Description

The proxy already computes two savings bases in `/stats`:
`savings_percent` (saved over everything forwarded, which recounts a
session's cached history every turn and drifts toward 0% on long
sessions) and `new_input_savings_percent` (compression-only saved over
the tokens that newly entered context: provider-billed uncached +
cache-write). Only the first one is shown anywhere: the dashboard
headline and `headroom savings` both render the whole-wire figure. A
user on a 300-turn Claude Code session sees 0.4% while compression is
working on every new turn, and a user comparing the CLI with a client
that reads the new-input field sees two unexplained numbers.

This surfaces the existing new-input figure as a second labelled view
next to the whole-wire one. No calculation changes.

- Dashboard: the headline "Tokens Saved" percentage leads with
`new_input_savings_percent` when the provider reported a cache breakdown
(`new_input_tokens > 0`), with a title explaining the basis; it falls
back to whole-wire otherwise (Bedrock, no cache data) so it never
collapses to 0% while compression is happening (#455). The "Of total
wire" line beneath it is unchanged.
- CLI `headroom savings`: each window line gains `· of new input NN.N%`
when that window has cache data, plus a one-line legend. Windows without
cache data print exactly what they print today.
- Ledger: `record_savings_event` takes optional `new_input_tokens` and
`deferred_tokens`; the proxy call site passes uncached + cache-write and
the tool-search deferral share. Events without a cache breakdown write
the same line as before and do not contribute to the new-input ratio on
either side, so the ratio never pairs one event's savings with another's
denominator. Numerator is compression-only (saved minus deferral),
matching `/stats`.

## Type of Change

- [x] New feature (non-breaking change that adds functionality)

## Changes Made

- `headroom/savings_ledger.py`: optional `new_input` / `deferred` event
fields; `_Bucket.new_input_tokens`, `new_input_saved`,
`new_input_savings_percent`; both exported in `to_dict()`.
- `headroom/proxy/prometheus_metrics.py`: ledger append passes
`new_input_tokens` (None when no cache breakdown) and `deferred_tokens`.
- `headroom/cli/savings.py`: `_window_line` appends the new-input figure
when present; legend line.
- `headroom/dashboard/templates/dashboard.html`: `headlineUsesNewInput`
gate, headline getter and title; stale "attempted denominator" comment
replaced.
- Tests: ledger pairing and no-cache-data isolation, CLI output with and
without cache data, proxy call-site kwargs with and without a cache
breakdown, dashboard headline order and titles.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
uv run --frozen --extra dev pytest tests/test_savings_ledger.py tests/test_dashboard_token_savings.py tests/test_savings_ledger_before_forwarded.py tests/test_savings_ledger_offload.py tests/test_proxy_project_savings.py tests/test_dashboard -q
46 passed, 3 skipped
uvx ruff check headroom/savings_ledger.py headroom/proxy/prometheus_metrics.py headroom/cli/savings.py tests/test_savings_ledger.py tests/test_dashboard_token_savings.py tests/test_savings_ledger_before_forwarded.py
All checks passed!
uv run --frozen --extra dev mypy headroom/savings_ledger.py headroom/cli/savings.py headroom/proxy/prometheus_metrics.py
Success: no issues found in 3 source files
```

## Real Behavior Proof

- Environment: macOS 15, Python 3.12, headroom-ai 0.37.0 as installed by
Headroom Desktop, real Claude Code traffic on 2026-09-08.
- Exact command / steps: Summed PERF lines for one opus-5 session (354
requests, 53.7M tokens forwarded, 45.0M cache reads, 1.17M cache writes)
and compared the two bases; ran `headroom savings` through `CliRunner`
against a temp ledger with one event carrying `new_input_tokens=900` and
one without.
- Observed result: whole-wire 0.42% versus new-input 2.9% on the same
session, which is the gap this view explains. CLI output: the window
line without cache data is byte-identical to today; with cache data it
ends in `· of new input 30.8%` and the legend prints once.
- Not tested: dashboard rendering in a browser (the template getters are
covered by the existing regex tests; no visual change beyond the number
and its tooltip).

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes. The dashboard headline
percentage and `headroom savings` lines show the new-input basis when
cache data exists; the whole-wire figure stays visible beside it. Ledger
lines gain two optional fields; old lines still parse.
- Kill switch / disable path: None; both figures are shown, nothing is
hidden.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert the commit; ledger lines with the extra fields
are ignored by the previous reader.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A

## Additional Notes

Why a second view and not a toggle: the two numbers answer different
questions ("how much of what I sent was cut" versus "how much of what I
newly added was cut") and hiding either one is what caused the
confusion. Documentation: the CLI legend and the dashboard tooltip carry
the definition; happy to add a docs line if you point me at the page.

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Jerrett Davis <mxjerrett@gmail.com>
This commit is contained in:
gglucass
2026-09-24 16:47:03 -05:00
committed by GitHub
co-authored by Claude Fable 5.1 Jerrett Davis
parent fa9edb3a29
commit e92cccca00
10 changed files with 395 additions and 22 deletions
+14 -1
View File
@@ -47,10 +47,18 @@ def _window_line(label: str, window: dict[str, Any]) -> str:
money = _money(cost)
if ceiling > 0 and abs(ceiling - cost) / ceiling >= 0.01:
money = f"{money} (list {_money(ceiling)})"
return (
line = (
f"{label:<12} {_bar(pct)} {pct:5.1f}% "
f"saved {_tokens(saved)} / {_tokens(before)} tokens {money}"
)
# Second basis, only when the window has provider cache data: the share of
# tokens that newly entered context. The bar's ratio counts a session's
# cached history on every turn, so a long session reads near 0% there
# while compression is working; this is the figure the dashboard leads with.
if int(window.get("new_input_tokens", 0) or 0) > 0:
new_pct = float(window.get("new_input_savings_percent", 0.0) or 0.0)
line += f" · of new input {new_pct:.1f}%"
return line
#: What each pricing basis means for a reader deciding how much to trust the
@@ -125,6 +133,11 @@ def savings(as_json: bool, days: int, reset: bool) -> None:
click.echo(_window_line("Today", report.windows["today"]))
click.echo(_window_line("Last 7 days", report.windows["last_7_days"]))
click.echo(_window_line("Last 30 days", report.windows["last_30_days"]))
if any(int(w.get("new_input_tokens", 0) or 0) > 0 for w in report.windows.values()):
click.echo(
" % is of all forwarded input (cached history recounted every turn); "
"'of new input' is of tokens that newly entered context."
)
if report.by_model:
click.echo("")
+11 -9
View File
@@ -225,19 +225,21 @@
<div class="text-xs text-gray-500 uppercase tracking-wide mb-1">Tokens Saved</div>
<div class="flex items-baseline gap-2">
<span class="text-3xl font-light tabular-nums text-accent" x-text="formatNumber(stats.tokens?.saved || 0)"></span>
<!-- Active compression ratio: savings as a fraction of tokens we
*attempted* to compress (extracted units + tool schema). When
the attempted denominator is missing (backend-routed paths
that don't yet wire per-message live-zone tracking), fall back
to the whole-request ratio so the headline doesn't collapse to
0% while compression is happening (issue #455). -->
<!-- Headline ratio: whole wire, savings as a fraction of ALL
forwarded input (#1653). That is the honest answer to "what
share of my billed input did Headroom remove", because
without Headroom the session's cached history would have
been billed too (as cache reads) and so belongs in the
denominator. The new-input basis, which is what compression
can actually touch, sits on the line below; it reads much
higher on long sessions and must not take this slot. -->
<span class="text-sm text-accent" x-text="headlineSavingsPercent.toFixed(1) + '%'" :title="headlineSavingsTitle"></span>
</div>
<div class="mt-1 text-xs text-gray-500 leading-relaxed">
<span x-text="'Proxy ' + formatNumber(stats.tokens?.proxy_compression_saved || 0) + ' (' + proxyShareOfTotal.toFixed(1) + '%)'"></span>
</div>
<div class="mt-1 text-xs text-gray-600 leading-relaxed">
<span x-text="'Of total wire: ' + (stats.tokens?.savings_percent || 0).toFixed(2) + '%'" title="Savings as fraction of all input tokens including frozen prefix"></span>
<div class="mt-1 text-xs text-gray-600 leading-relaxed" x-show="(stats.tokens?.new_input_tokens || 0) > 0">
<span x-text="'Of new input: ' + (stats.tokens?.new_input_savings_percent || 0).toFixed(2) + '%'" title="Savings as a fraction of the tokens that newly entered context (uncached + cache write), which is what compression can touch. The headline above recounts the session's cached history on every turn, so it reads low on a long session even when compression is working."></span>
</div>
<div class="mt-2 h-8">
<svg class="w-full h-full" viewBox="0 0 100 32" preserveAspectRatio="none" role="img"
@@ -2858,7 +2860,7 @@
},
get headlineSavingsTitle() {
return 'Of total wire input tokens';
return 'Of total wire input tokens: all forwarded input, cached history included';
},
// --- Expandable Rows ---
+5
View File
@@ -242,6 +242,11 @@ def build_prefix_cache_stats(
"cache_write_5m_requests": 0,
"cache_write_1h_requests": 0,
"uncached_input_tokens": 0,
# New-input basis, from the metrics object rather than the per-provider
# cache rows: its cohort is "newly billed input", which is not the
# cache-activity cohort those rows are gated on.
"new_input_tokens": int(getattr(metrics, "new_input_tokens_total", 0) or 0),
"new_input_saved_tokens": int(getattr(metrics, "new_input_saved_tokens_total", 0) or 0),
"requests": 0,
"hit_requests": 0,
"bust_count": 0,
+31 -1
View File
@@ -334,6 +334,16 @@ class PrometheusMetrics:
# Track per-model cache request count to distinguish cold starts from busts
self._cache_requests_by_model: dict[str, int] = defaultdict(int)
# New-input basis. The cohort is every request that newly BILLED input
# (uncached or cache-write tokens), which is not the same set as the
# cache accumulators above: those are gated on cache activity, so they
# both admit cache-read-only requests (numerator, no denominator) and
# drop uncached-only ones (real new input, dropped entirely). The
# ledger pairs the same two figures over the same predicate; keeping
# one gate here is what stops /stats and `headroom savings` disagreeing.
self.new_input_tokens_total: int = 0
self.new_input_saved_tokens_total: int = 0
# Prefix freeze stats (cache-aware compression)
self.prefix_freeze_busts_avoided: int = 0
self.prefix_freeze_tokens_preserved: int = 0
@@ -921,6 +931,11 @@ class PrometheusMetrics:
# denominator for the active-compression ratio.
self.attempted_input_tokens_total += max(0, int(attempted_input_tokens))
# New-input cohort, on the same predicate the ledger uses below.
if uncached_input_tokens > 0 or cache_write_tokens > 0:
self.new_input_tokens_total += uncached_input_tokens + cache_write_tokens
self.new_input_saved_tokens_total += max(0, int(tokens_saved))
# Track provider-specific prefix cache metrics
if cache_read_tokens > 0 or cache_write_tokens > 0:
pc = self.cache_by_provider[provider]
@@ -1061,7 +1076,13 @@ class PrometheusMetrics:
# sessions and drops deferral-only turns from the ledger entirely (#2795).
deferral_saved = max(0, int(tool_search_saved))
ledger_saved = tokens_saved + deferral_saved
if ledger_saved > 0 and not self._stateless:
# A request that newly billed input is written even when it saved
# nothing: the ledger's new-input basis needs the denominator from
# every such request (see record_savings_event). Same predicate as the
# `new_input_*_total` accumulators above, so the two rates share a
# cohort — /stats and `headroom savings` are the same measurement.
has_new_input = uncached_input_tokens > 0 or cache_write_tokens > 0
if (ledger_saved > 0 or has_new_input) and not self._stateless:
# `input_tokens` here is the optimized (post-compression) count
# that was actually forwarded — see emit_request_outcome, which
# passes `input_tokens=outcome.optimized_tokens`. The ledger's
@@ -1104,6 +1125,15 @@ class PrometheusMetrics:
uncached_input_tokens=uncached_input_tokens,
cache_inferred=cache_inferred,
provider=provider,
# Provider-billed new input (the /stats new_input denominator)
# and the deferral share of `saved`, so `headroom savings` can
# show the same new-input rate the dashboard headline does.
# Omitted when there is no cache breakdown (e.g. Bedrock), so
# the ledger never divides savings by themselves.
new_input_tokens=(
int(uncached_input_tokens) + int(cache_write_tokens) if has_new_input else None
),
deferred_tokens=deferral_saved,
)
otel_metrics = self._get_otel_metrics()
+16 -9
View File
@@ -4414,9 +4414,14 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
# any power over. Tokens Headroom removed never reached the
# provider at all, so they're added back to form the baseline.
_pc_totals = prefix_cache_stats.get("totals", {})
new_input_tokens = int(_pc_totals.get("uncached_input_tokens", 0) or 0) + int(
_pc_totals.get("cache_write_tokens", 0) or 0
)
new_input_tokens = int(_pc_totals.get("new_input_tokens", 0) or 0)
# Paired numerator: savings from the SAME requests that supplied the
# denominator, accumulated on one predicate in record_request.
# tokens_saved_total also counts requests with no usage breakdown
# (Bedrock, MCP tools), which would lend savings to a denominator they
# never entered: one qualified request at 50 percent plus one
# unqualified 10,000-token saving read as 99 percent.
new_input_saved_tokens = int(_pc_totals.get("new_input_saved_tokens", 0) or 0)
# Build human-readable summary
summary = _build_session_summary(proxy, m, prefix_cache_stats, total_tokens_before)
@@ -4669,14 +4674,16 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
# 200x into the denominator and long-running sessions
# (1M-context models never compact) read as ~0% no
# matter how well compression performs on new content.
# Guarded on new_input_tokens > 0 (not the full sum): the
# cache accumulators only see requests with cache
# activity, so a deployment with no cache metrics (e.g.
# Bedrock) would otherwise divide savings by themselves
# and report ~100%. No usage data -> report 0, not a lie.
# Guarded on new_input_tokens > 0 (not the full sum): a
# deployment that reports no usage breakdown at all (e.g.
# Bedrock) contributes to neither side, and would
# otherwise divide savings by themselves and report
# ~100%. No usage data -> report 0, not a lie. This is
# the same cohort `headroom savings` reports from the
# ledger, so the two figures cannot disagree.
"new_input_tokens": new_input_tokens,
"new_input_savings_percent": round(
(proxy_compression_tokens / (new_input_tokens + proxy_compression_tokens) * 100)
(new_input_saved_tokens / (new_input_tokens + new_input_saved_tokens) * 100)
if new_input_tokens > 0
else 0,
2,
+67 -1
View File
@@ -277,6 +277,8 @@ def record_savings_event(
uncached_input_tokens: Any = 0,
cache_inferred: Any = False,
provider: Any = None,
new_input_tokens: int | None = None,
deferred_tokens: int = 0,
) -> bool:
"""Append one savings event to the durable ledger. Never raises.
@@ -287,6 +289,21 @@ def record_savings_event(
proxy path does; the MCP path does not), both are recorded and the event
also carries a cache-aware ``cost_effective_usd``. ``cost_usd`` keeps its
original list-priced meaning either way, so existing readers are unaffected.
``new_input_tokens`` is the provider-billed input that newly entered
context this request (uncached + cache-write), the denominator /stats
reports as ``new_input_savings_percent``. ``deferred_tokens`` is the part
of ``saved`` that is tool-schema deferral, which rides the cached prefix
and is never part of new input; it is subtracted from the numerator so the
ledger pairs compression-only savings with new input exactly as /stats
does. Both are optional so MCP-tool events and older callers keep writing
the same line they always did.
A request that saved nothing but carries ``new_input_tokens`` is still
written, as a denominator-only observation: the new-input basis must see
every request that newly billed input, not only the ones compression
touched, or a 100-saved/100-new turn followed by a 0-saved/10,000-new turn
reads as 50 percent instead of under 1. Without ``new_input_tokens`` a
zero-saving request is skipped exactly as before.
"""
try:
@@ -296,7 +313,7 @@ def record_savings_event(
return False
saved = max(before - after, 0)
if saved <= 0:
if saved <= 0 and new_input_tokens is None:
return False
model_label = _normalize_model(model)
@@ -320,6 +337,13 @@ def record_savings_event(
"source": str(source or UNKNOWN),
"pid": os.getpid(),
}
if new_input_tokens is not None:
try:
event["new_input"] = max(int(new_input_tokens), 0)
event["deferred"] = min(max(int(deferred_tokens), 0), saved)
except (TypeError, ValueError):
event.pop("new_input", None)
event.pop("deferred", None)
# The savings split. Recorded only when the caller actually knows it —
# absent means "one undifferentiated figure", which is what every v1 event
@@ -443,6 +467,10 @@ class _Bucket:
#: Weakest pricing basis among the events in this bucket, so a total cannot
#: read as better-grounded than its worst contributing event.
basis: str | None = None
# Only events that recorded new input contribute to BOTH of these, so the
# ratio never pairs one event's savings with another event's denominator.
new_input_tokens: int = 0
new_input_saved: int = 0
def add(
self,
@@ -452,7 +480,15 @@ class _Bucket:
cost: float,
cost_effective: float | None = None,
basis: str | None = None,
new_input: int | None = None,
deferred: int = 0,
) -> None:
if saved <= 0 and new_input is not None:
# Denominator-only observation (see record_savings_event): it
# belongs in the new-input basis and nowhere else, so calls,
# before/saved and cost keep their saved-event semantics.
self.new_input_tokens += new_input
return
self.tokens_saved += saved
self.tokens_before += before
self.cost_usd += cost
@@ -464,6 +500,9 @@ class _Bucket:
self.cost_effective_usd += cost if cost_effective is None else cost_effective
self.calls += 1
self.basis = _weakest(self.basis, basis)
if new_input is not None:
self.new_input_tokens += new_input
self.new_input_saved += max(saved - deferred, 0)
@property
def savings_percent(self) -> float:
@@ -471,6 +510,20 @@ class _Bucket:
return 0.0
return round(self.tokens_saved / self.tokens_before * 100, 1)
@property
def new_input_savings_percent(self) -> float:
"""Compression-only savings as a share of what newly entered context.
The whole-wire ratio above recounts a session's cached history every
turn, so long sessions read as ~0% no matter how well compression does
on new content. Same definition as /stats ``new_input_savings_percent``:
saved / (new input + saved), since removed tokens never reached the
provider and are added back to form the baseline.
"""
if self.new_input_tokens <= 0:
return 0.0
return round(self.new_input_saved / (self.new_input_tokens + self.new_input_saved) * 100, 1)
def to_dict(self) -> dict[str, Any]:
return {
"tokens_saved": self.tokens_saved,
@@ -480,6 +533,8 @@ class _Bucket:
"basis": self.basis or BASIS_LIST,
"calls": self.calls,
"savings_percent": self.savings_percent,
"new_input_tokens": self.new_input_tokens,
"new_input_savings_percent": self.new_input_savings_percent,
}
@@ -590,6 +645,15 @@ def aggregate_savings(
# built from them widens to a union that no longer type-checks against
# `add`'s signature. Listed out per bucket rather than via a local
# helper, which would close over these loop variables.
new_input: int | None = None
deferred = 0
if event.get("new_input") is not None:
try:
new_input = max(int(event.get("new_input") or 0), 0)
deferred = max(int(event.get("deferred", 0) or 0), 0)
except (TypeError, ValueError):
new_input = None
targets = [windowed]
if ts >= today_cutoff:
targets.append(today)
@@ -605,6 +669,8 @@ def aggregate_savings(
cost=cost,
cost_effective=effective,
basis=basis,
new_input=new_input,
deferred=deferred,
)
model_rows = _ranked(by_model, "model")
+19 -1
View File
@@ -13,15 +13,33 @@ def _getter_body(html: str, name: str) -> str:
return match.group("body")
def test_token_savings_headline_uses_total_wire_denominator() -> None:
def test_token_savings_headline_leads_with_total_wire_then_new_input() -> None:
html = get_dashboard_html()
# Whole wire keeps the headline slot (#1653): it is the conservative,
# screenshot-able figure. The old attempted-denominator must stay gone.
headline_body = _getter_body(html, "headlineSavingsPercent")
assert "stats.tokens?.savings_percent" in headline_body
assert "stats.tokens?.proxy_savings_percent" in headline_body
assert "stats.tokens?.new_input_savings_percent" not in headline_body
assert "stats.tokens?.active_savings_percent" not in headline_body
assert "stats.tokens?.proxy_attempted_tokens" not in headline_body
title_body = _getter_body(html, "headlineSavingsTitle")
assert "Of total wire input tokens" in title_body
assert "Of compressible tokens attempted" not in title_body
# New input rides the line below, gated on a provider cache breakdown, and
# replaces the old "Of total wire" sublabel that duplicated the headline.
assert "'Of new input: '" in html
assert "'Of total wire: '" not in html
assert 'x-show="(stats.tokens?.new_input_tokens || 0) > 0"' in html
def test_new_input_explanation_lives_in_the_tooltip_not_the_card() -> None:
"""The card shows a number and its basis; the "why" is hover-only."""
html = get_dashboard_html()
card_line = next(line for line in html.splitlines() if "'Of new input: '" in line)
label, _, tooltip = card_line.partition("title=")
assert "what compression can touch" not in label
assert "what compression can touch" in tooltip
+71
View File
@@ -115,6 +115,50 @@ def test_windows_today_week_last30(monkeypatch, tmp_path):
assert report.windows["today"]["savings_percent"] == pytest.approx(50.0)
def test_new_input_basis_pairs_compression_only_with_new_input(monkeypatch, tmp_path):
_events_env(monkeypatch, tmp_path)
# 400 saved, of which 100 is tool-schema deferral; 900 tokens newly entered
# context. Compression-only 300 / (900 + 300) = 25%, same as /stats.
L.record_savings_event(
tokens_before=10_000,
tokens_after=9_600,
model=None,
client="claude-code",
new_input_tokens=900,
deferred_tokens=100,
)
# An event without a cache breakdown (MCP tool, Bedrock) keeps writing the
# old line and must not lend its savings to the new-input ratio.
L.record_savings_event(tokens_before=1000, tokens_after=500, model=None, client="mcp")
report = L.aggregate_savings()
window = report.windows["today"]
assert window["tokens_saved"] == 900
assert window["new_input_tokens"] == 900
assert window["new_input_savings_percent"] == pytest.approx(25.0)
clients = {row["client"]: row for row in report.by_client}
assert clients["mcp"]["new_input_tokens"] == 0
assert clients["mcp"]["new_input_savings_percent"] == 0.0
# The whole-wire ratio is untouched by the new field.
assert window["savings_percent"] == pytest.approx(900 / 11_000 * 100, abs=0.1)
def test_savings_cli_prints_new_input_line_only_with_cache_data(monkeypatch, tmp_path):
from click.testing import CliRunner
from headroom.cli.savings import savings
_events_env(monkeypatch, tmp_path)
L.record_savings_event(tokens_before=1000, tokens_after=500, model=None, client="mcp")
out = CliRunner().invoke(savings, []).output
assert "of new input" not in out
L.record_savings_event(
tokens_before=10_000, tokens_after=9_600, model=None, client="c", new_input_tokens=900
)
out = CliRunner().invoke(savings, []).output
assert "of new input 30.8%" in out
assert "newly entered context" in out
def test_retention_hard_capped_at_30_days(monkeypatch, tmp_path):
_events_env(monkeypatch, tmp_path)
now = datetime(2026, 6, 17, 12, 0, tzinfo=UTC)
@@ -450,3 +494,30 @@ def test_v1_events_still_aggregate_alongside_v2(tmp_path):
# ...but the v1 event's list dollar is still counted, not dropped.
assert lifetime["cost_effective_usd"] > 0.012
assert lifetime["basis"] == "list"
def test_zero_saving_request_keeps_its_place_in_the_new_input_denominator(monkeypatch, tmp_path):
"""100 saved on 100 new input, then 0 saved on 10,000 new input. The
new-input basis is 100 / (10,100 + 100), under 1 percent, not the 50
percent that only counting requests that saved would report. The
saved-event figures do not see the second request at all."""
_events_env(monkeypatch, tmp_path)
assert L.record_savings_event(
tokens_before=200, tokens_after=100, model=None, client="claude-code", new_input_tokens=100
)
assert L.record_savings_event(
tokens_before=10_000,
tokens_after=10_000,
model=None,
client="claude-code",
new_input_tokens=10_000,
)
# Without a new-input figure a zero-saving request is still not an event.
assert not L.record_savings_event(
tokens_before=500, tokens_after=500, model=None, client="claude-code"
)
window = L.aggregate_savings().windows["today"]
assert window["tokens_saved"] == 100
assert window["new_input_tokens"] == 10_100
assert window["new_input_savings_percent"] == pytest.approx(1.0, abs=0.05)
assert window["savings_percent"] == pytest.approx(50.0)
@@ -73,6 +73,27 @@ async def test_record_savings_event_uses_original_input_as_before(
assert call["saved_compression"] == 400
assert call["saved_tool_schema"] == 0
# With a provider cache breakdown the ledger also gets the /stats
# new-input denominator (uncached + cache write) and the deferral share of
# the saving, so `headroom savings` can pair compression-only with new input.
calls.clear()
await metrics.record_request(
provider="anthropic",
model="claude-opus-4-6",
input_tokens=600,
output_tokens=25,
tokens_saved=400,
latency_ms=10.0,
client="claude-code",
cache_read_tokens=5000,
cache_write_tokens=700,
uncached_input_tokens=200,
tool_search_saved=150,
)
assert calls[0]["tokens_before"] == 1000 + 150
assert calls[0]["new_input_tokens"] == 900
assert calls[0]["deferred_tokens"] == 150
@pytest.mark.asyncio
async def test_record_request_forwards_tool_savings_to_otel() -> None:
@@ -161,3 +182,46 @@ async def test_record_savings_event_written_for_deferral_only_turn(
assert len(calls) == 1
assert calls[0]["tokens_before"] == 50000 + 13182
assert calls[0]["tokens_after"] == 50000
@pytest.mark.asyncio
async def test_zero_saving_request_with_cache_breakdown_reaches_the_ledger(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A request that saved nothing but newly billed input is a denominator
observation for the new-input basis, so it must reach the ledger. One
without any breakdown is skipped as before."""
calls: list[dict[str, Any]] = []
monkeypatch.setattr(
prometheus_metrics.savings_ledger,
"record_savings_event",
lambda **kwargs: calls.append(kwargs),
)
metrics = prometheus_metrics.PrometheusMetrics(
savings_tracker=_FakeSavingsTracker(),
otel_metrics=_FakeOtelMetrics(),
)
await metrics.record_request(
provider="anthropic",
model="claude-opus-4-6",
input_tokens=10_000,
output_tokens=25,
tokens_saved=0,
latency_ms=10.0,
client="claude-code",
cache_read_tokens=5_000,
uncached_input_tokens=10_000,
)
assert len(calls) == 1
assert calls[0]["tokens_before"] == calls[0]["tokens_after"]
assert calls[0]["new_input_tokens"] == 10_000
await metrics.record_request(
provider="bedrock",
model="claude-opus-4-6",
input_tokens=10_000,
output_tokens=25,
tokens_saved=0,
latency_ms=10.0,
client="claude-code",
)
assert len(calls) == 1
@@ -11,8 +11,10 @@ from __future__ import annotations
import asyncio
import pytest
from fastapi.testclient import TestClient
from headroom import savings_ledger
from headroom.proxy.server import ProxyConfig, create_app
@@ -79,3 +81,98 @@ def test_stats_new_input_rate_is_zero_without_cache_usage_data(tmp_path, monkeyp
assert tokens["new_input_tokens"] == 0
assert tokens["new_input_savings_percent"] == 0
def test_stats_new_input_rate_pairs_savings_with_qualified_requests(tmp_path, monkeypatch):
"""The numerator must come from the same requests as the denominator. A
request with no cache breakdown (Bedrock, an MCP tool) never enters
new_input_tokens, so its savings must not lend themselves to that ratio:
one qualified 50 percent request plus one unqualified 10,000-token saving
used to read as 99 percent."""
with _make_client(tmp_path, monkeypatch) as client:
proxy = client.app.state.proxy
asyncio.run(
proxy.metrics.record_request(
provider="anthropic",
model="claude-opus-4-6",
input_tokens=1_100,
output_tokens=10,
tokens_saved=100,
latency_ms=10.0,
cache_read_tokens=1_000,
uncached_input_tokens=100,
)
)
assert client.get("/stats").json()["tokens"]["new_input_savings_percent"] == 50.0
asyncio.run(
proxy.metrics.record_request(
provider="bedrock",
model="claude-opus-4-6",
input_tokens=20_000,
output_tokens=10,
tokens_saved=10_000,
latency_ms=10.0,
)
)
tokens = client.get("/stats").json()["tokens"]
assert tokens["new_input_tokens"] == 100
assert tokens["new_input_savings_percent"] == 50.0
# The whole-wire figures still count the unqualified request.
assert tokens["saved"] >= 10_100
def test_stats_and_ledger_share_one_new_input_cohort(tmp_path, monkeypatch):
"""The dashboard rate and `headroom savings` must be the same measurement.
/stats accumulated its new-input pair under a cache-ACTIVITY gate while the
ledger writes under a newly-BILLED-input one, so the two admitted different
requests. An uncached-only turn (real new input, no cache read or write)
reached the ledger and nothing else, and a cache-read-only turn lent its
savings to the dashboard numerator with no denominator to match. The
reviewer's case: 100 saved / 100 new / 1,000 cache-read followed by
0 saved / 10,000 new left the dashboard at 50% and the ledger near 1%.
"""
ledger_path = tmp_path / "savings_events.jsonl"
monkeypatch.setattr(savings_ledger, "_resolve_path", lambda path=None: ledger_path)
with _make_client(tmp_path, monkeypatch) as client:
proxy = client.app.state.proxy
async def record(**kwargs) -> None:
await proxy.metrics.record_request(
provider="anthropic",
model="claude-opus-4-6",
output_tokens=10,
latency_ms=10.0,
**kwargs,
)
# Cache-read plus a little new input: in both cohorts.
asyncio.run(
record(
input_tokens=1_100,
tokens_saved=100,
cache_read_tokens=1_000,
uncached_input_tokens=100,
)
)
# Uncached-only: real new input, no cache activity at all. Used to be
# a ledger-only observation.
asyncio.run(record(input_tokens=10_000, tokens_saved=0, uncached_input_tokens=10_000))
# Cache-read-only: no new input, so it belongs to neither side despite
# saving tokens. Used to inflate the dashboard numerator alone.
asyncio.run(record(input_tokens=2_000, tokens_saved=500, cache_read_tokens=2_000))
tokens = client.get("/stats").json()["tokens"]
lifetime = savings_ledger.aggregate_savings(path=ledger_path).lifetime
assert tokens["new_input_tokens"] == 10_100
assert lifetime["new_input_tokens"] == 10_100
# 100 saved / (10,100 new + 100 saved) — not the old 50%.
assert tokens["new_input_savings_percent"] == pytest.approx(0.98, abs=0.05)
assert lifetime["new_input_savings_percent"] == pytest.approx(
tokens["new_input_savings_percent"], abs=0.05
)
# The cache-read-only turn's savings still count on the whole-wire figure.
assert tokens["saved"] == 600