feat(savings): carry exact cache-read cost in history rollups (#3734)

## Description

Hourly and daily savings rollups (bucket, `by_provider`, `by_model`) now
carry `cache_read_tokens_delta`, `cache_savings_usd_delta` and
`cache_read_cost_usd_delta`.

A consumer that wants the input spend Headroom can act on has to take
cache reads out of `total_input_cost_usd`, and the rollup gives it no
exact way to do that:

- **No read cost.** The only cache figure available is the read
*discount*, and backing the read cost out as `discount / 9` assumes
reads bill at 0.1x list. They bill at 0.025x on `claude-fable-5-1` and
0.05x on `claude-opus-5-5`, so the derived read cost is 4.3x and 2.1x
too high, the uncached spend comes out too low, and any rate built on it
is inflated.
- **No per-provider split.** With only a bucket-level figure, a consumer
has to apply one ratio to every provider, which misattributes spend
whenever two providers with different cache profiles share an hour.

`cache_savings_usd_delta` has the same name and semantics as the field
#2245 adds, so the two PRs compose.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `_build_rollup` diffs the cumulative `cache_read_tokens` /
`cache_savings_usd` on each checkpoint and adds the three fields to
every bucket, `by_provider` and `by_model` entry.
- The read cost is priced per checkpoint from its recorded model with
`_estimate_input_cost_usd`, the function that priced
`total_input_cost_usd`, memoized per model within one build.
- Checkpoints written before per-model attribution have no model to
price by; a bucket or breakdown containing one reports
`cache_read_cost_usd_delta: null` rather than a guess.
- Buckets and breakdowns now also appear for checkpoints whose only
movement was cache reads, matching the history append gate that already
records those.
- New `_empty_cache_delta()` helper for the zeroed fields.
- Two tests in `tests/test_proxy_savings_history.py`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_proxy_savings_history.py -q
======================== 41 passed, 1 warning in 6.02s =========================
$ uvx ruff check headroom/proxy/savings_tracker.py tests/test_proxy_savings_history.py
All checks passed!
$ uvx ruff format --check headroom/proxy/savings_tracker.py tests/test_proxy_savings_history.py
2 files already formatted
$ uv run --frozen --extra dev mypy headroom
headroom/integrations/langchain/chat_model.py:394: error: Function "RunnableBinding" could always be true in boolean context  [truthy-function]
Found 1 error in 1 file (checked 538 source files)
# the one mypy error is pre-existing and fails identically on main; savings_tracker.py is clean

# the two new tests against the unpatched savings_tracker.py (main):
FAILED tests/test_proxy_savings_history.py::test_savings_tracker_rollup_carries_exact_cache_read_cost_per_provider
FAILED tests/test_proxy_savings_history.py::test_savings_tracker_rollup_read_cost_unknown_for_model_less_checkpoints
```

## Real Behavior Proof

- Environment: macOS, headroom-ai 0.38.0 wheel as installed by Headroom
Desktop, real litellm catalog, the change exec'd into the installed
`savings_tracker` module; a live proxy's own
`/stats-history?history_mode=full` (5,000 stored checkpoints) and
`/transformations/feed?include_messages=0` from 2026-09-23.
- Exact command / steps: rebuilt that day's hourly rollup from the live
stored history with the patched `_build_rollup`; then matched 91 feed
entries to their checkpoints by timestamp, model and exact cache-read
count, and compared each request's recorded input cost against its read
cost from `cache_read_cost_usd_delta` and from `discount / 9`.
- Observed result: `discount / 9` put the read cost ABOVE the whole
request's input cost on 68 of 91 requests (impossible); the exact read
cost did so on 0 of 91. On clean `claude-opus-5-5` requests the
remainder after subtracting the exact read cost is exactly $5.00 per
MTok of new input, which is its cache-write price (1.25 x $4). Per
provider for the day: anthropic input rate 19.4% under `/ 9` vs 12.6%
exact; openai (0.1x reads, where `/ 9` is exact) differs only by the
per-provider split.
- Not tested: the dashboard templates (untouched; they read no new
keys); weekly and monthly rollups beyond the shared code path; providers
other than Anthropic and OpenAI.

## Runtime Rollout Safety

- Rollout-managed feature(s): None. Additive fields on existing
`/stats-history` rollup entries.
- Minimum rollout channel: N/A, not rollout-gated.
- Stable/default behavior changed: No existing key or value changes.
Three keys are added per rollup entry, and a bucket whose only movement
was cache reads now appears where it was previously omitted.
- Kill switch / disable path: None needed; consumers that do not read
the new keys are unaffected.
- Unsafe override required: No.
- Qualification impact: None; no compression, routing or cache-path
behaviour changes. `/stats` is untouched.
- Rollback path: Revert this PR. No persisted state or schema changes;
rollups are recomputed from stored history on every request.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A, no UI change.

## Additional Notes

- Documentation: no docs describe the rollup's per-entry fields, so
there was nothing to update.
- The wider rollup-consuming suites have 6 failures
(`test_cost_card_attribution.py`, `test_models.py::test_claude_info`)
that fail identically on `main`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
gglucass
2026-09-23 13:39:10 +02:00
committed by GitHub
co-authored by Claude Opus 5.5
parent d05bacb447
commit 1cb779e1bc
2 changed files with 183 additions and 2 deletions
+83 -2
View File
@@ -208,6 +208,19 @@ def _normalize_provider(value: Any) -> str:
MODEL_UNKNOWN = "unknown"
def _empty_cache_delta() -> dict[str, Any]:
"""Zeroed cache fields for a rollup bucket or one of its breakdowns.
``cache_read_cost_usd_delta`` is None when any contributing checkpoint's
reads could not be priced (see ``_build_rollup``).
"""
return {
"cache_read_tokens_delta": 0,
"cache_savings_usd_delta": 0.0,
"cache_read_cost_usd_delta": 0.0,
}
def _normalize_model(value: Any) -> str:
"""Normalize a model label, falling back to a stable sentinel.
@@ -1918,6 +1931,45 @@ class SavingsTracker:
prev_total_input_cost_usd = 0.0
prev_output_tokens = 0
prev_output_usd = 0.0
prev_cache_read_tokens = 0
prev_cache_savings_usd = 0.0
# What the bucket's cache reads actually COST, priced per checkpoint
# with the same function that put them into ``total_input_cost_usd``.
# Consumers need it to take reads out of the input bill, and cannot
# derive it from ``cache_savings_usd``: the read discount is not a
# fixed multiple of the read cost (reads bill at 0.1x on most models,
# 0.05x or 0.025x on others), so "discount / 9" misprices exactly the
# models with the steepest cache discount.
read_cost_per_token: dict[str, float] = {}
def _read_cost(model: str, reads: int) -> float | None:
if reads <= 0:
return 0.0
# Checkpoints written before per-model attribution carry no model,
# so their reads cannot be priced the way the request was. Report
# the bucket's read cost as unknown rather than guess.
if model == MODEL_UNKNOWN:
return None
if model not in read_cost_per_token:
read_cost_per_token[model] = (
_estimate_input_cost_usd(model, 1_000_000, cache_read_tokens=1_000_000)
/ 1_000_000
)
return reads * read_cost_per_token[model]
def _add_cache(
target: dict[str, Any], reads: int, discount: float, cost: float | None
) -> None:
target["cache_read_tokens_delta"] += reads
target["cache_savings_usd_delta"] = round(
target["cache_savings_usd_delta"] + discount, 6
)
if cost is None or target["cache_read_cost_usd_delta"] is None:
target["cache_read_cost_usd_delta"] = None
else:
target["cache_read_cost_usd_delta"] = round(
target["cache_read_cost_usd_delta"] + cost, 6
)
for point in history:
timestamp = _parse_timestamp(point["timestamp"])
@@ -1944,6 +1996,15 @@ class SavingsTracker:
delta_output_tokens = max(total_output_tokens - prev_output_tokens, 0)
delta_output_usd = max(total_output_usd - prev_output_usd, 0.0)
total_cache_read_tokens = _coerce_int(point.get("cache_read_tokens"))
total_cache_savings_usd = _coerce_float(point.get("cache_savings_usd"))
delta_cache_read_tokens = max(total_cache_read_tokens - prev_cache_read_tokens, 0)
delta_cache_savings_usd = max(total_cache_savings_usd - prev_cache_savings_usd, 0.0)
prev_cache_read_tokens = total_cache_read_tokens
prev_cache_savings_usd = total_cache_savings_usd
model = _normalize_model(point.get("model"))
delta_cache_read_cost_usd = _read_cost(model, delta_cache_read_tokens)
prev_total_tokens = total_tokens_saved
prev_total_usd = total_usd
prev_total_input_tokens = total_input_tokens
@@ -1965,6 +2026,7 @@ class SavingsTracker:
"total_input_cost_usd": total_input_cost_usd,
"output_tokens_saved_delta": 0,
"output_savings_usd_delta": 0.0,
**_empty_cache_delta(),
"by_provider": {},
"by_model": {},
},
@@ -1988,12 +2050,21 @@ class SavingsTracker:
entry["output_savings_usd_delta"] + delta_output_usd,
6,
)
_add_cache(
entry, delta_cache_read_tokens, delta_cache_savings_usd, delta_cache_read_cost_usd
)
# Attribute this checkpoint's delta to the provider that produced
# it. Each checkpoint comes from a single request, so its delta is
# wholly owned by one provider. Skip no-op checkpoints so providers
# only appear in a bucket where they actually moved a counter.
if delta_tokens or delta_usd or delta_input_tokens or delta_input_cost_usd:
if (
delta_tokens
or delta_usd
or delta_input_tokens
or delta_input_cost_usd
or delta_cache_read_tokens
):
provider = _normalize_provider(point.get("provider"))
prov = entry["by_provider"].setdefault(
provider,
@@ -2002,8 +2073,15 @@ class SavingsTracker:
"compression_savings_usd_delta": 0.0,
"total_input_tokens_delta": 0,
"total_input_cost_usd_delta": 0.0,
**_empty_cache_delta(),
},
)
_add_cache(
prov,
delta_cache_read_tokens,
delta_cache_savings_usd,
delta_cache_read_cost_usd,
)
prov["tokens_saved"] += delta_tokens
prov["compression_savings_usd_delta"] = round(
prov["compression_savings_usd_delta"] + delta_usd,
@@ -2015,7 +2093,6 @@ class SavingsTracker:
6,
)
model = _normalize_model(point.get("model"))
mod = entry["by_model"].setdefault(
model,
{
@@ -2023,8 +2100,12 @@ class SavingsTracker:
"compression_savings_usd_delta": 0.0,
"total_input_tokens_delta": 0,
"total_input_cost_usd_delta": 0.0,
**_empty_cache_delta(),
},
)
_add_cache(
mod, delta_cache_read_tokens, delta_cache_savings_usd, delta_cache_read_cost_usd
)
mod["tokens_saved"] += delta_tokens
mod["compression_savings_usd_delta"] = round(
mod["compression_savings_usd_delta"] + delta_usd,
+100
View File
@@ -917,6 +917,106 @@ def test_savings_tracker_rollup_attributes_savings_per_provider(tmp_path, monkey
assert third["by_provider"]["unknown"]["tokens_saved"] == 15
def test_savings_tracker_rollup_carries_exact_cache_read_cost_per_provider(tmp_path, monkeypatch):
# Reads on "steep-model" bill at 0.025x list (Fable-5.1-shaped pricing),
# on "flat-model" at the usual 0.1x. A consumer that backs the read cost
# out of the discount as "discount / 9" gets flat-model right and
# steep-model 4.3x too high, so the rollup has to carry the real figure.
fake_litellm = SimpleNamespace(
cost_per_token=lambda **_: (_ for _ in ()).throw(RuntimeError("unused")),
model_cost={
"steep-model": {
"input_cost_per_token": 1e-5,
"cache_read_input_token_cost": 2.5e-7,
},
"flat-model": {
"input_cost_per_token": 2e-6,
"cache_read_input_token_cost": 2e-7,
},
},
)
monkeypatch.setattr(savings_tracker_module, "LITELLM_AVAILABLE", True)
monkeypatch.setattr(savings_tracker_module, "litellm", fake_litellm)
tracker = SavingsTracker(
path=str(tmp_path / "proxy_savings.json"),
max_history_points=100,
max_history_age_days=30,
)
tracker.record_request(
model="steep-model",
provider="anthropic",
input_tokens=1_010_000,
tokens_saved=5_000,
cache_read_tokens=1_000_000,
uncached_input_tokens=10_000,
timestamp="2026-03-27T09:10:00Z",
)
tracker.record_request(
model="flat-model",
provider="openai",
input_tokens=150_000,
tokens_saved=2_000,
cache_read_tokens=100_000,
uncached_input_tokens=50_000,
timestamp="2026-03-27T09:20:00Z",
)
bucket = tracker.history_response()["series"]["hourly"][0]
anthropic = bucket["by_provider"]["anthropic"]
openai = bucket["by_provider"]["openai"]
assert anthropic["cache_read_tokens_delta"] == 1_000_000
assert anthropic["cache_read_cost_usd_delta"] == pytest.approx(0.25)
assert anthropic["cache_savings_usd_delta"] == pytest.approx(9.75)
# The read cost is exactly the read component of the recorded input cost,
# so input cost minus read cost is the uncached spend.
assert anthropic["total_input_cost_usd_delta"] - anthropic[
"cache_read_cost_usd_delta"
] == pytest.approx(10_000 * 1e-5)
assert openai["cache_read_tokens_delta"] == 100_000
assert openai["cache_read_cost_usd_delta"] == pytest.approx(0.02)
assert openai["cache_savings_usd_delta"] == pytest.approx(0.18)
# Bucket totals are the sum of the providers.
assert bucket["cache_read_tokens_delta"] == 1_100_000
assert bucket["cache_read_cost_usd_delta"] == pytest.approx(0.27)
assert bucket["cache_savings_usd_delta"] == pytest.approx(9.93)
assert bucket["by_model"]["steep-model"]["cache_read_cost_usd_delta"] == pytest.approx(0.25)
def test_savings_tracker_rollup_read_cost_unknown_for_model_less_checkpoints(tmp_path):
tracker = SavingsTracker(
path=str(tmp_path / "proxy_savings.json"),
max_history_points=100,
max_history_age_days=30,
)
history = [
{
"timestamp": "2026-03-27T09:00:00Z",
"provider": "anthropic",
"total_tokens_saved": 0,
"cache_read_tokens": 0,
},
# Written before per-model attribution: reads, but no model to price them.
{
"timestamp": "2026-03-27T09:30:00Z",
"provider": "anthropic",
"total_tokens_saved": 10,
"cache_read_tokens": 500,
"cache_savings_usd": 0.9,
},
]
bucket = tracker._build_rollup(history, "hour")[0]
assert bucket["cache_read_tokens_delta"] == 500
assert bucket["cache_savings_usd_delta"] == pytest.approx(0.9)
assert bucket["cache_read_cost_usd_delta"] is None
assert bucket["by_provider"]["anthropic"]["cache_read_cost_usd_delta"] is None
def test_savings_tracker_rollup_attributes_savings_per_model(tmp_path, monkeypatch):
path = tmp_path / "proxy_savings.json"
tracker = SavingsTracker(