feat(savings): carry cumulative actual output_tokens in history checkpoints (#2921)

## Description

Closes #3994

History checkpoints carry cumulative `cache_read_tokens` (and
`cache_savings_usd`), which lets a consumer diff consecutive checkpoints
into per-day / per-hour cache series - we use exactly that in the
Headroom desktop app to show window-scoped cache-hit rates. The
output-shaping dimension cannot do the same: checkpoints carry
cumulative `output_tokens_saved` but not cumulative ACTUAL output
tokens, so there is no denominator and no per-bucket output-reduction
rate (`saved / (saved + actual)`) can be derived from the payload.
Dashboards are left showing an all-time estimator figure under per-day
charts.

This PR adds one cumulative `output_tokens` counter, symmetric with the
existing cache fields:

- `record_request` already receives the request's actual output tokens
(main added the parameter and the `prometheus_metrics.py` caller in
#3520 to price `total_output_cost_usd`); this PR accumulates them as a
token count.
- The lifetime aggregate accumulates them as a lazy key, matching
`output_tokens_saved`'s legacy-tolerant shape (older state files keep
parsing; salvage behavior is identical to the existing output fields).
- Every history checkpoint emits the cumulative, and
`_normalize_history_entry` coerces it through reload so persisted
history retains the field (entries without the key normalize to 0, same
as the cache fields did when they were introduced).

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/savings_tracker.py`: lazy lifetime accumulation of
`output_tokens`; checkpoint emission (a request with actual output now
also writes a checkpoint, so its output lands in the window it happened
in); `_normalize_history_entry` coercion; lifetime restored from the
newest checkpoint on reload
- `headroom/proxy/prometheus_metrics.py`: no longer changed; #3520
already passes `output_tokens` to `record_request` on main
- `tests/test_proxy_savings_history.py`: new test covering accumulation
across requests, checkpoint values, reload survival, and legacy entries
without the key; two existing exact-shape fixtures extended with the new
field

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest tests/test_proxy_savings_history.py tests/test_output_shaping_rollup.py \
    tests/test_persistent_metrics_integration.py -q
======================== 45 passed, 1 warning in 8.46s =========================
$ ruff check (changed files)
All checks passed!
$ mypy headroom/proxy/savings_tracker.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS (Darwin 24.6.0), Python 3.10 dev venv of this
branch
- Exact command / steps: unit-level - the new test drives two
`record_request` calls with `output_tokens=200/300`, snapshots the
tracker, reloads it from disk, and normalizes a legacy entry
- Observed result: lifetime cumulative reaches 500; the two checkpoints
carry 200 and 500; a tracker reloaded from the persisted file still
reports 500 on the newest checkpoint; a legacy entry without the key
normalizes to 0 rather than erroring
- Repro (2026-10-06, at `acbf49c0` merged with upstream/main
`855390d6`): the script in #3994 records four requests through
`SavingsTracker.record_request`, reloads from disk and reads
`history_response()`. On main no checkpoint carries actual output and a
per-day output-reduction rate cannot be computed (the request with
output but no saving writes no checkpoint at all); on this branch the
checkpoints carry 1000/1600/2000/4000 and the per-day rates come out at
the expected 13.5% and 27.3%. 146 tests across 15 savings/history files
pass, ruff, ruff format and mypy clean.
- Not tested: a live proxy deployment of this exact commit. The consumer
that motivated it (Headroom desktop) currently approximates the same
series by polling `/stats` cumulatives and will switch to checkpoint
diffs once a wheel ships this field.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes - history checkpoints carry a new
cumulative `output_tokens` field and output-only requests now emit a
checkpoint; savings values are unchanged.
- Kill switch / disable path: Not applicable - the field is additive and
`_normalize_history_entry` coerces missing values to 0.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert; readers ignore the extra key and legacy entries
without it normalize to 0.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A - one new numeric field on an internal persistence schema.

## Additional Notes

- Documentation checklist item unchecked: the checkpoint schema is not
documented outside the code today; happy to add a schema note if
maintainers want one.
- Related: #2919 (cache-breakpoint diagnostics) consumes the cache half
of this checkpoint pattern; this PR completes the output half.
Independent - neither depends on the other.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2921"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->


### Maintainer current-main requalification

Exact head `d7bd7e066d89b02301ffcfb55be580a645e0ddc6`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.

Clean current-main integration, two intended tracker/test files; no
contributor counter/emission/legacy semantics rewritten.
Fresh45tests:43pass/2fail/1warning. Two failures match AST-identical
main tests (Windows directory fsync and positive price for offline
claude-sonnet-5); reused recorded exact0ddmain control from
pr2245-0dd-main-failure-control.log, not claimed rerun. They are
inherited, not PR blockers. Actual temp-file record/save/reopen
independently reproduces #3994: raw output checkpoint deltas
yield13.5%/27.3%, cumulative actual output survives reload and
reaches1065 after output-only25token request, which emits its own
checkpoint. Exact main tracker control lacks output field/rates and no
output-only checkpoint. Fresh capacity-two/cache-read retention probe
produces same earliest-surviving bucket carrying100prior saved tokens on
BOTH main and branch; outstanding retention issue inherited, not
approval hold for this feature. Legacy missing actual-output fields
normalize0; unknown older windows cannot be reconstructed. Raw
checkpoints are explicit approved spec scope; no CSV/rollup shape
expansion. No live proxy/provider/LLM/billing or production timing
claim. ChangedPython Rufflint/format,offline lock/whitespace pass. Full
native inputset/hashes/artifactchecksum verified before3982DLL reuse, no
fresh build. Current discussion/six threads adjudicated with author
replies and #3994 spec; no duplicate feedback. Exact published-head
hosted checks pending before formal approval; maintainers handle CI. No
merge.

Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
gglucass
2026-11-07 20:43:46 -06:00
committed by GitHub
co-authored by Claude Fable 5 JD Davis
parent 9b2dec56f9
commit 8388ff5efd
2 changed files with 115 additions and 1 deletions
+26 -1
View File
@@ -740,6 +740,7 @@ def _normalize_history_entry(entry: Any) -> dict[str, Any] | None:
total_input_tokens = 0
total_input_cost_usd = 0.0
output_tokens_saved = 0
output_tokens = 0
output_savings_usd = 0.0
tool_tokens_saved = 0
tool_schema_savings_usd = 0.0
@@ -760,6 +761,7 @@ def _normalize_history_entry(entry: Any) -> dict[str, Any] | None:
total_input_tokens = _coerce_int(entry.get("total_input_tokens"))
total_input_cost_usd = _coerce_float(entry.get("total_input_cost_usd"))
output_tokens_saved = _coerce_int(entry.get("output_tokens_saved"))
output_tokens = _coerce_int(entry.get("output_tokens"))
output_savings_usd = _coerce_float(entry.get("output_savings_usd"))
tool_tokens_saved = _coerce_int(entry.get("tool_tokens_saved"))
tool_schema_savings_usd = _coerce_float(entry.get("tool_schema_savings_usd"))
@@ -794,6 +796,7 @@ def _normalize_history_entry(entry: Any) -> dict[str, Any] | None:
"total_input_tokens": total_input_tokens,
"total_input_cost_usd": round(total_input_cost_usd, 6),
"output_tokens_saved": output_tokens_saved,
"output_tokens": output_tokens,
"output_savings_usd": round(output_savings_usd, 6),
"tool_tokens_saved": tool_tokens_saved,
"tool_schema_savings_usd": round(tool_schema_savings_usd, 6),
@@ -1185,6 +1188,7 @@ class SavingsTracker:
delta_tool_tokens_saved = max(_coerce_int(tool_search_saved), 0)
delta_input_tokens = _coerce_int(input_tokens)
delta_output_tokens_saved = max(_coerce_int(output_tokens_saved), 0)
delta_output_tokens = max(_coerce_int(output_tokens), 0)
delta_cache_read_tokens = _coerce_int(cache_read_tokens)
priced = estimated_savings_usd
# ``estimate_request_savings_usd`` prices FOUR buckets, but this method
@@ -1344,6 +1348,12 @@ class SavingsTracker:
lifetime["tool_schema_savings_usd"] + delta_tool_savings_usd,
6,
)
# Actual output tokens emitted (not saved): the cumulative the
# history checkpoints need so a per-bucket output-reduction rate
# (saved / (saved + actual)) can be diffed out of consecutive
# checkpoints, exactly like cache_read_tokens. Lazy key, same
# legacy-tolerant shape as output_tokens_saved above.
lifetime["output_tokens"] = lifetime.get("output_tokens", 0) + delta_output_tokens
lifetime["total_output_cost_usd"] = round(
lifetime.get("total_output_cost_usd", 0.0) + delta_output_cost_usd,
6,
@@ -1420,12 +1430,18 @@ class SavingsTracker:
# not lossy-compressed, to keep Bedrock's prompt cache warm. Gating
# on tokens_saved alone silently dropped every history point on
# those requests even though real cache-read savings occurred.
# Append whenever any savings mechanism produced a saving.
# Append whenever any savings mechanism produced a saving. Actual
# output activity must also emit: consumers derive per-bucket
# output-reduction rates by diffing consecutive checkpoints, so an
# output-only request that only bumped lifetime["output_tokens"]
# would otherwise be invisible to the window it lands in and
# overstate the reduction rate.
if (
delta_tokens_saved > 0
or delta_cache_read_tokens > 0
or delta_output_tokens_saved > 0
or delta_tool_tokens_saved > 0
or delta_output_tokens > 0
):
self._state["history"].append(
{
@@ -1440,6 +1456,7 @@ class SavingsTracker:
"total_input_tokens": lifetime["total_input_tokens"],
"total_input_cost_usd": lifetime["total_input_cost_usd"],
"output_tokens_saved": lifetime.get("output_tokens_saved", 0),
"output_tokens": lifetime.get("output_tokens", 0),
"output_savings_usd": lifetime.get("output_savings_usd", 0.0),
"tool_tokens_saved": lifetime["tool_tokens_saved"],
"tool_schema_savings_usd": lifetime["tool_schema_savings_usd"],
@@ -1834,6 +1851,7 @@ class SavingsTracker:
"total_input_cost_usd": 0.0,
"output_tokens_saved": 0,
"output_savings_usd": 0.0,
"output_tokens": 0,
"total_output_cost_usd": 0.0,
},
"display_session": _empty_display_session(),
@@ -1905,6 +1923,7 @@ class SavingsTracker:
is_v6_lifetime = False
lifetime_output_tokens_saved = 0
lifetime_output_savings_usd = 0.0
lifetime_output_tokens = 0
lifetime_output_cost_usd = 0.0
if isinstance(lifetime_raw, dict):
lifetime_requests = _coerce_int(lifetime_raw.get("requests"))
@@ -1925,6 +1944,7 @@ class SavingsTracker:
lifetime_basis = str(lifetime_raw.get("savings_basis") or BASIS_UNKNOWN)
lifetime_output_tokens_saved = _coerce_int(lifetime_raw.get("output_tokens_saved"))
lifetime_output_savings_usd = _coerce_float(lifetime_raw.get("output_savings_usd"))
lifetime_output_tokens = _coerce_int(lifetime_raw.get("output_tokens"))
lifetime_output_cost_usd = _coerce_float(lifetime_raw.get("total_output_cost_usd"))
if normalized_history:
@@ -1978,6 +1998,10 @@ class SavingsTracker:
lifetime_output_savings_usd,
_coerce_float(last.get("output_savings_usd")),
)
lifetime_output_tokens = max(
lifetime_output_tokens,
_coerce_int(last.get("output_tokens")),
)
lifetime_output_cost_usd = max(
lifetime_output_cost_usd,
_coerce_float(last.get("total_output_cost_usd")),
@@ -2020,6 +2044,7 @@ class SavingsTracker:
"total_input_cost_usd": round(lifetime_input_cost_usd, 6),
"output_tokens_saved": lifetime_output_tokens_saved,
"output_savings_usd": round(lifetime_output_savings_usd, 6),
"output_tokens": lifetime_output_tokens,
"total_output_cost_usd": round(lifetime_output_cost_usd, 6),
},
"display_session": _normalize_display_session(raw.get("display_session")),
+89
View File
@@ -84,6 +84,7 @@ def test_savings_tracker_helpers_normalize_inputs_and_paths(tmp_path, monkeypatc
"total_input_tokens": 0,
"total_input_cost_usd": 0.0,
"output_tokens_saved": 0,
"output_tokens": 0,
"output_savings_usd": 0.0,
"tool_tokens_saved": 0,
"tool_schema_savings_usd": 0.0,
@@ -150,6 +151,7 @@ def test_savings_tracker_sanitizes_legacy_state_and_applies_retention(tmp_path):
"total_input_cost_usd": 0.0,
"output_tokens_saved": 0,
"output_savings_usd": 0.0,
"output_tokens": 0,
"total_output_cost_usd": 0.0,
}
assert snapshot["display_session"] == savings_tracker_module._empty_display_session()
@@ -166,6 +168,7 @@ def test_savings_tracker_sanitizes_legacy_state_and_applies_retention(tmp_path):
"total_input_tokens": 0,
"total_input_cost_usd": 0.0,
"output_tokens_saved": 0,
"output_tokens": 0,
"output_savings_usd": 0.0,
"tool_tokens_saved": 0,
"tool_schema_savings_usd": 0.0,
@@ -202,6 +205,7 @@ def test_non_dict_savings_state_resets_to_default(tmp_path):
"total_input_cost_usd": 0.0,
"output_tokens_saved": 0,
"output_savings_usd": 0.0,
"output_tokens": 0,
"total_output_cost_usd": 0.0,
}
assert snapshot["display_session"] == savings_tracker_module._empty_display_session()
@@ -2133,3 +2137,88 @@ def test_normalize_history_entry_defaults_missing_cache_fields(tmp_path):
assert history[0]["cache_read_tokens"] == 0
assert history[0]["cache_savings_usd"] == 0.0
assert history[0]["total_tokens_saved"] == 40
def test_record_request_accumulates_output_tokens_into_checkpoints(tmp_path):
"""History checkpoints carry cumulative ACTUAL output tokens so a
per-bucket output-reduction rate can be diffed out of consecutive
checkpoints, exactly like cache_read_tokens. The cumulative must also
survive a reload through _normalize_history_entry."""
path = tmp_path / "savings.json"
tracker = SavingsTracker(path=path)
tracker.record_request(
model="claude-opus-5",
input_tokens=100,
tokens_saved=10,
output_tokens_saved=5,
output_tokens=200,
)
tracker.record_request(
model="claude-opus-5",
input_tokens=100,
tokens_saved=10,
output_tokens_saved=5,
output_tokens=300,
)
snapshot = tracker.snapshot()
assert snapshot["lifetime"]["output_tokens"] == 500
history = snapshot["history"]
assert [point["output_tokens"] for point in history[-2:]] == [200, 500]
# Reload: the normalizer must keep the field rather than strip it.
reloaded = SavingsTracker(path=path)
reloaded_history = reloaded.snapshot()["history"]
assert reloaded_history[-1]["output_tokens"] == 500
# ...and the lifetime cumulative must resume from it, not restart at 0, or
# the next checkpoint (50) reads as a negative delta to every consumer.
assert reloaded.snapshot()["lifetime"]["output_tokens"] == 500
reloaded.record_request(
model="claude-opus-5",
input_tokens=100,
tokens_saved=10,
output_tokens=50,
)
assert reloaded.snapshot()["history"][-1]["output_tokens"] == 550
# Legacy entries without the key normalize to zero, not an error.
legacy = savings_tracker_module._normalize_history_entry(
{"timestamp": "2026-08-11T10:00:00Z", "total_tokens_saved": 1}
)
assert legacy is not None
assert legacy["output_tokens"] == 0
def test_output_only_request_emits_checkpoint(tmp_path):
"""A request with actual output tokens but zero savings must still emit a
history checkpoint: it moves lifetime["output_tokens"], and if it were the
final request in a window, consumers diffing checkpoints would omit those
actual tokens and overstate the output-reduction rate."""
path = tmp_path / "savings.json"
tracker = SavingsTracker(path=path)
tracker.record_request(
model="claude-opus-5",
input_tokens=100,
tokens_saved=10,
output_tokens_saved=5,
output_tokens=200,
)
tracker.record_request(
model="claude-opus-5",
input_tokens=100,
tokens_saved=0,
output_tokens_saved=0,
output_tokens=300,
)
history = tracker.snapshot()["history"]
assert len(history) == 2
assert history[-1]["output_tokens"] == 500
assert history[-1]["output_tokens_saved"] == 5
assert history[-1]["total_tokens_saved"] == 10
# The output-only checkpoint must survive persistence and reload.
reloaded_history = SavingsTracker(path=path).snapshot()["history"]
assert len(reloaded_history) == 2
assert reloaded_history[-1]["output_tokens"] == 500