mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
feat(savings): carry cumulative actual output_tokens in history checkpoints (#2921)
## Description Closes #3994 History checkpoints carry cumulative `cache_read_tokens` (and `cache_savings_usd`), which lets a consumer diff consecutive checkpoints into per-day / per-hour cache series - we use exactly that in the Headroom desktop app to show window-scoped cache-hit rates. The output-shaping dimension cannot do the same: checkpoints carry cumulative `output_tokens_saved` but not cumulative ACTUAL output tokens, so there is no denominator and no per-bucket output-reduction rate (`saved / (saved + actual)`) can be derived from the payload. Dashboards are left showing an all-time estimator figure under per-day charts. This PR adds one cumulative `output_tokens` counter, symmetric with the existing cache fields: - `record_request` already receives the request's actual output tokens (main added the parameter and the `prometheus_metrics.py` caller in #3520 to price `total_output_cost_usd`); this PR accumulates them as a token count. - The lifetime aggregate accumulates them as a lazy key, matching `output_tokens_saved`'s legacy-tolerant shape (older state files keep parsing; salvage behavior is identical to the existing output fields). - Every history checkpoint emits the cumulative, and `_normalize_history_entry` coerces it through reload so persisted history retains the field (entries without the key normalize to 0, same as the cache fields did when they were introduced). ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/savings_tracker.py`: lazy lifetime accumulation of `output_tokens`; checkpoint emission (a request with actual output now also writes a checkpoint, so its output lands in the window it happened in); `_normalize_history_entry` coercion; lifetime restored from the newest checkpoint on reload - `headroom/proxy/prometheus_metrics.py`: no longer changed; #3520 already passes `output_tokens` to `record_request` on main - `tests/test_proxy_savings_history.py`: new test covering accumulation across requests, checkpoint values, reload survival, and legacy entries without the key; two existing exact-shape fixtures extended with the new field ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ pytest tests/test_proxy_savings_history.py tests/test_output_shaping_rollup.py \ tests/test_persistent_metrics_integration.py -q ======================== 45 passed, 1 warning in 8.46s ========================= $ ruff check (changed files) All checks passed! $ mypy headroom/proxy/savings_tracker.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS (Darwin 24.6.0), Python 3.10 dev venv of this branch - Exact command / steps: unit-level - the new test drives two `record_request` calls with `output_tokens=200/300`, snapshots the tracker, reloads it from disk, and normalizes a legacy entry - Observed result: lifetime cumulative reaches 500; the two checkpoints carry 200 and 500; a tracker reloaded from the persisted file still reports 500 on the newest checkpoint; a legacy entry without the key normalizes to 0 rather than erroring - Repro (2026-10-06, at `acbf49c0` merged with upstream/main `855390d6`): the script in #3994 records four requests through `SavingsTracker.record_request`, reloads from disk and reads `history_response()`. On main no checkpoint carries actual output and a per-day output-reduction rate cannot be computed (the request with output but no saving writes no checkpoint at all); on this branch the checkpoints carry 1000/1600/2000/4000 and the per-day rates come out at the expected 13.5% and 27.3%. 146 tests across 15 savings/history files pass, ruff, ruff format and mypy clean. - Not tested: a live proxy deployment of this exact commit. The consumer that motivated it (Headroom desktop) currently approximates the same series by polling `/stats` cumulatives and will switch to checkpoint diffs once a wheel ships this field. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: Stable. - Stable/default behavior changed: Yes - history checkpoints carry a new cumulative `output_tokens` field and output-only requests now emit a checkpoint; savings values are unchanged. - Kill switch / disable path: Not applicable - the field is additive and `_normalize_history_entry` coerces missing values to 0. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert; readers ignore the extra key and legacy entries without it normalize to 0. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A - one new numeric field on an internal persistence schema. ## Additional Notes - Documentation checklist item unchecked: the checkpoint schema is not documented outside the code today; happy to add a schema note if maintainers want one. - Related: #2919 (cache-breakpoint diagnostics) consumes the cache half of this checkpoint pattern; this PR completes the output half. Independent - neither depends on the other. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2921" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ### Maintainer current-main requalification Exact head `d7bd7e066d89b02301ffcfb55be580a645e0ddc6`, upstream main `0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`. Clean current-main integration, two intended tracker/test files; no contributor counter/emission/legacy semantics rewritten. Fresh45tests:43pass/2fail/1warning. Two failures match AST-identical main tests (Windows directory fsync and positive price for offline claude-sonnet-5); reused recorded exact0ddmain control from pr2245-0dd-main-failure-control.log, not claimed rerun. They are inherited, not PR blockers. Actual temp-file record/save/reopen independently reproduces #3994: raw output checkpoint deltas yield13.5%/27.3%, cumulative actual output survives reload and reaches1065 after output-only25token request, which emits its own checkpoint. Exact main tracker control lacks output field/rates and no output-only checkpoint. Fresh capacity-two/cache-read retention probe produces same earliest-surviving bucket carrying100prior saved tokens on BOTH main and branch; outstanding retention issue inherited, not approval hold for this feature. Legacy missing actual-output fields normalize0; unknown older windows cannot be reconstructed. Raw checkpoints are explicit approved spec scope; no CSV/rollup shape expansion. No live proxy/provider/LLM/billing or production timing claim. ChangedPython Rufflint/format,offline lock/whitespace pass. Full native inputset/hashes/artifactchecksum verified before3982DLL reuse, no fresh build. Current discussion/six threads adjudicated with author replies and #3994 spec; no duplicate feedback. Exact published-head hosted checks pending before formal approval; maintainers handle CI. No merge. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Fable 5
JD Davis
parent
9b2dec56f9
commit
8388ff5efd
@@ -740,6 +740,7 @@ def _normalize_history_entry(entry: Any) -> dict[str, Any] | None:
|
||||
total_input_tokens = 0
|
||||
total_input_cost_usd = 0.0
|
||||
output_tokens_saved = 0
|
||||
output_tokens = 0
|
||||
output_savings_usd = 0.0
|
||||
tool_tokens_saved = 0
|
||||
tool_schema_savings_usd = 0.0
|
||||
@@ -760,6 +761,7 @@ def _normalize_history_entry(entry: Any) -> dict[str, Any] | None:
|
||||
total_input_tokens = _coerce_int(entry.get("total_input_tokens"))
|
||||
total_input_cost_usd = _coerce_float(entry.get("total_input_cost_usd"))
|
||||
output_tokens_saved = _coerce_int(entry.get("output_tokens_saved"))
|
||||
output_tokens = _coerce_int(entry.get("output_tokens"))
|
||||
output_savings_usd = _coerce_float(entry.get("output_savings_usd"))
|
||||
tool_tokens_saved = _coerce_int(entry.get("tool_tokens_saved"))
|
||||
tool_schema_savings_usd = _coerce_float(entry.get("tool_schema_savings_usd"))
|
||||
@@ -794,6 +796,7 @@ def _normalize_history_entry(entry: Any) -> dict[str, Any] | None:
|
||||
"total_input_tokens": total_input_tokens,
|
||||
"total_input_cost_usd": round(total_input_cost_usd, 6),
|
||||
"output_tokens_saved": output_tokens_saved,
|
||||
"output_tokens": output_tokens,
|
||||
"output_savings_usd": round(output_savings_usd, 6),
|
||||
"tool_tokens_saved": tool_tokens_saved,
|
||||
"tool_schema_savings_usd": round(tool_schema_savings_usd, 6),
|
||||
@@ -1185,6 +1188,7 @@ class SavingsTracker:
|
||||
delta_tool_tokens_saved = max(_coerce_int(tool_search_saved), 0)
|
||||
delta_input_tokens = _coerce_int(input_tokens)
|
||||
delta_output_tokens_saved = max(_coerce_int(output_tokens_saved), 0)
|
||||
delta_output_tokens = max(_coerce_int(output_tokens), 0)
|
||||
delta_cache_read_tokens = _coerce_int(cache_read_tokens)
|
||||
priced = estimated_savings_usd
|
||||
# ``estimate_request_savings_usd`` prices FOUR buckets, but this method
|
||||
@@ -1344,6 +1348,12 @@ class SavingsTracker:
|
||||
lifetime["tool_schema_savings_usd"] + delta_tool_savings_usd,
|
||||
6,
|
||||
)
|
||||
# Actual output tokens emitted (not saved): the cumulative the
|
||||
# history checkpoints need so a per-bucket output-reduction rate
|
||||
# (saved / (saved + actual)) can be diffed out of consecutive
|
||||
# checkpoints, exactly like cache_read_tokens. Lazy key, same
|
||||
# legacy-tolerant shape as output_tokens_saved above.
|
||||
lifetime["output_tokens"] = lifetime.get("output_tokens", 0) + delta_output_tokens
|
||||
lifetime["total_output_cost_usd"] = round(
|
||||
lifetime.get("total_output_cost_usd", 0.0) + delta_output_cost_usd,
|
||||
6,
|
||||
@@ -1420,12 +1430,18 @@ class SavingsTracker:
|
||||
# not lossy-compressed, to keep Bedrock's prompt cache warm. Gating
|
||||
# on tokens_saved alone silently dropped every history point on
|
||||
# those requests even though real cache-read savings occurred.
|
||||
# Append whenever any savings mechanism produced a saving.
|
||||
# Append whenever any savings mechanism produced a saving. Actual
|
||||
# output activity must also emit: consumers derive per-bucket
|
||||
# output-reduction rates by diffing consecutive checkpoints, so an
|
||||
# output-only request that only bumped lifetime["output_tokens"]
|
||||
# would otherwise be invisible to the window it lands in and
|
||||
# overstate the reduction rate.
|
||||
if (
|
||||
delta_tokens_saved > 0
|
||||
or delta_cache_read_tokens > 0
|
||||
or delta_output_tokens_saved > 0
|
||||
or delta_tool_tokens_saved > 0
|
||||
or delta_output_tokens > 0
|
||||
):
|
||||
self._state["history"].append(
|
||||
{
|
||||
@@ -1440,6 +1456,7 @@ class SavingsTracker:
|
||||
"total_input_tokens": lifetime["total_input_tokens"],
|
||||
"total_input_cost_usd": lifetime["total_input_cost_usd"],
|
||||
"output_tokens_saved": lifetime.get("output_tokens_saved", 0),
|
||||
"output_tokens": lifetime.get("output_tokens", 0),
|
||||
"output_savings_usd": lifetime.get("output_savings_usd", 0.0),
|
||||
"tool_tokens_saved": lifetime["tool_tokens_saved"],
|
||||
"tool_schema_savings_usd": lifetime["tool_schema_savings_usd"],
|
||||
@@ -1834,6 +1851,7 @@ class SavingsTracker:
|
||||
"total_input_cost_usd": 0.0,
|
||||
"output_tokens_saved": 0,
|
||||
"output_savings_usd": 0.0,
|
||||
"output_tokens": 0,
|
||||
"total_output_cost_usd": 0.0,
|
||||
},
|
||||
"display_session": _empty_display_session(),
|
||||
@@ -1905,6 +1923,7 @@ class SavingsTracker:
|
||||
is_v6_lifetime = False
|
||||
lifetime_output_tokens_saved = 0
|
||||
lifetime_output_savings_usd = 0.0
|
||||
lifetime_output_tokens = 0
|
||||
lifetime_output_cost_usd = 0.0
|
||||
if isinstance(lifetime_raw, dict):
|
||||
lifetime_requests = _coerce_int(lifetime_raw.get("requests"))
|
||||
@@ -1925,6 +1944,7 @@ class SavingsTracker:
|
||||
lifetime_basis = str(lifetime_raw.get("savings_basis") or BASIS_UNKNOWN)
|
||||
lifetime_output_tokens_saved = _coerce_int(lifetime_raw.get("output_tokens_saved"))
|
||||
lifetime_output_savings_usd = _coerce_float(lifetime_raw.get("output_savings_usd"))
|
||||
lifetime_output_tokens = _coerce_int(lifetime_raw.get("output_tokens"))
|
||||
lifetime_output_cost_usd = _coerce_float(lifetime_raw.get("total_output_cost_usd"))
|
||||
|
||||
if normalized_history:
|
||||
@@ -1978,6 +1998,10 @@ class SavingsTracker:
|
||||
lifetime_output_savings_usd,
|
||||
_coerce_float(last.get("output_savings_usd")),
|
||||
)
|
||||
lifetime_output_tokens = max(
|
||||
lifetime_output_tokens,
|
||||
_coerce_int(last.get("output_tokens")),
|
||||
)
|
||||
lifetime_output_cost_usd = max(
|
||||
lifetime_output_cost_usd,
|
||||
_coerce_float(last.get("total_output_cost_usd")),
|
||||
@@ -2020,6 +2044,7 @@ class SavingsTracker:
|
||||
"total_input_cost_usd": round(lifetime_input_cost_usd, 6),
|
||||
"output_tokens_saved": lifetime_output_tokens_saved,
|
||||
"output_savings_usd": round(lifetime_output_savings_usd, 6),
|
||||
"output_tokens": lifetime_output_tokens,
|
||||
"total_output_cost_usd": round(lifetime_output_cost_usd, 6),
|
||||
},
|
||||
"display_session": _normalize_display_session(raw.get("display_session")),
|
||||
|
||||
@@ -84,6 +84,7 @@ def test_savings_tracker_helpers_normalize_inputs_and_paths(tmp_path, monkeypatc
|
||||
"total_input_tokens": 0,
|
||||
"total_input_cost_usd": 0.0,
|
||||
"output_tokens_saved": 0,
|
||||
"output_tokens": 0,
|
||||
"output_savings_usd": 0.0,
|
||||
"tool_tokens_saved": 0,
|
||||
"tool_schema_savings_usd": 0.0,
|
||||
@@ -150,6 +151,7 @@ def test_savings_tracker_sanitizes_legacy_state_and_applies_retention(tmp_path):
|
||||
"total_input_cost_usd": 0.0,
|
||||
"output_tokens_saved": 0,
|
||||
"output_savings_usd": 0.0,
|
||||
"output_tokens": 0,
|
||||
"total_output_cost_usd": 0.0,
|
||||
}
|
||||
assert snapshot["display_session"] == savings_tracker_module._empty_display_session()
|
||||
@@ -166,6 +168,7 @@ def test_savings_tracker_sanitizes_legacy_state_and_applies_retention(tmp_path):
|
||||
"total_input_tokens": 0,
|
||||
"total_input_cost_usd": 0.0,
|
||||
"output_tokens_saved": 0,
|
||||
"output_tokens": 0,
|
||||
"output_savings_usd": 0.0,
|
||||
"tool_tokens_saved": 0,
|
||||
"tool_schema_savings_usd": 0.0,
|
||||
@@ -202,6 +205,7 @@ def test_non_dict_savings_state_resets_to_default(tmp_path):
|
||||
"total_input_cost_usd": 0.0,
|
||||
"output_tokens_saved": 0,
|
||||
"output_savings_usd": 0.0,
|
||||
"output_tokens": 0,
|
||||
"total_output_cost_usd": 0.0,
|
||||
}
|
||||
assert snapshot["display_session"] == savings_tracker_module._empty_display_session()
|
||||
@@ -2133,3 +2137,88 @@ def test_normalize_history_entry_defaults_missing_cache_fields(tmp_path):
|
||||
assert history[0]["cache_read_tokens"] == 0
|
||||
assert history[0]["cache_savings_usd"] == 0.0
|
||||
assert history[0]["total_tokens_saved"] == 40
|
||||
|
||||
|
||||
def test_record_request_accumulates_output_tokens_into_checkpoints(tmp_path):
|
||||
"""History checkpoints carry cumulative ACTUAL output tokens so a
|
||||
per-bucket output-reduction rate can be diffed out of consecutive
|
||||
checkpoints, exactly like cache_read_tokens. The cumulative must also
|
||||
survive a reload through _normalize_history_entry."""
|
||||
path = tmp_path / "savings.json"
|
||||
tracker = SavingsTracker(path=path)
|
||||
tracker.record_request(
|
||||
model="claude-opus-5",
|
||||
input_tokens=100,
|
||||
tokens_saved=10,
|
||||
output_tokens_saved=5,
|
||||
output_tokens=200,
|
||||
)
|
||||
tracker.record_request(
|
||||
model="claude-opus-5",
|
||||
input_tokens=100,
|
||||
tokens_saved=10,
|
||||
output_tokens_saved=5,
|
||||
output_tokens=300,
|
||||
)
|
||||
|
||||
snapshot = tracker.snapshot()
|
||||
assert snapshot["lifetime"]["output_tokens"] == 500
|
||||
history = snapshot["history"]
|
||||
assert [point["output_tokens"] for point in history[-2:]] == [200, 500]
|
||||
|
||||
# Reload: the normalizer must keep the field rather than strip it.
|
||||
reloaded = SavingsTracker(path=path)
|
||||
reloaded_history = reloaded.snapshot()["history"]
|
||||
assert reloaded_history[-1]["output_tokens"] == 500
|
||||
|
||||
# ...and the lifetime cumulative must resume from it, not restart at 0, or
|
||||
# the next checkpoint (50) reads as a negative delta to every consumer.
|
||||
assert reloaded.snapshot()["lifetime"]["output_tokens"] == 500
|
||||
reloaded.record_request(
|
||||
model="claude-opus-5",
|
||||
input_tokens=100,
|
||||
tokens_saved=10,
|
||||
output_tokens=50,
|
||||
)
|
||||
assert reloaded.snapshot()["history"][-1]["output_tokens"] == 550
|
||||
|
||||
# Legacy entries without the key normalize to zero, not an error.
|
||||
legacy = savings_tracker_module._normalize_history_entry(
|
||||
{"timestamp": "2026-08-11T10:00:00Z", "total_tokens_saved": 1}
|
||||
)
|
||||
assert legacy is not None
|
||||
assert legacy["output_tokens"] == 0
|
||||
|
||||
|
||||
def test_output_only_request_emits_checkpoint(tmp_path):
|
||||
"""A request with actual output tokens but zero savings must still emit a
|
||||
history checkpoint: it moves lifetime["output_tokens"], and if it were the
|
||||
final request in a window, consumers diffing checkpoints would omit those
|
||||
actual tokens and overstate the output-reduction rate."""
|
||||
path = tmp_path / "savings.json"
|
||||
tracker = SavingsTracker(path=path)
|
||||
tracker.record_request(
|
||||
model="claude-opus-5",
|
||||
input_tokens=100,
|
||||
tokens_saved=10,
|
||||
output_tokens_saved=5,
|
||||
output_tokens=200,
|
||||
)
|
||||
tracker.record_request(
|
||||
model="claude-opus-5",
|
||||
input_tokens=100,
|
||||
tokens_saved=0,
|
||||
output_tokens_saved=0,
|
||||
output_tokens=300,
|
||||
)
|
||||
|
||||
history = tracker.snapshot()["history"]
|
||||
assert len(history) == 2
|
||||
assert history[-1]["output_tokens"] == 500
|
||||
assert history[-1]["output_tokens_saved"] == 5
|
||||
assert history[-1]["total_tokens_saved"] == 10
|
||||
|
||||
# The output-only checkpoint must survive persistence and reload.
|
||||
reloaded_history = SavingsTracker(path=path).snapshot()["history"]
|
||||
assert len(reloaded_history) == 2
|
||||
assert reloaded_history[-1]["output_tokens"] == 500
|
||||
|
||||
Reference in New Issue
Block a user