mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(savings): record tool-schema dollars disjointly beside the folded headline (#3170)
## Description Since the attribution unification (#2976, refined by #3123), `record_request` folds the tool-schema layer's dollars into `compression_savings_usd` — on the lifetime block, the display session, and every history checkpoint — while `total_tokens_saved` on those same checkpoints stays message-only. The two halves of one checkpoint now disagree about what "saved" means, and any consumer that derives a $/token rate from a checkpoint reads a figure inflated by `1 + tool/message`. That inflation is not hypothetical. On one real install, message compression was 3,299,618 tokens next to 18,435,490 tokens of tool-schema deferral (a 5.59x ratio — far from the 0.24x in the sample #2976 cites, this ratio varies wildly by workload). The blended savings rate a downstream consumer derived went from $4.99/M to $32.88/M — above the input list price of every model in the mix, a rate no genuine input-token saving can reach. There is also a smaller correctness gap: a request whose only saving is deferral (`tokens_saved == 0`, common on tool-heavy turns) appended no checkpoint at all, so its folded dollars landed in whichever later bucket happened to checkpoint next. #3155 fixed the token half of this per model, and stated the principle this PR extends to dollars: kept as its own bucket rather than widened in place, so the persisted meaning of an existing field never changes under a reader that predates it. This PR is deliberately **additive** — `compression_savings_usd` keeps its current folded semantics everywhere, so the dashboard, PERF headline, and every existing reader are unchanged. The layer is simply also recorded disjointly beside it, so message-only dollars are recoverable by subtraction and the checkpoint's token/dollar pair can be read consistently. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Record `tool_tokens_saved` / `tool_schema_savings_usd` on the lifetime block and display session, wired through `_default_state`, `_sanitize_state` (including the history-recovery max), and both normalizers — the fields survive restarts rather than resetting the way the ad-hoc output fields do. - Carry both fields on every history checkpoint, and extend the checkpoint append gate so a deferral-only request checkpoints too. - Expose `tool_tokens_saved_delta` / `tool_schema_savings_usd_delta` (plus cumulatives) on rollup buckets, following the exact delta convention of the existing series, and add the delta columns to the rollup CSV export. - State files written before these fields existed load with both at 0; legacy tuple-shaped history entries normalize the same way. - New test module `tests/test_tool_schema_savings_split.py` (6 tests) pinning the disjoint recording, reload durability, legacy-state defaults, the deferral-only checkpoint, the rollup deltas, and the subtraction identity; existing shape assertions in `tests/test_proxy_savings_history.py` updated for the two new fields. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output ```text $ uv run --frozen --extra dev pytest tests/test_proxy_savings_history.py \ tests/test_tool_schema_savings_split.py tests/test_per_model_tool_savings.py \ tests/test_savings_tool_search_aggregation.py tests/test_savings_tracker_zero_price.py \ tests/test_agent_savings.py ======================== 103 passed, 1 warning in 7.36s ======================== $ uv run --frozen --extra dev pytest tests/test_stats_new_input_savings_rate.py \ tests/test_dashboard_token_savings.py tests/test_proxy_project_savings.py \ tests/test_output_savings.py tests/test_savings_ledger.py =================== 76 passed, 2 skipped, 1 warning in 8.15s =================== $ uv run --frozen ruff check headroom/proxy/savings_tracker.py tests/test_tool_schema_savings_split.py tests/test_proxy_savings_history.py All checks passed! $ uv run --frozen ruff format --check headroom/proxy/savings_tracker.py tests/test_tool_schema_savings_split.py tests/test_proxy_savings_history.py 3 files already formatted $ uv run --frozen mypy headroom/proxy/savings_tracker.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS, Python 3.10 venv via `uv run --frozen`, this branch. - Exact command / steps: one `record_request` shaped like the reporting install's traffic (100 message tokens saved, 559 deferred schema tokens, claude-opus-5 pricing at $5/M input), then derive $/token from the appended checkpoint both ways. - Observed result: the naive checkpoint-derived rate reproduces the impossible $32.88/M seen in the field almost exactly, and subtracting the new disjoint field recovers the model's actual input rate: ```text checkpoint: total_tokens_saved=100 compression_savings_usd=0.003295 tool_tokens_saved=559 tool_schema_savings_usd=0.002795 folded-dollars / message-tokens rate: $32.95/M (input list price is $5/M) after subtracting the disjoint field: $5.00/M ``` - Not tested: a long-lived multi-process proxy soak, and the downstream consumer's subtraction path itself (that lands with the consumer release that reads the new fields). ## Runtime Rollout Safety - Rollout-managed feature(s): None -- persisted savings accounting only; no rollout-gated runtime behavior is touched. - Minimum rollout channel: N/A - Stable/default behavior changed: No. Additive fields beside existing ones; `compression_savings_usd` keeps its current folded semantics, so every existing reader sees identical values. - Kill switch / disable path: None needed -- old readers ignore the new fields, and state files written before them load with both at 0 (covered by tests). - Unsafe override required: No - Qualification impact: None beyond the unit suites run above; displayed figures are unchanged. - Rollback path: `git revert` -- the fields stop being written and remain optional on read, so state files produced while this was live stay loadable. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review Related: #3155 (token half of the same split), #3171 (companion PR on what the tool-schema bucket should be priced at — independent; either lands without the other). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
b73adaa0ce
commit
c719d4af55
@@ -612,6 +612,8 @@ def _normalize_history_entry(entry: Any) -> dict[str, Any] | None:
|
||||
total_input_cost_usd = 0.0
|
||||
output_tokens_saved = 0
|
||||
output_savings_usd = 0.0
|
||||
tool_tokens_saved = 0
|
||||
tool_schema_savings_usd = 0.0
|
||||
total_output_cost_usd = 0.0
|
||||
provider = PROVIDER_UNKNOWN
|
||||
model = MODEL_UNKNOWN
|
||||
@@ -629,6 +631,8 @@ def _normalize_history_entry(entry: Any) -> dict[str, Any] | None:
|
||||
total_input_cost_usd = _coerce_float(entry.get("total_input_cost_usd"))
|
||||
output_tokens_saved = _coerce_int(entry.get("output_tokens_saved"))
|
||||
output_savings_usd = _coerce_float(entry.get("output_savings_usd"))
|
||||
tool_tokens_saved = _coerce_int(entry.get("tool_tokens_saved"))
|
||||
tool_schema_savings_usd = _coerce_float(entry.get("tool_schema_savings_usd"))
|
||||
total_output_cost_usd = _coerce_float(entry.get("total_output_cost_usd"))
|
||||
provider = _normalize_provider(entry.get("provider"))
|
||||
model = _normalize_model(entry.get("model"))
|
||||
@@ -659,6 +663,8 @@ def _normalize_history_entry(entry: Any) -> dict[str, Any] | None:
|
||||
"total_input_cost_usd": round(total_input_cost_usd, 6),
|
||||
"output_tokens_saved": output_tokens_saved,
|
||||
"output_savings_usd": round(output_savings_usd, 6),
|
||||
"tool_tokens_saved": tool_tokens_saved,
|
||||
"tool_schema_savings_usd": round(tool_schema_savings_usd, 6),
|
||||
"total_output_cost_usd": round(total_output_cost_usd, 6),
|
||||
}
|
||||
|
||||
@@ -670,6 +676,8 @@ def _empty_display_session() -> dict[str, Any]:
|
||||
"compression_savings_usd": 0.0,
|
||||
"compression_savings_list_usd": 0.0,
|
||||
"savings_basis": BASIS_UNKNOWN,
|
||||
"tool_tokens_saved": 0,
|
||||
"tool_schema_savings_usd": 0.0,
|
||||
"cache_read_tokens": 0,
|
||||
"cache_savings_usd": 0.0,
|
||||
"total_input_tokens": 0,
|
||||
@@ -806,6 +814,11 @@ def _normalize_display_session(entry: Any) -> dict[str, Any]:
|
||||
"compression_savings_usd": round(savings_usd, 6),
|
||||
"compression_savings_list_usd": round(savings_list_usd, 6),
|
||||
"savings_basis": savings_basis,
|
||||
"tool_tokens_saved": _coerce_int(entry.get("tool_tokens_saved")),
|
||||
"tool_schema_savings_usd": round(
|
||||
_coerce_float(entry.get("tool_schema_savings_usd")),
|
||||
6,
|
||||
),
|
||||
"cache_read_tokens": _coerce_int(entry.get("cache_read_tokens")),
|
||||
"cache_savings_usd": round(
|
||||
_coerce_float(entry.get("cache_savings_usd")),
|
||||
@@ -1112,6 +1125,17 @@ class SavingsTracker:
|
||||
if priced is not None
|
||||
else _estimate_cache_savings_usd(model, delta_cache_read_tokens)
|
||||
)
|
||||
# The same figure that was just folded into ``delta_savings_usd``, kept
|
||||
# as its own delta so the layer is also recorded disjointly below. A
|
||||
# consumer can then recover message-only dollars by subtraction --
|
||||
# without this, checkpoint dollars (folded) over checkpoint tokens
|
||||
# (message-only) imply a $/token inflated by 1 + tool/message, which on
|
||||
# tool-heavy traffic (ratios of 5x+ observed) exceeds the model's own
|
||||
# input price. Without the priced breakdown nothing was folded either,
|
||||
# so 0.0 keeps the disjoint field consistent with the sum.
|
||||
delta_tool_savings_usd = (
|
||||
max(_coerce_float(priced.get("tool_schema")), 0.0) if priced is not None else 0.0
|
||||
)
|
||||
delta_input_cost_usd = _estimate_input_cost_usd(
|
||||
model,
|
||||
delta_input_tokens,
|
||||
@@ -1177,6 +1201,11 @@ class SavingsTracker:
|
||||
lifetime.get("output_savings_usd", 0.0) + delta_output_savings_usd,
|
||||
6,
|
||||
)
|
||||
lifetime["tool_tokens_saved"] += delta_tool_tokens_saved
|
||||
lifetime["tool_schema_savings_usd"] = round(
|
||||
lifetime["tool_schema_savings_usd"] + delta_tool_savings_usd,
|
||||
6,
|
||||
)
|
||||
lifetime["total_output_cost_usd"] = round(
|
||||
lifetime.get("total_output_cost_usd", 0.0) + delta_output_cost_usd,
|
||||
6,
|
||||
@@ -1204,6 +1233,11 @@ class SavingsTracker:
|
||||
6,
|
||||
)
|
||||
session["savings_basis"] = _blend_basis(session.get("savings_basis"), delta_basis)
|
||||
session["tool_tokens_saved"] += delta_tool_tokens_saved
|
||||
session["tool_schema_savings_usd"] = round(
|
||||
session["tool_schema_savings_usd"] + delta_tool_savings_usd,
|
||||
6,
|
||||
)
|
||||
session["cache_read_tokens"] += delta_cache_read_tokens
|
||||
session["cache_savings_usd"] = round(
|
||||
session["cache_savings_usd"] + delta_cache_savings_usd,
|
||||
@@ -1253,6 +1287,7 @@ class SavingsTracker:
|
||||
delta_tokens_saved > 0
|
||||
or delta_cache_read_tokens > 0
|
||||
or delta_output_tokens_saved > 0
|
||||
or delta_tool_tokens_saved > 0
|
||||
):
|
||||
self._state["history"].append(
|
||||
{
|
||||
@@ -1267,6 +1302,8 @@ class SavingsTracker:
|
||||
"total_input_cost_usd": lifetime["total_input_cost_usd"],
|
||||
"output_tokens_saved": lifetime.get("output_tokens_saved", 0),
|
||||
"output_savings_usd": lifetime.get("output_savings_usd", 0.0),
|
||||
"tool_tokens_saved": lifetime["tool_tokens_saved"],
|
||||
"tool_schema_savings_usd": lifetime["tool_schema_savings_usd"],
|
||||
"total_output_cost_usd": lifetime.get("total_output_cost_usd", 0.0),
|
||||
}
|
||||
)
|
||||
@@ -1598,6 +1635,8 @@ class SavingsTracker:
|
||||
"total_input_cost_usd",
|
||||
"output_tokens_saved_delta",
|
||||
"output_savings_usd_delta",
|
||||
"tool_tokens_saved_delta",
|
||||
"tool_schema_savings_usd_delta",
|
||||
"total_output_cost_usd_delta",
|
||||
]
|
||||
|
||||
@@ -1638,6 +1677,8 @@ class SavingsTracker:
|
||||
"compression_savings_usd": 0.0,
|
||||
"compression_savings_list_usd": 0.0,
|
||||
"savings_basis": BASIS_UNKNOWN,
|
||||
"tool_tokens_saved": 0,
|
||||
"tool_schema_savings_usd": 0.0,
|
||||
"cache_read_tokens": 0,
|
||||
"cache_savings_usd": 0.0,
|
||||
"total_input_tokens": 0,
|
||||
@@ -1703,6 +1744,8 @@ class SavingsTracker:
|
||||
lifetime_requests = 0
|
||||
lifetime_tokens_saved = 0
|
||||
lifetime_savings_usd = 0.0
|
||||
lifetime_tool_tokens_saved = 0
|
||||
lifetime_tool_savings_usd = 0.0
|
||||
lifetime_cache_read_tokens = 0
|
||||
lifetime_cache_savings_usd = 0.0
|
||||
lifetime_input_tokens = 0
|
||||
@@ -1718,6 +1761,8 @@ class SavingsTracker:
|
||||
lifetime_requests = _coerce_int(lifetime_raw.get("requests"))
|
||||
lifetime_tokens_saved = _coerce_int(lifetime_raw.get("tokens_saved"))
|
||||
lifetime_savings_usd = _coerce_float(lifetime_raw.get("compression_savings_usd"))
|
||||
lifetime_tool_tokens_saved = _coerce_int(lifetime_raw.get("tool_tokens_saved"))
|
||||
lifetime_tool_savings_usd = _coerce_float(lifetime_raw.get("tool_schema_savings_usd"))
|
||||
lifetime_cache_read_tokens = _coerce_int(lifetime_raw.get("cache_read_tokens"))
|
||||
lifetime_cache_savings_usd = _coerce_float(lifetime_raw.get("cache_savings_usd"))
|
||||
lifetime_input_tokens = _coerce_int(lifetime_raw.get("total_input_tokens"))
|
||||
@@ -1743,6 +1788,14 @@ class SavingsTracker:
|
||||
lifetime_savings_usd,
|
||||
_coerce_float(last["compression_savings_usd"]),
|
||||
)
|
||||
lifetime_tool_tokens_saved = max(
|
||||
lifetime_tool_tokens_saved,
|
||||
_coerce_int(last.get("tool_tokens_saved")),
|
||||
)
|
||||
lifetime_tool_savings_usd = max(
|
||||
lifetime_tool_savings_usd,
|
||||
_coerce_float(last.get("tool_schema_savings_usd")),
|
||||
)
|
||||
lifetime_input_tokens = max(
|
||||
lifetime_input_tokens,
|
||||
_coerce_int(last.get("total_input_tokens")),
|
||||
@@ -1797,6 +1850,8 @@ class SavingsTracker:
|
||||
"compression_savings_list_usd": round(lifetime_savings_list_usd, 6),
|
||||
"savings_basis": lifetime_basis,
|
||||
"savings_basis_migrated_at": migrated_at,
|
||||
"tool_tokens_saved": lifetime_tool_tokens_saved,
|
||||
"tool_schema_savings_usd": round(lifetime_tool_savings_usd, 6),
|
||||
"cache_read_tokens": lifetime_cache_read_tokens,
|
||||
"cache_savings_usd": round(lifetime_cache_savings_usd, 6),
|
||||
"total_input_tokens": lifetime_input_tokens,
|
||||
@@ -2083,6 +2138,8 @@ class SavingsTracker:
|
||||
prev_total_input_cost_usd = 0.0
|
||||
prev_output_tokens = 0
|
||||
prev_output_usd = 0.0
|
||||
prev_tool_tokens = 0
|
||||
prev_tool_usd = 0.0
|
||||
prev_output_cost_usd = 0.0
|
||||
prev_cache_read_tokens = 0
|
||||
prev_cache_savings_usd = 0.0
|
||||
@@ -2138,6 +2195,8 @@ class SavingsTracker:
|
||||
total_input_cost_usd = _coerce_float(point.get("total_input_cost_usd"))
|
||||
total_output_tokens = _coerce_int(point.get("output_tokens_saved"))
|
||||
total_output_usd = _coerce_float(point.get("output_savings_usd"))
|
||||
total_tool_tokens = _coerce_int(point.get("tool_tokens_saved"))
|
||||
total_tool_usd = _coerce_float(point.get("tool_schema_savings_usd"))
|
||||
total_output_cost_usd = _coerce_float(point.get("total_output_cost_usd"))
|
||||
delta_tokens = max(total_tokens_saved - prev_total_tokens, 0)
|
||||
delta_usd = max(total_usd - prev_total_usd, 0.0)
|
||||
@@ -2149,6 +2208,8 @@ class SavingsTracker:
|
||||
|
||||
delta_output_tokens = max(total_output_tokens - prev_output_tokens, 0)
|
||||
delta_output_usd = max(total_output_usd - prev_output_usd, 0.0)
|
||||
delta_tool_tokens = max(total_tool_tokens - prev_tool_tokens, 0)
|
||||
delta_tool_usd = max(total_tool_usd - prev_tool_usd, 0.0)
|
||||
delta_output_cost_usd = max(total_output_cost_usd - prev_output_cost_usd, 0.0)
|
||||
|
||||
total_cache_read_tokens = _coerce_int(point.get("cache_read_tokens"))
|
||||
@@ -2166,6 +2227,8 @@ class SavingsTracker:
|
||||
prev_total_input_cost_usd = total_input_cost_usd
|
||||
prev_output_tokens = total_output_tokens
|
||||
prev_output_usd = total_output_usd
|
||||
prev_tool_tokens = total_tool_tokens
|
||||
prev_tool_usd = total_tool_usd
|
||||
prev_output_cost_usd = total_output_cost_usd
|
||||
|
||||
entry = aggregated.setdefault(
|
||||
@@ -2182,6 +2245,10 @@ class SavingsTracker:
|
||||
"total_input_cost_usd": total_input_cost_usd,
|
||||
"output_tokens_saved_delta": 0,
|
||||
"output_savings_usd_delta": 0.0,
|
||||
"tool_tokens_saved_delta": 0,
|
||||
"tool_schema_savings_usd_delta": 0.0,
|
||||
"tool_tokens_saved": total_tool_tokens,
|
||||
"tool_schema_savings_usd": total_tool_usd,
|
||||
"total_output_cost_usd_delta": 0.0,
|
||||
"total_output_cost_usd": total_output_cost_usd,
|
||||
**_empty_cache_delta(),
|
||||
@@ -2208,6 +2275,13 @@ class SavingsTracker:
|
||||
entry["output_savings_usd_delta"] + delta_output_usd,
|
||||
6,
|
||||
)
|
||||
entry["tool_tokens_saved_delta"] += delta_tool_tokens
|
||||
entry["tool_schema_savings_usd_delta"] = round(
|
||||
entry["tool_schema_savings_usd_delta"] + delta_tool_usd,
|
||||
6,
|
||||
)
|
||||
entry["tool_tokens_saved"] = total_tool_tokens
|
||||
entry["tool_schema_savings_usd"] = round(total_tool_usd, 6)
|
||||
entry["total_output_cost_usd_delta"] = round(
|
||||
entry["total_output_cost_usd_delta"] + delta_output_cost_usd,
|
||||
6,
|
||||
|
||||
@@ -84,6 +84,8 @@ def test_savings_tracker_helpers_normalize_inputs_and_paths(tmp_path, monkeypatc
|
||||
"total_input_cost_usd": 0.0,
|
||||
"output_tokens_saved": 0,
|
||||
"output_savings_usd": 0.0,
|
||||
"tool_tokens_saved": 0,
|
||||
"tool_schema_savings_usd": 0.0,
|
||||
"total_output_cost_usd": 0.0,
|
||||
}
|
||||
assert savings_tracker_module._normalize_history_entry({"timestamp": "bad"}) is None
|
||||
@@ -139,6 +141,8 @@ def test_savings_tracker_sanitizes_legacy_state_and_applies_retention(tmp_path):
|
||||
"compression_savings_usd": pytest.approx(0.03),
|
||||
"compression_savings_list_usd": pytest.approx(0.03),
|
||||
"savings_basis": "list",
|
||||
"tool_tokens_saved": 0,
|
||||
"tool_schema_savings_usd": 0.0,
|
||||
"cache_read_tokens": 0,
|
||||
"cache_savings_usd": 0.0,
|
||||
"total_input_tokens": 0,
|
||||
@@ -161,6 +165,8 @@ def test_savings_tracker_sanitizes_legacy_state_and_applies_retention(tmp_path):
|
||||
"total_input_cost_usd": 0.0,
|
||||
"output_tokens_saved": 0,
|
||||
"output_savings_usd": 0.0,
|
||||
"tool_tokens_saved": 0,
|
||||
"tool_schema_savings_usd": 0.0,
|
||||
"total_output_cost_usd": 0.0,
|
||||
}
|
||||
]
|
||||
@@ -186,6 +192,8 @@ def test_non_dict_savings_state_resets_to_default(tmp_path):
|
||||
# Nothing priced yet, so there is no basis to report and nothing to
|
||||
# migrate -- a fresh default, not a migrated pre-v6 state.
|
||||
"savings_basis": "unknown",
|
||||
"tool_tokens_saved": 0,
|
||||
"tool_schema_savings_usd": 0.0,
|
||||
"cache_read_tokens": 0,
|
||||
"cache_savings_usd": 0.0,
|
||||
"total_input_tokens": 0,
|
||||
@@ -659,6 +667,8 @@ def test_display_session_rolls_after_inactivity_and_counts_zero_savings_requests
|
||||
"compression_savings_usd": pytest.approx(0.02),
|
||||
"compression_savings_list_usd": pytest.approx(0.02),
|
||||
"savings_basis": "list",
|
||||
"tool_tokens_saved": 0,
|
||||
"tool_schema_savings_usd": 0.0,
|
||||
"cache_read_tokens": 0,
|
||||
"cache_savings_usd": 0.0,
|
||||
"total_input_tokens": 200,
|
||||
@@ -695,6 +705,8 @@ def test_display_session_rolls_after_inactivity_and_counts_zero_savings_requests
|
||||
"compression_savings_usd": pytest.approx(0.005),
|
||||
"compression_savings_list_usd": pytest.approx(0.005),
|
||||
"savings_basis": "list",
|
||||
"tool_tokens_saved": 0,
|
||||
"tool_schema_savings_usd": 0.0,
|
||||
"cache_read_tokens": 0,
|
||||
"cache_savings_usd": 0.0,
|
||||
"total_input_tokens": 50,
|
||||
@@ -1433,6 +1445,7 @@ def test_stats_history_csv_export_is_frontend_friendly(tmp_path, monkeypatch):
|
||||
"compression_savings_usd,total_input_tokens_delta,total_input_tokens,"
|
||||
"total_input_cost_usd_delta,total_input_cost_usd,"
|
||||
"output_tokens_saved_delta,output_savings_usd_delta,"
|
||||
"tool_tokens_saved_delta,tool_schema_savings_usd_delta,"
|
||||
"total_output_cost_usd_delta"
|
||||
)
|
||||
assert len(lines) >= 2
|
||||
|
||||
@@ -0,0 +1,238 @@
|
||||
"""Tool-schema dollars must be recorded disjointly, not only folded.
|
||||
|
||||
Reported by a desktop consumer of the persisted savings state. Since the
|
||||
attribution unification, ``record_request`` folds the tool-schema layer's
|
||||
dollars into ``compression_savings_usd`` (lifetime, display_session, and every
|
||||
history checkpoint) while ``total_tokens_saved`` on the same checkpoint stays
|
||||
message-only. Any consumer deriving a $/token rate from a checkpoint therefore
|
||||
reads a figure inflated by ``1 + tool/message``:
|
||||
|
||||
- Traffic: message compression 3,299,618 tokens next to 18,435,490 tokens of
|
||||
tool-schema deferral (ratio 5.59x) on one real install.
|
||||
- Implied rate: a $4.99/M blended savings rate became $32.88/M after the fold,
|
||||
which exceeds the input list price of every model in the mix.
|
||||
- Reader impact: consumers that accumulated ``compression_savings_usd`` before
|
||||
the fold shipped had the field's meaning widened underneath them.
|
||||
|
||||
Same resolution shape as the per-model token fix: keep the folded headline
|
||||
exactly as it is, and record the layer disjointly beside it --
|
||||
``tool_tokens_saved`` / ``tool_schema_savings_usd`` on lifetime,
|
||||
display_session, and checkpoints, with matching ``_delta`` fields on rollup
|
||||
buckets -- so message-only dollars are recoverable by subtraction.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import pytest
|
||||
|
||||
from headroom.proxy.savings_tracker import SavingsTracker, _normalize_history_entry
|
||||
|
||||
MODEL = "claude-opus-5"
|
||||
|
||||
|
||||
def _iso(moment: datetime) -> str:
|
||||
return moment.strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
|
||||
def _recent() -> str:
|
||||
"""A timestamp inside the display-session window, anchored to wall clock.
|
||||
|
||||
``_display_session_snapshot_locked`` expires the session against
|
||||
``_utc_now()``, not against the recorded timestamp, so a frozen literal
|
||||
silently stops populating ``display_session`` once it ages past
|
||||
``DEFAULT_DISPLAY_SESSION_INACTIVITY_MINUTES``. This suite was written with
|
||||
a hardcoded date and went red three weeks later for exactly that reason.
|
||||
"""
|
||||
return _iso(datetime.now(timezone.utc) - timedelta(minutes=1))
|
||||
|
||||
|
||||
def _hour_base() -> datetime:
|
||||
"""A whole hour safely in the past, so derived buckets are stable and never future-dated."""
|
||||
now = datetime.now(timezone.utc).replace(minute=0, second=0, microsecond=0)
|
||||
return now - timedelta(hours=3)
|
||||
|
||||
|
||||
def _tracker(tmp_path) -> SavingsTracker:
|
||||
return SavingsTracker(
|
||||
path=str(tmp_path / "proxy_savings.json"),
|
||||
max_history_points=100,
|
||||
max_history_age_days=30,
|
||||
)
|
||||
|
||||
|
||||
def _record(
|
||||
tracker: SavingsTracker,
|
||||
*,
|
||||
tokens_saved: int = 100,
|
||||
tool_search_saved: int = 500,
|
||||
compression_usd: float = 1.0,
|
||||
tool_usd: float = 5.0,
|
||||
timestamp: str | None = None,
|
||||
) -> None:
|
||||
tracker.record_request(
|
||||
model=MODEL,
|
||||
input_tokens=1_000,
|
||||
tokens_saved=tokens_saved,
|
||||
tool_search_saved=tool_search_saved,
|
||||
estimated_savings_usd={
|
||||
"compression": compression_usd,
|
||||
"tool_schema": tool_usd,
|
||||
"output_shaping": 0.0,
|
||||
"provider_cache": 0.0,
|
||||
},
|
||||
timestamp=timestamp or _recent(),
|
||||
)
|
||||
|
||||
|
||||
def test_lifetime_and_session_record_tool_schema_disjointly(tmp_path):
|
||||
tracker = _tracker(tmp_path)
|
||||
_record(tracker)
|
||||
|
||||
snapshot = tracker.snapshot()
|
||||
for block in (snapshot["lifetime"], snapshot["display_session"]):
|
||||
# The folded headline is unchanged: compression + tool_schema.
|
||||
assert block["compression_savings_usd"] == 6.0
|
||||
# The disjoint record makes message-only dollars recoverable.
|
||||
assert block["tool_schema_savings_usd"] == 5.0
|
||||
assert block["compression_savings_usd"] - block["tool_schema_savings_usd"] == 1.0
|
||||
assert block["tool_tokens_saved"] == 500
|
||||
|
||||
|
||||
def test_checkpoints_carry_the_layer_and_survive_reload(tmp_path):
|
||||
tracker = _tracker(tmp_path)
|
||||
_record(tracker)
|
||||
tracker.flush()
|
||||
|
||||
point = tracker.snapshot()["history"][-1]
|
||||
assert point["tool_tokens_saved"] == 500
|
||||
assert point["tool_schema_savings_usd"] == 5.0
|
||||
# Checkpoint tokens stay message-only; the disjoint dollar field is what
|
||||
# lets a consumer pair a consistent numerator with that denominator.
|
||||
assert point["total_tokens_saved"] == 100
|
||||
|
||||
# A fresh tracker on the same file must not zero the layer (the output
|
||||
# fields historically reset on every restart; these must not).
|
||||
reloaded = SavingsTracker(
|
||||
path=str(tmp_path / "proxy_savings.json"),
|
||||
max_history_points=100,
|
||||
max_history_age_days=30,
|
||||
)
|
||||
lifetime = reloaded.snapshot()["lifetime"]
|
||||
assert lifetime["tool_tokens_saved"] == 500
|
||||
assert lifetime["tool_schema_savings_usd"] == 5.0
|
||||
|
||||
|
||||
def test_state_written_before_the_fields_existed_defaults_to_zero(tmp_path):
|
||||
path = tmp_path / "proxy_savings.json"
|
||||
path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"schema_version": 5,
|
||||
"lifetime": {
|
||||
"requests": 3,
|
||||
"tokens_saved": 900,
|
||||
"compression_savings_usd": 0.9,
|
||||
"cache_read_tokens": 0,
|
||||
"cache_savings_usd": 0.0,
|
||||
"total_input_tokens": 5_000,
|
||||
"total_input_cost_usd": 0.05,
|
||||
},
|
||||
"display_session": {},
|
||||
"history": [
|
||||
{
|
||||
"timestamp": "2026-08-20T10:00:00Z",
|
||||
"total_tokens_saved": 900,
|
||||
"compression_savings_usd": 0.9,
|
||||
"total_input_tokens": 5_000,
|
||||
"total_input_cost_usd": 0.05,
|
||||
}
|
||||
],
|
||||
"projects": {},
|
||||
"by_model": {},
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
tracker = SavingsTracker(path=str(path), max_history_points=100, max_history_age_days=30)
|
||||
lifetime = tracker.snapshot()["lifetime"]
|
||||
assert lifetime["tool_tokens_saved"] == 0
|
||||
assert lifetime["tool_schema_savings_usd"] == 0.0
|
||||
assert lifetime["tokens_saved"] == 900
|
||||
|
||||
# Legacy tuple-shaped history entries normalize the same way.
|
||||
normalized = _normalize_history_entry(["2026-08-20T10:00:00Z", 900, 0.9])
|
||||
assert normalized is not None
|
||||
assert normalized["tool_tokens_saved"] == 0
|
||||
assert normalized["tool_schema_savings_usd"] == 0.0
|
||||
|
||||
|
||||
def test_tool_only_request_appends_a_checkpoint(tmp_path):
|
||||
tracker = _tracker(tmp_path)
|
||||
# Deferral with zero message compression: real on tool-heavy traffic, and
|
||||
# previously invisible to the history because the append gate only looked
|
||||
# at message/cache/output deltas.
|
||||
_record(tracker, tokens_saved=0, compression_usd=0.0)
|
||||
|
||||
history = tracker.snapshot()["history"]
|
||||
assert len(history) == 1
|
||||
assert history[-1]["tool_tokens_saved"] == 500
|
||||
assert history[-1]["tool_schema_savings_usd"] == 5.0
|
||||
assert history[-1]["total_tokens_saved"] == 0
|
||||
|
||||
|
||||
def test_rollups_expose_tool_schema_deltas(tmp_path):
|
||||
tracker = _tracker(tmp_path)
|
||||
base = _hour_base()
|
||||
_record(tracker, timestamp=_iso(base + timedelta(minutes=10)))
|
||||
_record(
|
||||
tracker,
|
||||
tool_search_saved=250,
|
||||
tool_usd=2.5,
|
||||
timestamp=_iso(base + timedelta(minutes=40)),
|
||||
)
|
||||
_record(
|
||||
tracker,
|
||||
tool_search_saved=1_000,
|
||||
tool_usd=10.0,
|
||||
timestamp=_iso(base + timedelta(minutes=65)),
|
||||
)
|
||||
|
||||
hourly = tracker.history_response()["series"]["hourly"]
|
||||
assert [point["timestamp"] for point in hourly] == [
|
||||
_iso(base),
|
||||
_iso(base + timedelta(hours=1)),
|
||||
]
|
||||
|
||||
first, second = hourly
|
||||
# Deltas start from a zero baseline, so the series' first checkpoint
|
||||
# contributes its full cumulative -- the same convention as every other
|
||||
# rollup delta in this module. Bucket one therefore carries 500 + 250.
|
||||
assert first["tool_tokens_saved_delta"] == 750
|
||||
assert first["tool_schema_savings_usd_delta"] == 7.5
|
||||
assert first["tool_tokens_saved"] == 750
|
||||
assert second["tool_tokens_saved_delta"] == 1_000
|
||||
assert second["tool_schema_savings_usd_delta"] == 10.0
|
||||
assert second["tool_tokens_saved"] == 1_750
|
||||
|
||||
# The subtraction identity the split exists for: folded minus disjoint
|
||||
# equals message-only dollars, bucket by bucket (two $1 records, then one).
|
||||
assert first["compression_savings_usd_delta"] - first[
|
||||
"tool_schema_savings_usd_delta"
|
||||
] == pytest.approx(2.0)
|
||||
assert second["compression_savings_usd_delta"] - second[
|
||||
"tool_schema_savings_usd_delta"
|
||||
] == pytest.approx(1.0)
|
||||
|
||||
|
||||
def test_rollup_csv_exports_the_delta_columns(tmp_path):
|
||||
tracker = _tracker(tmp_path)
|
||||
base = _hour_base()
|
||||
_record(tracker, timestamp=_iso(base + timedelta(minutes=10)))
|
||||
_record(tracker, timestamp=_iso(base + timedelta(minutes=40)))
|
||||
|
||||
header = tracker.export_csv(series="hourly").splitlines()[0]
|
||||
assert "tool_tokens_saved_delta" in header
|
||||
assert "tool_schema_savings_usd_delta" in header
|
||||
Reference in New Issue
Block a user