fix(savings): record tool-schema dollars disjointly beside the folded headline (#3170)

## Description

Since the attribution unification (#2976, refined by #3123),
`record_request` folds the tool-schema layer's dollars into
`compression_savings_usd` — on the lifetime block, the display session,
and every history checkpoint — while `total_tokens_saved` on those same
checkpoints stays message-only. The two halves of one checkpoint now
disagree about what "saved" means, and any consumer that derives a
$/token rate from a checkpoint reads a figure inflated by `1 +
tool/message`.

That inflation is not hypothetical. On one real install, message
compression was 3,299,618 tokens next to 18,435,490 tokens of
tool-schema deferral (a 5.59x ratio — far from the 0.24x in the sample
#2976 cites, this ratio varies wildly by workload). The blended savings
rate a downstream consumer derived went from $4.99/M to $32.88/M — above
the input list price of every model in the mix, a rate no genuine
input-token saving can reach.

There is also a smaller correctness gap: a request whose only saving is
deferral (`tokens_saved == 0`, common on tool-heavy turns) appended no
checkpoint at all, so its folded dollars landed in whichever later
bucket happened to checkpoint next.

#3155 fixed the token half of this per model, and stated the principle
this PR extends to dollars: kept as its own bucket rather than widened
in place, so the persisted meaning of an existing field never changes
under a reader that predates it. This PR is deliberately **additive** —
`compression_savings_usd` keeps its current folded semantics everywhere,
so the dashboard, PERF headline, and every existing reader are
unchanged. The layer is simply also recorded disjointly beside it, so
message-only dollars are recoverable by subtraction and the checkpoint's
token/dollar pair can be read consistently.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Record `tool_tokens_saved` / `tool_schema_savings_usd` on the lifetime
block and display session, wired through `_default_state`,
`_sanitize_state` (including the history-recovery max), and both
normalizers — the fields survive restarts rather than resetting the way
the ad-hoc output fields do.
- Carry both fields on every history checkpoint, and extend the
checkpoint append gate so a deferral-only request checkpoints too.
- Expose `tool_tokens_saved_delta` / `tool_schema_savings_usd_delta`
(plus cumulatives) on rollup buckets, following the exact delta
convention of the existing series, and add the delta columns to the
rollup CSV export.
- State files written before these fields existed load with both at 0;
legacy tuple-shaped history entries normalize the same way.
- New test module `tests/test_tool_schema_savings_split.py` (6 tests)
pinning the disjoint recording, reload durability, legacy-state
defaults, the deferral-only checkpoint, the rollup deltas, and the
subtraction identity; existing shape assertions in
`tests/test_proxy_savings_history.py` updated for the two new fields.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_proxy_savings_history.py \
    tests/test_tool_schema_savings_split.py tests/test_per_model_tool_savings.py \
    tests/test_savings_tool_search_aggregation.py tests/test_savings_tracker_zero_price.py \
    tests/test_agent_savings.py
======================== 103 passed, 1 warning in 7.36s ========================

$ uv run --frozen --extra dev pytest tests/test_stats_new_input_savings_rate.py \
    tests/test_dashboard_token_savings.py tests/test_proxy_project_savings.py \
    tests/test_output_savings.py tests/test_savings_ledger.py
=================== 76 passed, 2 skipped, 1 warning in 8.15s ===================

$ uv run --frozen ruff check headroom/proxy/savings_tracker.py tests/test_tool_schema_savings_split.py tests/test_proxy_savings_history.py
All checks passed!
$ uv run --frozen ruff format --check headroom/proxy/savings_tracker.py tests/test_tool_schema_savings_split.py tests/test_proxy_savings_history.py
3 files already formatted
$ uv run --frozen mypy headroom/proxy/savings_tracker.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS, Python 3.10 venv via `uv run --frozen`, this
branch.
- Exact command / steps: one `record_request` shaped like the reporting
install's traffic (100 message tokens saved, 559 deferred schema tokens,
claude-opus-5 pricing at $5/M input), then derive $/token from the
appended checkpoint both ways.
- Observed result: the naive checkpoint-derived rate reproduces the
impossible $32.88/M seen in the field almost exactly, and subtracting
the new disjoint field recovers the model's actual input rate:

```text
checkpoint: total_tokens_saved=100 compression_savings_usd=0.003295 tool_tokens_saved=559 tool_schema_savings_usd=0.002795
folded-dollars / message-tokens rate: $32.95/M  (input list price is $5/M)
after subtracting the disjoint field:  $5.00/M
```

- Not tested: a long-lived multi-process proxy soak, and the downstream
consumer's subtraction path itself (that lands with the consumer release
that reads the new fields).

## Runtime Rollout Safety

- Rollout-managed feature(s): None -- persisted savings accounting only;
no rollout-gated runtime behavior is touched.
- Minimum rollout channel: N/A
- Stable/default behavior changed: No. Additive fields beside existing
ones; `compression_savings_usd` keeps its current folded semantics, so
every existing reader sees identical values.
- Kill switch / disable path: None needed -- old readers ignore the new
fields, and state files written before them load with both at 0 (covered
by tests).
- Unsafe override required: No
- Qualification impact: None beyond the unit suites run above; displayed
figures are unchanged.
- Rollback path: `git revert` -- the fields stop being written and
remain optional on read, so state files produced while this was live
stay loadable.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

Related: #3155 (token half of the same split), #3171 (companion PR on
what the tool-schema bucket should be priced at — independent; either
lands without the other).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
gglucass
2026-09-30 11:27:41 -05:00
committed by GitHub
co-authored by Claude Fable 5
parent b73adaa0ce
commit c719d4af55
3 changed files with 325 additions and 0 deletions
+74
View File
@@ -612,6 +612,8 @@ def _normalize_history_entry(entry: Any) -> dict[str, Any] | None:
total_input_cost_usd = 0.0
output_tokens_saved = 0
output_savings_usd = 0.0
tool_tokens_saved = 0
tool_schema_savings_usd = 0.0
total_output_cost_usd = 0.0
provider = PROVIDER_UNKNOWN
model = MODEL_UNKNOWN
@@ -629,6 +631,8 @@ def _normalize_history_entry(entry: Any) -> dict[str, Any] | None:
total_input_cost_usd = _coerce_float(entry.get("total_input_cost_usd"))
output_tokens_saved = _coerce_int(entry.get("output_tokens_saved"))
output_savings_usd = _coerce_float(entry.get("output_savings_usd"))
tool_tokens_saved = _coerce_int(entry.get("tool_tokens_saved"))
tool_schema_savings_usd = _coerce_float(entry.get("tool_schema_savings_usd"))
total_output_cost_usd = _coerce_float(entry.get("total_output_cost_usd"))
provider = _normalize_provider(entry.get("provider"))
model = _normalize_model(entry.get("model"))
@@ -659,6 +663,8 @@ def _normalize_history_entry(entry: Any) -> dict[str, Any] | None:
"total_input_cost_usd": round(total_input_cost_usd, 6),
"output_tokens_saved": output_tokens_saved,
"output_savings_usd": round(output_savings_usd, 6),
"tool_tokens_saved": tool_tokens_saved,
"tool_schema_savings_usd": round(tool_schema_savings_usd, 6),
"total_output_cost_usd": round(total_output_cost_usd, 6),
}
@@ -670,6 +676,8 @@ def _empty_display_session() -> dict[str, Any]:
"compression_savings_usd": 0.0,
"compression_savings_list_usd": 0.0,
"savings_basis": BASIS_UNKNOWN,
"tool_tokens_saved": 0,
"tool_schema_savings_usd": 0.0,
"cache_read_tokens": 0,
"cache_savings_usd": 0.0,
"total_input_tokens": 0,
@@ -806,6 +814,11 @@ def _normalize_display_session(entry: Any) -> dict[str, Any]:
"compression_savings_usd": round(savings_usd, 6),
"compression_savings_list_usd": round(savings_list_usd, 6),
"savings_basis": savings_basis,
"tool_tokens_saved": _coerce_int(entry.get("tool_tokens_saved")),
"tool_schema_savings_usd": round(
_coerce_float(entry.get("tool_schema_savings_usd")),
6,
),
"cache_read_tokens": _coerce_int(entry.get("cache_read_tokens")),
"cache_savings_usd": round(
_coerce_float(entry.get("cache_savings_usd")),
@@ -1112,6 +1125,17 @@ class SavingsTracker:
if priced is not None
else _estimate_cache_savings_usd(model, delta_cache_read_tokens)
)
# The same figure that was just folded into ``delta_savings_usd``, kept
# as its own delta so the layer is also recorded disjointly below. A
# consumer can then recover message-only dollars by subtraction --
# without this, checkpoint dollars (folded) over checkpoint tokens
# (message-only) imply a $/token inflated by 1 + tool/message, which on
# tool-heavy traffic (ratios of 5x+ observed) exceeds the model's own
# input price. Without the priced breakdown nothing was folded either,
# so 0.0 keeps the disjoint field consistent with the sum.
delta_tool_savings_usd = (
max(_coerce_float(priced.get("tool_schema")), 0.0) if priced is not None else 0.0
)
delta_input_cost_usd = _estimate_input_cost_usd(
model,
delta_input_tokens,
@@ -1177,6 +1201,11 @@ class SavingsTracker:
lifetime.get("output_savings_usd", 0.0) + delta_output_savings_usd,
6,
)
lifetime["tool_tokens_saved"] += delta_tool_tokens_saved
lifetime["tool_schema_savings_usd"] = round(
lifetime["tool_schema_savings_usd"] + delta_tool_savings_usd,
6,
)
lifetime["total_output_cost_usd"] = round(
lifetime.get("total_output_cost_usd", 0.0) + delta_output_cost_usd,
6,
@@ -1204,6 +1233,11 @@ class SavingsTracker:
6,
)
session["savings_basis"] = _blend_basis(session.get("savings_basis"), delta_basis)
session["tool_tokens_saved"] += delta_tool_tokens_saved
session["tool_schema_savings_usd"] = round(
session["tool_schema_savings_usd"] + delta_tool_savings_usd,
6,
)
session["cache_read_tokens"] += delta_cache_read_tokens
session["cache_savings_usd"] = round(
session["cache_savings_usd"] + delta_cache_savings_usd,
@@ -1253,6 +1287,7 @@ class SavingsTracker:
delta_tokens_saved > 0
or delta_cache_read_tokens > 0
or delta_output_tokens_saved > 0
or delta_tool_tokens_saved > 0
):
self._state["history"].append(
{
@@ -1267,6 +1302,8 @@ class SavingsTracker:
"total_input_cost_usd": lifetime["total_input_cost_usd"],
"output_tokens_saved": lifetime.get("output_tokens_saved", 0),
"output_savings_usd": lifetime.get("output_savings_usd", 0.0),
"tool_tokens_saved": lifetime["tool_tokens_saved"],
"tool_schema_savings_usd": lifetime["tool_schema_savings_usd"],
"total_output_cost_usd": lifetime.get("total_output_cost_usd", 0.0),
}
)
@@ -1598,6 +1635,8 @@ class SavingsTracker:
"total_input_cost_usd",
"output_tokens_saved_delta",
"output_savings_usd_delta",
"tool_tokens_saved_delta",
"tool_schema_savings_usd_delta",
"total_output_cost_usd_delta",
]
@@ -1638,6 +1677,8 @@ class SavingsTracker:
"compression_savings_usd": 0.0,
"compression_savings_list_usd": 0.0,
"savings_basis": BASIS_UNKNOWN,
"tool_tokens_saved": 0,
"tool_schema_savings_usd": 0.0,
"cache_read_tokens": 0,
"cache_savings_usd": 0.0,
"total_input_tokens": 0,
@@ -1703,6 +1744,8 @@ class SavingsTracker:
lifetime_requests = 0
lifetime_tokens_saved = 0
lifetime_savings_usd = 0.0
lifetime_tool_tokens_saved = 0
lifetime_tool_savings_usd = 0.0
lifetime_cache_read_tokens = 0
lifetime_cache_savings_usd = 0.0
lifetime_input_tokens = 0
@@ -1718,6 +1761,8 @@ class SavingsTracker:
lifetime_requests = _coerce_int(lifetime_raw.get("requests"))
lifetime_tokens_saved = _coerce_int(lifetime_raw.get("tokens_saved"))
lifetime_savings_usd = _coerce_float(lifetime_raw.get("compression_savings_usd"))
lifetime_tool_tokens_saved = _coerce_int(lifetime_raw.get("tool_tokens_saved"))
lifetime_tool_savings_usd = _coerce_float(lifetime_raw.get("tool_schema_savings_usd"))
lifetime_cache_read_tokens = _coerce_int(lifetime_raw.get("cache_read_tokens"))
lifetime_cache_savings_usd = _coerce_float(lifetime_raw.get("cache_savings_usd"))
lifetime_input_tokens = _coerce_int(lifetime_raw.get("total_input_tokens"))
@@ -1743,6 +1788,14 @@ class SavingsTracker:
lifetime_savings_usd,
_coerce_float(last["compression_savings_usd"]),
)
lifetime_tool_tokens_saved = max(
lifetime_tool_tokens_saved,
_coerce_int(last.get("tool_tokens_saved")),
)
lifetime_tool_savings_usd = max(
lifetime_tool_savings_usd,
_coerce_float(last.get("tool_schema_savings_usd")),
)
lifetime_input_tokens = max(
lifetime_input_tokens,
_coerce_int(last.get("total_input_tokens")),
@@ -1797,6 +1850,8 @@ class SavingsTracker:
"compression_savings_list_usd": round(lifetime_savings_list_usd, 6),
"savings_basis": lifetime_basis,
"savings_basis_migrated_at": migrated_at,
"tool_tokens_saved": lifetime_tool_tokens_saved,
"tool_schema_savings_usd": round(lifetime_tool_savings_usd, 6),
"cache_read_tokens": lifetime_cache_read_tokens,
"cache_savings_usd": round(lifetime_cache_savings_usd, 6),
"total_input_tokens": lifetime_input_tokens,
@@ -2083,6 +2138,8 @@ class SavingsTracker:
prev_total_input_cost_usd = 0.0
prev_output_tokens = 0
prev_output_usd = 0.0
prev_tool_tokens = 0
prev_tool_usd = 0.0
prev_output_cost_usd = 0.0
prev_cache_read_tokens = 0
prev_cache_savings_usd = 0.0
@@ -2138,6 +2195,8 @@ class SavingsTracker:
total_input_cost_usd = _coerce_float(point.get("total_input_cost_usd"))
total_output_tokens = _coerce_int(point.get("output_tokens_saved"))
total_output_usd = _coerce_float(point.get("output_savings_usd"))
total_tool_tokens = _coerce_int(point.get("tool_tokens_saved"))
total_tool_usd = _coerce_float(point.get("tool_schema_savings_usd"))
total_output_cost_usd = _coerce_float(point.get("total_output_cost_usd"))
delta_tokens = max(total_tokens_saved - prev_total_tokens, 0)
delta_usd = max(total_usd - prev_total_usd, 0.0)
@@ -2149,6 +2208,8 @@ class SavingsTracker:
delta_output_tokens = max(total_output_tokens - prev_output_tokens, 0)
delta_output_usd = max(total_output_usd - prev_output_usd, 0.0)
delta_tool_tokens = max(total_tool_tokens - prev_tool_tokens, 0)
delta_tool_usd = max(total_tool_usd - prev_tool_usd, 0.0)
delta_output_cost_usd = max(total_output_cost_usd - prev_output_cost_usd, 0.0)
total_cache_read_tokens = _coerce_int(point.get("cache_read_tokens"))
@@ -2166,6 +2227,8 @@ class SavingsTracker:
prev_total_input_cost_usd = total_input_cost_usd
prev_output_tokens = total_output_tokens
prev_output_usd = total_output_usd
prev_tool_tokens = total_tool_tokens
prev_tool_usd = total_tool_usd
prev_output_cost_usd = total_output_cost_usd
entry = aggregated.setdefault(
@@ -2182,6 +2245,10 @@ class SavingsTracker:
"total_input_cost_usd": total_input_cost_usd,
"output_tokens_saved_delta": 0,
"output_savings_usd_delta": 0.0,
"tool_tokens_saved_delta": 0,
"tool_schema_savings_usd_delta": 0.0,
"tool_tokens_saved": total_tool_tokens,
"tool_schema_savings_usd": total_tool_usd,
"total_output_cost_usd_delta": 0.0,
"total_output_cost_usd": total_output_cost_usd,
**_empty_cache_delta(),
@@ -2208,6 +2275,13 @@ class SavingsTracker:
entry["output_savings_usd_delta"] + delta_output_usd,
6,
)
entry["tool_tokens_saved_delta"] += delta_tool_tokens
entry["tool_schema_savings_usd_delta"] = round(
entry["tool_schema_savings_usd_delta"] + delta_tool_usd,
6,
)
entry["tool_tokens_saved"] = total_tool_tokens
entry["tool_schema_savings_usd"] = round(total_tool_usd, 6)
entry["total_output_cost_usd_delta"] = round(
entry["total_output_cost_usd_delta"] + delta_output_cost_usd,
6,
+13
View File
@@ -84,6 +84,8 @@ def test_savings_tracker_helpers_normalize_inputs_and_paths(tmp_path, monkeypatc
"total_input_cost_usd": 0.0,
"output_tokens_saved": 0,
"output_savings_usd": 0.0,
"tool_tokens_saved": 0,
"tool_schema_savings_usd": 0.0,
"total_output_cost_usd": 0.0,
}
assert savings_tracker_module._normalize_history_entry({"timestamp": "bad"}) is None
@@ -139,6 +141,8 @@ def test_savings_tracker_sanitizes_legacy_state_and_applies_retention(tmp_path):
"compression_savings_usd": pytest.approx(0.03),
"compression_savings_list_usd": pytest.approx(0.03),
"savings_basis": "list",
"tool_tokens_saved": 0,
"tool_schema_savings_usd": 0.0,
"cache_read_tokens": 0,
"cache_savings_usd": 0.0,
"total_input_tokens": 0,
@@ -161,6 +165,8 @@ def test_savings_tracker_sanitizes_legacy_state_and_applies_retention(tmp_path):
"total_input_cost_usd": 0.0,
"output_tokens_saved": 0,
"output_savings_usd": 0.0,
"tool_tokens_saved": 0,
"tool_schema_savings_usd": 0.0,
"total_output_cost_usd": 0.0,
}
]
@@ -186,6 +192,8 @@ def test_non_dict_savings_state_resets_to_default(tmp_path):
# Nothing priced yet, so there is no basis to report and nothing to
# migrate -- a fresh default, not a migrated pre-v6 state.
"savings_basis": "unknown",
"tool_tokens_saved": 0,
"tool_schema_savings_usd": 0.0,
"cache_read_tokens": 0,
"cache_savings_usd": 0.0,
"total_input_tokens": 0,
@@ -659,6 +667,8 @@ def test_display_session_rolls_after_inactivity_and_counts_zero_savings_requests
"compression_savings_usd": pytest.approx(0.02),
"compression_savings_list_usd": pytest.approx(0.02),
"savings_basis": "list",
"tool_tokens_saved": 0,
"tool_schema_savings_usd": 0.0,
"cache_read_tokens": 0,
"cache_savings_usd": 0.0,
"total_input_tokens": 200,
@@ -695,6 +705,8 @@ def test_display_session_rolls_after_inactivity_and_counts_zero_savings_requests
"compression_savings_usd": pytest.approx(0.005),
"compression_savings_list_usd": pytest.approx(0.005),
"savings_basis": "list",
"tool_tokens_saved": 0,
"tool_schema_savings_usd": 0.0,
"cache_read_tokens": 0,
"cache_savings_usd": 0.0,
"total_input_tokens": 50,
@@ -1433,6 +1445,7 @@ def test_stats_history_csv_export_is_frontend_friendly(tmp_path, monkeypatch):
"compression_savings_usd,total_input_tokens_delta,total_input_tokens,"
"total_input_cost_usd_delta,total_input_cost_usd,"
"output_tokens_saved_delta,output_savings_usd_delta,"
"tool_tokens_saved_delta,tool_schema_savings_usd_delta,"
"total_output_cost_usd_delta"
)
assert len(lines) >= 2
+238
View File
@@ -0,0 +1,238 @@
"""Tool-schema dollars must be recorded disjointly, not only folded.
Reported by a desktop consumer of the persisted savings state. Since the
attribution unification, ``record_request`` folds the tool-schema layer's
dollars into ``compression_savings_usd`` (lifetime, display_session, and every
history checkpoint) while ``total_tokens_saved`` on the same checkpoint stays
message-only. Any consumer deriving a $/token rate from a checkpoint therefore
reads a figure inflated by ``1 + tool/message``:
- Traffic: message compression 3,299,618 tokens next to 18,435,490 tokens of
tool-schema deferral (ratio 5.59x) on one real install.
- Implied rate: a $4.99/M blended savings rate became $32.88/M after the fold,
which exceeds the input list price of every model in the mix.
- Reader impact: consumers that accumulated ``compression_savings_usd`` before
the fold shipped had the field's meaning widened underneath them.
Same resolution shape as the per-model token fix: keep the folded headline
exactly as it is, and record the layer disjointly beside it --
``tool_tokens_saved`` / ``tool_schema_savings_usd`` on lifetime,
display_session, and checkpoints, with matching ``_delta`` fields on rollup
buckets -- so message-only dollars are recoverable by subtraction.
"""
from __future__ import annotations
import json
from datetime import datetime, timedelta, timezone
import pytest
from headroom.proxy.savings_tracker import SavingsTracker, _normalize_history_entry
MODEL = "claude-opus-5"
def _iso(moment: datetime) -> str:
return moment.strftime("%Y-%m-%dT%H:%M:%SZ")
def _recent() -> str:
"""A timestamp inside the display-session window, anchored to wall clock.
``_display_session_snapshot_locked`` expires the session against
``_utc_now()``, not against the recorded timestamp, so a frozen literal
silently stops populating ``display_session`` once it ages past
``DEFAULT_DISPLAY_SESSION_INACTIVITY_MINUTES``. This suite was written with
a hardcoded date and went red three weeks later for exactly that reason.
"""
return _iso(datetime.now(timezone.utc) - timedelta(minutes=1))
def _hour_base() -> datetime:
"""A whole hour safely in the past, so derived buckets are stable and never future-dated."""
now = datetime.now(timezone.utc).replace(minute=0, second=0, microsecond=0)
return now - timedelta(hours=3)
def _tracker(tmp_path) -> SavingsTracker:
return SavingsTracker(
path=str(tmp_path / "proxy_savings.json"),
max_history_points=100,
max_history_age_days=30,
)
def _record(
tracker: SavingsTracker,
*,
tokens_saved: int = 100,
tool_search_saved: int = 500,
compression_usd: float = 1.0,
tool_usd: float = 5.0,
timestamp: str | None = None,
) -> None:
tracker.record_request(
model=MODEL,
input_tokens=1_000,
tokens_saved=tokens_saved,
tool_search_saved=tool_search_saved,
estimated_savings_usd={
"compression": compression_usd,
"tool_schema": tool_usd,
"output_shaping": 0.0,
"provider_cache": 0.0,
},
timestamp=timestamp or _recent(),
)
def test_lifetime_and_session_record_tool_schema_disjointly(tmp_path):
tracker = _tracker(tmp_path)
_record(tracker)
snapshot = tracker.snapshot()
for block in (snapshot["lifetime"], snapshot["display_session"]):
# The folded headline is unchanged: compression + tool_schema.
assert block["compression_savings_usd"] == 6.0
# The disjoint record makes message-only dollars recoverable.
assert block["tool_schema_savings_usd"] == 5.0
assert block["compression_savings_usd"] - block["tool_schema_savings_usd"] == 1.0
assert block["tool_tokens_saved"] == 500
def test_checkpoints_carry_the_layer_and_survive_reload(tmp_path):
tracker = _tracker(tmp_path)
_record(tracker)
tracker.flush()
point = tracker.snapshot()["history"][-1]
assert point["tool_tokens_saved"] == 500
assert point["tool_schema_savings_usd"] == 5.0
# Checkpoint tokens stay message-only; the disjoint dollar field is what
# lets a consumer pair a consistent numerator with that denominator.
assert point["total_tokens_saved"] == 100
# A fresh tracker on the same file must not zero the layer (the output
# fields historically reset on every restart; these must not).
reloaded = SavingsTracker(
path=str(tmp_path / "proxy_savings.json"),
max_history_points=100,
max_history_age_days=30,
)
lifetime = reloaded.snapshot()["lifetime"]
assert lifetime["tool_tokens_saved"] == 500
assert lifetime["tool_schema_savings_usd"] == 5.0
def test_state_written_before_the_fields_existed_defaults_to_zero(tmp_path):
path = tmp_path / "proxy_savings.json"
path.write_text(
json.dumps(
{
"schema_version": 5,
"lifetime": {
"requests": 3,
"tokens_saved": 900,
"compression_savings_usd": 0.9,
"cache_read_tokens": 0,
"cache_savings_usd": 0.0,
"total_input_tokens": 5_000,
"total_input_cost_usd": 0.05,
},
"display_session": {},
"history": [
{
"timestamp": "2026-08-20T10:00:00Z",
"total_tokens_saved": 900,
"compression_savings_usd": 0.9,
"total_input_tokens": 5_000,
"total_input_cost_usd": 0.05,
}
],
"projects": {},
"by_model": {},
}
)
)
tracker = SavingsTracker(path=str(path), max_history_points=100, max_history_age_days=30)
lifetime = tracker.snapshot()["lifetime"]
assert lifetime["tool_tokens_saved"] == 0
assert lifetime["tool_schema_savings_usd"] == 0.0
assert lifetime["tokens_saved"] == 900
# Legacy tuple-shaped history entries normalize the same way.
normalized = _normalize_history_entry(["2026-08-20T10:00:00Z", 900, 0.9])
assert normalized is not None
assert normalized["tool_tokens_saved"] == 0
assert normalized["tool_schema_savings_usd"] == 0.0
def test_tool_only_request_appends_a_checkpoint(tmp_path):
tracker = _tracker(tmp_path)
# Deferral with zero message compression: real on tool-heavy traffic, and
# previously invisible to the history because the append gate only looked
# at message/cache/output deltas.
_record(tracker, tokens_saved=0, compression_usd=0.0)
history = tracker.snapshot()["history"]
assert len(history) == 1
assert history[-1]["tool_tokens_saved"] == 500
assert history[-1]["tool_schema_savings_usd"] == 5.0
assert history[-1]["total_tokens_saved"] == 0
def test_rollups_expose_tool_schema_deltas(tmp_path):
tracker = _tracker(tmp_path)
base = _hour_base()
_record(tracker, timestamp=_iso(base + timedelta(minutes=10)))
_record(
tracker,
tool_search_saved=250,
tool_usd=2.5,
timestamp=_iso(base + timedelta(minutes=40)),
)
_record(
tracker,
tool_search_saved=1_000,
tool_usd=10.0,
timestamp=_iso(base + timedelta(minutes=65)),
)
hourly = tracker.history_response()["series"]["hourly"]
assert [point["timestamp"] for point in hourly] == [
_iso(base),
_iso(base + timedelta(hours=1)),
]
first, second = hourly
# Deltas start from a zero baseline, so the series' first checkpoint
# contributes its full cumulative -- the same convention as every other
# rollup delta in this module. Bucket one therefore carries 500 + 250.
assert first["tool_tokens_saved_delta"] == 750
assert first["tool_schema_savings_usd_delta"] == 7.5
assert first["tool_tokens_saved"] == 750
assert second["tool_tokens_saved_delta"] == 1_000
assert second["tool_schema_savings_usd_delta"] == 10.0
assert second["tool_tokens_saved"] == 1_750
# The subtraction identity the split exists for: folded minus disjoint
# equals message-only dollars, bucket by bucket (two $1 records, then one).
assert first["compression_savings_usd_delta"] - first[
"tool_schema_savings_usd_delta"
] == pytest.approx(2.0)
assert second["compression_savings_usd_delta"] - second[
"tool_schema_savings_usd_delta"
] == pytest.approx(1.0)
def test_rollup_csv_exports_the_delta_columns(tmp_path):
tracker = _tracker(tmp_path)
base = _hour_base()
_record(tracker, timestamp=_iso(base + timedelta(minutes=10)))
_record(tracker, timestamp=_iso(base + timedelta(minutes=40)))
header = tracker.export_csv(series="hourly").splitlines()[0]
assert "tool_tokens_saved_delta" in header
assert "tool_schema_savings_usd_delta" in header