mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 18:41:07 +08:00
## Description Since the attribution unification (#2976, refined by #3123), `record_request` folds the tool-schema layer's dollars into `compression_savings_usd` — on the lifetime block, the display session, and every history checkpoint — while `total_tokens_saved` on those same checkpoints stays message-only. The two halves of one checkpoint now disagree about what "saved" means, and any consumer that derives a $/token rate from a checkpoint reads a figure inflated by `1 + tool/message`. That inflation is not hypothetical. On one real install, message compression was 3,299,618 tokens next to 18,435,490 tokens of tool-schema deferral (a 5.59x ratio — far from the 0.24x in the sample #2976 cites, this ratio varies wildly by workload). The blended savings rate a downstream consumer derived went from $4.99/M to $32.88/M — above the input list price of every model in the mix, a rate no genuine input-token saving can reach. There is also a smaller correctness gap: a request whose only saving is deferral (`tokens_saved == 0`, common on tool-heavy turns) appended no checkpoint at all, so its folded dollars landed in whichever later bucket happened to checkpoint next. #3155 fixed the token half of this per model, and stated the principle this PR extends to dollars: kept as its own bucket rather than widened in place, so the persisted meaning of an existing field never changes under a reader that predates it. This PR is deliberately **additive** — `compression_savings_usd` keeps its current folded semantics everywhere, so the dashboard, PERF headline, and every existing reader are unchanged. The layer is simply also recorded disjointly beside it, so message-only dollars are recoverable by subtraction and the checkpoint's token/dollar pair can be read consistently. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Record `tool_tokens_saved` / `tool_schema_savings_usd` on the lifetime block and display session, wired through `_default_state`, `_sanitize_state` (including the history-recovery max), and both normalizers — the fields survive restarts rather than resetting the way the ad-hoc output fields do. - Carry both fields on every history checkpoint, and extend the checkpoint append gate so a deferral-only request checkpoints too. - Expose `tool_tokens_saved_delta` / `tool_schema_savings_usd_delta` (plus cumulatives) on rollup buckets, following the exact delta convention of the existing series, and add the delta columns to the rollup CSV export. - State files written before these fields existed load with both at 0; legacy tuple-shaped history entries normalize the same way. - New test module `tests/test_tool_schema_savings_split.py` (6 tests) pinning the disjoint recording, reload durability, legacy-state defaults, the deferral-only checkpoint, the rollup deltas, and the subtraction identity; existing shape assertions in `tests/test_proxy_savings_history.py` updated for the two new fields. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output ```text $ uv run --frozen --extra dev pytest tests/test_proxy_savings_history.py \ tests/test_tool_schema_savings_split.py tests/test_per_model_tool_savings.py \ tests/test_savings_tool_search_aggregation.py tests/test_savings_tracker_zero_price.py \ tests/test_agent_savings.py ======================== 103 passed, 1 warning in 7.36s ======================== $ uv run --frozen --extra dev pytest tests/test_stats_new_input_savings_rate.py \ tests/test_dashboard_token_savings.py tests/test_proxy_project_savings.py \ tests/test_output_savings.py tests/test_savings_ledger.py =================== 76 passed, 2 skipped, 1 warning in 8.15s =================== $ uv run --frozen ruff check headroom/proxy/savings_tracker.py tests/test_tool_schema_savings_split.py tests/test_proxy_savings_history.py All checks passed! $ uv run --frozen ruff format --check headroom/proxy/savings_tracker.py tests/test_tool_schema_savings_split.py tests/test_proxy_savings_history.py 3 files already formatted $ uv run --frozen mypy headroom/proxy/savings_tracker.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS, Python 3.10 venv via `uv run --frozen`, this branch. - Exact command / steps: one `record_request` shaped like the reporting install's traffic (100 message tokens saved, 559 deferred schema tokens, claude-opus-5 pricing at $5/M input), then derive $/token from the appended checkpoint both ways. - Observed result: the naive checkpoint-derived rate reproduces the impossible $32.88/M seen in the field almost exactly, and subtracting the new disjoint field recovers the model's actual input rate: ```text checkpoint: total_tokens_saved=100 compression_savings_usd=0.003295 tool_tokens_saved=559 tool_schema_savings_usd=0.002795 folded-dollars / message-tokens rate: $32.95/M (input list price is $5/M) after subtracting the disjoint field: $5.00/M ``` - Not tested: a long-lived multi-process proxy soak, and the downstream consumer's subtraction path itself (that lands with the consumer release that reads the new fields). ## Runtime Rollout Safety - Rollout-managed feature(s): None -- persisted savings accounting only; no rollout-gated runtime behavior is touched. - Minimum rollout channel: N/A - Stable/default behavior changed: No. Additive fields beside existing ones; `compression_savings_usd` keeps its current folded semantics, so every existing reader sees identical values. - Kill switch / disable path: None needed -- old readers ignore the new fields, and state files written before them load with both at 0 (covered by tests). - Unsafe override required: No - Qualification impact: None beyond the unit suites run above; displayed figures are unchanged. - Rollback path: `git revert` -- the fields stop being written and remain optional on read, so state files produced while this was live stay loadable. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review Related: #3155 (token half of the same split), #3171 (companion PR on what the tool-schema bucket should be priced at — independent; either lands without the other). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
239 lines
8.8 KiB
Python
239 lines
8.8 KiB
Python
"""Tool-schema dollars must be recorded disjointly, not only folded.
|
|
|
|
Reported by a desktop consumer of the persisted savings state. Since the
|
|
attribution unification, ``record_request`` folds the tool-schema layer's
|
|
dollars into ``compression_savings_usd`` (lifetime, display_session, and every
|
|
history checkpoint) while ``total_tokens_saved`` on the same checkpoint stays
|
|
message-only. Any consumer deriving a $/token rate from a checkpoint therefore
|
|
reads a figure inflated by ``1 + tool/message``:
|
|
|
|
- Traffic: message compression 3,299,618 tokens next to 18,435,490 tokens of
|
|
tool-schema deferral (ratio 5.59x) on one real install.
|
|
- Implied rate: a $4.99/M blended savings rate became $32.88/M after the fold,
|
|
which exceeds the input list price of every model in the mix.
|
|
- Reader impact: consumers that accumulated ``compression_savings_usd`` before
|
|
the fold shipped had the field's meaning widened underneath them.
|
|
|
|
Same resolution shape as the per-model token fix: keep the folded headline
|
|
exactly as it is, and record the layer disjointly beside it --
|
|
``tool_tokens_saved`` / ``tool_schema_savings_usd`` on lifetime,
|
|
display_session, and checkpoints, with matching ``_delta`` fields on rollup
|
|
buckets -- so message-only dollars are recoverable by subtraction.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
from datetime import datetime, timedelta, timezone
|
|
|
|
import pytest
|
|
|
|
from headroom.proxy.savings_tracker import SavingsTracker, _normalize_history_entry
|
|
|
|
MODEL = "claude-opus-5"
|
|
|
|
|
|
def _iso(moment: datetime) -> str:
|
|
return moment.strftime("%Y-%m-%dT%H:%M:%SZ")
|
|
|
|
|
|
def _recent() -> str:
|
|
"""A timestamp inside the display-session window, anchored to wall clock.
|
|
|
|
``_display_session_snapshot_locked`` expires the session against
|
|
``_utc_now()``, not against the recorded timestamp, so a frozen literal
|
|
silently stops populating ``display_session`` once it ages past
|
|
``DEFAULT_DISPLAY_SESSION_INACTIVITY_MINUTES``. This suite was written with
|
|
a hardcoded date and went red three weeks later for exactly that reason.
|
|
"""
|
|
return _iso(datetime.now(timezone.utc) - timedelta(minutes=1))
|
|
|
|
|
|
def _hour_base() -> datetime:
|
|
"""A whole hour safely in the past, so derived buckets are stable and never future-dated."""
|
|
now = datetime.now(timezone.utc).replace(minute=0, second=0, microsecond=0)
|
|
return now - timedelta(hours=3)
|
|
|
|
|
|
def _tracker(tmp_path) -> SavingsTracker:
|
|
return SavingsTracker(
|
|
path=str(tmp_path / "proxy_savings.json"),
|
|
max_history_points=100,
|
|
max_history_age_days=30,
|
|
)
|
|
|
|
|
|
def _record(
|
|
tracker: SavingsTracker,
|
|
*,
|
|
tokens_saved: int = 100,
|
|
tool_search_saved: int = 500,
|
|
compression_usd: float = 1.0,
|
|
tool_usd: float = 5.0,
|
|
timestamp: str | None = None,
|
|
) -> None:
|
|
tracker.record_request(
|
|
model=MODEL,
|
|
input_tokens=1_000,
|
|
tokens_saved=tokens_saved,
|
|
tool_search_saved=tool_search_saved,
|
|
estimated_savings_usd={
|
|
"compression": compression_usd,
|
|
"tool_schema": tool_usd,
|
|
"output_shaping": 0.0,
|
|
"provider_cache": 0.0,
|
|
},
|
|
timestamp=timestamp or _recent(),
|
|
)
|
|
|
|
|
|
def test_lifetime_and_session_record_tool_schema_disjointly(tmp_path):
|
|
tracker = _tracker(tmp_path)
|
|
_record(tracker)
|
|
|
|
snapshot = tracker.snapshot()
|
|
for block in (snapshot["lifetime"], snapshot["display_session"]):
|
|
# The folded headline is unchanged: compression + tool_schema.
|
|
assert block["compression_savings_usd"] == 6.0
|
|
# The disjoint record makes message-only dollars recoverable.
|
|
assert block["tool_schema_savings_usd"] == 5.0
|
|
assert block["compression_savings_usd"] - block["tool_schema_savings_usd"] == 1.0
|
|
assert block["tool_tokens_saved"] == 500
|
|
|
|
|
|
def test_checkpoints_carry_the_layer_and_survive_reload(tmp_path):
|
|
tracker = _tracker(tmp_path)
|
|
_record(tracker)
|
|
tracker.flush()
|
|
|
|
point = tracker.snapshot()["history"][-1]
|
|
assert point["tool_tokens_saved"] == 500
|
|
assert point["tool_schema_savings_usd"] == 5.0
|
|
# Checkpoint tokens stay message-only; the disjoint dollar field is what
|
|
# lets a consumer pair a consistent numerator with that denominator.
|
|
assert point["total_tokens_saved"] == 100
|
|
|
|
# A fresh tracker on the same file must not zero the layer (the output
|
|
# fields historically reset on every restart; these must not).
|
|
reloaded = SavingsTracker(
|
|
path=str(tmp_path / "proxy_savings.json"),
|
|
max_history_points=100,
|
|
max_history_age_days=30,
|
|
)
|
|
lifetime = reloaded.snapshot()["lifetime"]
|
|
assert lifetime["tool_tokens_saved"] == 500
|
|
assert lifetime["tool_schema_savings_usd"] == 5.0
|
|
|
|
|
|
def test_state_written_before_the_fields_existed_defaults_to_zero(tmp_path):
|
|
path = tmp_path / "proxy_savings.json"
|
|
path.write_text(
|
|
json.dumps(
|
|
{
|
|
"schema_version": 5,
|
|
"lifetime": {
|
|
"requests": 3,
|
|
"tokens_saved": 900,
|
|
"compression_savings_usd": 0.9,
|
|
"cache_read_tokens": 0,
|
|
"cache_savings_usd": 0.0,
|
|
"total_input_tokens": 5_000,
|
|
"total_input_cost_usd": 0.05,
|
|
},
|
|
"display_session": {},
|
|
"history": [
|
|
{
|
|
"timestamp": "2026-08-20T10:00:00Z",
|
|
"total_tokens_saved": 900,
|
|
"compression_savings_usd": 0.9,
|
|
"total_input_tokens": 5_000,
|
|
"total_input_cost_usd": 0.05,
|
|
}
|
|
],
|
|
"projects": {},
|
|
"by_model": {},
|
|
}
|
|
)
|
|
)
|
|
|
|
tracker = SavingsTracker(path=str(path), max_history_points=100, max_history_age_days=30)
|
|
lifetime = tracker.snapshot()["lifetime"]
|
|
assert lifetime["tool_tokens_saved"] == 0
|
|
assert lifetime["tool_schema_savings_usd"] == 0.0
|
|
assert lifetime["tokens_saved"] == 900
|
|
|
|
# Legacy tuple-shaped history entries normalize the same way.
|
|
normalized = _normalize_history_entry(["2026-08-20T10:00:00Z", 900, 0.9])
|
|
assert normalized is not None
|
|
assert normalized["tool_tokens_saved"] == 0
|
|
assert normalized["tool_schema_savings_usd"] == 0.0
|
|
|
|
|
|
def test_tool_only_request_appends_a_checkpoint(tmp_path):
|
|
tracker = _tracker(tmp_path)
|
|
# Deferral with zero message compression: real on tool-heavy traffic, and
|
|
# previously invisible to the history because the append gate only looked
|
|
# at message/cache/output deltas.
|
|
_record(tracker, tokens_saved=0, compression_usd=0.0)
|
|
|
|
history = tracker.snapshot()["history"]
|
|
assert len(history) == 1
|
|
assert history[-1]["tool_tokens_saved"] == 500
|
|
assert history[-1]["tool_schema_savings_usd"] == 5.0
|
|
assert history[-1]["total_tokens_saved"] == 0
|
|
|
|
|
|
def test_rollups_expose_tool_schema_deltas(tmp_path):
|
|
tracker = _tracker(tmp_path)
|
|
base = _hour_base()
|
|
_record(tracker, timestamp=_iso(base + timedelta(minutes=10)))
|
|
_record(
|
|
tracker,
|
|
tool_search_saved=250,
|
|
tool_usd=2.5,
|
|
timestamp=_iso(base + timedelta(minutes=40)),
|
|
)
|
|
_record(
|
|
tracker,
|
|
tool_search_saved=1_000,
|
|
tool_usd=10.0,
|
|
timestamp=_iso(base + timedelta(minutes=65)),
|
|
)
|
|
|
|
hourly = tracker.history_response()["series"]["hourly"]
|
|
assert [point["timestamp"] for point in hourly] == [
|
|
_iso(base),
|
|
_iso(base + timedelta(hours=1)),
|
|
]
|
|
|
|
first, second = hourly
|
|
# Deltas start from a zero baseline, so the series' first checkpoint
|
|
# contributes its full cumulative -- the same convention as every other
|
|
# rollup delta in this module. Bucket one therefore carries 500 + 250.
|
|
assert first["tool_tokens_saved_delta"] == 750
|
|
assert first["tool_schema_savings_usd_delta"] == 7.5
|
|
assert first["tool_tokens_saved"] == 750
|
|
assert second["tool_tokens_saved_delta"] == 1_000
|
|
assert second["tool_schema_savings_usd_delta"] == 10.0
|
|
assert second["tool_tokens_saved"] == 1_750
|
|
|
|
# The subtraction identity the split exists for: folded minus disjoint
|
|
# equals message-only dollars, bucket by bucket (two $1 records, then one).
|
|
assert first["compression_savings_usd_delta"] - first[
|
|
"tool_schema_savings_usd_delta"
|
|
] == pytest.approx(2.0)
|
|
assert second["compression_savings_usd_delta"] - second[
|
|
"tool_schema_savings_usd_delta"
|
|
] == pytest.approx(1.0)
|
|
|
|
|
|
def test_rollup_csv_exports_the_delta_columns(tmp_path):
|
|
tracker = _tracker(tmp_path)
|
|
base = _hour_base()
|
|
_record(tracker, timestamp=_iso(base + timedelta(minutes=10)))
|
|
_record(tracker, timestamp=_iso(base + timedelta(minutes=40)))
|
|
|
|
header = tracker.export_csv(series="hourly").splitlines()[0]
|
|
assert "tool_tokens_saved_delta" in header
|
|
assert "tool_schema_savings_usd_delta" in header
|