Files
headroom/tests/test_tool_schema_savings_split.py
gglucassandClaude Fable 5 c719d4af55 fix(savings): record tool-schema dollars disjointly beside the folded headline (#3170)
## Description

Since the attribution unification (#2976, refined by #3123),
`record_request` folds the tool-schema layer's dollars into
`compression_savings_usd` — on the lifetime block, the display session,
and every history checkpoint — while `total_tokens_saved` on those same
checkpoints stays message-only. The two halves of one checkpoint now
disagree about what "saved" means, and any consumer that derives a
$/token rate from a checkpoint reads a figure inflated by `1 +
tool/message`.

That inflation is not hypothetical. On one real install, message
compression was 3,299,618 tokens next to 18,435,490 tokens of
tool-schema deferral (a 5.59x ratio — far from the 0.24x in the sample
#2976 cites, this ratio varies wildly by workload). The blended savings
rate a downstream consumer derived went from $4.99/M to $32.88/M — above
the input list price of every model in the mix, a rate no genuine
input-token saving can reach.

There is also a smaller correctness gap: a request whose only saving is
deferral (`tokens_saved == 0`, common on tool-heavy turns) appended no
checkpoint at all, so its folded dollars landed in whichever later
bucket happened to checkpoint next.

#3155 fixed the token half of this per model, and stated the principle
this PR extends to dollars: kept as its own bucket rather than widened
in place, so the persisted meaning of an existing field never changes
under a reader that predates it. This PR is deliberately **additive** —
`compression_savings_usd` keeps its current folded semantics everywhere,
so the dashboard, PERF headline, and every existing reader are
unchanged. The layer is simply also recorded disjointly beside it, so
message-only dollars are recoverable by subtraction and the checkpoint's
token/dollar pair can be read consistently.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Record `tool_tokens_saved` / `tool_schema_savings_usd` on the lifetime
block and display session, wired through `_default_state`,
`_sanitize_state` (including the history-recovery max), and both
normalizers — the fields survive restarts rather than resetting the way
the ad-hoc output fields do.
- Carry both fields on every history checkpoint, and extend the
checkpoint append gate so a deferral-only request checkpoints too.
- Expose `tool_tokens_saved_delta` / `tool_schema_savings_usd_delta`
(plus cumulatives) on rollup buckets, following the exact delta
convention of the existing series, and add the delta columns to the
rollup CSV export.
- State files written before these fields existed load with both at 0;
legacy tuple-shaped history entries normalize the same way.
- New test module `tests/test_tool_schema_savings_split.py` (6 tests)
pinning the disjoint recording, reload durability, legacy-state
defaults, the deferral-only checkpoint, the rollup deltas, and the
subtraction identity; existing shape assertions in
`tests/test_proxy_savings_history.py` updated for the two new fields.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_proxy_savings_history.py \
    tests/test_tool_schema_savings_split.py tests/test_per_model_tool_savings.py \
    tests/test_savings_tool_search_aggregation.py tests/test_savings_tracker_zero_price.py \
    tests/test_agent_savings.py
======================== 103 passed, 1 warning in 7.36s ========================

$ uv run --frozen --extra dev pytest tests/test_stats_new_input_savings_rate.py \
    tests/test_dashboard_token_savings.py tests/test_proxy_project_savings.py \
    tests/test_output_savings.py tests/test_savings_ledger.py
=================== 76 passed, 2 skipped, 1 warning in 8.15s ===================

$ uv run --frozen ruff check headroom/proxy/savings_tracker.py tests/test_tool_schema_savings_split.py tests/test_proxy_savings_history.py
All checks passed!
$ uv run --frozen ruff format --check headroom/proxy/savings_tracker.py tests/test_tool_schema_savings_split.py tests/test_proxy_savings_history.py
3 files already formatted
$ uv run --frozen mypy headroom/proxy/savings_tracker.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS, Python 3.10 venv via `uv run --frozen`, this
branch.
- Exact command / steps: one `record_request` shaped like the reporting
install's traffic (100 message tokens saved, 559 deferred schema tokens,
claude-opus-5 pricing at $5/M input), then derive $/token from the
appended checkpoint both ways.
- Observed result: the naive checkpoint-derived rate reproduces the
impossible $32.88/M seen in the field almost exactly, and subtracting
the new disjoint field recovers the model's actual input rate:

```text
checkpoint: total_tokens_saved=100 compression_savings_usd=0.003295 tool_tokens_saved=559 tool_schema_savings_usd=0.002795
folded-dollars / message-tokens rate: $32.95/M  (input list price is $5/M)
after subtracting the disjoint field:  $5.00/M
```

- Not tested: a long-lived multi-process proxy soak, and the downstream
consumer's subtraction path itself (that lands with the consumer release
that reads the new fields).

## Runtime Rollout Safety

- Rollout-managed feature(s): None -- persisted savings accounting only;
no rollout-gated runtime behavior is touched.
- Minimum rollout channel: N/A
- Stable/default behavior changed: No. Additive fields beside existing
ones; `compression_savings_usd` keeps its current folded semantics, so
every existing reader sees identical values.
- Kill switch / disable path: None needed -- old readers ignore the new
fields, and state files written before them load with both at 0 (covered
by tests).
- Unsafe override required: No
- Qualification impact: None beyond the unit suites run above; displayed
figures are unchanged.
- Rollback path: `git revert` -- the fields stop being written and
remain optional on read, so state files produced while this was live
stay loadable.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

Related: #3155 (token half of the same split), #3171 (companion PR on
what the tool-schema bucket should be priced at — independent; either
lands without the other).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-30 11:27:41 -05:00

239 lines
8.8 KiB
Python

"""Tool-schema dollars must be recorded disjointly, not only folded.
Reported by a desktop consumer of the persisted savings state. Since the
attribution unification, ``record_request`` folds the tool-schema layer's
dollars into ``compression_savings_usd`` (lifetime, display_session, and every
history checkpoint) while ``total_tokens_saved`` on the same checkpoint stays
message-only. Any consumer deriving a $/token rate from a checkpoint therefore
reads a figure inflated by ``1 + tool/message``:
- Traffic: message compression 3,299,618 tokens next to 18,435,490 tokens of
tool-schema deferral (ratio 5.59x) on one real install.
- Implied rate: a $4.99/M blended savings rate became $32.88/M after the fold,
which exceeds the input list price of every model in the mix.
- Reader impact: consumers that accumulated ``compression_savings_usd`` before
the fold shipped had the field's meaning widened underneath them.
Same resolution shape as the per-model token fix: keep the folded headline
exactly as it is, and record the layer disjointly beside it --
``tool_tokens_saved`` / ``tool_schema_savings_usd`` on lifetime,
display_session, and checkpoints, with matching ``_delta`` fields on rollup
buckets -- so message-only dollars are recoverable by subtraction.
"""
from __future__ import annotations
import json
from datetime import datetime, timedelta, timezone
import pytest
from headroom.proxy.savings_tracker import SavingsTracker, _normalize_history_entry
MODEL = "claude-opus-5"
def _iso(moment: datetime) -> str:
return moment.strftime("%Y-%m-%dT%H:%M:%SZ")
def _recent() -> str:
"""A timestamp inside the display-session window, anchored to wall clock.
``_display_session_snapshot_locked`` expires the session against
``_utc_now()``, not against the recorded timestamp, so a frozen literal
silently stops populating ``display_session`` once it ages past
``DEFAULT_DISPLAY_SESSION_INACTIVITY_MINUTES``. This suite was written with
a hardcoded date and went red three weeks later for exactly that reason.
"""
return _iso(datetime.now(timezone.utc) - timedelta(minutes=1))
def _hour_base() -> datetime:
"""A whole hour safely in the past, so derived buckets are stable and never future-dated."""
now = datetime.now(timezone.utc).replace(minute=0, second=0, microsecond=0)
return now - timedelta(hours=3)
def _tracker(tmp_path) -> SavingsTracker:
return SavingsTracker(
path=str(tmp_path / "proxy_savings.json"),
max_history_points=100,
max_history_age_days=30,
)
def _record(
tracker: SavingsTracker,
*,
tokens_saved: int = 100,
tool_search_saved: int = 500,
compression_usd: float = 1.0,
tool_usd: float = 5.0,
timestamp: str | None = None,
) -> None:
tracker.record_request(
model=MODEL,
input_tokens=1_000,
tokens_saved=tokens_saved,
tool_search_saved=tool_search_saved,
estimated_savings_usd={
"compression": compression_usd,
"tool_schema": tool_usd,
"output_shaping": 0.0,
"provider_cache": 0.0,
},
timestamp=timestamp or _recent(),
)
def test_lifetime_and_session_record_tool_schema_disjointly(tmp_path):
tracker = _tracker(tmp_path)
_record(tracker)
snapshot = tracker.snapshot()
for block in (snapshot["lifetime"], snapshot["display_session"]):
# The folded headline is unchanged: compression + tool_schema.
assert block["compression_savings_usd"] == 6.0
# The disjoint record makes message-only dollars recoverable.
assert block["tool_schema_savings_usd"] == 5.0
assert block["compression_savings_usd"] - block["tool_schema_savings_usd"] == 1.0
assert block["tool_tokens_saved"] == 500
def test_checkpoints_carry_the_layer_and_survive_reload(tmp_path):
tracker = _tracker(tmp_path)
_record(tracker)
tracker.flush()
point = tracker.snapshot()["history"][-1]
assert point["tool_tokens_saved"] == 500
assert point["tool_schema_savings_usd"] == 5.0
# Checkpoint tokens stay message-only; the disjoint dollar field is what
# lets a consumer pair a consistent numerator with that denominator.
assert point["total_tokens_saved"] == 100
# A fresh tracker on the same file must not zero the layer (the output
# fields historically reset on every restart; these must not).
reloaded = SavingsTracker(
path=str(tmp_path / "proxy_savings.json"),
max_history_points=100,
max_history_age_days=30,
)
lifetime = reloaded.snapshot()["lifetime"]
assert lifetime["tool_tokens_saved"] == 500
assert lifetime["tool_schema_savings_usd"] == 5.0
def test_state_written_before_the_fields_existed_defaults_to_zero(tmp_path):
path = tmp_path / "proxy_savings.json"
path.write_text(
json.dumps(
{
"schema_version": 5,
"lifetime": {
"requests": 3,
"tokens_saved": 900,
"compression_savings_usd": 0.9,
"cache_read_tokens": 0,
"cache_savings_usd": 0.0,
"total_input_tokens": 5_000,
"total_input_cost_usd": 0.05,
},
"display_session": {},
"history": [
{
"timestamp": "2026-08-20T10:00:00Z",
"total_tokens_saved": 900,
"compression_savings_usd": 0.9,
"total_input_tokens": 5_000,
"total_input_cost_usd": 0.05,
}
],
"projects": {},
"by_model": {},
}
)
)
tracker = SavingsTracker(path=str(path), max_history_points=100, max_history_age_days=30)
lifetime = tracker.snapshot()["lifetime"]
assert lifetime["tool_tokens_saved"] == 0
assert lifetime["tool_schema_savings_usd"] == 0.0
assert lifetime["tokens_saved"] == 900
# Legacy tuple-shaped history entries normalize the same way.
normalized = _normalize_history_entry(["2026-08-20T10:00:00Z", 900, 0.9])
assert normalized is not None
assert normalized["tool_tokens_saved"] == 0
assert normalized["tool_schema_savings_usd"] == 0.0
def test_tool_only_request_appends_a_checkpoint(tmp_path):
tracker = _tracker(tmp_path)
# Deferral with zero message compression: real on tool-heavy traffic, and
# previously invisible to the history because the append gate only looked
# at message/cache/output deltas.
_record(tracker, tokens_saved=0, compression_usd=0.0)
history = tracker.snapshot()["history"]
assert len(history) == 1
assert history[-1]["tool_tokens_saved"] == 500
assert history[-1]["tool_schema_savings_usd"] == 5.0
assert history[-1]["total_tokens_saved"] == 0
def test_rollups_expose_tool_schema_deltas(tmp_path):
tracker = _tracker(tmp_path)
base = _hour_base()
_record(tracker, timestamp=_iso(base + timedelta(minutes=10)))
_record(
tracker,
tool_search_saved=250,
tool_usd=2.5,
timestamp=_iso(base + timedelta(minutes=40)),
)
_record(
tracker,
tool_search_saved=1_000,
tool_usd=10.0,
timestamp=_iso(base + timedelta(minutes=65)),
)
hourly = tracker.history_response()["series"]["hourly"]
assert [point["timestamp"] for point in hourly] == [
_iso(base),
_iso(base + timedelta(hours=1)),
]
first, second = hourly
# Deltas start from a zero baseline, so the series' first checkpoint
# contributes its full cumulative -- the same convention as every other
# rollup delta in this module. Bucket one therefore carries 500 + 250.
assert first["tool_tokens_saved_delta"] == 750
assert first["tool_schema_savings_usd_delta"] == 7.5
assert first["tool_tokens_saved"] == 750
assert second["tool_tokens_saved_delta"] == 1_000
assert second["tool_schema_savings_usd_delta"] == 10.0
assert second["tool_tokens_saved"] == 1_750
# The subtraction identity the split exists for: folded minus disjoint
# equals message-only dollars, bucket by bucket (two $1 records, then one).
assert first["compression_savings_usd_delta"] - first[
"tool_schema_savings_usd_delta"
] == pytest.approx(2.0)
assert second["compression_savings_usd_delta"] - second[
"tool_schema_savings_usd_delta"
] == pytest.approx(1.0)
def test_rollup_csv_exports_the_delta_columns(tmp_path):
tracker = _tracker(tmp_path)
base = _hour_base()
_record(tracker, timestamp=_iso(base + timedelta(minutes=10)))
_record(tracker, timestamp=_iso(base + timedelta(minutes=40)))
header = tracker.export_csv(series="hourly").splitlines()[0]
assert "tool_tokens_saved_delta" in header
assert "tool_schema_savings_usd_delta" in header