mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
feat(savings): track lifetime output spend so a bill-share rate has a denominator (#3520)
## Description
`SavingsTracker` has accumulated `total_input_cost_usd` since it
existed, and
`output_savings_usd` was added later for the shaper (what it stopped the
model
emitting). There is no counterpart for what the model DID emit, so the
lifetime
block describes the whole input bill and none of the output bill.
That gap has a consequence beyond a missing field. Any figure answering
"what
share of my bill did Headroom remove" has to put compression, output
shaping
and tool-schema deferral in the numerator, but the only denominator
available
is input spend. Output-shaping dollars then sit over an input-only
denominator
and the rate reads high. I hit this concretely in a downstream
dashboard, where
an all-time rate built that way read 17.0% next to 11.6% for the same
month on
the same data, with 2.2pp of the gap being nothing but the extra layers.
This adds the missing denominator half. No existing number changes.
- `lifetime["total_output_cost_usd"]`: emitted output tokens priced at
the
model's output rate.
- Rollup series points gain `total_output_cost_usd_delta` and the
cumulative
`total_output_cost_usd`, matching how `total_input_cost_usd` already
appears,
so per-bucket rates are computable and not just an all-time one.
- Priced by `_estimate_output_cost_usd`, a named passthrough to
`_estimate_output_savings_usd`. Same rate table, same zero-price
handling,
same fallback, so output spend and output saving can never disagree
about a
model's price. Split out so the call sites read as spend and saving
rather
than one function doing double duty under a "savings" name.
- Fed by `output_tokens`, which `PrometheusMetrics.record_request`
already
accepts and simply never passed down.
## Type of Change
- [x] New feature (non-breaking change that adds functionality)
## Changes Made
- `headroom/proxy/savings_tracker.py`: `_estimate_output_cost_usd`;
`record_request(output_tokens=...)`; `lifetime["total_output_cost_usd"]`
accumulated and checkpointed into history; `_normalize_history_entry`
defaults it for legacy entries; `_build_rollup` carries the cumulative
and
the per-bucket delta; CSV export gains `total_output_cost_usd_delta`.
- `headroom/proxy/prometheus_metrics.py`: pass `output_tokens` through
to the
tracker.
- `tests/test_savings_tracker_output_cost.py`: new.
- `tests/test_proxy_savings_history.py`: three exact-shape assertions
gain the
new key (normalized entry, persisted checkpoint, CSV header).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
uv run --frozen --extra dev pytest tests/test_proxy_savings_history.py \
tests/test_savings_tracker_zero_price.py tests/test_cost_tracker_totals.py \
tests/test_proxy_project_savings.py tests/test_per_model_tool_savings.py \
tests/test_savings_ledger.py tests/test_output_savings.py \
tests/test_output_shaping_rollup.py tests/test_savings_tracker_output_cost.py \
tests/test_savings_tracker_litellm_resolution_cache.py -q
174 passed, 2 skipped, 1 warning in 31.98s
uvx ruff check headroom/proxy/savings_tracker.py headroom/proxy/prometheus_metrics.py \
tests/test_savings_tracker_output_cost.py tests/test_proxy_savings_history.py
All checks passed!
uvx ruff format --check <same four files>
4 files already formatted
uv run --frozen --extra dev mypy headroom/proxy/savings_tracker.py headroom/proxy/prometheus_metrics.py
Success: no issues found in 2 source files
```
## Real Behavior Proof
- Environment: macOS 15, Python 3.11, this branch off `main` at
4e1f7769. Live
figures pulled from a running headroom-ai 0.37.0 proxy on port 6767
carrying
real Claude Code traffic, 2026-09-10.
- Exact command / steps: confirmed the running proxy's `/stats`
`persistent_savings.lifetime` block has `total_input_cost_usd` and no
output
counterpart (`"total_output_cost_usd" in lifetime` -> `False`). Then
drove
the patched `SavingsTracker` with that same session's real shape:
`record_request(model="claude-opus-5", input_tokens=12_809_792,
tokens_saved=1_609_216, output_tokens=70_442)`.
- Observed result: input 12,809,792 tok -> `total_input_cost_usd`
$64.05;
output 70,442 tok -> `total_output_cost_usd` $1.76, the number that did
not
exist before. The bill-share rate on that session moves from 11.16% with
an
input-only denominator to 10.89% with input + output. The daily series
point
gained `total_output_cost_usd` and `total_output_cost_usd_delta` beside
the
existing `output_savings_usd_delta`.
- Not tested: the overstatement's size is workload-dependent and small
here
because this session is input-heavy (output is 0.5% of tokens, though at
~5x
the input rate). An output-heavy workload separates the two rates much
further. I also did not add a `/stats` field beyond what the lifetime
block
already passes through verbatim.
## Runtime Rollout Safety
- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: No. Every existing field keeps its
value;
this only adds keys.
- Kill switch / disable path: None needed, nothing reads the field yet.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert. Persisted state with the extra key is read by
the
previous version through `_normalize_history_entry`, which ignores
unknown
keys.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Screenshots (if applicable)
N/A
## Additional Notes
One known limitation, called out because a consumer will trip on it
otherwise:
the field accumulates from the first request after this ships, while
`total_input_cost_usd` carries the machine's full history. A combined
rate
computed on the day of the bump divides a full-history numerator by a
partial-history output denominator. That is the same ramp-up
`output_savings_usd`
had when it landed, so I followed the precedent rather than inventing a
backfill, but it means consumers should gate on `total_output_cost_usd >
0` and
treat a 0 as "no coverage" rather than "no output spend". Happy to add
an
explicit first-recorded-at marker instead if you would rather it be
self-describing.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5
JD Davis
parent
c32b315537
commit
c766bdb2b3
@@ -1030,6 +1030,7 @@ class PrometheusMetrics:
|
||||
total_input_tokens=total_input_tokens,
|
||||
total_input_cost_usd=total_input_cost_usd,
|
||||
output_tokens_saved=output_tokens_saved,
|
||||
output_tokens=output_tokens,
|
||||
estimated_savings_usd=savings_usd,
|
||||
)
|
||||
|
||||
|
||||
@@ -389,6 +389,17 @@ def _estimate_output_savings_usd(model: str, tokens_saved: int) -> float:
|
||||
return float(tokens_saved) * float(DEFAULT_FALLBACK_OUTPUT_COST_PER_TOKEN)
|
||||
|
||||
|
||||
def _estimate_output_cost_usd(model: str, output_tokens: int) -> float:
|
||||
"""Estimate what EMITTED output tokens cost, at the model's output rate.
|
||||
|
||||
Same rate table as ``_estimate_output_savings_usd``, which prices the
|
||||
tokens the shaper kept the model from emitting; this prices the ones it
|
||||
did emit. Split out so the call sites read as spend and saving rather than
|
||||
one function doing double duty.
|
||||
"""
|
||||
return _estimate_output_savings_usd(model, output_tokens)
|
||||
|
||||
|
||||
def _estimate_cache_savings_usd(model: str, cache_read_tokens: int) -> float:
|
||||
"""Estimate cache-read savings in USD — the discount delta vs list price.
|
||||
|
||||
@@ -601,6 +612,7 @@ def _normalize_history_entry(entry: Any) -> dict[str, Any] | None:
|
||||
total_input_cost_usd = 0.0
|
||||
output_tokens_saved = 0
|
||||
output_savings_usd = 0.0
|
||||
total_output_cost_usd = 0.0
|
||||
provider = PROVIDER_UNKNOWN
|
||||
model = MODEL_UNKNOWN
|
||||
|
||||
@@ -617,6 +629,7 @@ def _normalize_history_entry(entry: Any) -> dict[str, Any] | None:
|
||||
total_input_cost_usd = _coerce_float(entry.get("total_input_cost_usd"))
|
||||
output_tokens_saved = _coerce_int(entry.get("output_tokens_saved"))
|
||||
output_savings_usd = _coerce_float(entry.get("output_savings_usd"))
|
||||
total_output_cost_usd = _coerce_float(entry.get("total_output_cost_usd"))
|
||||
provider = _normalize_provider(entry.get("provider"))
|
||||
model = _normalize_model(entry.get("model"))
|
||||
elif isinstance(entry, list | tuple) and len(entry) >= 2:
|
||||
@@ -646,6 +659,7 @@ def _normalize_history_entry(entry: Any) -> dict[str, Any] | None:
|
||||
"total_input_cost_usd": round(total_input_cost_usd, 6),
|
||||
"output_tokens_saved": output_tokens_saved,
|
||||
"output_savings_usd": round(output_savings_usd, 6),
|
||||
"total_output_cost_usd": round(total_output_cost_usd, 6),
|
||||
}
|
||||
|
||||
|
||||
@@ -988,6 +1002,10 @@ class SavingsTracker:
|
||||
# message-only while per-model dollars did not.
|
||||
tool_search_saved: int = 0,
|
||||
output_tokens_saved: int = 0,
|
||||
# Tokens the model actually emitted, DISJOINT from
|
||||
# ``output_tokens_saved`` (which counts the ones it did not). Priced
|
||||
# into ``lifetime["total_output_cost_usd"]``; see the delta below.
|
||||
output_tokens: int = 0,
|
||||
provider: str | None = None,
|
||||
project: str | None = None,
|
||||
cache_read_tokens: int = 0,
|
||||
@@ -1079,6 +1097,16 @@ class SavingsTracker:
|
||||
if priced is not None
|
||||
else _estimate_output_savings_usd(model, delta_output_tokens_saved)
|
||||
)
|
||||
# Completion spend. ``total_input_cost_usd`` has never had an output
|
||||
# counterpart, so anything answering "what share of my bill did
|
||||
# Headroom remove" had to divide savings that include output shaping by
|
||||
# an input-only denominator, which overstates the rate. Estimated from
|
||||
# the same rate table as the output SAVINGS beside it, so the two are
|
||||
# consistent even when litellm has no price for the model.
|
||||
delta_output_cost_usd = _estimate_output_cost_usd(
|
||||
model,
|
||||
max(_coerce_int(output_tokens), 0),
|
||||
)
|
||||
delta_cache_savings_usd = (
|
||||
max(_coerce_float(priced.get("provider_cache")), 0.0)
|
||||
if priced is not None
|
||||
@@ -1149,6 +1177,10 @@ class SavingsTracker:
|
||||
lifetime.get("output_savings_usd", 0.0) + delta_output_savings_usd,
|
||||
6,
|
||||
)
|
||||
lifetime["total_output_cost_usd"] = round(
|
||||
lifetime.get("total_output_cost_usd", 0.0) + delta_output_cost_usd,
|
||||
6,
|
||||
)
|
||||
|
||||
session = self._state["display_session"]
|
||||
last_activity = _parse_timestamp(session.get("last_activity_at"))
|
||||
@@ -1235,6 +1267,7 @@ class SavingsTracker:
|
||||
"total_input_cost_usd": lifetime["total_input_cost_usd"],
|
||||
"output_tokens_saved": lifetime.get("output_tokens_saved", 0),
|
||||
"output_savings_usd": lifetime.get("output_savings_usd", 0.0),
|
||||
"total_output_cost_usd": lifetime.get("total_output_cost_usd", 0.0),
|
||||
}
|
||||
)
|
||||
self._trim_history_locked(reference_time=timestamp_dt)
|
||||
@@ -1565,6 +1598,7 @@ class SavingsTracker:
|
||||
"total_input_cost_usd",
|
||||
"output_tokens_saved_delta",
|
||||
"output_savings_usd_delta",
|
||||
"total_output_cost_usd_delta",
|
||||
]
|
||||
|
||||
buffer = StringIO()
|
||||
@@ -1608,6 +1642,9 @@ class SavingsTracker:
|
||||
"cache_savings_usd": 0.0,
|
||||
"total_input_tokens": 0,
|
||||
"total_input_cost_usd": 0.0,
|
||||
"output_tokens_saved": 0,
|
||||
"output_savings_usd": 0.0,
|
||||
"total_output_cost_usd": 0.0,
|
||||
},
|
||||
"display_session": _empty_display_session(),
|
||||
"history": [],
|
||||
@@ -1674,6 +1711,9 @@ class SavingsTracker:
|
||||
lifetime_basis = BASIS_UNKNOWN
|
||||
migrated_at = None
|
||||
is_v6_lifetime = False
|
||||
lifetime_output_tokens_saved = 0
|
||||
lifetime_output_savings_usd = 0.0
|
||||
lifetime_output_cost_usd = 0.0
|
||||
if isinstance(lifetime_raw, dict):
|
||||
lifetime_requests = _coerce_int(lifetime_raw.get("requests"))
|
||||
lifetime_tokens_saved = _coerce_int(lifetime_raw.get("tokens_saved"))
|
||||
@@ -1689,6 +1729,9 @@ class SavingsTracker:
|
||||
lifetime_raw.get("compression_savings_list_usd")
|
||||
)
|
||||
lifetime_basis = str(lifetime_raw.get("savings_basis") or BASIS_UNKNOWN)
|
||||
lifetime_output_tokens_saved = _coerce_int(lifetime_raw.get("output_tokens_saved"))
|
||||
lifetime_output_savings_usd = _coerce_float(lifetime_raw.get("output_savings_usd"))
|
||||
lifetime_output_cost_usd = _coerce_float(lifetime_raw.get("total_output_cost_usd"))
|
||||
|
||||
if normalized_history:
|
||||
last = normalized_history[-1]
|
||||
@@ -1708,6 +1751,22 @@ class SavingsTracker:
|
||||
lifetime_input_cost_usd,
|
||||
_coerce_float(last.get("total_input_cost_usd")),
|
||||
)
|
||||
# The output side accumulates the same way and is checkpointed the
|
||||
# same way, so it recovers the same way. Without this a restart
|
||||
# restarted the counters at 0 while history kept the old totals,
|
||||
# and the rollup's max(delta, 0) then swallowed the difference.
|
||||
lifetime_output_tokens_saved = max(
|
||||
lifetime_output_tokens_saved,
|
||||
_coerce_int(last.get("output_tokens_saved")),
|
||||
)
|
||||
lifetime_output_savings_usd = max(
|
||||
lifetime_output_savings_usd,
|
||||
_coerce_float(last.get("output_savings_usd")),
|
||||
)
|
||||
lifetime_output_cost_usd = max(
|
||||
lifetime_output_cost_usd,
|
||||
_coerce_float(last.get("total_output_cost_usd")),
|
||||
)
|
||||
|
||||
# v6 migration seed. Runs HERE, after the history back-fill above, not
|
||||
# beside the lifetime read: a state whose `lifetime` block is missing or
|
||||
@@ -1742,6 +1801,9 @@ class SavingsTracker:
|
||||
"cache_savings_usd": round(lifetime_cache_savings_usd, 6),
|
||||
"total_input_tokens": lifetime_input_tokens,
|
||||
"total_input_cost_usd": round(lifetime_input_cost_usd, 6),
|
||||
"output_tokens_saved": lifetime_output_tokens_saved,
|
||||
"output_savings_usd": round(lifetime_output_savings_usd, 6),
|
||||
"total_output_cost_usd": round(lifetime_output_cost_usd, 6),
|
||||
},
|
||||
"display_session": _normalize_display_session(raw.get("display_session")),
|
||||
"history": normalized_history,
|
||||
@@ -2021,6 +2083,7 @@ class SavingsTracker:
|
||||
prev_total_input_cost_usd = 0.0
|
||||
prev_output_tokens = 0
|
||||
prev_output_usd = 0.0
|
||||
prev_output_cost_usd = 0.0
|
||||
prev_cache_read_tokens = 0
|
||||
prev_cache_savings_usd = 0.0
|
||||
# What the bucket's cache reads actually COST, priced per checkpoint
|
||||
@@ -2075,6 +2138,7 @@ class SavingsTracker:
|
||||
total_input_cost_usd = _coerce_float(point.get("total_input_cost_usd"))
|
||||
total_output_tokens = _coerce_int(point.get("output_tokens_saved"))
|
||||
total_output_usd = _coerce_float(point.get("output_savings_usd"))
|
||||
total_output_cost_usd = _coerce_float(point.get("total_output_cost_usd"))
|
||||
delta_tokens = max(total_tokens_saved - prev_total_tokens, 0)
|
||||
delta_usd = max(total_usd - prev_total_usd, 0.0)
|
||||
delta_input_tokens = max(total_input_tokens - prev_total_input_tokens, 0)
|
||||
@@ -2085,6 +2149,7 @@ class SavingsTracker:
|
||||
|
||||
delta_output_tokens = max(total_output_tokens - prev_output_tokens, 0)
|
||||
delta_output_usd = max(total_output_usd - prev_output_usd, 0.0)
|
||||
delta_output_cost_usd = max(total_output_cost_usd - prev_output_cost_usd, 0.0)
|
||||
|
||||
total_cache_read_tokens = _coerce_int(point.get("cache_read_tokens"))
|
||||
total_cache_savings_usd = _coerce_float(point.get("cache_savings_usd"))
|
||||
@@ -2101,6 +2166,7 @@ class SavingsTracker:
|
||||
prev_total_input_cost_usd = total_input_cost_usd
|
||||
prev_output_tokens = total_output_tokens
|
||||
prev_output_usd = total_output_usd
|
||||
prev_output_cost_usd = total_output_cost_usd
|
||||
|
||||
entry = aggregated.setdefault(
|
||||
bucket_key,
|
||||
@@ -2116,6 +2182,8 @@ class SavingsTracker:
|
||||
"total_input_cost_usd": total_input_cost_usd,
|
||||
"output_tokens_saved_delta": 0,
|
||||
"output_savings_usd_delta": 0.0,
|
||||
"total_output_cost_usd_delta": 0.0,
|
||||
"total_output_cost_usd": total_output_cost_usd,
|
||||
**_empty_cache_delta(),
|
||||
"by_provider": {},
|
||||
"by_model": {},
|
||||
@@ -2140,6 +2208,11 @@ class SavingsTracker:
|
||||
entry["output_savings_usd_delta"] + delta_output_usd,
|
||||
6,
|
||||
)
|
||||
entry["total_output_cost_usd_delta"] = round(
|
||||
entry["total_output_cost_usd_delta"] + delta_output_cost_usd,
|
||||
6,
|
||||
)
|
||||
entry["total_output_cost_usd"] = round(total_output_cost_usd, 6)
|
||||
_add_cache(
|
||||
entry, delta_cache_read_tokens, delta_cache_savings_usd, delta_cache_read_cost_usd
|
||||
)
|
||||
|
||||
@@ -84,6 +84,7 @@ def test_savings_tracker_helpers_normalize_inputs_and_paths(tmp_path, monkeypatc
|
||||
"total_input_cost_usd": 0.0,
|
||||
"output_tokens_saved": 0,
|
||||
"output_savings_usd": 0.0,
|
||||
"total_output_cost_usd": 0.0,
|
||||
}
|
||||
assert savings_tracker_module._normalize_history_entry({"timestamp": "bad"}) is None
|
||||
assert savings_tracker_module._normalize_history_entry(object()) is None
|
||||
@@ -142,6 +143,9 @@ def test_savings_tracker_sanitizes_legacy_state_and_applies_retention(tmp_path):
|
||||
"cache_savings_usd": 0.0,
|
||||
"total_input_tokens": 0,
|
||||
"total_input_cost_usd": 0.0,
|
||||
"output_tokens_saved": 0,
|
||||
"output_savings_usd": 0.0,
|
||||
"total_output_cost_usd": 0.0,
|
||||
}
|
||||
assert snapshot["display_session"] == savings_tracker_module._empty_display_session()
|
||||
assert snapshot["history"] == [
|
||||
@@ -157,6 +161,7 @@ def test_savings_tracker_sanitizes_legacy_state_and_applies_retention(tmp_path):
|
||||
"total_input_cost_usd": 0.0,
|
||||
"output_tokens_saved": 0,
|
||||
"output_savings_usd": 0.0,
|
||||
"total_output_cost_usd": 0.0,
|
||||
}
|
||||
]
|
||||
assert snapshot["retention"] == {
|
||||
@@ -185,6 +190,9 @@ def test_non_dict_savings_state_resets_to_default(tmp_path):
|
||||
"cache_savings_usd": 0.0,
|
||||
"total_input_tokens": 0,
|
||||
"total_input_cost_usd": 0.0,
|
||||
"output_tokens_saved": 0,
|
||||
"output_savings_usd": 0.0,
|
||||
"total_output_cost_usd": 0.0,
|
||||
}
|
||||
assert snapshot["display_session"] == savings_tracker_module._empty_display_session()
|
||||
assert snapshot["history"] == []
|
||||
@@ -1424,7 +1432,8 @@ def test_stats_history_csv_export_is_frontend_friendly(tmp_path, monkeypatch):
|
||||
"timestamp,tokens_saved,compression_savings_usd_delta,total_tokens_saved,"
|
||||
"compression_savings_usd,total_input_tokens_delta,total_input_tokens,"
|
||||
"total_input_cost_usd_delta,total_input_cost_usd,"
|
||||
"output_tokens_saved_delta,output_savings_usd_delta"
|
||||
"output_tokens_saved_delta,output_savings_usd_delta,"
|
||||
"total_output_cost_usd_delta"
|
||||
)
|
||||
assert len(lines) >= 2
|
||||
assert "total_tokens_saved" in lines[0]
|
||||
|
||||
@@ -0,0 +1,164 @@
|
||||
"""Lifetime output spend, so a bill-share rate has a denominator that fits it.
|
||||
|
||||
`total_input_cost_usd` has been tracked since the tracker existed; the tokens
|
||||
the model EMITTED were priced nowhere durable. Any "what share of my bill did
|
||||
Headroom remove" figure therefore had to divide savings that include output
|
||||
shaping by an input-only denominator, which overstates the rate. These pin the
|
||||
new `total_output_cost_usd` on the lifetime block and its per-bucket delta on
|
||||
the rollup series, plus the legacy-checkpoint path.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import types
|
||||
|
||||
from headroom.proxy import savings_tracker as st
|
||||
from headroom.proxy.savings_tracker import SavingsTracker, _normalize_history_entry
|
||||
|
||||
|
||||
def _priced_litellm() -> types.SimpleNamespace:
|
||||
return types.SimpleNamespace(
|
||||
model_cost={
|
||||
"test-model": {
|
||||
"input_cost_per_token": 1e-06,
|
||||
"output_cost_per_token": 5e-06,
|
||||
}
|
||||
},
|
||||
cost_per_token=lambda **_kw: (0.0, 0.0),
|
||||
)
|
||||
|
||||
|
||||
def _tracker(tmp_path) -> SavingsTracker:
|
||||
return SavingsTracker(path=tmp_path / "savings.json", save_flush_every=1)
|
||||
|
||||
|
||||
def test_emitted_output_tokens_are_priced_onto_lifetime(monkeypatch, tmp_path):
|
||||
monkeypatch.setattr(st, "_get_litellm_module", _priced_litellm)
|
||||
tracker = _tracker(tmp_path)
|
||||
|
||||
tracker.record_request(
|
||||
model="test-model",
|
||||
input_tokens=1_000,
|
||||
tokens_saved=100,
|
||||
output_tokens=2_000,
|
||||
)
|
||||
|
||||
lifetime = tracker.snapshot()["lifetime"]
|
||||
# 2,000 emitted tokens at $5/M.
|
||||
assert lifetime["total_output_cost_usd"] == 0.01
|
||||
# The input side is untouched by the new field.
|
||||
assert lifetime["total_input_cost_usd"] > 0
|
||||
|
||||
|
||||
def test_output_cost_is_disjoint_from_output_savings(monkeypatch, tmp_path):
|
||||
"""Emitted and not-emitted are separate counts at the same rate."""
|
||||
monkeypatch.setattr(st, "_get_litellm_module", _priced_litellm)
|
||||
tracker = _tracker(tmp_path)
|
||||
|
||||
tracker.record_request(
|
||||
model="test-model",
|
||||
input_tokens=1_000,
|
||||
tokens_saved=0,
|
||||
output_tokens=2_000,
|
||||
output_tokens_saved=400,
|
||||
)
|
||||
|
||||
lifetime = tracker.snapshot()["lifetime"]
|
||||
assert lifetime["total_output_cost_usd"] == 0.01 # 2,000 emitted
|
||||
assert lifetime["output_savings_usd"] == 0.002 # 400 not emitted
|
||||
|
||||
|
||||
def test_bill_share_rate_is_computable_and_below_the_input_only_one(monkeypatch, tmp_path):
|
||||
"""The reason the field exists: an input-only denominator overstates."""
|
||||
monkeypatch.setattr(st, "_get_litellm_module", _priced_litellm)
|
||||
tracker = _tracker(tmp_path)
|
||||
|
||||
tracker.record_request(
|
||||
model="test-model",
|
||||
input_tokens=1_000,
|
||||
tokens_saved=100,
|
||||
output_tokens=2_000,
|
||||
output_tokens_saved=400,
|
||||
)
|
||||
|
||||
lifetime = tracker.snapshot()["lifetime"]
|
||||
saved = lifetime["compression_savings_usd"] + lifetime["output_savings_usd"]
|
||||
input_only = saved / (saved + lifetime["total_input_cost_usd"])
|
||||
whole_bill = saved / (
|
||||
saved + lifetime["total_input_cost_usd"] + lifetime["total_output_cost_usd"]
|
||||
)
|
||||
assert whole_bill < input_only
|
||||
|
||||
|
||||
def test_legacy_checkpoints_default_the_field_rather_than_dropping(tmp_path):
|
||||
normalized = _normalize_history_entry(
|
||||
{
|
||||
"timestamp": "2026-01-01T00:00:00+00:00",
|
||||
"total_tokens_saved": 10,
|
||||
"compression_savings_usd": 0.5,
|
||||
}
|
||||
)
|
||||
assert normalized is not None
|
||||
assert normalized["total_output_cost_usd"] == 0.0
|
||||
|
||||
|
||||
def test_rollup_series_carries_the_per_bucket_output_spend(monkeypatch, tmp_path):
|
||||
monkeypatch.setattr(st, "_get_litellm_module", _priced_litellm)
|
||||
tracker = _tracker(tmp_path)
|
||||
|
||||
for _ in range(3):
|
||||
tracker.record_request(
|
||||
model="test-model",
|
||||
input_tokens=1_000,
|
||||
tokens_saved=100,
|
||||
output_tokens=2_000,
|
||||
)
|
||||
|
||||
daily = tracker.history_response()["series"]["daily"]
|
||||
assert daily, "expected at least one bucket"
|
||||
bucket = daily[-1]
|
||||
# Three requests at $0.01 of emitted output each, as a delta and a total.
|
||||
assert bucket["total_output_cost_usd_delta"] == 0.03
|
||||
assert bucket["total_output_cost_usd"] == 0.03
|
||||
|
||||
|
||||
def test_lifetime_output_spend_survives_a_restart(monkeypatch, tmp_path):
|
||||
"""The field is a LIFETIME total, so a reload must not restart it at 0.
|
||||
|
||||
The lifetime block accumulates in memory and is checkpointed into history;
|
||||
on load the block has to be recovered from both. Without that, request two
|
||||
lands on a zeroed counter and the rollup's ``max(delta, 0)`` clamp hides
|
||||
the regression by reporting a flat total instead of a negative one.
|
||||
"""
|
||||
monkeypatch.setattr(st, "_get_litellm_module", _priced_litellm)
|
||||
path = tmp_path / "savings.json"
|
||||
|
||||
def record(tracker: SavingsTracker) -> None:
|
||||
tracker.record_request(
|
||||
model="test-model",
|
||||
input_tokens=1_000,
|
||||
tokens_saved=100,
|
||||
output_tokens=2_000,
|
||||
output_tokens_saved=400,
|
||||
)
|
||||
|
||||
first = SavingsTracker(path=path, save_flush_every=1)
|
||||
record(first)
|
||||
assert first.snapshot()["lifetime"]["total_output_cost_usd"] == 0.01
|
||||
|
||||
reloaded = SavingsTracker(path=path, save_flush_every=1)
|
||||
restored = reloaded.snapshot()["lifetime"]
|
||||
assert restored["total_output_cost_usd"] == 0.01
|
||||
assert restored["output_savings_usd"] == 0.002
|
||||
assert restored["output_tokens_saved"] == 400
|
||||
|
||||
record(reloaded)
|
||||
lifetime = reloaded.snapshot()["lifetime"]
|
||||
assert lifetime["total_output_cost_usd"] == 0.02
|
||||
assert lifetime["output_savings_usd"] == 0.004
|
||||
assert lifetime["output_tokens_saved"] == 800
|
||||
|
||||
bucket = reloaded.history_response()["series"]["daily"][-1]
|
||||
assert bucket["total_output_cost_usd"] == 0.02
|
||||
assert bucket["total_output_cost_usd_delta"] == 0.02
|
||||
assert bucket["output_savings_usd_delta"] == 0.004
|
||||
Reference in New Issue
Block a user