mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description Part of #4238 (design + PR plan). Step 1 of the savings redesign. This PR adds `headroom/proxy/savings.py`, a pure module that prices one request's removal as **counterfactual input cost − actual input cost**. It is not wired into any surface yet. A follow-up PR makes /stats, the durable ledger, `persistent_savings`, the dashboard and PERF all read this one record, so no surface computes savings its own way. ### Why The test: we ran Headroom Enterprise 0.4.3 in a Daytona sandbox with Codex CLI 0.162.1 on `gpt-5.6-sol` and the real GitHub MCP server, doing workshop task 1. We repeated the run with Claude Code on `claude-sonnet-5`. Each call was audited against the provider's own billing. What we found: | Run | Truth | /stats `cost.savings_usd` | `persistent_savings` | |---|---|---|---| | Codex | $0.0850 saved, 98,188 tokens | $0.0434 (−49%) | — | | Claude Code | $0.1883 saved, 747,493 tokens | $0.0874 (−54%) | $0.2654 (+41%) | The Codex /stats figure books only the conversation-novel tokens: re-sent history is $0 and OpenAI cache writes are ignored. On Claude Code, the /stats figure and `persistent_savings` disagree with each other. Four independent errors feed into this: - PERF `tok_before` is +4% on Codex. - PERF `tok_before` leaves out tool schemas on Claude Code. - Re-sent history is priced at $0 on Responses, but re-booked at the write rate on Anthropic. - `cache_write_tokens` is ignored. Each surface also prices on its own. ### The record Headroom's tokenizer gives per-item counts in Headroom units for the ORIGINAL and the SENT body (tools, system, each input item). The provider gives exact totals. They are only ever combined through ratios: - **Removed tokens:** - `R_old` is removed tokens in items within the longest prefix of the original body that is identical to the previous original body. - `R_new` is removed tokens in every other item. - Both are signed, so an item Headroom grew counts against the saving. - **Shared prefix and bust:** - `P_S` is the sent tokens shared with the previous sent body. - `bust` is old items whose sent copy Headroom changed. - **Ratios:** - `k = B/S` converts Headroom units to provider units. - `warm = min(1, (C/B)/(P_S/S))`. - `w = W/(U+W)`. - **Cost:** - `p_miss = w·p_write + (1−w)·p_in`. - `saved = k·R_old·[warm·p_read + (1−warm)·p_miss] + k·R_new·p_miss − k·bust·warm·(p_miss − p_read)`. - It is computed as explicit counterfactual and actual buckets, so the two context tiers are priced independently. - **Provider rules:** - Cache minimums: OpenAI needs 1024 tokens; Anthropic uses the catalog's per-model `prompt_cache_min_tokens`, e.g. 4096 on Haiku 4.5. - OpenAI cache reads come in 128-token blocks. - 5m and 1h writes are priced at their own rates. - **Missing usage:** priced at list and marked `estimated`. - **Never raises.** The bust charge is capped at what the request actually missed. Item-level `P_S` undercounts a token-level hit, and without the cap the counterfactual buckets went negative; this is covered by a test. Also in this PR: `pricing.counterfactual.catalog_min_cacheable_tokens`, with the guarded catalog lookup shared through `_catalog_row`. ## Type of Change - [x] New feature (non-breaking change that adds functionality) - [x] Bug fix (non-breaking change that fixes an issue): review fix `bb1275b7`, below ## Changes Made - `headroom/proxy/savings.py`: `ItemCounts`, `ItemMeasure`, `SavingsRecord`, `measure_items`, `compute_savings_record` (the record described above). - `headroom/pricing/counterfactual.py`: `catalog_min_cacheable_tokens`, with the guarded catalog lookup shared through `_catalog_row`. - `tests/test_savings_record.py`: pinned-catalog tests. - Review fix `bb1275b7` (JerrettDavis): the counterfactual's cache buckets are now built whole instead of as deltas clamped onto the actual buckets. - **Bug:** take a 1,000-token original item that Headroom sent as 10,000 tokens on successive requests (`ItemMeasure(10000, -9000, 0, 10000, 0)`, `CacheMix(read=9000, uncached=1000)`, OpenAI, $3/M in and $0.30/M read). The counterfactual inherited the actual reads and priced 1,000 tokens as reads: **$0.0003**. That prompt is below OpenAI's 1,024-token minimum, so it should cost **$0.003**. - **Fix: buckets.** The counterfactual's read, write and uncached buckets are nonnegative and sum to its billed input (`k × original`). - **Fix: reads.** Reads are the actual reads, plus the removed or busted old tokens that would have hit, minus the reads a grown old item gave up. They never exceed the counterfactual's billed input, and below its own cache minimum there are none. OpenAI reads come in 128-token blocks. - **Fix: writes.** A counterfactual smaller than anything shown cacheable has no writes. - **Catalog minimum.** The catalog's `prompt_cache_min_tokens` is used unless this request was actually cached or written below it. In that case the catalog is wrong and no minimum is applied, which is the direction that cannot inflate the saving. - **Pricing.** The part of each counterfactual bucket the request was billed for prices at billing rates; only the excess prices at estimate rates. When billing and estimate rates differ, a shrunk bucket can no longer drive the counterfactual below zero. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Unit tests pass. New file `tests/test_savings_record.py` (12 tests) uses a pinned catalog. It covers: - the worked example: k=1.0526, warm=0.97159, w=0.8, saved=$0.058618, matching the closed form to 1e-9 and also cf − actual; - a bust, an over-large bust, a cold cache, and 1h writes; - an inferred OpenAI write, and OpenAI 128-token blocks; - the per-model cache minimum, and the counterfactual's own context tier; - missing usage, an unpriced model, and the old/new item split. - [x] Related suites pass: `test_counterfactual_pricing`, `test_cost_cache_prices`, savings, cost and ledger suites (473 passed). - [x] `ruff check` / `ruff format --check` (0.16.8) and `mypy headroom` are clean. Review fix `bb1275b7` adds regression tests that assert cost, not just `warm`. **Fail before** is `bb1275b7`'s tests run against the parent's `savings.py`: 8 failed. **Pass after** is all 29 passing. - **Jerrett's exact case**, on OpenAI and Anthropic: actual $0.0057, counterfactual $0.003, saved −$0.0027. Before the fix the counterfactual was $0.0003. - **OpenAI boundaries:** originals of 1,023 / 1,024 / 1,151 / 1,152 / 2,000 tokens give 0 / 1,024 / 1,024 / 1,152 / 1,920 reads. Before the fix, 1,023 → 1,023 reads, 1,151 → 1,151, and 2,000 → 2,000. - **Anthropic (Haiku, minimum 4,096):** originals of 4,095 / 4,096 / 4,097 tokens give 0 / 4,096 / 4,097 reads. - **Too small to cache, with writes:** costs 1,000 × input with no write premium. Before the fix it was 1,000 × write. - **A grown old item alongside new removed items** loses its reads. Before the fix it kept all 9,000. - **No negative cost:** the counterfactual stays ≥ 0 when billing and estimate read rates differ. Before the fix it was −$0.0078 in the reviewer's probe. - **Guards that pass on both versions:** a catalog minimum the provider disproved is not applied; nothing removed saves exactly $0 on OpenAI and Anthropic. - **Unchanged:** the worked example, $0.058618. ### Test Output ```text $ UV_FROZEN=1 pytest tests/test_savings_record.py -q 29 passed in 1.98s # fail-before: the new tests against the parent savings.py 8 failed, 20 passed (test_an_old_item_headroom_grew_inherits_no_reads_below_the_cache_minimum: Obtained 0.0003, expected 0.003; ...) $ UV_FROZEN=1 pytest tests -q -k "savings or counterfactual or cost or ledger" 730 passed, 8 skipped, 16437 deselected $ ruff check . && ruff format --check headroom tests (ruff 0.17.0, as pinned in .pre-commit-config.yaml) All checks passed! $ mypy headroom --ignore-missing-imports headroom/integrations/langchain/chat_model.py:380: error: Function "RunnableBinding" could always be true [truthy-function] # the one error is in a file this PR does not touch (local langchain version); savings.py and counterfactual.py are clean ``` ## Real Behavior Proof - Environment: macOS, this branch, the live litellm catalog (`claude-sonnet-4-5`: $3/M input, $0.30/M read, $3.75/M write). - Exact command / steps: `compute_savings_record(ItemMeasure(95_000, 50_000, 10_000, 88_000, 0), CacheMix(uncached=2_000, read=90_000, write_5m=8_000), model="claude-sonnet-4-5", provider="anthropic")` - Observed result: `k=1.0526 warm=0.97159 w=0.80 saved_usd=0.058618 actual=0.063000 cf=0.121618`. With an over-large bust, `saved_usd=-0.003075`; the counterfactual stays ≥ 0. - Not tested: real traffic and provider-billed comparison. The module is not called by the proxy until the wiring PR, and the ±5% audit gate (Codex $0.0850, Claude Code $0.1883) lands with the audit eval PR. Review fix, on the live litellm catalog (`LITELLM_LOCAL_MODEL_COST_MAP=True`, `gpt-4o`: $2.50/M input, $1.25/M read), with Jerrett's measure: | | actual | counterfactual | saved | |---|---|---|---| | before (`5db1b343`) | $0.013750 | $0.001250 | −$0.012500 | | after (`bb1275b7`) | $0.013750 | $0.002500 | −$0.011250 | The worked example on `claude-sonnet-4-5` is unchanged: `saved_usd=0.058618`. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: no. The module is not called yet. - Kill switch / disable path: none needed; nothing calls the module until the wiring PR - Unsafe override required: no - Qualification impact: none - Rollback path: revert. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review Review notes: Plus `code-reviewer` passes on the review fix; its three BLOCKERs and two SHOULDs are fixed, and its final pass had no findings. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
369 lines
16 KiB
Python
369 lines
16 KiB
Python
"""The per-request savings record: counterfactual cost minus actual cost."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import dataclasses
|
|
|
|
import pytest
|
|
|
|
from headroom.pricing import counterfactual
|
|
from headroom.pricing.counterfactual import CacheMix, CacheRates
|
|
from headroom.proxy.savings import (
|
|
ItemCounts,
|
|
ItemMeasure,
|
|
compute_savings_record,
|
|
measure_items,
|
|
)
|
|
|
|
# $3/M input, $0.30/M cache read, $3.75/M 5m write, $6/M 1h write; the
|
|
# long-context tier doubles everything.
|
|
IN, READ, WRITE, WRITE_1H = 3e-6, 3e-7, 3.75e-6, 6e-6
|
|
BASE = CacheRates(read=READ, write_5m=WRITE, write_1h=WRITE_1H, uncached=IN)
|
|
LONG = CacheRates(read=2 * READ, write_5m=2 * WRITE, write_1h=2 * WRITE_1H, uncached=2 * IN)
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _fixed_catalog(monkeypatch):
|
|
"""Pin the catalog so the arithmetic is tested whatever litellm ships."""
|
|
|
|
def rates(model, *, long_context=False, provider=None, for_billing=False):
|
|
return None if model == "unpriced" else (LONG if long_context else BASE)
|
|
|
|
monkeypatch.setattr(counterfactual, "resolve_rates", rates)
|
|
monkeypatch.setattr(
|
|
counterfactual,
|
|
"catalog_min_cacheable_tokens",
|
|
lambda model: 4096 if model == "haiku" else 1024,
|
|
)
|
|
|
|
|
|
def _formula(m: ItemMeasure, u: CacheMix) -> float:
|
|
"""The closed form the record must agree with."""
|
|
billed = u.read + u.write_5m + u.uncached
|
|
k = billed / m.sent_tokens
|
|
warm = min(1.0, (u.read / billed) / (m.sent_shared_prefix / m.sent_tokens))
|
|
w = u.write_5m / (u.uncached + u.write_5m)
|
|
p_miss = w * WRITE + (1 - w) * IN
|
|
return (
|
|
k * m.removed_old * (warm * READ + (1 - warm) * p_miss)
|
|
+ k * m.removed_new * p_miss
|
|
- k * m.bust * warm * (p_miss - READ)
|
|
)
|
|
|
|
|
|
WARM = CacheMix(uncached=2_000, read=90_000, write_5m=8_000)
|
|
MEASURE = ItemMeasure(
|
|
sent_tokens=95_000, removed_old=50_000, removed_new=10_000, sent_shared_prefix=88_000, bust=0
|
|
)
|
|
|
|
|
|
def test_worked_example():
|
|
record = compute_savings_record(MEASURE, WARM, model="sonnet", provider="anthropic")
|
|
assert record.k == pytest.approx(1.0526, abs=1e-4)
|
|
assert record.warm == pytest.approx(0.97159, abs=1e-5)
|
|
assert record.write_share == pytest.approx(0.8)
|
|
assert record.saved_usd == pytest.approx(0.058618, abs=1e-6)
|
|
assert record.saved_usd == pytest.approx(_formula(MEASURE, WARM), abs=1e-9)
|
|
assert record.saved_usd == pytest.approx(
|
|
record.counterfactual_input_cost_usd - record.actual_input_cost_usd
|
|
)
|
|
assert record.saved_tokens == pytest.approx(100_000 / 95_000 * 60_000)
|
|
assert not record.estimated
|
|
|
|
|
|
def test_a_cache_bust_headroom_caused_is_charged_against_it():
|
|
busted = dataclasses.replace(MEASURE, bust=5_000)
|
|
record = compute_savings_record(busted, WARM, model="sonnet", provider="anthropic")
|
|
assert record.saved_usd == pytest.approx(_formula(busted, WARM), abs=1e-9)
|
|
assert record.saved_usd == pytest.approx(0.058618 - 0.016875, abs=1e-6)
|
|
|
|
|
|
def test_a_bust_never_claims_more_reads_than_the_request_missed():
|
|
"""Item-level P_S undercounts a token-level hit, so hit > sigma clips warm
|
|
to 1; the bust charge must still not drive the counterfactual negative."""
|
|
measure = ItemMeasure(
|
|
sent_tokens=10_000, removed_old=0, removed_new=0, sent_shared_prefix=2_000, bust=8_000
|
|
)
|
|
usage = CacheMix(uncached=500, read=9_000, write_5m=500)
|
|
record = compute_savings_record(measure, usage, model="sonnet", provider="anthropic")
|
|
assert record.counterfactual_input_cost_usd >= 0
|
|
# At most the 1,000 missed tokens could have been reads instead.
|
|
assert record.saved_usd == pytest.approx(-(500 * (IN - READ) + 500 * (WRITE - READ)))
|
|
|
|
|
|
def test_an_old_item_headroom_grew_inherits_no_reads_below_the_cache_minimum():
|
|
"""JerrettDavis's #4236 case: a 1,000-token item sent as 10,000 tokens on
|
|
successive requests, prompt cached. The 1,000-token counterfactual is
|
|
under the 1,024-token minimum, so none of it could have been a read."""
|
|
measure = ItemMeasure(
|
|
sent_tokens=10_000, removed_old=-9_000, removed_new=0, sent_shared_prefix=10_000, bust=0
|
|
)
|
|
usage = CacheMix(read=9_000, uncached=1_000)
|
|
for provider in ("openai", "anthropic"):
|
|
record = compute_savings_record(measure, usage, model="gpt", provider=provider)
|
|
assert record.actual_input_cost_usd == pytest.approx(0.0057)
|
|
assert record.counterfactual_input_cost_usd == pytest.approx(1_000 * IN) # $0.003
|
|
assert record.saved_usd == pytest.approx(-0.0027)
|
|
|
|
|
|
def test_summed_iteration_usage_does_not_calibrate_k():
|
|
"""Usage summed over server-side iterations describes more prompts than
|
|
the one sent body, so it must not scale the removal."""
|
|
summed = CacheMix(read=270_000, uncached=6_000, write_5m=24_000)
|
|
record = compute_savings_record(
|
|
MEASURE, summed, model="sonnet", provider="anthropic", estimated=True
|
|
)
|
|
assert record.k == 1.0
|
|
assert record.estimated
|
|
assert record.saved_tokens == 60_000
|
|
|
|
|
|
def test_a_cold_cache_prices_every_removed_token_as_a_miss():
|
|
cold = CacheMix(uncached=10_000, read=0, write_5m=90_000)
|
|
record = compute_savings_record(MEASURE, cold, model="sonnet", provider="anthropic")
|
|
assert record.warm == 0.0
|
|
p_miss = 0.9 * WRITE + 0.1 * IN
|
|
assert record.saved_usd == pytest.approx(100_000 / 95_000 * 60_000 * p_miss)
|
|
|
|
|
|
def test_one_hour_writes_price_at_their_own_rate():
|
|
usage = CacheMix(uncached=2_000, read=90_000, write_1h=8_000)
|
|
record = compute_savings_record(MEASURE, usage, model="sonnet", provider="anthropic")
|
|
k, warm = 100_000 / 95_000, record.warm
|
|
p_miss = 0.8 * WRITE_1H + 0.2 * IN
|
|
expected = k * 50_000 * (warm * READ + (1 - warm) * p_miss) + k * 10_000 * p_miss
|
|
assert record.saved_usd == pytest.approx(expected)
|
|
|
|
|
|
def test_an_inferred_openai_write_is_the_uncached_slice_not_a_write():
|
|
usage = CacheMix(read=90_000, write_total=10_000, uncached=10_000, inferred_write=True)
|
|
record = compute_savings_record(MEASURE, usage, model="gpt", provider="openai")
|
|
assert record.k == pytest.approx(100_000 / 95_000)
|
|
assert record.write_share == 0.0
|
|
|
|
|
|
def test_openai_reads_come_in_128_token_blocks():
|
|
measure = ItemMeasure(
|
|
sent_tokens=10_000, removed_old=1_000, removed_new=0, sent_shared_prefix=10_000, bust=0
|
|
)
|
|
usage = CacheMix(read=9_984, uncached=16) # 78 blocks
|
|
record = compute_savings_record(measure, usage, model="gpt", provider="openai")
|
|
# 1,000 * k removed reads would end mid-block: only whole blocks count.
|
|
total_read = 9_984 + 1_000 * record.k
|
|
reads = (total_read // 128) * 128 - 9_984
|
|
assert record.saved_usd == pytest.approx(reads * READ + (1_000 * record.k - reads) * IN)
|
|
|
|
|
|
def test_a_counterfactual_prefix_below_the_models_cache_minimum_is_never_warm():
|
|
# A 3,000-token old prefix: cacheable on Sonnet (1,024), not on Haiku (4,096).
|
|
measure = ItemMeasure(
|
|
sent_tokens=10_000, removed_old=1_000, removed_new=0, sent_shared_prefix=2_000, bust=0
|
|
)
|
|
usage = CacheMix(uncached=5_000, read=5_000)
|
|
assert compute_savings_record(measure, usage, model="sonnet").warm > 0
|
|
assert compute_savings_record(measure, usage, model="haiku").warm == 0.0
|
|
|
|
|
|
def test_the_counterfactual_is_priced_at_its_own_context_tier():
|
|
measure = ItemMeasure(
|
|
sent_tokens=150_000, removed_old=0, removed_new=100_000, sent_shared_prefix=0, bust=0
|
|
)
|
|
record = compute_savings_record(measure, CacheMix(uncached=150_000), model="sonnet")
|
|
assert record.actual_input_cost_usd == pytest.approx(150_000 * IN)
|
|
assert record.counterfactual_input_cost_usd == pytest.approx(250_000 * 2 * IN)
|
|
|
|
|
|
def test_missing_usage_prices_at_list_per_tier_and_says_so():
|
|
measure = dataclasses.replace(MEASURE, sent_tokens=150_000, removed_new=50_000)
|
|
record = compute_savings_record(measure, None, model="sonnet")
|
|
assert record.estimated
|
|
assert record.actual_input_cost_usd == pytest.approx(150_000 * IN)
|
|
assert record.counterfactual_input_cost_usd == pytest.approx(250_000 * 2 * IN)
|
|
|
|
|
|
def test_an_unpriced_model_reports_tokens_but_no_dollars():
|
|
record = compute_savings_record(MEASURE, WARM, model="unpriced")
|
|
assert not record.priced
|
|
assert record.saved_usd == 0.0
|
|
assert record.removed_tokens == 60_000
|
|
|
|
|
|
def test_items_split_into_old_and_new_against_the_previous_request():
|
|
def item(name: str, original: int, sent: int, sent_name: str | None = None) -> ItemCounts:
|
|
return ItemCounts(name, original, sent_name or name + "'", sent)
|
|
|
|
first = [item("tools", 1_000, 200), item("sys", 500, 500, "sys"), item("u1", 300, 300, "u1")]
|
|
second = [
|
|
item("tools", 1_000, 200),
|
|
item("sys", 500, 500, "sys"),
|
|
# Headroom compressed this item differently this time: its sent copy
|
|
# changed, so the provider missed it although the client resent it.
|
|
item("u1", 300, 250, "u1-recompressed"),
|
|
item("t1", 4_000, 1_000),
|
|
# An item Headroom grew counts against the removal.
|
|
item("t2", 100, 160),
|
|
]
|
|
assert measure_items(first, None) == ItemMeasure(
|
|
sent_tokens=1_000, removed_old=0, removed_new=800, sent_shared_prefix=0, bust=0
|
|
)
|
|
assert measure_items(second, first) == ItemMeasure(
|
|
sent_tokens=2_110,
|
|
removed_old=800 + 50,
|
|
removed_new=3_000 - 60,
|
|
sent_shared_prefix=700,
|
|
bust=250,
|
|
grown_new=60,
|
|
)
|
|
|
|
|
|
def test_billed_buckets_price_at_billing_rates(monkeypatch):
|
|
"""A catalog row without a cache-read rate bills reads via litellm's
|
|
calculator (``for_billing``); only the changed tokens use the estimate."""
|
|
billing = dataclasses.replace(BASE, read=5e-7)
|
|
|
|
def rates(model, *, long_context=False, provider=None, for_billing=False):
|
|
return billing if for_billing else BASE
|
|
|
|
monkeypatch.setattr(counterfactual, "resolve_rates", rates)
|
|
record = compute_savings_record(MEASURE, WARM, model="sonnet", provider="anthropic")
|
|
assert record.actual_input_cost_usd == pytest.approx(2_000 * IN + 90_000 * 5e-7 + 8_000 * WRITE)
|
|
assert record.saved_usd == pytest.approx(0.058618, abs=1e-6)
|
|
|
|
|
|
def _fully_cached_original_of(original: int, *, model: str, provider: str):
|
|
"""A 10,000-token sent body read wholly from cache, whose original was
|
|
``original`` tokens: the counterfactual is that original alone."""
|
|
measure = ItemMeasure(
|
|
sent_tokens=10_000,
|
|
removed_old=original - 10_000,
|
|
removed_new=0,
|
|
sent_shared_prefix=10_000,
|
|
bust=0,
|
|
)
|
|
return compute_savings_record(measure, CacheMix(read=10_000), model=model, provider=provider)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("original", "reads"),
|
|
[(1_023, 0), (1_024, 1_024), (1_151, 1_024), (1_152, 1_152), (2_000, 1_920)],
|
|
)
|
|
def test_an_openai_counterfactual_reads_whole_blocks_above_its_own_minimum(original, reads):
|
|
record = _fully_cached_original_of(original, model="gpt", provider="openai")
|
|
expected = reads * READ + (original - reads) * IN
|
|
assert record.counterfactual_input_cost_usd == pytest.approx(expected)
|
|
|
|
|
|
@pytest.mark.parametrize(("original", "reads"), [(4_095, 0), (4_096, 4_096), (4_097, 4_097)])
|
|
def test_an_anthropic_counterfactual_obeys_its_models_cache_minimum(original, reads):
|
|
record = _fully_cached_original_of(original, model="haiku", provider="anthropic")
|
|
expected = reads * READ + (original - reads) * IN
|
|
assert record.counterfactual_input_cost_usd == pytest.approx(expected)
|
|
|
|
|
|
def test_a_counterfactual_too_small_to_cache_pays_no_write_premium():
|
|
"""Below its minimum the counterfactual can be neither read nor written."""
|
|
measure = ItemMeasure(
|
|
sent_tokens=10_000, removed_old=-9_000, removed_new=0, sent_shared_prefix=10_000, bust=0
|
|
)
|
|
usage = CacheMix(read=9_000, write_5m=1_000)
|
|
record = compute_savings_record(measure, usage, model="sonnet", provider="anthropic")
|
|
assert record.counterfactual_input_cost_usd == pytest.approx(1_000 * IN)
|
|
|
|
|
|
def test_a_grown_old_item_takes_its_reads_out_even_with_new_items():
|
|
"""The new items keep the counterfactual above the minimum overall; its
|
|
1,000-token old prefix is still too small to have been read."""
|
|
measure = ItemMeasure(
|
|
sent_tokens=30_000,
|
|
removed_old=-9_000,
|
|
removed_new=20_000,
|
|
sent_shared_prefix=10_000,
|
|
bust=0,
|
|
)
|
|
usage = CacheMix(read=9_000, uncached=21_000)
|
|
record = compute_savings_record(measure, usage, model="gpt", provider="openai")
|
|
assert record.counterfactual_input_cost_usd == pytest.approx(41_000 * IN)
|
|
|
|
|
|
def test_a_catalog_minimum_the_provider_disproved_is_not_applied():
|
|
"""Haiku's catalog says 4,096, but the provider read 3,000: keeping the
|
|
counterfactual's 1,500 reads is the conservative price."""
|
|
measure = ItemMeasure(
|
|
sent_tokens=10_000, removed_old=-5_000, removed_new=0, sent_shared_prefix=10_000, bust=0
|
|
)
|
|
usage = CacheMix(read=3_000, uncached=7_000)
|
|
record = compute_savings_record(measure, usage, model="haiku", provider="anthropic")
|
|
assert record.counterfactual_input_cost_usd == pytest.approx(1_500 * READ + 3_500 * IN)
|
|
|
|
|
|
def test_nothing_removed_saves_nothing_whatever_the_read_alignment():
|
|
measure = ItemMeasure(
|
|
sent_tokens=10_000, removed_old=0, removed_new=0, sent_shared_prefix=2_000, bust=0
|
|
)
|
|
for provider, usage in (
|
|
("openai", CacheMix(read=9_000, uncached=1_000)),
|
|
("anthropic", CacheMix(read=900, write_5m=9_100)),
|
|
):
|
|
record = compute_savings_record(measure, usage, model="haiku", provider=provider)
|
|
assert record.saved_usd == pytest.approx(0.0, abs=1e-12)
|
|
|
|
|
|
def test_a_shrunk_bucket_is_never_priced_below_zero(monkeypatch):
|
|
"""Reads a grown item gave up price at the billing rate they were billed
|
|
at, not at the estimate rate, so the counterfactual cannot go negative."""
|
|
billing = dataclasses.replace(BASE, read=1.5e-6)
|
|
|
|
def rates(model, *, long_context=False, provider=None, for_billing=False):
|
|
return billing if for_billing else BASE
|
|
|
|
monkeypatch.setattr(counterfactual, "resolve_rates", rates)
|
|
measure = ItemMeasure(
|
|
sent_tokens=10_000, removed_old=-9_000, removed_new=0, sent_shared_prefix=10_000, bust=0
|
|
)
|
|
record = compute_savings_record(
|
|
measure, CacheMix(read=9_000, uncached=1_000), model="gpt", provider="openai"
|
|
)
|
|
assert record.counterfactual_input_cost_usd == pytest.approx(1_000 * IN)
|
|
|
|
|
|
def test_a_disproved_minimum_still_writes_nothing_smaller_than_was_cached():
|
|
"""Haiku's catalog minimum is disproved by a 2,000-token read; a 500-token
|
|
counterfactual is below anything shown cacheable, so it writes nothing."""
|
|
measure = ItemMeasure(
|
|
sent_tokens=3_000, removed_old=-2_500, removed_new=0, sent_shared_prefix=3_000, bust=0
|
|
)
|
|
usage = CacheMix(read=2_000, write_5m=1_000)
|
|
record = compute_savings_record(measure, usage, model="haiku", provider="anthropic")
|
|
reads = 2_000 + -2_500 * (2 / 3)
|
|
assert record.counterfactual_input_cost_usd == pytest.approx(reads * READ + (500 - reads) * IN)
|
|
|
|
|
|
def test_a_lost_baseline_prices_the_new_removal_at_the_read_floor():
|
|
"""With the conversation's baseline evicted, the new items may be old ones
|
|
the provider had cached. Pricing them as new, at the miss rate, booked
|
|
$0.2236 here; the read rate, the floor of any input token, books $0.0152.
|
|
The 1,000 tokens Headroom added to a new item still cost the miss rate."""
|
|
lost = ItemMeasure(
|
|
sent_tokens=95_000,
|
|
removed_old=0,
|
|
removed_new=60_000 - 1_000,
|
|
sent_shared_prefix=0,
|
|
bust=0,
|
|
aligned=False,
|
|
baseline_lost=True,
|
|
grown_new=1_000,
|
|
)
|
|
k, p_miss = 100_000 / 95_000, 0.8 * WRITE + 0.2 * IN
|
|
as_new = compute_savings_record(
|
|
dataclasses.replace(lost, baseline_lost=False), WARM, model="sonnet", provider="anthropic"
|
|
)
|
|
assert as_new.saved_usd == pytest.approx(k * 59_000 * p_miss) # $0.2236
|
|
|
|
record = compute_savings_record(lost, WARM, model="sonnet", provider="anthropic")
|
|
assert record.saved_usd == pytest.approx(k * 60_000 * READ - k * 1_000 * p_miss) # $0.0152
|
|
assert record.saved_usd == pytest.approx(0.015158, abs=1e-6)
|
|
assert record.estimated
|
|
# Without usage too: list price for the rest, the read rate for the removal.
|
|
unbilled = compute_savings_record(lost, None, model="sonnet", provider="anthropic")
|
|
assert unbilled.saved_usd == pytest.approx(60_000 * READ - 1_000 * IN)
|