mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(tests): ask litellm for a model it prices instead of naming a retired one (#3722)
## What broke `litellm` downloads `model_prices_and_context_window.json` from GitHub **at import time**, so `litellm.model_cost` is live third-party data we do not control. On 2026-09-23 BerriAI pruned retired Anthropic ids from that table. `claude-sonnet-4-20250514` disappeared, and every test that priced it started failing with `KeyError: 'input_cost_per_token'` — on every open PR simultaneously, with no change on our side. `main` was green at `5ff4ea1e` on 09-22 and the identical shards went red on #3720 and #3721, which share that base and touch no pricing code. ## What this does The model id in those tests is **incidental**. They assert that Headroom's cost arithmetic agrees with litellm's numbers — not that any particular model is priced correctly. So `tests/_pricing_models.py` asks for *a* model carrying the fields a test needs, newest first, and raises with an actionable message if litellm prices none of them. ```python MODEL = anthropic_pricing_model( "input_cost_per_token_above_200k_tokens", "cache_creation_input_token_cost", "cache_read_input_token_cost", ) ``` Retiring one candidate is now a no-op until the list runs dry. ## Why not pin to the wheel's vendored copy That was this PR's first attempt and **it was wrong** — I'm recording why, because it looks like the obvious fix. The live table and the copy vendored in the litellm wheel are *complementary, not ordered*: | | live | vendored | |---|---|---| | `claude-sonnet-4-20250514` | ✗ pruned | ✓ | | `claude-sonnet-5` | ✓ | ✗ predates it | | live-only entries | 1743 | — | | vendored-only entries | — | 314 | Pinning fixed the 25 `KeyError`s and **broke `test_proxy_savings_history`**, which prices `claude-sonnet-5`. Taking the union does not rescue it either: the vendored `claude-sonnet-4-20250514` entry is an older schema that has no `input_cost_per_token_above_200k_tokens` field at all, so `test_cost_card_attribution` still fails against it. ## A second, separate breakage in the same data drop `test_provider_prices_models_its_table_never_covered[groq/llama-3.3-70b-versatile]` was failing for a different reason: litellm dropped that model too, so `_get_pricing` fell through to the `$2.50/$10.00` GPT-4o default — which that test's own comment names as the "unknown default" it exists to catch. Repointed at `groq/llama-guard-3-8b`, which litellm still prices. **The product-side problem is filed separately as #3732**: an unresolvable model is silently billed at the GPT-4o tier, so cost tracking and `check_budget` — a hard control that refuses requests — can act on a fabricated number. For that groq model the default overstated output cost by ~12x. ## Verification Same six modules, same machine: ``` origin/main 25 failed, 82 passed this branch 107 passed ``` Ruff 0.16.7 `check` + `format --check` clean. ## Prior art #3683 / #3684 was the same class of failure (litellm started billing DeepSeek off-peak) and was patched narrowly at the assertion. This removes the category for model *identity*; #3732 covers what happens in production when identity goes missing.
This commit is contained in:
Binary file not shown.
|
Before Width: | Height: | Size: 260 KiB After Width: | Height: | Size: 273 KiB |
@@ -27,15 +27,25 @@ class LiteLLMModelPrefixRule:
|
||||
|
||||
# Aliases for models removed from LiteLLM's cost database (retired/renamed).
|
||||
# Maps old model name -> current LiteLLM key that has equivalent pricing.
|
||||
# Retired model ids, aliased to a CURRENTLY PRICED model of the same tier.
|
||||
#
|
||||
# An alias target must itself still be in litellm's table. These pointed at
|
||||
# claude-sonnet-4-20250514 until litellm pruned it on 2026-09-23, at which point
|
||||
# all three aliases resolved to nothing and every one of these models fell
|
||||
# through to the unknown-model default (the GPT-4o tier, $2.50/$10.00) -- the
|
||||
# same class of silent mispricing the Haiku note below warns about. The guard
|
||||
# test fails the build if a target is pruned again, so this cannot rot quietly.
|
||||
#
|
||||
# The undated target is deliberate: it survives a dated release being retired.
|
||||
MODEL_ALIASES: dict[str, str] = {
|
||||
# Claude 3.5 Sonnet retired Feb 2026, pricing same as claude-sonnet-4-20250514
|
||||
"claude-3-5-sonnet-20241022": "claude-sonnet-4-20250514",
|
||||
"claude-3-5-sonnet-20240620": "claude-sonnet-4-20250514",
|
||||
"claude-3-5-sonnet-20241022": "claude-sonnet-4-5",
|
||||
"claude-3-5-sonnet-20240620": "claude-sonnet-4-5",
|
||||
# Claude 3 Sonnet retired. It was a Sonnet-tier model ($3/$15 per 1M
|
||||
# in/out) — same price as claude-sonnet-4-20250514 — so alias it there.
|
||||
# The old target, claude-3-haiku-20240307 ($0.25/$1.25), is a different
|
||||
# (Haiku) tier and underpriced every cost/savings figure ~12x.
|
||||
"claude-3-sonnet-20240229": "claude-sonnet-4-20250514",
|
||||
"claude-3-sonnet-20240229": "claude-sonnet-4-5",
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
"""Pick a model litellm actually prices, instead of hardcoding one it may retire.
|
||||
|
||||
``litellm.model_cost`` is downloaded from GitHub at import time, so it is live
|
||||
third-party data. BerriAI prunes retired models from it: on 2026-09-23
|
||||
``claude-sonnet-4-20250514`` disappeared and every test that priced it began
|
||||
failing with ``KeyError: 'input_cost_per_token'`` — on every open pull request
|
||||
at once, with no change on our side.
|
||||
|
||||
The model id in those tests is incidental. They assert that Headroom's cost
|
||||
arithmetic agrees with litellm's numbers, not that any particular model is
|
||||
priced correctly, so the fix is to stop naming a specific release and instead
|
||||
ask for *a* model carrying the fields the test needs.
|
||||
|
||||
Pinning to the copy of the table vendored in the litellm wheel is not the
|
||||
alternative it looks like: the two maps are complementary, not ordered. The
|
||||
vendored map keeps retired ids but predates current models
|
||||
(``claude-sonnet-5``), and its older entries lack newer fields such as
|
||||
``input_cost_per_token_above_200k_tokens`` entirely.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
#: Preference order, newest first. A test takes the first entry that carries
|
||||
#: every field it needs, so retiring one is a no-op until the list runs dry.
|
||||
_CANDIDATES: tuple[str, ...] = (
|
||||
"claude-sonnet-4-5-20250929",
|
||||
"claude-sonnet-4-5",
|
||||
"claude-sonnet-4-20250514",
|
||||
"claude-opus-4-5-20251101",
|
||||
"claude-opus-4-5",
|
||||
)
|
||||
|
||||
_BASE_FIELDS = ("input_cost_per_token", "output_cost_per_token")
|
||||
|
||||
|
||||
def anthropic_pricing_model(*required_fields: str) -> str:
|
||||
"""Return a currently-priced Anthropic model carrying ``required_fields``.
|
||||
|
||||
Always includes the base input/output costs. Raises with an actionable
|
||||
message rather than skipping: if litellm prices none of these, the pricing
|
||||
tests are not measuring anything and that should be loud.
|
||||
"""
|
||||
import litellm
|
||||
|
||||
needed = set(_BASE_FIELDS) | set(required_fields)
|
||||
for model in _CANDIDATES:
|
||||
info = litellm.model_cost.get(model)
|
||||
if isinstance(info, dict) and needed <= set(info):
|
||||
return model
|
||||
|
||||
raise AssertionError(
|
||||
"litellm prices none of the candidate models with the fields "
|
||||
f"{sorted(needed)}. It most likely retired them from "
|
||||
"model_prices_and_context_window.json; add a current model id to "
|
||||
"_CANDIDATES in tests/_pricing_models.py (newest first)."
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def anthropic_model() -> str:
|
||||
"""Session fixture wrapper for tests that prefer injection over a constant."""
|
||||
return anthropic_pricing_model()
|
||||
@@ -22,13 +22,15 @@ from tests._dotenv import (
|
||||
importorskip_no_env_leak,
|
||||
load_env_overrides,
|
||||
)
|
||||
from tests._pricing_models import anthropic_pricing_model
|
||||
|
||||
_env_overrides = load_env_overrides()
|
||||
apply_dotenv = autouse_apply_env(_env_overrides)
|
||||
|
||||
importorskip_no_env_leak("litellm")
|
||||
|
||||
MODEL = "claude-sonnet-4-20250514"
|
||||
|
||||
MODEL = anthropic_pricing_model()
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
|
||||
@@ -16,13 +16,19 @@ from tests._dotenv import (
|
||||
importorskip_no_env_leak,
|
||||
load_env_overrides,
|
||||
)
|
||||
from tests._pricing_models import anthropic_pricing_model
|
||||
|
||||
_env_overrides = load_env_overrides()
|
||||
apply_dotenv = autouse_apply_env(_env_overrides)
|
||||
|
||||
importorskip_no_env_leak("litellm")
|
||||
|
||||
MODEL = "claude-sonnet-4-20250514"
|
||||
|
||||
MODEL = anthropic_pricing_model(
|
||||
"input_cost_per_token_above_200k_tokens",
|
||||
"cache_creation_input_token_cost",
|
||||
"cache_read_input_token_cost",
|
||||
)
|
||||
|
||||
|
||||
def _prices(model: str = MODEL) -> tuple[float, float, float]:
|
||||
|
||||
@@ -11,6 +11,7 @@ from tests._dotenv import (
|
||||
importorskip_no_env_leak,
|
||||
load_env_overrides,
|
||||
)
|
||||
from tests._pricing_models import anthropic_pricing_model
|
||||
|
||||
_env_overrides = load_env_overrides()
|
||||
apply_dotenv = autouse_apply_env(_env_overrides)
|
||||
@@ -18,12 +19,15 @@ apply_dotenv = autouse_apply_env(_env_overrides)
|
||||
importorskip_no_env_leak("litellm")
|
||||
|
||||
|
||||
MODEL = anthropic_pricing_model()
|
||||
|
||||
|
||||
def test_savings_at_list_price():
|
||||
"""savings_usd = tokens_saved * model list input price."""
|
||||
from headroom.proxy.server import CostTracker
|
||||
|
||||
ct = CostTracker()
|
||||
model = "claude-sonnet-4-20250514"
|
||||
model = MODEL
|
||||
|
||||
ct.record_tokens(
|
||||
model,
|
||||
@@ -54,7 +58,7 @@ def test_savings_monotonic():
|
||||
from headroom.proxy.server import CostTracker
|
||||
|
||||
ct = CostTracker()
|
||||
model = "claude-sonnet-4-20250514"
|
||||
model = MODEL
|
||||
|
||||
ct.record_tokens(model, tokens_saved=10_000, tokens_sent=5_000)
|
||||
stats1 = ct.stats()
|
||||
@@ -71,7 +75,7 @@ def test_savings_zero_when_no_tokens_saved():
|
||||
from headroom.proxy.server import CostTracker
|
||||
|
||||
ct = CostTracker()
|
||||
model = "claude-sonnet-4-20250514"
|
||||
model = MODEL
|
||||
|
||||
ct.record_tokens(model, tokens_saved=0, tokens_sent=5_000)
|
||||
stats = ct.stats()
|
||||
@@ -99,7 +103,7 @@ def test_multi_model_savings():
|
||||
|
||||
ct = CostTracker()
|
||||
|
||||
ct.record_tokens("claude-sonnet-4-20250514", tokens_saved=50_000, tokens_sent=10_000)
|
||||
ct.record_tokens(MODEL, tokens_saved=50_000, tokens_sent=10_000)
|
||||
ct.record_tokens("claude-haiku-4-5-20251001", tokens_saved=50_000, tokens_sent=10_000)
|
||||
|
||||
stats = ct.stats()
|
||||
@@ -117,7 +121,7 @@ def test_no_cost_without_headroom_field():
|
||||
from headroom.proxy.server import CostTracker
|
||||
|
||||
ct = CostTracker()
|
||||
ct.record_tokens("claude-sonnet-4-20250514", tokens_saved=10_000, tokens_sent=5_000)
|
||||
ct.record_tokens(MODEL, tokens_saved=10_000, tokens_sent=5_000)
|
||||
stats = ct.stats()
|
||||
|
||||
assert "cost_without_headroom_usd" not in stats
|
||||
@@ -137,7 +141,7 @@ def test_budget_enforced_after_recording_costs():
|
||||
|
||||
# ~$1.50+ of Sonnet input at list price — far over the budget
|
||||
ct.record_tokens(
|
||||
"claude-sonnet-4-20250514",
|
||||
MODEL,
|
||||
tokens_saved=0,
|
||||
tokens_sent=500_000,
|
||||
uncached_tokens=500_000,
|
||||
@@ -158,7 +162,7 @@ def test_budget_input_cost_counted_without_usage_breakdown():
|
||||
|
||||
ct = CostTracker(budget_limit_usd=100.0)
|
||||
ct.record_tokens(
|
||||
"claude-sonnet-4-20250514",
|
||||
MODEL,
|
||||
tokens_saved=0,
|
||||
tokens_sent=500_000,
|
||||
)
|
||||
|
||||
@@ -20,6 +20,7 @@ from tests._dotenv import (
|
||||
importorskip_no_env_leak,
|
||||
load_env_overrides,
|
||||
)
|
||||
from tests._pricing_models import anthropic_pricing_model
|
||||
|
||||
_env_overrides = load_env_overrides()
|
||||
apply_dotenv = autouse_apply_env(_env_overrides)
|
||||
@@ -40,7 +41,7 @@ from headroom.pricing.counterfactual import ( # noqa: E402
|
||||
weakest_basis,
|
||||
)
|
||||
|
||||
SONNET = "claude-sonnet-4-20250514"
|
||||
SONNET = anthropic_pricing_model()
|
||||
|
||||
|
||||
def _warm_anthropic() -> CacheMix:
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
"""An alias target that litellm no longer prices is worse than no alias.
|
||||
|
||||
`MODEL_ALIASES` exists so a retired model id still prices at its own tier.
|
||||
Every target pointed at `claude-sonnet-4-20250514` until litellm pruned it on
|
||||
2026-09-23; all three aliases then resolved to nothing and those models fell
|
||||
through to the unknown-model default -- the GPT-4o tier -- which silently
|
||||
misprices them. Nothing failed, because no test asserted the targets were live.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from tests._dotenv import importorskip_no_env_leak
|
||||
|
||||
importorskip_no_env_leak("litellm")
|
||||
|
||||
from headroom.pricing.litellm_model_resolution import MODEL_ALIASES # noqa: E402
|
||||
from headroom.pricing.litellm_pricing import get_model_pricing # noqa: E402
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("retired", "target"), sorted(MODEL_ALIASES.items()))
|
||||
def test_model_aliases_point_at_something_litellm_still_prices(retired: str, target: str) -> None:
|
||||
"""The target must be priced, or the alias is a silent no-op."""
|
||||
assert get_model_pricing(target) is not None, (
|
||||
f"MODEL_ALIASES[{retired!r}] points at {target!r}, which litellm no longer "
|
||||
"prices. Repoint it at a current model of the same tier "
|
||||
"(headroom/pricing/litellm_model_resolution.py)."
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("retired", "target"), sorted(MODEL_ALIASES.items()))
|
||||
def test_a_retired_id_prices_at_its_alias_rate(retired: str, target: str) -> None:
|
||||
"""The point of the alias: the retired id resolves to the target's numbers."""
|
||||
priced = get_model_pricing(retired)
|
||||
assert priced is not None, f"{retired!r} did not resolve through its alias"
|
||||
|
||||
target_priced = get_model_pricing(target)
|
||||
assert target_priced is not None
|
||||
assert (priced.input_cost_per_1m, priced.output_cost_per_1m) == (
|
||||
target_priced.input_cost_per_1m,
|
||||
target_priced.output_cost_per_1m,
|
||||
)
|
||||
|
||||
|
||||
def test_retired_sonnets_are_not_priced_at_the_unknown_default() -> None:
|
||||
"""Regression for the failure mode this whole file is about.
|
||||
|
||||
The unknown-model fallback is the GPT-4o tier. A Sonnet-tier model landing
|
||||
there is wrong in both directions and is exactly what happened when the
|
||||
alias targets went stale.
|
||||
"""
|
||||
gpt4o = get_model_pricing("gpt-4o")
|
||||
assert gpt4o is not None
|
||||
|
||||
for retired in MODEL_ALIASES:
|
||||
priced = get_model_pricing(retired)
|
||||
assert priced is not None
|
||||
assert (priced.input_cost_per_1m, priced.output_cost_per_1m) != (
|
||||
gpt4o.input_cost_per_1m,
|
||||
gpt4o.output_cost_per_1m,
|
||||
), f"{retired!r} is priced at the GPT-4o tier — its alias is not resolving"
|
||||
@@ -13,6 +13,7 @@ from headroom.models import (
|
||||
list_models,
|
||||
register_model,
|
||||
)
|
||||
from tests._pricing_models import anthropic_pricing_model
|
||||
|
||||
|
||||
class TestModelInfo:
|
||||
@@ -285,8 +286,9 @@ class TestBuiltInModels:
|
||||
info = get_model_info("claude-3-5-sonnet-20241022")
|
||||
assert info.provider == "anthropic"
|
||||
assert info.context_window == 200000
|
||||
# Pricing fetched from LiteLLM (falls back to alias for retired models)
|
||||
pricing = ModelRegistry.get_pricing("claude-sonnet-4-20250514")
|
||||
# Pricing comes from litellm's live table, so name a model it currently
|
||||
# prices; the retired-id path is the MODEL_ALIASES assertion below.
|
||||
pricing = ModelRegistry.get_pricing(anthropic_pricing_model())
|
||||
assert pricing is not None
|
||||
assert pricing[0] == 3.00 # input cost per 1M
|
||||
assert pricing[1] == 15.00 # output cost per 1M
|
||||
|
||||
@@ -41,7 +41,12 @@ def test_unwrapped_model_forms_drops_leading_segments() -> None:
|
||||
("bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0", 3.00, 15.00),
|
||||
("bedrock/us.anthropic.claude-3-5-sonnet-20241022-v2:0", 3.00, 15.00),
|
||||
("vertex_ai/claude-sonnet-4-5", 3.00, 15.00),
|
||||
("groq/llama-3.3-70b-versatile", 0.59, 0.79),
|
||||
# Any gateway-prefixed name here must be one litellm still prices:
|
||||
# when it prunes a model the unwrap finds nothing and _get_pricing
|
||||
# silently returns the $2.50/$10.00 GPT-4o default, which is what
|
||||
# this test exists to catch. litellm dropped
|
||||
# groq/llama-3.3-70b-versatile on 2026-09-23; see issue #3732.
|
||||
("groq/llama-guard-3-8b", 0.20, 0.20),
|
||||
# Non-OpenAI models reachable through the OpenAI-compatible passthrough.
|
||||
("gemini-2.5-flash", 0.30, 2.50),
|
||||
("deepseek-chat", 0.28, 0.42),
|
||||
|
||||
@@ -3,6 +3,7 @@ from __future__ import annotations
|
||||
from types import SimpleNamespace
|
||||
|
||||
from headroom.pricing import litellm_pricing
|
||||
from headroom.pricing.litellm_model_resolution import MODEL_ALIASES
|
||||
|
||||
|
||||
def test_litellm_helpers_when_dependency_is_unavailable(monkeypatch) -> None:
|
||||
@@ -72,9 +73,14 @@ def test_litellm_model_pricing_uses_provider_prefixes(monkeypatch) -> None:
|
||||
|
||||
|
||||
def test_litellm_model_pricing_uses_aliases_and_zero_cost_defaults(monkeypatch) -> None:
|
||||
# Keyed off the alias table rather than restating its target: this named
|
||||
# "claude-sonnet-4-20250514" and broke when the alias moved, even though the
|
||||
# behaviour under test (alias resolution + None costs reading as zero) was
|
||||
# unchanged.
|
||||
alias_target = MODEL_ALIASES["claude-3-5-sonnet-20241022"]
|
||||
fake_litellm = SimpleNamespace(
|
||||
model_cost={
|
||||
"claude-sonnet-4-20250514": {
|
||||
alias_target: {
|
||||
"input_cost_per_token": None,
|
||||
"output_cost_per_token": None,
|
||||
}
|
||||
|
||||
@@ -47,9 +47,15 @@ def test_retired_claude_3_sonnet_aliases_to_sonnet_tier_not_haiku() -> None:
|
||||
alias = MODEL_ALIASES["claude-3-sonnet-20240229"]
|
||||
|
||||
assert "haiku" not in alias
|
||||
assert "sonnet" in alias
|
||||
# Same-tier target as the other retired-Sonnet aliases.
|
||||
assert alias == MODEL_ALIASES["claude-3-5-sonnet-20241022"]
|
||||
assert alias == "claude-sonnet-4-20250514"
|
||||
# Deliberately not a specific release: this asserted
|
||||
# "claude-sonnet-4-20250514" until litellm pruned that id, at which point
|
||||
# the alias had to move and this test broke for naming the target rather
|
||||
# than the property it cares about. That the target is *priced*, and priced
|
||||
# at Sonnet rates rather than the unknown-model default, is asserted in
|
||||
# tests/test_model_alias_targets_are_priced.py.
|
||||
|
||||
|
||||
def test_resolve_litellm_model_name_returns_first_known_candidate() -> None:
|
||||
|
||||
@@ -8,6 +8,9 @@ from datetime import datetime, timedelta, timezone
|
||||
import pytest
|
||||
|
||||
from headroom import savings_ledger as L
|
||||
from tests._pricing_models import anthropic_pricing_model
|
||||
|
||||
MODEL = anthropic_pricing_model()
|
||||
|
||||
UTC = timezone.utc
|
||||
|
||||
@@ -417,7 +420,7 @@ def test_v1_events_still_aggregate_alongside_v2(tmp_path):
|
||||
"after": 6_000,
|
||||
"saved": 4_000,
|
||||
"cost_usd": 0.012,
|
||||
"model": "claude-sonnet-4-20250514",
|
||||
"model": MODEL,
|
||||
"client": "claude-code",
|
||||
"source": "proxy",
|
||||
}
|
||||
@@ -427,7 +430,7 @@ def test_v1_events_still_aggregate_alongside_v2(tmp_path):
|
||||
sl.record_savings_event(
|
||||
tokens_before=108_000,
|
||||
tokens_after=100_000,
|
||||
model="claude-sonnet-4-20250514",
|
||||
model=MODEL,
|
||||
client="claude-code",
|
||||
source="proxy",
|
||||
saved_compression=1_000,
|
||||
|
||||
@@ -25,6 +25,13 @@ from tests._dotenv import (
|
||||
importorskip_no_env_leak,
|
||||
load_env_overrides,
|
||||
)
|
||||
from tests._pricing_models import anthropic_pricing_model
|
||||
|
||||
MODEL = anthropic_pricing_model(
|
||||
"input_cost_per_token_above_200k_tokens",
|
||||
"cache_creation_input_token_cost",
|
||||
"cache_read_input_token_cost",
|
||||
)
|
||||
|
||||
_env_overrides = load_env_overrides()
|
||||
apply_dotenv = autouse_apply_env(_env_overrides)
|
||||
@@ -36,14 +43,14 @@ from headroom.proxy.savings_tracker import estimate_request_savings_usd # noqa:
|
||||
# One warm turn per provider, in that provider's own reporting shape.
|
||||
WARM_TURNS = {
|
||||
"anthropic": {
|
||||
"model": "claude-sonnet-4-20250514",
|
||||
"model": MODEL,
|
||||
"cache_read_tokens": 180_000,
|
||||
"cache_write_tokens": 2_000,
|
||||
"cache_write_5m_tokens": 2_000,
|
||||
"uncached_input_tokens": 500,
|
||||
},
|
||||
"bedrock": {
|
||||
"model": "claude-sonnet-4-20250514",
|
||||
"model": MODEL,
|
||||
"cache_read_tokens": 180_000,
|
||||
"cache_write_tokens": 2_000,
|
||||
"cache_write_5m_tokens": 2_000,
|
||||
@@ -132,7 +139,7 @@ def test_a_provider_that_reports_no_cache_data_prices_at_list_and_says_so():
|
||||
not presented as a measurement.
|
||||
"""
|
||||
priced = estimate_request_savings_usd(
|
||||
model="claude-sonnet-4-20250514",
|
||||
model=MODEL,
|
||||
compression_tokens_saved=5_000,
|
||||
tool_schema_tokens_saved=5_000,
|
||||
)
|
||||
@@ -180,7 +187,7 @@ def test_the_headline_degrades_to_the_weakest_provider_in_a_mixed_session():
|
||||
not present its total as catalog-grade.
|
||||
"""
|
||||
priced = estimate_request_savings_usd(
|
||||
model="claude-sonnet-4-20250514",
|
||||
model=MODEL,
|
||||
# Priced against a real mix...
|
||||
compression_tokens_saved=5_000,
|
||||
cache_read_tokens=100_000,
|
||||
@@ -193,7 +200,7 @@ def test_the_headline_degrades_to_the_weakest_provider_in_a_mixed_session():
|
||||
|
||||
# ...but the same layers with nothing reported cannot claim the same.
|
||||
blind = estimate_request_savings_usd(
|
||||
model="claude-sonnet-4-20250514",
|
||||
model=MODEL,
|
||||
compression_tokens_saved=5_000,
|
||||
)
|
||||
assert blind["basis"] == "no-mix"
|
||||
@@ -206,7 +213,7 @@ def test_long_context_requests_price_at_the_above_200k_tier():
|
||||
request's savings at base rates understates them by half.
|
||||
"""
|
||||
base = estimate_request_savings_usd(
|
||||
model="claude-sonnet-4-20250514",
|
||||
model=MODEL,
|
||||
compression_tokens_saved=5_000,
|
||||
cache_write_tokens=50_000,
|
||||
cache_write_5m_tokens=50_000,
|
||||
@@ -214,7 +221,7 @@ def test_long_context_requests_price_at_the_above_200k_tier():
|
||||
provider="anthropic",
|
||||
)
|
||||
long_ctx = estimate_request_savings_usd(
|
||||
model="claude-sonnet-4-20250514",
|
||||
model=MODEL,
|
||||
compression_tokens_saved=5_000,
|
||||
cache_write_tokens=250_000,
|
||||
cache_write_5m_tokens=250_000,
|
||||
|
||||
Reference in New Issue
Block a user