Files
headroom/tests/test_pricing.py
b0dfa33478 fix(pricing): DeepSeek V4.1-Flash peak/off-peak rate card (#3575)
## Description

DeepSeek restructured its model line-up and price card, and Headroom's
DeepSeek data was last verified 2026-06-19: the rows still carried the
pre-restructure flat rates, `deepseek-flash` was unknown, and nothing
could express the vendor's peak/off-peak split.

Source (fetched and re-verified today):
https://api-docs.deepseek.com/quick_start/pricing

| | `deepseek-flash` (V4.1-Flash) | `deepseek-v4-pro` (V4-Pro-0813) |
|---|---|---|
| Context / max output | 1M / 384K | 1M / 384K |
| Input (cache hit) | $0.003 off-peak / $0.006 peak | $0.022 / $0.044 |
| Input (cache miss) | $0.15 / $0.30 | $0.66 / $1.32 |
| Output | $0.60 / $1.20 | $1.98 / $3.96 |
| Vision | yes | no |

`deepseek-v4-flash` and `deepseek-v4-flash-vision-exp` are retired ids
still accepted on the wire — served by V4.1-Flash and billed at the
Flash price. Peak is exactly 2x off-peak; peak hours are 01:00-04:00 and
06:00-10:00 UTC, Monday-Friday.

Context note: this surfaced while working on #3036, which routes
DeepSeek traffic through the proxy and hard-codes the `deepseek-v4-*`
ids in its routing tests. That branch is not touched here, and no issue
is closed by this PR — it is a rate-card and model-data refresh.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Added `headroom/pricing/deepseek_tiers.py` as the single source of
truth: the off-peak `(cache-hit, cache-miss, output)` table for
`deepseek-flash` and `deepseek-v4-pro`, peak derived as exactly 2x, the
retired-id aliases, the Beijing peak-window predicate with the weekend
rule gated on its effective instant, and the `rates_for(model,
now=None)` seam.
- Made two cost seams tier-aware: `estimate_cost_from_tokens(...,
now=None)` prices DeepSeek flash/pro from the tier selected by the
request instant (litellm's `model_cost` holds one flat rate per model
and cannot express a tier), and `PricingRegistry.estimate_cost(...,
now=None)` prices them through its existing body from a synthesized rate
row so there is one arithmetic body rather than two.
- Gave `AnthropicProvider.estimate_cost` the same `now` passthrough and
derived its DeepSeek fallback table from the tier module.
- Derived every flat table from the tier module so no DeepSeek rate
literal is left outside it: the `deepseek_prices` rows, the
`litellm.model_cost` injection (now overriding and merging upstream
rather than filling absent keys), and the Anthropic fallback table.
Moved `LAST_UPDATED` / `SOURCE_URL` onto the module that owns the rates
and re-exported them so existing import paths keep working.
- Updated the model data: `deepseek-flash` registered with vision and
both retired ids as aliases, provider context rows, the Hugging Face
tokenizer map, the vendored LiteLLM snapshot, and the OpenCode guide's
published rate card.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_providers tests/test_pricing.py tests/test_pricing_deepseek_tiers.py \
    tests/test_pricing_from_litellm.py tests/test_pricing_litellm.py \
    tests/test_pricing_litellm_model_resolution.py tests/test_models.py \
    tests/test_huggingface_tokenizer_timeout.py tests/test_tokenizer_count_offload.py \
    tests/test_cost_pricing_warning_dedup.py -q
291 passed, 3 warnings in 9.80s

$ uv run ruff check .
All checks passed!

$ uv run mypy headroom
headroom\integrations\langchain\chat_model.py:394: error: Function "RunnableBinding" could
always be true in boolean context  [truthy-function]
Found 1 error in 1 file (checked 532 source files)

$ cargo test -p headroom-proxy --lib compression::model_limits
running 6 tests
test compression::model_limits::tests::vendored_json_parses_at_runtime ... ok
test compression::model_limits::tests::unknown_model_returns_default ... ok
test compression::model_limits::tests::sample_spec_entry_is_excluded ... ok
test compression::model_limits::tests::current_claude_models_present ... ok
test compression::model_limits::tests::current_gpt_models_present ... ok
test compression::model_limits::tests::empty_or_garbage_string_does_not_panic ... ok
test result: ok. 6 passed; 0 failed

$ uv run pytest tests/test_pricing_deepseek_tiers.py -q   # with litellm hidden
47 passed

$ uv run mypy headroom/pricing/deepseek_tiers.py headroom/pricing/registry.py \
    headroom/pricing/litellm_pricing.py headroom/providers/anthropic.py
Success: no issues found in 4 source files
```

The `mypy headroom` box is unticked on purpose: the single error above
is pre-existing on `main` in
`headroom/integrations/langchain/chat_model.py:394`, a file this PR does
not touch (see Additional Notes).

## Real Behavior Proof

- Environment: Windows 10, Python 3.13 in the repo's `.venv`, `uv`;
litellm 1.88.1 installed (whose bundled cost map predates these ids, so
the injected rows are what price them).
- Exact command / steps: pinned instants Monday 2026-08-17 02:00 UTC
(10:00 Beijing = peak) and 12:00 UTC (20:00 Beijing = off-peak), then:
`uv run python -c "from datetime import datetime,timezone; from
headroom.pricing.litellm_pricing import estimate_cost_from_tokens as e;
P=datetime(2026,8,17,2,tzinfo=timezone.utc);
O=datetime(2026,8,17,12,tzinfo=timezone.utc);
print(e('deepseek-flash',10**6,10**6,now=O),
e('deepseek-flash',10**6,10**6,now=P),
e('deepseek-flash',10**6,0,cached_tokens=10**6,now=O),
e('deepseek-v4-flash-vision-exp',10**6,10**6,now=O))"`
- Observed result: `0.75` (off-peak flash), `1.5` (peak flash, exactly
2x), `0.003` (fully cached prompt billed once at the cache-hit rate),
`0.75` (retired id prices as Flash). Also verified: 9 mutants of the new
logic (peak multiplier, weekend gate, window start, alias resolution,
cached-token handling, registry branch, provider passthrough) were each
caught by the new tests, and `tests/test_pricing_deepseek_tiers.py`
still passes with litellm absent (47 passed).
- Not tested: the full test suite (hours on this box, and much of it
cannot build a `TestClient` here because `NO_PROXY` contains a bare
`::1`, so `httpx.URLPattern` raises `InvalidURL: Invalid port: ':'` — a
pre-existing environment fault, confirmed by stashing this branch and
re-running); CNY billing; the proxy budget guard and savings ledger,
which remain flat by design (see Additional Notes).

## Runtime Rollout Safety

- Rollout-managed feature(s): DeepSeek pricing and model data consumed
by the pricing modules, providers, model registry and the Rust proxy's
context limits. No feature flags are involved.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes — DeepSeek costs, model ids and
capabilities change; the litellm `model_cost` DeepSeek rows now override
upstream; and one context-limit fallback now resolves
longest-prefix-first. All are itemised in Additional Notes.
- Kill switch / disable path: revert this branch (no flag). Operators
can override context limits without a release via
`HEADROOM_MODEL_LIMITS` or `~/.headroom/models.json`, and provider
pricing via the existing custom pricing config.
- Unsafe override required: No.
- Qualification impact: None — no schema, wire-format or session-format
change; the vendored JSON keeps its upstream shape.
- Rollback path: revert the merge commit; the only persisted artifact
change is the vendored JSON, which reverts with it.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Screenshots (if applicable)

Not applicable — no UI change.

## Additional Notes

**Behaviour changes worth a reviewer's attention**

1. Flat figures are the off-peak tier by design; only the two cost seams
are tier-aware. The proxy budget guard (`proxy/cost.py`), the savings
ledger, `ModelRegistry.estimate_cost` and
`OpenAICompatibleProvider.estimate_cost` remain flat or unpriced, so a
peak-hour DeepSeek request can be under-estimated by up to 2x there.
Making those tier-aware needs the request instant plumbed to each call
site — deliberately a separate change.
2. The litellm DeepSeek rows now override upstream instead of filling
only absent keys: upstream ships `deepseek-flash` at the *peak* rate, so
fill-if-absent would have made the flat figure depend on the installed
litellm version. The injection also merges (`{**upstream, **ours}`) so
capability flags survive.
3. DeepSeek caps moved `384_000` -> `393_216` in the litellm-shaped
tables (upstream's 384x1024 encoding of the published 384K). `ModelInfo`
keeps the published `384_000`; the litellm-shaped tables are
authoritative wherever litellm is consulted.
4. `OpenAICompatibleProvider`'s context fallback now matches the longest
prefix first, mirroring the long-standing behaviour in `OpenAIProvider`.
This fixes `deepseek-v4-flash-vision-exp` (was 1_048_576 via the shorter
`deepseek-v4` key, now 1_000_000) and also changes the window for
unlisted non-DeepSeek variants, e.g. `llama-3.1-8b-instruct` 8192 ->
128000, `mistral-large-2407` 32768 -> 128000, `starcoder2-15b` 8192 ->
16384.
5. `PricingRegistry.estimate_cost` no longer raises for a DeepSeek
flash/pro id absent from `prices` — the tier table is authoritative for
those four ids. Unknown non-DeepSeek ids still raise, and a flat row for
a tiered id is deliberately ignored.
6. Weekend off-peak applies only from 2026-08-22T16:00Z, so a back-dated
or replayed weekend request before that instant keeps the weekday
windows rather than being silently repriced. `now` exists only on
`AnthropicProvider.estimate_cost`; every other provider reads the wall
clock for live traffic.
7. CNY billing is not modelled. The zh-cn page lists the same tiers in
CNY, but the two lists are not one FX rate apart; supporting CNY-billed
accounts needs a switch over two published lists, not a baked constant.

**N/A / notes on checklist items**

- `Type checking passes (mypy headroom)` is unticked for the
pre-existing error above. `git diff --name-only origin/main..HEAD`
contains no `integrations/langchain` path, so this PR does not introduce
or touch it; `mypy` is clean on all four changed modules.
- Out of scope: the stale `deepseek-v3` / `deepseek-coder` registry rows
and the old `deepseek-chat` / `deepseek-reasoner` / `deepseek-v3.2`
context rows (the vendor no longer lists them; pruning them is a
behavioural change of its own).
- The design and implementation plan for this change live in a separate
workspace, so the decisions they record are inlined above rather than
linked.

---------

Co-authored-by: Zenjibad <316719033+Zenjibad@users.noreply.github.com>
Co-authored-by: DSH Agent <agent@local>
2026-09-24 11:31:44 -05:00

144 lines
5.4 KiB
Python

from __future__ import annotations
from dataclasses import FrozenInstanceError
from datetime import date, timedelta
import pytest
import headroom.pricing as pricing
from headroom.pricing.anthropic_prices import ANTHROPIC_PRICES, get_anthropic_registry
from headroom.pricing.openai_prices import OPENAI_PRICES, get_openai_registry
from headroom.pricing.registry import ModelPricing, PricingRegistry
def test_pricing_public_exports_and_provider_registries() -> None:
assert pricing.ModelPricing is ModelPricing
assert pricing.PricingRegistry is PricingRegistry
assert "get_openai_registry" in pricing.__all__
assert "get_anthropic_registry" in pricing.__all__
assert "estimate_cost" in pricing.__all__
openai_registry = get_openai_registry()
anthropic_registry = get_anthropic_registry()
assert openai_registry.source_url == "https://openai.com/api/pricing/"
assert anthropic_registry.source_url == "https://www.anthropic.com/pricing"
assert openai_registry.prices["gpt-4o"] == OPENAI_PRICES["gpt-4o"]
assert (
anthropic_registry.prices["claude-3-5-sonnet-20241022"]
== ANTHROPIC_PRICES["claude-3-5-sonnet-20241022"]
)
assert "get_deepseek_registry" in pricing.__all__
assert "DEEPSEEK_PRICES" in pricing.__all__
assert "OFF_PEAK_RATES_PER_1M" in pricing.__all__
assert "DeepSeekRates" in pricing.__all__
assert "off_peak_rates" in pricing.__all__
assert "rates_for" in pricing.__all__
deepseek_registry = pricing.get_deepseek_registry()
assert deepseek_registry.source_url == "https://api-docs.deepseek.com/quick_start/pricing"
flash = deepseek_registry.get_price("deepseek-v4-flash")
assert flash is not None
assert flash.input_per_1m == 0.15
assert flash.output_per_1m == 0.60
openai_registry.prices.pop("gpt-4o")
anthropic_registry.prices.pop("claude-3-5-sonnet-20241022")
assert "gpt-4o" in OPENAI_PRICES
assert "claude-3-5-sonnet-20241022" in ANTHROPIC_PRICES
def test_model_pricing_is_frozen() -> None:
model = ModelPricing(model="demo", provider="test", input_per_1m=1.5, output_per_1m=2.5)
with pytest.raises(FrozenInstanceError):
model.model = "other" # type: ignore[misc]
def test_registry_staleness_and_warning() -> None:
fresh = PricingRegistry(last_updated=date.today() - timedelta(days=30))
assert fresh.is_stale() is False
assert fresh.staleness_warning() is None
stale = PricingRegistry(
last_updated=date.today() - timedelta(days=31),
source_url="https://example.test/pricing",
)
assert stale.is_stale() is True
assert stale.staleness_warning() == (
f"Pricing data is 31 days old (last updated: {stale.last_updated})."
" Please verify at: https://example.test/pricing"
)
def test_registry_estimate_cost_with_all_token_types() -> None:
registry = PricingRegistry(
last_updated=date.today() - timedelta(days=31),
prices={
"demo": ModelPricing(
model="demo",
provider="test",
input_per_1m=2.0,
output_per_1m=4.0,
cached_input_per_1m=1.0,
batch_input_per_1m=0.5,
batch_output_per_1m=0.25,
)
},
)
estimate = registry.estimate_cost(
"demo",
input_tokens=1_000_000,
output_tokens=500_000,
cached_input_tokens=250_000,
batch_input_tokens=200_000,
batch_output_tokens=100_000,
)
assert estimate.cost_usd == pytest.approx(4.375)
assert estimate.breakdown == {
"input": {"tokens": 1_000_000, "rate_per_1m": 2.0, "cost_usd": 2.0},
"output": {"tokens": 500_000, "rate_per_1m": 4.0, "cost_usd": 2.0},
"cached_input": {"tokens": 250_000, "rate_per_1m": 1.0, "cost_usd": 0.25},
"batch_input": {"tokens": 200_000, "rate_per_1m": 0.5, "cost_usd": 0.1},
"batch_output": {"tokens": 100_000, "rate_per_1m": 0.25, "cost_usd": 0.025},
}
assert estimate.pricing_date == registry.last_updated
assert estimate.is_stale is True
assert estimate.warning == (
f"Pricing data is 31 days old (last updated: {registry.last_updated})."
)
def test_registry_estimate_cost_zero_usage_returns_empty_breakdown() -> None:
registry = PricingRegistry(
last_updated=date.today(),
prices={
"demo": ModelPricing(model="demo", provider="test", input_per_1m=1.0, output_per_1m=2.0)
},
)
estimate = registry.estimate_cost("demo")
assert estimate.cost_usd == 0.0
assert estimate.breakdown == {}
assert estimate.is_stale is False
assert estimate.warning is None
@pytest.mark.parametrize(
("kwargs", "message"),
[
({}, "Model 'missing' not found in registry"),
({"cached_input_tokens": 1}, "Model 'demo' does not have cached input pricing"),
({"batch_input_tokens": 1}, "Model 'demo' does not have batch input pricing"),
({"batch_output_tokens": 1}, "Model 'demo' does not have batch output pricing"),
],
)
def test_registry_estimate_cost_error_paths(kwargs: dict[str, int], message: str) -> None:
registry = PricingRegistry(
last_updated=date.today(),
prices={
"demo": ModelPricing(model="demo", provider="test", input_per_1m=1.0, output_per_1m=2.0)
},
)
with pytest.raises(ValueError, match=message):
registry.estimate_cost("missing" if not kwargs else "demo", **kwargs)