mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description DeepSeek restructured its model line-up and price card, and Headroom's DeepSeek data was last verified 2026-06-19: the rows still carried the pre-restructure flat rates, `deepseek-flash` was unknown, and nothing could express the vendor's peak/off-peak split. Source (fetched and re-verified today): https://api-docs.deepseek.com/quick_start/pricing | | `deepseek-flash` (V4.1-Flash) | `deepseek-v4-pro` (V4-Pro-0813) | |---|---|---| | Context / max output | 1M / 384K | 1M / 384K | | Input (cache hit) | $0.003 off-peak / $0.006 peak | $0.022 / $0.044 | | Input (cache miss) | $0.15 / $0.30 | $0.66 / $1.32 | | Output | $0.60 / $1.20 | $1.98 / $3.96 | | Vision | yes | no | `deepseek-v4-flash` and `deepseek-v4-flash-vision-exp` are retired ids still accepted on the wire — served by V4.1-Flash and billed at the Flash price. Peak is exactly 2x off-peak; peak hours are 01:00-04:00 and 06:00-10:00 UTC, Monday-Friday. Context note: this surfaced while working on #3036, which routes DeepSeek traffic through the proxy and hard-codes the `deepseek-v4-*` ids in its routing tests. That branch is not touched here, and no issue is closed by this PR — it is a rate-card and model-data refresh. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Added `headroom/pricing/deepseek_tiers.py` as the single source of truth: the off-peak `(cache-hit, cache-miss, output)` table for `deepseek-flash` and `deepseek-v4-pro`, peak derived as exactly 2x, the retired-id aliases, the Beijing peak-window predicate with the weekend rule gated on its effective instant, and the `rates_for(model, now=None)` seam. - Made two cost seams tier-aware: `estimate_cost_from_tokens(..., now=None)` prices DeepSeek flash/pro from the tier selected by the request instant (litellm's `model_cost` holds one flat rate per model and cannot express a tier), and `PricingRegistry.estimate_cost(..., now=None)` prices them through its existing body from a synthesized rate row so there is one arithmetic body rather than two. - Gave `AnthropicProvider.estimate_cost` the same `now` passthrough and derived its DeepSeek fallback table from the tier module. - Derived every flat table from the tier module so no DeepSeek rate literal is left outside it: the `deepseek_prices` rows, the `litellm.model_cost` injection (now overriding and merging upstream rather than filling absent keys), and the Anthropic fallback table. Moved `LAST_UPDATED` / `SOURCE_URL` onto the module that owns the rates and re-exported them so existing import paths keep working. - Updated the model data: `deepseek-flash` registered with vision and both retired ids as aliases, provider context rows, the Hugging Face tokenizer map, the vendored LiteLLM snapshot, and the OpenCode guide's published rate card. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_providers tests/test_pricing.py tests/test_pricing_deepseek_tiers.py \ tests/test_pricing_from_litellm.py tests/test_pricing_litellm.py \ tests/test_pricing_litellm_model_resolution.py tests/test_models.py \ tests/test_huggingface_tokenizer_timeout.py tests/test_tokenizer_count_offload.py \ tests/test_cost_pricing_warning_dedup.py -q 291 passed, 3 warnings in 9.80s $ uv run ruff check . All checks passed! $ uv run mypy headroom headroom\integrations\langchain\chat_model.py:394: error: Function "RunnableBinding" could always be true in boolean context [truthy-function] Found 1 error in 1 file (checked 532 source files) $ cargo test -p headroom-proxy --lib compression::model_limits running 6 tests test compression::model_limits::tests::vendored_json_parses_at_runtime ... ok test compression::model_limits::tests::unknown_model_returns_default ... ok test compression::model_limits::tests::sample_spec_entry_is_excluded ... ok test compression::model_limits::tests::current_claude_models_present ... ok test compression::model_limits::tests::current_gpt_models_present ... ok test compression::model_limits::tests::empty_or_garbage_string_does_not_panic ... ok test result: ok. 6 passed; 0 failed $ uv run pytest tests/test_pricing_deepseek_tiers.py -q # with litellm hidden 47 passed $ uv run mypy headroom/pricing/deepseek_tiers.py headroom/pricing/registry.py \ headroom/pricing/litellm_pricing.py headroom/providers/anthropic.py Success: no issues found in 4 source files ``` The `mypy headroom` box is unticked on purpose: the single error above is pre-existing on `main` in `headroom/integrations/langchain/chat_model.py:394`, a file this PR does not touch (see Additional Notes). ## Real Behavior Proof - Environment: Windows 10, Python 3.13 in the repo's `.venv`, `uv`; litellm 1.88.1 installed (whose bundled cost map predates these ids, so the injected rows are what price them). - Exact command / steps: pinned instants Monday 2026-08-17 02:00 UTC (10:00 Beijing = peak) and 12:00 UTC (20:00 Beijing = off-peak), then: `uv run python -c "from datetime import datetime,timezone; from headroom.pricing.litellm_pricing import estimate_cost_from_tokens as e; P=datetime(2026,8,17,2,tzinfo=timezone.utc); O=datetime(2026,8,17,12,tzinfo=timezone.utc); print(e('deepseek-flash',10**6,10**6,now=O), e('deepseek-flash',10**6,10**6,now=P), e('deepseek-flash',10**6,0,cached_tokens=10**6,now=O), e('deepseek-v4-flash-vision-exp',10**6,10**6,now=O))"` - Observed result: `0.75` (off-peak flash), `1.5` (peak flash, exactly 2x), `0.003` (fully cached prompt billed once at the cache-hit rate), `0.75` (retired id prices as Flash). Also verified: 9 mutants of the new logic (peak multiplier, weekend gate, window start, alias resolution, cached-token handling, registry branch, provider passthrough) were each caught by the new tests, and `tests/test_pricing_deepseek_tiers.py` still passes with litellm absent (47 passed). - Not tested: the full test suite (hours on this box, and much of it cannot build a `TestClient` here because `NO_PROXY` contains a bare `::1`, so `httpx.URLPattern` raises `InvalidURL: Invalid port: ':'` — a pre-existing environment fault, confirmed by stashing this branch and re-running); CNY billing; the proxy budget guard and savings ledger, which remain flat by design (see Additional Notes). ## Runtime Rollout Safety - Rollout-managed feature(s): DeepSeek pricing and model data consumed by the pricing modules, providers, model registry and the Rust proxy's context limits. No feature flags are involved. - Minimum rollout channel: Stable. - Stable/default behavior changed: Yes — DeepSeek costs, model ids and capabilities change; the litellm `model_cost` DeepSeek rows now override upstream; and one context-limit fallback now resolves longest-prefix-first. All are itemised in Additional Notes. - Kill switch / disable path: revert this branch (no flag). Operators can override context limits without a release via `HEADROOM_MODEL_LIMITS` or `~/.headroom/models.json`, and provider pricing via the existing custom pricing config. - Unsafe override required: No. - Qualification impact: None — no schema, wire-format or session-format change; the vendored JSON keeps its upstream shape. - Rollback path: revert the merge commit; the only persisted artifact change is the vendored JSON, which reverts with it. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Screenshots (if applicable) Not applicable — no UI change. ## Additional Notes **Behaviour changes worth a reviewer's attention** 1. Flat figures are the off-peak tier by design; only the two cost seams are tier-aware. The proxy budget guard (`proxy/cost.py`), the savings ledger, `ModelRegistry.estimate_cost` and `OpenAICompatibleProvider.estimate_cost` remain flat or unpriced, so a peak-hour DeepSeek request can be under-estimated by up to 2x there. Making those tier-aware needs the request instant plumbed to each call site — deliberately a separate change. 2. The litellm DeepSeek rows now override upstream instead of filling only absent keys: upstream ships `deepseek-flash` at the *peak* rate, so fill-if-absent would have made the flat figure depend on the installed litellm version. The injection also merges (`{**upstream, **ours}`) so capability flags survive. 3. DeepSeek caps moved `384_000` -> `393_216` in the litellm-shaped tables (upstream's 384x1024 encoding of the published 384K). `ModelInfo` keeps the published `384_000`; the litellm-shaped tables are authoritative wherever litellm is consulted. 4. `OpenAICompatibleProvider`'s context fallback now matches the longest prefix first, mirroring the long-standing behaviour in `OpenAIProvider`. This fixes `deepseek-v4-flash-vision-exp` (was 1_048_576 via the shorter `deepseek-v4` key, now 1_000_000) and also changes the window for unlisted non-DeepSeek variants, e.g. `llama-3.1-8b-instruct` 8192 -> 128000, `mistral-large-2407` 32768 -> 128000, `starcoder2-15b` 8192 -> 16384. 5. `PricingRegistry.estimate_cost` no longer raises for a DeepSeek flash/pro id absent from `prices` — the tier table is authoritative for those four ids. Unknown non-DeepSeek ids still raise, and a flat row for a tiered id is deliberately ignored. 6. Weekend off-peak applies only from 2026-08-22T16:00Z, so a back-dated or replayed weekend request before that instant keeps the weekday windows rather than being silently repriced. `now` exists only on `AnthropicProvider.estimate_cost`; every other provider reads the wall clock for live traffic. 7. CNY billing is not modelled. The zh-cn page lists the same tiers in CNY, but the two lists are not one FX rate apart; supporting CNY-billed accounts needs a switch over two published lists, not a baked constant. **N/A / notes on checklist items** - `Type checking passes (mypy headroom)` is unticked for the pre-existing error above. `git diff --name-only origin/main..HEAD` contains no `integrations/langchain` path, so this PR does not introduce or touch it; `mypy` is clean on all four changed modules. - Out of scope: the stale `deepseek-v3` / `deepseek-coder` registry rows and the old `deepseek-chat` / `deepseek-reasoner` / `deepseek-v3.2` context rows (the vendor no longer lists them; pruning them is a behavioural change of its own). - The design and implementation plan for this change live in a separate workspace, so the decisions they record are inlined above rather than linked. --------- Co-authored-by: Zenjibad <316719033+Zenjibad@users.noreply.github.com> Co-authored-by: DSH Agent <agent@local>
144 lines
5.4 KiB
Python
144 lines
5.4 KiB
Python
from __future__ import annotations
|
|
|
|
from dataclasses import FrozenInstanceError
|
|
from datetime import date, timedelta
|
|
|
|
import pytest
|
|
|
|
import headroom.pricing as pricing
|
|
from headroom.pricing.anthropic_prices import ANTHROPIC_PRICES, get_anthropic_registry
|
|
from headroom.pricing.openai_prices import OPENAI_PRICES, get_openai_registry
|
|
from headroom.pricing.registry import ModelPricing, PricingRegistry
|
|
|
|
|
|
def test_pricing_public_exports_and_provider_registries() -> None:
|
|
assert pricing.ModelPricing is ModelPricing
|
|
assert pricing.PricingRegistry is PricingRegistry
|
|
assert "get_openai_registry" in pricing.__all__
|
|
assert "get_anthropic_registry" in pricing.__all__
|
|
assert "estimate_cost" in pricing.__all__
|
|
|
|
openai_registry = get_openai_registry()
|
|
anthropic_registry = get_anthropic_registry()
|
|
assert openai_registry.source_url == "https://openai.com/api/pricing/"
|
|
assert anthropic_registry.source_url == "https://www.anthropic.com/pricing"
|
|
assert openai_registry.prices["gpt-4o"] == OPENAI_PRICES["gpt-4o"]
|
|
assert (
|
|
anthropic_registry.prices["claude-3-5-sonnet-20241022"]
|
|
== ANTHROPIC_PRICES["claude-3-5-sonnet-20241022"]
|
|
)
|
|
|
|
assert "get_deepseek_registry" in pricing.__all__
|
|
assert "DEEPSEEK_PRICES" in pricing.__all__
|
|
assert "OFF_PEAK_RATES_PER_1M" in pricing.__all__
|
|
assert "DeepSeekRates" in pricing.__all__
|
|
assert "off_peak_rates" in pricing.__all__
|
|
assert "rates_for" in pricing.__all__
|
|
deepseek_registry = pricing.get_deepseek_registry()
|
|
assert deepseek_registry.source_url == "https://api-docs.deepseek.com/quick_start/pricing"
|
|
flash = deepseek_registry.get_price("deepseek-v4-flash")
|
|
assert flash is not None
|
|
assert flash.input_per_1m == 0.15
|
|
assert flash.output_per_1m == 0.60
|
|
|
|
openai_registry.prices.pop("gpt-4o")
|
|
anthropic_registry.prices.pop("claude-3-5-sonnet-20241022")
|
|
assert "gpt-4o" in OPENAI_PRICES
|
|
assert "claude-3-5-sonnet-20241022" in ANTHROPIC_PRICES
|
|
|
|
|
|
def test_model_pricing_is_frozen() -> None:
|
|
model = ModelPricing(model="demo", provider="test", input_per_1m=1.5, output_per_1m=2.5)
|
|
with pytest.raises(FrozenInstanceError):
|
|
model.model = "other" # type: ignore[misc]
|
|
|
|
|
|
def test_registry_staleness_and_warning() -> None:
|
|
fresh = PricingRegistry(last_updated=date.today() - timedelta(days=30))
|
|
assert fresh.is_stale() is False
|
|
assert fresh.staleness_warning() is None
|
|
|
|
stale = PricingRegistry(
|
|
last_updated=date.today() - timedelta(days=31),
|
|
source_url="https://example.test/pricing",
|
|
)
|
|
assert stale.is_stale() is True
|
|
assert stale.staleness_warning() == (
|
|
f"Pricing data is 31 days old (last updated: {stale.last_updated})."
|
|
" Please verify at: https://example.test/pricing"
|
|
)
|
|
|
|
|
|
def test_registry_estimate_cost_with_all_token_types() -> None:
|
|
registry = PricingRegistry(
|
|
last_updated=date.today() - timedelta(days=31),
|
|
prices={
|
|
"demo": ModelPricing(
|
|
model="demo",
|
|
provider="test",
|
|
input_per_1m=2.0,
|
|
output_per_1m=4.0,
|
|
cached_input_per_1m=1.0,
|
|
batch_input_per_1m=0.5,
|
|
batch_output_per_1m=0.25,
|
|
)
|
|
},
|
|
)
|
|
|
|
estimate = registry.estimate_cost(
|
|
"demo",
|
|
input_tokens=1_000_000,
|
|
output_tokens=500_000,
|
|
cached_input_tokens=250_000,
|
|
batch_input_tokens=200_000,
|
|
batch_output_tokens=100_000,
|
|
)
|
|
|
|
assert estimate.cost_usd == pytest.approx(4.375)
|
|
assert estimate.breakdown == {
|
|
"input": {"tokens": 1_000_000, "rate_per_1m": 2.0, "cost_usd": 2.0},
|
|
"output": {"tokens": 500_000, "rate_per_1m": 4.0, "cost_usd": 2.0},
|
|
"cached_input": {"tokens": 250_000, "rate_per_1m": 1.0, "cost_usd": 0.25},
|
|
"batch_input": {"tokens": 200_000, "rate_per_1m": 0.5, "cost_usd": 0.1},
|
|
"batch_output": {"tokens": 100_000, "rate_per_1m": 0.25, "cost_usd": 0.025},
|
|
}
|
|
assert estimate.pricing_date == registry.last_updated
|
|
assert estimate.is_stale is True
|
|
assert estimate.warning == (
|
|
f"Pricing data is 31 days old (last updated: {registry.last_updated})."
|
|
)
|
|
|
|
|
|
def test_registry_estimate_cost_zero_usage_returns_empty_breakdown() -> None:
|
|
registry = PricingRegistry(
|
|
last_updated=date.today(),
|
|
prices={
|
|
"demo": ModelPricing(model="demo", provider="test", input_per_1m=1.0, output_per_1m=2.0)
|
|
},
|
|
)
|
|
estimate = registry.estimate_cost("demo")
|
|
assert estimate.cost_usd == 0.0
|
|
assert estimate.breakdown == {}
|
|
assert estimate.is_stale is False
|
|
assert estimate.warning is None
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("kwargs", "message"),
|
|
[
|
|
({}, "Model 'missing' not found in registry"),
|
|
({"cached_input_tokens": 1}, "Model 'demo' does not have cached input pricing"),
|
|
({"batch_input_tokens": 1}, "Model 'demo' does not have batch input pricing"),
|
|
({"batch_output_tokens": 1}, "Model 'demo' does not have batch output pricing"),
|
|
],
|
|
)
|
|
def test_registry_estimate_cost_error_paths(kwargs: dict[str, int], message: str) -> None:
|
|
registry = PricingRegistry(
|
|
last_updated=date.today(),
|
|
prices={
|
|
"demo": ModelPricing(model="demo", provider="test", input_per_1m=1.0, output_per_1m=2.0)
|
|
},
|
|
)
|
|
with pytest.raises(ValueError, match=message):
|
|
registry.estimate_cost("missing" if not kwargs else "demo", **kwargs)
|