Files
headroom/tests/test_models.py
b0dfa33478 fix(pricing): DeepSeek V4.1-Flash peak/off-peak rate card (#3575)
## Description

DeepSeek restructured its model line-up and price card, and Headroom's
DeepSeek data was last verified 2026-06-19: the rows still carried the
pre-restructure flat rates, `deepseek-flash` was unknown, and nothing
could express the vendor's peak/off-peak split.

Source (fetched and re-verified today):
https://api-docs.deepseek.com/quick_start/pricing

| | `deepseek-flash` (V4.1-Flash) | `deepseek-v4-pro` (V4-Pro-0813) |
|---|---|---|
| Context / max output | 1M / 384K | 1M / 384K |
| Input (cache hit) | $0.003 off-peak / $0.006 peak | $0.022 / $0.044 |
| Input (cache miss) | $0.15 / $0.30 | $0.66 / $1.32 |
| Output | $0.60 / $1.20 | $1.98 / $3.96 |
| Vision | yes | no |

`deepseek-v4-flash` and `deepseek-v4-flash-vision-exp` are retired ids
still accepted on the wire — served by V4.1-Flash and billed at the
Flash price. Peak is exactly 2x off-peak; peak hours are 01:00-04:00 and
06:00-10:00 UTC, Monday-Friday.

Context note: this surfaced while working on #3036, which routes
DeepSeek traffic through the proxy and hard-codes the `deepseek-v4-*`
ids in its routing tests. That branch is not touched here, and no issue
is closed by this PR — it is a rate-card and model-data refresh.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Added `headroom/pricing/deepseek_tiers.py` as the single source of
truth: the off-peak `(cache-hit, cache-miss, output)` table for
`deepseek-flash` and `deepseek-v4-pro`, peak derived as exactly 2x, the
retired-id aliases, the Beijing peak-window predicate with the weekend
rule gated on its effective instant, and the `rates_for(model,
now=None)` seam.
- Made two cost seams tier-aware: `estimate_cost_from_tokens(...,
now=None)` prices DeepSeek flash/pro from the tier selected by the
request instant (litellm's `model_cost` holds one flat rate per model
and cannot express a tier), and `PricingRegistry.estimate_cost(...,
now=None)` prices them through its existing body from a synthesized rate
row so there is one arithmetic body rather than two.
- Gave `AnthropicProvider.estimate_cost` the same `now` passthrough and
derived its DeepSeek fallback table from the tier module.
- Derived every flat table from the tier module so no DeepSeek rate
literal is left outside it: the `deepseek_prices` rows, the
`litellm.model_cost` injection (now overriding and merging upstream
rather than filling absent keys), and the Anthropic fallback table.
Moved `LAST_UPDATED` / `SOURCE_URL` onto the module that owns the rates
and re-exported them so existing import paths keep working.
- Updated the model data: `deepseek-flash` registered with vision and
both retired ids as aliases, provider context rows, the Hugging Face
tokenizer map, the vendored LiteLLM snapshot, and the OpenCode guide's
published rate card.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_providers tests/test_pricing.py tests/test_pricing_deepseek_tiers.py \
    tests/test_pricing_from_litellm.py tests/test_pricing_litellm.py \
    tests/test_pricing_litellm_model_resolution.py tests/test_models.py \
    tests/test_huggingface_tokenizer_timeout.py tests/test_tokenizer_count_offload.py \
    tests/test_cost_pricing_warning_dedup.py -q
291 passed, 3 warnings in 9.80s

$ uv run ruff check .
All checks passed!

$ uv run mypy headroom
headroom\integrations\langchain\chat_model.py:394: error: Function "RunnableBinding" could
always be true in boolean context  [truthy-function]
Found 1 error in 1 file (checked 532 source files)

$ cargo test -p headroom-proxy --lib compression::model_limits
running 6 tests
test compression::model_limits::tests::vendored_json_parses_at_runtime ... ok
test compression::model_limits::tests::unknown_model_returns_default ... ok
test compression::model_limits::tests::sample_spec_entry_is_excluded ... ok
test compression::model_limits::tests::current_claude_models_present ... ok
test compression::model_limits::tests::current_gpt_models_present ... ok
test compression::model_limits::tests::empty_or_garbage_string_does_not_panic ... ok
test result: ok. 6 passed; 0 failed

$ uv run pytest tests/test_pricing_deepseek_tiers.py -q   # with litellm hidden
47 passed

$ uv run mypy headroom/pricing/deepseek_tiers.py headroom/pricing/registry.py \
    headroom/pricing/litellm_pricing.py headroom/providers/anthropic.py
Success: no issues found in 4 source files
```

The `mypy headroom` box is unticked on purpose: the single error above
is pre-existing on `main` in
`headroom/integrations/langchain/chat_model.py:394`, a file this PR does
not touch (see Additional Notes).

## Real Behavior Proof

- Environment: Windows 10, Python 3.13 in the repo's `.venv`, `uv`;
litellm 1.88.1 installed (whose bundled cost map predates these ids, so
the injected rows are what price them).
- Exact command / steps: pinned instants Monday 2026-08-17 02:00 UTC
(10:00 Beijing = peak) and 12:00 UTC (20:00 Beijing = off-peak), then:
`uv run python -c "from datetime import datetime,timezone; from
headroom.pricing.litellm_pricing import estimate_cost_from_tokens as e;
P=datetime(2026,8,17,2,tzinfo=timezone.utc);
O=datetime(2026,8,17,12,tzinfo=timezone.utc);
print(e('deepseek-flash',10**6,10**6,now=O),
e('deepseek-flash',10**6,10**6,now=P),
e('deepseek-flash',10**6,0,cached_tokens=10**6,now=O),
e('deepseek-v4-flash-vision-exp',10**6,10**6,now=O))"`
- Observed result: `0.75` (off-peak flash), `1.5` (peak flash, exactly
2x), `0.003` (fully cached prompt billed once at the cache-hit rate),
`0.75` (retired id prices as Flash). Also verified: 9 mutants of the new
logic (peak multiplier, weekend gate, window start, alias resolution,
cached-token handling, registry branch, provider passthrough) were each
caught by the new tests, and `tests/test_pricing_deepseek_tiers.py`
still passes with litellm absent (47 passed).
- Not tested: the full test suite (hours on this box, and much of it
cannot build a `TestClient` here because `NO_PROXY` contains a bare
`::1`, so `httpx.URLPattern` raises `InvalidURL: Invalid port: ':'` — a
pre-existing environment fault, confirmed by stashing this branch and
re-running); CNY billing; the proxy budget guard and savings ledger,
which remain flat by design (see Additional Notes).

## Runtime Rollout Safety

- Rollout-managed feature(s): DeepSeek pricing and model data consumed
by the pricing modules, providers, model registry and the Rust proxy's
context limits. No feature flags are involved.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes — DeepSeek costs, model ids and
capabilities change; the litellm `model_cost` DeepSeek rows now override
upstream; and one context-limit fallback now resolves
longest-prefix-first. All are itemised in Additional Notes.
- Kill switch / disable path: revert this branch (no flag). Operators
can override context limits without a release via
`HEADROOM_MODEL_LIMITS` or `~/.headroom/models.json`, and provider
pricing via the existing custom pricing config.
- Unsafe override required: No.
- Qualification impact: None — no schema, wire-format or session-format
change; the vendored JSON keeps its upstream shape.
- Rollback path: revert the merge commit; the only persisted artifact
change is the vendored JSON, which reverts with it.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Screenshots (if applicable)

Not applicable — no UI change.

## Additional Notes

**Behaviour changes worth a reviewer's attention**

1. Flat figures are the off-peak tier by design; only the two cost seams
are tier-aware. The proxy budget guard (`proxy/cost.py`), the savings
ledger, `ModelRegistry.estimate_cost` and
`OpenAICompatibleProvider.estimate_cost` remain flat or unpriced, so a
peak-hour DeepSeek request can be under-estimated by up to 2x there.
Making those tier-aware needs the request instant plumbed to each call
site — deliberately a separate change.
2. The litellm DeepSeek rows now override upstream instead of filling
only absent keys: upstream ships `deepseek-flash` at the *peak* rate, so
fill-if-absent would have made the flat figure depend on the installed
litellm version. The injection also merges (`{**upstream, **ours}`) so
capability flags survive.
3. DeepSeek caps moved `384_000` -> `393_216` in the litellm-shaped
tables (upstream's 384x1024 encoding of the published 384K). `ModelInfo`
keeps the published `384_000`; the litellm-shaped tables are
authoritative wherever litellm is consulted.
4. `OpenAICompatibleProvider`'s context fallback now matches the longest
prefix first, mirroring the long-standing behaviour in `OpenAIProvider`.
This fixes `deepseek-v4-flash-vision-exp` (was 1_048_576 via the shorter
`deepseek-v4` key, now 1_000_000) and also changes the window for
unlisted non-DeepSeek variants, e.g. `llama-3.1-8b-instruct` 8192 ->
128000, `mistral-large-2407` 32768 -> 128000, `starcoder2-15b` 8192 ->
16384.
5. `PricingRegistry.estimate_cost` no longer raises for a DeepSeek
flash/pro id absent from `prices` — the tier table is authoritative for
those four ids. Unknown non-DeepSeek ids still raise, and a flat row for
a tiered id is deliberately ignored.
6. Weekend off-peak applies only from 2026-08-22T16:00Z, so a back-dated
or replayed weekend request before that instant keeps the weekday
windows rather than being silently repriced. `now` exists only on
`AnthropicProvider.estimate_cost`; every other provider reads the wall
clock for live traffic.
7. CNY billing is not modelled. The zh-cn page lists the same tiers in
CNY, but the two lists are not one FX rate apart; supporting CNY-billed
accounts needs a switch over two published lists, not a baked constant.

**N/A / notes on checklist items**

- `Type checking passes (mypy headroom)` is unticked for the
pre-existing error above. `git diff --name-only origin/main..HEAD`
contains no `integrations/langchain` path, so this PR does not introduce
or touch it; `mypy` is clean on all four changed modules.
- Out of scope: the stale `deepseek-v3` / `deepseek-coder` registry rows
and the old `deepseek-chat` / `deepseek-reasoner` / `deepseek-v3.2`
context rows (the vendor no longer lists them; pruning them is a
behavioural change of its own).
- The design and implementation plan for this change live in a separate
workspace, so the decisions they record are inlined above rather than
linked.

---------

Co-authored-by: Zenjibad <316719033+Zenjibad@users.noreply.github.com>
Co-authored-by: DSH Agent <agent@local>
2026-09-24 11:31:44 -05:00

341 lines
13 KiB
Python

"""Tests for the model registry and capabilities database."""
from __future__ import annotations
from unittest.mock import patch
import pytest
from headroom.models import (
ModelInfo,
ModelRegistry,
get_model_info,
list_models,
register_model,
)
from tests._pricing_models import anthropic_pricing_model
class TestModelInfo:
"""Tests for ModelInfo dataclass."""
def test_default_values(self):
"""Test default values."""
info = ModelInfo(name="test", provider="test-provider")
assert info.context_window == 128000
assert info.max_output_tokens == 4096
assert info.supports_tools is True
assert info.supports_vision is False
assert info.supports_streaming is True
def test_custom_values(self):
"""Test custom values."""
info = ModelInfo(
name="custom-model",
provider="custom",
context_window=32000,
max_output_tokens=8192,
supports_tools=False,
supports_vision=True,
)
assert info.context_window == 32000
assert info.max_output_tokens == 8192
assert info.supports_tools is False
assert info.supports_vision is True
def test_frozen(self):
"""Test that ModelInfo is frozen (immutable)."""
info = ModelInfo(name="test", provider="test")
with pytest.raises(AttributeError):
info.name = "changed"
class TestModelRegistry:
"""Tests for ModelRegistry."""
def test_get_openai_model(self):
"""Test getting OpenAI model info."""
info = ModelRegistry.get("gpt-4o")
assert info is not None
assert info.provider == "openai"
assert info.context_window == 128000
def test_get_anthropic_model(self):
"""Test getting Anthropic model info."""
info = ModelRegistry.get("claude-3-5-sonnet-20241022")
assert info is not None
assert info.provider == "anthropic"
assert info.context_window == 200000
def test_get_google_model(self):
"""Test getting Google model info."""
info = ModelRegistry.get("gemini-1.5-pro")
assert info is not None
assert info.provider == "google"
assert info.context_window == 2000000 # 2M!
def test_get_by_alias(self):
"""Test getting model by alias."""
info = ModelRegistry.get("gpt-4o-2024-11-20")
assert info is not None
assert info.name == "gpt-4o"
def test_get_unknown_model(self):
"""Test getting unknown model returns None."""
info = ModelRegistry.get("unknown-model-xyz")
assert info is None
def test_get_prefix_matching(self):
"""Test prefix matching for versioned models."""
info = ModelRegistry.get("gpt-4o-new-version")
assert info is not None
assert info.name == "gpt-4o"
def test_get_prefix_matching_prefers_longest_registered_name(self):
"""`gpt-4-32k-0613` must resolve to `gpt-4-32k` (32768), not the
shorter `gpt-4` (8192) that is registered first."""
info = ModelRegistry.get("gpt-4-32k-0613")
assert info is not None
assert info.name == "gpt-4-32k"
assert ModelRegistry.get_context_limit("gpt-4-32k-0613") == 32768
def test_get_prefix_matching_requires_version_boundary(self):
"""`gpt-4.1`/`gpt-4.5` are distinct models, not variants of `gpt-4`.
A `.`-separated suffix must not match `gpt-4`, so they no longer
inherit gpt-4's 8192-token window (they fall back to the default)."""
assert ModelRegistry.get("gpt-4.1") is None
assert ModelRegistry.get("gpt-4.5-preview") is None
# Not silently reported as an 8192-token model:
assert ModelRegistry.get_context_limit("gpt-4.1") != 8192
assert ModelRegistry.get_context_limit("gpt-4.1", default=100) == 100
def test_resolve_future_google_family_fallback(self):
"""Resolve should return provider-scoped fallbacks for plausible future models."""
with patch("headroom.models.registry.get_model_pricing", return_value=None):
info = ModelRegistry.resolve("gemini-3-pro-preview", provider="google")
assert info is not None
assert info.provider == "google"
assert info.context_window == 1000000
assert info.tokenizer_backend == "google"
def test_resolve_google_litellm_prefixed_family_fallback(self):
"""Resolve should support LiteLLM-style Gemini provider prefixes."""
with patch("headroom.models.registry.get_model_pricing", return_value=None):
info = ModelRegistry.resolve("gemini/gemini-3-pro-preview", provider="google")
assert info is not None
assert info.provider == "google"
assert info.context_window == 1000000
assert info.tokenizer_backend == "google"
def test_resolve_does_not_claim_unrelated_models_for_google(self):
"""Provider-scoped resolution should not mask unrelated model catalogs."""
assert ModelRegistry.resolve("not-a-google-model", provider="google") is None
assert ModelRegistry.resolve("gpt-4o", provider="google") is None
def test_register_custom_model(self):
"""Test registering custom model."""
info = ModelRegistry.register(
"my-custom-model",
provider="custom",
context_window=64000,
supports_vision=True,
)
assert info.name == "my-custom-model"
assert info.provider == "custom"
assert info.context_window == 64000
# Should be retrievable
retrieved = ModelRegistry.get("my-custom-model")
assert retrieved is not None
assert retrieved.context_window == 64000
def test_list_models_all(self):
"""Test listing all models."""
models = ModelRegistry.list_models()
assert len(models) > 0
def test_list_models_by_provider(self):
"""Test listing models by provider."""
openai_models = ModelRegistry.list_models(provider="openai")
assert len(openai_models) > 0
assert all(m.provider == "openai" for m in openai_models)
def test_list_models_with_tools(self):
"""Test listing models with tool support."""
models = ModelRegistry.list_models(supports_tools=True)
assert len(models) > 0
assert all(m.supports_tools for m in models)
def test_list_models_with_vision(self):
"""Test listing models with vision support."""
models = ModelRegistry.list_models(supports_vision=True)
assert len(models) > 0
assert all(m.supports_vision for m in models)
def test_list_models_min_context(self):
"""Test listing models with minimum context."""
models = ModelRegistry.list_models(min_context=1000000)
assert len(models) > 0
assert all(m.context_window >= 1000000 for m in models)
def test_list_providers(self):
"""Test listing all providers."""
providers = ModelRegistry.list_providers()
assert "openai" in providers
assert "anthropic" in providers
assert "google" in providers
def test_get_context_limit(self):
"""Test getting context limit."""
limit = ModelRegistry.get_context_limit("gpt-4o")
assert limit == 128000
def test_get_context_limit_unknown(self):
"""Test getting context limit for unknown model."""
limit = ModelRegistry.get_context_limit("unknown", default=32000)
assert limit == 32000
def test_estimate_cost(self):
"""Test cost estimation."""
cost = ModelRegistry.estimate_cost(
model="gpt-4o",
input_tokens=1000000,
output_tokens=500000,
)
assert cost is not None
# GPT-4o: $2.50/1M input + $10.00/1M output * 0.5 = $2.50 + $5.00 = $7.50
assert abs(cost - 7.50) < 0.01
def test_estimate_cost_with_cache(self):
"""Test cost estimation with cached tokens.
Note: LiteLLM's basic cost estimation doesn't support cached token pricing.
The cached_tokens parameter is accepted but not currently factored into cost.
"""
cost = ModelRegistry.estimate_cost(
model="gpt-4o",
input_tokens=1000000,
output_tokens=0,
cached_tokens=500000, # Not currently used by LiteLLM
)
assert cost is not None
# With LiteLLM, all 1M tokens are charged at input rate: $2.50
assert abs(cost - 2.50) < 0.01
def test_estimate_cost_unknown_model(self):
"""Test cost estimation for unknown model."""
cost = ModelRegistry.estimate_cost(
model="unknown-model",
input_tokens=1000,
output_tokens=500,
)
assert cost is None
class TestConvenienceFunctions:
"""Tests for convenience functions."""
def test_get_model_info(self):
"""Test get_model_info function."""
info = get_model_info("gpt-4o")
assert info is not None
assert info.name == "gpt-4o"
def test_list_models(self):
"""Test list_models function."""
models = list_models(provider="anthropic")
assert len(models) > 0
def test_register_model(self):
"""Test register_model function."""
info = register_model(
"test-function-model",
provider="test",
context_window=16000,
)
assert info.name == "test-function-model"
class TestBuiltInModels:
"""Tests for built-in model data."""
def test_gpt4o_info(self):
"""Test GPT-4o model info."""
info = get_model_info("gpt-4o")
assert info.provider == "openai"
assert info.context_window == 128000
assert info.supports_tools is True
assert info.supports_vision is True
# Pricing is now fetched from LiteLLM, not stored in ModelInfo
pricing = ModelRegistry.get_pricing("gpt-4o")
assert pricing is not None
assert pricing[0] == 2.50 # input cost per 1M
assert pricing[1] == 10.00 # output cost per 1M
def test_o1_info(self):
"""Test o1 model info."""
info = get_model_info("o1")
assert info.provider == "openai"
assert info.context_window == 200000 # 200K context
assert info.max_output_tokens == 100000 # 100K output
def test_claude_info(self):
"""Test Claude model info."""
info = get_model_info("claude-3-5-sonnet-20241022")
assert info.provider == "anthropic"
assert info.context_window == 200000
# Pricing comes from litellm's live table, so name a model it currently
# prices; the retired-id path is the MODEL_ALIASES assertion below.
pricing = ModelRegistry.get_pricing(anthropic_pricing_model())
assert pricing is not None
assert pricing[0] == 3.00 # input cost per 1M
assert pricing[1] == 15.00 # output cost per 1M
# Retired model alias should also resolve
alias_pricing = ModelRegistry.get_pricing("claude-3-5-sonnet-20241022")
assert alias_pricing is not None
def test_gemini_info(self):
"""Test Gemini model info."""
info = get_model_info("gemini-1.5-pro")
assert info.provider == "google"
assert info.context_window == 2000000 # 2M tokens!
def test_llama_info(self):
"""Test Llama model info."""
info = get_model_info("llama-3.1-8b")
assert info.provider == "meta"
assert info.context_window == 128000
assert info.tokenizer_backend == "huggingface"
def test_mistral_info(self):
"""Test Mistral model info."""
info = get_model_info("mistral-large")
assert info.provider == "mistral"
assert info.supports_tools is True
def test_deepseek_flash_is_registered_with_vision_and_legacy_aliases() -> None:
"""The current DeepSeek id carries V4.1-Flash capabilities; retired ids alias it."""
from headroom.models.registry import ModelRegistry
flash = ModelRegistry.get("deepseek-flash")
assert flash is not None
assert flash.provider == "deepseek"
assert flash.context_window == 1_000_000
assert flash.max_output_tokens == 384_000
assert flash.supports_vision is True
assert flash.supports_tools is True
assert flash.tokenizer_backend == "huggingface"
assert set(flash.aliases) == {"deepseek-v4-flash", "deepseek-v4-flash-vision-exp"}
for alias in ("deepseek-v4-flash", "deepseek-v4-flash-vision-exp"):
resolved = ModelRegistry.get(alias)
assert resolved is not None
assert resolved.name == "deepseek-flash"
pro = ModelRegistry.get("deepseek-v4-pro")
assert pro is not None
assert pro.supports_vision is False