mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Current maintainer qualification
Reviewed head 595803d786 after the
current-main sync. The PR-specific pricing implementation is unchanged
from the previous reviewed head.
Independent validation: 202 focused Python pricing/provider/savings
tests passed, four skipped. The Rust proxy binary built from this head.
Through the running Rust HTTP proxy and a separate loopback upstream,
requests carrying 100,000 and 100,001 input tokens produced the expected
native-ledger request costs of $0.0105 and $0.0525005 respectively,
including 1,000 output tokens. The default stable rollout channel was
used, with no unsafe feature override.
The HTTP replay uses controlled upstream usage; it does not claim live
Anthropic inference or independently verified provider billing. The
previously recorded missing native HTTP boundary check is now covered.
Current main is an ancestor. Hosted checks on this head are terminal: 49
successful, 2 skipped, with no failures. The prior shared Rust advisory
hold is cleared. Final merge approval remains human-only; nothing has
been merged by this review.
## Description
Claude Haiku 5.5 (`claude-haiku-5-5`) is released and had no rows in
Headroom's built-in Anthropic tables or the Rust proxy's vendored
LiteLLM snapshot. This PR adds it, including its second rate card for
prompts over 100K tokens, and makes every cost path read a model's
long-context threshold from its own catalog row instead of assuming
200K.
**Why the missing rows mispriced instead of failing:** without LiteLLM
data for the model, `AnthropicProvider` falls through to the `haiku`
pattern default ($0.80 / $4, 200K), which is 8x the real input rate and
a fifth of the real window. The Rust `context_window_for` is
exact-match, so the id fell through to the 128K default.
Haiku 5.5's rate card ([Anthropic
pricing](https://platform.claude.com/docs/en/about-claude/pricing),
matching upstream LiteLLM's `claude-haiku-5-5` rows): $0.10 / $0.50
input / output and $0.01 cache read for prompts up to 100K tokens; every
rate is 5x ($0.50 / $2.50 / $0.05) above 100K. 1M context, 128K max
output.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
All values in $ / MTok (input / output / cache read).
| Model | Before (fallback) | After | Context |
|---|---|---|---|
| `claude-haiku-5-5`, prompt <= 100K | *(no row: `haiku` tier default)*
$0.80 / $4 / $0.08 | **$0.10 / $0.50 / $0.01** | 200K -> **1M** |
| `claude-haiku-5-5`, prompt > 100K | $0.80 / $4 / $0.08 | **$0.50 /
$2.50 / $0.05** | |
- `headroom/providers/anthropic.py`:
- Adds Haiku 5.5 to `ANTHROPIC_PRICING` and `ANTHROPIC_CONTEXT_LIMITS`.
- The long-context premium becomes a small per-model table,
`_LONG_CONTEXT_TIERS` (prefixes, threshold, multipliers), so Haiku 5.5's
100K card sits next to the existing Sonnet 4 / 4.5 200K tier. The Sonnet
tier's threshold and multipliers are unchanged.
```diff
_apply_long_context_premium(model, pricing, input_tokens)
- if input_tokens <= 200K or model not in Sonnet 4 / 4.5: return pricing
- return pricing * Sonnet premium
+ for prefixes, threshold, premium in _LONG_CONTEXT_TIERS:
+ if model starts with prefixes:
+ return pricing if input_tokens <= threshold else pricing * premium
+ return pricing
```
- `crates/headroom-proxy/data/model_prices_and_context_window.json`:
appends the 18 upstream LiteLLM `claude-haiku-5-5` keys verbatim
(first-party, Bedrock regional/global/Mantle/GovCloud, Vertex, Azure,
OpenRouter, Perplexity). No existing entry changes or is removed;
extended rather than re-pulled for the reasons in #3841.
- **Per-model long-context tiers on the stats paths** (Devin finding 1).
Each catalog row publishes at most one `*_above_<N>k_tokens` tier (100K,
128K, 200K, 256K or 272K); the threshold now comes from the row, not a
shared 200K constant.
- `crates/headroom-proxy/src/observability/pricing.rs`: the price book
stores a `PriceCard` (base card + optional long-context card parsed from
the row's tier fields; a missing tier field falls back to that rate's
base price). `lookup` still returns the base card; `lookup_card` returns
both, and `PriceCard::for_prompt` picks one.
- `crates/headroom-proxy/src/observability/ledger.rs`: `record` picks
the card from the billed prompt (uncached + cache read + cache write)
before pricing spend and savings.
- `headroom/pricing/counterfactual.py`: `long_context_threshold(model)`
/ `long_context_tier(info)`; `resolve_rates` reads the row's own tier,
including its combined 1h + long-tier write rate where published.
`CacheMix.is_long_context` takes the threshold.
- `headroom/proxy/cost.py`, `headroom/proxy/savings_tracker.py`: bucket
and price at the model's own threshold.
- Side effect on other tiered rows, both intended: the Rust stats now
bill Sonnet 4 / 4.5 above 200K at its long card (the Python cost card
already did), and both paths switch GPT-5.x at 272K and the 128K-tier
rows at 128K with their own rates. Before, those rows crossed a 200K
line, found no `*_above_200k_tokens` fields and stayed on the base card.
- **Compression that crosses a tier credits the whole avoided premium**
(Devin re-review). When compression keeps a prompt under the threshold
its uncompressed form would have passed, the saving is the uncompressed
bill minus the actual bill, not the removed tokens at the cheap rate.
Rust: `PriceCard::compression_savings_usd` (input and output); Python:
`long_context_premium_avoided_usd`, used by
`estimate_request_savings_usd` and by `CostTracker` (which also adds the
output premium). The list companion still prices only the removed
tokens, so budget enforcement stays monotonic. The warm-prefix test in
`tests/test_cost_card_attribution.py` sent 200K with 10K saved, which
crosses Sonnet 4.5's tier; its counts are halved (same mix) so it stays
inside one tier.
- **Short ids no longer inherit a newer release** (Devin finding 2).
`get_context_limit` and `_get_pricing` matched substrings in both
directions, so `claude-haiku-5` resolved to Haiku 5.5 and
`claude-sonnet-4` to Sonnet 4.6's 1M window. `_is_release_of` accepts a
longer id containing the known one (dated or provider-wrapped) or an
undated / `-latest` alias of a dated key, and rejects ids one version
segment apart in either direction.
- `crates/headroom-proxy/src/compression/model_limits.rs`: the 1M-window
test now covers `claude-haiku-5-5`, `us.anthropic.claude-haiku-5-5` and
`vertex_ai/claude-haiku-5-5`.
- Rust tests: Haiku 5.5 base and >100K cards, per-model threshold
(Sonnet 4.5 at 200K), flat-rated model, missing tier field; ledger bills
150K in / 5K out at the long card, and cached tokens count toward the
threshold.
- `tests/test_providers/test_anthropic.py`: Haiku 5.5 price, dated-id
and 1M-context cases, cost cases at and above the 100K threshold, the
`claude-haiku-5` regression and an `_is_release_of` table.
- `tests/test_cost_card_attribution.py`,
`tests/test_savings_provider_matrix.py`: Haiku 5.5 cost card and savings
at 90K vs 150K.
- `tests/test_counterfactual_pricing.py`: the long-tier 1h test now
reads the published combined rate, and a synthetic row covers the
derived case.
- `tests/test_provider_model_fallback.py`: the "future haiku" probe
moves from `claude-haiku-5-20260101` to `claude-haiku-6-20270101`,
mirroring the opus probe change in #3841.
- `tests/test_thinking_compactor_version_gate.py`: `claude-haiku-5-5` is
treated as a Claude 5 model.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`) (ruff 0.16.8, on the changed
files)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ HOME=$(mktemp -d) pytest -q <119 pricing / cost / savings / stats / provider / learn modules>
2097 passed, 18 skipped, 5 warnings
$ cargo test -p headroom-proxy
577 passed, 0 failed (41 suites)
$ cargo clippy -p headroom-proxy --all-targets -- -D warnings
(clean)
$ cargo fmt -p headroom-proxy --check
(clean)
$ ruff check <changed .py files> # ruff 0.16.8
All checks passed!
$ ruff format --check <changed .py files>
(clean)
$ mypy headroom --ignore-missing-imports # mypy 1.20.2
Success: no issues found in 566 source files
```
`HOME` is isolated because a developer `~/.headroom/models.json` leaks
into `TestAnthropicConfigLoading`.
## Real Behavior Proof
- Environment: macOS, the repo `.venv`. `origin/main` and this PR head
were each checked out as a git worktree and run with the same
interpreter and an isolated `HOME`.
- Exact command / steps: a script built `AnthropicProvider()`, stubbed
`estimate_cost_from_tokens` and `_get_litellm_clients` to force the
built-in fallback (the LiteLLM-unavailable or offline path), then priced
two requests and read the context limit. For the Rust side it applied
`context_window_for` semantics (exact key, `mode == "chat"`,
`max_input_tokens`, 128K default) to each worktree's vendored JSON. For
the review fixes, a second script ran against the first PR commit
(`b0503796`) and this head: the fallback lookups for short ids, a
`CostTracker` 150K-in / 5K-out Haiku 5.5 turn with 10K saved, and
`resolve_rates(..., long_context=True)`; the Rust ledger case is the
`long_prompt_bills_at_the_models_long_context_card` test.
- Observed result: Haiku 5.5 now prices and sizes correctly on every
path, and short ids keep their tier default (tables below).
```text
origin/main this PR
context limit (Python fallback) 200,000 (inferred 'haiku' tier)
1,000,000
50K in / 5K out $0.060000 $0.007500
150K in (120K cached) / 5K out $0.053600 $0.033500 (100K+ card)
Rust context_window_for 128,000 1,000,000
```
Hand check of the last row: 30K × $0.50 + 120K × $0.05 + 5K × $2.50 per
MTok = $0.0335.
Review fixes, first PR commit vs this head:
```text
b0503796 this head
get_context_limit("claude-haiku-5") 1,000,000 200,000 ('haiku' tier)
_get_pricing("claude-haiku-5") $0.10 / $0.50 / $0.01 $0.80 / $4 / $0.08
get_context_limit("claude-sonnet-4") 1,000,000 (Sonnet 4.6) 200,000
(Sonnet 4)
CostTracker 150K in / 5K out: output cost $0.002500 $0.012500
CostTracker 10K saved: cache-aware savings $0.001000 $0.005000
resolve_rates(haiku-5-5, long).uncached $0.10 / MTok $0.50 / MTok
Rust ledger 150K in / 5K out $0.0175 (base card, computed) $0.0875
(ledger unit test)
```
Compression crossing the tier, Haiku 5.5, 110K uncached compressed to
90K (unit tests on this head; the "before" column is the old rule's
arithmetic):
```text
removed tokens at base this head
Rust ledger compression savings $0.002 $0.046
estimate_request_savings_usd["compression"] $0.002 $0.046
CostTracker cache-aware savings (1K output) $0.002 $0.048 (incl. $0.002
output premium)
```
- Not tested: live provider API billing. The controlled native HTTP
replay now covers the 100K boundary, as documented above.
- The Rust proxy end to end with a live >100K Haiku 5.5 request; the
ledger is covered by unit tests only.
- Bedrock-style ids (`us.anthropic.claude-haiku-5-5`) get the base card
without the 100K premium on the Python fallback path, the same
limitation the Sonnet 4 / 4.5 tier already has.
- No live API call was made.
## Maintainer follow-up qualification
Latest qualification head: `f885cc4c8afa574d867820583886687b0bae12a9`.
The pricing code and the 413-Python/73-Rust/TCP-replay validation below
are unchanged from `a45ae8c3e`. The first hosted shard-3 run exposed
five existing main integration failures:
`test_proxy_dependency_gate_matches_onnx_platform_markers` still patched
a removed eager-import hook. The follow-up aligns those existing tests
with lazy `find_spec` discovery and its missing-spec return value. The
five failures reproduce locally before; all eight dependency-contract
cases and 91 CLI/stateless/logging tests pass after, with
Ruff/format/whitespace passing. The current-head CI must finish before
approval.
Published maintainer follow-up at `a45ae8c3e` (normal push from the
unchanged `c326e2046` head): forward generated `output_tokens` into the
shared compression estimator through Prometheus and the durable lifetime
recorder, and add the long-card minus base-card output premium only when
compression crosses the model's threshold. `output_tokens_saved` remains
a separate output-shaping layer. The documented unpriced direct tracker
fallback is preserved.
Integrated current main `463c1a43e30fc33cb256a3f455daf28043e53b05` to
resolve the pricing conflict. The resolution retains canonical
model-specific cache billing, independent cache-rate provenance and
explicit-zero rates, while probing missing billed cache rates at the
model's own long-context threshold. New regressions cover both contracts
together.
Verification after the conflict resolution:
- 413 Python tests passed; one known Windows directory-fsync test
deselected after it failed identically on the unchanged PR head; three
existing warnings. This includes 20 new
output-premium/boundary/metadata/merged-cache-billing cases, pricing,
provider fallback, persistent metrics, cache TTL, tool-schema and output
savings suites.
- 73 Rust observability tests passed: `cargo test -p headroom-proxy
--lib observability --locked`.
- Ruff lint/format and whitespace checks pass. Focused Linux-platform
mypy passes for the four pricing/metrics modules. Full local mypy in
this extras-rich environment reports 19 errors in
`relevance/embedding.py`, `memory/backends/direct_mem0.py`,
`integrations/langchain/chat_model.py` and
`integrations/crewai/agents.py`; it is not claimed green.
- Before the fix, both public priced recorders report $0.046 for
110K→90K input with 1K generated output; after, both report $0.048.
Noncrossing and no-output controls pass, and separately saved output is
not conflated with the avoided premium.
Actual Windows/Python 3.12 TCP replay: real uvicorn proxy, separate
loopback HTTP upstream, and the real content-router/SmartCrusher
pipeline. A 304,903-character JSON tool result becomes 139,003
characters, saving 55,974 locally counted tokens. The controlled
upstream reports 90,000 input and 1,000 output tokens. On the unchanged
PR head the lifetime estimator reports $0.063987 and fails; with the fix
and after the main integration it reports the expected $0.065987. The
persistent Prometheus compression metric matches, and output-shaping
savings remain zero. This exercises real compression and handlers with
controlled usage/rates; no live provider billing or Rust HTTP runtime is
claimed.
Current-head GitHub CI and edited review bodies still require
qualification before approval.
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: cost estimates and the context window
reported for `claude-haiku-5-5` and the 18 added LiteLLM keys;
long-prompt spend and savings on tiered models now use each model's own
threshold; short ids such as `claude-haiku-5` fall back to their tier
default instead of a newer release
- Kill switch / disable path: n/a (data tables)
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the PR
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (n/a: no
user-facing docs list per-model prices)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
n/a
## Additional Notes
The 100K card applies on every path: the Python fallback
(`_LONG_CONTEXT_TIERS`), LiteLLM's own estimate, the cost card and
savings counterfactual, and the Rust stats ledger.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4037"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <tejas@headroomlabs.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
479 lines
20 KiB
Python
479 lines
20 KiB
Python
"""Tests for provider model fallback and configuration."""
|
|
|
|
import json
|
|
import logging
|
|
import os
|
|
import tempfile
|
|
from pathlib import Path
|
|
from unittest.mock import patch
|
|
|
|
import pytest
|
|
|
|
from headroom.providers.anthropic import (
|
|
AnthropicProvider,
|
|
_infer_model_tier,
|
|
)
|
|
from headroom.providers.anthropic import (
|
|
_load_custom_model_config as anthropic_load_config,
|
|
)
|
|
from headroom.providers.google import GeminiTokenCounter, GoogleProvider
|
|
from headroom.providers.openai import (
|
|
OpenAIProvider,
|
|
_infer_model_family,
|
|
)
|
|
from headroom.providers.openai import (
|
|
_load_custom_model_config as openai_load_config,
|
|
)
|
|
|
|
|
|
class TestGoogleModelFallback:
|
|
"""Tests for Google provider model fallback."""
|
|
|
|
def test_future_gemini_model_uses_registry_family_fallback(self):
|
|
"""Future Gemini models should not hard-fail token counting."""
|
|
provider = GoogleProvider()
|
|
|
|
with patch("headroom.models.registry.get_model_pricing", return_value=None):
|
|
assert provider.supports_model("gemini-3-pro-preview")
|
|
assert provider.get_context_limit("gemini-3-pro-preview") == 1000000
|
|
assert isinstance(
|
|
provider.get_token_counter("gemini-3-pro-preview"),
|
|
GeminiTokenCounter,
|
|
)
|
|
|
|
def test_litellm_prefixed_gemini_model_uses_registry_family_fallback(self):
|
|
"""LiteLLM-style Gemini ids should resolve through the Google provider."""
|
|
provider = GoogleProvider()
|
|
|
|
with patch("headroom.models.registry.get_model_pricing", return_value=None):
|
|
assert provider.supports_model("gemini/gemini-3-pro-preview")
|
|
assert provider.get_context_limit("gemini/gemini-3-pro-preview") == 1000000
|
|
|
|
def test_google_legacy_context_limits_are_preserved(self):
|
|
"""Moving lookup through ModelRegistry must keep legacy Gemini limits."""
|
|
provider = GoogleProvider()
|
|
|
|
with patch("headroom.models.registry.get_model_pricing", return_value=None):
|
|
assert provider.get_context_limit("gemini-1.5-pro-latest") == 2000000
|
|
assert provider.get_context_limit("gemini-1.0-pro") == 32768
|
|
|
|
def test_unknown_non_gemini_model_still_rejected(self):
|
|
"""The Google provider should not claim unrelated unknown models."""
|
|
provider = GoogleProvider()
|
|
|
|
assert not provider.supports_model("not-a-google-model")
|
|
assert not provider.supports_model("gpt-4o")
|
|
with pytest.raises(ValueError):
|
|
provider.get_token_counter("not-a-google-model")
|
|
|
|
|
|
class TestAnthropicModelFallback:
|
|
"""Tests for Anthropic provider model fallback."""
|
|
|
|
def test_known_claude_4_models(self):
|
|
"""Test that Claude 4/4.5 models are recognized."""
|
|
provider = AnthropicProvider()
|
|
|
|
# Claude Opus 4.5
|
|
assert provider.get_context_limit("claude-opus-4-5-20251101") == 200000
|
|
assert provider.supports_model("claude-opus-4-5-20251101")
|
|
|
|
# Claude Sonnet 4
|
|
assert provider.get_context_limit("claude-sonnet-4-20250514") == 200000
|
|
assert provider.supports_model("claude-sonnet-4-20250514")
|
|
|
|
# Claude Haiku 4
|
|
assert provider.get_context_limit("claude-haiku-4-5-20251001") == 200000
|
|
assert provider.supports_model("claude-haiku-4-5-20251001")
|
|
|
|
def test_pattern_based_inference_opus(self):
|
|
"""Test pattern-based inference for opus models."""
|
|
provider = AnthropicProvider()
|
|
|
|
# Future opus model should infer 200K and opus pricing
|
|
limit = provider.get_context_limit("claude-opus-6-20270101")
|
|
assert limit == 200000
|
|
|
|
pricing = provider._get_pricing("claude-opus-6-20270101")
|
|
assert pricing["input"] == 5.00
|
|
assert pricing["output"] == 25.00
|
|
|
|
def test_pattern_based_inference_sonnet(self):
|
|
"""Test pattern-based inference for sonnet models."""
|
|
provider = AnthropicProvider()
|
|
|
|
limit = provider.get_context_limit("claude-sonnet-6-20260101")
|
|
assert limit == 200000
|
|
|
|
pricing = provider._get_pricing("claude-sonnet-6-20260101")
|
|
assert pricing["input"] == 3.00
|
|
assert pricing["output"] == 15.00
|
|
|
|
def test_pattern_based_inference_haiku(self):
|
|
"""Test pattern-based inference for haiku models."""
|
|
provider = AnthropicProvider()
|
|
|
|
limit = provider.get_context_limit("claude-haiku-6-20270101")
|
|
assert limit == 200000
|
|
|
|
pricing = provider._get_pricing("claude-haiku-6-20270101")
|
|
assert pricing["input"] == 0.80
|
|
assert pricing["output"] == 4.00
|
|
|
|
def test_unknown_claude_model_fallback(self):
|
|
"""Test fallback for unknown Claude models."""
|
|
provider = AnthropicProvider()
|
|
|
|
# Unknown Claude model should get 200K default
|
|
limit = provider.get_context_limit("claude-unknown-model")
|
|
assert limit == 200000
|
|
|
|
# Should still support it
|
|
assert provider.supports_model("claude-unknown-model")
|
|
|
|
def test_no_exception_for_unknown_model(self):
|
|
"""Test that unknown models don't raise exceptions."""
|
|
provider = AnthropicProvider()
|
|
|
|
# Should not raise
|
|
limit = provider.get_context_limit("claude-future-model-xyz")
|
|
assert limit > 0
|
|
|
|
def test_infer_model_tier(self):
|
|
"""Test model tier inference."""
|
|
assert _infer_model_tier("claude-opus-4-5-20251101") == "opus"
|
|
assert _infer_model_tier("claude-sonnet-4-20250514") == "sonnet"
|
|
assert _infer_model_tier("claude-haiku-4-5-20251001") == "haiku"
|
|
assert _infer_model_tier("claude-3-5-sonnet-latest") == "sonnet"
|
|
assert _infer_model_tier("CLAUDE-OPUS-FUTURE") == "opus" # Case insensitive
|
|
assert _infer_model_tier("some-other-model") is None
|
|
|
|
def test_explicit_context_limits_override(self):
|
|
"""Test that explicit context_limits override defaults."""
|
|
provider = AnthropicProvider(context_limits={"custom-model": 500000})
|
|
|
|
assert provider.get_context_limit("custom-model") == 500000
|
|
|
|
def test_pricing_for_known_models(self):
|
|
"""Test pricing retrieval for known models."""
|
|
provider = AnthropicProvider()
|
|
|
|
# Claude Opus 4.5
|
|
pricing = provider._get_pricing("claude-opus-4-5-20251101")
|
|
assert pricing["input"] == 5.00
|
|
assert pricing["output"] == 25.00
|
|
assert pricing["cached_input"] == 0.50
|
|
|
|
def test_cost_estimation_for_new_models(self):
|
|
"""Test cost estimation works for new models."""
|
|
provider = AnthropicProvider()
|
|
|
|
cost = provider.estimate_cost(
|
|
input_tokens=1000000,
|
|
output_tokens=100000,
|
|
model="claude-opus-4-5-20251101",
|
|
cached_tokens=0,
|
|
)
|
|
|
|
# $5/1M input + $25/1M * 0.1M output = $5 + $2.5 = $7.5
|
|
assert cost == pytest.approx(7.5, rel=0.01)
|
|
|
|
|
|
class TestAnthropicConfigLoading:
|
|
"""Tests for Anthropic config file/env var loading."""
|
|
|
|
def test_load_from_env_var_json(self):
|
|
"""Test loading config from JSON env var."""
|
|
config = {"context_limits": {"test-model": 300000}}
|
|
|
|
with patch.dict(os.environ, {"HEADROOM_MODEL_LIMITS": json.dumps(config)}):
|
|
loaded = anthropic_load_config()
|
|
assert loaded["context_limits"]["test-model"] == 300000
|
|
|
|
def test_load_from_env_var_file(self):
|
|
"""Test loading config from file path in env var."""
|
|
config = {"context_limits": {"file-model": 400000}}
|
|
|
|
with tempfile.TemporaryDirectory() as tmpdir:
|
|
config_path = Path(tmpdir) / "model_limits.json"
|
|
config_path.write_text(json.dumps(config))
|
|
|
|
with patch.dict(os.environ, {"HEADROOM_MODEL_LIMITS": str(config_path)}):
|
|
loaded = anthropic_load_config()
|
|
assert loaded["context_limits"]["file-model"] == 400000
|
|
|
|
def test_load_from_config_file(self):
|
|
"""Test loading from ~/.headroom/models.json."""
|
|
config = {
|
|
"anthropic": {
|
|
"context_limits": {"config-model": 250000},
|
|
"pricing": {"config-model": {"input": 5.0, "output": 25.0}},
|
|
}
|
|
}
|
|
|
|
with tempfile.TemporaryDirectory() as tmpdir:
|
|
config_dir = Path(tmpdir) / ".headroom"
|
|
config_dir.mkdir()
|
|
config_file = config_dir / "models.json"
|
|
config_file.write_text(json.dumps(config))
|
|
|
|
with patch.object(Path, "home", return_value=Path(tmpdir)):
|
|
loaded = anthropic_load_config()
|
|
assert loaded["context_limits"]["config-model"] == 250000
|
|
|
|
def test_env_var_overrides_config_file(self):
|
|
"""Test that env var takes precedence over config file."""
|
|
env_config = {"context_limits": {"test-model": 100000}}
|
|
file_config = {"anthropic": {"context_limits": {"test-model": 200000}}}
|
|
|
|
with tempfile.TemporaryDirectory() as tmpdir:
|
|
config_dir = Path(tmpdir) / ".headroom"
|
|
config_dir.mkdir()
|
|
config_file = config_dir / "models.json"
|
|
config_file.write_text(json.dumps(file_config))
|
|
|
|
with patch.object(Path, "home", return_value=Path(tmpdir)):
|
|
with patch.dict(os.environ, {"HEADROOM_MODEL_LIMITS": json.dumps(env_config)}):
|
|
loaded = anthropic_load_config()
|
|
# Env var should win
|
|
assert loaded["context_limits"]["test-model"] == 100000
|
|
|
|
@pytest.mark.parametrize("raw", ["[1, 2, 3]", '"gpt-4"', "42", "true", "null"])
|
|
def test_non_object_env_var_falls_back_to_defaults(self, raw):
|
|
"""A valid-JSON-but-not-an-object env var must warn and use defaults,
|
|
not crash provider init with AttributeError on ``loaded.get``."""
|
|
with patch.dict(os.environ, {"HEADROOM_MODEL_LIMITS": raw}):
|
|
loaded = anthropic_load_config()
|
|
assert loaded == {"context_limits": {}, "pricing": {}}
|
|
|
|
def test_flat_shape_env_var_warns_that_it_had_no_effect(self, caplog):
|
|
"""A JSON *object* using the intuitive-but-wrong flat shape
|
|
``{"my-model": 262144}`` is silently ignored by the loader.
|
|
|
|
The unknown-model warning tells operators to "set HEADROOM_MODEL_LIMITS",
|
|
so the flat shape is the natural first guess. Without a diagnostic the
|
|
operator sees the conservative default silently persist and has no way
|
|
to tell the config was never applied. Assert we now say so explicitly.
|
|
"""
|
|
with patch.dict(os.environ, {"HEADROOM_MODEL_LIMITS": '{"qwen3.8": 262144}'}):
|
|
with caplog.at_level(logging.WARNING, logger="headroom.providers.anthropic"):
|
|
loaded = anthropic_load_config()
|
|
assert loaded == {"context_limits": {}, "pricing": {}}
|
|
assert "NO EFFECT" in caplog.text
|
|
assert "context_limits" in caplog.text
|
|
|
|
def test_correct_shape_env_var_does_not_warn(self, caplog):
|
|
"""The documented nested shape must apply cleanly and stay silent."""
|
|
cfg = '{"context_limits": {"qwen3.8": 262144}}'
|
|
with patch.dict(os.environ, {"HEADROOM_MODEL_LIMITS": cfg}):
|
|
with caplog.at_level(logging.WARNING, logger="headroom.providers.anthropic"):
|
|
loaded = anthropic_load_config()
|
|
assert loaded["context_limits"]["qwen3.8"] == 262144
|
|
assert "NO EFFECT" not in caplog.text
|
|
|
|
def test_other_provider_namespaced_env_var_does_not_warn(self, caplog):
|
|
"""``{"openai": {"context_limits": ...}}`` is the documented shape for the
|
|
OpenAI loader. The Anthropic loader consumes nothing from it, which is
|
|
correct, so it must not claim the config had no effect."""
|
|
cfg = '{"openai": {"context_limits": {"gpt-x": 400000}}}'
|
|
with patch.dict(os.environ, {"HEADROOM_MODEL_LIMITS": cfg}):
|
|
with caplog.at_level(logging.WARNING, logger="headroom.providers.anthropic"):
|
|
loaded = anthropic_load_config()
|
|
assert loaded == {"context_limits": {}, "pricing": {}}
|
|
assert "NO EFFECT" not in caplog.text
|
|
|
|
def test_anthropic_section_without_known_keys_still_warns(self, caplog):
|
|
"""An explicit ``anthropic`` section that carries none of the consumed
|
|
keys is the same silent no-op as the flat shape, so it warns."""
|
|
cfg = '{"anthropic": {"qwen3.8": 262144}}'
|
|
with patch.dict(os.environ, {"HEADROOM_MODEL_LIMITS": cfg}):
|
|
with caplog.at_level(logging.WARNING, logger="headroom.providers.anthropic"):
|
|
anthropic_load_config()
|
|
assert "NO EFFECT" in caplog.text
|
|
|
|
def test_non_object_config_file_falls_back_to_defaults(self):
|
|
"""A models.json whose top level is not an object must not crash."""
|
|
with tempfile.TemporaryDirectory() as tmpdir:
|
|
config_dir = Path(tmpdir) / ".headroom"
|
|
config_dir.mkdir()
|
|
(config_dir / "models.json").write_text("[1, 2, 3]")
|
|
|
|
with patch.object(Path, "home", return_value=Path(tmpdir)):
|
|
loaded = anthropic_load_config()
|
|
assert loaded == {"context_limits": {}, "pricing": {}}
|
|
|
|
|
|
class TestOpenAIModelFallback:
|
|
"""Tests for OpenAI provider model fallback."""
|
|
|
|
def test_known_models(self):
|
|
"""Test that known models work."""
|
|
provider = OpenAIProvider()
|
|
|
|
assert provider.get_context_limit("gpt-4o") == 128000
|
|
assert provider.get_context_limit("gpt-4o-mini") == 128000
|
|
assert provider.get_context_limit("o1") == 200000
|
|
assert provider.get_context_limit("o3-mini") == 200000
|
|
|
|
def test_pattern_based_inference_gpt4o(self):
|
|
"""Test pattern-based inference for gpt-4o models."""
|
|
provider = OpenAIProvider()
|
|
|
|
# Future gpt-4o model
|
|
limit = provider.get_context_limit("gpt-4o-2025-01-01")
|
|
assert limit == 128000
|
|
|
|
def test_pattern_based_inference_o1(self):
|
|
"""Test pattern-based inference for o1 models."""
|
|
provider = OpenAIProvider()
|
|
|
|
limit = provider.get_context_limit("o1-super-2025")
|
|
assert limit == 200000
|
|
|
|
def test_pattern_based_inference_o3(self):
|
|
"""Test pattern-based inference for o3 models."""
|
|
provider = OpenAIProvider()
|
|
|
|
limit = provider.get_context_limit("o3-large-2025")
|
|
assert limit == 200000
|
|
|
|
def test_unknown_model_fallback(self):
|
|
"""Test fallback for unknown models."""
|
|
provider = OpenAIProvider()
|
|
|
|
# Unknown model should get 128K default. Deliberately a name that
|
|
# matches no known family prefix -- this used to say "gpt-5-future",
|
|
# which stopped being unknown once gpt-5 was added to _CONTEXT_LIMITS.
|
|
limit = provider.get_context_limit("gpt-9-imaginary")
|
|
assert limit == 128000
|
|
|
|
def test_unknown_variant_inherits_its_family_limit(self):
|
|
"""An unrecognized variant of a *known* family takes that family's limit.
|
|
|
|
This is the same prefix inheritance that gives "gpt-4o-2024-11-20" the
|
|
gpt-4o limit, and it is strictly better than dropping such a model to
|
|
the generic 128K default.
|
|
"""
|
|
provider = OpenAIProvider()
|
|
|
|
assert provider.get_context_limit("gpt-5-future") == 272000
|
|
assert provider.get_context_limit("gpt-4.1-preview") == 1_047_576
|
|
|
|
def test_no_exception_for_unknown_model(self):
|
|
"""Test that unknown models don't raise exceptions."""
|
|
provider = OpenAIProvider()
|
|
|
|
# Should not raise
|
|
limit = provider.get_context_limit("gpt-future-xyz")
|
|
assert limit > 0
|
|
|
|
def test_infer_model_family(self):
|
|
"""Test model family inference."""
|
|
assert _infer_model_family("gpt-4o-2024-11-20") == "gpt-4o"
|
|
assert _infer_model_family("gpt-4-turbo-preview") == "gpt-4-turbo"
|
|
assert _infer_model_family("gpt-4") == "gpt-4"
|
|
assert _infer_model_family("gpt-3.5-turbo") == "gpt-3.5"
|
|
assert _infer_model_family("o1-preview") == "o1"
|
|
assert _infer_model_family("o3-mini") == "o3"
|
|
assert _infer_model_family("unknown") is None
|
|
|
|
def test_explicit_context_limits_override(self):
|
|
"""Test that explicit context_limits override defaults."""
|
|
provider = OpenAIProvider(context_limits={"custom-model": 500000})
|
|
|
|
assert provider.get_context_limit("custom-model") == 500000
|
|
|
|
def test_supports_model_expanded(self):
|
|
"""Test that supports_model works for new patterns."""
|
|
provider = OpenAIProvider()
|
|
|
|
# Should support any gpt-* or o1/o3
|
|
assert provider.supports_model("gpt-4o")
|
|
assert provider.supports_model("gpt-4o-future")
|
|
assert provider.supports_model("gpt-5-future")
|
|
assert provider.supports_model("o1-mega")
|
|
assert provider.supports_model("o3-ultra")
|
|
|
|
|
|
class TestOpenAIConfigLoading:
|
|
"""Tests for OpenAI config file/env var loading."""
|
|
|
|
def test_load_from_env_var_json(self):
|
|
"""Test loading config from JSON env var."""
|
|
config = {"openai": {"context_limits": {"test-model": 300000}}}
|
|
|
|
with patch.dict(os.environ, {"HEADROOM_MODEL_LIMITS": json.dumps(config)}):
|
|
loaded = openai_load_config()
|
|
assert loaded["context_limits"]["test-model"] == 300000
|
|
|
|
def test_load_pricing_from_config(self):
|
|
"""Test loading pricing from config."""
|
|
config = {"openai": {"pricing": {"test-model": [5.0, 15.0]}}}
|
|
|
|
with tempfile.TemporaryDirectory() as tmpdir:
|
|
config_path = Path(tmpdir) / "model_limits.json"
|
|
config_path.write_text(json.dumps(config))
|
|
|
|
with patch.dict(os.environ, {"HEADROOM_MODEL_LIMITS": str(config_path)}):
|
|
loaded = openai_load_config()
|
|
assert loaded["pricing"]["test-model"] == [5.0, 15.0]
|
|
|
|
@pytest.mark.parametrize("raw", ["[1, 2, 3]", '"gpt-4"', "42", "true", "null"])
|
|
def test_non_object_env_var_falls_back_to_defaults(self, raw):
|
|
"""A valid-JSON-but-not-an-object env var must warn and use defaults,
|
|
not crash provider init with AttributeError on ``loaded.get``."""
|
|
with patch.dict(os.environ, {"HEADROOM_MODEL_LIMITS": raw}):
|
|
loaded = openai_load_config()
|
|
assert loaded == {"context_limits": {}, "pricing": {}, "encodings": {}}
|
|
|
|
|
|
class TestCrossProviderConsistency:
|
|
"""Tests for consistency across providers."""
|
|
|
|
def test_both_providers_use_same_env_var(self):
|
|
"""Test that both providers use HEADROOM_MODEL_LIMITS."""
|
|
config = {
|
|
"anthropic": {"context_limits": {"anthropic-model": 100000}},
|
|
"openai": {"context_limits": {"openai-model": 200000}},
|
|
}
|
|
|
|
with patch.dict(os.environ, {"HEADROOM_MODEL_LIMITS": json.dumps(config)}):
|
|
anthropic = anthropic_load_config()
|
|
openai = openai_load_config()
|
|
|
|
assert anthropic["context_limits"]["anthropic-model"] == 100000
|
|
assert openai["context_limits"]["openai-model"] == 200000
|
|
|
|
def test_both_providers_never_raise_for_unknown_models(self):
|
|
"""Test that neither provider raises for unknown models."""
|
|
anthropic = AnthropicProvider()
|
|
openai = OpenAIProvider()
|
|
|
|
# Neither should raise
|
|
anthropic.get_context_limit("claude-future-model-xyz")
|
|
openai.get_context_limit("gpt-future-model-xyz")
|
|
|
|
def test_both_providers_warn_for_unknown_models(self):
|
|
"""Test that both providers warn for unknown models."""
|
|
# Clear warning caches
|
|
from headroom.providers import anthropic as anthropic_module
|
|
from headroom.providers import openai as openai_module
|
|
|
|
anthropic_module._UNKNOWN_MODEL_WARNINGS.clear()
|
|
openai_module._UNKNOWN_MODEL_WARNINGS.clear()
|
|
|
|
with (
|
|
patch.object(anthropic_module.logger, "warning") as anthropic_warning,
|
|
patch.object(openai_module.logger, "warning") as openai_warning,
|
|
):
|
|
anthropic = AnthropicProvider()
|
|
anthropic.get_context_limit("claude-test-unknown-model")
|
|
|
|
openai = OpenAIProvider()
|
|
openai.get_context_limit("gpt-test-unknown-model")
|
|
|
|
anthropic_warning.assert_called_once()
|
|
openai_warning.assert_called_once()
|
|
assert "claude-test-unknown-model" in anthropic_warning.call_args.args[0]
|
|
assert "gpt-test-unknown-model" in openai_warning.call_args.args[0]
|