mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 12:05:34 +08:00
## Description
`litellm.model_cost` is downloaded live. It no longer carries
`input_cost_per_token_above_200k_tokens` on any bare Anthropic id. That
broke two things.
1. **Production pricing (bug).** `AnthropicProvider.estimate_cost`
prices through LiteLLM first, so a 300K-token Sonnet 4.5 request was
priced at the base tier: **$0.975 instead of $1.9125**. That is about
half the real cost of exactly the long `[1m]` sessions. Now, when a
request is above the 200K threshold for a tiered model and the model's
LiteLLM entry has no above-threshold rate, the provider prices it from
its own table, which still applies Anthropic's premium (input 2x, output
1.5x, cache 2x). Every other request still goes through LiteLLM
unchanged.
2. **CI on `main` and every open PR.** `tests/_pricing_models.py` could
find no candidate with the full field set, so
`tests/test_cost_card_attribution.py` and
`tests/test_savings_provider_matrix.py` errored at collection. That hid
failure 1, which surfaced as soon as collection worked. The fix adds the
one id that still has every field, the Bedrock Sonnet 4.5 entry
`claude-sonnet-4-5-20250929-v1:0` (same rates), as the last-resort
candidate. These tests only check that Headroom's arithmetic agrees with
LiteLLM's numbers.
Refs #4136
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/providers/anthropic.py`:
`_litellm_lacks_long_context_rate()`; `estimate_cost` skips LiteLLM for
a long-context request whose entry lacks the above-200K rate, and uses
the manual table with its premium.
- `tests/test_providers/test_anthropic.py`: a hermetic regression test
that strips the rate from the LiteLLM entry and asserts $1.9125; it
fails on main.
- `tests/_pricing_models.py`: add `claude-sonnet-4-5-20250929-v1:0` as
the last-resort candidate, with a comment explaining why.
- `tests/test_counterfactual_pricing.py`:
`test_long_context_derives_the_one_hour_rate_and_says_so` priced the
long-context tier of a model chosen for the base fields only. It now
asks `anthropic_pricing_model()` for a model that has that tier. It also
failed on `main`; the collection errors had been hiding it in CI.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest -q -p no:cacheprovider tests/test_cost_card_attribution.py tests/test_savings_provider_matrix.py tests/test_cost_budget_basis.py tests/test_cost_tracker_counterfactual.py tests/test_counterfactual_pricing.py tests/test_savings_ledger.py tests/test_models.py tests/test_tokenizers.py tests/test_realignment_live_multi_turn.py
190 passed, 22 skipped in 4.79s
(on origin/main: 2 collection errors, plus test_long_context_derives_the_one_hour_rate_and_says_so failing)
$ pytest -q tests/test_providers/test_anthropic.py -k LongContext
10 passed (origin/main code: 3 failed, 7 passed: test_litellm_path[300000-...] x2 and the new hermetic test)
$ pytest -q tests/test_providers <all pricing/cost/savings tests>
629 passed, 14 skipped
$ uvx ruff@0.16.8 check / format --check (both files)
All checks passed! / 2 files already formatted
```
## Real Behavior Proof
- Environment: macOS, Python 3.12, repo .venv with litellm 1.101.0 and
the live LiteLLM price map as of 2026-10-09. Clean worktrees of
`origin/main` (a9815bbb) and of this branch.
- Exact command / steps: `python -c "from headroom.providers.anthropic
import AnthropicProvider as A; p=A();
print([round(p.estimate_cost(i,5000,'claude-sonnet-4-5',c),4) for i,c in
((100000,0),(300000,0),(300000,150000))])"` in each worktree. Then the
branch's `TestLongContextPricing` run against each.
- Observed result: on `main` the output is `[0.375, 0.975, 0.57]`: both
300K requests are billed at the base tier, about half the real cost. On
this branch it is `[0.375, 1.9125, 1.1025]`, Anthropic's published
long-context price; the 100K request is unchanged.
`TestLongContextPricing` gives 3 failed / 7 passed against main's code
and 10 passed on the branch. An untiered model (Opus 4.5, 300K) still
prices flat at 1.625.
- Not tested: A real Anthropic invoice. The expected values are
Anthropic's published rates, which the manual table and LiteLLM's former
entry both carry.
## Runtime Rollout Safety
- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A; cost reporting only, no request change.
- Stable/default behavior changed: Yes, only for Anthropic tiered models
above 200K input tokens whose LiteLLM entry lacks the long-context rate.
Those now report the premium cost instead of half of it.
- Kill switch / disable path: N/A.
- Unsafe override required: No.
- Qualification impact: Unblocks the Python test shards on every open
PR.
- Rollback path: Revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A
## Additional Notes
Follow-up: `headroom.pricing.counterfactual.resolve_rates(...,
long_context=True)` reads the same LiteLLM fields for its
cache-economics estimates and is not changed here.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
69 lines
2.8 KiB
Python
69 lines
2.8 KiB
Python
"""Pick a model litellm actually prices, instead of hardcoding one it may retire.
|
|
|
|
``litellm.model_cost`` is downloaded from GitHub at import time, so it is live
|
|
third-party data. BerriAI prunes retired models from it: on 2026-09-23
|
|
``claude-sonnet-4-20250514`` disappeared and every test that priced it began
|
|
failing with ``KeyError: 'input_cost_per_token'`` — on every open pull request
|
|
at once, with no change on our side.
|
|
|
|
The model id in those tests is incidental. They assert that Headroom's cost
|
|
arithmetic agrees with litellm's numbers, not that any particular model is
|
|
priced correctly, so the fix is to stop naming a specific release and instead
|
|
ask for *a* model carrying the fields the test needs.
|
|
|
|
Pinning to the copy of the table vendored in the litellm wheel is not the
|
|
alternative it looks like: the two maps are complementary, not ordered. The
|
|
vendored map keeps retired ids but predates current models
|
|
(``claude-sonnet-5``), and its older entries lack newer fields such as
|
|
``input_cost_per_token_above_200k_tokens`` entirely.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
#: Preference order, newest first. A test takes the first entry that carries
|
|
#: every field it needs, so retiring one is a no-op until the list runs dry.
|
|
_CANDIDATES: tuple[str, ...] = (
|
|
"claude-sonnet-4-5-20250929",
|
|
"claude-sonnet-4-5",
|
|
"claude-sonnet-4-20250514",
|
|
"claude-opus-4-5-20251101",
|
|
"claude-opus-4-5",
|
|
# Last resort. By 2026-10, litellm had dropped the 200k-tier fields from
|
|
# every bare Anthropic id; only this Bedrock id still carries the full set,
|
|
# at the same rates as claude-sonnet-4-5.
|
|
"claude-sonnet-4-5-20250929-v1:0",
|
|
)
|
|
|
|
_BASE_FIELDS = ("input_cost_per_token", "output_cost_per_token")
|
|
|
|
|
|
def anthropic_pricing_model(*required_fields: str) -> str:
|
|
"""Return a currently-priced Anthropic model carrying ``required_fields``.
|
|
|
|
Always includes the base input/output costs. Raises with an actionable
|
|
message rather than skipping: if litellm prices none of these, the pricing
|
|
tests are not measuring anything and that should be loud.
|
|
"""
|
|
import litellm
|
|
|
|
needed = set(_BASE_FIELDS) | set(required_fields)
|
|
for model in _CANDIDATES:
|
|
info = litellm.model_cost.get(model)
|
|
if isinstance(info, dict) and needed <= set(info):
|
|
return model
|
|
|
|
raise AssertionError(
|
|
"litellm prices none of the candidate models with the fields "
|
|
f"{sorted(needed)}. It most likely retired them from "
|
|
"model_prices_and_context_window.json; add a current model id to "
|
|
"_CANDIDATES in tests/_pricing_models.py (newest first)."
|
|
)
|
|
|
|
|
|
@pytest.fixture(scope="session")
|
|
def anthropic_model() -> str:
|
|
"""Session fixture wrapper for tests that prefer injection over a constant."""
|
|
return anthropic_pricing_model()
|