Files
headroom/tests/_pricing_models.py
Tejas ChopraandClaude Opus 5.5 8c1beb20ae fix(pricing): keep the long-context premium when litellm drops the 200k rate (#4109)
## Description

`litellm.model_cost` is downloaded live. It no longer carries
`input_cost_per_token_above_200k_tokens` on any bare Anthropic id. That
broke two things.

1. **Production pricing (bug).** `AnthropicProvider.estimate_cost`
prices through LiteLLM first, so a 300K-token Sonnet 4.5 request was
priced at the base tier: **$0.975 instead of $1.9125**. That is about
half the real cost of exactly the long `[1m]` sessions. Now, when a
request is above the 200K threshold for a tiered model and the model's
LiteLLM entry has no above-threshold rate, the provider prices it from
its own table, which still applies Anthropic's premium (input 2x, output
1.5x, cache 2x). Every other request still goes through LiteLLM
unchanged.
2. **CI on `main` and every open PR.** `tests/_pricing_models.py` could
find no candidate with the full field set, so
`tests/test_cost_card_attribution.py` and
`tests/test_savings_provider_matrix.py` errored at collection. That hid
failure 1, which surfaced as soon as collection worked. The fix adds the
one id that still has every field, the Bedrock Sonnet 4.5 entry
`claude-sonnet-4-5-20250929-v1:0` (same rates), as the last-resort
candidate. These tests only check that Headroom's arithmetic agrees with
LiteLLM's numbers.

Refs #4136

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/providers/anthropic.py`:
`_litellm_lacks_long_context_rate()`; `estimate_cost` skips LiteLLM for
a long-context request whose entry lacks the above-200K rate, and uses
the manual table with its premium.
- `tests/test_providers/test_anthropic.py`: a hermetic regression test
that strips the rate from the LiteLLM entry and asserts $1.9125; it
fails on main.
- `tests/_pricing_models.py`: add `claude-sonnet-4-5-20250929-v1:0` as
the last-resort candidate, with a comment explaining why.
- `tests/test_counterfactual_pricing.py`:
`test_long_context_derives_the_one_hour_rate_and_says_so` priced the
long-context tier of a model chosen for the base fields only. It now
asks `anthropic_pricing_model()` for a model that has that tier. It also
failed on `main`; the collection errors had been hiding it in CI.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest -q -p no:cacheprovider tests/test_cost_card_attribution.py tests/test_savings_provider_matrix.py tests/test_cost_budget_basis.py tests/test_cost_tracker_counterfactual.py tests/test_counterfactual_pricing.py tests/test_savings_ledger.py tests/test_models.py tests/test_tokenizers.py tests/test_realignment_live_multi_turn.py
190 passed, 22 skipped in 4.79s
(on origin/main: 2 collection errors, plus test_long_context_derives_the_one_hour_rate_and_says_so failing)
$ pytest -q tests/test_providers/test_anthropic.py -k LongContext
10 passed            (origin/main code: 3 failed, 7 passed: test_litellm_path[300000-...] x2 and the new hermetic test)
$ pytest -q tests/test_providers <all pricing/cost/savings tests>
629 passed, 14 skipped
$ uvx ruff@0.16.8 check / format --check (both files)
All checks passed! / 2 files already formatted
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, repo .venv with litellm 1.101.0 and
the live LiteLLM price map as of 2026-10-09. Clean worktrees of
`origin/main` (a9815bbb) and of this branch.
- Exact command / steps: `python -c "from headroom.providers.anthropic
import AnthropicProvider as A; p=A();
print([round(p.estimate_cost(i,5000,'claude-sonnet-4-5',c),4) for i,c in
((100000,0),(300000,0),(300000,150000))])"` in each worktree. Then the
branch's `TestLongContextPricing` run against each.
- Observed result: on `main` the output is `[0.375, 0.975, 0.57]`: both
300K requests are billed at the base tier, about half the real cost. On
this branch it is `[0.375, 1.9125, 1.1025]`, Anthropic's published
long-context price; the 100K request is unchanged.
`TestLongContextPricing` gives 3 failed / 7 passed against main's code
and 10 passed on the branch. An untiered model (Opus 4.5, 300K) still
prices flat at 1.625.
- Not tested: A real Anthropic invoice. The expected values are
Anthropic's published rates, which the manual table and LiteLLM's former
entry both carry.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A; cost reporting only, no request change.
- Stable/default behavior changed: Yes, only for Anthropic tiered models
above 200K input tokens whose LiteLLM entry lacks the long-context rate.
Those now report the premium cost instead of half of it.
- Kill switch / disable path: N/A.
- Unsafe override required: No.
- Qualification impact: Unblocks the Python test shards on every open
PR.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Follow-up: `headroom.pricing.counterfactual.resolve_rates(...,
long_context=True)` reads the same LiteLLM fields for its
cache-economics estimates and is not changed here.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 21:44:39 -07:00

69 lines
2.8 KiB
Python

"""Pick a model litellm actually prices, instead of hardcoding one it may retire.
``litellm.model_cost`` is downloaded from GitHub at import time, so it is live
third-party data. BerriAI prunes retired models from it: on 2026-09-23
``claude-sonnet-4-20250514`` disappeared and every test that priced it began
failing with ``KeyError: 'input_cost_per_token'`` — on every open pull request
at once, with no change on our side.
The model id in those tests is incidental. They assert that Headroom's cost
arithmetic agrees with litellm's numbers, not that any particular model is
priced correctly, so the fix is to stop naming a specific release and instead
ask for *a* model carrying the fields the test needs.
Pinning to the copy of the table vendored in the litellm wheel is not the
alternative it looks like: the two maps are complementary, not ordered. The
vendored map keeps retired ids but predates current models
(``claude-sonnet-5``), and its older entries lack newer fields such as
``input_cost_per_token_above_200k_tokens`` entirely.
"""
from __future__ import annotations
import pytest
#: Preference order, newest first. A test takes the first entry that carries
#: every field it needs, so retiring one is a no-op until the list runs dry.
_CANDIDATES: tuple[str, ...] = (
"claude-sonnet-4-5-20250929",
"claude-sonnet-4-5",
"claude-sonnet-4-20250514",
"claude-opus-4-5-20251101",
"claude-opus-4-5",
# Last resort. By 2026-10, litellm had dropped the 200k-tier fields from
# every bare Anthropic id; only this Bedrock id still carries the full set,
# at the same rates as claude-sonnet-4-5.
"claude-sonnet-4-5-20250929-v1:0",
)
_BASE_FIELDS = ("input_cost_per_token", "output_cost_per_token")
def anthropic_pricing_model(*required_fields: str) -> str:
"""Return a currently-priced Anthropic model carrying ``required_fields``.
Always includes the base input/output costs. Raises with an actionable
message rather than skipping: if litellm prices none of these, the pricing
tests are not measuring anything and that should be loud.
"""
import litellm
needed = set(_BASE_FIELDS) | set(required_fields)
for model in _CANDIDATES:
info = litellm.model_cost.get(model)
if isinstance(info, dict) and needed <= set(info):
return model
raise AssertionError(
"litellm prices none of the candidate models with the fields "
f"{sorted(needed)}. It most likely retired them from "
"model_prices_and_context_window.json; add a current model id to "
"_CANDIDATES in tests/_pricing_models.py (newest first)."
)
@pytest.fixture(scope="session")
def anthropic_model() -> str:
"""Session fixture wrapper for tests that prefer injection over a constant."""
return anthropic_pricing_model()