Commit Graph
2 Commits
Author SHA1 Message Date
Tejas ChopraandClaude Opus 5.5 8c1beb20ae fix(pricing): keep the long-context premium when litellm drops the 200k rate (#4109)
## Description

`litellm.model_cost` is downloaded live. It no longer carries
`input_cost_per_token_above_200k_tokens` on any bare Anthropic id. That
broke two things.

1. **Production pricing (bug).** `AnthropicProvider.estimate_cost`
prices through LiteLLM first, so a 300K-token Sonnet 4.5 request was
priced at the base tier: **$0.975 instead of $1.9125**. That is about
half the real cost of exactly the long `[1m]` sessions. Now, when a
request is above the 200K threshold for a tiered model and the model's
LiteLLM entry has no above-threshold rate, the provider prices it from
its own table, which still applies Anthropic's premium (input 2x, output
1.5x, cache 2x). Every other request still goes through LiteLLM
unchanged.
2. **CI on `main` and every open PR.** `tests/_pricing_models.py` could
find no candidate with the full field set, so
`tests/test_cost_card_attribution.py` and
`tests/test_savings_provider_matrix.py` errored at collection. That hid
failure 1, which surfaced as soon as collection worked. The fix adds the
one id that still has every field, the Bedrock Sonnet 4.5 entry
`claude-sonnet-4-5-20250929-v1:0` (same rates), as the last-resort
candidate. These tests only check that Headroom's arithmetic agrees with
LiteLLM's numbers.

Refs #4136

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/providers/anthropic.py`:
`_litellm_lacks_long_context_rate()`; `estimate_cost` skips LiteLLM for
a long-context request whose entry lacks the above-200K rate, and uses
the manual table with its premium.
- `tests/test_providers/test_anthropic.py`: a hermetic regression test
that strips the rate from the LiteLLM entry and asserts $1.9125; it
fails on main.
- `tests/_pricing_models.py`: add `claude-sonnet-4-5-20250929-v1:0` as
the last-resort candidate, with a comment explaining why.
- `tests/test_counterfactual_pricing.py`:
`test_long_context_derives_the_one_hour_rate_and_says_so` priced the
long-context tier of a model chosen for the base fields only. It now
asks `anthropic_pricing_model()` for a model that has that tier. It also
failed on `main`; the collection errors had been hiding it in CI.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest -q -p no:cacheprovider tests/test_cost_card_attribution.py tests/test_savings_provider_matrix.py tests/test_cost_budget_basis.py tests/test_cost_tracker_counterfactual.py tests/test_counterfactual_pricing.py tests/test_savings_ledger.py tests/test_models.py tests/test_tokenizers.py tests/test_realignment_live_multi_turn.py
190 passed, 22 skipped in 4.79s
(on origin/main: 2 collection errors, plus test_long_context_derives_the_one_hour_rate_and_says_so failing)
$ pytest -q tests/test_providers/test_anthropic.py -k LongContext
10 passed            (origin/main code: 3 failed, 7 passed: test_litellm_path[300000-...] x2 and the new hermetic test)
$ pytest -q tests/test_providers <all pricing/cost/savings tests>
629 passed, 14 skipped
$ uvx ruff@0.16.8 check / format --check (both files)
All checks passed! / 2 files already formatted
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, repo .venv with litellm 1.101.0 and
the live LiteLLM price map as of 2026-10-09. Clean worktrees of
`origin/main` (a9815bbb) and of this branch.
- Exact command / steps: `python -c "from headroom.providers.anthropic
import AnthropicProvider as A; p=A();
print([round(p.estimate_cost(i,5000,'claude-sonnet-4-5',c),4) for i,c in
((100000,0),(300000,0),(300000,150000))])"` in each worktree. Then the
branch's `TestLongContextPricing` run against each.
- Observed result: on `main` the output is `[0.375, 0.975, 0.57]`: both
300K requests are billed at the base tier, about half the real cost. On
this branch it is `[0.375, 1.9125, 1.1025]`, Anthropic's published
long-context price; the 100K request is unchanged.
`TestLongContextPricing` gives 3 failed / 7 passed against main's code
and 10 passed on the branch. An untiered model (Opus 4.5, 300K) still
prices flat at 1.625.
- Not tested: A real Anthropic invoice. The expected values are
Anthropic's published rates, which the manual table and LiteLLM's former
entry both carry.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A; cost reporting only, no request change.
- Stable/default behavior changed: Yes, only for Anthropic tiered models
above 200K input tokens whose LiteLLM entry lacks the long-context rate.
Those now report the premium cost instead of half of it.
- Kill switch / disable path: N/A.
- Unsafe override required: No.
- Qualification impact: Unblocks the Python test shards on every open
PR.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Follow-up: `headroom.pricing.counterfactual.resolve_rates(...,
long_context=True)` reads the same LiteLLM fields for its
cache-economics estimates and is not changed here.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 21:44:39 -07:00
Tejas Chopra 0ad5e68ca5 fix(tests): ask litellm for a model it prices instead of naming a retired one (#3722)
## What broke

`litellm` downloads `model_prices_and_context_window.json` from GitHub
**at import time**, so `litellm.model_cost` is live third-party data we
do not control.

On 2026-09-23 BerriAI pruned retired Anthropic ids from that table.
`claude-sonnet-4-20250514` disappeared, and every test that priced it
started failing with `KeyError: 'input_cost_per_token'` — on every open
PR simultaneously, with no change on our side. `main` was green at
`5ff4ea1e` on 09-22 and the identical shards went red on #3720 and
#3721, which share that base and touch no pricing code.

## What this does

The model id in those tests is **incidental**. They assert that
Headroom's cost arithmetic agrees with litellm's numbers — not that any
particular model is priced correctly. So `tests/_pricing_models.py` asks
for *a* model carrying the fields a test needs, newest first, and raises
with an actionable message if litellm prices none of them.

```python
MODEL = anthropic_pricing_model(
    "input_cost_per_token_above_200k_tokens",
    "cache_creation_input_token_cost",
    "cache_read_input_token_cost",
)
```

Retiring one candidate is now a no-op until the list runs dry.

## Why not pin to the wheel's vendored copy

That was this PR's first attempt and **it was wrong** — I'm recording
why, because it looks like the obvious fix.

The live table and the copy vendored in the litellm wheel are
*complementary, not ordered*:

| | live | vendored |
|---|---|---|
| `claude-sonnet-4-20250514` | ✗ pruned | ✓ |
| `claude-sonnet-5` | ✓ | ✗ predates it |
| live-only entries | 1743 | — |
| vendored-only entries | — | 314 |

Pinning fixed the 25 `KeyError`s and **broke
`test_proxy_savings_history`**, which prices `claude-sonnet-5`. Taking
the union does not rescue it either: the vendored
`claude-sonnet-4-20250514` entry is an older schema that has no
`input_cost_per_token_above_200k_tokens` field at all, so
`test_cost_card_attribution` still fails against it.

## A second, separate breakage in the same data drop


`test_provider_prices_models_its_table_never_covered[groq/llama-3.3-70b-versatile]`
was failing for a different reason: litellm dropped that model too, so
`_get_pricing` fell through to the `$2.50/$10.00` GPT-4o default — which
that test's own comment names as the "unknown default" it exists to
catch. Repointed at `groq/llama-guard-3-8b`, which litellm still prices.

**The product-side problem is filed separately as #3732**: an
unresolvable model is silently billed at the GPT-4o tier, so cost
tracking and `check_budget` — a hard control that refuses requests — can
act on a fabricated number. For that groq model the default overstated
output cost by ~12x.

## Verification

Same six modules, same machine:

```
origin/main          25 failed, 82 passed
this branch                    107 passed
```

Ruff 0.16.7 `check` + `format --check` clean.

## Prior art

#3683 / #3684 was the same class of failure (litellm started billing
DeepSeek off-peak) and was patched narrowly at the assertion. This
removes the category for model *identity*; #3732 covers what happens in
production when identity goes missing.
2026-09-23 11:03:26 +02:00