mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
feat/https-compose-caddy
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
8c1beb20ae |
fix(pricing): keep the long-context premium when litellm drops the 200k rate (#4109)
## Description
`litellm.model_cost` is downloaded live. It no longer carries
`input_cost_per_token_above_200k_tokens` on any bare Anthropic id. That
broke two things.
1. **Production pricing (bug).** `AnthropicProvider.estimate_cost`
prices through LiteLLM first, so a 300K-token Sonnet 4.5 request was
priced at the base tier: **$0.975 instead of $1.9125**. That is about
half the real cost of exactly the long `[1m]` sessions. Now, when a
request is above the 200K threshold for a tiered model and the model's
LiteLLM entry has no above-threshold rate, the provider prices it from
its own table, which still applies Anthropic's premium (input 2x, output
1.5x, cache 2x). Every other request still goes through LiteLLM
unchanged.
2. **CI on `main` and every open PR.** `tests/_pricing_models.py` could
find no candidate with the full field set, so
`tests/test_cost_card_attribution.py` and
`tests/test_savings_provider_matrix.py` errored at collection. That hid
failure 1, which surfaced as soon as collection worked. The fix adds the
one id that still has every field, the Bedrock Sonnet 4.5 entry
`claude-sonnet-4-5-20250929-v1:0` (same rates), as the last-resort
candidate. These tests only check that Headroom's arithmetic agrees with
LiteLLM's numbers.
Refs #4136
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/providers/anthropic.py`:
`_litellm_lacks_long_context_rate()`; `estimate_cost` skips LiteLLM for
a long-context request whose entry lacks the above-200K rate, and uses
the manual table with its premium.
- `tests/test_providers/test_anthropic.py`: a hermetic regression test
that strips the rate from the LiteLLM entry and asserts $1.9125; it
fails on main.
- `tests/_pricing_models.py`: add `claude-sonnet-4-5-20250929-v1:0` as
the last-resort candidate, with a comment explaining why.
- `tests/test_counterfactual_pricing.py`:
`test_long_context_derives_the_one_hour_rate_and_says_so` priced the
long-context tier of a model chosen for the base fields only. It now
asks `anthropic_pricing_model()` for a model that has that tier. It also
failed on `main`; the collection errors had been hiding it in CI.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest -q -p no:cacheprovider tests/test_cost_card_attribution.py tests/test_savings_provider_matrix.py tests/test_cost_budget_basis.py tests/test_cost_tracker_counterfactual.py tests/test_counterfactual_pricing.py tests/test_savings_ledger.py tests/test_models.py tests/test_tokenizers.py tests/test_realignment_live_multi_turn.py
190 passed, 22 skipped in 4.79s
(on origin/main: 2 collection errors, plus test_long_context_derives_the_one_hour_rate_and_says_so failing)
$ pytest -q tests/test_providers/test_anthropic.py -k LongContext
10 passed (origin/main code: 3 failed, 7 passed: test_litellm_path[300000-...] x2 and the new hermetic test)
$ pytest -q tests/test_providers <all pricing/cost/savings tests>
629 passed, 14 skipped
$ uvx ruff@0.16.8 check / format --check (both files)
All checks passed! / 2 files already formatted
```
## Real Behavior Proof
- Environment: macOS, Python 3.12, repo .venv with litellm 1.101.0 and
the live LiteLLM price map as of 2026-10-09. Clean worktrees of
`origin/main` (
|
||
|
|
0ad5e68ca5 |
fix(tests): ask litellm for a model it prices instead of naming a retired one (#3722)
## What broke `litellm` downloads `model_prices_and_context_window.json` from GitHub **at import time**, so `litellm.model_cost` is live third-party data we do not control. On 2026-09-23 BerriAI pruned retired Anthropic ids from that table. `claude-sonnet-4-20250514` disappeared, and every test that priced it started failing with `KeyError: 'input_cost_per_token'` — on every open PR simultaneously, with no change on our side. `main` was green at `5ff4ea1e` on 09-22 and the identical shards went red on #3720 and #3721, which share that base and touch no pricing code. ## What this does The model id in those tests is **incidental**. They assert that Headroom's cost arithmetic agrees with litellm's numbers — not that any particular model is priced correctly. So `tests/_pricing_models.py` asks for *a* model carrying the fields a test needs, newest first, and raises with an actionable message if litellm prices none of them. ```python MODEL = anthropic_pricing_model( "input_cost_per_token_above_200k_tokens", "cache_creation_input_token_cost", "cache_read_input_token_cost", ) ``` Retiring one candidate is now a no-op until the list runs dry. ## Why not pin to the wheel's vendored copy That was this PR's first attempt and **it was wrong** — I'm recording why, because it looks like the obvious fix. The live table and the copy vendored in the litellm wheel are *complementary, not ordered*: | | live | vendored | |---|---|---| | `claude-sonnet-4-20250514` | ✗ pruned | ✓ | | `claude-sonnet-5` | ✓ | ✗ predates it | | live-only entries | 1743 | — | | vendored-only entries | — | 314 | Pinning fixed the 25 `KeyError`s and **broke `test_proxy_savings_history`**, which prices `claude-sonnet-5`. Taking the union does not rescue it either: the vendored `claude-sonnet-4-20250514` entry is an older schema that has no `input_cost_per_token_above_200k_tokens` field at all, so `test_cost_card_attribution` still fails against it. ## A second, separate breakage in the same data drop `test_provider_prices_models_its_table_never_covered[groq/llama-3.3-70b-versatile]` was failing for a different reason: litellm dropped that model too, so `_get_pricing` fell through to the `$2.50/$10.00` GPT-4o default — which that test's own comment names as the "unknown default" it exists to catch. Repointed at `groq/llama-guard-3-8b`, which litellm still prices. **The product-side problem is filed separately as #3732**: an unresolvable model is silently billed at the GPT-4o tier, so cost tracking and `check_budget` — a hard control that refuses requests — can act on a fabricated number. For that groq model the default overstated output cost by ~12x. ## Verification Same six modules, same machine: ``` origin/main 25 failed, 82 passed this branch 107 passed ``` Ruff 0.16.7 `check` + `format --check` clean. ## Prior art #3683 / #3684 was the same class of failure (litellm started billing DeepSeek off-peak) and was patched narrowly at the assertion. This removes the category for model *identity*; #3732 covers what happens in production when identity goes missing. |