fix(pricing): add Claude Haiku 5.5 rates, 1M context and its 100K long-prompt card (#4037)

## Current maintainer qualification

Reviewed head 595803d786 after the
current-main sync. The PR-specific pricing implementation is unchanged
from the previous reviewed head.

Independent validation: 202 focused Python pricing/provider/savings
tests passed, four skipped. The Rust proxy binary built from this head.
Through the running Rust HTTP proxy and a separate loopback upstream,
requests carrying 100,000 and 100,001 input tokens produced the expected
native-ledger request costs of $0.0105 and $0.0525005 respectively,
including 1,000 output tokens. The default stable rollout channel was
used, with no unsafe feature override.

The HTTP replay uses controlled upstream usage; it does not claim live
Anthropic inference or independently verified provider billing. The
previously recorded missing native HTTP boundary check is now covered.

Current main is an ancestor. Hosted checks on this head are terminal: 49
successful, 2 skipped, with no failures. The prior shared Rust advisory
hold is cleared. Final merge approval remains human-only; nothing has
been merged by this review.
## Description

Claude Haiku 5.5 (`claude-haiku-5-5`) is released and had no rows in
Headroom's built-in Anthropic tables or the Rust proxy's vendored
LiteLLM snapshot. This PR adds it, including its second rate card for
prompts over 100K tokens, and makes every cost path read a model's
long-context threshold from its own catalog row instead of assuming
200K.

**Why the missing rows mispriced instead of failing:** without LiteLLM
data for the model, `AnthropicProvider` falls through to the `haiku`
pattern default ($0.80 / $4, 200K), which is 8x the real input rate and
a fifth of the real window. The Rust `context_window_for` is
exact-match, so the id fell through to the 128K default.

Haiku 5.5's rate card ([Anthropic
pricing](https://platform.claude.com/docs/en/about-claude/pricing),
matching upstream LiteLLM's `claude-haiku-5-5` rows): $0.10 / $0.50
input / output and $0.01 cache read for prompts up to 100K tokens; every
rate is 5x ($0.50 / $2.50 / $0.05) above 100K. 1M context, 128K max
output.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

All values in $ / MTok (input / output / cache read).

| Model | Before (fallback) | After | Context |
|---|---|---|---|
| `claude-haiku-5-5`, prompt <= 100K | *(no row: `haiku` tier default)*
$0.80 / $4 / $0.08 | **$0.10 / $0.50 / $0.01** | 200K -> **1M** |
| `claude-haiku-5-5`, prompt > 100K | $0.80 / $4 / $0.08 | **$0.50 /
$2.50 / $0.05** | |

- `headroom/providers/anthropic.py`:
- Adds Haiku 5.5 to `ANTHROPIC_PRICING` and `ANTHROPIC_CONTEXT_LIMITS`.
- The long-context premium becomes a small per-model table,
`_LONG_CONTEXT_TIERS` (prefixes, threshold, multipliers), so Haiku 5.5's
100K card sits next to the existing Sonnet 4 / 4.5 200K tier. The Sonnet
tier's threshold and multipliers are unchanged.
  ```diff
   _apply_long_context_premium(model, pricing, input_tokens)
- if input_tokens <= 200K or model not in Sonnet 4 / 4.5: return pricing
  -  return pricing * Sonnet premium
  +  for prefixes, threshold, premium in _LONG_CONTEXT_TIERS:
  +    if model starts with prefixes:
+ return pricing if input_tokens <= threshold else pricing * premium
  +  return pricing
  ```
- `crates/headroom-proxy/data/model_prices_and_context_window.json`:
appends the 18 upstream LiteLLM `claude-haiku-5-5` keys verbatim
(first-party, Bedrock regional/global/Mantle/GovCloud, Vertex, Azure,
OpenRouter, Perplexity). No existing entry changes or is removed;
extended rather than re-pulled for the reasons in #3841.
- **Per-model long-context tiers on the stats paths** (Devin finding 1).
Each catalog row publishes at most one `*_above_<N>k_tokens` tier (100K,
128K, 200K, 256K or 272K); the threshold now comes from the row, not a
shared 200K constant.
- `crates/headroom-proxy/src/observability/pricing.rs`: the price book
stores a `PriceCard` (base card + optional long-context card parsed from
the row's tier fields; a missing tier field falls back to that rate's
base price). `lookup` still returns the base card; `lookup_card` returns
both, and `PriceCard::for_prompt` picks one.
- `crates/headroom-proxy/src/observability/ledger.rs`: `record` picks
the card from the billed prompt (uncached + cache read + cache write)
before pricing spend and savings.
- `headroom/pricing/counterfactual.py`: `long_context_threshold(model)`
/ `long_context_tier(info)`; `resolve_rates` reads the row's own tier,
including its combined 1h + long-tier write rate where published.
`CacheMix.is_long_context` takes the threshold.
- `headroom/proxy/cost.py`, `headroom/proxy/savings_tracker.py`: bucket
and price at the model's own threshold.
- Side effect on other tiered rows, both intended: the Rust stats now
bill Sonnet 4 / 4.5 above 200K at its long card (the Python cost card
already did), and both paths switch GPT-5.x at 272K and the 128K-tier
rows at 128K with their own rates. Before, those rows crossed a 200K
line, found no `*_above_200k_tokens` fields and stayed on the base card.
- **Compression that crosses a tier credits the whole avoided premium**
(Devin re-review). When compression keeps a prompt under the threshold
its uncompressed form would have passed, the saving is the uncompressed
bill minus the actual bill, not the removed tokens at the cheap rate.
Rust: `PriceCard::compression_savings_usd` (input and output); Python:
`long_context_premium_avoided_usd`, used by
`estimate_request_savings_usd` and by `CostTracker` (which also adds the
output premium). The list companion still prices only the removed
tokens, so budget enforcement stays monotonic. The warm-prefix test in
`tests/test_cost_card_attribution.py` sent 200K with 10K saved, which
crosses Sonnet 4.5's tier; its counts are halved (same mix) so it stays
inside one tier.
- **Short ids no longer inherit a newer release** (Devin finding 2).
`get_context_limit` and `_get_pricing` matched substrings in both
directions, so `claude-haiku-5` resolved to Haiku 5.5 and
`claude-sonnet-4` to Sonnet 4.6's 1M window. `_is_release_of` accepts a
longer id containing the known one (dated or provider-wrapped) or an
undated / `-latest` alias of a dated key, and rejects ids one version
segment apart in either direction.
- `crates/headroom-proxy/src/compression/model_limits.rs`: the 1M-window
test now covers `claude-haiku-5-5`, `us.anthropic.claude-haiku-5-5` and
`vertex_ai/claude-haiku-5-5`.
- Rust tests: Haiku 5.5 base and >100K cards, per-model threshold
(Sonnet 4.5 at 200K), flat-rated model, missing tier field; ledger bills
150K in / 5K out at the long card, and cached tokens count toward the
threshold.
- `tests/test_providers/test_anthropic.py`: Haiku 5.5 price, dated-id
and 1M-context cases, cost cases at and above the 100K threshold, the
`claude-haiku-5` regression and an `_is_release_of` table.
- `tests/test_cost_card_attribution.py`,
`tests/test_savings_provider_matrix.py`: Haiku 5.5 cost card and savings
at 90K vs 150K.
- `tests/test_counterfactual_pricing.py`: the long-tier 1h test now
reads the published combined rate, and a synthetic row covers the
derived case.
- `tests/test_provider_model_fallback.py`: the "future haiku" probe
moves from `claude-haiku-5-20260101` to `claude-haiku-6-20270101`,
mirroring the opus probe change in #3841.
- `tests/test_thinking_compactor_version_gate.py`: `claude-haiku-5-5` is
treated as a Claude 5 model.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`) (ruff 0.16.8, on the changed
files)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ HOME=$(mktemp -d) pytest -q <119 pricing / cost / savings / stats / provider / learn modules>
2097 passed, 18 skipped, 5 warnings

$ cargo test -p headroom-proxy
577 passed, 0 failed (41 suites)
$ cargo clippy -p headroom-proxy --all-targets -- -D warnings
(clean)
$ cargo fmt -p headroom-proxy --check
(clean)

$ ruff check <changed .py files>          # ruff 0.16.8
All checks passed!
$ ruff format --check <changed .py files>
(clean)
$ mypy headroom --ignore-missing-imports  # mypy 1.20.2
Success: no issues found in 566 source files
```

`HOME` is isolated because a developer `~/.headroom/models.json` leaks
into `TestAnthropicConfigLoading`.

## Real Behavior Proof

- Environment: macOS, the repo `.venv`. `origin/main` and this PR head
were each checked out as a git worktree and run with the same
interpreter and an isolated `HOME`.
- Exact command / steps: a script built `AnthropicProvider()`, stubbed
`estimate_cost_from_tokens` and `_get_litellm_clients` to force the
built-in fallback (the LiteLLM-unavailable or offline path), then priced
two requests and read the context limit. For the Rust side it applied
`context_window_for` semantics (exact key, `mode == "chat"`,
`max_input_tokens`, 128K default) to each worktree's vendored JSON. For
the review fixes, a second script ran against the first PR commit
(`b0503796`) and this head: the fallback lookups for short ids, a
`CostTracker` 150K-in / 5K-out Haiku 5.5 turn with 10K saved, and
`resolve_rates(..., long_context=True)`; the Rust ledger case is the
`long_prompt_bills_at_the_models_long_context_card` test.
- Observed result: Haiku 5.5 now prices and sizes correctly on every
path, and short ids keep their tier default (tables below).
  ```text
origin/main this PR
context limit (Python fallback) 200,000 (inferred 'haiku' tier)
1,000,000
50K in / 5K out $0.060000 $0.007500
150K in (120K cached) / 5K out $0.053600 $0.033500 (100K+ card)
Rust context_window_for 128,000 1,000,000
  ```
Hand check of the last row: 30K × $0.50 + 120K × $0.05 + 5K × $2.50 per
MTok = $0.0335.

  Review fixes, first PR commit vs this head:
  ```text
b0503796 this head
get_context_limit("claude-haiku-5") 1,000,000 200,000 ('haiku' tier)
_get_pricing("claude-haiku-5") $0.10 / $0.50 / $0.01 $0.80 / $4 / $0.08
get_context_limit("claude-sonnet-4") 1,000,000 (Sonnet 4.6) 200,000
(Sonnet 4)
CostTracker 150K in / 5K out: output cost $0.002500 $0.012500
CostTracker 10K saved: cache-aware savings $0.001000 $0.005000
resolve_rates(haiku-5-5, long).uncached $0.10 / MTok $0.50 / MTok
Rust ledger 150K in / 5K out $0.0175 (base card, computed) $0.0875
(ledger unit test)
  ```

Compression crossing the tier, Haiku 5.5, 110K uncached compressed to
90K (unit tests on this head; the "before" column is the old rule's
arithmetic):
  ```text
removed tokens at base this head
Rust ledger compression savings $0.002 $0.046
estimate_request_savings_usd["compression"] $0.002 $0.046
CostTracker cache-aware savings (1K output) $0.002 $0.048 (incl. $0.002
output premium)
  ```
- Not tested: live provider API billing. The controlled native HTTP
replay now covers the 100K boundary, as documented above.
- The Rust proxy end to end with a live >100K Haiku 5.5 request; the
ledger is covered by unit tests only.
- Bedrock-style ids (`us.anthropic.claude-haiku-5-5`) get the base card
without the 100K premium on the Python fallback path, the same
limitation the Sonnet 4 / 4.5 tier already has.
  - No live API call was made.

## Maintainer follow-up qualification

Latest qualification head: `f885cc4c8afa574d867820583886687b0bae12a9`.
The pricing code and the 413-Python/73-Rust/TCP-replay validation below
are unchanged from `a45ae8c3e`. The first hosted shard-3 run exposed
five existing main integration failures:
`test_proxy_dependency_gate_matches_onnx_platform_markers` still patched
a removed eager-import hook. The follow-up aligns those existing tests
with lazy `find_spec` discovery and its missing-spec return value. The
five failures reproduce locally before; all eight dependency-contract
cases and 91 CLI/stateless/logging tests pass after, with
Ruff/format/whitespace passing. The current-head CI must finish before
approval.

Published maintainer follow-up at `a45ae8c3e` (normal push from the
unchanged `c326e2046` head): forward generated `output_tokens` into the
shared compression estimator through Prometheus and the durable lifetime
recorder, and add the long-card minus base-card output premium only when
compression crosses the model's threshold. `output_tokens_saved` remains
a separate output-shaping layer. The documented unpriced direct tracker
fallback is preserved.

Integrated current main `463c1a43e30fc33cb256a3f455daf28043e53b05` to
resolve the pricing conflict. The resolution retains canonical
model-specific cache billing, independent cache-rate provenance and
explicit-zero rates, while probing missing billed cache rates at the
model's own long-context threshold. New regressions cover both contracts
together.

Verification after the conflict resolution:

- 413 Python tests passed; one known Windows directory-fsync test
deselected after it failed identically on the unchanged PR head; three
existing warnings. This includes 20 new
output-premium/boundary/metadata/merged-cache-billing cases, pricing,
provider fallback, persistent metrics, cache TTL, tool-schema and output
savings suites.
- 73 Rust observability tests passed: `cargo test -p headroom-proxy
--lib observability --locked`.
- Ruff lint/format and whitespace checks pass. Focused Linux-platform
mypy passes for the four pricing/metrics modules. Full local mypy in
this extras-rich environment reports 19 errors in
`relevance/embedding.py`, `memory/backends/direct_mem0.py`,
`integrations/langchain/chat_model.py` and
`integrations/crewai/agents.py`; it is not claimed green.
- Before the fix, both public priced recorders report $0.046 for
110K→90K input with 1K generated output; after, both report $0.048.
Noncrossing and no-output controls pass, and separately saved output is
not conflated with the avoided premium.

Actual Windows/Python 3.12 TCP replay: real uvicorn proxy, separate
loopback HTTP upstream, and the real content-router/SmartCrusher
pipeline. A 304,903-character JSON tool result becomes 139,003
characters, saving 55,974 locally counted tokens. The controlled
upstream reports 90,000 input and 1,000 output tokens. On the unchanged
PR head the lifetime estimator reports $0.063987 and fails; with the fix
and after the main integration it reports the expected $0.065987. The
persistent Prometheus compression metric matches, and output-shaping
savings remain zero. This exercises real compression and handlers with
controlled usage/rates; no live provider billing or Rust HTTP runtime is
claimed.

Current-head GitHub CI and edited review bodies still require
qualification before approval.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: cost estimates and the context window
reported for `claude-haiku-5-5` and the 18 added LiteLLM keys;
long-prompt spend and savings on tiered models now use each model's own
threshold; short ids such as `claude-haiku-5` fall back to their tier
default instead of a newer release
- Kill switch / disable path: n/a (data tables)
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (n/a: no
user-facing docs list per-model prices)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

n/a

## Additional Notes

The 100K card applies on every path: the Python fallback
(`_LONG_CONTEXT_TIERS`), LiteLLM's own estimate, the cost card and
savings counterfactual, and the Rust stats ledger.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4037"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <tejas@headroomlabs.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kenneth Wong
2026-10-28 17:54:57 -05:00
committed by GitHub
co-authored by Kenneth Wong JerrettDavis Tejas Chopra Claude Opus 5.5
parent 79181ca0fe
commit 703755e873
16 changed files with 1902 additions and 120 deletions
@@ -43753,5 +43753,763 @@
"thinking_always_on": true,
"prompt_cache_min_tokens": 512,
"source": "https://cloud.google.com/gemini-enterprise-agent-platform/generative-ai/pricing"
},
"openrouter/anthropic/claude-haiku-5.5": {
"input_cost_per_token": 1e-07,
"output_cost_per_token": 5e-07,
"cache_read_input_token_cost": 1e-08,
"cache_creation_input_token_cost": 1.25e-07,
"cache_creation_input_token_cost_above_1hr": 2e-07,
"input_cost_per_token_above_100k_tokens": 5e-07,
"output_cost_per_token_above_100k_tokens": 2.5e-06,
"cache_read_input_token_cost_above_100k_tokens": 5e-08,
"cache_creation_input_token_cost_above_100k_tokens": 6.25e-07,
"cache_creation_input_token_cost_above_1hr_above_100k_tokens": 1e-06,
"litellm_provider": "openrouter",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"source": "https://openrouter.ai/api/v1/models",
"supports_function_calling": true,
"supports_tool_choice": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_vision": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_web_search": true,
"supports_adaptive_thinking": true,
"prompt_cache_min_tokens": 512,
"supports_sampling_params": false
},
"claude-haiku-5-5": {
"input_cost_per_token_above_100k_tokens_batches": 2.5e-07,
"output_cost_per_token_above_100k_tokens_batches": 1.25e-06,
"cache_creation_input_token_cost_above_100k_tokens_batches": 3.125e-07,
"cache_read_input_token_cost_above_100k_tokens_batches": 2.5e-08,
"supports_anthropic_compaction": true,
"cache_creation_input_token_cost": 1.25e-07,
"cache_creation_input_token_cost_above_1hr": 2e-07,
"cache_creation_input_token_cost_batches": 6.25e-08,
"cache_read_input_token_cost": 1e-08,
"cache_read_input_token_cost_batches": 5e-09,
"input_cost_per_token": 1e-07,
"input_cost_per_token_batches": 5e-08,
"litellm_provider": "anthropic",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 5e-07,
"output_cost_per_token_batches": 2.5e-07,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_adaptive_thinking": true,
"supports_mid_conversation_system": true,
"supports_assistant_prefill": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_native_structured_output": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_max_reasoning_effort": true,
"provider_specific_entry": {
"us": 1.1
},
"supports_output_config": true,
"supports_forced_tool_use": true,
"thinking_always_on": false,
"prompt_cache_min_tokens": 512,
"source": "https://platform.claude.com/docs/en/models/haiku-5-5/overview",
"supports_web_search": true,
"input_cost_per_token_above_100k_tokens": 5e-07,
"output_cost_per_token_above_100k_tokens": 2.5e-06,
"cache_creation_input_token_cost_above_100k_tokens": 6.25e-07,
"cache_creation_input_token_cost_above_1hr_above_100k_tokens": 1e-06,
"cache_read_input_token_cost_above_100k_tokens": 5e-08
},
"bedrock_mantle/anthropic.claude-haiku-5-5": {
"input_cost_per_token_above_100k_tokens": 5.5e-07,
"output_cost_per_token_above_100k_tokens": 2.75e-06,
"cache_creation_input_token_cost_above_100k_tokens": 6.875e-07,
"cache_creation_input_token_cost_above_1hr_above_100k_tokens": 1.1e-06,
"cache_read_input_token_cost_above_100k_tokens": 5.5e-08,
"bedrock_converse_supports_strict_tools": false,
"cache_creation_input_token_cost": 1.375e-07,
"cache_creation_input_token_cost_above_1hr": 2.2e-07,
"cache_read_input_token_cost": 1.1e-08,
"input_cost_per_token": 1.1e-07,
"litellm_provider": "bedrock_mantle",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 5.5e-07,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_adaptive_thinking": true,
"supports_mid_conversation_system": true,
"supports_assistant_prefill": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"bedrock_output_config_effort_ceiling": "xhigh",
"supports_parallel_tool_use_config": true,
"supports_forced_tool_use": true,
"thinking_always_on": false,
"prompt_cache_min_tokens": 512,
"source": "https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-anthropic-claude-haiku-5-5.html"
},
"bedrock_mantle/us-gov-west-1/anthropic.claude-haiku-5-5": {
"input_cost_per_token_above_100k_tokens": 6e-07,
"output_cost_per_token_above_100k_tokens": 3e-06,
"cache_creation_input_token_cost_above_100k_tokens": 7.5e-07,
"cache_creation_input_token_cost_above_1hr_above_100k_tokens": 1.2e-06,
"cache_read_input_token_cost_above_100k_tokens": 6e-08,
"bedrock_converse_supports_strict_tools": false,
"bedrock_output_config_effort_ceiling": "xhigh",
"cache_creation_input_token_cost": 1.5e-07,
"cache_creation_input_token_cost_above_1hr": 2.4e-07,
"cache_read_input_token_cost": 1.2e-08,
"input_cost_per_token": 1.2e-07,
"litellm_provider": "bedrock_mantle",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 6e-07,
"supports_forced_tool_use": true,
"thinking_always_on": false,
"prompt_cache_min_tokens": 512,
"supports_adaptive_thinking": true,
"supports_assistant_prefill": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_max_reasoning_effort": true,
"supports_mid_conversation_system": true,
"supports_native_structured_output": false,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"source": "https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-anthropic-claude-haiku-5-5.html"
},
"anthropic.claude-haiku-5-5": {
"input_cost_per_token_above_100k_tokens": 5e-07,
"output_cost_per_token_above_100k_tokens": 2.5e-06,
"cache_creation_input_token_cost_above_100k_tokens": 6.25e-07,
"cache_creation_input_token_cost_above_1hr_above_100k_tokens": 1e-06,
"cache_read_input_token_cost_above_100k_tokens": 5e-08,
"bedrock_converse_supports_strict_tools": false,
"cache_creation_input_token_cost": 1.25e-07,
"cache_creation_input_token_cost_above_1hr": 2e-07,
"cache_read_input_token_cost": 1e-08,
"input_cost_per_token": 1e-07,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 5e-07,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_adaptive_thinking": true,
"supports_mid_conversation_system": true,
"supports_assistant_prefill": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"bedrock_output_config_effort_ceiling": "xhigh",
"supports_parallel_tool_use_config": true,
"supports_forced_tool_use": true,
"thinking_always_on": false,
"prompt_cache_min_tokens": 512,
"source": "https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonBedrockFoundationModels/current/index.json"
},
"apac.anthropic.claude-haiku-5-5": {
"input_cost_per_token_above_100k_tokens": 5.5e-07,
"output_cost_per_token_above_100k_tokens": 2.75e-06,
"cache_creation_input_token_cost_above_100k_tokens": 6.875e-07,
"cache_creation_input_token_cost_above_1hr_above_100k_tokens": 1.1e-06,
"cache_read_input_token_cost_above_100k_tokens": 5.5e-08,
"bedrock_converse_supports_strict_tools": false,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"supports_adaptive_thinking": true,
"supports_mid_conversation_system": true,
"supports_assistant_prefill": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"bedrock_output_config_effort_ceiling": "xhigh",
"supports_parallel_tool_use_config": true,
"supports_forced_tool_use": true,
"thinking_always_on": false,
"prompt_cache_min_tokens": 512,
"cache_creation_input_token_cost_above_1hr": 2.2e-07,
"cache_creation_input_token_cost": 1.375e-07,
"input_cost_per_token": 1.1e-07,
"output_cost_per_token": 5.5e-07,
"cache_read_input_token_cost": 1.1e-08,
"source": "https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonBedrockFoundationModels/current/index.json"
},
"au.anthropic.claude-haiku-5-5": {
"input_cost_per_token_above_100k_tokens": 5.5e-07,
"output_cost_per_token_above_100k_tokens": 2.75e-06,
"cache_creation_input_token_cost_above_100k_tokens": 6.875e-07,
"cache_creation_input_token_cost_above_1hr_above_100k_tokens": 1.1e-06,
"cache_read_input_token_cost_above_100k_tokens": 5.5e-08,
"bedrock_converse_supports_strict_tools": false,
"cache_creation_input_token_cost": 1.375e-07,
"cache_creation_input_token_cost_above_1hr": 2.2e-07,
"cache_read_input_token_cost": 1.1e-08,
"input_cost_per_token": 1.1e-07,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 5.5e-07,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_adaptive_thinking": true,
"supports_mid_conversation_system": true,
"supports_assistant_prefill": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"bedrock_output_config_effort_ceiling": "xhigh",
"supports_parallel_tool_use_config": true,
"supports_forced_tool_use": true,
"thinking_always_on": false,
"prompt_cache_min_tokens": 512,
"source": "https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonBedrockFoundationModels/current/index.json"
},
"azure_ai/claude-haiku-5-5": {
"input_cost_per_token_above_100k_tokens": 5e-07,
"output_cost_per_token_above_100k_tokens": 2.5e-06,
"cache_creation_input_token_cost_above_100k_tokens": 6.25e-07,
"cache_creation_input_token_cost_above_1hr_above_100k_tokens": 1e-06,
"cache_read_input_token_cost_above_100k_tokens": 5e-08,
"supports_mid_conversation_system": true,
"cache_creation_input_token_cost": 1.25e-07,
"cache_creation_input_token_cost_above_1hr": 2e-07,
"cache_read_input_token_cost": 1e-08,
"deprecation_date": "2027-09-29",
"input_cost_per_token": 1e-07,
"litellm_provider": "azure_ai",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 5e-07,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_adaptive_thinking": true,
"supports_assistant_prefill": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_max_reasoning_effort": true,
"supports_forced_tool_use": true,
"thinking_always_on": false,
"prompt_cache_min_tokens": 512,
"source": "https://platform.claude.com/docs/en/models/haiku-5-5/migration-guide"
},
"bedrock/us-gov-east-1/anthropic.claude-haiku-5-5": {
"input_cost_per_token_above_100k_tokens": 6e-07,
"output_cost_per_token_above_100k_tokens": 3e-06,
"cache_creation_input_token_cost_above_100k_tokens": 7.5e-07,
"cache_creation_input_token_cost_above_1hr_above_100k_tokens": 1.2e-06,
"cache_read_input_token_cost_above_100k_tokens": 6e-08,
"bedrock_converse_supports_strict_tools": false,
"bedrock_output_config_effort_ceiling": "xhigh",
"cache_creation_input_token_cost": 1.5e-07,
"cache_creation_input_token_cost_above_1hr": 2.4e-07,
"cache_read_input_token_cost": 1.2e-08,
"input_cost_per_token": 1.2e-07,
"litellm_provider": "bedrock",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 6e-07,
"supports_forced_tool_use": true,
"thinking_always_on": false,
"prompt_cache_min_tokens": 512,
"source": "https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonBedrockFoundationModels/current/index.json",
"supports_adaptive_thinking": true,
"supports_assistant_prefill": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_max_reasoning_effort": true,
"supports_mid_conversation_system": true,
"supports_native_structured_output": false,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true
},
"bedrock/us-gov-west-1/anthropic.claude-haiku-5-5": {
"input_cost_per_token_above_100k_tokens": 6e-07,
"output_cost_per_token_above_100k_tokens": 3e-06,
"cache_creation_input_token_cost_above_100k_tokens": 7.5e-07,
"cache_creation_input_token_cost_above_1hr_above_100k_tokens": 1.2e-06,
"cache_read_input_token_cost_above_100k_tokens": 6e-08,
"bedrock_converse_supports_strict_tools": false,
"bedrock_output_config_effort_ceiling": "xhigh",
"cache_creation_input_token_cost": 1.5e-07,
"cache_creation_input_token_cost_above_1hr": 2.4e-07,
"cache_read_input_token_cost": 1.2e-08,
"input_cost_per_token": 1.2e-07,
"litellm_provider": "bedrock",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 6e-07,
"supports_forced_tool_use": true,
"thinking_always_on": false,
"prompt_cache_min_tokens": 512,
"source": "https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonBedrockFoundationModels/current/index.json",
"supports_adaptive_thinking": true,
"supports_assistant_prefill": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_max_reasoning_effort": true,
"supports_mid_conversation_system": true,
"supports_native_structured_output": false,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true
},
"eu.anthropic.claude-haiku-5-5": {
"input_cost_per_token_above_100k_tokens": 5.5e-07,
"output_cost_per_token_above_100k_tokens": 2.75e-06,
"cache_creation_input_token_cost_above_100k_tokens": 6.875e-07,
"cache_creation_input_token_cost_above_1hr_above_100k_tokens": 1.1e-06,
"cache_read_input_token_cost_above_100k_tokens": 5.5e-08,
"bedrock_converse_supports_strict_tools": false,
"cache_creation_input_token_cost": 1.375e-07,
"cache_creation_input_token_cost_above_1hr": 2.2e-07,
"cache_read_input_token_cost": 1.1e-08,
"input_cost_per_token": 1.1e-07,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 5.5e-07,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_adaptive_thinking": true,
"supports_mid_conversation_system": true,
"supports_assistant_prefill": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"bedrock_output_config_effort_ceiling": "xhigh",
"supports_parallel_tool_use_config": true,
"supports_forced_tool_use": true,
"thinking_always_on": false,
"prompt_cache_min_tokens": 512,
"source": "https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonBedrockFoundationModels/current/index.json"
},
"global.anthropic.claude-haiku-5-5": {
"input_cost_per_token_above_100k_tokens": 5e-07,
"output_cost_per_token_above_100k_tokens": 2.5e-06,
"cache_creation_input_token_cost_above_100k_tokens": 6.25e-07,
"cache_creation_input_token_cost_above_1hr_above_100k_tokens": 1e-06,
"cache_read_input_token_cost_above_100k_tokens": 5e-08,
"bedrock_converse_supports_strict_tools": false,
"cache_creation_input_token_cost": 1.25e-07,
"cache_creation_input_token_cost_above_1hr": 2e-07,
"cache_read_input_token_cost": 1e-08,
"input_cost_per_token": 1e-07,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 5e-07,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_adaptive_thinking": true,
"supports_mid_conversation_system": true,
"supports_assistant_prefill": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"bedrock_output_config_effort_ceiling": "xhigh",
"supports_parallel_tool_use_config": true,
"supports_forced_tool_use": true,
"thinking_always_on": false,
"prompt_cache_min_tokens": 512,
"source": "https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonBedrockFoundationModels/current/index.json"
},
"jp.anthropic.claude-haiku-5-5": {
"input_cost_per_token_above_100k_tokens": 5.5e-07,
"output_cost_per_token_above_100k_tokens": 2.75e-06,
"cache_creation_input_token_cost_above_100k_tokens": 6.875e-07,
"cache_creation_input_token_cost_above_1hr_above_100k_tokens": 1.1e-06,
"cache_read_input_token_cost_above_100k_tokens": 5.5e-08,
"bedrock_converse_supports_strict_tools": false,
"cache_creation_input_token_cost": 1.375e-07,
"cache_creation_input_token_cost_above_1hr": 2.2e-07,
"cache_read_input_token_cost": 1.1e-08,
"input_cost_per_token": 1.1e-07,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 5.5e-07,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_adaptive_thinking": true,
"supports_mid_conversation_system": true,
"supports_assistant_prefill": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"bedrock_output_config_effort_ceiling": "xhigh",
"supports_parallel_tool_use_config": true,
"supports_forced_tool_use": true,
"thinking_always_on": false,
"prompt_cache_min_tokens": 512,
"source": "https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonBedrockFoundationModels/current/index.json"
},
"perplexity/anthropic/claude-haiku-5-5": {
"litellm_provider": "perplexity",
"mode": "responses",
"supports_adaptive_thinking": true,
"supports_web_search": true,
"supports_function_calling": true,
"input_cost_per_token": 1e-07,
"output_cost_per_token": 5e-07,
"cache_read_input_token_cost": 1e-08,
"source": "https://docs.perplexity.ai/docs/agent-api/models"
},
"us-gov.anthropic.claude-haiku-5-5": {
"input_cost_per_token_above_100k_tokens": 6e-07,
"output_cost_per_token_above_100k_tokens": 3e-06,
"cache_creation_input_token_cost_above_100k_tokens": 7.5e-07,
"cache_creation_input_token_cost_above_1hr_above_100k_tokens": 1.2e-06,
"cache_read_input_token_cost_above_100k_tokens": 6e-08,
"bedrock_converse_supports_strict_tools": false,
"bedrock_output_config_effort_ceiling": "xhigh",
"cache_creation_input_token_cost": 1.5e-07,
"cache_creation_input_token_cost_above_1hr": 2.4e-07,
"cache_read_input_token_cost": 1.2e-08,
"input_cost_per_token": 1.2e-07,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 6e-07,
"supports_forced_tool_use": true,
"thinking_always_on": false,
"prompt_cache_min_tokens": 512,
"source": "https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonBedrockFoundationModels/current/index.json",
"supports_adaptive_thinking": true,
"supports_assistant_prefill": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_max_reasoning_effort": true,
"supports_mid_conversation_system": true,
"supports_native_structured_output": false,
"supports_output_config": true,
"supports_parallel_tool_use_config": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true
},
"us.anthropic.claude-haiku-5-5": {
"input_cost_per_token_above_100k_tokens": 5.5e-07,
"output_cost_per_token_above_100k_tokens": 2.75e-06,
"cache_creation_input_token_cost_above_100k_tokens": 6.875e-07,
"cache_creation_input_token_cost_above_1hr_above_100k_tokens": 1.1e-06,
"cache_read_input_token_cost_above_100k_tokens": 5.5e-08,
"bedrock_converse_supports_strict_tools": false,
"cache_creation_input_token_cost": 1.375e-07,
"cache_creation_input_token_cost_above_1hr": 2.2e-07,
"cache_read_input_token_cost": 1.1e-08,
"input_cost_per_token": 1.1e-07,
"litellm_provider": "bedrock_converse",
"supports_tool_search": true,
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 5.5e-07,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_adaptive_thinking": true,
"supports_mid_conversation_system": true,
"supports_assistant_prefill": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_native_structured_output": false,
"supports_max_reasoning_effort": true,
"supports_output_config": true,
"bedrock_output_config_effort_ceiling": "xhigh",
"supports_parallel_tool_use_config": true,
"supports_forced_tool_use": true,
"thinking_always_on": false,
"prompt_cache_min_tokens": 512,
"source": "https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonBedrockFoundationModels/current/index.json"
},
"vertex_ai/claude-haiku-5-5": {
"input_cost_per_token_above_100k_tokens": 5e-07,
"output_cost_per_token_above_100k_tokens": 2.5e-06,
"cache_creation_input_token_cost_above_100k_tokens": 6.25e-07,
"cache_creation_input_token_cost_above_1hr_above_100k_tokens": 1e-06,
"cache_read_input_token_cost_above_100k_tokens": 5e-08,
"regional_endpoint_uplift_multiplier": 1.1,
"supports_mid_conversation_system": true,
"cache_creation_input_token_cost": 1.25e-07,
"cache_creation_input_token_cost_above_1hr": 2e-07,
"cache_creation_input_token_cost_batches": 6.25e-08,
"cache_read_input_token_cost": 1e-08,
"cache_read_input_token_cost_batches": 5e-09,
"input_cost_per_token": 1e-07,
"input_cost_per_token_batches": 5e-08,
"litellm_provider": "vertex_ai-anthropic_models",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 5e-07,
"output_cost_per_token_batches": 2.5e-07,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_adaptive_thinking": true,
"supports_assistant_prefill": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_max_reasoning_effort": true,
"supports_forced_tool_use": true,
"thinking_always_on": false,
"prompt_cache_min_tokens": 512,
"source": "https://cloud.google.com/vertex-ai/generative-ai/pricing"
},
"vertex_ai/claude-haiku-5-5@default": {
"input_cost_per_token_above_100k_tokens": 5e-07,
"output_cost_per_token_above_100k_tokens": 2.5e-06,
"cache_creation_input_token_cost_above_100k_tokens": 6.25e-07,
"cache_creation_input_token_cost_above_1hr_above_100k_tokens": 1e-06,
"cache_read_input_token_cost_above_100k_tokens": 5e-08,
"regional_endpoint_uplift_multiplier": 1.1,
"supports_mid_conversation_system": true,
"cache_creation_input_token_cost": 1.25e-07,
"cache_creation_input_token_cost_above_1hr": 2e-07,
"cache_creation_input_token_cost_batches": 6.25e-08,
"cache_read_input_token_cost": 1e-08,
"cache_read_input_token_cost_batches": 5e-09,
"input_cost_per_token": 1e-07,
"input_cost_per_token_batches": 5e-08,
"litellm_provider": "vertex_ai-anthropic_models",
"max_input_tokens": 1000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 5e-07,
"output_cost_per_token_batches": 2.5e-07,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_adaptive_thinking": true,
"supports_assistant_prefill": false,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_sampling_params": false,
"supports_tool_choice": true,
"supports_vision": true,
"supports_xhigh_reasoning_effort": true,
"supports_max_reasoning_effort": true,
"supports_forced_tool_use": true,
"thinking_always_on": false,
"prompt_cache_min_tokens": 512,
"source": "https://cloud.google.com/vertex-ai/generative-ai/pricing"
}
}
@@ -187,8 +187,11 @@ mod tests {
"claude-opus-5-5",
"claude-opus-5",
"claude-fable-5-1",
"claude-haiku-5-5",
"global.anthropic.claude-sonnet-5-5",
"vertex_ai/claude-sonnet-5-5",
"us.anthropic.claude-haiku-5-5",
"vertex_ai/claude-haiku-5-5",
] {
assert_eq!(context_window_for(model), 1_000_000, "{model}");
}
@@ -660,25 +660,44 @@ impl Ledger {
let usd = if ev.failed {
EventUsd::default()
} else {
match pricing::lookup(&ev.model) {
Some(p) => EventUsd {
compression: p.compression_savings_usd(ev.tokens_saved()),
cache: p
.cache_savings_usd(ev.usage.cache_read_tokens, ev.usage.cache_write_tokens),
input_cost: p.input_cost_usd(
// When the response carried no usage block,
// bill the post-compression estimate instead
// so passthrough lanes still show spend.
if ev.usage.total_input() > 0 {
ev.usage.input_tokens
} else {
ev.tokens_after
},
ev.usage.cache_read_tokens,
ev.usage.cache_write_tokens,
),
output_cost: p.output_cost_usd(ev.usage.output_tokens),
},
match pricing::lookup_card(&ev.model) {
Some(card) => {
// When the response carried no usage block, bill
// the post-compression estimate instead so
// passthrough lanes still show spend.
let uncached = if ev.usage.total_input() > 0 {
ev.usage.input_tokens
} else {
ev.tokens_after
};
// The billed prompt picks the rate card: past a
// model's long-context threshold every rate below
// switches to the higher tier.
let p = card.for_prompt(
uncached
.saturating_add(ev.usage.cache_read_tokens)
.saturating_add(ev.usage.cache_write_tokens),
);
EventUsd {
compression: card.compression_savings_usd(
uncached,
ev.usage.cache_read_tokens,
ev.usage.cache_write_tokens,
ev.usage.output_tokens,
ev.tokens_saved(),
),
cache: p.cache_savings_usd(
ev.usage.cache_read_tokens,
ev.usage.cache_write_tokens,
),
input_cost: p.input_cost_usd(
uncached,
ev.usage.cache_read_tokens,
ev.usage.cache_write_tokens,
),
output_cost: p.output_cost_usd(ev.usage.output_tokens),
}
}
None => EventUsd::default(),
}
};
@@ -1277,6 +1296,66 @@ mod tests {
assert!(total > input, "total spend must exceed input-only spend");
}
/// A Haiku 5.5 prompt over 100K bills every rate at the 5x card:
/// 150K uncached in + 5K out is $0.075 + $0.0125, not the base
/// card's $0.015 + $0.0025.
#[test]
fn long_prompt_bills_at_the_models_long_context_card() {
let ledger = Ledger::in_memory();
let mut ev = event("claude-haiku-5-5");
ev.usage = Usage {
input_tokens: 150_000,
output_tokens: 5_000,
cache_read_tokens: 0,
cache_write_tokens: 0,
};
ledger.record(ev);
let v = ledger.stats_payload(true);
let input = v["session"]["input_cost_usd"].as_f64().unwrap();
let output = v["session"]["output_cost_usd"].as_f64().unwrap();
assert!((input - 0.075).abs() < 1e-9, "input: {input}");
assert!((output - 0.0125).abs() < 1e-9, "output: {output}");
}
/// Devin's example on #4037: 110K uncached compressed to 90K saves
/// $0.055 - $0.009 = $0.046 of input, not 20K x $0.10/M = $0.002.
#[test]
fn compression_below_the_threshold_credits_the_avoided_long_card() {
let ledger = Ledger::in_memory();
let mut ev = event("claude-haiku-5-5");
ev.tokens_before = 110_000;
ev.tokens_after = 90_000;
ev.usage = Usage {
input_tokens: 90_000,
output_tokens: 0,
cache_read_tokens: 0,
cache_write_tokens: 0,
};
ledger.record(ev);
let v = ledger.stats_payload(true);
let saved = v["session"]["compression_savings_usd"].as_f64().unwrap();
assert!((saved - 0.046).abs() < 1e-9, "saved: {saved}");
}
/// Cache reads count toward the threshold: 60K uncached + 60K
/// read is a 120K prompt, so it bills at the long card too.
#[test]
fn cached_tokens_count_toward_the_long_context_threshold() {
let ledger = Ledger::in_memory();
let mut ev = event("claude-haiku-5-5");
ev.usage = Usage {
input_tokens: 60_000,
output_tokens: 0,
cache_read_tokens: 60_000,
cache_write_tokens: 0,
};
ledger.record(ev);
let v = ledger.stats_payload(true);
let input = v["session"]["input_cost_usd"].as_f64().unwrap();
// 60K x $0.50/M + 60K x $0.05/M.
assert!((input - 0.033).abs() < 1e-9, "input: {input}");
}
#[test]
fn hostile_token_counts_clamp_to_ceiling() {
let ledger = Ledger::in_memory();
@@ -33,6 +33,17 @@
//! while its Bedrock form is present). Those price at $0 with a WARN
//! until the table is refreshed — see the module note below.
//!
//! # Long-context tiers
//!
//! Some models publish a second, higher rate card for long prompts
//! (LiteLLM's `*_above_<N>k_tokens` fields: Claude Haiku 5.5 above
//! 100K, Sonnet 4 / 4.5 above 200K, GPT-5.x above 272K). Above the
//! threshold the provider re-prices the WHOLE request — input, output
//! and cache alike — not just the tokens past it, so [`lookup_card`]
//! keeps both cards and [`PriceCard::for_prompt`] picks one from the
//! billed prompt. A tier field a row leaves out falls back to that
//! rate's base price, the same rule the Python cost path follows.
//!
//! # Unknown models price at $0
//!
//! A miss returns `None`; the ledger then records $0 for every USD
@@ -121,7 +132,71 @@ impl ModelPrice {
}
}
static BOOK: OnceLock<HashMap<String, ModelPrice>> = OnceLock::new();
/// A model's rate cards: the base one, plus the long-context one for
/// models that publish it (see the module doc).
#[derive(Debug, Clone, Copy, PartialEq)]
pub struct PriceCard {
pub base: ModelPrice,
pub long_context: Option<LongContextTier>,
}
/// The card a request bills at once its prompt passes `above_tokens`.
#[derive(Debug, Clone, Copy, PartialEq)]
pub struct LongContextTier {
/// Prompt size (uncached + cache read + cache write tokens) the
/// request must exceed for this card to apply.
pub above_tokens: u64,
pub price: ModelPrice,
}
impl PriceCard {
/// The rates a request with `prompt_tokens` of billed prompt pays.
pub fn for_prompt(&self, prompt_tokens: u64) -> ModelPrice {
match self.long_context {
Some(tier) if prompt_tokens > tier.above_tokens => tier.price,
_ => self.base,
}
}
/// USD compression saved on one request: what it would have cost
/// uncompressed minus what it did cost.
///
/// Inside one tier that is the removed tokens at that tier's input
/// rate. When the uncompressed prompt would have passed the
/// long-context threshold and the forwarded one does not, the
/// provider would have re-priced the WHOLE request — every input
/// bucket and the output — at the long card, so the saving is the
/// difference of the two full bills, not the removed tokens at the
/// cheap rate.
pub fn compression_savings_usd(
&self,
uncached_input_tokens: u64,
cache_read_tokens: u64,
cache_write_tokens: u64,
output_tokens: u64,
tokens_saved: u64,
) -> f64 {
let prompt = uncached_input_tokens
.saturating_add(cache_read_tokens)
.saturating_add(cache_write_tokens);
let sent = self.for_prompt(prompt);
let unsent = self.for_prompt(prompt.saturating_add(tokens_saved));
if unsent == sent {
return sent.compression_savings_usd(tokens_saved);
}
let without = unsent.input_cost_usd(
uncached_input_tokens.saturating_add(tokens_saved),
cache_read_tokens,
cache_write_tokens,
) + unsent.output_cost_usd(output_tokens);
let with =
sent.input_cost_usd(uncached_input_tokens, cache_read_tokens, cache_write_tokens)
+ sent.output_cost_usd(output_tokens);
(without - with).max(0.0)
}
}
static BOOK: OnceLock<HashMap<String, PriceCard>> = OnceLock::new();
/// Bounded once-per-model warn set so an unknown model logs one WARN,
/// not one per request. Capped so attacker-controlled model ids can't
@@ -140,11 +215,48 @@ pub fn warm() {
let _ = book();
}
fn book() -> &'static HashMap<String, ModelPrice> {
fn book() -> &'static HashMap<String, PriceCard> {
BOOK.get_or_init(parse_vendored)
}
fn parse_vendored() -> HashMap<String, ModelPrice> {
/// A finite, non-negative price field, or `None`.
fn price_field(spec: &serde_json::Map<String, serde_json::Value>, key: &str) -> Option<f64> {
spec.get(key)
.and_then(|v| v.as_f64())
.filter(|c| c.is_finite() && *c >= 0.0)
}
/// The row's long-context card, keyed off its
/// `input_cost_per_token_above_<N>k_tokens` field. Every row in the
/// vendored table publishes at most one such threshold.
fn parse_long_context_tier(
spec: &serde_json::Map<String, serde_json::Value>,
base: &ModelPrice,
) -> Option<LongContextTier> {
let (thousands, input) = spec.keys().find_map(|key| {
let n = key
.strip_prefix("input_cost_per_token_above_")?
.strip_suffix("k_tokens")?;
if n.is_empty() || !n.bytes().all(|b| b.is_ascii_digit()) {
return None;
}
Some((n.parse::<u64>().ok()?, price_field(spec, key)?))
})?;
let tiered = |field: &str, fallback: f64| {
price_field(spec, &format!("{field}_above_{thousands}k_tokens")).unwrap_or(fallback)
};
Some(LongContextTier {
above_tokens: thousands.saturating_mul(1000),
price: ModelPrice {
input,
output: tiered("output_cost_per_token", base.output),
cache_read: tiered("cache_read_input_token_cost", base.cache_read),
cache_write: tiered("cache_creation_input_token_cost", base.cache_write),
},
})
}
fn parse_vendored() -> HashMap<String, PriceCard> {
let raw: serde_json::Value = serde_json::from_str(VENDORED_JSON)
.expect("vendored LiteLLM JSON must parse — same invariant as model_limits");
let mut out = HashMap::new();
@@ -166,23 +278,17 @@ fn parse_vendored() -> HashMap<String, ModelPrice> {
if !input.is_finite() || !output.is_finite() || input < 0.0 || output < 0.0 {
continue;
}
let cache_read = spec
.get("cache_read_input_token_cost")
.and_then(|v| v.as_f64())
.filter(|c| c.is_finite() && *c >= 0.0)
.unwrap_or(input);
let cache_write = spec
.get("cache_creation_input_token_cost")
.and_then(|v| v.as_f64())
.filter(|c| c.is_finite() && *c >= 0.0)
.unwrap_or(input);
let base = ModelPrice {
input,
output,
cache_read: price_field(spec, "cache_read_input_token_cost").unwrap_or(input),
cache_write: price_field(spec, "cache_creation_input_token_cost").unwrap_or(input),
};
out.insert(
model_id.to_ascii_lowercase(),
ModelPrice {
input,
output,
cache_read,
cache_write,
PriceCard {
long_context: parse_long_context_tier(spec, &base),
base,
},
);
}
@@ -236,20 +342,26 @@ fn push_revision_trimmed(push: &mut impl FnMut(String), id: &str) {
}
}
/// Look up per-token pricing for a model id. `None` means "not in
/// the vendored table" — the caller records $0 for USD fields and
/// this module logs one WARN per distinct model id.
pub fn lookup(model: &str) -> Option<ModelPrice> {
/// Look up a model's rate cards. `None` means "not in the vendored
/// table" — the caller records $0 for USD fields and this module logs
/// one WARN per distinct model id.
pub fn lookup_card(model: &str) -> Option<PriceCard> {
let table = book();
for key in candidates(model) {
if let Some(p) = table.get(&key) {
return Some(*p);
if let Some(card) = table.get(&key) {
return Some(*card);
}
}
warn_once(model);
None
}
/// Look up a model's base per-token pricing — the card a prompt below
/// any long-context threshold pays. See [`lookup_card`].
pub fn lookup(model: &str) -> Option<ModelPrice> {
lookup_card(model).map(|card| card.base)
}
fn warn_once(model: &str) {
let set = WARNED.get_or_init(|| RwLock::new(std::collections::HashSet::new()));
{
@@ -288,6 +400,98 @@ mod tests {
assert!(p.cache_write > p.input, "cache writes carry a premium");
}
#[test]
fn claude_haiku_5_5_base_rate_card() {
let p = lookup("claude-haiku-5-5").expect("priced");
assert!((p.input - 1e-7).abs() < 1e-15, "input $0.10/MTok");
assert!((p.output - 5e-7).abs() < 1e-15, "output $0.50/MTok");
assert!((p.cache_read - 1e-8).abs() < 1e-15, "cache read $0.01/MTok");
assert!(
(p.cache_write - 1.25e-7).abs() < 1e-15,
"5m cache write $0.125/MTok"
);
}
/// Haiku 5.5 re-prices the whole request at 5x once the prompt
/// passes 100K; at exactly 100K it still pays the base card.
#[test]
fn claude_haiku_5_5_long_prompt_card_above_100k() {
let card = lookup_card("claude-haiku-5-5").expect("priced");
let tier = card.long_context.expect("Haiku 5.5 publishes a 100K tier");
assert_eq!(tier.above_tokens, 100_000);
assert_eq!(card.for_prompt(100_000), card.base);
let long = card.for_prompt(100_001);
assert!((long.input - 5e-7).abs() < 1e-15, "input $0.50/MTok");
assert!((long.output - 2.5e-6).abs() < 1e-15, "output $2.50/MTok");
assert!(
(long.cache_read - 5e-8).abs() < 1e-15,
"cache read $0.05/MTok"
);
assert!(
(long.cache_write - 6.25e-7).abs() < 1e-15,
"5m cache write $0.625/MTok"
);
}
/// The threshold comes from each row's own field name, not a
/// shared constant: Sonnet 4.5's card switches at 200K.
#[test]
fn long_context_threshold_is_read_per_model() {
let card = lookup_card("claude-sonnet-4-5").expect("priced");
let tier = card.long_context.expect("Sonnet 4.5 publishes a 200K tier");
assert_eq!(tier.above_tokens, 200_000);
assert_eq!(card.for_prompt(150_000), card.base);
assert!((card.for_prompt(200_001).input - 6e-6).abs() < 1e-15);
}
/// Compression that keeps a prompt under the threshold saves the
/// long card on the whole request, not just the removed tokens at
/// the base rate: 110K -> 90K uncached on Haiku 5.5, 1K output, is
/// (110K x $0.50 + 1K x $2.50) - (90K x $0.10 + 1K x $0.50) per MTok.
#[test]
fn compression_across_the_threshold_saves_the_whole_premium() {
let card = lookup_card("claude-haiku-5-5").expect("priced");
let usd = card.compression_savings_usd(90_000, 0, 0, 1_000, 20_000);
assert!((usd - (0.0575 - 0.0095)).abs() < 1e-9, "usd: {usd}");
// Inside one tier it stays removed tokens at that tier's rate.
let below = card.compression_savings_usd(50_000, 0, 0, 1_000, 20_000);
assert!((below - 20_000.0 * 1e-7).abs() < 1e-12, "below: {below}");
let above = card.compression_savings_usd(120_000, 0, 0, 1_000, 20_000);
assert!((above - 20_000.0 * 5e-7).abs() < 1e-12, "above: {above}");
}
/// A model without a published tier bills the base card at any size.
#[test]
fn flat_rated_model_has_no_long_context_tier() {
let card = lookup_card("claude-opus-4-6").expect("priced");
assert!(card.long_context.is_none());
assert_eq!(card.for_prompt(900_000), card.base);
}
/// A tier that omits a rate falls back to that rate's base price.
#[test]
fn missing_tier_field_falls_back_to_base_rate() {
let spec: serde_json::Map<String, serde_json::Value> = serde_json::from_str(
r#"{"input_cost_per_token_above_128k_tokens": 2e-6,
"output_cost_per_token_above_128k_tokens_priority": 9e-6}"#,
)
.unwrap();
let base = ModelPrice {
input: 1e-6,
output: 4e-6,
cache_read: 1e-7,
cache_write: 1e-6,
};
let tier = parse_long_context_tier(&spec, &base).expect("tier parsed");
assert_eq!(tier.above_tokens, 128_000);
assert_eq!(tier.price.input, 2e-6);
assert_eq!(
tier.price.output, base.output,
"a _priority variant is not the tier"
);
assert_eq!(tier.price.cache_read, base.cache_read);
}
/// Bedrock cross-region pricing is region-specific, so an exact
/// regional entry must win over the geo-stripped base id.
#[test]
+127 -22
View File
@@ -56,6 +56,7 @@ from __future__ import annotations
import logging
import math
import re
from dataclasses import dataclass
from enum import Enum
from functools import lru_cache
@@ -68,11 +69,16 @@ from headroom.pricing.cache_ttl import CACHE_WRITE_MULTIPLIERS
logger = logging.getLogger(__name__)
#: Context size at which the major catalogs publish a second, higher price tier
#: (LiteLLM spells it ``*_above_200k_tokens``). A request's billed prompt is
#: compared against this to pick which rate applies.
#: Prompt size at which most catalog rows publish a second, higher price tier
#: (LiteLLM spells it ``*_above_200k_tokens``). The default for
#: :func:`long_context_threshold`, which reads each model's own threshold from
#: its row: Claude Haiku 5.5's tier starts at 100K, GPT-5.x's at 272K.
LONG_CONTEXT_THRESHOLD_TOKENS = 200_000
#: A catalog row's long-context input rate; the number is the threshold in
#: thousands of tokens. Each row publishes at most one.
_LONG_CONTEXT_INPUT_FIELD = re.compile(r"^input_cost_per_token_above_(\d+)k_tokens$")
#: Provider-level cache discount ratios, as a fraction of the base input price.
#: FALLBACK ONLY — the per-model LiteLLM catalog is always preferred, because
#: these go stale per model and per context tier. The lightweight shared table
@@ -237,15 +243,18 @@ class CacheMix:
"""
return self.billed > 0
def is_long_context(self, *, local_tokens: int = 0) -> bool:
"""True when this request billed above the catalogs' 200k price tier.
def is_long_context(
self, *, local_tokens: int = 0, threshold: int = LONG_CONTEXT_THRESHOLD_TOKENS
) -> bool:
"""True when this request billed above ``threshold``, the model's long-context tier.
``local_tokens`` lets a caller contribute its own forwarded-token count
for requests where the provider reported no breakdown; the larger of the
two decides, so a long request is never priced at the cheap tier merely
because usage was missing.
because usage was missing. Pass :func:`long_context_threshold` for the
model so a tier that starts below 200K (Haiku 5.5: 100K) is honoured.
"""
return max(self.billed, _coerce_int(local_tokens)) > LONG_CONTEXT_THRESHOLD_TOKENS
return max(self.billed, _coerce_int(local_tokens)) > threshold
@dataclass(frozen=True)
@@ -358,6 +367,34 @@ def _litellm() -> Any | None:
_RATE_CACHE_MAXSIZE = 256
def long_context_tier(info: dict[str, Any]) -> tuple[int, str] | None:
"""The row's long-context tier as ``(threshold_tokens, field_suffix)``, if any."""
for key in info:
match = _LONG_CONTEXT_INPUT_FIELD.match(key)
if match:
thousands = int(match.group(1))
return thousands * 1000, f"_above_{thousands}k_tokens"
return None
@lru_cache(maxsize=_RATE_CACHE_MAXSIZE)
def long_context_threshold(model: str) -> int:
"""Billed-prompt size above which ``model`` pays its long-context rates.
Read from the catalog row's ``*_above_<N>k_tokens`` fields: 100K for Claude
Haiku 5.5, 200K for Sonnet 4 / 4.5, 272K for GPT-5.x. A model with no tier,
or one the catalog does not list, gets :data:`LONG_CONTEXT_THRESHOLD_TOKENS`;
that is harmless, because :func:`resolve_rates` then has no tier to switch
to and returns the base rates either way.
Memoized on the same bounded LRU as :func:`resolve_rates`, for the same
reason; tests that swap the catalog clear both.
"""
row = _catalog_row(model)
tier = long_context_tier(row[2]) if row is not None else None
return tier[0] if tier else LONG_CONTEXT_THRESHOLD_TOKENS
def _canonical_cache_rate(litellm: Any, model: str, field: str, *, long_context: bool) -> float:
"""Resolve an absent catalog slice through the model's billing calculator.
@@ -365,7 +402,7 @@ def _canonical_cache_rate(litellm: Any, model: str, field: str, *, long_context:
zero, list price, or a model-specific discount. Never invent our own ratio.
The result is cached with the other resolved rates, not probed per request.
"""
tokens = LONG_CONTEXT_THRESHOLD_TOKENS + 1 if long_context else 1
tokens = long_context_threshold(model) + 1 if long_context else 1
try:
input_cost, _ = litellm.cost_per_token(
model=model, prompt_tokens=tokens, completion_tokens=0, **{field: tokens}
@@ -413,11 +450,12 @@ def resolve_rates(
Preference order, strongest first:
1. **Catalog** — LiteLLM's per-model published rates, including the
long-context tier and, where the row has it,
long-context tier (the row's own ``*_above_<N>k_tokens`` fields, see
:func:`long_context_threshold`) and, where the row has it,
``cache_creation_input_token_cost_above_1hr``. Anthropic publishes a real
1h write rate (Sonnet: 2.00x base), and reading it beats deriving it.
2. **Catalog + TTL ratio** — a catalog row that prices 5m writes but not 1h
ones, which is every row at the above-200k tier. The 1h rate is derived
ones, which is every row at its long-context tier. The 1h rate is derived
from the tier's base input price and the structural multiplier in
:mod:`headroom.pricing.cache_ttl`. Only applied when the row shows a real
write premium: a provider that does not bill for cache writes at all
@@ -446,14 +484,17 @@ def resolve_rates(
# savings on a model that costs nothing.
if base is None:
return None
if long_context:
base = info.get("input_cost_per_token_above_200k_tokens") or base
tier = long_context_tier(info) if long_context else None
if tier:
long_base = info.get(f"input_cost_per_token{tier[1]}")
if long_base is not None:
base = long_base
base = float(base)
def _tier(field: str, default: float) -> float:
"""Read ``field``, preferring its above-200k variant on long requests."""
if long_context:
hi = info.get(f"{field}_above_200k_tokens")
"""Read ``field``, preferring its long-context variant on long requests."""
if tier:
hi = info.get(f"{field}{tier[1]}")
if hi is not None:
return float(hi)
value = info.get(field)
@@ -468,10 +509,10 @@ def resolve_rates(
# billed provider-reported writes require their own catalog rate.
write_5m = _tier("cache_creation_input_token_cost", missing_cache_rate)
read_is_catalog = info.get("cache_read_input_token_cost") is not None or (
long_context and info.get("cache_read_input_token_cost_above_200k_tokens") is not None
tier is not None and info.get(f"cache_read_input_token_cost{tier[1]}") is not None
)
write_is_catalog = info.get("cache_creation_input_token_cost") is not None or (
long_context and info.get("cache_creation_input_token_cost_above_200k_tokens") is not None
tier is not None and info.get(f"cache_creation_input_token_cost{tier[1]}") is not None
)
if for_billing and not read_is_catalog:
read = _canonical_cache_rate(
@@ -484,10 +525,12 @@ def resolve_rates(
basis = BASIS_CATALOG
write_1h_raw = info.get("cache_creation_input_token_cost_above_1hr")
if long_context:
# No catalog publishes a combined 1h + above-200k rate, so the long
# tier always derives. Ratio basis, and labelled as such.
write_1h_raw = None
if tier:
# The long tier's 1h rate is its own field
# (``cache_creation_input_token_cost_above_1hr_above_100k_tokens``);
# most tiered rows omit it, and the rate then derives below. Ratio
# basis, and labelled as such.
write_1h_raw = info.get(f"cache_creation_input_token_cost_above_1hr{tier[1]}")
if write_1h_raw is not None:
write_1h = float(write_1h_raw)
elif write_5m > base:
@@ -540,6 +583,63 @@ def resolve_rates(
)
def long_context_premium_avoided_usd(
model: str,
mix: CacheMix,
*,
local_tokens: int = 0,
output_tokens: int = 0,
provider: str | None = None,
) -> float:
"""Long-context premium a request's forwarded input and output avoided, in USD.
For a request whose uncompressed prompt would have passed ``model``'s
long-context threshold but whose forwarded prompt did not: the provider
re-prices the WHOLE request above the threshold, so compression saved the
long card on every forwarded input token, on top of the removed tokens
themselves. This is that difference, forwarded input at the long card minus
the same input at the base card. When ``output_tokens`` is supplied, include
the premium avoided on that generated output too. These tokens were still
generated; this is compression savings, not output-shaping savings.
``local_tokens`` stands in as uncached input when the provider reported no
breakdown. Returns 0.0 for a model with no tier or no catalog row.
"""
try:
long_rates = resolve_rates(model, long_context=True, provider=provider)
base_rates = resolve_rates(model, long_context=False, provider=provider)
except Exception: # pragma: no cover - defensive; pricing must never raise
return 0.0
if long_rates is None or base_rates is None:
return 0.0
if mix.has_signal():
n = mix.normalized()
split = TokenSplit(
read=float(n.read),
write_5m=float(n.write_5m),
write_1h=float(n.write_1h),
uncached=float(n.uncached),
)
else:
split = TokenSplit(uncached=float(max(_coerce_int(local_tokens), 0)))
input_premium = max(0.0, long_rates.price(split) - base_rates.price(split))
output_premium = 0.0
if max(_coerce_int(output_tokens), 0):
try:
row = _catalog_row(model)
info = row[2] if row is not None else {}
tier = long_context_tier(info)
base_output = info.get("output_cost_per_token")
long_output = info.get(f"output_cost_per_token{tier[1]}") if tier else None
if base_output is not None and long_output is not None:
output_premium = max(0.0, float(long_output) - float(base_output)) * max(
_coerce_int(output_tokens), 0
)
except Exception: # Pricing metadata must not break telemetry.
pass
return input_premium + output_premium
def catalog_min_cacheable_tokens(model: str) -> int | None:
"""The catalog's smallest prompt the provider will cache for ``model``.
@@ -613,7 +713,9 @@ def price_savings(
mix = (mix or CacheMix()).normalized()
if long_context is None:
long_context = mix.is_long_context(local_tokens=local_tokens)
long_context = mix.is_long_context(
local_tokens=local_tokens, threshold=long_context_threshold(model)
)
try:
rates = resolve_rates(model, long_context=long_context, provider=provider)
@@ -664,6 +766,9 @@ __all__ = [
"PricedSavings",
"Region",
"TokenSplit",
"long_context_premium_avoided_usd",
"long_context_threshold",
"long_context_tier",
"catalog_min_cacheable_tokens",
"price_savings",
"resolve_rates",
+106 -26
View File
@@ -132,6 +132,8 @@ ANTHROPIC_CONTEXT_LIMITS: dict[str, int] = {
"claude-sonnet-4-6": 1000000,
# Claude Sonnet 4.5
"claude-sonnet-4-5": 200000,
# Claude Haiku 5.5 - 1M context
"claude-haiku-5-5": 1000000,
# Claude 4 (Sonnet 4, Haiku 4)
"claude-sonnet-4-20250514": 200000,
"claude-haiku-4-5-20251001": 200000,
@@ -155,9 +157,9 @@ ANTHROPIC_CONTEXT_LIMITS: dict[str, int] = {
# NOTE: These are ESTIMATES. Always verify against actual Anthropic billing.
# Last updated: 2026-09-29 (platform.claude.com/docs/en/about-claude/pricing)
#
# Newer ids come before their prefixes: `_get_pricing` falls back to substring
# matching in insertion order, so a dated or suffixed "claude-sonnet-5-5-..."
# must hit the Sonnet 5.5 row before it can hit "claude-sonnet-5".
# Newer ids still come before their prefixes, but order no longer decides a
# match: `_is_release_of` refuses to read "claude-sonnet-5-5-..." as a release
# of "claude-sonnet-5".
ANTHROPIC_PRICING: dict[str, dict[str, float]] = {
# Claude Fable 5.1: $10 in / $50 out, cache read $0.25 (0.025x input).
"claude-fable-5-1": {"input": 10.00, "output": 50.00, "cached_input": 0.25},
@@ -183,6 +185,9 @@ ANTHROPIC_PRICING: dict[str, dict[str, float]] = {
# Claude Sonnet 4.6 / 4.5: $3 in / $15 out, cache read $0.30
"claude-sonnet-4-6": {"input": 3.00, "output": 15.00, "cached_input": 0.30},
"claude-sonnet-4-5": {"input": 3.00, "output": 15.00, "cached_input": 0.30},
# Claude Haiku 5.5: $0.10 in / $0.50 out, cache read $0.01, for prompts up
# to 100K tokens; longer prompts use a 5x rate card (`_LONG_CONTEXT_TIERS`).
"claude-haiku-5-5": {"input": 0.10, "output": 0.50, "cached_input": 0.01},
# Claude 4 (Sonnet/Haiku tier pricing)
"claude-sonnet-4-20250514": {"input": 3.00, "output": 15.00, "cached_input": 0.30},
"claude-haiku-4-5-20251001": {"input": 1.00, "output": 5.00, "cached_input": 0.10},
@@ -204,10 +209,10 @@ ANTHROPIC_PRICING: dict[str, dict[str, float]] = {
# derived from LiteLLM's `*_above_200k_tokens` fields ($3->$6 in, $15->$22.50
# out, $0.30->$0.60 cache read).
#
# Only the Sonnet 4 / 4.5 family is tiered: Opus, and Sonnet 4.6 onward, are
# flat-rated across their whole window. This is the same population that needs
# the `[1m]` suffix to reach 1M at all, so a session that fills the window this
# unlocks is billed at these rates.
# Among the 200K-threshold models only the Sonnet 4 / 4.5 family is tiered:
# Opus, and Sonnet 4.6 onward, are flat-rated across their whole window. This
# is the same population that needs the `[1m]` suffix to reach 1M at all, so a
# session that fills the window this unlocks is billed at these rates.
_LONG_CONTEXT_THRESHOLD = 200_000
_LONG_CONTEXT_PREMIUM: dict[str, float] = {"input": 2.0, "output": 1.5, "cached_input": 2.0}
_LONG_CONTEXT_TIERED_MODELS = (
@@ -216,6 +221,55 @@ _LONG_CONTEXT_TIERED_MODELS = (
"claude-4-sonnet-20250514",
)
# Claude Haiku 5.5 has its own rate card for prompts above 100K tokens: every
# rate is 5x (LiteLLM's `*_above_100k_tokens` fields: $0.10->$0.50 in,
# $0.50->$2.50 out, $0.01->$0.05 cache read).
_HAIKU_5_5_LONG_PROMPT_THRESHOLD = 100_000
_HAIKU_5_5_LONG_PROMPT_PREMIUM: dict[str, float] = {
"input": 5.0,
"output": 5.0,
"cached_input": 5.0,
}
# (model ids, prompt-token threshold, per-rate multipliers above it). Ids match
# releases and provider-wrapped forms too (`_long_context_tier`).
_LONG_CONTEXT_TIERS: tuple[tuple[tuple[str, ...], int, dict[str, float]], ...] = (
(_LONG_CONTEXT_TIERED_MODELS, _LONG_CONTEXT_THRESHOLD, _LONG_CONTEXT_PREMIUM),
(("claude-haiku-5-5",), _HAIKU_5_5_LONG_PROMPT_THRESHOLD, _HAIKU_5_5_LONG_PROMPT_PREMIUM),
)
# A dated release (`-20250929`) or `-latest` alias of a model id.
_RELEASE_SUFFIX_RE = re.compile(r"-(?:\d{8}|latest)")
# A further version segment (the `-5` in `claude-haiku-5-5`): another model.
_VERSION_SEGMENT_RE = re.compile(r"-\d{1,2}(?!\d)")
def _is_release_of(model: str, known_model: str) -> bool:
"""True when ``model`` names ``known_model``, or a release or alias of it.
The fallback tables are keyed by a mix of undated and dated ids, so a
lookup accepts:
- a longer id containing ``known_model``: a dated release or a provider
wrapped id (``claude-sonnet-4-5-20250929``,
``anthropic.claude-haiku-5-5-v1:0``);
- an undated alias of a dated key (``claude-3-5-sonnet`` for
``claude-3-5-sonnet-20241022``), or its ``-latest`` form.
An id one version segment away is a different model and does not match:
``claude-haiku-5`` is not ``claude-haiku-5-5``, and ``claude-sonnet-5-5``
is not ``claude-sonnet-5``.
"""
if model == known_model:
return True
if known_model.startswith(model):
return bool(_RELEASE_SUFFIX_RE.fullmatch(known_model[len(model) :]))
at = model.find(known_model)
if at < 0:
return False
return not _VERSION_SEGMENT_RE.match(model, at + len(known_model))
def _apply_long_context_premium(
model: str, pricing: dict[str, float], input_tokens: int
@@ -226,11 +280,24 @@ def _apply_long_context_premium(
LiteLLM entry has no above-threshold rate (see
``_litellm_lacks_long_context_rate``).
"""
if input_tokens <= _LONG_CONTEXT_THRESHOLD:
tier = _long_context_tier(model)
if tier is None or input_tokens <= tier[0]:
return pricing
if not any(model.startswith(tiered) for tiered in _LONG_CONTEXT_TIERED_MODELS):
return pricing
return {key: rate * _LONG_CONTEXT_PREMIUM.get(key, 1.0) for key, rate in pricing.items()}
_threshold, premium = tier
return {key: rate * premium.get(key, 1.0) for key, rate in pricing.items()}
def _long_context_tier(model: str) -> tuple[int, dict[str, float]] | None:
"""Return the ``(threshold, premium)`` of the long-context tier ``model`` is in.
Matches with ``_is_release_of``, as the manual rate lookup does, so a
Bedrock id such as ``us.anthropic.claude-haiku-5-5-v1:0`` that the manual
table prices as Haiku 5.5 also gets Haiku 5.5's long-prompt card.
"""
for known_models, threshold, premium in _LONG_CONTEXT_TIERS:
if any(_is_release_of(model, known) for known in known_models):
return threshold, premium
return None
def _litellm_lacks_long_context_rate(model: str, input_tokens: int) -> bool:
@@ -240,17 +307,29 @@ def _litellm_lacks_long_context_rate(model: str, input_tokens: int) -> bool:
from the bare Anthropic ids before, and pricing such a request through it then
bills the base tier: about half the real cost of a long session. The manual
table still carries the premium, so the caller uses it instead.
Each tier is checked against its own field, named after its threshold
(``*_above_200k_tokens`` for Sonnet 4 / 4.5, ``*_above_100k_tokens`` for
Haiku 5.5).
The row is resolved as ``estimate_cost_from_tokens`` resolves it, so a
gateway id such as ``anthropic/claude-haiku-5-5`` is checked against the
bare row it is billed from. Only Anthropic's own rows count: a reseller row
without the field (``perplexity/anthropic/claude-haiku-5-5``) is
deliberately flat-rated, not a dropped rate.
"""
if input_tokens <= _LONG_CONTEXT_THRESHOLD:
return False
if not any(model.startswith(tiered) for tiered in _LONG_CONTEXT_TIERED_MODELS):
return False
cost_data = get_litellm_model_cost()
for candidate in pricing_lookup_candidates(model):
info = cost_data.get(candidate)
if isinstance(info, dict):
return "input_cost_per_token_above_200k_tokens" not in info
return False # LiteLLM doesn't know the model; it returns None and we fall back anyway
row = next((c for c in pricing_lookup_candidates(model) if c in cost_data), None)
if row is None:
return False # LiteLLM doesn't know the model; it returns None and we fall back anyway
info = cost_data[row]
if not isinstance(info, dict) or info.get("litellm_provider") != "anthropic":
return False
tier = _long_context_tier(row)
if tier is None or input_tokens <= tier[0]:
return False
threshold, _premium = tier
return f"input_cost_per_token_above_{threshold // 1000}k_tokens" not in info
# Default limits for pattern-based inference
@@ -778,9 +857,10 @@ class AnthropicProvider(Provider):
if model in self._context_limits:
return self._context_limits[model]
# Check for partial matches (e.g., "claude-3-5-sonnet" matches "claude-3-5-sonnet-20241022")
# Releases and aliases of a known id (e.g. "claude-3-5-sonnet" matches
# "claude-3-5-sonnet-20241022"); see `_is_release_of`.
for known_model, limit in self._context_limits.items():
if model in known_model or known_model in model:
if _is_release_of(model, known_model):
return limit
# Try LiteLLM for context limit
@@ -853,8 +933,8 @@ class AnthropicProvider(Provider):
Tries LiteLLM first for up-to-date pricing, falls back to manual pricing.
Both paths apply Anthropic's long-context premium: on the Sonnet 4 / 4.5
family a prompt over 200K re-prices the whole request (see
``_LONG_CONTEXT_PREMIUM``).
family a prompt over 200K, and on Haiku 5.5 a prompt over 100K,
re-prices the whole request (see ``_LONG_CONTEXT_TIERS``).
``now`` selects a DeepSeek peak/off-peak tier (see
:mod:`headroom.pricing.deepseek_tiers`); ``None`` reads the wall clock.
@@ -898,9 +978,9 @@ class AnthropicProvider(Provider):
if model in self._pricing:
return self._pricing[model]
# Partial match
# Releases and aliases of a known id; see `_is_release_of`.
for known_model, prices in self._pricing.items():
if model in known_model or known_model in model:
if _is_release_of(model, known_model):
return prices
# Pattern-based inference
+70 -12
View File
@@ -133,10 +133,21 @@ class CostEntry(NamedTuple):
# (Anthropic's >200k rates differ), so they are never preferred over the catalog.
#: Context size at which the major catalogs publish a second, higher price
#: tier (LiteLLM spells it ``*_above_200k_tokens``). A request's billed prompt
#: is compared against this to pick which rate applies.
_LONG_CONTEXT_THRESHOLD_TOKENS = 200_000
def _long_context_threshold(model: str) -> int:
"""Billed-prompt size above which ``model`` pays its long-context rates.
Delegates to :func:`headroom.pricing.counterfactual.long_context_threshold`,
which reads the model's own tier from the catalog (Haiku 5.5: 100K, Sonnet
4.5: 200K). Imported at call time, like ``_get_cache_prices``: importing
``headroom.pricing`` pulls in litellm (~4s) and this module is on the
proxy's startup path. Falls back to 200K if the lookup fails.
"""
try:
from headroom.pricing.counterfactual import long_context_threshold
return long_context_threshold(model)
except Exception:
return 200_000
def _bucket_by_cache_mix(
@@ -906,6 +917,13 @@ class CostTracker:
self._saved_write_5m_by_tier: dict[tuple[str, bool], float] = {}
self._saved_write_1h_by_tier: dict[tuple[str, bool], float] = {}
self._saved_list_by_tier: dict[tuple[str, bool], float] = {}
# Forwarded tokens of requests that compression kept under the model's
# long-context threshold (their uncompressed prompt would have crossed
# it), as [read, write 5m, write 1h, uncached, output]. The provider
# re-prices the whole request above the threshold, so these tokens
# avoided the long card's premium; stats() prices that premium into
# the cache-aware savings.
self._tier_crossing_by_model: dict[str, list[float]] = {}
# Tool-schema deferral per model, DISJOINT from _tokens_saved_by_model
# (deferred schemas are never in the message counts). Tracked separately
# so the compression-only figure stays available; `stats()` reports both
@@ -964,6 +982,7 @@ class CostTracker:
self._saved_write_5m_by_tier.clear()
self._saved_write_1h_by_tier.clear()
self._saved_list_by_tier.clear()
self._tier_crossing_by_model.clear()
self._tool_saved_by_model.clear()
self._tool_saved_read_by_model.clear()
self._tool_saved_write_5m_by_model.clear()
@@ -1129,7 +1148,24 @@ class CostTracker:
write_5m_eff = 0 if cache_inferred else max(0, cache_write_5m_tokens)
write_1h_eff = 0 if cache_inferred else max(0, cache_write_1h_tokens)
billed_prompt = max(0, cache_read_tokens) + write_eff + max(0, uncached_tokens)
long_context = max(billed_prompt, tokens_sent) > _LONG_CONTEXT_THRESHOLD_TOKENS
threshold = _long_context_threshold(model)
forwarded_prompt = max(billed_prompt, tokens_sent)
long_context = forwarded_prompt > threshold
# Compression that kept the prompt under the threshold: the removed
# tokens would have billed at the long card, and so would everything
# forwarded (see `_tier_crossing_by_model`).
crossed = not long_context and forwarded_prompt + tokens_saved > threshold
if crossed:
w1h = min(write_1h_eff, write_eff)
row = self._tier_crossing_by_model.setdefault(key, [0.0] * 5)
if billed_prompt > 0:
row[0] += max(0, cache_read_tokens)
row[1] += write_eff - w1h
row[2] += w1h
row[3] += max(0, uncached_tokens)
else:
row[3] += max(0, tokens_sent)
row[4] += max(0, output_tokens)
if tokens_saved > 0:
# Message compression works the LIVE ZONE only: handlers freeze the
# cached prefix (system + prior turns) byte-for-byte for prefix-cache
@@ -1151,7 +1187,7 @@ class CostTracker:
uncached_tokens=uncached_tokens,
region="live_zone",
)
wkey = (key, long_context)
wkey = (key, long_context or crossed)
self._saved_write_5m_by_tier[wkey] = self._saved_write_5m_by_tier.get(wkey, 0.0) + c_w5m
self._saved_write_1h_by_tier[wkey] = self._saved_write_1h_by_tier.get(wkey, 0.0) + c_w1h
self._saved_list_by_tier[wkey] = self._saved_list_by_tier.get(wkey, 0.0) + c_list
@@ -1419,8 +1455,9 @@ class CostTracker:
def _get_output_price(self, model: str, *, long_context: bool = False) -> float | None:
"""Get the per-token completion price for a model, or None if unpriced.
``long_context`` selects the catalog's above-200k completion rate where
the model publishes one (Anthropic charges 1.5x there).
``long_context`` selects the completion rate of the model's
long-context tier where it publishes one (Sonnet 4.5 above 200K: 1.5x;
Haiku 5.5 above 100K: 5x).
"""
litellm = _get_litellm_module()
if litellm is None:
@@ -1432,7 +1469,11 @@ class CostTracker:
info = litellm.model_cost.get(resolved, {})
base = info.get("output_cost_per_token")
if long_context:
return info.get("output_cost_per_token_above_200k_tokens") or base or None
from headroom.pricing.counterfactual import long_context_tier
tier = long_context_tier(info)
if tier:
return info.get(f"output_cost_per_token{tier[1]}") or base or None
return base or None
except Exception as e:
_warn_pricing_once("output_price", "Failed to get output price", model, e)
@@ -1540,9 +1581,9 @@ class CostTracker:
``headroom.pricing.cache_ttl`` derives it. Previously every 1h write was
priced at the 5m rate.
``long_context`` picks the above-200k tier for each rate, falling back
per rate to the base one for a model that publishes no long-context
price.
``long_context`` picks the model's long-context tier for each rate,
falling back per rate to the base one for a model that publishes no
long-context price.
Imported at call time: ``headroom.pricing`` eagerly imports litellm
(~4s) and this module is on the proxy's startup path.
@@ -1721,6 +1762,23 @@ class CostTracker:
+ self._saved_list_by_tier.get(key, 0.0) * uncached_price
)
# The long card's premium that tier-crossing requests' forwarded
# tokens avoided (see `_tier_crossing_by_model`).
for model, crossing in self._tier_crossing_by_model.items():
long_prices = self._get_cache_prices(model, long_context=True)
base_prices = self._get_cache_prices(model)
if not long_prices or not base_prices:
continue
premium = sum(
tokens * max(0.0, hi - lo)
for tokens, hi, lo in zip(crossing[:4], long_prices, base_prices, strict=True)
)
long_out = self._get_output_price(model, long_context=True)
base_out = self._get_output_price(model)
if long_out and base_out:
premium += crossing[4] * max(0.0, long_out - base_out)
cache_aware_savings_usd += premium
# Tool-schema deferral, priced CACHE-AWARE rather than flat at list:
# had the schemas shipped they would have been part of the same prefix
# as the rest of the request — cold-written once at the provider's
+1
View File
@@ -932,6 +932,7 @@ class PrometheusMetrics:
compression_tokens_saved=tokens_saved,
tool_schema_tokens_saved=tool_search_saved,
output_tokens_saved=output_tokens_saved,
output_tokens=output_tokens,
cache_read_tokens=cache_read_tokens,
cache_write_tokens=cache_write_tokens,
cache_write_5m_tokens=cache_write_5m_tokens,
+29 -6
View File
@@ -543,6 +543,7 @@ def estimate_request_savings_usd(
compression_tokens_saved: int = 0,
tool_schema_tokens_saved: int = 0,
output_tokens_saved: int = 0,
output_tokens: int = 0,
cache_read_tokens: int = 0,
cache_write_tokens: int = 0,
cache_write_5m_tokens: int = 0,
@@ -588,6 +589,8 @@ def estimate_request_savings_usd(
CacheMix,
PricedSavings,
Region,
long_context_premium_avoided_usd,
long_context_threshold,
price_savings,
weakest_basis,
)
@@ -600,26 +603,45 @@ def estimate_request_savings_usd(
uncached_input_tokens=uncached_input_tokens,
cache_inferred=cache_inferred,
)
long_context = mix.is_long_context(local_tokens=local_input_tokens)
threshold = long_context_threshold(model)
long_context = mix.is_long_context(local_tokens=local_input_tokens, threshold=threshold)
# Compression that kept the prompt under the threshold avoided the long
# card on the whole request: the removed tokens are priced at that card,
# and the forwarded input and generated output premiums are added on top.
forwarded = max(mix.billed, max(_coerce_int(local_input_tokens), 0))
crossed = (
not long_context and forwarded + max(_coerce_int(compression_tokens_saved), 0) > threshold
)
def _price(tokens: Any, region: Region) -> PricedSavings:
def _price(tokens: Any, region: Region, *, long_tier: bool) -> PricedSavings:
return price_savings(
max(_coerce_int(tokens), 0),
model=model,
mix=mix,
region=region,
long_context=long_context,
long_context=long_tier,
provider=provider,
# Matches the blended rate the flat estimators fall back to, so an
# unpriceable model reports the same dollars it always has.
fallback_rate_per_token=DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN,
)
compression = _price(compression_tokens_saved, Region.LIVE_ZONE)
tool_schema = _price(tool_schema_tokens_saved, Region.PREFIX)
compression = _price(
compression_tokens_saved, Region.LIVE_ZONE, long_tier=long_context or crossed
)
tool_schema = _price(tool_schema_tokens_saved, Region.PREFIX, long_tier=long_context)
compression_usd = compression.usd
if crossed:
compression_usd += long_context_premium_avoided_usd(
model,
mix,
local_tokens=local_input_tokens,
output_tokens=output_tokens,
provider=provider,
)
return {
"compression": compression.usd,
"compression": compression_usd,
"compression_list": compression.usd_list,
"tool_schema": tool_schema.usd,
"tool_schema_list": tool_schema.usd_list,
@@ -1461,6 +1483,7 @@ class SavingsTracker:
priced = estimate_request_savings_usd(
model,
compression_tokens_saved=_coerce_int(metrics.get("tokens_saved")),
output_tokens=_coerce_int(metrics.get("output_tokens")),
cache_read_tokens=cache_read_tokens,
cache_write_tokens=cache_write_tokens,
cache_write_5m_tokens=_coerce_int(metrics.get("cache_write_5m_tokens")),
+67 -4
View File
@@ -11,6 +11,8 @@ from __future__ import annotations
from types import SimpleNamespace
import pytest
from tests._dotenv import (
autouse_apply_env,
importorskip_no_env_leak,
@@ -84,10 +86,11 @@ def test_warm_prefix_does_not_drag_the_live_delta_down_to_the_read_rate():
ct.record_tokens(
MODEL,
tokens_saved=10_000,
tokens_sent=200_000,
cache_read_tokens=190_000,
cache_write_tokens=5_000,
uncached_tokens=5_000,
# Inside one tier: the uncompressed 110K stays under the 200K threshold.
tokens_sent=100_000,
cache_read_tokens=95_000,
cache_write_tokens=2_500,
uncached_tokens=2_500,
)
stats = ct.stats()
@@ -173,6 +176,66 @@ def test_long_context_turn_is_priced_at_the_above_200k_rates():
assert abs(stats["cache_aware_savings_usd"] - 10_000 * long_input) < 1e-6
def test_haiku_5_5_prompt_over_100k_is_priced_at_its_long_prompt_card():
"""Haiku 5.5's tier starts at 100K: a 150K turn pays 5x on input and output."""
import litellm
from headroom.proxy.server import CostTracker
info = litellm.model_cost.get("claude-haiku-5-5", {})
if "input_cost_per_token_above_100k_tokens" not in info:
pytest.skip("installed LiteLLM catalog predates Claude Haiku 5.5")
ct = CostTracker()
ct.record_tokens(
"claude-haiku-5-5",
tokens_saved=10_000,
tokens_sent=150_000,
cache_read_tokens=0,
cache_write_tokens=0,
uncached_tokens=150_000,
output_tokens=5_000,
)
stats = ct.stats()
# $2.50/M output and $0.50/M input above 100K (base card: $0.50 / $0.10).
assert abs(stats["output_cost_usd"] - 5_000 * 2.5e-6) < 1e-9
assert abs(stats["cache_aware_savings_usd"] - 10_000 * 5e-7) < 1e-9
def test_compression_below_haiku_5_5_tier_credits_the_avoided_long_card():
"""110K compressed to 90K: the whole request avoided the 5x card.
Uncompressed bill: 110K x $0.50/M + 1K x $2.50/M = $0.0575.
Forwarded bill: 90K x $0.10/M + 1K x $0.50/M = $0.0095.
"""
import litellm
from headroom.proxy.server import CostTracker
if "input_cost_per_token_above_100k_tokens" not in litellm.model_cost.get(
"claude-haiku-5-5", {}
):
pytest.skip("installed LiteLLM catalog predates Claude Haiku 5.5")
ct = CostTracker()
ct.record_tokens(
"claude-haiku-5-5",
tokens_saved=20_000,
tokens_sent=90_000,
cache_read_tokens=0,
cache_write_tokens=0,
uncached_tokens=90_000,
output_tokens=1_000,
)
stats = ct.stats()
# The forwarded request still bills at the base card...
assert abs(stats["output_cost_usd"] - 1_000 * 5e-7) < 1e-12
# ...and the saving is the difference of the two whole bills.
assert abs(stats["cache_aware_savings_usd"] - (0.0575 - 0.0095)) < 1e-9
def _summary(cache_net_usd: float, cost_stats: dict) -> dict:
from headroom.proxy.cost import build_session_summary
+37 -4
View File
@@ -170,17 +170,50 @@ def test_untagged_writes_default_to_the_five_minute_ttl():
assert mix.write_1h == 0
def test_long_context_derives_the_one_hour_rate_and_says_so():
"""No catalog publishes a combined 1h + above-200k rate, so it derives."""
def test_long_context_reads_a_published_one_hour_rate():
"""A row's combined 1h + long-tier write rate is read, not derived."""
import litellm
from headroom.pricing.litellm_pricing import resolve_litellm_model
model = anthropic_pricing_model("input_cost_per_token_above_200k_tokens")
info = litellm.model_cost.get(resolve_litellm_model(model), {})
published = info.get("cache_creation_input_token_cost_above_1hr_above_200k_tokens")
if not published:
pytest.skip("catalog row publishes no combined 1h + above-200k rate")
rates = resolve_rates(model, long_context=True)
assert rates.basis == BASIS_CATALOG_TTL_RATIO
assert rates.write_1h / rates.uncached == pytest.approx(2.00, abs=1e-6)
assert rates.basis == BASIS_CATALOG
assert rates.write_1h == pytest.approx(float(published))
# And it is the EXPENSIVE tier, not the base one.
assert rates.uncached > resolve_rates(model).uncached
def test_long_context_derives_a_missing_one_hour_rate_and_says_so():
"""A long tier without its own 1h write rate derives it from the tier's input."""
import litellm
from headroom.pricing.litellm_pricing import resolve_litellm_model
model = "tiered-model-without-1h-for-test"
litellm.model_cost[resolve_litellm_model(model)] = {
"input_cost_per_token": 3e-6,
"cache_creation_input_token_cost": 3.75e-6,
"cache_creation_input_token_cost_above_1hr": 6e-6,
"input_cost_per_token_above_200k_tokens": 6e-6,
"cache_creation_input_token_cost_above_200k_tokens": 7.5e-6,
}
resolve_rates.cache_clear()
try:
rates = resolve_rates(model, long_context=True)
assert rates.basis == BASIS_CATALOG_TTL_RATIO
assert rates.uncached == pytest.approx(6e-6)
assert rates.write_1h / rates.uncached == pytest.approx(2.00, abs=1e-6)
finally:
litellm.model_cost.pop(resolve_litellm_model(model), None)
resolve_rates.cache_clear()
# ── Provider agnosticism: every harness, every backend ────────────────────
+158
View File
@@ -0,0 +1,158 @@
"""Generated output changes card when compression avoids a long-prompt tier."""
from __future__ import annotations
import pytest
from headroom.pricing.counterfactual import long_context_threshold, resolve_rates
from headroom.proxy.prometheus_metrics import PrometheusMetrics
from headroom.proxy.savings_tracker import SavingsTracker, estimate_request_savings_usd
@pytest.fixture
def haiku_card(monkeypatch):
import litellm
monkeypatch.setitem(
litellm.model_cost,
"claude-haiku-5-5",
{
"litellm_provider": "anthropic",
"input_cost_per_token": 1e-7,
"output_cost_per_token": 5e-7,
"input_cost_per_token_above_100k_tokens": 5e-7,
"output_cost_per_token_above_100k_tokens": 2.5e-6,
},
)
resolve_rates.cache_clear()
long_context_threshold.cache_clear()
yield
resolve_rates.cache_clear()
long_context_threshold.cache_clear()
@pytest.mark.asyncio
@pytest.mark.parametrize("through_metrics", [False, True], ids=["persistent", "metrics"])
@pytest.mark.parametrize(
("forwarded", "removed", "output", "expected"),
[
(90_000, 20_000, 1_000, 0.048),
(110_000, 10_000, 1_000, 0.005),
(90_000, 20_000, 0, 0.046),
(90_000, 10_000, 1_000, 0.001),
(100_000, 10_000, 1_000, 0.047),
(90_000, 0, 1_000, 0.0),
],
ids=[
"crosses",
"stays-above",
"no-output",
"original-at-threshold",
"forwarded-at-threshold",
"no-compression",
],
)
async def test_forwarded_output_premium_only_counts_when_compression_crosses_the_tier(
tmp_path, haiku_card, through_metrics, forwarded, removed, output, expected
):
tracker = SavingsTracker(path=str(tmp_path / "savings.json"), stateless=True)
request = {
"provider": "anthropic",
"model": "claude-haiku-5-5",
"input_tokens": forwarded,
"uncached_input_tokens": forwarded,
"tokens_saved": removed,
"output_tokens": output,
}
if through_metrics:
metrics = PrometheusMetrics(savings_tracker=tracker, stateless=True)
await metrics.record_request(**request, latency_ms=1)
else:
tracker.record_lifetime_request(**request, stack=None)
lifetime = tracker.lifetime_response()
assert lifetime["cost"]["compression_savings_usd"] == pytest.approx(expected)
assert lifetime["tokens"]["output"] == output
if through_metrics:
assert tracker.snapshot()["lifetime"]["output_savings_usd"] == 0
@pytest.mark.asyncio
async def test_avoided_output_premium_is_separate_from_output_shaping(tmp_path, haiku_card):
tracker = SavingsTracker(path=str(tmp_path / "savings.json"), stateless=True)
metrics = PrometheusMetrics(savings_tracker=tracker, stateless=True)
await metrics.record_request(
provider="anthropic",
model="claude-haiku-5-5",
input_tokens=90_000,
uncached_input_tokens=90_000,
tokens_saved=20_000,
output_tokens=1_000,
output_tokens_saved=100,
latency_ms=1,
)
lifetime = tracker.snapshot()["lifetime"]
assert lifetime["compression_savings_usd"] == pytest.approx(0.048)
assert lifetime["output_savings_usd"] == pytest.approx(0.00005)
@pytest.mark.parametrize(
("base", "long", "premium"),
[
(0.0, 2.5e-6, 0.0025),
(5e-7, 0.0, 0.0),
(None, 2.5e-6, 0.0),
(5e-7, None, 0.0),
(5e-7, "invalid", 0.0),
],
ids=["zero-base", "zero-long", "missing-base", "missing-long", "invalid-long"],
)
def test_optional_output_rate_metadata_is_fail_soft(haiku_card, base, long, premium):
import litellm
row = litellm.model_cost["claude-haiku-5-5"]
row["output_cost_per_token"] = base
row["output_cost_per_token_above_100k_tokens"] = long
priced = estimate_request_savings_usd(
"claude-haiku-5-5",
compression_tokens_saved=20_000,
uncached_input_tokens=90_000,
local_input_tokens=90_000,
output_tokens=1_000,
)
assert priced["compression"] == pytest.approx(0.046 + premium)
assert priced["compression_list"] == pytest.approx(0.01)
def test_billing_preserves_explicit_zero_cache_rates_at_the_model_tier(haiku_card):
import litellm
row = litellm.model_cost["claude-haiku-5-5"]
for field in (
"cache_read_input_token_cost",
"cache_creation_input_token_cost",
"cache_creation_input_token_cost_above_1hr",
):
row[f"{field}_above_100k_tokens"] = 0.0
rates = resolve_rates("claude-haiku-5-5", long_context=True, for_billing=True)
assert rates is not None
assert rates.uncached == 5e-7
assert (rates.read, rates.write_5m, rates.write_1h) == (0.0, 0.0, 0.0)
assert rates.read_is_catalog and rates.write_is_catalog
def test_missing_billed_cache_rates_probe_the_model_specific_threshold(haiku_card, monkeypatch):
import litellm
prompts = []
def canonical_cost(**kwargs):
prompts.append(kwargs["prompt_tokens"])
return kwargs["prompt_tokens"] * 3e-8, 0.0
monkeypatch.setattr(litellm, "cost_per_token", canonical_cost)
rates = resolve_rates("claude-haiku-5-5", long_context=True, for_billing=True)
assert rates is not None
assert rates.read == pytest.approx(3e-8)
assert rates.write_5m == pytest.approx(3e-8)
assert prompts == [100_001, 100_001]
assert not rates.read_is_catalog and not rates.write_is_catalog
+2 -2
View File
@@ -113,10 +113,10 @@ class TestAnthropicModelFallback:
"""Test pattern-based inference for haiku models."""
provider = AnthropicProvider()
limit = provider.get_context_limit("claude-haiku-5-20260101")
limit = provider.get_context_limit("claude-haiku-6-20270101")
assert limit == 200000
pricing = provider._get_pricing("claude-haiku-5-20260101")
pricing = provider._get_pricing("claude-haiku-6-20270101")
assert pricing["input"] == 0.80
assert pricing["output"] == 4.00
+162 -1
View File
@@ -180,6 +180,122 @@ class TestLongContextPricing:
cost = manual_provider.estimate_cost(300_000, 5_000, "claude-sonnet-4-5[1m]", 0)
assert cost == pytest.approx(1.9125, rel=1e-4)
# Haiku 5.5's tier starts at 100K and multiplies every rate by 5:
# 100K in / 5K out -> 100K*$0.10 + 5K*$0.50 = $0.0125
# 150K in / 5K out -> 150K*$0.50 + 5K*$2.50 = $0.0875 (long-prompt card)
# 150K in of which 100K cached, 5K out
# -> 50K*$0.50 + 100K*$0.05 + 5K*$2.50 = $0.0425
@pytest.mark.parametrize(
("input_tokens", "output_tokens", "cached_tokens", "expected"),
[
(100_000, 5_000, 0, 0.0125),
(150_000, 5_000, 0, 0.0875),
(150_000, 5_000, 100_000, 0.0425),
],
)
def test_haiku_5_5_long_prompt_card_starts_at_100k(
self, manual_provider, input_tokens, output_tokens, cached_tokens, expected
):
cost = manual_provider.estimate_cost(
input_tokens, output_tokens, "claude-haiku-5-5", cached_tokens
)
assert cost == pytest.approx(expected, rel=1e-4)
@pytest.mark.parametrize(
"model",
# Gateway ids resolve to the same bare row, so they are guarded too.
["claude-haiku-5-5", "anthropic/claude-haiku-5-5", "openrouter/anthropic/claude-haiku-5-5"],
)
def test_haiku_5_5_card_survives_litellm_dropping_the_100k_rate(
self, provider, monkeypatch, model
):
"""A Haiku 5.5 entry without the above-100K rate must not bill a long prompt at the base tier."""
litellm = pytest.importorskip("litellm")
import headroom.providers.anthropic as anthropic_module
# The LiteLLM pricer would return this sentinel; litellm caches model
# info, so the patched row below cannot be relied on to reach it.
monkeypatch.setattr(anthropic_module, "estimate_cost_from_tokens", lambda *a, **k: 0.42)
# Keeps the 200K field Sonnet's tier reads, so only Haiku's own field decides.
monkeypatch.setitem(
litellm.model_cost,
"claude-haiku-5-5",
{
"litellm_provider": "anthropic",
"mode": "chat",
"input_cost_per_token": 1e-07,
"output_cost_per_token": 5e-07,
"cache_read_input_token_cost": 1e-08,
"input_cost_per_token_above_200k_tokens": 5e-07,
},
)
assert provider.estimate_cost(150_000, 5_000, model, 0) == pytest.approx(0.0875, rel=1e-4)
# At the threshold the LiteLLM path still prices it.
assert provider.estimate_cost(100_000, 5_000, model, 0) == 0.42
def test_haiku_5_5_published_100k_rate_stays_on_litellm(self, provider, monkeypatch):
"""With its own above-100K rate published, LiteLLM keeps pricing a long Haiku prompt."""
litellm = pytest.importorskip("litellm")
import headroom.providers.anthropic as anthropic_module
monkeypatch.setitem(
litellm.model_cost,
"claude-haiku-5-5",
{
"litellm_provider": "anthropic",
"mode": "chat",
"input_cost_per_token": 1e-07,
"output_cost_per_token": 5e-07,
"input_cost_per_token_above_100k_tokens": 5e-07,
"output_cost_per_token_above_100k_tokens": 2.5e-06,
},
)
# Stub the LiteLLM pricer: litellm caches model info, so a patched row
# does not reliably reach its own cost_per_token. The manual table
# would return $0.0875, so the sentinel shows which path priced it.
monkeypatch.setattr(anthropic_module, "estimate_cost_from_tokens", lambda *a, **k: 0.42)
assert provider.estimate_cost(150_000, 5_000, "claude-haiku-5-5", 0) == 0.42
@pytest.mark.parametrize(
("model", "expected"),
[
# 150K*$0.50 + 5K*$2.50: Haiku 5.5's long-prompt card.
("anthropic.claude-haiku-5-5-v1:0", 0.0875),
("us.anthropic.claude-haiku-5-5-v1:0", 0.0875),
("claude-haiku-5-5-20261001", 0.0875),
# One version segment away is another model: the `haiku` tier
# default, 150K*$0.80 + 5K*$4, with no Haiku 5.5 card.
("claude-haiku-5", 0.14),
],
)
def test_wrapped_haiku_5_5_ids_get_the_long_prompt_card(self, manual_provider, model, expected):
"""An id the manual table prices as Haiku 5.5 is also billed on its long-prompt card."""
cost = manual_provider.estimate_cost(150_000, 5_000, model, 0)
assert cost == pytest.approx(expected, rel=1e-4)
def test_reseller_flat_haiku_5_5_row_keeps_its_litellm_rate(self, provider, monkeypatch):
"""A reseller row with no long-prompt tier is flat-rated, not a dropped Anthropic rate."""
litellm = pytest.importorskip("litellm")
import headroom.providers.anthropic as anthropic_module
monkeypatch.setitem(
litellm.model_cost,
"perplexity/anthropic/claude-haiku-5-5",
{
"litellm_provider": "perplexity",
"mode": "chat",
"input_cost_per_token": 1e-07,
"output_cost_per_token": 5e-07,
},
)
# Stub the LiteLLM pricer (see above); the manual card would be $0.0875.
monkeypatch.setattr(anthropic_module, "estimate_cost_from_tokens", lambda *a, **k: 0.42)
cost = provider.estimate_cost(150_000, 5_000, "perplexity/anthropic/claude-haiku-5-5", 0)
assert cost == 0.42
class TestLiteLLMCostHelper:
"""The shared helper each provider now uses for LiteLLM-backed pricing.
@@ -324,8 +440,10 @@ class TestAnthropicCostEstimation:
("claude-opus-5-5", {"input": 4.00, "output": 20.00, "cached_input": 0.20}),
("claude-opus-5", {"input": 5.00, "output": 25.00, "cached_input": 0.50}),
("claude-sonnet-5-5", {"input": 2.00, "output": 10.00, "cached_input": 0.20}),
("claude-haiku-5-5", {"input": 0.10, "output": 0.50, "cached_input": 0.01}),
# Suffixed ids must resolve to their own row, not a shorter prefix's.
("claude-sonnet-5-5[1m]", {"input": 2.00, "output": 10.00, "cached_input": 0.20}),
("claude-haiku-5-5-20261001", {"input": 0.10, "output": 0.50, "cached_input": 0.01}),
("claude-fable-5-1-20261001", {"input": 10.00, "output": 50.00, "cached_input": 0.25}),
("claude-opus-5-5-20261001", {"input": 4.00, "output": 20.00, "cached_input": 0.20}),
],
@@ -334,7 +452,50 @@ class TestAnthropicCostEstimation:
assert anthropic_provider._get_pricing(model) == expected
@pytest.mark.parametrize(
"model", ["claude-fable-5-1", "claude-opus-5-5", "claude-opus-5", "claude-sonnet-5-5"]
"model",
[
"claude-fable-5-1",
"claude-opus-5-5",
"claude-opus-5",
"claude-sonnet-5-5",
"claude-haiku-5-5",
],
)
def test_context_limit_claude_5_point_releases(self, anthropic_provider, model):
assert anthropic_provider.get_context_limit(model) == 1_000_000
def test_shorter_id_does_not_inherit_a_newer_release(self, monkeypatch):
"""``claude-haiku-5`` is not ``claude-haiku-5-5``: no 1M window, no 5.5 card.
It falls through to the ``haiku`` pattern default instead, as it did
before Haiku 5.5 had a row.
"""
import headroom.providers.anthropic as anthropic_module
monkeypatch.setattr(anthropic_module, "_get_litellm_clients", lambda: (None, None))
provider = anthropic_module.AnthropicProvider()
haiku_default = anthropic_module._PATTERN_DEFAULTS["haiku"]
assert provider.get_context_limit("claude-haiku-5") == haiku_default["context"]
assert provider._get_pricing("claude-haiku-5") == haiku_default["pricing"]
@pytest.mark.parametrize(
("model", "known_model", "expected"),
[
("claude-sonnet-4-5-20250929", "claude-sonnet-4-5", True),
("anthropic.claude-haiku-5-5-v1:0", "claude-haiku-5-5", True),
("claude-3-5-sonnet", "claude-3-5-sonnet-20241022", True),
("claude-3-5-haiku", "claude-3-5-haiku-latest", True),
("claude-sonnet-4", "claude-sonnet-4-20250514", True),
# One version segment away is another model, in either direction.
("claude-haiku-5", "claude-haiku-5-5", False),
("claude-sonnet-4", "claude-sonnet-4-6", False),
("claude-sonnet-5-5", "claude-sonnet-5", False),
("claude-sonnet-5-5-20261001", "claude-sonnet-5", False),
# A bare fragment is not an alias.
("sonnet", "claude-3-5-sonnet-20241022", False),
],
)
def test_is_release_of(self, model, known_model, expected):
from headroom.providers.anthropic import _is_release_of
assert _is_release_of(model, known_model) is expected
+55
View File
@@ -231,3 +231,58 @@ def test_long_context_requests_price_at_the_above_200k_tier():
assert long_ctx["compression"] > base["compression"]
assert long_ctx["compression_list"] > base["compression_list"]
def test_long_context_tier_threshold_is_read_per_model():
"""Claude Haiku 5.5's long-prompt card starts at 100K, not the usual 200K.
A 150K prompt sits between the two: it must be priced at Haiku's 5x card,
which a shared 200K threshold would miss.
"""
import litellm
from headroom.pricing.counterfactual import long_context_threshold
if "input_cost_per_token_above_100k_tokens" not in litellm.model_cost.get(
"claude-haiku-5-5", {}
):
pytest.skip("installed LiteLLM catalog predates Claude Haiku 5.5")
assert long_context_threshold("claude-haiku-5-5") == 100_000
assert long_context_threshold(MODEL) == 200_000
def _saved(uncached: int) -> dict:
return estimate_request_savings_usd(
model="claude-haiku-5-5",
compression_tokens_saved=1_000,
uncached_input_tokens=uncached,
provider="anthropic",
)
# 1K tokens at $0.10/M below the threshold, at $0.50/M above it.
assert _saved(90_000)["compression_list"] == pytest.approx(1_000 * 1e-7)
assert _saved(150_000)["compression_list"] == pytest.approx(1_000 * 5e-7)
def test_compression_that_crosses_the_tier_credits_the_whole_premium():
"""110K compressed to 90K on Haiku 5.5 avoids the 5x card on every token.
Uncompressed: 110K x $0.50/M = $0.055. Forwarded: 90K x $0.10/M = $0.009.
The saving is $0.046, not the removed 20K at the base rate ($0.002).
"""
import litellm
if "input_cost_per_token_above_100k_tokens" not in litellm.model_cost.get(
"claude-haiku-5-5", {}
):
pytest.skip("installed LiteLLM catalog predates Claude Haiku 5.5")
saved = estimate_request_savings_usd(
model="claude-haiku-5-5",
compression_tokens_saved=20_000,
uncached_input_tokens=90_000,
provider="anthropic",
)
assert saved["compression"] == pytest.approx(0.055 - 0.009)
# The list companion prices only the removed tokens, at the long card.
assert saved["compression_list"] == pytest.approx(20_000 * 5e-7)
@@ -32,6 +32,7 @@ from headroom.transforms.thinking_compactor import bills_prior_thinking
("claude-opus-4-8", True),
("claude-sonnet-5", True),
("claude-sonnet-5-20260101", True),
("claude-haiku-5-5", True),
# Legacy 3.x naming (with date) stays below threshold.
("claude-3-5-sonnet-20241022", False),
("claude-3-opus-20240229", False),