mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 18:41:07 +08:00
feat/https-compose-caddy
9
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
703755e873 |
fix(pricing): add Claude Haiku 5.5 rates, 1M context and its 100K long-prompt card (#4037)
## Current maintainer qualification
Reviewed head
|
||
|
|
0d99c56dc5 |
fix(pricing): add Claude Sonnet 5.5 / Opus 5.5 / Fable 5.1, correct Sonnet 5 rates (#3841)
## Description
Claude Sonnet 5.5 and Opus 5.5 are released, and Anthropic made Claude
Sonnet 5's launch price its standard rate. This PR updates Headroom's
built-in Anthropic fallback price and context tables and the Rust
proxy's vendored LiteLLM snapshot to match current [Anthropic
pricing](https://platform.claude.com/docs/en/about-claude/pricing)
(verified 2026-09-29).
From the pricing page, footnote 3:
> The $2/$10 per million input/output token pricing for Claude Sonnet 5,
announced at launch as introductory pricing through August 31, 2026, is
now the standard price. The previously scheduled increase to $3/$15 per
million input/output tokens on September 1, 2026 will not occur.
This reverses the deliberate $3/$15 pin from #1767, which assumed that
increase would happen.
**Why the missing rows mispriced instead of failing:**
`AnthropicProvider._get_pricing` falls back to substring matching in
dict insertion order, so `claude-sonnet-5-5` silently inherited the
`claude-sonnet-5` row and `claude-fable-5-1` inherited Fable 5's $1
cache read. The Rust `context_window_for` is exact-match, so every
Claude 5-generation id fell through to the 128K default.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
All values in $ / MTok (input / output / cache read).
| Model | Before (fallback) | After | Context |
|---|---|---|---|
| `claude-sonnet-5-5` | *(no row: matched `claude-sonnet-5`)* $3 / $15 /
$0.30 | **$2 / $10 / $0.20** | 1M |
| `claude-sonnet-5` | $3 / $15 / $0.30 | **$2 / $10 / $0.20** | 1M |
| `claude-opus-5-5` | *(no row: opus tier default)* $5 / $25 / $0.50 |
**$4 / $20 / $0.20** | 1M |
| `claude-opus-5` | *(no row: opus tier default)* | $5 / $25 / $0.50 |
1M |
| `claude-fable-5-1` | *(no row: matched `claude-fable-5`)* $10 / $50 /
$1.00 | **$10 / $50 / $0.25** | 1M |
- `headroom/providers/anthropic.py`:
- Adds the four new models to `ANTHROPIC_PRICING` and
`ANTHROPIC_CONTEXT_LIMITS` and corrects `claude-sonnet-5`.
- Newer ids are listed before their prefixes so dated or suffixed ids
(`…-5-5-20261001`, `…-5-5[1m]`) resolve to their own row.
- The `_PATTERN_DEFAULTS` tier rates are unchanged: they also cover
older unlisted models such as Sonnet 3.7.
- `crates/headroom-proxy/data/model_prices_and_context_window.json`:
appends the 113 missing upstream Claude 5-generation keys (first-party,
Bedrock regional/global/Mantle, Vertex, Azure, Databricks, OpenRouter
and similar). No existing entry changes or is removed. The snapshot is
extended rather than re-pulled with `scripts/refresh_model_limits.sh`:
- Upstream has since pruned 258 retired ids the proxy still serves, e.g.
`claude-3-7-sonnet-20250219` and `claude-opus-4-1`. Those would regress
from 200K to the 128K default.
- A full re-pull overwrites the hand-edited DeepSeek rate card from
#3575.
- `crates/headroom-proxy/src/compression/model_limits.rs`: new
`claude_5_generation_models_present` test.
- `tests/test_providers/test_anthropic.py`:
- The Sonnet 5 expectation moves to $2/$10/$0.20.
- New parametrized pricing and 1M-context tests for the new models,
including the prefix-collision cases.
- `tests/test_provider_model_fallback.py`: the "future opus" probe moves
from `claude-opus-5-20260101` to `claude-opus-6-20270101`. Opus 5 is now
a real 1M entry, so the old id no longer exercised the pattern fallback.
This mirrors the existing `claude-sonnet-6` probe.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`) (run on the changed module,
`headroom/providers/anthropic.py`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ cargo test -p headroom-proxy --lib compression::model_limits
test compression::model_limits::tests::claude_5_generation_models_present ... ok
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 271 filtered out
$ HOME=$(mktemp -d) pytest -q tests/test_providers/test_anthropic.py tests/test_provider_model_fallback.py \
tests/test_cost_tracker_totals.py tests/test_pricing_from_litellm.py tests/test_pricing.py \
tests/test_openai_pricing_resolution.py
138 passed, 2 warnings
$ ruff check headroom/providers/anthropic.py tests/test_providers/test_anthropic.py tests/test_provider_model_fallback.py
All checks passed!
$ ruff format --check (same files)
3 files already formatted
$ cargo fmt -p headroom-proxy -- --check
(clean)
$ mypy headroom/providers/anthropic.py
Success: no issues found in 1 source file
```
On the broader pricing, cost and savings subset, 773 pass. The same 12
failures and 3 collection errors occur on unmodified `main` in my
environment, all `ModuleNotFoundError` from optional dev deps (`respx`
and others) that aren't installed locally. `HOME` is isolated because a
developer `~/.headroom/models.json` leaks into
`TestAnthropicConfigLoading`.
## Real Behavior Proof
- Environment: macOS, the repo `.venv` (Python 3.10). `origin/main` and
this PR head were each checked out as a git worktree and run with the
same interpreter and an isolated `HOME`.
- Exact command / steps: priced the same request through the built-in
fallback on `origin/main` and on this PR head (steps below).
- A script built `AnthropicProvider()`, stubbed
`estimate_cost_from_tokens` to return `None` to force the built-in
fallback (the LiteLLM-unavailable or offline path), and priced 1M input
tokens (800K cached) plus 100K output tokens for each model.
- For the Rust side, `context_window_for` semantics (exact key, `mode ==
"chat"`, `max_input_tokens`, 128K default) were applied to each
worktree's vendored JSON.
- Observed result: `origin/main` bills Sonnet 5.5 at $3/$15 and gives
every Claude 5-generation id a 128K Rust window; this PR bills $2/$10
and gives them 1M.
```text
origin/main
claude-sonnet-5-5 rates={'input': 3.0, 'output': 15.0, 'cached_input':
0.3} cost=$2.3400
claude-sonnet-5 rates={'input': 3.0, 'output': 15.0, 'cached_input':
0.3} cost=$2.3400
claude-opus-5-5 rates={'input': 5.0, 'output': 25.0, 'cached_input':
0.5} cost=$3.9000
claude-fable-5-1 rates={'input': 10.0, 'output': 50.0, 'cached_input':
1.0} cost=$7.8000
rust context_window_for: sonnet-5-5 / global.anthropic.claude-sonnet-5-5
/ opus-5-5 / fable-5-1 = 128000
this PR
claude-sonnet-5-5 rates={'input': 2.0, 'output': 10.0, 'cached_input':
0.2} cost=$1.5600
claude-sonnet-5 rates={'input': 2.0, 'output': 10.0, 'cached_input':
0.2} cost=$1.5600
claude-opus-5-5 rates={'input': 4.0, 'output': 20.0, 'cached_input':
0.2} cost=$2.9600
claude-fable-5-1 rates={'input': 10.0, 'output': 50.0, 'cached_input':
0.25} cost=$7.2000
rust context_window_for: all four = 1000000
```
The live LiteLLM `model_cost` already reports the "after" rates for all
of these, so the primary pricing path is unchanged. This fixes the
fallback path and the Rust proxy.
- Not tested: no live upstream Sonnet 5.5 request, no dashboard browser
check (details below).
- No live upstream requests were sent against Claude Sonnet 5.5.
- The dashboard savings tile was not exercised in a browser. It consumes
the same `estimate_cost` path shown above.
## Runtime Rollout Safety
- Rollout-managed feature(s): none. This is static pricing and context
metadata.
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, fallback cost estimates and the
Rust compression budget for Claude 5-generation ids only (details
below).
- Fallback cost estimates for the listed models drop to Anthropic's
published rates.
- The Rust proxy's compression budget for Claude 5-generation ids rises
from 128K to their real 1M window. The proxy now compresses those
requests later instead of over-trimming them.
- Kill switch / disable path: `HEADROOM_MODEL_LIMITS` or
`~/.headroom/models.json` can override any model's context limit or
pricing.
- Unsafe override required: no
- Qualification impact: none expected. Existing models' rows and
vendored entries are byte-identical.
- Rollback path: revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (n/a: no
user-facing docs list model prices)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this) (my original commit did; @JerrettDavis removed it in
|
||
|
|
73a6edbe83 |
fix: never grep-fold timestamped logs, size-weight savings, warn on no-op model limits (#3419)
Three independent fixes from evaluating Headroom in front of a self-hosted vLLM gateway, plus review follow-ups.
- compaction: `_GREP_ROW_RE` matched timestamped log lines (`2026-09-02 14:30:00 [FATAL] ...`, syslog `Aug 16 11:03:22 ...`) as `path:line:content` rows, so search_heading hoisted the date+hour into a heading and the model saw `30:00 [FATAL] ...`. Byte-reversible, so the inverse check could not catch it; guard at the row matcher. Zero false positives on 5,921 real grep rows. Adds a `HEADROOM_LOSSLESS_COMPACTION=0` kill-switch, read per call so the proxy's runtime-env hot-sync applies.
- proxy/cost: `avg_compression_pct` is now weighted by original tokens instead of a mean of per-request ratios, so one tiny highly-compressible request no longer dominates the headline.
- providers/anthropic: warn when `HEADROOM_MODEL_LIMITS` parses but carries neither `context_limits` nor `pricing`, naming the expected shape. Stays quiet when another provider's namespaced section (e.g. `{"openai": {...}}`) carries the keys.
- docs: document `HEADROOM_LOSSLESS_COMPACTION` in the env table.
Co-authored-by: Morteza Rastgoo <5219339+Morteza-Rastgoo@users.noreply.github.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RbB9CAngCNrB3uXNqgHGZe
|
||
|
|
3ed8f76019 |
fix(providers): don't crash on a non-object HEADROOM_MODEL_LIMITS / models.json (#3089)
## Description
`_load_custom_model_config` in both `headroom/providers/anthropic.py`
and `headroom/providers/openai.py` loads the operator's custom model
configuration from `HEADROOM_MODEL_LIMITS` (a JSON string or a file
path) and `~/.headroom/models.json`, then reads it with
`loaded.get(...)`:
```python
loaded = json.loads(env_config) # or json.load(f)
anthropic_config = loaded.get("anthropic", loaded)
```
The `try` guards only `except (json.JSONDecodeError, OSError)`. When the
value is **valid JSON but not an object** (a JSON array, number, string,
bool, or `null`), `json.loads` succeeds and returns a non-dict, so
`loaded.get(...)` raises `AttributeError` — which is *not* one of the
caught types. Instead of the intended warn-and-fall-back-to-defaults, a
misconfigured `HEADROOM_MODEL_LIMITS` (e.g.
`HEADROOM_MODEL_LIMITS='[1,2,3]'` or `'"gpt-4"'`) crashes provider
initialization. The same gap exists in the `models.json` branch of both
providers.
## Fix
After each load, validate `isinstance(loaded, dict)` and raise
`ValueError` with a clear message, and broaden the handler from `except
(json.JSONDecodeError, OSError)` to `except (ValueError, OSError)`.
`json.JSONDecodeError` is a subclass of `ValueError`, so this strictly
supersets the previous handling: every previously-caught malformed value
still warns and falls back, and a valid-JSON-but-non-object value now
does too, instead of crashing.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/providers/anthropic.py` and `headroom/providers/openai.py`
(`_load_custom_model_config`): add an `isinstance(loaded, dict)` guard
(raising `ValueError`) after the env-var load and after the
`models.json` load, and change both `except` clauses to `(ValueError,
OSError)`.
- `tests/test_provider_model_fallback.py`: added parametrized
`test_non_object_env_var_falls_back_to_defaults` (array / string /
number / bool / null) for both providers, and
`test_non_object_config_file_falls_back_to_defaults` for a non-object
`models.json`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added
### Test Output
```text
tests/test_provider_model_fallback.py 44 passed
# uvx ruff@0.15.22 check -> All checks passed!
# uvx mypy@1.20.2 headroom/providers/anthropic.py headroom/providers/openai.py -> Success: no issues found in 2 source files
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.15.22 and mypy 1.20.2 via uvx.
- Exact command / steps: reverted both providers and ran the new
regressions to capture the bug (`python -m pytest
tests/test_provider_model_fallback.py::TestAnthropicConfigLoading::test_non_object_env_var_falls_back_to_defaults
tests/test_provider_model_fallback.py::TestOpenAIConfigLoading::test_non_object_env_var_falls_back_to_defaults
tests/test_provider_model_fallback.py::TestAnthropicConfigLoading::test_non_object_config_file_falls_back_to_defaults`
-> 11 failed with `AttributeError` on `loaded.get` across the
array/string/number/bool/null shapes); restored the fix; re-ran the full
file (`python -m pytest tests/test_provider_model_fallback.py` -> 44
passed); then `uvx ruff@0.15.22 format`, `uvx ruff@0.15.22 check`, and
`uvx mypy@1.20.2` on both providers.
- Observed result: before the fix, `HEADROOM_MODEL_LIMITS='[1,2,3]'` (or
`'"gpt-4"'`, `'42'`, `'true'`, `'null'`) raised `AttributeError` out of
`_load_custom_model_config`; after the fix the same values log a warning
and the loader returns the default `{"context_limits": {}, "pricing":
{}[, "encodings": {}]}`, and a well-formed object config is unchanged.
- Not tested: a live proxy boot with a corrupt `HEADROOM_MODEL_LIMITS`
(the loader is exercised directly, which is the exact function provider
init calls).
## Runtime Rollout Safety
- Rollout-managed feature(s): none. This is defensive parsing in the
provider model-config loader, not a rollout-channel-gated runtime
feature.
- Minimum rollout channel: N/A (no rollout-managed behavior).
- Stable/default behavior changed: only for a previously-crashing input.
A non-object `HEADROOM_MODEL_LIMITS` / `models.json` now warns and uses
built-in defaults instead of raising. Well-formed object configs are
parsed exactly as before.
- Kill switch / disable path: N/A — remove or correct the malformed
config value to load custom limits.
- Unsafe override required: no.
- Qualification impact: a corrupt or mistyped model-limits value
degrades to built-in defaults with a warning rather than failing
provider init.
- Rollback path: revert this PR; the loader returns to catching only
`json.JSONDecodeError`/`OSError`.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`: it is generated by
release-please from my Conventional Commit PR title
## Additional Notes
Both providers carry the same loader shape, so the guard and the widened
`except` are applied identically to keep them in sync. The message names
the offending source (`HEADROOM_MODEL_LIMITS` vs the resolved
config-file path) so the warning is actionable.
|
||
|
|
0cb72f45b2 |
fix(providers): stop a shorter model family shadowing a longer one (#2762)
## Description > **Stacked on #2761** — that PR splits `_lookup_encoding_name` out of `_get_encoding_name_for_model`, which this one builds on. Please merge #2761 first; the diff here will shrink to just this commit afterwards. `_MODEL_ENCODINGS` and `_CONTEXT_LIMITS` are matched by prefix, iterating in **plain dict order** — so the first *inserted* prefix wins rather than the most specific one. `gpt-4.1` matched the `gpt-4` entry: | model | resolved | actual | | |---|---|---|---| | `gpt-4.1` | 8192 | 1,047,576 | **128× under** | | `gpt-4.1-mini` | 8192 | 1,047,576 | **128× under** | | `gpt-4.1-nano` | 8192 | 1,047,576 | **128× under** | | `gpt-4-32k-0613` | 8192 | 32,768 | 4× under | | `gpt-5` / `-mini` / `-nano` | 128,000 | 400,000 | fell to unknown-model default | | `o4-mini` | 128,000 | 200,000 | fell to unknown-model default | A 128× under-estimate matters because the context limit is what tells the proxy how much headroom is left: it treats a 1M-context model as nearly full and compresses accordingly. The same shadowing picked the **wrong encoding** — `gpt-4.1` got `cl100k_base` instead of `o200k_base`. Measured cost of that: ```text cl100k o200k error python code 420 420 +0.0% json blob 555 555 +0.0% logs 580 580 +0.0% english 201 201 +0.0% CJK 600 450 +33.3% ``` So the encoding half is narrow but real — it only bites CJK content, which the repo already treats as a case worth testing (`tests/test_evals_cjk_tokenization.py`). **Scope honestly:** `get_context_limit` consults LiteLLM *before* this table, so the limit half only surfaces where LiteLLM is absent or does not know the model. That is not hypothetical — the `litellm` dependency carries a `python_version < '3.14'` marker (`pyproject.toml:56`), so **any install on Python 3.14+ has no LiteLLM** and this table is load-bearing. The encoding half never had a LiteLLM fallback and was always wrong. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Both prefix loops now iterate `sorted(..., key=len, reverse=True)` — longest prefix wins. This is the root-cause fix: it also protects the *next* model added to these tables. - Added the missing families: `gpt-4.1` (+`-mini`/`-nano`), `gpt-5` (+`-mini`/`-nano`), `o4-mini` to both tables. - Left `supports_model`'s prefix loop alone — it only returns a bool, so order cannot change its answer. Not touched: `_PRICING`. `gpt-4.1`/`gpt-5` also fall through to the GPT-4o pricing tier, which skews cost reporting, but that is a separate concern with its own verification burden (published rates, staleness window) and does not belong in a tokenizer-correctness fix. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check` + `ruff format`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output 12 of 24 new cases fail without the fix; the 12 that pass are the "must not regress" rows (`gpt-4`, `gpt-4-turbo`, `gpt-4o`, `o3`, `gpt-3.5-turbo`) — included precisely so the longest-prefix change can't quietly move them: ```text $ git stash push headroom/providers/openai.py && pytest tests/test_openai_model_table_resolution.py -q FAILED ...::test_context_limit_prefers_the_most_specific_prefix[gpt-4.1-mini-1047576] FAILED ...::test_context_limit_prefers_the_most_specific_prefix[gpt-4.1-nano-1047576] FAILED ...::test_context_limit_prefers_the_most_specific_prefix[gpt-4.1-2025-04-14-1047576] FAILED ...::test_context_limit_prefers_the_most_specific_prefix[gpt-4-32k-0613-32768] FAILED ...::test_context_limit_prefers_the_most_specific_prefix[gpt-5-400000] FAILED ...::test_context_limit_prefers_the_most_specific_prefix[gpt-5-mini-400000] FAILED ...::test_context_limit_prefers_the_most_specific_prefix[o4-mini-200000] FAILED ...::test_encoding_prefers_the_most_specific_prefix[gpt-4.1-o200k_base] FAILED ...::test_encoding_prefers_the_most_specific_prefix[gpt-4.1-mini-o200k_base] FAILED ...::test_encoding_prefers_the_most_specific_prefix[gpt-4.1-2025-04-14-o200k_base] FAILED ...::test_cjk_is_not_over_counted_for_gpt_41 12 failed, 12 passed in 0.70s $ git stash pop && pytest tests/test_openai_model_table_resolution.py -q 24 passed in 0.43s ``` Regression check — 119 suites touching openai / cost / savings / token / compress / outcome / budget, this branch vs clean `main` in the same environment, comparing failure *sets*: ```text branch : 5 failed, 1486 passed, 86 skipped in 111.01s main : 5 failed, 1453 passed, 86 skipped in 132.07s NEW failures introduced: (none) pre-existing on both: test_bundled_tools_savings.py::test_compressed_payload_preserves_answer_anthropic test_compress_route_tokenizer_by_model.py::...[deepseek/deepseek-v4] (needs `transformers`) test_compress_route_tokenizer_by_model.py::...[deepseek/deepseek-v4] (needs `transformers`) test_image_compressor_singleton_reuse.py::test_onnx_router_is_built_once_and_cached test_openai_streaming_backend.py::...test_litellm_vertex_streaming_preserves_max_tokens_and_vendor_fields ``` ```text $ ruff check headroom/providers/openai.py tests/test_openai_model_table_resolution.py All checks passed! $ mypy headroom/providers/openai.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - **Environment:** macOS, Python 3.13.7, isolated worktree at `upstream/main` (`ad56dd38`), no `litellm` installed (matching a Python 3.14+ install, where the dep marker excludes it). - **Exact command / steps:** resolve context limit + encoding for each model against published OpenAI values, before and after. - **Observed result:** ```text before after model limit enc model limit enc gpt-4.1 8192 cl100k gpt-4.1 1047576 o200k_base gpt-4.1-mini 8192 cl100k gpt-4.1-mini 1047576 o200k_base gpt-4.1-nano 8192 cl100k gpt-4.1-nano 1047576 o200k_base gpt-4.1-2025-04-14 8192 cl100k gpt-4.1-2025-04-14 1047576 o200k_base gpt-4-32k-0613 8192 cl100k gpt-4-32k-0613 32768 cl100k_base gpt-5 128000 o200k gpt-5 400000 o200k_base o4-mini 128000 o200k o4-mini 200000 o200k_base unchanged: gpt-4=8192/cl100k, gpt-4-turbo=128000/cl100k, gpt-4o=128000/o200k, o3=200000, gpt-3.5-turbo=16385/cl100k ``` |
||
|
|
e84ca980cf |
feat(anthropic): add Claude 5 family pricing & align current rates (#1767)
## Summary Adds Claude 5 generation metadata and aligns the Anthropic fallback pricing / context-limit tables in `headroom/providers/anthropic.py` with current [Anthropic pricing](https://platform.claude.com/docs/en/about-claude/pricing) (verified 2026-07-04). Several Claude 4.x entries carried stale rates, the new Claude 5 models (Fable 5, Opus 4.8, Sonnet 5) had no metadata, and the generic fallback tests disagreed with the provider tests. Supersedes #1485 (rebased onto latest `main`, squashed to one commit, extended with Sonnet 5 / Fable 5 and the requested fallback-test fixes). ## Changes All values `$ / MTok`; `cached_input` = prompt-cache read = 0.1× input. | Tier | Model | Before | After | Context | |---|---|---|---|---| | Fable | `claude-fable-5` | — *(new)* | $10 / $50 / $1.00 | **1M** | | Opus | `claude-opus-4-8` | — *(new)* | $5 / $25 / $0.50 | **1M** | | Opus | `claude-opus-4-7` | $15 / $75 / $1.50 | $5 / $25 / $0.50 | 1M | | Opus | `claude-opus-4-6` | $15 / $75 / $1.50 | $5 / $25 / $0.50 | 1M | | Opus | `claude-opus-4-5-20251101` | $15 / $75 / $1.50 | $5 / $25 / $0.50 | 200K | | Sonnet | `claude-sonnet-5` | — *(new)* | $3 / $15 / $0.30 | **1M** | | Sonnet | `claude-sonnet-4-6` | — *(new)* | $3 / $15 / $0.30 | **1M** | | Sonnet | `claude-sonnet-4-5` | — *(new)* | $3 / $15 / $0.30 | 200K | | Haiku | `claude-haiku-4-5-20251001` | $0.80 / $4 / $0.08 *(3.5 rates)* | $1 / $5 / $0.10 | 200K | `claude-sonnet-4-20250514` and all Claude 3.x / 3.5.x entries were already correct — left unchanged. The **1M-context** entries (Fable 5, Opus 4.8, Sonnet 5, Sonnet 4.6) are functional, not cosmetic: they ship in the long-context tier, and without explicit entries the `sonnet` / `opus` pattern defaults would report 200K. Sonnet 5 is pinned to the **standard** Sonnet tier ($3 / $15 / $0.30); Anthropic's introductory rate ($2 / $10 through Aug 31 2026) is intentionally not encoded to avoid a time-dependent fixture. ## Fallback-model tests (addresses review on #1485) `_PATTERN_DEFAULTS["opus"]` is aligned to the current Opus tier ($5 / $25 / $0.50) so the generic fallback suite and the provider-specific suite agree: - `test_pricing_for_known_models` — Opus 4.5 pins $5 / $25 / $0.50 - `test_pattern_based_inference_opus` — unknown-opus fallback now $5 / $25 - `test_cost_estimation_for_new_models` — fixture estimate corrected $22.5 → $7.5 - `test_pattern_based_inference_sonnet` — retargeted to `claude-sonnet-6-*` (the old `claude-sonnet-5-*` probe now prefix-matches the real `claude-sonnet-5` key) New provider coverage: - `test_get_context_limit_claude_5_family` — Fable 5 / Opus 4.8 / Sonnet 5 all 1M - `test_pricing_claude_5_family` — exact rate table for the 3 new models Full suite: **48 passed**. ## Source https://platform.claude.com/docs/en/about-claude/pricing |
||
|
|
17ecad9d89 |
fix(gemini): resolve Google model capabilities through ModelRegistry (#1276)
## Description Google model capability lookup was still tied to static provider tables for support checks and context limits. That made plausible future Gemini model ids fail token counting or context lookup even when they clearly belonged to the Google provider family. This change adds a tolerant `ModelRegistry.resolve()` runtime lookup path and routes the Google provider through it. Exact built-in registry matches still win first, LiteLLM pricing metadata can supply live limits when available, and provider-scoped family fallbacks cover future Gemini ids without letting Google claim unrelated models. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `ModelRegistry.resolve()` as a tolerant runtime capability resolver. - Added provider-scoped Google/Gemini family fallbacks for plausible future model ids. - Added support for LiteLLM-style `gemini/gemini-...` model ids in provider inference and family fallback matching. - Updated `GoogleProvider.supports_model()` and `GoogleProvider.get_context_limit()` to use the shared model registry path. - Added regression tests for future Gemini ids, legacy Gemini context limits, and unrelated model rejection. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text uv run --no-project --with pytest --with opentelemetry-api --with pydantic --with tiktoken --with litellm --with click --with rich python -B -m pytest tests/test_provider_model_fallback.py tests/test_models.py 65 passed uv run --no-project --with ruff ruff check headroom/models/registry.py headroom/providers/google.py tests/test_provider_model_fallback.py tests/test_models.py All checks passed! uv run --no-project --with ruff ruff format --check headroom/models/registry.py headroom/providers/google.py tests/test_provider_model_fallback.py tests/test_models.py 4 files already formatted ``` ## Real Behavior Proof - Environment: macOS arm64 local checkout, Python 3.13 virtualenv for editable install; deployed smoke test in a Cloud Run staging service using an earlier commit from this fork branch before the review follow-up. - Exact command / steps: installed `headroom-ai[langchain]` from the fork branch in the staging service, triggered long-context requests that activate Headroom's LangChain compression path, then checked Cloud Run logs after 2026-06-22 12:20 Europe/Paris. - Observed result: Headroom initialized successfully, compressed conversation memory (`23255 -> 5618 chars`), and no logs matched the previous model-resolution failure signatures (`not recognized as a Google model`, `Unknown context limit`). - Not tested: staging was not rerun after the `gemini/gemini-...` review follow-up; that prefix path is covered by local regression tests. Full repository `uv run pytest` on local macOS is currently blocked by a native `maturin`/`esaxx-rs` compile failure (`fatal error: 'cstdint' file not found`). Type checking was not run. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes - Documentation changes are not included because this is a runtime compatibility fix with no public API or user-facing configuration change. - Full local test execution should be retried in CI or a Linux environment where the native Rust extension build is healthy. Co-authored-by: Julien Guarino <julien.guarino@fashiondata.io> |
||
|
|
d00c6739e1 | Fix CI regressions for cache benchmark work | ||
|
|
905c229251 |
Add AST-based code compression and custom model configuration
CodeAwareCompressor: - Tree-sitter based AST parsing for Python, JS, TS, Go, Rust, Java, C, C++ - Preserves imports, signatures, type annotations, error handlers - Guarantees syntactically valid output - Uses tree-sitter-language-pack for broad language support ContentRouter: - Intelligent compression orchestrator - Auto-routes content to optimal compressor based on type detection - Source hint support for high-confidence routing Custom Model Configuration: - HEADROOM_MODEL_LIMITS env var and ~/.headroom/models.json support - Pattern-based inference for unknown models (opus/sonnet/haiku tiers) - Support for Claude 4.5, Claude 4, o3, o3-mini - Graceful fallback - never crashes on unknown models |