mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-23 02:49:17 +08:00
feat/https-compose-caddy
33
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
d21f74e396 |
fix(providers): count Anthropic tool_result/tool_use content blocks in OpenAI token counter (#2681)
## Description
The shared production `count_content_blocks` walker already counts
nested Anthropic tool-result/tool-use blocks on current main. This PR
adds direct OpenAI-provider regression coverage for that existing fix:
payload-sensitive counts, bounded nested media estimates, and the two
sample mapping orders. It makes no production counter change. Related
historical reports: #987, #828 and #813.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Add direct OpenAI regressions for tool-result/tool-use payload counts
and a lower count for a shorter tool-result fixture.
- Verify a nested 1 MB base64 image receives a bounded estimate rather
than text-tokenizing its bytes.
- Retain the sample mapping-order assertion with its scope clarified:
two sample maps have equal counts; this does not promise general
canonical key-order invariance. Main serializes the supplied mapping as
JSON.
- The production walker and both provider counters are unchanged by this
PR.
## Testing
- [x] Focused provider/shared-walker tests: 76 passed, one deprecation
warning.
- [x] Ruff lint and formatting pass on the changed test file.
- [x] Main-relative whitespace checks pass.
- [x] Independent fixture-count probe: 2809 tokens for the large
fixture, 26 for the short fixture.
- [x] Full hosted qualification passed on exact refreshed
head94c44fbe28065c7f31ef97ac2dcd51352d691dac (CI42232683655 and terminal
hosted rollup, no failures).
No old-main versus new-production-fix comparison is claimed; the
production fix is already on main.
## Real Behavior Proof
- Fresh maintainer environment: Windows, Python 3.12, tiktoken counter
for gpt-4o; current-main native DLL used for package startup, not
rebuilt here.
- Exact counter reproduction (run in the configured project
environment):
```python
from headroom.providers.openai import OpenAIProvider
c = OpenAIProvider().get_token_counter("gpt-4o")
big = "row data: " + ", ".join(f"item_{i}=value_{i}" for i in range(400))
small = "row data: item_0=value_0 [398 items compressed. hash=abc123]"
def message(text):
return [{"role": "user", "content": [{"type": "tool_result", "content": text}]}]
print(c.count_messages(message(big)), c.count_messages(message(small)))
```
- Observed: `2809 26`. These are counts for two supplied fixtures, not a
new compression run or an old-main/new-PR comparison; current main
already counts the blocks correctly.
- Focused verification: `pytest -q tests/test_providers/test_openai.py
tests/test_provider_counter_content_blocks.py` — 76 passed, one
deprecation warning.
- Not tested: full live `/v1/compress` HTTP flow, external provider
inference or a fresh branch-specific native build.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A
## Additional Notes
- The production fix for structured nested content landed on `main` via
the shared `count_content_blocks` walker; this branch adds the direct
OpenAI regressions on top of it.
- CHANGELOG is handled by Release Please for this repo, so that
checklist item is N/A.
````
````
Two things before you paste. First, check the boxes I left unchecked — I
marked mypy and "existing unit tests pass locally" as unchecked because
your own notes say you skipped the full suite, and I'd rather you not
assert something you didn't run. Tick them if you have since. Second, if
the template file in your checkout has drifted since #1839, diff against
`.github/PULL_REQUEST_TEMPLATE.md` before saving; the bot matches
headings literally.
Paste into the PR body, hit save, and the governance workflow should
re-run and swap the label.
If you'd rather I did the clicking, install the Claude for Chrome
extension and reconnect — then I can open the PR, fill the body, and
show you the diff before saving anything.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2681"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
### Maintainer current-main requalification
Exact head `94c44fbe28065c7f31ef97ac2dcd51352d691dac`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.
Clean current-main integration with upstream
|
||
|
|
7877241d84 |
feat(providers): add Hermes agent wrap support (#934)
## Description Adds first-class Hermes Agent support to `headroom wrap` while preserving the current provider architecture. The wrapper prepares both OpenAI- and Anthropic-compatible environment variables, supports argument forwarding and `--prepare-only`, and reports a clear installation error when the Hermes binary is unavailable. This update also changes the missing-binary test to assert the complete CLI output, resolving CodeQL's URL-substring sanitization false positive without weakening coverage. ## Changes Made - integrate Hermes launch-environment construction into the existing provider module - route OpenAI and Anthropic endpoints through Headroom - support Hermes argument forwarding and prepare-only setup - cover missing-binary, launch, and compression/passthrough behavior - use an exact expected error message in the CLI test for CodeQL compatibility **Provider contract:** `headroom wrap hermes` requires `--provider anthropic` or `--provider openai`. Any other selection, or no flag, exits with code 1 before the proxy or Hermes starts. ## Testing - [x] Unit tests pass - [x] Linting and formatting pass - [x] New tests added ### Test Output ```text 20 focused Hermes tests passed ruff check: passed ruff format --check: passed git diff --check: passed ``` - `uv run --extra dev pytest tests/test_providers/test_hermes.py tests/test_cli/test_wrap_hermes.py tests/test_hermes_passthrough_compression.py -q` (20 passed) - `uv run --extra dev ruff check tests/test_cli/test_wrap_hermes.py` - `uv run --extra dev ruff format --check tests/test_cli/test_wrap_hermes.py` - `git diff --check` ## UI evidence Not applicable. This PR changes CLI/provider behavior and tests; it does not alter a graphical UI. ## Checklist - [x] Tests added or updated - [x] Focused tests pass locally - [x] Formatting and lint checks pass - [x] No UI screenshots required - [x] No changelog entry required for this focused provider integration ## Type of Change - [ ] Bug fix - [x] New feature - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring ## Real Behavior Proof - Environment: current Hermes provider/wrapper test environment. - Exact command / steps: ran the Hermes provider, CLI wrapper, and passthrough/compression suites plus Ruff, formatting, and diff checks. - Observed result: 20 focused tests pass; OpenAI and Anthropic endpoints route through Headroom, arguments and prepare-only work, and missing binaries produce the exact expected error. - Not tested: a live Hermes Agent session against production providers. ## Runtime Rollout Safety - Rollout-managed feature(s): None. This adds a new opt-in `headroom wrap hermes` command. - Minimum rollout channel: N/A - Stable/default behavior changed: No. Other wrap commands and proxy behavior do not change. - Kill switch / disable path: Do not run `headroom wrap hermes`. Hermes can also be started without the wrapper. - Unsafe override required: No - Qualification impact: None - Rollback path: Revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/934" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ### Maintainer current-main requalification Exact head `ef4c2c11cd6359aaf4239f9cc394c28c23985c15`, upstream main `2d3ee51846435ef5b320eea4806af14e602b0d8c`. Clean mechanical synchronization with upstream main 2d3ee51846435ef5b320eea4806af14e602b0d8c; contributor routing policy unchanged. Current author replies and Devin findings reviewed. The previous default-routing blocker is addressed: the wrapper rejects absent/unsupported provider flags before calling _launch_tool, and permits explicitly selected anthropic/openai providers. The launch-boundary regressions exercise both rejected/default and supported paths. Explicit empty environments remain isolated, and the shared port callback is retained. Hermes/provider/port-alignment tests pass; changed-file Ruff lint/format and main-relative whitespace checks pass. Existing native extension supports Python startup, not a fresh native build. The author's Hermes0.22.2 WSL rejected/default and explicit-provider startup evidence is considered; this round does not claim live authenticated provider traffic. Project attribution, retrieval-plugin setup and a possible registry consolidation remain separate follow-ups rather than claims that the wrapper installs the plugin or supports other providers. No remaining finding in the previously requested provider-guard scope. Formal approval awaits exact-head hosted qualification, handled by maintainers. Author readiness state preserved; nothing merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
51258ef14a |
fix(providers): isolate resolved Anthropic context limits (#4593)
## Description
Addresses the order-dependent Anthropic context-limit resolution
reported in #4390. The proxy reuses its provider instance: resolving
`gpt-4` first currently makes a later `gpt-4-turbo` lookup inherit 8,192
tokens through alias matching, although LiteLLM reports 128,000.
Keep dynamically resolved limits in an exact-model cache so each model
reaches its own metadata lookup. The broader client-supplied
context-limit/header proposal in #4390 remains a separate feature.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Separate resolved limits from the built-in/configured entries used for
release and alias matching.
- Preserve exact-result caching, configured aliases, `[1m]` handling,
and recognition of previously resolved models.
- Cover both lookup orders, both LiteLLM limit fields, unknown-model and
Claude fallbacks, date aliases, repeated lookups, and explicit
overrides.
## Testing
- [x] Relevant unit tests pass
- [x] Linting passes (`python -m ruff check .`)
- [x] Type checking passes for the Linux target (`python -m mypy
headroom --ignore-missing-imports --platform linux`)
- [x] New regression tests added
- [x] Real provider-runtime replay performed
### Test Output
```text
New regressions against
|
||
|
|
703755e873 |
fix(pricing): add Claude Haiku 5.5 rates, 1M context and its 100K long-prompt card (#4037)
## Current maintainer qualification
Reviewed head
|
||
|
|
8c1beb20ae |
fix(pricing): keep the long-context premium when litellm drops the 200k rate (#4109)
## Description
`litellm.model_cost` is downloaded live. It no longer carries
`input_cost_per_token_above_200k_tokens` on any bare Anthropic id. That
broke two things.
1. **Production pricing (bug).** `AnthropicProvider.estimate_cost`
prices through LiteLLM first, so a 300K-token Sonnet 4.5 request was
priced at the base tier: **$0.975 instead of $1.9125**. That is about
half the real cost of exactly the long `[1m]` sessions. Now, when a
request is above the 200K threshold for a tiered model and the model's
LiteLLM entry has no above-threshold rate, the provider prices it from
its own table, which still applies Anthropic's premium (input 2x, output
1.5x, cache 2x). Every other request still goes through LiteLLM
unchanged.
2. **CI on `main` and every open PR.** `tests/_pricing_models.py` could
find no candidate with the full field set, so
`tests/test_cost_card_attribution.py` and
`tests/test_savings_provider_matrix.py` errored at collection. That hid
failure 1, which surfaced as soon as collection worked. The fix adds the
one id that still has every field, the Bedrock Sonnet 4.5 entry
`claude-sonnet-4-5-20250929-v1:0` (same rates), as the last-resort
candidate. These tests only check that Headroom's arithmetic agrees with
LiteLLM's numbers.
Refs #4136
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/providers/anthropic.py`:
`_litellm_lacks_long_context_rate()`; `estimate_cost` skips LiteLLM for
a long-context request whose entry lacks the above-200K rate, and uses
the manual table with its premium.
- `tests/test_providers/test_anthropic.py`: a hermetic regression test
that strips the rate from the LiteLLM entry and asserts $1.9125; it
fails on main.
- `tests/_pricing_models.py`: add `claude-sonnet-4-5-20250929-v1:0` as
the last-resort candidate, with a comment explaining why.
- `tests/test_counterfactual_pricing.py`:
`test_long_context_derives_the_one_hour_rate_and_says_so` priced the
long-context tier of a model chosen for the base fields only. It now
asks `anthropic_pricing_model()` for a model that has that tier. It also
failed on `main`; the collection errors had been hiding it in CI.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest -q -p no:cacheprovider tests/test_cost_card_attribution.py tests/test_savings_provider_matrix.py tests/test_cost_budget_basis.py tests/test_cost_tracker_counterfactual.py tests/test_counterfactual_pricing.py tests/test_savings_ledger.py tests/test_models.py tests/test_tokenizers.py tests/test_realignment_live_multi_turn.py
190 passed, 22 skipped in 4.79s
(on origin/main: 2 collection errors, plus test_long_context_derives_the_one_hour_rate_and_says_so failing)
$ pytest -q tests/test_providers/test_anthropic.py -k LongContext
10 passed (origin/main code: 3 failed, 7 passed: test_litellm_path[300000-...] x2 and the new hermetic test)
$ pytest -q tests/test_providers <all pricing/cost/savings tests>
629 passed, 14 skipped
$ uvx ruff@0.16.8 check / format --check (both files)
All checks passed! / 2 files already formatted
```
## Real Behavior Proof
- Environment: macOS, Python 3.12, repo .venv with litellm 1.101.0 and
the live LiteLLM price map as of 2026-10-09. Clean worktrees of
`origin/main` (
|
||
|
|
0a2c80d520 |
perf(tokenizer): memoize OpenAI token counts like the Anthropic counter (#3168)
## Description
`OpenAITokenCounter.count_text` re-ran `tiktoken.encode` on **every**
call:
```python
def count_text(self, text: str) -> int:
if not text:
return 0
try:
return len(self._encoding.encode(text))
except ValueError:
return len(self._encoding.encode(text, disallowed_special=()))
```
`AnthropicTokenCounter.count_text` already memoizes counts via
`TokenCountCache` (and its own comment even says the OpenAI counter
"Matches AnthropicTokenCounter.count_text" — but it did not: it had no
cache). Both counters are **per-model singletons**, reused across
requests (`OpenAIProvider`/`AnthropicProvider` cache them in
`self._token_counters`). So for OpenAI-model traffic, a stable system
prompt / conversation prefix, or a repeated tool result, was re-encoded
from scratch on every turn — and token counting runs on the request hot
path (often more than once per request across pipeline stages).
This adds the same `TokenCountCache` to `OpenAITokenCounter`, mirroring
the Anthropic counter. `count_text` is a pure function of its text, so a
replayed count is value-identical. The cache only admits strings `>=
256` chars (its existing admission policy), so tiny/rare strings — which
encode in microseconds — pay nothing and never evict the large entries
the cache exists for.
Benchmark (`OpenAITokenCounter("gpt-4o")`, an ~8 KB block, mean of 2000
calls):
```
re-encode each call : 928.7 us/call
cache hit : 0.1 us/call (~11000x faster)
same count : True
```
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/providers/openai.py`: `OpenAITokenCounter.__init__` now
creates a `TokenCountCache`; `count_text` checks/populates it and
delegates the actual encode to a new `_count_text_uncached` (which keeps
the existing `disallowed_special` fallback). Imported `TokenCountCache`.
- `tests/test_providers/test_openai.py`: added
`test_count_text_caches_large_strings` — a large string's count is
memoized and equals the uncached count; a short string stays below the
256-char admission floor.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
### Test Output
```text
tests/test_providers/test_openai.py -> 23 passed
uvx ruff@0.16.2 check headroom/providers/openai.py tests/test_providers/test_openai.py -> All checks passed!
uvx mypy@1.20.2 headroom/providers/openai.py -> Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
tiktoken installed, ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: benchmarked
`OpenAITokenCounter("gpt-4o").count_text` on an ~8 KB block, re-encoding
each call (928.7us) vs a cache hit (0.1us), confirming identical counts.
Removed the cache from `count_text` and ran `python -m pytest
tests/test_providers/test_openai.py -k caches_large_strings` -> failed
(`_count_cache.get(large)` was `None`); restored it -> 23 passed for the
file.
- Observed result: repeated counts of the same large text are served
from the cache instead of re-encoding, with the same value. Behavior for
uncached strings (including the `<|endoftext|>` literal-special-token
case) is unchanged.
- Not tested: an end-to-end proxy turn timing; the counter is exercised
directly, which is the object the pipeline resolves via
`OpenAIProvider.get_token_counter`.
## Runtime Rollout Safety
- Rollout-managed feature(s): none. This is an internal memoization in
the OpenAI token counter, not a rollout-channel-gated runtime feature.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no. Counts are identical; only
repeated counts of the same large text are faster. The cache is
per-counter, GIL-atomic (no lock), and bounded (entries + total chars).
- Kill switch / disable path: N/A.
- Unsafe override required: no.
- Qualification impact: less CPU spent re-encoding stable context on the
token-counting hot path for OpenAI-model traffic; counts unchanged.
- Rollback path: revert this PR; `count_text` returns to encoding on
every call.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal behavior, counts unchanged)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
This just brings the OpenAI counter in line with the Anthropic one,
which already carries `TokenCountCache` for exactly this reason. The
shared cache class is unchanged, so its admission/eviction bounds and
thread-safety argument apply identically here.
|
||
|
|
0d99c56dc5 |
fix(pricing): add Claude Sonnet 5.5 / Opus 5.5 / Fable 5.1, correct Sonnet 5 rates (#3841)
## Description
Claude Sonnet 5.5 and Opus 5.5 are released, and Anthropic made Claude
Sonnet 5's launch price its standard rate. This PR updates Headroom's
built-in Anthropic fallback price and context tables and the Rust
proxy's vendored LiteLLM snapshot to match current [Anthropic
pricing](https://platform.claude.com/docs/en/about-claude/pricing)
(verified 2026-09-29).
From the pricing page, footnote 3:
> The $2/$10 per million input/output token pricing for Claude Sonnet 5,
announced at launch as introductory pricing through August 31, 2026, is
now the standard price. The previously scheduled increase to $3/$15 per
million input/output tokens on September 1, 2026 will not occur.
This reverses the deliberate $3/$15 pin from #1767, which assumed that
increase would happen.
**Why the missing rows mispriced instead of failing:**
`AnthropicProvider._get_pricing` falls back to substring matching in
dict insertion order, so `claude-sonnet-5-5` silently inherited the
`claude-sonnet-5` row and `claude-fable-5-1` inherited Fable 5's $1
cache read. The Rust `context_window_for` is exact-match, so every
Claude 5-generation id fell through to the 128K default.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
All values in $ / MTok (input / output / cache read).
| Model | Before (fallback) | After | Context |
|---|---|---|---|
| `claude-sonnet-5-5` | *(no row: matched `claude-sonnet-5`)* $3 / $15 /
$0.30 | **$2 / $10 / $0.20** | 1M |
| `claude-sonnet-5` | $3 / $15 / $0.30 | **$2 / $10 / $0.20** | 1M |
| `claude-opus-5-5` | *(no row: opus tier default)* $5 / $25 / $0.50 |
**$4 / $20 / $0.20** | 1M |
| `claude-opus-5` | *(no row: opus tier default)* | $5 / $25 / $0.50 |
1M |
| `claude-fable-5-1` | *(no row: matched `claude-fable-5`)* $10 / $50 /
$1.00 | **$10 / $50 / $0.25** | 1M |
- `headroom/providers/anthropic.py`:
- Adds the four new models to `ANTHROPIC_PRICING` and
`ANTHROPIC_CONTEXT_LIMITS` and corrects `claude-sonnet-5`.
- Newer ids are listed before their prefixes so dated or suffixed ids
(`…-5-5-20261001`, `…-5-5[1m]`) resolve to their own row.
- The `_PATTERN_DEFAULTS` tier rates are unchanged: they also cover
older unlisted models such as Sonnet 3.7.
- `crates/headroom-proxy/data/model_prices_and_context_window.json`:
appends the 113 missing upstream Claude 5-generation keys (first-party,
Bedrock regional/global/Mantle, Vertex, Azure, Databricks, OpenRouter
and similar). No existing entry changes or is removed. The snapshot is
extended rather than re-pulled with `scripts/refresh_model_limits.sh`:
- Upstream has since pruned 258 retired ids the proxy still serves, e.g.
`claude-3-7-sonnet-20250219` and `claude-opus-4-1`. Those would regress
from 200K to the 128K default.
- A full re-pull overwrites the hand-edited DeepSeek rate card from
#3575.
- `crates/headroom-proxy/src/compression/model_limits.rs`: new
`claude_5_generation_models_present` test.
- `tests/test_providers/test_anthropic.py`:
- The Sonnet 5 expectation moves to $2/$10/$0.20.
- New parametrized pricing and 1M-context tests for the new models,
including the prefix-collision cases.
- `tests/test_provider_model_fallback.py`: the "future opus" probe moves
from `claude-opus-5-20260101` to `claude-opus-6-20270101`. Opus 5 is now
a real 1M entry, so the old id no longer exercised the pattern fallback.
This mirrors the existing `claude-sonnet-6` probe.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`) (run on the changed module,
`headroom/providers/anthropic.py`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ cargo test -p headroom-proxy --lib compression::model_limits
test compression::model_limits::tests::claude_5_generation_models_present ... ok
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 271 filtered out
$ HOME=$(mktemp -d) pytest -q tests/test_providers/test_anthropic.py tests/test_provider_model_fallback.py \
tests/test_cost_tracker_totals.py tests/test_pricing_from_litellm.py tests/test_pricing.py \
tests/test_openai_pricing_resolution.py
138 passed, 2 warnings
$ ruff check headroom/providers/anthropic.py tests/test_providers/test_anthropic.py tests/test_provider_model_fallback.py
All checks passed!
$ ruff format --check (same files)
3 files already formatted
$ cargo fmt -p headroom-proxy -- --check
(clean)
$ mypy headroom/providers/anthropic.py
Success: no issues found in 1 source file
```
On the broader pricing, cost and savings subset, 773 pass. The same 12
failures and 3 collection errors occur on unmodified `main` in my
environment, all `ModuleNotFoundError` from optional dev deps (`respx`
and others) that aren't installed locally. `HOME` is isolated because a
developer `~/.headroom/models.json` leaks into
`TestAnthropicConfigLoading`.
## Real Behavior Proof
- Environment: macOS, the repo `.venv` (Python 3.10). `origin/main` and
this PR head were each checked out as a git worktree and run with the
same interpreter and an isolated `HOME`.
- Exact command / steps: priced the same request through the built-in
fallback on `origin/main` and on this PR head (steps below).
- A script built `AnthropicProvider()`, stubbed
`estimate_cost_from_tokens` to return `None` to force the built-in
fallback (the LiteLLM-unavailable or offline path), and priced 1M input
tokens (800K cached) plus 100K output tokens for each model.
- For the Rust side, `context_window_for` semantics (exact key, `mode ==
"chat"`, `max_input_tokens`, 128K default) were applied to each
worktree's vendored JSON.
- Observed result: `origin/main` bills Sonnet 5.5 at $3/$15 and gives
every Claude 5-generation id a 128K Rust window; this PR bills $2/$10
and gives them 1M.
```text
origin/main
claude-sonnet-5-5 rates={'input': 3.0, 'output': 15.0, 'cached_input':
0.3} cost=$2.3400
claude-sonnet-5 rates={'input': 3.0, 'output': 15.0, 'cached_input':
0.3} cost=$2.3400
claude-opus-5-5 rates={'input': 5.0, 'output': 25.0, 'cached_input':
0.5} cost=$3.9000
claude-fable-5-1 rates={'input': 10.0, 'output': 50.0, 'cached_input':
1.0} cost=$7.8000
rust context_window_for: sonnet-5-5 / global.anthropic.claude-sonnet-5-5
/ opus-5-5 / fable-5-1 = 128000
this PR
claude-sonnet-5-5 rates={'input': 2.0, 'output': 10.0, 'cached_input':
0.2} cost=$1.5600
claude-sonnet-5 rates={'input': 2.0, 'output': 10.0, 'cached_input':
0.2} cost=$1.5600
claude-opus-5-5 rates={'input': 4.0, 'output': 20.0, 'cached_input':
0.2} cost=$2.9600
claude-fable-5-1 rates={'input': 10.0, 'output': 50.0, 'cached_input':
0.25} cost=$7.2000
rust context_window_for: all four = 1000000
```
The live LiteLLM `model_cost` already reports the "after" rates for all
of these, so the primary pricing path is unchanged. This fixes the
fallback path and the Rust proxy.
- Not tested: no live upstream Sonnet 5.5 request, no dashboard browser
check (details below).
- No live upstream requests were sent against Claude Sonnet 5.5.
- The dashboard savings tile was not exercised in a browser. It consumes
the same `estimate_cost` path shown above.
## Runtime Rollout Safety
- Rollout-managed feature(s): none. This is static pricing and context
metadata.
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, fallback cost estimates and the
Rust compression budget for Claude 5-generation ids only (details
below).
- Fallback cost estimates for the listed models drop to Anthropic's
published rates.
- The Rust proxy's compression budget for Claude 5-generation ids rises
from 128K to their real 1M window. The proxy now compresses those
requests later instead of over-trimming them.
- Kill switch / disable path: `HEADROOM_MODEL_LIMITS` or
`~/.headroom/models.json` can override any model's context limit or
pricing.
- Unsafe override required: no
- Qualification impact: none expected. Existing models' rows and
vendored entries are byte-identical.
- Rollback path: revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (n/a: no
user-facing docs list model prices)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this) (my original commit did; @JerrettDavis removed it in
|
||
|
|
cc6a07cd65 |
perf(providers): cache Google/Cohere token counters across requests (#3273)
## Description
`GoogleProvider.get_token_counter` and
`CohereProvider.get_token_counter` built a **fresh** counter on every
call:
```python
def get_token_counter(self, model: str) -> TokenCounter:
...
return GeminiTokenCounter(model, client=self._client) # new object every call
```
The Anthropic and OpenAI providers do **not** do this — they memoize
counters in a per-model dict (`self._token_counters`) and reuse them
across requests. That matters because each counter owns an
`EstimatingTokenCounter` carrying a `TokenCountCache` (the same
memoization added for the OpenAI counter in #3168). Providers are
long-lived singletons on the proxy hot path
(`ProxyProviderRuntime.pipeline_providers` is built once), so rebuilding
the counter every request threw that cache away every request — and a
stable prefix (system prompt + tools), which is identical turn to turn,
was re-tokenized from scratch on every request for Gemini and Cohere
deployments.
This adds the same `self._token_counters` memoization to both providers,
so the token-count cache (and any lazily-built genai model on the Gemini
counter) persists across requests, exactly like the Anthropic/OpenAI
providers.
Benchmark (`GoogleProvider`, ~11 KB stable prefix, mean of 2000
`get_token_counter(...).count_text(prefix)` calls):
```
fresh counter each call (old): 42.5 us/call
cached counter (new): 0.4 us/call
speedup: 107x
```
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/providers/google.py`: `GoogleProvider.__init__` now
initializes `self._token_counters: dict[str, TokenCounter] = {}`;
`get_token_counter` caches the counter per model instead of constructing
a new `GeminiTokenCounter` each call.
- `headroom/providers/cohere.py`: same change for `CohereProvider` /
`CohereTokenCounter`.
- `tests/test_providers/test_google.py` (new): asserts the counter is
memoized per model (same instance returned; distinct models get distinct
counters) and that an unknown model still raises.
- `tests/test_providers/test_cohere.py`: added the same per-model
caching assertion.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
### Test Output
```text
tests/test_providers/test_google.py tests/test_providers/test_cohere.py -> 20 passed
uvx ruff@0.16.2 check headroom/providers/{google,cohere}.py tests/test_providers/test_{google,cohere}.py -> All checks passed!
uvx mypy@1.20.2 headroom/providers/google.py headroom/providers/cohere.py -> Success: no issues found in 2 source files
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: stashed the two source edits and ran the two
new caching tests -> both FAILED (a second `get_token_counter` returned
a different instance); restored the edits -> 20 passed for the two
files. Benchmarked `GoogleProvider` on an ~11 KB stable prefix: fresh
counter per call 42.5us vs cached counter 0.4us (107x), same count.
- Observed result: `get_token_counter(model)` returns the same counter
instance across calls, so its `TokenCountCache` survives across requests
and the stable prefix is tokenized once instead of every turn. Token
counts are unchanged (same counter class, same estimation).
- Not tested: live Gemini/Cohere API `countTokens`/`tokenize` calls (no
API client configured in this environment); the estimation path and
counter identity are exercised.
## Runtime Rollout Safety
- Rollout-managed feature(s): none — no feature flag or rollout channel
involved.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no. Same counter class and same token
counts; only the counter's lifetime changes (reused instead of rebuilt).
- Kill switch / disable path: N/A (no config surface added).
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this commit; counters go back to being rebuilt
per call.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal behavior, counts unchanged)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
This brings the Google and Cohere providers in line with the Anthropic
and OpenAI providers, which already cache their counters in
`self._token_counters` for exactly this reason. No behavior change
beyond counter lifetime; the counters and the shared cache class are
otherwise untouched.
|
||
|
|
c81378c878 |
fix(grok): preserve xAI model context metadata (#3312) (#3315)
## Description Wrapped Grok Shell receives xAI model metadata with `context_length`, but Grok's custom-endpoint parser reads `context_window` and falls back to 256K when that alias is absent. Normalize the successful xAI model-list response at Headroom's provider metadata boundary, preserving the upstream value and the `grok-4.6` model id. Closes #3312. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Translate a positive xAI `context_length` value to Grok's missing `context_window` field for the resolved `api.x.ai` model list. - Preserve model ids, original xAI fields, explicit context aliases, response status, and non-xAI model metadata. - Leave backend selection unchanged, so this fixes the 256K context fallback without rerouting inference from ChatCompletions to Responses. - Remove entity validators when rewriting model-list bodies, preventing cached metadata from being validated against the unmodified upstream body. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_providers/test_grok_model_metadata.py tests/test_provider_model_metadata.py tests/test_provider_proxy_routes.py tests/test_provider_grok.py tests/test_cli/test_wrap_grok.py -q`) - [x] Linting passes (`uv run ruff check headroom/providers/grok/model_metadata.py headroom/providers/model_metadata.py headroom/proxy/handlers/openai.py tests/test_providers/test_grok_model_metadata.py tests/test_provider_model_metadata.py tests/test_provider_proxy_routes.py`) - [x] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text uv run pytest tests/test_providers/test_grok_model_metadata.py tests/test_provider_model_metadata.py tests/test_provider_proxy_routes.py tests/test_provider_grok.py tests/test_cli/test_wrap_grok.py -q 46 passed, 1 warning in 8.41s uv run ruff check headroom/providers/grok/model_metadata.py headroom/providers/model_metadata.py headroom/proxy/handlers/openai.py tests/test_providers/test_grok_model_metadata.py tests/test_provider_model_metadata.py tests/test_provider_proxy_routes.py All checks passed! uv run ruff format headroom/providers/grok/model_metadata.py headroom/providers/model_metadata.py headroom/proxy/handlers/openai.py tests/test_providers/test_grok_model_metadata.py tests/test_provider_model_metadata.py tests/test_provider_proxy_routes.py --check 6 files already formatted uv run mypy headroom Success: no issues found in 529 source files ``` ## Real Behavior Proof - Environment: Windows, Headroom development build, focused route tests with a local OpenAI-compatible response fixture. - Exact command / steps: Run `tests/test_provider_proxy_routes.py::test_issue_3312_grok_model_metadata`; the fixture sends an xAI-shaped `/v1/models` entry containing `id: grok-4.6` and `context_length: 500000` through the real provider route. - Observed result: The patched route returns `id: grok-4.6`, preserves `context_length: 500000`, and adds `context_window: 500000`. - Not tested: live Grok Shell 1.0.5 session against `api.x.ai` ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: Stable/default. - Stable/default behavior changed: Yes, wrapped xAI model discovery now exposes the upstream context window to Grok. - Kill switch / disable path: Not applicable, revert the change to disable it. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert the change; model-list responses return to the previous metadata shape. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md`, it is generated by release-please from the Conventional Commit PR title. ## Screenshots (if applicable) Not applicable for this provider metadata change. ## Additional Notes The change is scoped to the xAI model-list schema boundary. There is no runtime rollout switch; the default behavior changes only for wrapped xAI model discovery, and reverting the change restores the previous metadata response. Live Grok Shell 1.0.5 verification against `api.x.ai` was not run; the available local binary is Grok 1.0.0. |
||
|
|
b0dfa33478 |
fix(pricing): DeepSeek V4.1-Flash peak/off-peak rate card (#3575)
## Description DeepSeek restructured its model line-up and price card, and Headroom's DeepSeek data was last verified 2026-06-19: the rows still carried the pre-restructure flat rates, `deepseek-flash` was unknown, and nothing could express the vendor's peak/off-peak split. Source (fetched and re-verified today): https://api-docs.deepseek.com/quick_start/pricing | | `deepseek-flash` (V4.1-Flash) | `deepseek-v4-pro` (V4-Pro-0813) | |---|---|---| | Context / max output | 1M / 384K | 1M / 384K | | Input (cache hit) | $0.003 off-peak / $0.006 peak | $0.022 / $0.044 | | Input (cache miss) | $0.15 / $0.30 | $0.66 / $1.32 | | Output | $0.60 / $1.20 | $1.98 / $3.96 | | Vision | yes | no | `deepseek-v4-flash` and `deepseek-v4-flash-vision-exp` are retired ids still accepted on the wire — served by V4.1-Flash and billed at the Flash price. Peak is exactly 2x off-peak; peak hours are 01:00-04:00 and 06:00-10:00 UTC, Monday-Friday. Context note: this surfaced while working on #3036, which routes DeepSeek traffic through the proxy and hard-codes the `deepseek-v4-*` ids in its routing tests. That branch is not touched here, and no issue is closed by this PR — it is a rate-card and model-data refresh. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Added `headroom/pricing/deepseek_tiers.py` as the single source of truth: the off-peak `(cache-hit, cache-miss, output)` table for `deepseek-flash` and `deepseek-v4-pro`, peak derived as exactly 2x, the retired-id aliases, the Beijing peak-window predicate with the weekend rule gated on its effective instant, and the `rates_for(model, now=None)` seam. - Made two cost seams tier-aware: `estimate_cost_from_tokens(..., now=None)` prices DeepSeek flash/pro from the tier selected by the request instant (litellm's `model_cost` holds one flat rate per model and cannot express a tier), and `PricingRegistry.estimate_cost(..., now=None)` prices them through its existing body from a synthesized rate row so there is one arithmetic body rather than two. - Gave `AnthropicProvider.estimate_cost` the same `now` passthrough and derived its DeepSeek fallback table from the tier module. - Derived every flat table from the tier module so no DeepSeek rate literal is left outside it: the `deepseek_prices` rows, the `litellm.model_cost` injection (now overriding and merging upstream rather than filling absent keys), and the Anthropic fallback table. Moved `LAST_UPDATED` / `SOURCE_URL` onto the module that owns the rates and re-exported them so existing import paths keep working. - Updated the model data: `deepseek-flash` registered with vision and both retired ids as aliases, provider context rows, the Hugging Face tokenizer map, the vendored LiteLLM snapshot, and the OpenCode guide's published rate card. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_providers tests/test_pricing.py tests/test_pricing_deepseek_tiers.py \ tests/test_pricing_from_litellm.py tests/test_pricing_litellm.py \ tests/test_pricing_litellm_model_resolution.py tests/test_models.py \ tests/test_huggingface_tokenizer_timeout.py tests/test_tokenizer_count_offload.py \ tests/test_cost_pricing_warning_dedup.py -q 291 passed, 3 warnings in 9.80s $ uv run ruff check . All checks passed! $ uv run mypy headroom headroom\integrations\langchain\chat_model.py:394: error: Function "RunnableBinding" could always be true in boolean context [truthy-function] Found 1 error in 1 file (checked 532 source files) $ cargo test -p headroom-proxy --lib compression::model_limits running 6 tests test compression::model_limits::tests::vendored_json_parses_at_runtime ... ok test compression::model_limits::tests::unknown_model_returns_default ... ok test compression::model_limits::tests::sample_spec_entry_is_excluded ... ok test compression::model_limits::tests::current_claude_models_present ... ok test compression::model_limits::tests::current_gpt_models_present ... ok test compression::model_limits::tests::empty_or_garbage_string_does_not_panic ... ok test result: ok. 6 passed; 0 failed $ uv run pytest tests/test_pricing_deepseek_tiers.py -q # with litellm hidden 47 passed $ uv run mypy headroom/pricing/deepseek_tiers.py headroom/pricing/registry.py \ headroom/pricing/litellm_pricing.py headroom/providers/anthropic.py Success: no issues found in 4 source files ``` The `mypy headroom` box is unticked on purpose: the single error above is pre-existing on `main` in `headroom/integrations/langchain/chat_model.py:394`, a file this PR does not touch (see Additional Notes). ## Real Behavior Proof - Environment: Windows 10, Python 3.13 in the repo's `.venv`, `uv`; litellm 1.88.1 installed (whose bundled cost map predates these ids, so the injected rows are what price them). - Exact command / steps: pinned instants Monday 2026-08-17 02:00 UTC (10:00 Beijing = peak) and 12:00 UTC (20:00 Beijing = off-peak), then: `uv run python -c "from datetime import datetime,timezone; from headroom.pricing.litellm_pricing import estimate_cost_from_tokens as e; P=datetime(2026,8,17,2,tzinfo=timezone.utc); O=datetime(2026,8,17,12,tzinfo=timezone.utc); print(e('deepseek-flash',10**6,10**6,now=O), e('deepseek-flash',10**6,10**6,now=P), e('deepseek-flash',10**6,0,cached_tokens=10**6,now=O), e('deepseek-v4-flash-vision-exp',10**6,10**6,now=O))"` - Observed result: `0.75` (off-peak flash), `1.5` (peak flash, exactly 2x), `0.003` (fully cached prompt billed once at the cache-hit rate), `0.75` (retired id prices as Flash). Also verified: 9 mutants of the new logic (peak multiplier, weekend gate, window start, alias resolution, cached-token handling, registry branch, provider passthrough) were each caught by the new tests, and `tests/test_pricing_deepseek_tiers.py` still passes with litellm absent (47 passed). - Not tested: the full test suite (hours on this box, and much of it cannot build a `TestClient` here because `NO_PROXY` contains a bare `::1`, so `httpx.URLPattern` raises `InvalidURL: Invalid port: ':'` — a pre-existing environment fault, confirmed by stashing this branch and re-running); CNY billing; the proxy budget guard and savings ledger, which remain flat by design (see Additional Notes). ## Runtime Rollout Safety - Rollout-managed feature(s): DeepSeek pricing and model data consumed by the pricing modules, providers, model registry and the Rust proxy's context limits. No feature flags are involved. - Minimum rollout channel: Stable. - Stable/default behavior changed: Yes — DeepSeek costs, model ids and capabilities change; the litellm `model_cost` DeepSeek rows now override upstream; and one context-limit fallback now resolves longest-prefix-first. All are itemised in Additional Notes. - Kill switch / disable path: revert this branch (no flag). Operators can override context limits without a release via `HEADROOM_MODEL_LIMITS` or `~/.headroom/models.json`, and provider pricing via the existing custom pricing config. - Unsafe override required: No. - Qualification impact: None — no schema, wire-format or session-format change; the vendored JSON keeps its upstream shape. - Rollback path: revert the merge commit; the only persisted artifact change is the vendored JSON, which reverts with it. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Screenshots (if applicable) Not applicable — no UI change. ## Additional Notes **Behaviour changes worth a reviewer's attention** 1. Flat figures are the off-peak tier by design; only the two cost seams are tier-aware. The proxy budget guard (`proxy/cost.py`), the savings ledger, `ModelRegistry.estimate_cost` and `OpenAICompatibleProvider.estimate_cost` remain flat or unpriced, so a peak-hour DeepSeek request can be under-estimated by up to 2x there. Making those tier-aware needs the request instant plumbed to each call site — deliberately a separate change. 2. The litellm DeepSeek rows now override upstream instead of filling only absent keys: upstream ships `deepseek-flash` at the *peak* rate, so fill-if-absent would have made the flat figure depend on the installed litellm version. The injection also merges (`{**upstream, **ours}`) so capability flags survive. 3. DeepSeek caps moved `384_000` -> `393_216` in the litellm-shaped tables (upstream's 384x1024 encoding of the published 384K). `ModelInfo` keeps the published `384_000`; the litellm-shaped tables are authoritative wherever litellm is consulted. 4. `OpenAICompatibleProvider`'s context fallback now matches the longest prefix first, mirroring the long-standing behaviour in `OpenAIProvider`. This fixes `deepseek-v4-flash-vision-exp` (was 1_048_576 via the shorter `deepseek-v4` key, now 1_000_000) and also changes the window for unlisted non-DeepSeek variants, e.g. `llama-3.1-8b-instruct` 8192 -> 128000, `mistral-large-2407` 32768 -> 128000, `starcoder2-15b` 8192 -> 16384. 5. `PricingRegistry.estimate_cost` no longer raises for a DeepSeek flash/pro id absent from `prices` — the tier table is authoritative for those four ids. Unknown non-DeepSeek ids still raise, and a flat row for a tiered id is deliberately ignored. 6. Weekend off-peak applies only from 2026-08-22T16:00Z, so a back-dated or replayed weekend request before that instant keeps the weekday windows rather than being silently repriced. `now` exists only on `AnthropicProvider.estimate_cost`; every other provider reads the wall clock for live traffic. 7. CNY billing is not modelled. The zh-cn page lists the same tiers in CNY, but the two lists are not one FX rate apart; supporting CNY-billed accounts needs a switch over two published lists, not a baked constant. **N/A / notes on checklist items** - `Type checking passes (mypy headroom)` is unticked for the pre-existing error above. `git diff --name-only origin/main..HEAD` contains no `integrations/langchain` path, so this PR does not introduce or touch it; `mypy` is clean on all four changed modules. - Out of scope: the stale `deepseek-v3` / `deepseek-coder` registry rows and the old `deepseek-chat` / `deepseek-reasoner` / `deepseek-v3.2` context rows (the vendor no longer lists them; pruning them is a behavioural change of its own). - The design and implementation plan for this change live in a separate workspace, so the decisions they record are inlined above rather than linked. --------- Co-authored-by: Zenjibad <316719033+Zenjibad@users.noreply.github.com> Co-authored-by: DSH Agent <agent@local> |
||
|
|
89a58fd152 |
fix(pricing): price savings against the cache mix, not flat list price (#3699)
## The problem Every savings figure Headroom reports is a counterfactual: *what would this request have cost had we not removed those tokens?* We answered it with `tokens_saved * input_cost_per_token` — flat list price — which is true **exactly once**, for the first request of a cache window. After that, the tokens we removed would have been served from the provider's prompt cache at a fraction of list price (Anthropic 0.10x, OpenAI 0.50x, Gemini 0.10x). Measured on a real 238M-input-token corpus that was **87.8% cache reads**: | | | |---|---| | effective rate actually **paid** | $1.592/M | | rate savings were **priced at** | $4.339/M — **2.73x** | | reported compression savings | **$330.89** | | same tokens at the observed mix | ~$121 | For **tool-schema deferral** the gap is ~10x: tool definitions sit at the very front of the prompt and are the most cacheable tokens in it. ## The fix The correct rate depends on **where in the request** the tokens sat: | Region | What | Pricing | |---|---|---| | `PREFIX` | tool schemas — ahead of every cache breakpoint | warm turn = **entirely a cache read** (read-first, not pro-rata) | | `LIVE_ZONE` | the appended delta handlers compress | **never** a cache read; cold turn saves 1.25x list, not 1.0x | `headroom/pricing/counterfactual.py` is now the single place cache rates are resolved and counterfactual tokens are priced — for the live cost card, the persisted tracker and the durable ledger alike, so the three cannot drift apart again. It replaces three separate implementations, two of which were list-only. Verified, warm Anthropic turn, 8k deferred tool-schema tokens: `$0.0240 list → $0.0057` (4.2x correction). Cold turn: `$0.0240 → $0.0300` — correctly **above** list, the case list pricing *under*-reported. ## Also fixed - **1h-TTL writes were priced at the 5m rate.** The proxy counted and displayed the 5m/1h split but never priced it. LiteLLM publishes the 1h rate per model (Sonnet: 2.00x base); `pricing/cache_ttl.py` — written for exactly this and never imported by the cost path — derives it where a row omits one. - **Prefix tokens were bucketed pro-rata**, handing them a share of the 1.25x write premium they never paid (~2x high on a warm turn). - **OTel exporter crash risk**: it called `float()` over every value in the savings breakdown, so a non-numeric field would take the whole request-metrics path down. - **Ledger compaction thrash** (pre-existing, amplified here): `_maybe_compact` rewrote the entire file under `flock` on *every append* once it passed 1 MB with nothing out of retention to drop. Added a per-process back-off; retention is still enforced on read. ## Schema changes **Ledger v1 → v2**, **savings state v5 → v6.** v1 stored one list-priced dollar and no mix, so its history was not merely overstated — it was **unfixable**, because nothing on disk said what fraction of the removed tokens would have been cache reads. v2 stores the savings split and the observed mix, so a future basis change can re-derive the whole ledger. Pre-v6 state seeds both columns from the single figure it has and is labelled `list` for good; `savings_basis_migrated_at` marks where the honest numbers start. v1 ledger events still read and aggregate alongside v2. ## Honesty over precision Every figure now ships with its list-price ceiling and a `basis` label saying how soundly it was derived (`catalog`, `catalog+ttl-ratio`, `provider-ratio`, `list`, `no-mix`, `unpriced`). Totals report their **weakest** contributing basis, so one unpriceable request can't launder a total into looking catalog-grade. A request with no cache breakdown — the MCP tool path, a non-reporting gateway, a streaming turn with no usage frame — still prices at list and **says so**. ## Harness- and provider-agnostic The fix sits at the single `emit_request_outcome` → `record_request` chokepoint that all **32** handler emit sites route through (anthropic 6, openai 11, batch 5, gemini 3, streaming 3, bedrock 1, gateway_turn 2, server 1) — no per-handler work. `tests/test_savings_provider_matrix.py` asserts the matrix directly: | Provider | Reports | Handled | |---|---|---| | Anthropic / Bedrock / Vertex | read + write + 5m/1h + uncached | full | | OpenAI | read + uncached, write **inferred** | inferred write dropped — no premium OpenAI doesn't charge | | Gemini | read only | prices from reads alone | | gateway / MCP | nothing | list + labelled `no-mix` | ## Tests - **New:** 21 pricing + 13 provider-matrix + 3 ledger (compaction back-off, v1/v2 mixing) - **Updated:** 6 files for the deliberate schema change - Local sweep across everything touching savings/cost/pricing/metrics: **904 passed**; targeted re-run after review fixes: **125 passed** - `ruff check` + `format` clean under the **CI-pinned 0.16.7** (local 0.16.3 missed 5 C408s) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
1bea0ea31a |
test: track active LiteLLM DeepSeek pricing (#3161)
## Description Keep the LiteLLM DeepSeek V4 integration tests compatible with upstream-owned pricing entries. LiteLLM now publishes these models directly, so Headroom correctly preserves upstream values instead of installing its fallback values; the tests must validate the active entry rather than require fallback prices. Related: #3157 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Validate that active upstream DeepSeek V4 price entries contain positive input and output prices. - Compare `cost_per_token` results with the active LiteLLM model-cost entry. - Preserve the existing fallback-price and non-overwrite coverage. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text python -m pytest tests/test_providers/test_deepseek.py -q 20 passed in 4.63s ruff check tests/test_providers/test_deepseek.py All checks passed! ruff format --check tests/test_providers/test_deepseek.py 1 file already formatted pre-commit: Ruff alignment, merge-conflict check, Ruff, Ruff format, and mypy all passed ``` ## Real Behavior Proof - Environment: Windows, Python 3.13.13, LiteLLM model-cost data available. - Exact command / steps: `python -m pytest tests/test_providers/test_deepseek.py -q` - Observed result: all 20 DeepSeek provider and pricing tests pass against the active LiteLLM entries. - Not tested: provider API calls; this change only concerns local pricing metadata assertions. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: N/A. - Stable/default behavior changed: No runtime behavior changes. - Kill switch / disable path: N/A. - Unsafe override required: No. - Qualification impact: Restores deterministic CI coverage for upstream-owned pricing entries. - Rollback path: Revert this test-only commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code where needed - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] Existing tests prove the fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — test-only change. ## Additional Notes Documentation changes are not applicable because runtime behavior and public APIs are unchanged. |
||
|
|
6d2254dfb5 |
fix(anthropic): honor the [1m] 1M-context tier, and price it correctly (#3073)
Two coupled defects on Anthropic's 1M-context tier: Headroom **under-budgeted** those sessions and **under-priced** them by ~2x. The second gets worse once the first is fixed, so they ship together. --- # Part 1 — `[1m]` was lost before the context budget was sized `sanitize_anthropic_model_id()` strips a trailing `[1m]`, which is correct for the wire — upstream Anthropic rejects the suffix, and #2027 added the strip for exactly that reason. But `[1m]` is not only an ANSI artifact. Claude Code appends it to a model id to request the **1M context tier**, and only sends the `context-1m` beta header when it is present (#1158 — what `headroom wrap claude --1m` sets up). `get_context_limit()` sanitized *before* resolving, so the tier was gone by lookup time: ```python provider.get_context_limit("claude-sonnet-4-5[1m]") # 200_000 ← real window is 1M ``` The request still reached Anthropic correctly and still got a 1M window — the beta header goes through untouched. What broke is our **budget**: Headroom sized a 1M session at 200K and began compacting at a fifth of the available room. Models whose base is already 1M (`claude-opus-5`, `claude-sonnet-5`) resolved to 1M either way, which is why this went unnoticed. It bites the Sonnet 4 / 4.5 family — the models `[1m]` exists for. **Fix:** read the tier off the id *before* sanitizing; raise the resolved limit to at least 1M. `max()` rather than assignment, so a base wider than 1M keeps its own window. Detection is deliberately narrower than the sanitizer — only a literal `[1m]`; `[0m]`, `[1;32m]` and real `ESC[` sequences still strip without promoting. | model id | wire id (unchanged) | limit before | limit after | |---|---|---|---| | `claude-sonnet-4-5` | `claude-sonnet-4-5` | 200K | 200K | | `claude-sonnet-4-5[1m]` | `claude-sonnet-4-5` | **200K** | **1M** | | `claude-opus-5[1m]` | `claude-opus-5` | 1M | 1M | | `claude-sonnet-4-5[0m]` | `claude-sonnet-4-5` | 200K | 200K | | `ESC[1m claude-sonnet-4-5 ESC[0m` | `claude-sonnet-4-5` | 200K | 200K | The wire id is unchanged in every case, so #2027 holds — guarded by a regression test. --- # Part 2 — the pricing that reports those sessions was wrong ### 2a. The LiteLLM cost path was dead in every provider `litellm.completion_cost()` no longer accepts `prompt_tokens` / `completion_tokens`. Every call raised `TypeError`: ``` TypeError: completion_cost() got an unexpected keyword argument 'prompt_tokens' ``` All five providers — `anthropic`, `openai`, `google`, `cohere`, `litellm` — caught it with a bare `except` and silently fell through to their hand-maintained tables. The "up-to-date pricing from LiteLLM" the docstrings promise **has not run at all**. Anthropic additionally passed `input_tokens - cached_tokens`, the wrong convention (LiteLLM expects the cache-inclusive total), which would also have suppressed the long-context threshold even had the call worked. Replaced with `litellm.cost_per_token()` behind one shared helper, `pricing.litellm_pricing.estimate_cost_from_tokens()`, which reuses the existing gateway-alias candidate chain and returns `None` (not an exception) when LiteLLM can't price a model. ### 2b. Neither path applied Anthropic's long-context premium On the Sonnet 4 / 4.5 family a prompt over 200K re-prices the **whole** request — input 2×, output 1.5×, cache 2× — not just the tokens past the threshold. Rates confirmed from LiteLLM's `*_above_200k_tokens` fields. | request (`claude-sonnet-4-5`) | reported before | true | error | |---|---|---|---| | 100K in / 5K out | $0.3750 | $0.3750 | — | | 300K in / 5K out | $0.9750 | **$1.9125** | −49% | | 300K in (150K cached) / 5K out | $0.5700 | **$1.1025** | −48% | LiteLLM applies this itself once the call works. The manual fallback needed `_apply_long_context_premium()` — the LiteLLM dependency is gated `python_version < '3.14'`, so on 3.14 the fallback is the *only* path. **Both paths now agree to four decimal places on every case under test.** --- ## What I checked and did *not* change The fork report that prompted this claimed the Anthropic tables were materially stale ("Opus 4.x priced wrong"). **That does not hold.** I audited every entry against LiteLLM's vendored table: - **Anthropic** — every model LiteLLM knows matches exactly, Opus 4.x included. - **OpenAI** — all 17 entries match; the two that don't resolve are retired models. The defect was the mechanism, not the numbers, so the rate cards are untouched. One thing the repaired path fixes for free: OpenAI's cached-input discount is **50% on gpt-4o, 75% on gpt-4.1, 90% on gpt-5**, but the manual path applies a flat 50% estimate. With LiteLLM live, real per-model rates are used. The flat estimate remains only as the offline fallback. ## Scope **No Rust change needed.** `crates/headroom-proxy/src/compression/model_limits.rs` resolves context windows but has **no in-tree callers**; the Rust `[1m]` handling is wire-body sanitization only, correct as-is, and its integration tests assert behavior this PR does not touch. **Judgment call worth a reviewer's eye:** the `[1m]` marker is honored for *any* model, including ones with no 1M tier (`claude-haiku-4-5-20251001[1m]` → 1M). Gating on an allowlist would be more precise but reintroduces a hand-maintained table that rots — the failure mode `model_limits.rs` already documents against. Since `[1m]` is set by our own wrapper and Claude Code's opt-in, honoring it seemed the better default. Happy to tighten. ## Tests - `TestContext1MSuffix` — detection, the 200K→1M promotion, the `max()` floor, ANSI non-promotion, and the wire-id guard for #2027. - `TestLongContextPricing` — the premium on both paths (parametrized), threshold boundary (200,000 vs 200,001), an untiered model charged no premium, and the two halves meeting: a `[1m]` request gets both the 1M window and the premium rate. - `TestLiteLLMCostHelper` — unknown model returns `None`, a known model prices correctly, and `input_tokens` is cache-inclusive. Two existing tests were updated, both pinned to the broken behavior: - `test_estimate_cost_basic` probed a "per 1M" rate by sending exactly 1M tokens, which now crosses the 200K threshold. Re-probed at 100K. (Worth knowing: `claude-3-5-sonnet-20241022` is retired and no longer in LiteLLM, so the alias chain resolves it to `claude-sonnet-4-20250514` and it inherits that model's tier. Harmless — a 200K-window model can't exceed 200K in reality — but it explains the number.) - `test_litellm_provider_info_and_cost_fallbacks` monkeypatched `litellm.completion_cost`; repointed at the new helper seam. ``` ruff check / ruff format / mypy — clean across all six changed source files ``` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
0805e8e410 |
fix(providers/openai): bound tiktoken vocab loads with the guarded loader (#2554)
## Description
`headroom/providers/openai.py::_get_encoding` calls
`tiktoken.get_encoding` directly. tiktoken downloads missing
vocabularies via `requests.get` with **no timeout**, so on a network
that blackholes the vocab CDN (corporate firewall, SSL-intercepting
proxy), whichever thread first counts tokens for an OpenAI model — proxy
startup included — blocks indefinitely.
This is the provider-path hole left by #956: the tokenizer registry
already routes through a bounded loader
(`headroom/tokenizers/tiktoken_counter.py`, worker-thread load +
`HEADROOM_TIKTOKEN_LOAD_TIMEOUT_SECONDS`, default 10s) and falls back to
estimation, but the OpenAI provider path never got the same treatment.
Observed in production (Headroom Desktop fleet, Sentry): a proxy that
never finished booting, with a faulthandler dump wedged in
`tiktoken/registry.py` `get_encoding` on the main thread, reached from
the `headroom` CLI entrypoint via click. The desktop app now also
pre-seeds a persistent `TIKTOKEN_CACHE_DIR`, but the unbounded load
affects every deployment of the proxy, so it should be fixed here too.
Follow-up to #956.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `_get_encoding` now routes through the bounded `load_encoding` from
`headroom.tokenizers.tiktoken_counter` instead of calling
`tiktoken.get_encoding` directly, so a stalled vocab download raises
`TiktokenLoadError` after the timeout instead of hanging the calling
thread.
- `OpenAIProvider.get_token_counter` catches `TiktokenLoadError` and
falls back to `EstimatingTokenCounter`, cached per model so later
requests never re-block on the same failed download — mirroring
`TokenizerRegistry._create_tiktoken`.
- `TIKTOKEN_AVAILABLE` uses `importlib.util.find_spec` (the module-level
`import tiktoken` became unused; same pattern as `LITELLM_AVAILABLE`).
- Two regression tests (`TestGuardedEncodingLoad`) covering the
bounded-raise path and the cached estimation fallback.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uv run --frozen --extra dev pytest tests/test_providers/ tests/test_tokenizers/
======================= 117 passed, 4 warnings in 27.95s =======================
$ uvx ruff check headroom/providers/openai.py tests/test_providers/test_openai.py
All checks passed!
$ uvx ruff format --check headroom/providers/openai.py tests/test_providers/test_openai.py
2 files already formatted
$ uv run --frozen --extra dev mypy headroom/providers/openai.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: macOS (arm64), Python 3.12, uv-managed venv, branch off
`upstream/main` (
|
||
|
|
08466f3cae |
fix(providers/anthropic): don't crash token estimation on null tool_calls (#2472)
## Description
`AnthropicTokenCounter._count_message_estimated` (the
tiktoken-approximation fallback used when no Anthropic client is
available) counted OpenAI-format tool calls like this:
```python
if "tool_calls" in message:
for tool_call in message.get("tool_calls", []):
if isinstance(tool_call, dict):
func = tool_call.get("function", {})
...
```
The `if "tool_calls" in message` check only tests key presence, not the
value. OpenAI SDKs routinely include `"tool_calls": null` on an
assistant message with no tool calls, so `message.get("tool_calls", [])`
returned `None` (the default only applies when the key is absent) and
`for tool_call in None` raised `TypeError: 'NoneType' object is not
iterable`. That crashes token estimation for the entire request whenever
such a message is present. `tool_call.get("function", {})` had the same
gap for a `"function": null`.
## Fix
Iterate `message.get("tool_calls") or []` so a null or absent value
becomes an empty list, and read `function` with `or {}` for the same
reason. Valid tool calls are counted exactly as before.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/providers/anthropic.py`: value-guard `tool_calls` and
`function` in `_count_message_estimated`.
- `tests/test_providers/test_anthropic.py`: regression counting a
message list that includes `tool_calls: null` and a tool call with
`function: null`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_providers/test_anthropic.py -q
17 passed
# with the fix reverted, the new test fails with
# TypeError: 'NoneType' object is not iterable
$ uvx ruff@0.15.17 check headroom/providers/anthropic.py tests/test_providers/test_anthropic.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/providers/anthropic.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, project venv (`uv sync --extra
proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`, pytest in the venv.
- Exact command / steps: built a real
`AnthropicTokenCounter('claude-3-5-sonnet-20241022')` and called
`count_messages` / `_count_message_estimated` with an assistant message
carrying `tool_calls: null` and one carrying `function: null`, plus a
valid tool call; then reverted `anthropic.py` and re-ran.
- Observed result: with the fix the null shapes count without error and
a valid tool call still adds its name/arguments tokens (5 -> 11 on the
sample); with the fix reverted the `tool_calls: null` message raises
`TypeError: 'NoneType' object is not iterable`. Ran against the actual
module.
- Not tested: a live request from an SDK that emits `tool_calls: null`,
end to end.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
|
||
|
|
06add9e9d8 |
fix(providers): stop pricing modern content blocks at zero (#2760)
## Description Each token counter in `headroom/providers/` had grown its own shortened content-block walker, handling only the shapes its provider was expected to send. Everything else fell through and contributed **nothing**. Measured on one 6,800-char block, via `count_messages` of a single-block message — so 7–8 is message overhead alone: | block type | OpenAI ctr | Anthropic ctr | |---|---|---| | `text` (control) | 3409 | 3748 | | `tool_result` | **8** | 3748 | | `thinking` | **8** | **7** | | `document` | **8** | **7** | | `mcp_tool_result` | **8** | **7** | | `output_text` | **8** | **7** | | `refusal` | **8** | **7** | Two things make this worse than a coverage gap: 1. **Each counter zeroed blocks from its own provider.** `output_text` and `refusal` are OpenAI Responses shapes; `thinking` and `document` are Anthropic's. 2. **These are the counters the live pipelines use.** `proxy/server.py` builds them with `AnthropicProvider` / `OpenAIProvider`, so this is the main request path — not an edge case. #2743 fixed this for `/v1/compress` only, by routing that route to the registry tokenizers, whose `BaseTokenizer` walker is complete. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made Rather than add a **fifth** partial walker, the counters now delegate to the audited one: - `tokenizers/base.py` — new `count_content_blocks(parts, count_text_fn)` plus a thin `_DelegatingBlockCounter` adapter, since the provider counters are not `BaseTokenizer` subclasses. `BaseTokenizer` itself is untouched. - `providers/openai.py`, `providers/anthropic.py`, `providers/openai_compatible.py` — list-content branches delegate. **Why delegate instead of adding a `count_text(str(block))` catch-all:** that would serialize a base64 blob and price it as text. `tiktoken_counter.py` already documents the failure — a 1MB image becomes ~330K phantom tokens. The shared walker gives media a pixel/byte-based estimate. **Scope:** the three counters that accumulate token counts. `google.py` and `cohere.py` extract a *text string* first and count that, so the same defect there needs a differently-shaped fix — left as a follow-up. ## Testing - [x] Unit tests pass - [x] Linting passes (`ruff check` + `format --check`, pinned 0.15.17) - [x] New tests added - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_provider_counter_content_blocks.py -q 12 passed in 0.52s $ uvx ruff@0.15.17 check headroom/ tests/... --exclude headroom/dashboard/templates All checks passed! ``` **After the fix**, every shape lands within ~1% of the equivalent plain text, and media stays bounded: ```text block OpenAI Anthropic text (control) 3409 3748 tool_result 3409 3748 thinking 3419 3759 document 3423 3763 mcp_tool_result 3422 3762 output_text 3420 3760 refusal 3421 3761 image b64 200KB 1608 1607 <- pixel estimate, not ~50K as text ``` ## Real Behavior Proof — including a regression I caught **This change flipped an existing test**, and I only found it because every suite was run against clean `upstream/main` in the same environment with the failure sets diffed: ```text before the test rewrite: upstream/main : 1 failed, 104 passed this branch : 2 failed, 103 passed <- regression diff : + test_openai_compatible_token_counter_ignores_unhandled_content_shapes ``` That test asserted `content: [{"type": "image"}, 123] == 8` — i.e. it **pinned the defect**, that unhandled shapes contribute nothing. Rewritten as `..._prices_declared_media`: a declared image is now priced (1608) while a bare int is still correctly ignored (8), with the rationale in the docstring. ```text after the rewrite: upstream/main : 1 failed, 104 passed, 10 skipped, 25 errors this branch : 1 failed, 104 passed, 10 skipped, 25 errors failure sets : IDENTICAL ``` - **Pre-existing, not from this change:** the 1 failure and all 25 errors. The errors are all in `test_compress_route_tokenizer_by_model.py`, whose loopback `TestClient` fixture this throwaway env cannot satisfy. - **Environment note:** `content_router` and several suites need the compiled `headroom._core`, which isn't in a fresh worktree (gitignored, built in-place). I copied the built `.so` in to run these and removed it before committing. - **Not tested:** no live provider call, so the *absolute* accuracy of the 1600 image estimate against a real Anthropic/OpenAI bill is unverified — it is the value `BaseTokenizer` already used, and this PR only changes which blocks reach it. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] I did **not** edit `CHANGELOG.md` ## Related Fourth PR from one tokenizer-consistency audit: #2757 (litellm total prompt / `--budget`), #2758 (HuggingFace chat templates, `gpt-5`, gateway-wrapped names), #2759 (router token units). Plus #2756, which splits the local/provider token scales in `RequestOutcome`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
|
b976378c3e |
test(pricing): stop asserting DeepSeek pricing freshness on wall-clock time (#2428)
## Description The shared `test` job is currently failing on every open PR because of a wall-clock time-bomb in the DeepSeek pricing tests, not because of any code change. `tests/test_providers/test_deepseek.py::TestDeepSeekPricingModule::test_registry_staleness_and_source_url` asserted: ```python assert not registry.is_stale() ``` `PricingRegistry.is_stale()` returns `(date.today() - last_updated) > timedelta(days=30)`. The DeepSeek registry ships `LAST_UPDATED = date(2026, 6, 19)`, so this assertion holds only while the current date stays within 30 days of that constant. Once it lapses, the test fails on time alone, turning the `test` shard red for every unrelated PR in the repo. It is failing right now (31 days past `LAST_UPDATED`). This is not testing code behavior: it only checks that the machine's clock is within 30 days of a hardcoded date. The sibling Anthropic and OpenAI registries are 560 days old and make no such assertion, so DeepSeek is the odd one out here rather than a deliberate freshness gate. ## Fix Drop the freshness assertion and keep the meaningful `source_url` check, renaming the test to `test_registry_source_url` to match what it now verifies. The staleness mechanism stays fully and time-independently covered by `tests/test_pricing.py::test_registry_staleness_and_warning`, which builds registries with `date.today() - timedelta(days=30)` (asserts not stale) and `date.today() - timedelta(days=31)` (asserts stale) plus the warning text. So this removes a fragile environmental assertion without reducing real coverage, and aligns DeepSeek with the Anthropic/OpenAI registries. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `tests/test_providers/test_deepseek.py`: remove the wall-clock-dependent `assert not registry.is_stale()`, keep the `source_url` assertion, rename the test to `test_registry_source_url`, and add a comment explaining why freshness is not asserted here. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uvx ruff@0.15.17 check tests/test_providers/test_deepseek.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, project venv (`uv sync --extra proxy`), `uvx ruff@0.15.17`. - Exact command / steps: with the current date at 31 days past `LAST_UPDATED`, ran the registry's `is_stale()` and the fixed test body against the real modules, plus the mechanism test from `tests/test_pricing.py`. - Observed result: `get_deepseek_registry().is_stale()` is `True` on the current date (which is exactly what broke the old assertion); the fixed `test_registry_source_url` body passes regardless of the date; and `test_registry_staleness_and_warning` still passes, so the staleness mechanism remains covered. - Not tested: a live DeepSeek pricing fetch (out of scope; pricing values are unchanged). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable |
||
|
|
0c7087539d |
fix: add DeepSeek V4/V3.2/R1 tokenizer mappings and context limits (#912)
The tokenizer registry routed deepseek-v4-pro, deepseek-v4-flash,
deepseek-chat, deepseek-reasoner, and other modern DeepSeek models
to the 2023-era deepseek-llm-7b-base tokenizer via prefix fallback.
This caused token counts off by 30-50%, broken context-limit detection
(V4-Pro supports 1M but got 32K), and inaccurate savings reports.
## Fix
3 files, +43/-2:
- **huggingface.py**: 16 new MODEL_TO_TOKENIZER entries with verified
HuggingFace IDs (deepseek-ai/DeepSeek-V4-Pro, V4-Flash, V3.2,
V3-0324, R1, R1-0528, Reasoner, Chat, Coder-V2, etc.)
- **openai_compatible.py**: 17 new _DEFAULT_CONTEXT_LIMITS entries
(V4-Pro/Flash -> 1M, R1/Reasoner -> 131K, V3 -> 128K, etc.)
- **openai.py**: 8 new _CONTEXT_LIMITS entries for LiteLLM-fallback.
Existing mappings untouched (backward compatible).
## Real behavior proof
- **Setup**: Windows 11, Python 3.13.14, headroom-ai 0.2.15 wheel +
source checkout at v0.24.0. No Rust extension built (headroom._core
unavailable). Touched files are at parity with v0.24.0.
- **Steps after patch**:
```
python3 -c "
from headroom.tokenizers.huggingface import get_tokenizer_name
for m in
['deepseek-v4-pro','deepseek-chat','deepseek-reasoner','deepseek-v4-flash']:
print(f'{m} -> {get_tokenizer_name(m)}')
from headroom.tokenizers.registry import get_tokenizer
for m in ['deepseek-v4-pro','deepseek-chat','deepseek-reasoner']:
print(f'{m}: {get_tokenizer(m)}')
"
```
- **Observed result**:
```
deepseek-v4-pro -> deepseek-ai/DeepSeek-V4-Pro
deepseek-v4-flash -> deepseek-ai/DeepSeek-V4-Flash
deepseek-chat -> deepseek-ai/DeepSeek-V3
deepseek-reasoner -> deepseek-ai/DeepSeek-R1
```
Previously ALL resolved to deepseek-ai/deepseek-llm-7b-base.
TokenizerRegistry routes correctly. Context limits verified
(1M / 131K / 128K). compress() import smoke-tested OK.
- **Not tested**: full proxy e2e with a live DeepSeek API key
(no available key). HuggingFace AutoTokenizer download confirmed
for V4-Pro/V3/R1 but produced GBK decode errors from hf_hub on
this zh-CN Windows locale during config fetch -- a separate
huggingface_hub issue unrelated to this change.
<!-- headroom-maintainer-template-completion:start -->
## Description
This PR prepares `fix: add DeepSeek V4/V3.2/R1 tokenizer mappings and
context limits` for review by documenting the intended change,
validation evidence, and remaining merge-readiness context.
Linked issues: None declared.
## Type of Change
- [x] Bug fix
- [ ] New feature
- [ ] Documentation
- [ ] Refactor
- [ ] Tests only
## Changes Made
- Commit: fix: add DeepSeek V4/V3.2/R1 tokenizer mappings and context
limits
- Touches `headroom/providers/openai.py`
- Touches `headroom/providers/openai_compatible.py`
- Touches `headroom/tokenizers/huggingface.py`
## Testing
- [x] GitHub checks reviewed
- [x] Metadata/template validation
- [ ] Local functional testing
### Test Output
```text
gh pr view 912 --repo chopratejas/headroom --json statusCheckRollup
- PR Governance / label: SUCCESS
- external / GitGuardian Security Checks: SUCCESS
```
## Real Behavior Proof
- Environment: GitHub PR metadata and checks for `chopratejas/headroom`
PR #912.
- Exact command / steps: Reviewed PR title, commits, changed files,
linked issues, labels, and check rollup; appended this maintainer
template completion block without replacing the author's original
description.
- Observed result: PR body now contains all required governance
sections, checked readiness fields, and a non-placeholder validation
evidence block.
- Not tested: This pass updated PR metadata only; code validation
remains represented by the linked GitHub checks and any author-provided
evidence above.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
<!-- headroom-maintainer-template-completion:end -->
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
e22d7453d4 |
fix(proxy): strip 1m model suffix before upstream forwarding (#1840)
## Description Strips dangling terminal-style model suffixes like `[1m]` from Anthropic-compatible model ids before Headroom forwards `/v1/messages` upstream. Closes #1812 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Generalized `sanitize_anthropic_model_id()` so the existing dangling ANSI-style suffix cleanup applies to Anthropic-compatible non-Claude models, including `glm-5.2[1m]`. - Added a provider-level regression for `glm-5.2[1m] -> glm-5.2`. - Added a `/v1/messages` handler regression that captures the upstream request body and verifies Headroom forwards `glm-5.2`, not `glm-5.2[1m]`. ## Testing - [x] Unit tests pass (`pytest`) — focused local tests and full CI test matrix passed - [x] Linting passes (`ruff check .`) — local Ruff and CI lint passed - [x] Type checking passes (`mypy headroom`) — local mypy and CI lint passed - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ rtk proxy env HEADROOM_REQUIRE_RUST_CORE=false PYTHONPATH=/Users/vinaygupta/Desktop/git/headroom-fix-1812-1m-model-suffix /tmp/headroom-1812-testenv/bin/python -c '<inject local headroom._core test stub; pytest.main(["tests/test_providers/test_anthropic.py", "tests/test_proxy_anthropic_model_sanitization.py"])>' ============================= test session starts ============================== platform darwin -- Python 3.13.11, pytest-9.1.1, pluggy-1.6.0 -- /private/tmp/headroom-1812-testenv/bin/python collected 17 items tests/test_providers/test_anthropic.py::TestAnthropicModelSanitization::test_sanitize_model_id_removes_ansi_escape_sequences PASSED tests/test_providers/test_anthropic.py::TestAnthropicModelSanitization::test_sanitize_model_id_removes_displayed_style_suffix PASSED tests/test_providers/test_anthropic.py::TestAnthropicModelSanitization::test_sanitize_model_metadata_cleans_nested_model_ids PASSED tests/test_providers/test_anthropic.py::TestAnthropicTokenCounting::test_count_text_fallback PASSED tests/test_providers/test_anthropic.py::TestAnthropicTokenCounting::test_count_messages_basic PASSED tests/test_providers/test_anthropic.py::TestAnthropicTokenCounting::test_count_text_allows_literal_special_tokens PASSED tests/test_providers/test_anthropic.py::TestAnthropicModelLimits::test_get_context_limit_claude_sonnet PASSED tests/test_providers/test_anthropic.py::TestAnthropicModelLimits::test_get_context_limit_claude_opus PASSED tests/test_providers/test_anthropic.py::TestAnthropicModelLimits::test_get_context_limit_strips_ansi_model_suffix PASSED tests/test_providers/test_anthropic.py::TestAnthropicModelLimits::test_get_context_limit_claude_5_family PASSED tests/test_providers/test_anthropic.py::TestAnthropicModelLimits::test_supports_model_known PASSED tests/test_providers/test_anthropic.py::TestAnthropicModelLimits::test_supports_model_prefix PASSED tests/test_providers/test_anthropic.py::TestAnthropicModelLimits::test_token_counter_cache_uses_sanitized_model_id PASSED tests/test_providers/test_anthropic.py::TestAnthropicCostEstimation::test_estimate_cost_basic PASSED tests/test_providers/test_anthropic.py::TestAnthropicCostEstimation::test_pricing_lookup_strips_ansi_model_suffix PASSED tests/test_providers/test_anthropic.py::TestAnthropicCostEstimation::test_pricing_claude_5_family PASSED tests/test_proxy_anthropic_model_sanitization.py::test_anthropic_messages_strips_local_1m_model_suffix_before_forwarding PASSED ======================== 17 passed, 3 warnings in 2.11s ======================== $ rtk uvx ruff check headroom/providers/anthropic.py tests/test_providers/test_anthropic.py tests/test_proxy_anthropic_model_sanitization.py All checks passed! $ rtk uvx ruff format --check headroom/providers/anthropic.py tests/test_providers/test_anthropic.py tests/test_proxy_anthropic_model_sanitization.py 3 files already formatted ``` The normal editable test command was attempted but did not reach test execution in this local checkout because the native extension build failed: ```text $ rtk uv run pytest tests/test_providers/test_anthropic.py tests/test_proxy_anthropic_model_sanitization.py × Failed to build `headroom-ai @ file:///Users/vinaygupta/Desktop/git/headroom-fix-1812-1m-model-suffix` warning: esaxx-rs@0.1.10: src/esaxx.cpp:620:10: fatal error: 'cstdint' file not found error: failed to run custom build command for `esaxx-rs v0.1.10` ``` ## Real Behavior Proof - Environment: local macOS worktree from current upstream `main`; Python 3.13.11 throwaway test environment; `HEADROOM_REQUIRE_RUST_CORE=false`; in-memory `headroom._core` stub used only to avoid the local missing native extension during Python-level tests. - Exact command / steps: POST a TestClient `/v1/messages` request with `{"model": "glm-5.2[1m]", ...}` and replace `_retry_request` with a test double that records the upstream body. - Observed result: the recorded upstream request body contains `{"model": "glm-5.2"}` and `mutation_reasons == ["sanitize_model_id"]`, so the mutated JSON body is serialized instead of forwarding the original bytes. - Not tested: live Z.AI credentials/provider call; full local pytest; local `mypy headroom`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes - All non-skipped GitHub Actions checks are green after the rebase onto `main`; skipped jobs are path-gated. - No code comments were added because the fix reuses the existing sanitizer and mutation-tracking path. - Documentation and CHANGELOG updates are N/A for this narrow proxy compatibility fix. - The local pytest warnings were from the throwaway environment/test tooling (`asyncio_mode`, Starlette TestClient deprecation, and the existing AnthropicProvider no-client warning), not from the changed code path. |
||
|
|
e84ca980cf |
feat(anthropic): add Claude 5 family pricing & align current rates (#1767)
## Summary Adds Claude 5 generation metadata and aligns the Anthropic fallback pricing / context-limit tables in `headroom/providers/anthropic.py` with current [Anthropic pricing](https://platform.claude.com/docs/en/about-claude/pricing) (verified 2026-07-04). Several Claude 4.x entries carried stale rates, the new Claude 5 models (Fable 5, Opus 4.8, Sonnet 5) had no metadata, and the generic fallback tests disagreed with the provider tests. Supersedes #1485 (rebased onto latest `main`, squashed to one commit, extended with Sonnet 5 / Fable 5 and the requested fallback-test fixes). ## Changes All values `$ / MTok`; `cached_input` = prompt-cache read = 0.1× input. | Tier | Model | Before | After | Context | |---|---|---|---|---| | Fable | `claude-fable-5` | — *(new)* | $10 / $50 / $1.00 | **1M** | | Opus | `claude-opus-4-8` | — *(new)* | $5 / $25 / $0.50 | **1M** | | Opus | `claude-opus-4-7` | $15 / $75 / $1.50 | $5 / $25 / $0.50 | 1M | | Opus | `claude-opus-4-6` | $15 / $75 / $1.50 | $5 / $25 / $0.50 | 1M | | Opus | `claude-opus-4-5-20251101` | $15 / $75 / $1.50 | $5 / $25 / $0.50 | 200K | | Sonnet | `claude-sonnet-5` | — *(new)* | $3 / $15 / $0.30 | **1M** | | Sonnet | `claude-sonnet-4-6` | — *(new)* | $3 / $15 / $0.30 | **1M** | | Sonnet | `claude-sonnet-4-5` | — *(new)* | $3 / $15 / $0.30 | 200K | | Haiku | `claude-haiku-4-5-20251001` | $0.80 / $4 / $0.08 *(3.5 rates)* | $1 / $5 / $0.10 | 200K | `claude-sonnet-4-20250514` and all Claude 3.x / 3.5.x entries were already correct — left unchanged. The **1M-context** entries (Fable 5, Opus 4.8, Sonnet 5, Sonnet 4.6) are functional, not cosmetic: they ship in the long-context tier, and without explicit entries the `sonnet` / `opus` pattern defaults would report 200K. Sonnet 5 is pinned to the **standard** Sonnet tier ($3 / $15 / $0.30); Anthropic's introductory rate ($2 / $10 through Aug 31 2026) is intentionally not encoded to avoid a time-dependent fixture. ## Fallback-model tests (addresses review on #1485) `_PATTERN_DEFAULTS["opus"]` is aligned to the current Opus tier ($5 / $25 / $0.50) so the generic fallback suite and the provider-specific suite agree: - `test_pricing_for_known_models` — Opus 4.5 pins $5 / $25 / $0.50 - `test_pattern_based_inference_opus` — unknown-opus fallback now $5 / $25 - `test_cost_estimation_for_new_models` — fixture estimate corrected $22.5 → $7.5 - `test_pattern_based_inference_sonnet` — retargeted to `claude-sonnet-6-*` (the old `claude-sonnet-5-*` probe now prefix-matches the real `claude-sonnet-5` key) New provider coverage: - `test_get_context_limit_claude_5_family` — Fable 5 / Opus 4.8 / Sonnet 5 all 1M - `test_pricing_claude_5_family` — exact rate table for the 3 new models Full suite: **48 passed**. ## Source https://platform.claude.com/docs/en/about-claude/pricing |
||
|
|
0e6d922f88 |
feat(pricing): add DeepSeek V4 model pricing (deepseek-v4-flash, deepseek-v4-pro) (#1168)
## Description Adds pricing support for DeepSeek V4 models (`deepseek-v4-flash` and `deepseek-v4-pro`) when routing Headroom through `--anthropic-api-url https://api.deepseek.com/anthropic`. The vendored LiteLLM pricing database predates DeepSeek V4, so cost estimation silently returned `None` for these models. ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Changes Made - **`headroom/pricing/deepseek_prices.py`** — New pricing data module with `ModelPricing` dataclass entries for both V4 models, following the pattern of `anthropic_prices.py` - **`headroom/pricing/__init__.py`** — Exports `DEEPSEEK_PRICES`, `get_deepseek_registry()`, `DEEPSEEK_LAST_UPDATED` - **`headroom/pricing/litellm_pricing.py`** — Runtime injection of DeepSeek V4 pricing into `litellm.model_cost`, plus `deepseek-` prefix added to `resolve_litellm_model()` provider prefix list - **`headroom/providers/anthropic.py`** — DeepSeek fallback in `_get_pricing()` when model starts with `deepseek-` and LiteLLM is unavailable - **`crates/headroom-proxy/data/model_prices_and_context_window.json`** — Vendored JSON entries (bare + provider-prefixed) for Rust-side context window lookups - **`tests/test_providers/test_deepseek.py`** — 20 tests across 3 test classes (pricing data, LiteLLM injection, Anthropic fallback) - **`tests/test_pricing.py`** — Added DeepSeek export validation alongside existing OpenAI/Anthropic assertions ## Testing - [x] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ``` ========================= 137 passed, 8 warnings in 8.47s ========================= ``` ## Real Behavior Proof - Environment: Windows 10, Python 3.12, litellm 1.60+ - Exact command / steps: `python -c "from headroom.proxy.cost import CostTracker; t = CostTracker(); print(t.estimate_cost('deepseek-v4-flash', input_tokens=1000000, output_tokens=1000000))"` - Observed result: `$0.4200` (0.14 input + 0.28 output per 1M tokens) - Not tested: Live DeepSeek API routing via `--anthropic-api-url` (requires API key and Docker deployment) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes The 90% cache discount heuristic in `AnthropicProvider.estimate_cost()` (line 680) is a pre-existing pattern. DeepSeek V4 has much deeper cache discounts (98-99%), but the LiteLLM path currently falls through to the manual fallback which uses correct cached prices. A future improvement could prefer `cache_read_input_token_cost` from model info over the hardcoded `* 0.1` heuristic. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
d480c464e9 |
fix(tokenizers): treat literal special-token strings as plain text (#1244)
## Description
`tiktoken`'s `Encoding.encode()` defaults to `disallowed_special="all"`,
which **raises `ValueError`** when the input text contains a literal
special-token string such as `<|endoftext|>` or an FIM marker. Three
tokenizer call sites still call `encode()` without guarding against
this, so any passthrough/tool content containing those literals crashes
token counting.
In the proxy this aborts compression of `/v1/responses` requests. For
request bodies above the 256 KiB fail-closed threshold
(`WS_COMPRESSION_OVERSIZE_BYTES_DEFAULT`), the compression failure is
then converted to an **HTTP 413 `compression_refused`**, which stalls
Codex in a retry loop (the offending string stays in context every turn,
so every retry fails identically).
Observed in production with the token-mode proxy in front of Codex:
```text
WARNING /v1/responses compression failed (bytes=588269):
ValueError: Encountered text corresponding to disallowed special token '<|endoftext|>'.
ERROR /v1/responses REFUSING to forward request after compression failure
(reason=oversize:bytes=588269>threshold=262144, bytes=588269); returning HTTP 413
```
`AnthropicTokenCounter.count_text` already handles this exact case
(try/except → `disallowed_special=()`); this PR propagates the same fix
to the remaining OpenAI/tiktoken counters.
Closes # <!-- no issue filed; happy to open one if preferred -->
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/providers/openai.py` — `OpenAITokenCounter.count_text`: fall
back to `disallowed_special=()` on `ValueError`.
- `headroom/tokenizers/tiktoken_counter.py` — same fallback in
`TiktokenCounter.count_text` **and** `TiktokenCounter.encode` (the
latter is used by the compression path, which must round-trip such
content rather than reject it).
- Each fallback mirrors the existing `AnthropicTokenCounter.count_text`
idiom and comments.
- Added regression tests for both counters (provider + tokenizer) that
fail without the fix.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest -q tests/test_tokenizers.py tests/test_tokenizer.py \
tests/test_providers/test_openai.py tests/test_providers/test_anthropic.py
75 passed, 14 skipped, 2 warnings in 2.22s
$ pytest -q tests/test_proxy_count_tokens_integration.py \
tests/test_openai_responses_context_compaction.py \
tests/test_openai_codex_routing.py
23 passed, 20 skipped, 1 warning in 4.33s
$ ruff check <changed files> # All checks passed!
$ ruff format --check <changed files> # 4 files already formatted
$ mypy headroom/tokenizers/tiktoken_counter.py headroom/providers/openai.py
Success: no issues found in 2 source files
```
## Real Behavior Proof
- Environment: clean clone at `v0.26.0-41-g7c26a54d`, editable install
(`pip install -e ".[dev,proxy]"`), Python 3.14.
- Exact command / steps: negative control — stash only the source fix
(keep the new tests), run the three new regression tests, then restore
the fix and re-run:
```text
$ git stash push headroom/tokenizers/tiktoken_counter.py headroom/providers/openai.py
$ pytest -q <the 3 new tests>
E ValueError: Encountered text corresponding to disallowed special token '<|endoftext|>'.
3 failed in 0.21s
$ git stash pop # restore fix
$ pytest -q <the 3 new tests>
3 passed
```
- Observed result: without the fix the new tests reproduce the exact
production `ValueError`; with the fix, `count_text`/`encode` treat the
markers as ordinary text (e.g. `"x <|endoftext|> y"` → 16 tokens,
`decode(encode(text)) == text`).
- Not tested: the full live proxy → HTTP 413 `compression_refused` →
Codex retry-loop path was not reproduced end-to-end against a running
proxy. Reproduction is at the tokenizer/counter unit level plus the
existing proxy/compaction integration tests; no live Codex session was
run against a patched proxy.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
|
||
|
|
bcabc5cb11 |
fix(providers): update DeepSeek V3 context limit from 128K to 1M (#1038) (#1137)
## Description
Update `_DEFAULT_CONTEXT_LIMITS` so DeepSeek V3/V4 use their actual 1M
(1,048,576) context window instead of the outdated 128K. The hardcoded
128K causes Headroom to trigger compression far too early, defeating the
purpose of using a long-context model.
Closes #1038
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Update `deepseek` default from 32,768 to 1,048,576 (V3/V4 family
default)
- Update `deepseek-v3` from 128,000 to 1,048,576
- Update `deepseek-coder` from 16,384 to 128,000 (Coder V2+)
- Add `deepseek-v4` entry at 1,048,576
- `deepseek-v2` stays at 128,000 (unchanged)
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_providers/test_universal.py -v -x
37 passed, 3 skipped in 38.08s
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.11, headroom main (
|
||
|
|
0c5c89d05c |
fix(anthropic): strip styled Claude model ids (#651)
## Description Fixes #626 by normalizing Anthropic/Claude model ids that contain ANSI escape sequences or dangling style suffixes before provider lookups and upstream forwarding. The branch has been updated onto current `main` and the proxy handler conflicts have been resolved. ## Type of Change - [x] Bug fix - [ ] New feature - [ ] Documentation - [ ] Refactor - [x] Tests only ## Changes Made - Normalize Anthropic model ids before context/pricing lookup. - Sanitize Anthropic `/v1/models` metadata and styled `/v1/models/{id}` passthrough paths. - Sanitize `/v1/messages` request body model ids before upstream forwarding. - Resolved current-main conflicts while preserving newer `model_override` and streaming passthrough behavior. ## Testing - [x] Unit tests - [x] Route/proxy tests - [x] Lint/static checks - [ ] Manual testing ### Test Output ```text UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --with pytest --with fastapi --with httpx --with uvicorn --with h2 python -m pytest tests/test_providers/test_anthropic.py tests/test_provider_proxy_routes.py::test_anthropic_model_metadata_strips_ansi_model_ids tests/test_provider_proxy_routes.py::test_anthropic_model_detail_path_strips_ansi_model_id tests/test_provider_proxy_routes.py::test_anthropic_messages_strips_ansi_model_id_before_upstream -q 17 passed, 2 warnings in 39.91s UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --with ruff ruff check headroom/providers/anthropic.py headroom/proxy/handlers/openai.py headroom/proxy/handlers/anthropic.py tests/test_providers/test_anthropic.py tests/test_provider_proxy_routes.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13.3, focused local worktree for PR #651 after merging current `upstream/main`. - Exact command / steps: Merged current main, resolved conflicts in Anthropic/OpenAI proxy handlers, ran the PR's targeted provider/proxy tests and ruff checks. - Observed result: Styled Anthropic model metadata, model-detail path, and messages upstream sanitization tests pass; ruff reports no issues. - Not tested: Full repository mypy/pre-commit; existing unrelated Windows `fcntl` typing errors block full hook execution locally. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review <!-- headroom-maintainer-template-completion:start --> ## Description This PR prepares `fix(anthropic): strip styled Claude model ids` for review by documenting the intended change, validation evidence, and remaining merge-readiness context. Linked issues: #626 ## Type of Change - [x] Bug fix - [ ] New feature - [ ] Documentation - [ ] Refactor - [ ] Tests only ## Changes Made - Commit: fix(anthropic): normalize styled model ids - Commit: fix(proxy): strip styled Anthropic model ids - Commit: fix: format anthropic model sanitization - Commit: Merge remote-tracking branch 'upstream/main' into review/pr-651 - Touches `headroom/cache/dynamic_detector.py` - Touches `headroom/providers/anthropic.py` - Touches `headroom/proxy/handlers/anthropic.py` - Touches `headroom/proxy/handlers/openai.py` - Touches `tests/test_provider_proxy_routes.py` - Touches `tests/test_providers/test_anthropic.py` ## Testing - [x] GitHub checks reviewed - [x] Metadata/template validation - [ ] Local functional testing ### Test Output ```text gh pr view 651 --repo chopratejas/headroom --json statusCheckRollup - PR Governance / template: FAILURE - CI / changes: SUCCESS - Init E2E / docker-init-e2e: SUCCESS - Wrap E2E / docker-wrap-e2e: SUCCESS - Wrap Native E2E / wrap-native (ubuntu-latest): SUCCESS - Wrap Native E2E / wrap-native (macos-latest): SUCCESS - CI / commitlint: SUCCESS - PR Governance / label: SUCCESS - CI / lint: SUCCESS - CI / build-wheel: SUCCESS - CI / prefetch-model: SUCCESS - CI / build: SUCCESS ``` ## Real Behavior Proof - Environment: GitHub PR metadata and checks for `chopratejas/headroom` PR #651. - Exact command / steps: Reviewed PR title, commits, changed files, linked issues, labels, and check rollup; appended this maintainer template completion block without replacing the author's original description. - Observed result: PR body now contains all required governance sections, checked readiness fields, and a non-placeholder validation evidence block. - Not tested: This pass updated PR metadata only; code validation remains represented by the linked GitHub checks and any author-provided evidence above. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review <!-- headroom-maintainer-template-completion:end --> --------- Co-authored-by: Tejas Chopra <chopratejas@gmail.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
8310a495ba |
style: match CI ruff formatting
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
|
4576f9caba |
test: remove provider diff churn
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
|
7831620eca |
test: expand provider slice coverage
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
|
8e4d7759de | Harden cache validation reporting and TTL analysis | ||
|
|
7a34030b0d |
Fix mypy type errors in LiteLLM and OpenAI providers
- Add type: ignore[assignment] comments for optional litellm imports - Add None checks before accessing optional module functions - Handle nullable max_input_tokens and max_output_tokens values Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
|
905c229251 |
Add AST-based code compression and custom model configuration
CodeAwareCompressor: - Tree-sitter based AST parsing for Python, JS, TS, Go, Rust, Java, C, C++ - Preserves imports, signatures, type annotations, error handlers - Guarantees syntactically valid output - Uses tree-sitter-language-pack for broad language support ContentRouter: - Intelligent compression orchestrator - Auto-routes content to optimal compressor based on type detection - Source hint support for high-confidence routing Custom Model Configuration: - HEADROOM_MODEL_LIMITS env var and ~/.headroom/models.json support - Pattern-based inference for unknown models (opus/sonnet/haiku tiers) - Support for Claude 4.5, Claude 4, o3, o3-mini - Graceful fallback - never crashes on unknown models |
||
|
|
e4a41faa33 |
Fix all ruff lint and format errors for CI
- Fix E402: Move module-level imports to top of file - Fix F401: Add noqa for availability check imports - Fix F402: Rename loop variables shadowing imports - Fix E722: Replace bare except with except Exception - Fix B904: Add exception chaining (from e) - Fix F811: Remove duplicate imports - Fix B027: Add noqa for empty close() method - Fix E741: Rename ambiguous variable l -> label - Fix I001: Import sorting issues - Apply ruff format to all 106 files All 902 tests pass. |
||
|
|
175746cc26 |
Prepare for OSS release v0.2.0
This commit prepares Headroom for public open source release with comprehensive documentation, licensing, and community infrastructure. License & Legal: - Add Apache 2.0 LICENSE file - Add NOTICE file with third-party attributions - Add SECURITY.md for vulnerability reporting Community: - Add CONTRIBUTING.md with contribution guidelines - Add CODE_OF_CONDUCT.md (Contributor Covenant) - Add GitHub issue templates (bug report, feature request) - Add pull request template Documentation: - Update README.md with compelling value proposition - Add docs/getting-started.md - Add docs/proxy.md for proxy server documentation - Add docs/transforms.md for transform reference - Add docs/api.md for API reference - Add examples/README.md Package Infrastructure: - Add headroom/py.typed for PEP 561 compliance - Add headroom/cli.py for CLI entry point - Add .github/workflows/ci.yml for CI pipeline - Add .github/workflows/publish.yml for PyPI publishing - Update pyproject.toml with proper metadata New Features: - Add multi-provider support (Google, Cohere, LiteLLM, OpenAI-compatible) - Add universal tokenizer registry with multiple backends - Add model registry with pricing and context limits - Add production proxy server with caching and rate limiting Code Quality: - Fix 83 lint issues via ruff auto-fix - Fix version consistency (benchmarks 0.1.0 → 0.2.0) - Add skip decorators for optional dependency tests |
||
|
|
9c7d4512d6 |
Initial commit: Headroom SDK - LLM context optimization toolkit
A comprehensive SDK for optimizing LLM context windows, reducing token usage while preserving critical information for AI agents. Core Features: - SmartCrusher: Statistical compression of tool outputs (70-85% reduction) - CacheAligner: Prefix optimization for prompt cache hits - RollingWindow: Intelligent context window management - BM25/Hybrid relevance scoring for smart item selection Integrations: - OpenAI and Anthropic provider support - LangChain integration (ChatModel, Callbacks, Runnable) - MCP (Model Context Protocol) integration for tool compression Test Coverage: - 372 tests passing across all modules - 35 performance benchmarks - Real-world agent evaluations with 88% token savings Key Components: - headroom/transforms/: Core compression transforms - headroom/providers/: OpenAI and Anthropic support - headroom/integrations/: LangChain and MCP integrations - headroom/relevance/: BM25 and hybrid scoring - headroom/pricing/: Model pricing registry - benchmarks/: Performance benchmark suite - examples/: Usage examples and demos |