Commit Graph
33 Commits
Author SHA1 Message Date
d21f74e396 fix(providers): count Anthropic tool_result/tool_use content blocks in OpenAI token counter (#2681)
## Description

The shared production `count_content_blocks` walker already counts
nested Anthropic tool-result/tool-use blocks on current main. This PR
adds direct OpenAI-provider regression coverage for that existing fix:
payload-sensitive counts, bounded nested media estimates, and the two
sample mapping orders. It makes no production counter change. Related
historical reports: #987, #828 and #813.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Add direct OpenAI regressions for tool-result/tool-use payload counts
and a lower count for a shorter tool-result fixture.
- Verify a nested 1 MB base64 image receives a bounded estimate rather
than text-tokenizing its bytes.
- Retain the sample mapping-order assertion with its scope clarified:
two sample maps have equal counts; this does not promise general
canonical key-order invariance. Main serializes the supplied mapping as
JSON.
- The production walker and both provider counters are unchanged by this
PR.

## Testing

- [x] Focused provider/shared-walker tests: 76 passed, one deprecation
warning.
- [x] Ruff lint and formatting pass on the changed test file.
- [x] Main-relative whitespace checks pass.
- [x] Independent fixture-count probe: 2809 tokens for the large
fixture, 26 for the short fixture.
- [x] Full hosted qualification passed on exact refreshed
head94c44fbe28065c7f31ef97ac2dcd51352d691dac (CI42232683655 and terminal
hosted rollup, no failures).

No old-main versus new-production-fix comparison is claimed; the
production fix is already on main.

## Real Behavior Proof

- Fresh maintainer environment: Windows, Python 3.12, tiktoken counter
for gpt-4o; current-main native DLL used for package startup, not
rebuilt here.
- Exact counter reproduction (run in the configured project
environment):

```python
from headroom.providers.openai import OpenAIProvider
c = OpenAIProvider().get_token_counter("gpt-4o")
big = "row data: " + ", ".join(f"item_{i}=value_{i}" for i in range(400))
small = "row data: item_0=value_0 [398 items compressed. hash=abc123]"
def message(text):
    return [{"role": "user", "content": [{"type": "tool_result", "content": text}]}]
print(c.count_messages(message(big)), c.count_messages(message(small)))
```

- Observed: `2809 26`. These are counts for two supplied fixtures, not a
new compression run or an old-main/new-PR comparison; current main
already counts the blocks correctly.
- Focused verification: `pytest -q tests/test_providers/test_openai.py
tests/test_provider_counter_content_blocks.py` — 76 passed, one
deprecation warning.
- Not tested: full live `/v1/compress` HTTP flow, external provider
inference or a fresh branch-specific native build.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A

## Additional Notes

- The production fix for structured nested content landed on `main` via
the shared `count_content_blocks` walker; this branch adds the direct
OpenAI regressions on top of it.
- CHANGELOG is handled by Release Please for this repo, so that
checklist item is N/A.
````
````

Two things before you paste. First, check the boxes I left unchecked — I
marked mypy and "existing unit tests pass locally" as unchecked because
your own notes say you skipped the full suite, and I'd rather you not
assert something you didn't run. Tick them if you have since. Second, if
the template file in your checkout has drifted since #1839, diff against
`.github/PULL_REQUEST_TEMPLATE.md` before saving; the bot matches
headings literally.

Paste into the PR body, hit save, and the governance workflow should
re-run and swap the label.

If you'd rather I did the clicking, install the Claude for Chrome
extension and reconnect — then I can open the PR, fill the body, and
show you the diff before saving anything.
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2681"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer current-main requalification

Exact head `94c44fbe28065c7f31ef97ac2dcd51352d691dac`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Clean current-main integration with upstream
c01cf07d94. Production shared
content-block walker already exists on main; remaining branch scope is
direct OpenAI regression tests only. Re-read current discussion and all3
Devin notes.76 provider/shared-walker tests pass,1warning; changed-test
Ruff lint/format and whitespace pass. Maintainer wording-only test
rename/docstring limits mapping-order assertion to its actual two
fixtures; assertions and counter logic unchanged. Actual counter probe
reports2809 large-fixture tokens versus26 short-fixture tokens using
gpt-4o; this is fixture counting, not an HTTP compression or old/new
production comparison. PR body is corrected to test-only scope and
provides exact reproducible code; no general canonical key-order
guarantee. Existing current-main DLL supports startup only, no new
native build/provider call. Formal approval awaits exact-head hosted
qualification; maintainers handle CI. Nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-06 09:32:35 -06:00
7877241d84 feat(providers): add Hermes agent wrap support (#934)
## Description

Adds first-class Hermes Agent support to `headroom wrap` while
preserving the current provider architecture. The wrapper prepares both
OpenAI- and Anthropic-compatible environment variables, supports
argument forwarding and `--prepare-only`, and reports a clear
installation error when the Hermes binary is unavailable.

This update also changes the missing-binary test to assert the complete
CLI output, resolving CodeQL's URL-substring sanitization false positive
without weakening coverage.

## Changes Made

- integrate Hermes launch-environment construction into the existing
provider module
- route OpenAI and Anthropic endpoints through Headroom
- support Hermes argument forwarding and prepare-only setup
- cover missing-binary, launch, and compression/passthrough behavior
- use an exact expected error message in the CLI test for CodeQL
compatibility

**Provider contract:** `headroom wrap hermes` requires `--provider
anthropic` or `--provider openai`. Any other selection, or no flag,
exits with code 1 before the proxy or Hermes starts.

## Testing

- [x] Unit tests pass
- [x] Linting and formatting pass
- [x] New tests added

### Test Output

```text
20 focused Hermes tests passed
ruff check: passed
ruff format --check: passed
git diff --check: passed
```

- `uv run --extra dev pytest tests/test_providers/test_hermes.py
tests/test_cli/test_wrap_hermes.py
tests/test_hermes_passthrough_compression.py -q` (20 passed)
- `uv run --extra dev ruff check tests/test_cli/test_wrap_hermes.py`
- `uv run --extra dev ruff format --check
tests/test_cli/test_wrap_hermes.py`
- `git diff --check`

## UI evidence

Not applicable. This PR changes CLI/provider behavior and tests; it does
not alter a graphical UI.

## Checklist

- [x] Tests added or updated
- [x] Focused tests pass locally
- [x] Formatting and lint checks pass
- [x] No UI screenshots required
- [x] No changelog entry required for this focused provider integration


## Type of Change

- [ ] Bug fix
- [x] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring

## Real Behavior Proof

- Environment: current Hermes provider/wrapper test environment.
- Exact command / steps: ran the Hermes provider, CLI wrapper, and
passthrough/compression suites plus Ruff, formatting, and diff checks.
- Observed result: 20 focused tests pass; OpenAI and Anthropic endpoints
route through Headroom, arguments and prepare-only work, and missing
binaries produce the exact expected error.
- Not tested: a live Hermes Agent session against production providers.

## Runtime Rollout Safety

- Rollout-managed feature(s): None. This adds a new opt-in `headroom
wrap hermes` command.
- Minimum rollout channel: N/A
- Stable/default behavior changed: No. Other wrap commands and proxy
behavior do not change.
- Kill switch / disable path: Do not run `headroom wrap hermes`. Hermes
can also be started without the wrapper.
- Unsafe override required: No
- Qualification impact: None
- Rollback path: Revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/934"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->



### Maintainer current-main requalification

Exact head `ef4c2c11cd6359aaf4239f9cc394c28c23985c15`, upstream main
`2d3ee51846435ef5b320eea4806af14e602b0d8c`.

Clean mechanical synchronization with upstream main
2d3ee51846435ef5b320eea4806af14e602b0d8c; contributor routing policy
unchanged. Current author replies and Devin findings reviewed. The
previous default-routing blocker is addressed: the wrapper rejects
absent/unsupported provider flags before calling _launch_tool, and
permits explicitly selected anthropic/openai providers. The
launch-boundary regressions exercise both rejected/default and supported
paths. Explicit empty environments remain isolated, and the shared port
callback is retained.

Hermes/provider/port-alignment tests pass; changed-file Ruff lint/format
and main-relative whitespace checks pass. Existing native extension
supports Python startup, not a fresh native build. The author's
Hermes0.22.2 WSL rejected/default and explicit-provider startup evidence
is considered; this round does not claim live authenticated provider
traffic. Project attribution, retrieval-plugin setup and a possible
registry consolidation remain separate follow-ups rather than claims
that the wrapper installs the plugin or supports other providers.

No remaining finding in the previously requested provider-guard scope.
Formal approval awaits exact-head hosted qualification, handled by
maintainers. Author readiness state preserved; nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-05 21:11:37 -06:00
emme1tandJD Davis 51258ef14a fix(providers): isolate resolved Anthropic context limits (#4593)
## Description

Addresses the order-dependent Anthropic context-limit resolution
reported in #4390. The proxy reuses its provider instance: resolving
`gpt-4` first currently makes a later `gpt-4-turbo` lookup inherit 8,192
tokens through alias matching, although LiteLLM reports 128,000.

Keep dynamically resolved limits in an exact-model cache so each model
reaches its own metadata lookup. The broader client-supplied
context-limit/header proposal in #4390 remains a separate feature.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Separate resolved limits from the built-in/configured entries used for
release and alias matching.
- Preserve exact-result caching, configured aliases, `[1m]` handling,
and recognition of previously resolved models.
- Cover both lookup orders, both LiteLLM limit fields, unknown-model and
Claude fallbacks, date aliases, repeated lookups, and explicit
overrides.

## Testing

- [x] Relevant unit tests pass
- [x] Linting passes (`python -m ruff check .`)
- [x] Type checking passes for the Linux target (`python -m mypy
headroom --ignore-missing-imports --platform linux`)
- [x] New regression tests added
- [x] Real provider-runtime replay performed

### Test Output

```text
New regressions against fc1ba891: 14 failed, 3 passed
python -m pytest tests/test_providers tests/test_provider_model_fallback.py tests/test_provider_model_metadata.py tests/test_provider_registry.py tests/test_provider_registry_extended.py tests/test_proxy_anthropic_model_sanitization.py
304 passed

Makefile ci-precheck-python test selection, using the locally built Rust extension:
178 passed, 5 skipped

python -m ruff check .
All checks passed!
python -m ruff format . --check
1860 files already formatted
python -m mypy headroom --ignore-missing-imports --platform linux
Success: no issues found in 579 source files

cargo fmt --all -- --check: passed
cargo clippy --workspace -- -D warnings: passed
commitlint --from origin/main --to HEAD --config .commitlintrc.json: passed
```

Windows validation limits: native Windows mypy reports seven existing
POSIX-attribute errors (`os.fchmod`, `O_DIRECTORY`, `O_NOFOLLOW`,
`O_NONBLOCK`), with the same errors reproduced using the original
provider source. The local mypy hook was skipped after the equivalent
Linux-target check above passed. Other applicable pre-commit and
commit-message hooks passed.

`cargo test --workspace` compiled successfully with Rust 1.95.0,
`x86_64-pc-windows-gnu`, then the unchanged `headroom-core` test
executable exited with `0xc0000005` before running tests; invoking it
with `--list` produced the same failure. Rust tests therefore have no
local passing result. The Windows Make wrapper also fails to interpret
the shell recipe (`command was unexpected at this time`). Push uses the
hook's documented `--no-verify` option after the explicit checks above.
The Rust extension was built from this checkout, installed editable, and
import-verified with `DiffCompressor` and `SmartCrusher`.

## Real Behavior Proof

- Environment: Windows x64, Python 3.14.5, LiteLLM 1.101.3, default
`ProxyConfig`, locally built editable Headroom. Each sequence uses the
actual shared Anthropic provider from `build_proxy_provider_runtime` and
LiteLLM's installed model catalog.
- Exact steps, run with the checkout's Python:

```python
from headroom.providers.registry import build_proxy_provider_runtime
from headroom.proxy.models import ProxyConfig
import litellm

for order in [("gpt-4", "gpt-4-turbo"), ("gpt-4-turbo", "gpt-4")]:
    runtime = build_proxy_provider_runtime(ProxyConfig())
    provider = runtime.pipeline_provider("anthropic")
    assert runtime.pipeline_provider("anthropic") is provider
    actual = {model: provider.get_context_limit(model) for model in order}
    expected = {model: litellm.get_model_info(model)["max_input_tokens"] for model in order}
    print(order, actual, expected)
    assert actual == expected
```

- Before: short-ID-first returns `gpt-4=8192, gpt-4-turbo=8192`; reverse
order returns the correct values.
- After: both orders return `gpt-4=8192, gpt-4-turbo=128000`, matching
the real catalog.
- Not tested: live model-provider requests, OpenCode traffic, or
provider billing. The reproduction exercises the production
runtime/provider objects without mocks or online inference calls. The
complete Python suite was not run.

## Runtime Rollout Safety

- Rollout-managed features/channel: existing default provider
resolution; no new rollout flag.
- Stable/default behavior changed: resolved model IDs retain their own
context limits across successive requests.
- Disable/rollback: configure an exact context limit through the
existing configuration, or revert this commit.
- Unsafe override required: none.
- Qualification impact: provider-resolution regressions and runtime
replay above.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] Code follows the project's style guidelines
- [x] Added regressions that fail before the fix
- [x] Relevant new and existing Python tests pass locally
- [x] `CHANGELOG.md` is unchanged
- Documentation: existing configuration and public API remain
applicable.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4593"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-03 23:44:06 -06:00
703755e873 fix(pricing): add Claude Haiku 5.5 rates, 1M context and its 100K long-prompt card (#4037)
## Current maintainer qualification

Reviewed head 595803d786 after the
current-main sync. The PR-specific pricing implementation is unchanged
from the previous reviewed head.

Independent validation: 202 focused Python pricing/provider/savings
tests passed, four skipped. The Rust proxy binary built from this head.
Through the running Rust HTTP proxy and a separate loopback upstream,
requests carrying 100,000 and 100,001 input tokens produced the expected
native-ledger request costs of $0.0105 and $0.0525005 respectively,
including 1,000 output tokens. The default stable rollout channel was
used, with no unsafe feature override.

The HTTP replay uses controlled upstream usage; it does not claim live
Anthropic inference or independently verified provider billing. The
previously recorded missing native HTTP boundary check is now covered.

Current main is an ancestor. Hosted checks on this head are terminal: 49
successful, 2 skipped, with no failures. The prior shared Rust advisory
hold is cleared. Final merge approval remains human-only; nothing has
been merged by this review.
## Description

Claude Haiku 5.5 (`claude-haiku-5-5`) is released and had no rows in
Headroom's built-in Anthropic tables or the Rust proxy's vendored
LiteLLM snapshot. This PR adds it, including its second rate card for
prompts over 100K tokens, and makes every cost path read a model's
long-context threshold from its own catalog row instead of assuming
200K.

**Why the missing rows mispriced instead of failing:** without LiteLLM
data for the model, `AnthropicProvider` falls through to the `haiku`
pattern default ($0.80 / $4, 200K), which is 8x the real input rate and
a fifth of the real window. The Rust `context_window_for` is
exact-match, so the id fell through to the 128K default.

Haiku 5.5's rate card ([Anthropic
pricing](https://platform.claude.com/docs/en/about-claude/pricing),
matching upstream LiteLLM's `claude-haiku-5-5` rows): $0.10 / $0.50
input / output and $0.01 cache read for prompts up to 100K tokens; every
rate is 5x ($0.50 / $2.50 / $0.05) above 100K. 1M context, 128K max
output.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

All values in $ / MTok (input / output / cache read).

| Model | Before (fallback) | After | Context |
|---|---|---|---|
| `claude-haiku-5-5`, prompt <= 100K | *(no row: `haiku` tier default)*
$0.80 / $4 / $0.08 | **$0.10 / $0.50 / $0.01** | 200K -> **1M** |
| `claude-haiku-5-5`, prompt > 100K | $0.80 / $4 / $0.08 | **$0.50 /
$2.50 / $0.05** | |

- `headroom/providers/anthropic.py`:
- Adds Haiku 5.5 to `ANTHROPIC_PRICING` and `ANTHROPIC_CONTEXT_LIMITS`.
- The long-context premium becomes a small per-model table,
`_LONG_CONTEXT_TIERS` (prefixes, threshold, multipliers), so Haiku 5.5's
100K card sits next to the existing Sonnet 4 / 4.5 200K tier. The Sonnet
tier's threshold and multipliers are unchanged.
  ```diff
   _apply_long_context_premium(model, pricing, input_tokens)
- if input_tokens <= 200K or model not in Sonnet 4 / 4.5: return pricing
  -  return pricing * Sonnet premium
  +  for prefixes, threshold, premium in _LONG_CONTEXT_TIERS:
  +    if model starts with prefixes:
+ return pricing if input_tokens <= threshold else pricing * premium
  +  return pricing
  ```
- `crates/headroom-proxy/data/model_prices_and_context_window.json`:
appends the 18 upstream LiteLLM `claude-haiku-5-5` keys verbatim
(first-party, Bedrock regional/global/Mantle/GovCloud, Vertex, Azure,
OpenRouter, Perplexity). No existing entry changes or is removed;
extended rather than re-pulled for the reasons in #3841.
- **Per-model long-context tiers on the stats paths** (Devin finding 1).
Each catalog row publishes at most one `*_above_<N>k_tokens` tier (100K,
128K, 200K, 256K or 272K); the threshold now comes from the row, not a
shared 200K constant.
- `crates/headroom-proxy/src/observability/pricing.rs`: the price book
stores a `PriceCard` (base card + optional long-context card parsed from
the row's tier fields; a missing tier field falls back to that rate's
base price). `lookup` still returns the base card; `lookup_card` returns
both, and `PriceCard::for_prompt` picks one.
- `crates/headroom-proxy/src/observability/ledger.rs`: `record` picks
the card from the billed prompt (uncached + cache read + cache write)
before pricing spend and savings.
- `headroom/pricing/counterfactual.py`: `long_context_threshold(model)`
/ `long_context_tier(info)`; `resolve_rates` reads the row's own tier,
including its combined 1h + long-tier write rate where published.
`CacheMix.is_long_context` takes the threshold.
- `headroom/proxy/cost.py`, `headroom/proxy/savings_tracker.py`: bucket
and price at the model's own threshold.
- Side effect on other tiered rows, both intended: the Rust stats now
bill Sonnet 4 / 4.5 above 200K at its long card (the Python cost card
already did), and both paths switch GPT-5.x at 272K and the 128K-tier
rows at 128K with their own rates. Before, those rows crossed a 200K
line, found no `*_above_200k_tokens` fields and stayed on the base card.
- **Compression that crosses a tier credits the whole avoided premium**
(Devin re-review). When compression keeps a prompt under the threshold
its uncompressed form would have passed, the saving is the uncompressed
bill minus the actual bill, not the removed tokens at the cheap rate.
Rust: `PriceCard::compression_savings_usd` (input and output); Python:
`long_context_premium_avoided_usd`, used by
`estimate_request_savings_usd` and by `CostTracker` (which also adds the
output premium). The list companion still prices only the removed
tokens, so budget enforcement stays monotonic. The warm-prefix test in
`tests/test_cost_card_attribution.py` sent 200K with 10K saved, which
crosses Sonnet 4.5's tier; its counts are halved (same mix) so it stays
inside one tier.
- **Short ids no longer inherit a newer release** (Devin finding 2).
`get_context_limit` and `_get_pricing` matched substrings in both
directions, so `claude-haiku-5` resolved to Haiku 5.5 and
`claude-sonnet-4` to Sonnet 4.6's 1M window. `_is_release_of` accepts a
longer id containing the known one (dated or provider-wrapped) or an
undated / `-latest` alias of a dated key, and rejects ids one version
segment apart in either direction.
- `crates/headroom-proxy/src/compression/model_limits.rs`: the 1M-window
test now covers `claude-haiku-5-5`, `us.anthropic.claude-haiku-5-5` and
`vertex_ai/claude-haiku-5-5`.
- Rust tests: Haiku 5.5 base and >100K cards, per-model threshold
(Sonnet 4.5 at 200K), flat-rated model, missing tier field; ledger bills
150K in / 5K out at the long card, and cached tokens count toward the
threshold.
- `tests/test_providers/test_anthropic.py`: Haiku 5.5 price, dated-id
and 1M-context cases, cost cases at and above the 100K threshold, the
`claude-haiku-5` regression and an `_is_release_of` table.
- `tests/test_cost_card_attribution.py`,
`tests/test_savings_provider_matrix.py`: Haiku 5.5 cost card and savings
at 90K vs 150K.
- `tests/test_counterfactual_pricing.py`: the long-tier 1h test now
reads the published combined rate, and a synthetic row covers the
derived case.
- `tests/test_provider_model_fallback.py`: the "future haiku" probe
moves from `claude-haiku-5-20260101` to `claude-haiku-6-20270101`,
mirroring the opus probe change in #3841.
- `tests/test_thinking_compactor_version_gate.py`: `claude-haiku-5-5` is
treated as a Claude 5 model.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`) (ruff 0.16.8, on the changed
files)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ HOME=$(mktemp -d) pytest -q <119 pricing / cost / savings / stats / provider / learn modules>
2097 passed, 18 skipped, 5 warnings

$ cargo test -p headroom-proxy
577 passed, 0 failed (41 suites)
$ cargo clippy -p headroom-proxy --all-targets -- -D warnings
(clean)
$ cargo fmt -p headroom-proxy --check
(clean)

$ ruff check <changed .py files>          # ruff 0.16.8
All checks passed!
$ ruff format --check <changed .py files>
(clean)
$ mypy headroom --ignore-missing-imports  # mypy 1.20.2
Success: no issues found in 566 source files
```

`HOME` is isolated because a developer `~/.headroom/models.json` leaks
into `TestAnthropicConfigLoading`.

## Real Behavior Proof

- Environment: macOS, the repo `.venv`. `origin/main` and this PR head
were each checked out as a git worktree and run with the same
interpreter and an isolated `HOME`.
- Exact command / steps: a script built `AnthropicProvider()`, stubbed
`estimate_cost_from_tokens` and `_get_litellm_clients` to force the
built-in fallback (the LiteLLM-unavailable or offline path), then priced
two requests and read the context limit. For the Rust side it applied
`context_window_for` semantics (exact key, `mode == "chat"`,
`max_input_tokens`, 128K default) to each worktree's vendored JSON. For
the review fixes, a second script ran against the first PR commit
(`b0503796`) and this head: the fallback lookups for short ids, a
`CostTracker` 150K-in / 5K-out Haiku 5.5 turn with 10K saved, and
`resolve_rates(..., long_context=True)`; the Rust ledger case is the
`long_prompt_bills_at_the_models_long_context_card` test.
- Observed result: Haiku 5.5 now prices and sizes correctly on every
path, and short ids keep their tier default (tables below).
  ```text
origin/main this PR
context limit (Python fallback) 200,000 (inferred 'haiku' tier)
1,000,000
50K in / 5K out $0.060000 $0.007500
150K in (120K cached) / 5K out $0.053600 $0.033500 (100K+ card)
Rust context_window_for 128,000 1,000,000
  ```
Hand check of the last row: 30K × $0.50 + 120K × $0.05 + 5K × $2.50 per
MTok = $0.0335.

  Review fixes, first PR commit vs this head:
  ```text
b0503796 this head
get_context_limit("claude-haiku-5") 1,000,000 200,000 ('haiku' tier)
_get_pricing("claude-haiku-5") $0.10 / $0.50 / $0.01 $0.80 / $4 / $0.08
get_context_limit("claude-sonnet-4") 1,000,000 (Sonnet 4.6) 200,000
(Sonnet 4)
CostTracker 150K in / 5K out: output cost $0.002500 $0.012500
CostTracker 10K saved: cache-aware savings $0.001000 $0.005000
resolve_rates(haiku-5-5, long).uncached $0.10 / MTok $0.50 / MTok
Rust ledger 150K in / 5K out $0.0175 (base card, computed) $0.0875
(ledger unit test)
  ```

Compression crossing the tier, Haiku 5.5, 110K uncached compressed to
90K (unit tests on this head; the "before" column is the old rule's
arithmetic):
  ```text
removed tokens at base this head
Rust ledger compression savings $0.002 $0.046
estimate_request_savings_usd["compression"] $0.002 $0.046
CostTracker cache-aware savings (1K output) $0.002 $0.048 (incl. $0.002
output premium)
  ```
- Not tested: live provider API billing. The controlled native HTTP
replay now covers the 100K boundary, as documented above.
- The Rust proxy end to end with a live >100K Haiku 5.5 request; the
ledger is covered by unit tests only.
- Bedrock-style ids (`us.anthropic.claude-haiku-5-5`) get the base card
without the 100K premium on the Python fallback path, the same
limitation the Sonnet 4 / 4.5 tier already has.
  - No live API call was made.

## Maintainer follow-up qualification

Latest qualification head: `f885cc4c8afa574d867820583886687b0bae12a9`.
The pricing code and the 413-Python/73-Rust/TCP-replay validation below
are unchanged from `a45ae8c3e`. The first hosted shard-3 run exposed
five existing main integration failures:
`test_proxy_dependency_gate_matches_onnx_platform_markers` still patched
a removed eager-import hook. The follow-up aligns those existing tests
with lazy `find_spec` discovery and its missing-spec return value. The
five failures reproduce locally before; all eight dependency-contract
cases and 91 CLI/stateless/logging tests pass after, with
Ruff/format/whitespace passing. The current-head CI must finish before
approval.

Published maintainer follow-up at `a45ae8c3e` (normal push from the
unchanged `c326e2046` head): forward generated `output_tokens` into the
shared compression estimator through Prometheus and the durable lifetime
recorder, and add the long-card minus base-card output premium only when
compression crosses the model's threshold. `output_tokens_saved` remains
a separate output-shaping layer. The documented unpriced direct tracker
fallback is preserved.

Integrated current main `463c1a43e30fc33cb256a3f455daf28043e53b05` to
resolve the pricing conflict. The resolution retains canonical
model-specific cache billing, independent cache-rate provenance and
explicit-zero rates, while probing missing billed cache rates at the
model's own long-context threshold. New regressions cover both contracts
together.

Verification after the conflict resolution:

- 413 Python tests passed; one known Windows directory-fsync test
deselected after it failed identically on the unchanged PR head; three
existing warnings. This includes 20 new
output-premium/boundary/metadata/merged-cache-billing cases, pricing,
provider fallback, persistent metrics, cache TTL, tool-schema and output
savings suites.
- 73 Rust observability tests passed: `cargo test -p headroom-proxy
--lib observability --locked`.
- Ruff lint/format and whitespace checks pass. Focused Linux-platform
mypy passes for the four pricing/metrics modules. Full local mypy in
this extras-rich environment reports 19 errors in
`relevance/embedding.py`, `memory/backends/direct_mem0.py`,
`integrations/langchain/chat_model.py` and
`integrations/crewai/agents.py`; it is not claimed green.
- Before the fix, both public priced recorders report $0.046 for
110K→90K input with 1K generated output; after, both report $0.048.
Noncrossing and no-output controls pass, and separately saved output is
not conflated with the avoided premium.

Actual Windows/Python 3.12 TCP replay: real uvicorn proxy, separate
loopback HTTP upstream, and the real content-router/SmartCrusher
pipeline. A 304,903-character JSON tool result becomes 139,003
characters, saving 55,974 locally counted tokens. The controlled
upstream reports 90,000 input and 1,000 output tokens. On the unchanged
PR head the lifetime estimator reports $0.063987 and fails; with the fix
and after the main integration it reports the expected $0.065987. The
persistent Prometheus compression metric matches, and output-shaping
savings remain zero. This exercises real compression and handlers with
controlled usage/rates; no live provider billing or Rust HTTP runtime is
claimed.

Current-head GitHub CI and edited review bodies still require
qualification before approval.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: cost estimates and the context window
reported for `claude-haiku-5-5` and the 18 added LiteLLM keys;
long-prompt spend and savings on tiered models now use each model's own
threshold; short ids such as `claude-haiku-5` fall back to their tier
default instead of a newer release
- Kill switch / disable path: n/a (data tables)
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (n/a: no
user-facing docs list per-model prices)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

n/a

## Additional Notes

The 100K card applies on every path: the Python fallback
(`_LONG_CONTEXT_TIERS`), LiteLLM's own estimate, the cost card and
savings counterfactual, and the Rust stats ledger.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4037"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <tejas@headroomlabs.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 17:54:57 -05:00
Tejas ChopraandClaude Opus 5.5 8c1beb20ae fix(pricing): keep the long-context premium when litellm drops the 200k rate (#4109)
## Description

`litellm.model_cost` is downloaded live. It no longer carries
`input_cost_per_token_above_200k_tokens` on any bare Anthropic id. That
broke two things.

1. **Production pricing (bug).** `AnthropicProvider.estimate_cost`
prices through LiteLLM first, so a 300K-token Sonnet 4.5 request was
priced at the base tier: **$0.975 instead of $1.9125**. That is about
half the real cost of exactly the long `[1m]` sessions. Now, when a
request is above the 200K threshold for a tiered model and the model's
LiteLLM entry has no above-threshold rate, the provider prices it from
its own table, which still applies Anthropic's premium (input 2x, output
1.5x, cache 2x). Every other request still goes through LiteLLM
unchanged.
2. **CI on `main` and every open PR.** `tests/_pricing_models.py` could
find no candidate with the full field set, so
`tests/test_cost_card_attribution.py` and
`tests/test_savings_provider_matrix.py` errored at collection. That hid
failure 1, which surfaced as soon as collection worked. The fix adds the
one id that still has every field, the Bedrock Sonnet 4.5 entry
`claude-sonnet-4-5-20250929-v1:0` (same rates), as the last-resort
candidate. These tests only check that Headroom's arithmetic agrees with
LiteLLM's numbers.

Refs #4136

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/providers/anthropic.py`:
`_litellm_lacks_long_context_rate()`; `estimate_cost` skips LiteLLM for
a long-context request whose entry lacks the above-200K rate, and uses
the manual table with its premium.
- `tests/test_providers/test_anthropic.py`: a hermetic regression test
that strips the rate from the LiteLLM entry and asserts $1.9125; it
fails on main.
- `tests/_pricing_models.py`: add `claude-sonnet-4-5-20250929-v1:0` as
the last-resort candidate, with a comment explaining why.
- `tests/test_counterfactual_pricing.py`:
`test_long_context_derives_the_one_hour_rate_and_says_so` priced the
long-context tier of a model chosen for the base fields only. It now
asks `anthropic_pricing_model()` for a model that has that tier. It also
failed on `main`; the collection errors had been hiding it in CI.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest -q -p no:cacheprovider tests/test_cost_card_attribution.py tests/test_savings_provider_matrix.py tests/test_cost_budget_basis.py tests/test_cost_tracker_counterfactual.py tests/test_counterfactual_pricing.py tests/test_savings_ledger.py tests/test_models.py tests/test_tokenizers.py tests/test_realignment_live_multi_turn.py
190 passed, 22 skipped in 4.79s
(on origin/main: 2 collection errors, plus test_long_context_derives_the_one_hour_rate_and_says_so failing)
$ pytest -q tests/test_providers/test_anthropic.py -k LongContext
10 passed            (origin/main code: 3 failed, 7 passed: test_litellm_path[300000-...] x2 and the new hermetic test)
$ pytest -q tests/test_providers <all pricing/cost/savings tests>
629 passed, 14 skipped
$ uvx ruff@0.16.8 check / format --check (both files)
All checks passed! / 2 files already formatted
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, repo .venv with litellm 1.101.0 and
the live LiteLLM price map as of 2026-10-09. Clean worktrees of
`origin/main` (a9815bbb) and of this branch.
- Exact command / steps: `python -c "from headroom.providers.anthropic
import AnthropicProvider as A; p=A();
print([round(p.estimate_cost(i,5000,'claude-sonnet-4-5',c),4) for i,c in
((100000,0),(300000,0),(300000,150000))])"` in each worktree. Then the
branch's `TestLongContextPricing` run against each.
- Observed result: on `main` the output is `[0.375, 0.975, 0.57]`: both
300K requests are billed at the base tier, about half the real cost. On
this branch it is `[0.375, 1.9125, 1.1025]`, Anthropic's published
long-context price; the 100K request is unchanged.
`TestLongContextPricing` gives 3 failed / 7 passed against main's code
and 10 passed on the branch. An untiered model (Opus 4.5, 300K) still
prices flat at 1.625.
- Not tested: A real Anthropic invoice. The expected values are
Anthropic's published rates, which the manual table and LiteLLM's former
entry both carry.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A; cost reporting only, no request change.
- Stable/default behavior changed: Yes, only for Anthropic tiered models
above 200K input tokens whose LiteLLM entry lacks the long-context rate.
Those now report the premium cost instead of half of it.
- Kill switch / disable path: N/A.
- Unsafe override required: No.
- Qualification impact: Unblocks the Python test shards on every open
PR.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Follow-up: `headroom.pricing.counterfactual.resolve_rates(...,
long_context=True)` reads the same LiteLLM fields for its
cache-economics estimates and is not changed here.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 21:44:39 -07:00
Abhay Singh 0a2c80d520 perf(tokenizer): memoize OpenAI token counts like the Anthropic counter (#3168)
## Description

`OpenAITokenCounter.count_text` re-ran `tiktoken.encode` on **every**
call:

```python
def count_text(self, text: str) -> int:
    if not text:
        return 0
    try:
        return len(self._encoding.encode(text))
    except ValueError:
        return len(self._encoding.encode(text, disallowed_special=()))
```

`AnthropicTokenCounter.count_text` already memoizes counts via
`TokenCountCache` (and its own comment even says the OpenAI counter
"Matches AnthropicTokenCounter.count_text" — but it did not: it had no
cache). Both counters are **per-model singletons**, reused across
requests (`OpenAIProvider`/`AnthropicProvider` cache them in
`self._token_counters`). So for OpenAI-model traffic, a stable system
prompt / conversation prefix, or a repeated tool result, was re-encoded
from scratch on every turn — and token counting runs on the request hot
path (often more than once per request across pipeline stages).

This adds the same `TokenCountCache` to `OpenAITokenCounter`, mirroring
the Anthropic counter. `count_text` is a pure function of its text, so a
replayed count is value-identical. The cache only admits strings `>=
256` chars (its existing admission policy), so tiny/rare strings — which
encode in microseconds — pay nothing and never evict the large entries
the cache exists for.

Benchmark (`OpenAITokenCounter("gpt-4o")`, an ~8 KB block, mean of 2000
calls):

```
re-encode each call : 928.7 us/call
cache hit           :   0.1 us/call   (~11000x faster)
same count          : True
```

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/providers/openai.py`: `OpenAITokenCounter.__init__` now
creates a `TokenCountCache`; `count_text` checks/populates it and
delegates the actual encode to a new `_count_text_uncached` (which keeps
the existing `disallowed_special` fallback). Imported `TokenCountCache`.
- `tests/test_providers/test_openai.py`: added
`test_count_text_caches_large_strings` — a large string's count is
memoized and equals the uncached count; a short string stays below the
256-char admission floor.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
tests/test_providers/test_openai.py  ->  23 passed
uvx ruff@0.16.2 check headroom/providers/openai.py tests/test_providers/test_openai.py  ->  All checks passed!
uvx mypy@1.20.2 headroom/providers/openai.py  ->  Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
tiktoken installed, ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: benchmarked
`OpenAITokenCounter("gpt-4o").count_text` on an ~8 KB block, re-encoding
each call (928.7us) vs a cache hit (0.1us), confirming identical counts.
Removed the cache from `count_text` and ran `python -m pytest
tests/test_providers/test_openai.py -k caches_large_strings` -> failed
(`_count_cache.get(large)` was `None`); restored it -> 23 passed for the
file.
- Observed result: repeated counts of the same large text are served
from the cache instead of re-encoding, with the same value. Behavior for
uncached strings (including the `<|endoftext|>` literal-special-token
case) is unchanged.
- Not tested: an end-to-end proxy turn timing; the counter is exercised
directly, which is the object the pipeline resolves via
`OpenAIProvider.get_token_counter`.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is an internal memoization in
the OpenAI token counter, not a rollout-channel-gated runtime feature.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no. Counts are identical; only
repeated counts of the same large text are faster. The cache is
per-counter, GIL-atomic (no lock), and bounded (entries + total chars).
- Kill switch / disable path: N/A.
- Unsafe override required: no.
- Qualification impact: less CPU spent re-encoding stable context on the
token-counting hot path for OpenAI-model traffic; counts unchanged.
- Rollback path: revert this PR; `count_text` returns to encoding on
every call.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal behavior, counts unchanged)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

This just brings the OpenAI counter in line with the Anthropic one,
which already carries `TokenCountCache` for exactly this reason. The
shared cache class is unchanged, so its admission/eviction bounds and
thread-safety argument apply identically here.
2026-09-30 11:32:14 -05:00
Kenneth WongandJerrettDavis 0d99c56dc5 fix(pricing): add Claude Sonnet 5.5 / Opus 5.5 / Fable 5.1, correct Sonnet 5 rates (#3841)
## Description

Claude Sonnet 5.5 and Opus 5.5 are released, and Anthropic made Claude
Sonnet 5's launch price its standard rate. This PR updates Headroom's
built-in Anthropic fallback price and context tables and the Rust
proxy's vendored LiteLLM snapshot to match current [Anthropic
pricing](https://platform.claude.com/docs/en/about-claude/pricing)
(verified 2026-09-29).

From the pricing page, footnote 3:

> The $2/$10 per million input/output token pricing for Claude Sonnet 5,
announced at launch as introductory pricing through August 31, 2026, is
now the standard price. The previously scheduled increase to $3/$15 per
million input/output tokens on September 1, 2026 will not occur.

This reverses the deliberate $3/$15 pin from #1767, which assumed that
increase would happen.

**Why the missing rows mispriced instead of failing:**
`AnthropicProvider._get_pricing` falls back to substring matching in
dict insertion order, so `claude-sonnet-5-5` silently inherited the
`claude-sonnet-5` row and `claude-fable-5-1` inherited Fable 5's $1
cache read. The Rust `context_window_for` is exact-match, so every
Claude 5-generation id fell through to the 128K default.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

All values in $ / MTok (input / output / cache read).

| Model | Before (fallback) | After | Context |
|---|---|---|---|
| `claude-sonnet-5-5` | *(no row: matched `claude-sonnet-5`)* $3 / $15 /
$0.30 | **$2 / $10 / $0.20** | 1M |
| `claude-sonnet-5` | $3 / $15 / $0.30 | **$2 / $10 / $0.20** | 1M |
| `claude-opus-5-5` | *(no row: opus tier default)* $5 / $25 / $0.50 |
**$4 / $20 / $0.20** | 1M |
| `claude-opus-5` | *(no row: opus tier default)* | $5 / $25 / $0.50 |
1M |
| `claude-fable-5-1` | *(no row: matched `claude-fable-5`)* $10 / $50 /
$1.00 | **$10 / $50 / $0.25** | 1M |

- `headroom/providers/anthropic.py`:
- Adds the four new models to `ANTHROPIC_PRICING` and
`ANTHROPIC_CONTEXT_LIMITS` and corrects `claude-sonnet-5`.
- Newer ids are listed before their prefixes so dated or suffixed ids
(`…-5-5-20261001`, `…-5-5[1m]`) resolve to their own row.
- The `_PATTERN_DEFAULTS` tier rates are unchanged: they also cover
older unlisted models such as Sonnet 3.7.
- `crates/headroom-proxy/data/model_prices_and_context_window.json`:
appends the 113 missing upstream Claude 5-generation keys (first-party,
Bedrock regional/global/Mantle, Vertex, Azure, Databricks, OpenRouter
and similar). No existing entry changes or is removed. The snapshot is
extended rather than re-pulled with `scripts/refresh_model_limits.sh`:
- Upstream has since pruned 258 retired ids the proxy still serves, e.g.
`claude-3-7-sonnet-20250219` and `claude-opus-4-1`. Those would regress
from 200K to the 128K default.
- A full re-pull overwrites the hand-edited DeepSeek rate card from
#3575.
- `crates/headroom-proxy/src/compression/model_limits.rs`: new
`claude_5_generation_models_present` test.
- `tests/test_providers/test_anthropic.py`:
  - The Sonnet 5 expectation moves to $2/$10/$0.20.
- New parametrized pricing and 1M-context tests for the new models,
including the prefix-collision cases.
- `tests/test_provider_model_fallback.py`: the "future opus" probe moves
from `claude-opus-5-20260101` to `claude-opus-6-20270101`. Opus 5 is now
a real 1M entry, so the old id no longer exercised the pattern fallback.
This mirrors the existing `claude-sonnet-6` probe.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`) (run on the changed module,
`headroom/providers/anthropic.py`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ cargo test -p headroom-proxy --lib compression::model_limits
test compression::model_limits::tests::claude_5_generation_models_present ... ok
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 271 filtered out

$ HOME=$(mktemp -d) pytest -q tests/test_providers/test_anthropic.py tests/test_provider_model_fallback.py \
    tests/test_cost_tracker_totals.py tests/test_pricing_from_litellm.py tests/test_pricing.py \
    tests/test_openai_pricing_resolution.py
138 passed, 2 warnings

$ ruff check headroom/providers/anthropic.py tests/test_providers/test_anthropic.py tests/test_provider_model_fallback.py
All checks passed!
$ ruff format --check (same files)
3 files already formatted
$ cargo fmt -p headroom-proxy -- --check
(clean)
$ mypy headroom/providers/anthropic.py
Success: no issues found in 1 source file
```

On the broader pricing, cost and savings subset, 773 pass. The same 12
failures and 3 collection errors occur on unmodified `main` in my
environment, all `ModuleNotFoundError` from optional dev deps (`respx`
and others) that aren't installed locally. `HOME` is isolated because a
developer `~/.headroom/models.json` leaks into
`TestAnthropicConfigLoading`.

## Real Behavior Proof

- Environment: macOS, the repo `.venv` (Python 3.10). `origin/main` and
this PR head were each checked out as a git worktree and run with the
same interpreter and an isolated `HOME`.
- Exact command / steps: priced the same request through the built-in
fallback on `origin/main` and on this PR head (steps below).
- A script built `AnthropicProvider()`, stubbed
`estimate_cost_from_tokens` to return `None` to force the built-in
fallback (the LiteLLM-unavailable or offline path), and priced 1M input
tokens (800K cached) plus 100K output tokens for each model.
- For the Rust side, `context_window_for` semantics (exact key, `mode ==
"chat"`, `max_input_tokens`, 128K default) were applied to each
worktree's vendored JSON.
- Observed result: `origin/main` bills Sonnet 5.5 at $3/$15 and gives
every Claude 5-generation id a 128K Rust window; this PR bills $2/$10
and gives them 1M.
  ```text
  origin/main
claude-sonnet-5-5 rates={'input': 3.0, 'output': 15.0, 'cached_input':
0.3} cost=$2.3400
claude-sonnet-5 rates={'input': 3.0, 'output': 15.0, 'cached_input':
0.3} cost=$2.3400
claude-opus-5-5 rates={'input': 5.0, 'output': 25.0, 'cached_input':
0.5} cost=$3.9000
claude-fable-5-1 rates={'input': 10.0, 'output': 50.0, 'cached_input':
1.0} cost=$7.8000
rust context_window_for: sonnet-5-5 / global.anthropic.claude-sonnet-5-5
/ opus-5-5 / fable-5-1 = 128000

  this PR
claude-sonnet-5-5 rates={'input': 2.0, 'output': 10.0, 'cached_input':
0.2} cost=$1.5600
claude-sonnet-5 rates={'input': 2.0, 'output': 10.0, 'cached_input':
0.2} cost=$1.5600
claude-opus-5-5 rates={'input': 4.0, 'output': 20.0, 'cached_input':
0.2} cost=$2.9600
claude-fable-5-1 rates={'input': 10.0, 'output': 50.0, 'cached_input':
0.25} cost=$7.2000
    rust context_window_for: all four = 1000000
  ```
The live LiteLLM `model_cost` already reports the "after" rates for all
of these, so the primary pricing path is unchanged. This fixes the
fallback path and the Rust proxy.
- Not tested: no live upstream Sonnet 5.5 request, no dashboard browser
check (details below).
  - No live upstream requests were sent against Claude Sonnet 5.5.
- The dashboard savings tile was not exercised in a browser. It consumes
the same `estimate_cost` path shown above.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is static pricing and context
metadata.
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, fallback cost estimates and the
Rust compression budget for Claude 5-generation ids only (details
below).
- Fallback cost estimates for the listed models drop to Anthropic's
published rates.
- The Rust proxy's compression budget for Claude 5-generation ids rises
from 128K to their real 1M window. The proxy now compresses those
requests later instead of over-trimming them.
- Kill switch / disable path: `HEADROOM_MODEL_LIMITS` or
`~/.headroom/models.json` can override any model's context limit or
pricing.
- Unsafe override required: no
- Qualification impact: none expected. Existing models' rows and
vendored entries are byte-identical.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (n/a: no
user-facing docs list model prices)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this) (my original commit did; @JerrettDavis removed it in 2cfd4f64,
thanks)

## Screenshots (if applicable)

n/a

## Additional Notes

Not in scope: the static `_build_bedrock_fallback_map` and
`_VERTEX_MODEL_MAP` in `headroom/backends/litellm.py` stop at Claude
4.6. They are routing maps, not pricing, and are only used when
inference-profile discovery fails. They need verified per-model Bedrock
and Vertex ids, so they should be a separate PR.

https://claude.ai/code/session_01E5Hryq8EfuB4LAnu81TtJz

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-09-29 11:15:38 -05:00
Abhay Singh cc6a07cd65 perf(providers): cache Google/Cohere token counters across requests (#3273)
## Description

`GoogleProvider.get_token_counter` and
`CohereProvider.get_token_counter` built a **fresh** counter on every
call:

```python
def get_token_counter(self, model: str) -> TokenCounter:
    ...
    return GeminiTokenCounter(model, client=self._client)   # new object every call
```

The Anthropic and OpenAI providers do **not** do this — they memoize
counters in a per-model dict (`self._token_counters`) and reuse them
across requests. That matters because each counter owns an
`EstimatingTokenCounter` carrying a `TokenCountCache` (the same
memoization added for the OpenAI counter in #3168). Providers are
long-lived singletons on the proxy hot path
(`ProxyProviderRuntime.pipeline_providers` is built once), so rebuilding
the counter every request threw that cache away every request — and a
stable prefix (system prompt + tools), which is identical turn to turn,
was re-tokenized from scratch on every request for Gemini and Cohere
deployments.

This adds the same `self._token_counters` memoization to both providers,
so the token-count cache (and any lazily-built genai model on the Gemini
counter) persists across requests, exactly like the Anthropic/OpenAI
providers.

Benchmark (`GoogleProvider`, ~11 KB stable prefix, mean of 2000
`get_token_counter(...).count_text(prefix)` calls):

```
fresh counter each call (old):    42.5 us/call
cached counter          (new):     0.4 us/call
speedup: 107x
```

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/providers/google.py`: `GoogleProvider.__init__` now
initializes `self._token_counters: dict[str, TokenCounter] = {}`;
`get_token_counter` caches the counter per model instead of constructing
a new `GeminiTokenCounter` each call.
- `headroom/providers/cohere.py`: same change for `CohereProvider` /
`CohereTokenCounter`.
- `tests/test_providers/test_google.py` (new): asserts the counter is
memoized per model (same instance returned; distinct models get distinct
counters) and that an unknown model still raises.
- `tests/test_providers/test_cohere.py`: added the same per-model
caching assertion.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
tests/test_providers/test_google.py tests/test_providers/test_cohere.py  ->  20 passed
uvx ruff@0.16.2 check headroom/providers/{google,cohere}.py tests/test_providers/test_{google,cohere}.py  ->  All checks passed!
uvx mypy@1.20.2 headroom/providers/google.py headroom/providers/cohere.py  ->  Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: stashed the two source edits and ran the two
new caching tests -> both FAILED (a second `get_token_counter` returned
a different instance); restored the edits -> 20 passed for the two
files. Benchmarked `GoogleProvider` on an ~11 KB stable prefix: fresh
counter per call 42.5us vs cached counter 0.4us (107x), same count.
- Observed result: `get_token_counter(model)` returns the same counter
instance across calls, so its `TokenCountCache` survives across requests
and the stable prefix is tokenized once instead of every turn. Token
counts are unchanged (same counter class, same estimation).
- Not tested: live Gemini/Cohere API `countTokens`/`tokenize` calls (no
API client configured in this environment); the estimation path and
counter identity are exercised.

## Runtime Rollout Safety

- Rollout-managed feature(s): none — no feature flag or rollout channel
involved.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no. Same counter class and same token
counts; only the counter's lifetime changes (reused instead of rebuilt).
- Kill switch / disable path: N/A (no config surface added).
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this commit; counters go back to being rebuilt
per call.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal behavior, counts unchanged)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

This brings the Google and Cohere providers in line with the Anthropic
and OpenAI providers, which already cache their counters in
`self._token_counters` for exactly this reason. No behavior change
beyond counter lifetime; the counters and the shared cache class are
otherwise untouched.
2026-09-25 10:25:53 -05:00
Rod Boev c81378c878 fix(grok): preserve xAI model context metadata (#3312) (#3315)
## Description

Wrapped Grok Shell receives xAI model metadata with `context_length`,
but Grok's custom-endpoint parser reads `context_window` and falls back
to 256K when that alias is absent. Normalize the successful xAI
model-list response at Headroom's provider metadata boundary, preserving
the upstream value and the `grok-4.6` model id. Closes #3312.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Translate a positive xAI `context_length` value to Grok's missing
`context_window` field for the resolved `api.x.ai` model list.
- Preserve model ids, original xAI fields, explicit context aliases,
response status, and non-xAI model metadata.
- Leave backend selection unchanged, so this fixes the 256K context
fallback without rerouting inference from ChatCompletions to Responses.
- Remove entity validators when rewriting model-list bodies, preventing
cached metadata from being validated against the unmodified upstream
body.

## Testing

- [x] Unit tests pass (`uv run pytest
tests/test_providers/test_grok_model_metadata.py
tests/test_provider_model_metadata.py
tests/test_provider_proxy_routes.py tests/test_provider_grok.py
tests/test_cli/test_wrap_grok.py -q`)
- [x] Linting passes (`uv run ruff check
headroom/providers/grok/model_metadata.py
headroom/providers/model_metadata.py headroom/proxy/handlers/openai.py
tests/test_providers/test_grok_model_metadata.py
tests/test_provider_model_metadata.py
tests/test_provider_proxy_routes.py`)
- [x] Type checking passes (`uv run mypy headroom`)
- [x] New tests added for new functionality when applicable
- [ ] Manual testing performed

### Test Output

```text
uv run pytest tests/test_providers/test_grok_model_metadata.py tests/test_provider_model_metadata.py tests/test_provider_proxy_routes.py tests/test_provider_grok.py tests/test_cli/test_wrap_grok.py -q
46 passed, 1 warning in 8.41s

uv run ruff check headroom/providers/grok/model_metadata.py headroom/providers/model_metadata.py headroom/proxy/handlers/openai.py tests/test_providers/test_grok_model_metadata.py tests/test_provider_model_metadata.py tests/test_provider_proxy_routes.py
All checks passed!

uv run ruff format headroom/providers/grok/model_metadata.py headroom/providers/model_metadata.py headroom/proxy/handlers/openai.py tests/test_providers/test_grok_model_metadata.py tests/test_provider_model_metadata.py tests/test_provider_proxy_routes.py --check
6 files already formatted

uv run mypy headroom
Success: no issues found in 529 source files
```

## Real Behavior Proof

- Environment: Windows, Headroom development build, focused route tests
with a local OpenAI-compatible response fixture.
- Exact command / steps: Run
`tests/test_provider_proxy_routes.py::test_issue_3312_grok_model_metadata`;
the fixture sends an xAI-shaped `/v1/models` entry containing `id:
grok-4.6` and `context_length: 500000` through the real provider route.
- Observed result: The patched route returns `id: grok-4.6`, preserves
`context_length: 500000`, and adds `context_window: 500000`.
- Not tested: live Grok Shell 1.0.5 session against `api.x.ai`

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable/default.
- Stable/default behavior changed: Yes, wrapped xAI model discovery now
exposes the upstream context window to Grok.
- Kill switch / disable path: Not applicable, revert the change to
disable it.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert the change; model-list responses return to the
previous metadata shape.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`, it is generated by
release-please from the Conventional Commit PR title.

## Screenshots (if applicable)

Not applicable for this provider metadata change.

## Additional Notes

The change is scoped to the xAI model-list schema boundary. There is no
runtime rollout switch; the default behavior changes only for wrapped
xAI model discovery, and reverting the change restores the previous
metadata response. Live Grok Shell 1.0.5 verification against `api.x.ai`
was not run; the available local binary is Grok 1.0.0.
2026-09-25 00:00:59 -05:00
b0dfa33478 fix(pricing): DeepSeek V4.1-Flash peak/off-peak rate card (#3575)
## Description

DeepSeek restructured its model line-up and price card, and Headroom's
DeepSeek data was last verified 2026-06-19: the rows still carried the
pre-restructure flat rates, `deepseek-flash` was unknown, and nothing
could express the vendor's peak/off-peak split.

Source (fetched and re-verified today):
https://api-docs.deepseek.com/quick_start/pricing

| | `deepseek-flash` (V4.1-Flash) | `deepseek-v4-pro` (V4-Pro-0813) |
|---|---|---|
| Context / max output | 1M / 384K | 1M / 384K |
| Input (cache hit) | $0.003 off-peak / $0.006 peak | $0.022 / $0.044 |
| Input (cache miss) | $0.15 / $0.30 | $0.66 / $1.32 |
| Output | $0.60 / $1.20 | $1.98 / $3.96 |
| Vision | yes | no |

`deepseek-v4-flash` and `deepseek-v4-flash-vision-exp` are retired ids
still accepted on the wire — served by V4.1-Flash and billed at the
Flash price. Peak is exactly 2x off-peak; peak hours are 01:00-04:00 and
06:00-10:00 UTC, Monday-Friday.

Context note: this surfaced while working on #3036, which routes
DeepSeek traffic through the proxy and hard-codes the `deepseek-v4-*`
ids in its routing tests. That branch is not touched here, and no issue
is closed by this PR — it is a rate-card and model-data refresh.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Added `headroom/pricing/deepseek_tiers.py` as the single source of
truth: the off-peak `(cache-hit, cache-miss, output)` table for
`deepseek-flash` and `deepseek-v4-pro`, peak derived as exactly 2x, the
retired-id aliases, the Beijing peak-window predicate with the weekend
rule gated on its effective instant, and the `rates_for(model,
now=None)` seam.
- Made two cost seams tier-aware: `estimate_cost_from_tokens(...,
now=None)` prices DeepSeek flash/pro from the tier selected by the
request instant (litellm's `model_cost` holds one flat rate per model
and cannot express a tier), and `PricingRegistry.estimate_cost(...,
now=None)` prices them through its existing body from a synthesized rate
row so there is one arithmetic body rather than two.
- Gave `AnthropicProvider.estimate_cost` the same `now` passthrough and
derived its DeepSeek fallback table from the tier module.
- Derived every flat table from the tier module so no DeepSeek rate
literal is left outside it: the `deepseek_prices` rows, the
`litellm.model_cost` injection (now overriding and merging upstream
rather than filling absent keys), and the Anthropic fallback table.
Moved `LAST_UPDATED` / `SOURCE_URL` onto the module that owns the rates
and re-exported them so existing import paths keep working.
- Updated the model data: `deepseek-flash` registered with vision and
both retired ids as aliases, provider context rows, the Hugging Face
tokenizer map, the vendored LiteLLM snapshot, and the OpenCode guide's
published rate card.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_providers tests/test_pricing.py tests/test_pricing_deepseek_tiers.py \
    tests/test_pricing_from_litellm.py tests/test_pricing_litellm.py \
    tests/test_pricing_litellm_model_resolution.py tests/test_models.py \
    tests/test_huggingface_tokenizer_timeout.py tests/test_tokenizer_count_offload.py \
    tests/test_cost_pricing_warning_dedup.py -q
291 passed, 3 warnings in 9.80s

$ uv run ruff check .
All checks passed!

$ uv run mypy headroom
headroom\integrations\langchain\chat_model.py:394: error: Function "RunnableBinding" could
always be true in boolean context  [truthy-function]
Found 1 error in 1 file (checked 532 source files)

$ cargo test -p headroom-proxy --lib compression::model_limits
running 6 tests
test compression::model_limits::tests::vendored_json_parses_at_runtime ... ok
test compression::model_limits::tests::unknown_model_returns_default ... ok
test compression::model_limits::tests::sample_spec_entry_is_excluded ... ok
test compression::model_limits::tests::current_claude_models_present ... ok
test compression::model_limits::tests::current_gpt_models_present ... ok
test compression::model_limits::tests::empty_or_garbage_string_does_not_panic ... ok
test result: ok. 6 passed; 0 failed

$ uv run pytest tests/test_pricing_deepseek_tiers.py -q   # with litellm hidden
47 passed

$ uv run mypy headroom/pricing/deepseek_tiers.py headroom/pricing/registry.py \
    headroom/pricing/litellm_pricing.py headroom/providers/anthropic.py
Success: no issues found in 4 source files
```

The `mypy headroom` box is unticked on purpose: the single error above
is pre-existing on `main` in
`headroom/integrations/langchain/chat_model.py:394`, a file this PR does
not touch (see Additional Notes).

## Real Behavior Proof

- Environment: Windows 10, Python 3.13 in the repo's `.venv`, `uv`;
litellm 1.88.1 installed (whose bundled cost map predates these ids, so
the injected rows are what price them).
- Exact command / steps: pinned instants Monday 2026-08-17 02:00 UTC
(10:00 Beijing = peak) and 12:00 UTC (20:00 Beijing = off-peak), then:
`uv run python -c "from datetime import datetime,timezone; from
headroom.pricing.litellm_pricing import estimate_cost_from_tokens as e;
P=datetime(2026,8,17,2,tzinfo=timezone.utc);
O=datetime(2026,8,17,12,tzinfo=timezone.utc);
print(e('deepseek-flash',10**6,10**6,now=O),
e('deepseek-flash',10**6,10**6,now=P),
e('deepseek-flash',10**6,0,cached_tokens=10**6,now=O),
e('deepseek-v4-flash-vision-exp',10**6,10**6,now=O))"`
- Observed result: `0.75` (off-peak flash), `1.5` (peak flash, exactly
2x), `0.003` (fully cached prompt billed once at the cache-hit rate),
`0.75` (retired id prices as Flash). Also verified: 9 mutants of the new
logic (peak multiplier, weekend gate, window start, alias resolution,
cached-token handling, registry branch, provider passthrough) were each
caught by the new tests, and `tests/test_pricing_deepseek_tiers.py`
still passes with litellm absent (47 passed).
- Not tested: the full test suite (hours on this box, and much of it
cannot build a `TestClient` here because `NO_PROXY` contains a bare
`::1`, so `httpx.URLPattern` raises `InvalidURL: Invalid port: ':'` — a
pre-existing environment fault, confirmed by stashing this branch and
re-running); CNY billing; the proxy budget guard and savings ledger,
which remain flat by design (see Additional Notes).

## Runtime Rollout Safety

- Rollout-managed feature(s): DeepSeek pricing and model data consumed
by the pricing modules, providers, model registry and the Rust proxy's
context limits. No feature flags are involved.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes — DeepSeek costs, model ids and
capabilities change; the litellm `model_cost` DeepSeek rows now override
upstream; and one context-limit fallback now resolves
longest-prefix-first. All are itemised in Additional Notes.
- Kill switch / disable path: revert this branch (no flag). Operators
can override context limits without a release via
`HEADROOM_MODEL_LIMITS` or `~/.headroom/models.json`, and provider
pricing via the existing custom pricing config.
- Unsafe override required: No.
- Qualification impact: None — no schema, wire-format or session-format
change; the vendored JSON keeps its upstream shape.
- Rollback path: revert the merge commit; the only persisted artifact
change is the vendored JSON, which reverts with it.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Screenshots (if applicable)

Not applicable — no UI change.

## Additional Notes

**Behaviour changes worth a reviewer's attention**

1. Flat figures are the off-peak tier by design; only the two cost seams
are tier-aware. The proxy budget guard (`proxy/cost.py`), the savings
ledger, `ModelRegistry.estimate_cost` and
`OpenAICompatibleProvider.estimate_cost` remain flat or unpriced, so a
peak-hour DeepSeek request can be under-estimated by up to 2x there.
Making those tier-aware needs the request instant plumbed to each call
site — deliberately a separate change.
2. The litellm DeepSeek rows now override upstream instead of filling
only absent keys: upstream ships `deepseek-flash` at the *peak* rate, so
fill-if-absent would have made the flat figure depend on the installed
litellm version. The injection also merges (`{**upstream, **ours}`) so
capability flags survive.
3. DeepSeek caps moved `384_000` -> `393_216` in the litellm-shaped
tables (upstream's 384x1024 encoding of the published 384K). `ModelInfo`
keeps the published `384_000`; the litellm-shaped tables are
authoritative wherever litellm is consulted.
4. `OpenAICompatibleProvider`'s context fallback now matches the longest
prefix first, mirroring the long-standing behaviour in `OpenAIProvider`.
This fixes `deepseek-v4-flash-vision-exp` (was 1_048_576 via the shorter
`deepseek-v4` key, now 1_000_000) and also changes the window for
unlisted non-DeepSeek variants, e.g. `llama-3.1-8b-instruct` 8192 ->
128000, `mistral-large-2407` 32768 -> 128000, `starcoder2-15b` 8192 ->
16384.
5. `PricingRegistry.estimate_cost` no longer raises for a DeepSeek
flash/pro id absent from `prices` — the tier table is authoritative for
those four ids. Unknown non-DeepSeek ids still raise, and a flat row for
a tiered id is deliberately ignored.
6. Weekend off-peak applies only from 2026-08-22T16:00Z, so a back-dated
or replayed weekend request before that instant keeps the weekday
windows rather than being silently repriced. `now` exists only on
`AnthropicProvider.estimate_cost`; every other provider reads the wall
clock for live traffic.
7. CNY billing is not modelled. The zh-cn page lists the same tiers in
CNY, but the two lists are not one FX rate apart; supporting CNY-billed
accounts needs a switch over two published lists, not a baked constant.

**N/A / notes on checklist items**

- `Type checking passes (mypy headroom)` is unticked for the
pre-existing error above. `git diff --name-only origin/main..HEAD`
contains no `integrations/langchain` path, so this PR does not introduce
or touch it; `mypy` is clean on all four changed modules.
- Out of scope: the stale `deepseek-v3` / `deepseek-coder` registry rows
and the old `deepseek-chat` / `deepseek-reasoner` / `deepseek-v3.2`
context rows (the vendor no longer lists them; pruning them is a
behavioural change of its own).
- The design and implementation plan for this change live in a separate
workspace, so the decisions they record are inlined above rather than
linked.

---------

Co-authored-by: Zenjibad <316719033+Zenjibad@users.noreply.github.com>
Co-authored-by: DSH Agent <agent@local>
2026-09-24 11:31:44 -05:00
Tejas ChopraandClaude Opus 5 89a58fd152 fix(pricing): price savings against the cache mix, not flat list price (#3699)
## The problem

Every savings figure Headroom reports is a counterfactual: *what would
this request have cost had we not removed those tokens?* We answered it
with `tokens_saved * input_cost_per_token` — flat list price — which is
true **exactly once**, for the first request of a cache window. After
that, the tokens we removed would have been served from the provider's
prompt cache at a fraction of list price (Anthropic 0.10x, OpenAI 0.50x,
Gemini 0.10x).

Measured on a real 238M-input-token corpus that was **87.8% cache
reads**:

| | |
|---|---|
| effective rate actually **paid** | $1.592/M |
| rate savings were **priced at** | $4.339/M — **2.73x** |
| reported compression savings | **$330.89** |
| same tokens at the observed mix | ~$121 |

For **tool-schema deferral** the gap is ~10x: tool definitions sit at
the very front of the prompt and are the most cacheable tokens in it.

## The fix

The correct rate depends on **where in the request** the tokens sat:

| Region | What | Pricing |
|---|---|---|
| `PREFIX` | tool schemas — ahead of every cache breakpoint | warm turn
= **entirely a cache read** (read-first, not pro-rata) |
| `LIVE_ZONE` | the appended delta handlers compress | **never** a cache
read; cold turn saves 1.25x list, not 1.0x |

`headroom/pricing/counterfactual.py` is now the single place cache rates
are resolved and counterfactual tokens are priced — for the live cost
card, the persisted tracker and the durable ledger alike, so the three
cannot drift apart again. It replaces three separate implementations,
two of which were list-only.

Verified, warm Anthropic turn, 8k deferred tool-schema tokens: `$0.0240
list → $0.0057` (4.2x correction). Cold turn: `$0.0240 → $0.0300` —
correctly **above** list, the case list pricing *under*-reported.

## Also fixed

- **1h-TTL writes were priced at the 5m rate.** The proxy counted and
displayed the 5m/1h split but never priced it. LiteLLM publishes the 1h
rate per model (Sonnet: 2.00x base); `pricing/cache_ttl.py` — written
for exactly this and never imported by the cost path — derives it where
a row omits one.
- **Prefix tokens were bucketed pro-rata**, handing them a share of the
1.25x write premium they never paid (~2x high on a warm turn).
- **OTel exporter crash risk**: it called `float()` over every value in
the savings breakdown, so a non-numeric field would take the whole
request-metrics path down.
- **Ledger compaction thrash** (pre-existing, amplified here):
`_maybe_compact` rewrote the entire file under `flock` on *every append*
once it passed 1 MB with nothing out of retention to drop. Added a
per-process back-off; retention is still enforced on read.

## Schema changes

**Ledger v1 → v2**, **savings state v5 → v6.**

v1 stored one list-priced dollar and no mix, so its history was not
merely overstated — it was **unfixable**, because nothing on disk said
what fraction of the removed tokens would have been cache reads. v2
stores the savings split and the observed mix, so a future basis change
can re-derive the whole ledger.

Pre-v6 state seeds both columns from the single figure it has and is
labelled `list` for good; `savings_basis_migrated_at` marks where the
honest numbers start. v1 ledger events still read and aggregate
alongside v2.

## Honesty over precision

Every figure now ships with its list-price ceiling and a `basis` label
saying how soundly it was derived (`catalog`, `catalog+ttl-ratio`,
`provider-ratio`, `list`, `no-mix`, `unpriced`). Totals report their
**weakest** contributing basis, so one unpriceable request can't launder
a total into looking catalog-grade. A request with no cache breakdown —
the MCP tool path, a non-reporting gateway, a streaming turn with no
usage frame — still prices at list and **says so**.

## Harness- and provider-agnostic

The fix sits at the single `emit_request_outcome` → `record_request`
chokepoint that all **32** handler emit sites route through (anthropic
6, openai 11, batch 5, gemini 3, streaming 3, bedrock 1, gateway_turn 2,
server 1) — no per-handler work.

`tests/test_savings_provider_matrix.py` asserts the matrix directly:

| Provider | Reports | Handled |
|---|---|---|
| Anthropic / Bedrock / Vertex | read + write + 5m/1h + uncached | full
|
| OpenAI | read + uncached, write **inferred** | inferred write dropped
— no premium OpenAI doesn't charge |
| Gemini | read only | prices from reads alone |
| gateway / MCP | nothing | list + labelled `no-mix` |

## Tests

- **New:** 21 pricing + 13 provider-matrix + 3 ledger (compaction
back-off, v1/v2 mixing)
- **Updated:** 6 files for the deliberate schema change
- Local sweep across everything touching savings/cost/pricing/metrics:
**904 passed**; targeted re-run after review fixes: **125 passed**
- `ruff check` + `format` clean under the **CI-pinned 0.16.7** (local
0.16.3 missed 5 C408s)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-21 14:41:30 -07:00
JD Davis 1bea0ea31a test: track active LiteLLM DeepSeek pricing (#3161)
## Description

Keep the LiteLLM DeepSeek V4 integration tests compatible with
upstream-owned pricing entries. LiteLLM now publishes these models
directly, so Headroom correctly preserves upstream values instead of
installing its fallback values; the tests must validate the active entry
rather than require fallback prices.

Related: #3157

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Validate that active upstream DeepSeek V4 price entries contain
positive input and output prices.
- Compare `cost_per_token` results with the active LiteLLM model-cost
entry.
- Preserve the existing fallback-price and non-overwrite coverage.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
python -m pytest tests/test_providers/test_deepseek.py -q
20 passed in 4.63s

ruff check tests/test_providers/test_deepseek.py
All checks passed!

ruff format --check tests/test_providers/test_deepseek.py
1 file already formatted

pre-commit: Ruff alignment, merge-conflict check, Ruff, Ruff format, and mypy all passed
```

## Real Behavior Proof

- Environment: Windows, Python 3.13.13, LiteLLM model-cost data
available.
- Exact command / steps: `python -m pytest
tests/test_providers/test_deepseek.py -q`
- Observed result: all 20 DeepSeek provider and pricing tests pass
against the active LiteLLM entries.
- Not tested: provider API calls; this change only concerns local
pricing metadata assertions.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: No runtime behavior changes.
- Kill switch / disable path: N/A.
- Unsafe override required: No.
- Qualification impact: Restores deterministic CI coverage for
upstream-owned pricing entries.
- Rollback path: Revert this test-only commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code where needed
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] Existing tests prove the fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A — test-only change.

## Additional Notes

Documentation changes are not applicable because runtime behavior and
public APIs are unchanged.
2026-08-20 23:19:55 -05:00
6d2254dfb5 fix(anthropic): honor the [1m] 1M-context tier, and price it correctly (#3073)
Two coupled defects on Anthropic's 1M-context tier: Headroom
**under-budgeted** those sessions and **under-priced** them by ~2x. The
second gets worse once the first is fixed, so they ship together.

---

# Part 1 — `[1m]` was lost before the context budget was sized

`sanitize_anthropic_model_id()` strips a trailing `[1m]`, which is
correct for the wire — upstream Anthropic rejects the suffix, and #2027
added the strip for exactly that reason.

But `[1m]` is not only an ANSI artifact. Claude Code appends it to a
model id to request the **1M context tier**, and only sends the
`context-1m` beta header when it is present (#1158 — what `headroom wrap
claude --1m` sets up).

`get_context_limit()` sanitized *before* resolving, so the tier was gone
by lookup time:

```python
provider.get_context_limit("claude-sonnet-4-5[1m]")  # 200_000  ← real window is 1M
```

The request still reached Anthropic correctly and still got a 1M window
— the beta header goes through untouched. What broke is our **budget**:
Headroom sized a 1M session at 200K and began compacting at a fifth of
the available room.

Models whose base is already 1M (`claude-opus-5`, `claude-sonnet-5`)
resolved to 1M either way, which is why this went unnoticed. It bites
the Sonnet 4 / 4.5 family — the models `[1m]` exists for.

**Fix:** read the tier off the id *before* sanitizing; raise the
resolved limit to at least 1M. `max()` rather than assignment, so a base
wider than 1M keeps its own window. Detection is deliberately narrower
than the sanitizer — only a literal `[1m]`; `[0m]`, `[1;32m]` and real
`ESC[` sequences still strip without promoting.

| model id | wire id (unchanged) | limit before | limit after |
|---|---|---|---|
| `claude-sonnet-4-5` | `claude-sonnet-4-5` | 200K | 200K |
| `claude-sonnet-4-5[1m]` | `claude-sonnet-4-5` | **200K** | **1M** |
| `claude-opus-5[1m]` | `claude-opus-5` | 1M | 1M |
| `claude-sonnet-4-5[0m]` | `claude-sonnet-4-5` | 200K | 200K |
| `ESC[1m claude-sonnet-4-5 ESC[0m` | `claude-sonnet-4-5` | 200K | 200K
|

The wire id is unchanged in every case, so #2027 holds — guarded by a
regression test.

---

# Part 2 — the pricing that reports those sessions was wrong

### 2a. The LiteLLM cost path was dead in every provider

`litellm.completion_cost()` no longer accepts `prompt_tokens` /
`completion_tokens`. Every call raised `TypeError`:

```
TypeError: completion_cost() got an unexpected keyword argument 'prompt_tokens'
```

All five providers — `anthropic`, `openai`, `google`, `cohere`,
`litellm` — caught it with a bare `except` and silently fell through to
their hand-maintained tables. The "up-to-date pricing from LiteLLM" the
docstrings promise **has not run at all**. Anthropic additionally passed
`input_tokens - cached_tokens`, the wrong convention (LiteLLM expects
the cache-inclusive total), which would also have suppressed the
long-context threshold even had the call worked.

Replaced with `litellm.cost_per_token()` behind one shared helper,
`pricing.litellm_pricing.estimate_cost_from_tokens()`, which reuses the
existing gateway-alias candidate chain and returns `None` (not an
exception) when LiteLLM can't price a model.

### 2b. Neither path applied Anthropic's long-context premium

On the Sonnet 4 / 4.5 family a prompt over 200K re-prices the **whole**
request — input 2×, output 1.5×, cache 2× — not just the tokens past the
threshold. Rates confirmed from LiteLLM's `*_above_200k_tokens` fields.

| request (`claude-sonnet-4-5`) | reported before | true | error |
|---|---|---|---|
| 100K in / 5K out | $0.3750 | $0.3750 | — |
| 300K in / 5K out | $0.9750 | **$1.9125** | −49% |
| 300K in (150K cached) / 5K out | $0.5700 | **$1.1025** | −48% |

LiteLLM applies this itself once the call works. The manual fallback
needed `_apply_long_context_premium()` — the LiteLLM dependency is gated
`python_version < '3.14'`, so on 3.14 the fallback is the *only* path.
**Both paths now agree to four decimal places on every case under
test.**

---

## What I checked and did *not* change

The fork report that prompted this claimed the Anthropic tables were
materially stale ("Opus 4.x priced wrong"). **That does not hold.** I
audited every entry against LiteLLM's vendored table:

- **Anthropic** — every model LiteLLM knows matches exactly, Opus 4.x
included.
- **OpenAI** — all 17 entries match; the two that don't resolve are
retired models.

The defect was the mechanism, not the numbers, so the rate cards are
untouched.

One thing the repaired path fixes for free: OpenAI's cached-input
discount is **50% on gpt-4o, 75% on gpt-4.1, 90% on gpt-5**, but the
manual path applies a flat 50% estimate. With LiteLLM live, real
per-model rates are used. The flat estimate remains only as the offline
fallback.

## Scope

**No Rust change needed.**
`crates/headroom-proxy/src/compression/model_limits.rs` resolves context
windows but has **no in-tree callers**; the Rust `[1m]` handling is
wire-body sanitization only, correct as-is, and its integration tests
assert behavior this PR does not touch.

**Judgment call worth a reviewer's eye:** the `[1m]` marker is honored
for *any* model, including ones with no 1M tier
(`claude-haiku-4-5-20251001[1m]` → 1M). Gating on an allowlist would be
more precise but reintroduces a hand-maintained table that rots — the
failure mode `model_limits.rs` already documents against. Since `[1m]`
is set by our own wrapper and Claude Code's opt-in, honoring it seemed
the better default. Happy to tighten.

## Tests

- `TestContext1MSuffix` — detection, the 200K→1M promotion, the `max()`
floor, ANSI non-promotion, and the wire-id guard for #2027.
- `TestLongContextPricing` — the premium on both paths (parametrized),
threshold boundary (200,000 vs 200,001), an untiered model charged no
premium, and the two halves meeting: a `[1m]` request gets both the 1M
window and the premium rate.
- `TestLiteLLMCostHelper` — unknown model returns `None`, a known model
prices correctly, and `input_tokens` is cache-inclusive.

Two existing tests were updated, both pinned to the broken behavior:
- `test_estimate_cost_basic` probed a "per 1M" rate by sending exactly
1M tokens, which now crosses the 200K threshold. Re-probed at 100K.
(Worth knowing: `claude-3-5-sonnet-20241022` is retired and no longer in
LiteLLM, so the alias chain resolves it to `claude-sonnet-4-20250514`
and it inherits that model's tier. Harmless — a 200K-window model can't
exceed 200K in reality — but it explains the number.)
- `test_litellm_provider_info_and_cost_fallbacks` monkeypatched
`litellm.completion_cost`; repointed at the new helper seam.

```
ruff check / ruff format / mypy — clean across all six changed source files
```

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-16 22:46:12 -07:00
0805e8e410 fix(providers/openai): bound tiktoken vocab loads with the guarded loader (#2554)
## Description

`headroom/providers/openai.py::_get_encoding` calls
`tiktoken.get_encoding` directly. tiktoken downloads missing
vocabularies via `requests.get` with **no timeout**, so on a network
that blackholes the vocab CDN (corporate firewall, SSL-intercepting
proxy), whichever thread first counts tokens for an OpenAI model — proxy
startup included — blocks indefinitely.

This is the provider-path hole left by #956: the tokenizer registry
already routes through a bounded loader
(`headroom/tokenizers/tiktoken_counter.py`, worker-thread load +
`HEADROOM_TIKTOKEN_LOAD_TIMEOUT_SECONDS`, default 10s) and falls back to
estimation, but the OpenAI provider path never got the same treatment.

Observed in production (Headroom Desktop fleet, Sentry): a proxy that
never finished booting, with a faulthandler dump wedged in
`tiktoken/registry.py` `get_encoding` on the main thread, reached from
the `headroom` CLI entrypoint via click. The desktop app now also
pre-seeds a persistent `TIKTOKEN_CACHE_DIR`, but the unbounded load
affects every deployment of the proxy, so it should be fixed here too.

Follow-up to #956.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `_get_encoding` now routes through the bounded `load_encoding` from
`headroom.tokenizers.tiktoken_counter` instead of calling
`tiktoken.get_encoding` directly, so a stalled vocab download raises
`TiktokenLoadError` after the timeout instead of hanging the calling
thread.
- `OpenAIProvider.get_token_counter` catches `TiktokenLoadError` and
falls back to `EstimatingTokenCounter`, cached per model so later
requests never re-block on the same failed download — mirroring
`TokenizerRegistry._create_tiktoken`.
- `TIKTOKEN_AVAILABLE` uses `importlib.util.find_spec` (the module-level
`import tiktoken` became unused; same pattern as `LITELLM_AVAILABLE`).
- Two regression tests (`TestGuardedEncodingLoad`) covering the
bounded-raise path and the cached estimation fallback.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_providers/ tests/test_tokenizers/
======================= 117 passed, 4 warnings in 27.95s =======================

$ uvx ruff check headroom/providers/openai.py tests/test_providers/test_openai.py
All checks passed!
$ uvx ruff format --check headroom/providers/openai.py tests/test_providers/test_openai.py
2 files already formatted

$ uv run --frozen --extra dev mypy headroom/providers/openai.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS (arm64), Python 3.12, uv-managed venv, branch off
`upstream/main` (a6d4921e).
- Exact command / steps: the stalled download is simulated in
`TestGuardedEncodingLoad` by monkeypatching
`tiktoken_counter.load_encoding` to raise `TiktokenLoadError`;
`OpenAITokenCounter("gpt-4o")` then raises the bounded error, and
`OpenAIProvider.get_token_counter("gpt-4o")` returns a working
`EstimatingTokenCounter` and reuses the same instance on the second
call.
- Observed result: tests pass (see output above); with an empty tiktoken
on-disk cache, the encoding load path is identical to the one in the
production faulthandler dump.
- Not tested: an end-to-end run against a genuinely blackholed vocab CDN
(needs a firewalled network); the timeout mechanism itself is #956's
code, already covered by its own tests.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I commented my code, particularly in hard-to-understand areas
- [x] I made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I added tests that prove my fix is effective or that my feature
works
- [x] New and existing unit tests pass locally with my changes
- [ ] I updated CHANGELOG.md if applicable (N/A — release-please
generates it from the PR title; changelog-guard forbids manual edits)

## Screenshots (if applicable)

N/A — no UI change.

## Additional Notes

The estimation fallback is sticky for the process lifetime (per-model
cache + the loader's fail-fast set from #956): a network that recovers
mid-session keeps estimation until restart. That matches the registry's
existing behavior, so no new divergence is introduced.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
2026-08-12 00:20:16 -05:00
Abhay Singh 08466f3cae fix(providers/anthropic): don't crash token estimation on null tool_calls (#2472)
## Description

`AnthropicTokenCounter._count_message_estimated` (the
tiktoken-approximation fallback used when no Anthropic client is
available) counted OpenAI-format tool calls like this:

```python
if "tool_calls" in message:
    for tool_call in message.get("tool_calls", []):
        if isinstance(tool_call, dict):
            func = tool_call.get("function", {})
            ...
```

The `if "tool_calls" in message` check only tests key presence, not the
value. OpenAI SDKs routinely include `"tool_calls": null` on an
assistant message with no tool calls, so `message.get("tool_calls", [])`
returned `None` (the default only applies when the key is absent) and
`for tool_call in None` raised `TypeError: 'NoneType' object is not
iterable`. That crashes token estimation for the entire request whenever
such a message is present. `tool_call.get("function", {})` had the same
gap for a `"function": null`.

## Fix

Iterate `message.get("tool_calls") or []` so a null or absent value
becomes an empty list, and read `function` with `or {}` for the same
reason. Valid tool calls are counted exactly as before.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/providers/anthropic.py`: value-guard `tool_calls` and
`function` in `_count_message_estimated`.
- `tests/test_providers/test_anthropic.py`: regression counting a
message list that includes `tool_calls: null` and a tool call with
`function: null`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_providers/test_anthropic.py -q
17 passed

# with the fix reverted, the new test fails with
# TypeError: 'NoneType' object is not iterable

$ uvx ruff@0.15.17 check headroom/providers/anthropic.py tests/test_providers/test_anthropic.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/providers/anthropic.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, project venv (`uv sync --extra
proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`, pytest in the venv.
- Exact command / steps: built a real
`AnthropicTokenCounter('claude-3-5-sonnet-20241022')` and called
`count_messages` / `_count_message_estimated` with an assistant message
carrying `tool_calls: null` and one carrying `function: null`, plus a
valid tool call; then reverted `anthropic.py` and re-ran.
- Observed result: with the fix the null shapes count without error and
a valid tool call still adds its name/arguments tokens (5 -> 11 on the
sample); with the fix reverted the `tool_calls: null` message raises
`TypeError: 'NoneType' object is not iterable`. Ran against the actual
module.
- Not tested: a live request from an SDK that emits `tool_calls: null`,
end to end.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
2026-08-12 00:13:14 -05:00
Tejas Chopra 06add9e9d8 fix(providers): stop pricing modern content blocks at zero (#2760)
## Description

Each token counter in `headroom/providers/` had grown its own shortened
content-block walker, handling only the shapes its provider was expected
to send. Everything else fell through and contributed **nothing**.

Measured on one 6,800-char block, via `count_messages` of a single-block
message — so 7–8 is message overhead alone:

| block type | OpenAI ctr | Anthropic ctr |
|---|---|---|
| `text` (control) | 3409 | 3748 |
| `tool_result` | **8** | 3748 |
| `thinking` | **8** | **7** |
| `document` | **8** | **7** |
| `mcp_tool_result` | **8** | **7** |
| `output_text` | **8** | **7** |
| `refusal` | **8** | **7** |

Two things make this worse than a coverage gap:

1. **Each counter zeroed blocks from its own provider.** `output_text`
and `refusal` are OpenAI Responses shapes; `thinking` and `document` are
Anthropic's.
2. **These are the counters the live pipelines use.** `proxy/server.py`
builds them with `AnthropicProvider` / `OpenAIProvider`, so this is the
main request path — not an edge case. #2743 fixed this for
`/v1/compress` only, by routing that route to the registry tokenizers,
whose `BaseTokenizer` walker is complete.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

Rather than add a **fifth** partial walker, the counters now delegate to
the audited one:

- `tokenizers/base.py` — new `count_content_blocks(parts,
count_text_fn)` plus a thin `_DelegatingBlockCounter` adapter, since the
provider counters are not `BaseTokenizer` subclasses. `BaseTokenizer`
itself is untouched.
- `providers/openai.py`, `providers/anthropic.py`,
`providers/openai_compatible.py` — list-content branches delegate.

**Why delegate instead of adding a `count_text(str(block))` catch-all:**
that would serialize a base64 blob and price it as text.
`tiktoken_counter.py` already documents the failure — a 1MB image
becomes ~330K phantom tokens. The shared walker gives media a
pixel/byte-based estimate.

**Scope:** the three counters that accumulate token counts. `google.py`
and `cohere.py` extract a *text string* first and count that, so the
same defect there needs a differently-shaped fix — left as a follow-up.

## Testing

- [x] Unit tests pass
- [x] Linting passes (`ruff check` + `format --check`, pinned 0.15.17)
- [x] New tests added
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_provider_counter_content_blocks.py -q
12 passed in 0.52s

$ uvx ruff@0.15.17 check headroom/ tests/... --exclude headroom/dashboard/templates
All checks passed!
```

**After the fix**, every shape lands within ~1% of the equivalent plain
text, and media stays bounded:

```text
block                  OpenAI  Anthropic
text (control)           3409       3748
tool_result              3409       3748
thinking                 3419       3759
document                 3423       3763
mcp_tool_result          3422       3762
output_text              3420       3760
refusal                  3421       3761
image b64 200KB          1608       1607   <- pixel estimate, not ~50K as text
```

## Real Behavior Proof — including a regression I caught

**This change flipped an existing test**, and I only found it because
every suite was run against clean `upstream/main` in the same
environment with the failure sets diffed:

```text
before the test rewrite:
  upstream/main : 1 failed, 104 passed
  this branch   : 2 failed, 103 passed        <- regression
  diff          : + test_openai_compatible_token_counter_ignores_unhandled_content_shapes
```

That test asserted `content: [{"type": "image"}, 123] == 8` — i.e. it
**pinned the defect**, that unhandled shapes contribute nothing.
Rewritten as `..._prices_declared_media`: a declared image is now priced
(1608) while a bare int is still correctly ignored (8), with the
rationale in the docstring.

```text
after the rewrite:
  upstream/main : 1 failed, 104 passed, 10 skipped, 25 errors
  this branch   : 1 failed, 104 passed, 10 skipped, 25 errors
  failure sets  : IDENTICAL
```

- **Pre-existing, not from this change:** the 1 failure and all 25
errors. The errors are all in
`test_compress_route_tokenizer_by_model.py`, whose loopback `TestClient`
fixture this throwaway env cannot satisfy.
- **Environment note:** `content_router` and several suites need the
compiled `headroom._core`, which isn't in a fresh worktree (gitignored,
built in-place). I copied the built `.so` in to run these and removed it
before committing.
- **Not tested:** no live provider call, so the *absolute* accuracy of
the 1600 image estimate against a real Anthropic/OpenAI bill is
unverified — it is the value `BaseTokenizer` already used, and this PR
only changes which blocks reach it.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] I did **not** edit `CHANGELOG.md`

## Related

Fourth PR from one tokenizer-consistency audit: #2757 (litellm total
prompt / `--budget`), #2758 (HuggingFace chat templates, `gpt-5`,
gateway-wrapped names), #2759 (router token units). Plus #2756, which
splits the local/provider token scales in `RequestOutcome`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-08-03 22:40:38 -07:00
Abhay Singh b976378c3e test(pricing): stop asserting DeepSeek pricing freshness on wall-clock time (#2428)
## Description

The shared `test` job is currently failing on every open PR because of a
wall-clock time-bomb in the DeepSeek pricing tests, not because of any
code change.


`tests/test_providers/test_deepseek.py::TestDeepSeekPricingModule::test_registry_staleness_and_source_url`
asserted:

```python
assert not registry.is_stale()
```

`PricingRegistry.is_stale()` returns `(date.today() - last_updated) >
timedelta(days=30)`. The DeepSeek registry ships `LAST_UPDATED =
date(2026, 6, 19)`, so this assertion holds only while the current date
stays within 30 days of that constant. Once it lapses, the test fails on
time alone, turning the `test` shard red for every unrelated PR in the
repo. It is failing right now (31 days past `LAST_UPDATED`).

This is not testing code behavior: it only checks that the machine's
clock is within 30 days of a hardcoded date. The sibling Anthropic and
OpenAI registries are 560 days old and make no such assertion, so
DeepSeek is the odd one out here rather than a deliberate freshness
gate.

## Fix

Drop the freshness assertion and keep the meaningful `source_url` check,
renaming the test to `test_registry_source_url` to match what it now
verifies.

The staleness mechanism stays fully and time-independently covered by
`tests/test_pricing.py::test_registry_staleness_and_warning`, which
builds registries with `date.today() - timedelta(days=30)` (asserts not
stale) and `date.today() - timedelta(days=31)` (asserts stale) plus the
warning text. So this removes a fragile environmental assertion without
reducing real coverage, and aligns DeepSeek with the Anthropic/OpenAI
registries.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `tests/test_providers/test_deepseek.py`: remove the
wall-clock-dependent `assert not registry.is_stale()`, keep the
`source_url` assertion, rename the test to `test_registry_source_url`,
and add a comment explaining why freshness is not asserted here.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uvx ruff@0.15.17 check tests/test_providers/test_deepseek.py
All checks passed!
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, project venv (`uv sync --extra
proxy`), `uvx ruff@0.15.17`.
- Exact command / steps: with the current date at 31 days past
`LAST_UPDATED`, ran the registry's `is_stale()` and the fixed test body
against the real modules, plus the mechanism test from
`tests/test_pricing.py`.
- Observed result: `get_deepseek_registry().is_stale()` is `True` on the
current date (which is exactly what broke the old assertion); the fixed
`test_registry_source_url` body passes regardless of the date; and
`test_registry_staleness_and_warning` still passes, so the staleness
mechanism remains covered.
- Not tested: a live DeepSeek pricing fetch (out of scope; pricing
values are unchanged).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
2026-07-19 22:14:34 -07:00
yiihaoandJerrettDavis 0c7087539d fix: add DeepSeek V4/V3.2/R1 tokenizer mappings and context limits (#912)
The tokenizer registry routed deepseek-v4-pro, deepseek-v4-flash,
deepseek-chat, deepseek-reasoner, and other modern DeepSeek models
to the 2023-era deepseek-llm-7b-base tokenizer via prefix fallback.

This caused token counts off by 30-50%, broken context-limit detection
(V4-Pro supports 1M but got 32K), and inaccurate savings reports.

## Fix

3 files, +43/-2:

- **huggingface.py**: 16 new MODEL_TO_TOKENIZER entries with verified
  HuggingFace IDs (deepseek-ai/DeepSeek-V4-Pro, V4-Flash, V3.2,
  V3-0324, R1, R1-0528, Reasoner, Chat, Coder-V2, etc.)

- **openai_compatible.py**: 17 new _DEFAULT_CONTEXT_LIMITS entries
  (V4-Pro/Flash -> 1M, R1/Reasoner -> 131K, V3 -> 128K, etc.)

- **openai.py**: 8 new _CONTEXT_LIMITS entries for LiteLLM-fallback.

Existing mappings untouched (backward compatible).

## Real behavior proof

- **Setup**: Windows 11, Python 3.13.14, headroom-ai 0.2.15 wheel +
  source checkout at v0.24.0. No Rust extension built (headroom._core
  unavailable). Touched files are at parity with v0.24.0.

- **Steps after patch**:
  ```
  python3 -c "
  from headroom.tokenizers.huggingface import get_tokenizer_name
for m in
['deepseek-v4-pro','deepseek-chat','deepseek-reasoner','deepseek-v4-flash']:
      print(f'{m} -> {get_tokenizer_name(m)}')
  from headroom.tokenizers.registry import get_tokenizer
  for m in ['deepseek-v4-pro','deepseek-chat','deepseek-reasoner']:
      print(f'{m}: {get_tokenizer(m)}')
  "
  ```

- **Observed result**:
  ```
  deepseek-v4-pro  -> deepseek-ai/DeepSeek-V4-Pro
  deepseek-v4-flash -> deepseek-ai/DeepSeek-V4-Flash
  deepseek-chat    -> deepseek-ai/DeepSeek-V3
  deepseek-reasoner -> deepseek-ai/DeepSeek-R1
  ```
  Previously ALL resolved to deepseek-ai/deepseek-llm-7b-base.
  TokenizerRegistry routes correctly. Context limits verified
  (1M / 131K / 128K). compress() import smoke-tested OK.

- **Not tested**: full proxy e2e with a live DeepSeek API key
  (no available key). HuggingFace AutoTokenizer download confirmed
  for V4-Pro/V3/R1 but produced GBK decode errors from hf_hub on
  this zh-CN Windows locale during config fetch -- a separate
  huggingface_hub issue unrelated to this change.


<!-- headroom-maintainer-template-completion:start -->

## Description

This PR prepares `fix: add DeepSeek V4/V3.2/R1 tokenizer mappings and
context limits` for review by documenting the intended change,
validation evidence, and remaining merge-readiness context.

Linked issues: None declared.

## Type of Change

- [x] Bug fix
- [ ] New feature
- [ ] Documentation
- [ ] Refactor
- [ ] Tests only

## Changes Made

- Commit: fix: add DeepSeek V4/V3.2/R1 tokenizer mappings and context
limits
- Touches `headroom/providers/openai.py`
- Touches `headroom/providers/openai_compatible.py`
- Touches `headroom/tokenizers/huggingface.py`

## Testing

- [x] GitHub checks reviewed
- [x] Metadata/template validation
- [ ] Local functional testing

### Test Output

```text
gh pr view 912 --repo chopratejas/headroom --json statusCheckRollup
- PR Governance / label: SUCCESS
- external / GitGuardian Security Checks: SUCCESS
```

## Real Behavior Proof

- Environment: GitHub PR metadata and checks for `chopratejas/headroom`
PR #912.
- Exact command / steps: Reviewed PR title, commits, changed files,
linked issues, labels, and check rollup; appended this maintainer
template completion block without replacing the author's original
description.
- Observed result: PR body now contains all required governance
sections, checked readiness fields, and a non-placeholder validation
evidence block.
- Not tested: This pass updated PR metadata only; code validation
remains represented by the linked GitHub checks and any author-provided
evidence above.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

<!-- headroom-maintainer-template-completion:end -->

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-15 09:24:44 -05:00
Vinay Gupta e22d7453d4 fix(proxy): strip 1m model suffix before upstream forwarding (#1840)
## Description

Strips dangling terminal-style model suffixes like `[1m]` from
Anthropic-compatible model ids before Headroom forwards `/v1/messages`
upstream.

Closes #1812

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Generalized `sanitize_anthropic_model_id()` so the existing dangling
ANSI-style suffix cleanup applies to Anthropic-compatible non-Claude
models, including `glm-5.2[1m]`.
- Added a provider-level regression for `glm-5.2[1m] -> glm-5.2`.
- Added a `/v1/messages` handler regression that captures the upstream
request body and verifies Headroom forwards `glm-5.2`, not
`glm-5.2[1m]`.

## Testing

- [x] Unit tests pass (`pytest`) — focused local tests and full CI test
matrix passed
- [x] Linting passes (`ruff check .`) — local Ruff and CI lint passed
- [x] Type checking passes (`mypy headroom`) — local mypy and CI lint
passed
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ rtk proxy env HEADROOM_REQUIRE_RUST_CORE=false PYTHONPATH=/Users/vinaygupta/Desktop/git/headroom-fix-1812-1m-model-suffix /tmp/headroom-1812-testenv/bin/python -c '<inject local headroom._core test stub; pytest.main(["tests/test_providers/test_anthropic.py", "tests/test_proxy_anthropic_model_sanitization.py"])>'
============================= test session starts ==============================
platform darwin -- Python 3.13.11, pytest-9.1.1, pluggy-1.6.0 -- /private/tmp/headroom-1812-testenv/bin/python
collected 17 items

tests/test_providers/test_anthropic.py::TestAnthropicModelSanitization::test_sanitize_model_id_removes_ansi_escape_sequences PASSED
tests/test_providers/test_anthropic.py::TestAnthropicModelSanitization::test_sanitize_model_id_removes_displayed_style_suffix PASSED
tests/test_providers/test_anthropic.py::TestAnthropicModelSanitization::test_sanitize_model_metadata_cleans_nested_model_ids PASSED
tests/test_providers/test_anthropic.py::TestAnthropicTokenCounting::test_count_text_fallback PASSED
tests/test_providers/test_anthropic.py::TestAnthropicTokenCounting::test_count_messages_basic PASSED
tests/test_providers/test_anthropic.py::TestAnthropicTokenCounting::test_count_text_allows_literal_special_tokens PASSED
tests/test_providers/test_anthropic.py::TestAnthropicModelLimits::test_get_context_limit_claude_sonnet PASSED
tests/test_providers/test_anthropic.py::TestAnthropicModelLimits::test_get_context_limit_claude_opus PASSED
tests/test_providers/test_anthropic.py::TestAnthropicModelLimits::test_get_context_limit_strips_ansi_model_suffix PASSED
tests/test_providers/test_anthropic.py::TestAnthropicModelLimits::test_get_context_limit_claude_5_family PASSED
tests/test_providers/test_anthropic.py::TestAnthropicModelLimits::test_supports_model_known PASSED
tests/test_providers/test_anthropic.py::TestAnthropicModelLimits::test_supports_model_prefix PASSED
tests/test_providers/test_anthropic.py::TestAnthropicModelLimits::test_token_counter_cache_uses_sanitized_model_id PASSED
tests/test_providers/test_anthropic.py::TestAnthropicCostEstimation::test_estimate_cost_basic PASSED
tests/test_providers/test_anthropic.py::TestAnthropicCostEstimation::test_pricing_lookup_strips_ansi_model_suffix PASSED
tests/test_providers/test_anthropic.py::TestAnthropicCostEstimation::test_pricing_claude_5_family PASSED
tests/test_proxy_anthropic_model_sanitization.py::test_anthropic_messages_strips_local_1m_model_suffix_before_forwarding PASSED

======================== 17 passed, 3 warnings in 2.11s ========================

$ rtk uvx ruff check headroom/providers/anthropic.py tests/test_providers/test_anthropic.py tests/test_proxy_anthropic_model_sanitization.py
All checks passed!

$ rtk uvx ruff format --check headroom/providers/anthropic.py tests/test_providers/test_anthropic.py tests/test_proxy_anthropic_model_sanitization.py
3 files already formatted
```

The normal editable test command was attempted but did not reach test
execution in this local checkout because the native extension build
failed:

```text
$ rtk uv run pytest tests/test_providers/test_anthropic.py tests/test_proxy_anthropic_model_sanitization.py
× Failed to build `headroom-ai @ file:///Users/vinaygupta/Desktop/git/headroom-fix-1812-1m-model-suffix`
warning: esaxx-rs@0.1.10: src/esaxx.cpp:620:10: fatal error: 'cstdint' file not found
error: failed to run custom build command for `esaxx-rs v0.1.10`
```

## Real Behavior Proof

- Environment: local macOS worktree from current upstream `main`; Python
3.13.11 throwaway test environment; `HEADROOM_REQUIRE_RUST_CORE=false`;
in-memory `headroom._core` stub used only to avoid the local missing
native extension during Python-level tests.
- Exact command / steps: POST a TestClient `/v1/messages` request with
`{"model": "glm-5.2[1m]", ...}` and replace `_retry_request` with a test
double that records the upstream body.
- Observed result: the recorded upstream request body contains
`{"model": "glm-5.2"}` and `mutation_reasons == ["sanitize_model_id"]`,
so the mutated JSON body is serialized instead of forwarding the
original bytes.
- Not tested: live Z.AI credentials/provider call; full local pytest;
local `mypy headroom`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A

## Additional Notes

- All non-skipped GitHub Actions checks are green after the rebase onto
`main`; skipped jobs are path-gated.
- No code comments were added because the fix reuses the existing
sanitizer and mutation-tracking path.
- Documentation and CHANGELOG updates are N/A for this narrow proxy
compatibility fix.
- The local pytest warnings were from the throwaway environment/test
tooling (`asyncio_mode`, Starlette TestClient deprecation, and the
existing AnthropicProvider no-client warning), not from the changed code
path.
2026-07-06 08:33:45 -07:00
Kenneth Wong e84ca980cf feat(anthropic): add Claude 5 family pricing & align current rates (#1767)
## Summary

Adds Claude 5 generation metadata and aligns the Anthropic fallback
pricing / context-limit tables in `headroom/providers/anthropic.py` with
current [Anthropic
pricing](https://platform.claude.com/docs/en/about-claude/pricing)
(verified 2026-07-04). Several Claude 4.x entries carried stale rates,
the new Claude 5 models (Fable 5, Opus 4.8, Sonnet 5) had no metadata,
and the generic fallback tests disagreed with the provider tests.

Supersedes #1485 (rebased onto latest `main`, squashed to one commit,
extended with Sonnet 5 / Fable 5 and the requested fallback-test fixes).

## Changes

All values `$ / MTok`; `cached_input` = prompt-cache read = 0.1× input.

| Tier | Model | Before | After | Context |
|---|---|---|---|---|
| Fable | `claude-fable-5` | — *(new)* | $10 / $50 / $1.00 | **1M** |
| Opus | `claude-opus-4-8` | — *(new)* | $5 / $25 / $0.50 | **1M** |
| Opus | `claude-opus-4-7` | $15 / $75 / $1.50 | $5 / $25 / $0.50 | 1M |
| Opus | `claude-opus-4-6` | $15 / $75 / $1.50 | $5 / $25 / $0.50 | 1M |
| Opus | `claude-opus-4-5-20251101` | $15 / $75 / $1.50 | $5 / $25 /
$0.50 | 200K |
| Sonnet | `claude-sonnet-5` | — *(new)* | $3 / $15 / $0.30 | **1M** |
| Sonnet | `claude-sonnet-4-6` | — *(new)* | $3 / $15 / $0.30 | **1M** |
| Sonnet | `claude-sonnet-4-5` | — *(new)* | $3 / $15 / $0.30 | 200K |
| Haiku | `claude-haiku-4-5-20251001` | $0.80 / $4 / $0.08 *(3.5 rates)*
| $1 / $5 / $0.10 | 200K |

`claude-sonnet-4-20250514` and all Claude 3.x / 3.5.x entries were
already correct — left unchanged.

The **1M-context** entries (Fable 5, Opus 4.8, Sonnet 5, Sonnet 4.6) are
functional, not cosmetic: they ship in the long-context tier, and
without explicit entries the `sonnet` / `opus` pattern defaults would
report 200K.

Sonnet 5 is pinned to the **standard** Sonnet tier ($3 / $15 / $0.30);
Anthropic's introductory rate ($2 / $10 through Aug 31 2026) is
intentionally not encoded to avoid a time-dependent fixture.

## Fallback-model tests (addresses review on #1485)

`_PATTERN_DEFAULTS["opus"]` is aligned to the current Opus tier ($5 /
$25 / $0.50) so the generic fallback suite and the provider-specific
suite agree:

- `test_pricing_for_known_models` — Opus 4.5 pins $5 / $25 / $0.50
- `test_pattern_based_inference_opus` — unknown-opus fallback now $5 /
$25
- `test_cost_estimation_for_new_models` — fixture estimate corrected
$22.5 → $7.5
- `test_pattern_based_inference_sonnet` — retargeted to
`claude-sonnet-6-*` (the old `claude-sonnet-5-*` probe now
prefix-matches the real `claude-sonnet-5` key)

New provider coverage:

- `test_get_context_limit_claude_5_family` — Fable 5 / Opus 4.8 / Sonnet
5 all 1M
- `test_pricing_claude_5_family` — exact rate table for the 3 new models

Full suite: **48 passed**.

## Source

https://platform.claude.com/docs/en/about-claude/pricing
2026-07-03 12:09:35 -07:00
AliandClaude 0e6d922f88 feat(pricing): add DeepSeek V4 model pricing (deepseek-v4-flash, deepseek-v4-pro) (#1168)
## Description

Adds pricing support for DeepSeek V4 models (`deepseek-v4-flash` and
`deepseek-v4-pro`) when routing Headroom through `--anthropic-api-url
https://api.deepseek.com/anthropic`. The vendored LiteLLM pricing
database predates DeepSeek V4, so cost estimation silently returned
`None` for these models.

## Type of Change

- [x] New feature (non-breaking change that adds functionality)

## Changes Made

- **`headroom/pricing/deepseek_prices.py`** — New pricing data module
with `ModelPricing` dataclass entries for both V4 models, following the
pattern of `anthropic_prices.py`
- **`headroom/pricing/__init__.py`** — Exports `DEEPSEEK_PRICES`,
`get_deepseek_registry()`, `DEEPSEEK_LAST_UPDATED`
- **`headroom/pricing/litellm_pricing.py`** — Runtime injection of
DeepSeek V4 pricing into `litellm.model_cost`, plus `deepseek-` prefix
added to `resolve_litellm_model()` provider prefix list
- **`headroom/providers/anthropic.py`** — DeepSeek fallback in
`_get_pricing()` when model starts with `deepseek-` and LiteLLM is
unavailable
- **`crates/headroom-proxy/data/model_prices_and_context_window.json`**
— Vendored JSON entries (bare + provider-prefixed) for Rust-side context
window lookups
- **`tests/test_providers/test_deepseek.py`** — 20 tests across 3 test
classes (pricing data, LiteLLM injection, Anthropic fallback)
- **`tests/test_pricing.py`** — Added DeepSeek export validation
alongside existing OpenAI/Anthropic assertions

## Testing

- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```
========================= 137 passed, 8 warnings in 8.47s =========================
```

## Real Behavior Proof

- Environment: Windows 10, Python 3.12, litellm 1.60+
- Exact command / steps: `python -c "from headroom.proxy.cost import
CostTracker; t = CostTracker();
print(t.estimate_cost('deepseek-v4-flash', input_tokens=1000000,
output_tokens=1000000))"`
- Observed result: `$0.4200` (0.14 input + 0.28 output per 1M tokens)
- Not tested: Live DeepSeek API routing via `--anthropic-api-url`
(requires API key and Docker deployment)

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

The 90% cache discount heuristic in `AnthropicProvider.estimate_cost()`
(line 680) is a pre-existing pattern. DeepSeek V4 has much deeper cache
discounts (98-99%), but the LiteLLM path currently falls through to the
manual fallback which uses correct cached prices. A future improvement
could prefer `cache_read_input_token_cost` from model info over the
hardcoded `* 0.1` heuristic.

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-06-24 09:44:27 -05:00
Umut Keltek d480c464e9 fix(tokenizers): treat literal special-token strings as plain text (#1244)
## Description

`tiktoken`'s `Encoding.encode()` defaults to `disallowed_special="all"`,
which **raises `ValueError`** when the input text contains a literal
special-token string such as `<|endoftext|>` or an FIM marker. Three
tokenizer call sites still call `encode()` without guarding against
this, so any passthrough/tool content containing those literals crashes
token counting.

In the proxy this aborts compression of `/v1/responses` requests. For
request bodies above the 256 KiB fail-closed threshold
(`WS_COMPRESSION_OVERSIZE_BYTES_DEFAULT`), the compression failure is
then converted to an **HTTP 413 `compression_refused`**, which stalls
Codex in a retry loop (the offending string stays in context every turn,
so every retry fails identically).

Observed in production with the token-mode proxy in front of Codex:

```text
WARNING /v1/responses compression failed (bytes=588269):
  ValueError: Encountered text corresponding to disallowed special token '<|endoftext|>'.
ERROR   /v1/responses REFUSING to forward request after compression failure
  (reason=oversize:bytes=588269>threshold=262144, bytes=588269); returning HTTP 413
```

`AnthropicTokenCounter.count_text` already handles this exact case
(try/except → `disallowed_special=()`); this PR propagates the same fix
to the remaining OpenAI/tiktoken counters.

Closes # <!-- no issue filed; happy to open one if preferred -->

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/providers/openai.py` — `OpenAITokenCounter.count_text`: fall
back to `disallowed_special=()` on `ValueError`.
- `headroom/tokenizers/tiktoken_counter.py` — same fallback in
`TiktokenCounter.count_text` **and** `TiktokenCounter.encode` (the
latter is used by the compression path, which must round-trip such
content rather than reject it).
- Each fallback mirrors the existing `AnthropicTokenCounter.count_text`
idiom and comments.
- Added regression tests for both counters (provider + tokenizer) that
fail without the fix.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q tests/test_tokenizers.py tests/test_tokenizer.py \
            tests/test_providers/test_openai.py tests/test_providers/test_anthropic.py
75 passed, 14 skipped, 2 warnings in 2.22s

$ pytest -q tests/test_proxy_count_tokens_integration.py \
            tests/test_openai_responses_context_compaction.py \
            tests/test_openai_codex_routing.py
23 passed, 20 skipped, 1 warning in 4.33s

$ ruff check <changed files>         # All checks passed!
$ ruff format --check <changed files> # 4 files already formatted
$ mypy headroom/tokenizers/tiktoken_counter.py headroom/providers/openai.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: clean clone at `v0.26.0-41-g7c26a54d`, editable install
(`pip install -e ".[dev,proxy]"`), Python 3.14.
- Exact command / steps: negative control — stash only the source fix
(keep the new tests), run the three new regression tests, then restore
the fix and re-run:

```text
$ git stash push headroom/tokenizers/tiktoken_counter.py headroom/providers/openai.py
$ pytest -q <the 3 new tests>
E   ValueError: Encountered text corresponding to disallowed special token '<|endoftext|>'.
3 failed in 0.21s
$ git stash pop      # restore fix
$ pytest -q <the 3 new tests>
3 passed
```

- Observed result: without the fix the new tests reproduce the exact
production `ValueError`; with the fix, `count_text`/`encode` treat the
markers as ordinary text (e.g. `"x <|endoftext|> y"` → 16 tokens,
`decode(encode(text)) == text`).
- Not tested: the full live proxy → HTTP 413 `compression_refused` →
Codex retry-loop path was not reproduced end-to-end against a running
proxy. Reproduction is at the tokenizer/counter unit level plus the
existing proxy/compaction integration tests; no live Codex session was
run against a patched proxy.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review
2026-06-21 19:39:04 -07:00
Shengbo_WangandClaude Opus 4.6 bcabc5cb11 fix(providers): update DeepSeek V3 context limit from 128K to 1M (#1038) (#1137)
## Description

Update `_DEFAULT_CONTEXT_LIMITS` so DeepSeek V3/V4 use their actual 1M
(1,048,576) context window instead of the outdated 128K. The hardcoded
128K causes Headroom to trigger compression far too early, defeating the
purpose of using a long-context model.

Closes #1038

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Update `deepseek` default from 32,768 to 1,048,576 (V3/V4 family
default)
- Update `deepseek-v3` from 128,000 to 1,048,576
- Update `deepseek-coder` from 16,384 to 128,000 (Coder V2+)
- Add `deepseek-v4` entry at 1,048,576
- `deepseek-v2` stays at 128,000 (unchanged)

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_providers/test_universal.py -v -x
37 passed, 3 skipped in 38.08s
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.11, headroom main (6904d47)
- Exact command / steps: python -m pytest
tests/test_providers/test_universal.py::TestOpenAICompatibleProvider::test_get_context_limit_deepseek_v3_is_1m
-v
- Observed result: PASSED - deepseek-v3 returns 1048576, deepseek-v4
returns 1048576, deepseek returns 1048576, deepseek-v2 returns 128000
- Not tested: no manual proxy testing with a live DeepSeek endpoint

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

The context limit values are based on the DeepSeek V3 technical report
and API documentation, which specify a 1M-token context window. DeepSeek
Coder V2+ also supports 128K, up from the original Coder's 16K.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-06-18 10:38:24 -07:00
0c5c89d05c fix(anthropic): strip styled Claude model ids (#651)
## Description

Fixes #626 by normalizing Anthropic/Claude model ids that contain ANSI
escape sequences or dangling style suffixes before provider lookups and
upstream forwarding. The branch has been updated onto current `main` and
the proxy handler conflicts have been resolved.

## Type of Change

- [x] Bug fix
- [ ] New feature
- [ ] Documentation
- [ ] Refactor
- [x] Tests only

## Changes Made

- Normalize Anthropic model ids before context/pricing lookup.
- Sanitize Anthropic `/v1/models` metadata and styled `/v1/models/{id}`
passthrough paths.
- Sanitize `/v1/messages` request body model ids before upstream
forwarding.
- Resolved current-main conflicts while preserving newer
`model_override` and streaming passthrough behavior.

## Testing

- [x] Unit tests
- [x] Route/proxy tests
- [x] Lint/static checks
- [ ] Manual testing

### Test Output

```text
UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --with pytest --with fastapi --with httpx --with uvicorn --with h2 python -m pytest tests/test_providers/test_anthropic.py tests/test_provider_proxy_routes.py::test_anthropic_model_metadata_strips_ansi_model_ids tests/test_provider_proxy_routes.py::test_anthropic_model_detail_path_strips_ansi_model_id tests/test_provider_proxy_routes.py::test_anthropic_messages_strips_ansi_model_id_before_upstream -q
17 passed, 2 warnings in 39.91s

UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --with ruff ruff check headroom/providers/anthropic.py headroom/proxy/handlers/openai.py headroom/proxy/handlers/anthropic.py tests/test_providers/test_anthropic.py tests/test_provider_proxy_routes.py
All checks passed!
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13.3, focused local worktree for PR
#651 after merging current `upstream/main`.
- Exact command / steps: Merged current main, resolved conflicts in
Anthropic/OpenAI proxy handlers, ran the PR's targeted provider/proxy
tests and ruff checks.
- Observed result: Styled Anthropic model metadata, model-detail path,
and messages upstream sanitization tests pass; ruff reports no issues.
- Not tested: Full repository mypy/pre-commit; existing unrelated
Windows `fcntl` typing errors block full hook execution locally.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review


<!-- headroom-maintainer-template-completion:start -->

## Description

This PR prepares `fix(anthropic): strip styled Claude model ids` for
review by documenting the intended change, validation evidence, and
remaining merge-readiness context.

Linked issues: #626

## Type of Change

- [x] Bug fix
- [ ] New feature
- [ ] Documentation
- [ ] Refactor
- [ ] Tests only

## Changes Made

- Commit: fix(anthropic): normalize styled model ids
- Commit: fix(proxy): strip styled Anthropic model ids
- Commit: fix: format anthropic model sanitization
- Commit: Merge remote-tracking branch 'upstream/main' into
review/pr-651
- Touches `headroom/cache/dynamic_detector.py`
- Touches `headroom/providers/anthropic.py`
- Touches `headroom/proxy/handlers/anthropic.py`
- Touches `headroom/proxy/handlers/openai.py`
- Touches `tests/test_provider_proxy_routes.py`
- Touches `tests/test_providers/test_anthropic.py`

## Testing

- [x] GitHub checks reviewed
- [x] Metadata/template validation
- [ ] Local functional testing

### Test Output

```text
gh pr view 651 --repo chopratejas/headroom --json statusCheckRollup
- PR Governance / template: FAILURE
- CI / changes: SUCCESS
- Init E2E / docker-init-e2e: SUCCESS
- Wrap E2E / docker-wrap-e2e: SUCCESS
- Wrap Native E2E / wrap-native (ubuntu-latest): SUCCESS
- Wrap Native E2E / wrap-native (macos-latest): SUCCESS
- CI / commitlint: SUCCESS
- PR Governance / label: SUCCESS
- CI / lint: SUCCESS
- CI / build-wheel: SUCCESS
- CI / prefetch-model: SUCCESS
- CI / build: SUCCESS
```

## Real Behavior Proof

- Environment: GitHub PR metadata and checks for `chopratejas/headroom`
PR #651.
- Exact command / steps: Reviewed PR title, commits, changed files,
linked issues, labels, and check rollup; appended this maintainer
template completion block without replacing the author's original
description.
- Observed result: PR body now contains all required governance
sections, checked readiness fields, and a non-placeholder validation
evidence block.
- Not tested: This pass updated PR metadata only; code validation
remains represented by the linked GitHub checks and any author-provided
evidence above.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

<!-- headroom-maintainer-template-completion:end -->

---------

Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-06-13 13:46:21 -05:00
JerrettDavisandCopilot 8310a495ba style: match CI ruff formatting
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-22 22:21:00 -05:00
JerrettDavisandCopilot 4576f9caba test: remove provider diff churn
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-22 22:13:13 -05:00
JerrettDavisandCopilot 7831620eca test: expand provider slice coverage
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-22 21:58:03 -05:00
JerrettDavis 8e4d7759de Harden cache validation reporting and TTL analysis 2026-04-06 20:11:21 -05:00
chopratejasandClaude Opus 4.5 7a34030b0d Fix mypy type errors in LiteLLM and OpenAI providers
- Add type: ignore[assignment] comments for optional litellm imports
- Add None checks before accessing optional module functions
- Handle nullable max_input_tokens and max_output_tokens values

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-16 16:42:11 -08:00
chopratejas 905c229251 Add AST-based code compression and custom model configuration
CodeAwareCompressor:
- Tree-sitter based AST parsing for Python, JS, TS, Go, Rust, Java, C, C++
- Preserves imports, signatures, type annotations, error handlers
- Guarantees syntactically valid output
- Uses tree-sitter-language-pack for broad language support

ContentRouter:
- Intelligent compression orchestrator
- Auto-routes content to optimal compressor based on type detection
- Source hint support for high-confidence routing

Custom Model Configuration:
- HEADROOM_MODEL_LIMITS env var and ~/.headroom/models.json support
- Pattern-based inference for unknown models (opus/sonnet/haiku tiers)
- Support for Claude 4.5, Claude 4, o3, o3-mini
- Graceful fallback - never crashes on unknown models
2026-01-14 13:46:55 -08:00
chopratejas e4a41faa33 Fix all ruff lint and format errors for CI
- Fix E402: Move module-level imports to top of file
- Fix F401: Add noqa for availability check imports
- Fix F402: Rename loop variables shadowing imports
- Fix E722: Replace bare except with except Exception
- Fix B904: Add exception chaining (from e)
- Fix F811: Remove duplicate imports
- Fix B027: Add noqa for empty close() method
- Fix E741: Rename ambiguous variable l -> label
- Fix I001: Import sorting issues
- Apply ruff format to all 106 files

All 902 tests pass.
2026-01-10 15:33:44 -08:00
chopratejas 175746cc26 Prepare for OSS release v0.2.0
This commit prepares Headroom for public open source release with
comprehensive documentation, licensing, and community infrastructure.

License & Legal:
- Add Apache 2.0 LICENSE file
- Add NOTICE file with third-party attributions
- Add SECURITY.md for vulnerability reporting

Community:
- Add CONTRIBUTING.md with contribution guidelines
- Add CODE_OF_CONDUCT.md (Contributor Covenant)
- Add GitHub issue templates (bug report, feature request)
- Add pull request template

Documentation:
- Update README.md with compelling value proposition
- Add docs/getting-started.md
- Add docs/proxy.md for proxy server documentation
- Add docs/transforms.md for transform reference
- Add docs/api.md for API reference
- Add examples/README.md

Package Infrastructure:
- Add headroom/py.typed for PEP 561 compliance
- Add headroom/cli.py for CLI entry point
- Add .github/workflows/ci.yml for CI pipeline
- Add .github/workflows/publish.yml for PyPI publishing
- Update pyproject.toml with proper metadata

New Features:
- Add multi-provider support (Google, Cohere, LiteLLM, OpenAI-compatible)
- Add universal tokenizer registry with multiple backends
- Add model registry with pricing and context limits
- Add production proxy server with caching and rate limiting

Code Quality:
- Fix 83 lint issues via ruff auto-fix
- Fix version consistency (benchmarks 0.1.0 → 0.2.0)
- Add skip decorators for optional dependency tests
2026-01-07 11:36:44 -08:00
chopratejas 9c7d4512d6 Initial commit: Headroom SDK - LLM context optimization toolkit
A comprehensive SDK for optimizing LLM context windows, reducing token
usage while preserving critical information for AI agents.

Core Features:
- SmartCrusher: Statistical compression of tool outputs (70-85% reduction)
- CacheAligner: Prefix optimization for prompt cache hits
- RollingWindow: Intelligent context window management
- BM25/Hybrid relevance scoring for smart item selection

Integrations:
- OpenAI and Anthropic provider support
- LangChain integration (ChatModel, Callbacks, Runnable)
- MCP (Model Context Protocol) integration for tool compression

Test Coverage:
- 372 tests passing across all modules
- 35 performance benchmarks
- Real-world agent evaluations with 88% token savings

Key Components:
- headroom/transforms/: Core compression transforms
- headroom/providers/: OpenAI and Anthropic support
- headroom/integrations/: LangChain and MCP integrations
- headroom/relevance/: BM25 and hybrid scoring
- headroom/pricing/: Model pricing registry
- benchmarks/: Performance benchmark suite
- examples/: Usage examples and demos
2026-01-06 23:16:58 -08:00