Commit Graph
7 Commits
Author SHA1 Message Date
yi111andJD Davis d503c575ea fix: prevent HF tokenizer downloads in offline mode (#3783)
## Description

`HEADROOM_OFFLINE=1` must not fall through from a cold HuggingFace
tokenizer cache to a network download. This change applies the existing
offline environment defaults before importing Transformers and returns
to estimation after a cache miss when offline mode is active.

Closes #3777

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Apply `apply_offline_env()` before the Transformers import when
`HEADROOM_OFFLINE` is enabled.
- Skip the network fallback and use estimation after an offline cache
miss.
- Add regression coverage for cold-cache no-egress behavior, explicit
offline-flag overrides, and warm-cache loading.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
22 passed in 1.09s
2 files already formatted
All checks passed!
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Linux x86_64, Python 3.13.15, `uv` 0.12.15, Transformers
5.x, empty temporary `HF_HOME`, and a loopback HTTP server used as
`HF_ENDPOINT`; no provider or HuggingFace credentials.
- Exact command / steps: ran `env -u HF_HUB_OFFLINE -u
TRANSFORMERS_OFFLINE PYTHONPATH=. uv run --no-project --with
'transformers>=5.5,<6' --with opentelemetry-api --with pydantic --with
tiktoken --with click --with rich --with pyyaml --with tomlkit python -`
with a script that started the loopback server, set `HF_HOME` to an
empty temporary directory, set `HEADROOM_OFFLINE=1`, called
`_load_tokenizer("Qwen/Qwen2.5-7B")`, and recorded requests and cache
entries.
- Observed result: `_load_tokenizer` returned `None` (estimation
fallback), the loopback HuggingFace endpoint received no requests, the
empty `HF_HOME` stayed empty, and both `HF_HUB_OFFLINE` and
`TRANSFORMERS_OFFLINE` were set to `1`:

```text
{"cache_entries": [], "hf_hub_offline": "1", "loader_http_requests": [], "result": null, "transformers_offline": "1"}
```

- Not tested: a real external `huggingface.co` endpoint, a full proxy
request, or a real warm tokenizer cache; those paths are covered by the
existing loader tests and the local proof isolates the changed behavior.

## Runtime Rollout Safety

- Rollout-managed feature(s): HuggingFace tokenizer network fallback.
- Minimum rollout channel: next patch release.
- Stable/default behavior changed: only when `HEADROOM_OFFLINE` is
enabled; online behavior remains unchanged.
- Kill switch / disable path: unset `HEADROOM_OFFLINE` and restart the
process; existing explicit `HF_HUB_OFFLINE` and `TRANSFORMERS_OFFLINE`
overrides remain respected by `apply_offline_env()`.
- Unsafe override required: none.
- Qualification impact: no provider, model, or API qualification
changes.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title

## Screenshots (if applicable)

Not applicable.

## Additional Notes

No dependencies or generated files were added. The implementation reuses
the existing offline-mode contract rather than introducing a second
configuration switch.


<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3783"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-04 13:55:13 -07:00
b0dfa33478 fix(pricing): DeepSeek V4.1-Flash peak/off-peak rate card (#3575)
## Description

DeepSeek restructured its model line-up and price card, and Headroom's
DeepSeek data was last verified 2026-06-19: the rows still carried the
pre-restructure flat rates, `deepseek-flash` was unknown, and nothing
could express the vendor's peak/off-peak split.

Source (fetched and re-verified today):
https://api-docs.deepseek.com/quick_start/pricing

| | `deepseek-flash` (V4.1-Flash) | `deepseek-v4-pro` (V4-Pro-0813) |
|---|---|---|
| Context / max output | 1M / 384K | 1M / 384K |
| Input (cache hit) | $0.003 off-peak / $0.006 peak | $0.022 / $0.044 |
| Input (cache miss) | $0.15 / $0.30 | $0.66 / $1.32 |
| Output | $0.60 / $1.20 | $1.98 / $3.96 |
| Vision | yes | no |

`deepseek-v4-flash` and `deepseek-v4-flash-vision-exp` are retired ids
still accepted on the wire — served by V4.1-Flash and billed at the
Flash price. Peak is exactly 2x off-peak; peak hours are 01:00-04:00 and
06:00-10:00 UTC, Monday-Friday.

Context note: this surfaced while working on #3036, which routes
DeepSeek traffic through the proxy and hard-codes the `deepseek-v4-*`
ids in its routing tests. That branch is not touched here, and no issue
is closed by this PR — it is a rate-card and model-data refresh.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Added `headroom/pricing/deepseek_tiers.py` as the single source of
truth: the off-peak `(cache-hit, cache-miss, output)` table for
`deepseek-flash` and `deepseek-v4-pro`, peak derived as exactly 2x, the
retired-id aliases, the Beijing peak-window predicate with the weekend
rule gated on its effective instant, and the `rates_for(model,
now=None)` seam.
- Made two cost seams tier-aware: `estimate_cost_from_tokens(...,
now=None)` prices DeepSeek flash/pro from the tier selected by the
request instant (litellm's `model_cost` holds one flat rate per model
and cannot express a tier), and `PricingRegistry.estimate_cost(...,
now=None)` prices them through its existing body from a synthesized rate
row so there is one arithmetic body rather than two.
- Gave `AnthropicProvider.estimate_cost` the same `now` passthrough and
derived its DeepSeek fallback table from the tier module.
- Derived every flat table from the tier module so no DeepSeek rate
literal is left outside it: the `deepseek_prices` rows, the
`litellm.model_cost` injection (now overriding and merging upstream
rather than filling absent keys), and the Anthropic fallback table.
Moved `LAST_UPDATED` / `SOURCE_URL` onto the module that owns the rates
and re-exported them so existing import paths keep working.
- Updated the model data: `deepseek-flash` registered with vision and
both retired ids as aliases, provider context rows, the Hugging Face
tokenizer map, the vendored LiteLLM snapshot, and the OpenCode guide's
published rate card.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_providers tests/test_pricing.py tests/test_pricing_deepseek_tiers.py \
    tests/test_pricing_from_litellm.py tests/test_pricing_litellm.py \
    tests/test_pricing_litellm_model_resolution.py tests/test_models.py \
    tests/test_huggingface_tokenizer_timeout.py tests/test_tokenizer_count_offload.py \
    tests/test_cost_pricing_warning_dedup.py -q
291 passed, 3 warnings in 9.80s

$ uv run ruff check .
All checks passed!

$ uv run mypy headroom
headroom\integrations\langchain\chat_model.py:394: error: Function "RunnableBinding" could
always be true in boolean context  [truthy-function]
Found 1 error in 1 file (checked 532 source files)

$ cargo test -p headroom-proxy --lib compression::model_limits
running 6 tests
test compression::model_limits::tests::vendored_json_parses_at_runtime ... ok
test compression::model_limits::tests::unknown_model_returns_default ... ok
test compression::model_limits::tests::sample_spec_entry_is_excluded ... ok
test compression::model_limits::tests::current_claude_models_present ... ok
test compression::model_limits::tests::current_gpt_models_present ... ok
test compression::model_limits::tests::empty_or_garbage_string_does_not_panic ... ok
test result: ok. 6 passed; 0 failed

$ uv run pytest tests/test_pricing_deepseek_tiers.py -q   # with litellm hidden
47 passed

$ uv run mypy headroom/pricing/deepseek_tiers.py headroom/pricing/registry.py \
    headroom/pricing/litellm_pricing.py headroom/providers/anthropic.py
Success: no issues found in 4 source files
```

The `mypy headroom` box is unticked on purpose: the single error above
is pre-existing on `main` in
`headroom/integrations/langchain/chat_model.py:394`, a file this PR does
not touch (see Additional Notes).

## Real Behavior Proof

- Environment: Windows 10, Python 3.13 in the repo's `.venv`, `uv`;
litellm 1.88.1 installed (whose bundled cost map predates these ids, so
the injected rows are what price them).
- Exact command / steps: pinned instants Monday 2026-08-17 02:00 UTC
(10:00 Beijing = peak) and 12:00 UTC (20:00 Beijing = off-peak), then:
`uv run python -c "from datetime import datetime,timezone; from
headroom.pricing.litellm_pricing import estimate_cost_from_tokens as e;
P=datetime(2026,8,17,2,tzinfo=timezone.utc);
O=datetime(2026,8,17,12,tzinfo=timezone.utc);
print(e('deepseek-flash',10**6,10**6,now=O),
e('deepseek-flash',10**6,10**6,now=P),
e('deepseek-flash',10**6,0,cached_tokens=10**6,now=O),
e('deepseek-v4-flash-vision-exp',10**6,10**6,now=O))"`
- Observed result: `0.75` (off-peak flash), `1.5` (peak flash, exactly
2x), `0.003` (fully cached prompt billed once at the cache-hit rate),
`0.75` (retired id prices as Flash). Also verified: 9 mutants of the new
logic (peak multiplier, weekend gate, window start, alias resolution,
cached-token handling, registry branch, provider passthrough) were each
caught by the new tests, and `tests/test_pricing_deepseek_tiers.py`
still passes with litellm absent (47 passed).
- Not tested: the full test suite (hours on this box, and much of it
cannot build a `TestClient` here because `NO_PROXY` contains a bare
`::1`, so `httpx.URLPattern` raises `InvalidURL: Invalid port: ':'` — a
pre-existing environment fault, confirmed by stashing this branch and
re-running); CNY billing; the proxy budget guard and savings ledger,
which remain flat by design (see Additional Notes).

## Runtime Rollout Safety

- Rollout-managed feature(s): DeepSeek pricing and model data consumed
by the pricing modules, providers, model registry and the Rust proxy's
context limits. No feature flags are involved.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes — DeepSeek costs, model ids and
capabilities change; the litellm `model_cost` DeepSeek rows now override
upstream; and one context-limit fallback now resolves
longest-prefix-first. All are itemised in Additional Notes.
- Kill switch / disable path: revert this branch (no flag). Operators
can override context limits without a release via
`HEADROOM_MODEL_LIMITS` or `~/.headroom/models.json`, and provider
pricing via the existing custom pricing config.
- Unsafe override required: No.
- Qualification impact: None — no schema, wire-format or session-format
change; the vendored JSON keeps its upstream shape.
- Rollback path: revert the merge commit; the only persisted artifact
change is the vendored JSON, which reverts with it.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Screenshots (if applicable)

Not applicable — no UI change.

## Additional Notes

**Behaviour changes worth a reviewer's attention**

1. Flat figures are the off-peak tier by design; only the two cost seams
are tier-aware. The proxy budget guard (`proxy/cost.py`), the savings
ledger, `ModelRegistry.estimate_cost` and
`OpenAICompatibleProvider.estimate_cost` remain flat or unpriced, so a
peak-hour DeepSeek request can be under-estimated by up to 2x there.
Making those tier-aware needs the request instant plumbed to each call
site — deliberately a separate change.
2. The litellm DeepSeek rows now override upstream instead of filling
only absent keys: upstream ships `deepseek-flash` at the *peak* rate, so
fill-if-absent would have made the flat figure depend on the installed
litellm version. The injection also merges (`{**upstream, **ours}`) so
capability flags survive.
3. DeepSeek caps moved `384_000` -> `393_216` in the litellm-shaped
tables (upstream's 384x1024 encoding of the published 384K). `ModelInfo`
keeps the published `384_000`; the litellm-shaped tables are
authoritative wherever litellm is consulted.
4. `OpenAICompatibleProvider`'s context fallback now matches the longest
prefix first, mirroring the long-standing behaviour in `OpenAIProvider`.
This fixes `deepseek-v4-flash-vision-exp` (was 1_048_576 via the shorter
`deepseek-v4` key, now 1_000_000) and also changes the window for
unlisted non-DeepSeek variants, e.g. `llama-3.1-8b-instruct` 8192 ->
128000, `mistral-large-2407` 32768 -> 128000, `starcoder2-15b` 8192 ->
16384.
5. `PricingRegistry.estimate_cost` no longer raises for a DeepSeek
flash/pro id absent from `prices` — the tier table is authoritative for
those four ids. Unknown non-DeepSeek ids still raise, and a flat row for
a tiered id is deliberately ignored.
6. Weekend off-peak applies only from 2026-08-22T16:00Z, so a back-dated
or replayed weekend request before that instant keeps the weekday
windows rather than being silently repriced. `now` exists only on
`AnthropicProvider.estimate_cost`; every other provider reads the wall
clock for live traffic.
7. CNY billing is not modelled. The zh-cn page lists the same tiers in
CNY, but the two lists are not one FX rate apart; supporting CNY-billed
accounts needs a switch over two published lists, not a baked constant.

**N/A / notes on checklist items**

- `Type checking passes (mypy headroom)` is unticked for the
pre-existing error above. `git diff --name-only origin/main..HEAD`
contains no `integrations/langchain` path, so this PR does not introduce
or touch it; `mypy` is clean on all four changed modules.
- Out of scope: the stale `deepseek-v3` / `deepseek-coder` registry rows
and the old `deepseek-chat` / `deepseek-reasoner` / `deepseek-v3.2`
context rows (the vendor no longer lists them; pruning them is a
behavioural change of its own).
- The design and implementation plan for this change live in a separate
workspace, so the decisions they record are inlined above rather than
linked.

---------

Co-authored-by: Zenjibad <316719033+Zenjibad@users.noreply.github.com>
Co-authored-by: DSH Agent <agent@local>
2026-09-24 11:31:44 -05:00
Tejas ChopraandClaude Opus 5 34da21e89e fix(tokenizers): disable trust_remote_code and allowlist HF tokenizer repos (#3723)
## A-1 (S1, remote code execution): HF tokenizer loader ran caller-named
repositories

`headroom/tokenizers/huggingface.py` called
`AutoTokenizer.from_pretrained(...,
trust_remote_code=True)` on an identifier that traces back to a proxied
request body.
`get_tokenizer_name()` ended in *"assume the model name is the tokenizer
name"*, so an
unrecognised `model` field was handed straight to the loader, which
resolves it against the
HuggingFace Hub and — with `trust_remote_code=True` — executes whatever
Python that
repository ships. Publish a repo, name it in a request body, get code
execution in the proxy
process. Nothing constrained the identifier.

### The fix

1. **`trust_remote_code=False` on every path**, passed explicitly rather
than leaning on a
   library default upstream can change.
2. **`ALLOWED_TOKENIZER_REPOS`**, derived from the mappings the product
actually ships
(`MODEL_TO_TOKENIZER.values()`, so adding a family allows it and nothing
drifts in).
`_load_tokenizer` checks it *before* importing transformers: an
unallowlisted name is
refused outright and never becomes a Hub lookup. It fails closed to
character estimation
rather than silently substituting a different vocabulary under the
caller's name.
3. **`get_tokenizer_name()` fails closed.** The fallthrough now returns
`DEFAULT_TOKENIZER`
instead of echoing the caller's string. A full repo id is accepted only
when it is on the
allowlist, and then the value returned is *our* canonical spelling,
never the caller's.

Two layers on purpose: the resolver is the normal path, the loader is
the chokepoint —
`_load_tokenizer` is importable and callable with an arbitrary string.

`benchmarks/comprehensive_eval.py` also passed `trust_remote_code=True`
to
`datasets.load_dataset`. The brief only named the two tokenizer call
sites; that third one is
fixed here too, and the tree-wide test below now holds the line for all
of them.

### Operator escape hatch

`HEADROOM_HF_TOKENIZER_ALLOWLIST` (comma-separated repo ids) lets an
operator admit private
or self-hosted tokenizer repositories. It is deliberately
environment-only — server-side
configuration, never anything a client can reach through a request body
— and repos added
there are still loaded with `trust_remote_code=False`.

### Default change a customer could depend on

An unrecognised model previously resolved to *itself* as a tokenizer
name. In practice a
made-up id failed to resolve on the Hub and counting fell back to
estimation, so most
deployments see no change. Two cases do change:

- A caller who relied on passing an **arbitrary HF repo id** not in
`MODEL_TO_TOKENIZER`
(e.g. a private fine-tune) and getting exact counts now gets the default
tokenizer
  instead. Remedy: add the repo to `HEADROOM_HF_TOKENIZER_ALLOWLIST`.
- Unknown models now resolve to `DEFAULT_TOKENIZER` (Llama 3), so a
deployment with that
vocabulary in its HF cache gets a real BPE count where it previously got
estimation.

Repo ids that already resolved correctly are unchanged. Alias and prefix
matching run first
and are untouched, so existing precedence — including
`mistralai/Mixtral-8x7B-v0.1` hitting
the `mistral` prefix — resolves exactly as before; there is an explicit
test pinning that so
a future reorder has to acknowledge it.

### Tests

- `tests/test_huggingface_tokenizer_allowlist.py` — a crafted `model`
never reaches
`from_pretrained`; every resolution lands inside the shipped set; the
loader refuses an
unallowlisted repo without touching the Hub; `trust_remote_code` is
`False` on both the
cache-only and bounded-network calls; every shipped alias still resolves
to its mapping;
the operator env hatch admits only what it names; and, since matching is
case-insensitive, the string handed to the Hub is the allowlist's own
spelling
  rather than the caller's variant.
- `tests/test_no_trust_remote_code.py` — tree-wide scan over every
git-tracked file
  asserting every `trust_remote_code` assignment is `False`.
- `tests/test_huggingface_tokenizer_timeout.py` — the existing GH #1701
timeout tests used
placeholder repo ids (`some/model`, `slow/model`, `offline/model`) which
the allowlist now
rejects before the timeout logic runs, short-circuiting what those tests
exist to prove.
Switched to real shipped repo ids so they still exercise the
bounded-load path.

### Fail-before / pass-after evidence

Method: commit, `git checkout origin/main --
headroom/tokenizers/huggingface.py
benchmarks/comprehensive_eval.py` (source reverted, new tests kept),
run, then
`git checkout HEAD -- ...` to restore.

**Before (source reverted, tests kept) — 10 failed, 4 passed:**

```
tests/test_huggingface_tokenizer_allowlist.py FFFF..F..FFF               [ 85%]
tests/test_no_trust_remote_code.py FF                                    [100%]

=================================== FAILURES ===================================
E   AssertionError: attacker-named repository was handed to from_pretrained
    assert 'attacker-controlled/headroom-rce-poc' not in ['attacker-controlled/headroom-rce-poc']

E   AssertionError: 'attacker-controlled/headroom-rce-poc' escaped the allowlist
    assert 'attacker-controlled/headroom-rce-poc' in frozenset({'01-ai/Yi-1.5-6B', ...})
     +  where 'attacker-controlled/headroom-rce-poc' = get_tokenizer_name('attacker-controlled/headroom-rce-poc')

E   AttributeError: module 'headroom.tokenizers.huggingface' has no attribute 'DEFAULT_TOKENIZER'

E   AssertionError: trust_remote_code must be False everywhere — it executes repository-supplied Python in this process:
      benchmarks/comprehensive_eval.py:44: trust_remote_code=True,
      headroom/tokenizers/huggingface.py:162: trust_remote_code=True,
      headroom/tokenizers/huggingface.py:184: trust_remote_code=True,

E   AssertionError: assert {'True'} == {'False'}

=========================== short test summary info ============================
FAILED tests/test_huggingface_tokenizer_allowlist.py::test_crafted_model_never_reaches_the_loader
FAILED tests/test_huggingface_tokenizer_allowlist.py::test_crafted_model_resolves_to_a_shipped_tokenizer
FAILED tests/test_huggingface_tokenizer_allowlist.py::test_crafted_model_resolves_to_the_declared_default
FAILED tests/test_huggingface_tokenizer_allowlist.py::test_loader_fetches_the_allowlists_own_spelling
FAILED tests/test_huggingface_tokenizer_allowlist.py::test_remote_code_disabled_on_cache_and_network_paths
FAILED tests/test_huggingface_tokenizer_allowlist.py::test_repo_id_matching_is_case_insensitive_but_returns_our_spelling
FAILED tests/test_huggingface_tokenizer_allowlist.py::test_operator_env_allowlist_admits_a_private_repo
FAILED tests/test_huggingface_tokenizer_allowlist.py::test_empty_operator_env_admits_nothing_extra
FAILED tests/test_no_trust_remote_code.py::test_remote_code_is_never_enabled_anywhere_in_the_tree
FAILED tests/test_no_trust_remote_code.py::test_the_tokenizer_loader_sets_the_flag_explicitly
========================= 10 failed, 4 passed in 1.47s =========================
```

**After (source restored):**

```
tests/test_huggingface_tokenizer_allowlist.py ............               [ 54%]
tests/test_no_trust_remote_code.py ..                                    [ 63%]
tests/test_huggingface_tokenizer_timeout.py ........                     [100%]

============================== 22 passed in 2.73s ==============================
```

Wider tokenizer suites (`test_tokenizers.py`, `test_tokenizer.py`,
`test_tokenizers/`,
`test_tokenizer_encoding_resolution.py`,
`test_tokenizer_selection_coverage.py`,
`test_provider_tokenizer_one_ruler.py`,
`test_tokenizer_count_offload.py`,
`test_strands_tokenizer.py`,
`test_compress_route_tokenizer_by_model.py`,
`test_evals_cjk_tokenization.py`): **166 passed, 14 skipped**.

Lint: `ruff 0.16.7 check .` and `format --check .` both clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-23 20:20:56 +02:00
Rod BoevandJerrettDavis 551f473e04 fix(proxy): accept Codex websocket before upstream retries (#2203)
## Description

Codex Desktop could abandon Headroom's local `/v1/responses` WebSocket
handshake before Headroom's upstream retry strategy had a chance to
recover. The ChatGPT-auth path waited for an upstream opening handshake
with a minimum 30-second timeout before sending the local 101, while the
reported Codex Desktop handshake expired after about 34 seconds.

This change accepts validated ChatGPT-auth Codex WebSockets before
opening the upstream connection, then keeps the existing upstream
retries and HTTP fallback behind the established local session. API-key
sessions retain connect-before-accept behavior so upstream `x-codex-*`
headers can still be attached to their client-facing 101. The change is
scoped to the pre-101 timing failure and does not address the separate
large-context streaming investigation in #1944.

Closes #2184

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Accept ChatGPT-auth Codex WebSocket clients before the upstream
connect and retry loop.
- Preserve API-key connect-before-accept ordering and upstream
`x-codex-*` handshake-header forwarding.
- Keep upstream retry, relay, usage-state refresh, and WebSocket-to-HTTP
fallback behavior after the local 101.
- Add a deterministic regression that blocks the first upstream opening
handshake and proves the local acceptance deadline is independent of it.

## Testing

- [x] Unit tests pass (`uv run pytest
tests/test_openai_codex_ws_lifecycle.py -q`)
- [x] Linting passes (`uv run ruff check
headroom/proxy/handlers/openai.py
tests/test_openai_codex_ws_lifecycle.py`)
- [ ] Type checking passes (`uv run mypy headroom`)
- [x] New tests added for new functionality when applicable
- [ ] Manual testing performed

### Test Output

```text
uv run pytest tests/test_openai_codex_ws_lifecycle.py -q
28 passed in 2.02s
uv run ruff check headroom/proxy/handlers/openai.py tests/test_openai_codex_ws_lifecycle.py
All checks passed!
uv run ruff format headroom/proxy/handlers/openai.py tests/test_openai_codex_ws_lifecycle.py --check
2 files already formatted
```

## Real Behavior Proof

- Environment: Python 3.12, synced development worktree, local fake
Codex client and upstream WebSocket, no live provider
- Exact command / steps: Run `uv run pytest
tests/test_openai_codex_ws_lifecycle.py::test_chatgpt_ws_accepts_before_stalled_upstream_connect
-q`; the fake upstream blocks its first opening handshake while the
client enforces a bounded local-accept deadline.
- Observed result: `1 passed in 0.46s`; the ChatGPT-auth client receives
its local 101 before the blocked upstream connect is released, and the
handler continues into its existing upstream recovery path.
- Not tested: live Codex Desktop pre-turn compaction against ChatGPT
subscription infrastructure

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

`CHANGELOG.md` is unchanged because the release pipeline generates it
from conventional commits. No user documentation changes are required;
the handler comments and ordered-flow docstring are updated with the
auth-mode-specific behavior. The broader #1944 large-context disconnect
surface remains out of scope.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-15 18:18:06 +00:00
yiihaoandJerrettDavis 0c7087539d fix: add DeepSeek V4/V3.2/R1 tokenizer mappings and context limits (#912)
The tokenizer registry routed deepseek-v4-pro, deepseek-v4-flash,
deepseek-chat, deepseek-reasoner, and other modern DeepSeek models
to the 2023-era deepseek-llm-7b-base tokenizer via prefix fallback.

This caused token counts off by 30-50%, broken context-limit detection
(V4-Pro supports 1M but got 32K), and inaccurate savings reports.

## Fix

3 files, +43/-2:

- **huggingface.py**: 16 new MODEL_TO_TOKENIZER entries with verified
  HuggingFace IDs (deepseek-ai/DeepSeek-V4-Pro, V4-Flash, V3.2,
  V3-0324, R1, R1-0528, Reasoner, Chat, Coder-V2, etc.)

- **openai_compatible.py**: 17 new _DEFAULT_CONTEXT_LIMITS entries
  (V4-Pro/Flash -> 1M, R1/Reasoner -> 131K, V3 -> 128K, etc.)

- **openai.py**: 8 new _CONTEXT_LIMITS entries for LiteLLM-fallback.

Existing mappings untouched (backward compatible).

## Real behavior proof

- **Setup**: Windows 11, Python 3.13.14, headroom-ai 0.2.15 wheel +
  source checkout at v0.24.0. No Rust extension built (headroom._core
  unavailable). Touched files are at parity with v0.24.0.

- **Steps after patch**:
  ```
  python3 -c "
  from headroom.tokenizers.huggingface import get_tokenizer_name
for m in
['deepseek-v4-pro','deepseek-chat','deepseek-reasoner','deepseek-v4-flash']:
      print(f'{m} -> {get_tokenizer_name(m)}')
  from headroom.tokenizers.registry import get_tokenizer
  for m in ['deepseek-v4-pro','deepseek-chat','deepseek-reasoner']:
      print(f'{m}: {get_tokenizer(m)}')
  "
  ```

- **Observed result**:
  ```
  deepseek-v4-pro  -> deepseek-ai/DeepSeek-V4-Pro
  deepseek-v4-flash -> deepseek-ai/DeepSeek-V4-Flash
  deepseek-chat    -> deepseek-ai/DeepSeek-V3
  deepseek-reasoner -> deepseek-ai/DeepSeek-R1
  ```
  Previously ALL resolved to deepseek-ai/deepseek-llm-7b-base.
  TokenizerRegistry routes correctly. Context limits verified
  (1M / 131K / 128K). compress() import smoke-tested OK.

- **Not tested**: full proxy e2e with a live DeepSeek API key
  (no available key). HuggingFace AutoTokenizer download confirmed
  for V4-Pro/V3/R1 but produced GBK decode errors from hf_hub on
  this zh-CN Windows locale during config fetch -- a separate
  huggingface_hub issue unrelated to this change.


<!-- headroom-maintainer-template-completion:start -->

## Description

This PR prepares `fix: add DeepSeek V4/V3.2/R1 tokenizer mappings and
context limits` for review by documenting the intended change,
validation evidence, and remaining merge-readiness context.

Linked issues: None declared.

## Type of Change

- [x] Bug fix
- [ ] New feature
- [ ] Documentation
- [ ] Refactor
- [ ] Tests only

## Changes Made

- Commit: fix: add DeepSeek V4/V3.2/R1 tokenizer mappings and context
limits
- Touches `headroom/providers/openai.py`
- Touches `headroom/providers/openai_compatible.py`
- Touches `headroom/tokenizers/huggingface.py`

## Testing

- [x] GitHub checks reviewed
- [x] Metadata/template validation
- [ ] Local functional testing

### Test Output

```text
gh pr view 912 --repo chopratejas/headroom --json statusCheckRollup
- PR Governance / label: SUCCESS
- external / GitGuardian Security Checks: SUCCESS
```

## Real Behavior Proof

- Environment: GitHub PR metadata and checks for `chopratejas/headroom`
PR #912.
- Exact command / steps: Reviewed PR title, commits, changed files,
linked issues, labels, and check rollup; appended this maintainer
template completion block without replacing the author's original
description.
- Observed result: PR body now contains all required governance
sections, checked readiness fields, and a non-placeholder validation
evidence block.
- Not tested: This pass updated PR metadata only; code validation
remains represented by the linked GitHub checks and any author-provided
evidence above.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

<!-- headroom-maintainer-template-completion:end -->

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-15 09:24:44 -05:00
Abhay SinghandTejas Chopra e0232df9b4 fix(tokenizers): resolve HF tokenizer names by most-specific prefix (#2096)
## Description

`get_tokenizer_name` can pick the wrong tokenizer for a versioned model,
which silently produces wrong token counts.

For a model that isn't a literal key in `MODEL_TO_TOKENIZER`, it falls
back to prefix matching:

```python
for key, value in MODEL_TO_TOKENIZER.items():
    if model_lower.startswith(key):
        return value
```

That returns the first key the model merely *starts with*, in
dict-insertion order. The table lists short family keys before their
more-specific siblings — `"qwen"` (→ `Qwen/Qwen-7B`) appears before
`"qwen2"`/`"qwen2-7b"`/`"qwen2.5"`. So
`get_tokenizer_name("qwen2-7b-instruct")` matches `"qwen"` first and
returns the **Qwen1** tokenizer, not Qwen2. Qwen1 and Qwen2 have
different vocabularies, so every `count_text`/`count_messages` for that
model is off. `qwen2.5-*` and `deepseek-v2.x` are mis-resolved the same
way.

The sibling tiktoken resolver already documents and guards this exact
pitfall — `get_encoding_for_model` uses an explicit most-specific-first
prefix list with a comment that scanning "for the first key that merely
starts with the prefix is order-dependent and wrong." The HuggingFace
resolver is the one that still scans insertion order.

## Fix

Match the **longest** (most-specific) prefix instead of the first in
insertion order:

```python
for key in sorted(MODEL_TO_TOKENIZER, key=len, reverse=True):
    if model_lower.startswith(key):
        return MODEL_TO_TOKENIZER[key]
```

Direct-key lookups and the shorter-family fallback (e.g. `deepseek-chat`
→ `deepseek-ai/deepseek-llm-7b-base`) are unchanged.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/tokenizers/huggingface.py`: `get_tokenizer_name` prefix
matching now iterates keys longest-first and returns the most-specific
match.
- `tests/test_huggingface_tokenizer_timeout.py`: add
`test_get_tokenizer_name_prefers_most_specific_prefix`.
- `CHANGELOG.md`: Bug Fixes entry.

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uvx ruff@0.15.17 check headroom/tokenizers/huggingface.py tests/test_huggingface_tokenizer_timeout.py
All checks passed!
$ python -m py_compile headroom/tokenizers/huggingface.py tests/test_huggingface_tokenizer_timeout.py
OK
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and a full `pytest`
gets OOM-killed on this box, so I verified against the real key table
with a dependency-free script that parses `MODEL_TO_TOKENIZER` out of
the source and runs both the old (insertion-order) and new
(longest-first) scans, then left the full pytest to CI.
- Exact command / steps: resolved `qwen2-7b-instruct`, `qwen2.5-turbo`,
and `deepseek-v2.5` under both strategies, plus `deepseek-chat` as a
regression guard.
- Observed result: old scan returns `Qwen/Qwen-7B` (Qwen1) for both
qwen2 models and `deepseek-ai/deepseek-llm-7b-base` (v1) for
`deepseek-v2.5`; new scan returns `Qwen/Qwen2-7B`, `Qwen/Qwen2.5-7B`,
and `deepseek-ai/DeepSeek-V2` respectively. `deepseek-chat` resolves
identically under both (`deepseek-ai/deepseek-llm-7b-base`), so the
existing timeout test's model is unaffected.
- Not tested: loading the actual HuggingFace tokenizers
(network/`transformers`); full local `pytest` deferred to CI (OOM, per
above).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

The "unit tests pass locally" and "type checking" boxes are unchecked
because the full suite imports the ML stack, which I can't run in this
environment; the change is a localized swap of the prefix-scan order in
a pure function, verified by the standalone proof (run against the real
key table) and the new regression test for CI. This mirrors the
same-class fix already present in the sibling tiktoken resolver.

Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
2026-07-13 10:50:44 -04:00
ParideboyandClaude Fable 5 46d5d685d9 fix(proxy): bound HF tokenizer load and offload token counting off event loop (#1738)
## Description

Fixes #1701.

On Windows, `headroom proxy --anthropic-api-url
https://api.deepseek.com/anthropic` froze: the first `/v1/messages`
request took ~610s (`optimization_latency_ms=609972`) with only
router/lifecycle markers, and afterwards the whole server was a zombie —
`/livez`, `/readyz` and `/health` hung until the process was killed.
`HEADROOM_DETECT_BACKEND=python` was already set, so this was not the
#575/#845 native-detect deadlock.

Root cause: DeepSeek model names route to the HuggingFace tokenizer
backend (`MODEL_PATTERNS` in `headroom/tokenizers/registry.py`).
`HuggingFaceTokenizer` loads lazily, so the registry's construction-time
fallback never fires; the first `count_messages` calls
`AutoTokenizer.from_pretrained(..., trust_remote_code=True)` — unbounded
network downloads/retries — and this ran **synchronously inside the
async Anthropic messages handler** (`get_tokenizer(model)` +
`tokenizer.count_messages(messages)`), outside the 30s
`_run_compression_in_executor` bound. huggingface_hub retry chains on a
restricted network easily reach ~10 minutes, blocking the entire asyncio
event loop; subsequent on-loop counting kept it pinned. tiktoken got a
bounded eager load for the same bug class long ago (#956); the HF
backend never did.

## Type of Change

- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Refactoring (no functional changes)

## Changes Made

- `headroom/tokenizers/huggingface.py`: `_load_tokenizer` now tries the
local HF cache first (`local_files_only=True`, no network), then bounds
the network load with `HEADROOM_HF_TOKENIZER_LOAD_TIMEOUT_SECS` (default
10s; `0` disables network loads) on a daemon thread. Timeouts/failures
return `None` (cached by `lru_cache`, so the hub is probed at most once
per process per tokenizer) and `count_messages` fails open to char-based
estimation via the existing `_use_fallback()` path.
- `headroom/proxy/handlers/anthropic.py`: new
`AnthropicHandlerMixin._count_tokens_offloaded(model, messages)` runs
`get_tokenizer` + `count_messages` on the compression executor bounded
by `COMPRESSION_TIMEOUT_SECONDS`, failing open to
`EstimatingTokenCounter` (downgrade logged once per model). Used in
`handle_anthropic_messages` (the issue's hot path, both count sites) and
`handle_anthropic_batch_create`; the batch path's inline
`anthropic_pipeline.apply()` is now offloaded via
`_run_compression_in_executor` (mirrors the #1612 image-compression
offload).
- `headroom/proxy/handlers/batch.py`: the two remaining inline
`openai_pipeline.apply()` calls (`handle_google_batch_create`,
`_compress_batch_jsonl`) are offloaded the same way; existing `except`
blocks keep the pass-through fail-open semantics.
- Tests: `tests/test_huggingface_tokenizer_timeout.py` (cache-first,
bounded timeout, failure caching, timeout=0, fail-open estimation),
`tests/test_tokenizer_count_offload.py` (wiring guards, runs on
`headroom-compress` worker, event loop stays responsive during slow
tokenizer work, fail-open), plus `_run_compression_in_executor` stub on
the batch test double.

## Testing

- [x] All existing tests pass
- [x] Added new tests for the changes
- [ ] Manual testing performed

```
$ python -m pytest tests/test_huggingface_tokenizer_timeout.py tests/test_tokenizer_count_offload.py tests/test_image_compression_offload.py tests/test_gemini_compression_offload.py tests/test_tokenizers tests/test_proxy_handlers_batch.py -q
50 passed

$ ruff check .          # No issues found
$ ruff format --check . # 1043 files already formatted
$ mypy headroom --ignore-missing-imports  # 0 errors
```

## Real Behavior Proof

- Environment: Windows 11 Pro (10.0.26200), Python 3.13, local checkout
of this branch with the Rust core built.
- Exact command / steps: `python -m pytest
tests/test_tokenizer_count_offload.py -q` — includes
`test_count_tokens_offloaded_keeps_loop_responsive`, which reproduces
the issue's mechanism: a tokenizer whose `count_messages` blocks
(stand-in for the unbounded `AutoTokenizer.from_pretrained` network
load) while an asyncio ticker measures event-loop liveness. Also `python
-m pytest tests/test_huggingface_tokenizer_timeout.py -q` with a
`from_pretrained` stub that sleeps 60s and
`HEADROOM_HF_TOKENIZER_LOAD_TIMEOUT_SECS=0.2`.
- Observed result: with the fix, the slow count runs on a
`headroom-compress` worker thread and the loop keeps ticking (`ticks >=
5`; inline it yields ~0 — the zombie). The 60s-hung HF load unblocks at
the 0.2s timeout, falls back to estimation, and the second call returns
instantly (failure cached, no re-probe). All 10 new tests pass.
- Not tested: live reproduction against `api.deepseek.com` from a
network where HF hub downloads stall (the reporter's exact environment);
actual HF vocab download timing on a healthy network.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 12:46:26 -05:00