mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-23 02:49:17 +08:00
feat/https-compose-caddy
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
d503c575ea |
fix: prevent HF tokenizer downloads in offline mode (#3783)
## Description `HEADROOM_OFFLINE=1` must not fall through from a cold HuggingFace tokenizer cache to a network download. This change applies the existing offline environment defaults before importing Transformers and returns to estimation after a cache miss when offline mode is active. Closes #3777 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Apply `apply_offline_env()` before the Transformers import when `HEADROOM_OFFLINE` is enabled. - Skip the network fallback and use estimation after an offline cache miss. - Add regression coverage for cold-cache no-egress behavior, explicit offline-flag overrides, and warm-cache loading. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text 22 passed in 1.09s 2 files already formatted All checks passed! Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Linux x86_64, Python 3.13.15, `uv` 0.12.15, Transformers 5.x, empty temporary `HF_HOME`, and a loopback HTTP server used as `HF_ENDPOINT`; no provider or HuggingFace credentials. - Exact command / steps: ran `env -u HF_HUB_OFFLINE -u TRANSFORMERS_OFFLINE PYTHONPATH=. uv run --no-project --with 'transformers>=5.5,<6' --with opentelemetry-api --with pydantic --with tiktoken --with click --with rich --with pyyaml --with tomlkit python -` with a script that started the loopback server, set `HF_HOME` to an empty temporary directory, set `HEADROOM_OFFLINE=1`, called `_load_tokenizer("Qwen/Qwen2.5-7B")`, and recorded requests and cache entries. - Observed result: `_load_tokenizer` returned `None` (estimation fallback), the loopback HuggingFace endpoint received no requests, the empty `HF_HOME` stayed empty, and both `HF_HUB_OFFLINE` and `TRANSFORMERS_OFFLINE` were set to `1`: ```text {"cache_entries": [], "hf_hub_offline": "1", "loader_http_requests": [], "result": null, "transformers_offline": "1"} ``` - Not tested: a real external `huggingface.co` endpoint, a full proxy request, or a real warm tokenizer cache; those paths are covered by the existing loader tests and the local proof isolates the changed behavior. ## Runtime Rollout Safety - Rollout-managed feature(s): HuggingFace tokenizer network fallback. - Minimum rollout channel: next patch release. - Stable/default behavior changed: only when `HEADROOM_OFFLINE` is enabled; online behavior remains unchanged. - Kill switch / disable path: unset `HEADROOM_OFFLINE` and restart the process; existing explicit `HF_HUB_OFFLINE` and `TRANSFORMERS_OFFLINE` overrides remain respected by `apply_offline_env()`. - Unsafe override required: none. - Qualification impact: no provider, model, or API qualification changes. - Rollback path: revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title ## Screenshots (if applicable) Not applicable. ## Additional Notes No dependencies or generated files were added. The implementation reuses the existing offline-mode contract rather than introducing a second configuration switch. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3783" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
b0dfa33478 |
fix(pricing): DeepSeek V4.1-Flash peak/off-peak rate card (#3575)
## Description DeepSeek restructured its model line-up and price card, and Headroom's DeepSeek data was last verified 2026-06-19: the rows still carried the pre-restructure flat rates, `deepseek-flash` was unknown, and nothing could express the vendor's peak/off-peak split. Source (fetched and re-verified today): https://api-docs.deepseek.com/quick_start/pricing | | `deepseek-flash` (V4.1-Flash) | `deepseek-v4-pro` (V4-Pro-0813) | |---|---|---| | Context / max output | 1M / 384K | 1M / 384K | | Input (cache hit) | $0.003 off-peak / $0.006 peak | $0.022 / $0.044 | | Input (cache miss) | $0.15 / $0.30 | $0.66 / $1.32 | | Output | $0.60 / $1.20 | $1.98 / $3.96 | | Vision | yes | no | `deepseek-v4-flash` and `deepseek-v4-flash-vision-exp` are retired ids still accepted on the wire — served by V4.1-Flash and billed at the Flash price. Peak is exactly 2x off-peak; peak hours are 01:00-04:00 and 06:00-10:00 UTC, Monday-Friday. Context note: this surfaced while working on #3036, which routes DeepSeek traffic through the proxy and hard-codes the `deepseek-v4-*` ids in its routing tests. That branch is not touched here, and no issue is closed by this PR — it is a rate-card and model-data refresh. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Added `headroom/pricing/deepseek_tiers.py` as the single source of truth: the off-peak `(cache-hit, cache-miss, output)` table for `deepseek-flash` and `deepseek-v4-pro`, peak derived as exactly 2x, the retired-id aliases, the Beijing peak-window predicate with the weekend rule gated on its effective instant, and the `rates_for(model, now=None)` seam. - Made two cost seams tier-aware: `estimate_cost_from_tokens(..., now=None)` prices DeepSeek flash/pro from the tier selected by the request instant (litellm's `model_cost` holds one flat rate per model and cannot express a tier), and `PricingRegistry.estimate_cost(..., now=None)` prices them through its existing body from a synthesized rate row so there is one arithmetic body rather than two. - Gave `AnthropicProvider.estimate_cost` the same `now` passthrough and derived its DeepSeek fallback table from the tier module. - Derived every flat table from the tier module so no DeepSeek rate literal is left outside it: the `deepseek_prices` rows, the `litellm.model_cost` injection (now overriding and merging upstream rather than filling absent keys), and the Anthropic fallback table. Moved `LAST_UPDATED` / `SOURCE_URL` onto the module that owns the rates and re-exported them so existing import paths keep working. - Updated the model data: `deepseek-flash` registered with vision and both retired ids as aliases, provider context rows, the Hugging Face tokenizer map, the vendored LiteLLM snapshot, and the OpenCode guide's published rate card. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_providers tests/test_pricing.py tests/test_pricing_deepseek_tiers.py \ tests/test_pricing_from_litellm.py tests/test_pricing_litellm.py \ tests/test_pricing_litellm_model_resolution.py tests/test_models.py \ tests/test_huggingface_tokenizer_timeout.py tests/test_tokenizer_count_offload.py \ tests/test_cost_pricing_warning_dedup.py -q 291 passed, 3 warnings in 9.80s $ uv run ruff check . All checks passed! $ uv run mypy headroom headroom\integrations\langchain\chat_model.py:394: error: Function "RunnableBinding" could always be true in boolean context [truthy-function] Found 1 error in 1 file (checked 532 source files) $ cargo test -p headroom-proxy --lib compression::model_limits running 6 tests test compression::model_limits::tests::vendored_json_parses_at_runtime ... ok test compression::model_limits::tests::unknown_model_returns_default ... ok test compression::model_limits::tests::sample_spec_entry_is_excluded ... ok test compression::model_limits::tests::current_claude_models_present ... ok test compression::model_limits::tests::current_gpt_models_present ... ok test compression::model_limits::tests::empty_or_garbage_string_does_not_panic ... ok test result: ok. 6 passed; 0 failed $ uv run pytest tests/test_pricing_deepseek_tiers.py -q # with litellm hidden 47 passed $ uv run mypy headroom/pricing/deepseek_tiers.py headroom/pricing/registry.py \ headroom/pricing/litellm_pricing.py headroom/providers/anthropic.py Success: no issues found in 4 source files ``` The `mypy headroom` box is unticked on purpose: the single error above is pre-existing on `main` in `headroom/integrations/langchain/chat_model.py:394`, a file this PR does not touch (see Additional Notes). ## Real Behavior Proof - Environment: Windows 10, Python 3.13 in the repo's `.venv`, `uv`; litellm 1.88.1 installed (whose bundled cost map predates these ids, so the injected rows are what price them). - Exact command / steps: pinned instants Monday 2026-08-17 02:00 UTC (10:00 Beijing = peak) and 12:00 UTC (20:00 Beijing = off-peak), then: `uv run python -c "from datetime import datetime,timezone; from headroom.pricing.litellm_pricing import estimate_cost_from_tokens as e; P=datetime(2026,8,17,2,tzinfo=timezone.utc); O=datetime(2026,8,17,12,tzinfo=timezone.utc); print(e('deepseek-flash',10**6,10**6,now=O), e('deepseek-flash',10**6,10**6,now=P), e('deepseek-flash',10**6,0,cached_tokens=10**6,now=O), e('deepseek-v4-flash-vision-exp',10**6,10**6,now=O))"` - Observed result: `0.75` (off-peak flash), `1.5` (peak flash, exactly 2x), `0.003` (fully cached prompt billed once at the cache-hit rate), `0.75` (retired id prices as Flash). Also verified: 9 mutants of the new logic (peak multiplier, weekend gate, window start, alias resolution, cached-token handling, registry branch, provider passthrough) were each caught by the new tests, and `tests/test_pricing_deepseek_tiers.py` still passes with litellm absent (47 passed). - Not tested: the full test suite (hours on this box, and much of it cannot build a `TestClient` here because `NO_PROXY` contains a bare `::1`, so `httpx.URLPattern` raises `InvalidURL: Invalid port: ':'` — a pre-existing environment fault, confirmed by stashing this branch and re-running); CNY billing; the proxy budget guard and savings ledger, which remain flat by design (see Additional Notes). ## Runtime Rollout Safety - Rollout-managed feature(s): DeepSeek pricing and model data consumed by the pricing modules, providers, model registry and the Rust proxy's context limits. No feature flags are involved. - Minimum rollout channel: Stable. - Stable/default behavior changed: Yes — DeepSeek costs, model ids and capabilities change; the litellm `model_cost` DeepSeek rows now override upstream; and one context-limit fallback now resolves longest-prefix-first. All are itemised in Additional Notes. - Kill switch / disable path: revert this branch (no flag). Operators can override context limits without a release via `HEADROOM_MODEL_LIMITS` or `~/.headroom/models.json`, and provider pricing via the existing custom pricing config. - Unsafe override required: No. - Qualification impact: None — no schema, wire-format or session-format change; the vendored JSON keeps its upstream shape. - Rollback path: revert the merge commit; the only persisted artifact change is the vendored JSON, which reverts with it. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Screenshots (if applicable) Not applicable — no UI change. ## Additional Notes **Behaviour changes worth a reviewer's attention** 1. Flat figures are the off-peak tier by design; only the two cost seams are tier-aware. The proxy budget guard (`proxy/cost.py`), the savings ledger, `ModelRegistry.estimate_cost` and `OpenAICompatibleProvider.estimate_cost` remain flat or unpriced, so a peak-hour DeepSeek request can be under-estimated by up to 2x there. Making those tier-aware needs the request instant plumbed to each call site — deliberately a separate change. 2. The litellm DeepSeek rows now override upstream instead of filling only absent keys: upstream ships `deepseek-flash` at the *peak* rate, so fill-if-absent would have made the flat figure depend on the installed litellm version. The injection also merges (`{**upstream, **ours}`) so capability flags survive. 3. DeepSeek caps moved `384_000` -> `393_216` in the litellm-shaped tables (upstream's 384x1024 encoding of the published 384K). `ModelInfo` keeps the published `384_000`; the litellm-shaped tables are authoritative wherever litellm is consulted. 4. `OpenAICompatibleProvider`'s context fallback now matches the longest prefix first, mirroring the long-standing behaviour in `OpenAIProvider`. This fixes `deepseek-v4-flash-vision-exp` (was 1_048_576 via the shorter `deepseek-v4` key, now 1_000_000) and also changes the window for unlisted non-DeepSeek variants, e.g. `llama-3.1-8b-instruct` 8192 -> 128000, `mistral-large-2407` 32768 -> 128000, `starcoder2-15b` 8192 -> 16384. 5. `PricingRegistry.estimate_cost` no longer raises for a DeepSeek flash/pro id absent from `prices` — the tier table is authoritative for those four ids. Unknown non-DeepSeek ids still raise, and a flat row for a tiered id is deliberately ignored. 6. Weekend off-peak applies only from 2026-08-22T16:00Z, so a back-dated or replayed weekend request before that instant keeps the weekday windows rather than being silently repriced. `now` exists only on `AnthropicProvider.estimate_cost`; every other provider reads the wall clock for live traffic. 7. CNY billing is not modelled. The zh-cn page lists the same tiers in CNY, but the two lists are not one FX rate apart; supporting CNY-billed accounts needs a switch over two published lists, not a baked constant. **N/A / notes on checklist items** - `Type checking passes (mypy headroom)` is unticked for the pre-existing error above. `git diff --name-only origin/main..HEAD` contains no `integrations/langchain` path, so this PR does not introduce or touch it; `mypy` is clean on all four changed modules. - Out of scope: the stale `deepseek-v3` / `deepseek-coder` registry rows and the old `deepseek-chat` / `deepseek-reasoner` / `deepseek-v3.2` context rows (the vendor no longer lists them; pruning them is a behavioural change of its own). - The design and implementation plan for this change live in a separate workspace, so the decisions they record are inlined above rather than linked. --------- Co-authored-by: Zenjibad <316719033+Zenjibad@users.noreply.github.com> Co-authored-by: DSH Agent <agent@local> |
||
|
|
34da21e89e |
fix(tokenizers): disable trust_remote_code and allowlist HF tokenizer repos (#3723)
## A-1 (S1, remote code execution): HF tokenizer loader ran caller-named repositories `headroom/tokenizers/huggingface.py` called `AutoTokenizer.from_pretrained(..., trust_remote_code=True)` on an identifier that traces back to a proxied request body. `get_tokenizer_name()` ended in *"assume the model name is the tokenizer name"*, so an unrecognised `model` field was handed straight to the loader, which resolves it against the HuggingFace Hub and — with `trust_remote_code=True` — executes whatever Python that repository ships. Publish a repo, name it in a request body, get code execution in the proxy process. Nothing constrained the identifier. ### The fix 1. **`trust_remote_code=False` on every path**, passed explicitly rather than leaning on a library default upstream can change. 2. **`ALLOWED_TOKENIZER_REPOS`**, derived from the mappings the product actually ships (`MODEL_TO_TOKENIZER.values()`, so adding a family allows it and nothing drifts in). `_load_tokenizer` checks it *before* importing transformers: an unallowlisted name is refused outright and never becomes a Hub lookup. It fails closed to character estimation rather than silently substituting a different vocabulary under the caller's name. 3. **`get_tokenizer_name()` fails closed.** The fallthrough now returns `DEFAULT_TOKENIZER` instead of echoing the caller's string. A full repo id is accepted only when it is on the allowlist, and then the value returned is *our* canonical spelling, never the caller's. Two layers on purpose: the resolver is the normal path, the loader is the chokepoint — `_load_tokenizer` is importable and callable with an arbitrary string. `benchmarks/comprehensive_eval.py` also passed `trust_remote_code=True` to `datasets.load_dataset`. The brief only named the two tokenizer call sites; that third one is fixed here too, and the tree-wide test below now holds the line for all of them. ### Operator escape hatch `HEADROOM_HF_TOKENIZER_ALLOWLIST` (comma-separated repo ids) lets an operator admit private or self-hosted tokenizer repositories. It is deliberately environment-only — server-side configuration, never anything a client can reach through a request body — and repos added there are still loaded with `trust_remote_code=False`. ### Default change a customer could depend on An unrecognised model previously resolved to *itself* as a tokenizer name. In practice a made-up id failed to resolve on the Hub and counting fell back to estimation, so most deployments see no change. Two cases do change: - A caller who relied on passing an **arbitrary HF repo id** not in `MODEL_TO_TOKENIZER` (e.g. a private fine-tune) and getting exact counts now gets the default tokenizer instead. Remedy: add the repo to `HEADROOM_HF_TOKENIZER_ALLOWLIST`. - Unknown models now resolve to `DEFAULT_TOKENIZER` (Llama 3), so a deployment with that vocabulary in its HF cache gets a real BPE count where it previously got estimation. Repo ids that already resolved correctly are unchanged. Alias and prefix matching run first and are untouched, so existing precedence — including `mistralai/Mixtral-8x7B-v0.1` hitting the `mistral` prefix — resolves exactly as before; there is an explicit test pinning that so a future reorder has to acknowledge it. ### Tests - `tests/test_huggingface_tokenizer_allowlist.py` — a crafted `model` never reaches `from_pretrained`; every resolution lands inside the shipped set; the loader refuses an unallowlisted repo without touching the Hub; `trust_remote_code` is `False` on both the cache-only and bounded-network calls; every shipped alias still resolves to its mapping; the operator env hatch admits only what it names; and, since matching is case-insensitive, the string handed to the Hub is the allowlist's own spelling rather than the caller's variant. - `tests/test_no_trust_remote_code.py` — tree-wide scan over every git-tracked file asserting every `trust_remote_code` assignment is `False`. - `tests/test_huggingface_tokenizer_timeout.py` — the existing GH #1701 timeout tests used placeholder repo ids (`some/model`, `slow/model`, `offline/model`) which the allowlist now rejects before the timeout logic runs, short-circuiting what those tests exist to prove. Switched to real shipped repo ids so they still exercise the bounded-load path. ### Fail-before / pass-after evidence Method: commit, `git checkout origin/main -- headroom/tokenizers/huggingface.py benchmarks/comprehensive_eval.py` (source reverted, new tests kept), run, then `git checkout HEAD -- ...` to restore. **Before (source reverted, tests kept) — 10 failed, 4 passed:** ``` tests/test_huggingface_tokenizer_allowlist.py FFFF..F..FFF [ 85%] tests/test_no_trust_remote_code.py FF [100%] =================================== FAILURES =================================== E AssertionError: attacker-named repository was handed to from_pretrained assert 'attacker-controlled/headroom-rce-poc' not in ['attacker-controlled/headroom-rce-poc'] E AssertionError: 'attacker-controlled/headroom-rce-poc' escaped the allowlist assert 'attacker-controlled/headroom-rce-poc' in frozenset({'01-ai/Yi-1.5-6B', ...}) + where 'attacker-controlled/headroom-rce-poc' = get_tokenizer_name('attacker-controlled/headroom-rce-poc') E AttributeError: module 'headroom.tokenizers.huggingface' has no attribute 'DEFAULT_TOKENIZER' E AssertionError: trust_remote_code must be False everywhere — it executes repository-supplied Python in this process: benchmarks/comprehensive_eval.py:44: trust_remote_code=True, headroom/tokenizers/huggingface.py:162: trust_remote_code=True, headroom/tokenizers/huggingface.py:184: trust_remote_code=True, E AssertionError: assert {'True'} == {'False'} =========================== short test summary info ============================ FAILED tests/test_huggingface_tokenizer_allowlist.py::test_crafted_model_never_reaches_the_loader FAILED tests/test_huggingface_tokenizer_allowlist.py::test_crafted_model_resolves_to_a_shipped_tokenizer FAILED tests/test_huggingface_tokenizer_allowlist.py::test_crafted_model_resolves_to_the_declared_default FAILED tests/test_huggingface_tokenizer_allowlist.py::test_loader_fetches_the_allowlists_own_spelling FAILED tests/test_huggingface_tokenizer_allowlist.py::test_remote_code_disabled_on_cache_and_network_paths FAILED tests/test_huggingface_tokenizer_allowlist.py::test_repo_id_matching_is_case_insensitive_but_returns_our_spelling FAILED tests/test_huggingface_tokenizer_allowlist.py::test_operator_env_allowlist_admits_a_private_repo FAILED tests/test_huggingface_tokenizer_allowlist.py::test_empty_operator_env_admits_nothing_extra FAILED tests/test_no_trust_remote_code.py::test_remote_code_is_never_enabled_anywhere_in_the_tree FAILED tests/test_no_trust_remote_code.py::test_the_tokenizer_loader_sets_the_flag_explicitly ========================= 10 failed, 4 passed in 1.47s ========================= ``` **After (source restored):** ``` tests/test_huggingface_tokenizer_allowlist.py ............ [ 54%] tests/test_no_trust_remote_code.py .. [ 63%] tests/test_huggingface_tokenizer_timeout.py ........ [100%] ============================== 22 passed in 2.73s ============================== ``` Wider tokenizer suites (`test_tokenizers.py`, `test_tokenizer.py`, `test_tokenizers/`, `test_tokenizer_encoding_resolution.py`, `test_tokenizer_selection_coverage.py`, `test_provider_tokenizer_one_ruler.py`, `test_tokenizer_count_offload.py`, `test_strands_tokenizer.py`, `test_compress_route_tokenizer_by_model.py`, `test_evals_cjk_tokenization.py`): **166 passed, 14 skipped**. Lint: `ruff 0.16.7 check .` and `format --check .` both clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
551f473e04 |
fix(proxy): accept Codex websocket before upstream retries (#2203)
## Description Codex Desktop could abandon Headroom's local `/v1/responses` WebSocket handshake before Headroom's upstream retry strategy had a chance to recover. The ChatGPT-auth path waited for an upstream opening handshake with a minimum 30-second timeout before sending the local 101, while the reported Codex Desktop handshake expired after about 34 seconds. This change accepts validated ChatGPT-auth Codex WebSockets before opening the upstream connection, then keeps the existing upstream retries and HTTP fallback behind the established local session. API-key sessions retain connect-before-accept behavior so upstream `x-codex-*` headers can still be attached to their client-facing 101. The change is scoped to the pre-101 timing failure and does not address the separate large-context streaming investigation in #1944. Closes #2184 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Accept ChatGPT-auth Codex WebSocket clients before the upstream connect and retry loop. - Preserve API-key connect-before-accept ordering and upstream `x-codex-*` handshake-header forwarding. - Keep upstream retry, relay, usage-state refresh, and WebSocket-to-HTTP fallback behavior after the local 101. - Add a deterministic regression that blocks the first upstream opening handshake and proves the local acceptance deadline is independent of it. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_openai_codex_ws_lifecycle.py -q`) - [x] Linting passes (`uv run ruff check headroom/proxy/handlers/openai.py tests/test_openai_codex_ws_lifecycle.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text uv run pytest tests/test_openai_codex_ws_lifecycle.py -q 28 passed in 2.02s uv run ruff check headroom/proxy/handlers/openai.py tests/test_openai_codex_ws_lifecycle.py All checks passed! uv run ruff format headroom/proxy/handlers/openai.py tests/test_openai_codex_ws_lifecycle.py --check 2 files already formatted ``` ## Real Behavior Proof - Environment: Python 3.12, synced development worktree, local fake Codex client and upstream WebSocket, no live provider - Exact command / steps: Run `uv run pytest tests/test_openai_codex_ws_lifecycle.py::test_chatgpt_ws_accepts_before_stalled_upstream_connect -q`; the fake upstream blocks its first opening handshake while the client enforces a bounded local-accept deadline. - Observed result: `1 passed in 0.46s`; the ChatGPT-auth client receives its local 101 before the blocked upstream connect is released, and the handler continues into its existing upstream recovery path. - Not tested: live Codex Desktop pre-turn compaction against ChatGPT subscription infrastructure ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes `CHANGELOG.md` is unchanged because the release pipeline generates it from conventional commits. No user documentation changes are required; the handler comments and ordered-flow docstring are updated with the auth-mode-specific behavior. The broader #1944 large-context disconnect surface remains out of scope. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
0c7087539d |
fix: add DeepSeek V4/V3.2/R1 tokenizer mappings and context limits (#912)
The tokenizer registry routed deepseek-v4-pro, deepseek-v4-flash,
deepseek-chat, deepseek-reasoner, and other modern DeepSeek models
to the 2023-era deepseek-llm-7b-base tokenizer via prefix fallback.
This caused token counts off by 30-50%, broken context-limit detection
(V4-Pro supports 1M but got 32K), and inaccurate savings reports.
## Fix
3 files, +43/-2:
- **huggingface.py**: 16 new MODEL_TO_TOKENIZER entries with verified
HuggingFace IDs (deepseek-ai/DeepSeek-V4-Pro, V4-Flash, V3.2,
V3-0324, R1, R1-0528, Reasoner, Chat, Coder-V2, etc.)
- **openai_compatible.py**: 17 new _DEFAULT_CONTEXT_LIMITS entries
(V4-Pro/Flash -> 1M, R1/Reasoner -> 131K, V3 -> 128K, etc.)
- **openai.py**: 8 new _CONTEXT_LIMITS entries for LiteLLM-fallback.
Existing mappings untouched (backward compatible).
## Real behavior proof
- **Setup**: Windows 11, Python 3.13.14, headroom-ai 0.2.15 wheel +
source checkout at v0.24.0. No Rust extension built (headroom._core
unavailable). Touched files are at parity with v0.24.0.
- **Steps after patch**:
```
python3 -c "
from headroom.tokenizers.huggingface import get_tokenizer_name
for m in
['deepseek-v4-pro','deepseek-chat','deepseek-reasoner','deepseek-v4-flash']:
print(f'{m} -> {get_tokenizer_name(m)}')
from headroom.tokenizers.registry import get_tokenizer
for m in ['deepseek-v4-pro','deepseek-chat','deepseek-reasoner']:
print(f'{m}: {get_tokenizer(m)}')
"
```
- **Observed result**:
```
deepseek-v4-pro -> deepseek-ai/DeepSeek-V4-Pro
deepseek-v4-flash -> deepseek-ai/DeepSeek-V4-Flash
deepseek-chat -> deepseek-ai/DeepSeek-V3
deepseek-reasoner -> deepseek-ai/DeepSeek-R1
```
Previously ALL resolved to deepseek-ai/deepseek-llm-7b-base.
TokenizerRegistry routes correctly. Context limits verified
(1M / 131K / 128K). compress() import smoke-tested OK.
- **Not tested**: full proxy e2e with a live DeepSeek API key
(no available key). HuggingFace AutoTokenizer download confirmed
for V4-Pro/V3/R1 but produced GBK decode errors from hf_hub on
this zh-CN Windows locale during config fetch -- a separate
huggingface_hub issue unrelated to this change.
<!-- headroom-maintainer-template-completion:start -->
## Description
This PR prepares `fix: add DeepSeek V4/V3.2/R1 tokenizer mappings and
context limits` for review by documenting the intended change,
validation evidence, and remaining merge-readiness context.
Linked issues: None declared.
## Type of Change
- [x] Bug fix
- [ ] New feature
- [ ] Documentation
- [ ] Refactor
- [ ] Tests only
## Changes Made
- Commit: fix: add DeepSeek V4/V3.2/R1 tokenizer mappings and context
limits
- Touches `headroom/providers/openai.py`
- Touches `headroom/providers/openai_compatible.py`
- Touches `headroom/tokenizers/huggingface.py`
## Testing
- [x] GitHub checks reviewed
- [x] Metadata/template validation
- [ ] Local functional testing
### Test Output
```text
gh pr view 912 --repo chopratejas/headroom --json statusCheckRollup
- PR Governance / label: SUCCESS
- external / GitGuardian Security Checks: SUCCESS
```
## Real Behavior Proof
- Environment: GitHub PR metadata and checks for `chopratejas/headroom`
PR #912.
- Exact command / steps: Reviewed PR title, commits, changed files,
linked issues, labels, and check rollup; appended this maintainer
template completion block without replacing the author's original
description.
- Observed result: PR body now contains all required governance
sections, checked readiness fields, and a non-placeholder validation
evidence block.
- Not tested: This pass updated PR metadata only; code validation
remains represented by the linked GitHub checks and any author-provided
evidence above.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
<!-- headroom-maintainer-template-completion:end -->
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
e0232df9b4 |
fix(tokenizers): resolve HF tokenizer names by most-specific prefix (#2096)
## Description
`get_tokenizer_name` can pick the wrong tokenizer for a versioned model,
which silently produces wrong token counts.
For a model that isn't a literal key in `MODEL_TO_TOKENIZER`, it falls
back to prefix matching:
```python
for key, value in MODEL_TO_TOKENIZER.items():
if model_lower.startswith(key):
return value
```
That returns the first key the model merely *starts with*, in
dict-insertion order. The table lists short family keys before their
more-specific siblings — `"qwen"` (→ `Qwen/Qwen-7B`) appears before
`"qwen2"`/`"qwen2-7b"`/`"qwen2.5"`. So
`get_tokenizer_name("qwen2-7b-instruct")` matches `"qwen"` first and
returns the **Qwen1** tokenizer, not Qwen2. Qwen1 and Qwen2 have
different vocabularies, so every `count_text`/`count_messages` for that
model is off. `qwen2.5-*` and `deepseek-v2.x` are mis-resolved the same
way.
The sibling tiktoken resolver already documents and guards this exact
pitfall — `get_encoding_for_model` uses an explicit most-specific-first
prefix list with a comment that scanning "for the first key that merely
starts with the prefix is order-dependent and wrong." The HuggingFace
resolver is the one that still scans insertion order.
## Fix
Match the **longest** (most-specific) prefix instead of the first in
insertion order:
```python
for key in sorted(MODEL_TO_TOKENIZER, key=len, reverse=True):
if model_lower.startswith(key):
return MODEL_TO_TOKENIZER[key]
```
Direct-key lookups and the shorter-family fallback (e.g. `deepseek-chat`
→ `deepseek-ai/deepseek-llm-7b-base`) are unchanged.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/tokenizers/huggingface.py`: `get_tokenizer_name` prefix
matching now iterates keys longest-first and returns the most-specific
match.
- `tests/test_huggingface_tokenizer_timeout.py`: add
`test_get_tokenizer_name_prefers_most_specific_prefix`.
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/tokenizers/huggingface.py tests/test_huggingface_tokenizer_timeout.py
All checks passed!
$ python -m py_compile headroom/tokenizers/huggingface.py tests/test_huggingface_tokenizer_timeout.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and a full `pytest`
gets OOM-killed on this box, so I verified against the real key table
with a dependency-free script that parses `MODEL_TO_TOKENIZER` out of
the source and runs both the old (insertion-order) and new
(longest-first) scans, then left the full pytest to CI.
- Exact command / steps: resolved `qwen2-7b-instruct`, `qwen2.5-turbo`,
and `deepseek-v2.5` under both strategies, plus `deepseek-chat` as a
regression guard.
- Observed result: old scan returns `Qwen/Qwen-7B` (Qwen1) for both
qwen2 models and `deepseek-ai/deepseek-llm-7b-base` (v1) for
`deepseek-v2.5`; new scan returns `Qwen/Qwen2-7B`, `Qwen/Qwen2.5-7B`,
and `deepseek-ai/DeepSeek-V2` respectively. `deepseek-chat` resolves
identically under both (`deepseek-ai/deepseek-llm-7b-base`), so the
existing timeout test's model is unaffected.
- Not tested: loading the actual HuggingFace tokenizers
(network/`transformers`); full local `pytest` deferred to CI (OOM, per
above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" and "type checking" boxes are unchecked
because the full suite imports the ML stack, which I can't run in this
environment; the change is a localized swap of the prefix-scan order in
a pure function, verified by the standalone proof (run against the real
key table) and the new regression test for CI. This mirrors the
same-class fix already present in the sibling tiktoken resolver.
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
|
46d5d685d9 |
fix(proxy): bound HF tokenizer load and offload token counting off event loop (#1738)
## Description Fixes #1701. On Windows, `headroom proxy --anthropic-api-url https://api.deepseek.com/anthropic` froze: the first `/v1/messages` request took ~610s (`optimization_latency_ms=609972`) with only router/lifecycle markers, and afterwards the whole server was a zombie — `/livez`, `/readyz` and `/health` hung until the process was killed. `HEADROOM_DETECT_BACKEND=python` was already set, so this was not the #575/#845 native-detect deadlock. Root cause: DeepSeek model names route to the HuggingFace tokenizer backend (`MODEL_PATTERNS` in `headroom/tokenizers/registry.py`). `HuggingFaceTokenizer` loads lazily, so the registry's construction-time fallback never fires; the first `count_messages` calls `AutoTokenizer.from_pretrained(..., trust_remote_code=True)` — unbounded network downloads/retries — and this ran **synchronously inside the async Anthropic messages handler** (`get_tokenizer(model)` + `tokenizer.count_messages(messages)`), outside the 30s `_run_compression_in_executor` bound. huggingface_hub retry chains on a restricted network easily reach ~10 minutes, blocking the entire asyncio event loop; subsequent on-loop counting kept it pinned. tiktoken got a bounded eager load for the same bug class long ago (#956); the HF backend never did. ## Type of Change - [x] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Refactoring (no functional changes) ## Changes Made - `headroom/tokenizers/huggingface.py`: `_load_tokenizer` now tries the local HF cache first (`local_files_only=True`, no network), then bounds the network load with `HEADROOM_HF_TOKENIZER_LOAD_TIMEOUT_SECS` (default 10s; `0` disables network loads) on a daemon thread. Timeouts/failures return `None` (cached by `lru_cache`, so the hub is probed at most once per process per tokenizer) and `count_messages` fails open to char-based estimation via the existing `_use_fallback()` path. - `headroom/proxy/handlers/anthropic.py`: new `AnthropicHandlerMixin._count_tokens_offloaded(model, messages)` runs `get_tokenizer` + `count_messages` on the compression executor bounded by `COMPRESSION_TIMEOUT_SECONDS`, failing open to `EstimatingTokenCounter` (downgrade logged once per model). Used in `handle_anthropic_messages` (the issue's hot path, both count sites) and `handle_anthropic_batch_create`; the batch path's inline `anthropic_pipeline.apply()` is now offloaded via `_run_compression_in_executor` (mirrors the #1612 image-compression offload). - `headroom/proxy/handlers/batch.py`: the two remaining inline `openai_pipeline.apply()` calls (`handle_google_batch_create`, `_compress_batch_jsonl`) are offloaded the same way; existing `except` blocks keep the pass-through fail-open semantics. - Tests: `tests/test_huggingface_tokenizer_timeout.py` (cache-first, bounded timeout, failure caching, timeout=0, fail-open estimation), `tests/test_tokenizer_count_offload.py` (wiring guards, runs on `headroom-compress` worker, event loop stays responsive during slow tokenizer work, fail-open), plus `_run_compression_in_executor` stub on the batch test double. ## Testing - [x] All existing tests pass - [x] Added new tests for the changes - [ ] Manual testing performed ``` $ python -m pytest tests/test_huggingface_tokenizer_timeout.py tests/test_tokenizer_count_offload.py tests/test_image_compression_offload.py tests/test_gemini_compression_offload.py tests/test_tokenizers tests/test_proxy_handlers_batch.py -q 50 passed $ ruff check . # No issues found $ ruff format --check . # 1043 files already formatted $ mypy headroom --ignore-missing-imports # 0 errors ``` ## Real Behavior Proof - Environment: Windows 11 Pro (10.0.26200), Python 3.13, local checkout of this branch with the Rust core built. - Exact command / steps: `python -m pytest tests/test_tokenizer_count_offload.py -q` — includes `test_count_tokens_offloaded_keeps_loop_responsive`, which reproduces the issue's mechanism: a tokenizer whose `count_messages` blocks (stand-in for the unbounded `AutoTokenizer.from_pretrained` network load) while an asyncio ticker measures event-loop liveness. Also `python -m pytest tests/test_huggingface_tokenizer_timeout.py -q` with a `from_pretrained` stub that sleeps 60s and `HEADROOM_HF_TOKENIZER_LOAD_TIMEOUT_SECS=0.2`. - Observed result: with the fix, the slow count runs on a `headroom-compress` worker thread and the loop keeps ticking (`ticks >= 5`; inline it yields ~0 — the zombie). The 60s-hung HF load unblocks at the 0.2s timeout, falls back to estimation, and the second call returns instantly (failure cached, no re-probe). All 10 new tests pass. - Not tested: live reproduction against `api.deepseek.com` from a network where HF hub downloads stall (the reporter's exact environment); actual HF vocab download timing on a healthy network. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |