mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 18:41:07 +08:00
fix/plugin-launcher-isolated-python
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
9648d2d867 |
fix(tokenizers): resolve chatgpt-4o-* aliases to o200k_base (#4672)
## Description `chatgpt-4o-latest` (and dated `chatgpt-4o-*` aliases) are GPT-4o but matched no prefix. `TokenizerRegistry` returned the estimating counter, and `get_encoding_for_model` fell back to `cl100k_base` instead of `o200k_base`. This adds the `chatgpt-4o-` prefix to both tables. No dependency change. Closes #4268 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/tokenizers/registry.py`: add `^chatgpt-4o-` -> tiktoken. - `headroom/tokenizers/tiktoken_counter.py`: add `chatgpt-4o-` -> `o200k_base`. - `tests/test_tokenizer_encoding_resolution.py`: new cases (lowercase, mixed case, cold-cache registry path). They fail on `main` and pass with the fix. ## Testing - [x] Unit tests pass (`pytest`): `tests/test_tokenizer_encoding_resolution.py` only, 31 passed (3 failed before the fix) - [x] Linting passes (`ruff check` / `ruff format --check` on touched files) - [ ] Type checking passes (`mypy headroom`): not run - [x] New tests added ## Real Behavior Proof - Environment: macOS, Python 3.12, real `tiktoken`, no mocks. - Exact command / steps: script comparing `tiktoken.encoding_for_model("chatgpt-4o-latest")` with `get_tokenizer(...)` and `TiktokenCounter(...)` on 30 repeats of a Chinese sentence. - Observed result: reference 450 tokens. Before: registry 400 (`EstimatingTokenCounter`), explicit `TiktokenCounter` 600. After: both 450. - Not tested: full test suite, mypy, CI hooks (commitlint/pre-push), Rust build, non-macOS platforms. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] I did **not** edit `CHANGELOG.md` This change was prepared with AI assistance (Claude Code). Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
|
e9331b6b96 |
fix(tokenizers): resolve tiktoken encoding for gateway-prefixed model ids (#4637)
## Description Since #2758 the registry unwraps gateway prefixes to pick the tiktoken backend for ids like `azure/gpt-4o` or `openrouter/openai/gpt-5`, but `get_encoding_for_model` still gets the wrapped name. It matches none of the prefixes and falls back to `cl100k_base`, so every wrapped gpt-4o / 4.1 / 5 / o-series id over-counts CJK by ~33% compared to the bare id. `OpenAIProvider.get_token_counter("openai/gpt-4o")` ends up on the same path. `get_encoding_for_model` now matches on the last path segment. No key in `MODEL_TO_ENCODING` contains a `/`, so bare ids resolve exactly as before. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/tokenizers/tiktoken_counter.py`: strip the gateway prefix before the encoding lookup. - `tests/test_tokenizer_encoding_resolution.py`: wrapped ids in the resolution table, plus an end-to-end count check through `get_tokenizer`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) (2 errors that are also on `main`, see notes; the changed file is clean) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # main, new tests only 6 failed, 22 passed in 1.34s # this branch $ uv run pytest tests/test_tokenizer_encoding_resolution.py tests/test_tokenizer_selection_coverage.py 48 passed in 1.46s # every test file that touches tokenizers / token counts (165 files) 3502 passed, 300 skipped in 413.62s $ uv run ruff check . && uv run ruff format --check . All checks passed! 1889 files already formatted ``` ## Real Behavior Proof - Environment: macOS 26.7.2 (arm64), Python 3.13.12, `uv sync --extra dev`, no provider calls (token counting only) - Exact command / steps: run the public tokenizer/compression script below against main and this branch. ```python from headroom import compress from headroom.tokenizers import get_tokenizer text = "这是一个测试文档,用于验证分词器的差异。" * 30 messages = [{"role": "user", "content": text}] for model in ["gpt-4o", "openai/gpt-4o", "azure/gpt-4o", "openrouter/openai/gpt-5"]: tok = get_tokenizer(model) r = compress(messages, model=model) print(model, tok, tok.count_text(text), r.tokens_before) ``` - Observed result: wrapped modern OpenAI model IDs now select o200k_base and match the bare-model counts, as shown below. ```text # main gpt-4o TiktokenCounter(model='gpt-4o', encoding='o200k_base') 450 457 openai/gpt-4o TiktokenCounter(model='openai/gpt-4o', encoding='cl100k_base') 600 607 azure/gpt-4o TiktokenCounter(model='azure/gpt-4o', encoding='cl100k_base') 600 607 openrouter/openai/gpt-5 TiktokenCounter(model='openrouter/openai/gpt-5', encoding='cl100k_base') 600 607 # this branch gpt-4o TiktokenCounter(model='gpt-4o', encoding='o200k_base') 450 457 openai/gpt-4o TiktokenCounter(model='openai/gpt-4o', encoding='o200k_base') 450 457 azure/gpt-4o TiktokenCounter(model='azure/gpt-4o', encoding='o200k_base') 450 457 openrouter/openai/gpt-5 TiktokenCounter(model='openrouter/openai/gpt-5', encoding='o200k_base') 450 457 ``` - Not tested: a live request through the proxy to Azure/OpenRouter; Windows. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: token counts for gateway-prefixed OpenAI ids now match the bare id - Kill switch / disable path: n/a - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - No docs change needed. - The full suite has 18 failures locally that also fail on `main` here (`test_proxy_uds.py`, `test_wrap_persistent.py`, `test_mcp_reconcile.py`, macOS socket paths), plus one that only fails in the full run. `mypy headroom` reports 2 errors that are also on `main` (`memory/adapters/embedders.py`, `integrations/langchain/chat_model.py`). <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4637" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Maintainer Windows/Python 3.12 validation on this exact head: all 48 focused tokenizer resolution/selection tests passed; lint, formatting, and whitespace checks passed. The public `get_tokenizer`/`headroom.compress` script above produced 450 text tokens and 457 input tokens for all four model IDs; `OpenAIProvider().get_token_counter("openai/gpt-4o")` also returned 450. Current main `31dcaf41` is already included. This used the existing native extension and no provider calls; no live gateway or fresh native build is claimed. Hosted qualification is running. |
||
|
|
fc4680b37a |
fix(tokenizers): resolve gpt-5 and mixed-case model names to the right encoding (#2776)
## Description Two defects in `get_encoding_for_model`, both reachable through the normal `get_tokenizer()` path. **1. `gpt-5` had no prefix entry.** It fell through to `DEFAULT_ENCODING` (`cl100k_base`) instead of `o200k_base`. Same class as the `o4` gap already patched in that tuple. cl100k emits ~33% more tokens than o200k on CJK, so every gpt-5 count was inflated there: ```text CJK sample (30x repeated sentence) o200k_base (correct) 450 tokens cl100k_base (actual) 600 tokens +33.3% ``` Note #2758 taught the *registry* that `gpt-5` → the tiktoken backend; this is the next hop, where that backend picks its *encoding*. So gpt-5 got the right tokenizer family and the wrong encoding inside it. **2. Resolution was case-sensitive.** `TokenizerRegistry.get` lowercases only its **cache key**, then constructs the counter from the caller's original string (`_create_tokenizer(model, backend)`). An uppercase deployment name — routine on Azure, where the deployment name is user-chosen — arrived verbatim, matched no prefix, and took the default encoding. The cache makes this one genuinely unpleasant: key lowercased, construction not, so **the encoding a model receives depends on the casing of whichever request warmed the cache first**, and can differ across restarts. ```text cold cache, uppercase resolved first: GPT-4o -> cl100k_base CJK=600 WRONG GPT-4.1 -> cl100k_base CJK=600 WRONG Gpt-4O-Mini -> cl100k_base CJK=600 WRONG gpt-4o -> o200k_base CJK=450 ok ``` I nearly filed this as "not reachable" — my first check ran the lowercase spelling first, which populated the shared lowercased cache key and masked it completely. The tests call `clear_cache()` so the uppercase spelling resolves cold, which is the failing order. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Added `("gpt-5", "o200k_base")` to the ordered prefix tuple. - `get_encoding_for_model` now lowercases its input. The lowercasing is deliberately scoped to this function rather than the registry: every `MODEL_TO_ENCODING` key is already lowercase (asserted), so it is safe here — whereas lowercasing in `TokenizerRegistry` would break HuggingFace repo ids, which *are* case-sensitive (`Qwen/Qwen3-Coder`). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check` + `ruff format`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality ### Test Output ```text $ pytest tests/test_tokenizer_encoding_resolution.py -q 21 passed $ git stash push headroom/ && pytest tests/test_tokenizer_encoding_resolution.py -q # gpt-5 family, every uppercase case, and both cache-order tests fail ``` Targeted run across the tokenizer/pricing/provider suites, including `test_evals_cjk_tokenization.py` since CJK is the affected content type: ```text $ pytest tests/test_utils.py tests/test_reporting.py tests/test_cost_pricing_warning_dedup.py \ tests/test_pricing.py tests/test_pricing_litellm.py tests/test_provider_model_fallback.py \ tests/test_models.py tests/test_savings_ledger.py tests/test_tokenizers.py \ tests/test_tokenizer.py tests/test_tokenizer_selection_coverage.py \ tests/test_provider_tokenizer_one_ruler.py tests/test_openai_model_table_resolution.py \ tests/test_evals_cjk_tokenization.py -q 233 passed, 16 skipped ``` ```text $ ruff check <changed files> All checks passed! $ mypy headroom/tokenizers/tiktoken_counter.py # only pre-existing release_version.py tomllib redef, present on main ``` Deferring the full suite to CI — this environment has no maturin/Rust core, so the native-dependent shards can't run locally. ## Real Behavior Proof - **Environment:** macOS, Python 3.13.7, isolated worktree at `upstream/main` (`0cb72f45`). - **Exact command / steps:** `TokenizerRegistry.clear_cache()`, then resolve each spelling cold and count a CJK sample. - **Observed result:** ```text before after gpt-5 cl100k_base CJK=600 o200k_base CJK=450 gpt-5-mini cl100k_base CJK=600 o200k_base CJK=450 GPT-4o cl100k_base CJK=600 o200k_base CJK=450 GPT-4.1 cl100k_base CJK=600 o200k_base CJK=450 Gpt-4O-Mini cl100k_base CJK=600 o200k_base CJK=450 GPT-4 cl100k_base CJK=600 cl100k_base CJK=600 (unchanged, correct) gpt-4o o200k_base CJK=450 o200k_base CJK=450 (unchanged) gpt-3.5-turbo cl100k_base CJK=600 cl100k_base CJK=600 (unchanged) ``` `GPT-4` was previously "correct" only by accident — it missed every prefix and landed on `DEFAULT_ENCODING`, which happens to be `cl100k_base`. It is now correct by resolution. |