mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(tokenizers): resolve tiktoken encoding for gateway-prefixed model ids (#4637)
## Description Since #2758 the registry unwraps gateway prefixes to pick the tiktoken backend for ids like `azure/gpt-4o` or `openrouter/openai/gpt-5`, but `get_encoding_for_model` still gets the wrapped name. It matches none of the prefixes and falls back to `cl100k_base`, so every wrapped gpt-4o / 4.1 / 5 / o-series id over-counts CJK by ~33% compared to the bare id. `OpenAIProvider.get_token_counter("openai/gpt-4o")` ends up on the same path. `get_encoding_for_model` now matches on the last path segment. No key in `MODEL_TO_ENCODING` contains a `/`, so bare ids resolve exactly as before. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/tokenizers/tiktoken_counter.py`: strip the gateway prefix before the encoding lookup. - `tests/test_tokenizer_encoding_resolution.py`: wrapped ids in the resolution table, plus an end-to-end count check through `get_tokenizer`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) (2 errors that are also on `main`, see notes; the changed file is clean) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # main, new tests only 6 failed, 22 passed in 1.34s # this branch $ uv run pytest tests/test_tokenizer_encoding_resolution.py tests/test_tokenizer_selection_coverage.py 48 passed in 1.46s # every test file that touches tokenizers / token counts (165 files) 3502 passed, 300 skipped in 413.62s $ uv run ruff check . && uv run ruff format --check . All checks passed! 1889 files already formatted ``` ## Real Behavior Proof - Environment: macOS 26.7.2 (arm64), Python 3.13.12, `uv sync --extra dev`, no provider calls (token counting only) - Exact command / steps: run the public tokenizer/compression script below against main and this branch. ```python from headroom import compress from headroom.tokenizers import get_tokenizer text = "这是一个测试文档,用于验证分词器的差异。" * 30 messages = [{"role": "user", "content": text}] for model in ["gpt-4o", "openai/gpt-4o", "azure/gpt-4o", "openrouter/openai/gpt-5"]: tok = get_tokenizer(model) r = compress(messages, model=model) print(model, tok, tok.count_text(text), r.tokens_before) ``` - Observed result: wrapped modern OpenAI model IDs now select o200k_base and match the bare-model counts, as shown below. ```text # main gpt-4o TiktokenCounter(model='gpt-4o', encoding='o200k_base') 450 457 openai/gpt-4o TiktokenCounter(model='openai/gpt-4o', encoding='cl100k_base') 600 607 azure/gpt-4o TiktokenCounter(model='azure/gpt-4o', encoding='cl100k_base') 600 607 openrouter/openai/gpt-5 TiktokenCounter(model='openrouter/openai/gpt-5', encoding='cl100k_base') 600 607 # this branch gpt-4o TiktokenCounter(model='gpt-4o', encoding='o200k_base') 450 457 openai/gpt-4o TiktokenCounter(model='openai/gpt-4o', encoding='o200k_base') 450 457 azure/gpt-4o TiktokenCounter(model='azure/gpt-4o', encoding='o200k_base') 450 457 openrouter/openai/gpt-5 TiktokenCounter(model='openrouter/openai/gpt-5', encoding='o200k_base') 450 457 ``` - Not tested: a live request through the proxy to Azure/OpenRouter; Windows. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: token counts for gateway-prefixed OpenAI ids now match the bare id - Kill switch / disable path: n/a - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - No docs change needed. - The full suite has 18 failures locally that also fail on `main` here (`test_proxy_uds.py`, `test_wrap_persistent.py`, `test_mcp_reconcile.py`, macOS socket paths), plus one that only fails in the full run. `mypy headroom` reports 2 errors that are also on `main` (`memory/adapters/embedders.py`, `integrations/langchain/chat_model.py`). <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4637" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Maintainer Windows/Python 3.12 validation on this exact head: all 48 focused tokenizer resolution/selection tests passed; lint, formatting, and whitespace checks passed. The public `get_tokenizer`/`headroom.compress` script above produced 450 text tokens and 457 input tokens for all four model IDs; `OpenAIProvider().get_token_counter("openai/gpt-4o")` also returned 450. Current main `31dcaf41` is already included. This used the existing native extension and no provider calls; no live gateway or fresh native build is claimed. Hosted qualification is running.
This commit is contained in:
@@ -170,7 +170,10 @@ def get_encoding_for_model(model: str) -> str:
|
||||
# verbatim, matched no prefix, and silently took DEFAULT_ENCODING -- so the
|
||||
# encoding a model got depended on the casing of whichever request warmed
|
||||
# the cache first, and flipped across restarts.
|
||||
model = model.lower()
|
||||
# Gateway-wrapped ids ("azure/gpt-4o", "openrouter/openai/gpt-5") are picked
|
||||
# for tiktoken by the registry's unwrapping but arrive here as-is, so match on
|
||||
# the bare id or they all take DEFAULT_ENCODING.
|
||||
model = model.lower().rsplit("/", 1)[-1]
|
||||
|
||||
# Direct lookup
|
||||
if model in MODEL_TO_ENCODING:
|
||||
|
||||
@@ -46,6 +46,12 @@ CJK = "这是一个测试文档,用于验证分词器的差异。" * 30
|
||||
("O4-Mini", "o200k_base"),
|
||||
("GPT-4", "cl100k_base"),
|
||||
("GPT-4-Turbo", "cl100k_base"),
|
||||
# Gateway prefixes must resolve like the bare id.
|
||||
("azure/gpt-4o", "o200k_base"),
|
||||
("openai/gpt-4.1-mini", "o200k_base"),
|
||||
("openrouter/openai/gpt-5", "o200k_base"),
|
||||
("Azure/GPT-4o", "o200k_base"),
|
||||
("openai/gpt-4", "cl100k_base"),
|
||||
# Must not regress.
|
||||
("gpt-4o", "o200k_base"),
|
||||
("gpt-4.1", "o200k_base"),
|
||||
@@ -73,6 +79,15 @@ def test_cold_cache_uppercase_still_gets_the_right_encoding(model: str) -> None:
|
||||
assert get_tokenizer(model).count_text(CJK) == o200k
|
||||
|
||||
|
||||
@pytest.mark.parametrize("model", ["azure/gpt-4o", "openrouter/openai/gpt-5"])
|
||||
def test_gateway_wrapped_model_counts_like_bare_id(model: str) -> None:
|
||||
tiktoken = pytest.importorskip("tiktoken")
|
||||
o200k = len(tiktoken.get_encoding("o200k_base").encode(CJK))
|
||||
|
||||
TokenizerRegistry.clear_cache()
|
||||
assert get_tokenizer(model).count_text(CJK) == o200k
|
||||
|
||||
|
||||
def test_casing_is_not_load_order_dependent() -> None:
|
||||
"""The same model must count identically whichever spelling arrives first."""
|
||||
TokenizerRegistry.clear_cache()
|
||||
|
||||
Reference in New Issue
Block a user