fix(tokenizers): resolve tiktoken encoding for gateway-prefixed model ids (#4637)

## Description

Since #2758 the registry unwraps gateway prefixes to pick the tiktoken
backend for ids like `azure/gpt-4o` or `openrouter/openai/gpt-5`, but
`get_encoding_for_model` still gets the wrapped name. It matches none of
the prefixes and falls back to `cl100k_base`, so every wrapped gpt-4o /
4.1 / 5 / o-series id over-counts CJK by ~33% compared to the bare id.
`OpenAIProvider.get_token_counter("openai/gpt-4o")` ends up on the same
path.

`get_encoding_for_model` now matches on the last path segment. No key in
`MODEL_TO_ENCODING` contains a `/`, so bare ids resolve exactly as
before.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/tokenizers/tiktoken_counter.py`: strip the gateway prefix
before the encoding lookup.
- `tests/test_tokenizer_encoding_resolution.py`: wrapped ids in the
resolution table, plus an end-to-end count check through
`get_tokenizer`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) (2 errors that are also on
`main`, see notes; the changed file is clean)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# main, new tests only
6 failed, 22 passed in 1.34s

# this branch
$ uv run pytest tests/test_tokenizer_encoding_resolution.py tests/test_tokenizer_selection_coverage.py
48 passed in 1.46s

# every test file that touches tokenizers / token counts (165 files)
3502 passed, 300 skipped in 413.62s

$ uv run ruff check . && uv run ruff format --check .
All checks passed!
1889 files already formatted
```

## Real Behavior Proof

- Environment: macOS 26.7.2 (arm64), Python 3.13.12, `uv sync --extra
dev`, no provider calls (token counting only)
- Exact command / steps: run the public tokenizer/compression script
below against main and this branch.

  ```python
  from headroom import compress
  from headroom.tokenizers import get_tokenizer

  text = "这是一个测试文档,用于验证分词器的差异。" * 30
  messages = [{"role": "user", "content": text}]
for model in ["gpt-4o", "openai/gpt-4o", "azure/gpt-4o",
"openrouter/openai/gpt-5"]:
      tok = get_tokenizer(model)
      r = compress(messages, model=model)
      print(model, tok, tok.count_text(text), r.tokens_before)
  ```

- Observed result: wrapped modern OpenAI model IDs now select o200k_base
and match the bare-model counts, as shown below.

  ```text
  # main
gpt-4o TiktokenCounter(model='gpt-4o', encoding='o200k_base') 450 457
openai/gpt-4o TiktokenCounter(model='openai/gpt-4o',
encoding='cl100k_base') 600 607
azure/gpt-4o TiktokenCounter(model='azure/gpt-4o',
encoding='cl100k_base') 600 607
openrouter/openai/gpt-5 TiktokenCounter(model='openrouter/openai/gpt-5',
encoding='cl100k_base') 600 607

  # this branch
gpt-4o TiktokenCounter(model='gpt-4o', encoding='o200k_base') 450 457
openai/gpt-4o TiktokenCounter(model='openai/gpt-4o',
encoding='o200k_base') 450 457
azure/gpt-4o TiktokenCounter(model='azure/gpt-4o',
encoding='o200k_base') 450 457
openrouter/openai/gpt-5 TiktokenCounter(model='openrouter/openai/gpt-5',
encoding='o200k_base') 450 457
  ```

- Not tested: a live request through the proxy to Azure/OpenRouter;
Windows.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: token counts for gateway-prefixed
OpenAI ids now match the bare id
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- No docs change needed.
- The full suite has 18 failures locally that also fail on `main` here
(`test_proxy_uds.py`, `test_wrap_persistent.py`,
`test_mcp_reconcile.py`, macOS socket paths), plus one that only fails
in the full run. `mypy headroom` reports 2 errors that are also on
`main` (`memory/adapters/embedders.py`,
`integrations/langchain/chat_model.py`).

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4637"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Maintainer Windows/Python 3.12 validation on this exact head: all 48
focused tokenizer resolution/selection tests passed; lint, formatting,
and whitespace checks passed. The public
`get_tokenizer`/`headroom.compress` script above produced 450 text
tokens and 457 input tokens for all four model IDs;
`OpenAIProvider().get_token_counter("openai/gpt-4o")` also returned 450.
Current main `31dcaf41` is already included. This used the existing
native extension and no provider calls; no live gateway or fresh native
build is claimed. Hosted qualification is running.
This commit is contained in:
Alvi Rahman
2026-11-04 22:37:48 -06:00
committed by GitHub
parent 50bf775e8c
commit e9331b6b96
2 changed files with 19 additions and 1 deletions
+4 -1
View File
@@ -170,7 +170,10 @@ def get_encoding_for_model(model: str) -> str:
# verbatim, matched no prefix, and silently took DEFAULT_ENCODING -- so the
# encoding a model got depended on the casing of whichever request warmed
# the cache first, and flipped across restarts.
model = model.lower()
# Gateway-wrapped ids ("azure/gpt-4o", "openrouter/openai/gpt-5") are picked
# for tiktoken by the registry's unwrapping but arrive here as-is, so match on
# the bare id or they all take DEFAULT_ENCODING.
model = model.lower().rsplit("/", 1)[-1]
# Direct lookup
if model in MODEL_TO_ENCODING:
@@ -46,6 +46,12 @@ CJK = "这是一个测试文档,用于验证分词器的差异。" * 30
("O4-Mini", "o200k_base"),
("GPT-4", "cl100k_base"),
("GPT-4-Turbo", "cl100k_base"),
# Gateway prefixes must resolve like the bare id.
("azure/gpt-4o", "o200k_base"),
("openai/gpt-4.1-mini", "o200k_base"),
("openrouter/openai/gpt-5", "o200k_base"),
("Azure/GPT-4o", "o200k_base"),
("openai/gpt-4", "cl100k_base"),
# Must not regress.
("gpt-4o", "o200k_base"),
("gpt-4.1", "o200k_base"),
@@ -73,6 +79,15 @@ def test_cold_cache_uppercase_still_gets_the_right_encoding(model: str) -> None:
assert get_tokenizer(model).count_text(CJK) == o200k
@pytest.mark.parametrize("model", ["azure/gpt-4o", "openrouter/openai/gpt-5"])
def test_gateway_wrapped_model_counts_like_bare_id(model: str) -> None:
tiktoken = pytest.importorskip("tiktoken")
o200k = len(tiktoken.get_encoding("o200k_base").encode(CJK))
TokenizerRegistry.clear_cache()
assert get_tokenizer(model).count_text(CJK) == o200k
def test_casing_is_not_load_order_dependent() -> None:
"""The same model must count identically whichever spelling arrives first."""
TokenizerRegistry.clear_cache()