Commit Graph
3 Commits
Author SHA1 Message Date
Rudy Mizrahi CelekliandClaude Sonnet 5.5 9648d2d867 fix(tokenizers): resolve chatgpt-4o-* aliases to o200k_base (#4672)
## Description

`chatgpt-4o-latest` (and dated `chatgpt-4o-*` aliases) are GPT-4o but
matched no prefix. `TokenizerRegistry` returned the estimating counter,
and `get_encoding_for_model` fell back to `cl100k_base` instead of
`o200k_base`. This adds the `chatgpt-4o-` prefix to both tables. No
dependency change.

Closes #4268

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/tokenizers/registry.py`: add `^chatgpt-4o-` -> tiktoken.
- `headroom/tokenizers/tiktoken_counter.py`: add `chatgpt-4o-` ->
`o200k_base`.
- `tests/test_tokenizer_encoding_resolution.py`: new cases (lowercase,
mixed case, cold-cache registry path). They fail on `main` and pass with
the fix.

## Testing

- [x] Unit tests pass (`pytest`):
`tests/test_tokenizer_encoding_resolution.py` only, 31 passed (3 failed
before the fix)
- [x] Linting passes (`ruff check` / `ruff format --check` on touched
files)
- [ ] Type checking passes (`mypy headroom`): not run
- [x] New tests added

## Real Behavior Proof

- Environment: macOS, Python 3.12, real `tiktoken`, no mocks.
- Exact command / steps: script comparing
`tiktoken.encoding_for_model("chatgpt-4o-latest")` with
`get_tokenizer(...)` and `TiktokenCounter(...)` on 30 repeats of a
Chinese sentence.
- Observed result: reference 450 tokens. Before: registry 400
(`EstimatingTokenCounter`), explicit `TiktokenCounter` 600. After: both
450.
- Not tested: full test suite, mypy, CI hooks (commitlint/pre-push),
Rust build, non-macOS platforms.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] I did **not** edit `CHANGELOG.md`

This change was prepared with AI assistance (Claude Code).

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-06 21:22:00 -06:00
Alvi Rahman e9331b6b96 fix(tokenizers): resolve tiktoken encoding for gateway-prefixed model ids (#4637)
## Description

Since #2758 the registry unwraps gateway prefixes to pick the tiktoken
backend for ids like `azure/gpt-4o` or `openrouter/openai/gpt-5`, but
`get_encoding_for_model` still gets the wrapped name. It matches none of
the prefixes and falls back to `cl100k_base`, so every wrapped gpt-4o /
4.1 / 5 / o-series id over-counts CJK by ~33% compared to the bare id.
`OpenAIProvider.get_token_counter("openai/gpt-4o")` ends up on the same
path.

`get_encoding_for_model` now matches on the last path segment. No key in
`MODEL_TO_ENCODING` contains a `/`, so bare ids resolve exactly as
before.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/tokenizers/tiktoken_counter.py`: strip the gateway prefix
before the encoding lookup.
- `tests/test_tokenizer_encoding_resolution.py`: wrapped ids in the
resolution table, plus an end-to-end count check through
`get_tokenizer`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) (2 errors that are also on
`main`, see notes; the changed file is clean)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# main, new tests only
6 failed, 22 passed in 1.34s

# this branch
$ uv run pytest tests/test_tokenizer_encoding_resolution.py tests/test_tokenizer_selection_coverage.py
48 passed in 1.46s

# every test file that touches tokenizers / token counts (165 files)
3502 passed, 300 skipped in 413.62s

$ uv run ruff check . && uv run ruff format --check .
All checks passed!
1889 files already formatted
```

## Real Behavior Proof

- Environment: macOS 26.7.2 (arm64), Python 3.13.12, `uv sync --extra
dev`, no provider calls (token counting only)
- Exact command / steps: run the public tokenizer/compression script
below against main and this branch.

  ```python
  from headroom import compress
  from headroom.tokenizers import get_tokenizer

  text = "这是一个测试文档,用于验证分词器的差异。" * 30
  messages = [{"role": "user", "content": text}]
for model in ["gpt-4o", "openai/gpt-4o", "azure/gpt-4o",
"openrouter/openai/gpt-5"]:
      tok = get_tokenizer(model)
      r = compress(messages, model=model)
      print(model, tok, tok.count_text(text), r.tokens_before)
  ```

- Observed result: wrapped modern OpenAI model IDs now select o200k_base
and match the bare-model counts, as shown below.

  ```text
  # main
gpt-4o TiktokenCounter(model='gpt-4o', encoding='o200k_base') 450 457
openai/gpt-4o TiktokenCounter(model='openai/gpt-4o',
encoding='cl100k_base') 600 607
azure/gpt-4o TiktokenCounter(model='azure/gpt-4o',
encoding='cl100k_base') 600 607
openrouter/openai/gpt-5 TiktokenCounter(model='openrouter/openai/gpt-5',
encoding='cl100k_base') 600 607

  # this branch
gpt-4o TiktokenCounter(model='gpt-4o', encoding='o200k_base') 450 457
openai/gpt-4o TiktokenCounter(model='openai/gpt-4o',
encoding='o200k_base') 450 457
azure/gpt-4o TiktokenCounter(model='azure/gpt-4o',
encoding='o200k_base') 450 457
openrouter/openai/gpt-5 TiktokenCounter(model='openrouter/openai/gpt-5',
encoding='o200k_base') 450 457
  ```

- Not tested: a live request through the proxy to Azure/OpenRouter;
Windows.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: token counts for gateway-prefixed
OpenAI ids now match the bare id
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- No docs change needed.
- The full suite has 18 failures locally that also fail on `main` here
(`test_proxy_uds.py`, `test_wrap_persistent.py`,
`test_mcp_reconcile.py`, macOS socket paths), plus one that only fails
in the full run. `mypy headroom` reports 2 errors that are also on
`main` (`memory/adapters/embedders.py`,
`integrations/langchain/chat_model.py`).

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4637"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Maintainer Windows/Python 3.12 validation on this exact head: all 48
focused tokenizer resolution/selection tests passed; lint, formatting,
and whitespace checks passed. The public
`get_tokenizer`/`headroom.compress` script above produced 450 text
tokens and 457 input tokens for all four model IDs;
`OpenAIProvider().get_token_counter("openai/gpt-4o")` also returned 450.
Current main `31dcaf41` is already included. This used the existing
native extension and no provider calls; no live gateway or fresh native
build is claimed. Hosted qualification is running.
2026-11-04 22:37:48 -06:00
Tejas Chopra fc4680b37a fix(tokenizers): resolve gpt-5 and mixed-case model names to the right encoding (#2776)
## Description

Two defects in `get_encoding_for_model`, both reachable through the
normal `get_tokenizer()` path.

**1. `gpt-5` had no prefix entry.** It fell through to
`DEFAULT_ENCODING` (`cl100k_base`) instead of `o200k_base`. Same class
as the `o4` gap already patched in that tuple. cl100k emits ~33% more
tokens than o200k on CJK, so every gpt-5 count was inflated there:

```text
CJK sample (30x repeated sentence)
  o200k_base (correct)   450 tokens
  cl100k_base (actual)   600 tokens    +33.3%
```

Note #2758 taught the *registry* that `gpt-5` → the tiktoken backend;
this is the next hop, where that backend picks its *encoding*. So gpt-5
got the right tokenizer family and the wrong encoding inside it.

**2. Resolution was case-sensitive.** `TokenizerRegistry.get` lowercases
only its **cache key**, then constructs the counter from the caller's
original string (`_create_tokenizer(model, backend)`). An uppercase
deployment name — routine on Azure, where the deployment name is
user-chosen — arrived verbatim, matched no prefix, and took the default
encoding.

The cache makes this one genuinely unpleasant: key lowercased,
construction not, so **the encoding a model receives depends on the
casing of whichever request warmed the cache first**, and can differ
across restarts.

```text
cold cache, uppercase resolved first:
  GPT-4o        -> cl100k_base   CJK=600   WRONG
  GPT-4.1       -> cl100k_base   CJK=600   WRONG
  Gpt-4O-Mini   -> cl100k_base   CJK=600   WRONG
  gpt-4o        -> o200k_base    CJK=450   ok
```

I nearly filed this as "not reachable" — my first check ran the
lowercase spelling first, which populated the shared lowercased cache
key and masked it completely. The tests call `clear_cache()` so the
uppercase spelling resolves cold, which is the failing order.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Added `("gpt-5", "o200k_base")` to the ordered prefix tuple.
- `get_encoding_for_model` now lowercases its input.

The lowercasing is deliberately scoped to this function rather than the
registry: every `MODEL_TO_ENCODING` key is already lowercase (asserted),
so it is safe here — whereas lowercasing in `TokenizerRegistry` would
break HuggingFace repo ids, which *are* case-sensitive
(`Qwen/Qwen3-Coder`).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check` + `ruff format`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality

### Test Output

```text
$ pytest tests/test_tokenizer_encoding_resolution.py -q
21 passed

$ git stash push headroom/ && pytest tests/test_tokenizer_encoding_resolution.py -q
# gpt-5 family, every uppercase case, and both cache-order tests fail
```

Targeted run across the tokenizer/pricing/provider suites, including
`test_evals_cjk_tokenization.py` since CJK is the affected content type:

```text
$ pytest tests/test_utils.py tests/test_reporting.py tests/test_cost_pricing_warning_dedup.py \
         tests/test_pricing.py tests/test_pricing_litellm.py tests/test_provider_model_fallback.py \
         tests/test_models.py tests/test_savings_ledger.py tests/test_tokenizers.py \
         tests/test_tokenizer.py tests/test_tokenizer_selection_coverage.py \
         tests/test_provider_tokenizer_one_ruler.py tests/test_openai_model_table_resolution.py \
         tests/test_evals_cjk_tokenization.py -q
233 passed, 16 skipped
```

```text
$ ruff check <changed files>          All checks passed!
$ mypy headroom/tokenizers/tiktoken_counter.py
# only pre-existing release_version.py tomllib redef, present on main
```

Deferring the full suite to CI — this environment has no maturin/Rust
core, so the native-dependent shards can't run locally.

## Real Behavior Proof

- **Environment:** macOS, Python 3.13.7, isolated worktree at
`upstream/main` (`0cb72f45`).
- **Exact command / steps:** `TokenizerRegistry.clear_cache()`, then
resolve each spelling cold and count a CJK sample.
- **Observed result:**

```text
                 before                    after
gpt-5            cl100k_base  CJK=600      o200k_base  CJK=450
gpt-5-mini       cl100k_base  CJK=600      o200k_base  CJK=450
GPT-4o           cl100k_base  CJK=600      o200k_base  CJK=450
GPT-4.1          cl100k_base  CJK=600      o200k_base  CJK=450
Gpt-4O-Mini      cl100k_base  CJK=600      o200k_base  CJK=450
GPT-4            cl100k_base  CJK=600      cl100k_base CJK=600   (unchanged, correct)
gpt-4o           o200k_base   CJK=450      o200k_base  CJK=450   (unchanged)
gpt-3.5-turbo    cl100k_base  CJK=600      cl100k_base CJK=600   (unchanged)
```

`GPT-4` was previously "correct" only by accident — it missed every
prefix and landed on `DEFAULT_ENCODING`, which happens to be
`cl100k_base`. It is now correct by resolution.
2026-08-04 11:31:16 -07:00