Files
headroom/tests/test_tokenizers.py
Ehsan-hashandJD Davis fe01961b29 fix(tokenizers): honor registered model factories (#4437)
## Description

`register_tokenizer(model, factory=...)` currently stores the factory
under a model name in the backend registry. Automatic lookup selects a
backend instead, so the factory is never called and a custom model
silently receives estimated counts.

Keep model factories separate and consult them during automatic lookup.
This makes the documented factory form usable without changing explicit
backend selection.

Closes #4434

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Resolve registered model factories lazily, preserving the caller's
model spelling and case-insensitive cache lookup.
- Remove stale instance or factory registrations when replacing one with
the other, and invalidate cached counters.
- Retry model factories after failures so a cached estimate cannot mask
a later strict lookup or recovery.
- Include factory-backed model names in `list_registered()`, including
instance-to-factory replacement.
- Add regressions for lookup, caching, replacement, backend isolation,
strict failure/recovery, registration listings and forced-backend
selection.

## Testing

- [x] Focused and affected tokenizer/provider/compression tests pass.
- [x] Linting and formatting pass with pinned Ruff 0.17.0.
- [x] Type checking passes through the repository's pre-commit hook
(`mypy headroom`).
- [x] New regression fails before the fix and passes afterward.
- [x] Public API reproduction and error/fallback controls run locally.
- [x] Required pre-push gate passes: Rust workspace checks, extension
build, Python suite and commitlint.

### Test Output

```text
Before: EstimatingTokenCounter, count 2, factory_calls []
After: CharacterCounter, count 8, factory_calls ['CUSTOM-FACTORY-MODEL'], cached True

Baseline new regression: 1 failed, 69 deselected in 1.51s
Final affected set: 133 passed, 14 skipped in 14.97s
ruff check .: All checks passed!
ruff format --check .: 1812 files already formatted
Pre-commit: Ruff, formatting, mypy, conflict markers and commitlint passed
Pre-push: Rust fmt/clippy/workspace tests and extension rebuild passed
Pre-push Python suite: 177 passed, 4 skipped, 1 warning in 8.86s
ci-precheck PASSED
```

The 14 skips require the optional `mistral-common` package. The
compression-route tests ran and passed. The complete all-extras Python
suite was not run.

The pre-push suite's warning is the expected Anthropic-provider
approximation notice when no client is supplied.

## Real Behavior Proof

- Environment: macOS arm64, Python 3.12.15, editable source checkout; no
provider or model request. Offline mode enabled.
- Exact command / steps: register a factory and count a fixed
eight-character string using the public API, as below. Run `python -m
pytest -q tests/test_tokenizers.py
tests/test_tokenizer_selection_coverage.py
tests/test_provider_tokenizer_one_ruler.py
tests/test_compress_route_tokenizer_by_model.py
tests/test_tokenizer_encoding_resolution.py` for the affected set.
- Observed result: the factory is called once with the original model
string, returns a counter that reports eight, and repeated
case-insensitive lookup reuses that counter. Before the fix, no factory
call occurs and the estimator reports two. A deliberately failing
factory falls back by default; a subsequent strict lookup raises
`ValueError` without clearing the cache. When that factory becomes
available, the next lookup recovers. Factory-backed model names remain
listed once after replacing an instance.
- Not tested: live provider requests, optional Mistral package tests,
the complete all-extras Python suite, upstream CI or maintainer
acceptance.

```python
from headroom.tokenizers import CharacterCounter, get_tokenizer, register_tokenizer

calls = []

def factory(model):
    calls.append(model)
    return CharacterCounter(chars_per_token=1.0)

register_tokenizer("custom-factory-model", factory=factory)
first = get_tokenizer("CUSTOM-FACTORY-MODEL")
second = get_tokenizer("custom-factory-model")
print(type(first).__name__, first.count_text("abcdefgh"), calls, first is second)
# CharacterCounter 8 ['CUSTOM-FACTORY-MODEL'] True
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none; existing public tokenizer
registration only.
- Minimum rollout channel: not applicable, no rollout channel added.
- Stable/default behavior changed: explicitly registered model factories
are honored during automatic lookup and included in registration
listings. Failed model factories remain retryable. Unregistered models
and explicit backend selection retain their existing paths and fallback
caching.
- Kill switch / disable path: choose an explicit backend, or register a
prebuilt counter instead.
- Unsafe override required: none.
- Qualification impact: fixes an existing documented API without new
dependencies, provider access or network behavior.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] I did **not** edit `CHANGELOG.md`, which is generated by
release-please

## Additional Notes

The existing docstring already describes factory registration. A
separate independent review examined the complete patch. No dependency
or lockfile changes are included. Local checks establish the observed
behavior; upstream CI and maintainer review remain pending.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4437"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 22:47:22 -06:00

1003 lines
38 KiB
Python

"""Tests for the pluggable tokenizer system."""
from __future__ import annotations
import pytest
from headroom.tokenizers import (
BaseTokenizer,
CharacterCounter,
EstimatingTokenCounter,
TiktokenCounter,
TokenCounter,
TokenizerRegistry,
get_mistral_tokenizer,
get_tokenizer,
is_mistral_tokenizer_available,
list_supported_models,
register_tokenizer,
)
from headroom.tokenizers.base import coerce_countable_text
class TestTiktokenCounter:
"""Tests for TiktokenCounter."""
def test_init_default_model(self):
"""Test initialization with default model."""
counter = TiktokenCounter()
assert counter.model == "gpt-4o"
assert counter.encoding_name == "o200k_base"
def test_init_gpt4_model(self):
"""Test initialization with GPT-4."""
counter = TiktokenCounter("gpt-4")
assert counter.model == "gpt-4"
assert counter.encoding_name == "cl100k_base"
def test_unknown_gpt4_snapshot_uses_cl100k(self):
"""Unknown gpt-4 (non-o, non-turbo) snapshots must use cl100k_base.
Regression: the prefix matcher scanned MODEL_TO_ENCODING for the
first key starting with the prefix. For prefix "gpt-4" that matched
the "gpt-4o" entry first and wrongly returned o200k_base for any
gpt-4 snapshot not in the table (e.g. a future dated build).
"""
from headroom.tokenizers.tiktoken_counter import get_encoding_for_model
assert get_encoding_for_model("gpt-4-2025-01-01") == "cl100k_base"
assert get_encoding_for_model("gpt-4-future") == "cl100k_base"
# gpt-4o snapshots still resolve to o200k_base (most-specific first).
assert get_encoding_for_model("gpt-4o-2099-12-31") == "o200k_base"
# gpt-4-turbo snapshots use cl100k_base.
assert get_encoding_for_model("gpt-4-turbo-2099") == "cl100k_base"
def test_gpt41_and_o4_families_use_o200k(self):
"""gpt-4.1 / gpt-4.5 / o4 use o200k_base, not cl100k_base.
Regression: gpt-4.1* and gpt-4.5* matched the broad "gpt-4" prefix and
resolved to cl100k_base, and o4* matched no prefix and fell to the
cl100k_base default — both wrong encodings for those models.
"""
from headroom.tokenizers.tiktoken_counter import get_encoding_for_model
assert get_encoding_for_model("gpt-4.1") == "o200k_base"
assert get_encoding_for_model("gpt-4.1-mini") == "o200k_base"
assert get_encoding_for_model("gpt-4.5-preview") == "o200k_base"
assert get_encoding_for_model("o4-mini") == "o200k_base"
# A plain gpt-4 snapshot must still use cl100k_base (not shadowed).
assert get_encoding_for_model("gpt-4-0613") == "cl100k_base"
def test_count_text_empty(self):
"""Test counting empty text."""
counter = TiktokenCounter()
assert counter.count_text("") == 0
def test_count_text_simple(self):
"""Test counting simple text."""
counter = TiktokenCounter()
count = counter.count_text("Hello, world!")
assert count > 0
assert count < 10 # Should be a few tokens
def test_count_text_unicode(self):
"""Test counting text with unicode."""
counter = TiktokenCounter()
count = counter.count_text("Hello, 世界!")
assert count > 0
def test_count_messages_single(self):
"""Test counting single message."""
counter = TiktokenCounter()
messages = [{"role": "user", "content": "Hello!"}]
count = counter.count_messages(messages)
assert count > 0
def test_count_messages_with_tool_calls(self):
"""Test counting messages with tool calls."""
counter = TiktokenCounter()
messages = [
{"role": "user", "content": "Search for Python"},
{
"role": "assistant",
"tool_calls": [
{
"id": "call_123",
"type": "function",
"function": {
"name": "search",
"arguments": '{"query": "Python"}',
},
}
],
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": "Results...",
},
]
count = counter.count_messages(messages)
assert count > 0
def test_count_messages_image_block_is_not_stringified(self):
"""An Anthropic-style image block must be priced as an image, not text.
Over the wire the image arrives as a base64 string inside list content.
The old count_messages else-branch stringified any non-text/non-image_url
part and tokenized it as text, so a 1MB image counted as ~330K phantom
tokens. The base handler prices image blocks by a bounded estimate, so the
count must stay small regardless of the base64 payload size.
"""
import base64
counter = TiktokenCounter()
blob = base64.b64encode(b"\x89PNG\r\n\x1a\n" + b"\x00" * 200_000).decode()
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": "What is in this screenshot?"},
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": blob,
},
},
],
}
]
count = counter.count_messages(messages)
# The base64 blob alone would be tens of thousands of text tokens; a
# bounded image estimate keeps the whole message well under that.
assert count < 5000, count
assert count < len(blob) // 10
def test_encode_decode_roundtrip(self):
"""Test encode/decode roundtrip."""
counter = TiktokenCounter()
text = "Hello, world!"
tokens = counter.encode(text)
decoded = counter.decode(tokens)
assert decoded == text
def test_count_text_allows_literal_special_tokens(self):
"""count_text must not raise on literal tiktoken special-token strings.
Regression: passthrough/tool content containing "<|endoftext|>" (or FIM
markers) made tiktoken raise ValueError under its default
disallowed_special="all", aborting token counting for the whole request.
Through the proxy this surfaced as an HTTP 413 compression_refused.
"""
counter = TiktokenCounter("gpt-4o")
text = "before <|endoftext|> after <|fim_prefix|> end"
# Must not raise; markers are counted as ordinary text.
count = counter.count_text(text)
assert count > counter.count_text("before after end")
def test_encode_allows_literal_special_tokens(self):
"""encode must treat literal special-token strings as ordinary text."""
counter = TiktokenCounter("gpt-4o")
text = "x <|endoftext|> y"
tokens = counter.encode(text)
assert isinstance(tokens, list) and len(tokens) > 0
# Encoding as ordinary text round-trips back to the original literal.
assert counter.decode(tokens) == text
def test_repr(self):
"""Test string representation."""
counter = TiktokenCounter("gpt-4o")
assert "TiktokenCounter" in repr(counter)
assert "gpt-4o" in repr(counter)
class TestEstimatingTokenCounter:
"""Tests for EstimatingTokenCounter."""
def test_init_default(self):
"""Test initialization with defaults."""
counter = EstimatingTokenCounter()
assert counter._fixed_ratio is None
def test_init_fixed_ratio(self):
"""Test initialization with fixed ratio."""
counter = EstimatingTokenCounter(chars_per_token=3.5)
assert counter._fixed_ratio == 3.5
def test_count_text_empty(self):
"""Test counting empty text."""
counter = EstimatingTokenCounter()
assert counter.count_text("") == 0
def test_count_text_simple(self):
"""Test counting simple text."""
counter = EstimatingTokenCounter()
text = "Hello, world!"
count = counter.count_text(text)
assert count > 0
# Rough estimate: 13 chars / 4 chars per token ≈ 3-4 tokens
assert 2 <= count <= 6
def test_count_text_fixed_ratio(self):
"""Test counting with fixed ratio."""
counter = EstimatingTokenCounter(chars_per_token=5.0)
text = "x" * 50 # 50 chars
count = counter.count_text(text)
assert count == 10 # 50 / 5 = 10
def test_count_text_fixed_ratio_prices_cjk(self):
"""The fixed-ratio path must price dense scripts (CJK) like the auto path.
The registry builds provider counters (Anthropic 3.5, Google 4.0, ...)
with a fixed ratio; before the fix CJK was priced at the Latin ratio, so
a large CJK context read as ~40-55% of its true size and could skip
compression."""
counter = EstimatingTokenCounter(chars_per_token=3.5)
cjk = "これはテストです" * 100 # 800 dense-script chars, no ASCII
count = counter.count_text(cjk)
# ~1 token per 1.5 chars (CHARS_PER_TOKEN_CJK), not 1 per 3.5.
assert count == pytest.approx(len(cjk) / 1.5, rel=0.05)
# Far higher than the old Latin-ratio estimate.
assert count > len(cjk) / 3.5 * 2
# ASCII is unaffected by the fixed ratio.
assert counter.count_text("x" * 35) == 10
def test_count_text_minimum_one(self):
"""Test minimum of 1 token."""
counter = EstimatingTokenCounter()
assert counter.count_text("x") >= 1
def test_count_messages(self):
"""Test counting messages."""
counter = EstimatingTokenCounter()
messages = [
{"role": "user", "content": "Hello!"},
{"role": "assistant", "content": "Hi there!"},
]
count = counter.count_messages(messages)
assert count > 0
def test_json_detection(self):
"""Test JSON content detection."""
counter = EstimatingTokenCounter()
json_text = '{"name": "test", "value": 123}'
# Should use JSON ratio
count = counter.count_text(json_text)
assert count > 0
def test_code_detection(self):
"""Test code content detection."""
counter = EstimatingTokenCounter()
code_text = """
def hello():
return "Hello, world!"
"""
count = counter.count_text(code_text)
assert count > 0
def test_count_text_cjk_not_underestimated(self):
"""CJK text must not be priced at the Latin ~4-chars/token ratio.
Regression: count_text divided the whole string length by the Latin
ratio (4.0), so 100 Chinese characters estimated ~25 tokens while real
tokenizers (cl100k_base / DeepSeek / Qwen) produce ~60-150. Dense
scripts tokenize at roughly one token per character, so the estimate
must be far above len/4 and on the order of the character count.
"""
counter = EstimatingTokenCounter()
text = "你好世界" * 25 # 100 CJK characters
count = counter.count_text(text)
# Old behavior returned len/4 == 25; require clearly above that floor.
assert count > len(text) / 3
# And in the right ballpark for one-token-per-char scripts.
assert count >= int(len(text) * 0.6)
def test_count_text_cjk_japanese_and_korean(self):
"""Japanese (Kana) and Korean (Hangul) are also dense scripts."""
counter = EstimatingTokenCounter()
for text in ("こんにちは世界" * 10, "안녕하세요" * 10):
count = counter.count_text(text)
assert count >= int(len(text) * 0.6)
def test_count_text_mixed_latin_cjk(self):
"""Mixed text prices the Latin part and the CJK part independently."""
counter = EstimatingTokenCounter()
latin = "The quick brown fox jumps over the lazy dog. " # 45 chars
cjk = "今天天气很好" # 6 CJK chars
mixed = counter.count_text(latin + cjk)
# Must exceed the all-Latin estimate of the same length, since the CJK
# tail is priced denser than 4 chars/token.
latin_only = counter.count_text(latin + "x" * len(cjk))
assert mixed > latin_only
def test_count_text_latin_unchanged(self):
"""Pure-Latin estimates are unchanged by the CJK adjustment."""
counter = EstimatingTokenCounter()
text = "Hello, world!"
assert 2 <= counter.count_text(text) <= 6
def test_repr(self):
"""Test string representation."""
counter = EstimatingTokenCounter()
assert "EstimatingTokenCounter" in repr(counter)
class TestCharacterCounter:
"""Tests for CharacterCounter."""
def test_init_default(self):
"""Test initialization with default ratio."""
counter = CharacterCounter()
assert counter.chars_per_token == 4.0
def test_init_custom_ratio(self):
"""Test initialization with custom ratio."""
counter = CharacterCounter(chars_per_token=3.5)
assert counter.chars_per_token == 3.5
def test_count_text(self):
"""Test counting text."""
counter = CharacterCounter(chars_per_token=4.0)
text = "x" * 40 # 40 chars
count = counter.count_text(text)
assert count == 10 # 40 / 4 = 10
def test_count_text_empty(self):
"""Test counting empty text."""
counter = CharacterCounter()
assert counter.count_text("") == 0
class TestTokenizerRegistry:
"""Tests for TokenizerRegistry."""
def test_get_openai_model(self):
"""Test getting tokenizer for OpenAI model."""
tokenizer = get_tokenizer("gpt-4o")
assert isinstance(tokenizer, TiktokenCounter)
def test_get_anthropic_model(self):
"""Anthropic (Claude) has no public tokenizer, so it is priced against a
real BPE proxy (tiktoken o200k_base) rather than a character estimate —
for consistent, monotone before/after counts. Falls back to the estimator
only when the tiktoken vocab can't be loaded."""
tokenizer = get_tokenizer("claude-3-sonnet")
if isinstance(tokenizer, TiktokenCounter):
assert tokenizer.encoding_name == "o200k_base"
else: # vocab unavailable in this environment → documented fallback
assert isinstance(tokenizer, EstimatingTokenCounter)
def test_get_unknown_model_fallback(self):
"""Test fallback for unknown model."""
tokenizer = get_tokenizer("unknown-model-xyz")
assert isinstance(tokenizer, EstimatingTokenCounter)
def test_get_kimi_moonshot_calibrated_estimator(self):
"""Kimi/Moonshot resolves to the calibrated (3.1 chars/tok) estimator
across every serving form — Fireworks body, litellm slug, native — so
the size-gates aren't starved by the ~20% under-count of the default
adaptive estimator (measured on a SWE-bench Kimi-K2.7-code run)."""
for m in (
"accounts/fireworks/models/kimi-k2p7-code", # Fireworks body model
"fireworks_ai/kimi-k2p7-code-high", # litellm slug
"moonshotai/Kimi-K2-Instruct", # native
"KIMI-K2P7-CODE", # case-insensitive
):
tk = get_tokenizer(m)
assert isinstance(tk, EstimatingTokenCounter), m
assert tk._fixed_ratio == 3.1, f"{m}: expected 3.1, got {tk._fixed_ratio}"
# calibrated estimate must beat the default adaptive on Kimi-like code
# (which the default under-counts): denser ratio -> more tokens.
code = 'def f(x):\n return {"a": 1, "b": [2, 3]}\n' * 200
kimi = get_tokenizer("fireworks_ai/kimi-k2p7-code-high").count_text(code)
default = get_tokenizer("unknown-model-xyz").count_text(code)
assert kimi > default, (kimi, default)
def test_get_with_specific_backend(self):
"""Test forcing specific backend."""
tokenizer = get_tokenizer("any-model", backend="estimation")
assert isinstance(tokenizer, EstimatingTokenCounter)
def test_register_custom_tokenizer(self):
"""Test registering custom tokenizer."""
custom = EstimatingTokenCounter(chars_per_token=3.0)
register_tokenizer("my-custom-model", tokenizer=custom)
retrieved = get_tokenizer("my-custom-model")
assert retrieved is custom
def test_register_custom_factory_is_lazy_case_insensitive_and_replaceable(self, monkeypatch):
"""Model factories should override auto detection without polluting backends."""
registry = TokenizerRegistry()
for attr in ("_tokenizers", "_factories", "_cache"):
monkeypatch.setattr(registry, attr, getattr(registry, attr).copy())
monkeypatch.setattr(
registry,
"_model_factories",
getattr(registry, "_model_factories", {}).copy(),
raising=False,
)
model = "Custom-Factory-Model"
calls: list[str] = []
custom = CharacterCounter(chars_per_token=1.0)
def factory(requested_model: str) -> CharacterCounter:
calls.append(requested_model)
return custom
register_tokenizer(model, factory=factory)
assert model.lower() not in registry.list_backends()
first = get_tokenizer(model.upper())
assert first is custom
assert calls == [model.upper()]
assert get_tokenizer(model.lower()) is first
assert calls == [model.upper()]
forced = get_tokenizer(model, backend="estimation")
assert isinstance(forced, EstimatingTokenCounter)
assert calls == [model.upper()]
replacement = EstimatingTokenCounter(chars_per_token=2.0)
register_tokenizer(model, tokenizer=replacement)
assert get_tokenizer(model.lower()) is replacement
replacement_calls: list[str] = []
replacement_custom = CharacterCounter(chars_per_token=2.0)
def replacement_factory(requested_model: str) -> CharacterCounter:
replacement_calls.append(requested_model)
return replacement_custom
register_tokenizer(model.lower(), factory=replacement_factory)
recreated = get_tokenizer(model)
assert recreated is replacement_custom
assert replacement_calls == [model]
assert get_tokenizer(model.upper()) is recreated
assert replacement_calls == [model]
def test_model_factory_failure_does_not_poison_strict_or_recovery(self, monkeypatch):
"""A fallback from a model factory must not hide later retries."""
registry = TokenizerRegistry()
for attr in ("_tokenizers", "_factories", "_cache"):
monkeypatch.setattr(registry, attr, getattr(registry, attr).copy())
monkeypatch.setattr(
registry,
"_model_factories",
getattr(registry, "_model_factories", {}).copy(),
raising=False,
)
model = "Transient-Factory-Model"
state = {"available": False}
calls: list[str] = []
custom = CharacterCounter(chars_per_token=1.0)
def factory(requested_model: str) -> CharacterCounter:
calls.append(requested_model)
if not state["available"]:
raise RuntimeError("temporary factory outage")
return custom
register_tokenizer(model, factory=factory)
first = get_tokenizer(model)
with pytest.raises(ValueError, match="No tokenizer available"):
get_tokenizer(model, fallback=False)
state["available"] = True
recovered = get_tokenizer(model, fallback=False)
assert isinstance(first, EstimatingTokenCounter)
assert recovered is custom
assert calls == [model, model, model]
def test_list_registered_includes_model_factories_and_replacements(self, monkeypatch):
"""Registered model names should include both instances and factories."""
registry = TokenizerRegistry()
for attr in ("_tokenizers", "_factories", "_cache"):
monkeypatch.setattr(registry, attr, getattr(registry, attr).copy())
monkeypatch.setattr(
registry,
"_model_factories",
getattr(registry, "_model_factories", {}).copy(),
raising=False,
)
model = "Listed-Factory-Model"
register_tokenizer(model, factory=lambda _: CharacterCounter(chars_per_token=1.0))
assert model.lower() in registry.list_registered()
register_tokenizer(model, tokenizer=CharacterCounter(chars_per_token=2.0))
assert model.lower() in registry.list_registered()
register_tokenizer(model, factory=lambda _: CharacterCounter(chars_per_token=3.0))
registered = registry.list_registered()
assert model.lower() in registered
assert registered.count(model.lower()) == 1
def test_list_supported_models(self):
"""Test listing supported models."""
models = list_supported_models()
assert isinstance(models, dict)
assert "gpt-4o" in str(models) or "^gpt-4o" in str(models)
def test_clear_cache(self):
"""Test clearing tokenizer cache."""
# Get a tokenizer to populate cache
get_tokenizer("gpt-4o")
# Clear cache
TokenizerRegistry.clear_cache()
# Should still work after clearing
tokenizer = get_tokenizer("gpt-4o")
assert tokenizer is not None
class TestTokenCounterProtocol:
"""Tests for TokenCounter protocol."""
def test_tiktoken_implements_protocol(self):
"""Test TiktokenCounter implements protocol."""
counter = TiktokenCounter()
assert isinstance(counter, TokenCounter)
def test_estimating_implements_protocol(self):
"""Test EstimatingTokenCounter implements protocol."""
counter = EstimatingTokenCounter()
assert isinstance(counter, TokenCounter)
def test_character_implements_protocol(self):
"""Test CharacterCounter implements protocol."""
counter = CharacterCounter()
assert isinstance(counter, TokenCounter)
class TestBaseTokenizer:
"""Tests for BaseTokenizer base class."""
def test_message_overhead_constant(self):
"""Test message overhead constant."""
assert BaseTokenizer.MESSAGE_OVERHEAD == 4
def test_reply_overhead_constant(self):
"""Test reply overhead constant."""
assert BaseTokenizer.REPLY_OVERHEAD == 3
class TestMistralTokenizer:
"""Tests for Mistral tokenizer using official mistral-common."""
def test_is_available(self):
"""Test availability check."""
result = is_mistral_tokenizer_available()
assert isinstance(result, bool)
@pytest.mark.skipif(
not is_mistral_tokenizer_available(),
reason="mistral-common not installed",
)
def test_get_mistral_tokenizer_class(self):
"""Test getting MistralTokenizer class."""
MistralTokenizer = get_mistral_tokenizer()
assert MistralTokenizer is not None
assert hasattr(MistralTokenizer, "count_text")
@pytest.mark.skipif(
not is_mistral_tokenizer_available(),
reason="mistral-common not installed",
)
def test_init_default_model(self):
"""Test initialization with default model."""
MistralTokenizer = get_mistral_tokenizer()
counter = MistralTokenizer()
assert counter.model == "mistral-large"
assert counter.version == "v3"
@pytest.mark.skipif(
not is_mistral_tokenizer_available(),
reason="mistral-common not installed",
)
def test_init_mixtral_model(self):
"""Test initialization with Mixtral model (uses v1)."""
MistralTokenizer = get_mistral_tokenizer()
counter = MistralTokenizer("mixtral-8x7b")
assert counter.version == "v1"
@pytest.mark.skipif(
not is_mistral_tokenizer_available(),
reason="mistral-common not installed",
)
def test_count_text_empty(self):
"""Test counting empty text."""
MistralTokenizer = get_mistral_tokenizer()
counter = MistralTokenizer()
assert counter.count_text("") == 0
@pytest.mark.skipif(
not is_mistral_tokenizer_available(),
reason="mistral-common not installed",
)
def test_count_text_simple(self):
"""Test counting simple text."""
MistralTokenizer = get_mistral_tokenizer()
counter = MistralTokenizer()
count = counter.count_text("Hello, world!")
assert count > 0
assert count < 10
@pytest.mark.skipif(
not is_mistral_tokenizer_available(),
reason="mistral-common not installed",
)
def test_count_text_unicode(self):
"""Test counting text with unicode."""
MistralTokenizer = get_mistral_tokenizer()
counter = MistralTokenizer()
count = counter.count_text("Hello, 世界!")
assert count > 0
@pytest.mark.skipif(
not is_mistral_tokenizer_available(),
reason="mistral-common not installed",
)
def test_count_messages(self):
"""Test counting messages."""
MistralTokenizer = get_mistral_tokenizer()
counter = MistralTokenizer()
messages = [
{"role": "user", "content": "Hello!"},
{"role": "assistant", "content": "Hi there!"},
]
count = counter.count_messages(messages)
assert count > 0
@pytest.mark.skipif(
not is_mistral_tokenizer_available(),
reason="mistral-common not installed",
)
def test_count_messages_with_system(self):
"""Test counting messages with system prompt."""
MistralTokenizer = get_mistral_tokenizer()
counter = MistralTokenizer()
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Hello!"},
]
count = counter.count_messages(messages)
assert count > 0
@pytest.mark.skipif(
not is_mistral_tokenizer_available(),
reason="mistral-common not installed",
)
def test_encode_decode_roundtrip(self):
"""Test encode/decode roundtrip."""
MistralTokenizer = get_mistral_tokenizer()
counter = MistralTokenizer()
text = "Hello, world!"
tokens = counter.encode(text)
decoded = counter.decode(tokens)
assert decoded == text
@pytest.mark.skipif(
not is_mistral_tokenizer_available(),
reason="mistral-common not installed",
)
def test_implements_protocol(self):
"""Test MistralTokenizer implements TokenCounter protocol."""
MistralTokenizer = get_mistral_tokenizer()
counter = MistralTokenizer()
assert isinstance(counter, TokenCounter)
@pytest.mark.skipif(
not is_mistral_tokenizer_available(),
reason="mistral-common not installed",
)
def test_repr(self):
"""Test string representation."""
MistralTokenizer = get_mistral_tokenizer()
counter = MistralTokenizer("mistral-large")
assert "MistralTokenizer" in repr(counter)
assert "mistral-large" in repr(counter)
@pytest.mark.skipif(
not is_mistral_tokenizer_available(),
reason="mistral-common not installed",
)
def test_registry_returns_mistral_for_mistral_models(self):
"""Test registry returns Mistral tokenizer for Mistral models."""
tokenizer = get_tokenizer("mistral-large")
MistralTokenizer = get_mistral_tokenizer()
assert isinstance(tokenizer, MistralTokenizer)
@pytest.mark.skipif(
not is_mistral_tokenizer_available(),
reason="mistral-common not installed",
)
def test_registry_returns_mistral_for_mixtral(self):
"""Test registry returns Mistral tokenizer for Mixtral models."""
tokenizer = get_tokenizer("mixtral-8x7b")
MistralTokenizer = get_mistral_tokenizer()
assert isinstance(tokenizer, MistralTokenizer)
@pytest.mark.skipif(
not is_mistral_tokenizer_available(),
reason="mistral-common not installed",
)
def test_registry_returns_mistral_for_codestral(self):
"""Test registry returns Mistral tokenizer for Codestral models."""
tokenizer = get_tokenizer("codestral")
MistralTokenizer = get_mistral_tokenizer()
assert isinstance(tokenizer, MistralTokenizer)
class TestLargeToolBlobEstimation:
"""Oversized tool blobs are token-estimated without serializing them in full."""
def test_oversized_tool_blob_count_text_is_bounded(self, monkeypatch):
"""Regression: count_text over a multi-megabyte serialized blob froze the
event loop (~seconds). json.dumps itself is cheap; count_text over the
whole string is the cost, so its input must stay bounded for oversized
blobs.
"""
tok = EstimatingTokenCounter()
sizes: list[int] = []
real_count_text = tok.count_text
def spy(text):
sizes.append(len(text))
return real_count_text(text)
monkeypatch.setattr(tok, "count_text", spy)
messages = [
{
"role": "user",
"content": [
{"type": "tool_result", "content": {"small": "x"}},
{"type": "tool_result", "content": {"data": "A" * 4_000_000}},
],
}
]
tok.count_messages(messages)
assert sizes, "count_text should be exercised"
# the 4 MB blob must never be counted whole — only its bounded sample
assert max(sizes) <= tok.SAMPLE_CHARS + tok.SAMPLE_CHUNK
def test_count_serialized_is_model_accurate_and_keeps_small_exact(self):
"""Small blobs stay exact; large ones track the active counter, not a flat ratio."""
import json
tok = EstimatingTokenCounter(chars_per_token=3.5) # Claude-like ratio
small = {"k": "v"}
assert tok._count_serialized(small) == tok.count_text(json.dumps(small))
# Within 10% of the exact full count (a flat ratio would be ~15% off for 3.5).
big = {"k": "A" * 200_000}
exact = tok.count_text(json.dumps(big))
assert abs(tok._count_serialized(big) - exact) / exact < 0.10
def test_tool_result_list_recurses_into_image_block(self):
"""A tool that returns an image nests a base64 block inside a
`tool_result` list. Serializing it prices the base64 as text (a 50-200x
overcount); recursing into the block prices the image at ~1600 tokens."""
tok = EstimatingTokenCounter()
messages = [
{
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": "t1",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": "A" * 280_000, # ~280KB base64
},
}
],
}
],
}
]
count = tok.count_messages(messages)
# The image is priced structurally (~1600), not as a huge text blob.
assert count < 5_000
def test_oversized_estimate_never_overcounts(self):
"""R4 (prefer false negatives): a token-dense head + sparse tail must not
over-count. Counting per leaf cannot extrapolate a dense front slice to the
whole the way scaling one sample could.
"""
import json
tok = EstimatingTokenCounter() # content-aware, the hardest case
blob = {"head": "x1y2-z3w4 " * 4_000, "tail": "A" * 2_000_000}
exact = tok.count_text(json.dumps(blob))
assert tok._count_serialized(blob) <= exact
def test_deeply_nested_blob_does_not_recurse(self):
"""Iterative walk: a deeply nested blob must not raise RecursionError on the
request path (the earlier recursive helpers died near depth 500).
"""
deep: dict = {}
cur = deep
for _ in range(2_000):
cur["n"] = {}
cur = cur["n"]
cur["leaf"] = "x" * 60_000
assert EstimatingTokenCounter()._count_serialized(deep) >= 0
class TestThinkingBlockCounting:
"""Thinking blocks price their text plus the decoded signature bytes."""
def test_signature_prices_between_zero_and_the_json_catch_all(self):
counter = EstimatingTokenCounter()
text = "consider the failing test " * 20
block = {"type": "thinking", "thinking": text, "signature": "A" * 4000}
signed = {"role": "assistant", "content": [block]}
unsigned = {"role": "assistant", "content": [{"type": "thinking", "thinking": text}]}
# The signature is encrypted reasoning the server replays as billed
# input, so it adds to the text at decoded bytes / 4 (4000 * 3/4 / 4)...
delta = counter.count_messages([signed]) - counter.count_messages([unsigned])
assert delta == 750
# ...which is well under the JSON catch-all's base64-as-prose price.
assert delta < counter._count_serialized(block)
def test_omitted_display_block_is_not_free(self):
# display: "omitted" (the 5.x default) returns an empty thinking field
# and carries the whole reasoning in the signature.
counter = EstimatingTokenCounter()
omitted = {
"role": "assistant",
"content": [{"type": "thinking", "thinking": "", "signature": "A" * 4000}],
}
empty = {"role": "assistant", "content": [{"type": "thinking", "thinking": ""}]}
assert counter.count_messages([omitted]) > counter.count_messages([empty])
def test_provider_walker_shares_the_thinking_rule(self):
from headroom.tokenizers.base import count_content_blocks
counter = EstimatingTokenCounter()
text = "consider the failing test " * 20
signed = [{"type": "thinking", "thinking": text, "signature": "A" * 4000}]
unsigned = [{"type": "thinking", "thinking": text}]
assert count_content_blocks(signed, counter.count_text) == counter._count_content_parts(
signed
)
assert (
count_content_blocks(signed, counter.count_text)
- count_content_blocks(unsigned, counter.count_text)
== 750
)
class TestDictValuedTextFields:
"""Structured (non-string) text fields must not crash token counting.
Regression (GH #4273): a dict-valued ``text`` block reached
``tiktoken.encode`` unchanged and raised ``TypeError: unhashable type:
'dict'``, aborting compression for the whole request. The fix reuses
:func:`coerce_countable_text` so a non-string text-ish field is priced
like the JSON string it should have been.
"""
@pytest.mark.parametrize(
"counter_cls",
[TiktokenCounter, EstimatingTokenCounter],
ids=["tiktoken", "estimator"],
)
def test_dict_text_block_counts_like_coerced_string(self, counter_cls):
counter = counter_cls()
text_dict = {"message": "non-string provider text"}
expected = coerce_countable_text(text_dict)
bad = {"role": "user", "content": [{"type": "text", "text": text_dict}]}
good = {"role": "user", "content": [{"type": "text", "text": expected}]}
assert counter.count_messages([bad]) == counter.count_messages([good])
@pytest.mark.parametrize(
"counter_cls",
[TiktokenCounter, EstimatingTokenCounter],
ids=["tiktoken", "estimator"],
)
def test_dict_text_block_does_not_mutate_input(self, counter_cls):
import copy
counter = counter_cls()
message = {
"role": "user",
"content": [{"type": "text", "text": {"message": "non-string provider text"}}],
}
original = copy.deepcopy(message)
counter.count_messages([message])
assert message == original
@pytest.mark.parametrize(
"counter_cls",
[TiktokenCounter, EstimatingTokenCounter],
ids=["tiktoken", "estimator"],
)
@pytest.mark.parametrize(
"block",
[
{"type": "text", "text": {"message": "non-string provider text"}},
{"type": "image_url", "image_url": {"url": "data:image/png;base64," + "A" * 20000}},
],
ids=["text", "image"],
)
def test_unwrapped_content_block_counts_like_list_form(self, counter_cls, block):
# An image must keep its fixed estimate, not be priced as base64 text.
counter = counter_cls()
bare = {"role": "user", "content": block}
wrapped = {"role": "user", "content": [block]}
assert counter.count_messages([bare]) == counter.count_messages([wrapped])
@pytest.mark.parametrize(
"counter_cls",
[TiktokenCounter, EstimatingTokenCounter],
ids=["tiktoken", "estimator"],
)
def test_dict_role_counts_like_coerced_string(self, counter_cls):
counter = counter_cls()
role_dict = {"role": "user"}
expected = coerce_countable_text(role_dict)
bad = {"role": role_dict, "content": "hi"}
good = {"role": expected, "content": "hi"}
assert counter.count_messages([bad]) == counter.count_messages([good])
@pytest.mark.parametrize(
"counter_cls",
[TiktokenCounter, EstimatingTokenCounter],
ids=["tiktoken", "estimator"],
)
def test_dict_tool_result_content_counts_like_coerced_string(self, counter_cls):
counter = counter_cls()
tr_dict = {"output": "non-string provider text"}
expected = coerce_countable_text(tr_dict)
bad = {
"role": "user",
"content": [{"type": "tool_result", "content": tr_dict}],
}
good = {
"role": "user",
"content": [{"type": "tool_result", "content": expected}],
}
assert counter.count_messages([bad]) == counter.count_messages([good])
@pytest.mark.parametrize(
"counter_cls",
[TiktokenCounter, EstimatingTokenCounter],
ids=["tiktoken", "estimator"],
)
def test_dict_thinking_counts_like_coerced_string(self, counter_cls):
counter = counter_cls()
thinking_dict = {"reasoning": "non-string provider text"}
expected = coerce_countable_text(thinking_dict)
bad = {"role": "assistant", "content": [{"type": "thinking", "thinking": thinking_dict}]}
good = {"role": "assistant", "content": [{"type": "thinking", "thinking": expected}]}
assert counter.count_messages([bad]) == counter.count_messages([good])