mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
perf(tokenizer): memoize OpenAI token counts like the Anthropic counter (#3168)
## Description
`OpenAITokenCounter.count_text` re-ran `tiktoken.encode` on **every**
call:
```python
def count_text(self, text: str) -> int:
if not text:
return 0
try:
return len(self._encoding.encode(text))
except ValueError:
return len(self._encoding.encode(text, disallowed_special=()))
```
`AnthropicTokenCounter.count_text` already memoizes counts via
`TokenCountCache` (and its own comment even says the OpenAI counter
"Matches AnthropicTokenCounter.count_text" — but it did not: it had no
cache). Both counters are **per-model singletons**, reused across
requests (`OpenAIProvider`/`AnthropicProvider` cache them in
`self._token_counters`). So for OpenAI-model traffic, a stable system
prompt / conversation prefix, or a repeated tool result, was re-encoded
from scratch on every turn — and token counting runs on the request hot
path (often more than once per request across pipeline stages).
This adds the same `TokenCountCache` to `OpenAITokenCounter`, mirroring
the Anthropic counter. `count_text` is a pure function of its text, so a
replayed count is value-identical. The cache only admits strings `>=
256` chars (its existing admission policy), so tiny/rare strings — which
encode in microseconds — pay nothing and never evict the large entries
the cache exists for.
Benchmark (`OpenAITokenCounter("gpt-4o")`, an ~8 KB block, mean of 2000
calls):
```
re-encode each call : 928.7 us/call
cache hit : 0.1 us/call (~11000x faster)
same count : True
```
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/providers/openai.py`: `OpenAITokenCounter.__init__` now
creates a `TokenCountCache`; `count_text` checks/populates it and
delegates the actual encode to a new `_count_text_uncached` (which keeps
the existing `disallowed_special` fallback). Imported `TokenCountCache`.
- `tests/test_providers/test_openai.py`: added
`test_count_text_caches_large_strings` — a large string's count is
memoized and equals the uncached count; a short string stays below the
256-char admission floor.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
### Test Output
```text
tests/test_providers/test_openai.py -> 23 passed
uvx ruff@0.16.2 check headroom/providers/openai.py tests/test_providers/test_openai.py -> All checks passed!
uvx mypy@1.20.2 headroom/providers/openai.py -> Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
tiktoken installed, ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: benchmarked
`OpenAITokenCounter("gpt-4o").count_text` on an ~8 KB block, re-encoding
each call (928.7us) vs a cache hit (0.1us), confirming identical counts.
Removed the cache from `count_text` and ran `python -m pytest
tests/test_providers/test_openai.py -k caches_large_strings` -> failed
(`_count_cache.get(large)` was `None`); restored it -> 23 passed for the
file.
- Observed result: repeated counts of the same large text are served
from the cache instead of re-encoding, with the same value. Behavior for
uncached strings (including the `<|endoftext|>` literal-special-token
case) is unchanged.
- Not tested: an end-to-end proxy turn timing; the counter is exercised
directly, which is the object the pipeline resolves via
`OpenAIProvider.get_token_counter`.
## Runtime Rollout Safety
- Rollout-managed feature(s): none. This is an internal memoization in
the OpenAI token counter, not a rollout-channel-gated runtime feature.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no. Counts are identical; only
repeated counts of the same large text are faster. The cache is
per-counter, GIL-atomic (no lock), and bounded (entries + total chars).
- Kill switch / disable path: N/A.
- Unsafe override required: no.
- Qualification impact: less CPU spent re-encoding stable context on the
token-counting hot path for OpenAI-model traffic; counts unchanged.
- Rollback path: revert this PR; `count_text` returns to encoding on
every call.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal behavior, counts unchanged)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
This just brings the OpenAI counter in line with the Anthropic one,
which already carries `TokenCountCache` for exactly this reason. The
shared cache class is unchanged, so its admission/eviction bounds and
thread-safety argument apply identically here.
This commit is contained in:
@@ -17,7 +17,11 @@ from typing import Any, cast
|
||||
|
||||
from headroom import paths as _paths
|
||||
from headroom.pricing.litellm_pricing import estimate_cost_from_tokens
|
||||
from headroom.tokenizers.base import coerce_countable_text, count_content_blocks
|
||||
from headroom.tokenizers.base import (
|
||||
TokenCountCache,
|
||||
coerce_countable_text,
|
||||
count_content_blocks,
|
||||
)
|
||||
|
||||
from .base import Provider, TokenCounter
|
||||
|
||||
@@ -366,11 +370,26 @@ class OpenAITokenCounter:
|
||||
self.model = model
|
||||
encoding_name = _get_encoding_name_for_model(model, custom_encodings)
|
||||
self._encoding = _get_encoding(encoding_name)
|
||||
# count_text is a pure function of its text, and this counter is a
|
||||
# per-model singleton reused across requests (OpenAIProvider caches it),
|
||||
# so a stable prefix/system prompt or a repeated tool result is otherwise
|
||||
# re-encoded on every turn. Cache the count like AnthropicTokenCounter
|
||||
# already does (the cache only admits large strings, so tiny/rare ones
|
||||
# pay nothing).
|
||||
self._count_cache = TokenCountCache()
|
||||
|
||||
def count_text(self, text: str) -> int:
|
||||
"""Count tokens in text."""
|
||||
if not text:
|
||||
return 0
|
||||
cached = self._count_cache.get(text)
|
||||
if cached is not None:
|
||||
return cached
|
||||
count = self._count_text_uncached(text)
|
||||
self._count_cache.put(text, count)
|
||||
return count
|
||||
|
||||
def _count_text_uncached(self, text: str) -> int:
|
||||
try:
|
||||
return len(self._encoding.encode(text))
|
||||
except ValueError:
|
||||
|
||||
@@ -33,6 +33,27 @@ class TestOpenAITokenCounting:
|
||||
count = openai_tokenizer.count_text(text)
|
||||
assert count > openai_tokenizer.count_text("before after")
|
||||
|
||||
def test_count_text_caches_large_strings(self):
|
||||
"""count_text memoizes large strings so a stable prefix / repeated tool
|
||||
result is not re-encoded on every turn (the counter is a per-model
|
||||
singleton). Cached counts must equal the uncached count, and tiny
|
||||
strings must stay below the admission floor."""
|
||||
from headroom.providers.openai import OpenAITokenCounter
|
||||
|
||||
counter = OpenAITokenCounter("gpt-4o")
|
||||
large = "lorem ipsum dolor sit amet consectetur adipiscing elit " * 8 # > 256 chars
|
||||
assert len(large) >= 256
|
||||
|
||||
direct = counter._count_text_uncached(large)
|
||||
assert counter._count_cache.get(large) is None # cold before first count
|
||||
assert counter.count_text(large) == direct # value-identical to uncached
|
||||
assert counter._count_cache.get(large) == direct # now memoized
|
||||
|
||||
# A short string encodes in microseconds and stays below the floor, so
|
||||
# it never evicts the large entries the cache exists for.
|
||||
counter.count_text("hi there")
|
||||
assert counter._count_cache.get("hi there") is None
|
||||
|
||||
def test_count_messages_single(self, openai_tokenizer):
|
||||
messages = [{"role": "user", "content": "Hello"}]
|
||||
count = openai_tokenizer.count_messages(messages)
|
||||
|
||||
Reference in New Issue
Block a user