perf(tokenizer): memoize OpenAI token counts like the Anthropic counter (#3168)

## Description

`OpenAITokenCounter.count_text` re-ran `tiktoken.encode` on **every**
call:

```python
def count_text(self, text: str) -> int:
    if not text:
        return 0
    try:
        return len(self._encoding.encode(text))
    except ValueError:
        return len(self._encoding.encode(text, disallowed_special=()))
```

`AnthropicTokenCounter.count_text` already memoizes counts via
`TokenCountCache` (and its own comment even says the OpenAI counter
"Matches AnthropicTokenCounter.count_text" — but it did not: it had no
cache). Both counters are **per-model singletons**, reused across
requests (`OpenAIProvider`/`AnthropicProvider` cache them in
`self._token_counters`). So for OpenAI-model traffic, a stable system
prompt / conversation prefix, or a repeated tool result, was re-encoded
from scratch on every turn — and token counting runs on the request hot
path (often more than once per request across pipeline stages).

This adds the same `TokenCountCache` to `OpenAITokenCounter`, mirroring
the Anthropic counter. `count_text` is a pure function of its text, so a
replayed count is value-identical. The cache only admits strings `>=
256` chars (its existing admission policy), so tiny/rare strings — which
encode in microseconds — pay nothing and never evict the large entries
the cache exists for.

Benchmark (`OpenAITokenCounter("gpt-4o")`, an ~8 KB block, mean of 2000
calls):

```
re-encode each call : 928.7 us/call
cache hit           :   0.1 us/call   (~11000x faster)
same count          : True
```

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/providers/openai.py`: `OpenAITokenCounter.__init__` now
creates a `TokenCountCache`; `count_text` checks/populates it and
delegates the actual encode to a new `_count_text_uncached` (which keeps
the existing `disallowed_special` fallback). Imported `TokenCountCache`.
- `tests/test_providers/test_openai.py`: added
`test_count_text_caches_large_strings` — a large string's count is
memoized and equals the uncached count; a short string stays below the
256-char admission floor.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
tests/test_providers/test_openai.py  ->  23 passed
uvx ruff@0.16.2 check headroom/providers/openai.py tests/test_providers/test_openai.py  ->  All checks passed!
uvx mypy@1.20.2 headroom/providers/openai.py  ->  Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
tiktoken installed, ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: benchmarked
`OpenAITokenCounter("gpt-4o").count_text` on an ~8 KB block, re-encoding
each call (928.7us) vs a cache hit (0.1us), confirming identical counts.
Removed the cache from `count_text` and ran `python -m pytest
tests/test_providers/test_openai.py -k caches_large_strings` -> failed
(`_count_cache.get(large)` was `None`); restored it -> 23 passed for the
file.
- Observed result: repeated counts of the same large text are served
from the cache instead of re-encoding, with the same value. Behavior for
uncached strings (including the `<|endoftext|>` literal-special-token
case) is unchanged.
- Not tested: an end-to-end proxy turn timing; the counter is exercised
directly, which is the object the pipeline resolves via
`OpenAIProvider.get_token_counter`.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is an internal memoization in
the OpenAI token counter, not a rollout-channel-gated runtime feature.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no. Counts are identical; only
repeated counts of the same large text are faster. The cache is
per-counter, GIL-atomic (no lock), and bounded (entries + total chars).
- Kill switch / disable path: N/A.
- Unsafe override required: no.
- Qualification impact: less CPU spent re-encoding stable context on the
token-counting hot path for OpenAI-model traffic; counts unchanged.
- Rollback path: revert this PR; `count_text` returns to encoding on
every call.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal behavior, counts unchanged)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

This just brings the OpenAI counter in line with the Anthropic one,
which already carries `TokenCountCache` for exactly this reason. The
shared cache class is unchanged, so its admission/eviction bounds and
thread-safety argument apply identically here.
This commit is contained in:
Abhay Singh
2026-09-30 11:32:14 -05:00
committed by GitHub
parent fed7281120
commit 0a2c80d520
2 changed files with 41 additions and 1 deletions
+20 -1
View File
@@ -17,7 +17,11 @@ from typing import Any, cast
from headroom import paths as _paths
from headroom.pricing.litellm_pricing import estimate_cost_from_tokens
from headroom.tokenizers.base import coerce_countable_text, count_content_blocks
from headroom.tokenizers.base import (
TokenCountCache,
coerce_countable_text,
count_content_blocks,
)
from .base import Provider, TokenCounter
@@ -366,11 +370,26 @@ class OpenAITokenCounter:
self.model = model
encoding_name = _get_encoding_name_for_model(model, custom_encodings)
self._encoding = _get_encoding(encoding_name)
# count_text is a pure function of its text, and this counter is a
# per-model singleton reused across requests (OpenAIProvider caches it),
# so a stable prefix/system prompt or a repeated tool result is otherwise
# re-encoded on every turn. Cache the count like AnthropicTokenCounter
# already does (the cache only admits large strings, so tiny/rare ones
# pay nothing).
self._count_cache = TokenCountCache()
def count_text(self, text: str) -> int:
"""Count tokens in text."""
if not text:
return 0
cached = self._count_cache.get(text)
if cached is not None:
return cached
count = self._count_text_uncached(text)
self._count_cache.put(text, count)
return count
def _count_text_uncached(self, text: str) -> int:
try:
return len(self._encoding.encode(text))
except ValueError:
+21
View File
@@ -33,6 +33,27 @@ class TestOpenAITokenCounting:
count = openai_tokenizer.count_text(text)
assert count > openai_tokenizer.count_text("before after")
def test_count_text_caches_large_strings(self):
"""count_text memoizes large strings so a stable prefix / repeated tool
result is not re-encoded on every turn (the counter is a per-model
singleton). Cached counts must equal the uncached count, and tiny
strings must stay below the admission floor."""
from headroom.providers.openai import OpenAITokenCounter
counter = OpenAITokenCounter("gpt-4o")
large = "lorem ipsum dolor sit amet consectetur adipiscing elit " * 8 # > 256 chars
assert len(large) >= 256
direct = counter._count_text_uncached(large)
assert counter._count_cache.get(large) is None # cold before first count
assert counter.count_text(large) == direct # value-identical to uncached
assert counter._count_cache.get(large) == direct # now memoized
# A short string encodes in microseconds and stays below the floor, so
# it never evicts the large entries the cache exists for.
counter.count_text("hi there")
assert counter._count_cache.get("hi there") is None
def test_count_messages_single(self, openai_tokenizer):
messages = [{"role": "user", "content": "Hello"}]
count = openai_tokenizer.count_messages(messages)