mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(tokenizers): don't cache token-count inputs larger than max_chars (#4469)
## Description
`TokenCountCache.put` only checked the size budget before admitting a
new key, never the size of the key itself. A single text longer than
`max_chars` (8M chars by default) was stored and stayed alive until the
next new-key put cleared the cache, pinning a large request body in
memory for the tokenizer's lifetime.
Fixes #4264
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/tokenizers/base.py`: `put` returns early when `len(text) >
max_chars`. The check sits before the eviction clear, so warm entries
are kept. Re-count branch and soft budget for ordinary entries are
unchanged.
- `tests/test_token_count_cache.py`: oversized text is not cached, warm
entry survives an oversized put, text of exactly `max_chars` is still
admitted.
## Testing
- [x] Unit tests added/updated
- [x] Ran tests locally
```
$ python -m pytest tests/test_token_count_cache.py -q
============================= 21 passed in 2.57s ==============================
$ ruff check headroom/tokenizers/base.py tests/test_token_count_cache.py
All checks passed!
$ ruff format --check headroom/tokenizers/base.py tests/test_token_count_cache.py
2 files already formatted
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.13, real `EstimatingTokenCounter`,
branch at 1ffe0993 vs upstream main `base.py`
- Exact command / steps: count a 9,600,000-char string via
`EstimatingTokenCounter().count_text(s)` under `tracemalloc`, then `del
s; gc.collect()` and read retained bytes
- Observed result: upstream main: count 2400000, retained_bytes 9600105
(string pinned by the cache). This branch: count 2400000, retained_bytes
32. Same token count, oversized body released.
- Not tested: full proxy end-to-end memory profile under live traffic
```
$ python repro.py # upstream main base.py
count 2400000 retained_bytes 9600105
$ python repro.py # this branch
count 2400000 retained_bytes 32
```
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: stable
- Stable/default behavior changed: only oversized texts skip the count
cache; returned counts are identical
- Kill switch / disable path: none needed; revert the commit
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4469"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JD Davis
parent
50af2ff7a8
commit
aebb729ef3
@@ -78,12 +78,15 @@ class TokenCountCache:
|
||||
must not run the eviction check (a dict hit would clear the whole
|
||||
cache and force a full re-encode of every cached message on the
|
||||
next miss — the storm this exists to prevent).
|
||||
Oversized texts are never admitted, so warm entries survive.
|
||||
"""
|
||||
if text in self._counts:
|
||||
self._counts[text] = count
|
||||
return
|
||||
if len(text) < self._min_chars:
|
||||
return
|
||||
if len(text) > self._max_chars:
|
||||
return
|
||||
if len(self._counts) >= self._max_entries or self._chars >= self._max_chars:
|
||||
self._counts.clear()
|
||||
self._chars = 0
|
||||
|
||||
@@ -157,3 +157,22 @@ def test_recput_of_below_min_key_never_admits_and_never_evicts() -> None:
|
||||
assert cache._counts == {"aaaaaaaaaa-1": 10}
|
||||
cache.put("x", 5) # re-put: still no admission, no eviction
|
||||
assert cache._counts == {"aaaaaaaaaa-1": 10}
|
||||
|
||||
|
||||
def test_oversized_text_is_not_cached() -> None:
|
||||
cache = TokenCountCache(min_chars=1, max_chars=10)
|
||||
cache.put("x" * 20, 5)
|
||||
assert cache.get("x" * 20) is None
|
||||
|
||||
|
||||
def test_warm_entry_survives_oversized_put() -> None:
|
||||
cache = TokenCountCache(min_chars=1, max_chars=10)
|
||||
cache.put("warm", 7)
|
||||
cache.put("x" * 20, 5)
|
||||
assert cache.get("warm") == 7
|
||||
|
||||
|
||||
def test_text_of_exactly_max_chars_is_admitted() -> None:
|
||||
cache = TokenCountCache(min_chars=1, max_chars=10)
|
||||
cache.put("x" * 10, 5)
|
||||
assert cache.get("x" * 10) == 5
|
||||
|
||||
Reference in New Issue
Block a user