fix(tokenizers): don't cache token-count inputs larger than max_chars (#4469)

## Description

`TokenCountCache.put` only checked the size budget before admitting a
new key, never the size of the key itself. A single text longer than
`max_chars` (8M chars by default) was stored and stayed alive until the
next new-key put cleared the cache, pinning a large request body in
memory for the tokenizer's lifetime.

Fixes #4264

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/tokenizers/base.py`: `put` returns early when `len(text) >
max_chars`. The check sits before the eviction clear, so warm entries
are kept. Re-count branch and soft budget for ordinary entries are
unchanged.
- `tests/test_token_count_cache.py`: oversized text is not cached, warm
entry survives an oversized put, text of exactly `max_chars` is still
admitted.

## Testing

- [x] Unit tests added/updated
- [x] Ran tests locally

```
$ python -m pytest tests/test_token_count_cache.py -q
============================= 21 passed in 2.57s ==============================
$ ruff check headroom/tokenizers/base.py tests/test_token_count_cache.py
All checks passed!
$ ruff format --check headroom/tokenizers/base.py tests/test_token_count_cache.py
2 files already formatted
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13, real `EstimatingTokenCounter`,
branch at 1ffe0993 vs upstream main `base.py`
- Exact command / steps: count a 9,600,000-char string via
`EstimatingTokenCounter().count_text(s)` under `tracemalloc`, then `del
s; gc.collect()` and read retained bytes
- Observed result: upstream main: count 2400000, retained_bytes 9600105
(string pinned by the cache). This branch: count 2400000, retained_bytes
32. Same token count, oversized body released.
- Not tested: full proxy end-to-end memory profile under live traffic

```
$ python repro.py   # upstream main base.py
count 2400000 retained_bytes 9600105
$ python repro.py   # this branch
count 2400000 retained_bytes 32
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: stable
- Stable/default behavior changed: only oversized texts skip the count
cache; returned counts are identical
- Kill switch / disable path: none needed; revert the commit
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4469"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Parideboy
2026-10-28 23:33:57 -05:00
committed by GitHub
co-authored by Claude Opus 5.5 JD Davis
parent 50af2ff7a8
commit aebb729ef3
2 changed files with 22 additions and 0 deletions
+3
View File
@@ -78,12 +78,15 @@ class TokenCountCache:
must not run the eviction check (a dict hit would clear the whole
cache and force a full re-encode of every cached message on the
next miss — the storm this exists to prevent).
Oversized texts are never admitted, so warm entries survive.
"""
if text in self._counts:
self._counts[text] = count
return
if len(text) < self._min_chars:
return
if len(text) > self._max_chars:
return
if len(self._counts) >= self._max_entries or self._chars >= self._max_chars:
self._counts.clear()
self._chars = 0
+19
View File
@@ -157,3 +157,22 @@ def test_recput_of_below_min_key_never_admits_and_never_evicts() -> None:
assert cache._counts == {"aaaaaaaaaa-1": 10}
cache.put("x", 5) # re-put: still no admission, no eviction
assert cache._counts == {"aaaaaaaaaa-1": 10}
def test_oversized_text_is_not_cached() -> None:
cache = TokenCountCache(min_chars=1, max_chars=10)
cache.put("x" * 20, 5)
assert cache.get("x" * 20) is None
def test_warm_entry_survives_oversized_put() -> None:
cache = TokenCountCache(min_chars=1, max_chars=10)
cache.put("warm", 7)
cache.put("x" * 20, 5)
assert cache.get("warm") == 7
def test_text_of_exactly_max_chars_is_admitted() -> None:
cache = TokenCountCache(min_chars=1, max_chars=10)
cache.put("x" * 10, 5)
assert cache.get("x" * 10) == 5