fix(proxy): bound the model-keyed tokenizer, beacon and litellm caches (#4497)

## Description

Three process-wide caches are keyed by the `model` field of the request
body and never evict, so a client that sends a new model name on every
request grows each of them without limit (#4479):

- `TokenizerRegistry._cache` in `headroom/tokenizers/registry.py`. Each
entry is a tokenizer that owns its own `TokenCountCache`.
- `_model_cache` behind the beacon's `_public_model` in
`headroom/telemetry/session.py`.
- `_resolved_model_cache` behind `resolve_litellm_model` in
`headroom/pricing/litellm_pricing.py`.

#4407 bounded the per-model maps in the cost and savings trackers. This
PR bounds these three caches the way
`savings_tracker._resolve_litellm_model` was bounded in #2860: a fixed
entry count with least-recently-used eviction. The caps are 64
tokenizers and 256 entries for each of the two name caches. A model in
steady use is never evicted. An evicted model is recomputed the next
time it is used, so the cap never changes a result.

Keys are not clipped. A cache key decides which tokenizer, pricing key
or beacon model a request gets, so clipping would let two different
model names share an entry and get the wrong answer. With the count
capped, the size of each entry is already limited by the request body
cap.

Fixes #4479

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/utils.py`: adds `remember_bounded(cache, key, value,
maxsize)`. It stores `key` as the newest entry of a plain dict and
evicts from the front, since dict order is recency order. It is safe
without a lock: if two threads race, a key may be evicted early or the
cache may sit one entry over the cap for a moment, but it never raises.
The dicts stay plain dicts so the existing tests that clear, seed or
`len()` `_resolved_model_cache` keep working.
- `headroom/tokenizers/registry.py`: adds
`TokenizerRegistry._CACHE_MAXSIZE = 64`. A cache hit refreshes the
entry's recency, and both the created-tokenizer path and the
estimation-fallback path insert through `remember_bounded`. `register()`
and `clear_cache()` are unchanged.
- `headroom/telemetry/session.py`: adds `_MODEL_CACHE_MAXSIZE = 256` and
the same hit/insert handling. Cached `None` results ("not a public
model") still count as hits. The "ponytail: unbounded dict" comment is
replaced.
- `headroom/pricing/litellm_pricing.py`: adds
`_RESOLVED_MODEL_CACHE_MAXSIZE = 256` and the same treatment in
`resolve_litellm_model`.
- `tests/test_bounded_model_caches.py`: covers eviction order for the
helper and, for each cache, that it stays bounded, that a model in
steady use survives (LRU, not FIFO), and that an evicted name is
recomputed. It also covers the registry's estimation-fallback path and a
dict resized by another thread mid-eviction, so every changed line is
exercised.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`): no new errors in the
touched modules. The only error reported is in
`headroom/memory/adapters/embedders.py:389`, which is the same on
`main`.
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_bounded_model_caches.py -q
11 passed in 3.23s

# the six cache tests against the original registry.py / session.py / litellm_pricing.py
$ git stash push headroom/tokenizers/registry.py headroom/telemetry/session.py headroom/pricing/litellm_pricing.py && pytest tests/test_bounded_model_caches.py -q
7 failed, 4 passed

# suites that touch these caches
$ pytest tests/test_tokenizers.py tests/test_tokenizer_encoding_resolution.py tests/test_tokenizers \
    tests/test_proxy_streaming_resilience.py tests/test_telemetry_session_selfcheck.py \
    tests/test_savings_tracker_litellm_resolution_cache.py tests/test_gateway_sidecar_ports.py \
    tests/test_beacon_schema_v2.py tests/test_beacon_gzip.py tests/test_bounded_model_caches.py -q
189 passed, 14 skipped

$ pytest tests/test_cost_pricing_warning_dedup.py tests/test_counterfactual_pricing.py tests/test_litellm_*.py tests/test_utils*.py tests/test_{pricing,telemetry,tokeniz,session,beacon,cost}*.py -q
660 passed, 18 skipped

$ ruff check . && ruff format --check .
All checks passed!
1835 files already formatted

# full suite (Linux, Python 3.12, --timeout=120)
$ pytest tests -q --timeout=120
8 failed, 17053 passed, 596 skipped
```

None of the 8 full-suite failures comes from this change:

- `test_mcp_reconcile::test_ordinary_install_does_not_adopt_serena`,
`test_wrap_sighup_proxy_leak::test_sighup_on_launch_tool_releases_the_proxy`,
both `test_cc_switch_reconciler` tests and
`test_proxy_uds::test_prepare_only_chmods_directories_it_creates` fail
the same way on an unmodified `main` checkout in this environment (no
installed `headroom` console script, sandbox permissions).
- `test_direct_chat_ccr_resolution.py` hangs at `TestClient(app)`
startup until the timeout, and a different case fails on each run. On
unmodified `main` it failed in 2 of 3 isolated runs.
- `test_offline_egress_chokepoint.py` and `test_stateless_no_disk.py`
pass when run on their own on this branch.

## Real Behavior Proof

- Environment: Linux x86_64 (Amazon Linux 2023), Python 3.12.14, this
branch on top of `main` at `17d38e56` (0.42.0), with the published
0.42.0 `headroom._core` wheel binary. No Rust toolchain was available
here, so `_core` was not rebuilt; this PR does not touch Rust.
- Exact command / steps: the reproduction from #4479, run against the
real modules with no stubs. After one warm-up call per cache, call
`get_tokenizer(f"gpt-4o-x{i}-" + "m" * 10_000).count_text(...)`,
`session._public_model(f"custom-{i}-" + "m" * 10_000)` and
`litellm_pricing.resolve_litellm_model(f"custom-{i}-" + "m" * 10_000)`
for 2,000 distinct names each. Measure `len(cache)` and the
`tracemalloc` growth across each loop. Then check that `gpt-4o` still
resolves through both name caches.
- Observed result: on `main`, all three caches grow to 2,001 entries
(40.7 MB, 20.2 MB and 20.2 MB traced). On this branch they stop at 64,
256 and 256 entries (1.3 MB, 2.6 MB and 2.6 MB), and `gpt-4o` still
resolves through both name caches:

```text
# main (17d38e56)
registry._cache entries: 2001 | traced growth: 40.7 MB
telemetry.session._model_cache entries: 2001 | traced growth: 20.2 MB
litellm_pricing._resolved_model_cache entries: 2001 | traced growth: 20.2 MB
public model still resolves: gpt-4o | gpt-4o

# this branch
registry._cache entries: 64 | traced growth: 1.3 MB
telemetry.session._model_cache entries: 256 | traced growth: 2.6 MB
litellm_pricing._resolved_model_cache entries: 256 | traced growth: 2.6 MB
public model still resolves: gpt-4o | gpt-4o
```

- Not tested: a long-running proxy process taking real HTTP traffic with
rotating model names (the run above calls the same functions the proxy
calls, in-process); Windows and macOS; `mypy headroom` for the whole
package (only the four touched modules were checked).

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: No change for normal traffic (far
fewer distinct models than the caps). Past the caps, the least recently
used entry is dropped and recomputed if that model comes back.
- Kill switch / disable path: Not applicable.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

- No user documentation needed: these caches are internal.
- Cap sizes: 256 matches
`savings_tracker._MODEL_RESOLUTION_CACHE_MAXSIZE`. The tokenizer
registry uses 64 because each entry owns a `TokenCountCache` of up to
8,000,000 characters.
- Open PR #4437 (#4434) edits `TokenizerRegistry.register()`. This PR
only touches `get()` and the class attributes, so the two should not
conflict.

---
🤖 Opened with [Superhuman](https://github.com/gaurav0107/superhuman), an
open-source contribution agent.


### Maintainer current-main requalification

Exact head `a29cc2d40f5c8111cd079949201d8660c8a50130`, upstream main
`e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`.

Updated against upstream/main e4a60ed7, resolving the tokenizer-registry
overlap by retaining main's registered-model factories and dispatch,
including its rule not to cache estimation after a failed model-specific
factory. Applied the contributor's bounded recency insertion at the
successful cache writes and ordinary fallback writes. No contributor
policy or correctness logic was replaced. Re-reviewed cache helper/call
sites and existing discussion; cached None, full model keys and
recomputation after eviction remain intact.

The combined tree passes 111 bounded-cache/tokenizer/encoding tests with
14 optional skips, plus the adjacent tokenizer load-timeout, telemetry
selfcheck and pricing-resolution suites. Ruff lint/format and whitespace
checks pass. Existing rotating-2,000-name runtime evidence remains
scoped to in-process module calls; an existing development native
extension was used locally, not rebuilt. The shared test-only Gemini
fixture repair was applied for independently established main failures;
all ten cases pass. Formal approval awaits exact-head hosted
qualification.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


Maintainer commit-message qualification repair: current head
bf0037dd29. Corrected only the maintainer
main-sync commit message to Conventional Commit format. The complete
file tree is byte-identical to the previously tested head; exact-head
hosted checks are being refreshed. Nothing merged.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Gaurav Dubey
2026-11-05 10:43:25 -06:00
committed by GitHub
co-authored by JerrettDavis
parent 3afaa97ffc
commit e51b8065a4
5 changed files with 267 additions and 14 deletions
+9 -3
View File
@@ -27,6 +27,7 @@ from headroom.pricing.litellm_model_resolution import (
pricing_lookup_candidates,
resolve_litellm_model_name,
)
from headroom.utils import remember_bounded
# litellm calls `dotenv.load_dotenv()` during its own import, which loads
# the project `.env` into `os.environ`. We don't want that side effect —
@@ -59,7 +60,10 @@ except ImportError:
litellm = None # type: ignore[assignment]
LITELLM_AVAILABLE = False
# Least-recently-used first. Keyed by the client-supplied model name, so the
# entry count is capped (#4479); an evicted name is simply resolved again.
_resolved_model_cache: dict[str, str] = {}
_RESOLVED_MODEL_CACHE_MAXSIZE = 256
logger = logging.getLogger("headroom.pricing")
@@ -113,14 +117,16 @@ def resolve_litellm_model(model: str) -> str:
the bare-prefix rules. Shared by the live (cost.py) and persisted
(savings_tracker) pricing paths so both figures price identically.
"""
if model in _resolved_model_cache:
return _resolved_model_cache[model]
cached = _resolved_model_cache.get(model)
if cached is not None:
remember_bounded(_resolved_model_cache, model, cached, _RESOLVED_MODEL_CACHE_MAXSIZE)
return cached
priced: str | None = None
alias = _static_alias_map()
if alias:
priced = _reduce_to_priced_key(alias.get(model, model))
resolved = priced if priced is not None else _resolve_litellm_model_uncached(model)
_resolved_model_cache[model] = resolved
remember_bounded(_resolved_model_cache, model, resolved, _RESOLVED_MODEL_CACHE_MAXSIZE)
return resolved
+13 -5
View File
@@ -120,6 +120,7 @@ from pathlib import Path
from typing import Any
from headroom.offline import pin_litellm_cost_map_local
from headroom.utils import remember_bounded
logger = logging.getLogger(__name__)
@@ -511,7 +512,11 @@ def _config_snapshot() -> dict[str, str]:
return out
# Least-recently-used first. ``model`` comes from the request body, so the
# entry count is capped (#4479); a model that falls out is simply looked up
# again the next time it is seen.
_model_cache: dict[str, str | None] = {}
_MODEL_CACHE_MAXSIZE = 256
def _public_model(model: str) -> str | None:
@@ -530,8 +535,13 @@ def _public_model(model: str) -> str | None:
"""
if not model:
return None
if model in _model_cache:
return _model_cache[model]
try:
cached = _model_cache[model]
except KeyError:
pass
else:
remember_bounded(_model_cache, model, cached, _MODEL_CACHE_MAXSIZE)
return cached
resolved: str | None = None
try:
@@ -546,9 +556,7 @@ def _public_model(model: str) -> str | None:
except Exception:
resolved = None
# ponytail: unbounded dict, but it is keyed by distinct model ids seen in
# one process — a handful. Cap it if a router ever fans out over hundreds.
_model_cache[model] = resolved
remember_bounded(_model_cache, model, resolved, _MODEL_CACHE_MAXSIZE)
return resolved
+13 -5
View File
@@ -11,6 +11,7 @@ import re
from collections.abc import Callable
from typing import TYPE_CHECKING
from ..utils import remember_bounded
from .base import TokenCounter
from .estimator import EstimatingTokenCounter
@@ -144,8 +145,13 @@ class TokenizerRegistry:
# Registered model factories (model -> factory function)
_model_factories: dict[str, Callable[[str], TokenCounter]] = {}
# Cache for auto-detected tokenizers
# Cache for auto-detected tokenizers, least-recently-used first. ``model``
# is the ``model`` field of a proxied request body, so the entry count is
# capped (#4479): each entry owns its own TokenCountCache, and a client
# sending a new name per request must not grow this without limit. Real
# traffic uses a handful of models, far below the cap.
_cache: dict[str, TokenCounter] = {}
_CACHE_MAXSIZE = 64
def __new__(cls) -> TokenizerRegistry:
"""Singleton pattern."""
@@ -198,8 +204,10 @@ class TokenizerRegistry:
# Check cache
cache_key = f"{model_lower}:{backend or 'auto'}"
if cache_key in registry._cache:
return registry._cache[cache_key]
cached = registry._cache.get(cache_key)
if cached is not None:
remember_bounded(registry._cache, cache_key, cached, cls._CACHE_MAXSIZE)
return cached
uses_model_factory = backend is None and model_lower in registry._model_factories
@@ -209,7 +217,7 @@ class TokenizerRegistry:
tokenizer = registry._model_factories[model_lower](model)
else:
tokenizer = registry._create_tokenizer(model, backend)
registry._cache[cache_key] = tokenizer
remember_bounded(registry._cache, cache_key, tokenizer, cls._CACHE_MAXSIZE)
return tokenizer
except Exception as e:
if fallback:
@@ -218,7 +226,7 @@ class TokenizerRegistry:
)
tokenizer = EstimatingTokenCounter()
if not uses_model_factory:
registry._cache[cache_key] = tokenizer
remember_bounded(registry._cache, cache_key, tokenizer, cls._CACHE_MAXSIZE)
return tokenizer
raise ValueError(f"No tokenizer available for {model}: {e}") from e
+39 -1
View File
@@ -9,7 +9,10 @@ import re
import uuid
import warnings
from datetime import datetime, timezone
from typing import Any
from typing import Any, TypeVar
_K = TypeVar("_K")
_V = TypeVar("_V")
# Marker format for Headroom modifications
MARKER_PREFIX = "<headroom:"
@@ -42,6 +45,41 @@ def compute_short_hash(data: str | bytes, length: int = 16) -> str:
return compute_hash(data)[:length]
def remember_bounded(cache: dict[_K, _V], key: _K, value: _V, maxsize: int) -> None:
"""Store ``key`` as the newest entry of ``cache``, keeping at most ``maxsize``.
``cache`` is a plain dict used as a least-recently-used cache: insertion
order is recency order, so the first key is the stalest and is evicted
first. Call this again on a cache hit to mark the key recently used.
Meant for caches keyed by client-controlled strings, such as a proxied
request's ``model`` field: the entry count stays bounded however many
distinct keys arrive, and an evicted key is recomputed on its next use
rather than answered wrongly.
Safe to call from several threads without a lock: a race can evict a key
early or leave the cache briefly one entry over ``maxsize``, never raise.
Args:
cache: The dict to update in place.
key: Cache key.
value: Value to store.
maxsize: Maximum number of entries. ``<= 0`` stores nothing.
"""
if maxsize <= 0:
cache.clear()
return
cache.pop(key, None)
while len(cache) >= maxsize:
try:
oldest = next(iter(cache))
except (StopIteration, RuntimeError):
# Emptied, or resized by another thread mid-iteration.
break
cache.pop(oldest, None)
cache[key] = value
def fast_hash(data: str | bytes, length: int = 16) -> str:
"""Fast non-cryptographic content hash for caches and dedup.
+193
View File
@@ -0,0 +1,193 @@
"""Regression: caches keyed by the request's ``model`` string stay bounded (#4479).
``model`` comes straight off a proxied request body, so a client that sends a
new name on every request must not be able to grow a process-wide cache
without limit. #4407 bounded the per-model maps in the cost and savings
trackers; these tests pin the same property for the three remaining caches:
- ``TokenizerRegistry._cache`` (``headroom/tokenizers/registry.py``)
- the beacon's ``_model_cache`` behind ``_public_model``
(``headroom/telemetry/session.py``)
- ``_resolved_model_cache`` behind ``resolve_litellm_model``
(``headroom/pricing/litellm_pricing.py``)
Each is least-recently-used: a name in steady use is never evicted, and an
evicted name is recomputed on its next use rather than answered wrongly.
"""
from __future__ import annotations
import pytest
from headroom.pricing import litellm_pricing as lp
from headroom.telemetry import session
from headroom.tokenizers.estimator import EstimatingTokenCounter
from headroom.tokenizers.registry import TokenizerRegistry
from headroom.utils import remember_bounded
# --------------------------------------------------------------------------
# remember_bounded
# --------------------------------------------------------------------------
def test_remember_bounded_evicts_least_recently_inserted():
cache: dict[str, int] = {}
for i in range(5):
remember_bounded(cache, f"k{i}", i, maxsize=3)
assert list(cache) == ["k2", "k3", "k4"]
def test_remember_bounded_reinsert_moves_key_to_newest():
cache: dict[str, int] = {}
for i in range(3):
remember_bounded(cache, f"k{i}", i, maxsize=3)
remember_bounded(cache, "k0", 0, maxsize=3) # a hit re-records the key
remember_bounded(cache, "k3", 3, maxsize=3)
assert list(cache) == ["k2", "k0", "k3"]
def test_remember_bounded_nonpositive_maxsize_stores_nothing():
cache: dict[str, int] = {"stale": 1}
remember_bounded(cache, "k", 1, maxsize=0)
assert cache == {}
def test_remember_bounded_tolerates_a_dict_resized_by_another_thread():
class ResizedDuringIteration(dict):
def __iter__(self):
raise RuntimeError("dictionary changed size during iteration")
cache = ResizedDuringIteration({"k0": 0, "k1": 1})
remember_bounded(cache, "k2", 2, maxsize=2)
# The eviction is skipped for this call rather than raising on the request
# path; the next insert that can iterate trims the cache again.
assert dict.__getitem__(cache, "k2") == 2
# --------------------------------------------------------------------------
# TokenizerRegistry._cache
# --------------------------------------------------------------------------
@pytest.fixture
def clean_tokenizer_cache():
TokenizerRegistry.clear_cache()
yield
TokenizerRegistry.clear_cache()
def test_tokenizer_registry_cache_is_bounded(clean_tokenizer_cache):
cap = TokenizerRegistry._CACHE_MAXSIZE
for i in range(cap + 50):
TokenizerRegistry.get(f"client-model-{i}-" + "m" * 1_000)
assert len(TokenizerRegistry._cache) == cap
def test_tokenizer_registry_estimation_fallback_is_bounded(clean_tokenizer_cache):
cap = TokenizerRegistry._CACHE_MAXSIZE
for i in range(cap + 10):
tokenizer = TokenizerRegistry.get(f"client-model-{i}", backend="no-such-backend")
assert isinstance(tokenizer, EstimatingTokenCounter)
assert len(TokenizerRegistry._cache) == cap
def test_tokenizer_registry_keeps_a_model_in_steady_use(clean_tokenizer_cache):
cap = TokenizerRegistry._CACHE_MAXSIZE
hot = TokenizerRegistry.get("gpt-4o")
for i in range(cap + 50):
TokenizerRegistry.get(f"client-model-{i}")
assert TokenizerRegistry.get("gpt-4o") is hot
oldest = TokenizerRegistry.get("client-model-0")
assert "client-model-0:auto" in TokenizerRegistry._cache
assert oldest is TokenizerRegistry.get("client-model-0")
# --------------------------------------------------------------------------
# telemetry.session._model_cache
# --------------------------------------------------------------------------
@pytest.fixture
def clean_public_model_cache():
session._model_cache.clear()
yield
session._model_cache.clear()
def test_public_model_cache_is_bounded(clean_public_model_cache):
cap = session._MODEL_CACHE_MAXSIZE
for i in range(cap + 50):
assert session._public_model(f"custom-{i}-" + "m" * 1_000) is None
assert len(session._model_cache) == cap
def test_public_model_evicted_name_is_recomputed(clean_public_model_cache, monkeypatch):
calls: list[str] = []
real_pin = session.pin_litellm_cost_map_local
def counting_pin() -> None:
calls.append("probe")
real_pin()
monkeypatch.setattr(session, "pin_litellm_cost_map_local", counting_pin)
cap = session._MODEL_CACHE_MAXSIZE
session._public_model("seed-model")
for i in range(cap):
session._public_model(f"custom-{i}")
probes_before = len(calls)
assert session._public_model("seed-model") is None
assert len(calls) == probes_before + 1
assert len(session._model_cache) == cap
# --------------------------------------------------------------------------
# litellm_pricing._resolved_model_cache
# --------------------------------------------------------------------------
@pytest.fixture
def counted_resolution(monkeypatch):
lp._resolved_model_cache.clear()
calls: dict[str, int] = {}
def fake_uncached(model: str) -> str:
calls[model] = calls.get(model, 0) + 1
return f"resolved/{model}"
monkeypatch.setattr(lp, "_resolve_litellm_model_uncached", fake_uncached)
monkeypatch.delenv("HEADROOM_MODEL_ALIAS_MAP", raising=False)
yield calls
lp._resolved_model_cache.clear()
def test_resolved_model_cache_is_bounded(counted_resolution):
cap = lp._RESOLVED_MODEL_CACHE_MAXSIZE
for i in range(cap + 50):
assert lp.resolve_litellm_model(f"custom-{i}") == f"resolved/custom-{i}"
assert len(lp._resolved_model_cache) == cap
def test_resolved_model_cache_is_lru(counted_resolution):
cap = lp._RESOLVED_MODEL_CACHE_MAXSIZE
lp.resolve_litellm_model("hot-model")
lp.resolve_litellm_model("cold-model")
for i in range(cap):
lp.resolve_litellm_model(f"custom-{i}")
lp.resolve_litellm_model("hot-model")
assert counted_resolution["hot-model"] == 1
assert lp.resolve_litellm_model("cold-model") == "resolved/cold-model"
assert counted_resolution["cold-model"] == 2