Files
headroom/tests/test_bounded_model_caches.py
Gaurav DubeyandJerrettDavis e51b8065a4 fix(proxy): bound the model-keyed tokenizer, beacon and litellm caches (#4497)
## Description

Three process-wide caches are keyed by the `model` field of the request
body and never evict, so a client that sends a new model name on every
request grows each of them without limit (#4479):

- `TokenizerRegistry._cache` in `headroom/tokenizers/registry.py`. Each
entry is a tokenizer that owns its own `TokenCountCache`.
- `_model_cache` behind the beacon's `_public_model` in
`headroom/telemetry/session.py`.
- `_resolved_model_cache` behind `resolve_litellm_model` in
`headroom/pricing/litellm_pricing.py`.

#4407 bounded the per-model maps in the cost and savings trackers. This
PR bounds these three caches the way
`savings_tracker._resolve_litellm_model` was bounded in #2860: a fixed
entry count with least-recently-used eviction. The caps are 64
tokenizers and 256 entries for each of the two name caches. A model in
steady use is never evicted. An evicted model is recomputed the next
time it is used, so the cap never changes a result.

Keys are not clipped. A cache key decides which tokenizer, pricing key
or beacon model a request gets, so clipping would let two different
model names share an entry and get the wrong answer. With the count
capped, the size of each entry is already limited by the request body
cap.

Fixes #4479

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/utils.py`: adds `remember_bounded(cache, key, value,
maxsize)`. It stores `key` as the newest entry of a plain dict and
evicts from the front, since dict order is recency order. It is safe
without a lock: if two threads race, a key may be evicted early or the
cache may sit one entry over the cap for a moment, but it never raises.
The dicts stay plain dicts so the existing tests that clear, seed or
`len()` `_resolved_model_cache` keep working.
- `headroom/tokenizers/registry.py`: adds
`TokenizerRegistry._CACHE_MAXSIZE = 64`. A cache hit refreshes the
entry's recency, and both the created-tokenizer path and the
estimation-fallback path insert through `remember_bounded`. `register()`
and `clear_cache()` are unchanged.
- `headroom/telemetry/session.py`: adds `_MODEL_CACHE_MAXSIZE = 256` and
the same hit/insert handling. Cached `None` results ("not a public
model") still count as hits. The "ponytail: unbounded dict" comment is
replaced.
- `headroom/pricing/litellm_pricing.py`: adds
`_RESOLVED_MODEL_CACHE_MAXSIZE = 256` and the same treatment in
`resolve_litellm_model`.
- `tests/test_bounded_model_caches.py`: covers eviction order for the
helper and, for each cache, that it stays bounded, that a model in
steady use survives (LRU, not FIFO), and that an evicted name is
recomputed. It also covers the registry's estimation-fallback path and a
dict resized by another thread mid-eviction, so every changed line is
exercised.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`): no new errors in the
touched modules. The only error reported is in
`headroom/memory/adapters/embedders.py:389`, which is the same on
`main`.
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_bounded_model_caches.py -q
11 passed in 3.23s

# the six cache tests against the original registry.py / session.py / litellm_pricing.py
$ git stash push headroom/tokenizers/registry.py headroom/telemetry/session.py headroom/pricing/litellm_pricing.py && pytest tests/test_bounded_model_caches.py -q
7 failed, 4 passed

# suites that touch these caches
$ pytest tests/test_tokenizers.py tests/test_tokenizer_encoding_resolution.py tests/test_tokenizers \
    tests/test_proxy_streaming_resilience.py tests/test_telemetry_session_selfcheck.py \
    tests/test_savings_tracker_litellm_resolution_cache.py tests/test_gateway_sidecar_ports.py \
    tests/test_beacon_schema_v2.py tests/test_beacon_gzip.py tests/test_bounded_model_caches.py -q
189 passed, 14 skipped

$ pytest tests/test_cost_pricing_warning_dedup.py tests/test_counterfactual_pricing.py tests/test_litellm_*.py tests/test_utils*.py tests/test_{pricing,telemetry,tokeniz,session,beacon,cost}*.py -q
660 passed, 18 skipped

$ ruff check . && ruff format --check .
All checks passed!
1835 files already formatted

# full suite (Linux, Python 3.12, --timeout=120)
$ pytest tests -q --timeout=120
8 failed, 17053 passed, 596 skipped
```

None of the 8 full-suite failures comes from this change:

- `test_mcp_reconcile::test_ordinary_install_does_not_adopt_serena`,
`test_wrap_sighup_proxy_leak::test_sighup_on_launch_tool_releases_the_proxy`,
both `test_cc_switch_reconciler` tests and
`test_proxy_uds::test_prepare_only_chmods_directories_it_creates` fail
the same way on an unmodified `main` checkout in this environment (no
installed `headroom` console script, sandbox permissions).
- `test_direct_chat_ccr_resolution.py` hangs at `TestClient(app)`
startup until the timeout, and a different case fails on each run. On
unmodified `main` it failed in 2 of 3 isolated runs.
- `test_offline_egress_chokepoint.py` and `test_stateless_no_disk.py`
pass when run on their own on this branch.

## Real Behavior Proof

- Environment: Linux x86_64 (Amazon Linux 2023), Python 3.12.14, this
branch on top of `main` at `17d38e56` (0.42.0), with the published
0.42.0 `headroom._core` wheel binary. No Rust toolchain was available
here, so `_core` was not rebuilt; this PR does not touch Rust.
- Exact command / steps: the reproduction from #4479, run against the
real modules with no stubs. After one warm-up call per cache, call
`get_tokenizer(f"gpt-4o-x{i}-" + "m" * 10_000).count_text(...)`,
`session._public_model(f"custom-{i}-" + "m" * 10_000)` and
`litellm_pricing.resolve_litellm_model(f"custom-{i}-" + "m" * 10_000)`
for 2,000 distinct names each. Measure `len(cache)` and the
`tracemalloc` growth across each loop. Then check that `gpt-4o` still
resolves through both name caches.
- Observed result: on `main`, all three caches grow to 2,001 entries
(40.7 MB, 20.2 MB and 20.2 MB traced). On this branch they stop at 64,
256 and 256 entries (1.3 MB, 2.6 MB and 2.6 MB), and `gpt-4o` still
resolves through both name caches:

```text
# main (17d38e56)
registry._cache entries: 2001 | traced growth: 40.7 MB
telemetry.session._model_cache entries: 2001 | traced growth: 20.2 MB
litellm_pricing._resolved_model_cache entries: 2001 | traced growth: 20.2 MB
public model still resolves: gpt-4o | gpt-4o

# this branch
registry._cache entries: 64 | traced growth: 1.3 MB
telemetry.session._model_cache entries: 256 | traced growth: 2.6 MB
litellm_pricing._resolved_model_cache entries: 256 | traced growth: 2.6 MB
public model still resolves: gpt-4o | gpt-4o
```

- Not tested: a long-running proxy process taking real HTTP traffic with
rotating model names (the run above calls the same functions the proxy
calls, in-process); Windows and macOS; `mypy headroom` for the whole
package (only the four touched modules were checked).

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: No change for normal traffic (far
fewer distinct models than the caps). Past the caps, the least recently
used entry is dropped and recomputed if that model comes back.
- Kill switch / disable path: Not applicable.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

- No user documentation needed: these caches are internal.
- Cap sizes: 256 matches
`savings_tracker._MODEL_RESOLUTION_CACHE_MAXSIZE`. The tokenizer
registry uses 64 because each entry owns a `TokenCountCache` of up to
8,000,000 characters.
- Open PR #4437 (#4434) edits `TokenizerRegistry.register()`. This PR
only touches `get()` and the class attributes, so the two should not
conflict.

---
🤖 Opened with [Superhuman](https://github.com/gaurav0107/superhuman), an
open-source contribution agent.


### Maintainer current-main requalification

Exact head `a29cc2d40f5c8111cd079949201d8660c8a50130`, upstream main
`e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`.

Updated against upstream/main e4a60ed7, resolving the tokenizer-registry
overlap by retaining main's registered-model factories and dispatch,
including its rule not to cache estimation after a failed model-specific
factory. Applied the contributor's bounded recency insertion at the
successful cache writes and ordinary fallback writes. No contributor
policy or correctness logic was replaced. Re-reviewed cache helper/call
sites and existing discussion; cached None, full model keys and
recomputation after eviction remain intact.

The combined tree passes 111 bounded-cache/tokenizer/encoding tests with
14 optional skips, plus the adjacent tokenizer load-timeout, telemetry
selfcheck and pricing-resolution suites. Ruff lint/format and whitespace
checks pass. Existing rotating-2,000-name runtime evidence remains
scoped to in-process module calls; an existing development native
extension was used locally, not rebuilt. The shared test-only Gemini
fixture repair was applied for independently established main failures;
all ten cases pass. Formal approval awaits exact-head hosted
qualification.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


Maintainer commit-message qualification repair: current head
bf0037dd29. Corrected only the maintainer
main-sync commit message to Conventional Commit format. The complete
file tree is byte-identical to the previously tested head; exact-head
hosted checks are being refreshed. Nothing merged.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-05 10:43:25 -06:00

194 lines
6.4 KiB
Python

"""Regression: caches keyed by the request's ``model`` string stay bounded (#4479).
``model`` comes straight off a proxied request body, so a client that sends a
new name on every request must not be able to grow a process-wide cache
without limit. #4407 bounded the per-model maps in the cost and savings
trackers; these tests pin the same property for the three remaining caches:
- ``TokenizerRegistry._cache`` (``headroom/tokenizers/registry.py``)
- the beacon's ``_model_cache`` behind ``_public_model``
(``headroom/telemetry/session.py``)
- ``_resolved_model_cache`` behind ``resolve_litellm_model``
(``headroom/pricing/litellm_pricing.py``)
Each is least-recently-used: a name in steady use is never evicted, and an
evicted name is recomputed on its next use rather than answered wrongly.
"""
from __future__ import annotations
import pytest
from headroom.pricing import litellm_pricing as lp
from headroom.telemetry import session
from headroom.tokenizers.estimator import EstimatingTokenCounter
from headroom.tokenizers.registry import TokenizerRegistry
from headroom.utils import remember_bounded
# --------------------------------------------------------------------------
# remember_bounded
# --------------------------------------------------------------------------
def test_remember_bounded_evicts_least_recently_inserted():
cache: dict[str, int] = {}
for i in range(5):
remember_bounded(cache, f"k{i}", i, maxsize=3)
assert list(cache) == ["k2", "k3", "k4"]
def test_remember_bounded_reinsert_moves_key_to_newest():
cache: dict[str, int] = {}
for i in range(3):
remember_bounded(cache, f"k{i}", i, maxsize=3)
remember_bounded(cache, "k0", 0, maxsize=3) # a hit re-records the key
remember_bounded(cache, "k3", 3, maxsize=3)
assert list(cache) == ["k2", "k0", "k3"]
def test_remember_bounded_nonpositive_maxsize_stores_nothing():
cache: dict[str, int] = {"stale": 1}
remember_bounded(cache, "k", 1, maxsize=0)
assert cache == {}
def test_remember_bounded_tolerates_a_dict_resized_by_another_thread():
class ResizedDuringIteration(dict):
def __iter__(self):
raise RuntimeError("dictionary changed size during iteration")
cache = ResizedDuringIteration({"k0": 0, "k1": 1})
remember_bounded(cache, "k2", 2, maxsize=2)
# The eviction is skipped for this call rather than raising on the request
# path; the next insert that can iterate trims the cache again.
assert dict.__getitem__(cache, "k2") == 2
# --------------------------------------------------------------------------
# TokenizerRegistry._cache
# --------------------------------------------------------------------------
@pytest.fixture
def clean_tokenizer_cache():
TokenizerRegistry.clear_cache()
yield
TokenizerRegistry.clear_cache()
def test_tokenizer_registry_cache_is_bounded(clean_tokenizer_cache):
cap = TokenizerRegistry._CACHE_MAXSIZE
for i in range(cap + 50):
TokenizerRegistry.get(f"client-model-{i}-" + "m" * 1_000)
assert len(TokenizerRegistry._cache) == cap
def test_tokenizer_registry_estimation_fallback_is_bounded(clean_tokenizer_cache):
cap = TokenizerRegistry._CACHE_MAXSIZE
for i in range(cap + 10):
tokenizer = TokenizerRegistry.get(f"client-model-{i}", backend="no-such-backend")
assert isinstance(tokenizer, EstimatingTokenCounter)
assert len(TokenizerRegistry._cache) == cap
def test_tokenizer_registry_keeps_a_model_in_steady_use(clean_tokenizer_cache):
cap = TokenizerRegistry._CACHE_MAXSIZE
hot = TokenizerRegistry.get("gpt-4o")
for i in range(cap + 50):
TokenizerRegistry.get(f"client-model-{i}")
assert TokenizerRegistry.get("gpt-4o") is hot
oldest = TokenizerRegistry.get("client-model-0")
assert "client-model-0:auto" in TokenizerRegistry._cache
assert oldest is TokenizerRegistry.get("client-model-0")
# --------------------------------------------------------------------------
# telemetry.session._model_cache
# --------------------------------------------------------------------------
@pytest.fixture
def clean_public_model_cache():
session._model_cache.clear()
yield
session._model_cache.clear()
def test_public_model_cache_is_bounded(clean_public_model_cache):
cap = session._MODEL_CACHE_MAXSIZE
for i in range(cap + 50):
assert session._public_model(f"custom-{i}-" + "m" * 1_000) is None
assert len(session._model_cache) == cap
def test_public_model_evicted_name_is_recomputed(clean_public_model_cache, monkeypatch):
calls: list[str] = []
real_pin = session.pin_litellm_cost_map_local
def counting_pin() -> None:
calls.append("probe")
real_pin()
monkeypatch.setattr(session, "pin_litellm_cost_map_local", counting_pin)
cap = session._MODEL_CACHE_MAXSIZE
session._public_model("seed-model")
for i in range(cap):
session._public_model(f"custom-{i}")
probes_before = len(calls)
assert session._public_model("seed-model") is None
assert len(calls) == probes_before + 1
assert len(session._model_cache) == cap
# --------------------------------------------------------------------------
# litellm_pricing._resolved_model_cache
# --------------------------------------------------------------------------
@pytest.fixture
def counted_resolution(monkeypatch):
lp._resolved_model_cache.clear()
calls: dict[str, int] = {}
def fake_uncached(model: str) -> str:
calls[model] = calls.get(model, 0) + 1
return f"resolved/{model}"
monkeypatch.setattr(lp, "_resolve_litellm_model_uncached", fake_uncached)
monkeypatch.delenv("HEADROOM_MODEL_ALIAS_MAP", raising=False)
yield calls
lp._resolved_model_cache.clear()
def test_resolved_model_cache_is_bounded(counted_resolution):
cap = lp._RESOLVED_MODEL_CACHE_MAXSIZE
for i in range(cap + 50):
assert lp.resolve_litellm_model(f"custom-{i}") == f"resolved/custom-{i}"
assert len(lp._resolved_model_cache) == cap
def test_resolved_model_cache_is_lru(counted_resolution):
cap = lp._RESOLVED_MODEL_CACHE_MAXSIZE
lp.resolve_litellm_model("hot-model")
lp.resolve_litellm_model("cold-model")
for i in range(cap):
lp.resolve_litellm_model(f"custom-{i}")
lp.resolve_litellm_model("hot-model")
assert counted_resolution["hot-model"] == 1
assert lp.resolve_litellm_model("cold-model") == "resolved/cold-model"
assert counted_resolution["cold-model"] == 2