mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description Three process-wide caches are keyed by the `model` field of the request body and never evict, so a client that sends a new model name on every request grows each of them without limit (#4479): - `TokenizerRegistry._cache` in `headroom/tokenizers/registry.py`. Each entry is a tokenizer that owns its own `TokenCountCache`. - `_model_cache` behind the beacon's `_public_model` in `headroom/telemetry/session.py`. - `_resolved_model_cache` behind `resolve_litellm_model` in `headroom/pricing/litellm_pricing.py`. #4407 bounded the per-model maps in the cost and savings trackers. This PR bounds these three caches the way `savings_tracker._resolve_litellm_model` was bounded in #2860: a fixed entry count with least-recently-used eviction. The caps are 64 tokenizers and 256 entries for each of the two name caches. A model in steady use is never evicted. An evicted model is recomputed the next time it is used, so the cap never changes a result. Keys are not clipped. A cache key decides which tokenizer, pricing key or beacon model a request gets, so clipping would let two different model names share an entry and get the wrong answer. With the count capped, the size of each entry is already limited by the request body cap. Fixes #4479 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/utils.py`: adds `remember_bounded(cache, key, value, maxsize)`. It stores `key` as the newest entry of a plain dict and evicts from the front, since dict order is recency order. It is safe without a lock: if two threads race, a key may be evicted early or the cache may sit one entry over the cap for a moment, but it never raises. The dicts stay plain dicts so the existing tests that clear, seed or `len()` `_resolved_model_cache` keep working. - `headroom/tokenizers/registry.py`: adds `TokenizerRegistry._CACHE_MAXSIZE = 64`. A cache hit refreshes the entry's recency, and both the created-tokenizer path and the estimation-fallback path insert through `remember_bounded`. `register()` and `clear_cache()` are unchanged. - `headroom/telemetry/session.py`: adds `_MODEL_CACHE_MAXSIZE = 256` and the same hit/insert handling. Cached `None` results ("not a public model") still count as hits. The "ponytail: unbounded dict" comment is replaced. - `headroom/pricing/litellm_pricing.py`: adds `_RESOLVED_MODEL_CACHE_MAXSIZE = 256` and the same treatment in `resolve_litellm_model`. - `tests/test_bounded_model_caches.py`: covers eviction order for the helper and, for each cache, that it stays bounded, that a model in steady use survives (LRU, not FIFO), and that an evicted name is recomputed. It also covers the registry's estimation-fallback path and a dict resized by another thread mid-eviction, so every changed line is exercised. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`): no new errors in the touched modules. The only error reported is in `headroom/memory/adapters/embedders.py:389`, which is the same on `main`. - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_bounded_model_caches.py -q 11 passed in 3.23s # the six cache tests against the original registry.py / session.py / litellm_pricing.py $ git stash push headroom/tokenizers/registry.py headroom/telemetry/session.py headroom/pricing/litellm_pricing.py && pytest tests/test_bounded_model_caches.py -q 7 failed, 4 passed # suites that touch these caches $ pytest tests/test_tokenizers.py tests/test_tokenizer_encoding_resolution.py tests/test_tokenizers \ tests/test_proxy_streaming_resilience.py tests/test_telemetry_session_selfcheck.py \ tests/test_savings_tracker_litellm_resolution_cache.py tests/test_gateway_sidecar_ports.py \ tests/test_beacon_schema_v2.py tests/test_beacon_gzip.py tests/test_bounded_model_caches.py -q 189 passed, 14 skipped $ pytest tests/test_cost_pricing_warning_dedup.py tests/test_counterfactual_pricing.py tests/test_litellm_*.py tests/test_utils*.py tests/test_{pricing,telemetry,tokeniz,session,beacon,cost}*.py -q 660 passed, 18 skipped $ ruff check . && ruff format --check . All checks passed! 1835 files already formatted # full suite (Linux, Python 3.12, --timeout=120) $ pytest tests -q --timeout=120 8 failed, 17053 passed, 596 skipped ``` None of the 8 full-suite failures comes from this change: - `test_mcp_reconcile::test_ordinary_install_does_not_adopt_serena`, `test_wrap_sighup_proxy_leak::test_sighup_on_launch_tool_releases_the_proxy`, both `test_cc_switch_reconciler` tests and `test_proxy_uds::test_prepare_only_chmods_directories_it_creates` fail the same way on an unmodified `main` checkout in this environment (no installed `headroom` console script, sandbox permissions). - `test_direct_chat_ccr_resolution.py` hangs at `TestClient(app)` startup until the timeout, and a different case fails on each run. On unmodified `main` it failed in 2 of 3 isolated runs. - `test_offline_egress_chokepoint.py` and `test_stateless_no_disk.py` pass when run on their own on this branch. ## Real Behavior Proof - Environment: Linux x86_64 (Amazon Linux 2023), Python 3.12.14, this branch on top of `main` at `17d38e56` (0.42.0), with the published 0.42.0 `headroom._core` wheel binary. No Rust toolchain was available here, so `_core` was not rebuilt; this PR does not touch Rust. - Exact command / steps: the reproduction from #4479, run against the real modules with no stubs. After one warm-up call per cache, call `get_tokenizer(f"gpt-4o-x{i}-" + "m" * 10_000).count_text(...)`, `session._public_model(f"custom-{i}-" + "m" * 10_000)` and `litellm_pricing.resolve_litellm_model(f"custom-{i}-" + "m" * 10_000)` for 2,000 distinct names each. Measure `len(cache)` and the `tracemalloc` growth across each loop. Then check that `gpt-4o` still resolves through both name caches. - Observed result: on `main`, all three caches grow to 2,001 entries (40.7 MB, 20.2 MB and 20.2 MB traced). On this branch they stop at 64, 256 and 256 entries (1.3 MB, 2.6 MB and 2.6 MB), and `gpt-4o` still resolves through both name caches: ```text # main (17d38e56) registry._cache entries: 2001 | traced growth: 40.7 MB telemetry.session._model_cache entries: 2001 | traced growth: 20.2 MB litellm_pricing._resolved_model_cache entries: 2001 | traced growth: 20.2 MB public model still resolves: gpt-4o | gpt-4o # this branch registry._cache entries: 64 | traced growth: 1.3 MB telemetry.session._model_cache entries: 256 | traced growth: 2.6 MB litellm_pricing._resolved_model_cache entries: 256 | traced growth: 2.6 MB public model still resolves: gpt-4o | gpt-4o ``` - Not tested: a long-running proxy process taking real HTTP traffic with rotating model names (the run above calls the same functions the proxy calls, in-process); Windows and macOS; `mypy headroom` for the whole package (only the four touched modules were checked). ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: Stable. - Stable/default behavior changed: No change for normal traffic (far fewer distinct models than the caps). Past the caps, the least recently used entry is dropped and recomputed if that model comes back. - Kill switch / disable path: Not applicable. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable. ## Additional Notes - No user documentation needed: these caches are internal. - Cap sizes: 256 matches `savings_tracker._MODEL_RESOLUTION_CACHE_MAXSIZE`. The tokenizer registry uses 64 because each entry owns a `TokenCountCache` of up to 8,000,000 characters. - Open PR #4437 (#4434) edits `TokenizerRegistry.register()`. This PR only touches `get()` and the class attributes, so the two should not conflict. --- 🤖 Opened with [Superhuman](https://github.com/gaurav0107/superhuman), an open-source contribution agent. ### Maintainer current-main requalification Exact head `a29cc2d40f5c8111cd079949201d8660c8a50130`, upstream main `e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`. Updated against upstream/maine4a60ed7, resolving the tokenizer-registry overlap by retaining main's registered-model factories and dispatch, including its rule not to cache estimation after a failed model-specific factory. Applied the contributor's bounded recency insertion at the successful cache writes and ordinary fallback writes. No contributor policy or correctness logic was replaced. Re-reviewed cache helper/call sites and existing discussion; cached None, full model keys and recomputation after eviction remain intact. The combined tree passes 111 bounded-cache/tokenizer/encoding tests with 14 optional skips, plus the adjacent tokenizer load-timeout, telemetry selfcheck and pricing-resolution suites. Ruff lint/format and whitespace checks pass. Existing rotating-2,000-name runtime evidence remains scoped to in-process module calls; an existing development native extension was used locally, not rebuilt. The shared test-only Gemini fixture repair was applied for independently established main failures; all ten cases pass. Formal approval awaits exact-head hosted qualification. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. Maintainer commit-message qualification repair: current headbf0037dd29. Corrected only the maintainer main-sync commit message to Conventional Commit format. The complete file tree is byte-identical to the previously tested head; exact-head hosted checks are being refreshed. Nothing merged. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
194 lines
6.4 KiB
Python
194 lines
6.4 KiB
Python
"""Regression: caches keyed by the request's ``model`` string stay bounded (#4479).
|
|
|
|
``model`` comes straight off a proxied request body, so a client that sends a
|
|
new name on every request must not be able to grow a process-wide cache
|
|
without limit. #4407 bounded the per-model maps in the cost and savings
|
|
trackers; these tests pin the same property for the three remaining caches:
|
|
|
|
- ``TokenizerRegistry._cache`` (``headroom/tokenizers/registry.py``)
|
|
- the beacon's ``_model_cache`` behind ``_public_model``
|
|
(``headroom/telemetry/session.py``)
|
|
- ``_resolved_model_cache`` behind ``resolve_litellm_model``
|
|
(``headroom/pricing/litellm_pricing.py``)
|
|
|
|
Each is least-recently-used: a name in steady use is never evicted, and an
|
|
evicted name is recomputed on its next use rather than answered wrongly.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from headroom.pricing import litellm_pricing as lp
|
|
from headroom.telemetry import session
|
|
from headroom.tokenizers.estimator import EstimatingTokenCounter
|
|
from headroom.tokenizers.registry import TokenizerRegistry
|
|
from headroom.utils import remember_bounded
|
|
|
|
# --------------------------------------------------------------------------
|
|
# remember_bounded
|
|
# --------------------------------------------------------------------------
|
|
|
|
|
|
def test_remember_bounded_evicts_least_recently_inserted():
|
|
cache: dict[str, int] = {}
|
|
for i in range(5):
|
|
remember_bounded(cache, f"k{i}", i, maxsize=3)
|
|
|
|
assert list(cache) == ["k2", "k3", "k4"]
|
|
|
|
|
|
def test_remember_bounded_reinsert_moves_key_to_newest():
|
|
cache: dict[str, int] = {}
|
|
for i in range(3):
|
|
remember_bounded(cache, f"k{i}", i, maxsize=3)
|
|
|
|
remember_bounded(cache, "k0", 0, maxsize=3) # a hit re-records the key
|
|
remember_bounded(cache, "k3", 3, maxsize=3)
|
|
|
|
assert list(cache) == ["k2", "k0", "k3"]
|
|
|
|
|
|
def test_remember_bounded_nonpositive_maxsize_stores_nothing():
|
|
cache: dict[str, int] = {"stale": 1}
|
|
remember_bounded(cache, "k", 1, maxsize=0)
|
|
|
|
assert cache == {}
|
|
|
|
|
|
def test_remember_bounded_tolerates_a_dict_resized_by_another_thread():
|
|
class ResizedDuringIteration(dict):
|
|
def __iter__(self):
|
|
raise RuntimeError("dictionary changed size during iteration")
|
|
|
|
cache = ResizedDuringIteration({"k0": 0, "k1": 1})
|
|
remember_bounded(cache, "k2", 2, maxsize=2)
|
|
|
|
# The eviction is skipped for this call rather than raising on the request
|
|
# path; the next insert that can iterate trims the cache again.
|
|
assert dict.__getitem__(cache, "k2") == 2
|
|
|
|
|
|
# --------------------------------------------------------------------------
|
|
# TokenizerRegistry._cache
|
|
# --------------------------------------------------------------------------
|
|
|
|
|
|
@pytest.fixture
|
|
def clean_tokenizer_cache():
|
|
TokenizerRegistry.clear_cache()
|
|
yield
|
|
TokenizerRegistry.clear_cache()
|
|
|
|
|
|
def test_tokenizer_registry_cache_is_bounded(clean_tokenizer_cache):
|
|
cap = TokenizerRegistry._CACHE_MAXSIZE
|
|
for i in range(cap + 50):
|
|
TokenizerRegistry.get(f"client-model-{i}-" + "m" * 1_000)
|
|
|
|
assert len(TokenizerRegistry._cache) == cap
|
|
|
|
|
|
def test_tokenizer_registry_estimation_fallback_is_bounded(clean_tokenizer_cache):
|
|
cap = TokenizerRegistry._CACHE_MAXSIZE
|
|
for i in range(cap + 10):
|
|
tokenizer = TokenizerRegistry.get(f"client-model-{i}", backend="no-such-backend")
|
|
assert isinstance(tokenizer, EstimatingTokenCounter)
|
|
|
|
assert len(TokenizerRegistry._cache) == cap
|
|
|
|
|
|
def test_tokenizer_registry_keeps_a_model_in_steady_use(clean_tokenizer_cache):
|
|
cap = TokenizerRegistry._CACHE_MAXSIZE
|
|
hot = TokenizerRegistry.get("gpt-4o")
|
|
for i in range(cap + 50):
|
|
TokenizerRegistry.get(f"client-model-{i}")
|
|
assert TokenizerRegistry.get("gpt-4o") is hot
|
|
|
|
oldest = TokenizerRegistry.get("client-model-0")
|
|
assert "client-model-0:auto" in TokenizerRegistry._cache
|
|
assert oldest is TokenizerRegistry.get("client-model-0")
|
|
|
|
|
|
# --------------------------------------------------------------------------
|
|
# telemetry.session._model_cache
|
|
# --------------------------------------------------------------------------
|
|
|
|
|
|
@pytest.fixture
|
|
def clean_public_model_cache():
|
|
session._model_cache.clear()
|
|
yield
|
|
session._model_cache.clear()
|
|
|
|
|
|
def test_public_model_cache_is_bounded(clean_public_model_cache):
|
|
cap = session._MODEL_CACHE_MAXSIZE
|
|
for i in range(cap + 50):
|
|
assert session._public_model(f"custom-{i}-" + "m" * 1_000) is None
|
|
|
|
assert len(session._model_cache) == cap
|
|
|
|
|
|
def test_public_model_evicted_name_is_recomputed(clean_public_model_cache, monkeypatch):
|
|
calls: list[str] = []
|
|
real_pin = session.pin_litellm_cost_map_local
|
|
|
|
def counting_pin() -> None:
|
|
calls.append("probe")
|
|
real_pin()
|
|
|
|
monkeypatch.setattr(session, "pin_litellm_cost_map_local", counting_pin)
|
|
|
|
cap = session._MODEL_CACHE_MAXSIZE
|
|
session._public_model("seed-model")
|
|
for i in range(cap):
|
|
session._public_model(f"custom-{i}")
|
|
probes_before = len(calls)
|
|
|
|
assert session._public_model("seed-model") is None
|
|
assert len(calls) == probes_before + 1
|
|
assert len(session._model_cache) == cap
|
|
|
|
|
|
# --------------------------------------------------------------------------
|
|
# litellm_pricing._resolved_model_cache
|
|
# --------------------------------------------------------------------------
|
|
|
|
|
|
@pytest.fixture
|
|
def counted_resolution(monkeypatch):
|
|
lp._resolved_model_cache.clear()
|
|
calls: dict[str, int] = {}
|
|
|
|
def fake_uncached(model: str) -> str:
|
|
calls[model] = calls.get(model, 0) + 1
|
|
return f"resolved/{model}"
|
|
|
|
monkeypatch.setattr(lp, "_resolve_litellm_model_uncached", fake_uncached)
|
|
monkeypatch.delenv("HEADROOM_MODEL_ALIAS_MAP", raising=False)
|
|
yield calls
|
|
lp._resolved_model_cache.clear()
|
|
|
|
|
|
def test_resolved_model_cache_is_bounded(counted_resolution):
|
|
cap = lp._RESOLVED_MODEL_CACHE_MAXSIZE
|
|
for i in range(cap + 50):
|
|
assert lp.resolve_litellm_model(f"custom-{i}") == f"resolved/custom-{i}"
|
|
|
|
assert len(lp._resolved_model_cache) == cap
|
|
|
|
|
|
def test_resolved_model_cache_is_lru(counted_resolution):
|
|
cap = lp._RESOLVED_MODEL_CACHE_MAXSIZE
|
|
lp.resolve_litellm_model("hot-model")
|
|
lp.resolve_litellm_model("cold-model")
|
|
for i in range(cap):
|
|
lp.resolve_litellm_model(f"custom-{i}")
|
|
lp.resolve_litellm_model("hot-model")
|
|
|
|
assert counted_resolution["hot-model"] == 1
|
|
|
|
assert lp.resolve_litellm_model("cold-model") == "resolved/cold-model"
|
|
assert counted_resolution["cold-model"] == 2
|