mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): bound the model-keyed tokenizer, beacon and litellm caches (#4497)
## Description Three process-wide caches are keyed by the `model` field of the request body and never evict, so a client that sends a new model name on every request grows each of them without limit (#4479): - `TokenizerRegistry._cache` in `headroom/tokenizers/registry.py`. Each entry is a tokenizer that owns its own `TokenCountCache`. - `_model_cache` behind the beacon's `_public_model` in `headroom/telemetry/session.py`. - `_resolved_model_cache` behind `resolve_litellm_model` in `headroom/pricing/litellm_pricing.py`. #4407 bounded the per-model maps in the cost and savings trackers. This PR bounds these three caches the way `savings_tracker._resolve_litellm_model` was bounded in #2860: a fixed entry count with least-recently-used eviction. The caps are 64 tokenizers and 256 entries for each of the two name caches. A model in steady use is never evicted. An evicted model is recomputed the next time it is used, so the cap never changes a result. Keys are not clipped. A cache key decides which tokenizer, pricing key or beacon model a request gets, so clipping would let two different model names share an entry and get the wrong answer. With the count capped, the size of each entry is already limited by the request body cap. Fixes #4479 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/utils.py`: adds `remember_bounded(cache, key, value, maxsize)`. It stores `key` as the newest entry of a plain dict and evicts from the front, since dict order is recency order. It is safe without a lock: if two threads race, a key may be evicted early or the cache may sit one entry over the cap for a moment, but it never raises. The dicts stay plain dicts so the existing tests that clear, seed or `len()` `_resolved_model_cache` keep working. - `headroom/tokenizers/registry.py`: adds `TokenizerRegistry._CACHE_MAXSIZE = 64`. A cache hit refreshes the entry's recency, and both the created-tokenizer path and the estimation-fallback path insert through `remember_bounded`. `register()` and `clear_cache()` are unchanged. - `headroom/telemetry/session.py`: adds `_MODEL_CACHE_MAXSIZE = 256` and the same hit/insert handling. Cached `None` results ("not a public model") still count as hits. The "ponytail: unbounded dict" comment is replaced. - `headroom/pricing/litellm_pricing.py`: adds `_RESOLVED_MODEL_CACHE_MAXSIZE = 256` and the same treatment in `resolve_litellm_model`. - `tests/test_bounded_model_caches.py`: covers eviction order for the helper and, for each cache, that it stays bounded, that a model in steady use survives (LRU, not FIFO), and that an evicted name is recomputed. It also covers the registry's estimation-fallback path and a dict resized by another thread mid-eviction, so every changed line is exercised. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`): no new errors in the touched modules. The only error reported is in `headroom/memory/adapters/embedders.py:389`, which is the same on `main`. - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_bounded_model_caches.py -q 11 passed in 3.23s # the six cache tests against the original registry.py / session.py / litellm_pricing.py $ git stash push headroom/tokenizers/registry.py headroom/telemetry/session.py headroom/pricing/litellm_pricing.py && pytest tests/test_bounded_model_caches.py -q 7 failed, 4 passed # suites that touch these caches $ pytest tests/test_tokenizers.py tests/test_tokenizer_encoding_resolution.py tests/test_tokenizers \ tests/test_proxy_streaming_resilience.py tests/test_telemetry_session_selfcheck.py \ tests/test_savings_tracker_litellm_resolution_cache.py tests/test_gateway_sidecar_ports.py \ tests/test_beacon_schema_v2.py tests/test_beacon_gzip.py tests/test_bounded_model_caches.py -q 189 passed, 14 skipped $ pytest tests/test_cost_pricing_warning_dedup.py tests/test_counterfactual_pricing.py tests/test_litellm_*.py tests/test_utils*.py tests/test_{pricing,telemetry,tokeniz,session,beacon,cost}*.py -q 660 passed, 18 skipped $ ruff check . && ruff format --check . All checks passed! 1835 files already formatted # full suite (Linux, Python 3.12, --timeout=120) $ pytest tests -q --timeout=120 8 failed, 17053 passed, 596 skipped ``` None of the 8 full-suite failures comes from this change: - `test_mcp_reconcile::test_ordinary_install_does_not_adopt_serena`, `test_wrap_sighup_proxy_leak::test_sighup_on_launch_tool_releases_the_proxy`, both `test_cc_switch_reconciler` tests and `test_proxy_uds::test_prepare_only_chmods_directories_it_creates` fail the same way on an unmodified `main` checkout in this environment (no installed `headroom` console script, sandbox permissions). - `test_direct_chat_ccr_resolution.py` hangs at `TestClient(app)` startup until the timeout, and a different case fails on each run. On unmodified `main` it failed in 2 of 3 isolated runs. - `test_offline_egress_chokepoint.py` and `test_stateless_no_disk.py` pass when run on their own on this branch. ## Real Behavior Proof - Environment: Linux x86_64 (Amazon Linux 2023), Python 3.12.14, this branch on top of `main` at `17d38e56` (0.42.0), with the published 0.42.0 `headroom._core` wheel binary. No Rust toolchain was available here, so `_core` was not rebuilt; this PR does not touch Rust. - Exact command / steps: the reproduction from #4479, run against the real modules with no stubs. After one warm-up call per cache, call `get_tokenizer(f"gpt-4o-x{i}-" + "m" * 10_000).count_text(...)`, `session._public_model(f"custom-{i}-" + "m" * 10_000)` and `litellm_pricing.resolve_litellm_model(f"custom-{i}-" + "m" * 10_000)` for 2,000 distinct names each. Measure `len(cache)` and the `tracemalloc` growth across each loop. Then check that `gpt-4o` still resolves through both name caches. - Observed result: on `main`, all three caches grow to 2,001 entries (40.7 MB, 20.2 MB and 20.2 MB traced). On this branch they stop at 64, 256 and 256 entries (1.3 MB, 2.6 MB and 2.6 MB), and `gpt-4o` still resolves through both name caches: ```text # main (17d38e56) registry._cache entries: 2001 | traced growth: 40.7 MB telemetry.session._model_cache entries: 2001 | traced growth: 20.2 MB litellm_pricing._resolved_model_cache entries: 2001 | traced growth: 20.2 MB public model still resolves: gpt-4o | gpt-4o # this branch registry._cache entries: 64 | traced growth: 1.3 MB telemetry.session._model_cache entries: 256 | traced growth: 2.6 MB litellm_pricing._resolved_model_cache entries: 256 | traced growth: 2.6 MB public model still resolves: gpt-4o | gpt-4o ``` - Not tested: a long-running proxy process taking real HTTP traffic with rotating model names (the run above calls the same functions the proxy calls, in-process); Windows and macOS; `mypy headroom` for the whole package (only the four touched modules were checked). ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: Stable. - Stable/default behavior changed: No change for normal traffic (far fewer distinct models than the caps). Past the caps, the least recently used entry is dropped and recomputed if that model comes back. - Kill switch / disable path: Not applicable. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable. ## Additional Notes - No user documentation needed: these caches are internal. - Cap sizes: 256 matches `savings_tracker._MODEL_RESOLUTION_CACHE_MAXSIZE`. The tokenizer registry uses 64 because each entry owns a `TokenCountCache` of up to 8,000,000 characters. - Open PR #4437 (#4434) edits `TokenizerRegistry.register()`. This PR only touches `get()` and the class attributes, so the two should not conflict. --- 🤖 Opened with [Superhuman](https://github.com/gaurav0107/superhuman), an open-source contribution agent. ### Maintainer current-main requalification Exact head `a29cc2d40f5c8111cd079949201d8660c8a50130`, upstream main `e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`. Updated against upstream/maine4a60ed7, resolving the tokenizer-registry overlap by retaining main's registered-model factories and dispatch, including its rule not to cache estimation after a failed model-specific factory. Applied the contributor's bounded recency insertion at the successful cache writes and ordinary fallback writes. No contributor policy or correctness logic was replaced. Re-reviewed cache helper/call sites and existing discussion; cached None, full model keys and recomputation after eviction remain intact. The combined tree passes 111 bounded-cache/tokenizer/encoding tests with 14 optional skips, plus the adjacent tokenizer load-timeout, telemetry selfcheck and pricing-resolution suites. Ruff lint/format and whitespace checks pass. Existing rotating-2,000-name runtime evidence remains scoped to in-process module calls; an existing development native extension was used locally, not rebuilt. The shared test-only Gemini fixture repair was applied for independently established main failures; all ten cases pass. Formal approval awaits exact-head hosted qualification. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. Maintainer commit-message qualification repair: current headbf0037dd29. Corrected only the maintainer main-sync commit message to Conventional Commit format. The complete file tree is byte-identical to the previously tested head; exact-head hosted checks are being refreshed. Nothing merged. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
JerrettDavis
parent
3afaa97ffc
commit
e51b8065a4
@@ -27,6 +27,7 @@ from headroom.pricing.litellm_model_resolution import (
|
||||
pricing_lookup_candidates,
|
||||
resolve_litellm_model_name,
|
||||
)
|
||||
from headroom.utils import remember_bounded
|
||||
|
||||
# litellm calls `dotenv.load_dotenv()` during its own import, which loads
|
||||
# the project `.env` into `os.environ`. We don't want that side effect —
|
||||
@@ -59,7 +60,10 @@ except ImportError:
|
||||
litellm = None # type: ignore[assignment]
|
||||
LITELLM_AVAILABLE = False
|
||||
|
||||
# Least-recently-used first. Keyed by the client-supplied model name, so the
|
||||
# entry count is capped (#4479); an evicted name is simply resolved again.
|
||||
_resolved_model_cache: dict[str, str] = {}
|
||||
_RESOLVED_MODEL_CACHE_MAXSIZE = 256
|
||||
|
||||
logger = logging.getLogger("headroom.pricing")
|
||||
|
||||
@@ -113,14 +117,16 @@ def resolve_litellm_model(model: str) -> str:
|
||||
the bare-prefix rules. Shared by the live (cost.py) and persisted
|
||||
(savings_tracker) pricing paths so both figures price identically.
|
||||
"""
|
||||
if model in _resolved_model_cache:
|
||||
return _resolved_model_cache[model]
|
||||
cached = _resolved_model_cache.get(model)
|
||||
if cached is not None:
|
||||
remember_bounded(_resolved_model_cache, model, cached, _RESOLVED_MODEL_CACHE_MAXSIZE)
|
||||
return cached
|
||||
priced: str | None = None
|
||||
alias = _static_alias_map()
|
||||
if alias:
|
||||
priced = _reduce_to_priced_key(alias.get(model, model))
|
||||
resolved = priced if priced is not None else _resolve_litellm_model_uncached(model)
|
||||
_resolved_model_cache[model] = resolved
|
||||
remember_bounded(_resolved_model_cache, model, resolved, _RESOLVED_MODEL_CACHE_MAXSIZE)
|
||||
return resolved
|
||||
|
||||
|
||||
|
||||
@@ -120,6 +120,7 @@ from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from headroom.offline import pin_litellm_cost_map_local
|
||||
from headroom.utils import remember_bounded
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -511,7 +512,11 @@ def _config_snapshot() -> dict[str, str]:
|
||||
return out
|
||||
|
||||
|
||||
# Least-recently-used first. ``model`` comes from the request body, so the
|
||||
# entry count is capped (#4479); a model that falls out is simply looked up
|
||||
# again the next time it is seen.
|
||||
_model_cache: dict[str, str | None] = {}
|
||||
_MODEL_CACHE_MAXSIZE = 256
|
||||
|
||||
|
||||
def _public_model(model: str) -> str | None:
|
||||
@@ -530,8 +535,13 @@ def _public_model(model: str) -> str | None:
|
||||
"""
|
||||
if not model:
|
||||
return None
|
||||
if model in _model_cache:
|
||||
return _model_cache[model]
|
||||
try:
|
||||
cached = _model_cache[model]
|
||||
except KeyError:
|
||||
pass
|
||||
else:
|
||||
remember_bounded(_model_cache, model, cached, _MODEL_CACHE_MAXSIZE)
|
||||
return cached
|
||||
|
||||
resolved: str | None = None
|
||||
try:
|
||||
@@ -546,9 +556,7 @@ def _public_model(model: str) -> str | None:
|
||||
except Exception:
|
||||
resolved = None
|
||||
|
||||
# ponytail: unbounded dict, but it is keyed by distinct model ids seen in
|
||||
# one process — a handful. Cap it if a router ever fans out over hundreds.
|
||||
_model_cache[model] = resolved
|
||||
remember_bounded(_model_cache, model, resolved, _MODEL_CACHE_MAXSIZE)
|
||||
return resolved
|
||||
|
||||
|
||||
|
||||
@@ -11,6 +11,7 @@ import re
|
||||
from collections.abc import Callable
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from ..utils import remember_bounded
|
||||
from .base import TokenCounter
|
||||
from .estimator import EstimatingTokenCounter
|
||||
|
||||
@@ -144,8 +145,13 @@ class TokenizerRegistry:
|
||||
# Registered model factories (model -> factory function)
|
||||
_model_factories: dict[str, Callable[[str], TokenCounter]] = {}
|
||||
|
||||
# Cache for auto-detected tokenizers
|
||||
# Cache for auto-detected tokenizers, least-recently-used first. ``model``
|
||||
# is the ``model`` field of a proxied request body, so the entry count is
|
||||
# capped (#4479): each entry owns its own TokenCountCache, and a client
|
||||
# sending a new name per request must not grow this without limit. Real
|
||||
# traffic uses a handful of models, far below the cap.
|
||||
_cache: dict[str, TokenCounter] = {}
|
||||
_CACHE_MAXSIZE = 64
|
||||
|
||||
def __new__(cls) -> TokenizerRegistry:
|
||||
"""Singleton pattern."""
|
||||
@@ -198,8 +204,10 @@ class TokenizerRegistry:
|
||||
|
||||
# Check cache
|
||||
cache_key = f"{model_lower}:{backend or 'auto'}"
|
||||
if cache_key in registry._cache:
|
||||
return registry._cache[cache_key]
|
||||
cached = registry._cache.get(cache_key)
|
||||
if cached is not None:
|
||||
remember_bounded(registry._cache, cache_key, cached, cls._CACHE_MAXSIZE)
|
||||
return cached
|
||||
|
||||
uses_model_factory = backend is None and model_lower in registry._model_factories
|
||||
|
||||
@@ -209,7 +217,7 @@ class TokenizerRegistry:
|
||||
tokenizer = registry._model_factories[model_lower](model)
|
||||
else:
|
||||
tokenizer = registry._create_tokenizer(model, backend)
|
||||
registry._cache[cache_key] = tokenizer
|
||||
remember_bounded(registry._cache, cache_key, tokenizer, cls._CACHE_MAXSIZE)
|
||||
return tokenizer
|
||||
except Exception as e:
|
||||
if fallback:
|
||||
@@ -218,7 +226,7 @@ class TokenizerRegistry:
|
||||
)
|
||||
tokenizer = EstimatingTokenCounter()
|
||||
if not uses_model_factory:
|
||||
registry._cache[cache_key] = tokenizer
|
||||
remember_bounded(registry._cache, cache_key, tokenizer, cls._CACHE_MAXSIZE)
|
||||
return tokenizer
|
||||
raise ValueError(f"No tokenizer available for {model}: {e}") from e
|
||||
|
||||
|
||||
+39
-1
@@ -9,7 +9,10 @@ import re
|
||||
import uuid
|
||||
import warnings
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
from typing import Any, TypeVar
|
||||
|
||||
_K = TypeVar("_K")
|
||||
_V = TypeVar("_V")
|
||||
|
||||
# Marker format for Headroom modifications
|
||||
MARKER_PREFIX = "<headroom:"
|
||||
@@ -42,6 +45,41 @@ def compute_short_hash(data: str | bytes, length: int = 16) -> str:
|
||||
return compute_hash(data)[:length]
|
||||
|
||||
|
||||
def remember_bounded(cache: dict[_K, _V], key: _K, value: _V, maxsize: int) -> None:
|
||||
"""Store ``key`` as the newest entry of ``cache``, keeping at most ``maxsize``.
|
||||
|
||||
``cache`` is a plain dict used as a least-recently-used cache: insertion
|
||||
order is recency order, so the first key is the stalest and is evicted
|
||||
first. Call this again on a cache hit to mark the key recently used.
|
||||
|
||||
Meant for caches keyed by client-controlled strings, such as a proxied
|
||||
request's ``model`` field: the entry count stays bounded however many
|
||||
distinct keys arrive, and an evicted key is recomputed on its next use
|
||||
rather than answered wrongly.
|
||||
|
||||
Safe to call from several threads without a lock: a race can evict a key
|
||||
early or leave the cache briefly one entry over ``maxsize``, never raise.
|
||||
|
||||
Args:
|
||||
cache: The dict to update in place.
|
||||
key: Cache key.
|
||||
value: Value to store.
|
||||
maxsize: Maximum number of entries. ``<= 0`` stores nothing.
|
||||
"""
|
||||
if maxsize <= 0:
|
||||
cache.clear()
|
||||
return
|
||||
cache.pop(key, None)
|
||||
while len(cache) >= maxsize:
|
||||
try:
|
||||
oldest = next(iter(cache))
|
||||
except (StopIteration, RuntimeError):
|
||||
# Emptied, or resized by another thread mid-iteration.
|
||||
break
|
||||
cache.pop(oldest, None)
|
||||
cache[key] = value
|
||||
|
||||
|
||||
def fast_hash(data: str | bytes, length: int = 16) -> str:
|
||||
"""Fast non-cryptographic content hash for caches and dedup.
|
||||
|
||||
|
||||
@@ -0,0 +1,193 @@
|
||||
"""Regression: caches keyed by the request's ``model`` string stay bounded (#4479).
|
||||
|
||||
``model`` comes straight off a proxied request body, so a client that sends a
|
||||
new name on every request must not be able to grow a process-wide cache
|
||||
without limit. #4407 bounded the per-model maps in the cost and savings
|
||||
trackers; these tests pin the same property for the three remaining caches:
|
||||
|
||||
- ``TokenizerRegistry._cache`` (``headroom/tokenizers/registry.py``)
|
||||
- the beacon's ``_model_cache`` behind ``_public_model``
|
||||
(``headroom/telemetry/session.py``)
|
||||
- ``_resolved_model_cache`` behind ``resolve_litellm_model``
|
||||
(``headroom/pricing/litellm_pricing.py``)
|
||||
|
||||
Each is least-recently-used: a name in steady use is never evicted, and an
|
||||
evicted name is recomputed on its next use rather than answered wrongly.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from headroom.pricing import litellm_pricing as lp
|
||||
from headroom.telemetry import session
|
||||
from headroom.tokenizers.estimator import EstimatingTokenCounter
|
||||
from headroom.tokenizers.registry import TokenizerRegistry
|
||||
from headroom.utils import remember_bounded
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# remember_bounded
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_remember_bounded_evicts_least_recently_inserted():
|
||||
cache: dict[str, int] = {}
|
||||
for i in range(5):
|
||||
remember_bounded(cache, f"k{i}", i, maxsize=3)
|
||||
|
||||
assert list(cache) == ["k2", "k3", "k4"]
|
||||
|
||||
|
||||
def test_remember_bounded_reinsert_moves_key_to_newest():
|
||||
cache: dict[str, int] = {}
|
||||
for i in range(3):
|
||||
remember_bounded(cache, f"k{i}", i, maxsize=3)
|
||||
|
||||
remember_bounded(cache, "k0", 0, maxsize=3) # a hit re-records the key
|
||||
remember_bounded(cache, "k3", 3, maxsize=3)
|
||||
|
||||
assert list(cache) == ["k2", "k0", "k3"]
|
||||
|
||||
|
||||
def test_remember_bounded_nonpositive_maxsize_stores_nothing():
|
||||
cache: dict[str, int] = {"stale": 1}
|
||||
remember_bounded(cache, "k", 1, maxsize=0)
|
||||
|
||||
assert cache == {}
|
||||
|
||||
|
||||
def test_remember_bounded_tolerates_a_dict_resized_by_another_thread():
|
||||
class ResizedDuringIteration(dict):
|
||||
def __iter__(self):
|
||||
raise RuntimeError("dictionary changed size during iteration")
|
||||
|
||||
cache = ResizedDuringIteration({"k0": 0, "k1": 1})
|
||||
remember_bounded(cache, "k2", 2, maxsize=2)
|
||||
|
||||
# The eviction is skipped for this call rather than raising on the request
|
||||
# path; the next insert that can iterate trims the cache again.
|
||||
assert dict.__getitem__(cache, "k2") == 2
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# TokenizerRegistry._cache
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def clean_tokenizer_cache():
|
||||
TokenizerRegistry.clear_cache()
|
||||
yield
|
||||
TokenizerRegistry.clear_cache()
|
||||
|
||||
|
||||
def test_tokenizer_registry_cache_is_bounded(clean_tokenizer_cache):
|
||||
cap = TokenizerRegistry._CACHE_MAXSIZE
|
||||
for i in range(cap + 50):
|
||||
TokenizerRegistry.get(f"client-model-{i}-" + "m" * 1_000)
|
||||
|
||||
assert len(TokenizerRegistry._cache) == cap
|
||||
|
||||
|
||||
def test_tokenizer_registry_estimation_fallback_is_bounded(clean_tokenizer_cache):
|
||||
cap = TokenizerRegistry._CACHE_MAXSIZE
|
||||
for i in range(cap + 10):
|
||||
tokenizer = TokenizerRegistry.get(f"client-model-{i}", backend="no-such-backend")
|
||||
assert isinstance(tokenizer, EstimatingTokenCounter)
|
||||
|
||||
assert len(TokenizerRegistry._cache) == cap
|
||||
|
||||
|
||||
def test_tokenizer_registry_keeps_a_model_in_steady_use(clean_tokenizer_cache):
|
||||
cap = TokenizerRegistry._CACHE_MAXSIZE
|
||||
hot = TokenizerRegistry.get("gpt-4o")
|
||||
for i in range(cap + 50):
|
||||
TokenizerRegistry.get(f"client-model-{i}")
|
||||
assert TokenizerRegistry.get("gpt-4o") is hot
|
||||
|
||||
oldest = TokenizerRegistry.get("client-model-0")
|
||||
assert "client-model-0:auto" in TokenizerRegistry._cache
|
||||
assert oldest is TokenizerRegistry.get("client-model-0")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# telemetry.session._model_cache
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def clean_public_model_cache():
|
||||
session._model_cache.clear()
|
||||
yield
|
||||
session._model_cache.clear()
|
||||
|
||||
|
||||
def test_public_model_cache_is_bounded(clean_public_model_cache):
|
||||
cap = session._MODEL_CACHE_MAXSIZE
|
||||
for i in range(cap + 50):
|
||||
assert session._public_model(f"custom-{i}-" + "m" * 1_000) is None
|
||||
|
||||
assert len(session._model_cache) == cap
|
||||
|
||||
|
||||
def test_public_model_evicted_name_is_recomputed(clean_public_model_cache, monkeypatch):
|
||||
calls: list[str] = []
|
||||
real_pin = session.pin_litellm_cost_map_local
|
||||
|
||||
def counting_pin() -> None:
|
||||
calls.append("probe")
|
||||
real_pin()
|
||||
|
||||
monkeypatch.setattr(session, "pin_litellm_cost_map_local", counting_pin)
|
||||
|
||||
cap = session._MODEL_CACHE_MAXSIZE
|
||||
session._public_model("seed-model")
|
||||
for i in range(cap):
|
||||
session._public_model(f"custom-{i}")
|
||||
probes_before = len(calls)
|
||||
|
||||
assert session._public_model("seed-model") is None
|
||||
assert len(calls) == probes_before + 1
|
||||
assert len(session._model_cache) == cap
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# litellm_pricing._resolved_model_cache
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def counted_resolution(monkeypatch):
|
||||
lp._resolved_model_cache.clear()
|
||||
calls: dict[str, int] = {}
|
||||
|
||||
def fake_uncached(model: str) -> str:
|
||||
calls[model] = calls.get(model, 0) + 1
|
||||
return f"resolved/{model}"
|
||||
|
||||
monkeypatch.setattr(lp, "_resolve_litellm_model_uncached", fake_uncached)
|
||||
monkeypatch.delenv("HEADROOM_MODEL_ALIAS_MAP", raising=False)
|
||||
yield calls
|
||||
lp._resolved_model_cache.clear()
|
||||
|
||||
|
||||
def test_resolved_model_cache_is_bounded(counted_resolution):
|
||||
cap = lp._RESOLVED_MODEL_CACHE_MAXSIZE
|
||||
for i in range(cap + 50):
|
||||
assert lp.resolve_litellm_model(f"custom-{i}") == f"resolved/custom-{i}"
|
||||
|
||||
assert len(lp._resolved_model_cache) == cap
|
||||
|
||||
|
||||
def test_resolved_model_cache_is_lru(counted_resolution):
|
||||
cap = lp._RESOLVED_MODEL_CACHE_MAXSIZE
|
||||
lp.resolve_litellm_model("hot-model")
|
||||
lp.resolve_litellm_model("cold-model")
|
||||
for i in range(cap):
|
||||
lp.resolve_litellm_model(f"custom-{i}")
|
||||
lp.resolve_litellm_model("hot-model")
|
||||
|
||||
assert counted_resolution["hot-model"] == 1
|
||||
|
||||
assert lp.resolve_litellm_model("cold-model") == "resolved/cold-model"
|
||||
assert counted_resolution["cold-model"] == 2
|
||||
Reference in New Issue
Block a user