mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
perf(cache): LRU-bound the compression-feedback tool-pattern map (#3342)
## Description `CompressionFeedback._tool_patterns` is a `dict[tool_name, LocalToolPattern]`, and `tool_name` is client-controlled — MCP servers can expose arbitrarily named tools. It was only ever grown (`record_compression` / `record_retrieval`) or cleared wholesale (`clear()`), with no per-key eviction. `get_compression_feedback()` returns a process-global singleton, so the outer key set accumulated one entry per distinct tool name for the whole process lifetime. The interior per-pattern structures are all already capped (`common_queries` → 100, `queried_fields` → 50, `signature_hashes` → 100, strategy dicts pruned). Only the outer map was left unbounded — the same class of miss as the sibling caches that this codebase otherwise bounds carefully. This makes `_tool_patterns` an LRU `OrderedDict` capped at `_MAX_TRACKED_TOOLS` (1024 — well above any real deployment's tool count): recording a tool moves it to the most-recent end, and a new tool past the cap evicts the least-recently-recorded one. Both record paths go through a single `_touch_tool_pattern` helper. Reproduction (before): 1,524 distinct tool names → 1,524 entries. After: 1,024 (capped); the oldest are evicted, a re-recorded tool is refreshed and survives. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cache/compression_feedback.py`: `_tool_patterns` is now an `OrderedDict`; added `_MAX_TRACKED_TOOLS = 1024` and a `_touch_tool_pattern` helper (get-or-create with LRU eviction + `move_to_end`); `record_compression` and `record_retrieval` both use it, replacing the duplicated `if tool_name not in ...: ...` blocks. - `tests/test_ccr_feedback.py`: added `test_tool_patterns_are_lru_bounded` — filling to the cap then recording a new tool keeps the map at the cap, evicts the LRU tool, and a refreshed tool survives. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output ```text tests/test_ccr_feedback.py -> 21 passed uvx ruff@0.16.2 check headroom/cache/compression_feedback.py tests/test_ccr_feedback.py -> All checks passed! uvx mypy@1.20.2 headroom/cache/compression_feedback.py -> Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1, ruff 0.16.2 and mypy 1.20.2 via uvx. - Exact command / steps: recorded compression events for 1,524 distinct tool names; before the fix `len(_tool_patterns) == 1524` (unbounded), after `== 1024` with the oldest evicted and the newest kept. Verified LRU refresh: after filling to the cap, re-recording `tool_0` then adding a new tool evicts `tool_1` (the LRU) while `tool_0` survives. - Observed result: the tracked-tool map is bounded regardless of client tool-name cardinality; active tools are retained. - Not tested: a live long-running heap-growth measurement (the unbounded growth is shown directly by the entry-count reproduction). ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: N/A. - Stable/default behavior changed: only past 1024 distinct tools, where the least-recently-recorded tool's learned pattern is dropped (it would be re-learned if the tool becomes active again). Below the cap, behavior is identical. - Kill switch / disable path: N/A (raise `_MAX_TRACKED_TOOLS` to relax). - Unsafe override required: no. - Qualification impact: none. - Rollback path: revert this commit; `_tool_patterns` goes back to an unbounded dict. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (N/A: internal cache bound) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Consistent with the LRU/`max_*` bounding this codebase already applies to `_pattern_counts` (traffic learner), the session trackers, and the various compression caches; this closes the one companion map that was left uncapped.
This commit is contained in:
+32
-10
@@ -30,6 +30,7 @@ from __future__ import annotations
|
||||
import re
|
||||
import threading
|
||||
import time
|
||||
from collections import OrderedDict
|
||||
from dataclasses import dataclass, field
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
@@ -38,6 +39,13 @@ from .compression_strategy_outcomes import CompressionStrategyOutcomes
|
||||
if TYPE_CHECKING:
|
||||
from .compression_store import CompressionStore, RetrievalEvent
|
||||
|
||||
# Cap on distinct tools tracked in ``_tool_patterns``. ``tool_name`` is
|
||||
# client-controlled — MCP servers can expose arbitrarily named tools — so the
|
||||
# outer key set is LRU-bounded like the interior per-pattern structures already
|
||||
# are (see LocalToolPattern's capped queries/fields/hashes). Well above the tool
|
||||
# count of any real deployment.
|
||||
_MAX_TRACKED_TOOLS = 1024
|
||||
|
||||
|
||||
@dataclass
|
||||
class LocalToolPattern:
|
||||
@@ -185,8 +193,10 @@ class CompressionFeedback:
|
||||
self._enable_learning = enable_learning
|
||||
self._lock = threading.Lock()
|
||||
|
||||
# Learned patterns per tool
|
||||
self._tool_patterns: dict[str, LocalToolPattern] = {}
|
||||
# Learned patterns per tool. LRU-ordered so the tracked-tool set can be
|
||||
# capped (tool_name is client-controlled); most-recently-recorded tools
|
||||
# stay, one-off tool names are evicted past _MAX_TRACKED_TOOLS.
|
||||
self._tool_patterns: OrderedDict[str, LocalToolPattern] = OrderedDict()
|
||||
|
||||
# Time-based tracking
|
||||
self._last_analysis: float = 0.0
|
||||
@@ -208,6 +218,24 @@ class CompressionFeedback:
|
||||
self._store = get_compression_store()
|
||||
return self._store
|
||||
|
||||
def _touch_tool_pattern(self, tool_name: str) -> LocalToolPattern:
|
||||
"""Get-or-create a tool's pattern, LRU-capping the tracked-tool set.
|
||||
|
||||
``tool_name`` is client-controlled, so the outer dict is bounded: past
|
||||
``_MAX_TRACKED_TOOLS`` the least-recently-recorded tool is evicted, and
|
||||
recording a tool moves it to the most-recent end. The caller holds
|
||||
``self._lock``.
|
||||
"""
|
||||
pattern = self._tool_patterns.get(tool_name)
|
||||
if pattern is None:
|
||||
if len(self._tool_patterns) >= _MAX_TRACKED_TOOLS:
|
||||
self._tool_patterns.popitem(last=False)
|
||||
pattern = LocalToolPattern(tool_name=tool_name)
|
||||
self._tool_patterns[tool_name] = pattern
|
||||
else:
|
||||
self._tool_patterns.move_to_end(tool_name)
|
||||
return pattern
|
||||
|
||||
def record_compression(
|
||||
self,
|
||||
tool_name: str | None,
|
||||
@@ -233,10 +261,7 @@ class CompressionFeedback:
|
||||
with self._lock:
|
||||
self._total_compressions += 1
|
||||
|
||||
if tool_name not in self._tool_patterns:
|
||||
self._tool_patterns[tool_name] = LocalToolPattern(tool_name=tool_name)
|
||||
|
||||
pattern = self._tool_patterns[tool_name]
|
||||
pattern = self._touch_tool_pattern(tool_name)
|
||||
pattern.total_compressions += 1
|
||||
pattern.last_compression = time.time()
|
||||
|
||||
@@ -289,10 +314,7 @@ class CompressionFeedback:
|
||||
with self._lock:
|
||||
self._total_retrievals += 1
|
||||
|
||||
if tool_name not in self._tool_patterns:
|
||||
self._tool_patterns[tool_name] = LocalToolPattern(tool_name=tool_name)
|
||||
|
||||
pattern = self._tool_patterns[tool_name]
|
||||
pattern = self._touch_tool_pattern(tool_name)
|
||||
pattern.total_retrievals += 1
|
||||
pattern.last_retrieval = time.time()
|
||||
|
||||
|
||||
@@ -41,6 +41,29 @@ class TestCompressionFeedback:
|
||||
assert "test_tool" in patterns
|
||||
assert patterns["test_tool"].total_compressions == 2
|
||||
|
||||
def test_tool_patterns_are_lru_bounded(self):
|
||||
"""``tool_name`` is client-controlled (MCP servers expose arbitrarily
|
||||
named tools), so the per-tool pattern map must not grow without bound on
|
||||
the process-global feedback singleton. It is LRU-capped: past
|
||||
``_MAX_TRACKED_TOOLS`` the least-recently-recorded tool is evicted, while
|
||||
a re-recorded tool is refreshed and survives."""
|
||||
from headroom.cache.compression_feedback import _MAX_TRACKED_TOOLS
|
||||
|
||||
feedback = CompressionFeedback()
|
||||
for i in range(_MAX_TRACKED_TOOLS):
|
||||
feedback.record_compression(f"tool_{i}", 100, 10)
|
||||
assert len(feedback._tool_patterns) == _MAX_TRACKED_TOOLS
|
||||
|
||||
# Refresh the oldest tool so it becomes most-recently-used.
|
||||
feedback.record_compression("tool_0", 100, 10)
|
||||
# A brand-new tool past the cap evicts the current LRU (tool_1), not tool_0.
|
||||
feedback.record_compression("tool_new", 100, 10)
|
||||
|
||||
assert len(feedback._tool_patterns) == _MAX_TRACKED_TOOLS # still capped
|
||||
assert "tool_new" in feedback._tool_patterns
|
||||
assert "tool_0" in feedback._tool_patterns # refreshed -> survived
|
||||
assert "tool_1" not in feedback._tool_patterns # evicted as LRU
|
||||
|
||||
def test_record_retrieval(self):
|
||||
"""Recording retrieval events updates patterns."""
|
||||
feedback = CompressionFeedback()
|
||||
|
||||
Reference in New Issue
Block a user