perf(cache): LRU-bound the compression-feedback tool-pattern map (#3342)

## Description

`CompressionFeedback._tool_patterns` is a `dict[tool_name,
LocalToolPattern]`, and `tool_name` is client-controlled — MCP servers
can expose arbitrarily named tools. It was only ever grown
(`record_compression` / `record_retrieval`) or cleared wholesale
(`clear()`), with no per-key eviction. `get_compression_feedback()`
returns a process-global singleton, so the outer key set accumulated one
entry per distinct tool name for the whole process lifetime.

The interior per-pattern structures are all already capped
(`common_queries` → 100, `queried_fields` → 50, `signature_hashes` →
100, strategy dicts pruned). Only the outer map was left unbounded — the
same class of miss as the sibling caches that this codebase otherwise
bounds carefully.

This makes `_tool_patterns` an LRU `OrderedDict` capped at
`_MAX_TRACKED_TOOLS` (1024 — well above any real deployment's tool
count): recording a tool moves it to the most-recent end, and a new tool
past the cap evicts the least-recently-recorded one. Both record paths
go through a single `_touch_tool_pattern` helper.

Reproduction (before): 1,524 distinct tool names → 1,524 entries. After:
1,024 (capped); the oldest are evicted, a re-recorded tool is refreshed
and survives.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cache/compression_feedback.py`: `_tool_patterns` is now an
`OrderedDict`; added `_MAX_TRACKED_TOOLS = 1024` and a
`_touch_tool_pattern` helper (get-or-create with LRU eviction +
`move_to_end`); `record_compression` and `record_retrieval` both use it,
replacing the duplicated `if tool_name not in ...: ...` blocks.
- `tests/test_ccr_feedback.py`: added
`test_tool_patterns_are_lru_bounded` — filling to the cap then recording
a new tool keeps the map at the cap, evicts the LRU tool, and a
refreshed tool survives.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
tests/test_ccr_feedback.py  ->  21 passed
uvx ruff@0.16.2 check headroom/cache/compression_feedback.py tests/test_ccr_feedback.py  ->  All checks passed!
uvx mypy@1.20.2 headroom/cache/compression_feedback.py  ->  Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: recorded compression events for 1,524 distinct
tool names; before the fix `len(_tool_patterns) == 1524` (unbounded),
after `== 1024` with the oldest evicted and the newest kept. Verified
LRU refresh: after filling to the cap, re-recording `tool_0` then adding
a new tool evicts `tool_1` (the LRU) while `tool_0` survives.
- Observed result: the tracked-tool map is bounded regardless of client
tool-name cardinality; active tools are retained.
- Not tested: a live long-running heap-growth measurement (the unbounded
growth is shown directly by the entry-count reproduction).

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: only past 1024 distinct tools, where
the least-recently-recorded tool's learned pattern is dropped (it would
be re-learned if the tool becomes active again). Below the cap, behavior
is identical.
- Kill switch / disable path: N/A (raise `_MAX_TRACKED_TOOLS` to relax).
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this commit; `_tool_patterns` goes back to an
unbounded dict.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal cache bound)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Consistent with the LRU/`max_*` bounding this codebase already applies
to `_pattern_counts` (traffic learner), the session trackers, and the
various compression caches; this closes the one companion map that was
left uncapped.
This commit is contained in:
Abhay Singh
2026-09-24 22:31:14 -05:00
committed by GitHub
parent c7ebaedd74
commit 2d10b10ae4
2 changed files with 55 additions and 10 deletions
+32 -10
View File
@@ -30,6 +30,7 @@ from __future__ import annotations
import re
import threading
import time
from collections import OrderedDict
from dataclasses import dataclass, field
from typing import TYPE_CHECKING, Any
@@ -38,6 +39,13 @@ from .compression_strategy_outcomes import CompressionStrategyOutcomes
if TYPE_CHECKING:
from .compression_store import CompressionStore, RetrievalEvent
# Cap on distinct tools tracked in ``_tool_patterns``. ``tool_name`` is
# client-controlled — MCP servers can expose arbitrarily named tools — so the
# outer key set is LRU-bounded like the interior per-pattern structures already
# are (see LocalToolPattern's capped queries/fields/hashes). Well above the tool
# count of any real deployment.
_MAX_TRACKED_TOOLS = 1024
@dataclass
class LocalToolPattern:
@@ -185,8 +193,10 @@ class CompressionFeedback:
self._enable_learning = enable_learning
self._lock = threading.Lock()
# Learned patterns per tool
self._tool_patterns: dict[str, LocalToolPattern] = {}
# Learned patterns per tool. LRU-ordered so the tracked-tool set can be
# capped (tool_name is client-controlled); most-recently-recorded tools
# stay, one-off tool names are evicted past _MAX_TRACKED_TOOLS.
self._tool_patterns: OrderedDict[str, LocalToolPattern] = OrderedDict()
# Time-based tracking
self._last_analysis: float = 0.0
@@ -208,6 +218,24 @@ class CompressionFeedback:
self._store = get_compression_store()
return self._store
def _touch_tool_pattern(self, tool_name: str) -> LocalToolPattern:
"""Get-or-create a tool's pattern, LRU-capping the tracked-tool set.
``tool_name`` is client-controlled, so the outer dict is bounded: past
``_MAX_TRACKED_TOOLS`` the least-recently-recorded tool is evicted, and
recording a tool moves it to the most-recent end. The caller holds
``self._lock``.
"""
pattern = self._tool_patterns.get(tool_name)
if pattern is None:
if len(self._tool_patterns) >= _MAX_TRACKED_TOOLS:
self._tool_patterns.popitem(last=False)
pattern = LocalToolPattern(tool_name=tool_name)
self._tool_patterns[tool_name] = pattern
else:
self._tool_patterns.move_to_end(tool_name)
return pattern
def record_compression(
self,
tool_name: str | None,
@@ -233,10 +261,7 @@ class CompressionFeedback:
with self._lock:
self._total_compressions += 1
if tool_name not in self._tool_patterns:
self._tool_patterns[tool_name] = LocalToolPattern(tool_name=tool_name)
pattern = self._tool_patterns[tool_name]
pattern = self._touch_tool_pattern(tool_name)
pattern.total_compressions += 1
pattern.last_compression = time.time()
@@ -289,10 +314,7 @@ class CompressionFeedback:
with self._lock:
self._total_retrievals += 1
if tool_name not in self._tool_patterns:
self._tool_patterns[tool_name] = LocalToolPattern(tool_name=tool_name)
pattern = self._tool_patterns[tool_name]
pattern = self._touch_tool_pattern(tool_name)
pattern.total_retrievals += 1
pattern.last_retrieval = time.time()
+23
View File
@@ -41,6 +41,29 @@ class TestCompressionFeedback:
assert "test_tool" in patterns
assert patterns["test_tool"].total_compressions == 2
def test_tool_patterns_are_lru_bounded(self):
"""``tool_name`` is client-controlled (MCP servers expose arbitrarily
named tools), so the per-tool pattern map must not grow without bound on
the process-global feedback singleton. It is LRU-capped: past
``_MAX_TRACKED_TOOLS`` the least-recently-recorded tool is evicted, while
a re-recorded tool is refreshed and survives."""
from headroom.cache.compression_feedback import _MAX_TRACKED_TOOLS
feedback = CompressionFeedback()
for i in range(_MAX_TRACKED_TOOLS):
feedback.record_compression(f"tool_{i}", 100, 10)
assert len(feedback._tool_patterns) == _MAX_TRACKED_TOOLS
# Refresh the oldest tool so it becomes most-recently-used.
feedback.record_compression("tool_0", 100, 10)
# A brand-new tool past the cap evicts the current LRU (tool_1), not tool_0.
feedback.record_compression("tool_new", 100, 10)
assert len(feedback._tool_patterns) == _MAX_TRACKED_TOOLS # still capped
assert "tool_new" in feedback._tool_patterns
assert "tool_0" in feedback._tool_patterns # refreshed -> survived
assert "tool_1" not in feedback._tool_patterns # evicted as LRU
def test_record_retrieval(self):
"""Recording retrieval events updates patterns."""
feedback = CompressionFeedback()