Files
headroom/tests/test_ccr_feedback.py
Abhay Singh 2d10b10ae4 perf(cache): LRU-bound the compression-feedback tool-pattern map (#3342)
## Description

`CompressionFeedback._tool_patterns` is a `dict[tool_name,
LocalToolPattern]`, and `tool_name` is client-controlled — MCP servers
can expose arbitrarily named tools. It was only ever grown
(`record_compression` / `record_retrieval`) or cleared wholesale
(`clear()`), with no per-key eviction. `get_compression_feedback()`
returns a process-global singleton, so the outer key set accumulated one
entry per distinct tool name for the whole process lifetime.

The interior per-pattern structures are all already capped
(`common_queries` → 100, `queried_fields` → 50, `signature_hashes` →
100, strategy dicts pruned). Only the outer map was left unbounded — the
same class of miss as the sibling caches that this codebase otherwise
bounds carefully.

This makes `_tool_patterns` an LRU `OrderedDict` capped at
`_MAX_TRACKED_TOOLS` (1024 — well above any real deployment's tool
count): recording a tool moves it to the most-recent end, and a new tool
past the cap evicts the least-recently-recorded one. Both record paths
go through a single `_touch_tool_pattern` helper.

Reproduction (before): 1,524 distinct tool names → 1,524 entries. After:
1,024 (capped); the oldest are evicted, a re-recorded tool is refreshed
and survives.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cache/compression_feedback.py`: `_tool_patterns` is now an
`OrderedDict`; added `_MAX_TRACKED_TOOLS = 1024` and a
`_touch_tool_pattern` helper (get-or-create with LRU eviction +
`move_to_end`); `record_compression` and `record_retrieval` both use it,
replacing the duplicated `if tool_name not in ...: ...` blocks.
- `tests/test_ccr_feedback.py`: added
`test_tool_patterns_are_lru_bounded` — filling to the cap then recording
a new tool keeps the map at the cap, evicts the LRU tool, and a
refreshed tool survives.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
tests/test_ccr_feedback.py  ->  21 passed
uvx ruff@0.16.2 check headroom/cache/compression_feedback.py tests/test_ccr_feedback.py  ->  All checks passed!
uvx mypy@1.20.2 headroom/cache/compression_feedback.py  ->  Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: recorded compression events for 1,524 distinct
tool names; before the fix `len(_tool_patterns) == 1524` (unbounded),
after `== 1024` with the oldest evicted and the newest kept. Verified
LRU refresh: after filling to the cap, re-recording `tool_0` then adding
a new tool evicts `tool_1` (the LRU) while `tool_0` survives.
- Observed result: the tracked-tool map is bounded regardless of client
tool-name cardinality; active tools are retained.
- Not tested: a live long-running heap-growth measurement (the unbounded
growth is shown directly by the entry-count reproduction).

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: only past 1024 distinct tools, where
the least-recently-recorded tool's learned pattern is dropped (it would
be re-learned if the tool becomes active again). Below the cap, behavior
is identical.
- Kill switch / disable path: N/A (raise `_MAX_TRACKED_TOOLS` to relax).
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this commit; `_tool_patterns` goes back to an
unbounded dict.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal cache bound)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Consistent with the LRU/`max_*` bounding this codebase already applies
to `_pattern_counts` (traffic learner), the session trackers, and the
various compression caches; this closes the one companion map that was
left uncapped.
2026-09-24 22:31:14 -05:00

423 lines
14 KiB
Python

"""Tests for CCR feedback loop and pattern learning."""
import time
import pytest
from headroom.cache.compression_feedback import (
CompressionFeedback,
LocalToolPattern,
get_compression_feedback,
reset_compression_feedback,
)
from headroom.cache.compression_store import (
CompressionStore,
RetrievalEvent,
reset_compression_store,
)
@pytest.fixture(autouse=True)
def reset_globals():
"""Reset global state before each test."""
reset_compression_feedback()
reset_compression_store()
yield
reset_compression_feedback()
reset_compression_store()
class TestCompressionFeedback:
"""Test CompressionFeedback analyzer."""
def test_record_compression(self):
"""Recording compression events updates tool patterns."""
feedback = CompressionFeedback()
feedback.record_compression("test_tool", 100, 10)
feedback.record_compression("test_tool", 200, 20)
patterns = feedback.get_all_patterns()
assert "test_tool" in patterns
assert patterns["test_tool"].total_compressions == 2
def test_tool_patterns_are_lru_bounded(self):
"""``tool_name`` is client-controlled (MCP servers expose arbitrarily
named tools), so the per-tool pattern map must not grow without bound on
the process-global feedback singleton. It is LRU-capped: past
``_MAX_TRACKED_TOOLS`` the least-recently-recorded tool is evicted, while
a re-recorded tool is refreshed and survives."""
from headroom.cache.compression_feedback import _MAX_TRACKED_TOOLS
feedback = CompressionFeedback()
for i in range(_MAX_TRACKED_TOOLS):
feedback.record_compression(f"tool_{i}", 100, 10)
assert len(feedback._tool_patterns) == _MAX_TRACKED_TOOLS
# Refresh the oldest tool so it becomes most-recently-used.
feedback.record_compression("tool_0", 100, 10)
# A brand-new tool past the cap evicts the current LRU (tool_1), not tool_0.
feedback.record_compression("tool_new", 100, 10)
assert len(feedback._tool_patterns) == _MAX_TRACKED_TOOLS # still capped
assert "tool_new" in feedback._tool_patterns
assert "tool_0" in feedback._tool_patterns # refreshed -> survived
assert "tool_1" not in feedback._tool_patterns # evicted as LRU
def test_record_retrieval(self):
"""Recording retrieval events updates patterns."""
feedback = CompressionFeedback()
feedback.record_compression("test_tool", 100, 10)
event = RetrievalEvent(
hash="abc123",
query="find errors",
items_retrieved=50,
total_items=100,
tool_name="test_tool",
timestamp=time.time(),
retrieval_type="search",
)
feedback.record_retrieval(event)
patterns = feedback.get_all_patterns()
assert patterns["test_tool"].total_retrievals == 1
assert patterns["test_tool"].search_retrievals == 1
def test_retrieval_rate_calculation(self):
"""Retrieval rate is calculated correctly."""
feedback = CompressionFeedback()
# 10 compressions
for _ in range(10):
feedback.record_compression("test_tool", 100, 10)
# 5 retrievals (50% retrieval rate)
for _ in range(5):
event = RetrievalEvent(
hash="abc123",
query=None,
items_retrieved=100,
total_items=100,
tool_name="test_tool",
timestamp=time.time(),
retrieval_type="full",
)
feedback.record_retrieval(event)
pattern = feedback.get_all_patterns()["test_tool"]
assert pattern.retrieval_rate == 0.5
assert pattern.full_retrieval_rate == 1.0 # All were full retrievals
def test_eviction_success_is_not_counted_as_retrieval(self):
"""An eviction-without-retrieval is a compression success, not a retrieval.
The event arrives with retrieval_type="eviction_success". Because that
isn't "full" it used to fall into the search_retrievals branch and
inflate retrieval_rate/search_rate, driving get_compression_hints toward
less aggressive compression — the inverse of the intended signal. It must
leave the retrieval counters untouched.
"""
feedback = CompressionFeedback()
feedback.record_compression("test_tool", 100, 10)
event = RetrievalEvent(
hash="abc123",
query=None,
items_retrieved=0,
total_items=100,
tool_name="test_tool",
timestamp=time.time(),
retrieval_type="eviction_success",
)
feedback.record_retrieval(event, strategy="smart")
pattern = feedback.get_all_patterns()["test_tool"]
assert pattern.total_retrievals == 0
assert pattern.search_retrievals == 0
assert pattern.retrieval_rate == 0.0 # a successful compression, not a retrieval
# A genuine retrieval afterward is still counted.
feedback.record_retrieval(
RetrievalEvent(
hash="def456",
query="find errors",
items_retrieved=50,
total_items=100,
tool_name="test_tool",
timestamp=time.time(),
retrieval_type="search",
)
)
pattern = feedback.get_all_patterns()["test_tool"]
assert pattern.total_retrievals == 1
assert pattern.search_retrievals == 1
def test_hints_default_with_no_data(self):
"""Default hints returned when no data exists."""
feedback = CompressionFeedback()
hints = feedback.get_compression_hints("unknown_tool")
assert hints.max_items == 15 # Default
assert hints.skip_compression is False
assert "No pattern data" in hints.reason
def test_hints_insufficient_samples(self):
"""Default hints returned with insufficient samples."""
feedback = CompressionFeedback()
# Only 3 compressions (need 5 for hints)
for _ in range(3):
feedback.record_compression("test_tool", 100, 10)
hints = feedback.get_compression_hints("test_tool")
assert hints.max_items == 15 # Default
assert "Insufficient data" in hints.reason
def test_hints_high_retrieval_rate_less_aggressive(self):
"""High retrieval rate results in less aggressive compression."""
feedback = CompressionFeedback()
# 10 compressions
for _ in range(10):
feedback.record_compression("test_tool", 100, 10)
# 6 retrievals (60% retrieval rate - HIGH)
for _ in range(6):
event = RetrievalEvent(
hash="abc123",
query="search query",
items_retrieved=50,
total_items=100,
tool_name="test_tool",
timestamp=time.time(),
retrieval_type="search",
)
feedback.record_retrieval(event)
hints = feedback.get_compression_hints("test_tool")
assert hints.max_items > 15 # Should be more than default
assert hints.aggressiveness < 0.7 # Less aggressive
assert "High retrieval rate" in hints.reason or "less aggressive" in hints.reason.lower()
def test_hints_very_high_full_retrieval_skips_compression(self):
"""Very high full retrieval rate recommends skipping compression."""
feedback = CompressionFeedback()
# 10 compressions
for _ in range(10):
feedback.record_compression("test_tool", 100, 10)
# 9 FULL retrievals (90% retrieval rate, all full)
for _ in range(9):
event = RetrievalEvent(
hash="abc123",
query=None,
items_retrieved=100,
total_items=100,
tool_name="test_tool",
timestamp=time.time(),
retrieval_type="full",
)
feedback.record_retrieval(event)
hints = feedback.get_compression_hints("test_tool")
assert hints.skip_compression is True
assert "skip compression" in hints.reason.lower()
def test_hints_low_retrieval_rate_aggressive(self):
"""Low retrieval rate means current compression is effective."""
feedback = CompressionFeedback()
# 10 compressions
for _ in range(10):
feedback.record_compression("test_tool", 100, 10)
# Only 1 retrieval (10% - LOW)
event = RetrievalEvent(
hash="abc123",
query=None,
items_retrieved=100,
total_items=100,
tool_name="test_tool",
timestamp=time.time(),
retrieval_type="full",
)
feedback.record_retrieval(event)
hints = feedback.get_compression_hints("test_tool")
assert hints.max_items == 15 # Default/aggressive
assert "effective" in hints.reason.lower() or "Low retrieval" in hints.reason
def test_common_queries_tracked(self):
"""Common search queries are tracked per tool."""
feedback = CompressionFeedback()
feedback.record_compression("test_tool", 100, 10)
queries = ["find errors", "find errors", "status:failed", "error"]
for q in queries:
event = RetrievalEvent(
hash="abc123",
query=q,
items_retrieved=10,
total_items=100,
tool_name="test_tool",
timestamp=time.time(),
retrieval_type="search",
)
feedback.record_retrieval(event)
pattern = feedback.get_all_patterns()["test_tool"]
assert "find errors" in pattern.common_queries
assert pattern.common_queries["find errors"] == 2
def test_queried_fields_extracted(self):
"""Field names are extracted from queries."""
feedback = CompressionFeedback()
feedback.record_compression("test_tool", 100, 10)
# Query with field:value patterns
event = RetrievalEvent(
hash="abc123",
query="status:error id=12345",
items_retrieved=10,
total_items=100,
tool_name="test_tool",
timestamp=time.time(),
retrieval_type="search",
)
feedback.record_retrieval(event)
pattern = feedback.get_all_patterns()["test_tool"]
assert "status" in pattern.queried_fields
assert "id" in pattern.queried_fields
def test_preserve_fields_in_hints(self):
"""Frequently queried fields appear in hints."""
feedback = CompressionFeedback()
# Multiple compressions
for _ in range(10):
feedback.record_compression("test_tool", 100, 10)
# Multiple queries with same fields
for _ in range(5):
event = RetrievalEvent(
hash="abc123",
query="status:error code:500",
items_retrieved=10,
total_items=100,
tool_name="test_tool",
timestamp=time.time(),
retrieval_type="search",
)
feedback.record_retrieval(event)
hints = feedback.get_compression_hints("test_tool")
# Even if retrieval rate triggers hints, preserve_fields should be populated
assert len(hints.preserve_fields) > 0
def test_stats_returns_overview(self):
"""get_stats returns comprehensive overview."""
feedback = CompressionFeedback()
feedback.record_compression("tool_a", 100, 10)
feedback.record_compression("tool_b", 200, 20)
stats = feedback.get_stats()
assert stats["total_compressions"] == 2
assert stats["tools_tracked"] == 2
assert "tool_a" in stats["tool_patterns"]
assert "tool_b" in stats["tool_patterns"]
def test_clear_resets_state(self):
"""clear() removes all learned patterns."""
feedback = CompressionFeedback()
feedback.record_compression("test_tool", 100, 10)
feedback.clear()
assert len(feedback.get_all_patterns()) == 0
stats = feedback.get_stats()
assert stats["total_compressions"] == 0
class TestLocalToolPattern:
"""Test LocalToolPattern dataclass."""
def test_retrieval_rate_zero_compressions(self):
"""Retrieval rate is 0 when no compressions."""
pattern = LocalToolPattern(tool_name="test")
assert pattern.retrieval_rate == 0.0
def test_full_retrieval_rate_zero_retrievals(self):
"""Full retrieval rate is 0 when no retrievals."""
pattern = LocalToolPattern(tool_name="test")
assert pattern.full_retrieval_rate == 0.0
def test_search_rate_calculation(self):
"""Search rate is calculated correctly."""
pattern = LocalToolPattern(
tool_name="test",
total_retrievals=10,
full_retrievals=3,
search_retrievals=7,
)
assert pattern.search_rate == 0.7
class TestGlobalFeedback:
"""Test global feedback singleton."""
def test_singleton_returns_same_instance(self):
"""get_compression_feedback returns same instance."""
fb1 = get_compression_feedback()
fb2 = get_compression_feedback()
assert fb1 is fb2
def test_reset_clears_singleton(self):
"""reset_compression_feedback creates new instance."""
fb1 = get_compression_feedback()
fb1.record_compression("test", 100, 10)
reset_compression_feedback()
fb2 = get_compression_feedback()
assert len(fb2.get_all_patterns()) == 0
class TestFeedbackIntegrationWithStore:
"""Test feedback integration with CompressionStore."""
def test_store_notifies_feedback_on_retrieval(self):
"""CompressionStore adds events to pending for feedback processing."""
store = CompressionStore()
# Store content
hash_key = store.store(
original='[{"id": 1}, {"id": 2}]',
compressed='[{"id": 1}]',
original_item_count=2,
compressed_item_count=1,
tool_name="test_tool",
)
# Retrieve (should log event)
store.retrieve(hash_key)
# Process pending events (uses global feedback)
store.process_pending_feedback()
# Now global feedback should have the retrieval
feedback = get_compression_feedback()
patterns = feedback.get_all_patterns()
assert "test_tool" in patterns
assert patterns["test_tool"].total_retrievals == 1