mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description `CompressionFeedback._tool_patterns` is a `dict[tool_name, LocalToolPattern]`, and `tool_name` is client-controlled — MCP servers can expose arbitrarily named tools. It was only ever grown (`record_compression` / `record_retrieval`) or cleared wholesale (`clear()`), with no per-key eviction. `get_compression_feedback()` returns a process-global singleton, so the outer key set accumulated one entry per distinct tool name for the whole process lifetime. The interior per-pattern structures are all already capped (`common_queries` → 100, `queried_fields` → 50, `signature_hashes` → 100, strategy dicts pruned). Only the outer map was left unbounded — the same class of miss as the sibling caches that this codebase otherwise bounds carefully. This makes `_tool_patterns` an LRU `OrderedDict` capped at `_MAX_TRACKED_TOOLS` (1024 — well above any real deployment's tool count): recording a tool moves it to the most-recent end, and a new tool past the cap evicts the least-recently-recorded one. Both record paths go through a single `_touch_tool_pattern` helper. Reproduction (before): 1,524 distinct tool names → 1,524 entries. After: 1,024 (capped); the oldest are evicted, a re-recorded tool is refreshed and survives. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cache/compression_feedback.py`: `_tool_patterns` is now an `OrderedDict`; added `_MAX_TRACKED_TOOLS = 1024` and a `_touch_tool_pattern` helper (get-or-create with LRU eviction + `move_to_end`); `record_compression` and `record_retrieval` both use it, replacing the duplicated `if tool_name not in ...: ...` blocks. - `tests/test_ccr_feedback.py`: added `test_tool_patterns_are_lru_bounded` — filling to the cap then recording a new tool keeps the map at the cap, evicts the LRU tool, and a refreshed tool survives. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output ```text tests/test_ccr_feedback.py -> 21 passed uvx ruff@0.16.2 check headroom/cache/compression_feedback.py tests/test_ccr_feedback.py -> All checks passed! uvx mypy@1.20.2 headroom/cache/compression_feedback.py -> Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1, ruff 0.16.2 and mypy 1.20.2 via uvx. - Exact command / steps: recorded compression events for 1,524 distinct tool names; before the fix `len(_tool_patterns) == 1524` (unbounded), after `== 1024` with the oldest evicted and the newest kept. Verified LRU refresh: after filling to the cap, re-recording `tool_0` then adding a new tool evicts `tool_1` (the LRU) while `tool_0` survives. - Observed result: the tracked-tool map is bounded regardless of client tool-name cardinality; active tools are retained. - Not tested: a live long-running heap-growth measurement (the unbounded growth is shown directly by the entry-count reproduction). ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: N/A. - Stable/default behavior changed: only past 1024 distinct tools, where the least-recently-recorded tool's learned pattern is dropped (it would be re-learned if the tool becomes active again). Below the cap, behavior is identical. - Kill switch / disable path: N/A (raise `_MAX_TRACKED_TOOLS` to relax). - Unsafe override required: no. - Qualification impact: none. - Rollback path: revert this commit; `_tool_patterns` goes back to an unbounded dict. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (N/A: internal cache bound) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Consistent with the LRU/`max_*` bounding this codebase already applies to `_pattern_counts` (traffic learner), the session trackers, and the various compression caches; this closes the one companion map that was left uncapped.
423 lines
14 KiB
Python
423 lines
14 KiB
Python
"""Tests for CCR feedback loop and pattern learning."""
|
|
|
|
import time
|
|
|
|
import pytest
|
|
|
|
from headroom.cache.compression_feedback import (
|
|
CompressionFeedback,
|
|
LocalToolPattern,
|
|
get_compression_feedback,
|
|
reset_compression_feedback,
|
|
)
|
|
from headroom.cache.compression_store import (
|
|
CompressionStore,
|
|
RetrievalEvent,
|
|
reset_compression_store,
|
|
)
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def reset_globals():
|
|
"""Reset global state before each test."""
|
|
reset_compression_feedback()
|
|
reset_compression_store()
|
|
yield
|
|
reset_compression_feedback()
|
|
reset_compression_store()
|
|
|
|
|
|
class TestCompressionFeedback:
|
|
"""Test CompressionFeedback analyzer."""
|
|
|
|
def test_record_compression(self):
|
|
"""Recording compression events updates tool patterns."""
|
|
feedback = CompressionFeedback()
|
|
|
|
feedback.record_compression("test_tool", 100, 10)
|
|
feedback.record_compression("test_tool", 200, 20)
|
|
|
|
patterns = feedback.get_all_patterns()
|
|
assert "test_tool" in patterns
|
|
assert patterns["test_tool"].total_compressions == 2
|
|
|
|
def test_tool_patterns_are_lru_bounded(self):
|
|
"""``tool_name`` is client-controlled (MCP servers expose arbitrarily
|
|
named tools), so the per-tool pattern map must not grow without bound on
|
|
the process-global feedback singleton. It is LRU-capped: past
|
|
``_MAX_TRACKED_TOOLS`` the least-recently-recorded tool is evicted, while
|
|
a re-recorded tool is refreshed and survives."""
|
|
from headroom.cache.compression_feedback import _MAX_TRACKED_TOOLS
|
|
|
|
feedback = CompressionFeedback()
|
|
for i in range(_MAX_TRACKED_TOOLS):
|
|
feedback.record_compression(f"tool_{i}", 100, 10)
|
|
assert len(feedback._tool_patterns) == _MAX_TRACKED_TOOLS
|
|
|
|
# Refresh the oldest tool so it becomes most-recently-used.
|
|
feedback.record_compression("tool_0", 100, 10)
|
|
# A brand-new tool past the cap evicts the current LRU (tool_1), not tool_0.
|
|
feedback.record_compression("tool_new", 100, 10)
|
|
|
|
assert len(feedback._tool_patterns) == _MAX_TRACKED_TOOLS # still capped
|
|
assert "tool_new" in feedback._tool_patterns
|
|
assert "tool_0" in feedback._tool_patterns # refreshed -> survived
|
|
assert "tool_1" not in feedback._tool_patterns # evicted as LRU
|
|
|
|
def test_record_retrieval(self):
|
|
"""Recording retrieval events updates patterns."""
|
|
feedback = CompressionFeedback()
|
|
feedback.record_compression("test_tool", 100, 10)
|
|
|
|
event = RetrievalEvent(
|
|
hash="abc123",
|
|
query="find errors",
|
|
items_retrieved=50,
|
|
total_items=100,
|
|
tool_name="test_tool",
|
|
timestamp=time.time(),
|
|
retrieval_type="search",
|
|
)
|
|
feedback.record_retrieval(event)
|
|
|
|
patterns = feedback.get_all_patterns()
|
|
assert patterns["test_tool"].total_retrievals == 1
|
|
assert patterns["test_tool"].search_retrievals == 1
|
|
|
|
def test_retrieval_rate_calculation(self):
|
|
"""Retrieval rate is calculated correctly."""
|
|
feedback = CompressionFeedback()
|
|
|
|
# 10 compressions
|
|
for _ in range(10):
|
|
feedback.record_compression("test_tool", 100, 10)
|
|
|
|
# 5 retrievals (50% retrieval rate)
|
|
for _ in range(5):
|
|
event = RetrievalEvent(
|
|
hash="abc123",
|
|
query=None,
|
|
items_retrieved=100,
|
|
total_items=100,
|
|
tool_name="test_tool",
|
|
timestamp=time.time(),
|
|
retrieval_type="full",
|
|
)
|
|
feedback.record_retrieval(event)
|
|
|
|
pattern = feedback.get_all_patterns()["test_tool"]
|
|
assert pattern.retrieval_rate == 0.5
|
|
assert pattern.full_retrieval_rate == 1.0 # All were full retrievals
|
|
|
|
def test_eviction_success_is_not_counted_as_retrieval(self):
|
|
"""An eviction-without-retrieval is a compression success, not a retrieval.
|
|
|
|
The event arrives with retrieval_type="eviction_success". Because that
|
|
isn't "full" it used to fall into the search_retrievals branch and
|
|
inflate retrieval_rate/search_rate, driving get_compression_hints toward
|
|
less aggressive compression — the inverse of the intended signal. It must
|
|
leave the retrieval counters untouched.
|
|
"""
|
|
feedback = CompressionFeedback()
|
|
feedback.record_compression("test_tool", 100, 10)
|
|
|
|
event = RetrievalEvent(
|
|
hash="abc123",
|
|
query=None,
|
|
items_retrieved=0,
|
|
total_items=100,
|
|
tool_name="test_tool",
|
|
timestamp=time.time(),
|
|
retrieval_type="eviction_success",
|
|
)
|
|
feedback.record_retrieval(event, strategy="smart")
|
|
|
|
pattern = feedback.get_all_patterns()["test_tool"]
|
|
assert pattern.total_retrievals == 0
|
|
assert pattern.search_retrievals == 0
|
|
assert pattern.retrieval_rate == 0.0 # a successful compression, not a retrieval
|
|
|
|
# A genuine retrieval afterward is still counted.
|
|
feedback.record_retrieval(
|
|
RetrievalEvent(
|
|
hash="def456",
|
|
query="find errors",
|
|
items_retrieved=50,
|
|
total_items=100,
|
|
tool_name="test_tool",
|
|
timestamp=time.time(),
|
|
retrieval_type="search",
|
|
)
|
|
)
|
|
pattern = feedback.get_all_patterns()["test_tool"]
|
|
assert pattern.total_retrievals == 1
|
|
assert pattern.search_retrievals == 1
|
|
|
|
def test_hints_default_with_no_data(self):
|
|
"""Default hints returned when no data exists."""
|
|
feedback = CompressionFeedback()
|
|
|
|
hints = feedback.get_compression_hints("unknown_tool")
|
|
|
|
assert hints.max_items == 15 # Default
|
|
assert hints.skip_compression is False
|
|
assert "No pattern data" in hints.reason
|
|
|
|
def test_hints_insufficient_samples(self):
|
|
"""Default hints returned with insufficient samples."""
|
|
feedback = CompressionFeedback()
|
|
|
|
# Only 3 compressions (need 5 for hints)
|
|
for _ in range(3):
|
|
feedback.record_compression("test_tool", 100, 10)
|
|
|
|
hints = feedback.get_compression_hints("test_tool")
|
|
|
|
assert hints.max_items == 15 # Default
|
|
assert "Insufficient data" in hints.reason
|
|
|
|
def test_hints_high_retrieval_rate_less_aggressive(self):
|
|
"""High retrieval rate results in less aggressive compression."""
|
|
feedback = CompressionFeedback()
|
|
|
|
# 10 compressions
|
|
for _ in range(10):
|
|
feedback.record_compression("test_tool", 100, 10)
|
|
|
|
# 6 retrievals (60% retrieval rate - HIGH)
|
|
for _ in range(6):
|
|
event = RetrievalEvent(
|
|
hash="abc123",
|
|
query="search query",
|
|
items_retrieved=50,
|
|
total_items=100,
|
|
tool_name="test_tool",
|
|
timestamp=time.time(),
|
|
retrieval_type="search",
|
|
)
|
|
feedback.record_retrieval(event)
|
|
|
|
hints = feedback.get_compression_hints("test_tool")
|
|
|
|
assert hints.max_items > 15 # Should be more than default
|
|
assert hints.aggressiveness < 0.7 # Less aggressive
|
|
assert "High retrieval rate" in hints.reason or "less aggressive" in hints.reason.lower()
|
|
|
|
def test_hints_very_high_full_retrieval_skips_compression(self):
|
|
"""Very high full retrieval rate recommends skipping compression."""
|
|
feedback = CompressionFeedback()
|
|
|
|
# 10 compressions
|
|
for _ in range(10):
|
|
feedback.record_compression("test_tool", 100, 10)
|
|
|
|
# 9 FULL retrievals (90% retrieval rate, all full)
|
|
for _ in range(9):
|
|
event = RetrievalEvent(
|
|
hash="abc123",
|
|
query=None,
|
|
items_retrieved=100,
|
|
total_items=100,
|
|
tool_name="test_tool",
|
|
timestamp=time.time(),
|
|
retrieval_type="full",
|
|
)
|
|
feedback.record_retrieval(event)
|
|
|
|
hints = feedback.get_compression_hints("test_tool")
|
|
|
|
assert hints.skip_compression is True
|
|
assert "skip compression" in hints.reason.lower()
|
|
|
|
def test_hints_low_retrieval_rate_aggressive(self):
|
|
"""Low retrieval rate means current compression is effective."""
|
|
feedback = CompressionFeedback()
|
|
|
|
# 10 compressions
|
|
for _ in range(10):
|
|
feedback.record_compression("test_tool", 100, 10)
|
|
|
|
# Only 1 retrieval (10% - LOW)
|
|
event = RetrievalEvent(
|
|
hash="abc123",
|
|
query=None,
|
|
items_retrieved=100,
|
|
total_items=100,
|
|
tool_name="test_tool",
|
|
timestamp=time.time(),
|
|
retrieval_type="full",
|
|
)
|
|
feedback.record_retrieval(event)
|
|
|
|
hints = feedback.get_compression_hints("test_tool")
|
|
|
|
assert hints.max_items == 15 # Default/aggressive
|
|
assert "effective" in hints.reason.lower() or "Low retrieval" in hints.reason
|
|
|
|
def test_common_queries_tracked(self):
|
|
"""Common search queries are tracked per tool."""
|
|
feedback = CompressionFeedback()
|
|
feedback.record_compression("test_tool", 100, 10)
|
|
|
|
queries = ["find errors", "find errors", "status:failed", "error"]
|
|
for q in queries:
|
|
event = RetrievalEvent(
|
|
hash="abc123",
|
|
query=q,
|
|
items_retrieved=10,
|
|
total_items=100,
|
|
tool_name="test_tool",
|
|
timestamp=time.time(),
|
|
retrieval_type="search",
|
|
)
|
|
feedback.record_retrieval(event)
|
|
|
|
pattern = feedback.get_all_patterns()["test_tool"]
|
|
assert "find errors" in pattern.common_queries
|
|
assert pattern.common_queries["find errors"] == 2
|
|
|
|
def test_queried_fields_extracted(self):
|
|
"""Field names are extracted from queries."""
|
|
feedback = CompressionFeedback()
|
|
feedback.record_compression("test_tool", 100, 10)
|
|
|
|
# Query with field:value patterns
|
|
event = RetrievalEvent(
|
|
hash="abc123",
|
|
query="status:error id=12345",
|
|
items_retrieved=10,
|
|
total_items=100,
|
|
tool_name="test_tool",
|
|
timestamp=time.time(),
|
|
retrieval_type="search",
|
|
)
|
|
feedback.record_retrieval(event)
|
|
|
|
pattern = feedback.get_all_patterns()["test_tool"]
|
|
assert "status" in pattern.queried_fields
|
|
assert "id" in pattern.queried_fields
|
|
|
|
def test_preserve_fields_in_hints(self):
|
|
"""Frequently queried fields appear in hints."""
|
|
feedback = CompressionFeedback()
|
|
|
|
# Multiple compressions
|
|
for _ in range(10):
|
|
feedback.record_compression("test_tool", 100, 10)
|
|
|
|
# Multiple queries with same fields
|
|
for _ in range(5):
|
|
event = RetrievalEvent(
|
|
hash="abc123",
|
|
query="status:error code:500",
|
|
items_retrieved=10,
|
|
total_items=100,
|
|
tool_name="test_tool",
|
|
timestamp=time.time(),
|
|
retrieval_type="search",
|
|
)
|
|
feedback.record_retrieval(event)
|
|
|
|
hints = feedback.get_compression_hints("test_tool")
|
|
|
|
# Even if retrieval rate triggers hints, preserve_fields should be populated
|
|
assert len(hints.preserve_fields) > 0
|
|
|
|
def test_stats_returns_overview(self):
|
|
"""get_stats returns comprehensive overview."""
|
|
feedback = CompressionFeedback()
|
|
|
|
feedback.record_compression("tool_a", 100, 10)
|
|
feedback.record_compression("tool_b", 200, 20)
|
|
|
|
stats = feedback.get_stats()
|
|
|
|
assert stats["total_compressions"] == 2
|
|
assert stats["tools_tracked"] == 2
|
|
assert "tool_a" in stats["tool_patterns"]
|
|
assert "tool_b" in stats["tool_patterns"]
|
|
|
|
def test_clear_resets_state(self):
|
|
"""clear() removes all learned patterns."""
|
|
feedback = CompressionFeedback()
|
|
feedback.record_compression("test_tool", 100, 10)
|
|
|
|
feedback.clear()
|
|
|
|
assert len(feedback.get_all_patterns()) == 0
|
|
stats = feedback.get_stats()
|
|
assert stats["total_compressions"] == 0
|
|
|
|
|
|
class TestLocalToolPattern:
|
|
"""Test LocalToolPattern dataclass."""
|
|
|
|
def test_retrieval_rate_zero_compressions(self):
|
|
"""Retrieval rate is 0 when no compressions."""
|
|
pattern = LocalToolPattern(tool_name="test")
|
|
assert pattern.retrieval_rate == 0.0
|
|
|
|
def test_full_retrieval_rate_zero_retrievals(self):
|
|
"""Full retrieval rate is 0 when no retrievals."""
|
|
pattern = LocalToolPattern(tool_name="test")
|
|
assert pattern.full_retrieval_rate == 0.0
|
|
|
|
def test_search_rate_calculation(self):
|
|
"""Search rate is calculated correctly."""
|
|
pattern = LocalToolPattern(
|
|
tool_name="test",
|
|
total_retrievals=10,
|
|
full_retrievals=3,
|
|
search_retrievals=7,
|
|
)
|
|
assert pattern.search_rate == 0.7
|
|
|
|
|
|
class TestGlobalFeedback:
|
|
"""Test global feedback singleton."""
|
|
|
|
def test_singleton_returns_same_instance(self):
|
|
"""get_compression_feedback returns same instance."""
|
|
fb1 = get_compression_feedback()
|
|
fb2 = get_compression_feedback()
|
|
assert fb1 is fb2
|
|
|
|
def test_reset_clears_singleton(self):
|
|
"""reset_compression_feedback creates new instance."""
|
|
fb1 = get_compression_feedback()
|
|
fb1.record_compression("test", 100, 10)
|
|
|
|
reset_compression_feedback()
|
|
|
|
fb2 = get_compression_feedback()
|
|
assert len(fb2.get_all_patterns()) == 0
|
|
|
|
|
|
class TestFeedbackIntegrationWithStore:
|
|
"""Test feedback integration with CompressionStore."""
|
|
|
|
def test_store_notifies_feedback_on_retrieval(self):
|
|
"""CompressionStore adds events to pending for feedback processing."""
|
|
store = CompressionStore()
|
|
|
|
# Store content
|
|
hash_key = store.store(
|
|
original='[{"id": 1}, {"id": 2}]',
|
|
compressed='[{"id": 1}]',
|
|
original_item_count=2,
|
|
compressed_item_count=1,
|
|
tool_name="test_tool",
|
|
)
|
|
|
|
# Retrieve (should log event)
|
|
store.retrieve(hash_key)
|
|
|
|
# Process pending events (uses global feedback)
|
|
store.process_pending_feedback()
|
|
|
|
# Now global feedback should have the retrieval
|
|
feedback = get_compression_feedback()
|
|
patterns = feedback.get_all_patterns()
|
|
assert "test_tool" in patterns
|
|
assert patterns["test_tool"].total_retrievals == 1
|