mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description
This change isolates TOIN learning by tenant in shared-proxy
deployments. Each pattern key gains a tenant namespace derived from a
configurable sanitized header, a stable auth/token hash, or the
backwards-compatible global namespace. Legacy keys migrate to global and
every resolution path emits a structured event.
## Threat model
Pre-F3, every request's tool-call patterns aggregated into one global
pool keyed by `(auth_mode, model_family, sig_hash)`. That works for an
OSS single-tenant deploy, but in a multi-tenant SaaS / shared-proxy
deploy:
1. **Cross-tenant pattern leakage.** Tenant A's compression patterns
silently train recommendations the proxy then applies to tenant B's
requests, degrading both tenants' compression quality.
2. **Structural fingerprint exposure.** TOIN dumps include
`tool_signature_hash` per slice. With one shared pool, a user with read
access to a dump file learns the structural fingerprint of every other
tenant's tool surface.
F3 partitions: every TOIN store key gets a `tenant_key` prefix derived
from the inbound request. Two distinct tenants therefore train two
isolated pattern pools.
## Tenant key derivation rules
| Source | When | Key derivation |
|------------|--------------------------------------------------------------|-------------------------------------------------------------------|
| `header` | `HEADROOM_TENANT_KEY_HEADER` (default
`X-Headroom-Tenant-ID`) is present and survives sanitization | Sanitized
header value (ASCII alphanum + `-_`, max 64 chars) |
| `hash` | No header but bearer token + `auth_mode` (from F1) present |
`sha256(f"{auth_mode}:{api_key_prefix}")[:24]` (api_key_prefix = first 8
chars of bearer); same SHA-256[:24] idiom as PR #395 |
| `global` | Neither header nor bearer present (OSS single-tenant / dev)
| Literal `"global"` namespace; pre-F3 patterns migrate here on first
load |
Header path takes precedence over hash. Header values that sanitize to
empty (pure unicode / control chars only) fall through to the next rule.
## Configuration
Single env var: `HEADROOM_TENANT_KEY_HEADER`. Default
`X-Headroom-Tenant-ID`. The header NAME is configurable; the
sanitization rules and the `"global"` literal are NOT — they're
load-bearing for namespace isolation.
## Structured log fields
Every resolution path emits exactly one `tenant_key_resolved` log event
(no silent fallbacks per `feedback_no_silent_fallbacks.md`):
| Field | Always present | Notes |
|--------------|----------------|------------------------------------------------|
| `event` | yes | Always `"tenant_key_resolved"` |
| `tenant_key` | yes | Resolved key (or `"global"`) |
| `source` | yes | One of `header` / `hash` / `global` |
| `header` | header path | Header name that matched |
| `auth_mode` | hash path | `payg` / `oauth` / `subscription` |
Operators can alert on a sustained `source="global"` rate to spot
mis-configured upstream auth proxies.
## Implementation
- **NEW** `headroom/proxy/tenant_key.py`: pure resolver. Public surface
= `resolve_tenant_key(request) -> tuple[str, str]`. Plus
`set_request_tenant_key()` / `get_current_tenant_key()` ContextVar pair
so TOIN's deep-stack `record_compression` calls observe the right tenant
slice.
- `headroom/telemetry/toin.py`: extends `PatternKey` from a 3-tuple to a
4-tuple `(tenant_key, auth_mode, model_family, sig_hash)`.
`_deserialize_pattern_key` is backward-compat with both pre-F3 formats
(3-tuple from PR-B5, bare `sig_hash` from v1.0). `record_compression` /
`record_retrieval` / `get_pattern` accept an optional `tenant_key`
kwarg; when absent they read from the ContextVar.
- `headroom/cli/toin_publish.py`: each `[[recommendation]]` row now
carries `tenant_key`; rows sort by `(tenant_key, auth_mode,
model_family, sig_hash)` for clean diffs.
- `headroom/proxy/handlers/anthropic.py` +
`headroom/proxy/handlers/openai.py`: call `resolve_tenant_key(request)`
once at request entry, right after the F1 `classify_auth_mode` block.
Three call sites total (1 Anthropic, 2 OpenAI). Result attached to
`request.state.tenant_key` / `request.state.tenant_key_source` AND set
on the ContextVar.
## Realignment build constraints (per project memory)
- **configurable**: `HEADROOM_TENANT_KEY_HEADER` env var.
- **no hardcodes**: header name + `"global"` literal are the only
constants; both motivated and documented in code.
- **no regexes**: sanitization uses string ops +
`unicodedata.category()`.
- **no silent fallbacks**: every resolution emits a structured log
event.
- **comprehensive tests**: header / hash / global paths + sanitization
edges + tenant-isolation property test.
- **structured logs**: `event=tenant_key_resolved` with `tenant_key`,
`source`, optional `header` / `auth_mode`.
## Test plan
- [x] `pytest tests/test_tenant_key.py` (29 cases — header / hash /
global paths, structured-log emission, sanitization edges, ContextVar
round-trip)
- [x] `pytest tests/test_toin_observation_only.py` (6 new isolation
cases — two-tenant divergence, ContextVar read, explicit kwarg
precedence, default migration to `"global"`)
- [x] `pytest tests/test_toin*.py` (existing TOIN tests updated for the
4-tuple key shape)
- [x] `pytest tests/test_proxy_tenant_key_integration.py` (4 cases —
handler wires `request.state.tenant_key` + ContextVar through all three
resolution paths)
- [x] `pytest tests/test_critical_fixes.py tests/test_critical_gaps.py
tests/test_proxy_ccr.py` (legacy 3-tuple references updated)
- [x] `make ci-precheck-python` — 176 tests pass (passed via pre-push
hook)
- [x] `make ci-precheck-commitlint` — clean (passed via pre-push hook)
Refs: F1 (PR #399, auth_mode classifier), F2.1 (PR #400, per-auth-mode
CompressionPolicy gates).
## Type of Change
- [x] Bug fix / tenant-isolation hardening
- [ ] New feature
- [ ] Breaking change
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring
## Changes Made
- Add a pure tenant-key resolver and request ContextVar.
- Extend TOIN pattern keys and published recommendations with a tenant
namespace while preserving legacy deserialization.
- Resolve and attach tenant identity at Anthropic and OpenAI request
entry.
- Add isolation, migration, sanitization, header, CCR, handler, and
executor coverage.
## Testing
- [x] Focused tenant/TOIN/proxy tests pass
- [x] Python CI precheck passes
- [x] Commitlint precheck passes
### Test Output
```text
176 focused/precheck tests passed
Current-main isolation, migration, header, CCR, and executor validation: green
```
## Real Behavior Proof
- Environment: current-main reconciled Python proxy/TOIN test
environment.
- Exact command / steps: ran tenant resolver, TOIN observation/publish,
handler integration, header-isolation, CCR, and compression-executor
suites.
- Observed result: distinct tenants train isolated pattern pools; legacy
patterns migrate to global; request handlers propagate the resolved
namespace through deep TOIN calls.
- Not tested: production multi-tenant traffic volume or an external TOIN
dump consumer.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
---------
Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
306 lines
11 KiB
Python
306 lines
11 KiB
Python
"""Tests demonstrating critical fixes for TOIN/CCR implementation.
|
|
|
|
These tests verify the before/after behavior of critical bug fixes:
|
|
1. TOIN confidence math error (line 721)
|
|
2. TOIN double-count bug (lines 354-358)
|
|
3. compression_feedback.py race condition (lines 481-491)
|
|
4. Unbounded strategy dicts in compression_feedback.py
|
|
5. SmartCrusher integration with TOIN
|
|
"""
|
|
|
|
import time
|
|
from unittest.mock import patch
|
|
|
|
|
|
class TestTOINConfidenceMathFix:
|
|
"""Test for CRITICAL: Confidence calculation math error in toin.py:721.
|
|
|
|
BUG: `user_boost = min(0.3, pattern.user_count / 10 * 0.1)`
|
|
Due to operator precedence: user_count / 10 * 0.1 = user_count * 0.01
|
|
- 3 users: 0.03 boost (too small)
|
|
- 10 users: 0.1 boost
|
|
- 30 users needed to hit 0.3 cap!
|
|
|
|
FIX: Should be `min(0.3, pattern.user_count * 0.03)` for meaningful boost
|
|
- 3 users: 0.09 boost
|
|
- 10 users: 0.3 boost (capped)
|
|
"""
|
|
|
|
def test_confidence_user_boost_at_3_users(self):
|
|
"""With 3 users (min for network effect), boost should be meaningful."""
|
|
from headroom.telemetry.toin import (
|
|
TOINConfig,
|
|
ToolIntelligenceNetwork,
|
|
ToolPattern,
|
|
reset_toin,
|
|
)
|
|
|
|
reset_toin()
|
|
config = TOINConfig(min_users_for_network_effect=3)
|
|
toin = ToolIntelligenceNetwork(config)
|
|
|
|
# Create pattern with 3 users (correct API: tool_signature_hash is first arg)
|
|
pattern = ToolPattern(
|
|
tool_signature_hash="test123",
|
|
user_count=3,
|
|
sample_size=100, # Good sample size
|
|
)
|
|
|
|
confidence = toin._calculate_confidence(pattern)
|
|
|
|
# Sample confidence = min(0.7, 100/100) = 0.7
|
|
# User boost for 3 users should be meaningful (>= 0.05)
|
|
# FIX: With user_count * 0.03: boost = 0.09, total = 0.79
|
|
# BUG: With user_count * 0.01: boost = 0.03, total = 0.73
|
|
|
|
# After fix, confidence should be at least 0.75
|
|
assert confidence >= 0.75, (
|
|
f"Confidence {confidence} too low for 3 users - user boost not meaningful"
|
|
)
|
|
|
|
def test_confidence_user_boost_at_10_users(self):
|
|
"""With 10 users, boost should hit or approach cap."""
|
|
from headroom.telemetry.toin import (
|
|
TOINConfig,
|
|
ToolIntelligenceNetwork,
|
|
ToolPattern,
|
|
reset_toin,
|
|
)
|
|
|
|
reset_toin()
|
|
config = TOINConfig(min_users_for_network_effect=3)
|
|
toin = ToolIntelligenceNetwork(config)
|
|
|
|
pattern = ToolPattern(
|
|
tool_signature_hash="test123",
|
|
user_count=10,
|
|
sample_size=100,
|
|
)
|
|
|
|
confidence = toin._calculate_confidence(pattern)
|
|
|
|
# With 10 users, should be near cap (0.95)
|
|
# Sample confidence = 0.7, user boost should be 0.3 (capped)
|
|
# Total = min(0.95, 0.7 + 0.3) = 0.95
|
|
# BUG: user_boost = 0.1, total = 0.8
|
|
|
|
assert confidence >= 0.9, f"Confidence {confidence} too low for 10 users"
|
|
|
|
|
|
class TestTOINDoubleCountFix:
|
|
"""Test for CRITICAL: Double-count bug in toin.py:354-358.
|
|
|
|
BUG: When _seen_instance_hashes hits cap (100), new instance_ids are NOT stored
|
|
but user_count IS incremented. Next call with same instance_id:
|
|
- `if self._instance_id not in pattern._seen_instance_hashes` → True (not stored!)
|
|
- user_count incremented AGAIN → Double counting!
|
|
|
|
FIX: Use a separate set to track ALL seen instances (no cap for lookup),
|
|
OR check if we already tracked overflow for this instance.
|
|
"""
|
|
|
|
def test_user_count_no_double_counting_after_cap(self):
|
|
"""Same instance shouldn't be counted twice even after cap hit."""
|
|
from headroom.telemetry.models import ToolSignature
|
|
from headroom.telemetry.toin import TOINConfig, ToolIntelligenceNetwork, reset_toin
|
|
|
|
reset_toin()
|
|
toin = ToolIntelligenceNetwork(TOINConfig())
|
|
|
|
# Create a signature using the correct factory method
|
|
items = [{"field1": "value1", "field2": 123}]
|
|
sig = ToolSignature.from_items(items)
|
|
|
|
# Simulate 101 unique instances (exceed the 100 cap)
|
|
# First, fill up the cap with 100 unique instances
|
|
original_instance_id = toin._instance_id
|
|
for i in range(100):
|
|
toin._instance_id = f"instance_{i}"
|
|
toin.record_compression(sig, 100, 10, 1000, 100, strategy="test_strategy")
|
|
|
|
# Now add one more instance (exceeds cap)
|
|
toin._instance_id = "instance_100"
|
|
toin.record_compression(sig, 100, 10, 1000, 100, strategy="test_strategy")
|
|
|
|
# Get the pattern
|
|
with toin._lock:
|
|
pattern = toin._patterns[("global", "unknown", "unknown", sig.structure_hash)]
|
|
user_count_after_101 = pattern.user_count
|
|
|
|
# Now call again with same instance (instance_100)
|
|
# BUG: This would increment user_count again because instance_100
|
|
# was not stored (cap hit) so the check passes again
|
|
toin.record_compression(sig, 100, 10, 1000, 100, strategy="test_strategy")
|
|
|
|
with toin._lock:
|
|
pattern = toin._patterns[("global", "unknown", "unknown", sig.structure_hash)]
|
|
user_count_after_102 = pattern.user_count
|
|
|
|
# Restore instance_id
|
|
toin._instance_id = original_instance_id
|
|
|
|
# User count should NOT increase for same instance
|
|
assert user_count_after_102 == user_count_after_101, (
|
|
f"Double-counting bug: user_count went from {user_count_after_101} to "
|
|
f"{user_count_after_102} for same instance after cap hit"
|
|
)
|
|
|
|
|
|
class TestCompressionFeedbackRaceCondition:
|
|
"""Test for CRITICAL: Race condition in compression_feedback.py:481-491.
|
|
|
|
BUG: _last_event_timestamp is read (line 481) and written (line 491)
|
|
WITHOUT holding the lock. Another thread calling record_retrieval()
|
|
between these could cause events to be missed or double-counted.
|
|
|
|
FIX: Move timestamp filtering and update inside the lock.
|
|
"""
|
|
|
|
def test_analyze_from_store_thread_safety(self):
|
|
"""Concurrent analyze_from_store and record_retrieval should not lose events."""
|
|
from headroom.cache.compression_feedback import (
|
|
CompressionFeedback,
|
|
reset_compression_feedback,
|
|
)
|
|
from headroom.cache.compression_store import CompressionStore, RetrievalEvent
|
|
|
|
reset_compression_feedback()
|
|
|
|
# Create store with mock events
|
|
store = CompressionStore()
|
|
feedback = CompressionFeedback(store=store, analysis_interval=0.0) # No rate limiting
|
|
|
|
# Pre-populate some events with correct API
|
|
base_time = time.time()
|
|
events_recorded = []
|
|
|
|
def add_retrieval_event(tool_name: str, timestamp: float):
|
|
event = RetrievalEvent(
|
|
hash="test_hash",
|
|
query=None,
|
|
items_retrieved=10,
|
|
total_items=100,
|
|
tool_name=tool_name,
|
|
timestamp=timestamp,
|
|
retrieval_type="full",
|
|
)
|
|
# Directly add to feedback (simulating what analyze_from_store does)
|
|
feedback.record_retrieval(event)
|
|
events_recorded.append(event)
|
|
|
|
# Record some events
|
|
for i in range(10):
|
|
add_retrieval_event(f"tool_{i % 3}", base_time + i)
|
|
|
|
with feedback._lock:
|
|
total_retrievals = feedback._total_retrievals
|
|
patterns_count = len(feedback._tool_patterns)
|
|
|
|
# All 10 events should be recorded
|
|
assert total_retrievals == 10, f"Expected 10 retrievals, got {total_retrievals}"
|
|
# Should have 3 unique tools (tool_0, tool_1, tool_2)
|
|
assert patterns_count == 3, f"Expected 3 tool patterns, got {patterns_count}"
|
|
|
|
def test_timestamp_filtering_inside_lock(self):
|
|
"""Verify that timestamp filtering happens atomically with update."""
|
|
from headroom.cache.compression_feedback import (
|
|
CompressionFeedback,
|
|
reset_compression_feedback,
|
|
)
|
|
from headroom.cache.compression_store import CompressionStore, RetrievalEvent
|
|
|
|
reset_compression_feedback()
|
|
store = CompressionStore()
|
|
feedback = CompressionFeedback(store=store, analysis_interval=0.0)
|
|
|
|
# Manually set last event timestamp
|
|
feedback._last_event_timestamp = 100.0
|
|
|
|
# Create mock store with events (correct API)
|
|
mock_events = [
|
|
RetrievalEvent(
|
|
hash="h1",
|
|
query=None,
|
|
items_retrieved=5,
|
|
total_items=50,
|
|
tool_name="tool_a",
|
|
timestamp=99.0,
|
|
retrieval_type="full",
|
|
),
|
|
RetrievalEvent(
|
|
hash="h2",
|
|
query=None,
|
|
items_retrieved=5,
|
|
total_items=50,
|
|
tool_name="tool_b",
|
|
timestamp=101.0,
|
|
retrieval_type="full",
|
|
),
|
|
RetrievalEvent(
|
|
hash="h3",
|
|
query="test",
|
|
items_retrieved=5,
|
|
total_items=50,
|
|
tool_name="tool_c",
|
|
timestamp=102.0,
|
|
retrieval_type="search",
|
|
),
|
|
]
|
|
|
|
# Mock store.get_retrieval_events
|
|
with patch.object(store, "get_retrieval_events", return_value=mock_events):
|
|
feedback.analyze_from_store()
|
|
|
|
# Only events with timestamp > 100.0 should be processed (h2, h3)
|
|
with feedback._lock:
|
|
total = feedback._total_retrievals
|
|
# The timestamp should now be 102.0 (max of processed events)
|
|
last_ts = feedback._last_event_timestamp
|
|
|
|
assert total == 2, f"Expected 2 new events processed, got {total}"
|
|
assert last_ts == 102.0, f"Expected last_event_timestamp=102.0, got {last_ts}"
|
|
|
|
|
|
class TestUnboundedStrategyDicts:
|
|
"""Test for HIGH: Unbounded strategy_compressions/strategy_retrievals dicts.
|
|
|
|
BUG: Unlike common_queries (truncated at 100) and queried_fields (truncated at 50),
|
|
the strategy dicts have no size limits and could grow unbounded.
|
|
|
|
FIX: Add truncation logic similar to other dicts.
|
|
"""
|
|
|
|
def test_strategy_dicts_have_size_limits(self):
|
|
"""Strategy dicts should be bounded to prevent memory leaks."""
|
|
from headroom.cache.compression_feedback import (
|
|
CompressionFeedback,
|
|
reset_compression_feedback,
|
|
)
|
|
from headroom.cache.compression_store import CompressionStore
|
|
|
|
reset_compression_feedback()
|
|
store = CompressionStore()
|
|
feedback = CompressionFeedback(store=store)
|
|
|
|
# Record many compressions with different strategies
|
|
for i in range(200):
|
|
feedback.record_compression(
|
|
tool_name="test_tool",
|
|
original_count=100,
|
|
compressed_count=10,
|
|
strategy=f"strategy_{i}", # 200 unique strategies
|
|
)
|
|
|
|
with feedback._lock:
|
|
pattern = feedback._tool_patterns.get("test_tool")
|
|
strategy_count = len(pattern.strategy_compressions) if pattern else 0
|
|
|
|
# Strategy dict should be bounded (e.g., to 50 like queried_fields)
|
|
assert strategy_count <= 50, (
|
|
f"strategy_compressions has {strategy_count} entries, should be <= 50"
|
|
)
|
|
|
|
|
|
class TestAllFixesIntegrated:
|
|
"""Integration tests ensuring all fixes work together."""
|