mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description Several storage paths turn a real failure into a quiet miss. They either log nothing, or log a line that does not say which operation or file failed. When CCR or memory retrieval misses, an operator cannot tell corruption or a broken plugin from a genuine "not found". This PR adds the missing log lines. Return values and control flow do not change. Refs #4136 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/storage/__init__.py` (`create_storage`): - A `headroom.storage_backend` plugin that raised on load was swallowed by `except Exception: pass`. So was a scheme with no plugin. Both then fell back to `SQLiteStorage(store_url)`, which opens a SQLite file named after the URL. - Both cases now log a warning that names the scheme. The URL itself is not logged, because it can carry credentials. - A plugin that raises is logged with the exception type only, because its text can echo the URL. A separate DEBUG line gives `log_safety.describe_exception` (exception types and code locations, never the message), so no level carries the URL or its password. - The module gets a `logger`. - `headroom/cache/backends/sqlite.py` (CCR store): - `_entry_from_json` returned `None` with no log for a row that fails to decode or rebuild. So `get()` reported a plain miss for an entry that exists but is unreadable. - It now logs a warning with the hash, the db path and the exception type. `items()` decodes every row on each eviction pass, so the warning fires once per hash. - The once-per-hash guard is `log_safety.WarnOnce`. Within a one-hour window it warns for at most 1,000 hashes, then logs one overflow warning, and later rows log at DEBUG only, so repeated eviction scans over many bad rows cannot flood the log. When the window ends, a row that is still broken warns again, about once an hour. A hash is used up only while WARNING is enabled. - CCR keys come from model-issued retrieve calls, so they are logged through `log_safety.safe_id`: a repr (newlines escaped) capped at 80 characters. - Failed `delete`/`exists`/`clear`/`count`/`keys`/`items`/`get_stats` all logged as `"op"`. Each now logs its real name, with the key for `get`/`set`/`delete`/`exists`. - The transient-error warning now includes the db path. - `headroom/memory/traffic_learner.py`: the two read-only SQLite reads returned `[]` on `OperationalError`/`DatabaseError` with no log: the hydrate path and `_load_persisted_patterns_from_sqlite`. Both now warn with the db path. Also: - dropped a redundant `conn.close()` in an `except` whose `finally` already closes; - the two best-effort `conn.close()` calls now catch `sqlite3.Error` and log at debug, instead of `except Exception: pass`. - `headroom/memory/backends/direct_mem0.py`: `get_memory`/`delete_memory` turned any Qdrant/Neo4j exception into "not found". They now warn with the memory id and the exception type first. - The exception text is not logged at any level, because backend errors can carry stored memory or connection details. A DEBUG line gives `describe_exception`. - Memory ids come from the model, so they are logged through `log_safety.safe_id`, which escapes newlines and caps the length. - mem0 2.x `Memory.delete` raises exactly `ValueError(f"Memory with id {memory_id} not found")` for an unknown id. Only that message, for the requested id, logs at DEBUG; the delete is model-driven, so it is expected. Any other `ValueError`, such as a missing collection, still warns. - When a corrupt CCR store is recreated, the unreadable-row guard starts fresh, because the new file has no rows. - `headroom/learn/registry.py`: a built-in learn plugin that fails to import was logged at debug, so `headroom learn --agent <name>` only said "Unknown agent". It is now a warning with the traceback. Built-in plugins use only the stdlib, so this never fires on a healthy install. - `headroom/cache/compression_store.py`: the "Failed to load CCR backend" warning now says it falls back to the in-process store, and what that costs. This matches the SQLite branch next to it. It logs the exception type only, because the backend's text can echo `HEADROOM_REDIS_URL`. DEBUG gets `describe_exception`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest -q -p no:cacheprovider tests/test_storage_backends.py tests/test_storage tests/test_ccr_sqlite_backend.py tests/test_compression_store.py tests/test_learn/test_registry.py tests/test_memory/test_traffic_learner.py tests/test_memory/test_direct_mem0.py tests/test_private_state_files.py tests/test_adapter_hooks.py tests/test_cache/test_backends.py tests/test_learn/test_opencode_scanner.py tests/test_openai_responses_traffic_learner.py tests/test_learn/test_section_lifecycle.py tests/test_learn/test_worktree_merge.py tests/test_memory_handler_native_ops.py ================= 537 passed, 1 skipped, 5 warnings in 34.37s ================== # The three new tests against origin/main (product code reverted): FAILED tests/test_storage_backends.py::test_create_storage_warns_when_a_backend_plugin_fails_to_load FAILED tests/test_ccr_sqlite_backend.py::TestSQLiteBackend::test_unreadable_row_is_logged_once_and_healthy_rows_are_not FAILED tests/test_learn/test_registry.py::TestBuiltinLoadFailure::test_broken_builtin_plugin_is_logged_as_a_warning $ uvx ruff@0.16.8 check <changed files> All checks passed! # Added in review: the hydration-failure warning; mem0 delete/get paths (only mem0's exact missing-memory message is quiet; a missing collection warns); newline and 100,000-char model ids; two scans over more bad rows than the cap; and storage-plugin, CCR-backend and mem0 failures whose records are rendered with logging.Formatter at DEBUG and WARNING and checked for credential and content canaries. # Those tests against the previous head6c1ad49c, in a scratch worktree: FAILED tests/test_ccr_sqlite_backend.py::TestSQLiteBackend::test_repeated_scans_over_the_cap_stay_bounded FAILED tests/test_storage_backends.py::test_create_storage_warns_when_a_backend_plugin_fails_to_load FAILED tests/test_memory/test_direct_mem0.py::test_backend_failures_warn_without_exception_text[delete_memory-exc1-True] FAILED tests/test_memory/test_direct_mem0.py::test_backend_failures_warn_without_exception_text[delete_memory-exc4-True] FAILED tests/test_memory/test_direct_mem0.py::test_backend_failures_warn_without_exception_text[get_memory-exc5-True] FAILED tests/test_adapter_hooks.py::TestCCREntryPointLoading::test_failing_backend_plugin_logs_no_credentials_at_any_level $ mypy <6 changed product files> --ignore-missing-imports Success: no issues found in 6 source files ``` ## Real Behavior Proof - Environment: macOS, Python 3.12.13, repo .venv, branch checkout at72a166cb(basea9815bbbplus main and #4138 merged in), temp HOME / HEADROOM_WORKSPACE_DIR / CODEX_HOME, offline. - Exact command / steps: two runs from this checkout, listed below. - Ran the shared smoke harness, which starts a real proxy from this checkout against a local fake upstream and sends one Anthropic and one OpenAI request with a compressible tool result. - Ran a direct, unmocked script at DEBUG. A storage plugin raises an error that quotes its `pg://user:CRED_CANARY@...` URL. Then 5 unreadable rows are written into a real CCR SQLite store (the cap is lowered to 3 for the run), and `items()` scans it twice. - Observed result: as listed below. The raw output is in the blocks after this list. - Both proxy requests return HTTP 200 with savings, and the proxy log has no new warnings. - The storage fallback logs the scheme and exception type at WARNING, and only types and code locations at DEBUG. `CRED_CANARY` appears nowhere in the output. - The CCR store warns for 3 rows, then once for the overflow, and the second scan (inside the same hour) adds nothing. All 5 rows are kept, and reads miss as before. - Not tested: the two paths listed below. - A live Qdrant/Neo4j outage on the DirectMem0 path. Unit tests with a raising client cover it. - A real third-party storage or CCR plugin. ```text $ bash smoke.sh <worktree> branch: cleanup/memory-cache-storage-logging @72a166cbhealth: 200 POST /v1/messages -> HTTP200 smoke-ok POST /v1/chat/completions -> HTTP200 smoke-ok stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82 proxy.log WARNING/ERROR lines: 2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you 4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base 4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection. $ HOME=$TMP/home HEADROOM_WORKSPACE_DIR=$TMP/ws CODEX_HOME=$TMP/codex python -u run.py # logging at DEBUG INFO headroom._ort - Pinned ORT_DYLIB_PATH to bundled ONNX Runtime: /Users/tejas/headroom/.venv/lib/python3.12/site-packages/onnxruntime/capi/libonnxruntime.1.26.0.dylib WARNING headroom.storage - Failed to load storage backend for scheme 'pg' (<ValueError>); falling back to SQLite storage DEBUG headroom.storage - Storage backend 'pg' load failure: ValueError at run.py:11 in factory <- headroom/storage/__init__.py:70 in create_storage WARNING headroom.cache.backends.sqlite - CCR SQLite entry 'bad0' in $TMP/ws/ccr_store.db is unreadable (JSONDecodeError); treating as a miss WARNING headroom.cache.backends.sqlite - CCR SQLite entry 'bad1' in $TMP/ws/ccr_store.db is unreadable (JSONDecodeError); treating as a miss WARNING headroom.cache.backends.sqlite - CCR SQLite entry 'bad2' in $TMP/ws/ccr_store.db is unreadable (JSONDecodeError); treating as a miss WARNING headroom.cache.backends.sqlite - More than 3 distinct unreadable CCR SQLite entries; further ones are logged at debug only scan 1 -> [] scan 2 -> [] rows kept: 5 ``` Update 2026-10-15 (head `4e78ca73`). Plugin-load warnings now name the exception through `safe_id`, which renders `<ValueError>`, or `<object>` for a class whose name is not a plain identifier. The learn-plugin failures no longer put a traceback in the WARNING. This is a direct, unmocked run at DEBUG with `HEADROOM_DEBUG_DUMP` unset. A storage plugin and an external learn plugin raise errors quoting `pg://user:CRED_CANARY@db`, one of them through a class named `PRIVATE_CREDENTIAL_CANARY\nFORGED_LOG_ENTRY`. `grep -c 'CANARY|FORGED'` on the output gives 0. ```text WARNING headroom.storage - Failed to load storage backend for scheme 'pg' (<ValueError>); falling back to SQLite storage DEBUG headroom.storage - Storage backend 'pg' load failure: ValueError at headroom/storage/__init__.py:70 in create_storage WARNING headroom.storage - Failed to load storage backend for scheme 'pg' (<object>); falling back to SQLite storage DEBUG headroom.storage - Storage backend 'pg' load failure: <exception> at headroom/storage/__init__.py:70 in create_storage WARNING headroom.learn.registry - Failed to load external learn plugin 'broken' (<object>) DEBUG headroom.learn.registry - External learn plugin 'broken' failure: <exception> at headroom/learn/registry.py:52 in _discover learn plugins: ['claude', 'codex', 'gemini', 'grok', 'opencode'] ``` ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: N/A; logging only. - Stable/default behavior changed: No. Only log output changes. All return values and fallbacks are as before. - Kill switch / disable path: Standard logging level configuration for the `headroom.*` loggers. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A. ## Additional Notes Part of maintainer-requested cleanup #4136 (dead code, duplication, debuggability), split into small PRs for review. - Documentation: N/A, because no user-facing option changes. - On the DirectMem0 path, deleting an unknown id stays quiet at the default level. Only real backend failures warn. - Depends on #4138; its two files show in this diff until it merges. - The `describe_exception` calls run only when `logger.isEnabledFor(logging.DEBUG)`, so a failure path does no formatting work at INFO. That PR also touches `headroom/proxy/handlers/_debug_dump.py`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
647 lines
23 KiB
Python
647 lines
23 KiB
Python
"""Tests for pluggable adapter hooks.
|
|
|
|
Validates the extension points that allow SaaS packages to inject
|
|
custom backends for CCR, Storage, and TOIN without forking OSS.
|
|
|
|
Tests cover:
|
|
1. TOIN backend protocol conformance (FileSystemTOINBackend)
|
|
2. TOIN backend wiring (ToolIntelligenceNetwork with custom backend)
|
|
3. TOIN entry_point loading (_create_default_toin_backend)
|
|
4. CCR ContextVar scoping (set/clear/get request compression store)
|
|
5. CCR entry_point loading (_create_default_ccr_backend)
|
|
6. Storage entry_point loading (create_storage with custom scheme)
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import logging
|
|
import threading
|
|
from typing import Any
|
|
|
|
import pytest
|
|
|
|
from headroom.cache.backends import CompressionStoreBackend, InMemoryBackend
|
|
from headroom.cache.compression_store import (
|
|
CompressionStore,
|
|
clear_request_compression_store,
|
|
get_compression_store,
|
|
reset_compression_store,
|
|
set_request_compression_store,
|
|
)
|
|
from headroom.storage import Storage, create_storage
|
|
from headroom.telemetry.backends import FileSystemTOINBackend, TOINBackend
|
|
from headroom.telemetry.models import ToolSignature
|
|
from headroom.telemetry.toin import (
|
|
TOINConfig,
|
|
ToolIntelligenceNetwork,
|
|
_create_default_toin_backend,
|
|
get_toin,
|
|
reset_toin,
|
|
)
|
|
|
|
# =============================================================================
|
|
# Fixtures
|
|
# =============================================================================
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _clean_globals():
|
|
"""Reset global singletons before and after each test."""
|
|
reset_toin()
|
|
reset_compression_store()
|
|
clear_request_compression_store()
|
|
yield
|
|
reset_toin()
|
|
reset_compression_store()
|
|
clear_request_compression_store()
|
|
|
|
|
|
@pytest.fixture
|
|
def tmp_toin_path(tmp_path):
|
|
"""Temporary file path for TOIN storage."""
|
|
return str(tmp_path / "toin.json")
|
|
|
|
|
|
@pytest.fixture
|
|
def fs_backend(tmp_toin_path):
|
|
"""FileSystemTOINBackend with a temp path."""
|
|
return FileSystemTOINBackend(tmp_toin_path)
|
|
|
|
|
|
def _make_tool_signature() -> ToolSignature:
|
|
"""Create a ToolSignature for testing."""
|
|
return ToolSignature.from_items(
|
|
[
|
|
{"id": 1, "name": "test", "score": 0.95},
|
|
{"id": 2, "name": "test2", "score": 0.80},
|
|
]
|
|
)
|
|
|
|
|
|
# =============================================================================
|
|
# 1. TOIN Backend Protocol Conformance
|
|
# =============================================================================
|
|
|
|
|
|
class TestTOINBackendProtocol:
|
|
"""Verify FileSystemTOINBackend satisfies the TOINBackend protocol."""
|
|
|
|
def test_filesystem_backend_is_toin_backend(self, fs_backend):
|
|
"""FileSystemTOINBackend must satisfy the runtime-checkable TOINBackend protocol."""
|
|
assert isinstance(fs_backend, TOINBackend)
|
|
|
|
def test_load_empty(self, fs_backend):
|
|
"""load() returns empty dict when no file exists."""
|
|
result = fs_backend.load()
|
|
assert result == {}
|
|
|
|
def test_save_and_load_roundtrip(self, fs_backend):
|
|
"""Data survives a save/load cycle."""
|
|
data = {
|
|
"version": "1.0",
|
|
"patterns": {
|
|
"abc123": {
|
|
"tool_signature_hash": "abc123",
|
|
"total_compressions": 42,
|
|
}
|
|
},
|
|
}
|
|
fs_backend.save(data)
|
|
loaded = fs_backend.load()
|
|
assert loaded == data
|
|
|
|
def test_save_creates_parent_dirs(self, tmp_path):
|
|
"""save() creates parent directories if they don't exist."""
|
|
deep_path = str(tmp_path / "a" / "b" / "c" / "toin.json")
|
|
backend = FileSystemTOINBackend(deep_path)
|
|
backend.save({"version": "1.0"})
|
|
loaded = backend.load()
|
|
assert loaded["version"] == "1.0"
|
|
|
|
def test_save_atomic_on_failure(self, tmp_path):
|
|
"""If save fails mid-write, original data is preserved."""
|
|
path = str(tmp_path / "toin.json")
|
|
backend = FileSystemTOINBackend(path)
|
|
|
|
# Save initial data
|
|
backend.save({"version": "1.0", "state": "original"})
|
|
|
|
# Corrupt the temp dir to force failure (make dir read-only)
|
|
# This is OS-dependent; we verify the load still returns original
|
|
loaded = backend.load()
|
|
assert loaded["state"] == "original"
|
|
|
|
def test_load_corrupted_file(self, tmp_toin_path):
|
|
"""load() returns empty dict on corrupted JSON."""
|
|
with open(tmp_toin_path, "w") as f:
|
|
f.write("{invalid json!!")
|
|
|
|
backend = FileSystemTOINBackend(tmp_toin_path)
|
|
result = backend.load()
|
|
assert result == {}
|
|
|
|
def test_save_overwrites(self, fs_backend):
|
|
"""save() overwrites previous data completely."""
|
|
fs_backend.save({"version": "1.0", "old": True})
|
|
fs_backend.save({"version": "2.0", "new": True})
|
|
loaded = fs_backend.load()
|
|
assert loaded == {"version": "2.0", "new": True}
|
|
assert "old" not in loaded
|
|
|
|
|
|
class TestCustomTOINBackend:
|
|
"""Verify any dict-based backend satisfies the protocol."""
|
|
|
|
def test_dict_backend_satisfies_protocol(self):
|
|
"""A minimal dict-backed implementation passes protocol check."""
|
|
|
|
class DictBackend:
|
|
def __init__(self):
|
|
self._data: dict[str, Any] = {}
|
|
|
|
def load(self) -> dict[str, Any]:
|
|
return dict(self._data)
|
|
|
|
def save(self, data: dict[str, Any]) -> None:
|
|
self._data = dict(data)
|
|
|
|
backend = DictBackend()
|
|
assert isinstance(backend, TOINBackend)
|
|
|
|
backend.save({"key": "value"})
|
|
assert backend.load() == {"key": "value"}
|
|
|
|
|
|
# =============================================================================
|
|
# 2. TOIN Backend Wiring
|
|
# =============================================================================
|
|
|
|
|
|
class TestTOINBackendWiring:
|
|
"""Verify ToolIntelligenceNetwork correctly delegates to backends."""
|
|
|
|
def test_toin_with_custom_backend(self):
|
|
"""TOIN uses custom backend for save/load."""
|
|
store: dict[str, Any] = {}
|
|
|
|
class MemBackend:
|
|
def load(self) -> dict[str, Any]:
|
|
return dict(store)
|
|
|
|
def save(self, data: dict[str, Any]) -> None:
|
|
store.clear()
|
|
store.update(data)
|
|
|
|
config = TOINConfig(enabled=True, storage_path="", auto_save_interval=0)
|
|
toin = ToolIntelligenceNetwork(config, backend=MemBackend())
|
|
|
|
sig = _make_tool_signature()
|
|
toin.record_compression(
|
|
tool_signature=sig,
|
|
original_count=10,
|
|
compressed_count=3,
|
|
original_tokens=500,
|
|
compressed_tokens=150,
|
|
strategy="smart_crusher",
|
|
)
|
|
|
|
# Save should go to our backend
|
|
toin.save()
|
|
assert "patterns" in store
|
|
assert len(store["patterns"]) == 1
|
|
|
|
# Create a new TOIN instance with same backend — should load patterns
|
|
toin2 = ToolIntelligenceNetwork(config, backend=MemBackend())
|
|
stats = toin2.get_stats()
|
|
assert stats["patterns_tracked"] == 1
|
|
assert stats["total_compressions"] == 1
|
|
|
|
def test_toin_with_none_backend_and_no_path(self):
|
|
"""TOIN works without persistence when no backend and no path."""
|
|
config = TOINConfig(enabled=True, storage_path="")
|
|
toin = ToolIntelligenceNetwork(config, backend=None)
|
|
|
|
# save() should be a no-op
|
|
toin.save()
|
|
stats = toin.get_stats()
|
|
assert stats["patterns_tracked"] == 0
|
|
|
|
def test_toin_default_filesystem_backend(self, tmp_toin_path):
|
|
"""TOIN creates FileSystemTOINBackend when storage_path is set and no backend given."""
|
|
config = TOINConfig(enabled=True, storage_path=tmp_toin_path)
|
|
toin = ToolIntelligenceNetwork(config)
|
|
|
|
sig = _make_tool_signature()
|
|
toin.record_compression(
|
|
tool_signature=sig,
|
|
original_count=5,
|
|
compressed_count=2,
|
|
original_tokens=100,
|
|
compressed_tokens=40,
|
|
strategy="default",
|
|
)
|
|
toin.save()
|
|
|
|
# Verify file was written
|
|
with open(tmp_toin_path) as f:
|
|
data = json.load(f)
|
|
assert "patterns" in data
|
|
assert len(data["patterns"]) == 1
|
|
|
|
|
|
# =============================================================================
|
|
# 3. TOIN Entry Point Loading
|
|
# =============================================================================
|
|
|
|
|
|
class TestTOINEntryPointLoading:
|
|
"""Verify _create_default_toin_backend() env-based loading."""
|
|
|
|
def test_no_env_returns_none(self, monkeypatch):
|
|
"""No HEADROOM_TOIN_BACKEND env → returns None (use default)."""
|
|
monkeypatch.delenv("HEADROOM_TOIN_BACKEND", raising=False)
|
|
assert _create_default_toin_backend() is None
|
|
|
|
def test_empty_env_returns_none(self, monkeypatch):
|
|
"""Empty HEADROOM_TOIN_BACKEND → returns None."""
|
|
monkeypatch.setenv("HEADROOM_TOIN_BACKEND", "")
|
|
assert _create_default_toin_backend() is None
|
|
|
|
def test_filesystem_env_returns_none(self, monkeypatch):
|
|
"""HEADROOM_TOIN_BACKEND=filesystem → returns None (use default)."""
|
|
monkeypatch.setenv("HEADROOM_TOIN_BACKEND", "filesystem")
|
|
assert _create_default_toin_backend() is None
|
|
|
|
def test_unknown_backend_returns_none(self, monkeypatch):
|
|
"""Unknown backend name with no entry point → returns None with warning."""
|
|
monkeypatch.setenv("HEADROOM_TOIN_BACKEND", "nonexistent_backend_xyz")
|
|
result = _create_default_toin_backend()
|
|
assert result is None
|
|
|
|
def test_get_toin_respects_env_backend(self, monkeypatch, tmp_toin_path):
|
|
"""get_toin() uses _create_default_toin_backend() on first call."""
|
|
monkeypatch.delenv("HEADROOM_TOIN_BACKEND", raising=False)
|
|
monkeypatch.setenv("HEADROOM_TOIN_PATH", tmp_toin_path)
|
|
|
|
toin = get_toin()
|
|
assert toin is not None
|
|
# Should use FileSystemTOINBackend since no HEADROOM_TOIN_BACKEND set
|
|
assert toin._backend is not None
|
|
|
|
|
|
# =============================================================================
|
|
# 4. CCR ContextVar Scoping
|
|
# =============================================================================
|
|
|
|
|
|
class TestCCRContextVarScoping:
|
|
"""Verify request-scoped CCR stores work correctly."""
|
|
|
|
def test_default_returns_global(self):
|
|
"""Without request scope, get_compression_store() returns global singleton."""
|
|
store1 = get_compression_store()
|
|
store2 = get_compression_store()
|
|
assert store1 is store2
|
|
|
|
def test_set_request_store_overrides_global(self):
|
|
"""set_request_compression_store() makes get return the request store."""
|
|
global_store = get_compression_store()
|
|
request_store = CompressionStore(max_entries=10)
|
|
|
|
set_request_compression_store(request_store)
|
|
assert get_compression_store() is request_store
|
|
assert get_compression_store() is not global_store
|
|
|
|
def test_clear_request_store_restores_global(self):
|
|
"""clear_request_compression_store() restores the global store."""
|
|
global_store = get_compression_store()
|
|
request_store = CompressionStore(max_entries=10)
|
|
|
|
set_request_compression_store(request_store)
|
|
assert get_compression_store() is request_store
|
|
|
|
clear_request_compression_store()
|
|
assert get_compression_store() is global_store
|
|
|
|
def test_request_store_isolated_per_thread(self):
|
|
"""ContextVars are per-thread — each thread sees its own store."""
|
|
global_store = get_compression_store()
|
|
results: dict[str, CompressionStore | None] = {}
|
|
|
|
def worker(name: str, store: CompressionStore | None):
|
|
if store:
|
|
set_request_compression_store(store)
|
|
results[name] = get_compression_store()
|
|
if store:
|
|
clear_request_compression_store()
|
|
|
|
store_a = CompressionStore(max_entries=5)
|
|
store_b = CompressionStore(max_entries=7)
|
|
|
|
t1 = threading.Thread(target=worker, args=("t1", store_a))
|
|
t2 = threading.Thread(target=worker, args=("t2", store_b))
|
|
t3 = threading.Thread(target=worker, args=("t3", None))
|
|
|
|
t1.start()
|
|
t2.start()
|
|
t3.start()
|
|
t1.join()
|
|
t2.join()
|
|
t3.join()
|
|
|
|
assert results["t1"] is store_a
|
|
assert results["t2"] is store_b
|
|
assert results["t3"] is global_store
|
|
|
|
def test_request_store_data_isolation(self):
|
|
"""Data stored in request-scoped store doesn't leak to global."""
|
|
global_store = get_compression_store()
|
|
|
|
request_store = CompressionStore(max_entries=10)
|
|
set_request_compression_store(request_store)
|
|
|
|
# Store data in request store
|
|
active_store = get_compression_store()
|
|
hash_key = active_store.store(
|
|
original='[{"id": 1}]',
|
|
compressed='[{"id": 1}]',
|
|
original_tokens=10,
|
|
compressed_tokens=10,
|
|
)
|
|
|
|
# Verify it's in request store
|
|
assert active_store.retrieve(hash_key) is not None
|
|
# Verify it's NOT in global store
|
|
assert global_store.retrieve(hash_key) is None
|
|
|
|
clear_request_compression_store()
|
|
|
|
|
|
# =============================================================================
|
|
# 5. CCR Entry Point Loading
|
|
# =============================================================================
|
|
|
|
|
|
class TestCCREntryPointLoading:
|
|
"""Verify _create_default_ccr_backend() env-based loading."""
|
|
|
|
def test_no_env_returns_sqlite(self, monkeypatch, tmp_path):
|
|
"""No HEADROOM_CCR_BACKEND → SQLiteBackend (the persistent default;
|
|
restart survival + cross-worker sharing for the 30-min TTL)."""
|
|
monkeypatch.delenv("HEADROOM_CCR_BACKEND", raising=False)
|
|
monkeypatch.setenv("HEADROOM_CCR_SQLITE_PATH", str(tmp_path / "ccr.db"))
|
|
from headroom.cache.compression_store import _create_default_ccr_backend
|
|
|
|
backend = _create_default_ccr_backend()
|
|
assert backend is not None
|
|
assert backend.get_stats()["backend_type"] == "sqlite"
|
|
|
|
def test_memory_env_returns_none(self, monkeypatch):
|
|
"""HEADROOM_CCR_BACKEND=memory → returns None (use default)."""
|
|
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "memory")
|
|
from headroom.cache.compression_store import _create_default_ccr_backend
|
|
|
|
assert _create_default_ccr_backend() is None
|
|
|
|
def test_unknown_backend_returns_none(self, monkeypatch):
|
|
"""Unknown backend with no entry point → returns None."""
|
|
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "nonexistent_backend_xyz")
|
|
from headroom.cache.compression_store import _create_default_ccr_backend
|
|
|
|
assert _create_default_ccr_backend() is None
|
|
|
|
def test_failing_backend_plugin_logs_no_credentials_at_any_level(self, monkeypatch):
|
|
"""A plugin error that quotes HEADROOM_REDIS_URL must not reach the log, at
|
|
WARNING or DEBUG, rendered the way a handler would (exc_info included)."""
|
|
import logging
|
|
from types import SimpleNamespace
|
|
|
|
from headroom.cache.compression_store import _create_default_ccr_backend
|
|
|
|
def _factory(url: str, tenant_prefix: str) -> None:
|
|
raise ConnectionError(f"cannot reach {url} holding CONTENT_CANARY")
|
|
|
|
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "redisx")
|
|
monkeypatch.setenv("HEADROOM_REDIS_URL", "redis://user:CRED_CANARY@cache:6379/0")
|
|
monkeypatch.setattr(
|
|
"importlib.metadata.entry_points",
|
|
lambda group: [SimpleNamespace(name="redisx", load=lambda: _factory)],
|
|
)
|
|
records: list[logging.LogRecord] = []
|
|
|
|
class _Collect(logging.Handler):
|
|
def emit(self, record: logging.LogRecord) -> None:
|
|
records.append(record)
|
|
|
|
logger = logging.getLogger("headroom.cache.compression_store")
|
|
handler = _Collect(level=logging.DEBUG)
|
|
previous = logger.level
|
|
logger.addHandler(handler)
|
|
logger.setLevel(logging.DEBUG)
|
|
try:
|
|
assert _create_default_ccr_backend() is None
|
|
finally:
|
|
logger.removeHandler(handler)
|
|
logger.setLevel(previous)
|
|
|
|
assert {r.levelno for r in records} >= {logging.WARNING, logging.DEBUG}
|
|
for record in records:
|
|
rendered = logging.Formatter().format(record)
|
|
assert "CRED_CANARY" not in rendered
|
|
assert "CONTENT_CANARY" not in rendered
|
|
assert any("ConnectionError" in r.getMessage() for r in records)
|
|
|
|
@pytest.mark.parametrize("level", [logging.INFO, logging.DEBUG])
|
|
def test_backend_plugin_class_name_cannot_leak_or_forge_log_lines(self, monkeypatch, level):
|
|
"""A plugin chooses its exception's class name; the WARNING must not carry it."""
|
|
from types import SimpleNamespace
|
|
|
|
from headroom.cache.compression_store import _create_default_ccr_backend
|
|
|
|
error_type = type("PRIVATE_CREDENTIAL_CANARY\nFORGED_LOG_ENTRY", (RuntimeError,), {})
|
|
|
|
def _factory(url: str, tenant_prefix: str) -> None:
|
|
raise error_type("plugin text")
|
|
|
|
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
|
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "redisx")
|
|
monkeypatch.setattr(
|
|
"importlib.metadata.entry_points",
|
|
lambda group: [SimpleNamespace(name="redisx", load=lambda: _factory)],
|
|
)
|
|
records: list[logging.LogRecord] = []
|
|
|
|
class _Collect(logging.Handler):
|
|
def emit(self, record: logging.LogRecord) -> None:
|
|
records.append(record)
|
|
|
|
logger = logging.getLogger("headroom.cache.compression_store")
|
|
handler = _Collect(level=logging.DEBUG)
|
|
previous = logger.level
|
|
logger.addHandler(handler)
|
|
logger.setLevel(level)
|
|
try:
|
|
assert _create_default_ccr_backend() is None
|
|
finally:
|
|
logger.removeHandler(handler)
|
|
logger.setLevel(previous)
|
|
|
|
assert len([r for r in records if r.levelno == logging.WARNING]) == 1
|
|
for record in records:
|
|
rendered = logging.Formatter().format(record)
|
|
assert "PRIVATE_CREDENTIAL_CANARY" not in rendered
|
|
assert "FORGED_LOG_ENTRY" not in rendered
|
|
assert "\n" not in rendered
|
|
|
|
def test_inmemory_backend_satisfies_protocol(self):
|
|
"""InMemoryBackend satisfies the CompressionStoreBackend protocol."""
|
|
backend = InMemoryBackend()
|
|
assert isinstance(backend, CompressionStoreBackend)
|
|
|
|
|
|
# =============================================================================
|
|
# 6. Storage Entry Point Loading
|
|
# =============================================================================
|
|
|
|
|
|
class TestStorageEntryPointLoading:
|
|
"""Verify create_storage() scheme-based loading."""
|
|
|
|
def test_sqlite_scheme(self, tmp_path):
|
|
"""sqlite:// scheme creates SQLiteStorage."""
|
|
from headroom.storage.sqlite import SQLiteStorage
|
|
|
|
store = create_storage(f"sqlite:///{tmp_path}/test.db")
|
|
assert isinstance(store, SQLiteStorage)
|
|
|
|
def test_jsonl_scheme(self, tmp_path):
|
|
"""jsonl:// scheme creates JSONLStorage."""
|
|
from headroom.storage.jsonl import JSONLStorage
|
|
|
|
store = create_storage(f"jsonl:///{tmp_path}/test.jsonl")
|
|
assert isinstance(store, JSONLStorage)
|
|
|
|
def test_unknown_scheme_without_entry_point(self, tmp_path):
|
|
"""Unknown scheme without entry point falls back to SQLiteStorage."""
|
|
from headroom.storage.sqlite import SQLiteStorage
|
|
|
|
# This should fall back to SQLite (legacy behavior)
|
|
store = create_storage(str(tmp_path / "test.db"))
|
|
assert isinstance(store, SQLiteStorage)
|
|
|
|
def test_storage_base_is_abstract(self):
|
|
"""Storage ABC requires all methods to be implemented."""
|
|
assert hasattr(Storage, "save")
|
|
assert hasattr(Storage, "get")
|
|
assert hasattr(Storage, "query")
|
|
assert hasattr(Storage, "count")
|
|
assert hasattr(Storage, "iter_all")
|
|
assert hasattr(Storage, "get_summary_stats")
|
|
|
|
|
|
# =============================================================================
|
|
# 7. Integration: Full Adapter Lifecycle
|
|
# =============================================================================
|
|
|
|
|
|
class TestAdapterLifecycle:
|
|
"""End-to-end test of the adapter pattern."""
|
|
|
|
def test_ccr_with_custom_backend(self):
|
|
"""CompressionStore works with a custom backend implementation."""
|
|
|
|
class ListBackend:
|
|
"""Minimal backend that tracks all operations."""
|
|
|
|
def __init__(self):
|
|
self._store: dict[str, Any] = {}
|
|
self.ops: list[str] = []
|
|
|
|
def get(self, hash_key):
|
|
self.ops.append(f"get:{hash_key[:8]}")
|
|
return self._store.get(hash_key)
|
|
|
|
def set(self, hash_key, entry):
|
|
self.ops.append(f"set:{hash_key[:8]}")
|
|
self._store[hash_key] = entry
|
|
|
|
def delete(self, hash_key):
|
|
self.ops.append(f"delete:{hash_key[:8]}")
|
|
if hash_key in self._store:
|
|
del self._store[hash_key]
|
|
return True
|
|
return False
|
|
|
|
def exists(self, hash_key):
|
|
return hash_key in self._store
|
|
|
|
def clear(self):
|
|
self._store.clear()
|
|
|
|
def count(self):
|
|
return len(self._store)
|
|
|
|
def keys(self):
|
|
return list(self._store.keys())
|
|
|
|
def items(self):
|
|
return list(self._store.items())
|
|
|
|
def get_stats(self):
|
|
return {"backend_type": "list", "entry_count": len(self._store)}
|
|
|
|
backend = ListBackend()
|
|
store = CompressionStore(backend=backend)
|
|
|
|
# Store and retrieve
|
|
hash_key = store.store(
|
|
original='[{"id": 1, "name": "test"}]',
|
|
compressed='[{"id": 1}]',
|
|
original_tokens=50,
|
|
compressed_tokens=20,
|
|
)
|
|
entry = store.retrieve(hash_key)
|
|
|
|
assert entry is not None
|
|
assert entry.original_tokens == 50
|
|
assert any("set:" in op for op in backend.ops)
|
|
assert any("get:" in op for op in backend.ops)
|
|
|
|
def test_toin_save_load_preserves_patterns(self, tmp_toin_path):
|
|
"""Patterns survive save/load via backend.
|
|
|
|
PR-B5 retired the request-time `get_recommendation()` API
|
|
(it now returns None with a deprecation warning). Stats and
|
|
on-disk patterns must still survive save/load — that's the
|
|
observation API B5 preserves.
|
|
"""
|
|
config = TOINConfig(storage_path=tmp_toin_path)
|
|
toin = ToolIntelligenceNetwork(config)
|
|
|
|
sig = _make_tool_signature()
|
|
|
|
# Record multiple events
|
|
for _i in range(15):
|
|
toin.record_compression(
|
|
tool_signature=sig,
|
|
original_count=50,
|
|
compressed_count=10,
|
|
original_tokens=1000,
|
|
compressed_tokens=200,
|
|
strategy="smart_crusher",
|
|
)
|
|
|
|
toin.save()
|
|
|
|
# New instance loads from same backend
|
|
toin2 = ToolIntelligenceNetwork(TOINConfig(storage_path=tmp_toin_path))
|
|
stats = toin2.get_stats()
|
|
assert stats["patterns_tracked"] >= 1
|
|
assert stats["total_compressions"] >= 15
|
|
|
|
# PR-B5: get_recommendation is observation-only and returns None.
|
|
# Recommendations now flow through the publish CLI →
|
|
# recommendations.toml → Rust loader path.
|
|
assert toin2.get_recommendation(sig) is None
|