Files
headroom/tests/test_learn/test_registry.py
4e6e955cad fix(memory): log storage and SQLite failures that currently make retrieval miss silently (#4111)
## Description

Several storage paths turn a real failure into a quiet miss. They either
log nothing, or log a line that does not say which operation or file
failed. When CCR or memory retrieval misses, an operator cannot tell
corruption or a broken plugin from a genuine "not found". This PR adds
the missing log lines. Return values and control flow do not change.

Refs #4136

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/storage/__init__.py` (`create_storage`):
- A `headroom.storage_backend` plugin that raised on load was swallowed
by `except Exception: pass`. So was a scheme with no plugin. Both then
fell back to `SQLiteStorage(store_url)`, which opens a SQLite file named
after the URL.
- Both cases now log a warning that names the scheme. The URL itself is
not logged, because it can carry credentials.
- A plugin that raises is logged with the exception type only, because
its text can echo the URL. A separate DEBUG line gives
`log_safety.describe_exception` (exception types and code locations,
never the message), so no level carries the URL or its password.
  - The module gets a `logger`.
- `headroom/cache/backends/sqlite.py` (CCR store):
- `_entry_from_json` returned `None` with no log for a row that fails to
decode or rebuild. So `get()` reported a plain miss for an entry that
exists but is unreadable.
- It now logs a warning with the hash, the db path and the exception
type. `items()` decodes every row on each eviction pass, so the warning
fires once per hash.
- The once-per-hash guard is `log_safety.WarnOnce`. Within a one-hour
window it warns for at most 1,000 hashes, then logs one overflow
warning, and later rows log at DEBUG only, so repeated eviction scans
over many bad rows cannot flood the log. When the window ends, a row
that is still broken warns again, about once an hour. A hash is used up
only while WARNING is enabled.
- CCR keys come from model-issued retrieve calls, so they are logged
through `log_safety.safe_id`: a repr (newlines escaped) capped at 80
characters.
- Failed `delete`/`exists`/`clear`/`count`/`keys`/`items`/`get_stats`
all logged as `"op"`. Each now logs its real name, with the key for
`get`/`set`/`delete`/`exists`.
  - The transient-error warning now includes the db path.
- `headroom/memory/traffic_learner.py`: the two read-only SQLite reads
returned `[]` on `OperationalError`/`DatabaseError` with no log: the
hydrate path and `_load_persisted_patterns_from_sqlite`. Both now warn
with the db path. Also:
- dropped a redundant `conn.close()` in an `except` whose `finally`
already closes;
- the two best-effort `conn.close()` calls now catch `sqlite3.Error` and
log at debug, instead of `except Exception: pass`.
- `headroom/memory/backends/direct_mem0.py`:
`get_memory`/`delete_memory` turned any Qdrant/Neo4j exception into "not
found". They now warn with the memory id and the exception type first.
- The exception text is not logged at any level, because backend errors
can carry stored memory or connection details. A DEBUG line gives
`describe_exception`.
- Memory ids come from the model, so they are logged through
`log_safety.safe_id`, which escapes newlines and caps the length.
- mem0 2.x `Memory.delete` raises exactly `ValueError(f"Memory with id
{memory_id} not found")` for an unknown id. Only that message, for the
requested id, logs at DEBUG; the delete is model-driven, so it is
expected. Any other `ValueError`, such as a missing collection, still
warns.
- When a corrupt CCR store is recreated, the unreadable-row guard starts
fresh, because the new file has no rows.
- `headroom/learn/registry.py`: a built-in learn plugin that fails to
import was logged at debug, so `headroom learn --agent <name>` only said
"Unknown agent". It is now a warning with the traceback. Built-in
plugins use only the stdlib, so this never fires on a healthy install.
- `headroom/cache/compression_store.py`: the "Failed to load CCR
backend" warning now says it falls back to the in-process store, and
what that costs. This matches the SQLite branch next to it. It logs the
exception type only, because the backend's text can echo
`HEADROOM_REDIS_URL`. DEBUG gets `describe_exception`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q -p no:cacheprovider tests/test_storage_backends.py tests/test_storage tests/test_ccr_sqlite_backend.py tests/test_compression_store.py tests/test_learn/test_registry.py tests/test_memory/test_traffic_learner.py tests/test_memory/test_direct_mem0.py tests/test_private_state_files.py tests/test_adapter_hooks.py tests/test_cache/test_backends.py tests/test_learn/test_opencode_scanner.py tests/test_openai_responses_traffic_learner.py tests/test_learn/test_section_lifecycle.py tests/test_learn/test_worktree_merge.py tests/test_memory_handler_native_ops.py
================= 537 passed, 1 skipped, 5 warnings in 34.37s ==================
# The three new tests against origin/main (product code reverted):
FAILED tests/test_storage_backends.py::test_create_storage_warns_when_a_backend_plugin_fails_to_load
FAILED tests/test_ccr_sqlite_backend.py::TestSQLiteBackend::test_unreadable_row_is_logged_once_and_healthy_rows_are_not
FAILED tests/test_learn/test_registry.py::TestBuiltinLoadFailure::test_broken_builtin_plugin_is_logged_as_a_warning
$ uvx ruff@0.16.8 check <changed files>
All checks passed!
# Added in review: the hydration-failure warning; mem0 delete/get paths (only mem0's exact missing-memory message is quiet; a missing collection warns); newline and 100,000-char model ids; two scans over more bad rows than the cap; and storage-plugin, CCR-backend and mem0 failures whose records are rendered with logging.Formatter at DEBUG and WARNING and checked for credential and content canaries.
# Those tests against the previous head 6c1ad49c, in a scratch worktree:
FAILED tests/test_ccr_sqlite_backend.py::TestSQLiteBackend::test_repeated_scans_over_the_cap_stay_bounded
FAILED tests/test_storage_backends.py::test_create_storage_warns_when_a_backend_plugin_fails_to_load
FAILED tests/test_memory/test_direct_mem0.py::test_backend_failures_warn_without_exception_text[delete_memory-exc1-True]
FAILED tests/test_memory/test_direct_mem0.py::test_backend_failures_warn_without_exception_text[delete_memory-exc4-True]
FAILED tests/test_memory/test_direct_mem0.py::test_backend_failures_warn_without_exception_text[get_memory-exc5-True]
FAILED tests/test_adapter_hooks.py::TestCCREntryPointLoading::test_failing_backend_plugin_logs_no_credentials_at_any_level
$ mypy <6 changed product files> --ignore-missing-imports
Success: no issues found in 6 source files
```

## Real Behavior Proof

- Environment: macOS, Python 3.12.13, repo .venv, branch checkout at
72a166cb (base a9815bbb plus main and #4138 merged in), temp HOME /
HEADROOM_WORKSPACE_DIR / CODEX_HOME, offline.
- Exact command / steps: two runs from this checkout, listed below.
- Ran the shared smoke harness, which starts a real proxy from this
checkout against a local fake upstream and sends one Anthropic and one
OpenAI request with a compressible tool result.
- Ran a direct, unmocked script at DEBUG. A storage plugin raises an
error that quotes its `pg://user:CRED_CANARY@...` URL. Then 5 unreadable
rows are written into a real CCR SQLite store (the cap is lowered to 3
for the run), and `items()` scans it twice.
- Observed result: as listed below. The raw output is in the blocks
after this list.
- Both proxy requests return HTTP 200 with savings, and the proxy log
has no new warnings.
- The storage fallback logs the scheme and exception type at WARNING,
and only types and code locations at DEBUG. `CRED_CANARY` appears
nowhere in the output.
- The CCR store warns for 3 rows, then once for the overflow, and the
second scan (inside the same hour) adds nothing. All 5 rows are kept,
and reads miss as before.
- Not tested: the two paths listed below.
- A live Qdrant/Neo4j outage on the DirectMem0 path. Unit tests with a
raising client cover it.
  - A real third-party storage or CCR plugin.

```text
$ bash smoke.sh <worktree>
branch: cleanup/memory-cache-storage-logging @ 72a166cb
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines:
   2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you 
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.

$ HOME=$TMP/home HEADROOM_WORKSPACE_DIR=$TMP/ws CODEX_HOME=$TMP/codex python -u run.py   # logging at DEBUG
INFO headroom._ort - Pinned ORT_DYLIB_PATH to bundled ONNX Runtime: /Users/tejas/headroom/.venv/lib/python3.12/site-packages/onnxruntime/capi/libonnxruntime.1.26.0.dylib
WARNING headroom.storage - Failed to load storage backend for scheme 'pg' (<ValueError>); falling back to SQLite storage
DEBUG headroom.storage - Storage backend 'pg' load failure: ValueError at run.py:11 in factory <- headroom/storage/__init__.py:70 in create_storage
WARNING headroom.cache.backends.sqlite - CCR SQLite entry 'bad0' in $TMP/ws/ccr_store.db is unreadable (JSONDecodeError); treating as a miss
WARNING headroom.cache.backends.sqlite - CCR SQLite entry 'bad1' in $TMP/ws/ccr_store.db is unreadable (JSONDecodeError); treating as a miss
WARNING headroom.cache.backends.sqlite - CCR SQLite entry 'bad2' in $TMP/ws/ccr_store.db is unreadable (JSONDecodeError); treating as a miss
WARNING headroom.cache.backends.sqlite - More than 3 distinct unreadable CCR SQLite entries; further ones are logged at debug only
scan 1 -> []
scan 2 -> []
rows kept: 5
```

Update 2026-10-15 (head `4e78ca73`). Plugin-load warnings now name the
exception through `safe_id`, which renders `<ValueError>`, or `<object>`
for a class whose name is not a plain identifier. The learn-plugin
failures no longer put a traceback in the WARNING. This is a direct,
unmocked run at DEBUG with `HEADROOM_DEBUG_DUMP` unset. A storage plugin
and an external learn plugin raise errors quoting
`pg://user:CRED_CANARY@db`, one of them through a class named
`PRIVATE_CREDENTIAL_CANARY\nFORGED_LOG_ENTRY`. `grep -c 'CANARY|FORGED'`
on the output gives 0.

```text
WARNING headroom.storage - Failed to load storage backend for scheme 'pg' (<ValueError>); falling back to SQLite storage
DEBUG headroom.storage - Storage backend 'pg' load failure: ValueError at headroom/storage/__init__.py:70 in create_storage
WARNING headroom.storage - Failed to load storage backend for scheme 'pg' (<object>); falling back to SQLite storage
DEBUG headroom.storage - Storage backend 'pg' load failure: <exception> at headroom/storage/__init__.py:70 in create_storage
WARNING headroom.learn.registry - Failed to load external learn plugin 'broken' (<object>)
DEBUG headroom.learn.registry - External learn plugin 'broken' failure: <exception> at headroom/learn/registry.py:52 in _discover
learn plugins: ['claude', 'codex', 'gemini', 'grok', 'opencode']
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A; logging only.
- Stable/default behavior changed: No. Only log output changes. All
return values and fallbacks are as before.
- Kill switch / disable path: Standard logging level configuration for
the `headroom.*` loggers.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A.

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.

- Documentation: N/A, because no user-facing option changes.
- On the DirectMem0 path, deleting an unknown id stays quiet at the
default level. Only real backend failures warn.
- Depends on #4138; its two files show in this diff until it merges.
- The `describe_exception` calls run only when
`logger.isEnabledFor(logging.DEBUG)`, so a failure path does no
formatting work at INFO. That PR also touches
`headroom/proxy/handlers/_debug_dump.py`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-16 14:30:34 -07:00

219 lines
7.4 KiB
Python

"""Tests for the learn plugin registry."""
from __future__ import annotations
import contextlib
import importlib
import logging
from unittest.mock import MagicMock, patch
import pytest
from headroom.learn.base import LearnPlugin
from headroom.learn.registry import (
auto_detect_plugins,
available_agent_names,
get_plugin,
get_registry,
reset_registry,
)
from headroom.learn.scanner import ConversationScanner
@pytest.fixture(autouse=True)
def _clean_registry():
"""Reset registry before/after each test."""
reset_registry()
yield
reset_registry()
class TestBuiltinDiscovery:
def test_discovers_three_builtin_plugins(self):
reg = get_registry()
assert "claude" in reg
assert "codex" in reg
assert "gemini" in reg
assert len(reg) >= 3
def test_all_plugins_are_learn_plugins(self):
for name, plugin in get_registry().items():
assert isinstance(plugin, LearnPlugin), f"{name} is not a LearnPlugin"
def test_all_plugins_are_conversation_scanners(self):
"""Backwards compat: plugins must also be ConversationScanners."""
for name, plugin in get_registry().items():
assert isinstance(plugin, ConversationScanner), f"{name} is not a ConversationScanner"
def test_plugins_have_identity(self):
for name, plugin in get_registry().items():
assert plugin.name == name
assert plugin.display_name # non-empty
assert plugin.description # non-empty
class TestGetPlugin:
def test_get_existing_plugin(self):
plugin = get_plugin("claude")
assert plugin.name == "claude"
assert plugin.display_name == "Claude Code"
def test_get_unknown_raises_keyerror(self):
with pytest.raises(KeyError, match="Unknown agent.*cursor"):
get_plugin("cursor")
def test_error_message_lists_available(self):
with pytest.raises(KeyError, match="claude"):
get_plugin("nonexistent")
class TestAutoDetect:
def test_filters_to_detected_only(self):
"""Only plugins where detect() returns True are included."""
detected = auto_detect_plugins()
for plugin in detected:
assert plugin.detect()
def test_returns_empty_when_nothing_detected(self):
"""All plugins returning False → empty list."""
registry = get_registry()
patches = [patch.object(registry[name], "detect", return_value=False) for name in registry]
with contextlib.ExitStack() as stack:
for p in patches:
stack.enter_context(p)
assert auto_detect_plugins() == []
class TestAvailableNames:
def test_returns_sorted_list(self):
names = available_agent_names()
assert names == sorted(names)
assert "claude" in names
assert "codex" in names
assert "gemini" in names
class TestExternalPlugin:
def test_external_plugin_via_entry_point(self):
"""Mock an external plugin registered via entry_points."""
mock_plugin = MagicMock(spec=LearnPlugin)
mock_plugin.name = "cursor"
mock_plugin.display_name = "Cursor"
mock_plugin.description = "Cursor IDE (~/.cursor/)"
mock_ep = MagicMock()
mock_ep.load.return_value = mock_plugin
mock_ep.name = "cursor"
with patch("importlib.metadata.entry_points", return_value=[mock_ep]):
reset_registry()
reg = get_registry()
assert "cursor" in reg
assert reg["cursor"].name == "cursor"
class TestBuiltinLoadFailure:
def test_broken_builtin_plugin_is_logged_as_a_warning(self):
records: list[logging.LogRecord] = []
class _Collect(logging.Handler):
def emit(self, record: logging.LogRecord) -> None:
records.append(record)
handler = _Collect(level=logging.DEBUG)
logger = logging.getLogger("headroom.learn.registry")
logger.addHandler(handler)
real_import = importlib.import_module
def _import(name: str, *args, **kwargs): # noqa: ANN002, ANN003, ANN202
if name == "headroom.learn.plugins.grok":
raise ImportError("simulated breakage")
return real_import(name, *args, **kwargs)
try:
get_registry()
assert [r for r in records if r.levelno >= logging.WARNING] == []
reset_registry()
with patch("headroom.learn.registry.importlib.import_module", _import):
reg = get_registry()
assert "grok" not in reg
assert "claude" in reg
warnings = [r for r in records if r.levelno == logging.WARNING]
assert len(warnings) == 1
assert "grok" in warnings[0].getMessage()
finally:
logger.removeHandler(handler)
_MALFORMED_ERROR = type("PRIVATE_CREDENTIAL_CANARY\nFORGED_LOG_ENTRY", (RuntimeError,), {})
@pytest.mark.parametrize("level", [logging.INFO, logging.DEBUG])
@pytest.mark.parametrize("source", ["builtin", "external"])
def test_plugin_load_failure_logs_no_exception_text(monkeypatch, level, source) -> None:
"""A failing plugin's exception message and class name stay out of every record."""
records: list[logging.LogRecord] = []
class _Collect(logging.Handler):
def emit(self, record: logging.LogRecord) -> None:
records.append(record)
error = _MALFORMED_ERROR("cannot open postgres://user:CRED_CANARY@db/learn")
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
if source == "builtin":
real_import = importlib.import_module
def _import(name: str, *args, **kwargs): # noqa: ANN002, ANN003, ANN202
if name == "headroom.learn.plugins.grok":
raise error
return real_import(name, *args, **kwargs)
monkeypatch.setattr("headroom.learn.registry.importlib.import_module", _import)
else:
broken_ep = MagicMock()
broken_ep.name = "broken"
broken_ep.load.side_effect = error
monkeypatch.setattr("importlib.metadata.entry_points", lambda group: [broken_ep])
logger = logging.getLogger("headroom.learn.registry")
handler = _Collect(level=logging.DEBUG)
previous = logger.level
logger.addHandler(handler)
logger.setLevel(level)
try:
reset_registry()
assert "claude" in get_registry()
finally:
logger.removeHandler(handler)
logger.setLevel(previous)
reset_registry()
warnings = [r for r in records if r.levelno == logging.WARNING]
assert len(warnings) == 1
assert ("grok" if source == "builtin" else "broken") in warnings[0].getMessage()
for record in records:
rendered = logging.Formatter().format(record)
assert "CRED_CANARY" not in rendered
assert "PRIVATE_CREDENTIAL_CANARY" not in rendered
assert "FORGED_LOG_ENTRY" not in rendered
assert "\n" not in rendered
class TestResetRegistry:
def test_reset_clears_cache(self):
reg1 = get_registry()
assert reg1 is get_registry() # Same object (cached)
reset_registry()
reg2 = get_registry()
assert reg2 is not reg1 # New object (cache cleared)
class TestPluginCreateWriter:
def test_all_plugins_create_valid_writers(self):
from headroom.learn.writer import ContextWriter
for name, plugin in get_registry().items():
writer = plugin.create_writer()
assert isinstance(writer, ContextWriter), f"{name} writer is not a ContextWriter"