mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(memory): log storage and SQLite failures that currently make retrieval miss silently (#4111)
## Description Several storage paths turn a real failure into a quiet miss. They either log nothing, or log a line that does not say which operation or file failed. When CCR or memory retrieval misses, an operator cannot tell corruption or a broken plugin from a genuine "not found". This PR adds the missing log lines. Return values and control flow do not change. Refs #4136 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/storage/__init__.py` (`create_storage`): - A `headroom.storage_backend` plugin that raised on load was swallowed by `except Exception: pass`. So was a scheme with no plugin. Both then fell back to `SQLiteStorage(store_url)`, which opens a SQLite file named after the URL. - Both cases now log a warning that names the scheme. The URL itself is not logged, because it can carry credentials. - A plugin that raises is logged with the exception type only, because its text can echo the URL. A separate DEBUG line gives `log_safety.describe_exception` (exception types and code locations, never the message), so no level carries the URL or its password. - The module gets a `logger`. - `headroom/cache/backends/sqlite.py` (CCR store): - `_entry_from_json` returned `None` with no log for a row that fails to decode or rebuild. So `get()` reported a plain miss for an entry that exists but is unreadable. - It now logs a warning with the hash, the db path and the exception type. `items()` decodes every row on each eviction pass, so the warning fires once per hash. - The once-per-hash guard is `log_safety.WarnOnce`. Within a one-hour window it warns for at most 1,000 hashes, then logs one overflow warning, and later rows log at DEBUG only, so repeated eviction scans over many bad rows cannot flood the log. When the window ends, a row that is still broken warns again, about once an hour. A hash is used up only while WARNING is enabled. - CCR keys come from model-issued retrieve calls, so they are logged through `log_safety.safe_id`: a repr (newlines escaped) capped at 80 characters. - Failed `delete`/`exists`/`clear`/`count`/`keys`/`items`/`get_stats` all logged as `"op"`. Each now logs its real name, with the key for `get`/`set`/`delete`/`exists`. - The transient-error warning now includes the db path. - `headroom/memory/traffic_learner.py`: the two read-only SQLite reads returned `[]` on `OperationalError`/`DatabaseError` with no log: the hydrate path and `_load_persisted_patterns_from_sqlite`. Both now warn with the db path. Also: - dropped a redundant `conn.close()` in an `except` whose `finally` already closes; - the two best-effort `conn.close()` calls now catch `sqlite3.Error` and log at debug, instead of `except Exception: pass`. - `headroom/memory/backends/direct_mem0.py`: `get_memory`/`delete_memory` turned any Qdrant/Neo4j exception into "not found". They now warn with the memory id and the exception type first. - The exception text is not logged at any level, because backend errors can carry stored memory or connection details. A DEBUG line gives `describe_exception`. - Memory ids come from the model, so they are logged through `log_safety.safe_id`, which escapes newlines and caps the length. - mem0 2.x `Memory.delete` raises exactly `ValueError(f"Memory with id {memory_id} not found")` for an unknown id. Only that message, for the requested id, logs at DEBUG; the delete is model-driven, so it is expected. Any other `ValueError`, such as a missing collection, still warns. - When a corrupt CCR store is recreated, the unreadable-row guard starts fresh, because the new file has no rows. - `headroom/learn/registry.py`: a built-in learn plugin that fails to import was logged at debug, so `headroom learn --agent <name>` only said "Unknown agent". It is now a warning with the traceback. Built-in plugins use only the stdlib, so this never fires on a healthy install. - `headroom/cache/compression_store.py`: the "Failed to load CCR backend" warning now says it falls back to the in-process store, and what that costs. This matches the SQLite branch next to it. It logs the exception type only, because the backend's text can echo `HEADROOM_REDIS_URL`. DEBUG gets `describe_exception`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest -q -p no:cacheprovider tests/test_storage_backends.py tests/test_storage tests/test_ccr_sqlite_backend.py tests/test_compression_store.py tests/test_learn/test_registry.py tests/test_memory/test_traffic_learner.py tests/test_memory/test_direct_mem0.py tests/test_private_state_files.py tests/test_adapter_hooks.py tests/test_cache/test_backends.py tests/test_learn/test_opencode_scanner.py tests/test_openai_responses_traffic_learner.py tests/test_learn/test_section_lifecycle.py tests/test_learn/test_worktree_merge.py tests/test_memory_handler_native_ops.py ================= 537 passed, 1 skipped, 5 warnings in 34.37s ================== # The three new tests against origin/main (product code reverted): FAILED tests/test_storage_backends.py::test_create_storage_warns_when_a_backend_plugin_fails_to_load FAILED tests/test_ccr_sqlite_backend.py::TestSQLiteBackend::test_unreadable_row_is_logged_once_and_healthy_rows_are_not FAILED tests/test_learn/test_registry.py::TestBuiltinLoadFailure::test_broken_builtin_plugin_is_logged_as_a_warning $ uvx ruff@0.16.8 check <changed files> All checks passed! # Added in review: the hydration-failure warning; mem0 delete/get paths (only mem0's exact missing-memory message is quiet; a missing collection warns); newline and 100,000-char model ids; two scans over more bad rows than the cap; and storage-plugin, CCR-backend and mem0 failures whose records are rendered with logging.Formatter at DEBUG and WARNING and checked for credential and content canaries. # Those tests against the previous head6c1ad49c, in a scratch worktree: FAILED tests/test_ccr_sqlite_backend.py::TestSQLiteBackend::test_repeated_scans_over_the_cap_stay_bounded FAILED tests/test_storage_backends.py::test_create_storage_warns_when_a_backend_plugin_fails_to_load FAILED tests/test_memory/test_direct_mem0.py::test_backend_failures_warn_without_exception_text[delete_memory-exc1-True] FAILED tests/test_memory/test_direct_mem0.py::test_backend_failures_warn_without_exception_text[delete_memory-exc4-True] FAILED tests/test_memory/test_direct_mem0.py::test_backend_failures_warn_without_exception_text[get_memory-exc5-True] FAILED tests/test_adapter_hooks.py::TestCCREntryPointLoading::test_failing_backend_plugin_logs_no_credentials_at_any_level $ mypy <6 changed product files> --ignore-missing-imports Success: no issues found in 6 source files ``` ## Real Behavior Proof - Environment: macOS, Python 3.12.13, repo .venv, branch checkout at72a166cb(basea9815bbbplus main and #4138 merged in), temp HOME / HEADROOM_WORKSPACE_DIR / CODEX_HOME, offline. - Exact command / steps: two runs from this checkout, listed below. - Ran the shared smoke harness, which starts a real proxy from this checkout against a local fake upstream and sends one Anthropic and one OpenAI request with a compressible tool result. - Ran a direct, unmocked script at DEBUG. A storage plugin raises an error that quotes its `pg://user:CRED_CANARY@...` URL. Then 5 unreadable rows are written into a real CCR SQLite store (the cap is lowered to 3 for the run), and `items()` scans it twice. - Observed result: as listed below. The raw output is in the blocks after this list. - Both proxy requests return HTTP 200 with savings, and the proxy log has no new warnings. - The storage fallback logs the scheme and exception type at WARNING, and only types and code locations at DEBUG. `CRED_CANARY` appears nowhere in the output. - The CCR store warns for 3 rows, then once for the overflow, and the second scan (inside the same hour) adds nothing. All 5 rows are kept, and reads miss as before. - Not tested: the two paths listed below. - A live Qdrant/Neo4j outage on the DirectMem0 path. Unit tests with a raising client cover it. - A real third-party storage or CCR plugin. ```text $ bash smoke.sh <worktree> branch: cleanup/memory-cache-storage-logging @72a166cbhealth: 200 POST /v1/messages -> HTTP200 smoke-ok POST /v1/chat/completions -> HTTP200 smoke-ok stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82 proxy.log WARNING/ERROR lines: 2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you 4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base 4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection. $ HOME=$TMP/home HEADROOM_WORKSPACE_DIR=$TMP/ws CODEX_HOME=$TMP/codex python -u run.py # logging at DEBUG INFO headroom._ort - Pinned ORT_DYLIB_PATH to bundled ONNX Runtime: /Users/tejas/headroom/.venv/lib/python3.12/site-packages/onnxruntime/capi/libonnxruntime.1.26.0.dylib WARNING headroom.storage - Failed to load storage backend for scheme 'pg' (<ValueError>); falling back to SQLite storage DEBUG headroom.storage - Storage backend 'pg' load failure: ValueError at run.py:11 in factory <- headroom/storage/__init__.py:70 in create_storage WARNING headroom.cache.backends.sqlite - CCR SQLite entry 'bad0' in $TMP/ws/ccr_store.db is unreadable (JSONDecodeError); treating as a miss WARNING headroom.cache.backends.sqlite - CCR SQLite entry 'bad1' in $TMP/ws/ccr_store.db is unreadable (JSONDecodeError); treating as a miss WARNING headroom.cache.backends.sqlite - CCR SQLite entry 'bad2' in $TMP/ws/ccr_store.db is unreadable (JSONDecodeError); treating as a miss WARNING headroom.cache.backends.sqlite - More than 3 distinct unreadable CCR SQLite entries; further ones are logged at debug only scan 1 -> [] scan 2 -> [] rows kept: 5 ``` Update 2026-10-15 (head `4e78ca73`). Plugin-load warnings now name the exception through `safe_id`, which renders `<ValueError>`, or `<object>` for a class whose name is not a plain identifier. The learn-plugin failures no longer put a traceback in the WARNING. This is a direct, unmocked run at DEBUG with `HEADROOM_DEBUG_DUMP` unset. A storage plugin and an external learn plugin raise errors quoting `pg://user:CRED_CANARY@db`, one of them through a class named `PRIVATE_CREDENTIAL_CANARY\nFORGED_LOG_ENTRY`. `grep -c 'CANARY|FORGED'` on the output gives 0. ```text WARNING headroom.storage - Failed to load storage backend for scheme 'pg' (<ValueError>); falling back to SQLite storage DEBUG headroom.storage - Storage backend 'pg' load failure: ValueError at headroom/storage/__init__.py:70 in create_storage WARNING headroom.storage - Failed to load storage backend for scheme 'pg' (<object>); falling back to SQLite storage DEBUG headroom.storage - Storage backend 'pg' load failure: <exception> at headroom/storage/__init__.py:70 in create_storage WARNING headroom.learn.registry - Failed to load external learn plugin 'broken' (<object>) DEBUG headroom.learn.registry - External learn plugin 'broken' failure: <exception> at headroom/learn/registry.py:52 in _discover learn plugins: ['claude', 'codex', 'gemini', 'grok', 'opencode'] ``` ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: N/A; logging only. - Stable/default behavior changed: No. Only log output changes. All return values and fallbacks are as before. - Kill switch / disable path: Standard logging level configuration for the `headroom.*` loggers. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A. ## Additional Notes Part of maintainer-requested cleanup #4136 (dead code, duplication, debuggability), split into small PRs for review. - Documentation: N/A, because no user-facing option changes. - On the DirectMem0 path, deleting an unknown id stays quiet at the default level. Only real backend failures warn. - Depends on #4138; its two files show in this diff until it merges. - The `describe_exception` calls run only when `logger.isEnabledFor(logging.DEBUG)`, so a failure path does no formatting work at INFO. That PR also touches `headroom/proxy/handlers/_debug_dump.py`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JerrettDavis
parent
ea947ca421
commit
4e6e955cad
Vendored
+42
-15
@@ -29,6 +29,7 @@ from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from ...fileperms import ensure_private_file
|
||||
from ...log_safety import WarnOnce, safe_id
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ..compression_store import CompressionEntry
|
||||
@@ -51,6 +52,11 @@ DROP INDEX IF EXISTS idx_ccr_expiry;
|
||||
# Purge expired rows at most this often (seconds). Purging is hygiene,
|
||||
# not correctness — CompressionStore checks TTL on every get().
|
||||
_PURGE_INTERVAL = 60.0
|
||||
# items() decodes every row on each eviction pass, so an unreadable row warns
|
||||
# once per hash per hour (WarnOnce's window), for at most this many hashes per
|
||||
# hour; further ones log at debug only. A row that stays broken warns again
|
||||
# about once an hour, which keeps it visible without flooding.
|
||||
_MAX_REPORTED_UNREADABLE = 1000
|
||||
|
||||
|
||||
def default_db_path() -> Path:
|
||||
@@ -83,6 +89,9 @@ class SQLiteBackend:
|
||||
self._path.parent.mkdir(parents=True, exist_ok=True)
|
||||
self._lock = threading.Lock()
|
||||
self._last_purge = 0.0
|
||||
self._unreadable_warned = WarnOnce(
|
||||
_MAX_REPORTED_UNREADABLE, "unreadable CCR SQLite entries"
|
||||
)
|
||||
self._conn = self._open()
|
||||
|
||||
@staticmethod
|
||||
@@ -147,7 +156,9 @@ class SQLiteBackend:
|
||||
log and treat the operation as a miss; never destroy data over a
|
||||
transient error."""
|
||||
if not self._is_corruption(error):
|
||||
logger.warning("CCR SQLite %s failed (transient, no reset): %s", op, error)
|
||||
logger.warning(
|
||||
"CCR SQLite %s failed on %s (transient, no reset): %s", op, self._path, error
|
||||
)
|
||||
return
|
||||
logger.warning(
|
||||
"CCR SQLite store at %s is corrupt (%s); recreating. "
|
||||
@@ -161,26 +172,42 @@ class SQLiteBackend:
|
||||
except Exception: # noqa: BLE001 - best-effort close on corrupt handle
|
||||
pass
|
||||
self._path.unlink(missing_ok=True)
|
||||
self._unreadable_warned = WarnOnce(
|
||||
_MAX_REPORTED_UNREADABLE, "unreadable CCR SQLite entries"
|
||||
)
|
||||
self._conn = self._open()
|
||||
|
||||
def _entry_from_json(self, raw: str) -> CompressionEntry | None:
|
||||
def _report_unreadable(self, hash_key: str, reason: object) -> None:
|
||||
first = self._unreadable_warned.first(hash_key, logger)
|
||||
logger.log(
|
||||
logging.WARNING if first else logging.DEBUG,
|
||||
"CCR SQLite entry %s in %s is unreadable (%s); treating as a miss",
|
||||
safe_id(hash_key),
|
||||
self._path,
|
||||
reason,
|
||||
)
|
||||
|
||||
def _entry_from_json(self, raw: str, hash_key: str) -> CompressionEntry | None:
|
||||
from ..compression_store import CompressionEntry
|
||||
|
||||
try:
|
||||
data = json.loads(raw)
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
except (json.JSONDecodeError, TypeError) as e:
|
||||
self._report_unreadable(hash_key, type(e).__name__)
|
||||
return None
|
||||
if not isinstance(data, dict):
|
||||
self._report_unreadable(hash_key, "not a JSON object")
|
||||
return None
|
||||
known = {f.name for f in fields(CompressionEntry)}
|
||||
try:
|
||||
return CompressionEntry(**{k: v for k, v in data.items() if k in known})
|
||||
except (TypeError, ValueError):
|
||||
except (TypeError, ValueError) as e:
|
||||
# A blob that parses as JSON but is missing a required field (schema
|
||||
# drift across an upgrade, a partially written row) must degrade to a
|
||||
# miss, not raise. Otherwise a single bad row crashes get() — and,
|
||||
# via items(), _clean_expired() runs it on every store's eviction, so
|
||||
# one poison row would break all reads, evictions, and stores.
|
||||
self._report_unreadable(hash_key, type(e).__name__)
|
||||
return None
|
||||
|
||||
def _purge_expired(self, now: float) -> int:
|
||||
@@ -217,11 +244,11 @@ class SQLiteBackend:
|
||||
(hash_key,),
|
||||
).fetchone()
|
||||
except sqlite3.DatabaseError as e:
|
||||
self._handle_db_error(e, "get")
|
||||
self._handle_db_error(e, f"get {safe_id(hash_key)}")
|
||||
return None
|
||||
if row is None:
|
||||
return None
|
||||
return self._entry_from_json(row[0])
|
||||
return self._entry_from_json(row[0], hash_key)
|
||||
|
||||
def set(self, hash_key: str, entry: CompressionEntry) -> None:
|
||||
payload = json.dumps(asdict(entry), ensure_ascii=False)
|
||||
@@ -235,7 +262,7 @@ class SQLiteBackend:
|
||||
self._conn.commit()
|
||||
self._maybe_purge()
|
||||
except sqlite3.DatabaseError as e:
|
||||
self._handle_db_error(e, "set")
|
||||
self._handle_db_error(e, f"set {safe_id(hash_key)}")
|
||||
|
||||
def delete(self, hash_key: str) -> bool:
|
||||
with self._lock:
|
||||
@@ -247,7 +274,7 @@ class SQLiteBackend:
|
||||
self._conn.commit()
|
||||
return cur.rowcount > 0
|
||||
except sqlite3.DatabaseError as e:
|
||||
self._handle_db_error(e, "op")
|
||||
self._handle_db_error(e, f"delete {safe_id(hash_key)}")
|
||||
return False
|
||||
|
||||
def exists(self, hash_key: str) -> bool:
|
||||
@@ -258,7 +285,7 @@ class SQLiteBackend:
|
||||
(hash_key,),
|
||||
).fetchone()
|
||||
except sqlite3.DatabaseError as e:
|
||||
self._handle_db_error(e, "op")
|
||||
self._handle_db_error(e, f"exists {safe_id(hash_key)}")
|
||||
return False
|
||||
return row is not None
|
||||
|
||||
@@ -268,14 +295,14 @@ class SQLiteBackend:
|
||||
self._conn.execute("DELETE FROM ccr_entries")
|
||||
self._conn.commit()
|
||||
except sqlite3.DatabaseError as e:
|
||||
self._handle_db_error(e, "op")
|
||||
self._handle_db_error(e, "clear")
|
||||
|
||||
def count(self) -> int:
|
||||
with self._lock:
|
||||
try:
|
||||
row = self._conn.execute("SELECT COUNT(*) FROM ccr_entries").fetchone()
|
||||
except sqlite3.DatabaseError as e:
|
||||
self._handle_db_error(e, "op")
|
||||
self._handle_db_error(e, "count")
|
||||
return 0
|
||||
return int(row[0])
|
||||
|
||||
@@ -294,7 +321,7 @@ class SQLiteBackend:
|
||||
try:
|
||||
rows = self._conn.execute("SELECT hash FROM ccr_entries").fetchall()
|
||||
except sqlite3.DatabaseError as e:
|
||||
self._handle_db_error(e, "op")
|
||||
self._handle_db_error(e, "keys")
|
||||
return []
|
||||
return [r[0] for r in rows]
|
||||
|
||||
@@ -303,11 +330,11 @@ class SQLiteBackend:
|
||||
try:
|
||||
rows = self._conn.execute("SELECT hash, entry_json FROM ccr_entries").fetchall()
|
||||
except sqlite3.DatabaseError as e:
|
||||
self._handle_db_error(e, "op")
|
||||
self._handle_db_error(e, "items")
|
||||
return []
|
||||
out: list[tuple[str, CompressionEntry]] = []
|
||||
for hash_key, raw in rows:
|
||||
entry = self._entry_from_json(raw)
|
||||
entry = self._entry_from_json(raw, hash_key)
|
||||
if entry is not None:
|
||||
out.append((hash_key, entry))
|
||||
return out
|
||||
@@ -317,7 +344,7 @@ class SQLiteBackend:
|
||||
try:
|
||||
count_row = self._conn.execute("SELECT COUNT(*) FROM ccr_entries").fetchone()
|
||||
except sqlite3.DatabaseError as e:
|
||||
self._handle_db_error(e, "op")
|
||||
self._handle_db_error(e, "stats")
|
||||
count_row = (0,)
|
||||
try:
|
||||
bytes_used = self._path.stat().st_size
|
||||
|
||||
Vendored
+12
-1
@@ -43,6 +43,8 @@ from contextvars import ContextVar
|
||||
from dataclasses import dataclass, field, replace
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from ..log_safety import describe_exception, safe_id
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ..memory.tracker import ComponentStats
|
||||
from .backends import CompressionStoreBackend
|
||||
@@ -1164,7 +1166,16 @@ def _create_default_ccr_backend() -> CompressionStoreBackend | None:
|
||||
backend: CompressionStoreBackend = fn(**kwargs)
|
||||
return backend
|
||||
except Exception as e:
|
||||
logger.warning("Failed to load CCR backend %s: %s", backend_type, e)
|
||||
# The backend's exception can echo HEADROOM_REDIS_URL (credentials included),
|
||||
# so no level logs its message; DEBUG gets describe_exception (types and frames).
|
||||
logger.warning(
|
||||
"Failed to load CCR backend %s (%s); falling back to the in-process store. "
|
||||
"Retrieval will not survive proxy restarts or cross worker processes.",
|
||||
backend_type,
|
||||
safe_id(e),
|
||||
)
|
||||
if logger.isEnabledFor(logging.DEBUG):
|
||||
logger.debug("CCR backend %s load failure: %s", backend_type, describe_exception(e))
|
||||
return None
|
||||
|
||||
|
||||
|
||||
@@ -12,6 +12,7 @@ import importlib
|
||||
import logging
|
||||
import pkgutil
|
||||
|
||||
from ..log_safety import describe_exception, safe_id
|
||||
from .base import LearnPlugin
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -34,8 +35,13 @@ def _discover() -> dict[str, LearnPlugin]:
|
||||
if isinstance(p, LearnPlugin):
|
||||
plugins[p.name] = p
|
||||
logger.debug("Loaded built-in learn plugin: %s", p.name)
|
||||
except Exception:
|
||||
logger.debug("Failed to load built-in plugin: %s", mod_name, exc_info=True)
|
||||
except Exception as e:
|
||||
# Exception text can quote paths or credentials: type only at WARNING.
|
||||
logger.warning("Failed to load built-in learn plugin %s (%s)", mod_name, safe_id(e))
|
||||
if logger.isEnabledFor(logging.DEBUG):
|
||||
logger.debug(
|
||||
"Built-in learn plugin %s failure: %s", mod_name, describe_exception(e)
|
||||
)
|
||||
|
||||
# 2. External: entry_points(group="headroom.learn_plugin")
|
||||
try:
|
||||
@@ -56,8 +62,16 @@ def _discover() -> dict[str, LearnPlugin]:
|
||||
if isinstance(p, LearnPlugin):
|
||||
plugins[p.name] = p # External overrides built-in on name collision
|
||||
logger.debug("Loaded external learn plugin: %s (%s)", p.name, ep.name)
|
||||
except Exception:
|
||||
logger.warning("Failed to load external learn plugin: %s", ep.name, exc_info=True)
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
"Failed to load external learn plugin %s (%s)", safe_id(ep.name), safe_id(e)
|
||||
)
|
||||
if logger.isEnabledFor(logging.DEBUG):
|
||||
logger.debug(
|
||||
"External learn plugin %s failure: %s",
|
||||
safe_id(ep.name),
|
||||
describe_exception(e),
|
||||
)
|
||||
except Exception:
|
||||
logger.debug("Entry point discovery failed", exc_info=True)
|
||||
|
||||
|
||||
@@ -57,6 +57,7 @@ from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
|
||||
from headroom.log_safety import describe_exception, safe_id
|
||||
from headroom.memory import qdrant_env
|
||||
from headroom.memory.models import Memory
|
||||
from headroom.memory.ports import MemorySearchResult
|
||||
@@ -921,7 +922,29 @@ class DirectMem0Adapter:
|
||||
try:
|
||||
await asyncio.to_thread(self._mem0_client.delete, memory_id=memory_id)
|
||||
return True
|
||||
except Exception:
|
||||
except Exception as e:
|
||||
# mem0 (Memory.delete, mem0ai 2.x) raises exactly
|
||||
# ValueError(f"Memory with id {memory_id} not found") for an unknown id.
|
||||
# The delete is model-driven, so a stale or invented id is expected. Any
|
||||
# other ValueError (e.g. a missing collection) is a real failure.
|
||||
if isinstance(e, ValueError) and str(e).startswith(
|
||||
f"Memory with id {memory_id} not found"
|
||||
):
|
||||
logger.debug("DirectMem0: delete_memory(%s): memory not found", safe_id(memory_id))
|
||||
return False
|
||||
# Backend exception text can carry stored memory or connection details,
|
||||
# so no level logs its message; DEBUG gets describe_exception.
|
||||
logger.warning(
|
||||
"DirectMem0: delete_memory(%s) failed (%s)",
|
||||
safe_id(memory_id),
|
||||
safe_id(e),
|
||||
)
|
||||
if logger.isEnabledFor(logging.DEBUG):
|
||||
logger.debug(
|
||||
"DirectMem0: delete_memory(%s) failure: %s",
|
||||
safe_id(memory_id),
|
||||
describe_exception(e),
|
||||
)
|
||||
return False
|
||||
|
||||
async def get_memory(self, memory_id: str) -> Memory | None:
|
||||
@@ -949,7 +972,14 @@ class DirectMem0Adapter:
|
||||
valid_from=_utcnow(),
|
||||
metadata=result.get("metadata") or {},
|
||||
)
|
||||
except Exception:
|
||||
except Exception as e:
|
||||
logger.warning("DirectMem0: get_memory(%s) failed (%s)", safe_id(memory_id), safe_id(e))
|
||||
if logger.isEnabledFor(logging.DEBUG):
|
||||
logger.debug(
|
||||
"DirectMem0: get_memory(%s) failure: %s",
|
||||
safe_id(memory_id),
|
||||
describe_exception(e),
|
||||
)
|
||||
return None
|
||||
|
||||
@property
|
||||
|
||||
@@ -1374,7 +1374,8 @@ class TrafficLearner:
|
||||
uri = f"file:{db_path}?mode=ro"
|
||||
try:
|
||||
conn = sqlite3.connect(uri, uri=True)
|
||||
except sqlite3.OperationalError:
|
||||
except sqlite3.OperationalError as e:
|
||||
logger.warning("Traffic learner hydrate: failed to open %s: %s", db_path, e)
|
||||
return []
|
||||
try:
|
||||
rows = conn.execute(
|
||||
@@ -1390,13 +1391,14 @@ class TrafficLearner:
|
||||
"LIMIT ?",
|
||||
(self._dedup_window,),
|
||||
).fetchall()
|
||||
except sqlite3.DatabaseError:
|
||||
except sqlite3.DatabaseError as e:
|
||||
logger.warning("Traffic learner hydrate: failed to read %s: %s", db_path, e)
|
||||
return []
|
||||
finally:
|
||||
try:
|
||||
conn.close()
|
||||
except Exception:
|
||||
pass
|
||||
except sqlite3.Error as e:
|
||||
logger.debug("Traffic learner hydrate: closing %s failed: %s", db_path, e)
|
||||
return [(row[0], row[1] or "", row[2] or "{}") for row in rows]
|
||||
|
||||
try:
|
||||
@@ -1781,7 +1783,8 @@ def _load_persisted_patterns_from_sqlite(db_path: Path) -> list[ExtractedPattern
|
||||
patterns: dict[str, ExtractedPattern] = {}
|
||||
try:
|
||||
conn = sqlite3.connect(uri, uri=True)
|
||||
except sqlite3.OperationalError:
|
||||
except sqlite3.OperationalError as e:
|
||||
logger.warning("Traffic learner: failed to open persisted patterns %s: %s", db_path, e)
|
||||
return []
|
||||
try:
|
||||
conn.row_factory = sqlite3.Row
|
||||
@@ -1790,14 +1793,14 @@ def _load_persisted_patterns_from_sqlite(db_path: Path) -> list[ExtractedPattern
|
||||
"FROM memories "
|
||||
"WHERE json_extract(metadata, '$.source') = 'traffic_learner'"
|
||||
).fetchall()
|
||||
except sqlite3.DatabaseError:
|
||||
conn.close()
|
||||
except sqlite3.DatabaseError as e:
|
||||
logger.warning("Traffic learner: failed to read persisted patterns %s: %s", db_path, e)
|
||||
return []
|
||||
finally:
|
||||
try:
|
||||
conn.close()
|
||||
except Exception:
|
||||
pass
|
||||
except sqlite3.Error as e:
|
||||
logger.debug("Traffic learner: closing persisted patterns %s failed: %s", db_path, e)
|
||||
|
||||
for row in rows:
|
||||
content = row["content"] or ""
|
||||
|
||||
@@ -1,7 +1,10 @@
|
||||
"""Storage modules for Headroom SDK."""
|
||||
|
||||
import logging
|
||||
import os
|
||||
|
||||
from headroom.log_safety import describe_exception, safe_id
|
||||
|
||||
from .base import Storage
|
||||
from .jsonl import JSONLStorage
|
||||
from .sqlite import SQLiteStorage
|
||||
@@ -12,6 +15,8 @@ __all__ = [
|
||||
"JSONLStorage",
|
||||
]
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _builtin_storage_path(store_url: str, scheme: str) -> str:
|
||||
"""Return the filesystem portion of a built-in storage URL."""
|
||||
@@ -64,7 +69,23 @@ def create_storage(store_url: str) -> Storage:
|
||||
create_fn = ep.load()
|
||||
result: Storage = create_fn(store_url)
|
||||
return result
|
||||
except Exception:
|
||||
pass
|
||||
logger.warning(
|
||||
"No headroom.storage_backend entry point for scheme %r; "
|
||||
"falling back to SQLite storage",
|
||||
scheme,
|
||||
)
|
||||
except Exception as e:
|
||||
# The plugin's exception can echo the store URL, credentials included,
|
||||
# so no level logs its message; DEBUG gets describe_exception (types and frames).
|
||||
logger.warning(
|
||||
"Failed to load storage backend for scheme %r (%s); "
|
||||
"falling back to SQLite storage",
|
||||
scheme,
|
||||
safe_id(e),
|
||||
)
|
||||
if logger.isEnabledFor(logging.DEBUG):
|
||||
logger.debug(
|
||||
"Storage backend %r load failure: %s", scheme, describe_exception(e)
|
||||
)
|
||||
# Default to SQLite (legacy behavior)
|
||||
return SQLiteStorage(store_url)
|
||||
|
||||
@@ -15,6 +15,7 @@ Tests cover:
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import threading
|
||||
from typing import Any
|
||||
|
||||
@@ -410,6 +411,89 @@ class TestCCREntryPointLoading:
|
||||
|
||||
assert _create_default_ccr_backend() is None
|
||||
|
||||
def test_failing_backend_plugin_logs_no_credentials_at_any_level(self, monkeypatch):
|
||||
"""A plugin error that quotes HEADROOM_REDIS_URL must not reach the log, at
|
||||
WARNING or DEBUG, rendered the way a handler would (exc_info included)."""
|
||||
import logging
|
||||
from types import SimpleNamespace
|
||||
|
||||
from headroom.cache.compression_store import _create_default_ccr_backend
|
||||
|
||||
def _factory(url: str, tenant_prefix: str) -> None:
|
||||
raise ConnectionError(f"cannot reach {url} holding CONTENT_CANARY")
|
||||
|
||||
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "redisx")
|
||||
monkeypatch.setenv("HEADROOM_REDIS_URL", "redis://user:CRED_CANARY@cache:6379/0")
|
||||
monkeypatch.setattr(
|
||||
"importlib.metadata.entry_points",
|
||||
lambda group: [SimpleNamespace(name="redisx", load=lambda: _factory)],
|
||||
)
|
||||
records: list[logging.LogRecord] = []
|
||||
|
||||
class _Collect(logging.Handler):
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
records.append(record)
|
||||
|
||||
logger = logging.getLogger("headroom.cache.compression_store")
|
||||
handler = _Collect(level=logging.DEBUG)
|
||||
previous = logger.level
|
||||
logger.addHandler(handler)
|
||||
logger.setLevel(logging.DEBUG)
|
||||
try:
|
||||
assert _create_default_ccr_backend() is None
|
||||
finally:
|
||||
logger.removeHandler(handler)
|
||||
logger.setLevel(previous)
|
||||
|
||||
assert {r.levelno for r in records} >= {logging.WARNING, logging.DEBUG}
|
||||
for record in records:
|
||||
rendered = logging.Formatter().format(record)
|
||||
assert "CRED_CANARY" not in rendered
|
||||
assert "CONTENT_CANARY" not in rendered
|
||||
assert any("ConnectionError" in r.getMessage() for r in records)
|
||||
|
||||
@pytest.mark.parametrize("level", [logging.INFO, logging.DEBUG])
|
||||
def test_backend_plugin_class_name_cannot_leak_or_forge_log_lines(self, monkeypatch, level):
|
||||
"""A plugin chooses its exception's class name; the WARNING must not carry it."""
|
||||
from types import SimpleNamespace
|
||||
|
||||
from headroom.cache.compression_store import _create_default_ccr_backend
|
||||
|
||||
error_type = type("PRIVATE_CREDENTIAL_CANARY\nFORGED_LOG_ENTRY", (RuntimeError,), {})
|
||||
|
||||
def _factory(url: str, tenant_prefix: str) -> None:
|
||||
raise error_type("plugin text")
|
||||
|
||||
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
||||
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "redisx")
|
||||
monkeypatch.setattr(
|
||||
"importlib.metadata.entry_points",
|
||||
lambda group: [SimpleNamespace(name="redisx", load=lambda: _factory)],
|
||||
)
|
||||
records: list[logging.LogRecord] = []
|
||||
|
||||
class _Collect(logging.Handler):
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
records.append(record)
|
||||
|
||||
logger = logging.getLogger("headroom.cache.compression_store")
|
||||
handler = _Collect(level=logging.DEBUG)
|
||||
previous = logger.level
|
||||
logger.addHandler(handler)
|
||||
logger.setLevel(level)
|
||||
try:
|
||||
assert _create_default_ccr_backend() is None
|
||||
finally:
|
||||
logger.removeHandler(handler)
|
||||
logger.setLevel(previous)
|
||||
|
||||
assert len([r for r in records if r.levelno == logging.WARNING]) == 1
|
||||
for record in records:
|
||||
rendered = logging.Formatter().format(record)
|
||||
assert "PRIVATE_CREDENTIAL_CANARY" not in rendered
|
||||
assert "FORGED_LOG_ENTRY" not in rendered
|
||||
assert "\n" not in rendered
|
||||
|
||||
def test_inmemory_backend_satisfies_protocol(self):
|
||||
"""InMemoryBackend satisfies the CompressionStoreBackend protocol."""
|
||||
backend = InMemoryBackend()
|
||||
|
||||
@@ -7,6 +7,7 @@ worker processes — neither holds for the in-memory dict.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import sqlite3
|
||||
import time
|
||||
@@ -221,7 +222,7 @@ class TestSQLiteBackend:
|
||||
monkeypatch.setattr(
|
||||
backend,
|
||||
"_entry_from_json",
|
||||
lambda _raw: pytest.fail("new insertion decoded existing CCR payloads"),
|
||||
lambda *_args: pytest.fail("new insertion decoded existing CCR payloads"),
|
||||
)
|
||||
|
||||
new_hash = store.store("new original", "new compact", ttl=60)
|
||||
@@ -283,6 +284,83 @@ class TestSQLiteBackend:
|
||||
store.store(original="fresh original", compressed="c", original_item_count=1)
|
||||
assert b.get("good") is not None
|
||||
|
||||
def test_unreadable_row_is_logged_once_and_healthy_rows_are_not(self, db_path):
|
||||
"""An unreadable row must say so in the log (once per hash), so an
|
||||
operator can tell corruption from a genuine retrieval miss."""
|
||||
records: list[logging.LogRecord] = []
|
||||
|
||||
class _Collect(logging.Handler):
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
records.append(record)
|
||||
|
||||
handler = _Collect(level=logging.DEBUG)
|
||||
logger = logging.getLogger("headroom.cache.backends.sqlite")
|
||||
logger.addHandler(handler)
|
||||
try:
|
||||
b = SQLiteBackend(db_path)
|
||||
b.set("good", make_entry("good"))
|
||||
assert b.get("good") is not None
|
||||
assert [r for r in records if "unreadable" in r.getMessage()] == []
|
||||
|
||||
with b._lock:
|
||||
b._conn.execute(
|
||||
"INSERT OR REPLACE INTO ccr_entries "
|
||||
"(hash, entry_json, created_at, ttl) VALUES (?, ?, ?, ?)",
|
||||
("broken", "{not json", time.time(), 1800),
|
||||
)
|
||||
b._conn.commit()
|
||||
|
||||
assert b.get("broken") is None
|
||||
b.items()
|
||||
b.items()
|
||||
unreadable = [r for r in records if "unreadable" in r.getMessage()]
|
||||
assert len(unreadable) == 1
|
||||
assert unreadable[0].levelno == logging.WARNING
|
||||
assert "broken" in unreadable[0].getMessage()
|
||||
assert "good" not in unreadable[0].getMessage()
|
||||
finally:
|
||||
logger.removeHandler(handler)
|
||||
|
||||
def test_repeated_scans_over_the_cap_stay_bounded(self, db_path, monkeypatch):
|
||||
"""More bad rows than the cap, scanned twice: at most cap row warnings plus
|
||||
one overflow warning in total, and the rows stay in place and still miss."""
|
||||
import headroom.cache.backends.sqlite as sqlite_mod
|
||||
|
||||
monkeypatch.setattr(sqlite_mod, "_MAX_REPORTED_UNREADABLE", 2)
|
||||
records: list[logging.LogRecord] = []
|
||||
|
||||
class _Collect(logging.Handler):
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
records.append(record)
|
||||
|
||||
handler = _Collect(level=logging.DEBUG)
|
||||
logger = logging.getLogger("headroom.cache.backends.sqlite")
|
||||
logger.addHandler(handler)
|
||||
keys = [f"bad{i}" for i in range(5)]
|
||||
try:
|
||||
b = SQLiteBackend(db_path)
|
||||
with b._lock:
|
||||
for key in keys:
|
||||
b._conn.execute(
|
||||
"INSERT OR REPLACE INTO ccr_entries "
|
||||
"(hash, entry_json, created_at, ttl) VALUES (?, ?, ?, ?)",
|
||||
(key, "{not json", time.time(), 1800),
|
||||
)
|
||||
b._conn.commit()
|
||||
assert b.items() == []
|
||||
assert b.items() == []
|
||||
assert b.get("bad4") is None
|
||||
assert b.count() == len(keys)
|
||||
finally:
|
||||
logger.removeHandler(handler)
|
||||
|
||||
warnings = [r.getMessage() for r in records if r.levelno == logging.WARNING]
|
||||
row_warnings = [m for m in warnings if "is unreadable" in m]
|
||||
assert len(row_warnings) == 2
|
||||
assert all("JSONDecodeError" in m for m in row_warnings)
|
||||
assert len([m for m in warnings if "More than 2 warnings about" in m]) == 1
|
||||
assert len(warnings) == 3
|
||||
|
||||
def test_store_ttl_enforcement_via_compression_store(self, db_path):
|
||||
"""TTL checks stay in CompressionStore; expired entries miss."""
|
||||
store = CompressionStore(backend=SQLiteBackend(db_path))
|
||||
|
||||
@@ -3,6 +3,8 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import importlib
|
||||
import logging
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
@@ -110,6 +112,94 @@ class TestExternalPlugin:
|
||||
assert reg["cursor"].name == "cursor"
|
||||
|
||||
|
||||
class TestBuiltinLoadFailure:
|
||||
def test_broken_builtin_plugin_is_logged_as_a_warning(self):
|
||||
records: list[logging.LogRecord] = []
|
||||
|
||||
class _Collect(logging.Handler):
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
records.append(record)
|
||||
|
||||
handler = _Collect(level=logging.DEBUG)
|
||||
logger = logging.getLogger("headroom.learn.registry")
|
||||
logger.addHandler(handler)
|
||||
real_import = importlib.import_module
|
||||
|
||||
def _import(name: str, *args, **kwargs): # noqa: ANN002, ANN003, ANN202
|
||||
if name == "headroom.learn.plugins.grok":
|
||||
raise ImportError("simulated breakage")
|
||||
return real_import(name, *args, **kwargs)
|
||||
|
||||
try:
|
||||
get_registry()
|
||||
assert [r for r in records if r.levelno >= logging.WARNING] == []
|
||||
|
||||
reset_registry()
|
||||
with patch("headroom.learn.registry.importlib.import_module", _import):
|
||||
reg = get_registry()
|
||||
assert "grok" not in reg
|
||||
assert "claude" in reg
|
||||
warnings = [r for r in records if r.levelno == logging.WARNING]
|
||||
assert len(warnings) == 1
|
||||
assert "grok" in warnings[0].getMessage()
|
||||
finally:
|
||||
logger.removeHandler(handler)
|
||||
|
||||
|
||||
_MALFORMED_ERROR = type("PRIVATE_CREDENTIAL_CANARY\nFORGED_LOG_ENTRY", (RuntimeError,), {})
|
||||
|
||||
|
||||
@pytest.mark.parametrize("level", [logging.INFO, logging.DEBUG])
|
||||
@pytest.mark.parametrize("source", ["builtin", "external"])
|
||||
def test_plugin_load_failure_logs_no_exception_text(monkeypatch, level, source) -> None:
|
||||
"""A failing plugin's exception message and class name stay out of every record."""
|
||||
records: list[logging.LogRecord] = []
|
||||
|
||||
class _Collect(logging.Handler):
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
records.append(record)
|
||||
|
||||
error = _MALFORMED_ERROR("cannot open postgres://user:CRED_CANARY@db/learn")
|
||||
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
||||
if source == "builtin":
|
||||
real_import = importlib.import_module
|
||||
|
||||
def _import(name: str, *args, **kwargs): # noqa: ANN002, ANN003, ANN202
|
||||
if name == "headroom.learn.plugins.grok":
|
||||
raise error
|
||||
return real_import(name, *args, **kwargs)
|
||||
|
||||
monkeypatch.setattr("headroom.learn.registry.importlib.import_module", _import)
|
||||
else:
|
||||
broken_ep = MagicMock()
|
||||
broken_ep.name = "broken"
|
||||
broken_ep.load.side_effect = error
|
||||
monkeypatch.setattr("importlib.metadata.entry_points", lambda group: [broken_ep])
|
||||
|
||||
logger = logging.getLogger("headroom.learn.registry")
|
||||
handler = _Collect(level=logging.DEBUG)
|
||||
previous = logger.level
|
||||
logger.addHandler(handler)
|
||||
logger.setLevel(level)
|
||||
try:
|
||||
reset_registry()
|
||||
assert "claude" in get_registry()
|
||||
finally:
|
||||
logger.removeHandler(handler)
|
||||
logger.setLevel(previous)
|
||||
reset_registry()
|
||||
|
||||
warnings = [r for r in records if r.levelno == logging.WARNING]
|
||||
assert len(warnings) == 1
|
||||
assert ("grok" if source == "builtin" else "broken") in warnings[0].getMessage()
|
||||
for record in records:
|
||||
rendered = logging.Formatter().format(record)
|
||||
assert "CRED_CANARY" not in rendered
|
||||
assert "PRIVATE_CREDENTIAL_CANARY" not in rendered
|
||||
assert "FORGED_LOG_ENTRY" not in rendered
|
||||
assert "\n" not in rendered
|
||||
|
||||
|
||||
class TestResetRegistry:
|
||||
def test_reset_clears_cache(self):
|
||||
reg1 = get_registry()
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import threading
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
@@ -182,3 +183,152 @@ async def test_wait_for_task_timeout_does_not_cancel_background_write() -> None:
|
||||
"status": "completed",
|
||||
"result": True,
|
||||
}
|
||||
|
||||
|
||||
class _RaisingClient:
|
||||
def __init__(self, exc: Exception) -> None:
|
||||
self._exc = exc
|
||||
|
||||
def delete(self, memory_id: str) -> None:
|
||||
raise self._exc
|
||||
|
||||
def get(self, memory_id: str) -> None:
|
||||
raise self._exc
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("op", ["get_memory", "delete_memory"])
|
||||
@pytest.mark.parametrize("level", [logging.INFO, logging.DEBUG])
|
||||
async def test_malformed_exception_names_cannot_leak_from_memory_operations(
|
||||
op, level, monkeypatch, caplog
|
||||
) -> None:
|
||||
error_type = type("PRIVATE_CREDENTIAL_CANARY\nFORGED_LOG_ENTRY", (RuntimeError,), {})
|
||||
adapter = _adapter()
|
||||
adapter._ensure_initialized = AsyncMock()
|
||||
adapter._mem0_client = _RaisingClient(error_type("backend content"))
|
||||
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
||||
with caplog.at_level(level, logger="headroom.memory.backends.direct_mem0"):
|
||||
result = await getattr(adapter, op)("m1")
|
||||
assert result is (None if op == "get_memory" else False)
|
||||
assert len([record for record in caplog.records if record.levelno == logging.WARNING]) == 1
|
||||
for record in caplog.records:
|
||||
rendered = logging.Formatter().format(record)
|
||||
assert "PRIVATE_CREDENTIAL_CANARY" not in rendered
|
||||
assert "FORGED_LOG_ENTRY" not in rendered
|
||||
assert "\n" not in rendered
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
("op", "exc", "warns"),
|
||||
[
|
||||
("delete_memory", ValueError("Memory with id m1\nFAKE LOG LINE not found"), False),
|
||||
("delete_memory", ValueError("payload user:secret rejected"), True),
|
||||
("delete_memory", ValueError("Collection mem0 not found"), True),
|
||||
("delete_memory", ValueError("Memory with id other-id not found"), True),
|
||||
("delete_memory", RuntimeError("qdrant at user:secret@host down"), True),
|
||||
("get_memory", RuntimeError("qdrant at user:secret@host down"), True),
|
||||
],
|
||||
)
|
||||
async def test_backend_failures_warn_without_exception_text(
|
||||
op: str, exc: Exception, warns: bool
|
||||
) -> None:
|
||||
"""A missing id on delete is expected (DEBUG); other failures warn with the
|
||||
exception type only, never its text."""
|
||||
adapter = _adapter()
|
||||
adapter._ensure_initialized = AsyncMock() # type: ignore[method-assign]
|
||||
adapter._mem0_client = _RaisingClient(exc)
|
||||
|
||||
records: list[logging.LogRecord] = []
|
||||
|
||||
class _Collect(logging.Handler):
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
records.append(record)
|
||||
|
||||
logger = logging.getLogger("headroom.memory.backends.direct_mem0")
|
||||
handler = _Collect(level=logging.DEBUG)
|
||||
previous = logger.level
|
||||
logger.addHandler(handler)
|
||||
logger.setLevel(logging.DEBUG)
|
||||
try:
|
||||
# Model-supplied id with a newline: must not split the log line.
|
||||
result = await getattr(adapter, op)("m1\nFAKE LOG LINE")
|
||||
finally:
|
||||
logger.removeHandler(handler)
|
||||
logger.setLevel(previous)
|
||||
|
||||
assert not result
|
||||
# No level may carry the exception text, rendered as a handler would.
|
||||
for record in records:
|
||||
assert "secret" not in logging.Formatter().format(record)
|
||||
warnings = [r for r in records if r.levelname == "WARNING"]
|
||||
if warns:
|
||||
assert len(warnings) == 1
|
||||
assert type(exc).__name__ in warnings[0].getMessage()
|
||||
assert "secret" not in warnings[0].getMessage()
|
||||
assert warnings[0].exc_info is None
|
||||
assert "\n" not in warnings[0].getMessage()
|
||||
else:
|
||||
assert warnings == []
|
||||
assert any("not found" in r.getMessage() for r in records)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_huge_model_supplied_id_is_capped_in_the_warning() -> None:
|
||||
"""A model can invent an id of any size; the warning must stay bounded."""
|
||||
adapter = _adapter()
|
||||
adapter._ensure_initialized = AsyncMock() # type: ignore[method-assign]
|
||||
adapter._mem0_client = _RaisingClient(RuntimeError("backend down"))
|
||||
|
||||
records: list[logging.LogRecord] = []
|
||||
|
||||
class _Collect(logging.Handler):
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
records.append(record)
|
||||
|
||||
logger = logging.getLogger("headroom.memory.backends.direct_mem0")
|
||||
handler = _Collect(level=logging.WARNING)
|
||||
logger.addHandler(handler)
|
||||
try:
|
||||
assert await adapter.delete_memory("x" * 100_000) is False
|
||||
assert await adapter.delete_memory("short-id") is False
|
||||
finally:
|
||||
logger.removeHandler(handler)
|
||||
|
||||
huge, short = (r.getMessage() for r in records)
|
||||
assert len(huge) < 200
|
||||
assert "…(+" in huge
|
||||
assert "'short-id'" in short and "…" not in short
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("level", [logging.INFO, logging.DEBUG])
|
||||
async def test_get_memory_survives_an_out_of_range_errno(level: int) -> None:
|
||||
"""An OSError whose errno does not fit a C int must not escape get_memory,
|
||||
whether or not DEBUG (which builds the exception description) is on."""
|
||||
errno_value = 10 ** len("x" * 10000) # built at runtime, far past any C int
|
||||
adapter = _adapter()
|
||||
adapter._ensure_initialized = AsyncMock() # type: ignore[method-assign]
|
||||
adapter._mem0_client = _RaisingClient(OSError(errno_value, "backend down"))
|
||||
|
||||
logger = logging.getLogger("headroom.memory.backends.direct_mem0")
|
||||
records: list[logging.LogRecord] = []
|
||||
|
||||
class _Collect(logging.Handler):
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
records.append(record)
|
||||
|
||||
handler = _Collect(level=logging.DEBUG)
|
||||
previous = logger.level
|
||||
logger.addHandler(handler)
|
||||
logger.setLevel(level)
|
||||
try:
|
||||
assert await adapter.get_memory("m1") is None
|
||||
finally:
|
||||
logger.removeHandler(handler)
|
||||
logger.setLevel(previous)
|
||||
|
||||
assert [r.levelno for r in records if r.levelno == logging.WARNING] == [logging.WARNING]
|
||||
assert any(r.levelno == logging.DEBUG for r in records) is (level == logging.DEBUG)
|
||||
for record in records:
|
||||
logging.Formatter().format(record)
|
||||
|
||||
@@ -1686,6 +1686,38 @@ class TestHydrateEdgeCases:
|
||||
assert learner._saved_hashes == set()
|
||||
assert learner._persisted_ids == {}
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_unreadable_db_warns_on_hydration(self, tmp_path):
|
||||
"""A memory.db that cannot be read must say so; a missing one stays quiet."""
|
||||
import logging
|
||||
|
||||
records: list[logging.LogRecord] = []
|
||||
|
||||
class _Collect(logging.Handler):
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
records.append(record)
|
||||
|
||||
handler = _Collect(level=logging.DEBUG)
|
||||
logger = logging.getLogger("headroom.memory.traffic_learner")
|
||||
logger.addHandler(handler)
|
||||
try:
|
||||
await TrafficLearner(
|
||||
backend=_FakeBackend(tmp_path / "not-there.db"), min_evidence=1
|
||||
)._hydrate_persisted_state()
|
||||
assert [r for r in records if r.levelno >= logging.WARNING] == []
|
||||
|
||||
bad = tmp_path / "memory.db"
|
||||
bad.write_bytes(b"this is not a sqlite database" * 100)
|
||||
learner = TrafficLearner(backend=_FakeBackend(bad), min_evidence=1)
|
||||
await learner._hydrate_persisted_state()
|
||||
warnings = [r for r in records if r.levelno == logging.WARNING]
|
||||
assert len(warnings) == 1
|
||||
assert "hydrate" in warnings[0].getMessage()
|
||||
assert str(bad) in warnings[0].getMessage()
|
||||
assert learner._saved_hashes == set()
|
||||
finally:
|
||||
logger.removeHandler(handler)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_hydration_is_bounded_to_dedup_window(self, tmp_path):
|
||||
"""A persisted history larger than dedup_window must not start the
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import weakref
|
||||
from collections.abc import Callable, Iterator
|
||||
from dataclasses import dataclass
|
||||
@@ -168,6 +169,116 @@ def test_create_storage_builtin_entrypoint_and_fallback(
|
||||
plain.close()
|
||||
|
||||
|
||||
class _ListHandler(logging.Handler):
|
||||
def __init__(self) -> None:
|
||||
super().__init__(level=logging.DEBUG)
|
||||
self.records: list[logging.LogRecord] = []
|
||||
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
self.records.append(record)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def storage_log() -> Iterator[_ListHandler]:
|
||||
handler = _ListHandler()
|
||||
logger = logging.getLogger("headroom.storage")
|
||||
previous = logger.level
|
||||
logger.addHandler(handler)
|
||||
logger.setLevel(logging.DEBUG)
|
||||
yield handler
|
||||
logger.removeHandler(handler)
|
||||
logger.setLevel(previous)
|
||||
|
||||
|
||||
def _fallback_warnings(handler: _ListHandler) -> list[str]:
|
||||
return [
|
||||
r.getMessage()
|
||||
for r in handler.records
|
||||
if r.levelno == logging.WARNING and "falling back to SQLite" in r.getMessage()
|
||||
]
|
||||
|
||||
|
||||
def test_create_storage_warns_when_a_backend_plugin_fails_to_load(
|
||||
monkeypatch: pytest.MonkeyPatch, storage_log: _ListHandler
|
||||
) -> None:
|
||||
monkeypatch.setattr("headroom.storage.SQLiteStorage", lambda db_path: SimpleNamespace())
|
||||
|
||||
created = DummyStorage()
|
||||
monkeypatch.setattr(
|
||||
"importlib.metadata.entry_points",
|
||||
lambda group: [SimpleNamespace(name="custom", load=lambda: lambda url: created)],
|
||||
)
|
||||
assert create_storage("custom://user:secret@host/db") is created
|
||||
assert _fallback_warnings(storage_log) == []
|
||||
|
||||
monkeypatch.setattr(
|
||||
"importlib.metadata.entry_points",
|
||||
lambda group: (_ for _ in ()).throw(RuntimeError("boom")),
|
||||
)
|
||||
create_storage("custom://user:secret@host/db")
|
||||
warnings = _fallback_warnings(storage_log)
|
||||
assert len(warnings) == 1
|
||||
assert "'custom'" in warnings[0]
|
||||
assert "secret" not in warnings[0]
|
||||
|
||||
monkeypatch.setattr("importlib.metadata.entry_points", lambda group: [])
|
||||
create_storage("other://host/db")
|
||||
assert any("'other'" in w for w in _fallback_warnings(storage_log))
|
||||
|
||||
def _rejecting_factory(url: str) -> DummyStorage:
|
||||
raise ValueError(f"cannot connect to {url} while storing CONTENT_CANARY")
|
||||
|
||||
monkeypatch.setattr(
|
||||
"importlib.metadata.entry_points",
|
||||
lambda group: [SimpleNamespace(name="leaky", load=lambda: _rejecting_factory)],
|
||||
)
|
||||
storage_log.records.clear()
|
||||
create_storage("leaky://user:secret@host/db")
|
||||
warning_records = [r for r in storage_log.records if r.levelno == logging.WARNING]
|
||||
assert len(warning_records) == 1
|
||||
assert "ValueError" in warning_records[0].getMessage()
|
||||
assert "secret" not in warning_records[0].getMessage()
|
||||
assert warning_records[0].exc_info is None
|
||||
# Every level, rendered the way a log handler would (exc_info included).
|
||||
assert storage_log.records
|
||||
for record in storage_log.records:
|
||||
rendered = logging.Formatter().format(record)
|
||||
assert "secret" not in rendered
|
||||
assert "CONTENT_CANARY" not in rendered
|
||||
debug = [r for r in storage_log.records if r.levelno == logging.DEBUG]
|
||||
assert debug and "ValueError" in logging.Formatter().format(debug[0])
|
||||
|
||||
storage_log.records.clear()
|
||||
create_storage("metrics.db")
|
||||
assert _fallback_warnings(storage_log) == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("level", [logging.INFO, logging.DEBUG])
|
||||
def test_storage_plugin_class_name_cannot_leak_or_forge_log_lines(
|
||||
monkeypatch: pytest.MonkeyPatch, storage_log: _ListHandler, level: int
|
||||
) -> None:
|
||||
error_type = type("PRIVATE_CREDENTIAL_CANARY\nFORGED_LOG_ENTRY", (RuntimeError,), {})
|
||||
|
||||
def _rejecting_factory(url: str) -> DummyStorage:
|
||||
raise error_type("plugin text")
|
||||
|
||||
fallback = SimpleNamespace()
|
||||
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
||||
monkeypatch.setattr("headroom.storage.SQLiteStorage", lambda db_path: fallback)
|
||||
monkeypatch.setattr(
|
||||
"importlib.metadata.entry_points",
|
||||
lambda group: [SimpleNamespace(name="plugin", load=lambda: _rejecting_factory)],
|
||||
)
|
||||
logging.getLogger("headroom.storage").setLevel(level)
|
||||
assert create_storage("plugin://host/db") is fallback
|
||||
assert len(_fallback_warnings(storage_log)) == 1
|
||||
for record in storage_log.records:
|
||||
rendered = logging.Formatter().format(record)
|
||||
assert "PRIVATE_CREDENTIAL_CANARY" not in rendered
|
||||
assert "FORGED_LOG_ENTRY" not in rendered
|
||||
assert "\n" not in rendered
|
||||
|
||||
|
||||
def test_jsonl_storage_round_trip_query_count_and_summary(tmp_path: Path) -> None:
|
||||
storage = JSONLStorage(str(tmp_path / "metrics.jsonl"))
|
||||
now = datetime(2026, 4, 23, 12, 0, 0)
|
||||
|
||||
Reference in New Issue
Block a user