mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description
`MemoryHandler._ensure_initialized` documents a "fail-open contract — no
exception propagates to request handlers", but only
`asyncio.TimeoutError` and `asyncio.CancelledError` are caught:
```python
try:
await asyncio.wait_for(_do_init(), timeout=STARTUP_INIT_TIMEOUT_SECONDS)
except asyncio.TimeoutError:
... # fail open
except asyncio.CancelledError:
... # reset + re-raise
```
Any *other* exception raised synchronously during backend init — e.g.
`sqlite3.OperationalError: unable to open database file` — escapes both
handlers, propagates up through `handle_anthropic_messages` →
`memory_handler` → `LocalBackend._init_locked` →
`SQLiteMemoryStore._init_db`, and **500s the whole request**.
This is exactly the failure in issue #3251: on the documented macOS
Docker Desktop `persistent-docker --memory` deploy, the per-project
SQLite memory store cannot be opened in WAL mode over the bind-mount (a
known Docker Desktop/VirtioFS + SQLite interaction), and **every**
request returns `500 Internal Server Error` — even though compression,
caching, and tool-search savings don't need memory at all. An optional
subsystem should never take down core proxy traffic; that is Ask #3 of
the issue ("a memory-store open failure on the request path should
degrade gracefully instead of 500ing").
The environmental root cause (bind-mount + WAL) is out of scope here;
this fixes the graceful-degradation gap so `--memory` being unavailable
disables memory context for the request rather than killing it.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/memory_handler.py`: added an `except Exception` branch
to `_ensure_initialized` that mirrors the timeout branch — best-effort
close the possibly-half-assigned backend, null it, leave
`_initialized=False` so a later request retries, log an error, and
**return** (fail open) instead of propagating. Ordering is preserved:
`TimeoutError` (an `Exception`) is still caught first, and
`CancelledError` (a `BaseException`, not an `Exception`) still falls
through to its own re-raising branch.
- `tests/test_memory_handler_concurrent_init.py`: added
`test_ensure_initialized_fails_open_on_backend_init_error` — a
`LocalBackend` whose init raises `sqlite3.OperationalError`; asserts
`_ensure_initialized` does not raise, cleans up the backend, stays
uninitialized, and retries on the next call.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
### Test Output
```text
tests/test_memory_handler_concurrent_init.py -> 9 passed in 1.80s
(the new test FAILS on pre-fix code — the OperationalError propagates — verified via git stash)
uvx ruff@0.16.2 check headroom/proxy/memory_handler.py tests/test_memory_handler_concurrent_init.py -> All checks passed!
uvx mypy@1.20.2 headroom/proxy/memory_handler.py -> Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: monkeypatched `LocalBackend` so its
`_ensure_initialized` raises `sqlite3.OperationalError("unable to open
database file")` (the #3251 failure). Before the fix, `await
handler._ensure_initialized()` re-raised the error (which on the real
path becomes a request 500). After the fix, it returns cleanly with
`_initialized=False`, `_backend=None`, the half-init backend `close()`d
once, and a later call retries.
- Observed result: a backend init failure no longer propagates; memory
is skipped for the request and retried later; timeout and cancellation
behavior are unchanged.
- Not tested: no live macOS Docker Desktop repro of the bind-mount/WAL
open failure (environment-specific); the fail-open path is exercised by
injecting the same exception the deploy hits.
## Runtime Rollout Safety
- Rollout-managed feature(s): none. This is the memory-subsystem
lazy-init on the proxy request path, not a rollout-channel-gated runtime
feature.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: only the failure path — a backend
that cannot initialize now disables memory for the request instead of
500ing it. Successful init, the timeout branch, and the cancellation
branch are unchanged.
- Kill switch / disable path: memory is already opt-in
(`MemoryConfig.enabled`); this makes a broken backend behave like
memory-off for affected requests.
- Unsafe override required: no.
- Qualification impact: none for healthy deployments; broken-memory
deployments go from all-requests-500 to requests-served-without-memory.
- Rollback path: revert this PR.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal behavior)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
Addresses Ask #3 of #3251 (graceful degradation). The environmental root
cause (Docker Desktop macOS bind-mount + SQLite WAL) and Asks #1/#2
(repro / named-volume preset) are deployment concerns left out of this
code fix.
Co-authored-by: JD Davis <mxjerrett@gmail.com>
357 lines
12 KiB
Python
357 lines
12 KiB
Python
"""Concurrency + timeout tests for MemoryHandler._ensure_initialized (Unit 1).
|
|
|
|
Covers:
|
|
- 10 concurrent first-callers trigger exactly one backend init
|
|
(singleflight via asyncio.Lock + double-check).
|
|
- Initialization timeout is surfaced via log + leaves _initialized=False so
|
|
a subsequent call can retry (fail-open contract).
|
|
- End-to-end real-backend sanity test (no monkeypatching of internals)
|
|
that exercises LocalBackend so the system-wide check is satisfied.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
import functools
|
|
import os
|
|
from typing import Any
|
|
from unittest.mock import patch
|
|
|
|
import pytest
|
|
|
|
from headroom.proxy.memory_handler import (
|
|
STARTUP_INIT_TIMEOUT_SECONDS,
|
|
MemoryConfig,
|
|
MemoryHandler,
|
|
)
|
|
from tests._skip_helpers import external_model_skip_reason
|
|
|
|
|
|
def skip_offline_model_failures(func):
|
|
"""Skip real-backend smoke tests when the local embedder cannot start offline."""
|
|
|
|
@functools.wraps(func)
|
|
async def wrapper(*args, **kwargs):
|
|
try:
|
|
return await func(*args, **kwargs)
|
|
except Exception as exc:
|
|
reason = external_model_skip_reason(exc)
|
|
if reason is not None:
|
|
pytest.skip(reason)
|
|
raise
|
|
|
|
return wrapper
|
|
|
|
|
|
# -------------------------------------------------------------------
|
|
# Singleflight under concurrent callers
|
|
# -------------------------------------------------------------------
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_concurrent_ensure_initialized_runs_init_once(tmp_path, monkeypatch):
|
|
hits = {"n": 0}
|
|
start_event = asyncio.Event()
|
|
release_event = asyncio.Event()
|
|
|
|
class FakeLocalBackend:
|
|
def __init__(self, config): # noqa: D401
|
|
self.config = config
|
|
|
|
async def _ensure_initialized(self) -> None:
|
|
hits["n"] += 1
|
|
start_event.set()
|
|
# Simulate a slow cold-start so concurrent callers pile up on
|
|
# the lock. Without singleflight, hits would exceed 1.
|
|
await release_event.wait()
|
|
|
|
async def close(self) -> None:
|
|
pass
|
|
|
|
import headroom.memory.backends.local as local_mod
|
|
|
|
monkeypatch.setattr(local_mod, "LocalBackend", FakeLocalBackend)
|
|
|
|
handler = MemoryHandler(
|
|
MemoryConfig(enabled=True, backend="local", db_path=str(tmp_path / "mem.db"))
|
|
)
|
|
|
|
async def caller() -> None:
|
|
await handler._ensure_initialized()
|
|
|
|
# Fire 10 concurrent first-callers.
|
|
tasks = [asyncio.create_task(caller()) for _ in range(10)]
|
|
# Let one task enter the critical section, then release it.
|
|
await start_event.wait()
|
|
release_event.set()
|
|
await asyncio.gather(*tasks)
|
|
|
|
assert hits["n"] == 1, f"expected 1 backend init under concurrency, got {hits['n']}"
|
|
assert handler._initialized is True
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_ensure_initialized_noop_when_disabled(tmp_path):
|
|
handler = MemoryHandler(
|
|
MemoryConfig(enabled=False, backend="local", db_path=str(tmp_path / "mem.db"))
|
|
)
|
|
await handler._ensure_initialized()
|
|
assert handler._initialized is False
|
|
assert handler._backend is None
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_ensure_initialized_fails_open_on_backend_init_error(tmp_path, monkeypatch):
|
|
"""A backend that cannot open must NOT propagate — memory is optional.
|
|
|
|
Regression (#3251): a SQLite ``unable to open database file`` on a Docker
|
|
Desktop macOS bind-mount escaped ``_ensure_initialized`` (which only caught
|
|
TimeoutError/CancelledError) and 500'd every request. It must fail open: log,
|
|
leave ``_initialized=False`` and ``_backend=None``, and let the request
|
|
proceed without memory.
|
|
"""
|
|
import sqlite3
|
|
|
|
closed = {"n": 0}
|
|
|
|
class BrokenLocalBackend:
|
|
def __init__(self, config):
|
|
self.config = config
|
|
|
|
async def _ensure_initialized(self) -> None:
|
|
# Mirrors the real failure: sqlite3.connect raising mid-init after
|
|
# the backend object has already been assigned to self._backend.
|
|
raise sqlite3.OperationalError("unable to open database file")
|
|
|
|
async def close(self) -> None:
|
|
closed["n"] += 1
|
|
|
|
import headroom.memory.backends.local as local_mod
|
|
|
|
monkeypatch.setattr(local_mod, "LocalBackend", BrokenLocalBackend)
|
|
|
|
handler = MemoryHandler(
|
|
MemoryConfig(enabled=True, backend="local", db_path=str(tmp_path / "mem.db"))
|
|
)
|
|
|
|
# Must not raise — the whole point.
|
|
await handler._ensure_initialized()
|
|
|
|
assert handler._initialized is False
|
|
assert handler._backend is None
|
|
# The half-assigned backend was cleaned up.
|
|
assert closed["n"] == 1
|
|
|
|
# A later call retries (and fails open again) rather than short-circuiting.
|
|
await handler._ensure_initialized()
|
|
assert handler._initialized is False
|
|
|
|
|
|
# -------------------------------------------------------------------
|
|
# Timeout fail-open
|
|
# -------------------------------------------------------------------
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_ensure_initialized_timeout_leaves_handler_unready(tmp_path, monkeypatch):
|
|
class HangingBackend:
|
|
def __init__(self, config):
|
|
self.config = config
|
|
|
|
async def _ensure_initialized(self) -> None:
|
|
# Hang forever — timeout must cancel this.
|
|
await asyncio.Event().wait()
|
|
|
|
async def close(self) -> None:
|
|
pass
|
|
|
|
import headroom.memory.backends.local as local_mod
|
|
|
|
monkeypatch.setattr(local_mod, "LocalBackend", HangingBackend)
|
|
|
|
handler = MemoryHandler(
|
|
MemoryConfig(enabled=True, backend="local", db_path=str(tmp_path / "mem.db"))
|
|
)
|
|
|
|
# Attach a handler directly to the module logger — caplog has trouble
|
|
# when third-party conftest monkeys with propagation settings.
|
|
import logging as _logging
|
|
|
|
mem_logger = _logging.getLogger("headroom.proxy.memory_handler")
|
|
captured: list[_logging.LogRecord] = []
|
|
|
|
class _ListHandler(_logging.Handler):
|
|
def emit(self, record: _logging.LogRecord) -> None:
|
|
captured.append(record)
|
|
|
|
handler_log = _ListHandler(level=_logging.ERROR)
|
|
mem_logger.addHandler(handler_log)
|
|
prev_level = mem_logger.level
|
|
mem_logger.setLevel(_logging.DEBUG)
|
|
try:
|
|
# Shrink the module-level timeout to keep the test fast.
|
|
with patch("headroom.proxy.memory_handler.STARTUP_INIT_TIMEOUT_SECONDS", 0.1):
|
|
await handler._ensure_initialized()
|
|
finally:
|
|
mem_logger.removeHandler(handler_log)
|
|
mem_logger.setLevel(prev_level)
|
|
|
|
# Fail-open: no exception, _initialized=False, error logged.
|
|
assert handler._initialized is False
|
|
found_timeout_log = any("timed out" in rec.getMessage().lower() for rec in captured)
|
|
assert found_timeout_log, (
|
|
f"expected 'timed out' log record; got: {[(r.levelname, r.getMessage()) for r in captured]}"
|
|
)
|
|
|
|
# Confirm the default constant is unchanged (sanity).
|
|
assert STARTUP_INIT_TIMEOUT_SECONDS == 30.0
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_ensure_initialized_timeout_nulls_partially_initialized_backend(
|
|
tmp_path, monkeypatch
|
|
):
|
|
"""If wait_for fires while _init_backend_locked has already set
|
|
``self._backend`` but before ``self._initialized = True``, the timeout
|
|
handler must null ``_backend``. Otherwise callers doing
|
|
``if self.memory_handler._backend:`` see a truthy-but-broken backend.
|
|
"""
|
|
|
|
close_hits = {"n": 0}
|
|
|
|
class SlowBackend:
|
|
def __init__(self, config):
|
|
self.config = config
|
|
|
|
async def _ensure_initialized(self) -> None:
|
|
# Hang long enough to blow the 0.01s timeout below.
|
|
await asyncio.sleep(5.0)
|
|
|
|
async def close(self) -> None:
|
|
close_hits["n"] += 1
|
|
|
|
import headroom.memory.backends.local as local_mod
|
|
|
|
monkeypatch.setattr(local_mod, "LocalBackend", SlowBackend)
|
|
|
|
handler = MemoryHandler(
|
|
MemoryConfig(enabled=True, backend="local", db_path=str(tmp_path / "mem.db"))
|
|
)
|
|
|
|
with patch("headroom.proxy.memory_handler.STARTUP_INIT_TIMEOUT_SECONDS", 0.01):
|
|
await handler._ensure_initialized()
|
|
|
|
# Both must be consistent after timeout.
|
|
assert handler._initialized is False
|
|
assert handler._backend is None
|
|
assert close_hits["n"] == 1
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_ensure_initialized_cancellation_propagates_and_resets_state(tmp_path, monkeypatch):
|
|
"""External cancellation of an in-flight ``_ensure_initialized`` must
|
|
propagate (CancelledError is BaseException — not a swallowable error)
|
|
and leave the handler in a clean state."""
|
|
|
|
close_hits = {"n": 0}
|
|
|
|
class HangingBackend:
|
|
def __init__(self, config):
|
|
self.config = config
|
|
|
|
async def _ensure_initialized(self) -> None:
|
|
await asyncio.Event().wait()
|
|
|
|
async def close(self) -> None:
|
|
close_hits["n"] += 1
|
|
|
|
import headroom.memory.backends.local as local_mod
|
|
|
|
monkeypatch.setattr(local_mod, "LocalBackend", HangingBackend)
|
|
|
|
handler = MemoryHandler(
|
|
MemoryConfig(enabled=True, backend="local", db_path=str(tmp_path / "mem.db"))
|
|
)
|
|
|
|
task = asyncio.create_task(handler._ensure_initialized())
|
|
# Give the task a tick to enter _init_backend_locked and assign _backend.
|
|
await asyncio.sleep(0.05)
|
|
task.cancel()
|
|
with pytest.raises(asyncio.CancelledError):
|
|
await task
|
|
|
|
assert handler._initialized is False
|
|
assert handler._backend is None
|
|
assert close_hits["n"] == 1
|
|
|
|
|
|
# -------------------------------------------------------------------
|
|
# Real backend init (no monkeypatching) — integration smoke test
|
|
# -------------------------------------------------------------------
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
@skip_offline_model_failures
|
|
async def test_real_localbackend_initializes_via_public_entrypoint(tmp_path):
|
|
"""End-to-end sanity check: the public ``ensure_initialized`` path works
|
|
against a real LocalBackend. This catches regressions where the new
|
|
asyncio.Lock layering (Unit 1) breaks the actual init chain."""
|
|
pytest.importorskip("sqlite3") # bundled with Python but be explicit
|
|
# Skip on environments without an embedder backend available.
|
|
try:
|
|
import onnxruntime # noqa: F401
|
|
except Exception:
|
|
try:
|
|
import sentence_transformers # noqa: F401
|
|
except Exception: # pragma: no cover - env-dependent
|
|
pytest.skip("No embedder backend available in this environment")
|
|
|
|
handler = MemoryHandler(
|
|
MemoryConfig(
|
|
enabled=True,
|
|
backend="local",
|
|
db_path=str(tmp_path / "mem.db"),
|
|
)
|
|
)
|
|
|
|
# Concurrent callers should still produce one initialized backend.
|
|
await asyncio.gather(*[handler.ensure_initialized() for _ in range(5)])
|
|
assert handler._initialized is True
|
|
assert handler._backend is not None
|
|
|
|
# warmup_embedder is best-effort; on a real backend it should succeed.
|
|
warmed = await handler.warmup_embedder()
|
|
if not warmed and os.environ.get("TRANSFORMERS_OFFLINE") == "1":
|
|
pytest.skip("Skipped because required Hugging Face model files are unavailable offline")
|
|
assert warmed is True
|
|
await handler.close()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_warmup_embedder_returns_false_without_backend():
|
|
handler = MemoryHandler(MemoryConfig(enabled=True, backend="local"))
|
|
# _initialized=False, _backend=None → no-op, no crash.
|
|
result = await handler.warmup_embedder()
|
|
assert result is False
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_warmup_embedder_swallows_exceptions():
|
|
handler = MemoryHandler(MemoryConfig(enabled=True, backend="local"))
|
|
handler._initialized = True
|
|
|
|
class HM:
|
|
class _E:
|
|
async def embed(self, _text: Any) -> Any:
|
|
raise RuntimeError("synthetic embedder failure")
|
|
|
|
_embedder = _E()
|
|
|
|
class Backend:
|
|
_hierarchical_memory = HM()
|
|
|
|
handler._backend = Backend()
|
|
# Must not raise; returns False.
|
|
result = await handler.warmup_embedder()
|
|
assert result is False
|