mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description When `--memory` is enabled, memory backend initialization or database access failures (such as `sqlite3.OperationalError: unable to open database file` on bind mounts or invalid SQLite stores) caused request handlers to raise unhandled exceptions and abort client requests with HTTP 500. This change ensures that memory lookup and backend initialization failures degrade gracefully by logging a warning and continuing request processing without memory context (`memory_context = None`). Closes #3251 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - In `headroom/proxy/memory_handler.py:search_and_format_context`: wrapped `_ensure_initialized()` and `_resolve_for_request()` in a `try...except Exception` block, logging a warning and returning `None` on failure. - In `headroom/proxy/handlers/anthropic.py:handle_anthropic_messages`: added an `except Exception as e:` handler for memory context lookup to prevent lookup errors from raising unhandled exceptions out of the request handler. - In `headroom/proxy/handlers/openai.py`: added `except Exception as e:` handlers in Responses API (`handle_openai_responses`) and WebSocket (`handle_realtime_websocket`) context search blocks. - Added regression tests in `tests/test_anthropic_pre_upstream_backpressure.py` and `tests/test_memory_handler_native_ops.py`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text tests/test_anthropic_pre_upstream_backpressure.py .. [100%] 2 passed in 1.56s tests/test_memory_handler_native_ops.py .................................. [100%] 34 passed in 1.57s ``` ## Real Behavior Proof - Environment: Linux x86_64, Python 3.14 - Exact command / steps: `pytest -q tests/test_anthropic_pre_upstream_backpressure.py -k test_memory_context && pytest -q tests/test_memory_handler_native_ops.py` - Observed result: Verified `test_memory_context_exception_fails_open_and_releases_semaphore` and `test_search_and_format_context_degrades_gracefully_on_backend_error` pass cleanly. - Not tested: None ## Runtime Rollout Safety - Rollout-managed feature(s): None - Minimum rollout channel: stable - Stable/default behavior changed: No (gracefully falls back when memory store is unavailable rather than returning 500) - Kill switch / disable path: Disable memory with `--memory false` or omitting `--memory` - Unsafe override required: None - Qualification impact: None - Rollback path: git revert ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did not edit `CHANGELOG.md` (release-please handles this) ## Additional Notes Ensures optional memory subsystem failure does not take down core proxy traffic. ## Current maintainer verification (October 9 UTC) Exact synchronized head: `b222647f31e0fbdfd8e33279f4c3c21e3cd4e40f`, containing main `01d9194854233ef6ed04005cf0a5b8c32bf65d6d` through a clean merge. The updated WebSocket regression gates each response pair on its matching upstream request and keeps the client open through both completions; the handler call is bounded by a five-second timeout. No contributor logic was changed during synchronization. Both affected routing/WebSocket files pass: 96 tests on Windows/Python 3.12 with the compatible existing native extension. Additional reviewer probes pass with deliberately yielding upstream sends at 0, 1, and 10 ms, addressing the reported scheduling race. These are local mocked-socket tests; no live external-provider or client session is claimed. GitHub fork-workflow approval was provided for this reviewed head; Current-head hosted snapshot completed with 30 successful checks, 12 skips, and one failed test shard. The failed per-frame memory recovery test is a PR regression: this file has 1 failure/10 passes locally on the PR, versus all 11 passes on main. Failed lookup currently adds memory instructions instead of preserving the original WebSocket frame; the new maintainer review requests that correction. The 96 routing/lifecycle tests and three scheduling probes above remain passing evidence with this explicit limit. ### Maintainer current-main requalification Exact head `3ca4324e69bd46a1c07288fb1b17c12b6fbcc9a7`, upstream main `2d3ee51846435ef5b320eea4806af14e602b0d8c`. Synchronized with upstream main2d3ee51846. The memory-handler conflict retains main's unresolved-project early return before the PR's initialization/resolve failure boundary. The test conflict retains main's streaming-tool regression and the PR's independent lookup-failure regression as complete separate functions. No contributor failure policy rewritten or main project-isolation guard dropped. The author fix addresses the previous WebSocket finding: the inner lookup catch no longer allows later augmentation after failure, so the outer frame boundary forwards the original raw frame and later frames can retry. The existing per-frame original-request/recovery regression now passes. All 205 focused HTTP, WebSocket, native-memory-operation and backpressure tests pass; current author/Devin discussion reviewed, including the restored deterministic coordination and five-second test deadline. Changed Python lint/format and main-relative whitespace pass. Python uses the existing native extension for startup, not a fresh native build. No live memory-cloud or model-provider claim. No remaining finding in the reviewed failure/recovery scope; formal approval awaits exact-head hosted qualification, handled by maintainers. Nothing merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. ### Maintainer current-main requalification Exact head `742b5aeeb1e5c75cfa6a8b51d9b69d0a3fef36c8`, upstream main `c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`. Clean mechanical synchronization with upstream mainc01cf07d94. Re-reviewed memory initialization/lookup failure boundaries and current author/Devin discussion. Main's unresolved-project early return remains before backend initialization and resolution. Failed WebSocket lookup reaches the outer fallback before augmentation, preserving the original raw frame while later frames can retry. The previously fixed deterministic two-turn coordination and five-second test deadline remain intact. No contributor failure/isolation policy rewritten. Fresh HTTP/WebSocket/native-memory-operation/backpressure verification: 205 passed, one test-client deprecation warning. All seven changed Python files pass Ruff lint/format and main-relative whitespace checks; owned tree clean. Current-main native DLL supports local startup; no live cloud/model-provider or branch-specific native-build claim. No remaining finding in this failure/recovery scope; formal approval awaits exact-head hosted qualification. Maintainers handle hosted checks; nothing merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
JerrettDavis
parent
cde584d7fe
commit
00d4f441d3
@@ -3226,6 +3226,12 @@ class AnthropicHandlerMixin:
|
||||
f"{self.config.anthropic_pre_upstream_memory_context_timeout_seconds:.1f}s; "
|
||||
"continuing without it"
|
||||
)
|
||||
except Exception as e:
|
||||
memory_context = None
|
||||
logger.warning(
|
||||
f"[{request_id}] Memory: Context lookup failed: {e}; "
|
||||
"continuing without it"
|
||||
)
|
||||
try:
|
||||
if memory_context:
|
||||
from headroom.proxy.helpers import (
|
||||
|
||||
@@ -6647,6 +6647,11 @@ class OpenAIHandlerMixin:
|
||||
f"[{request_id}] Memory context lookup exceeded "
|
||||
f"{RESPONSES_CONTEXT_SEARCH_TIMEOUT_SECONDS:.1f}s; continuing without it"
|
||||
)
|
||||
except Exception as e:
|
||||
memory_context = None
|
||||
logger.warning(
|
||||
f"[{request_id}] Memory context lookup failed: {e}; continuing without it"
|
||||
)
|
||||
if memory_context:
|
||||
from headroom.proxy.helpers import (
|
||||
append_text_to_latest_user_input_item,
|
||||
|
||||
@@ -782,11 +782,19 @@ class MemoryHandler:
|
||||
|
||||
if self.is_project_unresolved(request_context):
|
||||
return None
|
||||
await self._ensure_initialized()
|
||||
if not self._backend:
|
||||
return None
|
||||
try:
|
||||
await self._ensure_initialized()
|
||||
if not self._backend:
|
||||
return None
|
||||
|
||||
backend, scope, effective_user_id = self._resolve_for_request(user_id, request_context)
|
||||
backend, scope, effective_user_id = self._resolve_for_request(user_id, request_context)
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
"Memory: Failed to initialize or resolve backend for user %s: %s",
|
||||
user_id,
|
||||
e,
|
||||
)
|
||||
return None
|
||||
|
||||
# Fail-closed when the router was unable to resolve a project in
|
||||
# PROJECT mode and `unresolved_project_fallback="empty"` (the
|
||||
|
||||
@@ -736,6 +736,52 @@ def test_memory_context_timeout_fails_open_and_releases_semaphore():
|
||||
anyio.run(_run)
|
||||
|
||||
|
||||
def test_memory_context_exception_fails_open_and_releases_semaphore():
|
||||
class _MemoryHandler:
|
||||
def __init__(self) -> None:
|
||||
self.config = SimpleNamespace(inject_context=True, inject_tools=False)
|
||||
self.initialized = False
|
||||
self.backend = None
|
||||
|
||||
async def search_and_format_context(self, _user_id, _messages, **_kwargs):
|
||||
import sqlite3
|
||||
|
||||
raise sqlite3.OperationalError("unable to open database file")
|
||||
|
||||
def inject_tools(self, tools, _provider):
|
||||
return tools, False
|
||||
|
||||
def get_beta_headers(self) -> dict[str, str]:
|
||||
return {}
|
||||
|
||||
def has_memory_tool_calls(self, _response, _provider) -> bool:
|
||||
return False
|
||||
|
||||
async def handle_memory_tool_calls(self, _response, _user_id, _provider, **_kwargs):
|
||||
return []
|
||||
|
||||
async def _run() -> None:
|
||||
sem = asyncio.Semaphore(1)
|
||||
handler = _DummyAnthropicHandler(anthropic_pre_upstream_sem=sem)
|
||||
handler.memory_handler = _MemoryHandler()
|
||||
req = _build_request(
|
||||
{
|
||||
"model": "claude-3-5-sonnet-latest",
|
||||
"messages": [{"role": "user", "content": "hello"}],
|
||||
},
|
||||
{
|
||||
"authorization": "Bearer sk-ant-api-test",
|
||||
"x-headroom-user-id": "user-1",
|
||||
},
|
||||
)
|
||||
response = await handler.handle_anthropic_messages(req)
|
||||
assert response.status_code == 200
|
||||
assert sem._value == 1
|
||||
|
||||
with _tokenizer_patch():
|
||||
anyio.run(_run)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# /livez stays fast under Anthropic pre-upstream contention. #
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
@@ -1807,3 +1807,17 @@ async def test_memory_list_dispatched_via_execute_memory_tool(handler: MemoryHan
|
||||
payload = json.loads(out)
|
||||
assert payload["status"] == "ok"
|
||||
assert payload["memories"][0]["id"] == "m1"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_and_format_context_degrades_gracefully_on_backend_error(
|
||||
handler: MemoryHandler, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
import sqlite3
|
||||
|
||||
async def fail_init() -> None:
|
||||
raise sqlite3.OperationalError("unable to open database file")
|
||||
|
||||
monkeypatch.setattr(handler, "_ensure_initialized", fail_init)
|
||||
result = await handler.search_and_format_context("u1", [{"role": "user", "content": "hello"}])
|
||||
assert result is None
|
||||
|
||||
@@ -1094,3 +1094,69 @@ def test_responses_savings_accept_a_session_header_as_identity(monkeypatch):
|
||||
_post_responses(handler, body, {"session_id": "sess-1"})
|
||||
_post_responses(handler, body, {"session_id": "sess-2"})
|
||||
assert _booked(handler) == [100, 0, 100]
|
||||
|
||||
|
||||
def test_handle_openai_responses_memory_lookup_exception_fails_open_and_forwards_original_request(
|
||||
monkeypatch,
|
||||
):
|
||||
import sqlite3
|
||||
|
||||
class _FailingContextMemoryHandler:
|
||||
def __init__(self) -> None:
|
||||
self.config = SimpleNamespace(
|
||||
inject_context=True,
|
||||
inject_tools=False,
|
||||
project_root_override=str(Path(".").resolve()),
|
||||
)
|
||||
self.call_count = 0
|
||||
|
||||
async def search_and_format_context(self, user_id, messages, **kwargs):
|
||||
self.call_count += 1
|
||||
raise sqlite3.OperationalError("unable to open database file")
|
||||
|
||||
def compute_memory_tool_definitions(self, provider: str) -> list[dict]:
|
||||
return []
|
||||
|
||||
def has_memory_tool_calls(self, response, provider: str) -> bool:
|
||||
return False
|
||||
|
||||
handler = _DummyOpenAIHandler()
|
||||
memory_handler = _FailingContextMemoryHandler()
|
||||
handler.memory_handler = memory_handler
|
||||
handler.session_tracker_store = SimpleNamespace(
|
||||
compute_session_id=lambda *a, **k: "sess-api-memory-fallback",
|
||||
)
|
||||
|
||||
monkeypatch.setattr("headroom.tokenizers.get_tokenizer", lambda model: _DummyTokenizer())
|
||||
|
||||
req1 = _build_request(
|
||||
{"model": "gpt-4o-mini", "input": "turn 1 original", "store": False},
|
||||
{
|
||||
"Authorization": "Bearer sk-test",
|
||||
"User-Agent": "codex-cli/0.5",
|
||||
"x-headroom-user-id": "user-1",
|
||||
},
|
||||
)
|
||||
req2 = _build_request(
|
||||
{"model": "gpt-4o-mini", "input": "turn 2 original", "store": False},
|
||||
{
|
||||
"Authorization": "Bearer sk-test",
|
||||
"User-Agent": "codex-cli/0.5",
|
||||
"x-headroom-user-id": "user-1",
|
||||
},
|
||||
)
|
||||
|
||||
# First request: lookup fails, original body forwarded without memory, response succeeds
|
||||
res1 = anyio.run(handler.handle_openai_responses, req1)
|
||||
assert res1.status_code == 200
|
||||
assert handler.captured_request is not None
|
||||
_, _, _, body1 = handler.captured_request
|
||||
assert body1["input"] == "turn 1 original"
|
||||
|
||||
# Subsequent request: handler continues serving traffic normally
|
||||
res2 = anyio.run(handler.handle_openai_responses, req2)
|
||||
assert res2.status_code == 200
|
||||
assert handler.captured_request is not None
|
||||
_, _, _, body2 = handler.captured_request
|
||||
assert body2["input"] == "turn 2 original"
|
||||
assert memory_handler.call_count == 2
|
||||
|
||||
@@ -2203,6 +2203,113 @@ async def test_ws_memory_enabled_non_memory_response_streams_completion():
|
||||
assert len(upstream.sent) == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ws_memory_context_lookup_exception_fails_open_and_survives_subsequent_traffic():
|
||||
import sqlite3
|
||||
|
||||
class _FailingWsContextMemoryHandler(_MemoryWsHandler):
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
self.config.inject_context = True
|
||||
self.config.inject_tools = False
|
||||
self.call_count = 0
|
||||
|
||||
async def search_and_format_context(self, user_id, messages, **kwargs):
|
||||
self.call_count += 1
|
||||
raise sqlite3.OperationalError("unable to open database file")
|
||||
|
||||
first_frame = json.dumps(
|
||||
{
|
||||
"type": "response.create",
|
||||
"response": {
|
||||
"model": "gpt-5.4",
|
||||
"input": "turn 1 original",
|
||||
},
|
||||
}
|
||||
)
|
||||
second_frame = json.dumps(
|
||||
{
|
||||
"type": "response.create",
|
||||
"response": {
|
||||
"model": "gpt-5.4",
|
||||
"input": "turn 2 original",
|
||||
},
|
||||
}
|
||||
)
|
||||
turn1_events = [
|
||||
json.dumps({"type": "response.created", "response": {"id": "r-1"}}),
|
||||
json.dumps({"type": "response.completed", "response": {"id": "r-1"}}),
|
||||
]
|
||||
turn2_events = [
|
||||
json.dumps({"type": "response.created", "response": {"id": "r-2"}}),
|
||||
json.dumps({"type": "response.completed", "response": {"id": "r-2"}}),
|
||||
]
|
||||
expected_events = turn1_events + turn2_events
|
||||
|
||||
first_sent = asyncio.Event()
|
||||
second_sent = asyncio.Event()
|
||||
turn1_done = asyncio.Event()
|
||||
turn2_done = asyncio.Event()
|
||||
both_completed = asyncio.Event()
|
||||
|
||||
class _CoordinatedUpstream(_FakeUpstream):
|
||||
async def send(self, payload: str) -> None:
|
||||
self.sent.append(payload)
|
||||
if len(self.sent) == 1:
|
||||
first_sent.set()
|
||||
elif len(self.sent) == 2:
|
||||
second_sent.set()
|
||||
|
||||
async def _iter(self):
|
||||
await first_sent.wait()
|
||||
for ev in turn1_events:
|
||||
yield ev
|
||||
await second_sent.wait()
|
||||
for ev in turn2_events:
|
||||
yield ev
|
||||
await both_completed.wait()
|
||||
|
||||
class _CoordinatedClientWebSocket(_FakeWebSocket):
|
||||
async def send_text(self, text: str) -> None:
|
||||
await super().send_text(text)
|
||||
if '"id": "r-1"' in text and "response.completed" in text:
|
||||
turn1_done.set()
|
||||
elif '"id": "r-2"' in text and "response.completed" in text:
|
||||
turn2_done.set()
|
||||
both_completed.set()
|
||||
|
||||
async def receive_text(self) -> str:
|
||||
if not first_sent.is_set():
|
||||
return first_frame
|
||||
await turn1_done.wait()
|
||||
if not second_sent.is_set():
|
||||
return second_frame
|
||||
await turn2_done.wait()
|
||||
raise _FakeWebSocketDisconnect("client closed")
|
||||
|
||||
upstream = _CoordinatedUpstream([])
|
||||
fake_ws_mod = _make_fake_websockets_module(upstream)
|
||||
client_ws = _CoordinatedClientWebSocket([])
|
||||
client_ws.headers["x-headroom-user-id"] = "user-1"
|
||||
handler = _DummyOpenAIHandler()
|
||||
memory_handler = _FailingWsContextMemoryHandler()
|
||||
handler.memory_handler = memory_handler
|
||||
|
||||
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
|
||||
await asyncio.wait_for(handler.handle_openai_responses_ws(client_ws), timeout=5.0)
|
||||
|
||||
# Both turns must reach upstream without injected memory
|
||||
assert len(upstream.sent) == 2
|
||||
sent0 = json.loads(upstream.sent[0])["response"]
|
||||
sent1 = json.loads(upstream.sent[1])["response"]
|
||||
assert sent0["input"] == "turn 1 original"
|
||||
assert sent1["input"] == "turn 2 original"
|
||||
|
||||
# Streaming socket connection survived and client received all upstream events
|
||||
assert client_ws.sent_text == expected_events
|
||||
assert memory_handler.call_count == 2
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ws_late_memory_call_after_streamed_message_passes_through():
|
||||
message_item = {
|
||||
|
||||
Reference in New Issue
Block a user