fix(memory): gracefully degrade on context lookup and initialization errors (#3251) (#4049)

## Description

When `--memory` is enabled, memory backend initialization or database
access failures (such as `sqlite3.OperationalError: unable to open
database file` on bind mounts or invalid SQLite stores) caused request
handlers to raise unhandled exceptions and abort client requests with
HTTP 500. This change ensures that memory lookup and backend
initialization failures degrade gracefully by logging a warning and
continuing request processing without memory context (`memory_context =
None`).

Closes #3251

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- In `headroom/proxy/memory_handler.py:search_and_format_context`:
wrapped `_ensure_initialized()` and `_resolve_for_request()` in a
`try...except Exception` block, logging a warning and returning `None`
on failure.
- In `headroom/proxy/handlers/anthropic.py:handle_anthropic_messages`:
added an `except Exception as e:` handler for memory context lookup to
prevent lookup errors from raising unhandled exceptions out of the
request handler.
- In `headroom/proxy/handlers/openai.py`: added `except Exception as e:`
handlers in Responses API (`handle_openai_responses`) and WebSocket
(`handle_realtime_websocket`) context search blocks.
- Added regression tests in
`tests/test_anthropic_pre_upstream_backpressure.py` and
`tests/test_memory_handler_native_ops.py`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
tests/test_anthropic_pre_upstream_backpressure.py .. [100%]
2 passed in 1.56s

tests/test_memory_handler_native_ops.py .................................. [100%]
34 passed in 1.57s
```

## Real Behavior Proof

- Environment: Linux x86_64, Python 3.14
- Exact command / steps: `pytest -q
tests/test_anthropic_pre_upstream_backpressure.py -k test_memory_context
&& pytest -q tests/test_memory_handler_native_ops.py`
- Observed result: Verified
`test_memory_context_exception_fails_open_and_releases_semaphore` and
`test_search_and_format_context_degrades_gracefully_on_backend_error`
pass cleanly.
- Not tested: None

## Runtime Rollout Safety

- Rollout-managed feature(s): None
- Minimum rollout channel: stable
- Stable/default behavior changed: No (gracefully falls back when memory
store is unavailable rather than returning 500)
- Kill switch / disable path: Disable memory with `--memory false` or
omitting `--memory`
- Unsafe override required: None
- Qualification impact: None
- Rollback path: git revert

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did not edit `CHANGELOG.md` (release-please handles this)

## Additional Notes

Ensures optional memory subsystem failure does not take down core proxy
traffic.


## Current maintainer verification (October 9 UTC)

Exact synchronized head: `b222647f31e0fbdfd8e33279f4c3c21e3cd4e40f`,
containing main `01d9194854233ef6ed04005cf0a5b8c32bf65d6d` through a
clean merge. The updated WebSocket regression gates each response pair
on its matching upstream request and keeps the client open through both
completions; the handler call is bounded by a five-second timeout. No
contributor logic was changed during synchronization.

Both affected routing/WebSocket files pass: 96 tests on Windows/Python
3.12 with the compatible existing native extension. Additional reviewer
probes pass with deliberately yielding upstream sends at 0, 1, and 10
ms, addressing the reported scheduling race. These are local
mocked-socket tests; no live external-provider or client session is
claimed. GitHub fork-workflow approval was provided for this reviewed
head; Current-head hosted snapshot completed with 30 successful checks,
12 skips, and one failed test shard. The failed per-frame memory
recovery test is a PR regression: this file has 1 failure/10 passes
locally on the PR, versus all 11 passes on main. Failed lookup currently
adds memory instructions instead of preserving the original WebSocket
frame; the new maintainer review requests that correction. The 96
routing/lifecycle tests and three scheduling probes above remain passing
evidence with this explicit limit.


### Maintainer current-main requalification

Exact head `3ca4324e69bd46a1c07288fb1b17c12b6fbcc9a7`, upstream main
`2d3ee51846435ef5b320eea4806af14e602b0d8c`.

Synchronized with upstream main
2d3ee51846. The memory-handler conflict
retains main's unresolved-project early return before the PR's
initialization/resolve failure boundary. The test conflict retains
main's streaming-tool regression and the PR's independent lookup-failure
regression as complete separate functions. No contributor failure policy
rewritten or main project-isolation guard dropped.

The author fix addresses the previous WebSocket finding: the inner
lookup catch no longer allows later augmentation after failure, so the
outer frame boundary forwards the original raw frame and later frames
can retry. The existing per-frame original-request/recovery regression
now passes. All 205 focused HTTP, WebSocket, native-memory-operation and
backpressure tests pass; current author/Devin discussion reviewed,
including the restored deterministic coordination and five-second test
deadline. Changed Python lint/format and main-relative whitespace pass.

Python uses the existing native extension for startup, not a fresh
native build. No live memory-cloud or model-provider claim. No remaining
finding in the reviewed failure/recovery scope; formal approval awaits
exact-head hosted qualification, handled by maintainers. Nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `742b5aeeb1e5c75cfa6a8b51d9b69d0a3fef36c8`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Clean mechanical synchronization with upstream main
c01cf07d94. Re-reviewed memory
initialization/lookup failure boundaries and current author/Devin
discussion. Main's unresolved-project early return remains before
backend initialization and resolution. Failed WebSocket lookup reaches
the outer fallback before augmentation, preserving the original raw
frame while later frames can retry. The previously fixed deterministic
two-turn coordination and five-second test deadline remain intact. No
contributor failure/isolation policy rewritten.

Fresh HTTP/WebSocket/native-memory-operation/backpressure verification:
205 passed, one test-client deprecation warning. All seven changed
Python files pass Ruff lint/format and main-relative whitespace checks;
owned tree clean. Current-main native DLL supports local startup; no
live cloud/model-provider or branch-specific native-build claim. No
remaining finding in this failure/recovery scope; formal approval awaits
exact-head hosted qualification. Maintainers handle hosted checks;
nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Abhinav Maurya
2026-11-06 09:25:11 -06:00
committed by GitHub
co-authored by JerrettDavis
parent cde584d7fe
commit 00d4f441d3
7 changed files with 256 additions and 4 deletions
+6
View File
@@ -3226,6 +3226,12 @@ class AnthropicHandlerMixin:
f"{self.config.anthropic_pre_upstream_memory_context_timeout_seconds:.1f}s; "
"continuing without it"
)
except Exception as e:
memory_context = None
logger.warning(
f"[{request_id}] Memory: Context lookup failed: {e}; "
"continuing without it"
)
try:
if memory_context:
from headroom.proxy.helpers import (
+5
View File
@@ -6647,6 +6647,11 @@ class OpenAIHandlerMixin:
f"[{request_id}] Memory context lookup exceeded "
f"{RESPONSES_CONTEXT_SEARCH_TIMEOUT_SECONDS:.1f}s; continuing without it"
)
except Exception as e:
memory_context = None
logger.warning(
f"[{request_id}] Memory context lookup failed: {e}; continuing without it"
)
if memory_context:
from headroom.proxy.helpers import (
append_text_to_latest_user_input_item,
+12 -4
View File
@@ -782,11 +782,19 @@ class MemoryHandler:
if self.is_project_unresolved(request_context):
return None
await self._ensure_initialized()
if not self._backend:
return None
try:
await self._ensure_initialized()
if not self._backend:
return None
backend, scope, effective_user_id = self._resolve_for_request(user_id, request_context)
backend, scope, effective_user_id = self._resolve_for_request(user_id, request_context)
except Exception as e:
logger.warning(
"Memory: Failed to initialize or resolve backend for user %s: %s",
user_id,
e,
)
return None
# Fail-closed when the router was unable to resolve a project in
# PROJECT mode and `unresolved_project_fallback="empty"` (the
@@ -736,6 +736,52 @@ def test_memory_context_timeout_fails_open_and_releases_semaphore():
anyio.run(_run)
def test_memory_context_exception_fails_open_and_releases_semaphore():
class _MemoryHandler:
def __init__(self) -> None:
self.config = SimpleNamespace(inject_context=True, inject_tools=False)
self.initialized = False
self.backend = None
async def search_and_format_context(self, _user_id, _messages, **_kwargs):
import sqlite3
raise sqlite3.OperationalError("unable to open database file")
def inject_tools(self, tools, _provider):
return tools, False
def get_beta_headers(self) -> dict[str, str]:
return {}
def has_memory_tool_calls(self, _response, _provider) -> bool:
return False
async def handle_memory_tool_calls(self, _response, _user_id, _provider, **_kwargs):
return []
async def _run() -> None:
sem = asyncio.Semaphore(1)
handler = _DummyAnthropicHandler(anthropic_pre_upstream_sem=sem)
handler.memory_handler = _MemoryHandler()
req = _build_request(
{
"model": "claude-3-5-sonnet-latest",
"messages": [{"role": "user", "content": "hello"}],
},
{
"authorization": "Bearer sk-ant-api-test",
"x-headroom-user-id": "user-1",
},
)
response = await handler.handle_anthropic_messages(req)
assert response.status_code == 200
assert sem._value == 1
with _tokenizer_patch():
anyio.run(_run)
# --------------------------------------------------------------------------- #
# /livez stays fast under Anthropic pre-upstream contention. #
# --------------------------------------------------------------------------- #
+14
View File
@@ -1807,3 +1807,17 @@ async def test_memory_list_dispatched_via_execute_memory_tool(handler: MemoryHan
payload = json.loads(out)
assert payload["status"] == "ok"
assert payload["memories"][0]["id"] == "m1"
@pytest.mark.asyncio
async def test_search_and_format_context_degrades_gracefully_on_backend_error(
handler: MemoryHandler, monkeypatch: pytest.MonkeyPatch
) -> None:
import sqlite3
async def fail_init() -> None:
raise sqlite3.OperationalError("unable to open database file")
monkeypatch.setattr(handler, "_ensure_initialized", fail_init)
result = await handler.search_and_format_context("u1", [{"role": "user", "content": "hello"}])
assert result is None
+66
View File
@@ -1094,3 +1094,69 @@ def test_responses_savings_accept_a_session_header_as_identity(monkeypatch):
_post_responses(handler, body, {"session_id": "sess-1"})
_post_responses(handler, body, {"session_id": "sess-2"})
assert _booked(handler) == [100, 0, 100]
def test_handle_openai_responses_memory_lookup_exception_fails_open_and_forwards_original_request(
monkeypatch,
):
import sqlite3
class _FailingContextMemoryHandler:
def __init__(self) -> None:
self.config = SimpleNamespace(
inject_context=True,
inject_tools=False,
project_root_override=str(Path(".").resolve()),
)
self.call_count = 0
async def search_and_format_context(self, user_id, messages, **kwargs):
self.call_count += 1
raise sqlite3.OperationalError("unable to open database file")
def compute_memory_tool_definitions(self, provider: str) -> list[dict]:
return []
def has_memory_tool_calls(self, response, provider: str) -> bool:
return False
handler = _DummyOpenAIHandler()
memory_handler = _FailingContextMemoryHandler()
handler.memory_handler = memory_handler
handler.session_tracker_store = SimpleNamespace(
compute_session_id=lambda *a, **k: "sess-api-memory-fallback",
)
monkeypatch.setattr("headroom.tokenizers.get_tokenizer", lambda model: _DummyTokenizer())
req1 = _build_request(
{"model": "gpt-4o-mini", "input": "turn 1 original", "store": False},
{
"Authorization": "Bearer sk-test",
"User-Agent": "codex-cli/0.5",
"x-headroom-user-id": "user-1",
},
)
req2 = _build_request(
{"model": "gpt-4o-mini", "input": "turn 2 original", "store": False},
{
"Authorization": "Bearer sk-test",
"User-Agent": "codex-cli/0.5",
"x-headroom-user-id": "user-1",
},
)
# First request: lookup fails, original body forwarded without memory, response succeeds
res1 = anyio.run(handler.handle_openai_responses, req1)
assert res1.status_code == 200
assert handler.captured_request is not None
_, _, _, body1 = handler.captured_request
assert body1["input"] == "turn 1 original"
# Subsequent request: handler continues serving traffic normally
res2 = anyio.run(handler.handle_openai_responses, req2)
assert res2.status_code == 200
assert handler.captured_request is not None
_, _, _, body2 = handler.captured_request
assert body2["input"] == "turn 2 original"
assert memory_handler.call_count == 2
+107
View File
@@ -2203,6 +2203,113 @@ async def test_ws_memory_enabled_non_memory_response_streams_completion():
assert len(upstream.sent) == 1
@pytest.mark.asyncio
async def test_ws_memory_context_lookup_exception_fails_open_and_survives_subsequent_traffic():
import sqlite3
class _FailingWsContextMemoryHandler(_MemoryWsHandler):
def __init__(self) -> None:
super().__init__()
self.config.inject_context = True
self.config.inject_tools = False
self.call_count = 0
async def search_and_format_context(self, user_id, messages, **kwargs):
self.call_count += 1
raise sqlite3.OperationalError("unable to open database file")
first_frame = json.dumps(
{
"type": "response.create",
"response": {
"model": "gpt-5.4",
"input": "turn 1 original",
},
}
)
second_frame = json.dumps(
{
"type": "response.create",
"response": {
"model": "gpt-5.4",
"input": "turn 2 original",
},
}
)
turn1_events = [
json.dumps({"type": "response.created", "response": {"id": "r-1"}}),
json.dumps({"type": "response.completed", "response": {"id": "r-1"}}),
]
turn2_events = [
json.dumps({"type": "response.created", "response": {"id": "r-2"}}),
json.dumps({"type": "response.completed", "response": {"id": "r-2"}}),
]
expected_events = turn1_events + turn2_events
first_sent = asyncio.Event()
second_sent = asyncio.Event()
turn1_done = asyncio.Event()
turn2_done = asyncio.Event()
both_completed = asyncio.Event()
class _CoordinatedUpstream(_FakeUpstream):
async def send(self, payload: str) -> None:
self.sent.append(payload)
if len(self.sent) == 1:
first_sent.set()
elif len(self.sent) == 2:
second_sent.set()
async def _iter(self):
await first_sent.wait()
for ev in turn1_events:
yield ev
await second_sent.wait()
for ev in turn2_events:
yield ev
await both_completed.wait()
class _CoordinatedClientWebSocket(_FakeWebSocket):
async def send_text(self, text: str) -> None:
await super().send_text(text)
if '"id": "r-1"' in text and "response.completed" in text:
turn1_done.set()
elif '"id": "r-2"' in text and "response.completed" in text:
turn2_done.set()
both_completed.set()
async def receive_text(self) -> str:
if not first_sent.is_set():
return first_frame
await turn1_done.wait()
if not second_sent.is_set():
return second_frame
await turn2_done.wait()
raise _FakeWebSocketDisconnect("client closed")
upstream = _CoordinatedUpstream([])
fake_ws_mod = _make_fake_websockets_module(upstream)
client_ws = _CoordinatedClientWebSocket([])
client_ws.headers["x-headroom-user-id"] = "user-1"
handler = _DummyOpenAIHandler()
memory_handler = _FailingWsContextMemoryHandler()
handler.memory_handler = memory_handler
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
await asyncio.wait_for(handler.handle_openai_responses_ws(client_ws), timeout=5.0)
# Both turns must reach upstream without injected memory
assert len(upstream.sent) == 2
sent0 = json.loads(upstream.sent[0])["response"]
sent1 = json.loads(upstream.sent[1])["response"]
assert sent0["input"] == "turn 1 original"
assert sent1["input"] == "turn 2 original"
# Streaming socket connection survived and client received all upstream events
assert client_ws.sent_text == expected_events
assert memory_handler.call_count == 2
@pytest.mark.asyncio
async def test_ws_late_memory_call_after_streamed_message_passes_through():
message_item = {