Commit Graph
7 Commits
Author SHA1 Message Date
Abhay SinghandJD Davis 3eba2d6066 fix(proxy/memory): fail open when backend init errors instead of 500ing the request (#3257)
## Description

`MemoryHandler._ensure_initialized` documents a "fail-open contract — no
exception propagates to request handlers", but only
`asyncio.TimeoutError` and `asyncio.CancelledError` are caught:

```python
try:
    await asyncio.wait_for(_do_init(), timeout=STARTUP_INIT_TIMEOUT_SECONDS)
except asyncio.TimeoutError:
    ... # fail open
except asyncio.CancelledError:
    ... # reset + re-raise
```

Any *other* exception raised synchronously during backend init — e.g.
`sqlite3.OperationalError: unable to open database file` — escapes both
handlers, propagates up through `handle_anthropic_messages` →
`memory_handler` → `LocalBackend._init_locked` →
`SQLiteMemoryStore._init_db`, and **500s the whole request**.

This is exactly the failure in issue #3251: on the documented macOS
Docker Desktop `persistent-docker --memory` deploy, the per-project
SQLite memory store cannot be opened in WAL mode over the bind-mount (a
known Docker Desktop/VirtioFS + SQLite interaction), and **every**
request returns `500 Internal Server Error` — even though compression,
caching, and tool-search savings don't need memory at all. An optional
subsystem should never take down core proxy traffic; that is Ask #3 of
the issue ("a memory-store open failure on the request path should
degrade gracefully instead of 500ing").

The environmental root cause (bind-mount + WAL) is out of scope here;
this fixes the graceful-degradation gap so `--memory` being unavailable
disables memory context for the request rather than killing it.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/memory_handler.py`: added an `except Exception` branch
to `_ensure_initialized` that mirrors the timeout branch — best-effort
close the possibly-half-assigned backend, null it, leave
`_initialized=False` so a later request retries, log an error, and
**return** (fail open) instead of propagating. Ordering is preserved:
`TimeoutError` (an `Exception`) is still caught first, and
`CancelledError` (a `BaseException`, not an `Exception`) still falls
through to its own re-raising branch.
- `tests/test_memory_handler_concurrent_init.py`: added
`test_ensure_initialized_fails_open_on_backend_init_error` — a
`LocalBackend` whose init raises `sqlite3.OperationalError`; asserts
`_ensure_initialized` does not raise, cleans up the backend, stays
uninitialized, and retries on the next call.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
tests/test_memory_handler_concurrent_init.py  ->  9 passed in 1.80s
(the new test FAILS on pre-fix code — the OperationalError propagates — verified via git stash)
uvx ruff@0.16.2 check headroom/proxy/memory_handler.py tests/test_memory_handler_concurrent_init.py  ->  All checks passed!
uvx mypy@1.20.2 headroom/proxy/memory_handler.py  ->  Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: monkeypatched `LocalBackend` so its
`_ensure_initialized` raises `sqlite3.OperationalError("unable to open
database file")` (the #3251 failure). Before the fix, `await
handler._ensure_initialized()` re-raised the error (which on the real
path becomes a request 500). After the fix, it returns cleanly with
`_initialized=False`, `_backend=None`, the half-init backend `close()`d
once, and a later call retries.
- Observed result: a backend init failure no longer propagates; memory
is skipped for the request and retried later; timeout and cancellation
behavior are unchanged.
- Not tested: no live macOS Docker Desktop repro of the bind-mount/WAL
open failure (environment-specific); the fail-open path is exercised by
injecting the same exception the deploy hits.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is the memory-subsystem
lazy-init on the proxy request path, not a rollout-channel-gated runtime
feature.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: only the failure path — a backend
that cannot initialize now disables memory for the request instead of
500ing it. Successful init, the timeout branch, and the cancellation
branch are unchanged.
- Kill switch / disable path: memory is already opt-in
(`MemoryConfig.enabled`); this makes a broken backend behave like
memory-off for affected requests.
- Unsafe override required: no.
- Qualification impact: none for healthy deployments; broken-memory
deployments go from all-requests-500 to requests-served-without-memory.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal behavior)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Addresses Ask #3 of #3251 (graceful degradation). The environmental root
cause (Docker Desktop macOS bind-mount + SQLite WAL) and Asks #1/#2
(repro / named-volume preset) are deployment concerns left out of this
code fix.

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-09-09 13:33:46 -05:00
Rod Boev 2cae13dd79 fix(ccr): skip Anthropic marker emission when tool injection is deferred (#1273)
## Description

Anthropic request-side CCR can still compress a turn into retrieval
markers after the frozen-prefix cache guard suppresses
`headroom_retrieve` registration. That leaves the model with marker-only
context it cannot redeem, so the proxy silently drops recoverable data
on exactly the turns where cache preservation deferred tool injection.
This change couples the Anthropic request-side CCR path to tool
availability so a turn never emits retrieve-only markers without the
retrieval tool, even when token mode or cache-mode prefix replay could
otherwise reuse already-compressed marker text. Closes #1006

After a collaborator merged current `main` into this branch, CI also
picked up unrelated offline-memory failures from the merged base. Those
follow-up changes are test-only: they keep the offline Hugging Face
cache lanes skipping cleanly instead of failing in memory tests that are
outside the CCR runtime path.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Couple Anthropic request-side CCR compression to the same
frozen-prefix guard that already defers `headroom_retrieve`
registration.
- Keep the existing cache-preservation behavior: frozen-prefix turns
stop emitting CCR retrieval markers instead of forcing tool injection
into the cached prefix.
- Make the skip decision use the effective frozen prefix after
token-mode reclamping, so turns that genuinely reclamp to zero still
keep normal reversible CCR behavior.
- Bypass cached marker reuse in both token mode and cache-mode prefix
replay when tool injection is deferred.
- Add focused regressions for the Anthropic request-path seams under
this bug:
  - frozen-prefix turns do not emit marker-only payloads
  - unfrozen turns still keep normal reversible CCR behavior
  - token-mode reclamp back to zero still compresses normally
- existing `headroom_retrieve` tools keep reversible CCR on frozen turns
- cache-mode delta reuse and exact-prefix replay both forward original
content when retrieval is unavailable
- Add a `CHANGELOG.md` entry because the proxy's user-visible Anthropic
CCR behavior changes.
- Add a shared test skip helper for offline Hugging Face cache misses
and apply it to the merged `main` memory tests that were failing only in
the offline CI shards after the branch picked up current `main`.

## Testing

- [x] Unit tests pass (`uv run pytest
tests/test_proxy/test_anthropic_ccr_deferred_injection.py`)
- [x] Linting passes (`uv run ruff check . && uv run ruff format .
--check`)
- [ ] Type checking passes (`uv run mypy headroom`)
- [x] New tests added for new functionality when applicable
- [ ] Manual testing performed

### Test Output

```text
uv run pytest tests/test_proxy/test_anthropic_ccr_deferred_injection.py
14 passed, 1 warning in 34.19s

uv run pytest tests/test_memory/test_skip_helpers.py tests/test_memory/test_embedder_mps_serialization.py::test_cpu_uses_shared_executor tests/test_memory/test_hierarchical.py::TestLocalEmbedder::test_embed_single tests/test_memory_bridge.py::TestMemoryBridgeImport::test_import_claude_code_memory tests/test_memory_handler_concurrent_init.py::test_real_localbackend_initializes_via_public_entrypoint tests/test_memory_system.py::TestLocalBackend::test_save_memory_basic -q
4 passed, 5 skipped, 11 warnings in 7.65s

uv run ruff check .
All checks passed!

uv run ruff format . --check
968 files already formatted
```

## Real Behavior Proof

- Environment: local FastAPI `TestClient` for the Anthropic request
path, plus Windows Python 3.12 offline-memory repros with
`TRANSFORMERS_OFFLINE=1`
- Exact command / steps: Run the focused CCR regression command and the
offline-memory repro subset below on the merged branch state.
- `uv run pytest
tests/test_proxy/test_anthropic_ccr_deferred_injection.py`
- `uv run pytest tests/test_memory/test_skip_helpers.py
tests/test_memory/test_embedder_mps_serialization.py::test_cpu_uses_shared_executor
tests/test_memory/test_hierarchical.py::TestLocalEmbedder::test_embed_single
tests/test_memory_bridge.py::TestMemoryBridgeImport::test_import_claude_code_memory
tests/test_memory_handler_concurrent_init.py::test_real_localbackend_initializes_via_public_entrypoint
tests/test_memory_system.py::TestLocalBackend::test_save_memory_basic
-q`
- Scenario coverage from the CCR pytest command:
- frozen prefix with deferred tool injection and cached marker text
available in token mode
  - unfrozen turn with normal CCR marker emission
  - token mode where the tracked frozen prefix reclamps back to zero
  - frozen prefix where the client already supplied `headroom_retrieve`
- cache-mode append-only delta reuse with a previously forwarded
compressed prefix
- cache-mode exact-prefix replay where the previous forwarded prefix
already contained a marker
- Scenario coverage from the offline-memory repro command:
  - direct local embedder startup on CPU with no cached HF model
- hierarchical memory embedder startup with offline model cache missing
  - bridge import through `LocalBackend`
  - `MemoryHandler` public init warmup path
  - `LocalBackend` save path under the offline lane
- Observed result: The CCR regression keeps marker-free forwarding on
frozen turns without tool availability, and the merged-`main`
offline-memory lanes now skip cleanly instead of failing unrelated CI
shards.
- frozen-prefix Anthropic turns without tool availability forward the
original long transcript across both token-mode and cache-mode reuse
paths, while unfrozen turns, reclamped token-mode turns, and frozen
turns that already advertise `headroom_retrieve` keep the reversible CCR
marker path
- the merged-`main` offline-memory regressions now skip cleanly when the
Hugging Face cache is unavailable instead of failing unrelated CI shards
- Not tested: live Zed session cache-hit behavior, provider latency
under real Anthropic upstreams, and online Hugging Face download lanes

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

- Runtime scope is still intentionally narrow to Anthropic request-side
CCR. OpenAI, Gemini, streaming, and response-side CCR behavior are
unchanged.
- The only non-CCR diff is the test-only offline-memory follow-up
required after current `main` was merged into the branch.
- The focused proxy pytest run still emits the Windows-local
`StarletteDeprecationWarning` from `fastapi.testclient`'s `httpx`
bridge. The offline-memory repro command also emits existing datetime
deprecation warnings and a pytest teardown warning around skipped
offline lanes; none of those warnings were introduced by the CCR runtime
change.
2026-06-23 12:48:05 -05:00
Adryan Eka Vandra 4e9348dec4 fix: add anthropic pre-upstream timeouts 2026-04-21 09:48:32 +07:00
Adryan Eka Vandra b71b659e1b fix(ci): restore repro harness test in dev installs
Add websockets to the dev extra so the repro harness smoke test can import
its websocket client dependency in the CI test matrix. Also apply ruff
formatting to the files the formatter check was rejecting so the 3.12 lint
job passes.
2026-04-20 22:12:01 +07:00
Adryan Eka Vandra 196d8d69a7 fix(proxy): propagate CancelledError from memory init cleanly
External cancellation while _ensure_initialized is in flight raises
CancelledError (BaseException). The existing 'except asyncio.TimeoutError:'
branch does not catch it, and caller 'except Exception:' blocks don't
either, so it propagated as-is with _backend possibly still assigned.

Add an explicit 'except asyncio.CancelledError:' handler that nulls
_backend, clears _initialized, logs at info, and re-raises. Cancellation
is a shutdown signal, not an error to swallow — but the state must be
clean so a later retry starts fresh.
2026-04-20 22:08:09 +07:00
Adryan Eka Vandra 3f1f82d7e2 fix(proxy): reset memory backend to None on init timeout
If asyncio.wait_for fires while LocalBackend._ensure_initialized() is
running, _init_backend_locked may have already assigned self._backend
(the LocalBackend constructor) before its own await raised or was
cancelled. Callers doing if self.memory_handler._backend: then see
a truthy-but-broken backend.

Null self._backend in the TimeoutError handler so the post-timeout
state is consistent: _initialized=False AND _backend=None.
2026-04-20 22:08:09 +07:00
Adryan Eka Vandra 7e2b078b76 feat(proxy): share cold-start warmup across pipelines + lock memory init
Unit 1 of the codex-proxy-resilience plan. Preload now iterates BOTH
the Anthropic and OpenAI transform pipelines, dedupes shared transforms
by id(), and merges the status dict into a typed WarmupRegistry so
/debug/warmup (Unit 5) and /readyz have one source of truth.

MemoryHandler._ensure_initialized gains an asyncio.Lock with a
double-checked pattern so concurrent first callers share one backend
init instead of racing. The body is wrapped in asyncio.wait_for with
STARTUP_INIT_TIMEOUT_SECONDS (default 30s) — on timeout _initialized
stays False so subsequent requests retry (fail-open contract).

Startup now also forces one embedder warm-up encode so the ONNX graph
is compiled synchronously at startup instead of lazily on the first
request. Best-effort: failures are logged, never raised.
2026-04-20 22:01:41 +07:00