mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
fix/plugin-launcher-isolated-python
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
3eba2d6066 |
fix(proxy/memory): fail open when backend init errors instead of 500ing the request (#3257)
## Description
`MemoryHandler._ensure_initialized` documents a "fail-open contract — no
exception propagates to request handlers", but only
`asyncio.TimeoutError` and `asyncio.CancelledError` are caught:
```python
try:
await asyncio.wait_for(_do_init(), timeout=STARTUP_INIT_TIMEOUT_SECONDS)
except asyncio.TimeoutError:
... # fail open
except asyncio.CancelledError:
... # reset + re-raise
```
Any *other* exception raised synchronously during backend init — e.g.
`sqlite3.OperationalError: unable to open database file` — escapes both
handlers, propagates up through `handle_anthropic_messages` →
`memory_handler` → `LocalBackend._init_locked` →
`SQLiteMemoryStore._init_db`, and **500s the whole request**.
This is exactly the failure in issue #3251: on the documented macOS
Docker Desktop `persistent-docker --memory` deploy, the per-project
SQLite memory store cannot be opened in WAL mode over the bind-mount (a
known Docker Desktop/VirtioFS + SQLite interaction), and **every**
request returns `500 Internal Server Error` — even though compression,
caching, and tool-search savings don't need memory at all. An optional
subsystem should never take down core proxy traffic; that is Ask #3 of
the issue ("a memory-store open failure on the request path should
degrade gracefully instead of 500ing").
The environmental root cause (bind-mount + WAL) is out of scope here;
this fixes the graceful-degradation gap so `--memory` being unavailable
disables memory context for the request rather than killing it.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/memory_handler.py`: added an `except Exception` branch
to `_ensure_initialized` that mirrors the timeout branch — best-effort
close the possibly-half-assigned backend, null it, leave
`_initialized=False` so a later request retries, log an error, and
**return** (fail open) instead of propagating. Ordering is preserved:
`TimeoutError` (an `Exception`) is still caught first, and
`CancelledError` (a `BaseException`, not an `Exception`) still falls
through to its own re-raising branch.
- `tests/test_memory_handler_concurrent_init.py`: added
`test_ensure_initialized_fails_open_on_backend_init_error` — a
`LocalBackend` whose init raises `sqlite3.OperationalError`; asserts
`_ensure_initialized` does not raise, cleans up the backend, stays
uninitialized, and retries on the next call.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
### Test Output
```text
tests/test_memory_handler_concurrent_init.py -> 9 passed in 1.80s
(the new test FAILS on pre-fix code — the OperationalError propagates — verified via git stash)
uvx ruff@0.16.2 check headroom/proxy/memory_handler.py tests/test_memory_handler_concurrent_init.py -> All checks passed!
uvx mypy@1.20.2 headroom/proxy/memory_handler.py -> Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: monkeypatched `LocalBackend` so its
`_ensure_initialized` raises `sqlite3.OperationalError("unable to open
database file")` (the #3251 failure). Before the fix, `await
handler._ensure_initialized()` re-raised the error (which on the real
path becomes a request 500). After the fix, it returns cleanly with
`_initialized=False`, `_backend=None`, the half-init backend `close()`d
once, and a later call retries.
- Observed result: a backend init failure no longer propagates; memory
is skipped for the request and retried later; timeout and cancellation
behavior are unchanged.
- Not tested: no live macOS Docker Desktop repro of the bind-mount/WAL
open failure (environment-specific); the fail-open path is exercised by
injecting the same exception the deploy hits.
## Runtime Rollout Safety
- Rollout-managed feature(s): none. This is the memory-subsystem
lazy-init on the proxy request path, not a rollout-channel-gated runtime
feature.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: only the failure path — a backend
that cannot initialize now disables memory for the request instead of
500ing it. Successful init, the timeout branch, and the cancellation
branch are unchanged.
- Kill switch / disable path: memory is already opt-in
(`MemoryConfig.enabled`); this makes a broken backend behave like
memory-off for affected requests.
- Unsafe override required: no.
- Qualification impact: none for healthy deployments; broken-memory
deployments go from all-requests-500 to requests-served-without-memory.
- Rollback path: revert this PR.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal behavior)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
Addresses Ask #3 of #3251 (graceful degradation). The environmental root
cause (Docker Desktop macOS bind-mount + SQLite WAL) and Asks #1/#2
(repro / named-volume preset) are deployment concerns left out of this
code fix.
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
|
2cae13dd79 |
fix(ccr): skip Anthropic marker emission when tool injection is deferred (#1273)
## Description Anthropic request-side CCR can still compress a turn into retrieval markers after the frozen-prefix cache guard suppresses `headroom_retrieve` registration. That leaves the model with marker-only context it cannot redeem, so the proxy silently drops recoverable data on exactly the turns where cache preservation deferred tool injection. This change couples the Anthropic request-side CCR path to tool availability so a turn never emits retrieve-only markers without the retrieval tool, even when token mode or cache-mode prefix replay could otherwise reuse already-compressed marker text. Closes #1006 After a collaborator merged current `main` into this branch, CI also picked up unrelated offline-memory failures from the merged base. Those follow-up changes are test-only: they keep the offline Hugging Face cache lanes skipping cleanly instead of failing in memory tests that are outside the CCR runtime path. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Couple Anthropic request-side CCR compression to the same frozen-prefix guard that already defers `headroom_retrieve` registration. - Keep the existing cache-preservation behavior: frozen-prefix turns stop emitting CCR retrieval markers instead of forcing tool injection into the cached prefix. - Make the skip decision use the effective frozen prefix after token-mode reclamping, so turns that genuinely reclamp to zero still keep normal reversible CCR behavior. - Bypass cached marker reuse in both token mode and cache-mode prefix replay when tool injection is deferred. - Add focused regressions for the Anthropic request-path seams under this bug: - frozen-prefix turns do not emit marker-only payloads - unfrozen turns still keep normal reversible CCR behavior - token-mode reclamp back to zero still compresses normally - existing `headroom_retrieve` tools keep reversible CCR on frozen turns - cache-mode delta reuse and exact-prefix replay both forward original content when retrieval is unavailable - Add a `CHANGELOG.md` entry because the proxy's user-visible Anthropic CCR behavior changes. - Add a shared test skip helper for offline Hugging Face cache misses and apply it to the merged `main` memory tests that were failing only in the offline CI shards after the branch picked up current `main`. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_proxy/test_anthropic_ccr_deferred_injection.py`) - [x] Linting passes (`uv run ruff check . && uv run ruff format . --check`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text uv run pytest tests/test_proxy/test_anthropic_ccr_deferred_injection.py 14 passed, 1 warning in 34.19s uv run pytest tests/test_memory/test_skip_helpers.py tests/test_memory/test_embedder_mps_serialization.py::test_cpu_uses_shared_executor tests/test_memory/test_hierarchical.py::TestLocalEmbedder::test_embed_single tests/test_memory_bridge.py::TestMemoryBridgeImport::test_import_claude_code_memory tests/test_memory_handler_concurrent_init.py::test_real_localbackend_initializes_via_public_entrypoint tests/test_memory_system.py::TestLocalBackend::test_save_memory_basic -q 4 passed, 5 skipped, 11 warnings in 7.65s uv run ruff check . All checks passed! uv run ruff format . --check 968 files already formatted ``` ## Real Behavior Proof - Environment: local FastAPI `TestClient` for the Anthropic request path, plus Windows Python 3.12 offline-memory repros with `TRANSFORMERS_OFFLINE=1` - Exact command / steps: Run the focused CCR regression command and the offline-memory repro subset below on the merged branch state. - `uv run pytest tests/test_proxy/test_anthropic_ccr_deferred_injection.py` - `uv run pytest tests/test_memory/test_skip_helpers.py tests/test_memory/test_embedder_mps_serialization.py::test_cpu_uses_shared_executor tests/test_memory/test_hierarchical.py::TestLocalEmbedder::test_embed_single tests/test_memory_bridge.py::TestMemoryBridgeImport::test_import_claude_code_memory tests/test_memory_handler_concurrent_init.py::test_real_localbackend_initializes_via_public_entrypoint tests/test_memory_system.py::TestLocalBackend::test_save_memory_basic -q` - Scenario coverage from the CCR pytest command: - frozen prefix with deferred tool injection and cached marker text available in token mode - unfrozen turn with normal CCR marker emission - token mode where the tracked frozen prefix reclamps back to zero - frozen prefix where the client already supplied `headroom_retrieve` - cache-mode append-only delta reuse with a previously forwarded compressed prefix - cache-mode exact-prefix replay where the previous forwarded prefix already contained a marker - Scenario coverage from the offline-memory repro command: - direct local embedder startup on CPU with no cached HF model - hierarchical memory embedder startup with offline model cache missing - bridge import through `LocalBackend` - `MemoryHandler` public init warmup path - `LocalBackend` save path under the offline lane - Observed result: The CCR regression keeps marker-free forwarding on frozen turns without tool availability, and the merged-`main` offline-memory lanes now skip cleanly instead of failing unrelated CI shards. - frozen-prefix Anthropic turns without tool availability forward the original long transcript across both token-mode and cache-mode reuse paths, while unfrozen turns, reclamped token-mode turns, and frozen turns that already advertise `headroom_retrieve` keep the reversible CCR marker path - the merged-`main` offline-memory regressions now skip cleanly when the Hugging Face cache is unavailable instead of failing unrelated CI shards - Not tested: live Zed session cache-hit behavior, provider latency under real Anthropic upstreams, and online Hugging Face download lanes ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - Runtime scope is still intentionally narrow to Anthropic request-side CCR. OpenAI, Gemini, streaming, and response-side CCR behavior are unchanged. - The only non-CCR diff is the test-only offline-memory follow-up required after current `main` was merged into the branch. - The focused proxy pytest run still emits the Windows-local `StarletteDeprecationWarning` from `fastapi.testclient`'s `httpx` bridge. The offline-memory repro command also emits existing datetime deprecation warnings and a pytest teardown warning around skipped offline lanes; none of those warnings were introduced by the CCR runtime change. |
||
|
|
4e9348dec4 | fix: add anthropic pre-upstream timeouts | ||
|
|
b71b659e1b |
fix(ci): restore repro harness test in dev installs
Add websockets to the dev extra so the repro harness smoke test can import its websocket client dependency in the CI test matrix. Also apply ruff formatting to the files the formatter check was rejecting so the 3.12 lint job passes. |
||
|
|
196d8d69a7 |
fix(proxy): propagate CancelledError from memory init cleanly
External cancellation while _ensure_initialized is in flight raises CancelledError (BaseException). The existing 'except asyncio.TimeoutError:' branch does not catch it, and caller 'except Exception:' blocks don't either, so it propagated as-is with _backend possibly still assigned. Add an explicit 'except asyncio.CancelledError:' handler that nulls _backend, clears _initialized, logs at info, and re-raises. Cancellation is a shutdown signal, not an error to swallow — but the state must be clean so a later retry starts fresh. |
||
|
|
3f1f82d7e2 |
fix(proxy): reset memory backend to None on init timeout
If asyncio.wait_for fires while LocalBackend._ensure_initialized() is running, _init_backend_locked may have already assigned self._backend (the LocalBackend constructor) before its own await raised or was cancelled. Callers doing if self.memory_handler._backend: then see a truthy-but-broken backend. Null self._backend in the TimeoutError handler so the post-timeout state is consistent: _initialized=False AND _backend=None. |
||
|
|
7e2b078b76 |
feat(proxy): share cold-start warmup across pipelines + lock memory init
Unit 1 of the codex-proxy-resilience plan. Preload now iterates BOTH the Anthropic and OpenAI transform pipelines, dedupes shared transforms by id(), and merges the status dict into a typed WarmupRegistry so /debug/warmup (Unit 5) and /readyz have one source of truth. MemoryHandler._ensure_initialized gains an asyncio.Lock with a double-checked pattern so concurrent first callers share one backend init instead of racing. The body is wrapped in asyncio.wait_for with STARTUP_INIT_TIMEOUT_SECONDS (default 30s) — on timeout _initialized stays False so subsequent requests retry (fail-open contract). Startup now also forces one embedder warm-up encode so the ONNX graph is compiled synchronously at startup instead of lazily on the first request. Best-effort: failures are logged, never raised. |