fix(proxy): skip the memory-context search in cache mode (#4606)

## Description

In `--mode cache`, the Anthropic `/v1/messages` handler ran the full
memory-context search (`search_and_format_context`: embed the query,
query the vector index and graph) on every request, and only then
checked `is_cache_mode()` and discarded the result with
`decision=skipped_cache_mode`. Cache mode never injects memory context,
so the search was pure latency before the request goes upstream.

This PR checks cache mode first and skips the search entirely. The
`skipped_cache_mode` decision is still logged, and memory tools are
still injected.

Closes #4600

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Performance improvement

## Changes Made

- `headroom/proxy/handlers/anthropic.py`: when `inject_context` is on
and the proxy is in cache mode, log `skipped_cache_mode` and skip
`search_and_format_context`. The old post-search cache-mode branch is
removed.
- Corrected the gate comment: `decide_memory_injection` only skips for
`HEADROOM_MEMORY_INJECTION_MODE=disabled`, not `tool` (as noted in the
issue).
- `tests/test_proxy_system_prompt_immutable.py`: added
`test_cache_mode_skips_memory_context_search`, which sends a cache-mode
request with memory on and asserts the search is never awaited and the
latest user turn is forwarded unchanged.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`): see Additional Notes
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

New test, before the fix:

```text
E   AssertionError: Expected mock to not have been awaited. Awaited 1 times.
FAILED tests/test_proxy_system_prompt_immutable.py::test_cache_mode_skips_memory_context_search
```

After the fix:

```text
$ pytest tests/test_proxy_system_prompt_immutable.py -q
8 passed

$ pytest tests/test_proxy_anthropic_cache_stability.py tests/test_proxy_memory_integration.py \
    tests/test_memory_tool_mode.py tests/test_memory_decision.py \
    tests/test_anthropic_pre_upstream_backpressure.py tests/test_proxy_pipeline_lifecycle.py \
    tests/test_proxy_byte_faithful_forwarding.py -q
170 passed, 12 skipped

$ ruff check headroom/proxy/handlers/anthropic.py tests/test_proxy_system_prompt_immutable.py
All checks passed!
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, local memory backend
(`all-MiniLM-L6-v2` ONNX embedder, hnswlib vector index),
`memory_storage_mode=global` with 50 seeded memories.
- Exact command / steps: an in-process benchmark that starts the real
proxy app (`create_app` + `TestClient`) with `--mode cache` and memory
enabled, stubs only the upstream call, and sends 100 `/v1/messages`
requests. It counts calls to `search_and_format_context` and times each
request end to end. Two alternating rounds of `main` (fc1ba891) vs this
branch.
- Observed result: in cache mode, `main` ran the memory search on 100 of
100 requests (p50 ~123 ms); this PR ran it on 0 of 100 (p50 3–5 ms),
about 120 ms saved per request. Full numbers below.

| Run | Searches in cache mode | p50 | p95 | mean |
|---|---|---|---|---|
| `main`, round 1 | 100 / 100 | 122.7 ms | 152.9 ms | 126.5 ms |
| this PR, round 1 | 0 / 100 | 2.9 ms | 3.6 ms | 3.0 ms |
| `main`, round 2 | 100 / 100 | 124.0 ms | 266.7 ms | 157.6 ms |
| this PR, round 2 | 0 / 100 | 4.8 ms | 10.1 ms | 5.6 ms |

About 120 ms p50 is removed from every cache-mode request. This matches
the issue's production numbers (`memory_context` p50 114 ms on v0.42.0,
p95 up to 1,169 ms on v0.40.0).
- Not tested: the benchmark store returned no hits on search (the
reopened HNSW index comes back empty, which looks like the reload defect
in #4362), so the `main` numbers cover embedding plus index and graph
query, but not result formatting. With real hits the saving is equal or
larger. Not run against a live upstream.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no. Cache mode already discarded the
search result; only the wasted work is removed. Token mode is unchanged.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation: n/a
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- `skipped_cache_mode` is now logged on every cache-mode request with
`inject_context` on. Before, it was logged only when the search returned
something.
- The OpenAI and Gemini handlers also call `search_and_format_context`,
but they inject the result even in cache mode, so their search is not
wasted. Whether those paths should skip injection in cache mode is a
separate question and is left out of this PR.
- `mypy headroom/proxy/handlers/anthropic.py` stops in my local
environment on a `numpy` stub (`Type statement is only supported in
Python 3.12 and greater`) before reaching project code. That is
unrelated to this change, and CI runs mypy.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4606"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
SD
2026-11-03 19:35:14 -06:00
committed by GitHub
co-authored by JD Davis
parent 75df01d627
commit 053925b03c
2 changed files with 55 additions and 12 deletions
+15 -12
View File
@@ -2970,7 +2970,7 @@ class AnthropicHandlerMixin:
# Memory: Inject context and tools — gated on MemoryDecision.
# ``inject`` is False under bypass, missing handler, missing
# user_id, or HEADROOM_MEMORY_INJECTION_MODE in disabled/tool.
# user_id, or HEADROOM_MEMORY_INJECTION_MODE=disabled.
# Pre-PR-this the gate was a raw conjunction that silently
# ignored bypass; now bypass is honoured here on Anthropic
# /v1/messages just as on /v1/responses.
@@ -2980,8 +2980,21 @@ class AnthropicHandlerMixin:
# streaming path withholds their calls from the client.
server_memory_tool_names: frozenset[str] = frozenset()
if memory_decision.inject:
if self.memory_handler.config.inject_context and is_cache_mode(self.config.mode):
# Cache mode never injects context, so the next-turn prefix
# bytes stay byte-equal to this turn's bytes. Skip the search
# too: its result could only be discarded (#4600).
from headroom.proxy.helpers import log_memory_injection
log_memory_injection(
request_id=request_id,
session_id=session_id,
decision="skipped_cache_mode",
bytes_injected=0,
query=extract_user_query(optimized_messages) or "",
)
# Search and inject memory context
if self.memory_handler.config.inject_context:
elif self.memory_handler.config.inject_context:
try:
async with stage_timer.measure("memory_context"):
memory_context = await asyncio.wait_for(
@@ -3019,16 +3032,6 @@ class AnthropicHandlerMixin:
bytes_injected=0,
query=user_query,
)
elif is_cache_mode(self.config.mode):
# Cache mode: skip injection entirely so the next-turn
# prefix bytes remain byte-equal to this turn's bytes.
log_memory_injection(
request_id=request_id,
session_id=session_id,
decision="skipped_cache_mode",
bytes_injected=0,
query=user_query,
)
else:
# P0-1 fix: route exclusively to the live zone tail
# (latest non-frozen user turn). System prompt + frozen
@@ -193,6 +193,46 @@ def test_memory_context_appears_in_latest_user_message_tail() -> None:
assert sent["messages"][2]["content"].endswith("MEMCTX")
def test_cache_mode_skips_memory_context_search() -> None:
"""Cache mode never injects memory context, so it must not search for it."""
captured: dict[str, object] = {}
with _make_proxy_client() as client:
proxy = client.app.state.proxy
proxy.config.mode = "cache"
proxy.config.optimize = False
proxy.config.image_optimize = False
proxy.config.ccr_proactive_expansion = False
_install_session_tracker(proxy, frozen_count=1)
_install_memory_handler(proxy)
_install_capture_retry(proxy, captured)
response = client.post(
"/v1/messages",
headers={
"x-api-key": "test-key",
"anthropic-version": "2023-06-01",
"x-headroom-user-id": "u1",
},
json={
"model": "claude-sonnet-4-6",
"max_tokens": 64,
"system": "base system",
"messages": [
{"role": "user", "content": "frozen prefix"},
{"role": "assistant", "content": "ack"},
{"role": "user", "content": "latest user"},
],
},
)
assert response.status_code == 200
proxy.memory_handler.search_and_format_context.assert_not_awaited()
sent = captured["body"]
assert isinstance(sent, dict)
assert sent["messages"][2]["content"] == "latest user"
def test_memory_context_byte_deterministic_for_same_query() -> None:
"""Two identical inbound requests produce identical outbound bytes."""
capture_a: dict[str, object] = {}