mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): skip the memory-context search in cache mode (#4606)
## Description
In `--mode cache`, the Anthropic `/v1/messages` handler ran the full
memory-context search (`search_and_format_context`: embed the query,
query the vector index and graph) on every request, and only then
checked `is_cache_mode()` and discarded the result with
`decision=skipped_cache_mode`. Cache mode never injects memory context,
so the search was pure latency before the request goes upstream.
This PR checks cache mode first and skips the search entirely. The
`skipped_cache_mode` decision is still logged, and memory tools are
still injected.
Closes #4600
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Performance improvement
## Changes Made
- `headroom/proxy/handlers/anthropic.py`: when `inject_context` is on
and the proxy is in cache mode, log `skipped_cache_mode` and skip
`search_and_format_context`. The old post-search cache-mode branch is
removed.
- Corrected the gate comment: `decide_memory_injection` only skips for
`HEADROOM_MEMORY_INJECTION_MODE=disabled`, not `tool` (as noted in the
issue).
- `tests/test_proxy_system_prompt_immutable.py`: added
`test_cache_mode_skips_memory_context_search`, which sends a cache-mode
request with memory on and asserts the search is never awaited and the
latest user turn is forwarded unchanged.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`): see Additional Notes
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
New test, before the fix:
```text
E AssertionError: Expected mock to not have been awaited. Awaited 1 times.
FAILED tests/test_proxy_system_prompt_immutable.py::test_cache_mode_skips_memory_context_search
```
After the fix:
```text
$ pytest tests/test_proxy_system_prompt_immutable.py -q
8 passed
$ pytest tests/test_proxy_anthropic_cache_stability.py tests/test_proxy_memory_integration.py \
tests/test_memory_tool_mode.py tests/test_memory_decision.py \
tests/test_anthropic_pre_upstream_backpressure.py tests/test_proxy_pipeline_lifecycle.py \
tests/test_proxy_byte_faithful_forwarding.py -q
170 passed, 12 skipped
$ ruff check headroom/proxy/handlers/anthropic.py tests/test_proxy_system_prompt_immutable.py
All checks passed!
```
## Real Behavior Proof
- Environment: macOS, Python 3.12, local memory backend
(`all-MiniLM-L6-v2` ONNX embedder, hnswlib vector index),
`memory_storage_mode=global` with 50 seeded memories.
- Exact command / steps: an in-process benchmark that starts the real
proxy app (`create_app` + `TestClient`) with `--mode cache` and memory
enabled, stubs only the upstream call, and sends 100 `/v1/messages`
requests. It counts calls to `search_and_format_context` and times each
request end to end. Two alternating rounds of `main` (fc1ba891) vs this
branch.
- Observed result: in cache mode, `main` ran the memory search on 100 of
100 requests (p50 ~123 ms); this PR ran it on 0 of 100 (p50 3–5 ms),
about 120 ms saved per request. Full numbers below.
| Run | Searches in cache mode | p50 | p95 | mean |
|---|---|---|---|---|
| `main`, round 1 | 100 / 100 | 122.7 ms | 152.9 ms | 126.5 ms |
| this PR, round 1 | 0 / 100 | 2.9 ms | 3.6 ms | 3.0 ms |
| `main`, round 2 | 100 / 100 | 124.0 ms | 266.7 ms | 157.6 ms |
| this PR, round 2 | 0 / 100 | 4.8 ms | 10.1 ms | 5.6 ms |
About 120 ms p50 is removed from every cache-mode request. This matches
the issue's production numbers (`memory_context` p50 114 ms on v0.42.0,
p95 up to 1,169 ms on v0.40.0).
- Not tested: the benchmark store returned no hits on search (the
reopened HNSW index comes back empty, which looks like the reload defect
in #4362), so the `main` numbers cover embedding plus index and graph
query, but not result formatting. With real hits the saving is equal or
larger. Not run against a live upstream.
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no. Cache mode already discarded the
search result; only the wasted work is removed. Token mode is unchanged.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation: n/a
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
- `skipped_cache_mode` is now logged on every cache-mode request with
`inject_context` on. Before, it was logged only when the search returned
something.
- The OpenAI and Gemini handlers also call `search_and_format_context`,
but they inject the result even in cache mode, so their search is not
wasted. Whether those paths should skip injection in cache mode is a
separate question and is left out of this PR.
- `mypy headroom/proxy/handlers/anthropic.py` stops in my local
environment on a `numpy` stub (`Type statement is only supported in
Python 3.12 and greater`) before reaching project code. That is
unrelated to this change, and CI runs mypy.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4606"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
@@ -2970,7 +2970,7 @@ class AnthropicHandlerMixin:
|
||||
|
||||
# Memory: Inject context and tools — gated on MemoryDecision.
|
||||
# ``inject`` is False under bypass, missing handler, missing
|
||||
# user_id, or HEADROOM_MEMORY_INJECTION_MODE in disabled/tool.
|
||||
# user_id, or HEADROOM_MEMORY_INJECTION_MODE=disabled.
|
||||
# Pre-PR-this the gate was a raw conjunction that silently
|
||||
# ignored bypass; now bypass is honoured here on Anthropic
|
||||
# /v1/messages just as on /v1/responses.
|
||||
@@ -2980,8 +2980,21 @@ class AnthropicHandlerMixin:
|
||||
# streaming path withholds their calls from the client.
|
||||
server_memory_tool_names: frozenset[str] = frozenset()
|
||||
if memory_decision.inject:
|
||||
if self.memory_handler.config.inject_context and is_cache_mode(self.config.mode):
|
||||
# Cache mode never injects context, so the next-turn prefix
|
||||
# bytes stay byte-equal to this turn's bytes. Skip the search
|
||||
# too: its result could only be discarded (#4600).
|
||||
from headroom.proxy.helpers import log_memory_injection
|
||||
|
||||
log_memory_injection(
|
||||
request_id=request_id,
|
||||
session_id=session_id,
|
||||
decision="skipped_cache_mode",
|
||||
bytes_injected=0,
|
||||
query=extract_user_query(optimized_messages) or "",
|
||||
)
|
||||
# Search and inject memory context
|
||||
if self.memory_handler.config.inject_context:
|
||||
elif self.memory_handler.config.inject_context:
|
||||
try:
|
||||
async with stage_timer.measure("memory_context"):
|
||||
memory_context = await asyncio.wait_for(
|
||||
@@ -3019,16 +3032,6 @@ class AnthropicHandlerMixin:
|
||||
bytes_injected=0,
|
||||
query=user_query,
|
||||
)
|
||||
elif is_cache_mode(self.config.mode):
|
||||
# Cache mode: skip injection entirely so the next-turn
|
||||
# prefix bytes remain byte-equal to this turn's bytes.
|
||||
log_memory_injection(
|
||||
request_id=request_id,
|
||||
session_id=session_id,
|
||||
decision="skipped_cache_mode",
|
||||
bytes_injected=0,
|
||||
query=user_query,
|
||||
)
|
||||
else:
|
||||
# P0-1 fix: route exclusively to the live zone tail
|
||||
# (latest non-frozen user turn). System prompt + frozen
|
||||
|
||||
@@ -193,6 +193,46 @@ def test_memory_context_appears_in_latest_user_message_tail() -> None:
|
||||
assert sent["messages"][2]["content"].endswith("MEMCTX")
|
||||
|
||||
|
||||
def test_cache_mode_skips_memory_context_search() -> None:
|
||||
"""Cache mode never injects memory context, so it must not search for it."""
|
||||
captured: dict[str, object] = {}
|
||||
with _make_proxy_client() as client:
|
||||
proxy = client.app.state.proxy
|
||||
proxy.config.mode = "cache"
|
||||
proxy.config.optimize = False
|
||||
proxy.config.image_optimize = False
|
||||
proxy.config.ccr_proactive_expansion = False
|
||||
|
||||
_install_session_tracker(proxy, frozen_count=1)
|
||||
_install_memory_handler(proxy)
|
||||
_install_capture_retry(proxy, captured)
|
||||
|
||||
response = client.post(
|
||||
"/v1/messages",
|
||||
headers={
|
||||
"x-api-key": "test-key",
|
||||
"anthropic-version": "2023-06-01",
|
||||
"x-headroom-user-id": "u1",
|
||||
},
|
||||
json={
|
||||
"model": "claude-sonnet-4-6",
|
||||
"max_tokens": 64,
|
||||
"system": "base system",
|
||||
"messages": [
|
||||
{"role": "user", "content": "frozen prefix"},
|
||||
{"role": "assistant", "content": "ack"},
|
||||
{"role": "user", "content": "latest user"},
|
||||
],
|
||||
},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
proxy.memory_handler.search_and_format_context.assert_not_awaited()
|
||||
sent = captured["body"]
|
||||
assert isinstance(sent, dict)
|
||||
assert sent["messages"][2]["content"] == "latest user"
|
||||
|
||||
|
||||
def test_memory_context_byte_deterministic_for_same_query() -> None:
|
||||
"""Two identical inbound requests produce identical outbound bytes."""
|
||||
capture_a: dict[str, object] = {}
|
||||
|
||||
Reference in New Issue
Block a user