fix(proxy): skip memory-context search in cache mode on OpenAI routes (#4632)

## Description

Follow-up to #4600 / #4606 as requested in #4622. PR #4606 implemented
this optimization for Anthropic `/v1/messages`. This PR brings the three
OpenAI routes (`handle_openai_chat`, `handle_openai_responses`, and
`handle_openai_responses_ws`) into alignment.

In `--mode cache`, memory context injection is disabled so prompt prefix
bytes remain byte-stable across turns. Previously, all three OpenAI
paths still executed the memory-context search
(`search_and_format_context`: embedding query, vector index lookups,
graph searches) and appended the result to the latest user turn in cache
mode, introducing latency (~1–37ms+ per turn) and altering the forwarded
user turn.

With this PR:
1. `handle_openai_chat`: checks `is_cache_mode(self.config.mode)` before
executing `search_and_format_context`, emits
`decision="skipped_cache_mode"`, and keeps the user turn intact.
2. `handle_openai_responses`: checks `is_cache_mode(self.config.mode)`
before searching, emits `decision="skipped_cache_mode"`, and keeps
`body["input"]` intact.
3. `handle_openai_responses_ws`: checks
`is_cache_mode(self.config.mode)` before measuring `memory_context`,
emits `decision="skipped_cache_mode"`, and leaves
`ws_response_body["input"]` untouched.
4. Memory tool injection remains completely unchanged across all routes.

Closes #4622

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Performance improvement

## Changes Made

- `headroom/proxy/handlers/openai.py`: Check
`is_cache_mode(self.config.mode)` when `inject_context` is enabled
across `handle_openai_chat`, `handle_openai_responses`, and
`handle_openai_responses_ws`. Emit `decision="skipped_cache_mode"` with
`bytes_injected=0` and bypass `search_and_format_context`.
- `tests/test_proxy_system_prompt_immutable.py`: Added
`test_cache_mode_skips_memory_context_search_openai_chat` and
`test_cache_mode_skips_memory_context_search_openai_responses` to verify
that `search_and_format_context` is never awaited in cache mode and user
message / input fields remain byte-equal to client values.

## Testing

- [x] Unit tests pass (`pytest
tests/test_proxy_system_prompt_immutable.py`)
- [x] Linting passes (`ruff check headroom/proxy/handlers/openai.py
tests/test_proxy_system_prompt_immutable.py`)
- [x] Formatting passes (`ruff format --check
headroom/proxy/handlers/openai.py
tests/test_proxy_system_prompt_immutable.py`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_proxy_system_prompt_immutable.py -q
10 passed, 1 warning in 31.92s

$ ruff check headroom/proxy/handlers/openai.py tests/test_proxy_system_prompt_immutable.py
All checks passed!

$ ruff format --check headroom/proxy/handlers/openai.py tests/test_proxy_system_prompt_immutable.py
2 files already formatted
```

## Real Behavior Proof

- Environment: Linux / WSL2 Ubuntu, Python 3.13, local proxy test
harness (`create_app` + `TestClient`).
- Exact command / steps: Executed requests against
`/v1/chat/completions` and `/v1/responses` with proxy configured in
`--mode cache` and memory context enabled. Verified upstream payload
captures and mock call history.
- Observed result: In cache mode, the memory search is bypassed on all
OpenAI requests; search_and_format_context was never awaited (0 calls),
user message content and input fields remain byte-equal without injected
memory context, and proxy emitted decision=skipped_cache_mode with
bytes_injected=0.
- Not tested: Live WebSocket connection to OpenAI upstream with
streaming audio/realtime endpoints.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no. Cache mode requires byte
stability across turns; memory search is skipped in cache mode. Token
mode is untouched.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: git revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation: n/a
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- Aligns OpenAI Chat, Responses, and WebSocket routes with Anthropic
behavior established in #4606.
- Preserves memory tool injection without modification.

Co-authored-by: shaiknaziya20 <naziyashaik5512@gmail.com>


### Maintainer combined-head validation

Reviewed head `66a09054548423e33af47121ac7e87134d023fe4` includes a
clean main sync through `12dca26aa` plus test-only maintainer
follow-ups: the existing WebSocket dummy config now explicitly uses
token mode, and a two-frame cache-mode regression covers skipped
searches, unchanged input, retained memory tools, passthrough
cancellation frames, and skip telemetry. No contributor runtime logic
was changed.

Windows / Python 3.12.13, with the existing local Rust extension:
`.venv/Scripts/python.exe -m pytest
tests/test_proxy_system_prompt_immutable.py
tests/test_codex_ws_per_frame_memory.py
tests/test_openai_codex_ws_lifecycle.py -q --tb=short
--disable-warnings` => **89 passed**. The new WebSocket regression fails
on current main because both frames still execute memory searches. Ruff
check/format and main-relative whitespace checks pass.

Additional runtime proof used a real Uvicorn proxy and a separate local
HTTP upstream, both bound to ephemeral loopback ports, with production
HTTP forwarding intact (no `_retry_request` patch). A deterministic
recording memory backend returned `LOOPBACK_MEMORY` when searched, so
the probe can distinguish an actual skip from an unconfigured/disabled
memory path. The probe sent the same authenticated Chat and Responses
requests in token mode, then cache mode, using user text `latest user
prompt` and fixed Responses instructions.

Exact invocation from the isolated worktree:
`PYTHONPATH=G:/git/headroom-review4632 HEADROOM_OFFLINE=1
G:/git/headroom/.venv/Scripts/python.exe
G:/git/headroom/.codex/review4632-loopback.py` (environment variables
set with PowerShell `$env:` assignments). Results: **four HTTP 200
responses**; token mode performed one memory search per route and
forwarded the returned memory text; cache mode performed zero searches
per route, forwarded the original user text unchanged, and recorded
`memory_skip_reason=cache_mode`. The proxy and upstream were shut down
after the run.

Limits: the loopback provider and recording memory backend are
deterministic local fixtures; no external provider, embedding model,
production memory service, load test, or real upstream WebSocket was
exercised. WebSocket multi-frame/input/tool/telemetry behavior is
covered by the focused regression above. Updated-head hosted
qualification is inspected by maintainers; no contributor CI action is
needed. Nothing merged.

---------

Co-authored-by: shaiknaziya20 <naziyashaik5512@gmail.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
AKSHITH BHARADWAJ D L T
2026-11-04 12:37:00 -06:00
committed by GitHub
co-authored by shaiknaziya20 JerrettDavis
parent 1bbc0ea89e
commit 365f6fe523
4 changed files with 208 additions and 3 deletions
+54 -3
View File
@@ -4372,7 +4372,24 @@ class OpenAIHandlerMixin:
# /v1/responses) — pre-PR-this site was the only chat
# path without one.
try:
if self.memory_handler.config.inject_context:
if self.memory_handler.config.inject_context and is_cache_mode(self.config.mode):
# Cache mode never injects context, so the next-turn prefix
# bytes stay byte-equal to this turn's bytes. Skip the search
# too: its result could only be discarded (#4600, #4622).
from headroom.proxy.helpers import log_memory_injection
from headroom.utils import extract_user_query
if tags is not None:
tags["memory_skip_reason"] = "cache_mode"
log_memory_injection(
request_id=request_id,
session_id=None,
decision="skipped_cache_mode",
bytes_injected=0,
query=extract_user_query(optimized_messages) or "",
tags=tags,
)
elif self.memory_handler.config.inject_context:
memory_context = await asyncio.wait_for(
self.memory_handler.search_and_format_context(
memory_user_id,
@@ -6211,7 +6228,24 @@ class OpenAIHandlerMixin:
# (latest non-frozen user item). Instructions are part of the
# cache hot zone and must never be mutated — invariant I2.
# See REALIGNMENT/03-phase-A-lockdown.md PR-A2.
if self.memory_handler.config.inject_context:
if self.memory_handler.config.inject_context and is_cache_mode(self.config.mode):
# Cache mode never injects context, so the next-turn prefix
# bytes stay byte-equal to this turn's bytes. Skip the search
# too: its result could only be discarded (#4600, #4622).
from headroom.proxy.helpers import log_memory_injection
from headroom.utils import extract_user_query
if tags is not None:
tags["memory_skip_reason"] = "cache_mode"
log_memory_injection(
request_id=request_id,
session_id=None,
decision="skipped_cache_mode",
bytes_injected=0,
query=extract_user_query(optimized_messages) or "",
tags=tags,
)
elif self.memory_handler.config.inject_context:
try:
memory_context = await asyncio.wait_for(
self.memory_handler.search_and_format_context(
@@ -8276,7 +8310,24 @@ class OpenAIHandlerMixin:
)
# Inject memory context into instructions
if self.memory_handler.config.inject_context:
if self.memory_handler.config.inject_context and is_cache_mode(
self.config.mode
):
# Cache mode never injects context, so the next-turn prefix
# bytes stay byte-equal to this turn's bytes. Skip the search
# too: its result could only be discarded (#4600, #4622).
if ws_tags is not None:
ws_tags["memory_skip_reason"] = "cache_mode"
ws_input = ws_response_body.get("input", "")
log_memory_injection(
request_id=request_id,
session_id=session_id,
decision="skipped_cache_mode",
bytes_injected=0,
query=ws_input if isinstance(ws_input, str) else "",
tags=ws_tags,
)
elif self.memory_handler.config.inject_context:
ws_input = ws_response_body.get("input", "")
ws_instructions = ws_response_body.get("instructions")
ws_msgs: list[dict[str, Any]] = []
+44
View File
@@ -196,6 +196,50 @@ async def test_memory_lookup_runs_for_each_issue_artifact_frame_and_preserves_no
assert forwarded_turns[0]["response"]["tools"] == forwarded_turns[1]["response"]["tools"]
@pytest.mark.asyncio
async def test_cache_mode_skips_context_on_each_frame_and_preserves_tools():
upstream = _FakeUpstream(
[
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
],
hold_after_events=True,
)
first, later = _issue_2059_inputs()
cancel = json.dumps({"type": "response.cancel"})
frames = [
_turn_with_tools(first, _client_response_tools()),
cancel,
_turn_with_tools(later, _client_response_tools()),
]
client_ws = _FakeWebSocket(frames=frames)
handler = _DummyOpenAIHandler()
handler.config.mode = "cache"
memory = _MemoryHandler()
handler.memory_handler = memory
with (
patch.dict(sys.modules, {"websockets": _make_fake_websockets_module(upstream)}),
patch("headroom.proxy.helpers.log_memory_injection") as log_injection,
):
await handler.handle_openai_responses_ws(client_ws)
assert memory.queries == []
assert upstream.sent[1] == cancel
forwarded = [json.loads(upstream.sent[i])["response"] for i in (0, 2)]
assert [response["input"] for response in forwarded] == [first, later]
expected_tools = [*_client_response_tools(), *_expected_memory_response_tools()]
assert all(response["tools"] == expected_tools for response in forwarded)
skips = [
call.kwargs
for call in log_injection.call_args_list
if call.kwargs.get("decision") == "skipped_cache_mode"
]
assert len(skips) == 2
assert all(call["bytes_injected"] == 0 for call in skips)
assert all(call["tags"]["memory_skip_reason"] == "cache_mode" for call in skips)
@pytest.mark.asyncio
@pytest.mark.parametrize("tools", [None, []], ids=["omitted", "empty"])
async def test_ws_responses_do_not_inject_memory_tools_without_client_tools(tools):
+1
View File
@@ -129,6 +129,7 @@ class _DummyOpenAIHandler(OpenAIHandlerMixin):
self.metrics = _DummyMetrics()
self.config = SimpleNamespace(
optimize=False,
mode="token",
retry_max_attempts=1,
retry_base_delay_ms=1,
retry_max_delay_ms=1,
+109
View File
@@ -21,6 +21,11 @@ from fastapi.testclient import TestClient
from headroom.proxy.server import ProxyConfig, create_app
@pytest.fixture(autouse=True)
def _stub_env(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("HEADROOM_REQUIRE_RUST_CORE", "false")
class _FakePrefixTracker:
def __init__(self, frozen_count: int):
self._frozen_count = frozen_count
@@ -474,3 +479,107 @@ def test_memory_context_appears_in_responses_api_input_tail() -> None:
assert any("MEMCTX" in t for t in text_parts)
elif isinstance(input_field, str):
assert input_field.endswith("MEMCTX")
def test_cache_mode_skips_memory_context_search_openai_chat() -> None:
"""Cache mode never injects memory context on OpenAI chat, so it must not search for it."""
captured: dict[str, object] = {}
with _make_proxy_client() as client:
proxy = client.app.state.proxy
proxy.config.mode = "cache"
proxy.config.optimize = False
proxy.config.image_optimize = False
proxy.config.ccr_proactive_expansion = False
_install_session_tracker(proxy, frozen_count=1)
_install_memory_handler(proxy)
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
captured["body"] = body
return httpx.Response(
200,
json={
"id": "chatcmpl_1",
"object": "chat.completion",
"choices": [
{
"index": 0,
"message": {"role": "assistant", "content": "ack"},
"finish_reason": "stop",
}
],
"usage": {"prompt_tokens": 10, "completion_tokens": 2, "total_tokens": 12},
},
)
proxy._retry_request = _fake_retry
response = client.post(
"/v1/chat/completions",
headers={
"authorization": "Bearer sk-test",
"x-headroom-user-id": "u1",
},
json={
"model": "gpt-4o",
"messages": [
{"role": "system", "content": "base system"},
{"role": "user", "content": "frozen prefix"},
{"role": "assistant", "content": "ack"},
{"role": "user", "content": "latest user"},
],
},
)
assert response.status_code == 200
proxy.memory_handler.search_and_format_context.assert_not_awaited()
sent = captured["body"]
assert isinstance(sent, dict)
assert sent["messages"][3]["content"] == "latest user"
def test_cache_mode_skips_memory_context_search_openai_responses() -> None:
"""Cache mode never injects memory context on OpenAI responses, so it must not search for it."""
captured: dict[str, object] = {}
with _make_responses_proxy_client() as client:
proxy = client.app.state.proxy
proxy.config.mode = "cache"
proxy.config.optimize = False
proxy.config.image_optimize = False
_install_memory_handler(proxy)
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
captured["body"] = body
return httpx.Response(
200,
json={
"id": "resp_1",
"object": "response",
"model": "gpt-5.4",
"output": [],
"usage": {"input_tokens": 5, "output_tokens": 1, "total_tokens": 6},
},
)
proxy._retry_request = _fake_retry
response = client.post(
"/v1/responses",
headers={
"authorization": "Bearer sk-test",
"x-headroom-user-id": "u1",
},
json={
"model": "gpt-5.4",
"instructions": "You are a Codex assistant.",
"input": "latest user string input",
},
)
assert response.status_code == 200
proxy.memory_handler.search_and_format_context.assert_not_awaited()
assert "body" in captured
sent = captured["body"]
assert isinstance(sent, dict)
assert sent.get("input") == "latest user string input"
assert sent.get("instructions") == "You are a Codex assistant."