fix(proxy): skip memory-context search on OpenAI paths in cache mode (#4634)

## Description

Follow-up to #4606. In `--mode cache`, the proxy preserves prompt cache
stability across turns by never mutating or appending to user turns.
While #4606 addressed this for Anthropic's `/v1/messages` by skipping
`search_and_format_context`, the three OpenAI-compatible paths
(`/v1/chat/completions`, `/v1/responses`, and `/v1/responses` WebSocket)
still executed the memory search and injected memory context into the
user turn even in cache mode.

This PR aligns the OpenAI handlers with Anthropic: when in cache mode
with `inject_context` enabled, it logs `skipped_cache_mode`, stamps
`memory_skip_reason="cache_mode"` in the request/frame tags, and skips
`search_and_format_context`. Memory tool injection and user input bytes
remain unchanged.

Closes #4622

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Performance improvement

## Changes Made

- `headroom/proxy/handlers/openai.py`:
- `handle_openai_chat`: skip `search_and_format_context` when
`is_cache_mode` is active, stamp `tags["memory_skip_reason"] =
"cache_mode"`, log `decision="skipped_cache_mode"`, and keep the latest
user message unmutated.
- `handle_openai_responses`: skip `search_and_format_context` when
`is_cache_mode` is active, stamp `tags["memory_skip_reason"] =
"cache_mode"`, and log `decision="skipped_cache_mode"`.
- `handle_openai_responses_ws`: skip `search_and_format_context` when
`is_cache_mode` is active, stamp `ws_tags["memory_skip_reason"] =
"cache_mode"`, log `decision="skipped_cache_mode"`, and avoid input
mutation.
- `tests/test_proxy_system_prompt_immutable.py`:
- Updated `test_cache_mode_skips_memory_context_search_openai_chat` to
verify `memory_skip_reason="cache_mode"` on recorded outcomes and
preservation of memory tools.
- Updated `test_cache_mode_skips_memory_context_search_openai_responses`
to verify `memory_skip_reason="cache_mode"` on recorded outcomes and
preservation of memory tools.
- `tests/test_codex_ws_per_frame_memory.py`:
- Expanded `test_cache_mode_skips_memory_context_search_ws` with
multiple response.create frames verifying input preservation, memory
tool preservation, and `memory_skip_reason="cache_mode"` in turn outcome
tags.

## Testing

- [x] Unit tests pass:
  - `tests/test_proxy_system_prompt_immutable.py`: 10 passed
  - `tests/test_codex_ws_per_frame_memory.py`: 12 passed
- [x] Linting passes: `ruff check` on modified files
- [x] Formatting passes: `ruff format --check` on modified files
- [x] New regression tests added for all 3 OpenAI paths in cache mode
- [x] Manual testing performed

### Test Output

```text
tests/test_proxy_system_prompt_immutable.py .......... [ 45%]
tests/test_codex_ws_per_frame_memory.py ............   [100%]
======================= 22 passed, 4 warnings in 18.06s =======================
```

## Real Behavior Proof

- Environment: Windows, Python 3.12.16, `HEADROOM_OFFLINE=1`.
- Exact command / steps:
1. Ran `pytest tests/test_proxy_system_prompt_immutable.py` exercising
`/v1/chat/completions` and `/v1/responses` in cache mode with mock
memory handler and outcome recorder.
2. Ran `pytest tests/test_codex_ws_per_frame_memory.py -k
test_cache_mode_skips_memory_context_search_ws` exercising two
`response.create` frames in cache mode with mock WebSocket and outcome
recorder.
3. Checked that `memory_handler.search_and_format_context` is never
awaited on any path.
4. Checked that `memory_skip_reason == "cache_mode"` reaches
`RequestOutcome.tags` across Chat, Responses, and WebSocket turns.
5. Checked that input bytes and client/memory tools (`client_tool`,
`mem_search`) remain untouched in the forwarded payloads.
- Observed result:
- `memory_handler.search_and_format_context.assert_not_awaited()` passed
on all routes.
- `outcome.tags.get("memory_skip_reason") == "cache_mode"` passed on
Chat, Responses, and WS frames.
- Forwarded inputs matched raw client inputs without injected memory
context.
- Forwarded tools retained `client_tool` and memory tools
(`mem_search`).
  - 22 tests passed across both test files.
- Not tested: Live cloud provider endpoints.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: yes; in cache mode, memory context is
no longer injected into OpenAI requests, ensuring next-turn prefix bytes
remain byte-identical, and request/frame tags now record
`memory_skip_reason="cache_mode"`.
- Kill switch / disable path: N/A (revert commit)
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I did not edit `CHANGELOG.md`

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Luis Araya
2026-11-04 14:38:43 -06:00
committed by GitHub
co-authored by JerrettDavis
parent 365f6fe523
commit 36540379e0
3 changed files with 194 additions and 21 deletions
+19 -21
View File
@@ -4372,15 +4372,12 @@ class OpenAIHandlerMixin:
# /v1/responses) — pre-PR-this site was the only chat
# path without one.
try:
if self.memory_handler.config.inject_context and is_cache_mode(self.config.mode):
# Cache mode never injects context, so the next-turn prefix
# bytes stay byte-equal to this turn's bytes. Skip the search
# too: its result could only be discarded (#4600, #4622).
if self.memory_handler.config.inject_context and is_cache_mode(
getattr(getattr(self, "config", None), "mode", "token")
):
from headroom.proxy.helpers import log_memory_injection
from headroom.utils import extract_user_query
if tags is not None:
tags["memory_skip_reason"] = "cache_mode"
tags["memory_skip_reason"] = "cache_mode"
log_memory_injection(
request_id=request_id,
session_id=None,
@@ -6228,15 +6225,12 @@ class OpenAIHandlerMixin:
# (latest non-frozen user item). Instructions are part of the
# cache hot zone and must never be mutated — invariant I2.
# See REALIGNMENT/03-phase-A-lockdown.md PR-A2.
if self.memory_handler.config.inject_context and is_cache_mode(self.config.mode):
# Cache mode never injects context, so the next-turn prefix
# bytes stay byte-equal to this turn's bytes. Skip the search
# too: its result could only be discarded (#4600, #4622).
if self.memory_handler.config.inject_context and is_cache_mode(
getattr(getattr(self, "config", None), "mode", "token")
):
from headroom.proxy.helpers import log_memory_injection
from headroom.utils import extract_user_query
if tags is not None:
tags["memory_skip_reason"] = "cache_mode"
tags["memory_skip_reason"] = "cache_mode"
log_memory_injection(
request_id=request_id,
session_id=None,
@@ -8311,20 +8305,24 @@ class OpenAIHandlerMixin:
# Inject memory context into instructions
if self.memory_handler.config.inject_context and is_cache_mode(
self.config.mode
getattr(getattr(self, "config", None), "mode", "token")
):
# Cache mode never injects context, so the next-turn prefix
# bytes stay byte-equal to this turn's bytes. Skip the search
# too: its result could only be discarded (#4600, #4622).
if ws_tags is not None:
ws_tags["memory_skip_reason"] = "cache_mode"
from headroom.utils import extract_user_query
ws_tags["memory_skip_reason"] = "cache_mode"
ws_input = ws_response_body.get("input", "")
ws_instructions = ws_response_body.get("instructions")
ws_msgs: list[dict[str, Any]] = []
if ws_instructions:
ws_msgs.append({"role": "system", "content": ws_instructions})
if isinstance(ws_input, str) and ws_input:
ws_msgs.append({"role": "user", "content": ws_input})
log_memory_injection(
request_id=request_id,
session_id=session_id,
decision="skipped_cache_mode",
bytes_injected=0,
query=ws_input if isinstance(ws_input, str) else "",
query=extract_user_query(ws_msgs) or "",
tags=ws_tags,
)
elif self.memory_handler.config.inject_context:
+45
View File
@@ -296,6 +296,51 @@ async def test_memory_lookup_skips_input_bearing_non_create_first_frame():
assert f"current memory: {later_input}" in forwarded_later["response"]["input"]
@pytest.mark.asyncio
async def test_cache_mode_skips_memory_context_search_ws():
upstream = _FakeUpstream(
[
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
json.dumps({"type": "response.created", "response": {"id": "r_2"}}),
json.dumps({"type": "response.completed", "response": {"id": "r_2"}}),
],
hold_after_events=True,
)
first_turn, later_turn = _issue_2059_turns()
client_frames = [
_turn_with_tools(json.loads(first_turn)["response"]["input"], _client_response_tools()),
_turn_with_tools(json.loads(later_turn)["response"]["input"], _client_response_tools()),
]
client_ws = _FakeWebSocket(frames=client_frames)
handler = _DummyOpenAIHandler()
handler.config.mode = "cache"
memory = _MemoryHandler()
handler.memory_handler = memory
recorded_outcomes = []
async def _capture_outcome(outcome):
recorded_outcomes.append(outcome)
handler._record_request_outcome = _capture_outcome
with patch.dict(sys.modules, {"websockets": _make_fake_websockets_module(upstream)}):
await handler.handle_openai_responses_ws(client_ws)
assert memory.queries == []
assert len(upstream.sent) == 2
for i, sent_raw in enumerate(upstream.sent):
sent = json.loads(sent_raw)
original = json.loads(client_frames[i])
assert sent["response"]["input"] == original["response"]["input"]
expected_tools = [*_client_response_tools(), *_expected_memory_response_tools()]
assert sent["response"]["tools"] == expected_tools
assert len(recorded_outcomes) >= 1
for outcome in recorded_outcomes:
assert outcome.tags.get("memory_skip_reason") == "cache_mode"
@pytest.mark.asyncio
async def test_memory_lookup_skips_bypassed_frames():
upstream = _FakeUpstream(
+130
View File
@@ -583,3 +583,133 @@ def test_cache_mode_skips_memory_context_search_openai_responses() -> None:
assert isinstance(sent, dict)
assert sent.get("input") == "latest user string input"
assert sent.get("instructions") == "You are a Codex assistant."
def test_cache_mode_chat_preserves_tools_and_outcome_skip_reason() -> None:
"""Cache mode never injects memory context on OpenAI /v1/chat/completions."""
captured: dict[str, object] = {}
recorded_outcomes = []
with _make_proxy_client() as client:
proxy = client.app.state.proxy
proxy.config.mode = "cache"
proxy.config.optimize = False
proxy.config.image_optimize = False
proxy.config.ccr_proactive_expansion = False
async def _capture_outcome(outcome):
recorded_outcomes.append(outcome)
proxy._record_request_outcome = _capture_outcome
_install_session_tracker(proxy, frozen_count=1)
_install_memory_handler(proxy)
proxy.memory_handler.config.inject_tools = True
proxy.memory_handler.compute_memory_tool_definitions = lambda _provider: [
{"type": "function", "function": {"name": "mem_search", "parameters": {}}}
]
_install_capture_retry(proxy, captured)
response = client.post(
"/v1/chat/completions",
headers={
"authorization": "Bearer sk-test",
"x-headroom-user-id": "u1",
},
json={
"model": "gpt-4o",
"messages": [
{"role": "system", "content": "base system"},
{"role": "user", "content": "latest user"},
],
"tools": [
{
"type": "function",
"function": {"name": "client_tool", "parameters": {}},
}
],
},
)
assert response.status_code == 200
proxy.memory_handler.search_and_format_context.assert_not_awaited()
sent = captured["body"]
assert isinstance(sent, dict)
assert sent["messages"][1]["content"] == "latest user"
# Memory tools remain present even when context search is skipped
assert "tools" in sent
tool_names = [t.get("function", {}).get("name") for t in sent["tools"]]
assert "client_tool" in tool_names
assert "mem_search" in tool_names
# Outcome tags must record memory_skip_reason="cache_mode"
assert len(recorded_outcomes) >= 1
assert recorded_outcomes[0].tags.get("memory_skip_reason") == "cache_mode"
def test_cache_mode_responses_preserves_tools_and_outcome_skip_reason() -> None:
"""Cache mode never injects memory context on OpenAI /v1/responses."""
captured: dict[str, object] = {}
recorded_outcomes = []
with _make_responses_proxy_client() as client:
proxy = client.app.state.proxy
proxy.config.mode = "cache"
proxy.config.optimize = False
proxy.config.image_optimize = False
async def _capture_outcome(outcome):
recorded_outcomes.append(outcome)
proxy._record_request_outcome = _capture_outcome
_install_memory_handler(proxy)
proxy.memory_handler.config.inject_tools = True
proxy.memory_handler.compute_memory_tool_definitions = lambda _provider: [
{"type": "function", "function": {"name": "mem_search", "parameters": {}}}
]
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
captured["body"] = body
return httpx.Response(
200,
json={
"id": "resp_1",
"object": "response",
"model": "gpt-5.4",
"output": [],
"usage": {"input_tokens": 5, "output_tokens": 1, "total_tokens": 6},
},
)
proxy._retry_request = _fake_retry
response = client.post(
"/v1/responses",
headers={
"authorization": "Bearer sk-test",
"x-headroom-user-id": "u1",
},
json={
"model": "gpt-5.4",
"instructions": "You are a Codex assistant.",
"input": "latest user prompt",
"tools": [
{
"type": "function",
"name": "client_tool",
"description": "client tool",
"parameters": {},
}
],
},
)
assert response.status_code == 200
proxy.memory_handler.search_and_format_context.assert_not_awaited()
assert "body" in captured
sent = captured["body"]
assert isinstance(sent, dict)
assert sent.get("input") == "latest user prompt"
assert "tools" in sent
tool_names = [t.get("name") for t in sent["tools"]]
assert "client_tool" in tool_names
assert "mem_search" in tool_names
assert len(recorded_outcomes) >= 1
assert recorded_outcomes[0].tags.get("memory_skip_reason") == "cache_mode"