Files
headroom/tests/test_codex_ws_per_frame_memory.py
Luis ArayaandJerrettDavis 36540379e0 fix(proxy): skip memory-context search on OpenAI paths in cache mode (#4634)
## Description

Follow-up to #4606. In `--mode cache`, the proxy preserves prompt cache
stability across turns by never mutating or appending to user turns.
While #4606 addressed this for Anthropic's `/v1/messages` by skipping
`search_and_format_context`, the three OpenAI-compatible paths
(`/v1/chat/completions`, `/v1/responses`, and `/v1/responses` WebSocket)
still executed the memory search and injected memory context into the
user turn even in cache mode.

This PR aligns the OpenAI handlers with Anthropic: when in cache mode
with `inject_context` enabled, it logs `skipped_cache_mode`, stamps
`memory_skip_reason="cache_mode"` in the request/frame tags, and skips
`search_and_format_context`. Memory tool injection and user input bytes
remain unchanged.

Closes #4622

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Performance improvement

## Changes Made

- `headroom/proxy/handlers/openai.py`:
- `handle_openai_chat`: skip `search_and_format_context` when
`is_cache_mode` is active, stamp `tags["memory_skip_reason"] =
"cache_mode"`, log `decision="skipped_cache_mode"`, and keep the latest
user message unmutated.
- `handle_openai_responses`: skip `search_and_format_context` when
`is_cache_mode` is active, stamp `tags["memory_skip_reason"] =
"cache_mode"`, and log `decision="skipped_cache_mode"`.
- `handle_openai_responses_ws`: skip `search_and_format_context` when
`is_cache_mode` is active, stamp `ws_tags["memory_skip_reason"] =
"cache_mode"`, log `decision="skipped_cache_mode"`, and avoid input
mutation.
- `tests/test_proxy_system_prompt_immutable.py`:
- Updated `test_cache_mode_skips_memory_context_search_openai_chat` to
verify `memory_skip_reason="cache_mode"` on recorded outcomes and
preservation of memory tools.
- Updated `test_cache_mode_skips_memory_context_search_openai_responses`
to verify `memory_skip_reason="cache_mode"` on recorded outcomes and
preservation of memory tools.
- `tests/test_codex_ws_per_frame_memory.py`:
- Expanded `test_cache_mode_skips_memory_context_search_ws` with
multiple response.create frames verifying input preservation, memory
tool preservation, and `memory_skip_reason="cache_mode"` in turn outcome
tags.

## Testing

- [x] Unit tests pass:
  - `tests/test_proxy_system_prompt_immutable.py`: 10 passed
  - `tests/test_codex_ws_per_frame_memory.py`: 12 passed
- [x] Linting passes: `ruff check` on modified files
- [x] Formatting passes: `ruff format --check` on modified files
- [x] New regression tests added for all 3 OpenAI paths in cache mode
- [x] Manual testing performed

### Test Output

```text
tests/test_proxy_system_prompt_immutable.py .......... [ 45%]
tests/test_codex_ws_per_frame_memory.py ............   [100%]
======================= 22 passed, 4 warnings in 18.06s =======================
```

## Real Behavior Proof

- Environment: Windows, Python 3.12.16, `HEADROOM_OFFLINE=1`.
- Exact command / steps:
1. Ran `pytest tests/test_proxy_system_prompt_immutable.py` exercising
`/v1/chat/completions` and `/v1/responses` in cache mode with mock
memory handler and outcome recorder.
2. Ran `pytest tests/test_codex_ws_per_frame_memory.py -k
test_cache_mode_skips_memory_context_search_ws` exercising two
`response.create` frames in cache mode with mock WebSocket and outcome
recorder.
3. Checked that `memory_handler.search_and_format_context` is never
awaited on any path.
4. Checked that `memory_skip_reason == "cache_mode"` reaches
`RequestOutcome.tags` across Chat, Responses, and WebSocket turns.
5. Checked that input bytes and client/memory tools (`client_tool`,
`mem_search`) remain untouched in the forwarded payloads.
- Observed result:
- `memory_handler.search_and_format_context.assert_not_awaited()` passed
on all routes.
- `outcome.tags.get("memory_skip_reason") == "cache_mode"` passed on
Chat, Responses, and WS frames.
- Forwarded inputs matched raw client inputs without injected memory
context.
- Forwarded tools retained `client_tool` and memory tools
(`mem_search`).
  - 22 tests passed across both test files.
- Not tested: Live cloud provider endpoints.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: yes; in cache mode, memory context is
no longer injected into OpenAI requests, ensuring next-turn prefix bytes
remain byte-identical, and request/frame tags now record
`memory_skip_reason="cache_mode"`.
- Kill switch / disable path: N/A (revert commit)
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I did not edit `CHANGELOG.md`

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-04 14:38:43 -06:00

548 lines
20 KiB
Python

from __future__ import annotations
import asyncio
import json
import re
import sys
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import patch
import pytest
from tests.test_openai_codex_ws_lifecycle import (
_DummyOpenAIHandler,
_FakeUpstream,
_FakeWebSocket,
_make_fake_websockets_module,
)
class _MemoryHandler:
def __init__(self) -> None:
self.config = SimpleNamespace(
inject_context=True,
inject_tools=True,
project_root_override=str(Path(__file__).resolve().parent),
)
self.queries: list[str] = []
async def search_and_format_context(self, _user_id, messages, **_kwargs):
current_turn = messages[-1]["content"] if messages else ""
self.queries.append(current_turn)
return f"current memory: {current_turn}"
def compute_memory_tool_definitions(self, _provider):
return [
{
"type": "function",
"function": {
"name": "memory_search",
"description": "search",
"parameters": {"type": "object"},
},
},
{
"type": "function",
"function": {
"name": "memory_save",
"description": "save",
"parameters": {"type": "object"},
},
},
]
def _expected_memory_response_tools() -> list[dict[str, object]]:
expected: list[dict[str, object]] = []
for tool in _MemoryHandler().compute_memory_tool_definitions("openai"):
function = tool["function"]
expected.append(
{
"type": "function",
"name": function["name"],
"description": function["description"],
"parameters": function["parameters"],
}
)
return expected
def _turn(text: str) -> str:
return json.dumps({"type": "response.create", "response": {"input": text}})
def _turn_with_tools(text: str, tools: list[dict[str, object]] | None) -> str:
response: dict[str, object] = {"input": text}
if tools is not None:
response["tools"] = tools
return json.dumps({"type": "response.create", "response": response})
def _client_response_tools() -> list[dict[str, object]]:
return [
{
"type": "function",
"name": "client_tool",
"description": "client tool",
"parameters": {"type": "object", "properties": {}},
}
]
def _direct_turn(text: str) -> str:
return json.dumps({"input": text})
def _issue_2059_artifact_path() -> Path:
return Path(__file__).resolve().parent / "fixtures" / "issues" / "headroom_issue_2059.json"
def _issue_2059_turns() -> tuple[str, str]:
issue_path = _issue_2059_artifact_path()
issue = json.loads(issue_path.read_text(encoding="utf-8"))
match = re.search(r"```json\s*(.*?)```", issue["body"], re.DOTALL)
assert match is not None, "issue 2059 artifact must contain a JSON code sample"
frames = [line.strip() for line in match.group(1).splitlines() if line.strip()]
assert len(frames) == 2, "issue 2059 artifact must contain exactly two frames"
return frames[0], frames[1]
def _issue_2059_inputs() -> tuple[str, str]:
first, later = _issue_2059_turns()
return (
json.loads(first)["response"]["input"],
json.loads(later)["response"]["input"],
)
def _list_turn(text: str, *, instructions: str) -> str:
return json.dumps(
{
"type": "response.create",
"response": {
"instructions": instructions,
"input": [
{
"type": "message",
"role": "user",
"content": [{"type": "input_text", "text": text}],
}
],
},
}
)
class _FlakyMemoryHandler(_MemoryHandler):
def __init__(self, *, fail_on: set[str]) -> None:
super().__init__()
self.fail_on = set(fail_on)
async def search_and_format_context(self, _user_id, messages, **_kwargs):
current_turn = messages[-1]["content"] if messages else ""
self.queries.append(current_turn)
if current_turn in self.fail_on:
raise RuntimeError(f"memory failed for {current_turn}")
return f"current memory: {current_turn}"
class _ToolFailingMemoryHandler(_MemoryHandler):
def __init__(self) -> None:
super().__init__()
self._fail_next_tools = True
def compute_memory_tool_definitions(self, _provider):
if self._fail_next_tools:
self._fail_next_tools = False
raise RuntimeError("memory tool preparation failed")
return super().compute_memory_tool_definitions(_provider)
@pytest.mark.asyncio
async def test_memory_lookup_runs_for_each_issue_artifact_frame_and_preserves_non_create_frames():
upstream = _FakeUpstream(
[
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
],
hold_after_events=True,
)
first_turn, later_turn = _issue_2059_turns()
first_input, later_input = _issue_2059_inputs()
client_frames = [
_turn_with_tools(json.loads(first_turn)["response"]["input"], _client_response_tools()),
json.dumps({"type": "response.cancel"}),
_turn_with_tools(json.loads(later_turn)["response"]["input"], _client_response_tools()),
]
client_ws = _FakeWebSocket(frames=client_frames)
handler = _DummyOpenAIHandler()
memory = _MemoryHandler()
handler.memory_handler = memory
with patch.dict(sys.modules, {"websockets": _make_fake_websockets_module(upstream)}):
await handler.handle_openai_responses_ws(client_ws)
assert memory.queries == [first_input, later_input]
assert upstream.sent[1] == client_frames[1]
forwarded_turns = [
json.loads(frame) for frame in upstream.sent if "response" in json.loads(frame)
]
assert f"current memory: {first_input}" in forwarded_turns[0]["response"]["input"]
assert f"current memory: {later_input}" in forwarded_turns[1]["response"]["input"]
expected_tools = [*_client_response_tools(), *_expected_memory_response_tools()]
for frame in forwarded_turns:
assert frame["response"]["tools"] == expected_tools
assert forwarded_turns[0]["response"]["tools"] == forwarded_turns[1]["response"]["tools"]
@pytest.mark.asyncio
async def test_cache_mode_skips_context_on_each_frame_and_preserves_tools():
upstream = _FakeUpstream(
[
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
],
hold_after_events=True,
)
first, later = _issue_2059_inputs()
cancel = json.dumps({"type": "response.cancel"})
frames = [
_turn_with_tools(first, _client_response_tools()),
cancel,
_turn_with_tools(later, _client_response_tools()),
]
client_ws = _FakeWebSocket(frames=frames)
handler = _DummyOpenAIHandler()
handler.config.mode = "cache"
memory = _MemoryHandler()
handler.memory_handler = memory
with (
patch.dict(sys.modules, {"websockets": _make_fake_websockets_module(upstream)}),
patch("headroom.proxy.helpers.log_memory_injection") as log_injection,
):
await handler.handle_openai_responses_ws(client_ws)
assert memory.queries == []
assert upstream.sent[1] == cancel
forwarded = [json.loads(upstream.sent[i])["response"] for i in (0, 2)]
assert [response["input"] for response in forwarded] == [first, later]
expected_tools = [*_client_response_tools(), *_expected_memory_response_tools()]
assert all(response["tools"] == expected_tools for response in forwarded)
skips = [
call.kwargs
for call in log_injection.call_args_list
if call.kwargs.get("decision") == "skipped_cache_mode"
]
assert len(skips) == 2
assert all(call["bytes_injected"] == 0 for call in skips)
assert all(call["tags"]["memory_skip_reason"] == "cache_mode" for call in skips)
@pytest.mark.asyncio
@pytest.mark.parametrize("tools", [None, []], ids=["omitted", "empty"])
async def test_ws_responses_do_not_inject_memory_tools_without_client_tools(tools):
upstream = _FakeUpstream(
[
json.dumps({"type": "response.created", "response": {"id": "r_no_tools"}}),
json.dumps({"type": "response.completed", "response": {"id": "r_no_tools"}}),
]
)
client_ws = _FakeWebSocket(
frames=[_turn_with_tools("no tools requested", tools)],
headers={"authorization": "Bearer test", "x-client": "unknown-client"},
)
handler = _DummyOpenAIHandler()
handler.memory_handler = _MemoryHandler()
with patch.dict(sys.modules, {"websockets": _make_fake_websockets_module(upstream)}):
await handler.handle_openai_responses_ws(client_ws)
forwarded = [json.loads(frame) for frame in upstream.sent if "response" in json.loads(frame)]
assert forwarded
assert forwarded[0]["response"].get("tools", []) == []
@pytest.mark.asyncio
async def test_memory_lookup_skips_input_bearing_non_create_first_frame():
upstream = _FakeUpstream(
[
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
],
hold_after_events=True,
)
_first_input, later_input = _issue_2059_inputs()
cancel_frame = json.dumps(
{
"type": "response.cancel",
"response_id": "r_1",
"input": "must not query",
}
)
later_turn = _issue_2059_turns()[1]
client_ws = _FakeWebSocket(frames=[cancel_frame, later_turn])
handler = _DummyOpenAIHandler()
memory = _MemoryHandler()
handler.memory_handler = memory
with patch.dict(sys.modules, {"websockets": _make_fake_websockets_module(upstream)}):
await handler.handle_openai_responses_ws(client_ws)
assert memory.queries == [later_input]
assert upstream.sent[0] == cancel_frame
forwarded_later = json.loads(upstream.sent[1])
assert f"current memory: {later_input}" in forwarded_later["response"]["input"]
@pytest.mark.asyncio
async def test_cache_mode_skips_memory_context_search_ws():
upstream = _FakeUpstream(
[
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
json.dumps({"type": "response.created", "response": {"id": "r_2"}}),
json.dumps({"type": "response.completed", "response": {"id": "r_2"}}),
],
hold_after_events=True,
)
first_turn, later_turn = _issue_2059_turns()
client_frames = [
_turn_with_tools(json.loads(first_turn)["response"]["input"], _client_response_tools()),
_turn_with_tools(json.loads(later_turn)["response"]["input"], _client_response_tools()),
]
client_ws = _FakeWebSocket(frames=client_frames)
handler = _DummyOpenAIHandler()
handler.config.mode = "cache"
memory = _MemoryHandler()
handler.memory_handler = memory
recorded_outcomes = []
async def _capture_outcome(outcome):
recorded_outcomes.append(outcome)
handler._record_request_outcome = _capture_outcome
with patch.dict(sys.modules, {"websockets": _make_fake_websockets_module(upstream)}):
await handler.handle_openai_responses_ws(client_ws)
assert memory.queries == []
assert len(upstream.sent) == 2
for i, sent_raw in enumerate(upstream.sent):
sent = json.loads(sent_raw)
original = json.loads(client_frames[i])
assert sent["response"]["input"] == original["response"]["input"]
expected_tools = [*_client_response_tools(), *_expected_memory_response_tools()]
assert sent["response"]["tools"] == expected_tools
assert len(recorded_outcomes) >= 1
for outcome in recorded_outcomes:
assert outcome.tags.get("memory_skip_reason") == "cache_mode"
@pytest.mark.asyncio
async def test_memory_lookup_skips_bypassed_frames():
upstream = _FakeUpstream(
[
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
],
hold_after_events=True,
)
first, later = _issue_2059_turns()
client_ws = _FakeWebSocket(
frames=[first, later],
headers={"authorization": "Bearer test", "x-headroom-bypass": "true"},
)
handler = _DummyOpenAIHandler()
memory = _MemoryHandler()
handler.memory_handler = memory
with patch.dict(sys.modules, {"websockets": _make_fake_websockets_module(upstream)}):
await handler.handle_openai_responses_ws(client_ws)
assert memory.queries == []
assert upstream.sent == [first, later]
@pytest.mark.asyncio
async def test_memory_lookup_keeps_legacy_direct_first_frame():
upstream = _FakeUpstream(
[
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
],
hold_after_events=True,
)
first_input, later_input = _issue_2059_inputs()
first = _direct_turn(first_input)
later = _issue_2059_turns()[1]
client_ws = _FakeWebSocket(frames=[first, later])
handler = _DummyOpenAIHandler()
memory = _MemoryHandler()
handler.memory_handler = memory
with patch.dict(sys.modules, {"websockets": _make_fake_websockets_module(upstream)}):
await handler.handle_openai_responses_ws(client_ws)
assert memory.queries == [first_input, later_input]
forwarded_first = json.loads(upstream.sent[0])
forwarded_later = json.loads(upstream.sent[1])
assert f"current memory: {first_input}" in forwarded_first["input"]
assert f"current memory: {later_input}" in forwarded_later["response"]["input"]
@pytest.mark.asyncio
async def test_memory_lookup_skips_disabled_memory(monkeypatch):
monkeypatch.setenv("HEADROOM_MEMORY_INJECTION_MODE", "disabled")
upstream = _FakeUpstream(
[
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
],
hold_after_events=True,
)
first, later = _issue_2059_turns()
client_ws = _FakeWebSocket(frames=[first, later])
handler = _DummyOpenAIHandler()
memory = _MemoryHandler()
handler.memory_handler = memory
with patch.dict(sys.modules, {"websockets": _make_fake_websockets_module(upstream)}):
await handler.handle_openai_responses_ws(client_ws)
assert memory.queries == []
assert upstream.sent == [first, later]
@pytest.mark.asyncio
async def test_memory_lookup_fails_open_and_recovers_on_later_frame():
first, later = _issue_2059_turns()
first_input, later_input = _issue_2059_inputs()
upstream = _FakeUpstream([], hold_after_events=True)
client_ws = _FakeWebSocket(frames=[first, later], hold_after_initial=True)
handler = _DummyOpenAIHandler()
memory = _FlakyMemoryHandler(fail_on={first_input})
handler.memory_handler = memory
async def _trigger() -> None:
await asyncio.sleep(0.05)
client_ws.trigger_disconnect()
with patch.dict(sys.modules, {"websockets": _make_fake_websockets_module(upstream)}):
trigger_task = asyncio.create_task(_trigger())
try:
await handler.handle_openai_responses_ws(client_ws)
finally:
trigger_task.cancel()
try:
await trigger_task
except asyncio.CancelledError:
pass
assert memory.queries == [first_input, later_input]
assert upstream.sent[0] == first
assert f"current memory: {later_input}" in json.loads(upstream.sent[1])["response"]["input"]
@pytest.mark.asyncio
async def test_memory_lookup_fails_open_when_tool_preparation_raises():
first, later = _issue_2059_turns()
first_input, later_input = _issue_2059_inputs()
upstream = _FakeUpstream([], hold_after_events=True)
client_ws = _FakeWebSocket(frames=[first, later], hold_after_initial=True)
handler = _DummyOpenAIHandler()
memory = _ToolFailingMemoryHandler()
handler.memory_handler = memory
async def _trigger() -> None:
await asyncio.sleep(0.05)
client_ws.trigger_disconnect()
with patch.dict(sys.modules, {"websockets": _make_fake_websockets_module(upstream)}):
trigger_task = asyncio.create_task(_trigger())
try:
await handler.handle_openai_responses_ws(client_ws)
finally:
trigger_task.cancel()
try:
await trigger_task
except asyncio.CancelledError:
pass
assert memory.queries == [first_input, later_input]
assert upstream.sent[0] == first
assert f"current memory: {later_input}" in json.loads(upstream.sent[1])["response"]["input"]
@pytest.mark.asyncio
async def test_memory_lookup_preserves_list_shaped_later_frame_input():
first, _later = _issue_2059_turns()
list_frame = _list_turn(
"later turn with list payload",
instructions="list payload instructions",
)
expected_input = json.loads(list_frame)["response"]["input"]
upstream = _FakeUpstream([], hold_after_events=True)
client_ws = _FakeWebSocket(frames=[first, list_frame], hold_after_initial=True)
handler = _DummyOpenAIHandler()
memory = _MemoryHandler()
handler.memory_handler = memory
async def _trigger() -> None:
await asyncio.sleep(0.05)
client_ws.trigger_disconnect()
with patch.dict(sys.modules, {"websockets": _make_fake_websockets_module(upstream)}):
trigger_task = asyncio.create_task(_trigger())
try:
await handler.handle_openai_responses_ws(client_ws)
finally:
trigger_task.cancel()
try:
await trigger_task
except asyncio.CancelledError:
pass
forwarded_later = json.loads(upstream.sent[1])
assert forwarded_later["response"]["input"] == expected_input
assert memory.queries[-1] == "list payload instructions"
@pytest.mark.asyncio
async def test_later_frame_compression_receives_memory_prepared_input():
first, later = _issue_2059_turns()
_first_input, later_input = _issue_2059_inputs()
upstream = _FakeUpstream([], hold_after_events=True)
client_ws = _FakeWebSocket(frames=[first, later], hold_after_initial=True)
handler = _DummyOpenAIHandler()
handler.config.optimize = True
memory = _MemoryHandler()
handler.memory_handler = memory
seen_inputs: list[object] = []
def _capture_compress(payload, *, model, request_id, timing=None):
seen_inputs.append(payload["input"])
return payload, False, 0, [], "test_noop", 10, 10, 0
async def _trigger() -> None:
await asyncio.sleep(0.05)
client_ws.trigger_disconnect()
handler._compress_openai_responses_payload = _capture_compress # type: ignore[method-assign]
with patch.dict(sys.modules, {"websockets": _make_fake_websockets_module(upstream)}):
trigger_task = asyncio.create_task(_trigger())
try:
await handler.handle_openai_responses_ws(client_ws)
finally:
trigger_task.cancel()
try:
await trigger_task
except asyncio.CancelledError:
pass
assert len(seen_inputs) == 2
assert f"current memory: {later_input}" in str(seen_inputs[1])