mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): skip memory-context search on OpenAI paths in cache mode (#4634)
## Description Follow-up to #4606. In `--mode cache`, the proxy preserves prompt cache stability across turns by never mutating or appending to user turns. While #4606 addressed this for Anthropic's `/v1/messages` by skipping `search_and_format_context`, the three OpenAI-compatible paths (`/v1/chat/completions`, `/v1/responses`, and `/v1/responses` WebSocket) still executed the memory search and injected memory context into the user turn even in cache mode. This PR aligns the OpenAI handlers with Anthropic: when in cache mode with `inject_context` enabled, it logs `skipped_cache_mode`, stamps `memory_skip_reason="cache_mode"` in the request/frame tags, and skips `search_and_format_context`. Memory tool injection and user input bytes remain unchanged. Closes #4622 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Performance improvement ## Changes Made - `headroom/proxy/handlers/openai.py`: - `handle_openai_chat`: skip `search_and_format_context` when `is_cache_mode` is active, stamp `tags["memory_skip_reason"] = "cache_mode"`, log `decision="skipped_cache_mode"`, and keep the latest user message unmutated. - `handle_openai_responses`: skip `search_and_format_context` when `is_cache_mode` is active, stamp `tags["memory_skip_reason"] = "cache_mode"`, and log `decision="skipped_cache_mode"`. - `handle_openai_responses_ws`: skip `search_and_format_context` when `is_cache_mode` is active, stamp `ws_tags["memory_skip_reason"] = "cache_mode"`, log `decision="skipped_cache_mode"`, and avoid input mutation. - `tests/test_proxy_system_prompt_immutable.py`: - Updated `test_cache_mode_skips_memory_context_search_openai_chat` to verify `memory_skip_reason="cache_mode"` on recorded outcomes and preservation of memory tools. - Updated `test_cache_mode_skips_memory_context_search_openai_responses` to verify `memory_skip_reason="cache_mode"` on recorded outcomes and preservation of memory tools. - `tests/test_codex_ws_per_frame_memory.py`: - Expanded `test_cache_mode_skips_memory_context_search_ws` with multiple response.create frames verifying input preservation, memory tool preservation, and `memory_skip_reason="cache_mode"` in turn outcome tags. ## Testing - [x] Unit tests pass: - `tests/test_proxy_system_prompt_immutable.py`: 10 passed - `tests/test_codex_ws_per_frame_memory.py`: 12 passed - [x] Linting passes: `ruff check` on modified files - [x] Formatting passes: `ruff format --check` on modified files - [x] New regression tests added for all 3 OpenAI paths in cache mode - [x] Manual testing performed ### Test Output ```text tests/test_proxy_system_prompt_immutable.py .......... [ 45%] tests/test_codex_ws_per_frame_memory.py ............ [100%] ======================= 22 passed, 4 warnings in 18.06s ======================= ``` ## Real Behavior Proof - Environment: Windows, Python 3.12.16, `HEADROOM_OFFLINE=1`. - Exact command / steps: 1. Ran `pytest tests/test_proxy_system_prompt_immutable.py` exercising `/v1/chat/completions` and `/v1/responses` in cache mode with mock memory handler and outcome recorder. 2. Ran `pytest tests/test_codex_ws_per_frame_memory.py -k test_cache_mode_skips_memory_context_search_ws` exercising two `response.create` frames in cache mode with mock WebSocket and outcome recorder. 3. Checked that `memory_handler.search_and_format_context` is never awaited on any path. 4. Checked that `memory_skip_reason == "cache_mode"` reaches `RequestOutcome.tags` across Chat, Responses, and WebSocket turns. 5. Checked that input bytes and client/memory tools (`client_tool`, `mem_search`) remain untouched in the forwarded payloads. - Observed result: - `memory_handler.search_and_format_context.assert_not_awaited()` passed on all routes. - `outcome.tags.get("memory_skip_reason") == "cache_mode"` passed on Chat, Responses, and WS frames. - Forwarded inputs matched raw client inputs without injected memory context. - Forwarded tools retained `client_tool` and memory tools (`mem_search`). - 22 tests passed across both test files. - Not tested: Live cloud provider endpoints. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: N/A - Stable/default behavior changed: yes; in cache mode, memory context is no longer injected into OpenAI requests, ensuring next-turn prefix bytes remain byte-identical, and request/frame tags now record `memory_skip_reason="cache_mode"`. - Kill switch / disable path: N/A (revert commit) - Unsafe override required: no - Qualification impact: none - Rollback path: revert this PR ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I did not edit `CHANGELOG.md` --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
JerrettDavis
parent
365f6fe523
commit
36540379e0
@@ -4372,15 +4372,12 @@ class OpenAIHandlerMixin:
|
||||
# /v1/responses) — pre-PR-this site was the only chat
|
||||
# path without one.
|
||||
try:
|
||||
if self.memory_handler.config.inject_context and is_cache_mode(self.config.mode):
|
||||
# Cache mode never injects context, so the next-turn prefix
|
||||
# bytes stay byte-equal to this turn's bytes. Skip the search
|
||||
# too: its result could only be discarded (#4600, #4622).
|
||||
if self.memory_handler.config.inject_context and is_cache_mode(
|
||||
getattr(getattr(self, "config", None), "mode", "token")
|
||||
):
|
||||
from headroom.proxy.helpers import log_memory_injection
|
||||
from headroom.utils import extract_user_query
|
||||
|
||||
if tags is not None:
|
||||
tags["memory_skip_reason"] = "cache_mode"
|
||||
tags["memory_skip_reason"] = "cache_mode"
|
||||
log_memory_injection(
|
||||
request_id=request_id,
|
||||
session_id=None,
|
||||
@@ -6228,15 +6225,12 @@ class OpenAIHandlerMixin:
|
||||
# (latest non-frozen user item). Instructions are part of the
|
||||
# cache hot zone and must never be mutated — invariant I2.
|
||||
# See REALIGNMENT/03-phase-A-lockdown.md PR-A2.
|
||||
if self.memory_handler.config.inject_context and is_cache_mode(self.config.mode):
|
||||
# Cache mode never injects context, so the next-turn prefix
|
||||
# bytes stay byte-equal to this turn's bytes. Skip the search
|
||||
# too: its result could only be discarded (#4600, #4622).
|
||||
if self.memory_handler.config.inject_context and is_cache_mode(
|
||||
getattr(getattr(self, "config", None), "mode", "token")
|
||||
):
|
||||
from headroom.proxy.helpers import log_memory_injection
|
||||
from headroom.utils import extract_user_query
|
||||
|
||||
if tags is not None:
|
||||
tags["memory_skip_reason"] = "cache_mode"
|
||||
tags["memory_skip_reason"] = "cache_mode"
|
||||
log_memory_injection(
|
||||
request_id=request_id,
|
||||
session_id=None,
|
||||
@@ -8311,20 +8305,24 @@ class OpenAIHandlerMixin:
|
||||
|
||||
# Inject memory context into instructions
|
||||
if self.memory_handler.config.inject_context and is_cache_mode(
|
||||
self.config.mode
|
||||
getattr(getattr(self, "config", None), "mode", "token")
|
||||
):
|
||||
# Cache mode never injects context, so the next-turn prefix
|
||||
# bytes stay byte-equal to this turn's bytes. Skip the search
|
||||
# too: its result could only be discarded (#4600, #4622).
|
||||
if ws_tags is not None:
|
||||
ws_tags["memory_skip_reason"] = "cache_mode"
|
||||
from headroom.utils import extract_user_query
|
||||
|
||||
ws_tags["memory_skip_reason"] = "cache_mode"
|
||||
ws_input = ws_response_body.get("input", "")
|
||||
ws_instructions = ws_response_body.get("instructions")
|
||||
ws_msgs: list[dict[str, Any]] = []
|
||||
if ws_instructions:
|
||||
ws_msgs.append({"role": "system", "content": ws_instructions})
|
||||
if isinstance(ws_input, str) and ws_input:
|
||||
ws_msgs.append({"role": "user", "content": ws_input})
|
||||
log_memory_injection(
|
||||
request_id=request_id,
|
||||
session_id=session_id,
|
||||
decision="skipped_cache_mode",
|
||||
bytes_injected=0,
|
||||
query=ws_input if isinstance(ws_input, str) else "",
|
||||
query=extract_user_query(ws_msgs) or "",
|
||||
tags=ws_tags,
|
||||
)
|
||||
elif self.memory_handler.config.inject_context:
|
||||
|
||||
@@ -296,6 +296,51 @@ async def test_memory_lookup_skips_input_bearing_non_create_first_frame():
|
||||
assert f"current memory: {later_input}" in forwarded_later["response"]["input"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cache_mode_skips_memory_context_search_ws():
|
||||
upstream = _FakeUpstream(
|
||||
[
|
||||
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
|
||||
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
|
||||
json.dumps({"type": "response.created", "response": {"id": "r_2"}}),
|
||||
json.dumps({"type": "response.completed", "response": {"id": "r_2"}}),
|
||||
],
|
||||
hold_after_events=True,
|
||||
)
|
||||
first_turn, later_turn = _issue_2059_turns()
|
||||
client_frames = [
|
||||
_turn_with_tools(json.loads(first_turn)["response"]["input"], _client_response_tools()),
|
||||
_turn_with_tools(json.loads(later_turn)["response"]["input"], _client_response_tools()),
|
||||
]
|
||||
client_ws = _FakeWebSocket(frames=client_frames)
|
||||
handler = _DummyOpenAIHandler()
|
||||
handler.config.mode = "cache"
|
||||
memory = _MemoryHandler()
|
||||
handler.memory_handler = memory
|
||||
recorded_outcomes = []
|
||||
|
||||
async def _capture_outcome(outcome):
|
||||
recorded_outcomes.append(outcome)
|
||||
|
||||
handler._record_request_outcome = _capture_outcome
|
||||
|
||||
with patch.dict(sys.modules, {"websockets": _make_fake_websockets_module(upstream)}):
|
||||
await handler.handle_openai_responses_ws(client_ws)
|
||||
|
||||
assert memory.queries == []
|
||||
assert len(upstream.sent) == 2
|
||||
for i, sent_raw in enumerate(upstream.sent):
|
||||
sent = json.loads(sent_raw)
|
||||
original = json.loads(client_frames[i])
|
||||
assert sent["response"]["input"] == original["response"]["input"]
|
||||
expected_tools = [*_client_response_tools(), *_expected_memory_response_tools()]
|
||||
assert sent["response"]["tools"] == expected_tools
|
||||
|
||||
assert len(recorded_outcomes) >= 1
|
||||
for outcome in recorded_outcomes:
|
||||
assert outcome.tags.get("memory_skip_reason") == "cache_mode"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_memory_lookup_skips_bypassed_frames():
|
||||
upstream = _FakeUpstream(
|
||||
|
||||
@@ -583,3 +583,133 @@ def test_cache_mode_skips_memory_context_search_openai_responses() -> None:
|
||||
assert isinstance(sent, dict)
|
||||
assert sent.get("input") == "latest user string input"
|
||||
assert sent.get("instructions") == "You are a Codex assistant."
|
||||
|
||||
|
||||
def test_cache_mode_chat_preserves_tools_and_outcome_skip_reason() -> None:
|
||||
"""Cache mode never injects memory context on OpenAI /v1/chat/completions."""
|
||||
captured: dict[str, object] = {}
|
||||
recorded_outcomes = []
|
||||
with _make_proxy_client() as client:
|
||||
proxy = client.app.state.proxy
|
||||
proxy.config.mode = "cache"
|
||||
proxy.config.optimize = False
|
||||
proxy.config.image_optimize = False
|
||||
proxy.config.ccr_proactive_expansion = False
|
||||
|
||||
async def _capture_outcome(outcome):
|
||||
recorded_outcomes.append(outcome)
|
||||
|
||||
proxy._record_request_outcome = _capture_outcome
|
||||
|
||||
_install_session_tracker(proxy, frozen_count=1)
|
||||
_install_memory_handler(proxy)
|
||||
proxy.memory_handler.config.inject_tools = True
|
||||
proxy.memory_handler.compute_memory_tool_definitions = lambda _provider: [
|
||||
{"type": "function", "function": {"name": "mem_search", "parameters": {}}}
|
||||
]
|
||||
_install_capture_retry(proxy, captured)
|
||||
|
||||
response = client.post(
|
||||
"/v1/chat/completions",
|
||||
headers={
|
||||
"authorization": "Bearer sk-test",
|
||||
"x-headroom-user-id": "u1",
|
||||
},
|
||||
json={
|
||||
"model": "gpt-4o",
|
||||
"messages": [
|
||||
{"role": "system", "content": "base system"},
|
||||
{"role": "user", "content": "latest user"},
|
||||
],
|
||||
"tools": [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {"name": "client_tool", "parameters": {}},
|
||||
}
|
||||
],
|
||||
},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
proxy.memory_handler.search_and_format_context.assert_not_awaited()
|
||||
sent = captured["body"]
|
||||
assert isinstance(sent, dict)
|
||||
assert sent["messages"][1]["content"] == "latest user"
|
||||
# Memory tools remain present even when context search is skipped
|
||||
assert "tools" in sent
|
||||
tool_names = [t.get("function", {}).get("name") for t in sent["tools"]]
|
||||
assert "client_tool" in tool_names
|
||||
assert "mem_search" in tool_names
|
||||
# Outcome tags must record memory_skip_reason="cache_mode"
|
||||
assert len(recorded_outcomes) >= 1
|
||||
assert recorded_outcomes[0].tags.get("memory_skip_reason") == "cache_mode"
|
||||
|
||||
|
||||
def test_cache_mode_responses_preserves_tools_and_outcome_skip_reason() -> None:
|
||||
"""Cache mode never injects memory context on OpenAI /v1/responses."""
|
||||
captured: dict[str, object] = {}
|
||||
recorded_outcomes = []
|
||||
with _make_responses_proxy_client() as client:
|
||||
proxy = client.app.state.proxy
|
||||
proxy.config.mode = "cache"
|
||||
proxy.config.optimize = False
|
||||
proxy.config.image_optimize = False
|
||||
|
||||
async def _capture_outcome(outcome):
|
||||
recorded_outcomes.append(outcome)
|
||||
|
||||
proxy._record_request_outcome = _capture_outcome
|
||||
_install_memory_handler(proxy)
|
||||
proxy.memory_handler.config.inject_tools = True
|
||||
proxy.memory_handler.compute_memory_tool_definitions = lambda _provider: [
|
||||
{"type": "function", "function": {"name": "mem_search", "parameters": {}}}
|
||||
]
|
||||
|
||||
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
||||
captured["body"] = body
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"id": "resp_1",
|
||||
"object": "response",
|
||||
"model": "gpt-5.4",
|
||||
"output": [],
|
||||
"usage": {"input_tokens": 5, "output_tokens": 1, "total_tokens": 6},
|
||||
},
|
||||
)
|
||||
|
||||
proxy._retry_request = _fake_retry
|
||||
|
||||
response = client.post(
|
||||
"/v1/responses",
|
||||
headers={
|
||||
"authorization": "Bearer sk-test",
|
||||
"x-headroom-user-id": "u1",
|
||||
},
|
||||
json={
|
||||
"model": "gpt-5.4",
|
||||
"instructions": "You are a Codex assistant.",
|
||||
"input": "latest user prompt",
|
||||
"tools": [
|
||||
{
|
||||
"type": "function",
|
||||
"name": "client_tool",
|
||||
"description": "client tool",
|
||||
"parameters": {},
|
||||
}
|
||||
],
|
||||
},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
proxy.memory_handler.search_and_format_context.assert_not_awaited()
|
||||
assert "body" in captured
|
||||
sent = captured["body"]
|
||||
assert isinstance(sent, dict)
|
||||
assert sent.get("input") == "latest user prompt"
|
||||
assert "tools" in sent
|
||||
tool_names = [t.get("name") for t in sent["tools"]]
|
||||
assert "client_tool" in tool_names
|
||||
assert "mem_search" in tool_names
|
||||
assert len(recorded_outcomes) >= 1
|
||||
assert recorded_outcomes[0].tags.get("memory_skip_reason") == "cache_mode"
|
||||
|
||||
Reference in New Issue
Block a user