mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): skip memory-context search in cache mode on OpenAI routes (#4632)
## Description Follow-up to #4600 / #4606 as requested in #4622. PR #4606 implemented this optimization for Anthropic `/v1/messages`. This PR brings the three OpenAI routes (`handle_openai_chat`, `handle_openai_responses`, and `handle_openai_responses_ws`) into alignment. In `--mode cache`, memory context injection is disabled so prompt prefix bytes remain byte-stable across turns. Previously, all three OpenAI paths still executed the memory-context search (`search_and_format_context`: embedding query, vector index lookups, graph searches) and appended the result to the latest user turn in cache mode, introducing latency (~1–37ms+ per turn) and altering the forwarded user turn. With this PR: 1. `handle_openai_chat`: checks `is_cache_mode(self.config.mode)` before executing `search_and_format_context`, emits `decision="skipped_cache_mode"`, and keeps the user turn intact. 2. `handle_openai_responses`: checks `is_cache_mode(self.config.mode)` before searching, emits `decision="skipped_cache_mode"`, and keeps `body["input"]` intact. 3. `handle_openai_responses_ws`: checks `is_cache_mode(self.config.mode)` before measuring `memory_context`, emits `decision="skipped_cache_mode"`, and leaves `ws_response_body["input"]` untouched. 4. Memory tool injection remains completely unchanged across all routes. Closes #4622 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Performance improvement ## Changes Made - `headroom/proxy/handlers/openai.py`: Check `is_cache_mode(self.config.mode)` when `inject_context` is enabled across `handle_openai_chat`, `handle_openai_responses`, and `handle_openai_responses_ws`. Emit `decision="skipped_cache_mode"` with `bytes_injected=0` and bypass `search_and_format_context`. - `tests/test_proxy_system_prompt_immutable.py`: Added `test_cache_mode_skips_memory_context_search_openai_chat` and `test_cache_mode_skips_memory_context_search_openai_responses` to verify that `search_and_format_context` is never awaited in cache mode and user message / input fields remain byte-equal to client values. ## Testing - [x] Unit tests pass (`pytest tests/test_proxy_system_prompt_immutable.py`) - [x] Linting passes (`ruff check headroom/proxy/handlers/openai.py tests/test_proxy_system_prompt_immutable.py`) - [x] Formatting passes (`ruff format --check headroom/proxy/handlers/openai.py tests/test_proxy_system_prompt_immutable.py`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_proxy_system_prompt_immutable.py -q 10 passed, 1 warning in 31.92s $ ruff check headroom/proxy/handlers/openai.py tests/test_proxy_system_prompt_immutable.py All checks passed! $ ruff format --check headroom/proxy/handlers/openai.py tests/test_proxy_system_prompt_immutable.py 2 files already formatted ``` ## Real Behavior Proof - Environment: Linux / WSL2 Ubuntu, Python 3.13, local proxy test harness (`create_app` + `TestClient`). - Exact command / steps: Executed requests against `/v1/chat/completions` and `/v1/responses` with proxy configured in `--mode cache` and memory context enabled. Verified upstream payload captures and mock call history. - Observed result: In cache mode, the memory search is bypassed on all OpenAI requests; search_and_format_context was never awaited (0 calls), user message content and input fields remain byte-equal without injected memory context, and proxy emitted decision=skipped_cache_mode with bytes_injected=0. - Not tested: Live WebSocket connection to OpenAI upstream with streaming audio/realtime endpoints. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: no. Cache mode requires byte stability across turns; memory search is skipped in cache mode. Token mode is untouched. - Kill switch / disable path: n/a - Unsafe override required: no - Qualification impact: none - Rollback path: git revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation: n/a - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes - Aligns OpenAI Chat, Responses, and WebSocket routes with Anthropic behavior established in #4606. - Preserves memory tool injection without modification. Co-authored-by: shaiknaziya20 <naziyashaik5512@gmail.com> ### Maintainer combined-head validation Reviewed head `66a09054548423e33af47121ac7e87134d023fe4` includes a clean main sync through `12dca26aa` plus test-only maintainer follow-ups: the existing WebSocket dummy config now explicitly uses token mode, and a two-frame cache-mode regression covers skipped searches, unchanged input, retained memory tools, passthrough cancellation frames, and skip telemetry. No contributor runtime logic was changed. Windows / Python 3.12.13, with the existing local Rust extension: `.venv/Scripts/python.exe -m pytest tests/test_proxy_system_prompt_immutable.py tests/test_codex_ws_per_frame_memory.py tests/test_openai_codex_ws_lifecycle.py -q --tb=short --disable-warnings` => **89 passed**. The new WebSocket regression fails on current main because both frames still execute memory searches. Ruff check/format and main-relative whitespace checks pass. Additional runtime proof used a real Uvicorn proxy and a separate local HTTP upstream, both bound to ephemeral loopback ports, with production HTTP forwarding intact (no `_retry_request` patch). A deterministic recording memory backend returned `LOOPBACK_MEMORY` when searched, so the probe can distinguish an actual skip from an unconfigured/disabled memory path. The probe sent the same authenticated Chat and Responses requests in token mode, then cache mode, using user text `latest user prompt` and fixed Responses instructions. Exact invocation from the isolated worktree: `PYTHONPATH=G:/git/headroom-review4632 HEADROOM_OFFLINE=1 G:/git/headroom/.venv/Scripts/python.exe G:/git/headroom/.codex/review4632-loopback.py` (environment variables set with PowerShell `$env:` assignments). Results: **four HTTP 200 responses**; token mode performed one memory search per route and forwarded the returned memory text; cache mode performed zero searches per route, forwarded the original user text unchanged, and recorded `memory_skip_reason=cache_mode`. The proxy and upstream were shut down after the run. Limits: the loopback provider and recording memory backend are deterministic local fixtures; no external provider, embedding model, production memory service, load test, or real upstream WebSocket was exercised. WebSocket multi-frame/input/tool/telemetry behavior is covered by the focused regression above. Updated-head hosted qualification is inspected by maintainers; no contributor CI action is needed. Nothing merged. --------- Co-authored-by: shaiknaziya20 <naziyashaik5512@gmail.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
shaiknaziya20
JerrettDavis
parent
1bbc0ea89e
commit
365f6fe523
@@ -4372,7 +4372,24 @@ class OpenAIHandlerMixin:
|
||||
# /v1/responses) — pre-PR-this site was the only chat
|
||||
# path without one.
|
||||
try:
|
||||
if self.memory_handler.config.inject_context:
|
||||
if self.memory_handler.config.inject_context and is_cache_mode(self.config.mode):
|
||||
# Cache mode never injects context, so the next-turn prefix
|
||||
# bytes stay byte-equal to this turn's bytes. Skip the search
|
||||
# too: its result could only be discarded (#4600, #4622).
|
||||
from headroom.proxy.helpers import log_memory_injection
|
||||
from headroom.utils import extract_user_query
|
||||
|
||||
if tags is not None:
|
||||
tags["memory_skip_reason"] = "cache_mode"
|
||||
log_memory_injection(
|
||||
request_id=request_id,
|
||||
session_id=None,
|
||||
decision="skipped_cache_mode",
|
||||
bytes_injected=0,
|
||||
query=extract_user_query(optimized_messages) or "",
|
||||
tags=tags,
|
||||
)
|
||||
elif self.memory_handler.config.inject_context:
|
||||
memory_context = await asyncio.wait_for(
|
||||
self.memory_handler.search_and_format_context(
|
||||
memory_user_id,
|
||||
@@ -6211,7 +6228,24 @@ class OpenAIHandlerMixin:
|
||||
# (latest non-frozen user item). Instructions are part of the
|
||||
# cache hot zone and must never be mutated — invariant I2.
|
||||
# See REALIGNMENT/03-phase-A-lockdown.md PR-A2.
|
||||
if self.memory_handler.config.inject_context:
|
||||
if self.memory_handler.config.inject_context and is_cache_mode(self.config.mode):
|
||||
# Cache mode never injects context, so the next-turn prefix
|
||||
# bytes stay byte-equal to this turn's bytes. Skip the search
|
||||
# too: its result could only be discarded (#4600, #4622).
|
||||
from headroom.proxy.helpers import log_memory_injection
|
||||
from headroom.utils import extract_user_query
|
||||
|
||||
if tags is not None:
|
||||
tags["memory_skip_reason"] = "cache_mode"
|
||||
log_memory_injection(
|
||||
request_id=request_id,
|
||||
session_id=None,
|
||||
decision="skipped_cache_mode",
|
||||
bytes_injected=0,
|
||||
query=extract_user_query(optimized_messages) or "",
|
||||
tags=tags,
|
||||
)
|
||||
elif self.memory_handler.config.inject_context:
|
||||
try:
|
||||
memory_context = await asyncio.wait_for(
|
||||
self.memory_handler.search_and_format_context(
|
||||
@@ -8276,7 +8310,24 @@ class OpenAIHandlerMixin:
|
||||
)
|
||||
|
||||
# Inject memory context into instructions
|
||||
if self.memory_handler.config.inject_context:
|
||||
if self.memory_handler.config.inject_context and is_cache_mode(
|
||||
self.config.mode
|
||||
):
|
||||
# Cache mode never injects context, so the next-turn prefix
|
||||
# bytes stay byte-equal to this turn's bytes. Skip the search
|
||||
# too: its result could only be discarded (#4600, #4622).
|
||||
if ws_tags is not None:
|
||||
ws_tags["memory_skip_reason"] = "cache_mode"
|
||||
ws_input = ws_response_body.get("input", "")
|
||||
log_memory_injection(
|
||||
request_id=request_id,
|
||||
session_id=session_id,
|
||||
decision="skipped_cache_mode",
|
||||
bytes_injected=0,
|
||||
query=ws_input if isinstance(ws_input, str) else "",
|
||||
tags=ws_tags,
|
||||
)
|
||||
elif self.memory_handler.config.inject_context:
|
||||
ws_input = ws_response_body.get("input", "")
|
||||
ws_instructions = ws_response_body.get("instructions")
|
||||
ws_msgs: list[dict[str, Any]] = []
|
||||
|
||||
@@ -196,6 +196,50 @@ async def test_memory_lookup_runs_for_each_issue_artifact_frame_and_preserves_no
|
||||
assert forwarded_turns[0]["response"]["tools"] == forwarded_turns[1]["response"]["tools"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cache_mode_skips_context_on_each_frame_and_preserves_tools():
|
||||
upstream = _FakeUpstream(
|
||||
[
|
||||
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
|
||||
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
|
||||
],
|
||||
hold_after_events=True,
|
||||
)
|
||||
first, later = _issue_2059_inputs()
|
||||
cancel = json.dumps({"type": "response.cancel"})
|
||||
frames = [
|
||||
_turn_with_tools(first, _client_response_tools()),
|
||||
cancel,
|
||||
_turn_with_tools(later, _client_response_tools()),
|
||||
]
|
||||
client_ws = _FakeWebSocket(frames=frames)
|
||||
handler = _DummyOpenAIHandler()
|
||||
handler.config.mode = "cache"
|
||||
memory = _MemoryHandler()
|
||||
handler.memory_handler = memory
|
||||
|
||||
with (
|
||||
patch.dict(sys.modules, {"websockets": _make_fake_websockets_module(upstream)}),
|
||||
patch("headroom.proxy.helpers.log_memory_injection") as log_injection,
|
||||
):
|
||||
await handler.handle_openai_responses_ws(client_ws)
|
||||
|
||||
assert memory.queries == []
|
||||
assert upstream.sent[1] == cancel
|
||||
forwarded = [json.loads(upstream.sent[i])["response"] for i in (0, 2)]
|
||||
assert [response["input"] for response in forwarded] == [first, later]
|
||||
expected_tools = [*_client_response_tools(), *_expected_memory_response_tools()]
|
||||
assert all(response["tools"] == expected_tools for response in forwarded)
|
||||
skips = [
|
||||
call.kwargs
|
||||
for call in log_injection.call_args_list
|
||||
if call.kwargs.get("decision") == "skipped_cache_mode"
|
||||
]
|
||||
assert len(skips) == 2
|
||||
assert all(call["bytes_injected"] == 0 for call in skips)
|
||||
assert all(call["tags"]["memory_skip_reason"] == "cache_mode" for call in skips)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("tools", [None, []], ids=["omitted", "empty"])
|
||||
async def test_ws_responses_do_not_inject_memory_tools_without_client_tools(tools):
|
||||
|
||||
@@ -129,6 +129,7 @@ class _DummyOpenAIHandler(OpenAIHandlerMixin):
|
||||
self.metrics = _DummyMetrics()
|
||||
self.config = SimpleNamespace(
|
||||
optimize=False,
|
||||
mode="token",
|
||||
retry_max_attempts=1,
|
||||
retry_base_delay_ms=1,
|
||||
retry_max_delay_ms=1,
|
||||
|
||||
@@ -21,6 +21,11 @@ from fastapi.testclient import TestClient
|
||||
from headroom.proxy.server import ProxyConfig, create_app
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _stub_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv("HEADROOM_REQUIRE_RUST_CORE", "false")
|
||||
|
||||
|
||||
class _FakePrefixTracker:
|
||||
def __init__(self, frozen_count: int):
|
||||
self._frozen_count = frozen_count
|
||||
@@ -474,3 +479,107 @@ def test_memory_context_appears_in_responses_api_input_tail() -> None:
|
||||
assert any("MEMCTX" in t for t in text_parts)
|
||||
elif isinstance(input_field, str):
|
||||
assert input_field.endswith("MEMCTX")
|
||||
|
||||
|
||||
def test_cache_mode_skips_memory_context_search_openai_chat() -> None:
|
||||
"""Cache mode never injects memory context on OpenAI chat, so it must not search for it."""
|
||||
captured: dict[str, object] = {}
|
||||
with _make_proxy_client() as client:
|
||||
proxy = client.app.state.proxy
|
||||
proxy.config.mode = "cache"
|
||||
proxy.config.optimize = False
|
||||
proxy.config.image_optimize = False
|
||||
proxy.config.ccr_proactive_expansion = False
|
||||
|
||||
_install_session_tracker(proxy, frozen_count=1)
|
||||
_install_memory_handler(proxy)
|
||||
|
||||
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
||||
captured["body"] = body
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"id": "chatcmpl_1",
|
||||
"object": "chat.completion",
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"message": {"role": "assistant", "content": "ack"},
|
||||
"finish_reason": "stop",
|
||||
}
|
||||
],
|
||||
"usage": {"prompt_tokens": 10, "completion_tokens": 2, "total_tokens": 12},
|
||||
},
|
||||
)
|
||||
|
||||
proxy._retry_request = _fake_retry
|
||||
|
||||
response = client.post(
|
||||
"/v1/chat/completions",
|
||||
headers={
|
||||
"authorization": "Bearer sk-test",
|
||||
"x-headroom-user-id": "u1",
|
||||
},
|
||||
json={
|
||||
"model": "gpt-4o",
|
||||
"messages": [
|
||||
{"role": "system", "content": "base system"},
|
||||
{"role": "user", "content": "frozen prefix"},
|
||||
{"role": "assistant", "content": "ack"},
|
||||
{"role": "user", "content": "latest user"},
|
||||
],
|
||||
},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
proxy.memory_handler.search_and_format_context.assert_not_awaited()
|
||||
sent = captured["body"]
|
||||
assert isinstance(sent, dict)
|
||||
assert sent["messages"][3]["content"] == "latest user"
|
||||
|
||||
|
||||
def test_cache_mode_skips_memory_context_search_openai_responses() -> None:
|
||||
"""Cache mode never injects memory context on OpenAI responses, so it must not search for it."""
|
||||
captured: dict[str, object] = {}
|
||||
with _make_responses_proxy_client() as client:
|
||||
proxy = client.app.state.proxy
|
||||
proxy.config.mode = "cache"
|
||||
proxy.config.optimize = False
|
||||
proxy.config.image_optimize = False
|
||||
_install_memory_handler(proxy)
|
||||
|
||||
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
||||
captured["body"] = body
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"id": "resp_1",
|
||||
"object": "response",
|
||||
"model": "gpt-5.4",
|
||||
"output": [],
|
||||
"usage": {"input_tokens": 5, "output_tokens": 1, "total_tokens": 6},
|
||||
},
|
||||
)
|
||||
|
||||
proxy._retry_request = _fake_retry
|
||||
|
||||
response = client.post(
|
||||
"/v1/responses",
|
||||
headers={
|
||||
"authorization": "Bearer sk-test",
|
||||
"x-headroom-user-id": "u1",
|
||||
},
|
||||
json={
|
||||
"model": "gpt-5.4",
|
||||
"instructions": "You are a Codex assistant.",
|
||||
"input": "latest user string input",
|
||||
},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
proxy.memory_handler.search_and_format_context.assert_not_awaited()
|
||||
assert "body" in captured
|
||||
sent = captured["body"]
|
||||
assert isinstance(sent, dict)
|
||||
assert sent.get("input") == "latest user string input"
|
||||
assert sent.get("instructions") == "You are a Codex assistant."
|
||||
|
||||
Reference in New Issue
Block a user