mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): drop web_search_call history from a Responses request with no web_search tool (#4202)
## Description
Codex on a custom model provider (how `headroom wrap codex` configures
it) compacts locally: it resends the whole history to `/v1/responses`
with `tools: []`. When that history holds a hosted `web_search_call`
item, the ChatGPT backend answers 200 and then sends the in-band error
"response protection is unavailable". Codex ignores in-band `error`
events and reports "stream closed before response.completed". It retries
five times, then runs the same compaction before every later prompt, so
the session never answers again. Headroom forwards that body unchanged
today, on HTTP and on the Codex WebSocket.
The backend behaviour comes from a user report. A compaction body
captured from the failing session held 10 `web_search_call` items. Sent
straight to `chatgpt.com/backend-api/codex/responses`, with no Headroom
in the path, it failed every time. The same body completed with the 10
items removed, and also completed with `tools: [{"type":
"web_search"}]`. A normal turn on the same history, which declares the
tool, is unaffected.
The same failure appears without Headroom:
- openai/codex#51962: Codex's built-in provider, remote compaction of a
resumed history. Removing only the `web_search_call` items fixes it
there too.
- openai/codex#52016: the same two error strings with direct ChatGPT
login.
This PR drops `web_search_call` items from a Responses `input` when the
request declares no web_search tool, before the body leaves the proxy,
and marks the body mutated so the edited bytes are sent. Only the search
action (query or URL) leaves the history. The search results were never
in it. The answer the model gave afterwards is a separate `message`
item, and it stays. A bypass request (`x-headroom-bypass: true`) is left
untouched. Every other body is forwarded exactly as before.
This works around a backend behaviour; it does not fix a Headroom bug.
It should be removed once the backend accepts such a body.
Closes #4201
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **The drop.** `_drop_undeclared_web_search_calls(payload)`
(`headroom/proxy/handlers/openai.py`, next to
`_ensure_chatgpt_responses_store_false`) drops every `web_search_call`
item from `input`. It logs `event=web_search_history request_id=...
dropped=N reason=no_web_search_tool`. It does nothing in these cases:
- A tool whose `type` starts with `web_search` is declared
(`web_search`, `web_search_preview`, the dated variants), in top-level
`tools` or in a Codex `additional_tools` input item. Codex 0.149 and
later sends tools that way, with no top-level `tools` (#3185). Without
this check a normal Codex turn would lose its search history.
- The request continues a `previous_response_id`, because its
declarations may live server-side.
- `input` is not a list, or holds no `web_search_call`.
- `HEADROOM_WEB_SEARCH_HISTORY=0`.
- **HTTP `/v1/responses`** (`handle_openai_responses`): the drop runs
before the handler first reads `input`, so compression, memory and
telemetry see the same body that is sent. It is skipped when `_bypass`
is set (`x-headroom-bypass: true` or `x-headroom-mode: passthrough`). A
drop marks the `BodyMutationTracker` with
`responses_web_search_history`, and both the stream and the buffered
forwarder pick canonical bytes from that flag. With no drop, the
client's bytes go out verbatim, as before.
- **Codex WebSocket `/v1/responses`:** the drop is called from
`_normalize_ws_response_create_for_upstream`. Every `response.create`
frame passes through there on its way upstream: the first frame, the
WS-to-HTTP fallback that replays it, later frames, and memory
continuations.
- On a bypass session (`_ws_bypass`, the same header check on the
WebSocket handshake headers) the drop is skipped and every frame goes
out as the client sent it.
- `_ws_bypass` is now resolved just before that helper is defined,
instead of a few lines later, so the helper can read it. Its value and
log line are unchanged.
- `_drop_undeclared_web_search_calls_from_frame` returns the same string
object unless it dropped something. A frame is parsed only if it holds
the literal `"web_search_call"` or any `\u` escape, since JSON can spell
the type as `"web_search\u005fcall"` and a letter or underscore can only
be escaped that way; other frames skip the parse.
- **With `optimize` off:** the drop still runs, like the forced
`store=false`, because it is a repair the request needs in order to
succeed, not an optimization.
- **Kill switch:** `HEADROOM_WEB_SEARCH_HISTORY=0`, read per request
through `runtime_env.getenv`.
- It is registered in `RUNTIME_ENV_KNOBS`, so `headroom wrap` pushes it
to a reused proxy through `POST /admin/runtime-env`, and `/health`
reports it under `config.runtime_env`.
- The allowlist table in `docs/content/docs/security-model.mdx` gets the
matching row.
- **Tests in `tests/test_openai_responses_web_search_history.py` (22
tests):**
- The decision. Drops when `tools` is empty, absent, function-only, or
holds a function named `web_search`. Keeps with each web_search type
spelling, with a web_search tool in `additional_tools`, with
`previous_response_id`, with string `input`, with no search history, and
under the kill switch.
- The frame helper: wrapped and flattened frames, and the same object
back for anything it leaves alone.
- Through the app:
- HTTP stream and buffered: upstream receives the body minus those
items, and the outbound log names the reason.
- HTTP byte-identical passthrough for a declared tool and for a history
with no searches.
- HTTP bypass, stream and buffered: byte-identical, no
`web_search_history` line.
- WS first frame, and a WS later frame after a normal turn that is
forwarded unchanged.
- WS bypass session: first and later frame unchanged.
- The WS-to-HTTP fallback.
- **Test in `tests/test_runtime_env.py`:**
`test_override_disables_web_search_history_drop_without_env` checks that
a runtime-env override of `0` turns the drop off with no environment
variable set, and that `explicit_env` puts the knob in the wrap payload.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
What was run:
- Unit tests: the new file, `tests/test_runtime_env.py`, and every
existing test file for the Responses handler, the Codex WebSocket, Codex
routing and byte-faithful forwarding. Not the whole suite.
- Lint and types: `ruff check`, `ruff format --check` (ruff 0.16.8, the
pre-commit pin) and `mypy` on the changed files. Not the whole tree.
- Every run used a temporary `HOME`, with `HEADROOM_SAVINGS_EVENTS_PATH`
in that directory and `HEADROOM_BEACON=off`.
### Test Output
With the two drop call sites removed from `openai.py`, which is upstream
`main` behaviour (the helpers stay so the test module imports):
```text
$ python -m pytest tests/test_openai_responses_web_search_history.py -q -p no:cacheprovider
FAILED tests/test_openai_responses_web_search_history.py::test_http_compaction_request_reaches_upstream_without_web_search_calls[stream]
FAILED tests/test_openai_responses_web_search_history.py::test_http_compaction_request_reaches_upstream_without_web_search_calls[buffered]
FAILED tests/test_openai_responses_web_search_history.py::test_ws_first_frame_reaches_upstream_without_web_search_calls
FAILED tests/test_openai_responses_web_search_history.py::test_ws_later_frame_reaches_upstream_without_web_search_calls
FAILED tests/test_openai_responses_web_search_history.py::test_ws_http_fallback_gets_the_frame_without_web_search_calls
=================== 5 failed, 17 passed, 1 warning in 2.17s ====================
```
With the bypass gates and the knob registration removed, which is this
PR's first revision, 45d134db:
```text
$ python -m pytest tests/test_openai_responses_web_search_history.py tests/test_runtime_env.py -q -p no:cacheprovider
E assert b'{"model":"g...store":false}' == b'{\n "model...re": false\n}'
E assert b'{"model":"g...store":false}' == b'{\n "model...re": false\n}'
E assert ['{"type": "r...re": false}}'] == ['{"type": "r...re": false}}']
E AssertionError: assert 1 == 0
FAILED tests/test_openai_responses_web_search_history.py::test_http_bypass_request_is_forwarded_byte_identical[stream]
FAILED tests/test_openai_responses_web_search_history.py::test_http_bypass_request_is_forwarded_byte_identical[buffered]
FAILED tests/test_openai_responses_web_search_history.py::test_ws_bypass_session_frames_reach_upstream_unchanged
FAILED tests/test_runtime_env.py::test_override_disables_web_search_history_drop_without_env
=================== 4 failed, 42 passed, 1 warning in 3.54s ====================
```
With the fix (418faa61):
```text
$ python -m pytest tests/test_openai_responses_web_search_history.py tests/test_runtime_env.py -q -p no:cacheprovider
======================== 46 passed, 1 warning in 8.57s =========================
$ python -m pytest tests/test_openai_responses_web_search_history.py tests/test_runtime_env.py tests/test_proxy_byte_faithful_forwarding.py tests/test_ccr_response_handler_openai_responses.py tests/test_codex_responses_passthrough_bytes.py tests/test_codex_responses_waste_signals.py tests/test_codex_ws_compression_scheduler.py tests/test_codex_ws_per_frame_memory.py tests/test_codex_ws_savings_deferral.py tests/test_openai_codex_routing.py tests/test_openai_codex_ws_lifecycle.py tests/test_openai_codex_ws_timings.py tests/test_openai_responses_additional_tools.py tests/test_openai_responses_buffered_sse.py tests/test_openai_responses_compression_units.py tests/test_openai_responses_context_compaction.py tests/test_openai_responses_null_arguments.py tests/test_openai_responses_output_shaper.py tests/test_openai_responses_read_protection.py tests/test_openai_responses_t3_replay_regression.py tests/test_openai_responses_traffic_learner.py tests/test_output_shaper_responses.py tests/test_provider_codex_responses.py tests/test_provider_openai_responses.py tests/test_proxy_openai_cache_key_integration.py tests/test_proxy_openai_cache_stability.py tests/test_proxy_openai_rate_limit_key.py tests/test_proxy_openai_responses_bypass.py tests/test_proxy_openai_responses_integration.py tests/test_proxy_openai_responses_stream_ccr.py tests/test_proxy_openai.py tests/test_responses_cross_turn_dedup.py tests/test_responses_pyo3_compression.py tests/test_responses_savings_denominator.py tests/test_responses_ws_pyo3_compression.py -q -p no:cacheprovider
================= 552 passed, 14 skipped, 2 warnings in 24.94s =================
$ ruff check headroom/proxy/handlers/openai.py headroom/proxy/runtime_env.py tests/test_openai_responses_web_search_history.py tests/test_runtime_env.py && ruff format --check headroom/proxy/handlers/openai.py headroom/proxy/runtime_env.py tests/test_openai_responses_web_search_history.py tests/test_runtime_env.py # ruff 0.16.8
All checks passed!
4 files already formatted
$ mypy headroom/proxy/handlers/openai.py headroom/proxy/runtime_env.py
Success: no issues found in 2 source files
```
## Real Behavior Proof
- Environment: macOS 27.0.1 arm64, Python 3.11.13, a temporary `HOME`.
Upstream `main` f1bc2609 (exported with `git archive`) compared with
this branch at 418faa61, in the same venv.
- Exact command / steps: `repro_web_search_history.py` (below), run as
`PYTHONPATH=<tree> LITELLM_LOCAL_MODEL_COST_MAP=True python
repro_web_search_history.py`.
- It boots the real app with `create_app(ProxyConfig(optimize=False,
openai_api_url=<mock>))` inside a lifespan `TestClient`.
- The OpenAI upstream points at a local mock that records the bytes it
receives: an HTTP server for the POST and a `websockets` server for the
WebSocket.
- It sends a Codex-shaped compaction body (`tools: []`, two
`web_search_call` items, one `search` and one `open_page`) over HTTP
`POST /v1/responses` (stream) and over the Codex WebSocket.
- Each transport also gets two controls: the same history with `tools:
[{"type": "web_search"}]`, and the compaction body sent with
`x-headroom-bypass: true`.
- Observed result: main forwards all six requests with their
`web_search_call` items; this branch drops them from the `tools: []`
compaction body on HTTP and WebSocket and leaves both controls
byte-identical. Details:
- On `main`, all six requests reach upstream byte-identical,
`web_search_call` items included.
- On the branch, the compaction body loses its two items on both
transports and the drop is logged.
- On the branch, both controls go out byte-identical on both transports,
including the bypass request.
```text
=== main @ f1bc2609
== HTTP POST /v1/responses ==
-- compaction, tools: []
client sent : ['message', 'web_search_call', 'web_search_call', 'message', 'message']
upstream got: ['message', 'web_search_call', 'web_search_call', 'message', 'message']
byte-identical: True
log: event=outbound_request forwarder=streaming method=POST body_bytes=739 body_mutated=false mutation_reasons= source=passthrough
-- control, web_search declared
client sent : ['message', 'web_search_call', 'web_search_call', 'message', 'message']
upstream got: ['message', 'web_search_call', 'web_search_call', 'message', 'message']
byte-identical: True
log: event=outbound_request forwarder=streaming method=POST body_bytes=761 body_mutated=false mutation_reasons= source=passthrough
-- control, tools: [] with x-headroom-bypass: true
client sent : ['message', 'web_search_call', 'web_search_call', 'message', 'message']
upstream got: ['message', 'web_search_call', 'web_search_call', 'message', 'message']
byte-identical: True
log: event=outbound_request forwarder=streaming method=POST body_bytes=739 body_mutated=false mutation_reasons= source=passthrough
== WebSocket /v1/responses (response.create) ==
-- compaction, tools: []
client sent : ['message', 'web_search_call', 'web_search_call', 'message', 'message']
upstream got: ['message', 'web_search_call', 'web_search_call', 'message', 'message']
byte-identical: True
log: (no matching log line)
-- control, web_search declared
client sent : ['message', 'web_search_call', 'web_search_call', 'message', 'message']
upstream got: ['message', 'web_search_call', 'web_search_call', 'message', 'message']
byte-identical: True
log: (no matching log line)
-- control, tools: [] with x-headroom-bypass: true
client sent : ['message', 'web_search_call', 'web_search_call', 'message', 'message']
upstream got: ['message', 'web_search_call', 'web_search_call', 'message', 'message']
byte-identical: True
log: (no matching log line)
=== branch @ 418faa61
== HTTP POST /v1/responses ==
-- compaction, tools: []
client sent : ['message', 'web_search_call', 'web_search_call', 'message', 'message']
upstream got: ['message', 'message', 'message']
byte-identical: False
log: event=web_search_history dropped=2 reason=no_web_search_tool
log: event=outbound_request forwarder=streaming method=POST body_bytes=443 body_mutated=true mutation_reasons=responses_web_search_history source=canonical
-- control, web_search declared
client sent : ['message', 'web_search_call', 'web_search_call', 'message', 'message']
upstream got: ['message', 'web_search_call', 'web_search_call', 'message', 'message']
byte-identical: True
log: event=outbound_request forwarder=streaming method=POST body_bytes=761 body_mutated=false mutation_reasons= source=passthrough
-- control, tools: [] with x-headroom-bypass: true
client sent : ['message', 'web_search_call', 'web_search_call', 'message', 'message']
upstream got: ['message', 'web_search_call', 'web_search_call', 'message', 'message']
byte-identical: True
log: event=outbound_request forwarder=streaming method=POST body_bytes=739 body_mutated=false mutation_reasons= source=passthrough
== WebSocket /v1/responses (response.create) ==
-- compaction, tools: []
client sent : ['message', 'web_search_call', 'web_search_call', 'message', 'message']
upstream got: ['message', 'message', 'message']
byte-identical: False
log: event=web_search_history dropped=2 reason=no_web_search_tool
-- control, web_search declared
client sent : ['message', 'web_search_call', 'web_search_call', 'message', 'message']
upstream got: ['message', 'web_search_call', 'web_search_call', 'message', 'message']
byte-identical: True
log: (no matching log line)
-- control, tools: [] with x-headroom-bypass: true
client sent : ['message', 'web_search_call', 'web_search_call', 'message', 'message']
upstream got: ['message', 'web_search_call', 'web_search_call', 'message', 'message']
byte-identical: True
log: (no matching log line)
```
The backend side was measured before this PR, from a user report. The
captured compaction body was sent straight to `chatgpt.com` with no
proxy in the path. As sent, it failed with "response protection is
unavailable". With its 10 `web_search_call` items removed, or with a
web_search tool declared, it completed. Headroom Desktop has carried the
same drop as a downstream patch since 2026-10-08, on the HTTP forwarders
only. Through that patch, the forked session that had failed six
compactions in a row compacted on the first try
(`event=web_search_history dropped=10`).
- Not tested: a live ChatGPT backend run of this branch, among others:
- This branch was not run against the live ChatGPT backend, and no live
Codex session went through it. The request shape and the backend A/B
above come from the downstream patch, which makes the same edit to the
same body.
- The WebSocket path has unit and mock coverage only. The downstream
verification ran over HTTP, because Headroom Desktop configures Codex
with `supports_websockets = false`. `headroom wrap codex` sets
`supports_websockets = true`, so most upstream users take the WebSocket
path.
- No captured Codex 0.149+ compaction body with `additional_tools` items
was available.
- The `headroom wrap` push of the knob to a reused proxy was not run end
to end. The test covers `explicit_env` and the override reaching the
reader.
- Linux and Windows were not run.
<details>
<summary>repro_web_search_history.py</summary>
```python
"""Repro: web_search_call history on a Responses request with no web_search tool.
Boots the real Headroom proxy app (create_app + ProxyConfig, optimize off so
compression cannot be the variable) in-process, points its OpenAI upstream at
a local mock, and sends a Codex local-compaction body (full history,
``tools: []``, two hosted ``web_search_call`` items) over:
1. HTTP POST /v1/responses (stream)
2. the Codex WebSocket /v1/responses (first response.create frame)
plus two controls on each transport that must reach upstream unchanged: the
same history with a web_search tool declared (a normal Codex turn), and the
compaction body sent with ``x-headroom-bypass: true``.
The mock records what upstream received. No network beyond 127.0.0.1, no
real provider call.
Run against a source tree:
PYTHONPATH=<tree> LITELLM_LOCAL_MODEL_COST_MAP=True <venv>/bin/python repro_web_search_history.py
"""
from __future__ import annotations
import asyncio
import json
import logging
import os
import re
import sys
import tempfile
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
_TMP = tempfile.mkdtemp(prefix="wsh-repro-")
os.environ["HOME"] = _TMP
os.environ["HEADROOM_SAVINGS_EVENTS_PATH"] = os.path.join(_TMP, "savings_events.jsonl")
os.environ.setdefault("HEADROOM_BEACON", "off")
os.environ.setdefault("HEADROOM_TELEMETRY", "off")
import headroom # noqa: E402
import websockets # noqa: E402
from fastapi.testclient import TestClient # noqa: E402
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
HEADERS = {
"authorization": "Bearer sk-test-not-a-real-key",
"user-agent": "codex_cli_rs/0.161.0",
"originator": "codex_cli_rs",
"content-type": "application/json",
}
BYPASS_HEADERS = {**HEADERS, "x-headroom-bypass": "true"}
CASES = (
(False, HEADERS, "compaction, tools: []"),
(True, HEADERS, "control, web_search declared"),
(False, BYPASS_HEADERS, "control, tools: [] with x-headroom-bypass: true"),
)
def compaction_body(*, declare_web_search: bool) -> dict:
return {
"model": "gpt-5.5",
"instructions": "Codex base instructions.",
"input": [
{
"type": "message",
"role": "user",
"content": [{"type": "input_text", "text": "what changed in the release?"}],
},
{
"type": "web_search_call",
"id": "ws_1",
"status": "completed",
"action": {"type": "search", "query": "release notes"},
},
{
"type": "web_search_call",
"id": "ws_2",
"status": "completed",
"action": {"type": "open_page", "url": "https://example.com/notes"},
},
{
"type": "message",
"role": "assistant",
"content": [{"type": "output_text", "text": "Three fixes shipped."}],
},
{
"type": "message",
"role": "user",
"content": [{"type": "input_text", "text": "Summarize the conversation."}],
},
],
"tools": [{"type": "web_search"}] if declare_web_search else [],
"tool_choice": "auto",
"stream": True,
"store": False,
}
def types(payload: dict) -> list[str]:
return [item["type"] for item in payload["input"]]
SSE = (
b'event: response.created\ndata: {"type":"response.created","response":{"id":"r_1"}}\n\n'
b'event: response.completed\ndata: {"type":"response.completed","response":{"id":"r_1",'
b'"usage":{"input_tokens":10,"output_tokens":2,"total_tokens":12}}}\n\n'
)
class _HttpUpstream(BaseHTTPRequestHandler):
received: list[bytes] = []
def do_POST(self) -> None: # noqa: N802
self.received.append(self.rfile.read(int(self.headers["content-length"])))
self.send_response(200)
self.send_header("content-type", "text/event-stream")
self.send_header("content-length", str(len(SSE)))
self.end_headers()
self.wfile.write(SSE)
def log_message(self, *args) -> None:
pass
class _WsUpstream:
def __init__(self) -> None:
self.received: list[str] = []
self.port = 0
ready = threading.Event()
async def handler(conn) -> None:
async for frame in conn:
self.received.append(frame)
await conn.send(json.dumps({"type": "response.created", "response": {"id": "r_1"}}))
await conn.send(
json.dumps({"type": "response.completed", "response": {"id": "r_1"}})
)
async def main() -> None:
async with websockets.serve(handler, "127.0.0.1", 0) as server:
self.port = server.sockets[0].getsockname()[1]
ready.set()
await asyncio.Future()
threading.Thread(target=lambda: asyncio.run(main()), daemon=True).start()
ready.wait(5)
def make_client(upstream_url: str) -> TestClient:
config = ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
ccr_inject_tool=False,
ccr_handle_responses=False,
ccr_context_tracking=False,
image_optimize=False,
openai_api_url=upstream_url,
)
return TestClient(create_app(config))
class _Lines(logging.Handler):
def __init__(self) -> None:
super().__init__(logging.INFO)
self.lines: list[str] = []
def emit(self, record: logging.LogRecord) -> None:
msg = record.getMessage()
if "web_search_history" in msg or (
"event=outbound_request" in msg and "/v1/responses" in msg
):
self.lines.append(msg)
def report(label: str, sent_bytes: bytes | str, got: bytes | str, logs: _Lines) -> None:
sent = json.loads(sent_bytes)
wire = json.loads(got)
sent_payload = sent.get("response", sent)
wire_payload = wire.get("response", wire)
print(f"-- {label}")
print(f" client sent : {types(sent_payload)}")
print(f" upstream got: {types(wire_payload)}")
print(f" byte-identical: {sent_bytes == got}")
for line in logs.lines or ["(no matching log line)"]:
print(" log: " + re.sub(r" (path|request_id)=\S+", "", line))
logs.lines.clear()
def main() -> int:
proxy_logger = logging.getLogger("headroom.proxy")
proxy_logger.setLevel(logging.INFO)
logs = _Lines()
proxy_logger.addHandler(logs)
print(f"headroom source: {os.path.dirname(headroom.__file__)}")
http_server = ThreadingHTTPServer(("127.0.0.1", 0), _HttpUpstream)
threading.Thread(target=http_server.serve_forever, daemon=True).start()
print("== HTTP POST /v1/responses ==")
with make_client(f"http://127.0.0.1:{http_server.server_port}") as client:
for declare, headers, label in CASES:
body = json.dumps(compaction_body(declare_web_search=declare)).encode()
with client.stream("POST", "/v1/responses", headers=headers, content=body) as response:
assert response.status_code == 200, response.status_code
response.read()
report(label, body, _HttpUpstream.received[-1], logs)
ws_upstream = _WsUpstream()
print("== WebSocket /v1/responses (response.create) ==")
with make_client(f"http://127.0.0.1:{ws_upstream.port}") as client:
for declare, headers, label in CASES:
frame = json.dumps(
{"type": "response.create", "response": compaction_body(declare_web_search=declare)}
)
before = len(ws_upstream.received)
with client.websocket_connect("/v1/responses", headers=headers) as ws:
ws.send_text(frame)
while json.loads(ws.receive_text()).get("type") != "response.completed":
pass
assert len(ws_upstream.received) == before + 1
report(label, frame, ws_upstream.received[-1], logs)
return 0
if __name__ == "__main__":
sys.exit(main())
```
</details>
## Runtime Rollout Safety
- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes, for `/v1/responses` requests
(HTTP and Codex WebSocket) that meet all of these:
- `input` holds `web_search_call` items;
- neither `tools` nor an `additional_tools` item declares a web_search
tool;
- no `previous_response_id` is set;
- the request does not carry `x-headroom-bypass: true` or
`x-headroom-mode: passthrough`.
Those requests now go upstream without the `web_search_call` items,
re-serialized: canonical bytes on HTTP, `json.dumps` of the frame on the
WebSocket. Any other edit already made to the body rides along, as it
does whenever the body is mutated. One example is the forced
`store=false` on ChatGPT auth. Every other request is forwarded as
before, byte-identical when nothing else changed it. Bypass requests
stay byte-identical. Chat Completions, Anthropic, Gemini and the
`/v1/responses/{subpath}` passthrough are not touched.
- Kill switch / disable path: `HEADROOM_WEB_SEARCH_HISTORY=0` (also
`false`, `no`, `off`). It is read per request and registered as a
runtime-env knob, so there are three ways to set it:
- in the proxy's environment, which takes effect when the proxy next
starts;
- exported in the shell of a `headroom wrap` that reuses a running proxy
(wrap pushes it, no restart);
- posted to `POST /admin/runtime-env` (no restart).
A single request can also opt out with `x-headroom-bypass: true`.
- Unsafe override required: No.
- Qualification impact: only requests the backend was already failing
are changed, so no completed request's compression or cache result
moves. Such a request now completes, without reusing the previous turn's
cached prefix past the first dropped item.
- Over HTTP the drop runs before compression, so token counts and
savings are computed on the edited body, and the removed items are not
counted as saved.
- On the WebSocket it runs after compression, so the items appear in
both the before and after counts, and tokens saved is unchanged.
- Rollback path: set the kill switch, or revert these commits. Either
restores byte-identical forwarding of these bodies, and the failing
compactions with it.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A
## Additional Notes
- Review follow-ups:
- 2099603a: bypass requests are no longer rewritten, on HTTP or on the
WebSocket.
- 418faa61: registers the kill switch as a runtime-env knob, so a reused
proxy honours it.
- Documentation: the `POST /admin/runtime-env` allowlist table in
`security-model.mdx` lists the new knob. No other page lists per-path
proxy repairs.
- Why not `body_forwarding.select_outbound_body`: it is the one place
both HTTP forwarders pick their bytes, and the downstream patch wraps it
there. But it is provider-agnostic byte policy and never sees a
WebSocket frame. A Responses-specific edit belongs in the Responses
handler, and the WebSocket needs it too.
- Why not gate on the ChatGPT backend: the failure was measured on
`chatgpt.com`, but the rule is upstream-neutral: the history references
a hosted tool the request does not declare. Dropping the item costs only
the search action. The drop also applies to API-key and custom-upstream
Responses traffic. A reviewer who prefers ChatGPT-auth only can gate the
HTTP call on `is_chatgpt_auth`; the WebSocket closure has the same flag
in scope.
- The `previous_response_id` skip is conservative. No Codex compaction
sets it, and an incremental request's earlier `additional_tools`
declarations are not in its body.
- Related: #3185 / #3186 (`additional_tools`), openai/codex#51962,
openai/codex#52016.
- Remove this workaround once a `tools: []` Responses body carrying a
`web_search_call` completes on the ChatGPT backend. The kill switch
makes that check a one-variable replay.
- Review fix fa7cea7d: the WebSocket prescreen no longer skips a frame
that spells the type with JSON escapes (`"web_search\u005fcall"`).
`test_frame_helper_decodes_escaped_type_spellings` and
`test_ws_escaped_web_search_call_frame[normal|bypass]` cover the helper
and the WS handler; both fail on 418faa61.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JD Davis
parent
ad62166ba3
commit
2a70d12819
@@ -284,6 +284,7 @@ accept arbitrary environment variables — writes are restricted to a hardcoded
|
||||
| `HEADROOM_INTERCEPT_READ_MIN_CHARS` | ast-grep read-interception threshold |
|
||||
| `HEADROOM_VERIFY_TRUNCATION_ON_DISK` | Verify ast-grep Read truncation against disk when no client banner is found. Always returns `UNKNOWN` on Windows, where `dir_fd` / `O_NOFOLLOW` support is unavailable. |
|
||||
| `HEADROOM_VERIFY_TRUNCATION_MAX_BYTES` | Max on-disk file size (bytes) the disk-verify fallback will read |
|
||||
| `HEADROOM_WEB_SEARCH_HISTORY` | Drop `web_search_call` history from a Responses request that declares no web_search tool (default on; `0` disables) |
|
||||
| `HEADROOM_BUFFERED_UPSTREAM_STREAM` | Send buffered CCR requests (and their continuations) upstream as `stream: true` and fold the reply (default on; `0` sends `stream: false`) |
|
||||
|
||||
Keys outside this list are ignored, and `/admin/upstream` is read-only — there is no
|
||||
|
||||
@@ -1129,6 +1129,109 @@ def _ensure_chatgpt_responses_store_false(
|
||||
return False
|
||||
|
||||
|
||||
def _web_search_history_drop_enabled() -> bool:
|
||||
from headroom.proxy import runtime_env
|
||||
|
||||
return (runtime_env.getenv("HEADROOM_WEB_SEARCH_HISTORY", "1") or "1").strip().lower() not in (
|
||||
"0",
|
||||
"false",
|
||||
"no",
|
||||
"off",
|
||||
)
|
||||
|
||||
|
||||
def _drop_undeclared_web_search_calls(
|
||||
payload: dict[str, Any],
|
||||
*,
|
||||
request_id: str | None = None,
|
||||
) -> int:
|
||||
"""Drop hosted ``web_search_call`` items from a request that cannot search.
|
||||
|
||||
Workaround for an OpenAI backend behaviour, not a Headroom bug. Codex on a
|
||||
custom model provider (how Headroom wraps it) compacts locally: it resends
|
||||
the whole history to ``/v1/responses`` with ``tools: []``. When that
|
||||
history holds a hosted ``web_search_call`` item, the ChatGPT backend
|
||||
answers 200 and then the in-band error "response protection is
|
||||
unavailable". Codex ignores in-band ``error`` events, reports "stream
|
||||
closed before response.completed", retries, and runs the same compaction
|
||||
before every later prompt, so the session never answers again. The same
|
||||
body completes with the ``web_search_call`` items removed, or with a
|
||||
web_search tool declared.
|
||||
|
||||
Only the search action (its query or URL) leaves the history; the results
|
||||
were never in it, and the model's answer that followed is a separate
|
||||
``message`` item that stays. A tool counts as declared in top-level
|
||||
``tools`` or in a Codex ``additional_tools`` input item (Codex >= 0.149
|
||||
sends tools that way), under any ``web_search*`` type spelling. A request
|
||||
continuing ``previous_response_id`` is left alone: its declarations may
|
||||
live server-side.
|
||||
|
||||
Mutates *payload* in place and returns the number of items dropped
|
||||
(0 = untouched). Callers skip it for a bypass request
|
||||
(``x-headroom-bypass: true``), which asked for its bytes untouched.
|
||||
Disable with ``HEADROOM_WEB_SEARCH_HISTORY=0`` (a registered runtime-env
|
||||
knob, so ``headroom wrap`` also pushes it to a reused proxy). Remove once
|
||||
the backend accepts such a body.
|
||||
"""
|
||||
items = payload.get("input")
|
||||
if not isinstance(items, list) or payload.get("previous_response_id"):
|
||||
return 0
|
||||
if not any(isinstance(item, dict) and item.get("type") == "web_search_call" for item in items):
|
||||
return 0
|
||||
tools = payload.get("tools")
|
||||
declared = list(tools) if isinstance(tools, list) else []
|
||||
for item in items:
|
||||
if isinstance(item, dict) and item.get("type") == "additional_tools":
|
||||
declared.extend(item["tools"] if isinstance(item.get("tools"), list) else [])
|
||||
if any(
|
||||
isinstance(tool, dict) and str(tool.get("type") or "").startswith("web_search")
|
||||
for tool in declared
|
||||
):
|
||||
return 0
|
||||
if not _web_search_history_drop_enabled():
|
||||
return 0
|
||||
kept = [
|
||||
item
|
||||
for item in items
|
||||
if not (isinstance(item, dict) and item.get("type") == "web_search_call")
|
||||
]
|
||||
payload["input"] = kept
|
||||
dropped = len(items) - len(kept)
|
||||
logger.info(
|
||||
"event=web_search_history request_id=%s dropped=%d reason=no_web_search_tool",
|
||||
request_id or "-",
|
||||
dropped,
|
||||
)
|
||||
return dropped
|
||||
|
||||
|
||||
def _drop_undeclared_web_search_calls_from_frame(
|
||||
raw_msg: str,
|
||||
*,
|
||||
request_id: str | None = None,
|
||||
) -> str:
|
||||
"""Apply :func:`_drop_undeclared_web_search_calls` to a WS ``response.create`` frame.
|
||||
|
||||
Returns *raw_msg* unchanged (same object) unless items were dropped. The
|
||||
substring prescreen keeps every frame without a ``web_search_call`` from
|
||||
paying a parse. JSON can spell the type with ``\\uXXXX`` escapes
|
||||
(``"web_search\\u005fcall"``), and a letter or underscore can only be
|
||||
escaped that way, so a frame holding any ``\\u`` is parsed too.
|
||||
"""
|
||||
if '"web_search_call"' not in raw_msg and "\\u" not in raw_msg:
|
||||
return raw_msg
|
||||
try:
|
||||
frame = json.loads(raw_msg)
|
||||
except (json.JSONDecodeError, TypeError, ValueError):
|
||||
return raw_msg
|
||||
if not isinstance(frame, dict) or frame.get("type") != "response.create":
|
||||
return raw_msg
|
||||
payload = frame["response"] if isinstance(frame.get("response"), dict) else frame
|
||||
if not _drop_undeclared_web_search_calls(payload, request_id=request_id):
|
||||
return raw_msg
|
||||
return json.dumps(frame)
|
||||
|
||||
|
||||
_RESPONSES_OUTPUT_ITEM_TYPES = frozenset(
|
||||
{
|
||||
"custom_tool_call_output",
|
||||
@@ -5771,6 +5874,12 @@ class OpenAIHandlerMixin:
|
||||
# minimal `messages` list purely for downstream memory injection and
|
||||
# telemetry; list-typed `input` is consulted directly by the unit
|
||||
# extraction helpers.
|
||||
#
|
||||
# Before anything reads `input`: drop web_search_call history from a
|
||||
# request that declares no web_search tool (Codex local compaction).
|
||||
# A bypass request asked for its bytes untouched, so it is left alone.
|
||||
if not _bypass and _drop_undeclared_web_search_calls(body, request_id=request_id):
|
||||
body_mutation_tracker.mark_mutated("responses_web_search_history")
|
||||
input_data = body.get("input", "")
|
||||
instructions = body.get("instructions")
|
||||
|
||||
@@ -7673,6 +7782,14 @@ class OpenAIHandlerMixin:
|
||||
model or "unknown",
|
||||
)
|
||||
|
||||
# Resolved before the frame helpers below, which read it.
|
||||
_ws_bypass = self._headroom_bypass_enabled(ws_headers)
|
||||
if _ws_bypass:
|
||||
logger.info(
|
||||
"[%s] WS /v1/responses passthrough reason=bypass_header mutation=disabled",
|
||||
request_id,
|
||||
)
|
||||
|
||||
def _normalize_ws_response_create_for_upstream(raw_msg: str) -> str:
|
||||
"""Optionally flatten Codex WS response.create frames for gateways.
|
||||
|
||||
@@ -7680,7 +7797,18 @@ class OpenAIHandlerMixin:
|
||||
OpenAI-compatible gateways accept the WebSocket upgrade but expect
|
||||
model/input/tools at the top level of response.create frames.
|
||||
Preserve default upstream behavior unless explicitly enabled.
|
||||
|
||||
Every response.create frame headed upstream (first frame, which
|
||||
the HTTP fallback also replays, later frames, memory
|
||||
continuations) passes through here, so it also drops
|
||||
web_search_call history the frame declares no tool for (see
|
||||
``_drop_undeclared_web_search_calls``), except on a bypass
|
||||
session, whose frames stay as the client sent them.
|
||||
"""
|
||||
if not _ws_bypass:
|
||||
raw_msg = _drop_undeclared_web_search_calls_from_frame(
|
||||
raw_msg, request_id=request_id
|
||||
)
|
||||
if os.environ.get(
|
||||
"HEADROOM_OPENAI_WS_FLATTEN_RESPONSE_CREATE", ""
|
||||
).strip().lower() not in (
|
||||
|
||||
@@ -84,6 +84,12 @@ RUNTIME_ENV_KNOBS: tuple[Knob, ...] = (
|
||||
"int",
|
||||
"Max on-disk file size (bytes) the disk-verify fallback will read.",
|
||||
),
|
||||
Knob(
|
||||
"HEADROOM_WEB_SEARCH_HISTORY",
|
||||
"bool",
|
||||
"Drop web_search_call history from a Responses request that declares no "
|
||||
"web_search tool (default on; 0 disables).",
|
||||
),
|
||||
Knob(
|
||||
"HEADROOM_BUFFERED_UPSTREAM_STREAM",
|
||||
"bool",
|
||||
|
||||
@@ -0,0 +1,413 @@
|
||||
"""web_search_call history on a Responses request that declares no web_search tool.
|
||||
|
||||
Codex on a custom provider compacts locally: it resends the whole history with
|
||||
``tools: []``. When that history holds a hosted ``web_search_call`` item, the
|
||||
ChatGPT backend answers 200 and then the in-band error "response protection is
|
||||
unavailable", and the Codex session never answers again. The proxy drops those
|
||||
items from such a request (HTTP and WebSocket) and leaves every other body
|
||||
byte-identical.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import copy
|
||||
import json
|
||||
import sys
|
||||
from typing import Any
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
|
||||
from headroom.proxy.handlers.openai import (
|
||||
_drop_undeclared_web_search_calls,
|
||||
_drop_undeclared_web_search_calls_from_frame,
|
||||
)
|
||||
from tests.test_openai_codex_ws_lifecycle import (
|
||||
_codex_lite_headers,
|
||||
_DummyOpenAIHandler,
|
||||
_FakeUpstream,
|
||||
_FakeWebSocket,
|
||||
_make_fake_websockets_module,
|
||||
)
|
||||
from tests.test_proxy_byte_faithful_forwarding import (
|
||||
_make_no_optimize_app,
|
||||
_start_proxy_log_capture,
|
||||
_stop_proxy_log_capture,
|
||||
)
|
||||
|
||||
|
||||
def _user(text: str) -> dict[str, Any]:
|
||||
return {"type": "message", "role": "user", "content": [{"type": "input_text", "text": text}]}
|
||||
|
||||
|
||||
def _search(n: int) -> dict[str, Any]:
|
||||
return {
|
||||
"type": "web_search_call",
|
||||
"id": f"ws_{n}",
|
||||
"status": "completed",
|
||||
"action": {"type": "search", "query": f"query {n}"},
|
||||
}
|
||||
|
||||
|
||||
def _answer(text: str) -> dict[str, Any]:
|
||||
return {
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"content": [{"type": "output_text", "text": text}],
|
||||
}
|
||||
|
||||
|
||||
def _compaction_body(**overrides: Any) -> dict[str, Any]:
|
||||
"""A Codex local-compaction request: full history, ``tools: []``."""
|
||||
body: dict[str, Any] = {
|
||||
"model": "gpt-5.5",
|
||||
"instructions": "Summarize the conversation.",
|
||||
"input": [
|
||||
_user("look this up"),
|
||||
{"type": "reasoning", "id": "rs_1", "summary": [], "encrypted_content": "enc"},
|
||||
_search(1),
|
||||
_search(2),
|
||||
_answer("found it"),
|
||||
_user("Summarize the conversation so far."),
|
||||
],
|
||||
"tools": [],
|
||||
"stream": True,
|
||||
"store": False,
|
||||
}
|
||||
body.update(overrides)
|
||||
return body
|
||||
|
||||
|
||||
def _types(body: dict[str, Any]) -> list[str]:
|
||||
return [item["type"] for item in body["input"]]
|
||||
|
||||
|
||||
# --- the decision ----------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"tools",
|
||||
[
|
||||
pytest.param([], id="empty"),
|
||||
pytest.param(None, id="absent"),
|
||||
pytest.param([{"type": "function", "name": "shell"}], id="function-only"),
|
||||
pytest.param([{"type": "function", "name": "web_search"}], id="function-named-web_search"),
|
||||
],
|
||||
)
|
||||
def test_drops_web_search_calls_when_no_web_search_tool_is_declared(tools) -> None:
|
||||
body = _compaction_body()
|
||||
if tools is None:
|
||||
del body["tools"]
|
||||
else:
|
||||
body["tools"] = tools
|
||||
|
||||
assert _drop_undeclared_web_search_calls(body, request_id="r1") == 2
|
||||
# Only the search actions leave; the answer and reasoning stay in order.
|
||||
assert _types(body) == ["message", "reasoning", "message", "message"]
|
||||
assert body["input"][2] == _answer("found it")
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"tool_type",
|
||||
["web_search", "web_search_preview", "web_search_preview_2025_03_11", "web_search_2025_08_26"],
|
||||
)
|
||||
def test_keeps_web_search_calls_when_a_web_search_tool_is_declared(tool_type: str) -> None:
|
||||
body = _compaction_body(tools=[{"type": "function", "name": "shell"}, {"type": tool_type}])
|
||||
before = copy.deepcopy(body)
|
||||
|
||||
assert _drop_undeclared_web_search_calls(body) == 0
|
||||
assert body == before
|
||||
|
||||
|
||||
def test_keeps_web_search_calls_declared_through_codex_additional_tools() -> None:
|
||||
# Codex >= 0.149 sends tools as an ``additional_tools`` input item and no
|
||||
# top-level ``tools``; a normal turn there must not lose its history.
|
||||
body = _compaction_body()
|
||||
del body["tools"]
|
||||
body["input"].insert(0, {"type": "additional_tools", "tools": [{"type": "web_search"}]})
|
||||
before = copy.deepcopy(body)
|
||||
|
||||
assert _drop_undeclared_web_search_calls(body) == 0
|
||||
assert body == before
|
||||
|
||||
|
||||
def test_leaves_unaffected_bodies_alone() -> None:
|
||||
no_search = _compaction_body()
|
||||
no_search["input"] = [item for item in no_search["input"] if item["type"] != "web_search_call"]
|
||||
continued = _compaction_body(previous_response_id="resp_1")
|
||||
string_input = _compaction_body(input="hello")
|
||||
for body in (no_search, continued, string_input):
|
||||
before = copy.deepcopy(body)
|
||||
assert _drop_undeclared_web_search_calls(body) == 0
|
||||
assert body == before
|
||||
|
||||
|
||||
def test_kill_switch_disables_the_drop(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv("HEADROOM_WEB_SEARCH_HISTORY", "0")
|
||||
body = _compaction_body()
|
||||
before = copy.deepcopy(body)
|
||||
|
||||
assert _drop_undeclared_web_search_calls(body) == 0
|
||||
assert body == before
|
||||
|
||||
|
||||
def test_frame_helper_returns_the_same_string_unless_it_drops() -> None:
|
||||
wrapped = json.dumps({"type": "response.create", "response": _compaction_body()})
|
||||
out = _drop_undeclared_web_search_calls_from_frame(wrapped)
|
||||
assert _types(json.loads(out)["response"]) == ["message", "reasoning", "message", "message"]
|
||||
|
||||
flat = json.dumps({"type": "response.create", **_compaction_body()})
|
||||
assert "web_search_call" not in _drop_undeclared_web_search_calls_from_frame(flat)
|
||||
|
||||
for raw in (
|
||||
json.dumps(
|
||||
{
|
||||
"type": "response.create",
|
||||
"response": _compaction_body(tools=[{"type": "web_search"}]),
|
||||
}
|
||||
),
|
||||
json.dumps({"type": "response.cancel", "response": _compaction_body()}),
|
||||
json.dumps({"type": "response.create", "response": {"model": "m", "input": "hi"}}),
|
||||
"not json",
|
||||
):
|
||||
assert _drop_undeclared_web_search_calls_from_frame(raw) is raw
|
||||
|
||||
|
||||
def test_frame_helper_decodes_escaped_type_spellings() -> None:
|
||||
# json.loads reads "web_search_call" as web_search_call, and so does
|
||||
# the backend, so the prescreen must not skip it.
|
||||
escaped = (
|
||||
'{"type":"response.create","response":{"input":'
|
||||
'[{"type":"web_search\\u005fcall","id":"ws_1"}],"tools":[]}}'
|
||||
)
|
||||
assert json.loads(escaped)["response"]["input"][0]["type"] == "web_search_call"
|
||||
assert json.loads(_drop_undeclared_web_search_calls_from_frame(escaped))["response"] == {
|
||||
"input": [],
|
||||
"tools": [],
|
||||
}
|
||||
|
||||
# Escapes alone change nothing: declared tool, no search history.
|
||||
for raw in (
|
||||
'{"type":"response.create","response":{"input":'
|
||||
'[{"type":"web_search\\u005fcall","id":"ws_1"}],"tools":[{"type":"web_search"}]}}',
|
||||
'{"type":"response.create","response":{"input":'
|
||||
'[{"type":"message","role":"user","content":"caf\\u00e9"}],"tools":[]}}',
|
||||
):
|
||||
assert _drop_undeclared_web_search_calls_from_frame(raw) is raw
|
||||
|
||||
|
||||
# --- HTTP /v1/responses ----------------------------------------------------
|
||||
|
||||
_CODEX_HEADERS = {
|
||||
"authorization": "Bearer test-token",
|
||||
"chatgpt-account-id": "acct_test",
|
||||
"originator": "codex_cli_rs",
|
||||
"content-type": "application/json",
|
||||
}
|
||||
|
||||
|
||||
def _post(
|
||||
body_bytes: bytes, *, stream: bool, extra_headers: dict[str, str] | None = None
|
||||
) -> tuple[bytes, list[str]]:
|
||||
headers = {**_CODEX_HEADERS, **(extra_headers or {})}
|
||||
client, transport = _make_no_optimize_app()
|
||||
proxy_logger, handler, prev_level, records = _start_proxy_log_capture()
|
||||
try:
|
||||
if stream:
|
||||
with client.stream(
|
||||
"POST", "/v1/responses", headers=headers, content=body_bytes
|
||||
) as response:
|
||||
assert response.status_code == 200
|
||||
for _ in response.iter_bytes():
|
||||
pass
|
||||
else:
|
||||
response = client.post("/v1/responses", headers=headers, content=body_bytes)
|
||||
assert response.status_code == 200, response.text
|
||||
finally:
|
||||
_stop_proxy_log_capture(proxy_logger, handler, prev_level)
|
||||
assert transport.captured_body is not None
|
||||
return transport.captured_body, [record.getMessage() for record in records]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("stream", [True, False], ids=["stream", "buffered"])
|
||||
def test_http_compaction_request_reaches_upstream_without_web_search_calls(stream: bool) -> None:
|
||||
body = _compaction_body(stream=stream)
|
||||
sent, logs = _post(json.dumps(body, indent=2).encode(), stream=stream)
|
||||
|
||||
wire = json.loads(sent)
|
||||
assert _types(wire) == ["message", "reasoning", "message", "message"]
|
||||
expected = copy.deepcopy(body)
|
||||
expected["input"] = [item for item in body["input"] if item["type"] != "web_search_call"]
|
||||
assert wire == expected
|
||||
assert any("event=web_search_history" in m and "dropped=2" in m for m in logs), logs
|
||||
assert any(
|
||||
"event=outbound_request" in m and "responses_web_search_history" in m for m in logs
|
||||
), logs
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"body",
|
||||
[
|
||||
pytest.param(_compaction_body(tools=[{"type": "web_search"}]), id="tool-declared"),
|
||||
pytest.param(
|
||||
{**_compaction_body(), "input": [_user("hi"), _answer("hello")]}, id="no-search-history"
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_http_unaffected_request_is_forwarded_byte_identical(body: dict[str, Any]) -> None:
|
||||
body_bytes = json.dumps(body, indent=2).encode()
|
||||
sent, logs = _post(body_bytes, stream=True)
|
||||
|
||||
assert sent == body_bytes
|
||||
assert not any("event=web_search_history" in m for m in logs)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("stream", [True, False], ids=["stream", "buffered"])
|
||||
def test_http_bypass_request_is_forwarded_byte_identical(stream: bool) -> None:
|
||||
# A bypass client asked Headroom to stay out of its bytes, so even a body
|
||||
# the backend will refuse goes out exactly as sent.
|
||||
body_bytes = json.dumps(_compaction_body(stream=stream), indent=2).encode()
|
||||
sent, logs = _post(body_bytes, stream=stream, extra_headers={"x-headroom-bypass": "true"})
|
||||
|
||||
assert sent == body_bytes
|
||||
assert not any("event=web_search_history" in m for m in logs)
|
||||
|
||||
|
||||
# --- Codex WebSocket /v1/responses ------------------------------------------
|
||||
|
||||
|
||||
def _frame(body: dict[str, Any]) -> str:
|
||||
return json.dumps({"type": "response.create", "response": body})
|
||||
|
||||
|
||||
def _upstream_events() -> list[str]:
|
||||
return [
|
||||
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
|
||||
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ws_first_frame_reaches_upstream_without_web_search_calls() -> None:
|
||||
upstream = _FakeUpstream(_upstream_events())
|
||||
client_ws = _FakeWebSocket(frames=[_frame(_compaction_body())])
|
||||
|
||||
with patch.dict(sys.modules, {"websockets": _make_fake_websockets_module(upstream)}):
|
||||
await _DummyOpenAIHandler().handle_openai_responses_ws(client_ws)
|
||||
|
||||
assert len(upstream.sent) == 1
|
||||
assert _types(json.loads(upstream.sent[0])["response"]) == [
|
||||
"message",
|
||||
"reasoning",
|
||||
"message",
|
||||
"message",
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ws_bypass_session_frames_reach_upstream_unchanged() -> None:
|
||||
compaction = _frame(_compaction_body())
|
||||
upstream = _FakeUpstream([], hold_after_events=True)
|
||||
client_ws = _FakeWebSocket(
|
||||
frames=[compaction, compaction],
|
||||
headers={"authorization": "Bearer test", "x-headroom-bypass": "true"},
|
||||
hold_after_initial=True,
|
||||
)
|
||||
|
||||
async def _disconnect_soon() -> None:
|
||||
await asyncio.sleep(0.05)
|
||||
client_ws.trigger_disconnect()
|
||||
|
||||
with patch.dict(sys.modules, {"websockets": _make_fake_websockets_module(upstream)}):
|
||||
trigger = asyncio.create_task(_disconnect_soon())
|
||||
try:
|
||||
await asyncio.wait_for(
|
||||
_DummyOpenAIHandler().handle_openai_responses_ws(client_ws), timeout=2.0
|
||||
)
|
||||
finally:
|
||||
trigger.cancel()
|
||||
|
||||
# First frame and a later frame both go out as the client sent them.
|
||||
assert upstream.sent == [compaction, compaction]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("bypass", [False, True], ids=["normal", "bypass"])
|
||||
async def test_ws_escaped_web_search_call_frame(bypass: bool) -> None:
|
||||
escaped = (
|
||||
'{"type":"response.create","response":{"model":"gpt-5-codex","input":'
|
||||
'[{"type":"web_search\\u005fcall","id":"ws_1"}],"tools":[]}}'
|
||||
)
|
||||
headers = {"authorization": "Bearer test"}
|
||||
if bypass:
|
||||
headers["x-headroom-bypass"] = "true"
|
||||
upstream = _FakeUpstream([], hold_after_events=True)
|
||||
client_ws = _FakeWebSocket(frames=[escaped], headers=headers, hold_after_initial=True)
|
||||
|
||||
async def _disconnect_soon() -> None:
|
||||
await asyncio.sleep(0.05)
|
||||
client_ws.trigger_disconnect()
|
||||
|
||||
with patch.dict(sys.modules, {"websockets": _make_fake_websockets_module(upstream)}):
|
||||
trigger = asyncio.create_task(_disconnect_soon())
|
||||
try:
|
||||
await asyncio.wait_for(
|
||||
_DummyOpenAIHandler().handle_openai_responses_ws(client_ws), timeout=2.0
|
||||
)
|
||||
finally:
|
||||
trigger.cancel()
|
||||
|
||||
assert len(upstream.sent) == 1
|
||||
if bypass:
|
||||
assert upstream.sent[0] == escaped
|
||||
else:
|
||||
assert json.loads(upstream.sent[0])["response"]["input"] == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ws_later_frame_reaches_upstream_without_web_search_calls() -> None:
|
||||
normal_turn = _frame(_compaction_body(tools=[{"type": "web_search"}]))
|
||||
compaction = _frame(_compaction_body())
|
||||
upstream = _FakeUpstream([], hold_after_events=True)
|
||||
client_ws = _FakeWebSocket(frames=[normal_turn, compaction], hold_after_initial=True)
|
||||
|
||||
async def _disconnect_soon() -> None:
|
||||
await asyncio.sleep(0.05)
|
||||
client_ws.trigger_disconnect()
|
||||
|
||||
with patch.dict(sys.modules, {"websockets": _make_fake_websockets_module(upstream)}):
|
||||
trigger = asyncio.create_task(_disconnect_soon())
|
||||
try:
|
||||
await asyncio.wait_for(
|
||||
_DummyOpenAIHandler().handle_openai_responses_ws(client_ws), timeout=2.0
|
||||
)
|
||||
finally:
|
||||
trigger.cancel()
|
||||
|
||||
assert len(upstream.sent) == 2
|
||||
# The turn that declares the tool goes out as the client sent it.
|
||||
assert upstream.sent[0] == normal_turn
|
||||
assert "web_search_call" not in upstream.sent[1]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ws_http_fallback_gets_the_frame_without_web_search_calls() -> None:
|
||||
fake_ws_mod = _make_fake_websockets_module(None, connect_error=RuntimeError("HTTP 500"))
|
||||
client_ws = _FakeWebSocket(
|
||||
frames=[_frame(_compaction_body())], headers=_codex_lite_headers(chatgpt=True)
|
||||
)
|
||||
handler = _DummyOpenAIHandler()
|
||||
fallback_frames: list[str] = []
|
||||
|
||||
async def _fallback(websocket, body, first_msg_raw, upstream_headers, request_id):
|
||||
fallback_frames.append(first_msg_raw)
|
||||
return 0, 0, 0, 0, 0
|
||||
|
||||
handler._ws_http_fallback = _fallback # type: ignore[assignment]
|
||||
|
||||
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
|
||||
await handler.handle_openai_responses_ws(client_ws)
|
||||
|
||||
assert len(fallback_frames) == 1
|
||||
assert "web_search_call" not in fallback_frames[0]
|
||||
@@ -129,6 +129,20 @@ def test_override_changes_astgrep_threshold_without_env():
|
||||
assert astgrep._min_chars_to_rewrite() == 500
|
||||
|
||||
|
||||
def test_override_disables_web_search_history_drop_without_env():
|
||||
from headroom.proxy.handlers.openai import _drop_undeclared_web_search_calls
|
||||
|
||||
def body() -> dict:
|
||||
return {"input": [{"type": "web_search_call", "id": "ws_1"}], "tools": []}
|
||||
|
||||
assert _drop_undeclared_web_search_calls(body()) == 1
|
||||
rt.set_overrides({"HEADROOM_WEB_SEARCH_HISTORY": "0"})
|
||||
assert _drop_undeclared_web_search_calls(body()) == 0
|
||||
assert rt.explicit_env({"HEADROOM_WEB_SEARCH_HISTORY": "0"}) == {
|
||||
"HEADROOM_WEB_SEARCH_HISTORY": "0"
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# /health surface + /admin/runtime-env hot-reload endpoint
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
Reference in New Issue
Block a user