Commit Graph
5 Commits
Author SHA1 Message Date
2a70d12819 fix(proxy): drop web_search_call history from a Responses request with no web_search tool (#4202)
## Description

Codex on a custom model provider (how `headroom wrap codex` configures
it) compacts locally: it resends the whole history to `/v1/responses`
with `tools: []`. When that history holds a hosted `web_search_call`
item, the ChatGPT backend answers 200 and then sends the in-band error
"response protection is unavailable". Codex ignores in-band `error`
events and reports "stream closed before response.completed". It retries
five times, then runs the same compaction before every later prompt, so
the session never answers again. Headroom forwards that body unchanged
today, on HTTP and on the Codex WebSocket.

The backend behaviour comes from a user report. A compaction body
captured from the failing session held 10 `web_search_call` items. Sent
straight to `chatgpt.com/backend-api/codex/responses`, with no Headroom
in the path, it failed every time. The same body completed with the 10
items removed, and also completed with `tools: [{"type":
"web_search"}]`. A normal turn on the same history, which declares the
tool, is unaffected.

The same failure appears without Headroom:
- openai/codex#51962: Codex's built-in provider, remote compaction of a
resumed history. Removing only the `web_search_call` items fixes it
there too.
- openai/codex#52016: the same two error strings with direct ChatGPT
login.

This PR drops `web_search_call` items from a Responses `input` when the
request declares no web_search tool, before the body leaves the proxy,
and marks the body mutated so the edited bytes are sent. Only the search
action (query or URL) leaves the history. The search results were never
in it. The answer the model gave afterwards is a separate `message`
item, and it stays. A bypass request (`x-headroom-bypass: true`) is left
untouched. Every other body is forwarded exactly as before.

This works around a backend behaviour; it does not fix a Headroom bug.
It should be removed once the backend accepts such a body.

Closes #4201

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **The drop.** `_drop_undeclared_web_search_calls(payload)`
(`headroom/proxy/handlers/openai.py`, next to
`_ensure_chatgpt_responses_store_false`) drops every `web_search_call`
item from `input`. It logs `event=web_search_history request_id=...
dropped=N reason=no_web_search_tool`. It does nothing in these cases:
- A tool whose `type` starts with `web_search` is declared
(`web_search`, `web_search_preview`, the dated variants), in top-level
`tools` or in a Codex `additional_tools` input item. Codex 0.149 and
later sends tools that way, with no top-level `tools` (#3185). Without
this check a normal Codex turn would lose its search history.
- The request continues a `previous_response_id`, because its
declarations may live server-side.
  - `input` is not a list, or holds no `web_search_call`.
  - `HEADROOM_WEB_SEARCH_HISTORY=0`.
- **HTTP `/v1/responses`** (`handle_openai_responses`): the drop runs
before the handler first reads `input`, so compression, memory and
telemetry see the same body that is sent. It is skipped when `_bypass`
is set (`x-headroom-bypass: true` or `x-headroom-mode: passthrough`). A
drop marks the `BodyMutationTracker` with
`responses_web_search_history`, and both the stream and the buffered
forwarder pick canonical bytes from that flag. With no drop, the
client's bytes go out verbatim, as before.
- **Codex WebSocket `/v1/responses`:** the drop is called from
`_normalize_ws_response_create_for_upstream`. Every `response.create`
frame passes through there on its way upstream: the first frame, the
WS-to-HTTP fallback that replays it, later frames, and memory
continuations.
- On a bypass session (`_ws_bypass`, the same header check on the
WebSocket handshake headers) the drop is skipped and every frame goes
out as the client sent it.
- `_ws_bypass` is now resolved just before that helper is defined,
instead of a few lines later, so the helper can read it. Its value and
log line are unchanged.
- `_drop_undeclared_web_search_calls_from_frame` returns the same string
object unless it dropped something. A frame is parsed only if it holds
the literal `"web_search_call"` or any `\u` escape, since JSON can spell
the type as `"web_search\u005fcall"` and a letter or underscore can only
be escaped that way; other frames skip the parse.
- **With `optimize` off:** the drop still runs, like the forced
`store=false`, because it is a repair the request needs in order to
succeed, not an optimization.
- **Kill switch:** `HEADROOM_WEB_SEARCH_HISTORY=0`, read per request
through `runtime_env.getenv`.
- It is registered in `RUNTIME_ENV_KNOBS`, so `headroom wrap` pushes it
to a reused proxy through `POST /admin/runtime-env`, and `/health`
reports it under `config.runtime_env`.
- The allowlist table in `docs/content/docs/security-model.mdx` gets the
matching row.
- **Tests in `tests/test_openai_responses_web_search_history.py` (22
tests):**
- The decision. Drops when `tools` is empty, absent, function-only, or
holds a function named `web_search`. Keeps with each web_search type
spelling, with a web_search tool in `additional_tools`, with
`previous_response_id`, with string `input`, with no search history, and
under the kill switch.
- The frame helper: wrapped and flattened frames, and the same object
back for anything it leaves alone.
  - Through the app:
- HTTP stream and buffered: upstream receives the body minus those
items, and the outbound log names the reason.
- HTTP byte-identical passthrough for a declared tool and for a history
with no searches.
- HTTP bypass, stream and buffered: byte-identical, no
`web_search_history` line.
- WS first frame, and a WS later frame after a normal turn that is
forwarded unchanged.
    - WS bypass session: first and later frame unchanged.
    - The WS-to-HTTP fallback.
- **Test in `tests/test_runtime_env.py`:**
`test_override_disables_web_search_history_drop_without_env` checks that
a runtime-env override of `0` turns the drop off with no environment
variable set, and that `explicit_env` puts the knob in the wrap payload.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

What was run:
- Unit tests: the new file, `tests/test_runtime_env.py`, and every
existing test file for the Responses handler, the Codex WebSocket, Codex
routing and byte-faithful forwarding. Not the whole suite.
- Lint and types: `ruff check`, `ruff format --check` (ruff 0.16.8, the
pre-commit pin) and `mypy` on the changed files. Not the whole tree.
- Every run used a temporary `HOME`, with `HEADROOM_SAVINGS_EVENTS_PATH`
in that directory and `HEADROOM_BEACON=off`.

### Test Output

With the two drop call sites removed from `openai.py`, which is upstream
`main` behaviour (the helpers stay so the test module imports):

```text
$ python -m pytest tests/test_openai_responses_web_search_history.py -q -p no:cacheprovider
FAILED tests/test_openai_responses_web_search_history.py::test_http_compaction_request_reaches_upstream_without_web_search_calls[stream]
FAILED tests/test_openai_responses_web_search_history.py::test_http_compaction_request_reaches_upstream_without_web_search_calls[buffered]
FAILED tests/test_openai_responses_web_search_history.py::test_ws_first_frame_reaches_upstream_without_web_search_calls
FAILED tests/test_openai_responses_web_search_history.py::test_ws_later_frame_reaches_upstream_without_web_search_calls
FAILED tests/test_openai_responses_web_search_history.py::test_ws_http_fallback_gets_the_frame_without_web_search_calls
=================== 5 failed, 17 passed, 1 warning in 2.17s ====================
```

With the bypass gates and the knob registration removed, which is this
PR's first revision, 45d134db:

```text
$ python -m pytest tests/test_openai_responses_web_search_history.py tests/test_runtime_env.py -q -p no:cacheprovider
E   assert b'{"model":"g...store":false}' == b'{\n  "model...re": false\n}'
E   assert b'{"model":"g...store":false}' == b'{\n  "model...re": false\n}'
E   assert ['{"type": "r...re": false}}'] == ['{"type": "r...re": false}}']
E   AssertionError: assert 1 == 0
FAILED tests/test_openai_responses_web_search_history.py::test_http_bypass_request_is_forwarded_byte_identical[stream]
FAILED tests/test_openai_responses_web_search_history.py::test_http_bypass_request_is_forwarded_byte_identical[buffered]
FAILED tests/test_openai_responses_web_search_history.py::test_ws_bypass_session_frames_reach_upstream_unchanged
FAILED tests/test_runtime_env.py::test_override_disables_web_search_history_drop_without_env
=================== 4 failed, 42 passed, 1 warning in 3.54s ====================
```

With the fix (418faa61):

```text
$ python -m pytest tests/test_openai_responses_web_search_history.py tests/test_runtime_env.py -q -p no:cacheprovider
======================== 46 passed, 1 warning in 8.57s =========================

$ python -m pytest tests/test_openai_responses_web_search_history.py tests/test_runtime_env.py tests/test_proxy_byte_faithful_forwarding.py tests/test_ccr_response_handler_openai_responses.py tests/test_codex_responses_passthrough_bytes.py tests/test_codex_responses_waste_signals.py tests/test_codex_ws_compression_scheduler.py tests/test_codex_ws_per_frame_memory.py tests/test_codex_ws_savings_deferral.py tests/test_openai_codex_routing.py tests/test_openai_codex_ws_lifecycle.py tests/test_openai_codex_ws_timings.py tests/test_openai_responses_additional_tools.py tests/test_openai_responses_buffered_sse.py tests/test_openai_responses_compression_units.py tests/test_openai_responses_context_compaction.py tests/test_openai_responses_null_arguments.py tests/test_openai_responses_output_shaper.py tests/test_openai_responses_read_protection.py tests/test_openai_responses_t3_replay_regression.py tests/test_openai_responses_traffic_learner.py tests/test_output_shaper_responses.py tests/test_provider_codex_responses.py tests/test_provider_openai_responses.py tests/test_proxy_openai_cache_key_integration.py tests/test_proxy_openai_cache_stability.py tests/test_proxy_openai_rate_limit_key.py tests/test_proxy_openai_responses_bypass.py tests/test_proxy_openai_responses_integration.py tests/test_proxy_openai_responses_stream_ccr.py tests/test_proxy_openai.py tests/test_responses_cross_turn_dedup.py tests/test_responses_pyo3_compression.py tests/test_responses_savings_denominator.py tests/test_responses_ws_pyo3_compression.py -q -p no:cacheprovider
================= 552 passed, 14 skipped, 2 warnings in 24.94s =================

$ ruff check headroom/proxy/handlers/openai.py headroom/proxy/runtime_env.py tests/test_openai_responses_web_search_history.py tests/test_runtime_env.py && ruff format --check headroom/proxy/handlers/openai.py headroom/proxy/runtime_env.py tests/test_openai_responses_web_search_history.py tests/test_runtime_env.py   # ruff 0.16.8
All checks passed!
4 files already formatted

$ mypy headroom/proxy/handlers/openai.py headroom/proxy/runtime_env.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: macOS 27.0.1 arm64, Python 3.11.13, a temporary `HOME`.
Upstream `main` f1bc2609 (exported with `git archive`) compared with
this branch at 418faa61, in the same venv.
- Exact command / steps: `repro_web_search_history.py` (below), run as
`PYTHONPATH=<tree> LITELLM_LOCAL_MODEL_COST_MAP=True python
repro_web_search_history.py`.
- It boots the real app with `create_app(ProxyConfig(optimize=False,
openai_api_url=<mock>))` inside a lifespan `TestClient`.
- The OpenAI upstream points at a local mock that records the bytes it
receives: an HTTP server for the POST and a `websockets` server for the
WebSocket.
- It sends a Codex-shaped compaction body (`tools: []`, two
`web_search_call` items, one `search` and one `open_page`) over HTTP
`POST /v1/responses` (stream) and over the Codex WebSocket.
- Each transport also gets two controls: the same history with `tools:
[{"type": "web_search"}]`, and the compaction body sent with
`x-headroom-bypass: true`.
- Observed result: main forwards all six requests with their
`web_search_call` items; this branch drops them from the `tools: []`
compaction body on HTTP and WebSocket and leaves both controls
byte-identical. Details:
- On `main`, all six requests reach upstream byte-identical,
`web_search_call` items included.
- On the branch, the compaction body loses its two items on both
transports and the drop is logged.
- On the branch, both controls go out byte-identical on both transports,
including the bypass request.

```text
=== main @ f1bc2609
== HTTP POST /v1/responses ==
-- compaction, tools: []
   client sent : ['message', 'web_search_call', 'web_search_call', 'message', 'message']
   upstream got: ['message', 'web_search_call', 'web_search_call', 'message', 'message']
   byte-identical: True
   log: event=outbound_request forwarder=streaming method=POST body_bytes=739 body_mutated=false mutation_reasons= source=passthrough
-- control, web_search declared
   client sent : ['message', 'web_search_call', 'web_search_call', 'message', 'message']
   upstream got: ['message', 'web_search_call', 'web_search_call', 'message', 'message']
   byte-identical: True
   log: event=outbound_request forwarder=streaming method=POST body_bytes=761 body_mutated=false mutation_reasons= source=passthrough
-- control, tools: [] with x-headroom-bypass: true
   client sent : ['message', 'web_search_call', 'web_search_call', 'message', 'message']
   upstream got: ['message', 'web_search_call', 'web_search_call', 'message', 'message']
   byte-identical: True
   log: event=outbound_request forwarder=streaming method=POST body_bytes=739 body_mutated=false mutation_reasons= source=passthrough
== WebSocket /v1/responses (response.create) ==
-- compaction, tools: []
   client sent : ['message', 'web_search_call', 'web_search_call', 'message', 'message']
   upstream got: ['message', 'web_search_call', 'web_search_call', 'message', 'message']
   byte-identical: True
   log: (no matching log line)
-- control, web_search declared
   client sent : ['message', 'web_search_call', 'web_search_call', 'message', 'message']
   upstream got: ['message', 'web_search_call', 'web_search_call', 'message', 'message']
   byte-identical: True
   log: (no matching log line)
-- control, tools: [] with x-headroom-bypass: true
   client sent : ['message', 'web_search_call', 'web_search_call', 'message', 'message']
   upstream got: ['message', 'web_search_call', 'web_search_call', 'message', 'message']
   byte-identical: True
   log: (no matching log line)
=== branch @ 418faa61
== HTTP POST /v1/responses ==
-- compaction, tools: []
   client sent : ['message', 'web_search_call', 'web_search_call', 'message', 'message']
   upstream got: ['message', 'message', 'message']
   byte-identical: False
   log: event=web_search_history dropped=2 reason=no_web_search_tool
   log: event=outbound_request forwarder=streaming method=POST body_bytes=443 body_mutated=true mutation_reasons=responses_web_search_history source=canonical
-- control, web_search declared
   client sent : ['message', 'web_search_call', 'web_search_call', 'message', 'message']
   upstream got: ['message', 'web_search_call', 'web_search_call', 'message', 'message']
   byte-identical: True
   log: event=outbound_request forwarder=streaming method=POST body_bytes=761 body_mutated=false mutation_reasons= source=passthrough
-- control, tools: [] with x-headroom-bypass: true
   client sent : ['message', 'web_search_call', 'web_search_call', 'message', 'message']
   upstream got: ['message', 'web_search_call', 'web_search_call', 'message', 'message']
   byte-identical: True
   log: event=outbound_request forwarder=streaming method=POST body_bytes=739 body_mutated=false mutation_reasons= source=passthrough
== WebSocket /v1/responses (response.create) ==
-- compaction, tools: []
   client sent : ['message', 'web_search_call', 'web_search_call', 'message', 'message']
   upstream got: ['message', 'message', 'message']
   byte-identical: False
   log: event=web_search_history dropped=2 reason=no_web_search_tool
-- control, web_search declared
   client sent : ['message', 'web_search_call', 'web_search_call', 'message', 'message']
   upstream got: ['message', 'web_search_call', 'web_search_call', 'message', 'message']
   byte-identical: True
   log: (no matching log line)
-- control, tools: [] with x-headroom-bypass: true
   client sent : ['message', 'web_search_call', 'web_search_call', 'message', 'message']
   upstream got: ['message', 'web_search_call', 'web_search_call', 'message', 'message']
   byte-identical: True
   log: (no matching log line)
```

The backend side was measured before this PR, from a user report. The
captured compaction body was sent straight to `chatgpt.com` with no
proxy in the path. As sent, it failed with "response protection is
unavailable". With its 10 `web_search_call` items removed, or with a
web_search tool declared, it completed. Headroom Desktop has carried the
same drop as a downstream patch since 2026-10-08, on the HTTP forwarders
only. Through that patch, the forked session that had failed six
compactions in a row compacted on the first try
(`event=web_search_history dropped=10`).

- Not tested: a live ChatGPT backend run of this branch, among others:
- This branch was not run against the live ChatGPT backend, and no live
Codex session went through it. The request shape and the backend A/B
above come from the downstream patch, which makes the same edit to the
same body.
- The WebSocket path has unit and mock coverage only. The downstream
verification ran over HTTP, because Headroom Desktop configures Codex
with `supports_websockets = false`. `headroom wrap codex` sets
`supports_websockets = true`, so most upstream users take the WebSocket
path.
- No captured Codex 0.149+ compaction body with `additional_tools` items
was available.
- The `headroom wrap` push of the knob to a reused proxy was not run end
to end. The test covers `explicit_env` and the override reaching the
reader.
  - Linux and Windows were not run.

<details>
<summary>repro_web_search_history.py</summary>

```python
"""Repro: web_search_call history on a Responses request with no web_search tool.

Boots the real Headroom proxy app (create_app + ProxyConfig, optimize off so
compression cannot be the variable) in-process, points its OpenAI upstream at
a local mock, and sends a Codex local-compaction body (full history,
``tools: []``, two hosted ``web_search_call`` items) over:

  1. HTTP POST /v1/responses (stream)
  2. the Codex WebSocket /v1/responses (first response.create frame)

plus two controls on each transport that must reach upstream unchanged: the
same history with a web_search tool declared (a normal Codex turn), and the
compaction body sent with ``x-headroom-bypass: true``.

The mock records what upstream received. No network beyond 127.0.0.1, no
real provider call.

Run against a source tree:
  PYTHONPATH=<tree> LITELLM_LOCAL_MODEL_COST_MAP=True <venv>/bin/python repro_web_search_history.py
"""

from __future__ import annotations

import asyncio
import json
import logging
import os
import re
import sys
import tempfile
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

_TMP = tempfile.mkdtemp(prefix="wsh-repro-")
os.environ["HOME"] = _TMP
os.environ["HEADROOM_SAVINGS_EVENTS_PATH"] = os.path.join(_TMP, "savings_events.jsonl")
os.environ.setdefault("HEADROOM_BEACON", "off")
os.environ.setdefault("HEADROOM_TELEMETRY", "off")

import headroom  # noqa: E402
import websockets  # noqa: E402
from fastapi.testclient import TestClient  # noqa: E402

from headroom.proxy.server import ProxyConfig, create_app  # noqa: E402

HEADERS = {
    "authorization": "Bearer sk-test-not-a-real-key",
    "user-agent": "codex_cli_rs/0.161.0",
    "originator": "codex_cli_rs",
    "content-type": "application/json",
}


BYPASS_HEADERS = {**HEADERS, "x-headroom-bypass": "true"}
CASES = (
    (False, HEADERS, "compaction, tools: []"),
    (True, HEADERS, "control, web_search declared"),
    (False, BYPASS_HEADERS, "control, tools: [] with x-headroom-bypass: true"),
)


def compaction_body(*, declare_web_search: bool) -> dict:
    return {
        "model": "gpt-5.5",
        "instructions": "Codex base instructions.",
        "input": [
            {
                "type": "message",
                "role": "user",
                "content": [{"type": "input_text", "text": "what changed in the release?"}],
            },
            {
                "type": "web_search_call",
                "id": "ws_1",
                "status": "completed",
                "action": {"type": "search", "query": "release notes"},
            },
            {
                "type": "web_search_call",
                "id": "ws_2",
                "status": "completed",
                "action": {"type": "open_page", "url": "https://example.com/notes"},
            },
            {
                "type": "message",
                "role": "assistant",
                "content": [{"type": "output_text", "text": "Three fixes shipped."}],
            },
            {
                "type": "message",
                "role": "user",
                "content": [{"type": "input_text", "text": "Summarize the conversation."}],
            },
        ],
        "tools": [{"type": "web_search"}] if declare_web_search else [],
        "tool_choice": "auto",
        "stream": True,
        "store": False,
    }


def types(payload: dict) -> list[str]:
    return [item["type"] for item in payload["input"]]


SSE = (
    b'event: response.created\ndata: {"type":"response.created","response":{"id":"r_1"}}\n\n'
    b'event: response.completed\ndata: {"type":"response.completed","response":{"id":"r_1",'
    b'"usage":{"input_tokens":10,"output_tokens":2,"total_tokens":12}}}\n\n'
)


class _HttpUpstream(BaseHTTPRequestHandler):
    received: list[bytes] = []

    def do_POST(self) -> None:  # noqa: N802
        self.received.append(self.rfile.read(int(self.headers["content-length"])))
        self.send_response(200)
        self.send_header("content-type", "text/event-stream")
        self.send_header("content-length", str(len(SSE)))
        self.end_headers()
        self.wfile.write(SSE)

    def log_message(self, *args) -> None:
        pass


class _WsUpstream:
    def __init__(self) -> None:
        self.received: list[str] = []
        self.port = 0
        ready = threading.Event()

        async def handler(conn) -> None:
            async for frame in conn:
                self.received.append(frame)
                await conn.send(json.dumps({"type": "response.created", "response": {"id": "r_1"}}))
                await conn.send(
                    json.dumps({"type": "response.completed", "response": {"id": "r_1"}})
                )

        async def main() -> None:
            async with websockets.serve(handler, "127.0.0.1", 0) as server:
                self.port = server.sockets[0].getsockname()[1]
                ready.set()
                await asyncio.Future()

        threading.Thread(target=lambda: asyncio.run(main()), daemon=True).start()
        ready.wait(5)


def make_client(upstream_url: str) -> TestClient:
    config = ProxyConfig(
        optimize=False,
        cache_enabled=False,
        rate_limit_enabled=False,
        cost_tracking_enabled=False,
        log_requests=False,
        ccr_inject_tool=False,
        ccr_handle_responses=False,
        ccr_context_tracking=False,
        image_optimize=False,
        openai_api_url=upstream_url,
    )
    return TestClient(create_app(config))


class _Lines(logging.Handler):
    def __init__(self) -> None:
        super().__init__(logging.INFO)
        self.lines: list[str] = []

    def emit(self, record: logging.LogRecord) -> None:
        msg = record.getMessage()
        if "web_search_history" in msg or (
            "event=outbound_request" in msg and "/v1/responses" in msg
        ):
            self.lines.append(msg)


def report(label: str, sent_bytes: bytes | str, got: bytes | str, logs: _Lines) -> None:
    sent = json.loads(sent_bytes)
    wire = json.loads(got)
    sent_payload = sent.get("response", sent)
    wire_payload = wire.get("response", wire)
    print(f"-- {label}")
    print(f"   client sent : {types(sent_payload)}")
    print(f"   upstream got: {types(wire_payload)}")
    print(f"   byte-identical: {sent_bytes == got}")
    for line in logs.lines or ["(no matching log line)"]:
        print("   log: " + re.sub(r" (path|request_id)=\S+", "", line))
    logs.lines.clear()


def main() -> int:
    proxy_logger = logging.getLogger("headroom.proxy")
    proxy_logger.setLevel(logging.INFO)
    logs = _Lines()
    proxy_logger.addHandler(logs)
    print(f"headroom source: {os.path.dirname(headroom.__file__)}")

    http_server = ThreadingHTTPServer(("127.0.0.1", 0), _HttpUpstream)
    threading.Thread(target=http_server.serve_forever, daemon=True).start()
    print("== HTTP POST /v1/responses ==")
    with make_client(f"http://127.0.0.1:{http_server.server_port}") as client:
        for declare, headers, label in CASES:
            body = json.dumps(compaction_body(declare_web_search=declare)).encode()
            with client.stream("POST", "/v1/responses", headers=headers, content=body) as response:
                assert response.status_code == 200, response.status_code
                response.read()
            report(label, body, _HttpUpstream.received[-1], logs)

    ws_upstream = _WsUpstream()
    print("== WebSocket /v1/responses (response.create) ==")
    with make_client(f"http://127.0.0.1:{ws_upstream.port}") as client:
        for declare, headers, label in CASES:
            frame = json.dumps(
                {"type": "response.create", "response": compaction_body(declare_web_search=declare)}
            )
            before = len(ws_upstream.received)
            with client.websocket_connect("/v1/responses", headers=headers) as ws:
                ws.send_text(frame)
                while json.loads(ws.receive_text()).get("type") != "response.completed":
                    pass
            assert len(ws_upstream.received) == before + 1
            report(label, frame, ws_upstream.received[-1], logs)
    return 0


if __name__ == "__main__":
    sys.exit(main())
```

</details>

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes, for `/v1/responses` requests
(HTTP and Codex WebSocket) that meet all of these:
  - `input` holds `web_search_call` items;
- neither `tools` nor an `additional_tools` item declares a web_search
tool;
  - no `previous_response_id` is set;
- the request does not carry `x-headroom-bypass: true` or
`x-headroom-mode: passthrough`.

Those requests now go upstream without the `web_search_call` items,
re-serialized: canonical bytes on HTTP, `json.dumps` of the frame on the
WebSocket. Any other edit already made to the body rides along, as it
does whenever the body is mutated. One example is the forced
`store=false` on ChatGPT auth. Every other request is forwarded as
before, byte-identical when nothing else changed it. Bypass requests
stay byte-identical. Chat Completions, Anthropic, Gemini and the
`/v1/responses/{subpath}` passthrough are not touched.
- Kill switch / disable path: `HEADROOM_WEB_SEARCH_HISTORY=0` (also
`false`, `no`, `off`). It is read per request and registered as a
runtime-env knob, so there are three ways to set it:
- in the proxy's environment, which takes effect when the proxy next
starts;
- exported in the shell of a `headroom wrap` that reuses a running proxy
(wrap pushes it, no restart);
  - posted to `POST /admin/runtime-env` (no restart).

  A single request can also opt out with `x-headroom-bypass: true`.
- Unsafe override required: No.
- Qualification impact: only requests the backend was already failing
are changed, so no completed request's compression or cache result
moves. Such a request now completes, without reusing the previous turn's
cached prefix past the first dropped item.
- Over HTTP the drop runs before compression, so token counts and
savings are computed on the edited body, and the removed items are not
counted as saved.
- On the WebSocket it runs after compression, so the items appear in
both the before and after counts, and tokens saved is unchanged.
- Rollback path: set the kill switch, or revert these commits. Either
restores byte-identical forwarding of these bodies, and the failing
compactions with it.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Review follow-ups:
- 2099603a: bypass requests are no longer rewritten, on HTTP or on the
WebSocket.
- 418faa61: registers the kill switch as a runtime-env knob, so a reused
proxy honours it.
- Documentation: the `POST /admin/runtime-env` allowlist table in
`security-model.mdx` lists the new knob. No other page lists per-path
proxy repairs.
- Why not `body_forwarding.select_outbound_body`: it is the one place
both HTTP forwarders pick their bytes, and the downstream patch wraps it
there. But it is provider-agnostic byte policy and never sees a
WebSocket frame. A Responses-specific edit belongs in the Responses
handler, and the WebSocket needs it too.
- Why not gate on the ChatGPT backend: the failure was measured on
`chatgpt.com`, but the rule is upstream-neutral: the history references
a hosted tool the request does not declare. Dropping the item costs only
the search action. The drop also applies to API-key and custom-upstream
Responses traffic. A reviewer who prefers ChatGPT-auth only can gate the
HTTP call on `is_chatgpt_auth`; the WebSocket closure has the same flag
in scope.
- The `previous_response_id` skip is conservative. No Codex compaction
sets it, and an incremental request's earlier `additional_tools`
declarations are not in its body.
- Related: #3185 / #3186 (`additional_tools`), openai/codex#51962,
openai/codex#52016.
- Remove this workaround once a `tools: []` Responses body carrying a
`web_search_call` completes on the ChatGPT backend. The kill switch
makes that check a one-variable replay.
- Review fix fa7cea7d: the WebSocket prescreen no longer skips a frame
that spells the type with JSON escapes (`"web_search\u005fcall"`).
`test_frame_helper_decodes_escaped_type_spellings` and
`test_ws_escaped_web_search_call_frame[normal|bypass]` cover the helper
and the WS handler; both fail on 418faa61.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-24 08:39:00 -07:00
Anupam MedirattaandClaude Sonnet 5 97aa949924 fix: fix security issue in server.py (#3593)
## Description
Add an explicit 64 KiB request-body limit to `POST /admin/runtime-env`
to prevent
unbounded request-body buffering and JSON parsing.

## Type of Change
- [x] Security fix
- [x] Bug fix

## Changes Made
- Enforce the 64 KiB request-body limit against bytes actually consumed
from the
ASGI request stream (`request.stream()`), aborting with 413 as soon as
the
  cumulative size exceeds the cap.
- Parse JSON only after the bounded body has been fully read;
`request.json()`
  (which buffers unconditionally) is no longer called.
- Do not rely on the client-controlled `Content-Length` header as the
  enforcement boundary.
- Added endpoint-level regression tests for: a body at/under the limit
with no
  `Content-Length`, an oversized chunked/streamed body, and a body whose
  declared `Content-Length` understates the actual bytes delivered.
- Removed an unrelated `min_tokens` → `min_ntoks` log-label rename that
had been
  bundled into the original commit.

## Testing
- New: 64 KiB-or-smaller request without `Content-Length` → 200
- New: oversized streamed/chunked request → 413, JSON parser never
invoked
- New: `Content-Length` smaller than actual streamed body → 413, JSON
parser never invoked
- Existing `tests/test_runtime_env.py` suite (23/23 passing)

## Real Behavior Proof
`POST /admin/runtime-env` rejects bodies exceeding 64 KiB before any
JSON
parsing occurs, regardless of what `Content-Length` claims or whether
the
request is chunked.

## Runtime Rollout Safety
The change is limited to request-body handling for `/admin/runtime-env`
(loopback-only, same-origin-guarded admin endpoint). Requests within the
64 KiB limit behave exactly as before.

## Review Readiness
- [x] I have performed a self-review
- [x] Tests cover the security boundary
- [x] This PR is ready for human review

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-17 15:47:04 -07:00
Tejas Chopra 1390d89715 fix(shaper): measure output-token steering, and stop claiming savings we did not measure (#3377)
## What this does

Verbosity steering already shipped (BETA, L2). This branch measures it
properly, fixes two ways it was misbehaving, and changes what Headroom
is willing to *claim* about it.

Net effect for a user: steering saves more than it did, on more models,
and the dashboard stops showing a savings percentage that was never
measured on their traffic.

## Three changes worth reviewing separately

### 1. The L3 steering text had two defects, both measured

**"Reproduce code, error strings, numbers, units, and API or CLI names
exactly"** read as a licence to emit code. Against the same SWE-bench
turn it left gpt-5.2 unchanged but inflated gpt-5.1 by 58% (1631 → 2573
output tokens), cutting L3's saving on that model from 51% to 23%.
Replaced with a completeness floor that forbids dropping task-critical
content — negations included — without instructing the model to
reproduce anything. The whole OpenAI family went from *negative* to
+31.2%.

**"reference by path and line"** was too soft. Told to stop restating
code, models wrote "change the check in the mixin" and the reader had to
ask which file. Measured with a blind judge playing the engineer who
must apply the fix: of four blocker categories only "under-specified
location" moved between L0 and L3 (0 → 3 across 9 paired tasks). Notably
the *predicted* failure — "omit rationale" → "but why?" — scored zero at
both levels. Making the citation obligatory took that category to 0 and
turn inflation from +23.1% to 0.0%.

Suppressing content is safe. Suppressing the pointer to it is not: the
reader buys it back at the cost of a whole turn, which at agentic
context sizes is far more expensive than the tokens saved.

### 2. The savings table now ships empty

`MODELLED_REDUCTION` previously carried built-in factors. It now ships
`{}` plus a `register_modelled_factors()` seam.

A credible reduction factor is not a constant. It depends on the model
family, the shape of the turn, and the exact steering text — and
producing one means running a paired benchmark across models and paying
for both arms. A constant compiled into the source is a number measured
on somebody else's traffic.

So `estimate_from_model` returns `None` for every level and the
dashboard renders its "not measured" fallback. That is the correct
rendering of *not measured*; a number nobody measured is not. Both real
paths are unchanged and documented at the table: run a holdout (which
outranks this tier entirely), or register factors from a benchmark.

The old built-in figure is a good argument for this: it claimed `(0.336,
0.738)` from five synthetic prompts, measured against a version of the
L3 string that has since changed twice. It over-reported by roughly 2×
at the optimistic end — the direction that matters for a number on a
dashboard.

### 3. Everything else

Thinking tokens split from visible output (`thinking_tokens.py`), the
estimator that was wired but never ran now runs, and the
`OUTCOME_OBSERVED` pipeline stage.

An earlier commit in this range adds a `headroom-output` plugin and a
writable `PRE_SEND_PARAMS` body hook; a later one removes both. Live
measurement showed per-turn effort routing ~15× underwater on mechanical
turns, so the hypothesis it was built on did not survive. The net diff
has neither, and a test now pins the hook as deliberately absent — an
unused writable-body seam is a liability, not an extension point.

## How it was measured

SWE-bench Verified, 12 instances carrying the real source files each
gold patch touches (6k–48k chars), stratified by the dataset's
difficulty label, run at L0 and L3 across six models and paired on
instance. Truncated pairs excluded — a capped response's length is the
ceiling, not the model's choice.

| | pairs | reduction |
|---|---|---|
| Anthropic (opus, sonnet, haiku) | 33 | 18.2% |
| OpenAI (terra, luna, sol) | 28 | 31.2% |

61 usable pairs, median +31.8%, mean +23.0% (SE 5.3, t=4.4), all six
positive in sign, 13% of individual pairs negative.

**On power, since it is easy to over-read the per-model figures:** only
luna (t=6.0), terra (t=3.0) and sonnet (t=2.7) are individually
distinguishable from zero. sol, haiku and opus are not (t≈1.3, 1.3,
0.7). They are single samples per instance against a model whose
*unsteered* output varies by a median 33% run to run — measured
directly, by re-running the L0 arm on identical prompts. The aggregate
effect is solid; a single-model number from this run is not.

## Risk

- Steering is the one lever that mutates the provider prefix-cache key,
so the text is byte-stable and applied once per conversation;
`steering_allowed_for` suppresses it entirely in `mode="cache"`.
- The feature stays opt-in (`available_in=STABLE`, no
`default_enabled_in`). L3 is a visible change to every response and
should be a choice.
- `shaper_enabled_for` keeps byte-faithful forwarding intact: default-on
plus `optimize=False` means don't shape, because creating a `system`
field where the client sent none breaks the contract.

## Testing

612 targeted tests pass; full suite green apart from
`test_mcp_reconcile.py::test_ordinary_install_does_not_adopt_serena`,
which fails identically on `main`.

The final commit fixes a UP037 lint error that arrived with the
savings-rendering work on main. It is unrelated to this branch but fails
CI lint for anything opened on top of it.
2026-09-01 15:28:09 +05:30
3077ac81e8 feat: add deterministic runtime rollout controls (#1490)
## Description

Establish one centrally resolved, observable, deterministic, versioned
runtime rollout-control mechanism for Headroom. Runtime rollout controls
which behaviors an already-built artifact may expose; it does not select
or qualify a Headroom release/version.

## Type of Change

- [x] New feature (non-breaking change that adds functionality)
- [x] Bug fix (non-breaking change that fixes rollout enforcement
regressions)
- [x] Documentation update
- [x] Code refactoring (no functional changes)

## Changes Made

- Added `RolloutChannel`, `HEADROOM_ROLLOUT_CHANNEL`,
`--rollout-channel`, and a versioned immutable `RolloutSnapshot` shared
by Python configuration boundaries.
- Added schema/policy versions, canonical registry and snapshot SHA-256
identities, per-feature decision reasons, disable precedence, unsafe
qualification poisoning, strict CLI validation, and fail-closed
environment handling.
- Added `headroom rollout status --json`, Python `/stats.rollout`, and
Rust `/rollout/status` runtime provenance.
- Added equivalent Rust snapshot semantics and shared Python/Rust policy
vectors while retaining language-specific feature registries.
- Enforced rollout policy at alternate Python server composition roots
so `HEADROOM_READ_MATURATION=1` cannot bypass its beta gate.
- Preserved typed rollout snapshots across multi-worker serialization
with schema, policy, registry, snapshot-digest, type, and feature-name
validation.
- Made loopback runtime output-shaper updates replace the immutable
snapshot atomically for request readers, retain explicit request/disable
provenance, preserve channel and kill-switch precedence, invalidate
cached stats, and return the effective rollout decision.
- Made `headroom learn --verbosity --apply` report a channel-blocked
update instead of claiming the shaper is live.
- Made explicit CLI feature flags fail loudly when their current channel
blocks them.
- Made persistent interceptor installation select canary automatically,
or reject an explicitly insufficient channel unless the break-glass
override is set.
- Updated architecture, proxy, rollout, learn, and output-shaper
documentation with required channels and hot-reload semantics.

## Testing

- [x] Unit tests pass
- [x] Linting passes (`ruff check .` and `ruff format --check .`)
- [x] Type checking passes (`mypy headroom --ignore-missing-imports`)
- [x] New regression tests added for every corrected behavior
- [x] Rust tests and production-target Clippy pass
- [x] Documentation build passes

### Test Output

```text
Focused rollout coverage suite
57 passed; headroom.rollout + rollout CLI: 98% coverage

Affected proxy/rollout/transform/governance suites
222 passed; 0 failed

Final changed regression suites
100 passed; 0 failed

Cross-module hot-reload isolation regression
6 passed; 0 failed

cargo test -p headroom-core -p headroom-proxy --quiet
headroom-core: 924 passed; 1 ignored
headroom-proxy and integration suites: all passed

cargo clippy -p headroom-core -p headroom-proxy --lib --bins -- -D warnings
cargo fmt --all -- --check
ruff check .
ruff format --check .
mypy headroom --ignore-missing-imports
git diff --check
All passed

cd docs && npm run build
Compiled successfully; 164 static pages generated
```

The unsharded Windows-only CI selection exposed unrelated baseline
failures, principally the existing `sqlite:///C:\\...` URL parser
producing an invalid `\\C:\\...` path. At commit `8e793a80`, all 52
completed GitHub checks passed; the only other conclusions are expected
skips and superseded governance jobs.

## Real Behavior Proof

- **Environment:** Windows checkout on Python 3.13.3 and the current
Rust workspace, based on upstream `main` at `93f2d7a2`.
- **Exact command / steps:** Exercised canary and beta feature requests
through CLI status, Python `/stats.rollout`, Rust `/rollout/status`,
multi-worker payload round trips, loopback `/admin/runtime-env`, real
proxy request shaping before/after hot reload, installer manifest
generation, and shared Python/Rust policy vectors.
- **Observed result:** Stable blocks unstable requests; disable wins
over explicit/default/legacy/unsafe paths; unsafe state reports
`qualification_eligible=false`; worker handoff rejects tampering;
running output shaping changes only when the effective beta policy
permits it; explicit blocked flags fail with actionable diagnostics.
- **Not tested:** Live production traffic requiring provider
credentials, or future artifact qualification/promotion automation
(intentionally out of scope).

## Runtime Rollout Safety

- **Rollout-managed features:** Python `tool_result_interceptors`,
`proxy_output_shaper`, `read_maturation`; Rust `native_bedrock`,
`openai_responses_streaming`, `canary_probe`.
- **Minimum rollout channel:** Registry-defined per feature; process
default is `stable`.
- **Stable/default behavior changed:** No unstable feature becomes
enabled by default. Explicit blocked CLI flags now fail instead of
silently doing nothing.
- **Kill switch / disable path:**
`HEADROOM_DISABLE_FEATURES=<comma-separated feature names>`; explicit
disable has highest precedence, including over the unsafe override.
- **Unsafe override required:** No.
`HEADROOM_UNSAFE_ALLOW_UNSTABLE_FEATURES=1` is break-glass only and
makes qualification evidence ineligible.
- **Qualification impact:** Adds machine-readable policy/snapshot
identities and eligibility; does not implement qualification itself.
- **Rollback path:** Set the named disable list for operational
rollback, lower the channel, or revert this PR.

## Review Readiness

- [x] I have performed a full diff review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented hard-to-understand areas
- [x] I have made corresponding documentation changes
- [x] My changes generate no new warnings
- [x] I added tests that reproduce and prevent every regression fixed
during review
- [x] New and existing affected tests pass locally
- [x] I did **not** edit `CHANGELOG.md`; release-please generates it
from the Conventional Commit PR title

## Additional Notes

Out of scope: artifact candidates, benchmark orchestration,
qualification manifests/gates, promotion automation, release branches,
publication guards, and release-risk classification. Those workflows can
consume the rollout registry digest, runtime snapshot digest, decision
reasons, and qualification eligibility through supported black-box
interfaces.

---------

Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
Co-authored-by: JD Davis <jd@JDH-AIR-00.local>
2026-08-12 23:16:54 -05:00
Tejas Chopra 6904d47a01 feat(proxy): hot-reload live env knobs so a reused proxy picks them up without a restart (#1090)
## Description

A small class of env vars is read by the proxy **live, per request** —
the output-shaper family (`HEADROOM_OUTPUT_SHAPER`,
`HEADROOM_VERBOSITY_LEVEL`, `HEADROOM_EFFORT_ROUTER`,
`HEADROOM_MECHANICAL_EFFORT`, `HEADROOM_VERBOSITY_AUTOTUNE`,
`HEADROOM_OUTPUT_HOLDOUT`), or captured at import
(`HEADROOM_INTERCEPT_READ_MIN_CHARS`). The proxy reads them from its own
process environment, fixed at launch. But `headroom wrap` reuses an
already-running proxy (it restarts only on startup-config drift), so a
value exported *after* the proxy started silently no-op'd — e.g. `export
HEADROOM_OUTPUT_SHAPER=1` had zero effect on a reused proxy on `:8787`.

This PR makes those live knobs **hot-reloadable**: `headroom wrap`
pushes them to the running proxy, which applies them in memory — no
restart (a restart would cold-start the ML stack, drop in-flight
requests, and lose CCR/router caches).

_No linked issue._

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/runtime_env.py` (new): single source of truth
registering the live knobs + a thread-safe process-global override
store. `getenv()` (override-then-env) is a drop-in for `os.environ.get`;
behaviour is byte-identical when no override is set.
- Readers rerouted through `runtime_env.getenv`: `output_shaper.py`, the
anthropic holdout read, and the ast-grep threshold (now a live read, not
an import-time constant).
- Proxy: loopback-only `POST /admin/runtime-env` applies overrides in
memory; `/health` → `config.runtime_env` surfaces the live values so
reuse is observable.
- `wrap`: after attaching to a proxy (all call sites), best-effort push
of the session's **explicitly-set** knobs. No-ops if nothing is set,
`--no-proxy`, the proxy is unreachable, or it predates the endpoint
(404). Only explicitly-set knobs are pushed, so a session never clobbers
another with a default it never asked for.
- Docs: README + output-token-reduction guide document the
global-override caveat.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_runtime_env.py -q
16 passed

$ python -m pytest tests/test_runtime_env.py tests/test_output_shaper.py -q
50 passed

$ ruff check headroom/proxy/runtime_env.py headroom/proxy/output_shaper.py headroom/proxy/handlers/anthropic.py headroom/proxy/interceptors/astgrep.py headroom/proxy/server.py headroom/cli/wrap.py
All checks passed!

$ mypy headroom/proxy/runtime_env.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: local macOS, Python 3.12 `.venv`, branch
`fix/runtime-env-hot-reload` at the PR head.
- Exact command / steps: ran the test suites above. The 16 new
`test_runtime_env` tests exercise the registry/store, overrides reaching
the shaper + the ast-grep threshold, the `POST /admin/runtime-env` apply
+ `/health` reflect + loopback-only 404 + 400-on-non-object, and the
wrap push payload / no-op / error-swallow paths.
- Observed result: 50 passed; ruff + mypy clean on the changed modules;
an override set via the endpoint is read by `getenv()` at the shaper and
surfaced in `/health` config.
- Not tested: a literal two-terminal manual session (start a proxy,
`headroom wrap` a second session, `export HEADROOM_OUTPUT_SHAPER=1`,
confirm the reused proxy picks it up). The behaviour is covered by the
endpoint + wrap-push integration tests, but was not exercised by hand
here.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

- **Inherent caveat (documented):** overrides are global to the proxy —
one process serves every attached wrapper, so the last explicit setting
wins. No mechanism (restart or hot-reload) can give two sessions on one
shared proxy different output-shaper settings.
- **Scope:** startup-captured settings (`HEADROOM_TARGET_RATIO` etc.)
are intentionally out of scope — a fresh proxy already gets them and
they ride the existing `/health` config channel.
- **Merge blocker:** this branch is currently **CONFLICTING with
`main`** and needs a rebase/merge before it can land.
- CHANGELOG.md left unchanged — releases are managed by release-please
from conventional commits.
2026-06-18 09:50:50 -07:00