mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description
Add an optional on_responses_create turn hook so extensions can replace
an OpenAI Responses request before model-dependent proxy processing. The
hook applies to HTTP requests and WebSocket response.create frames,
including subsequent frames, and respects the Headroom bypass header.
## Type of Change
- [x] New feature (optional turn-hook method)
## Changes Made
- Expose ResponsesCreate with request body, headers, transport, and
connection identity.
- Run ordered synchronous/asynchronous hooks and isolate hook
exceptions.
- Apply replacement bodies before compression, model-dependent
accounting, and HTTP dispatch.
- Apply the hook to initial and subsequent WebSocket create frames.
- Add handler-level regression coverage for replacement, no-op, bypass,
and streaming behavior.
## Testing
- [x] Maintainer focused hook and WebSocket lifecycle suites passed.
- [x] Touched-file Ruff and whitespace checks passed.
### Test Output
```text
python -m pytest tests/test_responses_create_hooks.py tests/test_turn_hooks.py tests/test_openai_codex_ws_lifecycle.py tests/test_original_snapshot_in_place_hook.py -q
76 passed, 1 warning in 13.93s
ruff check headroom/proxy/turn_hooks.py headroom/proxy/handlers/openai.py tests/test_responses_create_hooks.py
All checks passed!
```
The latest main sync incorporates the merged snapshot-preservation fix
from #4241. All 76 focused hook, WebSocket, and snapshot tests passed
after that clean sync. Hosted CI is qualifying the newly pushed head.
Maintainer tests used the available development native extension; no
newly built native binary or independent live-provider validation is
claimed.
## Real Behavior Proof
- Environment: isolated Windows/Python 3.12.13 maintainer worktree, real
HTTP/WebSocket handlers exercised through repository test fixtures.
- Exact command / steps: run the four pytest suites listed above;
inspect replacement, bypass, no-op, and WebSocket lifecycle cases.
- Observed result: all 76 focused tests passed, including routed
model/effort reaching the captured upstream request and preserving
no-op/bypass behavior.
- Not tested: independent live inference through an installed routing
extension. The original author-supplied live/replay evidence is
preserved below and remains attributed to the author.
## Runtime Rollout Safety
- Rollout-managed feature(s): no new global rollout switch; the optional
hook runs only when registered.
- Minimum rollout channel: existing extension deployment policy applies.
- Stable/default behavior changed: registered hooks may replace
Responses bodies before proxy processing; deployments without this hook
retain the existing path.
- Kill switch / disable path: disable or unregister the consuming
extension; the Headroom bypass header skips this hook.
- Unsafe override required: no.
- Qualification impact: routing/model/stream changes require
extension-specific runtime qualification in addition to the core tests.
- Rollback path: disable the extension or retain the prior qualified
release.
## Review Readiness
- [ ] I have performed a self-review
- [ ] This PR is ready for human review
Maintainer qualification is recorded above; author attestations have not
been filled on the author's behalf.
## Additional Notes
The original author description and supplied evidence follow, with
headings demoted for the required template structure. The read-only body
view is shallow: hooks must construct replacements rather than mutate
nested request values.
#### Summary
Proxy extensions can now route Codex's `/v1/responses` traffic on the
WebSocket transport as well as over HTTP. This adds one optional
turn-hook method, `on_responses_create(ResponsesCreate) -> dict | None`.
It runs on every HTTP `/v1/responses` body and on every WebSocket
`response.create` frame, at the same point in the pipeline on both. A
hook reads `request.body`, which is a read-only view. To change the
request it returns a replacement body (for example with a new `model` or
`reasoning`); returning `None` leaves the request alone. This mirrors
how `on_response` already works. Chained hooks each see the previous
hook's replacement. `ResponsesCreate` carries `body`, `headers`,
`transport` (`"http"` or `"websocket"`) and `connection_id`.
`connection_id` is the WS session id and is `None` over HTTP. A
`previous_response_id` chain lives within one connection, so a hook that
holds state across a chain's frames should key it on `connection_id`.
#### The test we ran
- **Setup:** a Daytona sandbox running Headroom Enterprise 0.4.3 (core
0.39.1), Codex CLI 0.162.1 with the real GitHub MCP server, and
RouteMeGood in apply mode.
- **HTTP Codex:** 0 RouteMeGood decisions before routemegood#33 and 3
after it, on a probe.
- **Wrapped Codex:** still 0 decisions. `headroom wrap codex` writes
`supports_websockets = true`, so wrapped Codex sends `/v1/responses`
over a WebSocket. That traffic never passes through
`@app.middleware("http")`, so routing silently did nothing.
#### Why a new hook rather than a writable `TurnContext.model`
Turn hooks already run on the WS path. They run inside
`_compress_openai_responses_payload`, and that call site is the wrong
place for a router to change the model:
- It runs only when `config.optimize` is on, and on the executor under
the compression timeout. A timeout or a compression failure would drop
the routing decision.
- Built-in tool-search deferral
(`inject_tool_search_deferral_openai(model)`) has already run against
the *old* model. Routing to a model without tool search would forward a
`tool_search` tool the new model rejects.
- The handler's `model` (tokenizer, pricing, metrics, outcome) is
captured before compression, so a changed `ctx.model` would be priced as
the old model.
`on_responses_create` runs before all of that:
- **HTTP:** right after the inbound debug capture and before `model =
body["model"]`.
- **First WS frame:** before the ChatGPT `store=false` force, memory and
compression.
- **Later WS frames:** before memory and compression.
So everything downstream sees the routed model: compression, deferral,
`store=false`, token counting, pricing, and the WS memory-continuation
template, which is rebuilt from the outbound frame. It is skipped under
`x-headroom-bypass`, and it is inert when no hook defines it.
- **`previous_response_id`:** passed through untouched. OpenAI allows a
model switch mid-chain, and the continuation template already drops
`previous_response_id`.
- **Byte-faithful forwarding (#1542):** the HTTP body is marked mutated
only when a hook returns a replacement. An unchanged request still
forwards the client's original bytes, and nothing has to serialize the
body to find out.
- **A failing hook leaves no trace:** a hook never edits the body in
place, so a hook that raises cannot leave a half-made edit behind.
Without that, pricing would see the edit while the forwarded bytes did
not.
- **`stream` is read after the hook:** a replacement that turns
streaming on takes the streaming dispatch path.
- **Headers:** WS handshake headers are handed over read-only
(`MappingProxyType`).
#### Before / after
| | before | after |
|---|---|---|
| Unit: WS `response.create` frames, hook sets `gpt-5.4-mini` +
`reasoning.effort=low` | upstream frames carry the client's `gpt-5.5` |
both frames carry `gpt-5.4-mini` / `low` (compression off: routing
doesn't depend on it) |
| Unit: HTTP `/v1/responses`, same hook | upstream body +
`RequestOutcome.model` = `gpt-5.5` | upstream body and outcome =
`gpt-5.4-mini` |
| Live, local proxy (steps below) | 0 decisions (2 WS create frames, 0
HTTP POSTs); RouteMeGood logs `this Headroom has no Responses create
hook: Codex over WebSocket passes through unrouted` | 2 decisions, one
per WS create frame |
##### Live check: steps
- **Machine:** macOS (Darwin 25.6), Python 3.12.13.
- **Codex:** codex-cli 0.153.2 with ChatGPT auth.
- **Extension:** RouteMeGood from the stacked PR branch, in shadow mode.
- **Proxy:** the extension's `.venv`, started from this branch:
```sh
HEADROOM_BEACON=off ROUTEMEGOOD_BEACON=off HEADROOM_ROUTEMEGOOD_LICENSE_DEV=1 ROUTEMEGOOD_MODE=shadow ROUTEMEGOOD_HARNESS=codex ROUTEMEGOOD_PERSIST=0 ROUTEMEGOOD_REQUIRE_CREDS=0 PYTHONPATH=<this branch>:<routemegood feat/responses-ws> python -m headroom.cli proxy --port 8799 --proxy-extension routemegood
codex exec --skip-git-repo-check -c model_provider=headroom -c 'model_providers.headroom={name="OpenAI via Headroom proxy",base_url="http://127.0.0.1:8799/v1",supports_websockets=true,requires_openai_auth=true,wire_api="responses"}' "Reply with the single word: pong"
curl -s localhost:8799/routemegood/stats
```
**Observed:**
- The proxy log showed `WS /v1/responses accepted
(route=chatgpt_subscription ...)`, then `completed (... client_frames=2
...)`.
- There were no HTTP `POST /v1/responses`.
- Codex printed `pong`.
- **After:** `/routemegood/stats` returned
`{"decisions":2,"harness":"codex","picked":{"gpt-5.6-terra":2},...}`.
- **Before:** I ran the same steps with `PYTHONPATH` set to a `main`
checkout (976aa714). That gave `{"decisions":0,"note":"no inference
traffic seen yet"}` and the one-line degrade warning.
**Not tested live:** apply mode, i.e. whether the rewritten model
reaches OpenAI's servers. Without a learned predictor RouteMeGood keeps
the caller's model. The unit tests above cover the rewrite. The live run
was in shadow mode, and with no learned predictor RouteMeGood keeps the
caller's model, so it shows decisions being *made* on WS. The rewrite
reaching the upstream is shown by the unit tests. The Daytona sandbox
was not used for this check: another audit is running through its proxy.
#### Tests
`tests/test_responses_create_hooks.py`, all through the handlers:
- WS: the hook's model and effort reach the upstream on the first and
later frames. `previous_response_id` is kept and non-create frames are
untouched.
- WS: the bypass header skips the hook.
- WS: a hook that raises does not break the session.
- HTTP: the upstream body and the recorded outcome both carry the routed
model.
- HTTP: a no-op hook keeps `body_mutated=False`, so the original bytes
are forwarded.
- HTTP: the bypass header skips the hook.
The WS and HTTP routing tests fail on `main`. Touched-area suites pass:
532 tests across turn hooks, Codex WS lifecycle, memory, savings,
compression, routing, byte-faithful forwarding and the extension
ordering suites. ruff check, ruff format and mypy are clean.
The RouteMeGood side: headroomlabs-ai/routemegood PR stacked on #33.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
297 lines
9.5 KiB
Python
297 lines
9.5 KiB
Python
"""``on_responses_create``: one routing seam for OpenAI Responses on both transports.
|
|
|
|
Codex sends ``/v1/responses`` over a WebSocket when wrapped, and an HTTP
|
|
middleware never sees those frames, so a router installed as middleware did
|
|
nothing for wrapped Codex. A turn hook's ``on_responses_create`` runs on the
|
|
HTTP body and on every WebSocket ``response.create`` frame, before the proxy's
|
|
own stages, so a model/effort rewrite reaches the upstream and the accounting.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import sys
|
|
from typing import Any
|
|
from unittest.mock import patch
|
|
|
|
import pytest
|
|
|
|
from headroom.proxy.turn_hooks import (
|
|
ResponsesCreate,
|
|
clear_turn_hooks,
|
|
register_turn_hook,
|
|
)
|
|
from tests.test_openai_codex_ws_lifecycle import (
|
|
_DummyOpenAIHandler,
|
|
_FakeUpstream,
|
|
_FakeWebSocket,
|
|
_make_fake_websockets_module,
|
|
)
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _clean_registry():
|
|
clear_turn_hooks()
|
|
yield
|
|
clear_turn_hooks()
|
|
|
|
|
|
class _Router:
|
|
name = "test-router"
|
|
|
|
def __init__(self) -> None:
|
|
self.seen: list[tuple[str, str, str | None]] = []
|
|
self.connections: list[str | None] = []
|
|
|
|
async def on_responses_create(self, request: ResponsesCreate) -> dict:
|
|
self.seen.append(
|
|
(request.transport, str(request.body.get("model")), request.headers.get("user-agent"))
|
|
)
|
|
self.connections.append(request.connection_id)
|
|
return {**request.body, "model": "gpt-5.4-mini", "reasoning": {"effort": "low"}}
|
|
|
|
|
|
def _create(text: str, **extra: Any) -> str:
|
|
return json.dumps(
|
|
{"type": "response.create", "response": {"model": "gpt-5.5", "input": text, **extra}}
|
|
)
|
|
|
|
|
|
def _upstream() -> _FakeUpstream:
|
|
return _FakeUpstream(
|
|
[
|
|
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
|
|
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
|
|
],
|
|
hold_after_events=True,
|
|
)
|
|
|
|
|
|
async def _run_ws(frames: list[str], headers: dict[str, str] | None = None) -> list[dict]:
|
|
upstream = _upstream()
|
|
client_ws = _FakeWebSocket(frames=frames, headers=headers)
|
|
with patch.dict(sys.modules, {"websockets": _make_fake_websockets_module(upstream)}):
|
|
await _DummyOpenAIHandler().handle_openai_responses_ws(client_ws)
|
|
return [json.loads(frame) for frame in upstream.sent]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_ws_create_frames_carry_the_hooks_model_and_effort_upstream() -> None:
|
|
router = _Router()
|
|
register_turn_hook(router)
|
|
|
|
# previous_response_id chains are left to the client and OpenAI: a model
|
|
# switch mid-chain is allowed, so the hook's rewrite rides alongside it.
|
|
sent = await _run_ws(
|
|
[
|
|
_create("first"),
|
|
json.dumps({"type": "response.cancel"}),
|
|
_create("next", previous_response_id="r_1"),
|
|
]
|
|
)
|
|
|
|
creates = [frame["response"] for frame in sent if frame.get("type") == "response.create"]
|
|
assert [c["model"] for c in creates] == ["gpt-5.4-mini", "gpt-5.4-mini"]
|
|
assert [c["reasoning"] for c in creates] == [{"effort": "low"}] * 2
|
|
assert creates[1]["previous_response_id"] == "r_1"
|
|
assert {"type": "response.cancel"} in sent
|
|
# Compression is off in the dummy handler: routing must not depend on it.
|
|
assert router.seen == [
|
|
("websocket", "gpt-5.5", "codex-cli/0.5"),
|
|
("websocket", "gpt-5.5", "codex-cli/0.5"),
|
|
]
|
|
# One connection, one id: what a hook keys a previous_response_id chain on.
|
|
assert router.connections[0] and len(set(router.connections)) == 1
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_ws_bypass_header_skips_the_hook() -> None:
|
|
router = _Router()
|
|
register_turn_hook(router)
|
|
frames = [_create("first"), _create("next")]
|
|
|
|
sent = await _run_ws(frames, headers={"authorization": "Bearer t", "x-headroom-bypass": "true"})
|
|
|
|
assert router.seen == []
|
|
assert [frame["response"]["model"] for frame in sent] == ["gpt-5.5", "gpt-5.5"]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_hook_that_raises_does_not_break_the_session() -> None:
|
|
class _Broken:
|
|
name = "broken"
|
|
|
|
async def on_responses_create(self, request: ResponsesCreate) -> None:
|
|
# The body is read-only: a hook cannot half-apply an edit and then fail.
|
|
request.body["model"] = "gpt-5.4-mini" # type: ignore[index]
|
|
|
|
register_turn_hook(_Broken())
|
|
frames = [_create("first")]
|
|
|
|
sent = await _run_ws(frames)
|
|
|
|
assert sent == [json.loads(frames[0])]
|
|
|
|
|
|
class _NoOp:
|
|
name = "no-op"
|
|
|
|
async def on_responses_create(self, request: ResponsesCreate) -> None:
|
|
return None
|
|
|
|
|
|
def _post_http(raw: bytes, headers: dict[str, str] | None = None) -> tuple[Any, list, list]:
|
|
"""POST raw bytes to /v1/responses.
|
|
|
|
Returns the response, what reached the upstream call as
|
|
``(body, body_mutated)`` pairs, and the recorded outcomes.
|
|
"""
|
|
pytest.importorskip("fastapi")
|
|
httpx = pytest.importorskip("httpx")
|
|
from fastapi.testclient import TestClient
|
|
|
|
from headroom.proxy.loopback_guard import require_loopback
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
|
|
app = create_app(
|
|
ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
)
|
|
)
|
|
app.dependency_overrides[require_loopback] = lambda: None
|
|
sent: list[Any] = []
|
|
outcomes: list[Any] = []
|
|
|
|
with TestClient(app) as client:
|
|
server = app.state.proxy
|
|
|
|
async def fake_retry(method, url, headers, body, stream=False, **kwargs):
|
|
sent.append((_as_json(body), kwargs.get("body_mutated")))
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "resp_1",
|
|
"object": "response",
|
|
"status": "completed",
|
|
"model": "gpt-5.4-mini",
|
|
"output": [],
|
|
"usage": {"input_tokens": 10, "output_tokens": 2},
|
|
},
|
|
request=httpx.Request("POST", url),
|
|
)
|
|
|
|
record = server._record_request_outcome
|
|
|
|
async def capture(outcome):
|
|
outcomes.append(outcome)
|
|
await record(outcome)
|
|
|
|
server._retry_request = fake_retry
|
|
server._record_request_outcome = capture
|
|
resp = client.post(
|
|
"/v1/responses",
|
|
content=raw,
|
|
headers={
|
|
"Authorization": "Bearer sk-test",
|
|
"Content-Type": "application/json",
|
|
"User-Agent": "codex_exec/0.162.1",
|
|
**(headers or {}),
|
|
},
|
|
)
|
|
return resp, sent, outcomes
|
|
|
|
|
|
def _as_json(body: Any) -> dict:
|
|
return json.loads(body) if isinstance(body, (bytes, str)) else body
|
|
|
|
|
|
_RAW = json.dumps({"model": "gpt-5.5", "input": "hi", "stream": False}).encode()
|
|
|
|
|
|
def test_http_body_carries_the_hooks_model_and_the_outcome_prices_it() -> None:
|
|
router = _Router()
|
|
register_turn_hook(router)
|
|
|
|
resp, sent, outcomes = _post_http(_RAW)
|
|
|
|
assert resp.status_code == 200, resp.text
|
|
assert router.seen == [("http", "gpt-5.5", "codex_exec/0.162.1")]
|
|
assert router.connections == [None]
|
|
body, mutated = sent[0]
|
|
assert body["model"] == "gpt-5.4-mini"
|
|
assert body["reasoning"] == {"effort": "low"}
|
|
assert mutated is True
|
|
assert outcomes and all(o.model == "gpt-5.4-mini" for o in outcomes)
|
|
|
|
|
|
def test_http_hook_that_changes_nothing_keeps_the_clients_bytes() -> None:
|
|
register_turn_hook(_NoOp())
|
|
|
|
resp, sent, _ = _post_http(_RAW)
|
|
|
|
assert resp.status_code == 200, resp.text
|
|
# Unmutated is what sends the client's original bytes (#1542).
|
|
assert sent == [(json.loads(_RAW), False)]
|
|
|
|
|
|
def test_http_bypass_header_skips_the_hook() -> None:
|
|
router = _Router()
|
|
register_turn_hook(router)
|
|
|
|
resp, sent, _ = _post_http(_RAW, headers={"x-headroom-bypass": "true"})
|
|
|
|
assert resp.status_code == 200, resp.text
|
|
assert router.seen == []
|
|
assert sent == [(json.loads(_RAW), False)]
|
|
|
|
|
|
def test_http_hook_that_turns_on_streaming_takes_the_streaming_path() -> None:
|
|
class _Streamer:
|
|
name = "streamer"
|
|
|
|
async def on_responses_create(self, request: ResponsesCreate) -> dict:
|
|
return {**request.body, "stream": True}
|
|
|
|
register_turn_hook(_Streamer())
|
|
pytest.importorskip("fastapi")
|
|
from fastapi.responses import JSONResponse
|
|
from fastapi.testclient import TestClient
|
|
|
|
from headroom.proxy.loopback_guard import require_loopback
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
|
|
app = create_app(
|
|
ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
)
|
|
)
|
|
app.dependency_overrides[require_loopback] = lambda: None
|
|
streamed: list[dict] = []
|
|
|
|
with TestClient(app) as client:
|
|
server = app.state.proxy
|
|
|
|
async def fake_stream(url, headers, body, *args, **kwargs):
|
|
streamed.append(body)
|
|
return JSONResponse({"ok": True})
|
|
|
|
async def no_buffered(*args, **kwargs):
|
|
raise AssertionError("a streaming upstream request took the buffered path")
|
|
|
|
server._stream_response = fake_stream
|
|
server._retry_request = no_buffered
|
|
resp = client.post(
|
|
"/v1/responses",
|
|
content=_RAW,
|
|
headers={"Authorization": "Bearer sk-test", "Content-Type": "application/json"},
|
|
)
|
|
|
|
assert resp.status_code == 200, resp.text
|
|
assert [b["stream"] for b in streamed] == [True]
|