Files
headroom/tests/test_ws_http_fallback.py
46306024e7 fix(openai): read Responses cache_write_tokens instead of inferring writes (#4237)
## Type of Change
- [x] Bug fix (Responses usage accounting)

## Description

OpenAI's Responses usage now reports billed cache writes as
`input_tokens_details.cache_write_tokens`. They are billed at the write
rate: $5/M on `gpt-5.6-sol`, against $4/M input. Every Headroom
Responses path ignored that field and inferred writes as `input −
cached` instead. As a result:
- On HTTP, writes were flagged as inferred and dropped from pricing, so
the written tokens were priced as uncached input at list.
- On the Codex WebSocket path the inferred write was never flagged, so
the uncached slice was billed twice: once as uncached and once as a
write.

Part of the savings redesign (step 2). The counterfactual record (#4236)
needs the true write share `w = W/(U+W)`.

### The test that found it

We ran Headroom Enterprise 0.4.3 in a Daytona sandbox with Codex CLI
0.162.1 on `gpt-5.6-sol` and the real GitHub MCP server, doing workshop
task 1. We audited 13 calls against the provider's billing.
- Every call reported `cache_write_tokens`. For call 2, usage was input
12,497, cached 9,568, write 2,926 (3 uncached).
- PERF printed `cache_write=0` on every line.
- Of the true $0.0850 saved, $0.0497 came from avoided cache **writes**,
which core could not see.
- /stats `total_cost_usd` undercounted the real bill by 8–10%.

## Changes Made

- `_openai_responses_cache_split(input, cached, reported_write)` returns
`(read, write, uncached, inferred)`. A reported write is used as-is; the
write is inferred only when the field is absent.
- Callers updated:
  - `_extract_responses_usage` (WS) returns a 6th `inferred` element.
- The WS per-turn and session-end outcomes now pass `cache_inferred`.
Before this they never set it.
  - The WS→HTTP fallback returns the flag.
- HTTP `/v1/responses` keeps the turn-hook re-drive reads and adds the
re-drives' reported writes (`TurnHookUsage.cache_write_tokens`).
- The passthrough normalizer, the streaming usage parser (mapped onto
the existing real-write branch) and `gateway_turn.normalize_usage` (Kong
relays) also read the field.
- Not changed: Chat Completions, which reports no write counter, and the
`/v1/usage` relay, which doesn't map Responses-shaped details at all.
That was already a gap for reads and is left as a follow-up.

## Before / after (audit replay, 13 Codex calls, current pricers)

| | inferred (main) | reported (this PR) | provider truth |
|---|---|---|---|
| cost-tracker input cost | $0.2329 | $0.2649 | — (audit total $0.305
includes output) |
| SavingsTracker compression $ (novel tokens) | $0.0434 | $0.0542 |
$0.0850 total saved |

The rest of the gap is re-sent history and the per-request record, which
come in the follow-up PRs of the redesign.

## Testing

- [x] New regression tests:
  - `test_ws_session_metrics_use_reported_cache_writes` (WS)
  - `test_fallback_returns_reported_cache_writes_as_real`
- `test_responses_reported_cache_writes_include_the_redrive` (HTTP +
turn hook)
  - `test_normalize_openai_responses_reported_cache_writes` (gateway)
- [x] The existing WS test now pins `cache_inferred is True` when no
write is reported.
- [x] Passed (275, 14 skipped): `test_ws_http_fallback`,
`test_openai_codex_ws_lifecycle`, `test_gateway_turn_unit`,
`test_proxy_budget`, `test_turn_hook_usage`,
`test_proxy_handler_helpers`, `test_proxy_openai_responses_integration`,
`test_proxy_cache_telemetry`, `test_direct_chat_ccr_resolution`,
`test_request_outcome`.
- `test_turn_hook_usage` needs `LITELLM_LOCAL_MODEL_COST_MAP=True`
locally: respx blocks litellm's remote price-map fetch. It fails the
same way on main without it.
- [x] `ruff check` / `ruff format --check` (0.16.8) and `mypy headroom`
are clean.


### Test Output

```text
Maintainer mixed-WebSocket-session probe:
turn 1: input=100 read=75 write=25 uncached=25 cache_inferred=true
turn 2: input=100 read=75 write=20 uncached=5 cache_inferred=false
```

The probe exercised the actual WebSocket handler using the existing
lifecycle test harness. It verifies separate per-turn records for this
mixed session; it does not establish behavior for every suppressed
memory-tool re-drive. The prior qualified head completed hosted CI with
30 successful and 12 skipped checks. The current main-synced head is
being qualified again.

## Real Behavior Proof

- Environment: macOS, this branch; the audit's recorded provider usage
for the 13 Codex calls.
- Exact command / steps: replayed each call's usage through
`CostTracker.record_tokens` and `estimate_request_savings_usd`, first
with the inferred split (main) and then with the reported split (this
PR). Script: `replay_c.py` in the audit scratchpad.
- Observed result: input cost $0.2329 → $0.2649 and compression dollars
$0.0434 → $0.0542, as in the table above. Call 2 now records read 9,568
/ write 2,926 / uncached 3 instead of write 2,929 inferred.
- Not tested: a live Codex session against this branch. The live re-run
comes with the audit-eval PR of the redesign.

## Runtime Rollout Safety
- Rollout-managed feature(s): no new rollout-managed feature switch.
- Minimum rollout channel: existing Responses transport policy applies.
- Unsafe override required: no.
- Qualification impact: reported cache writes affect recorded cost and
savings; hosted checks and human review remain required.

- Stable/default behavior changed: Responses accounting prices
explicitly reported writes as billed writes; inferred writes remain
marked as estimates.
- Yes for OpenAI Responses cost and savings dollars. The newly reported
writes are now priced at the write rate.
- WS turns without a reported write are now flagged inferred. That
removes the double-billed uncached slice, so WS cost totals go down for
older responses.
  - WS turns now count toward `cache_inferred_turns`.
- Kill switch / disable path: none needed; this follows the provider's
own usage.
- Rollback path: revert.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4237"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

## Review Readiness
- [ ] I have performed a self-review
- [ ] This PR is ready for human review

The maintainer has recorded independent qualification above. Author
readiness attestations have not been filled on the author's behalf.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-16 14:32:23 -07:00

489 lines
18 KiB
Python

"""Tests for WebSocket HTTP fallback in the OpenAI handler.
When the upstream WebSocket connection to OpenAI fails (HTTP 500),
the proxy should transparently fall back to HTTP POST streaming
and relay SSE events over the client WebSocket.
"""
from __future__ import annotations
import asyncio
import json
from types import SimpleNamespace
import httpx
class FakeWebSocket:
"""Minimal WebSocket mock for testing."""
def __init__(self):
self.sent_texts: list[str] = []
self.closed = False
async def send_text(self, data: str) -> None:
self.sent_texts.append(data)
async def close(self, code: int = 1000, reason: str = "") -> None:
self.closed = True
class FakeStreamResponse:
"""Mock httpx streaming response."""
def __init__(
self,
status_code: int = 200,
sse_events: list[str] | None = None,
headers: dict[str, str] | None = None,
):
self.status_code = status_code
self._events = sse_events or []
self.headers = headers or {}
async def aiter_text(self):
for event in self._events:
yield event
async def aiter_bytes(self):
yield b"error body"
async def __aenter__(self):
return self
async def __aexit__(self, *args):
pass
class FakeHttpClient:
"""Mock httpx.AsyncClient with stream support."""
def __init__(self, response: FakeStreamResponse):
self._response = response
def stream(self, method, url, **kwargs):
return self._response
def _make_handler():
"""Create a minimal OpenAIHandlerMixin-like object."""
from headroom.proxy.handlers.openai import OpenAIHandlerMixin
obj = object.__new__(OpenAIHandlerMixin)
obj.OPENAI_API_URL = "https://api.openai.com"
obj.http_client = None
# Deliberately odd values: a test that asserts the real defaults would pass
# even if the timeout stopped being read from config at all.
obj.config = SimpleNamespace(
retry_max_attempts=3,
retry_base_delay_ms=0,
retry_max_delay_ms=0,
connect_timeout_seconds=7,
write_timeout_seconds=33,
request_timeout_seconds=999,
)
return obj
class TestWsHttpFallbackTimeout:
"""The fallback POST must honor the configured timeouts.
It passed a bare ``timeout=120.0``, and httpx expands a float across all
four phases -- so connect and pool silently got 120s instead of the
configured value, and the send ignored ``write_timeout_seconds``. Only the
SSE read bound was ever meant to be 120s here.
"""
def test_connect_and_pool_come_from_config(self):
handler = _make_handler()
timeout = handler._ws_http_fallback_timeout()
assert timeout.connect == 7
assert timeout.pool == 7
def test_write_comes_from_config(self):
handler = _make_handler()
timeout = handler._ws_http_fallback_timeout()
assert timeout.write == 33
def test_the_read_bound_is_not_smeared_across_the_other_phases(self):
"""The regression itself: one float became all four phases."""
handler = _make_handler()
timeout = handler._ws_http_fallback_timeout()
assert timeout.read == 120.0
assert timeout.connect != timeout.read
assert timeout.write != timeout.read
assert timeout.pool != timeout.read
def test_read_stays_the_paths_own_sse_stall_bound(self):
"""`read` is the gap between SSE events, so it is not the generic cap."""
handler = _make_handler()
timeout = handler._ws_http_fallback_timeout()
assert timeout.read == 120.0
assert timeout.read != handler.config.request_timeout_seconds
def test_the_fallback_request_actually_uses_it(self):
"""Pin the wiring, not just the helper."""
handler = _make_handler()
ws = FakeWebSocket()
captured_kwargs = {}
class CapturingClient:
def stream(self, method, url, **kwargs):
captured_kwargs.update(kwargs)
return FakeStreamResponse(200, ["data: [DONE]\n\n"])
handler.http_client = CapturingClient()
body = {"model": "gpt-5.4", "input": "test"}
asyncio.run(handler._ws_http_fallback(ws, body, json.dumps(body), {}, "req_timeout"))
timeout = captured_kwargs["timeout"]
assert isinstance(timeout, httpx.Timeout)
assert timeout.connect == 7
assert timeout.write == 33
assert timeout.read == 120.0
class TestWsHttpFallback:
def test_fallback_relays_sse_events(self):
"""HTTP fallback should relay SSE data lines as WS text messages."""
handler = _make_handler()
ws = FakeWebSocket()
sse_lines = [
'event: response.created\ndata: {"type":"response.created","response":{"id":"r1"}}\n\n',
'event: response.output_item.added\ndata: {"type":"response.output_item.added"}\n\n',
'event: response.completed\ndata: {"type":"response.completed"}\n\n',
"data: [DONE]\n\n",
]
response = FakeStreamResponse(200, sse_lines)
handler.http_client = FakeHttpClient(response)
body = {"model": "gpt-5.4", "input": "hi"}
first_msg_raw = json.dumps({"type": "response.create", "response": body})
asyncio.run(
handler._ws_http_fallback(
ws, body, first_msg_raw, {"Authorization": "Bearer test"}, "req_1"
)
)
assert len(ws.sent_texts) == 3 # 3 data events, [DONE] skipped
assert '"response.created"' in ws.sent_texts[0]
assert '"response.output_item.added"' in ws.sent_texts[1]
assert '"response.completed"' in ws.sent_texts[2]
assert ws.closed
def test_fallback_sends_error_on_non_200(self):
"""HTTP fallback should send error event on non-200 response."""
handler = _make_handler()
ws = FakeWebSocket()
response = FakeStreamResponse(status_code=401)
handler.http_client = FakeHttpClient(response)
body = {"model": "gpt-5.4", "input": "hi"}
asyncio.run(
handler._ws_http_fallback(
ws, body, json.dumps(body), {"Authorization": "Bearer bad"}, "req_2"
)
)
assert len(ws.sent_texts) == 1
event = json.loads(ws.sent_texts[0])
assert event["type"] == "error"
assert "401" in event["error"]["message"]
def test_fallback_sets_stream_true(self):
"""HTTP fallback should force stream=True in request body.
After PR-A3 (byte-faithful Python forwarders) the fallback sends
the request body as raw bytes via `content=`, not via the `json=`
kwarg. The test extracts the posted JSON from the captured bytes.
"""
handler = _make_handler()
ws = FakeWebSocket()
captured_kwargs: dict = {}
class CapturingClient:
def stream(self, method, url, **kwargs):
captured_kwargs.update(kwargs)
return FakeStreamResponse(200, ["data: [DONE]\n\n"])
handler.http_client = CapturingClient()
body = {"model": "gpt-5.4", "input": "test", "stream": False}
asyncio.run(handler._ws_http_fallback(ws, body, json.dumps(body), {}, "req_3"))
posted = json.loads(captured_kwargs["content"])
assert posted["stream"] is True
def test_fallback_unwraps_response_create_envelope(self):
"""HTTP fallback should unwrap WS response.create wrapper for HTTP POST."""
handler = _make_handler()
ws = FakeWebSocket()
captured_kwargs: dict = {}
class CapturingClient:
def stream(self, method, url, **kwargs):
captured_kwargs.update(kwargs)
return FakeStreamResponse(200, ["data: [DONE]\n\n"])
handler.http_client = CapturingClient()
# WS sends wrapped format: {"type": "response.create", "response": {...}}
inner = {
"model": "gpt-5.4",
"input": [{"role": "user", "content": [{"type": "input_text", "text": "hi"}]}],
}
ws_msg = {"type": "response.create", "response": inner}
asyncio.run(handler._ws_http_fallback(ws, ws_msg, json.dumps(ws_msg), {}, "req_unwrap"))
posted = json.loads(captured_kwargs["content"])
# Should be the inner response, not the wrapper
assert "type" not in posted # no "response.create" type field
assert posted["model"] == "gpt-5.4"
assert posted["stream"] is True
assert "input" in posted
def test_fallback_strips_top_level_response_create_type(self):
"""HTTP fallback should strip top-level response.create metadata."""
handler = _make_handler()
ws = FakeWebSocket()
captured_kwargs: dict = {}
class CapturingClient:
def stream(self, method, url, **kwargs):
captured_kwargs.update(kwargs)
return FakeStreamResponse(200, ["data: [DONE]\n\n"])
handler.http_client = CapturingClient()
body = {"type": "response.create", "model": "gpt-5.4", "input": "hi"}
asyncio.run(handler._ws_http_fallback(ws, body, json.dumps(body), {}, "req_type_strip"))
posted = json.loads(captured_kwargs["content"])
assert posted["model"] == "gpt-5.4"
assert posted["stream"] is True
assert "type" not in posted
def test_fallback_handles_http_exception(self):
"""HTTP fallback should send error event when HTTP request fails."""
handler = _make_handler()
ws = FakeWebSocket()
class FailingClient:
def stream(self, method, url, **kwargs):
raise ConnectionError("upstream unreachable")
handler.http_client = FailingClient()
body = {"model": "gpt-5.4", "input": "test"}
asyncio.run(handler._ws_http_fallback(ws, body, json.dumps(body), {}, "req_4"))
assert len(ws.sent_texts) == 1
event = json.loads(ws.sent_texts[0])
assert event["type"] == "error"
assert "unreachable" in event["error"]["message"]
def test_fallback_retries_connect_timeout(self):
"""HTTP fallback should retry transient connect timeouts."""
handler = _make_handler()
ws = FakeWebSocket()
attempts = {"count": 0}
class FlakyClient:
def stream(self, method, url, **kwargs):
attempts["count"] += 1
if attempts["count"] == 1:
raise httpx.ConnectTimeout("timed out")
return FakeStreamResponse(200, ['data: {"type":"response.completed"}\n\n'])
handler.http_client = FlakyClient()
body = {"model": "gpt-5.4", "input": "test"}
asyncio.run(handler._ws_http_fallback(ws, body, json.dumps(body), {}, "req_retry"))
assert attempts["count"] == 2
assert len(ws.sent_texts) == 1
assert json.loads(ws.sent_texts[0])["type"] == "response.completed"
def test_fallback_routes_chatgpt_auth_to_chatgpt_domain(self):
"""ChatGPT session auth should route to chatgpt.com, not api.openai.com."""
handler = _make_handler()
ws = FakeWebSocket()
captured_url = {}
class CapturingClient:
def stream(self, method, url, **kwargs):
captured_url["url"] = url
return FakeStreamResponse(200, ["data: [DONE]\n\n"])
handler.http_client = CapturingClient()
body = {"model": "gpt-5.4", "input": "test"}
# ChatGPT session auth includes this header
headers = {
"Authorization": "Bearer chatgpt-session-token",
"ChatGPT-Account-ID": "acct_abc123",
}
asyncio.run(handler._ws_http_fallback(ws, body, json.dumps(body), headers, "req_5"))
assert "chatgpt.com" in captured_url["url"]
assert "api.openai.com" not in captured_url["url"]
def test_fallback_chatgpt_auth_forces_store_false(self):
"""ChatGPT Responses backend requires explicit store=false."""
handler = _make_handler()
ws = FakeWebSocket()
captured_kwargs: dict = {}
class CapturingClient:
def stream(self, method, url, **kwargs):
captured_kwargs.update(kwargs)
return FakeStreamResponse(200, ["data: [DONE]\n\n"])
handler.http_client = CapturingClient()
body = {"model": "gpt-5.4", "input": "test", "store": True}
headers = {
"Authorization": "Bearer chatgpt-session-token",
"ChatGPT-Account-ID": "acct_abc123",
}
asyncio.run(handler._ws_http_fallback(ws, body, json.dumps(body), headers, "req_store"))
posted = json.loads(captured_kwargs["content"])
assert posted["store"] is False
assert posted["stream"] is True
def test_fallback_routes_api_key_to_openai(self):
"""API key auth should route to api.openai.com."""
handler = _make_handler()
ws = FakeWebSocket()
captured_url = {}
class CapturingClient:
def stream(self, method, url, **kwargs):
captured_url["url"] = url
return FakeStreamResponse(200, ["data: [DONE]\n\n"])
handler.http_client = CapturingClient()
body = {"model": "gpt-5.4", "input": "test"}
# API key auth — no ChatGPT-Account-ID header
headers = {"Authorization": "Bearer sk-abc123"}
asyncio.run(handler._ws_http_fallback(ws, body, json.dumps(body), headers, "req_6"))
assert "api.openai.com" in captured_url["url"]
def test_fallback_returns_provider_usage_from_completed_event(self):
"""The fallback must surface the provider's input usage (#2957).
Otherwise the WS session-end outcome records input_tokens=0 for a large
request and savings percentages blow past 100.
"""
handler = _make_handler()
ws = FakeWebSocket()
completed = {
"type": "response.completed",
"response": {
"usage": {
"input_tokens": 31055,
"output_tokens": 246,
"input_tokens_details": {"cached_tokens": 20000},
}
},
}
sse_lines = [
'data: {"type":"response.created","response":{"id":"r1"}}\n\n',
f"data: {json.dumps(completed)}\n\n",
"data: [DONE]\n\n",
]
handler.http_client = FakeHttpClient(FakeStreamResponse(200, sse_lines))
body = {"model": "gpt-5.4", "input": "big context"}
usage = asyncio.run(handler._ws_http_fallback(ws, body, json.dumps(body), {}, "req_usage"))
input_tokens, output_tokens, cache_read, _cache_write, uncached, inferred = usage
assert inferred is True # no cache_write_tokens on this frame
assert input_tokens == 31055
assert output_tokens == 246
assert cache_read == 20000
assert uncached == 31055 - 20000
def test_fallback_returns_zero_usage_without_completed_event(self):
handler = _make_handler()
ws = FakeWebSocket()
handler.http_client = FakeHttpClient(
FakeStreamResponse(200, ['data: {"type":"response.created"}\n\n', "data: [DONE]\n\n"])
)
usage = asyncio.run(
handler._ws_http_fallback(
ws, {"model": "gpt-5.4", "input": "hi"}, json.dumps({"input": "hi"}), {}, "req_none"
)
)
assert usage == (0, 0, 0, 0, 0, True)
def test_fallback_returns_reported_cache_writes_as_real(self):
"""A reported write must not come back flagged inferred: pricing drops
inferred writes, so the written tokens would be billed at nothing."""
handler = _make_handler()
completed = {
"type": "response.completed",
"response": {
"usage": {
"input_tokens": 100,
"output_tokens": 5,
"input_tokens_details": {"cached_tokens": 75, "cache_write_tokens": 20},
}
},
}
handler.http_client = FakeHttpClient(
FakeStreamResponse(200, [f"data: {json.dumps(completed)}\n\n", "data: [DONE]\n\n"])
)
body = {"model": "gpt-5.4", "input": "hi"}
usage = asyncio.run(
handler._ws_http_fallback(FakeWebSocket(), body, json.dumps(body), {}, "req_w")
)
assert usage == (100, 5, 75, 20, 5, False)
def test_fallback_refreshes_codex_rate_limit_state(self, monkeypatch):
"""A successful fallback refreshes Codex /stats from response headers.
The fallback can't forward headers onto the (already-accepted) client
101, but it should still keep Python /stats in sync so the gauge does
not go stale when the WS upgrade fails and we drop to HTTP.
"""
handler = _make_handler()
ws = FakeWebSocket()
captured: dict[str, dict[str, str]] = {}
class _FakeState:
def update_from_headers(self, hdrs):
captured["headers"] = dict(hdrs)
import headroom.subscription.codex_rate_limits as crl
monkeypatch.setattr(crl, "get_codex_rate_limit_state", lambda: _FakeState())
response = FakeStreamResponse(
200,
['data: {"type":"response.completed"}\n\n', "data: [DONE]\n\n"],
headers={
"x-codex-primary-used-percent": "42",
"content-type": "text/event-stream",
},
)
handler.http_client = FakeHttpClient(response)
body = {"model": "gpt-5.4", "input": "hi"}
asyncio.run(handler._ws_http_fallback(ws, body, json.dumps(body), {}, "req_capture"))
assert captured["headers"]["x-codex-primary-used-percent"] == "42"