fix(openai): read Responses cache_write_tokens instead of inferring writes (#4237)

## Type of Change
- [x] Bug fix (Responses usage accounting)

## Description

OpenAI's Responses usage now reports billed cache writes as
`input_tokens_details.cache_write_tokens`. They are billed at the write
rate: $5/M on `gpt-5.6-sol`, against $4/M input. Every Headroom
Responses path ignored that field and inferred writes as `input −
cached` instead. As a result:
- On HTTP, writes were flagged as inferred and dropped from pricing, so
the written tokens were priced as uncached input at list.
- On the Codex WebSocket path the inferred write was never flagged, so
the uncached slice was billed twice: once as uncached and once as a
write.

Part of the savings redesign (step 2). The counterfactual record (#4236)
needs the true write share `w = W/(U+W)`.

### The test that found it

We ran Headroom Enterprise 0.4.3 in a Daytona sandbox with Codex CLI
0.162.1 on `gpt-5.6-sol` and the real GitHub MCP server, doing workshop
task 1. We audited 13 calls against the provider's billing.
- Every call reported `cache_write_tokens`. For call 2, usage was input
12,497, cached 9,568, write 2,926 (3 uncached).
- PERF printed `cache_write=0` on every line.
- Of the true $0.0850 saved, $0.0497 came from avoided cache **writes**,
which core could not see.
- /stats `total_cost_usd` undercounted the real bill by 8–10%.

## Changes Made

- `_openai_responses_cache_split(input, cached, reported_write)` returns
`(read, write, uncached, inferred)`. A reported write is used as-is; the
write is inferred only when the field is absent.
- Callers updated:
  - `_extract_responses_usage` (WS) returns a 6th `inferred` element.
- The WS per-turn and session-end outcomes now pass `cache_inferred`.
Before this they never set it.
  - The WS→HTTP fallback returns the flag.
- HTTP `/v1/responses` keeps the turn-hook re-drive reads and adds the
re-drives' reported writes (`TurnHookUsage.cache_write_tokens`).
- The passthrough normalizer, the streaming usage parser (mapped onto
the existing real-write branch) and `gateway_turn.normalize_usage` (Kong
relays) also read the field.
- Not changed: Chat Completions, which reports no write counter, and the
`/v1/usage` relay, which doesn't map Responses-shaped details at all.
That was already a gap for reads and is left as a follow-up.

## Before / after (audit replay, 13 Codex calls, current pricers)

| | inferred (main) | reported (this PR) | provider truth |
|---|---|---|---|
| cost-tracker input cost | $0.2329 | $0.2649 | — (audit total $0.305
includes output) |
| SavingsTracker compression $ (novel tokens) | $0.0434 | $0.0542 |
$0.0850 total saved |

The rest of the gap is re-sent history and the per-request record, which
come in the follow-up PRs of the redesign.

## Testing

- [x] New regression tests:
  - `test_ws_session_metrics_use_reported_cache_writes` (WS)
  - `test_fallback_returns_reported_cache_writes_as_real`
- `test_responses_reported_cache_writes_include_the_redrive` (HTTP +
turn hook)
  - `test_normalize_openai_responses_reported_cache_writes` (gateway)
- [x] The existing WS test now pins `cache_inferred is True` when no
write is reported.
- [x] Passed (275, 14 skipped): `test_ws_http_fallback`,
`test_openai_codex_ws_lifecycle`, `test_gateway_turn_unit`,
`test_proxy_budget`, `test_turn_hook_usage`,
`test_proxy_handler_helpers`, `test_proxy_openai_responses_integration`,
`test_proxy_cache_telemetry`, `test_direct_chat_ccr_resolution`,
`test_request_outcome`.
- `test_turn_hook_usage` needs `LITELLM_LOCAL_MODEL_COST_MAP=True`
locally: respx blocks litellm's remote price-map fetch. It fails the
same way on main without it.
- [x] `ruff check` / `ruff format --check` (0.16.8) and `mypy headroom`
are clean.


### Test Output

```text
Maintainer mixed-WebSocket-session probe:
turn 1: input=100 read=75 write=25 uncached=25 cache_inferred=true
turn 2: input=100 read=75 write=20 uncached=5 cache_inferred=false
```

The probe exercised the actual WebSocket handler using the existing
lifecycle test harness. It verifies separate per-turn records for this
mixed session; it does not establish behavior for every suppressed
memory-tool re-drive. The prior qualified head completed hosted CI with
30 successful and 12 skipped checks. The current main-synced head is
being qualified again.

## Real Behavior Proof

- Environment: macOS, this branch; the audit's recorded provider usage
for the 13 Codex calls.
- Exact command / steps: replayed each call's usage through
`CostTracker.record_tokens` and `estimate_request_savings_usd`, first
with the inferred split (main) and then with the reported split (this
PR). Script: `replay_c.py` in the audit scratchpad.
- Observed result: input cost $0.2329 → $0.2649 and compression dollars
$0.0434 → $0.0542, as in the table above. Call 2 now records read 9,568
/ write 2,926 / uncached 3 instead of write 2,929 inferred.
- Not tested: a live Codex session against this branch. The live re-run
comes with the audit-eval PR of the redesign.

## Runtime Rollout Safety
- Rollout-managed feature(s): no new rollout-managed feature switch.
- Minimum rollout channel: existing Responses transport policy applies.
- Unsafe override required: no.
- Qualification impact: reported cache writes affect recorded cost and
savings; hosted checks and human review remain required.

- Stable/default behavior changed: Responses accounting prices
explicitly reported writes as billed writes; inferred writes remain
marked as estimates.
- Yes for OpenAI Responses cost and savings dollars. The newly reported
writes are now priced at the write rate.
- WS turns without a reported write are now flagged inferred. That
removes the double-billed uncached slice, so WS cost totals go down for
older responses.
  - WS turns now count toward `cache_inferred_turns`.
- Kill switch / disable path: none needed; this follows the provider's
own usage.
- Rollback path: revert.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4237"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

## Review Readiness
- [ ] I have performed a self-review
- [ ] This PR is ready for human review

The maintainer has recorded independent qualification above. Author
readiness attestations have not been filled on the author's behalf.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Tejas Chopra
2026-10-16 14:32:23 -07:00
committed by GitHub
co-authored by Claude Opus 5.5 JD Davis
parent 1a40b3ce67
commit 46306024e7
9 changed files with 317 additions and 29 deletions
+12 -4
View File
@@ -342,7 +342,9 @@ def normalize_usage(usage: dict[str, Any]) -> NormalizedUsage:
# ``input_tokens`` too, so the key name cannot be the discriminator. Keying
# off Anthropic's cache fields can misread only a relay that mixes the two
# vocabularies, and then it errs towards not inflating the prompt.
input_includes_cache = not (
# Responses' ``input_tokens_details`` marks an inclusive total even when a
# relay also flattened a top-level read beside it.
input_includes_cache = "input_tokens_details" in usage or not (
"cache_read_input_tokens" in usage or "cache_creation_input_tokens" in usage
)
output_tokens = _usage_int(usage, "output_tokens")
@@ -351,10 +353,16 @@ def normalize_usage(usage: dict[str, Any]) -> NormalizedUsage:
cache_read = _usage_int(usage, "cache_read_input_tokens")
cache_write = _usage_int(usage, "cache_creation_input_tokens")
if cache_write is None:
# Responses reports billed writes as input_tokens_details.cache_write_tokens;
# chat reports none. Read whether or not a top-level read was relayed.
details = usage.get("input_tokens_details")
if isinstance(details, dict):
cache_write = _usage_int(details, "cache_write_tokens")
if cache_read is None:
# OpenAI reports reads only (no write counter exists), nested under
# prompt_tokens_details (chat), input_tokens_details (Responses) or —
# in Kong's flattened log shape — at the top level.
# OpenAI nests reads under prompt_tokens_details (chat),
# input_tokens_details (Responses) or — in Kong's flattened log shape —
# puts them at the top level.
for details_key in ("prompt_tokens_details", "input_tokens_details"):
details = usage.get(details_key)
if isinstance(details, dict):
+105 -20
View File
@@ -351,13 +351,22 @@ class TurnHookUsage:
the safe direction for a bill.
"""
__slots__ = ("_seen", "input_tokens", "output_tokens", "cache_read_tokens", "extra_calls")
__slots__ = (
"_seen",
"input_tokens",
"output_tokens",
"cache_read_tokens",
"cache_write_tokens",
"extra_calls",
)
def __init__(self) -> None:
self._seen: list[tuple[int, Any, int, int, int]] = []
self._seen: list[tuple[int, Any, int, int, int, int]] = []
self.input_tokens = 0
self.output_tokens = 0
self.cache_read_tokens = 0
#: Reported (Responses ``cache_write_tokens``) writes only; 0 otherwise.
self.cache_write_tokens = 0
self.extra_calls = 0
def record(
@@ -381,17 +390,19 @@ class TurnHookUsage:
usage_int(usage.get(input_key)),
usage_int(usage.get(output_key)),
cached,
_reported_cache_write(details) or 0,
)
else:
entry = (id(payload), payload, 0, 0, 0)
entry = (id(payload), payload, 0, 0, 0, 0)
self._seen.append(entry)
def settle(self, final: Any) -> None:
"""Total everything except ``final``, which the usage block will read."""
self.input_tokens = self.output_tokens = self.cache_read_tokens = 0
self.cache_write_tokens = 0
self.extra_calls = 0
dropped = False
for _ident, payload, tin, tout, cached in self._seen:
for _ident, payload, tin, tout, cached, written in self._seen:
if not dropped and payload is final:
dropped = True
continue
@@ -399,6 +410,7 @@ class TurnHookUsage:
self.input_tokens += tin
self.output_tokens += tout
self.cache_read_tokens += cached
self.cache_write_tokens += written
def _sanitize_forwarded_response_headers(
@@ -614,7 +626,13 @@ def _passthrough_usage_from_json(payload: Any, provider: str | None = None) -> d
"input_tokens": billed_input_from_usage(usage, provider) or usage_int(input_tokens),
"output_tokens": usage_int(output_tokens),
"cache_read_input_tokens": usage_int(usage.get("cache_read_input_tokens", cache_read)),
"cache_creation_input_tokens": usage_int(usage.get("cache_creation_input_tokens")),
# Responses spells a billed write ``input_tokens_details.cache_write_tokens``.
"cache_creation_input_tokens": usage_int(
usage.get(
"cache_creation_input_tokens",
details.get("cache_write_tokens") if isinstance(details, dict) else None,
)
),
}
return {}
@@ -1650,6 +1668,48 @@ def _infer_openai_cache_write_tokens(input_tokens: int, cache_read_tokens: int)
return max(input_tokens - cache_read_tokens, 0)
def _reported_cache_write(details: Any) -> int | None:
"""``input_tokens_details.cache_write_tokens``, or ``None`` when not reported."""
if not isinstance(details, dict) or details.get("cache_write_tokens") is None:
return None
return usage_int(details.get("cache_write_tokens"))
def _hook_reported_write(final_write: int | None, hook_usage: Any) -> int | None:
"""The final response's reported write plus any turn-hook re-drives' writes.
A final response without the field still carries the re-drives' reported
writes: treating its own write as 0 prices the same as inferring it, since
an inferred write is dropped from pricing anyway.
"""
hook_writes = hook_usage.cache_write_tokens if hook_usage.extra_calls else 0
if final_write is None:
return hook_writes or None
return final_write + hook_writes
def _openai_responses_cache_split(
input_tokens: int, cached: int, reported_write: int | None
) -> tuple[int, int, int, bool]:
"""Split a Responses ``input_tokens`` total into ``(read, write, uncached, inferred)``.
Responses usage now reports cache writes itself as
``input_tokens_details.cache_write_tokens``, billed at the write rate
(above list input on current models). When present it is the truth and the
three buckets are disjoint. Older responses omit it; the write is then
inferred from the uncached slice and flagged, so pricing drops it rather
than billing the same tokens twice.
"""
if reported_write is not None:
return cached, reported_write, max(input_tokens - cached - reported_write, 0), False
return (
cached,
_infer_openai_cache_write_tokens(input_tokens, cached),
max(input_tokens - cached, 0),
True,
)
def _deferrable_savings_delta(input_delta: int, saved_delta: int) -> int:
"""Gate a WS Responses turn's compression-savings delta on input accounting.
@@ -1670,32 +1730,36 @@ def _deferrable_savings_delta(input_delta: int, saved_delta: int) -> int:
return saved_delta
def _extract_responses_usage(event: dict[str, Any]) -> tuple[int, int, int, int, int]:
def _extract_responses_usage(event: dict[str, Any]) -> tuple[int, int, int, int, int, bool]:
"""Return input/output/cache usage from a Responses event.
Codex WebSocket streams include usage on ``response.completed`` events.
The shape mirrors HTTP Responses usage:
``response.usage.input_tokens`` plus
``response.usage.input_tokens_details.cached_tokens``.
``response.usage.input_tokens_details.cached_tokens`` / ``cache_write_tokens``.
The last element says whether the write was inferred (see
:func:`_openai_responses_cache_split`).
"""
if event.get("type") != "response.completed":
return 0, 0, 0, 0, 0
return 0, 0, 0, 0, 0, True
response = event.get("response")
if not isinstance(response, dict):
response = {}
usage = response.get("usage") or event.get("usage")
if not isinstance(usage, dict):
return 0, 0, 0, 0, 0
return 0, 0, 0, 0, 0, True
input_tokens = usage_int(usage.get("input_tokens"))
output_tokens = usage_int(usage.get("output_tokens"))
details = usage.get("input_tokens_details")
cached_tokens = usage_int(details.get("cached_tokens")) if isinstance(details, dict) else 0
cache_write_tokens = _infer_openai_cache_write_tokens(input_tokens, cached_tokens)
uncached_tokens = max(input_tokens - cached_tokens, 0)
return input_tokens, output_tokens, cached_tokens, cache_write_tokens, uncached_tokens
cached, write, uncached, inferred = _openai_responses_cache_split(
input_tokens,
usage_int(details.get("cached_tokens")) if isinstance(details, dict) else 0,
_reported_cache_write(details),
)
return input_tokens, output_tokens, cached, write, uncached, inferred
def _prefers_http1_passthrough(base_url: str) -> bool:
@@ -6538,6 +6602,7 @@ class OpenAIHandlerMixin:
total_input_tokens = original_tokens # fallback
output_tokens = 0
cache_read_tokens = 0
details: Any = None
resp_json = None
try:
resp_json = response.json()
@@ -6757,11 +6822,17 @@ class OpenAIHandlerMixin:
# Cost is recorded once by the outcome funnel below; here we only
# compute the cache-write / uncached split the funnel needs.
# (Recording here too double-counted spend + budget on this path.)
cache_write_tokens = _infer_openai_cache_write_tokens(
# Reads include the hook's re-drives added above.
(
cache_read_tokens,
cache_write_tokens,
uncached_input_tokens,
cache_inferred,
) = _openai_responses_cache_split(
total_input_tokens,
cache_read_tokens,
_hook_reported_write(_reported_cache_write(details), _resp_hook_usage),
)
uncached_input_tokens = max(0, total_input_tokens - cache_read_tokens)
# Was: optimized := provider count, then original := max(original,
# optimized + saved) to stop `attempted` exceeding `original`.
@@ -6803,7 +6874,7 @@ class OpenAIHandlerMixin:
cache_read_tokens=cache_read_tokens,
cache_write_tokens=cache_write_tokens,
uncached_input_tokens=uncached_input_tokens,
cache_inferred=True,
cache_inferred=cache_inferred,
total_latency_ms=total_latency,
overhead_ms=optimization_latency,
transforms_applied=tuple(transforms_applied),
@@ -7564,6 +7635,11 @@ class OpenAIHandlerMixin:
ws_cache_read_tokens_total = 0
ws_cache_write_tokens_total = 0
ws_uncached_input_tokens_total = 0
# Whether the latest usage frame's write was inferred rather than
# reported (see _openai_responses_cache_split). One flag for the
# summed deltas: a session that mixed reported and unreported
# frames would misprice one kind, which Codex does not do.
ws_cache_inferred = True
ws_recorded_input_tokens_total = 0
ws_recorded_output_tokens_total = 0
ws_recorded_cache_read_tokens_total = 0
@@ -8890,7 +8966,7 @@ class OpenAIHandlerMixin:
nonlocal ws_memory_response_pending
nonlocal ws_input_tokens_total, ws_output_tokens_total
nonlocal ws_cache_read_tokens_total, ws_cache_write_tokens_total
nonlocal ws_uncached_input_tokens_total
nonlocal ws_uncached_input_tokens_total, ws_cache_inferred
nonlocal ws_recorded_input_tokens_total
nonlocal ws_recorded_output_tokens_total
nonlocal ws_recorded_cache_read_tokens_total
@@ -9020,6 +9096,7 @@ class OpenAIHandlerMixin:
cache_read_tokens=max(0, cache_read_delta),
cache_write_tokens=max(0, cache_write_delta),
uncached_input_tokens=max(0, uncached_delta),
cache_inferred=ws_cache_inferred,
total_latency_ms=latency_ms,
overhead_ms=overhead_delta_ms,
ttfb_ms=ttfb_for_record_ms,
@@ -9164,8 +9241,10 @@ class OpenAIHandlerMixin:
usage_cache_read_tokens,
usage_cache_write_tokens,
usage_uncached_tokens,
usage_cache_inferred,
) = _extract_responses_usage(event)
if usage_input_tokens or usage_output_tokens:
ws_cache_inferred = usage_cache_inferred
ws_input_tokens_total += usage_input_tokens
ws_output_tokens_total += usage_output_tokens
ws_cache_read_tokens_total += usage_cache_read_tokens
@@ -9559,6 +9638,7 @@ class OpenAIHandlerMixin:
fb_cache_read_tokens,
fb_cache_write_tokens,
fb_uncached_tokens,
ws_cache_inferred,
) = await self._ws_http_fallback(
websocket, body, first_msg_raw, upstream_headers, request_id
)
@@ -9690,6 +9770,7 @@ class OpenAIHandlerMixin:
cache_read_tokens=residual_cache_read_tokens,
cache_write_tokens=residual_cache_write_tokens,
uncached_input_tokens=residual_uncached_input_tokens,
cache_inferred=ws_cache_inferred,
total_latency_ms=ws_session_duration_ms,
overhead_ms=final_overhead_delta_ms,
ttfb_ms=final_ttfb_ms,
@@ -9836,7 +9917,7 @@ class OpenAIHandlerMixin:
first_msg_raw: str,
upstream_headers: dict[str, str],
request_id: str,
) -> tuple[int, int, int, int, int]:
) -> tuple[int, int, int, int, int, bool]:
"""Fall back to HTTP POST streaming when upstream WS fails.
Converts the WS ``response.create`` message to an HTTP POST to
@@ -9850,7 +9931,9 @@ class OpenAIHandlerMixin:
authoritative wire-token count; otherwise a fallback recorded
``input_tokens=0`` and savings percentages blew past 100 (#2957).
"""
fallback_usage = [0, 0, 0, 0, 0]
# Five summed counters, then whether the latest frame's write was
# inferred (see _openai_responses_cache_split).
fallback_usage: list[int] = [0, 0, 0, 0, 0, True]
def _accumulate_usage(data_str: str) -> None:
try:
@@ -9858,8 +9941,10 @@ class OpenAIHandlerMixin:
except (json.JSONDecodeError, TypeError):
return
if isinstance(event, dict) and event.get("type") == "response.completed":
for i, value in enumerate(_extract_responses_usage(event)):
usage = _extract_responses_usage(event)
for i, value in enumerate(usage[:5]):
fallback_usage[i] += value
fallback_usage[5] = usage[5]
# Route to correct endpoint based on auth mode
is_chatgpt_fallback = has_chatgpt_account_header(upstream_headers)
+12 -2
View File
@@ -421,6 +421,13 @@ class StreamingMixin:
usage_found["cache_read_input_tokens"] = usage_int(
details.get("cached_tokens")
)
# Responses reports billed cache writes itself; the
# Anthropic-shape key routes them down the real-write
# branch instead of inferring them from uncached.
if details.get("cache_write_tokens") is not None:
usage_found["cache_creation_input_tokens"] = usage_int(
details.get("cache_write_tokens")
)
elif provider == "gemini":
usage_meta = _sse_dict(data.get("usageMetadata"))
@@ -2425,12 +2432,15 @@ class StreamingMixin:
# Bedrock/Anthropic shape when present. Fall back to
# inferring write count from total - read for OpenAI
# shape (which has no separate write counter).
cache_creation_input_tokens = stream_state.get("cache_creation_input_tokens") or 0
# None means "not reported"; an explicit 0 is a reported zero
# and must not fall through to the inferred branch.
reported_write = stream_state.get("cache_creation_input_tokens")
cache_creation_input_tokens = reported_write or 0
if upstream_input is None:
cache_write_tokens = 0
uncached_input_tokens = 0
cache_inferred = False
elif cache_creation_input_tokens > 0:
elif reported_write is not None:
cache_write_tokens = cache_creation_input_tokens
uncached_input_tokens = max(
upstream_input - cache_read_tokens - cache_write_tokens, 0
+26
View File
@@ -448,6 +448,32 @@ def test_normalize_openai_responses_shape() -> None:
assert u == NormalizedUsage(10, 2, 8, None, input_includes_cache=True)
def test_normalize_openai_responses_reported_cache_writes() -> None:
u = normalize_usage(
{
"input_tokens": 10,
"output_tokens": 2,
"input_tokens_details": {"cached_tokens": 6, "cache_write_tokens": 3},
}
)
assert u == NormalizedUsage(10, 2, 6, 3, input_includes_cache=True)
def test_normalize_reads_nested_writes_beside_a_top_level_read() -> None:
u = normalize_usage(
{
"input_tokens": 100,
"cache_read_input_tokens": 70,
"input_tokens_details": {"cached_tokens": 70, "cache_write_tokens": 20},
}
)
assert (u.cache_read, u.cache_write) == (70, 20)
# The Responses total already contains both: billing must not add them.
done = gt.complete_outcome(_outcome(), u, status=200, latency_ms=1)
assert done.provider_input_tokens == 100
assert done.uncached_input_tokens == 10
def test_normalize_kong_flat_shape() -> None:
u = normalize_usage({"prompt_tokens": 50, "completion_tokens": 5, "cached_tokens": 40})
assert u == NormalizedUsage(50, 5, 40, None, input_includes_cache=True)
+37
View File
@@ -899,6 +899,43 @@ async def test_ws_session_metrics_include_response_completed_usage():
assert recorded["cache_read_tokens"] == 75
assert recorded["cache_write_tokens"] == 25
assert recorded["uncached_input_tokens"] == 25
# The write above is the uncached slice, not a billed write: flagged, or
# pricing bills those 25 tokens twice.
assert recorded["cache_inferred"] is True
@pytest.mark.asyncio
async def test_ws_session_metrics_use_reported_cache_writes():
"""Responses now reports billed cache writes; they are the truth, not the
uncached remainder, and pricing must treat them as real writes."""
upstream_events = [
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
json.dumps(
{
"type": "response.completed",
"response": {
"id": "r_1",
"usage": {
"input_tokens": 100,
"input_tokens_details": {"cached_tokens": 75, "cache_write_tokens": 20},
"output_tokens": 12,
},
},
}
),
]
fake_ws_mod = _make_fake_websockets_module(_FakeUpstream(upstream_events))
handler = _DummyOpenAIHandler()
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
await handler.handle_openai_responses_ws(_FakeWebSocket(frames=[_first_frame()]))
recorded = handler.metrics.recorded_requests[-1]
assert recorded["cache_read_tokens"] == 75
assert recorded["cache_write_tokens"] == 20
assert recorded["uncached_input_tokens"] == 5
assert recorded["cache_inferred"] is False
@pytest.mark.asyncio
+1 -1
View File
@@ -718,7 +718,7 @@ def test_websocket_http_fallback_blocked_when_budget_exhausted() -> None:
async def _run() -> None:
proxy = _ws_budget_proxy(retry_max_attempts=1)
proxy._ws_http_fallback = AsyncMock(return_value=(0, 0, 0, 0, 0))
proxy._ws_http_fallback = AsyncMock(return_value=(0, 0, 0, 0, 0, True))
async def _fake_connect(*args, **kwargs):
_spend_past_budget(proxy)
+36
View File
@@ -7,6 +7,7 @@ with nothing published, every path is exactly what it was before.
from __future__ import annotations
import asyncio
import json
from types import SimpleNamespace
import pytest
@@ -626,3 +627,38 @@ def test_the_resolver_follows_a_reassigned_default():
assert first.default is None
h.anthropic_backend = DEFAULT
assert BR(h.anthropic_backend).default is DEFAULT
@pytest.mark.parametrize(("written", "inferred"), [(0, False), (20, False), (None, True)])
def test_backend_openai_stream_reads_reported_cache_writes(written, inferred):
"""A reported write, zero included, is the provider's word; only an absent
field falls back to inferring writes from the uncached slice."""
from types import MethodType
from headroom.proxy.handlers.streaming import StreamingMixin
details = {"cached_tokens": 75}
if written is not None:
details["cache_write_tokens"] = written
usage = {"input_tokens": 100, "output_tokens": 5, "input_tokens_details": details}
class Backend(_Backend):
async def stream_openai_message(self, body, headers):
yield f"data: {json.dumps({'type': 'response.completed', 'response': {'usage': usage}})}\n\n"
yield "data: [DONE]\n\n"
outcomes = []
async def record(outcome):
outcomes.append(outcome)
handler = _handler(Backend("openai"))
handler._record_request_outcome = record
handler._parse_sse_usage_from_buffer = MethodType(
StreamingMixin._parse_sse_usage_from_buffer, handler
)
asyncio.run(_drive_openai_with_outcome(handler, handler.anthropic_backend))
o = outcomes[0]
assert o.cache_inferred is inferred
assert o.cache_write_tokens == (25 if written is None else written)
+62
View File
@@ -320,6 +320,68 @@ def test_responses_bills_the_original_plus_the_redrive(monkeypatch, _no_hooks) -
assert o.output_tokens == 90, f"want A+B=90, got {o.output_tokens}"
@respx.mock
def test_responses_reported_cache_writes_include_the_redrive(monkeypatch, _no_hooks) -> None:
"""Responses reports billed writes itself; both calls' writes are writes,
not an uncached remainder priced at list."""
register_turn_hook(_RedriveOnce())
app, outcomes = _app_and_outcomes(monkeypatch)
def body(rid: str, total: int, cached: int, written: int) -> dict:
return {
"id": rid,
"output": [{"type": "message", "role": "assistant", "content": []}],
"usage": {
"input_tokens": total,
"output_tokens": 10,
"input_tokens_details": {"cached_tokens": cached, "cache_write_tokens": written},
},
}
sent = iter([body("a", 400, 300, 80), body("b", 500, 400, 60)])
respx.post("https://api.openai.com/v1/responses").mock(
side_effect=lambda request: httpx.Response(200, json=next(sent))
)
with TestClient(app) as client:
r = client.post(
"/v1/responses",
json={
"model": "gpt-4o",
"input": [{"type": "message", "role": "user", "content": []}],
"stream": False,
},
headers={"authorization": "Bearer sk-test"},
)
assert r.status_code == 200
o = outcomes[-1]
assert (o.provider_input_tokens, o.cache_read_tokens, o.cache_write_tokens) == (900, 700, 140)
assert o.uncached_input_tokens == 60
assert o.cache_inferred is False
def test_redrive_writes_survive_a_final_response_without_the_field() -> None:
"""Treating the final call's own write as 0 prices the same as inferring
it (inferred writes are dropped), so the re-drives' real writes stay."""
from headroom.proxy.handlers.openai import _hook_reported_write
u = TurnHookUsage()
a = {
"usage": {
"input_tokens": 100,
"output_tokens": 1,
"input_tokens_details": {"cached_tokens": 60, "cache_write_tokens": 30},
}
}
b = {"usage": {"input_tokens": 100, "output_tokens": 1}}
u.record(a, **RESPONSES_USAGE_KEYS)
u.record(b, **RESPONSES_USAGE_KEYS)
u.settle(b)
assert _hook_reported_write(None, u) == 30
assert _hook_reported_write(5, u) == 35
assert _hook_reported_write(None, TurnHookUsage()) is None
@respx.mock
def test_no_hook_registered_bills_exactly_the_one_call(monkeypatch, _no_hooks) -> None:
"""The regression guard in the other direction: with no hook, accounting must
+26 -2
View File
@@ -410,7 +410,8 @@ class TestWsHttpFallback:
body = {"model": "gpt-5.4", "input": "big context"}
usage = asyncio.run(handler._ws_http_fallback(ws, body, json.dumps(body), {}, "req_usage"))
input_tokens, output_tokens, cache_read, _cache_write, uncached = usage
input_tokens, output_tokens, cache_read, _cache_write, uncached, inferred = usage
assert inferred is True # no cache_write_tokens on this frame
assert input_tokens == 31055
assert output_tokens == 246
assert cache_read == 20000
@@ -427,7 +428,30 @@ class TestWsHttpFallback:
ws, {"model": "gpt-5.4", "input": "hi"}, json.dumps({"input": "hi"}), {}, "req_none"
)
)
assert usage == (0, 0, 0, 0, 0)
assert usage == (0, 0, 0, 0, 0, True)
def test_fallback_returns_reported_cache_writes_as_real(self):
"""A reported write must not come back flagged inferred: pricing drops
inferred writes, so the written tokens would be billed at nothing."""
handler = _make_handler()
completed = {
"type": "response.completed",
"response": {
"usage": {
"input_tokens": 100,
"output_tokens": 5,
"input_tokens_details": {"cached_tokens": 75, "cache_write_tokens": 20},
}
},
}
handler.http_client = FakeHttpClient(
FakeStreamResponse(200, [f"data: {json.dumps(completed)}\n\n", "data: [DONE]\n\n"])
)
body = {"model": "gpt-5.4", "input": "hi"}
usage = asyncio.run(
handler._ws_http_fallback(FakeWebSocket(), body, json.dumps(body), {}, "req_w")
)
assert usage == (100, 5, 75, 20, 5, False)
def test_fallback_refreshes_codex_rate_limit_state(self, monkeypatch):
"""A successful fallback refreshes Codex /stats from response headers.