mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(openai): read Responses cache_write_tokens instead of inferring writes (#4237)
## Type of Change - [x] Bug fix (Responses usage accounting) ## Description OpenAI's Responses usage now reports billed cache writes as `input_tokens_details.cache_write_tokens`. They are billed at the write rate: $5/M on `gpt-5.6-sol`, against $4/M input. Every Headroom Responses path ignored that field and inferred writes as `input − cached` instead. As a result: - On HTTP, writes were flagged as inferred and dropped from pricing, so the written tokens were priced as uncached input at list. - On the Codex WebSocket path the inferred write was never flagged, so the uncached slice was billed twice: once as uncached and once as a write. Part of the savings redesign (step 2). The counterfactual record (#4236) needs the true write share `w = W/(U+W)`. ### The test that found it We ran Headroom Enterprise 0.4.3 in a Daytona sandbox with Codex CLI 0.162.1 on `gpt-5.6-sol` and the real GitHub MCP server, doing workshop task 1. We audited 13 calls against the provider's billing. - Every call reported `cache_write_tokens`. For call 2, usage was input 12,497, cached 9,568, write 2,926 (3 uncached). - PERF printed `cache_write=0` on every line. - Of the true $0.0850 saved, $0.0497 came from avoided cache **writes**, which core could not see. - /stats `total_cost_usd` undercounted the real bill by 8–10%. ## Changes Made - `_openai_responses_cache_split(input, cached, reported_write)` returns `(read, write, uncached, inferred)`. A reported write is used as-is; the write is inferred only when the field is absent. - Callers updated: - `_extract_responses_usage` (WS) returns a 6th `inferred` element. - The WS per-turn and session-end outcomes now pass `cache_inferred`. Before this they never set it. - The WS→HTTP fallback returns the flag. - HTTP `/v1/responses` keeps the turn-hook re-drive reads and adds the re-drives' reported writes (`TurnHookUsage.cache_write_tokens`). - The passthrough normalizer, the streaming usage parser (mapped onto the existing real-write branch) and `gateway_turn.normalize_usage` (Kong relays) also read the field. - Not changed: Chat Completions, which reports no write counter, and the `/v1/usage` relay, which doesn't map Responses-shaped details at all. That was already a gap for reads and is left as a follow-up. ## Before / after (audit replay, 13 Codex calls, current pricers) | | inferred (main) | reported (this PR) | provider truth | |---|---|---|---| | cost-tracker input cost | $0.2329 | $0.2649 | — (audit total $0.305 includes output) | | SavingsTracker compression $ (novel tokens) | $0.0434 | $0.0542 | $0.0850 total saved | The rest of the gap is re-sent history and the per-request record, which come in the follow-up PRs of the redesign. ## Testing - [x] New regression tests: - `test_ws_session_metrics_use_reported_cache_writes` (WS) - `test_fallback_returns_reported_cache_writes_as_real` - `test_responses_reported_cache_writes_include_the_redrive` (HTTP + turn hook) - `test_normalize_openai_responses_reported_cache_writes` (gateway) - [x] The existing WS test now pins `cache_inferred is True` when no write is reported. - [x] Passed (275, 14 skipped): `test_ws_http_fallback`, `test_openai_codex_ws_lifecycle`, `test_gateway_turn_unit`, `test_proxy_budget`, `test_turn_hook_usage`, `test_proxy_handler_helpers`, `test_proxy_openai_responses_integration`, `test_proxy_cache_telemetry`, `test_direct_chat_ccr_resolution`, `test_request_outcome`. - `test_turn_hook_usage` needs `LITELLM_LOCAL_MODEL_COST_MAP=True` locally: respx blocks litellm's remote price-map fetch. It fails the same way on main without it. - [x] `ruff check` / `ruff format --check` (0.16.8) and `mypy headroom` are clean. ### Test Output ```text Maintainer mixed-WebSocket-session probe: turn 1: input=100 read=75 write=25 uncached=25 cache_inferred=true turn 2: input=100 read=75 write=20 uncached=5 cache_inferred=false ``` The probe exercised the actual WebSocket handler using the existing lifecycle test harness. It verifies separate per-turn records for this mixed session; it does not establish behavior for every suppressed memory-tool re-drive. The prior qualified head completed hosted CI with 30 successful and 12 skipped checks. The current main-synced head is being qualified again. ## Real Behavior Proof - Environment: macOS, this branch; the audit's recorded provider usage for the 13 Codex calls. - Exact command / steps: replayed each call's usage through `CostTracker.record_tokens` and `estimate_request_savings_usd`, first with the inferred split (main) and then with the reported split (this PR). Script: `replay_c.py` in the audit scratchpad. - Observed result: input cost $0.2329 → $0.2649 and compression dollars $0.0434 → $0.0542, as in the table above. Call 2 now records read 9,568 / write 2,926 / uncached 3 instead of write 2,929 inferred. - Not tested: a live Codex session against this branch. The live re-run comes with the audit-eval PR of the redesign. ## Runtime Rollout Safety - Rollout-managed feature(s): no new rollout-managed feature switch. - Minimum rollout channel: existing Responses transport policy applies. - Unsafe override required: no. - Qualification impact: reported cache writes affect recorded cost and savings; hosted checks and human review remain required. - Stable/default behavior changed: Responses accounting prices explicitly reported writes as billed writes; inferred writes remain marked as estimates. - Yes for OpenAI Responses cost and savings dollars. The newly reported writes are now priced at the write rate. - WS turns without a reported write are now flagged inferred. That removes the double-billed uncached slice, so WS cost totals go down for older responses. - WS turns now count toward `cache_inferred_turns`. - Kill switch / disable path: none needed; this follows the provider's own usage. - Rollback path: revert. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4237" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ## Review Readiness - [ ] I have performed a self-review - [ ] This PR is ready for human review The maintainer has recorded independent qualification above. Author readiness attestations have not been filled on the author's behalf. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JD Davis
parent
1a40b3ce67
commit
46306024e7
@@ -342,7 +342,9 @@ def normalize_usage(usage: dict[str, Any]) -> NormalizedUsage:
|
||||
# ``input_tokens`` too, so the key name cannot be the discriminator. Keying
|
||||
# off Anthropic's cache fields can misread only a relay that mixes the two
|
||||
# vocabularies, and then it errs towards not inflating the prompt.
|
||||
input_includes_cache = not (
|
||||
# Responses' ``input_tokens_details`` marks an inclusive total even when a
|
||||
# relay also flattened a top-level read beside it.
|
||||
input_includes_cache = "input_tokens_details" in usage or not (
|
||||
"cache_read_input_tokens" in usage or "cache_creation_input_tokens" in usage
|
||||
)
|
||||
output_tokens = _usage_int(usage, "output_tokens")
|
||||
@@ -351,10 +353,16 @@ def normalize_usage(usage: dict[str, Any]) -> NormalizedUsage:
|
||||
|
||||
cache_read = _usage_int(usage, "cache_read_input_tokens")
|
||||
cache_write = _usage_int(usage, "cache_creation_input_tokens")
|
||||
if cache_write is None:
|
||||
# Responses reports billed writes as input_tokens_details.cache_write_tokens;
|
||||
# chat reports none. Read whether or not a top-level read was relayed.
|
||||
details = usage.get("input_tokens_details")
|
||||
if isinstance(details, dict):
|
||||
cache_write = _usage_int(details, "cache_write_tokens")
|
||||
if cache_read is None:
|
||||
# OpenAI reports reads only (no write counter exists), nested under
|
||||
# prompt_tokens_details (chat), input_tokens_details (Responses) or —
|
||||
# in Kong's flattened log shape — at the top level.
|
||||
# OpenAI nests reads under prompt_tokens_details (chat),
|
||||
# input_tokens_details (Responses) or — in Kong's flattened log shape —
|
||||
# puts them at the top level.
|
||||
for details_key in ("prompt_tokens_details", "input_tokens_details"):
|
||||
details = usage.get(details_key)
|
||||
if isinstance(details, dict):
|
||||
|
||||
@@ -351,13 +351,22 @@ class TurnHookUsage:
|
||||
the safe direction for a bill.
|
||||
"""
|
||||
|
||||
__slots__ = ("_seen", "input_tokens", "output_tokens", "cache_read_tokens", "extra_calls")
|
||||
__slots__ = (
|
||||
"_seen",
|
||||
"input_tokens",
|
||||
"output_tokens",
|
||||
"cache_read_tokens",
|
||||
"cache_write_tokens",
|
||||
"extra_calls",
|
||||
)
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._seen: list[tuple[int, Any, int, int, int]] = []
|
||||
self._seen: list[tuple[int, Any, int, int, int, int]] = []
|
||||
self.input_tokens = 0
|
||||
self.output_tokens = 0
|
||||
self.cache_read_tokens = 0
|
||||
#: Reported (Responses ``cache_write_tokens``) writes only; 0 otherwise.
|
||||
self.cache_write_tokens = 0
|
||||
self.extra_calls = 0
|
||||
|
||||
def record(
|
||||
@@ -381,17 +390,19 @@ class TurnHookUsage:
|
||||
usage_int(usage.get(input_key)),
|
||||
usage_int(usage.get(output_key)),
|
||||
cached,
|
||||
_reported_cache_write(details) or 0,
|
||||
)
|
||||
else:
|
||||
entry = (id(payload), payload, 0, 0, 0)
|
||||
entry = (id(payload), payload, 0, 0, 0, 0)
|
||||
self._seen.append(entry)
|
||||
|
||||
def settle(self, final: Any) -> None:
|
||||
"""Total everything except ``final``, which the usage block will read."""
|
||||
self.input_tokens = self.output_tokens = self.cache_read_tokens = 0
|
||||
self.cache_write_tokens = 0
|
||||
self.extra_calls = 0
|
||||
dropped = False
|
||||
for _ident, payload, tin, tout, cached in self._seen:
|
||||
for _ident, payload, tin, tout, cached, written in self._seen:
|
||||
if not dropped and payload is final:
|
||||
dropped = True
|
||||
continue
|
||||
@@ -399,6 +410,7 @@ class TurnHookUsage:
|
||||
self.input_tokens += tin
|
||||
self.output_tokens += tout
|
||||
self.cache_read_tokens += cached
|
||||
self.cache_write_tokens += written
|
||||
|
||||
|
||||
def _sanitize_forwarded_response_headers(
|
||||
@@ -614,7 +626,13 @@ def _passthrough_usage_from_json(payload: Any, provider: str | None = None) -> d
|
||||
"input_tokens": billed_input_from_usage(usage, provider) or usage_int(input_tokens),
|
||||
"output_tokens": usage_int(output_tokens),
|
||||
"cache_read_input_tokens": usage_int(usage.get("cache_read_input_tokens", cache_read)),
|
||||
"cache_creation_input_tokens": usage_int(usage.get("cache_creation_input_tokens")),
|
||||
# Responses spells a billed write ``input_tokens_details.cache_write_tokens``.
|
||||
"cache_creation_input_tokens": usage_int(
|
||||
usage.get(
|
||||
"cache_creation_input_tokens",
|
||||
details.get("cache_write_tokens") if isinstance(details, dict) else None,
|
||||
)
|
||||
),
|
||||
}
|
||||
|
||||
return {}
|
||||
@@ -1650,6 +1668,48 @@ def _infer_openai_cache_write_tokens(input_tokens: int, cache_read_tokens: int)
|
||||
return max(input_tokens - cache_read_tokens, 0)
|
||||
|
||||
|
||||
def _reported_cache_write(details: Any) -> int | None:
|
||||
"""``input_tokens_details.cache_write_tokens``, or ``None`` when not reported."""
|
||||
if not isinstance(details, dict) or details.get("cache_write_tokens") is None:
|
||||
return None
|
||||
return usage_int(details.get("cache_write_tokens"))
|
||||
|
||||
|
||||
def _hook_reported_write(final_write: int | None, hook_usage: Any) -> int | None:
|
||||
"""The final response's reported write plus any turn-hook re-drives' writes.
|
||||
|
||||
A final response without the field still carries the re-drives' reported
|
||||
writes: treating its own write as 0 prices the same as inferring it, since
|
||||
an inferred write is dropped from pricing anyway.
|
||||
"""
|
||||
hook_writes = hook_usage.cache_write_tokens if hook_usage.extra_calls else 0
|
||||
if final_write is None:
|
||||
return hook_writes or None
|
||||
return final_write + hook_writes
|
||||
|
||||
|
||||
def _openai_responses_cache_split(
|
||||
input_tokens: int, cached: int, reported_write: int | None
|
||||
) -> tuple[int, int, int, bool]:
|
||||
"""Split a Responses ``input_tokens`` total into ``(read, write, uncached, inferred)``.
|
||||
|
||||
Responses usage now reports cache writes itself as
|
||||
``input_tokens_details.cache_write_tokens``, billed at the write rate
|
||||
(above list input on current models). When present it is the truth and the
|
||||
three buckets are disjoint. Older responses omit it; the write is then
|
||||
inferred from the uncached slice and flagged, so pricing drops it rather
|
||||
than billing the same tokens twice.
|
||||
"""
|
||||
if reported_write is not None:
|
||||
return cached, reported_write, max(input_tokens - cached - reported_write, 0), False
|
||||
return (
|
||||
cached,
|
||||
_infer_openai_cache_write_tokens(input_tokens, cached),
|
||||
max(input_tokens - cached, 0),
|
||||
True,
|
||||
)
|
||||
|
||||
|
||||
def _deferrable_savings_delta(input_delta: int, saved_delta: int) -> int:
|
||||
"""Gate a WS Responses turn's compression-savings delta on input accounting.
|
||||
|
||||
@@ -1670,32 +1730,36 @@ def _deferrable_savings_delta(input_delta: int, saved_delta: int) -> int:
|
||||
return saved_delta
|
||||
|
||||
|
||||
def _extract_responses_usage(event: dict[str, Any]) -> tuple[int, int, int, int, int]:
|
||||
def _extract_responses_usage(event: dict[str, Any]) -> tuple[int, int, int, int, int, bool]:
|
||||
"""Return input/output/cache usage from a Responses event.
|
||||
|
||||
Codex WebSocket streams include usage on ``response.completed`` events.
|
||||
The shape mirrors HTTP Responses usage:
|
||||
``response.usage.input_tokens`` plus
|
||||
``response.usage.input_tokens_details.cached_tokens``.
|
||||
``response.usage.input_tokens_details.cached_tokens`` / ``cache_write_tokens``.
|
||||
The last element says whether the write was inferred (see
|
||||
:func:`_openai_responses_cache_split`).
|
||||
"""
|
||||
|
||||
if event.get("type") != "response.completed":
|
||||
return 0, 0, 0, 0, 0
|
||||
return 0, 0, 0, 0, 0, True
|
||||
|
||||
response = event.get("response")
|
||||
if not isinstance(response, dict):
|
||||
response = {}
|
||||
usage = response.get("usage") or event.get("usage")
|
||||
if not isinstance(usage, dict):
|
||||
return 0, 0, 0, 0, 0
|
||||
return 0, 0, 0, 0, 0, True
|
||||
|
||||
input_tokens = usage_int(usage.get("input_tokens"))
|
||||
output_tokens = usage_int(usage.get("output_tokens"))
|
||||
details = usage.get("input_tokens_details")
|
||||
cached_tokens = usage_int(details.get("cached_tokens")) if isinstance(details, dict) else 0
|
||||
cache_write_tokens = _infer_openai_cache_write_tokens(input_tokens, cached_tokens)
|
||||
uncached_tokens = max(input_tokens - cached_tokens, 0)
|
||||
return input_tokens, output_tokens, cached_tokens, cache_write_tokens, uncached_tokens
|
||||
cached, write, uncached, inferred = _openai_responses_cache_split(
|
||||
input_tokens,
|
||||
usage_int(details.get("cached_tokens")) if isinstance(details, dict) else 0,
|
||||
_reported_cache_write(details),
|
||||
)
|
||||
return input_tokens, output_tokens, cached, write, uncached, inferred
|
||||
|
||||
|
||||
def _prefers_http1_passthrough(base_url: str) -> bool:
|
||||
@@ -6538,6 +6602,7 @@ class OpenAIHandlerMixin:
|
||||
total_input_tokens = original_tokens # fallback
|
||||
output_tokens = 0
|
||||
cache_read_tokens = 0
|
||||
details: Any = None
|
||||
resp_json = None
|
||||
try:
|
||||
resp_json = response.json()
|
||||
@@ -6757,11 +6822,17 @@ class OpenAIHandlerMixin:
|
||||
# Cost is recorded once by the outcome funnel below; here we only
|
||||
# compute the cache-write / uncached split the funnel needs.
|
||||
# (Recording here too double-counted spend + budget on this path.)
|
||||
cache_write_tokens = _infer_openai_cache_write_tokens(
|
||||
# Reads include the hook's re-drives added above.
|
||||
(
|
||||
cache_read_tokens,
|
||||
cache_write_tokens,
|
||||
uncached_input_tokens,
|
||||
cache_inferred,
|
||||
) = _openai_responses_cache_split(
|
||||
total_input_tokens,
|
||||
cache_read_tokens,
|
||||
_hook_reported_write(_reported_cache_write(details), _resp_hook_usage),
|
||||
)
|
||||
uncached_input_tokens = max(0, total_input_tokens - cache_read_tokens)
|
||||
|
||||
# Was: optimized := provider count, then original := max(original,
|
||||
# optimized + saved) to stop `attempted` exceeding `original`.
|
||||
@@ -6803,7 +6874,7 @@ class OpenAIHandlerMixin:
|
||||
cache_read_tokens=cache_read_tokens,
|
||||
cache_write_tokens=cache_write_tokens,
|
||||
uncached_input_tokens=uncached_input_tokens,
|
||||
cache_inferred=True,
|
||||
cache_inferred=cache_inferred,
|
||||
total_latency_ms=total_latency,
|
||||
overhead_ms=optimization_latency,
|
||||
transforms_applied=tuple(transforms_applied),
|
||||
@@ -7564,6 +7635,11 @@ class OpenAIHandlerMixin:
|
||||
ws_cache_read_tokens_total = 0
|
||||
ws_cache_write_tokens_total = 0
|
||||
ws_uncached_input_tokens_total = 0
|
||||
# Whether the latest usage frame's write was inferred rather than
|
||||
# reported (see _openai_responses_cache_split). One flag for the
|
||||
# summed deltas: a session that mixed reported and unreported
|
||||
# frames would misprice one kind, which Codex does not do.
|
||||
ws_cache_inferred = True
|
||||
ws_recorded_input_tokens_total = 0
|
||||
ws_recorded_output_tokens_total = 0
|
||||
ws_recorded_cache_read_tokens_total = 0
|
||||
@@ -8890,7 +8966,7 @@ class OpenAIHandlerMixin:
|
||||
nonlocal ws_memory_response_pending
|
||||
nonlocal ws_input_tokens_total, ws_output_tokens_total
|
||||
nonlocal ws_cache_read_tokens_total, ws_cache_write_tokens_total
|
||||
nonlocal ws_uncached_input_tokens_total
|
||||
nonlocal ws_uncached_input_tokens_total, ws_cache_inferred
|
||||
nonlocal ws_recorded_input_tokens_total
|
||||
nonlocal ws_recorded_output_tokens_total
|
||||
nonlocal ws_recorded_cache_read_tokens_total
|
||||
@@ -9020,6 +9096,7 @@ class OpenAIHandlerMixin:
|
||||
cache_read_tokens=max(0, cache_read_delta),
|
||||
cache_write_tokens=max(0, cache_write_delta),
|
||||
uncached_input_tokens=max(0, uncached_delta),
|
||||
cache_inferred=ws_cache_inferred,
|
||||
total_latency_ms=latency_ms,
|
||||
overhead_ms=overhead_delta_ms,
|
||||
ttfb_ms=ttfb_for_record_ms,
|
||||
@@ -9164,8 +9241,10 @@ class OpenAIHandlerMixin:
|
||||
usage_cache_read_tokens,
|
||||
usage_cache_write_tokens,
|
||||
usage_uncached_tokens,
|
||||
usage_cache_inferred,
|
||||
) = _extract_responses_usage(event)
|
||||
if usage_input_tokens or usage_output_tokens:
|
||||
ws_cache_inferred = usage_cache_inferred
|
||||
ws_input_tokens_total += usage_input_tokens
|
||||
ws_output_tokens_total += usage_output_tokens
|
||||
ws_cache_read_tokens_total += usage_cache_read_tokens
|
||||
@@ -9559,6 +9638,7 @@ class OpenAIHandlerMixin:
|
||||
fb_cache_read_tokens,
|
||||
fb_cache_write_tokens,
|
||||
fb_uncached_tokens,
|
||||
ws_cache_inferred,
|
||||
) = await self._ws_http_fallback(
|
||||
websocket, body, first_msg_raw, upstream_headers, request_id
|
||||
)
|
||||
@@ -9690,6 +9770,7 @@ class OpenAIHandlerMixin:
|
||||
cache_read_tokens=residual_cache_read_tokens,
|
||||
cache_write_tokens=residual_cache_write_tokens,
|
||||
uncached_input_tokens=residual_uncached_input_tokens,
|
||||
cache_inferred=ws_cache_inferred,
|
||||
total_latency_ms=ws_session_duration_ms,
|
||||
overhead_ms=final_overhead_delta_ms,
|
||||
ttfb_ms=final_ttfb_ms,
|
||||
@@ -9836,7 +9917,7 @@ class OpenAIHandlerMixin:
|
||||
first_msg_raw: str,
|
||||
upstream_headers: dict[str, str],
|
||||
request_id: str,
|
||||
) -> tuple[int, int, int, int, int]:
|
||||
) -> tuple[int, int, int, int, int, bool]:
|
||||
"""Fall back to HTTP POST streaming when upstream WS fails.
|
||||
|
||||
Converts the WS ``response.create`` message to an HTTP POST to
|
||||
@@ -9850,7 +9931,9 @@ class OpenAIHandlerMixin:
|
||||
authoritative wire-token count; otherwise a fallback recorded
|
||||
``input_tokens=0`` and savings percentages blew past 100 (#2957).
|
||||
"""
|
||||
fallback_usage = [0, 0, 0, 0, 0]
|
||||
# Five summed counters, then whether the latest frame's write was
|
||||
# inferred (see _openai_responses_cache_split).
|
||||
fallback_usage: list[int] = [0, 0, 0, 0, 0, True]
|
||||
|
||||
def _accumulate_usage(data_str: str) -> None:
|
||||
try:
|
||||
@@ -9858,8 +9941,10 @@ class OpenAIHandlerMixin:
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
return
|
||||
if isinstance(event, dict) and event.get("type") == "response.completed":
|
||||
for i, value in enumerate(_extract_responses_usage(event)):
|
||||
usage = _extract_responses_usage(event)
|
||||
for i, value in enumerate(usage[:5]):
|
||||
fallback_usage[i] += value
|
||||
fallback_usage[5] = usage[5]
|
||||
|
||||
# Route to correct endpoint based on auth mode
|
||||
is_chatgpt_fallback = has_chatgpt_account_header(upstream_headers)
|
||||
|
||||
@@ -421,6 +421,13 @@ class StreamingMixin:
|
||||
usage_found["cache_read_input_tokens"] = usage_int(
|
||||
details.get("cached_tokens")
|
||||
)
|
||||
# Responses reports billed cache writes itself; the
|
||||
# Anthropic-shape key routes them down the real-write
|
||||
# branch instead of inferring them from uncached.
|
||||
if details.get("cache_write_tokens") is not None:
|
||||
usage_found["cache_creation_input_tokens"] = usage_int(
|
||||
details.get("cache_write_tokens")
|
||||
)
|
||||
|
||||
elif provider == "gemini":
|
||||
usage_meta = _sse_dict(data.get("usageMetadata"))
|
||||
@@ -2425,12 +2432,15 @@ class StreamingMixin:
|
||||
# Bedrock/Anthropic shape when present. Fall back to
|
||||
# inferring write count from total - read for OpenAI
|
||||
# shape (which has no separate write counter).
|
||||
cache_creation_input_tokens = stream_state.get("cache_creation_input_tokens") or 0
|
||||
# None means "not reported"; an explicit 0 is a reported zero
|
||||
# and must not fall through to the inferred branch.
|
||||
reported_write = stream_state.get("cache_creation_input_tokens")
|
||||
cache_creation_input_tokens = reported_write or 0
|
||||
if upstream_input is None:
|
||||
cache_write_tokens = 0
|
||||
uncached_input_tokens = 0
|
||||
cache_inferred = False
|
||||
elif cache_creation_input_tokens > 0:
|
||||
elif reported_write is not None:
|
||||
cache_write_tokens = cache_creation_input_tokens
|
||||
uncached_input_tokens = max(
|
||||
upstream_input - cache_read_tokens - cache_write_tokens, 0
|
||||
|
||||
@@ -448,6 +448,32 @@ def test_normalize_openai_responses_shape() -> None:
|
||||
assert u == NormalizedUsage(10, 2, 8, None, input_includes_cache=True)
|
||||
|
||||
|
||||
def test_normalize_openai_responses_reported_cache_writes() -> None:
|
||||
u = normalize_usage(
|
||||
{
|
||||
"input_tokens": 10,
|
||||
"output_tokens": 2,
|
||||
"input_tokens_details": {"cached_tokens": 6, "cache_write_tokens": 3},
|
||||
}
|
||||
)
|
||||
assert u == NormalizedUsage(10, 2, 6, 3, input_includes_cache=True)
|
||||
|
||||
|
||||
def test_normalize_reads_nested_writes_beside_a_top_level_read() -> None:
|
||||
u = normalize_usage(
|
||||
{
|
||||
"input_tokens": 100,
|
||||
"cache_read_input_tokens": 70,
|
||||
"input_tokens_details": {"cached_tokens": 70, "cache_write_tokens": 20},
|
||||
}
|
||||
)
|
||||
assert (u.cache_read, u.cache_write) == (70, 20)
|
||||
# The Responses total already contains both: billing must not add them.
|
||||
done = gt.complete_outcome(_outcome(), u, status=200, latency_ms=1)
|
||||
assert done.provider_input_tokens == 100
|
||||
assert done.uncached_input_tokens == 10
|
||||
|
||||
|
||||
def test_normalize_kong_flat_shape() -> None:
|
||||
u = normalize_usage({"prompt_tokens": 50, "completion_tokens": 5, "cached_tokens": 40})
|
||||
assert u == NormalizedUsage(50, 5, 40, None, input_includes_cache=True)
|
||||
|
||||
@@ -899,6 +899,43 @@ async def test_ws_session_metrics_include_response_completed_usage():
|
||||
assert recorded["cache_read_tokens"] == 75
|
||||
assert recorded["cache_write_tokens"] == 25
|
||||
assert recorded["uncached_input_tokens"] == 25
|
||||
# The write above is the uncached slice, not a billed write: flagged, or
|
||||
# pricing bills those 25 tokens twice.
|
||||
assert recorded["cache_inferred"] is True
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ws_session_metrics_use_reported_cache_writes():
|
||||
"""Responses now reports billed cache writes; they are the truth, not the
|
||||
uncached remainder, and pricing must treat them as real writes."""
|
||||
|
||||
upstream_events = [
|
||||
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
|
||||
json.dumps(
|
||||
{
|
||||
"type": "response.completed",
|
||||
"response": {
|
||||
"id": "r_1",
|
||||
"usage": {
|
||||
"input_tokens": 100,
|
||||
"input_tokens_details": {"cached_tokens": 75, "cache_write_tokens": 20},
|
||||
"output_tokens": 12,
|
||||
},
|
||||
},
|
||||
}
|
||||
),
|
||||
]
|
||||
fake_ws_mod = _make_fake_websockets_module(_FakeUpstream(upstream_events))
|
||||
handler = _DummyOpenAIHandler()
|
||||
|
||||
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
|
||||
await handler.handle_openai_responses_ws(_FakeWebSocket(frames=[_first_frame()]))
|
||||
|
||||
recorded = handler.metrics.recorded_requests[-1]
|
||||
assert recorded["cache_read_tokens"] == 75
|
||||
assert recorded["cache_write_tokens"] == 20
|
||||
assert recorded["uncached_input_tokens"] == 5
|
||||
assert recorded["cache_inferred"] is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
@@ -718,7 +718,7 @@ def test_websocket_http_fallback_blocked_when_budget_exhausted() -> None:
|
||||
|
||||
async def _run() -> None:
|
||||
proxy = _ws_budget_proxy(retry_max_attempts=1)
|
||||
proxy._ws_http_fallback = AsyncMock(return_value=(0, 0, 0, 0, 0))
|
||||
proxy._ws_http_fallback = AsyncMock(return_value=(0, 0, 0, 0, 0, True))
|
||||
|
||||
async def _fake_connect(*args, **kwargs):
|
||||
_spend_past_budget(proxy)
|
||||
|
||||
@@ -7,6 +7,7 @@ with nothing published, every path is exactly what it was before.
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
@@ -626,3 +627,38 @@ def test_the_resolver_follows_a_reassigned_default():
|
||||
assert first.default is None
|
||||
h.anthropic_backend = DEFAULT
|
||||
assert BR(h.anthropic_backend).default is DEFAULT
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("written", "inferred"), [(0, False), (20, False), (None, True)])
|
||||
def test_backend_openai_stream_reads_reported_cache_writes(written, inferred):
|
||||
"""A reported write, zero included, is the provider's word; only an absent
|
||||
field falls back to inferring writes from the uncached slice."""
|
||||
from types import MethodType
|
||||
|
||||
from headroom.proxy.handlers.streaming import StreamingMixin
|
||||
|
||||
details = {"cached_tokens": 75}
|
||||
if written is not None:
|
||||
details["cache_write_tokens"] = written
|
||||
usage = {"input_tokens": 100, "output_tokens": 5, "input_tokens_details": details}
|
||||
|
||||
class Backend(_Backend):
|
||||
async def stream_openai_message(self, body, headers):
|
||||
yield f"data: {json.dumps({'type': 'response.completed', 'response': {'usage': usage}})}\n\n"
|
||||
yield "data: [DONE]\n\n"
|
||||
|
||||
outcomes = []
|
||||
|
||||
async def record(outcome):
|
||||
outcomes.append(outcome)
|
||||
|
||||
handler = _handler(Backend("openai"))
|
||||
handler._record_request_outcome = record
|
||||
handler._parse_sse_usage_from_buffer = MethodType(
|
||||
StreamingMixin._parse_sse_usage_from_buffer, handler
|
||||
)
|
||||
asyncio.run(_drive_openai_with_outcome(handler, handler.anthropic_backend))
|
||||
|
||||
o = outcomes[0]
|
||||
assert o.cache_inferred is inferred
|
||||
assert o.cache_write_tokens == (25 if written is None else written)
|
||||
|
||||
@@ -320,6 +320,68 @@ def test_responses_bills_the_original_plus_the_redrive(monkeypatch, _no_hooks) -
|
||||
assert o.output_tokens == 90, f"want A+B=90, got {o.output_tokens}"
|
||||
|
||||
|
||||
@respx.mock
|
||||
def test_responses_reported_cache_writes_include_the_redrive(monkeypatch, _no_hooks) -> None:
|
||||
"""Responses reports billed writes itself; both calls' writes are writes,
|
||||
not an uncached remainder priced at list."""
|
||||
register_turn_hook(_RedriveOnce())
|
||||
app, outcomes = _app_and_outcomes(monkeypatch)
|
||||
|
||||
def body(rid: str, total: int, cached: int, written: int) -> dict:
|
||||
return {
|
||||
"id": rid,
|
||||
"output": [{"type": "message", "role": "assistant", "content": []}],
|
||||
"usage": {
|
||||
"input_tokens": total,
|
||||
"output_tokens": 10,
|
||||
"input_tokens_details": {"cached_tokens": cached, "cache_write_tokens": written},
|
||||
},
|
||||
}
|
||||
|
||||
sent = iter([body("a", 400, 300, 80), body("b", 500, 400, 60)])
|
||||
respx.post("https://api.openai.com/v1/responses").mock(
|
||||
side_effect=lambda request: httpx.Response(200, json=next(sent))
|
||||
)
|
||||
|
||||
with TestClient(app) as client:
|
||||
r = client.post(
|
||||
"/v1/responses",
|
||||
json={
|
||||
"model": "gpt-4o",
|
||||
"input": [{"type": "message", "role": "user", "content": []}],
|
||||
"stream": False,
|
||||
},
|
||||
headers={"authorization": "Bearer sk-test"},
|
||||
)
|
||||
assert r.status_code == 200
|
||||
o = outcomes[-1]
|
||||
assert (o.provider_input_tokens, o.cache_read_tokens, o.cache_write_tokens) == (900, 700, 140)
|
||||
assert o.uncached_input_tokens == 60
|
||||
assert o.cache_inferred is False
|
||||
|
||||
|
||||
def test_redrive_writes_survive_a_final_response_without_the_field() -> None:
|
||||
"""Treating the final call's own write as 0 prices the same as inferring
|
||||
it (inferred writes are dropped), so the re-drives' real writes stay."""
|
||||
from headroom.proxy.handlers.openai import _hook_reported_write
|
||||
|
||||
u = TurnHookUsage()
|
||||
a = {
|
||||
"usage": {
|
||||
"input_tokens": 100,
|
||||
"output_tokens": 1,
|
||||
"input_tokens_details": {"cached_tokens": 60, "cache_write_tokens": 30},
|
||||
}
|
||||
}
|
||||
b = {"usage": {"input_tokens": 100, "output_tokens": 1}}
|
||||
u.record(a, **RESPONSES_USAGE_KEYS)
|
||||
u.record(b, **RESPONSES_USAGE_KEYS)
|
||||
u.settle(b)
|
||||
assert _hook_reported_write(None, u) == 30
|
||||
assert _hook_reported_write(5, u) == 35
|
||||
assert _hook_reported_write(None, TurnHookUsage()) is None
|
||||
|
||||
|
||||
@respx.mock
|
||||
def test_no_hook_registered_bills_exactly_the_one_call(monkeypatch, _no_hooks) -> None:
|
||||
"""The regression guard in the other direction: with no hook, accounting must
|
||||
|
||||
@@ -410,7 +410,8 @@ class TestWsHttpFallback:
|
||||
body = {"model": "gpt-5.4", "input": "big context"}
|
||||
usage = asyncio.run(handler._ws_http_fallback(ws, body, json.dumps(body), {}, "req_usage"))
|
||||
|
||||
input_tokens, output_tokens, cache_read, _cache_write, uncached = usage
|
||||
input_tokens, output_tokens, cache_read, _cache_write, uncached, inferred = usage
|
||||
assert inferred is True # no cache_write_tokens on this frame
|
||||
assert input_tokens == 31055
|
||||
assert output_tokens == 246
|
||||
assert cache_read == 20000
|
||||
@@ -427,7 +428,30 @@ class TestWsHttpFallback:
|
||||
ws, {"model": "gpt-5.4", "input": "hi"}, json.dumps({"input": "hi"}), {}, "req_none"
|
||||
)
|
||||
)
|
||||
assert usage == (0, 0, 0, 0, 0)
|
||||
assert usage == (0, 0, 0, 0, 0, True)
|
||||
|
||||
def test_fallback_returns_reported_cache_writes_as_real(self):
|
||||
"""A reported write must not come back flagged inferred: pricing drops
|
||||
inferred writes, so the written tokens would be billed at nothing."""
|
||||
handler = _make_handler()
|
||||
completed = {
|
||||
"type": "response.completed",
|
||||
"response": {
|
||||
"usage": {
|
||||
"input_tokens": 100,
|
||||
"output_tokens": 5,
|
||||
"input_tokens_details": {"cached_tokens": 75, "cache_write_tokens": 20},
|
||||
}
|
||||
},
|
||||
}
|
||||
handler.http_client = FakeHttpClient(
|
||||
FakeStreamResponse(200, [f"data: {json.dumps(completed)}\n\n", "data: [DONE]\n\n"])
|
||||
)
|
||||
body = {"model": "gpt-5.4", "input": "hi"}
|
||||
usage = asyncio.run(
|
||||
handler._ws_http_fallback(FakeWebSocket(), body, json.dumps(body), {}, "req_w")
|
||||
)
|
||||
assert usage == (100, 5, 75, 20, 5, False)
|
||||
|
||||
def test_fallback_refreshes_codex_rate_limit_state(self, monkeypatch):
|
||||
"""A successful fallback refreshes Codex /stats from response headers.
|
||||
|
||||
Reference in New Issue
Block a user