Commit Graph
2 Commits
Author SHA1 Message Date
46306024e7 fix(openai): read Responses cache_write_tokens instead of inferring writes (#4237)
## Type of Change
- [x] Bug fix (Responses usage accounting)

## Description

OpenAI's Responses usage now reports billed cache writes as
`input_tokens_details.cache_write_tokens`. They are billed at the write
rate: $5/M on `gpt-5.6-sol`, against $4/M input. Every Headroom
Responses path ignored that field and inferred writes as `input −
cached` instead. As a result:
- On HTTP, writes were flagged as inferred and dropped from pricing, so
the written tokens were priced as uncached input at list.
- On the Codex WebSocket path the inferred write was never flagged, so
the uncached slice was billed twice: once as uncached and once as a
write.

Part of the savings redesign (step 2). The counterfactual record (#4236)
needs the true write share `w = W/(U+W)`.

### The test that found it

We ran Headroom Enterprise 0.4.3 in a Daytona sandbox with Codex CLI
0.162.1 on `gpt-5.6-sol` and the real GitHub MCP server, doing workshop
task 1. We audited 13 calls against the provider's billing.
- Every call reported `cache_write_tokens`. For call 2, usage was input
12,497, cached 9,568, write 2,926 (3 uncached).
- PERF printed `cache_write=0` on every line.
- Of the true $0.0850 saved, $0.0497 came from avoided cache **writes**,
which core could not see.
- /stats `total_cost_usd` undercounted the real bill by 8–10%.

## Changes Made

- `_openai_responses_cache_split(input, cached, reported_write)` returns
`(read, write, uncached, inferred)`. A reported write is used as-is; the
write is inferred only when the field is absent.
- Callers updated:
  - `_extract_responses_usage` (WS) returns a 6th `inferred` element.
- The WS per-turn and session-end outcomes now pass `cache_inferred`.
Before this they never set it.
  - The WS→HTTP fallback returns the flag.
- HTTP `/v1/responses` keeps the turn-hook re-drive reads and adds the
re-drives' reported writes (`TurnHookUsage.cache_write_tokens`).
- The passthrough normalizer, the streaming usage parser (mapped onto
the existing real-write branch) and `gateway_turn.normalize_usage` (Kong
relays) also read the field.
- Not changed: Chat Completions, which reports no write counter, and the
`/v1/usage` relay, which doesn't map Responses-shaped details at all.
That was already a gap for reads and is left as a follow-up.

## Before / after (audit replay, 13 Codex calls, current pricers)

| | inferred (main) | reported (this PR) | provider truth |
|---|---|---|---|
| cost-tracker input cost | $0.2329 | $0.2649 | — (audit total $0.305
includes output) |
| SavingsTracker compression $ (novel tokens) | $0.0434 | $0.0542 |
$0.0850 total saved |

The rest of the gap is re-sent history and the per-request record, which
come in the follow-up PRs of the redesign.

## Testing

- [x] New regression tests:
  - `test_ws_session_metrics_use_reported_cache_writes` (WS)
  - `test_fallback_returns_reported_cache_writes_as_real`
- `test_responses_reported_cache_writes_include_the_redrive` (HTTP +
turn hook)
  - `test_normalize_openai_responses_reported_cache_writes` (gateway)
- [x] The existing WS test now pins `cache_inferred is True` when no
write is reported.
- [x] Passed (275, 14 skipped): `test_ws_http_fallback`,
`test_openai_codex_ws_lifecycle`, `test_gateway_turn_unit`,
`test_proxy_budget`, `test_turn_hook_usage`,
`test_proxy_handler_helpers`, `test_proxy_openai_responses_integration`,
`test_proxy_cache_telemetry`, `test_direct_chat_ccr_resolution`,
`test_request_outcome`.
- `test_turn_hook_usage` needs `LITELLM_LOCAL_MODEL_COST_MAP=True`
locally: respx blocks litellm's remote price-map fetch. It fails the
same way on main without it.
- [x] `ruff check` / `ruff format --check` (0.16.8) and `mypy headroom`
are clean.


### Test Output

```text
Maintainer mixed-WebSocket-session probe:
turn 1: input=100 read=75 write=25 uncached=25 cache_inferred=true
turn 2: input=100 read=75 write=20 uncached=5 cache_inferred=false
```

The probe exercised the actual WebSocket handler using the existing
lifecycle test harness. It verifies separate per-turn records for this
mixed session; it does not establish behavior for every suppressed
memory-tool re-drive. The prior qualified head completed hosted CI with
30 successful and 12 skipped checks. The current main-synced head is
being qualified again.

## Real Behavior Proof

- Environment: macOS, this branch; the audit's recorded provider usage
for the 13 Codex calls.
- Exact command / steps: replayed each call's usage through
`CostTracker.record_tokens` and `estimate_request_savings_usd`, first
with the inferred split (main) and then with the reported split (this
PR). Script: `replay_c.py` in the audit scratchpad.
- Observed result: input cost $0.2329 → $0.2649 and compression dollars
$0.0434 → $0.0542, as in the table above. Call 2 now records read 9,568
/ write 2,926 / uncached 3 instead of write 2,929 inferred.
- Not tested: a live Codex session against this branch. The live re-run
comes with the audit-eval PR of the redesign.

## Runtime Rollout Safety
- Rollout-managed feature(s): no new rollout-managed feature switch.
- Minimum rollout channel: existing Responses transport policy applies.
- Unsafe override required: no.
- Qualification impact: reported cache writes affect recorded cost and
savings; hosted checks and human review remain required.

- Stable/default behavior changed: Responses accounting prices
explicitly reported writes as billed writes; inferred writes remain
marked as estimates.
- Yes for OpenAI Responses cost and savings dollars. The newly reported
writes are now priced at the write rate.
- WS turns without a reported write are now flagged inferred. That
removes the double-billed uncached slice, so WS cost totals go down for
older responses.
  - WS turns now count toward `cache_inferred_turns`.
- Kill switch / disable path: none needed; this follows the provider's
own usage.
- Rollback path: revert.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4237"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

## Review Readiness
- [ ] I have performed a self-review
- [ ] This PR is ready for human review

The maintainer has recorded independent qualification above. Author
readiness attestations have not been filled on the author's behalf.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-16 14:32:23 -07:00
f734c57341 fix(proxy): enforce budget limits on OpenAI and Gemini routes (#3579)
## Description

Enforces budget limit checks (`--budget`) on OpenAI and Gemini
generation routes. Previously, `CostTracker.check_budget()` was only
called within the Anthropic handler
(`headroom/proxy/handlers/anthropic.py`), allowing requests hitting
`/v1/chat/completions`, `/v1/responses`, and Gemini
`/v1beta/models/...:generateContent` to bypass budget enforcement
entirely.

Furthermore, enforces budget checks on every WebSocket `response.create`
frame before forwarding upstream (in addition to the upgrade handshake
preflight), ensuring that long-lived WebSocket sessions cannot continue
sending paid generations once the budget has been exhausted.

Refs #3374

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added `self.cost_tracker.check_budget()` check to `handle_openai_chat`
(`/v1/chat/completions`), raising `HTTPException(429)` when budget is
exceeded.
- Added `self.cost_tracker.check_budget()` check to
`handle_openai_responses` (`/v1/responses`), raising
`HTTPException(429)`.
- Added handshake preflight budget check to `handle_openai_responses_ws`
WebSocket upgrade, closing with code 1008 if budget is exhausted.
- Added per-frame budget enforcement in `handle_openai_responses_ws`
before forwarding any initial or late `response.create` message upstream
(including `_client_to_upstream` and HTTP fallback), consistently
closing with code 1008 and budget denial detail.
- Preserved `termination_cause = "budget_exceeded"` during WebSocket
teardown so metrics and logs accurately record the rejection.
- Added `self.cost_tracker.check_budget()` check to
`handle_gemini_generate_content`,
`handle_gemini_stream_generate_content`, and
`handle_google_cloudcode_stream`.
- Added integration tests in `tests/test_proxy_budget.py` covering zero
budget rejection, unlimited budget allowance, dynamic budget exhaustion
across all providers, WebSocket preflight rejection, and regression
tests proving late `response.create` frames on long-lived WebSockets are
blocked and not forwarded upstream once budget is exhausted.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
============================= test session starts ==============================
platform darwin -- Python 3.13.9, pytest-8.4.2, pluggy-1.5.0
rootdir: /Users/wilson/headroom
configfile: pyproject.toml
collected 10 items

tests/test_proxy_budget.py::test_zero_budget_blocks_openai_chat_completions PASSED [ 10%]
tests/test_proxy_budget.py::test_zero_budget_blocks_openai_responses PASSED [ 20%]
tests/test_proxy_budget.py::test_zero_budget_blocks_gemini_generate_content PASSED [ 30%]
tests/test_proxy_budget.py::test_zero_budget_blocks_gemini_stream_generate_content PASSED [ 40%]
tests/test_proxy_budget.py::test_zero_budget_blocks_anthropic_messages_parity PASSED [ 50%]
tests/test_proxy_budget.py::test_unlimited_budget_allows_all_generation_routes PASSED [ 60%]
tests/test_proxy_budget.py::test_zero_budget_websocket_preflight_rejected PASSED [ 70%]
tests/test_proxy_budget.py::test_dynamic_budget_exhaustion_blocks_all_providers PASSED [ 80%]
tests/test_proxy_budget.py::test_websocket_per_turn_budget_enforcement_blocks_late_response_create PASSED [ 90%]
tests/test_proxy_budget.py::test_websocket_turn_completion_usage_exhausts_budget_and_blocks_next_turn PASSED [100%]

======================== 10 passed, 1 warning in 3.57s =========================
```

## Real Behavior Proof

- Environment: macOS 15.3 (Darwin 24.6.0 arm64), Python 3.13.9, FastAPI
0.115.6, httpx 0.28.1, pytest 8.4.2
- Exact command / steps: Run `pytest tests/test_proxy_budget.py -v` to
boot test proxy client with ProxyConfig(budget_limit_usd=...) against
mock upstreams, issuing HTTP and WebSocket requests across routes and
multi-turn sessions.
- Observed result: HTTP routes reject with 429 'Budget exceeded for
daily period'; WebSocket preflight rejects with 1008; long-lived
WebSockets opened under budget allow turn 1, record spend/exhaust
budget, then reject late `response.create` frames with code 1008 and
halt upstream forwarding (`upstream.sent` length = 1).
- Not tested: Live external upstream cloud billing endpoints (mock
transport used to isolate proxy logic and prevent live API charges)

## Runtime Rollout Safety

- Rollout-managed feature(s): None
- Minimum rollout channel: stable
- Stable/default behavior changed: No (when --budget is unset,
budget_limit_usd defaults to None, preserving existing behavior)
- Kill switch / disable path: Omit --budget or set --budget ""
- Unsafe override required: None
- Qualification impact: None
- Rollback path: git revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Ensures behavioral parity across Anthropic, OpenAI (HTTP and WebSocket),
and Gemini generation routes when budget caps are configured.

---------

Signed-off-by: Johnny Wilson Dougherty <192861341+JohnnyWilson-Portfolio@users.noreply.github.com>
Co-authored-by: Johnny Wilson Dougherty <192861341+JohnnyWilson-Portfolio@users.noreply.github.com>
Co-authored-by: Kevin Costner <120246174+kevincostner17@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-09-21 10:01:54 -07:00