mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 18:41:07 +08:00
feat/https-compose-caddy
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
46306024e7 |
fix(openai): read Responses cache_write_tokens instead of inferring writes (#4237)
## Type of Change - [x] Bug fix (Responses usage accounting) ## Description OpenAI's Responses usage now reports billed cache writes as `input_tokens_details.cache_write_tokens`. They are billed at the write rate: $5/M on `gpt-5.6-sol`, against $4/M input. Every Headroom Responses path ignored that field and inferred writes as `input − cached` instead. As a result: - On HTTP, writes were flagged as inferred and dropped from pricing, so the written tokens were priced as uncached input at list. - On the Codex WebSocket path the inferred write was never flagged, so the uncached slice was billed twice: once as uncached and once as a write. Part of the savings redesign (step 2). The counterfactual record (#4236) needs the true write share `w = W/(U+W)`. ### The test that found it We ran Headroom Enterprise 0.4.3 in a Daytona sandbox with Codex CLI 0.162.1 on `gpt-5.6-sol` and the real GitHub MCP server, doing workshop task 1. We audited 13 calls against the provider's billing. - Every call reported `cache_write_tokens`. For call 2, usage was input 12,497, cached 9,568, write 2,926 (3 uncached). - PERF printed `cache_write=0` on every line. - Of the true $0.0850 saved, $0.0497 came from avoided cache **writes**, which core could not see. - /stats `total_cost_usd` undercounted the real bill by 8–10%. ## Changes Made - `_openai_responses_cache_split(input, cached, reported_write)` returns `(read, write, uncached, inferred)`. A reported write is used as-is; the write is inferred only when the field is absent. - Callers updated: - `_extract_responses_usage` (WS) returns a 6th `inferred` element. - The WS per-turn and session-end outcomes now pass `cache_inferred`. Before this they never set it. - The WS→HTTP fallback returns the flag. - HTTP `/v1/responses` keeps the turn-hook re-drive reads and adds the re-drives' reported writes (`TurnHookUsage.cache_write_tokens`). - The passthrough normalizer, the streaming usage parser (mapped onto the existing real-write branch) and `gateway_turn.normalize_usage` (Kong relays) also read the field. - Not changed: Chat Completions, which reports no write counter, and the `/v1/usage` relay, which doesn't map Responses-shaped details at all. That was already a gap for reads and is left as a follow-up. ## Before / after (audit replay, 13 Codex calls, current pricers) | | inferred (main) | reported (this PR) | provider truth | |---|---|---|---| | cost-tracker input cost | $0.2329 | $0.2649 | — (audit total $0.305 includes output) | | SavingsTracker compression $ (novel tokens) | $0.0434 | $0.0542 | $0.0850 total saved | The rest of the gap is re-sent history and the per-request record, which come in the follow-up PRs of the redesign. ## Testing - [x] New regression tests: - `test_ws_session_metrics_use_reported_cache_writes` (WS) - `test_fallback_returns_reported_cache_writes_as_real` - `test_responses_reported_cache_writes_include_the_redrive` (HTTP + turn hook) - `test_normalize_openai_responses_reported_cache_writes` (gateway) - [x] The existing WS test now pins `cache_inferred is True` when no write is reported. - [x] Passed (275, 14 skipped): `test_ws_http_fallback`, `test_openai_codex_ws_lifecycle`, `test_gateway_turn_unit`, `test_proxy_budget`, `test_turn_hook_usage`, `test_proxy_handler_helpers`, `test_proxy_openai_responses_integration`, `test_proxy_cache_telemetry`, `test_direct_chat_ccr_resolution`, `test_request_outcome`. - `test_turn_hook_usage` needs `LITELLM_LOCAL_MODEL_COST_MAP=True` locally: respx blocks litellm's remote price-map fetch. It fails the same way on main without it. - [x] `ruff check` / `ruff format --check` (0.16.8) and `mypy headroom` are clean. ### Test Output ```text Maintainer mixed-WebSocket-session probe: turn 1: input=100 read=75 write=25 uncached=25 cache_inferred=true turn 2: input=100 read=75 write=20 uncached=5 cache_inferred=false ``` The probe exercised the actual WebSocket handler using the existing lifecycle test harness. It verifies separate per-turn records for this mixed session; it does not establish behavior for every suppressed memory-tool re-drive. The prior qualified head completed hosted CI with 30 successful and 12 skipped checks. The current main-synced head is being qualified again. ## Real Behavior Proof - Environment: macOS, this branch; the audit's recorded provider usage for the 13 Codex calls. - Exact command / steps: replayed each call's usage through `CostTracker.record_tokens` and `estimate_request_savings_usd`, first with the inferred split (main) and then with the reported split (this PR). Script: `replay_c.py` in the audit scratchpad. - Observed result: input cost $0.2329 → $0.2649 and compression dollars $0.0434 → $0.0542, as in the table above. Call 2 now records read 9,568 / write 2,926 / uncached 3 instead of write 2,929 inferred. - Not tested: a live Codex session against this branch. The live re-run comes with the audit-eval PR of the redesign. ## Runtime Rollout Safety - Rollout-managed feature(s): no new rollout-managed feature switch. - Minimum rollout channel: existing Responses transport policy applies. - Unsafe override required: no. - Qualification impact: reported cache writes affect recorded cost and savings; hosted checks and human review remain required. - Stable/default behavior changed: Responses accounting prices explicitly reported writes as billed writes; inferred writes remain marked as estimates. - Yes for OpenAI Responses cost and savings dollars. The newly reported writes are now priced at the write rate. - WS turns without a reported write are now flagged inferred. That removes the double-billed uncached slice, so WS cost totals go down for older responses. - WS turns now count toward `cache_inferred_turns`. - Kill switch / disable path: none needed; this follows the provider's own usage. - Rollback path: revert. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4237" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ## Review Readiness - [ ] I have performed a self-review - [ ] This PR is ready for human review The maintainer has recorded independent qualification above. Author readiness attestations have not been filled on the author's behalf. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
f734c57341 |
fix(proxy): enforce budget limits on OpenAI and Gemini routes (#3579)
## Description Enforces budget limit checks (`--budget`) on OpenAI and Gemini generation routes. Previously, `CostTracker.check_budget()` was only called within the Anthropic handler (`headroom/proxy/handlers/anthropic.py`), allowing requests hitting `/v1/chat/completions`, `/v1/responses`, and Gemini `/v1beta/models/...:generateContent` to bypass budget enforcement entirely. Furthermore, enforces budget checks on every WebSocket `response.create` frame before forwarding upstream (in addition to the upgrade handshake preflight), ensuring that long-lived WebSocket sessions cannot continue sending paid generations once the budget has been exhausted. Refs #3374 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `self.cost_tracker.check_budget()` check to `handle_openai_chat` (`/v1/chat/completions`), raising `HTTPException(429)` when budget is exceeded. - Added `self.cost_tracker.check_budget()` check to `handle_openai_responses` (`/v1/responses`), raising `HTTPException(429)`. - Added handshake preflight budget check to `handle_openai_responses_ws` WebSocket upgrade, closing with code 1008 if budget is exhausted. - Added per-frame budget enforcement in `handle_openai_responses_ws` before forwarding any initial or late `response.create` message upstream (including `_client_to_upstream` and HTTP fallback), consistently closing with code 1008 and budget denial detail. - Preserved `termination_cause = "budget_exceeded"` during WebSocket teardown so metrics and logs accurately record the rejection. - Added `self.cost_tracker.check_budget()` check to `handle_gemini_generate_content`, `handle_gemini_stream_generate_content`, and `handle_google_cloudcode_stream`. - Added integration tests in `tests/test_proxy_budget.py` covering zero budget rejection, unlimited budget allowance, dynamic budget exhaustion across all providers, WebSocket preflight rejection, and regression tests proving late `response.create` frames on long-lived WebSockets are blocked and not forwarded upstream once budget is exhausted. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text ============================= test session starts ============================== platform darwin -- Python 3.13.9, pytest-8.4.2, pluggy-1.5.0 rootdir: /Users/wilson/headroom configfile: pyproject.toml collected 10 items tests/test_proxy_budget.py::test_zero_budget_blocks_openai_chat_completions PASSED [ 10%] tests/test_proxy_budget.py::test_zero_budget_blocks_openai_responses PASSED [ 20%] tests/test_proxy_budget.py::test_zero_budget_blocks_gemini_generate_content PASSED [ 30%] tests/test_proxy_budget.py::test_zero_budget_blocks_gemini_stream_generate_content PASSED [ 40%] tests/test_proxy_budget.py::test_zero_budget_blocks_anthropic_messages_parity PASSED [ 50%] tests/test_proxy_budget.py::test_unlimited_budget_allows_all_generation_routes PASSED [ 60%] tests/test_proxy_budget.py::test_zero_budget_websocket_preflight_rejected PASSED [ 70%] tests/test_proxy_budget.py::test_dynamic_budget_exhaustion_blocks_all_providers PASSED [ 80%] tests/test_proxy_budget.py::test_websocket_per_turn_budget_enforcement_blocks_late_response_create PASSED [ 90%] tests/test_proxy_budget.py::test_websocket_turn_completion_usage_exhausts_budget_and_blocks_next_turn PASSED [100%] ======================== 10 passed, 1 warning in 3.57s ========================= ``` ## Real Behavior Proof - Environment: macOS 15.3 (Darwin 24.6.0 arm64), Python 3.13.9, FastAPI 0.115.6, httpx 0.28.1, pytest 8.4.2 - Exact command / steps: Run `pytest tests/test_proxy_budget.py -v` to boot test proxy client with ProxyConfig(budget_limit_usd=...) against mock upstreams, issuing HTTP and WebSocket requests across routes and multi-turn sessions. - Observed result: HTTP routes reject with 429 'Budget exceeded for daily period'; WebSocket preflight rejects with 1008; long-lived WebSockets opened under budget allow turn 1, record spend/exhaust budget, then reject late `response.create` frames with code 1008 and halt upstream forwarding (`upstream.sent` length = 1). - Not tested: Live external upstream cloud billing endpoints (mock transport used to isolate proxy logic and prevent live API charges) ## Runtime Rollout Safety - Rollout-managed feature(s): None - Minimum rollout channel: stable - Stable/default behavior changed: No (when --budget is unset, budget_limit_usd defaults to None, preserving existing behavior) - Kill switch / disable path: Omit --budget or set --budget "" - Unsafe override required: None - Qualification impact: None - Rollback path: git revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Ensures behavioral parity across Anthropic, OpenAI (HTTP and WebSocket), and Gemini generation routes when budget caps are configured. --------- Signed-off-by: Johnny Wilson Dougherty <192861341+JohnnyWilson-Portfolio@users.noreply.github.com> Co-authored-by: Johnny Wilson Dougherty <192861341+JohnnyWilson-Portfolio@users.noreply.github.com> Co-authored-by: Kevin Costner <120246174+kevincostner17@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |