mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
feat/https-compose-caddy
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
f59c8ffcd8 |
fix(proxy): do not rebuild a failed turn-hook re-drive as HTTP 200 (#4398)
## Description
The Anthropic buffered turn-hook re-drive (`_turn_hook_call_model` in
`headroom/proxy/handlers/anthropic.py`) parsed the continuation body
without checking its status, so an upstream 4xx/5xx error envelope was
returned to the client as a 200 response. This raises on a non-2xx
status, as the CCR continuation path already does; `run_response_hooks`
catches the exception and keeps the original response.
Fixes #4394
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Raise `RuntimeError` in `_turn_hook_call_model` when the re-drive
status is not 2xx (5 lines).
- Add `test_anthropic_hook_redrive_error_is_not_rebuilt_as_200` to
`tests/test_turn_hook_usage.py`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
# new test, source change reverted
E KeyError: 'id'
1 failed, 14 deselected
# with the fix: tests/test_turn_hook_usage.py tests/test_turn_hooks.py tests/test_turn_hook_seams.py tests/test_openai_chat_turn_hooks.py
46 passed, 1 warning
ruff check: All checks passed!
ruff format --check: 2 files already formatted
```
## Real Behavior Proof
- Environment: Linux (WSL2), Python 3.12.15, the real `python -m
headroom.proxy.server` with `ANTHROPIC_TARGET_API_URL` at a local stub
(first POST 200, later POSTs 500), Rust core from the headroom-ai 0.41.0
wheel, a re-drive turn hook loaded through the
`headroom.proxy_extension` entry point
- Exact command / steps: one non-streaming `POST /v1/messages`, run with
the status check reverted and on this branch (details below)
- Observed result: reverted, the client gets HTTP 200 carrying the
upstream error envelope; this branch logs the hook `RuntimeError` and
the client gets HTTP 200 with the original first response
- Not tested: real Anthropic, a production hook, streaming, the OpenAI
handler paths, a Rust core built from this checkout
Details:
Setup: WSL2 Ubuntu (Linux 6.18 x86_64), Python 3.12.15, uvicorn 0.54.0,
fastapi 0.158.1, starlette 1.9.0. Worktree at PR head `e6d16cdc` (base
`e995eb9f`).
`headroom/_core.abi3.so` was copied (untracked, not committed) from the
official `headroom_ai-0.41.0-cp310-abi3-manylinux_2_28_x86_64.whl`
because the checkout has no built Rust core.
Stub upstream (`stub2.py`, 127.0.0.1:18794): the first POST to
`/v1/messages` returns 200 with a normal message (`id: msg_first`, text
"first answer"); every later POST returns 500
`{"type":"error","error":{"type":"api_error","message":"Internal server
error"}}`.
Proxy: the real server, `ANTHROPIC_TARGET_API_URL=http://127.0.0.1:18794
HEADROOM_PROXY_EXTENSIONS=redrive_ext PYTHONPATH=.:ext python -P -m
headroom.proxy.server --host 127.0.0.1 --port 18793 --no-cache
--no-rate-limit`.
The turn hook is loaded through the real extension entry point group
`headroom.proxy_extension` (a throwaway `redrive_ext` dist-info on
PYTHONPATH; its hook does `return await call_model(ctx.messages)` once
in `on_response`, the same as the PR's test hook). No hook ships in the
repo, so a hook that re-drives is the only way to reach this path.
Client (`hook_client.py`): one non-streaming `POST /v1/messages` with
`x-api-key: sk-ant-test`.
Before = `headroom/proxy/handlers/anthropic.py` with the PR's 5-line
status check reverted (`git apply -R`). After = PR head.
### Before (PR change reverted)
```
registered turn hook: proof_redrive (priority 100)
proxy extension installed: redrive_ext
client: HTTP 200 {"type": "error", "error": {"type": "api_error", "message": "Internal server error"}}
--- stub upstream received
STUB-UPSTREAM call #1 POST /v1/messages -> 200
STUB-UPSTREAM call #2 POST /v1/messages -> 500
STUB-UPSTREAM call #3 POST /v1/messages -> 500
STUB-UPSTREAM call #4 POST /v1/messages -> 500
```
The client gets HTTP 200 whose body is the upstream error JSON, and the
good first response (`msg_first`) is lost. A client that checks the
status code sees success.
### After (PR head
|
||
|
|
46306024e7 |
fix(openai): read Responses cache_write_tokens instead of inferring writes (#4237)
## Type of Change - [x] Bug fix (Responses usage accounting) ## Description OpenAI's Responses usage now reports billed cache writes as `input_tokens_details.cache_write_tokens`. They are billed at the write rate: $5/M on `gpt-5.6-sol`, against $4/M input. Every Headroom Responses path ignored that field and inferred writes as `input − cached` instead. As a result: - On HTTP, writes were flagged as inferred and dropped from pricing, so the written tokens were priced as uncached input at list. - On the Codex WebSocket path the inferred write was never flagged, so the uncached slice was billed twice: once as uncached and once as a write. Part of the savings redesign (step 2). The counterfactual record (#4236) needs the true write share `w = W/(U+W)`. ### The test that found it We ran Headroom Enterprise 0.4.3 in a Daytona sandbox with Codex CLI 0.162.1 on `gpt-5.6-sol` and the real GitHub MCP server, doing workshop task 1. We audited 13 calls against the provider's billing. - Every call reported `cache_write_tokens`. For call 2, usage was input 12,497, cached 9,568, write 2,926 (3 uncached). - PERF printed `cache_write=0` on every line. - Of the true $0.0850 saved, $0.0497 came from avoided cache **writes**, which core could not see. - /stats `total_cost_usd` undercounted the real bill by 8–10%. ## Changes Made - `_openai_responses_cache_split(input, cached, reported_write)` returns `(read, write, uncached, inferred)`. A reported write is used as-is; the write is inferred only when the field is absent. - Callers updated: - `_extract_responses_usage` (WS) returns a 6th `inferred` element. - The WS per-turn and session-end outcomes now pass `cache_inferred`. Before this they never set it. - The WS→HTTP fallback returns the flag. - HTTP `/v1/responses` keeps the turn-hook re-drive reads and adds the re-drives' reported writes (`TurnHookUsage.cache_write_tokens`). - The passthrough normalizer, the streaming usage parser (mapped onto the existing real-write branch) and `gateway_turn.normalize_usage` (Kong relays) also read the field. - Not changed: Chat Completions, which reports no write counter, and the `/v1/usage` relay, which doesn't map Responses-shaped details at all. That was already a gap for reads and is left as a follow-up. ## Before / after (audit replay, 13 Codex calls, current pricers) | | inferred (main) | reported (this PR) | provider truth | |---|---|---|---| | cost-tracker input cost | $0.2329 | $0.2649 | — (audit total $0.305 includes output) | | SavingsTracker compression $ (novel tokens) | $0.0434 | $0.0542 | $0.0850 total saved | The rest of the gap is re-sent history and the per-request record, which come in the follow-up PRs of the redesign. ## Testing - [x] New regression tests: - `test_ws_session_metrics_use_reported_cache_writes` (WS) - `test_fallback_returns_reported_cache_writes_as_real` - `test_responses_reported_cache_writes_include_the_redrive` (HTTP + turn hook) - `test_normalize_openai_responses_reported_cache_writes` (gateway) - [x] The existing WS test now pins `cache_inferred is True` when no write is reported. - [x] Passed (275, 14 skipped): `test_ws_http_fallback`, `test_openai_codex_ws_lifecycle`, `test_gateway_turn_unit`, `test_proxy_budget`, `test_turn_hook_usage`, `test_proxy_handler_helpers`, `test_proxy_openai_responses_integration`, `test_proxy_cache_telemetry`, `test_direct_chat_ccr_resolution`, `test_request_outcome`. - `test_turn_hook_usage` needs `LITELLM_LOCAL_MODEL_COST_MAP=True` locally: respx blocks litellm's remote price-map fetch. It fails the same way on main without it. - [x] `ruff check` / `ruff format --check` (0.16.8) and `mypy headroom` are clean. ### Test Output ```text Maintainer mixed-WebSocket-session probe: turn 1: input=100 read=75 write=25 uncached=25 cache_inferred=true turn 2: input=100 read=75 write=20 uncached=5 cache_inferred=false ``` The probe exercised the actual WebSocket handler using the existing lifecycle test harness. It verifies separate per-turn records for this mixed session; it does not establish behavior for every suppressed memory-tool re-drive. The prior qualified head completed hosted CI with 30 successful and 12 skipped checks. The current main-synced head is being qualified again. ## Real Behavior Proof - Environment: macOS, this branch; the audit's recorded provider usage for the 13 Codex calls. - Exact command / steps: replayed each call's usage through `CostTracker.record_tokens` and `estimate_request_savings_usd`, first with the inferred split (main) and then with the reported split (this PR). Script: `replay_c.py` in the audit scratchpad. - Observed result: input cost $0.2329 → $0.2649 and compression dollars $0.0434 → $0.0542, as in the table above. Call 2 now records read 9,568 / write 2,926 / uncached 3 instead of write 2,929 inferred. - Not tested: a live Codex session against this branch. The live re-run comes with the audit-eval PR of the redesign. ## Runtime Rollout Safety - Rollout-managed feature(s): no new rollout-managed feature switch. - Minimum rollout channel: existing Responses transport policy applies. - Unsafe override required: no. - Qualification impact: reported cache writes affect recorded cost and savings; hosted checks and human review remain required. - Stable/default behavior changed: Responses accounting prices explicitly reported writes as billed writes; inferred writes remain marked as estimates. - Yes for OpenAI Responses cost and savings dollars. The newly reported writes are now priced at the write rate. - WS turns without a reported write are now flagged inferred. That removes the double-billed uncached slice, so WS cost totals go down for older responses. - WS turns now count toward `cache_inferred_turns`. - Kill switch / disable path: none needed; this follows the provider's own usage. - Rollback path: revert. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4237" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ## Review Readiness - [ ] I have performed a self-review - [ ] This PR is ready for human review The maintainer has recorded independent qualification above. Author readiness attestations have not been filled on the author's behalf. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
3145242645 |
Unify savings attribution across stats, perf, metrics, and dashboard (#2976)
## Summary Adds a small provider-neutral savings attribution seam. Named sources can attach realized or projected token/USD deltas to a request without changing headline arithmetic or introducing private-package inventory into OSS. Also fixes the Anthropic buffered lifecycle so normal successful responses run response hooks, applies stream-safety filtering, includes tool savings in per-model perf totals, and surfaces the same breakdown in request logs, `/stats`, `headroom perf`, Prometheus, OTEL, and the dashboard. ## Why Request-local savings were split between canonical token deltas, process-global extension counters, and tool-only tags. This made correct headline totals possible while losing attribution in perf, recent requests, metrics, and the dashboard. Normal Anthropic responses also skipped response hooks unless CCR ran. ## Validation - 74 focused tests passed: turn hooks, OpenAI hook lifecycle, outcome funnel, perf formats, and tool-search repair - Ruff passes on all changed Python files - Existing compression-observability suite: 11 passed; 2 tokenizer-cache tests require network access to fetch the tiktoken vocabulary ## Compatibility No named private packages or private inventory are encoded in OSS. Existing hooks remain source-compatible because all new TurnContext fields are optional. --------- Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> |
||
|
|
675d13f08d |
fix(proxy/openai): run response hooks on Responses, and bill their re-drives (#2872)
The Responses path runs `run_request_hooks` but never `run_response_hooks` — only `handle_openai_chat` does. So a turn hook can shrink a Responses turn and then never be asked to resolve what the model did about it: the model's injected tool call goes straight to a client that has no such tool. That asymmetry is why tool-belt deferral has to be disabled wholesale on the Responses API, which is the surface Codex uses. ## 1. Wire the response side Mirrors the chat-completions block. **Buffered path only**, for the same reason CCR already forces `stream:false` when it needs to intercept: you cannot re-drive a turn whose bytes are already flowing. ## 2. Honour `stream_safe_only` on the Responses request path It was the one hook call site that ignored the flag. A re-driving hook would run its shrink on a streamed turn and then have no response side to finish it — latent until (1) lands, live afterwards. `stream` is not a parameter of `_compress_openai_responses_payload`, but the payload it is compressing carries the flag. It is read **before** CCR may force `stream:false` further down, so this is the client's request rather than the effective one — conservative in the safe direction: at worst a CCR-buffered turn misses a saving, never a stranded tool call. Fold-only hooks that declare `stream_safe = True` are unaffected. ## 3. Bill what the re-drives cost Both handlers read usage from the **final** upstream response, so every intermediate call a hook made was free as far as Headroom was concerned. For a token-saving feature that is not a rounding error. A tool-search reload is a whole extra model call; counting only the last one lets the feature hide its own overhead behind the saving it is claiming, and the numbers come out better than the truth. `TurnHookUsage` accumulates input/output/cached across re-drives; both HTTP paths fold it into their totals. The two surfaces report the same three quantities under different names (`prompt_tokens` vs `input_tokens`), so the key pair is passed in. Expect measured cost to go **up** and savings percentage to go **down** on any deployment running a re-driving hook. That is the correction, not a regression. ## Also: restore the body after the hooks A re-drive rewrites `body[input]` / `body[messages]` / `body[tools]` so the next upstream call carries the hook's turn. Everything downstream — CCR's `_responses_input_to_items(body["input"])`, usage accounting, observability — is describing the request the *client* made, not the proxy's internal detour. Without the restore, a turn that both reloaded a tool and hit CCR retrieval hands CCR the proxy's synthetic items. The chat path had the same leak (`body["messages"]` stayed rewritten); both are fixed the same way. ## Known gap A re-drive on the custom backend path (`send_openai_message`) is still not folded into that request's accounting — its usage is recorded elsewhere. Commented at the call site rather than silently skipped. ## Blast radius **Inert unless a turn hook is registered**, so no behaviour change for a stock OSS proxy. `TurnHookUsage` starts at zero and stays there on every path that does not re-drive. ## Verification - `tests/test_turn_hook_usage.py` — 5 new tests: per-surface key names, accumulation across rounds, negative counts floored not subtracted, and that an unreadable shape still counts the call (a silent zero there looks exactly like "the hook cost nothing") - 434 passing across `turn_hook`, `extension`, `tool_search`, `responses` and `openai_chat` suites - `ruff check` + `ruff format` clean 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |