Commit Graph
4 Commits
Author SHA1 Message Date
f59c8ffcd8 fix(proxy): do not rebuild a failed turn-hook re-drive as HTTP 200 (#4398)
## Description

The Anthropic buffered turn-hook re-drive (`_turn_hook_call_model` in
`headroom/proxy/handlers/anthropic.py`) parsed the continuation body
without checking its status, so an upstream 4xx/5xx error envelope was
returned to the client as a 200 response. This raises on a non-2xx
status, as the CCR continuation path already does; `run_response_hooks`
catches the exception and keeps the original response.

Fixes #4394

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Raise `RuntimeError` in `_turn_hook_call_model` when the re-drive
status is not 2xx (5 lines).
- Add `test_anthropic_hook_redrive_error_is_not_rebuilt_as_200` to
`tests/test_turn_hook_usage.py`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
# new test, source change reverted
E   KeyError: 'id'
1 failed, 14 deselected

# with the fix: tests/test_turn_hook_usage.py tests/test_turn_hooks.py tests/test_turn_hook_seams.py tests/test_openai_chat_turn_hooks.py
46 passed, 1 warning

ruff check: All checks passed!
ruff format --check: 2 files already formatted
```

## Real Behavior Proof

- Environment: Linux (WSL2), Python 3.12.15, the real `python -m
headroom.proxy.server` with `ANTHROPIC_TARGET_API_URL` at a local stub
(first POST 200, later POSTs 500), Rust core from the headroom-ai 0.41.0
wheel, a re-drive turn hook loaded through the
`headroom.proxy_extension` entry point
- Exact command / steps: one non-streaming `POST /v1/messages`, run with
the status check reverted and on this branch (details below)
- Observed result: reverted, the client gets HTTP 200 carrying the
upstream error envelope; this branch logs the hook `RuntimeError` and
the client gets HTTP 200 with the original first response
- Not tested: real Anthropic, a production hook, streaming, the OpenAI
handler paths, a Rust core built from this checkout

Details:

Setup: WSL2 Ubuntu (Linux 6.18 x86_64), Python 3.12.15, uvicorn 0.54.0,
fastapi 0.158.1, starlette 1.9.0. Worktree at PR head `e6d16cdc` (base
`e995eb9f`).
`headroom/_core.abi3.so` was copied (untracked, not committed) from the
official `headroom_ai-0.41.0-cp310-abi3-manylinux_2_28_x86_64.whl`
because the checkout has no built Rust core.

Stub upstream (`stub2.py`, 127.0.0.1:18794): the first POST to
`/v1/messages` returns 200 with a normal message (`id: msg_first`, text
"first answer"); every later POST returns 500
`{"type":"error","error":{"type":"api_error","message":"Internal server
error"}}`.
Proxy: the real server, `ANTHROPIC_TARGET_API_URL=http://127.0.0.1:18794
HEADROOM_PROXY_EXTENSIONS=redrive_ext PYTHONPATH=.:ext python -P -m
headroom.proxy.server --host 127.0.0.1 --port 18793 --no-cache
--no-rate-limit`.
The turn hook is loaded through the real extension entry point group
`headroom.proxy_extension` (a throwaway `redrive_ext` dist-info on
PYTHONPATH; its hook does `return await call_model(ctx.messages)` once
in `on_response`, the same as the PR's test hook). No hook ships in the
repo, so a hook that re-drives is the only way to reach this path.
Client (`hook_client.py`): one non-streaming `POST /v1/messages` with
`x-api-key: sk-ant-test`.

Before = `headroom/proxy/handlers/anthropic.py` with the PR's 5-line
status check reverted (`git apply -R`). After = PR head.

### Before (PR change reverted)

```
registered turn hook: proof_redrive (priority 100)
proxy extension installed: redrive_ext
client: HTTP 200 {"type": "error", "error": {"type": "api_error", "message": "Internal server error"}}
--- stub upstream received
STUB-UPSTREAM call #1 POST /v1/messages -> 200
STUB-UPSTREAM call #2 POST /v1/messages -> 500
STUB-UPSTREAM call #3 POST /v1/messages -> 500
STUB-UPSTREAM call #4 POST /v1/messages -> 500
```

The client gets HTTP 200 whose body is the upstream error JSON, and the
good first response (`msg_first`) is lost. A client that checks the
status code sees success.

### After (PR head e6d16cdc)

```
client: HTTP 200 {"id": "msg_first", "type": "message", "role": "assistant", "model": "claude-sonnet-4-5", "content": [{"type": "text", "text": "first answer"}], "stop_reason": "end_turn", "usage": {"input_tokens": 100, "output_tokens": 10}}
--- proxy log
ERROR - turn hook 'proof_redrive' on_response failed
  File ".../headroom/proxy/handlers/anthropic.py", line 4993, in _turn_hook_call_model
RuntimeError: turn hook re-drive returned non-success status 500
--- stub upstream received
STUB-UPSTREAM call #1 POST /v1/messages -> 200
STUB-UPSTREAM call #2 POST /v1/messages -> 500
STUB-UPSTREAM call #3 POST /v1/messages -> 500
STUB-UPSTREAM call #4 POST /v1/messages -> 500
```

The failed re-drive is logged by the existing hook runner
(`run_response_hooks` logs and skips a hook that raises) and the client
receives the original successful response. The upstream sees the same 4
calls in both runs (the 3 later ones are the proxy's own retries of the
500).

### Commands

```
python -P stub2.py 18794 hook.stub.log &
HEADROOM_PROXY_EXTENSIONS=redrive_ext ANTHROPIC_TARGET_API_URL=http://127.0.0.1:18794 PYTHONPATH=.:ext \
  python -P -m headroom.proxy.server --host 127.0.0.1 --port 18793 --no-cache --no-rate-limit &
python -P hook_client.py 18793
```

Scripts and raw outputs: `/var/tmp/cb-pub/proof/` (WSL): `run_hook.sh`,
`stub2.py`, `hook_client.py`, `ext/`, `before_hook.out`,
`after_hook.out`.

### Not tested

- Real Anthropic API (stub upstream only) and other failure statuses
than 500.
- A production turn hook; the hook here is a minimal stand-in that
re-drives once.
- Streaming requests (hooks that may re-drive are buffered-only).
- The OpenAI handlers' hook paths.
- The Rust core is the 0.41.0 wheel's, not one built from this checkout.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only when a turn hook re-drive fails
upstream (client now gets the original response instead of an error
envelope as 200)
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none known
- Rollback path: revert the commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Only the four listed test files were run, not the whole suite. Produced
with AI assistance (Claude Code) operated by me (HetCreep); the
reproduction was run on my machine on a clone of the repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4398"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-24 08:37:56 -07:00
46306024e7 fix(openai): read Responses cache_write_tokens instead of inferring writes (#4237)
## Type of Change
- [x] Bug fix (Responses usage accounting)

## Description

OpenAI's Responses usage now reports billed cache writes as
`input_tokens_details.cache_write_tokens`. They are billed at the write
rate: $5/M on `gpt-5.6-sol`, against $4/M input. Every Headroom
Responses path ignored that field and inferred writes as `input −
cached` instead. As a result:
- On HTTP, writes were flagged as inferred and dropped from pricing, so
the written tokens were priced as uncached input at list.
- On the Codex WebSocket path the inferred write was never flagged, so
the uncached slice was billed twice: once as uncached and once as a
write.

Part of the savings redesign (step 2). The counterfactual record (#4236)
needs the true write share `w = W/(U+W)`.

### The test that found it

We ran Headroom Enterprise 0.4.3 in a Daytona sandbox with Codex CLI
0.162.1 on `gpt-5.6-sol` and the real GitHub MCP server, doing workshop
task 1. We audited 13 calls against the provider's billing.
- Every call reported `cache_write_tokens`. For call 2, usage was input
12,497, cached 9,568, write 2,926 (3 uncached).
- PERF printed `cache_write=0` on every line.
- Of the true $0.0850 saved, $0.0497 came from avoided cache **writes**,
which core could not see.
- /stats `total_cost_usd` undercounted the real bill by 8–10%.

## Changes Made

- `_openai_responses_cache_split(input, cached, reported_write)` returns
`(read, write, uncached, inferred)`. A reported write is used as-is; the
write is inferred only when the field is absent.
- Callers updated:
  - `_extract_responses_usage` (WS) returns a 6th `inferred` element.
- The WS per-turn and session-end outcomes now pass `cache_inferred`.
Before this they never set it.
  - The WS→HTTP fallback returns the flag.
- HTTP `/v1/responses` keeps the turn-hook re-drive reads and adds the
re-drives' reported writes (`TurnHookUsage.cache_write_tokens`).
- The passthrough normalizer, the streaming usage parser (mapped onto
the existing real-write branch) and `gateway_turn.normalize_usage` (Kong
relays) also read the field.
- Not changed: Chat Completions, which reports no write counter, and the
`/v1/usage` relay, which doesn't map Responses-shaped details at all.
That was already a gap for reads and is left as a follow-up.

## Before / after (audit replay, 13 Codex calls, current pricers)

| | inferred (main) | reported (this PR) | provider truth |
|---|---|---|---|
| cost-tracker input cost | $0.2329 | $0.2649 | — (audit total $0.305
includes output) |
| SavingsTracker compression $ (novel tokens) | $0.0434 | $0.0542 |
$0.0850 total saved |

The rest of the gap is re-sent history and the per-request record, which
come in the follow-up PRs of the redesign.

## Testing

- [x] New regression tests:
  - `test_ws_session_metrics_use_reported_cache_writes` (WS)
  - `test_fallback_returns_reported_cache_writes_as_real`
- `test_responses_reported_cache_writes_include_the_redrive` (HTTP +
turn hook)
  - `test_normalize_openai_responses_reported_cache_writes` (gateway)
- [x] The existing WS test now pins `cache_inferred is True` when no
write is reported.
- [x] Passed (275, 14 skipped): `test_ws_http_fallback`,
`test_openai_codex_ws_lifecycle`, `test_gateway_turn_unit`,
`test_proxy_budget`, `test_turn_hook_usage`,
`test_proxy_handler_helpers`, `test_proxy_openai_responses_integration`,
`test_proxy_cache_telemetry`, `test_direct_chat_ccr_resolution`,
`test_request_outcome`.
- `test_turn_hook_usage` needs `LITELLM_LOCAL_MODEL_COST_MAP=True`
locally: respx blocks litellm's remote price-map fetch. It fails the
same way on main without it.
- [x] `ruff check` / `ruff format --check` (0.16.8) and `mypy headroom`
are clean.


### Test Output

```text
Maintainer mixed-WebSocket-session probe:
turn 1: input=100 read=75 write=25 uncached=25 cache_inferred=true
turn 2: input=100 read=75 write=20 uncached=5 cache_inferred=false
```

The probe exercised the actual WebSocket handler using the existing
lifecycle test harness. It verifies separate per-turn records for this
mixed session; it does not establish behavior for every suppressed
memory-tool re-drive. The prior qualified head completed hosted CI with
30 successful and 12 skipped checks. The current main-synced head is
being qualified again.

## Real Behavior Proof

- Environment: macOS, this branch; the audit's recorded provider usage
for the 13 Codex calls.
- Exact command / steps: replayed each call's usage through
`CostTracker.record_tokens` and `estimate_request_savings_usd`, first
with the inferred split (main) and then with the reported split (this
PR). Script: `replay_c.py` in the audit scratchpad.
- Observed result: input cost $0.2329 → $0.2649 and compression dollars
$0.0434 → $0.0542, as in the table above. Call 2 now records read 9,568
/ write 2,926 / uncached 3 instead of write 2,929 inferred.
- Not tested: a live Codex session against this branch. The live re-run
comes with the audit-eval PR of the redesign.

## Runtime Rollout Safety
- Rollout-managed feature(s): no new rollout-managed feature switch.
- Minimum rollout channel: existing Responses transport policy applies.
- Unsafe override required: no.
- Qualification impact: reported cache writes affect recorded cost and
savings; hosted checks and human review remain required.

- Stable/default behavior changed: Responses accounting prices
explicitly reported writes as billed writes; inferred writes remain
marked as estimates.
- Yes for OpenAI Responses cost and savings dollars. The newly reported
writes are now priced at the write rate.
- WS turns without a reported write are now flagged inferred. That
removes the double-billed uncached slice, so WS cost totals go down for
older responses.
  - WS turns now count toward `cache_inferred_turns`.
- Kill switch / disable path: none needed; this follows the provider's
own usage.
- Rollback path: revert.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4237"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

## Review Readiness
- [ ] I have performed a self-review
- [ ] This PR is ready for human review

The maintainer has recorded independent qualification above. Author
readiness attestations have not been filled on the author's behalf.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-16 14:32:23 -07:00
Tejas ChopraandTejas Chopra 3145242645 Unify savings attribution across stats, perf, metrics, and dashboard (#2976)
## Summary

Adds a small provider-neutral savings attribution seam. Named sources
can attach realized or projected token/USD deltas to a request without
changing headline arithmetic or introducing private-package inventory
into OSS.

Also fixes the Anthropic buffered lifecycle so normal successful
responses run response hooks, applies stream-safety filtering, includes
tool savings in per-model perf totals, and surfaces the same breakdown
in request logs, `/stats`, `headroom perf`, Prometheus, OTEL, and the
dashboard.

## Why

Request-local savings were split between canonical token deltas,
process-global extension counters, and tool-only tags. This made correct
headline totals possible while losing attribution in perf, recent
requests, metrics, and the dashboard. Normal Anthropic responses also
skipped response hooks unless CCR ran.

## Validation

- 74 focused tests passed: turn hooks, OpenAI hook lifecycle, outcome
funnel, perf formats, and tool-search repair
- Ruff passes on all changed Python files
- Existing compression-observability suite: 11 passed; 2 tokenizer-cache
tests require network access to fetch the tiktoken vocabulary

## Compatibility

No named private packages or private inventory are encoded in OSS.
Existing hooks remain source-compatible because all new TurnContext
fields are optional.

---------

Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
2026-08-13 17:13:23 -07:00
Tejas ChopraandClaude Opus 5 675d13f08d fix(proxy/openai): run response hooks on Responses, and bill their re-drives (#2872)
The Responses path runs `run_request_hooks` but never
`run_response_hooks` —
only `handle_openai_chat` does. So a turn hook can shrink a Responses
turn and
then never be asked to resolve what the model did about it: the model's
injected
tool call goes straight to a client that has no such tool.

That asymmetry is why tool-belt deferral has to be disabled wholesale on
the
Responses API, which is the surface Codex uses.

## 1. Wire the response side

Mirrors the chat-completions block. **Buffered path only**, for the same
reason
CCR already forces `stream:false` when it needs to intercept: you cannot
re-drive a turn whose bytes are already flowing.

## 2. Honour `stream_safe_only` on the Responses request path

It was the one hook call site that ignored the flag. A re-driving hook
would run
its shrink on a streamed turn and then have no response side to finish
it —
latent until (1) lands, live afterwards.

`stream` is not a parameter of `_compress_openai_responses_payload`, but
the
payload it is compressing carries the flag. It is read **before** CCR
may force
`stream:false` further down, so this is the client's request rather than
the
effective one — conservative in the safe direction: at worst a
CCR-buffered turn
misses a saving, never a stranded tool call.

Fold-only hooks that declare `stream_safe = True` are unaffected.

## 3. Bill what the re-drives cost

Both handlers read usage from the **final** upstream response, so every
intermediate call a hook made was free as far as Headroom was concerned.

For a token-saving feature that is not a rounding error. A tool-search
reload is
a whole extra model call; counting only the last one lets the feature
hide its
own overhead behind the saving it is claiming, and the numbers come out
better
than the truth.

`TurnHookUsage` accumulates input/output/cached across re-drives; both
HTTP
paths fold it into their totals. The two surfaces report the same three
quantities under different names (`prompt_tokens` vs `input_tokens`), so
the key
pair is passed in.

Expect measured cost to go **up** and savings percentage to go **down**
on any
deployment running a re-driving hook. That is the correction, not a
regression.

## Also: restore the body after the hooks

A re-drive rewrites `body[input]` / `body[messages]` / `body[tools]` so
the next
upstream call carries the hook's turn. Everything downstream — CCR's
`_responses_input_to_items(body["input"])`, usage accounting,
observability — is
describing the request the *client* made, not the proxy's internal
detour.

Without the restore, a turn that both reloaded a tool and hit CCR
retrieval
hands CCR the proxy's synthetic items. The chat path had the same leak
(`body["messages"]` stayed rewritten); both are fixed the same way.

## Known gap

A re-drive on the custom backend path (`send_openai_message`) is still
not
folded into that request's accounting — its usage is recorded elsewhere.
Commented at the call site rather than silently skipped.

## Blast radius

**Inert unless a turn hook is registered**, so no behaviour change for a
stock
OSS proxy. `TurnHookUsage` starts at zero and stays there on every path
that
does not re-drive.

## Verification

- `tests/test_turn_hook_usage.py` — 5 new tests: per-surface key names,
accumulation across rounds, negative counts floored not subtracted, and
that
an unreadable shape still counts the call (a silent zero there looks
exactly
  like "the hook cost nothing")
- 434 passing across `turn_hook`, `extension`, `tool_search`,
`responses` and
  `openai_chat` suites
- `ruff check` + `ruff format` clean

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 10:09:24 -07:00