mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
feat/https-compose-caddy
14
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
d6df127d57 |
feat(compress): accept Gemini and Bedrock Converse bodies on the gateway turn (#4679)
## Description
`/v1/compress` in gateway mode (a request with a `gateway` block) now
accepts two more native request bodies:
- Gemini `generateContent`
- Bedrock Converse
It compresses their text slots and returns `body` in the same native
shape. This lets a gateway like LiteLLM compress native Gemini and
Bedrock passthrough traffic, which it couldn't before.
`config.mode: "ccr"` is rejected for these shapes with a 400. Every
other mode is marker-free for them.
(Was stacked on #4617, which has merged; the branch now carries current
`main`.)
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
### How
This works the same way `gateway_responses` does for Codex: it builds a
view instead of translating the body.
- Every compressible text slot becomes one chat-shaped pseudo message.
- Compressed text is written back by its path in the body.
- Everything else stays byte-identical: `thoughtSignature`, thought
parts, `functionCall`, `inlineData`, `toolUse`, `cachePoint`,
`reasoningContent`, `tools`/`toolConfig`.
- Tool calls appear in the view, so tool-name rules still apply. `Read`
stays verbatim, for example.
- A JSON slot is only written back if it still parses to the same JSON
type.
- If the compressed list doesn't line up with the slots, the original
transcript is forwarded and a warning is logged.
### CCR on native bodies (review fix)
A native body can't carry `headroom_retrieve`. Its `tools`/`toolConfig`
are forwarded untouched, and `create_ccr_tool_definition` only builds
chat, Anthropic and Responses shapes. It also never gets a re-drive. So
a CCR marker in a native body points at content the model can never
fetch. Before this fix, `config.mode: "ccr"` on a Gemini or Converse
gateway turn did exactly that: log output came back as a marker with
`ccr_hashes: [<hash>]` and `obligations: []`.
- `handle_compress` now answers `ccr` on a native view with `400
invalid_request`. The message names the shape, explains why, and lists
the supported modes.
- The check runs where the mode is chosen, right after `attach_view` has
recorded the shape. That is before any compression, session work or CCR
store write.
- **The rejection does not depend on `gateway.can_redrive`.** Even a
gateway that can re-drive has no way to put the retrieval tool into a
native request, so there is no capability under which native `ccr`
works.
- **Without a `gateway` block nothing changes.** A Gemini body is
already a 400 (no `messages`). A Converse body goes through the plain
messages path, where its untyped blocks are opaque: 0 tokens saved, no
markers in any mode, `ccr` included (checked). So the rule is
consistent: reject `ccr` wherever Headroom reads a native shape.
- The bypass header still returns the body untouched with no
compression, in any mode.
- I checked every other mode on the same marker-producing input
(default, `lossless`, `lossy_inline`, `lossless_then_lossy`): no marker
and no hash in any of them. These modes build pipelines with
`ccr_inject_marker=False, ccr_enabled=False`. Stale-Read notices are
hashless in marker-free modes since #4678.
- **Second defect, same family:** with `HEADROOM_DEDUPE=1`, cross-turn
dedup wrote `[↑NL same as msg 4]` into a Converse tool result. That `4`
counts view messages (system blocks, tool calls, result slots), not the
caller's `messages`, so it named the wrong message. Native views now set
`cross_turn_dedup_recoverable=False`, as `lossless` mode already did.
- The `not self.native_shape` guards on CCR arming and re-drive in
`GatewayTurn.finish` stay as defence in depth.
### Caller contract
**Gemini**
- Send top-level `contents` as a list, with no `messages` or `input`.
- `systemInstruction`/`system_instruction`, `tools` and
`generationConfig` are optional and pass through.
**Converse**
- Send `messages` where every message has role `user` or `assistant`,
and every content block has no `type` key.
- At least one block must use a Converse key: `text`, `toolUse`,
`toolResult`, `cachePoint`, and so on.
- `system` is optional, as a list of blocks.
**Both shapes**
- `config.mode`: omit it (marker-free default), or use `lossless`,
`lossy_inline` or `lossless_then_lossy`. `ccr` is a 400.
- Only gateway turns get a view. Without a `gateway` block the endpoint
behaves exactly as before. A client-sent `_headroom_native_view` key is
dropped, and `BODY_CONTROL_KEYS` strips the real one before the body is
returned.
- `config.frozen_message_count` and `config.protect_recent` count native
messages: Gemini `contents` entries and Converse `messages`.
`session.frozen_message_count` comes back in native units. The
`/v1/usage` reply's `frozen_message_count` still counts view messages,
and the docs say so.
- No re-drive is ever armed, so `obligations` never contains `redrive`,
and request hooks run stream-safe-only. Tool compaction and output
steering are skipped.
- Relayed usage can be:
- Gemini `usageMetadata`, where `promptTokenCount` already includes
`cachedContentTokenCount`.
- Converse `usage`, where `inputTokens` excludes `cacheReadInputTokens`
and `cacheWriteInputTokens`.
- The whole provider response that carries either.
- Every gateway answer, fail-open ones included, carries `wire_shape`:
`gemini`, `converse`, `openai_responses` or `messages`. A caller that
sent a native body and gets anything else, or no field at all, is
talking to an older Headroom and should forward its original body.
- Top-level `messages` in the answer is the Converse list for Converse
and the chat-shaped view for Gemini.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`): no new errors; the 8
reported are in `ccr/mcp_server.py` and `cache/prefix_tracker.py` and
are identical on `main`
- [x] New tests added for new functionality
- [x] Manual testing performed (local replay, below)
New in `tests/gateway/test_native_shapes.py`:
- `test_ccr_mode_is_rejected_for_native_bodies[gemini|converse]`: 400,
`invalid_request`, message asserted.
- `test_other_modes_forward_native_bodies_without_ccr_markers`: 4 modes
x 2 shapes, on log output large enough to produce a marker. Checks no
hash, no marker, the shape echoed, non-message keys byte-identical and
the native list mapped back.
- `test_ccr_still_works_on_a_chat_gateway_turn`: the same log output in
a chat body with `ccr` still carries a marker. This proves the input
above is marker-producing, and that chat `ccr` didn't regress.
- `test_native_bodies_get_no_dedup_pointer_into_view_messages`.
- `test_native_turns_are_never_asked_to_redrive` now runs in the default
mode, because `ccr` is a 400.
### Test Output
```text
# before the fix (new tests only, rest of the file passing)
FAILED tests/gateway/test_native_shapes.py::test_ccr_mode_is_rejected_for_native_bodies[gemini]
FAILED tests/gateway/test_native_shapes.py::test_ccr_mode_is_rejected_for_native_bodies[converse]
FAILED tests/gateway/test_native_shapes.py::test_native_bodies_get_no_dedup_pointer_into_view_messages
3 failed, 25 passed
# after
tests/gateway/test_native_shapes.py: 28 passed
# tests/gateway tests/test_gateway_turn_unit.py tests/test_compress_api.py
# tests/test_proxy_compress_endpoint.py tests/test_compress_session_mode.py
# tests/test_compress_session_shared.py tests/test_compress_passthrough.py tests/test_compress_failure.py
1 failed, 507 passed, 4 skipped
# the one failure, test_lossy_inline_emits_no_ccr_markers[1-json_minifier], fails identically on main
|
||
|
|
94a99ed20f |
feat(proxy): give turn hooks one conversation id per client conversation (#4496)
## Description Plugins that pin wording per conversation (headroom-condense, and through it skill-search and tool-search) cannot see a real conversation id. Today: - **Claude:** they hash the first typed prompt, so two sessions that open with the same words share a key. - **Codex:** they get no key at all, so every new skill or MCP description waits out the one-hour hold. This PR adds an optional `TurnContext.conversation` that core fills from the client's own conversation id, so hooks can pin per real conversation. It is item #4 of the compression-gaps plan; the plugin work for #3, #6, #8 and #9 needs it to stay cache-safe. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **New field:** `TurnContext.conversation: str | None = None`. It is an opaque key that stays the same for every turn and retry of one client conversation, or `None` when the client names none. Hooks get a digest, never the raw id. | Path | Source | | --- | --- | | `/v1/messages` | sha256 of `X-Claude-Code-Session-Id` (Claude Code sends it on every request), falling back to `X-Headroom-Session-Id`; each source has its own namespace | | `/v1/responses` (HTTP and WebSocket) | the new `turn_conversation_key`, computed from the body before compression and passed through `_compress_openai_responses_payload_in_executor` | | `/v1/chat/completions` | `X-Headroom-Session-Id` only | | `/v1/compress` (gateway) | the gateway session id; `None` without one | - **Responses key.** `turn_conversation_key` uses the same identity and digest as `savings_conversation_key` (`thread_id` before `session_id`, then `client_metadata.*`, then the session header). Unlike the savings key, it also applies on `previous_response_id` turns and on bodies that name a server-side `conversation` (a string, or an object with an `id`), so the key does not drop out after the first WebSocket frame. `savings_conversation_key` keeps its existing exclusions. - **Gateway seam.** `CompressTurn.prepare()` is a third-party seam, so its signature is unchanged. Instead, the handler sets an optional writable `conversation` attribute on the turn before calling `prepare`. This is documented in `compress_turn.py`. - **No new ids.** Nothing is generated. The per-connection WebSocket uuid is not used, because a reconnect would change it. - **One path left out.** The catch-all passthrough Responses route still passes `None`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`): 3 errors, all in files this PR does not touch (embedders, model_metadata, langchain) - [x] New tests added for new functionality - [x] Manual testing performed - **New tests:** - `tests/test_turn_context_conversation.py`: - Anthropic: the same session gives one key, a different session with the same prompt gives a different key, the fallback header gets its own key, and no header gives `None`. - Responses HTTP: the same thread gives one key, including on a `previous_response_id` turn, and a different thread gives a different key. - Responses server-side `conversation`: string and object forms give one key across turns; a different conversation gives a different key; the savings key stays `None`. - Responses WebSocket: first and later frames share one key; a different thread gets a different key. - Chat Completions: covered the same way. - `tests/gateway/test_contract_request_half.py`: per-session gateway keys, and `None` without a session. - Every new test fails on `main`. - **Updated test:** `tests/test_proxy_openai.py` now pins `conversation=None` for a body with no ids. ### Test Output ```text pytest (turn hooks, gateway, Codex WS lifecycle, proxy openai, conversation savings, passthrough, hook usage): 411 passed, 5 skipped after the server-side conversation fix, on the synced head: 101 passed (conversation, gateway, Responses suites) ruff@0.17.0 check / format --check: clean mypy headroom --ignore-missing-imports: 3 errors, none in changed files ``` `tests/test_turn_hook_usage.py` needs `LITELLM_LOCAL_MODEL_COST_MAP=True` in a fresh venv; that is an existing litellm price-map fetch, and it fails the same way on `main`. ## Real Behavior Proof - Environment: - macOS 27.0.1 (Darwin 27.0.0, arm64), Python 3.12.13. - The proxy ran from this branch in a fresh `uv` venv (`uv venv -p 3.12 && uv pip install -e ".[dev]"`) on 127.0.0.1:9100, with default `ProxyConfig` (optimize on). - Upstream was the real Anthropic API, with my own API key passed by environment variable. - Client was Claude Code 2.1.311 (`claude --version`) with `--model claude-haiku-5-5`. - Exact command / steps: 1. Start the proxy with a scratch-only launcher (not committed). It calls `register_turn_hook()` with a hook whose `on_request` appends `ctx.provider`, `ctx.model`, `ctx.conversation` and `len(ctx.messages)` to a JSONL file, then calls `run_server(ProxyConfig(host="127.0.0.1", port=9100))`. Run it with `PYTHONSAFEPATH=1 HEADROOM_TELEMETRY=off LITELLM_LOCAL_MODEL_COST_MAP=True python run_proxy.py hooks.jsonl`, from a directory outside the repo. 2. Point Claude Code at the proxy: `export ANTHROPIC_BASE_URL=http://127.0.0.1:9100 CLAUDE_CONFIG_DIR=<fresh empty dir>`. 3. Start two sessions with the same prompt: `claude -p "say hi" --model claude-haiku-5-5 --output-format json`, run twice. Note each `session_id`. 4. Add two more turns to session 1: `claude -p "now say bye" --resume <session 1 id> --model claude-haiku-5-5`, then `claude -p "one more word please" --resume <session 1 id> --model claude-haiku-5-5`. 5. Read `hooks.jsonl`, and compare each key with sha256(`claude-code\0` + session_id). - Observed result (from `hooks.jsonl`, keys truncated): | Session | Turn | Messages | `ctx.conversation` | | --- | --- | --- | --- | | `305d64a9…` | first | 2 | `306306eeb9bc164b…` | | `4e50609f…` (same prompt) | first | 2 | `4a9e51bc7f8005f2…` | | `305d64a9…` | first `--resume` | 4 | `306306eeb9bc164b…` | | `305d64a9…` | second `--resume` | 6 | `306306eeb9bc164b…` | Each key equals sha256(`claude-code\0<session id>`) of the session id that Claude Code reported. All three runs answered normally, and the proxy log had no errors. - Not tested: - Codex over Responses HTTP or WebSocket (unit and handler tests cover these). - Chat Completions with `X-Headroom-Session-Id`. - The `/v1/compress` gateway path. - The server-side `conversation` body form, live (covered by the handler test). - The `X-Headroom-Session-Id` fallback on `/v1/messages`. - Claude Code `/compact` and subagents. - Bedrock and Vertex upstreams. - Linux and Windows. - A real plugin consuming the key. That is wired in headroomlabs-ai/headroom-condense#7; skill-search and tool-search adoption comes next. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: stable. - Stable/default behavior changed: no request or response bytes change. Registered turn hooks receive one new optional field; hooks that do not read it behave as before. - Kill switch / disable path: none needed; a hook that should not use the key ignores it (headroom-condense falls back to its previous key when the field is `None` or absent). - Unsafe override required: no. - Qualification impact: none. - Rollback path: revert this PR. Plugins read the field with `getattr`, so they keep working on cores without it. Cache note: compare keys only within one endpoint, since the same header gives different keys on `/v1/messages` and `/v1/responses`. Each key comes from a client id that stays the same for the whole conversation (Claude Code session ids survive retries, `--resume` and `/compact`; Codex `thread_id` wins over `session_id`, so guardian and subagent requests do not collide), and the Responses key is taken before compression rewrites the body, so a pinning hook keeps the same pin on every turn. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A (proxy-side change). ## Additional Notes - "No new warnings" is unchecked because the run was not compared warning by warning against `main`. - Follow-ups: skill-search and tool-search switch to `conversation_for(ctx)` once headroom-condense#7 is released. 🤖 Generated with [Claude Code](https://claude.com/claude-code) ## Maintainer post-merge qualification Published `819f82117a4737de621528e9c49aee6aeeb4a793` includes upstream main `e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`. 28 conversation/Responses/per-frame-memory/disconnect tests pass; Ruff/format/diff pass. Clean main merge; hook conversation policy unchanged, no new code finding. Author explicitly reserves unchecked self-review/readiness attestations; existing hold remains. Contributor correctness logic was not rewritten. Existing Changes Requested decision retained without duplicate findings. Scoped Windows validation uses the existing development native extension, not a fresh build. Hosted qualification is handled by maintainers; nothing merged. ### Maintainer current-main requalification Exact head `965ff21ffbea9b446e5e39269812bb0d8c43c108`, upstream main `ecbaf86d78cf52d185fbffa306157b9a4cc1b906`. Mechanically updated from upstream/main |
||
|
|
33f067a243 |
feat(proxy): add lossless /v1/compress mode and detect provider by wire shape (#4617)
## Description Two gaps stop a gateway from using `/v1/compress` well. Both showed up while fixing LiteLLM's `headroom` guardrail, which is the follow-up PR that needs this one. 1. **No mode compresses messages losslessly without markers.** A gateway with no retrieval loop can't resolve a marker or call `headroom_retrieve`, so it needs a mode with nothing to retrieve. The existing marker-free modes still run lossy compressors. 2. **Hooks guess the provider from the model name.** A gateway can send any client's shape to any model. Claude Code's Anthropic body routed to Gemini looked like OpenAI, so skill condensing found nothing to change. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) ## Changes Made **`config.mode="lossless"`** - Runs format-native folds only, with no markers and no CCR store writes. A registered lossless provider still applies. - Turns off stale-Read pointers, which drop the file body and need the retrieval tool. - Turns off cross-turn dedup pointers, which a client that never numbers its messages can't resolve. - Scope: the guarantee covers message compression (lossless codecs, no CCR). On a gateway turn, tool schema compaction and registered turn hooks still run as configured, as in every other mode; description truncation only happens when an operator sets `HEADROOM_TOOL_DESC_MAX_CHARS`. **`provider_for_body`** - Hooks and the session tracker now use the body's wire shape, and fall back to the model name when the shape proves nothing, such as plain text with no system prompt or tools. - Anthropic's own tool types (server tools such as `web_search_20250305`, client tools such as `bash_20250124`) end in an eight-digit date and count as the Anthropic shape, so a first-turn body whose only tool is a server tool is read correctly. - `route.provider` still follows the model. - The cache breakpoint budget runs when either the shape or the model is Anthropic, and never on a Responses body. - `gateway_response_fields` loses its unused `provider` parameter, and `RESPONSE_RECEIVED` keeps reporting the model's provider. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`, `ruff format --check .`, on the pinned 0.17.0) - [ ] Type checking passes (`mypy headroom`): one error, in `headroom/integrations/langchain/chat_model.py`, which this PR doesn't touch (local mypy 1.19.1) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text pytest tests/test_proxy_compress_endpoint.py tests/gateway tests/test_gateway_turn_unit.py tests/test_compress_route_tokenizer_by_model.py 425 passed, 4 skipped Regression tests with the fixes removed: 8 failed, 7 passed ``` ## Real Behavior Proof - **Environment:** LiteLLM main with the follow-up guardrail change, this branch as the sidecar, and a fake upstream that simulates prompt caching. No real provider API was called. - **Exact command / steps:** replayed a scripted 17-call agent session with exact-duplicate outputs removed, once direct to the fake upstream and once through LiteLLM and `/v1/compress`. - **Observed result:** 0 errors, 0 rewrites of already-sent results, `Read` and `Edit` results unchanged, about 1.3s added in total. | Client | Prompt cost vs direct | |---|---| | Anthropic Messages, Claude Code cache markers | −3.9% | | Chat Completions | −3.8% | | Responses | −4.1% | The script's outputs are synthetic, so treat these as a comparison between setups, not a forecast for real traffic. ## Known limits - The default mode and `lossy_inline` can still return a stale-Read `Retrieve original: hash=` marker, although the docs call them marker-free. This predates the PR and needs a separate fix. Proxy `--lossless` has no read-lifecycle guard either, from reading the code. - With no `ContentRouter`, `_derived_compress_pipeline` falls back to the default pipeline, so `lossless` wouldn't hold. Production always has a router, so only test stubs reach this. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4617" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ## Maintainer post-merge revisit: main |
||
|
|
55602fc1e2 |
fix(proxy): cap cache_control breakpoints on OpenAI chat path (#4471) (#4491)
## Description On the OpenAI `/chat/completions` path in cache mode, earlier messages are replayed with old `cache_control` markers. The 4-marker budget guard `enforce_cache_breakpoint_budget` now runs on that path after the mirror step, skipped for `x-headroom-bypass`, with `client_messages=None` to avoid the block normalizer inventing markers; `tool_calls` markers are counted/dropped oldest-first. Closes #4471 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `handlers/openai.py`: run `enforce_cache_breakpoint_budget` as last-stop guard right after the mirror step (skipped for `x-headroom-bypass`), passing `client_messages=None` to avoid the block normalizer inventing markers; same guard the gateway extension already uses. - `helpers.py`: `count_cache_breakpoints` counts `tool_calls[*].cache_control`; the oldest-first drop pass also strips those markers. - Mirror step left as-is: its documented bail-out on count mismatch stays, the budget guard now bounds the total with the bypass gate and `client_messages=None`. ## Testing - [x] Unit tests added/updated - [x] Existing tests pass ```text $ pytest tests/test_openai_message_level_cache_control.py tests/gateway/test_cache_breakpoint_budget.py tests/test_cache_breakpoint_diagnostics.py <prefix-tracker/mirror tests> -q 1 failed, 115 passed, 8 skipped in 10.69s FAILED tests/test_cache_breakpoint_diagnostics.py::test_runtime_log_says_why_it_is_missing # pre-existing on main: Windows WinError 183 on temp log dir $ ruff check / ruff format --check: clean ``` ## Real Behavior Proof - Environment: Windows 11, Python uv tool env with headroom._core built - Exact command / steps: unit-level repro of the issue body shape (4 client markers + replayed prefix carrying an old message-level and tool_calls marker, counts differ) through enforce_cache_breakpoint_budget - Observed result: 6 markers in → ≤4 out, newest tail marker kept, stats["repaired"] True - Not tested: live LiteLLM + Vertex gateway end-to-end ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: stable - Stable/default behavior changed: only when a chat request exceeds 4 cache_control markers (previously a guaranteed upstream 400) - Kill switch / disable path: none needed; under-limit requests return by identity unchanged - Unsafe override required: no - Qualification impact: none - Rollback path: revert this PR ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4491" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
a90421083a |
feat(proxy): report the same tool savings on gateway /v1/compress as the proxy path (#4467)
## Description Gateway-mode `/v1/compress` now reports the same savings as the proxy path. The why is in #4466. Closes #4466 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [x] Documentation update ## Changes Made Two commits, one per gap: 1. **`fix(proxy): count tool-schema compaction ...`** - The gateway turn counts the tool definitions before and after compaction. - The handler adds both counts to `tokens_before` / `tokens_after`, with the proxy path's guard (`0 < after < before`). - Tools the client already marked `defer_loading` are skipped. 2. **`feat(proxy): report tool_tokens_saved and tool_tokens_deferred ...`** - Gateway-mode answers add `tool_tokens_saved` (from the tags `/stats` sums) and `tool_tokens_deferred` (the estimated share). - Docs: `proxy.mdx`, `wiki/proxy.md`. Non-gateway and fail-open answers do not change. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`, on the changed files) - [x] Type checking passes (`mypy`, on the changed files) - [x] New tests added for new functionality ```text $ pytest -q tests/gateway/ tests/test_gateway_turn_unit.py tests/test_proxy_compress_endpoint.py 380 passed, 5 skipped ``` The tests also pass on the first commit alone (`tests/gateway/`: 280 passed). The new test for client-deferred tools fails without its fix. ## Real Behavior Proof - Environment: macOS, Python 3.12. Before: image `headroom:0.41.0-nonroot`. After: this branch, `HEADROOM_OFFLINE=1 headroom proxy`. - Exact command / steps: start each build, then send it the same gateway-mode `POST /v1/compress` — one message plus 5 tools whose schemas carry `title` / `examples` / `default` (the annotation keys schema compaction strips), with `"gateway": {"can_redrive": false, "can_relay_response": false}` so the request is a single call. Read `tokens_before`, `tokens_after`, `tokens_saved` and the `tool_*` fields out of the JSON answer. No provider is contacted. - Observed result: same request, same tools. On 0.41.0 the tool definitions are invisible to the counts, so `tokens_saved` is 0 and neither `tool_*` field is present. On this branch the compaction is counted, so `tokens_before` / `tokens_after` carry the tool bytes and `tokens_saved` is the 240 tokens compaction stripped (`tool_tokens_saved` / `tool_tokens_deferred` stay 0 here: no tool-search extension ran, and the extension is what books deferral). ```text 0.41.0: tokens_before 13, tokens_after 13, tokens_saved 0 (no tool_* fields) branch: tokens_before 830, tokens_after 590, tokens_saved 240, tool_tokens_saved 0, tool_tokens_deferred 0 ``` - Not tested: a live run with the headroom-tool-search extension (deferral is covered by unit tests), and Responses-API bodies. ## Runtime Rollout Safety - Rollout-managed feature(s): None gate this change — the gateway `/v1/compress` accounting is not behind a registry feature. One interaction: `proxy_output_shaper` (registry feature, available `stable`, deliberately not default-enabled) consumes the augmented count. `tokens_before` is folded at `headroom/proxy/handlers/openai.py:11064` and then handed to the turn at `:11083`, which passes it as `input_tokens` to `shape_gateway_body` (`headroom/proxy/gateway_extension.py:213`) → `stratum_key` → `input_bucket`. A turn whose tool definitions push it across one of the 2k / 8k / 32k / 128k boundaries therefore lands in a bigger stratum, which picks a different output-savings baseline. The chat path already has this coupling. - Minimum rollout channel: n/a — no channel gate, so the change takes effect wherever gateway mode is served, on every channel. - Stable/default behavior changed: Yes, gateway-mode answers only. `tokens_before` / `tokens_after` now include the tool definitions before and after schema compaction, so `tokens_saved` covers the compaction saving the same way the chat handlers already do, and two fields are added. Non-gateway and fail-open answers are byte-identical. - Kill switch / disable path: None. Without compaction there is nothing to fold, and the two fields are additive on answers; a gateway that does not want them ignores them. - Unsafe override required: No. Nothing crosses a channel boundary. - Qualification impact: None. No registry entry is added, so `registry_digest` and `snapshot_digest` are unchanged. The gateway `tokens_saved` figure itself changes, which a benchmark comparing arms across this commit will see. - Rollback path: Revert the two commits. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] I have made corresponding changes to the documentation - [x] I have added tests that my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes - The fields are emitted for any claimed compress turn. Only the gateway extension claims turns today. - If a response-half hook rewrote the tool tags during a re-drive, `/stats` would no longer match the answer. Core has no such hook. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
85520c2941 |
fix(proxy): net tools a turn hook adds against the saving it books (#4057)
## Description A turn hook that folds messages but **adds** a tool definition saved the difference, not the fold alone. headroom-skill-search does exactly this: it rewrites the skill list in the messages and appends a `search_skills` tool. Two places over-credited it: - **Hook runner (`run_request_hooks`):** - It clamped the message and tool halves separately at zero. - So the added tool never reduced the hook's ledger entry (`/stats` `savings.by_source.skill_search`). - **Anthropic and OpenAI chat handlers:** - Tool shrinkage from a hook goes into `turn_hook_tools_saved_tokens`, but tool growth went nowhere. - So the headline `tokens_saved` ignored a tool that is sent on every request. Found while auditing the enterprise plugins for the provider-unit savings work (#4052 / #4055). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `turn_hooks.run_request_hooks`: signed message and tool deltas, netted. An entry is recorded only when the net is a saving. `details.tool_tokens_saved` is negative when tools were added. - Anthropic handler: tool growth from turn hooks (`_th_tool_growth`) is added to `optimized_tokens` at the recount. This is the same place compaction deltas are folded, so `tok_before - tok_after == tok_saved` stays coherent. - OpenAI chat handler: the same, in its post-hook recount. - `turn_hook_tools_saved_tokens` keeps its non-negative meaning (shrinkage only). Every surface that sums it is unchanged. ## Testing - [x] Unit tests pass (`pytest`): related suites - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy`) on changed modules - [x] New tests added for new functionality - [x] Manual testing performed (plugin suites against this core) ### Test Output ```text tests/test_hook_net_tool_additions.py: 6 passed - ledger entry is net of the added tool; no entry when the tool outweighs the fold - Anthropic /v1/messages and OpenAI /v1/chat/completions: tokens_saved == fold - added tool - a later hook that grows the request more than was saved cancels the earlier credit - a later partial loss caps the earlier credit at the sequence net, and the breakdown (message + tool) sums to the capped tokens (uncapped_* kept) tests/gateway/test_hook_tool_growth.py: /v1/compress tokens_after includes the added tool -> each fails on the commit before its fix (e.g. 1499 == 1499 - 117, 0 > 100, ('fold', 200) == ('fold', 142), (200 + 0) == 142) gateway + turn_hook / tool_search / tool_schema / savings_attribution / extension / anthropic / openai / responses / compress suites: 2101 passed, 34 skipped Plugin suites against this core (PYTHONPATH): skill-search 41, tool-search 144, lossless-guard 237 passed ruff check / format: clean; mypy (changed modules): Success ``` ## Real Behavior Proof - Environment: macOS, Python 3.12, `LITELLM_LOCAL_MODEL_COST_MAP=True`; the real proxy app (`create_app`) through FastAPI TestClient, upstream mocked with respx. - Exact command / steps: `pytest tests/test_hook_net_tool_additions.py tests/gateway/test_hook_tool_growth.py` (a hook shaped like SkillSearchHook folds messages and adds a search tool, on `/v1/messages`, `/v1/chat/completions` and `/v1/compress`), then the skill-search / tool-search / lossless-guard suites with `PYTHONPATH` set to this branch. - Observed result: on every path `tokens_saved` and the `skill_search` ledger entry equal the message fold minus the added tool's tokens; across several hooks the total credit never exceeds the sequence's net saving and each entry's breakdown sums to its tokens. - Live skill-search session (2026-10-15, branch head `7a81ec9f`): - Setup: a real proxy run from this branch with headroom-skill-search 0.2.0 (`--proxy-extension skill_search`, `HEADROOM_SKILL_SEARCH_HINT_MODE=first_sentence`, `HEADROOM_BEACON=off`), real Anthropic upstream, `claude-haiku-4-5`. Each request was a non-streaming curl to `/v1/messages` with 22 skills listed the way Claude Code lists them. - Turn 1: HTTP 200, body 9116 B → 2744 B. - The `skill_search` ledger booked `message_tokens_saved=1327` and `tool_tokens_saved=-108`; the −108 is the added `search_skills` tool. That gives `tokens=1219`, the signed net this PR books. - Anthropic `count_tokens` on the original body: 2415. Input billed through the proxy: 1153–1276. - Headline: - On this branch alone it read `tok_before=424 tok_after=532 tok_saved=0`. The cause is a bug on main, not in this PR: `snapshot_original_messages` aliases the live messages when only turn hooks are registered, and skill search folds text in place, so the recorded original was the folded one. The fix is #4241. - With #4241 applied: `tok_before=1751 tok_after=532 tok_saved=1219` = 1327 − 108, the same as the ledger. - Turn 2: the model called `search_skills` and the hook served `image-resize`. After one reload round the final answer came back HTTP 200 `end_turn`. Headline `tok_saved=1219`; no new ledger entry was booked. - `/stats` `savings.by_source.skill_search`: 1 event, 1219 tokens. - Not tested: streaming (skill search acts only on buffered turns with at least 18 skills), and the OpenAI path live. - Merge order: the headline matches the ledger only once #4241 is on main. Merge #4241 first. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: n/a - Stable/default behavior changed: yes, accounting only (requests are byte-identical). Headline, `/v1/compress` `tokens_saved` and `skill_search` attribution drop by the size of the tool the hook adds, and a later hook's growth offsets earlier hook credits. - Kill switch / disable path: n/a - Unsafe override required: no - Qualification impact: one extra tool-array count per hooked OpenAI chat request. - Rollback path: revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes - **Docs:** none needed; this is an internal accounting fix. - **Stacked on #4056** (base `fix/hook-tool-deferral-accounting`). Both PRs change the same lines in `run_request_hooks`, so this branch carries the resolution: the signed tool delta is measured with `without_deferral_flags`, and the OpenAI tool-growth count strips the flag too. Merge #4056 first. The combined branch passes 844 core tests and the skill-search, tool-search and lossless-guard suites. 🤖 Generated with [Claude Code](https://claude.com/claude-code) ## Current maintainer verification (October 8) Exact synchronized head: `de0208ae9dbabe7967d4b781e67364f6ee8b141d`, containing main `01d9194854233ef6ed04005cf0a5b8c32bf65d6d`. The main merge preserves the contributor's signed sequence accounting and net-saving helper, along with main's deferral-filtered tool counters. Parent #4056 has merged, so the PR now targets main directly. No contributor logic was changed. 25 focused hook/gateway/deferral tests pass. Two additional isolated reviewer probes force only the session's final message recount to fail; positive and negative net-savings cases both pass, reaching the fallback without breaking the hook counter. This is local TestClient/native-core evidence, not a live skill-search session. Current-head hosted qualification is pending. The previously acknowledged live skill-search runtime proof remains outstanding; the new tests close the accounting finding but do not close that evidence requirement. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
5cac7ab693 |
fix(gateway): report why /v1/compress/response rejected a body (#4081)
## Description `POST /v1/compress/response` (the response half of the gateway contract) caught every body-read failure and answered `"Invalid JSON in request body."`. That covered malformed JSON, but also an oversize body, a failed decompression, invalid UTF-8 and a JSON array. A Kong or LiteLLM integrator debugging a 400 could not tell which one it was. The other handlers (anthropic, openai, gemini, batch; about 10 call sites) already use `helpers.invalid_request_body_message(exc)`. It picks a fixed string by exception type, so it never echoes the exception text that CodeQL's `py/stack-trace-exposure` flags, and it logs the exception type on the proxy side. This handler now uses it too. That gives specific messages for malformed JSON, a non-object, invalid UTF-8 and an oversize body. A failed decompression (for example a bad gzip stream) still gets the generic `Invalid request body`, as it does on every other handler. A separate decompression message would mean changing the shared helper for all handlers, so that is out of scope here. The `if not isinstance(body, dict)` branch that followed is removed. `_read_request_json` raises `RequestBodyNotObject` for a non-object before it can return one, so the branch could not be reached. Its case is now reported as `Invalid request body: must be a JSON object`. The status code (400) and the error `type` (`invalid_request`) are unchanged. Only the `message` text differs. No test, doc or consumer repo matched on the old string: I grepped `tests`, `docs`, `headroom` and the gateway integrations, which branch on the status code. Refs #4136 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/gateway_turn.py`: `handle_compress_response` answers 400 with `invalid_request_body_message(exc)`, and the unreachable non-dict check is gone. - `tests/gateway/test_contract_response_half.py`: add `test_invalid_body_400_says_why`, parametrized over three bodies. Each case asserts a 400, `invalid_request`, and a message that names the reason: - malformed JSON; - a JSON array; - invalid UTF-8. All three cases fail on main. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest -q -p no:cacheprovider tests/gateway/test_contract_response_half.py -k invalid # before the fix FAILED tests/gateway/test_contract_response_half.py::test_invalid_body_400_says_why[malformed] FAILED tests/gateway/test_contract_response_half.py::test_invalid_body_400_says_why[not-object] FAILED tests/gateway/test_contract_response_half.py::test_invalid_body_400_says_why[not-utf8] ================== 3 failed, 6 passed, 28 deselected in 7.18s ================== $ pytest -q -p no:cacheprovider tests/gateway/ tests/test_gateway_turn_unit.py tests/test_invalid_request_body_message.py # after ================== 323 passed, 4 skipped in 171.41s (0:02:51) ================== $ uvx ruff@0.16.8 check headroom/proxy/gateway_turn.py tests/gateway/test_contract_response_half.py All checks passed! $ mypy headroom/proxy/gateway_turn.py --ignore-missing-imports Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS, Python 3.12.13, branch checkout at |
||
|
|
ee731d7ea8 |
fix(proxy): refuse token-less non-loopback binds, gate operator routes (#3852)
## Why Four related gaps in the proxy's network gate (findings 01-F1, 01-F2, 01-F5, 01-F9): - **Open bind.** A non-loopback bind with no `HEADROOM_PROXY_TOKEN` only logged a warning, then served the `/v1/*` relay, with the operator's upstream credentials, to every peer that could reach the port. - **Missing peer address.** `is_loopback_host(None)` returned `True`. A Unix-domain socket, an ASGI adapter that leaves `scope["client"]` unset, or a hand-built request therefore turned off the token gate, the `/admin` and `/debug` guards and the WebSocket gate. - **Operator data routes.** `/stats-history` (including CSV export), `/quota`, `/subscription-window` and `/metrics` were served to any network caller. - **WebSocket loopback exemption** checked only the peer IP, not the `Host` header, so it had no DNS-rebinding defence. ## What changes - New `headroom/proxy/bind_policy.py`. A non-loopback bind (`0.0.0.0`, `::`, a LAN address or a hostname) with no token is refused at three places: `create_app` (embedders and the multi-worker factory), `run_server` (one exit with code 2 before uvicorn forks) and the `headroom proxy` CLI (a clear error before the banner, exit code 1). An absent host means uvicorn's default `127.0.0.1`. - `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` acknowledges an open bind when the runtime already restricts reachability, such as a container published on `127.0.0.1`. The `proxy_open_bind` warning is still logged on that path, and the banner says the bind is open. - `is_loopback_host(None)` now returns `False` (fail closed), matching `identity.py`. - `/stats-history`, `/quota` and `/subscription-window` answer only: a non-loopback caller that passed the token gate, a loopback caller (peer and `Host` header both loopback), a trusted dashboard client, or, under an acknowledged token-less bind, the container's own default-gateway TCP peer with a loopback `Host` (the host browser reaching a `127.0.0.1`-published container). Everyone else gets 404, as with `/admin`. Settings writes keep their stricter rule with no token short-cut. - `/metrics` answers a non-loopback caller that passed the token gate, a loopback caller, a connecting peer in `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` (whatever it forwards), or a resolved client in the dashboard CIDRs. For CIDR peers a hostname `Host` is fine, so Prometheus scrapers work; a cross-origin browser request is still refused. - The token short-cut applies only to non-loopback requests. The gate exempts loopback peers without checking `Host`, so a loopback request has not proven it holds the token and must pass the loopback `Host` check. This blocks a DNS-rebound browser on the operator's machine. - The `/dashboard` shell stays ungated. It is a static template, like the already public `/dashboard/static` assets. Gating it would break the dashboard for containers published on host loopback, where the peer is the bridge gateway and `Host` is `localhost`. The data routes behind it are gated. - The WebSocket loopback exemption requires a loopback peer and a loopback `Host` header. A loopback peer with a foreign `Host` can still connect with the token. - Launchers that always publish on `127.0.0.1` set the acknowledgement themselves: `headroom install` (docker preset), `scripts/install.sh` and `scripts/install.ps1` (through one helper that adds the acknowledgement only together with the `127.0.0.1` publication), and `docker/docker-compose.native.yml`. - Docs: `proxy.mdx` documents the new variable and which routes the token also gates; `docker-install` (docs and wiki) and the `docker-compose.yml` comment show the acknowledgement in the loopback-published example. `e2e/docker-bind-security.sh` sets it and checks that the same launch without it refuses to start. ## Behaviour change (breaking) - `headroom proxy --host 0.0.0.0`, or `HEADROOM_HOST` set to any non-loopback address, without `HEADROOM_PROXY_TOKEN` now refuses to start. The error names both remedies. Set a token, bind loopback, or set `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` when the port is already confined. Anyone running the image by hand with `--host 0.0.0.0` and no token must add one of these. - Without a token, `/stats-history`, `/quota`, `/subscription-window` and `/metrics` return 404 to non-loopback callers outside a trusted CIDR. Prometheus scrapers on another host must be in `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` or send the token. - Code that relied on a request with no peer address being treated as loopback now gets the non-loopback path. - The default loopback run is unchanged. ## Tests - `tests/test_proxy_bind_policy.py`: the refuse/allow matrix (non-loopback, loopback, unset host, token, acknowledgement values), the warning on the acknowledged path, `create_app`, the `run_server` exit before uvicorn, and the CLI (refusal, `HEADROOM_HOST` from env, acknowledged start, offline banner, start with a token). - `tests/test_proxy_loopback_gating.py`: each data route against non-loopback, loopback, trusted-CIDR and token callers; the dashboard shell stays reachable; the host of an acknowledged, loopback-published container reads the data routes while another bridge peer, a forwarded gateway address and a foreign `Host` do not; CSV export is not served to network callers; `/metrics` across loopback, both CIDR lists, out-of-range peers, cross-origin browsers and token callers, including a trusted gateway relaying a scraper outside its own range. A regression test shows that a token on the proxy does not exempt a loopback peer from the `Host` check; it fails without the fix. - `tests/test_proxy_hardening.py`: the HTTP gate with no peer address, WebSocket `Host` cases, `require_loopback` and `/debug` with no peer. - Installer tests: `headroom install` adds the acknowledgement, and the shell and PowerShell wrappers add it only together with the loopback publication. Launcher-level tests start the proxy with the environment and `--host` that `build_runtime_command()` and `docker-compose.native.yml` produce and check that the host browser (bridge-gateway peer) gets `/stats-history` and another container does not. - Tests that bound `0.0.0.0` for unrelated reasons now set a token or the acknowledgement. Tests that read operator routes use an explicit loopback peer. On `main`, 14 of the gating and hardening tests fail and the bind-policy module does not exist. Manual runs: token-less `--host 0.0.0.0` exits with the refusal from both the CLI and `run_server`; with the acknowledgement, the data routes are 404 to a LAN peer and to a loopback peer with a foreign `Host`; with a token, a LAN peer gets 401 without it and 200 with it; the default loopback bind starts as before. Not tested: a real Unix-domain-socket deployment and a real Prometheus scrape (both covered by tests only). ## Merge order Part of a set. Suggested order: #3852 → #3891 → #3849 → #3860 → #3862 → #3863. This PR goes first: it is the gate the others assume, and it is the release-note event. ## Not in this PR - Enforcing the token for loopback callers when one is configured. Revised 2026-10-02 after review: `/metrics` matches gateway CIDRs against the TCP peer, and the operator data routes serve the host of a loopback-published container (bridge-gateway peer under the launchers' bind acknowledgement) so the default Docker dashboard works. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3852" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
befdb52fd5 |
fix(proxy): bill the whole prompt on the gateway path so savings read true (#3632)
## Description `/stats` reported **97% of tokens saved** on a real Claude Code session through Kong. The compression is real, the percentage was not. Anthropic reports a cached turn as `input_tokens` = the *uncached remainder*, with the cached prefix in `cache_read_input_tokens` / `cache_creation_input_tokens`. The direct Anthropic handler already sums all three into `provider_input_tokens` ([handlers/anthropic.py:3587](https://github.com/headroomlabs-ai/headroom/blob/main/headroom/proxy/handlers/anthropic.py#L3587)). The gateway path recorded `usage.input_tokens` alone, so `tokens.input` held the uncached sliver and every ratio over it was inflated. On a fresh 5-call run the old formula reports **100.0%**, because the denominator was 10 tokens against an 113,490-token prompt. Affects anything speaking the `/v1/compress` contract: the Kong plugin, the LiteLLM package, custom gateways. The direct proxy path was already correct. Second, smaller problem: tokens alone mislead here even when counted right. Deferred tool schemas sit at the front of the prompt, so after turn 1 they would have been cache reads at roughly a tenth of list input. "90% of tokens" is nowhere near "90% of the bill" (measured on this laptop's demo traffic: 604 calls, ~$3.12 saved of ~$23.34, about 13%). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `gateway_turn.complete_outcome`: records the whole billed prompt in `provider_input_tokens` and the remainder in `uncached_input_tokens`, matching the direct handler - `gateway_turn.normalize_usage`: new `input_includes_cache`, decided by the presence of Anthropic's cache fields rather than the key name. OpenAI's Responses API also calls its field `input_tokens` but already counts the cached prefix in it, so keying off the name would double-count every Responses prompt - `/stats` `savings.by_layer.tool_search.usd`: the cache-aware dollar value of deferral, the same figure the cost card already uses - `/stats` `tokens.cost_weighted_savings_percent`: the saving in money beside the saving in tokens - dashboard: a "Cost-weighted: X%" line under Tokens Saved, and the "Of total wire" tooltip now says the denominator includes the cached prefix - tests: gateway shapes (Anthropic split, OpenAI chat, OpenAI Responses), and `/stats` honesty on cache-heavy traffic No request is shaped differently and no bill changes. Reporting only. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/gateway 250 passed, 4 skipped in 122.95s $ pytest tests/test_gateway_turn_unit.py tests/test_dashboard_token_savings.py \ tests/test_dashboard_accessibility.py tests/test_dashboard_static_assets.py 64 passed in 10.58s $ pytest tests/gateway tests/test_gateway_turn_unit.py tests/test_stats_new_input_savings_rate.py \ tests/test_stats_tool_schema_savings_are_honest.py tests/test_proxy_savings_history.py \ tests/test_proxy_stats_recent_requests.py tests/test_proxy_dashboard_stats_cache.py \ tests/test_cost_card_attribution.py tests/test_cost_tracker_totals.py \ tests/test_cost_budget_total_prompt.py tests/test_cost_tracker_counterfactual.py 376 passed, 5 skipped (one pre-existing failure, test_outcome_anthropic_cache_fields, asserted provider_input_tokens == 300 for a 2,800-token prompt: updated here) # the new tests, without the source change FAILED tests/test_gateway_turn_unit.py::test_complete_outcome_bills_the_whole_prompt_on_a_cached_anthropic_turn FAILED tests/test_gateway_turn_unit.py::test_complete_outcome_does_not_double_count_openai_responses_cached_tokens FAILED tests/test_gateway_turn_unit.py::test_complete_outcome_does_not_double_count_openai_cached_tokens FAILED tests/test_stats_tool_schema_savings_are_honest.py::test_stats_publishes_the_cost_weighted_saving_next_to_the_token_saving FAILED tests/test_stats_tool_schema_savings_are_honest.py::test_tool_schema_layer_carries_its_dollar_value $ ruff check / ruff format --check (0.16.7) on the changed files All checks passed! / 5 files already formatted $ mypy --ignore-missing-imports headroom/proxy/gateway_turn.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS, Python 3.12, Headroom built from this branch, licensed extensions installed, Kong 3.9 (DB-less, `kong-plugin-headroom`) in Docker in front of `api.anthropic.com`, Claude Code 2.1.273 as the client - Exact command / steps: same stack before and after the change; Claude Code fixes three planted bugs in a small Python project through Kong (5 model calls), then `GET /stats` - Observed result, one run on the fixed build: ```text billed prompt 113,490 = cache_read 86,430 + cache_write 27,050 + uncached 10 old formula would have reported: 100.0% (denominator was just the 10 uncached tokens) fixed formula reports: 32.52% tokens.cost_weighted_savings_percent: 27.7% savings.by_layer.tool_search: 54,590 tokens, $0.043 cost card: paid $0.10, saved $0.04 (27.7%) ``` - On a longer real session through the same stack (110 calls, Opus + Sonnet), the old figure was 97%; recomputing that session's rows at provider cache rates gives ~15% of input tokens and ~13% of spend - Not tested: non-Anthropic gateways end to end (OpenAI and Responses shapes are covered by unit tests over `normalize_usage` / `complete_outcome`, not a live gateway run) ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: reported numbers only. Gateway users' "tokens saved %" drops sharply (97% -> ~15% on a long session) because it was wrong in our favour. Worth a release note so nobody reads it as a regression - Kill switch / disable path: n/a - Unsafe override required: no - Qualification impact: `tokens.input` on the gateway path now counts the cached prefix, so any external dashboard differencing it against `original_tokens` sees a step change - Rollback path: revert this PR ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Found while building a Kong + Headroom demo: the dashboard claimed 90%+ tokens saved while the invoice moved about 13%. The dollar figures were already cache-aware; only the token percentage was wrong, plus the missing per-layer dollars. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
c02929846c |
fix(proxy): make the output shaper actually reduce output in the default mode (#3773)
## What's wrong
**`HEADROOM_OUTPUT_SHAPER=1` reduced nothing.**
`HEADROOM_MODE` defaults to `cache` (`server.py:5707`) and the `coding`
profile pins it. In cache mode `resolve_verbosity_level` forced the
level to `0` unconditionally, deliberately outranking even an explicit
`HEADROOM_VERBOSITY_LEVEL`:
```python
if not settings.steering_enabled:
# It deliberately outranks the manual override below: a level set in the
# environment must not reintroduce a prefix mutation the mode exists
# to prevent.
return 0, "cache_mode"
```
So for effectively every deployment, the shaper turned on, labelled its
requests, and left the body untouched. Both documented knobs were
silently ignored, with no warning and no override.
## Why the ban was too broad
It guards the right hazard with the wrong instrument.
What costs a prefix cache is the steering block **changing** — appearing
on a conversation already in flight, or moving between levels — because
it lives in the `system` array, inside the cached prefix of every
message-level `cache_control` breakpoint a harness sets.
What costs nothing is the block **being there**. `shape_request` applies
it on every request with no turn-kind gating, and the text is
byte-stable per level and idempotent via the `<headroom_output_shaping>`
sentinel. A level fixed at startup lands in turn 1's prefix and is
byte-identical on every turn after.
The file already said this and then contradicted itself: the module
docstring claims the block is appended "after any `cache_control`
breakpoint, so the provider prefix cache is preserved", while
`steering_allowed_for` says steering "writes into that key".
## Measured
Anthropic, `claude-sonnet-4-5`, 9.1k-token cached system prompt, four
sequential turns, `cache_control` on both the system block and the
trailing message. Arms isolated by a nonce — without that the second arm
reads the first's cache and both cold-start numbers are meaningless.
| turn | control write / read | L2 write / read |
|---|---|---|
| 1 | 9124 / 0 | 9206 / 0 |
| 2 | 100 / 9124 | 157 / 9206 |
| 3 | 126 / 9224 | 137 / 9363 |
| 4 | 161 / 9350 | 132 / 9500 |
**Identical caching pattern. No bust on any turn.** The whole difference
is the block's own ~82 tokens riding in the cached prefix — about
$0.0003 once and $0.000025 per turn at Sonnet rates.
Against that, a separate paired A/B on a long explain-with-code turn (3
pairs, non-overlapping ranges): **2375 → 1663 output tokens, −30%**, ≈
$0.011/turn. Roughly 450x the cache cost.
## The change
`resolve_verbosity_level` allows through the two sources that **cannot
move**:
| source | when |
|---|---|
| `env_pinned` | `HEADROOM_VERBOSITY_LEVEL` set explicitly |
| `cache_mode_default` | otherwise — the settings level, fixed at
startup |
The second is what makes `HEADROOM_OUTPUT_SHAPER=1` sufficient alone.
The shaper is opt-in (the `proxy_output_shaper` rollout spec
deliberately does not default-enable it), so an enabled shaper is
already an explicit request.
The controller and learned profile stay out of cache mode entirely —
they rewrite their files while conversations are open. Skipping the
lookup also spares the default mode two filesystem stats per request.
Autotune being ignored is warned once, naming `HEADROOM_MODE=token`.
**Default level 3 → 2**, named `DEFAULT_VERBOSITY_LEVEL`. L3 instructs
the model to drop content ("give conclusions only; omit rationale") —
the same reasoning the rollout spec gives for not defaulting the shaper
on at all. L2 only forbids ceremony and restatement, and is what the
−30% above was measured at. This also settles an existing inconsistency:
`from_env` defaulted to 3 but fell back to **2** on a malformed value.
## ⚠️ Behaviour changes
Three, all disclosed:
1. **Cache mode with the shaper on now steers.** That is the point.
Blast radius is exactly "operators who set `HEADROOM_OUTPUT_SHAPER=1`" —
the people who asked for output shaping and were not getting it.
2. **A pinned `HEADROOM_VERBOSITY_LEVEL` is now honoured in cache
mode**, where it was previously discarded.
3. **The default level drops 3 → 2**, including in token mode. Existing
token-mode users with the shaper on and no level set get *less*
aggressive shaping: lower savings, lower risk of the terseness reading
as regression.
Five existing tests encoded the old policy and are inverted here
deliberately, including the gateway-contract one
(`test_cache_mode_labels_but_never_steers` →
`test_cache_mode_steers_at_the_default_level`), which verifies the
change end-to-end through the real `/v1/compress` path rather than only
at the unit level.
`/stats` needed no change: it builds the same settings the handlers do
and reads the resolved level, so it follows automatically.
## Tests
New coverage:
- `HEADROOM_OUTPUT_SHAPER=1` alone resolves `(2, "cache_mode_default")`
- a pinned level resolves `(4, "env_pinned")` in cache mode
- the learned profile and the controller are both ignored there
- the ignored-autotune warning fires exactly once and names
`HEADROOM_MODE=token`
- cache mode consults **no workspace files** (asserted by making
`workspace_dir()` raise)
- **ten turns of a growing conversation produce a byte-identical
`system` array**, with the client's own `cache_control` still on the
first block — the cache-safety claim asserted rather than argued
340 tests pass across shaper / verbosity / steering / output_reduction /
cache_stability / output_savings / gateway_turn. `ruff check`, `ruff
format --check` (0.16.7) and `mypy headroom --ignore-missing-imports`
clean.
Local full-suite runs error en masse in a way unrelated to this change
(files that error in the full run pass individually), so CI is the
arbiter on the complete suite.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
|
a9757c9c7e |
fix(tool-search): make the core-tools override authoritative and the warning recurrent (#3745)
Three defects found while tracing why a customer benchmark showed **+108% cost** with Headroom in front of Claude Code. Follow-on to #3739. ## 1. The override was not authoritative on the OpenAI path `helpers.py` unioned `terminal` into the resident set *after* applying the operator's override. Setting `HEADROOM_TOOL_SEARCH_CORE_TOOLS` to a list that excluded `terminal` got it back anyway, silently — the one tool most likely to carry a large schema. `resolved_core_tools` now takes provider additions as an `extra` argument folded into the **default** only. An explicit override replaces everything, provider additions included. That is what an override means. ## 2. The warning armed once per process The "client tool search is disabled" warning was already `logger.warning` — the defect was that it armed once per process. A long-lived proxy reported the condition on the first request and went quiet forever. **Twelve benchmark runs produced a single log line**, which is why this took so long to find. It now re-arms hourly. ## 3. One knob, two spellings Core read `HEADROOM_TOOL_SEARCH_CORE_TOOLS`; the plugin read `HEADROOM_TOOL_SEARCH_CORE`. Both now resolve to the same setting — canonical wins, legacy is honoured and warns once naming the replacement. No existing configuration breaks. ## Testing 40 tests in `test_tool_search_client_deferral.py`, 1115 passing across the related suites. ruff clean. ## Note on defaults This PR does **not** change what is resident by default. Claude Code defers all its built-ins as of v2.1.69, and matching that is the obvious next step — but there are open client-side bugs against it (Grep/Glob disappearing on macOS/Linux under `ENABLE_TOOL_SEARCH=true`, strict endpoints 400ing the first request, signed-thinking-block errors on Opus 4.8). Those are upstream, not ours; they are the reason this PR makes the knob a real off-switch first. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ### Update after self-review Two live bugs, **one of them introduced by this PR**: ``` HEADROOM_TOOL_SEARCH_CORE_TOOLS="bash, read, terminal" -> resident set: [' read', ' terminal', 'bash'] -> read: DEFERRED, terminal: DEFERRED ``` The override never stripped whitespace — `part.strip()` gated only the emptiness test, while the key function lowercases and strips leading underscores but not spaces. So the natural spelling of a list defers exactly the tools you asked to pin. This was latent before: the OpenAI injector re-unioned its resident names *after* the override and masked it. **Removing that union in this PR is what made it bite.** Separately, an override could defer the client's own `ToolSearch` — Claude Code reaches tools held in a local registry through it and nothing else can, so an override that omitted it orphaned that whole category. A search-tool floor now survives every override. Also: the hourly window was stamped only when the hint **fired**, so once the operator fixed the condition the stamp stopped advancing and every later request paid the full O(tools) scan for the life of the process — a hot-path regression this PR added. It now throttles *scans*, and claiming the slot before scanning collapses the old two-call protocol. The legacy-var flag is now lock-guarded with a reset helper, the gateway reference hook honours `resolved_core_tools()` instead of the raw default (it ignored the override entirely), and the hint test no longer monkeypatches stdlib `time.monotonic` process-wide. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
be00a798c3 |
feat(proxy): accept the OpenAI Responses shape on the gateway turn (#3661)
## Description
`POST /v1/compress` accepted two wire shapes that both carry a
`messages` list: Anthropic Messages and OpenAI Chat Completions. Codex
speaks the third one — OpenAI Responses — where the transcript is
`input` (typed items) and the system prompt is `instructions`. That body
was rejected outright:
```
HTTP 400 {"error":{"type":"invalid_request","message":"Missing required field: messages"}}
```
So a gateway (Kong, LiteLLM) in front of Codex failed open on every turn
and Headroom did nothing for it. This adds the third shape.
Closes #
## Type of Change
- [x] New feature (non-breaking change that adds functionality)
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
**A view, not a translation.** A Responses transcript carries fields the
provider hands back to itself — `reasoning.encrypted_content` above all,
plus `call_id` pairing and Codex's
`internal_chat_message_metadata_passthrough`. Translating to chat
messages and back loses them *silently*: the request still looks valid.
`_responses_input_to_waste_messages` already carries that warning —
telemetry-only, "never used as a compression input".
So `headroom/proxy/gateway_responses.py` builds a view:
- Every compressible text slot becomes exactly one pseudo message. Never
joined, so putting the text back never has to guess how to split it.
- Items that are not text slots produce no message at all. The
compressor cannot see or rewrite them.
- Write-back is positional; a view that does not line up forwards the
original rather than guessing.
**Call inputs are slots too.** Measured across 51 real Codex sessions:
`custom_tool_call.input` is 8.9% of transcript bytes and is resent every
turn. `function_call.arguments` is a JSON document the provider parses,
so it is rewritten only when the replacement still parses (`_json_safe`)
— it is 0.3% of bytes and real ones carry encrypted payloads, so little
is lost when the guard refuses.
**Three bugs this turned up**, each with a test named after it:
| | |
|---|---|
| `build_provider_body` added `messages` unconditionally | the
`x-headroom-bypass` header — the path whose whole job is to change
nothing — turned every Codex request into a provider 400 |
| `_redrive_payload` did the same on the second leg | every re-drive on
a Responses turn was malformed, CCR and tool-router alike |
| `create_ccr_tool_definition` fell through to the nested chat shape |
CCR injected a tool the provider rejects, on exactly the turns where
markers made it necessary |
The response half needed nothing: `normalize_usage` already read
Responses usage via `input_tokens_details.cached_tokens`.
`wire_shape` is deliberately separate from `provider` on `PendingTurn`:
`provider` picks pricing, tokenizer and the upstream the gateway is
advised to call, and all three want `"openai"` for a Responses turn
while the tool-call shape and the re-drive body want
`"openai_responses"`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```
tests/gateway + test_gateway_turn_unit + test_gateway_sidecar_ports
319 passed, 4 skipped
tests -k "ccr or compress"
2307 passed, 72 skipped, 10860 deselected
ruff check All checks passed!
mypy Success: no issues found in 1 source file
```
**Verified against the live OpenAI Responses API**, because a malformed
body only shows up when a real provider validates it:
```
=== compressed Codex body ===
compress 2,436 -> 1,632 tokens shape input=True messages=False
provider HTTP 200
=== tool-router re-drive (both legs) ===
compress tools -> 1 names=['search_tools']
provider round 0 tool_calls=['search_tools']
RE-DRIVE 1: input=True messages=False tools=11 shape_ok=True
provider round 1 tool_calls=['zz_obscure_capability_47']
action=done after 1 re-drive(s)
=== CCR re-drive ===
compress 85,105 -> 41,903 ccr_hashes=1 tools=['headroom_retrieve']
provider tool_calls=['headroom_retrieve']
RE-DRIVE 1: input=True messages=False shape_ok=True
...
answer: 'ZQ-7741-KLM' <- buried at line 2500 of a 4000-line folded output
```
Chat Completions and Anthropic bodies were re-checked live alongside
Responses and are byte-identical in behaviour.
## Known limitations
- Codex streams by default, and both plugins set `can_redrive =
allow_redrive and not stream`. Tool-router and CCR savings therefore do
not apply to streaming turns; schema compaction and content compression
still do.
- `reasoning` items stay opaque by design.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
|
aceff2ea9d |
fix(proxy): gateway turn fixes found by benchmarking behind a gateway (#3619)
## Description Fixes five defects found by running real Claude Code sessions through a gateway that uses the `/v1/compress` turn contract (#3605). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - **Prompt text is never compressed on replaying callers.** The coding profile compresses user messages, and on turn 1 that rewrote a plain agent's task ("Fix this bug in the library source" became "Fix bug in source", and test ids lost their `tests/test_` prefix). When `prefix_replay_guaranteed` is set, `content_router` protects the opening user messages and the newest user message, for string and list content alike. The one exception is a newest user message that answers a fenced shell command in the assistant turn before it: that is a text harness's tool observation and still compresses. Tool results still compress, and cache freeze is unchanged. - **Anthropic's four cache breakpoint limit.** One benchmark request failed with `max 4 cache_control blocks, found 5`. Session turns in the OpenAI handler now re-place markers with `normalize_message_cache_control`. A last-stop guard, `enforce_cache_breakpoint_budget`, runs in the gateway extension for Anthropic bodies: it re-places markers, then drops the oldest extras, and logs `event=cache_breakpoint_budget` when it repairs. - **Failure relay without a body.** `POST /v1/compress/response` rejected an upstream 4xx/5xx without a response body as `missing_response`. A body is now required only on 2xx. - **Usage totals after a re-drive.** The done payload carries `billed_usage` summed across rounds. After one or more re-drives, `response` is never null: it is the hook's replacement or the latest provider response (including when the hook hands back that same object), with `usage` set to the totals in its own key shape (chat or Anthropic). Turns with no re-drive keep `response: null`. Before, the client saw the last round only. - **Cold Kompress stall.** When native Kompress init is deferred at startup, a daemon thread preloads it after a short delay. `HEADROOM_KOMPRESS_WARMUP=0` disables it. It is off under pytest and skipped on glibc < 2.28 (the #1908 segfault hosts). - Docs: `proxy.mdx` and `wiki/proxy.md` describe the 2xx-only body rule and `billed_usage`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) on the changed modules - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_transforms_content_router.py tests/test_compress_session_mode.py \ tests/test_compress_route_tokenizer_by_model.py tests/gateway tests/test_proxy \ tests/test_kompress_preload_deferral.py 672 passed, 4 skipped in 228.49s $ uvx ruff@0.16.3 format --check <changed files> && uvx ruff@0.16.3 check <changed files> 12 files already formatted All checks passed! $ mypy <changed modules> Success: no issues found in 6 source files ``` New tests: 5 prompt-protection tests in `test_transforms_content_router.py` (3 fail without the fix), `tests/gateway/test_cache_breakpoint_budget.py`, failure-relay and billed-usage tests in `test_redrive_loop.py`, `billed_usage` in the response-half contract test, `tests/test_proxy/test_kompress_background_warmup.py`. ## Real Behavior Proof - Environment: headroom proxy on this branch behind a gateway, Claude Code with claude-sonnet-5, real Anthropic calls. - Exact command / steps: 8 smoke sessions (32 calls) on SWE-bench-style tasks, with a spend logger recording every provider round. - Observed result: 0 errors. Task statements reached the provider verbatim. On every re-driven call (2 and 3 rounds) the client's usage equalled the sum of the per-round provider usage. Kompress warmed in the background in about 4 s at startup; in a replay of the slow session the first code-aware turn went from 5.4 s to 1.3 s. - Not tested: glibc < 2.28 hosts (warm-up is skipped there by design). The original body behind the 5-breakpoint error was not captured, so the guard is covered by unit tests rather than a live repro. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: stable. - Stable/default behavior changed: yes. Prompt text on replaying callers is no longer compressed, the done payload adds `billed_usage`, and Kompress preloads in the background when deferred. - Kill switch / disable path: `HEADROOM_KOMPRESS_WARMUP=0` for the warm-up. The other changes are correctness fixes with no switch. - Unsafe override required: no. - Qualification impact: none expected; the breakpoint guard only acts on bodies that would otherwise get a 400. - Rollback path: revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Open follow-up, not in this PR: steady-state code-aware compression falls back to Kompress at 0.3 to 1.3 s per tool result even when it saves nothing. A gate for that is worth a separate change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
d9757dcc6b |
fix(proxy): gateway turn contract for /v1/compress behind a compress-turn seam (#3605)
## Description Adds a gateway turn contract to `POST /v1/compress`, allowing an API gateway to submit a complete provider request and later report the provider response. Requests without a top-level `gateway` object remain unchanged. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Adds the request half to `/v1/compress` and the response half at `/v1/compress/response`, including bounded pending turns, idempotent rounds, usage relay, and redrive obligations. - Adds a reusable six-stage compress-turn extension seam and turn-hook priority/provider-header support. - Keeps native tool-schema deferral in the licensed extension while sharing protocol-shape helpers with core. - Accepts Anthropic, OpenAI chat, OpenAI Responses, and Kong-shaped usage payloads. - Documents the gateway contract in `docs/content/docs/proxy.mdx` and `wiki/proxy.md`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text tests/gateway + gateway unit + turn-hook seam + test_issue_746_tool_search: 358 passed ruff 0.16.3 lint/format and mypy on touched modules: clean Kong 3.9 Docker e2e (HEADROOM_KONG_E2E=1): 4 passed ``` ## Real Behavior Proof - Environment: Kong 3.9 Docker e2e and real-provider Claude Code through Kong on SWE-bench-mini. - Exact command / steps: `HEADROOM_KONG_E2E=1` gateway e2e; real-provider traffic through the gateway contract with usage relay and licensed native deferral. - Observed result: four Kong e2e tests passed; prefix cache remained intact with zero busts; relayed usage matched Anthropic token-for-token; licensed tool-search deferral reduced cache-read tokens per turn by 27–30%. - Not tested: production rollout at scale. ## Runtime Rollout Safety - Rollout-managed feature(s): gateway contract and in-process pending-turn registry. - Minimum rollout channel: controlled gateway deployment before broader enablement. - Stable/default behavior changed: only requests containing a top-level `gateway` object enter the new contract; ordinary `/v1/compress` bodies are unchanged. - Kill switch / disable path: set `HEADROOM_GATEWAY_CONTRACT=0`. - Unsafe override required: no. - Qualification impact: gateway callers must redeem advertised `redrive` and `relay_usage` obligations. - Rollback path: disable the contract with `HEADROOM_GATEWAY_CONTRACT=0` or revert the feature commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable; this is an API contract change. ## Additional Notes Pending turns are process-local and bounded by `HEADROOM_GATEWAY_TURN_TTL_SECONDS`, `HEADROOM_GATEWAY_MAX_PENDING_TURNS`, and `HEADROOM_GATEWAY_MAX_REDRIVES`. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |