Commit Graph
14 Commits
Author SHA1 Message Date
d6df127d57 feat(compress): accept Gemini and Bedrock Converse bodies on the gateway turn (#4679)
## Description

`/v1/compress` in gateway mode (a request with a `gateway` block) now
accepts two more native request bodies:
- Gemini `generateContent`
- Bedrock Converse

It compresses their text slots and returns `body` in the same native
shape. This lets a gateway like LiteLLM compress native Gemini and
Bedrock passthrough traffic, which it couldn't before.

`config.mode: "ccr"` is rejected for these shapes with a 400. Every
other mode is marker-free for them.

(Was stacked on #4617, which has merged; the branch now carries current
`main`.)

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

### How

This works the same way `gateway_responses` does for Codex: it builds a
view instead of translating the body.
- Every compressible text slot becomes one chat-shaped pseudo message.
- Compressed text is written back by its path in the body.
- Everything else stays byte-identical: `thoughtSignature`, thought
parts, `functionCall`, `inlineData`, `toolUse`, `cachePoint`,
`reasoningContent`, `tools`/`toolConfig`.
- Tool calls appear in the view, so tool-name rules still apply. `Read`
stays verbatim, for example.
- A JSON slot is only written back if it still parses to the same JSON
type.
- If the compressed list doesn't line up with the slots, the original
transcript is forwarded and a warning is logged.

### CCR on native bodies (review fix)

A native body can't carry `headroom_retrieve`. Its `tools`/`toolConfig`
are forwarded untouched, and `create_ccr_tool_definition` only builds
chat, Anthropic and Responses shapes. It also never gets a re-drive. So
a CCR marker in a native body points at content the model can never
fetch. Before this fix, `config.mode: "ccr"` on a Gemini or Converse
gateway turn did exactly that: log output came back as a marker with
`ccr_hashes: [<hash>]` and `obligations: []`.

- `handle_compress` now answers `ccr` on a native view with `400
invalid_request`. The message names the shape, explains why, and lists
the supported modes.
- The check runs where the mode is chosen, right after `attach_view` has
recorded the shape. That is before any compression, session work or CCR
store write.
- **The rejection does not depend on `gateway.can_redrive`.** Even a
gateway that can re-drive has no way to put the retrieval tool into a
native request, so there is no capability under which native `ccr`
works.
- **Without a `gateway` block nothing changes.** A Gemini body is
already a 400 (no `messages`). A Converse body goes through the plain
messages path, where its untyped blocks are opaque: 0 tokens saved, no
markers in any mode, `ccr` included (checked). So the rule is
consistent: reject `ccr` wherever Headroom reads a native shape.
- The bypass header still returns the body untouched with no
compression, in any mode.
- I checked every other mode on the same marker-producing input
(default, `lossless`, `lossy_inline`, `lossless_then_lossy`): no marker
and no hash in any of them. These modes build pipelines with
`ccr_inject_marker=False, ccr_enabled=False`. Stale-Read notices are
hashless in marker-free modes since #4678.
- **Second defect, same family:** with `HEADROOM_DEDUPE=1`, cross-turn
dedup wrote `[↑NL same as msg 4]` into a Converse tool result. That `4`
counts view messages (system blocks, tool calls, result slots), not the
caller's `messages`, so it named the wrong message. Native views now set
`cross_turn_dedup_recoverable=False`, as `lossless` mode already did.
- The `not self.native_shape` guards on CCR arming and re-drive in
`GatewayTurn.finish` stay as defence in depth.

### Caller contract

**Gemini**
- Send top-level `contents` as a list, with no `messages` or `input`.
- `systemInstruction`/`system_instruction`, `tools` and
`generationConfig` are optional and pass through.

**Converse**
- Send `messages` where every message has role `user` or `assistant`,
and every content block has no `type` key.
- At least one block must use a Converse key: `text`, `toolUse`,
`toolResult`, `cachePoint`, and so on.
- `system` is optional, as a list of blocks.

**Both shapes**
- `config.mode`: omit it (marker-free default), or use `lossless`,
`lossy_inline` or `lossless_then_lossy`. `ccr` is a 400.
- Only gateway turns get a view. Without a `gateway` block the endpoint
behaves exactly as before. A client-sent `_headroom_native_view` key is
dropped, and `BODY_CONTROL_KEYS` strips the real one before the body is
returned.
- `config.frozen_message_count` and `config.protect_recent` count native
messages: Gemini `contents` entries and Converse `messages`.
`session.frozen_message_count` comes back in native units. The
`/v1/usage` reply's `frozen_message_count` still counts view messages,
and the docs say so.
- No re-drive is ever armed, so `obligations` never contains `redrive`,
and request hooks run stream-safe-only. Tool compaction and output
steering are skipped.
- Relayed usage can be:
- Gemini `usageMetadata`, where `promptTokenCount` already includes
`cachedContentTokenCount`.
- Converse `usage`, where `inputTokens` excludes `cacheReadInputTokens`
and `cacheWriteInputTokens`.
  - The whole provider response that carries either.
- Every gateway answer, fail-open ones included, carries `wire_shape`:
`gemini`, `converse`, `openai_responses` or `messages`. A caller that
sent a native body and gets anything else, or no field at all, is
talking to an older Headroom and should forward its original body.
- Top-level `messages` in the answer is the Converse list for Converse
and the chat-shaped view for Gemini.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`): no new errors; the 8
reported are in `ccr/mcp_server.py` and `cache/prefix_tracker.py` and
are identical on `main`
- [x] New tests added for new functionality
- [x] Manual testing performed (local replay, below)

New in `tests/gateway/test_native_shapes.py`:
- `test_ccr_mode_is_rejected_for_native_bodies[gemini|converse]`: 400,
`invalid_request`, message asserted.
- `test_other_modes_forward_native_bodies_without_ccr_markers`: 4 modes
x 2 shapes, on log output large enough to produce a marker. Checks no
hash, no marker, the shape echoed, non-message keys byte-identical and
the native list mapped back.
- `test_ccr_still_works_on_a_chat_gateway_turn`: the same log output in
a chat body with `ccr` still carries a marker. This proves the input
above is marker-producing, and that chat `ccr` didn't regress.
- `test_native_bodies_get_no_dedup_pointer_into_view_messages`.
- `test_native_turns_are_never_asked_to_redrive` now runs in the default
mode, because `ccr` is a 400.

### Test Output

```text
# before the fix (new tests only, rest of the file passing)
FAILED tests/gateway/test_native_shapes.py::test_ccr_mode_is_rejected_for_native_bodies[gemini]
FAILED tests/gateway/test_native_shapes.py::test_ccr_mode_is_rejected_for_native_bodies[converse]
FAILED tests/gateway/test_native_shapes.py::test_native_bodies_get_no_dedup_pointer_into_view_messages
3 failed, 25 passed

# after
tests/gateway/test_native_shapes.py: 28 passed

# tests/gateway tests/test_gateway_turn_unit.py tests/test_compress_api.py
# tests/test_proxy_compress_endpoint.py tests/test_compress_session_mode.py
# tests/test_compress_session_shared.py tests/test_compress_passthrough.py tests/test_compress_failure.py
1 failed, 507 passed, 4 skipped
# the one failure, test_lossy_inline_emits_no_ccr_markers[1-json_minifier], fails identically on main fda75de5

uvx ruff@0.17.3 check / format --check (changed files): All checks passed! / 3 files already formatted
```

## Real Behavior Proof

These are local synthetic replays. They are not provider qualification.

- Environment: macOS, Python venv, branch head `530c20aa`. Headroom runs
in-process via `create_app(ProxyConfig(optimize=True,
cache_enabled=False, rate_limit_enabled=False,
cost_tracking_enabled=False, log_requests=False, image_optimize=False))`
behind `fastapi.testclient.TestClient` from a loopback client. The
"provider" is an `httpx.MockTransport` that records each body it
receives and answers with a Gemini `generateContent` or Bedrock Converse
response, including usage. `LITELLM_LOCAL_MODEL_COST_MAP=True`. No real
provider calls.
- Exact command / steps: `LITELLM_LOCAL_MODEL_COST_MAP=True python -I -c
"import sys; sys.path.insert(0,'<worktree>');
exec(open('native_replay.py').read())"`. For each shape the script does
three things:
1. POSTs the native body plus `gateway: {can_redrive, session_affinity,
can_relay_response: true}` to `/v1/compress`.
2. Sends the returned `body` through the mock upstream at the real
Gemini or Bedrock URL.
3. POSTs the provider's whole response to `/v1/compress/response` with
the `turn_id`.

It then sends both native bodies again with `config.mode: "ccr"`. The
tool output is the 399-line grep listing used in the tests. The `ccr`
requests use 3000 log lines, which produce a marker in `ccr` mode.
- Observed result:

```text
[gemini] request half: wire_shape=gemini obligations=['relay_usage'] tokens 7246->3863 ccr_hashes=[] marker_in_body=False
[gemini] all 399 grep lines still readable in the forwarded body=True
[gemini] non-message keys unchanged=True first two contents unchanged=True
[gemini] provider got 16125 bytes (raw 29694), control keys forwarded=[], host=generativelanguage.googleapis.com
[gemini] response half: 200 {'action': 'done', ..., 'billed_usage': {'input_tokens': 900, 'output_tokens': 12, 'cache_read_input_tokens': 512}}
[converse] request half: wire_shape=converse obligations=['relay_usage'] tokens 8823->4843 ccr_hashes=[] marker_in_body=False
[converse] all 399 grep lines still readable in the forwarded body=True
[converse] non-message keys unchanged=True first two messages unchanged=True
[converse] provider got 16184 bytes (raw 29761), control keys forwarded=[], host=bedrock-runtime.us-west-2.amazonaws.com
[converse] response half: 200 {'action': 'done', ..., 'billed_usage': {'input_tokens': 300, 'output_tokens': 12, 'cache_read_input_tokens': 600, 'cache_creation_input_tokens': 0}}
[gemini ccr] 400 {'error': {'type': 'invalid_request', 'message': "config.mode='ccr' is not supported for gemini bodies: Headroom cannot add its retrieval tool to a native Gemini or Bedrock Converse request, so the model could not resolve a CCR marker. Use lossless, lossy_inline or lossless_then_lossy, or omit config.mode for the default marker-free mode."}}
[converse ccr] 400 {'error': {'type': 'invalid_request', 'message': "config.mode='ccr' is not supported for converse bodies: ..."}}
provider calls: 2
```

Before the fix, the same `ccr` requests returned 200 with `ccr_hashes`
of length 1, a `hash=` marker in the forwarded body, and `obligations:
[]`.
- Not tested:
- live Gemini (AI Studio or Vertex) and live Bedrock Converse: the
provider was mocked, so provider acceptance of the rewritten bodies is
unverified;
  - real Kong `ai-proxy` or a real LiteLLM gateway in front of Headroom;
- streaming callers (`streamGenerateContent`, `ConverseStream`): the
request half is identical, but no streamed response was relayed;
  - Gemini `thoughtsTokenCount` in output accounting;
  - session mode with a Redis store for native bodies;
- multimodal parts (`inlineData`, `image`, `document`) beyond their
byte-identical pass-through in unit tests.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. Native shapes only activate for
gateway-mode `/v1/compress` callers that send a `gateway` block and a
native body.
- Minimum rollout channel: n/a
- Stable/default behavior changed: no for existing callers. Chat,
Anthropic and Responses bodies are unchanged. A Converse body without a
`gateway` block goes through the plain path as before. Before this PR, a
native body with a `gateway` block was misread as chat (Converse) or
rejected (Gemini).
- Kill switch / disable path: the gateway omits the `gateway` block, or
forwards its original body when `wire_shape` doesn't match.
- Unsafe override required: no
- Qualification impact: none for existing paths
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Known follow-ups, not in this PR:
- A `functionResponse.response` with any string field never compresses
its nested dict values. The JSON slot is only used when there are no
string fields. This is a missed saving, not a correctness bug.
- `handlers/gemini.py` (the proxy path) and `gateway_native.py` map
Gemini parts with different rules, and their part-key lists are kept in
step by hand. They should share one constant.
- `protect_recent` translation and stream-safe hook gating for native
turns have no dedicated test.
- Gemini `thoughtsTokenCount` is not added to output tokens when usage
is relayed, so output savings attribution undercounts for thinking
models.
- This is not specific to native shapes: the marker-free default folds
very repetitive logs to a single line (for example `[3000 lines omitted:
3000 INFO]`) on chat bodies too, with no retrieval path.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-15 01:52:50 +05:30
94a99ed20f feat(proxy): give turn hooks one conversation id per client conversation (#4496)
## Description

Plugins that pin wording per conversation (headroom-condense, and
through it skill-search and tool-search) cannot see a real conversation
id. Today:

- **Claude:** they hash the first typed prompt, so two sessions that
open with the same words share a key.
- **Codex:** they get no key at all, so every new skill or MCP
description waits out the one-hour hold.

This PR adds an optional `TurnContext.conversation` that core fills from
the client's own conversation id, so hooks can pin per real
conversation. It is item #4 of the compression-gaps plan; the plugin
work for #3, #6, #8 and #9 needs it to stay cache-safe.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **New field:** `TurnContext.conversation: str | None = None`. It is an
opaque key that stays the same for every turn and retry of one client
conversation, or `None` when the client names none. Hooks get a digest,
never the raw id.

  | Path | Source |
  | --- | --- |
| `/v1/messages` | sha256 of `X-Claude-Code-Session-Id` (Claude Code
sends it on every request), falling back to `X-Headroom-Session-Id`;
each source has its own namespace |
| `/v1/responses` (HTTP and WebSocket) | the new
`turn_conversation_key`, computed from the body before compression and
passed through `_compress_openai_responses_payload_in_executor` |
  | `/v1/chat/completions` | `X-Headroom-Session-Id` only |
| `/v1/compress` (gateway) | the gateway session id; `None` without one
|

- **Responses key.** `turn_conversation_key` uses the same identity and
digest as `savings_conversation_key` (`thread_id` before `session_id`,
then `client_metadata.*`, then the session header). Unlike the savings
key, it also applies on `previous_response_id` turns and on bodies that
name a server-side `conversation` (a string, or an object with an `id`),
so the key does not drop out after the first WebSocket frame.
`savings_conversation_key` keeps its existing exclusions.
- **Gateway seam.** `CompressTurn.prepare()` is a third-party seam, so
its signature is unchanged. Instead, the handler sets an optional
writable `conversation` attribute on the turn before calling `prepare`.
This is documented in `compress_turn.py`.
- **No new ids.** Nothing is generated. The per-connection WebSocket
uuid is not used, because a reconnect would change it.
- **One path left out.** The catch-all passthrough Responses route still
passes `None`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`): 3 errors, all in files
this PR does not touch (embedders, model_metadata, langchain)
- [x] New tests added for new functionality
- [x] Manual testing performed

- **New tests:**
  - `tests/test_turn_context_conversation.py`:
- Anthropic: the same session gives one key, a different session with
the same prompt gives a different key, the fallback header gets its own
key, and no header gives `None`.
- Responses HTTP: the same thread gives one key, including on a
`previous_response_id` turn, and a different thread gives a different
key.
- Responses server-side `conversation`: string and object forms give one
key across turns; a different conversation gives a different key; the
savings key stays `None`.
- Responses WebSocket: first and later frames share one key; a different
thread gets a different key.
    - Chat Completions: covered the same way.
- `tests/gateway/test_contract_request_half.py`: per-session gateway
keys, and `None` without a session.
  - Every new test fails on `main`.
- **Updated test:** `tests/test_proxy_openai.py` now pins
`conversation=None` for a body with no ids.

### Test Output

```text
pytest (turn hooks, gateway, Codex WS lifecycle, proxy openai, conversation savings, passthrough, hook usage): 411 passed, 5 skipped
after the server-side conversation fix, on the synced head: 101 passed (conversation, gateway, Responses suites)
ruff@0.17.0 check / format --check: clean
mypy headroom --ignore-missing-imports: 3 errors, none in changed files
```

`tests/test_turn_hook_usage.py` needs
`LITELLM_LOCAL_MODEL_COST_MAP=True` in a fresh venv; that is an existing
litellm price-map fetch, and it fails the same way on `main`.

## Real Behavior Proof

- Environment:
  - macOS 27.0.1 (Darwin 27.0.0, arm64), Python 3.12.13.
- The proxy ran from this branch in a fresh `uv` venv (`uv venv -p 3.12
&& uv pip install -e ".[dev]"`) on 127.0.0.1:9100, with default
`ProxyConfig` (optimize on).
- Upstream was the real Anthropic API, with my own API key passed by
environment variable.
- Client was Claude Code 2.1.311 (`claude --version`) with `--model
claude-haiku-5-5`.
- Exact command / steps:
1. Start the proxy with a scratch-only launcher (not committed). It
calls `register_turn_hook()` with a hook whose `on_request` appends
`ctx.provider`, `ctx.model`, `ctx.conversation` and `len(ctx.messages)`
to a JSONL file, then calls `run_server(ProxyConfig(host="127.0.0.1",
port=9100))`. Run it with `PYTHONSAFEPATH=1 HEADROOM_TELEMETRY=off
LITELLM_LOCAL_MODEL_COST_MAP=True python run_proxy.py hooks.jsonl`, from
a directory outside the repo.
2. Point Claude Code at the proxy: `export
ANTHROPIC_BASE_URL=http://127.0.0.1:9100 CLAUDE_CONFIG_DIR=<fresh empty
dir>`.
3. Start two sessions with the same prompt: `claude -p "say hi" --model
claude-haiku-5-5 --output-format json`, run twice. Note each
`session_id`.
4. Add two more turns to session 1: `claude -p "now say bye" --resume
<session 1 id> --model claude-haiku-5-5`, then `claude -p "one more word
please" --resume <session 1 id> --model claude-haiku-5-5`.
5. Read `hooks.jsonl`, and compare each key with sha256(`claude-code\0`
+ session_id).
- Observed result (from `hooks.jsonl`, keys truncated):

  | Session | Turn | Messages | `ctx.conversation` |
  | --- | --- | --- | --- |
  | `305d64a9…` | first | 2 | `306306eeb9bc164b…` |
  | `4e50609f…` (same prompt) | first | 2 | `4a9e51bc7f8005f2…` |
  | `305d64a9…` | first `--resume` | 4 | `306306eeb9bc164b…` |
  | `305d64a9…` | second `--resume` | 6 | `306306eeb9bc164b…` |

Each key equals sha256(`claude-code\0<session id>`) of the session id
that Claude Code reported. All three runs answered normally, and the
proxy log had no errors.
- Not tested:
- Codex over Responses HTTP or WebSocket (unit and handler tests cover
these).
  - Chat Completions with `X-Headroom-Session-Id`.
  - The `/v1/compress` gateway path.
- The server-side `conversation` body form, live (covered by the handler
test).
  - The `X-Headroom-Session-Id` fallback on `/v1/messages`.
  - Claude Code `/compact` and subagents.
  - Bedrock and Vertex upstreams.
  - Linux and Windows.
- A real plugin consuming the key. That is wired in
headroomlabs-ai/headroom-condense#7; skill-search and tool-search
adoption comes next.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: stable.
- Stable/default behavior changed: no request or response bytes change.
Registered turn hooks receive one new optional field; hooks that do not
read it behave as before.
- Kill switch / disable path: none needed; a hook that should not use
the key ignores it (headroom-condense falls back to its previous key
when the field is `None` or absent).
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this PR. Plugins read the field with `getattr`,
so they keep working on cores without it.

Cache note: compare keys only within one endpoint, since the same header
gives different keys on `/v1/messages` and `/v1/responses`. Each key
comes from a client id that stays the same for the whole conversation
(Claude Code session ids survive retries, `--resume` and `/compact`;
Codex `thread_id` wins over `session_id`, so guardian and subagent
requests do not collide), and the Responses key is taken before
compression rewrites the body, so a pinning hook keeps the same pin on
every turn.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A (proxy-side change).

## Additional Notes

- "No new warnings" is unchecked because the run was not compared
warning by warning against `main`.
- Follow-ups: skill-search and tool-search switch to
`conversation_for(ctx)` once headroom-condense#7 is released.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


## Maintainer post-merge qualification
Published `819f82117a4737de621528e9c49aee6aeeb4a793` includes upstream
main `e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`. 28
conversation/Responses/per-frame-memory/disconnect tests pass;
Ruff/format/diff pass. Clean main merge; hook conversation policy
unchanged, no new code finding. Author explicitly reserves unchecked
self-review/readiness attestations; existing hold remains. Contributor
correctness logic was not rewritten. Existing Changes Requested decision
retained without duplicate findings. Scoped Windows validation uses the
existing development native extension, not a fresh build. Hosted
qualification is handled by maintainers; nothing merged.


### Maintainer current-main requalification

Exact head `965ff21ffbea9b446e5e39269812bb0d8c43c108`, upstream main
`ecbaf86d78cf52d185fbffa306157b9a4cc1b906`.

Mechanically updated from upstream/main
ecbaf86d78 with no conflicts and no
contributor conversation/ownership policy changes. Re-reviewed current
discussion and the integration delta. All 38 focused
conversation-context, Responses, per-frame WebSocket memory, disconnect
and Gemini tests pass; Ruff0.17.3 lint/format and whitespace checks
pass. Existing runtime evidence remains scoped; local tests use the
existing native extension, not a fresh build.

No new source blocker was found. The existing readiness action remains:
the author has explicitly reserved and still not checked the self-review
and ready-for-human-review attestations. They are not asserted on the
author's behalf. Preserve the existing Changes Requested readiness hold
until the author completes that action. Maintainers handle exact-head
hosted checks; no contributor CI action requested. Nothing merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `db69aa725d085d75b0881f0b2e1f2237480ee0c6`, upstream main
`8d3e9e5ca7cd645eb73084b848edbea5fc796e1a`.

Clean mechanical sync with upstream
main8d3e9e5ca7cd645eb73084b848edbea5fc796e1a. Re-reviewed
conversation-context integration and fresh author/Devin discussion
without changing contributor identity/ownership
policy.38conversation/Responses/per-frame-WebSocket/disconnect/Gemini
tests pass;10changed Python files pass Ruff0.17.3check/format and
whitespace checks. Existing runtime evidence remains scoped; copied
extension supports Python startup, not a fresh native build.

No new source blocker found. The current body still leaves author
self-review and ready-for-human-review attestations unchecked and
reserves those actions for the author. Preserve that existing readiness
hold; no boxes were ticked or attested on their behalf. This is not an
inherited-main CI defect. Maintainers handle exact-head hosted checks;
nothing merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `7411de5607beabfcbbd63b134795b0053da0e681`, upstream main
`7592e681459f14a2041eb5f524a658cb6ad1dd92`.

Clean current-main
sync7592e681459f14a2041eb5f524a658cb6ad1dd92,including
latestmemoryfixes. Conversationidentity/ownership
policyunchanged.46conversation/Responses/perframeWS/disconnect/memory-regression/Gemini
tests pass;whitespace passes. PriorRuff0.17.3check/format applies to
unchangedPRfiles.
Freshbodystillreservesuncheckedself-review/ready-for-humanreviewboxesfortheauthor.

No
newsourceblockerfound;retainexistingauthor-readinessholdwithouttickingboxesonbehalf.
Existingruntimeevidence scoped,no freshnativebuildclaim.
Maintainershandlechecks;nothingmerged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `cdf48ae9561dd2075625a4085594ac429a421231`, upstream main
`2d3ee51846435ef5b320eea4806af14e602b0d8c`.

Clean integration with upstream/main
2d3ee51846 after the external 0.43.1
release merge. Exact headroom/crates/tests comparison against previously
reviewed 7411de560 shows no changes. Re-reviewed generated metadata; 78
release/version/MCP tests pass on this tree and whitespace passes.
Earlier 46 conversation/Responses/WebSocket/disconnect/memory/Gemini
passes remain scoped to unchanged inputs.

No new source blocker found in reviewed scope. Existing author-owned
readiness declaration remains the hold; fresh body must be checked
before publication without checking author attestations on their behalf.
No contributor identity/ownership policy rewritten, no fresh
native-build/provider claim. Maintainers handle checks; nothing merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `d343c187f73b6a9005f0268e280ec657f435f1df`, upstream main
`ea59ac3eef3a3dc02076e9fe4bb30d572ca9c1a9`.

Clean integration with upstream/main
ea59ac3eef after the Click/macOS setup
merge. Re-reviewed the OpenAI response replay transport-header overlap;
conversation and existing tool-name map parameters remain additive.14
conversation/context-compaction tests and3 targeted route/gateway cases
pass (71 unrelated cases deselected); Ruff and whitespace pass. No new
source blocker found. Fresh body still leaves author-owned
self-review/readiness attestations unchecked; preserve the existing hold
without attesting on their behalf. Existing wider runtime evidence
remains scoped; no fresh native-build or live-provider claim.
Maintainers handle hosted checks; nothing merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `741582654b9d0817ca8318af6aaf2dd000d1ec81`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Final post-Dependabot requalification against upstream/main
c01cf07d94. All six Dependabot updates are
now merged. Relative to this branch's freshly tested prepared tree, the
final main sync changes only the three intended Cargo.lock records;
Python/source/test inputs are unchanged. Re-reviewed the integration and
confirmed complete native source/lock/toolchain objects match the tested
combined build and exact fresh native DLL used for this branch's local
tests.

Focused fresh-native qualification: 18 passed, 70 deselected, 1 warning
in 20.19s. Selected real dependency overlays and native DLL paths are
recorded in final4496-fresh-native-tests.log; the DLL provenance/input
hashes are in final-native-artifact-proof.json. No redundant rerun is
implied after this Cargo-only merge. Whitespace passes. Prior source
review and scoped real-behavior evidence remain applicable to unchanged
contributor inputs.

No new source blocker found. Fresh body still leaves author self-review
and ready-for-human-review attestations unchecked. Retain the
author-owned readiness hold without checking those boxes on their
behalf.

Maintainers handle hosted checks; no external-provider claim,
contributor semantic rewrite or non-Dependabot merge.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `cb5c34b6221880a811dd94d21c5e3f3e874ee565`, upstream main
`da6d9736cd82074988c7d0a2360b49d9d8b6c0a0`.

Current-main requalification against
da6d9736cd. All69 integration patches
exactly equal c01cf07-to-current-main delta after diff-location
normalization; conversation identity/ownership policy unchanged.88
conversation,Responses context-compaction,OpenAI route,gateway
request-contract tests pass;1 deprecation warning. Post-suite PyTorch
cache-stat logging reports closed-stream logging errors during process
exit; pytest exit0 and test results are retained without claiming full
runtime qualification.31 changed Python files pass Ruff lint/format;
offline lock and whitespace pass. Existing pr3982 native DLL verified
full input hashes/artifact checksum, no fresh build claimed.
Author/Devin discussion considered; previous substantive Devin findings
resolved. No new source blocker found. Author explicitly reserves
self-review/ready-for-human-review attestations and both remain
unchecked; retain existing readiness hold without attesting on their
behalf. Maintainers handle hosted checks; no contributor logic rewrite
or merge.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `951bdf5853aaf8757ba46253763c91b6601b5c6d`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.

Clean integration with upstream/main
0ddaa56c21. All 19 integration patches
exactly match the prior-reviewed-main da6d9736 to current-main delta
after diff-location normalization; conversation identity/ownership
policy is unchanged. Fresh conversation/turn-hook/Responses/gateway
request-contract, doctor and incoming Vertex/backend/CLI suite: 367
passed, 11 skipped, 1 warning. Fourteen changed Python files pass Ruff
lint/format; offline lock and whitespace checks pass.

Author discussion and Devin findings read: both historical Devin items
are marked resolved, including server-side conversation IDs. No new
source blocker found. Existing author-reserved
self-review/ready-for-human-review attestations remain a hold; this
maintainer does not attest on the author's behalf. Static source/test
scope and security inspection found no new workflow/dependency/setup
path; the inherited flagged workflow-validation job is excluded by this
exact PR's unchanged workflow paths. Conversation digests are not an
authentication boundary. Verified recorded native input-set/hashes and
artifact checksum before DLL reuse; no fresh build or new live-client
qualification claimed. Maintainers handle exact-head hosted checks; no
PR merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-09 14:56:34 -06:00
33f067a243 feat(proxy): add lossless /v1/compress mode and detect provider by wire shape (#4617)
## Description

Two gaps stop a gateway from using `/v1/compress` well. Both showed up
while fixing LiteLLM's `headroom` guardrail, which is the follow-up PR
that needs this one.

1. **No mode compresses messages losslessly without markers.** A gateway
with no retrieval loop can't resolve a marker or call
`headroom_retrieve`, so it needs a mode with nothing to retrieve. The
existing marker-free modes still run lossy compressors.
2. **Hooks guess the provider from the model name.** A gateway can send
any client's shape to any model. Claude Code's Anthropic body routed to
Gemini looked like OpenAI, so skill condensing found nothing to change.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)

## Changes Made

**`config.mode="lossless"`**
- Runs format-native folds only, with no markers and no CCR store
writes. A registered lossless provider still applies.
- Turns off stale-Read pointers, which drop the file body and need the
retrieval tool.
- Turns off cross-turn dedup pointers, which a client that never numbers
its messages can't resolve.
- Scope: the guarantee covers message compression (lossless codecs, no
CCR). On a gateway turn, tool schema compaction and registered turn
hooks still run as configured, as in every other mode; description
truncation only happens when an operator sets
`HEADROOM_TOOL_DESC_MAX_CHARS`.

**`provider_for_body`**
- Hooks and the session tracker now use the body's wire shape, and fall
back to the model name when the shape proves nothing, such as plain text
with no system prompt or tools.
- Anthropic's own tool types (server tools such as
`web_search_20250305`, client tools such as `bash_20250124`) end in an
eight-digit date and count as the Anthropic shape, so a first-turn body
whose only tool is a server tool is read correctly.
- `route.provider` still follows the model.
- The cache breakpoint budget runs when either the shape or the model is
Anthropic, and never on a Responses body.
- `gateway_response_fields` loses its unused `provider` parameter, and
`RESPONSE_RECEIVED` keeps reporting the model's provider.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`, `ruff format --check .`, on the
pinned 0.17.0)
- [ ] Type checking passes (`mypy headroom`): one error, in
`headroom/integrations/langchain/chat_model.py`, which this PR doesn't
touch (local mypy 1.19.1)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
pytest tests/test_proxy_compress_endpoint.py tests/gateway tests/test_gateway_turn_unit.py tests/test_compress_route_tokenizer_by_model.py
425 passed, 4 skipped

Regression tests with the fixes removed: 8 failed, 7 passed
```

## Real Behavior Proof

- **Environment:** LiteLLM main with the follow-up guardrail change,
this branch as the sidecar, and a fake upstream that simulates prompt
caching. No real provider API was called.
- **Exact command / steps:** replayed a scripted 17-call agent session
with exact-duplicate outputs removed, once direct to the fake upstream
and once through LiteLLM and `/v1/compress`.
- **Observed result:** 0 errors, 0 rewrites of already-sent results,
`Read` and `Edit` results unchanged, about 1.3s added in total.

| Client | Prompt cost vs direct |
|---|---|
| Anthropic Messages, Claude Code cache markers | −3.9% |
| Chat Completions | −3.8% |
| Responses | −4.1% |

The script's outputs are synthetic, so treat these as a comparison
between setups, not a forecast for real traffic.

## Known limits

- The default mode and `lossy_inline` can still return a stale-Read
`Retrieve original: hash=` marker, although the docs call them
marker-free. This predates the PR and needs a separate fix. Proxy
`--lossless` has no read-lifecycle guard either, from reading the code.
- With no `ContentRouter`, `_derived_compress_pipeline` falls back to
the default pipeline, so `lossless` wouldn't hold. Production always has
a router, so only test stubs reach this.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4617"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

## Maintainer post-merge revisit: main e9331b6b
Published head `e50b001bd05c9bc85405accca49cbef0e7786e51` incorporates
upstream main `e9331b6b963b5b6554f45943abed2366aaa25ea6` after #4142,
#4637 and #4641 merged. 164 endpoint/gateway/tokenizer/admission tests
pass; one json_minifier strategy assertion fails identically on actual
main e9331b6b. Ruff/format/diff pass. Import-only conflict resolved by
retaining both provider_for_body and RequestBodyTooLarge. Both existing
holds reconfirmed: lossless HTTP output removes unique description
suffix via tool_desc_compaction; Anthropic server tool with Gemini model
is classified OpenAI.

Contributor correctness logic was not rewritten. The existing Changes
Requested decision remains applicable; unchanged findings are not
reposted as duplicate reviews. Scoped Windows qualification uses the
existing development native extension, not a fresh build. Hosted checks
must finish independently; no contributor CI action requested and
nothing merged by this review.


## Maintainer post-merge revisit: main cc1eb540
Published `38b871120e318f6eaa1cb2ce16f96d4f6cc2b1e8` includes upstream
main `cc1eb5402aade58204b96b6b4b4d280cdd5456d2`. 90 gateway/tokenizer
tests pass; both existing HTTP/provider counterexamples reconfirmed.
Clean merge; Ruff/format/diff pass. Contributor correctness logic was
not rewritten. The existing Changes Requested decision remains
applicable; unchanged findings are not reposted. Scoped Windows
qualification uses the existing development native extension, not a
fresh build. Hosted checks must finish independently; nothing merged by
this review.


### Maintainer current-main requalification

Exact head `33395b5cdd7ee9facce9f97f2a3250f5e6e49cb2`, upstream main
`ecbaf86d78cf52d185fbffa306157b9a4cc1b906`.

Mechanically merged upstream/main
ecbaf86d78 without changing contributor
policy logic. Re-reviewed the combined gateway/provider delta and
existing comments. All 90 focused gateway/tokenizer tests and ten
inherited Gemini fixture cases pass; Ruff0.17.3 lint/format and
whitespace checks pass. The shared Gemini repair was already present, so
no duplicate fixture commit was added.

Both existing correctness findings independently remain on this combined
tree. An HTTP200 /v1/compress request with config.mode=lossless still
drops UNIQUE_REQUIREMENT_AT_END from a long tool description, reporting
tool_desc_compaction. An Anthropic web_search_20250305 tool body with a
Gemini-named model still returns provider openai. Existing requested
semantic fixes/regressions remain necessary; maintainers are handling CI
and no contributor CI execution is requested. No contributor logic was
rewritten and nothing merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `4c40adb53a68f382e32f00620adcfaf1cff477f5`, upstream main
`8d3e9e5ca7cd645eb73084b848edbea5fc796e1a`.

Clean mechanical sync with upstream
main8d3e9e5ca7cd645eb73084b848edbea5fc796e1a including intervening
release/runtime/cache/CCR/SSE/HTTP1.1 merges. No contributor
gateway/provider policy rewritten. Fresh author/Devin discussion
considered.100focused gateway-contract/tokenizer/Gemini tests
pass;7changed Python files pass Ruff0.17.3check/format and whitespace
checks pass. Last main delta after test start changes only
Rust/OpenClaw, leaving tested Python inputs unchanged.

Both existing blockers remain freshly reproduced from this worktree
(source import path verified): real app POST/v1/compress lossless
returns200 but drops UNIQUE_REQUIREMENT_AT_END from tool description via
tool_desc_compaction; Anthropic server-tool wire shape with Gemini model
returns provider openai. Preserve promised lossless content and classify
server-tool wire forms correctly, with regressions. No external provider
session/nativebuild claimed. Copied extension supports Python startup;
maintainers handle hosted checks. Nothing merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `8b7b2b0d02daf1a031828ecb13095ab315431f7e`, upstream main
`917edc62230c3c4a8a415e2053da68dab1f003dd`.

Current-main integration against917edc62230c3c4a8a415e2053da68dab1f003dd
is clean. Compared entire headroom/crates/Python manifests+lock/tests
inputs to prior qualified4c40adb53: byte-identical. Intervening changes
are reviewed npm maintenance and release-envelope recovery workflow; no
contributor gateway policy changed.75new-base
release-workflow/normalizer tests pass,whitespace passes.
Prior100gateway/tokenizer/Gemini results remain scoped to those
identical inputs rather than relabeled fresh.

Both gateway blockers freshly rechecked from worktree: Anthropic
server-tool+Gemini body classifiesopenai; actual lossless/v1/compress
returns200 but drops unique description suffix via tool_desc_compaction.
Retain existing losslessness/classification corrections. No fresh
nativebuild/provider claim. Maintainers handle exactheadCI;nothing
merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `490ff6defdebddda9106e9e143d42d1e463b6324`, upstream main
`7592e681459f14a2041eb5f524a658cb6ad1dd92`.

Clean integration with upstream/main
7592e68145. Reviewed both intervening
memory fixes and fresh PR discussion. Gateway/runtime/Rust inputs are
unchanged from the previously reviewed head; only the two main memory
fixes and their regressions enter this merge. Nine memory/Gemini
regression tests pass; main-relative whitespace passes.

Fresh actual-app probes retain both existing blockers: Anthropic
server-tool content with a Gemini model classifies as OpenAI; lossless
/v1/compress removes the unique tool-description suffix through
tool_desc_compaction. No contributor policy was rewritten. Existing
native extension used for Python startup, not a fresh native-build
claim. Maintainers handle hosted checks; nothing merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `d3cb3a02af03991a696461160ee0db322dd8bf48`, upstream main
`2d3ee51846435ef5b320eea4806af14e602b0d8c`.

Clean integration with upstream/main
2d3ee51846 after the external 0.43.1
release merge. Exact headroom/crates/tests comparison against prior
reviewed 490ff6def shows no changes. Re-reviewed generated release
metadata; 78 release/version/MCP tests pass on this composed tree, and
whitespace passes. Previous nine memory/Gemini regressions and fresh
gateway probes remain scoped to identical runtime inputs.

Both established blockers remain: lossless /v1/compress drops the unique
tool-description suffix, and Anthropic server-tool content with a Gemini
model classifies as OpenAI. Runtime probes were not relabeled as rerun
after this metadata-only merge. No contributor policy rewritten; no
fresh native-build claim. Maintainers handle exact-head checks; nothing
merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `e0c873630e9ffc7bc0d0d2fdd22c483b731e49a3`, upstream main
`910588e6b15ab54c4099d03703e5fa20c8abb757`.

Clean mechanical synchronization with upstream
main910588e6b15ab54c4099d03703e5fa20c8abb757, including the externally
merged runtime fixes, docs dependency update and tree-sitter lock
repair. No contributor gateway/provider policy rewritten. Current
author/Devin discussion considered. All90 gateway
contract/tokenizer-by-model tests pass on the preceding composed runtime
tree; the final dependency-only merge changes only pyproject/uv.lock,
leaving those gateway/tokenizer sources unchanged. No redundant rerun on
the final merge is claimed. Changed Python Ruff lint/format and
main-relative whitespace pass.

Both existing blockers freshly reproduce on the composed runtime paths:
lossless /v1/compress returns200 but drops UNIQUE_REQUIREMENT_AT_END
from the tool description and reports tool_desc_compaction; Anthropic
server-tool content with a Gemini model is classified as OpenAI.
Preserve the lossless original payload and classify the full request
protocol consistently, with endpoint regressions for these inputs. The
main fixes do not close either defect.

Tests use the existing native core for startup, not a fresh native
build; no live provider traffic was made. Maintainers handle hosted
checks; no inherited-main pipeline failure attributed and nothing
merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `507433606f698249e546e1418928caee99d0c901`, upstream main
`7877241d84c93b3ec487e07324245838ddae85cc`.

Clean mechanical integration with
upstream/main7877241d84c93b3ec487e07324245838ddae85cc. Gateway
production/contracts remain byte-identical to the preceding reviewed
head; re-reviewed the new CLI/Hermes integration.127 gateway
request/response contract and model-tokenizer tests pass; whitespace
passes. Both existing holds freshly reproduce on this exact tree:
lossless /v1/compress returns200 while dropping
UNIQUE_REQUIREMENT_AT_END and reporting tool_desc_compaction; Anthropic
server-tool content with a Gemini model is classified as OpenAI.
Preserve the lossless original payload and classify the full request
protocol consistently, with endpoint regressions. No contributor
gateway/provider policy rewritten. Existing native core used for
startup; no fresh native build or live provider claim. Maintainers
handle hosted checks; nothing merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `b0daece6b10f623a6e7b0dc3eb01c73e05fa3209`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Final post-Dependabot requalification against upstream/main
c01cf07d94. All six Dependabot updates are
now merged. Relative to this branch's freshly tested prepared tree, the
final main sync changes only the three intended Cargo.lock records;
Python/source/test inputs are unchanged. Re-reviewed the integration and
confirmed complete native source/lock/toolchain objects match the tested
combined build and exact fresh native DLL used for this branch's local
tests.

Focused fresh-native qualification: 127 passed, 3 warnings in 104.78s
(0:01:44). Selected real dependency overlays and native DLL paths are
recorded in final4617-fresh-native-tests.log; the DLL provenance/input
hashes are in final-native-artifact-proof.json. No redundant rerun is
implied after this Cargo-only merge. Whitespace passes. Prior source
review and scoped real-behavior evidence remain applicable to unchanged
contributor inputs.

Both existing blockers freshly reproduce using the combined native DLL:
lossless /v1/compress returns200 but removes UNIQUE_REQUIREMENT_AT_END
and reports tool_desc_compaction; Anthropic server-tool content with a
Gemini model is classified as OpenAI. Preserve the original lossless
payload and classify the complete wire protocol, with endpoint
regressions.

Maintainers handle hosted checks; no external-provider claim,
contributor semantic rewrite or non-Dependabot merge.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `ec6abc0228a93c5eab1881938041dfee344b1e3a`, upstream main
`2973ca593d2fe65c3259941fdff8009175eff456`.

Requalified after human merges #4667/#4669/#4670 against upstream main
ce42754912. All 11 integration patches
exactly match the prior-main-to-current-main delta after ignoring diff
locations; no contributor gateway/provider policy changed. Fresh gateway
request/response contracts, model-tokenizer, gateway unit,
Kompress-disable and must-keep suites: 207 passed, 3 warnings. Six
changed Python files pass Ruff lint/format; offline lock validation and
whitespace pass. Existing matching-core DLL used; native
source/lock/toolchain unchanged by these main merges, with no fresh
native build or live-provider claim. Both existing blockers freshly
reproduce: lossless /v1/compress returns 200 while removing
UNIQUE_REQUIREMENT_AT_END and reporting tool_desc_compaction; Anthropic
server-tool content with a Gemini model is classified as OpenAI.
Preserve original lossless payload and recognize complete wire protocol
with endpoint regressions. Author and Devin discussions checked;
existing findings remain actionable. Maintainers handle hosted checks;
nothing merged.
Subsequent human merge #4671 advanced main to
2973ca593d before publication; guard
stopped the old candidate. Cleanly integrated the new workflow-only
delta and re-reviewed action pinning with 62 workflow/release tests
passing. Runtime/native sources are byte-identical to the immediately
preceding 207-test and real-behavior qualification; those results apply
to unchanged inputs. All prior-main hosted results are historical for
readiness.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `9f857218486fa451ecacc675863a64bc95f45bb0`, upstream main
`2b0315b2d4b0d54947050bbdad2de9e89df06378`.

Current-main requalification
against2b0315b2d4b0d54947050bbdad2de9e89df06378 after later human
merges. All37 integration patches exactly match2973ca59-to-current-main
delta after diff-location normalization; contributor gateway/provider
policy unchanged.393gateway request/response
contracts,model-tokenizer,gateway
unit,Kompress-disable,must-keep,doctor,search tests
pass;3warnings.27changedPythonfiles pass Ruff lint/format;offline
lock/whitespacepass. Existing pr3982 native DLL verified every tracked
native-input hash/artifactchecksum;no fresh build claimed. Author/Devin
discussion checked. Both correctness blockers freshly reproduce:lossless
/v1/compress returns200while droppingUNIQUE_REQUIREMENT_AT_END and
reportingtool_desc_compaction;Anthropicserver-toolcontent
withGeminimodel classifiedOpenAI. Preserve original lossless payload and
classify full request wire protocol consistently,with endpoint
regressions. No live provider qualification,contributor logic rewrite,or
inherited-main CI fault attributed. Maintainers handle hosted
checks;nothing merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `41b2145418db7f5980cd52e6235ab330fcd31ac0`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.

Clean integration with upstream/main
0ddaa56c21. All 17 integration patches
exactly match the prior-reviewed-main to current-main delta after
diff-location normalization; contributor gateway/provider policy is
unchanged. Fresh gateway/tokenizer/search/doctor and incoming
Vertex/backend/CLI qualification: 455 passed, 11 skipped, 3 warnings.
Additional gateway request/response contracts and compress-endpoint
suite: 158 passed, 1 warning. Twelve changed Python files pass Ruff
lint/format; offline lock and whitespace checks pass.

Author discussion and outstanding Devin findings read. Both existing
correctness findings freshly reproduce on this exact tree: lossless
/v1/compress returns 200 but drops UNIQUE_REQUIREMENT_AT_END through
tool_desc_compaction; Anthropic server-tool content with a Gemini model
classifies as OpenAI. Existing requested corrections remain; no new
duplicate finding. Local app probes are not live provider proof. Native
DLL reused only after verifying the complete tracked input set, every
recorded input SHA256 and artifact checksum against
pr3982-c01-native-proof.json; no fresh build claimed. Maintainers handle
exact-head hosted checks. No PR merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-07 20:36:11 -06:00
55602fc1e2 fix(proxy): cap cache_control breakpoints on OpenAI chat path (#4471) (#4491)
## Description

On the OpenAI `/chat/completions` path in cache mode, earlier messages
are replayed with old `cache_control` markers. The 4-marker budget guard
`enforce_cache_breakpoint_budget` now runs on that path after the mirror
step, skipped for `x-headroom-bypass`, with `client_messages=None` to
avoid the block normalizer inventing markers; `tool_calls` markers are
counted/dropped oldest-first.

Closes #4471

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `handlers/openai.py`: run `enforce_cache_breakpoint_budget` as
last-stop guard right after the mirror step (skipped for
`x-headroom-bypass`), passing `client_messages=None` to avoid the block
normalizer inventing markers; same guard the gateway extension already
uses.
- `helpers.py`: `count_cache_breakpoints` counts
`tool_calls[*].cache_control`; the oldest-first drop pass also strips
those markers.
- Mirror step left as-is: its documented bail-out on count mismatch
stays, the budget guard now bounds the total with the bypass gate and
`client_messages=None`.

## Testing

- [x] Unit tests added/updated
- [x] Existing tests pass

```text
$ pytest tests/test_openai_message_level_cache_control.py tests/gateway/test_cache_breakpoint_budget.py tests/test_cache_breakpoint_diagnostics.py <prefix-tracker/mirror tests> -q
1 failed, 115 passed, 8 skipped in 10.69s
FAILED tests/test_cache_breakpoint_diagnostics.py::test_runtime_log_says_why_it_is_missing   # pre-existing on main: Windows WinError 183 on temp log dir
$ ruff check / ruff format --check: clean
```

## Real Behavior Proof

- Environment: Windows 11, Python uv tool env with headroom._core built
- Exact command / steps: unit-level repro of the issue body shape (4
client markers + replayed prefix carrying an old message-level and
tool_calls marker, counts differ) through
enforce_cache_breakpoint_budget
- Observed result: 6 markers in → ≤4 out, newest tail marker kept,
stats["repaired"] True
- Not tested: live LiteLLM + Vertex gateway end-to-end

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: stable
- Stable/default behavior changed: only when a chat request exceeds 4
cache_control markers (previously a guaranteed upstream 400)
- Kill switch / disable path: none needed; under-limit requests return
by identity unchanged
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4491"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-28 23:38:31 -05:00
AlessandroSpallinaandClaude Opus 5.5 a90421083a feat(proxy): report the same tool savings on gateway /v1/compress as the proxy path (#4467)
## Description

Gateway-mode `/v1/compress` now reports the same savings as the proxy
path. The why is in #4466.

Closes #4466

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [x] Documentation update

## Changes Made

Two commits, one per gap:

1. **`fix(proxy): count tool-schema compaction ...`**
- The gateway turn counts the tool definitions before and after
compaction.
- The handler adds both counts to `tokens_before` / `tokens_after`, with
the proxy path's guard (`0 < after < before`).
   - Tools the client already marked `defer_loading` are skipped.
2. **`feat(proxy): report tool_tokens_saved and tool_tokens_deferred
...`**
- Gateway-mode answers add `tool_tokens_saved` (from the tags `/stats`
sums) and `tool_tokens_deferred` (the estimated share).
   - Docs: `proxy.mdx`, `wiki/proxy.md`.

Non-gateway and fail-open answers do not change.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`, on the changed files)
- [x] Type checking passes (`mypy`, on the changed files)
- [x] New tests added for new functionality

```text
$ pytest -q tests/gateway/ tests/test_gateway_turn_unit.py tests/test_proxy_compress_endpoint.py
380 passed, 5 skipped
```

The tests also pass on the first commit alone (`tests/gateway/`: 280
passed). The new test for client-deferred tools fails without its fix.

## Real Behavior Proof

- Environment: macOS, Python 3.12. Before: image
`headroom:0.41.0-nonroot`. After: this branch, `HEADROOM_OFFLINE=1
headroom proxy`.
- Exact command / steps: start each build, then send it the same
gateway-mode `POST /v1/compress` — one message plus 5 tools whose
schemas carry `title` / `examples` / `default` (the annotation keys
schema compaction strips), with `"gateway": {"can_redrive": false,
"can_relay_response": false}` so the request is a single call. Read
`tokens_before`, `tokens_after`, `tokens_saved` and the `tool_*` fields
out of the JSON answer. No provider is contacted.
- Observed result: same request, same tools. On 0.41.0 the tool
definitions are invisible to the counts, so `tokens_saved` is 0 and
neither `tool_*` field is present. On this branch the compaction is
counted, so `tokens_before` / `tokens_after` carry the tool bytes and
`tokens_saved` is the 240 tokens compaction stripped
(`tool_tokens_saved` / `tool_tokens_deferred` stay 0 here: no
tool-search extension ran, and the extension is what books deferral).

```text
0.41.0:  tokens_before 13,  tokens_after 13,  tokens_saved 0    (no tool_* fields)
branch:  tokens_before 830, tokens_after 590, tokens_saved 240, tool_tokens_saved 0, tool_tokens_deferred 0
```

- Not tested: a live run with the headroom-tool-search extension
(deferral is covered by unit tests), and Responses-API bodies.

## Runtime Rollout Safety

- Rollout-managed feature(s): None gate this change — the gateway
`/v1/compress` accounting is not behind a registry feature. One
interaction: `proxy_output_shaper` (registry feature, available
`stable`, deliberately not default-enabled) consumes the augmented
count. `tokens_before` is folded at
`headroom/proxy/handlers/openai.py:11064` and then handed to the turn at
`:11083`, which passes it as `input_tokens` to `shape_gateway_body`
(`headroom/proxy/gateway_extension.py:213`) → `stratum_key` →
`input_bucket`. A turn whose tool definitions push it across one of the
2k / 8k / 32k / 128k boundaries therefore lands in a bigger stratum,
which picks a different output-savings baseline. The chat path already
has this coupling.
- Minimum rollout channel: n/a — no channel gate, so the change takes
effect wherever gateway mode is served, on every channel.
- Stable/default behavior changed: Yes, gateway-mode answers only.
`tokens_before` / `tokens_after` now include the tool definitions before
and after schema compaction, so `tokens_saved` covers the compaction
saving the same way the chat handlers already do, and two fields are
added. Non-gateway and fail-open answers are byte-identical.
- Kill switch / disable path: None. Without compaction there is nothing
to fold, and the two fields are additive on answers; a gateway that does
not want them ignores them.
- Unsafe override required: No. Nothing crosses a channel boundary.
- Qualification impact: None. No registry entry is added, so
`registry_digest` and `snapshot_digest` are unchanged. The gateway
`tokens_saved` figure itself changes, which a benchmark comparing arms
across this commit will see.
- Rollback path: Revert the two commits.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] I have made corresponding changes to the documentation
- [x] I have added tests that my fix is effective or that my feature
works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- The fields are emitted for any claimed compress turn. Only the gateway
extension claims turns today.
- If a response-half hook rewrote the tool tags during a re-drive,
`/stats` would no longer match the answer. Core has no such hook.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 08:45:53 -07:00
85520c2941 fix(proxy): net tools a turn hook adds against the saving it books (#4057)
## Description

A turn hook that folds messages but **adds** a tool definition saved the
difference, not the fold alone. headroom-skill-search does exactly this:
it rewrites the skill list in the messages and appends a `search_skills`
tool. Two places over-credited it:

- **Hook runner (`run_request_hooks`):**
  - It clamped the message and tool halves separately at zero.
- So the added tool never reduced the hook's ledger entry (`/stats`
`savings.by_source.skill_search`).
- **Anthropic and OpenAI chat handlers:**
- Tool shrinkage from a hook goes into `turn_hook_tools_saved_tokens`,
but tool growth went nowhere.
- So the headline `tokens_saved` ignored a tool that is sent on every
request.

Found while auditing the enterprise plugins for the provider-unit
savings work (#4052 / #4055).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `turn_hooks.run_request_hooks`: signed message and tool deltas,
netted. An entry is recorded only when the net is a saving.
`details.tool_tokens_saved` is negative when tools were added.
- Anthropic handler: tool growth from turn hooks (`_th_tool_growth`) is
added to `optimized_tokens` at the recount. This is the same place
compaction deltas are folded, so `tok_before - tok_after == tok_saved`
stays coherent.
- OpenAI chat handler: the same, in its post-hook recount.
- `turn_hook_tools_saved_tokens` keeps its non-negative meaning
(shrinkage only). Every surface that sums it is unchanged.

## Testing

- [x] Unit tests pass (`pytest`): related suites
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy`) on changed modules
- [x] New tests added for new functionality
- [x] Manual testing performed (plugin suites against this core)

### Test Output

```text
tests/test_hook_net_tool_additions.py: 6 passed
  - ledger entry is net of the added tool; no entry when the tool outweighs the fold
  - Anthropic /v1/messages and OpenAI /v1/chat/completions: tokens_saved == fold - added tool
  - a later hook that grows the request more than was saved cancels the earlier credit
  - a later partial loss caps the earlier credit at the sequence net, and the
    breakdown (message + tool) sums to the capped tokens (uncapped_* kept)
tests/gateway/test_hook_tool_growth.py: /v1/compress tokens_after includes the added tool
  -> each fails on the commit before its fix (e.g. 1499 == 1499 - 117, 0 > 100,
     ('fold', 200) == ('fold', 142), (200 + 0) == 142)
gateway + turn_hook / tool_search / tool_schema / savings_attribution / extension /
anthropic / openai / responses / compress suites: 2101 passed, 34 skipped
Plugin suites against this core (PYTHONPATH): skill-search 41, tool-search 144, lossless-guard 237 passed
ruff check / format: clean; mypy (changed modules): Success
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, `LITELLM_LOCAL_MODEL_COST_MAP=True`;
the real proxy app (`create_app`) through FastAPI TestClient, upstream
mocked with respx.
- Exact command / steps: `pytest tests/test_hook_net_tool_additions.py
tests/gateway/test_hook_tool_growth.py` (a hook shaped like
SkillSearchHook folds messages and adds a search tool, on
`/v1/messages`, `/v1/chat/completions` and `/v1/compress`), then the
skill-search / tool-search / lossless-guard suites with `PYTHONPATH` set
to this branch.
- Observed result: on every path `tokens_saved` and the `skill_search`
ledger entry equal the message fold minus the added tool's tokens;
across several hooks the total credit never exceeds the sequence's net
saving and each entry's breakdown sums to its tokens.
- Live skill-search session (2026-10-15, branch head `7a81ec9f`):
- Setup: a real proxy run from this branch with headroom-skill-search
0.2.0 (`--proxy-extension skill_search`,
`HEADROOM_SKILL_SEARCH_HINT_MODE=first_sentence`,
`HEADROOM_BEACON=off`), real Anthropic upstream, `claude-haiku-4-5`.
Each request was a non-streaming curl to `/v1/messages` with 22 skills
listed the way Claude Code lists them.
  - Turn 1: HTTP 200, body 9116 B → 2744 B.
- The `skill_search` ledger booked `message_tokens_saved=1327` and
`tool_tokens_saved=-108`; the −108 is the added `search_skills` tool.
That gives `tokens=1219`, the signed net this PR books.
- Anthropic `count_tokens` on the original body: 2415. Input billed
through the proxy: 1153–1276.
  - Headline:
- On this branch alone it read `tok_before=424 tok_after=532
tok_saved=0`. The cause is a bug on main, not in this PR:
`snapshot_original_messages` aliases the live messages when only turn
hooks are registered, and skill search folds text in place, so the
recorded original was the folded one. The fix is #4241.
- With #4241 applied: `tok_before=1751 tok_after=532 tok_saved=1219` =
1327 − 108, the same as the ledger.
- Turn 2: the model called `search_skills` and the hook served
`image-resize`. After one reload round the final answer came back HTTP
200 `end_turn`. Headline `tok_saved=1219`; no new ledger entry was
booked.
  - `/stats` `savings.by_source.skill_search`: 1 event, 1219 tokens.
- Not tested: streaming (skill search acts only on buffered turns with
at least 18 skills), and the OpenAI path live.
- Merge order: the headline matches the ledger only once #4241 is on
main. Merge #4241 first.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, accounting only (requests are
byte-identical). Headline, `/v1/compress` `tokens_saved` and
`skill_search` attribution drop by the size of the tool the hook adds,
and a later hook's growth offsets earlier hook credits.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: one extra tool-array count per hooked OpenAI
chat request.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- **Docs:** none needed; this is an internal accounting fix.
- **Stacked on #4056** (base `fix/hook-tool-deferral-accounting`). Both
PRs change the same lines in `run_request_hooks`, so this branch carries
the resolution: the signed tool delta is measured with
`without_deferral_flags`, and the OpenAI tool-growth count strips the
flag too. Merge #4056 first. The combined branch passes 844 core tests
and the skill-search, tool-search and lossless-guard suites.

🤖 Generated with [Claude Code](https://claude.com/claude-code)



## Current maintainer verification (October 8)

Exact synchronized head: `de0208ae9dbabe7967d4b781e67364f6ee8b141d`,
containing main `01d9194854233ef6ed04005cf0a5b8c32bf65d6d`. The main
merge preserves the contributor's signed sequence accounting and
net-saving helper, along with main's deferral-filtered tool counters.
Parent #4056 has merged, so the PR now targets main directly. No
contributor logic was changed.

25 focused hook/gateway/deferral tests pass. Two additional isolated
reviewer probes force only the session's final message recount to fail;
positive and negative net-savings cases both pass, reaching the fallback
without breaking the hook counter. This is local TestClient/native-core
evidence, not a live skill-search session. Current-head hosted
qualification is pending.

The previously acknowledged live skill-search runtime proof remains
outstanding; the new tests close the accounting finding but do not close
that evidence requirement.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-16 14:30:00 -07:00
Tejas Chopra 5cac7ab693 fix(gateway): report why /v1/compress/response rejected a body (#4081)
## Description

`POST /v1/compress/response` (the response half of the gateway contract)
caught every body-read failure and answered `"Invalid JSON in request
body."`. That covered malformed JSON, but also an oversize body, a
failed decompression, invalid UTF-8 and a JSON array. A Kong or LiteLLM
integrator debugging a 400 could not tell which one it was.

The other handlers (anthropic, openai, gemini, batch; about 10 call
sites) already use `helpers.invalid_request_body_message(exc)`. It picks
a fixed string by exception type, so it never echoes the exception text
that CodeQL's `py/stack-trace-exposure` flags, and it logs the exception
type on the proxy side. This handler now uses it too. That gives
specific messages for malformed JSON, a non-object, invalid UTF-8 and an
oversize body. A failed decompression (for example a bad gzip stream)
still gets the generic `Invalid request body`, as it does on every other
handler. A separate decompression message would mean changing the shared
helper for all handlers, so that is out of scope here.

The `if not isinstance(body, dict)` branch that followed is removed.
`_read_request_json` raises `RequestBodyNotObject` for a non-object
before it can return one, so the branch could not be reached. Its case
is now reported as `Invalid request body: must be a JSON object`.

The status code (400) and the error `type` (`invalid_request`) are
unchanged. Only the `message` text differs. No test, doc or consumer
repo matched on the old string: I grepped `tests`, `docs`, `headroom`
and the gateway integrations, which branch on the status code.

Refs #4136

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/gateway_turn.py`: `handle_compress_response` answers
400 with `invalid_request_body_message(exc)`, and the unreachable
non-dict check is gone.
- `tests/gateway/test_contract_response_half.py`: add
`test_invalid_body_400_says_why`, parametrized over three bodies. Each
case asserts a 400, `invalid_request`, and a message that names the
reason:
  - malformed JSON;
  - a JSON array;
  - invalid UTF-8.

  All three cases fail on main.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q -p no:cacheprovider tests/gateway/test_contract_response_half.py -k invalid   # before the fix
FAILED tests/gateway/test_contract_response_half.py::test_invalid_body_400_says_why[malformed]
FAILED tests/gateway/test_contract_response_half.py::test_invalid_body_400_says_why[not-object]
FAILED tests/gateway/test_contract_response_half.py::test_invalid_body_400_says_why[not-utf8]
================== 3 failed, 6 passed, 28 deselected in 7.18s ==================
$ pytest -q -p no:cacheprovider tests/gateway/ tests/test_gateway_turn_unit.py tests/test_invalid_request_body_message.py   # after
================== 323 passed, 4 skipped in 171.41s (0:02:51) ==================
$ uvx ruff@0.16.8 check headroom/proxy/gateway_turn.py tests/gateway/test_contract_response_half.py
All checks passed!
$ mypy headroom/proxy/gateway_turn.py --ignore-missing-imports
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS, Python 3.12.13, branch checkout at ae6a419a, using
the repo .venv. A temp HOME, workspace and CODEX_HOME were used, with
offline mode on.
- Exact command / steps:
1. Ran the shared smoke harness, which starts a real proxy from this
checkout against a local fake upstream and sends one Anthropic and one
OpenAI request.
2. Started a real proxy with `python -m headroom.cli proxy --port <p>`
and used curl to POST five bodies to
`http://127.0.0.1:<p>/v1/compress/response`: four bad ones, and one
valid body with an unknown turn as a control.
- Observed result:
```text
branch: cleanup/proxy-core-a-compress-response-errors @ ae6a419a
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines (<tmp>):
(smoke dir: <tmp>)
```
```text
malformed JSON         -> {"error":{"type":"invalid_request","message":"Invalid request body: malformed JSON"}} HTTP400
JSON array             -> {"error":{"type":"invalid_request","message":"Invalid request body: must be a JSON object"}} HTTP400
invalid UTF-8          -> {"error":{"type":"invalid_request","message":"Invalid request body: not valid UTF-8 (possibly compressed?)"}} HTTP400
bad gzip               -> {"error":{"type":"invalid_request","message":"Invalid request body"}} HTTP400
valid, unknown turn    -> {"error":{"type":"unknown_turn","message":"No pending turn 't-x' (never registered, already finished, or expired)."}} HTTP404
```
Every bad body is a 400 with type `invalid_request`, and the message
names the reason. The exception is bad gzip, which gets the generic
message, as on every other handler. On main, all four said `Invalid JSON
in request body.`. The control request still reaches turn lookup (404
`unknown_turn`).
- Not tested: A live Kong or LiteLLM gateway. Both key on the status and
`type`, which are unchanged.

## Runtime Rollout Safety

- Rollout-managed feature(s): None (the gateway contract is controlled
by `HEADROOM_GATEWAY_CONTRACT`, unchanged).
- Minimum rollout channel: N/A; error-message text only.
- Stable/default behavior changed: Only the `error.message` text on a
400 for a bad body. The status, error type and every success path are
unchanged.
- Kill switch / disable path: N/A.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Part of a maintainer-requested codebase cleanup (dead code, duplication,
debuggability), split into small PRs for review. It is part of
maintainer-requested cleanup #4136.

No docs quote the old message, so none needed changing. The change is
small and self-explanatory, so no new comments were added.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-10-10 09:05:53 -07:00
ee731d7ea8 fix(proxy): refuse token-less non-loopback binds, gate operator routes (#3852)
## Why

Four related gaps in the proxy's network gate (findings 01-F1, 01-F2,
01-F5, 01-F9):

- **Open bind.** A non-loopback bind with no `HEADROOM_PROXY_TOKEN` only
logged a warning, then served the `/v1/*` relay, with the operator's
upstream credentials, to every peer that could reach the port.
- **Missing peer address.** `is_loopback_host(None)` returned `True`. A
Unix-domain socket, an ASGI adapter that leaves `scope["client"]` unset,
or a hand-built request therefore turned off the token gate, the
`/admin` and `/debug` guards and the WebSocket gate.
- **Operator data routes.** `/stats-history` (including CSV export),
`/quota`, `/subscription-window` and `/metrics` were served to any
network caller.
- **WebSocket loopback exemption** checked only the peer IP, not the
`Host` header, so it had no DNS-rebinding defence.

## What changes

- New `headroom/proxy/bind_policy.py`. A non-loopback bind (`0.0.0.0`,
`::`, a LAN address or a hostname) with no token is refused at three
places: `create_app` (embedders and the multi-worker factory),
`run_server` (one exit with code 2 before uvicorn forks) and the
`headroom proxy` CLI (a clear error before the banner, exit code 1). An
absent host means uvicorn's default `127.0.0.1`.
- `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` acknowledges an open bind when
the runtime already restricts reachability, such as a container
published on `127.0.0.1`. The `proxy_open_bind` warning is still logged
on that path, and the banner says the bind is open.
- `is_loopback_host(None)` now returns `False` (fail closed), matching
`identity.py`.
- `/stats-history`, `/quota` and `/subscription-window` answer only: a
non-loopback caller that passed the token gate, a loopback caller (peer
and `Host` header both loopback), a trusted dashboard client, or, under
an acknowledged token-less bind, the container's own default-gateway TCP
peer with a loopback `Host` (the host browser reaching a
`127.0.0.1`-published container). Everyone else gets 404, as with
`/admin`. Settings writes keep their stricter rule with no token
short-cut.
- `/metrics` answers a non-loopback caller that passed the token gate, a
loopback caller, a connecting peer in
`HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` (whatever it forwards), or a
resolved client in the dashboard CIDRs. For CIDR peers a hostname `Host`
is fine, so Prometheus scrapers work; a cross-origin browser request is
still refused.
- The token short-cut applies only to non-loopback requests. The gate
exempts loopback peers without checking `Host`, so a loopback request
has not proven it holds the token and must pass the loopback `Host`
check. This blocks a DNS-rebound browser on the operator's machine.
- The `/dashboard` shell stays ungated. It is a static template, like
the already public `/dashboard/static` assets. Gating it would break the
dashboard for containers published on host loopback, where the peer is
the bridge gateway and `Host` is `localhost`. The data routes behind it
are gated.
- The WebSocket loopback exemption requires a loopback peer and a
loopback `Host` header. A loopback peer with a foreign `Host` can still
connect with the token.
- Launchers that always publish on `127.0.0.1` set the acknowledgement
themselves: `headroom install` (docker preset), `scripts/install.sh` and
`scripts/install.ps1` (through one helper that adds the acknowledgement
only together with the `127.0.0.1` publication), and
`docker/docker-compose.native.yml`.
- Docs: `proxy.mdx` documents the new variable and which routes the
token also gates; `docker-install` (docs and wiki) and the
`docker-compose.yml` comment show the acknowledgement in the
loopback-published example. `e2e/docker-bind-security.sh` sets it and
checks that the same launch without it refuses to start.

## Behaviour change (breaking)

- `headroom proxy --host 0.0.0.0`, or `HEADROOM_HOST` set to any
non-loopback address, without `HEADROOM_PROXY_TOKEN` now refuses to
start. The error names both remedies. Set a token, bind loopback, or set
`HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` when the port is already
confined. Anyone running the image by hand with `--host 0.0.0.0` and no
token must add one of these.
- Without a token, `/stats-history`, `/quota`, `/subscription-window`
and `/metrics` return 404 to non-loopback callers outside a trusted
CIDR. Prometheus scrapers on another host must be in
`HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` or send the token.
- Code that relied on a request with no peer address being treated as
loopback now gets the non-loopback path.
- The default loopback run is unchanged.

## Tests

- `tests/test_proxy_bind_policy.py`: the refuse/allow matrix
(non-loopback, loopback, unset host, token, acknowledgement values), the
warning on the acknowledged path, `create_app`, the `run_server` exit
before uvicorn, and the CLI (refusal, `HEADROOM_HOST` from env,
acknowledged start, offline banner, start with a token).
- `tests/test_proxy_loopback_gating.py`: each data route against
non-loopback, loopback, trusted-CIDR and token callers; the dashboard
shell stays reachable; the host of an acknowledged, loopback-published
container reads the data routes while another bridge peer, a forwarded
gateway address and a foreign `Host` do not; CSV export is not served to
network callers; `/metrics` across loopback, both CIDR lists,
out-of-range peers, cross-origin browsers and token callers, including a
trusted gateway relaying a scraper outside its own range. A regression
test shows that a token on the proxy does not exempt a loopback peer
from the `Host` check; it fails without the fix.
- `tests/test_proxy_hardening.py`: the HTTP gate with no peer address,
WebSocket `Host` cases, `require_loopback` and `/debug` with no peer.
- Installer tests: `headroom install` adds the acknowledgement, and the
shell and PowerShell wrappers add it only together with the loopback
publication. Launcher-level tests start the proxy with the environment
and `--host` that `build_runtime_command()` and
`docker-compose.native.yml` produce and check that the host browser
(bridge-gateway peer) gets `/stats-history` and another container does
not.
- Tests that bound `0.0.0.0` for unrelated reasons now set a token or
the acknowledgement. Tests that read operator routes use an explicit
loopback peer.

On `main`, 14 of the gating and hardening tests fail and the bind-policy
module does not exist. Manual runs: token-less `--host 0.0.0.0` exits
with the refusal from both the CLI and `run_server`; with the
acknowledgement, the data routes are 404 to a LAN peer and to a loopback
peer with a foreign `Host`; with a token, a LAN peer gets 401 without it
and 200 with it; the default loopback bind starts as before.

Not tested: a real Unix-domain-socket deployment and a real Prometheus
scrape (both covered by tests only).

## Merge order

Part of a set. Suggested order: #3852 → #3891 → #3849 → #3860 → #3862 →
#3863. This PR goes first: it is the gate the others assume, and it is
the release-note event.

## Not in this PR

- Enforcing the token for loopback callers when one is configured.

Revised 2026-10-02 after review: `/metrics` matches gateway CIDRs
against the TCP peer, and the operator data routes serve the host of a
loopback-published container (bridge-gateway peer under the launchers'
bind acknowledgement) so the default Docker dashboard works.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3852"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-05 08:27:35 -05:00
befdb52fd5 fix(proxy): bill the whole prompt on the gateway path so savings read true (#3632)
## Description

`/stats` reported **97% of tokens saved** on a real Claude Code session
through Kong. The compression is real, the percentage was not.

Anthropic reports a cached turn as `input_tokens` = the *uncached
remainder*, with the cached prefix in `cache_read_input_tokens` /
`cache_creation_input_tokens`. The direct Anthropic handler already sums
all three into `provider_input_tokens`
([handlers/anthropic.py:3587](https://github.com/headroomlabs-ai/headroom/blob/main/headroom/proxy/handlers/anthropic.py#L3587)).
The gateway path recorded `usage.input_tokens` alone, so `tokens.input`
held the uncached sliver and every ratio over it was inflated. On a
fresh 5-call run the old formula reports **100.0%**, because the
denominator was 10 tokens against an 113,490-token prompt.

Affects anything speaking the `/v1/compress` contract: the Kong plugin,
the LiteLLM package, custom gateways. The direct proxy path was already
correct.

Second, smaller problem: tokens alone mislead here even when counted
right. Deferred tool schemas sit at the front of the prompt, so after
turn 1 they would have been cache reads at roughly a tenth of list
input. "90% of tokens" is nowhere near "90% of the bill" (measured on
this laptop's demo traffic: 604 calls, ~$3.12 saved of ~$23.34, about
13%).

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `gateway_turn.complete_outcome`: records the whole billed prompt in
`provider_input_tokens` and the remainder in `uncached_input_tokens`,
matching the direct handler
- `gateway_turn.normalize_usage`: new `input_includes_cache`, decided by
the presence of Anthropic's cache fields rather than the key name.
OpenAI's Responses API also calls its field `input_tokens` but already
counts the cached prefix in it, so keying off the name would
double-count every Responses prompt
- `/stats` `savings.by_layer.tool_search.usd`: the cache-aware dollar
value of deferral, the same figure the cost card already uses
- `/stats` `tokens.cost_weighted_savings_percent`: the saving in money
beside the saving in tokens
- dashboard: a "Cost-weighted: X%" line under Tokens Saved, and the "Of
total wire" tooltip now says the denominator includes the cached prefix
- tests: gateway shapes (Anthropic split, OpenAI chat, OpenAI
Responses), and `/stats` honesty on cache-heavy traffic

No request is shaped differently and no bill changes. Reporting only.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/gateway
250 passed, 4 skipped in 122.95s

$ pytest tests/test_gateway_turn_unit.py tests/test_dashboard_token_savings.py \
         tests/test_dashboard_accessibility.py tests/test_dashboard_static_assets.py
64 passed in 10.58s

$ pytest tests/gateway tests/test_gateway_turn_unit.py tests/test_stats_new_input_savings_rate.py \
         tests/test_stats_tool_schema_savings_are_honest.py tests/test_proxy_savings_history.py \
         tests/test_proxy_stats_recent_requests.py tests/test_proxy_dashboard_stats_cache.py \
         tests/test_cost_card_attribution.py tests/test_cost_tracker_totals.py \
         tests/test_cost_budget_total_prompt.py tests/test_cost_tracker_counterfactual.py
376 passed, 5 skipped   (one pre-existing failure, test_outcome_anthropic_cache_fields, asserted
                         provider_input_tokens == 300 for a 2,800-token prompt: updated here)

# the new tests, without the source change
FAILED tests/test_gateway_turn_unit.py::test_complete_outcome_bills_the_whole_prompt_on_a_cached_anthropic_turn
FAILED tests/test_gateway_turn_unit.py::test_complete_outcome_does_not_double_count_openai_responses_cached_tokens
FAILED tests/test_gateway_turn_unit.py::test_complete_outcome_does_not_double_count_openai_cached_tokens
FAILED tests/test_stats_tool_schema_savings_are_honest.py::test_stats_publishes_the_cost_weighted_saving_next_to_the_token_saving
FAILED tests/test_stats_tool_schema_savings_are_honest.py::test_tool_schema_layer_carries_its_dollar_value

$ ruff check / ruff format --check (0.16.7) on the changed files
All checks passed! / 5 files already formatted
$ mypy --ignore-missing-imports headroom/proxy/gateway_turn.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, Headroom built from this branch,
licensed extensions installed, Kong 3.9 (DB-less,
`kong-plugin-headroom`) in Docker in front of `api.anthropic.com`,
Claude Code 2.1.273 as the client
- Exact command / steps: same stack before and after the change; Claude
Code fixes three planted bugs in a small Python project through Kong (5
model calls), then `GET /stats`
- Observed result, one run on the fixed build:

```text
billed prompt 113,490 = cache_read 86,430 + cache_write 27,050 + uncached 10
old formula would have reported: 100.0%   (denominator was just the 10 uncached tokens)
fixed formula reports:            32.52%
tokens.cost_weighted_savings_percent: 27.7%
savings.by_layer.tool_search: 54,590 tokens, $0.043
cost card: paid $0.10, saved $0.04 (27.7%)
```

- On a longer real session through the same stack (110 calls, Opus +
Sonnet), the old figure was 97%; recomputing that session's rows at
provider cache rates gives ~15% of input tokens and ~13% of spend
- Not tested: non-Anthropic gateways end to end (OpenAI and Responses
shapes are covered by unit tests over `normalize_usage` /
`complete_outcome`, not a live gateway run)

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: reported numbers only. Gateway users'
"tokens saved %" drops sharply (97% -> ~15% on a long session) because
it was wrong in our favour. Worth a release note so nobody reads it as a
regression
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: `tokens.input` on the gateway path now counts
the cached prefix, so any external dashboard differencing it against
`original_tokens` sees a step change
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Found while building a Kong + Headroom demo: the dashboard claimed 90%+
tokens saved while the invoice moved about 13%. The dollar figures were
already cache-aware; only the token percentage was wrong, plus the
missing per-layer dollars.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-04 15:43:11 -07:00
Tejas Chopra c02929846c fix(proxy): make the output shaper actually reduce output in the default mode (#3773)
## What's wrong

**`HEADROOM_OUTPUT_SHAPER=1` reduced nothing.**

`HEADROOM_MODE` defaults to `cache` (`server.py:5707`) and the `coding`
profile pins it. In cache mode `resolve_verbosity_level` forced the
level to `0` unconditionally, deliberately outranking even an explicit
`HEADROOM_VERBOSITY_LEVEL`:

```python
if not settings.steering_enabled:
    # It deliberately outranks the manual override below: a level set in the
    # environment must not reintroduce a prefix mutation the mode exists
    # to prevent.
    return 0, "cache_mode"
```

So for effectively every deployment, the shaper turned on, labelled its
requests, and left the body untouched. Both documented knobs were
silently ignored, with no warning and no override.

## Why the ban was too broad

It guards the right hazard with the wrong instrument.

What costs a prefix cache is the steering block **changing** — appearing
on a conversation already in flight, or moving between levels — because
it lives in the `system` array, inside the cached prefix of every
message-level `cache_control` breakpoint a harness sets.

What costs nothing is the block **being there**. `shape_request` applies
it on every request with no turn-kind gating, and the text is
byte-stable per level and idempotent via the `<headroom_output_shaping>`
sentinel. A level fixed at startup lands in turn 1's prefix and is
byte-identical on every turn after.

The file already said this and then contradicted itself: the module
docstring claims the block is appended "after any `cache_control`
breakpoint, so the provider prefix cache is preserved", while
`steering_allowed_for` says steering "writes into that key".

## Measured

Anthropic, `claude-sonnet-4-5`, 9.1k-token cached system prompt, four
sequential turns, `cache_control` on both the system block and the
trailing message. Arms isolated by a nonce — without that the second arm
reads the first's cache and both cold-start numbers are meaningless.

| turn | control write / read | L2 write / read |
|---|---|---|
| 1 | 9124 / 0 | 9206 / 0 |
| 2 | 100 / 9124 | 157 / 9206 |
| 3 | 126 / 9224 | 137 / 9363 |
| 4 | 161 / 9350 | 132 / 9500 |

**Identical caching pattern. No bust on any turn.** The whole difference
is the block's own ~82 tokens riding in the cached prefix — about
$0.0003 once and $0.000025 per turn at Sonnet rates.

Against that, a separate paired A/B on a long explain-with-code turn (3
pairs, non-overlapping ranges): **2375 → 1663 output tokens, −30%**, ≈
$0.011/turn. Roughly 450x the cache cost.

## The change

`resolve_verbosity_level` allows through the two sources that **cannot
move**:

| source | when |
|---|---|
| `env_pinned` | `HEADROOM_VERBOSITY_LEVEL` set explicitly |
| `cache_mode_default` | otherwise — the settings level, fixed at
startup |

The second is what makes `HEADROOM_OUTPUT_SHAPER=1` sufficient alone.
The shaper is opt-in (the `proxy_output_shaper` rollout spec
deliberately does not default-enable it), so an enabled shaper is
already an explicit request.

The controller and learned profile stay out of cache mode entirely —
they rewrite their files while conversations are open. Skipping the
lookup also spares the default mode two filesystem stats per request.
Autotune being ignored is warned once, naming `HEADROOM_MODE=token`.

**Default level 3 → 2**, named `DEFAULT_VERBOSITY_LEVEL`. L3 instructs
the model to drop content ("give conclusions only; omit rationale") —
the same reasoning the rollout spec gives for not defaulting the shaper
on at all. L2 only forbids ceremony and restatement, and is what the
−30% above was measured at. This also settles an existing inconsistency:
`from_env` defaulted to 3 but fell back to **2** on a malformed value.

## ⚠️ Behaviour changes

Three, all disclosed:

1. **Cache mode with the shaper on now steers.** That is the point.
Blast radius is exactly "operators who set `HEADROOM_OUTPUT_SHAPER=1`" —
the people who asked for output shaping and were not getting it.
2. **A pinned `HEADROOM_VERBOSITY_LEVEL` is now honoured in cache
mode**, where it was previously discarded.
3. **The default level drops 3 → 2**, including in token mode. Existing
token-mode users with the shaper on and no level set get *less*
aggressive shaping: lower savings, lower risk of the terseness reading
as regression.

Five existing tests encoded the old policy and are inverted here
deliberately, including the gateway-contract one
(`test_cache_mode_labels_but_never_steers` →
`test_cache_mode_steers_at_the_default_level`), which verifies the
change end-to-end through the real `/v1/compress` path rather than only
at the unit level.

`/stats` needed no change: it builds the same settings the handlers do
and reads the resolved level, so it follows automatically.

## Tests

New coverage:

- `HEADROOM_OUTPUT_SHAPER=1` alone resolves `(2, "cache_mode_default")`
- a pinned level resolves `(4, "env_pinned")` in cache mode
- the learned profile and the controller are both ignored there
- the ignored-autotune warning fires exactly once and names
`HEADROOM_MODE=token`
- cache mode consults **no workspace files** (asserted by making
`workspace_dir()` raise)
- **ten turns of a growing conversation produce a byte-identical
`system` array**, with the client's own `cache_control` still on the
first block — the cache-safety claim asserted rather than argued

340 tests pass across shaper / verbosity / steering / output_reduction /
cache_stability / output_savings / gateway_turn. `ruff check`, `ruff
format --check` (0.16.7) and `mypy headroom --ignore-missing-imports`
clean.

Local full-suite runs error en masse in a way unrelated to this change
(files that error in the full run pass individually), so CI is the
arbiter on the complete suite.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-09-24 14:18:17 +02:00
Tejas ChopraandClaude Opus 5 a9757c9c7e fix(tool-search): make the core-tools override authoritative and the warning recurrent (#3745)
Three defects found while tracing why a customer benchmark showed
**+108% cost** with Headroom in front of Claude Code. Follow-on to
#3739.

## 1. The override was not authoritative on the OpenAI path

`helpers.py` unioned `terminal` into the resident set *after* applying
the operator's override. Setting `HEADROOM_TOOL_SEARCH_CORE_TOOLS` to a
list that excluded `terminal` got it back anyway, silently — the one
tool most likely to carry a large schema.

`resolved_core_tools` now takes provider additions as an `extra`
argument folded into the **default** only. An explicit override replaces
everything, provider additions included. That is what an override means.

## 2. The warning armed once per process

The "client tool search is disabled" warning was already
`logger.warning` — the defect was that it armed once per process. A
long-lived proxy reported the condition on the first request and went
quiet forever. **Twelve benchmark runs produced a single log line**,
which is why this took so long to find. It now re-arms hourly.

## 3. One knob, two spellings

Core read `HEADROOM_TOOL_SEARCH_CORE_TOOLS`; the plugin read
`HEADROOM_TOOL_SEARCH_CORE`. Both now resolve to the same setting —
canonical wins, legacy is honoured and warns once naming the
replacement. No existing configuration breaks.

## Testing

40 tests in `test_tool_search_client_deferral.py`, 1115 passing across
the related suites. ruff clean.

## Note on defaults

This PR does **not** change what is resident by default. Claude Code
defers all its built-ins as of v2.1.69, and matching that is the obvious
next step — but there are open client-side bugs against it (Grep/Glob
disappearing on macOS/Linux under `ENABLE_TOOL_SEARCH=true`, strict
endpoints 400ing the first request, signed-thinking-block errors on Opus
4.8). Those are upstream, not ours; they are the reason this PR makes
the knob a real off-switch first.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

### Update after self-review

Two live bugs, **one of them introduced by this PR**:

```
HEADROOM_TOOL_SEARCH_CORE_TOOLS="bash, read, terminal"
-> resident set: [' read', ' terminal', 'bash']
-> read: DEFERRED, terminal: DEFERRED
```

The override never stripped whitespace — `part.strip()` gated only the
emptiness test, while the key function lowercases and strips leading
underscores but not spaces. So the natural spelling of a list defers
exactly the tools you asked to pin. This was latent before: the OpenAI
injector re-unioned its resident names *after* the override and masked
it. **Removing that union in this PR is what made it bite.**

Separately, an override could defer the client's own `ToolSearch` —
Claude Code reaches tools held in a local registry through it and
nothing else can, so an override that omitted it orphaned that whole
category. A search-tool floor now survives every override.

Also: the hourly window was stamped only when the hint **fired**, so
once the operator fixed the condition the stamp stopped advancing and
every later request paid the full O(tools) scan for the life of the
process — a hot-path regression this PR added. It now throttles *scans*,
and claiming the slot before scanning collapses the old two-call
protocol. The legacy-var flag is now lock-guarded with a reset helper,
the gateway reference hook honours `resolved_core_tools()` instead of
the raw default (it ignored the override entirely), and the hint test no
longer monkeypatches stdlib `time.monotonic` process-wide.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-23 23:11:26 +02:00
Tejas ChopraandClaude Opus 5 be00a798c3 feat(proxy): accept the OpenAI Responses shape on the gateway turn (#3661)
## Description

`POST /v1/compress` accepted two wire shapes that both carry a
`messages` list: Anthropic Messages and OpenAI Chat Completions. Codex
speaks the third one — OpenAI Responses — where the transcript is
`input` (typed items) and the system prompt is `instructions`. That body
was rejected outright:

```
HTTP 400 {"error":{"type":"invalid_request","message":"Missing required field: messages"}}
```

So a gateway (Kong, LiteLLM) in front of Codex failed open on every turn
and Headroom did nothing for it. This adds the third shape.

Closes #

## Type of Change

- [x] New feature (non-breaking change that adds functionality)
- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

**A view, not a translation.** A Responses transcript carries fields the
provider hands back to itself — `reasoning.encrypted_content` above all,
plus `call_id` pairing and Codex's
`internal_chat_message_metadata_passthrough`. Translating to chat
messages and back loses them *silently*: the request still looks valid.
`_responses_input_to_waste_messages` already carries that warning —
telemetry-only, "never used as a compression input".

So `headroom/proxy/gateway_responses.py` builds a view:

- Every compressible text slot becomes exactly one pseudo message. Never
joined, so putting the text back never has to guess how to split it.
- Items that are not text slots produce no message at all. The
compressor cannot see or rewrite them.
- Write-back is positional; a view that does not line up forwards the
original rather than guessing.

**Call inputs are slots too.** Measured across 51 real Codex sessions:
`custom_tool_call.input` is 8.9% of transcript bytes and is resent every
turn. `function_call.arguments` is a JSON document the provider parses,
so it is rewritten only when the replacement still parses (`_json_safe`)
— it is 0.3% of bytes and real ones carry encrypted payloads, so little
is lost when the guard refuses.

**Three bugs this turned up**, each with a test named after it:

| | |
|---|---|
| `build_provider_body` added `messages` unconditionally | the
`x-headroom-bypass` header — the path whose whole job is to change
nothing — turned every Codex request into a provider 400 |
| `_redrive_payload` did the same on the second leg | every re-drive on
a Responses turn was malformed, CCR and tool-router alike |
| `create_ccr_tool_definition` fell through to the nested chat shape |
CCR injected a tool the provider rejects, on exactly the turns where
markers made it necessary |

The response half needed nothing: `normalize_usage` already read
Responses usage via `input_tokens_details.cached_tokens`.

`wire_shape` is deliberately separate from `provider` on `PendingTurn`:
`provider` picks pricing, tokenizer and the upstream the gateway is
advised to call, and all three want `"openai"` for a Responses turn
while the tool-call shape and the re-drive body want
`"openai_responses"`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```
tests/gateway + test_gateway_turn_unit + test_gateway_sidecar_ports
  319 passed, 4 skipped

tests -k "ccr or compress"
  2307 passed, 72 skipped, 10860 deselected

ruff check   All checks passed!
mypy         Success: no issues found in 1 source file
```

**Verified against the live OpenAI Responses API**, because a malformed
body only shows up when a real provider validates it:

```
=== compressed Codex body ===
  compress   2,436 -> 1,632 tokens   shape input=True messages=False
  provider   HTTP 200

=== tool-router re-drive (both legs) ===
  compress   tools -> 1  names=['search_tools']
  provider   round 0  tool_calls=['search_tools']
  RE-DRIVE 1: input=True messages=False tools=11  shape_ok=True
  provider   round 1  tool_calls=['zz_obscure_capability_47']
  action=done after 1 re-drive(s)

=== CCR re-drive ===
  compress   85,105 -> 41,903  ccr_hashes=1  tools=['headroom_retrieve']
  provider   tool_calls=['headroom_retrieve']
  RE-DRIVE 1: input=True messages=False shape_ok=True
  ...
  answer: 'ZQ-7741-KLM'   <- buried at line 2500 of a 4000-line folded output
```

Chat Completions and Anthropic bodies were re-checked live alongside
Responses and are byte-identical in behaviour.

## Known limitations

- Codex streams by default, and both plugins set `can_redrive =
allow_redrive and not stream`. Tool-router and CCR savings therefore do
not apply to streaming turns; schema compaction and content compression
still do.
- `reasoning` items stay opaque by design.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 15:49:45 -07:00
Tejas ChopraandClaude Opus 5 aceff2ea9d fix(proxy): gateway turn fixes found by benchmarking behind a gateway (#3619)
## Description

Fixes five defects found by running real Claude Code sessions through a
gateway that uses the `/v1/compress` turn contract (#3605).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- **Prompt text is never compressed on replaying callers.** The coding
profile compresses user messages, and on turn 1 that rewrote a plain
agent's task ("Fix this bug in the library source" became "Fix bug in
source", and test ids lost their `tests/test_` prefix). When
`prefix_replay_guaranteed` is set, `content_router` protects the opening
user messages and the newest user message, for string and list content
alike. The one exception is a newest user message that answers a fenced
shell command in the assistant turn before it: that is a text harness's
tool observation and still compresses. Tool results still compress, and
cache freeze is unchanged.
- **Anthropic's four cache breakpoint limit.** One benchmark request
failed with `max 4 cache_control blocks, found 5`. Session turns in the
OpenAI handler now re-place markers with
`normalize_message_cache_control`. A last-stop guard,
`enforce_cache_breakpoint_budget`, runs in the gateway extension for
Anthropic bodies: it re-places markers, then drops the oldest extras,
and logs `event=cache_breakpoint_budget` when it repairs.
- **Failure relay without a body.** `POST /v1/compress/response`
rejected an upstream 4xx/5xx without a response body as
`missing_response`. A body is now required only on 2xx.
- **Usage totals after a re-drive.** The done payload carries
`billed_usage` summed across rounds. After one or more re-drives,
`response` is never null: it is the hook's replacement or the latest
provider response (including when the hook hands back that same object),
with `usage` set to the totals in its own key shape (chat or Anthropic).
Turns with no re-drive keep `response: null`. Before, the client saw the
last round only.
- **Cold Kompress stall.** When native Kompress init is deferred at
startup, a daemon thread preloads it after a short delay.
`HEADROOM_KOMPRESS_WARMUP=0` disables it. It is off under pytest and
skipped on glibc < 2.28 (the #1908 segfault hosts).
- Docs: `proxy.mdx` and `wiki/proxy.md` describe the 2xx-only body rule
and `billed_usage`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`) on the changed modules
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_transforms_content_router.py tests/test_compress_session_mode.py \
    tests/test_compress_route_tokenizer_by_model.py tests/gateway tests/test_proxy \
    tests/test_kompress_preload_deferral.py
672 passed, 4 skipped in 228.49s

$ uvx ruff@0.16.3 format --check <changed files> && uvx ruff@0.16.3 check <changed files>
12 files already formatted
All checks passed!

$ mypy <changed modules>
Success: no issues found in 6 source files
```

New tests: 5 prompt-protection tests in
`test_transforms_content_router.py` (3 fail without the fix),
`tests/gateway/test_cache_breakpoint_budget.py`, failure-relay and
billed-usage tests in `test_redrive_loop.py`, `billed_usage` in the
response-half contract test,
`tests/test_proxy/test_kompress_background_warmup.py`.

## Real Behavior Proof

- Environment: headroom proxy on this branch behind a gateway, Claude
Code with claude-sonnet-5, real Anthropic calls.
- Exact command / steps: 8 smoke sessions (32 calls) on SWE-bench-style
tasks, with a spend logger recording every provider round.
- Observed result: 0 errors. Task statements reached the provider
verbatim. On every re-driven call (2 and 3 rounds) the client's usage
equalled the sum of the per-round provider usage. Kompress warmed in the
background in about 4 s at startup; in a replay of the slow session the
first code-aware turn went from 5.4 s to 1.3 s.
- Not tested: glibc < 2.28 hosts (warm-up is skipped there by design).
The original body behind the 5-breakpoint error was not captured, so the
guard is covered by unit tests rather than a live repro.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: stable.
- Stable/default behavior changed: yes. Prompt text on replaying callers
is no longer compressed, the done payload adds `billed_usage`, and
Kompress preloads in the background when deferred.
- Kill switch / disable path: `HEADROOM_KOMPRESS_WARMUP=0` for the
warm-up. The other changes are correctness fixes with no switch.
- Unsafe override required: no.
- Qualification impact: none expected; the breakpoint guard only acts on
bodies that would otherwise get a 400.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Open follow-up, not in this PR: steady-state code-aware compression
falls back to Kompress at 0.3 to 1.3 s per tool result even when it
saves nothing. A gate for that is worth a separate change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 08:24:53 -07:00
Tejas ChopraandClaude Fable 5.1 d9757dcc6b fix(proxy): gateway turn contract for /v1/compress behind a compress-turn seam (#3605)
## Description

Adds a gateway turn contract to `POST /v1/compress`, allowing an API
gateway to submit a complete provider request and later report the
provider response. Requests without a top-level `gateway` object remain
unchanged.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Adds the request half to `/v1/compress` and the response half at
`/v1/compress/response`, including bounded pending turns, idempotent
rounds, usage relay, and redrive obligations.
- Adds a reusable six-stage compress-turn extension seam and turn-hook
priority/provider-header support.
- Keeps native tool-schema deferral in the licensed extension while
sharing protocol-shape helpers with core.
- Accepts Anthropic, OpenAI chat, OpenAI Responses, and Kong-shaped
usage payloads.
- Documents the gateway contract in `docs/content/docs/proxy.mdx` and
`wiki/proxy.md`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
tests/gateway + gateway unit + turn-hook seam + test_issue_746_tool_search:
358 passed

ruff 0.16.3 lint/format and mypy on touched modules:
clean

Kong 3.9 Docker e2e (HEADROOM_KONG_E2E=1):
4 passed
```

## Real Behavior Proof

- Environment: Kong 3.9 Docker e2e and real-provider Claude Code through
Kong on SWE-bench-mini.
- Exact command / steps: `HEADROOM_KONG_E2E=1` gateway e2e;
real-provider traffic through the gateway contract with usage relay and
licensed native deferral.
- Observed result: four Kong e2e tests passed; prefix cache remained
intact with zero busts; relayed usage matched Anthropic token-for-token;
licensed tool-search deferral reduced cache-read tokens per turn by
27–30%.
- Not tested: production rollout at scale.

## Runtime Rollout Safety

- Rollout-managed feature(s): gateway contract and in-process
pending-turn registry.
- Minimum rollout channel: controlled gateway deployment before broader
enablement.
- Stable/default behavior changed: only requests containing a top-level
`gateway` object enter the new contract; ordinary `/v1/compress` bodies
are unchanged.
- Kill switch / disable path: set `HEADROOM_GATEWAY_CONTRACT=0`.
- Unsafe override required: no.
- Qualification impact: gateway callers must redeem advertised `redrive`
and `relay_usage` obligations.
- Rollback path: disable the contract with `HEADROOM_GATEWAY_CONTRACT=0`
or revert the feature commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable; this is an API contract change.

## Additional Notes

Pending turns are process-local and bounded by
`HEADROOM_GATEWAY_TURN_TTL_SECONDS`,
`HEADROOM_GATEWAY_MAX_PENDING_TURNS`, and
`HEADROOM_GATEWAY_MAX_REDRIVES`.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-15 16:56:07 -07:00