feat(proxy): add lossless /v1/compress mode and detect provider by wire shape (#4617)

## Description

Two gaps stop a gateway from using `/v1/compress` well. Both showed up
while fixing LiteLLM's `headroom` guardrail, which is the follow-up PR
that needs this one.

1. **No mode compresses messages losslessly without markers.** A gateway
with no retrieval loop can't resolve a marker or call
`headroom_retrieve`, so it needs a mode with nothing to retrieve. The
existing marker-free modes still run lossy compressors.
2. **Hooks guess the provider from the model name.** A gateway can send
any client's shape to any model. Claude Code's Anthropic body routed to
Gemini looked like OpenAI, so skill condensing found nothing to change.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)

## Changes Made

**`config.mode="lossless"`**
- Runs format-native folds only, with no markers and no CCR store
writes. A registered lossless provider still applies.
- Turns off stale-Read pointers, which drop the file body and need the
retrieval tool.
- Turns off cross-turn dedup pointers, which a client that never numbers
its messages can't resolve.
- Scope: the guarantee covers message compression (lossless codecs, no
CCR). On a gateway turn, tool schema compaction and registered turn
hooks still run as configured, as in every other mode; description
truncation only happens when an operator sets
`HEADROOM_TOOL_DESC_MAX_CHARS`.

**`provider_for_body`**
- Hooks and the session tracker now use the body's wire shape, and fall
back to the model name when the shape proves nothing, such as plain text
with no system prompt or tools.
- Anthropic's own tool types (server tools such as
`web_search_20250305`, client tools such as `bash_20250124`) end in an
eight-digit date and count as the Anthropic shape, so a first-turn body
whose only tool is a server tool is read correctly.
- `route.provider` still follows the model.
- The cache breakpoint budget runs when either the shape or the model is
Anthropic, and never on a Responses body.
- `gateway_response_fields` loses its unused `provider` parameter, and
`RESPONSE_RECEIVED` keeps reporting the model's provider.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`, `ruff format --check .`, on the
pinned 0.17.0)
- [ ] Type checking passes (`mypy headroom`): one error, in
`headroom/integrations/langchain/chat_model.py`, which this PR doesn't
touch (local mypy 1.19.1)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
pytest tests/test_proxy_compress_endpoint.py tests/gateway tests/test_gateway_turn_unit.py tests/test_compress_route_tokenizer_by_model.py
425 passed, 4 skipped

Regression tests with the fixes removed: 8 failed, 7 passed
```

## Real Behavior Proof

- **Environment:** LiteLLM main with the follow-up guardrail change,
this branch as the sidecar, and a fake upstream that simulates prompt
caching. No real provider API was called.
- **Exact command / steps:** replayed a scripted 17-call agent session
with exact-duplicate outputs removed, once direct to the fake upstream
and once through LiteLLM and `/v1/compress`.
- **Observed result:** 0 errors, 0 rewrites of already-sent results,
`Read` and `Edit` results unchanged, about 1.3s added in total.

| Client | Prompt cost vs direct |
|---|---|
| Anthropic Messages, Claude Code cache markers | −3.9% |
| Chat Completions | −3.8% |
| Responses | −4.1% |

The script's outputs are synthetic, so treat these as a comparison
between setups, not a forecast for real traffic.

## Known limits

- The default mode and `lossy_inline` can still return a stale-Read
`Retrieve original: hash=` marker, although the docs call them
marker-free. This predates the PR and needs a separate fix. Proxy
`--lossless` has no read-lifecycle guard either, from reading the code.
- With no `ContentRouter`, `_derived_compress_pipeline` falls back to
the default pipeline, so `lossless` wouldn't hold. Production always has
a router, so only test stubs reach this.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4617"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

## Maintainer post-merge revisit: main e9331b6b
Published head `e50b001bd05c9bc85405accca49cbef0e7786e51` incorporates
upstream main `e9331b6b963b5b6554f45943abed2366aaa25ea6` after #4142,
#4637 and #4641 merged. 164 endpoint/gateway/tokenizer/admission tests
pass; one json_minifier strategy assertion fails identically on actual
main e9331b6b. Ruff/format/diff pass. Import-only conflict resolved by
retaining both provider_for_body and RequestBodyTooLarge. Both existing
holds reconfirmed: lossless HTTP output removes unique description
suffix via tool_desc_compaction; Anthropic server tool with Gemini model
is classified OpenAI.

Contributor correctness logic was not rewritten. The existing Changes
Requested decision remains applicable; unchanged findings are not
reposted as duplicate reviews. Scoped Windows qualification uses the
existing development native extension, not a fresh build. Hosted checks
must finish independently; no contributor CI action requested and
nothing merged by this review.


## Maintainer post-merge revisit: main cc1eb540
Published `38b871120e318f6eaa1cb2ce16f96d4f6cc2b1e8` includes upstream
main `cc1eb5402aade58204b96b6b4b4d280cdd5456d2`. 90 gateway/tokenizer
tests pass; both existing HTTP/provider counterexamples reconfirmed.
Clean merge; Ruff/format/diff pass. Contributor correctness logic was
not rewritten. The existing Changes Requested decision remains
applicable; unchanged findings are not reposted. Scoped Windows
qualification uses the existing development native extension, not a
fresh build. Hosted checks must finish independently; nothing merged by
this review.


### Maintainer current-main requalification

Exact head `33395b5cdd7ee9facce9f97f2a3250f5e6e49cb2`, upstream main
`ecbaf86d78cf52d185fbffa306157b9a4cc1b906`.

Mechanically merged upstream/main
ecbaf86d78 without changing contributor
policy logic. Re-reviewed the combined gateway/provider delta and
existing comments. All 90 focused gateway/tokenizer tests and ten
inherited Gemini fixture cases pass; Ruff0.17.3 lint/format and
whitespace checks pass. The shared Gemini repair was already present, so
no duplicate fixture commit was added.

Both existing correctness findings independently remain on this combined
tree. An HTTP200 /v1/compress request with config.mode=lossless still
drops UNIQUE_REQUIREMENT_AT_END from a long tool description, reporting
tool_desc_compaction. An Anthropic web_search_20250305 tool body with a
Gemini-named model still returns provider openai. Existing requested
semantic fixes/regressions remain necessary; maintainers are handling CI
and no contributor CI execution is requested. No contributor logic was
rewritten and nothing merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `4c40adb53a68f382e32f00620adcfaf1cff477f5`, upstream main
`8d3e9e5ca7cd645eb73084b848edbea5fc796e1a`.

Clean mechanical sync with upstream
main8d3e9e5ca7cd645eb73084b848edbea5fc796e1a including intervening
release/runtime/cache/CCR/SSE/HTTP1.1 merges. No contributor
gateway/provider policy rewritten. Fresh author/Devin discussion
considered.100focused gateway-contract/tokenizer/Gemini tests
pass;7changed Python files pass Ruff0.17.3check/format and whitespace
checks pass. Last main delta after test start changes only
Rust/OpenClaw, leaving tested Python inputs unchanged.

Both existing blockers remain freshly reproduced from this worktree
(source import path verified): real app POST/v1/compress lossless
returns200 but drops UNIQUE_REQUIREMENT_AT_END from tool description via
tool_desc_compaction; Anthropic server-tool wire shape with Gemini model
returns provider openai. Preserve promised lossless content and classify
server-tool wire forms correctly, with regressions. No external provider
session/nativebuild claimed. Copied extension supports Python startup;
maintainers handle hosted checks. Nothing merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `8b7b2b0d02daf1a031828ecb13095ab315431f7e`, upstream main
`917edc62230c3c4a8a415e2053da68dab1f003dd`.

Current-main integration against917edc62230c3c4a8a415e2053da68dab1f003dd
is clean. Compared entire headroom/crates/Python manifests+lock/tests
inputs to prior qualified4c40adb53: byte-identical. Intervening changes
are reviewed npm maintenance and release-envelope recovery workflow; no
contributor gateway policy changed.75new-base
release-workflow/normalizer tests pass,whitespace passes.
Prior100gateway/tokenizer/Gemini results remain scoped to those
identical inputs rather than relabeled fresh.

Both gateway blockers freshly rechecked from worktree: Anthropic
server-tool+Gemini body classifiesopenai; actual lossless/v1/compress
returns200 but drops unique description suffix via tool_desc_compaction.
Retain existing losslessness/classification corrections. No fresh
nativebuild/provider claim. Maintainers handle exactheadCI;nothing
merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `490ff6defdebddda9106e9e143d42d1e463b6324`, upstream main
`7592e681459f14a2041eb5f524a658cb6ad1dd92`.

Clean integration with upstream/main
7592e68145. Reviewed both intervening
memory fixes and fresh PR discussion. Gateway/runtime/Rust inputs are
unchanged from the previously reviewed head; only the two main memory
fixes and their regressions enter this merge. Nine memory/Gemini
regression tests pass; main-relative whitespace passes.

Fresh actual-app probes retain both existing blockers: Anthropic
server-tool content with a Gemini model classifies as OpenAI; lossless
/v1/compress removes the unique tool-description suffix through
tool_desc_compaction. No contributor policy was rewritten. Existing
native extension used for Python startup, not a fresh native-build
claim. Maintainers handle hosted checks; nothing merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `d3cb3a02af03991a696461160ee0db322dd8bf48`, upstream main
`2d3ee51846435ef5b320eea4806af14e602b0d8c`.

Clean integration with upstream/main
2d3ee51846 after the external 0.43.1
release merge. Exact headroom/crates/tests comparison against prior
reviewed 490ff6def shows no changes. Re-reviewed generated release
metadata; 78 release/version/MCP tests pass on this composed tree, and
whitespace passes. Previous nine memory/Gemini regressions and fresh
gateway probes remain scoped to identical runtime inputs.

Both established blockers remain: lossless /v1/compress drops the unique
tool-description suffix, and Anthropic server-tool content with a Gemini
model classifies as OpenAI. Runtime probes were not relabeled as rerun
after this metadata-only merge. No contributor policy rewritten; no
fresh native-build claim. Maintainers handle exact-head checks; nothing
merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `e0c873630e9ffc7bc0d0d2fdd22c483b731e49a3`, upstream main
`910588e6b15ab54c4099d03703e5fa20c8abb757`.

Clean mechanical synchronization with upstream
main910588e6b15ab54c4099d03703e5fa20c8abb757, including the externally
merged runtime fixes, docs dependency update and tree-sitter lock
repair. No contributor gateway/provider policy rewritten. Current
author/Devin discussion considered. All90 gateway
contract/tokenizer-by-model tests pass on the preceding composed runtime
tree; the final dependency-only merge changes only pyproject/uv.lock,
leaving those gateway/tokenizer sources unchanged. No redundant rerun on
the final merge is claimed. Changed Python Ruff lint/format and
main-relative whitespace pass.

Both existing blockers freshly reproduce on the composed runtime paths:
lossless /v1/compress returns200 but drops UNIQUE_REQUIREMENT_AT_END
from the tool description and reports tool_desc_compaction; Anthropic
server-tool content with a Gemini model is classified as OpenAI.
Preserve the lossless original payload and classify the full request
protocol consistently, with endpoint regressions for these inputs. The
main fixes do not close either defect.

Tests use the existing native core for startup, not a fresh native
build; no live provider traffic was made. Maintainers handle hosted
checks; no inherited-main pipeline failure attributed and nothing
merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `507433606f698249e546e1418928caee99d0c901`, upstream main
`7877241d84c93b3ec487e07324245838ddae85cc`.

Clean mechanical integration with
upstream/main7877241d84c93b3ec487e07324245838ddae85cc. Gateway
production/contracts remain byte-identical to the preceding reviewed
head; re-reviewed the new CLI/Hermes integration.127 gateway
request/response contract and model-tokenizer tests pass; whitespace
passes. Both existing holds freshly reproduce on this exact tree:
lossless /v1/compress returns200 while dropping
UNIQUE_REQUIREMENT_AT_END and reporting tool_desc_compaction; Anthropic
server-tool content with a Gemini model is classified as OpenAI.
Preserve the lossless original payload and classify the full request
protocol consistently, with endpoint regressions. No contributor
gateway/provider policy rewritten. Existing native core used for
startup; no fresh native build or live provider claim. Maintainers
handle hosted checks; nothing merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `b0daece6b10f623a6e7b0dc3eb01c73e05fa3209`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Final post-Dependabot requalification against upstream/main
c01cf07d94. All six Dependabot updates are
now merged. Relative to this branch's freshly tested prepared tree, the
final main sync changes only the three intended Cargo.lock records;
Python/source/test inputs are unchanged. Re-reviewed the integration and
confirmed complete native source/lock/toolchain objects match the tested
combined build and exact fresh native DLL used for this branch's local
tests.

Focused fresh-native qualification: 127 passed, 3 warnings in 104.78s
(0:01:44). Selected real dependency overlays and native DLL paths are
recorded in final4617-fresh-native-tests.log; the DLL provenance/input
hashes are in final-native-artifact-proof.json. No redundant rerun is
implied after this Cargo-only merge. Whitespace passes. Prior source
review and scoped real-behavior evidence remain applicable to unchanged
contributor inputs.

Both existing blockers freshly reproduce using the combined native DLL:
lossless /v1/compress returns200 but removes UNIQUE_REQUIREMENT_AT_END
and reports tool_desc_compaction; Anthropic server-tool content with a
Gemini model is classified as OpenAI. Preserve the original lossless
payload and classify the complete wire protocol, with endpoint
regressions.

Maintainers handle hosted checks; no external-provider claim,
contributor semantic rewrite or non-Dependabot merge.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `ec6abc0228a93c5eab1881938041dfee344b1e3a`, upstream main
`2973ca593d2fe65c3259941fdff8009175eff456`.

Requalified after human merges #4667/#4669/#4670 against upstream main
ce42754912. All 11 integration patches
exactly match the prior-main-to-current-main delta after ignoring diff
locations; no contributor gateway/provider policy changed. Fresh gateway
request/response contracts, model-tokenizer, gateway unit,
Kompress-disable and must-keep suites: 207 passed, 3 warnings. Six
changed Python files pass Ruff lint/format; offline lock validation and
whitespace pass. Existing matching-core DLL used; native
source/lock/toolchain unchanged by these main merges, with no fresh
native build or live-provider claim. Both existing blockers freshly
reproduce: lossless /v1/compress returns 200 while removing
UNIQUE_REQUIREMENT_AT_END and reporting tool_desc_compaction; Anthropic
server-tool content with a Gemini model is classified as OpenAI.
Preserve original lossless payload and recognize complete wire protocol
with endpoint regressions. Author and Devin discussions checked;
existing findings remain actionable. Maintainers handle hosted checks;
nothing merged.
Subsequent human merge #4671 advanced main to
2973ca593d before publication; guard
stopped the old candidate. Cleanly integrated the new workflow-only
delta and re-reviewed action pinning with 62 workflow/release tests
passing. Runtime/native sources are byte-identical to the immediately
preceding 207-test and real-behavior qualification; those results apply
to unchanged inputs. All prior-main hosted results are historical for
readiness.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `9f857218486fa451ecacc675863a64bc95f45bb0`, upstream main
`2b0315b2d4b0d54947050bbdad2de9e89df06378`.

Current-main requalification
against2b0315b2d4b0d54947050bbdad2de9e89df06378 after later human
merges. All37 integration patches exactly match2973ca59-to-current-main
delta after diff-location normalization; contributor gateway/provider
policy unchanged.393gateway request/response
contracts,model-tokenizer,gateway
unit,Kompress-disable,must-keep,doctor,search tests
pass;3warnings.27changedPythonfiles pass Ruff lint/format;offline
lock/whitespacepass. Existing pr3982 native DLL verified every tracked
native-input hash/artifactchecksum;no fresh build claimed. Author/Devin
discussion checked. Both correctness blockers freshly reproduce:lossless
/v1/compress returns200while droppingUNIQUE_REQUIREMENT_AT_END and
reportingtool_desc_compaction;Anthropicserver-toolcontent
withGeminimodel classifiedOpenAI. Preserve original lossless payload and
classify full request wire protocol consistently,with endpoint
regressions. No live provider qualification,contributor logic rewrite,or
inherited-main CI fault attributed. Maintainers handle hosted
checks;nothing merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.


### Maintainer current-main requalification

Exact head `41b2145418db7f5980cd52e6235ab330fcd31ac0`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.

Clean integration with upstream/main
0ddaa56c21. All 17 integration patches
exactly match the prior-reviewed-main to current-main delta after
diff-location normalization; contributor gateway/provider policy is
unchanged. Fresh gateway/tokenizer/search/doctor and incoming
Vertex/backend/CLI qualification: 455 passed, 11 skipped, 3 warnings.
Additional gateway request/response contracts and compress-endpoint
suite: 158 passed, 1 warning. Twelve changed Python files pass Ruff
lint/format; offline lock and whitespace checks pass.

Author discussion and outstanding Devin findings read. Both existing
correctness findings freshly reproduce on this exact tree: lossless
/v1/compress returns 200 but drops UNIQUE_REQUIREMENT_AT_END through
tool_desc_compaction; Anthropic server-tool content with a Gemini model
classifies as OpenAI. Existing requested corrections remain; no new
duplicate finding. Local app probes are not live provider proof. Native
DLL reused only after verifying the complete tracked input set, every
recorded input SHA256 and artifact checksum against
pr3982-c01-native-proof.json; no fresh build claimed. Maintainers handle
exact-head hosted checks. No PR merged.


Exact-head hosted checks are being handled by maintainers. Existing
correctness/qualification holds remain; nothing merged.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Tejas Chopra
2026-11-07 20:36:11 -06:00
committed by GitHub
co-authored by Claude Opus 5.5 JD Davis
parent 8f50884bd5
commit 33f067a243
10 changed files with 367 additions and 31 deletions
+2 -1
View File
@@ -637,7 +637,7 @@ Keep carrying both yourself and send them upstream unchanged. Two consequences w
| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `mode` | string | unset | `ccr`, `lossy_inline`, or `lossless_then_lossy`. Unset selects the default marker-free pipeline. Any other value is a `400`. |
| `mode` | string | unset | `ccr`, `lossy_inline`, `lossless_then_lossy`, or `lossless`. Unset selects the default marker-free pipeline. Any other value is a `400`. |
| `frozen_message_count` | integer | unset | Pin a prefix: the first N messages are returned byte-for-byte unchanged while staying visible to cross-message transforms like dedup. Set it to the number of messages the provider has already cached so compression cannot rewrite the prefix and bust that cache. Must be a non-negative integer; anything else is a `400`. |
| `session_id` | string | unset | Opt into [session mode](#session-mode): Headroom keeps this conversation's replay state itself, so you can resend the raw conversation every turn and get a byte-identical prefix back. Non-empty, at most 256 characters; anything else is a `400`. Cannot be combined with `compress_user_messages` (`400`). |
| `compress_user_messages` | boolean | `false` | Also compress user-role messages. |
@@ -650,6 +650,7 @@ Keep carrying both yourself and send them upstream unchanged. Two consequences w
- **unset (default)** — marker-free. Emits no `<<ccr:…>>` retrieval markers and writes nothing to the CCR store, so you can forward the returned messages straight to a provider. This is the right mode for a gateway or guardrail that just swaps `messages` and forwards.
- **`ccr`** — emits CCR markers and writes to the store. Only for callers that also inject the `headroom_retrieve` tool *and* can reach `/v1/retrieve` (itself loopback-only). Markers are a dangling pointer for the model otherwise.
- **`lossy_inline`** (alias `lossless_then_lossy`) — runs the lossless byte/data fold first, then compresses the folded remainder. Marker-free.
- **`lossless`** — format-native folds only, with no markers and no retrieval. For a sidecar with no retrieval loop; a registered lossless provider still applies. Keep the mode stable within a `session_id`, because session mode replays earlier turns byte-for-byte.
#### Response
+9 -6
View File
@@ -42,6 +42,7 @@ from headroom.proxy.gateway_turn import (
new_turn_id,
on_turn_expired,
parse_gateway_block,
provider_for_body,
provider_for_model,
register_pending_turn,
shape_gateway_body,
@@ -80,7 +81,7 @@ class GatewayTurn:
self.body = body
self.caps = caps
self.client = client
self.provider = provider_for_model(str(body.get("model") or ""))
self.provider = provider_for_body(body, str(body.get("model") or ""))
# The wire shape is a property of the body, not of the model name, and
# it is fixed for the life of the turn: the response half has to write
# a re-drive in the same shape the request half read.
@@ -100,7 +101,6 @@ class GatewayTurn:
request=self.request,
body=self.body,
caps=self.caps,
provider=self.provider,
messages=messages,
tools=self.body.get("tools"),
obligations=[],
@@ -116,7 +116,7 @@ class GatewayTurn:
bind_scope(tags, self.request.scope)
self.model_name = model_name
self.tags = tags
self.provider = provider_for_model(model_name)
self.provider = provider_for_body(self.body, model_name)
self._transformer = RequestTransformer(
provider=self.provider,
model_name=model_name,
@@ -198,7 +198,6 @@ class GatewayTurn:
request=self.request,
body=self.body,
caps=self.caps,
provider=self.provider,
messages=messages,
tools=result.tools,
obligations=self._obligations,
@@ -213,9 +212,13 @@ class GatewayTurn:
input_tokens=tokens_before,
transforms=transforms,
)
if self.provider == "anthropic":
if not self.responses_shape and "anthropic" in (
self.provider,
provider_for_model(self.model_name),
):
# Last stop before the body leaves: a request over Anthropic's
# cache_control budget is a guaranteed 400 on the provider.
# cache_control budget is a guaranteed 400 on the provider. The
# budget belongs to the upstream, so a chat body to Claude counts.
from headroom.proxy.helpers import enforce_cache_breakpoint_budget
provider_body = self._fields["body"]
+66 -8
View File
@@ -41,6 +41,7 @@ import dataclasses
import json
import logging
import os
import re
import threading
import time
import uuid
@@ -50,6 +51,7 @@ from dataclasses import dataclass, field
from typing import Any
from headroom.proxy.gateway_responses import VIEW_MARKER as GATEWAY_RESPONSES_VIEW_MARKER
from headroom.proxy.gateway_responses import carries_view, is_responses_body
from headroom.proxy.outcome import RequestOutcome
from headroom.proxy.tool_schema_savings_policy import without_deferral_flags
from headroom.proxy.turn_hooks import (
@@ -208,11 +210,67 @@ def _parse_request_headers(raw: Any) -> dict[str, str]:
def provider_for_model(model_name: str) -> str:
"""The tracker-provider inference ``/v1/compress`` already uses."""
"""The provider a model name implies, for routing advice and event pricing."""
lowered = (model_name or "").lower()
return "anthropic" if ("claude" in lowered or "anthropic" in lowered) else "openai"
_OPENAI_ONLY_ROLES = frozenset({"system", "developer", "tool", "function"})
_OPENAI_ONLY_BLOCKS = frozenset({"image_url", "input_audio", "input_text"})
_ANTHROPIC_ONLY_BLOCKS = frozenset(
{"tool_use", "tool_result", "thinking", "redacted_thinking", "server_tool_use", "document"}
)
# Anthropic's own tools (server tools such as ``web_search_20250305`` and
# client tools such as ``bash_20250124``) carry a dated ``type`` and no
# ``input_schema``; OpenAI chat tools are ``function`` or ``custom``.
_ANTHROPIC_DATED_TOOL_TYPE = re.compile(r"_\d{8}$")
def _wire_provider_from_shape(body: dict[str, Any]) -> str | None:
"""The provider a body's shape proves, or None when only the model can tell."""
if carries_view(body) or is_responses_body(body):
return "openai"
if body.get("system") is not None:
return "anthropic"
tools = body.get("tools")
for tool in tools if isinstance(tools, list) else ():
if isinstance(tool, dict) and "input_schema" in tool:
return "anthropic"
tool_type = tool.get("type") if isinstance(tool, dict) else None
if isinstance(tool_type, str) and _ANTHROPIC_DATED_TOOL_TYPE.search(tool_type):
return "anthropic"
if isinstance(tool, dict) and "function" in tool:
return "openai"
messages = body.get("messages")
for message in messages if isinstance(messages, list) else ():
if not isinstance(message, dict):
continue
if message.get("role") in _OPENAI_ONLY_ROLES or message.get("tool_calls"):
return "openai"
content = message.get("content")
for block in content if isinstance(content, list) else ():
block_type = block.get("type") if isinstance(block, dict) else None
if not isinstance(block_type, str):
continue
if block_type in _ANTHROPIC_ONLY_BLOCKS or block_type.endswith("_tool_result"):
return "anthropic"
if block_type in _OPENAI_ONLY_BLOCKS:
return "openai"
return None
def provider_for_body(body: dict[str, Any], model_name: str) -> str:
"""The wire shape hooks and steering must treat a gateway body as.
A gateway can send any client's shape to any model: Claude Code's Anthropic
body to Gemini, an OpenAI chat body to Claude. The shape wins over the model
name; ``route.provider`` stays advice about the model.
"""
return _wire_provider_from_shape(body) or provider_for_model(model_name)
# --------------------------------------------------------------------------- #
# Usage normalization #
# --------------------------------------------------------------------------- #
@@ -560,9 +618,9 @@ class PendingTurn:
ccr_armed: bool = False
# The wire shape this turn arrived in, when it is not the one `provider`
# implies: "openai_responses" for a Codex body. Deliberately separate from
# `provider`, which picks pricing, tokenizer and the upstream the gateway
# is advised to call -- all three want "openai" for a Responses turn while
# the tool-call shape and the re-drive body want "openai_responses".
# `provider`, the chat-level wire shape (see `provider_for_body`), which
# wants "openai" for a Responses turn while the tool-call shape and the
# re-drive body want "openai_responses".
wire_shape: str | None = None
@property
@@ -1113,7 +1171,6 @@ def build_provider_body(
transcript the provider will reject.
"""
from headroom.proxy.gateway_responses import apply_view as apply_responses_view
from headroom.proxy.gateway_responses import carries_view, is_responses_body
out = {k: v for k, v in body.items() if k not in BODY_CONTROL_KEYS}
if carries_view(body):
@@ -1175,7 +1232,6 @@ def gateway_response_fields(
request: Any,
body: dict[str, Any],
caps: GatewayCapabilities,
provider: str,
messages: list[dict[str, Any]],
tools: Any,
obligations: list[str],
@@ -1188,7 +1244,9 @@ def gateway_response_fields(
return {
"body": provider_body,
"turn_id": turn_id,
"route": build_route(request, provider_body, provider),
"route": build_route(
request, provider_body, provider_for_model(str(provider_body.get("model") or ""))
),
"obligations": list(obligations),
"gateway": caps.echo(),
"headers": dict(headers or {}),
@@ -1468,7 +1526,7 @@ def _emit_response_received(proxy: Any, turn: PendingTurn, response: Any, status
PipelineStage.RESPONSE_RECEIVED,
operation="gateway.turn",
request_id=(turn.outcome_draft.request_id if turn.outcome_draft else turn.turn_id),
provider=turn.provider,
provider=provider_for_model(turn.model),
model=turn.model,
response=response,
metadata={
+32 -10
View File
@@ -79,7 +79,7 @@ import httpx
from headroom.agent_savings import proxy_pipeline_kwargs
from headroom.ccr.marker_resolution import resolve_markers_in_response
from headroom.ccr.tool_calls import has_ccr_retrieve_tool
from headroom.config import is_tool_excluded, unwrap_tool_call_name
from headroom.config import ReadLifecycleConfig, is_tool_excluded, unwrap_tool_call_name
from headroom.copilot_auth import (
apply_copilot_api_auth,
build_copilot_upstream_url,
@@ -1925,9 +1925,8 @@ WS_FIRST_FRAME_TIMEOUT_SECONDS = 60.0
# Accepted values for ``config.mode`` on POST /v1/compress. Unset/None means
# the default marker-free pipeline. Anything else is a 400 rather than a
# silent fall-through to the default (a typo or an unsupported mode like
# "lossless" would otherwise look like it worked).
COMPRESS_MODES = ("ccr", "lossy_inline", "lossless_then_lossy")
# silent fall-through to the default (a typo would otherwise look like it worked).
COMPRESS_MODES = ("ccr", "lossy_inline", "lossless_then_lossy", "lossless")
# Max wait for a sidecar session's turn lock, on the executor. MUST stay
# well below COMPRESSION_TIMEOUT_SECONDS: with an untimed acquire, a slow
@@ -11029,6 +11028,25 @@ class OpenAIHandlerMixin:
smart_crusher_lossless_only=False, # keep SmartCrusher lossy
)
def _lossless_pipeline(self) -> Any:
"""Pipeline for ``/v1/compress`` ``config.mode="lossless"``.
The per-request form of the proxy's ``--lossless``: format-native folds
only, with no markers and no CCR store, so a sidecar with no retrieval
loop loses nothing. A registered lossless provider still competes.
``handle_compress`` also turns off cross-turn dedup pointers, which a
client that never numbers its messages cannot resolve.
"""
return self._derived_compress_pipeline(
"lossless",
lossless=True,
smart_crusher_lossless_only=True,
ccr_inject_marker=False,
ccr_enabled=False,
# Stale-Read pointers drop the file body and need the retrieval tool.
read_lifecycle=ReadLifecycleConfig(enabled=False),
)
async def handle_compress(self, request: Request) -> JSONResponse:
"""Compress messages without calling an LLM.
@@ -11043,6 +11061,9 @@ class OpenAIHandlerMixin:
``headroom_retrieve`` tool and can reach loopback ``/v1/retrieve``.
* ``"lossy_inline"`` (alias ``"lossless_then_lossy"``) — marker-free,
lossless fold first then Kompress the remainder.
* ``"lossless"`` — marker-free, lossless folds only. This covers message
compression; on a gateway turn, tool schema compaction and registered
turn hooks still run as configured.
Any other ``config.mode`` value is a 400 (see ``COMPRESS_MODES``).
@@ -11068,6 +11089,7 @@ class OpenAIHandlerMixin:
from headroom.proxy.gateway_responses import build_view as build_responses_view
from headroom.proxy.gateway_responses import is_responses_body
from headroom.proxy.gateway_responses import mark_view as mark_responses_view
from headroom.proxy.gateway_turn import provider_for_body
from headroom.proxy.helpers import RequestBodyTooLarge, _read_request_json
def _gateway_invalid(e: CompressTurnError) -> JSONResponse:
@@ -11330,7 +11352,9 @@ class OpenAIHandlerMixin:
}
},
)
if mode in ("lossy_inline", "lossless_then_lossy"):
if mode == "lossless":
pipeline = self._lossless_pipeline()
elif mode in ("lossy_inline", "lossless_then_lossy"):
pipeline = self._lossy_inline_pipeline()
elif mode == "ccr":
# Markers + store writes wanted, so inherit the live router config
@@ -11347,6 +11371,8 @@ class OpenAIHandlerMixin:
"model_limit": context_limit,
**proxy_pipeline_kwargs(self.config),
}
if mode == "lossless":
pipeline_kwargs["cross_turn_dedup_recoverable"] = False
if compress_user_messages:
pipeline_kwargs["compress_user_messages"] = True
if target_ratio is not None:
@@ -11383,11 +11409,7 @@ class OpenAIHandlerMixin:
# trick SessionTrackerStore uses for its synthetic lineage
# keys). A plain "compress:" string prefix was spoofable.
_session_key = f"compress\x00{session_id}"
_tracker_provider = (
"anthropic"
if ("claude" in model_name.lower() or "anthropic" in model_name.lower())
else "openai"
)
_tracker_provider = provider_for_body(body, model_name)
comp_cache = self._get_compression_cache(_session_key)
session_tracker = self.session_tracker_store.get_or_create(
_session_key, _tracker_provider
+1 -1
View File
@@ -50,7 +50,7 @@ class TurnContext:
(or just sent); a hook's ``on_request`` may mutate them in place.
"""
provider: str # "anthropic" | "openai" | "google" | ...
provider: str # the wire shape, not the model's vendor: "anthropic" | "openai" | ...
model: str
messages: list[dict[str, Any]]
tools: Any = None # provider-native tools value (list, or None)
+1 -1
View File
@@ -9,7 +9,7 @@ export type HeadroomMode = "audit" | "optimize" | "simulate";
/** Pipeline mode accepted by POST /v1/compress. Unset = default marker-free
* pipeline. Any other value is rejected (400) by the proxy. */
export type CompressMode = "ccr" | "lossy_inline" | "lossless_then_lossy";
export type CompressMode = "ccr" | "lossy_inline" | "lossless_then_lossy" | "lossless";
/** Config accepted by the POST /v1/compress endpoint. camelCase keys are
* snake_cased on the wire (e.g. targetRatio -> target_ratio). Every field is
+167
View File
@@ -467,3 +467,170 @@ def test_turn_ids_are_unique_per_request(headroom_client) -> None:
"""I-KEYS: every request half mints a fresh id, registered or not."""
ids = {compress(headroom_client, _body(gateway={})).json()["turn_id"] for _ in range(5)}
assert len(ids) == 5
@pytest.mark.parametrize(
("label", "shape", "model", "provider", "route_provider"),
[
(
"anthropic_system",
{"messages": [{"role": "user", "content": "x"}], "system": SYSTEM_PROMPT},
"gemini/gemini-2.5-pro",
"anthropic",
"openai",
),
(
"anthropic_tools",
{"messages": [{"role": "user", "content": "x"}], "tools": anthropic_tools()},
"gemini/gemini-2.5-pro",
"anthropic",
"openai",
),
(
"anthropic_history_with_server_tool",
{
"messages": anthropic_tool_history(),
"tools": [{"type": "web_search_20250305", "name": "web_search"}],
},
"gpt-5.5",
"anthropic",
"openai",
),
(
"anthropic_first_turn_server_tool_only",
{
"messages": [{"role": "user", "content": "x"}],
"tools": [{"type": "web_search_20250305", "name": "web_search"}],
},
"gemini/gemini-2.5-pro",
"anthropic",
"openai",
),
(
"openai_tools_to_claude",
{"messages": [{"role": "user", "content": "x"}], "tools": openai_tools()},
"claude-opus-5-5",
"openai",
"anthropic",
),
(
"openai_tool_row_to_claude",
{"messages": [{"role": "tool", "tool_call_id": "c", "content": "x"}]},
"claude-opus-5-5",
"openai",
"anthropic",
),
(
"responses_input_to_claude",
{"input": [{"role": "user", "content": "x"}], "instructions": "be brief"},
"claude-opus-5-5",
"openai",
"anthropic",
),
(
"chat_system_role_to_claude",
{
"messages": [
{"role": "system", "content": "be brief"},
{"role": "user", "content": "x"},
]
},
"claude-opus-5-5",
"openai",
"anthropic",
),
(
"chat_image_to_claude",
{
"messages": [
{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "https://x.test/a.png"}}
],
}
]
},
"claude-opus-5-5",
"openai",
"anthropic",
),
(
"anthropic_thinking_to_gpt",
{
"messages": [
{"role": "user", "content": "x"},
{
"role": "assistant",
"content": [{"type": "thinking", "thinking": "t", "signature": "s"}],
},
{"role": "user", "content": "y"},
]
},
"gpt-5.5",
"anthropic",
"openai",
),
(
"anthropic_server_tool_result_to_gpt",
{
"messages": [
{"role": "user", "content": "x"},
{
"role": "assistant",
"content": [
{"type": "web_search_tool_result", "tool_use_id": "s", "content": []}
],
},
{"role": "user", "content": "y"},
],
"tools": [{"type": "web_search_20250305", "name": "web_search"}],
},
"gpt-5.5",
"anthropic",
"openai",
),
(
"plain_text_follows_model",
{"messages": [{"role": "user", "content": "x"}], "system": None},
"claude-opus-5-5",
"anthropic",
"anthropic",
),
],
)
def test_hooks_see_the_wire_shape_and_route_follows_the_model(
headroom_client, label, shape, model, provider, route_provider
) -> None:
"""A gateway can send any client's shape to any model."""
seen: list[str] = []
class Recorder:
name = f"provider_recorder_{label}"
stream_safe = True
def on_request(self, ctx: Any) -> None:
seen.append(ctx.provider)
register_turn_hook(Recorder())
data = compress(headroom_client, {**shape, "model": model, "gateway": {}}).json()
assert seen == [provider]
assert data["route"]["provider"] == route_provider
def test_chat_body_to_claude_stays_within_the_cache_breakpoint_budget(headroom_client) -> None:
"""The budget belongs to the upstream model, not to the body's wire shape."""
marked = [
{"role": "user", "content": [{"type": "text", "text": f"m{i}", "cache_control": {}}]}
for i in range(7)
]
body = _body(messages=marked, tools=openai_tools(), model="claude-opus-5-5", gateway={})
data = compress(headroom_client, body).json()
sent = [
block
for message in data["body"]["messages"]
for block in message["content"]
if isinstance(block, dict) and "cache_control" in block
]
assert len(sent) <= 4
assert "cache_breakpoint_budget" in data["transforms_applied"]
@@ -121,7 +121,7 @@ def test_openai_shape_still_counted_for_every_model_family(client, model):
assert body["tokens_saved"] > 0, f"{model} reported no savings: {body}"
@pytest.mark.parametrize("mode", [None, "ccr", "lossy_inline", "lossless_then_lossy"])
@pytest.mark.parametrize("mode", [None, "ccr", "lossy_inline", "lossless_then_lossy", "lossless"])
def test_every_mode_counts_anthropic_shape(client, mode):
"""mode="ccr" used to share the request pipeline, which pinned the OpenAI counter."""
payload: dict = {"messages": _anthropic_messages(), "model": "claude-sonnet-4-6"}
+87 -2
View File
@@ -514,6 +514,91 @@ class TestCompressEndpointLossyInlineMode:
with TestClient(app, base_url="http://127.0.0.1", client=("127.0.0.1", 12345)) as c:
yield c
def test_lossless_mode_folds_search_output_and_keeps_json_rows(self, client):
grep_output = "\n".join(
f"headroom/proxy/handlers/openai.py:{i}: value_{i} = compute(value_{i})"
for i in range(1, 400)
)
messages = [
{"role": "user", "content": "Find the values."},
{
"role": "assistant",
"content": None,
"tool_calls": [
{
"id": "call_grep",
"type": "function",
"function": {"name": "Grep", "arguments": "{}"},
}
],
},
{"role": "tool", "tool_call_id": "call_grep", "content": grep_output},
*self._big_tool_message(),
]
response = client.post(
"/v1/compress",
json={"messages": messages, "model": "gpt-4", "config": {"mode": "lossless"}},
)
assert response.status_code == 200
body = response.json()
assert not body.get("ccr_hashes")
folded = body["messages"][2]["content"]
assert len(folded) < len(grep_output)
assert all(f"value_{i} = compute(value_{i})" in folded for i in range(1, 400))
json_rows = json.loads(messages[4]["content"])
assert all(row["description"] in body["messages"][4]["content"] for row in json_rows)
@staticmethod
def _read_call(call_id, path, name="Read"):
return {
"role": "assistant",
"content": None,
"tool_calls": [
{
"id": call_id,
"type": "function",
"function": {"name": name, "arguments": json.dumps({"file_path": path})},
}
],
}
def test_lossless_mode_keeps_a_read_the_conversation_later_edited(self, client):
source = "\n".join(f"def function_{i}(): return {i}" for i in range(400))
messages = [
{"role": "user", "content": "Change a.py."},
self._read_call("call_read", "/x/a.py"),
{"role": "tool", "tool_call_id": "call_read", "content": source},
self._read_call("call_edit", "/x/a.py", name="Edit"),
{"role": "tool", "tool_call_id": "call_edit", "content": "Edited /x/a.py"},
{"role": "user", "content": "Now what?"},
]
response = client.post(
"/v1/compress",
json={"messages": messages, "model": "gpt-4", "config": {"mode": "lossless"}},
)
assert response.status_code == 200
body = response.json()
assert not body.get("ccr_hashes")
assert body["messages"][2]["content"] == source
def test_lossless_mode_emits_no_dedup_pointer(self, client, monkeypatch):
monkeypatch.setenv("HEADROOM_DEDUPE", "1")
output = "\n".join(f"2026-01-01 worker-{i} finished job {i}" for i in range(400))
messages = [{"role": "user", "content": "Run it twice."}]
for call_id in ("call_1", "call_2"):
messages.append(self._read_call(call_id, "/x/run.sh", name="Bash"))
messages.append({"role": "tool", "tool_call_id": call_id, "content": output})
messages.append({"role": "user", "content": "Same?"})
response = client.post(
"/v1/compress",
json={"messages": messages, "model": "gpt-4", "config": {"mode": "lossless"}},
)
assert response.status_code == 200
second = response.json()["messages"][4]["content"]
assert "same as msg" not in second
assert "finished job 399" in second
@pytest.mark.parametrize(
("repetitions", "strategy"), [(1, "json_minifier"), (20, "smart_crusher")]
)
@@ -597,7 +682,7 @@ class TestCompressEndpointModeValidation:
@pytest.mark.parametrize(
"bad_mode",
["lossless", "CCR", "ccr ", "no_ccr", "", 7, ["ccr"]],
["LOSSLESS", "CCR", "ccr ", "no_ccr", "", 7, ["ccr"]],
)
def test_unknown_mode_returns_400(self, client, bad_mode):
response = client.post(
@@ -613,7 +698,7 @@ class TestCompressEndpointModeValidation:
assert data["error"]["type"] == "invalid_request"
message = data["error"]["message"]
# The message must tell the caller what IS valid.
for valid in ("ccr", "lossy_inline", "lossless_then_lossy"):
for valid in ("ccr", "lossy_inline", "lossless_then_lossy", "lossless"):
assert valid in message
@pytest.mark.parametrize(
+1 -1
View File
@@ -283,7 +283,7 @@ Compression-only endpoint. Compresses messages without ever making a **completio
"model": "gpt-4o", // tokenizer + context limit
"token_budget": 8000, // optional: override the context limit
"config": { // optional
"mode": "lossy_inline", // ccr | lossy_inline | lossless_then_lossy
"mode": "lossy_inline", // ccr | lossy_inline | lossless_then_lossy | lossless
"frozen_message_count": 12, // pin an already-cached prefix
"session_id": "conv-8f1c", // session mode: Headroom keeps the replay state
"compress_user_messages": false,