Files
headroom/wiki/proxy.md
d6df127d57 feat(compress): accept Gemini and Bedrock Converse bodies on the gateway turn (#4679)
## Description

`/v1/compress` in gateway mode (a request with a `gateway` block) now
accepts two more native request bodies:
- Gemini `generateContent`
- Bedrock Converse

It compresses their text slots and returns `body` in the same native
shape. This lets a gateway like LiteLLM compress native Gemini and
Bedrock passthrough traffic, which it couldn't before.

`config.mode: "ccr"` is rejected for these shapes with a 400. Every
other mode is marker-free for them.

(Was stacked on #4617, which has merged; the branch now carries current
`main`.)

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

### How

This works the same way `gateway_responses` does for Codex: it builds a
view instead of translating the body.
- Every compressible text slot becomes one chat-shaped pseudo message.
- Compressed text is written back by its path in the body.
- Everything else stays byte-identical: `thoughtSignature`, thought
parts, `functionCall`, `inlineData`, `toolUse`, `cachePoint`,
`reasoningContent`, `tools`/`toolConfig`.
- Tool calls appear in the view, so tool-name rules still apply. `Read`
stays verbatim, for example.
- A JSON slot is only written back if it still parses to the same JSON
type.
- If the compressed list doesn't line up with the slots, the original
transcript is forwarded and a warning is logged.

### CCR on native bodies (review fix)

A native body can't carry `headroom_retrieve`. Its `tools`/`toolConfig`
are forwarded untouched, and `create_ccr_tool_definition` only builds
chat, Anthropic and Responses shapes. It also never gets a re-drive. So
a CCR marker in a native body points at content the model can never
fetch. Before this fix, `config.mode: "ccr"` on a Gemini or Converse
gateway turn did exactly that: log output came back as a marker with
`ccr_hashes: [<hash>]` and `obligations: []`.

- `handle_compress` now answers `ccr` on a native view with `400
invalid_request`. The message names the shape, explains why, and lists
the supported modes.
- The check runs where the mode is chosen, right after `attach_view` has
recorded the shape. That is before any compression, session work or CCR
store write.
- **The rejection does not depend on `gateway.can_redrive`.** Even a
gateway that can re-drive has no way to put the retrieval tool into a
native request, so there is no capability under which native `ccr`
works.
- **Without a `gateway` block nothing changes.** A Gemini body is
already a 400 (no `messages`). A Converse body goes through the plain
messages path, where its untyped blocks are opaque: 0 tokens saved, no
markers in any mode, `ccr` included (checked). So the rule is
consistent: reject `ccr` wherever Headroom reads a native shape.
- The bypass header still returns the body untouched with no
compression, in any mode.
- I checked every other mode on the same marker-producing input
(default, `lossless`, `lossy_inline`, `lossless_then_lossy`): no marker
and no hash in any of them. These modes build pipelines with
`ccr_inject_marker=False, ccr_enabled=False`. Stale-Read notices are
hashless in marker-free modes since #4678.
- **Second defect, same family:** with `HEADROOM_DEDUPE=1`, cross-turn
dedup wrote `[↑NL same as msg 4]` into a Converse tool result. That `4`
counts view messages (system blocks, tool calls, result slots), not the
caller's `messages`, so it named the wrong message. Native views now set
`cross_turn_dedup_recoverable=False`, as `lossless` mode already did.
- The `not self.native_shape` guards on CCR arming and re-drive in
`GatewayTurn.finish` stay as defence in depth.

### Caller contract

**Gemini**
- Send top-level `contents` as a list, with no `messages` or `input`.
- `systemInstruction`/`system_instruction`, `tools` and
`generationConfig` are optional and pass through.

**Converse**
- Send `messages` where every message has role `user` or `assistant`,
and every content block has no `type` key.
- At least one block must use a Converse key: `text`, `toolUse`,
`toolResult`, `cachePoint`, and so on.
- `system` is optional, as a list of blocks.

**Both shapes**
- `config.mode`: omit it (marker-free default), or use `lossless`,
`lossy_inline` or `lossless_then_lossy`. `ccr` is a 400.
- Only gateway turns get a view. Without a `gateway` block the endpoint
behaves exactly as before. A client-sent `_headroom_native_view` key is
dropped, and `BODY_CONTROL_KEYS` strips the real one before the body is
returned.
- `config.frozen_message_count` and `config.protect_recent` count native
messages: Gemini `contents` entries and Converse `messages`.
`session.frozen_message_count` comes back in native units. The
`/v1/usage` reply's `frozen_message_count` still counts view messages,
and the docs say so.
- No re-drive is ever armed, so `obligations` never contains `redrive`,
and request hooks run stream-safe-only. Tool compaction and output
steering are skipped.
- Relayed usage can be:
- Gemini `usageMetadata`, where `promptTokenCount` already includes
`cachedContentTokenCount`.
- Converse `usage`, where `inputTokens` excludes `cacheReadInputTokens`
and `cacheWriteInputTokens`.
  - The whole provider response that carries either.
- Every gateway answer, fail-open ones included, carries `wire_shape`:
`gemini`, `converse`, `openai_responses` or `messages`. A caller that
sent a native body and gets anything else, or no field at all, is
talking to an older Headroom and should forward its original body.
- Top-level `messages` in the answer is the Converse list for Converse
and the chat-shaped view for Gemini.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`): no new errors; the 8
reported are in `ccr/mcp_server.py` and `cache/prefix_tracker.py` and
are identical on `main`
- [x] New tests added for new functionality
- [x] Manual testing performed (local replay, below)

New in `tests/gateway/test_native_shapes.py`:
- `test_ccr_mode_is_rejected_for_native_bodies[gemini|converse]`: 400,
`invalid_request`, message asserted.
- `test_other_modes_forward_native_bodies_without_ccr_markers`: 4 modes
x 2 shapes, on log output large enough to produce a marker. Checks no
hash, no marker, the shape echoed, non-message keys byte-identical and
the native list mapped back.
- `test_ccr_still_works_on_a_chat_gateway_turn`: the same log output in
a chat body with `ccr` still carries a marker. This proves the input
above is marker-producing, and that chat `ccr` didn't regress.
- `test_native_bodies_get_no_dedup_pointer_into_view_messages`.
- `test_native_turns_are_never_asked_to_redrive` now runs in the default
mode, because `ccr` is a 400.

### Test Output

```text
# before the fix (new tests only, rest of the file passing)
FAILED tests/gateway/test_native_shapes.py::test_ccr_mode_is_rejected_for_native_bodies[gemini]
FAILED tests/gateway/test_native_shapes.py::test_ccr_mode_is_rejected_for_native_bodies[converse]
FAILED tests/gateway/test_native_shapes.py::test_native_bodies_get_no_dedup_pointer_into_view_messages
3 failed, 25 passed

# after
tests/gateway/test_native_shapes.py: 28 passed

# tests/gateway tests/test_gateway_turn_unit.py tests/test_compress_api.py
# tests/test_proxy_compress_endpoint.py tests/test_compress_session_mode.py
# tests/test_compress_session_shared.py tests/test_compress_passthrough.py tests/test_compress_failure.py
1 failed, 507 passed, 4 skipped
# the one failure, test_lossy_inline_emits_no_ccr_markers[1-json_minifier], fails identically on main fda75de5

uvx ruff@0.17.3 check / format --check (changed files): All checks passed! / 3 files already formatted
```

## Real Behavior Proof

These are local synthetic replays. They are not provider qualification.

- Environment: macOS, Python venv, branch head `530c20aa`. Headroom runs
in-process via `create_app(ProxyConfig(optimize=True,
cache_enabled=False, rate_limit_enabled=False,
cost_tracking_enabled=False, log_requests=False, image_optimize=False))`
behind `fastapi.testclient.TestClient` from a loopback client. The
"provider" is an `httpx.MockTransport` that records each body it
receives and answers with a Gemini `generateContent` or Bedrock Converse
response, including usage. `LITELLM_LOCAL_MODEL_COST_MAP=True`. No real
provider calls.
- Exact command / steps: `LITELLM_LOCAL_MODEL_COST_MAP=True python -I -c
"import sys; sys.path.insert(0,'<worktree>');
exec(open('native_replay.py').read())"`. For each shape the script does
three things:
1. POSTs the native body plus `gateway: {can_redrive, session_affinity,
can_relay_response: true}` to `/v1/compress`.
2. Sends the returned `body` through the mock upstream at the real
Gemini or Bedrock URL.
3. POSTs the provider's whole response to `/v1/compress/response` with
the `turn_id`.

It then sends both native bodies again with `config.mode: "ccr"`. The
tool output is the 399-line grep listing used in the tests. The `ccr`
requests use 3000 log lines, which produce a marker in `ccr` mode.
- Observed result:

```text
[gemini] request half: wire_shape=gemini obligations=['relay_usage'] tokens 7246->3863 ccr_hashes=[] marker_in_body=False
[gemini] all 399 grep lines still readable in the forwarded body=True
[gemini] non-message keys unchanged=True first two contents unchanged=True
[gemini] provider got 16125 bytes (raw 29694), control keys forwarded=[], host=generativelanguage.googleapis.com
[gemini] response half: 200 {'action': 'done', ..., 'billed_usage': {'input_tokens': 900, 'output_tokens': 12, 'cache_read_input_tokens': 512}}
[converse] request half: wire_shape=converse obligations=['relay_usage'] tokens 8823->4843 ccr_hashes=[] marker_in_body=False
[converse] all 399 grep lines still readable in the forwarded body=True
[converse] non-message keys unchanged=True first two messages unchanged=True
[converse] provider got 16184 bytes (raw 29761), control keys forwarded=[], host=bedrock-runtime.us-west-2.amazonaws.com
[converse] response half: 200 {'action': 'done', ..., 'billed_usage': {'input_tokens': 300, 'output_tokens': 12, 'cache_read_input_tokens': 600, 'cache_creation_input_tokens': 0}}
[gemini ccr] 400 {'error': {'type': 'invalid_request', 'message': "config.mode='ccr' is not supported for gemini bodies: Headroom cannot add its retrieval tool to a native Gemini or Bedrock Converse request, so the model could not resolve a CCR marker. Use lossless, lossy_inline or lossless_then_lossy, or omit config.mode for the default marker-free mode."}}
[converse ccr] 400 {'error': {'type': 'invalid_request', 'message': "config.mode='ccr' is not supported for converse bodies: ..."}}
provider calls: 2
```

Before the fix, the same `ccr` requests returned 200 with `ccr_hashes`
of length 1, a `hash=` marker in the forwarded body, and `obligations:
[]`.
- Not tested:
- live Gemini (AI Studio or Vertex) and live Bedrock Converse: the
provider was mocked, so provider acceptance of the rewritten bodies is
unverified;
  - real Kong `ai-proxy` or a real LiteLLM gateway in front of Headroom;
- streaming callers (`streamGenerateContent`, `ConverseStream`): the
request half is identical, but no streamed response was relayed;
  - Gemini `thoughtsTokenCount` in output accounting;
  - session mode with a Redis store for native bodies;
- multimodal parts (`inlineData`, `image`, `document`) beyond their
byte-identical pass-through in unit tests.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. Native shapes only activate for
gateway-mode `/v1/compress` callers that send a `gateway` block and a
native body.
- Minimum rollout channel: n/a
- Stable/default behavior changed: no for existing callers. Chat,
Anthropic and Responses bodies are unchanged. A Converse body without a
`gateway` block goes through the plain path as before. Before this PR, a
native body with a `gateway` block was misread as chat (Converse) or
rejected (Gemini).
- Kill switch / disable path: the gateway omits the `gateway` block, or
forwards its original body when `wire_shape` doesn't match.
- Unsafe override required: no
- Qualification impact: none for existing paths
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Known follow-ups, not in this PR:
- A `functionResponse.response` with any string field never compresses
its nested dict values. The JSON slot is only used when there are no
string fields. This is a missed saving, not a correctness bug.
- `handlers/gemini.py` (the proxy path) and `gateway_native.py` map
Gemini parts with different rules, and their part-key lists are kept in
step by hand. They should share one constant.
- `protect_recent` translation and stream-safe hook gating for native
turns have no dedicated test.
- Gemini `thoughtsTokenCount` is not added to output tokens when usage
is relayed, so output savings attribution undercounts for thinking
models.
- This is not specific to native shapes: the marker-free default folds
very repetitive logs to a single line (for example `[3000 lines omitted:
3000 INFO]`) on chat bodies too, with no retrieval path.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-15 01:52:50 +05:30

26 KiB

Proxy Server Documentation

The Headroom proxy server is a production-ready HTTP server that applies context optimization to all requests passing through it.

The proxy exposes compression-as-a-service via the POST /v1/compress endpoint — used by the TypeScript SDK, LiteLLM's headroom guardrail, and gateway sidecars. It is loopback-only by default; see the endpoint section below.

Starting the Proxy

# Basic usage
headroom proxy

# Deliberate public access, with the existing token protocol
HEADROOM_PROXY_TOKEN='replace-with-a-secret' headroom proxy --host 0.0.0.0 --port 8080
# Send `Authorization: Bearer replace-with-a-secret` or
# `X-Headroom-Proxy-Token: replace-with-a-secret` from the caller.

# With logging and budget
headroom proxy \
  --log-file /var/log/headroom.jsonl \
  --budget 100.0

Common agent CLI entrypoints

# Claude Code
ANTHROPIC_BASE_URL=http://localhost:8787 claude

# GitHub Copilot CLI
headroom wrap copilot -- --model claude-sonnet-4-20250514

# OpenAI-compatible clients
OPENAI_BASE_URL=http://localhost:8787/v1 your-app

headroom wrap copilot uses Copilot CLI's BYOK provider settings under the hood. In provider-type=auto, it chooses Headroom's Anthropic route for the default proxy backend and the OpenAI-compatible /v1 route for translated backends such as anyllm and LiteLLM.

Anonymous aggregate telemetry is off by default (opt-in). Opt in with HEADROOM_TELEMETRY=on or headroom proxy --telemetry. Downstream apps can set HEADROOM_SDK=headroom-app to override the anonymous telemetry sdk label; the default remains proxy.

Operational OTEL metrics are configured separately and are off by default. Install headroom-ai[proxy,otel] and set:

HEADROOM_OTEL_METRICS_ENABLED=1
HEADROOM_OTEL_METRICS_EXPORTER=otlp_http
HEADROOM_OTEL_METRICS_ENDPOINT=http://127.0.0.1:4318/v1/metrics
HEADROOM_OTEL_SERVICE_NAME=headroom-proxy

Use HEADROOM_OTEL_METRICS_EXPORTER=console for local smoke testing. HEADROOM_TELEMETRY controls the anonymous data-flywheel beacon only; it does not disable or enable OTEL export.

Langfuse can be enabled alongside this OTEL path for trace ingestion. Langfuse does not ingest OTEL metrics, so Headroom keeps metrics and Langfuse traces as complementary signals:

HEADROOM_LANGFUSE_ENABLED=1
LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_BASE_URL=https://cloud.langfuse.com

When configured, Headroom emits OTLP traces for the shared compression pipeline to Langfuse while continuing to expose metrics through /metrics and OTEL metric exporters.

Command Line Options

Core Options

Option Default Description
--host 127.0.0.1 Host to bind to
--port 8787 Port to bind to
--mode cache Run mode: token (maximize compression) or cache (freeze prior turns)
--no-optimize false Disable optimization (passthrough mode)
--no-cache false Disable semantic caching
--no-rate-limit false Disable rate limiting
--log-file None Path to JSONL log file
--budget None Daily budget limit in USD
--code-aware / --no-code-aware disabled Enable or disable AST-based code compression. Requires headroom-ai[code] (env: HEADROOM_CODE_AWARE_ENABLED=1 to enable)
--anthropic-api-url https://api.anthropic.com Custom Anthropic API URL endpoint
--openai-api-url https://api.openai.com Custom OpenAI API URL endpoint
--anthropic-extra-headers unset JSON object of extra headers merged into (and overriding) forwarded Anthropic requests, e.g. '{"Api-Key": "..."}'
--openai-extra-headers unset JSON object of extra headers merged into (and overriding) forwarded OpenAI requests

Run Modes

Headroom proxy has two explicit run modes:

  • token mode: prioritize token reduction. Prior history may be rewritten when that improves compression.
  • cache mode: prioritize provider prefix cache stability. Prior turns are frozen; only the newest turn is mutable.

Set via CLI or env:

headroom proxy --mode token
HEADROOM_MODE=cache headroom proxy

When to pick each:

  • token: best for maximizing immediate compression savings.
  • cache: best for long conversations where preserving prior-turn bytes improves prefix-cache reuse.

Legacy values (token_headroom, cost_savings) are still accepted as aliases.

Context Management Options

Context management in the proxy is handled automatically by the compression pipeline. CCR (Compress-Cache-Retrieve) ensures that when content is compressed or messages are dropped, the original data remains accessible for the LLM to retrieve on demand. See CCR documentation for details.

Key CCR-related proxy flags:

Option Description
--no-ccr Disable CCR entirely — no retrieval markers in compressed output and no injected headroom_retrieve tool (lossy, no recovery path)
--no-ccr-proactive-expansion Disable proactive context expansion before the LLM asks

ML Compression — RETIRED --llmlingua flag

The --llmlingua / --llmlingua-device / --llmlingua-rate flags and the headroom-ai[llmlingua] extra were retired and replaced by Kompress (ModernBERT). For the current opt-in path, install headroom-ai[ml] and see transforms.md and ARCHITECTURE.md.

API Endpoints

Liveness

curl http://localhost:8787/livez

Response:

{
  "service": "headroom-proxy",
  "status": "healthy",
  "alive": true,
  "version": "0.5.21",
  "timestamp": "2026-04-10T16:36:25Z",
  "uptime_seconds": 12.483
}

Readiness

curl http://localhost:8787/readyz

Response:

{
  "service": "headroom-proxy",
  "status": "healthy",
  "ready": true,
  "version": "0.5.21",
  "timestamp": "2026-04-10T16:36:25Z",
  "uptime_seconds": 12.483,
  "checks": {
    "startup": {"enabled": true, "ready": true, "status": "healthy"},
    "http_client": {"enabled": true, "ready": true, "status": "healthy"},
    "cache": {"enabled": true, "ready": true, "status": "healthy"},
    "rate_limiter": {"enabled": true, "ready": true, "status": "healthy"},
    "memory": {"enabled": false, "ready": true, "status": "disabled"}
  }
}

/readyz returns HTTP 503 when Headroom has not completed startup or a required enabled subsystem is unavailable. This is the endpoint used by the container health checks.

Aggregate Health

curl http://localhost:8787/health

Response:

{
  "status": "healthy",
  "ready": true,
  "version": "0.5.21",
  "config": {
    "backend": "anthropic",
    "optimize": true,
    "cache": true,
    "rate_limit": true
  },
  "checks": {
    "startup": {"enabled": true, "ready": true, "status": "healthy"},
    "http_client": {"enabled": true, "ready": true, "status": "healthy"}
  }
}

Detailed Statistics

curl http://localhost:8787/stats

/stats remains the live/session-oriented endpoint and now also includes a persistent_savings block with durable proxy compression lifetime totals plus a small recent preview. The existing savings_history field is still present and remains session-scoped for backward compatibility.

For providers that return cache-write TTL bucket usage, /stats also includes observed TTL breakdowns under prefix_cache:

  • observed_ttl_buckets.5m.tokens
  • observed_ttl_buckets.1h.tokens
  • observed_ttl_mix

These are provider-reported observations, not configured TTL and not remaining expiration time.

Historical Savings

curl http://localhost:8787/stats-history

/stats-history exposes durable proxy compression history for dashboards and other Headroom frontends. It returns:

  • lifetime proxy compression totals
  • compact checkpoint history by default, with history_mode=full available for export/debug flows
  • derived hourly, daily, weekly, and monthly rollups for charts
  • a history_summary block describing stored versus returned checkpoint counts
  • UTC timestamps throughout

By default the proxy stores this history at ${HEADROOM_WORKSPACE_DIR}/proxy_savings.json (i.e. ~/.headroom/proxy_savings.json when HEADROOM_WORKSPACE_DIR is unset). Set HEADROOM_SAVINGS_PATH to override the location directly, or set HEADROOM_WORKSPACE_DIR to relocate the full state root. See the Filesystem Contract.

/dashboard uses this endpoint directly for its historical view, including the daily/weekly/monthly rollups and built-in JSON / CSV export buttons.

curl "http://localhost:8787/stats-history?format=csv&series=weekly"
curl "http://localhost:8787/stats-history?format=csv&series=monthly"
curl "http://localhost:8787/stats-history?history_mode=full"

Prometheus Metrics

curl http://localhost:8787/metrics

/metrics remains the built-in Prometheus-formatted operational view. The proxy now also emits the same operational events through the OTEL facade when OTEL metrics are configured.

LLM APIs

The proxy supports both Anthropic and OpenAI API formats:

# Anthropic format
POST /v1/messages

# OpenAI format
POST /v1/chat/completions

POST /v1/compress

Compression-only endpoint. Compresses messages without ever making a completion request to an LLM provider — no generation, no provider API key. Used by the TypeScript SDK, LiteLLM's headroom guardrail, and gateway sidecars.

It does run local ML models. Compression is ML-backed: Kompress is a ModernBERT encoder scoring tokens for retention (classification, not generation) and Magika classifies content types, both in-process by default. If HEADROOM_KOMPRESS_ENDPOINT is set, Kompress inference is offloaded over HTTP to that model server — real egress from the sidecar. Only inference goes remote; the CCR store and markers stay proxy-local. HEADROOM_DISABLE_KOMPRESS=1 gives structural compression only.

Loopback-only by default. Non-loopback callers get 404 (not 403 — the route stays invisible to scanners). Set HEADROOM_COMPRESS_ALLOW_REMOTE=1 to allow remote callers.

No format conversion. messages may be OpenAI-shaped (role: "tool" + tool_call_id) or Anthropic-shaped (tool_use / tool_result content blocks); the same shape comes back. In gateway mode it may also be a native Gemini generateContent body (contents) or a Bedrock Converse body (untyped text/toolUse/toolResult/cachePoint blocks); their text and tool-result slots are compressed in place and body keeps the native shape, with tools/toolConfig, steering and re-drive left out. config.mode: "ccr" is a 400 for these bodies (no retrieval tool can be added). model selects the tokenizer and context limit — send the real name, including gateway-prefixed forms like bedrock/anthropic.claude-3-5-sonnet.

system and tools are ignored outside gateway mode. Anthropic sends both out of band. Without a gateway block this endpoint accepts them without complaint (200, no warning) and returns neither, so neither is compressed — keep carrying them yourself. That means the Anthropic system prompt is not compressed here, and tool-schema compaction / tool-search deferral are not reachable this way. Gateway mode (below) passes tools through and compacts them; in OSS, native tool-search deferral is still proxy-only — the headroom-tool-search extension adds it to gateway mode as a turn hook.

Request:

{
  "messages": [...],          // either wire format
  "model": "gpt-4o",          // tokenizer + context limit
  "token_budget": 8000,       // optional: override the context limit
  "config": {                 // optional
    "mode": "lossy_inline",       // ccr | lossy_inline | lossless_then_lossy | lossless
    "frozen_message_count": 12,   // pin an already-cached prefix
    "session_id": "conv-8f1c",    // session mode: Headroom keeps the replay state
    "compress_user_messages": false,
    "target_ratio": 0.5,
    "protect_recent": 2,
    "protect_analysis_context": true
  }
}

Response:

{
  "messages": [...],            // compressed messages
  "tokens_before": 15000,
  "tokens_after": 3500,
  "tokens_saved": 11500,
  "compression_ratio": 0.23,    // tokens_after / tokens_before — LOWER is better
  "transforms_applied": ["router:smart_crusher:0.35"],
  "transforms_summary": {"router:smart_crusher:0.35": 1},
  "ccr_hashes": [],             // non-empty only with mode="ccr"
  "session": {"id": "conv-8f1c", "frozen_message_count": 12, "cached_prefix_replayed": false}  // session mode only
}

Headers:

  • x-headroom-bypass: true — skip compression, return messages as-is with zeroed metrics

Error responses: 400 (missing/invalid fields, bad config.mode, config.frozen_message_count or config.session_id, session_id with compress_user_messages, malformed gateway block), 401 (bad HEADROOM_PROXY_TOKEN), 404 (non-loopback without HEADROOM_COMPRESS_ALLOW_REMOTE=1), 503 (compression failed; in session mode also compression_timeout — retry the turn)

Fail-open: without a session, on timeout you get 200 with the original messages plus compression_skipped: true and skip_reason: "compression_timeout". In session mode a timeout or busy session lock is a 503 instead, because handing back originals would desync the replay state.

Multi-turn callers — don't lose the prefix cache. Without config.session_id this endpoint is stateless: unlike the proxy's own request path (which runs a CacheAligner and tracks provider cache hits across turns), it has no idea what the provider already cached. Either let Headroom keep the state (session mode, below) or keep it yourself with frozen_message_count.

The provider caches the bytes you forwarded, which compression already changed — so your originals and the cached prefix are no longer the same thing, and it is the forwarded version you must keep reproducing. Compression also varies with position: an older tool result can fall outside the recent-read protection window as the conversation grows and be compressed harder than last turn, so re-compression is not guaranteed to reproduce earlier output either. Two rules:

  1. Pass config.frozen_message_count = the number of leading messages already cached upstream.
  2. Send back the messages you previously forwarded, not the pristine originals. frozen_message_count returns leading messages exactly as passed in, so feeding it originals hands the provider different bytes than last turn and busts the cache anyway.
forwarded = []


def next_turn(new_messages):
    r = requests.post(
        f"{proxy}/v1/compress",
        json={
            "messages": forwarded + new_messages,
            "model": "claude-sonnet-4-6",
            "config": {"frozen_message_count": len(forwarded)},
        },
    ).json()
    forwarded[:] = r["messages"]  # next turn's frozen prefix
    return forwarded

Note protect_recent is not a substitute — it guards the newest messages, while frozen_message_count guards the oldest, which is the cached end.

Session mode. Pass config.session_id (non-empty, at most 256 chars) and Headroom keeps the conversation's replay state itself — the same per-session compression cache and prefix tracker the proxy uses — so a gateway that owns routing can resend the raw conversation every turn and get a byte-identical prefix back. Everything previously returned for the session comes back unchanged; only the new tail is compressed; an explicit frozen_message_count still wins when larger. Forward the returned messages verbatim. compress_user_messages is refused (400). State is in-process by default (replay cache: HEADROOM_COMPRESSION_CACHE_TTL_SECONDS, default 3900; tracker state: 10 idle minutes), so multi-process deployments must pin a session to one process. With HEADROOM_COMPRESS_SESSION_STORE=redis://... (needs the redis extra, included in every Docker image) each session's tracker is also kept in Redis and any replica can serve its next turn and its /v1/usage; the CCR store follows to Redis unless HEADROOM_CCR_BACKEND names another. The per-session lock stays per process: two concurrent turns of one session on two replicas both run and the first write wins (compare-and-set). Gateway-mode pending turns stay in-process, so re-drive still needs real affinity. An unreachable Redis never fails a turn: the replica serves from its own memory (one WARNING when calls start failing, INFO when they recover, Redis skipped 5 s after a connection error). A replica with no local state then recompresses the history, slower, and can return a different prefix (a provider prompt-cache miss) with cached_prefix_replayed: false. Run Redis with high availability. HEADROOM_COMPRESS_SESSION_FROM_HEADER=1 lets the x-headroom-session-id header stand in for config.session_id (off by default).

The response's session object: id is the session id the turn ran under (config.session_id, or the header above). frozen_message_count is how many leading messages the pipeline was told not to recompress: the leading run Headroom can replay from the session's compression cache (plain messages always count, a tool result only once its content is cached, the last message never), or config.frozen_message_count when larger; it comes from Headroom's state, not the provider, and can be non-zero on a first turn. cached_prefix_replayed is true when the pipeline's output for some leading messages differed from what was returned last turn and Headroom overwrote them with last turn's bytes; false when the serving process holds no previous turn for the session (first turn, expired state, a replica that never served or loaded it), when the pipeline already matched (the usual case), or when the replay was skipped as unsafe (messages do not line up with last turn's, or the replay would be larger), so false does not mean the prefix matches last turn's. It is not a provider prompt-cache hit/miss signal; read that from the provider's usage (cache_read_input_tokens, cached_tokens).

Optionally relay the provider's usage for attribution with POST /v1/usage {"session_id": "...", "usage": {...}}. Anthropic cache_read_input_tokens / cache_creation_input_tokens and OpenAI prompt_tokens_details.cached_tokens / flat cached_tokens are accepted; a block with none of them is a 400. Replies {"session_id", "frozen_message_count", "applied"} (applied: false, reason: "no_cache_signal" when the only present field is 0), 404 unknown_session, or 503 session_busy (retry). Telemetry only — it never raises the frozen count.

Gateway mode (two-half turn contract). Add a top-level gateway object ({} is enough) and one model turn becomes two calls, so a gateway that never lets Headroom see the provider response can still run transforms whose reload step needs it — the proxy's "no shrink without reload" rule at the API boundary. gateway fields: can_redrive (default false: the gateway can call the provider again with a request Headroom hands it; send false when streaming), can_relay_response (default false: the gateway will post status/usage after each turn), session_affinity (default true; false disables re-driving because pending turns are in-process, with or without HEADROOM_COMPRESS_SESSION_STORE), plugin_version (diagnostics). Every other top-level field (system, tools, temperature, …) passes through; tools is compacted deterministically (tool_schema_compaction; tool_desc_compaction with HEADROOM_TOOL_DESC_MAX_CHARS); extension turn hooks run, stream-safe-only unless can_redrive and session_affinity both hold. The response adds body (the complete provider request — forward it as-is; never contains config, gateway, token_budget), turn_id, route ({model, provider, service_tier, reason}, advisory; a routing extension's model is also written into body.model), obligations (redrive and/or relay_usage) and a gateway echo. Fail-open answers carry the same keys with originals and obligations: []. Savings follow the proxy path: when compaction shortened tools, tokens_before/tokens_after also count the tool definitions, and the answer adds tool_tokens_saved (tool-search deferral plus tools a turn hook removed, both net; tokens_saved + tool_tokens_saved is the /stats headline) and tool_tokens_deferred (its deferral share, booked estimated); both are absent on fail-open answers. A turn is registered only when obligations is non-empty; with relay_usage the /stats record waits for the response half. Knobs: HEADROOM_GATEWAY_TURN_TTL_SECONDS (120), HEADROOM_GATEWAY_MAX_PENDING_TURNS (10000), HEADROOM_GATEWAY_MAX_REDRIVES (8). With config.mode: "ccr" and re-drive allowed, headroom_retrieve is injected into body.tools (ccr_tool_injected) and the response half answers the model's retrieval calls itself.

POST /v1/compress/response (same exposure rules as /v1/compress) is the response half: {"turn_id", "status": 200, "latency_ms", "usage": {...}, "response": {...}}. usage accepts Anthropic, OpenAI chat (prompt_tokens_details.cached_tokens), OpenAI Responses (input_tokens_details.cached_tokens) and Kong's flat cached_tokens shapes, or the whole provider body (a nested usage key is descended into once); billed counters are summed across re-drive rounds. response is required when the turn carries redrive and the provider call succeeded; a failed call closes the turn with its status alone. Answers: {"action": "done", "turn_id", "response": <replacement, the latest provider response after a re-drive, or null — forward what you hold>, "frozen_message_count", "usage_applied", "rounds", "billed_usage"} or {"action": "redrive", "turn_id", "request": <full provider body to send>, "round"} — post the provider's JSON back under the same turn_id; billed_usage sums every counter (cache reads and writes included) across rounds in Anthropic keys, and a re-driven turn's response is never null and reports that total as its usage; past HEADROOM_GATEWAY_MAX_REDRIVES the turn ends with done and the latest provider response. Errors: 400 invalid_request / missing_response, 404 unknown_turn (unregistered, finished, or expired), 409 turn_busy.

Kong plugin. kong-plugin-headroom (its own repo; luarocks install kong-plugin-headroom) implements both halves for Kong Gateway 3.9 (session id from a header, usage relay from the log phase, re-drive loop in access). The contract itself is installed in Headroom through the compress-turn seam (headroom.proxy.compress_turn, HEADROOM_GATEWAY_CONTRACT), the same seam a third-party contract would use.

Using with Claude Code

# Start proxy
headroom proxy --port 8787

# In another terminal
ANTHROPIC_BASE_URL=http://localhost:8787 claude

Using with Cursor

  1. Start the proxy: headroom proxy
  2. In Cursor settings, set the base URL to http://localhost:8787

Using with OpenAI SDK

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8787/v1",
    api_key="your-api-key",  # Still needed for upstream
)

Features

ML Compression (Opt-In, Kompress)

The earlier LLMLingua-2 integration documented in this section (--llmlingua, --llmlingua-device, --llmlingua-rate, headroom-ai[llmlingua], LLMLinguaCompressor) was retired and replaced by Kompress (ModernBERT). Install with pip install 'headroom-ai[ml]'. See transforms.md and ARCHITECTURE.md for current configuration.

Semantic Caching

The proxy caches responses for repeated queries:

  • LRU eviction with configurable max entries
  • TTL-based expiration
  • Cache key based on message content hash

Rate Limiting

Token bucket rate limiting protects against runaway costs:

  • Configurable requests per minute
  • Configurable tokens per minute
  • Per-API-key tracking

Cost Tracking

Track spending and enforce budgets:

  • Real-time cost estimation
  • Budget periods: hourly, daily, monthly
  • Automatic request rejection when over budget

Prometheus Metrics

Export metrics for monitoring:

headroom_requests_total
headroom_tokens_saved_total
headroom_cost_usd_total
headroom_latency_ms_sum

Configuration via Environment

# For deliberate public access, configure the existing token and send it from callers.
export HEADROOM_HOST=0.0.0.0
export HEADROOM_PROXY_TOKEN='replace-with-a-secret'
export HEADROOM_PORT=8787
export HEADROOM_BUDGET=100.0

# Route OpenAI passthrough requests to a custom endpoint
export OPENAI_TARGET_API_URL=https://custom.openai.endpoint.com

# Route Anthropic passthrough requests to a custom endpoint
export ANTHROPIC_TARGET_API_URL=https://litellm.company.internal

headroom proxy

Running in Production

For production deployments:

# Use a process manager
pip install gunicorn

# Run with gunicorn — server.py has no module-level `app`; FastAPI is built
# by the create_app() factory, so gunicorn needs --factory
gunicorn headroom.proxy.server:create_app \
  --workers 4 \
  --bind 0.0.0.0:8787 \
  --worker-class uvicorn.workers.UvicornWorker \
  --factory
# Callers must send `Authorization: Bearer replace-with-a-secret` or
# `X-Headroom-Proxy-Token: replace-with-a-secret`.

Or with Docker:

FROM python:3.11-slim
RUN apt-get update && apt-get install -y --no-install-recommends build-essential \
    && pip install "headroom-ai[proxy]" \
    && apt-get purge -y build-essential && apt-get autoremove -y \
    && rm -rf /var/lib/apt/lists/*
EXPOSE 8787
CMD ["headroom", "proxy", "--host", "0.0.0.0", "--port", "8787"]

Run this image with an explicit token and a deliberate publication choice:

docker run --rm -p 127.0.0.1:8787:8787 \
  -e HEADROOM_PROXY_TOKEN='replace-with-a-secret' \
  headroom-proxy

Callers must send Authorization: Bearer replace-with-a-secret or X-Headroom-Proxy-Token: replace-with-a-secret. For network access, replace the host-side 127.0.0.1 with an intentional public address and keep the token configured.

Note: build-essential is required at install time because headroom-ai includes hnswlib, a C++ extension that must be compiled from source. It is removed after installation to keep the image slim.