mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description
`/v1/compress` in gateway mode (a request with a `gateway` block) now
accepts two more native request bodies:
- Gemini `generateContent`
- Bedrock Converse
It compresses their text slots and returns `body` in the same native
shape. This lets a gateway like LiteLLM compress native Gemini and
Bedrock passthrough traffic, which it couldn't before.
`config.mode: "ccr"` is rejected for these shapes with a 400. Every
other mode is marker-free for them.
(Was stacked on #4617, which has merged; the branch now carries current
`main`.)
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
### How
This works the same way `gateway_responses` does for Codex: it builds a
view instead of translating the body.
- Every compressible text slot becomes one chat-shaped pseudo message.
- Compressed text is written back by its path in the body.
- Everything else stays byte-identical: `thoughtSignature`, thought
parts, `functionCall`, `inlineData`, `toolUse`, `cachePoint`,
`reasoningContent`, `tools`/`toolConfig`.
- Tool calls appear in the view, so tool-name rules still apply. `Read`
stays verbatim, for example.
- A JSON slot is only written back if it still parses to the same JSON
type.
- If the compressed list doesn't line up with the slots, the original
transcript is forwarded and a warning is logged.
### CCR on native bodies (review fix)
A native body can't carry `headroom_retrieve`. Its `tools`/`toolConfig`
are forwarded untouched, and `create_ccr_tool_definition` only builds
chat, Anthropic and Responses shapes. It also never gets a re-drive. So
a CCR marker in a native body points at content the model can never
fetch. Before this fix, `config.mode: "ccr"` on a Gemini or Converse
gateway turn did exactly that: log output came back as a marker with
`ccr_hashes: [<hash>]` and `obligations: []`.
- `handle_compress` now answers `ccr` on a native view with `400
invalid_request`. The message names the shape, explains why, and lists
the supported modes.
- The check runs where the mode is chosen, right after `attach_view` has
recorded the shape. That is before any compression, session work or CCR
store write.
- **The rejection does not depend on `gateway.can_redrive`.** Even a
gateway that can re-drive has no way to put the retrieval tool into a
native request, so there is no capability under which native `ccr`
works.
- **Without a `gateway` block nothing changes.** A Gemini body is
already a 400 (no `messages`). A Converse body goes through the plain
messages path, where its untyped blocks are opaque: 0 tokens saved, no
markers in any mode, `ccr` included (checked). So the rule is
consistent: reject `ccr` wherever Headroom reads a native shape.
- The bypass header still returns the body untouched with no
compression, in any mode.
- I checked every other mode on the same marker-producing input
(default, `lossless`, `lossy_inline`, `lossless_then_lossy`): no marker
and no hash in any of them. These modes build pipelines with
`ccr_inject_marker=False, ccr_enabled=False`. Stale-Read notices are
hashless in marker-free modes since #4678.
- **Second defect, same family:** with `HEADROOM_DEDUPE=1`, cross-turn
dedup wrote `[↑NL same as msg 4]` into a Converse tool result. That `4`
counts view messages (system blocks, tool calls, result slots), not the
caller's `messages`, so it named the wrong message. Native views now set
`cross_turn_dedup_recoverable=False`, as `lossless` mode already did.
- The `not self.native_shape` guards on CCR arming and re-drive in
`GatewayTurn.finish` stay as defence in depth.
### Caller contract
**Gemini**
- Send top-level `contents` as a list, with no `messages` or `input`.
- `systemInstruction`/`system_instruction`, `tools` and
`generationConfig` are optional and pass through.
**Converse**
- Send `messages` where every message has role `user` or `assistant`,
and every content block has no `type` key.
- At least one block must use a Converse key: `text`, `toolUse`,
`toolResult`, `cachePoint`, and so on.
- `system` is optional, as a list of blocks.
**Both shapes**
- `config.mode`: omit it (marker-free default), or use `lossless`,
`lossy_inline` or `lossless_then_lossy`. `ccr` is a 400.
- Only gateway turns get a view. Without a `gateway` block the endpoint
behaves exactly as before. A client-sent `_headroom_native_view` key is
dropped, and `BODY_CONTROL_KEYS` strips the real one before the body is
returned.
- `config.frozen_message_count` and `config.protect_recent` count native
messages: Gemini `contents` entries and Converse `messages`.
`session.frozen_message_count` comes back in native units. The
`/v1/usage` reply's `frozen_message_count` still counts view messages,
and the docs say so.
- No re-drive is ever armed, so `obligations` never contains `redrive`,
and request hooks run stream-safe-only. Tool compaction and output
steering are skipped.
- Relayed usage can be:
- Gemini `usageMetadata`, where `promptTokenCount` already includes
`cachedContentTokenCount`.
- Converse `usage`, where `inputTokens` excludes `cacheReadInputTokens`
and `cacheWriteInputTokens`.
- The whole provider response that carries either.
- Every gateway answer, fail-open ones included, carries `wire_shape`:
`gemini`, `converse`, `openai_responses` or `messages`. A caller that
sent a native body and gets anything else, or no field at all, is
talking to an older Headroom and should forward its original body.
- Top-level `messages` in the answer is the Converse list for Converse
and the chat-shaped view for Gemini.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`): no new errors; the 8
reported are in `ccr/mcp_server.py` and `cache/prefix_tracker.py` and
are identical on `main`
- [x] New tests added for new functionality
- [x] Manual testing performed (local replay, below)
New in `tests/gateway/test_native_shapes.py`:
- `test_ccr_mode_is_rejected_for_native_bodies[gemini|converse]`: 400,
`invalid_request`, message asserted.
- `test_other_modes_forward_native_bodies_without_ccr_markers`: 4 modes
x 2 shapes, on log output large enough to produce a marker. Checks no
hash, no marker, the shape echoed, non-message keys byte-identical and
the native list mapped back.
- `test_ccr_still_works_on_a_chat_gateway_turn`: the same log output in
a chat body with `ccr` still carries a marker. This proves the input
above is marker-producing, and that chat `ccr` didn't regress.
- `test_native_bodies_get_no_dedup_pointer_into_view_messages`.
- `test_native_turns_are_never_asked_to_redrive` now runs in the default
mode, because `ccr` is a 400.
### Test Output
```text
# before the fix (new tests only, rest of the file passing)
FAILED tests/gateway/test_native_shapes.py::test_ccr_mode_is_rejected_for_native_bodies[gemini]
FAILED tests/gateway/test_native_shapes.py::test_ccr_mode_is_rejected_for_native_bodies[converse]
FAILED tests/gateway/test_native_shapes.py::test_native_bodies_get_no_dedup_pointer_into_view_messages
3 failed, 25 passed
# after
tests/gateway/test_native_shapes.py: 28 passed
# tests/gateway tests/test_gateway_turn_unit.py tests/test_compress_api.py
# tests/test_proxy_compress_endpoint.py tests/test_compress_session_mode.py
# tests/test_compress_session_shared.py tests/test_compress_passthrough.py tests/test_compress_failure.py
1 failed, 507 passed, 4 skipped
# the one failure, test_lossy_inline_emits_no_ccr_markers[1-json_minifier], fails identically on main fda75de5
uvx ruff@0.17.3 check / format --check (changed files): All checks passed! / 3 files already formatted
```
## Real Behavior Proof
These are local synthetic replays. They are not provider qualification.
- Environment: macOS, Python venv, branch head `530c20aa`. Headroom runs
in-process via `create_app(ProxyConfig(optimize=True,
cache_enabled=False, rate_limit_enabled=False,
cost_tracking_enabled=False, log_requests=False, image_optimize=False))`
behind `fastapi.testclient.TestClient` from a loopback client. The
"provider" is an `httpx.MockTransport` that records each body it
receives and answers with a Gemini `generateContent` or Bedrock Converse
response, including usage. `LITELLM_LOCAL_MODEL_COST_MAP=True`. No real
provider calls.
- Exact command / steps: `LITELLM_LOCAL_MODEL_COST_MAP=True python -I -c
"import sys; sys.path.insert(0,'<worktree>');
exec(open('native_replay.py').read())"`. For each shape the script does
three things:
1. POSTs the native body plus `gateway: {can_redrive, session_affinity,
can_relay_response: true}` to `/v1/compress`.
2. Sends the returned `body` through the mock upstream at the real
Gemini or Bedrock URL.
3. POSTs the provider's whole response to `/v1/compress/response` with
the `turn_id`.
It then sends both native bodies again with `config.mode: "ccr"`. The
tool output is the 399-line grep listing used in the tests. The `ccr`
requests use 3000 log lines, which produce a marker in `ccr` mode.
- Observed result:
```text
[gemini] request half: wire_shape=gemini obligations=['relay_usage'] tokens 7246->3863 ccr_hashes=[] marker_in_body=False
[gemini] all 399 grep lines still readable in the forwarded body=True
[gemini] non-message keys unchanged=True first two contents unchanged=True
[gemini] provider got 16125 bytes (raw 29694), control keys forwarded=[], host=generativelanguage.googleapis.com
[gemini] response half: 200 {'action': 'done', ..., 'billed_usage': {'input_tokens': 900, 'output_tokens': 12, 'cache_read_input_tokens': 512}}
[converse] request half: wire_shape=converse obligations=['relay_usage'] tokens 8823->4843 ccr_hashes=[] marker_in_body=False
[converse] all 399 grep lines still readable in the forwarded body=True
[converse] non-message keys unchanged=True first two messages unchanged=True
[converse] provider got 16184 bytes (raw 29761), control keys forwarded=[], host=bedrock-runtime.us-west-2.amazonaws.com
[converse] response half: 200 {'action': 'done', ..., 'billed_usage': {'input_tokens': 300, 'output_tokens': 12, 'cache_read_input_tokens': 600, 'cache_creation_input_tokens': 0}}
[gemini ccr] 400 {'error': {'type': 'invalid_request', 'message': "config.mode='ccr' is not supported for gemini bodies: Headroom cannot add its retrieval tool to a native Gemini or Bedrock Converse request, so the model could not resolve a CCR marker. Use lossless, lossy_inline or lossless_then_lossy, or omit config.mode for the default marker-free mode."}}
[converse ccr] 400 {'error': {'type': 'invalid_request', 'message': "config.mode='ccr' is not supported for converse bodies: ..."}}
provider calls: 2
```
Before the fix, the same `ccr` requests returned 200 with `ccr_hashes`
of length 1, a `hash=` marker in the forwarded body, and `obligations:
[]`.
- Not tested:
- live Gemini (AI Studio or Vertex) and live Bedrock Converse: the
provider was mocked, so provider acceptance of the rewritten bodies is
unverified;
- real Kong `ai-proxy` or a real LiteLLM gateway in front of Headroom;
- streaming callers (`streamGenerateContent`, `ConverseStream`): the
request half is identical, but no streamed response was relayed;
- Gemini `thoughtsTokenCount` in output accounting;
- session mode with a Redis store for native bodies;
- multimodal parts (`inlineData`, `image`, `document`) beyond their
byte-identical pass-through in unit tests.
## Runtime Rollout Safety
- Rollout-managed feature(s): none. Native shapes only activate for
gateway-mode `/v1/compress` callers that send a `gateway` block and a
native body.
- Minimum rollout channel: n/a
- Stable/default behavior changed: no for existing callers. Chat,
Anthropic and Responses bodies are unchanged. A Converse body without a
`gateway` block goes through the plain path as before. Before this PR, a
native body with a `gateway` block was misread as chat (Converse) or
rejected (Gemini).
- Kill switch / disable path: the gateway omits the `gateway` block, or
forwards its original body when `wire_shape` doesn't match.
- Unsafe override required: no
- Qualification impact: none for existing paths
- Rollback path: revert this PR
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
Known follow-ups, not in this PR:
- A `functionResponse.response` with any string field never compresses
its nested dict values. The JSON slot is only used when there are no
string fields. This is a missed saving, not a correctness bug.
- `handlers/gemini.py` (the proxy path) and `gateway_native.py` map
Gemini parts with different rules, and their part-key lists are kept in
step by hand. They should share one constant.
- `protect_recent` translation and stream-safe hook gating for native
turns have no dedicated test.
- Gemini `thoughtsTokenCount` is not added to output tokens when usage
is relayed, so output savings attribution undercounts for thinking
models.
- This is not specific to native shapes: the marker-free default folds
very repetitive logs to a single line (for example `[3000 lines omitted:
3000 INFO]`) on chat bodies too, with no retrieval path.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
492 lines
26 KiB
Markdown
492 lines
26 KiB
Markdown
# Proxy Server Documentation
|
|
|
|
The Headroom proxy server is a production-ready HTTP server that applies context optimization to all requests passing through it.
|
|
|
|
> The proxy exposes compression-as-a-service via the `POST /v1/compress` endpoint — used by the [TypeScript SDK](typescript-sdk.md), LiteLLM's `headroom` guardrail, and gateway sidecars. It is loopback-only by default; see the endpoint section below.
|
|
|
|
## Starting the Proxy
|
|
|
|
```bash
|
|
# Basic usage
|
|
headroom proxy
|
|
|
|
# Deliberate public access, with the existing token protocol
|
|
HEADROOM_PROXY_TOKEN='replace-with-a-secret' headroom proxy --host 0.0.0.0 --port 8080
|
|
# Send `Authorization: Bearer replace-with-a-secret` or
|
|
# `X-Headroom-Proxy-Token: replace-with-a-secret` from the caller.
|
|
|
|
# With logging and budget
|
|
headroom proxy \
|
|
--log-file /var/log/headroom.jsonl \
|
|
--budget 100.0
|
|
```
|
|
|
|
### Common agent CLI entrypoints
|
|
|
|
```bash
|
|
# Claude Code
|
|
ANTHROPIC_BASE_URL=http://localhost:8787 claude
|
|
|
|
# GitHub Copilot CLI
|
|
headroom wrap copilot -- --model claude-sonnet-4-20250514
|
|
|
|
# OpenAI-compatible clients
|
|
OPENAI_BASE_URL=http://localhost:8787/v1 your-app
|
|
```
|
|
|
|
`headroom wrap copilot` uses Copilot CLI's BYOK provider settings under the hood. In `provider-type=auto`, it chooses Headroom's Anthropic route for the default proxy backend and the OpenAI-compatible `/v1` route for translated backends such as `anyllm` and LiteLLM.
|
|
|
|
Anonymous aggregate telemetry is **off by default** (opt-in). Opt in with `HEADROOM_TELEMETRY=on` or `headroom proxy --telemetry`. Downstream apps can set `HEADROOM_SDK=headroom-app` to override the anonymous telemetry `sdk` label; the default remains `proxy`.
|
|
|
|
Operational OTEL metrics are configured separately and are **off by default**. Install `headroom-ai[proxy,otel]` and set:
|
|
|
|
```bash
|
|
HEADROOM_OTEL_METRICS_ENABLED=1
|
|
HEADROOM_OTEL_METRICS_EXPORTER=otlp_http
|
|
HEADROOM_OTEL_METRICS_ENDPOINT=http://127.0.0.1:4318/v1/metrics
|
|
HEADROOM_OTEL_SERVICE_NAME=headroom-proxy
|
|
```
|
|
|
|
Use `HEADROOM_OTEL_METRICS_EXPORTER=console` for local smoke testing. `HEADROOM_TELEMETRY` controls the anonymous data-flywheel beacon only; it does not disable or enable OTEL export.
|
|
|
|
Langfuse can be enabled alongside this OTEL path for **trace ingestion**. Langfuse does **not** ingest OTEL metrics, so Headroom keeps metrics and Langfuse traces as complementary signals:
|
|
|
|
```bash
|
|
HEADROOM_LANGFUSE_ENABLED=1
|
|
LANGFUSE_PUBLIC_KEY=pk-lf-...
|
|
LANGFUSE_SECRET_KEY=sk-lf-...
|
|
LANGFUSE_BASE_URL=https://cloud.langfuse.com
|
|
```
|
|
|
|
When configured, Headroom emits OTLP traces for the shared compression pipeline to Langfuse while continuing to expose metrics through `/metrics` and OTEL metric exporters.
|
|
|
|
## Command Line Options
|
|
|
|
### Core Options
|
|
|
|
| Option | Default | Description |
|
|
|--------|---------|-------------|
|
|
| `--host` | `127.0.0.1` | Host to bind to |
|
|
| `--port` | `8787` | Port to bind to |
|
|
| `--mode` | `cache` | Run mode: `token` (maximize compression) or `cache` (freeze prior turns) |
|
|
| `--no-optimize` | `false` | Disable optimization (passthrough mode) |
|
|
| `--no-cache` | `false` | Disable semantic caching |
|
|
| `--no-rate-limit` | `false` | Disable rate limiting |
|
|
| `--log-file` | None | Path to JSONL log file |
|
|
| `--budget` | None | Daily budget limit in USD |
|
|
| `--code-aware` / `--no-code-aware` | disabled | Enable or disable AST-based code compression. Requires `headroom-ai[code]` (env: HEADROOM_CODE_AWARE_ENABLED=1 to enable) |
|
|
| `--anthropic-api-url` | `https://api.anthropic.com` | Custom Anthropic API URL endpoint |
|
|
| `--openai-api-url` | `https://api.openai.com` | Custom OpenAI API URL endpoint |
|
|
| `--anthropic-extra-headers` | unset | JSON object of extra headers merged into (and overriding) forwarded Anthropic requests, e.g. `'{"Api-Key": "..."}'` |
|
|
| `--openai-extra-headers` | unset | JSON object of extra headers merged into (and overriding) forwarded OpenAI requests |
|
|
|
|
### Run Modes
|
|
|
|
Headroom proxy has two explicit run modes:
|
|
|
|
- `token` mode: prioritize token reduction. Prior history may be rewritten when that improves compression.
|
|
- `cache` mode: prioritize provider prefix cache stability. Prior turns are frozen; only the newest turn is mutable.
|
|
|
|
Set via CLI or env:
|
|
|
|
```bash
|
|
headroom proxy --mode token
|
|
HEADROOM_MODE=cache headroom proxy
|
|
```
|
|
|
|
When to pick each:
|
|
|
|
- `token`: best for maximizing immediate compression savings.
|
|
- `cache`: best for long conversations where preserving prior-turn bytes improves prefix-cache reuse.
|
|
|
|
Legacy values (`token_headroom`, `cost_savings`) are still accepted as aliases.
|
|
|
|
### Context Management Options
|
|
|
|
Context management in the proxy is handled automatically by the compression pipeline. CCR (Compress-Cache-Retrieve) ensures that when content is compressed or messages are dropped, the original data remains accessible for the LLM to retrieve on demand. See [CCR documentation](ccr.md) for details.
|
|
|
|
Key CCR-related proxy flags:
|
|
|
|
| Option | Description |
|
|
|--------|-------------|
|
|
| `--no-ccr` | Disable CCR entirely — no retrieval markers in compressed output and no injected `headroom_retrieve` tool (lossy, no recovery path) |
|
|
| `--no-ccr-proactive-expansion` | Disable proactive context expansion before the LLM asks |
|
|
|
|
### ML Compression — RETIRED `--llmlingua` flag
|
|
|
|
The `--llmlingua` / `--llmlingua-device` / `--llmlingua-rate` flags and
|
|
the `headroom-ai[llmlingua]` extra were retired and replaced by Kompress
|
|
(ModernBERT). For the current opt-in path, install `headroom-ai[ml]`
|
|
and see [transforms.md](transforms.md) and [ARCHITECTURE.md](ARCHITECTURE.md).
|
|
|
|
## API Endpoints
|
|
|
|
### Liveness
|
|
|
|
```bash
|
|
curl http://localhost:8787/livez
|
|
```
|
|
|
|
Response:
|
|
```json
|
|
{
|
|
"service": "headroom-proxy",
|
|
"status": "healthy",
|
|
"alive": true,
|
|
"version": "0.5.21",
|
|
"timestamp": "2026-04-10T16:36:25Z",
|
|
"uptime_seconds": 12.483
|
|
}
|
|
```
|
|
|
|
### Readiness
|
|
|
|
```bash
|
|
curl http://localhost:8787/readyz
|
|
```
|
|
|
|
Response:
|
|
```json
|
|
{
|
|
"service": "headroom-proxy",
|
|
"status": "healthy",
|
|
"ready": true,
|
|
"version": "0.5.21",
|
|
"timestamp": "2026-04-10T16:36:25Z",
|
|
"uptime_seconds": 12.483,
|
|
"checks": {
|
|
"startup": {"enabled": true, "ready": true, "status": "healthy"},
|
|
"http_client": {"enabled": true, "ready": true, "status": "healthy"},
|
|
"cache": {"enabled": true, "ready": true, "status": "healthy"},
|
|
"rate_limiter": {"enabled": true, "ready": true, "status": "healthy"},
|
|
"memory": {"enabled": false, "ready": true, "status": "disabled"}
|
|
}
|
|
}
|
|
```
|
|
|
|
`/readyz` returns HTTP 503 when Headroom has not completed startup or a required enabled subsystem is unavailable. This is the endpoint used by the container health checks.
|
|
|
|
### Aggregate Health
|
|
|
|
```bash
|
|
curl http://localhost:8787/health
|
|
```
|
|
|
|
Response:
|
|
```json
|
|
{
|
|
"status": "healthy",
|
|
"ready": true,
|
|
"version": "0.5.21",
|
|
"config": {
|
|
"backend": "anthropic",
|
|
"optimize": true,
|
|
"cache": true,
|
|
"rate_limit": true
|
|
},
|
|
"checks": {
|
|
"startup": {"enabled": true, "ready": true, "status": "healthy"},
|
|
"http_client": {"enabled": true, "ready": true, "status": "healthy"}
|
|
}
|
|
}
|
|
```
|
|
|
|
### Detailed Statistics
|
|
|
|
```bash
|
|
curl http://localhost:8787/stats
|
|
```
|
|
|
|
`/stats` remains the live/session-oriented endpoint and now also includes a
|
|
`persistent_savings` block with durable proxy compression lifetime totals plus a
|
|
small recent preview. The existing `savings_history` field is still present and
|
|
remains session-scoped for backward compatibility.
|
|
|
|
For providers that return cache-write TTL bucket usage, `/stats` also includes
|
|
observed TTL breakdowns under `prefix_cache`:
|
|
|
|
- `observed_ttl_buckets.5m.tokens`
|
|
- `observed_ttl_buckets.1h.tokens`
|
|
- `observed_ttl_mix`
|
|
|
|
These are provider-reported observations, not configured TTL and not remaining
|
|
expiration time.
|
|
|
|
### Historical Savings
|
|
|
|
```bash
|
|
curl http://localhost:8787/stats-history
|
|
```
|
|
|
|
`/stats-history` exposes durable proxy compression history for dashboards and
|
|
other Headroom frontends. It returns:
|
|
|
|
- lifetime proxy compression totals
|
|
- compact checkpoint history by default, with `history_mode=full` available for
|
|
export/debug flows
|
|
- derived hourly, daily, weekly, and monthly rollups for charts
|
|
- a `history_summary` block describing stored versus returned checkpoint counts
|
|
- UTC timestamps throughout
|
|
|
|
By default the proxy stores this history at
|
|
`${HEADROOM_WORKSPACE_DIR}/proxy_savings.json` (i.e.
|
|
`~/.headroom/proxy_savings.json` when `HEADROOM_WORKSPACE_DIR` is unset).
|
|
Set `HEADROOM_SAVINGS_PATH` to override the location directly, or set
|
|
`HEADROOM_WORKSPACE_DIR` to relocate the full state root. See the
|
|
[Filesystem Contract](filesystem-contract.md).
|
|
|
|
`/dashboard` uses this endpoint directly for its historical view, including the
|
|
daily/weekly/monthly rollups and built-in JSON / CSV export buttons.
|
|
|
|
```bash
|
|
curl "http://localhost:8787/stats-history?format=csv&series=weekly"
|
|
curl "http://localhost:8787/stats-history?format=csv&series=monthly"
|
|
curl "http://localhost:8787/stats-history?history_mode=full"
|
|
```
|
|
|
|
### Prometheus Metrics
|
|
|
|
```bash
|
|
curl http://localhost:8787/metrics
|
|
```
|
|
|
|
`/metrics` remains the built-in Prometheus-formatted operational view. The proxy now also emits the same operational events through the OTEL facade when OTEL metrics are configured.
|
|
|
|
### LLM APIs
|
|
|
|
The proxy supports both Anthropic and OpenAI API formats:
|
|
|
|
```bash
|
|
# Anthropic format
|
|
POST /v1/messages
|
|
|
|
# OpenAI format
|
|
POST /v1/chat/completions
|
|
```
|
|
|
|
### `POST /v1/compress`
|
|
|
|
Compression-only endpoint. Compresses messages without ever making a **completion request to an LLM provider** — no generation, no provider API key. Used by the [TypeScript SDK](typescript-sdk.md), LiteLLM's `headroom` guardrail, and gateway sidecars.
|
|
|
|
**It does run local ML models.** Compression is ML-backed: Kompress is a ModernBERT encoder scoring tokens for retention (classification, not generation) and Magika classifies content types, both in-process by default. If `HEADROOM_KOMPRESS_ENDPOINT` is set, Kompress inference is offloaded over HTTP to that model server — real egress from the sidecar. Only inference goes remote; the CCR store and markers stay proxy-local. `HEADROOM_DISABLE_KOMPRESS=1` gives structural compression only.
|
|
|
|
**Loopback-only by default.** Non-loopback callers get **404** (not 403 — the route stays invisible to scanners). Set `HEADROOM_COMPRESS_ALLOW_REMOTE=1` to allow remote callers.
|
|
|
|
**No format conversion.** `messages` may be OpenAI-shaped (`role: "tool"` + `tool_call_id`) or Anthropic-shaped (`tool_use` / `tool_result` content blocks); the same shape comes back. In gateway mode it may also be a native Gemini `generateContent` body (`contents`) or a Bedrock Converse body (untyped `text`/`toolUse`/`toolResult`/`cachePoint` blocks); their text and tool-result slots are compressed in place and `body` keeps the native shape, with `tools`/`toolConfig`, steering and re-drive left out. `config.mode: "ccr"` is a 400 for these bodies (no retrieval tool can be added). `model` selects the tokenizer and context limit — send the real name, including gateway-prefixed forms like `bedrock/anthropic.claude-3-5-sonnet`.
|
|
|
|
**`system` and `tools` are ignored outside gateway mode.** Anthropic sends both out of band. Without a `gateway` block this endpoint accepts them without complaint (200, no warning) and returns neither, so neither is compressed — keep carrying them yourself. That means the Anthropic system prompt is not compressed here, and tool-schema compaction / tool-search deferral are not reachable this way. Gateway mode (below) passes `tools` through and compacts them; in OSS, native tool-search deferral is still proxy-only — the headroom-tool-search extension adds it to gateway mode as a turn hook.
|
|
|
|
**Request:**
|
|
```json
|
|
{
|
|
"messages": [...], // either wire format
|
|
"model": "gpt-4o", // tokenizer + context limit
|
|
"token_budget": 8000, // optional: override the context limit
|
|
"config": { // optional
|
|
"mode": "lossy_inline", // ccr | lossy_inline | lossless_then_lossy | lossless
|
|
"frozen_message_count": 12, // pin an already-cached prefix
|
|
"session_id": "conv-8f1c", // session mode: Headroom keeps the replay state
|
|
"compress_user_messages": false,
|
|
"target_ratio": 0.5,
|
|
"protect_recent": 2,
|
|
"protect_analysis_context": true
|
|
}
|
|
}
|
|
```
|
|
|
|
**Response:**
|
|
```json
|
|
{
|
|
"messages": [...], // compressed messages
|
|
"tokens_before": 15000,
|
|
"tokens_after": 3500,
|
|
"tokens_saved": 11500,
|
|
"compression_ratio": 0.23, // tokens_after / tokens_before — LOWER is better
|
|
"transforms_applied": ["router:smart_crusher:0.35"],
|
|
"transforms_summary": {"router:smart_crusher:0.35": 1},
|
|
"ccr_hashes": [], // non-empty only with mode="ccr"
|
|
"session": {"id": "conv-8f1c", "frozen_message_count": 12, "cached_prefix_replayed": false} // session mode only
|
|
}
|
|
```
|
|
|
|
**Headers:**
|
|
- `x-headroom-bypass: true` — skip compression, return messages as-is with zeroed metrics
|
|
|
|
**Error responses:** 400 (missing/invalid fields, bad `config.mode`, `config.frozen_message_count` or `config.session_id`, `session_id` with `compress_user_messages`, malformed `gateway` block), 401 (bad `HEADROOM_PROXY_TOKEN`), 404 (non-loopback without `HEADROOM_COMPRESS_ALLOW_REMOTE=1`), 503 (compression failed; in session mode also `compression_timeout` — retry the turn)
|
|
|
|
**Fail-open:** without a session, on timeout you get 200 with the original messages plus `compression_skipped: true` and `skip_reason: "compression_timeout"`. In session mode a timeout or busy session lock is a 503 instead, because handing back originals would desync the replay state.
|
|
|
|
**Multi-turn callers — don't lose the prefix cache.** Without `config.session_id` this endpoint is stateless: unlike the proxy's own request path (which runs a CacheAligner and tracks provider cache hits across turns), it has no idea what the provider already cached. Either let Headroom keep the state (session mode, below) or keep it yourself with `frozen_message_count`.
|
|
|
|
The provider caches the bytes you *forwarded*, which compression already changed — so your originals and the cached prefix are no longer the same thing, and it is the forwarded version you must keep reproducing. Compression also varies with position: an older tool result can fall outside the recent-read protection window as the conversation grows and be compressed harder than last turn, so re-compression is not guaranteed to reproduce earlier output either. Two rules:
|
|
|
|
1. Pass `config.frozen_message_count` = the number of leading messages already cached upstream.
|
|
2. Send back the messages you **previously forwarded**, not the pristine originals. `frozen_message_count` returns leading messages exactly as passed in, so feeding it originals hands the provider different bytes than last turn and busts the cache anyway.
|
|
|
|
```python
|
|
forwarded = []
|
|
|
|
|
|
def next_turn(new_messages):
|
|
r = requests.post(
|
|
f"{proxy}/v1/compress",
|
|
json={
|
|
"messages": forwarded + new_messages,
|
|
"model": "claude-sonnet-4-6",
|
|
"config": {"frozen_message_count": len(forwarded)},
|
|
},
|
|
).json()
|
|
forwarded[:] = r["messages"] # next turn's frozen prefix
|
|
return forwarded
|
|
```
|
|
|
|
Note `protect_recent` is not a substitute — it guards the newest messages, while `frozen_message_count` guards the oldest, which is the cached end.
|
|
|
|
**Session mode.** Pass `config.session_id` (non-empty, at most 256 chars) and Headroom keeps the conversation's replay state itself — the same per-session compression cache and prefix tracker the proxy uses — so a gateway that owns routing can resend the raw conversation every turn and get a byte-identical prefix back. Everything previously returned for the session comes back unchanged; only the new tail is compressed; an explicit `frozen_message_count` still wins when larger. Forward the returned messages verbatim. `compress_user_messages` is refused (400). State is in-process by default (replay cache: `HEADROOM_COMPRESSION_CACHE_TTL_SECONDS`, default 3900; tracker state: 10 idle minutes), so multi-process deployments must pin a session to one process. With `HEADROOM_COMPRESS_SESSION_STORE=redis://...` (needs the `redis` extra, included in every Docker image) each session's tracker is also kept in Redis and any replica can serve its next turn and its `/v1/usage`; the CCR store follows to Redis unless `HEADROOM_CCR_BACKEND` names another. The per-session lock stays per process: two concurrent turns of one session on two replicas both run and the first write wins (compare-and-set). Gateway-mode pending turns stay in-process, so re-drive still needs real affinity. An unreachable Redis never fails a turn: the replica serves from its own memory (one `WARNING` when calls start failing, `INFO` when they recover, Redis skipped 5 s after a connection error). A replica with no local state then recompresses the history, slower, and can return a different prefix (a provider prompt-cache miss) with `cached_prefix_replayed: false`. Run Redis with high availability. `HEADROOM_COMPRESS_SESSION_FROM_HEADER=1` lets the `x-headroom-session-id` header stand in for `config.session_id` (off by default).
|
|
|
|
The response's `session` object: `id` is the session id the turn ran under (`config.session_id`, or the header above). `frozen_message_count` is how many leading messages the pipeline was told not to recompress: the leading run Headroom can replay from the session's compression cache (plain messages always count, a tool result only once its content is cached, the last message never), or `config.frozen_message_count` when larger; it comes from Headroom's state, not the provider, and can be non-zero on a first turn. `cached_prefix_replayed` is `true` when the pipeline's output for some leading messages differed from what was returned last turn and Headroom overwrote them with last turn's bytes; `false` when the serving process holds no previous turn for the session (first turn, expired state, a replica that never served or loaded it), when the pipeline already matched (the usual case), or when the replay was skipped as unsafe (messages do not line up with last turn's, or the replay would be larger), so `false` does not mean the prefix matches last turn's. It is **not** a provider prompt-cache hit/miss signal; read that from the provider's usage (`cache_read_input_tokens`, `cached_tokens`).
|
|
|
|
Optionally relay the provider's usage for attribution with `POST /v1/usage` `{"session_id": "...", "usage": {...}}`. Anthropic `cache_read_input_tokens` / `cache_creation_input_tokens` and OpenAI `prompt_tokens_details.cached_tokens` / flat `cached_tokens` are accepted; a block with none of them is a 400. Replies `{"session_id", "frozen_message_count", "applied"}` (`applied: false, reason: "no_cache_signal"` when the only present field is 0), 404 `unknown_session`, or 503 `session_busy` (retry). Telemetry only — it never raises the frozen count.
|
|
|
|
**Gateway mode (two-half turn contract).** Add a top-level `gateway` object (`{}` is enough) and one model turn becomes two calls, so a gateway that never lets Headroom see the provider response can still run transforms whose reload step needs it — the proxy's "no shrink without reload" rule at the API boundary. `gateway` fields: `can_redrive` (default `false`: the gateway can call the provider again with a request Headroom hands it; send `false` when streaming), `can_relay_response` (default `false`: the gateway will post status/usage after each turn), `session_affinity` (default `true`; `false` disables re-driving because pending turns are in-process, with or without `HEADROOM_COMPRESS_SESSION_STORE`), `plugin_version` (diagnostics). Every other top-level field (`system`, `tools`, `temperature`, …) passes through; `tools` is compacted deterministically (`tool_schema_compaction`; `tool_desc_compaction` with `HEADROOM_TOOL_DESC_MAX_CHARS`); extension turn hooks run, stream-safe-only unless `can_redrive` and `session_affinity` both hold. The response adds `body` (the complete provider request — forward it as-is; never contains `config`, `gateway`, `token_budget`), `turn_id`, `route` (`{model, provider, service_tier, reason}`, advisory; a routing extension's model is also written into `body.model`), `obligations` (`redrive` and/or `relay_usage`) and a `gateway` echo. Fail-open answers carry the same keys with originals and `obligations: []`. Savings follow the proxy path: when compaction shortened `tools`, `tokens_before`/`tokens_after` also count the tool definitions, and the answer adds `tool_tokens_saved` (tool-search deferral plus tools a turn hook removed, both net; `tokens_saved + tool_tokens_saved` is the `/stats` headline) and `tool_tokens_deferred` (its deferral share, booked `estimated`); both are absent on fail-open answers. A turn is registered only when `obligations` is non-empty; with `relay_usage` the `/stats` record waits for the response half. Knobs: `HEADROOM_GATEWAY_TURN_TTL_SECONDS` (120), `HEADROOM_GATEWAY_MAX_PENDING_TURNS` (10000), `HEADROOM_GATEWAY_MAX_REDRIVES` (8). With `config.mode: "ccr"` and re-drive allowed, `headroom_retrieve` is injected into `body.tools` (`ccr_tool_injected`) and the response half answers the model's retrieval calls itself.
|
|
|
|
`POST /v1/compress/response` (same exposure rules as `/v1/compress`) is the response half: `{"turn_id", "status": 200, "latency_ms", "usage": {...}, "response": {...}}`. `usage` accepts Anthropic, OpenAI chat (`prompt_tokens_details.cached_tokens`), OpenAI Responses (`input_tokens_details.cached_tokens`) and Kong's flat `cached_tokens` shapes, or the whole provider body (a nested `usage` key is descended into once); billed counters are summed across re-drive rounds. `response` is required when the turn carries `redrive` and the provider call succeeded; a failed call closes the turn with its `status` alone. Answers: `{"action": "done", "turn_id", "response": <replacement, the latest provider response after a re-drive, or null — forward what you hold>, "frozen_message_count", "usage_applied", "rounds", "billed_usage"}` or `{"action": "redrive", "turn_id", "request": <full provider body to send>, "round"}` — post the provider's JSON back under the same `turn_id`; `billed_usage` sums every counter (cache reads and writes included) across rounds in Anthropic keys, and a re-driven turn's `response` is never null and reports that total as its `usage`; past `HEADROOM_GATEWAY_MAX_REDRIVES` the turn ends with `done` and the latest provider response. Errors: 400 `invalid_request` / `missing_response`, 404 `unknown_turn` (unregistered, finished, or expired), 409 `turn_busy`.
|
|
|
|
**Kong plugin.** [kong-plugin-headroom](https://github.com/headroomlabs-ai/kong-plugin-headroom) (its own repo; `luarocks install kong-plugin-headroom`) implements both halves for Kong Gateway 3.9 (session id from a header, usage relay from the `log` phase, re-drive loop in `access`). The contract itself is installed in Headroom through the compress-turn seam (`headroom.proxy.compress_turn`, `HEADROOM_GATEWAY_CONTRACT`), the same seam a third-party contract would use.
|
|
|
|
## Using with Claude Code
|
|
|
|
```bash
|
|
# Start proxy
|
|
headroom proxy --port 8787
|
|
|
|
# In another terminal
|
|
ANTHROPIC_BASE_URL=http://localhost:8787 claude
|
|
```
|
|
|
|
## Using with Cursor
|
|
|
|
1. Start the proxy: `headroom proxy`
|
|
2. In Cursor settings, set the base URL to `http://localhost:8787`
|
|
|
|
## Using with OpenAI SDK
|
|
|
|
```python
|
|
from openai import OpenAI
|
|
|
|
client = OpenAI(
|
|
base_url="http://localhost:8787/v1",
|
|
api_key="your-api-key", # Still needed for upstream
|
|
)
|
|
```
|
|
|
|
## Features
|
|
|
|
### ML Compression (Opt-In, Kompress)
|
|
|
|
> The earlier LLMLingua-2 integration documented in this section
|
|
> (`--llmlingua`, `--llmlingua-device`, `--llmlingua-rate`,
|
|
> `headroom-ai[llmlingua]`, `LLMLinguaCompressor`) was retired and
|
|
> replaced by **Kompress** (ModernBERT). Install with `pip install
|
|
> 'headroom-ai[ml]'`. See [transforms.md](transforms.md) and
|
|
> [ARCHITECTURE.md](ARCHITECTURE.md) for current configuration.
|
|
|
|
### Semantic Caching
|
|
|
|
The proxy caches responses for repeated queries:
|
|
|
|
- LRU eviction with configurable max entries
|
|
- TTL-based expiration
|
|
- Cache key based on message content hash
|
|
|
|
### Rate Limiting
|
|
|
|
Token bucket rate limiting protects against runaway costs:
|
|
|
|
- Configurable requests per minute
|
|
- Configurable tokens per minute
|
|
- Per-API-key tracking
|
|
|
|
### Cost Tracking
|
|
|
|
Track spending and enforce budgets:
|
|
|
|
- Real-time cost estimation
|
|
- Budget periods: hourly, daily, monthly
|
|
- Automatic request rejection when over budget
|
|
|
|
### Prometheus Metrics
|
|
|
|
Export metrics for monitoring:
|
|
|
|
```
|
|
headroom_requests_total
|
|
headroom_tokens_saved_total
|
|
headroom_cost_usd_total
|
|
headroom_latency_ms_sum
|
|
```
|
|
|
|
## Configuration via Environment
|
|
|
|
```bash
|
|
# For deliberate public access, configure the existing token and send it from callers.
|
|
export HEADROOM_HOST=0.0.0.0
|
|
export HEADROOM_PROXY_TOKEN='replace-with-a-secret'
|
|
export HEADROOM_PORT=8787
|
|
export HEADROOM_BUDGET=100.0
|
|
|
|
# Route OpenAI passthrough requests to a custom endpoint
|
|
export OPENAI_TARGET_API_URL=https://custom.openai.endpoint.com
|
|
|
|
# Route Anthropic passthrough requests to a custom endpoint
|
|
export ANTHROPIC_TARGET_API_URL=https://litellm.company.internal
|
|
|
|
headroom proxy
|
|
```
|
|
|
|
## Running in Production
|
|
|
|
For production deployments:
|
|
|
|
```bash
|
|
# Use a process manager
|
|
pip install gunicorn
|
|
|
|
# Run with gunicorn — server.py has no module-level `app`; FastAPI is built
|
|
# by the create_app() factory, so gunicorn needs --factory
|
|
gunicorn headroom.proxy.server:create_app \
|
|
--workers 4 \
|
|
--bind 0.0.0.0:8787 \
|
|
--worker-class uvicorn.workers.UvicornWorker \
|
|
--factory
|
|
# Callers must send `Authorization: Bearer replace-with-a-secret` or
|
|
# `X-Headroom-Proxy-Token: replace-with-a-secret`.
|
|
```
|
|
|
|
Or with Docker:
|
|
|
|
```dockerfile
|
|
FROM python:3.11-slim
|
|
RUN apt-get update && apt-get install -y --no-install-recommends build-essential \
|
|
&& pip install "headroom-ai[proxy]" \
|
|
&& apt-get purge -y build-essential && apt-get autoremove -y \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
EXPOSE 8787
|
|
CMD ["headroom", "proxy", "--host", "0.0.0.0", "--port", "8787"]
|
|
```
|
|
|
|
Run this image with an explicit token and a deliberate publication choice:
|
|
|
|
```bash
|
|
docker run --rm -p 127.0.0.1:8787:8787 \
|
|
-e HEADROOM_PROXY_TOKEN='replace-with-a-secret' \
|
|
headroom-proxy
|
|
```
|
|
|
|
Callers must send `Authorization: Bearer replace-with-a-secret` or
|
|
`X-Headroom-Proxy-Token: replace-with-a-secret`. For network access, replace
|
|
the host-side `127.0.0.1` with an intentional public address and keep the
|
|
token configured.
|
|
|
|
> **Note:** `build-essential` is required at install time because `headroom-ai` includes `hnswlib`, a C++ extension that must be compiled from source. It is removed after installation to keep the image slim.
|