mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description
Translated OpenAI streams (`/v1/chat/completions` through a backend such
as `anyllm` or LiteLLM) recorded the **first nonzero** usage snapshot
instead of the last one.
`_absorb` in `_stream_openai_via_backend` wrote a counter only when the
accumulator was still falsy:
```python
if key in usage and not stream_state.get(key):
stream_state[key] = usage[key]
```
A provider that reports climbing cumulative snapshots on successive
content chunks therefore had every later frame thrown away. Replaying
the two frames from the issue through the real proxy path, `/stats`
recorded 10 input / 1 output tokens and `/metrics` recorded 2 cache
reads — while the client received both frames intact. The bug is in
accounting after delivery, not a dropped upstream event.
Closes #4263
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `_parse_sse_usage_from_buffer` (openai branch): absent counters no
longer default to `0`. A frame that omits a field was previously
indistinguishable from one reporting an explicit zero, which is a
prerequisite for "absent keeps the last value".
- `_absorb`: assigns unconditionally, so the latest cumulative snapshot
wins. An absent field keeps its earlier value; an explicit zero still
replaces it.
- Removed `_parse_completion_tokens_from_sse_chunk`. It existed to catch
completion-only frames, but the buffered SSE parser already handles
those, and its `not stream_state["output_tokens"]` guard latched the
output count a second time.
- Replaced that helper's unit tests with eight regressions driving the
real `_stream_openai_via_backend` path through stream finalization, in
Chat and Responses shapes.
- `docs/content/docs/proxy.mdx`: one paragraph on the `POST
/v1/chat/completions` page documenting the accounting rule, in the
published docs site.
Net: -100/+113 lines, of which most deletions are the removed helper and
its implementation-only tests.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_streaming_usage_parser.py -q
........ [100%]
8 passed in 1.01s
$ ruff check headroom/proxy/handlers/streaming.py tests/test_streaming_usage_parser.py
All checks passed!
$ ruff format --check headroom/proxy/handlers/streaming.py tests/test_streaming_usage_parser.py
2 files already formatted
$ mypy headroom/proxy/handlers/streaming.py
Success: no issues found in 1 source file
```
Regression set — the new tests against the unfixed handler (`git stash`
of `streaming.py` only):
```text
$ python -m pytest tests/test_streaming_usage_parser.py -q
FAILED ...[latest-snapshot-chat] assert (10, 1, 2, 3) == (13, 5, 9, 1)
FAILED ...[latest-snapshot-responses]
FAILED ...[partial-chat] assert (10, 1, 2, 3) == (10, 5, 2, 3)
FAILED ...[partial-responses]
FAILED ...[explicit-zero-chat] assert (10, 1, 2, 3) == (0, 0, 0, 0)
FAILED ...[explicit-zero-responses]
6 failed, 2 passed in 1.07s
```
Six fail before the fix, all eight pass after — matching the issue
exactly.
Wider streaming and parser suites:
```text
$ python -m pytest tests/test_streaming_usage_parser.py tests/test_streaming_sse_malformed_events.py \
tests/test_provider_billed_input.py tests/test_proxy_cache_ttl_metrics.py tests/test_proxy_handler_helpers.py \
tests/test_proxy_streaming_request_logger.py tests/test_route_advice.py tests/test_h2_stream_reset_retry.py \
tests/test_proxy_streaming_resilience.py tests/test_proxy_streaming_ratelimit_headers.py -q
3 failed, 607 passed in 10.21s
```
The three failures are
`tests/test_proxy_cache_ttl_metrics.py::test_stats_endpoint_reports_*`,
which raise `SystemExit: 78` from `_check_rust_core()` in `server.py`.
They are environmental (unbuilt Rust core in this checkout) and fail
identically on unmodified `upstream/main` with this branch stashed —
unrelated to this change.
## Real Behavior Proof
- Environment: Windows, Python 3.13.11, this branch, `PYTHONPATH` set to
the checkout.
- Exact command / steps: replayed the issue's two usage snapshots as a
Chat stream through `_stream_openai_via_backend` with a stub backend,
then read the resulting `RequestOutcome`.
- Observed result: the corrected counts from the issue; previously 10 /
1 / 2.
```text
provider_input_tokens=13 (expect 13)
output_tokens =5 (expect 5)
cache_read_tokens =9 (expect 9)
```
- Not tested: a live `anyllm`/LiteLLM upstream over real HTTP (no
credentials in this environment). The replay drives the identical
handler and parser path the proxy uses; the frames are byte-for-byte the
issue's.
## Runtime Rollout Safety
- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: token accounting for translated
OpenAI streams now records the final snapshot. This is the fix; counts
move up from truncated to complete.
- Kill switch / disable path: none beyond reverting this commit.
- Unsafe override required: none.
- Qualification impact: none.
- Rollback path: `git revert`.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
Authorship is credited to @oxura, who diagnosed this in #4263 and wrote
the original patch on their fork; they were holding it back on the
ten-open-PR limit. This PR carries their change rebased onto current
`upstream/main` (their branch was cut from `fbf7bc32`, and
`streaming.py` has since drifted ~200 lines), with the commit message
and tests adapted to the current tree. Their `Authored-by` trailer is on
the commit.
One hunk from their patch was redirected: they documented the rule in
`wiki/proxy.md`, which `AGENTS.md:38` says is unpublished and must not
receive new content, so the paragraph now lives on the `POST
/v1/chat/completions` page in `docs/content/docs/proxy.mdx`. Thanks to
Devin Review for flagging it.
Co-authored-by: oxura <grozafroza44@gmail.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
1225 lines
90 KiB
Plaintext
1225 lines
90 KiB
Plaintext
---
|
||
title: Proxy Server
|
||
description: Run the Headroom proxy to compress LLM traffic for any client — Claude Code, Cursor, OpenAI SDK, or custom apps.
|
||
---
|
||
|
||
The Headroom proxy is a standalone HTTP server that compresses all LLM traffic passing through it. Point any client at the proxy and get automatic context optimization.
|
||
|
||
Running a local OpenAI-compatible model? See [Local LLM prefill benchmarking](/docs/local-llm-prefill) for a baseline-vs-optimized workflow that measures prompt-processing savings with the dashboard.
|
||
|
||
## Starting the proxy
|
||
|
||
```bash
|
||
# Basic usage
|
||
headroom proxy
|
||
|
||
# Deliberate public access, with the existing token protocol
|
||
HEADROOM_PROXY_TOKEN='replace-with-a-secret' headroom proxy --host 0.0.0.0 --port 8080
|
||
# Send `Authorization: Bearer replace-with-a-secret` or
|
||
# `X-Headroom-Proxy-Token: replace-with-a-secret` from the caller.
|
||
|
||
# With logging and budget
|
||
headroom proxy \
|
||
--log-file /var/log/headroom.jsonl \
|
||
--budget 100.0
|
||
```
|
||
|
||
Two telemetry switches exist. `HEADROOM_TELEMETRY=on` (or `--telemetry`) turns on **local-only** in-process usage stats that power your own `/stats`, `/metrics`, and dashboard — off by default, and nothing under this switch leaves the machine.
|
||
|
||
Separately, an **anonymous usage beacon is on by default** (opt-out). Once a session goes idle it POSTs a small summary of how compression behaved to a Headroom Labs collector.
|
||
|
||
**Why it exists.** Compression quality is the product. The beacon tells us when a release changes it — a ratio that regresses, a content type that starts getting skipped, a provider path that begins failing — across real workloads rather than only our own test corpus. That is the whole purpose; there is no other use.
|
||
|
||
**What it sends.** Counters and identifiers, never free text. The collector applies an allowlist server-side and drops everything else before storage ([`deploy/beacon/worker.js`](https://github.com/headroomlabs-ai/headroom/blob/main/deploy/beacon/worker.js)): token totals, compression ratios and rates, skip reasons, source and provider names, model ids, failure counts and status codes, and a routing summary. Alongside those, seven resource attributes: service name and version, install id, install mode, stack, OS type and CPU architecture. A typical event is about 2KB.
|
||
|
||
**What it never sends.** Your prompts, your completions, your code, file paths, file contents, environment variables, or anything else derived from request or response bodies. None of these are on the allowlist, so even if a future client sent one the collector would discard it.
|
||
|
||
The install id is a random UUID generated on first run and stored in your config directory. It is deliberately not derived from hostname, MAC address, or any hardware property — delete the file and you get a new one.
|
||
|
||
Disable it with `HEADROOM_BEACON=off`, `HEADROOM_TELEMETRY=off` (or `--no-telemetry`), the `DO_NOT_TRACK=1` convention, or `HEADROOM_OFFLINE=1` (which also disables update checks and license reporting). `HEADROOM_BEACON=on` keeps the beacon on even with local stats off. `headroom telemetry` names the setting that turned it off.
|
||
|
||
## CLI options
|
||
|
||
### Core
|
||
|
||
| Option | Default | Description |
|
||
|--------|---------|-------------|
|
||
| `--host` | `127.0.0.1` | Host to bind to |
|
||
| `--port` | `8787` | Port to bind to |
|
||
| `--uds` | None | Serve on a Unix domain socket at this path instead of `--host`/`--port`. POSIX only — Windows has no `AF_UNIX` support in Python or asyncio. See [Serving on a Unix socket](#serving-on-a-unix-socket) |
|
||
| `--workers` | `1` | Number of Uvicorn worker processes |
|
||
| `--limit-concurrency` | `1000` | Maximum concurrent connections before Uvicorn returns 503 |
|
||
| `--max-connections` | `500` | Maximum upstream HTTP connections |
|
||
| `--max-keepalive` | `100` | Maximum upstream keep-alive connections |
|
||
| `--http-proxy` | None | HTTP proxy URL for upstream provider requests only; HTTPS provider APIs use CONNECT |
|
||
| `--mode` | `cache` | Optimization mode: `token` prioritizes compression, `cache` preserves provider prefix-cache stability. Default is `cache` (see [Savings profiles](#savings-profiles)) |
|
||
| `--no-optimize` | `false` | Disable optimization (passthrough mode) |
|
||
| `--no-cache` | `false` | Disable semantic caching |
|
||
| `--no-rate-limit` | `false` | Disable rate limiting |
|
||
| `--log-file` | None | Path to JSONL log file |
|
||
| `--log-messages` | `false` | Store full request/response content for the live feed |
|
||
| `--budget` | None | Daily budget limit in USD |
|
||
| `--openai-api-url` | `https://api.openai.com` | Custom OpenAI API URL |
|
||
| `--provider-name` | Detected from `--openai-api-url` | Display name for the OpenAI-compatible upstream on the dashboard (e.g. `OpenRouter`). Well-known hosts (OpenRouter, Groq, Together, Azure OpenAI, …) are detected automatically; this overrides them. Routing and pricing are unaffected. |
|
||
| `--anthropic-api-url` | Anthropic default | Custom Anthropic API URL |
|
||
| `--gemini-api-url` | Gemini default | Custom Gemini API URL |
|
||
| `--backend` | `anthropic` | Backend: `anthropic`, `bedrock`, `openrouter`, `anyllm`, or `litellm-<provider>` |
|
||
| `--bedrock-api-url` | None | Bedrock InvokeModel upstream for the `/model/{id}/invoke` passthrough routes (see [Bedrock via a local gateway](#bedrock-via-a-local-gateway)) |
|
||
| `--factory-api-url` | None | Factory Droid upstream (env: `FACTORY_TARGET_API_URL`). Registers `/api/*` routes: `/api/llm/a/v1/messages` is compressed, other `/api/*` paths pass through verbatim. Paths outside `/api/` are never sent to Factory. Set automatically by `headroom wrap droid` (see [Factory Droid](/docs/droid)) |
|
||
| `--telemetry` | `false` | Enable local, in-process usage stats (for your own `/stats` and dashboard; nothing leaves the machine) |
|
||
| `--no-telemetry` | `false` | Force local telemetry off (already the default) |
|
||
| `--stateless` | `false` | Disable filesystem writes and keep runtime state in memory |
|
||
|
||
Use `--http-proxy` or `HEADROOM_HTTP_PROXY` when only provider API traffic should go through a proxy:
|
||
|
||
```bash
|
||
headroom proxy --http-proxy http://proxy.internal:8080
|
||
```
|
||
|
||
Avoid setting process-wide variables such as `HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, or `NO_PROXY` for this use case. HTTPX reads those variables too, but Headroom also inherits them into tool executions, so they can proxy unrelated tool traffic.
|
||
|
||
### Context management
|
||
|
||
| Option | Default | Description |
|
||
|--------|---------|-------------|
|
||
| `--mode token` | | Prioritize token compression; prior turns may be rewritten for maximum savings. |
|
||
| `--mode cache` | default | Freeze prior turns to maximize provider prefix-cache hit rate. This is the effective default (see [Savings profiles](#savings-profiles)). |
|
||
| `--intercept-tool-results` | `false` | Opt into canary tool-result interceptors such as ast-grep Read outlining. Requires `HEADROOM_ROLLOUT_CHANNEL=canary` (or `dev`). |
|
||
| `--no-read-lifecycle` | `false` | Disable stale/superseded Read-output compression. |
|
||
| `--code-aware` / `--no-code-aware` | disabled | Enable or disable AST-based code compression. Requires `headroom-ai[code]`. |
|
||
| `--code-graph` | `false` | Enable the proxy's live code-graph file watcher for the current project. |
|
||
|
||
#### Code-memory MCP (Serena)
|
||
|
||
`headroom wrap` registers no code-memory MCP by default. Pass `--code-memory serena` (or set `HEADROOM_CODE_MEMORY=serena`) to register **[Serena](https://github.com/oraios/serena)** for semantic, symbol-level code navigation. Serena runs on demand via `uvx` — Headroom downloads and executes no binary of its own — and indexes the current project locally. It is opt-in because its ~20 tool definitions add about 6K tokens to every request where the agent does not defer tools, and indexing runs at each launch.
|
||
|
||
> **Upgrading?** Earlier releases registered Serena by default. On your next `headroom wrap`, Headroom removes the Serena entry it installed and says so; pass `--code-memory serena` to keep it. For Claude Code the entry is per project, so it is removed the next time you run `headroom wrap claude` (or `headroom unwrap claude`) in that directory. A Serena you registered yourself is left alone.
|
||
|
||
For Claude Code, the entry is written at `local` scope, which means it applies to the directory you wrapped and nothing else. That keeps a Serena that cannot start from breaking Claude Code sessions in unrelated projects. Pass `--code-memory-scope user` (or set `HEADROOM_CODE_MEMORY_SCOPE=user`) to register it machine-wide instead, which is how releases before #2787 behaved. If an earlier version already registered Serena machine-wide, the next `headroom wrap claude --code-memory serena` moves it into the project scope for you — but only when Headroom installed that entry itself; a Serena you registered by hand is left alone.
|
||
|
||
> **Upgrading from tokensave?** Earlier releases registered a `tokensave` MCP server (a downloaded Rust binary). tokensave has been retired in favour of Serena. On your next `headroom wrap` / `headroom unwrap`, Headroom removes the `tokensave` MCP entry it installed — nothing to migrate, since it was just an index rebuilt from your source. The leftover `tokensave` binary in `~/.local/bin` and any `.tokensave/` folders are unused and safe to delete.
|
||
|
||
By default, the proxy uses the shared **ContentRouter** pipeline. It routes text, logs, JSON, code, images, and tool outputs through the currently enabled compressors and preserves reversible CCR markers where applicable.
|
||
|
||
```bash
|
||
# Maximize compression
|
||
headroom proxy --mode token
|
||
|
||
# Preserve provider prefix cache stability
|
||
headroom proxy --mode cache
|
||
```
|
||
|
||
### Savings profiles
|
||
|
||
`HEADROOM_SAVINGS_PROFILE` selects a named profile that seeds Headroom's whole compression posture — proxy mode, keep-ratio, which messages are compressed, and `force_kompress` — at proxy startup. It is read by `headroom proxy` and by the `headroom wrap` subprocesses. When unset, the default profile is `coding`.
|
||
|
||
| Profile | Target savings | Mode | Notes |
|
||
|---------|----------------|------|-------|
|
||
| `coding` | ~50% of new input tokens (emergent) | `cache` | **Default.** Delta-only compression at ~0 prefix-cache busts; never lossy-compresses file reads. Whole-request and lifetime ratios are much lower in cache mode. |
|
||
| `balanced` | ~70% of new input tokens | `token` | Moderate compression with structural compaction. |
|
||
| `agent-90` | ~90% of new input tokens | `token` | Aggressive; pins a `0.10` keep-ratio and forces Kompress. |
|
||
| `general` | ~60% of new input tokens (emergent) | `token` | Non-coding workloads. |
|
||
|
||
An unrecognized `HEADROOM_SAVINGS_PROFILE` value logs a warning and falls back to `coding` (the same default an unset variable resolves to) — the proxy never fails to start over a bad profile name. `balanced` is used only as a last-resort fallback when the runtime has no `coding` entry at all (old-runtime/new-client version skew). See `headroom/agent_savings.py` for each profile's full set of knobs.
|
||
|
||
Because the default `coding` profile uses **cache** mode (and the proxy's own default mode is also `cache`), Headroom runs in cache mode out of the box. For a direct `headroom proxy` invocation, mode precedence is `--mode` > `HEADROOM_MODE` > the `cache` default. The selected `HEADROOM_SAVINGS_PROFILE` does not provide a third-level mode fallback after startup; wrapped launches may seed `HEADROOM_MODE` before starting the proxy. To run token mode, pass `--mode token` or set `HEADROOM_MODE=token`:
|
||
|
||
```bash
|
||
# Aggressive ~90% token-savings profile
|
||
HEADROOM_SAVINGS_PROFILE=agent-90 headroom proxy --port 8787
|
||
```
|
||
|
||
### Optional features
|
||
|
||
| Option | Default | Description |
|
||
|--------|---------|-------------|
|
||
| `--memory` | `false` | Enable persistent user memory and provider-appropriate memory tools. In `--mode cache`, automatic context injection is skipped only for Anthropic `/v1/messages` to preserve prefix stability; OpenAI chat/responses and Gemini append memory to the live-zone tail. Use `--mode token` for automatic memory context on the Anthropic path. |
|
||
| `--memory-db-path` | `{cwd}/.headroom/memory.db` | Override the memory SQLite path |
|
||
| `--no-memory-tools` | `false` | Disable automatic memory tool injection |
|
||
| `--no-memory-context` | `false` | Disable automatic memory context injection |
|
||
| `--memory-top-k` | `10` | Number of memories to inject as context |
|
||
| `--learn` | `false` | Enable live traffic learning; implies `--memory` |
|
||
| `--no-learn` | `false` | Explicitly disable traffic learning |
|
||
| `--min-evidence` | `5` | Minimum observations before a learned pattern is persisted; `1` saves the first sighting |
|
||
| `--codex-wire-debug` | `false` | Write local Codex wire snapshots and matching proxy log traces |
|
||
| `HEADROOM_COMPRESS_PASSTHROUGH` | `0` | Also compress custom proxy paths that fall through to the catch-all handler (OpenAI Responses-shaped bodies, path ends in `/responses`). No `--compress-passthrough` flag on `headroom proxy`; the direct module entry point (`python -m headroom.proxy.server`) does accept one |
|
||
|
||
```bash
|
||
headroom proxy --memory
|
||
headroom proxy --learn --min-evidence 3
|
||
headroom proxy --codex-wire-debug
|
||
HEADROOM_COMPRESS_PASSTHROUGH=1 headroom proxy
|
||
```
|
||
|
||
Live learning includes observations received while a save is queued or in flight.
|
||
A pattern is marked as saved only after its evidence total is persisted. A full
|
||
queue, failed save or cancellation retains the accumulated evidence for the next
|
||
sighting and the shutdown snapshot.
|
||
|
||
With LocalBackend, recovery reuses a row that committed before indexing failed
|
||
instead of inserting another memory, and refreshes its indexes. Reconciliation
|
||
is idempotent even when a cancelled SQLite update finishes late.
|
||
|
||
Automatic memory lookup renders a separate, bounded embedding query:
|
||
|
||
| Input | Maximum characters |
|
||
| --- | ---: |
|
||
| Entire query, including delimiters | 4096 |
|
||
| Latest user message tail | 2048 |
|
||
| Each recent tool-output tail | 512 |
|
||
| Each recent assistant-turn tail | 512 |
|
||
|
||
These are source-size limits, not token estimates. The local ONNX and
|
||
sentence-transformers embedders then use their actual tokenizer and configured
|
||
input window, including special tokens. When both sources are present, they
|
||
reserve room for the question and context separately; unused space can go to
|
||
the other source. Newest tool evidence is at the end of the context so retaining
|
||
its tail does not silently discard it behind a long question.
|
||
|
||
Remote embedders that do not expose a local tokenizer retain the character
|
||
limits; these do not guarantee an arbitrary remote model's token window.
|
||
The raw `MemoryQuery` fields stay unchanged, and SDK callers that render without
|
||
a budget still receive the full sources.
|
||
|
||
Anthropic's pre-upstream lookup keeps its existing 2.0-second deadline,
|
||
configured by `ProxyConfig.anthropic_pre_upstream_memory_context_timeout_seconds`.
|
||
Bounding the input limits embedding work; it is not a guarantee against cold
|
||
model initialization or a slow embedding server. Memory failures include the
|
||
exception representation, so an otherwise empty timeout is identifiable as
|
||
`TimeoutError()`.
|
||
|
||
|
||
<Callout type="info" title="LLMLingua removed from the proxy CLI">
|
||
The old LLMLingua proxy toggles are no longer part of the CLI. Headroom's proxy compression path uses ContentRouter plus the current built-in compressors, including Kompress where applicable.
|
||
</Callout>
|
||
|
||
## Savings profiles
|
||
|
||
The proxy uses a **savings profile** to control compression behavior — which messages get compressed, how aggressively, and whether to prioritize provider prefix-cache stability or raw savings. Only the env var survives across related tools (`headroom wrap` passes it to the proxy it launches).
|
||
|
||
```bash
|
||
# Switch to a different profile
|
||
HEADROOM_SAVINGS_PROFILE=agent-90 headroom proxy
|
||
```
|
||
|
||
### Built-in profiles
|
||
|
||
| Profile | Target savings | proxy_mode | force_kompress | Best for |
|
||
|---------|---------------|------------|---------------|----------|
|
||
| `coding` (default) | ~50% of new input tokens (emergent) | `cache` | No | Coding agents — preserves Anthropic prefix-cache stability. Whole-request and lifetime ratios are much lower in cache mode. |
|
||
| `agent-90` | ~90% of new input tokens | `token` | Yes | Non-coding, cost-sensitive, or high-volume workloads |
|
||
| `balanced` | ~70% of new input tokens | `token` | No | General-purpose moderate compression |
|
||
| `general` | ~60% of new input tokens (emergent) | `token` | No | Non-coding chat, little code in context |
|
||
|
||
**`coding` (default)** — Optimizes for coding-agent workloads with Anthropic. Uses **cache mode** (`proxy_mode="cache"`): compresses only the newest delta in each turn so the provider's prefix-cache is never busted. User messages are compressed, system prompts preserved (hottest cache). `protect_recent` is `0` — in cache mode the delta already *is* the newest turn, so a turn-count guard would suppress the only thing cache mode compresses; byte-exact fidelity instead comes from `protect_reads=True` (file reads are never lossy-compressed). Lossless-first with lossy fallback; tool search and cross-turn dedup enabled. This is the profile that `headroom wrap` uses.
|
||
|
||
**`agent-90`** — Forces ML-based (Kompress) compression with a 10% keep-ratio, ignoring the lossless path. Compresses both user and system messages. Designed for non-coding or cost-sensitive workloads where maximum compression is the goal.
|
||
|
||
**`balanced`** — Token-mode compression with a 30% keep-ratio. Uses the standard lossless pipeline (does not force Kompress). Protects 4 recent turns. A safe general-purpose profile.
|
||
|
||
**`general`** — Token-mode compression for non-coding conversations. No turn protection (`protect_recent=0` — nothing code-positional to preserve) and does not compress user or system messages. Uses the standard lossless pipeline.
|
||
|
||
### Profile defaults and overrides
|
||
|
||
Profiles seed compression settings. For a direct `headroom proxy` invocation, explicit mode settings take precedence in this order: `--mode` (CLI) > `HEADROOM_MODE` (environment) > the `cache` default. The selected profile does not replace that final fallback, but wrapped launches may pre-seed `HEADROOM_MODE`. The `coding` profile defaults to `cache`, and an explicit `--mode token` still takes effect:
|
||
|
||
```bash
|
||
# Coding's default mode is cache
|
||
headroom proxy
|
||
# An explicit CLI mode overrides the profile default
|
||
headroom proxy --mode token
|
||
# An explicit environment mode also overrides the profile default
|
||
HEADROOM_MODE=token HEADROOM_SAVINGS_PROFILE=coding headroom proxy
|
||
```
|
||
|
||
You can also choose a profile whose default is token mode:
|
||
|
||
```bash
|
||
HEADROOM_SAVINGS_PROFILE=agent-90 headroom proxy --mode token
|
||
```
|
||
|
||
### Extending a profile with env overrides
|
||
|
||
Profile defaults are applied only when the corresponding env var is not already set. You can start from a named profile and override individual settings:
|
||
|
||
```bash
|
||
# Start from coding but force Kompress on
|
||
HEADROOM_SAVINGS_PROFILE=coding HEADROOM_FORCE_KOMPRESS=1 headroom proxy
|
||
|
||
# Start from balanced but lower the keep-ratio
|
||
HEADROOM_SAVINGS_PROFILE=balanced HEADROOM_TARGET_RATIO=0.15 headroom proxy
|
||
```
|
||
|
||
### Custom profiles
|
||
|
||
For permanent custom profiles, see the profile definitions in `headroom/agent_savings.py`. Each profile is an `AgentSavingsProfile` dataclass with fields for compression mode, target ratio, turn protection, and pipeline toggles.
|
||
|
||
## Serving on a Unix socket
|
||
|
||
`--uds` binds an `AF_UNIX` socket instead of a TCP port. Request handling is
|
||
identical — only the transport changes.
|
||
|
||
```bash
|
||
headroom proxy --uds ~/.headroom/run/proxy.sock
|
||
```
|
||
|
||
Two reasons to prefer it over a loopback port:
|
||
|
||
- **No port to collide with, and no port exposed.** Access is governed by
|
||
filesystem permissions rather than by anything listening on the network.
|
||
- **Deployments where a port is awkward.** Containers and systemd units can share a
|
||
socket through a bind mount or a runtime directory without publishing a port.
|
||
|
||
<Callout type="warning">
|
||
A socket does **not** re-enable Claude Code's Remote Control. Setting
|
||
`ANTHROPIC_UNIX_SOCKET` passes that feature's host check but makes Claude Code
|
||
treat the session as API-key auth, which fails a separate subscription-auth
|
||
requirement — see [Why the Unix socket route does not
|
||
help](/docs/troubleshooting#why-the-unix-socket-route-does-not-help).
|
||
</Callout>
|
||
|
||
Point any Unix-socket-capable client at the path. With curl:
|
||
|
||
```bash
|
||
curl --unix-socket ~/.headroom/run/proxy.sock http://localhost/health
|
||
```
|
||
|
||
### Permissions and lifecycle
|
||
|
||
A Unix socket carries no credentials of its own, so **the mode of the directory
|
||
holding it is the access-control boundary**.
|
||
|
||
If the parent directory does not exist, Headroom creates it `0700`. If it already
|
||
exists, Headroom does not touch its permissions — something else owns that
|
||
directory's policy, and silently tightening it could lock out whatever put it
|
||
there. Instead it refuses to start when an existing parent is writable by other
|
||
users and not sticky, since anyone on the host could then replace the socket.
|
||
(A sticky directory such as `/tmp` is accepted: others may create their own
|
||
entries there but cannot unlink or rename yours.)
|
||
|
||
On startup Headroom removes a stale socket left behind by a crashed process, but
|
||
it refuses to start when:
|
||
|
||
- something is still listening on the path (two proxies would silently split traffic),
|
||
- the path exists and is not a socket (far more likely a typo'd argument pointing
|
||
at real data than a leftover), or
|
||
- the path exceeds the platform's `sun_path` limit — 108 bytes on Linux, 104 on
|
||
macOS and the BSDs. Keep it short; a long path fails inside `bind()` with an
|
||
error that names nothing.
|
||
|
||
<Callout type="warn">
|
||
`--uds` is POSIX-only. On Windows the command exits immediately with an error
|
||
telling you to use `--port`; Python exposes no `socket.AF_UNIX` there and
|
||
asyncio has no Windows UDS transport.
|
||
</Callout>
|
||
|
||
## Configuration in depth
|
||
|
||
Proxy behavior is set by three layers, **each overriding the one before**:
|
||
|
||
1. **Savings profile** (`HEADROOM_SAVINGS_PROFILE`) — seeds a whole posture (mode, keep-ratio, which roles get compressed, Kompress on/off). Default `coding`. See [Savings profiles](#savings-profiles).
|
||
2. **Environment variables** — nearly every CLI flag has an `HEADROOM_*` twin, which is what you'll use in Docker, systemd, or CI.
|
||
3. **CLI flags** — the most explicit; they win over env and profile.
|
||
|
||
The tables below group the knobs by what they control. They aren't exhaustive (`headroom proxy --help` prints the full list), but they cover what real deployments actually touch. Unless noted, every option is off/unset by default and safe to ignore.
|
||
|
||
### Compression tuning
|
||
|
||
Fine-grained control over what gets compressed and how hard. Most users pick a [profile](#savings-profiles) instead and never touch these.
|
||
|
||
| Flag / env | Default | Effect |
|
||
|---|---|---|
|
||
| `--mode` / `HEADROOM_MODE` | `cache` | `cache` compresses only the newest delta (prefix-cache safe); `token` maximizes removal. |
|
||
| `--target-ratio` / `HEADROOM_TARGET_RATIO` | unset | Keep-ratio for ML text compression; lower = more aggressive (e.g. `0.10`). |
|
||
| `HEADROOM_MIN_TOKENS` | `500` | Minimum block size before a tool output is compressed. |
|
||
| `HEADROOM_SHELL_RESULT_MIN_CHARS` | `2000` | Shell tool results shorter than this many characters bypass lossy compression. For top-level text blocks, the threshold uses their combined text length. Byte-lossless folds such as search grouping may still apply. Applies to `bash`, `shell`, `local_shell`, and aliases such as `run_shell_command`/`execute_command`; set `0` to restore lossy compression. |
|
||
| `HEADROOM_COMPRESS_USER_MESSAGES` | profile | Compress text inside user-role messages. The `coding` profile turns this on; set `0` to keep user text verbatim under it. Anthropic `tool_result` blocks compress either way. No `--compress-user-messages` flag on `headroom proxy`; the direct module entry point (`python -m headroom.proxy.server`) does accept one. |
|
||
| `HEADROOM_COMPRESS_SYSTEM_MESSAGES` | unset | Compress system prompts. Off by default to keep the hottest cache prefix stable. |
|
||
| `HEADROOM_PROTECT_RECENT` | profile | Never compress the N most recent turns. |
|
||
| `--protect-tool-results` / `HEADROOM_PROTECT_TOOL_RESULTS` | empty | Comma-separated tool names whose output is never lossy-compressed. |
|
||
| `--compressor` (repeatable) / `HEADROOM_COMPRESSORS` | all | Restrict to specific compressors: `smart_crusher,kompress,code_aware,search,log,tabular,config,html,image`. Leaving out `image` also turns off the proxy's image stage. |
|
||
| `--code-aware` / `--no-code-aware` | off | AST-based [code compression](/docs/code-compression). Requires `headroom-ai[code]`. |
|
||
|
||
### Kompress (ML compression)
|
||
|
||
Kompress is the ModernBERT/ONNX compressor that ContentRouter falls back to for prose and unstructured text. It can run in-process or be offloaded to a hosted endpoint.
|
||
|
||
| Flag / env | Default | Effect |
|
||
|---|---|---|
|
||
| `--disable-kompress` / `HEADROOM_DISABLE_KOMPRESS` | `false` | Turn off ML compression; keep the structural compressors. |
|
||
| `--disable-kompress-anthropic` / `--disable-kompress-openai` | inherit | Per-provider override. |
|
||
| `HEADROOM_FORCE_KOMPRESS_ALL` | `false` | Route *all* content through Kompress, bypassing per-type selection. No `--force-kompress-all` flag on `headroom proxy`; the direct module entry point (`python -m headroom.proxy.server`) does accept one. |
|
||
| `HEADROOM_KOMPRESS_ENDPOINT` | none | Offload ML compression to a remote `/compress` endpoint (e.g. a Modal deployment) instead of running the model locally. |
|
||
| `HEADROOM_KOMPRESS_ENDPOINT_TOKEN` | none | Bearer token for the remote endpoint. |
|
||
| `HEADROOM_KOMPRESS_BACKEND` | `auto` | Compute backend: `auto`, `onnx_cpu`, `onnx_coreml`, `pytorch`, `pytorch_mps`. |
|
||
|
||
### Reversible compression (CCR) and lossless mode
|
||
|
||
By default Headroom stores originals so the model can recover them via `headroom_retrieve`. See [Reversible Compression](/docs/ccr).
|
||
|
||
| Flag / env | Default | Effect |
|
||
|---|---|---|
|
||
| `--no-ccr` / `HEADROOM_NO_CCR` | CCR on | Disable retrieval markers **and** the injected `headroom_retrieve` tool. |
|
||
| `--lossless` / `HEADROOM_LOSSLESS` | `false` | Format-native lossless compaction only — no CCR marker, no retrieval tool. |
|
||
| `--ccr-buffered-chat-streaming` / `HEADROOM_CCR_BUFFERED_CHAT_STREAMING` | `false` | Answer `headroom_retrieve` on streaming `/v1/chat/completions`. A turn carrying a redeemable marker runs as `stream: false` upstream and is re-emitted as `chat.completion.chunk` frames, so its text arrives all at once; other turns keep their real stream |
|
||
| `--ccr-inject-system-instructions` / `HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS` | `false` | Append compressed context instructions to the system message listing active retrieval hashes (disabled under `--no-ccr`/`--lossless`; skipped in cache mode once prefix freezes). |
|
||
| `--no-ccr-proactive-expansion` | expansion on | Stop proactively re-expanding compressed content when the model appears to need it. |
|
||
|
||
### File-read handling
|
||
|
||
Coding agents re-read the same files repeatedly; these control how stale reads are handled without busting the prefix cache.
|
||
|
||
| Flag / env | Default | Effect |
|
||
|---|---|---|
|
||
| `--no-read-lifecycle` | lifecycle on | Stop replacing stale/superseded file reads with CCR markers. |
|
||
| `--read-maturation` / `HEADROOM_READ_MATURATION` | `false` | *(Beta)* Keep freshly-read files verbatim while in use, then compress them once the file quiesces, unless the provider cache already holds them. Requires `HEADROOM_ROLLOUT_CHANNEL=beta` (or `dev`). |
|
||
| `--read-maturation-quiesce-turns` | `5` | Turns of no change before a held read is admitted. |
|
||
|
||
### Reliability: timeouts, retries, limits
|
||
|
||
| Flag / env | Default | Effect |
|
||
|---|---|---|
|
||
| `--request-timeout-seconds` / `HEADROOM_REQUEST_TIMEOUT` | `300` | Upstream request timeout (seconds). |
|
||
| `--connect-timeout-seconds` | `10` | Upstream connect timeout (seconds). |
|
||
| `--retry-max-attempts` | `3` | Upstream retries on transient failure. |
|
||
| `--limit-concurrency` | `1000` | Concurrent connections before returning 503. |
|
||
| `--rpm` / `--tpm` | `60` / unlimited | Requests- and tokens-per-minute rate limits (disable with `--no-rate-limit`). |
|
||
| `--budget` / `--budget-period` | none / `daily` | Spend cap in USD per period; over-budget requests get 429. |
|
||
| `--workers` / `HEADROOM_WORKERS` | `1` | Uvicorn worker processes. |
|
||
|
||
### Tool search and MCP
|
||
|
||
Defers large tool schemas so they don't sit in every request. See [MCP](/docs/mcp).
|
||
|
||
| Env | Scope | Effect |
|
||
|---|---|---|
|
||
| `HEADROOM_TOOL_SEARCH` | proxy (server-side) | Defer MCP/system tool schemas behind a `search_tools` tool. **On by default** for Anthropic requests carrying enough tools to be worth it; set `HEADROOM_TOOL_SEARCH=0` to opt out. |
|
||
| `ENABLE_TOOL_SEARCH` | client (Claude Code) | Keep Claude Code's own deferred tool-loading active behind a custom base URL ([issue #746](https://github.com/headroomlabs-ai/headroom/issues/746)). Set automatically by `headroom wrap`. |
|
||
|
||
### Cost-aware model routing
|
||
|
||
Rewrite the upstream model per request — for example, send small, tool-free calls to a cheaper model. Opt-in and off by default. Configure with `HEADROOM_MODEL_ROUTER_ENABLED` plus `HEADROOM_MODEL_ROUTES`; see [Cost-aware model routing](/docs/configuration#cost-aware-model-routing).
|
||
|
||
### Observability
|
||
|
||
| Flag / env | Default | Effect |
|
||
|---|---|---|
|
||
| `--telemetry` / `HEADROOM_TELEMETRY` | off | **Local-only** usage stats for your own `/stats`, `/metrics`, and dashboard. Nothing leaves the machine. |
|
||
| `--log-file` / `HEADROOM_LOG_FILE` | none | JSONL request/response log. |
|
||
| `--log-messages` | `false` | Include full message bodies in the log (may contain sensitive data). |
|
||
| `HEADROOM_LOG_PAYLOAD_PREVIEW` | `off` | Include up to 4 KB of the retrieved [CCR](/docs/ccr) payload in each `headroom_retrieve` line of the runtime log (`~/.headroom/logs/proxy-<port>.log`). Off by default: the payload is verbatim tool-result content and can carry source code, credentials the redactor does not recognize, or customer data; the log otherwise records byte counts only. Turn it on with `1`/`true`/`yes`/`on`. Do not share a log captured with this on. |
|
||
| `HEADROOM_LOG_LEVEL` | `warning` | uvicorn's log level (`critical`, `error`, `warning`, `info`, `debug`, `trace`). Raise to `info` for the per-request access log when diagnosing a deployed proxy. An unrecognized value warns and falls back to `warning`. |
|
||
| `HEADROOM_OTEL_METRICS_ENABLED` | `false` | Export OpenTelemetry metrics (`HEADROOM_OTEL_METRICS_ENDPOINT`, …). See [OTLP export](/docs/metrics#opentelemetry-otlp-export). |
|
||
| `HEADROOM_LANGFUSE_ENABLED` | `false` | Emit Langfuse traces (`LANGFUSE_PUBLIC_KEY` / `LANGFUSE_SECRET_KEY`). |
|
||
|
||
See [Metrics](/docs/metrics) for the Prometheus and Grafana setup.
|
||
|
||
#### Who can read the logs
|
||
|
||
Headroom writes the runtime log (`~/.headroom/logs/proxy-<port>.log`) and, when
|
||
`--log-file` is set, the JSONL request log. Both can hold request and response
|
||
content — `--log-messages` bodies, wire-debug dumps, and CCR payload previews
|
||
when they are enabled.
|
||
|
||
- **Linux and macOS:** every log file Headroom creates is `0600`, owner-read/write
|
||
only, and so is every rotated backup (`proxy-8000.log.1`, …). This is not
|
||
conditional on which content switch is on. If the log path already exists as a
|
||
symlink, Headroom refuses to log rather than write through it, and says so.
|
||
- **Windows:** Headroom restricts the log file too, through an NTFS ACL —
|
||
Python has no ACL API, so this goes directly through `ctypes` rather than a
|
||
third-party dependency. The DACL it sets grants access to exactly the
|
||
current user, replacing (not merging with) whatever was inherited from the
|
||
parent directory. This can fail for reasons unrelated to Headroom's own
|
||
code — a non-NTFS volume, a missing privilege — so the proxy re-reads the
|
||
file afterward to confirm, and only logs a warning at startup if that
|
||
check fails. If you see that warning, keep `%USERPROFILE%\.headroom` (or
|
||
your chosen `--log-file` path) off shared volumes and out of directories
|
||
with broadened permissions.
|
||
|
||
A log shipper or sidecar running as a different user or group cannot read the
|
||
runtime log on Linux/macOS. Run the collector as the same user, or copy the file
|
||
with a privileged job.
|
||
|
||
|
||
### Security and networking
|
||
|
||
| Flag / env | Default | Effect |
|
||
|---|---|---|
|
||
| `HEADROOM_PROXY_TOKEN` | none | Require a token from non-loopback callers, as `X-Headroom-Proxy-Token: <token>` or `Authorization: Bearer <token>`. The token is removed from the request before it is forwarded upstream; clients that also send a provider credential should use `X-Headroom-Proxy-Token` and keep `Authorization` for the provider. Also gates the read-only operator routes (`/stats-history`, `/quota`, `/subscription-window`, `/metrics`), which otherwise answer loopback and trusted-CIDR callers only. A token holder also sees the per-request metadata `/stats` and `/stats-lifetime` withhold from other network callers. That metadata covers every caller of the proxy, so only share the token between clients that may see each other's traffic metadata; see [What the proxy token unlocks](/docs/security-model#what-the-proxy-token-unlocks). |
|
||
| `HEADROOM_ALLOW_UNAUTHENTICATED_BIND` | `false` | Acknowledge a non-loopback bind (`--host 0.0.0.0`, `::`, a LAN address) with **no** `HEADROOM_PROXY_TOKEN`. Without a token or this flag the proxy refuses to start, because `/v1/*` would relay upstream with the operator's credentials for every peer that can reach the port. Only set it when the runtime already restricts reachability (a container published on `127.0.0.1`). |
|
||
| `HEADROOM_COMPRESS_ALLOW_REMOTE` | `false` | Allow non-loopback callers to reach [`POST /v1/compress`](#post-v1compress) and `POST /v1/usage`. Required to run Headroom as a remote gateway/sidecar; without it remote callers get `404`. |
|
||
| `HEADROOM_OFFLINE` | `false` | Air-gap mode: refuse every connection Headroom itself initiates — telemetry, update checks, license reporting, model/tokenizer/binary/dataset downloads, litellm's pricing-table refresh, CrewAI's install tracking, remote Kompress, OTLP and Langfuse export, Copilot auth, the subscription pollers, the OpenAI embedders, Headroom Cloud compression and the TLS diagnostics (`headroom doctor --network` reports them as skipped). Four things still work on purpose: forwarding your own requests to the upstream you configured, operator-configured local endpoints (the Ollama embedder), loopback health probes, and paths an `is_offline()` check already disables. Refusals surface as messages, not tracebacks. See the air-gap notes in `docs/metrics-technical-guide.md`. |
|
||
| `HEADROOM_LICENSE` | unset | Licence token, read by core and every licensed extension. `HEADROOM_LICENSE_KEY` is a deprecated alias for one release and logs a warning. A licence on its own sends nothing to Headroom. |
|
||
| `HEADROOM_USAGE_REPORTING` | `false` | Set `1` to let the proxy validate the licence and post aggregate usage counts (no content) to the Headroom cloud. Off unless set; `HEADROOM_OFFLINE` still overrides it. |
|
||
| `--stateless` / `HEADROOM_STATELESS` | `false` | Keep all state in memory; no filesystem writes (disables logs, memory, TOIN). |
|
||
| `HEADROOM_STRIP_INTERNAL_HEADERS` | `enabled` | Strip internal `x-headroom-*` headers before forwarding upstream. |
|
||
| `HEADROOM_TLS_STRICT` | strict | Set `0` to relax CA-constraint checks behind a corporate TLS-inspection proxy. |
|
||
| `HEADROOM_CERT_STORE` | `system,bundled` | Trusted roots for upstream TLS: OS certificate store and/or certifi. See [Corporate Networks](/docs/corporate-networks). |
|
||
| `HEADROOM_CA_BUNDLE` | unset | Extra PEM roots (e.g. a Zscaler root) added to the trusted set. |
|
||
|
||
### Performance
|
||
|
||
| Flag / env | Default | Effect |
|
||
|---|---|---|
|
||
| `--embedding-server` / `HEADROOM_EMBEDDING_SERVER` | off | Share one ONNX embedder across workers (~600 MB RSS saved). |
|
||
| `--compression-max-workers` / `HEADROOM_COMPRESSION_MAX_WORKERS` | CPU count | Bound the CPU-bound compression threadpool. |
|
||
|
||
<Callout type="info" title="Full JSON config">
|
||
For programmatic deployment you can pass an entire proxy config as JSON via `HEADROOM_PROXY_CONFIG_JSON`, or point `HEADROOM_CONFIG_DIR` / `HEADROOM_WORKSPACE_DIR` at custom roots (see [Filesystem Contract](/docs/filesystem-contract)).
|
||
</Callout>
|
||
|
||
## API endpoints
|
||
|
||
### `GET /health`
|
||
|
||
```bash
|
||
curl http://localhost:8787/health
|
||
```
|
||
|
||
```json
|
||
{
|
||
"status": "healthy",
|
||
"optimize": true,
|
||
"stats": {
|
||
"total_requests": 42,
|
||
"tokens_saved": 15000,
|
||
"savings_percent": 45.2
|
||
}
|
||
}
|
||
```
|
||
|
||
### `GET /stats`
|
||
|
||
Live session statistics plus durable `persistent_savings` totals. Stored at `~/.headroom/proxy_savings.json` (override with `HEADROOM_SAVINGS_PATH`).
|
||
|
||
```bash
|
||
curl http://localhost:8787/stats
|
||
```
|
||
|
||
### `GET /stats-history`
|
||
|
||
Durable history with hourly, daily, weekly, and monthly rollups. Powers the `/dashboard` view.
|
||
|
||
```bash
|
||
curl http://localhost:8787/stats-history
|
||
curl "http://localhost:8787/stats-history?format=csv&series=weekly"
|
||
```
|
||
|
||
### `GET /metrics`
|
||
|
||
Prometheus-format metrics for monitoring.
|
||
|
||
```bash
|
||
curl http://localhost:8787/metrics
|
||
```
|
||
|
||
```
|
||
headroom_requests_total{mode="optimize"} 1234
|
||
headroom_tokens_saved_total 5678900
|
||
headroom_persistent_savings_tokens_saved_total 5678900
|
||
headroom_compression_ratio_bucket{le="0.5"} 890
|
||
headroom_latency_seconds_bucket{le="0.01"} 800
|
||
headroom_cache_hits_total 456
|
||
```
|
||
|
||
`headroom_tokens_saved_total` is the runtime counter for the current proxy process. Use `headroom_persistent_savings_tokens_saved_total` for durable lifetime savings that match `/stats.persistent_savings`.
|
||
|
||
### `POST /v1/messages`
|
||
|
||
Anthropic API format. The proxy compresses messages, forwards to Anthropic, and returns the response.
|
||
|
||
### `POST /v1/chat/completions`
|
||
|
||
OpenAI API format. The proxy compresses messages, forwards to OpenAI, and returns the response.
|
||
|
||
On translated backends (`anyllm`, `litellm-<provider>`), streaming token accounting uses the latest provider usage snapshot. Providers may repeat a cumulative `usage` block on several content chunks before the terminal frame; the final counts win. A frame that omits a field leaves the earlier value alone, while an explicitly reported zero replaces it — so `/stats` and the cache-hit-rate metrics report what the provider last said, not the first nonzero count it sent.
|
||
|
||
### `POST /v1/responses`
|
||
|
||
OpenAI Responses API format. The proxy compresses `input` payloads where applicable, forwards the request, and returns the response.
|
||
|
||
For Codex-compatible clients, the proxy also accepts these alias paths and routes them through the same handler:
|
||
- `POST /v1/codex/responses`
|
||
- `POST /backend-api/responses`
|
||
- `POST /backend-api/codex/responses`
|
||
|
||
Matching WebSocket and subpath aliases are also supported for Codex flows.
|
||
|
||
### Codex Live voice WebSocket
|
||
|
||
The proxy relays Codex Live voice frames without parsing or transforming them.
|
||
These paths use the same transparent transport:
|
||
|
||
- `ws://localhost:8787/v1/live`
|
||
- `ws://localhost:8787/v1/codex/live`
|
||
- `ws://localhost:8787/backend-api/live`
|
||
- `ws://localhost:8787/backend-api/codex/live`
|
||
|
||
Subscription authentication uses the derived ChatGPT backend path. API-key
|
||
authentication preserves the selected OpenAI-compatible base URL and inbound
|
||
path. The backend Live suffix defaults to `/live` and can be corrected with
|
||
`HEADROOM_CODEX_LIVE_WS_PATH` if the upstream contract changes. The exact
|
||
ChatGPT backend path is not confirmed by this proxy documentation.
|
||
|
||
### `POST /v1internal:streamGenerateContent`
|
||
|
||
Google Cloud Code Assist / Antigravity compatibility endpoint used by Pi-style `google-gemini-cli` and `google-antigravity` providers.
|
||
|
||
The proxy also accepts:
|
||
- `POST /v1/v1internal:streamGenerateContent`
|
||
|
||
### Antigravity IDE
|
||
|
||
Google's Antigravity IDE can route its model traffic through the proxy:
|
||
|
||
```bash
|
||
headroom wrap antigravity # Start proxy + print Antigravity setup instructions
|
||
headroom mcp install --agent antigravity # Register the Headroom MCP server in Antigravity
|
||
```
|
||
|
||
The Antigravity IDE does not support third-party model providers — its
|
||
agent models are Google-managed, so there is no settings panel to point
|
||
at the proxy. What the proxy does offer Antigravity users:
|
||
|
||
- MCP tools: `mcp install` writes the Headroom server into Antigravity's
|
||
global `mcp_config.json` under `~/.gemini/` (preferring the install's
|
||
current location when one already exists) and refuses to overwrite a
|
||
config file that does not parse as JSON.
|
||
- Native traffic: `google-antigravity` clients are accepted on
|
||
[`POST /v1internal:streamGenerateContent`](#post-v1internalstreamgeneratecontent)
|
||
and routed upstream like any other Cloud Code request.
|
||
|
||
The Antigravity CLI (`agy`), unlike the IDE, accepts a custom endpoint
|
||
via its `GOOGLE_GEMINI_BASE_URL` setting — the one Antigravity surface
|
||
where a custom model endpoint is supported.
|
||
|
||
### `POST /v1/compress`
|
||
|
||
Compression-only endpoint. Compresses messages and returns them without ever making a **completion request to an LLM provider** — no generation, no provider API key, no upstream chat call. Used by the TypeScript SDK, by LiteLLM's `headroom` guardrail, and by API gateways running Headroom as a sidecar.
|
||
|
||
<Callout type="info" title="It does run local ML models">
|
||
"No LLM call" means no *generative* request to a provider. Compression itself is ML-backed: **Kompress** is a ModernBERT encoder that scores tokens for retention (classification, not generation), and Magika classifies content types. Both run in-process by default, so budget CPU and memory for the sidecar accordingly.
|
||
|
||
If `HEADROOM_KOMPRESS_ENDPOINT` is set, Kompress inference is offloaded over HTTP to that model server — **real egress from the sidecar**, which matters if you deployed it expecting none. Only inference goes remote: the CCR store and retrieval markers stay proxy-local, and original content never persists off-box. Leave the variable unset to keep everything in-process, or run with `HEADROOM_DISABLE_KOMPRESS=1` for structural compression only.
|
||
</Callout>
|
||
|
||
<Callout type="warning" title="Loopback-only by default">
|
||
This route is restricted to loopback callers and answers everyone else with **`404`**, not `403` — deliberately, so it stays invisible to external scanners. A gateway calling it from another host or pod therefore sees what looks like a missing route.
|
||
|
||
Both the client IP and the inbound `Host:` header must name loopback. In container environments (Docker/Podman), requests forwarded from the container host gateway are also admitted under loopback `Host:` headers. To allow arbitrary remote callers, set `HEADROOM_COMPRESS_ALLOW_REMOTE=1`. `HEADROOM_PROXY_TOKEN` still applies if set. The same policy applies to [`POST /v1/usage`](#relaying-provider-usage-post-v1usage).
|
||
</Callout>
|
||
|
||
#### Message format
|
||
|
||
The endpoint does **no format conversion**. Whatever shape you send in `messages` is the shape you get back, and both wire formats are compressed natively:
|
||
|
||
- **OpenAI shape** — `role: "tool"` messages with `tool_call_id`, assistant `tool_calls`
|
||
- **Anthropic shape** — content-block lists with `tool_use` / `tool_result` / `thinking` blocks
|
||
|
||
So an Anthropic-native caller does not need to convert to OpenAI format first. Block types, `tool_use_id`s and message order are all preserved.
|
||
|
||
`model` selects the tokenizer (per-model, from Headroom's tokenizer registry) and the context limit. Send the real model name — including gateway-prefixed forms like `bedrock/anthropic.claude-3-5-sonnet` or `vertex_ai/claude-sonnet-4@20250514` — so token counts and compression aggressiveness are right.
|
||
|
||
#### Request
|
||
|
||
| Field | Type | Required | Description |
|
||
| --- | --- | --- | --- |
|
||
| `messages` | array | yes | Messages to compress, in either wire format. `400` if missing. Empty array returns immediately with zero metrics. |
|
||
| `model` | string | yes | Model name. Drives tokenizer + context-limit resolution. `400` if missing. |
|
||
| `token_budget` | integer | no | Overrides the model's context limit. Used by callers that need to fit a tighter budget. |
|
||
| `config` | object | no | Compression options, below. A non-object value is ignored rather than rejected. |
|
||
| `gateway` | object | no | Switches on [gateway mode](#gateway-mode-two-half-turn-contract). May be `{}`. Fields below. |
|
||
|
||
<Callout type="warning" title="system and tools are ignored outside gateway mode">
|
||
Without a `gateway` block, only the four fields above are read. Anthropic sends `system` and `tools` **out of band**, alongside `messages` — this endpoint accepts them without complaint (you get a `200`, no warning) and returns neither, so neither is compressed.
|
||
|
||
Keep carrying both yourself and send them upstream unchanged. Two consequences worth knowing:
|
||
|
||
- An Anthropic system prompt is not compressed here, even though it is resent on every request.
|
||
- Tool-schema compaction, tool-search deferral and output steering are not reachable this way — on tool-heavy traffic those can be the largest share of available savings. [Gateway mode](#gateway-mode-two-half-turn-contract) passes `tools` and `system` through and applies all three.
|
||
</Callout>
|
||
|
||
`config` fields:
|
||
|
||
| Field | Type | Default | Description |
|
||
| --- | --- | --- | --- |
|
||
| `mode` | string | unset | `ccr`, `lossy_inline`, `lossless_then_lossy`, or `lossless`. Unset selects the default marker-free pipeline. Any other value is a `400`. |
|
||
| `frozen_message_count` | integer | unset | Pin a prefix: the first N messages are returned byte-for-byte unchanged while staying visible to cross-message transforms like dedup. Set it to the number of messages the provider has already cached so compression cannot rewrite the prefix and bust that cache. Must be a non-negative integer; anything else is a `400`. |
|
||
| `session_id` | string | unset | Opt into [session mode](#session-mode): Headroom keeps this conversation's replay state itself, so you can resend the raw conversation every turn and get a byte-identical prefix back. Non-empty, at most 256 characters; anything else is a `400`. Cannot be combined with `compress_user_messages` (`400`). |
|
||
| `compress_user_messages` | boolean | `false` | Also compress user-role messages. |
|
||
| `target_ratio` | number | unset | Target compression ratio. |
|
||
| `protect_recent` | integer | unset | Leave recent code uncompressed in the last N messages (non-code tool payloads are not shielded). |
|
||
| `protect_analysis_context` | boolean | unset | Preserve analysis context blocks. |
|
||
|
||
**`config.mode` values:**
|
||
|
||
- **unset (default)** — marker-free. Emits no `<<ccr:…>>` retrieval markers and writes nothing to the CCR store, so you can forward the returned messages straight to a provider. This is the right mode for a gateway or guardrail that just swaps `messages` and forwards. A file read that a later edit made stale is still replaced, with a note telling the model to re-read the file and no retrieval hash.
|
||
- **`ccr`** — emits CCR markers and writes to the store. Only for callers that also inject the `headroom_retrieve` tool *and* can reach `/v1/retrieve` (itself loopback-only). Markers are a dangling pointer for the model otherwise.
|
||
- **`lossy_inline`** (alias `lossless_then_lossy`) — runs the lossless byte/data fold first, then compresses the folded remainder. Marker-free.
|
||
- **`lossless`** — format-native folds only, with no markers and no retrieval. For a sidecar with no retrieval loop; a registered lossless provider still applies. Keep the mode stable within a `session_id`, because session mode replays earlier turns byte-for-byte.
|
||
|
||
#### Response
|
||
|
||
| Field | Type | Description |
|
||
| --- | --- | --- |
|
||
| `messages` | array | Compressed messages, in the shape you sent. |
|
||
| `tokens_before` | integer | Token count before compression. |
|
||
| `tokens_after` | integer | Token count after compression. |
|
||
| `tokens_saved` | integer | `tokens_before - tokens_after`. |
|
||
| `compression_ratio` | number | `tokens_after / tokens_before` — so **lower is better**. A ratio of `0.23` means a 77% reduction, not 23%. `1.0` when nothing was compressed. |
|
||
| `transforms_applied` | array | Transform labels that ran. |
|
||
| `transforms_summary` | object | Per-transform counts. |
|
||
| `ccr_hashes` | array | Retrieval hashes for markers inserted (empty unless `mode: "ccr"`). |
|
||
| `session` | object | [Session mode](#session-mode) only: `{ "id", "frozen_message_count", "cached_prefix_replayed" }`, described there. |
|
||
|
||
```json
|
||
{
|
||
"messages": [{ "role": "user", "content": "..." }],
|
||
"tokens_before": 15000,
|
||
"tokens_after": 3500,
|
||
"tokens_saved": 11500,
|
||
"compression_ratio": 0.23,
|
||
"transforms_applied": ["router:smart_crusher:0.35"],
|
||
"transforms_summary": { "router:smart_crusher:0.35": 1 },
|
||
"ccr_hashes": []
|
||
}
|
||
```
|
||
|
||
#### Headers
|
||
|
||
`x-headroom-bypass: true` (case-insensitive) skips compression entirely and echoes your messages back with zeroed metrics. The bypass and empty-messages responses omit `transforms_summary`.
|
||
|
||
#### Errors and fail-open
|
||
|
||
| Status | Body | When |
|
||
| --- | --- | --- |
|
||
| `400` | `error.type = "invalid_request"` | Missing `messages` or `model`, malformed JSON, invalid `config.mode`, `config.frozen_message_count` or `config.session_id`, `session_id` combined with `compress_user_messages`, or a malformed `gateway` block. |
|
||
| `401` | — | `HEADROOM_PROXY_TOKEN` is set and the bearer token is missing or wrong. |
|
||
| `404` | — | Non-loopback caller without `HEADROOM_COMPRESS_ALLOW_REMOTE=1`. |
|
||
| `503` | `error.type = "compression_error"` | Compression failed unexpectedly. |
|
||
| `503` | `error.type = "compression_timeout"` | Session mode only: the turn timed out or the session's lock was busy. Retry the turn. |
|
||
|
||
Without a session, compression **fails open on timeout**: you get `200` with your original messages, zeroed metrics, plus `compression_skipped: true` and `skip_reason: "compression_timeout"`. Always check `compression_skipped` if you need to know whether compression actually ran. In session mode the same timeout is a `503` instead — see below for why.
|
||
|
||
Requests are recorded under `provider="compress"` in `/stats` and `/metrics`.
|
||
|
||
#### Multi-turn usage: keeping the prefix cache
|
||
|
||
This is the single most important thing to get right, and the default is not safe for an agent loop.
|
||
|
||
When Headroom proxies a request itself it watches the provider's cache hit rate turn over turn and freezes the already-cached prefix. Without `config.session_id`, `/v1/compress` cannot do that: it sees one isolated call and has no idea what the provider already cached. You have two options — let Headroom keep the state ([session mode](#session-mode), below) or keep it yourself with `frozen_message_count`, which the rest of this section covers.
|
||
|
||
The provider caches the bytes you **forwarded**. Compression changed those bytes, so your original messages and the ones the provider cached are no longer the same thing — and it is the forwarded version you have to keep reproducing. Send the pristine originals again next turn and the provider sees a different prefix and re-reads it from scratch. On Anthropic a cache read is ~90% cheaper than fresh input, so that can easily cost more than the compression saves.
|
||
|
||
Compression is also not uniform over a conversation: how hard a message is compressed depends partly on how far it now sits from the end, so an older tool result can fall outside the recent-read protection window as the conversation grows and be compressed harder than it was last turn. Another reason not to rely on re-compression reproducing earlier output.
|
||
|
||
Two rules:
|
||
|
||
1. **Pass `config.frozen_message_count`** — how many leading messages the provider has already cached.
|
||
2. **Send back your own previous output, not the original messages.** `frozen_message_count` returns those leading messages *exactly as you passed them in* — it pins whatever you hand it. Hand it pristine originals and you get pristine originals back, which is precisely the prefix the provider does not have.
|
||
|
||
```python
|
||
# Keep what you FORWARDED, not what you started with.
|
||
forwarded: list[dict] = []
|
||
|
||
def next_turn(new_messages: list[dict]) -> list[dict]:
|
||
body = {
|
||
"messages": forwarded + new_messages,
|
||
"model": "claude-sonnet-4-6",
|
||
# Everything already forwarded is already cached upstream — pin it.
|
||
"config": {"frozen_message_count": len(forwarded)},
|
||
}
|
||
result = requests.post(f"{proxy}/v1/compress", json=body).json()
|
||
forwarded[:] = result["messages"] # becomes next turn's frozen prefix
|
||
return forwarded
|
||
```
|
||
|
||
<Callout type="warning" title="Re-sending pristine messages every turn silently busts the cache">
|
||
Compressing the full original conversation on each turn looks correct — you get a `200` and a positive `tokens_saved` — but the leading messages come back different from the ones the provider cached. You pay for compression *and* for a cache miss. Nothing in the response tells you this happened; watch your provider's cache-read tokens.
|
||
</Callout>
|
||
|
||
Also for multi-turn callers:
|
||
|
||
- **Leave `config.mode` unset.** The default is marker-free, which is what a forward-only caller wants.
|
||
- **Send the real model name** so the tokenizer and context limit resolve correctly — including gateway-prefixed forms.
|
||
- **`protect_recent` is not a substitute.** It guards the newest messages; `frozen_message_count` guards the oldest, which is the cached end.
|
||
|
||
#### Session mode
|
||
|
||
Pass `config.session_id` and Headroom keeps the conversation's replay state itself — the same per-session compression cache and prefix tracker the proxy path uses — so a gateway that owns routing can resend the **raw** conversation every turn and still get a byte-identical prefix back. Whatever this endpoint previously returned for the session is treated as the provider's cache contract: every already-returned message comes back unchanged, even where recompressing it would do better, and only the new tail is compressed. Message `cache_control` markers stay where you put them. If you send none, none come back. An explicit `config.frozen_message_count` still wins when it is larger.
|
||
|
||
```json
|
||
{
|
||
"messages": [{ "role": "user", "content": "..." }],
|
||
"model": "claude-sonnet-4-6",
|
||
"config": { "session_id": "conv-8f1c" }
|
||
}
|
||
```
|
||
|
||
The response gains a `session` object:
|
||
|
||
```json
|
||
"session": { "id": "conv-8f1c", "frozen_message_count": 12, "cached_prefix_replayed": false }
|
||
```
|
||
|
||
| Field | Type | Description |
|
||
| --- | --- | --- |
|
||
| `id` | string | The session id this turn ran under: `config.session_id` as sent, or the `x-headroom-session-id` header when `HEADROOM_COMPRESS_SESSION_FROM_HEADER=1` and `config.session_id` is absent. |
|
||
| `frozen_message_count` | integer | How many leading messages the pipeline was told not to recompress this turn. Headroom counts the leading run it can replay from the session's compression cache: plain messages always count, a tool result counts only once its content is in that cache, and the last message never counts. An explicit `config.frozen_message_count` wins when larger. It is derived from Headroom's own state, not from the provider, and it can be non-zero on a first turn. |
|
||
| `cached_prefix_replayed` | boolean | `true` when the pipeline's output for some leading messages differed from what this endpoint returned for the session last turn, and Headroom overwrote those messages with last turn's bytes. `false` when nothing was overwritten: when the serving process holds no previous turn for the session (a first turn, expired state, or a replica that never served or loaded it), when the pipeline already produced last turn's bytes (the usual case), or when the replay was skipped as unsafe (this turn's messages do not line up with last turn's, or the replay would be larger than the pipeline's output). So `false` does not mean the returned prefix matches last turn's. |
|
||
|
||
<Callout type="warning" title="cached_prefix_replayed is not a prompt-cache hit">
|
||
`cached_prefix_replayed` reports Headroom's own repair step. It says nothing about whether the provider served the prefix from its prompt cache. `false` is the normal value on a healthy session, and it is also what a turn that changed the prefix returns when there was nothing to replay. Read cache hits and misses from the provider's usage (`cache_read_input_tokens`, `prompt_tokens_details.cached_tokens`).
|
||
</Callout>
|
||
|
||
What differs from the stateless contract:
|
||
|
||
- **Forward the returned messages verbatim.** They are what the next turn replays.
|
||
- **`compress_user_messages` is refused** (`400`). User-message rewrites are not byte-replayable across turns, so they would bust the prefix later.
|
||
- **Timeouts and busy sessions are a `503`, never fail-open.** Handing back originals would leave the caller forwarding bytes the replay state does not know about, so a compression timeout answers `503` with `error.type = "compression_timeout"`; retry the turn and it lands on consistent state. Turns for one session are serialized under a per-session lock, and a concurrent turn for the same session that waits more than 10 seconds gets the same `503`.
|
||
- **State is in-process by default.** The replay cache lives for `HEADROOM_COMPRESSION_CACHE_TTL_SECONDS` of idle time (3900 by default, floor 600); the tracker's usage state expires after 10 idle minutes. Without a shared store, multi-process deployments (gunicorn workers, replicas) must pin a session to one process — hash the upstream on the session id — or replay silently misses.
|
||
- **`HEADROOM_COMPRESS_SESSION_STORE=redis://...` shares sessions across replicas.** Every replica also keeps each session's tracker in Redis, so any replica can serve the session's next turn and its `POST /v1/usage`. It needs the `redis` extra, which every Docker image includes. It also selects the Redis CCR store unless `HEADROOM_CCR_BACKEND` names another, so CCR markers resolve on any replica. Two limits remain. The per-session lock is per process: two concurrent turns of one session on two replicas both run, the first write wins (compare-and-set), and the next turn continues from the winner. Gateway-mode pending turns stay in-process, so `POST /v1/compress/response` must reach the replica that answered the request half; re-drive therefore still needs real affinity (`gateway.session_affinity`).
|
||
- **An unreachable Redis does not fail turns.** A Redis error is a miss: the replica serves from its own memory, logs one `WARNING` when a kind of call (read, write) starts failing and an `INFO` when it works again, and skips Redis for 5 seconds after a connection error (each command times out after 1 second). The cost is the sharing. A replica with no local state for the session recompresses the history, which is slower, and the prefix it returns can differ from last turn's, which is a provider prompt-cache miss. `cached_prefix_replayed` stays `false` on those turns. A Redis that comes back empty is refilled as each session's next turn writes it whole. Run Redis with high availability if prompt-cache hits matter.
|
||
- `HEADROOM_COMPRESS_SESSION_FROM_HEADER=1` lets the `x-headroom-session-id` header stand in for `config.session_id`. Off by default: gateways commonly stamp that header on all traffic, and a value shared across conversations (Claude Code subagents do this) would blend them into one replay state.
|
||
|
||
**Relaying provider usage: `POST /v1/usage`.** Session mode never sees the provider's response, so cache-hit attribution and savings accounting for the session come from this optional relay (same exposure rules as `/v1/compress`). Post the provider's `usage` block under the session id:
|
||
|
||
```json
|
||
{
|
||
"session_id": "conv-8f1c",
|
||
"usage": { "cache_read_input_tokens": 11800, "cache_creation_input_tokens": 400 }
|
||
}
|
||
```
|
||
|
||
OpenAI-shaped usage is accepted too: `prompt_tokens_details.cached_tokens` or a flat `cached_tokens` is read as the cache-read count (OpenAI reports no write counter). A block carrying none of these fields is a `400` — it has no cache signal, and reading it as zeros would wipe the tracker's cached-prefix state.
|
||
|
||
The reply is `{ "session_id": "conv-8f1c", "frozen_message_count": 12, "applied": true }`. `applied: false` with `reason: "no_cache_signal"` means the only present field was `0` (a fully cold turn can only be asserted with both fields present). `404` with `error.type = "unknown_session"` means the tracker's state for that session has expired — replay may still be active; only this telemetry landed nowhere. `503` with `error.type = "session_busy"` means a compress turn for the session is in flight; retry.
|
||
|
||
The relay is telemetry only. It never raises the frozen count, because a provider-confirmed count could freeze a message whose cache entry has since been evicted. Skipping it costs attribution, never correctness.
|
||
|
||
#### Gateway mode (two-half turn contract)
|
||
|
||
An API gateway that owns routing (Kong, LiteLLM, a custom sidecar) never lets Headroom see the provider's response, which rules out every transform whose reload step needs it. Gateway mode splits one model turn into two HTTP calls: the **request half** (`POST /v1/compress` with a top-level `gateway` object) returns the complete provider body plus the *obligations* the gateway must redeem, and the **response half** (`POST /v1/compress/response`) takes the provider's answer and either finishes the turn or asks the gateway to call the provider again.
|
||
|
||
The rule behind it is the proxy's own **no shrink without reload**, moved to the API boundary: a transform whose reload step needs the response only runs when the gateway has said it can re-drive. A body without a `gateway` key never touches any of this — stateless and session behaviour are unchanged.
|
||
|
||
**The `gateway` block.** Presence is the switch: `"gateway": {}` is gateway mode with every default. Unknown keys are ignored (a newer plugin can talk to an older Headroom); a non-object block or a non-boolean field is a `400`.
|
||
|
||
| Field | Type | Default | Description |
|
||
| --- | --- | --- | --- |
|
||
| `can_redrive` | boolean | `false` | The gateway can call the provider again with a request Headroom hands it and post the result back. Unlocks re-driving turn hooks and, with `config.mode: "ccr"`, in-band CCR retrieval. Send `false` for streaming requests. |
|
||
| `can_relay_response` | boolean | `false` | The gateway will post the provider's status and usage to `/v1/compress/response` after every turn. |
|
||
| `session_affinity` | boolean | `true` | The gateway pins this session to one Headroom replica. Pending turns are in-process state, so `false` disables re-driving for the turn even when `can_redrive` is set. `HEADROOM_COMPRESS_SESSION_STORE` shares replay state, not pending turns, so it does not lift this. |
|
||
| `plugin_version` | string | unset | Diagnostics only. |
|
||
| `request_headers` | object | `{}` | The client's request headers the gateway is willing to hand over, as lowercase name → string value. Only `anthropic-beta` is read today: when a turn hook asks for an `anthropic-beta` token, the client's tokens go first in the returned `headers`; every other name is ignored, so a gateway may pass its whole header map. A non-object block or a non-string value is a `400`. Never forwarded anywhere by Headroom. |
|
||
|
||
**What changes on the request half.** Every top-level field other than `messages`, `model`, `token_budget`, `config` and `gateway` passes through into the returned `body`: `system`, `tools`, `temperature`, `stream`, `metadata`, anything the provider accepts. `tools` is compacted deterministically (`tool_schema_compaction` in `transforms_applied`, plus `tool_desc_compaction` when `HEADROOM_TOOL_DESC_MAX_CHARS` is set), so identical tools come back byte-identical every turn. Two more request-only, streaming-safe levers run here exactly as they do on the proxy path:
|
||
|
||
- **Tool-search deferral (Anthropic) is an extension's job on this path.** OSS core does not mark tools `defer_loading` on `/v1/compress`; the [headroom-tool-search](https://github.com/headroomlabs-ai/headroom-tool-search) extension's native tier does, through the turn-hook contract, when it is installed, enabled (`HEADROOM_PROXY_EXTENSIONS=tool_search`) and licensed. It applies the same primitives as the proxy path (12 or more Anthropic-shaped tools on a first-party Claude model, the same resident set, the same client-already-searching check, `HEADROOM_TOOL_SEARCH=0` as the kill switch), is a pure function of the input tools so the tools prefix keeps prompt-caching turn over turn, and is named `tool_search:native_deferral:<n>tools:<n>tok` in `transforms_applied` and credited as `tool_schema_tokens_saved` in `/stats`. Without the extension the tools pass through compacted and `headers` is `{}`. Tool-search history repair stays in core regardless: once a transcript carries Anthropic's tool-search blocks, a later request whose tools cannot support them (a side request with a smaller tools array) has those blocks replaced in place by an inert placeholder text block, as on the proxy path (`router:tool_search_repair:<n>blocks`).
|
||
- **Output steering.** When the output shaper is on (`HEADROOM_OUTPUT_SHAPER=1` or the `proxy_output_shaper` rollout; off in `mode="cache"` for steering), the byte-stable verbosity block is appended to the tail of `system` (Anthropic) or the last system/developer message (OpenAI chat), the conversation-stable `HEADROOM_OUTPUT_HOLDOUT` arm is honoured, and the `output_shaper:stratum:…` / `output_shaper:control:…` label rides `transforms_applied` — it is what turns the response half's relayed `output_tokens` into output savings in `/stats`. The block is identical every turn, so the prefix still caches; it is applied to the returned `body` only, never to the session snapshot, so it is re-appended once per turn rather than accumulating.
|
||
|
||
Both fail open: any error leaves the body as built and `headers` empty.
|
||
|
||
**Gemini and Bedrock Converse bodies.** Besides `messages` (OpenAI chat or Anthropic) and `input` (OpenAI Responses), the request half accepts a native Gemini `generateContent` body (`contents`, with `systemInstruction` or `system_instruction`, and no `messages`) and a Bedrock Converse body (`messages` whose content blocks carry no `type`: `text`, `toolUse`, `toolResult`, `cachePoint`, …, with `system` as a list of blocks). Send `model` as usual. Text parts, `functionResponse.response` strings and Converse `toolResult` text and `json` blocks are compressed, and `body` comes back in the same shape with every other part byte-identical (`thoughtSignature`, `functionCall`, `inlineData`, `toolUse`, `cachePoint`, `reasoningContent`); a JSON slot is only rewritten when the result still parses. Their `tools`/`toolConfig` pass through untouched, output steering is skipped, and no re-drive is armed, so `obligations` never contains `redrive` for these shapes. Top-level `messages` in the answer is the chat-shaped view for Gemini and the Converse list for Converse. Only a gateway turn gets this treatment: without a `gateway` block the endpoint behaves as before. `config.frozen_message_count` and `config.protect_recent` count native messages (Gemini `contents` entries, Converse `messages`), and `session.frozen_message_count` comes back in the same units; the `/v1/usage` reply's `frozen_message_count` counts Headroom's internal view messages instead. Relayed usage may be Gemini's `usageMetadata` (`promptTokenCount`, `candidatesTokenCount`, `cachedContentTokenCount`) or Converse's `usage` (`inputTokens`, `outputTokens`, `cacheReadInputTokens`, `cacheWriteInputTokens`), or the whole provider response carrying either. Request hooks run stream-safe-only for these shapes. `config.mode: "ccr"` is rejected with a `400` (`invalid_request`) for these shapes, whatever the gateway's capabilities: Headroom cannot add `headroom_retrieve` to their tools, so a marker could never be resolved. Every other mode is marker-free for them, and cross-turn dedup pointers are off because they would count Headroom's view messages.
|
||
|
||
Turn hooks registered by proxy extensions run here as they do on the proxy path. Request hooks run stream-safe-only unless `can_redrive` and `session_affinity` are both `true` — a hook that must see the response is skipped when the gateway cannot come back with one. Hook savings show up as `turn_hook` and `turn_hook:tools:<n>tok`. In session mode hooks run after the replay overlay and before the snapshot, so a hook that is not a pure function of its input busts the prefix exactly as it would on the proxy.
|
||
|
||
The response keeps every field above and adds six:
|
||
|
||
```json
|
||
{
|
||
"messages": [{ "role": "user", "content": "..." }],
|
||
"tokens_before": 15000,
|
||
"tokens_after": 3500,
|
||
"body": {
|
||
"model": "gpt-4o",
|
||
"messages": [{ "role": "user", "content": "..." }],
|
||
"tools": [{ "type": "function", "function": { "name": "search", "parameters": {} } }],
|
||
"temperature": 0.2
|
||
},
|
||
"turn_id": "3d2f6c0e0b0a4e1a9c1b2f3d4e5f6a7b",
|
||
"route": { "model": "gpt-4o", "provider": "openai", "service_tier": null, "reason": "" },
|
||
"obligations": ["relay_usage"],
|
||
"gateway": { "can_redrive": false, "can_relay_response": true, "session_affinity": true },
|
||
"headers": {}
|
||
}
|
||
```
|
||
|
||
| Field | Description |
|
||
| --- | --- |
|
||
| `body` | The complete provider request: every pass-through field, final `messages` (the same list as top-level `messages`) and final `tools`. Never contains `config`, `gateway` or `token_budget`. Forward it as-is. |
|
||
| `headers` | Request headers the gateway must **set** on the provider request (name → value), on the original forward and on every `redrive`. `{}` when nothing is needed. Produced by turn hooks through `TurnContext.provider_headers`: only allow-listed names are honoured (today `anthropic-beta`), and `anthropic-beta` is merged behind the client's value from `gateway.request_headers` (deduplicated, client tokens first — the same merge the proxy path uses for every Headroom-required beta). The tool-search extension's native tier uses it for `advanced-tool-use-2025-11-20`. Fail-open answers always carry `{}`. |
|
||
| `turn_id` | Handle for the response half. Always present, but a turn is only *registered* — and worth posting back — when `obligations` is non-empty. |
|
||
| `route` | Advisory. `provider` is inferred from the model name (`anthropic` for names containing `claude` or `anthropic`, else `openai`) unless a routing extension published advice; when it did, its model is also written into `body.model`, so `route.model` and `body.model` never disagree. `service_tier` echoes the request. Headroom never switches the upstream itself. |
|
||
| `obligations` | `redrive` — a transform ran whose reload step needs the response; post `response` to `/v1/compress/response` and honour `redrive` answers. `relay_usage` — you offered `can_relay_response`; post status and usage when the provider answers. |
|
||
| `gateway` | Echo of the capabilities Headroom honoured. |
|
||
| `wire_shape` | The request shape Headroom read: `gemini`, `converse`, `openai_responses`, or `messages` (Chat Completions or Anthropic). `body` comes back in that shape. A caller that sent a native body and gets anything else, or no field (older Headroom), should forward its original body. |
|
||
|
||
Fail-open answers (bypass header, empty `messages`, stateless timeout) carry the same six keys with your original body, `obligations: []` and `headers: {}`.
|
||
|
||
**Savings on a gateway answer.** The token counts follow the proxy path, so a gateway can report the same figure `/stats` does. When compaction shortened `tools`, `tokens_before` and `tokens_after` also count the tool definitions before and after it, so `tokens_saved` includes the compaction saving. Two more fields carry the tool definitions kept out of the context, which no message count sees; both are absent on fail-open answers:
|
||
|
||
| Field | Description |
|
||
| --- | --- |
|
||
| `tool_tokens_saved` | Tool-search deferral plus tools a turn hook removed, net (`tool_search_deferred_tokens` after reconciliation + `turn_hook_tools_saved_tokens` after netting against message growth). `tokens_saved + tool_tokens_saved` is the `/stats` headline for the turn. |
|
||
| `tool_tokens_deferred` | The deferral share of `tool_tokens_saved`. `/stats` books it as `estimated`: a deferred tool the model loads later is billed after all. Subtract it for a figure of measured savings only. |
|
||
|
||
When `relay_usage` is owed, the turn's `/stats` record is deferred until the response half arrives with billed input and output, cache counts, provider status and latency; a turn that never comes back is recorded as-is when it expires. Pending turns live `HEADROOM_GATEWAY_TURN_TTL_SECONDS` (default `120`, refreshed on every `redrive`), at most `HEADROOM_GATEWAY_MAX_PENDING_TURNS` at a time (default `10000`, oldest evicted first), and a turn may re-drive at most `HEADROOM_GATEWAY_MAX_REDRIVES` times (default `8`).
|
||
|
||
**`config.mode: "ccr"` on this contract.** When markers were inserted and re-driving is allowed, Headroom injects the `headroom_retrieve` tool into `body.tools` (`ccr_tool_injected` in `transforms_applied`), adds the `redrive` obligation, and the response half answers the model's retrieval calls itself — the gateway just keeps re-driving. Without `can_redrive`, `ccr` mode behaves as it does today: the markers are yours to resolve. Gemini and Converse bodies cannot use `ccr` mode (see above).
|
||
|
||
#### `POST /v1/compress/response`
|
||
|
||
The response half. Same exposure rules as `/v1/compress`: loopback-only unless `HEADROOM_COMPRESS_ALLOW_REMOTE=1`, and `HEADROOM_PROXY_TOKEN` applies.
|
||
|
||
| Field | Type | Required | Description |
|
||
| --- | --- | --- | --- |
|
||
| `turn_id` | string | yes | From the request half. |
|
||
| `status` | integer | no | Provider HTTP status, `100`–`599`. Default `200`. Hooks only start on a `200`. |
|
||
| `latency_ms` | number | no | Gateway-measured provider latency, added to the turn's recorded latency. Must be non-negative. |
|
||
| `round` | integer | no | Which provider call this response answers: `0` for the original forward, `n` for the `n`-th re-drive (the `round` this endpoint returned). Makes the call idempotent: a retried relay for a round already answered gets the same `redrive` answer again instead of being fed to the hook, and its usage is not billed twice. A round ahead of the turn is a `400`. |
|
||
| `usage` | object | no | The provider's usage block, in any accepted shape (below). |
|
||
| `response` | object | no | The full provider response JSON. **Required** when the turn carries `redrive` and the provider call succeeded (`2xx`). A failed call closes the turn with its `status` alone. |
|
||
| `model` | string | no | Provider-reported model. Accepted, not read today. |
|
||
|
||
Accepted `usage` shapes — the whole provider response body works too, since a nested `usage` key is descended into once:
|
||
|
||
- Anthropic: `input_tokens`, `output_tokens`, `cache_read_input_tokens`, `cache_creation_input_tokens`
|
||
- OpenAI chat: `prompt_tokens`, `completion_tokens`, `prompt_tokens_details.cached_tokens`
|
||
- OpenAI Responses: `input_tokens`, `output_tokens`, `input_tokens_details.cached_tokens`
|
||
- Kong's flattened `ai.proxy` log shape: `prompt_tokens`, `completion_tokens`, top-level `cached_tokens`
|
||
|
||
Booleans and negative counts are a `400`. Usage with no cache field is accepted but not applied to the session tracker; the apply rule is `/v1/usage`'s (a positive count, or both cache fields present at `0`). Across re-drive rounds the billed counters are summed and the cache fields come from the latest round that carried them.
|
||
|
||
There are two answers. `done` ends the turn:
|
||
|
||
```json
|
||
{
|
||
"action": "done",
|
||
"turn_id": "3d2f6c0e0b0a4e1a9c1b2f3d4e5f6a7b",
|
||
"response": null,
|
||
"frozen_message_count": 12,
|
||
"usage_applied": true,
|
||
"rounds": 0,
|
||
"billed_usage": { "input_tokens": 3120, "output_tokens": 212, "cache_read_input_tokens": 20414 }
|
||
}
|
||
```
|
||
|
||
`response` is what to send to the client. On a turn with no re-drive it is the hook's replacement when there is one, otherwise `null` and you forward the provider response you already hold. After one or more re-drives it is never `null`: it is the hook's replacement or the latest provider response, with `usage` set to the whole turn's bill. `frozen_message_count` is the session's count after the usage was applied (`null` for a stateless turn). `usage_applied` is `false` when the usage carried no cache signal, the session had expired, or its lock was busy — never an error, since the turn is finishing and there is nothing to retry with. `billed_usage` is what the provider billed for the whole turn: every counter, cache reads and writes included, summed across all rounds, in Anthropic keys (`null` when no usage was relayed). That total is the re-driven `response`'s `usage`, since the client made one call. After `done` the turn is gone.
|
||
|
||
`redrive` asks for another provider call:
|
||
|
||
```json
|
||
{
|
||
"action": "redrive",
|
||
"turn_id": "3d2f6c0e0b0a4e1a9c1b2f3d4e5f6a7b",
|
||
"request": { "model": "gpt-4o", "messages": [], "tools": [] },
|
||
"round": 1
|
||
}
|
||
```
|
||
|
||
Send `request` to the provider exactly as given — it is the provider body with the hook's messages and possibly reloaded tools — and post the provider's JSON back under the same `turn_id`. Past `HEADROOM_GATEWAY_MAX_REDRIVES` the turn ends with `done` and the latest provider response. Hook failures never fail the turn either: the answer is `done`, with the latest provider response after a re-drive, or `response: null` before one, and the gateway forwards what it gets.
|
||
|
||
| Status | Body | When |
|
||
| --- | --- | --- |
|
||
| `400` | `error.type = "invalid_request"` | Malformed JSON, missing or invalid `turn_id`, `status` not an HTTP status code, negative `latency_ms`, non-object `usage` or `response`, boolean or negative usage counts. |
|
||
| `400` | `error.type = "missing_response"` | The turn carries `redrive` and `response` is absent. |
|
||
| `404` | `error.type = "unknown_turn"` | Never registered (the request half returned no obligations), already finished, or expired. |
|
||
| `409` | `error.type = "turn_busy"` | A response-half call for this turn is already being driven. |
|
||
| `400` | `error.type = "round_out_of_sequence"` | `round` is ahead of the round the turn is waiting for. |
|
||
| `409` | `error.type = "round_already_answered"` | `round` names a round that was already answered with `done`. |
|
||
|
||
#### Kong plugin
|
||
|
||
[kong-plugin-headroom](https://github.com/headroomlabs-ai/kong-plugin-headroom) is a Kong Gateway 3.9 plugin, in its own repository and installable with `luarocks install kong-plugin-headroom` or as a Konnect custom plugin, that implements both halves: it sends every chat request through `/v1/compress` with `config.session_id` and a `gateway` block, swaps `body` in, relays usage from the `log` phase, and runs the re-drive loop in `access` when a `redrive` obligation comes back. Install steps, configuration and limits are in its README. Headroom's docker end-to-end test (`tests/gateway/test_kong_integration.py`, `HEADROOM_KONG_E2E=1`) expects a checkout of it next to this repository or at `HEADROOM_KONG_PLUGIN_DIR`.
|
||
|
||
**The contract is itself an extension.** Core exposes a compress-turn seam (`headroom.proxy.compress_turn`): an extension registers a contract with `register_compress_turn_extension`, and per request it may claim the body (`begin`), prepare executor-side work (`prepare`), rewrite the outbound messages at the one safe moment in session mode (`transform`, after the prefix replay and before the tracker records what was returned), add response fields and transform labels (`finish`), take ownership of the outcome (`commit`), and answer the fail-open cases (`fail_open_fields`). The gateway turn contract described above is installed through that seam by `headroom.proxy.gateway_extension` (on by default; `HEADROOM_GATEWAY_CONTRACT=0` leaves it out, after which a `gateway` block is ignored and the endpoint answers in legacy shape). A third-party contract's `install(app, config)` finds the `/v1/compress` exposure policy on `app.state.compress_route_dependencies` for any route it adds.
|
||
|
||
<Callout type="info" title="Not the same as HEADROOM_KOMPRESS_ENDPOINT">
|
||
`HEADROOM_KOMPRESS_ENDPOINT` points *outbound* at a remote Kompress ML model server that happens to expose a `/compress` path. It is unrelated to this inbound endpoint.
|
||
</Callout>
|
||
|
||
## Agent wrapping
|
||
|
||
Use `headroom wrap` to launch supported CLI agents through the local proxy:
|
||
|
||
```bash
|
||
# Claude Code
|
||
headroom wrap claude
|
||
|
||
# Claude Code extension in VS Code (configures settings, then starts the proxy)
|
||
headroom wrap vscode-claude
|
||
|
||
# OpenAI Codex
|
||
headroom wrap codex
|
||
|
||
# Aider
|
||
headroom wrap aider
|
||
|
||
# Cursor (starts the proxy and prints settings to paste into Cursor)
|
||
headroom wrap cursor
|
||
|
||
# Grok Build (updates ~/.grok/config.toml and starts the proxy)
|
||
headroom wrap grok-build
|
||
```
|
||
|
||
Wrapping can reuse an existing proxy instead of starting a new process. Its
|
||
startup-only settings do not change when you export different values in the
|
||
calling shell. `wrap` warns when an explicit `HEADROOM_MODE`,
|
||
`HEADROOM_MIN_TOKENS`, or `HEADROOM_EXCLUDE_TOOLS` disagrees with the settings
|
||
reported by that proxy. It does not restart a shared proxy or interrupt other
|
||
clients. Restart the proxy with the desired settings, or use `--port` to start
|
||
a separate instance. Tool exclusions are compared case-insensitively using the
|
||
effective router exclusions, including built-ins and tools protected by
|
||
`HEADROOM_PROTECT_TOOL_RESULTS`. Older proxies that do not report a setting
|
||
cannot be compared for that setting.
|
||
|
||
|
||
`headroom wrap codex` keeps routing scoped to the Codex process it launches: it
|
||
passes the proxy URL as a `--config` override and never writes it to
|
||
`~/.codex/config.toml`. Codex runs any session started with command-line
|
||
overrides in embedded mode, so a wrapped session does not use Codex's shared
|
||
background server and Codex prints "Running without the shared background
|
||
server". If you want the shared server, write the proxy routing to Codex's own
|
||
config instead, then start `codex` directly:
|
||
|
||
```bash
|
||
headroom init codex # this project's .codex/config.toml (trusted projects)
|
||
headroom init -g codex # ~/.codex/config.toml, for every project
|
||
```
|
||
|
||
That routing is persistent: it applies to every Codex session in that scope
|
||
until you remove it. `headroom unwrap codex` removes the `-g` routing; for a
|
||
project, delete the `# --- Headroom init provider ---` block from
|
||
`.codex/config.toml`. Unwrap removes only the routing: the Headroom
|
||
`SessionStart` hook that `init` adds to Codex's `hooks.json` stays, and still
|
||
starts a proxy when a Codex session begins until you delete it. On Windows,
|
||
nothing starts the proxy for this routing, so Codex cannot connect while the
|
||
proxy is down; use `headroom install apply` if you need a supervised proxy.
|
||
|
||
Cursor reads model endpoints from its settings UI, so `headroom wrap cursor`
|
||
does not rewrite Cursor configuration or launch the app. After it starts the
|
||
proxy, copy the printed base URL into Cursor's model settings.
|
||
|
||
Grok Build reads model endpoints from `~/.grok/config.toml`. `headroom wrap grok-build`
|
||
injects or updates `[model.grok-build] base_url` to point at the local proxy, then
|
||
run `grok` from the same project directory. See [Grok Build Integration](/docs/grok-build).
|
||
|
||
The official Claude Code extension reads Claude Code's user settings rather than
|
||
the terminal environment. Use `headroom wrap vscode-claude`, reload VS Code after
|
||
the first run, and keep the wrapper running. See the
|
||
[VS Code Claude Code guide](/docs/vscode-claude-code) for verification and undo
|
||
steps.
|
||
|
||
For environment-driven clients, you can also set the base URL manually:
|
||
|
||
```bash
|
||
# Claude Code
|
||
ANTHROPIC_BASE_URL=http://localhost:8787 claude
|
||
|
||
# Any OpenAI-compatible CLI client that reads OPENAI_BASE_URL
|
||
OPENAI_BASE_URL=http://localhost:8787/v1 your-client
|
||
```
|
||
|
||
## Cloud providers
|
||
|
||
```bash
|
||
# AWS Bedrock
|
||
headroom proxy --backend bedrock --region us-east-1
|
||
|
||
# Google Vertex AI
|
||
headroom proxy --backend vertex_ai --region us-central1
|
||
|
||
# Azure OpenAI
|
||
headroom proxy --backend azure
|
||
|
||
# OpenRouter (400+ models)
|
||
OPENROUTER_API_KEY=sk-or-... headroom proxy --backend openrouter
|
||
```
|
||
|
||
### Google Vertex AI
|
||
|
||
`--backend vertex_ai` delegates to [LiteLLM](https://docs.litellm.ai/docs/providers/vertex),
|
||
which brings two requirements that are easy to miss:
|
||
|
||
**1. Install the Vertex SDK.** `google-cloud-aiplatform` ships in the dedicated
|
||
`[vertex]` extra, kept out of `[proxy]` because it and its transitive tree add
|
||
roughly 175 MB and only this backend needs it:
|
||
|
||
```bash
|
||
pip install "headroom-ai[proxy,vertex]"
|
||
```
|
||
|
||
Without it the proxy **refuses to start**, naming the fix and exiting `2`:
|
||
|
||
```text
|
||
Cannot start proxy: Vertex backend selected but the Vertex SDK is missing. ...
|
||
```
|
||
|
||
This is a startup check rather than a per-request failure, because the backend
|
||
would otherwise construct cleanly and then fail on every call with an opaque
|
||
`litellm.BadRequestError: … No module named 'vertexai'`.
|
||
|
||
**2. Set the LiteLLM project/location variables.** LiteLLM reads the GCP project
|
||
and region from `VERTEXAI_PROJECT` / `VERTEXAI_LOCATION` — these are **not** the
|
||
standard Google Cloud variables (`GOOGLE_CLOUD_PROJECT` / `GOOGLE_CLOUD_LOCATION`)
|
||
used by `gcloud` and current Google SDKs:
|
||
|
||
```bash
|
||
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json # or use ADC
|
||
export VERTEXAI_PROJECT=<your-gcp-project>
|
||
export VERTEXAI_LOCATION=us-central1
|
||
|
||
headroom proxy --backend vertex_ai --region us-central1
|
||
```
|
||
|
||
<Callout type="warn">
|
||
If `VERTEXAI_PROJECT` is unset, requests do not fail loudly — they can silently
|
||
resolve against your Application Default Credentials' default quota project,
|
||
billing a different GCP project than you intended. Set it explicitly even if
|
||
`GOOGLE_CLOUD_PROJECT` is already exported.
|
||
</Callout>
|
||
|
||
**Backend name aliases.** `vertex_ai`, `vertex`, `google-vertex`, `googlevertex`,
|
||
`litellm-vertex`, and `litellm-vertex_ai` are all normalized to the same
|
||
LiteLLM-backed `vertex_ai` backend — CLI help text and older docs use these
|
||
spellings interchangeably.
|
||
|
||
Running Claude Code against Claude models on Vertex? See
|
||
[Claude Code on Vertex AI](/docs/claude-code-vertex) for the recommended
|
||
native Vertex-mode flow that reuses Claude Code's own GCP auth.
|
||
|
||
<Callout type="info" title="Not the same as the LiteLLM callback">
|
||
`--backend vertex_ai` runs Headroom **as a proxy that itself calls Vertex via
|
||
LiteLLM**. The [LiteLLM integration page](/docs/litellm) documents the inverse:
|
||
adding Headroom as a compression callback inside your own LiteLLM app. Despite
|
||
the shared name, they are different mechanisms.
|
||
</Callout>
|
||
|
||
### Native Vertex passthrough routes
|
||
|
||
Separately from `--backend vertex_ai`, the proxy always registers routes that
|
||
mirror Vertex's native REST shape verbatim — no backend flag needed:
|
||
|
||
```text
|
||
/{api_version}/projects/{project}/locations/{location}/publishers/{publisher}/models/{model}:generateContent
|
||
/{api_version}/projects/{project}/locations/{location}/publishers/{publisher}/models/{model}:streamGenerateContent
|
||
/{api_version}/projects/{project}/locations/{location}/publishers/{publisher}/models/{model}:countTokens
|
||
/{api_version}/projects/{project}/locations/{location}/publishers/{publisher}/models/{model}:rawPredict
|
||
/{api_version}/projects/{project}/locations/{location}/publishers/{publisher}/models/{model}:streamRawPredict
|
||
```
|
||
|
||
Requests with `publisher=google` (Gemini models) go through Headroom's full
|
||
Gemini optimization handler; `publisher=anthropic` (Claude on Vertex) is
|
||
forwarded to Vertex directly, preserving your bearer token. Any client that
|
||
already speaks the native Vertex REST API can simply point its endpoint at the
|
||
proxy — this is the mechanism `headroom wrap claude` uses in
|
||
[Vertex mode](/docs/claude-code-vertex).
|
||
|
||
<Callout type="warn">
|
||
These routes need **neither** `--backend vertex_ai` nor the `[vertex]` extra. Do
|
||
not combine the two: with the backend flag set, Claude publisher requests are
|
||
re-routed through LiteLLM, which derives the project, region and model from its
|
||
own configuration and ignores the ones in your URL — a request that works
|
||
without the flag can return a `404` for a model you never asked for.
|
||
</Callout>
|
||
|
||
### Reading Vertex errors
|
||
|
||
When Vertex rejects a request with `401`, `403`, `404` or `429`, Headroom appends
|
||
a `[headroom] hint: …` note to `error.message`, sets an `x-headroom-hint`
|
||
response header, and logs a warning. The hint names the likely cause — expired
|
||
ADC, a partner model not yet enabled in Model Garden, or a model that is not
|
||
served in the requested location. Successful responses are never modified.
|
||
|
||
Hints are fixed text: they never echo your request path, project, credentials or
|
||
the underlying exception. With `--backend vertex`, a backend that cannot serve
|
||
(missing Vertex SDK, no ADC) returns the standard sanitized proxy error plus the
|
||
same kind of fixed hint; the raw provider detail is only in the proxy log, under
|
||
the response's `request_id`.
|
||
|
||
### Bedrock via a local gateway
|
||
|
||
`--backend bedrock` accepts **Anthropic** input (`/v1/messages`) and re-signs to AWS. Some setups are the other way around: the client already speaks **Bedrock** (e.g. Claude Code with `CLAUDE_CODE_USE_BEDROCK=1`, or any AWS SDK pointed at a custom endpoint), sending `POST /model/{id}/invoke` to a local gateway that re-signs and forwards to AWS (LiteLLM, LocalStack, a corporate Bedrock proxy).
|
||
|
||
`--bedrock-api-url` lets Headroom sit in that chain. It registers passthrough routes for `/model/{id}/invoke` and `/model/{id}/invoke-with-response-stream`, compresses the request body with the same pipeline as `/v1/messages`, and forwards to the gateway:
|
||
|
||
```bash
|
||
headroom proxy --bedrock-api-url http://127.0.0.1:4000
|
||
# then point the client's Bedrock endpoint at Headroom:
|
||
AWS_ENDPOINT_URL_BEDROCK_RUNTIME=http://127.0.0.1:8787 your-bedrock-client
|
||
```
|
||
|
||
The routes are registered **only** when `--bedrock-api-url` (or `BEDROCK_TARGET_API_URL`) is set — otherwise Bedrock requests fall through unchanged.
|
||
|
||
<Callout type="warn">
|
||
Rewriting the request body invalidates the caller's **SigV4** signature (it covers a hash of the body). Point `--bedrock-api-url` at a gateway that re-signs or does not verify the inbound signature — **never raw AWS**, which would reject the request with 403. For direct-to-AWS compression, use `--backend bedrock` (which re-signs). The two are complementary.
|
||
</Callout>
|
||
|
||
## Environment variables
|
||
|
||
```bash
|
||
# For deliberate public access, configure the existing token and send it from callers.
|
||
export HEADROOM_HOST=0.0.0.0
|
||
export HEADROOM_PROXY_TOKEN='replace-with-a-secret'
|
||
export HEADROOM_PORT=8787
|
||
export HEADROOM_BUDGET=100.0
|
||
|
||
# Route OpenAI passthrough requests to a custom endpoint
|
||
export OPENAI_TARGET_API_URL=https://custom.openai.endpoint.com
|
||
|
||
# Route Anthropic passthrough requests to a custom endpoint
|
||
export ANTHROPIC_TARGET_API_URL=https://litellm.company.internal
|
||
|
||
# Compress Bedrock InvokeModel traffic, forwarding to a re-signing gateway
|
||
export BEDROCK_TARGET_API_URL=http://127.0.0.1:4000
|
||
|
||
headroom proxy
|
||
```
|
||
|
||
## Production deployment
|
||
|
||
### gunicorn
|
||
|
||
```bash
|
||
pip install gunicorn
|
||
|
||
gunicorn "headroom.proxy.server:create_app()" \
|
||
--workers 4 \
|
||
--bind 0.0.0.0:8787 \
|
||
--worker-class uvicorn.workers.UvicornWorker
|
||
# Callers must send `Authorization: Bearer replace-with-a-secret` or
|
||
# `X-Headroom-Proxy-Token: replace-with-a-secret`.
|
||
```
|
||
|
||
`headroom.proxy.server` has no module-level `app` object — it exposes a `create_app(config: ProxyConfig | None = None) -> FastAPI` factory, so gunicorn's import string must call it.
|
||
|
||
### Docker
|
||
|
||
```dockerfile
|
||
FROM python:3.11-slim
|
||
RUN apt-get update && apt-get install -y --no-install-recommends build-essential \
|
||
&& pip install "headroom-ai[proxy]" \
|
||
&& apt-get purge -y build-essential && apt-get autoremove -y \
|
||
&& rm -rf /var/lib/apt/lists/*
|
||
EXPOSE 8787
|
||
CMD ["headroom", "proxy", "--host", "0.0.0.0", "--port", "8787"]
|
||
```
|
||
|
||
Run this image with an explicit token and a deliberate publication choice:
|
||
|
||
```bash
|
||
docker run --rm -p 127.0.0.1:8787:8787 \
|
||
-e HEADROOM_PROXY_TOKEN='replace-with-a-secret' \
|
||
headroom-proxy
|
||
```
|
||
|
||
Callers must send `Authorization: Bearer replace-with-a-secret` or
|
||
`X-Headroom-Proxy-Token: replace-with-a-secret`. For network access, replace
|
||
the host-side `127.0.0.1` with an intentional public address and keep the
|
||
token configured.
|
||
|
||
<Callout type="info" title="Build dependencies">
|
||
`build-essential` is required at install time because `headroom-ai` includes `hnswlib`, a C++ extension compiled from source. It is removed after installation to keep the image slim.
|
||
</Callout>
|