mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description Both named sandbox runs (Codex, Claude Code: "install headroom-ai and add POST /chat that compresses messages") built exactly this endpoint from the installed source, because no docs page described it. The unbranded run and the web query "fastapi reduce openai token cost middleware" found nothing of ours. The quickstart led with TypeScript, which needs the proxy anyway, and installed `[all]`, which pulls torch, the weight the chat engines objected to when they chose to hand-roll a trimmer instead. Part of the agent-readiness work: README (#4530), llms.txt (#4531), AGENTS.md (#4532), skill (#4533), PyPI README (#4534) link to `/docs/fastapi`, so this PR should merge first or alongside them. ## Type of Change - [x] Documentation update ## Changes Made - `docs/content/docs/fastapi.mdx` (new, first under Integrations): `POST /chat` on `compress()`, an offline test request with expected output, the one-line forward to OpenAI/Anthropic, what is kept, and a note that `CompressionMiddleware` is for proxy-shaped apps, not apps that call a provider. - `quickstart.mdx`: Python tab first in all four Tabs blocks; base install with an `[all]` note; a "no provider key needed" callout with the zero-network one-liner. - `index.mdx`: description carries the task phrases (reduce input tokens, token cost, context window) so the generated `/llms.txt` does too. ## Testing - [x] Manual testing performed - [x] `npm run types:check` in `docs/` (fumadocs-mdx + next typegen + tsc): exit 0 ### Test Output ```text types:check (docs/): ✓ Types generated successfully, exit 0 Page test request, published 0.42.0 wheel, 3 runs: 20031 -> 95 FATAL kept: True ['router:protected:user_message', 'router:log:0.00'] ``` ## Real Behavior Proof - Environment: macOS arm64, Python 3.12 venv with the **published** `headroom-ai==0.42.0` wheel plus `fastapi`, `uvicorn[standard]`; run from outside the checkout (inside the checkout, the source `headroom/` dir shadows the wheel and has no compiled `_core`). - Exact command / steps: `HEADROOM_OFFLINE=1 uvicorn main:app --port 8767` with the page's `main.py`, then the page's test request, three times. - Observed result: as pasted below. ```text 20031 -> 95 FATAL kept: True ['router:protected:user_message', 'router:log:0.00'] ``` - Not tested: the `/answer` snippet against a live provider (needs a key); rendering on the deployed Vercel site (Vercel preview will show it). ## Runtime Rollout Safety - Rollout-managed feature(s): none; documentation and metadata only, no runtime code path changes - Minimum rollout channel: n/a - Stable/default behavior changed: no - Kill switch / disable path: n/a (revert the PR) - Unsafe override required: no - Qualification impact: none - Rollback path: revert the commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes The example file was dropped from this PR on request: as a network-facing module under `examples/` it became a CodeQL taint source and surfaced seven library sinks as new alerts. The page carries the full code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
125 lines
4.3 KiB
Plaintext
125 lines
4.3 KiB
Plaintext
---
|
|
title: FastAPI
|
|
description: Cut LLM input tokens in a FastAPI app. Compress tool outputs, logs and chat history with compress() before calling OpenAI or Anthropic. No API key needed to try it.
|
|
---
|
|
|
|
Headroom's `compress()` is a function over your message list, so in FastAPI it
|
|
is one call before your model call. This page builds a `POST /chat` endpoint
|
|
that returns the compressed messages and token counts, then shows the one-line
|
|
change that forwards them to a provider.
|
|
|
|
## Install
|
|
|
|
```bash
|
|
pip install fastapi "uvicorn[standard]" headroom-ai
|
|
```
|
|
|
|
No provider key, no model download. The base package is enough for
|
|
`compress()`; `[ml]` adds the Kompress text model later if you want more
|
|
savings on prose-heavy tool output.
|
|
|
|
## Endpoint
|
|
|
|
```python
|
|
# main.py
|
|
from typing import Any
|
|
|
|
from fastapi import FastAPI
|
|
from pydantic import BaseModel
|
|
|
|
from headroom import compress
|
|
|
|
app = FastAPI()
|
|
|
|
|
|
class ChatIn(BaseModel):
|
|
messages: list[dict[str, Any]] # OpenAI or Anthropic message dicts
|
|
model: str = "gpt-4o" # used for token counting and the context limit
|
|
|
|
|
|
@app.post("/chat")
|
|
def chat(body: ChatIn) -> dict[str, Any]:
|
|
result = compress(body.messages, model=body.model)
|
|
return {
|
|
"messages": result.messages,
|
|
"tokens_before": result.tokens_before,
|
|
"tokens_after": result.tokens_after,
|
|
"tokens_saved": result.tokens_saved,
|
|
"transforms": result.transforms_applied,
|
|
}
|
|
```
|
|
|
|
Save it as `main.py` and run it with no network at all:
|
|
|
|
```bash
|
|
HEADROOM_OFFLINE=1 uvicorn main:app --port 8000
|
|
```
|
|
|
|
Then post a 3,000-line tool output with one error in it:
|
|
|
|
```bash
|
|
python - <<'EOF_PY'
|
|
import json, urllib.request
|
|
log = "\n".join([f"INFO step {i} ok" for i in range(3000)] + ["FATAL disk full"])
|
|
msgs = [
|
|
{"role": "user", "content": "What failed?"},
|
|
{"role": "assistant", "content": None, "tool_calls": [{"id": "c1", "type": "function", "function": {"name": "logs", "arguments": "{}"}}]},
|
|
{"role": "tool", "tool_call_id": "c1", "content": log},
|
|
]
|
|
req = urllib.request.Request("http://127.0.0.1:8000/chat", data=json.dumps({"messages": msgs}).encode(), headers={"content-type": "application/json"})
|
|
out = json.load(urllib.request.urlopen(req))
|
|
print(out["tokens_before"], "->", out["tokens_after"], "FATAL kept:", "FATAL" in json.dumps(out["messages"]))
|
|
EOF_PY
|
|
```
|
|
|
|
Expected output is on the order of `20000 -> 95 FATAL kept: True`: the
|
|
repeated `INFO` lines collapse, the `FATAL` line survives.
|
|
|
|
## Forward to the provider
|
|
|
|
Replace `messages=body.messages` with `messages=result.messages` in your
|
|
existing call. Nothing else changes:
|
|
|
|
```python
|
|
from openai import OpenAI
|
|
|
|
client = OpenAI()
|
|
|
|
|
|
@app.post("/answer")
|
|
def answer(body: ChatIn) -> dict[str, Any]:
|
|
result = compress(body.messages, model=body.model)
|
|
completion = client.chat.completions.create(model=body.model, messages=result.messages)
|
|
return {
|
|
"answer": completion.choices[0].message.content,
|
|
"tokens_saved": result.tokens_saved,
|
|
}
|
|
```
|
|
|
|
For Anthropic, pass `result.messages` to `client.messages.create` the same way.
|
|
|
|
## What it keeps
|
|
|
|
Error lines, stack traces, first and last items, and statistical outliers in
|
|
JSON arrays are kept. Repeated and low-information lines are dropped. Originals
|
|
are cached locally; the model can ask for them back when it has the retrieval
|
|
tool, which the [proxy](/docs/proxy) and the [MCP server](/docs/mcp) provide
|
|
([CCR](/docs/ccr)). The plain `compress()` call above does not expose it.
|
|
Compression applies to tool and assistant messages; user messages are left alone
|
|
unless you pass `compress_user_messages=True`. Short prompts and prose compress
|
|
very little; see [Limitations](/docs/limitations).
|
|
|
|
## Not this page
|
|
|
|
`headroom.integrations.asgi.CompressionMiddleware` is for FastAPI apps that
|
|
*serve* an LLM-shaped endpoint (`/v1/messages`, `/v1/chat/completions`), such
|
|
as a proxy you built. For an app that *calls* a provider, use `compress()` as
|
|
above.
|
|
|
|
## Settings you will reach for
|
|
|
|
- `HEADROOM_OFFLINE=1`: no update check, no model or pricing-table downloads, no beacon. Use it in CI and sandboxes. See [Security model](/docs/security-model).
|
|
- `compress(..., model_limit=N)`: tell Headroom the model's window is N tokens, so it compresses harder as the prompt approaches it. It never truncates history.
|
|
- `compress(..., protect_recent=K)`: leave source code in the last K messages uncompressed.
|
|
- [Configuration](/docs/configuration) has the rest.
|