mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:07:29 +08:00
Audited all 52 pages of docs.headroomlabs.ai/docs against source, executing code rather than reading it. **36 pages changed.** Every figure that could not be traced to a benchmark, a test, or a committed artifact was measured or removed — none was replaced with a different guess. ## The site was telling users things that were not true **Telemetry.** `proxy.mdx` stated *"nothing is sent to Headroom Labs"* and that the anonymous beacon *"has been removed from the code."* Neither is true — `BEACON_DEFAULT_ON = True` (`headroom/telemetry/beacon.py:56`), `is_beacon_enabled()` is fail-open, and there is a live POST at `headroom/telemetry/session.py:75`. `proxy.mdx` and `limitations.mdx` now describe both switches accurately, say why the beacon exists (compression quality is the product; it shows when a release regresses a ratio across real workloads), and list the 12 fields the collector allowlists instead of asserting "content-free". The install id is disclosed for what it is: a random UUID, not a machine fingerprint, resettable by deleting the file. **Numbers.** The landing page's "Proof" table claimed 92/92/47/73% savings from a harness that seeded nothing, so it could not reproduce a figure even in principle. | Scenario | Claimed | Measured | |---|---|---| | Code search (100 results) | 92% | **21%** | | SRE incident debugging | 92% | **57%** | | Codebase exploration | 47% | **42%** | | GitHub issue triage | 73% | **30%** | `real_world_agent_benchmark.py` now seeds, and `benchmarks/index_proof_table.py` measures the same four scenarios through the real tokenizer and the real `compress()` path with no network call. Stable across five seeds; the page cites the command. Latency was wrong in the *other* direction — 189 ms and 943 ms published against **0.21 ms and 0.80 ms** measured. The docs made the product look ~1000× slower than it is. **Examples.** Every `agno` example was broken at line one, including Quick Start: `HeadroomAgnoModel`'s first dataclass field is `id`, so a positional wrapped model bound to the wrong field and raised `ValueError`. `api-reference.mdx` — the canonical signature reference — had 13 broken examples, including three `AttributeError`s in the first three lines of its `simulate()` block. Four integration pages omitted the `[proxy]` extra and then told the reader to run `headroom proxy`, which fails immediately. `ccr.mdx` showed SDK-level retrieval that only exists in the proxy handlers. Also: six Prometheus metric names that are never emitted, a Grafana dashboard filtering on a label no metric carries, a documented deploy workflow that does not exist, a 365-day retention default the code caps at 30, and CCR TTL stated as 5 minutes against an actual 1800s. ## Canonical URL bug `robots.ts`, `sitemap.ts` and `layout.tsx` fell back to a hardcoded `headroom-docs.vercel.app` when `NEXT_PUBLIC_SITE_URL` was unset — so the live site served a sitemap and `Host:` on the wrong domain and pointed crawlers at the wrong canonical. Verified in the build output: robots and all 53 sitemap URLs now carry `docs.headroomlabs.ai`. Setting `NEXT_PUBLIC_SITE_URL=https://docs.headroomlabs.ai` in Vercel is still worth doing so previews match, but the code no longer depends on it. ## Verification Four adversarial judges re-checked this work. They confirmed the landing-page table reproduces digit-for-digit across five seeds, found **zero cross-page contradictions**, independently regenerated all nine surviving compression percentages, and proved `headroom_keep_turns` dead at runtime with a mocked-transport test. They also caught two links this audit had reversed to the deprecated org, and 18 examples it had missed — both fixed here. `next build` passes with all 52 pages prerendered, matching the pre-change baseline. ## Follow-up, not in this PR - GitHub org migration (`chopratejas` → `headroomlabs-ai`) across source, manifests and packaging, plus stale-domain fixes in `llms.txt`, the dashboard UI and `pyproject.toml`. - `wiki/` disposition: it is unpublished and unbuilt, and `wiki/benchmarks.md` carried an invented production-telemetry section (250+ instances, 1.4 billion tokens saved) that exists nowhere in the repo. - `langchain.mdx` imports are dead against LangChain 1.x (`langchain.memory`, `langchain.retrievers`, `create_openai_tools_agent`). Needs a pin-vs-rewrite decision. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01HRaEPjQcCSAwD2xbQMNENu --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
163 lines
4.9 KiB
Plaintext
163 lines
4.9 KiB
Plaintext
---
|
|
title: Image Compression
|
|
description: ML-powered image compression that reduces vision model token usage by 40-90% while maintaining answer accuracy.
|
|
---
|
|
|
|
Vision models charge by the token, and images are expensive. A single 1024x1024 image costs ~765 tokens on OpenAI. Headroom's image compression uses a trained ML router to analyze your query and automatically select the optimal compression technique, saving 40-90% of image tokens.
|
|
|
|
## How It Works
|
|
|
|
```
|
|
User uploads image + asks question
|
|
|
|
|
[Query Analysis]
|
|
TrainedRouter (MiniLM from HuggingFace)
|
|
Classifies: "What animal is this?" -> full_low
|
|
|
|
|
[Image Analysis]
|
|
SigLIP analyzes image properties
|
|
(has text? complex? fine details?)
|
|
|
|
|
[Apply Compression]
|
|
OpenAI: detail="low"
|
|
Anthropic: Resize to 512px
|
|
Google: Resize to 768px
|
|
|
|
|
Compressed request to LLM
|
|
```
|
|
|
|
The router is a fine-tuned MiniLM classifier (`chopratejas/technique-router` on HuggingFace) with 93.7% accuracy across 1,157 training examples.
|
|
|
|
## Compression Techniques
|
|
|
|
| Technique | Savings | When Used | Example Query |
|
|
|---|---|---|---|
|
|
| `full_low` | ~87% | General understanding | "What is this?", "Describe the scene" |
|
|
| `preserve` | 0% | Fine details needed | "Count the whiskers", "Read the serial number" |
|
|
| `crop` | 50-90% | Region-specific queries | "What's in the corner?", "Focus on the background" |
|
|
| `transcode` | ~99% | Text extraction | "Read the sign", "Transcribe the document" |
|
|
|
|
## Quick Start
|
|
|
|
### With Headroom Proxy (Zero Code Changes)
|
|
|
|
```bash
|
|
# Start the proxy
|
|
headroom proxy --port 8787
|
|
|
|
# Connect your client -- images are compressed automatically
|
|
ANTHROPIC_BASE_URL=http://localhost:8787 claude
|
|
```
|
|
|
|
### With HeadroomClient
|
|
|
|
```python
|
|
from headroom import HeadroomClient, OpenAIProvider
|
|
from openai import OpenAI
|
|
|
|
client = HeadroomClient(
|
|
original_client=OpenAI(),
|
|
provider=OpenAIProvider(),
|
|
default_mode="optimize",
|
|
)
|
|
|
|
response = client.chat.completions.create(
|
|
model="gpt-4o",
|
|
messages=[{
|
|
"role": "user",
|
|
"content": [
|
|
{"type": "text", "text": "What animal is this?"},
|
|
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}
|
|
]
|
|
}]
|
|
)
|
|
# Image automatically compressed with detail="low" (87% savings)
|
|
```
|
|
|
|
### Direct API
|
|
|
|
```python
|
|
from headroom.image import ImageCompressor
|
|
|
|
compressor = ImageCompressor()
|
|
|
|
# Compress images in messages
|
|
compressed_messages = compressor.compress(messages, provider="openai")
|
|
|
|
# Check savings
|
|
print(f"Saved {compressor.last_savings:.0f}% tokens")
|
|
print(f"Technique: {compressor.last_result.technique.value}")
|
|
```
|
|
|
|
## Provider Support
|
|
|
|
The compressor adapts its strategy per provider:
|
|
|
|
| Provider | Compression Method | Details |
|
|
|---|---|---|
|
|
| **OpenAI** | Sets `detail="low"` | Native detail parameter |
|
|
| **Anthropic** | Resizes to 512px | PIL-based resize |
|
|
| **Google Gemini** | Resizes to 768px | Optimized for Gemini's 768x768 tile system |
|
|
|
|
### Token Savings by Provider
|
|
|
|
**OpenAI** (1024x1024 image):
|
|
|
|
| Technique | Before | After | Savings |
|
|
|---|---|---|---|
|
|
| `full_low` | 765 tokens | 85 tokens | 89% |
|
|
| `preserve` | 765 tokens | 765 tokens | 0% |
|
|
|
|
**Anthropic** (1024x1024 image):
|
|
|
|
| Before | After | Savings |
|
|
|---|---|---|
|
|
| ~1,398 tokens | ~349 tokens | 75% |
|
|
|
|
**Google Gemini** (1536x1536 image):
|
|
|
|
| Before | After | Savings |
|
|
|---|---|---|
|
|
| 1,032 tokens (4 tiles) | 258 tokens (1 tile) | 75% |
|
|
|
|
Unlike the OpenAI and Anthropic figures above (computed from the formulas in
|
|
`headroom/image/tile_optimizer.py` and verified to match this page's numbers
|
|
exactly), Headroom has no Gemini token-counting formula in the codebase --
|
|
these figures describe Google's publicly documented tile pricing, not
|
|
something this repo computes or tests.
|
|
|
|
## Configuration
|
|
|
|
```python
|
|
from headroom.image import ImageCompressor
|
|
|
|
compressor = ImageCompressor(
|
|
model_id="chopratejas/technique-router", # HuggingFace model
|
|
use_siglip=True, # Enable image analysis
|
|
device="cuda", # Use GPU if available (auto, cuda, cpu, mps)
|
|
)
|
|
```
|
|
|
|
### Proxy Configuration
|
|
|
|
```bash
|
|
# Image optimization is part of the ContentRouter path when the optional image
|
|
# dependencies are installed. There are no public proxy CLI image toggles in
|
|
# the current release.
|
|
headroom proxy
|
|
```
|
|
|
|
## Performance
|
|
|
|
| Metric | Value |
|
|
|---|---|
|
|
| Image resize | ~5-20ms (measured ~9ms median for a 1024px -> 512px PIL/LANCZOS resize) |
|
|
| First request | Model download, cached after (duration depends on network) |
|
|
| Router accuracy | 93.7% (fine-tuned MiniLM classifier, 1,157 training examples -- see `headroom/image/trained_router.py`) |
|
|
| Model size | ~100MB (PyTorch MiniLM router) or ~32MB (ONNX INT8 router, preferred in production) |
|
|
| GPU memory (SigLIP) | ~400MB |
|
|
|
|
<Callout type="info" title="Automatic with the proxy">
|
|
When using the Headroom proxy, image compression happens automatically on every request that contains images. No code changes needed.
|
|
</Callout>
|