# Headroom Metrics — Dashboard Guide What each metric shows, so you can build panels against it. **Two endpoints.** Both are on the proxy (default `:8787`). | Surface | How to get it | Use it for | |---|---|---| | **Prometheus** — `GET /metrics` | Always on, no config | Everything below. Start here. | | **OpenTelemetry** — OTLP/HTTP | `HEADROOM_OTEL_METRICS_ENABLED=1` + `pip install "headroom-ai[proxy,otel]"` | Same data, dotted names, plus per-tenant labels | Names differ between them: Prometheus uses `headroom_tokens_saved_total` (**milliseconds** for timings), OTel uses `headroom.proxy.tokens.saved` (**seconds**). Both are listed below. --- ## The savings panel — start here **`headroom.proxy.tokens.saved`** is the headline number. It already combines compression + tool-schema deferral — no need to add anything to it. | Metric | What it shows | |---|---| | **`headroom.proxy.tokens.saved`** *(OTel)* | **Total input tokens Headroom kept out of the request.** Compression + tool savings, combined. This is your hero number. | | `headroom.proxy.savings.usd{source}` *(OTel)* | **Dollars saved**, split by layer: `compression`, `tool_schema`, `output_shaping`, `provider_cache`. Sum for the total. | | `headroom_persistent_savings_tokens_saved_total` | Same tokens-saved number, but **survives proxy restarts**. Use for "lifetime saved" tiles. | | `headroom_persistent_savings_compression_savings_usd_total` | **Lifetime dollars saved**, durable across restarts. | | `headroom_tokens_input_total` | Input tokens actually sent upstream (post-compression). The denominator for a reduction %. | | `headroom_tokens_output_total` | Output tokens returned by the provider. | ```promql # Hero tile: tokens saved per second rate(headroom_tokens_saved_total[5m]) + sum(rate(headroom_savings_attributed_tokens_total{source="tool_search",realized="true"}[5m])) # Context reduction % 100 * rate(headroom_tokens_saved_total[5m]) / clamp_min(rate(headroom_tokens_input_total[5m]) + rate(headroom_tokens_saved_total[5m]), 1) # Lifetime tiles (survive restart) headroom_persistent_savings_tokens_saved_total headroom_persistent_savings_compression_savings_usd_total ``` > **One catch on the Prometheus side.** `headroom_tokens_saved_total` is compression **only** — it leaves out tool-schema deferral. The OTel `headroom.proxy.tokens.saved` includes both. That's why the query above adds the `tool_search` term back in. On tool-heavy workloads the gap is large. --- ## Latency panel All Prometheus timings are in **milliseconds**, exposed as `_sum` / `_count` / `_min` / `_max`. Build means with `rate(sum)/rate(count)`. | Metric | What it shows | |---|---| | **`headroom_overhead_ms_*`** | **Latency Headroom itself adds.** Handler entry → end of compression. Excludes the LLM call. This is the "what does this cost us" number. | | `headroom_latency_ms_*` | Total request duration, including the provider. | | `headroom_ttfb_ms_*` | Time to first byte from upstream. Streaming requests only. | | `headroom_stage_timing_ms_*{path,stage}` | Where time went inside the handler — `compression_first_stage`, `upstream_connect`, `memory_context`, etc. | | `headroom_transform_timing_ms_*{transform}` | Time per compression transform. Use to find a slow transform. | ```promql # Headroom's added overhead, mean ms rate(headroom_overhead_ms_sum[5m]) / rate(headroom_overhead_ms_count[5m]) # End-to-end, mean ms rate(headroom_latency_ms_sum[5m]) / rate(headroom_latency_ms_count[5m]) # Slowest stages topk(5, rate(headroom_stage_timing_ms_sum[5m]) / rate(headroom_stage_timing_ms_count[5m])) ``` > **No percentiles are available.** There are no histogram buckets on `/metrics`, and the OTel histograms ship with default buckets that put every request into one bucket, so `histogram_quantile()` returns nonsense. **Means work fine.** For real p95/p99 today, use the `headroom perf` CLI. > > Also: divide each `_sum` by **its own** `_count`. Overhead and TTFB are only sampled when > 0, so their counts are smaller than the latency count. --- ## Cache panel | Metric | What it shows | |---|---| | `headroom_provider_cache_hit_requests_total{provider}` | Requests that read from the provider's prompt cache. | | `headroom_provider_cache_requests_total{provider}` | Requests with any cache activity. **The correct denominator for hit rate.** | | `headroom_cache_read_tokens_total{provider}` | Tokens served from cache (the discounted ones). | | `headroom_cache_write_tokens_total{provider}` | Tokens written into cache (these carry a premium). | | `headroom_cache_write_ttl_tokens_total{provider,ttl}` | Cache writes split by TTL — `5m` vs `1h`. | | `headroom_uncached_input_tokens_total{provider}` | Input tokens that missed cache entirely. | | `headroom_cache_bust_total` | Requests where compression broke a cached prefix. **Should stay near zero.** | | `headroom_cache_miss_attribution_total{provider,reason}` | Why a cached prefix missed — `ttl_expiry`, `prefix_change`, `unknown`. | ```promql # Cache hit rate by provider sum by (provider) (rate(headroom_provider_cache_hit_requests_total[5m])) / sum by (provider) (rate(headroom_provider_cache_requests_total[5m])) # Compression breaking cache — alert if this rises rate(headroom_cache_bust_total[5m]) ``` > **Don't use `headroom_requests_cached_total` as a hit rate.** It mixes the provider's prompt cache with Headroom's own response cache into one boolean, so it measures neither. --- ## Traffic & health panel | Metric | What it shows | |---|---| | `headroom_requests_total` | **Completed** requests — the denominator for success-side stats, not total traffic. A request that ended in a 4xx, 5xx or 429 never reaches it. Unlabelled. | | `headroom_inbound_requests_total` | Every inbound HTTP request the proxy accepted, whatever the outcome. This is the all-traffic counter. | | `headroom_requests_by_provider{provider}` | Traffic split by provider — `anthropic`, `openai`, `gemini`, `bedrock`… **Completed requests only**, same denominator as `headroom_requests_total`. Use `headroom_requests_failed_total{provider}` for the failure side. | | `headroom_requests_by_model{model}` | Traffic split by model. Capped at 1024 distinct; overflow lands in `model="other"`. Completed requests only. | | `headroom_requests_failed_total{provider}` | Requests that failed upstream — **4xx and 5xx**, excluding 429. Excluded from `headroom_requests_total`, so add it back when you compute a rate. | | `headroom_requests_rate_limited_total{source}` | Requests rejected with 429, by **either** Headroom's own rate limiter (`source="headroom"`) or the upstream provider (`source="upstream"`). Excluded from `headroom_requests_total`. | | `headroom_compression_failed_total{reason}` | Compression failures — `timeout` or `error`. Fails open, so traffic keeps flowing but savings quietly stop. **Worth an alert.** | | `headroom_compression_quarantine_total{event}` | Compression disabled after repeated timeouts — `activated`, `skipped`, `released`. | | `headroom_inbound_requests_active` | In-flight requests, gauge. Counts all HTTP including `/metrics`. | | `headroom_active_ws_sessions` | Live Codex WebSocket sessions, gauge. | ### `headroom_requests_total` is a denominator, not traffic `headroom_requests_total` counts requests that **completed** — it has always been incremented only on the success path, and since the outcome funnel started short-circuiting at `>= 400`, every 4xx, 5xx and 429 drops out of it. So: - `headroom_requests_total` + `headroom_requests_failed_total` + `headroom_requests_rate_limited_total` ≈ the requests the proxy forwarded upstream. - `headroom_inbound_requests_total` is the honest all-traffic counter (it also counts `/metrics`, `/stats` and other non-proxy HTTP). That is why the failure-rate query below adds `failed` back into its own denominator, and why you should not "simplify" it to `… / rate(headroom_requests_total[5m])`: failures are not in `requests_total`, so that form divides failures by successes and over-reports the rate — badly, during exactly the incident you wrote it for (at 100% failures the denominator goes to zero). ```promql # Failure rate. The denominator deliberately re-adds `failed`: a failed request # is NOT in headroom_requests_total, so requests_total alone is successes-only. rate(headroom_requests_failed_total[5m]) / clamp_min(rate(headroom_requests_total[5m]) + rate(headroom_requests_failed_total[5m]), 1) # Failure rate by provider — which upstream is actually broken. sum by (provider) (rate(headroom_requests_failed_total[5m])) / clamp_min( sum by (provider) (rate(headroom_requests_by_provider[5m])) + sum by (provider) (rate(headroom_requests_failed_total[5m])), 1 ) # Who is throttling you: your own limiter (raise the cap) vs the provider # (back off, shard keys). Alert on these separately — they are different actions. sum by (source) (rate(headroom_requests_rate_limited_total[5m])) # Provider throttling only — the one that means "slow down". rate(headroom_requests_rate_limited_total{source="upstream"}[5m]) # Savings silently stopped sum by (reason) (rate(headroom_compression_failed_total[5m])) # Traffic mix sum by (provider) (rate(headroom_requests_by_provider[5m])) ``` > **Migration — these two counters gained labels.** `headroom_requests_failed_total` is now > labelled by `provider` and `headroom_requests_rate_limited_total` by `source`, so an > unaggregated query that used to return one series now returns several, and a panel or > alert that graphed the bare counter will fan out into one line per label value. > `sum without (provider) (rate(headroom_requests_failed_total[5m]))` and > `sum without (source) (rate(headroom_requests_rate_limited_total[5m]))` reproduce the old > single-series behaviour exactly. Both `source` series are exported from process start > (including at zero); `failed` exports one series per provider that has actually failed, so > it has no series at all until the first failure. > > The same split is available outside Prometheus: `/stats` carries > `requests.rate_limited_by_source` and `requests.failed_by_provider` alongside the > unlabelled `requests.rate_limited` / `requests.failed` totals (which are unchanged), the > lifetime aggregate carries the same two maps, and the OTel counters > `headroom.proxy.requests.rate_limited` / `.failed` carry `source` / `provider` attributes. --- ## Anthropic subscription panel Only if you're on an Anthropic OAuth/subscription plan. OTel only, gauges, no labels. | Metric | What it shows | |---|---| | `headroom.subscription.5h_utilization_pct` | How much of the 5-hour rate-limit window is used (0–100). | | `headroom.subscription.7d_utilization_pct` | Same for the 7-day window. | | `headroom.subscription.5h_seconds_to_reset` | Seconds until the 5-hour window resets. | | `headroom.subscription.7d_seconds_to_reset` | Seconds until the 7-day window resets. | | `headroom.subscription.overage_usd` | Extra-usage credits consumed, in dollars. | --- ## Attribution — where savings came from | Metric | What it shows | |---|---| | `headroom_savings_attributed_tokens_total{source,realized}` | Tokens saved, broken out by named source. `source="tool_search"` is tool-schema deferral. | | `headroom_savings_attributed_usd_total{source,realized}` | Dollars saved by source. **Gauge, can go negative** — don't `rate()` it. | | `headroom_savings_attribution_events_total{source,realized}` | How often each source contributed. | | `headroom_waste_signal_tokens_total{signal}` | Wasteful patterns *detected* in the input — `json_bloat`, `base64`, `repetition`, `reread`… This is diagnosis, **not savings**. | These rows *explain* the headline total — they are never added to it. --- ## Compression internals | Metric | What it shows | |---|---| | `headroom.compression.tokens.input` *(OTel)* | Tokens going into the compression pipeline. | | `headroom.compression.tokens.output` *(OTel)* | Tokens coming out. | | `headroom.compression.tokens.saved` *(OTel)* | The difference. Pipeline-level view of compression only. | | `headroom.compression.runs` *(OTel)* | Pipeline executions. Note: **per pipeline run, not per request.** | | `headroom.compression.pipeline.duration` *(OTel, seconds)* | How long the pipeline took. | | `headroom.compression.transforms{transform}` *(OTel)* | Which transforms fired. **High cardinality — drop or aggregate at the collector.** | --- ## Five things that will break a dashboard 1. **Only savings counters survive a restart.** 55 of 60 Prometheus families reset to zero when the proxy restarts. Only `headroom_persistent_savings_*` is durable, and it needs `HEADROOM_WORKSPACE_DIR` on a persistent volume — otherwise it resets on every deploy. 2. **No percentiles anywhere.** Use means. See the latency section. 3. **`headroom_latency_ms` measures differently for streaming.** On streaming requests the timer starts *after* compression, so end-to-end is `latency + overhead`. On non-streaming it's just `latency`. Don't mix both in one panel. 4. **A 5xx erases its own savings.** Requests that fail upstream are dropped from every savings and token counter. During a provider incident, savings rates look artificially clean while throughput falls. 5. **`/metrics` needs auth if you set a proxy token.** With `HEADROOM_PROXY_TOKEN` set, any non-loopback scraper must send `Authorization: Bearer ` (or `X-Headroom-Proxy-Token: `). Loopback is always exempt. The proxy removes the token from every request before forwarding it, so it never reaches a model provider. --- ## Metrics the docs mention that don't exist If panels came back empty, this is probably why. These names appear in the published docs but not in the code: `headroom_compression_ratio` · `headroom_latency_seconds` (and `_bucket`) · `headroom_cache_hits_total` · `headroom_cache_misses_total` · `headroom_cost_usd_total` · the `mode="optimize"` label on `headroom_requests_total` The shipped `examples/grafana/headroom-dashboard.json` also filters every panel on `pool` and `hook` labels that no metric emits — the dropdowns will be permanently empty. Its metric names are otherwise correct. --- ## Setup reference ```bash # Prometheus — nothing to do, GET /metrics is always on # OpenTelemetry pip install "headroom-ai[proxy,otel]" export HEADROOM_OTEL_METRICS_ENABLED=1 export HEADROOM_OTEL_METRICS_ENDPOINT=https://otel.corp.example/v1/metrics export HEADROOM_OTEL_METRICS_HEADERS="authorization=Bearer XXX" export HEADROOM_OTEL_RESOURCE_ATTRIBUTES="service.instance.id=$HOSTNAME" ``` | Variable | Default | Notes | |---|---|---| | `HEADROOM_OTEL_METRICS_ENABLED` | `0` | Master switch | | `HEADROOM_OTEL_METRICS_EXPORTER` | `otlp_http` | Or `console`. No gRPC exporter exists. | | `HEADROOM_OTEL_METRICS_ENDPOINT` | unset | Passed verbatim — `/v1/metrics` is **not** appended | | `HEADROOM_OTEL_METRICS_HEADERS` | unset | `k=v,k2=v2` | | `HEADROOM_OTEL_METRICS_EXPORT_INTERVAL_MS` | `10000` | | | `HEADROOM_OTEL_SERVICE_NAME` | `headroom-proxy` | | | `HEADROOM_OTEL_RESOURCE_ATTRIBUTES` | unset | **Set `service.instance.id` here** — Headroom doesn't, and replicas will collide | Verify with `curl -s localhost:8787/stats | jq .otel`. **Multi-tenant labels:** `register_otel_metric_attribute_provider()` adds request-scoped attributes (tenant, team, cost centre) to every OTel datapoint. Max 16 attributes, 256 chars each. **Air-gapped deployments:** `HEADROOM_OFFLINE=1` refuses every connection Headroom itself decides to make to a destination Headroom itself chose. That is the whole list, not a sample: the anonymous usage beacon (which is **on by default**), the update check, the license/usage reporter, **OTLP metric and Langfuse trace export**, HuggingFace / Kompress / fastembed model and tokenizer downloads (Python and Rust), the remote Kompress endpoint, the `headroom install` release-binary and codebase-memory-mcp downloads, litellm's import-time pricing-table refresh, CrewAI's import-time install tracking, the BFCL eval-dataset fetch and the provider SDK clients the eval harness drives, GitHub Copilot device-flow auth and token exchange, the Anthropic / Codex / Copilot subscription pollers, the OpenAI embedders, the Headroom Cloud compression modes in the ASGI and LiteLLM integrations, and the TLS diagnostics (`headroom doctor --network` endpoint checks and the certificate-chain re-probe after an upstream TLS failure). Four things are deliberately still allowed, and they are the complete set of exceptions: 1. **Your traffic through the proxy.** Requests you send *through* Headroom are still forwarded to the upstream you configured. That is your traffic, not Headroom's, and an air-gapped deployment points it at an on-prem endpoint — refusing it would mean refusing to be a proxy. 2. **Operator-configured local endpoints.** Today that is exactly one thing: the Ollama embedder, whose address comes entirely from your configuration and defaults to `http://localhost:11434`. No hard-coded internet host is permitted under this exception. 3. **Loopback.** Health and readiness probes against your own proxy on `127.0.0.1` (`headroom doctor`, the installers, the MCP sidecar). 4. **Paths an `is_offline()` check already makes unreachable**, where no connection is ever built in the first place. Those four are enumerated with written reasons in `_EGRESS_ALLOWLIST` in `tests/test_offline_egress_chokepoint.py`, which fails the build if a new egress path appears that is neither guarded nor one of them. There is no category for "known violation": a path that can dial the internet with the flag set is a bug. A refusal reaches you as a message — a Click error on the CLI, a named "model unavailable" from a model loader, a logged line from a background poller — never as a traceback and never as a silent degradation. Enforcing the same policy at the network layer as well is still good practice; it is no longer the only thing standing between you and Headroom-initiated egress. litellm's pricing-table refresh is refused by setting `LITELLM_LOCAL_MODEL_COST_MAP=True` before litellm is imported, so it holds only when Headroom is the first thing in the process to import litellm. A host that imports litellm before Headroom, such as a LiteLLM proxy loading the Headroom callback, has made that fetch already and must set the variable itself. Offline, cost and savings figures use the pricing table bundled with the installed litellm release, so a model released after it prices as unknown. OTLP export is refused loudly: with `HEADROOM_OFFLINE=1` set, `HEADROOM_OTEL_METRICS_ENABLED=1` plus the default `otlp_http` exporter makes the proxy exit 78 at startup with an explanation, rather than quietly dropping metrics. There is no exemption for a collector that looks local — an in-cluster address is not reliably distinguishable from an internet one. For metrics under an air-gap, either scrape `localhost:8787/metrics` or set `HEADROOM_OTEL_METRICS_EXPORTER=console`, both of which stay on-box. `HEADROOM_KOMPRESS_ENDPOINT` and `HEADROOM_LANGFUSE_ENABLED` are refused the same way, for the same reason. ---