Files
headroom/docs/metrics-technical-guide.md
3fef15d376 fix(offline): keep litellm's pricing table local under HEADROOM_OFFLINE (#4242)
## Description

`HEADROOM_OFFLINE=1` still lets litellm reach GitHub. `import litellm`
downloads `model_prices_and_context_window.json` from
`raw.githubusercontent.com` during the import (three attempts) unless
`LITELLM_LOCAL_MODEL_COST_MAP` is `true`. Headroom imports litellm from
24 places, among them the pricing helpers, the LiteLLM provider and
backend, the proxy's cost and savings trackers, and both Headroom Cloud
integrations. With the air-gap switch on, the first of those imports
dialled out.

Nothing in the chokepoint suite caught it, because no Headroom-built
client is involved, so no `guard_egress` site can dominate it.
`TestHeadroomCloudCompressionOffline` only passed when an earlier test
had already imported litellm. Run on its own (`pytest
tests/test_offline_egress_chokepoint.py -k HeadroomCloud`), it fails on
`main` with `SocketOpened: ... ('raw.githubusercontent.com', 443)`.

The only lever is the env var, and litellm reads it once, during its own
import. So:

- `pin_litellm_cost_map_local()` in `headroom/offline.py` sets
`LITELLM_LOCAL_MODEL_COST_MAP=True` while `HEADROOM_OFFLINE` is on.
- Every litellm import in `headroom/` calls it first, in the same scope.
A pin only at package import would miss a switch turned on later:
`is_offline()` is read when checked, and
`create_app(ProxyConfig(offline=True))` sets it after `import headroom`.
- The pin is Headroom's, so it is undone: called with the switch off, it
restores the value it replaced if the variable still holds the pin. A
process that leaves offline mode before its first litellm import gets
the online pricing refresh back.
- It overwrites the variable rather than using `setdefault`. litellm
only accepts `true`, so an operator value of `1` (the form every
Headroom switch takes), `False` or an empty string would otherwise
reopen the fetch.
- A meta-test fails the build for any litellm import without a preceding
pin, the same pattern as the existing site scanner.

No tracking issue; found while chasing a test that only failed when run
on its own.

Review found a second import-time leak on the same path:
`headroom/integrations/__init__.py` imports the CrewAI integration
whenever CrewAI is installed, and `import crewai` requests a Scarf pixel
from `api.scarf.sh` on a background thread unless
`CREWAI_DISABLE_TELEMETRY` is `true`. So `import
headroom.integrations.litellm_callback` still dialled out under the
switch. `pin_crewai_telemetry_off()` handles it the same way, through a
shared `_OfflineEnvPin`, and the meta-test now covers `crewai` imports
too.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/offline.py`: new `pin_litellm_cost_map_local()` (a no-op
unless `HEADROOM_OFFLINE` is on), documented in the module's list of
refused egress.
- All 24 litellm import sites call it immediately before importing:
`pricing/`, `providers/` (litellm, anthropic, openai, cohere, google),
`backends/litellm.py`, `proxy/` (cost, savings_tracker, route_advice),
`integrations/litellm_callback.py`, `telemetry/session.py`,
`perf/analyzer.py`, `learn/analyzer.py`, `cli/` (evals, learn) and
`evals/`.
- `pin_crewai_telemetry_off()` forces `CREWAI_DISABLE_TELEMETRY=true`
while offline (restored the same way), called before the one `crewai`
import (`headroom/integrations/crewai/agents.py`). Both pins share
`_OfflineEnvPin`.
- `headroom/__init__.py` also calls it, which covers a host that imports
litellm itself after importing Headroom.
- `tests/test_offline_egress_chokepoint.py`, new
`TestLiteLLMCostMapOffline`:
- subprocess probes (litellm only fetches on its first import, which an
earlier test usually did already) for four litellm-backed modules, with
the switch set both before and after `import headroom`;
  - operator values `1`, `False` and empty;
- leaving offline mode restores the replaced value (absent or `False`),
and an operator change made after the pin is kept;
  - `test_every_litellm_import_is_pinned_first`, the AST meta-test;
- the `offline` fixture now restores `LITELLM_LOCAL_MODEL_COST_MAP` at
teardown.
- `TestCrewAITelemetryOffline`: a stand-in `crewai` package with crewai
1.6.1's import-time tracking (no extra or CI job installs CrewAI),
probed early and late for `headroom.integrations.crewai` and
`headroom.integrations.litellm_callback`, plus operator values, a
restore test, the same probes against a real CrewAI when installed, and
`test_every_crewai_import_is_pinned_first`. A sanity test checks that
the stand-in does dial when online.
- `docs/metrics-technical-guide.md` and `docs/content/docs/proxy.mdx`
add the refresh and CrewAI's tracking to the refused list. The guide
also states the limits: a host that imports litellm *before* Headroom
has already made the fetch and must set the variable itself, and offline
prices come from the table bundled with the installed litellm release.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q tests/test_offline_egress_chokepoint.py tests/test_proxy_hardening.py
216 passed, 3 skipped, 1 warning in 38.44s

$ pytest -q --continue-on-collection-errors tests/test_providers tests/test_proxy tests/test_pricing* tests/test_perf* \
    tests/test_telemetry* tests/test_learn* tests/test_route_advice* tests/test_backend* tests/test_litellm* tests/test_cost* \
    tests/test_savings* tests/test_counterfactual* tests/test_evals* tests/test_integrations*
1742 passed, 337 skipped, 6 warnings, 1 error in 65.33s
# the 1 error is test_langchain_1x_compat.py failing collection (NameError: tool) because langchain_core
# is not installed in my venv; it fails identically on main df11f274

$ ruff check headroom tests/test_offline_egress_chokepoint.py tests/test_proxy_hardening.py   # ruff 0.17.0
All checks passed!
$ ruff format --check headroom tests/test_offline_egress_chokepoint.py tests/test_proxy_hardening.py
573 files already formatted
$ mypy <the 23 changed headroom/ files>
Success: no issues found in 23 source files

# The new tests fail without the fix: removing one pin call, and reverting to setdefault, gives
FAILED ...::test_operator_value_cannot_reopen_the_fetch[1]       assert ['raw.githubusercontent.com', ...] == []
FAILED ...::test_operator_value_cannot_reopen_the_fetch[False]
FAILED ...::test_operator_value_cannot_reopen_the_fetch[]
FAILED ...::test_every_litellm_import_is_pinned_first           names the unpinned import in headroom/backends/litellm.py
```

## Real Behavior Proof

- Environment: macOS arm64, Python 3.10.18 venv, litellm 1.101.0 (the
`uv.lock` version), `main` at df11f274 in a worktree vs this branch, the
same venv for both, `LITELLM_LOCAL_MODEL_COST_MAP` unset unless shown.
- Exact command / steps: a probe script replaces
`socket.create_connection` with a trap that records the address and
raises, runs `import headroom`, sets `HEADROOM_OFFLINE=1` before
(`early`) or after (`late`) that import, then imports the module and
prints the recorded attempts. Run with `HEADROOM_OFFLINE=1 python
probe2.py early headroom.integrations.asgi`, then the same for
`headroom.backends.litellm`, then `python probe2.py late
headroom.backends.litellm`, then `HEADROOM_OFFLINE=1
LITELLM_LOCAL_MODEL_COST_MAP=1 python probe2.py early
headroom.providers.litellm`.
- Observed result: main df11f274 gives `attempts=3
['raw.githubusercontent.com:443']` for all four runs; this branch gives
`attempts=0 []` for all four, with litellm loaded in every case.
- CrewAI: in a venv with crewai 1.6.1 + litellm 1.101.0 (the review
setup) and CrewAI telemetry at its default, `pytest
tests/test_offline_egress_chokepoint.py -k 'LiteLLMCostMap or CrewAI'`
gives 28 passed, including both real-CrewAI imports. Without the
`agents.py` pin, 10 CrewAI checks fail with `attempts=['api.scarf.sh']`.
- Not tested: a live proxy on an air-gapped host. The probe covers the
import paths; the proxy's lazy imports (`proxy/cost.py`,
`proxy/savings_tracker.py`, `proxy/route_advice.py`) are covered by the
meta-test, not by a running proxy.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: stable.
- Stable/default behavior changed: only with HEADROOM_OFFLINE set;
litellm then uses its bundled pricing table, which is what it already
fell back to after the three failed fetches. Without the switch nothing
changes.
- Kill switch / disable path: unset HEADROOM_OFFLINE.
- Unsafe override required: none.
- Qualification impact: none.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- A host process that imports litellm before Headroom (a LiteLLM proxy
that loads the Headroom callback) has already fetched by the time any
Headroom code runs. That connection isn't Headroom's, so the docs state
the condition instead of claiming it.
- `HF_HUB_OFFLINE` and `TRANSFORMERS_OFFLINE` keep their `setdefault` in
`apply_offline_env()`: the Hub path is protected by `guard_egress` in
`onnx_runtime.py`, not by those variables.
- Possible follow-up, not in this PR: show which cost map litellm loaded
(litellm's `get_model_cost_map_source_info()`) in the offline startup
banner or `headroom doctor`.

---------

Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-24 08:45:38 -07:00

19 KiB
Raw Permalink Blame History

Headroom Metrics — Dashboard Guide

What each metric shows, so you can build panels against it.

Two endpoints. Both are on the proxy (default :8787).

Surface How to get it Use it for
Prometheus — GET /metrics Always on, no config Everything below. Start here.
OpenTelemetry — OTLP/HTTP HEADROOM_OTEL_METRICS_ENABLED=1 + pip install "headroom-ai[proxy,otel]" Same data, dotted names, plus per-tenant labels

Names differ between them: Prometheus uses headroom_tokens_saved_total (milliseconds for timings), OTel uses headroom.proxy.tokens.saved (seconds). Both are listed below.


The savings panel — start here

headroom.proxy.tokens.saved is the headline number. It already combines compression + tool-schema deferral — no need to add anything to it.

Metric What it shows
headroom.proxy.tokens.saved (OTel) Total input tokens Headroom kept out of the request. Compression + tool savings, combined. This is your hero number.
headroom.proxy.savings.usd{source} (OTel) Dollars saved, split by layer: compression, tool_schema, output_shaping, provider_cache. Sum for the total.
headroom_persistent_savings_tokens_saved_total Same tokens-saved number, but survives proxy restarts. Use for "lifetime saved" tiles.
headroom_persistent_savings_compression_savings_usd_total Lifetime dollars saved, durable across restarts.
headroom_tokens_input_total Input tokens actually sent upstream (post-compression). The denominator for a reduction %.
headroom_tokens_output_total Output tokens returned by the provider.
# Hero tile: tokens saved per second
rate(headroom_tokens_saved_total[5m])
  + sum(rate(headroom_savings_attributed_tokens_total{source="tool_search",realized="true"}[5m]))

# Context reduction %
100 * rate(headroom_tokens_saved_total[5m])
    / clamp_min(rate(headroom_tokens_input_total[5m]) + rate(headroom_tokens_saved_total[5m]), 1)

# Lifetime tiles (survive restart)
headroom_persistent_savings_tokens_saved_total
headroom_persistent_savings_compression_savings_usd_total

One catch on the Prometheus side. headroom_tokens_saved_total is compression only — it leaves out tool-schema deferral. The OTel headroom.proxy.tokens.saved includes both. That's why the query above adds the tool_search term back in. On tool-heavy workloads the gap is large.


Latency panel

All Prometheus timings are in milliseconds, exposed as _sum / _count / _min / _max. Build means with rate(sum)/rate(count).

Metric What it shows
headroom_overhead_ms_* Latency Headroom itself adds. Handler entry → end of compression. Excludes the LLM call. This is the "what does this cost us" number.
headroom_latency_ms_* Total request duration, including the provider.
headroom_ttfb_ms_* Time to first byte from upstream. Streaming requests only.
headroom_stage_timing_ms_*{path,stage} Where time went inside the handler — compression_first_stage, upstream_connect, memory_context, etc.
headroom_transform_timing_ms_*{transform} Time per compression transform. Use to find a slow transform.
# Headroom's added overhead, mean ms
rate(headroom_overhead_ms_sum[5m]) / rate(headroom_overhead_ms_count[5m])

# End-to-end, mean ms
rate(headroom_latency_ms_sum[5m]) / rate(headroom_latency_ms_count[5m])

# Slowest stages
topk(5, rate(headroom_stage_timing_ms_sum[5m]) / rate(headroom_stage_timing_ms_count[5m]))

No percentiles are available. There are no histogram buckets on /metrics, and the OTel histograms ship with default buckets that put every request into one bucket, so histogram_quantile() returns nonsense. Means work fine. For real p95/p99 today, use the headroom perf CLI.

Also: divide each _sum by its own _count. Overhead and TTFB are only sampled when > 0, so their counts are smaller than the latency count.


Cache panel

Metric What it shows
headroom_provider_cache_hit_requests_total{provider} Requests that read from the provider's prompt cache.
headroom_provider_cache_requests_total{provider} Requests with any cache activity. The correct denominator for hit rate.
headroom_cache_read_tokens_total{provider} Tokens served from cache (the discounted ones).
headroom_cache_write_tokens_total{provider} Tokens written into cache (these carry a premium).
headroom_cache_write_ttl_tokens_total{provider,ttl} Cache writes split by TTL — 5m vs 1h.
headroom_uncached_input_tokens_total{provider} Input tokens that missed cache entirely.
headroom_cache_bust_total Requests where compression broke a cached prefix. Should stay near zero.
headroom_cache_miss_attribution_total{provider,reason} Why a cached prefix missed — ttl_expiry, prefix_change, unknown.
# Cache hit rate by provider
sum by (provider) (rate(headroom_provider_cache_hit_requests_total[5m]))
  / sum by (provider) (rate(headroom_provider_cache_requests_total[5m]))

# Compression breaking cache — alert if this rises
rate(headroom_cache_bust_total[5m])

Don't use headroom_requests_cached_total as a hit rate. It mixes the provider's prompt cache with Headroom's own response cache into one boolean, so it measures neither.


Traffic & health panel

Metric What it shows
headroom_requests_total Completed requests — the denominator for success-side stats, not total traffic. A request that ended in a 4xx, 5xx or 429 never reaches it. Unlabelled.
headroom_inbound_requests_total Every inbound HTTP request the proxy accepted, whatever the outcome. This is the all-traffic counter.
headroom_requests_by_provider{provider} Traffic split by provider — anthropic, openai, gemini, bedrock… Completed requests only, same denominator as headroom_requests_total. Use headroom_requests_failed_total{provider} for the failure side.
headroom_requests_by_model{model} Traffic split by model. Capped at 1024 distinct; overflow lands in model="other". Completed requests only.
headroom_requests_failed_total{provider} Requests that failed upstream — 4xx and 5xx, excluding 429. Excluded from headroom_requests_total, so add it back when you compute a rate.
headroom_requests_rate_limited_total{source} Requests rejected with 429, by either Headroom's own rate limiter (source="headroom") or the upstream provider (source="upstream"). Excluded from headroom_requests_total.
headroom_compression_failed_total{reason} Compression failures — timeout or error. Fails open, so traffic keeps flowing but savings quietly stop. Worth an alert.
headroom_compression_quarantine_total{event} Compression disabled after repeated timeouts — activated, skipped, released.
headroom_inbound_requests_active In-flight requests, gauge. Counts all HTTP including /metrics.
headroom_active_ws_sessions Live Codex WebSocket sessions, gauge.

headroom_requests_total is a denominator, not traffic

headroom_requests_total counts requests that completed — it has always been incremented only on the success path, and since the outcome funnel started short-circuiting at >= 400, every 4xx, 5xx and 429 drops out of it. So:

  • headroom_requests_total + headroom_requests_failed_total + headroom_requests_rate_limited_total ≈ the requests the proxy forwarded upstream.
  • headroom_inbound_requests_total is the honest all-traffic counter (it also counts /metrics, /stats and other non-proxy HTTP).

That is why the failure-rate query below adds failed back into its own denominator, and why you should not "simplify" it to … / rate(headroom_requests_total[5m]): failures are not in requests_total, so that form divides failures by successes and over-reports the rate — badly, during exactly the incident you wrote it for (at 100% failures the denominator goes to zero).

# Failure rate. The denominator deliberately re-adds `failed`: a failed request
# is NOT in headroom_requests_total, so requests_total alone is successes-only.
rate(headroom_requests_failed_total[5m])
  / clamp_min(rate(headroom_requests_total[5m]) + rate(headroom_requests_failed_total[5m]), 1)

# Failure rate by provider — which upstream is actually broken.
sum by (provider) (rate(headroom_requests_failed_total[5m]))
  / clamp_min(
      sum by (provider) (rate(headroom_requests_by_provider[5m]))
        + sum by (provider) (rate(headroom_requests_failed_total[5m])),
      1
    )

# Who is throttling you: your own limiter (raise the cap) vs the provider
# (back off, shard keys). Alert on these separately — they are different actions.
sum by (source) (rate(headroom_requests_rate_limited_total[5m]))

# Provider throttling only — the one that means "slow down".
rate(headroom_requests_rate_limited_total{source="upstream"}[5m])

# Savings silently stopped
sum by (reason) (rate(headroom_compression_failed_total[5m]))

# Traffic mix
sum by (provider) (rate(headroom_requests_by_provider[5m]))

Migration — these two counters gained labels. headroom_requests_failed_total is now labelled by provider and headroom_requests_rate_limited_total by source, so an unaggregated query that used to return one series now returns several, and a panel or alert that graphed the bare counter will fan out into one line per label value. sum without (provider) (rate(headroom_requests_failed_total[5m])) and sum without (source) (rate(headroom_requests_rate_limited_total[5m])) reproduce the old single-series behaviour exactly. Both source series are exported from process start (including at zero); failed exports one series per provider that has actually failed, so it has no series at all until the first failure.

The same split is available outside Prometheus: /stats carries requests.rate_limited_by_source and requests.failed_by_provider alongside the unlabelled requests.rate_limited / requests.failed totals (which are unchanged), the lifetime aggregate carries the same two maps, and the OTel counters headroom.proxy.requests.rate_limited / .failed carry source / provider attributes.


Anthropic subscription panel

Only if you're on an Anthropic OAuth/subscription plan. OTel only, gauges, no labels.

Metric What it shows
headroom.subscription.5h_utilization_pct How much of the 5-hour rate-limit window is used (0–100).
headroom.subscription.7d_utilization_pct Same for the 7-day window.
headroom.subscription.5h_seconds_to_reset Seconds until the 5-hour window resets.
headroom.subscription.7d_seconds_to_reset Seconds until the 7-day window resets.
headroom.subscription.overage_usd Extra-usage credits consumed, in dollars.

Attribution — where savings came from

Metric What it shows
headroom_savings_attributed_tokens_total{source,realized} Tokens saved, broken out by named source. source="tool_search" is tool-schema deferral.
headroom_savings_attributed_usd_total{source,realized} Dollars saved by source. Gauge, can go negative — don't rate() it.
headroom_savings_attribution_events_total{source,realized} How often each source contributed.
headroom_waste_signal_tokens_total{signal} Wasteful patterns detected in the input — json_bloat, base64, repetition, reread… This is diagnosis, not savings.

These rows explain the headline total — they are never added to it.


Compression internals

Metric What it shows
headroom.compression.tokens.input (OTel) Tokens going into the compression pipeline.
headroom.compression.tokens.output (OTel) Tokens coming out.
headroom.compression.tokens.saved (OTel) The difference. Pipeline-level view of compression only.
headroom.compression.runs (OTel) Pipeline executions. Note: per pipeline run, not per request.
headroom.compression.pipeline.duration (OTel, seconds) How long the pipeline took.
headroom.compression.transforms{transform} (OTel) Which transforms fired. High cardinality — drop or aggregate at the collector.

Five things that will break a dashboard

  1. Only savings counters survive a restart. 55 of 60 Prometheus families reset to zero when the proxy restarts. Only headroom_persistent_savings_* is durable, and it needs HEADROOM_WORKSPACE_DIR on a persistent volume — otherwise it resets on every deploy.

  2. No percentiles anywhere. Use means. See the latency section.

  3. headroom_latency_ms measures differently for streaming. On streaming requests the timer starts after compression, so end-to-end is latency + overhead. On non-streaming it's just latency. Don't mix both in one panel.

  4. A 5xx erases its own savings. Requests that fail upstream are dropped from every savings and token counter. During a provider incident, savings rates look artificially clean while throughput falls.

  5. /metrics needs auth if you set a proxy token. With HEADROOM_PROXY_TOKEN set, any non-loopback scraper must send Authorization: Bearer <token> (or X-Headroom-Proxy-Token: <token>). Loopback is always exempt. The proxy removes the token from every request before forwarding it, so it never reaches a model provider.


Metrics the docs mention that don't exist

If panels came back empty, this is probably why. These names appear in the published docs but not in the code:

headroom_compression_ratio · headroom_latency_seconds (and _bucket) · headroom_cache_hits_total · headroom_cache_misses_total · headroom_cost_usd_total · the mode="optimize" label on headroom_requests_total

The shipped examples/grafana/headroom-dashboard.json also filters every panel on pool and hook labels that no metric emits — the dropdowns will be permanently empty. Its metric names are otherwise correct.


Setup reference

# Prometheus — nothing to do, GET /metrics is always on

# OpenTelemetry
pip install "headroom-ai[proxy,otel]"
export HEADROOM_OTEL_METRICS_ENABLED=1
export HEADROOM_OTEL_METRICS_ENDPOINT=https://otel.corp.example/v1/metrics
export HEADROOM_OTEL_METRICS_HEADERS="authorization=Bearer XXX"
export HEADROOM_OTEL_RESOURCE_ATTRIBUTES="service.instance.id=$HOSTNAME"
Variable Default Notes
HEADROOM_OTEL_METRICS_ENABLED 0 Master switch
HEADROOM_OTEL_METRICS_EXPORTER otlp_http Or console. No gRPC exporter exists.
HEADROOM_OTEL_METRICS_ENDPOINT unset Passed verbatim — /v1/metrics is not appended
HEADROOM_OTEL_METRICS_HEADERS unset k=v,k2=v2
HEADROOM_OTEL_METRICS_EXPORT_INTERVAL_MS 10000
HEADROOM_OTEL_SERVICE_NAME headroom-proxy
HEADROOM_OTEL_RESOURCE_ATTRIBUTES unset Set service.instance.id here — Headroom doesn't, and replicas will collide

Verify with curl -s localhost:8787/stats | jq .otel.

Multi-tenant labels: register_otel_metric_attribute_provider() adds request-scoped attributes (tenant, team, cost centre) to every OTel datapoint. Max 16 attributes, 256 chars each.

Air-gapped deployments: HEADROOM_OFFLINE=1 refuses every connection Headroom itself decides to make to a destination Headroom itself chose. That is the whole list, not a sample: the anonymous usage beacon (which is on by default), the update check, the license/usage reporter, OTLP metric and Langfuse trace export, HuggingFace / Kompress / fastembed model and tokenizer downloads (Python and Rust), the remote Kompress endpoint, the headroom install release-binary and codebase-memory-mcp downloads, litellm's import-time pricing-table refresh, CrewAI's import-time install tracking, the BFCL eval-dataset fetch and the provider SDK clients the eval harness drives, GitHub Copilot device-flow auth and token exchange, the Anthropic / Codex / Copilot subscription pollers, the OpenAI embedders, the Headroom Cloud compression modes in the ASGI and LiteLLM integrations, and the TLS diagnostics (headroom doctor --network endpoint checks and the certificate-chain re-probe after an upstream TLS failure).

Four things are deliberately still allowed, and they are the complete set of exceptions:

  1. Your traffic through the proxy. Requests you send through Headroom are still forwarded to the upstream you configured. That is your traffic, not Headroom's, and an air-gapped deployment points it at an on-prem endpoint — refusing it would mean refusing to be a proxy.
  2. Operator-configured local endpoints. Today that is exactly one thing: the Ollama embedder, whose address comes entirely from your configuration and defaults to http://localhost:11434. No hard-coded internet host is permitted under this exception.
  3. Loopback. Health and readiness probes against your own proxy on 127.0.0.1 (headroom doctor, the installers, the MCP sidecar).
  4. Paths an is_offline() check already makes unreachable, where no connection is ever built in the first place.

Those four are enumerated with written reasons in _EGRESS_ALLOWLIST in tests/test_offline_egress_chokepoint.py, which fails the build if a new egress path appears that is neither guarded nor one of them. There is no category for "known violation": a path that can dial the internet with the flag set is a bug. A refusal reaches you as a message — a Click error on the CLI, a named "model unavailable" from a model loader, a logged line from a background poller — never as a traceback and never as a silent degradation. Enforcing the same policy at the network layer as well is still good practice; it is no longer the only thing standing between you and Headroom-initiated egress.

litellm's pricing-table refresh is refused by setting LITELLM_LOCAL_MODEL_COST_MAP=True before litellm is imported, so it holds only when Headroom is the first thing in the process to import litellm. A host that imports litellm before Headroom, such as a LiteLLM proxy loading the Headroom callback, has made that fetch already and must set the variable itself. Offline, cost and savings figures use the pricing table bundled with the installed litellm release, so a model released after it prices as unknown.

OTLP export is refused loudly: with HEADROOM_OFFLINE=1 set, HEADROOM_OTEL_METRICS_ENABLED=1 plus the default otlp_http exporter makes the proxy exit 78 at startup with an explanation, rather than quietly dropping metrics. There is no exemption for a collector that looks local — an in-cluster address is not reliably distinguishable from an internet one. For metrics under an air-gap, either scrape localhost:8787/metrics or set HEADROOM_OTEL_METRICS_EXPORTER=console, both of which stay on-box. HEADROOM_KOMPRESS_ENDPOINT and HEADROOM_LANGFUSE_ENABLED are refused the same way, for the same reason.