Commit Graph
5 Commits
Author SHA1 Message Date
3fef15d376 fix(offline): keep litellm's pricing table local under HEADROOM_OFFLINE (#4242)
## Description

`HEADROOM_OFFLINE=1` still lets litellm reach GitHub. `import litellm`
downloads `model_prices_and_context_window.json` from
`raw.githubusercontent.com` during the import (three attempts) unless
`LITELLM_LOCAL_MODEL_COST_MAP` is `true`. Headroom imports litellm from
24 places, among them the pricing helpers, the LiteLLM provider and
backend, the proxy's cost and savings trackers, and both Headroom Cloud
integrations. With the air-gap switch on, the first of those imports
dialled out.

Nothing in the chokepoint suite caught it, because no Headroom-built
client is involved, so no `guard_egress` site can dominate it.
`TestHeadroomCloudCompressionOffline` only passed when an earlier test
had already imported litellm. Run on its own (`pytest
tests/test_offline_egress_chokepoint.py -k HeadroomCloud`), it fails on
`main` with `SocketOpened: ... ('raw.githubusercontent.com', 443)`.

The only lever is the env var, and litellm reads it once, during its own
import. So:

- `pin_litellm_cost_map_local()` in `headroom/offline.py` sets
`LITELLM_LOCAL_MODEL_COST_MAP=True` while `HEADROOM_OFFLINE` is on.
- Every litellm import in `headroom/` calls it first, in the same scope.
A pin only at package import would miss a switch turned on later:
`is_offline()` is read when checked, and
`create_app(ProxyConfig(offline=True))` sets it after `import headroom`.
- The pin is Headroom's, so it is undone: called with the switch off, it
restores the value it replaced if the variable still holds the pin. A
process that leaves offline mode before its first litellm import gets
the online pricing refresh back.
- It overwrites the variable rather than using `setdefault`. litellm
only accepts `true`, so an operator value of `1` (the form every
Headroom switch takes), `False` or an empty string would otherwise
reopen the fetch.
- A meta-test fails the build for any litellm import without a preceding
pin, the same pattern as the existing site scanner.

No tracking issue; found while chasing a test that only failed when run
on its own.

Review found a second import-time leak on the same path:
`headroom/integrations/__init__.py` imports the CrewAI integration
whenever CrewAI is installed, and `import crewai` requests a Scarf pixel
from `api.scarf.sh` on a background thread unless
`CREWAI_DISABLE_TELEMETRY` is `true`. So `import
headroom.integrations.litellm_callback` still dialled out under the
switch. `pin_crewai_telemetry_off()` handles it the same way, through a
shared `_OfflineEnvPin`, and the meta-test now covers `crewai` imports
too.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/offline.py`: new `pin_litellm_cost_map_local()` (a no-op
unless `HEADROOM_OFFLINE` is on), documented in the module's list of
refused egress.
- All 24 litellm import sites call it immediately before importing:
`pricing/`, `providers/` (litellm, anthropic, openai, cohere, google),
`backends/litellm.py`, `proxy/` (cost, savings_tracker, route_advice),
`integrations/litellm_callback.py`, `telemetry/session.py`,
`perf/analyzer.py`, `learn/analyzer.py`, `cli/` (evals, learn) and
`evals/`.
- `pin_crewai_telemetry_off()` forces `CREWAI_DISABLE_TELEMETRY=true`
while offline (restored the same way), called before the one `crewai`
import (`headroom/integrations/crewai/agents.py`). Both pins share
`_OfflineEnvPin`.
- `headroom/__init__.py` also calls it, which covers a host that imports
litellm itself after importing Headroom.
- `tests/test_offline_egress_chokepoint.py`, new
`TestLiteLLMCostMapOffline`:
- subprocess probes (litellm only fetches on its first import, which an
earlier test usually did already) for four litellm-backed modules, with
the switch set both before and after `import headroom`;
  - operator values `1`, `False` and empty;
- leaving offline mode restores the replaced value (absent or `False`),
and an operator change made after the pin is kept;
  - `test_every_litellm_import_is_pinned_first`, the AST meta-test;
- the `offline` fixture now restores `LITELLM_LOCAL_MODEL_COST_MAP` at
teardown.
- `TestCrewAITelemetryOffline`: a stand-in `crewai` package with crewai
1.6.1's import-time tracking (no extra or CI job installs CrewAI),
probed early and late for `headroom.integrations.crewai` and
`headroom.integrations.litellm_callback`, plus operator values, a
restore test, the same probes against a real CrewAI when installed, and
`test_every_crewai_import_is_pinned_first`. A sanity test checks that
the stand-in does dial when online.
- `docs/metrics-technical-guide.md` and `docs/content/docs/proxy.mdx`
add the refresh and CrewAI's tracking to the refused list. The guide
also states the limits: a host that imports litellm *before* Headroom
has already made the fetch and must set the variable itself, and offline
prices come from the table bundled with the installed litellm release.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q tests/test_offline_egress_chokepoint.py tests/test_proxy_hardening.py
216 passed, 3 skipped, 1 warning in 38.44s

$ pytest -q --continue-on-collection-errors tests/test_providers tests/test_proxy tests/test_pricing* tests/test_perf* \
    tests/test_telemetry* tests/test_learn* tests/test_route_advice* tests/test_backend* tests/test_litellm* tests/test_cost* \
    tests/test_savings* tests/test_counterfactual* tests/test_evals* tests/test_integrations*
1742 passed, 337 skipped, 6 warnings, 1 error in 65.33s
# the 1 error is test_langchain_1x_compat.py failing collection (NameError: tool) because langchain_core
# is not installed in my venv; it fails identically on main df11f274

$ ruff check headroom tests/test_offline_egress_chokepoint.py tests/test_proxy_hardening.py   # ruff 0.17.0
All checks passed!
$ ruff format --check headroom tests/test_offline_egress_chokepoint.py tests/test_proxy_hardening.py
573 files already formatted
$ mypy <the 23 changed headroom/ files>
Success: no issues found in 23 source files

# The new tests fail without the fix: removing one pin call, and reverting to setdefault, gives
FAILED ...::test_operator_value_cannot_reopen_the_fetch[1]       assert ['raw.githubusercontent.com', ...] == []
FAILED ...::test_operator_value_cannot_reopen_the_fetch[False]
FAILED ...::test_operator_value_cannot_reopen_the_fetch[]
FAILED ...::test_every_litellm_import_is_pinned_first           names the unpinned import in headroom/backends/litellm.py
```

## Real Behavior Proof

- Environment: macOS arm64, Python 3.10.18 venv, litellm 1.101.0 (the
`uv.lock` version), `main` at df11f274 in a worktree vs this branch, the
same venv for both, `LITELLM_LOCAL_MODEL_COST_MAP` unset unless shown.
- Exact command / steps: a probe script replaces
`socket.create_connection` with a trap that records the address and
raises, runs `import headroom`, sets `HEADROOM_OFFLINE=1` before
(`early`) or after (`late`) that import, then imports the module and
prints the recorded attempts. Run with `HEADROOM_OFFLINE=1 python
probe2.py early headroom.integrations.asgi`, then the same for
`headroom.backends.litellm`, then `python probe2.py late
headroom.backends.litellm`, then `HEADROOM_OFFLINE=1
LITELLM_LOCAL_MODEL_COST_MAP=1 python probe2.py early
headroom.providers.litellm`.
- Observed result: main df11f274 gives `attempts=3
['raw.githubusercontent.com:443']` for all four runs; this branch gives
`attempts=0 []` for all four, with litellm loaded in every case.
- CrewAI: in a venv with crewai 1.6.1 + litellm 1.101.0 (the review
setup) and CrewAI telemetry at its default, `pytest
tests/test_offline_egress_chokepoint.py -k 'LiteLLMCostMap or CrewAI'`
gives 28 passed, including both real-CrewAI imports. Without the
`agents.py` pin, 10 CrewAI checks fail with `attempts=['api.scarf.sh']`.
- Not tested: a live proxy on an air-gapped host. The probe covers the
import paths; the proxy's lazy imports (`proxy/cost.py`,
`proxy/savings_tracker.py`, `proxy/route_advice.py`) are covered by the
meta-test, not by a running proxy.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: stable.
- Stable/default behavior changed: only with HEADROOM_OFFLINE set;
litellm then uses its bundled pricing table, which is what it already
fell back to after the three failed fetches. Without the switch nothing
changes.
- Kill switch / disable path: unset HEADROOM_OFFLINE.
- Unsafe override required: none.
- Qualification impact: none.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- A host process that imports litellm before Headroom (a LiteLLM proxy
that loads the Headroom callback) has already fetched by the time any
Headroom code runs. That connection isn't Headroom's, so the docs state
the condition instead of claiming it.
- `HF_HUB_OFFLINE` and `TRANSFORMERS_OFFLINE` keep their `setdefault` in
`apply_offline_env()`: the Hub path is protected by `guard_egress` in
`onnx_runtime.py`, not by those variables.
- Possible follow-up, not in this PR: show which cost map litellm loaded
(litellm's `get_model_cost_map_source_info()`) in the offline startup
banner or `headroom doctor`.

---------

Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-24 08:45:38 -07:00
Tejas ChopraandClaude Opus 5 119d1a1939 fix(offline): make HEADROOM_OFFLINE a real air-gap via one chokepoint (#3729)
## Description

**Revised 2026-10-05 after review: merged current `main`, routed main's
new TLS-diagnostic egress (`doctor --network` and the TLS-error chain
re-probe) through `guard_egress`, made the test socket trap let loopback
through so the async offline tests run on Windows, and closed five
model-loader findings from Devin Review.**

`HEADROOM_OFFLINE=1` is documented as an air-gap switch, but several
egress paths never consulted `headroom/offline.py`. This PR makes
`guard_egress` the single chokepoint every Headroom-initiated outbound
connection goes through, with a per-site meta-test that fails on any new
unguarded egress. Full detail of every revision is under *Changes Made*.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change) — outbound calls that previously worked with
`HEADROOM_OFFLINE=1` set now refuse; see *Behaviour change customers can
see*
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

### This revision (2026-10-05, after @JerrettDavis's review of
2026-09-24 and comment of 2026-10-03)

* **Merged current `main`.** Conflicts in `docs/content/docs/proxy.mdx`
(kept main's new rows, this PR's offline row),
`headroom/image/onnx_router.py` (both imports) and
`headroom/proxy/server.py` lifespan (main's rust-core passthrough block,
then this PR's `_configure_observability_or_refuse()`).
* **`doctor --network` / TLS diagnostics now go through the guard.**
`tls_diagnostics.probe_endpoint` calls `guard_egress` before both the
certificate-chain handshake and the `httpx.Client` GET; `headroom
doctor` turns the refusal into one `network · skipped: HEADROOM_OFFLINE
is set, no endpoint was contacted` row (status `skip`, so it does not
fail the doctor) instead of a page of "unreachable". `_probe_uncached` —
the chain handshake used both by doctor and by the request path's
TLS-error re-probe (`describe_upstream_failure`) — guards before its
socket; being diagnostic-only on the request path, it reports `skipped:
…` and the TLS explanation is still produced without the chain. No
allowlist widening: the inventory test passes because the `httpx.Client`
site is now dominated by a guard.
* **Socket trap is Windows-safe.** `no_sockets` now passes connects
whose address is a loopback **IP literal** (the Windows Proactor
self-pipe is a `127.0.0.1` listen + connect) and still fails every other
address, including hostnames such as `localhost`. `TestSocketTrap`
reproduces the Windows path on every platform by forcing
`socket._fallback_socketpair` (the implementation Windows uses for
`socket.socketpair()`) under `asyncio.run()`, and pins that outbound
IPv4/IPv6 `connect` / `connect_ex` / `create_connection` are still
caught.
* **Devin Review findings (`40e2d449`).**
* **Kompress tokenizer bypass.** Cached ONNX weights plus an uncached
ModernBERT tokenizer fell through to an unguarded
`from_pretrained(local_files_only=False)`. That call is now guarded and
becomes `KompressModelNotCached`, the same translation `_hf_artifact`
uses.
* **Kompress ONNX candidates.** An uncached first candidate no longer
ends the loop. The refusal is logged once, then the remaining candidates
are checked cache-only, so a cached fp32 fallback still loads offline.
* **fastembed.** `huggingface_hub` reads `HF_HUB_OFFLINE` once, at
import, into `constants.HF_HUB_OFFLINE`. The cache-only attempt
therefore forces that constant as well as the env var, and restores it
afterwards.
* **Image router.** It now resolves every artifact before publishing the
session, which doubles as the "loaded" flag. A refused secondary
artifact no longer leaves a half-loaded router that never retries.
* **Eval runner.** The local Ollama provider is no longer refused, for
the same operator-endpoint reason as the Ollama embedder. Anthropic and
OpenAI are still refused.
* **Sweep of main for other new egress since this PR was written:** the
scanner's only new finding was `tls_diagnostics.py`. Manually checked
the rest of the diff: the new `reqwest` client in
`crates/headroom-proxy` and the keep-alive `httpx.AsyncClient` in
`proxy/server.py` are upstream forwarding (`user-traffic`); the new HF
tokenizer allowlist in `tokenizers/huggingface.py` sits behind the
existing `is_offline()` gate (`gated`); `port_discovery.py`,
`upstream_pinning.py`, `license_env.py` and `credential_policy.py` open
no new external connections.

### A-2 (S2): the air-gap guarantee was not real

`HEADROOM_OFFLINE=1` is documented as an air-gap switch, but several
egress paths never consulted `headroom/offline.py`. A customer who set
it and believed they were air-gapped was wrong.

**This PR has been revised after an adversarial review found that the
first version's guarantee was largely unenforced.** The review is
reproduced in the fixes below; read the *What this does and does not
promise* section for the honest scope.

### The chokepoint

`headroom/offline.py` provides `guard_egress(purpose, destination)` and
a named `OfflineEgressBlocked`. It refuses loudly — a silent skip is
indistinguishable from success at the call site, which is how these
holes survived. It is always called **before the HTTP client is
constructed**, not merely before the request, so no pooled connection is
ever opened. `crates/headroom-core/src/offline.rs` is the same switch,
same env var, same rules, for the Rust core.

#### Paths routed through it

| path | previously |
| --- | --- |
| `headroom/transforms/kompress_remote.py` | built an `httpx.Client` and
`POST`ed every compressible block to the remote Kompress endpoint |
| `headroom/observability/metrics.py` | started an `OTLPMetricExporter`
plus a background `PeriodicExportingMetricReader` timer |
| `headroom/observability/tracing.py` | started an `OTLPSpanExporter` to
Langfuse Cloud — the metric exporter's twin, missed the first time |
| `headroom/onnx_runtime.py` | `hf_hub_download` on a cache miss: the
**Python** half of the HF fetch, every ONNX model in the tree resolves
through it |
| `crates/…/tokenizer/hf_impl.rs` | fetched `tokenizer.json` from
huggingface.co |
| `crates/…/transforms/kompress.rs` | fetched the **Kompress model**
from huggingface.co |
| `crates/…/relevance/embedding.rs` | fastembed fetched ONNX weights
from huggingface.co |

The three Rust sites are the same shape and the same argument:
`HF_HUB_OFFLINE` only shortcuts the *cache lookup*, and on a miss
`hf-hub` still dials out. The first version of this PR guarded one of
the three and left the other two open.

Remote Kompress is guarded twice — in `__init__`, and again at the top
of `compress()` **outside** the fail-open `try`, because the router
caches one compressor for the process lifetime and `compress()` converts
every exception into a silent passthrough.

`onnx_runtime.hf_hub_download_local_first` is guarded on the **network
fallback only**, so a pre-seeded air-gapped cache still resolves. The
three Rust fetches refuse even on a warm cache: `hf-hub` 0.5 exposes no
cache-hit/cache-miss split at that layer, so "guard before the client
exists" and "serve from the cache" cannot both hold.
`Kompress::from_files` / `HfTokenizer::from_file` are the network-free
paths for a pre-seeded Rust deployment. Splitting the Rust fetches
cache-first is the obvious follow-up and is **not** done here.

### The refusal now survives the fail-open handlers

The original docstring said "anything that catches broad `Exception`
around an egress call should re-raise this". **Nothing did** — there
were zero re-raise sites in the package — and the refusal was swallowed
on the way out of a request by `thinking_compactor._memo_compact`
(wrapped directly around the guarded `kompress.compress` call),
`ContentRouter._kompress_model_ready` (which reported a policy refusal
as "model ready"), the startup preload, and the OpenAI
reasoning-compaction path. A full `/v1/messages` request with
`HEADROOM_OFFLINE=1` + `HEADROOM_KOMPRESS_ENDPOINT` returned **200
uncompressed** with `guard_egress` having fired twice.

**`OfflineEgressBlocked` now derives from `BaseException`, not
`Exception`.** A convention that has to be re-applied at every future
`except Exception:` in a 100k-line codebase is not a guarantee; the type
hierarchy is. Same reason `KeyboardInterrupt` and `SystemExit` sit
outside `Exception`: a policy decision is control flow, not a runtime
error to degrade around. All four sites propagate with no edit to any of
them.

⚠️ **This is a behaviour change for callers.** `except Exception` no
longer catches it anywhere, including in third-party code. `except
BaseException` still does; those are swept by `TestBroadHandlerSweep`,
which requires each to re-raise or carry a written reason.
`content_router`'s native-detect degrade path now re-raises it
explicitly, alongside the cancellation `BaseException`s it already let
through.

#### Where the refusal deliberately becomes a degradation

Escaping `except Exception` is the point, but it made one class of
caller worse. Fetching **public model weights** is not data leaving the
box, and an air-gapped proxy with a cold HF cache would have started
failing user requests where it previously degraded — punishing the
request for a decision the operator made about the host. So each
optional model loader translates the refusal into its own "model
unavailable" error, **explicitly**, with the switch named in the log and
the refusal chained:

| boundary | translated to | the caller's existing degradation |
| --- | --- | --- |
| `kompress_compressor._hf_artifact` | `KompressModelNotCached` | non-ML
compression path |
| `image/onnx_router._hf_artifact` | `RuntimeError` | `ImageCompressor`
→ `Technique.PRESERVE` |
| `memory/adapters/embedders` ONNX load | `RuntimeError` | local
embedder unavailable |
| the two Kompress download daemon threads | logged, `False`/return |
keeps whatever is cached |

Behaviour for a cold air-gapped cache is exactly what it was. What
changed is that the log now names the switch, and the translation is a
reviewed decision at a named boundary rather than something inherited
from whichever `except Exception` happened to be in the stack. The
daemon threads matter separately: an unhandled `BaseException` there
reaches `threading.excepthook` as a bare traceback on every air-gapped
boot.

Paths that **send data off the box** — remote Kompress, OTLP, Langfuse,
the beacon — are untouched and still refuse loudly.

### Startup refuses a contradictory configuration

`configure_otel_metrics` was called **outside** `lifespan`'s `try`,
above the line that sets `app.state.startup_error`, so an
`OfflineEgressBlocked` escaped uvicorn as an unhandled error — killing a
proxy that had been serving traffic, with a traceback instead of an
explanation.

It now runs through `_configure_observability_or_refuse()`, which turns
a refusal into the shape `_check_rust_core` uses ten lines above: what
is wrong, how to fix it, **exit 78** (`EX_CONFIG`). A preflight covers
the remote-Kompress contradiction the same way. So the ordinary case is
settled at startup with an actionable message, and `BaseException` is
the backstop rather than the user experience.

```
FATAL: OTLP metric export (HEADROOM_OTEL_METRICS_ENABLED) needs outbound network access, but HEADROOM_OFFLINE=1 forbids it.
    refused: HEADROOM_OFFLINE is set: refusing outbound network access for OTLP metric export to …
    fix:     set HEADROOM_OTEL_METRICS_EXPORTER=console or scrape /metrics (both stay on-box), or unset HEADROOM_OTEL_METRICS_ENABLED
```

### Python/Rust normalisation parity

Python trimmed with `str.strip()`, Rust with `str::trim()`. Not the same
set: `str.isspace()` includes U+001C–U+001F, the Unicode `White_Space`
property does not, so `HEADROOM_OFFLINE=$'\x1c1'` read as **offline to
Python and online to Rust** — one variable, one half of the process
air-gapped. Both sides now trim an enumerated `TRIM_CHARS` /
`_TRIM_CHARS`, and the parity test compares the trim sets and a shared
normalisation table, not just the accepted tokens. (Case folding needed
no change: no non-ASCII character lowercases into any character of
`1`/`true`/`yes`/`on`.)

### The meta-test, rebuilt

The first version was neutered by one line:

```python
if "guard_egress" in source:
    continue
```

A whole-file exemption keyed on the string appearing *anywhere*,
including in a docstring — the exact blanket exemption the allowlist's
own comment says must never happen. It made the site-count assertion
unreachable for every guarded file, so "adding a second client trips
this test" only held for files with **no** guard at all. Two bypasses
were demonstrated against it: a second unguarded `httpx.Client` in an
already-guarded file, and a brand-new unguarded module with the word in
its docstring.

It now decides **per egress site**:

* **AST, not text.** Comments and docstrings vouch for nothing.
`self.client.post()` is no longer read as `requests.post`, and the
`httpx.post` example in `compress.py`'s docstring is not a site.
* **Dominance.** A site is guarded only when a `guard_egress` call is in
its block or an enclosing one, earlier in that block, in the same
function. A guard in a sibling `if`/`except` arm, in a nested `def`, or
after the site does not count. A guard in a helper the function calls
does not count either — conservative on purpose.
* **`TestSiteScannerRules`** tests the scanner itself, pinning both
demonstrated bypasses plus the one-line `def f(): return
httpx.Client().post(...)` that the old `startswith("def ")` skip made
invisible. A meta-test nobody tests is a comment.
* **Wider callee set:** module-level `httpx` verbs
(`headroom/cli/mcp.py` was a live unmatched site), `hf_hub_download`
(skipping `local_files_only=True`, which opens no socket), fastembed's
`TextEmbedding`, the OTLP exporters, and the OpenAI/Anthropic SDK
clients.
* **`crates/` is swept too** (`TestRustEgressChokepointCoverage`). It
was outside the scan entirely, which is how two Rust downloads stayed
open. Text-level rather than AST — no Rust parser here — so "guarded"
means a `guard_egress` earlier in the same `fn` at no deeper indent.
Weaker, and labelled as such; the runtime assertions live in each
crate's `#[test]`s.
* **Callees are resolved through the module's import table before
matching.** The scanner compared the text `ast.unparse` produced, which
is whatever local name the module bound — so `import httpx as h` hid a
client from a test whose entire job is to notice clients. That was not
hypothetical: `copilot_auth.py` does `from urllib import request as
urllib_request`, so its GitHub calls rendered as
`urllib_request.urlopen` and **five call sites, including the shared
`_urlopen` helper every Copilot request goes through, were invisible**.
`_canonical_callee` rewrites the leading name first; the alias defence
cuts both ways, so `guard_egress as ge` still counts as a guard. Imports
are collected from the whole tree, not just module level, because half
the egress here is imported inside the function that uses it.
* Allowlist counts now mean **unguarded** sites only, so a file can
legitimately guard some of its egress and still carry an entry for the
rest.
* **There is no longer a category for "known violation".**
`test_no_category_permits_a_known_violation` pins the permitted set to
`loopback` / `gated` / `user-traffic` / `operator-endpoint` /
`cache-only` and rejects "out of scope" as a reason.

### The offline policy, in plain words

`HEADROOM_OFFLINE=1` **refuses every connection Headroom itself decides
to make to a destination Headroom itself chose.** That is the rule, not
a sample list: telemetry, update check, license/usage reporter, OTLP and
Langfuse export, HuggingFace / Kompress / fastembed model and tokenizer
downloads (Python and Rust), release-binary and codebase-memory-mcp
downloads, eval dataset downloads and the provider SDK clients the eval
harness drives, GitHub Copilot device-flow auth and token exchange, the
Anthropic / Codex / Copilot subscription pollers, both OpenAI embedders,
both Headroom Cloud compression integrations, and the TLS diagnostics
(`headroom doctor --network` endpoint checks and the certificate-chain
re-probe after an upstream TLS failure).

**Four things are deliberately still allowed, and they are the complete
set of exceptions.** Every one of them is either physically incapable of
leaving the box or is not Headroom-initiated traffic:

| category | what it covers | why it survives the switch |
| --- | --- | --- |
| `user-traffic` | `proxy/server.py` (2 sites) | the caller's own
request forwarded to the upstream the operator configured. A proxy that
refuses this is not a proxy; an air-gapped install points it at an
on-prem endpoint. |
| `operator-endpoint` | `memory/adapters/embedders.py` (1 site) | the
Ollama embedder, whose address comes entirely from operator config and
defaults to `http://localhost:11434`. Exactly one thing qualifies today,
and a hard-coded internet host may never hide here. |
| `loopback` | 8 files, 12 sites | hard-coded `127.0.0.1` probes against
the operator's own local proxy (`headroom doctor`, the installers, the
MCP sidecar, the test harness). |
| `gated` | 3 files, 3 sites | the beacon, the update check and the
usage reporter are already unreachable behind an `is_offline()` check,
so no connection is ever built. |

Plus one mechanical exemption, `cache-only` (`relevance/embedding.py`, 1
site): the first of the two fastembed loads runs with `HF_HUB_OFFLINE`
forced to `1`, so `huggingface_hub` resolves from the local cache or
raises **without opening a socket**; the network retry directly below it
calls `guard_egress`. That is what lets a pre-seeded air-gapped host
keep loading the model while a cold one refuses.

#### What changed since the review

The previous revision of this PR recorded **nine egress-capable files**
under an `unguarded` allowlist category meaning "reachable under
`HEADROOM_OFFLINE`, out of scope for A-2": Copilot auth/token exchange,
three subscription pollers, the two binary/CBM downloads, the
eval-dataset fetch, and both Headroom Cloud compression integrations —
plus the eval harness's LLM clients, both OpenAI embedders and the
fastembed weights. @JerrettDavis blocked on exactly that, and he was
right: recording that a switch does not do what it says is not the same
as making it do it.

**All of them are now guarded, and the `unguarded` category is
deleted.** `test_no_category_permits_a_known_violation` pins the five
permitted categories by name and rejects "out of scope" as an allowlist
reason, so re-introducing a bless-it-and-move-on category takes a
deliberate edit with a reviewer looking at it.

#### ⚠️ Behaviour change customers can see

**Outbound calls that previously worked with `HEADROOM_OFFLINE=1` set
now refuse.** That is the point of the PR, but it is a real change for
anyone who set the flag and kept using these features:

* `headroom auth` / `headroom copilot-auth login` cannot complete a
GitHub device flow, and a cached Copilot OAuth token cannot be exchanged
for an API token. A Copilot subscription is unusable on an air-gapped
box regardless, but the failure now arrives at the moment of the
attempt.
* `headroom install` will not download release binaries or
codebase-memory-mcp. Pre-stage them, or install with the flag unset.
* The Anthropic / Codex / Copilot subscription and quota panels stay
empty, with the reason logged and (for Copilot quota) shown in the
panel's own error slot.
* Headroom Cloud compression mode in the ASGI middleware and the LiteLLM
callback refuses instead of falling through to local compression —
loudly, because silently compressing locally would hide that the
deployment is no longer doing what it was configured to do.
* The eval harness refuses to build provider clients or download BFCL.
* The OpenAI embedders (memory adapter and direct mem0 backend) refuse.
Use the ONNX or Ollama embedder on an air-gapped box.
* `EmbeddingScorer` raises "model unavailable" on a **cold** fastembed
cache. A warm one is unaffected.

#### Refusals reach a human, never a traceback

Following the pattern the earlier commits set for the Kompress download
threads, the model layer and the startup banner:

* **CLI:** `OfflineAwareGroup` (`headroom/cli/main.py`) is the one place
that translates the refusal into a Click `Error:` line, for every
subcommand at once. `OfflineEgressBlocked` stays a `BaseException`
everywhere inside, so nothing else can downgrade it; only the last frame
before the terminal is allowed to make it readable.
* **Background pollers:** each catches `OfflineEgressBlocked`
specifically, calls `offline.note_refusal` (WARNING once per purpose,
DEBUG after, so a 30-second poller does not flood the log), and returns
its ordinary "no data" value. An un-caught `BaseException` out of a bare
`create_task` surfaces as "Task exception was never retrieved" whenever
the GC gets to it — a traceback with no explanation in it.
* **Model loaders:** `EmbeddingScorer._get_model` translates into a
named "model unavailable" `RuntimeError`, the way
`kompress_compressor._hf_artifact` and `image/onnx_router._hf_artifact`
already do.

### 🚩 Two decisions that need a founder call

Both were implemented the **safe** way — refuse, loudly and legibly —
because that is the defensible default for a switch an operator sets to
mean "nothing leaves this box". Neither is obviously right, and a
permissive default should not be chosen quietly:

1. **Should OTLP / telemetry export be permitted under
`HEADROOM_OFFLINE`?** Today it refuses, and a contradictory config
(`HEADROOM_OTEL_METRICS_ENABLED=1` + offline) exits 78 at startup with
an explanation. The argument for permitting it is that a regulated
customer's OTLP collector is usually in-cluster, and losing
observability is a real operational cost. The argument against — and the
reason it is refused today — is that the guard **cannot reliably tell an
in-cluster collector from an internet host**: DNS, service meshes and
egress proxies all blur it, so any exemption is really "trust the
endpoint the config names". A customer with an in-cluster collector
currently has to scrape `localhost:8787/metrics` or use
`HEADROOM_OTEL_METRICS_EXPORTER=console`. If we want to permit it, the
honest shape is an explicit second flag (e.g.
`HEADROOM_OFFLINE_ALLOW_OTLP=1`) rather than widening the policy.
2. **Should the subscription pollers refuse, or degrade silently?**
Today all three refuse and say so once per purpose at WARNING. The
argument for silence is that these are cosmetic panels and a WARNING on
an air-gapped box is noise about a decision the operator already made.
The argument for the current behaviour is that a usage panel that
silently stops updating is exactly the "why is this empty?" confusion
the switch was supposed to end, and once-per-purpose logging bounds the
noise. Changing this is a one-line change in `offline.note_refusal`.

### Tests

`tests/test_offline_egress_chokepoint.py` — 110 tests. Per-path socket
assertions (a `no_sockets` fixture booby-traps
`socket.create_connection`, `socket.socket.connect` and `connect_ex`, so
asserting "no socket" rather than "raises" catches a guard placed after
the client is built), the scanner-rule tests, the two coverage sweeps,
the broad-handler sweep, the startup-refusal tests, the model-loader
degradation tests, the cross-language parity tests, and the doc-claim
test. Plus 8 Rust tests in `headroom-core`.

### Fail-before / pass-after

Per fix, work committed, then sources reverted while keeping the new
tests, run, restore. No `git stash`.

**Meta-test (1).** Added a probe module whose docstring mentions
`guard_egress` and a second unguarded `httpx.Client` to the
already-guarded `kompress_remote.py`. The old meta-test: `3 passed`. The
new one:

```
  headroom/_a2_probe_tmp.py
        line 7: httpx.Client().get("https://evil.example.com")
        (not guarded, not allowlisted)
  headroom/transforms/kompress_remote.py
        line 154: return httpx.Client(timeout=1.0)
        (not guarded, not allowlisted)
```

**Rust guards (2).** Guard calls commented out, tests kept:

```
panicked at crates/headroom-core/src/transforms/kompress.rs:688:
expected an offline refusal, got a network error: no loadable ONNX artifact in
`headroomlabs-ai/this-model-does-not-exist-a2` (tried [...]): request error: http status: 401
```

The model id cannot exist, so a `Hub` error can only come from a real
round-trip. That **401 is the proof a socket was opened to
huggingface.co with `HEADROOM_OFFLINE=1` set**, and `Offline` can only
come from returning before `Api::new()`.

**Refusal / startup (3, 4).**
`headroom/{offline,proxy/server,transforms/content_router}.py` reverted
to the PR tip, tests kept: `11 failed, 6 passed` — including
`test_thinking_compactor_does_not_swallow_it`,
`test_kompress_model_ready_does_not_report_a_refusal_as_ready`,
`test_no_broad_handler_swallows_the_refusal` and all five
`TestStartupRefusal` tests.

**New Python guards (5).**
`headroom/{onnx_runtime,observability/tracing}.py` reverted to
`origin/main`, tests kept: `4 failed` — the two socket tests plus both
coverage assertions.

**Normalisation (6).** Both `offline` modules reverted to the PR tip,
parity tests kept: `6 failed`, with `HEADROOM_OFFLINE="\x1c1"` reading
as offline in Python and not in Rust.

**Docs (7).** Both doc files reverted, test kept:

```
AssertionError: docs/metrics-technical-guide.md says 'disables all outbound traffic', but
_EGRESS_ALLOWLIST still records these as reachable under HEADROOM_OFFLINE: [...16 files...]
```

#### This revision (the review fixes)

Committed first, then `git checkout origin/main --
headroom/{binaries,cli/main,copilot_auth,graph/installer,integrations/asgi,integrations/litellm_callback,memory/adapters/embedders,memory/backends/direct_mem0,relevance/embedding}.py
headroom/{evals,subscription}/` with the new tests kept:

```
23 failed, 79 passed
```

The 23 are the five Copilot-auth socket tests, the four poller tests,
the two install-download tests, the BFCL test, both Headroom Cloud
tests, the two cloud-embedder tests, the four fastembed tests, both
CLI-translation tests — and
`test_every_egress_site_is_guarded_or_allowlisted`, which lists every
reverted file as "(not guarded, not allowlisted)".

The aliased-import blind spot, isolated (canonicalisation stubbed back
to the identity, i.e. the pre-fix behaviour, against the same inputs):

```
before:  INVISIBLE: import httpx as h
         INVISIBLE: from urllib import request as urllib_request
         INVISIBLE: from openai import AsyncOpenAI as AO
         INVISIBLE: import httpx as h (inside a function)
         aliased guard recognised: [False]
         copilot_auth.py sites seen: 4
after:   DETECTED (all four); aliased guard recognised: [True]
         copilot_auth.py sites seen: 6
```

**Pass-after (this revision):** `102 passed` in
`tests/test_offline_egress_chokepoint.py`; `391 passed, 86 skipped`
across the copilot-auth / copilot-quota / codex-rate-limits / binaries /
integrations / CLI subsets; `544 passed, 15 skipped` across the
relevance / embedding / mem0 / asgi / litellm / subscription / installer
subsets; `160 passed, 6 skipped` across the evals and CLI doctor/learn
subsets; `59 passed` across the proxy-startup subset; `cargo test -p
headroom-core offline` → `8 passed`.

**Pass-after (earlier revisions):** `70 passed` (pytest), `942 passed`
(`cargo test -p headroom-core --lib`), `720 passed, 4 skipped` across
the offline/otel/kompress/telemetry/onnx/tracing/relevance subsets, and
`1868 passed, 30 skipped` across the
kompress/offline/image/embedder/memory subsets after the model-loader
boundary change. The one failure in the full run,
`test_mcp_reconcile.py::test_ordinary_install_does_not_adopt_serena`,
reproduces unchanged at the PR tip and is unrelated. Cost/pricing tests
are the known `origin/main` litellm breakage (#3722) and were not run.

### Checks

- `uvx ruff@0.16.7 check .` → All checks passed
- `uvx ruff@0.16.7 format --check .` → 1622 files already formatted
- `cargo fmt --check` → clean; `cargo clippy -p headroom-core
--all-targets` → no warnings
- `CHANGELOG.md` untouched
- CodeQL: the four high alerts were all in
`tests/test_offline_egress_chokepoint.py` and are fixed honestly rather
than suppressed. Three `py/incomplete-url-substring-sanitization` host
assertions (`"host" in url`, which is also true of
`https://host.attacker.test`) now parse with `urllib.parse.urlsplit` and
compare the hostname exactly — which is what the tests meant. The
`py/redos` regex scraping Rust `char` literals had a nested `(...)+`
that can backtrack exponentially; a Rust char literal is exactly one
character, so it is now a single bounded alternation.

### Follow-ups (not in this PR)

1. ~~Guard the 16 `unguarded` files, each with its own user-facing
message.~~ Done in this PR; the `unguarded` category is deleted.
2. Split the three Rust HF fetches cache-first so a pre-seeded
air-gapped cache resolves without egress, the way
`hf_hub_download_local_first` already does on the Python side.
3. The Rust sweep is a text heuristic. An AST-level dominance check (via
`syn`, in a Rust test) would make the two halves symmetric.
4. The control-plane heartbeat lives in a different repository and is
still out of scope.
5. The scanner has no entry for transformers `from_pretrained` /
`SentenceTransformer`. The Kompress tokenizer site Devin found is
guarded now, but `models/ml_models.py` (SigLIP, technique router,
sentence-transformers) and the Kompress PyTorch backend still rely on
`HF_HUB_OFFLINE` alone, which `apply_offline_env` sets with `setdefault`
and only in the proxy process. Each needs the cache-first split the
tokenizer has, plus a scanner entry.


<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3729"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

## Testing

- [x] Unit tests pass (`pytest`) — subsets below
- [x] Linting passes (`ruff check .`) — touched files, `ruff@0.16.7`
- [ ] Type checking passes (`mypy headroom`) — not run
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# this revision
$ LITELLM_LOCAL_MODEL_COST_MAP=True uv run --frozen --extra dev --extra proxy pytest \
    tests/test_offline_egress_chokepoint.py tests/test_cli_doctor.py tests/test_corporate_tls.py -q
245 passed
# after the Devin fixes
$ pytest tests/test_offline_egress_chokepoint.py + kompress/onnx_router/relevance/embedding/evals subsets
361 passed, 6 skipped
# fail-before for the Devin fixes (4 sources reverted, tests kept): 6 failed + the fallback test (fails once its fake
# raises OSError instead of LocalEntryNotFoundError, which conftest converts to a skip)

# fail-before, TLS guard (tls_diagnostics.py + doctor.py reverted, tests kept)
FAILED TestDoctorNetworkOffline::test_endpoint_check_refuses_before_any_connection
FAILED TestDoctorNetworkOffline::test_doctor_reports_the_network_checks_as_skipped
FAILED TestDoctorNetworkOffline::test_chain_probe_opens_no_socket
FAILED TestEgressChokepointCoverage::test_every_egress_site_is_guarded_or_allowlisted
   headroom/proxy/tls_diagnostics.py line 473: with httpx.Client(...)  (not guarded, not allowlisted)
# with only tls_diagnostics.py reverted, the doctor test fails on content, not on a missing helper:
   [('net api.anthropic.com', 'fail'), ('net api.openai.com', 'fail'), ...] != [('network', 'skip')]

# fail-before, socket trap (old fixture body, new tests kept)
FAILED TestSocketTrap::test_event_loop_wakeup_pipe_is_not_egress
   SocketOpened: outbound connection attempted while offline: (<socket ...>, ('127.0.0.1', 50151))
# — the same csock.connect inside socketpair() that Jerrett's Windows trace shows.

$ uvx ruff@0.16.7 check / format --check (touched files)
All checks passed! / 4 files already formatted
```

Earlier-revision test evidence is in *Fail-before / pass-after* under
*Changes Made*.

## Real Behavior Proof

- Environment: macOS, Python 3.14, this branch (merged with `main` at
`befdb52f`), no proxy running.
- Exact command / steps: `HEADROOM_OFFLINE=1 headroom doctor --network
--port 59999`, run for real. A second run went through
`headroom.cli.main` with a `sys.addaudithook` recording every
`socket.getaddrinfo` / `socket.connect`.
- Observed result: the doctor table shows `network · skip · skipped:
HEADROOM_OFFLINE is set, no endpoint was contacted`, with the refusal
naming `https://api.anthropic.com/v1/models` as the hint. The audit hook
recorded only `127.0.0.1` lookups and connects (the doctor's own proxy
probes on :59999 and :8787) and **zero external lookups or connects**.
Exit code 2 comes from the unrelated "proxy not reachable" failures.
- Not tested: Windows itself. The Windows mechanism is reproduced by
forcing `socket._fallback_socketpair`, which is the code Windows runs
for `socket.socketpair()`; the CI Windows job (`test-windows-newline`)
only runs `windows_newline`-marked tests and was not extended here.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; behaviour keyed on the existing
`HEADROOM_OFFLINE` switch.
- Minimum rollout channel: stable.
- Stable/default behavior changed: only when `HEADROOM_OFFLINE` is set
(refusals listed above). Unset, nothing changes.
- Kill switch / disable path: unset `HEADROOM_OFFLINE`.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Two founder decisions remain flagged above (OTLP under offline; pollers
loud vs silent). Windows CI coverage for this module is not added: no
unit-test job runs the full module on Windows today, and adding one is a
workflow change better made separately.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-05 15:20:39 -05:00
Satya BharadwajandTejas Chopra 0147cf0220 fix(security): stop forwarding the proxy token to upstream providers (#3891)
## Description

`HEADROOM_PROXY_TOKEN` is accepted as either `X-Headroom-Proxy-Token:
<token>` or `Authorization: Bearer <token>` (`read_proxy_token`,
`server.py`). The provider handlers build their upstream headers from
the inbound request and only strip `x-headroom-*`, so a token sent in
`Authorization` was forwarded verbatim to Anthropic / OpenAI / the
catch-all upstream.

That form is not unusual:
- `docs/metrics-technical-guide.md` documents `Authorization: Bearer
<token>`.
- The TypeScript SDK (`sdk/typescript/src/client.ts`, `rawFetch`) sets
`Authorization: Bearer <apiKey>` whenever the caller has no provider
credential.
- An Anthropic client using `x-api-key` for the provider can use
`Authorization` for the proxy.

In each case the proxy's shared secret ended up in a third party's
request logs. For Anthropic it also changed auth-mode classification,
because an extra bearer turned a PAYG request into OAuth.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **`scrub_proxy_token_headers(scope, token_bytes)`** in
`headroom/proxy/server.py` removes the credential from the ASGI scope
once, so no handler or transport can forward it. About 96 call sites
build upstream headers from `request.headers`, so the fix is not made
per call site.
- `Authorization` is dropped only when it is `Bearer <configured
token>`. Parsing matches `read_proxy_token` (case-insensitive scheme,
stripped credential) and the comparison is constant-time. Any other
value is a provider credential and is kept.
- `X-Headroom-Proxy-Token` is always dropped, including when
`HEADROOM_STRIP_INTERNAL_HEADERS=disabled`.
- **The HTTP `_security_gate`** calls it after the token check and on
the exempt paths (loopback callers, health probes), just before
`call_next`.
- **`WebSocketAuthMiddleware`** calls it on the loopback, authenticated
and no-token paths before handing off to the app. This covers the
`/v1/responses` and `/v1/live` WebSocket relays.
- **Docs:** `docs/content/docs/proxy.mdx` and
`docs/metrics-technical-guide.md` now list both header forms and say the
token is removed before forwarding.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

New tests in `tests/test_proxy_hardening.py`:
- **End-to-end (`TestProxyTokenIsNotForwardedUpstream`):** these capture
the request that actually leaves the proxy via a mock `httpx` transport
installed after app startup. Cases:
- `/v1/messages` with the token in `Authorization`, and with a lowercase
`bearer` scheme
- `/v1/chat/completions` with `X-Headroom-Proxy-Token` plus a provider
bearer
  - `HEADROOM_STRIP_INTERNAL_HEADERS=disabled`
  - a loopback caller
  - catch-all passthrough (`GET /v1/models`)
- **Helper unit tests (`TestScrubProxyTokenHeaders`):** non-matching
bearer, non-bearer scheme, duplicate `Authorization`, no configured
token, non-ASCII value.
- **WebSocket (`TestWebSocketScrubsProxyToken`):** middleware-level for
loopback and remote callers, plus route-level on `/v1/responses`.

With the scrub stubbed out, 12 of the new tests fail. The one that still
passes (custom header, default strip mode) is already covered by the
existing `x-headroom-*` strip.

### Test Output

```text
$ pytest -q tests/test_proxy_hardening.py
48 passed, 1 warning in 13.90s

$ pytest -q tests/test_proxy*.py tests/test_identity_resolution.py tests/test_codex_live.py
4 failed, 947 passed, 107 skipped, 10 warnings in 167.26s

$ ruff check . && ruff format --check .
All checks passed!
1653 files already formatted

$ mypy headroom --ignore-missing-imports
Success: no issues found in 542 source files
```

The 4 failures are sandbox network blocks (tiktoken download from
`openaipublic.blob.core.windows.net`, real upstream calls). They fail
identically on the parent commit without this change:
-
`test_proxy_ccr.py::TestEndToEndTOINIntegration::test_pipeline_compresses_tool_output_and_records_toin`
-
`test_proxy_mode_benchmark.py::test_local_mode_benchmark_shows_compression_and_cache_tradeoff`
-
`test_proxy_passthrough_integration.py::TestPassthroughErrorHandling::test_missing_auth_header_openai`
-
`test_proxy_passthrough_integration.py::test_head_root_returns_200_not_405`

## Real Behavior Proof

- Environment: Linux, Python 3.11, editable install (`.[proxy]`),
FastAPI `TestClient` with a capturing `httpx` transport as the upstream.
- Exact command / steps: `pytest -q tests/test_proxy_hardening.py -k
"NotForwardedUpstream or Scrub"`
- Observed result: the captured upstream request has no `authorization`
header when it carried the proxy token, and keeps `x-api-key` or the
provider bearer unchanged.
- Not tested: a live proxy against real Anthropic/OpenAI endpoints.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is a security fix in the
inbound auth middleware.
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, but only for requests carrying
the proxy token. That credential is no longer forwarded upstream. With
no `HEADROOM_PROXY_TOKEN` configured, the only change is that
`X-Headroom-Proxy-Token` is now dropped even when
`HEADROOM_STRIP_INTERNAL_HEADERS=disabled`.
- Kill switch / disable path: none (deliberately; forwarding the secret
is never correct).
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- **Edge case:** an operator who set `HEADROOM_PROXY_TOKEN` to the same
value as their provider API key will now have that `Authorization`
header removed. The two are indistinguishable, and reusing the provider
key as the proxy secret is exactly the leak this closes. Such
deployments should set a distinct proxy token.
- **Why run after the gate:** the scrub runs *after* the credential
check rather than as a separate outer middleware. An outer middleware
would hide the token from the gate and turn authenticated requests into
401s.
- **Follow-up (not in this PR):** `sdk/typescript/src/client.ts` could
stop mirroring the proxy token into `Authorization` now that
`X-Headroom-Proxy-Token` is always sent. It is harmless after this
change.
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3891"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Tejas Chopra <tejas@headroomlabs.ai>
2026-10-02 11:00:02 -05:00
Tejas ChopraandClaude Opus 5 5ff4ea1ef9 fix(metrics): split rate-limited by source and label failures by provider (#3707)
Closes #3696. #3615 moved the outcome-funnel short-circuit from `>= 500`
to `>= 400` — correct, and staying — but it silently changed what two
metrics mean and the docs were never updated.

**⚠️ This changes exported series. See Migration.**

## What was wrong

- `headroom_requests_failed_total` was documented as "Upstream 5xx
errors"; it now counts 4xx too.
- `headroom_requests_rate_limited_total` was documented as "Requests
**Headroom** rejected via its own rate limiter (**not upstream 429s**)"
— #3615 inverted exactly that.
- 4xx also dropped out of `headroom_requests_by_provider` (only
`record_request` touches it), leaving Prometheus with no way to tell
which upstream was failing.

## Changes


**`headroom_requests_rate_limited_total{source="headroom"|"upstream"}`**
— two 429s mean opposite things to an operator: `headroom` is our
limiter refusing (raise the cap), `upstream` is the provider refusing
(back off / shard keys). Closed label set, so cardinality stays bounded;
unknown values clamp to `headroom`, the counter's pre-#3615 meaning, so
an un-updated caller cannot invent a label. Threaded at all five call
sites (`openai.py:3554, 5681`, `anthropic.py:1256`, `gemini.py:405` →
`headroom`; `outcome.py:543`, the #3615 funnel → `upstream`).

**`headroom_requests_failed_total{provider}`** — uses
`outcome.provider`, already Copilot-relabelled by the funnel, so no
second convention. Matches the existing
`upstream_connection_errors_by_provider` shape.

**All four backends**, so no sink reports a differently-shaped counter:
Prometheus, persistent/lifetime
(`SavingsTracker.record_lifetime_rate_limited`), OTel (attribute on
`headroom.proxy.requests.rate_limited`), and `/stats` JSON.
`SCHEMA_VERSION` stays 5 — the new maps are purely additive, and old
state files load with them empty rather than back-filling invented
history.

**Docs**: `docs/metrics-technical-guide.md`,
`docs/content/docs/metrics.mdx`, `wiki/metrics.md`, plus tooltips on the
dashboard's Failed / Rate Limited tiles so the UI stops implying "Rate
Limited" means Headroom only. `requests_total` is now documented as a
completed-request denominator with `inbound_requests_total` named as the
all-traffic counter, and the failure-rate PromQL carries a "why" on its
denominator so nobody "simplifies" it. Checked and found clean:
`examples/grafana/headroom-dashboard.json`, `docs/observability.md`,
`docs/content/docs/proxy.mdx`, `wiki/proxy.md`, `README.md`.

## Second commit: a bug found while reviewing the first

Adding the label turned `headroom_requests_failed_total` from a counter
that **always** exported `0` into one that exports **no sample at all**
on a healthy proxy — confirmed by rendering `/metrics` on a fresh
instance:

```
headroom_requests_failed_total:       0 sample(s)
headroom_requests_rate_limited_total: 2 sample(s)
```

That breaks the failure-rate query documented in this very PR: an empty
numerator empties the whole PromQL expression, so the panel reads "No
data" instead of 0% while everything is fine, and `absent()` alerts
would fire on healthy proxies. The rate-limit series avoided this by
seeding its closed label set; providers are open-ended so the same trick
does not apply.

Fix: seed the `unknown` bucket at zero. It is a real bucket —
`record_failed` uses it for unattributed failures — so this invents no
provider. Guarded by a new test.

## Migration

Both counters now export **only** labelled series; an unlabelled sample
alongside would double-count under `sum()`. Unaggregated queries return
multiple series — `sum without (source)` / `sum without (provider)`
reproduces the previous value exactly. Noted in the docs and an mdx
Callout.

## Verification

- **16 tests fail on pre-change source** (verified by reverting only
`headroom/`), pass after.
- Targeted metrics suites → **58 passed**. Broad sweep `-k "metric or
stats or outcome or rate_limit or savings_tracker or persistent"` → 634
passed / 44 skipped. Handler, dashboard and savings-history suites → 90
+ 79 passed.
- Ruff 0.16.7 (the CI pin, via `scripts/verify-ruff-version.py
--print-version`): clean.

`CHANGELOG.md` intentionally untouched — `changelog-guard.yml` fails any
PR that edits it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 07:26:40 -07:00
250ede2f7f fix(reporting): show net vs gross savings, real skip thresholds, and the effective profile (#3123)
## Description

Six reporting/config defects found while investigating a user reporting
~1% savings on Claude Code. **None of these changes how much Headroom
compresses** — all of them change whether an operator can tell what it
did. Every one was found by reading that user's own 227,777 lines of
proxy logs against the code.

## Changes Made

- **`perf/analyzer`: parse and render `tok_inflated`.** Every PERF line
carried it; nothing downstream read it. The report could print
`321,239,562 -> 313,274,727` directly above `8,455,763 saved` — two
figures that differ by exactly the 490,928 tokens of inflation it
omitted.
- **`content_router`: report the real skip thresholds.** The routing
summary hardcoded `skipped (<50 words)` regardless of what was in force.
Wrong number (the message gate is `min_tokens`, 10–250 by profile),
wrong unit (tokens and characters, never words), and it merged two
different gates under one label.
- **`perf/analyzer`: disclose that Transform Effectiveness is partial.**
It is built only from `pipeline.py`'s `Transform NAME:` lines.
`compression_units.py` / `compression_batches.py` contain zero logging
calls, so the table read `content_router: 189,783 saved` against a PERF
total 44x larger. Reports the divergence rather than a coverage ratio —
the two are different populations and neither contains the other (those
lines carry no request_id, fire per stage, and are emitted before the
forwarder decides).
- **`perf/analyzer`: disclose the routing denominator.** Percentages
were taken over 4 of the router's 17 outcome buckets, silently dropping
buckets larger than several it displayed.
- **`savings_tracker`: stop dropping tool-schema dollars.**
`estimate_request_savings_usd` prices four buckets; `record_request`
read three. `tool_schema` was computed and discarded, so a quarter of
the token headline never reached "Cost saved". The two inputs are
disjoint (verified at the call site), so this is additive, not
double-counting.
- **`agent_savings`: an unknown profile no longer degrades to
`balanced`.** `balanced` is a different product posture from the default
`coding`: cache→token mode, dedup off, tool-search off, user messages
uncompressed, message floor 25x higher, block floor 20x higher. A typo
in `HEADROOM_SAVINGS_PROFILE` silently reconfigured the whole proxy. Now
degrades to `DEFAULT_PROFILE` and names the resolved profile in the
warning.
- **`agent_savings`: give `min_chars_for_block` a config-object path.**
Every other router pipeline kwarg travels on the config object; this one
alone was env-only, so an unseeded proxy applied every sibling `coding`
knob while this floor stayed at 500 instead of 25.
- **`server`: log the resolved compression posture at startup**, reading
cross-turn dedup off the constructed router rather than the environment
(the router resolves it as `config OR env`, so reading env alone would
be a guess).

## Testing

- [x] Unit tests pass, [x] ruff, [x] mypy, [x] new tests added

```text
uv run pytest tests/ -k "content_router or agent_savings or perf or analyzer or savings or proxy_server or cli_perf or prometheus"
620 passed, 25 skipped
uv run mypy headroom  # Success
uv run ruff check . && ruff format --check .  # clean
```

## Real behavior proof

- **Setup:** macOS arm64, Python 3.12, this branch. Input: 60 MB /
227,777 lines of real proxy logs from the reporting user (6 rotated
files, 2,792 PERF lines, 2026-08-17 → 2026-08-19).
- **Steps:** pointed `headroom.perf.analyzer.LOG_DIR` at that directory
and rendered the report before and after the patch.
- **After-fix output (real data, unmodified):**

```text
Requests:     2792
Tokens:       321,288,161 -> 313,323,326 (2.6% messages)
Tokens saved: 11,158,901 (3.4% reduction)
  · inflated      490,928 (net message reduction 7,964,835)
  · messages       8,455,763
  · tool schemas   2,703,138
  ! stage-level total 190,641 != PERF message total 8,455,763 — this table sees only
    engines that emit a Transform line, counts per stage, and does not check whether
    the mutation shipped
  Skipped:     44641 (77%) — below size floor
  (shares are of these 4 buckets only, n=58319; see `[router] route_counts=` for the
   full outcome space)
```

The arithmetic now closes on the page: `8,455,763 - 490,928 =
7,964,835`, matching the token delta exactly. Before the patch none of
the three annotated lines existed and the `Skipped` line claimed `<50
words`.

- **Profile resolution verified by execution**, not inspection —
subprocesses with controlled env:

```text
vanilla (nothing set)          mode=cache dedupe=1 tool_search=1 min_tokens=10  min_chars=25
HEADROOM_SAVINGS_PROFILE=coding  mode=cache dedupe=1 tool_search=1 min_tokens=10  min_chars=25
unknown profile name (before)  mode=token dedupe=0 tool_search=0 min_tokens=250 min_chars=500
unknown profile name (after)   -> resolves to `coding`, warning names it
coding, seeding never runs     min_chars=25 (was 500 before this patch)
```

- **Not tested:** live paid Anthropic traffic. These are
reporting/config surfaces; the wire path is untouched by this PR.

## Review readiness

- [x] Self-reviewed. Three overclaims in my own first draft were
corrected before this PR: a false subset claim in the Transform
Effectiveness note, a comment asserting `min_chars_for_block` was the
*only* env-only field (it is the only env-only *router pipeline kwarg*;
`cross_turn_dedup`, `tool_search`, `protect_reads`, `code_aware`,
`effort_router`, `lossless` remain env-only via a different mechanism
and are **not** fixed here), and a money-path expression that relied on
`a + b if c else d` grouping.

## Known remaining (deliberately out of scope)

- `Requests: N` still overcounts: the Codex WS forwarder reuses one
`request_id` across every turn (one observed 156x), plus ~18 duplicate
PERF emissions.
- `compression_units.py` / `compression_batches.py` remain unlogged —
this PR *discloses* the blind spot rather than closing it.
- The headline stays **gross**. True net is `11,158,901 - 490,928 =
10,667,973` (3.3%, not 3.4%). Making net the headline lowers every
user's reported savings ~4.4%; that is a product call, not mine, so the
inflation is surfaced beside it instead.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-19 00:16:37 -07:00