fix(offline): make HEADROOM_OFFLINE a real air-gap via one chokepoint (#3729)

## Description

**Revised 2026-10-05 after review: merged current `main`, routed main's
new TLS-diagnostic egress (`doctor --network` and the TLS-error chain
re-probe) through `guard_egress`, made the test socket trap let loopback
through so the async offline tests run on Windows, and closed five
model-loader findings from Devin Review.**

`HEADROOM_OFFLINE=1` is documented as an air-gap switch, but several
egress paths never consulted `headroom/offline.py`. This PR makes
`guard_egress` the single chokepoint every Headroom-initiated outbound
connection goes through, with a per-site meta-test that fails on any new
unguarded egress. Full detail of every revision is under *Changes Made*.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change) — outbound calls that previously worked with
`HEADROOM_OFFLINE=1` set now refuse; see *Behaviour change customers can
see*
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

### This revision (2026-10-05, after @JerrettDavis's review of
2026-09-24 and comment of 2026-10-03)

* **Merged current `main`.** Conflicts in `docs/content/docs/proxy.mdx`
(kept main's new rows, this PR's offline row),
`headroom/image/onnx_router.py` (both imports) and
`headroom/proxy/server.py` lifespan (main's rust-core passthrough block,
then this PR's `_configure_observability_or_refuse()`).
* **`doctor --network` / TLS diagnostics now go through the guard.**
`tls_diagnostics.probe_endpoint` calls `guard_egress` before both the
certificate-chain handshake and the `httpx.Client` GET; `headroom
doctor` turns the refusal into one `network · skipped: HEADROOM_OFFLINE
is set, no endpoint was contacted` row (status `skip`, so it does not
fail the doctor) instead of a page of "unreachable". `_probe_uncached` —
the chain handshake used both by doctor and by the request path's
TLS-error re-probe (`describe_upstream_failure`) — guards before its
socket; being diagnostic-only on the request path, it reports `skipped:
…` and the TLS explanation is still produced without the chain. No
allowlist widening: the inventory test passes because the `httpx.Client`
site is now dominated by a guard.
* **Socket trap is Windows-safe.** `no_sockets` now passes connects
whose address is a loopback **IP literal** (the Windows Proactor
self-pipe is a `127.0.0.1` listen + connect) and still fails every other
address, including hostnames such as `localhost`. `TestSocketTrap`
reproduces the Windows path on every platform by forcing
`socket._fallback_socketpair` (the implementation Windows uses for
`socket.socketpair()`) under `asyncio.run()`, and pins that outbound
IPv4/IPv6 `connect` / `connect_ex` / `create_connection` are still
caught.
* **Devin Review findings (`40e2d449`).**
* **Kompress tokenizer bypass.** Cached ONNX weights plus an uncached
ModernBERT tokenizer fell through to an unguarded
`from_pretrained(local_files_only=False)`. That call is now guarded and
becomes `KompressModelNotCached`, the same translation `_hf_artifact`
uses.
* **Kompress ONNX candidates.** An uncached first candidate no longer
ends the loop. The refusal is logged once, then the remaining candidates
are checked cache-only, so a cached fp32 fallback still loads offline.
* **fastembed.** `huggingface_hub` reads `HF_HUB_OFFLINE` once, at
import, into `constants.HF_HUB_OFFLINE`. The cache-only attempt
therefore forces that constant as well as the env var, and restores it
afterwards.
* **Image router.** It now resolves every artifact before publishing the
session, which doubles as the "loaded" flag. A refused secondary
artifact no longer leaves a half-loaded router that never retries.
* **Eval runner.** The local Ollama provider is no longer refused, for
the same operator-endpoint reason as the Ollama embedder. Anthropic and
OpenAI are still refused.
* **Sweep of main for other new egress since this PR was written:** the
scanner's only new finding was `tls_diagnostics.py`. Manually checked
the rest of the diff: the new `reqwest` client in
`crates/headroom-proxy` and the keep-alive `httpx.AsyncClient` in
`proxy/server.py` are upstream forwarding (`user-traffic`); the new HF
tokenizer allowlist in `tokenizers/huggingface.py` sits behind the
existing `is_offline()` gate (`gated`); `port_discovery.py`,
`upstream_pinning.py`, `license_env.py` and `credential_policy.py` open
no new external connections.

### A-2 (S2): the air-gap guarantee was not real

`HEADROOM_OFFLINE=1` is documented as an air-gap switch, but several
egress paths never consulted `headroom/offline.py`. A customer who set
it and believed they were air-gapped was wrong.

**This PR has been revised after an adversarial review found that the
first version's guarantee was largely unenforced.** The review is
reproduced in the fixes below; read the *What this does and does not
promise* section for the honest scope.

### The chokepoint

`headroom/offline.py` provides `guard_egress(purpose, destination)` and
a named `OfflineEgressBlocked`. It refuses loudly — a silent skip is
indistinguishable from success at the call site, which is how these
holes survived. It is always called **before the HTTP client is
constructed**, not merely before the request, so no pooled connection is
ever opened. `crates/headroom-core/src/offline.rs` is the same switch,
same env var, same rules, for the Rust core.

#### Paths routed through it

| path | previously |
| --- | --- |
| `headroom/transforms/kompress_remote.py` | built an `httpx.Client` and
`POST`ed every compressible block to the remote Kompress endpoint |
| `headroom/observability/metrics.py` | started an `OTLPMetricExporter`
plus a background `PeriodicExportingMetricReader` timer |
| `headroom/observability/tracing.py` | started an `OTLPSpanExporter` to
Langfuse Cloud — the metric exporter's twin, missed the first time |
| `headroom/onnx_runtime.py` | `hf_hub_download` on a cache miss: the
**Python** half of the HF fetch, every ONNX model in the tree resolves
through it |
| `crates/…/tokenizer/hf_impl.rs` | fetched `tokenizer.json` from
huggingface.co |
| `crates/…/transforms/kompress.rs` | fetched the **Kompress model**
from huggingface.co |
| `crates/…/relevance/embedding.rs` | fastembed fetched ONNX weights
from huggingface.co |

The three Rust sites are the same shape and the same argument:
`HF_HUB_OFFLINE` only shortcuts the *cache lookup*, and on a miss
`hf-hub` still dials out. The first version of this PR guarded one of
the three and left the other two open.

Remote Kompress is guarded twice — in `__init__`, and again at the top
of `compress()` **outside** the fail-open `try`, because the router
caches one compressor for the process lifetime and `compress()` converts
every exception into a silent passthrough.

`onnx_runtime.hf_hub_download_local_first` is guarded on the **network
fallback only**, so a pre-seeded air-gapped cache still resolves. The
three Rust fetches refuse even on a warm cache: `hf-hub` 0.5 exposes no
cache-hit/cache-miss split at that layer, so "guard before the client
exists" and "serve from the cache" cannot both hold.
`Kompress::from_files` / `HfTokenizer::from_file` are the network-free
paths for a pre-seeded Rust deployment. Splitting the Rust fetches
cache-first is the obvious follow-up and is **not** done here.

### The refusal now survives the fail-open handlers

The original docstring said "anything that catches broad `Exception`
around an egress call should re-raise this". **Nothing did** — there
were zero re-raise sites in the package — and the refusal was swallowed
on the way out of a request by `thinking_compactor._memo_compact`
(wrapped directly around the guarded `kompress.compress` call),
`ContentRouter._kompress_model_ready` (which reported a policy refusal
as "model ready"), the startup preload, and the OpenAI
reasoning-compaction path. A full `/v1/messages` request with
`HEADROOM_OFFLINE=1` + `HEADROOM_KOMPRESS_ENDPOINT` returned **200
uncompressed** with `guard_egress` having fired twice.

**`OfflineEgressBlocked` now derives from `BaseException`, not
`Exception`.** A convention that has to be re-applied at every future
`except Exception:` in a 100k-line codebase is not a guarantee; the type
hierarchy is. Same reason `KeyboardInterrupt` and `SystemExit` sit
outside `Exception`: a policy decision is control flow, not a runtime
error to degrade around. All four sites propagate with no edit to any of
them.

⚠️ **This is a behaviour change for callers.** `except Exception` no
longer catches it anywhere, including in third-party code. `except
BaseException` still does; those are swept by `TestBroadHandlerSweep`,
which requires each to re-raise or carry a written reason.
`content_router`'s native-detect degrade path now re-raises it
explicitly, alongside the cancellation `BaseException`s it already let
through.

#### Where the refusal deliberately becomes a degradation

Escaping `except Exception` is the point, but it made one class of
caller worse. Fetching **public model weights** is not data leaving the
box, and an air-gapped proxy with a cold HF cache would have started
failing user requests where it previously degraded — punishing the
request for a decision the operator made about the host. So each
optional model loader translates the refusal into its own "model
unavailable" error, **explicitly**, with the switch named in the log and
the refusal chained:

| boundary | translated to | the caller's existing degradation |
| --- | --- | --- |
| `kompress_compressor._hf_artifact` | `KompressModelNotCached` | non-ML
compression path |
| `image/onnx_router._hf_artifact` | `RuntimeError` | `ImageCompressor`
→ `Technique.PRESERVE` |
| `memory/adapters/embedders` ONNX load | `RuntimeError` | local
embedder unavailable |
| the two Kompress download daemon threads | logged, `False`/return |
keeps whatever is cached |

Behaviour for a cold air-gapped cache is exactly what it was. What
changed is that the log now names the switch, and the translation is a
reviewed decision at a named boundary rather than something inherited
from whichever `except Exception` happened to be in the stack. The
daemon threads matter separately: an unhandled `BaseException` there
reaches `threading.excepthook` as a bare traceback on every air-gapped
boot.

Paths that **send data off the box** — remote Kompress, OTLP, Langfuse,
the beacon — are untouched and still refuse loudly.

### Startup refuses a contradictory configuration

`configure_otel_metrics` was called **outside** `lifespan`'s `try`,
above the line that sets `app.state.startup_error`, so an
`OfflineEgressBlocked` escaped uvicorn as an unhandled error — killing a
proxy that had been serving traffic, with a traceback instead of an
explanation.

It now runs through `_configure_observability_or_refuse()`, which turns
a refusal into the shape `_check_rust_core` uses ten lines above: what
is wrong, how to fix it, **exit 78** (`EX_CONFIG`). A preflight covers
the remote-Kompress contradiction the same way. So the ordinary case is
settled at startup with an actionable message, and `BaseException` is
the backstop rather than the user experience.

```
FATAL: OTLP metric export (HEADROOM_OTEL_METRICS_ENABLED) needs outbound network access, but HEADROOM_OFFLINE=1 forbids it.
    refused: HEADROOM_OFFLINE is set: refusing outbound network access for OTLP metric export to …
    fix:     set HEADROOM_OTEL_METRICS_EXPORTER=console or scrape /metrics (both stay on-box), or unset HEADROOM_OTEL_METRICS_ENABLED
```

### Python/Rust normalisation parity

Python trimmed with `str.strip()`, Rust with `str::trim()`. Not the same
set: `str.isspace()` includes U+001C–U+001F, the Unicode `White_Space`
property does not, so `HEADROOM_OFFLINE=$'\x1c1'` read as **offline to
Python and online to Rust** — one variable, one half of the process
air-gapped. Both sides now trim an enumerated `TRIM_CHARS` /
`_TRIM_CHARS`, and the parity test compares the trim sets and a shared
normalisation table, not just the accepted tokens. (Case folding needed
no change: no non-ASCII character lowercases into any character of
`1`/`true`/`yes`/`on`.)

### The meta-test, rebuilt

The first version was neutered by one line:

```python
if "guard_egress" in source:
    continue
```

A whole-file exemption keyed on the string appearing *anywhere*,
including in a docstring — the exact blanket exemption the allowlist's
own comment says must never happen. It made the site-count assertion
unreachable for every guarded file, so "adding a second client trips
this test" only held for files with **no** guard at all. Two bypasses
were demonstrated against it: a second unguarded `httpx.Client` in an
already-guarded file, and a brand-new unguarded module with the word in
its docstring.

It now decides **per egress site**:

* **AST, not text.** Comments and docstrings vouch for nothing.
`self.client.post()` is no longer read as `requests.post`, and the
`httpx.post` example in `compress.py`'s docstring is not a site.
* **Dominance.** A site is guarded only when a `guard_egress` call is in
its block or an enclosing one, earlier in that block, in the same
function. A guard in a sibling `if`/`except` arm, in a nested `def`, or
after the site does not count. A guard in a helper the function calls
does not count either — conservative on purpose.
* **`TestSiteScannerRules`** tests the scanner itself, pinning both
demonstrated bypasses plus the one-line `def f(): return
httpx.Client().post(...)` that the old `startswith("def ")` skip made
invisible. A meta-test nobody tests is a comment.
* **Wider callee set:** module-level `httpx` verbs
(`headroom/cli/mcp.py` was a live unmatched site), `hf_hub_download`
(skipping `local_files_only=True`, which opens no socket), fastembed's
`TextEmbedding`, the OTLP exporters, and the OpenAI/Anthropic SDK
clients.
* **`crates/` is swept too** (`TestRustEgressChokepointCoverage`). It
was outside the scan entirely, which is how two Rust downloads stayed
open. Text-level rather than AST — no Rust parser here — so "guarded"
means a `guard_egress` earlier in the same `fn` at no deeper indent.
Weaker, and labelled as such; the runtime assertions live in each
crate's `#[test]`s.
* **Callees are resolved through the module's import table before
matching.** The scanner compared the text `ast.unparse` produced, which
is whatever local name the module bound — so `import httpx as h` hid a
client from a test whose entire job is to notice clients. That was not
hypothetical: `copilot_auth.py` does `from urllib import request as
urllib_request`, so its GitHub calls rendered as
`urllib_request.urlopen` and **five call sites, including the shared
`_urlopen` helper every Copilot request goes through, were invisible**.
`_canonical_callee` rewrites the leading name first; the alias defence
cuts both ways, so `guard_egress as ge` still counts as a guard. Imports
are collected from the whole tree, not just module level, because half
the egress here is imported inside the function that uses it.
* Allowlist counts now mean **unguarded** sites only, so a file can
legitimately guard some of its egress and still carry an entry for the
rest.
* **There is no longer a category for "known violation".**
`test_no_category_permits_a_known_violation` pins the permitted set to
`loopback` / `gated` / `user-traffic` / `operator-endpoint` /
`cache-only` and rejects "out of scope" as a reason.

### The offline policy, in plain words

`HEADROOM_OFFLINE=1` **refuses every connection Headroom itself decides
to make to a destination Headroom itself chose.** That is the rule, not
a sample list: telemetry, update check, license/usage reporter, OTLP and
Langfuse export, HuggingFace / Kompress / fastembed model and tokenizer
downloads (Python and Rust), release-binary and codebase-memory-mcp
downloads, eval dataset downloads and the provider SDK clients the eval
harness drives, GitHub Copilot device-flow auth and token exchange, the
Anthropic / Codex / Copilot subscription pollers, both OpenAI embedders,
both Headroom Cloud compression integrations, and the TLS diagnostics
(`headroom doctor --network` endpoint checks and the certificate-chain
re-probe after an upstream TLS failure).

**Four things are deliberately still allowed, and they are the complete
set of exceptions.** Every one of them is either physically incapable of
leaving the box or is not Headroom-initiated traffic:

| category | what it covers | why it survives the switch |
| --- | --- | --- |
| `user-traffic` | `proxy/server.py` (2 sites) | the caller's own
request forwarded to the upstream the operator configured. A proxy that
refuses this is not a proxy; an air-gapped install points it at an
on-prem endpoint. |
| `operator-endpoint` | `memory/adapters/embedders.py` (1 site) | the
Ollama embedder, whose address comes entirely from operator config and
defaults to `http://localhost:11434`. Exactly one thing qualifies today,
and a hard-coded internet host may never hide here. |
| `loopback` | 8 files, 12 sites | hard-coded `127.0.0.1` probes against
the operator's own local proxy (`headroom doctor`, the installers, the
MCP sidecar, the test harness). |
| `gated` | 3 files, 3 sites | the beacon, the update check and the
usage reporter are already unreachable behind an `is_offline()` check,
so no connection is ever built. |

Plus one mechanical exemption, `cache-only` (`relevance/embedding.py`, 1
site): the first of the two fastembed loads runs with `HF_HUB_OFFLINE`
forced to `1`, so `huggingface_hub` resolves from the local cache or
raises **without opening a socket**; the network retry directly below it
calls `guard_egress`. That is what lets a pre-seeded air-gapped host
keep loading the model while a cold one refuses.

#### What changed since the review

The previous revision of this PR recorded **nine egress-capable files**
under an `unguarded` allowlist category meaning "reachable under
`HEADROOM_OFFLINE`, out of scope for A-2": Copilot auth/token exchange,
three subscription pollers, the two binary/CBM downloads, the
eval-dataset fetch, and both Headroom Cloud compression integrations —
plus the eval harness's LLM clients, both OpenAI embedders and the
fastembed weights. @JerrettDavis blocked on exactly that, and he was
right: recording that a switch does not do what it says is not the same
as making it do it.

**All of them are now guarded, and the `unguarded` category is
deleted.** `test_no_category_permits_a_known_violation` pins the five
permitted categories by name and rejects "out of scope" as an allowlist
reason, so re-introducing a bless-it-and-move-on category takes a
deliberate edit with a reviewer looking at it.

#### ⚠️ Behaviour change customers can see

**Outbound calls that previously worked with `HEADROOM_OFFLINE=1` set
now refuse.** That is the point of the PR, but it is a real change for
anyone who set the flag and kept using these features:

* `headroom auth` / `headroom copilot-auth login` cannot complete a
GitHub device flow, and a cached Copilot OAuth token cannot be exchanged
for an API token. A Copilot subscription is unusable on an air-gapped
box regardless, but the failure now arrives at the moment of the
attempt.
* `headroom install` will not download release binaries or
codebase-memory-mcp. Pre-stage them, or install with the flag unset.
* The Anthropic / Codex / Copilot subscription and quota panels stay
empty, with the reason logged and (for Copilot quota) shown in the
panel's own error slot.
* Headroom Cloud compression mode in the ASGI middleware and the LiteLLM
callback refuses instead of falling through to local compression —
loudly, because silently compressing locally would hide that the
deployment is no longer doing what it was configured to do.
* The eval harness refuses to build provider clients or download BFCL.
* The OpenAI embedders (memory adapter and direct mem0 backend) refuse.
Use the ONNX or Ollama embedder on an air-gapped box.
* `EmbeddingScorer` raises "model unavailable" on a **cold** fastembed
cache. A warm one is unaffected.

#### Refusals reach a human, never a traceback

Following the pattern the earlier commits set for the Kompress download
threads, the model layer and the startup banner:

* **CLI:** `OfflineAwareGroup` (`headroom/cli/main.py`) is the one place
that translates the refusal into a Click `Error:` line, for every
subcommand at once. `OfflineEgressBlocked` stays a `BaseException`
everywhere inside, so nothing else can downgrade it; only the last frame
before the terminal is allowed to make it readable.
* **Background pollers:** each catches `OfflineEgressBlocked`
specifically, calls `offline.note_refusal` (WARNING once per purpose,
DEBUG after, so a 30-second poller does not flood the log), and returns
its ordinary "no data" value. An un-caught `BaseException` out of a bare
`create_task` surfaces as "Task exception was never retrieved" whenever
the GC gets to it — a traceback with no explanation in it.
* **Model loaders:** `EmbeddingScorer._get_model` translates into a
named "model unavailable" `RuntimeError`, the way
`kompress_compressor._hf_artifact` and `image/onnx_router._hf_artifact`
already do.

### 🚩 Two decisions that need a founder call

Both were implemented the **safe** way — refuse, loudly and legibly —
because that is the defensible default for a switch an operator sets to
mean "nothing leaves this box". Neither is obviously right, and a
permissive default should not be chosen quietly:

1. **Should OTLP / telemetry export be permitted under
`HEADROOM_OFFLINE`?** Today it refuses, and a contradictory config
(`HEADROOM_OTEL_METRICS_ENABLED=1` + offline) exits 78 at startup with
an explanation. The argument for permitting it is that a regulated
customer's OTLP collector is usually in-cluster, and losing
observability is a real operational cost. The argument against — and the
reason it is refused today — is that the guard **cannot reliably tell an
in-cluster collector from an internet host**: DNS, service meshes and
egress proxies all blur it, so any exemption is really "trust the
endpoint the config names". A customer with an in-cluster collector
currently has to scrape `localhost:8787/metrics` or use
`HEADROOM_OTEL_METRICS_EXPORTER=console`. If we want to permit it, the
honest shape is an explicit second flag (e.g.
`HEADROOM_OFFLINE_ALLOW_OTLP=1`) rather than widening the policy.
2. **Should the subscription pollers refuse, or degrade silently?**
Today all three refuse and say so once per purpose at WARNING. The
argument for silence is that these are cosmetic panels and a WARNING on
an air-gapped box is noise about a decision the operator already made.
The argument for the current behaviour is that a usage panel that
silently stops updating is exactly the "why is this empty?" confusion
the switch was supposed to end, and once-per-purpose logging bounds the
noise. Changing this is a one-line change in `offline.note_refusal`.

### Tests

`tests/test_offline_egress_chokepoint.py` — 110 tests. Per-path socket
assertions (a `no_sockets` fixture booby-traps
`socket.create_connection`, `socket.socket.connect` and `connect_ex`, so
asserting "no socket" rather than "raises" catches a guard placed after
the client is built), the scanner-rule tests, the two coverage sweeps,
the broad-handler sweep, the startup-refusal tests, the model-loader
degradation tests, the cross-language parity tests, and the doc-claim
test. Plus 8 Rust tests in `headroom-core`.

### Fail-before / pass-after

Per fix, work committed, then sources reverted while keeping the new
tests, run, restore. No `git stash`.

**Meta-test (1).** Added a probe module whose docstring mentions
`guard_egress` and a second unguarded `httpx.Client` to the
already-guarded `kompress_remote.py`. The old meta-test: `3 passed`. The
new one:

```
  headroom/_a2_probe_tmp.py
        line 7: httpx.Client().get("https://evil.example.com")
        (not guarded, not allowlisted)
  headroom/transforms/kompress_remote.py
        line 154: return httpx.Client(timeout=1.0)
        (not guarded, not allowlisted)
```

**Rust guards (2).** Guard calls commented out, tests kept:

```
panicked at crates/headroom-core/src/transforms/kompress.rs:688:
expected an offline refusal, got a network error: no loadable ONNX artifact in
`headroomlabs-ai/this-model-does-not-exist-a2` (tried [...]): request error: http status: 401
```

The model id cannot exist, so a `Hub` error can only come from a real
round-trip. That **401 is the proof a socket was opened to
huggingface.co with `HEADROOM_OFFLINE=1` set**, and `Offline` can only
come from returning before `Api::new()`.

**Refusal / startup (3, 4).**
`headroom/{offline,proxy/server,transforms/content_router}.py` reverted
to the PR tip, tests kept: `11 failed, 6 passed` — including
`test_thinking_compactor_does_not_swallow_it`,
`test_kompress_model_ready_does_not_report_a_refusal_as_ready`,
`test_no_broad_handler_swallows_the_refusal` and all five
`TestStartupRefusal` tests.

**New Python guards (5).**
`headroom/{onnx_runtime,observability/tracing}.py` reverted to
`origin/main`, tests kept: `4 failed` — the two socket tests plus both
coverage assertions.

**Normalisation (6).** Both `offline` modules reverted to the PR tip,
parity tests kept: `6 failed`, with `HEADROOM_OFFLINE="\x1c1"` reading
as offline in Python and not in Rust.

**Docs (7).** Both doc files reverted, test kept:

```
AssertionError: docs/metrics-technical-guide.md says 'disables all outbound traffic', but
_EGRESS_ALLOWLIST still records these as reachable under HEADROOM_OFFLINE: [...16 files...]
```

#### This revision (the review fixes)

Committed first, then `git checkout origin/main --
headroom/{binaries,cli/main,copilot_auth,graph/installer,integrations/asgi,integrations/litellm_callback,memory/adapters/embedders,memory/backends/direct_mem0,relevance/embedding}.py
headroom/{evals,subscription}/` with the new tests kept:

```
23 failed, 79 passed
```

The 23 are the five Copilot-auth socket tests, the four poller tests,
the two install-download tests, the BFCL test, both Headroom Cloud
tests, the two cloud-embedder tests, the four fastembed tests, both
CLI-translation tests — and
`test_every_egress_site_is_guarded_or_allowlisted`, which lists every
reverted file as "(not guarded, not allowlisted)".

The aliased-import blind spot, isolated (canonicalisation stubbed back
to the identity, i.e. the pre-fix behaviour, against the same inputs):

```
before:  INVISIBLE: import httpx as h
         INVISIBLE: from urllib import request as urllib_request
         INVISIBLE: from openai import AsyncOpenAI as AO
         INVISIBLE: import httpx as h (inside a function)
         aliased guard recognised: [False]
         copilot_auth.py sites seen: 4
after:   DETECTED (all four); aliased guard recognised: [True]
         copilot_auth.py sites seen: 6
```

**Pass-after (this revision):** `102 passed` in
`tests/test_offline_egress_chokepoint.py`; `391 passed, 86 skipped`
across the copilot-auth / copilot-quota / codex-rate-limits / binaries /
integrations / CLI subsets; `544 passed, 15 skipped` across the
relevance / embedding / mem0 / asgi / litellm / subscription / installer
subsets; `160 passed, 6 skipped` across the evals and CLI doctor/learn
subsets; `59 passed` across the proxy-startup subset; `cargo test -p
headroom-core offline` → `8 passed`.

**Pass-after (earlier revisions):** `70 passed` (pytest), `942 passed`
(`cargo test -p headroom-core --lib`), `720 passed, 4 skipped` across
the offline/otel/kompress/telemetry/onnx/tracing/relevance subsets, and
`1868 passed, 30 skipped` across the
kompress/offline/image/embedder/memory subsets after the model-loader
boundary change. The one failure in the full run,
`test_mcp_reconcile.py::test_ordinary_install_does_not_adopt_serena`,
reproduces unchanged at the PR tip and is unrelated. Cost/pricing tests
are the known `origin/main` litellm breakage (#3722) and were not run.

### Checks

- `uvx ruff@0.16.7 check .` → All checks passed
- `uvx ruff@0.16.7 format --check .` → 1622 files already formatted
- `cargo fmt --check` → clean; `cargo clippy -p headroom-core
--all-targets` → no warnings
- `CHANGELOG.md` untouched
- CodeQL: the four high alerts were all in
`tests/test_offline_egress_chokepoint.py` and are fixed honestly rather
than suppressed. Three `py/incomplete-url-substring-sanitization` host
assertions (`"host" in url`, which is also true of
`https://host.attacker.test`) now parse with `urllib.parse.urlsplit` and
compare the hostname exactly — which is what the tests meant. The
`py/redos` regex scraping Rust `char` literals had a nested `(...)+`
that can backtrack exponentially; a Rust char literal is exactly one
character, so it is now a single bounded alternation.

### Follow-ups (not in this PR)

1. ~~Guard the 16 `unguarded` files, each with its own user-facing
message.~~ Done in this PR; the `unguarded` category is deleted.
2. Split the three Rust HF fetches cache-first so a pre-seeded
air-gapped cache resolves without egress, the way
`hf_hub_download_local_first` already does on the Python side.
3. The Rust sweep is a text heuristic. An AST-level dominance check (via
`syn`, in a Rust test) would make the two halves symmetric.
4. The control-plane heartbeat lives in a different repository and is
still out of scope.
5. The scanner has no entry for transformers `from_pretrained` /
`SentenceTransformer`. The Kompress tokenizer site Devin found is
guarded now, but `models/ml_models.py` (SigLIP, technique router,
sentence-transformers) and the Kompress PyTorch backend still rely on
`HF_HUB_OFFLINE` alone, which `apply_offline_env` sets with `setdefault`
and only in the proxy process. Each needs the cache-first split the
tokenizer has, plus a scanner entry.


<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3729"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

## Testing

- [x] Unit tests pass (`pytest`) — subsets below
- [x] Linting passes (`ruff check .`) — touched files, `ruff@0.16.7`
- [ ] Type checking passes (`mypy headroom`) — not run
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# this revision
$ LITELLM_LOCAL_MODEL_COST_MAP=True uv run --frozen --extra dev --extra proxy pytest \
    tests/test_offline_egress_chokepoint.py tests/test_cli_doctor.py tests/test_corporate_tls.py -q
245 passed
# after the Devin fixes
$ pytest tests/test_offline_egress_chokepoint.py + kompress/onnx_router/relevance/embedding/evals subsets
361 passed, 6 skipped
# fail-before for the Devin fixes (4 sources reverted, tests kept): 6 failed + the fallback test (fails once its fake
# raises OSError instead of LocalEntryNotFoundError, which conftest converts to a skip)

# fail-before, TLS guard (tls_diagnostics.py + doctor.py reverted, tests kept)
FAILED TestDoctorNetworkOffline::test_endpoint_check_refuses_before_any_connection
FAILED TestDoctorNetworkOffline::test_doctor_reports_the_network_checks_as_skipped
FAILED TestDoctorNetworkOffline::test_chain_probe_opens_no_socket
FAILED TestEgressChokepointCoverage::test_every_egress_site_is_guarded_or_allowlisted
   headroom/proxy/tls_diagnostics.py line 473: with httpx.Client(...)  (not guarded, not allowlisted)
# with only tls_diagnostics.py reverted, the doctor test fails on content, not on a missing helper:
   [('net api.anthropic.com', 'fail'), ('net api.openai.com', 'fail'), ...] != [('network', 'skip')]

# fail-before, socket trap (old fixture body, new tests kept)
FAILED TestSocketTrap::test_event_loop_wakeup_pipe_is_not_egress
   SocketOpened: outbound connection attempted while offline: (<socket ...>, ('127.0.0.1', 50151))
# — the same csock.connect inside socketpair() that Jerrett's Windows trace shows.

$ uvx ruff@0.16.7 check / format --check (touched files)
All checks passed! / 4 files already formatted
```

Earlier-revision test evidence is in *Fail-before / pass-after* under
*Changes Made*.

## Real Behavior Proof

- Environment: macOS, Python 3.14, this branch (merged with `main` at
`befdb52f`), no proxy running.
- Exact command / steps: `HEADROOM_OFFLINE=1 headroom doctor --network
--port 59999`, run for real. A second run went through
`headroom.cli.main` with a `sys.addaudithook` recording every
`socket.getaddrinfo` / `socket.connect`.
- Observed result: the doctor table shows `network · skip · skipped:
HEADROOM_OFFLINE is set, no endpoint was contacted`, with the refusal
naming `https://api.anthropic.com/v1/models` as the hint. The audit hook
recorded only `127.0.0.1` lookups and connects (the doctor's own proxy
probes on :59999 and :8787) and **zero external lookups or connects**.
Exit code 2 comes from the unrelated "proxy not reachable" failures.
- Not tested: Windows itself. The Windows mechanism is reproduced by
forcing `socket._fallback_socketpair`, which is the code Windows runs
for `socket.socketpair()`; the CI Windows job (`test-windows-newline`)
only runs `windows_newline`-marked tests and was not extended here.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; behaviour keyed on the existing
`HEADROOM_OFFLINE` switch.
- Minimum rollout channel: stable.
- Stable/default behavior changed: only when `HEADROOM_OFFLINE` is set
(refusals listed above). Unset, nothing changes.
- Kill switch / disable path: unset `HEADROOM_OFFLINE`.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Two founder decisions remain flagged above (OTLP under offline; pollers
loud vs silent). Windows CI coverage for this module is not added: no
unit-test job runs the full module on Windows today, and adding one is a
workflow change better made separately.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Tejas Chopra
2026-10-05 15:20:39 -05:00
committed by GitHub
co-authored by Claude Opus 5
parent ee6cfcccf6
commit 119d1a1939
40 changed files with 4319 additions and 100 deletions
+19
View File
@@ -8,6 +8,7 @@ pub mod auth_mode;
pub mod cache_control;
pub mod ccr;
pub mod compression_policy;
pub mod offline;
#[cfg(feature = "ml")]
mod onnx_cpu;
pub mod relevance;
@@ -29,6 +30,24 @@ pub fn hello() -> &'static str {
"headroom-core"
}
#[cfg(test)]
pub(crate) mod test_support {
use std::sync::{Mutex, MutexGuard};
/// The process environment is global while `cargo test` runs tests in
/// parallel threads inside one process, so a test that sets an env var can
/// be observed by an unrelated sibling mid-assertion. Every test that
/// mutates the environment holds this for its whole body. Same shape as
/// `REGISTRY_LOCK` in `tokenizer::registry`.
static ENV_LOCK: Mutex<()> = Mutex::new(());
/// Recovers from poisoning on purpose: one panicking test should not
/// cascade into every later test that touches the environment.
pub(crate) fn env_lock() -> MutexGuard<'static, ()> {
ENV_LOCK.lock().unwrap_or_else(|e| e.into_inner())
}
}
#[cfg(test)]
mod tests {
use super::*;
+181
View File
@@ -0,0 +1,181 @@
//! Air-gap / no-egress master switch (`HEADROOM_OFFLINE`) — Rust half.
//!
//! The Python side lives in `headroom/offline.py`. This is the same switch
//! read by the same environment variable with the same truthiness rules, so a
//! deployment that sets `HEADROOM_OFFLINE=1` gets identical behaviour whether
//! the egress attempt originates in the Python proxy or in a Rust core path.
//!
//! Two functions, mirroring the Python module deliberately:
//!
//! - [`is_offline`] — the predicate, for code that wants to take a different
//! route (use a cached artifact, skip an optional refresh).
//! - [`guard_egress`] — the chokepoint, for code that is about to open a
//! socket. It returns [`OfflineEgressBlocked`], a distinct error type, so a
//! caller that otherwise degrades gracefully on network failure can still
//! tell "the operator air-gapped this box" from "the network was flaky" and
//! refuse loudly instead of silently falling back.
//!
//! # Parity contract
//!
//! `HEADROOM_OFFLINE` is true for `1`, `true`, `yes`, `on` — trimmed and
//! ASCII-case-insensitive — and false for anything else, including unset and
//! empty. Any change here must be mirrored in `headroom/offline.py`'s
//! `_TRUE_VALUES`, and vice versa: the two implementations are a pair, and a
//! deployment that reads as offline to Python but online to Rust is precisely
//! the failure this module exists to prevent.
//!
//! The parity covers **normalisation**, not just the accepted values. This
//! used to call `str::trim()` while Python called `str.strip()`, and those are
//! not the same set: Python's strips U+001C-U+001F (the ASCII file/group/
//! record/unit separators) because `str.isspace()` includes them, Rust's does
//! not because the Unicode `White_Space` property does not. `HEADROOM_OFFLINE`
//! set to `"\x1c1"` read as offline to Python and online to Rust — one
//! environment variable, one process air-gapped and the other not. Both sides
//! now trim exactly [`TRIM_CHARS`].
use std::env;
use thiserror::Error;
/// The environment variable that selects fully-offline operation.
pub const OFFLINE_ENV: &str = "HEADROOM_OFFLINE";
/// Values that read as "yes, offline". Kept byte-identical to the Python
/// side's `_TRUE_VALUES` (see the parity contract in the module docs).
const TRUE_VALUES: [&str; 4] = ["1", "true", "yes", "on"];
/// Whitespace trimmed off the raw value before matching. Enumerated rather
/// than left to `str::trim()`, which is the Unicode `White_Space` property and
/// does not agree with Python's `str.strip()`. Kept byte-identical to the
/// Python side's `_TRIM_CHARS` (see the parity contract in the module docs).
const TRIM_CHARS: [char; 6] = [' ', '\t', '\n', '\r', '\u{b}', '\u{c}'];
/// An egress attempt was refused because `HEADROOM_OFFLINE` is in force.
///
/// Carries the human-readable `purpose` and `destination` handed to
/// [`guard_egress`] so the operator learns which feature to turn off rather
/// than just that "something" was blocked.
#[derive(Debug, Clone, Error)]
#[error(
"{OFFLINE_ENV} is set: refusing outbound network access for {purpose} to \
{destination}. Unset {OFFLINE_ENV}, or turn off the feature that needs \
this connection."
)]
pub struct OfflineEgressBlocked {
pub purpose: String,
pub destination: String,
}
/// Return `true` when `HEADROOM_OFFLINE` selects fully-offline operation.
pub fn is_offline() -> bool {
match env::var(OFFLINE_ENV) {
Ok(raw) => {
let normalized = raw.trim_matches(TRIM_CHARS.as_slice()).to_ascii_lowercase();
TRUE_VALUES.contains(&normalized.as_str())
}
// Unset, or not valid UTF-8 — neither is an opt-in to offline mode.
Err(_) => false,
}
}
/// The chokepoint every Rust egress path must call before opening a socket.
///
/// `Ok(())` when online; `Err(OfflineEgressBlocked)` when `HEADROOM_OFFLINE`
/// is set. Call it *before* constructing the HTTP client, not merely before
/// the request — several clients (`ureq`, `reqwest`) resolve DNS or warm a
/// connection pool during setup, so a guard placed at the request would leak
/// the very packets the air-gap switch promises not to send.
pub fn guard_egress(purpose: &str, destination: &str) -> Result<(), OfflineEgressBlocked> {
if is_offline() {
return Err(OfflineEgressBlocked {
purpose: purpose.to_string(),
destination: destination.to_string(),
});
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use crate::test_support::env_lock;
#[test]
fn unset_is_online() {
let _guard = env_lock();
env::remove_var(OFFLINE_ENV);
assert!(!is_offline());
assert!(guard_egress("anything", "https://example.invalid").is_ok());
}
#[test]
fn truthy_values_match_python() {
let _guard = env_lock();
// Same set, same trimming, same case-insensitivity as
// headroom/offline.py::_TRUE_VALUES. If this list drifts, the two
// halves of the switch disagree and the air-gap has a hole.
for raw in ["1", "true", "TRUE", "Yes", " on ", "\ton\n"] {
env::set_var(OFFLINE_ENV, raw);
assert!(is_offline(), "{raw:?} should read as offline");
}
env::remove_var(OFFLINE_ENV);
}
#[test]
fn falsy_values_stay_online() {
let _guard = env_lock();
for raw in ["", "0", "off", "no", "false", "maybe", " "] {
env::set_var(OFFLINE_ENV, raw);
assert!(!is_offline(), "{raw:?} should read as online");
}
env::remove_var(OFFLINE_ENV);
}
/// Normalisation parity, not just value parity. Every case here is
/// duplicated verbatim in `tests/test_offline_egress_chokepoint.py`'s
/// `test_normalisation_matches_the_rust_side`, and the two must agree.
///
/// The `\x1c` cases are the regression: they are whitespace to Python's
/// `str.strip()` and not to Rust's `str::trim()`, so before `TRIM_CHARS`
/// existed `"\x1c1"` air-gapped the Python proxy and left the Rust core
/// dialling out.
#[test]
fn normalisation_matches_python() {
let _guard = env_lock();
for (raw, expected) in [
("1", true),
(" 1 ", true),
("\t1\n", true),
("\r\n TRUE \r\n", true),
("\u{b}yes\u{c}", true),
("\u{1c}1", false),
("1\u{1f}", false),
("\u{a0}1", false),
("\u{2007}on", false),
("", false),
(" ", false),
] {
env::set_var(OFFLINE_ENV, raw);
assert_eq!(is_offline(), expected, "{raw:?}");
}
env::remove_var(OFFLINE_ENV);
}
#[test]
fn guard_reports_purpose_and_destination() {
let _guard = env_lock();
env::set_var(OFFLINE_ENV, "1");
let err = guard_egress("HuggingFace tokenizer download", "hf.co/acme/model")
.expect_err("guard must refuse while offline");
assert_eq!(err.purpose, "HuggingFace tokenizer download");
assert_eq!(err.destination, "hf.co/acme/model");
let rendered = err.to_string();
assert!(rendered.contains(OFFLINE_ENV), "{rendered}");
assert!(
rendered.contains("HuggingFace tokenizer download"),
"{rendered}"
);
assert!(rendered.contains("hf.co/acme/model"), "{rendered}");
env::remove_var(OFFLINE_ENV);
}
}
@@ -98,6 +98,22 @@ impl EmbeddingScorer {
/// quality/speed tradeoff for compression-relevance scoring on
/// short snippets.
pub fn try_new_with_model(model_kind: EmbeddingModel) -> Result<Self, String> {
let name = format!("{:?}", model_kind);
// Air-gap chokepoint, FIRST thing in the function. `TextEmbedding::
// try_new` resolves the model's ONNX weights through `hf-hub`, which
// downloads from huggingface.co on a cache miss even with
// `HF_HUB_OFFLINE` set — the same hole the tokenizer and Kompress model
// fetches had. It sits ahead of the AVX2 and ort-loader probes below so
// that an air-gapped box reports the policy refusal rather than
// whichever local prerequisite happened to be missing as well.
//
// The refusal is soft here by design: this function already returns
// `Err(String)` for "ONNX runtime unavailable" and every caller degrades
// to the BM25 scorer, so an air-gapped box loses embedding relevance
// instead of failing a request. The message names the switch so the
// operator can tell the two causes apart in the log.
crate::offline::guard_egress("fastembed embedding model download", &name)
.map_err(|e| format!("EmbeddingScorer model load refused: {e}"))?;
// fastembed links the precompiled ONNX Runtime binary, which contains
// AVX2 instructions on x86. Loading/running it on a non-AVX2 CPU traps
// with SIGILL (issue #1723) — an uncatchable native fault. Bail early so
@@ -113,7 +129,6 @@ impl EmbeddingScorer {
// `dynamic_ort_loader_ready`).
crate::transforms::magika_detector::dynamic_ort_loader_ready()
.map_err(|e| format!("EmbeddingScorer: ONNX Runtime unavailable: {e}"))?;
let name = format!("{:?}", model_kind);
let model = TextEmbedding::try_new(InitOptions::new(model_kind))
.map_err(|e| format!("EmbeddingScorer model load failed: {}", e))?;
Ok(EmbeddingScorer {
@@ -308,6 +323,28 @@ mod tests {
std::env::var("RUN_FASTEMBED_TESTS").is_ok()
}
/// `try_new_with_model` must refuse before fastembed resolves the model,
/// and must say so in the returned message rather than looking like an
/// ordinary load failure — the caller only ever sees the `String`.
///
/// Not gated on `RUN_FASTEMBED_TESTS`: the whole point is that no network
/// call happens, so this is safe to run in CI. If the guard were removed
/// this test would either download ~30 MB or fail with a load error, and
/// neither says "refused".
#[cfg(feature = "ml")]
#[test]
fn try_new_refuses_while_offline() {
let _guard = crate::test_support::env_lock();
std::env::set_var(crate::offline::OFFLINE_ENV, "1");
let result = EmbeddingScorer::try_new();
std::env::remove_var(crate::offline::OFFLINE_ENV);
let err = result.err().expect("guard must refuse while offline");
assert!(err.contains("refused"), "{err}");
assert!(err.contains(crate::offline::OFFLINE_ENV), "{err}");
assert!(err.contains("fastembed embedding model download"), "{err}");
}
/// Construct a stub scorer with `model = None` for offline-safe
/// tests of the unavailable-path behavior.
#[cfg(feature = "ml")]
@@ -49,6 +49,13 @@ pub enum HfTokenizerError {
#[source]
source: Box<dyn std::error::Error + Send + Sync>,
},
/// `HEADROOM_OFFLINE` is set, so the Hub fetch was refused before any
/// socket was opened. Deliberately a separate variant from [`Self::Hub`]:
/// a caller that retries or falls back on a transient download failure
/// must not do either here — the operator asked for no egress, and
/// retrying an air-gap refusal is just a slower refusal.
#[error(transparent)]
Offline(#[from] crate::offline::OfflineEgressBlocked),
}
/// Token counter backed by a HuggingFace `tokenizer.json`.
@@ -110,11 +117,35 @@ impl HfTokenizer {
/// processes hit the on-disk cache.
///
/// Errors:
/// - [`HfTokenizerError::Offline`] when `HEADROOM_OFFLINE` is set. Refused
/// before any socket is opened; see below.
/// - [`HfTokenizerError::Hub`] for download failures (no network, 404,
/// 401 on a gated model without `HF_TOKEN`).
/// - [`HfTokenizerError::Load`] if the downloaded bytes don't parse as
/// a valid `tokenizer.json`. Should not happen for healthy HF repos.
pub fn from_pretrained(repo: &str) -> Result<Self, HfTokenizerError> {
// Air-gap chokepoint, matching `headroom/offline.py`'s `guard_egress`.
// This is the Rust half of the same switch, so an operator who sets
// HEADROOM_OFFLINE=1 gets the same refusal whichever runtime reaches
// for the Hub.
//
// It has to sit here rather than around the `get` below: `Api::new`
// builds the `ureq` agent and resolves the endpoint, and hf-hub's own
// HF_HUB_OFFLINE handling only covers the cache lookup — a cache miss
// still dials out. Guarding first means the air-gapped case never
// constructs a client at all.
//
// The refusal is hard on purpose. Silently returning a cached or
// estimating tokenizer would leave token counts subtly wrong with no
// signal; the caller (`tokenizer::registry::try_register_hf`) already
// propagates the error so the operator sees which repo was wanted.
//
// Note it refuses even on a warm cache: `hf-hub` 0.5 exposes no
// cache-hit/cache-miss split at this layer, so "guard before the
// client exists" and "serve from the cache" cannot both hold here. An
// air-gapped deployment with pre-seeded artifacts should point at them
// with `from_file`, which is network-free by construction.
crate::offline::guard_egress("HuggingFace tokenizer download", repo)?;
let api = hf_hub::api::sync::Api::new().map_err(|e| HfTokenizerError::Hub {
repo: repo.to_string(),
source: Box::new(e),
@@ -260,6 +291,41 @@ mod tests {
assert!(Arc::ptr_eq(&a.inner, &b.inner));
}
/// A-2: `HEADROOM_OFFLINE=1` must stop the Hub fetch before a socket exists.
///
/// The repo name is deliberately one that cannot exist, which is what makes
/// this a real assertion rather than a tautology: without the guard the call
/// reaches `hf-hub`, dials `huggingface.co`, gets a 401/404 and returns
/// `HfTokenizerError::Hub`. Only the guard can produce `Offline`, and only
/// by returning before `Api::new` — so `Offline` *is* the "no socket was
/// opened" assertion for this path. (A packet-level assertion is not
/// available to a `cargo test` unit test the way `socket.socket` patching is
/// in Python; this typed-error distinction is the equivalent signal.)
#[test]
fn from_pretrained_refuses_while_offline() {
let _guard = crate::test_support::env_lock();
std::env::set_var(crate::offline::OFFLINE_ENV, "1");
let result = HfTokenizer::from_pretrained("headroomlabs-ai/this-repo-does-not-exist-a2");
std::env::remove_var(crate::offline::OFFLINE_ENV);
match result {
Err(HfTokenizerError::Offline(blocked)) => {
assert_eq!(blocked.purpose, "HuggingFace tokenizer download");
assert_eq!(
blocked.destination,
"headroomlabs-ai/this-repo-does-not-exist-a2"
);
}
Err(other) => panic!("expected an offline refusal, got a network error: {other}"),
Ok(_) => panic!("expected an offline refusal, got a tokenizer"),
}
}
// The "switch off ⇒ guard is a no-op" half is covered in `crate::offline`'s
// own tests rather than here: asserting it through `from_pretrained` would
// mean actually dialling huggingface.co, which is exactly the thing a unit
// test must not do.
#[test]
fn from_file_loads_a_real_file() {
// Round-trip via a temp file to cover the on-disk constructor.
@@ -180,6 +180,12 @@ pub enum KompressError {
#[source]
source: Box<dyn std::error::Error + Send + Sync>,
},
/// `HEADROOM_OFFLINE` is set, so the Hub fetch was refused before any
/// socket was opened. A separate variant from [`Self::Hub`] for the same
/// reason as [`crate::tokenizer::HfTokenizerError::Offline`]: retrying or
/// falling back on a policy refusal is just a slower refusal.
#[error(transparent)]
Offline(#[from] crate::offline::OfflineEgressBlocked),
}
// ─── Compressor ─────────────────────────────────────────────────────────
@@ -255,6 +261,22 @@ impl Kompress {
/// downloading on miss). Blocking — call off the hot path. Tries the
/// [`ONNX_CANDIDATES`] in order.
pub fn from_pretrained(config: KompressConfig) -> Result<Self, KompressError> {
// Air-gap chokepoint — the same one `hf_impl.rs` uses for the
// tokenizer, for the same reason. `HF_HUB_OFFLINE` only shortcuts the
// cache *lookup*; on a cache miss `hf-hub` still dials huggingface.co,
// so the Kompress model download is a live egress path under
// `HEADROOM_OFFLINE=1` and was not covered.
//
// It sits before `Api::new()` because that already resolves the Hub
// endpoint and builds the `ureq` agent, so guarding at the `get` call
// would leak the setup packets the switch promises not to send.
//
// Trade-off, stated plainly: this refuses even when the artifacts are
// already in `~/.cache/huggingface/hub`, because `hf-hub` 0.5 gives no
// cache-hit/cache-miss split at this layer. An air-gapped caller that
// has pre-seeded artifacts should use [`Self::from_files`], which is
// network-free by construction and unaffected by this guard.
crate::offline::guard_egress("Kompress model download", &config.model_id)?;
let api = hf_hub::api::sync::Api::new().map_err(|e| KompressError::Hub {
repo: config.model_id.clone(),
source: Box::new(e),
@@ -635,6 +657,38 @@ fn hf_hub_roots() -> Vec<PathBuf> {
mod tests {
use super::*;
/// `from_pretrained` must refuse before `Api::new()` when the operator
/// air-gapped the box.
///
/// The model id cannot exist, which is what makes this a real assertion
/// rather than a tautology: without the guard the call reaches `hf-hub`,
/// dials `huggingface.co` and comes back `KompressError::Hub`. Only the
/// guard can produce `Offline`, and only by returning before the client is
/// built — so `Offline` *is* the "no socket was opened" assertion here.
#[test]
fn from_pretrained_refuses_while_offline() {
let _guard = crate::test_support::env_lock();
std::env::set_var(crate::offline::OFFLINE_ENV, "1");
let config = KompressConfig {
model_id: "headroomlabs-ai/this-model-does-not-exist-a2".to_string(),
..KompressConfig::default()
};
let result = Kompress::from_pretrained(config);
std::env::remove_var(crate::offline::OFFLINE_ENV);
match result {
Err(KompressError::Offline(blocked)) => {
assert_eq!(blocked.purpose, "Kompress model download");
assert_eq!(
blocked.destination,
"headroomlabs-ai/this-model-does-not-exist-a2"
);
}
Err(other) => panic!("expected an offline refusal, got a network error: {other}"),
Ok(_) => panic!("expected an offline refusal, got a loaded model"),
}
}
#[test]
fn config_defaults_match_kompress_v2_base() {
let c = KompressConfig::default();
+1 -1
View File
@@ -341,7 +341,7 @@ with a privileged job.
| `HEADROOM_PROXY_TOKEN` | none | Require a token from non-loopback callers, as `X-Headroom-Proxy-Token: <token>` or `Authorization: Bearer <token>`. The token is removed from the request before it is forwarded upstream; clients that also send a provider credential should use `X-Headroom-Proxy-Token` and keep `Authorization` for the provider. Also gates the read-only operator routes (`/stats-history`, `/quota`, `/subscription-window`, `/metrics`), which otherwise answer loopback and trusted-CIDR callers only. |
| `HEADROOM_ALLOW_UNAUTHENTICATED_BIND` | `false` | Acknowledge a non-loopback bind (`--host 0.0.0.0`, `::`, a LAN address) with **no** `HEADROOM_PROXY_TOKEN`. Without a token or this flag the proxy refuses to start, because `/v1/*` would relay upstream with the operator's credentials for every peer that can reach the port. Only set it when the runtime already restricts reachability (a container published on `127.0.0.1`). |
| `HEADROOM_COMPRESS_ALLOW_REMOTE` | `false` | Allow non-loopback callers to reach [`POST /v1/compress`](#post-v1compress) and `POST /v1/usage`. Required to run Headroom as a remote gateway/sidecar; without it remote callers get `404`. |
| `--offline` / `HEADROOM_OFFLINE` | `false` | Air-gap mode: hard-disable **all** egress (telemetry, update checks, license reporting, model downloads). |
| `--offline` / `HEADROOM_OFFLINE` | `false` | Air-gap mode: refuse every connection Headroom itself initiates — telemetry, update checks, license reporting, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP and Langfuse export, Copilot auth, the subscription pollers, the OpenAI embedders, Headroom Cloud compression and the TLS diagnostics (`headroom doctor --network` reports them as skipped). Four things still work on purpose: forwarding your own requests to the upstream you configured, operator-configured local endpoints (the Ollama embedder), loopback health probes, and paths an `is_offline()` check already disables. Refusals surface as messages, not tracebacks. See the air-gap notes in `docs/metrics-technical-guide.md`. |
| `HEADROOM_LICENSE` | unset | Licence token, read by core and every licensed extension. `HEADROOM_LICENSE_KEY` is a deprecated alias for one release and logs a warning. A licence on its own sends nothing to Headroom. |
| `HEADROOM_USAGE_REPORTING` | `false` | Set `1` to let the proxy validate the licence and post aggregate usage counts (no content) to the Headroom cloud. Off unless set; `HEADROOM_OFFLINE` still overrides it. |
| `--stateless` / `HEADROOM_STATELESS` | `false` | Keep all state in memory; no filesystem writes (disables logs, memory, TOIN). |
+12 -1
View File
@@ -268,6 +268,17 @@ Verify with `curl -s localhost:8787/stats | jq .otel`.
**Multi-tenant labels:** `register_otel_metric_attribute_provider()` adds request-scoped attributes (tenant, team, cost centre) to every OTel datapoint. Max 16 attributes, 256 chars each.
**Air-gapped deployments:** `HEADROOM_OFFLINE=1` disables all outbound traffic — the anonymous usage beacon (which is **on by default**), the update check, and model downloads.
**Air-gapped deployments:** `HEADROOM_OFFLINE=1` refuses every connection Headroom itself decides to make to a destination Headroom itself chose. That is the whole list, not a sample: the anonymous usage beacon (which is **on by default**), the update check, the license/usage reporter, **OTLP metric and Langfuse trace export**, HuggingFace / Kompress / fastembed model and tokenizer downloads (Python and Rust), the remote Kompress endpoint, the `headroom install` release-binary and codebase-memory-mcp downloads, the BFCL eval-dataset fetch and the provider SDK clients the eval harness drives, GitHub Copilot device-flow auth and token exchange, the Anthropic / Codex / Copilot subscription pollers, the OpenAI embedders, the Headroom Cloud compression modes in the ASGI and LiteLLM integrations, and the TLS diagnostics (`headroom doctor --network` endpoint checks and the certificate-chain re-probe after an upstream TLS failure).
Four things are deliberately still allowed, and they are the complete set of exceptions:
1. **Your traffic through the proxy.** Requests you send *through* Headroom are still forwarded to the upstream you configured. That is your traffic, not Headroom's, and an air-gapped deployment points it at an on-prem endpoint — refusing it would mean refusing to be a proxy.
2. **Operator-configured local endpoints.** Today that is exactly one thing: the Ollama embedder, whose address comes entirely from your configuration and defaults to `http://localhost:11434`. No hard-coded internet host is permitted under this exception.
3. **Loopback.** Health and readiness probes against your own proxy on `127.0.0.1` (`headroom doctor`, the installers, the MCP sidecar).
4. **Paths an `is_offline()` check already makes unreachable**, where no connection is ever built in the first place.
Those four are enumerated with written reasons in `_EGRESS_ALLOWLIST` in `tests/test_offline_egress_chokepoint.py`, which fails the build if a new egress path appears that is neither guarded nor one of them. There is no category for "known violation": a path that can dial the internet with the flag set is a bug. A refusal reaches you as a message — a Click error on the CLI, a named "model unavailable" from a model loader, a logged line from a background poller — never as a traceback and never as a silent degradation. Enforcing the same policy at the network layer as well is still good practice; it is no longer the only thing standing between you and Headroom-initiated egress.
OTLP export is refused loudly: with `HEADROOM_OFFLINE=1` set, `HEADROOM_OTEL_METRICS_ENABLED=1` plus the default `otlp_http` exporter makes the proxy exit 78 at startup with an explanation, rather than quietly dropping metrics. There is no exemption for a collector that looks local — an in-cluster address is not reliably distinguishable from an internet one. For metrics under an air-gap, either scrape `localhost:8787/metrics` or set `HEADROOM_OTEL_METRICS_EXPORTER=console`, both of which stay on-box. `HEADROOM_KOMPRESS_ENDPOINT` and `HEADROOM_LANGFUSE_ENABLED` are refused the same way, for the same reason.
---
+6
View File
@@ -51,6 +51,7 @@ from pathlib import Path
from typing import Any
from headroom._subprocess import run
from headroom.offline import guard_egress
logger = logging.getLogger(__name__)
@@ -266,6 +267,11 @@ def _mirror_url(url: str) -> str:
def _download(url: str, dest: Path, *, progress: bool = True) -> None:
if os.environ.get("HEADROOM_BINARIES_OFFLINE"):
raise OfflineError(f"offline mode (HEADROOM_BINARIES_OFFLINE=1) but fetch required: {url}")
# HEADROOM_BINARIES_OFFLINE above is the narrow, binaries-only switch; this
# is the deployment-wide one. An operator who set the air-gap switch should
# not have to also discover a second, differently-named flag to stop
# Headroom fetching release assets from GitHub at install time.
guard_egress("release binary download", url)
if not _has_writable_existing_parent(dest.parent):
raise OSError(f"binary cache directory parent is not writable: {dest.parent}")
dest.parent.mkdir(parents=True, exist_ok=True)
+23 -2
View File
@@ -843,6 +843,28 @@ def run_network_probes(extra_urls: Sequence[str]) -> tuple[list[Any], set[str]]:
return reports, required
def network_checks(extra_urls: Sequence[str]) -> list[CheckResult]:
"""The ``--network`` rows, or one skipped row when ``HEADROOM_OFFLINE`` is set.
Every probe here dials an internet host, so under the air-gap switch none
of them runs; saying so beats a page of "unreachable" failures.
"""
from headroom.offline import OFFLINE_ENV, OfflineEgressBlocked
try:
reports, required = run_network_probes(extra_urls)
except OfflineEgressBlocked as blocked:
return [
CheckResult(
name="network",
status=SKIP,
summary=f"skipped: {OFFLINE_ENV} is set, no endpoint was contacted",
hint=str(blocked),
)
]
return check_network_endpoints(reports, required)
def check_shell_env(environ: Mapping[str, str], port: int) -> CheckResult:
"""Is the *current shell* pointed at the proxy for ad-hoc runs?"""
name = "shell env"
@@ -1233,8 +1255,7 @@ def doctor(
if proxy_env_check is not None:
checks.append(proxy_env_check)
if network or network_urls:
reports, required = run_network_probes(network_urls)
checks.extend(check_network_endpoints(reports, required))
checks.extend(network_checks(network_urls))
auth_conflict_check = check_claude_auth_conflict(
claude_settings_path(),
project_claude_settings,
+31 -1
View File
@@ -2,9 +2,39 @@
import click
from headroom.offline import OfflineEgressBlocked
CLI_CONTEXT_SETTINGS = {"help_option_names": ["--help", "-?"]}
class OfflineAwareGroup(click.Group):
"""The CLI's single translation point for an air-gap refusal.
``OfflineEgressBlocked`` is a ``BaseException`` on purpose, so no
``except Exception`` anywhere in the tree can downgrade a policy refusal
into "that feature is having a bad day". The cost of that choice is that
nothing catches it either — Click's own error handling only knows about
``ClickException``/``Abort``, so an operator who ran ``headroom auth
copilot`` on an air-gapped box would have got a stack trace ending in a
type they have never heard of.
Translating here, once, at the outermost boundary of every subcommand,
turns it into the ordinary ``Error: ...`` line Click prints for any other
bad invocation — with the switch, the feature and the host already in the
message, because :class:`OfflineEgressBlocked` builds them in. The guard
keeps its unswallowable property everywhere inside; only the last frame
before the user's terminal is allowed to make it readable.
"""
def invoke(self, ctx: click.Context) -> object:
try:
return super().invoke(ctx)
except OfflineEgressBlocked as blocked:
# ``from None``: the traceback is Headroom's own plumbing and
# tells the operator nothing they can act on. The message does.
raise click.ClickException(str(blocked)) from None
def get_version() -> str:
"""Get the current version."""
try:
@@ -15,7 +45,7 @@ def get_version() -> str:
return "unknown"
@click.group(context_settings=CLI_CONTEXT_SETTINGS)
@click.group(cls=OfflineAwareGroup, context_settings=CLI_CONTEXT_SETTINGS)
@click.version_option(get_version(), "--version", "-v", prog_name="headroom")
@click.pass_context
def main(ctx: click.Context) -> None:
+21 -1
View File
@@ -26,6 +26,7 @@ from headroom import paths
from headroom._subprocess import run
from headroom.copilot_linux_secret import read_copilot_oauth_token as read_linux_secret_token
from headroom.copilot_macos_keychain import read_copilot_oauth_token as read_macos_keychain_token
from headroom.offline import guard_egress
from headroom.proxy import ssl_context as proxy_ssl_context
logger = logging.getLogger(__name__)
@@ -79,8 +80,20 @@ _EXPIRY_KEYS = ("expires_at", "expiresAt", "expiry", "expires")
def _urlopen(request: urllib_request.Request, *, timeout: float) -> Any:
"""Open a GitHub request with Headroom's configured corporate trust roots."""
"""Open a GitHub request with Headroom's configured corporate trust roots.
Every GitHub call in this module bottoms out here, so this is also the
backstop for the air-gap switch. The four call sites guard individually
too — they can name the specific step ("device-flow authorisation",
"token exchange") in the refusal, which this frame cannot — but a fifth
call site added later inherits the refusal from here whether its author
thought about ``HEADROOM_OFFLINE`` or not.
"""
guard_egress(
"GitHub Copilot authentication",
urlparse(request.full_url).hostname or request.full_url,
)
context = proxy_ssl_context.build_urlopen_context()
if context is not None:
return urllib_request.urlopen(request, timeout=timeout, context=context)
@@ -699,6 +712,7 @@ def start_copilot_device_authorization(
"""Start the GitHub Copilot OAuth device-code flow."""
urls = _github_oauth_urls(domain)
guard_egress("GitHub Copilot device-flow authorisation", domain)
body = urlencode({"client_id": COPILOT_CHAT_OAUTH_CLIENT_ID, "scope": "read:user"}).encode(
"utf-8"
)
@@ -730,6 +744,7 @@ def poll_copilot_device_authorization(
"""Poll GitHub until the device-code OAuth flow returns an access token."""
urls = _github_oauth_urls(domain)
guard_egress("GitHub Copilot device-flow authorisation", domain)
deadline = time.time() + max(1, expires_in)
poll_interval = max(1, interval)
while time.time() < deadline:
@@ -1423,6 +1438,7 @@ def _fetch_copilot_user_info(token: str) -> dict[str, Any] | None:
if not token:
return None
guard_egress("GitHub Copilot account lookup", urlparse(_user_info_url()).hostname)
headers = _copilot_token_exchange_headers(token)
request = urllib_request.Request(_user_info_url(), headers=headers, method="GET")
try:
@@ -1540,6 +1556,10 @@ class CopilotTokenProvider:
@staticmethod
def _exchange_token_sync(headers: dict[str, str]) -> dict[str, Any]:
guard_egress(
"GitHub Copilot API token exchange",
urlparse(_token_exchange_url()).hostname,
)
request = urllib_request.Request(_token_exchange_url(), headers=headers, method="GET")
try:
with _urlopen(request, timeout=10.0) as response:
+2
View File
@@ -46,6 +46,7 @@ from headroom.evals.metrics import (
compute_rouge_l,
compute_semantic_similarity,
)
from headroom.offline import guard_egress
from headroom.transforms.content_router import ContentRouter, ContentRouterConfig
logger = logging.getLogger(__name__)
@@ -1029,6 +1030,7 @@ class BatchCompressionEvaluator:
def _init_llm_client(self) -> Any:
"""Initialize LLM client."""
guard_egress(f"{self.provider} API for the batch compression benchmark")
if self.provider == "anthropic":
try:
import anthropic
+46 -22
View File
@@ -28,10 +28,14 @@ Custom:
from __future__ import annotations
import json
import logging
from pathlib import Path
from typing import Any
from headroom.evals.core import EvalCase, EvalSuite
from headroom.offline import guard_egress, is_offline
logger = logging.getLogger(__name__)
def _check_datasets_installed() -> None:
@@ -45,6 +49,36 @@ def _check_datasets_installed() -> None:
) from e
def load_hf_dataset(path: str, *args: Any, **kwargs: Any) -> Any:
"""``datasets.load_dataset``, cache first under ``HEADROOM_OFFLINE``.
``load_dataset`` takes no ``local_files_only``, and with an explicit
``HF_HUB_OFFLINE=0`` it fetches from the Hub even when ``HEADROOM_OFFLINE``
is set. So, as ``relevance/embedding._load_text_embedding`` does for
fastembed: under the switch, try the load with the datasets and
huggingface_hub offline constants forced on (a pre-seeded cache still
loads), and only on a miss reach :func:`guard_egress`, which refuses
before any remote attempt. Online, nothing changes.
"""
from datasets import load_dataset
if is_offline():
import datasets.config as datasets_config
from huggingface_hub import constants as hf_constants
previous = (datasets_config.HF_HUB_OFFLINE, hf_constants.HF_HUB_OFFLINE)
datasets_config.HF_HUB_OFFLINE = True
hf_constants.HF_HUB_OFFLINE = True
try:
return load_dataset(path, *args, **kwargs)
except Exception as cache_miss: # noqa: BLE001 - any local-lookup failure
logger.debug("eval dataset %s not in the local cache: %s", path, cache_miss)
finally:
datasets_config.HF_HUB_OFFLINE, hf_constants.HF_HUB_OFFLINE = previous
guard_egress(f"HuggingFace eval-dataset download ({path})", "huggingface.co")
return load_dataset(path, *args, **kwargs)
# =============================================================================
# RAG / RETRIEVAL DATASETS
# =============================================================================
@@ -69,9 +103,8 @@ def load_hotpotqa(
EvalSuite with HotpotQA cases
"""
_check_datasets_installed()
from datasets import load_dataset
ds = load_dataset("hotpotqa/hotpot_qa", "fullwiki", split=split)
ds = load_hf_dataset("hotpotqa/hotpot_qa", "fullwiki", split=split)
cases: list[EvalCase] = []
for i, item in enumerate(ds):
@@ -121,9 +154,8 @@ def load_natural_questions(
EvalSuite with Natural Questions cases
"""
_check_datasets_installed()
from datasets import load_dataset
ds = load_dataset("google-research-datasets/natural_questions", "default", split=split)
ds = load_hf_dataset("google-research-datasets/natural_questions", "default", split=split)
cases: list[EvalCase] = []
for i, item in enumerate(ds):
@@ -198,9 +230,8 @@ def load_triviaqa(
EvalSuite with TriviaQA cases
"""
_check_datasets_installed()
from datasets import load_dataset
ds = load_dataset("trivia_qa", subset, split=split)
ds = load_hf_dataset("trivia_qa", subset, split=split)
cases: list[EvalCase] = []
for i, item in enumerate(ds):
@@ -269,9 +300,8 @@ def load_msmarco(
EvalSuite with MS MARCO cases
"""
_check_datasets_installed()
from datasets import load_dataset
ds = load_dataset("microsoft/ms_marco", "v2.1", split=split)
ds = load_hf_dataset("microsoft/ms_marco", "v2.1", split=split)
cases: list[EvalCase] = []
for i, item in enumerate(ds):
@@ -337,9 +367,8 @@ def load_squad(
EvalSuite with SQuAD cases
"""
_check_datasets_installed()
from datasets import load_dataset
ds = load_dataset("rajpurkar/squad_v2", split=split)
ds = load_hf_dataset("rajpurkar/squad_v2", split=split)
cases: list[EvalCase] = []
for i, item in enumerate(ds):
@@ -398,10 +427,9 @@ def load_longbench(
EvalSuite with LongBench cases
"""
_check_datasets_installed()
from datasets import load_dataset
try:
ds = load_dataset("THUDM/LongBench", task, split="test")
ds = load_hf_dataset("THUDM/LongBench", task, split="test")
except Exception as e:
raise ValueError(f"Failed to load LongBench task '{task}': {e}") from e
@@ -460,10 +488,9 @@ def load_multi_wiki_qa(
EvalSuite with multi-wiki-qa cases
"""
_check_datasets_installed()
from datasets import load_dataset
try:
ds = load_dataset("alexandrainst/multi-wiki-qa", lang, split=f"train[:{n}]")
ds = load_hf_dataset("alexandrainst/multi-wiki-qa", lang, split=f"train[:{n}]")
except Exception as e:
raise ValueError(f"Failed to load multi-wiki-qa '{lang}': {e}") from e
@@ -513,9 +540,8 @@ def load_narrativeqa(
EvalSuite with NarrativeQA cases
"""
_check_datasets_installed()
from datasets import load_dataset
ds = load_dataset("deepmind/narrativeqa", split=split)
ds = load_hf_dataset("deepmind/narrativeqa", split=split)
cases: list[EvalCase] = []
for i, item in enumerate(ds):
@@ -590,6 +616,7 @@ def load_bfcl(
import urllib.request
base_url = "https://huggingface.co/datasets/gorilla-llm/Berkeley-Function-Calling-Leaderboard/resolve/main"
guard_egress("BFCL eval-dataset download", base_url)
data_file = f"BFCL_v3_{category}.json"
gt_file = f"possible_answer/BFCL_v3_{category}.json"
@@ -676,10 +703,9 @@ def load_toolbench(
EvalSuite with ToolBench cases
"""
_check_datasets_installed()
from datasets import load_dataset
try:
ds = load_dataset("ToolBench/ToolBench", category, split="test")
ds = load_hf_dataset("ToolBench/ToolBench", category, split="test")
except Exception as e:
raise ValueError(f"Failed to load ToolBench category '{category}': {e}") from e
@@ -759,10 +785,9 @@ def load_codesearchnet(
EvalSuite with CodeSearchNet cases
"""
_check_datasets_installed()
from datasets import load_dataset
try:
ds = load_dataset("code_search_net", language, split=split)
ds = load_hf_dataset("code_search_net", language, split=split)
except Exception as e:
raise ValueError(f"Failed to load CodeSearchNet for '{language}': {e}") from e
@@ -815,9 +840,8 @@ def load_humaneval(
EvalSuite with HumanEval cases
"""
_check_datasets_installed()
from datasets import load_dataset
ds = load_dataset("openai_humaneval", split="test")
ds = load_hf_dataset("openai_humaneval", split="test")
cases: list[EvalCase] = []
for i, item in enumerate(ds):
+4
View File
@@ -15,6 +15,8 @@ from collections.abc import Callable
from dataclasses import dataclass, field
from typing import TYPE_CHECKING, Any
from headroom.offline import guard_egress
if TYPE_CHECKING:
from headroom.transforms.html_extractor import HTMLExtractor
@@ -255,6 +257,7 @@ class HTMLExtractionEvaluator:
def _create_judge(self) -> Callable[[str, str, str], tuple[float, str]]:
"""Create the LLM judge function."""
guard_egress(f"{self.provider} API for the HTML-extraction eval judge")
if self.provider == "openai":
try:
from openai import OpenAI
@@ -355,6 +358,7 @@ class HTMLExtractionEvaluator:
def _get_answer(self, content: str, question: str) -> str:
"""Get LLM answer for a question given content."""
guard_egress(f"{self.provider} API for the HTML-extraction eval answers")
prompt = f"""Based on the following content, answer the question.
Content:
+11 -5
View File
@@ -142,12 +142,15 @@ def evaluate_scrapinghub_benchmark(
print(f"F1: {result.avg_f1:.3f} (baseline: {result.baseline_f1})")
"""
try:
from datasets import load_dataset
import datasets # noqa: F401
except ImportError:
raise ImportError(
"HuggingFace datasets required. Install with: pip install datasets"
) from None
# Cache first under HEADROOM_OFFLINE, guard_egress before any download.
from headroom.evals.datasets import load_hf_dataset
if extractor is None:
from headroom.transforms.html_extractor import HTMLExtractor
@@ -155,7 +158,7 @@ def evaluate_scrapinghub_benchmark(
# Load the benchmark dataset
logger.info("Loading Scrapinghub article extraction benchmark...")
dataset = load_dataset("allenai/scrapinghub-article-extraction-benchmark")
dataset = load_hf_dataset("allenai/scrapinghub-article-extraction-benchmark")
samples = dataset["train"]
if max_samples:
@@ -274,10 +277,13 @@ def evaluate_qa_accuracy_preservation(
QAAccuracyResult showing whether accuracy is preserved
"""
try:
from datasets import load_dataset
import datasets # noqa: F401
except ImportError:
raise ImportError("HuggingFace datasets required") from None
# Cache first under HEADROOM_OFFLINE, guard_egress before any download.
from headroom.evals.datasets import load_hf_dataset
if extractor is None:
from headroom.transforms.html_extractor import HTMLExtractor
@@ -287,9 +293,9 @@ def evaluate_qa_accuracy_preservation(
logger.info(f"Loading {dataset_name} dataset...")
if dataset_name == "squad":
dataset = load_dataset("rajpurkar/squad_v2", split="validation")
dataset = load_hf_dataset("rajpurkar/squad_v2", split="validation")
elif dataset_name == "hotpotqa":
dataset = load_dataset("hotpotqa/hotpot_qa", "fullwiki", split="validation")
dataset = load_hf_dataset("hotpotqa/hotpot_qa", "fullwiki", split="validation")
else:
raise ValueError(f"Unknown dataset: {dataset_name}")
+4
View File
@@ -17,6 +17,8 @@ from __future__ import annotations
import logging
from collections.abc import Callable
from headroom.offline import guard_egress
logger = logging.getLogger(__name__)
# Prompt template for LLM judge
@@ -73,6 +75,7 @@ def create_openai_judge(
"OpenAI package required for LLM judge. Install with: pip install openai"
) from e
guard_egress("OpenAI API for the memory eval LLM judge", "api.openai.com")
client = OpenAI(api_key=api_key) if api_key else OpenAI()
def judge(question: str, ground_truth: str, prediction: str) -> tuple[float, str]:
@@ -124,6 +127,7 @@ def create_anthropic_judge(
"Anthropic package required for LLM judge. Install with: pip install anthropic"
) from e
guard_egress("Anthropic API for the memory eval LLM judge", "api.anthropic.com")
client = anthropic.Anthropic(api_key=api_key) if api_key else anthropic.Anthropic()
def judge(question: str, ground_truth: str, prediction: str) -> tuple[float, str]:
+3
View File
@@ -32,6 +32,8 @@ from datetime import datetime
from pathlib import Path
from typing import Any
from headroom.offline import guard_egress
logger = logging.getLogger(__name__)
@@ -259,6 +261,7 @@ def compare_prompts(
"or pass api_key parameter."
)
guard_egress("OpenAI API for the prompt-comparison judge", "api.openai.com")
client = OpenAI(api_key=resolved_api_key)
# Build the judge prompt
+12 -1
View File
@@ -25,6 +25,7 @@ from headroom.evals.core import (
EvalSuiteResult,
)
from headroom.evals.metrics import compute_semantic_similarity
from headroom.offline import guard_egress
from headroom.transforms.content_router import ContentRouter, ContentRouterConfig
from headroom.transforms.smart_crusher import SmartCrusherConfig
@@ -95,8 +96,13 @@ class BeforeAfterRunner:
self._judge_fn: Any = None
def _init_llm_client(self) -> Any:
"""Initialize the appropriate LLM client."""
"""Initialize the appropriate LLM client.
Ollama is not guarded: like the Ollama embedder it is an
operator-configured endpoint that defaults to localhost.
"""
if self.llm_config.provider == "anthropic":
guard_egress("anthropic API for the before/after eval runner")
try:
import anthropic
@@ -106,6 +112,7 @@ class BeforeAfterRunner:
"anthropic package required. Install with: pip install anthropic"
) from e
elif self.llm_config.provider == "openai":
guard_egress("openai API for the before/after eval runner")
try:
import openai
@@ -128,6 +135,10 @@ class BeforeAfterRunner:
def _init_proxy_client(self) -> Any:
"""Initialize an OpenAI client pointing at the Headroom proxy."""
guard_egress(
"LLM calls through the Headroom proxy for the before/after eval runner",
self.llm_config.headroom_proxy_url,
)
import openai
return openai.OpenAI(
+3
View File
@@ -12,6 +12,8 @@ import zipfile
from pathlib import Path
from urllib.request import urlopen
from headroom.offline import guard_egress
logger = logging.getLogger(__name__)
CBM_VERSION = "v0.8.1"
@@ -98,6 +100,7 @@ def download_cbm(version: str | None = None) -> Path:
plat = _detect_platform()
filename = _asset_filename(plat)
url = f"{GITHUB_RELEASE_URL}/{version}/{filename}"
guard_egress("codebase-memory-mcp binary download", url)
CBM_BIN_DIR.mkdir(parents=True, exist_ok=True)
target_path = CBM_BIN_DIR / _installed_name(plat)
+40 -11
View File
@@ -22,6 +22,7 @@ from typing import Any
import numpy as np
from headroom.image.image_types import ImageMemo, ImageSignals, RouteDecision, Technique
from headroom.offline import OFFLINE_ENV, OfflineEgressBlocked
from headroom.onnx_runtime import create_cpu_session_options, hf_hub_download_local_first
logger = logging.getLogger(__name__)
@@ -67,6 +68,25 @@ def _classifier_intra_op_threads() -> int:
# ImageSignals, RouteDecision, Technique imported from trained_router
def _hf_artifact(repo: str, filename: str) -> str:
"""``hf_hub_download_local_first``, translating the air-gap refusal.
``OfflineEgressBlocked`` is a ``BaseException`` so that it cannot be
swallowed by accident. Here it is handled on purpose: ``ImageCompressor``
already degrades a router that will not load to ``Technique.PRESERVE``, and
an air-gapped box with no cached router should get that same degradation
rather than a failed request. Re-raised as a plain ``RuntimeError``, with
the switch named in the message, so the existing handler sees it and the
log says why.
"""
try:
return hf_hub_download_local_first(repo, filename)
except OfflineEgressBlocked as blocked:
raise RuntimeError(
f"{filename} for {repo} is not cached and {OFFLINE_ENV} forbids fetching it: {blocked}"
) from blocked
class OnnxTechniqueRouter:
"""ONNX-based technique router — no PyTorch dependency.
@@ -98,8 +118,15 @@ class OnnxTechniqueRouter:
logger.info("Loading technique-router ONNX INT8...")
model_path = hf_hub_download_local_first(_TECHNIQUE_ROUTER_REPO, "model_quantized.onnx")
self._classifier_session = ort.InferenceSession(
# Resolve every artifact before publishing anything: the session is the
# "loaded" flag, so setting it before a later artifact fails (e.g. not
# cached under HEADROOM_OFFLINE) would leave a half-loaded router that
# never retries.
model_path = _hf_artifact(_TECHNIQUE_ROUTER_REPO, "model_quantized.onnx")
tokenizer_path = _hf_artifact(_TECHNIQUE_ROUTER_REPO, "tokenizer.json")
config_path = _hf_artifact(_TECHNIQUE_ROUTER_REPO, "config.json")
session = ort.InferenceSession(
model_path,
create_cpu_session_options(
ort,
@@ -109,18 +136,18 @@ class OnnxTechniqueRouter:
providers=["CPUExecutionProvider"],
)
tokenizer_path = hf_hub_download_local_first(_TECHNIQUE_ROUTER_REPO, "tokenizer.json")
self._tokenizer = Tokenizer.from_file(tokenizer_path)
self._tokenizer.enable_truncation(max_length=64)
self._tokenizer.enable_padding(length=64)
tokenizer = Tokenizer.from_file(tokenizer_path)
tokenizer.enable_truncation(max_length=64)
tokenizer.enable_padding(length=64)
# Load label mapping
import json
config_path = hf_hub_download_local_first(_TECHNIQUE_ROUTER_REPO, "config.json")
with open(config_path) as f:
config = json.load(f)
self._id2label = {int(k): v for k, v in config.get("id2label", {}).items()}
self._tokenizer = tokenizer
self._classifier_session = session
logger.info(
f"Technique router loaded: {len(self._id2label)} classes, "
@@ -136,16 +163,18 @@ class OnnxTechniqueRouter:
logger.info("Loading SigLIP ONNX INT8 image encoder...")
model_path = hf_hub_download_local_first(_SIGLIP_ENCODER_REPO, "image_encoder_int8.onnx")
self._siglip_session = ort.InferenceSession(
# Both artifacts first, session last — see _load_classifier.
model_path = _hf_artifact(_SIGLIP_ENCODER_REPO, "image_encoder_int8.onnx")
embeddings_path = _hf_artifact(_SIGLIP_ENCODER_REPO, "text_embeddings.npz")
session = ort.InferenceSession(
model_path,
create_cpu_session_options(ort),
providers=["CPUExecutionProvider"],
)
embeddings_path = hf_hub_download_local_first(_SIGLIP_ENCODER_REPO, "text_embeddings.npz")
loaded = np.load(embeddings_path)
self._text_embeddings = {k: loaded[k] for k in loaded.files}
self._siglip_session = session
logger.info(
f"SigLIP image encoder loaded: ONNX INT8 "
+14 -1
View File
@@ -34,6 +34,8 @@ from typing import Any
from starlette.types import ASGIApp, Receive, Scope, Send
from headroom.offline import guard_egress
logger = logging.getLogger(__name__)
_DEFAULT_CLOUD_URL = "https://api.headroomlabs.ai"
@@ -204,7 +206,18 @@ class CompressionMiddleware:
}
async def _cloud_compress(self, messages: list[dict], model: str) -> dict[str, Any] | None:
"""Compress via Headroom Cloud API (managed CCR, TOIN, analytics)."""
"""Compress via Headroom Cloud API (managed CCR, TOIN, analytics).
This is the one path in this file that puts the caller's prompt
content on the wire to a Headroom-operated host, so it is exactly what
HEADROOM_OFFLINE exists to stop. "Opt-in by configuration" was the old
reason for leaving it open, and it is not good enough: an operator who
sets an air-gap switch is overriding earlier configuration on purpose.
The refusal is loud rather than a silent fall-through to local
compression, because silently compressing locally would hide the fact
that the deployment is no longer doing what it was configured to do.
"""
guard_egress("Headroom Cloud compression", self._api_url)
if self._client is None:
try:
import httpx
+14 -1
View File
@@ -20,6 +20,8 @@ import json
import logging
from typing import Any
from headroom.offline import guard_egress
logger = logging.getLogger(__name__)
_DEFAULT_CLOUD_URL = "https://api.headroomlabs.ai"
@@ -184,7 +186,18 @@ class HeadroomCallback(_CustomLogger):
}
async def _cloud_compress(self, messages: list[dict], model: str) -> dict[str, Any] | None:
"""Compress via Headroom Cloud API (managed CCR, TOIN, analytics)."""
"""Compress via Headroom Cloud API (managed CCR, TOIN, analytics).
This is the one path in this file that puts the caller's prompt
content on the wire to a Headroom-operated host, so it is exactly what
HEADROOM_OFFLINE exists to stop. "Opt-in by configuration" was the old
reason for leaving it open, and it is not good enough: an operator who
sets an air-gap switch is overriding earlier configuration on purpose.
The refusal is loud rather than a silent fall-through to local
compression, because silently compressing locally would hide the fact
that the deployment is no longer doing what it was configured to do.
"""
guard_egress("Headroom Cloud compression", self._api_url)
if self._client is None:
try:
import httpx
+16 -2
View File
@@ -21,6 +21,7 @@ from typing import TYPE_CHECKING, Any, cast
import numpy as np
from headroom.models.config import ML_MODEL_DEFAULTS
from headroom.offline import OFFLINE_ENV, OfflineEgressBlocked, guard_egress
from headroom.onnx_runtime import create_cpu_session_options, hf_hub_download_local_first
if TYPE_CHECKING:
@@ -450,8 +451,20 @@ class OnnxLocalEmbedder:
logger.info("Loading ONNX embedding model (all-MiniLM-L6-v2, ~86MB)...")
# Prefer local cache to avoid a redundant network HEAD on warm starts.
model_path = hf_hub_download_local_first(self.ONNX_REPO, "model.onnx")
tok_path = hf_hub_download_local_first(self.ONNX_REPO, "tokenizer.json")
# OfflineEgressBlocked is a BaseException so it cannot be swallowed by
# accident; here the degradation is deliberate and written down. A model
# download is not data leaving the box, and an air-gapped deployment
# without a cached embedder should get "the local embedder is
# unavailable" — the caller's existing failure mode — rather than a
# BaseException escaping into the memory pipeline.
try:
model_path = hf_hub_download_local_first(self.ONNX_REPO, "model.onnx")
tok_path = hf_hub_download_local_first(self.ONNX_REPO, "tokenizer.json")
except OfflineEgressBlocked as blocked:
raise RuntimeError(
f"local ONNX embedder {self.ONNX_REPO} is not cached and "
f"{OFFLINE_ENV} forbids fetching it: {blocked}"
) from blocked
# Keep a small thread pool for Docker compatibility and disable ORT's
# CPU memory arena/pattern caches so long-running proxy workers do not
@@ -655,6 +668,7 @@ class OpenAIEmbedder:
@cached_property
def _async_client(self) -> Any:
"""Lazy initialization of async OpenAI client."""
guard_egress("OpenAI embedding API", "api.openai.com")
from openai import AsyncOpenAI
return AsyncOpenAI(api_key=self._api_key)
+7 -1
View File
@@ -60,6 +60,7 @@ from typing import Any
from headroom.memory import qdrant_env
from headroom.memory.models import Memory
from headroom.memory.ports import MemorySearchResult
from headroom.offline import guard_egress
logger = logging.getLogger(__name__)
@@ -169,7 +170,12 @@ class DirectMem0Adapter:
if self._initialized:
return
# Initialize embedder (OpenAI)
# Initialize embedder (OpenAI). The air-gap switch outranks the
# backend configuration: this embeds the user's memories by shipping
# them to api.openai.com, which is precisely what an operator who set
# HEADROOM_OFFLINE is refusing. Use the ONNX or Ollama embedder
# instead on an air-gapped box.
guard_egress("OpenAI embedding API (direct mem0 backend)", "api.openai.com")
try:
from openai import OpenAI
+59 -5
View File
@@ -32,6 +32,7 @@ import logging
from threading import RLock
from typing import TYPE_CHECKING, Any
from ..offline import OFFLINE_ENV, OfflineEgressBlocked
from .config import ML_MODEL_DEFAULTS
if TYPE_CHECKING:
@@ -40,6 +41,42 @@ if TYPE_CHECKING:
logger = logging.getLogger(__name__)
class ModelUnavailableOffline(RuntimeError):
"""A model is not in the local cache and ``HEADROOM_OFFLINE`` forbids fetching it.
The legible form of an air-gap refusal at a model-loader boundary. Fetching
public weights is not data leaving the box, so the loader degrades instead
of failing the request: every caller of this registry already treats a
load failure as "feature unavailable" (the image router preserves the
image, the memory embedder falls back, ...). The original
:class:`~headroom.offline.OfflineEgressBlocked` is chained.
"""
def __init__(self, purpose: str, name_or_path: str) -> None:
self.purpose = purpose
self.name_or_path = name_or_path
super().__init__(
f"{purpose} {name_or_path!r} is not in the local HuggingFace cache and "
f"{OFFLINE_ENV} forbids downloading it. Pre-seed the cache to use this feature."
)
def _load_local_first(loader: Any, name_or_path: str, *, purpose: str, **kwargs: Any) -> Any:
"""Cache-first load with the air-gap refusal translated and logged.
Thin wrapper over :func:`headroom.onnx_runtime.hf_from_pretrained_local_first`
(the one place the cache-first + ``guard_egress`` order lives).
"""
from ..onnx_runtime import hf_from_pretrained_local_first
try:
return hf_from_pretrained_local_first(loader, name_or_path, purpose=purpose, **kwargs)
except OfflineEgressBlocked as blocked:
unavailable = ModelUnavailableOffline(purpose, name_or_path)
logger.warning("%s (%s)", unavailable, blocked)
raise unavailable from blocked
class MLModelRegistry:
"""Singleton registry for shared ML model instances.
@@ -168,7 +205,12 @@ class MLModelRegistry:
if device is None:
device = cls._detect_device()
model = SentenceTransformer(model_name, device=device)
model = _load_local_first(
SentenceTransformer,
model_name,
purpose="SentenceTransformer model",
device=device,
)
instance._models[key] = model
logger.info(f"Loaded SentenceTransformer: {model_name} on {device}")
@@ -207,8 +249,12 @@ class MLModelRegistry:
if device is None:
device = cls._detect_device()
model = AutoModel.from_pretrained(model_name)
processor = AutoProcessor.from_pretrained(model_name)
model = _load_local_first(
AutoModel.from_pretrained, model_name, purpose="SigLIP model"
)
processor = _load_local_first(
AutoProcessor.from_pretrained, model_name, purpose="SigLIP processor"
)
# Move to device and set eval mode
if device != "cpu":
@@ -295,8 +341,16 @@ class MLModelRegistry:
if device is None:
device = cls._detect_device()
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForSequenceClassification.from_pretrained(model_path)
tokenizer = _load_local_first(
AutoTokenizer.from_pretrained,
model_path,
purpose="technique-router tokenizer",
)
model = _load_local_first(
AutoModelForSequenceClassification.from_pretrained,
model_path,
purpose="technique-router model",
)
# Move to device and set eval mode
if device != "cpu":
+17
View File
@@ -13,6 +13,7 @@ from opentelemetry import metrics
from opentelemetry.metrics import CallbackOptions, Observation
from headroom._version import get_version
from headroom.offline import guard_egress
logger = logging.getLogger(__name__)
@@ -717,6 +718,22 @@ def configure_otel_metrics(config: OTelMetricsConfig | None = None) -> HeadroomO
if resolved.exporter == "console":
exporter = ConsoleMetricExporter()
else:
# Air-gap chokepoint. OTLP/HTTP export dials a collector and then keeps
# POSTing on a background timer (PeriodicExportingMetricReader), so it
# is exactly the kind of long-lived, easy-to-forget egress
# HEADROOM_OFFLINE is supposed to cover — and it was not covered.
#
# The guard runs before OTLPMetricExporter is constructed, because the
# exporter opens its session eagerly. No exemption for a collector that
# looks local: the endpoint may be unset here (the OTEL SDK then falls
# back to OTEL_EXPORTER_OTLP_ENDPOINT or the 4318 default) and a
# sidecar address is not distinguishable from an internet one. An
# operator who wants in-cluster metrics under an air-gap should use the
# "console" exporter or scrape /metrics, both of which stay on-box.
guard_egress(
"OTLP metric export",
resolved.endpoint or "the OTEL SDK's default OTLP endpoint",
)
exporter_kwargs: dict[str, Any] = {}
if resolved.endpoint is not None:
exporter_kwargs["endpoint"] = resolved.endpoint
+13
View File
@@ -11,6 +11,8 @@ from typing import Any
from opentelemetry import trace
from headroom.offline import guard_egress
from .metrics import _headroom_version, _parse_bool, _parse_key_value_pairs
logger = logging.getLogger(__name__)
@@ -158,6 +160,17 @@ def configure_langfuse_tracing(
)
return get_headroom_tracer()
# Air-gap chokepoint, the exact counterpart of the one in metrics.py.
# Langfuse OTLP tracing ships every span off the box on a background
# BatchSpanProcessor timer — and unlike the metrics exporter, the endpoint
# here is Langfuse Cloud unless the operator overrode it. It was missed
# when the metric exporter was guarded; the two are the same egress shape
# and now refuse the same way. Guarded before OTLPSpanExporter is
# constructed, because the exporter opens its session eagerly.
guard_egress(
"Langfuse OTLP trace export",
resolved.endpoint or "the OTEL SDK's default OTLP endpoint",
)
resource = Resource.create(
{
SERVICE_NAME: resolved.service_name,
+189 -6
View File
@@ -1,10 +1,59 @@
"""Air-gap / no-egress master switch (``HEADROOM_OFFLINE``).
A single predicate the individual egress paths consult so a regulated or
air-gapped deployment can disable **all** outbound network access with one
flag: the telemetry beacon, the update check, the license/usage reporter, and
HuggingFace model downloads. Each of those already had its own opt-out; this
is the one switch that turns them all off together and fails closed.
THE POLICY, in one sentence: with ``HEADROOM_OFFLINE`` set, Headroom refuses
every connection **Headroom itself decides to make to a destination Headroom
itself chose**, and keeps working for the two kinds of traffic an air-gapped
deployment exists to carry.
Refused (this is not a sample — it is the rule, and the meta-test enforces it):
the telemetry beacon, the update check, the license/usage reporter, OTLP and
Langfuse export, HuggingFace/Kompress/fastembed model downloads, release-binary
and codebase-memory-mcp downloads, eval dataset downloads and the provider SDK
clients the eval harness drives, GitHub Copilot device-flow auth and token
exchange, the Anthropic / Codex / Copilot subscription pollers, the OpenAI
embedders, both Headroom Cloud compression integrations, and the TLS
diagnostics (``doctor --network`` endpoint checks and the certificate-chain
re-probe after an upstream TLS failure).
Permitted, explicitly and by name — an air-gapped deployment needs these and
none of them is Headroom phoning home:
``user-traffic``
The proxy forwarding the caller's own request to the upstream the operator
configured. Forwarding a request the user's client made, to the endpoint
the user pointed Headroom at, is the proxy's entire job. An air-gapped
install points it at an on-prem model endpoint.
``operator-endpoint``
Headroom dialling an address that comes **entirely** from operator
configuration and whose default is loopback — today that is exactly one
thing, the Ollama embedder (``http://localhost:11434``). No hard-coded
internet host may hide behind this category.
``loopback``
Hard-coded ``127.0.0.1`` traffic to the operator's own local proxy: the
readiness and health probes, ``headroom doctor``, the MCP sidecar.
``gated``
A path that an ``is_offline()`` check upstream already makes unreachable,
so the connection is never even built.
Those four are the complete list of exceptions. There is no "known violation"
category: a path that dials out under ``HEADROOM_OFFLINE`` for any other
reason is a bug, not an entry in a table.
:func:`guard_egress` is the chokepoint: a path that is about to open an
outbound connection calls it and gets a loud :class:`OfflineEgressBlocked`
instead of a socket. Prefer it over a bare ``if is_offline(): return`` at any
call site that actually dials out — the meta-test in
``tests/test_offline_egress_chokepoint.py`` enumerates the outbound clients in
``headroom/`` and ``crates/`` and requires each **site** to have a guard that
dominates it or to be counted in an allowlist under one of the four categories
above, so the guard is what keeps a newly-added egress path from silently
escaping the air-gap.
A refusal must reach a human as a sentence, never as a traceback and never as
a silent degradation. Interactive paths get that from the CLI boundary in
``headroom/cli/main.py``, which turns the refusal into a Click error; model
loaders translate it into their own "model unavailable"; background pollers
call :func:`note_refusal` and stop.
Kept at the top level (depends only on the stdlib) so any layer — telemetry,
proxy, model code — can import it without creating a package cycle.
@@ -12,16 +61,32 @@ proxy, model code — can import it without creating a package cycle.
from __future__ import annotations
import logging
import os
_TRUE_VALUES = frozenset({"1", "true", "yes", "on"})
# Whitespace trimmed off the raw value before matching. Enumerated rather than
# left to ``str.strip()``'s default, because that default is Python's
# ``str.isspace()`` — which includes U+001C-U+001F (the ASCII file/group/record
# /unit separators) — while Rust's ``str::trim()`` is the Unicode White_Space
# property, which does not. ``HEADROOM_OFFLINE=$'\x1c1'`` therefore read as
# offline to Python and online to Rust: one process air-gapped, the other not,
# from one environment variable. Neither side's default is more right than the
# other, so both now trim exactly this set. Mirrored by ``TRIM_CHARS`` in
# ``crates/headroom-core/src/offline.rs``; the parity test compares them.
#
# Case folding needs no such treatment: Python's ``str.lower()`` and Rust's
# ``to_ascii_lowercase()`` differ only on non-ASCII input, and no non-ASCII
# character lowercases into any character of "1"/"true"/"yes"/"on".
_TRIM_CHARS = " \t\n\r\x0b\x0c"
OFFLINE_ENV = "HEADROOM_OFFLINE"
def is_offline() -> bool:
"""Return True when ``HEADROOM_OFFLINE`` selects fully-offline operation."""
return os.environ.get(OFFLINE_ENV, "").strip().lower() in _TRUE_VALUES
return os.environ.get(OFFLINE_ENV, "").strip(_TRIM_CHARS).lower() in _TRUE_VALUES
def apply_offline_env() -> None:
@@ -34,3 +99,121 @@ def apply_offline_env() -> None:
if is_offline():
os.environ.setdefault("HF_HUB_OFFLINE", "1")
os.environ.setdefault("TRANSFORMERS_OFFLINE", "1")
class OfflineEgressBlocked(BaseException):
"""Raised by :func:`guard_egress` when ``HEADROOM_OFFLINE`` is in force.
A distinct, named type so callers can tell "the operator air-gapped this
box" apart from "the network was flaky". That distinction matters because
several Headroom egress paths deliberately fail OPEN on network errors
(remote Kompress passes content through verbatim, the license reporter
falls back to a cached grant). Fail-open is right for a flaky endpoint and
WRONG for a policy refusal: swallowing this turns the air-gap switch back
into a suggestion.
**It derives from BaseException, not Exception, on purpose.** The first
version of this class was a ``RuntimeError`` whose docstring asked every
broad handler to re-raise it. Nothing did — there were zero re-raise sites
in the package — and the refusal was swallowed by four reachable
``except Exception`` blocks on the way out of a single ``/v1/messages``
request, which returned 200 with the content uncompressed while the guard
had fired twice. A convention that has to be re-applied at every future
``except Exception`` in a 100k-line codebase is not a guarantee; the type
hierarchy is. This is the same reason ``KeyboardInterrupt`` and
``SystemExit`` sit outside ``Exception``: a policy decision is control
flow, not a runtime error to degrade around.
Consequences to know about:
* ``except Exception`` no longer catches it anywhere — including in
third-party code (httpx, the OTEL SDK, Starlette's error middleware).
* ``except BaseException`` still does. Those are rare, they are enumerated
by ``tests/test_offline_egress_chokepoint.py``'s broad-handler sweep, and
each must either re-raise or carry a written reason.
* Reaching this at request time means the operator changed the environment
under a running proxy; the ordinary contradictory-configuration case is
caught at startup by ``headroom/proxy/server.py``'s offline preflight,
which exits 78 with an explanation instead.
* Where a refusal SHOULD become a degradation, the boundary that owns the
degradation says so explicitly. Fetching public model weights is not data
leaving the box, so each optional model loader catches this and re-raises
its own "model unavailable" error with the switch named — see
``kompress_compressor._hf_artifact``, ``image/onnx_router._hf_artifact``
and the ONNX embedder in ``memory/adapters/embedders.py``. Translating is
fine; inheriting a translation from whichever ``except Exception`` happens
to be in the stack is what this type prevents.
"""
def __init__(self, purpose: str, destination: str | None = None) -> None:
self.purpose = purpose
self.destination = destination
where = f" to {destination}" if destination else ""
super().__init__(
f"{OFFLINE_ENV} is set: refusing outbound network access for "
f"{purpose}{where}. Unset {OFFLINE_ENV}, or turn off the feature "
f"that needs this connection."
)
def guard_egress(purpose: str, destination: str | None = None) -> None:
"""The single chokepoint every Headroom-initiated egress path must call.
Raise :class:`OfflineEgressBlocked` when ``HEADROOM_OFFLINE`` selects
offline operation; return silently otherwise.
Call it BEFORE the socket exists — before constructing the client, not
just before the request — so a pooled/keep-alive connection is never even
opened. ``purpose`` and ``destination`` land verbatim in the message, so
an operator who trips this learns which feature to turn off.
Why raise instead of returning a no-op result: a silent skip is
indistinguishable from success at the call site, so a future refactor can
quietly reintroduce egress and nothing fails. A loud, named exception is
what makes the meta-test in ``tests/test_offline_egress_chokepoint.py``
able to assert "every egress path is behind this or allowlisted".
Deliberately NOT exempted:
* Loopback / in-cluster destinations. The guard cannot reliably tell an
in-cluster collector from an internet host (DNS, proxies and sidecars
all blur it), and ``HEADROOM_OFFLINE`` is documented as "no outbound
traffic". Paths that only ever talk to ``127.0.0.1`` — the readiness
probes, the proxy's own ``/livez`` check — simply do not call the guard
and are allowlisted by name in the meta-test instead.
* The proxy's own request forwarding to the caller's configured upstream.
That is the caller's traffic, not Headroom phoning home; an air-gapped
deployment points it at an on-prem endpoint and still needs it to work.
"""
if is_offline():
raise OfflineEgressBlocked(purpose, destination)
# Purposes already reported, so a poller that runs every 30s says it once at
# WARNING and then keeps quiet. Module-level rather than per-object because
# several of these pollers are recreated per poll.
_REPORTED_REFUSALS: set[str] = set()
def note_refusal(blocked: OfflineEgressBlocked, log: logging.Logger) -> str:
"""Report an air-gap refusal on a background path and return its message.
Background pollers and other fire-and-forget tasks cannot let a
``BaseException`` escape — an unhandled task exception is a traceback in
the log, not an explanation, and asyncio may not surface it until the
task is garbage collected. They also must not swallow the refusal
silently, because "the subscription window stopped updating" with no
stated cause is the exact confusion this switch was supposed to end.
So: catch :class:`OfflineEgressBlocked` specifically, call this, and stop.
The first refusal for a given purpose is logged at WARNING with the switch
named; later ones drop to DEBUG so a 30-second poller does not flood the
log with a decision the operator already made.
"""
message = str(blocked)
if blocked.purpose in _REPORTED_REFUSALS:
log.debug("%s", message)
else:
_REPORTED_REFUSALS.add(blocked.purpose)
log.warning("%s", message)
return message
+87
View File
@@ -8,6 +8,8 @@ import os
import sys
from typing import Any
from headroom.offline import guard_egress
logger = logging.getLogger(__name__)
# Override for the CPU memory-arena default below: "1"/"true" forces the
@@ -121,6 +123,10 @@ def hf_hub_download_local_first(
Absolute path to the local cached file.
Raises:
OfflineEgressBlocked: when ``HEADROOM_OFFLINE`` is set and the file is
not already cached. A cache HIT still succeeds — the guard sits on
the network fallback only, so a pre-seeded air-gapped deployment
keeps working, which is the whole point of pre-seeding.
Any exception raised by ``hf_hub_download`` on a genuine download failure,
or the local-lookup error when ``allow_network`` is ``False`` and the
file is not cached.
@@ -135,6 +141,17 @@ def hf_hub_download_local_first(
except (LocalEntryNotFoundError, EntryNotFoundError, OSError):
if not allow_network:
raise
# Air-gap chokepoint: this is the Python half of the HuggingFace fetch
# the Rust core already guards. ``apply_offline_env`` sets
# ``HF_HUB_OFFLINE=1``, but only with ``setdefault`` (an explicit
# ``HF_HUB_OFFLINE=0`` wins) and only inside the proxy process — the CLI
# and library entry points never call it, so the flag alone is not the
# guarantee. Guarding here is, and it costs nothing on the cache-hit
# path above.
guard_egress(
f"HuggingFace download of {repo_id}/{filename}",
"huggingface.co",
)
return str(hf_hub_download(repo_id, filename, revision=revision))
@@ -157,6 +174,76 @@ def hf_entry_known_absent(repo_id: str, filename: str, *, revision: str | None =
return result is _CACHED_NO_EXIST
def _hf_not_cached_errors() -> tuple[type[BaseException], ...]:
"""Errors a ``local_files_only=True`` load raises on a plain cache miss.
Transformers and sentence-transformers both surface a miss as ``OSError``;
huggingface_hub's own lookup errors are listed explicitly in case a loader
lets one through unwrapped.
"""
try:
from huggingface_hub.errors import EntryNotFoundError, LocalEntryNotFoundError
except Exception: # pragma: no cover - huggingface_hub ships with every HF loader
return (OSError,)
return (LocalEntryNotFoundError, EntryNotFoundError, OSError)
def hf_from_pretrained_local_first(
loader: Any,
name_or_path: str,
*,
purpose: str,
allow_network: bool = True,
**kwargs: Any,
) -> Any:
"""Load a HuggingFace model/tokenizer/processor, local cache first.
The ``from_pretrained`` twin of :func:`hf_hub_download_local_first`, for
every loader that takes ``local_files_only`` — ``AutoModel.from_pretrained``,
``AutoTokenizer.from_pretrained``, ``AutoProcessor.from_pretrained``,
``SentenceTransformer`` and friends. ``loader`` is that callable, passed
uncalled; ``kwargs`` are forwarded to both attempts.
1. ``local_files_only=True``: a pure cache lookup that cannot open a socket.
A warm (or pre-seeded air-gapped) cache loads here and never touches the
network, which also skips the Hub re-validation round-trips a plain
``from_pretrained`` makes on every load.
2. On a cache miss, :func:`guard_egress` runs BEFORE the remote attempt.
Why the guard and not ``HF_HUB_OFFLINE``: ``apply_offline_env`` sets it with
``setdefault``, so an explicit ``HF_HUB_OFFLINE=0`` wins, and only the proxy
process calls it at all. Under that configuration huggingface_hub's offline
constant is false even with ``TRANSFORMERS_OFFLINE=1``, and a bare
``from_pretrained`` downloads. ``HEADROOM_OFFLINE`` is the master switch;
this guard is what makes it one for model loaders.
Args:
loader: The ``from_pretrained``-style callable.
name_or_path: Hub repo id or local directory.
purpose: What is being loaded, for the refusal message ("SigLIP model").
allow_network: When ``False`` a cache miss re-raises the local-lookup
error instead of falling back to a download.
**kwargs: Forwarded to ``loader`` (``revision=``, ``device=``, ...).
Raises:
OfflineEgressBlocked: ``HEADROOM_OFFLINE`` is set and the model is not
cached. Callers translate this into their own "model unavailable".
The loader's own error on a cache miss when ``allow_network`` is
``False``, when ``name_or_path`` is a local directory (nothing remote
to fall back to), or when the remote attempt itself fails.
"""
kwargs.pop("local_files_only", None)
try:
return loader(name_or_path, local_files_only=True, **kwargs)
except _hf_not_cached_errors():
# A local directory that failed to load has no remote to fall back to;
# surface the real error rather than a download attempt or a refusal.
if not allow_network or os.path.isdir(name_or_path):
raise
guard_egress(f"HuggingFace download of {purpose} ({name_or_path})", "huggingface.co")
return loader(name_or_path, local_files_only=False, **kwargs)
def create_cpu_session_options(
ort: Any,
*,
+109 -9
View File
@@ -97,7 +97,13 @@ from headroom.observability import (
shutdown_headroom_tracing,
shutdown_otel_metrics,
)
from headroom.offline import apply_offline_env, is_offline
from headroom.offline import (
OFFLINE_ENV,
OfflineEgressBlocked,
apply_offline_env,
guard_egress,
is_offline,
)
from headroom.pipeline import PipelineExtensionManager, PipelineStage
from headroom.providers.proxy_routes import register_provider_routes
from headroom.providers.registry import (
@@ -698,6 +704,89 @@ def _check_rust_core() -> tuple[str, str | None]:
return ("loaded", None)
def _refuse_offline_egress(blocked: OfflineEgressBlocked, feature: str, fix: str) -> None:
"""Turn a startup-time air-gap refusal into a clean exit, never a traceback.
``OfflineEgressBlocked`` derives from ``BaseException``, so nothing in the
lifespan's own ``except Exception`` will catch it and it would otherwise
escape uvicorn as an unhandled error — killing a proxy that had been
serving traffic, with a stack trace instead of an explanation, and before
``app.state.startup_error`` exists to record why. ``_check_rust_core``
above already settled the shape for a startup refusal: say what is wrong,
say how to fix it, exit 78 (``EX_CONFIG``).
"""
msg = (
f"FATAL: {feature} needs outbound network access, but "
f"{OFFLINE_ENV}=1 forbids it.\n"
f" refused: {blocked}\n"
f" fix: {fix}\n"
)
logger.error(
"event=offline_egress_refused feature=%r reason=%r action=exit_78",
feature,
str(blocked),
)
print(msg, file=sys.stderr, flush=True)
sys.exit(_EXIT_CONFIG)
def _configure_observability_or_refuse() -> None:
"""Install the metric/trace exporters, refusing cleanly under an air-gap.
Both exporters call ``guard_egress`` before they build a client, so under
``HEADROOM_OFFLINE`` they raise rather than dial. Catching that here is what
keeps the refusal a refusal: these two calls sit OUTSIDE ``lifespan``'s own
``try``/``except Exception``, and ``OfflineEgressBlocked`` is a
``BaseException``, so an uncaught one escapes uvicorn as an unhandled error
— a proxy that had been serving traffic dies on a traceback with
``app.state.startup_error`` never set.
"""
_preflight_offline_egress()
try:
configure_otel_metrics(OTelMetricsConfig.from_env(default_service_name="headroom-proxy"))
except OfflineEgressBlocked as blocked:
_refuse_offline_egress(
blocked,
"OTLP metric export (HEADROOM_OTEL_METRICS_ENABLED)",
"set HEADROOM_OTEL_METRICS_EXPORTER=console or scrape /metrics "
"(both stay on-box), or unset HEADROOM_OTEL_METRICS_ENABLED",
)
try:
configure_langfuse_tracing(
LangfuseTracingConfig.from_env(default_service_name="headroom-proxy")
)
except OfflineEgressBlocked as blocked:
_refuse_offline_egress(
blocked,
"Langfuse OTLP trace export (HEADROOM_LANGFUSE_ENABLED)",
"unset HEADROOM_LANGFUSE_ENABLED; Langfuse ingestion has no on-box mode",
)
def _preflight_offline_egress() -> None:
"""Refuse, at startup, any configuration that asks for egress while offline.
Runs before the observability exporters are configured and before the
compressors are preloaded. Without it the same contradiction surfaces later
and worse: the OTLP exporter raised out of ``lifespan`` with a raw
traceback, and the remote Kompress endpoint was not noticed until a request
reached for the compressor.
"""
if not is_offline():
return
endpoint = os.environ.get("HEADROOM_KOMPRESS_ENDPOINT", "").strip()
if endpoint:
try:
guard_egress("remote Kompress inference", endpoint)
except OfflineEgressBlocked as blocked:
_refuse_offline_egress(
blocked,
"remote Kompress compression (HEADROOM_KOMPRESS_ENDPOINT)",
"unset HEADROOM_KOMPRESS_ENDPOINT to use the local Kompress "
"model, or unset " + OFFLINE_ENV + " if this box is not air-gapped",
)
# Compression pipeline timeout in seconds
@@ -3104,15 +3193,29 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
# Air-gap master switch. Propagate config.offline to the env so the
# env-based egress predicates (telemetry, update check, license) all honor
# it, force HF/transformers offline before any model code loads, and
# announce that every outbound path is disabled.
# it, force HF/transformers offline before any model code loads, and say
# what is covered.
#
# The banner names the exception rather than claiming a whole-process kill
# switch, and the exception is permanent rather than a to-do: a proxy that
# refused to forward the caller's request to the caller's own upstream
# would not be a proxy. Every connection Headroom itself initiates IS
# refused; tests/test_offline_egress_chokepoint.py fails the build if a new
# egress path appears that is neither guarded nor one of the four written
# exceptions. TestDocsMatchTheGuarantee scans this file for the absolute
# phrasings and fails while any remain.
if config.offline:
os.environ.setdefault("HEADROOM_OFFLINE", "1")
if is_offline():
apply_offline_env()
logger.warning(
"event=proxy_offline_mode air-gap active — all outbound egress disabled "
"(telemetry, update check, license reporter, HuggingFace downloads)"
"event=proxy_offline_mode air-gap active — every connection Headroom "
"initiates is refused (telemetry, update check, license reporter, "
"model/tokenizer/binary/dataset downloads, remote Kompress, "
"OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI "
"embedders, Headroom Cloud compression). Still allowed on purpose: "
"forwarding your requests to the upstream you configured, "
"operator-configured local endpoints, loopback health probes."
)
proxy = HeadroomProxy(config)
@@ -3231,10 +3334,7 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
_rust_core_error,
)
configure_otel_metrics(OTelMetricsConfig.from_env(default_service_name="headroom-proxy"))
configure_langfuse_tracing(
LangfuseTracingConfig.from_env(default_service_name="headroom-proxy")
)
_configure_observability_or_refuse()
app.state.started_at = time.time()
app.state.ready = False
+16
View File
@@ -33,6 +33,8 @@ from dataclasses import asdict, dataclass, field
from typing import Any
from urllib.parse import urlsplit
from headroom.offline import OfflineEgressBlocked, guard_egress, note_refusal
logger = logging.getLogger("headroom.proxy")
DOCS_URL = "https://docs.headroomlabs.ai/docs/corporate-networks"
@@ -302,6 +304,14 @@ def probe_presented_chain(
def _probe_uncached(host: str, port: int, *, timeout: float, allow_private: bool) -> ChainInfo:
proxy = _proxy_for(host)
info = ChainInfo(host=host, port=port, reachable=False, via_proxy=proxy)
# A second connection Headroom opens on its own, so HEADROOM_OFFLINE refuses
# it. It only enriches a diagnostic: on the request path the refusal becomes
# "no chain to report", and the TLS explanation is still given without it.
try:
guard_egress("TLS certificate-chain probe", f"{host}:{port}")
except OfflineEgressBlocked as blocked:
info.error = f"skipped: {note_refusal(blocked, logger)}"
return info
# Certificate-inspection handshake only: it reads the presented chain and
# closes without sending a byte, so it deliberately skips verification (the
# chain is what we are diagnosing). It still refuses TLS < 1.2, and it only
@@ -459,7 +469,13 @@ def probe_endpoint(name: str, url: str, *, timeout: float = 8.0) -> EndpointRepo
Reports the presented chain (and any inspection vendor), whether the chain
verifies under the policy Headroom will use, and whether the response is a
gateway block page instead of the service.
Raises :class:`OfflineEgressBlocked` under ``HEADROOM_OFFLINE`` before
either connection is attempted; ``headroom doctor`` reports that as a
skipped check.
"""
guard_egress("doctor --network endpoint check", url)
import httpx
from headroom.proxy.ssl_context import build_httpx_verify
+68 -5
View File
@@ -30,6 +30,8 @@ import logging
import os
from typing import TYPE_CHECKING
from headroom.offline import OfflineEgressBlocked, guard_egress
from .base import RelevanceScore, RelevanceScorer
# numpy is an optional dependency - import lazily
@@ -57,6 +59,57 @@ if TYPE_CHECKING:
logger = logging.getLogger(__name__)
def _load_text_embedding(kwargs: dict[str, str]) -> TextEmbedding:
"""Build a fastembed ``TextEmbedding``, cache first and network second.
``onnx_runtime.hf_hub_download_local_first`` can hang the air-gap guard on
an explicit ``local_files_only=True`` attempt because ``hf_hub_download``
takes that flag. fastembed's constructor does not, so the same cache-hit /
network-fallback split has to be made here: try the load with the whole
HuggingFace stack forced offline, and only reach for the guard when that
fails, which is exactly the case where a socket would have been opened.
This keeps the behaviour an air-gapped deployment actually wants — a
pre-seeded cache still loads under ``HEADROOM_OFFLINE`` — while making a
cold cache refuse instead of quietly dialling huggingface.co. A plain
unconditional guard would have broken the pre-seeded case, which is the
reason this path was left unguarded before.
The ``HF_HUB_OFFLINE`` flip is process-global for the duration of the
call. That is acceptable here because loading is one-shot and memoised by
the caller, and because the value it forces is *stricter* than whatever it
replaces, so a concurrent HuggingFace call can only be refused locally,
never sent somewhere it would not otherwise go.
"""
from fastembed import TextEmbedding
from huggingface_hub import constants as hf_constants
# huggingface_hub reads HF_HUB_OFFLINE once, at import, into
# ``constants.HF_HUB_OFFLINE``; if it was imported before this call the
# env var alone changes nothing. Force both.
previous = os.environ.get("HF_HUB_OFFLINE")
previous_constant = hf_constants.HF_HUB_OFFLINE
os.environ["HF_HUB_OFFLINE"] = "1"
hf_constants.HF_HUB_OFFLINE = True
try:
return TextEmbedding(**kwargs)
except Exception as cache_miss: # noqa: BLE001 - any local-lookup failure
logger.debug("fastembed cache lookup failed, falling back to network: %s", cache_miss)
finally:
hf_constants.HF_HUB_OFFLINE = previous_constant
if previous is None:
os.environ.pop("HF_HUB_OFFLINE", None)
else:
os.environ["HF_HUB_OFFLINE"] = previous
guard_egress(
f"fastembed weight download for {kwargs['model_name']}",
"huggingface.co",
)
return TextEmbedding(**kwargs)
# Default model name. Same string used by the Rust embedding scorer.
DEFAULT_MODEL_NAME = "BAAI/bge-small-en-v1.5"
@@ -169,14 +222,24 @@ class EmbeddingScorer(RelevanceScorer):
)
if self._model is None:
from fastembed import TextEmbedding
revision = _pinned_revision(self.model_name)
kwargs = {"model_name": self.model_name}
if revision is not None:
# fastembed forwards **kwargs to snapshot_download(revision=...).
self._model = TextEmbedding(model_name=self.model_name, revision=revision)
else:
self._model = TextEmbedding(model_name=self.model_name)
kwargs["revision"] = revision
try:
self._model = _load_text_embedding(kwargs)
except OfflineEgressBlocked as blocked:
# Translate at the boundary that owns the degradation, the way
# the Kompress and ONNX loaders do. Public model weights are
# not data leaving the box, so what the caller needs to hear
# is "this model is not available here and why", not an
# air-gap type it has never seen.
raise RuntimeError(
f"Embedding model {self.model_name!r} is unavailable: {blocked}. "
"Pre-seed the fastembed/HuggingFace cache on this host, or "
"use the BM25-only scorer."
) from None
return self._model
def _encode(self, texts: list[str]):
+13
View File
@@ -21,6 +21,7 @@ from typing import Any
import httpx
from headroom.offline import OfflineEgressBlocked, guard_egress, note_refusal
from headroom.subscription.models import SubscriptionSnapshot
logger = logging.getLogger(__name__)
@@ -106,6 +107,10 @@ class SubscriptionClient:
}
try:
# First statement in the block that opens the socket, so the guard
# provably runs before the client is built (which is also what the
# egress meta-test's dominance rule checks).
guard_egress("Anthropic subscription usage polling", _USAGE_URL)
async with httpx.AsyncClient(timeout=self._timeout) as client:
resp = await client.get(_USAGE_URL, headers=headers)
@@ -123,6 +128,14 @@ class SubscriptionClient:
data: dict[str, Any] = resp.json()
return SubscriptionSnapshot.from_api_response(data, token=resolved)
except OfflineEgressBlocked as blocked:
# Refuse, but as a sentence in the log rather than an unhandled
# BaseException inside a background poll. The caller reads `None`
# as "no window data", which is the truth here: an air-gapped box
# cannot know the subscription window, and pretending otherwise
# would be worse than saying so once and moving on.
note_refusal(blocked, logger)
return None
except httpx.TimeoutException:
logger.debug("Timeout fetching Anthropic subscription window")
return None
@@ -50,6 +50,7 @@ from threading import Lock
import httpx
from headroom.offline import OfflineEgressBlocked, guard_egress, is_offline, note_refusal
from headroom.subscription.base import QuotaTracker
logger = logging.getLogger(__name__)
@@ -450,6 +451,7 @@ def _build_usage_headers(request_headers: dict[str, str]) -> dict[str, str] | No
async def _fetch_and_store_usage(url: str, headers: dict[str, str]) -> None:
state = get_codex_rate_limit_state()
try:
guard_egress("Codex rate-limit polling", url)
async with httpx.AsyncClient(timeout=_USAGE_POLL_TIMEOUT_S) as client:
resp = await client.get(url, headers=headers)
if resp.status_code == 200:
@@ -459,6 +461,12 @@ async def _fetch_and_store_usage(url: str, headers: dict[str, str]) -> None:
logger.debug("codex usage poll: 200 but no usable rate-limit data")
else:
logger.debug("codex usage poll: HTTP %s", resp.status_code)
except OfflineEgressBlocked as blocked:
# Background task: report the refusal as a sentence. Letting a
# BaseException out of a bare ``create_task`` surfaces as "Task
# exception was never retrieved" whenever the GC gets round to it,
# which is a traceback with no explanation in it.
note_refusal(blocked, logger)
except Exception as exc: # pragma: no cover - network/JSON defensive
logger.debug("codex usage poll failed: %s", exc)
finally:
@@ -488,6 +496,12 @@ def maybe_schedule_usage_poll(
headers = _build_usage_headers(request_headers)
if headers is None:
return False
if is_offline():
# Cheap pre-check so an air-gapped proxy does not spawn (and log) a
# doomed task on every single Codex request. The guard inside
# _fetch_and_store_usage is still the thing that makes the refusal
# true; this only keeps the log and the event loop quiet.
return False
try:
loop = asyncio.get_running_loop()
except RuntimeError:
+9
View File
@@ -41,6 +41,7 @@ from dataclasses import dataclass, field
from threading import Lock
from typing import Any
from headroom.offline import OfflineEgressBlocked, guard_egress, note_refusal
from headroom.subscription.base import QuotaTracker
logger = logging.getLogger(__name__)
@@ -314,6 +315,7 @@ class _CopilotQuotaTracker(QuotaTracker):
}
try:
guard_egress("GitHub Copilot quota polling", _GITHUB_API_BASE)
async with aiohttp.ClientSession() as session:
async with session.get(
url, headers=headers, timeout=aiohttp.ClientTimeout(total=10)
@@ -333,6 +335,13 @@ class _CopilotQuotaTracker(QuotaTracker):
return
data = await resp.json()
except OfflineEgressBlocked as blocked:
# Say it twice, in the two places an operator looks: once in the
# log, and once in the quota panel's own error slot, so the empty
# panel carries its reason instead of looking like a broken poll.
with self._lock:
self._state.last_error = note_refusal(blocked, logger)
return
except Exception as exc:
with self._lock:
self._state.last_error = str(exc)
+5 -1
View File
@@ -226,7 +226,7 @@ def _load_tokenizer(tokenizer_name: str):
# vetted, not a variant the caller chose.
tokenizer_name = repo
from headroom.offline import apply_offline_env, is_offline
from headroom.offline import apply_offline_env, guard_egress, is_offline
offline = is_offline()
if offline:
@@ -268,6 +268,10 @@ def _load_tokenizer(tokenizer_name: str):
def _download() -> None:
try:
# Defence in depth behind the is_offline() return above: the switch
# could flip between that check and this thread, and the guard is
# what the egress meta-test recognises as dominating the fetch.
guard_egress(f"HuggingFace tokenizer download ({tokenizer_name})", "huggingface.co")
result.append(
AutoTokenizer.from_pretrained(
tokenizer_name,
+5 -2
View File
@@ -63,6 +63,7 @@ from ..config import (
is_tool_excluded,
unwrap_tool_call,
)
from ..offline import OfflineEgressBlocked
from ..parser import CCR_RETRIEVAL_MARKER_RE
from ..tokenizer import Tokenizer
from ..tokenizers.base import count_content_blocks
@@ -1185,7 +1186,7 @@ def _detect_content(content: str) -> DetectionResult:
# "json_array"); translate to the Python `ContentType` enum so
# downstream mapping keys match.
content_type = ContentType(rust_result.content_type)
except (KeyboardInterrupt, SystemExit, GeneratorExit):
except (KeyboardInterrupt, SystemExit, GeneratorExit, OfflineEgressBlocked):
raise
except BaseException as exc: # noqa: BLE001
# A native Rust panic surfaces as pyo3_runtime.PanicException, which
@@ -1194,7 +1195,9 @@ def _detect_content(content: str) -> DetectionResult:
# (panic, or an unrecognized content-type tag) degrades to the
# pure-Python detector instead of aborting the request. See #1123.
# Guard: don't swallow cancellation/control-flow BaseExceptions such
# as asyncio.CancelledError — keep them propagating.
# as asyncio.CancelledError — keep them propagating. OfflineEgressBlocked
# is in that list for the same reason: it is a policy refusal, not a
# detector failure, and degrading it here would hide the air-gap switch.
if isinstance(exc, asyncio.CancelledError):
raise
if isinstance(exc, TimeoutError):
+122 -21
View File
@@ -28,11 +28,13 @@ from functools import lru_cache
from typing import Any, Literal
from ..config import TransformResult
from ..offline import OFFLINE_ENV, OfflineEgressBlocked
from ..onnx_runtime import (
ONNX_CPU_ARENA_ENV,
_resolve_revision,
create_cpu_session_options,
hf_entry_known_absent,
hf_from_pretrained_local_first,
hf_hub_download_local_first,
trim_process_heap,
)
@@ -220,6 +222,70 @@ class KompressModelNotCached(RuntimeError):
"""
def _hf_artifact(model_id: str, filename: str, *, allow_network: bool) -> str:
"""``hf_hub_download_local_first`` with the air-gap refusal translated.
``OfflineEgressBlocked`` is a ``BaseException`` so that no ``except
Exception:`` can silently degrade an air-gap refusal into "that feature
stopped working". At THIS boundary that degradation is the correct
behaviour and is written down rather than inherited: a model download is
not data leaving the box, every caller below already handles
"the model is not available locally", and failing a user's request because
an optional compressor could not fetch public weights would punish the
request for a decision the operator made about the host.
So the refusal is logged with the switch named and re-raised as
:class:`KompressModelNotCached`, which is exactly what it means: under
``HEADROOM_OFFLINE`` the only artifacts that will ever be available are the
cached ones. The original refusal is chained, so it is still in the
traceback.
"""
try:
return hf_hub_download_local_first(model_id, filename, allow_network=allow_network)
except OfflineEgressBlocked as blocked:
logger.warning(
"Kompress: %r for %s is not cached and %s forbids fetching it (%s). "
"Compression falls back to the non-ML path.",
filename,
model_id,
OFFLINE_ENV,
blocked,
)
raise KompressModelNotCached(model_id) from blocked
def _hf_pretrained(
loader: Any, name: str, *, purpose: str, allow_download: bool, **kwargs: Any
) -> Any:
"""``hf_from_pretrained_local_first`` with Kompress's two translations.
The ``from_pretrained`` counterpart of :func:`_hf_artifact`: cache first,
``guard_egress`` before any remote attempt, and both "not cached while
downloads are disallowed" and "not cached while ``HEADROOM_OFFLINE`` is set"
surface as :class:`KompressModelNotCached` — the error every Kompress caller
already treats as "use the non-ML path". A genuine remote failure while
downloads are allowed propagates unchanged.
"""
try:
return hf_from_pretrained_local_first(
loader, name, purpose=purpose, allow_network=allow_download, **kwargs
)
except OfflineEgressBlocked as blocked:
logger.warning(
"Kompress: the %s (%s) is not cached and %s forbids fetching it (%s). "
"Compression falls back to the non-ML path.",
purpose,
name,
OFFLINE_ENV,
blocked,
)
raise KompressModelNotCached(name) from blocked
except _NOT_CACHED_ERRORS as exc:
if allow_download:
raise
raise KompressModelNotCached(name) from exc
# Model cache: model_id -> (model, tokenizer, backend)
# Supports multiple models loaded simultaneously.
_kompress_cache: dict[str, tuple[Any, Any, str]] = {}
@@ -611,11 +677,14 @@ def _get_model_class() -> type:
super().__init__()
# Same pin as the tokenizer: the fine-tuned heads were trained on
# this encoder snapshot. Unknown repos resolve to None (floating).
self.encoder = AutoModel.from_pretrained(
# Cache first, air-gap guard before any download (see _hf_pretrained).
self.encoder = _hf_pretrained(
AutoModel.from_pretrained,
model_name,
purpose="Kompress PyTorch encoder",
allow_download=allow_download,
attn_implementation="eager",
revision=_resolve_revision(model_name, revision),
local_files_only=not allow_download,
)
hidden_size = self.encoder.config.hidden_size # 768
@@ -750,12 +819,19 @@ def _create_onnx_session(
"""
last_err: Exception | None = None
cache_miss = False
refused: KompressModelNotCached | None = None
ort: Any = None
for filename in _onnx_filename_candidates():
try:
onnx_path = hf_hub_download_local_first(
model_id, filename, allow_network=allow_download
onnx_path = _hf_artifact(
model_id, filename, allow_network=allow_download and refused is None
)
except KompressModelNotCached as exc:
# Only _hf_artifact's air-gap translation raises this here. It says
# THIS candidate is not cached; a later one may be. Keep looking,
# cache-only, so the refusal is logged once and never retried.
refused = exc
continue
except Exception as exc:
last_err = exc
cache_miss = cache_miss or isinstance(exc, _NOT_CACHED_ERRORS)
@@ -781,6 +857,8 @@ def _create_onnx_session(
model_id,
exc,
)
if refused is not None:
raise refused
if not allow_download and cache_miss:
raise KompressModelNotCached(model_id) from last_err
raise FileNotFoundError(
@@ -860,8 +938,11 @@ def _load_modernbert_tokenizer(auto_tokenizer: Any, *, allow_download: bool) ->
Same files, same tokenizer, so the loaded object is identical; this only
changes whether the Hub is consulted to confirm what is already on disk.
Mirrors ``onnx_runtime.hf_hub_download_local_first``, which the ONNX half of
this loader already uses.
Goes through ``onnx_runtime.hf_from_pretrained_local_first`` (via
:func:`_hf_pretrained`), the ``from_pretrained`` twin of the
``hf_hub_download_local_first`` the ONNX half of this loader uses, so a cache
miss under ``HEADROOM_OFFLINE`` is refused before any remote attempt even
when ``HF_HUB_OFFLINE=0`` is set explicitly.
The load is pinned to the same immutable revision as the model artifacts
(``onnx_runtime._PINNED_REVISIONS``): the ONNX export was produced against
@@ -869,17 +950,12 @@ def _load_modernbert_tokenizer(auto_tokenizer: Any, *, allow_download: bool) ->
the shipped weights without any code change. ``HEADROOM_HF_PIN=off`` floats
it, exactly as for the weights.
"""
revision = _resolve_revision(_MODERNBERT_TOKENIZER_REPO, None)
try:
return auto_tokenizer.from_pretrained(
_MODERNBERT_TOKENIZER_REPO, revision=revision, local_files_only=True
)
except _NOT_CACHED_ERRORS as exc:
if not allow_download:
raise KompressModelNotCached(_MODERNBERT_TOKENIZER_REPO) from exc
# Genuine cache miss and downloading is permitted: fetch it at the pin.
return auto_tokenizer.from_pretrained(
_MODERNBERT_TOKENIZER_REPO, revision=revision, local_files_only=False
return _hf_pretrained(
auto_tokenizer.from_pretrained,
_MODERNBERT_TOKENIZER_REPO,
purpose="ModernBERT tokenizer",
allow_download=allow_download,
revision=_resolve_revision(_MODERNBERT_TOKENIZER_REPO, None),
)
@@ -954,7 +1030,7 @@ def _load_pytorch_weights(model: Any, model_id: str, *, allow_download: bool) ->
HuggingFace Hub's own cache of confirmed-404 lookups.
"""
try:
ckpt_path = hf_hub_download_local_first(model_id, "merged.pt", allow_network=allow_download)
ckpt_path = _hf_artifact(model_id, "merged.pt", allow_network=allow_download)
except _NOT_CACHED_ERRORS as exc:
if not allow_download:
if not hf_entry_known_absent(model_id, "merged.pt"):
@@ -971,9 +1047,7 @@ def _load_pytorch_weights(model: Any, model_id: str, *, allow_download: bool) ->
# merged.pt genuinely does not exist in this repo (confirmed by a
# real network lookup, not just a cache miss) - fall back to the
# plain format instead of treating it as a download failure.
weights_path = hf_hub_download_local_first(
model_id, "model.safetensors", allow_network=allow_download
)
weights_path = _hf_artifact(model_id, "model.safetensors", allow_network=allow_download)
_load_plain_state_dict(model, weights_path, model_id)
return
raise
@@ -1218,6 +1292,21 @@ def _background_download(model_id: str, device: str) -> None:
logger.info("Kompress: downloading model %s in the background ...", model_id)
_load_kompress(model_id, device, allow_download=True)
logger.info("Kompress: background model download complete for %s", model_id)
except OfflineEgressBlocked as blocked:
# Explicit, because OfflineEgressBlocked is a BaseException and would
# otherwise reach threading.excepthook as a bare traceback on every
# air-gapped startup with a cold cache. This is a background *refresh*,
# not the request path: the refusal is reported once, at WARNING, and
# the compressor keeps whatever is already cached. Do NOT widen this to
# `except Exception` — that is the shape the guard exists to defeat.
_record_download_failure(model_id)
logger.warning(
"Kompress: background model download refused for %s (%s is set): %s. "
"The model will only load if its artifacts are already cached.",
model_id,
OFFLINE_ENV,
blocked,
)
except Exception as exc:
_record_download_failure(model_id)
logger.warning("Kompress: background model download failed for %s: %s", model_id, exc)
@@ -1277,6 +1366,18 @@ def prefetch_kompress_artifacts(model_id: str = HF_MODEL_ID) -> bool:
try:
hf_hub_download_local_first(model_id, filename, allow_network=True)
return True
except OfflineEgressBlocked as blocked:
# Same reasoning as _background_download: explicit because it is a
# BaseException, reported rather than swallowed, and terminal for
# the whole loop — every candidate would be refused for the same
# reason, so retrying them just logs the same refusal four times.
logger.warning(
"Kompress: artifact prefetch refused for %s (%s is set): %s",
model_id,
OFFLINE_ENV,
blocked,
)
return False
except Exception as exc:
logger.debug("Kompress prefetch: %r unavailable for %s: %s", filename, model_id, exc)
return False
+18
View File
@@ -57,6 +57,7 @@ import logging
import httpx
from ..offline import guard_egress
from .kompress_compressor import (
KompressConfig,
KompressResult,
@@ -139,6 +140,13 @@ class RemoteKompressCompressor:
# without needing a separate auth-scheme setting.
if headers:
self._headers.update(headers)
# Air-gap chokepoint. HEADROOM_KOMPRESS_ENDPOINT points at a box that is
# by definition off this host, so under HEADROOM_OFFLINE this class must
# not exist at all. Guarding at construction (not at the POST) means the
# connection pool is never created and the operator hears about the
# contradictory configuration the first time the router reaches for
# Kompress, rather than discovering it in a packet capture.
guard_egress("remote Kompress inference", self._url)
# httpx.Client is safe to share across the proxy's worker threads.
self._client = httpx.Client(timeout=timeout)
@@ -212,6 +220,16 @@ class RemoteKompressCompressor:
if n_words < max(_MIN_WORDS, self.config.min_input_words):
return self._passthrough(content, n_words)
# Re-check outside the fail-open try. __init__ already refused if the
# flag was set then, but the env can be flipped after this object was
# cached on the ContentRouter (the router builds it once per instance
# and reuses it for the process lifetime), and the security property we
# owe is "no socket", not "no construction". It is outside the try on
# purpose: the except below turns every exception into a silent
# pass-through, and a policy refusal that degrades silently is exactly
# the hole this guard closes.
guard_egress("remote Kompress inference", self._url)
try:
resp = self._client.post(
self._url,
File diff suppressed because it is too large Load Diff