mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(offline): make HEADROOM_OFFLINE a real air-gap via one chokepoint (#3729)
## Description **Revised 2026-10-05 after review: merged current `main`, routed main's new TLS-diagnostic egress (`doctor --network` and the TLS-error chain re-probe) through `guard_egress`, made the test socket trap let loopback through so the async offline tests run on Windows, and closed five model-loader findings from Devin Review.** `HEADROOM_OFFLINE=1` is documented as an air-gap switch, but several egress paths never consulted `headroom/offline.py`. This PR makes `guard_egress` the single chokepoint every Headroom-initiated outbound connection goes through, with a per-site meta-test that fails on any new unguarded egress. Full detail of every revision is under *Changes Made*. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [x] Breaking change (fix or feature that would cause existing functionality to change) — outbound calls that previously worked with `HEADROOM_OFFLINE=1` set now refuse; see *Behaviour change customers can see* - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made ### This revision (2026-10-05, after @JerrettDavis's review of 2026-09-24 and comment of 2026-10-03) * **Merged current `main`.** Conflicts in `docs/content/docs/proxy.mdx` (kept main's new rows, this PR's offline row), `headroom/image/onnx_router.py` (both imports) and `headroom/proxy/server.py` lifespan (main's rust-core passthrough block, then this PR's `_configure_observability_or_refuse()`). * **`doctor --network` / TLS diagnostics now go through the guard.** `tls_diagnostics.probe_endpoint` calls `guard_egress` before both the certificate-chain handshake and the `httpx.Client` GET; `headroom doctor` turns the refusal into one `network · skipped: HEADROOM_OFFLINE is set, no endpoint was contacted` row (status `skip`, so it does not fail the doctor) instead of a page of "unreachable". `_probe_uncached` — the chain handshake used both by doctor and by the request path's TLS-error re-probe (`describe_upstream_failure`) — guards before its socket; being diagnostic-only on the request path, it reports `skipped: …` and the TLS explanation is still produced without the chain. No allowlist widening: the inventory test passes because the `httpx.Client` site is now dominated by a guard. * **Socket trap is Windows-safe.** `no_sockets` now passes connects whose address is a loopback **IP literal** (the Windows Proactor self-pipe is a `127.0.0.1` listen + connect) and still fails every other address, including hostnames such as `localhost`. `TestSocketTrap` reproduces the Windows path on every platform by forcing `socket._fallback_socketpair` (the implementation Windows uses for `socket.socketpair()`) under `asyncio.run()`, and pins that outbound IPv4/IPv6 `connect` / `connect_ex` / `create_connection` are still caught. * **Devin Review findings (`40e2d449`).** * **Kompress tokenizer bypass.** Cached ONNX weights plus an uncached ModernBERT tokenizer fell through to an unguarded `from_pretrained(local_files_only=False)`. That call is now guarded and becomes `KompressModelNotCached`, the same translation `_hf_artifact` uses. * **Kompress ONNX candidates.** An uncached first candidate no longer ends the loop. The refusal is logged once, then the remaining candidates are checked cache-only, so a cached fp32 fallback still loads offline. * **fastembed.** `huggingface_hub` reads `HF_HUB_OFFLINE` once, at import, into `constants.HF_HUB_OFFLINE`. The cache-only attempt therefore forces that constant as well as the env var, and restores it afterwards. * **Image router.** It now resolves every artifact before publishing the session, which doubles as the "loaded" flag. A refused secondary artifact no longer leaves a half-loaded router that never retries. * **Eval runner.** The local Ollama provider is no longer refused, for the same operator-endpoint reason as the Ollama embedder. Anthropic and OpenAI are still refused. * **Sweep of main for other new egress since this PR was written:** the scanner's only new finding was `tls_diagnostics.py`. Manually checked the rest of the diff: the new `reqwest` client in `crates/headroom-proxy` and the keep-alive `httpx.AsyncClient` in `proxy/server.py` are upstream forwarding (`user-traffic`); the new HF tokenizer allowlist in `tokenizers/huggingface.py` sits behind the existing `is_offline()` gate (`gated`); `port_discovery.py`, `upstream_pinning.py`, `license_env.py` and `credential_policy.py` open no new external connections. ### A-2 (S2): the air-gap guarantee was not real `HEADROOM_OFFLINE=1` is documented as an air-gap switch, but several egress paths never consulted `headroom/offline.py`. A customer who set it and believed they were air-gapped was wrong. **This PR has been revised after an adversarial review found that the first version's guarantee was largely unenforced.** The review is reproduced in the fixes below; read the *What this does and does not promise* section for the honest scope. ### The chokepoint `headroom/offline.py` provides `guard_egress(purpose, destination)` and a named `OfflineEgressBlocked`. It refuses loudly — a silent skip is indistinguishable from success at the call site, which is how these holes survived. It is always called **before the HTTP client is constructed**, not merely before the request, so no pooled connection is ever opened. `crates/headroom-core/src/offline.rs` is the same switch, same env var, same rules, for the Rust core. #### Paths routed through it | path | previously | | --- | --- | | `headroom/transforms/kompress_remote.py` | built an `httpx.Client` and `POST`ed every compressible block to the remote Kompress endpoint | | `headroom/observability/metrics.py` | started an `OTLPMetricExporter` plus a background `PeriodicExportingMetricReader` timer | | `headroom/observability/tracing.py` | started an `OTLPSpanExporter` to Langfuse Cloud — the metric exporter's twin, missed the first time | | `headroom/onnx_runtime.py` | `hf_hub_download` on a cache miss: the **Python** half of the HF fetch, every ONNX model in the tree resolves through it | | `crates/…/tokenizer/hf_impl.rs` | fetched `tokenizer.json` from huggingface.co | | `crates/…/transforms/kompress.rs` | fetched the **Kompress model** from huggingface.co | | `crates/…/relevance/embedding.rs` | fastembed fetched ONNX weights from huggingface.co | The three Rust sites are the same shape and the same argument: `HF_HUB_OFFLINE` only shortcuts the *cache lookup*, and on a miss `hf-hub` still dials out. The first version of this PR guarded one of the three and left the other two open. Remote Kompress is guarded twice — in `__init__`, and again at the top of `compress()` **outside** the fail-open `try`, because the router caches one compressor for the process lifetime and `compress()` converts every exception into a silent passthrough. `onnx_runtime.hf_hub_download_local_first` is guarded on the **network fallback only**, so a pre-seeded air-gapped cache still resolves. The three Rust fetches refuse even on a warm cache: `hf-hub` 0.5 exposes no cache-hit/cache-miss split at that layer, so "guard before the client exists" and "serve from the cache" cannot both hold. `Kompress::from_files` / `HfTokenizer::from_file` are the network-free paths for a pre-seeded Rust deployment. Splitting the Rust fetches cache-first is the obvious follow-up and is **not** done here. ### The refusal now survives the fail-open handlers The original docstring said "anything that catches broad `Exception` around an egress call should re-raise this". **Nothing did** — there were zero re-raise sites in the package — and the refusal was swallowed on the way out of a request by `thinking_compactor._memo_compact` (wrapped directly around the guarded `kompress.compress` call), `ContentRouter._kompress_model_ready` (which reported a policy refusal as "model ready"), the startup preload, and the OpenAI reasoning-compaction path. A full `/v1/messages` request with `HEADROOM_OFFLINE=1` + `HEADROOM_KOMPRESS_ENDPOINT` returned **200 uncompressed** with `guard_egress` having fired twice. **`OfflineEgressBlocked` now derives from `BaseException`, not `Exception`.** A convention that has to be re-applied at every future `except Exception:` in a 100k-line codebase is not a guarantee; the type hierarchy is. Same reason `KeyboardInterrupt` and `SystemExit` sit outside `Exception`: a policy decision is control flow, not a runtime error to degrade around. All four sites propagate with no edit to any of them. ⚠️ **This is a behaviour change for callers.** `except Exception` no longer catches it anywhere, including in third-party code. `except BaseException` still does; those are swept by `TestBroadHandlerSweep`, which requires each to re-raise or carry a written reason. `content_router`'s native-detect degrade path now re-raises it explicitly, alongside the cancellation `BaseException`s it already let through. #### Where the refusal deliberately becomes a degradation Escaping `except Exception` is the point, but it made one class of caller worse. Fetching **public model weights** is not data leaving the box, and an air-gapped proxy with a cold HF cache would have started failing user requests where it previously degraded — punishing the request for a decision the operator made about the host. So each optional model loader translates the refusal into its own "model unavailable" error, **explicitly**, with the switch named in the log and the refusal chained: | boundary | translated to | the caller's existing degradation | | --- | --- | --- | | `kompress_compressor._hf_artifact` | `KompressModelNotCached` | non-ML compression path | | `image/onnx_router._hf_artifact` | `RuntimeError` | `ImageCompressor` → `Technique.PRESERVE` | | `memory/adapters/embedders` ONNX load | `RuntimeError` | local embedder unavailable | | the two Kompress download daemon threads | logged, `False`/return | keeps whatever is cached | Behaviour for a cold air-gapped cache is exactly what it was. What changed is that the log now names the switch, and the translation is a reviewed decision at a named boundary rather than something inherited from whichever `except Exception` happened to be in the stack. The daemon threads matter separately: an unhandled `BaseException` there reaches `threading.excepthook` as a bare traceback on every air-gapped boot. Paths that **send data off the box** — remote Kompress, OTLP, Langfuse, the beacon — are untouched and still refuse loudly. ### Startup refuses a contradictory configuration `configure_otel_metrics` was called **outside** `lifespan`'s `try`, above the line that sets `app.state.startup_error`, so an `OfflineEgressBlocked` escaped uvicorn as an unhandled error — killing a proxy that had been serving traffic, with a traceback instead of an explanation. It now runs through `_configure_observability_or_refuse()`, which turns a refusal into the shape `_check_rust_core` uses ten lines above: what is wrong, how to fix it, **exit 78** (`EX_CONFIG`). A preflight covers the remote-Kompress contradiction the same way. So the ordinary case is settled at startup with an actionable message, and `BaseException` is the backstop rather than the user experience. ``` FATAL: OTLP metric export (HEADROOM_OTEL_METRICS_ENABLED) needs outbound network access, but HEADROOM_OFFLINE=1 forbids it. refused: HEADROOM_OFFLINE is set: refusing outbound network access for OTLP metric export to … fix: set HEADROOM_OTEL_METRICS_EXPORTER=console or scrape /metrics (both stay on-box), or unset HEADROOM_OTEL_METRICS_ENABLED ``` ### Python/Rust normalisation parity Python trimmed with `str.strip()`, Rust with `str::trim()`. Not the same set: `str.isspace()` includes U+001C–U+001F, the Unicode `White_Space` property does not, so `HEADROOM_OFFLINE=$'\x1c1'` read as **offline to Python and online to Rust** — one variable, one half of the process air-gapped. Both sides now trim an enumerated `TRIM_CHARS` / `_TRIM_CHARS`, and the parity test compares the trim sets and a shared normalisation table, not just the accepted tokens. (Case folding needed no change: no non-ASCII character lowercases into any character of `1`/`true`/`yes`/`on`.) ### The meta-test, rebuilt The first version was neutered by one line: ```python if "guard_egress" in source: continue ``` A whole-file exemption keyed on the string appearing *anywhere*, including in a docstring — the exact blanket exemption the allowlist's own comment says must never happen. It made the site-count assertion unreachable for every guarded file, so "adding a second client trips this test" only held for files with **no** guard at all. Two bypasses were demonstrated against it: a second unguarded `httpx.Client` in an already-guarded file, and a brand-new unguarded module with the word in its docstring. It now decides **per egress site**: * **AST, not text.** Comments and docstrings vouch for nothing. `self.client.post()` is no longer read as `requests.post`, and the `httpx.post` example in `compress.py`'s docstring is not a site. * **Dominance.** A site is guarded only when a `guard_egress` call is in its block or an enclosing one, earlier in that block, in the same function. A guard in a sibling `if`/`except` arm, in a nested `def`, or after the site does not count. A guard in a helper the function calls does not count either — conservative on purpose. * **`TestSiteScannerRules`** tests the scanner itself, pinning both demonstrated bypasses plus the one-line `def f(): return httpx.Client().post(...)` that the old `startswith("def ")` skip made invisible. A meta-test nobody tests is a comment. * **Wider callee set:** module-level `httpx` verbs (`headroom/cli/mcp.py` was a live unmatched site), `hf_hub_download` (skipping `local_files_only=True`, which opens no socket), fastembed's `TextEmbedding`, the OTLP exporters, and the OpenAI/Anthropic SDK clients. * **`crates/` is swept too** (`TestRustEgressChokepointCoverage`). It was outside the scan entirely, which is how two Rust downloads stayed open. Text-level rather than AST — no Rust parser here — so "guarded" means a `guard_egress` earlier in the same `fn` at no deeper indent. Weaker, and labelled as such; the runtime assertions live in each crate's `#[test]`s. * **Callees are resolved through the module's import table before matching.** The scanner compared the text `ast.unparse` produced, which is whatever local name the module bound — so `import httpx as h` hid a client from a test whose entire job is to notice clients. That was not hypothetical: `copilot_auth.py` does `from urllib import request as urllib_request`, so its GitHub calls rendered as `urllib_request.urlopen` and **five call sites, including the shared `_urlopen` helper every Copilot request goes through, were invisible**. `_canonical_callee` rewrites the leading name first; the alias defence cuts both ways, so `guard_egress as ge` still counts as a guard. Imports are collected from the whole tree, not just module level, because half the egress here is imported inside the function that uses it. * Allowlist counts now mean **unguarded** sites only, so a file can legitimately guard some of its egress and still carry an entry for the rest. * **There is no longer a category for "known violation".** `test_no_category_permits_a_known_violation` pins the permitted set to `loopback` / `gated` / `user-traffic` / `operator-endpoint` / `cache-only` and rejects "out of scope" as a reason. ### The offline policy, in plain words `HEADROOM_OFFLINE=1` **refuses every connection Headroom itself decides to make to a destination Headroom itself chose.** That is the rule, not a sample list: telemetry, update check, license/usage reporter, OTLP and Langfuse export, HuggingFace / Kompress / fastembed model and tokenizer downloads (Python and Rust), release-binary and codebase-memory-mcp downloads, eval dataset downloads and the provider SDK clients the eval harness drives, GitHub Copilot device-flow auth and token exchange, the Anthropic / Codex / Copilot subscription pollers, both OpenAI embedders, both Headroom Cloud compression integrations, and the TLS diagnostics (`headroom doctor --network` endpoint checks and the certificate-chain re-probe after an upstream TLS failure). **Four things are deliberately still allowed, and they are the complete set of exceptions.** Every one of them is either physically incapable of leaving the box or is not Headroom-initiated traffic: | category | what it covers | why it survives the switch | | --- | --- | --- | | `user-traffic` | `proxy/server.py` (2 sites) | the caller's own request forwarded to the upstream the operator configured. A proxy that refuses this is not a proxy; an air-gapped install points it at an on-prem endpoint. | | `operator-endpoint` | `memory/adapters/embedders.py` (1 site) | the Ollama embedder, whose address comes entirely from operator config and defaults to `http://localhost:11434`. Exactly one thing qualifies today, and a hard-coded internet host may never hide here. | | `loopback` | 8 files, 12 sites | hard-coded `127.0.0.1` probes against the operator's own local proxy (`headroom doctor`, the installers, the MCP sidecar, the test harness). | | `gated` | 3 files, 3 sites | the beacon, the update check and the usage reporter are already unreachable behind an `is_offline()` check, so no connection is ever built. | Plus one mechanical exemption, `cache-only` (`relevance/embedding.py`, 1 site): the first of the two fastembed loads runs with `HF_HUB_OFFLINE` forced to `1`, so `huggingface_hub` resolves from the local cache or raises **without opening a socket**; the network retry directly below it calls `guard_egress`. That is what lets a pre-seeded air-gapped host keep loading the model while a cold one refuses. #### What changed since the review The previous revision of this PR recorded **nine egress-capable files** under an `unguarded` allowlist category meaning "reachable under `HEADROOM_OFFLINE`, out of scope for A-2": Copilot auth/token exchange, three subscription pollers, the two binary/CBM downloads, the eval-dataset fetch, and both Headroom Cloud compression integrations — plus the eval harness's LLM clients, both OpenAI embedders and the fastembed weights. @JerrettDavis blocked on exactly that, and he was right: recording that a switch does not do what it says is not the same as making it do it. **All of them are now guarded, and the `unguarded` category is deleted.** `test_no_category_permits_a_known_violation` pins the five permitted categories by name and rejects "out of scope" as an allowlist reason, so re-introducing a bless-it-and-move-on category takes a deliberate edit with a reviewer looking at it. #### ⚠️ Behaviour change customers can see **Outbound calls that previously worked with `HEADROOM_OFFLINE=1` set now refuse.** That is the point of the PR, but it is a real change for anyone who set the flag and kept using these features: * `headroom auth` / `headroom copilot-auth login` cannot complete a GitHub device flow, and a cached Copilot OAuth token cannot be exchanged for an API token. A Copilot subscription is unusable on an air-gapped box regardless, but the failure now arrives at the moment of the attempt. * `headroom install` will not download release binaries or codebase-memory-mcp. Pre-stage them, or install with the flag unset. * The Anthropic / Codex / Copilot subscription and quota panels stay empty, with the reason logged and (for Copilot quota) shown in the panel's own error slot. * Headroom Cloud compression mode in the ASGI middleware and the LiteLLM callback refuses instead of falling through to local compression — loudly, because silently compressing locally would hide that the deployment is no longer doing what it was configured to do. * The eval harness refuses to build provider clients or download BFCL. * The OpenAI embedders (memory adapter and direct mem0 backend) refuse. Use the ONNX or Ollama embedder on an air-gapped box. * `EmbeddingScorer` raises "model unavailable" on a **cold** fastembed cache. A warm one is unaffected. #### Refusals reach a human, never a traceback Following the pattern the earlier commits set for the Kompress download threads, the model layer and the startup banner: * **CLI:** `OfflineAwareGroup` (`headroom/cli/main.py`) is the one place that translates the refusal into a Click `Error:` line, for every subcommand at once. `OfflineEgressBlocked` stays a `BaseException` everywhere inside, so nothing else can downgrade it; only the last frame before the terminal is allowed to make it readable. * **Background pollers:** each catches `OfflineEgressBlocked` specifically, calls `offline.note_refusal` (WARNING once per purpose, DEBUG after, so a 30-second poller does not flood the log), and returns its ordinary "no data" value. An un-caught `BaseException` out of a bare `create_task` surfaces as "Task exception was never retrieved" whenever the GC gets to it — a traceback with no explanation in it. * **Model loaders:** `EmbeddingScorer._get_model` translates into a named "model unavailable" `RuntimeError`, the way `kompress_compressor._hf_artifact` and `image/onnx_router._hf_artifact` already do. ### 🚩 Two decisions that need a founder call Both were implemented the **safe** way — refuse, loudly and legibly — because that is the defensible default for a switch an operator sets to mean "nothing leaves this box". Neither is obviously right, and a permissive default should not be chosen quietly: 1. **Should OTLP / telemetry export be permitted under `HEADROOM_OFFLINE`?** Today it refuses, and a contradictory config (`HEADROOM_OTEL_METRICS_ENABLED=1` + offline) exits 78 at startup with an explanation. The argument for permitting it is that a regulated customer's OTLP collector is usually in-cluster, and losing observability is a real operational cost. The argument against — and the reason it is refused today — is that the guard **cannot reliably tell an in-cluster collector from an internet host**: DNS, service meshes and egress proxies all blur it, so any exemption is really "trust the endpoint the config names". A customer with an in-cluster collector currently has to scrape `localhost:8787/metrics` or use `HEADROOM_OTEL_METRICS_EXPORTER=console`. If we want to permit it, the honest shape is an explicit second flag (e.g. `HEADROOM_OFFLINE_ALLOW_OTLP=1`) rather than widening the policy. 2. **Should the subscription pollers refuse, or degrade silently?** Today all three refuse and say so once per purpose at WARNING. The argument for silence is that these are cosmetic panels and a WARNING on an air-gapped box is noise about a decision the operator already made. The argument for the current behaviour is that a usage panel that silently stops updating is exactly the "why is this empty?" confusion the switch was supposed to end, and once-per-purpose logging bounds the noise. Changing this is a one-line change in `offline.note_refusal`. ### Tests `tests/test_offline_egress_chokepoint.py` — 110 tests. Per-path socket assertions (a `no_sockets` fixture booby-traps `socket.create_connection`, `socket.socket.connect` and `connect_ex`, so asserting "no socket" rather than "raises" catches a guard placed after the client is built), the scanner-rule tests, the two coverage sweeps, the broad-handler sweep, the startup-refusal tests, the model-loader degradation tests, the cross-language parity tests, and the doc-claim test. Plus 8 Rust tests in `headroom-core`. ### Fail-before / pass-after Per fix, work committed, then sources reverted while keeping the new tests, run, restore. No `git stash`. **Meta-test (1).** Added a probe module whose docstring mentions `guard_egress` and a second unguarded `httpx.Client` to the already-guarded `kompress_remote.py`. The old meta-test: `3 passed`. The new one: ``` headroom/_a2_probe_tmp.py line 7: httpx.Client().get("https://evil.example.com") (not guarded, not allowlisted) headroom/transforms/kompress_remote.py line 154: return httpx.Client(timeout=1.0) (not guarded, not allowlisted) ``` **Rust guards (2).** Guard calls commented out, tests kept: ``` panicked at crates/headroom-core/src/transforms/kompress.rs:688: expected an offline refusal, got a network error: no loadable ONNX artifact in `headroomlabs-ai/this-model-does-not-exist-a2` (tried [...]): request error: http status: 401 ``` The model id cannot exist, so a `Hub` error can only come from a real round-trip. That **401 is the proof a socket was opened to huggingface.co with `HEADROOM_OFFLINE=1` set**, and `Offline` can only come from returning before `Api::new()`. **Refusal / startup (3, 4).** `headroom/{offline,proxy/server,transforms/content_router}.py` reverted to the PR tip, tests kept: `11 failed, 6 passed` — including `test_thinking_compactor_does_not_swallow_it`, `test_kompress_model_ready_does_not_report_a_refusal_as_ready`, `test_no_broad_handler_swallows_the_refusal` and all five `TestStartupRefusal` tests. **New Python guards (5).** `headroom/{onnx_runtime,observability/tracing}.py` reverted to `origin/main`, tests kept: `4 failed` — the two socket tests plus both coverage assertions. **Normalisation (6).** Both `offline` modules reverted to the PR tip, parity tests kept: `6 failed`, with `HEADROOM_OFFLINE="\x1c1"` reading as offline in Python and not in Rust. **Docs (7).** Both doc files reverted, test kept: ``` AssertionError: docs/metrics-technical-guide.md says 'disables all outbound traffic', but _EGRESS_ALLOWLIST still records these as reachable under HEADROOM_OFFLINE: [...16 files...] ``` #### This revision (the review fixes) Committed first, then `git checkout origin/main -- headroom/{binaries,cli/main,copilot_auth,graph/installer,integrations/asgi,integrations/litellm_callback,memory/adapters/embedders,memory/backends/direct_mem0,relevance/embedding}.py headroom/{evals,subscription}/` with the new tests kept: ``` 23 failed, 79 passed ``` The 23 are the five Copilot-auth socket tests, the four poller tests, the two install-download tests, the BFCL test, both Headroom Cloud tests, the two cloud-embedder tests, the four fastembed tests, both CLI-translation tests — and `test_every_egress_site_is_guarded_or_allowlisted`, which lists every reverted file as "(not guarded, not allowlisted)". The aliased-import blind spot, isolated (canonicalisation stubbed back to the identity, i.e. the pre-fix behaviour, against the same inputs): ``` before: INVISIBLE: import httpx as h INVISIBLE: from urllib import request as urllib_request INVISIBLE: from openai import AsyncOpenAI as AO INVISIBLE: import httpx as h (inside a function) aliased guard recognised: [False] copilot_auth.py sites seen: 4 after: DETECTED (all four); aliased guard recognised: [True] copilot_auth.py sites seen: 6 ``` **Pass-after (this revision):** `102 passed` in `tests/test_offline_egress_chokepoint.py`; `391 passed, 86 skipped` across the copilot-auth / copilot-quota / codex-rate-limits / binaries / integrations / CLI subsets; `544 passed, 15 skipped` across the relevance / embedding / mem0 / asgi / litellm / subscription / installer subsets; `160 passed, 6 skipped` across the evals and CLI doctor/learn subsets; `59 passed` across the proxy-startup subset; `cargo test -p headroom-core offline` → `8 passed`. **Pass-after (earlier revisions):** `70 passed` (pytest), `942 passed` (`cargo test -p headroom-core --lib`), `720 passed, 4 skipped` across the offline/otel/kompress/telemetry/onnx/tracing/relevance subsets, and `1868 passed, 30 skipped` across the kompress/offline/image/embedder/memory subsets after the model-loader boundary change. The one failure in the full run, `test_mcp_reconcile.py::test_ordinary_install_does_not_adopt_serena`, reproduces unchanged at the PR tip and is unrelated. Cost/pricing tests are the known `origin/main` litellm breakage (#3722) and were not run. ### Checks - `uvx ruff@0.16.7 check .` → All checks passed - `uvx ruff@0.16.7 format --check .` → 1622 files already formatted - `cargo fmt --check` → clean; `cargo clippy -p headroom-core --all-targets` → no warnings - `CHANGELOG.md` untouched - CodeQL: the four high alerts were all in `tests/test_offline_egress_chokepoint.py` and are fixed honestly rather than suppressed. Three `py/incomplete-url-substring-sanitization` host assertions (`"host" in url`, which is also true of `https://host.attacker.test`) now parse with `urllib.parse.urlsplit` and compare the hostname exactly — which is what the tests meant. The `py/redos` regex scraping Rust `char` literals had a nested `(...)+` that can backtrack exponentially; a Rust char literal is exactly one character, so it is now a single bounded alternation. ### Follow-ups (not in this PR) 1. ~~Guard the 16 `unguarded` files, each with its own user-facing message.~~ Done in this PR; the `unguarded` category is deleted. 2. Split the three Rust HF fetches cache-first so a pre-seeded air-gapped cache resolves without egress, the way `hf_hub_download_local_first` already does on the Python side. 3. The Rust sweep is a text heuristic. An AST-level dominance check (via `syn`, in a Rust test) would make the two halves symmetric. 4. The control-plane heartbeat lives in a different repository and is still out of scope. 5. The scanner has no entry for transformers `from_pretrained` / `SentenceTransformer`. The Kompress tokenizer site Devin found is guarded now, but `models/ml_models.py` (SigLIP, technique router, sentence-transformers) and the Kompress PyTorch backend still rely on `HF_HUB_OFFLINE` alone, which `apply_offline_env` sets with `setdefault` and only in the proxy process. Each needs the cache-first split the tokenizer has, plus a scanner entry. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3729" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ## Testing - [x] Unit tests pass (`pytest`) — subsets below - [x] Linting passes (`ruff check .`) — touched files, `ruff@0.16.7` - [ ] Type checking passes (`mypy headroom`) — not run - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # this revision $ LITELLM_LOCAL_MODEL_COST_MAP=True uv run --frozen --extra dev --extra proxy pytest \ tests/test_offline_egress_chokepoint.py tests/test_cli_doctor.py tests/test_corporate_tls.py -q 245 passed # after the Devin fixes $ pytest tests/test_offline_egress_chokepoint.py + kompress/onnx_router/relevance/embedding/evals subsets 361 passed, 6 skipped # fail-before for the Devin fixes (4 sources reverted, tests kept): 6 failed + the fallback test (fails once its fake # raises OSError instead of LocalEntryNotFoundError, which conftest converts to a skip) # fail-before, TLS guard (tls_diagnostics.py + doctor.py reverted, tests kept) FAILED TestDoctorNetworkOffline::test_endpoint_check_refuses_before_any_connection FAILED TestDoctorNetworkOffline::test_doctor_reports_the_network_checks_as_skipped FAILED TestDoctorNetworkOffline::test_chain_probe_opens_no_socket FAILED TestEgressChokepointCoverage::test_every_egress_site_is_guarded_or_allowlisted headroom/proxy/tls_diagnostics.py line 473: with httpx.Client(...) (not guarded, not allowlisted) # with only tls_diagnostics.py reverted, the doctor test fails on content, not on a missing helper: [('net api.anthropic.com', 'fail'), ('net api.openai.com', 'fail'), ...] != [('network', 'skip')] # fail-before, socket trap (old fixture body, new tests kept) FAILED TestSocketTrap::test_event_loop_wakeup_pipe_is_not_egress SocketOpened: outbound connection attempted while offline: (<socket ...>, ('127.0.0.1', 50151)) # — the same csock.connect inside socketpair() that Jerrett's Windows trace shows. $ uvx ruff@0.16.7 check / format --check (touched files) All checks passed! / 4 files already formatted ``` Earlier-revision test evidence is in *Fail-before / pass-after* under *Changes Made*. ## Real Behavior Proof - Environment: macOS, Python 3.14, this branch (merged with `main` at `befdb52f`), no proxy running. - Exact command / steps: `HEADROOM_OFFLINE=1 headroom doctor --network --port 59999`, run for real. A second run went through `headroom.cli.main` with a `sys.addaudithook` recording every `socket.getaddrinfo` / `socket.connect`. - Observed result: the doctor table shows `network · skip · skipped: HEADROOM_OFFLINE is set, no endpoint was contacted`, with the refusal naming `https://api.anthropic.com/v1/models` as the hint. The audit hook recorded only `127.0.0.1` lookups and connects (the doctor's own proxy probes on :59999 and :8787) and **zero external lookups or connects**. Exit code 2 comes from the unrelated "proxy not reachable" failures. - Not tested: Windows itself. The Windows mechanism is reproduced by forcing `socket._fallback_socketpair`, which is the code Windows runs for `socket.socketpair()`; the CI Windows job (`test-windows-newline`) only runs `windows_newline`-marked tests and was not extended here. ## Runtime Rollout Safety - Rollout-managed feature(s): none; behaviour keyed on the existing `HEADROOM_OFFLINE` switch. - Minimum rollout channel: stable. - Stable/default behavior changed: only when `HEADROOM_OFFLINE` is set (refusals listed above). Unset, nothing changes. - Kill switch / disable path: unset `HEADROOM_OFFLINE`. - Unsafe override required: no. - Qualification impact: none. - Rollback path: revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Two founder decisions remain flagged above (OTLP under offline; pollers loud vs silent). Windows CI coverage for this module is not added: no unit-test job runs the full module on Windows today, and adding one is a workflow change better made separately. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
ee6cfcccf6
commit
119d1a1939
@@ -8,6 +8,7 @@ pub mod auth_mode;
|
||||
pub mod cache_control;
|
||||
pub mod ccr;
|
||||
pub mod compression_policy;
|
||||
pub mod offline;
|
||||
#[cfg(feature = "ml")]
|
||||
mod onnx_cpu;
|
||||
pub mod relevance;
|
||||
@@ -29,6 +30,24 @@ pub fn hello() -> &'static str {
|
||||
"headroom-core"
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) mod test_support {
|
||||
use std::sync::{Mutex, MutexGuard};
|
||||
|
||||
/// The process environment is global while `cargo test` runs tests in
|
||||
/// parallel threads inside one process, so a test that sets an env var can
|
||||
/// be observed by an unrelated sibling mid-assertion. Every test that
|
||||
/// mutates the environment holds this for its whole body. Same shape as
|
||||
/// `REGISTRY_LOCK` in `tokenizer::registry`.
|
||||
static ENV_LOCK: Mutex<()> = Mutex::new(());
|
||||
|
||||
/// Recovers from poisoning on purpose: one panicking test should not
|
||||
/// cascade into every later test that touches the environment.
|
||||
pub(crate) fn env_lock() -> MutexGuard<'static, ()> {
|
||||
ENV_LOCK.lock().unwrap_or_else(|e| e.into_inner())
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
@@ -0,0 +1,181 @@
|
||||
//! Air-gap / no-egress master switch (`HEADROOM_OFFLINE`) — Rust half.
|
||||
//!
|
||||
//! The Python side lives in `headroom/offline.py`. This is the same switch
|
||||
//! read by the same environment variable with the same truthiness rules, so a
|
||||
//! deployment that sets `HEADROOM_OFFLINE=1` gets identical behaviour whether
|
||||
//! the egress attempt originates in the Python proxy or in a Rust core path.
|
||||
//!
|
||||
//! Two functions, mirroring the Python module deliberately:
|
||||
//!
|
||||
//! - [`is_offline`] — the predicate, for code that wants to take a different
|
||||
//! route (use a cached artifact, skip an optional refresh).
|
||||
//! - [`guard_egress`] — the chokepoint, for code that is about to open a
|
||||
//! socket. It returns [`OfflineEgressBlocked`], a distinct error type, so a
|
||||
//! caller that otherwise degrades gracefully on network failure can still
|
||||
//! tell "the operator air-gapped this box" from "the network was flaky" and
|
||||
//! refuse loudly instead of silently falling back.
|
||||
//!
|
||||
//! # Parity contract
|
||||
//!
|
||||
//! `HEADROOM_OFFLINE` is true for `1`, `true`, `yes`, `on` — trimmed and
|
||||
//! ASCII-case-insensitive — and false for anything else, including unset and
|
||||
//! empty. Any change here must be mirrored in `headroom/offline.py`'s
|
||||
//! `_TRUE_VALUES`, and vice versa: the two implementations are a pair, and a
|
||||
//! deployment that reads as offline to Python but online to Rust is precisely
|
||||
//! the failure this module exists to prevent.
|
||||
//!
|
||||
//! The parity covers **normalisation**, not just the accepted values. This
|
||||
//! used to call `str::trim()` while Python called `str.strip()`, and those are
|
||||
//! not the same set: Python's strips U+001C-U+001F (the ASCII file/group/
|
||||
//! record/unit separators) because `str.isspace()` includes them, Rust's does
|
||||
//! not because the Unicode `White_Space` property does not. `HEADROOM_OFFLINE`
|
||||
//! set to `"\x1c1"` read as offline to Python and online to Rust — one
|
||||
//! environment variable, one process air-gapped and the other not. Both sides
|
||||
//! now trim exactly [`TRIM_CHARS`].
|
||||
|
||||
use std::env;
|
||||
|
||||
use thiserror::Error;
|
||||
|
||||
/// The environment variable that selects fully-offline operation.
|
||||
pub const OFFLINE_ENV: &str = "HEADROOM_OFFLINE";
|
||||
|
||||
/// Values that read as "yes, offline". Kept byte-identical to the Python
|
||||
/// side's `_TRUE_VALUES` (see the parity contract in the module docs).
|
||||
const TRUE_VALUES: [&str; 4] = ["1", "true", "yes", "on"];
|
||||
|
||||
/// Whitespace trimmed off the raw value before matching. Enumerated rather
|
||||
/// than left to `str::trim()`, which is the Unicode `White_Space` property and
|
||||
/// does not agree with Python's `str.strip()`. Kept byte-identical to the
|
||||
/// Python side's `_TRIM_CHARS` (see the parity contract in the module docs).
|
||||
const TRIM_CHARS: [char; 6] = [' ', '\t', '\n', '\r', '\u{b}', '\u{c}'];
|
||||
|
||||
/// An egress attempt was refused because `HEADROOM_OFFLINE` is in force.
|
||||
///
|
||||
/// Carries the human-readable `purpose` and `destination` handed to
|
||||
/// [`guard_egress`] so the operator learns which feature to turn off rather
|
||||
/// than just that "something" was blocked.
|
||||
#[derive(Debug, Clone, Error)]
|
||||
#[error(
|
||||
"{OFFLINE_ENV} is set: refusing outbound network access for {purpose} to \
|
||||
{destination}. Unset {OFFLINE_ENV}, or turn off the feature that needs \
|
||||
this connection."
|
||||
)]
|
||||
pub struct OfflineEgressBlocked {
|
||||
pub purpose: String,
|
||||
pub destination: String,
|
||||
}
|
||||
|
||||
/// Return `true` when `HEADROOM_OFFLINE` selects fully-offline operation.
|
||||
pub fn is_offline() -> bool {
|
||||
match env::var(OFFLINE_ENV) {
|
||||
Ok(raw) => {
|
||||
let normalized = raw.trim_matches(TRIM_CHARS.as_slice()).to_ascii_lowercase();
|
||||
TRUE_VALUES.contains(&normalized.as_str())
|
||||
}
|
||||
// Unset, or not valid UTF-8 — neither is an opt-in to offline mode.
|
||||
Err(_) => false,
|
||||
}
|
||||
}
|
||||
|
||||
/// The chokepoint every Rust egress path must call before opening a socket.
|
||||
///
|
||||
/// `Ok(())` when online; `Err(OfflineEgressBlocked)` when `HEADROOM_OFFLINE`
|
||||
/// is set. Call it *before* constructing the HTTP client, not merely before
|
||||
/// the request — several clients (`ureq`, `reqwest`) resolve DNS or warm a
|
||||
/// connection pool during setup, so a guard placed at the request would leak
|
||||
/// the very packets the air-gap switch promises not to send.
|
||||
pub fn guard_egress(purpose: &str, destination: &str) -> Result<(), OfflineEgressBlocked> {
|
||||
if is_offline() {
|
||||
return Err(OfflineEgressBlocked {
|
||||
purpose: purpose.to_string(),
|
||||
destination: destination.to_string(),
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::test_support::env_lock;
|
||||
|
||||
#[test]
|
||||
fn unset_is_online() {
|
||||
let _guard = env_lock();
|
||||
env::remove_var(OFFLINE_ENV);
|
||||
assert!(!is_offline());
|
||||
assert!(guard_egress("anything", "https://example.invalid").is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn truthy_values_match_python() {
|
||||
let _guard = env_lock();
|
||||
// Same set, same trimming, same case-insensitivity as
|
||||
// headroom/offline.py::_TRUE_VALUES. If this list drifts, the two
|
||||
// halves of the switch disagree and the air-gap has a hole.
|
||||
for raw in ["1", "true", "TRUE", "Yes", " on ", "\ton\n"] {
|
||||
env::set_var(OFFLINE_ENV, raw);
|
||||
assert!(is_offline(), "{raw:?} should read as offline");
|
||||
}
|
||||
env::remove_var(OFFLINE_ENV);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn falsy_values_stay_online() {
|
||||
let _guard = env_lock();
|
||||
for raw in ["", "0", "off", "no", "false", "maybe", " "] {
|
||||
env::set_var(OFFLINE_ENV, raw);
|
||||
assert!(!is_offline(), "{raw:?} should read as online");
|
||||
}
|
||||
env::remove_var(OFFLINE_ENV);
|
||||
}
|
||||
|
||||
/// Normalisation parity, not just value parity. Every case here is
|
||||
/// duplicated verbatim in `tests/test_offline_egress_chokepoint.py`'s
|
||||
/// `test_normalisation_matches_the_rust_side`, and the two must agree.
|
||||
///
|
||||
/// The `\x1c` cases are the regression: they are whitespace to Python's
|
||||
/// `str.strip()` and not to Rust's `str::trim()`, so before `TRIM_CHARS`
|
||||
/// existed `"\x1c1"` air-gapped the Python proxy and left the Rust core
|
||||
/// dialling out.
|
||||
#[test]
|
||||
fn normalisation_matches_python() {
|
||||
let _guard = env_lock();
|
||||
for (raw, expected) in [
|
||||
("1", true),
|
||||
(" 1 ", true),
|
||||
("\t1\n", true),
|
||||
("\r\n TRUE \r\n", true),
|
||||
("\u{b}yes\u{c}", true),
|
||||
("\u{1c}1", false),
|
||||
("1\u{1f}", false),
|
||||
("\u{a0}1", false),
|
||||
("\u{2007}on", false),
|
||||
("", false),
|
||||
(" ", false),
|
||||
] {
|
||||
env::set_var(OFFLINE_ENV, raw);
|
||||
assert_eq!(is_offline(), expected, "{raw:?}");
|
||||
}
|
||||
env::remove_var(OFFLINE_ENV);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn guard_reports_purpose_and_destination() {
|
||||
let _guard = env_lock();
|
||||
env::set_var(OFFLINE_ENV, "1");
|
||||
let err = guard_egress("HuggingFace tokenizer download", "hf.co/acme/model")
|
||||
.expect_err("guard must refuse while offline");
|
||||
assert_eq!(err.purpose, "HuggingFace tokenizer download");
|
||||
assert_eq!(err.destination, "hf.co/acme/model");
|
||||
let rendered = err.to_string();
|
||||
assert!(rendered.contains(OFFLINE_ENV), "{rendered}");
|
||||
assert!(
|
||||
rendered.contains("HuggingFace tokenizer download"),
|
||||
"{rendered}"
|
||||
);
|
||||
assert!(rendered.contains("hf.co/acme/model"), "{rendered}");
|
||||
env::remove_var(OFFLINE_ENV);
|
||||
}
|
||||
}
|
||||
@@ -98,6 +98,22 @@ impl EmbeddingScorer {
|
||||
/// quality/speed tradeoff for compression-relevance scoring on
|
||||
/// short snippets.
|
||||
pub fn try_new_with_model(model_kind: EmbeddingModel) -> Result<Self, String> {
|
||||
let name = format!("{:?}", model_kind);
|
||||
// Air-gap chokepoint, FIRST thing in the function. `TextEmbedding::
|
||||
// try_new` resolves the model's ONNX weights through `hf-hub`, which
|
||||
// downloads from huggingface.co on a cache miss even with
|
||||
// `HF_HUB_OFFLINE` set — the same hole the tokenizer and Kompress model
|
||||
// fetches had. It sits ahead of the AVX2 and ort-loader probes below so
|
||||
// that an air-gapped box reports the policy refusal rather than
|
||||
// whichever local prerequisite happened to be missing as well.
|
||||
//
|
||||
// The refusal is soft here by design: this function already returns
|
||||
// `Err(String)` for "ONNX runtime unavailable" and every caller degrades
|
||||
// to the BM25 scorer, so an air-gapped box loses embedding relevance
|
||||
// instead of failing a request. The message names the switch so the
|
||||
// operator can tell the two causes apart in the log.
|
||||
crate::offline::guard_egress("fastembed embedding model download", &name)
|
||||
.map_err(|e| format!("EmbeddingScorer model load refused: {e}"))?;
|
||||
// fastembed links the precompiled ONNX Runtime binary, which contains
|
||||
// AVX2 instructions on x86. Loading/running it on a non-AVX2 CPU traps
|
||||
// with SIGILL (issue #1723) — an uncatchable native fault. Bail early so
|
||||
@@ -113,7 +129,6 @@ impl EmbeddingScorer {
|
||||
// `dynamic_ort_loader_ready`).
|
||||
crate::transforms::magika_detector::dynamic_ort_loader_ready()
|
||||
.map_err(|e| format!("EmbeddingScorer: ONNX Runtime unavailable: {e}"))?;
|
||||
let name = format!("{:?}", model_kind);
|
||||
let model = TextEmbedding::try_new(InitOptions::new(model_kind))
|
||||
.map_err(|e| format!("EmbeddingScorer model load failed: {}", e))?;
|
||||
Ok(EmbeddingScorer {
|
||||
@@ -308,6 +323,28 @@ mod tests {
|
||||
std::env::var("RUN_FASTEMBED_TESTS").is_ok()
|
||||
}
|
||||
|
||||
/// `try_new_with_model` must refuse before fastembed resolves the model,
|
||||
/// and must say so in the returned message rather than looking like an
|
||||
/// ordinary load failure — the caller only ever sees the `String`.
|
||||
///
|
||||
/// Not gated on `RUN_FASTEMBED_TESTS`: the whole point is that no network
|
||||
/// call happens, so this is safe to run in CI. If the guard were removed
|
||||
/// this test would either download ~30 MB or fail with a load error, and
|
||||
/// neither says "refused".
|
||||
#[cfg(feature = "ml")]
|
||||
#[test]
|
||||
fn try_new_refuses_while_offline() {
|
||||
let _guard = crate::test_support::env_lock();
|
||||
std::env::set_var(crate::offline::OFFLINE_ENV, "1");
|
||||
let result = EmbeddingScorer::try_new();
|
||||
std::env::remove_var(crate::offline::OFFLINE_ENV);
|
||||
|
||||
let err = result.err().expect("guard must refuse while offline");
|
||||
assert!(err.contains("refused"), "{err}");
|
||||
assert!(err.contains(crate::offline::OFFLINE_ENV), "{err}");
|
||||
assert!(err.contains("fastembed embedding model download"), "{err}");
|
||||
}
|
||||
|
||||
/// Construct a stub scorer with `model = None` for offline-safe
|
||||
/// tests of the unavailable-path behavior.
|
||||
#[cfg(feature = "ml")]
|
||||
|
||||
@@ -49,6 +49,13 @@ pub enum HfTokenizerError {
|
||||
#[source]
|
||||
source: Box<dyn std::error::Error + Send + Sync>,
|
||||
},
|
||||
/// `HEADROOM_OFFLINE` is set, so the Hub fetch was refused before any
|
||||
/// socket was opened. Deliberately a separate variant from [`Self::Hub`]:
|
||||
/// a caller that retries or falls back on a transient download failure
|
||||
/// must not do either here — the operator asked for no egress, and
|
||||
/// retrying an air-gap refusal is just a slower refusal.
|
||||
#[error(transparent)]
|
||||
Offline(#[from] crate::offline::OfflineEgressBlocked),
|
||||
}
|
||||
|
||||
/// Token counter backed by a HuggingFace `tokenizer.json`.
|
||||
@@ -110,11 +117,35 @@ impl HfTokenizer {
|
||||
/// processes hit the on-disk cache.
|
||||
///
|
||||
/// Errors:
|
||||
/// - [`HfTokenizerError::Offline`] when `HEADROOM_OFFLINE` is set. Refused
|
||||
/// before any socket is opened; see below.
|
||||
/// - [`HfTokenizerError::Hub`] for download failures (no network, 404,
|
||||
/// 401 on a gated model without `HF_TOKEN`).
|
||||
/// - [`HfTokenizerError::Load`] if the downloaded bytes don't parse as
|
||||
/// a valid `tokenizer.json`. Should not happen for healthy HF repos.
|
||||
pub fn from_pretrained(repo: &str) -> Result<Self, HfTokenizerError> {
|
||||
// Air-gap chokepoint, matching `headroom/offline.py`'s `guard_egress`.
|
||||
// This is the Rust half of the same switch, so an operator who sets
|
||||
// HEADROOM_OFFLINE=1 gets the same refusal whichever runtime reaches
|
||||
// for the Hub.
|
||||
//
|
||||
// It has to sit here rather than around the `get` below: `Api::new`
|
||||
// builds the `ureq` agent and resolves the endpoint, and hf-hub's own
|
||||
// HF_HUB_OFFLINE handling only covers the cache lookup — a cache miss
|
||||
// still dials out. Guarding first means the air-gapped case never
|
||||
// constructs a client at all.
|
||||
//
|
||||
// The refusal is hard on purpose. Silently returning a cached or
|
||||
// estimating tokenizer would leave token counts subtly wrong with no
|
||||
// signal; the caller (`tokenizer::registry::try_register_hf`) already
|
||||
// propagates the error so the operator sees which repo was wanted.
|
||||
//
|
||||
// Note it refuses even on a warm cache: `hf-hub` 0.5 exposes no
|
||||
// cache-hit/cache-miss split at this layer, so "guard before the
|
||||
// client exists" and "serve from the cache" cannot both hold here. An
|
||||
// air-gapped deployment with pre-seeded artifacts should point at them
|
||||
// with `from_file`, which is network-free by construction.
|
||||
crate::offline::guard_egress("HuggingFace tokenizer download", repo)?;
|
||||
let api = hf_hub::api::sync::Api::new().map_err(|e| HfTokenizerError::Hub {
|
||||
repo: repo.to_string(),
|
||||
source: Box::new(e),
|
||||
@@ -260,6 +291,41 @@ mod tests {
|
||||
assert!(Arc::ptr_eq(&a.inner, &b.inner));
|
||||
}
|
||||
|
||||
/// A-2: `HEADROOM_OFFLINE=1` must stop the Hub fetch before a socket exists.
|
||||
///
|
||||
/// The repo name is deliberately one that cannot exist, which is what makes
|
||||
/// this a real assertion rather than a tautology: without the guard the call
|
||||
/// reaches `hf-hub`, dials `huggingface.co`, gets a 401/404 and returns
|
||||
/// `HfTokenizerError::Hub`. Only the guard can produce `Offline`, and only
|
||||
/// by returning before `Api::new` — so `Offline` *is* the "no socket was
|
||||
/// opened" assertion for this path. (A packet-level assertion is not
|
||||
/// available to a `cargo test` unit test the way `socket.socket` patching is
|
||||
/// in Python; this typed-error distinction is the equivalent signal.)
|
||||
#[test]
|
||||
fn from_pretrained_refuses_while_offline() {
|
||||
let _guard = crate::test_support::env_lock();
|
||||
std::env::set_var(crate::offline::OFFLINE_ENV, "1");
|
||||
let result = HfTokenizer::from_pretrained("headroomlabs-ai/this-repo-does-not-exist-a2");
|
||||
std::env::remove_var(crate::offline::OFFLINE_ENV);
|
||||
|
||||
match result {
|
||||
Err(HfTokenizerError::Offline(blocked)) => {
|
||||
assert_eq!(blocked.purpose, "HuggingFace tokenizer download");
|
||||
assert_eq!(
|
||||
blocked.destination,
|
||||
"headroomlabs-ai/this-repo-does-not-exist-a2"
|
||||
);
|
||||
}
|
||||
Err(other) => panic!("expected an offline refusal, got a network error: {other}"),
|
||||
Ok(_) => panic!("expected an offline refusal, got a tokenizer"),
|
||||
}
|
||||
}
|
||||
|
||||
// The "switch off ⇒ guard is a no-op" half is covered in `crate::offline`'s
|
||||
// own tests rather than here: asserting it through `from_pretrained` would
|
||||
// mean actually dialling huggingface.co, which is exactly the thing a unit
|
||||
// test must not do.
|
||||
|
||||
#[test]
|
||||
fn from_file_loads_a_real_file() {
|
||||
// Round-trip via a temp file to cover the on-disk constructor.
|
||||
|
||||
@@ -180,6 +180,12 @@ pub enum KompressError {
|
||||
#[source]
|
||||
source: Box<dyn std::error::Error + Send + Sync>,
|
||||
},
|
||||
/// `HEADROOM_OFFLINE` is set, so the Hub fetch was refused before any
|
||||
/// socket was opened. A separate variant from [`Self::Hub`] for the same
|
||||
/// reason as [`crate::tokenizer::HfTokenizerError::Offline`]: retrying or
|
||||
/// falling back on a policy refusal is just a slower refusal.
|
||||
#[error(transparent)]
|
||||
Offline(#[from] crate::offline::OfflineEgressBlocked),
|
||||
}
|
||||
|
||||
// ─── Compressor ─────────────────────────────────────────────────────────
|
||||
@@ -255,6 +261,22 @@ impl Kompress {
|
||||
/// downloading on miss). Blocking — call off the hot path. Tries the
|
||||
/// [`ONNX_CANDIDATES`] in order.
|
||||
pub fn from_pretrained(config: KompressConfig) -> Result<Self, KompressError> {
|
||||
// Air-gap chokepoint — the same one `hf_impl.rs` uses for the
|
||||
// tokenizer, for the same reason. `HF_HUB_OFFLINE` only shortcuts the
|
||||
// cache *lookup*; on a cache miss `hf-hub` still dials huggingface.co,
|
||||
// so the Kompress model download is a live egress path under
|
||||
// `HEADROOM_OFFLINE=1` and was not covered.
|
||||
//
|
||||
// It sits before `Api::new()` because that already resolves the Hub
|
||||
// endpoint and builds the `ureq` agent, so guarding at the `get` call
|
||||
// would leak the setup packets the switch promises not to send.
|
||||
//
|
||||
// Trade-off, stated plainly: this refuses even when the artifacts are
|
||||
// already in `~/.cache/huggingface/hub`, because `hf-hub` 0.5 gives no
|
||||
// cache-hit/cache-miss split at this layer. An air-gapped caller that
|
||||
// has pre-seeded artifacts should use [`Self::from_files`], which is
|
||||
// network-free by construction and unaffected by this guard.
|
||||
crate::offline::guard_egress("Kompress model download", &config.model_id)?;
|
||||
let api = hf_hub::api::sync::Api::new().map_err(|e| KompressError::Hub {
|
||||
repo: config.model_id.clone(),
|
||||
source: Box::new(e),
|
||||
@@ -635,6 +657,38 @@ fn hf_hub_roots() -> Vec<PathBuf> {
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// `from_pretrained` must refuse before `Api::new()` when the operator
|
||||
/// air-gapped the box.
|
||||
///
|
||||
/// The model id cannot exist, which is what makes this a real assertion
|
||||
/// rather than a tautology: without the guard the call reaches `hf-hub`,
|
||||
/// dials `huggingface.co` and comes back `KompressError::Hub`. Only the
|
||||
/// guard can produce `Offline`, and only by returning before the client is
|
||||
/// built — so `Offline` *is* the "no socket was opened" assertion here.
|
||||
#[test]
|
||||
fn from_pretrained_refuses_while_offline() {
|
||||
let _guard = crate::test_support::env_lock();
|
||||
std::env::set_var(crate::offline::OFFLINE_ENV, "1");
|
||||
let config = KompressConfig {
|
||||
model_id: "headroomlabs-ai/this-model-does-not-exist-a2".to_string(),
|
||||
..KompressConfig::default()
|
||||
};
|
||||
let result = Kompress::from_pretrained(config);
|
||||
std::env::remove_var(crate::offline::OFFLINE_ENV);
|
||||
|
||||
match result {
|
||||
Err(KompressError::Offline(blocked)) => {
|
||||
assert_eq!(blocked.purpose, "Kompress model download");
|
||||
assert_eq!(
|
||||
blocked.destination,
|
||||
"headroomlabs-ai/this-model-does-not-exist-a2"
|
||||
);
|
||||
}
|
||||
Err(other) => panic!("expected an offline refusal, got a network error: {other}"),
|
||||
Ok(_) => panic!("expected an offline refusal, got a loaded model"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn config_defaults_match_kompress_v2_base() {
|
||||
let c = KompressConfig::default();
|
||||
|
||||
@@ -341,7 +341,7 @@ with a privileged job.
|
||||
| `HEADROOM_PROXY_TOKEN` | none | Require a token from non-loopback callers, as `X-Headroom-Proxy-Token: <token>` or `Authorization: Bearer <token>`. The token is removed from the request before it is forwarded upstream; clients that also send a provider credential should use `X-Headroom-Proxy-Token` and keep `Authorization` for the provider. Also gates the read-only operator routes (`/stats-history`, `/quota`, `/subscription-window`, `/metrics`), which otherwise answer loopback and trusted-CIDR callers only. |
|
||||
| `HEADROOM_ALLOW_UNAUTHENTICATED_BIND` | `false` | Acknowledge a non-loopback bind (`--host 0.0.0.0`, `::`, a LAN address) with **no** `HEADROOM_PROXY_TOKEN`. Without a token or this flag the proxy refuses to start, because `/v1/*` would relay upstream with the operator's credentials for every peer that can reach the port. Only set it when the runtime already restricts reachability (a container published on `127.0.0.1`). |
|
||||
| `HEADROOM_COMPRESS_ALLOW_REMOTE` | `false` | Allow non-loopback callers to reach [`POST /v1/compress`](#post-v1compress) and `POST /v1/usage`. Required to run Headroom as a remote gateway/sidecar; without it remote callers get `404`. |
|
||||
| `--offline` / `HEADROOM_OFFLINE` | `false` | Air-gap mode: hard-disable **all** egress (telemetry, update checks, license reporting, model downloads). |
|
||||
| `--offline` / `HEADROOM_OFFLINE` | `false` | Air-gap mode: refuse every connection Headroom itself initiates — telemetry, update checks, license reporting, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP and Langfuse export, Copilot auth, the subscription pollers, the OpenAI embedders, Headroom Cloud compression and the TLS diagnostics (`headroom doctor --network` reports them as skipped). Four things still work on purpose: forwarding your own requests to the upstream you configured, operator-configured local endpoints (the Ollama embedder), loopback health probes, and paths an `is_offline()` check already disables. Refusals surface as messages, not tracebacks. See the air-gap notes in `docs/metrics-technical-guide.md`. |
|
||||
| `HEADROOM_LICENSE` | unset | Licence token, read by core and every licensed extension. `HEADROOM_LICENSE_KEY` is a deprecated alias for one release and logs a warning. A licence on its own sends nothing to Headroom. |
|
||||
| `HEADROOM_USAGE_REPORTING` | `false` | Set `1` to let the proxy validate the licence and post aggregate usage counts (no content) to the Headroom cloud. Off unless set; `HEADROOM_OFFLINE` still overrides it. |
|
||||
| `--stateless` / `HEADROOM_STATELESS` | `false` | Keep all state in memory; no filesystem writes (disables logs, memory, TOIN). |
|
||||
|
||||
@@ -268,6 +268,17 @@ Verify with `curl -s localhost:8787/stats | jq .otel`.
|
||||
|
||||
**Multi-tenant labels:** `register_otel_metric_attribute_provider()` adds request-scoped attributes (tenant, team, cost centre) to every OTel datapoint. Max 16 attributes, 256 chars each.
|
||||
|
||||
**Air-gapped deployments:** `HEADROOM_OFFLINE=1` disables all outbound traffic — the anonymous usage beacon (which is **on by default**), the update check, and model downloads.
|
||||
**Air-gapped deployments:** `HEADROOM_OFFLINE=1` refuses every connection Headroom itself decides to make to a destination Headroom itself chose. That is the whole list, not a sample: the anonymous usage beacon (which is **on by default**), the update check, the license/usage reporter, **OTLP metric and Langfuse trace export**, HuggingFace / Kompress / fastembed model and tokenizer downloads (Python and Rust), the remote Kompress endpoint, the `headroom install` release-binary and codebase-memory-mcp downloads, the BFCL eval-dataset fetch and the provider SDK clients the eval harness drives, GitHub Copilot device-flow auth and token exchange, the Anthropic / Codex / Copilot subscription pollers, the OpenAI embedders, the Headroom Cloud compression modes in the ASGI and LiteLLM integrations, and the TLS diagnostics (`headroom doctor --network` endpoint checks and the certificate-chain re-probe after an upstream TLS failure).
|
||||
|
||||
Four things are deliberately still allowed, and they are the complete set of exceptions:
|
||||
|
||||
1. **Your traffic through the proxy.** Requests you send *through* Headroom are still forwarded to the upstream you configured. That is your traffic, not Headroom's, and an air-gapped deployment points it at an on-prem endpoint — refusing it would mean refusing to be a proxy.
|
||||
2. **Operator-configured local endpoints.** Today that is exactly one thing: the Ollama embedder, whose address comes entirely from your configuration and defaults to `http://localhost:11434`. No hard-coded internet host is permitted under this exception.
|
||||
3. **Loopback.** Health and readiness probes against your own proxy on `127.0.0.1` (`headroom doctor`, the installers, the MCP sidecar).
|
||||
4. **Paths an `is_offline()` check already makes unreachable**, where no connection is ever built in the first place.
|
||||
|
||||
Those four are enumerated with written reasons in `_EGRESS_ALLOWLIST` in `tests/test_offline_egress_chokepoint.py`, which fails the build if a new egress path appears that is neither guarded nor one of them. There is no category for "known violation": a path that can dial the internet with the flag set is a bug. A refusal reaches you as a message — a Click error on the CLI, a named "model unavailable" from a model loader, a logged line from a background poller — never as a traceback and never as a silent degradation. Enforcing the same policy at the network layer as well is still good practice; it is no longer the only thing standing between you and Headroom-initiated egress.
|
||||
|
||||
OTLP export is refused loudly: with `HEADROOM_OFFLINE=1` set, `HEADROOM_OTEL_METRICS_ENABLED=1` plus the default `otlp_http` exporter makes the proxy exit 78 at startup with an explanation, rather than quietly dropping metrics. There is no exemption for a collector that looks local — an in-cluster address is not reliably distinguishable from an internet one. For metrics under an air-gap, either scrape `localhost:8787/metrics` or set `HEADROOM_OTEL_METRICS_EXPORTER=console`, both of which stay on-box. `HEADROOM_KOMPRESS_ENDPOINT` and `HEADROOM_LANGFUSE_ENABLED` are refused the same way, for the same reason.
|
||||
|
||||
---
|
||||
|
||||
@@ -51,6 +51,7 @@ from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from headroom._subprocess import run
|
||||
from headroom.offline import guard_egress
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -266,6 +267,11 @@ def _mirror_url(url: str) -> str:
|
||||
def _download(url: str, dest: Path, *, progress: bool = True) -> None:
|
||||
if os.environ.get("HEADROOM_BINARIES_OFFLINE"):
|
||||
raise OfflineError(f"offline mode (HEADROOM_BINARIES_OFFLINE=1) but fetch required: {url}")
|
||||
# HEADROOM_BINARIES_OFFLINE above is the narrow, binaries-only switch; this
|
||||
# is the deployment-wide one. An operator who set the air-gap switch should
|
||||
# not have to also discover a second, differently-named flag to stop
|
||||
# Headroom fetching release assets from GitHub at install time.
|
||||
guard_egress("release binary download", url)
|
||||
if not _has_writable_existing_parent(dest.parent):
|
||||
raise OSError(f"binary cache directory parent is not writable: {dest.parent}")
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
+23
-2
@@ -843,6 +843,28 @@ def run_network_probes(extra_urls: Sequence[str]) -> tuple[list[Any], set[str]]:
|
||||
return reports, required
|
||||
|
||||
|
||||
def network_checks(extra_urls: Sequence[str]) -> list[CheckResult]:
|
||||
"""The ``--network`` rows, or one skipped row when ``HEADROOM_OFFLINE`` is set.
|
||||
|
||||
Every probe here dials an internet host, so under the air-gap switch none
|
||||
of them runs; saying so beats a page of "unreachable" failures.
|
||||
"""
|
||||
from headroom.offline import OFFLINE_ENV, OfflineEgressBlocked
|
||||
|
||||
try:
|
||||
reports, required = run_network_probes(extra_urls)
|
||||
except OfflineEgressBlocked as blocked:
|
||||
return [
|
||||
CheckResult(
|
||||
name="network",
|
||||
status=SKIP,
|
||||
summary=f"skipped: {OFFLINE_ENV} is set, no endpoint was contacted",
|
||||
hint=str(blocked),
|
||||
)
|
||||
]
|
||||
return check_network_endpoints(reports, required)
|
||||
|
||||
|
||||
def check_shell_env(environ: Mapping[str, str], port: int) -> CheckResult:
|
||||
"""Is the *current shell* pointed at the proxy for ad-hoc runs?"""
|
||||
name = "shell env"
|
||||
@@ -1233,8 +1255,7 @@ def doctor(
|
||||
if proxy_env_check is not None:
|
||||
checks.append(proxy_env_check)
|
||||
if network or network_urls:
|
||||
reports, required = run_network_probes(network_urls)
|
||||
checks.extend(check_network_endpoints(reports, required))
|
||||
checks.extend(network_checks(network_urls))
|
||||
auth_conflict_check = check_claude_auth_conflict(
|
||||
claude_settings_path(),
|
||||
project_claude_settings,
|
||||
|
||||
+31
-1
@@ -2,9 +2,39 @@
|
||||
|
||||
import click
|
||||
|
||||
from headroom.offline import OfflineEgressBlocked
|
||||
|
||||
CLI_CONTEXT_SETTINGS = {"help_option_names": ["--help", "-?"]}
|
||||
|
||||
|
||||
class OfflineAwareGroup(click.Group):
|
||||
"""The CLI's single translation point for an air-gap refusal.
|
||||
|
||||
``OfflineEgressBlocked`` is a ``BaseException`` on purpose, so no
|
||||
``except Exception`` anywhere in the tree can downgrade a policy refusal
|
||||
into "that feature is having a bad day". The cost of that choice is that
|
||||
nothing catches it either — Click's own error handling only knows about
|
||||
``ClickException``/``Abort``, so an operator who ran ``headroom auth
|
||||
copilot`` on an air-gapped box would have got a stack trace ending in a
|
||||
type they have never heard of.
|
||||
|
||||
Translating here, once, at the outermost boundary of every subcommand,
|
||||
turns it into the ordinary ``Error: ...`` line Click prints for any other
|
||||
bad invocation — with the switch, the feature and the host already in the
|
||||
message, because :class:`OfflineEgressBlocked` builds them in. The guard
|
||||
keeps its unswallowable property everywhere inside; only the last frame
|
||||
before the user's terminal is allowed to make it readable.
|
||||
"""
|
||||
|
||||
def invoke(self, ctx: click.Context) -> object:
|
||||
try:
|
||||
return super().invoke(ctx)
|
||||
except OfflineEgressBlocked as blocked:
|
||||
# ``from None``: the traceback is Headroom's own plumbing and
|
||||
# tells the operator nothing they can act on. The message does.
|
||||
raise click.ClickException(str(blocked)) from None
|
||||
|
||||
|
||||
def get_version() -> str:
|
||||
"""Get the current version."""
|
||||
try:
|
||||
@@ -15,7 +45,7 @@ def get_version() -> str:
|
||||
return "unknown"
|
||||
|
||||
|
||||
@click.group(context_settings=CLI_CONTEXT_SETTINGS)
|
||||
@click.group(cls=OfflineAwareGroup, context_settings=CLI_CONTEXT_SETTINGS)
|
||||
@click.version_option(get_version(), "--version", "-v", prog_name="headroom")
|
||||
@click.pass_context
|
||||
def main(ctx: click.Context) -> None:
|
||||
|
||||
@@ -26,6 +26,7 @@ from headroom import paths
|
||||
from headroom._subprocess import run
|
||||
from headroom.copilot_linux_secret import read_copilot_oauth_token as read_linux_secret_token
|
||||
from headroom.copilot_macos_keychain import read_copilot_oauth_token as read_macos_keychain_token
|
||||
from headroom.offline import guard_egress
|
||||
from headroom.proxy import ssl_context as proxy_ssl_context
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -79,8 +80,20 @@ _EXPIRY_KEYS = ("expires_at", "expiresAt", "expiry", "expires")
|
||||
|
||||
|
||||
def _urlopen(request: urllib_request.Request, *, timeout: float) -> Any:
|
||||
"""Open a GitHub request with Headroom's configured corporate trust roots."""
|
||||
"""Open a GitHub request with Headroom's configured corporate trust roots.
|
||||
|
||||
Every GitHub call in this module bottoms out here, so this is also the
|
||||
backstop for the air-gap switch. The four call sites guard individually
|
||||
too — they can name the specific step ("device-flow authorisation",
|
||||
"token exchange") in the refusal, which this frame cannot — but a fifth
|
||||
call site added later inherits the refusal from here whether its author
|
||||
thought about ``HEADROOM_OFFLINE`` or not.
|
||||
"""
|
||||
|
||||
guard_egress(
|
||||
"GitHub Copilot authentication",
|
||||
urlparse(request.full_url).hostname or request.full_url,
|
||||
)
|
||||
context = proxy_ssl_context.build_urlopen_context()
|
||||
if context is not None:
|
||||
return urllib_request.urlopen(request, timeout=timeout, context=context)
|
||||
@@ -699,6 +712,7 @@ def start_copilot_device_authorization(
|
||||
"""Start the GitHub Copilot OAuth device-code flow."""
|
||||
|
||||
urls = _github_oauth_urls(domain)
|
||||
guard_egress("GitHub Copilot device-flow authorisation", domain)
|
||||
body = urlencode({"client_id": COPILOT_CHAT_OAUTH_CLIENT_ID, "scope": "read:user"}).encode(
|
||||
"utf-8"
|
||||
)
|
||||
@@ -730,6 +744,7 @@ def poll_copilot_device_authorization(
|
||||
"""Poll GitHub until the device-code OAuth flow returns an access token."""
|
||||
|
||||
urls = _github_oauth_urls(domain)
|
||||
guard_egress("GitHub Copilot device-flow authorisation", domain)
|
||||
deadline = time.time() + max(1, expires_in)
|
||||
poll_interval = max(1, interval)
|
||||
while time.time() < deadline:
|
||||
@@ -1423,6 +1438,7 @@ def _fetch_copilot_user_info(token: str) -> dict[str, Any] | None:
|
||||
if not token:
|
||||
return None
|
||||
|
||||
guard_egress("GitHub Copilot account lookup", urlparse(_user_info_url()).hostname)
|
||||
headers = _copilot_token_exchange_headers(token)
|
||||
request = urllib_request.Request(_user_info_url(), headers=headers, method="GET")
|
||||
try:
|
||||
@@ -1540,6 +1556,10 @@ class CopilotTokenProvider:
|
||||
|
||||
@staticmethod
|
||||
def _exchange_token_sync(headers: dict[str, str]) -> dict[str, Any]:
|
||||
guard_egress(
|
||||
"GitHub Copilot API token exchange",
|
||||
urlparse(_token_exchange_url()).hostname,
|
||||
)
|
||||
request = urllib_request.Request(_token_exchange_url(), headers=headers, method="GET")
|
||||
try:
|
||||
with _urlopen(request, timeout=10.0) as response:
|
||||
|
||||
@@ -46,6 +46,7 @@ from headroom.evals.metrics import (
|
||||
compute_rouge_l,
|
||||
compute_semantic_similarity,
|
||||
)
|
||||
from headroom.offline import guard_egress
|
||||
from headroom.transforms.content_router import ContentRouter, ContentRouterConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -1029,6 +1030,7 @@ class BatchCompressionEvaluator:
|
||||
|
||||
def _init_llm_client(self) -> Any:
|
||||
"""Initialize LLM client."""
|
||||
guard_egress(f"{self.provider} API for the batch compression benchmark")
|
||||
if self.provider == "anthropic":
|
||||
try:
|
||||
import anthropic
|
||||
|
||||
+46
-22
@@ -28,10 +28,14 @@ Custom:
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from headroom.evals.core import EvalCase, EvalSuite
|
||||
from headroom.offline import guard_egress, is_offline
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _check_datasets_installed() -> None:
|
||||
@@ -45,6 +49,36 @@ def _check_datasets_installed() -> None:
|
||||
) from e
|
||||
|
||||
|
||||
def load_hf_dataset(path: str, *args: Any, **kwargs: Any) -> Any:
|
||||
"""``datasets.load_dataset``, cache first under ``HEADROOM_OFFLINE``.
|
||||
|
||||
``load_dataset`` takes no ``local_files_only``, and with an explicit
|
||||
``HF_HUB_OFFLINE=0`` it fetches from the Hub even when ``HEADROOM_OFFLINE``
|
||||
is set. So, as ``relevance/embedding._load_text_embedding`` does for
|
||||
fastembed: under the switch, try the load with the datasets and
|
||||
huggingface_hub offline constants forced on (a pre-seeded cache still
|
||||
loads), and only on a miss reach :func:`guard_egress`, which refuses
|
||||
before any remote attempt. Online, nothing changes.
|
||||
"""
|
||||
from datasets import load_dataset
|
||||
|
||||
if is_offline():
|
||||
import datasets.config as datasets_config
|
||||
from huggingface_hub import constants as hf_constants
|
||||
|
||||
previous = (datasets_config.HF_HUB_OFFLINE, hf_constants.HF_HUB_OFFLINE)
|
||||
datasets_config.HF_HUB_OFFLINE = True
|
||||
hf_constants.HF_HUB_OFFLINE = True
|
||||
try:
|
||||
return load_dataset(path, *args, **kwargs)
|
||||
except Exception as cache_miss: # noqa: BLE001 - any local-lookup failure
|
||||
logger.debug("eval dataset %s not in the local cache: %s", path, cache_miss)
|
||||
finally:
|
||||
datasets_config.HF_HUB_OFFLINE, hf_constants.HF_HUB_OFFLINE = previous
|
||||
guard_egress(f"HuggingFace eval-dataset download ({path})", "huggingface.co")
|
||||
return load_dataset(path, *args, **kwargs)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# RAG / RETRIEVAL DATASETS
|
||||
# =============================================================================
|
||||
@@ -69,9 +103,8 @@ def load_hotpotqa(
|
||||
EvalSuite with HotpotQA cases
|
||||
"""
|
||||
_check_datasets_installed()
|
||||
from datasets import load_dataset
|
||||
|
||||
ds = load_dataset("hotpotqa/hotpot_qa", "fullwiki", split=split)
|
||||
ds = load_hf_dataset("hotpotqa/hotpot_qa", "fullwiki", split=split)
|
||||
|
||||
cases: list[EvalCase] = []
|
||||
for i, item in enumerate(ds):
|
||||
@@ -121,9 +154,8 @@ def load_natural_questions(
|
||||
EvalSuite with Natural Questions cases
|
||||
"""
|
||||
_check_datasets_installed()
|
||||
from datasets import load_dataset
|
||||
|
||||
ds = load_dataset("google-research-datasets/natural_questions", "default", split=split)
|
||||
ds = load_hf_dataset("google-research-datasets/natural_questions", "default", split=split)
|
||||
|
||||
cases: list[EvalCase] = []
|
||||
for i, item in enumerate(ds):
|
||||
@@ -198,9 +230,8 @@ def load_triviaqa(
|
||||
EvalSuite with TriviaQA cases
|
||||
"""
|
||||
_check_datasets_installed()
|
||||
from datasets import load_dataset
|
||||
|
||||
ds = load_dataset("trivia_qa", subset, split=split)
|
||||
ds = load_hf_dataset("trivia_qa", subset, split=split)
|
||||
|
||||
cases: list[EvalCase] = []
|
||||
for i, item in enumerate(ds):
|
||||
@@ -269,9 +300,8 @@ def load_msmarco(
|
||||
EvalSuite with MS MARCO cases
|
||||
"""
|
||||
_check_datasets_installed()
|
||||
from datasets import load_dataset
|
||||
|
||||
ds = load_dataset("microsoft/ms_marco", "v2.1", split=split)
|
||||
ds = load_hf_dataset("microsoft/ms_marco", "v2.1", split=split)
|
||||
|
||||
cases: list[EvalCase] = []
|
||||
for i, item in enumerate(ds):
|
||||
@@ -337,9 +367,8 @@ def load_squad(
|
||||
EvalSuite with SQuAD cases
|
||||
"""
|
||||
_check_datasets_installed()
|
||||
from datasets import load_dataset
|
||||
|
||||
ds = load_dataset("rajpurkar/squad_v2", split=split)
|
||||
ds = load_hf_dataset("rajpurkar/squad_v2", split=split)
|
||||
|
||||
cases: list[EvalCase] = []
|
||||
for i, item in enumerate(ds):
|
||||
@@ -398,10 +427,9 @@ def load_longbench(
|
||||
EvalSuite with LongBench cases
|
||||
"""
|
||||
_check_datasets_installed()
|
||||
from datasets import load_dataset
|
||||
|
||||
try:
|
||||
ds = load_dataset("THUDM/LongBench", task, split="test")
|
||||
ds = load_hf_dataset("THUDM/LongBench", task, split="test")
|
||||
except Exception as e:
|
||||
raise ValueError(f"Failed to load LongBench task '{task}': {e}") from e
|
||||
|
||||
@@ -460,10 +488,9 @@ def load_multi_wiki_qa(
|
||||
EvalSuite with multi-wiki-qa cases
|
||||
"""
|
||||
_check_datasets_installed()
|
||||
from datasets import load_dataset
|
||||
|
||||
try:
|
||||
ds = load_dataset("alexandrainst/multi-wiki-qa", lang, split=f"train[:{n}]")
|
||||
ds = load_hf_dataset("alexandrainst/multi-wiki-qa", lang, split=f"train[:{n}]")
|
||||
except Exception as e:
|
||||
raise ValueError(f"Failed to load multi-wiki-qa '{lang}': {e}") from e
|
||||
|
||||
@@ -513,9 +540,8 @@ def load_narrativeqa(
|
||||
EvalSuite with NarrativeQA cases
|
||||
"""
|
||||
_check_datasets_installed()
|
||||
from datasets import load_dataset
|
||||
|
||||
ds = load_dataset("deepmind/narrativeqa", split=split)
|
||||
ds = load_hf_dataset("deepmind/narrativeqa", split=split)
|
||||
|
||||
cases: list[EvalCase] = []
|
||||
for i, item in enumerate(ds):
|
||||
@@ -590,6 +616,7 @@ def load_bfcl(
|
||||
import urllib.request
|
||||
|
||||
base_url = "https://huggingface.co/datasets/gorilla-llm/Berkeley-Function-Calling-Leaderboard/resolve/main"
|
||||
guard_egress("BFCL eval-dataset download", base_url)
|
||||
data_file = f"BFCL_v3_{category}.json"
|
||||
gt_file = f"possible_answer/BFCL_v3_{category}.json"
|
||||
|
||||
@@ -676,10 +703,9 @@ def load_toolbench(
|
||||
EvalSuite with ToolBench cases
|
||||
"""
|
||||
_check_datasets_installed()
|
||||
from datasets import load_dataset
|
||||
|
||||
try:
|
||||
ds = load_dataset("ToolBench/ToolBench", category, split="test")
|
||||
ds = load_hf_dataset("ToolBench/ToolBench", category, split="test")
|
||||
except Exception as e:
|
||||
raise ValueError(f"Failed to load ToolBench category '{category}': {e}") from e
|
||||
|
||||
@@ -759,10 +785,9 @@ def load_codesearchnet(
|
||||
EvalSuite with CodeSearchNet cases
|
||||
"""
|
||||
_check_datasets_installed()
|
||||
from datasets import load_dataset
|
||||
|
||||
try:
|
||||
ds = load_dataset("code_search_net", language, split=split)
|
||||
ds = load_hf_dataset("code_search_net", language, split=split)
|
||||
except Exception as e:
|
||||
raise ValueError(f"Failed to load CodeSearchNet for '{language}': {e}") from e
|
||||
|
||||
@@ -815,9 +840,8 @@ def load_humaneval(
|
||||
EvalSuite with HumanEval cases
|
||||
"""
|
||||
_check_datasets_installed()
|
||||
from datasets import load_dataset
|
||||
|
||||
ds = load_dataset("openai_humaneval", split="test")
|
||||
ds = load_hf_dataset("openai_humaneval", split="test")
|
||||
|
||||
cases: list[EvalCase] = []
|
||||
for i, item in enumerate(ds):
|
||||
|
||||
@@ -15,6 +15,8 @@ from collections.abc import Callable
|
||||
from dataclasses import dataclass, field
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from headroom.offline import guard_egress
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from headroom.transforms.html_extractor import HTMLExtractor
|
||||
|
||||
@@ -255,6 +257,7 @@ class HTMLExtractionEvaluator:
|
||||
|
||||
def _create_judge(self) -> Callable[[str, str, str], tuple[float, str]]:
|
||||
"""Create the LLM judge function."""
|
||||
guard_egress(f"{self.provider} API for the HTML-extraction eval judge")
|
||||
if self.provider == "openai":
|
||||
try:
|
||||
from openai import OpenAI
|
||||
@@ -355,6 +358,7 @@ class HTMLExtractionEvaluator:
|
||||
|
||||
def _get_answer(self, content: str, question: str) -> str:
|
||||
"""Get LLM answer for a question given content."""
|
||||
guard_egress(f"{self.provider} API for the HTML-extraction eval answers")
|
||||
prompt = f"""Based on the following content, answer the question.
|
||||
|
||||
Content:
|
||||
|
||||
@@ -142,12 +142,15 @@ def evaluate_scrapinghub_benchmark(
|
||||
print(f"F1: {result.avg_f1:.3f} (baseline: {result.baseline_f1})")
|
||||
"""
|
||||
try:
|
||||
from datasets import load_dataset
|
||||
import datasets # noqa: F401
|
||||
except ImportError:
|
||||
raise ImportError(
|
||||
"HuggingFace datasets required. Install with: pip install datasets"
|
||||
) from None
|
||||
|
||||
# Cache first under HEADROOM_OFFLINE, guard_egress before any download.
|
||||
from headroom.evals.datasets import load_hf_dataset
|
||||
|
||||
if extractor is None:
|
||||
from headroom.transforms.html_extractor import HTMLExtractor
|
||||
|
||||
@@ -155,7 +158,7 @@ def evaluate_scrapinghub_benchmark(
|
||||
|
||||
# Load the benchmark dataset
|
||||
logger.info("Loading Scrapinghub article extraction benchmark...")
|
||||
dataset = load_dataset("allenai/scrapinghub-article-extraction-benchmark")
|
||||
dataset = load_hf_dataset("allenai/scrapinghub-article-extraction-benchmark")
|
||||
samples = dataset["train"]
|
||||
|
||||
if max_samples:
|
||||
@@ -274,10 +277,13 @@ def evaluate_qa_accuracy_preservation(
|
||||
QAAccuracyResult showing whether accuracy is preserved
|
||||
"""
|
||||
try:
|
||||
from datasets import load_dataset
|
||||
import datasets # noqa: F401
|
||||
except ImportError:
|
||||
raise ImportError("HuggingFace datasets required") from None
|
||||
|
||||
# Cache first under HEADROOM_OFFLINE, guard_egress before any download.
|
||||
from headroom.evals.datasets import load_hf_dataset
|
||||
|
||||
if extractor is None:
|
||||
from headroom.transforms.html_extractor import HTMLExtractor
|
||||
|
||||
@@ -287,9 +293,9 @@ def evaluate_qa_accuracy_preservation(
|
||||
logger.info(f"Loading {dataset_name} dataset...")
|
||||
|
||||
if dataset_name == "squad":
|
||||
dataset = load_dataset("rajpurkar/squad_v2", split="validation")
|
||||
dataset = load_hf_dataset("rajpurkar/squad_v2", split="validation")
|
||||
elif dataset_name == "hotpotqa":
|
||||
dataset = load_dataset("hotpotqa/hotpot_qa", "fullwiki", split="validation")
|
||||
dataset = load_hf_dataset("hotpotqa/hotpot_qa", "fullwiki", split="validation")
|
||||
else:
|
||||
raise ValueError(f"Unknown dataset: {dataset_name}")
|
||||
|
||||
|
||||
@@ -17,6 +17,8 @@ from __future__ import annotations
|
||||
import logging
|
||||
from collections.abc import Callable
|
||||
|
||||
from headroom.offline import guard_egress
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Prompt template for LLM judge
|
||||
@@ -73,6 +75,7 @@ def create_openai_judge(
|
||||
"OpenAI package required for LLM judge. Install with: pip install openai"
|
||||
) from e
|
||||
|
||||
guard_egress("OpenAI API for the memory eval LLM judge", "api.openai.com")
|
||||
client = OpenAI(api_key=api_key) if api_key else OpenAI()
|
||||
|
||||
def judge(question: str, ground_truth: str, prediction: str) -> tuple[float, str]:
|
||||
@@ -124,6 +127,7 @@ def create_anthropic_judge(
|
||||
"Anthropic package required for LLM judge. Install with: pip install anthropic"
|
||||
) from e
|
||||
|
||||
guard_egress("Anthropic API for the memory eval LLM judge", "api.anthropic.com")
|
||||
client = anthropic.Anthropic(api_key=api_key) if api_key else anthropic.Anthropic()
|
||||
|
||||
def judge(question: str, ground_truth: str, prediction: str) -> tuple[float, str]:
|
||||
|
||||
@@ -32,6 +32,8 @@ from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from headroom.offline import guard_egress
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@@ -259,6 +261,7 @@ def compare_prompts(
|
||||
"or pass api_key parameter."
|
||||
)
|
||||
|
||||
guard_egress("OpenAI API for the prompt-comparison judge", "api.openai.com")
|
||||
client = OpenAI(api_key=resolved_api_key)
|
||||
|
||||
# Build the judge prompt
|
||||
|
||||
@@ -25,6 +25,7 @@ from headroom.evals.core import (
|
||||
EvalSuiteResult,
|
||||
)
|
||||
from headroom.evals.metrics import compute_semantic_similarity
|
||||
from headroom.offline import guard_egress
|
||||
from headroom.transforms.content_router import ContentRouter, ContentRouterConfig
|
||||
from headroom.transforms.smart_crusher import SmartCrusherConfig
|
||||
|
||||
@@ -95,8 +96,13 @@ class BeforeAfterRunner:
|
||||
self._judge_fn: Any = None
|
||||
|
||||
def _init_llm_client(self) -> Any:
|
||||
"""Initialize the appropriate LLM client."""
|
||||
"""Initialize the appropriate LLM client.
|
||||
|
||||
Ollama is not guarded: like the Ollama embedder it is an
|
||||
operator-configured endpoint that defaults to localhost.
|
||||
"""
|
||||
if self.llm_config.provider == "anthropic":
|
||||
guard_egress("anthropic API for the before/after eval runner")
|
||||
try:
|
||||
import anthropic
|
||||
|
||||
@@ -106,6 +112,7 @@ class BeforeAfterRunner:
|
||||
"anthropic package required. Install with: pip install anthropic"
|
||||
) from e
|
||||
elif self.llm_config.provider == "openai":
|
||||
guard_egress("openai API for the before/after eval runner")
|
||||
try:
|
||||
import openai
|
||||
|
||||
@@ -128,6 +135,10 @@ class BeforeAfterRunner:
|
||||
|
||||
def _init_proxy_client(self) -> Any:
|
||||
"""Initialize an OpenAI client pointing at the Headroom proxy."""
|
||||
guard_egress(
|
||||
"LLM calls through the Headroom proxy for the before/after eval runner",
|
||||
self.llm_config.headroom_proxy_url,
|
||||
)
|
||||
import openai
|
||||
|
||||
return openai.OpenAI(
|
||||
|
||||
@@ -12,6 +12,8 @@ import zipfile
|
||||
from pathlib import Path
|
||||
from urllib.request import urlopen
|
||||
|
||||
from headroom.offline import guard_egress
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
CBM_VERSION = "v0.8.1"
|
||||
@@ -98,6 +100,7 @@ def download_cbm(version: str | None = None) -> Path:
|
||||
plat = _detect_platform()
|
||||
filename = _asset_filename(plat)
|
||||
url = f"{GITHUB_RELEASE_URL}/{version}/{filename}"
|
||||
guard_egress("codebase-memory-mcp binary download", url)
|
||||
|
||||
CBM_BIN_DIR.mkdir(parents=True, exist_ok=True)
|
||||
target_path = CBM_BIN_DIR / _installed_name(plat)
|
||||
|
||||
@@ -22,6 +22,7 @@ from typing import Any
|
||||
import numpy as np
|
||||
|
||||
from headroom.image.image_types import ImageMemo, ImageSignals, RouteDecision, Technique
|
||||
from headroom.offline import OFFLINE_ENV, OfflineEgressBlocked
|
||||
from headroom.onnx_runtime import create_cpu_session_options, hf_hub_download_local_first
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -67,6 +68,25 @@ def _classifier_intra_op_threads() -> int:
|
||||
# ImageSignals, RouteDecision, Technique imported from trained_router
|
||||
|
||||
|
||||
def _hf_artifact(repo: str, filename: str) -> str:
|
||||
"""``hf_hub_download_local_first``, translating the air-gap refusal.
|
||||
|
||||
``OfflineEgressBlocked`` is a ``BaseException`` so that it cannot be
|
||||
swallowed by accident. Here it is handled on purpose: ``ImageCompressor``
|
||||
already degrades a router that will not load to ``Technique.PRESERVE``, and
|
||||
an air-gapped box with no cached router should get that same degradation
|
||||
rather than a failed request. Re-raised as a plain ``RuntimeError``, with
|
||||
the switch named in the message, so the existing handler sees it and the
|
||||
log says why.
|
||||
"""
|
||||
try:
|
||||
return hf_hub_download_local_first(repo, filename)
|
||||
except OfflineEgressBlocked as blocked:
|
||||
raise RuntimeError(
|
||||
f"{filename} for {repo} is not cached and {OFFLINE_ENV} forbids fetching it: {blocked}"
|
||||
) from blocked
|
||||
|
||||
|
||||
class OnnxTechniqueRouter:
|
||||
"""ONNX-based technique router — no PyTorch dependency.
|
||||
|
||||
@@ -98,8 +118,15 @@ class OnnxTechniqueRouter:
|
||||
|
||||
logger.info("Loading technique-router ONNX INT8...")
|
||||
|
||||
model_path = hf_hub_download_local_first(_TECHNIQUE_ROUTER_REPO, "model_quantized.onnx")
|
||||
self._classifier_session = ort.InferenceSession(
|
||||
# Resolve every artifact before publishing anything: the session is the
|
||||
# "loaded" flag, so setting it before a later artifact fails (e.g. not
|
||||
# cached under HEADROOM_OFFLINE) would leave a half-loaded router that
|
||||
# never retries.
|
||||
model_path = _hf_artifact(_TECHNIQUE_ROUTER_REPO, "model_quantized.onnx")
|
||||
tokenizer_path = _hf_artifact(_TECHNIQUE_ROUTER_REPO, "tokenizer.json")
|
||||
config_path = _hf_artifact(_TECHNIQUE_ROUTER_REPO, "config.json")
|
||||
|
||||
session = ort.InferenceSession(
|
||||
model_path,
|
||||
create_cpu_session_options(
|
||||
ort,
|
||||
@@ -109,18 +136,18 @@ class OnnxTechniqueRouter:
|
||||
providers=["CPUExecutionProvider"],
|
||||
)
|
||||
|
||||
tokenizer_path = hf_hub_download_local_first(_TECHNIQUE_ROUTER_REPO, "tokenizer.json")
|
||||
self._tokenizer = Tokenizer.from_file(tokenizer_path)
|
||||
self._tokenizer.enable_truncation(max_length=64)
|
||||
self._tokenizer.enable_padding(length=64)
|
||||
tokenizer = Tokenizer.from_file(tokenizer_path)
|
||||
tokenizer.enable_truncation(max_length=64)
|
||||
tokenizer.enable_padding(length=64)
|
||||
|
||||
# Load label mapping
|
||||
import json
|
||||
|
||||
config_path = hf_hub_download_local_first(_TECHNIQUE_ROUTER_REPO, "config.json")
|
||||
with open(config_path) as f:
|
||||
config = json.load(f)
|
||||
self._id2label = {int(k): v for k, v in config.get("id2label", {}).items()}
|
||||
self._tokenizer = tokenizer
|
||||
self._classifier_session = session
|
||||
|
||||
logger.info(
|
||||
f"Technique router loaded: {len(self._id2label)} classes, "
|
||||
@@ -136,16 +163,18 @@ class OnnxTechniqueRouter:
|
||||
|
||||
logger.info("Loading SigLIP ONNX INT8 image encoder...")
|
||||
|
||||
model_path = hf_hub_download_local_first(_SIGLIP_ENCODER_REPO, "image_encoder_int8.onnx")
|
||||
self._siglip_session = ort.InferenceSession(
|
||||
# Both artifacts first, session last — see _load_classifier.
|
||||
model_path = _hf_artifact(_SIGLIP_ENCODER_REPO, "image_encoder_int8.onnx")
|
||||
embeddings_path = _hf_artifact(_SIGLIP_ENCODER_REPO, "text_embeddings.npz")
|
||||
|
||||
session = ort.InferenceSession(
|
||||
model_path,
|
||||
create_cpu_session_options(ort),
|
||||
providers=["CPUExecutionProvider"],
|
||||
)
|
||||
|
||||
embeddings_path = hf_hub_download_local_first(_SIGLIP_ENCODER_REPO, "text_embeddings.npz")
|
||||
loaded = np.load(embeddings_path)
|
||||
self._text_embeddings = {k: loaded[k] for k in loaded.files}
|
||||
self._siglip_session = session
|
||||
|
||||
logger.info(
|
||||
f"SigLIP image encoder loaded: ONNX INT8 "
|
||||
|
||||
@@ -34,6 +34,8 @@ from typing import Any
|
||||
|
||||
from starlette.types import ASGIApp, Receive, Scope, Send
|
||||
|
||||
from headroom.offline import guard_egress
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_DEFAULT_CLOUD_URL = "https://api.headroomlabs.ai"
|
||||
@@ -204,7 +206,18 @@ class CompressionMiddleware:
|
||||
}
|
||||
|
||||
async def _cloud_compress(self, messages: list[dict], model: str) -> dict[str, Any] | None:
|
||||
"""Compress via Headroom Cloud API (managed CCR, TOIN, analytics)."""
|
||||
"""Compress via Headroom Cloud API (managed CCR, TOIN, analytics).
|
||||
|
||||
This is the one path in this file that puts the caller's prompt
|
||||
content on the wire to a Headroom-operated host, so it is exactly what
|
||||
HEADROOM_OFFLINE exists to stop. "Opt-in by configuration" was the old
|
||||
reason for leaving it open, and it is not good enough: an operator who
|
||||
sets an air-gap switch is overriding earlier configuration on purpose.
|
||||
The refusal is loud rather than a silent fall-through to local
|
||||
compression, because silently compressing locally would hide the fact
|
||||
that the deployment is no longer doing what it was configured to do.
|
||||
"""
|
||||
guard_egress("Headroom Cloud compression", self._api_url)
|
||||
if self._client is None:
|
||||
try:
|
||||
import httpx
|
||||
|
||||
@@ -20,6 +20,8 @@ import json
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from headroom.offline import guard_egress
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_DEFAULT_CLOUD_URL = "https://api.headroomlabs.ai"
|
||||
@@ -184,7 +186,18 @@ class HeadroomCallback(_CustomLogger):
|
||||
}
|
||||
|
||||
async def _cloud_compress(self, messages: list[dict], model: str) -> dict[str, Any] | None:
|
||||
"""Compress via Headroom Cloud API (managed CCR, TOIN, analytics)."""
|
||||
"""Compress via Headroom Cloud API (managed CCR, TOIN, analytics).
|
||||
|
||||
This is the one path in this file that puts the caller's prompt
|
||||
content on the wire to a Headroom-operated host, so it is exactly what
|
||||
HEADROOM_OFFLINE exists to stop. "Opt-in by configuration" was the old
|
||||
reason for leaving it open, and it is not good enough: an operator who
|
||||
sets an air-gap switch is overriding earlier configuration on purpose.
|
||||
The refusal is loud rather than a silent fall-through to local
|
||||
compression, because silently compressing locally would hide the fact
|
||||
that the deployment is no longer doing what it was configured to do.
|
||||
"""
|
||||
guard_egress("Headroom Cloud compression", self._api_url)
|
||||
if self._client is None:
|
||||
try:
|
||||
import httpx
|
||||
|
||||
@@ -21,6 +21,7 @@ from typing import TYPE_CHECKING, Any, cast
|
||||
import numpy as np
|
||||
|
||||
from headroom.models.config import ML_MODEL_DEFAULTS
|
||||
from headroom.offline import OFFLINE_ENV, OfflineEgressBlocked, guard_egress
|
||||
from headroom.onnx_runtime import create_cpu_session_options, hf_hub_download_local_first
|
||||
|
||||
if TYPE_CHECKING:
|
||||
@@ -450,8 +451,20 @@ class OnnxLocalEmbedder:
|
||||
logger.info("Loading ONNX embedding model (all-MiniLM-L6-v2, ~86MB)...")
|
||||
|
||||
# Prefer local cache to avoid a redundant network HEAD on warm starts.
|
||||
model_path = hf_hub_download_local_first(self.ONNX_REPO, "model.onnx")
|
||||
tok_path = hf_hub_download_local_first(self.ONNX_REPO, "tokenizer.json")
|
||||
# OfflineEgressBlocked is a BaseException so it cannot be swallowed by
|
||||
# accident; here the degradation is deliberate and written down. A model
|
||||
# download is not data leaving the box, and an air-gapped deployment
|
||||
# without a cached embedder should get "the local embedder is
|
||||
# unavailable" — the caller's existing failure mode — rather than a
|
||||
# BaseException escaping into the memory pipeline.
|
||||
try:
|
||||
model_path = hf_hub_download_local_first(self.ONNX_REPO, "model.onnx")
|
||||
tok_path = hf_hub_download_local_first(self.ONNX_REPO, "tokenizer.json")
|
||||
except OfflineEgressBlocked as blocked:
|
||||
raise RuntimeError(
|
||||
f"local ONNX embedder {self.ONNX_REPO} is not cached and "
|
||||
f"{OFFLINE_ENV} forbids fetching it: {blocked}"
|
||||
) from blocked
|
||||
|
||||
# Keep a small thread pool for Docker compatibility and disable ORT's
|
||||
# CPU memory arena/pattern caches so long-running proxy workers do not
|
||||
@@ -655,6 +668,7 @@ class OpenAIEmbedder:
|
||||
@cached_property
|
||||
def _async_client(self) -> Any:
|
||||
"""Lazy initialization of async OpenAI client."""
|
||||
guard_egress("OpenAI embedding API", "api.openai.com")
|
||||
from openai import AsyncOpenAI
|
||||
|
||||
return AsyncOpenAI(api_key=self._api_key)
|
||||
|
||||
@@ -60,6 +60,7 @@ from typing import Any
|
||||
from headroom.memory import qdrant_env
|
||||
from headroom.memory.models import Memory
|
||||
from headroom.memory.ports import MemorySearchResult
|
||||
from headroom.offline import guard_egress
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -169,7 +170,12 @@ class DirectMem0Adapter:
|
||||
if self._initialized:
|
||||
return
|
||||
|
||||
# Initialize embedder (OpenAI)
|
||||
# Initialize embedder (OpenAI). The air-gap switch outranks the
|
||||
# backend configuration: this embeds the user's memories by shipping
|
||||
# them to api.openai.com, which is precisely what an operator who set
|
||||
# HEADROOM_OFFLINE is refusing. Use the ONNX or Ollama embedder
|
||||
# instead on an air-gapped box.
|
||||
guard_egress("OpenAI embedding API (direct mem0 backend)", "api.openai.com")
|
||||
try:
|
||||
from openai import OpenAI
|
||||
|
||||
|
||||
@@ -32,6 +32,7 @@ import logging
|
||||
from threading import RLock
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from ..offline import OFFLINE_ENV, OfflineEgressBlocked
|
||||
from .config import ML_MODEL_DEFAULTS
|
||||
|
||||
if TYPE_CHECKING:
|
||||
@@ -40,6 +41,42 @@ if TYPE_CHECKING:
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class ModelUnavailableOffline(RuntimeError):
|
||||
"""A model is not in the local cache and ``HEADROOM_OFFLINE`` forbids fetching it.
|
||||
|
||||
The legible form of an air-gap refusal at a model-loader boundary. Fetching
|
||||
public weights is not data leaving the box, so the loader degrades instead
|
||||
of failing the request: every caller of this registry already treats a
|
||||
load failure as "feature unavailable" (the image router preserves the
|
||||
image, the memory embedder falls back, ...). The original
|
||||
:class:`~headroom.offline.OfflineEgressBlocked` is chained.
|
||||
"""
|
||||
|
||||
def __init__(self, purpose: str, name_or_path: str) -> None:
|
||||
self.purpose = purpose
|
||||
self.name_or_path = name_or_path
|
||||
super().__init__(
|
||||
f"{purpose} {name_or_path!r} is not in the local HuggingFace cache and "
|
||||
f"{OFFLINE_ENV} forbids downloading it. Pre-seed the cache to use this feature."
|
||||
)
|
||||
|
||||
|
||||
def _load_local_first(loader: Any, name_or_path: str, *, purpose: str, **kwargs: Any) -> Any:
|
||||
"""Cache-first load with the air-gap refusal translated and logged.
|
||||
|
||||
Thin wrapper over :func:`headroom.onnx_runtime.hf_from_pretrained_local_first`
|
||||
(the one place the cache-first + ``guard_egress`` order lives).
|
||||
"""
|
||||
from ..onnx_runtime import hf_from_pretrained_local_first
|
||||
|
||||
try:
|
||||
return hf_from_pretrained_local_first(loader, name_or_path, purpose=purpose, **kwargs)
|
||||
except OfflineEgressBlocked as blocked:
|
||||
unavailable = ModelUnavailableOffline(purpose, name_or_path)
|
||||
logger.warning("%s (%s)", unavailable, blocked)
|
||||
raise unavailable from blocked
|
||||
|
||||
|
||||
class MLModelRegistry:
|
||||
"""Singleton registry for shared ML model instances.
|
||||
|
||||
@@ -168,7 +205,12 @@ class MLModelRegistry:
|
||||
if device is None:
|
||||
device = cls._detect_device()
|
||||
|
||||
model = SentenceTransformer(model_name, device=device)
|
||||
model = _load_local_first(
|
||||
SentenceTransformer,
|
||||
model_name,
|
||||
purpose="SentenceTransformer model",
|
||||
device=device,
|
||||
)
|
||||
instance._models[key] = model
|
||||
logger.info(f"Loaded SentenceTransformer: {model_name} on {device}")
|
||||
|
||||
@@ -207,8 +249,12 @@ class MLModelRegistry:
|
||||
if device is None:
|
||||
device = cls._detect_device()
|
||||
|
||||
model = AutoModel.from_pretrained(model_name)
|
||||
processor = AutoProcessor.from_pretrained(model_name)
|
||||
model = _load_local_first(
|
||||
AutoModel.from_pretrained, model_name, purpose="SigLIP model"
|
||||
)
|
||||
processor = _load_local_first(
|
||||
AutoProcessor.from_pretrained, model_name, purpose="SigLIP processor"
|
||||
)
|
||||
|
||||
# Move to device and set eval mode
|
||||
if device != "cpu":
|
||||
@@ -295,8 +341,16 @@ class MLModelRegistry:
|
||||
if device is None:
|
||||
device = cls._detect_device()
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_path)
|
||||
model = AutoModelForSequenceClassification.from_pretrained(model_path)
|
||||
tokenizer = _load_local_first(
|
||||
AutoTokenizer.from_pretrained,
|
||||
model_path,
|
||||
purpose="technique-router tokenizer",
|
||||
)
|
||||
model = _load_local_first(
|
||||
AutoModelForSequenceClassification.from_pretrained,
|
||||
model_path,
|
||||
purpose="technique-router model",
|
||||
)
|
||||
|
||||
# Move to device and set eval mode
|
||||
if device != "cpu":
|
||||
|
||||
@@ -13,6 +13,7 @@ from opentelemetry import metrics
|
||||
from opentelemetry.metrics import CallbackOptions, Observation
|
||||
|
||||
from headroom._version import get_version
|
||||
from headroom.offline import guard_egress
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -717,6 +718,22 @@ def configure_otel_metrics(config: OTelMetricsConfig | None = None) -> HeadroomO
|
||||
if resolved.exporter == "console":
|
||||
exporter = ConsoleMetricExporter()
|
||||
else:
|
||||
# Air-gap chokepoint. OTLP/HTTP export dials a collector and then keeps
|
||||
# POSTing on a background timer (PeriodicExportingMetricReader), so it
|
||||
# is exactly the kind of long-lived, easy-to-forget egress
|
||||
# HEADROOM_OFFLINE is supposed to cover — and it was not covered.
|
||||
#
|
||||
# The guard runs before OTLPMetricExporter is constructed, because the
|
||||
# exporter opens its session eagerly. No exemption for a collector that
|
||||
# looks local: the endpoint may be unset here (the OTEL SDK then falls
|
||||
# back to OTEL_EXPORTER_OTLP_ENDPOINT or the 4318 default) and a
|
||||
# sidecar address is not distinguishable from an internet one. An
|
||||
# operator who wants in-cluster metrics under an air-gap should use the
|
||||
# "console" exporter or scrape /metrics, both of which stay on-box.
|
||||
guard_egress(
|
||||
"OTLP metric export",
|
||||
resolved.endpoint or "the OTEL SDK's default OTLP endpoint",
|
||||
)
|
||||
exporter_kwargs: dict[str, Any] = {}
|
||||
if resolved.endpoint is not None:
|
||||
exporter_kwargs["endpoint"] = resolved.endpoint
|
||||
|
||||
@@ -11,6 +11,8 @@ from typing import Any
|
||||
|
||||
from opentelemetry import trace
|
||||
|
||||
from headroom.offline import guard_egress
|
||||
|
||||
from .metrics import _headroom_version, _parse_bool, _parse_key_value_pairs
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -158,6 +160,17 @@ def configure_langfuse_tracing(
|
||||
)
|
||||
return get_headroom_tracer()
|
||||
|
||||
# Air-gap chokepoint, the exact counterpart of the one in metrics.py.
|
||||
# Langfuse OTLP tracing ships every span off the box on a background
|
||||
# BatchSpanProcessor timer — and unlike the metrics exporter, the endpoint
|
||||
# here is Langfuse Cloud unless the operator overrode it. It was missed
|
||||
# when the metric exporter was guarded; the two are the same egress shape
|
||||
# and now refuse the same way. Guarded before OTLPSpanExporter is
|
||||
# constructed, because the exporter opens its session eagerly.
|
||||
guard_egress(
|
||||
"Langfuse OTLP trace export",
|
||||
resolved.endpoint or "the OTEL SDK's default OTLP endpoint",
|
||||
)
|
||||
resource = Resource.create(
|
||||
{
|
||||
SERVICE_NAME: resolved.service_name,
|
||||
|
||||
+189
-6
@@ -1,10 +1,59 @@
|
||||
"""Air-gap / no-egress master switch (``HEADROOM_OFFLINE``).
|
||||
|
||||
A single predicate the individual egress paths consult so a regulated or
|
||||
air-gapped deployment can disable **all** outbound network access with one
|
||||
flag: the telemetry beacon, the update check, the license/usage reporter, and
|
||||
HuggingFace model downloads. Each of those already had its own opt-out; this
|
||||
is the one switch that turns them all off together and fails closed.
|
||||
THE POLICY, in one sentence: with ``HEADROOM_OFFLINE`` set, Headroom refuses
|
||||
every connection **Headroom itself decides to make to a destination Headroom
|
||||
itself chose**, and keeps working for the two kinds of traffic an air-gapped
|
||||
deployment exists to carry.
|
||||
|
||||
Refused (this is not a sample — it is the rule, and the meta-test enforces it):
|
||||
the telemetry beacon, the update check, the license/usage reporter, OTLP and
|
||||
Langfuse export, HuggingFace/Kompress/fastembed model downloads, release-binary
|
||||
and codebase-memory-mcp downloads, eval dataset downloads and the provider SDK
|
||||
clients the eval harness drives, GitHub Copilot device-flow auth and token
|
||||
exchange, the Anthropic / Codex / Copilot subscription pollers, the OpenAI
|
||||
embedders, both Headroom Cloud compression integrations, and the TLS
|
||||
diagnostics (``doctor --network`` endpoint checks and the certificate-chain
|
||||
re-probe after an upstream TLS failure).
|
||||
|
||||
Permitted, explicitly and by name — an air-gapped deployment needs these and
|
||||
none of them is Headroom phoning home:
|
||||
|
||||
``user-traffic``
|
||||
The proxy forwarding the caller's own request to the upstream the operator
|
||||
configured. Forwarding a request the user's client made, to the endpoint
|
||||
the user pointed Headroom at, is the proxy's entire job. An air-gapped
|
||||
install points it at an on-prem model endpoint.
|
||||
``operator-endpoint``
|
||||
Headroom dialling an address that comes **entirely** from operator
|
||||
configuration and whose default is loopback — today that is exactly one
|
||||
thing, the Ollama embedder (``http://localhost:11434``). No hard-coded
|
||||
internet host may hide behind this category.
|
||||
``loopback``
|
||||
Hard-coded ``127.0.0.1`` traffic to the operator's own local proxy: the
|
||||
readiness and health probes, ``headroom doctor``, the MCP sidecar.
|
||||
``gated``
|
||||
A path that an ``is_offline()`` check upstream already makes unreachable,
|
||||
so the connection is never even built.
|
||||
|
||||
Those four are the complete list of exceptions. There is no "known violation"
|
||||
category: a path that dials out under ``HEADROOM_OFFLINE`` for any other
|
||||
reason is a bug, not an entry in a table.
|
||||
|
||||
:func:`guard_egress` is the chokepoint: a path that is about to open an
|
||||
outbound connection calls it and gets a loud :class:`OfflineEgressBlocked`
|
||||
instead of a socket. Prefer it over a bare ``if is_offline(): return`` at any
|
||||
call site that actually dials out — the meta-test in
|
||||
``tests/test_offline_egress_chokepoint.py`` enumerates the outbound clients in
|
||||
``headroom/`` and ``crates/`` and requires each **site** to have a guard that
|
||||
dominates it or to be counted in an allowlist under one of the four categories
|
||||
above, so the guard is what keeps a newly-added egress path from silently
|
||||
escaping the air-gap.
|
||||
|
||||
A refusal must reach a human as a sentence, never as a traceback and never as
|
||||
a silent degradation. Interactive paths get that from the CLI boundary in
|
||||
``headroom/cli/main.py``, which turns the refusal into a Click error; model
|
||||
loaders translate it into their own "model unavailable"; background pollers
|
||||
call :func:`note_refusal` and stop.
|
||||
|
||||
Kept at the top level (depends only on the stdlib) so any layer — telemetry,
|
||||
proxy, model code — can import it without creating a package cycle.
|
||||
@@ -12,16 +61,32 @@ proxy, model code — can import it without creating a package cycle.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
|
||||
_TRUE_VALUES = frozenset({"1", "true", "yes", "on"})
|
||||
|
||||
# Whitespace trimmed off the raw value before matching. Enumerated rather than
|
||||
# left to ``str.strip()``'s default, because that default is Python's
|
||||
# ``str.isspace()`` — which includes U+001C-U+001F (the ASCII file/group/record
|
||||
# /unit separators) — while Rust's ``str::trim()`` is the Unicode White_Space
|
||||
# property, which does not. ``HEADROOM_OFFLINE=$'\x1c1'`` therefore read as
|
||||
# offline to Python and online to Rust: one process air-gapped, the other not,
|
||||
# from one environment variable. Neither side's default is more right than the
|
||||
# other, so both now trim exactly this set. Mirrored by ``TRIM_CHARS`` in
|
||||
# ``crates/headroom-core/src/offline.rs``; the parity test compares them.
|
||||
#
|
||||
# Case folding needs no such treatment: Python's ``str.lower()`` and Rust's
|
||||
# ``to_ascii_lowercase()`` differ only on non-ASCII input, and no non-ASCII
|
||||
# character lowercases into any character of "1"/"true"/"yes"/"on".
|
||||
_TRIM_CHARS = " \t\n\r\x0b\x0c"
|
||||
|
||||
OFFLINE_ENV = "HEADROOM_OFFLINE"
|
||||
|
||||
|
||||
def is_offline() -> bool:
|
||||
"""Return True when ``HEADROOM_OFFLINE`` selects fully-offline operation."""
|
||||
return os.environ.get(OFFLINE_ENV, "").strip().lower() in _TRUE_VALUES
|
||||
return os.environ.get(OFFLINE_ENV, "").strip(_TRIM_CHARS).lower() in _TRUE_VALUES
|
||||
|
||||
|
||||
def apply_offline_env() -> None:
|
||||
@@ -34,3 +99,121 @@ def apply_offline_env() -> None:
|
||||
if is_offline():
|
||||
os.environ.setdefault("HF_HUB_OFFLINE", "1")
|
||||
os.environ.setdefault("TRANSFORMERS_OFFLINE", "1")
|
||||
|
||||
|
||||
class OfflineEgressBlocked(BaseException):
|
||||
"""Raised by :func:`guard_egress` when ``HEADROOM_OFFLINE`` is in force.
|
||||
|
||||
A distinct, named type so callers can tell "the operator air-gapped this
|
||||
box" apart from "the network was flaky". That distinction matters because
|
||||
several Headroom egress paths deliberately fail OPEN on network errors
|
||||
(remote Kompress passes content through verbatim, the license reporter
|
||||
falls back to a cached grant). Fail-open is right for a flaky endpoint and
|
||||
WRONG for a policy refusal: swallowing this turns the air-gap switch back
|
||||
into a suggestion.
|
||||
|
||||
**It derives from BaseException, not Exception, on purpose.** The first
|
||||
version of this class was a ``RuntimeError`` whose docstring asked every
|
||||
broad handler to re-raise it. Nothing did — there were zero re-raise sites
|
||||
in the package — and the refusal was swallowed by four reachable
|
||||
``except Exception`` blocks on the way out of a single ``/v1/messages``
|
||||
request, which returned 200 with the content uncompressed while the guard
|
||||
had fired twice. A convention that has to be re-applied at every future
|
||||
``except Exception`` in a 100k-line codebase is not a guarantee; the type
|
||||
hierarchy is. This is the same reason ``KeyboardInterrupt`` and
|
||||
``SystemExit`` sit outside ``Exception``: a policy decision is control
|
||||
flow, not a runtime error to degrade around.
|
||||
|
||||
Consequences to know about:
|
||||
|
||||
* ``except Exception`` no longer catches it anywhere — including in
|
||||
third-party code (httpx, the OTEL SDK, Starlette's error middleware).
|
||||
* ``except BaseException`` still does. Those are rare, they are enumerated
|
||||
by ``tests/test_offline_egress_chokepoint.py``'s broad-handler sweep, and
|
||||
each must either re-raise or carry a written reason.
|
||||
* Reaching this at request time means the operator changed the environment
|
||||
under a running proxy; the ordinary contradictory-configuration case is
|
||||
caught at startup by ``headroom/proxy/server.py``'s offline preflight,
|
||||
which exits 78 with an explanation instead.
|
||||
* Where a refusal SHOULD become a degradation, the boundary that owns the
|
||||
degradation says so explicitly. Fetching public model weights is not data
|
||||
leaving the box, so each optional model loader catches this and re-raises
|
||||
its own "model unavailable" error with the switch named — see
|
||||
``kompress_compressor._hf_artifact``, ``image/onnx_router._hf_artifact``
|
||||
and the ONNX embedder in ``memory/adapters/embedders.py``. Translating is
|
||||
fine; inheriting a translation from whichever ``except Exception`` happens
|
||||
to be in the stack is what this type prevents.
|
||||
"""
|
||||
|
||||
def __init__(self, purpose: str, destination: str | None = None) -> None:
|
||||
self.purpose = purpose
|
||||
self.destination = destination
|
||||
where = f" to {destination}" if destination else ""
|
||||
super().__init__(
|
||||
f"{OFFLINE_ENV} is set: refusing outbound network access for "
|
||||
f"{purpose}{where}. Unset {OFFLINE_ENV}, or turn off the feature "
|
||||
f"that needs this connection."
|
||||
)
|
||||
|
||||
|
||||
def guard_egress(purpose: str, destination: str | None = None) -> None:
|
||||
"""The single chokepoint every Headroom-initiated egress path must call.
|
||||
|
||||
Raise :class:`OfflineEgressBlocked` when ``HEADROOM_OFFLINE`` selects
|
||||
offline operation; return silently otherwise.
|
||||
|
||||
Call it BEFORE the socket exists — before constructing the client, not
|
||||
just before the request — so a pooled/keep-alive connection is never even
|
||||
opened. ``purpose`` and ``destination`` land verbatim in the message, so
|
||||
an operator who trips this learns which feature to turn off.
|
||||
|
||||
Why raise instead of returning a no-op result: a silent skip is
|
||||
indistinguishable from success at the call site, so a future refactor can
|
||||
quietly reintroduce egress and nothing fails. A loud, named exception is
|
||||
what makes the meta-test in ``tests/test_offline_egress_chokepoint.py``
|
||||
able to assert "every egress path is behind this or allowlisted".
|
||||
|
||||
Deliberately NOT exempted:
|
||||
|
||||
* Loopback / in-cluster destinations. The guard cannot reliably tell an
|
||||
in-cluster collector from an internet host (DNS, proxies and sidecars
|
||||
all blur it), and ``HEADROOM_OFFLINE`` is documented as "no outbound
|
||||
traffic". Paths that only ever talk to ``127.0.0.1`` — the readiness
|
||||
probes, the proxy's own ``/livez`` check — simply do not call the guard
|
||||
and are allowlisted by name in the meta-test instead.
|
||||
* The proxy's own request forwarding to the caller's configured upstream.
|
||||
That is the caller's traffic, not Headroom phoning home; an air-gapped
|
||||
deployment points it at an on-prem endpoint and still needs it to work.
|
||||
"""
|
||||
if is_offline():
|
||||
raise OfflineEgressBlocked(purpose, destination)
|
||||
|
||||
|
||||
# Purposes already reported, so a poller that runs every 30s says it once at
|
||||
# WARNING and then keeps quiet. Module-level rather than per-object because
|
||||
# several of these pollers are recreated per poll.
|
||||
_REPORTED_REFUSALS: set[str] = set()
|
||||
|
||||
|
||||
def note_refusal(blocked: OfflineEgressBlocked, log: logging.Logger) -> str:
|
||||
"""Report an air-gap refusal on a background path and return its message.
|
||||
|
||||
Background pollers and other fire-and-forget tasks cannot let a
|
||||
``BaseException`` escape — an unhandled task exception is a traceback in
|
||||
the log, not an explanation, and asyncio may not surface it until the
|
||||
task is garbage collected. They also must not swallow the refusal
|
||||
silently, because "the subscription window stopped updating" with no
|
||||
stated cause is the exact confusion this switch was supposed to end.
|
||||
|
||||
So: catch :class:`OfflineEgressBlocked` specifically, call this, and stop.
|
||||
The first refusal for a given purpose is logged at WARNING with the switch
|
||||
named; later ones drop to DEBUG so a 30-second poller does not flood the
|
||||
log with a decision the operator already made.
|
||||
"""
|
||||
message = str(blocked)
|
||||
if blocked.purpose in _REPORTED_REFUSALS:
|
||||
log.debug("%s", message)
|
||||
else:
|
||||
_REPORTED_REFUSALS.add(blocked.purpose)
|
||||
log.warning("%s", message)
|
||||
return message
|
||||
|
||||
@@ -8,6 +8,8 @@ import os
|
||||
import sys
|
||||
from typing import Any
|
||||
|
||||
from headroom.offline import guard_egress
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Override for the CPU memory-arena default below: "1"/"true" forces the
|
||||
@@ -121,6 +123,10 @@ def hf_hub_download_local_first(
|
||||
Absolute path to the local cached file.
|
||||
|
||||
Raises:
|
||||
OfflineEgressBlocked: when ``HEADROOM_OFFLINE`` is set and the file is
|
||||
not already cached. A cache HIT still succeeds — the guard sits on
|
||||
the network fallback only, so a pre-seeded air-gapped deployment
|
||||
keeps working, which is the whole point of pre-seeding.
|
||||
Any exception raised by ``hf_hub_download`` on a genuine download failure,
|
||||
or the local-lookup error when ``allow_network`` is ``False`` and the
|
||||
file is not cached.
|
||||
@@ -135,6 +141,17 @@ def hf_hub_download_local_first(
|
||||
except (LocalEntryNotFoundError, EntryNotFoundError, OSError):
|
||||
if not allow_network:
|
||||
raise
|
||||
# Air-gap chokepoint: this is the Python half of the HuggingFace fetch
|
||||
# the Rust core already guards. ``apply_offline_env`` sets
|
||||
# ``HF_HUB_OFFLINE=1``, but only with ``setdefault`` (an explicit
|
||||
# ``HF_HUB_OFFLINE=0`` wins) and only inside the proxy process — the CLI
|
||||
# and library entry points never call it, so the flag alone is not the
|
||||
# guarantee. Guarding here is, and it costs nothing on the cache-hit
|
||||
# path above.
|
||||
guard_egress(
|
||||
f"HuggingFace download of {repo_id}/{filename}",
|
||||
"huggingface.co",
|
||||
)
|
||||
return str(hf_hub_download(repo_id, filename, revision=revision))
|
||||
|
||||
|
||||
@@ -157,6 +174,76 @@ def hf_entry_known_absent(repo_id: str, filename: str, *, revision: str | None =
|
||||
return result is _CACHED_NO_EXIST
|
||||
|
||||
|
||||
def _hf_not_cached_errors() -> tuple[type[BaseException], ...]:
|
||||
"""Errors a ``local_files_only=True`` load raises on a plain cache miss.
|
||||
|
||||
Transformers and sentence-transformers both surface a miss as ``OSError``;
|
||||
huggingface_hub's own lookup errors are listed explicitly in case a loader
|
||||
lets one through unwrapped.
|
||||
"""
|
||||
try:
|
||||
from huggingface_hub.errors import EntryNotFoundError, LocalEntryNotFoundError
|
||||
except Exception: # pragma: no cover - huggingface_hub ships with every HF loader
|
||||
return (OSError,)
|
||||
return (LocalEntryNotFoundError, EntryNotFoundError, OSError)
|
||||
|
||||
|
||||
def hf_from_pretrained_local_first(
|
||||
loader: Any,
|
||||
name_or_path: str,
|
||||
*,
|
||||
purpose: str,
|
||||
allow_network: bool = True,
|
||||
**kwargs: Any,
|
||||
) -> Any:
|
||||
"""Load a HuggingFace model/tokenizer/processor, local cache first.
|
||||
|
||||
The ``from_pretrained`` twin of :func:`hf_hub_download_local_first`, for
|
||||
every loader that takes ``local_files_only`` — ``AutoModel.from_pretrained``,
|
||||
``AutoTokenizer.from_pretrained``, ``AutoProcessor.from_pretrained``,
|
||||
``SentenceTransformer`` and friends. ``loader`` is that callable, passed
|
||||
uncalled; ``kwargs`` are forwarded to both attempts.
|
||||
|
||||
1. ``local_files_only=True``: a pure cache lookup that cannot open a socket.
|
||||
A warm (or pre-seeded air-gapped) cache loads here and never touches the
|
||||
network, which also skips the Hub re-validation round-trips a plain
|
||||
``from_pretrained`` makes on every load.
|
||||
2. On a cache miss, :func:`guard_egress` runs BEFORE the remote attempt.
|
||||
|
||||
Why the guard and not ``HF_HUB_OFFLINE``: ``apply_offline_env`` sets it with
|
||||
``setdefault``, so an explicit ``HF_HUB_OFFLINE=0`` wins, and only the proxy
|
||||
process calls it at all. Under that configuration huggingface_hub's offline
|
||||
constant is false even with ``TRANSFORMERS_OFFLINE=1``, and a bare
|
||||
``from_pretrained`` downloads. ``HEADROOM_OFFLINE`` is the master switch;
|
||||
this guard is what makes it one for model loaders.
|
||||
|
||||
Args:
|
||||
loader: The ``from_pretrained``-style callable.
|
||||
name_or_path: Hub repo id or local directory.
|
||||
purpose: What is being loaded, for the refusal message ("SigLIP model").
|
||||
allow_network: When ``False`` a cache miss re-raises the local-lookup
|
||||
error instead of falling back to a download.
|
||||
**kwargs: Forwarded to ``loader`` (``revision=``, ``device=``, ...).
|
||||
|
||||
Raises:
|
||||
OfflineEgressBlocked: ``HEADROOM_OFFLINE`` is set and the model is not
|
||||
cached. Callers translate this into their own "model unavailable".
|
||||
The loader's own error on a cache miss when ``allow_network`` is
|
||||
``False``, when ``name_or_path`` is a local directory (nothing remote
|
||||
to fall back to), or when the remote attempt itself fails.
|
||||
"""
|
||||
kwargs.pop("local_files_only", None)
|
||||
try:
|
||||
return loader(name_or_path, local_files_only=True, **kwargs)
|
||||
except _hf_not_cached_errors():
|
||||
# A local directory that failed to load has no remote to fall back to;
|
||||
# surface the real error rather than a download attempt or a refusal.
|
||||
if not allow_network or os.path.isdir(name_or_path):
|
||||
raise
|
||||
guard_egress(f"HuggingFace download of {purpose} ({name_or_path})", "huggingface.co")
|
||||
return loader(name_or_path, local_files_only=False, **kwargs)
|
||||
|
||||
|
||||
def create_cpu_session_options(
|
||||
ort: Any,
|
||||
*,
|
||||
|
||||
+109
-9
@@ -97,7 +97,13 @@ from headroom.observability import (
|
||||
shutdown_headroom_tracing,
|
||||
shutdown_otel_metrics,
|
||||
)
|
||||
from headroom.offline import apply_offline_env, is_offline
|
||||
from headroom.offline import (
|
||||
OFFLINE_ENV,
|
||||
OfflineEgressBlocked,
|
||||
apply_offline_env,
|
||||
guard_egress,
|
||||
is_offline,
|
||||
)
|
||||
from headroom.pipeline import PipelineExtensionManager, PipelineStage
|
||||
from headroom.providers.proxy_routes import register_provider_routes
|
||||
from headroom.providers.registry import (
|
||||
@@ -698,6 +704,89 @@ def _check_rust_core() -> tuple[str, str | None]:
|
||||
return ("loaded", None)
|
||||
|
||||
|
||||
def _refuse_offline_egress(blocked: OfflineEgressBlocked, feature: str, fix: str) -> None:
|
||||
"""Turn a startup-time air-gap refusal into a clean exit, never a traceback.
|
||||
|
||||
``OfflineEgressBlocked`` derives from ``BaseException``, so nothing in the
|
||||
lifespan's own ``except Exception`` will catch it and it would otherwise
|
||||
escape uvicorn as an unhandled error — killing a proxy that had been
|
||||
serving traffic, with a stack trace instead of an explanation, and before
|
||||
``app.state.startup_error`` exists to record why. ``_check_rust_core``
|
||||
above already settled the shape for a startup refusal: say what is wrong,
|
||||
say how to fix it, exit 78 (``EX_CONFIG``).
|
||||
"""
|
||||
msg = (
|
||||
f"FATAL: {feature} needs outbound network access, but "
|
||||
f"{OFFLINE_ENV}=1 forbids it.\n"
|
||||
f" refused: {blocked}\n"
|
||||
f" fix: {fix}\n"
|
||||
)
|
||||
logger.error(
|
||||
"event=offline_egress_refused feature=%r reason=%r action=exit_78",
|
||||
feature,
|
||||
str(blocked),
|
||||
)
|
||||
print(msg, file=sys.stderr, flush=True)
|
||||
sys.exit(_EXIT_CONFIG)
|
||||
|
||||
|
||||
def _configure_observability_or_refuse() -> None:
|
||||
"""Install the metric/trace exporters, refusing cleanly under an air-gap.
|
||||
|
||||
Both exporters call ``guard_egress`` before they build a client, so under
|
||||
``HEADROOM_OFFLINE`` they raise rather than dial. Catching that here is what
|
||||
keeps the refusal a refusal: these two calls sit OUTSIDE ``lifespan``'s own
|
||||
``try``/``except Exception``, and ``OfflineEgressBlocked`` is a
|
||||
``BaseException``, so an uncaught one escapes uvicorn as an unhandled error
|
||||
— a proxy that had been serving traffic dies on a traceback with
|
||||
``app.state.startup_error`` never set.
|
||||
"""
|
||||
_preflight_offline_egress()
|
||||
try:
|
||||
configure_otel_metrics(OTelMetricsConfig.from_env(default_service_name="headroom-proxy"))
|
||||
except OfflineEgressBlocked as blocked:
|
||||
_refuse_offline_egress(
|
||||
blocked,
|
||||
"OTLP metric export (HEADROOM_OTEL_METRICS_ENABLED)",
|
||||
"set HEADROOM_OTEL_METRICS_EXPORTER=console or scrape /metrics "
|
||||
"(both stay on-box), or unset HEADROOM_OTEL_METRICS_ENABLED",
|
||||
)
|
||||
try:
|
||||
configure_langfuse_tracing(
|
||||
LangfuseTracingConfig.from_env(default_service_name="headroom-proxy")
|
||||
)
|
||||
except OfflineEgressBlocked as blocked:
|
||||
_refuse_offline_egress(
|
||||
blocked,
|
||||
"Langfuse OTLP trace export (HEADROOM_LANGFUSE_ENABLED)",
|
||||
"unset HEADROOM_LANGFUSE_ENABLED; Langfuse ingestion has no on-box mode",
|
||||
)
|
||||
|
||||
|
||||
def _preflight_offline_egress() -> None:
|
||||
"""Refuse, at startup, any configuration that asks for egress while offline.
|
||||
|
||||
Runs before the observability exporters are configured and before the
|
||||
compressors are preloaded. Without it the same contradiction surfaces later
|
||||
and worse: the OTLP exporter raised out of ``lifespan`` with a raw
|
||||
traceback, and the remote Kompress endpoint was not noticed until a request
|
||||
reached for the compressor.
|
||||
"""
|
||||
if not is_offline():
|
||||
return
|
||||
endpoint = os.environ.get("HEADROOM_KOMPRESS_ENDPOINT", "").strip()
|
||||
if endpoint:
|
||||
try:
|
||||
guard_egress("remote Kompress inference", endpoint)
|
||||
except OfflineEgressBlocked as blocked:
|
||||
_refuse_offline_egress(
|
||||
blocked,
|
||||
"remote Kompress compression (HEADROOM_KOMPRESS_ENDPOINT)",
|
||||
"unset HEADROOM_KOMPRESS_ENDPOINT to use the local Kompress "
|
||||
"model, or unset " + OFFLINE_ENV + " if this box is not air-gapped",
|
||||
)
|
||||
|
||||
|
||||
# Compression pipeline timeout in seconds
|
||||
|
||||
|
||||
@@ -3104,15 +3193,29 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
|
||||
# Air-gap master switch. Propagate config.offline to the env so the
|
||||
# env-based egress predicates (telemetry, update check, license) all honor
|
||||
# it, force HF/transformers offline before any model code loads, and
|
||||
# announce that every outbound path is disabled.
|
||||
# it, force HF/transformers offline before any model code loads, and say
|
||||
# what is covered.
|
||||
#
|
||||
# The banner names the exception rather than claiming a whole-process kill
|
||||
# switch, and the exception is permanent rather than a to-do: a proxy that
|
||||
# refused to forward the caller's request to the caller's own upstream
|
||||
# would not be a proxy. Every connection Headroom itself initiates IS
|
||||
# refused; tests/test_offline_egress_chokepoint.py fails the build if a new
|
||||
# egress path appears that is neither guarded nor one of the four written
|
||||
# exceptions. TestDocsMatchTheGuarantee scans this file for the absolute
|
||||
# phrasings and fails while any remain.
|
||||
if config.offline:
|
||||
os.environ.setdefault("HEADROOM_OFFLINE", "1")
|
||||
if is_offline():
|
||||
apply_offline_env()
|
||||
logger.warning(
|
||||
"event=proxy_offline_mode air-gap active — all outbound egress disabled "
|
||||
"(telemetry, update check, license reporter, HuggingFace downloads)"
|
||||
"event=proxy_offline_mode air-gap active — every connection Headroom "
|
||||
"initiates is refused (telemetry, update check, license reporter, "
|
||||
"model/tokenizer/binary/dataset downloads, remote Kompress, "
|
||||
"OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI "
|
||||
"embedders, Headroom Cloud compression). Still allowed on purpose: "
|
||||
"forwarding your requests to the upstream you configured, "
|
||||
"operator-configured local endpoints, loopback health probes."
|
||||
)
|
||||
|
||||
proxy = HeadroomProxy(config)
|
||||
@@ -3231,10 +3334,7 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
_rust_core_error,
|
||||
)
|
||||
|
||||
configure_otel_metrics(OTelMetricsConfig.from_env(default_service_name="headroom-proxy"))
|
||||
configure_langfuse_tracing(
|
||||
LangfuseTracingConfig.from_env(default_service_name="headroom-proxy")
|
||||
)
|
||||
_configure_observability_or_refuse()
|
||||
|
||||
app.state.started_at = time.time()
|
||||
app.state.ready = False
|
||||
|
||||
@@ -33,6 +33,8 @@ from dataclasses import asdict, dataclass, field
|
||||
from typing import Any
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from headroom.offline import OfflineEgressBlocked, guard_egress, note_refusal
|
||||
|
||||
logger = logging.getLogger("headroom.proxy")
|
||||
|
||||
DOCS_URL = "https://docs.headroomlabs.ai/docs/corporate-networks"
|
||||
@@ -302,6 +304,14 @@ def probe_presented_chain(
|
||||
def _probe_uncached(host: str, port: int, *, timeout: float, allow_private: bool) -> ChainInfo:
|
||||
proxy = _proxy_for(host)
|
||||
info = ChainInfo(host=host, port=port, reachable=False, via_proxy=proxy)
|
||||
# A second connection Headroom opens on its own, so HEADROOM_OFFLINE refuses
|
||||
# it. It only enriches a diagnostic: on the request path the refusal becomes
|
||||
# "no chain to report", and the TLS explanation is still given without it.
|
||||
try:
|
||||
guard_egress("TLS certificate-chain probe", f"{host}:{port}")
|
||||
except OfflineEgressBlocked as blocked:
|
||||
info.error = f"skipped: {note_refusal(blocked, logger)}"
|
||||
return info
|
||||
# Certificate-inspection handshake only: it reads the presented chain and
|
||||
# closes without sending a byte, so it deliberately skips verification (the
|
||||
# chain is what we are diagnosing). It still refuses TLS < 1.2, and it only
|
||||
@@ -459,7 +469,13 @@ def probe_endpoint(name: str, url: str, *, timeout: float = 8.0) -> EndpointRepo
|
||||
Reports the presented chain (and any inspection vendor), whether the chain
|
||||
verifies under the policy Headroom will use, and whether the response is a
|
||||
gateway block page instead of the service.
|
||||
|
||||
Raises :class:`OfflineEgressBlocked` under ``HEADROOM_OFFLINE`` before
|
||||
either connection is attempted; ``headroom doctor`` reports that as a
|
||||
skipped check.
|
||||
"""
|
||||
guard_egress("doctor --network endpoint check", url)
|
||||
|
||||
import httpx
|
||||
|
||||
from headroom.proxy.ssl_context import build_httpx_verify
|
||||
|
||||
@@ -30,6 +30,8 @@ import logging
|
||||
import os
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from headroom.offline import OfflineEgressBlocked, guard_egress
|
||||
|
||||
from .base import RelevanceScore, RelevanceScorer
|
||||
|
||||
# numpy is an optional dependency - import lazily
|
||||
@@ -57,6 +59,57 @@ if TYPE_CHECKING:
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _load_text_embedding(kwargs: dict[str, str]) -> TextEmbedding:
|
||||
"""Build a fastembed ``TextEmbedding``, cache first and network second.
|
||||
|
||||
``onnx_runtime.hf_hub_download_local_first`` can hang the air-gap guard on
|
||||
an explicit ``local_files_only=True`` attempt because ``hf_hub_download``
|
||||
takes that flag. fastembed's constructor does not, so the same cache-hit /
|
||||
network-fallback split has to be made here: try the load with the whole
|
||||
HuggingFace stack forced offline, and only reach for the guard when that
|
||||
fails, which is exactly the case where a socket would have been opened.
|
||||
|
||||
This keeps the behaviour an air-gapped deployment actually wants — a
|
||||
pre-seeded cache still loads under ``HEADROOM_OFFLINE`` — while making a
|
||||
cold cache refuse instead of quietly dialling huggingface.co. A plain
|
||||
unconditional guard would have broken the pre-seeded case, which is the
|
||||
reason this path was left unguarded before.
|
||||
|
||||
The ``HF_HUB_OFFLINE`` flip is process-global for the duration of the
|
||||
call. That is acceptable here because loading is one-shot and memoised by
|
||||
the caller, and because the value it forces is *stricter* than whatever it
|
||||
replaces, so a concurrent HuggingFace call can only be refused locally,
|
||||
never sent somewhere it would not otherwise go.
|
||||
"""
|
||||
from fastembed import TextEmbedding
|
||||
from huggingface_hub import constants as hf_constants
|
||||
|
||||
# huggingface_hub reads HF_HUB_OFFLINE once, at import, into
|
||||
# ``constants.HF_HUB_OFFLINE``; if it was imported before this call the
|
||||
# env var alone changes nothing. Force both.
|
||||
previous = os.environ.get("HF_HUB_OFFLINE")
|
||||
previous_constant = hf_constants.HF_HUB_OFFLINE
|
||||
os.environ["HF_HUB_OFFLINE"] = "1"
|
||||
hf_constants.HF_HUB_OFFLINE = True
|
||||
try:
|
||||
return TextEmbedding(**kwargs)
|
||||
except Exception as cache_miss: # noqa: BLE001 - any local-lookup failure
|
||||
logger.debug("fastembed cache lookup failed, falling back to network: %s", cache_miss)
|
||||
finally:
|
||||
hf_constants.HF_HUB_OFFLINE = previous_constant
|
||||
if previous is None:
|
||||
os.environ.pop("HF_HUB_OFFLINE", None)
|
||||
else:
|
||||
os.environ["HF_HUB_OFFLINE"] = previous
|
||||
|
||||
guard_egress(
|
||||
f"fastembed weight download for {kwargs['model_name']}",
|
||||
"huggingface.co",
|
||||
)
|
||||
return TextEmbedding(**kwargs)
|
||||
|
||||
|
||||
# Default model name. Same string used by the Rust embedding scorer.
|
||||
DEFAULT_MODEL_NAME = "BAAI/bge-small-en-v1.5"
|
||||
|
||||
@@ -169,14 +222,24 @@ class EmbeddingScorer(RelevanceScorer):
|
||||
)
|
||||
|
||||
if self._model is None:
|
||||
from fastembed import TextEmbedding
|
||||
|
||||
revision = _pinned_revision(self.model_name)
|
||||
kwargs = {"model_name": self.model_name}
|
||||
if revision is not None:
|
||||
# fastembed forwards **kwargs to snapshot_download(revision=...).
|
||||
self._model = TextEmbedding(model_name=self.model_name, revision=revision)
|
||||
else:
|
||||
self._model = TextEmbedding(model_name=self.model_name)
|
||||
kwargs["revision"] = revision
|
||||
try:
|
||||
self._model = _load_text_embedding(kwargs)
|
||||
except OfflineEgressBlocked as blocked:
|
||||
# Translate at the boundary that owns the degradation, the way
|
||||
# the Kompress and ONNX loaders do. Public model weights are
|
||||
# not data leaving the box, so what the caller needs to hear
|
||||
# is "this model is not available here and why", not an
|
||||
# air-gap type it has never seen.
|
||||
raise RuntimeError(
|
||||
f"Embedding model {self.model_name!r} is unavailable: {blocked}. "
|
||||
"Pre-seed the fastembed/HuggingFace cache on this host, or "
|
||||
"use the BM25-only scorer."
|
||||
) from None
|
||||
return self._model
|
||||
|
||||
def _encode(self, texts: list[str]):
|
||||
|
||||
@@ -21,6 +21,7 @@ from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
from headroom.offline import OfflineEgressBlocked, guard_egress, note_refusal
|
||||
from headroom.subscription.models import SubscriptionSnapshot
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -106,6 +107,10 @@ class SubscriptionClient:
|
||||
}
|
||||
|
||||
try:
|
||||
# First statement in the block that opens the socket, so the guard
|
||||
# provably runs before the client is built (which is also what the
|
||||
# egress meta-test's dominance rule checks).
|
||||
guard_egress("Anthropic subscription usage polling", _USAGE_URL)
|
||||
async with httpx.AsyncClient(timeout=self._timeout) as client:
|
||||
resp = await client.get(_USAGE_URL, headers=headers)
|
||||
|
||||
@@ -123,6 +128,14 @@ class SubscriptionClient:
|
||||
data: dict[str, Any] = resp.json()
|
||||
return SubscriptionSnapshot.from_api_response(data, token=resolved)
|
||||
|
||||
except OfflineEgressBlocked as blocked:
|
||||
# Refuse, but as a sentence in the log rather than an unhandled
|
||||
# BaseException inside a background poll. The caller reads `None`
|
||||
# as "no window data", which is the truth here: an air-gapped box
|
||||
# cannot know the subscription window, and pretending otherwise
|
||||
# would be worse than saying so once and moving on.
|
||||
note_refusal(blocked, logger)
|
||||
return None
|
||||
except httpx.TimeoutException:
|
||||
logger.debug("Timeout fetching Anthropic subscription window")
|
||||
return None
|
||||
|
||||
@@ -50,6 +50,7 @@ from threading import Lock
|
||||
|
||||
import httpx
|
||||
|
||||
from headroom.offline import OfflineEgressBlocked, guard_egress, is_offline, note_refusal
|
||||
from headroom.subscription.base import QuotaTracker
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -450,6 +451,7 @@ def _build_usage_headers(request_headers: dict[str, str]) -> dict[str, str] | No
|
||||
async def _fetch_and_store_usage(url: str, headers: dict[str, str]) -> None:
|
||||
state = get_codex_rate_limit_state()
|
||||
try:
|
||||
guard_egress("Codex rate-limit polling", url)
|
||||
async with httpx.AsyncClient(timeout=_USAGE_POLL_TIMEOUT_S) as client:
|
||||
resp = await client.get(url, headers=headers)
|
||||
if resp.status_code == 200:
|
||||
@@ -459,6 +461,12 @@ async def _fetch_and_store_usage(url: str, headers: dict[str, str]) -> None:
|
||||
logger.debug("codex usage poll: 200 but no usable rate-limit data")
|
||||
else:
|
||||
logger.debug("codex usage poll: HTTP %s", resp.status_code)
|
||||
except OfflineEgressBlocked as blocked:
|
||||
# Background task: report the refusal as a sentence. Letting a
|
||||
# BaseException out of a bare ``create_task`` surfaces as "Task
|
||||
# exception was never retrieved" whenever the GC gets round to it,
|
||||
# which is a traceback with no explanation in it.
|
||||
note_refusal(blocked, logger)
|
||||
except Exception as exc: # pragma: no cover - network/JSON defensive
|
||||
logger.debug("codex usage poll failed: %s", exc)
|
||||
finally:
|
||||
@@ -488,6 +496,12 @@ def maybe_schedule_usage_poll(
|
||||
headers = _build_usage_headers(request_headers)
|
||||
if headers is None:
|
||||
return False
|
||||
if is_offline():
|
||||
# Cheap pre-check so an air-gapped proxy does not spawn (and log) a
|
||||
# doomed task on every single Codex request. The guard inside
|
||||
# _fetch_and_store_usage is still the thing that makes the refusal
|
||||
# true; this only keeps the log and the event loop quiet.
|
||||
return False
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
except RuntimeError:
|
||||
|
||||
@@ -41,6 +41,7 @@ from dataclasses import dataclass, field
|
||||
from threading import Lock
|
||||
from typing import Any
|
||||
|
||||
from headroom.offline import OfflineEgressBlocked, guard_egress, note_refusal
|
||||
from headroom.subscription.base import QuotaTracker
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -314,6 +315,7 @@ class _CopilotQuotaTracker(QuotaTracker):
|
||||
}
|
||||
|
||||
try:
|
||||
guard_egress("GitHub Copilot quota polling", _GITHUB_API_BASE)
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.get(
|
||||
url, headers=headers, timeout=aiohttp.ClientTimeout(total=10)
|
||||
@@ -333,6 +335,13 @@ class _CopilotQuotaTracker(QuotaTracker):
|
||||
return
|
||||
|
||||
data = await resp.json()
|
||||
except OfflineEgressBlocked as blocked:
|
||||
# Say it twice, in the two places an operator looks: once in the
|
||||
# log, and once in the quota panel's own error slot, so the empty
|
||||
# panel carries its reason instead of looking like a broken poll.
|
||||
with self._lock:
|
||||
self._state.last_error = note_refusal(blocked, logger)
|
||||
return
|
||||
except Exception as exc:
|
||||
with self._lock:
|
||||
self._state.last_error = str(exc)
|
||||
|
||||
@@ -226,7 +226,7 @@ def _load_tokenizer(tokenizer_name: str):
|
||||
# vetted, not a variant the caller chose.
|
||||
tokenizer_name = repo
|
||||
|
||||
from headroom.offline import apply_offline_env, is_offline
|
||||
from headroom.offline import apply_offline_env, guard_egress, is_offline
|
||||
|
||||
offline = is_offline()
|
||||
if offline:
|
||||
@@ -268,6 +268,10 @@ def _load_tokenizer(tokenizer_name: str):
|
||||
|
||||
def _download() -> None:
|
||||
try:
|
||||
# Defence in depth behind the is_offline() return above: the switch
|
||||
# could flip between that check and this thread, and the guard is
|
||||
# what the egress meta-test recognises as dominating the fetch.
|
||||
guard_egress(f"HuggingFace tokenizer download ({tokenizer_name})", "huggingface.co")
|
||||
result.append(
|
||||
AutoTokenizer.from_pretrained(
|
||||
tokenizer_name,
|
||||
|
||||
@@ -63,6 +63,7 @@ from ..config import (
|
||||
is_tool_excluded,
|
||||
unwrap_tool_call,
|
||||
)
|
||||
from ..offline import OfflineEgressBlocked
|
||||
from ..parser import CCR_RETRIEVAL_MARKER_RE
|
||||
from ..tokenizer import Tokenizer
|
||||
from ..tokenizers.base import count_content_blocks
|
||||
@@ -1185,7 +1186,7 @@ def _detect_content(content: str) -> DetectionResult:
|
||||
# "json_array"); translate to the Python `ContentType` enum so
|
||||
# downstream mapping keys match.
|
||||
content_type = ContentType(rust_result.content_type)
|
||||
except (KeyboardInterrupt, SystemExit, GeneratorExit):
|
||||
except (KeyboardInterrupt, SystemExit, GeneratorExit, OfflineEgressBlocked):
|
||||
raise
|
||||
except BaseException as exc: # noqa: BLE001
|
||||
# A native Rust panic surfaces as pyo3_runtime.PanicException, which
|
||||
@@ -1194,7 +1195,9 @@ def _detect_content(content: str) -> DetectionResult:
|
||||
# (panic, or an unrecognized content-type tag) degrades to the
|
||||
# pure-Python detector instead of aborting the request. See #1123.
|
||||
# Guard: don't swallow cancellation/control-flow BaseExceptions such
|
||||
# as asyncio.CancelledError — keep them propagating.
|
||||
# as asyncio.CancelledError — keep them propagating. OfflineEgressBlocked
|
||||
# is in that list for the same reason: it is a policy refusal, not a
|
||||
# detector failure, and degrading it here would hide the air-gap switch.
|
||||
if isinstance(exc, asyncio.CancelledError):
|
||||
raise
|
||||
if isinstance(exc, TimeoutError):
|
||||
|
||||
@@ -28,11 +28,13 @@ from functools import lru_cache
|
||||
from typing import Any, Literal
|
||||
|
||||
from ..config import TransformResult
|
||||
from ..offline import OFFLINE_ENV, OfflineEgressBlocked
|
||||
from ..onnx_runtime import (
|
||||
ONNX_CPU_ARENA_ENV,
|
||||
_resolve_revision,
|
||||
create_cpu_session_options,
|
||||
hf_entry_known_absent,
|
||||
hf_from_pretrained_local_first,
|
||||
hf_hub_download_local_first,
|
||||
trim_process_heap,
|
||||
)
|
||||
@@ -220,6 +222,70 @@ class KompressModelNotCached(RuntimeError):
|
||||
"""
|
||||
|
||||
|
||||
def _hf_artifact(model_id: str, filename: str, *, allow_network: bool) -> str:
|
||||
"""``hf_hub_download_local_first`` with the air-gap refusal translated.
|
||||
|
||||
``OfflineEgressBlocked`` is a ``BaseException`` so that no ``except
|
||||
Exception:`` can silently degrade an air-gap refusal into "that feature
|
||||
stopped working". At THIS boundary that degradation is the correct
|
||||
behaviour and is written down rather than inherited: a model download is
|
||||
not data leaving the box, every caller below already handles
|
||||
"the model is not available locally", and failing a user's request because
|
||||
an optional compressor could not fetch public weights would punish the
|
||||
request for a decision the operator made about the host.
|
||||
|
||||
So the refusal is logged with the switch named and re-raised as
|
||||
:class:`KompressModelNotCached`, which is exactly what it means: under
|
||||
``HEADROOM_OFFLINE`` the only artifacts that will ever be available are the
|
||||
cached ones. The original refusal is chained, so it is still in the
|
||||
traceback.
|
||||
"""
|
||||
try:
|
||||
return hf_hub_download_local_first(model_id, filename, allow_network=allow_network)
|
||||
except OfflineEgressBlocked as blocked:
|
||||
logger.warning(
|
||||
"Kompress: %r for %s is not cached and %s forbids fetching it (%s). "
|
||||
"Compression falls back to the non-ML path.",
|
||||
filename,
|
||||
model_id,
|
||||
OFFLINE_ENV,
|
||||
blocked,
|
||||
)
|
||||
raise KompressModelNotCached(model_id) from blocked
|
||||
|
||||
|
||||
def _hf_pretrained(
|
||||
loader: Any, name: str, *, purpose: str, allow_download: bool, **kwargs: Any
|
||||
) -> Any:
|
||||
"""``hf_from_pretrained_local_first`` with Kompress's two translations.
|
||||
|
||||
The ``from_pretrained`` counterpart of :func:`_hf_artifact`: cache first,
|
||||
``guard_egress`` before any remote attempt, and both "not cached while
|
||||
downloads are disallowed" and "not cached while ``HEADROOM_OFFLINE`` is set"
|
||||
surface as :class:`KompressModelNotCached` — the error every Kompress caller
|
||||
already treats as "use the non-ML path". A genuine remote failure while
|
||||
downloads are allowed propagates unchanged.
|
||||
"""
|
||||
try:
|
||||
return hf_from_pretrained_local_first(
|
||||
loader, name, purpose=purpose, allow_network=allow_download, **kwargs
|
||||
)
|
||||
except OfflineEgressBlocked as blocked:
|
||||
logger.warning(
|
||||
"Kompress: the %s (%s) is not cached and %s forbids fetching it (%s). "
|
||||
"Compression falls back to the non-ML path.",
|
||||
purpose,
|
||||
name,
|
||||
OFFLINE_ENV,
|
||||
blocked,
|
||||
)
|
||||
raise KompressModelNotCached(name) from blocked
|
||||
except _NOT_CACHED_ERRORS as exc:
|
||||
if allow_download:
|
||||
raise
|
||||
raise KompressModelNotCached(name) from exc
|
||||
|
||||
|
||||
# Model cache: model_id -> (model, tokenizer, backend)
|
||||
# Supports multiple models loaded simultaneously.
|
||||
_kompress_cache: dict[str, tuple[Any, Any, str]] = {}
|
||||
@@ -611,11 +677,14 @@ def _get_model_class() -> type:
|
||||
super().__init__()
|
||||
# Same pin as the tokenizer: the fine-tuned heads were trained on
|
||||
# this encoder snapshot. Unknown repos resolve to None (floating).
|
||||
self.encoder = AutoModel.from_pretrained(
|
||||
# Cache first, air-gap guard before any download (see _hf_pretrained).
|
||||
self.encoder = _hf_pretrained(
|
||||
AutoModel.from_pretrained,
|
||||
model_name,
|
||||
purpose="Kompress PyTorch encoder",
|
||||
allow_download=allow_download,
|
||||
attn_implementation="eager",
|
||||
revision=_resolve_revision(model_name, revision),
|
||||
local_files_only=not allow_download,
|
||||
)
|
||||
hidden_size = self.encoder.config.hidden_size # 768
|
||||
|
||||
@@ -750,12 +819,19 @@ def _create_onnx_session(
|
||||
"""
|
||||
last_err: Exception | None = None
|
||||
cache_miss = False
|
||||
refused: KompressModelNotCached | None = None
|
||||
ort: Any = None
|
||||
for filename in _onnx_filename_candidates():
|
||||
try:
|
||||
onnx_path = hf_hub_download_local_first(
|
||||
model_id, filename, allow_network=allow_download
|
||||
onnx_path = _hf_artifact(
|
||||
model_id, filename, allow_network=allow_download and refused is None
|
||||
)
|
||||
except KompressModelNotCached as exc:
|
||||
# Only _hf_artifact's air-gap translation raises this here. It says
|
||||
# THIS candidate is not cached; a later one may be. Keep looking,
|
||||
# cache-only, so the refusal is logged once and never retried.
|
||||
refused = exc
|
||||
continue
|
||||
except Exception as exc:
|
||||
last_err = exc
|
||||
cache_miss = cache_miss or isinstance(exc, _NOT_CACHED_ERRORS)
|
||||
@@ -781,6 +857,8 @@ def _create_onnx_session(
|
||||
model_id,
|
||||
exc,
|
||||
)
|
||||
if refused is not None:
|
||||
raise refused
|
||||
if not allow_download and cache_miss:
|
||||
raise KompressModelNotCached(model_id) from last_err
|
||||
raise FileNotFoundError(
|
||||
@@ -860,8 +938,11 @@ def _load_modernbert_tokenizer(auto_tokenizer: Any, *, allow_download: bool) ->
|
||||
|
||||
Same files, same tokenizer, so the loaded object is identical; this only
|
||||
changes whether the Hub is consulted to confirm what is already on disk.
|
||||
Mirrors ``onnx_runtime.hf_hub_download_local_first``, which the ONNX half of
|
||||
this loader already uses.
|
||||
Goes through ``onnx_runtime.hf_from_pretrained_local_first`` (via
|
||||
:func:`_hf_pretrained`), the ``from_pretrained`` twin of the
|
||||
``hf_hub_download_local_first`` the ONNX half of this loader uses, so a cache
|
||||
miss under ``HEADROOM_OFFLINE`` is refused before any remote attempt even
|
||||
when ``HF_HUB_OFFLINE=0`` is set explicitly.
|
||||
|
||||
The load is pinned to the same immutable revision as the model artifacts
|
||||
(``onnx_runtime._PINNED_REVISIONS``): the ONNX export was produced against
|
||||
@@ -869,17 +950,12 @@ def _load_modernbert_tokenizer(auto_tokenizer: Any, *, allow_download: bool) ->
|
||||
the shipped weights without any code change. ``HEADROOM_HF_PIN=off`` floats
|
||||
it, exactly as for the weights.
|
||||
"""
|
||||
revision = _resolve_revision(_MODERNBERT_TOKENIZER_REPO, None)
|
||||
try:
|
||||
return auto_tokenizer.from_pretrained(
|
||||
_MODERNBERT_TOKENIZER_REPO, revision=revision, local_files_only=True
|
||||
)
|
||||
except _NOT_CACHED_ERRORS as exc:
|
||||
if not allow_download:
|
||||
raise KompressModelNotCached(_MODERNBERT_TOKENIZER_REPO) from exc
|
||||
# Genuine cache miss and downloading is permitted: fetch it at the pin.
|
||||
return auto_tokenizer.from_pretrained(
|
||||
_MODERNBERT_TOKENIZER_REPO, revision=revision, local_files_only=False
|
||||
return _hf_pretrained(
|
||||
auto_tokenizer.from_pretrained,
|
||||
_MODERNBERT_TOKENIZER_REPO,
|
||||
purpose="ModernBERT tokenizer",
|
||||
allow_download=allow_download,
|
||||
revision=_resolve_revision(_MODERNBERT_TOKENIZER_REPO, None),
|
||||
)
|
||||
|
||||
|
||||
@@ -954,7 +1030,7 @@ def _load_pytorch_weights(model: Any, model_id: str, *, allow_download: bool) ->
|
||||
HuggingFace Hub's own cache of confirmed-404 lookups.
|
||||
"""
|
||||
try:
|
||||
ckpt_path = hf_hub_download_local_first(model_id, "merged.pt", allow_network=allow_download)
|
||||
ckpt_path = _hf_artifact(model_id, "merged.pt", allow_network=allow_download)
|
||||
except _NOT_CACHED_ERRORS as exc:
|
||||
if not allow_download:
|
||||
if not hf_entry_known_absent(model_id, "merged.pt"):
|
||||
@@ -971,9 +1047,7 @@ def _load_pytorch_weights(model: Any, model_id: str, *, allow_download: bool) ->
|
||||
# merged.pt genuinely does not exist in this repo (confirmed by a
|
||||
# real network lookup, not just a cache miss) - fall back to the
|
||||
# plain format instead of treating it as a download failure.
|
||||
weights_path = hf_hub_download_local_first(
|
||||
model_id, "model.safetensors", allow_network=allow_download
|
||||
)
|
||||
weights_path = _hf_artifact(model_id, "model.safetensors", allow_network=allow_download)
|
||||
_load_plain_state_dict(model, weights_path, model_id)
|
||||
return
|
||||
raise
|
||||
@@ -1218,6 +1292,21 @@ def _background_download(model_id: str, device: str) -> None:
|
||||
logger.info("Kompress: downloading model %s in the background ...", model_id)
|
||||
_load_kompress(model_id, device, allow_download=True)
|
||||
logger.info("Kompress: background model download complete for %s", model_id)
|
||||
except OfflineEgressBlocked as blocked:
|
||||
# Explicit, because OfflineEgressBlocked is a BaseException and would
|
||||
# otherwise reach threading.excepthook as a bare traceback on every
|
||||
# air-gapped startup with a cold cache. This is a background *refresh*,
|
||||
# not the request path: the refusal is reported once, at WARNING, and
|
||||
# the compressor keeps whatever is already cached. Do NOT widen this to
|
||||
# `except Exception` — that is the shape the guard exists to defeat.
|
||||
_record_download_failure(model_id)
|
||||
logger.warning(
|
||||
"Kompress: background model download refused for %s (%s is set): %s. "
|
||||
"The model will only load if its artifacts are already cached.",
|
||||
model_id,
|
||||
OFFLINE_ENV,
|
||||
blocked,
|
||||
)
|
||||
except Exception as exc:
|
||||
_record_download_failure(model_id)
|
||||
logger.warning("Kompress: background model download failed for %s: %s", model_id, exc)
|
||||
@@ -1277,6 +1366,18 @@ def prefetch_kompress_artifacts(model_id: str = HF_MODEL_ID) -> bool:
|
||||
try:
|
||||
hf_hub_download_local_first(model_id, filename, allow_network=True)
|
||||
return True
|
||||
except OfflineEgressBlocked as blocked:
|
||||
# Same reasoning as _background_download: explicit because it is a
|
||||
# BaseException, reported rather than swallowed, and terminal for
|
||||
# the whole loop — every candidate would be refused for the same
|
||||
# reason, so retrying them just logs the same refusal four times.
|
||||
logger.warning(
|
||||
"Kompress: artifact prefetch refused for %s (%s is set): %s",
|
||||
model_id,
|
||||
OFFLINE_ENV,
|
||||
blocked,
|
||||
)
|
||||
return False
|
||||
except Exception as exc:
|
||||
logger.debug("Kompress prefetch: %r unavailable for %s: %s", filename, model_id, exc)
|
||||
return False
|
||||
|
||||
@@ -57,6 +57,7 @@ import logging
|
||||
|
||||
import httpx
|
||||
|
||||
from ..offline import guard_egress
|
||||
from .kompress_compressor import (
|
||||
KompressConfig,
|
||||
KompressResult,
|
||||
@@ -139,6 +140,13 @@ class RemoteKompressCompressor:
|
||||
# without needing a separate auth-scheme setting.
|
||||
if headers:
|
||||
self._headers.update(headers)
|
||||
# Air-gap chokepoint. HEADROOM_KOMPRESS_ENDPOINT points at a box that is
|
||||
# by definition off this host, so under HEADROOM_OFFLINE this class must
|
||||
# not exist at all. Guarding at construction (not at the POST) means the
|
||||
# connection pool is never created and the operator hears about the
|
||||
# contradictory configuration the first time the router reaches for
|
||||
# Kompress, rather than discovering it in a packet capture.
|
||||
guard_egress("remote Kompress inference", self._url)
|
||||
# httpx.Client is safe to share across the proxy's worker threads.
|
||||
self._client = httpx.Client(timeout=timeout)
|
||||
|
||||
@@ -212,6 +220,16 @@ class RemoteKompressCompressor:
|
||||
if n_words < max(_MIN_WORDS, self.config.min_input_words):
|
||||
return self._passthrough(content, n_words)
|
||||
|
||||
# Re-check outside the fail-open try. __init__ already refused if the
|
||||
# flag was set then, but the env can be flipped after this object was
|
||||
# cached on the ContentRouter (the router builds it once per instance
|
||||
# and reuses it for the process lifetime), and the security property we
|
||||
# owe is "no socket", not "no construction". It is outside the try on
|
||||
# purpose: the except below turns every exception into a silent
|
||||
# pass-through, and a policy refusal that degrades silently is exactly
|
||||
# the hole this guard closes.
|
||||
guard_egress("remote Kompress inference", self._url)
|
||||
|
||||
try:
|
||||
resp = self._client.post(
|
||||
self._url,
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user