mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
feat/https-compose-caddy
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
3fef15d376 |
fix(offline): keep litellm's pricing table local under HEADROOM_OFFLINE (#4242)
## Description
`HEADROOM_OFFLINE=1` still lets litellm reach GitHub. `import litellm`
downloads `model_prices_and_context_window.json` from
`raw.githubusercontent.com` during the import (three attempts) unless
`LITELLM_LOCAL_MODEL_COST_MAP` is `true`. Headroom imports litellm from
24 places, among them the pricing helpers, the LiteLLM provider and
backend, the proxy's cost and savings trackers, and both Headroom Cloud
integrations. With the air-gap switch on, the first of those imports
dialled out.
Nothing in the chokepoint suite caught it, because no Headroom-built
client is involved, so no `guard_egress` site can dominate it.
`TestHeadroomCloudCompressionOffline` only passed when an earlier test
had already imported litellm. Run on its own (`pytest
tests/test_offline_egress_chokepoint.py -k HeadroomCloud`), it fails on
`main` with `SocketOpened: ... ('raw.githubusercontent.com', 443)`.
The only lever is the env var, and litellm reads it once, during its own
import. So:
- `pin_litellm_cost_map_local()` in `headroom/offline.py` sets
`LITELLM_LOCAL_MODEL_COST_MAP=True` while `HEADROOM_OFFLINE` is on.
- Every litellm import in `headroom/` calls it first, in the same scope.
A pin only at package import would miss a switch turned on later:
`is_offline()` is read when checked, and
`create_app(ProxyConfig(offline=True))` sets it after `import headroom`.
- The pin is Headroom's, so it is undone: called with the switch off, it
restores the value it replaced if the variable still holds the pin. A
process that leaves offline mode before its first litellm import gets
the online pricing refresh back.
- It overwrites the variable rather than using `setdefault`. litellm
only accepts `true`, so an operator value of `1` (the form every
Headroom switch takes), `False` or an empty string would otherwise
reopen the fetch.
- A meta-test fails the build for any litellm import without a preceding
pin, the same pattern as the existing site scanner.
No tracking issue; found while chasing a test that only failed when run
on its own.
Review found a second import-time leak on the same path:
`headroom/integrations/__init__.py` imports the CrewAI integration
whenever CrewAI is installed, and `import crewai` requests a Scarf pixel
from `api.scarf.sh` on a background thread unless
`CREWAI_DISABLE_TELEMETRY` is `true`. So `import
headroom.integrations.litellm_callback` still dialled out under the
switch. `pin_crewai_telemetry_off()` handles it the same way, through a
shared `_OfflineEnvPin`, and the meta-test now covers `crewai` imports
too.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/offline.py`: new `pin_litellm_cost_map_local()` (a no-op
unless `HEADROOM_OFFLINE` is on), documented in the module's list of
refused egress.
- All 24 litellm import sites call it immediately before importing:
`pricing/`, `providers/` (litellm, anthropic, openai, cohere, google),
`backends/litellm.py`, `proxy/` (cost, savings_tracker, route_advice),
`integrations/litellm_callback.py`, `telemetry/session.py`,
`perf/analyzer.py`, `learn/analyzer.py`, `cli/` (evals, learn) and
`evals/`.
- `pin_crewai_telemetry_off()` forces `CREWAI_DISABLE_TELEMETRY=true`
while offline (restored the same way), called before the one `crewai`
import (`headroom/integrations/crewai/agents.py`). Both pins share
`_OfflineEnvPin`.
- `headroom/__init__.py` also calls it, which covers a host that imports
litellm itself after importing Headroom.
- `tests/test_offline_egress_chokepoint.py`, new
`TestLiteLLMCostMapOffline`:
- subprocess probes (litellm only fetches on its first import, which an
earlier test usually did already) for four litellm-backed modules, with
the switch set both before and after `import headroom`;
- operator values `1`, `False` and empty;
- leaving offline mode restores the replaced value (absent or `False`),
and an operator change made after the pin is kept;
- `test_every_litellm_import_is_pinned_first`, the AST meta-test;
- the `offline` fixture now restores `LITELLM_LOCAL_MODEL_COST_MAP` at
teardown.
- `TestCrewAITelemetryOffline`: a stand-in `crewai` package with crewai
1.6.1's import-time tracking (no extra or CI job installs CrewAI),
probed early and late for `headroom.integrations.crewai` and
`headroom.integrations.litellm_callback`, plus operator values, a
restore test, the same probes against a real CrewAI when installed, and
`test_every_crewai_import_is_pinned_first`. A sanity test checks that
the stand-in does dial when online.
- `docs/metrics-technical-guide.md` and `docs/content/docs/proxy.mdx`
add the refresh and CrewAI's tracking to the refused list. The guide
also states the limits: a host that imports litellm *before* Headroom
has already made the fetch and must set the variable itself, and offline
prices come from the table bundled with the installed litellm release.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest -q tests/test_offline_egress_chokepoint.py tests/test_proxy_hardening.py
216 passed, 3 skipped, 1 warning in 38.44s
$ pytest -q --continue-on-collection-errors tests/test_providers tests/test_proxy tests/test_pricing* tests/test_perf* \
tests/test_telemetry* tests/test_learn* tests/test_route_advice* tests/test_backend* tests/test_litellm* tests/test_cost* \
tests/test_savings* tests/test_counterfactual* tests/test_evals* tests/test_integrations*
1742 passed, 337 skipped, 6 warnings, 1 error in 65.33s
# the 1 error is test_langchain_1x_compat.py failing collection (NameError: tool) because langchain_core
# is not installed in my venv; it fails identically on main
|
||
|
|
119d1a1939 |
fix(offline): make HEADROOM_OFFLINE a real air-gap via one chokepoint (#3729)
## Description **Revised 2026-10-05 after review: merged current `main`, routed main's new TLS-diagnostic egress (`doctor --network` and the TLS-error chain re-probe) through `guard_egress`, made the test socket trap let loopback through so the async offline tests run on Windows, and closed five model-loader findings from Devin Review.** `HEADROOM_OFFLINE=1` is documented as an air-gap switch, but several egress paths never consulted `headroom/offline.py`. This PR makes `guard_egress` the single chokepoint every Headroom-initiated outbound connection goes through, with a per-site meta-test that fails on any new unguarded egress. Full detail of every revision is under *Changes Made*. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [x] Breaking change (fix or feature that would cause existing functionality to change) — outbound calls that previously worked with `HEADROOM_OFFLINE=1` set now refuse; see *Behaviour change customers can see* - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made ### This revision (2026-10-05, after @JerrettDavis's review of 2026-09-24 and comment of 2026-10-03) * **Merged current `main`.** Conflicts in `docs/content/docs/proxy.mdx` (kept main's new rows, this PR's offline row), `headroom/image/onnx_router.py` (both imports) and `headroom/proxy/server.py` lifespan (main's rust-core passthrough block, then this PR's `_configure_observability_or_refuse()`). * **`doctor --network` / TLS diagnostics now go through the guard.** `tls_diagnostics.probe_endpoint` calls `guard_egress` before both the certificate-chain handshake and the `httpx.Client` GET; `headroom doctor` turns the refusal into one `network · skipped: HEADROOM_OFFLINE is set, no endpoint was contacted` row (status `skip`, so it does not fail the doctor) instead of a page of "unreachable". `_probe_uncached` — the chain handshake used both by doctor and by the request path's TLS-error re-probe (`describe_upstream_failure`) — guards before its socket; being diagnostic-only on the request path, it reports `skipped: …` and the TLS explanation is still produced without the chain. No allowlist widening: the inventory test passes because the `httpx.Client` site is now dominated by a guard. * **Socket trap is Windows-safe.** `no_sockets` now passes connects whose address is a loopback **IP literal** (the Windows Proactor self-pipe is a `127.0.0.1` listen + connect) and still fails every other address, including hostnames such as `localhost`. `TestSocketTrap` reproduces the Windows path on every platform by forcing `socket._fallback_socketpair` (the implementation Windows uses for `socket.socketpair()`) under `asyncio.run()`, and pins that outbound IPv4/IPv6 `connect` / `connect_ex` / `create_connection` are still caught. * **Devin Review findings (`40e2d449`).** * **Kompress tokenizer bypass.** Cached ONNX weights plus an uncached ModernBERT tokenizer fell through to an unguarded `from_pretrained(local_files_only=False)`. That call is now guarded and becomes `KompressModelNotCached`, the same translation `_hf_artifact` uses. * **Kompress ONNX candidates.** An uncached first candidate no longer ends the loop. The refusal is logged once, then the remaining candidates are checked cache-only, so a cached fp32 fallback still loads offline. * **fastembed.** `huggingface_hub` reads `HF_HUB_OFFLINE` once, at import, into `constants.HF_HUB_OFFLINE`. The cache-only attempt therefore forces that constant as well as the env var, and restores it afterwards. * **Image router.** It now resolves every artifact before publishing the session, which doubles as the "loaded" flag. A refused secondary artifact no longer leaves a half-loaded router that never retries. * **Eval runner.** The local Ollama provider is no longer refused, for the same operator-endpoint reason as the Ollama embedder. Anthropic and OpenAI are still refused. * **Sweep of main for other new egress since this PR was written:** the scanner's only new finding was `tls_diagnostics.py`. Manually checked the rest of the diff: the new `reqwest` client in `crates/headroom-proxy` and the keep-alive `httpx.AsyncClient` in `proxy/server.py` are upstream forwarding (`user-traffic`); the new HF tokenizer allowlist in `tokenizers/huggingface.py` sits behind the existing `is_offline()` gate (`gated`); `port_discovery.py`, `upstream_pinning.py`, `license_env.py` and `credential_policy.py` open no new external connections. ### A-2 (S2): the air-gap guarantee was not real `HEADROOM_OFFLINE=1` is documented as an air-gap switch, but several egress paths never consulted `headroom/offline.py`. A customer who set it and believed they were air-gapped was wrong. **This PR has been revised after an adversarial review found that the first version's guarantee was largely unenforced.** The review is reproduced in the fixes below; read the *What this does and does not promise* section for the honest scope. ### The chokepoint `headroom/offline.py` provides `guard_egress(purpose, destination)` and a named `OfflineEgressBlocked`. It refuses loudly — a silent skip is indistinguishable from success at the call site, which is how these holes survived. It is always called **before the HTTP client is constructed**, not merely before the request, so no pooled connection is ever opened. `crates/headroom-core/src/offline.rs` is the same switch, same env var, same rules, for the Rust core. #### Paths routed through it | path | previously | | --- | --- | | `headroom/transforms/kompress_remote.py` | built an `httpx.Client` and `POST`ed every compressible block to the remote Kompress endpoint | | `headroom/observability/metrics.py` | started an `OTLPMetricExporter` plus a background `PeriodicExportingMetricReader` timer | | `headroom/observability/tracing.py` | started an `OTLPSpanExporter` to Langfuse Cloud — the metric exporter's twin, missed the first time | | `headroom/onnx_runtime.py` | `hf_hub_download` on a cache miss: the **Python** half of the HF fetch, every ONNX model in the tree resolves through it | | `crates/…/tokenizer/hf_impl.rs` | fetched `tokenizer.json` from huggingface.co | | `crates/…/transforms/kompress.rs` | fetched the **Kompress model** from huggingface.co | | `crates/…/relevance/embedding.rs` | fastembed fetched ONNX weights from huggingface.co | The three Rust sites are the same shape and the same argument: `HF_HUB_OFFLINE` only shortcuts the *cache lookup*, and on a miss `hf-hub` still dials out. The first version of this PR guarded one of the three and left the other two open. Remote Kompress is guarded twice — in `__init__`, and again at the top of `compress()` **outside** the fail-open `try`, because the router caches one compressor for the process lifetime and `compress()` converts every exception into a silent passthrough. `onnx_runtime.hf_hub_download_local_first` is guarded on the **network fallback only**, so a pre-seeded air-gapped cache still resolves. The three Rust fetches refuse even on a warm cache: `hf-hub` 0.5 exposes no cache-hit/cache-miss split at that layer, so "guard before the client exists" and "serve from the cache" cannot both hold. `Kompress::from_files` / `HfTokenizer::from_file` are the network-free paths for a pre-seeded Rust deployment. Splitting the Rust fetches cache-first is the obvious follow-up and is **not** done here. ### The refusal now survives the fail-open handlers The original docstring said "anything that catches broad `Exception` around an egress call should re-raise this". **Nothing did** — there were zero re-raise sites in the package — and the refusal was swallowed on the way out of a request by `thinking_compactor._memo_compact` (wrapped directly around the guarded `kompress.compress` call), `ContentRouter._kompress_model_ready` (which reported a policy refusal as "model ready"), the startup preload, and the OpenAI reasoning-compaction path. A full `/v1/messages` request with `HEADROOM_OFFLINE=1` + `HEADROOM_KOMPRESS_ENDPOINT` returned **200 uncompressed** with `guard_egress` having fired twice. **`OfflineEgressBlocked` now derives from `BaseException`, not `Exception`.** A convention that has to be re-applied at every future `except Exception:` in a 100k-line codebase is not a guarantee; the type hierarchy is. Same reason `KeyboardInterrupt` and `SystemExit` sit outside `Exception`: a policy decision is control flow, not a runtime error to degrade around. All four sites propagate with no edit to any of them. ⚠️ **This is a behaviour change for callers.** `except Exception` no longer catches it anywhere, including in third-party code. `except BaseException` still does; those are swept by `TestBroadHandlerSweep`, which requires each to re-raise or carry a written reason. `content_router`'s native-detect degrade path now re-raises it explicitly, alongside the cancellation `BaseException`s it already let through. #### Where the refusal deliberately becomes a degradation Escaping `except Exception` is the point, but it made one class of caller worse. Fetching **public model weights** is not data leaving the box, and an air-gapped proxy with a cold HF cache would have started failing user requests where it previously degraded — punishing the request for a decision the operator made about the host. So each optional model loader translates the refusal into its own "model unavailable" error, **explicitly**, with the switch named in the log and the refusal chained: | boundary | translated to | the caller's existing degradation | | --- | --- | --- | | `kompress_compressor._hf_artifact` | `KompressModelNotCached` | non-ML compression path | | `image/onnx_router._hf_artifact` | `RuntimeError` | `ImageCompressor` → `Technique.PRESERVE` | | `memory/adapters/embedders` ONNX load | `RuntimeError` | local embedder unavailable | | the two Kompress download daemon threads | logged, `False`/return | keeps whatever is cached | Behaviour for a cold air-gapped cache is exactly what it was. What changed is that the log now names the switch, and the translation is a reviewed decision at a named boundary rather than something inherited from whichever `except Exception` happened to be in the stack. The daemon threads matter separately: an unhandled `BaseException` there reaches `threading.excepthook` as a bare traceback on every air-gapped boot. Paths that **send data off the box** — remote Kompress, OTLP, Langfuse, the beacon — are untouched and still refuse loudly. ### Startup refuses a contradictory configuration `configure_otel_metrics` was called **outside** `lifespan`'s `try`, above the line that sets `app.state.startup_error`, so an `OfflineEgressBlocked` escaped uvicorn as an unhandled error — killing a proxy that had been serving traffic, with a traceback instead of an explanation. It now runs through `_configure_observability_or_refuse()`, which turns a refusal into the shape `_check_rust_core` uses ten lines above: what is wrong, how to fix it, **exit 78** (`EX_CONFIG`). A preflight covers the remote-Kompress contradiction the same way. So the ordinary case is settled at startup with an actionable message, and `BaseException` is the backstop rather than the user experience. ``` FATAL: OTLP metric export (HEADROOM_OTEL_METRICS_ENABLED) needs outbound network access, but HEADROOM_OFFLINE=1 forbids it. refused: HEADROOM_OFFLINE is set: refusing outbound network access for OTLP metric export to … fix: set HEADROOM_OTEL_METRICS_EXPORTER=console or scrape /metrics (both stay on-box), or unset HEADROOM_OTEL_METRICS_ENABLED ``` ### Python/Rust normalisation parity Python trimmed with `str.strip()`, Rust with `str::trim()`. Not the same set: `str.isspace()` includes U+001C–U+001F, the Unicode `White_Space` property does not, so `HEADROOM_OFFLINE=$'\x1c1'` read as **offline to Python and online to Rust** — one variable, one half of the process air-gapped. Both sides now trim an enumerated `TRIM_CHARS` / `_TRIM_CHARS`, and the parity test compares the trim sets and a shared normalisation table, not just the accepted tokens. (Case folding needed no change: no non-ASCII character lowercases into any character of `1`/`true`/`yes`/`on`.) ### The meta-test, rebuilt The first version was neutered by one line: ```python if "guard_egress" in source: continue ``` A whole-file exemption keyed on the string appearing *anywhere*, including in a docstring — the exact blanket exemption the allowlist's own comment says must never happen. It made the site-count assertion unreachable for every guarded file, so "adding a second client trips this test" only held for files with **no** guard at all. Two bypasses were demonstrated against it: a second unguarded `httpx.Client` in an already-guarded file, and a brand-new unguarded module with the word in its docstring. It now decides **per egress site**: * **AST, not text.** Comments and docstrings vouch for nothing. `self.client.post()` is no longer read as `requests.post`, and the `httpx.post` example in `compress.py`'s docstring is not a site. * **Dominance.** A site is guarded only when a `guard_egress` call is in its block or an enclosing one, earlier in that block, in the same function. A guard in a sibling `if`/`except` arm, in a nested `def`, or after the site does not count. A guard in a helper the function calls does not count either — conservative on purpose. * **`TestSiteScannerRules`** tests the scanner itself, pinning both demonstrated bypasses plus the one-line `def f(): return httpx.Client().post(...)` that the old `startswith("def ")` skip made invisible. A meta-test nobody tests is a comment. * **Wider callee set:** module-level `httpx` verbs (`headroom/cli/mcp.py` was a live unmatched site), `hf_hub_download` (skipping `local_files_only=True`, which opens no socket), fastembed's `TextEmbedding`, the OTLP exporters, and the OpenAI/Anthropic SDK clients. * **`crates/` is swept too** (`TestRustEgressChokepointCoverage`). It was outside the scan entirely, which is how two Rust downloads stayed open. Text-level rather than AST — no Rust parser here — so "guarded" means a `guard_egress` earlier in the same `fn` at no deeper indent. Weaker, and labelled as such; the runtime assertions live in each crate's `#[test]`s. * **Callees are resolved through the module's import table before matching.** The scanner compared the text `ast.unparse` produced, which is whatever local name the module bound — so `import httpx as h` hid a client from a test whose entire job is to notice clients. That was not hypothetical: `copilot_auth.py` does `from urllib import request as urllib_request`, so its GitHub calls rendered as `urllib_request.urlopen` and **five call sites, including the shared `_urlopen` helper every Copilot request goes through, were invisible**. `_canonical_callee` rewrites the leading name first; the alias defence cuts both ways, so `guard_egress as ge` still counts as a guard. Imports are collected from the whole tree, not just module level, because half the egress here is imported inside the function that uses it. * Allowlist counts now mean **unguarded** sites only, so a file can legitimately guard some of its egress and still carry an entry for the rest. * **There is no longer a category for "known violation".** `test_no_category_permits_a_known_violation` pins the permitted set to `loopback` / `gated` / `user-traffic` / `operator-endpoint` / `cache-only` and rejects "out of scope" as a reason. ### The offline policy, in plain words `HEADROOM_OFFLINE=1` **refuses every connection Headroom itself decides to make to a destination Headroom itself chose.** That is the rule, not a sample list: telemetry, update check, license/usage reporter, OTLP and Langfuse export, HuggingFace / Kompress / fastembed model and tokenizer downloads (Python and Rust), release-binary and codebase-memory-mcp downloads, eval dataset downloads and the provider SDK clients the eval harness drives, GitHub Copilot device-flow auth and token exchange, the Anthropic / Codex / Copilot subscription pollers, both OpenAI embedders, both Headroom Cloud compression integrations, and the TLS diagnostics (`headroom doctor --network` endpoint checks and the certificate-chain re-probe after an upstream TLS failure). **Four things are deliberately still allowed, and they are the complete set of exceptions.** Every one of them is either physically incapable of leaving the box or is not Headroom-initiated traffic: | category | what it covers | why it survives the switch | | --- | --- | --- | | `user-traffic` | `proxy/server.py` (2 sites) | the caller's own request forwarded to the upstream the operator configured. A proxy that refuses this is not a proxy; an air-gapped install points it at an on-prem endpoint. | | `operator-endpoint` | `memory/adapters/embedders.py` (1 site) | the Ollama embedder, whose address comes entirely from operator config and defaults to `http://localhost:11434`. Exactly one thing qualifies today, and a hard-coded internet host may never hide here. | | `loopback` | 8 files, 12 sites | hard-coded `127.0.0.1` probes against the operator's own local proxy (`headroom doctor`, the installers, the MCP sidecar, the test harness). | | `gated` | 3 files, 3 sites | the beacon, the update check and the usage reporter are already unreachable behind an `is_offline()` check, so no connection is ever built. | Plus one mechanical exemption, `cache-only` (`relevance/embedding.py`, 1 site): the first of the two fastembed loads runs with `HF_HUB_OFFLINE` forced to `1`, so `huggingface_hub` resolves from the local cache or raises **without opening a socket**; the network retry directly below it calls `guard_egress`. That is what lets a pre-seeded air-gapped host keep loading the model while a cold one refuses. #### What changed since the review The previous revision of this PR recorded **nine egress-capable files** under an `unguarded` allowlist category meaning "reachable under `HEADROOM_OFFLINE`, out of scope for A-2": Copilot auth/token exchange, three subscription pollers, the two binary/CBM downloads, the eval-dataset fetch, and both Headroom Cloud compression integrations — plus the eval harness's LLM clients, both OpenAI embedders and the fastembed weights. @JerrettDavis blocked on exactly that, and he was right: recording that a switch does not do what it says is not the same as making it do it. **All of them are now guarded, and the `unguarded` category is deleted.** `test_no_category_permits_a_known_violation` pins the five permitted categories by name and rejects "out of scope" as an allowlist reason, so re-introducing a bless-it-and-move-on category takes a deliberate edit with a reviewer looking at it. #### ⚠️ Behaviour change customers can see **Outbound calls that previously worked with `HEADROOM_OFFLINE=1` set now refuse.** That is the point of the PR, but it is a real change for anyone who set the flag and kept using these features: * `headroom auth` / `headroom copilot-auth login` cannot complete a GitHub device flow, and a cached Copilot OAuth token cannot be exchanged for an API token. A Copilot subscription is unusable on an air-gapped box regardless, but the failure now arrives at the moment of the attempt. * `headroom install` will not download release binaries or codebase-memory-mcp. Pre-stage them, or install with the flag unset. * The Anthropic / Codex / Copilot subscription and quota panels stay empty, with the reason logged and (for Copilot quota) shown in the panel's own error slot. * Headroom Cloud compression mode in the ASGI middleware and the LiteLLM callback refuses instead of falling through to local compression — loudly, because silently compressing locally would hide that the deployment is no longer doing what it was configured to do. * The eval harness refuses to build provider clients or download BFCL. * The OpenAI embedders (memory adapter and direct mem0 backend) refuse. Use the ONNX or Ollama embedder on an air-gapped box. * `EmbeddingScorer` raises "model unavailable" on a **cold** fastembed cache. A warm one is unaffected. #### Refusals reach a human, never a traceback Following the pattern the earlier commits set for the Kompress download threads, the model layer and the startup banner: * **CLI:** `OfflineAwareGroup` (`headroom/cli/main.py`) is the one place that translates the refusal into a Click `Error:` line, for every subcommand at once. `OfflineEgressBlocked` stays a `BaseException` everywhere inside, so nothing else can downgrade it; only the last frame before the terminal is allowed to make it readable. * **Background pollers:** each catches `OfflineEgressBlocked` specifically, calls `offline.note_refusal` (WARNING once per purpose, DEBUG after, so a 30-second poller does not flood the log), and returns its ordinary "no data" value. An un-caught `BaseException` out of a bare `create_task` surfaces as "Task exception was never retrieved" whenever the GC gets to it — a traceback with no explanation in it. * **Model loaders:** `EmbeddingScorer._get_model` translates into a named "model unavailable" `RuntimeError`, the way `kompress_compressor._hf_artifact` and `image/onnx_router._hf_artifact` already do. ### 🚩 Two decisions that need a founder call Both were implemented the **safe** way — refuse, loudly and legibly — because that is the defensible default for a switch an operator sets to mean "nothing leaves this box". Neither is obviously right, and a permissive default should not be chosen quietly: 1. **Should OTLP / telemetry export be permitted under `HEADROOM_OFFLINE`?** Today it refuses, and a contradictory config (`HEADROOM_OTEL_METRICS_ENABLED=1` + offline) exits 78 at startup with an explanation. The argument for permitting it is that a regulated customer's OTLP collector is usually in-cluster, and losing observability is a real operational cost. The argument against — and the reason it is refused today — is that the guard **cannot reliably tell an in-cluster collector from an internet host**: DNS, service meshes and egress proxies all blur it, so any exemption is really "trust the endpoint the config names". A customer with an in-cluster collector currently has to scrape `localhost:8787/metrics` or use `HEADROOM_OTEL_METRICS_EXPORTER=console`. If we want to permit it, the honest shape is an explicit second flag (e.g. `HEADROOM_OFFLINE_ALLOW_OTLP=1`) rather than widening the policy. 2. **Should the subscription pollers refuse, or degrade silently?** Today all three refuse and say so once per purpose at WARNING. The argument for silence is that these are cosmetic panels and a WARNING on an air-gapped box is noise about a decision the operator already made. The argument for the current behaviour is that a usage panel that silently stops updating is exactly the "why is this empty?" confusion the switch was supposed to end, and once-per-purpose logging bounds the noise. Changing this is a one-line change in `offline.note_refusal`. ### Tests `tests/test_offline_egress_chokepoint.py` — 110 tests. Per-path socket assertions (a `no_sockets` fixture booby-traps `socket.create_connection`, `socket.socket.connect` and `connect_ex`, so asserting "no socket" rather than "raises" catches a guard placed after the client is built), the scanner-rule tests, the two coverage sweeps, the broad-handler sweep, the startup-refusal tests, the model-loader degradation tests, the cross-language parity tests, and the doc-claim test. Plus 8 Rust tests in `headroom-core`. ### Fail-before / pass-after Per fix, work committed, then sources reverted while keeping the new tests, run, restore. No `git stash`. **Meta-test (1).** Added a probe module whose docstring mentions `guard_egress` and a second unguarded `httpx.Client` to the already-guarded `kompress_remote.py`. The old meta-test: `3 passed`. The new one: ``` headroom/_a2_probe_tmp.py line 7: httpx.Client().get("https://evil.example.com") (not guarded, not allowlisted) headroom/transforms/kompress_remote.py line 154: return httpx.Client(timeout=1.0) (not guarded, not allowlisted) ``` **Rust guards (2).** Guard calls commented out, tests kept: ``` panicked at crates/headroom-core/src/transforms/kompress.rs:688: expected an offline refusal, got a network error: no loadable ONNX artifact in `headroomlabs-ai/this-model-does-not-exist-a2` (tried [...]): request error: http status: 401 ``` The model id cannot exist, so a `Hub` error can only come from a real round-trip. That **401 is the proof a socket was opened to huggingface.co with `HEADROOM_OFFLINE=1` set**, and `Offline` can only come from returning before `Api::new()`. **Refusal / startup (3, 4).** `headroom/{offline,proxy/server,transforms/content_router}.py` reverted to the PR tip, tests kept: `11 failed, 6 passed` — including `test_thinking_compactor_does_not_swallow_it`, `test_kompress_model_ready_does_not_report_a_refusal_as_ready`, `test_no_broad_handler_swallows_the_refusal` and all five `TestStartupRefusal` tests. **New Python guards (5).** `headroom/{onnx_runtime,observability/tracing}.py` reverted to `origin/main`, tests kept: `4 failed` — the two socket tests plus both coverage assertions. **Normalisation (6).** Both `offline` modules reverted to the PR tip, parity tests kept: `6 failed`, with `HEADROOM_OFFLINE="\x1c1"` reading as offline in Python and not in Rust. **Docs (7).** Both doc files reverted, test kept: ``` AssertionError: docs/metrics-technical-guide.md says 'disables all outbound traffic', but _EGRESS_ALLOWLIST still records these as reachable under HEADROOM_OFFLINE: [...16 files...] ``` #### This revision (the review fixes) Committed first, then `git checkout origin/main -- headroom/{binaries,cli/main,copilot_auth,graph/installer,integrations/asgi,integrations/litellm_callback,memory/adapters/embedders,memory/backends/direct_mem0,relevance/embedding}.py headroom/{evals,subscription}/` with the new tests kept: ``` 23 failed, 79 passed ``` The 23 are the five Copilot-auth socket tests, the four poller tests, the two install-download tests, the BFCL test, both Headroom Cloud tests, the two cloud-embedder tests, the four fastembed tests, both CLI-translation tests — and `test_every_egress_site_is_guarded_or_allowlisted`, which lists every reverted file as "(not guarded, not allowlisted)". The aliased-import blind spot, isolated (canonicalisation stubbed back to the identity, i.e. the pre-fix behaviour, against the same inputs): ``` before: INVISIBLE: import httpx as h INVISIBLE: from urllib import request as urllib_request INVISIBLE: from openai import AsyncOpenAI as AO INVISIBLE: import httpx as h (inside a function) aliased guard recognised: [False] copilot_auth.py sites seen: 4 after: DETECTED (all four); aliased guard recognised: [True] copilot_auth.py sites seen: 6 ``` **Pass-after (this revision):** `102 passed` in `tests/test_offline_egress_chokepoint.py`; `391 passed, 86 skipped` across the copilot-auth / copilot-quota / codex-rate-limits / binaries / integrations / CLI subsets; `544 passed, 15 skipped` across the relevance / embedding / mem0 / asgi / litellm / subscription / installer subsets; `160 passed, 6 skipped` across the evals and CLI doctor/learn subsets; `59 passed` across the proxy-startup subset; `cargo test -p headroom-core offline` → `8 passed`. **Pass-after (earlier revisions):** `70 passed` (pytest), `942 passed` (`cargo test -p headroom-core --lib`), `720 passed, 4 skipped` across the offline/otel/kompress/telemetry/onnx/tracing/relevance subsets, and `1868 passed, 30 skipped` across the kompress/offline/image/embedder/memory subsets after the model-loader boundary change. The one failure in the full run, `test_mcp_reconcile.py::test_ordinary_install_does_not_adopt_serena`, reproduces unchanged at the PR tip and is unrelated. Cost/pricing tests are the known `origin/main` litellm breakage (#3722) and were not run. ### Checks - `uvx ruff@0.16.7 check .` → All checks passed - `uvx ruff@0.16.7 format --check .` → 1622 files already formatted - `cargo fmt --check` → clean; `cargo clippy -p headroom-core --all-targets` → no warnings - `CHANGELOG.md` untouched - CodeQL: the four high alerts were all in `tests/test_offline_egress_chokepoint.py` and are fixed honestly rather than suppressed. Three `py/incomplete-url-substring-sanitization` host assertions (`"host" in url`, which is also true of `https://host.attacker.test`) now parse with `urllib.parse.urlsplit` and compare the hostname exactly — which is what the tests meant. The `py/redos` regex scraping Rust `char` literals had a nested `(...)+` that can backtrack exponentially; a Rust char literal is exactly one character, so it is now a single bounded alternation. ### Follow-ups (not in this PR) 1. ~~Guard the 16 `unguarded` files, each with its own user-facing message.~~ Done in this PR; the `unguarded` category is deleted. 2. Split the three Rust HF fetches cache-first so a pre-seeded air-gapped cache resolves without egress, the way `hf_hub_download_local_first` already does on the Python side. 3. The Rust sweep is a text heuristic. An AST-level dominance check (via `syn`, in a Rust test) would make the two halves symmetric. 4. The control-plane heartbeat lives in a different repository and is still out of scope. 5. The scanner has no entry for transformers `from_pretrained` / `SentenceTransformer`. The Kompress tokenizer site Devin found is guarded now, but `models/ml_models.py` (SigLIP, technique router, sentence-transformers) and the Kompress PyTorch backend still rely on `HF_HUB_OFFLINE` alone, which `apply_offline_env` sets with `setdefault` and only in the proxy process. Each needs the cache-first split the tokenizer has, plus a scanner entry. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3729" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ## Testing - [x] Unit tests pass (`pytest`) — subsets below - [x] Linting passes (`ruff check .`) — touched files, `ruff@0.16.7` - [ ] Type checking passes (`mypy headroom`) — not run - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # this revision $ LITELLM_LOCAL_MODEL_COST_MAP=True uv run --frozen --extra dev --extra proxy pytest \ tests/test_offline_egress_chokepoint.py tests/test_cli_doctor.py tests/test_corporate_tls.py -q 245 passed # after the Devin fixes $ pytest tests/test_offline_egress_chokepoint.py + kompress/onnx_router/relevance/embedding/evals subsets 361 passed, 6 skipped # fail-before for the Devin fixes (4 sources reverted, tests kept): 6 failed + the fallback test (fails once its fake # raises OSError instead of LocalEntryNotFoundError, which conftest converts to a skip) # fail-before, TLS guard (tls_diagnostics.py + doctor.py reverted, tests kept) FAILED TestDoctorNetworkOffline::test_endpoint_check_refuses_before_any_connection FAILED TestDoctorNetworkOffline::test_doctor_reports_the_network_checks_as_skipped FAILED TestDoctorNetworkOffline::test_chain_probe_opens_no_socket FAILED TestEgressChokepointCoverage::test_every_egress_site_is_guarded_or_allowlisted headroom/proxy/tls_diagnostics.py line 473: with httpx.Client(...) (not guarded, not allowlisted) # with only tls_diagnostics.py reverted, the doctor test fails on content, not on a missing helper: [('net api.anthropic.com', 'fail'), ('net api.openai.com', 'fail'), ...] != [('network', 'skip')] # fail-before, socket trap (old fixture body, new tests kept) FAILED TestSocketTrap::test_event_loop_wakeup_pipe_is_not_egress SocketOpened: outbound connection attempted while offline: (<socket ...>, ('127.0.0.1', 50151)) # — the same csock.connect inside socketpair() that Jerrett's Windows trace shows. $ uvx ruff@0.16.7 check / format --check (touched files) All checks passed! / 4 files already formatted ``` Earlier-revision test evidence is in *Fail-before / pass-after* under *Changes Made*. ## Real Behavior Proof - Environment: macOS, Python 3.14, this branch (merged with `main` at `befdb52f`), no proxy running. - Exact command / steps: `HEADROOM_OFFLINE=1 headroom doctor --network --port 59999`, run for real. A second run went through `headroom.cli.main` with a `sys.addaudithook` recording every `socket.getaddrinfo` / `socket.connect`. - Observed result: the doctor table shows `network · skip · skipped: HEADROOM_OFFLINE is set, no endpoint was contacted`, with the refusal naming `https://api.anthropic.com/v1/models` as the hint. The audit hook recorded only `127.0.0.1` lookups and connects (the doctor's own proxy probes on :59999 and :8787) and **zero external lookups or connects**. Exit code 2 comes from the unrelated "proxy not reachable" failures. - Not tested: Windows itself. The Windows mechanism is reproduced by forcing `socket._fallback_socketpair`, which is the code Windows runs for `socket.socketpair()`; the CI Windows job (`test-windows-newline`) only runs `windows_newline`-marked tests and was not extended here. ## Runtime Rollout Safety - Rollout-managed feature(s): none; behaviour keyed on the existing `HEADROOM_OFFLINE` switch. - Minimum rollout channel: stable. - Stable/default behavior changed: only when `HEADROOM_OFFLINE` is set (refusals listed above). Unset, nothing changes. - Kill switch / disable path: unset `HEADROOM_OFFLINE`. - Unsafe override required: no. - Qualification impact: none. - Rollback path: revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Two founder decisions remain flagged above (OTLP under offline; pollers loud vs silent). Windows CI coverage for this module is not added: no unit-test job runs the full module on Windows today, and adding one is a workflow change better made separately. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
0147cf0220 |
fix(security): stop forwarding the proxy token to upstream providers (#3891)
## Description `HEADROOM_PROXY_TOKEN` is accepted as either `X-Headroom-Proxy-Token: <token>` or `Authorization: Bearer <token>` (`read_proxy_token`, `server.py`). The provider handlers build their upstream headers from the inbound request and only strip `x-headroom-*`, so a token sent in `Authorization` was forwarded verbatim to Anthropic / OpenAI / the catch-all upstream. That form is not unusual: - `docs/metrics-technical-guide.md` documents `Authorization: Bearer <token>`. - The TypeScript SDK (`sdk/typescript/src/client.ts`, `rawFetch`) sets `Authorization: Bearer <apiKey>` whenever the caller has no provider credential. - An Anthropic client using `x-api-key` for the provider can use `Authorization` for the proxy. In each case the proxy's shared secret ended up in a third party's request logs. For Anthropic it also changed auth-mode classification, because an extra bearer turned a PAYG request into OAuth. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **`scrub_proxy_token_headers(scope, token_bytes)`** in `headroom/proxy/server.py` removes the credential from the ASGI scope once, so no handler or transport can forward it. About 96 call sites build upstream headers from `request.headers`, so the fix is not made per call site. - `Authorization` is dropped only when it is `Bearer <configured token>`. Parsing matches `read_proxy_token` (case-insensitive scheme, stripped credential) and the comparison is constant-time. Any other value is a provider credential and is kept. - `X-Headroom-Proxy-Token` is always dropped, including when `HEADROOM_STRIP_INTERNAL_HEADERS=disabled`. - **The HTTP `_security_gate`** calls it after the token check and on the exempt paths (loopback callers, health probes), just before `call_next`. - **`WebSocketAuthMiddleware`** calls it on the loopback, authenticated and no-token paths before handing off to the app. This covers the `/v1/responses` and `/v1/live` WebSocket relays. - **Docs:** `docs/content/docs/proxy.mdx` and `docs/metrics-technical-guide.md` now list both header forms and say the token is removed before forwarding. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed New tests in `tests/test_proxy_hardening.py`: - **End-to-end (`TestProxyTokenIsNotForwardedUpstream`):** these capture the request that actually leaves the proxy via a mock `httpx` transport installed after app startup. Cases: - `/v1/messages` with the token in `Authorization`, and with a lowercase `bearer` scheme - `/v1/chat/completions` with `X-Headroom-Proxy-Token` plus a provider bearer - `HEADROOM_STRIP_INTERNAL_HEADERS=disabled` - a loopback caller - catch-all passthrough (`GET /v1/models`) - **Helper unit tests (`TestScrubProxyTokenHeaders`):** non-matching bearer, non-bearer scheme, duplicate `Authorization`, no configured token, non-ASCII value. - **WebSocket (`TestWebSocketScrubsProxyToken`):** middleware-level for loopback and remote callers, plus route-level on `/v1/responses`. With the scrub stubbed out, 12 of the new tests fail. The one that still passes (custom header, default strip mode) is already covered by the existing `x-headroom-*` strip. ### Test Output ```text $ pytest -q tests/test_proxy_hardening.py 48 passed, 1 warning in 13.90s $ pytest -q tests/test_proxy*.py tests/test_identity_resolution.py tests/test_codex_live.py 4 failed, 947 passed, 107 skipped, 10 warnings in 167.26s $ ruff check . && ruff format --check . All checks passed! 1653 files already formatted $ mypy headroom --ignore-missing-imports Success: no issues found in 542 source files ``` The 4 failures are sandbox network blocks (tiktoken download from `openaipublic.blob.core.windows.net`, real upstream calls). They fail identically on the parent commit without this change: - `test_proxy_ccr.py::TestEndToEndTOINIntegration::test_pipeline_compresses_tool_output_and_records_toin` - `test_proxy_mode_benchmark.py::test_local_mode_benchmark_shows_compression_and_cache_tradeoff` - `test_proxy_passthrough_integration.py::TestPassthroughErrorHandling::test_missing_auth_header_openai` - `test_proxy_passthrough_integration.py::test_head_root_returns_200_not_405` ## Real Behavior Proof - Environment: Linux, Python 3.11, editable install (`.[proxy]`), FastAPI `TestClient` with a capturing `httpx` transport as the upstream. - Exact command / steps: `pytest -q tests/test_proxy_hardening.py -k "NotForwardedUpstream or Scrub"` - Observed result: the captured upstream request has no `authorization` header when it carried the proxy token, and keeps `x-api-key` or the provider bearer unchanged. - Not tested: a live proxy against real Anthropic/OpenAI endpoints. ## Runtime Rollout Safety - Rollout-managed feature(s): none. This is a security fix in the inbound auth middleware. - Minimum rollout channel: n/a - Stable/default behavior changed: yes, but only for requests carrying the proxy token. That credential is no longer forwarded upstream. With no `HEADROOM_PROXY_TOKEN` configured, the only change is that `X-Headroom-Proxy-Token` is now dropped even when `HEADROOM_STRIP_INTERNAL_HEADERS=disabled`. - Kill switch / disable path: none (deliberately; forwarding the secret is never correct). - Unsafe override required: no - Qualification impact: none - Rollback path: revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - **Edge case:** an operator who set `HEADROOM_PROXY_TOKEN` to the same value as their provider API key will now have that `Authorization` header removed. The two are indistinguishable, and reusing the provider key as the proxy secret is exactly the leak this closes. Such deployments should set a distinct proxy token. - **Why run after the gate:** the scrub runs *after* the credential check rather than as a separate outer middleware. An outer middleware would hide the token from the gate and turn authenticated requests into 401s. - **Follow-up (not in this PR):** `sdk/typescript/src/client.ts` could stop mirroring the proxy token into `Authorization` now that `X-Headroom-Proxy-Token` is always sent. It is harmless after this change. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3891" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Tejas Chopra <tejas@headroomlabs.ai> |
||
|
|
5ff4ea1ef9 |
fix(metrics): split rate-limited by source and label failures by provider (#3707)
Closes #3696. #3615 moved the outcome-funnel short-circuit from `>= 500` to `>= 400` — correct, and staying — but it silently changed what two metrics mean and the docs were never updated. **⚠️ This changes exported series. See Migration.** ## What was wrong - `headroom_requests_failed_total` was documented as "Upstream 5xx errors"; it now counts 4xx too. - `headroom_requests_rate_limited_total` was documented as "Requests **Headroom** rejected via its own rate limiter (**not upstream 429s**)" — #3615 inverted exactly that. - 4xx also dropped out of `headroom_requests_by_provider` (only `record_request` touches it), leaving Prometheus with no way to tell which upstream was failing. ## Changes **`headroom_requests_rate_limited_total{source="headroom"|"upstream"}`** — two 429s mean opposite things to an operator: `headroom` is our limiter refusing (raise the cap), `upstream` is the provider refusing (back off / shard keys). Closed label set, so cardinality stays bounded; unknown values clamp to `headroom`, the counter's pre-#3615 meaning, so an un-updated caller cannot invent a label. Threaded at all five call sites (`openai.py:3554, 5681`, `anthropic.py:1256`, `gemini.py:405` → `headroom`; `outcome.py:543`, the #3615 funnel → `upstream`). **`headroom_requests_failed_total{provider}`** — uses `outcome.provider`, already Copilot-relabelled by the funnel, so no second convention. Matches the existing `upstream_connection_errors_by_provider` shape. **All four backends**, so no sink reports a differently-shaped counter: Prometheus, persistent/lifetime (`SavingsTracker.record_lifetime_rate_limited`), OTel (attribute on `headroom.proxy.requests.rate_limited`), and `/stats` JSON. `SCHEMA_VERSION` stays 5 — the new maps are purely additive, and old state files load with them empty rather than back-filling invented history. **Docs**: `docs/metrics-technical-guide.md`, `docs/content/docs/metrics.mdx`, `wiki/metrics.md`, plus tooltips on the dashboard's Failed / Rate Limited tiles so the UI stops implying "Rate Limited" means Headroom only. `requests_total` is now documented as a completed-request denominator with `inbound_requests_total` named as the all-traffic counter, and the failure-rate PromQL carries a "why" on its denominator so nobody "simplifies" it. Checked and found clean: `examples/grafana/headroom-dashboard.json`, `docs/observability.md`, `docs/content/docs/proxy.mdx`, `wiki/proxy.md`, `README.md`. ## Second commit: a bug found while reviewing the first Adding the label turned `headroom_requests_failed_total` from a counter that **always** exported `0` into one that exports **no sample at all** on a healthy proxy — confirmed by rendering `/metrics` on a fresh instance: ``` headroom_requests_failed_total: 0 sample(s) headroom_requests_rate_limited_total: 2 sample(s) ``` That breaks the failure-rate query documented in this very PR: an empty numerator empties the whole PromQL expression, so the panel reads "No data" instead of 0% while everything is fine, and `absent()` alerts would fire on healthy proxies. The rate-limit series avoided this by seeding its closed label set; providers are open-ended so the same trick does not apply. Fix: seed the `unknown` bucket at zero. It is a real bucket — `record_failed` uses it for unattributed failures — so this invents no provider. Guarded by a new test. ## Migration Both counters now export **only** labelled series; an unlabelled sample alongside would double-count under `sum()`. Unaggregated queries return multiple series — `sum without (source)` / `sum without (provider)` reproduces the previous value exactly. Noted in the docs and an mdx Callout. ## Verification - **16 tests fail on pre-change source** (verified by reverting only `headroom/`), pass after. - Targeted metrics suites → **58 passed**. Broad sweep `-k "metric or stats or outcome or rate_limit or savings_tracker or persistent"` → 634 passed / 44 skipped. Handler, dashboard and savings-history suites → 90 + 79 passed. - Ruff 0.16.7 (the CI pin, via `scripts/verify-ruff-version.py --print-version`): clean. `CHANGELOG.md` intentionally untouched — `changelog-guard.yml` fails any PR that edits it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
250ede2f7f |
fix(reporting): show net vs gross savings, real skip thresholds, and the effective profile (#3123)
## Description
Six reporting/config defects found while investigating a user reporting
~1% savings on Claude Code. **None of these changes how much Headroom
compresses** — all of them change whether an operator can tell what it
did. Every one was found by reading that user's own 227,777 lines of
proxy logs against the code.
## Changes Made
- **`perf/analyzer`: parse and render `tok_inflated`.** Every PERF line
carried it; nothing downstream read it. The report could print
`321,239,562 -> 313,274,727` directly above `8,455,763 saved` — two
figures that differ by exactly the 490,928 tokens of inflation it
omitted.
- **`content_router`: report the real skip thresholds.** The routing
summary hardcoded `skipped (<50 words)` regardless of what was in force.
Wrong number (the message gate is `min_tokens`, 10–250 by profile),
wrong unit (tokens and characters, never words), and it merged two
different gates under one label.
- **`perf/analyzer`: disclose that Transform Effectiveness is partial.**
It is built only from `pipeline.py`'s `Transform NAME:` lines.
`compression_units.py` / `compression_batches.py` contain zero logging
calls, so the table read `content_router: 189,783 saved` against a PERF
total 44x larger. Reports the divergence rather than a coverage ratio —
the two are different populations and neither contains the other (those
lines carry no request_id, fire per stage, and are emitted before the
forwarder decides).
- **`perf/analyzer`: disclose the routing denominator.** Percentages
were taken over 4 of the router's 17 outcome buckets, silently dropping
buckets larger than several it displayed.
- **`savings_tracker`: stop dropping tool-schema dollars.**
`estimate_request_savings_usd` prices four buckets; `record_request`
read three. `tool_schema` was computed and discarded, so a quarter of
the token headline never reached "Cost saved". The two inputs are
disjoint (verified at the call site), so this is additive, not
double-counting.
- **`agent_savings`: an unknown profile no longer degrades to
`balanced`.** `balanced` is a different product posture from the default
`coding`: cache→token mode, dedup off, tool-search off, user messages
uncompressed, message floor 25x higher, block floor 20x higher. A typo
in `HEADROOM_SAVINGS_PROFILE` silently reconfigured the whole proxy. Now
degrades to `DEFAULT_PROFILE` and names the resolved profile in the
warning.
- **`agent_savings`: give `min_chars_for_block` a config-object path.**
Every other router pipeline kwarg travels on the config object; this one
alone was env-only, so an unseeded proxy applied every sibling `coding`
knob while this floor stayed at 500 instead of 25.
- **`server`: log the resolved compression posture at startup**, reading
cross-turn dedup off the constructed router rather than the environment
(the router resolves it as `config OR env`, so reading env alone would
be a guess).
## Testing
- [x] Unit tests pass, [x] ruff, [x] mypy, [x] new tests added
```text
uv run pytest tests/ -k "content_router or agent_savings or perf or analyzer or savings or proxy_server or cli_perf or prometheus"
620 passed, 25 skipped
uv run mypy headroom # Success
uv run ruff check . && ruff format --check . # clean
```
## Real behavior proof
- **Setup:** macOS arm64, Python 3.12, this branch. Input: 60 MB /
227,777 lines of real proxy logs from the reporting user (6 rotated
files, 2,792 PERF lines, 2026-08-17 → 2026-08-19).
- **Steps:** pointed `headroom.perf.analyzer.LOG_DIR` at that directory
and rendered the report before and after the patch.
- **After-fix output (real data, unmodified):**
```text
Requests: 2792
Tokens: 321,288,161 -> 313,323,326 (2.6% messages)
Tokens saved: 11,158,901 (3.4% reduction)
· inflated 490,928 (net message reduction 7,964,835)
· messages 8,455,763
· tool schemas 2,703,138
! stage-level total 190,641 != PERF message total 8,455,763 — this table sees only
engines that emit a Transform line, counts per stage, and does not check whether
the mutation shipped
Skipped: 44641 (77%) — below size floor
(shares are of these 4 buckets only, n=58319; see `[router] route_counts=` for the
full outcome space)
```
The arithmetic now closes on the page: `8,455,763 - 490,928 =
7,964,835`, matching the token delta exactly. Before the patch none of
the three annotated lines existed and the `Skipped` line claimed `<50
words`.
- **Profile resolution verified by execution**, not inspection —
subprocesses with controlled env:
```text
vanilla (nothing set) mode=cache dedupe=1 tool_search=1 min_tokens=10 min_chars=25
HEADROOM_SAVINGS_PROFILE=coding mode=cache dedupe=1 tool_search=1 min_tokens=10 min_chars=25
unknown profile name (before) mode=token dedupe=0 tool_search=0 min_tokens=250 min_chars=500
unknown profile name (after) -> resolves to `coding`, warning names it
coding, seeding never runs min_chars=25 (was 500 before this patch)
```
- **Not tested:** live paid Anthropic traffic. These are
reporting/config surfaces; the wire path is untouched by this PR.
## Review readiness
- [x] Self-reviewed. Three overclaims in my own first draft were
corrected before this PR: a false subset claim in the Transform
Effectiveness note, a comment asserting `min_chars_for_block` was the
*only* env-only field (it is the only env-only *router pipeline kwarg*;
`cross_turn_dedup`, `tool_search`, `protect_reads`, `code_aware`,
`effort_router`, `lossless` remain env-only via a different mechanism
and are **not** fixed here), and a money-path expression that relied on
`a + b if c else d` grouping.
## Known remaining (deliberately out of scope)
- `Requests: N` still overcounts: the Codex WS forwarder reuses one
`request_id` across every turn (one observed 156x), plus ~18 duplicate
PERF emissions.
- `compression_units.py` / `compression_batches.py` remain unlogged —
this PR *discloses* the blind spot rather than closing it.
- The headline stays **gross**. True net is `11,158,901 - 490,928 =
10,667,973` (3.3%, not 3.4%). Making net the headline lowers every
user's reported savings ~4.4%; that is a product call, not mine, so the
inflation is surfaced beside it instead.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|