fix: make Headroom work behind Zscaler and other TLS-inspecting networks (#3831)

## Description

Customers behind **Zscaler** (and Netskope, Palo Alto, Cisco Secure
Access, …) report that Headroom "cannot work": every upstream request
fails with `CERTIFICATE_VERIFY_FAILED`, and the agent only sees a bare
502.

These gateways inspect TLS by re-signing every certificate with a
company root. IT installs that root in the **OS certificate store**
(macOS Keychain, Windows cert store), which is why browsers, `curl` and
Claude Code keep working. Headroom never read that store:

- The httpx upstream client trusted certifi only.
- On macOS, the stdlib default reads the static file
`/etc/ssl/cert.pem`, not the Keychain.
- The Rust proxy trusted Mozilla's webpki roots only.

Only 3 of about 40 outbound call sites used the shared CA helper. We had
been patching one path at a time (#741, #998, #1308, #3244, #3344,
#3358, #810, #2029). This PR sets **one trust policy, based on the OS
store, used everywhere.**

Closes #3358

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

**1. Verify upstream TLS through the OS store by default**
(`headroom/proxy/ssl_context.py`)
- New `HEADROOM_CERT_STORE`, default `system,bundled` (same shape as
Claude Code's `CLAUDE_CODE_CERT_STORE`). It uses
[`truststore`](https://truststore.readthedocs.io/), which pip has used
by default since 24.2, plus certifi's public roots.
- The same policy covers the HTTP upstream client, both WebSocket
upstreams, and the `urlopen` paths (Copilot auth).
- **The builders now always return a concrete, verifying `SSLContext`**,
never a `bool`. The legacy default is built explicitly as httpx's
`verify=True` equivalent (certifi).
- OS-store contexts are cached per configuration. They set
`VERIFY_X509_PARTIAL_CHAIN` so an exported intermediate still works as a
trust anchor (matching `create_default_context()` on 3.13). On macOS and
Windows the OS verifies the chain, so the Python 3.13 strict-mode
rejection (#1308) no longer happens.
- `HEADROOM_CA_BUNDLE` and `NODE_EXTRA_CA_CERTS` **add** roots.
`SSL_CERT_FILE` and `REQUESTS_CA_BUNDLE` keep their **replacement**
meaning and turn the OS store off.
- `HEADROOM_CERT_STORE=bundled` restores the old behavior. The aliases
`bundle`, `certifi`, `os` and `native` are also accepted.
- `truststore.inject_into_ssl()` (covers `huggingface_hub`, tiktoken and
OTEL) runs only in processes the Headroom CLI owns. The CLI marks them
with `HEADROOM_PROCESS_TRUST=1`, and proxy workers inherit it.
Applications that embed Headroom keep their own process-wide TLS
behavior.
- `truststore` is added as a core dependency: pure Python, with no
dependencies of its own.

**2. Errors that say what's wrong**
(`headroom/proxy/tls_diagnostics.py`)
- When the certificate check fails, the Anthropic, streaming (SSE) and
OpenAI passthrough errors now name who signed the certificate and give
the fix. For example: *"…The certificate was issued by 'O=Zscaler Inc.,
CN=Zscaler Intermediate Root CA': Zscaler is inspecting this connection.
Fix (pick one): …"*
- The issuer comes from a separate handshake that inspects the
certificate only:
  - It sends no data and requires **TLS ≥ 1.2**.
- It has a 4-second timeout and is cached for 5 minutes per host. A
burst of failures shares a single handshake.
  - It uses the same environment-variable proxies as the real client.
  - It targets the host that actually failed.
- **Security:** on the request path it skips hosts that resolve to
non-public addresses. It also connects to the exact address it checked,
instead of looking the host up again. A per-request upstream therefore
can't be used to read an internal service's certificate or to rebind
DNS.

**3. `headroom doctor`**
- A new **tls trust** row reports the running proxy's policy, read from
`/health`.
- A **proxy env** warning appears when `HTTP(S)_PROXY` is set but
`NO_PROXY` doesn't exempt loopback.
- `--network` (plus a repeatable `--network-url`) probes the provider
and model hosts. For each it shows the certificate issuer, whether
Headroom trusts it, and whether a gateway block page came back. On
failure it prints a paste-ready request for IT.

**4. `headroom wrap`**
- Merges `NO_PROXY` and `no_proxy`, appends `127.0.0.1,localhost,::1`,
and writes the result to both, so a corporate proxy doesn't capture the
agent's traffic to Headroom.
- The notice goes to stderr, because the JSON that `--prepare-only`
prints on stdout is parsed by `install.sh`.

**5. Rust proxy** (`crates/headroom-proxy`)
- reqwest now enables `rustls-tls-native-roots`, and
`HEADROOM_CA_BUNDLE` / `NODE_EXTRA_CA_CERTS` add roots.
- **Fixes a bug already on `main`: `wss://` upstreams panicked.** Both
ring and aws-lc-rs are compiled into this binary, so
`rustls::ClientConfig::builder()`, which
`tokio_tungstenite::connect_async` calls, panics with "no process-level
CryptoProvider available". WebSocket upstreams now use an explicit
config: the ring provider, webpki + OS + additive roots, and ALPN
`http/1.1`.

**6. Docs**
- New page **Corporate Networks**: trust settings, HTTP proxies, a
domain table for IT, and notes on Caution/Isolate, SSE buffering,
process bypass, tenant restrictions and `HF_ENDPOINT` mirrors.
- README section updated, correcting a wrong claim about which trust
store the Rust core uses.

**7. `uv.lock`**
- `main`'s lockfile plus `truststore` only. The diff is 11 lines: the
new package entry and two lines in `headroom-ai`'s dependency lists.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_corporate_tls.py tests/test_ssl_context.py tests/test_copilot_auth.py tests/test_cli_doctor.py -q
============================= 278 passed in 2.88s ==============================

$ pytest <117 related files: doctor, wrap, streaming, codex, copilot, ssl, upstream, websocket, passthrough> -q
====== 4 failed, 1876 passed, 25 skipped, 7 warnings in 132.42s (0:02:12) ======
# the 4 failures: tests/test_codex_live.py ordering issue, identical on main (see Additional Notes)

$ cargo test -p headroom-proxy
passed=478 failed=0 suites=37
$ cargo clippy -p headroom-proxy --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s)

$ ruff check .                                            -> All checks passed!
$ ruff format --check .                                   -> 1660 files already formatted
$ mypy headroom --ignore-missing-imports                  -> Success: no issues found in 544 source files
$ uv export --frozen --no-dev --no-hashes | grep truststore -> truststore==0.10.4
```

`tests/test_corporate_tls.py` runs a **fake Zscaler** HTTPS server. Its
root is `O=Zscaler Inc.` with **non-critical basicConstraints**, like
the real 2014 root. The tests check that:

- certifi alone rejects it;
- the default policy plus `HEADROOM_CA_BUNDLE` or `NODE_EXTRA_CA_CERTS`
accepts it;
- the probe names Zscaler, and the request path skips private hosts;
- a streaming request surfaces the actionable message in a 502 SSE
error;
- a burst of failures shares one handshake;
- the notices and JSON output from `wrap` go to the right streams;
- process injection is gated as described.

## Real Behavior Proof

- **Environment:**
  - macOS (Darwin 25.6, Apple Silicon), Python 3.12.13 venv
  - Python 3.13.15 via `uv run --python 3.13`
  - Linux `python:3.13-slim` in Docker
  - Rust proxy built from this branch
- **Steps and results:**
1. **Standalone fake-Zscaler script** (same root as above) on Python
3.13:
- old stdlib path, `ssl.create_default_context(cafile=root)` →
`CERTIFICATE_VERIFY_FAILED: Missing Authority Key Identifier`
     - new `truststore` context + root → `200`
     - identical results on macOS and in `python:3.13-slim`.
2. **`headroom doctor --network`** from this branch on an uninspected
network: every endpoint shows `reachable, certificate trusted`
(api.anthropic.com 401, api.openai.com 401, api.githubcopilot.com 400,
huggingface.co 200, openaipublic 404); the tls trust row shows `OS
certificate store + certifi`.
3. **Rust `wss://` connection** (temporary test) to
`wss://echo.websocket.org`:
- `tokio_tungstenite::connect_async` as on `main` → **panic** (no
process-level CryptoProvider)
     - `connect_async_tls_with_config` with the new config → `Ok(())`
- **Not tested:**
  - A real Zscaler, Netskope or Palo Alto tenant.
- Windows. `truststore` supports the Windows cert store, but no Windows
run was done here.
- A root installed into the OS store. That needs admin rights, so the
additive-bundle path stood in for it; it goes through the same OS
verifier.

## Runtime Rollout Safety

- **Rollout-managed feature(s):** none. This is trust-store selection,
not a rollout-gated feature.
- **Minimum rollout channel:** n/a.
- **Stable/default behavior changed:** yes. Upstream TLS now also trusts
the OS certificate store; certifi is still loaded, so public endpoints
behave the same. The Rust `wss://` upstream no longer panics.
- **Kill switch / disable path:** `HEADROOM_CERT_STORE=bundled` restores
certifi-only trust. An explicit `SSL_CERT_FILE` / `REQUESTS_CA_BUNDLE`
behaves exactly as before.
- **Unsafe override required:** none. Nothing here disables certificate
verification.
- **Qualification impact:** none expected. Default behavior on
uninspected networks is unchanged.
- **Rollback path:** revert the PR, or set
`HEADROOM_CERT_STORE=bundled`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

n/a (CLI and proxy behavior).

## Additional Notes

**Review response** (@JerrettDavis):
- **CodeQL:** the 5 alerts that were still open on `efe173c9` had two
causes.
- **Certificate-validation alerts:** CodeQL traced a `None` returned by
the optional-context helpers through the `configured or default`
fallback into `verify=`. All contexts are now built by a single
`_build_context()`, and every branch returns a verifying `SSLContext`.
`build_httpx_verify()` is typed `-> ssl.SSLContext`.
- **TLS-version alerts:** every context the module creates now sets an
explicit `minimum_version = TLSv1_2`. That includes the
certificate-inspection probe, whose no-validation scope stays confined
to that handshake behind the public-address / environment-proxy CONNECT
guard. The test server also requires TLS 1.2, and the host assertion in
the test is a regex.
  - **Verified locally with CodeQL 2.27.1** before pushing:
- the same 5 alerts reproduce on `efe173c9`, and 0 remain on the fix
commit;
- `python-security-extended` over every Python file this PR changes
finds the same 14 pre-existing alerts on `main`'s versions and on the
branch's versions, so the PR adds none.
- **`uv.lock`:** rebuilt as `main`'s lockfile plus `truststore` only (11
lines). Re-locking with current uv also rewrites markers across the
file, because `main`'s lockfile predates the current `pyproject.toml`.
That is left for a separate lock refresh.

**Earlier review:** a high-effort code review of this branch produced 10
findings plus 3 minor notes. Commit `fix: address review findings…`
resolves 9 of the 10 and two of the notes; the third note (lockfile
markers) checked out as a false alarm. The remaining finding is the last
follow-up below.

**Follow-ups (not in this PR):**
- SSE keep-alives and HTTP/1.1 fallback when a gateway buffers a stream.
- A `wrap` pre-flight that steps aside when the upstream is unreachable.
- The Rust core's fastembed/hf-hub embedding download, which hard-codes
WebPki roots (needs an hf-hub patch; BM25 is the fallback, and
`HF_ENDPOINT` is documented).
- A CI job with a fake root installed in the OS store on macOS and
Windows.
- Map certificate failures once, in a shared layer, so batch, Gemini and
the other handlers get the TLS hint too.

**Known unrelated failures:**
- 4 `tests/test_codex_live.py` tests fail when run after other files in
this set, both on `main` and on this branch. They pass when run on their
own.
- After merging the latest `main` (thanks for the refresh),
`tests/test_cli/test_wrap_helpers.py::test_start_proxy_strips_ambient_worker_configuration`
and `::test_start_proxy_timeout_kills_failed_new_process` fail locally
with `TypeError: <lambda>() takes 0 positional arguments but 1 was
given`. They fail the same way on a clean checkout of current `main`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Tejas Chopra
2026-09-28 18:06:09 -07:00
committed by GitHub
co-authored by Claude Opus 5.5 JerrettDavis
parent bb8c285714
commit 66258c40a4
26 changed files with 2105 additions and 92 deletions
Generated
+6
View File
@@ -1963,6 +1963,9 @@ dependencies = [
"prometheus",
"proptest",
"reqwest",
"rustls",
"rustls-native-certs",
"rustls-pki-types",
"serde",
"serde_json",
"sha2",
@@ -1977,6 +1980,7 @@ dependencies = [
"tracing-subscriber",
"url",
"uuid",
"webpki-roots 0.26.11",
"wiremock",
]
@@ -3642,6 +3646,7 @@ dependencies = [
"pin-project-lite",
"quinn",
"rustls",
"rustls-native-certs",
"rustls-pki-types",
"serde",
"serde_json",
@@ -4451,6 +4456,7 @@ dependencies = [
"futures-util",
"log",
"rustls",
"rustls-native-certs",
"rustls-pki-types",
"tokio",
"tokio-rustls",
+18 -10
View File
@@ -525,11 +525,19 @@ published for Windows (`win_amd64`), Linux (`x86_64` / `aarch64`) and macOS
(Apple Silicon and Intel), so those platforms never need a local Rust toolchain —
the Rust-first step above is only for the sdist fallback when no wheel matches.
Two runtime assets are fetched over TLS. If they are blocked, trust your
corporate CA through `REQUESTS_CA_BUNDLE` / `SSL_CERT_FILE` / `CURL_CA_BUNDLE`:
**At runtime** Headroom verifies upstream TLS against the operating system's
certificate store — where IT installs the Zscaler / Netskope / Palo Alto root —
plus certifi's public roots, so a managed machine normally needs no setup.
`headroom doctor --network` shows who signed each upstream certificate, whether
Headroom trusts it, and whether a gateway block page is in the way. If the root
is not in the OS store, export it as PEM and set
`HEADROOM_CA_BUNDLE=/path/to/root.pem` (`NODE_EXTRA_CA_CERTS` also works). Full
guide and the domain list for IT: [Corporate networks](https://docs.headroomlabs.ai/docs/corporate-networks).
- **`cdn.pyke.io`** — the ONNX Runtime for the Rust core. Or pre-provide it with `ORT_STRATEGY=system` and `ORT_LIB_LOCATION=/path/to/onnxruntime`.
- **`huggingface.co`** — the `kompress-base` model. Pre-download it and run with `HF_HUB_OFFLINE=1`, or point `HF_ENDPOINT` at a trusted mirror.
Two download hosts can still need attention:
- **`cdn.pyke.io`** — the ONNX Runtime for the Rust core, fetched with Mozilla's roots only. Pre-provide it with `ORT_STRATEGY=system` and `ORT_LIB_LOCATION=/path/to/onnxruntime`.
- **`huggingface.co`** — the `kompress-base` model (Python, OS store) and the Rust core's embedding model (Mozilla's roots only). If the domain is blocked, pre-download and run with `HF_HUB_OFFLINE=1`, or point `HF_ENDPOINT` at a trusted mirror.
Running with compression disabled (pure gateway) needs neither asset.
@@ -552,8 +560,9 @@ export ORT_DYLIB_PATH="$(brew --prefix onnxruntime)/lib/libonnxruntime.dylib"
`ORT_PREFER_DYNAMIC_LINK=1` is required — without it `ORT_STRATEGY=system` still
attempts static linking, which the Homebrew keg does not provide.
**"Basic Constraints of CA cert not marked critical"** is a different failure. If
TLS fails with:
**"Basic Constraints of CA cert not marked critical"** only occurs with
`HEADROOM_CERT_STORE=bundled` (the OS store verifies these roots fine). If TLS
fails with:
```
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed:
@@ -575,10 +584,9 @@ signature, expiry and hostname checks all stay on.
HEADROOM_TLS_STRICT=0 headroom proxy --port 8787
```
The Rust core's ONNX download uses a separate TLS stack (rustls / OS trust store)
and is unaffected by `HEADROOM_TLS_STRICT`. On Windows the corporate root must be
in the **machine** certificate store — browsers already trust it there — or
pre-provision ONNX Runtime with `ORT_STRATEGY=system` to skip the download.
The Rust core's downloads use rustls with Mozilla's roots and are unaffected by
`HEADROOM_TLS_STRICT` and the CA variables; pre-provision ONNX Runtime with
`ORT_STRATEGY=system` and models via `HF_ENDPOINT` as above.
</details>
+12 -2
View File
@@ -22,8 +22,11 @@ tower = { workspace = true }
tower-http = { version = "0.7", features = ["trace", "request-id", "util"] }
tracing = { workspace = true }
tracing-subscriber = { version = "0.3", features = ["json", "env-filter", "fmt"] }
reqwest = { version = "0.12", default-features = false, features = ["stream", "rustls-tls", "http2"] }
tokio-tungstenite = { version = "0.30", default-features = false, features = ["connect", "rustls-tls-webpki-roots"] }
# Native roots on top of the bundled webpki set: corporate TLS-inspection
# roots (Zscaler, Netskope, ...) are installed in the OS store, never in
# Mozilla's list, so webpki alone rejects every re-signed upstream certificate.
reqwest = { version = "0.12", default-features = false, features = ["stream", "rustls-tls", "rustls-tls-native-roots", "http2"] }
tokio-tungstenite = { version = "0.30", default-features = false, features = ["connect", "rustls-tls-webpki-roots", "rustls-tls-native-roots"] }
clap = { workspace = true, features = ["derive", "env"] }
serde = { workspace = true }
serde_json = { workspace = true }
@@ -93,6 +96,13 @@ async-trait = "0.1"
# the core uses for the CCR cache_key, so no additional hash backend
# enters the dep tree.
md-5 = "0.11"
# Explicit rustls config for WebSocket upstreams (see src/tls.rs). Both ring
# and aws-lc-rs are compiled into this binary, so rustls cannot pick a default
# provider and `ClientConfig::builder()` panics; we name ring explicitly.
rustls = { version = "0.23", default-features = false, features = ["ring", "std", "tls12", "logging"] }
rustls-native-certs = "0.8"
rustls-pki-types = { version = "1", features = ["std"] }
webpki-roots = "0.26"
[dev-dependencies]
tower = { workspace = true, features = ["util"] }
+1
View File
@@ -13,6 +13,7 @@ pub mod observability;
pub mod proxy;
pub mod responses_items;
pub mod sse;
pub mod tls;
pub mod vertex;
pub mod websocket;
+7 -1
View File
@@ -97,7 +97,13 @@ const DRIFT_DETECTOR_CAPACITY: usize = 1000;
impl AppState {
pub fn new(config: Config) -> Result<Self, ProxyError> {
let client = reqwest::Client::builder()
let mut builder = reqwest::Client::builder();
// Corporate roots handed over as PEM files, added on top of the
// bundled + OS roots (see `crate::tls`).
for cert in crate::tls::extra_root_certificates() {
builder = builder.add_root_certificate(cert);
}
let client = builder
.connect_timeout(config.upstream_connect_timeout)
.timeout(config.upstream_timeout)
// Don't auto-follow redirects: pass them through verbatim.
+183
View File
@@ -0,0 +1,183 @@
//! Upstream TLS trust for the Rust proxy.
//!
//! Upstreams are verified against Mozilla's bundled roots plus the operating
//! system store, which is where IT installs a corporate TLS-inspection root
//! (Zscaler, Netskope, ...). Deployments without such a store (containers,
//! CI) hand the root over as a PEM file instead; these env vars add it,
//! matching the Python proxy's policy:
//!
//! * `HEADROOM_CA_BUNDLE` — Headroom's own knob.
//! * `NODE_EXTRA_CA_CERTS` — what Claude Code users already set.
//!
//! Both are additive. `SSL_CERT_FILE` / `SSL_CERT_DIR` are honored by
//! `rustls-native-certs` itself, as a replacement for the OS store.
//!
//! HTTP upstreams (reqwest) get the extras via [`extra_root_certificates`];
//! WebSocket upstreams get a complete rustls config from
//! [`websocket_tls_config`].
use std::path::Path;
use std::sync::{Arc, OnceLock};
use rustls_pki_types::pem::PemObject;
use rustls_pki_types::CertificateDer;
/// Env vars naming additive PEM bundles, in the order they are loaded.
pub const ADDITIVE_CA_VARS: [&str; 2] = ["HEADROOM_CA_BUNDLE", "NODE_EXTRA_CA_CERTS"];
/// Root certificates from the additive env vars, for reqwest. Missing or
/// unparsable files are logged and skipped: a bad path must not take the
/// proxy down.
pub fn extra_root_certificates() -> Vec<reqwest::Certificate> {
extra_root_ders()
.iter()
.filter_map(|der| reqwest::Certificate::from_der(der.as_ref()).ok())
.collect()
}
/// rustls client config for `wss://` upstreams: webpki + OS + additive roots,
/// on an explicit crypto provider. Built once per process.
pub fn websocket_tls_config() -> Arc<rustls::ClientConfig> {
static CONFIG: OnceLock<Arc<rustls::ClientConfig>> = OnceLock::new();
CONFIG
.get_or_init(|| Arc::new(build_websocket_tls_config(&extra_root_ders())))
.clone()
}
fn build_websocket_tls_config(extra: &[CertificateDer<'static>]) -> rustls::ClientConfig {
let mut roots = rustls::RootCertStore::empty();
roots.extend(webpki_roots::TLS_SERVER_ROOTS.iter().cloned());
let native = rustls_native_certs::load_native_certs();
if !native.errors.is_empty() {
tracing::warn!(
event = "tls_native_roots_errors",
errors = ?native.errors,
"some OS root certificates could not be loaded"
);
}
roots.add_parsable_certificates(native.certs);
roots.add_parsable_certificates(extra.iter().cloned());
let provider = Arc::new(rustls::crypto::ring::default_provider());
let mut config = rustls::ClientConfig::builder_with_provider(provider)
.with_safe_default_protocol_versions()
.expect("ring supports the default TLS versions")
.with_root_certificates(roots)
.with_no_client_auth();
// A WebSocket upgrade is an HTTP/1.1 request.
config.alpn_protocols = vec![b"http/1.1".to_vec()];
config
}
fn extra_root_ders() -> Vec<CertificateDer<'static>> {
extra_root_ders_from(|var| std::env::var(var).ok())
}
fn extra_root_ders_from(lookup: impl Fn(&str) -> Option<String>) -> Vec<CertificateDer<'static>> {
let mut certs = Vec::new();
for var in ADDITIVE_CA_VARS {
let Some(path) = lookup(var).filter(|p| !p.is_empty()) else {
continue;
};
match load_pem_bundle(Path::new(&path)) {
Ok(found) => {
tracing::info!(
event = "tls_ca_bundle_loaded",
env_var = var,
path = %path,
certificates = found.len(),
"loaded extra upstream root certificates"
);
certs.extend(found);
}
Err(err) => {
tracing::warn!(
event = "tls_ca_bundle_skipped",
env_var = var,
path = %path,
error = %err,
"could not load extra root certificates; skipping"
);
}
}
}
certs
}
fn load_pem_bundle(path: &Path) -> Result<Vec<CertificateDer<'static>>, String> {
let certs = CertificateDer::pem_file_iter(path)
.map_err(|e| e.to_string())?
.collect::<Result<Vec<_>, _>>()
.map_err(|e| e.to_string())?;
if certs.is_empty() {
return Err("no PEM certificates found".to_string());
}
Ok(certs)
}
#[cfg(test)]
mod tests {
use super::*;
// Throwaway self-signed CA, only parsed here, never used for a handshake.
const TEST_CA: &str = "-----BEGIN CERTIFICATE-----
MIIBuTCCAV+gAwIBAgIUQJOjUYts91bSLsemwb5EuzdkOnMwCgYIKoZIzj0EAwIw
MjEVMBMGA1UECgwMWnNjYWxlciBJbmMuMRkwFwYDVQQDDBBIZWFkcm9vbSBUZXN0
IENBMB4XDTI2MDkyODE0MjM1MloXDTM2MDkyNTE0MjM1MlowMjEVMBMGA1UECgwM
WnNjYWxlciBJbmMuMRkwFwYDVQQDDBBIZWFkcm9vbSBUZXN0IENBMFkwEwYHKoZI
zj0CAQYIKoZIzj0DAQcDQgAEKZ0h9e4jj/eJiBVh4eMZ3d+pcugxj/hEhqzoAo9r
+7KL4dGgqrg2GH4IP6sfKNdssKHshDcjz+HWiKVGyCx08KNTMFEwHQYDVR0OBBYE
FPFmGy3B6aRYxGMB1P/up4Y05PxbMB8GA1UdIwQYMBaAFPFmGy3B6aRYxGMB1P/u
p4Y05PxbMA8GA1UdEwEB/wQFMAMBAf8wCgYIKoZIzj0EAwIDSAAwRQIgZTHC/ZR0
SI8i1dYBBaT8a2e/CVF6MmgXObYWEg2YYk8CIQC8e2+jUZDkYV+Ct3y0JQO7OSfp
+Yyjbym7JFsfsp4fVQ==
-----END CERTIFICATE-----
";
fn write_temp(name: &str, contents: &str) -> String {
let dir = std::env::temp_dir().join(format!("headroom-tls-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
let path = dir.join(name);
std::fs::write(&path, contents).unwrap();
path.display().to_string()
}
#[test]
fn unset_and_empty_vars_add_nothing() {
assert!(extra_root_ders_from(|_| None).is_empty());
assert!(extra_root_ders_from(|_| Some(String::new())).is_empty());
}
#[test]
fn missing_file_is_skipped() {
let certs = extra_root_ders_from(|var| {
(var == "HEADROOM_CA_BUNDLE").then(|| "/nonexistent/headroom-ca.pem".to_string())
});
assert!(certs.is_empty());
}
#[test]
fn non_pem_file_is_skipped() {
let p = write_temp("not-a-cert.pem", "hello");
let certs = extra_root_ders_from(|var| (var == "NODE_EXTRA_CA_CERTS").then(|| p.clone()));
assert!(certs.is_empty());
}
#[test]
fn both_additive_vars_are_loaded() {
let p = write_temp("corp-root.pem", TEST_CA);
let certs = extra_root_ders_from(|_| Some(p.clone()));
assert_eq!(certs.len(), ADDITIVE_CA_VARS.len());
assert!(reqwest::Certificate::from_der(certs[0].as_ref()).is_ok());
}
#[test]
fn websocket_config_builds_without_a_process_default_provider() {
// rustls' own `ClientConfig::builder()` panics in this binary (two
// providers compiled in); ours must not, and must trust the extras.
let p = write_temp("corp-root-ws.pem", TEST_CA);
let extra = extra_root_ders_from(|var| (var == "HEADROOM_CA_BUNDLE").then(|| p.clone()));
let config = build_websocket_tls_config(&extra);
assert_eq!(config.alpn_protocols, vec![b"http/1.1".to_vec()]);
}
}
+9 -3
View File
@@ -151,9 +151,15 @@ async fn run_ws_pump(
}
}
let (upstream_ws, _resp) = tokio_tungstenite::connect_async(req)
.await
.map_err(|e| format!("upstream ws connect: {e}"))?;
// Explicit TLS config: corporate roots (OS store + HEADROOM_CA_BUNDLE) and
// a named crypto provider. tokio-tungstenite's default path calls
// `rustls::ClientConfig::builder()`, which panics in this binary because
// two rustls providers are compiled in.
let connector = tokio_tungstenite::Connector::Rustls(crate::tls::websocket_tls_config());
let (upstream_ws, _resp) =
tokio_tungstenite::connect_async_tls_with_config(req, None, false, Some(connector))
.await
.map_err(|e| format!("upstream ws connect: {e}"))?;
let (mut upstream_sink, mut upstream_stream) = upstream_ws.split();
let (mut client_sink, mut client_stream) = client_ws.split();
+97
View File
@@ -0,0 +1,97 @@
---
title: Corporate Networks (Zscaler, Netskope, TLS Inspection)
description: Run Headroom behind Zscaler, Netskope, Palo Alto, Cisco Secure Access and other TLS-inspecting gateways, and what to ask IT for when something is still blocked.
---
Many companies route laptop traffic through a gateway that **inspects TLS**: it opens each HTTPS connection, checks it, and re-signs it with the company's own root certificate. IT installs that root in the operating system's certificate store, which is why browsers, `curl` and Claude Code keep working.
Headroom verifies upstream certificates against the **same OS certificate store** (macOS Keychain, Windows certificate store, the system bundle on Linux), plus the public roots bundled with Python. On a managed machine this usually needs no configuration.
## Quick check
```bash
headroom doctor --network
```
For each endpoint Headroom uses, this prints who signed the certificate the network presented (for example `Zscaler Intermediate Root CA`), whether Headroom trusts it, and whether a gateway block page was returned instead of the service. If anything fails, the last row is a short request you can send to IT as-is.
Add `--network-url https://your-gateway.example.com/v1/models` to test a custom upstream too.
## If a certificate is still not trusted
The error Claude Code (or Codex) shows will read like:
> Headroom could not verify the TLS certificate for api.anthropic.com (unable to get local issuer certificate). The certificate was issued by 'O=Zscaler Inc., CN=Zscaler Intermediate Root CA': Zscaler is inspecting this connection. ...
Pick one fix:
1. **Ask IT to deploy the root to the OS certificate store.** This is the normal setup for Zscaler Client Connector ("Install Zscaler SSL Certificate") and most MDM profiles. Nothing else to change.
2. **Point Headroom at the root as a file.** Export it (your browser's certificate viewer, or IT) as PEM and set:
```bash
export HEADROOM_CA_BUNDLE=/path/to/company-root.pem
```
`HEADROOM_CA_BUNDLE` and `NODE_EXTRA_CA_CERTS` are **added** to the trusted set. If you already set `NODE_EXTRA_CA_CERTS` for Claude Code, Headroom picks it up.
3. **Ask IT to exempt the provider domains from TLS inspection** (see the list below).
<Callout type="warn" title="Don't disable verification">
Never turn certificate checking off to get past this. Headroom has no switch for that on purpose.
</Callout>
## Trust settings
| Env var | Default | Effect |
|---|---|---|
| `HEADROOM_CERT_STORE` | `system,bundled` | Where trusted roots come from: `system` = OS certificate store, `bundled` = certifi's public roots. Set `bundled` to go back to the previous behavior (public roots only). Mirrors Claude Code's `CLAUDE_CODE_CERT_STORE`. |
| `HEADROOM_CA_BUNDLE` | unset | Extra PEM roots, added to the above. |
| `NODE_EXTRA_CA_CERTS` | unset | Same as `HEADROOM_CA_BUNDLE` (shared with Claude Code). |
| `SSL_CERT_FILE` / `REQUESTS_CA_BUNDLE` | unset | **Replace** every other source: only this bundle is trusted. Use only if the bundle is complete. |
| `HEADROOM_TLS_STRICT` | strict | `0` relaxes OpenSSL's RFC 5280 strict checks on the bundled path. Not needed with the OS store; kept for `HEADROOM_CERT_STORE=bundled`. |
The same policy covers the proxy's HTTP and WebSocket upstreams, GitHub Copilot sign-in, and model and tokenizer downloads made from Python. The Rust proxy binary (`headroom-proxy`, used in the Docker image) trusts Mozilla's roots plus the OS store for all upstreams, and adds `HEADROOM_CA_BUNDLE` / `NODE_EXTRA_CA_CERTS` for HTTP upstreams.
One gap remains: the embedding model that the Rust core fetches for relevance scoring still trusts Mozilla's roots only. Behind inspection that download fails and Headroom falls back to BM25 scoring; use `HF_ENDPOINT` (below) or pre-populate the cache to keep embeddings.
In a container there is usually no corporate root in the OS store: mount it and set `HEADROOM_CA_BUNDLE`.
## HTTP proxies
If your company uses an explicit proxy (`HTTPS_PROXY` set, or a PAC file), Headroom's upstream calls go through it. The agent's calls **to** Headroom must not: they target `127.0.0.1`, which the corporate proxy cannot reach. `headroom wrap` adds `127.0.0.1,localhost,::1` to `NO_PROXY` for the processes it launches; if you start the agent yourself, set:
```bash
export NO_PROXY="$NO_PROXY,127.0.0.1,localhost,::1"
```
PAC files are not evaluated by Python. Export the proxy the PAC file resolves to as `HTTPS_PROXY`.
## What to ask IT for
Headroom runs on the developer's machine and talks to the same services the AI tools already use, plus two download hosts:
| Domain | Used for | Required |
|---|---|---|
| `api.anthropic.com` | Claude / Claude Code | yes, for Anthropic |
| `api.openai.com`, `chatgpt.com` | OpenAI API, Codex | yes, for OpenAI / Codex |
| `api.githubcopilot.com`, `api.github.com` | GitHub Copilot | for Copilot |
| `huggingface.co` and the storage hosts it redirects to (`cdn-lfs.hf.co`, `cas-bridge.xethub.hf.co`) | Compression models (downloaded once) | recommended |
| `openaipublic.blob.core.windows.net` | Tokenizer vocabulary (downloaded once) | recommended |
Things that help:
- **Deploy the inspection root to the OS store.** This is enough for Headroom; no per-app certificate work is needed.
- **Do not route these domains to "Caution" or "Isolate".** An API client cannot click through an interstitial page; it sees HTML where it expected JSON.
- **Streaming:** some gateways buffer server-sent events while inspecting them, so responses arrive in bursts or time out. If users see stalled streams, exempt the provider domains from inspection or DLP buffering.
- **Please don't exempt `python` as a process** (for example Zscaler's process-based bypass). Headroom does not need it once the root is trusted, and it would exempt every Python program on the machine.
If your company uses Anthropic **tenant restrictions**, those require inspection of `api.anthropic.com` to stay on. That works with Headroom as long as the root is trusted.
## Model downloads when huggingface.co is blocked
Headroom degrades instead of failing: without the compression model it falls back to rule-based compression, and without the tokenizer vocabulary it estimates token counts. To keep ML compression on a locked-down network, point Headroom at an internal Hugging Face mirror (Artifactory, Nexus):
```bash
export HF_ENDPOINT=https://artifactory.example.com/api/huggingface/huggingface
```
Or pre-populate `~/.cache/huggingface` from a machine that has access.
+1
View File
@@ -24,6 +24,7 @@
"failure-learning",
"---Proxy Server---",
"proxy",
"corporate-networks",
"local-llm-prefill",
"---Integrations---",
"vercel-ai-sdk",
+2
View File
@@ -339,6 +339,8 @@ with a privileged job.
| `--stateless` / `HEADROOM_STATELESS` | `false` | Keep all state in memory; no filesystem writes (disables logs, memory, TOIN). |
| `HEADROOM_STRIP_INTERNAL_HEADERS` | `enabled` | Strip internal `x-headroom-*` headers before forwarding upstream. |
| `HEADROOM_TLS_STRICT` | strict | Set `0` to relax CA-constraint checks behind a corporate TLS-inspection proxy. |
| `HEADROOM_CERT_STORE` | `system,bundled` | Trusted roots for upstream TLS: OS certificate store and/or certifi. See [Corporate Networks](/docs/corporate-networks). |
| `HEADROOM_CA_BUNDLE` | unset | Extra PEM roots (e.g. a Zscaler root) added to the trusted set. |
### Performance
+193 -1
View File
@@ -482,6 +482,167 @@ def _codex_block_missing_openai_auth(
return False
def check_trust_policy(policy: Mapping[str, Any]) -> CheckResult:
"""Which certificates the proxy trusts for its upstream connections.
Corporate TLS inspection (Zscaler, Netskope, ...) re-signs every upstream
certificate with a company root that IT installs in the OS store. The proxy
only accepts it when the OS store is in use or the root is supplied as a
bundle, so this line answers "will Headroom trust what my network presents?"
"""
replacement = policy.get("replacement_bundle")
additive = policy.get("additive_bundles") or []
extras = ", ".join(f"{b['env_var']}={b['path']}" for b in additive)
if replacement:
return CheckResult(
name="tls trust",
status=PASS,
summary=f"only {replacement['env_var']}={replacement['path']}",
hint=(
f"{replacement['env_var']} replaces every other trust source; the OS "
"certificate store is not consulted. Unset it to trust the OS store."
),
)
if policy.get("system_store_active"):
sources = "OS certificate store" + (
" + certifi" if "bundled" in policy.get("cert_store", []) else ""
)
return CheckResult(
name="tls trust",
status=PASS,
summary=sources + (f" + {extras}" if extras else ""),
)
reason = (
"HEADROOM_CERT_STORE excludes 'system'"
if policy.get("system_store_available")
else "the truststore package is not installed"
)
return CheckResult(
name="tls trust",
status=WARN,
summary="certifi bundle only" + (f" + {extras}" if extras else ""),
hint=(
f"The OS certificate store is not used ({reason}), so a corporate "
"TLS-inspection root is only trusted if you pass it via HEADROOM_CA_BUNDLE."
),
)
def check_proxy_env(environ: Mapping[str, str]) -> CheckResult | None:
"""Warn when an HTTP(S) proxy would capture the agent's loopback traffic."""
from headroom.proxy.tls_diagnostics import loopback_no_proxy_gap
gap = loopback_no_proxy_gap(environ)
if gap is None:
return None
return CheckResult(
name="proxy env",
status=WARN,
summary=f"{gap} is set but NO_PROXY does not exempt 127.0.0.1/localhost",
hint=(
"Agents may send their requests to Headroom through the corporate proxy, "
"which cannot reach this machine. `headroom wrap` fixes this for the "
"processes it launches; for other setups add "
"NO_PROXY=127.0.0.1,localhost,::1."
),
)
# (label, url, required). Required endpoints fail the check; the rest only warn
# because Headroom degrades (no ML compression, estimated token counts) rather
# than breaking when they are unreachable.
NETWORK_ENDPOINTS: tuple[tuple[str, str, bool], ...] = (
("api.anthropic.com", "https://api.anthropic.com/v1/models", True),
("api.openai.com", "https://api.openai.com/v1/models", True),
("api.githubcopilot.com", "https://api.githubcopilot.com/models", False),
("huggingface.co (models)", "https://huggingface.co/api/models?limit=1", False),
(
"openaipublic (tiktoken)",
"https://openaipublic.blob.core.windows.net/encodings/",
False,
),
)
def check_network_endpoints(reports: Sequence[Any], required: set[str]) -> list[CheckResult]:
"""Turn :class:`EndpointReport`s into doctor rows, plus an IT summary row."""
results: list[CheckResult] = []
vendors: set[str] = set()
failing_hosts: list[str] = []
for report in reports:
vendor = report.chain.inspection_vendor
if vendor:
vendors.add(vendor)
inspected = f"; TLS inspected by {vendor}" if vendor else ""
host = urlsplit_host(report.url)
if report.ok:
results.append(
CheckResult(
name=f"net {report.name}",
status=PASS,
summary=(
f"reachable, certificate trusted (HTTP {report.status}, "
f"{report.elapsed_ms:.0f} ms){inspected}"
),
)
)
continue
failing_hosts.append(host)
status = FAIL if report.name in required else WARN
if report.block_page:
summary = f"blocked by a network gateway (HTTP {report.status}){inspected}"
elif report.chain.issuer and "verify" in (report.error or "").lower():
summary = f"certificate not trusted (issued by {report.chain.issuer})"
else:
summary = (report.error or "unreachable").split(". ")[0][:160]
results.append(
CheckResult(
name=f"net {report.name}", status=status, summary=summary, hint=report.error
)
)
if failing_hosts:
who = " / ".join(sorted(vendors)) or "your TLS-inspection gateway"
results.append(
CheckResult(
name="it request",
status=WARN,
summary=f"{len(failing_hosts)} endpoint(s) need a network change",
hint=(
f"Text for your IT team: Headroom (a local AI coding proxy) needs HTTPS "
f"access to {', '.join(failing_hosts)}. Either confirm the {who} root "
"certificate is deployed to the operating system certificate store on "
"developer machines, or exempt these domains from TLS inspection and "
"cloud-app blocking. Please do not exempt 'python' as a process; the "
"certificate fix is sufficient."
),
)
)
return results
def urlsplit_host(url: str) -> str:
from urllib.parse import urlsplit
return urlsplit(url).hostname or url
def run_network_probes(extra_urls: Sequence[str]) -> tuple[list[Any], set[str]]:
"""Probe every endpoint concurrently; returns (reports, required names)."""
from concurrent.futures import ThreadPoolExecutor
from headroom.proxy.tls_diagnostics import probe_endpoint
targets = [(label, url) for label, url, _ in NETWORK_ENDPOINTS]
required = {label for label, _, req in NETWORK_ENDPOINTS if req}
for url in extra_urls:
label = urlsplit_host(url)
targets.append((label, url))
required.add(label)
with ThreadPoolExecutor(max_workers=len(targets)) as pool:
reports = list(pool.map(lambda t: probe_endpoint(*t), targets))
return reports, required
def check_shell_env(environ: Mapping[str, str], port: int) -> CheckResult:
"""Is the *current shell* pointed at the proxy for ad-hoc runs?"""
name = "shell env"
@@ -751,7 +912,23 @@ def _render(checks: list[CheckResult], port: int, installed: str) -> None:
help="Proxy port to check (default: 8787, env: HEADROOM_PORT)",
)
@click.option("--json", "emit_json", is_flag=True, help="Emit JSON instead of formatted output.")
def doctor(port: int, emit_json: bool) -> None:
@click.option(
"--network",
is_flag=True,
help=(
"Also test HTTPS to the provider APIs and model hosts: who signed the "
"certificate (Zscaler, Netskope, ...), whether Headroom trusts it, and "
"whether a gateway block page is in the way."
),
)
@click.option(
"--network-url",
"network_urls",
multiple=True,
metavar="URL",
help="Extra upstream URL to test with --network (repeatable), e.g. a custom gateway.",
)
def doctor(port: int, emit_json: bool, network: bool, network_urls: tuple[str, ...]) -> None:
"""Check that the Headroom proxy and client routing are working.
\b
@@ -783,6 +960,21 @@ def doctor(port: int, emit_json: bool) -> None:
check_savings(stats, savings_path()),
check_budget(stats),
]
from headroom.proxy.ssl_context import describe_trust_policy
# Prefer the running proxy's own policy (its env can differ from this
# shell's under launchd/systemd); fall back to what this shell would use.
proxy_tls = ((health or {}).get("config") or {}).get("tls")
trust_row = check_trust_policy(proxy_tls or describe_trust_policy())
if not proxy_tls:
trust_row.summary += " (this shell; proxy not reporting)"
checks.append(trust_row)
proxy_env_check = check_proxy_env(os.environ)
if proxy_env_check is not None:
checks.append(proxy_env_check)
if network or network_urls:
reports, required = run_network_probes(network_urls)
checks.extend(check_network_endpoints(reports, required))
auth_conflict_check = check_claude_auth_conflict(
claude_settings_path(),
project_claude_settings,
+10
View File
@@ -45,6 +45,16 @@ def main(ctx: click.Context) -> None:
except Exception: # noqa: BLE001 — settings load must never break the CLI
pass
# Verify TLS through the OS trust store for every client in this process
# (after settings.json, which may set HEADROOM_CERT_STORE). Corporate
# TLS-inspection roots live there; certifi does not have them.
try:
from headroom.proxy.ssl_context import ensure_process_trust
ensure_process_trust()
except Exception: # noqa: BLE001 — trust wiring must never break the CLI
pass
# Fire a rate-limited, opt-out background check for newer releases so other
# surfaces (e.g. the proxy banner) can show an "update available" notice.
# Never blocks, never raises; skipped for `update` (it checks explicitly).
+42 -1
View File
@@ -33,7 +33,7 @@ import subprocess
import sys
import time
import urllib.parse
from collections.abc import Callable, Mapping
from collections.abc import Callable, Mapping, MutableMapping
from contextlib import contextmanager
from functools import wraps
from pathlib import Path
@@ -5194,6 +5194,47 @@ def wrap(ctx: click.Context) -> None:
"""
if _should_purge_context_tools(ctx):
_report_context_tool_purge()
# Every wrap path copies os.environ into the launched agent and the proxy,
# and probes the proxy over loopback itself, so one fix here covers all.
added = _ensure_loopback_no_proxy(os.environ)
if added:
# stderr: `wrap <tool> --prepare-only` prints machine-read JSON on stdout.
click.echo(
f" Added 127.0.0.1,localhost,::1 to {'/'.join(added)} so the agent reaches "
"Headroom directly instead of through your HTTP proxy.",
err=True,
)
_LOOPBACK_NO_PROXY_HOSTS = ("127.0.0.1", "localhost", "::1")
def _ensure_loopback_no_proxy(env: MutableMapping[str, str]) -> list[str]:
"""Exempt loopback from ``HTTP(S)_PROXY`` in ``env``; return the vars written.
With a corporate proxy exported and no loopback entry in ``NO_PROXY``, an
agent sends its ``http://127.0.0.1:<port>`` requests to the corporate proxy,
which cannot reach this machine and answers with a block page or 502.
``NO_PROXY`` and ``no_proxy`` are merged before writing both, because tools
disagree on which spelling wins and dropping either side's entries would
re-route the user's own internal hosts.
"""
from headroom.proxy.tls_diagnostics import loopback_no_proxy_gap
if loopback_no_proxy_gap(env) is None:
return []
merged: list[str] = []
for var in ("NO_PROXY", "no_proxy"):
for entry in env.get(var, "").split(","):
entry = entry.strip()
if entry and entry not in merged:
merged.append(entry)
merged.extend(h for h in _LOOPBACK_NO_PROXY_HOSTS if h not in merged)
value = ",".join(merged)
written = [var for var in ("NO_PROXY", "no_proxy") if env.get(var) != value]
for var in written:
env[var] = value
return written
@main.group()
+2 -1
View File
@@ -23,6 +23,7 @@ from headroom.proxy.helpers import (
merge_extra_headers,
sanitize_forwarded_response_headers,
)
from headroom.proxy.ssl_context import build_websocket_ssl
from headroom.proxy.ws_headers import WS_HOP_BY_HOP_HEADERS
logger = logging.getLogger("headroom.providers.codex.live")
@@ -226,7 +227,7 @@ async def handle_codex_live_websocket(
additional_headers=forwarded_headers,
# websockets types wire protocol tokens as a nominal wrapper.
subprotocols=cast(Any, subprotocols or None),
ssl=True if upstream_url.startswith("wss://") else None,
ssl=build_websocket_ssl() if upstream_url.startswith("wss://") else None,
open_timeout=max(30, getattr(config, "connect_timeout_seconds", 10) * 3),
close_timeout=10,
ping_interval=20,
+15 -1
View File
@@ -5106,6 +5106,19 @@ class AnthropicHandlerMixin:
# Log full error details internally for debugging
logger.error(f"[{request_id}] Request failed: {type(e).__name__}: {e}")
# An untrusted TLS-inspection root is the one failure worth
# spelling out: it is environmental, never transient, and the
# message names only the certificate issuer and the fix.
from headroom.proxy.tls_diagnostics import describe_upstream_failure_async
# Probe the host that actually failed: `url` is the per-request
# upstream (Copilot, a custom gateway) once it has been built;
# an exception raised before that point never reached a host.
_failed_url = locals().get("url")
tls_hint = await describe_upstream_failure_async(
e, _failed_url if isinstance(_failed_url, str) else self.ANTHROPIC_API_URL
)
# Return sanitized error message to client (don't expose internal details)
return JSONResponse(
status_code=502,
@@ -5113,7 +5126,8 @@ class AnthropicHandlerMixin:
"type": "error",
"error": {
"type": "api_error",
"message": "An error occurred while processing your request. Please try again.",
"message": tls_hint
or "An error occurred while processing your request. Please try again.",
},
},
)
+23 -6
View File
@@ -15,6 +15,7 @@ import logging
import os
import re
import secrets
import ssl
import threading
import time
import uuid
@@ -7227,10 +7228,15 @@ class OpenAIHandlerMixin:
# --- Connect to upstream OpenAI WebSocket ---
logger.info(f"[{request_id}] WS /v1/responses connecting to {upstream_url}")
# Use ssl=True to let the websockets library handle SSL natively.
# Manual ssl.create_default_context() + certifi doesn't load the
# Windows system cert store, causing HTTP 500 on wss:// connections.
use_ssl: bool | None = True if upstream_url.startswith("wss://") else None
# Same trust policy as the HTTP upstream client: the OS trust store
# (Windows machine store, macOS Keychain) plus any configured
# corporate bundle. Plain certifi misses both, causing HTTP 500 on
# wss:// connections behind TLS inspection.
from headroom.proxy.ssl_context import build_websocket_ssl
use_ssl: ssl.SSLContext | bool | None = (
build_websocket_ssl() if upstream_url.startswith("wss://") else None
)
ws_connected = False
ws_connect_attempts = max(1, getattr(self.config, "retry_max_attempts", 3))
@@ -9296,6 +9302,11 @@ class OpenAIHandlerMixin:
f"[{request_id}] WS upstream failed ({_ws_detail}), "
f"falling back to HTTP POST streaming"
)
# Logs the certificate issuer + fix once per host when the
# failure is an untrusted TLS-inspection root.
from headroom.proxy.tls_diagnostics import describe_upstream_failure_async
await describe_upstream_failure_async(ws_err, upstream_url)
(
fb_input_tokens,
fb_output_tokens,
@@ -10967,12 +10978,15 @@ class OpenAIHandlerMixin:
url,
e,
)
from headroom.proxy.tls_diagnostics import describe_upstream_failure_async
tls_hint = await describe_upstream_failure_async(e, url)
return Response(
content=json.dumps(
{
"error": {
"type": "connection_error",
"message": f"Failed to connect to upstream API: {e}",
"message": tls_hint or f"Failed to connect to upstream API: {e}",
}
}
),
@@ -11155,12 +11169,15 @@ class OpenAIHandlerMixin:
url,
e,
)
from headroom.proxy.tls_diagnostics import describe_upstream_failure_async
tls_hint = await describe_upstream_failure_async(e, url)
return Response(
content=json.dumps(
{
"error": {
"type": "connection_error",
"message": f"Failed to connect to upstream API: {e}",
"message": tls_hint or f"Failed to connect to upstream API: {e}",
}
}
),
+8 -1
View File
@@ -1057,13 +1057,20 @@ class StreamingMixin:
error_msg = str(e) or repr(e)
logger.error(f"[{request_id}] Connection error to upstream API: {error_msg}")
self.metrics.record_upstream_connection_error(provider)
# A corporate TLS-inspection root the proxy does not trust shows up
# here as CERTIFICATE_VERIFY_FAILED. Name the issuer and the fix so
# the agent's error line is actionable instead of a bare 502.
from headroom.proxy.tls_diagnostics import describe_upstream_failure_async
tls_hint = await describe_upstream_failure_async(e, url)
client_message = tls_hint or f"Failed to connect to upstream API: {error_msg}"
async def _error_gen():
error_event = {
"type": "error",
"error": {
"type": "connection_error",
"message": f"Failed to connect to upstream API: {error_msg}",
"message": client_message,
},
}
yield f"event: error\ndata: {json.dumps(error_event)}\n\n".encode()
+15 -3
View File
@@ -182,7 +182,11 @@ from headroom.proxy.rate_limiter import TokenBucketRateLimiter # noqa: F401
from headroom.proxy.request_logger import RequestLogger # noqa: F401
from headroom.proxy.savings_tracker import LITELLM_AVAILABLE
from headroom.proxy.semantic_cache import SemanticCache # noqa: F401
from headroom.proxy.ssl_context import build_httpx_verify
from headroom.proxy.ssl_context import (
build_httpx_verify,
describe_trust_policy,
ensure_process_trust_if_owned,
)
from headroom.proxy.tool_schema_savings_policy import tool_schema_saved_from_tags
from headroom.proxy.upstream_pinning import install_upstream_pinning
from headroom.proxy.warmup import WarmupRegistry
@@ -1964,9 +1968,14 @@ class HeadroomProxy(
operation="proxy.startup",
metadata={"port": self.config.port, "host": self.config.host},
)
# Resolve TLS verification: a custom CA bundle (corporate PKI) if one
# is configured, else a strict-relaxed default context when
# Resolve TLS verification: the OS trust store (where IT installs a
# corporate TLS-inspection root) plus certifi by default, else a custom
# CA bundle, else a strict-relaxed default context when
# HEADROOM_TLS_STRICT=0, else httpx's default strict verification.
# Injection covers the third-party clients that build their own
# contexts; it also runs here because uvicorn workers can import the
# app without going through the CLI (they inherit the CLI's marker).
ensure_process_trust_if_owned()
_verify = build_httpx_verify()
_http2, _client_kwargs = _provider_httpx_client_options(self.config, _verify)
# `install_upstream_pinning` is what makes the SSRF guard's verdict
@@ -3535,6 +3544,9 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
profile_kwargs.get("target_ratio", config.target_ratio),
)
payload["config"] = {
# What *this* process trusts upstream; doctor reads it so it
# reports the proxy's policy, not the calling shell's.
"tls": describe_trust_policy(),
"backend": config.backend,
"optimize": config.optimize,
"cache": config.cache_enabled,
+355 -53
View File
@@ -1,16 +1,27 @@
"""SSL context builder for the Headroom upstream httpx client.
"""TLS trust policy for every outbound connection Headroom makes.
Respects the standard CA-bundle environment variables used by Python
(``SSL_CERT_FILE``), requests (``REQUESTS_CA_BUNDLE``), and Node.js /
Claude Code (``NODE_EXTRA_CA_CERTS``) so that enterprise / corporate
deployments with custom certificate authorities work without extra
configuration.
Corporate networks (Zscaler, Netskope, Palo Alto, Cisco Secure Access, ...)
inspect TLS by re-signing every certificate with a company root. IT installs
that root in the **operating system** trust store (macOS Keychain, Windows
certificate store), which is why browsers, curl and Claude Code keep working.
Python does not read that store: httpx trusts certifi's bundle and the stdlib
reads a static OpenSSL file, so Headroom used to reject the re-signed chain
and fail every upstream request with ``CERTIFICATE_VERIFY_FAILED``.
Priority order (first match wins):
Default policy (``HEADROOM_CERT_STORE=system,bundled``, mirroring Claude
Code's ``CLAUDE_CODE_CERT_STORE``): verify through the OS trust store via
``truststore`` *and* trust certifi's public bundle. On macOS and Windows the
OS verifier does the chain check, so a corporate root that IT pushed to the
machine just works and OpenSSL's strict-mode quirks never apply.
Sources, in the order they are resolved:
1. ``SSL_CERT_FILE`` — replacement semantics (only these CAs are trusted)
2. ``REQUESTS_CA_BUNDLE`` — replacement semantics
3. ``NODE_EXTRA_CA_CERTS`` — **additive** semantics (extra roots loaded
on top of the default/system trust store, matching Node.js behavior)
3. Otherwise the ``HEADROOM_CERT_STORE`` sources (``system``, ``bundled``),
plus the **additive** bundles ``HEADROOM_CA_BUNDLE`` and
``NODE_EXTRA_CA_CERTS`` (extra roots on top, matching Node.js behavior).
``HEADROOM_CERT_STORE=bundled`` restores the pre-0.40 behavior (no OS store).
Strict-mode toggle (``HEADROOM_TLS_STRICT``):
Python 3.13 + OpenSSL 3.x enable ``VERIFY_X509_STRICT`` by default, which
@@ -34,10 +45,37 @@ from __future__ import annotations
import logging
import os
import ssl
from collections.abc import Callable
from typing import Any, cast
logger = logging.getLogger("headroom.proxy")
CERT_STORE_ENV = "HEADROOM_CERT_STORE"
EXTRA_CA_ENV = "HEADROOM_CA_BUNDLE"
_CERT_STORE_SOURCES = frozenset({"system", "bundled"})
# Spellings people reach for; a typo'd opt-out must not silently re-enable a source.
_CERT_STORE_ALIASES = {
"bundle": "bundled",
"certifi": "bundled",
"os": "system",
"native": "system",
}
_DEFAULT_CERT_STORE = frozenset({"system", "bundled"})
# Additive bundles, loaded on top of whatever trust store is in force.
_ADDITIVE_CA_VARS = (EXTRA_CA_ENV, "NODE_EXTRA_CA_CERTS")
# Set by Headroom's CLI (and inherited by the proxy processes it spawns) so
# proxy startup only injects process-wide when Headroom owns the process.
PROCESS_TRUST_ENV = "HEADROOM_PROCESS_TRUST"
_process_trust_injected = False
# Built contexts, keyed by ALPN + every env var that shapes them. Building one
# parses certifi's ~140 roots, so it must not run per WebSocket connection.
_system_ctx_cache: dict[tuple[Any, ...], ssl.SSLContext] = {}
_REPLACEMENT_CA_VARS = (
"SSL_CERT_FILE",
"REQUESTS_CA_BUNDLE",
@@ -90,16 +128,22 @@ def _relax_x509_strict_for_custom_ca(ctx: ssl.SSLContext, *, path: str) -> ssl.S
return _clear_x509_strict(ctx, reason=f"custom_ca:{path}")
def _require_tls12(ctx: ssl.SSLContext) -> ssl.SSLContext:
"""Refuse TLS < 1.2 explicitly (Python's default, stated so it cannot regress)."""
ctx.minimum_version = ssl.TLSVersion.TLSv1_2
return ctx
def _replacement_ca_context(path: str) -> ssl.SSLContext:
"""Build a replacement trust-store context from a CA bundle path."""
ctx = ssl.create_default_context(cafile=path)
ctx = _require_tls12(ssl.create_default_context(cafile=path))
ctx.set_alpn_protocols(["h2", "http/1.1"])
return _relax_x509_strict_for_custom_ca(ctx, path=path)
def _additive_ca_context(path: str) -> ssl.SSLContext:
"""Build an additive trust-store context from a CA bundle path."""
ctx = ssl.create_default_context()
ctx = _require_tls12(ssl.create_default_context())
ctx.load_verify_locations(cafile=path)
ctx.set_alpn_protocols(["h2", "http/1.1"])
return _relax_x509_strict_for_custom_ca(ctx, path=path)
@@ -135,22 +179,157 @@ def find_ca_bundle() -> ssl.SSLContext | None:
path,
)
node_path = os.environ.get("NODE_EXTRA_CA_CERTS")
if node_path and os.path.isfile(node_path):
logger.info(
"event=ssl_ca_bundle_loaded env_var=NODE_EXTRA_CA_CERTS path=%s additive=true",
node_path,
)
return _additive_ca_context(node_path)
if node_path and not os.path.isfile(node_path):
logger.warning(
"event=ssl_ca_bundle_missing env_var=NODE_EXTRA_CA_CERTS path=%r (skipped)",
node_path,
)
additive = _additive_ca_paths()
if additive:
ctx = _additive_ca_context(additive[0])
for path in additive[1:]:
ctx.load_verify_locations(cafile=path)
return ctx
return None
def _additive_ca_paths(*, log: bool = True) -> list[str]:
"""Existing files named by the additive CA vars (logged unless ``log=False``)."""
paths: list[str] = []
for var in _ADDITIVE_CA_VARS:
path = os.environ.get(var)
if not path:
continue
if not log:
if os.path.isfile(path):
paths.append(path)
continue
if os.path.isfile(path):
logger.info(
"event=ssl_ca_bundle_loaded env_var=%s path=%s additive=true",
var,
path,
)
paths.append(path)
else:
logger.warning(
"event=ssl_ca_bundle_missing env_var=%s path=%r (skipped)",
var,
path,
)
return paths
def _replacement_ca_var_set() -> bool:
"""True when ``SSL_CERT_FILE``/``REQUESTS_CA_BUNDLE`` name an existing file."""
return any(
(path := os.environ.get(var)) and os.path.isfile(path) for var in _REPLACEMENT_CA_VARS
)
def cert_store_sources() -> frozenset[str]:
"""Trust sources selected by ``HEADROOM_CERT_STORE`` (default: system,bundled).
Unknown tokens are ignored with a warning; a value with no known token falls
back to the default rather than trusting nothing.
"""
raw = os.environ.get(CERT_STORE_ENV)
if raw is None or not raw.strip():
return _DEFAULT_CERT_STORE
tokens = {
_CERT_STORE_ALIASES.get(t.strip().lower(), t.strip().lower())
for t in raw.split(",")
if t.strip()
}
unknown = tokens - _CERT_STORE_SOURCES
if unknown:
logger.warning(
"event=ssl_cert_store_unknown env_var=%s ignored=%s",
CERT_STORE_ENV,
",".join(sorted(unknown)),
)
chosen = frozenset(tokens & _CERT_STORE_SOURCES)
return chosen or _DEFAULT_CERT_STORE
def system_trust_available() -> bool:
"""True when the ``truststore`` package can verify through the OS store."""
try:
import truststore # noqa: F401
except Exception:
return False
return True
def _system_store_enabled() -> bool:
return (
"system" in cert_store_sources()
and not _replacement_ca_var_set()
and system_trust_available()
)
def _build_system_context(alpn: list[str]) -> ssl.SSLContext:
"""A context that verifies through the OS trust store. Raises if it cannot.
``truststore`` hands chain validation to macOS Security.framework / Windows
CryptoAPI (OpenSSL + the system bundle on Linux). Certificates loaded with
``load_verify_locations`` are trusted *in addition* to the OS roots, so
certifi (``bundled``) and the additive bundle vars stack on top. Built once
per configuration and cached.
"""
key = (
tuple(alpn),
os.environ.get(CERT_STORE_ENV),
*(os.environ.get(var) for var in _ADDITIVE_CA_VARS),
)
cached = _system_ctx_cache.get(key)
if cached is not None:
return cached
import truststore
ctx: ssl.SSLContext = _require_tls12(truststore.SSLContext(ssl.PROTOCOL_TLS_CLIENT))
# Match create_default_context() on 3.13+: an explicitly trusted
# intermediate (a common "Zscaler Intermediate Root CA" export) is a
# valid anchor. Only affects the OpenSSL (Linux) backend.
partial_chain = getattr(ssl, "VERIFY_X509_PARTIAL_CHAIN", 0)
if partial_chain:
ctx.verify_flags |= partial_chain
if "bundled" in cert_store_sources():
try:
import certifi
ctx.load_verify_locations(cafile=certifi.where())
except Exception: # certifi is optional; the OS store still applies
logger.debug("event=ssl_certifi_unavailable")
for path in _additive_ca_paths():
ctx.load_verify_locations(cafile=path)
ctx.set_alpn_protocols(alpn)
_system_ctx_cache[key] = ctx
return ctx
def describe_trust_policy() -> dict[str, Any]:
"""What trust configuration is in force, for ``/health`` and ``doctor``."""
replacement = next(
(
{"env_var": var, "path": os.environ[var]}
for var in _REPLACEMENT_CA_VARS
if os.environ.get(var) and os.path.isfile(os.environ[var])
),
None,
)
return {
"cert_store": sorted(cert_store_sources()),
"system_store_available": system_trust_available(),
"system_store_active": _system_store_enabled(),
"replacement_bundle": replacement,
"additive_bundles": [
{"env_var": var, "path": os.environ[var]}
for var in _ADDITIVE_CA_VARS
if os.environ.get(var) and os.path.isfile(os.environ[var])
],
"tls_strict_disabled": tls_strict_disabled(),
"process_injected": _process_trust_injected,
}
def _default_strict_relaxed_context() -> ssl.SSLContext:
"""Default trust store, but with ``VERIFY_X509_STRICT`` cleared.
@@ -159,51 +338,115 @@ def _default_strict_relaxed_context() -> ssl.SSLContext:
tolerate a non-critical ``basicConstraints`` CA. Mirrors what httpx builds
for ``verify=True`` (default context + ALPN), minus the strict flag.
"""
ctx = ssl.create_default_context()
ctx = _require_tls12(ssl.create_default_context())
ctx.set_alpn_protocols(["h2", "http/1.1"])
return _clear_x509_strict(ctx, reason="env_toggle")
def build_httpx_verify() -> ssl.SSLContext | bool:
"""Return the value for httpx's ``verify=`` parameter.
def _replacement_ca_path() -> str | None:
"""The existing file named by ``SSL_CERT_FILE``/``REQUESTS_CA_BUNDLE``, if any."""
for var in _REPLACEMENT_CA_VARS:
path = os.environ.get(var)
if path and os.path.isfile(path):
return path
return None
Resolution order:
1. A custom CA bundle env var (``SSL_CERT_FILE`` / ``REQUESTS_CA_BUNDLE`` /
``NODE_EXTRA_CA_CERTS``) → a context trusting that bundle, with strict
mode already relaxed (corporate PKI signal).
2. No bundle, but ``HEADROOM_TLS_STRICT=0`` → the default trust store with
``VERIFY_X509_STRICT`` cleared, so a corporate root that's installed in
the OS store but trips RFC 5280 strict mode still validates.
3. Otherwise → ``True`` (httpx's default strict verification).
def _has_configured_trust() -> bool:
"""True when Headroom's trust settings call for something other than a library default."""
return (
_system_store_enabled()
or _replacement_ca_path() is not None
or bool(_additive_ca_paths(log=False))
or tls_strict_disabled()
)
Returning ``True`` rather than a hand-built context in the common case
keeps httpx's own default behavior (including its certifi fallback) intact.
def _build_context(alpn: list[str], fallback: Callable[[], ssl.SSLContext]) -> ssl.SSLContext:
"""Build the verifying context Headroom's trust configuration calls for.
Every branch returns a concrete, certificate-verifying ``SSLContext``;
``fallback`` supplies the library-default equivalent. Resolution order:
0. The OS trust store (plus certifi and the additive bundle vars) when
``HEADROOM_CERT_STORE`` includes ``system`` — the default — and no
replacement bundle var is set.
1. ``SSL_CERT_FILE`` / ``REQUESTS_CA_BUNDLE`` → only that bundle, strict
mode relaxed (corporate PKI signal).
2. ``HEADROOM_CA_BUNDLE`` / ``NODE_EXTRA_CA_CERTS`` → default roots plus
those, strict mode relaxed.
3. ``HEADROOM_TLS_STRICT=0`` → the default trust store with
``VERIFY_X509_STRICT`` cleared.
4. Otherwise ``fallback()``.
"""
ca_ctx = find_ca_bundle()
if ca_ctx is not None:
return ca_ctx
if tls_strict_disabled():
return _default_strict_relaxed_context()
return True
if _system_store_enabled():
try:
return _build_system_context(alpn)
except Exception as exc:
# Never let the OS-store path take the proxy down: fall back to the
# bundled behavior and say so.
logger.warning("event=ssl_system_store_failed error=%r (falling back to bundled)", exc)
replacement = _replacement_ca_path()
additive = _additive_ca_paths()
if replacement is not None:
logger.info("event=ssl_ca_bundle_loaded path=%s", replacement)
ctx = _replacement_ca_context(replacement)
elif additive:
ctx = _additive_ca_context(additive[0])
for path in additive[1:]:
ctx.load_verify_locations(cafile=path)
elif tls_strict_disabled():
ctx = _default_strict_relaxed_context()
else:
ctx = fallback()
ctx.set_alpn_protocols(alpn)
return ctx
def _bundled_default_context() -> ssl.SSLContext:
"""Exactly what httpx builds for ``verify=True``: certifi, strict defaults.
Built here rather than passing ``True`` so every upstream client receives a
concrete, always-verifying ``SSLContext``: no code path can hand httpx a
value that disables verification.
"""
cert_dir = os.environ.get("SSL_CERT_DIR")
if cert_dir:
ctx = ssl.create_default_context(capath=cert_dir)
else:
import certifi
ctx = ssl.create_default_context(cafile=certifi.where())
return _require_tls12(ctx)
def _stdlib_default_context() -> ssl.SSLContext:
"""What ``ssl=True`` / urlopen build: the stdlib default trust store."""
return _require_tls12(ssl.create_default_context())
def build_httpx_verify() -> ssl.SSLContext:
"""Return the value for httpx's ``verify=`` parameter: always a verifying context.
Headroom's configured trust (see :func:`_build_context`) when there is one,
else httpx's own default (certifi) built explicitly.
"""
return _build_context(["h2", "http/1.1"], _bundled_default_context)
def build_urlopen_context() -> ssl.SSLContext | None:
"""Return Headroom's configured TLS context for ``urllib.request.urlopen``.
``urlopen`` already handles Python's default trust configuration when no
explicit context is passed. Return only a custom context here so callers
retain that default while sharing Headroom's corporate CA and strict-mode
handling when it is configured.
``None`` means "use urlopen's own default" (Python's default trust store),
which is what the legacy ``HEADROOM_CERT_STORE=bundled`` path wants when no
bundle or strict-mode toggle is configured. urllib.request/http.client only
implements HTTP/1.1 framing, so the context only offers ``http/1.1``:
offering h2 can make a TLS-inspecting MITM negotiate a protocol it cannot
parse.
"""
verify = build_httpx_verify()
if not isinstance(verify, ssl.SSLContext):
if not _has_configured_trust():
return None
# urllib.request/http.client only implements HTTP/1.1 framing. Offering h2
# can make a TLS-inspecting MITM negotiate a protocol it cannot parse.
verify.set_alpn_protocols(["http/1.1"])
return verify
return _build_context(["http/1.1"], _stdlib_default_context)
def apply_global_tls_relaxation() -> bool:
@@ -257,3 +500,62 @@ def apply_global_tls_relaxation() -> bool:
_u3ssl.create_urllib3_context = _relaxed_create_urllib3_context # type: ignore[assignment]
logger.info("event=ssl_x509_strict_disabled reason=urllib3_global_patch")
return True
def build_websocket_ssl() -> ssl.SSLContext:
"""Return the value for ``websockets.connect(ssl=...)`` on a ``wss://`` URL.
WebSocket upgrades are HTTP/1.1, so the context only offers ``http/1.1``.
Without configured trust it is what ``ssl=True`` would build (the stdlib
default, which on Windows loads the machine store).
"""
return _build_context(["http/1.1"], _stdlib_default_context)
def ensure_process_trust() -> bool:
"""Make every TLS client in this *application* process use the OS store.
Marks the process (``HEADROOM_PROCESS_TRUST=1``) so proxy processes the CLI
spawns inherit the decision; see :func:`ensure_process_trust_if_owned`.
Headroom's own upstream clients get an explicit context from the builders
above, but third-party code (``huggingface_hub`` model downloads, tiktoken
vocab fetches, OTEL exporters, ...) builds its own. ``truststore``'s
process-wide injection covers those in one place instead of chasing every
call site. It is only ever called from Headroom's entry points (CLI and
proxy startup) — never on library import — because injecting into an
embedding application's process is not ours to decide.
Skipped when the operator pinned a replacement bundle (``SSL_CERT_FILE`` /
``REQUESTS_CA_BUNDLE``) or opted out via ``HEADROOM_CERT_STORE``.
Idempotent. Returns True when injection is in place.
"""
global _process_trust_injected
os.environ[PROCESS_TRUST_ENV] = "1"
if _process_trust_injected:
return True
if not _system_store_enabled():
return False
try:
import truststore
truststore.inject_into_ssl()
except Exception as exc:
logger.warning("event=ssl_process_trust_failed error=%r", exc)
return False
_process_trust_injected = True
logger.info("event=ssl_process_trust_injected source=os_trust_store")
return True
def ensure_process_trust_if_owned() -> bool:
""":func:`ensure_process_trust`, but only in a process Headroom's CLI owns.
Proxy startup calls this: uvicorn workers spawned by ``headroom proxy``
inherit ``HEADROOM_PROCESS_TRUST=1`` and get injection, while an application
that embeds the proxy app keeps its own process-wide TLS behavior (its
Headroom upstream clients still get the OS store via explicit contexts).
"""
if os.environ.get(PROCESS_TRUST_ENV) != "1":
return False
return ensure_process_trust()
+529
View File
@@ -0,0 +1,529 @@
"""Explain upstream TLS failures in words a user (and their IT team) can act on.
Behind a corporate TLS-inspection gateway every certificate Headroom sees is
re-signed by the company's root. When that root is not trusted, the only thing
that reaches the user today is ``CERTIFICATE_VERIFY_FAILED`` wrapped in a 502.
This module turns that into: *who* signed the certificate (e.g. "Zscaler
Intermediate Root CA"), what that means, and the exact fix.
It is used in two places:
* the proxy's upstream error paths, so the agent's error line names the cause
(:func:`describe_upstream_failure`), and
* ``headroom doctor --network`` (:func:`probe_endpoint`).
To learn the issuer, the probe performs a second, *unverified* TLS handshake
purely to read the presented certificate. Nothing is sent over that
connection: it is closed straight after the handshake.
"""
from __future__ import annotations
import base64
import logging
import os
import socket
import ssl
import tempfile
import threading
import time
import urllib.request
from collections.abc import Mapping
from dataclasses import asdict, dataclass, field
from typing import Any
from urllib.parse import urlsplit
logger = logging.getLogger("headroom.proxy")
DOCS_URL = "https://docs.headroomlabs.ai/docs/corporate-networks"
# Issuer substrings (lower-cased) of common TLS-inspection products. Matching is
# on the issuer's organization / common name, so a customer-uploaded
# intermediate ("Acme Corp Zscaler SubCA") still resolves to the product.
_INSPECTION_VENDORS: tuple[tuple[str, str], ...] = (
("zscaler", "Zscaler"),
("netskope", "Netskope"),
("palo alto", "Palo Alto Networks"),
("globalprotect", "Palo Alto Networks"),
("prisma", "Palo Alto Networks"),
("cisco umbrella", "Cisco Umbrella"),
("cisco secure access", "Cisco Secure Access"),
("fortinet", "Fortinet"),
("fortigate", "Fortinet"),
("forcepoint", "Forcepoint"),
("websense", "Forcepoint"),
("blue coat", "Symantec / Broadcom"),
("bluecoat", "Symantec / Broadcom"),
("symantec", "Symantec / Broadcom"),
("cloudflare gateway", "Cloudflare Gateway"),
("cloudflare for teams", "Cloudflare Gateway"),
("iboss", "iboss"),
("check point", "Check Point"),
("checkpoint", "Check Point"),
("sophos", "Sophos"),
("skyhigh", "Skyhigh Security"),
("mcafee web gateway", "Skyhigh Security"),
("menlo security", "Menlo Security"),
("barracuda", "Barracuda"),
)
# ``_ssl.Certificate.public_bytes`` format flag (Python 3.13+ chain API).
_ENCODING_DER = getattr(ssl._ssl, "ENCODING_DER", 1) # type: ignore[attr-defined]
_PROBE_TIMEOUT_S = 4.0
_PROBE_CACHE_TTL_S = 300.0
_probe_cache: dict[tuple[str, int, bool], tuple[float, ChainInfo]] = {}
_probe_lock = threading.Lock()
# One in-flight probe per (host, port): a burst of failing requests waits for
# the first handshake instead of each opening its own.
_probe_inflight: dict[tuple[str, int, bool], threading.Lock] = {}
_logged_hosts: set[str] = set()
@dataclass
class ChainInfo:
"""What the network presented for ``host`` — no trust decision implied."""
host: str
port: int
reachable: bool
issuer: str | None = None
subject: str | None = None
inspection_vendor: str | None = None
via_proxy: str | None = None
error: str | None = None
chain_issuers: list[str] = field(default_factory=list)
def to_dict(self) -> dict[str, Any]:
return asdict(self)
# ---------------------------------------------------------------------------
# Exception inspection
# ---------------------------------------------------------------------------
def find_cert_verify_error(exc: BaseException | None) -> ssl.SSLCertVerificationError | None:
"""Return the certificate-verification error buried in ``exc``'s chain.
httpx wraps httpcore, which wraps the ``ssl`` error, and websockets
re-raises it directly; walk ``__cause__``/``__context__`` (bounded) and the
exception args to find it.
"""
seen: set[int] = set()
stack: list[BaseException] = [exc] if exc is not None else []
while stack:
cur = stack.pop()
if id(cur) in seen or len(seen) > 32:
continue
seen.add(id(cur))
if isinstance(cur, ssl.SSLCertVerificationError):
return cur
for nxt in (cur.__cause__, cur.__context__):
if isinstance(nxt, BaseException):
stack.append(nxt)
for arg in getattr(cur, "args", ()):
if isinstance(arg, BaseException):
stack.append(arg)
return None
def _looks_like_cert_failure(exc: BaseException) -> bool:
# Some stacks (httpcore on certain paths) stringify the ssl error into a
# plain ConnectError without chaining it.
text = str(exc)
return "CERTIFICATE_VERIFY_FAILED" in text or "certificate verify failed" in text
# ---------------------------------------------------------------------------
# Presented-chain probe
# ---------------------------------------------------------------------------
def identify_inspection_vendor(*names: str | None) -> str | None:
"""Map certificate issuer / subject strings to a TLS-inspection product."""
for name in names:
if not name:
continue
lowered = name.lower()
for needle, vendor in _INSPECTION_VENDORS:
if needle in lowered:
return vendor
return None
def _name_to_str(name: Any) -> str | None:
"""Render a decoded ``getpeercert()``-style name tuple as ``O=..., CN=...``."""
if not name:
return None
parts: list[str] = []
wanted = {"organizationName": "O", "organizationalUnitName": "OU", "commonName": "CN"}
for rdn in name:
for key, value in rdn:
if key in wanted:
parts.append(f"{wanted[key]}={value}")
return ", ".join(parts) or None
def _decode_der(der: bytes) -> dict[str, Any]:
"""Decode a DER certificate into ``getpeercert()`` form without verifying it.
Uses ``cryptography`` when installed, else CPython's own decoder (the one
``ssl`` uses internally), which needs a PEM file on disk.
"""
try:
from cryptography import x509
from cryptography.x509.oid import NameOID
cert = x509.load_der_x509_certificate(der)
def _rdns(n: Any) -> tuple[tuple[tuple[str, str], ...], ...]:
mapping = {
NameOID.ORGANIZATION_NAME: "organizationName",
NameOID.ORGANIZATIONAL_UNIT_NAME: "organizationalUnitName",
NameOID.COMMON_NAME: "commonName",
}
return tuple(((mapping[a.oid], str(a.value)),) for a in n if a.oid in mapping)
return {"issuer": _rdns(cert.issuer), "subject": _rdns(cert.subject)}
except Exception:
pass
pem = ssl.DER_cert_to_PEM_cert(der)
fd, path = tempfile.mkstemp(suffix=".pem")
try:
with os.fdopen(fd, "w") as fh:
fh.write(pem)
decoded: dict[str, Any] = ssl._ssl._test_decode_cert(path) # type: ignore[attr-defined]
return decoded
except Exception:
return {}
finally:
try:
os.unlink(path)
except OSError:
pass
def _proxy_for(host: str) -> str | None:
"""The HTTPS proxy the upstream client would use for ``host``, if any.
Environment only, like httpx's ``trust_env``: the OS proxy settings that
``urllib.request.getproxies`` also reads on macOS/Windows are not what the
proxy's requests use, so probing through them would diagnose another path.
"""
try:
if urllib.request.proxy_bypass_environment(host): # type: ignore[attr-defined]
return None
env = urllib.request.getproxies_environment()
except Exception:
return None
return env.get("https") or env.get("all") or None
def _open_tunnel(host: str, port: int, proxy_url: str, timeout: float) -> socket.socket:
"""Open a TCP connection to ``host:port`` through an HTTP CONNECT proxy."""
parts = urlsplit(proxy_url if "://" in proxy_url else f"http://{proxy_url}")
if parts.scheme not in ("http", ""):
raise OSError(f"unsupported proxy scheme {parts.scheme!r} for the TLS probe")
sock = socket.create_connection((parts.hostname or "", parts.port or 8080), timeout=timeout)
request = f"CONNECT {host}:{port} HTTP/1.1\r\nHost: {host}:{port}\r\n"
if parts.username:
token = base64.b64encode(f"{parts.username}:{parts.password or ''}".encode()).decode()
request += f"Proxy-Authorization: Basic {token}\r\n"
sock.sendall((request + "\r\n").encode())
reply = b""
while b"\r\n\r\n" not in reply and len(reply) < 8192:
chunk = sock.recv(1024)
if not chunk:
break
reply += chunk
status_line = reply.split(b"\r\n", 1)[0].decode("latin-1", "replace")
if " 200" not in status_line:
sock.close()
raise OSError(f"proxy refused CONNECT: {status_line or 'no reply'}")
return sock
def _public_address(host: str, port: int, timeout: float) -> tuple[str, int] | None:
"""Resolve ``host`` once and return a public address, or None if it is not public.
The request-path probe can be pointed at a per-request upstream, so it must
not become a way to read the certificate of an internal service. Returning
the checked address (instead of re-resolving) keeps DNS rebinding out too.
"""
import ipaddress
# No timeout knob exists for getaddrinfo; the caller already runs off the
# event loop, and the process-wide socket default must not be touched here.
del timeout
infos = socket.getaddrinfo(host, port, type=socket.SOCK_STREAM)
for _family, _type, _proto, _canon, sockaddr in infos:
ip = ipaddress.ip_address(sockaddr[0])
if not ip.is_global:
return None
return (str(infos[0][4][0]), port) if infos else None
def probe_presented_chain(
host: str,
port: int = 443,
*,
timeout: float = _PROBE_TIMEOUT_S,
use_cache: bool = True,
allow_private: bool = False,
) -> ChainInfo:
"""Handshake with ``host`` *without verification* and report who signed it.
Goes through the same explicit HTTPS proxy the environment configures, so an
explicit-proxy deployment sees the same re-signed chain the proxy would.
Results are cached for a few minutes so a burst of failing requests costs a
single probe. Hosts resolving to private, loopback or link-local addresses
are skipped unless ``allow_private`` (``doctor``, which the user runs on
purpose, sets it).
"""
# allow_private is part of the key so a doctor/test probe of a private host
# can never be served to the request path.
key = (host, port, allow_private)
if not use_cache:
return _probe_uncached(host, port, timeout=timeout, allow_private=allow_private)
with _probe_lock:
gate = _probe_inflight.setdefault(key, threading.Lock())
with gate:
with _probe_lock:
cached = _probe_cache.get(key)
if cached and time.monotonic() - cached[0] < _PROBE_CACHE_TTL_S:
return cached[1]
info = _probe_uncached(host, port, timeout=timeout, allow_private=allow_private)
with _probe_lock:
_probe_cache[key] = (time.monotonic(), info)
return info
def _probe_uncached(host: str, port: int, *, timeout: float, allow_private: bool) -> ChainInfo:
proxy = _proxy_for(host)
info = ChainInfo(host=host, port=port, reachable=False, via_proxy=proxy)
# Certificate-inspection handshake only: it reads the presented chain and
# closes without sending a byte, so it deliberately skips verification (the
# chain is what we are diagnosing). It still refuses TLS < 1.2, and it only
# connects through the public-address / env-proxy CONNECT guard below.
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
ctx.minimum_version = ssl.TLSVersion.TLSv1_2
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
try:
if proxy:
raw = _open_tunnel(host, port, proxy, timeout)
else:
target = (host, port) if allow_private else _public_address(host, port, timeout)
if target is None:
info.error = "skipped: host resolves to a non-public address"
return info
raw = socket.create_connection(target, timeout=timeout)
with ctx.wrap_socket(raw, server_hostname=host) as tls:
info.reachable = True
ders: list[bytes] = []
get_chain = getattr(tls, "get_unverified_chain", None) # Python 3.13+
if get_chain is not None:
try:
ders = [
c if isinstance(c, bytes) else c.public_bytes(_ENCODING_DER)
for c in get_chain() or []
]
except Exception:
ders = []
if not ders:
leaf = tls.getpeercert(binary_form=True)
ders = [leaf] if leaf else []
decoded = [_decode_der(d) for d in ders]
if decoded:
info.subject = _name_to_str(decoded[0].get("subject"))
info.issuer = _name_to_str(decoded[0].get("issuer"))
info.chain_issuers = [s for d in decoded if (s := _name_to_str(d.get("issuer")))]
info.inspection_vendor = identify_inspection_vendor(info.issuer, *info.chain_issuers)
except Exception as exc:
info.error = f"{type(exc).__name__}: {exc}"
return info
# ---------------------------------------------------------------------------
# User-facing explanation
# ---------------------------------------------------------------------------
def _fix_lines(vendor: str | None) -> list[str]:
who = f"{vendor} " if vendor else "your network's "
return [
f"Headroom trusts the operating system's certificate store, where IT normally "
f"installs the {who}root certificate. If your browser and Claude Code work but "
f"Headroom does not, that root is not in this machine's store.",
"Fix (pick one): ask IT to install the root in the OS certificate store; or export "
"it to a PEM file and set HEADROOM_CA_BUNDLE=/path/to/root.pem (NODE_EXTRA_CA_CERTS "
"also works); or ask IT to exempt the provider domains from TLS inspection.",
"Run `headroom doctor --network` for a full report.",
]
def explain_cert_failure(host: str, verify_message: str | None, info: ChainInfo | None) -> str:
"""One paragraph a user can act on (and forward to IT)."""
reason = verify_message or "certificate verify failed"
head = f"Headroom could not verify the TLS certificate for {host} ({reason})."
if info is not None and info.issuer:
if info.inspection_vendor:
head += (
f" The certificate was issued by '{info.issuer}': {info.inspection_vendor} "
f"is inspecting this connection."
)
else:
head += f" The certificate was issued by '{info.issuer}'."
return " ".join([head, *_fix_lines(info.inspection_vendor if info else None), DOCS_URL])
def describe_upstream_failure(
exc: BaseException, url: str | None, *, probe: bool = True, allow_private: bool = False
) -> str | None:
"""Actionable message for a certificate-verification failure, else ``None``.
``None`` means "not a TLS trust problem" and the caller keeps its existing
message. Safe to call from a request's error path: the probe is bounded by a
short timeout and cached per host. Blocking — async callers should use
:func:`describe_upstream_failure_async`.
"""
cert_err = find_cert_verify_error(exc)
if cert_err is None and not _looks_like_cert_failure(exc):
return None
parts = urlsplit(url or "")
host = parts.hostname or "the upstream API"
port = parts.port or (443 if parts.scheme in ("https", "wss", "") else 80)
info = (
probe_presented_chain(host, port, allow_private=allow_private)
if probe and parts.hostname
else None
)
verify_message = getattr(cert_err, "verify_message", None) if cert_err else None
message = explain_cert_failure(host, verify_message, info)
if host not in _logged_hosts:
_logged_hosts.add(host)
logger.error("event=upstream_tls_untrusted host=%s detail=%s", host, message)
return message
async def describe_upstream_failure_async(
exc: BaseException, url: str | None, *, probe: bool = True
) -> str | None:
""":func:`describe_upstream_failure` off the event loop."""
import asyncio
if find_cert_verify_error(exc) is None and not _looks_like_cert_failure(exc):
return None
return await asyncio.to_thread(describe_upstream_failure, exc, url, probe=probe)
# ---------------------------------------------------------------------------
# Endpoint check for `headroom doctor --network`
# ---------------------------------------------------------------------------
@dataclass
class EndpointReport:
"""Result of checking one endpoint Headroom needs."""
name: str
url: str
ok: bool
chain: ChainInfo
verified_with: str | None = None
error: str | None = None
status: int | None = None
block_page: bool = False
elapsed_ms: float | None = None
def to_dict(self) -> dict[str, Any]:
d = asdict(self)
d["chain"] = self.chain.to_dict()
return d
def _looks_like_block_page(status: int, content_type: str, body: bytes) -> bool:
"""A TLS-inspection gateway's interstitial rather than the real API."""
if "text/html" not in content_type.lower():
return False
text = body[:8192].decode("utf-8", "replace").lower()
if identify_inspection_vendor(text) is not None:
return True
return status in (403, 407, 451) and ("access denied" in text or "blocked" in text)
def probe_endpoint(name: str, url: str, *, timeout: float = 8.0) -> EndpointReport:
"""Check ``url`` end to end with Headroom's real trust policy.
Reports the presented chain (and any inspection vendor), whether the chain
verifies under the policy Headroom will use, and whether the response is a
gateway block page instead of the service.
"""
import httpx
from headroom.proxy.ssl_context import build_httpx_verify
parts = urlsplit(url)
chain = probe_presented_chain(
parts.hostname or "", parts.port or 443, use_cache=False, allow_private=True
)
started = time.monotonic()
try:
with httpx.Client(verify=build_httpx_verify(), timeout=timeout) as client:
resp = client.get(url, headers={"User-Agent": "headroom-doctor"})
elapsed = (time.monotonic() - started) * 1000
block = _looks_like_block_page(
resp.status_code, resp.headers.get("content-type", ""), resp.content
)
return EndpointReport(
name=name,
url=url,
ok=not block,
chain=chain,
verified_with="headroom trust policy",
status=resp.status_code,
block_page=block,
elapsed_ms=round(elapsed, 1),
error="gateway returned a block/warning page instead of the service" if block else None,
)
except Exception as exc:
cert_err = find_cert_verify_error(exc)
if cert_err is not None or _looks_like_cert_failure(exc):
error = explain_cert_failure(
parts.hostname or url, getattr(cert_err, "verify_message", None), chain
)
else:
error = f"{type(exc).__name__}: {exc}"
return EndpointReport(name=name, url=url, ok=False, chain=chain, error=error)
def loopback_no_proxy_gap(environ: Mapping[str, str]) -> str | None:
"""Name the proxy var that would capture loopback traffic, if NO_PROXY misses it.
With ``HTTP(S)_PROXY`` set and no loopback entry in ``NO_PROXY``, an agent's
requests to ``http://127.0.0.1:8787`` can be sent to the corporate proxy,
which cannot reach this laptop.
"""
proxy_var = next(
(
v
for v in (
"HTTP_PROXY",
"http_proxy",
"HTTPS_PROXY",
"https_proxy",
"ALL_PROXY",
"all_proxy",
)
if environ.get(v)
),
None,
)
if proxy_var is None:
return None
no_proxy = ",".join(environ.get(v, "") for v in ("NO_PROXY", "no_proxy")).lower()
entries = {e.strip() for e in no_proxy.split(",") if e.strip()}
if "*" in entries or {"127.0.0.1", "localhost"} <= entries:
return None
return proxy_var
+4
View File
@@ -65,6 +65,10 @@ dependencies = [
"pyyaml>=6.0", # omp wrap: parse/merge omp's models.yml registry
"tomli>=2.0.0; python_version < '3.11'", # tomllib backport for helper scripts
"tomlkit>=0.13.0,<1.0", # Loss-minimizing Codex config recovery
# Verify upstream TLS through the OS trust store (macOS Keychain, Windows
# cert store), where IT installs corporate TLS-inspection roots (Zscaler,
# Netskope). Pure Python, no deps; the same library pip uses by default.
"truststore>=0.10.0",
]
[project.optional-dependencies]
+23
View File
@@ -19,6 +19,29 @@ import pytest
from tests._skip_helpers import external_model_skip_reason
@pytest.fixture(autouse=True)
def _undo_process_trust_injection():
"""Undo ``truststore.inject_into_ssl()`` after any test that triggered it.
Invoking the CLI (``CliRunner`` on ``main``) or starting the proxy calls
``ensure_process_trust()``, which swaps ``ssl.SSLContext`` process-wide.
Left in place it leaks into later tests (server-side test contexts, OpenSSL
store stats), so restore the stdlib class after each test.
"""
yield
try:
from headroom.proxy import ssl_context
except Exception:
return
os.environ.pop(ssl_context.PROCESS_TRUST_ENV, None)
ssl_context._system_ctx_cache.clear()
if ssl_context._process_trust_injected:
import truststore
truststore.extract_from_ssl()
ssl_context._process_trust_injected = False
# A live `headroom` dev session exports HEADROOM_* into the shell (and the
# Claude wrap adds ANTHROPIC_CUSTOM_HEADERS). Click `envvar=` options pick
# those up inside CliRunner, so assertions would see the developer's proxy
+3 -3
View File
@@ -19,7 +19,7 @@ def test_device_authorization_uses_form_encoded_request(monkeypatch: pytest.Monk
captured = {}
def fake_urlopen(request, timeout): # noqa: ANN001, ANN202
def fake_urlopen(request, timeout, context=None): # noqa: ANN001, ANN202
captured["request"] = request
return io.BytesIO(b'{"device_code":"d","user_code":"u"}')
@@ -37,7 +37,7 @@ def test_device_poll_uses_form_encoded_request(monkeypatch: pytest.MonkeyPatch)
captured = {}
def fake_urlopen(request, timeout): # noqa: ANN001, ANN202
def fake_urlopen(request, timeout, context=None): # noqa: ANN001, ANN202
captured["request"] = request
return io.BytesIO(b'{"access_token":"gho-test"}')
@@ -1601,7 +1601,7 @@ def test_exchange_token_sync_raises_for_http_error(monkeypatch: pytest.MonkeyPat
def close(self) -> None:
return None
def fake_urlopen(request, timeout: float): # noqa: ANN001, ANN202
def fake_urlopen(request, timeout: float, context=None): # noqa: ANN001, ANN202
raise urllib_error.HTTPError(
url=request.full_url,
code=404,
+513
View File
@@ -0,0 +1,513 @@
"""Headroom behind a corporate TLS-inspection gateway ("fake Zscaler").
A local HTTPS server presents a certificate signed by a root that looks like
Zscaler's: ``O=Zscaler Inc.``, and — like the real 2014 Zscaler root —
``basicConstraints`` present but **not critical**, which Python 3.13's
``VERIFY_X509_STRICT`` rejects. These tests pin the behavior customers see:
* certifi alone rejects the chain (the original bug);
* the default trust policy plus the corporate root (``HEADROOM_CA_BUNDLE`` or
``NODE_EXTRA_CA_CERTS``) accepts it, with no strict-mode opt-out;
* the failure explanation names Zscaler and the fix;
* process-wide injection respects the operator's replacement bundle;
* ``wrap``'s loopback ``NO_PROXY`` merge and the new ``doctor`` rows.
The OS-store half of the default policy cannot be exercised without admin
rights to install a root, so it is covered by the ``truststore`` context type
and by the additive-bundle path, which goes through the same verifier.
"""
from __future__ import annotations
import datetime as _dt
import http.server
import json
import re
import ssl
import threading
from collections.abc import Iterator
from pathlib import Path
import httpx
import pytest
from headroom.proxy import ssl_context, tls_diagnostics
from headroom.proxy.ssl_context import (
build_httpx_verify,
build_urlopen_context,
build_websocket_ssl,
cert_store_sources,
describe_trust_policy,
ensure_process_trust,
)
x509 = pytest.importorskip("cryptography.x509")
from cryptography.hazmat.primitives import hashes, serialization # noqa: E402
from cryptography.hazmat.primitives.asymmetric import ec # noqa: E402
from cryptography.x509.oid import NameOID # noqa: E402
_ENV_VARS = (
"SSL_CERT_FILE",
"REQUESTS_CA_BUNDLE",
"NODE_EXTRA_CA_CERTS",
"HEADROOM_CA_BUNDLE",
"HEADROOM_CERT_STORE",
"HEADROOM_TLS_STRICT",
"HTTP_PROXY",
"http_proxy",
"HTTPS_PROXY",
"https_proxy",
"ALL_PROXY",
"all_proxy",
"NO_PROXY",
"no_proxy",
)
@pytest.fixture(autouse=True)
def _clean_env(monkeypatch: pytest.MonkeyPatch) -> None:
for var in _ENV_VARS:
monkeypatch.delenv(var, raising=False)
tls_diagnostics._probe_cache.clear()
tls_diagnostics._logged_hosts.clear()
def _name(org: str, cn: str) -> x509.Name:
return x509.Name(
[
x509.NameAttribute(NameOID.ORGANIZATION_NAME, org),
x509.NameAttribute(NameOID.COMMON_NAME, cn),
]
)
@pytest.fixture(scope="module")
def fake_zscaler(tmp_path_factory: pytest.TempPathFactory) -> Iterator[dict[str, str | int]]:
"""HTTPS server on 127.0.0.1 behind a Zscaler-style re-signing root."""
d: Path = tmp_path_factory.mktemp("fake-zscaler")
now = _dt.datetime.now(_dt.timezone.utc)
root_key = ec.generate_private_key(ec.SECP256R1())
root_name = _name("Zscaler Inc.", "Zscaler Root CA")
root = (
x509.CertificateBuilder()
.subject_name(root_name)
.issuer_name(root_name)
.public_key(root_key.public_key())
.serial_number(x509.random_serial_number())
.not_valid_before(now - _dt.timedelta(days=1))
.not_valid_after(now + _dt.timedelta(days=30))
# Not critical, exactly like the real Zscaler root: the RFC 5280
# strict check in Python 3.13 rejects this.
.add_extension(x509.BasicConstraints(ca=True, path_length=None), critical=False)
.sign(root_key, hashes.SHA256())
)
leaf_key = ec.generate_private_key(ec.SECP256R1())
import ipaddress
leaf = (
x509.CertificateBuilder()
.subject_name(_name("Zscaler Inc.", "127.0.0.1"))
.issuer_name(root_name)
.public_key(leaf_key.public_key())
.serial_number(x509.random_serial_number())
.not_valid_before(now - _dt.timedelta(days=1))
.not_valid_after(now + _dt.timedelta(days=30))
.add_extension(
x509.SubjectAlternativeName(
[x509.DNSName("localhost"), x509.IPAddress(ipaddress.ip_address("127.0.0.1"))]
),
critical=False,
)
.add_extension(
x509.ExtendedKeyUsage([x509.oid.ExtendedKeyUsageOID.SERVER_AUTH]), critical=False
)
.sign(root_key, hashes.SHA256())
)
root_pem = d / "zscaler-root.pem"
root_pem.write_bytes(root.public_bytes(serialization.Encoding.PEM))
leaf_pem = d / "leaf.pem"
leaf_pem.write_bytes(
leaf.public_bytes(serialization.Encoding.PEM)
+ leaf_key.private_bytes(
serialization.Encoding.PEM,
serialization.PrivateFormat.PKCS8,
serialization.NoEncryption(),
)
)
class Handler(http.server.BaseHTTPRequestHandler):
def do_GET(self) -> None: # noqa: N802 - stdlib hook name
body = json.dumps({"ok": True}).encode()
self.send_response(200)
self.send_header("content-type", "application/json")
self.send_header("content-length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *args: object) -> None:
return
server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Handler)
server_ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
server_ctx.minimum_version = ssl.TLSVersion.TLSv1_2
server_ctx.load_cert_chain(str(leaf_pem))
server.socket = server_ctx.wrap_socket(server.socket, server_side=True)
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
try:
yield {
"url": f"https://127.0.0.1:{server.server_address[1]}/v1/models",
"port": server.server_address[1],
"root": str(root_pem),
}
finally:
server.shutdown()
server.server_close()
class TestTrustPolicy:
def test_default_sources_are_system_and_bundled(self) -> None:
assert cert_store_sources() == {"system", "bundled"}
def test_default_verify_is_os_store_context(self) -> None:
truststore = pytest.importorskip("truststore")
verify = build_httpx_verify()
assert isinstance(verify, truststore.SSLContext)
assert verify.verify_mode == ssl.CERT_REQUIRED
def test_bundled_restores_legacy_default(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("HEADROOM_CERT_STORE", "bundled")
truststore = pytest.importorskip("truststore")
for ctx in (build_httpx_verify(), build_websocket_ssl()):
assert isinstance(ctx, ssl.SSLContext)
assert not isinstance(ctx, truststore.SSLContext)
assert ctx.verify_mode == ssl.CERT_REQUIRED
assert build_urlopen_context() is None
def test_unknown_tokens_fall_back_to_default(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("HEADROOM_CERT_STORE", "keychain")
assert cert_store_sources() == {"system", "bundled"}
def test_replacement_bundle_disables_os_store(
self, monkeypatch: pytest.MonkeyPatch, fake_zscaler: dict
) -> None:
monkeypatch.setenv("SSL_CERT_FILE", fake_zscaler["root"])
policy = describe_trust_policy()
assert policy["system_store_active"] is False
assert policy["replacement_bundle"]["env_var"] == "SSL_CERT_FILE"
def test_websocket_and_urlopen_use_os_store(self) -> None:
truststore = pytest.importorskip("truststore")
assert isinstance(build_websocket_ssl(), truststore.SSLContext)
assert isinstance(build_urlopen_context(), truststore.SSLContext)
class TestFakeZscaler:
def test_certifi_alone_rejects_the_chain(
self, monkeypatch: pytest.MonkeyPatch, fake_zscaler: dict
) -> None:
"""The original bug: no corporate root, every upstream call fails."""
monkeypatch.setenv("HEADROOM_CERT_STORE", "bundled")
with pytest.raises(httpx.ConnectError) as info:
httpx.get(fake_zscaler["url"], verify=build_httpx_verify())
assert tls_diagnostics.find_cert_verify_error(info.value) is not None
@pytest.mark.parametrize("var", ["HEADROOM_CA_BUNDLE", "NODE_EXTRA_CA_CERTS"])
def test_default_policy_plus_corporate_root_succeeds(
self, monkeypatch: pytest.MonkeyPatch, fake_zscaler: dict, var: str
) -> None:
"""Additive root on top of the OS store, no HEADROOM_TLS_STRICT needed."""
pytest.importorskip("truststore")
monkeypatch.setenv(var, fake_zscaler["root"])
resp = httpx.get(fake_zscaler["url"], verify=build_httpx_verify())
assert resp.status_code == 200
assert resp.json() == {"ok": True}
def test_websocket_context_trusts_corporate_root(
self, monkeypatch: pytest.MonkeyPatch, fake_zscaler: dict
) -> None:
import socket
monkeypatch.setenv("HEADROOM_CA_BUNDLE", fake_zscaler["root"])
ctx = build_websocket_ssl()
assert isinstance(ctx, ssl.SSLContext)
with socket.create_connection(("127.0.0.1", fake_zscaler["port"]), timeout=5) as raw:
with ctx.wrap_socket(raw, server_hostname="127.0.0.1") as tls:
assert tls.version() is not None
def test_probe_identifies_zscaler(self, fake_zscaler: dict) -> None:
info = tls_diagnostics.probe_presented_chain(
"127.0.0.1", fake_zscaler["port"], allow_private=True
)
assert info.reachable
assert info.issuer is not None and "Zscaler Root CA" in info.issuer
assert info.inspection_vendor == "Zscaler"
def test_failure_message_names_vendor_and_fix(
self, monkeypatch: pytest.MonkeyPatch, fake_zscaler: dict
) -> None:
monkeypatch.setenv("HEADROOM_CERT_STORE", "bundled")
with pytest.raises(httpx.ConnectError) as info:
httpx.get(fake_zscaler["url"], verify=build_httpx_verify())
message = tls_diagnostics.describe_upstream_failure(
info.value, fake_zscaler["url"], allow_private=True
)
assert message is not None
assert "Zscaler" in message
assert "HEADROOM_CA_BUNDLE" in message
assert "headroom doctor --network" in message
def test_request_path_probe_skips_private_hosts(self, fake_zscaler: dict) -> None:
"""A per-request upstream must not reveal an internal service's certificate."""
info = tls_diagnostics.probe_presented_chain("127.0.0.1", fake_zscaler["port"])
assert not info.reachable
assert info.issuer is None
assert "non-public" in (info.error or "")
def test_non_tls_errors_are_left_alone(self) -> None:
err = httpx.ConnectError("connection refused")
assert tls_diagnostics.describe_upstream_failure(err, "https://x.invalid") is None
def test_doctor_endpoint_probe_reports_vendor(
self, monkeypatch: pytest.MonkeyPatch, fake_zscaler: dict
) -> None:
monkeypatch.setenv("HEADROOM_CERT_STORE", "bundled")
failing = tls_diagnostics.probe_endpoint("fake", fake_zscaler["url"])
assert not failing.ok
assert failing.chain.inspection_vendor == "Zscaler"
monkeypatch.setenv("HEADROOM_CA_BUNDLE", fake_zscaler["root"])
monkeypatch.delenv("HEADROOM_CERT_STORE")
passing = tls_diagnostics.probe_endpoint("fake", fake_zscaler["url"])
assert passing.ok and passing.status == 200
class TestProcessTrust:
@pytest.fixture(autouse=True)
def _reset(self, monkeypatch: pytest.MonkeyPatch) -> Iterator[list[int]]:
calls: list[int] = []
truststore = pytest.importorskip("truststore")
monkeypatch.setattr(truststore, "inject_into_ssl", lambda: calls.append(1))
monkeypatch.setattr(ssl_context, "_process_trust_injected", False)
yield calls
def test_injects_once(self, _reset: list[int]) -> None:
assert ensure_process_trust() is True
assert ensure_process_trust() is True
assert _reset == [1]
def test_respects_replacement_bundle(
self, monkeypatch: pytest.MonkeyPatch, fake_zscaler: dict, _reset: list[int]
) -> None:
monkeypatch.setenv("REQUESTS_CA_BUNDLE", fake_zscaler["root"])
assert ensure_process_trust() is False
assert _reset == []
def test_respects_opt_out(self, monkeypatch: pytest.MonkeyPatch, _reset: list[int]) -> None:
monkeypatch.setenv("HEADROOM_CERT_STORE", "bundled")
assert ensure_process_trust() is False
assert _reset == []
class TestLoopbackNoProxy:
def test_gap_detected_only_with_proxy(self) -> None:
assert tls_diagnostics.loopback_no_proxy_gap({}) is None
assert (
tls_diagnostics.loopback_no_proxy_gap({"HTTPS_PROXY": "http://proxy:8080"})
== "HTTPS_PROXY"
)
assert (
tls_diagnostics.loopback_no_proxy_gap(
{"HTTPS_PROXY": "http://p:1", "no_proxy": "localhost,127.0.0.1"}
)
is None
)
def test_wrap_merges_both_spellings(self) -> None:
from headroom.cli.wrap import _ensure_loopback_no_proxy
env = {"HTTPS_PROXY": "http://p:1", "NO_PROXY": "corp.internal", "no_proxy": ".svc"}
written = _ensure_loopback_no_proxy(env)
assert set(written) == {"NO_PROXY", "no_proxy"}
assert env["NO_PROXY"] == env["no_proxy"] == "corp.internal,.svc,127.0.0.1,localhost,::1"
assert _ensure_loopback_no_proxy(env) == []
def test_wrap_leaves_env_alone_without_proxy(self) -> None:
from headroom.cli.wrap import _ensure_loopback_no_proxy
env = {"NO_PROXY": "corp.internal"}
assert _ensure_loopback_no_proxy(env) == []
assert env == {"NO_PROXY": "corp.internal"}
class TestDoctorRows:
def test_trust_policy_rows(self, monkeypatch: pytest.MonkeyPatch, fake_zscaler: dict) -> None:
from headroom.cli.doctor import PASS, WARN, check_trust_policy
pytest.importorskip("truststore")
assert check_trust_policy(describe_trust_policy()).status == PASS
monkeypatch.setenv("HEADROOM_CERT_STORE", "bundled")
assert check_trust_policy(describe_trust_policy()).status == WARN
monkeypatch.setenv("SSL_CERT_FILE", fake_zscaler["root"])
row = check_trust_policy(describe_trust_policy())
assert row.status == PASS and "SSL_CERT_FILE" in row.summary
def test_network_rows_and_it_request(
self, monkeypatch: pytest.MonkeyPatch, fake_zscaler: dict
) -> None:
from headroom.cli.doctor import FAIL, check_network_endpoints
monkeypatch.setenv("HEADROOM_CERT_STORE", "bundled")
report = tls_diagnostics.probe_endpoint("api.example", fake_zscaler["url"])
rows = check_network_endpoints([report], required={"api.example"})
assert rows[0].status == FAIL
assert "Zscaler Root CA" in rows[0].summary
it_row = rows[-1]
assert it_row.name == "it request"
assert "Zscaler" in (it_row.hint or "")
assert "127.0.0.1" in (it_row.hint or "")
def test_block_page_detection(self) -> None:
html = b"<html><title>Zscaler</title>Website blocked by policy</html>"
assert tls_diagnostics._looks_like_block_page(403, "text/html", html)
assert not tls_diagnostics._looks_like_block_page(
200, "text/html", b"<html>Hugging Face privacy policy</html>"
)
assert not tls_diagnostics._looks_like_block_page(403, "application/json", b"{}")
class TestStreamingErrorSurface:
"""The agent's error line names the TLS cause instead of a bare 502."""
@pytest.mark.asyncio
async def test_cert_failure_reaches_client_as_actionable_sse_error(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
from unittest.mock import AsyncMock, MagicMock
from tests.test_h2_stream_reset_retry import _mock_proxy, _run_stream
issued_by = tls_diagnostics.ChainInfo(
host="api.anthropic.com",
port=443,
reachable=True,
issuer="O=Zscaler Inc., CN=Zscaler Intermediate Root CA",
inspection_vendor="Zscaler",
)
monkeypatch.setattr(tls_diagnostics, "probe_presented_chain", lambda *a, **k: issued_by)
cert_err = ssl.SSLCertVerificationError(
1, "certificate verify failed: unable to get local issuer certificate"
)
cert_err.verify_message = "unable to get local issuer certificate"
connect_err = httpx.ConnectError("[SSL: CERTIFICATE_VERIFY_FAILED]")
connect_err.__cause__ = cert_err
proxy = _mock_proxy()
proxy.http_client.build_request = MagicMock(return_value=MagicMock())
proxy.http_client.send = AsyncMock(side_effect=connect_err)
result = await _run_stream(proxy)
body = b"".join([chunk async for chunk in result.body_iterator]).decode()
assert result.status_code == 502
assert "Zscaler is inspecting this connection" in body
assert "HEADROOM_CA_BUNDLE" in body
assert re.search(r"certificate for api\.anthropic\.com \(", body)
class TestReviewFixes:
def test_opt_out_aliases_are_honored(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("HEADROOM_CERT_STORE", "bundle")
assert cert_store_sources() == {"bundled"}
monkeypatch.setenv("HEADROOM_CERT_STORE", "os")
assert cert_store_sources() == {"system"}
def test_system_context_is_built_once_per_config(
self, monkeypatch: pytest.MonkeyPatch, fake_zscaler: dict
) -> None:
pytest.importorskip("truststore")
ssl_context._system_ctx_cache.clear()
first = build_websocket_ssl()
assert build_websocket_ssl() is first
monkeypatch.setenv("HEADROOM_CA_BUNDLE", fake_zscaler["root"])
assert build_websocket_ssl() is not first
def test_partial_chain_is_enabled(self) -> None:
pytest.importorskip("truststore")
flag = getattr(ssl, "VERIFY_X509_PARTIAL_CHAIN", 0)
if not flag:
pytest.skip("OpenSSL build without partial-chain support")
ctx = build_httpx_verify()
assert isinstance(ctx, ssl.SSLContext)
assert ctx.verify_flags & flag
def test_proxy_startup_only_injects_in_cli_owned_process(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
truststore = pytest.importorskip("truststore")
calls: list[int] = []
monkeypatch.setattr(truststore, "inject_into_ssl", lambda: calls.append(1))
monkeypatch.setattr(ssl_context, "_process_trust_injected", False)
monkeypatch.delenv(ssl_context.PROCESS_TRUST_ENV, raising=False)
assert ssl_context.ensure_process_trust_if_owned() is False
assert calls == []
monkeypatch.setenv(ssl_context.PROCESS_TRUST_ENV, "1")
assert ssl_context.ensure_process_trust_if_owned() is True
assert calls == [1]
def test_concurrent_probes_share_one_handshake(self, monkeypatch: pytest.MonkeyPatch) -> None:
import time as _time
from concurrent.futures import ThreadPoolExecutor
calls: list[str] = []
def slow_probe(host: str, port: int, **_: object) -> tls_diagnostics.ChainInfo:
calls.append(host)
_time.sleep(0.2)
return tls_diagnostics.ChainInfo(host=host, port=port, reachable=True)
monkeypatch.setattr(tls_diagnostics, "_probe_uncached", slow_probe)
with ThreadPoolExecutor(8) as pool:
list(
pool.map(
lambda _: tls_diagnostics.probe_presented_chain("api.example.com"), range(8)
)
)
assert calls == ["api.example.com"]
def test_private_allowed_probe_is_not_served_to_request_path(self, fake_zscaler: dict) -> None:
port = fake_zscaler["port"]
allowed = tls_diagnostics.probe_presented_chain("127.0.0.1", port, allow_private=True)
assert allowed.issuer
guarded = tls_diagnostics.probe_presented_chain("127.0.0.1", port)
assert guarded.issuer is None
def test_probe_ignores_os_proxy_settings(self, monkeypatch: pytest.MonkeyPatch) -> None:
import urllib.request
monkeypatch.setattr(urllib.request, "getproxies", lambda: {"https": "http://os-proxy:3128"})
assert tls_diagnostics._proxy_for("api.anthropic.com") is None
monkeypatch.setenv("HTTPS_PROXY", "http://env-proxy:8080")
assert tls_diagnostics._proxy_for("api.anthropic.com") == "http://env-proxy:8080"
def test_wrap_notice_goes_to_stderr(
self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
) -> None:
"""`wrap <tool> --prepare-only` stdout is machine-read JSON; keep it clean."""
import click
from headroom.cli import wrap as wrap_mod
monkeypatch.setenv("HTTPS_PROXY", "http://proxy:8080")
# Registered so monkeypatch restores them: the callback writes os.environ.
monkeypatch.setenv("NO_PROXY", "")
monkeypatch.setenv("no_proxy", "")
monkeypatch.setattr(wrap_mod, "_should_purge_context_tools", lambda ctx: False)
with click.Context(wrap_mod.wrap) as ctx:
ctx.invoke(wrap_mod.wrap.callback)
out, err = capsys.readouterr()
assert out == ""
assert "NO_PROXY" in err
+22 -5
View File
@@ -62,14 +62,21 @@ def ca_pem_file(tmp_path):
def _clean_env(monkeypatch):
"""Remove all CA-bundle env vars + the strict toggle for a clean state."""
"""Remove all CA-bundle env vars + the strict toggle for a clean state.
Pins ``HEADROOM_CERT_STORE=bundled`` so these tests keep covering the
certifi / env-bundle resolution without the OS trust store in front of it;
the OS-store default is covered in ``test_corporate_tls.py``.
"""
for var in (
"SSL_CERT_FILE",
"REQUESTS_CA_BUNDLE",
"NODE_EXTRA_CA_CERTS",
"HEADROOM_CA_BUNDLE",
"HEADROOM_TLS_STRICT",
):
monkeypatch.delenv(var, raising=False)
monkeypatch.setenv("HEADROOM_CERT_STORE", "bundled")
def _default_x509_ca_count() -> int:
@@ -256,10 +263,20 @@ class TestTlsStrictDisabled:
class TestBuildHttpxVerify:
def test_default_returns_true(self, monkeypatch):
"""No CA bundle, strict on → httpx's own default verification."""
def test_default_is_httpx_equivalent_certifi_context(self, monkeypatch):
"""No CA bundle, strict on → what httpx builds for verify=True (certifi).
Always a concrete verifying context, never a boolean.
"""
import certifi
_clean_env(monkeypatch)
assert build_httpx_verify() is True
ctx = build_httpx_verify()
assert isinstance(ctx, ssl.SSLContext)
assert ctx.verify_mode == ssl.CERT_REQUIRED
assert ctx.check_hostname is True
expected = ssl.create_default_context(cafile=certifi.where())
assert ctx.cert_store_stats()["x509_ca"] == expected.cert_store_stats()["x509_ca"]
def test_toggle_off_returns_relaxed_context(self, monkeypatch):
"""No CA bundle, strict OFF → default trust store with strict cleared."""
@@ -306,7 +323,7 @@ class TestBuildUrlopenContext:
assert created_context.alpn_protocols == ["http/1.1"]
def test_default_returns_none(self, monkeypatch):
"""No CA bundle, strict on → build_httpx_verify() is True, nothing to restrict."""
"""No CA bundle, strict on → no configured context; urlopen keeps its default."""
_clean_env(monkeypatch)
assert build_urlopen_context() is None
Generated
+12 -1
View File
@@ -1847,7 +1847,7 @@ wheels = [
[[package]]
name = "headroom-ai"
version = "0.38.0"
version = "0.39.1"
source = { editable = "." }
dependencies = [
{ name = "ast-grep-cli" },
@@ -1860,6 +1860,7 @@ dependencies = [
{ name = "tiktoken" },
{ name = "tomli", marker = "python_full_version < '3.11'" },
{ name = "tomlkit" },
{ name = "truststore" },
]
[package.optional-dependencies]
@@ -2203,6 +2204,7 @@ requires-dist = [
{ name = "transformers", marker = "extra == 'voice'", specifier = ">=5.5.0,<6.0" },
{ name = "tree-sitter", marker = "extra == 'code'", specifier = ">=0.25.2,<0.27" },
{ name = "tree-sitter-language-pack", marker = "extra == 'code'", specifier = ">=0.10.0,<1.0" },
{ name = "truststore", specifier = ">=0.10.0" },
{ name = "uvicorn", marker = "extra == 'dev'", specifier = ">=0.23.0,<1.0" },
{ name = "uvicorn", marker = "extra == 'mcp'", specifier = ">=0.23.0,<1.0" },
{ name = "uvicorn", marker = "extra == 'proxy'", specifier = ">=0.23.0,<1.0" },
@@ -7280,6 +7282,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/f0/ac/229b7d4589d2e5937310e72c6d46e89599d16a4a12b479ffa1499fee8eb8/triton-3.7.1-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:10ba85fa2cca4a2fbdeb36bf1cb082f2c252bda55bf9fccd74f65ec5bc647e68", size = 197824404, upload-time = "2026-06-17T19:53:42.772Z" },
]
[[package]]
name = "truststore"
version = "0.10.4"
source = { registry = "https://pypi.org/simple/" }
sdist = { url = "https://files.pythonhosted.org/packages/53/a3/1585216310e344e8102c22482f6060c7a6ea0322b63e026372e6dcefcfd6/truststore-0.10.4.tar.gz", hash = "sha256:9d91bd436463ad5e4ee4aba766628dd6cd7010cf3e2461756b3303710eebc301", size = 26169, upload-time = "2025-08-12T18:49:02.73Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/19/97/56608b2249fe206a67cd573bc93cd9896e1efb9e98bce9c163bcdc704b88/truststore-0.10.4-py3-none-any.whl", hash = "sha256:adaeaecf1cbb5f4de3b1959b42d41f6fab57b2b1666adb59e89cb0b53361d981", size = 18660, upload-time = "2025-08-12T18:49:01.46Z" },
]
[[package]]
name = "typer"
version = "0.21.1"