mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:07:29 +08:00
feat/https-compose-caddy
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
22d0905a73 |
fix(cli): pass configured SSL context to stdlib urlopen calls (#4011)
## Description Fixes #3976. On `uv tool` installs and other environments where certifi is not wired into the stdlib OpenSSL trust path, `headroom update` and bundled-binary downloads can fail behind corporate TLS inspection. These urllib calls now use the configured Headroom trust context, matching the proxy's TLS policy. The lazy helper import still falls back to urllib's default when the optional trust module cannot be imported. Errors while constructing an explicitly configured context now propagate instead of silently replacing the requested trust roots with stdlib defaults. The update checker retains its outer error-handling behavior and returns no update on failure. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring ## Changes Made - Pass `build_urlopen_context()` to the PyPI version check and release-binary download paths. - Catch only `ImportError` around the lazy helper import; let trust-context construction errors reach the caller. - Add valid and invalid replacement-bundle tests for both helpers and verify the update check still handles the error through its existing outer handler. - Update mocked `urlopen` functions to accept the `context` keyword used by the production call path. ## Testing - [x] Targeted unit tests pass - [ ] Full test suite run - [x] Ruff lint and format checks pass for changed files - [x] New tests added for the requested behavior - [ ] Type checking run - [ ] Manual live-network testing performed ### Test Output ```text python -m pytest --disable-plugin-autoload -p pytest_asyncio.plugin tests/test_ssl_context.py tests/test_binaries.py tests/test_update_check.py 95 passed in 5.10s python -m ruff check headroom/binaries.py headroom/update_check.py tests/test_ssl_context.py tests/test_binaries.py All checks passed! python -m ruff format --check headroom/binaries.py headroom/update_check.py tests/test_ssl_context.py tests/test_binaries.py 4 files already formatted git diff --check passed ``` ## Real Behavior Proof **Original reproduction:** On macOS behind a corporate TLS-inspecting network, `headroom update` failed with `CERTIFICATE_VERIFY_FAILED` when the inspection root was present in the OS trust store but absent from Python's stdlib OpenSSL bundle. The same machine's proxy reached its upstream through the configured root. The original fix routed the direct urllib calls through `build_urlopen_context()`. **Review regression checks:** On Windows with Python 3.13.13, tests using a valid `REQUESTS_CA_BUNDLE` confirm both helpers return a verifying `SSLContext`. With an existing file containing invalid PEM, both helpers now raise `ssl.SSLError`; the update-check caller still returns `None` through its outer error handler. All 95 tests in the affected TLS, binary-download, and update-check suites pass. **Not tested:** A live corporate TLS-inspection connection, Windows `uv tool` end-to-end behavior behind such a network, the full repository test suite, and the separate eval dataset download path. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: Not applicable. - Stable/default behavior changed: An invalid explicitly configured trust bundle now fails visibly rather than silently falling back to stdlib roots; unconfigured default behavior is unchanged. - Kill switch / disable path: No new switch; existing TLS environment settings remain available. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert this PR's changes. ## Review Readiness - [x] I have performed a self-review. - [x] This PR is ready for human review. ## Checklist - [x] My code follows the project's style guidelines. - [x] I have performed a self-review of my code. - [ ] I have added comments where the code needs additional explanation. - [ ] I have made corresponding documentation changes (none needed for this internal error-handling change). - [x] The changed files pass Ruff checks and the focused tests add no warnings. - [x] I have added tests that prove the requested behavior. - [ ] The full repository unit-test suite was run. - [x] I did not edit `CHANGELOG.md`. --------- Co-authored-by: dajiaohuang <dajiaohuang@users.noreply.github.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
66258c40a4 |
fix: make Headroom work behind Zscaler and other TLS-inspecting networks (#3831)
## Description Customers behind **Zscaler** (and Netskope, Palo Alto, Cisco Secure Access, …) report that Headroom "cannot work": every upstream request fails with `CERTIFICATE_VERIFY_FAILED`, and the agent only sees a bare 502. These gateways inspect TLS by re-signing every certificate with a company root. IT installs that root in the **OS certificate store** (macOS Keychain, Windows cert store), which is why browsers, `curl` and Claude Code keep working. Headroom never read that store: - The httpx upstream client trusted certifi only. - On macOS, the stdlib default reads the static file `/etc/ssl/cert.pem`, not the Keychain. - The Rust proxy trusted Mozilla's webpki roots only. Only 3 of about 40 outbound call sites used the shared CA helper. We had been patching one path at a time (#741, #998, #1308, #3244, #3344, #3358, #810, #2029). This PR sets **one trust policy, based on the OS store, used everywhere.** Closes #3358 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made **1. Verify upstream TLS through the OS store by default** (`headroom/proxy/ssl_context.py`) - New `HEADROOM_CERT_STORE`, default `system,bundled` (same shape as Claude Code's `CLAUDE_CODE_CERT_STORE`). It uses [`truststore`](https://truststore.readthedocs.io/), which pip has used by default since 24.2, plus certifi's public roots. - The same policy covers the HTTP upstream client, both WebSocket upstreams, and the `urlopen` paths (Copilot auth). - **The builders now always return a concrete, verifying `SSLContext`**, never a `bool`. The legacy default is built explicitly as httpx's `verify=True` equivalent (certifi). - OS-store contexts are cached per configuration. They set `VERIFY_X509_PARTIAL_CHAIN` so an exported intermediate still works as a trust anchor (matching `create_default_context()` on 3.13). On macOS and Windows the OS verifies the chain, so the Python 3.13 strict-mode rejection (#1308) no longer happens. - `HEADROOM_CA_BUNDLE` and `NODE_EXTRA_CA_CERTS` **add** roots. `SSL_CERT_FILE` and `REQUESTS_CA_BUNDLE` keep their **replacement** meaning and turn the OS store off. - `HEADROOM_CERT_STORE=bundled` restores the old behavior. The aliases `bundle`, `certifi`, `os` and `native` are also accepted. - `truststore.inject_into_ssl()` (covers `huggingface_hub`, tiktoken and OTEL) runs only in processes the Headroom CLI owns. The CLI marks them with `HEADROOM_PROCESS_TRUST=1`, and proxy workers inherit it. Applications that embed Headroom keep their own process-wide TLS behavior. - `truststore` is added as a core dependency: pure Python, with no dependencies of its own. **2. Errors that say what's wrong** (`headroom/proxy/tls_diagnostics.py`) - When the certificate check fails, the Anthropic, streaming (SSE) and OpenAI passthrough errors now name who signed the certificate and give the fix. For example: *"…The certificate was issued by 'O=Zscaler Inc., CN=Zscaler Intermediate Root CA': Zscaler is inspecting this connection. Fix (pick one): …"* - The issuer comes from a separate handshake that inspects the certificate only: - It sends no data and requires **TLS ≥ 1.2**. - It has a 4-second timeout and is cached for 5 minutes per host. A burst of failures shares a single handshake. - It uses the same environment-variable proxies as the real client. - It targets the host that actually failed. - **Security:** on the request path it skips hosts that resolve to non-public addresses. It also connects to the exact address it checked, instead of looking the host up again. A per-request upstream therefore can't be used to read an internal service's certificate or to rebind DNS. **3. `headroom doctor`** - A new **tls trust** row reports the running proxy's policy, read from `/health`. - A **proxy env** warning appears when `HTTP(S)_PROXY` is set but `NO_PROXY` doesn't exempt loopback. - `--network` (plus a repeatable `--network-url`) probes the provider and model hosts. For each it shows the certificate issuer, whether Headroom trusts it, and whether a gateway block page came back. On failure it prints a paste-ready request for IT. **4. `headroom wrap`** - Merges `NO_PROXY` and `no_proxy`, appends `127.0.0.1,localhost,::1`, and writes the result to both, so a corporate proxy doesn't capture the agent's traffic to Headroom. - The notice goes to stderr, because the JSON that `--prepare-only` prints on stdout is parsed by `install.sh`. **5. Rust proxy** (`crates/headroom-proxy`) - reqwest now enables `rustls-tls-native-roots`, and `HEADROOM_CA_BUNDLE` / `NODE_EXTRA_CA_CERTS` add roots. - **Fixes a bug already on `main`: `wss://` upstreams panicked.** Both ring and aws-lc-rs are compiled into this binary, so `rustls::ClientConfig::builder()`, which `tokio_tungstenite::connect_async` calls, panics with "no process-level CryptoProvider available". WebSocket upstreams now use an explicit config: the ring provider, webpki + OS + additive roots, and ALPN `http/1.1`. **6. Docs** - New page **Corporate Networks**: trust settings, HTTP proxies, a domain table for IT, and notes on Caution/Isolate, SSE buffering, process bypass, tenant restrictions and `HF_ENDPOINT` mirrors. - README section updated, correcting a wrong claim about which trust store the Rust core uses. **7. `uv.lock`** - `main`'s lockfile plus `truststore` only. The diff is 11 lines: the new package entry and two lines in `headroom-ai`'s dependency lists. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_corporate_tls.py tests/test_ssl_context.py tests/test_copilot_auth.py tests/test_cli_doctor.py -q ============================= 278 passed in 2.88s ============================== $ pytest <117 related files: doctor, wrap, streaming, codex, copilot, ssl, upstream, websocket, passthrough> -q ====== 4 failed, 1876 passed, 25 skipped, 7 warnings in 132.42s (0:02:12) ====== # the 4 failures: tests/test_codex_live.py ordering issue, identical on main (see Additional Notes) $ cargo test -p headroom-proxy passed=478 failed=0 suites=37 $ cargo clippy -p headroom-proxy --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) $ ruff check . -> All checks passed! $ ruff format --check . -> 1660 files already formatted $ mypy headroom --ignore-missing-imports -> Success: no issues found in 544 source files $ uv export --frozen --no-dev --no-hashes | grep truststore -> truststore==0.10.4 ``` `tests/test_corporate_tls.py` runs a **fake Zscaler** HTTPS server. Its root is `O=Zscaler Inc.` with **non-critical basicConstraints**, like the real 2014 root. The tests check that: - certifi alone rejects it; - the default policy plus `HEADROOM_CA_BUNDLE` or `NODE_EXTRA_CA_CERTS` accepts it; - the probe names Zscaler, and the request path skips private hosts; - a streaming request surfaces the actionable message in a 502 SSE error; - a burst of failures shares one handshake; - the notices and JSON output from `wrap` go to the right streams; - process injection is gated as described. ## Real Behavior Proof - **Environment:** - macOS (Darwin 25.6, Apple Silicon), Python 3.12.13 venv - Python 3.13.15 via `uv run --python 3.13` - Linux `python:3.13-slim` in Docker - Rust proxy built from this branch - **Steps and results:** 1. **Standalone fake-Zscaler script** (same root as above) on Python 3.13: - old stdlib path, `ssl.create_default_context(cafile=root)` → `CERTIFICATE_VERIFY_FAILED: Missing Authority Key Identifier` - new `truststore` context + root → `200` - identical results on macOS and in `python:3.13-slim`. 2. **`headroom doctor --network`** from this branch on an uninspected network: every endpoint shows `reachable, certificate trusted` (api.anthropic.com 401, api.openai.com 401, api.githubcopilot.com 400, huggingface.co 200, openaipublic 404); the tls trust row shows `OS certificate store + certifi`. 3. **Rust `wss://` connection** (temporary test) to `wss://echo.websocket.org`: - `tokio_tungstenite::connect_async` as on `main` → **panic** (no process-level CryptoProvider) - `connect_async_tls_with_config` with the new config → `Ok(())` - **Not tested:** - A real Zscaler, Netskope or Palo Alto tenant. - Windows. `truststore` supports the Windows cert store, but no Windows run was done here. - A root installed into the OS store. That needs admin rights, so the additive-bundle path stood in for it; it goes through the same OS verifier. ## Runtime Rollout Safety - **Rollout-managed feature(s):** none. This is trust-store selection, not a rollout-gated feature. - **Minimum rollout channel:** n/a. - **Stable/default behavior changed:** yes. Upstream TLS now also trusts the OS certificate store; certifi is still loaded, so public endpoints behave the same. The Rust `wss://` upstream no longer panics. - **Kill switch / disable path:** `HEADROOM_CERT_STORE=bundled` restores certifi-only trust. An explicit `SSL_CERT_FILE` / `REQUESTS_CA_BUNDLE` behaves exactly as before. - **Unsafe override required:** none. Nothing here disables certificate verification. - **Qualification impact:** none expected. Default behavior on uninspected networks is unchanged. - **Rollback path:** revert the PR, or set `HEADROOM_CERT_STORE=bundled`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) n/a (CLI and proxy behavior). ## Additional Notes **Review response** (@JerrettDavis): - **CodeQL:** the 5 alerts that were still open on `efe173c9` had two causes. - **Certificate-validation alerts:** CodeQL traced a `None` returned by the optional-context helpers through the `configured or default` fallback into `verify=`. All contexts are now built by a single `_build_context()`, and every branch returns a verifying `SSLContext`. `build_httpx_verify()` is typed `-> ssl.SSLContext`. - **TLS-version alerts:** every context the module creates now sets an explicit `minimum_version = TLSv1_2`. That includes the certificate-inspection probe, whose no-validation scope stays confined to that handshake behind the public-address / environment-proxy CONNECT guard. The test server also requires TLS 1.2, and the host assertion in the test is a regex. - **Verified locally with CodeQL 2.27.1** before pushing: - the same 5 alerts reproduce on `efe173c9`, and 0 remain on the fix commit; - `python-security-extended` over every Python file this PR changes finds the same 14 pre-existing alerts on `main`'s versions and on the branch's versions, so the PR adds none. - **`uv.lock`:** rebuilt as `main`'s lockfile plus `truststore` only (11 lines). Re-locking with current uv also rewrites markers across the file, because `main`'s lockfile predates the current `pyproject.toml`. That is left for a separate lock refresh. **Earlier review:** a high-effort code review of this branch produced 10 findings plus 3 minor notes. Commit `fix: address review findings…` resolves 9 of the 10 and two of the notes; the third note (lockfile markers) checked out as a false alarm. The remaining finding is the last follow-up below. **Follow-ups (not in this PR):** - SSE keep-alives and HTTP/1.1 fallback when a gateway buffers a stream. - A `wrap` pre-flight that steps aside when the upstream is unreachable. - The Rust core's fastembed/hf-hub embedding download, which hard-codes WebPki roots (needs an hf-hub patch; BM25 is the fallback, and `HF_ENDPOINT` is documented). - A CI job with a fake root installed in the OS store on macOS and Windows. - Map certificate failures once, in a shared layer, so batch, Gemini and the other handlers get the TLS hint too. **Known unrelated failures:** - 4 `tests/test_codex_live.py` tests fail when run after other files in this set, both on `main` and on this branch. They pass when run on their own. - After merging the latest `main` (thanks for the refresh), `tests/test_cli/test_wrap_helpers.py::test_start_proxy_strips_ambient_worker_configuration` and `::test_start_proxy_timeout_kills_failed_new_process` fail locally with `TypeError: <lambda>() takes 0 positional arguments but 1 was given`. They fail the same way on a clean checkout of current `main`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
5baa439ebf |
fix(proxy): restrict urlopen SSL context ALPN to http/1.1 (#3567)
## Description `build_urlopen_context()` in `headroom/proxy/ssl_context.py` is only used by `copilot_auth.py`'s `_urlopen()` helper, which wraps stdlib `urllib.request`/`http.client` for the GitHub Copilot device-code auth flow. It was reusing whatever `build_httpx_verify()` returns, which (when a custom CA bundle or `HEADROOM_TLS_STRICT=0` is configured) advertises ALPN `["h2", "http/1.1"]` -- correct for the proxy's httpx upstream client, but `http.client` has no HTTP/2 support at all. Under a TLS-inspecting corporate MITM that honors ALPN (GlobalProtect, per the issue), the far end negotiates HTTP/2 with the client. `http.client` then tries to read the response as HTTP/1.1 and misreads the HTTP/2 connection-preface/frame bytes as a garbled status line, surfacing as `http.client.BadStatusLine: ]MAGIC: Bad MAGIC` on `POST https://github.com/login/device/code`. Closes #3344 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `build_urlopen_context()` now restricts the returned context's ALPN to `["http/1.1"]` only, since its sole caller (`urllib.request`) can't speak anything else. - Left `build_httpx_verify()`, `find_ca_bundle()`, and the shared context builders untouched -- the proxy's httpx client still offers `h2` as before. - Added regression tests in `tests/test_ssl_context.py` covering the custom-CA-bundle case, the `HEADROOM_TLS_STRICT=0` case, and the no-bundle/default case (unaffected, still returns `None`). ## Testing - [x] Unit tests pass (`pytest tests/test_ssl_context.py`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --no-project --with pytest --with urllib3 pytest tests/test_ssl_context.py -v ... tests/test_ssl_context.py::TestBuildUrlopenContext::test_custom_ca_context_only_offers_http_1_1 PASSED tests/test_ssl_context.py::TestBuildUrlopenContext::test_default_returns_none PASSED tests/test_ssl_context.py::TestBuildUrlopenContext::test_toggle_off_context_only_offers_http_1_1 PASSED ... ======================== 34 passed, 1 warning in 3.50s ========================= ``` I confirmed the new tests are real regression tests, not tautologies: reverting the one-line fix (removing the `set_alpn_protocols(["http/1.1"])` call) makes `test_custom_ca_context_only_offers_http_1_1` and `test_toggle_off_context_only_offers_http_1_1` fail, because the context would still carry `["h2", "http/1.1"]`. ## Real Behavior Proof - Environment: this repo's `main`, Python 3.13, pytest run via an isolated `uv` environment (the editable install's Rust extension build wasn't needed for this pure-Python module/test). - Exact command / steps: shown above. - Observed result: all 34 tests in `tests/test_ssl_context.py` pass, including the 3 new ones for `build_urlopen_context()`. - Not tested: an actual TLS-inspecting MITM / live GitHub Copilot device-auth flow -- I don't have that network setup available, so this is verified at the ALPN-configuration level (the confirmed root cause) rather than an end-to-end repro of the corporate proxy environment. ## Runtime Rollout Safety - Rollout-managed feature(s): none -- this only affects TLS context construction for the `urllib.request`-based Copilot device-auth path, gated the same way it already was (custom CA bundle env var, or `HEADROOM_TLS_STRICT=0`). - Minimum rollout channel: n/a - Stable/default behavior changed: no -- the no-bundle/default case (`build_httpx_verify()` returns `True`) is unchanged, still returns `None`. Only the ALPN list on an already-returned custom context is narrowed. - Kill switch / disable path: n/a -- restricting ALPN to `http/1.1` can only fix broken connections for this urlopen-only caller; it can't regress anything since `http.client` never spoke `h2` to begin with. - Unsafe override required: no - Qualification impact: none - Rollback path: revert the commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (n/a -- internal helper, no public docs reference its ALPN behavior) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` |
||
|
|
9fde127534 |
fix(proxy): relocate stray system-role messages to the top-level system param (#765) (#1357)
## Description On requests large enough to trigger compression, the proxy emitted an upstream Anthropic request whose `messages[0]` had `role: "system"`. Anthropic's Messages API rejects any `system` role inside `messages[]`: ``` 400 invalid_request_error: "messages.0: use the top-level 'system' parameter for the initial system prompt" ``` The original request correctly carries its system prompt in the top-level `system` parameter; a compression/transform/pipeline step relocates the harness system block into `messages[0]`, so the request fails outright (intermittent only because it requires a context large enough to compress). This adds a wire-contract guard in the Anthropic forwarder: as the **last** step before sending upstream (after every transform, memory injection, tool sort, and pipeline extension, covering both the Bedrock and direct paths), any stray `role="system"` message is relocated out of `messages[]` and merged back into the top-level `system` parameter. Content order is preserved (existing system first, relocated content after) and block-level `cache_control` survives. The guard is a no-op on the common path (no system-role entry → inputs pass through unchanged). Closes #765 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/helpers.py`: new pure helper `relocate_system_messages_to_top_level(messages, system) -> (clean_messages, new_system, changed)` plus `_system_message_to_blocks`. Handles `system` being `None`/`str`/`list`, never drops content, preserves order and content blocks. - `headroom/proxy/handlers/anthropic.py`: invoke the guard just before the byte-faithful forward block; on relocation, update `body["messages"]`/`body["system"]`, mark the body mutated (`system_role_relocated`) so the byte-faithful forwarder re-serializes, and log a warning. - `tests/test_proxy_handler_helpers.py`: 3 unit tests (relocate stray system into top-level, append-to-existing-system order, no-op without a system entry). - `CHANGELOG.md`: Bug Fixes entry. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --extra dev python -m pytest tests/test_proxy_handler_helpers.py -q 29 passed in 4.95s # Regression on the forward path (byte-faithful forwarding, system-prompt immutability, cache stability): $ uv run --extra dev python -m pytest tests/test_proxy_handler_helpers.py tests/test_proxy_byte_faithful_forwarding.py tests/test_proxy_system_prompt_immutable.py tests/test_proxy_anthropic_cache_stability.py -q 90 passed, 15 warnings in 29.72s $ uv run ruff check headroom/proxy/helpers.py headroom/proxy/handlers/anthropic.py tests/test_proxy_handler_helpers.py All checks passed! $ uv run ruff format --check ... # 3 files already formatted $ uv run mypy headroom/proxy/helpers.py headroom/proxy/handlers/anthropic.py Success: no issues found in 2 source files ``` ## Test verification (RED → GREEN) The new tests exercise the guard directly and import the new helper at module top, so reverting the production fix makes them fail at collection. **RED — production fix reverted (helper removed):** ```text ImportError while importing test module 'tests/test_proxy_handler_helpers.py'. E ImportError: cannot import name 'relocate_system_messages_to_top_level' from 'headroom.proxy.helpers' =========================== 1 error in 0.41s =============================== ``` **GREEN — production fix applied:** ```text tests/test_proxy_handler_helpers.py ... [100%] ======================= 3 passed, 26 deselected in 1.50s ======================= ``` ## Real Behavior Proof - Environment: Python 3.13, `uv run` in this repo, branch `fix/issue-765`. - Exact command / steps: ran the guard on a body in the exact #765 failure shape — `system: None` and a `role="system"` harness block at `messages[0]`: - Observed result: ```text BEFORE: messages[0].role = system (Anthropic 400 trigger) changed = True AFTER roles = ['user', 'assistant'] system param = [{"type": "text", "text": "You are Claude Code. <system-reminder>...</system-reminder>"}] OK: no role=system in messages[]; system content preserved in top-level param ``` The illegal `role="system"` entry is removed from `messages[]` and its content lands in the top-level `system` parameter — exactly the body Anthropic accepts. - Not tested: a full live 250k+-token Claude Code session against the real Anthropic API (needs a large live context + API key); the fix is validated at the request-shaping boundary the 400 is raised on. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes The guard intentionally fires at the forwarder boundary rather than in any single transform: the issue's captures show the relocation can originate from the compression path, and pipeline extensions / hooks can also mutate `messages` late. Enforcing Anthropic's wire contract once, at the point the body is serialized upstream, fixes the 400 regardless of which step introduced the stray entry and matches the architecture invariant "never produce a `system`-role entry within `messages[]`". --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
52068dd650 |
fix(tls): add HEADROOM_TLS_STRICT=0 toggle for corporate SSL inspection (#1308) (#1341)
## Description Behind a corporate TLS-inspection proxy (Zscaler, Netskope) on Python 3.13+, Headroom can't reach the network even with the corporate root correctly installed and trusted. Every path fails with: ``` [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Basic Constraints of CA cert not marked critical ``` This isn't a missing-CA problem — the cert is found and trusted. Python 3.13 + OpenSSL 3.x enable `VERIFY_X509_STRICT` by default, which enforces RFC 5280 §4.2.1.9 (a CA cert's `basicConstraints` MUST be marked critical). Inspection roots set `CA:TRUE` without the critical bit, so the chain is rejected. Adding the CA to a bundle does nothing — it's the strict check that fails, and the existing README section only covers `unable to get local issuer certificate`. There are two independent sources of the strict flag (both reported in the issue): Python's own `ssl.create_default_context()` (hits the httpx upstream client), and urllib3 ≥ 2.5's `create_urllib3_context()` (hits the `huggingface_hub` model-download path). Closes #1308 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `ssl_context.py`: added `HEADROOM_TLS_STRICT`. `tls_strict_disabled()` reads the toggle (off-values `0/false/no/off`, default strict). `build_httpx_verify()` resolves the httpx `verify=` value: a configured CA bundle wins; otherwise, when the toggle is off, a default-trust-store context with **only** `VERIFY_X509_STRICT` cleared (so a corporate root that lives in the OS store but trips strict mode still validates); otherwise `True` (httpx default). `apply_global_tls_relaxation()` monkeypatches urllib3's `create_urllib3_context` to drop the strict flag — idempotent, guarded, no-op if urllib3 is absent or the toggle is on. - `server.py`: the proxy's httpx upstream client now uses `build_httpx_verify()` instead of `find_ca_bundle()`-or-`True`. - `cli/proxy.py`: calls `apply_global_tls_relaxation()` at module import, before `huggingface_hub`/`requests` import and cache their context. - README: a distinct SSL-inspection subsection for the `Basic Constraints ... not marked critical` failure, separate from `unable to get local issuer certificate`. Documents that the Rust core's ONNX download (`cdn.pyke.io`) uses a separate stack (rustls/OS trust store) unaffected by the toggle — corporate root must be in the Windows **machine** store, or pre-provision via `ORT_STRATEGY=system`. Chain validation, signature, expiry, and hostname checks all stay on — `HEADROOM_TLS_STRICT=0` is strictly narrower than `verify=False`. Default is strict, matching Python's own default. ## Testing - [x] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_ssl_context.py -q 31 passed # 19 existing + 12 new (TestTlsStrictDisabled, TestBuildHttpxVerify, TestApplyGlobalTlsRelaxation). ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.10.11 (OpenSSL build that exposes `VERIFY_X509_STRICT`). - Exact command / steps: exercised the module directly — set/unset `HEADROOM_TLS_STRICT`, inspected the resolved httpx `verify` value and the urllib3 context's `verify_flags`. - Observed result: default → `verify=True` (strict preserved); `HEADROOM_TLS_STRICT=0` → httpx gets an `SSLContext` with `VERIFY_X509_STRICT` cleared but `verify_mode == CERT_REQUIRED` and the full default trust store (cert_store x509_ca > 1); `apply_global_tls_relaxation()` patches `urllib3.util.ssl_.create_urllib3_context` so new contexts have the strict flag cleared, and is idempotent. A configured `SSL_CERT_FILE` still wins over the toggle. - Not tested: an actual handshake through a live Zscaler/Netskope MITM on Python 3.13 — I don't have that environment. The fix targets exactly the flag the issue identifies (`VERIFY_X509_STRICT`) on both reported context builders; I verified the flag manipulation and resolution logic directly rather than simulating the proxy. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - The toggle is opt-in and defaults to strict, so behavior is unchanged unless a user explicitly sets `HEADROOM_TLS_STRICT=0`. It clears only the strict flag, never disables verification. - The httpx path uses an explicit context (clean, testable); the urllib3 path needs a monkeypatch because `huggingface_hub` → `requests` builds its context internally and never sees ours. - CHANGELOG.md isn't touched — release-please generates it from the `fix(tls):` commit subject. - I scoped this to the two Python TLS stacks the issue calls out and documented (rather than tried to patch) the separate Rust/ONNX path, since that one resolves through the OS trust store and isn't something this Python toggle can reach. |
||
|
|
561ba17ec2 |
fix(proxy): build SSL contexts for custom CA bundles (#1134)
## Description Build explicit `ssl.SSLContext` objects for custom CA bundles configured through `SSL_CERT_FILE` and `REQUESTS_CA_BUNDLE`. Python 3.13 / newer OpenSSL can reject some enterprise/private PKI roots that platform TLS stacks accept, for example roots without a `keyUsage` extension. The new custom-CA contexts keep certificate verification enabled while clearing only `ssl.VERIFY_X509_STRICT`. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Build replacement `SSLContext` objects for `SSL_CERT_FILE` and `REQUESTS_CA_BUNDLE` instead of passing raw CA bundle paths to httpx. - Clear only `ssl.VERIFY_X509_STRICT` for operator-provided custom CA contexts; certificate verification, hostname verification, expiry checks, and chain validation stay enabled. - Preserve `NODE_EXTRA_CA_CERTS` additive behavior by loading the extra CA bundle on top of the default/system trust store. - Update existing SSL context tests for replacement CA contexts, env-var priority, missing-path fallthrough, and strict-mode relaxation. - Add an Unreleased changelog entry for the proxy bug fix. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ PYTHONPATH=. .venv/bin/python -m pytest tests/test_ssl_context.py -q collected 12 items tests/test_ssl_context.py ............ [100%] 12 passed, 1 warning in 0.11s ``` ```text $ uv tool run ruff check headroom/proxy/ssl_context.py tests/test_ssl_context.py CHANGELOG.md All checks passed! $ uv tool run ruff format --check headroom/proxy/ssl_context.py tests/test_ssl_context.py 2 files already formatted ``` ## Real Behavior Proof - Environment: macOS, Python 3.13.2, Headroom checkout on this branch, `SSL_CERT_FILE` / `REQUESTS_CA_BUNDLE` / `NODE_EXTRA_CA_CERTS` pointed at an enterprise/private PKI CA bundle. Upstream HTTPS endpoint name is redacted. - Exact command / steps: Ran a Python smoke script that imports `headroom.proxy.ssl_context.find_ca_bundle()`, passes the returned verifier into `httpx.AsyncClient(verify=...)`, and performs a GET against the enterprise HTTPS endpoint that previously failed with OpenSSL strict verification. - Observed result: The request used an `SSLContext`, strict X.509 verification was disabled for that custom CA context, and the request reached the upstream HTTP response: ```text verify_type SSLContext strict_enabled False status 302 ``` - Not tested: Full `uv run pytest`, `uv sync --extra dev`, and `mypy headroom` in this local environment. The editable build currently fails before tests run because Cargo's `ort-sys` build script cannot download ORT prebuilt binaries due an unrelated local certificate verification error against the ORT CDN. No dependency or lockfile changes are included in this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A. ## Additional Notes No dependencies or lockfiles changed. Documentation is unchanged because this is a bug fix to existing custom CA environment-variable behavior rather than a new user-facing configuration surface. Signed-off-by: Mark Phelps <209477+markphelps@users.noreply.github.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
c98728363a |
fix(proxy): treat NODE_EXTRA_CA_CERTS as additive, not replacement (#998) (#1031)
## Description `find_ca_bundle()` returns the `NODE_EXTRA_CA_CERTS` path as a bare string passed to httpx's `verify=` parameter, which makes it the sole trust store. When that bundle contains only a private/internal root (the common corporate setup), all public upstreams (`api.anthropic.com`, `api.openai.com`) fail TLS verification with `CERTIFICATE_VERIFY_FAILED`, returning 502. This is the inverse of #741: that fix added corporate CA support, but this regression means public CAs are no longer trusted when the extra bundle is not a full superset of the public roots. The fix builds an `ssl.SSLContext` via `create_default_context()` (keeps system/default roots) then `load_verify_locations()` (adds the extra cert), matching Node.js additive semantics. `SSL_CERT_FILE` and `REQUESTS_CA_BUNDLE` keep their existing replacement semantics. Closes #998 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Split `NODE_EXTRA_CA_CERTS` handling out of the replacement-semantics loop in `find_ca_bundle()` - When `NODE_EXTRA_CA_CERTS` is the source, return an `ssl.SSLContext` with default roots plus the extra cert, instead of a bare path string - Set ALPN protocols (`h2`, `http/1.1`) on the context to preserve HTTP/2 negotiation - Updated `test_node_extra_ca_certs_returns_path` to assert `ssl.SSLContext` return type - Added `test_node_extra_ca_certs_is_additive` verifying the context contains more than just the extra cert (default roots preserved) ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] New tests added for new functionality ### Test Output ```text tests/test_ssl_context.py::TestFindCaBundleNoEnvVars::test_returns_none_when_no_env_var_set PASSED tests/test_ssl_context.py::TestFindCaBundleWithValidPem::test_ssl_cert_file_returns_path PASSED tests/test_ssl_context.py::TestFindCaBundleWithValidPem::test_requests_ca_bundle_returns_path PASSED tests/test_ssl_context.py::TestFindCaBundleWithValidPem::test_node_extra_ca_certs_returns_ssl_context PASSED tests/test_ssl_context.py::TestFindCaBundleWithValidPem::test_node_extra_ca_certs_is_additive PASSED tests/test_ssl_context.py::TestFindCaBundlePriority::test_ssl_cert_file_beats_requests_ca_bundle PASSED tests/test_ssl_context.py::TestFindCaBundlePriority::test_ssl_cert_file_beats_node_extra_ca_certs PASSED tests/test_ssl_context.py::TestFindCaBundlePriority::test_requests_ca_bundle_beats_node_extra_ca_certs PASSED tests/test_ssl_context.py::TestFindCaBundleNonexistentPaths::test_nonexistent_path_is_skipped PASSED tests/test_ssl_context.py::TestFindCaBundleNonexistentPaths::test_all_nonexistent_returns_none PASSED tests/test_ssl_context.py::TestFindCaBundleNonexistentPaths::test_first_nonexistent_falls_through_to_valid PASSED 11 passed in 0.91s ``` ## Real Behavior Proof - Environment: Windows 11 Home 10.0.26200, Python 3.10.18 - Exact command / steps: Ran `python -m pytest tests/test_ssl_context.py -v` after applying the fix. The new `test_node_extra_ca_certs_is_additive` test verifies `ctx.cert_store_stats()['x509_ca'] > 1`, confirming the default trust store roots are preserved alongside the extra cert. If replacement semantics were used, only the single test CA would be loaded. - Observed result: All 11 SSL context tests pass. The additive context reports 143 x509_ca certs (system defaults + test cert), confirming default roots are preserved. - Not tested: No live TLS handshake to a public upstream with a private-only `NODE_EXTRA_CA_CERTS` bundle, but the `cert_store_stats` assertion proves the default roots are loaded into the context. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes ## Additional Notes The fix aligns with the design proposed in #741 and #745's own description (which said it "builds an SSLContext") but which the merged implementation did not follow. The `server.py` consumer does not need changes because httpx accepts `ssl.SSLContext` for `verify=`. ALPN protocols are set on the context to maintain HTTP/2 negotiation parity with httpx's internally-built contexts. |
||
|
|
e50fbb3e0d |
fix(ssl): upstream httpx client inherits SSL_CERT_FILE, REQUESTS_CA_BUNDLE, NODE_EXTRA_CA_CERTS (#745)
Closes #741 ## What Headroom is commonly deployed alongside Claude Code, which sets `NODE_EXTRA_CA_CERTS` to a custom CA bundle for corporate or internal CAs. Node.js inherits this automatically; Python's `httpx` does not. Every upstream request silently failed with `SSL: CERTIFICATE_VERIFY_FAILED`, causing 502s and retry loops in the client. ## Changes - New `headroom/proxy/ssl_context.py` with `build_ssl_context()` helper that checks `SSL_CERT_FILE` → `REQUESTS_CA_BUNDLE` → `NODE_EXTRA_CA_CERTS` (first match wins) and builds an `ssl.SSLContext` with the custom CA bundle loaded - `HeadroomProxy.start()` calls `build_ssl_context()` and passes the result as `verify=` to `httpx.AsyncClient`; falls back to `verify=True` (default httpx behaviour) when no env var is set - Logs which env var and path was used at `INFO` level; warns on set-but-missing paths - 10 unit tests covering: no env var → `None`, each var returns `SSLContext`, priority order, nonexistent paths skipped ## Priority order 1. `SSL_CERT_FILE` — standard POSIX/Python ssl override 2. `REQUESTS_CA_BUNDLE` — standard requests/httpx convention 3. `NODE_EXTRA_CA_CERTS` — Node.js / Claude Code convention |