Commit Graph
8 Commits
Author SHA1 Message Date
22d0905a73 fix(cli): pass configured SSL context to stdlib urlopen calls (#4011)
## Description

Fixes #3976. On `uv tool` installs and other environments where certifi
is not
wired into the stdlib OpenSSL trust path, `headroom update` and
bundled-binary
downloads can fail behind corporate TLS inspection. These urllib calls
now use
the configured Headroom trust context, matching the proxy's TLS policy.

The lazy helper import still falls back to urllib's default when the
optional
trust module cannot be imported. Errors while constructing an explicitly
configured context now propagate instead of silently replacing the
requested
trust roots with stdlib defaults. The update checker retains its outer
error-handling behavior and returns no update on failure.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring

## Changes Made

- Pass `build_urlopen_context()` to the PyPI version check and
release-binary
  download paths.
- Catch only `ImportError` around the lazy helper import; let
trust-context
  construction errors reach the caller.
- Add valid and invalid replacement-bundle tests for both helpers and
verify
the update check still handles the error through its existing outer
handler.
- Update mocked `urlopen` functions to accept the `context` keyword used
by the
  production call path.

## Testing

- [x] Targeted unit tests pass
- [ ] Full test suite run
- [x] Ruff lint and format checks pass for changed files
- [x] New tests added for the requested behavior
- [ ] Type checking run
- [ ] Manual live-network testing performed

### Test Output

```text
python -m pytest --disable-plugin-autoload -p pytest_asyncio.plugin tests/test_ssl_context.py tests/test_binaries.py tests/test_update_check.py
95 passed in 5.10s

python -m ruff check headroom/binaries.py headroom/update_check.py tests/test_ssl_context.py tests/test_binaries.py
All checks passed!

python -m ruff format --check headroom/binaries.py headroom/update_check.py tests/test_ssl_context.py tests/test_binaries.py
4 files already formatted

git diff --check
passed
```

## Real Behavior Proof

**Original reproduction:** On macOS behind a corporate TLS-inspecting
network,
`headroom update` failed with `CERTIFICATE_VERIFY_FAILED` when the
inspection
root was present in the OS trust store but absent from Python's stdlib
OpenSSL
bundle. The same machine's proxy reached its upstream through the
configured
root. The original fix routed the direct urllib calls through
`build_urlopen_context()`.

**Review regression checks:** On Windows with Python 3.13.13, tests
using a
valid `REQUESTS_CA_BUNDLE` confirm both helpers return a verifying
`SSLContext`. With an existing file containing invalid PEM, both helpers
now
raise `ssl.SSLError`; the update-check caller still returns `None`
through its
outer error handler. All 95 tests in the affected TLS, binary-download,
and
update-check suites pass.

**Not tested:** A live corporate TLS-inspection connection, Windows `uv
tool`
end-to-end behavior behind such a network, the full repository test
suite, and
the separate eval dataset download path.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Not applicable.
- Stable/default behavior changed: An invalid explicitly configured
trust
bundle now fails visibly rather than silently falling back to stdlib
roots;
  unconfigured default behavior is unchanged.
- Kill switch / disable path: No new switch; existing TLS environment
settings
  remain available.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this PR's changes.

## Review Readiness

- [x] I have performed a self-review.
- [x] This PR is ready for human review.

## Checklist

- [x] My code follows the project's style guidelines.
- [x] I have performed a self-review of my code.
- [ ] I have added comments where the code needs additional explanation.
- [ ] I have made corresponding documentation changes (none needed for
this
  internal error-handling change).
- [x] The changed files pass Ruff checks and the focused tests add no
warnings.
- [x] I have added tests that prove the requested behavior.
- [ ] The full repository unit-test suite was run.
- [x] I did not edit `CHANGELOG.md`.

---------

Co-authored-by: dajiaohuang <dajiaohuang@users.noreply.github.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-07 20:51:49 -05:00
66258c40a4 fix: make Headroom work behind Zscaler and other TLS-inspecting networks (#3831)
## Description

Customers behind **Zscaler** (and Netskope, Palo Alto, Cisco Secure
Access, …) report that Headroom "cannot work": every upstream request
fails with `CERTIFICATE_VERIFY_FAILED`, and the agent only sees a bare
502.

These gateways inspect TLS by re-signing every certificate with a
company root. IT installs that root in the **OS certificate store**
(macOS Keychain, Windows cert store), which is why browsers, `curl` and
Claude Code keep working. Headroom never read that store:

- The httpx upstream client trusted certifi only.
- On macOS, the stdlib default reads the static file
`/etc/ssl/cert.pem`, not the Keychain.
- The Rust proxy trusted Mozilla's webpki roots only.

Only 3 of about 40 outbound call sites used the shared CA helper. We had
been patching one path at a time (#741, #998, #1308, #3244, #3344,
#3358, #810, #2029). This PR sets **one trust policy, based on the OS
store, used everywhere.**

Closes #3358

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

**1. Verify upstream TLS through the OS store by default**
(`headroom/proxy/ssl_context.py`)
- New `HEADROOM_CERT_STORE`, default `system,bundled` (same shape as
Claude Code's `CLAUDE_CODE_CERT_STORE`). It uses
[`truststore`](https://truststore.readthedocs.io/), which pip has used
by default since 24.2, plus certifi's public roots.
- The same policy covers the HTTP upstream client, both WebSocket
upstreams, and the `urlopen` paths (Copilot auth).
- **The builders now always return a concrete, verifying `SSLContext`**,
never a `bool`. The legacy default is built explicitly as httpx's
`verify=True` equivalent (certifi).
- OS-store contexts are cached per configuration. They set
`VERIFY_X509_PARTIAL_CHAIN` so an exported intermediate still works as a
trust anchor (matching `create_default_context()` on 3.13). On macOS and
Windows the OS verifies the chain, so the Python 3.13 strict-mode
rejection (#1308) no longer happens.
- `HEADROOM_CA_BUNDLE` and `NODE_EXTRA_CA_CERTS` **add** roots.
`SSL_CERT_FILE` and `REQUESTS_CA_BUNDLE` keep their **replacement**
meaning and turn the OS store off.
- `HEADROOM_CERT_STORE=bundled` restores the old behavior. The aliases
`bundle`, `certifi`, `os` and `native` are also accepted.
- `truststore.inject_into_ssl()` (covers `huggingface_hub`, tiktoken and
OTEL) runs only in processes the Headroom CLI owns. The CLI marks them
with `HEADROOM_PROCESS_TRUST=1`, and proxy workers inherit it.
Applications that embed Headroom keep their own process-wide TLS
behavior.
- `truststore` is added as a core dependency: pure Python, with no
dependencies of its own.

**2. Errors that say what's wrong**
(`headroom/proxy/tls_diagnostics.py`)
- When the certificate check fails, the Anthropic, streaming (SSE) and
OpenAI passthrough errors now name who signed the certificate and give
the fix. For example: *"…The certificate was issued by 'O=Zscaler Inc.,
CN=Zscaler Intermediate Root CA': Zscaler is inspecting this connection.
Fix (pick one): …"*
- The issuer comes from a separate handshake that inspects the
certificate only:
  - It sends no data and requires **TLS ≥ 1.2**.
- It has a 4-second timeout and is cached for 5 minutes per host. A
burst of failures shares a single handshake.
  - It uses the same environment-variable proxies as the real client.
  - It targets the host that actually failed.
- **Security:** on the request path it skips hosts that resolve to
non-public addresses. It also connects to the exact address it checked,
instead of looking the host up again. A per-request upstream therefore
can't be used to read an internal service's certificate or to rebind
DNS.

**3. `headroom doctor`**
- A new **tls trust** row reports the running proxy's policy, read from
`/health`.
- A **proxy env** warning appears when `HTTP(S)_PROXY` is set but
`NO_PROXY` doesn't exempt loopback.
- `--network` (plus a repeatable `--network-url`) probes the provider
and model hosts. For each it shows the certificate issuer, whether
Headroom trusts it, and whether a gateway block page came back. On
failure it prints a paste-ready request for IT.

**4. `headroom wrap`**
- Merges `NO_PROXY` and `no_proxy`, appends `127.0.0.1,localhost,::1`,
and writes the result to both, so a corporate proxy doesn't capture the
agent's traffic to Headroom.
- The notice goes to stderr, because the JSON that `--prepare-only`
prints on stdout is parsed by `install.sh`.

**5. Rust proxy** (`crates/headroom-proxy`)
- reqwest now enables `rustls-tls-native-roots`, and
`HEADROOM_CA_BUNDLE` / `NODE_EXTRA_CA_CERTS` add roots.
- **Fixes a bug already on `main`: `wss://` upstreams panicked.** Both
ring and aws-lc-rs are compiled into this binary, so
`rustls::ClientConfig::builder()`, which
`tokio_tungstenite::connect_async` calls, panics with "no process-level
CryptoProvider available". WebSocket upstreams now use an explicit
config: the ring provider, webpki + OS + additive roots, and ALPN
`http/1.1`.

**6. Docs**
- New page **Corporate Networks**: trust settings, HTTP proxies, a
domain table for IT, and notes on Caution/Isolate, SSE buffering,
process bypass, tenant restrictions and `HF_ENDPOINT` mirrors.
- README section updated, correcting a wrong claim about which trust
store the Rust core uses.

**7. `uv.lock`**
- `main`'s lockfile plus `truststore` only. The diff is 11 lines: the
new package entry and two lines in `headroom-ai`'s dependency lists.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_corporate_tls.py tests/test_ssl_context.py tests/test_copilot_auth.py tests/test_cli_doctor.py -q
============================= 278 passed in 2.88s ==============================

$ pytest <117 related files: doctor, wrap, streaming, codex, copilot, ssl, upstream, websocket, passthrough> -q
====== 4 failed, 1876 passed, 25 skipped, 7 warnings in 132.42s (0:02:12) ======
# the 4 failures: tests/test_codex_live.py ordering issue, identical on main (see Additional Notes)

$ cargo test -p headroom-proxy
passed=478 failed=0 suites=37
$ cargo clippy -p headroom-proxy --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s)

$ ruff check .                                            -> All checks passed!
$ ruff format --check .                                   -> 1660 files already formatted
$ mypy headroom --ignore-missing-imports                  -> Success: no issues found in 544 source files
$ uv export --frozen --no-dev --no-hashes | grep truststore -> truststore==0.10.4
```

`tests/test_corporate_tls.py` runs a **fake Zscaler** HTTPS server. Its
root is `O=Zscaler Inc.` with **non-critical basicConstraints**, like
the real 2014 root. The tests check that:

- certifi alone rejects it;
- the default policy plus `HEADROOM_CA_BUNDLE` or `NODE_EXTRA_CA_CERTS`
accepts it;
- the probe names Zscaler, and the request path skips private hosts;
- a streaming request surfaces the actionable message in a 502 SSE
error;
- a burst of failures shares one handshake;
- the notices and JSON output from `wrap` go to the right streams;
- process injection is gated as described.

## Real Behavior Proof

- **Environment:**
  - macOS (Darwin 25.6, Apple Silicon), Python 3.12.13 venv
  - Python 3.13.15 via `uv run --python 3.13`
  - Linux `python:3.13-slim` in Docker
  - Rust proxy built from this branch
- **Steps and results:**
1. **Standalone fake-Zscaler script** (same root as above) on Python
3.13:
- old stdlib path, `ssl.create_default_context(cafile=root)` →
`CERTIFICATE_VERIFY_FAILED: Missing Authority Key Identifier`
     - new `truststore` context + root → `200`
     - identical results on macOS and in `python:3.13-slim`.
2. **`headroom doctor --network`** from this branch on an uninspected
network: every endpoint shows `reachable, certificate trusted`
(api.anthropic.com 401, api.openai.com 401, api.githubcopilot.com 400,
huggingface.co 200, openaipublic 404); the tls trust row shows `OS
certificate store + certifi`.
3. **Rust `wss://` connection** (temporary test) to
`wss://echo.websocket.org`:
- `tokio_tungstenite::connect_async` as on `main` → **panic** (no
process-level CryptoProvider)
     - `connect_async_tls_with_config` with the new config → `Ok(())`
- **Not tested:**
  - A real Zscaler, Netskope or Palo Alto tenant.
- Windows. `truststore` supports the Windows cert store, but no Windows
run was done here.
- A root installed into the OS store. That needs admin rights, so the
additive-bundle path stood in for it; it goes through the same OS
verifier.

## Runtime Rollout Safety

- **Rollout-managed feature(s):** none. This is trust-store selection,
not a rollout-gated feature.
- **Minimum rollout channel:** n/a.
- **Stable/default behavior changed:** yes. Upstream TLS now also trusts
the OS certificate store; certifi is still loaded, so public endpoints
behave the same. The Rust `wss://` upstream no longer panics.
- **Kill switch / disable path:** `HEADROOM_CERT_STORE=bundled` restores
certifi-only trust. An explicit `SSL_CERT_FILE` / `REQUESTS_CA_BUNDLE`
behaves exactly as before.
- **Unsafe override required:** none. Nothing here disables certificate
verification.
- **Qualification impact:** none expected. Default behavior on
uninspected networks is unchanged.
- **Rollback path:** revert the PR, or set
`HEADROOM_CERT_STORE=bundled`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

n/a (CLI and proxy behavior).

## Additional Notes

**Review response** (@JerrettDavis):
- **CodeQL:** the 5 alerts that were still open on `efe173c9` had two
causes.
- **Certificate-validation alerts:** CodeQL traced a `None` returned by
the optional-context helpers through the `configured or default`
fallback into `verify=`. All contexts are now built by a single
`_build_context()`, and every branch returns a verifying `SSLContext`.
`build_httpx_verify()` is typed `-> ssl.SSLContext`.
- **TLS-version alerts:** every context the module creates now sets an
explicit `minimum_version = TLSv1_2`. That includes the
certificate-inspection probe, whose no-validation scope stays confined
to that handshake behind the public-address / environment-proxy CONNECT
guard. The test server also requires TLS 1.2, and the host assertion in
the test is a regex.
  - **Verified locally with CodeQL 2.27.1** before pushing:
- the same 5 alerts reproduce on `efe173c9`, and 0 remain on the fix
commit;
- `python-security-extended` over every Python file this PR changes
finds the same 14 pre-existing alerts on `main`'s versions and on the
branch's versions, so the PR adds none.
- **`uv.lock`:** rebuilt as `main`'s lockfile plus `truststore` only (11
lines). Re-locking with current uv also rewrites markers across the
file, because `main`'s lockfile predates the current `pyproject.toml`.
That is left for a separate lock refresh.

**Earlier review:** a high-effort code review of this branch produced 10
findings plus 3 minor notes. Commit `fix: address review findings…`
resolves 9 of the 10 and two of the notes; the third note (lockfile
markers) checked out as a false alarm. The remaining finding is the last
follow-up below.

**Follow-ups (not in this PR):**
- SSE keep-alives and HTTP/1.1 fallback when a gateway buffers a stream.
- A `wrap` pre-flight that steps aside when the upstream is unreachable.
- The Rust core's fastembed/hf-hub embedding download, which hard-codes
WebPki roots (needs an hf-hub patch; BM25 is the fallback, and
`HF_ENDPOINT` is documented).
- A CI job with a fake root installed in the OS store on macOS and
Windows.
- Map certificate failures once, in a shared layer, so batch, Gemini and
the other handlers get the TLS hint too.

**Known unrelated failures:**
- 4 `tests/test_codex_live.py` tests fail when run after other files in
this set, both on `main` and on this branch. They pass when run on their
own.
- After merging the latest `main` (thanks for the refresh),
`tests/test_cli/test_wrap_helpers.py::test_start_proxy_strips_ambient_worker_configuration`
and `::test_start_proxy_timeout_kills_failed_new_process` fail locally
with `TypeError: <lambda>() takes 0 positional arguments but 1 was
given`. They fail the same way on a clean checkout of current `main`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-09-28 18:06:09 -07:00
Waqas Ahmed 5baa439ebf fix(proxy): restrict urlopen SSL context ALPN to http/1.1 (#3567)
## Description

`build_urlopen_context()` in `headroom/proxy/ssl_context.py` is only
used by `copilot_auth.py`'s `_urlopen()` helper, which wraps stdlib
`urllib.request`/`http.client` for the GitHub Copilot device-code auth
flow. It was reusing whatever `build_httpx_verify()` returns, which
(when a custom CA bundle or `HEADROOM_TLS_STRICT=0` is configured)
advertises ALPN `["h2", "http/1.1"]` -- correct for the proxy's httpx
upstream client, but `http.client` has no HTTP/2 support at all.

Under a TLS-inspecting corporate MITM that honors ALPN (GlobalProtect,
per the issue), the far end negotiates HTTP/2 with the client.
`http.client` then tries to read the response as HTTP/1.1 and misreads
the HTTP/2 connection-preface/frame bytes as a garbled status line,
surfacing as `http.client.BadStatusLine: ]MAGIC: Bad MAGIC` on `POST
https://github.com/login/device/code`.

Closes #3344

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `build_urlopen_context()` now restricts the returned context's ALPN to
`["http/1.1"]` only, since its sole caller (`urllib.request`) can't
speak anything else.
- Left `build_httpx_verify()`, `find_ca_bundle()`, and the shared
context builders untouched -- the proxy's httpx client still offers `h2`
as before.
- Added regression tests in `tests/test_ssl_context.py` covering the
custom-CA-bundle case, the `HEADROOM_TLS_STRICT=0` case, and the
no-bundle/default case (unaffected, still returns `None`).

## Testing

- [x] Unit tests pass (`pytest tests/test_ssl_context.py`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --no-project --with pytest --with urllib3 pytest tests/test_ssl_context.py -v
...
tests/test_ssl_context.py::TestBuildUrlopenContext::test_custom_ca_context_only_offers_http_1_1 PASSED
tests/test_ssl_context.py::TestBuildUrlopenContext::test_default_returns_none PASSED
tests/test_ssl_context.py::TestBuildUrlopenContext::test_toggle_off_context_only_offers_http_1_1 PASSED
...
======================== 34 passed, 1 warning in 3.50s =========================
```

I confirmed the new tests are real regression tests, not tautologies:
reverting the one-line fix (removing the
`set_alpn_protocols(["http/1.1"])` call) makes
`test_custom_ca_context_only_offers_http_1_1` and
`test_toggle_off_context_only_offers_http_1_1` fail, because the context
would still carry `["h2", "http/1.1"]`.

## Real Behavior Proof

- Environment: this repo's `main`, Python 3.13, pytest run via an
isolated `uv` environment (the editable install's Rust extension build
wasn't needed for this pure-Python module/test).
- Exact command / steps: shown above.
- Observed result: all 34 tests in `tests/test_ssl_context.py` pass,
including the 3 new ones for `build_urlopen_context()`.
- Not tested: an actual TLS-inspecting MITM / live GitHub Copilot
device-auth flow -- I don't have that network setup available, so this
is verified at the ALPN-configuration level (the confirmed root cause)
rather than an end-to-end repro of the corporate proxy environment.

## Runtime Rollout Safety

- Rollout-managed feature(s): none -- this only affects TLS context
construction for the `urllib.request`-based Copilot device-auth path,
gated the same way it already was (custom CA bundle env var, or
`HEADROOM_TLS_STRICT=0`).
- Minimum rollout channel: n/a
- Stable/default behavior changed: no -- the no-bundle/default case
(`build_httpx_verify()` returns `True`) is unchanged, still returns
`None`. Only the ALPN list on an already-returned custom context is
narrowed.
- Kill switch / disable path: n/a -- restricting ALPN to `http/1.1` can
only fix broken connections for this urlopen-only caller; it can't
regress anything since `http.client` never spoke `h2` to begin with.
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (n/a --
internal helper, no public docs reference its ALPN behavior)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
2026-09-24 11:32:27 -05:00
9fde127534 fix(proxy): relocate stray system-role messages to the top-level system param (#765) (#1357)
## Description

On requests large enough to trigger compression, the proxy emitted an
upstream Anthropic request whose `messages[0]` had `role: "system"`.
Anthropic's Messages API rejects any `system` role inside `messages[]`:

```
400 invalid_request_error: "messages.0: use the top-level 'system' parameter for the initial system prompt"
```

The original request correctly carries its system prompt in the
top-level `system` parameter; a compression/transform/pipeline step
relocates the harness system block into `messages[0]`, so the request
fails outright (intermittent only because it requires a context large
enough to compress).

This adds a wire-contract guard in the Anthropic forwarder: as the
**last** step before sending upstream (after every transform, memory
injection, tool sort, and pipeline extension, covering both the Bedrock
and direct paths), any stray `role="system"` message is relocated out of
`messages[]` and merged back into the top-level `system` parameter.
Content order is preserved (existing system first, relocated content
after) and block-level `cache_control` survives. The guard is a no-op on
the common path (no system-role entry → inputs pass through unchanged).

Closes #765

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/helpers.py`: new pure helper
`relocate_system_messages_to_top_level(messages, system) ->
(clean_messages, new_system, changed)` plus `_system_message_to_blocks`.
Handles `system` being `None`/`str`/`list`, never drops content,
preserves order and content blocks.
- `headroom/proxy/handlers/anthropic.py`: invoke the guard just before
the byte-faithful forward block; on relocation, update
`body["messages"]`/`body["system"]`, mark the body mutated
(`system_role_relocated`) so the byte-faithful forwarder re-serializes,
and log a warning.
- `tests/test_proxy_handler_helpers.py`: 3 unit tests (relocate stray
system into top-level, append-to-existing-system order, no-op without a
system entry).
- `CHANGELOG.md`: Bug Fixes entry.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --extra dev python -m pytest tests/test_proxy_handler_helpers.py -q
29 passed in 4.95s

# Regression on the forward path (byte-faithful forwarding, system-prompt immutability, cache stability):
$ uv run --extra dev python -m pytest tests/test_proxy_handler_helpers.py tests/test_proxy_byte_faithful_forwarding.py tests/test_proxy_system_prompt_immutable.py tests/test_proxy_anthropic_cache_stability.py -q
90 passed, 15 warnings in 29.72s

$ uv run ruff check headroom/proxy/helpers.py headroom/proxy/handlers/anthropic.py tests/test_proxy_handler_helpers.py
All checks passed!

$ uv run ruff format --check ...   # 3 files already formatted
$ uv run mypy headroom/proxy/helpers.py headroom/proxy/handlers/anthropic.py
Success: no issues found in 2 source files
```

## Test verification (RED → GREEN)

The new tests exercise the guard directly and import the new helper at
module top, so reverting the production fix makes them fail at
collection.

**RED — production fix reverted (helper removed):**
```text
ImportError while importing test module 'tests/test_proxy_handler_helpers.py'.
E   ImportError: cannot import name 'relocate_system_messages_to_top_level' from 'headroom.proxy.helpers'
=========================== 1 error in 0.41s ===============================
```

**GREEN — production fix applied:**
```text
tests/test_proxy_handler_helpers.py ...                                  [100%]
======================= 3 passed, 26 deselected in 1.50s =======================
```

## Real Behavior Proof

- Environment: Python 3.13, `uv run` in this repo, branch
`fix/issue-765`.
- Exact command / steps: ran the guard on a body in the exact #765
failure shape — `system: None` and a `role="system"` harness block at
`messages[0]`:
- Observed result:
  ```text
  BEFORE: messages[0].role = system (Anthropic 400 trigger)
  changed       = True
  AFTER roles   = ['user', 'assistant']
system param = [{"type": "text", "text": "You are Claude Code.
<system-reminder>...</system-reminder>"}]
OK: no role=system in messages[]; system content preserved in top-level
param
  ```
The illegal `role="system"` entry is removed from `messages[]` and its
content lands in the top-level `system` parameter — exactly the body
Anthropic accepts.
- Not tested: a full live 250k+-token Claude Code session against the
real Anthropic API (needs a large live context + API key); the fix is
validated at the request-shaping boundary the 400 is raised on.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

The guard intentionally fires at the forwarder boundary rather than in
any single transform: the issue's captures show the relocation can
originate from the compression path, and pipeline extensions / hooks can
also mutate `messages` late. Enforcing Anthropic's wire contract once,
at the point the body is serialized upstream, fixes the 400 regardless
of which step introduced the stray entry and matches the architecture
invariant "never produce a `system`-role entry within `messages[]`".

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-08-13 11:52:09 -05:00
Lakshya Sharma 52068dd650 fix(tls): add HEADROOM_TLS_STRICT=0 toggle for corporate SSL inspection (#1308) (#1341)
## Description

Behind a corporate TLS-inspection proxy (Zscaler, Netskope) on Python
3.13+, Headroom can't reach the network even with the corporate root
correctly installed and trusted. Every path fails with:

```
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed:
Basic Constraints of CA cert not marked critical
```

This isn't a missing-CA problem — the cert is found and trusted. Python
3.13 + OpenSSL 3.x enable `VERIFY_X509_STRICT` by default, which
enforces RFC 5280 §4.2.1.9 (a CA cert's `basicConstraints` MUST be
marked critical). Inspection roots set `CA:TRUE` without the critical
bit, so the chain is rejected. Adding the CA to a bundle does nothing —
it's the strict check that fails, and the existing README section only
covers `unable to get local issuer certificate`.

There are two independent sources of the strict flag (both reported in
the issue): Python's own `ssl.create_default_context()` (hits the httpx
upstream client), and urllib3 ≥ 2.5's `create_urllib3_context()` (hits
the `huggingface_hub` model-download path).

Closes #1308

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `ssl_context.py`: added `HEADROOM_TLS_STRICT`. `tls_strict_disabled()`
reads the toggle (off-values `0/false/no/off`, default strict).
`build_httpx_verify()` resolves the httpx `verify=` value: a configured
CA bundle wins; otherwise, when the toggle is off, a default-trust-store
context with **only** `VERIFY_X509_STRICT` cleared (so a corporate root
that lives in the OS store but trips strict mode still validates);
otherwise `True` (httpx default). `apply_global_tls_relaxation()`
monkeypatches urllib3's `create_urllib3_context` to drop the strict flag
— idempotent, guarded, no-op if urllib3 is absent or the toggle is on.
- `server.py`: the proxy's httpx upstream client now uses
`build_httpx_verify()` instead of `find_ca_bundle()`-or-`True`.
- `cli/proxy.py`: calls `apply_global_tls_relaxation()` at module
import, before `huggingface_hub`/`requests` import and cache their
context.
- README: a distinct SSL-inspection subsection for the `Basic
Constraints ... not marked critical` failure, separate from `unable to
get local issuer certificate`. Documents that the Rust core's ONNX
download (`cdn.pyke.io`) uses a separate stack (rustls/OS trust store)
unaffected by the toggle — corporate root must be in the Windows
**machine** store, or pre-provision via `ORT_STRATEGY=system`.

Chain validation, signature, expiry, and hostname checks all stay on —
`HEADROOM_TLS_STRICT=0` is strictly narrower than `verify=False`.
Default is strict, matching Python's own default.

## Testing

- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_ssl_context.py -q
31 passed
# 19 existing + 12 new (TestTlsStrictDisabled, TestBuildHttpxVerify, TestApplyGlobalTlsRelaxation).
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.10.11 (OpenSSL build that exposes
`VERIFY_X509_STRICT`).
- Exact command / steps: exercised the module directly — set/unset
`HEADROOM_TLS_STRICT`, inspected the resolved httpx `verify` value and
the urllib3 context's `verify_flags`.
- Observed result: default → `verify=True` (strict preserved);
`HEADROOM_TLS_STRICT=0` → httpx gets an `SSLContext` with
`VERIFY_X509_STRICT` cleared but `verify_mode == CERT_REQUIRED` and the
full default trust store (cert_store x509_ca > 1);
`apply_global_tls_relaxation()` patches
`urllib3.util.ssl_.create_urllib3_context` so new contexts have the
strict flag cleared, and is idempotent. A configured `SSL_CERT_FILE`
still wins over the toggle.
- Not tested: an actual handshake through a live Zscaler/Netskope MITM
on Python 3.13 — I don't have that environment. The fix targets exactly
the flag the issue identifies (`VERIFY_X509_STRICT`) on both reported
context builders; I verified the flag manipulation and resolution logic
directly rather than simulating the proxy.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

- The toggle is opt-in and defaults to strict, so behavior is unchanged
unless a user explicitly sets `HEADROOM_TLS_STRICT=0`. It clears only
the strict flag, never disables verification.
- The httpx path uses an explicit context (clean, testable); the urllib3
path needs a monkeypatch because `huggingface_hub` → `requests` builds
its context internally and never sees ours.
- CHANGELOG.md isn't touched — release-please generates it from the
`fix(tls):` commit subject.
- I scoped this to the two Python TLS stacks the issue calls out and
documented (rather than tried to patch) the separate Rust/ONNX path,
since that one resolves through the OS trust store and isn't something
this Python toggle can reach.
2026-06-24 09:51:30 -05:00
Mark PhelpsandJD Davis 561ba17ec2 fix(proxy): build SSL contexts for custom CA bundles (#1134)
## Description

Build explicit `ssl.SSLContext` objects for custom CA bundles configured
through `SSL_CERT_FILE` and `REQUESTS_CA_BUNDLE`. Python 3.13 / newer
OpenSSL can reject some enterprise/private PKI roots that platform TLS
stacks accept, for example roots without a `keyUsage` extension. The new
custom-CA contexts keep certificate verification enabled while clearing
only `ssl.VERIFY_X509_STRICT`.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Build replacement `SSLContext` objects for `SSL_CERT_FILE` and
`REQUESTS_CA_BUNDLE` instead of passing raw CA bundle paths to httpx.
- Clear only `ssl.VERIFY_X509_STRICT` for operator-provided custom CA
contexts; certificate verification, hostname verification, expiry
checks, and chain validation stay enabled.
- Preserve `NODE_EXTRA_CA_CERTS` additive behavior by loading the extra
CA bundle on top of the default/system trust store.
- Update existing SSL context tests for replacement CA contexts, env-var
priority, missing-path fallthrough, and strict-mode relaxation.
- Add an Unreleased changelog entry for the proxy bug fix.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ PYTHONPATH=. .venv/bin/python -m pytest tests/test_ssl_context.py -q
collected 12 items

tests/test_ssl_context.py ............                                   [100%]

12 passed, 1 warning in 0.11s
```

```text
$ uv tool run ruff check headroom/proxy/ssl_context.py tests/test_ssl_context.py CHANGELOG.md
All checks passed!

$ uv tool run ruff format --check headroom/proxy/ssl_context.py tests/test_ssl_context.py
2 files already formatted
```

## Real Behavior Proof

- Environment: macOS, Python 3.13.2, Headroom checkout on this branch,
`SSL_CERT_FILE` / `REQUESTS_CA_BUNDLE` / `NODE_EXTRA_CA_CERTS` pointed
at an enterprise/private PKI CA bundle. Upstream HTTPS endpoint name is
redacted.
- Exact command / steps: Ran a Python smoke script that imports
`headroom.proxy.ssl_context.find_ca_bundle()`, passes the returned
verifier into `httpx.AsyncClient(verify=...)`, and performs a GET
against the enterprise HTTPS endpoint that previously failed with
OpenSSL strict verification.
- Observed result: The request used an `SSLContext`, strict X.509
verification was disabled for that custom CA context, and the request
reached the upstream HTTP response:

```text
verify_type SSLContext
strict_enabled False
status 302
```

- Not tested: Full `uv run pytest`, `uv sync --extra dev`, and `mypy
headroom` in this local environment. The editable build currently fails
before tests run because Cargo's `ort-sys` build script cannot download
ORT prebuilt binaries due an unrelated local certificate verification
error against the ORT CDN. No dependency or lockfile changes are
included in this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A.

## Additional Notes

No dependencies or lockfiles changed. Documentation is unchanged because
this is a bug fix to existing custom CA environment-variable behavior
rather than a new user-facing configuration surface.

Signed-off-by: Mark Phelps <209477+markphelps@users.noreply.github.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-06-22 18:48:19 -05:00
Shengbo_Wang c98728363a fix(proxy): treat NODE_EXTRA_CA_CERTS as additive, not replacement (#998) (#1031)
## Description

`find_ca_bundle()` returns the `NODE_EXTRA_CA_CERTS` path as a bare
string passed to httpx's `verify=` parameter, which makes it the sole
trust store. When that bundle contains only a private/internal root (the
common corporate setup), all public upstreams (`api.anthropic.com`,
`api.openai.com`) fail TLS verification with
`CERTIFICATE_VERIFY_FAILED`, returning 502.

This is the inverse of #741: that fix added corporate CA support, but
this regression means public CAs are no longer trusted when the extra
bundle is not a full superset of the public roots.

The fix builds an `ssl.SSLContext` via `create_default_context()` (keeps
system/default roots) then `load_verify_locations()` (adds the extra
cert), matching Node.js additive semantics. `SSL_CERT_FILE` and
`REQUESTS_CA_BUNDLE` keep their existing replacement semantics.

Closes #998

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Split `NODE_EXTRA_CA_CERTS` handling out of the replacement-semantics
loop in `find_ca_bundle()`
- When `NODE_EXTRA_CA_CERTS` is the source, return an `ssl.SSLContext`
with default roots plus the extra cert, instead of a bare path string
- Set ALPN protocols (`h2`, `http/1.1`) on the context to preserve
HTTP/2 negotiation
- Updated `test_node_extra_ca_certs_returns_path` to assert
`ssl.SSLContext` return type
- Added `test_node_extra_ca_certs_is_additive` verifying the context
contains more than just the extra cert (default roots preserved)

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] New tests added for new functionality

### Test Output

```text
tests/test_ssl_context.py::TestFindCaBundleNoEnvVars::test_returns_none_when_no_env_var_set PASSED
tests/test_ssl_context.py::TestFindCaBundleWithValidPem::test_ssl_cert_file_returns_path PASSED
tests/test_ssl_context.py::TestFindCaBundleWithValidPem::test_requests_ca_bundle_returns_path PASSED
tests/test_ssl_context.py::TestFindCaBundleWithValidPem::test_node_extra_ca_certs_returns_ssl_context PASSED
tests/test_ssl_context.py::TestFindCaBundleWithValidPem::test_node_extra_ca_certs_is_additive PASSED
tests/test_ssl_context.py::TestFindCaBundlePriority::test_ssl_cert_file_beats_requests_ca_bundle PASSED
tests/test_ssl_context.py::TestFindCaBundlePriority::test_ssl_cert_file_beats_node_extra_ca_certs PASSED
tests/test_ssl_context.py::TestFindCaBundlePriority::test_requests_ca_bundle_beats_node_extra_ca_certs PASSED
tests/test_ssl_context.py::TestFindCaBundleNonexistentPaths::test_nonexistent_path_is_skipped PASSED
tests/test_ssl_context.py::TestFindCaBundleNonexistentPaths::test_all_nonexistent_returns_none PASSED
tests/test_ssl_context.py::TestFindCaBundleNonexistentPaths::test_first_nonexistent_falls_through_to_valid PASSED

11 passed in 0.91s
```

## Real Behavior Proof

- Environment: Windows 11 Home 10.0.26200, Python 3.10.18
- Exact command / steps: Ran `python -m pytest tests/test_ssl_context.py
-v` after applying the fix. The new
`test_node_extra_ca_certs_is_additive` test verifies
`ctx.cert_store_stats()['x509_ca'] > 1`, confirming the default trust
store roots are preserved alongside the extra cert. If replacement
semantics were used, only the single test CA would be loaded.
- Observed result: All 11 SSL context tests pass. The additive context
reports 143 x509_ca certs (system defaults + test cert), confirming
default roots are preserved.
- Not tested: No live TLS handshake to a public upstream with a
private-only `NODE_EXTRA_CA_CERTS` bundle, but the `cert_store_stats`
assertion proves the default roots are loaded into the context.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes

## Additional Notes

The fix aligns with the design proposed in #741 and #745's own
description (which said it "builds an SSLContext") but which the merged
implementation did not follow. The `server.py` consumer does not need
changes because httpx accepts `ssl.SSLContext` for `verify=`. ALPN
protocols are set on the context to maintain HTTP/2 negotiation parity
with httpx's internally-built contexts.
2026-06-17 11:41:49 -05:00
yehsuf e50fbb3e0d fix(ssl): upstream httpx client inherits SSL_CERT_FILE, REQUESTS_CA_BUNDLE, NODE_EXTRA_CA_CERTS (#745)
Closes #741

## What

Headroom is commonly deployed alongside Claude Code, which sets
`NODE_EXTRA_CA_CERTS` to a custom CA bundle for corporate or internal
CAs. Node.js inherits this automatically; Python's `httpx` does not.
Every upstream request silently failed with `SSL:
CERTIFICATE_VERIFY_FAILED`, causing 502s and retry loops in the client.

## Changes

- New `headroom/proxy/ssl_context.py` with `build_ssl_context()` helper
that checks `SSL_CERT_FILE` → `REQUESTS_CA_BUNDLE` →
`NODE_EXTRA_CA_CERTS` (first match wins) and builds an `ssl.SSLContext`
with the custom CA bundle loaded
- `HeadroomProxy.start()` calls `build_ssl_context()` and passes the
result as `verify=` to `httpx.AsyncClient`; falls back to `verify=True`
(default httpx behaviour) when no env var is set
- Logs which env var and path was used at `INFO` level; warns on
set-but-missing paths
- 10 unit tests covering: no env var → `None`, each var returns
`SSLContext`, priority order, nonexistent paths skipped

## Priority order

1. `SSL_CERT_FILE` — standard POSIX/Python ssl override  
2. `REQUESTS_CA_BUNDLE` — standard requests/httpx convention  
3. `NODE_EXTRA_CA_CERTS` — Node.js / Claude Code convention
2026-06-08 21:10:47 -08:00