fix(cli): pass configured SSL context to stdlib urlopen calls (#4011)

## Description

Fixes #3976. On `uv tool` installs and other environments where certifi
is not
wired into the stdlib OpenSSL trust path, `headroom update` and
bundled-binary
downloads can fail behind corporate TLS inspection. These urllib calls
now use
the configured Headroom trust context, matching the proxy's TLS policy.

The lazy helper import still falls back to urllib's default when the
optional
trust module cannot be imported. Errors while constructing an explicitly
configured context now propagate instead of silently replacing the
requested
trust roots with stdlib defaults. The update checker retains its outer
error-handling behavior and returns no update on failure.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring

## Changes Made

- Pass `build_urlopen_context()` to the PyPI version check and
release-binary
  download paths.
- Catch only `ImportError` around the lazy helper import; let
trust-context
  construction errors reach the caller.
- Add valid and invalid replacement-bundle tests for both helpers and
verify
the update check still handles the error through its existing outer
handler.
- Update mocked `urlopen` functions to accept the `context` keyword used
by the
  production call path.

## Testing

- [x] Targeted unit tests pass
- [ ] Full test suite run
- [x] Ruff lint and format checks pass for changed files
- [x] New tests added for the requested behavior
- [ ] Type checking run
- [ ] Manual live-network testing performed

### Test Output

```text
python -m pytest --disable-plugin-autoload -p pytest_asyncio.plugin tests/test_ssl_context.py tests/test_binaries.py tests/test_update_check.py
95 passed in 5.10s

python -m ruff check headroom/binaries.py headroom/update_check.py tests/test_ssl_context.py tests/test_binaries.py
All checks passed!

python -m ruff format --check headroom/binaries.py headroom/update_check.py tests/test_ssl_context.py tests/test_binaries.py
4 files already formatted

git diff --check
passed
```

## Real Behavior Proof

**Original reproduction:** On macOS behind a corporate TLS-inspecting
network,
`headroom update` failed with `CERTIFICATE_VERIFY_FAILED` when the
inspection
root was present in the OS trust store but absent from Python's stdlib
OpenSSL
bundle. The same machine's proxy reached its upstream through the
configured
root. The original fix routed the direct urllib calls through
`build_urlopen_context()`.

**Review regression checks:** On Windows with Python 3.13.13, tests
using a
valid `REQUESTS_CA_BUNDLE` confirm both helpers return a verifying
`SSLContext`. With an existing file containing invalid PEM, both helpers
now
raise `ssl.SSLError`; the update-check caller still returns `None`
through its
outer error handler. All 95 tests in the affected TLS, binary-download,
and
update-check suites pass.

**Not tested:** A live corporate TLS-inspection connection, Windows `uv
tool`
end-to-end behavior behind such a network, the full repository test
suite, and
the separate eval dataset download path.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Not applicable.
- Stable/default behavior changed: An invalid explicitly configured
trust
bundle now fails visibly rather than silently falling back to stdlib
roots;
  unconfigured default behavior is unchanged.
- Kill switch / disable path: No new switch; existing TLS environment
settings
  remain available.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this PR's changes.

## Review Readiness

- [x] I have performed a self-review.
- [x] This PR is ready for human review.

## Checklist

- [x] My code follows the project's style guidelines.
- [x] I have performed a self-review of my code.
- [ ] I have added comments where the code needs additional explanation.
- [ ] I have made corresponding documentation changes (none needed for
this
  internal error-handling change).
- [x] The changed files pass Ruff checks and the focused tests add no
warnings.
- [x] I have added tests that prove the requested behavior.
- [ ] The full repository unit-test suite was run.
- [x] I did not edit `CHANGELOG.md`.

---------

Co-authored-by: dajiaohuang <dajiaohuang@users.noreply.github.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Wu Shuwen
2026-10-07 20:51:49 -05:00
committed by GitHub
co-authored by dajiaohuang JD Davis
parent a21879bd54
commit 22d0905a73
6 changed files with 221 additions and 12 deletions
+32 -2
View File
@@ -48,7 +48,10 @@ import urllib.request
import zipfile
from dataclasses import dataclass
from pathlib import Path
from typing import Any
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
import ssl
from headroom._subprocess import run
from headroom.offline import guard_egress
@@ -264,6 +267,25 @@ def _mirror_url(url: str) -> str:
# ---------- Download + verify --------------------------------------------- #
def _ssl_context() -> ssl.SSLContext | None:
"""Return Headroom's configured SSL context for urllib.request.urlopen, or
``None`` (urlopen's default) when the trust helper cannot be loaded.
This ensures binary downloads from GitHub releases (and any configured
mirror) trust the same roots as the proxy — including corporate
TLS-inspection roots installed in the OS store and bundles referenced via
``HEADROOM_CA_BUNDLE`` / ``SSL_CERT_FILE``. Without this, ``uv tool``
installs (where certifi may not yet be wired to the stdlib path) raise
``CERTIFICATE_VERIFY_FAILED`` even though certifi is already a dependency.
"""
try:
from headroom.proxy.ssl_context import build_urlopen_context
except ImportError:
return None
return build_urlopen_context()
def _download(url: str, dest: Path, *, progress: bool = True) -> None:
if os.environ.get("HEADROOM_BINARIES_OFFLINE"):
raise OfflineError(f"offline mode (HEADROOM_BINARIES_OFFLINE=1) but fetch required: {url}")
@@ -281,10 +303,18 @@ def _download(url: str, dest: Path, *, progress: bool = True) -> None:
if not final_url.startswith("https://"):
raise BinaryFetchError(f"refusing non-https download URL: {final_url!r}")
req = urllib.request.Request(final_url, headers={"User-Agent": "headroom-binaries/1"})
try:
ctx = _ssl_context()
except Exception as e:
raise BinaryFetchError(f"failed to configure TLS trust for binary download: {e}") from e
attempts = 3
for attempt in range(1, attempts + 1):
try:
with urllib.request.urlopen(req, timeout=60) as resp: # noqa: S310 (https)
if ctx is not None:
resp = urllib.request.urlopen(req, timeout=60, context=ctx) # noqa: S310 (https)
else:
resp = urllib.request.urlopen(req, timeout=60) # noqa: S310 (https)
with resp:
total = int(resp.headers.get("Content-Length") or 0)
_stream_to(resp, dest, total, label=dest.name, show_progress=progress)
return
+34 -2
View File
@@ -26,7 +26,10 @@ import threading
import time
import urllib.request
from pathlib import Path
from typing import Any
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
import ssl
logger = logging.getLogger(__name__)
@@ -187,17 +190,46 @@ def _select_latest(data: dict[str, Any], *, allow_pre: bool) -> str | None:
return None
def _urlopen_ssl_context() -> ssl.SSLContext | None:
"""Return an SSL context for urllib.request.urlopen that honors Headroom's
trust policy (OS store + certifi, HEADROOM_CA_BUNDLE, etc).
Imported lazily so this module's dependency footprint stays at stdlib +
packaging when the trust helpers aren't yet importable (e.g. during early
startup or in minimal test environments). Falls back to ``None`` (urlopen's
default) when the helper cannot be loaded.
"""
try:
from headroom.proxy.ssl_context import build_urlopen_context
except ImportError:
return None
return build_urlopen_context()
def fetch_latest_version(*, allow_pre: bool = False, timeout: float = 4.0) -> str | None:
"""Query the PyPI JSON API for the latest release. Returns None on any error.
Uses ``urllib`` (stdlib) so the base CLI install needs no HTTP dependency.
Passes Headroom's configured SSL context so corporate TLS-inspection roots
(Zscaler, Netskope, ...) installed in the OS trust store — or referenced via
``HEADROOM_CA_BUNDLE`` / ``SSL_CERT_FILE`` — are trusted the same way the
proxy trusts them.
"""
try:
from headroom.offline import guard_egress
guard_egress("Headroom update check", _PYPI_JSON_URL)
req = urllib.request.Request(
_PYPI_JSON_URL,
headers={"Accept": "application/json", "User-Agent": "headroom-update-check"},
)
with urllib.request.urlopen(req, timeout=timeout) as resp: # noqa: S310 — fixed https URL
ctx = _urlopen_ssl_context()
if ctx is not None:
resp = urllib.request.urlopen(req, timeout=timeout, context=ctx) # noqa: S310 — fixed https URL
else:
resp = urllib.request.urlopen(req, timeout=timeout) # noqa: S310 — fixed https URL
with resp:
data = json.loads(resp.read().decode("utf-8"))
return _select_latest(data, allow_pre=allow_pre)
except Exception:
+18 -3
View File
@@ -7,6 +7,7 @@ from __future__ import annotations
import hashlib
import io
import ssl
import sys
import tarfile
import zipfile
@@ -101,7 +102,7 @@ def fake_urlopen(monkeypatch):
"""Install a fake urllib.request.urlopen that serves registered URLs."""
served: dict[str, bytes] = {}
def fake(req, timeout=None): # noqa: ARG001
def fake(req, timeout=None, context=None): # noqa: ARG001
url = req.full_url if hasattr(req, "full_url") else req
if url not in served:
raise AssertionError(f"unexpected fetch for {url}")
@@ -218,6 +219,20 @@ def test_mirror_substitution(monkeypatch):
assert binaries._mirror_url("https://example.com/x") == "https://example.com/x"
def test_download_wraps_tls_context_error_and_preserves_cause(monkeypatch, tmp_path):
error = ssl.SSLError("invalid configured certificate bundle")
monkeypatch.setattr(
binaries,
"_ssl_context",
lambda: (_ for _ in ()).throw(error),
)
with pytest.raises(binaries.BinaryFetchError, match="failed to configure TLS trust") as exc:
binaries._download("https://example.invalid/tool.tar.gz", tmp_path / "tool.tar.gz")
assert exc.value.__cause__ is error
def test_mirror_url_with_query_params_strips_them_from_download_filename(
monkeypatch, fake_urlopen, allow_unverified
):
@@ -284,7 +299,7 @@ def test_download_retries_transient_network_failure(monkeypatch, tmp_path):
attempts = 0
sleeps: list[float] = []
def flaky_urlopen(req, timeout=None): # noqa: ARG001
def flaky_urlopen(req, timeout=None, context=None): # noqa: ARG001
nonlocal attempts
attempts += 1
if attempts < 3:
@@ -465,7 +480,7 @@ def test_ensure_tools_partial_failure_proxy_still_starts(monkeypatch, allow_unve
scc_asset = binaries._registry()["tools"]["scc"]["assets"]["darwin-aarch64"]
scc_tar = _make_tar_gz({"scc": b"ok"})
def selective_urlopen(req, timeout=None): # noqa: ARG001
def selective_urlopen(req, timeout=None, context=None): # noqa: ARG001
url = req.full_url if hasattr(req, "full_url") else req
if url == scc_asset["url"]:
return _FakeResponse(scc_tar)
+68
View File
@@ -96,6 +96,74 @@ def test_sg_command_reports_resolution_errors(
assert expected in result.output
@pytest.mark.parametrize(
("args", "expected", "exit_code"),
[
(("diff", "--version"), "error: failed to configure TLS trust", 2),
(("tools", "install", "--tool", "difft"), "difft: failed to configure TLS trust", 1),
],
)
def test_invalid_configured_ca_bundle_is_reported_without_fetching(
monkeypatch: pytest.MonkeyPatch,
runner: CliRunner,
tmp_path: Path,
args: tuple[str, ...],
expected: str,
exit_code: int,
) -> None:
invalid_bundle = tmp_path / "invalid.pem"
invalid_bundle.write_text("not a PEM certificate", encoding="ascii")
monkeypatch.setenv("REQUESTS_CA_BUNDLE", str(invalid_bundle))
for variable in (
"SSL_CERT_FILE",
"NODE_EXTRA_CA_CERTS",
"HEADROOM_CA_BUNDLE",
"HEADROOM_TLS_STRICT",
):
monkeypatch.delenv(variable, raising=False)
monkeypatch.setenv("HEADROOM_CERT_STORE", "bundled")
monkeypatch.setattr(cli_tools.binaries, "cache_dir", lambda: tmp_path / "cache")
monkeypatch.setattr(cli_tools.binaries, "_path_lookup", lambda _name: None)
monkeypatch.setattr(
cli_tools.binaries,
"detect_platform",
lambda: binaries.PlatformKey("linux", "x86_64", "gnu"),
)
monkeypatch.setattr(
cli_tools.binaries,
"_registry",
lambda: {
"tools": {
"difft": {
"version": "test",
"source": "github",
"assets": {
"linux-x86_64-gnu": {
"url": "https://example.invalid/difft.tar.gz",
"member": "difft",
"sha256": "0" * 64,
}
},
}
}
},
)
urlopen_calls: list[object] = []
def unexpected_urlopen(*call_args, **call_kwargs):
urlopen_calls.append((call_args, call_kwargs))
raise AssertionError("invalid configured trust must fail before urlopen")
monkeypatch.setattr(cli_tools.binaries.urllib.request, "urlopen", unexpected_urlopen)
result = runner.invoke(main, list(args))
assert result.exit_code == exit_code
assert expected in result.output
assert "NO_CERTIFICATE_OR_CRL_FOUND" in result.output
assert urlopen_calls == []
def test_tools_list_renders_registry(monkeypatch: pytest.MonkeyPatch, runner: CliRunner) -> None:
install_fake_rich(monkeypatch)
monkeypatch.setattr(
+21 -5
View File
@@ -57,6 +57,27 @@ PACKAGE_ROOT = REPO_ROOT / "headroom"
# ──────────────────────────── socket booby trap ────────────────────────────
class TestUpdateCheckOffline:
def test_explicit_update_check_never_constructs_request(self, monkeypatch, offline, no_sockets):
from click.testing import CliRunner
from headroom import update_check
from headroom.cli import main
attempts = []
def refuse_request(*args, **kwargs):
attempts.append((args, kwargs))
raise SocketOpened("update check constructed a request while offline")
monkeypatch.setenv("HEADROOM_UPDATE_CHECK", "on")
monkeypatch.setattr(update_check.urllib.request, "Request", refuse_request)
result = CliRunner().invoke(main, ["update", "--check"])
assert attempts == []
assert result.exit_code == 1
class SocketOpened(AssertionError):
"""Raised from the patched socket entry points.
@@ -2271,11 +2292,6 @@ _EGRESS_ALLOWLIST: dict[str, tuple[int, str]] = {
"deployment that points Headroom at an on-prem model endpoint, which "
"is the main reason such a deployment exists.",
),
"update_check.py": (
1,
"gated: is_update_check_enabled() returns False when is_offline(), so "
"the request is never built. See headroom/update_check.py.",
),
"telemetry/session.py": (
1,
"gated: the beacon upload is reached only via is_telemetry_enabled(), "
+48
View File
@@ -17,6 +17,7 @@ import ssl
import pytest
from headroom import binaries, update_check
from headroom.proxy import ssl_context
from headroom.proxy.ssl_context import (
apply_global_tls_relaxation,
@@ -341,6 +342,53 @@ class TestBuildUrlopenContext:
assert created_context.alpn_protocols == ["http/1.1"]
@pytest.mark.parametrize(
"context_helper",
[
pytest.param(binaries._ssl_context, id="binary-download"),
pytest.param(update_check._urlopen_ssl_context, id="update-check"),
],
)
def test_urlopen_context_helpers_use_valid_configured_bundle(
monkeypatch, ca_pem_file, context_helper
):
_clean_env(monkeypatch)
monkeypatch.setenv("REQUESTS_CA_BUNDLE", ca_pem_file)
context = context_helper()
assert isinstance(context, ssl.SSLContext)
assert context.verify_mode == ssl.CERT_REQUIRED
@pytest.mark.parametrize(
"context_helper",
[
pytest.param(binaries._ssl_context, id="binary-download"),
pytest.param(update_check._urlopen_ssl_context, id="update-check"),
],
)
def test_urlopen_context_helpers_propagate_invalid_configured_bundle(
monkeypatch, tmp_path, context_helper
):
_clean_env(monkeypatch)
invalid_bundle = tmp_path / "invalid.pem"
invalid_bundle.write_text("not a PEM certificate", encoding="ascii")
monkeypatch.setenv("REQUESTS_CA_BUNDLE", str(invalid_bundle))
with pytest.raises(ssl.SSLError):
context_helper()
def test_update_check_returns_none_when_configured_bundle_is_invalid(monkeypatch, tmp_path):
_clean_env(monkeypatch)
invalid_bundle = tmp_path / "invalid.pem"
invalid_bundle.write_text("not a PEM certificate", encoding="ascii")
monkeypatch.setenv("REQUESTS_CA_BUNDLE", str(invalid_bundle))
assert update_check.fetch_latest_version() is None
class TestApplyGlobalTlsRelaxation:
def test_noop_when_strict_on(self, monkeypatch):
_clean_env(monkeypatch)