Files
headroom/tests/test_rust_core_smoke.py
eaa16d9f63 fix(proxy): preserve Windows service when Rust core is blocked (#2989)
## Description

When `headroom._core` cannot load, `HEADROOM_REQUIRE_RUST_CORE=false`
should keep the Python proxy available in unoptimized passthrough mode.
A module-level native import made that escape hatch unreachable before
startup could read the setting. This addresses the degraded-service
portion of #2918 and advances #1466; neither issue is closed because
Authenticode/Smart App Control reputation work remains.

## Type of Change

- [x] Bug fix (non-breaking change)
- [x] Documentation update

## Changes Made

- Resolve Rust-backed error-detection registries and patterns lazily
while retaining Rust as the sole source of truth.
- Force `optimize=false` only when the operator explicitly allows a
missing Rust core, keeping traffic available and unmodified.
- Preserve the default fail-loud startup behavior.
- Add Windows Smart App Control, WDAC, antivirus, and Code Integrity
diagnostics.
- Add cold-interpreter coverage for the import boundary and degraded
startup behavior.

## Testing

- [x] Focused tests pass
- [x] Ruff check and formatting pass
- [x] `git diff --check` passes
- [x] New regression tests added

```text
uv run --extra dev pytest -q tests/test_error_detection.py tests/test_rust_core_smoke.py
16 passed in 1.64s (independent exact-head review run)

Additional author verification: 38 focused proxy/core/error-detection/parity/MCP contract tests passed.
```

CI is rerunning on the current head after its merge from `main`; no
failure should be inferred from an unrun or pending job.

## Real Behavior Proof

- Environment: cold CPython 3.12 interpreter with `headroom._core`
marked unavailable and `HEADROOM_REQUIRE_RUST_CORE=false`.
- Exact steps: import `ProxyConfig`/`create_app`, construct the
application with `optimize=True`, and enter the FastAPI lifespan through
the focused degraded-mode tests.
- Observed result: the server module and application construct
successfully; lifespan reports `rust_core=disabled` and changes the
shared config to `optimize=False`. Rust-backed constants still fail
loudly if explicitly used without the extension.
- Not tested: a production Windows host with Smart App Control actively
blocking the shipped `_core.pyd`; the failure boundary is reproduced
deterministically in a child interpreter.

## Runtime Rollout Safety

- Rollout-managed feature: explicit degraded mode controlled by
`HEADROOM_REQUIRE_RUST_CORE=false`.
- Stable/default behavior changed: no; the default still exits loudly
when the required core cannot load.
- Kill switch / disable path: unset the opt-out or set it to true to
restore fail-loud behavior.
- Unsafe override required: no.
- Qualification impact: cold-import, degraded-startup, parity, and
error-detection tests must remain green.
- Rollback path: revert this PR; no persistent data or migration is
involved.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] Code follows project style
- [x] Documentation is updated
- [x] New tests cover the changed behavior
- [x] No manual changelog edit

## Remaining Work

#2918 remains open for Authenticode signing and Smart App Control
reputation. #1466 remains the Windows compatibility umbrella.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-01 16:10:08 -05:00

294 lines
13 KiB
Python

"""Hotfix-A0 smoke tests: deployment-stage Rust core verification.
Background — Finding #2 of HEADROOM_PROXY_LOG_FINDINGS_2026_05_03.md.
A customer's production proxy was silently running without the
`headroom._core` PyO3 extension because the Docker image never built it
into the runtime layer. Diff compression failed 54 times in one day;
optimization failed 379 times. Once the failure rate hit ~100%, every
Rust port we'd shipped was providing zero customer value.
These tests pin the contract for the fix:
1. The PyO3 module exposes a `hello()` marker function returning
``"headroom-core"`` so the deployment smoke test has something stable
to latch onto. Reusing an existing function instead of inventing a
new one means we don't drift the Rust API surface for diagnostic
purposes.
2. The proxy lifespan refuses to start when ``headroom._core`` is
unimportable, exiting with `sysexits.h` ``EX_CONFIG`` (78) so process
supervisors recognize this as a deliberate configuration failure
rather than a crash they should retry forever.
3. An explicit opt-out — ``HEADROOM_REQUIRE_RUST_CORE=false`` — keeps
the dev-time `pip install -e .` workflow alive without forcing every
contributor to run maturin.
The opt-out test also verifies that `/health` surfaces the rust core
state so operators can alert on `rust_core != "loaded"` in production.
"""
from __future__ import annotations
import pytest
# ──────────────────────────────────────────────────────────────────────────
# 1. The Rust extension's `hello()` marker is stable and importable.
# ──────────────────────────────────────────────────────────────────────────
def test_rust_core_imports() -> None:
"""`headroom._core.hello()` returns the documented sentinel.
The deployment smoke test (`headroom.proxy.server._check_rust_core`)
asserts on the exact return value so a stale or mis-linked .so is
caught — not just a complete `ImportError`. If you change the marker
string, you also need to update the lifespan check to match.
"""
from headroom._core import hello
assert hello() == "headroom-core"
# ──────────────────────────────────────────────────────────────────────────
# 2. Default behavior: missing extension blocks startup with exit 78.
# ──────────────────────────────────────────────────────────────────────────
def test_proxy_refuses_to_start_when_rust_core_missing(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""When `headroom._core` raises ImportError on import and the opt-out
env var is not set, the lifespan smoke test must call `sys.exit(78)`.
We invoke the helper directly (rather than spinning up FastAPI)
because the helper is the single source of truth for the policy and
`sys.exit` propagates through the lifespan context manager
transparently. Hitting it directly keeps the test tight and avoids
the lifespan's 30+ side effects (OTel, Langfuse, beacon, ...).
"""
from headroom.proxy import server
# Ensure the env var is absent so the default fail-loud path runs.
monkeypatch.delenv("HEADROOM_REQUIRE_RUST_CORE", raising=False)
# Force the import to raise. Patching the symbol on the module is
# not enough because `_check_rust_core` runs `from headroom._core
# import hello` inside its body — we need the import machinery to
# raise. `sys.modules` is the cleanest hook for that.
import sys
# Pre-purge any cached binding so the import statement re-runs.
sys.modules.pop("headroom._core", None)
# Sentinel that pretends to be the module but raises on attribute
# access. `from X import Y` first looks up `X` in sys.modules; if
# present it skips re-import and uses it directly. Setting
# sys.modules["headroom._core"] to None forces a real ImportError.
sys.modules["headroom._core"] = None # type: ignore[assignment]
try:
with pytest.raises(SystemExit) as exc_info:
server._check_rust_core()
assert exc_info.value.code == 78, (
f"expected exit code 78 (EX_CONFIG), got {exc_info.value.code!r}"
)
finally:
# Restore the real module so subsequent tests can import it.
sys.modules.pop("headroom._core", None)
# ──────────────────────────────────────────────────────────────────────────
# 3. Opt-out path: HEADROOM_REQUIRE_RUST_CORE=false → degraded mode + /health.
# ──────────────────────────────────────────────────────────────────────────
def test_proxy_starts_in_degraded_mode_when_opt_out_set(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""With the opt-out env var set, the lifespan must start successfully
even when `headroom._core` is missing, and `/health` must surface
`rust_core: "disabled"` so operators can detect the degraded mode.
We spin up a real FastAPI app via `TestClient` because the spec
requires the health endpoint to reflect the lifespan state — the
helper alone doesn't tell us the wiring is right.
"""
from fastapi.testclient import TestClient
from headroom.proxy.server import ProxyConfig, create_app
monkeypatch.setenv("HEADROOM_REQUIRE_RUST_CORE", "false")
# Force the import to fail at lifespan time. Same trick as above:
# sys.modules["headroom._core"] = None makes Python treat the module
# as known-unimportable, raising ImportError on the next `from ...
# import` without re-trying the loader.
import sys
sys.modules.pop("headroom._core", None)
sys.modules["headroom._core"] = None # type: ignore[assignment]
config = ProxyConfig(
optimize=False,
image_optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
ccr_inject_tool=False,
ccr_handle_responses=False,
ccr_context_tracking=False,
)
app = create_app(config)
try:
with TestClient(app) as client:
response = client.get("/health")
assert response.status_code == 200
payload = response.json()
assert payload["rust_core"] == "disabled", (
f"expected rust_core=disabled, got {payload.get('rust_core')!r}; "
f"full payload keys: {sorted(payload.keys())}"
)
# The error reason should be carried through so operators can
# see *why* the extension isn't loaded.
assert "rust_core_error" in payload
assert (
"ModuleNotFoundError" in payload["rust_core_error"]
or "ImportError" in payload["rust_core_error"]
)
finally:
sys.modules.pop("headroom._core", None)
# ──────────────────────────────────────────────────────────────────────────
# 3b. Degraded mode must actually be reachable, and must mean passthrough.
# ──────────────────────────────────────────────────────────────────────────
def test_proxy_server_module_imports_without_rust_core() -> None:
"""`import headroom.proxy.server` must survive an unloadable `_core`.
Regression for issue #2918. `headroom.transforms.error_detection` used
to do a top-level `from headroom._core import ...`, and
`content_router` imports it at module level, so a Windows box where
Smart App Control blocked the unsigned `_core.pyd` couldn't even
import the server — the `HEADROOM_REQUIRE_RUST_CORE=false` opt-out the
other tests in this file exercise was unreachable in exactly the
situation it exists for, and users got `ConnectionRefused` instead of
a degraded proxy.
This has to run in a child interpreter: the tests above deliberately
import the server module first and only then poison `sys.modules`, so
an in-process check would pass on the cached module regardless.
"""
import subprocess
import sys
from pathlib import Path
result = subprocess.run(
[
sys.executable,
"-c",
"import sys\n"
"sys.modules['headroom._core'] = None\n"
"import headroom.proxy.server\n"
"print('OK')\n",
],
capture_output=True,
text=True,
cwd=Path(__file__).resolve().parents[1],
timeout=300,
)
assert result.returncode == 0, (
"importing headroom.proxy.server without a loadable Rust core failed — "
"some module on the proxy import path took a hard top-level "
f"`from headroom._core import ...` dependency again:\n{result.stderr}"
)
assert "OK" in result.stdout
def test_degraded_mode_forces_passthrough(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Degraded mode must turn optimization off, not just label itself.
Every compressor reaches `headroom._core` sooner or later, so a proxy
that starts with `rust_core: "disabled"` while `optimize` is still
true would answer each request with an ImportError. The point of the
opt-out is that the client keeps working, so the lifespan drops to the
same passthrough path `--no-optimize` uses.
"""
from fastapi.testclient import TestClient
from headroom.proxy.server import ProxyConfig, create_app
monkeypatch.setenv("HEADROOM_REQUIRE_RUST_CORE", "false")
import sys
sys.modules.pop("headroom._core", None)
sys.modules["headroom._core"] = None # type: ignore[assignment]
config = ProxyConfig(
optimize=True, # the point of the test: this must be flipped off
image_optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
ccr_inject_tool=False,
ccr_handle_responses=False,
ccr_context_tracking=False,
)
app = create_app(config)
try:
with TestClient(app) as client:
response = client.get("/health")
assert response.status_code == 200
assert response.json()["rust_core"] == "disabled"
assert config.optimize is False, (
"lifespan left optimize=True while the Rust core is unavailable — "
"every compressed request would raise ImportError"
)
finally:
sys.modules.pop("headroom._core", None)
# ──────────────────────────────────────────────────────────────────────────
# 4. Happy path through the helper: real extension present → status=loaded.
# ──────────────────────────────────────────────────────────────────────────
def test_check_rust_core_returns_loaded_when_extension_present(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""When `headroom._core` is loadable and `hello()` returns the
expected sentinel, `_check_rust_core` returns ``("loaded", None)``.
This covers the production-happy path so a future change that
breaks the marker check (e.g. tightening to require a JSON dict
return) will fail this test rather than degrading silently.
"""
from headroom.proxy import server
# Make sure the env var doesn't tip us into the disabled branch.
monkeypatch.delenv("HEADROOM_REQUIRE_RUST_CORE", raising=False)
status, error = server._check_rust_core()
assert status == "loaded"
assert error is None
def test_proxy_refuses_marker_mismatch(monkeypatch: pytest.MonkeyPatch) -> None:
"""A loadable but stale extension is still a required-core failure."""
import sys
from types import SimpleNamespace
from headroom.proxy import server
monkeypatch.delenv("HEADROOM_REQUIRE_RUST_CORE", raising=False)
original = sys.modules.get("headroom._core")
sys.modules["headroom._core"] = SimpleNamespace(hello=lambda: "stale-core")
try:
with pytest.raises(SystemExit) as exc_info:
server._check_rust_core()
assert exc_info.value.code == 78
finally:
if original is None:
sys.modules.pop("headroom._core", None)
else:
sys.modules["headroom._core"] = original