mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): preserve Windows service when Rust core is blocked (#2989)
## Description When `headroom._core` cannot load, `HEADROOM_REQUIRE_RUST_CORE=false` should keep the Python proxy available in unoptimized passthrough mode. A module-level native import made that escape hatch unreachable before startup could read the setting. This addresses the degraded-service portion of #2918 and advances #1466; neither issue is closed because Authenticode/Smart App Control reputation work remains. ## Type of Change - [x] Bug fix (non-breaking change) - [x] Documentation update ## Changes Made - Resolve Rust-backed error-detection registries and patterns lazily while retaining Rust as the sole source of truth. - Force `optimize=false` only when the operator explicitly allows a missing Rust core, keeping traffic available and unmodified. - Preserve the default fail-loud startup behavior. - Add Windows Smart App Control, WDAC, antivirus, and Code Integrity diagnostics. - Add cold-interpreter coverage for the import boundary and degraded startup behavior. ## Testing - [x] Focused tests pass - [x] Ruff check and formatting pass - [x] `git diff --check` passes - [x] New regression tests added ```text uv run --extra dev pytest -q tests/test_error_detection.py tests/test_rust_core_smoke.py 16 passed in 1.64s (independent exact-head review run) Additional author verification: 38 focused proxy/core/error-detection/parity/MCP contract tests passed. ``` CI is rerunning on the current head after its merge from `main`; no failure should be inferred from an unrun or pending job. ## Real Behavior Proof - Environment: cold CPython 3.12 interpreter with `headroom._core` marked unavailable and `HEADROOM_REQUIRE_RUST_CORE=false`. - Exact steps: import `ProxyConfig`/`create_app`, construct the application with `optimize=True`, and enter the FastAPI lifespan through the focused degraded-mode tests. - Observed result: the server module and application construct successfully; lifespan reports `rust_core=disabled` and changes the shared config to `optimize=False`. Rust-backed constants still fail loudly if explicitly used without the extension. - Not tested: a production Windows host with Smart App Control actively blocking the shipped `_core.pyd`; the failure boundary is reproduced deterministically in a child interpreter. ## Runtime Rollout Safety - Rollout-managed feature: explicit degraded mode controlled by `HEADROOM_REQUIRE_RUST_CORE=false`. - Stable/default behavior changed: no; the default still exits loudly when the required core cannot load. - Kill switch / disable path: unset the opt-out or set it to true to restore fail-loud behavior. - Unsafe override required: no. - Qualification impact: cold-import, degraded-startup, parity, and error-detection tests must remain green. - Rollback path: revert this PR; no persistent data or migration is involved. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] Code follows project style - [x] Documentation is updated - [x] New tests cover the changed behavior - [x] No manual changelog edit ## Remaining Work #2918 remains open for Authenticode signing and Smart App Control reputation. #1466 remains the Windows compatibility umbrella. Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5
JD Davis
parent
0712e048b0
commit
eaa16d9f63
@@ -643,13 +643,30 @@ def _check_rust_core() -> tuple[str, str | None]:
|
||||
return ("disabled", reason)
|
||||
# Fail loud. Print to stderr in addition to logging so operators
|
||||
# see it even if the logging handler is mis-configured.
|
||||
#
|
||||
# On Windows the usual cause is not a missing build but a present
|
||||
# binary the OS refuses to load: `_core.pyd` ships unsigned, so
|
||||
# Smart App Control / WDAC / antivirus can block it (issue #2918).
|
||||
# "rebuild the wheel" is useless advice there, so name the real
|
||||
# cause and point at the degraded mode instead.
|
||||
windows_hint = ""
|
||||
if sys.platform == "win32":
|
||||
windows_hint = (
|
||||
" windows: a 'DLL load failed' or 'blocked by application control policy'\n"
|
||||
" error usually means Smart App Control, WDAC, or antivirus\n"
|
||||
" blocked the (unsigned) _core.pyd rather than the wheel being\n"
|
||||
" broken. Check Event Viewer > Applications and Services Logs >\n"
|
||||
" Microsoft > Windows > CodeIntegrity/Operational for event 3033.\n"
|
||||
)
|
||||
msg = (
|
||||
f"FATAL: Rust extension `headroom._core` not loadable.\n"
|
||||
f" error: {reason}\n"
|
||||
f"{windows_hint}"
|
||||
f" fix: `make build-wheel && pip install --force-reinstall "
|
||||
f"target/wheels/headroom_*.whl`\n"
|
||||
f" opt-out: set {_RUST_CORE_REQUIRED_ENV}=false to start in "
|
||||
f"degraded Python-only mode\n"
|
||||
f"degraded Python-only mode (proxy stays up and forwards traffic "
|
||||
f"unoptimized)\n"
|
||||
)
|
||||
logger.error("event=rust_core_missing reason=%r action=exit_78", reason)
|
||||
print(msg, file=sys.stderr, flush=True)
|
||||
@@ -3096,6 +3113,25 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
app.state.rust_core_status = _rust_core_status
|
||||
app.state.rust_core_error = _rust_core_error
|
||||
|
||||
# Degraded mode has to mean something. Every compressor reaches
|
||||
# `headroom._core` eventually, so leaving `optimize` on while the
|
||||
# extension is unloadable turns each request into an ImportError
|
||||
# instead of a proxied response. Drop to the same passthrough path
|
||||
# `--no-optimize` uses: no savings, but the client keeps working
|
||||
# rather than getting a connection refused (issue #2918).
|
||||
#
|
||||
# Safe to mutate here: `HeadroomProxy` holds this exact config
|
||||
# object, its compressors are built lazily on first use, and
|
||||
# `proxy.startup()` has not run yet.
|
||||
if _rust_core_status == "disabled" and config.optimize:
|
||||
config.optimize = False
|
||||
logger.warning(
|
||||
"event=proxy_optimization_disabled reason=rust_core_unavailable "
|
||||
"detail=%r mode=passthrough; traffic is forwarded unmodified, "
|
||||
"restore `headroom._core` to re-enable compression",
|
||||
_rust_core_error,
|
||||
)
|
||||
|
||||
configure_otel_metrics(OTelMetricsConfig.from_env(default_service_name="headroom-proxy"))
|
||||
configure_langfuse_tracing(
|
||||
LangfuseTracingConfig.from_env(default_service_name="headroom-proxy")
|
||||
|
||||
@@ -7,7 +7,8 @@ Phase 3e.1 ported the keyword data + scoring logic to
|
||||
1. Pulls the keyword tables out of Rust via
|
||||
``headroom._core.keyword_registry_snapshot()`` so the Python side
|
||||
never re-declares them and cannot drift from the Rust source of
|
||||
truth.
|
||||
truth. The snapshot is taken on first use rather than at import —
|
||||
see :func:`_registry` for why that matters to the proxy.
|
||||
2. Re-exports the legacy ``frozenset`` and compiled-regex names
|
||||
(``ERROR_KEYWORDS``, ``ERROR_PATTERN``, ``PRIORITY_PATTERNS_TEXT``,
|
||||
…) so the existing callers in ``search_compressor``,
|
||||
@@ -36,17 +37,7 @@ The Rust implementation fixes two bugs the Python originals carried:
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import cast
|
||||
|
||||
from headroom._core import (
|
||||
content_has_error_indicators as _rust_content_has_error_indicators,
|
||||
)
|
||||
from headroom._core import (
|
||||
keyword_registry_snapshot as _rust_keyword_registry_snapshot,
|
||||
)
|
||||
from headroom._core import (
|
||||
score_line as _rust_score_line,
|
||||
)
|
||||
from typing import TYPE_CHECKING, Any, cast
|
||||
|
||||
|
||||
def score_line(line: str, context: str = "text") -> tuple[str | None, float, float]:
|
||||
@@ -62,19 +53,48 @@ def score_line(line: str, context: str = "text") -> tuple[str | None, float, flo
|
||||
``#[pyfunction]``s; this shim translates that into the explicit
|
||||
Python error every caller would expect.
|
||||
"""
|
||||
from headroom._core import score_line as _rust_score_line
|
||||
|
||||
result = _rust_score_line(line, context)
|
||||
if result is None:
|
||||
raise ValueError(f"unknown importance context: {context}")
|
||||
return cast("tuple[str | None, float, float]", result)
|
||||
|
||||
|
||||
_REGISTRY: dict[str, list[str]] = _rust_keyword_registry_snapshot()
|
||||
_REGISTRY: dict[str, list[str]] | None = None
|
||||
|
||||
|
||||
def _registry() -> dict[str, list[str]]:
|
||||
"""Return the Rust keyword tables, importing `headroom._core` on demand.
|
||||
|
||||
Deliberately NOT resolved at module import: `content_router` imports
|
||||
this module at *its* module level, so a hard import here is the one
|
||||
unguarded `headroom._core` edge on the whole proxy startup path. When
|
||||
the extension cannot load — a Windows build blocked by Smart App
|
||||
Control or antivirus, a wheel built without the extension — that made
|
||||
`import headroom.proxy.server` fail outright, which in turn made the
|
||||
proxy's own documented degraded mode
|
||||
(``HEADROOM_REQUIRE_RUST_CORE=false``) unreachable and reported the
|
||||
failure as a missing `headroom-ai[proxy]` extra. See issue #2918.
|
||||
|
||||
Resolving on first use keeps Rust the single source of truth (there is
|
||||
still no Python fallback keyword table) while letting the module import
|
||||
on a machine where the extension is unavailable. Callers that actually
|
||||
need the tables get the same `ImportError` they always did, just at
|
||||
first use instead of at import.
|
||||
"""
|
||||
global _REGISTRY
|
||||
if _REGISTRY is None:
|
||||
from headroom._core import keyword_registry_snapshot
|
||||
|
||||
_REGISTRY = keyword_registry_snapshot()
|
||||
return _REGISTRY
|
||||
|
||||
|
||||
def _alternation(words: list[str]) -> str:
|
||||
"""Compile a `\b(w1|w2|…)\b` regex source from the Rust-supplied list.
|
||||
|
||||
The keywords are static (compiled once on import) so we don't need
|
||||
The keywords are static (compiled once on first use) so we don't need
|
||||
`re.escape` for the current set, but using it keeps the shim
|
||||
correct if a future Rust update adds a regex meta-character.
|
||||
"""
|
||||
@@ -82,52 +102,109 @@ def _alternation(words: list[str]) -> str:
|
||||
return r"\b(" + "|".join(escaped) + r")\b"
|
||||
|
||||
|
||||
# ─── Canonical keyword sets (pulled from Rust at import time) ───────────────
|
||||
|
||||
ERROR_KEYWORDS: frozenset[str] = frozenset(_REGISTRY["error"])
|
||||
|
||||
# Importance keywords historically included the error set — preserve that
|
||||
# union so consumers iterating the set get the same membership as before.
|
||||
IMPORTANCE_KEYWORDS: frozenset[str] = frozenset(
|
||||
list(_REGISTRY["error"]) + list(_REGISTRY["importance"]) + list(_REGISTRY["warning"])
|
||||
)
|
||||
|
||||
SECURITY_KEYWORDS: frozenset[str] = frozenset(_REGISTRY["security"])
|
||||
|
||||
ERROR_INDICATOR_KEYWORDS: tuple[str, ...] = tuple(_REGISTRY["error_indicators"])
|
||||
# ─── Canonical keyword sets and compiled patterns ───────────────────────────
|
||||
#
|
||||
# Built from the Rust registry on first attribute access (PEP 562) and then
|
||||
# cached into the module globals, so each one is computed at most once — the
|
||||
# same "compile once" property the eager module-level assignments had. The
|
||||
# builders below are the only definition of each name; keep the keys in
|
||||
# `_LAZY_ATTRS` in sync with `__all__`.
|
||||
|
||||
|
||||
# ─── Compiled patterns ──────────────────────────────────────────────────────
|
||||
|
||||
ERROR_PATTERN: re.Pattern[str] = re.compile(_alternation(_REGISTRY["error"]), re.IGNORECASE)
|
||||
WARNING_PATTERN: re.Pattern[str] = re.compile(_alternation(_REGISTRY["warning"]), re.IGNORECASE)
|
||||
IMPORTANCE_PATTERN: re.Pattern[str] = re.compile(
|
||||
_alternation(_REGISTRY["importance"]), re.IGNORECASE
|
||||
)
|
||||
SECURITY_PATTERN: re.Pattern[str] = re.compile(_alternation(_REGISTRY["security"]), re.IGNORECASE)
|
||||
def _build_error_pattern() -> re.Pattern[str]:
|
||||
return re.compile(_alternation(_registry()["error"]), re.IGNORECASE)
|
||||
|
||||
|
||||
# ─── Per-context priority pattern lists ─────────────────────────────────────
|
||||
def _build_warning_pattern() -> re.Pattern[str]:
|
||||
return re.compile(_alternation(_registry()["warning"]), re.IGNORECASE)
|
||||
|
||||
PRIORITY_PATTERNS_SEARCH: list[re.Pattern[str]] = [
|
||||
ERROR_PATTERN,
|
||||
WARNING_PATTERN,
|
||||
IMPORTANCE_PATTERN,
|
||||
]
|
||||
|
||||
PRIORITY_PATTERNS_DIFF: list[re.Pattern[str]] = [
|
||||
ERROR_PATTERN,
|
||||
IMPORTANCE_PATTERN,
|
||||
SECURITY_PATTERN,
|
||||
]
|
||||
def _build_importance_pattern() -> re.Pattern[str]:
|
||||
return re.compile(_alternation(_registry()["importance"]), re.IGNORECASE)
|
||||
|
||||
# Markdown structural prefixes: matched on whole lines, anchored with `^`.
|
||||
# Pulled from Rust so the prefix table can't drift either.
|
||||
PRIORITY_PATTERNS_TEXT: list[re.Pattern[str]] = [
|
||||
ERROR_PATTERN,
|
||||
IMPORTANCE_PATTERN,
|
||||
*(re.compile("^" + re.escape(prefix)) for prefix in _REGISTRY["markdown_prefixes"]),
|
||||
]
|
||||
|
||||
def _build_security_pattern() -> re.Pattern[str]:
|
||||
return re.compile(_alternation(_registry()["security"]), re.IGNORECASE)
|
||||
|
||||
|
||||
_LAZY_ATTRS: dict[str, Any] = {
|
||||
"ERROR_KEYWORDS": lambda: frozenset(_registry()["error"]),
|
||||
# Importance keywords historically included the error set — preserve that
|
||||
# union so consumers iterating the set get the same membership as before.
|
||||
"IMPORTANCE_KEYWORDS": lambda: frozenset(
|
||||
list(_registry()["error"]) + list(_registry()["importance"]) + list(_registry()["warning"])
|
||||
),
|
||||
"SECURITY_KEYWORDS": lambda: frozenset(_registry()["security"]),
|
||||
"ERROR_INDICATOR_KEYWORDS": lambda: tuple(_registry()["error_indicators"]),
|
||||
"ERROR_PATTERN": _build_error_pattern,
|
||||
"WARNING_PATTERN": _build_warning_pattern,
|
||||
"IMPORTANCE_PATTERN": _build_importance_pattern,
|
||||
"SECURITY_PATTERN": _build_security_pattern,
|
||||
# Per-context priority pattern lists. Each goes through `_get` so the
|
||||
# shared compiled patterns above are reused rather than recompiled: a
|
||||
# plain `ERROR_PATTERN` reference inside a function body is a global
|
||||
# lookup, and global lookups do NOT fall through to `__getattr__`.
|
||||
"PRIORITY_PATTERNS_SEARCH": lambda: [
|
||||
_get("ERROR_PATTERN"),
|
||||
_get("WARNING_PATTERN"),
|
||||
_get("IMPORTANCE_PATTERN"),
|
||||
],
|
||||
"PRIORITY_PATTERNS_DIFF": lambda: [
|
||||
_get("ERROR_PATTERN"),
|
||||
_get("IMPORTANCE_PATTERN"),
|
||||
_get("SECURITY_PATTERN"),
|
||||
],
|
||||
# Markdown structural prefixes: matched on whole lines, anchored with `^`.
|
||||
# Pulled from Rust so the prefix table can't drift either.
|
||||
"PRIORITY_PATTERNS_TEXT": lambda: [
|
||||
_get("ERROR_PATTERN"),
|
||||
_get("IMPORTANCE_PATTERN"),
|
||||
*(re.compile("^" + re.escape(prefix)) for prefix in _registry()["markdown_prefixes"]),
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _get(name: str) -> Any:
|
||||
"""Read a lazy module constant from inside this module.
|
||||
|
||||
`__getattr__` only fires for attribute access on the module object, so
|
||||
code *within* the module has to route through here instead of reading
|
||||
the global directly.
|
||||
"""
|
||||
value = globals().get(name)
|
||||
if value is None:
|
||||
value = __getattr__(name)
|
||||
return value
|
||||
|
||||
|
||||
def __getattr__(name: str) -> Any:
|
||||
"""Build the Rust-derived module constants on first access (PEP 562)."""
|
||||
builder = _LAZY_ATTRS.get(name)
|
||||
if builder is None:
|
||||
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
|
||||
value = builder()
|
||||
globals()[name] = value
|
||||
return value
|
||||
|
||||
|
||||
def __dir__() -> list[str]:
|
||||
return sorted(set(globals()) | set(_LAZY_ATTRS))
|
||||
|
||||
|
||||
if TYPE_CHECKING:
|
||||
# Declared for type checkers and editors only — at runtime these are
|
||||
# produced by `__getattr__` above.
|
||||
ERROR_KEYWORDS: frozenset[str]
|
||||
IMPORTANCE_KEYWORDS: frozenset[str]
|
||||
SECURITY_KEYWORDS: frozenset[str]
|
||||
ERROR_INDICATOR_KEYWORDS: tuple[str, ...]
|
||||
ERROR_PATTERN: re.Pattern[str]
|
||||
WARNING_PATTERN: re.Pattern[str]
|
||||
IMPORTANCE_PATTERN: re.Pattern[str]
|
||||
SECURITY_PATTERN: re.Pattern[str]
|
||||
PRIORITY_PATTERNS_SEARCH: list[re.Pattern[str]]
|
||||
PRIORITY_PATTERNS_DIFF: list[re.Pattern[str]]
|
||||
PRIORITY_PATTERNS_TEXT: list[re.Pattern[str]]
|
||||
|
||||
|
||||
# ─── Triage helper ──────────────────────────────────────────────────────────
|
||||
@@ -142,6 +219,10 @@ def content_has_error_indicators(text: str) -> bool:
|
||||
Python tracebacks and similar substrings more than connection
|
||||
states.
|
||||
"""
|
||||
from headroom._core import (
|
||||
content_has_error_indicators as _rust_content_has_error_indicators,
|
||||
)
|
||||
|
||||
return bool(_rust_content_has_error_indicators(text))
|
||||
|
||||
|
||||
@@ -194,7 +275,7 @@ def content_has_strong_error_indicators(text: str) -> bool:
|
||||
"""
|
||||
lowered = _ZERO_RESULT_PATTERN.sub(" ", text.lower())
|
||||
hits = 0
|
||||
for keyword in ERROR_INDICATOR_KEYWORDS:
|
||||
for keyword in _get("ERROR_INDICATOR_KEYWORDS"):
|
||||
if keyword in lowered:
|
||||
hits += 1
|
||||
if hits >= 2:
|
||||
|
||||
@@ -2,8 +2,100 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import subprocess
|
||||
import sys
|
||||
import textwrap
|
||||
from pathlib import Path
|
||||
|
||||
from headroom.transforms.error_detection import content_has_strong_error_indicators
|
||||
|
||||
_REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def _run_without_rust_core(body: str) -> subprocess.CompletedProcess[str]:
|
||||
"""Run `body` in a child interpreter where `headroom._core` won't import.
|
||||
|
||||
Has to be a child process: by the time this test runs, the rest of the
|
||||
suite has already imported both `headroom._core` and the modules under
|
||||
test, so poisoning `sys.modules` in-process would prove nothing.
|
||||
|
||||
Setting ``sys.modules["headroom._core"] = None`` is Python's own
|
||||
"known-unimportable" marker, so the `from headroom._core import ...`
|
||||
statements raise `ImportError` exactly as they do when Windows Smart
|
||||
App Control blocks the compiled `_core.pyd` (issue #2918).
|
||||
"""
|
||||
script = "import sys\nsys.modules['headroom._core'] = None\n" + textwrap.dedent(body)
|
||||
return subprocess.run(
|
||||
[sys.executable, "-c", script],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
cwd=_REPO_ROOT,
|
||||
timeout=300,
|
||||
)
|
||||
|
||||
|
||||
def test_module_imports_without_the_rust_extension() -> None:
|
||||
"""`error_detection` must import even when `headroom._core` cannot load.
|
||||
|
||||
`content_router` imports this module at *its* module level, so a hard
|
||||
top-level `from headroom._core import ...` here was the single
|
||||
unguarded native edge on the proxy's whole startup path — it made
|
||||
`import headroom.proxy.server` fail outright, which made the proxy's
|
||||
documented `HEADROOM_REQUIRE_RUST_CORE=false` degraded mode
|
||||
unreachable (issue #2918).
|
||||
"""
|
||||
result = _run_without_rust_core(
|
||||
"""
|
||||
import headroom.transforms.error_detection as ed
|
||||
|
||||
assert callable(ed.content_has_strong_error_indicators)
|
||||
print("OK")
|
||||
"""
|
||||
)
|
||||
assert result.returncode == 0, (
|
||||
f"importing error_detection without the Rust core failed:\n{result.stderr}"
|
||||
)
|
||||
assert "OK" in result.stdout
|
||||
|
||||
|
||||
def test_rust_backed_constants_still_fail_loudly_when_used() -> None:
|
||||
"""Deferring the import must not turn into a silent Python fallback.
|
||||
|
||||
There is no Python copy of the keyword tables, so touching one of the
|
||||
Rust-derived constants without the extension has to raise `ImportError`
|
||||
rather than hand back an empty or made-up set.
|
||||
"""
|
||||
result = _run_without_rust_core(
|
||||
"""
|
||||
import headroom.transforms.error_detection as ed
|
||||
|
||||
for name in ("ERROR_PATTERN", "ERROR_KEYWORDS", "PRIORITY_PATTERNS_TEXT"):
|
||||
try:
|
||||
getattr(ed, name)
|
||||
except ImportError:
|
||||
continue
|
||||
raise AssertionError(f"{name} resolved without headroom._core")
|
||||
print("OK")
|
||||
"""
|
||||
)
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert "OK" in result.stdout
|
||||
|
||||
|
||||
def test_compiled_patterns_are_shared_not_rebuilt_per_access() -> None:
|
||||
"""The per-context lists must reuse the shared compiled patterns.
|
||||
|
||||
The eager version built each pattern exactly once and put the same
|
||||
object in every list; the lazy version has to keep that property so
|
||||
repeated access stays free.
|
||||
"""
|
||||
from headroom.transforms import error_detection as ed
|
||||
|
||||
assert ed.PRIORITY_PATTERNS_SEARCH[0] is ed.ERROR_PATTERN
|
||||
assert ed.PRIORITY_PATTERNS_DIFF[0] is ed.ERROR_PATTERN
|
||||
assert ed.PRIORITY_PATTERNS_TEXT[1] is ed.IMPORTANCE_PATTERN
|
||||
assert ed.PRIORITY_PATTERNS_SEARCH is ed.PRIORITY_PATTERNS_SEARCH
|
||||
|
||||
|
||||
def test_real_error_output_is_detected() -> None:
|
||||
text = "Traceback (most recent call last):\n ...\nValueError: fatal error during load"
|
||||
|
||||
@@ -156,6 +156,99 @@ def test_proxy_starts_in_degraded_mode_when_opt_out_set(
|
||||
sys.modules.pop("headroom._core", None)
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────
|
||||
# 3b. Degraded mode must actually be reachable, and must mean passthrough.
|
||||
# ──────────────────────────────────────────────────────────────────────────
|
||||
def test_proxy_server_module_imports_without_rust_core() -> None:
|
||||
"""`import headroom.proxy.server` must survive an unloadable `_core`.
|
||||
|
||||
Regression for issue #2918. `headroom.transforms.error_detection` used
|
||||
to do a top-level `from headroom._core import ...`, and
|
||||
`content_router` imports it at module level, so a Windows box where
|
||||
Smart App Control blocked the unsigned `_core.pyd` couldn't even
|
||||
import the server — the `HEADROOM_REQUIRE_RUST_CORE=false` opt-out the
|
||||
other tests in this file exercise was unreachable in exactly the
|
||||
situation it exists for, and users got `ConnectionRefused` instead of
|
||||
a degraded proxy.
|
||||
|
||||
This has to run in a child interpreter: the tests above deliberately
|
||||
import the server module first and only then poison `sys.modules`, so
|
||||
an in-process check would pass on the cached module regardless.
|
||||
"""
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
result = subprocess.run(
|
||||
[
|
||||
sys.executable,
|
||||
"-c",
|
||||
"import sys\n"
|
||||
"sys.modules['headroom._core'] = None\n"
|
||||
"import headroom.proxy.server\n"
|
||||
"print('OK')\n",
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
cwd=Path(__file__).resolve().parents[1],
|
||||
timeout=300,
|
||||
)
|
||||
assert result.returncode == 0, (
|
||||
"importing headroom.proxy.server without a loadable Rust core failed — "
|
||||
"some module on the proxy import path took a hard top-level "
|
||||
f"`from headroom._core import ...` dependency again:\n{result.stderr}"
|
||||
)
|
||||
assert "OK" in result.stdout
|
||||
|
||||
|
||||
def test_degraded_mode_forces_passthrough(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Degraded mode must turn optimization off, not just label itself.
|
||||
|
||||
Every compressor reaches `headroom._core` sooner or later, so a proxy
|
||||
that starts with `rust_core: "disabled"` while `optimize` is still
|
||||
true would answer each request with an ImportError. The point of the
|
||||
opt-out is that the client keeps working, so the lifespan drops to the
|
||||
same passthrough path `--no-optimize` uses.
|
||||
"""
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from headroom.proxy.server import ProxyConfig, create_app
|
||||
|
||||
monkeypatch.setenv("HEADROOM_REQUIRE_RUST_CORE", "false")
|
||||
|
||||
import sys
|
||||
|
||||
sys.modules.pop("headroom._core", None)
|
||||
sys.modules["headroom._core"] = None # type: ignore[assignment]
|
||||
|
||||
config = ProxyConfig(
|
||||
optimize=True, # the point of the test: this must be flipped off
|
||||
image_optimize=False,
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
cost_tracking_enabled=False,
|
||||
log_requests=False,
|
||||
ccr_inject_tool=False,
|
||||
ccr_handle_responses=False,
|
||||
ccr_context_tracking=False,
|
||||
)
|
||||
app = create_app(config)
|
||||
|
||||
try:
|
||||
with TestClient(app) as client:
|
||||
response = client.get("/health")
|
||||
assert response.status_code == 200
|
||||
assert response.json()["rust_core"] == "disabled"
|
||||
assert config.optimize is False, (
|
||||
"lifespan left optimize=True while the Rust core is unavailable — "
|
||||
"every compressed request would raise ImportError"
|
||||
)
|
||||
finally:
|
||||
sys.modules.pop("headroom._core", None)
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────
|
||||
# 4. Happy path through the helper: real extension present → status=loaded.
|
||||
# ──────────────────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -459,6 +459,64 @@ export HEADROOM_REQUIRE_RUST_CORE=false
|
||||
|
||||
---
|
||||
|
||||
### Windows: "DLL load failed while importing _core"
|
||||
|
||||
Symptoms: the proxy exits at startup with
|
||||
|
||||
```
|
||||
FATAL: Rust extension `headroom._core` not loadable.
|
||||
error: ImportError: DLL load failed while importing _core: ...
|
||||
```
|
||||
|
||||
and clients pointed at `ANTHROPIC_BASE_URL=http://127.0.0.1:8787` get
|
||||
`ConnectionRefused`.
|
||||
|
||||
The compiled extension `headroom\_core.pyd` ships unsigned. Windows Smart App
|
||||
Control, WDAC, or antivirus can therefore refuse to load it, and Smart App
|
||||
Control has no per-file allowlist: it is a system-wide, one-way switch. Confirm
|
||||
it is the OS blocking the file rather than a broken install:
|
||||
|
||||
```powershell
|
||||
# Where the extension lives (adjust for your install method)
|
||||
$core = "$env:APPDATA\uv\tools\headroom-ai\Lib\site-packages\headroom\_core.pyd"
|
||||
Get-AuthenticodeSignature $core | Format-List Status, StatusMessage
|
||||
|
||||
# Code Integrity block events, if any
|
||||
Get-WinEvent -LogName "Microsoft-Windows-CodeIntegrity/Operational" -MaxEvents 20 |
|
||||
Where-Object Id -in 3033, 3077, 3118 | Format-List TimeCreated, Id, Message
|
||||
```
|
||||
|
||||
A `Status: NotSigned` plus event 3033 or 3077 naming `_core.pyd` confirms it.
|
||||
|
||||
To keep the proxy serving while the block is in place, start it in degraded
|
||||
mode. The proxy stays up and forwards traffic **unoptimized** (you lose the token
|
||||
savings, not the connection), `/health` reports `"rust_core": "disabled"`, and
|
||||
the startup log carries one `event=proxy_optimization_disabled` warning so the
|
||||
degradation is visible:
|
||||
|
||||
```powershell
|
||||
$env:HEADROOM_REQUIRE_RUST_CORE = "false"
|
||||
headroom proxy
|
||||
```
|
||||
|
||||
For a scheduled-task or service deployment, bake it into the profile so the
|
||||
supervisor passes it to every restart. Supervisors start with a bare
|
||||
environment and will not pick up an export from your shell, so re-run
|
||||
`install apply` with the options you originally used plus `--env`:
|
||||
|
||||
```powershell
|
||||
headroom install apply --preset persistent-task --env HEADROOM_REQUIRE_RUST_CORE=false
|
||||
headroom install restart
|
||||
```
|
||||
|
||||
To get compression back you have to let the file load: exclude the `headroom`
|
||||
package directory in your antivirus, or turn Smart App Control off (Windows
|
||||
Security > App & browser control > Smart App Control). Note that turning it off
|
||||
is irreversible without reinstalling Windows, so prefer degraded mode unless you
|
||||
were going to disable it anyway.
|
||||
|
||||
---
|
||||
|
||||
## Error Reference
|
||||
|
||||
| Exception | Meaning | Solution |
|
||||
|
||||
Reference in New Issue
Block a user