mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description CI shard 4 was not merely slow: after thousands of fast tests it parked indefinitely inside `headroom._core.detect_content_type` at 0% CPU. The router watchdogged only the first native call and then permanently trusted direct calls via `_detect_native_verified`. Earlier suite activity can change ORT/native state after that first success, making a later call deadlock until GitHub cancels the job. This keeps every native call bounded by the existing watchdog, activates the process-wide pure-Python circuit breaker after a timeout, restores the test-job ceiling to 30 minutes, and removes a separate wall-clock scheduler assertion that generated false shard-1 failures despite the structural regression guards passing. No issue is auto-closed by this infrastructure repair. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Removed the unsafe process-lifetime `_detect_native_verified` fast path. - Kept every native detection call behind the existing bounded watchdog. - Preserved the process-wide fallback circuit breaker so only the first wedged call consumes the watchdog budget. - Added a success-then-hang regression test. - Isolated native circuit-breaker state in fallback exception tests. - Restored the CI test timeout from the temporary 90-minute diagnostic ceiling to 30 minutes. - Replaced the Codex scheduler's noise-sensitive p99/p50 assertion with its meaningful absolute regression ceiling while retaining source-level guards against the removed semaphore and nested executor. - Corrected import order and formatting defects inherited from current main so the synthetic merge commit passes repository-wide lint. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text Exact local shard-4 command with coverage: 2723 passed, 172 skipped, 8661 deselected in 108.40s Focused detector/router suite: 62 passed Codex scheduler suite: 3 passed, 1 skipped ruff check . All checks passed! ruff format --check . 1411 files already formatted mypy headroom/transforms/content_router.py Success: no issues found in 1 source file ``` Exact-head GitHub CI on `28f284c7a156d5be2fac2d21ba904a19d3389e6d` is entirely green. Test jobs 1–4, test-extras, test-agno, build, wheel, lint, CodeQL, dependency audit, secret scan, smoke, governance, and conflict checks all passed. Remaining skips are path-filtered jobs not applicable to this diff. ## Real Behavior Proof - Environment: macOS arm64/Python 3.13 locally; GitHub-hosted Ubuntu/Python 3.12 using the production CI workflow and prebuilt wheel. - Exact command / steps: reproduced `pytest tests scripts/tests --splits 4 --group 4 ...` hanging in native detection; sampled the parked process; reran with `pytest-timeout` to locate `_rust_detect`; applied the correction; reran the exact shard locally and all four CI shards remotely. - Observed result: local shard 4 completed in 1:48. GitHub shard 4's pytest step completed in 5:45 and its full job in 8:06 under the restored 30-minute ceiling. All four shards passed on the same head. - Not tested: deliberately wedging a real production ORT runtime outside the deterministic mocked regression; the watchdog behavior is covered with a native-call fake that succeeds once and then never returns. ## Runtime Rollout Safety - Rollout-managed feature(s): native content detection watchdog and fallback only. - Minimum rollout channel: normal patch release; no staged feature flag required. - Stable/default behavior changed: every native detection call remains watchdog-bounded instead of only the first successful call. - Kill switch / disable path: `HEADROOM_DETECT_BACKEND=python` bypasses native detection; `HEADROOM_DETECT_TIMEOUT_SECS` controls the watchdog budget. - Unsafe override required: none. - Qualification impact: full Python CI matrix must remain green; exact shard-4 completion is the primary qualification evidence. - Rollback path: human revert of this PR if bounded calls cause an unexpected regression; setting the Python backend provides an immediate operational fallback without code rollback. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation — inline lifecycle documentation and PR operational notes; no user-facing docs change is needed - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable; no UI change. ## Additional Notes Human review only. No merge or auto-merge action has been configured. The branch includes current main and preserves the MCP SDK compatibility cap `mcp>=1.28.1,<2.0.0`.
87 lines
3.3 KiB
Python
87 lines
3.3 KiB
Python
"""Native (Rust) content-detector failures must degrade to the pure-Python
|
|
detector instead of propagating out as an HTTP 500. Regression test for #1123."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
|
|
import pytest
|
|
|
|
import headroom._ort as ort_runtime
|
|
from headroom.transforms import content_router as cr
|
|
|
|
# Patch the native detector via its string target ("headroom._core.detect_content_type")
|
|
# rather than a module alias captured at import time. content_router._detect_content does a
|
|
# fresh `from headroom._core import detect_content_type` on every call, and other tests pop
|
|
# headroom._core out of sys.modules (e.g. test_rust_core_smoke), which rebuilds the module
|
|
# object. A captured alias would then go stale and the patch would miss the live module —
|
|
# the control-flow tests would silently run the real detector and never see the exception.
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _compatible_mock_native_runtime(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""Keep mocked native calls reachable regardless of prior test state."""
|
|
monkeypatch.setattr(ort_runtime, "rust_ort_runtime_compatible", lambda: True)
|
|
monkeypatch.setattr(cr, "_detect_native_unhealthy", False)
|
|
|
|
|
|
def test_falls_back_on_rust_exception(monkeypatch):
|
|
"""An ordinary exception from the native detector degrades to regex."""
|
|
|
|
def _boom(_content):
|
|
raise RuntimeError("simulated native failure")
|
|
|
|
monkeypatch.setenv("HEADROOM_DETECT_BACKEND", "rust")
|
|
monkeypatch.setattr("headroom._core.detect_content_type", _boom)
|
|
monkeypatch.setattr(cr, "_detect_panic_warned", False, raising=False)
|
|
|
|
# Must not raise; returns a usable detection result from the regex path.
|
|
result = cr._detect_content('{"a": 1, "b": [1, 2, 3]}')
|
|
assert result is not None
|
|
assert result.content_type is not None
|
|
|
|
|
|
def test_falls_back_on_baseexception_panic(monkeypatch):
|
|
"""A BaseException-derived panic (like pyo3's PanicException) is caught too."""
|
|
|
|
class FakePanic(BaseException):
|
|
pass
|
|
|
|
def _panic(_content):
|
|
raise FakePanic("simulated pyo3 panic")
|
|
|
|
monkeypatch.setenv("HEADROOM_DETECT_BACKEND", "rust")
|
|
monkeypatch.setattr("headroom._core.detect_content_type", _panic)
|
|
monkeypatch.setattr(cr, "_detect_panic_warned", False, raising=False)
|
|
|
|
result = cr._detect_content("some plain text content here")
|
|
assert result is not None
|
|
|
|
|
|
def test_control_flow_exceptions_propagate(monkeypatch):
|
|
"""KeyboardInterrupt/SystemExit must not be swallowed by the fallback."""
|
|
|
|
def _interrupt(_content):
|
|
raise KeyboardInterrupt
|
|
|
|
monkeypatch.setenv("HEADROOM_DETECT_BACKEND", "rust")
|
|
monkeypatch.setattr("headroom._core.detect_content_type", _interrupt)
|
|
monkeypatch.setattr(cr, "_detect_panic_warned", False, raising=False)
|
|
|
|
with pytest.raises(KeyboardInterrupt):
|
|
cr._detect_content("content")
|
|
|
|
|
|
def test_cancelled_error_propagates(monkeypatch):
|
|
"""asyncio.CancelledError must propagate, not be swallowed as a fallback."""
|
|
|
|
def _cancel(_content):
|
|
raise asyncio.CancelledError()
|
|
|
|
monkeypatch.setenv("HEADROOM_DETECT_BACKEND", "rust")
|
|
monkeypatch.setattr("headroom._core.detect_content_type", _cancel)
|
|
monkeypatch.setattr(cr, "_detect_panic_warned", False, raising=False)
|
|
|
|
with pytest.raises(asyncio.CancelledError):
|
|
cr._detect_content("content")
|