mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description Nothing in Headroom ever stops a wrap-spawned proxy whose wrappers are gone. Two leak paths, both by-products of deliberate hardening without a counterpart: 1. **Unclean wrapper death.** `_start_proxy` detaches the proxy (`start_new_session` on POSIX, breakaway console flags on Windows) on purpose, so a crashed wrapper cannot tree-kill a shared proxy out from under other clients. But when the wrapper dies without running cleanup (SIGKILL, crash, terminal close), its client marker goes stale, `_live_proxy_clients` unlinks it, and the proxy itself is never stopped. 2. **Graceful last-exit leak.** `_make_cleanup` only terminates the proxy when the exiting wrapper *started* it (`proxy_holder[0] is not None`). When the starter exits while other clients remain (correct), every later wrapper exits with no handle, so even a perfectly behaved multi-wrap topology leaks the proxy at the end. Observed in production: three `headroom proxy` processes still alive 21 days after their last session, holding ports and being silently reused by later wraps (which then inherited their stale version and captive routing; see the two companion PRs). The fix: wrap-spawned proxies carry `HEADROOM_WRAP_OWNED=1` and run an orphan-watchdog task that polls the per-port client-marker dir every 30s and gracefully stops the process once no live markers, no active WebSocket sessions, and no served-request activity remain for a grace period (default 900s, floor 60s, `HEADROOM_WRAP_ORPHAN_GRACE_SECONDS`). Standalone `headroom proxy` instances and persistent installs never get the marker env var and are never affected; multi-worker runs are excluded (WS sessions are per-worker, and wrap never requests multiple workers). Closes #3199 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/proxy/orphan_watchdog.py` (new): env config (`WRAP_OWNED_ENV`, grace parsing with floor), marker liveness (`pid_alive` + start-time identity, pruning stale markers), and `orphan_watchdog_loop` (idle clock reset by live markers, active WS sessions, or movement of the proxy's served-request counter — the last covers direct-HTTP and SSH-forwarded clients that write no marker). Stop is `signal.raise_signal(SIGTERM)`, which reaches uvicorn's handler on Unix and the emulated handler table on Windows (unlike `os.kill(pid, SIGTERM)`, which is `TerminateProcess` there). - `headroom/proxy/server.py`: lifespan spawns the watchdog when `orphan_watchdog_enabled()` and not a multi-worker run (`HEADROOM_PROXY_CONFIG_JSON` absent), and cancels it in teardown like the other periodic tasks. - `headroom/cli/wrap.py`: `_start_proxy` sets `HEADROOM_WRAP_OWNED=1` in the proxy subprocess env; `_proc_identity`/`_identity_mismatch` moved to `headroom/_subprocess.py` (`proc_identity`/`identity_mismatch`) so proxy code does not import the CLI layer — thin module-level delegates keep the existing monkeypatch-based tests working unchanged. - `tests/test_proxy/test_orphan_watchdog.py` (new, 13 tests): env parsing (default/override/floor/garbage), marker pruning (live, dead PID, recycled PID, garbage JSON, missing dir), loop semantics (exit after grace, hold on live client, hold on active WS session, hold on served-request movement, grace reset on returning client, missing registry counts as idle). - `tests/test_cli/test_wrap_claude_vertex_proxy_env.py`: `_start_proxy` marks the subprocess env wrap-owned. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_proxy/test_orphan_watchdog.py -q 13 passed in 2.77s $ uv run pytest tests/test_proxy tests/test_cli -q 973 passed, 1 skipped in 49.85s # 1 pre-existing unrelated failure # (test_litellm_vertex_backend_path_...) fails # identically on clean upstream/main $ uv run ruff format / ruff check on touched files All checks passed! $ uv run mypy headroom/_subprocess.py headroom/cli/wrap.py headroom/proxy/orphan_watchdog.py Success: no issues found in 3 source files ``` ## Real Behavior Proof - Environment: macOS 26.5 arm64, Python 3.13, local source build 0.37.0-dev (integration overlay with this PR applied). - Exact command / steps: start wrap-spawned proxies via `wrap._ensure_proxy(...)` (which goes through `_start_proxy` and thus `HEADROOM_WRAP_OWNED=1`), with zero registered clients; wait; inspect `~/.headroom/logs/proxy.log`. - Observed result: two such proxies (plain on 8789 PID 48365, augment-mode on 8791 PID 49729) self-exited at the 900s default grace: `event=proxy_shutdown reason=signal pid=49729` at 09:51:49 against a ~09:36 start, port released afterwards (`lsof` clean). A machine-wide sweep beforehand showed three 21-day-old orphans (PIDs 73974, 63454, 79339); with this build they can no longer accumulate. - Not tested: Windows signal path (reasoned from CPython/uvicorn semantics; unit-injected `stop` callable covers the loop side); the multi-worker exclusion with a real `--workers 2` run (guarded by env presence only); marker pruning under active PID recycling on Linux without psutil (`/proc` fallback is unit-tested via mocks). ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: N/A - Stable/default behavior changed: only for wrap-spawned proxies, which now self-exit after 15 clientless minutes. Standalone and persistent proxies are untouched. - Kill switch / disable path: omit `HEADROOM_WRAP_OWNED` (not settable by users of wrap; standalone proxies never have it), or set `HEADROOM_WRAP_ORPHAN_GRACE_SECONDS` very high. - Unsafe override required: no - Qualification impact: none - Rollback path: revert the commit; orphaned proxies leak again (status quo ante). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Adversarially reviewed by a second model (glm-5.2, high reasoning); initial verdict needs-work, all accepted findings applied (multi-worker exclusion, Windows-safe stop signal, served-request activity hold for marker-less clients, wider test timing margins) and re-tested green. The reviewer independently verified the boot-race ordering, teardown cancellation, and the kill-vs-self-exit race as already correct. Companion PRs (same incident): `-dev` version-restart fix, routing-config reuse check.
157 lines
5.1 KiB
Python
157 lines
5.1 KiB
Python
"""Regression tests for the Windows-safe PID liveness helper (#1544)."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import sys
|
|
import types
|
|
|
|
from headroom._subprocess import pid_alive
|
|
|
|
|
|
def test_pid_alive_rejects_non_positive() -> None:
|
|
assert pid_alive(0) is False
|
|
assert pid_alive(-1) is False
|
|
|
|
|
|
def test_pid_alive_prefers_psutil_without_signalling(monkeypatch) -> None:
|
|
monkeypatch.setitem(sys.modules, "psutil", types.SimpleNamespace(pid_exists=lambda pid: True))
|
|
|
|
def boom(pid: int, sig: int) -> None:
|
|
raise AssertionError("os.kill must not run when psutil answers")
|
|
|
|
monkeypatch.setattr("headroom._subprocess.os.kill", boom)
|
|
assert pid_alive(4321) is True
|
|
|
|
|
|
def test_pid_alive_systemerror_is_not_alive(monkeypatch) -> None:
|
|
"""WinError 87 surfaces as SystemError on Windows; it must read as 'not alive', not crash."""
|
|
monkeypatch.setitem(
|
|
sys.modules,
|
|
"psutil",
|
|
types.SimpleNamespace(pid_exists=lambda pid: (_ for _ in ()).throw(RuntimeError())),
|
|
)
|
|
monkeypatch.setattr(
|
|
"headroom._subprocess.os.kill",
|
|
lambda pid, sig: (_ for _ in ()).throw(SystemError("WinError 87")),
|
|
)
|
|
assert pid_alive(4321) is False
|
|
|
|
|
|
def test_pid_alive_overflow_is_not_alive(monkeypatch) -> None:
|
|
monkeypatch.setitem(
|
|
sys.modules,
|
|
"psutil",
|
|
types.SimpleNamespace(pid_exists=lambda pid: (_ for _ in ()).throw(RuntimeError())),
|
|
)
|
|
monkeypatch.setattr(
|
|
"headroom._subprocess.os.kill",
|
|
lambda pid, sig: (_ for _ in ()).throw(OverflowError()),
|
|
)
|
|
|
|
assert pid_alive(int("9" * 100)) is False
|
|
|
|
|
|
def test_pid_alive_only_uses_signal_zero(monkeypatch) -> None:
|
|
"""The liveness probe must never send a real (terminating) signal."""
|
|
monkeypatch.setitem(
|
|
sys.modules,
|
|
"psutil",
|
|
types.SimpleNamespace(pid_exists=lambda pid: (_ for _ in ()).throw(RuntimeError())),
|
|
)
|
|
sent: list[int] = []
|
|
monkeypatch.setattr("headroom._subprocess.os.kill", lambda pid, sig: sent.append(sig))
|
|
assert pid_alive(4321) is True
|
|
assert sent == [0]
|
|
|
|
|
|
def test_pid_alive_win32_no_psutil_never_calls_os_kill(monkeypatch) -> None:
|
|
"""On Windows without psutil, pid_alive must not call os.kill (it routes through TerminateProcess)."""
|
|
monkeypatch.setitem(
|
|
sys.modules,
|
|
"psutil",
|
|
types.SimpleNamespace(pid_exists=lambda pid: (_ for _ in ()).throw(RuntimeError())),
|
|
)
|
|
monkeypatch.setattr("headroom._subprocess.sys.platform", "win32")
|
|
|
|
fake_handle = 42
|
|
opened: list[int] = []
|
|
|
|
def fake_open_process(access, inherit, pid):
|
|
opened.append(pid)
|
|
return fake_handle
|
|
|
|
closed: list[int] = []
|
|
|
|
def fake_close_handle(handle):
|
|
closed.append(handle)
|
|
|
|
fake_kernel32 = types.SimpleNamespace(
|
|
OpenProcess=fake_open_process,
|
|
CloseHandle=fake_close_handle,
|
|
)
|
|
fake_ctypes = types.SimpleNamespace(windll=types.SimpleNamespace(kernel32=fake_kernel32))
|
|
monkeypatch.setitem(sys.modules, "ctypes", fake_ctypes)
|
|
|
|
def boom(pid: int, sig: int) -> None:
|
|
raise AssertionError("os.kill must not be called on Windows")
|
|
|
|
monkeypatch.setattr("headroom._subprocess.os.kill", boom)
|
|
|
|
assert pid_alive(4321) is True
|
|
assert opened == [4321]
|
|
assert closed == [fake_handle]
|
|
|
|
|
|
def test_pid_alive_win32_no_psutil_no_ctypes_returns_conservative(monkeypatch) -> None:
|
|
"""On Windows without psutil AND ctypes failure, return True (assume alive) rather than crash."""
|
|
monkeypatch.setitem(
|
|
sys.modules,
|
|
"psutil",
|
|
types.SimpleNamespace(pid_exists=lambda pid: (_ for _ in ()).throw(RuntimeError())),
|
|
)
|
|
monkeypatch.setattr("headroom._subprocess.sys.platform", "win32")
|
|
monkeypatch.setitem(
|
|
sys.modules,
|
|
"ctypes",
|
|
types.SimpleNamespace(
|
|
windll=types.SimpleNamespace(
|
|
kernel32=types.SimpleNamespace(
|
|
OpenProcess=lambda *a: (_ for _ in ()).throw(OSError("no kernel32")),
|
|
)
|
|
)
|
|
),
|
|
)
|
|
|
|
def boom(pid: int, sig: int) -> None:
|
|
raise AssertionError("os.kill must not be called on Windows")
|
|
|
|
monkeypatch.setattr("headroom._subprocess.os.kill", boom)
|
|
|
|
assert pid_alive(4321) is True
|
|
|
|
|
|
def test_pid_alive_win32_access_denied_returns_alive(monkeypatch) -> None:
|
|
"""OpenProcess returning NULL with ERROR_ACCESS_DENIED means the process exists but is protected."""
|
|
monkeypatch.setitem(
|
|
sys.modules,
|
|
"psutil",
|
|
types.SimpleNamespace(pid_exists=lambda pid: (_ for _ in ()).throw(RuntimeError())),
|
|
)
|
|
monkeypatch.setattr("headroom._subprocess.sys.platform", "win32")
|
|
|
|
ERROR_ACCESS_DENIED = 5
|
|
|
|
fake_kernel32 = types.SimpleNamespace(
|
|
OpenProcess=lambda access, inherit, pid: 0,
|
|
GetLastError=lambda: ERROR_ACCESS_DENIED,
|
|
)
|
|
fake_ctypes = types.SimpleNamespace(windll=types.SimpleNamespace(kernel32=fake_kernel32))
|
|
monkeypatch.setitem(sys.modules, "ctypes", fake_ctypes)
|
|
|
|
def boom(pid: int, sig: int) -> None:
|
|
raise AssertionError("os.kill must not be called on Windows")
|
|
|
|
monkeypatch.setattr("headroom._subprocess.os.kill", boom)
|
|
|
|
assert pid_alive(4321) is True
|