mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description Nothing in Headroom ever stops a wrap-spawned proxy whose wrappers are gone. Two leak paths, both by-products of deliberate hardening without a counterpart: 1. **Unclean wrapper death.** `_start_proxy` detaches the proxy (`start_new_session` on POSIX, breakaway console flags on Windows) on purpose, so a crashed wrapper cannot tree-kill a shared proxy out from under other clients. But when the wrapper dies without running cleanup (SIGKILL, crash, terminal close), its client marker goes stale, `_live_proxy_clients` unlinks it, and the proxy itself is never stopped. 2. **Graceful last-exit leak.** `_make_cleanup` only terminates the proxy when the exiting wrapper *started* it (`proxy_holder[0] is not None`). When the starter exits while other clients remain (correct), every later wrapper exits with no handle, so even a perfectly behaved multi-wrap topology leaks the proxy at the end. Observed in production: three `headroom proxy` processes still alive 21 days after their last session, holding ports and being silently reused by later wraps (which then inherited their stale version and captive routing; see the two companion PRs). The fix: wrap-spawned proxies carry `HEADROOM_WRAP_OWNED=1` and run an orphan-watchdog task that polls the per-port client-marker dir every 30s and gracefully stops the process once no live markers, no active WebSocket sessions, and no served-request activity remain for a grace period (default 900s, floor 60s, `HEADROOM_WRAP_ORPHAN_GRACE_SECONDS`). Standalone `headroom proxy` instances and persistent installs never get the marker env var and are never affected; multi-worker runs are excluded (WS sessions are per-worker, and wrap never requests multiple workers). Closes #3199 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/proxy/orphan_watchdog.py` (new): env config (`WRAP_OWNED_ENV`, grace parsing with floor), marker liveness (`pid_alive` + start-time identity, pruning stale markers), and `orphan_watchdog_loop` (idle clock reset by live markers, active WS sessions, or movement of the proxy's served-request counter — the last covers direct-HTTP and SSH-forwarded clients that write no marker). Stop is `signal.raise_signal(SIGTERM)`, which reaches uvicorn's handler on Unix and the emulated handler table on Windows (unlike `os.kill(pid, SIGTERM)`, which is `TerminateProcess` there). - `headroom/proxy/server.py`: lifespan spawns the watchdog when `orphan_watchdog_enabled()` and not a multi-worker run (`HEADROOM_PROXY_CONFIG_JSON` absent), and cancels it in teardown like the other periodic tasks. - `headroom/cli/wrap.py`: `_start_proxy` sets `HEADROOM_WRAP_OWNED=1` in the proxy subprocess env; `_proc_identity`/`_identity_mismatch` moved to `headroom/_subprocess.py` (`proc_identity`/`identity_mismatch`) so proxy code does not import the CLI layer — thin module-level delegates keep the existing monkeypatch-based tests working unchanged. - `tests/test_proxy/test_orphan_watchdog.py` (new, 13 tests): env parsing (default/override/floor/garbage), marker pruning (live, dead PID, recycled PID, garbage JSON, missing dir), loop semantics (exit after grace, hold on live client, hold on active WS session, hold on served-request movement, grace reset on returning client, missing registry counts as idle). - `tests/test_cli/test_wrap_claude_vertex_proxy_env.py`: `_start_proxy` marks the subprocess env wrap-owned. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_proxy/test_orphan_watchdog.py -q 13 passed in 2.77s $ uv run pytest tests/test_proxy tests/test_cli -q 973 passed, 1 skipped in 49.85s # 1 pre-existing unrelated failure # (test_litellm_vertex_backend_path_...) fails # identically on clean upstream/main $ uv run ruff format / ruff check on touched files All checks passed! $ uv run mypy headroom/_subprocess.py headroom/cli/wrap.py headroom/proxy/orphan_watchdog.py Success: no issues found in 3 source files ``` ## Real Behavior Proof - Environment: macOS 26.5 arm64, Python 3.13, local source build 0.37.0-dev (integration overlay with this PR applied). - Exact command / steps: start wrap-spawned proxies via `wrap._ensure_proxy(...)` (which goes through `_start_proxy` and thus `HEADROOM_WRAP_OWNED=1`), with zero registered clients; wait; inspect `~/.headroom/logs/proxy.log`. - Observed result: two such proxies (plain on 8789 PID 48365, augment-mode on 8791 PID 49729) self-exited at the 900s default grace: `event=proxy_shutdown reason=signal pid=49729` at 09:51:49 against a ~09:36 start, port released afterwards (`lsof` clean). A machine-wide sweep beforehand showed three 21-day-old orphans (PIDs 73974, 63454, 79339); with this build they can no longer accumulate. - Not tested: Windows signal path (reasoned from CPython/uvicorn semantics; unit-injected `stop` callable covers the loop side); the multi-worker exclusion with a real `--workers 2` run (guarded by env presence only); marker pruning under active PID recycling on Linux without psutil (`/proc` fallback is unit-tested via mocks). ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: N/A - Stable/default behavior changed: only for wrap-spawned proxies, which now self-exit after 15 clientless minutes. Standalone and persistent proxies are untouched. - Kill switch / disable path: omit `HEADROOM_WRAP_OWNED` (not settable by users of wrap; standalone proxies never have it), or set `HEADROOM_WRAP_ORPHAN_GRACE_SECONDS` very high. - Unsafe override required: no - Qualification impact: none - Rollback path: revert the commit; orphaned proxies leak again (status quo ante). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Adversarially reviewed by a second model (glm-5.2, high reasoning); initial verdict needs-work, all accepted findings applied (multi-worker exclusion, Windows-safe stop signal, served-request activity hold for marker-less clients, wider test timing margins) and re-tested green. The reviewer independently verified the boot-race ordering, teardown cancellation, and the kill-vs-self-exit race as already correct. Companion PRs (same incident): `-dev` version-restart fix, routing-config reuse check.
83 lines
3.3 KiB
Python
83 lines
3.3 KiB
Python
"""Tests for headroom._subprocess proc_identity / identity_mismatch.
|
|
|
|
proc_identity has platform-exclusive legs (psutil vs /proc), so no single CI
|
|
runner reaches both; these tests drive each leg explicitly with fakes.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import builtins
|
|
import io
|
|
import os
|
|
import sys
|
|
from types import SimpleNamespace
|
|
|
|
import pytest
|
|
|
|
from headroom import _subprocess as sub
|
|
|
|
|
|
class TestProcIdentity:
|
|
def test_psutil_leg_with_stub(self, monkeypatch) -> None:
|
|
# Deterministic: psutil is an optional dep, so drive the leg with a stub
|
|
# instead of importorskip (line must be covered on psutil-less CI too).
|
|
stub = SimpleNamespace(Process=lambda pid: SimpleNamespace(create_time=lambda: 1234.5))
|
|
monkeypatch.setitem(sys.modules, "psutil", stub)
|
|
assert sub.proc_identity(999) == ("psutil", 1234.5)
|
|
|
|
def test_psutil_leg_real_when_installed(self) -> None:
|
|
pytest.importorskip("psutil")
|
|
ident = sub.proc_identity(os.getpid())
|
|
assert ident is not None
|
|
src, start = ident
|
|
assert src == "psutil"
|
|
assert isinstance(start, float) and start > 0
|
|
|
|
def test_proc_leg_when_psutil_unavailable(self, monkeypatch) -> None:
|
|
monkeypatch.setitem(sys.modules, "psutil", None) # import raises ImportError
|
|
# /proc/<pid>/stat: fields after the final ")" are field 3 onwards, so
|
|
# field 22 (starttime) is index 19 of the split — here 987654.
|
|
stat = b"123 (python (test)) S 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 987654 21"
|
|
real_open = builtins.open
|
|
|
|
def fake_open(path, *args, **kwargs):
|
|
if str(path).startswith("/proc/"):
|
|
return io.BytesIO(stat)
|
|
return real_open(path, *args, **kwargs)
|
|
|
|
monkeypatch.setattr(builtins, "open", fake_open)
|
|
assert sub.proc_identity(123) == ("proc", 987654.0)
|
|
|
|
def test_returns_none_when_both_sources_fail(self, monkeypatch) -> None:
|
|
monkeypatch.setitem(sys.modules, "psutil", None)
|
|
real_open = builtins.open
|
|
|
|
def fake_open(path, *args, **kwargs):
|
|
if str(path).startswith("/proc/"):
|
|
raise OSError("no /proc here")
|
|
return real_open(path, *args, **kwargs)
|
|
|
|
monkeypatch.setattr(builtins, "open", fake_open)
|
|
assert sub.proc_identity(123) is None
|
|
|
|
|
|
class TestIdentityMismatch:
|
|
def test_legacy_record_never_mismatches(self) -> None:
|
|
assert sub.identity_mismatch(None, None, os.getpid()) is False
|
|
assert sub.identity_mismatch("psutil", "not-a-number", os.getpid()) is False
|
|
|
|
def test_unknown_identity_never_mismatches(self) -> None:
|
|
assert sub.identity_mismatch("psutil", 1.0, 123, identity_fn=lambda pid: None) is False
|
|
|
|
def test_source_mismatch_never_mismatches(self) -> None:
|
|
ident = lambda pid: ("proc", 1.0) # noqa: E731
|
|
assert sub.identity_mismatch("psutil", 1.0, 123, identity_fn=ident) is False
|
|
|
|
def test_same_start_time_is_not_a_mismatch(self) -> None:
|
|
ident = lambda pid: ("psutil", 1000.5) # noqa: E731
|
|
assert sub.identity_mismatch("psutil", 1000.2, 123, identity_fn=ident) is False
|
|
|
|
def test_distant_start_time_proves_recycling(self) -> None:
|
|
ident = lambda pid: ("psutil", 2000.0) # noqa: E731
|
|
assert sub.identity_mismatch("psutil", 1000.0, 123, identity_fn=ident) is True
|