Files
headroom/tests/test_pid_alive.py
rNoz 4227bd2419 fix(proxy): reap wrap-spawned proxies once no wrap clients remain (#3202)
## Description

Nothing in Headroom ever stops a wrap-spawned proxy whose wrappers are
gone. Two leak paths, both by-products of deliberate hardening without a
counterpart:

1. **Unclean wrapper death.** `_start_proxy` detaches the proxy
(`start_new_session` on POSIX, breakaway console flags on Windows) on
purpose, so a crashed wrapper cannot tree-kill a shared proxy out from
under other clients. But when the wrapper dies without running cleanup
(SIGKILL, crash, terminal close), its client marker goes stale,
`_live_proxy_clients` unlinks it, and the proxy itself is never stopped.
2. **Graceful last-exit leak.** `_make_cleanup` only terminates the
proxy when the exiting wrapper *started* it (`proxy_holder[0] is not
None`). When the starter exits while other clients remain (correct),
every later wrapper exits with no handle, so even a perfectly behaved
multi-wrap topology leaks the proxy at the end.

Observed in production: three `headroom proxy` processes still alive 21
days after their last session, holding ports and being silently reused
by later wraps (which then inherited their stale version and captive
routing; see the two companion PRs).

The fix: wrap-spawned proxies carry `HEADROOM_WRAP_OWNED=1` and run an
orphan-watchdog task that polls the per-port client-marker dir every 30s
and gracefully stops the process once no live markers, no active
WebSocket sessions, and no served-request activity remain for a grace
period (default 900s, floor 60s, `HEADROOM_WRAP_ORPHAN_GRACE_SECONDS`).
Standalone `headroom proxy` instances and persistent installs never get
the marker env var and are never affected; multi-worker runs are
excluded (WS sessions are per-worker, and wrap never requests multiple
workers).

Closes #3199

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/proxy/orphan_watchdog.py` (new): env config
(`WRAP_OWNED_ENV`, grace parsing with floor), marker liveness
(`pid_alive` + start-time identity, pruning stale markers), and
`orphan_watchdog_loop` (idle clock reset by live markers, active WS
sessions, or movement of the proxy's served-request counter — the last
covers direct-HTTP and SSH-forwarded clients that write no marker). Stop
is `signal.raise_signal(SIGTERM)`, which reaches uvicorn's handler on
Unix and the emulated handler table on Windows (unlike `os.kill(pid,
SIGTERM)`, which is `TerminateProcess` there).
- `headroom/proxy/server.py`: lifespan spawns the watchdog when
`orphan_watchdog_enabled()` and not a multi-worker run
(`HEADROOM_PROXY_CONFIG_JSON` absent), and cancels it in teardown like
the other periodic tasks.
- `headroom/cli/wrap.py`: `_start_proxy` sets `HEADROOM_WRAP_OWNED=1` in
the proxy subprocess env; `_proc_identity`/`_identity_mismatch` moved to
`headroom/_subprocess.py` (`proc_identity`/`identity_mismatch`) so proxy
code does not import the CLI layer — thin module-level delegates keep
the existing monkeypatch-based tests working unchanged.
- `tests/test_proxy/test_orphan_watchdog.py` (new, 13 tests): env
parsing (default/override/floor/garbage), marker pruning (live, dead
PID, recycled PID, garbage JSON, missing dir), loop semantics (exit
after grace, hold on live client, hold on active WS session, hold on
served-request movement, grace reset on returning client, missing
registry counts as idle).
- `tests/test_cli/test_wrap_claude_vertex_proxy_env.py`: `_start_proxy`
marks the subprocess env wrap-owned.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_proxy/test_orphan_watchdog.py -q
13 passed in 2.77s
$ uv run pytest tests/test_proxy tests/test_cli -q
973 passed, 1 skipped in 49.85s   # 1 pre-existing unrelated failure
                                  # (test_litellm_vertex_backend_path_...) fails
                                  # identically on clean upstream/main
$ uv run ruff format / ruff check on touched files
All checks passed!
$ uv run mypy headroom/_subprocess.py headroom/cli/wrap.py headroom/proxy/orphan_watchdog.py
Success: no issues found in 3 source files
```

## Real Behavior Proof

- Environment: macOS 26.5 arm64, Python 3.13, local source build
0.37.0-dev (integration overlay with this PR applied).
- Exact command / steps: start wrap-spawned proxies via
`wrap._ensure_proxy(...)` (which goes through `_start_proxy` and thus
`HEADROOM_WRAP_OWNED=1`), with zero registered clients; wait; inspect
`~/.headroom/logs/proxy.log`.
- Observed result: two such proxies (plain on 8789 PID 48365,
augment-mode on 8791 PID 49729) self-exited at the 900s default grace:
`event=proxy_shutdown reason=signal pid=49729` at 09:51:49 against a
~09:36 start, port released afterwards (`lsof` clean). A machine-wide
sweep beforehand showed three 21-day-old orphans (PIDs 73974, 63454,
79339); with this build they can no longer accumulate.
- Not tested: Windows signal path (reasoned from CPython/uvicorn
semantics; unit-injected `stop` callable covers the loop side); the
multi-worker exclusion with a real `--workers 2` run (guarded by env
presence only); marker pruning under active PID recycling on Linux
without psutil (`/proc` fallback is unit-tested via mocks).

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: only for wrap-spawned proxies, which
now self-exit after 15 clientless minutes. Standalone and persistent
proxies are untouched.
- Kill switch / disable path: omit `HEADROOM_WRAP_OWNED` (not settable
by users of wrap; standalone proxies never have it), or set
`HEADROOM_WRAP_ORPHAN_GRACE_SECONDS` very high.
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the commit; orphaned proxies leak again (status
quo ante).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Adversarially reviewed by a second model (glm-5.2, high reasoning);
initial verdict needs-work, all accepted findings applied (multi-worker
exclusion, Windows-safe stop signal, served-request activity hold for
marker-less clients, wider test timing margins) and re-tested green. The
reviewer independently verified the boot-race ordering, teardown
cancellation, and the kill-vs-self-exit race as already correct.
Companion PRs (same incident): `-dev` version-restart fix,
routing-config reuse check.
2026-09-28 13:31:49 -05:00

157 lines
5.1 KiB
Python

"""Regression tests for the Windows-safe PID liveness helper (#1544)."""
from __future__ import annotations
import sys
import types
from headroom._subprocess import pid_alive
def test_pid_alive_rejects_non_positive() -> None:
assert pid_alive(0) is False
assert pid_alive(-1) is False
def test_pid_alive_prefers_psutil_without_signalling(monkeypatch) -> None:
monkeypatch.setitem(sys.modules, "psutil", types.SimpleNamespace(pid_exists=lambda pid: True))
def boom(pid: int, sig: int) -> None:
raise AssertionError("os.kill must not run when psutil answers")
monkeypatch.setattr("headroom._subprocess.os.kill", boom)
assert pid_alive(4321) is True
def test_pid_alive_systemerror_is_not_alive(monkeypatch) -> None:
"""WinError 87 surfaces as SystemError on Windows; it must read as 'not alive', not crash."""
monkeypatch.setitem(
sys.modules,
"psutil",
types.SimpleNamespace(pid_exists=lambda pid: (_ for _ in ()).throw(RuntimeError())),
)
monkeypatch.setattr(
"headroom._subprocess.os.kill",
lambda pid, sig: (_ for _ in ()).throw(SystemError("WinError 87")),
)
assert pid_alive(4321) is False
def test_pid_alive_overflow_is_not_alive(monkeypatch) -> None:
monkeypatch.setitem(
sys.modules,
"psutil",
types.SimpleNamespace(pid_exists=lambda pid: (_ for _ in ()).throw(RuntimeError())),
)
monkeypatch.setattr(
"headroom._subprocess.os.kill",
lambda pid, sig: (_ for _ in ()).throw(OverflowError()),
)
assert pid_alive(int("9" * 100)) is False
def test_pid_alive_only_uses_signal_zero(monkeypatch) -> None:
"""The liveness probe must never send a real (terminating) signal."""
monkeypatch.setitem(
sys.modules,
"psutil",
types.SimpleNamespace(pid_exists=lambda pid: (_ for _ in ()).throw(RuntimeError())),
)
sent: list[int] = []
monkeypatch.setattr("headroom._subprocess.os.kill", lambda pid, sig: sent.append(sig))
assert pid_alive(4321) is True
assert sent == [0]
def test_pid_alive_win32_no_psutil_never_calls_os_kill(monkeypatch) -> None:
"""On Windows without psutil, pid_alive must not call os.kill (it routes through TerminateProcess)."""
monkeypatch.setitem(
sys.modules,
"psutil",
types.SimpleNamespace(pid_exists=lambda pid: (_ for _ in ()).throw(RuntimeError())),
)
monkeypatch.setattr("headroom._subprocess.sys.platform", "win32")
fake_handle = 42
opened: list[int] = []
def fake_open_process(access, inherit, pid):
opened.append(pid)
return fake_handle
closed: list[int] = []
def fake_close_handle(handle):
closed.append(handle)
fake_kernel32 = types.SimpleNamespace(
OpenProcess=fake_open_process,
CloseHandle=fake_close_handle,
)
fake_ctypes = types.SimpleNamespace(windll=types.SimpleNamespace(kernel32=fake_kernel32))
monkeypatch.setitem(sys.modules, "ctypes", fake_ctypes)
def boom(pid: int, sig: int) -> None:
raise AssertionError("os.kill must not be called on Windows")
monkeypatch.setattr("headroom._subprocess.os.kill", boom)
assert pid_alive(4321) is True
assert opened == [4321]
assert closed == [fake_handle]
def test_pid_alive_win32_no_psutil_no_ctypes_returns_conservative(monkeypatch) -> None:
"""On Windows without psutil AND ctypes failure, return True (assume alive) rather than crash."""
monkeypatch.setitem(
sys.modules,
"psutil",
types.SimpleNamespace(pid_exists=lambda pid: (_ for _ in ()).throw(RuntimeError())),
)
monkeypatch.setattr("headroom._subprocess.sys.platform", "win32")
monkeypatch.setitem(
sys.modules,
"ctypes",
types.SimpleNamespace(
windll=types.SimpleNamespace(
kernel32=types.SimpleNamespace(
OpenProcess=lambda *a: (_ for _ in ()).throw(OSError("no kernel32")),
)
)
),
)
def boom(pid: int, sig: int) -> None:
raise AssertionError("os.kill must not be called on Windows")
monkeypatch.setattr("headroom._subprocess.os.kill", boom)
assert pid_alive(4321) is True
def test_pid_alive_win32_access_denied_returns_alive(monkeypatch) -> None:
"""OpenProcess returning NULL with ERROR_ACCESS_DENIED means the process exists but is protected."""
monkeypatch.setitem(
sys.modules,
"psutil",
types.SimpleNamespace(pid_exists=lambda pid: (_ for _ in ()).throw(RuntimeError())),
)
monkeypatch.setattr("headroom._subprocess.sys.platform", "win32")
ERROR_ACCESS_DENIED = 5
fake_kernel32 = types.SimpleNamespace(
OpenProcess=lambda access, inherit, pid: 0,
GetLastError=lambda: ERROR_ACCESS_DENIED,
)
fake_ctypes = types.SimpleNamespace(windll=types.SimpleNamespace(kernel32=fake_kernel32))
monkeypatch.setitem(sys.modules, "ctypes", fake_ctypes)
def boom(pid: int, sig: int) -> None:
raise AssertionError("os.kill must not be called on Windows")
monkeypatch.setattr("headroom._subprocess.os.kill", boom)
assert pid_alive(4321) is True