Files
headroom/tests/test_subprocess_identity.py
rNoz 4227bd2419 fix(proxy): reap wrap-spawned proxies once no wrap clients remain (#3202)
## Description

Nothing in Headroom ever stops a wrap-spawned proxy whose wrappers are
gone. Two leak paths, both by-products of deliberate hardening without a
counterpart:

1. **Unclean wrapper death.** `_start_proxy` detaches the proxy
(`start_new_session` on POSIX, breakaway console flags on Windows) on
purpose, so a crashed wrapper cannot tree-kill a shared proxy out from
under other clients. But when the wrapper dies without running cleanup
(SIGKILL, crash, terminal close), its client marker goes stale,
`_live_proxy_clients` unlinks it, and the proxy itself is never stopped.
2. **Graceful last-exit leak.** `_make_cleanup` only terminates the
proxy when the exiting wrapper *started* it (`proxy_holder[0] is not
None`). When the starter exits while other clients remain (correct),
every later wrapper exits with no handle, so even a perfectly behaved
multi-wrap topology leaks the proxy at the end.

Observed in production: three `headroom proxy` processes still alive 21
days after their last session, holding ports and being silently reused
by later wraps (which then inherited their stale version and captive
routing; see the two companion PRs).

The fix: wrap-spawned proxies carry `HEADROOM_WRAP_OWNED=1` and run an
orphan-watchdog task that polls the per-port client-marker dir every 30s
and gracefully stops the process once no live markers, no active
WebSocket sessions, and no served-request activity remain for a grace
period (default 900s, floor 60s, `HEADROOM_WRAP_ORPHAN_GRACE_SECONDS`).
Standalone `headroom proxy` instances and persistent installs never get
the marker env var and are never affected; multi-worker runs are
excluded (WS sessions are per-worker, and wrap never requests multiple
workers).

Closes #3199

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/proxy/orphan_watchdog.py` (new): env config
(`WRAP_OWNED_ENV`, grace parsing with floor), marker liveness
(`pid_alive` + start-time identity, pruning stale markers), and
`orphan_watchdog_loop` (idle clock reset by live markers, active WS
sessions, or movement of the proxy's served-request counter — the last
covers direct-HTTP and SSH-forwarded clients that write no marker). Stop
is `signal.raise_signal(SIGTERM)`, which reaches uvicorn's handler on
Unix and the emulated handler table on Windows (unlike `os.kill(pid,
SIGTERM)`, which is `TerminateProcess` there).
- `headroom/proxy/server.py`: lifespan spawns the watchdog when
`orphan_watchdog_enabled()` and not a multi-worker run
(`HEADROOM_PROXY_CONFIG_JSON` absent), and cancels it in teardown like
the other periodic tasks.
- `headroom/cli/wrap.py`: `_start_proxy` sets `HEADROOM_WRAP_OWNED=1` in
the proxy subprocess env; `_proc_identity`/`_identity_mismatch` moved to
`headroom/_subprocess.py` (`proc_identity`/`identity_mismatch`) so proxy
code does not import the CLI layer — thin module-level delegates keep
the existing monkeypatch-based tests working unchanged.
- `tests/test_proxy/test_orphan_watchdog.py` (new, 13 tests): env
parsing (default/override/floor/garbage), marker pruning (live, dead
PID, recycled PID, garbage JSON, missing dir), loop semantics (exit
after grace, hold on live client, hold on active WS session, hold on
served-request movement, grace reset on returning client, missing
registry counts as idle).
- `tests/test_cli/test_wrap_claude_vertex_proxy_env.py`: `_start_proxy`
marks the subprocess env wrap-owned.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_proxy/test_orphan_watchdog.py -q
13 passed in 2.77s
$ uv run pytest tests/test_proxy tests/test_cli -q
973 passed, 1 skipped in 49.85s   # 1 pre-existing unrelated failure
                                  # (test_litellm_vertex_backend_path_...) fails
                                  # identically on clean upstream/main
$ uv run ruff format / ruff check on touched files
All checks passed!
$ uv run mypy headroom/_subprocess.py headroom/cli/wrap.py headroom/proxy/orphan_watchdog.py
Success: no issues found in 3 source files
```

## Real Behavior Proof

- Environment: macOS 26.5 arm64, Python 3.13, local source build
0.37.0-dev (integration overlay with this PR applied).
- Exact command / steps: start wrap-spawned proxies via
`wrap._ensure_proxy(...)` (which goes through `_start_proxy` and thus
`HEADROOM_WRAP_OWNED=1`), with zero registered clients; wait; inspect
`~/.headroom/logs/proxy.log`.
- Observed result: two such proxies (plain on 8789 PID 48365,
augment-mode on 8791 PID 49729) self-exited at the 900s default grace:
`event=proxy_shutdown reason=signal pid=49729` at 09:51:49 against a
~09:36 start, port released afterwards (`lsof` clean). A machine-wide
sweep beforehand showed three 21-day-old orphans (PIDs 73974, 63454,
79339); with this build they can no longer accumulate.
- Not tested: Windows signal path (reasoned from CPython/uvicorn
semantics; unit-injected `stop` callable covers the loop side); the
multi-worker exclusion with a real `--workers 2` run (guarded by env
presence only); marker pruning under active PID recycling on Linux
without psutil (`/proc` fallback is unit-tested via mocks).

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: only for wrap-spawned proxies, which
now self-exit after 15 clientless minutes. Standalone and persistent
proxies are untouched.
- Kill switch / disable path: omit `HEADROOM_WRAP_OWNED` (not settable
by users of wrap; standalone proxies never have it), or set
`HEADROOM_WRAP_ORPHAN_GRACE_SECONDS` very high.
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the commit; orphaned proxies leak again (status
quo ante).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Adversarially reviewed by a second model (glm-5.2, high reasoning);
initial verdict needs-work, all accepted findings applied (multi-worker
exclusion, Windows-safe stop signal, served-request activity hold for
marker-less clients, wider test timing margins) and re-tested green. The
reviewer independently verified the boot-race ordering, teardown
cancellation, and the kill-vs-self-exit race as already correct.
Companion PRs (same incident): `-dev` version-restart fix,
routing-config reuse check.
2026-09-28 13:31:49 -05:00

83 lines
3.3 KiB
Python

"""Tests for headroom._subprocess proc_identity / identity_mismatch.
proc_identity has platform-exclusive legs (psutil vs /proc), so no single CI
runner reaches both; these tests drive each leg explicitly with fakes.
"""
from __future__ import annotations
import builtins
import io
import os
import sys
from types import SimpleNamespace
import pytest
from headroom import _subprocess as sub
class TestProcIdentity:
def test_psutil_leg_with_stub(self, monkeypatch) -> None:
# Deterministic: psutil is an optional dep, so drive the leg with a stub
# instead of importorskip (line must be covered on psutil-less CI too).
stub = SimpleNamespace(Process=lambda pid: SimpleNamespace(create_time=lambda: 1234.5))
monkeypatch.setitem(sys.modules, "psutil", stub)
assert sub.proc_identity(999) == ("psutil", 1234.5)
def test_psutil_leg_real_when_installed(self) -> None:
pytest.importorskip("psutil")
ident = sub.proc_identity(os.getpid())
assert ident is not None
src, start = ident
assert src == "psutil"
assert isinstance(start, float) and start > 0
def test_proc_leg_when_psutil_unavailable(self, monkeypatch) -> None:
monkeypatch.setitem(sys.modules, "psutil", None) # import raises ImportError
# /proc/<pid>/stat: fields after the final ")" are field 3 onwards, so
# field 22 (starttime) is index 19 of the split — here 987654.
stat = b"123 (python (test)) S 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 987654 21"
real_open = builtins.open
def fake_open(path, *args, **kwargs):
if str(path).startswith("/proc/"):
return io.BytesIO(stat)
return real_open(path, *args, **kwargs)
monkeypatch.setattr(builtins, "open", fake_open)
assert sub.proc_identity(123) == ("proc", 987654.0)
def test_returns_none_when_both_sources_fail(self, monkeypatch) -> None:
monkeypatch.setitem(sys.modules, "psutil", None)
real_open = builtins.open
def fake_open(path, *args, **kwargs):
if str(path).startswith("/proc/"):
raise OSError("no /proc here")
return real_open(path, *args, **kwargs)
monkeypatch.setattr(builtins, "open", fake_open)
assert sub.proc_identity(123) is None
class TestIdentityMismatch:
def test_legacy_record_never_mismatches(self) -> None:
assert sub.identity_mismatch(None, None, os.getpid()) is False
assert sub.identity_mismatch("psutil", "not-a-number", os.getpid()) is False
def test_unknown_identity_never_mismatches(self) -> None:
assert sub.identity_mismatch("psutil", 1.0, 123, identity_fn=lambda pid: None) is False
def test_source_mismatch_never_mismatches(self) -> None:
ident = lambda pid: ("proc", 1.0) # noqa: E731
assert sub.identity_mismatch("psutil", 1.0, 123, identity_fn=ident) is False
def test_same_start_time_is_not_a_mismatch(self) -> None:
ident = lambda pid: ("psutil", 1000.5) # noqa: E731
assert sub.identity_mismatch("psutil", 1000.2, 123, identity_fn=ident) is False
def test_distant_start_time_proves_recycling(self) -> None:
ident = lambda pid: ("psutil", 2000.0) # noqa: E731
assert sub.identity_mismatch("psutil", 1000.0, 123, identity_fn=ident) is True