fix(proxy): reap wrap-spawned proxies once no wrap clients remain (#3202)

## Description

Nothing in Headroom ever stops a wrap-spawned proxy whose wrappers are
gone. Two leak paths, both by-products of deliberate hardening without a
counterpart:

1. **Unclean wrapper death.** `_start_proxy` detaches the proxy
(`start_new_session` on POSIX, breakaway console flags on Windows) on
purpose, so a crashed wrapper cannot tree-kill a shared proxy out from
under other clients. But when the wrapper dies without running cleanup
(SIGKILL, crash, terminal close), its client marker goes stale,
`_live_proxy_clients` unlinks it, and the proxy itself is never stopped.
2. **Graceful last-exit leak.** `_make_cleanup` only terminates the
proxy when the exiting wrapper *started* it (`proxy_holder[0] is not
None`). When the starter exits while other clients remain (correct),
every later wrapper exits with no handle, so even a perfectly behaved
multi-wrap topology leaks the proxy at the end.

Observed in production: three `headroom proxy` processes still alive 21
days after their last session, holding ports and being silently reused
by later wraps (which then inherited their stale version and captive
routing; see the two companion PRs).

The fix: wrap-spawned proxies carry `HEADROOM_WRAP_OWNED=1` and run an
orphan-watchdog task that polls the per-port client-marker dir every 30s
and gracefully stops the process once no live markers, no active
WebSocket sessions, and no served-request activity remain for a grace
period (default 900s, floor 60s, `HEADROOM_WRAP_ORPHAN_GRACE_SECONDS`).
Standalone `headroom proxy` instances and persistent installs never get
the marker env var and are never affected; multi-worker runs are
excluded (WS sessions are per-worker, and wrap never requests multiple
workers).

Closes #3199

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/proxy/orphan_watchdog.py` (new): env config
(`WRAP_OWNED_ENV`, grace parsing with floor), marker liveness
(`pid_alive` + start-time identity, pruning stale markers), and
`orphan_watchdog_loop` (idle clock reset by live markers, active WS
sessions, or movement of the proxy's served-request counter — the last
covers direct-HTTP and SSH-forwarded clients that write no marker). Stop
is `signal.raise_signal(SIGTERM)`, which reaches uvicorn's handler on
Unix and the emulated handler table on Windows (unlike `os.kill(pid,
SIGTERM)`, which is `TerminateProcess` there).
- `headroom/proxy/server.py`: lifespan spawns the watchdog when
`orphan_watchdog_enabled()` and not a multi-worker run
(`HEADROOM_PROXY_CONFIG_JSON` absent), and cancels it in teardown like
the other periodic tasks.
- `headroom/cli/wrap.py`: `_start_proxy` sets `HEADROOM_WRAP_OWNED=1` in
the proxy subprocess env; `_proc_identity`/`_identity_mismatch` moved to
`headroom/_subprocess.py` (`proc_identity`/`identity_mismatch`) so proxy
code does not import the CLI layer — thin module-level delegates keep
the existing monkeypatch-based tests working unchanged.
- `tests/test_proxy/test_orphan_watchdog.py` (new, 13 tests): env
parsing (default/override/floor/garbage), marker pruning (live, dead
PID, recycled PID, garbage JSON, missing dir), loop semantics (exit
after grace, hold on live client, hold on active WS session, hold on
served-request movement, grace reset on returning client, missing
registry counts as idle).
- `tests/test_cli/test_wrap_claude_vertex_proxy_env.py`: `_start_proxy`
marks the subprocess env wrap-owned.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_proxy/test_orphan_watchdog.py -q
13 passed in 2.77s
$ uv run pytest tests/test_proxy tests/test_cli -q
973 passed, 1 skipped in 49.85s   # 1 pre-existing unrelated failure
                                  # (test_litellm_vertex_backend_path_...) fails
                                  # identically on clean upstream/main
$ uv run ruff format / ruff check on touched files
All checks passed!
$ uv run mypy headroom/_subprocess.py headroom/cli/wrap.py headroom/proxy/orphan_watchdog.py
Success: no issues found in 3 source files
```

## Real Behavior Proof

- Environment: macOS 26.5 arm64, Python 3.13, local source build
0.37.0-dev (integration overlay with this PR applied).
- Exact command / steps: start wrap-spawned proxies via
`wrap._ensure_proxy(...)` (which goes through `_start_proxy` and thus
`HEADROOM_WRAP_OWNED=1`), with zero registered clients; wait; inspect
`~/.headroom/logs/proxy.log`.
- Observed result: two such proxies (plain on 8789 PID 48365,
augment-mode on 8791 PID 49729) self-exited at the 900s default grace:
`event=proxy_shutdown reason=signal pid=49729` at 09:51:49 against a
~09:36 start, port released afterwards (`lsof` clean). A machine-wide
sweep beforehand showed three 21-day-old orphans (PIDs 73974, 63454,
79339); with this build they can no longer accumulate.
- Not tested: Windows signal path (reasoned from CPython/uvicorn
semantics; unit-injected `stop` callable covers the loop side); the
multi-worker exclusion with a real `--workers 2` run (guarded by env
presence only); marker pruning under active PID recycling on Linux
without psutil (`/proc` fallback is unit-tested via mocks).

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: only for wrap-spawned proxies, which
now self-exit after 15 clientless minutes. Standalone and persistent
proxies are untouched.
- Kill switch / disable path: omit `HEADROOM_WRAP_OWNED` (not settable
by users of wrap; standalone proxies never have it), or set
`HEADROOM_WRAP_ORPHAN_GRACE_SECONDS` very high.
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the commit; orphaned proxies leak again (status
quo ante).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Adversarially reviewed by a second model (glm-5.2, high reasoning);
initial verdict needs-work, all accepted findings applied (multi-worker
exclusion, Windows-safe stop signal, served-request activity hold for
marker-less clients, wider test timing margins) and re-tested green. The
reviewer independently verified the boot-race ordering, teardown
cancellation, and the kill-vs-self-exit race as already correct.
Companion PRs (same incident): `-dev` version-restart fix,
routing-config reuse check.
This commit is contained in:
rNoz
2026-09-28 13:31:49 -05:00
committed by GitHub
parent 7790bdeebd
commit 4227bd2419
10 changed files with 1282 additions and 116 deletions
+57 -1
View File
@@ -38,11 +38,67 @@ def pid_alive(pid: int) -> bool:
os.kill(pid, 0)
except PermissionError:
return True
except (ProcessLookupError, OSError, SystemError):
except (ProcessLookupError, OSError, SystemError, OverflowError, ValueError):
return False
return True
def proc_identity(pid: int) -> tuple[str, float] | None:
"""Best-effort ``(source, start_time)`` identity for a PID.
Used to defeat PID reuse: a marker is only trusted while the live PID is
*the same process* that wrote it. Returns ``None`` when start time can't be
determined (e.g. macOS without psutil), in which case callers fall back to
existence-only liveness, no regression, just no reuse protection there.
The ``source`` tag ("psutil" vs "proc") guards against comparing values in
different units; we only compare like-for-like.
"""
try:
import psutil # type: ignore[import-untyped] # optional dependency; portable when present
return ("psutil", psutil.Process(pid).create_time())
except Exception:
pass
# Linux fallback: field 22 of /proc/<pid>/stat is starttime in clock ticks
# since boot, a stable per-process value. `comm` (field 2) may contain
# spaces/parens, so split after the final ')'.
try:
with open(f"/proc/{pid}/stat", "rb") as fh:
fields = fh.read().rpartition(b")")[2].split()
return ("proc", float(fields[19]))
except (OSError, IndexError, ValueError):
return None
def identity_mismatch(
src: Any,
recorded: Any,
pid: int,
*,
identity_fn: Any = None,
) -> bool:
"""True only if ``pid``'s current identity *provably* differs from the
recorded ``(src, recorded)`` identity (i.e. the PID was recycled).
Conservative by design: any uncertainty (unknown/legacy identity, unknown
start time, mismatched source) returns ``False``, never claim a mismatch
without proof, since the caller uses this to decide whether to trust or
discard state tied to a live PID. ``identity_fn`` defaults to
:func:`proc_identity` and exists so callers with their own (mockable)
identity source can share this comparison.
"""
if identity_fn is None:
identity_fn = proc_identity
if not isinstance(src, str) or not isinstance(recorded, int | float):
return False # legacy / identity-less record, can't tell
ident = identity_fn(pid)
if ident is None or ident[0] != src:
return False # can't compare like-for-like, don't claim mismatch
# Start times are stable per process; >1s apart means a different process.
return bool(abs(ident[1] - float(recorded)) > 1.0)
def run(*args: Any, **kwargs: Any) -> _sp.CompletedProcess:
if kwargs.get("text") or kwargs.get("universal_newlines"):
kwargs.setdefault("encoding", "utf-8")
+67 -103
View File
@@ -39,7 +39,8 @@ from functools import wraps
from pathlib import Path
from typing import Any, NamedTuple, cast
from headroom._subprocess import pid_alive, run
from headroom._subprocess import identity_mismatch as _shared_identity_mismatch
from headroom._subprocess import pid_alive, proc_identity, run
# Fix Windows cp1252 encoding — box-drawing characters require UTF-8
if sys.platform == "win32" and hasattr(sys.stdout, "buffer"):
@@ -662,7 +663,16 @@ def _start_proxy(
(see ``_find_available_port``).
"""
cmd = [sys.executable, "-m", "headroom.cli", "proxy", "--port", str(port)]
cmd = [
sys.executable,
"-m",
"headroom.cli",
"proxy",
"--port",
str(port),
"--workers",
"1",
]
# Forward HEADROOM_MODE env var so the proxy respects the user's mode choice
headroom_mode = os.environ.get("HEADROOM_MODE")
@@ -711,6 +721,10 @@ def _start_proxy(
# Ensure proxy subprocess uses UTF-8 (Windows defaults to cp1252)
proxy_env = os.environ.copy()
_scrub_copilot_proxy_seed_env(proxy_env)
# A wrap-owned proxy must be a single authoritative process so its orphan
# watchdog can observe all activity before deciding to self-terminate.
proxy_env.pop("HEADROOM_WORKERS", None)
proxy_env.pop("HEADROOM_PROXY_CONFIG_JSON", None)
proxy_env["PYTHONIOENCODING"] = "utf-8"
# `python -m headroom.cli` prepends the launch cwd to sys.path, so running
# `wrap` from a directory that contains a `headroom/` folder (most commonly a
@@ -729,6 +743,12 @@ def _start_proxy(
if agent_type != "unknown":
proxy_env["HEADROOM_AGENT_TYPE"] = agent_type
proxy_env.setdefault("HEADROOM_STACK", f"wrap_{agent_type}")
# Mark the proxy as wrap-spawned so its orphan watchdog (proxy side) stops
# it once no live wrap clients remain. Standalone `headroom proxy` and
# persistent installs never get this flag and never self-exit. The proxy
# is detached (setsid / breakaway flags), so without this a crashed or
# killed wrapper leaks the proxy forever and later wraps silently reuse it.
proxy_env["HEADROOM_WRAP_OWNED"] = "1"
savings_profile = _wrap_agent_savings_profile(agent_type)
if savings_profile is not None:
apply_agent_savings_env_defaults(proxy_env, savings_profile)
@@ -4555,31 +4575,13 @@ def _client_marker_path(port: int) -> Path:
def _proc_identity(pid: int) -> tuple[str, float] | None:
"""Best-effort ``(source, start_time)`` identity for a PID.
"""Module-level alias of :func:`headroom._subprocess.proc_identity`.
Used to defeat PID reuse: a marker is only trusted while the live PID is
*the same process* that wrote it. Returns ``None`` when start time can't be
determined (e.g. macOS without psutil), in which case callers fall back to
existence-only liveness — no regression, just no reuse protection there.
The ``source`` tag ("psutil" vs "proc") guards against comparing values in
different units; we only compare like-for-like.
Kept as a function (not an assignment) so tests can monkeypatch
``wrap._proc_identity`` and have ``_identity_mismatch`` pick the patch up
through the module-global lookup below.
"""
try:
import psutil # type: ignore[import-untyped] # optional dependency; portable when present
return ("psutil", psutil.Process(pid).create_time())
except Exception:
pass
# Linux fallback: field 22 of /proc/<pid>/stat is starttime in clock ticks
# since boot — a stable per-process value. `comm` (field 2) may contain
# spaces/parens, so split after the final ')'.
try:
with open(f"/proc/{pid}/stat", "rb") as fh:
fields = fh.read().rpartition(b")")[2].split()
return ("proc", float(fields[19]))
except (OSError, IndexError, ValueError):
return None
return proc_identity(pid)
def _register_proxy_client(port: int) -> None:
@@ -4616,21 +4618,9 @@ def _pid_alive(pid: int) -> bool:
def _identity_mismatch(src: Any, recorded: Any, pid: int) -> bool:
"""True only if ``pid``'s current identity *provably* differs from the
recorded ``(src, recorded)`` identity (i.e. the PID was recycled).
Conservative by design: any uncertainty (unknown/legacy identity, unknown
start time, mismatched source) returns ``False`` — never claim a mismatch
without proof, since the caller uses this to decide whether to trust or
discard state tied to a live PID.
"""
if not isinstance(src, str) or not isinstance(recorded, int | float):
return False # legacy / identity-less record — can't tell
ident = _proc_identity(pid)
if ident is None or ident[0] != src:
return False # can't compare like-for-like — don't claim mismatch
# Start times are stable per process; >1s apart means a different process.
return abs(ident[1] - float(recorded)) > 1.0
"""Delegate to the shared comparison, resolving the identity probe through
this module so monkeypatched ``_proc_identity`` keeps working in tests."""
return _shared_identity_mismatch(src, recorded, pid, identity_fn=_proc_identity)
def _marker_pid_reused(marker: Path, pid: int) -> bool:
@@ -4641,6 +4631,8 @@ def _marker_pid_reused(marker: Path, pid: int) -> bool:
rec = json.loads(_read_text(marker))
except (OSError, ValueError):
return False
if not isinstance(rec, dict):
return False
return _identity_mismatch(rec.get("start_src"), rec.get("start_time"), pid)
@@ -4671,61 +4663,21 @@ def _live_proxy_clients(port: int, *, exclude_self: bool = True) -> list[int]:
def _make_cleanup(proxy_proc_holder: list, port: int | list[int] = 8787) -> Any:
"""Create a cleanup function that terminates the proxy on exit.
Only kills the proxy when no other live headroom-wrapped clients remain,
tracked via per-PID marker files in ``paths.proxy_clients_dir(port)``.
"""Create marker cleanup for a successfully launched wrap session.
``port`` can be an ``int`` or a ``list[int]``. When a port fallback occurs
(``_ensure_proxy`` ups the port because the requested one is busy), the
caller can update ``port[0]`` in-place and the closure picks it up.
"""
def _other_clients_exist() -> bool:
p = port[0] if isinstance(port, list) else port
return len(_live_proxy_clients(p, exclude_self=True)) > 0
Normal wrapper exit only unregisters its marker. The wrap-owned proxy's
watchdog owns final shutdown so active markerless HTTP or WebSocket traffic
receives a full grace period. Startup rollback and explicit stop commands
retain direct process termination.
"""
def cleanup(signum: int | None = None, frame: Any = None) -> None:
p = port[0] if isinstance(port, list) else port
_unregister_proxy_client(p)
proc = proxy_proc_holder[0] if proxy_proc_holder else None
if proc:
if _other_clients_exist():
# Other clients still using the proxy — leave it running.
return
# Snapshot the serving PID before terminating the launcher. On
# Windows the detached serving child can briefly make /health
# unavailable while the launcher exits, causing the later safety
# probe to classify our own listener as "unidentified" and leave
# it orphaned. We still verify it through Headroom's health
# payload before trusting the PID.
serving_pid: int | None = None
if sys.platform == "win32" and _check_proxy(p):
running_config = _query_proxy_config(p)
try:
serving_pid = int(running_config["pid"]) if running_config else None
except (KeyError, TypeError, ValueError):
serving_pid = None
if proc.poll() is None:
proc.terminate()
try:
proc.wait(timeout=5)
except subprocess.TimeoutExpired:
proc.kill()
# On Windows the proxy launcher can exit while its detached
# serving child remains alive (the native runtime uses a child
# process). The detachment is intentional so an ungraceful
# terminal close cannot disrupt other wrappers, but a graceful
# Ctrl+C from the last wrapper must still stop the listener.
if sys.platform == "win32" and _check_proxy(p):
stop_status = _stop_local_proxy_for_unwrap(p)
if stop_status == "unidentified" and serving_pid is not None:
stop_status = "stopped" if _kill_proxy_by_pid(serving_pid, p) else "failed"
if stop_status not in {"stopped", "not_running"}:
click.echo(
f" Warning: proxy on port {p} remained running "
f"after shutdown ({stop_status})."
)
return cleanup
@@ -7698,6 +7650,7 @@ def opencode(
),
)
launch_started = False
try:
# If the proxy fell back to a different port, move our marker so
# cleanup tracking stays accurate and update MCP config.
@@ -7726,24 +7679,35 @@ def opencode(
# Proxy already started by _ensure_proxy above; tell _launch_tool to
# skip duplicate startup.
_launch_tool(
binary=opencode_bin,
args=opencode_args,
env=env,
port=actual_port,
no_proxy=True,
tool_label="OPENCODE",
env_vars_display=env_vars_display,
learn=learn,
memory=memory,
agent_type="opencode",
code_graph=code_graph,
backend=backend,
anyllm_provider=anyllm_provider,
region=region,
)
launch_started = True
try:
_launch_tool(
binary=opencode_bin,
args=opencode_args,
env=env,
port=actual_port,
no_proxy=True,
tool_label="OPENCODE",
env_vars_display=env_vars_display,
learn=learn,
memory=memory,
agent_type="opencode",
code_graph=code_graph,
backend=backend,
anyllm_provider=anyllm_provider,
region=region,
)
except SystemExit as exc:
# _launch_tool raises plain SystemExit after a child exits, but
# chains pre-launch failures as the cause.
if exc.__cause__ is not None:
launch_started = False
raise
except BaseException:
launch_started = False
raise
finally:
if _opencode_proxy and _opencode_proxy.poll() is None:
if not launch_started and _opencode_proxy and _opencode_proxy.poll() is None:
_other = _live_proxy_clients(actual_port, exclude_self=True)
if not _other:
_opencode_proxy.terminate()
+223
View File
@@ -0,0 +1,223 @@
"""Orphan watchdog for wrap-spawned proxies.
``headroom wrap`` starts its proxy detached (setsid on POSIX, breakaway
console flags on Windows) on purpose: a crashing wrapper must not tree-kill a
shared proxy out from under other clients. The flip side was a leak: once
every wrapper was gone, nothing ever stopped the proxy, so clientless proxies
accumulated for weeks and later wraps silently reused whatever they found on
the port (port-hijack incident, 2026-08-22). Cleanup on the wrapper side
cannot cover unclean exits, by definition nobody is alive to run it.
When the proxy was spawned by a wrap (``HEADROOM_WRAP_OWNED=1``, set in
``cli/wrap._start_proxy``), this module's background task stops the process
once no live wrap-client markers and no active sessions remain for a grace
period. Standalone ``headroom proxy`` instances and persistent installs never
get the marker env var and are never affected.
"""
from __future__ import annotations
import asyncio
import json
import logging
import os
import signal
import time
from collections.abc import Callable, Mapping
from pathlib import Path
from typing import Any
from headroom._subprocess import identity_mismatch, pid_alive
from headroom.paths import proxy_clients_dir
logger = logging.getLogger(__name__)
WRAP_OWNED_ENV = "HEADROOM_WRAP_OWNED"
GRACE_SECONDS_ENV = "HEADROOM_WRAP_ORPHAN_GRACE_SECONDS"
DEFAULT_GRACE_SECONDS = 900.0
# Floor so a typo ("5" for "5m") cannot make a proxy exit underneath a client
# that simply has not registered its marker yet.
MIN_GRACE_SECONDS = 60.0
DEFAULT_POLL_INTERVAL_SECONDS = 30.0
def orphan_watchdog_enabled(env: Mapping[str, str] | None = None) -> bool:
"""True when this proxy process was spawned by ``headroom wrap``."""
env = os.environ if env is None else env
return (env.get(WRAP_OWNED_ENV) or "").strip().lower() in ("1", "true", "yes", "on")
def orphan_grace_seconds(env: Mapping[str, str] | None = None) -> float:
"""Grace period with zero live clients before the watchdog exits the proxy."""
env = os.environ if env is None else env
raw = env.get(GRACE_SECONDS_ENV)
if raw:
try:
return max(MIN_GRACE_SECONDS, float(raw))
except ValueError:
logger.warning(
"event=orphan_watchdog_bad_grace value=%r default=%.0f", raw, DEFAULT_GRACE_SECONDS
)
return DEFAULT_GRACE_SECONDS
def _marker_pid_recycled(marker: Path, pid: int) -> bool:
"""True only if the live ``pid`` is provably a different process than the
one that wrote ``marker`` (PID recycled after a crash)."""
try:
record = json.loads(marker.read_text(encoding="utf-8"))
except (OSError, ValueError):
return False
if not isinstance(record, dict):
return False
return identity_mismatch(record.get("start_src"), record.get("start_time"), pid)
def live_client_pids(clients_dir: Path) -> list[int] | None:
"""Live wrap-client PIDs for a clients dir, pruning stale markers as we go.
Mirrors ``cli.wrap._live_proxy_clients`` without the self-exclusion; the
proxy is never its own client.
"""
try:
with os.scandir(clients_dir) as entries:
markers = [
Path(entry.path)
for entry in entries
if entry.name.endswith(".json") and entry.is_file()
]
except FileNotFoundError:
return []
except OSError:
logger.warning(
"event=orphan_watchdog_client_scan_failed dir=%s", clients_dir, exc_info=True
)
return None
live: list[int] = []
for marker in markers:
try:
pid = int(marker.stem)
except ValueError:
continue
alive = pid_alive(pid)
if not alive or _marker_pid_recycled(marker, pid):
try:
marker.unlink(missing_ok=True)
except OSError:
pass
continue
live.append(pid)
return live
def _active_session_count(proxy: Any) -> int | None:
registry = getattr(proxy, "ws_sessions", None)
if registry is None:
return 0
try:
return int(registry.active_count())
except Exception:
logger.warning("event=orphan_watchdog_session_count_failed", exc_info=True)
return None
def _active_request_count(proxy: Any) -> int | None:
"""Return active HTTP/WebSocket scope count, or None when unobservable."""
try:
count = proxy.active_request_count
except Exception:
logger.warning("event=orphan_watchdog_request_count_failed", exc_info=True)
return None
return count if isinstance(count, int) and count >= 0 else None
def _activity_generation(proxy: Any) -> int | None:
"""Return the monotonic ASGI activity generation, or None."""
try:
generation = proxy.activity_generation
except Exception:
logger.warning("event=orphan_watchdog_activity_generation_failed", exc_info=True)
return None
return generation if isinstance(generation, int) and generation >= 0 else None
def _default_stop() -> None:
# signal.raise_signal (not os.kill): on Windows CPython maps
# os.kill(pid, SIGTERM) to TerminateProcess, a hard kill that skips
# uvicorn's handler and the lifespan teardown; raise_signal dispatches
# through the emulated handler table, so uvicorn shuts down gracefully on
# both platforms.
signal.raise_signal(signal.SIGTERM)
async def orphan_watchdog_loop(
proxy: Any,
*,
grace_seconds: float,
interval_seconds: float = DEFAULT_POLL_INTERVAL_SECONDS,
stop: Callable[[], None] = _default_stop,
monotonic: Callable[[], float] = time.monotonic,
) -> None:
"""Poll wrap-client markers; ``stop()`` once clientless for ``grace_seconds``.
The grace window covers the boot race (the wrapper registers its marker
just before spawning the proxy, and re-registers on port fallback) and
short client restarts. Active WebSocket relay sessions, in-flight HTTP
requests, and any change to the ASGI activity generation suppress shutdown
even with no live markers. The idle clock runs only while every supported
activity signal is positively observable and idle.
"""
port = proxy.config.port
clients_dir = proxy_clients_dir(port)
idle_since: float | None = None
last_activity_generation = _activity_generation(proxy)
while True:
await asyncio.sleep(interval_seconds)
clients = live_client_pids(clients_dir)
active_sessions = _active_session_count(proxy)
active_requests = _active_request_count(proxy)
activity_generation = _activity_generation(proxy)
activity_changed = (
activity_generation is not None
and last_activity_generation is not None
and activity_generation != last_activity_generation
)
if activity_generation is not None:
last_activity_generation = activity_generation
activity_unknown = (
clients is None
or active_sessions is None
or active_requests is None
or activity_generation is None
)
if (
activity_unknown
or clients
or (active_sessions is not None and active_sessions > 0)
or (active_requests is not None and active_requests > 0)
or activity_changed
):
if idle_since is not None:
logger.info(
"event=orphan_watchdog_reset port=%d live_clients=%s "
"active_sessions=%s active_requests=%s",
port,
clients,
active_sessions,
active_requests,
)
idle_since = None
continue
now = monotonic()
if idle_since is None:
idle_since = now
continue
if now - idle_since >= grace_seconds:
logger.info(
"event=orphan_watchdog_exit port=%d idle_seconds=%.0f reason=no_live_wrap_clients",
port,
now - idle_since,
)
stop()
return
+63
View File
@@ -166,6 +166,11 @@ from headroom.proxy.modes import (
is_token_mode,
normalize_proxy_mode,
)
from headroom.proxy.orphan_watchdog import (
orphan_grace_seconds,
orphan_watchdog_enabled,
orphan_watchdog_loop,
)
from headroom.proxy.probe_recorder import probe_recorder_from_env
from headroom.proxy.project_context import (
classify_project,
@@ -1275,6 +1280,8 @@ class HeadroomProxy(
# Request counter for IDs
self._request_counter = 0
self._request_counter_lock = asyncio.Lock()
self._active_requests = 0
self._activity_generation = 0
# CCR tool injectors (one per provider)
self.anthropic_tool_injector = CCRToolInjector(
@@ -2367,6 +2374,16 @@ class HeadroomProxy(
self._request_counter += 1
return f"hr_{int(time.time())}_{self._request_counter:06d}"
@property
def active_request_count(self) -> int:
"""Number of active HTTP and WebSocket ASGI scopes."""
return self._active_requests
@property
def activity_generation(self) -> int:
"""Monotonic generation advanced at request entry and completion."""
return self._activity_generation
def _extract_tags(self, headers: dict) -> dict[str, str]:
"""Backwards-compat wrapper around :func:`extract_tags`.
@@ -2890,6 +2907,27 @@ class WebSocketProjectPrefixMiddleware:
await self.app(scope, receive, send)
class ActivityMiddleware:
"""Track HTTP and WebSocket scopes through their complete ASGI lifetime."""
def __init__(self, app: Any, *, proxy: HeadroomProxy) -> None:
self.app = app
self.proxy = proxy
async def __call__(self, scope: Any, receive: Any, send: Any) -> None:
if scope["type"] not in {"http", "websocket"}:
await self.app(scope, receive, send)
return
self.proxy._active_requests += 1
self.proxy._activity_generation += 1
try:
await self.app(scope, receive, send)
finally:
self.proxy._active_requests -= 1
self.proxy._activity_generation += 1
def create_app(config: ProxyConfig | None = None) -> FastAPI:
"""Create FastAPI application."""
if not FASTAPI_AVAILABLE:
@@ -3043,6 +3081,7 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
app.state.startup_error = None
app.state.periodic_toin_stats_task = None
app.state.periodic_malloc_trim_task = None
app.state.orphan_watchdog_task = None
try:
try:
@@ -3059,6 +3098,17 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
app.state.periodic_malloc_trim_task = asyncio.create_task(
trim_periodically(config.malloc_trim_interval_seconds)
)
# Only wrap-spawned proxies carry the env marker; standalone
# `headroom proxy` and persistent installs never self-exit.
# Multi-worker runs (HEADROOM_PROXY_CONFIG_JSON is set before
# forking workers) are excluded: WS sessions are per-worker,
# so no single worker can observe "zero active sessions", and
# wrap never requests multiple workers anyway.
if orphan_watchdog_enabled() and _MULTI_WORKER_CONFIG_ENV not in os.environ:
app.state.orphan_watchdog_task = asyncio.create_task(
orphan_watchdog_loop(proxy, grace_seconds=orphan_grace_seconds()),
name="headroom-orphan-watchdog",
)
if proxy.usage_reporter:
await proxy.usage_reporter.start(proxy)
if proxy.traffic_learner:
@@ -3128,6 +3178,16 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
)
app.state.periodic_malloc_trim_task = None
orphan_watchdog_task = app.state.orphan_watchdog_task
if orphan_watchdog_task is not None:
orphan_watchdog_task.cancel()
await _timed(
asyncio.gather(orphan_watchdog_task, return_exceptions=True),
label="orphan_watchdog.stop",
timeout=3.0,
)
app.state.orphan_watchdog_task = None
if _cc_reconciler is not None:
await _timed(_cc_reconciler.stop(), label="cc_reconciler.stop", timeout=3.0)
if _beacon_is_owner[0]:
@@ -5681,6 +5741,9 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
_install_gateway_contract(app, config, route_dependencies=_compress_dependencies)
register_provider_routes(app, proxy)
# Register last so this raw ASGI middleware is outermost and covers the
# complete response body, including responses produced by other middleware.
app.add_middleware(ActivityMiddleware, proxy=proxy)
return app
@@ -460,6 +460,29 @@ def test_start_proxy_sets_vertex_target_env_for_proxy_subprocess(
assert proxy_env["VERTEX_TARGET_API_URL"] == "https://vertex-gateway.internal/custom"
def test_start_proxy_marks_subprocess_as_wrap_owned(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""Wrap-spawned proxies carry HEADROOM_WRAP_OWNED=1 for the orphan watchdog."""
fake_proc = _FakeProxyProcess()
captured: dict[str, Any] = {}
monkeypatch.setattr(wrap_mod, "_get_log_path", lambda port=None: tmp_path / "proxy.log")
monkeypatch.setattr(wrap_mod, "_check_proxy", lambda _port: True)
monkeypatch.setattr(wrap_mod.time, "sleep", lambda _seconds: None)
def fake_popen(cmd: list[str], **kwargs: object) -> _FakeProxyProcess:
captured["kwargs"] = kwargs
return fake_proc
monkeypatch.setattr(wrap_mod.subprocess, "Popen", fake_popen)
proc = wrap_mod._start_proxy(8787, agent_type="codex")
assert proc is fake_proc
assert captured["kwargs"]["env"]["HEADROOM_WRAP_OWNED"] == "1"
def test_start_proxy_clears_inherited_vertex_target_env(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
+91 -12
View File
@@ -533,6 +533,62 @@ class _FakeProxyProc:
self.killed = True
def test_start_proxy_strips_ambient_worker_configuration(
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
) -> None:
monkeypatch.setenv("HEADROOM_WORKERS", "4")
monkeypatch.setenv("HEADROOM_PROXY_CONFIG_JSON", '{"port": 9999}')
monkeypatch.setenv("CLAUDE_CODE_USE_VERTEX", "1")
captured: dict[str, object] = {}
proc = _FakeProxyProc()
def fake_popen(command: list[str], **kwargs: object) -> _FakeProxyProc:
captured["command"] = command
captured["env"] = kwargs["env"]
return proc
monkeypatch.setattr(wrap_mod.subprocess, "Popen", fake_popen)
monkeypatch.setattr(wrap_mod, "_check_proxy", lambda port: True)
monkeypatch.setattr(wrap_mod, "_get_log_path", lambda: tmp_path / "proxy.log")
monkeypatch.setattr(
wrap_mod,
"_get_proxy_stdio_log_path",
lambda: tmp_path / "proxy-stdio.log",
)
monkeypatch.setattr(wrap_mod.time, "sleep", lambda seconds: None)
assert wrap_mod._start_proxy(8787) is proc
env = captured["env"]
assert isinstance(env, dict)
assert "HEADROOM_WORKERS" not in env
assert "HEADROOM_PROXY_CONFIG_JSON" not in env
assert env["HEADROOM_HTTP2"] == "false"
assert captured["command"][-2:] == ["--workers", "1"]
def test_start_proxy_timeout_kills_failed_new_process(
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
) -> None:
proc = _FakeProxyProc()
monkeypatch.setattr(wrap_mod.subprocess, "Popen", lambda *args, **kwargs: proc)
monkeypatch.setattr(wrap_mod, "_check_proxy", lambda port: False)
monkeypatch.setattr(wrap_mod, "_get_log_path", lambda: tmp_path / "proxy.log")
monkeypatch.setattr(
wrap_mod,
"_get_proxy_stdio_log_path",
lambda: tmp_path / "proxy-stdio.log",
)
monkeypatch.setattr(wrap_mod, "_resolve_wrap_proxy_timeout_seconds", lambda: 1)
monkeypatch.setattr(wrap_mod.time, "sleep", lambda seconds: None)
with pytest.raises(RuntimeError, match="failed to start"):
wrap_mod._start_proxy(8787)
assert proc.killed is True
class TestProxyClientRefCounting:
"""Proxy lifecycle is reference-counted via marker files, not pgrep."""
@@ -560,22 +616,21 @@ class TestProxyClientRefCounting:
marker.write_text(json.dumps(rec))
return marker
def test_cleanup_terminates_proxy_when_only_self_registered(self, clients_dir: Path) -> None:
"""The owner alone → no other clients → proxy is terminated on exit."""
def test_cleanup_unregisters_marker_without_terminating_proxy(self, clients_dir: Path) -> None:
"""Normal exit transfers final shutdown ownership to the watchdog."""
wrap_mod._register_proxy_client(self.PORT)
proc = _FakeProxyProc()
cleanup = wrap_mod._make_cleanup([proc], self.PORT)
cleanup()
assert proc.terminated is True
# Our own marker is removed before we count.
assert proc.terminated is False
assert wrap_mod._live_proxy_clients(self.PORT, exclude_self=False) == []
def test_cleanup_stops_detached_windows_serving_child(
def test_cleanup_leaves_detached_windows_serving_child_to_watchdog(
self, clients_dir: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Ctrl+C must stop the listener even when its launcher already exited."""
"""Windows cleanup follows the same marker-only ownership transfer."""
wrap_mod._register_proxy_client(self.PORT)
proc = _FakeProxyProc()
proc.poll = lambda: 0 # type: ignore[method-assign]
@@ -592,7 +647,7 @@ class TestProxyClientRefCounting:
wrap_mod._make_cleanup([proc], self.PORT)()
assert not proc.terminated
assert stopped == [self.PORT]
assert stopped == []
def test_kill_proxy_uses_taskkill_tree_on_windows(
self, monkeypatch: pytest.MonkeyPatch
@@ -617,10 +672,10 @@ class TestProxyClientRefCounting:
)
]
def test_cleanup_uses_pre_shutdown_pid_when_health_probe_races(
def test_cleanup_does_not_probe_or_kill_windows_serving_child(
self, clients_dir: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A transient post-terminate /health miss must not orphan the listener."""
"""The watchdog, not wrapper cleanup, owns normal listener shutdown."""
wrap_mod._register_proxy_client(self.PORT)
proc = _FakeProxyProc()
killed: list[tuple[int, int]] = []
@@ -636,8 +691,8 @@ class TestProxyClientRefCounting:
wrap_mod._make_cleanup([proc], self.PORT)()
assert proc.terminated
assert killed == [(456, self.PORT)]
assert not proc.terminated
assert killed == []
def test_cleanup_leaves_proxy_running_when_other_client_alive(self, clients_dir: Path) -> None:
"""A second live client (here: the test's parent) keeps the proxy up."""
@@ -665,6 +720,23 @@ class TestProxyClientRefCounting:
assert dead_pid not in live
assert not marker.exists()
def test_dead_client_marker_unlink_failure_is_tolerated(
self,
clients_dir: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
dead_pid = 358784
marker = self._write_marker(clients_dir, dead_pid)
monkeypatch.setattr(wrap_mod, "_pid_alive", lambda pid: pid != dead_pid)
def fail_unlink(*args: object, **kwargs: object) -> None:
raise OSError("read-only")
monkeypatch.setattr(Path, "unlink", fail_unlink)
assert wrap_mod._live_proxy_clients(self.PORT, exclude_self=True) == []
assert marker.exists()
def test_reused_pid_with_mismatched_identity_is_pruned(
self, clients_dir: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
@@ -719,6 +791,13 @@ class TestProxyClientRefCounting:
assert wrap_mod._live_proxy_clients(self.PORT, exclude_self=True) == []
def test_non_dict_marker_is_tolerated(self, clients_dir: Path) -> None:
live_pid = os.getppid()
marker = self._write_marker(clients_dir, live_pid)
marker.write_text("[]", encoding="utf-8")
assert wrap_mod._live_proxy_clients(self.PORT, exclude_self=True) == [live_pid]
def test_cleanup_does_not_shell_out_to_pgrep(
self, clients_dir: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
@@ -739,7 +818,7 @@ class TestProxyClientRefCounting:
cleanup = wrap_mod._make_cleanup([proc], self.PORT)
cleanup() # must not raise
assert proc.terminated is True
assert proc.terminated is False
def test_register_then_unregister_is_idempotent(self, clients_dir: Path) -> None:
"""Register adds exactly our marker; unregister removes it; re-call is safe."""
+92
View File
@@ -293,6 +293,98 @@ def test_wrap_opencode_copilot_subscription_cleans_up_proxy_on_config_failure(
assert proxy.wait_timeout == 5
@pytest.mark.parametrize("failure", ["spawn", "interrupt"])
def test_wrap_opencode_cleans_up_proxy_when_tool_launch_fails(
runner: CliRunner,
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
failure: str,
) -> None:
monkeypatch.chdir(tmp_path)
_set_test_home(monkeypatch, tmp_path)
class _FakeProxy:
def __init__(self) -> None:
self.terminated = False
def poll(self) -> None:
return None
def terminate(self) -> None:
self.terminated = True
def wait(self, timeout: float | None = None) -> int:
return 0
proxy = _FakeProxy()
def fail_before_spawn(**kwargs: object) -> None:
if failure == "interrupt":
raise KeyboardInterrupt
try:
raise OSError("spawn failed")
except OSError as error:
raise SystemExit(1) from error
with (
patch.object(wrap_mod.shutil, "which", return_value="opencode"),
patch.object(wrap_mod, "_ensure_proxy", return_value=(proxy, 9010)),
patch.object(wrap_mod, "_register_proxy_client"),
patch.object(wrap_mod, "_unregister_proxy_client"),
patch.object(wrap_mod, "_live_proxy_clients", return_value=[]),
patch.object(wrap_mod, "inject_opencode_provider_config"),
patch.object(wrap_mod, "_launch_tool", side_effect=fail_before_spawn),
):
result = runner.invoke(
main,
["wrap", "opencode", "--no-mcp", "--no-serena"],
)
assert result.exit_code == 1
assert proxy.terminated is True
def test_wrap_opencode_leaves_started_proxy_to_watchdog_after_normal_exit(
runner: CliRunner,
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.chdir(tmp_path)
_set_test_home(monkeypatch, tmp_path)
class _FakeProxy:
def __init__(self) -> None:
self.terminated = False
def poll(self) -> None:
return None
def terminate(self) -> None:
self.terminated = True
def wait(self, timeout: float | None = None) -> int:
return 0
proxy = _FakeProxy()
with (
patch.object(wrap_mod.shutil, "which", return_value="opencode"),
patch.object(wrap_mod, "_ensure_proxy", return_value=(proxy, 9010)),
patch.object(wrap_mod, "_register_proxy_client"),
patch.object(wrap_mod, "_unregister_proxy_client"),
patch.object(wrap_mod, "_live_proxy_clients", return_value=[]),
patch.object(wrap_mod, "inject_opencode_provider_config"),
patch.object(wrap_mod, "_launch_tool", side_effect=SystemExit(0)),
):
result = runner.invoke(
main,
["wrap", "opencode", "--no-mcp", "--no-serena"],
)
assert result.exit_code == 0, result.output
assert proxy.terminated is False
def test_wrap_opencode_sets_config_content_env(
runner: CliRunner,
tmp_path: Path,
+14
View File
@@ -37,6 +37,20 @@ def test_pid_alive_systemerror_is_not_alive(monkeypatch) -> None:
assert pid_alive(4321) is False
def test_pid_alive_overflow_is_not_alive(monkeypatch) -> None:
monkeypatch.setitem(
sys.modules,
"psutil",
types.SimpleNamespace(pid_exists=lambda pid: (_ for _ in ()).throw(RuntimeError())),
)
monkeypatch.setattr(
"headroom._subprocess.os.kill",
lambda pid, sig: (_ for _ in ()).throw(OverflowError()),
)
assert pid_alive(int("9" * 100)) is False
def test_pid_alive_only_uses_signal_zero(monkeypatch) -> None:
"""The liveness probe must never send a real (terminating) signal."""
monkeypatch.setitem(
+570
View File
@@ -0,0 +1,570 @@
"""Tests for the wrap-spawned proxy orphan watchdog."""
from __future__ import annotations
import asyncio
import json
import os
from pathlib import Path
from types import SimpleNamespace
import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
from httpx import ASGITransport, AsyncClient
from starlette.responses import StreamingResponse
import headroom.proxy.server as server
from headroom.proxy import orphan_watchdog as ow
from headroom.proxy.server import ProxyConfig
def _write_marker(clients_dir: Path, pid: int, **extra: object) -> Path:
clients_dir.mkdir(parents=True, exist_ok=True)
marker = clients_dir / f"{pid}.json"
payload = {"pid": pid, "started_at": 1.0}
payload.update(extra)
marker.write_text(json.dumps(payload), encoding="utf-8")
return marker
class _Proxy:
def __init__(self, *, port: int = 8787, active_sessions: int = 0) -> None:
self.config = SimpleNamespace(port=port)
self.ws_sessions = SimpleNamespace(active_count=lambda: active_sessions)
self._active_requests = 0
self._activity_generation = 0
@property
def active_request_count(self) -> int:
return self._active_requests
@property
def activity_generation(self) -> int:
return self._activity_generation
class TestEnvConfig:
def test_enabled_requires_wrap_owned_flag(self, monkeypatch) -> None:
monkeypatch.delenv(ow.WRAP_OWNED_ENV, raising=False)
assert ow.orphan_watchdog_enabled() is False
monkeypatch.setenv(ow.WRAP_OWNED_ENV, "1")
assert ow.orphan_watchdog_enabled() is True
monkeypatch.setenv(ow.WRAP_OWNED_ENV, "0")
assert ow.orphan_watchdog_enabled() is False
# Explicit mapping wins over os.environ.
assert ow.orphan_watchdog_enabled({ow.WRAP_OWNED_ENV: "true"}) is True
def test_grace_seconds_default_override_and_floor(self, monkeypatch) -> None:
monkeypatch.delenv(ow.GRACE_SECONDS_ENV, raising=False)
assert ow.orphan_grace_seconds() == ow.DEFAULT_GRACE_SECONDS
monkeypatch.setenv(ow.GRACE_SECONDS_ENV, "120")
assert ow.orphan_grace_seconds() == 120.0
# Below the floor: clamped, so a typo cannot exit a proxy under a
# client that has not registered yet.
monkeypatch.setenv(ow.GRACE_SECONDS_ENV, "5")
assert ow.orphan_grace_seconds() == ow.MIN_GRACE_SECONDS
monkeypatch.setenv(ow.GRACE_SECONDS_ENV, "not-a-number")
assert ow.orphan_grace_seconds() == ow.DEFAULT_GRACE_SECONDS
class TestLiveClientPids:
def test_live_marker_is_kept(self, tmp_path) -> None:
_write_marker(tmp_path, os.getpid())
assert ow.live_client_pids(tmp_path) == [os.getpid()]
def test_dead_pid_marker_is_pruned(self, tmp_path) -> None:
# PID 2**31 - 1 is never alive.
marker = _write_marker(tmp_path, 2**31 - 1)
assert ow.live_client_pids(tmp_path) == []
assert not marker.exists()
def test_recycled_pid_marker_is_pruned(self, tmp_path, monkeypatch) -> None:
# Live PID (ours) but a recorded identity that provably differs.
monkeypatch.setattr(ow, "identity_mismatch", lambda src, recorded, pid: True)
marker = _write_marker(tmp_path, os.getpid(), start_src="psutil", start_time=1.0)
assert ow.live_client_pids(tmp_path) == []
assert not marker.exists()
def test_garbage_and_non_numeric_markers_are_ignored(self, tmp_path) -> None:
(tmp_path / "not-a-pid.json").write_text("{}", encoding="utf-8")
(tmp_path / f"{os.getpid()}.json").write_text("not json", encoding="utf-8")
# Unparseable marker JSON is not proof of recycling: PID is alive, kept.
assert ow.live_client_pids(tmp_path) == [os.getpid()]
def test_oversized_numeric_marker_is_pruned(self, tmp_path) -> None:
marker = _write_marker(tmp_path, int("9" * 100))
assert ow.live_client_pids(tmp_path) == []
assert not marker.exists()
def test_non_dict_marker_is_not_recycling_proof(self, tmp_path) -> None:
(tmp_path / f"{os.getpid()}.json").write_text("[]", encoding="utf-8")
assert ow.live_client_pids(tmp_path) == [os.getpid()]
def test_missing_dir_is_empty(self, tmp_path) -> None:
assert ow.live_client_pids(tmp_path / "nope") == []
def test_client_scan_error_is_unknown(self, tmp_path, monkeypatch) -> None:
tmp_path.mkdir(exist_ok=True)
def fail_scandir(path: object): # type: ignore[no-untyped-def]
raise PermissionError("scan unavailable")
monkeypatch.setattr(ow.os, "scandir", fail_scandir)
assert ow.live_client_pids(tmp_path) is None
class TestWatchdogLoop:
@staticmethod
def _proxy(port: int = 8787, active_sessions: int = 0) -> _Proxy:
return _Proxy(port=port, active_sessions=active_sessions)
def test_stops_after_grace_with_no_clients(self, tmp_path, monkeypatch) -> None:
monkeypatch.setattr(ow, "proxy_clients_dir", lambda port: tmp_path)
stopped: list[bool] = []
asyncio.run(
ow.orphan_watchdog_loop(
self._proxy(),
grace_seconds=0.05,
interval_seconds=0.01,
stop=lambda: stopped.append(True),
)
)
assert stopped == [True]
def test_no_stop_while_client_alive(self, tmp_path, monkeypatch) -> None:
monkeypatch.setattr(ow, "proxy_clients_dir", lambda port: tmp_path)
_write_marker(tmp_path, os.getpid())
stopped: list[bool] = []
async def run() -> None:
task = asyncio.create_task(
ow.orphan_watchdog_loop(
self._proxy(),
grace_seconds=0.3,
interval_seconds=0.01,
stop=lambda: stopped.append(True),
)
)
await asyncio.sleep(0.4)
task.cancel()
with pytest.raises(asyncio.CancelledError):
await task
asyncio.run(run())
assert stopped == []
def test_no_stop_while_ws_session_active(self, tmp_path, monkeypatch) -> None:
monkeypatch.setattr(ow, "proxy_clients_dir", lambda port: tmp_path)
stopped: list[bool] = []
async def run() -> None:
task = asyncio.create_task(
ow.orphan_watchdog_loop(
self._proxy(active_sessions=1),
grace_seconds=0.3,
interval_seconds=0.01,
stop=lambda: stopped.append(True),
)
)
await asyncio.sleep(0.4)
task.cancel()
with pytest.raises(asyncio.CancelledError):
await task
asyncio.run(run())
assert stopped == []
def test_no_stop_while_request_activity_continues(self, tmp_path, monkeypatch) -> None:
"""Direct-HTTP / SSH-forwarded clients leave no marker and may hold no
WS session; activity generation movement must still hold the exit."""
monkeypatch.setattr(ow, "proxy_clients_dir", lambda port: tmp_path)
counter = {"n": 0}
proxy = self._proxy()
stopped: list[bool] = []
async def serve_traffic() -> None:
while True:
await asyncio.sleep(0.05)
counter["n"] += 1
proxy._activity_generation = counter["n"]
async def run() -> None:
traffic = asyncio.create_task(serve_traffic())
task = asyncio.create_task(
ow.orphan_watchdog_loop(
proxy,
grace_seconds=0.15,
interval_seconds=0.01,
stop=lambda: stopped.append(True),
)
)
await asyncio.sleep(0.4)
task.cancel()
traffic.cancel()
with pytest.raises(asyncio.CancelledError):
await task
asyncio.run(run())
assert stopped == []
def test_streaming_request_resets_full_grace_after_completion(
self, tmp_path, monkeypatch
) -> None:
monkeypatch.setattr(ow, "proxy_clients_dir", lambda port: tmp_path)
proxy = self._proxy()
app = FastAPI()
app.add_middleware(server.ActivityMiddleware, proxy=proxy)
stream_started = asyncio.Event()
release_stream = asyncio.Event()
stopped: list[bool] = []
@app.get("/stream")
async def stream() -> StreamingResponse:
async def body(): # type: ignore[no-untyped-def]
stream_started.set()
yield b"started"
await release_stream.wait()
yield b"finished"
return StreamingResponse(body())
async def run() -> None:
transport = ASGITransport(app=app)
async with AsyncClient(transport=transport, base_url="http://test") as client:
watchdog = asyncio.create_task(
ow.orphan_watchdog_loop(
proxy,
grace_seconds=0.08,
interval_seconds=0.01,
stop=lambda: stopped.append(True),
)
)
request = asyncio.create_task(client.get("/stream"))
await asyncio.wait_for(stream_started.wait(), timeout=1)
await asyncio.sleep(0.12)
assert stopped == []
assert proxy.active_request_count == 1
release_stream.set()
await request
await asyncio.sleep(0.04)
assert stopped == []
await asyncio.wait_for(watchdog, timeout=0.2)
asyncio.run(run())
assert stopped == [True]
def test_short_http_pulse_restarts_full_grace(self, tmp_path, monkeypatch) -> None:
monkeypatch.setattr(ow, "proxy_clients_dir", lambda port: tmp_path)
proxy = self._proxy()
app = FastAPI()
app.add_middleware(server.ActivityMiddleware, proxy=proxy)
stopped: list[bool] = []
@app.get("/pulse")
async def pulse() -> dict[str, bool]:
return {"ok": True}
async def run() -> None:
transport = ASGITransport(app=app)
async with AsyncClient(transport=transport, base_url="http://test") as client:
watchdog = asyncio.create_task(
ow.orphan_watchdog_loop(
proxy,
grace_seconds=0.08,
interval_seconds=0.02,
stop=lambda: stopped.append(True),
)
)
await asyncio.sleep(0.05)
before = proxy.activity_generation
response = await client.get("/pulse")
assert response.status_code == 200
assert proxy.active_request_count == 0
assert proxy.activity_generation == before + 2
# The pre-pulse idle window would have expired by now. The
# observed generation change must start a full new window.
await asyncio.sleep(0.06)
assert stopped == []
await asyncio.wait_for(watchdog, timeout=0.2)
asyncio.run(run())
assert stopped == [True]
def test_activity_middleware_tracks_held_websocket_scope(self, tmp_path, monkeypatch) -> None:
monkeypatch.setattr(ow, "proxy_clients_dir", lambda port: tmp_path)
proxy = self._proxy()
entered = asyncio.Event()
release = asyncio.Event()
stopped: list[bool] = []
async def held_app(scope, receive, send): # type: ignore[no-untyped-def]
entered.set()
await release.wait()
async def unused(): # type: ignore[no-untyped-def]
return {}
async def run() -> None:
middleware = server.ActivityMiddleware(held_app, proxy=proxy)
watchdog = asyncio.create_task(
ow.orphan_watchdog_loop(
proxy,
grace_seconds=0.05,
interval_seconds=0.01,
stop=lambda: stopped.append(True),
)
)
task = asyncio.create_task(middleware({"type": "websocket"}, unused, unused))
await asyncio.wait_for(entered.wait(), timeout=1)
assert proxy.active_request_count == 1
assert proxy.activity_generation == 1
await asyncio.sleep(0.08)
assert stopped == []
release.set()
await task
assert proxy.active_request_count == 0
assert proxy.activity_generation == 2
await asyncio.wait_for(watchdog, timeout=0.2)
asyncio.run(run())
assert stopped == [True]
def test_registry_failure_resets_grace_until_activity_is_observable(
self, tmp_path, monkeypatch
) -> None:
monkeypatch.setattr(ow, "proxy_clients_dir", lambda port: tmp_path)
observable = False
def active_count() -> int:
if not observable:
raise RuntimeError("registry unavailable")
return 0
proxy = self._proxy()
proxy.ws_sessions = SimpleNamespace(active_count=active_count)
stopped: list[bool] = []
async def run() -> None:
nonlocal observable
watchdog = asyncio.create_task(
ow.orphan_watchdog_loop(
proxy,
grace_seconds=0.08,
interval_seconds=0.01,
stop=lambda: stopped.append(True),
)
)
await asyncio.sleep(0.12)
assert stopped == []
observable = True
await asyncio.sleep(0.04)
assert stopped == []
await asyncio.wait_for(watchdog, timeout=0.2)
asyncio.run(run())
assert stopped == [True]
@pytest.mark.parametrize("unknown_signal", ["clients", "activity", "generation"])
def test_unknown_activity_signal_resets_grace_until_observable(
self, unknown_signal, tmp_path, monkeypatch
) -> None:
observable = False
proxy = self._proxy()
if unknown_signal == "clients":
monkeypatch.setattr(
ow,
"live_client_pids",
lambda clients_dir: [] if observable else None,
)
elif unknown_signal == "activity":
monkeypatch.setattr(ow, "proxy_clients_dir", lambda port: tmp_path)
proxy._active_requests = None
else:
monkeypatch.setattr(ow, "proxy_clients_dir", lambda port: tmp_path)
proxy._activity_generation = None
stopped: list[bool] = []
async def run() -> None:
nonlocal observable
watchdog = asyncio.create_task(
ow.orphan_watchdog_loop(
proxy,
grace_seconds=0.08,
interval_seconds=0.01,
stop=lambda: stopped.append(True),
)
)
await asyncio.sleep(0.12)
assert stopped == []
observable = True
proxy._active_requests = 0
proxy._activity_generation = 0
await asyncio.sleep(0.04)
assert stopped == []
await asyncio.wait_for(watchdog, timeout=0.2)
asyncio.run(run())
assert stopped == [True]
def test_grace_restarts_when_client_returns(self, tmp_path, monkeypatch) -> None:
monkeypatch.setattr(ow, "proxy_clients_dir", lambda port: tmp_path)
stopped: list[bool] = []
async def run() -> None:
task = asyncio.create_task(
ow.orphan_watchdog_loop(
self._proxy(),
grace_seconds=0.5,
interval_seconds=0.01,
stop=lambda: stopped.append(True),
)
)
# Let the idle clock accumulate, then register a live client: the
# clock must reset, not fire late.
await asyncio.sleep(0.2)
_write_marker(tmp_path, os.getpid())
await asyncio.sleep(0.7)
task.cancel()
with pytest.raises(asyncio.CancelledError):
await task
asyncio.run(run())
assert stopped == []
def test_missing_ws_registry_counts_as_idle(self, tmp_path, monkeypatch) -> None:
monkeypatch.setattr(ow, "proxy_clients_dir", lambda port: tmp_path)
proxy = self._proxy()
proxy.ws_sessions = None
stopped: list[bool] = []
asyncio.run(
ow.orphan_watchdog_loop(
proxy,
grace_seconds=0.05,
interval_seconds=0.01,
stop=lambda: stopped.append(True),
)
)
assert stopped == [True]
class TestDefensiveHelpers:
def test_non_dict_marker_json_is_not_recycling_proof(self, tmp_path) -> None:
# Valid JSON but not an object: only a dict record can prove recycling.
marker = _write_marker(tmp_path, os.getpid())
marker.write_text("[1, 2]", encoding="utf-8")
assert ow._marker_pid_recycled(marker, os.getpid()) is False
def test_unlink_oserror_is_tolerated(self, tmp_path, monkeypatch) -> None:
dead_pid = 2**31 - 1 # never alive
marker = _write_marker(tmp_path, dead_pid)
real_unlink = Path.unlink
def flaky_unlink(self: Path, *args: object, **kwargs: object) -> None:
if self == marker:
raise OSError("simulated EPERM")
real_unlink(self, *args, **kwargs)
monkeypatch.setattr(Path, "unlink", flaky_unlink)
# The scan tolerates the failed prune and still reports no live clients.
assert ow.live_client_pids(tmp_path) == []
assert marker.exists()
def test_active_session_count_reports_registry_errors_as_unknown(self) -> None:
def boom() -> int:
raise RuntimeError("registry exploded")
proxy = SimpleNamespace(ws_sessions=SimpleNamespace(active_count=boom))
assert ow._active_session_count(proxy) is None
def test_active_request_count_is_unknown_when_unavailable(self) -> None:
class BrokenProxy:
@property
def active_request_count(self) -> int:
raise RuntimeError("counter unavailable")
assert ow._active_request_count(SimpleNamespace()) is None
assert ow._active_request_count(BrokenProxy()) is None
def test_activity_generation_is_unknown_when_unavailable(self) -> None:
assert ow._activity_generation(SimpleNamespace()) is None
def test_default_stop_raises_sigterm(self, monkeypatch) -> None:
sent: list[int] = []
monkeypatch.setattr(ow.signal, "raise_signal", lambda sig: sent.append(sig))
ow._default_stop()
assert sent == [ow.signal.SIGTERM]
class TestServerWiring:
"""create_app must start the watchdog only for wrap-spawned single-worker
proxies, and cancel it cleanly on shutdown."""
@staticmethod
def _config() -> ProxyConfig:
return ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
periodic_malloc_trim_enabled=False,
)
@staticmethod
async def _fake_loop(*args: object, **kwargs: object) -> None:
await asyncio.sleep(3600)
def test_starts_and_stops_watchdog_when_wrap_owned(self, monkeypatch) -> None:
monkeypatch.setenv(ow.WRAP_OWNED_ENV, "1")
monkeypatch.delenv(server._MULTI_WORKER_CONFIG_ENV, raising=False)
monkeypatch.setattr(server, "orphan_watchdog_loop", self._fake_loop)
app = server.create_app(self._config())
with TestClient(app) as client:
task = client.app.state.orphan_watchdog_task
assert task is not None
assert not task.done()
# Lifespan teardown cancelled it and cleared the state slot.
assert app.state.orphan_watchdog_task is None
def test_activity_middleware_is_outermost(self) -> None:
app = server.create_app(self._config())
assert app.user_middleware[0].cls is server.ActivityMiddleware
assert app.state.proxy.active_request_count == 0
assert app.state.proxy.activity_generation == 0
def test_skips_watchdog_for_multi_worker(self, monkeypatch) -> None:
monkeypatch.setenv(ow.WRAP_OWNED_ENV, "1")
monkeypatch.setenv(server._MULTI_WORKER_CONFIG_ENV, "{}")
monkeypatch.setattr(server, "orphan_watchdog_loop", self._fake_loop)
app = server.create_app(self._config())
with TestClient(app) as client:
assert client.app.state.orphan_watchdog_task is None
def test_skips_watchdog_when_not_wrap_owned(self, monkeypatch) -> None:
monkeypatch.delenv(ow.WRAP_OWNED_ENV, raising=False)
monkeypatch.delenv(server._MULTI_WORKER_CONFIG_ENV, raising=False)
monkeypatch.setattr(server, "orphan_watchdog_loop", self._fake_loop)
app = server.create_app(self._config())
with TestClient(app) as client:
assert client.app.state.orphan_watchdog_task is None
+82
View File
@@ -0,0 +1,82 @@
"""Tests for headroom._subprocess proc_identity / identity_mismatch.
proc_identity has platform-exclusive legs (psutil vs /proc), so no single CI
runner reaches both; these tests drive each leg explicitly with fakes.
"""
from __future__ import annotations
import builtins
import io
import os
import sys
from types import SimpleNamespace
import pytest
from headroom import _subprocess as sub
class TestProcIdentity:
def test_psutil_leg_with_stub(self, monkeypatch) -> None:
# Deterministic: psutil is an optional dep, so drive the leg with a stub
# instead of importorskip (line must be covered on psutil-less CI too).
stub = SimpleNamespace(Process=lambda pid: SimpleNamespace(create_time=lambda: 1234.5))
monkeypatch.setitem(sys.modules, "psutil", stub)
assert sub.proc_identity(999) == ("psutil", 1234.5)
def test_psutil_leg_real_when_installed(self) -> None:
pytest.importorskip("psutil")
ident = sub.proc_identity(os.getpid())
assert ident is not None
src, start = ident
assert src == "psutil"
assert isinstance(start, float) and start > 0
def test_proc_leg_when_psutil_unavailable(self, monkeypatch) -> None:
monkeypatch.setitem(sys.modules, "psutil", None) # import raises ImportError
# /proc/<pid>/stat: fields after the final ")" are field 3 onwards, so
# field 22 (starttime) is index 19 of the split — here 987654.
stat = b"123 (python (test)) S 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 987654 21"
real_open = builtins.open
def fake_open(path, *args, **kwargs):
if str(path).startswith("/proc/"):
return io.BytesIO(stat)
return real_open(path, *args, **kwargs)
monkeypatch.setattr(builtins, "open", fake_open)
assert sub.proc_identity(123) == ("proc", 987654.0)
def test_returns_none_when_both_sources_fail(self, monkeypatch) -> None:
monkeypatch.setitem(sys.modules, "psutil", None)
real_open = builtins.open
def fake_open(path, *args, **kwargs):
if str(path).startswith("/proc/"):
raise OSError("no /proc here")
return real_open(path, *args, **kwargs)
monkeypatch.setattr(builtins, "open", fake_open)
assert sub.proc_identity(123) is None
class TestIdentityMismatch:
def test_legacy_record_never_mismatches(self) -> None:
assert sub.identity_mismatch(None, None, os.getpid()) is False
assert sub.identity_mismatch("psutil", "not-a-number", os.getpid()) is False
def test_unknown_identity_never_mismatches(self) -> None:
assert sub.identity_mismatch("psutil", 1.0, 123, identity_fn=lambda pid: None) is False
def test_source_mismatch_never_mismatches(self) -> None:
ident = lambda pid: ("proc", 1.0) # noqa: E731
assert sub.identity_mismatch("psutil", 1.0, 123, identity_fn=ident) is False
def test_same_start_time_is_not_a_mismatch(self) -> None:
ident = lambda pid: ("psutil", 1000.5) # noqa: E731
assert sub.identity_mismatch("psutil", 1000.2, 123, identity_fn=ident) is False
def test_distant_start_time_proves_recycling(self) -> None:
ident = lambda pid: ("psutil", 2000.0) # noqa: E731
assert sub.identity_mismatch("psutil", 1000.0, 123, identity_fn=ident) is True