mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): reap wrap-spawned proxies once no wrap clients remain (#3202)
## Description Nothing in Headroom ever stops a wrap-spawned proxy whose wrappers are gone. Two leak paths, both by-products of deliberate hardening without a counterpart: 1. **Unclean wrapper death.** `_start_proxy` detaches the proxy (`start_new_session` on POSIX, breakaway console flags on Windows) on purpose, so a crashed wrapper cannot tree-kill a shared proxy out from under other clients. But when the wrapper dies without running cleanup (SIGKILL, crash, terminal close), its client marker goes stale, `_live_proxy_clients` unlinks it, and the proxy itself is never stopped. 2. **Graceful last-exit leak.** `_make_cleanup` only terminates the proxy when the exiting wrapper *started* it (`proxy_holder[0] is not None`). When the starter exits while other clients remain (correct), every later wrapper exits with no handle, so even a perfectly behaved multi-wrap topology leaks the proxy at the end. Observed in production: three `headroom proxy` processes still alive 21 days after their last session, holding ports and being silently reused by later wraps (which then inherited their stale version and captive routing; see the two companion PRs). The fix: wrap-spawned proxies carry `HEADROOM_WRAP_OWNED=1` and run an orphan-watchdog task that polls the per-port client-marker dir every 30s and gracefully stops the process once no live markers, no active WebSocket sessions, and no served-request activity remain for a grace period (default 900s, floor 60s, `HEADROOM_WRAP_ORPHAN_GRACE_SECONDS`). Standalone `headroom proxy` instances and persistent installs never get the marker env var and are never affected; multi-worker runs are excluded (WS sessions are per-worker, and wrap never requests multiple workers). Closes #3199 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/proxy/orphan_watchdog.py` (new): env config (`WRAP_OWNED_ENV`, grace parsing with floor), marker liveness (`pid_alive` + start-time identity, pruning stale markers), and `orphan_watchdog_loop` (idle clock reset by live markers, active WS sessions, or movement of the proxy's served-request counter — the last covers direct-HTTP and SSH-forwarded clients that write no marker). Stop is `signal.raise_signal(SIGTERM)`, which reaches uvicorn's handler on Unix and the emulated handler table on Windows (unlike `os.kill(pid, SIGTERM)`, which is `TerminateProcess` there). - `headroom/proxy/server.py`: lifespan spawns the watchdog when `orphan_watchdog_enabled()` and not a multi-worker run (`HEADROOM_PROXY_CONFIG_JSON` absent), and cancels it in teardown like the other periodic tasks. - `headroom/cli/wrap.py`: `_start_proxy` sets `HEADROOM_WRAP_OWNED=1` in the proxy subprocess env; `_proc_identity`/`_identity_mismatch` moved to `headroom/_subprocess.py` (`proc_identity`/`identity_mismatch`) so proxy code does not import the CLI layer — thin module-level delegates keep the existing monkeypatch-based tests working unchanged. - `tests/test_proxy/test_orphan_watchdog.py` (new, 13 tests): env parsing (default/override/floor/garbage), marker pruning (live, dead PID, recycled PID, garbage JSON, missing dir), loop semantics (exit after grace, hold on live client, hold on active WS session, hold on served-request movement, grace reset on returning client, missing registry counts as idle). - `tests/test_cli/test_wrap_claude_vertex_proxy_env.py`: `_start_proxy` marks the subprocess env wrap-owned. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_proxy/test_orphan_watchdog.py -q 13 passed in 2.77s $ uv run pytest tests/test_proxy tests/test_cli -q 973 passed, 1 skipped in 49.85s # 1 pre-existing unrelated failure # (test_litellm_vertex_backend_path_...) fails # identically on clean upstream/main $ uv run ruff format / ruff check on touched files All checks passed! $ uv run mypy headroom/_subprocess.py headroom/cli/wrap.py headroom/proxy/orphan_watchdog.py Success: no issues found in 3 source files ``` ## Real Behavior Proof - Environment: macOS 26.5 arm64, Python 3.13, local source build 0.37.0-dev (integration overlay with this PR applied). - Exact command / steps: start wrap-spawned proxies via `wrap._ensure_proxy(...)` (which goes through `_start_proxy` and thus `HEADROOM_WRAP_OWNED=1`), with zero registered clients; wait; inspect `~/.headroom/logs/proxy.log`. - Observed result: two such proxies (plain on 8789 PID 48365, augment-mode on 8791 PID 49729) self-exited at the 900s default grace: `event=proxy_shutdown reason=signal pid=49729` at 09:51:49 against a ~09:36 start, port released afterwards (`lsof` clean). A machine-wide sweep beforehand showed three 21-day-old orphans (PIDs 73974, 63454, 79339); with this build they can no longer accumulate. - Not tested: Windows signal path (reasoned from CPython/uvicorn semantics; unit-injected `stop` callable covers the loop side); the multi-worker exclusion with a real `--workers 2` run (guarded by env presence only); marker pruning under active PID recycling on Linux without psutil (`/proc` fallback is unit-tested via mocks). ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: N/A - Stable/default behavior changed: only for wrap-spawned proxies, which now self-exit after 15 clientless minutes. Standalone and persistent proxies are untouched. - Kill switch / disable path: omit `HEADROOM_WRAP_OWNED` (not settable by users of wrap; standalone proxies never have it), or set `HEADROOM_WRAP_ORPHAN_GRACE_SECONDS` very high. - Unsafe override required: no - Qualification impact: none - Rollback path: revert the commit; orphaned proxies leak again (status quo ante). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Adversarially reviewed by a second model (glm-5.2, high reasoning); initial verdict needs-work, all accepted findings applied (multi-worker exclusion, Windows-safe stop signal, served-request activity hold for marker-less clients, wider test timing margins) and re-tested green. The reviewer independently verified the boot-race ordering, teardown cancellation, and the kill-vs-self-exit race as already correct. Companion PRs (same incident): `-dev` version-restart fix, routing-config reuse check.
This commit is contained in:
+57
-1
@@ -38,11 +38,67 @@ def pid_alive(pid: int) -> bool:
|
||||
os.kill(pid, 0)
|
||||
except PermissionError:
|
||||
return True
|
||||
except (ProcessLookupError, OSError, SystemError):
|
||||
except (ProcessLookupError, OSError, SystemError, OverflowError, ValueError):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def proc_identity(pid: int) -> tuple[str, float] | None:
|
||||
"""Best-effort ``(source, start_time)`` identity for a PID.
|
||||
|
||||
Used to defeat PID reuse: a marker is only trusted while the live PID is
|
||||
*the same process* that wrote it. Returns ``None`` when start time can't be
|
||||
determined (e.g. macOS without psutil), in which case callers fall back to
|
||||
existence-only liveness, no regression, just no reuse protection there.
|
||||
|
||||
The ``source`` tag ("psutil" vs "proc") guards against comparing values in
|
||||
different units; we only compare like-for-like.
|
||||
"""
|
||||
try:
|
||||
import psutil # type: ignore[import-untyped] # optional dependency; portable when present
|
||||
|
||||
return ("psutil", psutil.Process(pid).create_time())
|
||||
except Exception:
|
||||
pass
|
||||
# Linux fallback: field 22 of /proc/<pid>/stat is starttime in clock ticks
|
||||
# since boot, a stable per-process value. `comm` (field 2) may contain
|
||||
# spaces/parens, so split after the final ')'.
|
||||
try:
|
||||
with open(f"/proc/{pid}/stat", "rb") as fh:
|
||||
fields = fh.read().rpartition(b")")[2].split()
|
||||
return ("proc", float(fields[19]))
|
||||
except (OSError, IndexError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def identity_mismatch(
|
||||
src: Any,
|
||||
recorded: Any,
|
||||
pid: int,
|
||||
*,
|
||||
identity_fn: Any = None,
|
||||
) -> bool:
|
||||
"""True only if ``pid``'s current identity *provably* differs from the
|
||||
recorded ``(src, recorded)`` identity (i.e. the PID was recycled).
|
||||
|
||||
Conservative by design: any uncertainty (unknown/legacy identity, unknown
|
||||
start time, mismatched source) returns ``False``, never claim a mismatch
|
||||
without proof, since the caller uses this to decide whether to trust or
|
||||
discard state tied to a live PID. ``identity_fn`` defaults to
|
||||
:func:`proc_identity` and exists so callers with their own (mockable)
|
||||
identity source can share this comparison.
|
||||
"""
|
||||
if identity_fn is None:
|
||||
identity_fn = proc_identity
|
||||
if not isinstance(src, str) or not isinstance(recorded, int | float):
|
||||
return False # legacy / identity-less record, can't tell
|
||||
ident = identity_fn(pid)
|
||||
if ident is None or ident[0] != src:
|
||||
return False # can't compare like-for-like, don't claim mismatch
|
||||
# Start times are stable per process; >1s apart means a different process.
|
||||
return bool(abs(ident[1] - float(recorded)) > 1.0)
|
||||
|
||||
|
||||
def run(*args: Any, **kwargs: Any) -> _sp.CompletedProcess:
|
||||
if kwargs.get("text") or kwargs.get("universal_newlines"):
|
||||
kwargs.setdefault("encoding", "utf-8")
|
||||
|
||||
+67
-103
@@ -39,7 +39,8 @@ from functools import wraps
|
||||
from pathlib import Path
|
||||
from typing import Any, NamedTuple, cast
|
||||
|
||||
from headroom._subprocess import pid_alive, run
|
||||
from headroom._subprocess import identity_mismatch as _shared_identity_mismatch
|
||||
from headroom._subprocess import pid_alive, proc_identity, run
|
||||
|
||||
# Fix Windows cp1252 encoding — box-drawing characters require UTF-8
|
||||
if sys.platform == "win32" and hasattr(sys.stdout, "buffer"):
|
||||
@@ -662,7 +663,16 @@ def _start_proxy(
|
||||
(see ``_find_available_port``).
|
||||
"""
|
||||
|
||||
cmd = [sys.executable, "-m", "headroom.cli", "proxy", "--port", str(port)]
|
||||
cmd = [
|
||||
sys.executable,
|
||||
"-m",
|
||||
"headroom.cli",
|
||||
"proxy",
|
||||
"--port",
|
||||
str(port),
|
||||
"--workers",
|
||||
"1",
|
||||
]
|
||||
|
||||
# Forward HEADROOM_MODE env var so the proxy respects the user's mode choice
|
||||
headroom_mode = os.environ.get("HEADROOM_MODE")
|
||||
@@ -711,6 +721,10 @@ def _start_proxy(
|
||||
# Ensure proxy subprocess uses UTF-8 (Windows defaults to cp1252)
|
||||
proxy_env = os.environ.copy()
|
||||
_scrub_copilot_proxy_seed_env(proxy_env)
|
||||
# A wrap-owned proxy must be a single authoritative process so its orphan
|
||||
# watchdog can observe all activity before deciding to self-terminate.
|
||||
proxy_env.pop("HEADROOM_WORKERS", None)
|
||||
proxy_env.pop("HEADROOM_PROXY_CONFIG_JSON", None)
|
||||
proxy_env["PYTHONIOENCODING"] = "utf-8"
|
||||
# `python -m headroom.cli` prepends the launch cwd to sys.path, so running
|
||||
# `wrap` from a directory that contains a `headroom/` folder (most commonly a
|
||||
@@ -729,6 +743,12 @@ def _start_proxy(
|
||||
if agent_type != "unknown":
|
||||
proxy_env["HEADROOM_AGENT_TYPE"] = agent_type
|
||||
proxy_env.setdefault("HEADROOM_STACK", f"wrap_{agent_type}")
|
||||
# Mark the proxy as wrap-spawned so its orphan watchdog (proxy side) stops
|
||||
# it once no live wrap clients remain. Standalone `headroom proxy` and
|
||||
# persistent installs never get this flag and never self-exit. The proxy
|
||||
# is detached (setsid / breakaway flags), so without this a crashed or
|
||||
# killed wrapper leaks the proxy forever and later wraps silently reuse it.
|
||||
proxy_env["HEADROOM_WRAP_OWNED"] = "1"
|
||||
savings_profile = _wrap_agent_savings_profile(agent_type)
|
||||
if savings_profile is not None:
|
||||
apply_agent_savings_env_defaults(proxy_env, savings_profile)
|
||||
@@ -4555,31 +4575,13 @@ def _client_marker_path(port: int) -> Path:
|
||||
|
||||
|
||||
def _proc_identity(pid: int) -> tuple[str, float] | None:
|
||||
"""Best-effort ``(source, start_time)`` identity for a PID.
|
||||
"""Module-level alias of :func:`headroom._subprocess.proc_identity`.
|
||||
|
||||
Used to defeat PID reuse: a marker is only trusted while the live PID is
|
||||
*the same process* that wrote it. Returns ``None`` when start time can't be
|
||||
determined (e.g. macOS without psutil), in which case callers fall back to
|
||||
existence-only liveness — no regression, just no reuse protection there.
|
||||
|
||||
The ``source`` tag ("psutil" vs "proc") guards against comparing values in
|
||||
different units; we only compare like-for-like.
|
||||
Kept as a function (not an assignment) so tests can monkeypatch
|
||||
``wrap._proc_identity`` and have ``_identity_mismatch`` pick the patch up
|
||||
through the module-global lookup below.
|
||||
"""
|
||||
try:
|
||||
import psutil # type: ignore[import-untyped] # optional dependency; portable when present
|
||||
|
||||
return ("psutil", psutil.Process(pid).create_time())
|
||||
except Exception:
|
||||
pass
|
||||
# Linux fallback: field 22 of /proc/<pid>/stat is starttime in clock ticks
|
||||
# since boot — a stable per-process value. `comm` (field 2) may contain
|
||||
# spaces/parens, so split after the final ')'.
|
||||
try:
|
||||
with open(f"/proc/{pid}/stat", "rb") as fh:
|
||||
fields = fh.read().rpartition(b")")[2].split()
|
||||
return ("proc", float(fields[19]))
|
||||
except (OSError, IndexError, ValueError):
|
||||
return None
|
||||
return proc_identity(pid)
|
||||
|
||||
|
||||
def _register_proxy_client(port: int) -> None:
|
||||
@@ -4616,21 +4618,9 @@ def _pid_alive(pid: int) -> bool:
|
||||
|
||||
|
||||
def _identity_mismatch(src: Any, recorded: Any, pid: int) -> bool:
|
||||
"""True only if ``pid``'s current identity *provably* differs from the
|
||||
recorded ``(src, recorded)`` identity (i.e. the PID was recycled).
|
||||
|
||||
Conservative by design: any uncertainty (unknown/legacy identity, unknown
|
||||
start time, mismatched source) returns ``False`` — never claim a mismatch
|
||||
without proof, since the caller uses this to decide whether to trust or
|
||||
discard state tied to a live PID.
|
||||
"""
|
||||
if not isinstance(src, str) or not isinstance(recorded, int | float):
|
||||
return False # legacy / identity-less record — can't tell
|
||||
ident = _proc_identity(pid)
|
||||
if ident is None or ident[0] != src:
|
||||
return False # can't compare like-for-like — don't claim mismatch
|
||||
# Start times are stable per process; >1s apart means a different process.
|
||||
return abs(ident[1] - float(recorded)) > 1.0
|
||||
"""Delegate to the shared comparison, resolving the identity probe through
|
||||
this module so monkeypatched ``_proc_identity`` keeps working in tests."""
|
||||
return _shared_identity_mismatch(src, recorded, pid, identity_fn=_proc_identity)
|
||||
|
||||
|
||||
def _marker_pid_reused(marker: Path, pid: int) -> bool:
|
||||
@@ -4641,6 +4631,8 @@ def _marker_pid_reused(marker: Path, pid: int) -> bool:
|
||||
rec = json.loads(_read_text(marker))
|
||||
except (OSError, ValueError):
|
||||
return False
|
||||
if not isinstance(rec, dict):
|
||||
return False
|
||||
return _identity_mismatch(rec.get("start_src"), rec.get("start_time"), pid)
|
||||
|
||||
|
||||
@@ -4671,61 +4663,21 @@ def _live_proxy_clients(port: int, *, exclude_self: bool = True) -> list[int]:
|
||||
|
||||
|
||||
def _make_cleanup(proxy_proc_holder: list, port: int | list[int] = 8787) -> Any:
|
||||
"""Create a cleanup function that terminates the proxy on exit.
|
||||
|
||||
Only kills the proxy when no other live headroom-wrapped clients remain,
|
||||
tracked via per-PID marker files in ``paths.proxy_clients_dir(port)``.
|
||||
"""Create marker cleanup for a successfully launched wrap session.
|
||||
|
||||
``port`` can be an ``int`` or a ``list[int]``. When a port fallback occurs
|
||||
(``_ensure_proxy`` ups the port because the requested one is busy), the
|
||||
caller can update ``port[0]`` in-place and the closure picks it up.
|
||||
"""
|
||||
|
||||
def _other_clients_exist() -> bool:
|
||||
p = port[0] if isinstance(port, list) else port
|
||||
return len(_live_proxy_clients(p, exclude_self=True)) > 0
|
||||
Normal wrapper exit only unregisters its marker. The wrap-owned proxy's
|
||||
watchdog owns final shutdown so active markerless HTTP or WebSocket traffic
|
||||
receives a full grace period. Startup rollback and explicit stop commands
|
||||
retain direct process termination.
|
||||
"""
|
||||
|
||||
def cleanup(signum: int | None = None, frame: Any = None) -> None:
|
||||
p = port[0] if isinstance(port, list) else port
|
||||
_unregister_proxy_client(p)
|
||||
proc = proxy_proc_holder[0] if proxy_proc_holder else None
|
||||
if proc:
|
||||
if _other_clients_exist():
|
||||
# Other clients still using the proxy — leave it running.
|
||||
return
|
||||
# Snapshot the serving PID before terminating the launcher. On
|
||||
# Windows the detached serving child can briefly make /health
|
||||
# unavailable while the launcher exits, causing the later safety
|
||||
# probe to classify our own listener as "unidentified" and leave
|
||||
# it orphaned. We still verify it through Headroom's health
|
||||
# payload before trusting the PID.
|
||||
serving_pid: int | None = None
|
||||
if sys.platform == "win32" and _check_proxy(p):
|
||||
running_config = _query_proxy_config(p)
|
||||
try:
|
||||
serving_pid = int(running_config["pid"]) if running_config else None
|
||||
except (KeyError, TypeError, ValueError):
|
||||
serving_pid = None
|
||||
if proc.poll() is None:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
# On Windows the proxy launcher can exit while its detached
|
||||
# serving child remains alive (the native runtime uses a child
|
||||
# process). The detachment is intentional so an ungraceful
|
||||
# terminal close cannot disrupt other wrappers, but a graceful
|
||||
# Ctrl+C from the last wrapper must still stop the listener.
|
||||
if sys.platform == "win32" and _check_proxy(p):
|
||||
stop_status = _stop_local_proxy_for_unwrap(p)
|
||||
if stop_status == "unidentified" and serving_pid is not None:
|
||||
stop_status = "stopped" if _kill_proxy_by_pid(serving_pid, p) else "failed"
|
||||
if stop_status not in {"stopped", "not_running"}:
|
||||
click.echo(
|
||||
f" Warning: proxy on port {p} remained running "
|
||||
f"after shutdown ({stop_status})."
|
||||
)
|
||||
|
||||
return cleanup
|
||||
|
||||
@@ -7698,6 +7650,7 @@ def opencode(
|
||||
),
|
||||
)
|
||||
|
||||
launch_started = False
|
||||
try:
|
||||
# If the proxy fell back to a different port, move our marker so
|
||||
# cleanup tracking stays accurate and update MCP config.
|
||||
@@ -7726,24 +7679,35 @@ def opencode(
|
||||
|
||||
# Proxy already started by _ensure_proxy above; tell _launch_tool to
|
||||
# skip duplicate startup.
|
||||
_launch_tool(
|
||||
binary=opencode_bin,
|
||||
args=opencode_args,
|
||||
env=env,
|
||||
port=actual_port,
|
||||
no_proxy=True,
|
||||
tool_label="OPENCODE",
|
||||
env_vars_display=env_vars_display,
|
||||
learn=learn,
|
||||
memory=memory,
|
||||
agent_type="opencode",
|
||||
code_graph=code_graph,
|
||||
backend=backend,
|
||||
anyllm_provider=anyllm_provider,
|
||||
region=region,
|
||||
)
|
||||
launch_started = True
|
||||
try:
|
||||
_launch_tool(
|
||||
binary=opencode_bin,
|
||||
args=opencode_args,
|
||||
env=env,
|
||||
port=actual_port,
|
||||
no_proxy=True,
|
||||
tool_label="OPENCODE",
|
||||
env_vars_display=env_vars_display,
|
||||
learn=learn,
|
||||
memory=memory,
|
||||
agent_type="opencode",
|
||||
code_graph=code_graph,
|
||||
backend=backend,
|
||||
anyllm_provider=anyllm_provider,
|
||||
region=region,
|
||||
)
|
||||
except SystemExit as exc:
|
||||
# _launch_tool raises plain SystemExit after a child exits, but
|
||||
# chains pre-launch failures as the cause.
|
||||
if exc.__cause__ is not None:
|
||||
launch_started = False
|
||||
raise
|
||||
except BaseException:
|
||||
launch_started = False
|
||||
raise
|
||||
finally:
|
||||
if _opencode_proxy and _opencode_proxy.poll() is None:
|
||||
if not launch_started and _opencode_proxy and _opencode_proxy.poll() is None:
|
||||
_other = _live_proxy_clients(actual_port, exclude_self=True)
|
||||
if not _other:
|
||||
_opencode_proxy.terminate()
|
||||
|
||||
@@ -0,0 +1,223 @@
|
||||
"""Orphan watchdog for wrap-spawned proxies.
|
||||
|
||||
``headroom wrap`` starts its proxy detached (setsid on POSIX, breakaway
|
||||
console flags on Windows) on purpose: a crashing wrapper must not tree-kill a
|
||||
shared proxy out from under other clients. The flip side was a leak: once
|
||||
every wrapper was gone, nothing ever stopped the proxy, so clientless proxies
|
||||
accumulated for weeks and later wraps silently reused whatever they found on
|
||||
the port (port-hijack incident, 2026-08-22). Cleanup on the wrapper side
|
||||
cannot cover unclean exits, by definition nobody is alive to run it.
|
||||
|
||||
When the proxy was spawned by a wrap (``HEADROOM_WRAP_OWNED=1``, set in
|
||||
``cli/wrap._start_proxy``), this module's background task stops the process
|
||||
once no live wrap-client markers and no active sessions remain for a grace
|
||||
period. Standalone ``headroom proxy`` instances and persistent installs never
|
||||
get the marker env var and are never affected.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import signal
|
||||
import time
|
||||
from collections.abc import Callable, Mapping
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from headroom._subprocess import identity_mismatch, pid_alive
|
||||
from headroom.paths import proxy_clients_dir
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
WRAP_OWNED_ENV = "HEADROOM_WRAP_OWNED"
|
||||
GRACE_SECONDS_ENV = "HEADROOM_WRAP_ORPHAN_GRACE_SECONDS"
|
||||
DEFAULT_GRACE_SECONDS = 900.0
|
||||
# Floor so a typo ("5" for "5m") cannot make a proxy exit underneath a client
|
||||
# that simply has not registered its marker yet.
|
||||
MIN_GRACE_SECONDS = 60.0
|
||||
DEFAULT_POLL_INTERVAL_SECONDS = 30.0
|
||||
|
||||
|
||||
def orphan_watchdog_enabled(env: Mapping[str, str] | None = None) -> bool:
|
||||
"""True when this proxy process was spawned by ``headroom wrap``."""
|
||||
env = os.environ if env is None else env
|
||||
return (env.get(WRAP_OWNED_ENV) or "").strip().lower() in ("1", "true", "yes", "on")
|
||||
|
||||
|
||||
def orphan_grace_seconds(env: Mapping[str, str] | None = None) -> float:
|
||||
"""Grace period with zero live clients before the watchdog exits the proxy."""
|
||||
env = os.environ if env is None else env
|
||||
raw = env.get(GRACE_SECONDS_ENV)
|
||||
if raw:
|
||||
try:
|
||||
return max(MIN_GRACE_SECONDS, float(raw))
|
||||
except ValueError:
|
||||
logger.warning(
|
||||
"event=orphan_watchdog_bad_grace value=%r default=%.0f", raw, DEFAULT_GRACE_SECONDS
|
||||
)
|
||||
return DEFAULT_GRACE_SECONDS
|
||||
|
||||
|
||||
def _marker_pid_recycled(marker: Path, pid: int) -> bool:
|
||||
"""True only if the live ``pid`` is provably a different process than the
|
||||
one that wrote ``marker`` (PID recycled after a crash)."""
|
||||
try:
|
||||
record = json.loads(marker.read_text(encoding="utf-8"))
|
||||
except (OSError, ValueError):
|
||||
return False
|
||||
if not isinstance(record, dict):
|
||||
return False
|
||||
return identity_mismatch(record.get("start_src"), record.get("start_time"), pid)
|
||||
|
||||
|
||||
def live_client_pids(clients_dir: Path) -> list[int] | None:
|
||||
"""Live wrap-client PIDs for a clients dir, pruning stale markers as we go.
|
||||
|
||||
Mirrors ``cli.wrap._live_proxy_clients`` without the self-exclusion; the
|
||||
proxy is never its own client.
|
||||
"""
|
||||
try:
|
||||
with os.scandir(clients_dir) as entries:
|
||||
markers = [
|
||||
Path(entry.path)
|
||||
for entry in entries
|
||||
if entry.name.endswith(".json") and entry.is_file()
|
||||
]
|
||||
except FileNotFoundError:
|
||||
return []
|
||||
except OSError:
|
||||
logger.warning(
|
||||
"event=orphan_watchdog_client_scan_failed dir=%s", clients_dir, exc_info=True
|
||||
)
|
||||
return None
|
||||
|
||||
live: list[int] = []
|
||||
for marker in markers:
|
||||
try:
|
||||
pid = int(marker.stem)
|
||||
except ValueError:
|
||||
continue
|
||||
alive = pid_alive(pid)
|
||||
if not alive or _marker_pid_recycled(marker, pid):
|
||||
try:
|
||||
marker.unlink(missing_ok=True)
|
||||
except OSError:
|
||||
pass
|
||||
continue
|
||||
live.append(pid)
|
||||
return live
|
||||
|
||||
|
||||
def _active_session_count(proxy: Any) -> int | None:
|
||||
registry = getattr(proxy, "ws_sessions", None)
|
||||
if registry is None:
|
||||
return 0
|
||||
try:
|
||||
return int(registry.active_count())
|
||||
except Exception:
|
||||
logger.warning("event=orphan_watchdog_session_count_failed", exc_info=True)
|
||||
return None
|
||||
|
||||
|
||||
def _active_request_count(proxy: Any) -> int | None:
|
||||
"""Return active HTTP/WebSocket scope count, or None when unobservable."""
|
||||
try:
|
||||
count = proxy.active_request_count
|
||||
except Exception:
|
||||
logger.warning("event=orphan_watchdog_request_count_failed", exc_info=True)
|
||||
return None
|
||||
return count if isinstance(count, int) and count >= 0 else None
|
||||
|
||||
|
||||
def _activity_generation(proxy: Any) -> int | None:
|
||||
"""Return the monotonic ASGI activity generation, or None."""
|
||||
try:
|
||||
generation = proxy.activity_generation
|
||||
except Exception:
|
||||
logger.warning("event=orphan_watchdog_activity_generation_failed", exc_info=True)
|
||||
return None
|
||||
return generation if isinstance(generation, int) and generation >= 0 else None
|
||||
|
||||
|
||||
def _default_stop() -> None:
|
||||
# signal.raise_signal (not os.kill): on Windows CPython maps
|
||||
# os.kill(pid, SIGTERM) to TerminateProcess, a hard kill that skips
|
||||
# uvicorn's handler and the lifespan teardown; raise_signal dispatches
|
||||
# through the emulated handler table, so uvicorn shuts down gracefully on
|
||||
# both platforms.
|
||||
signal.raise_signal(signal.SIGTERM)
|
||||
|
||||
|
||||
async def orphan_watchdog_loop(
|
||||
proxy: Any,
|
||||
*,
|
||||
grace_seconds: float,
|
||||
interval_seconds: float = DEFAULT_POLL_INTERVAL_SECONDS,
|
||||
stop: Callable[[], None] = _default_stop,
|
||||
monotonic: Callable[[], float] = time.monotonic,
|
||||
) -> None:
|
||||
"""Poll wrap-client markers; ``stop()`` once clientless for ``grace_seconds``.
|
||||
|
||||
The grace window covers the boot race (the wrapper registers its marker
|
||||
just before spawning the proxy, and re-registers on port fallback) and
|
||||
short client restarts. Active WebSocket relay sessions, in-flight HTTP
|
||||
requests, and any change to the ASGI activity generation suppress shutdown
|
||||
even with no live markers. The idle clock runs only while every supported
|
||||
activity signal is positively observable and idle.
|
||||
"""
|
||||
port = proxy.config.port
|
||||
clients_dir = proxy_clients_dir(port)
|
||||
idle_since: float | None = None
|
||||
last_activity_generation = _activity_generation(proxy)
|
||||
while True:
|
||||
await asyncio.sleep(interval_seconds)
|
||||
clients = live_client_pids(clients_dir)
|
||||
active_sessions = _active_session_count(proxy)
|
||||
active_requests = _active_request_count(proxy)
|
||||
activity_generation = _activity_generation(proxy)
|
||||
activity_changed = (
|
||||
activity_generation is not None
|
||||
and last_activity_generation is not None
|
||||
and activity_generation != last_activity_generation
|
||||
)
|
||||
if activity_generation is not None:
|
||||
last_activity_generation = activity_generation
|
||||
activity_unknown = (
|
||||
clients is None
|
||||
or active_sessions is None
|
||||
or active_requests is None
|
||||
or activity_generation is None
|
||||
)
|
||||
if (
|
||||
activity_unknown
|
||||
or clients
|
||||
or (active_sessions is not None and active_sessions > 0)
|
||||
or (active_requests is not None and active_requests > 0)
|
||||
or activity_changed
|
||||
):
|
||||
if idle_since is not None:
|
||||
logger.info(
|
||||
"event=orphan_watchdog_reset port=%d live_clients=%s "
|
||||
"active_sessions=%s active_requests=%s",
|
||||
port,
|
||||
clients,
|
||||
active_sessions,
|
||||
active_requests,
|
||||
)
|
||||
idle_since = None
|
||||
continue
|
||||
now = monotonic()
|
||||
if idle_since is None:
|
||||
idle_since = now
|
||||
continue
|
||||
if now - idle_since >= grace_seconds:
|
||||
logger.info(
|
||||
"event=orphan_watchdog_exit port=%d idle_seconds=%.0f reason=no_live_wrap_clients",
|
||||
port,
|
||||
now - idle_since,
|
||||
)
|
||||
stop()
|
||||
return
|
||||
@@ -166,6 +166,11 @@ from headroom.proxy.modes import (
|
||||
is_token_mode,
|
||||
normalize_proxy_mode,
|
||||
)
|
||||
from headroom.proxy.orphan_watchdog import (
|
||||
orphan_grace_seconds,
|
||||
orphan_watchdog_enabled,
|
||||
orphan_watchdog_loop,
|
||||
)
|
||||
from headroom.proxy.probe_recorder import probe_recorder_from_env
|
||||
from headroom.proxy.project_context import (
|
||||
classify_project,
|
||||
@@ -1275,6 +1280,8 @@ class HeadroomProxy(
|
||||
# Request counter for IDs
|
||||
self._request_counter = 0
|
||||
self._request_counter_lock = asyncio.Lock()
|
||||
self._active_requests = 0
|
||||
self._activity_generation = 0
|
||||
|
||||
# CCR tool injectors (one per provider)
|
||||
self.anthropic_tool_injector = CCRToolInjector(
|
||||
@@ -2367,6 +2374,16 @@ class HeadroomProxy(
|
||||
self._request_counter += 1
|
||||
return f"hr_{int(time.time())}_{self._request_counter:06d}"
|
||||
|
||||
@property
|
||||
def active_request_count(self) -> int:
|
||||
"""Number of active HTTP and WebSocket ASGI scopes."""
|
||||
return self._active_requests
|
||||
|
||||
@property
|
||||
def activity_generation(self) -> int:
|
||||
"""Monotonic generation advanced at request entry and completion."""
|
||||
return self._activity_generation
|
||||
|
||||
def _extract_tags(self, headers: dict) -> dict[str, str]:
|
||||
"""Backwards-compat wrapper around :func:`extract_tags`.
|
||||
|
||||
@@ -2890,6 +2907,27 @@ class WebSocketProjectPrefixMiddleware:
|
||||
await self.app(scope, receive, send)
|
||||
|
||||
|
||||
class ActivityMiddleware:
|
||||
"""Track HTTP and WebSocket scopes through their complete ASGI lifetime."""
|
||||
|
||||
def __init__(self, app: Any, *, proxy: HeadroomProxy) -> None:
|
||||
self.app = app
|
||||
self.proxy = proxy
|
||||
|
||||
async def __call__(self, scope: Any, receive: Any, send: Any) -> None:
|
||||
if scope["type"] not in {"http", "websocket"}:
|
||||
await self.app(scope, receive, send)
|
||||
return
|
||||
|
||||
self.proxy._active_requests += 1
|
||||
self.proxy._activity_generation += 1
|
||||
try:
|
||||
await self.app(scope, receive, send)
|
||||
finally:
|
||||
self.proxy._active_requests -= 1
|
||||
self.proxy._activity_generation += 1
|
||||
|
||||
|
||||
def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
"""Create FastAPI application."""
|
||||
if not FASTAPI_AVAILABLE:
|
||||
@@ -3043,6 +3081,7 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
app.state.startup_error = None
|
||||
app.state.periodic_toin_stats_task = None
|
||||
app.state.periodic_malloc_trim_task = None
|
||||
app.state.orphan_watchdog_task = None
|
||||
|
||||
try:
|
||||
try:
|
||||
@@ -3059,6 +3098,17 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
app.state.periodic_malloc_trim_task = asyncio.create_task(
|
||||
trim_periodically(config.malloc_trim_interval_seconds)
|
||||
)
|
||||
# Only wrap-spawned proxies carry the env marker; standalone
|
||||
# `headroom proxy` and persistent installs never self-exit.
|
||||
# Multi-worker runs (HEADROOM_PROXY_CONFIG_JSON is set before
|
||||
# forking workers) are excluded: WS sessions are per-worker,
|
||||
# so no single worker can observe "zero active sessions", and
|
||||
# wrap never requests multiple workers anyway.
|
||||
if orphan_watchdog_enabled() and _MULTI_WORKER_CONFIG_ENV not in os.environ:
|
||||
app.state.orphan_watchdog_task = asyncio.create_task(
|
||||
orphan_watchdog_loop(proxy, grace_seconds=orphan_grace_seconds()),
|
||||
name="headroom-orphan-watchdog",
|
||||
)
|
||||
if proxy.usage_reporter:
|
||||
await proxy.usage_reporter.start(proxy)
|
||||
if proxy.traffic_learner:
|
||||
@@ -3128,6 +3178,16 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
)
|
||||
app.state.periodic_malloc_trim_task = None
|
||||
|
||||
orphan_watchdog_task = app.state.orphan_watchdog_task
|
||||
if orphan_watchdog_task is not None:
|
||||
orphan_watchdog_task.cancel()
|
||||
await _timed(
|
||||
asyncio.gather(orphan_watchdog_task, return_exceptions=True),
|
||||
label="orphan_watchdog.stop",
|
||||
timeout=3.0,
|
||||
)
|
||||
app.state.orphan_watchdog_task = None
|
||||
|
||||
if _cc_reconciler is not None:
|
||||
await _timed(_cc_reconciler.stop(), label="cc_reconciler.stop", timeout=3.0)
|
||||
if _beacon_is_owner[0]:
|
||||
@@ -5681,6 +5741,9 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
_install_gateway_contract(app, config, route_dependencies=_compress_dependencies)
|
||||
|
||||
register_provider_routes(app, proxy)
|
||||
# Register last so this raw ASGI middleware is outermost and covers the
|
||||
# complete response body, including responses produced by other middleware.
|
||||
app.add_middleware(ActivityMiddleware, proxy=proxy)
|
||||
|
||||
return app
|
||||
|
||||
|
||||
@@ -460,6 +460,29 @@ def test_start_proxy_sets_vertex_target_env_for_proxy_subprocess(
|
||||
assert proxy_env["VERTEX_TARGET_API_URL"] == "https://vertex-gateway.internal/custom"
|
||||
|
||||
|
||||
def test_start_proxy_marks_subprocess_as_wrap_owned(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
"""Wrap-spawned proxies carry HEADROOM_WRAP_OWNED=1 for the orphan watchdog."""
|
||||
fake_proc = _FakeProxyProcess()
|
||||
captured: dict[str, Any] = {}
|
||||
|
||||
monkeypatch.setattr(wrap_mod, "_get_log_path", lambda port=None: tmp_path / "proxy.log")
|
||||
monkeypatch.setattr(wrap_mod, "_check_proxy", lambda _port: True)
|
||||
monkeypatch.setattr(wrap_mod.time, "sleep", lambda _seconds: None)
|
||||
|
||||
def fake_popen(cmd: list[str], **kwargs: object) -> _FakeProxyProcess:
|
||||
captured["kwargs"] = kwargs
|
||||
return fake_proc
|
||||
|
||||
monkeypatch.setattr(wrap_mod.subprocess, "Popen", fake_popen)
|
||||
|
||||
proc = wrap_mod._start_proxy(8787, agent_type="codex")
|
||||
|
||||
assert proc is fake_proc
|
||||
assert captured["kwargs"]["env"]["HEADROOM_WRAP_OWNED"] == "1"
|
||||
|
||||
|
||||
def test_start_proxy_clears_inherited_vertex_target_env(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
|
||||
@@ -533,6 +533,62 @@ class _FakeProxyProc:
|
||||
self.killed = True
|
||||
|
||||
|
||||
def test_start_proxy_strips_ambient_worker_configuration(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
monkeypatch.setenv("HEADROOM_WORKERS", "4")
|
||||
monkeypatch.setenv("HEADROOM_PROXY_CONFIG_JSON", '{"port": 9999}')
|
||||
monkeypatch.setenv("CLAUDE_CODE_USE_VERTEX", "1")
|
||||
captured: dict[str, object] = {}
|
||||
proc = _FakeProxyProc()
|
||||
|
||||
def fake_popen(command: list[str], **kwargs: object) -> _FakeProxyProc:
|
||||
captured["command"] = command
|
||||
captured["env"] = kwargs["env"]
|
||||
return proc
|
||||
|
||||
monkeypatch.setattr(wrap_mod.subprocess, "Popen", fake_popen)
|
||||
monkeypatch.setattr(wrap_mod, "_check_proxy", lambda port: True)
|
||||
monkeypatch.setattr(wrap_mod, "_get_log_path", lambda: tmp_path / "proxy.log")
|
||||
monkeypatch.setattr(
|
||||
wrap_mod,
|
||||
"_get_proxy_stdio_log_path",
|
||||
lambda: tmp_path / "proxy-stdio.log",
|
||||
)
|
||||
monkeypatch.setattr(wrap_mod.time, "sleep", lambda seconds: None)
|
||||
|
||||
assert wrap_mod._start_proxy(8787) is proc
|
||||
env = captured["env"]
|
||||
assert isinstance(env, dict)
|
||||
assert "HEADROOM_WORKERS" not in env
|
||||
assert "HEADROOM_PROXY_CONFIG_JSON" not in env
|
||||
assert env["HEADROOM_HTTP2"] == "false"
|
||||
assert captured["command"][-2:] == ["--workers", "1"]
|
||||
|
||||
|
||||
def test_start_proxy_timeout_kills_failed_new_process(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
proc = _FakeProxyProc()
|
||||
monkeypatch.setattr(wrap_mod.subprocess, "Popen", lambda *args, **kwargs: proc)
|
||||
monkeypatch.setattr(wrap_mod, "_check_proxy", lambda port: False)
|
||||
monkeypatch.setattr(wrap_mod, "_get_log_path", lambda: tmp_path / "proxy.log")
|
||||
monkeypatch.setattr(
|
||||
wrap_mod,
|
||||
"_get_proxy_stdio_log_path",
|
||||
lambda: tmp_path / "proxy-stdio.log",
|
||||
)
|
||||
monkeypatch.setattr(wrap_mod, "_resolve_wrap_proxy_timeout_seconds", lambda: 1)
|
||||
monkeypatch.setattr(wrap_mod.time, "sleep", lambda seconds: None)
|
||||
|
||||
with pytest.raises(RuntimeError, match="failed to start"):
|
||||
wrap_mod._start_proxy(8787)
|
||||
|
||||
assert proc.killed is True
|
||||
|
||||
|
||||
class TestProxyClientRefCounting:
|
||||
"""Proxy lifecycle is reference-counted via marker files, not pgrep."""
|
||||
|
||||
@@ -560,22 +616,21 @@ class TestProxyClientRefCounting:
|
||||
marker.write_text(json.dumps(rec))
|
||||
return marker
|
||||
|
||||
def test_cleanup_terminates_proxy_when_only_self_registered(self, clients_dir: Path) -> None:
|
||||
"""The owner alone → no other clients → proxy is terminated on exit."""
|
||||
def test_cleanup_unregisters_marker_without_terminating_proxy(self, clients_dir: Path) -> None:
|
||||
"""Normal exit transfers final shutdown ownership to the watchdog."""
|
||||
wrap_mod._register_proxy_client(self.PORT)
|
||||
proc = _FakeProxyProc()
|
||||
cleanup = wrap_mod._make_cleanup([proc], self.PORT)
|
||||
|
||||
cleanup()
|
||||
|
||||
assert proc.terminated is True
|
||||
# Our own marker is removed before we count.
|
||||
assert proc.terminated is False
|
||||
assert wrap_mod._live_proxy_clients(self.PORT, exclude_self=False) == []
|
||||
|
||||
def test_cleanup_stops_detached_windows_serving_child(
|
||||
def test_cleanup_leaves_detached_windows_serving_child_to_watchdog(
|
||||
self, clients_dir: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""Ctrl+C must stop the listener even when its launcher already exited."""
|
||||
"""Windows cleanup follows the same marker-only ownership transfer."""
|
||||
wrap_mod._register_proxy_client(self.PORT)
|
||||
proc = _FakeProxyProc()
|
||||
proc.poll = lambda: 0 # type: ignore[method-assign]
|
||||
@@ -592,7 +647,7 @@ class TestProxyClientRefCounting:
|
||||
wrap_mod._make_cleanup([proc], self.PORT)()
|
||||
|
||||
assert not proc.terminated
|
||||
assert stopped == [self.PORT]
|
||||
assert stopped == []
|
||||
|
||||
def test_kill_proxy_uses_taskkill_tree_on_windows(
|
||||
self, monkeypatch: pytest.MonkeyPatch
|
||||
@@ -617,10 +672,10 @@ class TestProxyClientRefCounting:
|
||||
)
|
||||
]
|
||||
|
||||
def test_cleanup_uses_pre_shutdown_pid_when_health_probe_races(
|
||||
def test_cleanup_does_not_probe_or_kill_windows_serving_child(
|
||||
self, clients_dir: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""A transient post-terminate /health miss must not orphan the listener."""
|
||||
"""The watchdog, not wrapper cleanup, owns normal listener shutdown."""
|
||||
wrap_mod._register_proxy_client(self.PORT)
|
||||
proc = _FakeProxyProc()
|
||||
killed: list[tuple[int, int]] = []
|
||||
@@ -636,8 +691,8 @@ class TestProxyClientRefCounting:
|
||||
|
||||
wrap_mod._make_cleanup([proc], self.PORT)()
|
||||
|
||||
assert proc.terminated
|
||||
assert killed == [(456, self.PORT)]
|
||||
assert not proc.terminated
|
||||
assert killed == []
|
||||
|
||||
def test_cleanup_leaves_proxy_running_when_other_client_alive(self, clients_dir: Path) -> None:
|
||||
"""A second live client (here: the test's parent) keeps the proxy up."""
|
||||
@@ -665,6 +720,23 @@ class TestProxyClientRefCounting:
|
||||
assert dead_pid not in live
|
||||
assert not marker.exists()
|
||||
|
||||
def test_dead_client_marker_unlink_failure_is_tolerated(
|
||||
self,
|
||||
clients_dir: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
dead_pid = 358784
|
||||
marker = self._write_marker(clients_dir, dead_pid)
|
||||
monkeypatch.setattr(wrap_mod, "_pid_alive", lambda pid: pid != dead_pid)
|
||||
|
||||
def fail_unlink(*args: object, **kwargs: object) -> None:
|
||||
raise OSError("read-only")
|
||||
|
||||
monkeypatch.setattr(Path, "unlink", fail_unlink)
|
||||
|
||||
assert wrap_mod._live_proxy_clients(self.PORT, exclude_self=True) == []
|
||||
assert marker.exists()
|
||||
|
||||
def test_reused_pid_with_mismatched_identity_is_pruned(
|
||||
self, clients_dir: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
@@ -719,6 +791,13 @@ class TestProxyClientRefCounting:
|
||||
|
||||
assert wrap_mod._live_proxy_clients(self.PORT, exclude_self=True) == []
|
||||
|
||||
def test_non_dict_marker_is_tolerated(self, clients_dir: Path) -> None:
|
||||
live_pid = os.getppid()
|
||||
marker = self._write_marker(clients_dir, live_pid)
|
||||
marker.write_text("[]", encoding="utf-8")
|
||||
|
||||
assert wrap_mod._live_proxy_clients(self.PORT, exclude_self=True) == [live_pid]
|
||||
|
||||
def test_cleanup_does_not_shell_out_to_pgrep(
|
||||
self, clients_dir: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
@@ -739,7 +818,7 @@ class TestProxyClientRefCounting:
|
||||
cleanup = wrap_mod._make_cleanup([proc], self.PORT)
|
||||
cleanup() # must not raise
|
||||
|
||||
assert proc.terminated is True
|
||||
assert proc.terminated is False
|
||||
|
||||
def test_register_then_unregister_is_idempotent(self, clients_dir: Path) -> None:
|
||||
"""Register adds exactly our marker; unregister removes it; re-call is safe."""
|
||||
|
||||
@@ -293,6 +293,98 @@ def test_wrap_opencode_copilot_subscription_cleans_up_proxy_on_config_failure(
|
||||
assert proxy.wait_timeout == 5
|
||||
|
||||
|
||||
@pytest.mark.parametrize("failure", ["spawn", "interrupt"])
|
||||
def test_wrap_opencode_cleans_up_proxy_when_tool_launch_fails(
|
||||
runner: CliRunner,
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
failure: str,
|
||||
) -> None:
|
||||
monkeypatch.chdir(tmp_path)
|
||||
_set_test_home(monkeypatch, tmp_path)
|
||||
|
||||
class _FakeProxy:
|
||||
def __init__(self) -> None:
|
||||
self.terminated = False
|
||||
|
||||
def poll(self) -> None:
|
||||
return None
|
||||
|
||||
def terminate(self) -> None:
|
||||
self.terminated = True
|
||||
|
||||
def wait(self, timeout: float | None = None) -> int:
|
||||
return 0
|
||||
|
||||
proxy = _FakeProxy()
|
||||
|
||||
def fail_before_spawn(**kwargs: object) -> None:
|
||||
if failure == "interrupt":
|
||||
raise KeyboardInterrupt
|
||||
try:
|
||||
raise OSError("spawn failed")
|
||||
except OSError as error:
|
||||
raise SystemExit(1) from error
|
||||
|
||||
with (
|
||||
patch.object(wrap_mod.shutil, "which", return_value="opencode"),
|
||||
patch.object(wrap_mod, "_ensure_proxy", return_value=(proxy, 9010)),
|
||||
patch.object(wrap_mod, "_register_proxy_client"),
|
||||
patch.object(wrap_mod, "_unregister_proxy_client"),
|
||||
patch.object(wrap_mod, "_live_proxy_clients", return_value=[]),
|
||||
patch.object(wrap_mod, "inject_opencode_provider_config"),
|
||||
patch.object(wrap_mod, "_launch_tool", side_effect=fail_before_spawn),
|
||||
):
|
||||
result = runner.invoke(
|
||||
main,
|
||||
["wrap", "opencode", "--no-mcp", "--no-serena"],
|
||||
)
|
||||
|
||||
assert result.exit_code == 1
|
||||
assert proxy.terminated is True
|
||||
|
||||
|
||||
def test_wrap_opencode_leaves_started_proxy_to_watchdog_after_normal_exit(
|
||||
runner: CliRunner,
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.chdir(tmp_path)
|
||||
_set_test_home(monkeypatch, tmp_path)
|
||||
|
||||
class _FakeProxy:
|
||||
def __init__(self) -> None:
|
||||
self.terminated = False
|
||||
|
||||
def poll(self) -> None:
|
||||
return None
|
||||
|
||||
def terminate(self) -> None:
|
||||
self.terminated = True
|
||||
|
||||
def wait(self, timeout: float | None = None) -> int:
|
||||
return 0
|
||||
|
||||
proxy = _FakeProxy()
|
||||
|
||||
with (
|
||||
patch.object(wrap_mod.shutil, "which", return_value="opencode"),
|
||||
patch.object(wrap_mod, "_ensure_proxy", return_value=(proxy, 9010)),
|
||||
patch.object(wrap_mod, "_register_proxy_client"),
|
||||
patch.object(wrap_mod, "_unregister_proxy_client"),
|
||||
patch.object(wrap_mod, "_live_proxy_clients", return_value=[]),
|
||||
patch.object(wrap_mod, "inject_opencode_provider_config"),
|
||||
patch.object(wrap_mod, "_launch_tool", side_effect=SystemExit(0)),
|
||||
):
|
||||
result = runner.invoke(
|
||||
main,
|
||||
["wrap", "opencode", "--no-mcp", "--no-serena"],
|
||||
)
|
||||
|
||||
assert result.exit_code == 0, result.output
|
||||
assert proxy.terminated is False
|
||||
|
||||
|
||||
def test_wrap_opencode_sets_config_content_env(
|
||||
runner: CliRunner,
|
||||
tmp_path: Path,
|
||||
|
||||
@@ -37,6 +37,20 @@ def test_pid_alive_systemerror_is_not_alive(monkeypatch) -> None:
|
||||
assert pid_alive(4321) is False
|
||||
|
||||
|
||||
def test_pid_alive_overflow_is_not_alive(monkeypatch) -> None:
|
||||
monkeypatch.setitem(
|
||||
sys.modules,
|
||||
"psutil",
|
||||
types.SimpleNamespace(pid_exists=lambda pid: (_ for _ in ()).throw(RuntimeError())),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"headroom._subprocess.os.kill",
|
||||
lambda pid, sig: (_ for _ in ()).throw(OverflowError()),
|
||||
)
|
||||
|
||||
assert pid_alive(int("9" * 100)) is False
|
||||
|
||||
|
||||
def test_pid_alive_only_uses_signal_zero(monkeypatch) -> None:
|
||||
"""The liveness probe must never send a real (terminating) signal."""
|
||||
monkeypatch.setitem(
|
||||
|
||||
@@ -0,0 +1,570 @@
|
||||
"""Tests for the wrap-spawned proxy orphan watchdog."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
from starlette.responses import StreamingResponse
|
||||
|
||||
import headroom.proxy.server as server
|
||||
from headroom.proxy import orphan_watchdog as ow
|
||||
from headroom.proxy.server import ProxyConfig
|
||||
|
||||
|
||||
def _write_marker(clients_dir: Path, pid: int, **extra: object) -> Path:
|
||||
clients_dir.mkdir(parents=True, exist_ok=True)
|
||||
marker = clients_dir / f"{pid}.json"
|
||||
payload = {"pid": pid, "started_at": 1.0}
|
||||
payload.update(extra)
|
||||
marker.write_text(json.dumps(payload), encoding="utf-8")
|
||||
return marker
|
||||
|
||||
|
||||
class _Proxy:
|
||||
def __init__(self, *, port: int = 8787, active_sessions: int = 0) -> None:
|
||||
self.config = SimpleNamespace(port=port)
|
||||
self.ws_sessions = SimpleNamespace(active_count=lambda: active_sessions)
|
||||
self._active_requests = 0
|
||||
self._activity_generation = 0
|
||||
|
||||
@property
|
||||
def active_request_count(self) -> int:
|
||||
return self._active_requests
|
||||
|
||||
@property
|
||||
def activity_generation(self) -> int:
|
||||
return self._activity_generation
|
||||
|
||||
|
||||
class TestEnvConfig:
|
||||
def test_enabled_requires_wrap_owned_flag(self, monkeypatch) -> None:
|
||||
monkeypatch.delenv(ow.WRAP_OWNED_ENV, raising=False)
|
||||
assert ow.orphan_watchdog_enabled() is False
|
||||
monkeypatch.setenv(ow.WRAP_OWNED_ENV, "1")
|
||||
assert ow.orphan_watchdog_enabled() is True
|
||||
monkeypatch.setenv(ow.WRAP_OWNED_ENV, "0")
|
||||
assert ow.orphan_watchdog_enabled() is False
|
||||
# Explicit mapping wins over os.environ.
|
||||
assert ow.orphan_watchdog_enabled({ow.WRAP_OWNED_ENV: "true"}) is True
|
||||
|
||||
def test_grace_seconds_default_override_and_floor(self, monkeypatch) -> None:
|
||||
monkeypatch.delenv(ow.GRACE_SECONDS_ENV, raising=False)
|
||||
assert ow.orphan_grace_seconds() == ow.DEFAULT_GRACE_SECONDS
|
||||
monkeypatch.setenv(ow.GRACE_SECONDS_ENV, "120")
|
||||
assert ow.orphan_grace_seconds() == 120.0
|
||||
# Below the floor: clamped, so a typo cannot exit a proxy under a
|
||||
# client that has not registered yet.
|
||||
monkeypatch.setenv(ow.GRACE_SECONDS_ENV, "5")
|
||||
assert ow.orphan_grace_seconds() == ow.MIN_GRACE_SECONDS
|
||||
monkeypatch.setenv(ow.GRACE_SECONDS_ENV, "not-a-number")
|
||||
assert ow.orphan_grace_seconds() == ow.DEFAULT_GRACE_SECONDS
|
||||
|
||||
|
||||
class TestLiveClientPids:
|
||||
def test_live_marker_is_kept(self, tmp_path) -> None:
|
||||
_write_marker(tmp_path, os.getpid())
|
||||
assert ow.live_client_pids(tmp_path) == [os.getpid()]
|
||||
|
||||
def test_dead_pid_marker_is_pruned(self, tmp_path) -> None:
|
||||
# PID 2**31 - 1 is never alive.
|
||||
marker = _write_marker(tmp_path, 2**31 - 1)
|
||||
assert ow.live_client_pids(tmp_path) == []
|
||||
assert not marker.exists()
|
||||
|
||||
def test_recycled_pid_marker_is_pruned(self, tmp_path, monkeypatch) -> None:
|
||||
# Live PID (ours) but a recorded identity that provably differs.
|
||||
monkeypatch.setattr(ow, "identity_mismatch", lambda src, recorded, pid: True)
|
||||
marker = _write_marker(tmp_path, os.getpid(), start_src="psutil", start_time=1.0)
|
||||
assert ow.live_client_pids(tmp_path) == []
|
||||
assert not marker.exists()
|
||||
|
||||
def test_garbage_and_non_numeric_markers_are_ignored(self, tmp_path) -> None:
|
||||
(tmp_path / "not-a-pid.json").write_text("{}", encoding="utf-8")
|
||||
(tmp_path / f"{os.getpid()}.json").write_text("not json", encoding="utf-8")
|
||||
# Unparseable marker JSON is not proof of recycling: PID is alive, kept.
|
||||
assert ow.live_client_pids(tmp_path) == [os.getpid()]
|
||||
|
||||
def test_oversized_numeric_marker_is_pruned(self, tmp_path) -> None:
|
||||
marker = _write_marker(tmp_path, int("9" * 100))
|
||||
|
||||
assert ow.live_client_pids(tmp_path) == []
|
||||
assert not marker.exists()
|
||||
|
||||
def test_non_dict_marker_is_not_recycling_proof(self, tmp_path) -> None:
|
||||
(tmp_path / f"{os.getpid()}.json").write_text("[]", encoding="utf-8")
|
||||
assert ow.live_client_pids(tmp_path) == [os.getpid()]
|
||||
|
||||
def test_missing_dir_is_empty(self, tmp_path) -> None:
|
||||
assert ow.live_client_pids(tmp_path / "nope") == []
|
||||
|
||||
def test_client_scan_error_is_unknown(self, tmp_path, monkeypatch) -> None:
|
||||
tmp_path.mkdir(exist_ok=True)
|
||||
|
||||
def fail_scandir(path: object): # type: ignore[no-untyped-def]
|
||||
raise PermissionError("scan unavailable")
|
||||
|
||||
monkeypatch.setattr(ow.os, "scandir", fail_scandir)
|
||||
assert ow.live_client_pids(tmp_path) is None
|
||||
|
||||
|
||||
class TestWatchdogLoop:
|
||||
@staticmethod
|
||||
def _proxy(port: int = 8787, active_sessions: int = 0) -> _Proxy:
|
||||
return _Proxy(port=port, active_sessions=active_sessions)
|
||||
|
||||
def test_stops_after_grace_with_no_clients(self, tmp_path, monkeypatch) -> None:
|
||||
monkeypatch.setattr(ow, "proxy_clients_dir", lambda port: tmp_path)
|
||||
stopped: list[bool] = []
|
||||
|
||||
asyncio.run(
|
||||
ow.orphan_watchdog_loop(
|
||||
self._proxy(),
|
||||
grace_seconds=0.05,
|
||||
interval_seconds=0.01,
|
||||
stop=lambda: stopped.append(True),
|
||||
)
|
||||
)
|
||||
|
||||
assert stopped == [True]
|
||||
|
||||
def test_no_stop_while_client_alive(self, tmp_path, monkeypatch) -> None:
|
||||
monkeypatch.setattr(ow, "proxy_clients_dir", lambda port: tmp_path)
|
||||
_write_marker(tmp_path, os.getpid())
|
||||
stopped: list[bool] = []
|
||||
|
||||
async def run() -> None:
|
||||
task = asyncio.create_task(
|
||||
ow.orphan_watchdog_loop(
|
||||
self._proxy(),
|
||||
grace_seconds=0.3,
|
||||
interval_seconds=0.01,
|
||||
stop=lambda: stopped.append(True),
|
||||
)
|
||||
)
|
||||
await asyncio.sleep(0.4)
|
||||
task.cancel()
|
||||
with pytest.raises(asyncio.CancelledError):
|
||||
await task
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
assert stopped == []
|
||||
|
||||
def test_no_stop_while_ws_session_active(self, tmp_path, monkeypatch) -> None:
|
||||
monkeypatch.setattr(ow, "proxy_clients_dir", lambda port: tmp_path)
|
||||
stopped: list[bool] = []
|
||||
|
||||
async def run() -> None:
|
||||
task = asyncio.create_task(
|
||||
ow.orphan_watchdog_loop(
|
||||
self._proxy(active_sessions=1),
|
||||
grace_seconds=0.3,
|
||||
interval_seconds=0.01,
|
||||
stop=lambda: stopped.append(True),
|
||||
)
|
||||
)
|
||||
await asyncio.sleep(0.4)
|
||||
task.cancel()
|
||||
with pytest.raises(asyncio.CancelledError):
|
||||
await task
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
assert stopped == []
|
||||
|
||||
def test_no_stop_while_request_activity_continues(self, tmp_path, monkeypatch) -> None:
|
||||
"""Direct-HTTP / SSH-forwarded clients leave no marker and may hold no
|
||||
WS session; activity generation movement must still hold the exit."""
|
||||
monkeypatch.setattr(ow, "proxy_clients_dir", lambda port: tmp_path)
|
||||
counter = {"n": 0}
|
||||
proxy = self._proxy()
|
||||
stopped: list[bool] = []
|
||||
|
||||
async def serve_traffic() -> None:
|
||||
while True:
|
||||
await asyncio.sleep(0.05)
|
||||
counter["n"] += 1
|
||||
proxy._activity_generation = counter["n"]
|
||||
|
||||
async def run() -> None:
|
||||
traffic = asyncio.create_task(serve_traffic())
|
||||
task = asyncio.create_task(
|
||||
ow.orphan_watchdog_loop(
|
||||
proxy,
|
||||
grace_seconds=0.15,
|
||||
interval_seconds=0.01,
|
||||
stop=lambda: stopped.append(True),
|
||||
)
|
||||
)
|
||||
await asyncio.sleep(0.4)
|
||||
task.cancel()
|
||||
traffic.cancel()
|
||||
with pytest.raises(asyncio.CancelledError):
|
||||
await task
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
assert stopped == []
|
||||
|
||||
def test_streaming_request_resets_full_grace_after_completion(
|
||||
self, tmp_path, monkeypatch
|
||||
) -> None:
|
||||
monkeypatch.setattr(ow, "proxy_clients_dir", lambda port: tmp_path)
|
||||
proxy = self._proxy()
|
||||
app = FastAPI()
|
||||
app.add_middleware(server.ActivityMiddleware, proxy=proxy)
|
||||
stream_started = asyncio.Event()
|
||||
release_stream = asyncio.Event()
|
||||
stopped: list[bool] = []
|
||||
|
||||
@app.get("/stream")
|
||||
async def stream() -> StreamingResponse:
|
||||
async def body(): # type: ignore[no-untyped-def]
|
||||
stream_started.set()
|
||||
yield b"started"
|
||||
await release_stream.wait()
|
||||
yield b"finished"
|
||||
|
||||
return StreamingResponse(body())
|
||||
|
||||
async def run() -> None:
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
watchdog = asyncio.create_task(
|
||||
ow.orphan_watchdog_loop(
|
||||
proxy,
|
||||
grace_seconds=0.08,
|
||||
interval_seconds=0.01,
|
||||
stop=lambda: stopped.append(True),
|
||||
)
|
||||
)
|
||||
request = asyncio.create_task(client.get("/stream"))
|
||||
await asyncio.wait_for(stream_started.wait(), timeout=1)
|
||||
await asyncio.sleep(0.12)
|
||||
assert stopped == []
|
||||
assert proxy.active_request_count == 1
|
||||
|
||||
release_stream.set()
|
||||
await request
|
||||
await asyncio.sleep(0.04)
|
||||
assert stopped == []
|
||||
await asyncio.wait_for(watchdog, timeout=0.2)
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
assert stopped == [True]
|
||||
|
||||
def test_short_http_pulse_restarts_full_grace(self, tmp_path, monkeypatch) -> None:
|
||||
monkeypatch.setattr(ow, "proxy_clients_dir", lambda port: tmp_path)
|
||||
proxy = self._proxy()
|
||||
app = FastAPI()
|
||||
app.add_middleware(server.ActivityMiddleware, proxy=proxy)
|
||||
stopped: list[bool] = []
|
||||
|
||||
@app.get("/pulse")
|
||||
async def pulse() -> dict[str, bool]:
|
||||
return {"ok": True}
|
||||
|
||||
async def run() -> None:
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
watchdog = asyncio.create_task(
|
||||
ow.orphan_watchdog_loop(
|
||||
proxy,
|
||||
grace_seconds=0.08,
|
||||
interval_seconds=0.02,
|
||||
stop=lambda: stopped.append(True),
|
||||
)
|
||||
)
|
||||
await asyncio.sleep(0.05)
|
||||
before = proxy.activity_generation
|
||||
response = await client.get("/pulse")
|
||||
assert response.status_code == 200
|
||||
assert proxy.active_request_count == 0
|
||||
assert proxy.activity_generation == before + 2
|
||||
|
||||
# The pre-pulse idle window would have expired by now. The
|
||||
# observed generation change must start a full new window.
|
||||
await asyncio.sleep(0.06)
|
||||
assert stopped == []
|
||||
await asyncio.wait_for(watchdog, timeout=0.2)
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
assert stopped == [True]
|
||||
|
||||
def test_activity_middleware_tracks_held_websocket_scope(self, tmp_path, monkeypatch) -> None:
|
||||
monkeypatch.setattr(ow, "proxy_clients_dir", lambda port: tmp_path)
|
||||
proxy = self._proxy()
|
||||
entered = asyncio.Event()
|
||||
release = asyncio.Event()
|
||||
stopped: list[bool] = []
|
||||
|
||||
async def held_app(scope, receive, send): # type: ignore[no-untyped-def]
|
||||
entered.set()
|
||||
await release.wait()
|
||||
|
||||
async def unused(): # type: ignore[no-untyped-def]
|
||||
return {}
|
||||
|
||||
async def run() -> None:
|
||||
middleware = server.ActivityMiddleware(held_app, proxy=proxy)
|
||||
watchdog = asyncio.create_task(
|
||||
ow.orphan_watchdog_loop(
|
||||
proxy,
|
||||
grace_seconds=0.05,
|
||||
interval_seconds=0.01,
|
||||
stop=lambda: stopped.append(True),
|
||||
)
|
||||
)
|
||||
task = asyncio.create_task(middleware({"type": "websocket"}, unused, unused))
|
||||
await asyncio.wait_for(entered.wait(), timeout=1)
|
||||
assert proxy.active_request_count == 1
|
||||
assert proxy.activity_generation == 1
|
||||
await asyncio.sleep(0.08)
|
||||
assert stopped == []
|
||||
|
||||
release.set()
|
||||
await task
|
||||
assert proxy.active_request_count == 0
|
||||
assert proxy.activity_generation == 2
|
||||
await asyncio.wait_for(watchdog, timeout=0.2)
|
||||
|
||||
asyncio.run(run())
|
||||
assert stopped == [True]
|
||||
|
||||
def test_registry_failure_resets_grace_until_activity_is_observable(
|
||||
self, tmp_path, monkeypatch
|
||||
) -> None:
|
||||
monkeypatch.setattr(ow, "proxy_clients_dir", lambda port: tmp_path)
|
||||
observable = False
|
||||
|
||||
def active_count() -> int:
|
||||
if not observable:
|
||||
raise RuntimeError("registry unavailable")
|
||||
return 0
|
||||
|
||||
proxy = self._proxy()
|
||||
proxy.ws_sessions = SimpleNamespace(active_count=active_count)
|
||||
stopped: list[bool] = []
|
||||
|
||||
async def run() -> None:
|
||||
nonlocal observable
|
||||
watchdog = asyncio.create_task(
|
||||
ow.orphan_watchdog_loop(
|
||||
proxy,
|
||||
grace_seconds=0.08,
|
||||
interval_seconds=0.01,
|
||||
stop=lambda: stopped.append(True),
|
||||
)
|
||||
)
|
||||
await asyncio.sleep(0.12)
|
||||
assert stopped == []
|
||||
|
||||
observable = True
|
||||
await asyncio.sleep(0.04)
|
||||
assert stopped == []
|
||||
await asyncio.wait_for(watchdog, timeout=0.2)
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
assert stopped == [True]
|
||||
|
||||
@pytest.mark.parametrize("unknown_signal", ["clients", "activity", "generation"])
|
||||
def test_unknown_activity_signal_resets_grace_until_observable(
|
||||
self, unknown_signal, tmp_path, monkeypatch
|
||||
) -> None:
|
||||
observable = False
|
||||
proxy = self._proxy()
|
||||
if unknown_signal == "clients":
|
||||
monkeypatch.setattr(
|
||||
ow,
|
||||
"live_client_pids",
|
||||
lambda clients_dir: [] if observable else None,
|
||||
)
|
||||
elif unknown_signal == "activity":
|
||||
monkeypatch.setattr(ow, "proxy_clients_dir", lambda port: tmp_path)
|
||||
proxy._active_requests = None
|
||||
else:
|
||||
monkeypatch.setattr(ow, "proxy_clients_dir", lambda port: tmp_path)
|
||||
proxy._activity_generation = None
|
||||
stopped: list[bool] = []
|
||||
|
||||
async def run() -> None:
|
||||
nonlocal observable
|
||||
watchdog = asyncio.create_task(
|
||||
ow.orphan_watchdog_loop(
|
||||
proxy,
|
||||
grace_seconds=0.08,
|
||||
interval_seconds=0.01,
|
||||
stop=lambda: stopped.append(True),
|
||||
)
|
||||
)
|
||||
await asyncio.sleep(0.12)
|
||||
assert stopped == []
|
||||
|
||||
observable = True
|
||||
proxy._active_requests = 0
|
||||
proxy._activity_generation = 0
|
||||
await asyncio.sleep(0.04)
|
||||
assert stopped == []
|
||||
await asyncio.wait_for(watchdog, timeout=0.2)
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
assert stopped == [True]
|
||||
|
||||
def test_grace_restarts_when_client_returns(self, tmp_path, monkeypatch) -> None:
|
||||
monkeypatch.setattr(ow, "proxy_clients_dir", lambda port: tmp_path)
|
||||
stopped: list[bool] = []
|
||||
|
||||
async def run() -> None:
|
||||
task = asyncio.create_task(
|
||||
ow.orphan_watchdog_loop(
|
||||
self._proxy(),
|
||||
grace_seconds=0.5,
|
||||
interval_seconds=0.01,
|
||||
stop=lambda: stopped.append(True),
|
||||
)
|
||||
)
|
||||
# Let the idle clock accumulate, then register a live client: the
|
||||
# clock must reset, not fire late.
|
||||
await asyncio.sleep(0.2)
|
||||
_write_marker(tmp_path, os.getpid())
|
||||
await asyncio.sleep(0.7)
|
||||
task.cancel()
|
||||
with pytest.raises(asyncio.CancelledError):
|
||||
await task
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
assert stopped == []
|
||||
|
||||
def test_missing_ws_registry_counts_as_idle(self, tmp_path, monkeypatch) -> None:
|
||||
monkeypatch.setattr(ow, "proxy_clients_dir", lambda port: tmp_path)
|
||||
proxy = self._proxy()
|
||||
proxy.ws_sessions = None
|
||||
stopped: list[bool] = []
|
||||
|
||||
asyncio.run(
|
||||
ow.orphan_watchdog_loop(
|
||||
proxy,
|
||||
grace_seconds=0.05,
|
||||
interval_seconds=0.01,
|
||||
stop=lambda: stopped.append(True),
|
||||
)
|
||||
)
|
||||
|
||||
assert stopped == [True]
|
||||
|
||||
|
||||
class TestDefensiveHelpers:
|
||||
def test_non_dict_marker_json_is_not_recycling_proof(self, tmp_path) -> None:
|
||||
# Valid JSON but not an object: only a dict record can prove recycling.
|
||||
marker = _write_marker(tmp_path, os.getpid())
|
||||
marker.write_text("[1, 2]", encoding="utf-8")
|
||||
assert ow._marker_pid_recycled(marker, os.getpid()) is False
|
||||
|
||||
def test_unlink_oserror_is_tolerated(self, tmp_path, monkeypatch) -> None:
|
||||
dead_pid = 2**31 - 1 # never alive
|
||||
marker = _write_marker(tmp_path, dead_pid)
|
||||
real_unlink = Path.unlink
|
||||
|
||||
def flaky_unlink(self: Path, *args: object, **kwargs: object) -> None:
|
||||
if self == marker:
|
||||
raise OSError("simulated EPERM")
|
||||
real_unlink(self, *args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(Path, "unlink", flaky_unlink)
|
||||
# The scan tolerates the failed prune and still reports no live clients.
|
||||
assert ow.live_client_pids(tmp_path) == []
|
||||
assert marker.exists()
|
||||
|
||||
def test_active_session_count_reports_registry_errors_as_unknown(self) -> None:
|
||||
def boom() -> int:
|
||||
raise RuntimeError("registry exploded")
|
||||
|
||||
proxy = SimpleNamespace(ws_sessions=SimpleNamespace(active_count=boom))
|
||||
assert ow._active_session_count(proxy) is None
|
||||
|
||||
def test_active_request_count_is_unknown_when_unavailable(self) -> None:
|
||||
class BrokenProxy:
|
||||
@property
|
||||
def active_request_count(self) -> int:
|
||||
raise RuntimeError("counter unavailable")
|
||||
|
||||
assert ow._active_request_count(SimpleNamespace()) is None
|
||||
assert ow._active_request_count(BrokenProxy()) is None
|
||||
|
||||
def test_activity_generation_is_unknown_when_unavailable(self) -> None:
|
||||
assert ow._activity_generation(SimpleNamespace()) is None
|
||||
|
||||
def test_default_stop_raises_sigterm(self, monkeypatch) -> None:
|
||||
sent: list[int] = []
|
||||
monkeypatch.setattr(ow.signal, "raise_signal", lambda sig: sent.append(sig))
|
||||
ow._default_stop()
|
||||
assert sent == [ow.signal.SIGTERM]
|
||||
|
||||
|
||||
class TestServerWiring:
|
||||
"""create_app must start the watchdog only for wrap-spawned single-worker
|
||||
proxies, and cancel it cleanly on shutdown."""
|
||||
|
||||
@staticmethod
|
||||
def _config() -> ProxyConfig:
|
||||
return ProxyConfig(
|
||||
optimize=False,
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
cost_tracking_enabled=False,
|
||||
periodic_malloc_trim_enabled=False,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
async def _fake_loop(*args: object, **kwargs: object) -> None:
|
||||
await asyncio.sleep(3600)
|
||||
|
||||
def test_starts_and_stops_watchdog_when_wrap_owned(self, monkeypatch) -> None:
|
||||
monkeypatch.setenv(ow.WRAP_OWNED_ENV, "1")
|
||||
monkeypatch.delenv(server._MULTI_WORKER_CONFIG_ENV, raising=False)
|
||||
monkeypatch.setattr(server, "orphan_watchdog_loop", self._fake_loop)
|
||||
|
||||
app = server.create_app(self._config())
|
||||
with TestClient(app) as client:
|
||||
task = client.app.state.orphan_watchdog_task
|
||||
assert task is not None
|
||||
assert not task.done()
|
||||
# Lifespan teardown cancelled it and cleared the state slot.
|
||||
assert app.state.orphan_watchdog_task is None
|
||||
|
||||
def test_activity_middleware_is_outermost(self) -> None:
|
||||
app = server.create_app(self._config())
|
||||
assert app.user_middleware[0].cls is server.ActivityMiddleware
|
||||
assert app.state.proxy.active_request_count == 0
|
||||
assert app.state.proxy.activity_generation == 0
|
||||
|
||||
def test_skips_watchdog_for_multi_worker(self, monkeypatch) -> None:
|
||||
monkeypatch.setenv(ow.WRAP_OWNED_ENV, "1")
|
||||
monkeypatch.setenv(server._MULTI_WORKER_CONFIG_ENV, "{}")
|
||||
monkeypatch.setattr(server, "orphan_watchdog_loop", self._fake_loop)
|
||||
|
||||
app = server.create_app(self._config())
|
||||
with TestClient(app) as client:
|
||||
assert client.app.state.orphan_watchdog_task is None
|
||||
|
||||
def test_skips_watchdog_when_not_wrap_owned(self, monkeypatch) -> None:
|
||||
monkeypatch.delenv(ow.WRAP_OWNED_ENV, raising=False)
|
||||
monkeypatch.delenv(server._MULTI_WORKER_CONFIG_ENV, raising=False)
|
||||
monkeypatch.setattr(server, "orphan_watchdog_loop", self._fake_loop)
|
||||
|
||||
app = server.create_app(self._config())
|
||||
with TestClient(app) as client:
|
||||
assert client.app.state.orphan_watchdog_task is None
|
||||
@@ -0,0 +1,82 @@
|
||||
"""Tests for headroom._subprocess proc_identity / identity_mismatch.
|
||||
|
||||
proc_identity has platform-exclusive legs (psutil vs /proc), so no single CI
|
||||
runner reaches both; these tests drive each leg explicitly with fakes.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import builtins
|
||||
import io
|
||||
import os
|
||||
import sys
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from headroom import _subprocess as sub
|
||||
|
||||
|
||||
class TestProcIdentity:
|
||||
def test_psutil_leg_with_stub(self, monkeypatch) -> None:
|
||||
# Deterministic: psutil is an optional dep, so drive the leg with a stub
|
||||
# instead of importorskip (line must be covered on psutil-less CI too).
|
||||
stub = SimpleNamespace(Process=lambda pid: SimpleNamespace(create_time=lambda: 1234.5))
|
||||
monkeypatch.setitem(sys.modules, "psutil", stub)
|
||||
assert sub.proc_identity(999) == ("psutil", 1234.5)
|
||||
|
||||
def test_psutil_leg_real_when_installed(self) -> None:
|
||||
pytest.importorskip("psutil")
|
||||
ident = sub.proc_identity(os.getpid())
|
||||
assert ident is not None
|
||||
src, start = ident
|
||||
assert src == "psutil"
|
||||
assert isinstance(start, float) and start > 0
|
||||
|
||||
def test_proc_leg_when_psutil_unavailable(self, monkeypatch) -> None:
|
||||
monkeypatch.setitem(sys.modules, "psutil", None) # import raises ImportError
|
||||
# /proc/<pid>/stat: fields after the final ")" are field 3 onwards, so
|
||||
# field 22 (starttime) is index 19 of the split — here 987654.
|
||||
stat = b"123 (python (test)) S 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 987654 21"
|
||||
real_open = builtins.open
|
||||
|
||||
def fake_open(path, *args, **kwargs):
|
||||
if str(path).startswith("/proc/"):
|
||||
return io.BytesIO(stat)
|
||||
return real_open(path, *args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(builtins, "open", fake_open)
|
||||
assert sub.proc_identity(123) == ("proc", 987654.0)
|
||||
|
||||
def test_returns_none_when_both_sources_fail(self, monkeypatch) -> None:
|
||||
monkeypatch.setitem(sys.modules, "psutil", None)
|
||||
real_open = builtins.open
|
||||
|
||||
def fake_open(path, *args, **kwargs):
|
||||
if str(path).startswith("/proc/"):
|
||||
raise OSError("no /proc here")
|
||||
return real_open(path, *args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(builtins, "open", fake_open)
|
||||
assert sub.proc_identity(123) is None
|
||||
|
||||
|
||||
class TestIdentityMismatch:
|
||||
def test_legacy_record_never_mismatches(self) -> None:
|
||||
assert sub.identity_mismatch(None, None, os.getpid()) is False
|
||||
assert sub.identity_mismatch("psutil", "not-a-number", os.getpid()) is False
|
||||
|
||||
def test_unknown_identity_never_mismatches(self) -> None:
|
||||
assert sub.identity_mismatch("psutil", 1.0, 123, identity_fn=lambda pid: None) is False
|
||||
|
||||
def test_source_mismatch_never_mismatches(self) -> None:
|
||||
ident = lambda pid: ("proc", 1.0) # noqa: E731
|
||||
assert sub.identity_mismatch("psutil", 1.0, 123, identity_fn=ident) is False
|
||||
|
||||
def test_same_start_time_is_not_a_mismatch(self) -> None:
|
||||
ident = lambda pid: ("psutil", 1000.5) # noqa: E731
|
||||
assert sub.identity_mismatch("psutil", 1000.2, 123, identity_fn=ident) is False
|
||||
|
||||
def test_distant_start_time_proves_recycling(self) -> None:
|
||||
ident = lambda pid: ("psutil", 2000.0) # noqa: E731
|
||||
assert sub.identity_mismatch("psutil", 1000.0, 123, identity_fn=ident) is True
|
||||
Reference in New Issue
Block a user