mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(wrap): handle SIGHUP on the shared proxy paths, not just claude (#3331)
## Description
Closing a terminal (or `tmux kill-session`) delivers **SIGHUP**, not
SIGTERM. `claude()` learned to catch it in #1768 / #3205, but the two
shared paths that every *other* wrapped tool goes through still register
only SIGINT and SIGTERM:
| Path | Tools | SIGHUP before this PR |
| --- | --- | --- |
| `claude()` (`wrap.py:5196`) | claude | ✅ handled (#3205) |
| `_launch_tool` (`wrap.py:4713`) | codex, aider, copilot, goose,
openhands, … | ❌ |
| `_run_proxy_only_watcher` (`wrap.py:3319`) | cursor, cline, continue |
❌ |
Unhandled SIGHUP kills the wrapper outright, so the `finally: cleanup()`
that terminates the proxy never runs. The proxy sits in its own session,
survives, and is reparented to PID 1 — a leaked listener holding the
port. Nothing reaps it later either: the dead wrapper also never removed
its `clients/<port>/<pid>.json` marker, so `_make_cleanup`'s
`_other_clients_exist()` can see a phantom client and decline to
terminate the proxy.
`_run_proxy_only_watcher` is a notable case — it already special-cases
Windows' `SIGBREAK` while leaving the POSIX terminal-close signal
unhandled.
Found from a leaked proxy in the wild (8d19h old, `PPID 1`), not by
reading the code. Details under Real Behavior Proof.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `_launch_tool`: register `SIGHUP` → `_exit_on_signal` alongside
SIGTERM, guarded by `hasattr(signal, "SIGHUP")`.
- `_run_proxy_only_watcher`: register `SIGHUP` → `_signal_shutdown`
alongside SIGTERM, same guard.
- New `tests/test_cli/test_wrap_sighup_proxy_leak.py` — 3 tests, all
fail before / pass after.
Both hunks are the same 5-line shape the `claude` path already uses. No
new behavior, no new dependencies, no config or CLI surface — just
applied to the paths that were missed.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
Two tests call the shipping `_launch_tool` / `_run_proxy_only_watcher`
in-process, with only `_ensure_proxy` stubbed, and assert
`signal.getsignal(SIGHUP)` is the handler that was installed (a fixture
snapshots and restores process handlers). The third goes past the
limitation noted on the existing `claude` test ("Full signal delivery
isn't practical to exercise via CliRunner"): it spawns a **real**
wrapper subprocess, sends a **real** SIGHUP, and asserts the proxy is
actually reaped. It runs the wrapper in its own session and reaps the
group afterwards, so it leaks nothing on either outcome.
### Test Output
```text
########## WITHOUT FIX (tests kept, wrap.py reverted to HEAD~1) ##########
FAILED tests/test_cli/test_wrap_sighup_proxy_leak.py::test_launch_tool_installs_sighup_handler
FAILED tests/test_cli/test_wrap_sighup_proxy_leak.py::test_proxy_only_watcher_installs_sighup_handler
FAILED tests/test_cli/test_wrap_sighup_proxy_leak.py::test_sighup_on_launch_tool_reaps_the_proxy
======================== 3 failed, 1 warning in 16.57s =========================
E assert <Handlers.SIG_DFL: 0> is <function _exit_on_signal at 0x10b314360>
E assert <Handlers.SIG_DFL: 0> not in (<Handlers.SIG_DFL: 0>, <Handlers.SIG_IGN: 1>, None)
E AssertionError: proxy 75403 outlived the wrapper -- it would be reparented to PID 1 and leak
########## WITH FIX ##########
========================= 3 passed, 1 warning in 1.50s =========================
########## Regression: existing wrap suites + new file ##########
$ pytest tests/test_cli/test_wrap_stale_marker.py tests/test_cli/test_wrap_codex.py \
tests/test_wrap_concurrent_settings.py tests/test_cli/test_wrap_sighup_proxy_leak.py -q
======================= 114 passed, 1 warning in 58.33s ========================
########## Patch coverage of the four added lines ##########
line 3324 COVERED if hasattr(signal, "SIGHUP"):
line 3325 COVERED signal.signal(signal.SIGHUP, _signal_shutdown)
line 4724 COVERED if hasattr(signal, "SIGHUP"):
line 4725 COVERED signal.signal(signal.SIGHUP, _exit_on_signal)
########## ruff / mypy ##########
$ ruff check headroom/cli/wrap.py tests/test_cli/test_wrap_sighup_proxy_leak.py
All checks passed!
$ ruff format --check headroom/cli/wrap.py tests/test_cli/test_wrap_sighup_proxy_leak.py
2 files already formatted
$ mypy tests/test_cli/test_wrap_sighup_proxy_leak.py --ignore-missing-imports --follow-imports=skip
Success: no issues found in 1 source file
$ mypy headroom/cli/wrap.py --ignore-missing-imports --follow-imports=skip
before: 20 errors, after: 20 errors -> IDENTICAL, no new type errors
(20 pre-existing no-any-return errors, an artifact of --follow-imports=skip)
```
## Real Behavior Proof
- Environment: macOS 15 (Darwin 25.5.0, arm64), Python 3.13.14,
headroom-ai 0.32.1 at discovery; reproduced and fixed against `main` @
32d7ca4 (0.37.0). Leaking client was `codex-tui` 0.149.0 (Pattern-A →
`_launch_tool`).
- Exact command / steps: run any Pattern-A wrap (`headroom codex`), then
close the terminal window instead of pressing Ctrl-C. The wrapper
receives SIGHUP, dies unhandled, and `headroom proxy` is left behind on
PID 1. `kill -HUP <wrapper-pid>` reproduces it identically without the
GUI.
- Observed result: **before the fix**, a `headroom proxy --port 8787`
had been running 8 days 19 hours with `PPID 1`, no controlling TTY,
`STAT Ss` (its own session leader):
```
PID PPID PGID STAT STARTED ELAPSED COMMAND
25125 1 25125 Ss Thu Aug 20 00:38:08 2026 08-19:16:08 python -m
headroom.cli proxy --port 8787
```
It still held the port with no client attached, and the last real
request was ~4h earlier — everything after it is the 5-minute heartbeat:
```
$ lsof -nP -iTCP:8787
python3.1 25125 ... TCP 127.0.0.1:8787 (LISTEN) # LISTEN only, no
ESTABLISHED
16:04:21 - headroom.proxy - INFO - event=proxy_inbound_request ...
client=codex # last real request
19:50:14 - headroom.proxy - INFO - TOIN: 6230 patterns, 525344
compressions, ... # heartbeat only
```
Plus the phantom client marker the dead wrapper never removed — its PID
long gone, which is what keeps a later wrapper from reaping the proxy:
```
$ cat ~/.headroom/clients/8787/76759.json
{"pid": 76759, "started_at": 1787378087.6705542}
$ ps -p 76759
(no such process)
```
**After the fix**, the same scenario at process level is
`test_sighup_on_launch_tool_reaps_the_proxy` — a real SIGHUP to a real
wrapper. Before the patch the stand-in proxy outlives the wrapper
(`proxy 75403 outlived the wrapper`); after it, the proxy is reaped and
the wrapper exits 129. Separately confirmed with the shipping
`_exit_on_signal`: without the SIGHUP registration the `finally` block
never executed (exit 129, no cleanup marker written); with it, the
`finally` ran (exit 129, cleanup marker written).
- Not tested: Windows and Linux — verified on macOS only (CI covers the
other platforms). The `_run_proxy_only_watcher` path is covered by the
in-process registration test but was not driven through a real
cursor/cline session. No test against a live upstream provider: the
stand-in proxy is a plain child process, since what is under test is
process teardown, not proxying.
## Runtime Rollout Safety
- Rollout-managed feature(s): none. This is not behind a rollout channel
or feature flag.
- Minimum rollout channel: N/A — ships to all users with the release.
- Stable/default behavior changed: yes, and only in the failure case.
SIGHUP to a wrap wrapper previously killed the process with no cleanup;
it now runs the same `finally: cleanup()` that SIGTERM and Ctrl-C
already run. Ctrl-C (SIGINT), SIGTERM, and normal exit paths are
untouched. No API, CLI, config, or wire-format change.
- Kill switch / disable path: none added. The pre-existing behavior was
an unhandled fatal signal, so there is nothing to fall back to; a user
who wants the proxy to outlive its wrapper should use `headroom install`
(a supervised persistent deployment) rather than a leaked child.
- Unsafe override required: no.
- Qualification impact: none expected. The handler is registered under
`hasattr(signal, "SIGHUP")`, so Windows is unaffected; on POSIX the
added work is one `signal.signal` call at wrapper startup.
- Rollback path: revert this commit. The two hunks are self-contained (5
lines each, no shared state, no persisted format), so reverting restores
the prior behavior exactly.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A — process lifecycle change with no UI surface. Terminal output is
under Real Behavior Proof.
## Additional Notes
- **Documentation checklist item is N/A**: this restores the intended
documented behavior (the wrapper cleans up its proxy) rather than
changing it, so there is nothing to document.
- **Review feedback addressed** in `c49b15b`: the two registration tests
originally pattern-matched `inspect.getsource`, which never executes the
guarded lines — Codecov correctly flagged 0% patch coverage, and the
assertions couldn't distinguish a real handler from matching text. They
now execute the real registration in-process and assert on
`signal.getsignal`. The harness pid-file write also uses a context
manager instead of relying on CPython refcounting to flush.
- **Out of scope, flagging not fixing**: `install/runtime.py:299`
(`run_foreground`) has the same SIGINT/SIGTERM-only shape. It matters
much less — supervisors send SIGTERM — so I left it alone rather than
widen the diff. Happy to include it if you'd prefer.
- The pre-existing `claude`-path registration (`wrap.py:5199`/`5202`) is
still only covered by the source-inspection test from #3205 and shows as
uncovered. Not touched here, but the same in-process approach would
cover it if you want a follow-up.
---------
Co-authored-by: sesepark <250551845+sesepark@users.noreply.github.com>
This commit is contained in:
@@ -3382,6 +3382,11 @@ def _run_proxy_only_watcher(
|
||||
|
||||
signal.signal(signal.SIGINT, _signal_shutdown)
|
||||
signal.signal(signal.SIGTERM, _signal_shutdown)
|
||||
# Terminal close / tmux kill-session sends SIGHUP, not SIGTERM. Without
|
||||
# this the watcher dies unhandled and the proxy it spawned is reparented
|
||||
# to PID 1 and leaks. Mirrors the SIGHUP handling on the `claude` path.
|
||||
if hasattr(signal, "SIGHUP"):
|
||||
signal.signal(signal.SIGHUP, _signal_shutdown)
|
||||
# Windows exposes Ctrl+Break as SIGBREAK rather than SIGINT. Test runners,
|
||||
# IDE terminals, and process supervisors commonly use Ctrl+Break to target
|
||||
# a newly created process group, so route it through the same graceful
|
||||
@@ -4776,6 +4781,12 @@ def _launch_tool(
|
||||
cleanup = _make_cleanup(proxy_holder, port_holder)
|
||||
signal.signal(signal.SIGINT, _ignore_child_sigint)
|
||||
signal.signal(signal.SIGTERM, _exit_on_signal)
|
||||
# Terminal close / tmux kill-session sends SIGHUP, not SIGTERM. Without
|
||||
# this the wrapper dies unhandled, the `finally` below never runs, and
|
||||
# the proxy it spawned is reparented to PID 1 and leaks. Mirrors the
|
||||
# SIGHUP handling on the `claude` path.
|
||||
if hasattr(signal, "SIGHUP"):
|
||||
signal.signal(signal.SIGHUP, _exit_on_signal)
|
||||
|
||||
try:
|
||||
click.echo()
|
||||
|
||||
@@ -0,0 +1,224 @@
|
||||
"""SIGHUP must tear the proxy down on every wrap path, not just ``claude``.
|
||||
|
||||
Closing a terminal (or ``tmux kill-session``) delivers SIGHUP, not SIGTERM.
|
||||
``claude()`` learned to catch it in #1768/#3205, but the two shared paths that
|
||||
every other wrapped tool goes through did not:
|
||||
|
||||
* ``_launch_tool`` -- Pattern-A (codex, aider, copilot, goose, openhands, ...)
|
||||
* ``_run_proxy_only_watcher`` -- Pattern-B (cursor, cline, continue)
|
||||
|
||||
Unhandled SIGHUP kills the wrapper outright, so the ``finally: cleanup()`` that
|
||||
terminates the proxy never runs. The proxy is in its own session, survives, and
|
||||
is reparented to PID 1 -- a leaked listener that no later wrap invocation will
|
||||
ever reap.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import textwrap
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from headroom.cli import wrap as wrap_cli
|
||||
|
||||
requires_sighup = pytest.mark.skipif(not hasattr(signal, "SIGHUP"), reason="SIGHUP is POSIX-only")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def restore_signal_handlers() -> Iterator[None]:
|
||||
"""Snapshot and restore the handlers the wrap paths install.
|
||||
|
||||
These tests call the real registration code in-process, so without this the
|
||||
installed handlers would leak into the rest of the session.
|
||||
"""
|
||||
names = [s for s in ("SIGINT", "SIGTERM", "SIGHUP") if hasattr(signal, s)]
|
||||
saved = {getattr(signal, n): signal.getsignal(getattr(signal, n)) for n in names}
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
for sig, handler in saved.items():
|
||||
signal.signal(sig, handler)
|
||||
|
||||
|
||||
class _AlreadyExitedProxy:
|
||||
"""Stand-in proxy that reports as dead, so the watcher loop exits at once."""
|
||||
|
||||
def poll(self) -> int:
|
||||
return 0
|
||||
|
||||
def terminate(self) -> None: # pragma: no cover - not reached
|
||||
pass
|
||||
|
||||
def wait(self, timeout: float | None = None) -> int: # pragma: no cover
|
||||
return 0
|
||||
|
||||
|
||||
@requires_sighup
|
||||
def test_launch_tool_installs_sighup_handler(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, restore_signal_handlers: None
|
||||
) -> None:
|
||||
"""Pattern-A tools (codex et al.) share ``_launch_tool``'s handlers.
|
||||
|
||||
Asserts the handler is actually installed by running the real registration,
|
||||
rather than pattern-matching the source.
|
||||
"""
|
||||
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path / "workspace"))
|
||||
monkeypatch.setattr(wrap_cli, "_ensure_proxy", lambda *a, **k: (None, 18787))
|
||||
monkeypatch.setattr(wrap_cli, "_push_runtime_env", lambda *a, **k: None)
|
||||
|
||||
observed: dict[str, Any] = {}
|
||||
|
||||
def _capture_during_child_run(cmd: list[str], **kwargs: Any) -> Any:
|
||||
# Sampled where the real wrapper blocks on the child CLI -- the window
|
||||
# in which a terminal close actually arrives.
|
||||
observed["SIGHUP"] = signal.getsignal(signal.SIGHUP)
|
||||
observed["SIGTERM"] = signal.getsignal(signal.SIGTERM)
|
||||
return subprocess.CompletedProcess(cmd, 0)
|
||||
|
||||
monkeypatch.setattr(wrap_cli.subprocess, "run", _capture_during_child_run)
|
||||
|
||||
with pytest.raises(SystemExit):
|
||||
wrap_cli._launch_tool(
|
||||
binary=sys.executable,
|
||||
args=(),
|
||||
env={},
|
||||
port=18787,
|
||||
no_proxy=False,
|
||||
tool_label="sighup-test",
|
||||
env_vars_display=[],
|
||||
)
|
||||
|
||||
assert observed["SIGHUP"] is wrap_cli._exit_on_signal
|
||||
assert observed["SIGTERM"] is wrap_cli._exit_on_signal
|
||||
|
||||
|
||||
@requires_sighup
|
||||
def test_proxy_only_watcher_installs_sighup_handler(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, restore_signal_handlers: None
|
||||
) -> None:
|
||||
"""Pattern-B tools (cursor et al.) share ``_run_proxy_only_watcher``.
|
||||
|
||||
This path already special-cased Windows' SIGBREAK while leaving the POSIX
|
||||
terminal-close signal unhandled.
|
||||
"""
|
||||
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path / "workspace"))
|
||||
monkeypatch.setattr(wrap_cli, "_ensure_proxy", lambda *a, **k: (_AlreadyExitedProxy(), 18787))
|
||||
monkeypatch.setattr(wrap_cli, "_push_runtime_env", lambda *a, **k: None)
|
||||
|
||||
with pytest.raises(SystemExit):
|
||||
wrap_cli._run_proxy_only_watcher(
|
||||
agent_label="sighup-test",
|
||||
port=18787,
|
||||
no_proxy=False,
|
||||
learn=False,
|
||||
memory=False,
|
||||
agent_type="unknown",
|
||||
print_setup_lines=lambda _port: None,
|
||||
)
|
||||
|
||||
# `_signal_shutdown` is a closure, so identity against a module attribute
|
||||
# is not available -- assert a real handler replaced the default instead.
|
||||
handler = signal.getsignal(signal.SIGHUP)
|
||||
assert handler not in (signal.SIG_DFL, signal.SIG_IGN, None)
|
||||
assert getattr(handler, "__name__", "") == "_signal_shutdown"
|
||||
assert signal.getsignal(signal.SIGTERM) is handler
|
||||
|
||||
|
||||
# Harness driving the real `_launch_tool` under a real SIGHUP. Only
|
||||
# `_ensure_proxy` is stubbed -- to a live child process standing in for the
|
||||
# proxy -- so the signal handler, the `finally`, and `_make_cleanup`'s
|
||||
# terminate are all the shipping implementations.
|
||||
_HARNESS = textwrap.dedent(
|
||||
"""
|
||||
import os, subprocess, sys
|
||||
from headroom.cli import wrap
|
||||
|
||||
port = 18787
|
||||
proxy = subprocess.Popen([sys.executable, "-c", "import time; time.sleep(300)"])
|
||||
with open(sys.argv[1], "w") as fh:
|
||||
fh.write(str(proxy.pid))
|
||||
|
||||
wrap._ensure_proxy = lambda *a, **k: (proxy, port)
|
||||
wrap._live_proxy_clients = lambda *a, **k: [] # no other clients -> may reap
|
||||
wrap._push_runtime_env = lambda *a, **k: None
|
||||
|
||||
with open(sys.argv[2], "w"): # ready
|
||||
pass
|
||||
wrap._launch_tool(
|
||||
binary=sys.executable,
|
||||
args=("-c", "import time; time.sleep(300)"),
|
||||
env=dict(os.environ),
|
||||
port=port,
|
||||
no_proxy=False,
|
||||
tool_label="sighup-test",
|
||||
env_vars_display=[],
|
||||
)
|
||||
"""
|
||||
)
|
||||
|
||||
|
||||
def _wait_for(predicate: Any, timeout: float = 15.0, interval: float = 0.05) -> bool:
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
if predicate():
|
||||
return True
|
||||
time.sleep(interval)
|
||||
return False
|
||||
|
||||
|
||||
def _pid_alive(pid: int) -> bool:
|
||||
try:
|
||||
os.kill(pid, 0)
|
||||
except ProcessLookupError:
|
||||
return False
|
||||
except PermissionError:
|
||||
return True
|
||||
return True
|
||||
|
||||
|
||||
@requires_sighup
|
||||
def test_sighup_on_launch_tool_reaps_the_proxy(tmp_path: Path) -> None:
|
||||
"""End-to-end: real SIGHUP to a real wrapper must not leak the proxy."""
|
||||
harness = tmp_path / "harness.py"
|
||||
harness.write_text(_HARNESS, encoding="utf-8")
|
||||
pid_file = tmp_path / "proxy.pid"
|
||||
ready = tmp_path / "ready"
|
||||
|
||||
env = dict(os.environ)
|
||||
env["HEADROOM_WORKSPACE_DIR"] = str(tmp_path / "workspace")
|
||||
|
||||
# Own session/process group: the wrapper's children (stand-in proxy and
|
||||
# stand-in CLI) inherit it, so the cleanup below can reap the whole tree
|
||||
# without signalling the pytest process.
|
||||
wrapper = subprocess.Popen(
|
||||
[sys.executable, str(harness), str(pid_file), str(ready)],
|
||||
env=env,
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
start_new_session=True,
|
||||
)
|
||||
try:
|
||||
assert _wait_for(ready.exists), "harness never reached _launch_tool"
|
||||
proxy_pid = int(pid_file.read_text())
|
||||
assert _pid_alive(proxy_pid), "stand-in proxy should be running"
|
||||
|
||||
os.kill(wrapper.pid, signal.SIGHUP)
|
||||
|
||||
assert _wait_for(lambda: wrapper.poll() is not None), "wrapper survived SIGHUP"
|
||||
assert _wait_for(lambda: not _pid_alive(proxy_pid)), (
|
||||
f"proxy {proxy_pid} outlived the wrapper -- it would be reparented to PID 1 and leak"
|
||||
)
|
||||
finally:
|
||||
try:
|
||||
os.killpg(wrapper.pid, signal.SIGKILL)
|
||||
except (ProcessLookupError, PermissionError):
|
||||
pass
|
||||
wrapper.wait(timeout=10)
|
||||
Reference in New Issue
Block a user