fix(install/windows): back persistent-service with an always-on task (#1866) (#2460)

## Description

`headroom install apply --preset persistent-service` failed on Windows.
It registered `cmd.exe /c run-headroom.cmd` as a service via `sc.exe
create`, but a plain console process never performs the SCM
`StartServiceCtrlDispatcher` handshake, so Windows killed it after the
30s start timeout (`StartService FAILED 1053`) and left an empty
`deploy/default` profile. Fixes #1866.

Rather than add a service-wrapper dependency, the Windows service is now
backed by scheduled tasks: a `BootTrigger` task launches the foreground
proxy (`run-headroom`) at boot so it stays always-on, plus a 5-minute
health task as a watchdog. Both are registered from the same S4U/hidden
task XML introduced for #2453, so no console window ever flashes.

> Stacked on #2459 (the #2453 S4U/hidden XML machinery). Until that
merges, this PR's diff includes its commit; it collapses automatically
once #2459 lands.

## Type of Change

- [x] Bug fix (non-breaking change which fixes an issue)

## Changes Made

- `headroom/install/supervisors.py`: collapse the Windows SERVICE/TASK
branches of `install_supervisor` — SERVICE registers a boot task running
`run-headroom.cmd` (always-on) plus a 5-minute `ensure-headroom.cmd`
health task; TASK re-ensures on boot. Both use `_windows_task_xml`
(S4U/hidden). `start_supervisor`/`stop_supervisor` now `schtasks /Run` /
`/End` the startup task; `remove_supervisor` deletes both tasks and
best-effort deletes any legacy `sc.exe` service.
- `tests/test_install/test_supervisors.py`: assert SERVICE now emits
`windows-task` records (no `sc.exe create`), and that Windows
start/stop/remove drive the tasks.

## Testing

- [x] Unit tests pass

```
$ python -m pytest tests/test_install/test_supervisors.py -q
collected 29 items
tests\test_install\test_supervisors.py .............................     [100%]
============================= 29 passed in 0.35s ==============================
```

## Real Behavior Proof

- Environment: Windows 11 Pro 10.0.26200, Python 3.13.11
- Exact command / steps: python -m pytest
tests/test_install/test_supervisors.py -q; ruff check + ruff format
--check; mypy headroom/install/supervisors.py --ignore-missing-imports
- Observed result: 29 passed; ruff clean; mypy exit 0. SERVICE install
now returns windows-task records and issues no sc.exe create.
- Not tested: live `headroom install apply --preset persistent-service`
on a physical Windows host confirming the proxy answers /readyz after
reboot with no 1053 (no interactive Windows host in CI).

## Runtime Rollout Safety

- Rollout-managed feature(s): none. The install/supervisor path is not
in the `headroom.rollout` feature registry
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, for Windows `--preset
persistent-service` installs only. It registers a boot-triggered
scheduled task plus a 5-minute health task instead of an `sc.exe`
service; other presets and other platforms are unchanged
- Kill switch / disable path: none needed as a flag. `headroom install
remove` deletes both tasks and best-effort removes any legacy `sc.exe`
service, returning the host to no supervisor; this change replaces a
path that never worked (`StartService FAILED 1053`), not a working
default
- Unsafe override required: no
- Qualification impact: none. Install/supervisor logic only, no
routing/caching/backend-selection change
- Rollback path: revert this commit. `remove_supervisor` cleans up both
new tasks before rollback, so no orphaned scheduled tasks remain

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review


<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2460"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Parideboy
2026-10-08 11:03:32 -05:00
committed by GitHub
co-authored by Claude Opus 4.8 JerrettDavis
parent d091fbd9f2
commit 5682883bb0
2 changed files with 134 additions and 69 deletions
+40 -51
View File
@@ -527,44 +527,34 @@ def install_supervisor(manifest: DeploymentManifest, *, start: bool = True) -> l
records.append(ArtifactRecord(kind="plist", path=str(plist_path)))
return records
if _is_windows() and manifest.supervisor_kind == SupervisorKind.SERVICE.value:
# sc.exe's binPath= value embeds its own quotes (cmd.exe /c "<path>").
# Passing this as an argv list lets subprocess.list2cmdline re-quote the
# token and sc.exe mis-tokenizes it (issue #1654), so build the exact
# command line ourselves and hand subprocess a string.
run_cmd = windows_run_cmd_path(manifest.profile)
create_cmd = (
f"sc.exe create {manifest.service_name} "
f'binPath= "cmd.exe /c \\"{run_cmd}\\"" start= auto'
)
subprocess.run(create_cmd, check=True)
subprocess.run(
["sc.exe", "failure", manifest.service_name, "reset= 0", "actions= restart/5000"],
check=True,
)
records.append(ArtifactRecord(kind="windows-service", path=manifest.service_name))
return records
if _is_windows() and manifest.supervisor_kind == SupervisorKind.TASK.value:
if _is_windows() and manifest.supervisor_kind in (
SupervisorKind.SERVICE.value,
SupervisorKind.TASK.value,
):
startup_name = f"{manifest.service_name}-startup"
health_name = f"{manifest.service_name}-health"
startup_cmd = str(windows_ensure_cmd_path(manifest.profile))
startup_cmd = str(
windows_run_cmd_path(manifest.profile)
if manifest.supervisor_kind == SupervisorKind.SERVICE.value
else windows_ensure_cmd_path(manifest.profile)
)
health_cmd = str(windows_ensure_cmd_path(manifest.profile))
# Register from task XML (not schtasks flags) so the principal is S4U /
# hidden — flag-created tasks use an interactive token and flash a
# focus-stealing console on every run (issue #2453).
preferred_logon_type = "ServiceAccount" if manifest.scope == "system" else "S4U"
preferred_tasks = [
(startup_name, _windows_boot_trigger()),
(health_name, _windows_health_trigger()),
(startup_name, startup_cmd, _windows_boot_trigger()),
(health_name, health_cmd, _windows_health_trigger()),
]
registered_preferred_tasks: list[str] = []
try:
for name, trigger_xml in preferred_tasks:
for name, task_cmd, trigger_xml in preferred_tasks:
_register_windows_task(
name,
_windows_task_xml(
startup_cmd,
task_cmd,
trigger_xml=trigger_xml,
scope=manifest.scope,
logon_type=preferred_logon_type,
@@ -579,12 +569,13 @@ def install_supervisor(manifest: DeploymentManifest, *, start: bool = True) -> l
if exc.task_created:
must_delete.add(name)
_cleanup_windows_tasks_for_fallback(
[name for name, _trigger_xml in preferred_tasks], must_delete=must_delete
[name for name, _task_cmd, _trigger_xml in preferred_tasks],
must_delete=must_delete,
)
_register_windows_task(
health_name,
_windows_task_xml(
startup_cmd,
health_cmd,
trigger_xml=_windows_health_trigger(),
scope=manifest.scope,
logon_type="InteractiveToken",
@@ -652,8 +643,12 @@ def start_supervisor(manifest: DeploymentManifest) -> None:
plist_path = plist_dir / f"{label}.plist"
_bootstrap_with_retry(domain, plist_path, action="start")
return
if _is_windows() and manifest.supervisor_kind == SupervisorKind.SERVICE.value:
subprocess.run(["sc.exe", "start", manifest.service_name], check=True)
if _is_windows():
subprocess.run(
["schtasks", "/Change", "/TN", f"{manifest.service_name}-health", "/ENABLE"],
check=True,
)
subprocess.run(["schtasks", "/Run", "/TN", f"{manifest.service_name}-startup"], check=True)
def stop_supervisor(manifest: DeploymentManifest) -> None:
@@ -691,8 +686,11 @@ def stop_supervisor(manifest: DeploymentManifest) -> None:
f"launchctl bootout failed for {domain}/{label}: {detail or 'unknown error'}"
)
return
if _is_windows() and manifest.supervisor_kind == SupervisorKind.SERVICE.value:
subprocess.run(["sc.exe", "stop", manifest.service_name], check=True)
if _is_windows():
health = f"{manifest.service_name}-health"
subprocess.run(["schtasks", "/Change", "/TN", health, "/DISABLE"], check=True)
subprocess.run(["schtasks", "/End", "/TN", health], check=False)
subprocess.run(["schtasks", "/End", "/TN", f"{manifest.service_name}-startup"], check=True)
def remove_supervisor(manifest: DeploymentManifest) -> None:
@@ -772,32 +770,23 @@ def remove_supervisor(manifest: DeploymentManifest) -> None:
return
if _is_windows():
if manifest.supervisor_kind == SupervisorKind.SERVICE.value:
# Both presets are task-backed. Remove any legacy SCM service too,
# but do not hide a permission failure behind best-effort cleanup.
result = run(["sc.exe", "stop", manifest.service_name], capture_output=True, text=True)
_require_removal_success(result, "sc.exe stop", absent_codes={1060, 1062})
result = run(["sc.exe", "delete", manifest.service_name], capture_output=True, text=True)
_require_removal_success(result, "sc.exe delete", absent_codes={1060})
for suffix in ("startup", "health"):
result = run(
["sc.exe", "stop", manifest.service_name],
["schtasks", "/Delete", "/TN", f"{manifest.service_name}-{suffix}", "/F"],
capture_output=True,
text=True,
)
_require_removal_success(result, "sc.exe stop", absent_text="does not exist")
result = run(
["sc.exe", "delete", manifest.service_name],
capture_output=True,
text=True,
_require_removal_success(
result,
f"schtasks {suffix} delete",
absent_text="cannot find the file specified",
)
_require_removal_success(result, "sc.exe delete", absent_text="does not exist")
return
result = run(
["schtasks", "/Delete", "/TN", f"{manifest.service_name}-startup", "/F"],
capture_output=True,
text=True,
)
_require_removal_success(result, "schtasks startup delete", absent_text="does not exist")
result = run(
["schtasks", "/Delete", "/TN", f"{manifest.service_name}-health", "/F"],
capture_output=True,
text=True,
)
_require_removal_success(result, "schtasks health delete", absent_text="does not exist")
def _result_text(result: object) -> str:
+94 -18
View File
@@ -730,7 +730,7 @@ def test_install_supervisor_darwin_windows_and_unsupported(monkeypatch, tmp_path
)
calls: list[list[str]] = []
def fake_run(command, **kwargs):
def fake_run(command: list[str], **kwargs: object) -> object:
calls.append(command)
if isinstance(command, list) and len(command) > 1 and command[1] == "/Query":
return _LaunchctlResult(
@@ -771,15 +771,12 @@ def test_install_supervisor_darwin_windows_and_unsupported(monkeypatch, tmp_path
)
win_service = install_supervisor(_manifest(supervisor=SupervisorKind.SERVICE.value))
win_task = install_supervisor(_manifest(supervisor=SupervisorKind.TASK.value))
assert win_service[-1].kind == "windows-service"
# #1866: a console process can't be an SCM service, so SERVICE is task-backed
# too — no more `sc.exe create`. Both presets emit startup + health tasks.
assert win_service[-1].kind == "windows-task"
assert win_service[-2].path.endswith("-startup")
assert win_task[-2].path.endswith("-startup")
# Regression for #1654: the create command must be a single pre-quoted
# string (bypassing list2cmdline) with the inner quotes backslash-escaped
# and `start= auto` as a separate trailing token.
assert (
"sc.exe create headroom-default "
'binPath= "cmd.exe /c \\"C:\\tmp\\default\\run-headroom.cmd\\"" start= auto'
) in calls
assert not any(isinstance(c, str) and c.startswith("sc.exe create") for c in calls)
# #2453: tasks are registered from S4U/hidden XML via `schtasks /XML`, not
# interactive-token flag creation. Assert the startup and health tasks are
# each created from an XML file (the temp path varies).
@@ -846,7 +843,7 @@ def test_install_supervisor_retries_bootstrap_until_launchd_settles(
monkeypatch.setattr("headroom.install.supervisors.time.sleep", lambda _s: None)
bootstrap_attempts = 0
def fake_run(command, **kwargs):
def fake_run(command: list[str], **kwargs: object) -> object:
nonlocal bootstrap_attempts
if command[1] == "bootout":
return _LaunchctlResult(0)
@@ -881,7 +878,7 @@ def test_install_supervisor_raises_after_bootstrap_keeps_failing(
monkeypatch.setattr("headroom.install.supervisors.time.sleep", lambda _s: None)
monkeypatch.setattr("headroom.install.supervisors._MACOS_BOOTSTRAP_RETRIES", 3)
def fake_run(command, **kwargs):
def fake_run(command: list[str], **kwargs: object) -> object:
if command[1] == "bootout":
return _LaunchctlResult(0)
return _LaunchctlResult(5, stderr="Bootstrap failed: 5: Input/output error")
@@ -937,9 +934,16 @@ def test_start_and_stop_supervisor_darwin_windows_and_none(monkeypatch) -> None:
monkeypatch.setattr("headroom.install.supervisors.sys.platform", "win32")
start_supervisor(_manifest(supervisor=SupervisorKind.SERVICE.value))
stop_supervisor(_manifest(supervisor=SupervisorKind.SERVICE.value))
# #1866: Windows service is task-backed. start re-enables the health
# watchdog then runs the startup task. stop disables the watchdog, ends any
# in-flight health task, then ends the startup task last — so no running
# `ensure` can re-enable or restart the service after `stop` completes.
assert calls == [
["sc.exe", "start", "headroom-default"],
["sc.exe", "stop", "headroom-default"],
["schtasks", "/Change", "/TN", "headroom-default-health", "/ENABLE"],
["schtasks", "/Run", "/TN", "headroom-default-startup"],
["schtasks", "/Change", "/TN", "headroom-default-health", "/DISABLE"],
["schtasks", "/End", "/TN", "headroom-default-health"],
["schtasks", "/End", "/TN", "headroom-default-startup"],
]
@@ -951,7 +955,7 @@ def test_macos_start_bootstraps_when_job_not_registered(monkeypatch, tmp_path: P
monkeypatch.setattr(Path, "home", lambda: tmp_path)
calls: list[list[str]] = []
def fake_run(command, **kwargs):
def fake_run(command: list[str], **kwargs: object) -> object:
calls.append(command)
if command[1] == "kickstart":
return _LaunchctlResult(113, stderr="Could not find service")
@@ -977,7 +981,7 @@ def test_macos_start_retries_bootstrap_until_launchd_settles(monkeypatch, tmp_pa
monkeypatch.setattr("headroom.install.supervisors.time.sleep", lambda _s: None)
bootstrap_attempts = 0
def fake_run(command, **kwargs):
def fake_run(command: list[str], **kwargs: object) -> object:
nonlocal bootstrap_attempts
if command[1] == "kickstart":
return _LaunchctlResult(113)
@@ -999,7 +1003,7 @@ def test_macos_start_raises_after_bootstrap_keeps_failing(monkeypatch, tmp_path:
monkeypatch.setattr("headroom.install.supervisors.time.sleep", lambda _s: None)
monkeypatch.setattr("headroom.install.supervisors._MACOS_BOOTSTRAP_RETRIES", 3)
def fake_run(command, **kwargs):
def fake_run(command: list[str], **kwargs: object) -> object:
if command[1] == "kickstart":
return _LaunchctlResult(113)
return _LaunchctlResult(5, stderr="Bootstrap failed: 5: Input/output error")
@@ -1017,7 +1021,7 @@ def test_macos_stop_tolerates_missing_job(monkeypatch) -> None:
monkeypatch.setattr("headroom.install.supervisors.os.getuid", lambda: 77, raising=False)
calls: list[list[str]] = []
def fake_run(command, **kwargs):
def fake_run(command: list[str], **kwargs: object) -> object:
calls.append(command)
assert kwargs.get("check") is not True
return _LaunchctlResult(3, stderr="Boot-out failed: 3: No such process")
@@ -1169,9 +1173,81 @@ def test_remove_supervisor_darwin_and_windows(monkeypatch, tmp_path: Path) -> No
monkeypatch.setattr("headroom.install.supervisors.sys.platform", "win32")
remove_supervisor(_manifest(supervisor=SupervisorKind.SERVICE.value))
remove_supervisor(_manifest(supervisor=SupervisorKind.TASK.value))
assert calls == [
# #1866: both presets are task-backed, so removal deletes the two tasks. A
# best-effort `sc.exe` delete stays to clean up any legacy service.
windows_cleanup = [
["sc.exe", "stop", "headroom-default"],
["sc.exe", "delete", "headroom-default"],
["schtasks", "/Delete", "/TN", "headroom-default-startup", "/F"],
["schtasks", "/Delete", "/TN", "headroom-default-health", "/F"],
]
assert calls == windows_cleanup + windows_cleanup
@pytest.mark.parametrize(
("failed_command", "failure_code"), [("sc.exe", 5), ("schtasks", 1), ("schtasks", 5)]
)
def test_windows_task_backed_removal_surfaces_permission_failure(
monkeypatch: pytest.MonkeyPatch, failed_command: str, failure_code: int
) -> None:
monkeypatch.setattr("headroom.install.supervisors.sys.platform", "win32")
def fake_run(command: list[str], **kwargs: object) -> object:
failed = command[0] == failed_command
return type(
"Result",
(),
{
"returncode": failure_code if failed else 0,
"stdout": "",
"stderr": "Access denied" if failed else "",
},
)()
monkeypatch.setattr("headroom.install.supervisors.run", fake_run)
with pytest.raises(click.ClickException, match="failed: Access denied"):
remove_supervisor(_manifest(supervisor=SupervisorKind.SERVICE.value))
def test_windows_task_backed_removal_accepts_absent_legacy_service_and_tasks(
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.setattr("headroom.install.supervisors.sys.platform", "win32")
calls: list[list[str]] = []
def fake_run(command: list[str], **kwargs: object) -> object:
calls.append(command)
code = 1060 if command[0] == "sc.exe" else 1
text = (
"The specified service does not exist"
if code == 1060
else "ERROR: The system cannot find the file specified."
)
return type("Result", (), {"returncode": code, "stdout": "", "stderr": text})()
monkeypatch.setattr("headroom.install.supervisors.run", fake_run)
remove_supervisor(_manifest(supervisor=SupervisorKind.SERVICE.value))
assert [c[0] for c in calls] == ["sc.exe", "sc.exe", "schtasks", "schtasks"]
def test_windows_task_backed_removal_accepts_stopped_legacy_service(
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.setattr("headroom.install.supervisors.sys.platform", "win32")
calls: list[list[str]] = []
def fake_run(command: list[str], **kwargs: object) -> object:
calls.append(command)
return type(
"Result",
(),
{
"returncode": 1062 if command[:2] == ["sc.exe", "stop"] else 0,
"stdout": "",
"stderr": "The service has not been started",
},
)()
monkeypatch.setattr("headroom.install.supervisors.run", fake_run)
remove_supervisor(_manifest(supervisor=SupervisorKind.SERVICE.value))
assert len([c for c in calls if c[0] == "schtasks"]) == 2