mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description `headroom install apply --preset persistent-service` failed on Windows. It registered `cmd.exe /c run-headroom.cmd` as a service via `sc.exe create`, but a plain console process never performs the SCM `StartServiceCtrlDispatcher` handshake, so Windows killed it after the 30s start timeout (`StartService FAILED 1053`) and left an empty `deploy/default` profile. Fixes #1866. Rather than add a service-wrapper dependency, the Windows service is now backed by scheduled tasks: a `BootTrigger` task launches the foreground proxy (`run-headroom`) at boot so it stays always-on, plus a 5-minute health task as a watchdog. Both are registered from the same S4U/hidden task XML introduced for #2453, so no console window ever flashes. > Stacked on #2459 (the #2453 S4U/hidden XML machinery). Until that merges, this PR's diff includes its commit; it collapses automatically once #2459 lands. ## Type of Change - [x] Bug fix (non-breaking change which fixes an issue) ## Changes Made - `headroom/install/supervisors.py`: collapse the Windows SERVICE/TASK branches of `install_supervisor` — SERVICE registers a boot task running `run-headroom.cmd` (always-on) plus a 5-minute `ensure-headroom.cmd` health task; TASK re-ensures on boot. Both use `_windows_task_xml` (S4U/hidden). `start_supervisor`/`stop_supervisor` now `schtasks /Run` / `/End` the startup task; `remove_supervisor` deletes both tasks and best-effort deletes any legacy `sc.exe` service. - `tests/test_install/test_supervisors.py`: assert SERVICE now emits `windows-task` records (no `sc.exe create`), and that Windows start/stop/remove drive the tasks. ## Testing - [x] Unit tests pass ``` $ python -m pytest tests/test_install/test_supervisors.py -q collected 29 items tests\test_install\test_supervisors.py ............................. [100%] ============================= 29 passed in 0.35s ============================== ``` ## Real Behavior Proof - Environment: Windows 11 Pro 10.0.26200, Python 3.13.11 - Exact command / steps: python -m pytest tests/test_install/test_supervisors.py -q; ruff check + ruff format --check; mypy headroom/install/supervisors.py --ignore-missing-imports - Observed result: 29 passed; ruff clean; mypy exit 0. SERVICE install now returns windows-task records and issues no sc.exe create. - Not tested: live `headroom install apply --preset persistent-service` on a physical Windows host confirming the proxy answers /readyz after reboot with no 1053 (no interactive Windows host in CI). ## Runtime Rollout Safety - Rollout-managed feature(s): none. The install/supervisor path is not in the `headroom.rollout` feature registry - Minimum rollout channel: n/a - Stable/default behavior changed: yes, for Windows `--preset persistent-service` installs only. It registers a boot-triggered scheduled task plus a 5-minute health task instead of an `sc.exe` service; other presets and other platforms are unchanged - Kill switch / disable path: none needed as a flag. `headroom install remove` deletes both tasks and best-effort removes any legacy `sc.exe` service, returning the host to no supervisor; this change replaces a path that never worked (`StartService FAILED 1053`), not a working default - Unsafe override required: no - Qualification impact: none. Install/supervisor logic only, no routing/caching/backend-selection change - Rollback path: revert this commit. `remove_supervisor` cleans up both new tasks before rollback, so no orphaned scheduled tasks remain ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2460" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
JerrettDavis
parent
d091fbd9f2
commit
5682883bb0
@@ -527,44 +527,34 @@ def install_supervisor(manifest: DeploymentManifest, *, start: bool = True) -> l
|
||||
records.append(ArtifactRecord(kind="plist", path=str(plist_path)))
|
||||
return records
|
||||
|
||||
if _is_windows() and manifest.supervisor_kind == SupervisorKind.SERVICE.value:
|
||||
# sc.exe's binPath= value embeds its own quotes (cmd.exe /c "<path>").
|
||||
# Passing this as an argv list lets subprocess.list2cmdline re-quote the
|
||||
# token and sc.exe mis-tokenizes it (issue #1654), so build the exact
|
||||
# command line ourselves and hand subprocess a string.
|
||||
run_cmd = windows_run_cmd_path(manifest.profile)
|
||||
create_cmd = (
|
||||
f"sc.exe create {manifest.service_name} "
|
||||
f'binPath= "cmd.exe /c \\"{run_cmd}\\"" start= auto'
|
||||
)
|
||||
subprocess.run(create_cmd, check=True)
|
||||
subprocess.run(
|
||||
["sc.exe", "failure", manifest.service_name, "reset= 0", "actions= restart/5000"],
|
||||
check=True,
|
||||
)
|
||||
records.append(ArtifactRecord(kind="windows-service", path=manifest.service_name))
|
||||
return records
|
||||
|
||||
if _is_windows() and manifest.supervisor_kind == SupervisorKind.TASK.value:
|
||||
if _is_windows() and manifest.supervisor_kind in (
|
||||
SupervisorKind.SERVICE.value,
|
||||
SupervisorKind.TASK.value,
|
||||
):
|
||||
startup_name = f"{manifest.service_name}-startup"
|
||||
health_name = f"{manifest.service_name}-health"
|
||||
startup_cmd = str(windows_ensure_cmd_path(manifest.profile))
|
||||
startup_cmd = str(
|
||||
windows_run_cmd_path(manifest.profile)
|
||||
if manifest.supervisor_kind == SupervisorKind.SERVICE.value
|
||||
else windows_ensure_cmd_path(manifest.profile)
|
||||
)
|
||||
health_cmd = str(windows_ensure_cmd_path(manifest.profile))
|
||||
# Register from task XML (not schtasks flags) so the principal is S4U /
|
||||
# hidden — flag-created tasks use an interactive token and flash a
|
||||
# focus-stealing console on every run (issue #2453).
|
||||
preferred_logon_type = "ServiceAccount" if manifest.scope == "system" else "S4U"
|
||||
preferred_tasks = [
|
||||
(startup_name, _windows_boot_trigger()),
|
||||
(health_name, _windows_health_trigger()),
|
||||
(startup_name, startup_cmd, _windows_boot_trigger()),
|
||||
(health_name, health_cmd, _windows_health_trigger()),
|
||||
]
|
||||
registered_preferred_tasks: list[str] = []
|
||||
|
||||
try:
|
||||
for name, trigger_xml in preferred_tasks:
|
||||
for name, task_cmd, trigger_xml in preferred_tasks:
|
||||
_register_windows_task(
|
||||
name,
|
||||
_windows_task_xml(
|
||||
startup_cmd,
|
||||
task_cmd,
|
||||
trigger_xml=trigger_xml,
|
||||
scope=manifest.scope,
|
||||
logon_type=preferred_logon_type,
|
||||
@@ -579,12 +569,13 @@ def install_supervisor(manifest: DeploymentManifest, *, start: bool = True) -> l
|
||||
if exc.task_created:
|
||||
must_delete.add(name)
|
||||
_cleanup_windows_tasks_for_fallback(
|
||||
[name for name, _trigger_xml in preferred_tasks], must_delete=must_delete
|
||||
[name for name, _task_cmd, _trigger_xml in preferred_tasks],
|
||||
must_delete=must_delete,
|
||||
)
|
||||
_register_windows_task(
|
||||
health_name,
|
||||
_windows_task_xml(
|
||||
startup_cmd,
|
||||
health_cmd,
|
||||
trigger_xml=_windows_health_trigger(),
|
||||
scope=manifest.scope,
|
||||
logon_type="InteractiveToken",
|
||||
@@ -652,8 +643,12 @@ def start_supervisor(manifest: DeploymentManifest) -> None:
|
||||
plist_path = plist_dir / f"{label}.plist"
|
||||
_bootstrap_with_retry(domain, plist_path, action="start")
|
||||
return
|
||||
if _is_windows() and manifest.supervisor_kind == SupervisorKind.SERVICE.value:
|
||||
subprocess.run(["sc.exe", "start", manifest.service_name], check=True)
|
||||
if _is_windows():
|
||||
subprocess.run(
|
||||
["schtasks", "/Change", "/TN", f"{manifest.service_name}-health", "/ENABLE"],
|
||||
check=True,
|
||||
)
|
||||
subprocess.run(["schtasks", "/Run", "/TN", f"{manifest.service_name}-startup"], check=True)
|
||||
|
||||
|
||||
def stop_supervisor(manifest: DeploymentManifest) -> None:
|
||||
@@ -691,8 +686,11 @@ def stop_supervisor(manifest: DeploymentManifest) -> None:
|
||||
f"launchctl bootout failed for {domain}/{label}: {detail or 'unknown error'}"
|
||||
)
|
||||
return
|
||||
if _is_windows() and manifest.supervisor_kind == SupervisorKind.SERVICE.value:
|
||||
subprocess.run(["sc.exe", "stop", manifest.service_name], check=True)
|
||||
if _is_windows():
|
||||
health = f"{manifest.service_name}-health"
|
||||
subprocess.run(["schtasks", "/Change", "/TN", health, "/DISABLE"], check=True)
|
||||
subprocess.run(["schtasks", "/End", "/TN", health], check=False)
|
||||
subprocess.run(["schtasks", "/End", "/TN", f"{manifest.service_name}-startup"], check=True)
|
||||
|
||||
|
||||
def remove_supervisor(manifest: DeploymentManifest) -> None:
|
||||
@@ -772,32 +770,23 @@ def remove_supervisor(manifest: DeploymentManifest) -> None:
|
||||
return
|
||||
|
||||
if _is_windows():
|
||||
if manifest.supervisor_kind == SupervisorKind.SERVICE.value:
|
||||
# Both presets are task-backed. Remove any legacy SCM service too,
|
||||
# but do not hide a permission failure behind best-effort cleanup.
|
||||
result = run(["sc.exe", "stop", manifest.service_name], capture_output=True, text=True)
|
||||
_require_removal_success(result, "sc.exe stop", absent_codes={1060, 1062})
|
||||
result = run(["sc.exe", "delete", manifest.service_name], capture_output=True, text=True)
|
||||
_require_removal_success(result, "sc.exe delete", absent_codes={1060})
|
||||
for suffix in ("startup", "health"):
|
||||
result = run(
|
||||
["sc.exe", "stop", manifest.service_name],
|
||||
["schtasks", "/Delete", "/TN", f"{manifest.service_name}-{suffix}", "/F"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
_require_removal_success(result, "sc.exe stop", absent_text="does not exist")
|
||||
result = run(
|
||||
["sc.exe", "delete", manifest.service_name],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
_require_removal_success(
|
||||
result,
|
||||
f"schtasks {suffix} delete",
|
||||
absent_text="cannot find the file specified",
|
||||
)
|
||||
_require_removal_success(result, "sc.exe delete", absent_text="does not exist")
|
||||
return
|
||||
result = run(
|
||||
["schtasks", "/Delete", "/TN", f"{manifest.service_name}-startup", "/F"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
_require_removal_success(result, "schtasks startup delete", absent_text="does not exist")
|
||||
result = run(
|
||||
["schtasks", "/Delete", "/TN", f"{manifest.service_name}-health", "/F"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
_require_removal_success(result, "schtasks health delete", absent_text="does not exist")
|
||||
|
||||
|
||||
def _result_text(result: object) -> str:
|
||||
|
||||
@@ -730,7 +730,7 @@ def test_install_supervisor_darwin_windows_and_unsupported(monkeypatch, tmp_path
|
||||
)
|
||||
calls: list[list[str]] = []
|
||||
|
||||
def fake_run(command, **kwargs):
|
||||
def fake_run(command: list[str], **kwargs: object) -> object:
|
||||
calls.append(command)
|
||||
if isinstance(command, list) and len(command) > 1 and command[1] == "/Query":
|
||||
return _LaunchctlResult(
|
||||
@@ -771,15 +771,12 @@ def test_install_supervisor_darwin_windows_and_unsupported(monkeypatch, tmp_path
|
||||
)
|
||||
win_service = install_supervisor(_manifest(supervisor=SupervisorKind.SERVICE.value))
|
||||
win_task = install_supervisor(_manifest(supervisor=SupervisorKind.TASK.value))
|
||||
assert win_service[-1].kind == "windows-service"
|
||||
# #1866: a console process can't be an SCM service, so SERVICE is task-backed
|
||||
# too — no more `sc.exe create`. Both presets emit startup + health tasks.
|
||||
assert win_service[-1].kind == "windows-task"
|
||||
assert win_service[-2].path.endswith("-startup")
|
||||
assert win_task[-2].path.endswith("-startup")
|
||||
# Regression for #1654: the create command must be a single pre-quoted
|
||||
# string (bypassing list2cmdline) with the inner quotes backslash-escaped
|
||||
# and `start= auto` as a separate trailing token.
|
||||
assert (
|
||||
"sc.exe create headroom-default "
|
||||
'binPath= "cmd.exe /c \\"C:\\tmp\\default\\run-headroom.cmd\\"" start= auto'
|
||||
) in calls
|
||||
assert not any(isinstance(c, str) and c.startswith("sc.exe create") for c in calls)
|
||||
# #2453: tasks are registered from S4U/hidden XML via `schtasks /XML`, not
|
||||
# interactive-token flag creation. Assert the startup and health tasks are
|
||||
# each created from an XML file (the temp path varies).
|
||||
@@ -846,7 +843,7 @@ def test_install_supervisor_retries_bootstrap_until_launchd_settles(
|
||||
monkeypatch.setattr("headroom.install.supervisors.time.sleep", lambda _s: None)
|
||||
bootstrap_attempts = 0
|
||||
|
||||
def fake_run(command, **kwargs):
|
||||
def fake_run(command: list[str], **kwargs: object) -> object:
|
||||
nonlocal bootstrap_attempts
|
||||
if command[1] == "bootout":
|
||||
return _LaunchctlResult(0)
|
||||
@@ -881,7 +878,7 @@ def test_install_supervisor_raises_after_bootstrap_keeps_failing(
|
||||
monkeypatch.setattr("headroom.install.supervisors.time.sleep", lambda _s: None)
|
||||
monkeypatch.setattr("headroom.install.supervisors._MACOS_BOOTSTRAP_RETRIES", 3)
|
||||
|
||||
def fake_run(command, **kwargs):
|
||||
def fake_run(command: list[str], **kwargs: object) -> object:
|
||||
if command[1] == "bootout":
|
||||
return _LaunchctlResult(0)
|
||||
return _LaunchctlResult(5, stderr="Bootstrap failed: 5: Input/output error")
|
||||
@@ -937,9 +934,16 @@ def test_start_and_stop_supervisor_darwin_windows_and_none(monkeypatch) -> None:
|
||||
monkeypatch.setattr("headroom.install.supervisors.sys.platform", "win32")
|
||||
start_supervisor(_manifest(supervisor=SupervisorKind.SERVICE.value))
|
||||
stop_supervisor(_manifest(supervisor=SupervisorKind.SERVICE.value))
|
||||
# #1866: Windows service is task-backed. start re-enables the health
|
||||
# watchdog then runs the startup task. stop disables the watchdog, ends any
|
||||
# in-flight health task, then ends the startup task last — so no running
|
||||
# `ensure` can re-enable or restart the service after `stop` completes.
|
||||
assert calls == [
|
||||
["sc.exe", "start", "headroom-default"],
|
||||
["sc.exe", "stop", "headroom-default"],
|
||||
["schtasks", "/Change", "/TN", "headroom-default-health", "/ENABLE"],
|
||||
["schtasks", "/Run", "/TN", "headroom-default-startup"],
|
||||
["schtasks", "/Change", "/TN", "headroom-default-health", "/DISABLE"],
|
||||
["schtasks", "/End", "/TN", "headroom-default-health"],
|
||||
["schtasks", "/End", "/TN", "headroom-default-startup"],
|
||||
]
|
||||
|
||||
|
||||
@@ -951,7 +955,7 @@ def test_macos_start_bootstraps_when_job_not_registered(monkeypatch, tmp_path: P
|
||||
monkeypatch.setattr(Path, "home", lambda: tmp_path)
|
||||
calls: list[list[str]] = []
|
||||
|
||||
def fake_run(command, **kwargs):
|
||||
def fake_run(command: list[str], **kwargs: object) -> object:
|
||||
calls.append(command)
|
||||
if command[1] == "kickstart":
|
||||
return _LaunchctlResult(113, stderr="Could not find service")
|
||||
@@ -977,7 +981,7 @@ def test_macos_start_retries_bootstrap_until_launchd_settles(monkeypatch, tmp_pa
|
||||
monkeypatch.setattr("headroom.install.supervisors.time.sleep", lambda _s: None)
|
||||
bootstrap_attempts = 0
|
||||
|
||||
def fake_run(command, **kwargs):
|
||||
def fake_run(command: list[str], **kwargs: object) -> object:
|
||||
nonlocal bootstrap_attempts
|
||||
if command[1] == "kickstart":
|
||||
return _LaunchctlResult(113)
|
||||
@@ -999,7 +1003,7 @@ def test_macos_start_raises_after_bootstrap_keeps_failing(monkeypatch, tmp_path:
|
||||
monkeypatch.setattr("headroom.install.supervisors.time.sleep", lambda _s: None)
|
||||
monkeypatch.setattr("headroom.install.supervisors._MACOS_BOOTSTRAP_RETRIES", 3)
|
||||
|
||||
def fake_run(command, **kwargs):
|
||||
def fake_run(command: list[str], **kwargs: object) -> object:
|
||||
if command[1] == "kickstart":
|
||||
return _LaunchctlResult(113)
|
||||
return _LaunchctlResult(5, stderr="Bootstrap failed: 5: Input/output error")
|
||||
@@ -1017,7 +1021,7 @@ def test_macos_stop_tolerates_missing_job(monkeypatch) -> None:
|
||||
monkeypatch.setattr("headroom.install.supervisors.os.getuid", lambda: 77, raising=False)
|
||||
calls: list[list[str]] = []
|
||||
|
||||
def fake_run(command, **kwargs):
|
||||
def fake_run(command: list[str], **kwargs: object) -> object:
|
||||
calls.append(command)
|
||||
assert kwargs.get("check") is not True
|
||||
return _LaunchctlResult(3, stderr="Boot-out failed: 3: No such process")
|
||||
@@ -1169,9 +1173,81 @@ def test_remove_supervisor_darwin_and_windows(monkeypatch, tmp_path: Path) -> No
|
||||
monkeypatch.setattr("headroom.install.supervisors.sys.platform", "win32")
|
||||
remove_supervisor(_manifest(supervisor=SupervisorKind.SERVICE.value))
|
||||
remove_supervisor(_manifest(supervisor=SupervisorKind.TASK.value))
|
||||
assert calls == [
|
||||
# #1866: both presets are task-backed, so removal deletes the two tasks. A
|
||||
# best-effort `sc.exe` delete stays to clean up any legacy service.
|
||||
windows_cleanup = [
|
||||
["sc.exe", "stop", "headroom-default"],
|
||||
["sc.exe", "delete", "headroom-default"],
|
||||
["schtasks", "/Delete", "/TN", "headroom-default-startup", "/F"],
|
||||
["schtasks", "/Delete", "/TN", "headroom-default-health", "/F"],
|
||||
]
|
||||
assert calls == windows_cleanup + windows_cleanup
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("failed_command", "failure_code"), [("sc.exe", 5), ("schtasks", 1), ("schtasks", 5)]
|
||||
)
|
||||
def test_windows_task_backed_removal_surfaces_permission_failure(
|
||||
monkeypatch: pytest.MonkeyPatch, failed_command: str, failure_code: int
|
||||
) -> None:
|
||||
monkeypatch.setattr("headroom.install.supervisors.sys.platform", "win32")
|
||||
|
||||
def fake_run(command: list[str], **kwargs: object) -> object:
|
||||
failed = command[0] == failed_command
|
||||
return type(
|
||||
"Result",
|
||||
(),
|
||||
{
|
||||
"returncode": failure_code if failed else 0,
|
||||
"stdout": "",
|
||||
"stderr": "Access denied" if failed else "",
|
||||
},
|
||||
)()
|
||||
|
||||
monkeypatch.setattr("headroom.install.supervisors.run", fake_run)
|
||||
with pytest.raises(click.ClickException, match="failed: Access denied"):
|
||||
remove_supervisor(_manifest(supervisor=SupervisorKind.SERVICE.value))
|
||||
|
||||
|
||||
def test_windows_task_backed_removal_accepts_absent_legacy_service_and_tasks(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.setattr("headroom.install.supervisors.sys.platform", "win32")
|
||||
calls: list[list[str]] = []
|
||||
|
||||
def fake_run(command: list[str], **kwargs: object) -> object:
|
||||
calls.append(command)
|
||||
code = 1060 if command[0] == "sc.exe" else 1
|
||||
text = (
|
||||
"The specified service does not exist"
|
||||
if code == 1060
|
||||
else "ERROR: The system cannot find the file specified."
|
||||
)
|
||||
return type("Result", (), {"returncode": code, "stdout": "", "stderr": text})()
|
||||
|
||||
monkeypatch.setattr("headroom.install.supervisors.run", fake_run)
|
||||
remove_supervisor(_manifest(supervisor=SupervisorKind.SERVICE.value))
|
||||
assert [c[0] for c in calls] == ["sc.exe", "sc.exe", "schtasks", "schtasks"]
|
||||
|
||||
|
||||
def test_windows_task_backed_removal_accepts_stopped_legacy_service(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.setattr("headroom.install.supervisors.sys.platform", "win32")
|
||||
calls: list[list[str]] = []
|
||||
|
||||
def fake_run(command: list[str], **kwargs: object) -> object:
|
||||
calls.append(command)
|
||||
return type(
|
||||
"Result",
|
||||
(),
|
||||
{
|
||||
"returncode": 1062 if command[:2] == ["sc.exe", "stop"] else 0,
|
||||
"stdout": "",
|
||||
"stderr": "The service has not been started",
|
||||
},
|
||||
)()
|
||||
|
||||
monkeypatch.setattr("headroom.install.supervisors.run", fake_run)
|
||||
remove_supervisor(_manifest(supervisor=SupervisorKind.SERVICE.value))
|
||||
assert len([c for c in calls if c[0] == "schtasks"]) == 2
|
||||
|
||||
Reference in New Issue
Block a user