fix(install): make install status agree with doctor when the proxy is reachable (#3072) (#3141)

## Description

`headroom install status` could report `Status: stopped` next to
`Healthy: yes`, while `headroom doctor` reported the same proxy as
running (#3072, bug 2). The commands used independent process views:
`install status` trusted PID/container bookkeeping for `Status`, while
both commands used the live HTTP health endpoints to establish health.

This change treats a successful `/readyz` probe as authoritative
evidence that the deployment is running. If the proxy is not ready,
`runtime_status` still distinguishes a running-but-unhealthy process
from a stopped process.

The PR also adds a focused macOS pytest lane for CLI and install code,
closing the Darwin test-coverage gap described in #3065 without
duplicating the full four-shard Linux suite.

Bug 1 from #3072 (`HEADROOM_PORT` being ignored by `install apply` and
`deploy`) was already fixed on `main` by #3085 and is not reimplemented
here.

Closes #3072
Closes #3065

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Updated `headroom/cli/install.py` so `install_status` performs one
readiness probe and reports `Status: running` whenever `/readyz` is
healthy.
- Kept `runtime_status` as the fallback when the proxy is not ready,
preserving the distinction between running-but-unhealthy and stopped.
- Added regression coverage for the formerly contradictory `Status:
stopped` plus `Healthy: yes` output.
- Added regression coverage for the not-ready case where process status
must still come from `runtime_status`.
- Added a `test-macos` CI job that runs the CLI/install pytest subset on
`macos-latest`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_cli/test_install_cli.py -q
============================= test session starts ==============================
platform darwin -- Python 3.11.15, pytest-9.0.3, pluggy-1.6.0
collected 42 items

tests/test_cli/test_install_cli.py ..................................... [ 88%]
.....                                                                    [100%]

============================== 42 passed in 1.79s ==============================

$ uv run --frozen ruff check headroom/cli/install.py tests/test_cli/test_install_cli.py
All checks passed!

$ uv run --frozen ruff format --check headroom/cli/install.py tests/test_cli/test_install_cli.py
2 files already formatted

$ uv run --frozen mypy headroom/cli/install.py --ignore-missing-imports
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS (Apple Silicon), CPython 3.11.15 via `uv`, pytest
9.0.3.
- Exact command / steps: ran `uv run --frozen --extra dev pytest
tests/test_cli/test_install_cli.py -q`. The new regression test invokes
`headroom install status` through Click's `CliRunner` with `/readyz`
returning healthy while PID/runtime bookkeeping returns `stopped`.
- Observed result: the command exits successfully and prints `Status:
running` with `Healthy: yes`; `Status: stopped` is absent. A second test
verifies that when readiness is false, a live runtime still prints
`Status: running` with `Healthy: no`.
- Not tested: an end-to-end persistent-service installation with a real
OS supervisor or Docker daemon. The new macOS CI job itself will be
exercised by GitHub Actions rather than locally.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; this changes diagnostic display
logic and CI coverage only.
- Minimum rollout channel: stable/default patch release.
- Stable/default behavior changed: yes; `install status` now reports a
ready deployment as running even if PID/container bookkeeping says
stopped. Unready deployments retain the existing runtime-status
behavior.
- Kill switch / disable path: no runtime flag; revert this PR if the
readiness precedence causes an unforeseen regression.
- Unsafe override required: none.
- Qualification impact: `tests/test_cli/test_install_cli.py` and the new
`test-macos` CI job must remain green.
- Rollback path: revert commit `f3ae269b`; no persisted state, schema,
manifest, or configuration migration is involved.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable; this is terminal status output and CI configuration.

## Additional Notes

- Documentation was not changed because no command, option,
configuration contract, or documented workflow changed; this corrects
contradictory diagnostic output.
- #2787 was also reviewed as an urgent author-reported issue, but it
already has active implementation PR #2992. This PR deliberately avoids
duplicating or conflicting with that work.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3141"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer current-main requalification

Exact head `f550bf1d05873b20b773d935aa7cb93025b63262`, upstream main
`2d3ee51846435ef5b320eea4806af14e602b0d8c`.

Re-reviewed latest integration head 3c11d4a96 and current discussion;
clean integration with upstream/main
2d3ee51846. Install status still treats a
ready reachable proxy as running, preserving runtime-status fallback
when unhealthy. All 160 install-status/release/version/MCP tests pass;
no PR-specific production-code blocker found in scope.

Downloaded the latest failing native macOS job 126368233730 from run
41765101507: four persistent-wrap recovery tests still failed because
_Manifest/TaskManifest lacked runtime_kind. Repaired test doubles with
the Python runtime declaration and explicitly mocked the readiness
boundary those tests control through health probes; real ownership
classification remains exercised. Parametrized the four affected
recovery cases across Darwin/Linux/Windows by replacing only runtime
module's sys reference, without global platform emulation. All 83
persistent-wrap tests pass, preserving matching deployment recovery,
bound-socket recovery, healthy reuse, and task warning coverage. No
production recovery logic changed.

Changed-file Ruff check/format and whitespace pass. These local platform
variants are not native macOS validation; exact-head hosted macOS lane
must qualify before approval. Existing native extension supports Python
startup, not a fresh native build. Maintainers handle CI; nothing
merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: vishal veerareddy <vishalveera.reddy@servicenow.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Vishal Veera Reddy
2026-11-05 21:09:29 -06:00
committed by GitHub
co-authored by vishal veerareddy Claude Sonnet 5 JerrettDavis
parent ce14271ba8
commit 05431b264f
4 changed files with 96 additions and 7 deletions
+26
View File
@@ -438,6 +438,32 @@ jobs:
token: ${{ secrets.CODECOV_TOKEN }}
fail_ci_if_error: false
# Keep a small but real pytest lane on Darwin. The full four-shard suite is
# intentionally Linux-only for cost, but platform-gated CLI/proxy code must
# execute somewhere on macOS; otherwise Darwin regressions can merge green
# while being unreachable in every pytest job (#3065).
test-macos:
needs: changes
if: needs.changes.outputs.code == 'true'
runs-on: macos-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v6
with:
python-version: ${{ env.PY_VERSION }}
- name: Install package and focused test dependencies
run: |
python -m pip install --upgrade pip
pip install -e '.[proxy]' pytest pytest-asyncio numpy
- name: Run macOS CLI and install tests
run: |
pytest \
tests/test_cli \
tests/test_cli_*.py \
tests/test_install \
--tb=short -q
test-extras:
needs: [changes, build-wheel]
if: needs.changes.outputs.code == 'true'
+8 -2
View File
@@ -1166,14 +1166,20 @@ def install_status(profile: str) -> None:
manifest = _require_manifest(profile)
payload = probe_json(manifest.health_url.replace("/readyz", "/health"))
healthy = probe_ready(manifest.health_url)
# The health endpoint is the authoritative process signal. Detached runtimes
# can outlive a missing/stale PID file (for example after cleanup or a
# supervisor restart), while Docker's local CLI view can also be temporarily
# unavailable. Never report a reachable, ready proxy as "stopped" (#3072).
status = "running" if healthy else runtime_status(manifest)
click.echo(f"Profile: {manifest.profile}")
click.echo(f"Preset: {manifest.preset}")
click.echo(f"Runtime: {manifest.runtime_kind}")
click.echo(f"Supervisor: {manifest.supervisor_kind}")
click.echo(f"Scope: {manifest.scope}")
click.echo(f"Port: {manifest.port}")
click.echo(f"Status: {runtime_status(manifest)}")
click.echo(f"Healthy: {'yes' if probe_ready(manifest.health_url) else 'no'}")
click.echo(f"Status: {status}")
click.echo(f"Healthy: {'yes' if healthy else 'no'}")
if payload and isinstance(payload, dict):
click.echo(f"Health URL: {manifest.health_url.replace('/readyz', '/health')}")
# `config` may be a non-dict (null / string / list) if a different or
+37 -1
View File
@@ -497,6 +497,42 @@ def test_install_status_includes_protocol_targets_from_health_probe(
assert secret not in result.output
def test_install_status_treats_ready_proxy_as_running(monkeypatch) -> None:
"""Health wins when PID/container bookkeeping incorrectly says stopped.
Regression for #3072 bug 2: status previously printed the contradictory
pair ``Status: stopped`` and ``Healthy: yes`` while doctor correctly
reported the same proxy as running.
"""
manifest = _status_manifest("default")
monkeypatch.setattr(inst, "load_manifest", lambda profile: manifest)
monkeypatch.setattr(inst, "probe_json", lambda url: None)
monkeypatch.setattr(inst, "probe_ready", lambda url: True)
monkeypatch.setattr(inst, "runtime_status", lambda deployment: "stopped")
result = CliRunner().invoke(main, ["install", "status"])
assert result.exit_code == 0, result.output
assert "Status: running" in result.output
assert "Healthy: yes" in result.output
assert "Status: stopped" not in result.output
def test_install_status_keeps_runtime_status_when_not_ready(monkeypatch) -> None:
"""An unhealthy live process remains distinguishable from a stopped one."""
manifest = _status_manifest("default")
monkeypatch.setattr(inst, "load_manifest", lambda profile: manifest)
monkeypatch.setattr(inst, "probe_json", lambda url: None)
monkeypatch.setattr(inst, "probe_ready", lambda url: False)
monkeypatch.setattr(inst, "runtime_status", lambda deployment: "running")
result = CliRunner().invoke(main, ["install", "status"])
assert result.exit_code == 0, result.output
assert "Status: running" in result.output
assert "Healthy: no" in result.output
@pytest.mark.parametrize(
"payload",
[
@@ -604,7 +640,7 @@ def test_install_status_probes_live_health_response(
assert result.exit_code == 0, result.output
assert requests == ["/health", "/readyz"]
assert "Status: stopped" in result.output
assert "Status: running" in result.output
assert "Healthy: yes" in result.output
assert "Default backend: anthropic" in result.output
assert f"Anthropic target: {anthropic_target}" in result.output
+25 -4
View File
@@ -1,11 +1,14 @@
from __future__ import annotations
import errno
from types import SimpleNamespace
import click
import pytest
import headroom.cli.wrap as wrap_cli
import headroom.install.health as install_health
import headroom.install.runtime as install_runtime
@pytest.fixture(autouse=True)
@@ -17,6 +20,13 @@ def _no_attached_wrappers(monkeypatch: pytest.MonkeyPatch) -> None:
flaky. Individual tests override this to simulate attached wrappers.
"""
monkeypatch.setattr(wrap_cli, "_live_proxy_clients", lambda *a, **kw: [])
# These recovery tests control readiness through the health probe; process
# identity and launchd ownership have their own runtime-level coverage.
monkeypatch.setattr(
install_runtime,
"runtime_ready",
lambda manifest: install_health.probe_ready(manifest.health_url),
)
for name in (
"ANTHROPIC_TARGET_API_URL",
"OPENAI_TARGET_API_URL",
@@ -35,9 +45,18 @@ class _Manifest:
preset = "persistent-service"
supervisor_kind = "service"
health_url = "http://127.0.0.1:8787/readyz"
runtime_kind = "python"
def test_ensure_proxy_recovers_matching_persistent_deployment(monkeypatch) -> None:
@pytest.fixture(params=["darwin", "linux", "win32"])
def runtime_platform(monkeypatch: pytest.MonkeyPatch, request) -> None:
# Replace only this module's sys reference, never the process-wide object.
monkeypatch.setattr(install_runtime, "sys", SimpleNamespace(platform=request.param))
def test_ensure_proxy_recovers_matching_persistent_deployment(
monkeypatch, runtime_platform
) -> None:
calls: list[str] = []
monkeypatch.setattr(wrap_cli, "_check_proxy", lambda port: False)
@@ -66,7 +85,9 @@ def test_ensure_proxy_recovers_matching_persistent_deployment(monkeypatch) -> No
assert calls == ["start:default"]
def test_ensure_proxy_recovers_persistent_deployment_when_socket_is_bound(monkeypatch) -> None:
def test_ensure_proxy_recovers_persistent_deployment_when_socket_is_bound(
monkeypatch, runtime_platform
) -> None:
calls: list[str] = []
monkeypatch.setattr(wrap_cli, "_check_proxy", lambda port: True)
@@ -303,14 +324,14 @@ def test_find_persistent_manifest_prefers_default_profile(monkeypatch) -> None:
assert manifest.profile == "default"
def test_recover_persistent_proxy_reuses_healthy_deployment(monkeypatch) -> None:
def test_recover_persistent_proxy_reuses_healthy_deployment(monkeypatch, runtime_platform) -> None:
monkeypatch.setattr(wrap_cli, "_find_persistent_manifest", lambda port: _Manifest())
monkeypatch.setattr("headroom.install.health.probe_ready", lambda url: True)
assert wrap_cli._recover_persistent_proxy(8787) is True
def test_recover_persistent_proxy_warns_for_task_deployment(monkeypatch) -> None:
def test_recover_persistent_proxy_warns_for_task_deployment(monkeypatch, runtime_platform) -> None:
class TaskManifest(_Manifest):
supervisor_kind = "task"