mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 12:05:34 +08:00
## Description `headroom install status` could report `Status: stopped` next to `Healthy: yes`, while `headroom doctor` reported the same proxy as running (#3072, bug 2). The commands used independent process views: `install status` trusted PID/container bookkeeping for `Status`, while both commands used the live HTTP health endpoints to establish health. This change treats a successful `/readyz` probe as authoritative evidence that the deployment is running. If the proxy is not ready, `runtime_status` still distinguishes a running-but-unhealthy process from a stopped process. The PR also adds a focused macOS pytest lane for CLI and install code, closing the Darwin test-coverage gap described in #3065 without duplicating the full four-shard Linux suite. Bug 1 from #3072 (`HEADROOM_PORT` being ignored by `install apply` and `deploy`) was already fixed on `main` by #3085 and is not reimplemented here. Closes #3072 Closes #3065 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Updated `headroom/cli/install.py` so `install_status` performs one readiness probe and reports `Status: running` whenever `/readyz` is healthy. - Kept `runtime_status` as the fallback when the proxy is not ready, preserving the distinction between running-but-unhealthy and stopped. - Added regression coverage for the formerly contradictory `Status: stopped` plus `Healthy: yes` output. - Added regression coverage for the not-ready case where process status must still come from `runtime_status`. - Added a `test-macos` CI job that runs the CLI/install pytest subset on `macos-latest`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest tests/test_cli/test_install_cli.py -q ============================= test session starts ============================== platform darwin -- Python 3.11.15, pytest-9.0.3, pluggy-1.6.0 collected 42 items tests/test_cli/test_install_cli.py ..................................... [ 88%] ..... [100%] ============================== 42 passed in 1.79s ============================== $ uv run --frozen ruff check headroom/cli/install.py tests/test_cli/test_install_cli.py All checks passed! $ uv run --frozen ruff format --check headroom/cli/install.py tests/test_cli/test_install_cli.py 2 files already formatted $ uv run --frozen mypy headroom/cli/install.py --ignore-missing-imports Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS (Apple Silicon), CPython 3.11.15 via `uv`, pytest 9.0.3. - Exact command / steps: ran `uv run --frozen --extra dev pytest tests/test_cli/test_install_cli.py -q`. The new regression test invokes `headroom install status` through Click's `CliRunner` with `/readyz` returning healthy while PID/runtime bookkeeping returns `stopped`. - Observed result: the command exits successfully and prints `Status: running` with `Healthy: yes`; `Status: stopped` is absent. A second test verifies that when readiness is false, a live runtime still prints `Status: running` with `Healthy: no`. - Not tested: an end-to-end persistent-service installation with a real OS supervisor or Docker daemon. The new macOS CI job itself will be exercised by GitHub Actions rather than locally. ## Runtime Rollout Safety - Rollout-managed feature(s): none; this changes diagnostic display logic and CI coverage only. - Minimum rollout channel: stable/default patch release. - Stable/default behavior changed: yes; `install status` now reports a ready deployment as running even if PID/container bookkeeping says stopped. Unready deployments retain the existing runtime-status behavior. - Kill switch / disable path: no runtime flag; revert this PR if the readiness precedence causes an unforeseen regression. - Unsafe override required: none. - Qualification impact: `tests/test_cli/test_install_cli.py` and the new `test-macos` CI job must remain green. - Rollback path: revert commit `f3ae269b`; no persisted state, schema, manifest, or configuration migration is involved. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable; this is terminal status output and CI configuration. ## Additional Notes - Documentation was not changed because no command, option, configuration contract, or documented workflow changed; this corrects contradictory diagnostic output. - #2787 was also reviewed as an urgent author-reported issue, but it already has active implementation PR #2992. This PR deliberately avoids duplicating or conflicting with that work. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3141" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ### Maintainer current-main requalification Exact head `f550bf1d05873b20b773d935aa7cb93025b63262`, upstream main `2d3ee51846435ef5b320eea4806af14e602b0d8c`. Re-reviewed latest integration head3c11d4a96and current discussion; clean integration with upstream/main2d3ee51846. Install status still treats a ready reachable proxy as running, preserving runtime-status fallback when unhealthy. All 160 install-status/release/version/MCP tests pass; no PR-specific production-code blocker found in scope. Downloaded the latest failing native macOS job 126368233730 from run 41765101507: four persistent-wrap recovery tests still failed because _Manifest/TaskManifest lacked runtime_kind. Repaired test doubles with the Python runtime declaration and explicitly mocked the readiness boundary those tests control through health probes; real ownership classification remains exercised. Parametrized the four affected recovery cases across Darwin/Linux/Windows by replacing only runtime module's sys reference, without global platform emulation. All 83 persistent-wrap tests pass, preserving matching deployment recovery, bound-socket recovery, healthy reuse, and task warning coverage. No production recovery logic changed. Changed-file Ruff check/format and whitespace pass. These local platform variants are not native macOS validation; exact-head hosted macOS lane must qualify before approval. Existing native extension supports Python startup, not a fresh native build. Maintainers handle CI; nothing merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. --------- Co-authored-by: vishal veerareddy <vishalveera.reddy@servicenow.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
vishal veerareddy
Claude Sonnet 5
JerrettDavis
parent
ce14271ba8
commit
05431b264f
@@ -438,6 +438,32 @@ jobs:
|
||||
token: ${{ secrets.CODECOV_TOKEN }}
|
||||
fail_ci_if_error: false
|
||||
|
||||
# Keep a small but real pytest lane on Darwin. The full four-shard suite is
|
||||
# intentionally Linux-only for cost, but platform-gated CLI/proxy code must
|
||||
# execute somewhere on macOS; otherwise Darwin regressions can merge green
|
||||
# while being unreachable in every pytest job (#3065).
|
||||
test-macos:
|
||||
needs: changes
|
||||
if: needs.changes.outputs.code == 'true'
|
||||
runs-on: macos-latest
|
||||
timeout-minutes: 20
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
- uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: ${{ env.PY_VERSION }}
|
||||
- name: Install package and focused test dependencies
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install -e '.[proxy]' pytest pytest-asyncio numpy
|
||||
- name: Run macOS CLI and install tests
|
||||
run: |
|
||||
pytest \
|
||||
tests/test_cli \
|
||||
tests/test_cli_*.py \
|
||||
tests/test_install \
|
||||
--tb=short -q
|
||||
|
||||
test-extras:
|
||||
needs: [changes, build-wheel]
|
||||
if: needs.changes.outputs.code == 'true'
|
||||
|
||||
@@ -1166,14 +1166,20 @@ def install_status(profile: str) -> None:
|
||||
|
||||
manifest = _require_manifest(profile)
|
||||
payload = probe_json(manifest.health_url.replace("/readyz", "/health"))
|
||||
healthy = probe_ready(manifest.health_url)
|
||||
# The health endpoint is the authoritative process signal. Detached runtimes
|
||||
# can outlive a missing/stale PID file (for example after cleanup or a
|
||||
# supervisor restart), while Docker's local CLI view can also be temporarily
|
||||
# unavailable. Never report a reachable, ready proxy as "stopped" (#3072).
|
||||
status = "running" if healthy else runtime_status(manifest)
|
||||
click.echo(f"Profile: {manifest.profile}")
|
||||
click.echo(f"Preset: {manifest.preset}")
|
||||
click.echo(f"Runtime: {manifest.runtime_kind}")
|
||||
click.echo(f"Supervisor: {manifest.supervisor_kind}")
|
||||
click.echo(f"Scope: {manifest.scope}")
|
||||
click.echo(f"Port: {manifest.port}")
|
||||
click.echo(f"Status: {runtime_status(manifest)}")
|
||||
click.echo(f"Healthy: {'yes' if probe_ready(manifest.health_url) else 'no'}")
|
||||
click.echo(f"Status: {status}")
|
||||
click.echo(f"Healthy: {'yes' if healthy else 'no'}")
|
||||
if payload and isinstance(payload, dict):
|
||||
click.echo(f"Health URL: {manifest.health_url.replace('/readyz', '/health')}")
|
||||
# `config` may be a non-dict (null / string / list) if a different or
|
||||
|
||||
@@ -497,6 +497,42 @@ def test_install_status_includes_protocol_targets_from_health_probe(
|
||||
assert secret not in result.output
|
||||
|
||||
|
||||
def test_install_status_treats_ready_proxy_as_running(monkeypatch) -> None:
|
||||
"""Health wins when PID/container bookkeeping incorrectly says stopped.
|
||||
|
||||
Regression for #3072 bug 2: status previously printed the contradictory
|
||||
pair ``Status: stopped`` and ``Healthy: yes`` while doctor correctly
|
||||
reported the same proxy as running.
|
||||
"""
|
||||
manifest = _status_manifest("default")
|
||||
monkeypatch.setattr(inst, "load_manifest", lambda profile: manifest)
|
||||
monkeypatch.setattr(inst, "probe_json", lambda url: None)
|
||||
monkeypatch.setattr(inst, "probe_ready", lambda url: True)
|
||||
monkeypatch.setattr(inst, "runtime_status", lambda deployment: "stopped")
|
||||
|
||||
result = CliRunner().invoke(main, ["install", "status"])
|
||||
|
||||
assert result.exit_code == 0, result.output
|
||||
assert "Status: running" in result.output
|
||||
assert "Healthy: yes" in result.output
|
||||
assert "Status: stopped" not in result.output
|
||||
|
||||
|
||||
def test_install_status_keeps_runtime_status_when_not_ready(monkeypatch) -> None:
|
||||
"""An unhealthy live process remains distinguishable from a stopped one."""
|
||||
manifest = _status_manifest("default")
|
||||
monkeypatch.setattr(inst, "load_manifest", lambda profile: manifest)
|
||||
monkeypatch.setattr(inst, "probe_json", lambda url: None)
|
||||
monkeypatch.setattr(inst, "probe_ready", lambda url: False)
|
||||
monkeypatch.setattr(inst, "runtime_status", lambda deployment: "running")
|
||||
|
||||
result = CliRunner().invoke(main, ["install", "status"])
|
||||
|
||||
assert result.exit_code == 0, result.output
|
||||
assert "Status: running" in result.output
|
||||
assert "Healthy: no" in result.output
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"payload",
|
||||
[
|
||||
@@ -604,7 +640,7 @@ def test_install_status_probes_live_health_response(
|
||||
|
||||
assert result.exit_code == 0, result.output
|
||||
assert requests == ["/health", "/readyz"]
|
||||
assert "Status: stopped" in result.output
|
||||
assert "Status: running" in result.output
|
||||
assert "Healthy: yes" in result.output
|
||||
assert "Default backend: anthropic" in result.output
|
||||
assert f"Anthropic target: {anthropic_target}" in result.output
|
||||
|
||||
@@ -1,11 +1,14 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import errno
|
||||
from types import SimpleNamespace
|
||||
|
||||
import click
|
||||
import pytest
|
||||
|
||||
import headroom.cli.wrap as wrap_cli
|
||||
import headroom.install.health as install_health
|
||||
import headroom.install.runtime as install_runtime
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
@@ -17,6 +20,13 @@ def _no_attached_wrappers(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
flaky. Individual tests override this to simulate attached wrappers.
|
||||
"""
|
||||
monkeypatch.setattr(wrap_cli, "_live_proxy_clients", lambda *a, **kw: [])
|
||||
# These recovery tests control readiness through the health probe; process
|
||||
# identity and launchd ownership have their own runtime-level coverage.
|
||||
monkeypatch.setattr(
|
||||
install_runtime,
|
||||
"runtime_ready",
|
||||
lambda manifest: install_health.probe_ready(manifest.health_url),
|
||||
)
|
||||
for name in (
|
||||
"ANTHROPIC_TARGET_API_URL",
|
||||
"OPENAI_TARGET_API_URL",
|
||||
@@ -35,9 +45,18 @@ class _Manifest:
|
||||
preset = "persistent-service"
|
||||
supervisor_kind = "service"
|
||||
health_url = "http://127.0.0.1:8787/readyz"
|
||||
runtime_kind = "python"
|
||||
|
||||
|
||||
def test_ensure_proxy_recovers_matching_persistent_deployment(monkeypatch) -> None:
|
||||
@pytest.fixture(params=["darwin", "linux", "win32"])
|
||||
def runtime_platform(monkeypatch: pytest.MonkeyPatch, request) -> None:
|
||||
# Replace only this module's sys reference, never the process-wide object.
|
||||
monkeypatch.setattr(install_runtime, "sys", SimpleNamespace(platform=request.param))
|
||||
|
||||
|
||||
def test_ensure_proxy_recovers_matching_persistent_deployment(
|
||||
monkeypatch, runtime_platform
|
||||
) -> None:
|
||||
calls: list[str] = []
|
||||
|
||||
monkeypatch.setattr(wrap_cli, "_check_proxy", lambda port: False)
|
||||
@@ -66,7 +85,9 @@ def test_ensure_proxy_recovers_matching_persistent_deployment(monkeypatch) -> No
|
||||
assert calls == ["start:default"]
|
||||
|
||||
|
||||
def test_ensure_proxy_recovers_persistent_deployment_when_socket_is_bound(monkeypatch) -> None:
|
||||
def test_ensure_proxy_recovers_persistent_deployment_when_socket_is_bound(
|
||||
monkeypatch, runtime_platform
|
||||
) -> None:
|
||||
calls: list[str] = []
|
||||
|
||||
monkeypatch.setattr(wrap_cli, "_check_proxy", lambda port: True)
|
||||
@@ -303,14 +324,14 @@ def test_find_persistent_manifest_prefers_default_profile(monkeypatch) -> None:
|
||||
assert manifest.profile == "default"
|
||||
|
||||
|
||||
def test_recover_persistent_proxy_reuses_healthy_deployment(monkeypatch) -> None:
|
||||
def test_recover_persistent_proxy_reuses_healthy_deployment(monkeypatch, runtime_platform) -> None:
|
||||
monkeypatch.setattr(wrap_cli, "_find_persistent_manifest", lambda port: _Manifest())
|
||||
monkeypatch.setattr("headroom.install.health.probe_ready", lambda url: True)
|
||||
|
||||
assert wrap_cli._recover_persistent_proxy(8787) is True
|
||||
|
||||
|
||||
def test_recover_persistent_proxy_warns_for_task_deployment(monkeypatch) -> None:
|
||||
def test_recover_persistent_proxy_warns_for_task_deployment(monkeypatch, runtime_platform) -> None:
|
||||
class TaskManifest(_Manifest):
|
||||
supervisor_kind = "task"
|
||||
|
||||
|
||||
Reference in New Issue
Block a user