mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description Deploying on a non-default port produced a working proxy that the surrounding tooling reported as broken. Tracing the probe turned up a second defect upstream of it: the container could not change its port via `HEADROOM_PORT` at all. `headroom proxy --port` is a click option with `envvar="HEADROOM_PORT"`, and an explicit CLI argument beats the envvar. The Dockerfile baked `--port 8787` into CMD, so `HEADROOM_PORT` was inert for a plain `docker run` — while `docker-compose.yml` already passes `command: ["--host", "0.0.0.0"]` with no port, which is the intended shape. The Dockerfile was the outlier. The two halves have to move together. An earlier revision of this change fixed only the HEALTHCHECK; that made the probe follow `HEADROOM_PORT` while the listener stayed pinned to 8787, and the container went from misleadingly healthy to genuinely unhealthy. Evidence in Real Behavior Proof below. Closes #2432 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `Dockerfile`, both runtime stages: dropped the baked `--port` from CMD so `HEADROOM_PORT` stays authoritative. `docker run` with no env still binds 8787 (click's default), so the published default is unchanged. - `Dockerfile` (`runtime-slim-base`): shell-form HEALTHCHECK so `${HEADROOM_PORT:-8787}` expands at probe time. - `Dockerfile` (`runtime-slim`): distroless, no shell — the port is resolved inside Python instead, keeping exec form. - `headroom/cli/doctor.py`: added `resolve_probe_port()`. `--port` / `HEADROOM_PORT` wins, then an unambiguous deployment manifest (`default` profile, or a lone named profile), then 8787. `headroom deploy --port N` writes N into the deployment's environment, not into the shell `doctor` runs in, which is why the manifest tier is needed; with several named profiles and no explicit request there is no non-arbitrary pick, so it stays on the default and the existing per-deployment section reports each one. `list_manifests()` is now read once and shared with that check. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom --ignore-missing-imports`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output Full suite, run inside this repo's own builder image so the maturin-compiled Rust core is the one under test: ```text 9 failed, 9248 passed, 853 skipped, 2230 warnings in 384.56s (0:06:24) ``` All 9 failures reproduce unchanged on `origin/main` (c400f908) in the same container. Restricted to just those files, base and branch are identical: ```text origin/mainc400f908: 13 failed, 119 passed, 15 skipped, 66 warnings in 14.57s this branch: 13 failed, 119 passed, 15 skipped, 66 warnings in 14.58s ``` They are `test_cli/test_wrap_helpers.py`, `test_image_compression.py`, `test_memory_bridge.py`, `test_proxy/test_cc_switch_reconciler.py` and `test_startup_log_noise.py`. Counts differ between the full and file-scoped runs because `test_memory_bridge` and `test_startup_log_noise` are order-dependent — also true on the base. ```text $ mypy headroom --ignore-missing-imports pyproject.toml: note: unused section(s): module = ['mlx.*'] Success: no issues found in 505 source files ``` New and changed tests, green on this branch: ```text tests/test_docker_healthcheck_port.py ...... [100%] 6 passed in 0.15s tests/test_cli_doctor.py ............................. [100%] 67 passed in 0.68s ``` Both go red with only the tests applied on top of `origin/main`: ```text FAILED tests/test_docker_healthcheck_port.py::test_no_healthcheck_hardcodes_the_default_port FAILED tests/test_docker_healthcheck_port.py::test_every_healthcheck_resolves_the_port_from_the_environment FAILED tests/test_docker_healthcheck_port.py::test_distroless_stage_healthcheck_stays_shell_free FAILED tests/test_docker_healthcheck_port.py::test_debian_stage_healthcheck_expands_the_port_variable FAILED tests/test_docker_healthcheck_port.py::test_no_stage_bakes_a_port_into_cmd 5 failed, 1 passed in 0.23s tests/test_cli_doctor.py: E ImportError: cannot import name 'DEFAULT_PROXY_PORT' from 'headroom.cli.doctor' ``` ## Real Behavior Proof - Environment: macOS 15 (Darwin 25.3.0) arm64, Docker 29.1.3, image built from this branch with `docker build -t headroom:pr2432 .`; control is the published `ghcr.io/headroomlabs-ai/headroom:latest`. - Exact command / steps: started three containers — the patched image with `HEADROOM_PORT=8789`, the patched image with no env at all, and the published image with `HEADROOM_PORT=8790` as a control — then read `docker ps` health and probed `/readyz` from inside each. Full commands below. - Observed result: the patched image binds 8789 and its probe follows, reporting healthy; with no env it still binds 8787, so the published default is unchanged; the published image ignores `HEADROOM_PORT` entirely and stays on 8787. Output below. - Not tested: `headroom deploy` was not driven end to end — the manifest was written directly rather than produced by the deploy command, so the manifest *contents* are assumed to match what deploy records (`port` is the field `headroom install status` already reads). Non-Docker supervisors were not exercised. ### Containers ```bash docker run -d --name hr-fixed -e HEADROOM_PORT=8789 -p 8789:8789 headroom:pr2432 docker run -d --name hr-default -p 8791:8787 headroom:pr2432 docker run -d --name hr-control -e HEADROOM_PORT=8790 -p 8790:8790 \ ghcr.io/headroomlabs-ai/headroom:latest ``` ```text NAMES STATUS hr-fixed Up (healthy) HEADROOM_PORT=8789 -> listener on 8789 hr-default Up (healthy) no env -> listener on 8787 hr-control Up (healthy) HEADROOM_PORT=8790 -> listener on 8787 ``` Probing from inside each container confirms which port is actually serving: ```text hr-fixed: 8789=OK hr-default: 8787=OK hr-control: 8787=OK ``` `hr-control` is the published image ignoring `HEADROOM_PORT` entirely — the CMD half of the bug. Baked image config, before and after: ```text published: ["CMD","curl","--fail","--silent","http://127.0.0.1:8787/readyz"] this PR: ["CMD-SHELL","curl --fail --silent \"http://127.0.0.1:${HEADROOM_PORT:-8787}/readyz\""] ``` The distroless stage was verified the same way — `docker build --target runtime-slim` with `-e HEADROOM_PORT=8795` reports healthy and answers `GET /readyz` with 200 from the host. ### doctor Running the real CLI against a seeded `default` deployment manifest recording port 8789, with `HEADROOM_PORT` unset in the shell — the reported situation, where the port lives in the deployment rather than in the shell: ```text origin/mainc400f908: doctor probed port: 8787 this branch: doctor probed port: 8789 ``` ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — no UI surface. ## Additional Notes "I have made corresponding changes to the documentation" is checked in the sense that the `--port` help text now describes the new resolution order; no prose docs referenced the baked CMD port. `README.md` and the wiki reference `headroom proxy --port 8787` as a host command, which is unaffected. `docker/differential-network-capture/docker-compose.yml` passes `--port 8787` explicitly in its own `command:`, which still overrides everything and keeps working. Adjacent, deliberately not in this PR: `docker-compose.yml`'s healthcheck is hardcoded to 8787 the same way. It is inert today because that file never sets `HEADROOM_PORT`, and making it configurable also means parameterising the published `ports:` mapping — a separate change. Happy to open a follow-up if you want it. ## Runtime Rollout Safety - Rollout-managed feature(s): Docker runtime port selection and `headroom doctor` probe-port resolution. - Minimum rollout channel: Standard Docker image release and persistent Docker deployments. - Stable/default behavior changed: No. Unset `HEADROOM_PORT` continues to use port 8787. - Kill switch / disable path: No new feature flag; revert the image or this PR. - Unsafe override required: No. - Qualification impact: None. No data or schema migration; existing port-8787 deployments remain compatible. - Rollback path: Redeploy the previous image or revert this PR; no migration cleanup is required. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2436" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: JD Davis <mxjerrett@gmail.com>
143 lines
6.1 KiB
Python
143 lines
6.1 KiB
Python
"""Regression checks for Docker Compose persistence wiring."""
|
|
|
|
from __future__ import annotations
|
|
|
|
from pathlib import Path
|
|
|
|
import yaml
|
|
|
|
ROOT = Path(__file__).resolve().parents[1]
|
|
|
|
|
|
def test_top_level_compose_publishes_only_to_loopback() -> None:
|
|
"""Every published port must name an explicit loopback host IP.
|
|
|
|
None of the three services authenticates by default: the proxy's /v1/*
|
|
data plane is open without HEADROOM_PROXY_TOKEN, Qdrant has no API key,
|
|
and NEO4J_AUTH falls back to a password published in the compose file. A
|
|
bare "8787:8787" binds 0.0.0.0 on the host, so `docker compose up -d` on a
|
|
shared network would expose all three.
|
|
"""
|
|
compose = yaml.safe_load((ROOT / "docker-compose.yml").read_text(encoding="utf-8"))
|
|
|
|
published = [
|
|
(service, port)
|
|
for service, spec in compose["services"].items()
|
|
for port in spec.get("ports", [])
|
|
]
|
|
assert published, "expected the compose file to publish at least one port"
|
|
|
|
for service, port in published:
|
|
# Short syntax is "HOST_IP:HOST_PORT:CONTAINER_PORT"; anything with
|
|
# fewer than three segments is published on every interface.
|
|
assert isinstance(port, str), f"{service}: expected short-syntax port, got {port!r}"
|
|
segments = port.split(":")
|
|
assert len(segments) == 3, f"{service}: port {port!r} publishes on all interfaces"
|
|
assert segments[0] in {"127.0.0.1", "::1"}, (
|
|
f"{service}: port {port!r} publishes on {segments[0]}, not loopback"
|
|
)
|
|
|
|
|
|
def test_top_level_compose_pins_headroom_state_to_named_volume() -> None:
|
|
compose = (ROOT / "docker-compose.yml").read_text(encoding="utf-8")
|
|
|
|
assert "- headroom_workspace:/home/nonroot/.headroom" in compose
|
|
assert "- HOME=/home/nonroot" in compose
|
|
assert "- HEADROOM_WORKSPACE_DIR=/home/nonroot/.headroom" in compose
|
|
assert "- HEADROOM_CONFIG_DIR=/home/nonroot/.headroom/config" in compose
|
|
|
|
|
|
def test_top_level_compose_marks_source_build_version() -> None:
|
|
compose = (ROOT / "docker-compose.yml").read_text(encoding="utf-8")
|
|
dockerfile = (ROOT / "Dockerfile").read_text(encoding="utf-8")
|
|
dockerignore = (ROOT / ".dockerignore").read_text(encoding="utf-8")
|
|
|
|
assert "HEADROOM_BUILD_VERSION: ${HEADROOM_BUILD_VERSION:-source-build}" in compose
|
|
assert 'ARG HEADROOM_BUILD_VERSION=""' in dockerfile
|
|
assert "ARG PYTHON_SITE_PACKAGES" in dockerfile
|
|
assert "if not build_version:" in dockerfile
|
|
assert "source-build+g{revision}" in dockerfile
|
|
assert "source-build+sha256." in dockerfile
|
|
assert "_build_info.py" in dockerfile
|
|
assert "import headroom._version" not in dockerfile
|
|
assert ".git/*" in dockerignore
|
|
assert "!.git/HEAD" in dockerignore
|
|
assert "!.git/refs/**" in dockerignore
|
|
|
|
|
|
def test_native_compose_keeps_host_publication_loopback_and_optional_token() -> None:
|
|
compose = yaml.safe_load(
|
|
(ROOT / "docker" / "docker-compose.native.yml").read_text(encoding="utf-8")
|
|
)
|
|
proxy = compose["services"]["proxy"]
|
|
|
|
assert proxy["ports"] == ["127.0.0.1:${HEADROOM_PORT:-8787}:${HEADROOM_PORT:-8787}"]
|
|
assert proxy["environment"]["HEADROOM_HOST"] == "0.0.0.0"
|
|
assert proxy["environment"]["HEADROOM_PROXY_TOKEN"] == "${HEADROOM_PROXY_TOKEN:-}"
|
|
assert proxy["command"] == ["--host", "0.0.0.0", "--port", "${HEADROOM_PORT:-8787}"]
|
|
|
|
|
|
def test_native_compose_lets_the_host_dashboard_read_its_data(monkeypatch) -> None:
|
|
"""The compose proxy must serve /stats-history to the host browser, which
|
|
reaches the 127.0.0.1 publication as the compose network's gateway."""
|
|
from fastapi.testclient import TestClient
|
|
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
|
|
compose = yaml.safe_load(
|
|
(ROOT / "docker" / "docker-compose.native.yml").read_text(encoding="utf-8")
|
|
)
|
|
proxy = compose["services"]["proxy"]
|
|
assert proxy["ports"][0].startswith("127.0.0.1:")
|
|
|
|
for name, value in proxy["environment"].items():
|
|
if name != "HOME" and not name.endswith("_DIR"):
|
|
# "${VAR:-default}" with VAR unset, as compose resolves it.
|
|
value = value.split(":-", 1)[1].rstrip("}") if value.startswith("${") else value
|
|
monkeypatch.setenv(name, value)
|
|
monkeypatch.setenv("HEADROOM_CONTAINER_HOST_GATEWAY", "172.18.0.1")
|
|
host = proxy["command"][proxy["command"].index("--host") + 1]
|
|
app = create_app(ProxyConfig(host=host, optimize=False, cache_enabled=False))
|
|
|
|
browser = TestClient(app, base_url="http://127.0.0.1:8787", client=("172.18.0.1", 1))
|
|
assert browser.get("/stats-history").status_code == 200
|
|
other_container = TestClient(app, base_url="http://127.0.0.1:8787", client=("172.18.0.5", 1))
|
|
assert other_container.get("/stats-history").status_code == 404
|
|
|
|
|
|
def test_differential_capture_compose_keeps_proxy_internal_and_host_ports_loopback() -> None:
|
|
compose = yaml.safe_load(
|
|
(ROOT / "docker" / "differential-network-capture" / "docker-compose.yml").read_text(
|
|
encoding="utf-8"
|
|
)
|
|
)
|
|
services = compose["services"]
|
|
assert (
|
|
services["headroom-proxy"]["environment"]["HEADROOM_PROXY_TOKEN"]
|
|
== "${HEADROOM_PROXY_TOKEN:?Set HEADROOM_PROXY_TOKEN}"
|
|
)
|
|
assert (
|
|
services["mitm-headroom-client"]["environment"]["CAPTURE_PROXY_TOKEN"]
|
|
== "${HEADROOM_PROXY_TOKEN:?Set HEADROOM_PROXY_TOKEN}"
|
|
)
|
|
assert (
|
|
services["mitm-headroom-client"]["environment"]["CAPTURE_PROXY_TOKEN_HOSTS"]
|
|
== "headroom-proxy"
|
|
)
|
|
assert services["mitm-direct"]["ports"] == ["127.0.0.1:${DIRECT_MITM_PORT:-18080}:8080"]
|
|
assert services["mitm-headroom-upstream"]["ports"] == [
|
|
"127.0.0.1:${HEADROOM_MITM_PORT:-18081}:8080"
|
|
]
|
|
assert "ports" not in services["mitm-headroom-client"]
|
|
assert "HEADROOM_CLIENT_MITM_PORT" not in compose["services"]["mitm-headroom-client"].get(
|
|
"environment", {}
|
|
)
|
|
|
|
|
|
def test_published_dockerfile_stages_default_to_loopback() -> None:
|
|
dockerfile = (ROOT / "Dockerfile").read_text(encoding="utf-8")
|
|
|
|
assert dockerfile.count("ENV HEADROOM_HOST=127.0.0.1") == 2
|
|
assert 'CMD ["--port", "8787"]' not in dockerfile
|
|
assert 'CMD ["--host", "0.0.0.0"]' not in dockerfile
|