Files
headroom/tests/test_docker_compose_persistence.py
natoriandJD Davis 1b7977cecd fix: honor configured port in container startup and healthcheck (#2436)
## Description

Deploying on a non-default port produced a working proxy that the
surrounding
tooling reported as broken. Tracing the probe turned up a second defect
upstream of it: the container could not change its port via
`HEADROOM_PORT`
at all.

`headroom proxy --port` is a click option with `envvar="HEADROOM_PORT"`,
and an
explicit CLI argument beats the envvar. The Dockerfile baked `--port
8787` into
CMD, so `HEADROOM_PORT` was inert for a plain `docker run` — while
`docker-compose.yml` already passes `command: ["--host", "0.0.0.0"]`
with no
port, which is the intended shape. The Dockerfile was the outlier.

The two halves have to move together. An earlier revision of this change
fixed
only the HEALTHCHECK; that made the probe follow `HEADROOM_PORT` while
the
listener stayed pinned to 8787, and the container went from misleadingly
healthy to genuinely unhealthy. Evidence in Real Behavior Proof below.

Closes #2432

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `Dockerfile`, both runtime stages: dropped the baked `--port` from CMD
so
`HEADROOM_PORT` stays authoritative. `docker run` with no env still
binds
  8787 (click's default), so the published default is unchanged.
- `Dockerfile` (`runtime-slim-base`): shell-form HEALTHCHECK so
  `${HEADROOM_PORT:-8787}` expands at probe time.
- `Dockerfile` (`runtime-slim`): distroless, no shell — the port is
resolved
  inside Python instead, keeping exec form.
- `headroom/cli/doctor.py`: added `resolve_probe_port()`. `--port` /
`HEADROOM_PORT` wins, then an unambiguous deployment manifest (`default`
profile, or a lone named profile), then 8787. `headroom deploy --port N`
writes N into the deployment's environment, not into the shell `doctor`
runs
in, which is why the manifest tier is needed; with several named
profiles and
  no explicit request there is no non-arbitrary pick, so it stays on the
  default and the existing per-deployment section reports each one.
  `list_manifests()` is now read once and shared with that check.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom --ignore-missing-imports`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

Full suite, run inside this repo's own builder image so the
maturin-compiled
Rust core is the one under test:

```text
9 failed, 9248 passed, 853 skipped, 2230 warnings in 384.56s (0:06:24)
```

All 9 failures reproduce unchanged on `origin/main` (c400f908) in the
same
container. Restricted to just those files, base and branch are
identical:

```text
origin/main c400f908: 13 failed, 119 passed, 15 skipped, 66 warnings in 14.57s
this branch:          13 failed, 119 passed, 15 skipped, 66 warnings in 14.58s
```

They are `test_cli/test_wrap_helpers.py`, `test_image_compression.py`,
`test_memory_bridge.py`, `test_proxy/test_cc_switch_reconciler.py` and
`test_startup_log_noise.py`. Counts differ between the full and
file-scoped
runs because `test_memory_bridge` and `test_startup_log_noise` are
order-dependent — also true on the base.

```text
$ mypy headroom --ignore-missing-imports
pyproject.toml: note: unused section(s): module = ['mlx.*']
Success: no issues found in 505 source files
```

New and changed tests, green on this branch:

```text
tests/test_docker_healthcheck_port.py ......                   [100%]
6 passed in 0.15s

tests/test_cli_doctor.py .............................         [100%]
67 passed in 0.68s
```

Both go red with only the tests applied on top of `origin/main`:

```text
FAILED tests/test_docker_healthcheck_port.py::test_no_healthcheck_hardcodes_the_default_port
FAILED tests/test_docker_healthcheck_port.py::test_every_healthcheck_resolves_the_port_from_the_environment
FAILED tests/test_docker_healthcheck_port.py::test_distroless_stage_healthcheck_stays_shell_free
FAILED tests/test_docker_healthcheck_port.py::test_debian_stage_healthcheck_expands_the_port_variable
FAILED tests/test_docker_healthcheck_port.py::test_no_stage_bakes_a_port_into_cmd
5 failed, 1 passed in 0.23s

tests/test_cli_doctor.py:
E   ImportError: cannot import name 'DEFAULT_PROXY_PORT' from 'headroom.cli.doctor'
```

## Real Behavior Proof

- Environment: macOS 15 (Darwin 25.3.0) arm64, Docker 29.1.3, image
built from this branch with `docker build -t headroom:pr2432 .`; control
is the published `ghcr.io/headroomlabs-ai/headroom:latest`.
- Exact command / steps: started three containers — the patched image
with `HEADROOM_PORT=8789`, the patched image with no env at all, and the
published image with `HEADROOM_PORT=8790` as a control — then read
`docker ps` health and probed `/readyz` from inside each. Full commands
below.
- Observed result: the patched image binds 8789 and its probe follows,
reporting healthy; with no env it still binds 8787, so the published
default is unchanged; the published image ignores `HEADROOM_PORT`
entirely and stays on 8787. Output below.
- Not tested: `headroom deploy` was not driven end to end — the manifest
was written directly rather than produced by the deploy command, so the
manifest *contents* are assumed to match what deploy records (`port` is
the field `headroom install status` already reads). Non-Docker
supervisors were not exercised.

### Containers

```bash
docker run -d --name hr-fixed   -e HEADROOM_PORT=8789 -p 8789:8789 headroom:pr2432
docker run -d --name hr-default                       -p 8791:8787 headroom:pr2432
docker run -d --name hr-control -e HEADROOM_PORT=8790 -p 8790:8790 \
    ghcr.io/headroomlabs-ai/headroom:latest
```

```text
NAMES        STATUS
hr-fixed     Up (healthy)   HEADROOM_PORT=8789 -> listener on 8789
hr-default   Up (healthy)   no env             -> listener on 8787
hr-control   Up (healthy)   HEADROOM_PORT=8790 -> listener on 8787
```

Probing from inside each container confirms which port is actually
serving:

```text
hr-fixed:   8789=OK
hr-default: 8787=OK
hr-control: 8787=OK
```

`hr-control` is the published image ignoring `HEADROOM_PORT` entirely —
the CMD
half of the bug. Baked image config, before and after:

```text
published: ["CMD","curl","--fail","--silent","http://127.0.0.1:8787/readyz"]
this PR:   ["CMD-SHELL","curl --fail --silent \"http://127.0.0.1:${HEADROOM_PORT:-8787}/readyz\""]
```

The distroless stage was verified the same way — `docker build --target
runtime-slim` with `-e HEADROOM_PORT=8795` reports healthy and answers
`GET /readyz` with 200 from the host.

### doctor

Running the real CLI against a seeded `default` deployment manifest
recording
port 8789, with `HEADROOM_PORT` unset in the shell — the reported
situation,
where the port lives in the deployment rather than in the shell:

```text
origin/main c400f908: doctor probed port: 8787
this branch:          doctor probed port: 8789
```

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from
      my Conventional Commit PR title (a CI guard enforces this)

## Screenshots (if applicable)

N/A — no UI surface.

## Additional Notes

"I have made corresponding changes to the documentation" is checked in
the sense
that the `--port` help text now describes the new resolution order; no
prose
docs referenced the baked CMD port. `README.md` and the wiki reference
`headroom proxy --port 8787` as a host command, which is unaffected.

`docker/differential-network-capture/docker-compose.yml` passes `--port
8787`
explicitly in its own `command:`, which still overrides everything and
keeps
working.

Adjacent, deliberately not in this PR: `docker-compose.yml`'s
healthcheck is
hardcoded to 8787 the same way. It is inert today because that file
never sets
`HEADROOM_PORT`, and making it configurable also means parameterising
the
published `ports:` mapping — a separate change. Happy to open a
follow-up if
you want it.

## Runtime Rollout Safety

- Rollout-managed feature(s): Docker runtime port selection and
`headroom doctor` probe-port resolution.
- Minimum rollout channel: Standard Docker image release and persistent
Docker deployments.
- Stable/default behavior changed: No. Unset `HEADROOM_PORT` continues
to use port 8787.
- Kill switch / disable path: No new feature flag; revert the image or
this PR.
- Unsafe override required: No.
- Qualification impact: None. No data or schema migration; existing
port-8787 deployments remain compatible.
- Rollback path: Redeploy the previous image or revert this PR; no
migration cleanup is required.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2436"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-05 08:36:54 -05:00

143 lines
6.1 KiB
Python

"""Regression checks for Docker Compose persistence wiring."""
from __future__ import annotations
from pathlib import Path
import yaml
ROOT = Path(__file__).resolve().parents[1]
def test_top_level_compose_publishes_only_to_loopback() -> None:
"""Every published port must name an explicit loopback host IP.
None of the three services authenticates by default: the proxy's /v1/*
data plane is open without HEADROOM_PROXY_TOKEN, Qdrant has no API key,
and NEO4J_AUTH falls back to a password published in the compose file. A
bare "8787:8787" binds 0.0.0.0 on the host, so `docker compose up -d` on a
shared network would expose all three.
"""
compose = yaml.safe_load((ROOT / "docker-compose.yml").read_text(encoding="utf-8"))
published = [
(service, port)
for service, spec in compose["services"].items()
for port in spec.get("ports", [])
]
assert published, "expected the compose file to publish at least one port"
for service, port in published:
# Short syntax is "HOST_IP:HOST_PORT:CONTAINER_PORT"; anything with
# fewer than three segments is published on every interface.
assert isinstance(port, str), f"{service}: expected short-syntax port, got {port!r}"
segments = port.split(":")
assert len(segments) == 3, f"{service}: port {port!r} publishes on all interfaces"
assert segments[0] in {"127.0.0.1", "::1"}, (
f"{service}: port {port!r} publishes on {segments[0]}, not loopback"
)
def test_top_level_compose_pins_headroom_state_to_named_volume() -> None:
compose = (ROOT / "docker-compose.yml").read_text(encoding="utf-8")
assert "- headroom_workspace:/home/nonroot/.headroom" in compose
assert "- HOME=/home/nonroot" in compose
assert "- HEADROOM_WORKSPACE_DIR=/home/nonroot/.headroom" in compose
assert "- HEADROOM_CONFIG_DIR=/home/nonroot/.headroom/config" in compose
def test_top_level_compose_marks_source_build_version() -> None:
compose = (ROOT / "docker-compose.yml").read_text(encoding="utf-8")
dockerfile = (ROOT / "Dockerfile").read_text(encoding="utf-8")
dockerignore = (ROOT / ".dockerignore").read_text(encoding="utf-8")
assert "HEADROOM_BUILD_VERSION: ${HEADROOM_BUILD_VERSION:-source-build}" in compose
assert 'ARG HEADROOM_BUILD_VERSION=""' in dockerfile
assert "ARG PYTHON_SITE_PACKAGES" in dockerfile
assert "if not build_version:" in dockerfile
assert "source-build+g{revision}" in dockerfile
assert "source-build+sha256." in dockerfile
assert "_build_info.py" in dockerfile
assert "import headroom._version" not in dockerfile
assert ".git/*" in dockerignore
assert "!.git/HEAD" in dockerignore
assert "!.git/refs/**" in dockerignore
def test_native_compose_keeps_host_publication_loopback_and_optional_token() -> None:
compose = yaml.safe_load(
(ROOT / "docker" / "docker-compose.native.yml").read_text(encoding="utf-8")
)
proxy = compose["services"]["proxy"]
assert proxy["ports"] == ["127.0.0.1:${HEADROOM_PORT:-8787}:${HEADROOM_PORT:-8787}"]
assert proxy["environment"]["HEADROOM_HOST"] == "0.0.0.0"
assert proxy["environment"]["HEADROOM_PROXY_TOKEN"] == "${HEADROOM_PROXY_TOKEN:-}"
assert proxy["command"] == ["--host", "0.0.0.0", "--port", "${HEADROOM_PORT:-8787}"]
def test_native_compose_lets_the_host_dashboard_read_its_data(monkeypatch) -> None:
"""The compose proxy must serve /stats-history to the host browser, which
reaches the 127.0.0.1 publication as the compose network's gateway."""
from fastapi.testclient import TestClient
from headroom.proxy.server import ProxyConfig, create_app
compose = yaml.safe_load(
(ROOT / "docker" / "docker-compose.native.yml").read_text(encoding="utf-8")
)
proxy = compose["services"]["proxy"]
assert proxy["ports"][0].startswith("127.0.0.1:")
for name, value in proxy["environment"].items():
if name != "HOME" and not name.endswith("_DIR"):
# "${VAR:-default}" with VAR unset, as compose resolves it.
value = value.split(":-", 1)[1].rstrip("}") if value.startswith("${") else value
monkeypatch.setenv(name, value)
monkeypatch.setenv("HEADROOM_CONTAINER_HOST_GATEWAY", "172.18.0.1")
host = proxy["command"][proxy["command"].index("--host") + 1]
app = create_app(ProxyConfig(host=host, optimize=False, cache_enabled=False))
browser = TestClient(app, base_url="http://127.0.0.1:8787", client=("172.18.0.1", 1))
assert browser.get("/stats-history").status_code == 200
other_container = TestClient(app, base_url="http://127.0.0.1:8787", client=("172.18.0.5", 1))
assert other_container.get("/stats-history").status_code == 404
def test_differential_capture_compose_keeps_proxy_internal_and_host_ports_loopback() -> None:
compose = yaml.safe_load(
(ROOT / "docker" / "differential-network-capture" / "docker-compose.yml").read_text(
encoding="utf-8"
)
)
services = compose["services"]
assert (
services["headroom-proxy"]["environment"]["HEADROOM_PROXY_TOKEN"]
== "${HEADROOM_PROXY_TOKEN:?Set HEADROOM_PROXY_TOKEN}"
)
assert (
services["mitm-headroom-client"]["environment"]["CAPTURE_PROXY_TOKEN"]
== "${HEADROOM_PROXY_TOKEN:?Set HEADROOM_PROXY_TOKEN}"
)
assert (
services["mitm-headroom-client"]["environment"]["CAPTURE_PROXY_TOKEN_HOSTS"]
== "headroom-proxy"
)
assert services["mitm-direct"]["ports"] == ["127.0.0.1:${DIRECT_MITM_PORT:-18080}:8080"]
assert services["mitm-headroom-upstream"]["ports"] == [
"127.0.0.1:${HEADROOM_MITM_PORT:-18081}:8080"
]
assert "ports" not in services["mitm-headroom-client"]
assert "HEADROOM_CLIENT_MITM_PORT" not in compose["services"]["mitm-headroom-client"].get(
"environment", {}
)
def test_published_dockerfile_stages_default_to_loopback() -> None:
dockerfile = (ROOT / "Dockerfile").read_text(encoding="utf-8")
assert dockerfile.count("ENV HEADROOM_HOST=127.0.0.1") == 2
assert 'CMD ["--port", "8787"]' not in dockerfile
assert 'CMD ["--host", "0.0.0.0"]' not in dockerfile