fix: honor configured port in container startup and healthcheck (#2436)

## Description

Deploying on a non-default port produced a working proxy that the
surrounding
tooling reported as broken. Tracing the probe turned up a second defect
upstream of it: the container could not change its port via
`HEADROOM_PORT`
at all.

`headroom proxy --port` is a click option with `envvar="HEADROOM_PORT"`,
and an
explicit CLI argument beats the envvar. The Dockerfile baked `--port
8787` into
CMD, so `HEADROOM_PORT` was inert for a plain `docker run` — while
`docker-compose.yml` already passes `command: ["--host", "0.0.0.0"]`
with no
port, which is the intended shape. The Dockerfile was the outlier.

The two halves have to move together. An earlier revision of this change
fixed
only the HEALTHCHECK; that made the probe follow `HEADROOM_PORT` while
the
listener stayed pinned to 8787, and the container went from misleadingly
healthy to genuinely unhealthy. Evidence in Real Behavior Proof below.

Closes #2432

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `Dockerfile`, both runtime stages: dropped the baked `--port` from CMD
so
`HEADROOM_PORT` stays authoritative. `docker run` with no env still
binds
  8787 (click's default), so the published default is unchanged.
- `Dockerfile` (`runtime-slim-base`): shell-form HEALTHCHECK so
  `${HEADROOM_PORT:-8787}` expands at probe time.
- `Dockerfile` (`runtime-slim`): distroless, no shell — the port is
resolved
  inside Python instead, keeping exec form.
- `headroom/cli/doctor.py`: added `resolve_probe_port()`. `--port` /
`HEADROOM_PORT` wins, then an unambiguous deployment manifest (`default`
profile, or a lone named profile), then 8787. `headroom deploy --port N`
writes N into the deployment's environment, not into the shell `doctor`
runs
in, which is why the manifest tier is needed; with several named
profiles and
  no explicit request there is no non-arbitrary pick, so it stays on the
  default and the existing per-deployment section reports each one.
  `list_manifests()` is now read once and shared with that check.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom --ignore-missing-imports`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

Full suite, run inside this repo's own builder image so the
maturin-compiled
Rust core is the one under test:

```text
9 failed, 9248 passed, 853 skipped, 2230 warnings in 384.56s (0:06:24)
```

All 9 failures reproduce unchanged on `origin/main` (c400f908) in the
same
container. Restricted to just those files, base and branch are
identical:

```text
origin/main c400f908: 13 failed, 119 passed, 15 skipped, 66 warnings in 14.57s
this branch:          13 failed, 119 passed, 15 skipped, 66 warnings in 14.58s
```

They are `test_cli/test_wrap_helpers.py`, `test_image_compression.py`,
`test_memory_bridge.py`, `test_proxy/test_cc_switch_reconciler.py` and
`test_startup_log_noise.py`. Counts differ between the full and
file-scoped
runs because `test_memory_bridge` and `test_startup_log_noise` are
order-dependent — also true on the base.

```text
$ mypy headroom --ignore-missing-imports
pyproject.toml: note: unused section(s): module = ['mlx.*']
Success: no issues found in 505 source files
```

New and changed tests, green on this branch:

```text
tests/test_docker_healthcheck_port.py ......                   [100%]
6 passed in 0.15s

tests/test_cli_doctor.py .............................         [100%]
67 passed in 0.68s
```

Both go red with only the tests applied on top of `origin/main`:

```text
FAILED tests/test_docker_healthcheck_port.py::test_no_healthcheck_hardcodes_the_default_port
FAILED tests/test_docker_healthcheck_port.py::test_every_healthcheck_resolves_the_port_from_the_environment
FAILED tests/test_docker_healthcheck_port.py::test_distroless_stage_healthcheck_stays_shell_free
FAILED tests/test_docker_healthcheck_port.py::test_debian_stage_healthcheck_expands_the_port_variable
FAILED tests/test_docker_healthcheck_port.py::test_no_stage_bakes_a_port_into_cmd
5 failed, 1 passed in 0.23s

tests/test_cli_doctor.py:
E   ImportError: cannot import name 'DEFAULT_PROXY_PORT' from 'headroom.cli.doctor'
```

## Real Behavior Proof

- Environment: macOS 15 (Darwin 25.3.0) arm64, Docker 29.1.3, image
built from this branch with `docker build -t headroom:pr2432 .`; control
is the published `ghcr.io/headroomlabs-ai/headroom:latest`.
- Exact command / steps: started three containers — the patched image
with `HEADROOM_PORT=8789`, the patched image with no env at all, and the
published image with `HEADROOM_PORT=8790` as a control — then read
`docker ps` health and probed `/readyz` from inside each. Full commands
below.
- Observed result: the patched image binds 8789 and its probe follows,
reporting healthy; with no env it still binds 8787, so the published
default is unchanged; the published image ignores `HEADROOM_PORT`
entirely and stays on 8787. Output below.
- Not tested: `headroom deploy` was not driven end to end — the manifest
was written directly rather than produced by the deploy command, so the
manifest *contents* are assumed to match what deploy records (`port` is
the field `headroom install status` already reads). Non-Docker
supervisors were not exercised.

### Containers

```bash
docker run -d --name hr-fixed   -e HEADROOM_PORT=8789 -p 8789:8789 headroom:pr2432
docker run -d --name hr-default                       -p 8791:8787 headroom:pr2432
docker run -d --name hr-control -e HEADROOM_PORT=8790 -p 8790:8790 \
    ghcr.io/headroomlabs-ai/headroom:latest
```

```text
NAMES        STATUS
hr-fixed     Up (healthy)   HEADROOM_PORT=8789 -> listener on 8789
hr-default   Up (healthy)   no env             -> listener on 8787
hr-control   Up (healthy)   HEADROOM_PORT=8790 -> listener on 8787
```

Probing from inside each container confirms which port is actually
serving:

```text
hr-fixed:   8789=OK
hr-default: 8787=OK
hr-control: 8787=OK
```

`hr-control` is the published image ignoring `HEADROOM_PORT` entirely —
the CMD
half of the bug. Baked image config, before and after:

```text
published: ["CMD","curl","--fail","--silent","http://127.0.0.1:8787/readyz"]
this PR:   ["CMD-SHELL","curl --fail --silent \"http://127.0.0.1:${HEADROOM_PORT:-8787}/readyz\""]
```

The distroless stage was verified the same way — `docker build --target
runtime-slim` with `-e HEADROOM_PORT=8795` reports healthy and answers
`GET /readyz` with 200 from the host.

### doctor

Running the real CLI against a seeded `default` deployment manifest
recording
port 8789, with `HEADROOM_PORT` unset in the shell — the reported
situation,
where the port lives in the deployment rather than in the shell:

```text
origin/main c400f908: doctor probed port: 8787
this branch:          doctor probed port: 8789
```

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from
      my Conventional Commit PR title (a CI guard enforces this)

## Screenshots (if applicable)

N/A — no UI surface.

## Additional Notes

"I have made corresponding changes to the documentation" is checked in
the sense
that the `--port` help text now describes the new resolution order; no
prose
docs referenced the baked CMD port. `README.md` and the wiki reference
`headroom proxy --port 8787` as a host command, which is unaffected.

`docker/differential-network-capture/docker-compose.yml` passes `--port
8787`
explicitly in its own `command:`, which still overrides everything and
keeps
working.

Adjacent, deliberately not in this PR: `docker-compose.yml`'s
healthcheck is
hardcoded to 8787 the same way. It is inert today because that file
never sets
`HEADROOM_PORT`, and making it configurable also means parameterising
the
published `ports:` mapping — a separate change. Happy to open a
follow-up if
you want it.

## Runtime Rollout Safety

- Rollout-managed feature(s): Docker runtime port selection and
`headroom doctor` probe-port resolution.
- Minimum rollout channel: Standard Docker image release and persistent
Docker deployments.
- Stable/default behavior changed: No. Unset `HEADROOM_PORT` continues
to use port 8787.
- Kill switch / disable path: No new feature flag; revert the image or
this PR.
- Unsafe override required: No.
- Qualification impact: None. No data or schema migration; existing
port-8787 deployments remain compatible.
- Rollback path: Redeploy the previous image or revert this PR; no
migration cleanup is required.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2436"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
natori
2026-10-05 08:36:54 -05:00
committed by GitHub
co-authored by JD Davis
parent ef7605fdbb
commit 1b7977cecd
6 changed files with 218 additions and 10 deletions
+13 -4
View File
@@ -198,11 +198,15 @@ VOLUME ${RUNTIME_HOME}/.headroom
EXPOSE 8787
# Shell form so ${HEADROOM_PORT} expands at probe time: `headroom deploy
# --port N` sets it in the container env, and an exec-form CMD would keep
# probing 8787 and report a working deployment as unhealthy (issue #2432).
HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
CMD ["curl", "--fail", "--silent", "http://127.0.0.1:8787/readyz"]
CMD curl --fail --silent "http://127.0.0.1:${HEADROOM_PORT:-8787}/readyz"
ENTRYPOINT ["headroom", "proxy"]
CMD ["--port", "8787"]
# Keep host and port defaults in the CLI so HEADROOM_HOST and HEADROOM_PORT
# remain authoritative for the published image.
FROM ${DISTROLESS_IMAGE} AS runtime-slim
@@ -223,11 +227,16 @@ ENV HEADROOM_HOST=127.0.0.1 \
EXPOSE 8787
# This stage is distroless, so there is no shell to expand ${HEADROOM_PORT}.
# Resolve the port inside Python instead, keeping exec form (issue #2432).
# `or` rather than a get() default so an empty HEADROOM_PORT falls back the
# same way `${HEADROOM_PORT:-8787}` does in the shell-form stage above.
HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
CMD ["python3", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8787/readyz', timeout=5)"]
CMD ["python3", "-c", "import os, urllib.request; urllib.request.urlopen('http://127.0.0.1:' + (os.environ.get('HEADROOM_PORT') or '8787') + '/readyz', timeout=5)"]
ENTRYPOINT ["python3", "-m", "headroom.cli", "proxy"]
CMD ["--port", "8787"]
# Keep host and port defaults in the CLI so HEADROOM_HOST and HEADROOM_PORT
# remain authoritative for the published image.
# Default published image remains python-slim runtime
FROM runtime-slim-base AS runtime
+52 -4
View File
@@ -25,6 +25,7 @@ import click
from headroom._version import format_version_label, normalize_release_version
from headroom.install.health import probe_json
from headroom.install.models import DeploymentManifest
from headroom.install.paths import (
claude_settings_path,
codex_config_path,
@@ -53,6 +54,8 @@ WARN = "warn"
FAIL = "fail"
SKIP = "skip"
DEFAULT_PROXY_PORT = 8787
_LOOPBACK_URL_RE = re.compile(r"https?://(?:127\.0\.0\.1|localhost):(\d+)")
_CODEX_BASE_URL_RE = re.compile(r'(?m)^[ \t]*base_url\s*=\s*"([^"\r\n]+)"')
_CODEX_MODEL_PROVIDER_RE = re.compile(r'(?m)^[ \t]*model_provider\s*=\s*"([^"\r\n]+)"')
@@ -1099,14 +1102,53 @@ def _render(checks: list[CheckResult], port: int, installed: str) -> None:
console.print("\n[green bold]all checks passed[/green bold]")
def resolve_probe_port(
requested: int | None,
manifests: Sequence[DeploymentManifest],
) -> int:
"""Resolve the port `doctor` should probe.
Precedence: an explicit ``--port`` (or ``HEADROOM_PORT``) wins, then the
port a managed deployment recorded, then the built-in default.
The manifest tier matters because ``headroom deploy --port N`` writes N
into the deployment's own environment, not into the shell `doctor` later
runs in. Without it a healthy non-default deployment is reported as
unreachable, and the remediation hints point at the wrong port.
A deployment is only consulted when it is unambiguous: the ``default``
profile, or a lone profile under another name. With several named profiles
and no explicit request there is no non-arbitrary choice, so the built-in
default is used and the per-deployment section reports each one instead.
Args:
requested: Port supplied on the command line or via the environment.
manifests: Deployment manifests to consider.
Returns:
The port to probe.
"""
if requested is not None:
return requested
default_profile = next((m for m in manifests if m.profile == "default"), None)
if default_profile is not None:
return default_profile.port
if len(manifests) == 1:
return manifests[0].port
return DEFAULT_PROXY_PORT
@main.command()
@click.option(
"--port",
"-p",
default=8787,
default=None,
type=click.IntRange(1, 65535),
envvar="HEADROOM_PORT",
help="Proxy port to check (default: 8787, env: HEADROOM_PORT)",
help=(
"Proxy port to check (env: HEADROOM_PORT). Defaults to the port of a "
f"single managed deployment, else {DEFAULT_PROXY_PORT}."
),
)
@click.option("--json", "emit_json", is_flag=True, help="Emit JSON instead of formatted output.")
@click.option(
@@ -1125,7 +1167,12 @@ def _render(checks: list[CheckResult], port: int, installed: str) -> None:
metavar="URL",
help="Extra upstream URL to test with --network (repeatable), e.g. a custom gateway.",
)
def doctor(port: int, emit_json: bool, network: bool, network_urls: tuple[str, ...]) -> None:
def doctor(
port: int | None,
emit_json: bool,
network: bool,
network_urls: tuple[str, ...],
) -> None:
"""Check that the Headroom proxy and client routing are working.
\b
@@ -1134,13 +1181,14 @@ def doctor(port: int, emit_json: bool, network: bool, network_urls: tuple[str, .
1 warnings only (working, but not optimally wired)
2 at least one failure (proxy down / deployment down)
"""
manifests = list_manifests()
port = resolve_probe_port(port, manifests)
base_url = f"http://127.0.0.1:{port}"
livez = probe_json(f"{base_url}/livez")
readyz = probe_json(f"{base_url}/readyz", timeout=3.0) if livez else None
health = probe_json(f"{base_url}/health", timeout=5.0) if livez else None
stats = probe_json(f"{base_url}/stats", timeout=5.0) if livez else None
installed = get_version()
manifests = list_manifests()
live_elsewhere = None
if livez is None:
# Cheap, bounded: only ports Headroom recorded (HEADROOM_PORT, 8787,
+1 -1
View File
@@ -565,7 +565,7 @@ class TestDoctorAdditions:
monkeypatch.setattr(doctor_mod, "check_deployments", lambda manifests: None)
monkeypatch.setattr(doctor_mod, "claude_settings_path", lambda: home / "settings.json")
monkeypatch.setattr(doctor_mod, "savings_path", lambda: home / "savings.json")
result = CliRunner().invoke(main, ["doctor", "--json"])
result = CliRunner().invoke(main, ["doctor", "--json", "--port", "8787"])
payload = json.loads(result.output)
assert payload["exit_code"] == 2 # still a failure at the probed port
proxy = next(c for c in payload["checks"] if c["name"] == "proxy")
+48
View File
@@ -10,6 +10,7 @@ from click.testing import CliRunner
import headroom.cli.doctor as doctor_mod
from headroom.cli.doctor import (
DEFAULT_PROXY_PORT,
FAIL,
PASS,
SKIP,
@@ -25,6 +26,7 @@ from headroom.cli.doctor import (
check_savings,
check_shell_env,
check_version_drift,
resolve_probe_port,
)
from headroom.cli.main import main
from headroom.providers.claude.runtime import remote_control_gate_message
@@ -804,6 +806,7 @@ class TestBudget:
class _FakeManifest:
profile: str
health_url: str
port: int = 8787
class TestDeployments:
@@ -822,6 +825,33 @@ class TestDeployments:
assert "prod" in result.summary
class TestResolveProbePort:
def test_explicit_port_wins_over_deployment(self):
manifests = [_FakeManifest("default", "", port=8789)]
assert resolve_probe_port(9000, manifests) == 9000
def test_default_profile_deployment_port_is_preferred(self):
manifests = [
_FakeManifest("prod", "", port=9999),
_FakeManifest("default", "", port=8789),
]
assert resolve_probe_port(None, manifests) == 8789
def test_single_non_default_deployment_is_used(self):
assert resolve_probe_port(None, [_FakeManifest("prod", "", port=9999)]) == 9999
def test_builtin_default_without_deployments(self):
assert resolve_probe_port(None, []) == DEFAULT_PROXY_PORT
def test_ambiguous_named_deployments_fall_back_to_the_default(self):
"""Several named profiles and no request: any pick would be arbitrary."""
manifests = [
_FakeManifest("prod", "", port=9998),
_FakeManifest("staging", "", port=9999),
]
assert resolve_probe_port(None, manifests) == DEFAULT_PROXY_PORT
class TestDoctorCommand:
@pytest.fixture
def runner(self):
@@ -938,6 +968,24 @@ class TestDoctorCommand:
runner.invoke(main, ["doctor"], env={"HEADROOM_PORT": "9999"})
assert "http://127.0.0.1:9999/livez" in seen
def test_deployment_port_probed_when_shell_has_no_override(self, runner, isolated, monkeypatch):
"""`headroom deploy --port N` only sets HEADROOM_PORT inside the deployment."""
seen: list[str] = []
def recording_probe(url, timeout=2.0):
seen.append(url)
return None
monkeypatch.setattr(
doctor_mod,
"list_manifests",
lambda: [_FakeManifest("default", "http://127.0.0.1:8789/readyz", port=8789)],
)
monkeypatch.setattr(doctor_mod, "probe_json", recording_probe)
runner.invoke(main, ["doctor"])
assert "http://127.0.0.1:8789/livez" in seen
assert not any("127.0.0.1:8787" in url for url in seen)
class TestCostTrackerBudgetKeys:
def test_stats_exposes_budget_config(self):
+2 -1
View File
@@ -138,4 +138,5 @@ def test_published_dockerfile_stages_default_to_loopback() -> None:
dockerfile = (ROOT / "Dockerfile").read_text(encoding="utf-8")
assert dockerfile.count("ENV HEADROOM_HOST=127.0.0.1") == 2
assert dockerfile.count('CMD ["--port", "8787"]') == 2
assert 'CMD ["--port", "8787"]' not in dockerfile
assert 'CMD ["--host", "0.0.0.0"]' not in dockerfile
+102
View File
@@ -0,0 +1,102 @@
"""Regression checks for port-aware container startup and healthchecks (#2432).
`headroom proxy --port` is a click option with `envvar="HEADROOM_PORT"`, and an
explicit CLI argument beats the envvar. So a baked `--port` in CMD and a
hardcoded probe port in HEALTHCHECK are two halves of the same defect: the
first makes HEADROOM_PORT unable to move the listener, the second makes the
probe unable to follow it.
"""
from __future__ import annotations
import re
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
STAGE_RE = re.compile(r"^FROM\s+\S+\s+AS\s+(\S+)\s*$", re.IGNORECASE)
def _stage_body(dockerfile: str, stage: str) -> str:
"""Return the lines belonging to a single named build stage."""
body: list[str] = []
collecting = False
for line in dockerfile.splitlines():
match = STAGE_RE.match(line.strip())
if match:
collecting = match.group(1) == stage
continue
if collecting:
body.append(line)
assert body, f"stage {stage!r} not found in Dockerfile"
return "\n".join(body)
def _healthcheck_directives(dockerfile: str) -> list[str]:
"""Collect every HEALTHCHECK directive, joining backslash continuations."""
directives: list[str] = []
lines = dockerfile.splitlines()
for index, line in enumerate(lines):
if not line.lstrip().startswith("HEALTHCHECK"):
continue
parts = [line.rstrip("\\").strip()]
cursor = index
while lines[cursor].rstrip().endswith("\\") and cursor + 1 < len(lines):
cursor += 1
parts.append(lines[cursor].rstrip("\\").strip())
directives.append(" ".join(parts))
return directives
def test_dockerfile_defines_a_healthcheck_per_runtime_stage() -> None:
directives = _healthcheck_directives((ROOT / "Dockerfile").read_text(encoding="utf-8"))
assert len(directives) == 2
def test_no_healthcheck_hardcodes_the_default_port() -> None:
"""A hardcoded probe port reports a working non-8787 deployment as unhealthy."""
directives = _healthcheck_directives((ROOT / "Dockerfile").read_text(encoding="utf-8"))
offenders = [d for d in directives if "127.0.0.1:8787" in d]
assert offenders == []
def test_every_healthcheck_resolves_the_port_from_the_environment() -> None:
directives = _healthcheck_directives((ROOT / "Dockerfile").read_text(encoding="utf-8"))
assert directives, "expected at least one HEALTHCHECK directive"
for directive in directives:
assert "HEADROOM_PORT" in directive
def test_distroless_stage_healthcheck_stays_shell_free() -> None:
"""The distroless stage has no shell, so ${VAR} expansion is unavailable."""
dockerfile = (ROOT / "Dockerfile").read_text(encoding="utf-8")
directives = _healthcheck_directives(_stage_body(dockerfile, "runtime-slim"))
assert len(directives) == 1
assert "${HEADROOM_PORT" not in directives[0]
# `or` and not a get() default, so an empty value falls back like `:-` does.
assert "(os.environ.get('HEADROOM_PORT') or '8787')" in directives[0]
def test_debian_stage_healthcheck_expands_the_port_variable() -> None:
dockerfile = (ROOT / "Dockerfile").read_text(encoding="utf-8")
directives = _healthcheck_directives(_stage_body(dockerfile, "runtime-slim-base"))
assert len(directives) == 1
assert "${HEADROOM_PORT:-8787}" in directives[0]
def test_no_stage_bakes_a_port_into_cmd() -> None:
"""A baked `--port` outranks HEADROOM_PORT and pins the listener to 8787."""
dockerfile = (ROOT / "Dockerfile").read_text(encoding="utf-8")
cmds = [line for line in dockerfile.splitlines() if line.startswith("CMD ")]
assert not any("--port" in cmd for cmd in cmds)