Files
headroom/tests/test_cli/test_proxy_port_preflight.py
f3d240bcae fix(cli): say why the proxy failed to start (#4120)
## Description

Fixes two proxy startup failures that hid their cause from the user.

**1. `headroom proxy` on a port that is already in use.** It printed the
full banner, ending in "Press Ctrl+C to stop.", and then about 10 INFO
lines. The one line that mattered, uvicorn's `[Errno 48] ... address
already in use`, came next. After it came `event=proxy_shutdown
reason=signal` and a 10-line zero-valued session summary.

The command now checks the port before printing the banner and exits
with one clear line:

```
Error: Port 54351 on 127.0.0.1 is already in use by another process (Address already in use). Start this proxy on another port with --port N (env: HEADROOM_PORT).
```

When a Headroom proxy is the process holding the port, the message says
so instead. The check asks the address that is actually busy (IPv6
included), so a Headroom proxy on 127.0.0.1:P is never reported as the
owner of [::1]:P. When that address cannot be probed (a scoped IPv6
address, an unresolvable name), the message is the generic `already in
use (<reason>)`.

**2. `headroom wrap <tool>` when the proxy it starts dies during
startup.** The error quoted the last 500 characters of the proxy stdio
log. That was the wrong part of the log, for two reasons:
- The proxy logs its shutdown summary after the error, so the tail was
the summary box.
- The log is appended to across runs, so the tail could come from an
older run.

The error also never said where the log was. It now reads only this
run's output, at most its first 64 KiB and, when the head has no error,
its last 64 KiB (so a runaway log is never read whole, and an error
after a long burst of startup output is still found), quotes from the
first `ERROR` / `CRITICAL` / `Traceback` line or line-leading Click
`Error: ` (otherwise the end of the first non-blank window, so neither a
blank head nor a blank tail hides useful startup text), and adds `Full
log: <path>`. The timeout error now names the log too.

Refs #4136

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cli/proxy.py`: new `_tcp_port_in_use` / `_refuse_busy_port`.
They run after the bind-policy check and before the banner, and are
skipped for `--uds`. The probe mirrors uvicorn's own bind, so it never
refuses a bind that uvicorn would accept:
- It sets `SO_REUSEADDR`, as both uvicorn paths do. A port in TIME_WAIT
after a restart is therefore not reported as busy.
- With one worker, it binds every address the host resolves to, with
`IPV6_V6ONLY`, the same way asyncio `create_server` does.
- With `--workers > 1`, it binds one AF_INET or AF_INET6 socket, as
uvicorn's `Config.bind_socket` does.
- On Windows, and for any error other than EADDRINUSE, it does nothing
and leaves the case to uvicorn, exactly as today.
- It returns the busy address. `_probe_host_for` maps it to the address
to ask: wildcards go to loopback of the same family (`0.0.0.0` to
`127.0.0.1`, `::` to `::1`), a hostname is resolved for IPv4, and a
scoped IPv6 address returns None (generic message).
- `headroom/cli/port_discovery.py`: `probe_headroom_proxy` takes an
optional keyword `host` (default `127.0.0.1`, so existing callers are
unchanged) and brackets IPv6 in the URL.
- `headroom/cli/wrap.py`:
  - `_start_proxy` records the stdio log size before spawning the proxy.
- New `_proxy_startup_failure_excerpt` picks the error line from this
run's output, reading at most the first and last 64 KiB written after
the recorded offset.
  - Both startup errors now end with `Full log: <path>`.
- `tests/conftest.py` + `pyproject.toml`: an autouse guard, and its
`proxy_port_preflight` marker, turn the new check off unless a test opts
in. It follows the existing `proxy_dependency_gate` pattern.
- Without it, the 16 test files that invoke `headroom proxy` with a
stubbed `run_server` on the default port 8787 would fail whenever a
developer has their own proxy running.
- Tests:
- New `tests/test_cli/test_proxy_port_preflight.py`, which uses real
sockets:
    - busy port with workers 1 and 2;
    - port held by a Headroom proxy;
    - free port still starts;
- `test_ipv6_conflict_is_identified_on_the_requested_host`: a real
fake-Headroom `/livez` on 127.0.0.1:P and a plain listener on [::1]:P;
`--host ::1` must say "another process";
    - `test_unprobeable_host_gets_the_generic_message`.
  - New `test_startup_excerpt_reads_at_most_64_kib_of_this_run`.
- New
`test_start_proxy_exit_quotes_this_runs_error_not_summary_or_old_runs`.
  - The timeout test now also asserts the log path.
- The existing stdio-tail test now writes its log output from the fake
`Popen`, as the real child process does.
- The new tests fail on the old code (3 preflight tests; the excerpt and
timeout tests) and pass now. The three tests added for the review (IPv6
owner, unprobeable host, 64 KiB bound) fail on the PR's previous commit
03288c05. The late-error test (an error after 256 KiB of output is
quoted, at most 128 KiB read) fails on c9868420. The head-fallback test
(blank tail) fails on 062cf011, and the two Click `Error:` tests fail on
409fee89 (one of them also on 062cf011). The blank-head/useful-tail test
fails on 92b1bd08. The excerpt fixtures write bytes, so their byte
offsets also hold on Windows.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q -p no:cacheprovider tests/test_cli/test_proxy_port_preflight.py tests/test_cli_proxy_env.py tests/test_proxy_bind_policy.py tests/test_proxy_uds.py tests/test_cli/test_wrap_helpers.py tests/test_cli/test_wrap_proxy_detach.py tests/test_cli/test_port_alignment.py tests/test_malloc_tuning.py tests/test_telemetry_warning.py tests/test_cli_extension_seam.py tests/test_agent_savings.py tests/test_no_ccr_disables_response_handling.py tests/test_stateless_no_disk.py tests/test_cli_proxy_embedding_server.py tests/test_proxy_per_provider_kompress.py tests/test_savings_ledger.py
11 failed, 427 passed, 1 warning in 18.04s
# The 11 failures are all in tests/test_proxy_uds.py ("Socket path is 134 bytes, over this platform's 104-byte sun_path limit"). They are pre-existing on this machine: clean origin/main gives "11 failed, 19 passed" for that file.

# Regression tests against the old code:
3 failed, 1 passed   (test_proxy_port_preflight.py with the proxy.py change reverted)
2 failed, 8 passed   (test_cli_proxy_env.py -k "TestCLIWrapProxyTimeout or start_proxy" with the wrap.py change reverted)

# After the review fixes (c9868420):
$ pytest -q -p no:cacheprovider tests/test_cli/test_proxy_port_preflight.py tests/test_cli_proxy_env.py tests/test_cli/test_wrap_helpers.py tests/test_cli/test_wrap_proxy_detach.py tests/test_cli/test_port_alignment.py tests/test_proxy_bind_policy.py
247 passed
$ pytest -q tests/test_cli -k "port or discovery or livez"
126 passed, 989 deselected
# The 3 new review tests against 03288c05: 3 failed

$ uvx ruff@0.16.8 check headroom/cli/proxy.py headroom/cli/wrap.py tests/conftest.py tests/test_cli_proxy_env.py tests/test_cli/test_proxy_port_preflight.py
All checks passed!
$ mypy headroom/cli/proxy.py headroom/cli/wrap.py headroom/cli/port_discovery.py --ignore-missing-imports
Success: no issues found in 3 source files
```

## Real Behavior Proof

- Environment: macOS 25.6 with Python 3.12 (repo `.venv`); Linux checked
in Docker `python:3.12-slim-trixie` with uvicorn 0.40.0.
- Exact command / steps:
1. Bound a socket to 127.0.0.1:N, then ran `python -m headroom.cli proxy
--port N`.
2. Ran a script that compares `_tcp_port_in_use` with uvicorn's real
bind (asyncio `create_server` for one worker,
`uvicorn.Config(...).bind_socket()` for workers=2). It covered free
ports; listeners on 127.0.0.1, 0.0.0.0 and ::1; hosts 127.0.0.1,
0.0.0.0, localhost and ::; a port in TIME_WAIT; and a non-local host.
- Observed result:
1. rc=1, empty stdout, and stderr `Error: Port 54351 on 127.0.0.1 is
already in use by another process (Address already in use). Start this
proxy on another port with --port N (env: HEADROOM_PORT).` The banner
did not print and uvicorn was never reached.
2. On both macOS and Linux, the probe agreed with uvicorn in every case,
including the ones where the two platforms differ (a 127.0.0.1 listener
with host 0.0.0.0 is busy on Linux and free on macOS). TIME_WAIT is not
reported as busy. For a non-local host, the probe returns "not busy" and
leaves the EADDRNOTAVAIL error to uvicorn, unchanged.
- Direct runtime command (this PR's path). A script holds 127.0.0.1:P
with a fake Headroom `/livez` and [::1]:P with a plain listener, then
runs the real CLI, `python -m headroom.cli proxy --host <h> --port P`,
with temp `HOME`, `HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`:

  ```text
  $ headroom proxy --host ::1 --port 64747
  exit=1 stdout=''
stderr: Error: Port 64747 on ::1 is already in use by another process
(Address already in use). Start this proxy on another port with --port N
(env: HEADROOM_PORT).
  $ headroom proxy --host 127.0.0.1 --port 64747
  exit=1 stdout=''
stderr: Error: Port 64747 on 127.0.0.1 is already in use by a running
Headroom proxy (reuse it, or stop it first). Start this proxy on another
port with --port N (env: HEADROOM_PORT).
  # Same script on the previous commit 03288c05, --host ::1:
stderr: Error: Port 64976 on ::1 is already in use by a running Headroom
proxy (reuse it, or stop it first). ...
  ```

- Shared smoke harness (`smoke.sh <worktree>`: a real proxy from this
checkout against a local fake upstream, offline, temp HOME):

  ```text
  branch: cleanup/cli-integrations-proxy-start-errors @ 92b1bd08
  health: 200
  POST /v1/messages        -> HTTP200 smoke-ok
  POST /v1/chat/completions -> HTTP200 smoke-ok
  stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
  proxy.log WARNING/ERROR lines (<tmp>/ws/logs/proxy-49585.log):
2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active —
every connection Headroom initiates is refused (telemetry, update check,
license reporter, model/tokenizer/binary/dataset downloads, remote
Kompress, OTLP/Langfuse export, Copilot auth, subscription polling,
OpenAI embedders, Headroom Cloud compression). Still allowed on purpose:
forwarding your requests to the upstream you configured,
operator-configured local endpoints, loopback health probes.
2 headroom.transforms.kompress_compressor - WARNING - Kompress:
'onnx/kompress-int8-wo.onnx' for chopratejas/kompress-v2-base is not
cached and HEADROOM_OFFLINE forbids fetching it (HEADROOM_OFFLINE is
set: refusing outbound network access for HuggingFace download of
chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to
huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that
needs this connection.). Compression falls back to the non-ML path.
4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact
prefetch found no usable file for chopratejas/kompress-v2-base
4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact
prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is
set): HEADROOM_OFFLINE is set: refusing outbound network access for
HuggingFace download of
chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to
huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that
needs this connection.
  (smoke dir: <tmp>)
  ```

- Not tested: Windows (the check is skipped there by design), and a live
`wrap` session whose proxy crashes. The excerpt logic is covered by the
unit test.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A, not rollout-managed.
- Stable/default behavior changed: Yes, only on failure paths. A busy
port now fails before the banner (exit 1, same as before) with a clear
message; wrap's startup errors quote a different part of the log and add
the path.
- Kill switch / disable path: None needed. The probe only refuses on
EADDRINUSE, in the same cases where uvicorn itself would fail to bind.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.
- Restart race: the check runs before the heavy app import, a few
seconds earlier than uvicorn's bind used to. A restart whose old process
still holds the listening socket can now fail fast where it used to win
the race narrowly. The restart paths wait for the old process to exit
first (`stop_runtime` polls `pid_alive`; systemd and launchd wait for
exit), and the exit code is 1 either way.
- No docs change: the new behaviour is an error message.
- The audit had also proposed changing wrap's startup `except
RuntimeError` to raise `ClickException` (stderr). I dropped that: every
caller of `_ensure_proxy` wraps it in `except Exception: echo " Error:
..."; SystemExit(1)`, so the output would not change.
- The audit had proposed reusing wrap's `_port_bind_error` for the
probe. I did not, because that helper binds 127.0.0.1 only and without
`SO_REUSEADDR`, so it does not match uvicorn.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4120"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-12 18:59:35 -07:00

146 lines
5.0 KiB
Python

"""`headroom proxy` refuses a busy port before printing its banner.
Without the check the banner ends "Press Ctrl+C to stop", then uvicorn's bind
error is buried among startup and shutdown log lines.
"""
from __future__ import annotations
import json
import socket
import sys
import threading
from collections.abc import Iterator
from contextlib import contextmanager
from http.server import BaseHTTPRequestHandler, HTTPServer
from unittest.mock import patch
import pytest
from click.testing import CliRunner
from headroom.cli.main import main
pytestmark = [
pytest.mark.proxy_port_preflight,
pytest.mark.skipif(sys.platform == "win32", reason="the check is POSIX-only"),
]
@contextmanager
def _listening(host: str = "127.0.0.1") -> Iterator[int]:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind((host, 0))
sock.listen(1)
try:
yield sock.getsockname()[1]
finally:
sock.close()
def _free_port() -> int:
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.bind(("127.0.0.1", 0))
return sock.getsockname()[1]
def _invoke_proxy(*args: str) -> tuple[object, list[object]]:
calls: list[object] = []
with (
patch("headroom.proxy.server.run_server", lambda config, **kw: calls.append(config)),
patch("headroom.cli.port_discovery.probe_headroom_proxy", return_value=False),
):
result = CliRunner().invoke(main, ["proxy", *args])
return result, calls
@pytest.mark.parametrize("workers", ["1", "2"])
def test_busy_port_fails_before_banner_with_actionable_message(workers: str) -> None:
with _listening() as port:
result, calls = _invoke_proxy("--port", str(port), "--workers", workers)
assert result.exit_code == 1, result.output
assert f"Port {port} on 127.0.0.1 is already in use by another process" in result.output
assert "--port N" in result.output
assert "HEADROOM PROXY" not in result.output
assert calls == []
def test_busy_port_held_by_headroom_says_so() -> None:
with _listening() as port:
with (
patch("headroom.proxy.server.run_server", lambda config, **kw: None),
patch("headroom.cli.port_discovery.probe_headroom_proxy", return_value=True),
):
result = CliRunner().invoke(main, ["proxy", "--port", str(port)])
assert result.exit_code == 1, result.output
assert "already in use by a running Headroom proxy" in result.output
def test_free_port_starts_normally() -> None:
port = _free_port()
result, calls = _invoke_proxy("--port", str(port))
assert result.exit_code == 0, result.output
assert "HEADROOM PROXY" in result.output
assert len(calls) == 1
def _ipv6_loopback_available() -> bool:
try:
with socket.socket(socket.AF_INET6, socket.SOCK_STREAM) as sock:
sock.bind(("::1", 0))
except OSError:
return False
return True
class _HeadroomLivez(BaseHTTPRequestHandler):
def do_GET(self) -> None: # noqa: N802 - http.server API
body = json.dumps({"service": "headroom-proxy", "alive": True}).encode()
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *args: object) -> None:
pass
@pytest.mark.skipif(not _ipv6_loopback_available(), reason="needs IPv6 loopback")
def test_ipv6_conflict_is_identified_on_the_requested_host() -> None:
"""A Headroom proxy on 127.0.0.1:P must not be reported as the owner of [::1]:P."""
headroom_v4 = HTTPServer(("127.0.0.1", 0), _HeadroomLivez)
port = headroom_v4.server_address[1]
threading.Thread(target=headroom_v4.serve_forever, daemon=True).start()
other_v6 = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)
try:
other_v6.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 1)
other_v6.bind(("::1", port))
other_v6.listen(1)
with patch("headroom.proxy.server.run_server", lambda config, **kw: None):
v6 = CliRunner().invoke(main, ["proxy", "--host", "::1", "--port", str(port)])
finally:
other_v6.close()
headroom_v4.shutdown()
headroom_v4.server_close()
assert v6.exit_code == 1, v6.output
assert f"Port {port} on ::1 is already in use by another process" in v6.output
assert "Headroom proxy" not in v6.output
def test_unprobeable_host_gets_the_generic_message() -> None:
with _listening() as port:
with (
patch("headroom.proxy.server.run_server", lambda config, **kw: None),
patch("headroom.cli.proxy._probe_host_for", return_value=None),
patch("headroom.cli.port_discovery.probe_headroom_proxy", return_value=True),
):
result = CliRunner().invoke(main, ["proxy", "--port", str(port)])
assert result.exit_code == 1, result.output
assert f"Port {port} on 127.0.0.1 is already in use (" in result.output
assert "Headroom proxy" not in result.output