fix(cli): say why the proxy failed to start (#4120)

## Description

Fixes two proxy startup failures that hid their cause from the user.

**1. `headroom proxy` on a port that is already in use.** It printed the
full banner, ending in "Press Ctrl+C to stop.", and then about 10 INFO
lines. The one line that mattered, uvicorn's `[Errno 48] ... address
already in use`, came next. After it came `event=proxy_shutdown
reason=signal` and a 10-line zero-valued session summary.

The command now checks the port before printing the banner and exits
with one clear line:

```
Error: Port 54351 on 127.0.0.1 is already in use by another process (Address already in use). Start this proxy on another port with --port N (env: HEADROOM_PORT).
```

When a Headroom proxy is the process holding the port, the message says
so instead. The check asks the address that is actually busy (IPv6
included), so a Headroom proxy on 127.0.0.1:P is never reported as the
owner of [::1]:P. When that address cannot be probed (a scoped IPv6
address, an unresolvable name), the message is the generic `already in
use (<reason>)`.

**2. `headroom wrap <tool>` when the proxy it starts dies during
startup.** The error quoted the last 500 characters of the proxy stdio
log. That was the wrong part of the log, for two reasons:
- The proxy logs its shutdown summary after the error, so the tail was
the summary box.
- The log is appended to across runs, so the tail could come from an
older run.

The error also never said where the log was. It now reads only this
run's output, at most its first 64 KiB and, when the head has no error,
its last 64 KiB (so a runaway log is never read whole, and an error
after a long burst of startup output is still found), quotes from the
first `ERROR` / `CRITICAL` / `Traceback` line or line-leading Click
`Error: ` (otherwise the end of the first non-blank window, so neither a
blank head nor a blank tail hides useful startup text), and adds `Full
log: <path>`. The timeout error now names the log too.

Refs #4136

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cli/proxy.py`: new `_tcp_port_in_use` / `_refuse_busy_port`.
They run after the bind-policy check and before the banner, and are
skipped for `--uds`. The probe mirrors uvicorn's own bind, so it never
refuses a bind that uvicorn would accept:
- It sets `SO_REUSEADDR`, as both uvicorn paths do. A port in TIME_WAIT
after a restart is therefore not reported as busy.
- With one worker, it binds every address the host resolves to, with
`IPV6_V6ONLY`, the same way asyncio `create_server` does.
- With `--workers > 1`, it binds one AF_INET or AF_INET6 socket, as
uvicorn's `Config.bind_socket` does.
- On Windows, and for any error other than EADDRINUSE, it does nothing
and leaves the case to uvicorn, exactly as today.
- It returns the busy address. `_probe_host_for` maps it to the address
to ask: wildcards go to loopback of the same family (`0.0.0.0` to
`127.0.0.1`, `::` to `::1`), a hostname is resolved for IPv4, and a
scoped IPv6 address returns None (generic message).
- `headroom/cli/port_discovery.py`: `probe_headroom_proxy` takes an
optional keyword `host` (default `127.0.0.1`, so existing callers are
unchanged) and brackets IPv6 in the URL.
- `headroom/cli/wrap.py`:
  - `_start_proxy` records the stdio log size before spawning the proxy.
- New `_proxy_startup_failure_excerpt` picks the error line from this
run's output, reading at most the first and last 64 KiB written after
the recorded offset.
  - Both startup errors now end with `Full log: <path>`.
- `tests/conftest.py` + `pyproject.toml`: an autouse guard, and its
`proxy_port_preflight` marker, turn the new check off unless a test opts
in. It follows the existing `proxy_dependency_gate` pattern.
- Without it, the 16 test files that invoke `headroom proxy` with a
stubbed `run_server` on the default port 8787 would fail whenever a
developer has their own proxy running.
- Tests:
- New `tests/test_cli/test_proxy_port_preflight.py`, which uses real
sockets:
    - busy port with workers 1 and 2;
    - port held by a Headroom proxy;
    - free port still starts;
- `test_ipv6_conflict_is_identified_on_the_requested_host`: a real
fake-Headroom `/livez` on 127.0.0.1:P and a plain listener on [::1]:P;
`--host ::1` must say "another process";
    - `test_unprobeable_host_gets_the_generic_message`.
  - New `test_startup_excerpt_reads_at_most_64_kib_of_this_run`.
- New
`test_start_proxy_exit_quotes_this_runs_error_not_summary_or_old_runs`.
  - The timeout test now also asserts the log path.
- The existing stdio-tail test now writes its log output from the fake
`Popen`, as the real child process does.
- The new tests fail on the old code (3 preflight tests; the excerpt and
timeout tests) and pass now. The three tests added for the review (IPv6
owner, unprobeable host, 64 KiB bound) fail on the PR's previous commit
03288c05. The late-error test (an error after 256 KiB of output is
quoted, at most 128 KiB read) fails on c9868420. The head-fallback test
(blank tail) fails on 062cf011, and the two Click `Error:` tests fail on
409fee89 (one of them also on 062cf011). The blank-head/useful-tail test
fails on 92b1bd08. The excerpt fixtures write bytes, so their byte
offsets also hold on Windows.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q -p no:cacheprovider tests/test_cli/test_proxy_port_preflight.py tests/test_cli_proxy_env.py tests/test_proxy_bind_policy.py tests/test_proxy_uds.py tests/test_cli/test_wrap_helpers.py tests/test_cli/test_wrap_proxy_detach.py tests/test_cli/test_port_alignment.py tests/test_malloc_tuning.py tests/test_telemetry_warning.py tests/test_cli_extension_seam.py tests/test_agent_savings.py tests/test_no_ccr_disables_response_handling.py tests/test_stateless_no_disk.py tests/test_cli_proxy_embedding_server.py tests/test_proxy_per_provider_kompress.py tests/test_savings_ledger.py
11 failed, 427 passed, 1 warning in 18.04s
# The 11 failures are all in tests/test_proxy_uds.py ("Socket path is 134 bytes, over this platform's 104-byte sun_path limit"). They are pre-existing on this machine: clean origin/main gives "11 failed, 19 passed" for that file.

# Regression tests against the old code:
3 failed, 1 passed   (test_proxy_port_preflight.py with the proxy.py change reverted)
2 failed, 8 passed   (test_cli_proxy_env.py -k "TestCLIWrapProxyTimeout or start_proxy" with the wrap.py change reverted)

# After the review fixes (c9868420):
$ pytest -q -p no:cacheprovider tests/test_cli/test_proxy_port_preflight.py tests/test_cli_proxy_env.py tests/test_cli/test_wrap_helpers.py tests/test_cli/test_wrap_proxy_detach.py tests/test_cli/test_port_alignment.py tests/test_proxy_bind_policy.py
247 passed
$ pytest -q tests/test_cli -k "port or discovery or livez"
126 passed, 989 deselected
# The 3 new review tests against 03288c05: 3 failed

$ uvx ruff@0.16.8 check headroom/cli/proxy.py headroom/cli/wrap.py tests/conftest.py tests/test_cli_proxy_env.py tests/test_cli/test_proxy_port_preflight.py
All checks passed!
$ mypy headroom/cli/proxy.py headroom/cli/wrap.py headroom/cli/port_discovery.py --ignore-missing-imports
Success: no issues found in 3 source files
```

## Real Behavior Proof

- Environment: macOS 25.6 with Python 3.12 (repo `.venv`); Linux checked
in Docker `python:3.12-slim-trixie` with uvicorn 0.40.0.
- Exact command / steps:
1. Bound a socket to 127.0.0.1:N, then ran `python -m headroom.cli proxy
--port N`.
2. Ran a script that compares `_tcp_port_in_use` with uvicorn's real
bind (asyncio `create_server` for one worker,
`uvicorn.Config(...).bind_socket()` for workers=2). It covered free
ports; listeners on 127.0.0.1, 0.0.0.0 and ::1; hosts 127.0.0.1,
0.0.0.0, localhost and ::; a port in TIME_WAIT; and a non-local host.
- Observed result:
1. rc=1, empty stdout, and stderr `Error: Port 54351 on 127.0.0.1 is
already in use by another process (Address already in use). Start this
proxy on another port with --port N (env: HEADROOM_PORT).` The banner
did not print and uvicorn was never reached.
2. On both macOS and Linux, the probe agreed with uvicorn in every case,
including the ones where the two platforms differ (a 127.0.0.1 listener
with host 0.0.0.0 is busy on Linux and free on macOS). TIME_WAIT is not
reported as busy. For a non-local host, the probe returns "not busy" and
leaves the EADDRNOTAVAIL error to uvicorn, unchanged.
- Direct runtime command (this PR's path). A script holds 127.0.0.1:P
with a fake Headroom `/livez` and [::1]:P with a plain listener, then
runs the real CLI, `python -m headroom.cli proxy --host <h> --port P`,
with temp `HOME`, `HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`:

  ```text
  $ headroom proxy --host ::1 --port 64747
  exit=1 stdout=''
stderr: Error: Port 64747 on ::1 is already in use by another process
(Address already in use). Start this proxy on another port with --port N
(env: HEADROOM_PORT).
  $ headroom proxy --host 127.0.0.1 --port 64747
  exit=1 stdout=''
stderr: Error: Port 64747 on 127.0.0.1 is already in use by a running
Headroom proxy (reuse it, or stop it first). Start this proxy on another
port with --port N (env: HEADROOM_PORT).
  # Same script on the previous commit 03288c05, --host ::1:
stderr: Error: Port 64976 on ::1 is already in use by a running Headroom
proxy (reuse it, or stop it first). ...
  ```

- Shared smoke harness (`smoke.sh <worktree>`: a real proxy from this
checkout against a local fake upstream, offline, temp HOME):

  ```text
  branch: cleanup/cli-integrations-proxy-start-errors @ 92b1bd08
  health: 200
  POST /v1/messages        -> HTTP200 smoke-ok
  POST /v1/chat/completions -> HTTP200 smoke-ok
  stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
  proxy.log WARNING/ERROR lines (<tmp>/ws/logs/proxy-49585.log):
2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active —
every connection Headroom initiates is refused (telemetry, update check,
license reporter, model/tokenizer/binary/dataset downloads, remote
Kompress, OTLP/Langfuse export, Copilot auth, subscription polling,
OpenAI embedders, Headroom Cloud compression). Still allowed on purpose:
forwarding your requests to the upstream you configured,
operator-configured local endpoints, loopback health probes.
2 headroom.transforms.kompress_compressor - WARNING - Kompress:
'onnx/kompress-int8-wo.onnx' for chopratejas/kompress-v2-base is not
cached and HEADROOM_OFFLINE forbids fetching it (HEADROOM_OFFLINE is
set: refusing outbound network access for HuggingFace download of
chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to
huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that
needs this connection.). Compression falls back to the non-ML path.
4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact
prefetch found no usable file for chopratejas/kompress-v2-base
4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact
prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is
set): HEADROOM_OFFLINE is set: refusing outbound network access for
HuggingFace download of
chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to
huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that
needs this connection.
  (smoke dir: <tmp>)
  ```

- Not tested: Windows (the check is skipped there by design), and a live
`wrap` session whose proxy crashes. The excerpt logic is covered by the
unit test.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A, not rollout-managed.
- Stable/default behavior changed: Yes, only on failure paths. A busy
port now fails before the banner (exit 1, same as before) with a clear
message; wrap's startup errors quote a different part of the log and add
the path.
- Kill switch / disable path: None needed. The probe only refuses on
EADDRINUSE, in the same cases where uvicorn itself would fail to bind.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.
- Restart race: the check runs before the heavy app import, a few
seconds earlier than uvicorn's bind used to. A restart whose old process
still holds the listening socket can now fail fast where it used to win
the race narrowly. The restart paths wait for the old process to exit
first (`stop_runtime` polls `pid_alive`; systemd and launchd wait for
exit), and the exit code is 1 either way.
- No docs change: the new behaviour is an error message.
- The audit had also proposed changing wrap's startup `except
RuntimeError` to raise `ClickException` (stderr). I dropped that: every
caller of `_ensure_proxy` wraps it in `except Exception: echo " Error:
..."; SystemExit(1)`, so the output would not change.
- The audit had proposed reusing wrap's `_port_bind_error` for the
probe. I did not, because that helper binds 127.0.0.1 only and without
`SO_REUSEADDR`, so it does not match uvicorn.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4120"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Tejas Chopra
2026-10-12 18:59:35 -07:00
committed by GitHub
co-authored by Claude Opus 5.5 JerrettDavis
parent 6e14ee0bc2
commit f3d240bcae
7 changed files with 470 additions and 13 deletions
+7 -4
View File
@@ -120,16 +120,19 @@ def candidate_ports(
return seen[:_MAX_CANDIDATES]
def probe_headroom_proxy(port: int, timeout: float = _PROBE_TIMEOUT_SECONDS) -> bool:
"""True when a Headroom proxy answers ``/livez`` on loopback ``port``.
def probe_headroom_proxy(
port: int, timeout: float = _PROBE_TIMEOUT_SECONDS, *, host: str = "127.0.0.1"
) -> bool:
"""True when a Headroom proxy answers ``/livez`` on ``host:port`` (loopback by default).
Requires the ``service: headroom-proxy`` identity (or the legacy
``alive``+``version`` shape) so an unrelated local server on the port is
never mistaken for Headroom.
never mistaken for Headroom. ``host`` is a numeric IPv4 or IPv6 address.
"""
from headroom.install.health import probe_json
return is_headroom_livez(probe_json(f"http://127.0.0.1:{port}/livez", timeout=timeout))
netloc = f"[{host}]" if ":" in host else host
return is_headroom_livez(probe_json(f"http://{netloc}:{port}/livez", timeout=timeout))
def is_headroom_livez(payload: Mapping[str, Any] | None) -> bool:
+98
View File
@@ -1,8 +1,11 @@
"""Proxy server CLI commands."""
import errno
import ipaddress
import logging
import os
import platform
import socket
import sys
import warnings
from importlib.util import find_spec
@@ -195,6 +198,99 @@ def _get_env_int(name: str, default: int) -> int:
return default if value is None else value
def _tcp_port_in_use(host: str, port: int, *, workers: int) -> tuple[OSError, Any] | None:
"""Return ``(EADDRINUSE error, busy address)`` uvicorn would hit binding ``host:port``.
Returns None when the bind would succeed.
Mirrors uvicorn's own bind so the probe never refuses a bind uvicorn would
accept: ``SO_REUSEADDR`` (both uvicorn paths set it on POSIX, so a port in
TIME_WAIT after a restart is not "in use"); a single process binds every
address ``host`` resolves to (asyncio ``create_server``), while
``--workers > 1`` binds one AF_INET/AF_INET6 socket in the parent.
Windows ``SO_REUSEADDR`` semantics differ, so Windows is left to uvicorn,
as is any error other than EADDRINUSE (unresolvable host, address not on
this machine); uvicorn reports those as before.
"""
if sys.platform == "win32":
return None
if workers > 1:
family = socket.AF_INET6 if ":" in host else socket.AF_INET
targets: list[tuple[Any, Any]] = [(family, (host, port))]
else:
try:
infos = socket.getaddrinfo(
host or None, port, type=socket.SOCK_STREAM, flags=socket.AI_PASSIVE
)
except (OSError, UnicodeError):
return None
targets = [(info[0], info[4]) for info in infos]
for family, address in targets:
try:
with socket.socket(family, socket.SOCK_STREAM) as probe:
probe.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
if family == socket.AF_INET6 and workers <= 1:
# asyncio binds v4 and v6 separately; match it.
probe.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 1)
probe.bind(address)
except OSError as exc:
if exc.errno == errno.EADDRINUSE:
return exc, address
return None
_WILDCARD_PROBE_HOSTS = {"": "127.0.0.1", "0.0.0.0": "127.0.0.1", "::": "::1"}
def _probe_host_for(address: Any) -> str | None:
"""The numeric address to ask "is this Headroom?", or None when it can't be targeted.
A wildcard bind is reached through loopback of the same family. A
hostname (the ``--workers > 1`` path binds it unresolved) is resolved for
IPv4. A scoped IPv6 address cannot go in a probe URL.
"""
host = str(address[0])
if host in _WILDCARD_PROBE_HOSTS:
return _WILDCARD_PROBE_HOSTS[host]
if "%" in host:
return None
try:
ipaddress.ip_address(host)
except ValueError:
try:
infos = socket.getaddrinfo(host, None, socket.AF_INET, socket.SOCK_STREAM)
except (OSError, UnicodeError):
return None
return str(infos[0][4][0]) if infos else None
return host
def _refuse_busy_port(host: str, port: int, *, workers: int) -> None:
"""Fail before the banner when ``host:port`` is already taken.
Without this the banner prints "Press Ctrl+C to stop", then uvicorn's bind
error is buried among startup and shutdown log lines.
"""
busy = _tcp_port_in_use(host, port, workers=workers)
if busy is None:
return
error, address = busy
from headroom.cli.port_discovery import probe_headroom_proxy
reason = error.strerror or str(error)
probe_host = _probe_host_for(address)
if probe_host is None:
holder = f"({reason})"
elif probe_headroom_proxy(port, host=probe_host):
holder = "by a running Headroom proxy (reuse it, or stop it first)"
else:
holder = f"by another process ({reason})"
raise click.ClickException(
f"Port {port} on {host} is already in use {holder}. "
"Start this proxy on another port with --port N (env: HEADROOM_PORT)."
)
@main.command()
@click.option(
"--port",
@@ -1685,6 +1781,8 @@ Memory (Multi-Provider):
_bind = evaluate_bind_policy(config.host, config.proxy_token)
if _bind.refused:
raise click.ClickException(_bind.message())
if not config.uds:
_refuse_busy_port(config.host, config.port, workers=workers)
_auth_on = _bind.token_configured
_open_bind_note = (
f" · WARNING open bind, /v1/* UNAUTHENTICATED (acknowledged via {OPEN_BIND_ACK_ENV}=1)"
+64 -7
View File
@@ -695,6 +695,57 @@ def _get_proxy_stdio_log_path(port: int | None = None) -> Path:
return _get_log_path(port).with_name(_paths.proxy_stdio_log_path(port).name)
# A proxy that dies during startup logs its shutdown summary after the error,
# so the end of the log is the summary box, not the cause. Quote from the first
# error-looking line instead.
# Click's own failures (``click.ClickException``) print ``Error: ...`` at the
# start of a line.
_PROXY_STARTUP_ERROR_RE = re.compile(
r"\b(?:ERROR|CRITICAL)\b|Traceback \(most recent call last\)|^Error: ", re.MULTILINE
)
_PROXY_STARTUP_EXCERPT_CHARS = 500
# Never read a runaway log whole: look at most at the first and the last 64 KiB
# of this run's output. The error usually comes early, before the shutdown
# summary; the tail catches one that follows a long burst of startup output.
_PROXY_STARTUP_READ_BYTES = 64 * 1024
def _proxy_startup_failure_excerpt(stdio_log_path: Path, start: int) -> str:
"""Return the part of this run's stdio log that best explains a startup exit.
Reads only what was written after byte offset ``start`` (the log is shared
across runs): the first ``_PROXY_STARTUP_READ_BYTES`` of it and, when the run
wrote more and the head has no error, the last ``_PROXY_STARTUP_READ_BYTES``.
Prefers the first ERROR/CRITICAL/Traceback/``Error:`` line onwards, and falls
back to the last few hundred characters of the first non-blank window.
"""
try:
with open(stdio_log_path, "rb") as fh:
fh.seek(start)
windows = [fh.read(_PROXY_STARTUP_READ_BYTES)]
end = fh.seek(0, os.SEEK_END)
tail_start = max(start + _PROXY_STARTUP_READ_BYTES, end - _PROXY_STARTUP_READ_BYTES)
if end > tail_start and not _PROXY_STARTUP_ERROR_RE.search(
windows[0].decode("utf-8", errors="replace")
):
fh.seek(tail_start)
windows.append(fh.read(_PROXY_STARTUP_READ_BYTES))
except OSError:
return "(no log output)"
texts = [raw.decode("utf-8", errors="replace").strip() for raw in windows]
for output in texts:
match = _PROXY_STARTUP_ERROR_RE.search(output)
if match is not None:
line_start = output.rfind("\n", 0, match.start()) + 1
return output[line_start : line_start + _PROXY_STARTUP_EXCERPT_CHARS]
# No recognised error: quote the end of the first non-blank window, the head
# when it has text (the startup output), otherwise the tail.
for output in texts:
if output:
return output[-_PROXY_STARTUP_EXCERPT_CHARS:]
return "(no log output)"
def _start_proxy(
port: int,
*,
@@ -776,6 +827,12 @@ def _start_proxy(
timeout_seconds = _resolve_wrap_proxy_timeout_seconds()
log_path = _get_log_path(port)
stdio_log_path = _get_proxy_stdio_log_path(port)
# The stdio log is appended to across runs; remember where this run starts
# so a startup failure quotes this run's output, not an earlier one.
try:
stdio_log_start = stdio_log_path.stat().st_size
except OSError:
stdio_log_start = 0
stdio_log_file = open(stdio_log_path, "a", encoding="utf-8") # noqa: SIM115
# Ensure proxy subprocess uses UTF-8 (Windows defaults to cp1252)
@@ -896,17 +953,17 @@ def _start_proxy(
return proc
# Check if process died
if proc.poll() is not None:
# Read last few lines of log for error context
try:
tail = _read_text(stdio_log_path)[-500:]
except Exception:
tail = "(no log output)"
raise RuntimeError(f"Proxy exited with code {proc.returncode}: {tail}")
excerpt = _proxy_startup_failure_excerpt(stdio_log_path, stdio_log_start)
raise RuntimeError(
f"Proxy exited with code {proc.returncode}: {excerpt}\n"
f" Full log: {stdio_log_path}"
)
proc.kill()
raise RuntimeError(
f"Proxy failed to start on port {port} within {timeout_seconds} seconds. "
f"Set {_WRAP_PROXY_TIMEOUT_ENV} to a larger number of seconds for slow startup."
f"Set {_WRAP_PROXY_TIMEOUT_ENV} to a larger number of seconds for slow startup.\n"
f" Full log: {stdio_log_path}"
)
finally:
stdio_log_file.close()
+1
View File
@@ -534,6 +534,7 @@ markers = [
"real_llm: tests that hit real LLM APIs; skipped unless explicitly enabled",
"live: opt-in multi-turn tests that hit real upstream APIs; require provider keys",
"proxy_dependency_gate: exercises ensure_proxy_dependencies() without mocking",
"proxy_port_preflight: exercises the proxy busy-port check against real sockets",
"windows_newline: newline-contract tests (#3698); also run on windows-latest in CI",
]
+16
View File
@@ -63,6 +63,22 @@ def _skip_proxy_dependency_gate_unless_exercised(
monkeypatch.setattr(proxy, "ensure_proxy_dependencies", lambda: None)
# `headroom proxy` refuses to start when its port is already bound. Tests that
# invoke it with a stubbed run_server use the default 8787, which a developer's
# own running proxy holds, so the check is off unless a test exercises it.
@pytest.fixture(autouse=True)
def _skip_proxy_port_preflight_unless_exercised(
request: pytest.FixtureRequest, monkeypatch: pytest.MonkeyPatch
) -> None:
if request.node.get_closest_marker("proxy_port_preflight") is not None:
return
try:
from headroom.cli import proxy
except ModuleNotFoundError:
return
monkeypatch.setattr(proxy, "_refuse_busy_port", lambda *args, **kwargs: None)
@pytest.fixture(autouse=True)
def _scrub_developer_headroom_env(monkeypatch, tmp_path):
for key in list(os.environ):
+145
View File
@@ -0,0 +1,145 @@
"""`headroom proxy` refuses a busy port before printing its banner.
Without the check the banner ends "Press Ctrl+C to stop", then uvicorn's bind
error is buried among startup and shutdown log lines.
"""
from __future__ import annotations
import json
import socket
import sys
import threading
from collections.abc import Iterator
from contextlib import contextmanager
from http.server import BaseHTTPRequestHandler, HTTPServer
from unittest.mock import patch
import pytest
from click.testing import CliRunner
from headroom.cli.main import main
pytestmark = [
pytest.mark.proxy_port_preflight,
pytest.mark.skipif(sys.platform == "win32", reason="the check is POSIX-only"),
]
@contextmanager
def _listening(host: str = "127.0.0.1") -> Iterator[int]:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind((host, 0))
sock.listen(1)
try:
yield sock.getsockname()[1]
finally:
sock.close()
def _free_port() -> int:
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.bind(("127.0.0.1", 0))
return sock.getsockname()[1]
def _invoke_proxy(*args: str) -> tuple[object, list[object]]:
calls: list[object] = []
with (
patch("headroom.proxy.server.run_server", lambda config, **kw: calls.append(config)),
patch("headroom.cli.port_discovery.probe_headroom_proxy", return_value=False),
):
result = CliRunner().invoke(main, ["proxy", *args])
return result, calls
@pytest.mark.parametrize("workers", ["1", "2"])
def test_busy_port_fails_before_banner_with_actionable_message(workers: str) -> None:
with _listening() as port:
result, calls = _invoke_proxy("--port", str(port), "--workers", workers)
assert result.exit_code == 1, result.output
assert f"Port {port} on 127.0.0.1 is already in use by another process" in result.output
assert "--port N" in result.output
assert "HEADROOM PROXY" not in result.output
assert calls == []
def test_busy_port_held_by_headroom_says_so() -> None:
with _listening() as port:
with (
patch("headroom.proxy.server.run_server", lambda config, **kw: None),
patch("headroom.cli.port_discovery.probe_headroom_proxy", return_value=True),
):
result = CliRunner().invoke(main, ["proxy", "--port", str(port)])
assert result.exit_code == 1, result.output
assert "already in use by a running Headroom proxy" in result.output
def test_free_port_starts_normally() -> None:
port = _free_port()
result, calls = _invoke_proxy("--port", str(port))
assert result.exit_code == 0, result.output
assert "HEADROOM PROXY" in result.output
assert len(calls) == 1
def _ipv6_loopback_available() -> bool:
try:
with socket.socket(socket.AF_INET6, socket.SOCK_STREAM) as sock:
sock.bind(("::1", 0))
except OSError:
return False
return True
class _HeadroomLivez(BaseHTTPRequestHandler):
def do_GET(self) -> None: # noqa: N802 - http.server API
body = json.dumps({"service": "headroom-proxy", "alive": True}).encode()
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *args: object) -> None:
pass
@pytest.mark.skipif(not _ipv6_loopback_available(), reason="needs IPv6 loopback")
def test_ipv6_conflict_is_identified_on_the_requested_host() -> None:
"""A Headroom proxy on 127.0.0.1:P must not be reported as the owner of [::1]:P."""
headroom_v4 = HTTPServer(("127.0.0.1", 0), _HeadroomLivez)
port = headroom_v4.server_address[1]
threading.Thread(target=headroom_v4.serve_forever, daemon=True).start()
other_v6 = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)
try:
other_v6.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 1)
other_v6.bind(("::1", port))
other_v6.listen(1)
with patch("headroom.proxy.server.run_server", lambda config, **kw: None):
v6 = CliRunner().invoke(main, ["proxy", "--host", "::1", "--port", str(port)])
finally:
other_v6.close()
headroom_v4.shutdown()
headroom_v4.server_close()
assert v6.exit_code == 1, v6.output
assert f"Port {port} on ::1 is already in use by another process" in v6.output
assert "Headroom proxy" not in v6.output
def test_unprobeable_host_gets_the_generic_message() -> None:
with _listening() as port:
with (
patch("headroom.proxy.server.run_server", lambda config, **kw: None),
patch("headroom.cli.proxy._probe_host_for", return_value=None),
patch("headroom.cli.port_discovery.probe_headroom_proxy", return_value=True),
):
result = CliRunner().invoke(main, ["proxy", "--port", str(port)])
assert result.exit_code == 1, result.output
assert f"Port {port} on 127.0.0.1 is already in use (" in result.output
assert "Headroom proxy" not in result.output
+139 -2
View File
@@ -203,10 +203,15 @@ class TestCLIWrapProxyTimeout:
monkeypatch.setattr(wrap_mod, "_get_log_path", lambda port=None: tmp_path / "proxy.log")
monkeypatch.setattr(wrap_mod, "_check_proxy", lambda _port: False)
monkeypatch.setattr(wrap_mod.time, "sleep", lambda _seconds: None)
monkeypatch.setattr(wrap_mod.subprocess, "Popen", lambda *args, **kwargs: fake_proc)
def popen_writing_stdio(*args, **kwargs):
kwargs["stdout"].write("proxy stdio startup output\n")
kwargs["stdout"].flush()
return fake_proc
monkeypatch.setattr(wrap_mod.subprocess, "Popen", popen_writing_stdio)
(tmp_path / "proxy.log").write_text("canonical runtime log output")
(tmp_path / "proxy-stdio-8787.log").write_text("proxy stdio startup output")
with pytest.raises(RuntimeError) as excinfo:
wrap_mod._start_proxy(8787, agent_type="codex")
@@ -216,6 +221,137 @@ class TestCLIWrapProxyTimeout:
assert "proxy stdio startup output" in message
assert "canonical runtime log output" not in message
def test_start_proxy_exit_quotes_this_runs_error_not_summary_or_old_runs(
self, monkeypatch, tmp_path
):
"""The startup error names the cause and the log path.
The proxy logs its shutdown summary after the error, and the stdio log
is appended across runs, so the old ``[-500:]`` tail quoted the summary
box (or an earlier run) and never said where the full log was.
"""
fake_proc = _FakeProxyProcess()
fake_proc.returncode = 1
fake_proc.poll = lambda: fake_proc.returncode
monkeypatch.setenv(wrap_mod._WRAP_PROXY_TIMEOUT_ENV, "2")
monkeypatch.setattr(wrap_mod, "_get_log_path", lambda port=None: tmp_path / "proxy.log")
monkeypatch.setattr(wrap_mod, "_check_proxy", lambda _port: False)
monkeypatch.setattr(wrap_mod.time, "sleep", lambda _seconds: None)
stdio_log = tmp_path / "proxy-stdio-8787.log"
stdio_log.write_text("ERROR: stale failure from an earlier run\n")
summary = "".join(
f"2026-10-09 16:19:23,665 - headroom.proxy - INFO - Summary line {i}: 0\n"
for i in range(20)
)
def popen_failing_to_bind(*args, **kwargs):
kwargs["stdout"].write(
"2026-10-09 16:19:23,659 - headroom.proxy - INFO - Local telemetry: DISABLED\n"
"ERROR: [Errno 48] error while attempting to bind on address "
"('127.0.0.1', 8787): address already in use\n" + summary
)
kwargs["stdout"].flush()
return fake_proc
monkeypatch.setattr(wrap_mod.subprocess, "Popen", popen_failing_to_bind)
with pytest.raises(RuntimeError) as excinfo:
wrap_mod._start_proxy(8787, agent_type="codex")
message = str(excinfo.value)
assert "Proxy exited with code 1" in message
assert "[Errno 48] error while attempting to bind" in message
assert "stale failure from an earlier run" not in message
assert "Local telemetry" not in message
assert f"Full log: {stdio_log}" in message
def test_startup_excerpt_reads_at_most_64_kib_of_this_run(self, monkeypatch, tmp_path):
"""A runaway stdio log is never read whole: this run's first and last 64 KiB
at most, so an error after a long burst of output is still quoted."""
stdio_log = tmp_path / "proxy-stdio-8787.log"
earlier = "ERROR: an earlier run\n" * 1000
noise = "INFO noise line\n" * 16000 # ~256 KiB, no error line
late = "ERROR: late startup failure\n"
# Bytes, not text: the offset is a byte offset, and write_text would turn
# "\n" into "\r\n" on Windows.
stdio_log.write_bytes((earlier + noise + late + "INFO shutdown summary\n").encode())
read_sizes: list[int] = []
real_open = open
def tracking_open(*args, **kwargs):
handle = real_open(*args, **kwargs)
real_read = handle.read
def read(size=-1):
data = real_read(size)
read_sizes.append(len(data))
return data
handle.read = read
return handle
monkeypatch.setattr(wrap_mod, "open", tracking_open, raising=False)
excerpt = wrap_mod._proxy_startup_failure_excerpt(stdio_log, len(earlier.encode()))
assert "earlier run" not in excerpt
assert excerpt.startswith("ERROR: late startup failure")
assert sum(read_sizes) <= 2 * wrap_mod._PROXY_STARTUP_READ_BYTES
def test_startup_excerpt_prefers_an_error_in_the_first_64_kib(self, tmp_path):
stdio_log = tmp_path / "proxy-stdio-8787.log"
noise = "INFO noise line\n" * 16000
stdio_log.write_bytes(("ERROR: early failure\n" + noise + "ERROR: later echo\n").encode())
excerpt = wrap_mod._proxy_startup_failure_excerpt(stdio_log, 0)
assert excerpt.startswith("ERROR: early failure")
def test_startup_excerpt_without_error_quotes_the_end_of_the_head(self, tmp_path):
"""No recognised error line: quote the end of this run's first 64 KiB,
even when the tail window is blank."""
stdio_log = tmp_path / "proxy-stdio-8787.log"
head = "INFO noise line\n" * 3000 + "INFO last startup line\n" # < 64 KiB
stdio_log.write_bytes((head + " \n" * 70000).encode())
excerpt = wrap_mod._proxy_startup_failure_excerpt(stdio_log, 0)
assert excerpt.endswith("INFO last startup line")
def test_startup_excerpt_falls_back_to_a_useful_tail_after_a_blank_head(self, tmp_path):
"""A whitespace-only head must not hide a useful tail line."""
stdio_log = tmp_path / "proxy-stdio-8787.log"
stdio_log.write_bytes(b" " * (192 * 1024) + b"initialization could not bind the port\n")
excerpt = wrap_mod._proxy_startup_failure_excerpt(stdio_log, 0)
assert excerpt == "initialization could not bind the port"
def test_startup_excerpt_quotes_click_error_before_a_blank_tail(self, tmp_path):
"""A Click ``Error:`` line followed by a whitespace-only tail is quoted,
not reported as "(no log output)"."""
stdio_log = tmp_path / "proxy-stdio-8787.log"
stdio_log.write_bytes(("Error: address already in use\n" + " " * (192 * 1024)).encode())
excerpt = wrap_mod._proxy_startup_failure_excerpt(stdio_log, 0)
assert excerpt == "Error: address already in use"
def test_startup_excerpt_starts_at_a_click_style_error_line(self, tmp_path):
stdio_log = tmp_path / "proxy-stdio-8787.log"
stdio_log.write_bytes(
b"INFO starting\nINFO loading config\n"
b"Error: Port 8787 on 127.0.0.1 is already in use by another process.\n"
b"INFO shutdown summary\n"
)
excerpt = wrap_mod._proxy_startup_failure_excerpt(stdio_log, 0)
assert excerpt.startswith("Error: Port 8787 on 127.0.0.1 is already in use")
assert "INFO starting" not in excerpt
def test_timeout_error_names_configured_timeout_and_env_var(self, monkeypatch, tmp_path):
fake_proc = _FakeProxyProcess()
@@ -231,6 +367,7 @@ class TestCLIWrapProxyTimeout:
message = str(excinfo.value)
assert "within 2 seconds" in message
assert wrap_mod._WRAP_PROXY_TIMEOUT_ENV in message
assert f"Full log: {tmp_path / 'proxy-stdio-8787.log'}" in message
assert fake_proc.killed is True