mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(cli): say why the proxy failed to start (#4120)
## Description Fixes two proxy startup failures that hid their cause from the user. **1. `headroom proxy` on a port that is already in use.** It printed the full banner, ending in "Press Ctrl+C to stop.", and then about 10 INFO lines. The one line that mattered, uvicorn's `[Errno 48] ... address already in use`, came next. After it came `event=proxy_shutdown reason=signal` and a 10-line zero-valued session summary. The command now checks the port before printing the banner and exits with one clear line: ``` Error: Port 54351 on 127.0.0.1 is already in use by another process (Address already in use). Start this proxy on another port with --port N (env: HEADROOM_PORT). ``` When a Headroom proxy is the process holding the port, the message says so instead. The check asks the address that is actually busy (IPv6 included), so a Headroom proxy on 127.0.0.1:P is never reported as the owner of [::1]:P. When that address cannot be probed (a scoped IPv6 address, an unresolvable name), the message is the generic `already in use (<reason>)`. **2. `headroom wrap <tool>` when the proxy it starts dies during startup.** The error quoted the last 500 characters of the proxy stdio log. That was the wrong part of the log, for two reasons: - The proxy logs its shutdown summary after the error, so the tail was the summary box. - The log is appended to across runs, so the tail could come from an older run. The error also never said where the log was. It now reads only this run's output, at most its first 64 KiB and, when the head has no error, its last 64 KiB (so a runaway log is never read whole, and an error after a long burst of startup output is still found), quotes from the first `ERROR` / `CRITICAL` / `Traceback` line or line-leading Click `Error: ` (otherwise the end of the first non-blank window, so neither a blank head nor a blank tail hides useful startup text), and adds `Full log: <path>`. The timeout error now names the log too. Refs #4136 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cli/proxy.py`: new `_tcp_port_in_use` / `_refuse_busy_port`. They run after the bind-policy check and before the banner, and are skipped for `--uds`. The probe mirrors uvicorn's own bind, so it never refuses a bind that uvicorn would accept: - It sets `SO_REUSEADDR`, as both uvicorn paths do. A port in TIME_WAIT after a restart is therefore not reported as busy. - With one worker, it binds every address the host resolves to, with `IPV6_V6ONLY`, the same way asyncio `create_server` does. - With `--workers > 1`, it binds one AF_INET or AF_INET6 socket, as uvicorn's `Config.bind_socket` does. - On Windows, and for any error other than EADDRINUSE, it does nothing and leaves the case to uvicorn, exactly as today. - It returns the busy address. `_probe_host_for` maps it to the address to ask: wildcards go to loopback of the same family (`0.0.0.0` to `127.0.0.1`, `::` to `::1`), a hostname is resolved for IPv4, and a scoped IPv6 address returns None (generic message). - `headroom/cli/port_discovery.py`: `probe_headroom_proxy` takes an optional keyword `host` (default `127.0.0.1`, so existing callers are unchanged) and brackets IPv6 in the URL. - `headroom/cli/wrap.py`: - `_start_proxy` records the stdio log size before spawning the proxy. - New `_proxy_startup_failure_excerpt` picks the error line from this run's output, reading at most the first and last 64 KiB written after the recorded offset. - Both startup errors now end with `Full log: <path>`. - `tests/conftest.py` + `pyproject.toml`: an autouse guard, and its `proxy_port_preflight` marker, turn the new check off unless a test opts in. It follows the existing `proxy_dependency_gate` pattern. - Without it, the 16 test files that invoke `headroom proxy` with a stubbed `run_server` on the default port 8787 would fail whenever a developer has their own proxy running. - Tests: - New `tests/test_cli/test_proxy_port_preflight.py`, which uses real sockets: - busy port with workers 1 and 2; - port held by a Headroom proxy; - free port still starts; - `test_ipv6_conflict_is_identified_on_the_requested_host`: a real fake-Headroom `/livez` on 127.0.0.1:P and a plain listener on [::1]:P; `--host ::1` must say "another process"; - `test_unprobeable_host_gets_the_generic_message`. - New `test_startup_excerpt_reads_at_most_64_kib_of_this_run`. - New `test_start_proxy_exit_quotes_this_runs_error_not_summary_or_old_runs`. - The timeout test now also asserts the log path. - The existing stdio-tail test now writes its log output from the fake `Popen`, as the real child process does. - The new tests fail on the old code (3 preflight tests; the excerpt and timeout tests) and pass now. The three tests added for the review (IPv6 owner, unprobeable host, 64 KiB bound) fail on the PR's previous commit03288c05. The late-error test (an error after 256 KiB of output is quoted, at most 128 KiB read) fails onc9868420. The head-fallback test (blank tail) fails on062cf011, and the two Click `Error:` tests fail on409fee89(one of them also on062cf011). The blank-head/useful-tail test fails on92b1bd08. The excerpt fixtures write bytes, so their byte offsets also hold on Windows. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest -q -p no:cacheprovider tests/test_cli/test_proxy_port_preflight.py tests/test_cli_proxy_env.py tests/test_proxy_bind_policy.py tests/test_proxy_uds.py tests/test_cli/test_wrap_helpers.py tests/test_cli/test_wrap_proxy_detach.py tests/test_cli/test_port_alignment.py tests/test_malloc_tuning.py tests/test_telemetry_warning.py tests/test_cli_extension_seam.py tests/test_agent_savings.py tests/test_no_ccr_disables_response_handling.py tests/test_stateless_no_disk.py tests/test_cli_proxy_embedding_server.py tests/test_proxy_per_provider_kompress.py tests/test_savings_ledger.py 11 failed, 427 passed, 1 warning in 18.04s # The 11 failures are all in tests/test_proxy_uds.py ("Socket path is 134 bytes, over this platform's 104-byte sun_path limit"). They are pre-existing on this machine: clean origin/main gives "11 failed, 19 passed" for that file. # Regression tests against the old code: 3 failed, 1 passed (test_proxy_port_preflight.py with the proxy.py change reverted) 2 failed, 8 passed (test_cli_proxy_env.py -k "TestCLIWrapProxyTimeout or start_proxy" with the wrap.py change reverted) # After the review fixes (c9868420): $ pytest -q -p no:cacheprovider tests/test_cli/test_proxy_port_preflight.py tests/test_cli_proxy_env.py tests/test_cli/test_wrap_helpers.py tests/test_cli/test_wrap_proxy_detach.py tests/test_cli/test_port_alignment.py tests/test_proxy_bind_policy.py 247 passed $ pytest -q tests/test_cli -k "port or discovery or livez" 126 passed, 989 deselected # The 3 new review tests against03288c05: 3 failed $ uvx ruff@0.16.8 check headroom/cli/proxy.py headroom/cli/wrap.py tests/conftest.py tests/test_cli_proxy_env.py tests/test_cli/test_proxy_port_preflight.py All checks passed! $ mypy headroom/cli/proxy.py headroom/cli/wrap.py headroom/cli/port_discovery.py --ignore-missing-imports Success: no issues found in 3 source files ``` ## Real Behavior Proof - Environment: macOS 25.6 with Python 3.12 (repo `.venv`); Linux checked in Docker `python:3.12-slim-trixie` with uvicorn 0.40.0. - Exact command / steps: 1. Bound a socket to 127.0.0.1:N, then ran `python -m headroom.cli proxy --port N`. 2. Ran a script that compares `_tcp_port_in_use` with uvicorn's real bind (asyncio `create_server` for one worker, `uvicorn.Config(...).bind_socket()` for workers=2). It covered free ports; listeners on 127.0.0.1, 0.0.0.0 and ::1; hosts 127.0.0.1, 0.0.0.0, localhost and ::; a port in TIME_WAIT; and a non-local host. - Observed result: 1. rc=1, empty stdout, and stderr `Error: Port 54351 on 127.0.0.1 is already in use by another process (Address already in use). Start this proxy on another port with --port N (env: HEADROOM_PORT).` The banner did not print and uvicorn was never reached. 2. On both macOS and Linux, the probe agreed with uvicorn in every case, including the ones where the two platforms differ (a 127.0.0.1 listener with host 0.0.0.0 is busy on Linux and free on macOS). TIME_WAIT is not reported as busy. For a non-local host, the probe returns "not busy" and leaves the EADDRNOTAVAIL error to uvicorn, unchanged. - Direct runtime command (this PR's path). A script holds 127.0.0.1:P with a fake Headroom `/livez` and [::1]:P with a plain listener, then runs the real CLI, `python -m headroom.cli proxy --host <h> --port P`, with temp `HOME`, `HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`: ```text $ headroom proxy --host ::1 --port 64747 exit=1 stdout='' stderr: Error: Port 64747 on ::1 is already in use by another process (Address already in use). Start this proxy on another port with --port N (env: HEADROOM_PORT). $ headroom proxy --host 127.0.0.1 --port 64747 exit=1 stdout='' stderr: Error: Port 64747 on 127.0.0.1 is already in use by a running Headroom proxy (reuse it, or stop it first). Start this proxy on another port with --port N (env: HEADROOM_PORT). # Same script on the previous commit03288c05, --host ::1: stderr: Error: Port 64976 on ::1 is already in use by a running Headroom proxy (reuse it, or stop it first). ... ``` - Shared smoke harness (`smoke.sh <worktree>`: a real proxy from this checkout against a local fake upstream, offline, temp HOME): ```text branch: cleanup/cli-integrations-proxy-start-errors @92b1bd08health: 200 POST /v1/messages -> HTTP200 smoke-ok POST /v1/chat/completions -> HTTP200 smoke-ok stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82 proxy.log WARNING/ERROR lines (<tmp>/ws/logs/proxy-49585.log): 2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes. 2 headroom.transforms.kompress_compressor - WARNING - Kompress: 'onnx/kompress-int8-wo.onnx' for chopratejas/kompress-v2-base is not cached and HEADROOM_OFFLINE forbids fetching it (HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.). Compression falls back to the non-ML path. 4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base 4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection. (smoke dir: <tmp>) ``` - Not tested: Windows (the check is skipped there by design), and a live `wrap` session whose proxy crashes. The excerpt logic is covered by the unit test. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: N/A, not rollout-managed. - Stable/default behavior changed: Yes, only on failure paths. A busy port now fails before the banner (exit 1, same as before) with a clear message; wrap's startup errors quote a different part of the log and add the path. - Kill switch / disable path: None needed. The probe only refuses on EADDRINUSE, in the same cases where uvicorn itself would fail to bind. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - Part of maintainer-requested cleanup #4136 (dead code, duplication, debuggability), split into small PRs for review. - Restart race: the check runs before the heavy app import, a few seconds earlier than uvicorn's bind used to. A restart whose old process still holds the listening socket can now fail fast where it used to win the race narrowly. The restart paths wait for the old process to exit first (`stop_runtime` polls `pid_alive`; systemd and launchd wait for exit), and the exit code is 1 either way. - No docs change: the new behaviour is an error message. - The audit had also proposed changing wrap's startup `except RuntimeError` to raise `ClickException` (stderr). I dropped that: every caller of `_ensure_proxy` wraps it in `except Exception: echo " Error: ..."; SystemExit(1)`, so the output would not change. - The audit had proposed reusing wrap's `_port_bind_error` for the probe. I did not, because that helper binds 127.0.0.1 only and without `SO_REUSEADDR`, so it does not match uvicorn. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4120" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JerrettDavis
parent
6e14ee0bc2
commit
f3d240bcae
@@ -120,16 +120,19 @@ def candidate_ports(
|
||||
return seen[:_MAX_CANDIDATES]
|
||||
|
||||
|
||||
def probe_headroom_proxy(port: int, timeout: float = _PROBE_TIMEOUT_SECONDS) -> bool:
|
||||
"""True when a Headroom proxy answers ``/livez`` on loopback ``port``.
|
||||
def probe_headroom_proxy(
|
||||
port: int, timeout: float = _PROBE_TIMEOUT_SECONDS, *, host: str = "127.0.0.1"
|
||||
) -> bool:
|
||||
"""True when a Headroom proxy answers ``/livez`` on ``host:port`` (loopback by default).
|
||||
|
||||
Requires the ``service: headroom-proxy`` identity (or the legacy
|
||||
``alive``+``version`` shape) so an unrelated local server on the port is
|
||||
never mistaken for Headroom.
|
||||
never mistaken for Headroom. ``host`` is a numeric IPv4 or IPv6 address.
|
||||
"""
|
||||
from headroom.install.health import probe_json
|
||||
|
||||
return is_headroom_livez(probe_json(f"http://127.0.0.1:{port}/livez", timeout=timeout))
|
||||
netloc = f"[{host}]" if ":" in host else host
|
||||
return is_headroom_livez(probe_json(f"http://{netloc}:{port}/livez", timeout=timeout))
|
||||
|
||||
|
||||
def is_headroom_livez(payload: Mapping[str, Any] | None) -> bool:
|
||||
|
||||
@@ -1,8 +1,11 @@
|
||||
"""Proxy server CLI commands."""
|
||||
|
||||
import errno
|
||||
import ipaddress
|
||||
import logging
|
||||
import os
|
||||
import platform
|
||||
import socket
|
||||
import sys
|
||||
import warnings
|
||||
from importlib.util import find_spec
|
||||
@@ -195,6 +198,99 @@ def _get_env_int(name: str, default: int) -> int:
|
||||
return default if value is None else value
|
||||
|
||||
|
||||
def _tcp_port_in_use(host: str, port: int, *, workers: int) -> tuple[OSError, Any] | None:
|
||||
"""Return ``(EADDRINUSE error, busy address)`` uvicorn would hit binding ``host:port``.
|
||||
|
||||
Returns None when the bind would succeed.
|
||||
|
||||
Mirrors uvicorn's own bind so the probe never refuses a bind uvicorn would
|
||||
accept: ``SO_REUSEADDR`` (both uvicorn paths set it on POSIX, so a port in
|
||||
TIME_WAIT after a restart is not "in use"); a single process binds every
|
||||
address ``host`` resolves to (asyncio ``create_server``), while
|
||||
``--workers > 1`` binds one AF_INET/AF_INET6 socket in the parent.
|
||||
Windows ``SO_REUSEADDR`` semantics differ, so Windows is left to uvicorn,
|
||||
as is any error other than EADDRINUSE (unresolvable host, address not on
|
||||
this machine); uvicorn reports those as before.
|
||||
"""
|
||||
if sys.platform == "win32":
|
||||
return None
|
||||
if workers > 1:
|
||||
family = socket.AF_INET6 if ":" in host else socket.AF_INET
|
||||
targets: list[tuple[Any, Any]] = [(family, (host, port))]
|
||||
else:
|
||||
try:
|
||||
infos = socket.getaddrinfo(
|
||||
host or None, port, type=socket.SOCK_STREAM, flags=socket.AI_PASSIVE
|
||||
)
|
||||
except (OSError, UnicodeError):
|
||||
return None
|
||||
targets = [(info[0], info[4]) for info in infos]
|
||||
for family, address in targets:
|
||||
try:
|
||||
with socket.socket(family, socket.SOCK_STREAM) as probe:
|
||||
probe.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
|
||||
if family == socket.AF_INET6 and workers <= 1:
|
||||
# asyncio binds v4 and v6 separately; match it.
|
||||
probe.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 1)
|
||||
probe.bind(address)
|
||||
except OSError as exc:
|
||||
if exc.errno == errno.EADDRINUSE:
|
||||
return exc, address
|
||||
return None
|
||||
|
||||
|
||||
_WILDCARD_PROBE_HOSTS = {"": "127.0.0.1", "0.0.0.0": "127.0.0.1", "::": "::1"}
|
||||
|
||||
|
||||
def _probe_host_for(address: Any) -> str | None:
|
||||
"""The numeric address to ask "is this Headroom?", or None when it can't be targeted.
|
||||
|
||||
A wildcard bind is reached through loopback of the same family. A
|
||||
hostname (the ``--workers > 1`` path binds it unresolved) is resolved for
|
||||
IPv4. A scoped IPv6 address cannot go in a probe URL.
|
||||
"""
|
||||
host = str(address[0])
|
||||
if host in _WILDCARD_PROBE_HOSTS:
|
||||
return _WILDCARD_PROBE_HOSTS[host]
|
||||
if "%" in host:
|
||||
return None
|
||||
try:
|
||||
ipaddress.ip_address(host)
|
||||
except ValueError:
|
||||
try:
|
||||
infos = socket.getaddrinfo(host, None, socket.AF_INET, socket.SOCK_STREAM)
|
||||
except (OSError, UnicodeError):
|
||||
return None
|
||||
return str(infos[0][4][0]) if infos else None
|
||||
return host
|
||||
|
||||
|
||||
def _refuse_busy_port(host: str, port: int, *, workers: int) -> None:
|
||||
"""Fail before the banner when ``host:port`` is already taken.
|
||||
|
||||
Without this the banner prints "Press Ctrl+C to stop", then uvicorn's bind
|
||||
error is buried among startup and shutdown log lines.
|
||||
"""
|
||||
busy = _tcp_port_in_use(host, port, workers=workers)
|
||||
if busy is None:
|
||||
return
|
||||
error, address = busy
|
||||
from headroom.cli.port_discovery import probe_headroom_proxy
|
||||
|
||||
reason = error.strerror or str(error)
|
||||
probe_host = _probe_host_for(address)
|
||||
if probe_host is None:
|
||||
holder = f"({reason})"
|
||||
elif probe_headroom_proxy(port, host=probe_host):
|
||||
holder = "by a running Headroom proxy (reuse it, or stop it first)"
|
||||
else:
|
||||
holder = f"by another process ({reason})"
|
||||
raise click.ClickException(
|
||||
f"Port {port} on {host} is already in use {holder}. "
|
||||
"Start this proxy on another port with --port N (env: HEADROOM_PORT)."
|
||||
)
|
||||
|
||||
|
||||
@main.command()
|
||||
@click.option(
|
||||
"--port",
|
||||
@@ -1685,6 +1781,8 @@ Memory (Multi-Provider):
|
||||
_bind = evaluate_bind_policy(config.host, config.proxy_token)
|
||||
if _bind.refused:
|
||||
raise click.ClickException(_bind.message())
|
||||
if not config.uds:
|
||||
_refuse_busy_port(config.host, config.port, workers=workers)
|
||||
_auth_on = _bind.token_configured
|
||||
_open_bind_note = (
|
||||
f" · WARNING open bind, /v1/* UNAUTHENTICATED (acknowledged via {OPEN_BIND_ACK_ENV}=1)"
|
||||
|
||||
+64
-7
@@ -695,6 +695,57 @@ def _get_proxy_stdio_log_path(port: int | None = None) -> Path:
|
||||
return _get_log_path(port).with_name(_paths.proxy_stdio_log_path(port).name)
|
||||
|
||||
|
||||
# A proxy that dies during startup logs its shutdown summary after the error,
|
||||
# so the end of the log is the summary box, not the cause. Quote from the first
|
||||
# error-looking line instead.
|
||||
# Click's own failures (``click.ClickException``) print ``Error: ...`` at the
|
||||
# start of a line.
|
||||
_PROXY_STARTUP_ERROR_RE = re.compile(
|
||||
r"\b(?:ERROR|CRITICAL)\b|Traceback \(most recent call last\)|^Error: ", re.MULTILINE
|
||||
)
|
||||
_PROXY_STARTUP_EXCERPT_CHARS = 500
|
||||
# Never read a runaway log whole: look at most at the first and the last 64 KiB
|
||||
# of this run's output. The error usually comes early, before the shutdown
|
||||
# summary; the tail catches one that follows a long burst of startup output.
|
||||
_PROXY_STARTUP_READ_BYTES = 64 * 1024
|
||||
|
||||
|
||||
def _proxy_startup_failure_excerpt(stdio_log_path: Path, start: int) -> str:
|
||||
"""Return the part of this run's stdio log that best explains a startup exit.
|
||||
|
||||
Reads only what was written after byte offset ``start`` (the log is shared
|
||||
across runs): the first ``_PROXY_STARTUP_READ_BYTES`` of it and, when the run
|
||||
wrote more and the head has no error, the last ``_PROXY_STARTUP_READ_BYTES``.
|
||||
Prefers the first ERROR/CRITICAL/Traceback/``Error:`` line onwards, and falls
|
||||
back to the last few hundred characters of the first non-blank window.
|
||||
"""
|
||||
try:
|
||||
with open(stdio_log_path, "rb") as fh:
|
||||
fh.seek(start)
|
||||
windows = [fh.read(_PROXY_STARTUP_READ_BYTES)]
|
||||
end = fh.seek(0, os.SEEK_END)
|
||||
tail_start = max(start + _PROXY_STARTUP_READ_BYTES, end - _PROXY_STARTUP_READ_BYTES)
|
||||
if end > tail_start and not _PROXY_STARTUP_ERROR_RE.search(
|
||||
windows[0].decode("utf-8", errors="replace")
|
||||
):
|
||||
fh.seek(tail_start)
|
||||
windows.append(fh.read(_PROXY_STARTUP_READ_BYTES))
|
||||
except OSError:
|
||||
return "(no log output)"
|
||||
texts = [raw.decode("utf-8", errors="replace").strip() for raw in windows]
|
||||
for output in texts:
|
||||
match = _PROXY_STARTUP_ERROR_RE.search(output)
|
||||
if match is not None:
|
||||
line_start = output.rfind("\n", 0, match.start()) + 1
|
||||
return output[line_start : line_start + _PROXY_STARTUP_EXCERPT_CHARS]
|
||||
# No recognised error: quote the end of the first non-blank window, the head
|
||||
# when it has text (the startup output), otherwise the tail.
|
||||
for output in texts:
|
||||
if output:
|
||||
return output[-_PROXY_STARTUP_EXCERPT_CHARS:]
|
||||
return "(no log output)"
|
||||
|
||||
|
||||
def _start_proxy(
|
||||
port: int,
|
||||
*,
|
||||
@@ -776,6 +827,12 @@ def _start_proxy(
|
||||
timeout_seconds = _resolve_wrap_proxy_timeout_seconds()
|
||||
log_path = _get_log_path(port)
|
||||
stdio_log_path = _get_proxy_stdio_log_path(port)
|
||||
# The stdio log is appended to across runs; remember where this run starts
|
||||
# so a startup failure quotes this run's output, not an earlier one.
|
||||
try:
|
||||
stdio_log_start = stdio_log_path.stat().st_size
|
||||
except OSError:
|
||||
stdio_log_start = 0
|
||||
stdio_log_file = open(stdio_log_path, "a", encoding="utf-8") # noqa: SIM115
|
||||
|
||||
# Ensure proxy subprocess uses UTF-8 (Windows defaults to cp1252)
|
||||
@@ -896,17 +953,17 @@ def _start_proxy(
|
||||
return proc
|
||||
# Check if process died
|
||||
if proc.poll() is not None:
|
||||
# Read last few lines of log for error context
|
||||
try:
|
||||
tail = _read_text(stdio_log_path)[-500:]
|
||||
except Exception:
|
||||
tail = "(no log output)"
|
||||
raise RuntimeError(f"Proxy exited with code {proc.returncode}: {tail}")
|
||||
excerpt = _proxy_startup_failure_excerpt(stdio_log_path, stdio_log_start)
|
||||
raise RuntimeError(
|
||||
f"Proxy exited with code {proc.returncode}: {excerpt}\n"
|
||||
f" Full log: {stdio_log_path}"
|
||||
)
|
||||
|
||||
proc.kill()
|
||||
raise RuntimeError(
|
||||
f"Proxy failed to start on port {port} within {timeout_seconds} seconds. "
|
||||
f"Set {_WRAP_PROXY_TIMEOUT_ENV} to a larger number of seconds for slow startup."
|
||||
f"Set {_WRAP_PROXY_TIMEOUT_ENV} to a larger number of seconds for slow startup.\n"
|
||||
f" Full log: {stdio_log_path}"
|
||||
)
|
||||
finally:
|
||||
stdio_log_file.close()
|
||||
|
||||
@@ -534,6 +534,7 @@ markers = [
|
||||
"real_llm: tests that hit real LLM APIs; skipped unless explicitly enabled",
|
||||
"live: opt-in multi-turn tests that hit real upstream APIs; require provider keys",
|
||||
"proxy_dependency_gate: exercises ensure_proxy_dependencies() without mocking",
|
||||
"proxy_port_preflight: exercises the proxy busy-port check against real sockets",
|
||||
"windows_newline: newline-contract tests (#3698); also run on windows-latest in CI",
|
||||
]
|
||||
|
||||
|
||||
@@ -63,6 +63,22 @@ def _skip_proxy_dependency_gate_unless_exercised(
|
||||
monkeypatch.setattr(proxy, "ensure_proxy_dependencies", lambda: None)
|
||||
|
||||
|
||||
# `headroom proxy` refuses to start when its port is already bound. Tests that
|
||||
# invoke it with a stubbed run_server use the default 8787, which a developer's
|
||||
# own running proxy holds, so the check is off unless a test exercises it.
|
||||
@pytest.fixture(autouse=True)
|
||||
def _skip_proxy_port_preflight_unless_exercised(
|
||||
request: pytest.FixtureRequest, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
if request.node.get_closest_marker("proxy_port_preflight") is not None:
|
||||
return
|
||||
try:
|
||||
from headroom.cli import proxy
|
||||
except ModuleNotFoundError:
|
||||
return
|
||||
monkeypatch.setattr(proxy, "_refuse_busy_port", lambda *args, **kwargs: None)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _scrub_developer_headroom_env(monkeypatch, tmp_path):
|
||||
for key in list(os.environ):
|
||||
|
||||
@@ -0,0 +1,145 @@
|
||||
"""`headroom proxy` refuses a busy port before printing its banner.
|
||||
|
||||
Without the check the banner ends "Press Ctrl+C to stop", then uvicorn's bind
|
||||
error is buried among startup and shutdown log lines.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import socket
|
||||
import sys
|
||||
import threading
|
||||
from collections.abc import Iterator
|
||||
from contextlib import contextmanager
|
||||
from http.server import BaseHTTPRequestHandler, HTTPServer
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
from click.testing import CliRunner
|
||||
|
||||
from headroom.cli.main import main
|
||||
|
||||
pytestmark = [
|
||||
pytest.mark.proxy_port_preflight,
|
||||
pytest.mark.skipif(sys.platform == "win32", reason="the check is POSIX-only"),
|
||||
]
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _listening(host: str = "127.0.0.1") -> Iterator[int]:
|
||||
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||
sock.bind((host, 0))
|
||||
sock.listen(1)
|
||||
try:
|
||||
yield sock.getsockname()[1]
|
||||
finally:
|
||||
sock.close()
|
||||
|
||||
|
||||
def _free_port() -> int:
|
||||
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
|
||||
sock.bind(("127.0.0.1", 0))
|
||||
return sock.getsockname()[1]
|
||||
|
||||
|
||||
def _invoke_proxy(*args: str) -> tuple[object, list[object]]:
|
||||
calls: list[object] = []
|
||||
with (
|
||||
patch("headroom.proxy.server.run_server", lambda config, **kw: calls.append(config)),
|
||||
patch("headroom.cli.port_discovery.probe_headroom_proxy", return_value=False),
|
||||
):
|
||||
result = CliRunner().invoke(main, ["proxy", *args])
|
||||
return result, calls
|
||||
|
||||
|
||||
@pytest.mark.parametrize("workers", ["1", "2"])
|
||||
def test_busy_port_fails_before_banner_with_actionable_message(workers: str) -> None:
|
||||
with _listening() as port:
|
||||
result, calls = _invoke_proxy("--port", str(port), "--workers", workers)
|
||||
|
||||
assert result.exit_code == 1, result.output
|
||||
assert f"Port {port} on 127.0.0.1 is already in use by another process" in result.output
|
||||
assert "--port N" in result.output
|
||||
assert "HEADROOM PROXY" not in result.output
|
||||
assert calls == []
|
||||
|
||||
|
||||
def test_busy_port_held_by_headroom_says_so() -> None:
|
||||
with _listening() as port:
|
||||
with (
|
||||
patch("headroom.proxy.server.run_server", lambda config, **kw: None),
|
||||
patch("headroom.cli.port_discovery.probe_headroom_proxy", return_value=True),
|
||||
):
|
||||
result = CliRunner().invoke(main, ["proxy", "--port", str(port)])
|
||||
|
||||
assert result.exit_code == 1, result.output
|
||||
assert "already in use by a running Headroom proxy" in result.output
|
||||
|
||||
|
||||
def test_free_port_starts_normally() -> None:
|
||||
port = _free_port()
|
||||
result, calls = _invoke_proxy("--port", str(port))
|
||||
|
||||
assert result.exit_code == 0, result.output
|
||||
assert "HEADROOM PROXY" in result.output
|
||||
assert len(calls) == 1
|
||||
|
||||
|
||||
def _ipv6_loopback_available() -> bool:
|
||||
try:
|
||||
with socket.socket(socket.AF_INET6, socket.SOCK_STREAM) as sock:
|
||||
sock.bind(("::1", 0))
|
||||
except OSError:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
class _HeadroomLivez(BaseHTTPRequestHandler):
|
||||
def do_GET(self) -> None: # noqa: N802 - http.server API
|
||||
body = json.dumps({"service": "headroom-proxy", "alive": True}).encode()
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def log_message(self, *args: object) -> None:
|
||||
pass
|
||||
|
||||
|
||||
@pytest.mark.skipif(not _ipv6_loopback_available(), reason="needs IPv6 loopback")
|
||||
def test_ipv6_conflict_is_identified_on_the_requested_host() -> None:
|
||||
"""A Headroom proxy on 127.0.0.1:P must not be reported as the owner of [::1]:P."""
|
||||
headroom_v4 = HTTPServer(("127.0.0.1", 0), _HeadroomLivez)
|
||||
port = headroom_v4.server_address[1]
|
||||
threading.Thread(target=headroom_v4.serve_forever, daemon=True).start()
|
||||
other_v6 = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)
|
||||
try:
|
||||
other_v6.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 1)
|
||||
other_v6.bind(("::1", port))
|
||||
other_v6.listen(1)
|
||||
with patch("headroom.proxy.server.run_server", lambda config, **kw: None):
|
||||
v6 = CliRunner().invoke(main, ["proxy", "--host", "::1", "--port", str(port)])
|
||||
finally:
|
||||
other_v6.close()
|
||||
headroom_v4.shutdown()
|
||||
headroom_v4.server_close()
|
||||
|
||||
assert v6.exit_code == 1, v6.output
|
||||
assert f"Port {port} on ::1 is already in use by another process" in v6.output
|
||||
assert "Headroom proxy" not in v6.output
|
||||
|
||||
|
||||
def test_unprobeable_host_gets_the_generic_message() -> None:
|
||||
with _listening() as port:
|
||||
with (
|
||||
patch("headroom.proxy.server.run_server", lambda config, **kw: None),
|
||||
patch("headroom.cli.proxy._probe_host_for", return_value=None),
|
||||
patch("headroom.cli.port_discovery.probe_headroom_proxy", return_value=True),
|
||||
):
|
||||
result = CliRunner().invoke(main, ["proxy", "--port", str(port)])
|
||||
|
||||
assert result.exit_code == 1, result.output
|
||||
assert f"Port {port} on 127.0.0.1 is already in use (" in result.output
|
||||
assert "Headroom proxy" not in result.output
|
||||
+139
-2
@@ -203,10 +203,15 @@ class TestCLIWrapProxyTimeout:
|
||||
monkeypatch.setattr(wrap_mod, "_get_log_path", lambda port=None: tmp_path / "proxy.log")
|
||||
monkeypatch.setattr(wrap_mod, "_check_proxy", lambda _port: False)
|
||||
monkeypatch.setattr(wrap_mod.time, "sleep", lambda _seconds: None)
|
||||
monkeypatch.setattr(wrap_mod.subprocess, "Popen", lambda *args, **kwargs: fake_proc)
|
||||
|
||||
def popen_writing_stdio(*args, **kwargs):
|
||||
kwargs["stdout"].write("proxy stdio startup output\n")
|
||||
kwargs["stdout"].flush()
|
||||
return fake_proc
|
||||
|
||||
monkeypatch.setattr(wrap_mod.subprocess, "Popen", popen_writing_stdio)
|
||||
|
||||
(tmp_path / "proxy.log").write_text("canonical runtime log output")
|
||||
(tmp_path / "proxy-stdio-8787.log").write_text("proxy stdio startup output")
|
||||
|
||||
with pytest.raises(RuntimeError) as excinfo:
|
||||
wrap_mod._start_proxy(8787, agent_type="codex")
|
||||
@@ -216,6 +221,137 @@ class TestCLIWrapProxyTimeout:
|
||||
assert "proxy stdio startup output" in message
|
||||
assert "canonical runtime log output" not in message
|
||||
|
||||
def test_start_proxy_exit_quotes_this_runs_error_not_summary_or_old_runs(
|
||||
self, monkeypatch, tmp_path
|
||||
):
|
||||
"""The startup error names the cause and the log path.
|
||||
|
||||
The proxy logs its shutdown summary after the error, and the stdio log
|
||||
is appended across runs, so the old ``[-500:]`` tail quoted the summary
|
||||
box (or an earlier run) and never said where the full log was.
|
||||
"""
|
||||
fake_proc = _FakeProxyProcess()
|
||||
fake_proc.returncode = 1
|
||||
fake_proc.poll = lambda: fake_proc.returncode
|
||||
|
||||
monkeypatch.setenv(wrap_mod._WRAP_PROXY_TIMEOUT_ENV, "2")
|
||||
monkeypatch.setattr(wrap_mod, "_get_log_path", lambda port=None: tmp_path / "proxy.log")
|
||||
monkeypatch.setattr(wrap_mod, "_check_proxy", lambda _port: False)
|
||||
monkeypatch.setattr(wrap_mod.time, "sleep", lambda _seconds: None)
|
||||
|
||||
stdio_log = tmp_path / "proxy-stdio-8787.log"
|
||||
stdio_log.write_text("ERROR: stale failure from an earlier run\n")
|
||||
summary = "".join(
|
||||
f"2026-10-09 16:19:23,665 - headroom.proxy - INFO - Summary line {i}: 0\n"
|
||||
for i in range(20)
|
||||
)
|
||||
|
||||
def popen_failing_to_bind(*args, **kwargs):
|
||||
kwargs["stdout"].write(
|
||||
"2026-10-09 16:19:23,659 - headroom.proxy - INFO - Local telemetry: DISABLED\n"
|
||||
"ERROR: [Errno 48] error while attempting to bind on address "
|
||||
"('127.0.0.1', 8787): address already in use\n" + summary
|
||||
)
|
||||
kwargs["stdout"].flush()
|
||||
return fake_proc
|
||||
|
||||
monkeypatch.setattr(wrap_mod.subprocess, "Popen", popen_failing_to_bind)
|
||||
|
||||
with pytest.raises(RuntimeError) as excinfo:
|
||||
wrap_mod._start_proxy(8787, agent_type="codex")
|
||||
|
||||
message = str(excinfo.value)
|
||||
assert "Proxy exited with code 1" in message
|
||||
assert "[Errno 48] error while attempting to bind" in message
|
||||
assert "stale failure from an earlier run" not in message
|
||||
assert "Local telemetry" not in message
|
||||
assert f"Full log: {stdio_log}" in message
|
||||
|
||||
def test_startup_excerpt_reads_at_most_64_kib_of_this_run(self, monkeypatch, tmp_path):
|
||||
"""A runaway stdio log is never read whole: this run's first and last 64 KiB
|
||||
at most, so an error after a long burst of output is still quoted."""
|
||||
stdio_log = tmp_path / "proxy-stdio-8787.log"
|
||||
earlier = "ERROR: an earlier run\n" * 1000
|
||||
noise = "INFO noise line\n" * 16000 # ~256 KiB, no error line
|
||||
late = "ERROR: late startup failure\n"
|
||||
# Bytes, not text: the offset is a byte offset, and write_text would turn
|
||||
# "\n" into "\r\n" on Windows.
|
||||
stdio_log.write_bytes((earlier + noise + late + "INFO shutdown summary\n").encode())
|
||||
read_sizes: list[int] = []
|
||||
real_open = open
|
||||
|
||||
def tracking_open(*args, **kwargs):
|
||||
handle = real_open(*args, **kwargs)
|
||||
real_read = handle.read
|
||||
|
||||
def read(size=-1):
|
||||
data = real_read(size)
|
||||
read_sizes.append(len(data))
|
||||
return data
|
||||
|
||||
handle.read = read
|
||||
return handle
|
||||
|
||||
monkeypatch.setattr(wrap_mod, "open", tracking_open, raising=False)
|
||||
|
||||
excerpt = wrap_mod._proxy_startup_failure_excerpt(stdio_log, len(earlier.encode()))
|
||||
|
||||
assert "earlier run" not in excerpt
|
||||
assert excerpt.startswith("ERROR: late startup failure")
|
||||
assert sum(read_sizes) <= 2 * wrap_mod._PROXY_STARTUP_READ_BYTES
|
||||
|
||||
def test_startup_excerpt_prefers_an_error_in_the_first_64_kib(self, tmp_path):
|
||||
stdio_log = tmp_path / "proxy-stdio-8787.log"
|
||||
noise = "INFO noise line\n" * 16000
|
||||
stdio_log.write_bytes(("ERROR: early failure\n" + noise + "ERROR: later echo\n").encode())
|
||||
|
||||
excerpt = wrap_mod._proxy_startup_failure_excerpt(stdio_log, 0)
|
||||
|
||||
assert excerpt.startswith("ERROR: early failure")
|
||||
|
||||
def test_startup_excerpt_without_error_quotes_the_end_of_the_head(self, tmp_path):
|
||||
"""No recognised error line: quote the end of this run's first 64 KiB,
|
||||
even when the tail window is blank."""
|
||||
stdio_log = tmp_path / "proxy-stdio-8787.log"
|
||||
head = "INFO noise line\n" * 3000 + "INFO last startup line\n" # < 64 KiB
|
||||
stdio_log.write_bytes((head + " \n" * 70000).encode())
|
||||
|
||||
excerpt = wrap_mod._proxy_startup_failure_excerpt(stdio_log, 0)
|
||||
|
||||
assert excerpt.endswith("INFO last startup line")
|
||||
|
||||
def test_startup_excerpt_falls_back_to_a_useful_tail_after_a_blank_head(self, tmp_path):
|
||||
"""A whitespace-only head must not hide a useful tail line."""
|
||||
stdio_log = tmp_path / "proxy-stdio-8787.log"
|
||||
stdio_log.write_bytes(b" " * (192 * 1024) + b"initialization could not bind the port\n")
|
||||
|
||||
excerpt = wrap_mod._proxy_startup_failure_excerpt(stdio_log, 0)
|
||||
|
||||
assert excerpt == "initialization could not bind the port"
|
||||
|
||||
def test_startup_excerpt_quotes_click_error_before_a_blank_tail(self, tmp_path):
|
||||
"""A Click ``Error:`` line followed by a whitespace-only tail is quoted,
|
||||
not reported as "(no log output)"."""
|
||||
stdio_log = tmp_path / "proxy-stdio-8787.log"
|
||||
stdio_log.write_bytes(("Error: address already in use\n" + " " * (192 * 1024)).encode())
|
||||
|
||||
excerpt = wrap_mod._proxy_startup_failure_excerpt(stdio_log, 0)
|
||||
|
||||
assert excerpt == "Error: address already in use"
|
||||
|
||||
def test_startup_excerpt_starts_at_a_click_style_error_line(self, tmp_path):
|
||||
stdio_log = tmp_path / "proxy-stdio-8787.log"
|
||||
stdio_log.write_bytes(
|
||||
b"INFO starting\nINFO loading config\n"
|
||||
b"Error: Port 8787 on 127.0.0.1 is already in use by another process.\n"
|
||||
b"INFO shutdown summary\n"
|
||||
)
|
||||
|
||||
excerpt = wrap_mod._proxy_startup_failure_excerpt(stdio_log, 0)
|
||||
|
||||
assert excerpt.startswith("Error: Port 8787 on 127.0.0.1 is already in use")
|
||||
assert "INFO starting" not in excerpt
|
||||
|
||||
def test_timeout_error_names_configured_timeout_and_env_var(self, monkeypatch, tmp_path):
|
||||
fake_proc = _FakeProxyProcess()
|
||||
|
||||
@@ -231,6 +367,7 @@ class TestCLIWrapProxyTimeout:
|
||||
message = str(excinfo.value)
|
||||
assert "within 2 seconds" in message
|
||||
assert wrap_mod._WRAP_PROXY_TIMEOUT_ENV in message
|
||||
assert f"Full log: {tmp_path / 'proxy-stdio-8787.log'}" in message
|
||||
assert fake_proc.killed is True
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user