mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description Fixes two proxy startup failures that hid their cause from the user. **1. `headroom proxy` on a port that is already in use.** It printed the full banner, ending in "Press Ctrl+C to stop.", and then about 10 INFO lines. The one line that mattered, uvicorn's `[Errno 48] ... address already in use`, came next. After it came `event=proxy_shutdown reason=signal` and a 10-line zero-valued session summary. The command now checks the port before printing the banner and exits with one clear line: ``` Error: Port 54351 on 127.0.0.1 is already in use by another process (Address already in use). Start this proxy on another port with --port N (env: HEADROOM_PORT). ``` When a Headroom proxy is the process holding the port, the message says so instead. The check asks the address that is actually busy (IPv6 included), so a Headroom proxy on 127.0.0.1:P is never reported as the owner of [::1]:P. When that address cannot be probed (a scoped IPv6 address, an unresolvable name), the message is the generic `already in use (<reason>)`. **2. `headroom wrap <tool>` when the proxy it starts dies during startup.** The error quoted the last 500 characters of the proxy stdio log. That was the wrong part of the log, for two reasons: - The proxy logs its shutdown summary after the error, so the tail was the summary box. - The log is appended to across runs, so the tail could come from an older run. The error also never said where the log was. It now reads only this run's output, at most its first 64 KiB and, when the head has no error, its last 64 KiB (so a runaway log is never read whole, and an error after a long burst of startup output is still found), quotes from the first `ERROR` / `CRITICAL` / `Traceback` line or line-leading Click `Error: ` (otherwise the end of the first non-blank window, so neither a blank head nor a blank tail hides useful startup text), and adds `Full log: <path>`. The timeout error now names the log too. Refs #4136 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cli/proxy.py`: new `_tcp_port_in_use` / `_refuse_busy_port`. They run after the bind-policy check and before the banner, and are skipped for `--uds`. The probe mirrors uvicorn's own bind, so it never refuses a bind that uvicorn would accept: - It sets `SO_REUSEADDR`, as both uvicorn paths do. A port in TIME_WAIT after a restart is therefore not reported as busy. - With one worker, it binds every address the host resolves to, with `IPV6_V6ONLY`, the same way asyncio `create_server` does. - With `--workers > 1`, it binds one AF_INET or AF_INET6 socket, as uvicorn's `Config.bind_socket` does. - On Windows, and for any error other than EADDRINUSE, it does nothing and leaves the case to uvicorn, exactly as today. - It returns the busy address. `_probe_host_for` maps it to the address to ask: wildcards go to loopback of the same family (`0.0.0.0` to `127.0.0.1`, `::` to `::1`), a hostname is resolved for IPv4, and a scoped IPv6 address returns None (generic message). - `headroom/cli/port_discovery.py`: `probe_headroom_proxy` takes an optional keyword `host` (default `127.0.0.1`, so existing callers are unchanged) and brackets IPv6 in the URL. - `headroom/cli/wrap.py`: - `_start_proxy` records the stdio log size before spawning the proxy. - New `_proxy_startup_failure_excerpt` picks the error line from this run's output, reading at most the first and last 64 KiB written after the recorded offset. - Both startup errors now end with `Full log: <path>`. - `tests/conftest.py` + `pyproject.toml`: an autouse guard, and its `proxy_port_preflight` marker, turn the new check off unless a test opts in. It follows the existing `proxy_dependency_gate` pattern. - Without it, the 16 test files that invoke `headroom proxy` with a stubbed `run_server` on the default port 8787 would fail whenever a developer has their own proxy running. - Tests: - New `tests/test_cli/test_proxy_port_preflight.py`, which uses real sockets: - busy port with workers 1 and 2; - port held by a Headroom proxy; - free port still starts; - `test_ipv6_conflict_is_identified_on_the_requested_host`: a real fake-Headroom `/livez` on 127.0.0.1:P and a plain listener on [::1]:P; `--host ::1` must say "another process"; - `test_unprobeable_host_gets_the_generic_message`. - New `test_startup_excerpt_reads_at_most_64_kib_of_this_run`. - New `test_start_proxy_exit_quotes_this_runs_error_not_summary_or_old_runs`. - The timeout test now also asserts the log path. - The existing stdio-tail test now writes its log output from the fake `Popen`, as the real child process does. - The new tests fail on the old code (3 preflight tests; the excerpt and timeout tests) and pass now. The three tests added for the review (IPv6 owner, unprobeable host, 64 KiB bound) fail on the PR's previous commit03288c05. The late-error test (an error after 256 KiB of output is quoted, at most 128 KiB read) fails onc9868420. The head-fallback test (blank tail) fails on062cf011, and the two Click `Error:` tests fail on409fee89(one of them also on062cf011). The blank-head/useful-tail test fails on92b1bd08. The excerpt fixtures write bytes, so their byte offsets also hold on Windows. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest -q -p no:cacheprovider tests/test_cli/test_proxy_port_preflight.py tests/test_cli_proxy_env.py tests/test_proxy_bind_policy.py tests/test_proxy_uds.py tests/test_cli/test_wrap_helpers.py tests/test_cli/test_wrap_proxy_detach.py tests/test_cli/test_port_alignment.py tests/test_malloc_tuning.py tests/test_telemetry_warning.py tests/test_cli_extension_seam.py tests/test_agent_savings.py tests/test_no_ccr_disables_response_handling.py tests/test_stateless_no_disk.py tests/test_cli_proxy_embedding_server.py tests/test_proxy_per_provider_kompress.py tests/test_savings_ledger.py 11 failed, 427 passed, 1 warning in 18.04s # The 11 failures are all in tests/test_proxy_uds.py ("Socket path is 134 bytes, over this platform's 104-byte sun_path limit"). They are pre-existing on this machine: clean origin/main gives "11 failed, 19 passed" for that file. # Regression tests against the old code: 3 failed, 1 passed (test_proxy_port_preflight.py with the proxy.py change reverted) 2 failed, 8 passed (test_cli_proxy_env.py -k "TestCLIWrapProxyTimeout or start_proxy" with the wrap.py change reverted) # After the review fixes (c9868420): $ pytest -q -p no:cacheprovider tests/test_cli/test_proxy_port_preflight.py tests/test_cli_proxy_env.py tests/test_cli/test_wrap_helpers.py tests/test_cli/test_wrap_proxy_detach.py tests/test_cli/test_port_alignment.py tests/test_proxy_bind_policy.py 247 passed $ pytest -q tests/test_cli -k "port or discovery or livez" 126 passed, 989 deselected # The 3 new review tests against03288c05: 3 failed $ uvx ruff@0.16.8 check headroom/cli/proxy.py headroom/cli/wrap.py tests/conftest.py tests/test_cli_proxy_env.py tests/test_cli/test_proxy_port_preflight.py All checks passed! $ mypy headroom/cli/proxy.py headroom/cli/wrap.py headroom/cli/port_discovery.py --ignore-missing-imports Success: no issues found in 3 source files ``` ## Real Behavior Proof - Environment: macOS 25.6 with Python 3.12 (repo `.venv`); Linux checked in Docker `python:3.12-slim-trixie` with uvicorn 0.40.0. - Exact command / steps: 1. Bound a socket to 127.0.0.1:N, then ran `python -m headroom.cli proxy --port N`. 2. Ran a script that compares `_tcp_port_in_use` with uvicorn's real bind (asyncio `create_server` for one worker, `uvicorn.Config(...).bind_socket()` for workers=2). It covered free ports; listeners on 127.0.0.1, 0.0.0.0 and ::1; hosts 127.0.0.1, 0.0.0.0, localhost and ::; a port in TIME_WAIT; and a non-local host. - Observed result: 1. rc=1, empty stdout, and stderr `Error: Port 54351 on 127.0.0.1 is already in use by another process (Address already in use). Start this proxy on another port with --port N (env: HEADROOM_PORT).` The banner did not print and uvicorn was never reached. 2. On both macOS and Linux, the probe agreed with uvicorn in every case, including the ones where the two platforms differ (a 127.0.0.1 listener with host 0.0.0.0 is busy on Linux and free on macOS). TIME_WAIT is not reported as busy. For a non-local host, the probe returns "not busy" and leaves the EADDRNOTAVAIL error to uvicorn, unchanged. - Direct runtime command (this PR's path). A script holds 127.0.0.1:P with a fake Headroom `/livez` and [::1]:P with a plain listener, then runs the real CLI, `python -m headroom.cli proxy --host <h> --port P`, with temp `HOME`, `HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`: ```text $ headroom proxy --host ::1 --port 64747 exit=1 stdout='' stderr: Error: Port 64747 on ::1 is already in use by another process (Address already in use). Start this proxy on another port with --port N (env: HEADROOM_PORT). $ headroom proxy --host 127.0.0.1 --port 64747 exit=1 stdout='' stderr: Error: Port 64747 on 127.0.0.1 is already in use by a running Headroom proxy (reuse it, or stop it first). Start this proxy on another port with --port N (env: HEADROOM_PORT). # Same script on the previous commit03288c05, --host ::1: stderr: Error: Port 64976 on ::1 is already in use by a running Headroom proxy (reuse it, or stop it first). ... ``` - Shared smoke harness (`smoke.sh <worktree>`: a real proxy from this checkout against a local fake upstream, offline, temp HOME): ```text branch: cleanup/cli-integrations-proxy-start-errors @92b1bd08health: 200 POST /v1/messages -> HTTP200 smoke-ok POST /v1/chat/completions -> HTTP200 smoke-ok stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82 proxy.log WARNING/ERROR lines (<tmp>/ws/logs/proxy-49585.log): 2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes. 2 headroom.transforms.kompress_compressor - WARNING - Kompress: 'onnx/kompress-int8-wo.onnx' for chopratejas/kompress-v2-base is not cached and HEADROOM_OFFLINE forbids fetching it (HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.). Compression falls back to the non-ML path. 4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base 4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection. (smoke dir: <tmp>) ``` - Not tested: Windows (the check is skipped there by design), and a live `wrap` session whose proxy crashes. The excerpt logic is covered by the unit test. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: N/A, not rollout-managed. - Stable/default behavior changed: Yes, only on failure paths. A busy port now fails before the banner (exit 1, same as before) with a clear message; wrap's startup errors quote a different part of the log and add the path. - Kill switch / disable path: None needed. The probe only refuses on EADDRINUSE, in the same cases where uvicorn itself would fail to bind. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - Part of maintainer-requested cleanup #4136 (dead code, duplication, debuggability), split into small PRs for review. - Restart race: the check runs before the heavy app import, a few seconds earlier than uvicorn's bind used to. A restart whose old process still holds the listening socket can now fail fast where it used to win the race narrowly. The restart paths wait for the old process to exit first (`stop_runtime` polls `pid_alive`; systemd and launchd wait for exit), and the exit code is 1 either way. - No docs change: the new behaviour is an error message. - The audit had also proposed changing wrap's startup `except RuntimeError` to raise `ClickException` (stderr). I dropped that: every caller of `_ensure_proxy` wraps it in `except Exception: echo " Error: ..."; SystemExit(1)`, so the output would not change. - The audit had proposed reusing wrap's `_port_bind_error` for the probe. I did not, because that helper binds 127.0.0.1 only and without `SO_REUSEADDR`, so it does not match uvicorn. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4120" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
146 lines
5.0 KiB
Python
146 lines
5.0 KiB
Python
"""`headroom proxy` refuses a busy port before printing its banner.
|
|
|
|
Without the check the banner ends "Press Ctrl+C to stop", then uvicorn's bind
|
|
error is buried among startup and shutdown log lines.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import socket
|
|
import sys
|
|
import threading
|
|
from collections.abc import Iterator
|
|
from contextlib import contextmanager
|
|
from http.server import BaseHTTPRequestHandler, HTTPServer
|
|
from unittest.mock import patch
|
|
|
|
import pytest
|
|
from click.testing import CliRunner
|
|
|
|
from headroom.cli.main import main
|
|
|
|
pytestmark = [
|
|
pytest.mark.proxy_port_preflight,
|
|
pytest.mark.skipif(sys.platform == "win32", reason="the check is POSIX-only"),
|
|
]
|
|
|
|
|
|
@contextmanager
|
|
def _listening(host: str = "127.0.0.1") -> Iterator[int]:
|
|
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
|
sock.bind((host, 0))
|
|
sock.listen(1)
|
|
try:
|
|
yield sock.getsockname()[1]
|
|
finally:
|
|
sock.close()
|
|
|
|
|
|
def _free_port() -> int:
|
|
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
|
|
sock.bind(("127.0.0.1", 0))
|
|
return sock.getsockname()[1]
|
|
|
|
|
|
def _invoke_proxy(*args: str) -> tuple[object, list[object]]:
|
|
calls: list[object] = []
|
|
with (
|
|
patch("headroom.proxy.server.run_server", lambda config, **kw: calls.append(config)),
|
|
patch("headroom.cli.port_discovery.probe_headroom_proxy", return_value=False),
|
|
):
|
|
result = CliRunner().invoke(main, ["proxy", *args])
|
|
return result, calls
|
|
|
|
|
|
@pytest.mark.parametrize("workers", ["1", "2"])
|
|
def test_busy_port_fails_before_banner_with_actionable_message(workers: str) -> None:
|
|
with _listening() as port:
|
|
result, calls = _invoke_proxy("--port", str(port), "--workers", workers)
|
|
|
|
assert result.exit_code == 1, result.output
|
|
assert f"Port {port} on 127.0.0.1 is already in use by another process" in result.output
|
|
assert "--port N" in result.output
|
|
assert "HEADROOM PROXY" not in result.output
|
|
assert calls == []
|
|
|
|
|
|
def test_busy_port_held_by_headroom_says_so() -> None:
|
|
with _listening() as port:
|
|
with (
|
|
patch("headroom.proxy.server.run_server", lambda config, **kw: None),
|
|
patch("headroom.cli.port_discovery.probe_headroom_proxy", return_value=True),
|
|
):
|
|
result = CliRunner().invoke(main, ["proxy", "--port", str(port)])
|
|
|
|
assert result.exit_code == 1, result.output
|
|
assert "already in use by a running Headroom proxy" in result.output
|
|
|
|
|
|
def test_free_port_starts_normally() -> None:
|
|
port = _free_port()
|
|
result, calls = _invoke_proxy("--port", str(port))
|
|
|
|
assert result.exit_code == 0, result.output
|
|
assert "HEADROOM PROXY" in result.output
|
|
assert len(calls) == 1
|
|
|
|
|
|
def _ipv6_loopback_available() -> bool:
|
|
try:
|
|
with socket.socket(socket.AF_INET6, socket.SOCK_STREAM) as sock:
|
|
sock.bind(("::1", 0))
|
|
except OSError:
|
|
return False
|
|
return True
|
|
|
|
|
|
class _HeadroomLivez(BaseHTTPRequestHandler):
|
|
def do_GET(self) -> None: # noqa: N802 - http.server API
|
|
body = json.dumps({"service": "headroom-proxy", "alive": True}).encode()
|
|
self.send_response(200)
|
|
self.send_header("Content-Type", "application/json")
|
|
self.send_header("Content-Length", str(len(body)))
|
|
self.end_headers()
|
|
self.wfile.write(body)
|
|
|
|
def log_message(self, *args: object) -> None:
|
|
pass
|
|
|
|
|
|
@pytest.mark.skipif(not _ipv6_loopback_available(), reason="needs IPv6 loopback")
|
|
def test_ipv6_conflict_is_identified_on_the_requested_host() -> None:
|
|
"""A Headroom proxy on 127.0.0.1:P must not be reported as the owner of [::1]:P."""
|
|
headroom_v4 = HTTPServer(("127.0.0.1", 0), _HeadroomLivez)
|
|
port = headroom_v4.server_address[1]
|
|
threading.Thread(target=headroom_v4.serve_forever, daemon=True).start()
|
|
other_v6 = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)
|
|
try:
|
|
other_v6.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 1)
|
|
other_v6.bind(("::1", port))
|
|
other_v6.listen(1)
|
|
with patch("headroom.proxy.server.run_server", lambda config, **kw: None):
|
|
v6 = CliRunner().invoke(main, ["proxy", "--host", "::1", "--port", str(port)])
|
|
finally:
|
|
other_v6.close()
|
|
headroom_v4.shutdown()
|
|
headroom_v4.server_close()
|
|
|
|
assert v6.exit_code == 1, v6.output
|
|
assert f"Port {port} on ::1 is already in use by another process" in v6.output
|
|
assert "Headroom proxy" not in v6.output
|
|
|
|
|
|
def test_unprobeable_host_gets_the_generic_message() -> None:
|
|
with _listening() as port:
|
|
with (
|
|
patch("headroom.proxy.server.run_server", lambda config, **kw: None),
|
|
patch("headroom.cli.proxy._probe_host_for", return_value=None),
|
|
patch("headroom.cli.port_discovery.probe_headroom_proxy", return_value=True),
|
|
):
|
|
result = CliRunner().invoke(main, ["proxy", "--port", str(port)])
|
|
|
|
assert result.exit_code == 1, result.output
|
|
assert f"Port {port} on 127.0.0.1 is already in use (" in result.output
|
|
assert "Headroom proxy" not in result.output
|