mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description Every proxy instance wrote the same two fixed files: `~/.headroom/logs/proxy.log` (runtime log, via the `RotatingFileHandler` in `_setup_file_logging`) and `~/.headroom/logs/proxy-stdio.log` (subprocess stdout/stderr from `_start_proxy`). The name is a constant (`paths.py` `_PROXY_LOG_FILE = "proxy.log"`), so N concurrent proxies on different ports (wrapped copilot, droid, claude; plus any persistent `install agent` deployment) run N independent rotating handlers over the *same* file and truncate/rename each other's history. This fix keys both files by port (`proxy-<port>.log`, `proxy-stdio-<port>.log`) so each proxy rotates only its own file. `headroom perf` aggregates every per-port file and still reads a lone legacy `proxy.log` when that is all that exists (backward compat); the `proxy-stdio-*` captures (raw stdout/stderr, not PERF records) are excluded from the parser. This matters more now that wrap-spawned proxies self-exit after an idle grace (orphan-proxy watchdog, #3202): the process is gone, so its per-port log is the only forensic evidence and must survive an unrelated proxy's rotation. Closes #3203 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/paths.py`: `proxy_log_path(port=None)` and new `proxy_stdio_log_path(port=None)` are the single source of truth for the filenames (`proxy-<port>.log` / `proxy-stdio-<port>.log` with a port, legacy names without). The `_PROXY_LOG_FILE` constant stays only as the readers' fallback name. - `headroom/proxy/helpers.py`: `_setup_file_logging(port=None)` targets `proxy_log_path(port)`. The duplicate-handler guard runs BEFORE the `RotatingFileHandler` is constructed (construction opens the file), so a second `create_app` on a different port in one process cannot leave an empty stray `proxy-<port>.log` or leak an fd; a handler pointing at a different port is replaced and closed so per-port keying holds. - `headroom/proxy/server.py`: `create_app` resolves `config` first, then calls `_setup_file_logging(config.port)` (the proxy always knows its own port). - `headroom/cli/wrap.py`: `_get_log_path(port=None)` / `_get_proxy_stdio_log_path(port=None)` are per-port; `_start_proxy` passes its `port`. The stdio filename comes from `paths.proxy_stdio_log_path` (single source) and its directory from `_get_log_path`. - `headroom/perf/analyzer.py`: `parse_log_files` selects PERF logs by a positive filename match (`_PERF_LOG_FILE_RE` = `proxy(-<digits>)?.log(.N)?`) rather than a `proxy-stdio` blacklist, so `proxy-stdio-*.log` and any other `proxy-<word>.log` are never ingested; ordering is deterministic (mtime, then path). - `headroom/cli/perf.py`, `wiki/cli.md`: doc/help text updated for per-port + legacy. - `tests/test_perf_per_port_logs.py` (new): per-port path helpers; a second `_setup_file_logging` on a new port reroutes cleanly with no cross-file leakage (regression for the guard-ordering bug); and `parse_log_files` aggregates per-port + legacy while excluding stdio and other non-PERF `proxy-*.log`. - `tests/test_cli_proxy_env.py`, `tests/test_agent_savings.py`, `tests/test_cli/test_wrap_{codex,claude_vertex_proxy_env,proxy_detach}.py`: the `_get_log_path` monkeypatches take the new optional `port`, and the two stdio-filename assertions/fixtures in `test_cli_proxy_env.py` use the per-port name. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) — plus `ruff format --check` - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/bin/python -m pytest tests/test_perf_per_port_logs.py tests/test_cli_proxy_env.py tests/test_agent_savings.py -q -p no:randomly 115 passed, 1 skipped in 4.44s $ .venv/bin/python -m pytest tests/test_cli_perf_format.py tests/test_learn/test_analyzer.py -q 107 passed in 4.16s $ .venv/bin/python -m pytest tests/test_cli/test_wrap_proxy_detach.py tests/test_cli/test_wrap_codex.py tests/test_cli/test_wrap_claude_vertex_proxy_env.py -q 120 passed in 171.19s $ .venv/bin/ruff check <touched> && .venv/bin/ruff format --check <touched> All checks passed! $ .venv/bin/mypy headroom/paths.py headroom/cli/wrap.py headroom/proxy/helpers.py headroom/proxy/server.py headroom/perf/analyzer.py Success: no issues found in 5 source files ``` Rust gate (`cargo clippy/test/fmt`) not run: Python-only change, no crate touched. ## Real Behavior Proof - Environment: macOS 26.5 arm64, Python 3.13, local editable source build. - Exact command / steps: `headroom proxy --port 8890` and `headroom proxy --port 8891` started concurrently; inspected `~/.headroom/logs/`; then `headroom perf --hours 1 --format json`. - Observed result: each proxy wrote its own file (`proxy-8890.log`, `proxy-8891.log`, independent), the pre-existing shared `proxy.log` was not touched, and `headroom perf` reported `log_files_read: 4`, aggregating per-port + legacy with no error. `test_setup_file_logging_reroutes_on_port_change` additionally drives the production helper twice in one process and asserts each port's records land only in its own file. - Not tested: Windows path semantics (filename construction is platform-independent `pathlib`); a legacy `proxy.log` written by an OLD binary running beside a new per-port one end to end (readers handle it, but not exercised live). ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: N/A (no flag; readers accept both layouts). - Stable/default behavior changed: log filenames change from `proxy.log` to `proxy-<port>.log`; `headroom perf` reads both, so history is not orphaned. - Kill switch / disable path: N/A (no flag). - Unsafe override required: no. - Qualification impact: none — readers stay backward compatible with a lone legacy `proxy.log`. - Rollback path: revert; a downgraded binary writes legacy `proxy.log` again and readers still aggregate any per-port files left behind. Disk growth: each per-port family keeps the prior bound (`maxBytes=10 MB × backupCount=5` = 60 MB), but there is now one family per distinct port used instead of one global family. Ports are usually stable (default 8787; wrap auto-selects a nearby free port), so in practice a handful of families; the theoretical worst case is bounded by the port scan (`_find_available_port`, ≤100) → ≤ ~6 GB. No automatic pruning; stale `proxy-<port>.log*` families can be deleted safely (perf reads whatever remains). A retention policy is intentionally out of scope. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — log-filename change; no UI. ## Additional Notes Follow-up (separate SDK PR, not here): the TypeScript SDK helper `sdk/typescript/src/paths.ts::proxyLogPath()` still returns legacy `proxy.log` with no port argument and is pinned by `paths.test.ts`. It mirrors `paths.py`, so it should gain an optional port argument in an SDK PR (kept separate to stay one-surface, and because that package's vitest suite is the right gate for it).
147 lines
5.9 KiB
Python
147 lines
5.9 KiB
Python
"""Detachment of the shared proxy subprocess in ``headroom wrap``.
|
|
|
|
``_start_proxy`` launches the proxy that every wrapped agent on a port shares.
|
|
It must outlive an *ungraceful* close of the agent that happened to start it
|
|
(closing the terminal window, taskkill, a crash) -- otherwise the OS tree-kills
|
|
the proxy and breaks the other live clients, bypassing the marker-based
|
|
reference counting in ``_make_cleanup``.
|
|
|
|
On Windows that means detaching from the launcher's console and Job object via
|
|
creation flags; on POSIX ``start_new_session`` already detaches via setsid().
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
import pytest
|
|
|
|
from headroom.cli import wrap as wrap_cli
|
|
|
|
# Windows-only creation flag; not always exported by the host's ``subprocess``.
|
|
_CREATE_BREAKAWAY_FROM_JOB = 0x01000000
|
|
|
|
|
|
class _FakeProc:
|
|
"""Stand-in for a live proxy process (``poll() is None``)."""
|
|
|
|
returncode = 0
|
|
|
|
def poll(self) -> None:
|
|
return None
|
|
|
|
|
|
def _capture_popen_kwargs(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> dict[str, Any]:
|
|
"""Invoke ``_start_proxy`` with all I/O stubbed; return the Popen kwargs."""
|
|
captured: dict[str, Any] = {}
|
|
|
|
def _fake_popen(cmd: Any, **kwargs: Any) -> _FakeProc:
|
|
captured.clear()
|
|
captured.update(kwargs)
|
|
return _FakeProc()
|
|
|
|
monkeypatch.setattr(wrap_cli.subprocess, "Popen", _fake_popen)
|
|
monkeypatch.setattr(wrap_cli, "_check_proxy", lambda port: True)
|
|
monkeypatch.setattr(wrap_cli.time, "sleep", lambda _seconds: None)
|
|
monkeypatch.setattr(wrap_cli, "_get_log_path", lambda port=None: tmp_path / "proxy.log")
|
|
monkeypatch.setattr(wrap_cli, "_resolve_wrap_proxy_timeout_seconds", lambda: 1)
|
|
|
|
wrap_cli._start_proxy(8787)
|
|
return captured
|
|
|
|
|
|
def test_start_proxy_detaches_on_windows(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
|
|
# Force the Windows branch regardless of the host OS, and supply the
|
|
# Windows-only ``subprocess`` constants the host lacks on POSIX.
|
|
monkeypatch.setattr(wrap_cli.sys, "platform", "win32")
|
|
monkeypatch.setattr(wrap_cli.subprocess, "CREATE_NO_WINDOW", 0x08000000, raising=False)
|
|
monkeypatch.setattr(wrap_cli.subprocess, "CREATE_NEW_PROCESS_GROUP", 0x200, raising=False)
|
|
|
|
flags = _capture_popen_kwargs(monkeypatch, tmp_path)["creationflags"]
|
|
|
|
assert flags & 0x08000000 # CREATE_NO_WINDOW: own hidden console
|
|
assert flags & 0x200 # CREATE_NEW_PROCESS_GROUP: ignore the parent's Ctrl-C
|
|
assert flags & _CREATE_BREAKAWAY_FROM_JOB # survive Job kill-on-close
|
|
|
|
|
|
def test_start_proxy_keeps_creationflags_zero_off_windows(
|
|
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
|
) -> None:
|
|
monkeypatch.setattr(wrap_cli.sys, "platform", "linux")
|
|
|
|
kwargs = _capture_popen_kwargs(monkeypatch, tmp_path)
|
|
|
|
# POSIX detaches via setsid(); no Windows creation flags are applied.
|
|
assert kwargs["creationflags"] == 0
|
|
assert isinstance(kwargs["start_new_session"], bool)
|
|
|
|
|
|
def test_start_proxy_retries_without_breakaway_when_job_forbids_it(
|
|
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
|
) -> None:
|
|
# Force the Windows branch and supply its constants, as above.
|
|
monkeypatch.setattr(wrap_cli.sys, "platform", "win32")
|
|
monkeypatch.setattr(wrap_cli.subprocess, "CREATE_NO_WINDOW", 0x08000000, raising=False)
|
|
monkeypatch.setattr(wrap_cli.subprocess, "CREATE_NEW_PROCESS_GROUP", 0x200, raising=False)
|
|
|
|
# First spawn raises OSError (the launcher's Job forbids breakaway); the
|
|
# second must succeed without CREATE_BREAKAWAY_FROM_JOB.
|
|
seen: list[int] = []
|
|
|
|
def _flaky_popen(cmd: Any, **kwargs: Any) -> _FakeProc:
|
|
seen.append(kwargs["creationflags"])
|
|
if len(seen) == 1:
|
|
raise OSError("a process in the job cannot break away")
|
|
return _FakeProc()
|
|
|
|
monkeypatch.setattr(wrap_cli.subprocess, "Popen", _flaky_popen)
|
|
monkeypatch.setattr(wrap_cli, "_check_proxy", lambda port: True)
|
|
monkeypatch.setattr(wrap_cli.time, "sleep", lambda _seconds: None)
|
|
monkeypatch.setattr(wrap_cli, "_get_log_path", lambda port=None: tmp_path / "proxy.log")
|
|
monkeypatch.setattr(wrap_cli, "_resolve_wrap_proxy_timeout_seconds", lambda: 1)
|
|
|
|
wrap_cli._start_proxy(8787)
|
|
|
|
assert len(seen) == 2 # the call was retried exactly once
|
|
assert seen[0] & _CREATE_BREAKAWAY_FROM_JOB # first attempt requests breakaway
|
|
assert not seen[1] & _CREATE_BREAKAWAY_FROM_JOB # retry drops it
|
|
assert seen[1] & 0x08000000 # but still gets its own hidden console
|
|
assert seen[1] & 0x200 # and still gets its own process group
|
|
|
|
|
|
class _TrackingFile:
|
|
"""Records whether the parent closed its copy of the stdio log handle."""
|
|
|
|
def __init__(self) -> None:
|
|
self.closed = False
|
|
|
|
def close(self) -> None:
|
|
self.closed = True
|
|
|
|
|
|
def test_start_proxy_closes_log_when_both_spawns_fail(
|
|
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
|
) -> None:
|
|
# Both spawn attempts raise (e.g. the Job forbids breakaway AND the retry
|
|
# still fails). The parent must not leak the stdio log file handle.
|
|
monkeypatch.setattr(wrap_cli.sys, "platform", "win32")
|
|
monkeypatch.setattr(wrap_cli.subprocess, "CREATE_NO_WINDOW", 0x08000000, raising=False)
|
|
monkeypatch.setattr(wrap_cli.subprocess, "CREATE_NEW_PROCESS_GROUP", 0x200, raising=False)
|
|
|
|
log_file = _TrackingFile()
|
|
monkeypatch.setattr(wrap_cli, "open", lambda *a, **k: log_file, raising=False)
|
|
|
|
def _always_fails(cmd: Any, **kwargs: Any) -> _FakeProc:
|
|
raise OSError("spawn failed on both attempts")
|
|
|
|
monkeypatch.setattr(wrap_cli.subprocess, "Popen", _always_fails)
|
|
monkeypatch.setattr(wrap_cli.time, "sleep", lambda _seconds: None)
|
|
monkeypatch.setattr(wrap_cli, "_get_log_path", lambda port=None: tmp_path / "proxy.log")
|
|
monkeypatch.setattr(wrap_cli, "_resolve_wrap_proxy_timeout_seconds", lambda: 1)
|
|
|
|
with pytest.raises(OSError):
|
|
wrap_cli._start_proxy(8787)
|
|
|
|
assert log_file.closed # finally closed the handle despite the failure
|