mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): stop headroom logger from suppressing propagation to stdout/stderr (#3096)
## Description `_setup_file_logging()` in `headroom/proxy/helpers.py` unconditionally set `headroom_logger.propagate = False`, on the theory that `wrap.py` redirects stderr into the same `proxy.log` file the rotating handler writes to, so propagation would double-write records. That premise is wrong: `wrap.py` redirects the proxy subprocess's stdout/stderr to a separate sibling file, `proxy-stdio.log` (see `_get_proxy_stdio_log_path()`), never `proxy.log`. Severing propagation bought no deduplication — it only stopped every `headroom.*` log record from reaching root's `logging.basicConfig()` handler (`proxy/server.py`), which is what writes to stderr/stdout. In containerized deployments (Docker/Kubernetes), stdout/stderr is the only channel the platform collects, so this silently dropped all application logs. Closes #3087 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Removed `headroom_logger.propagate = False` from `_setup_file_logging()` in `headroom/proxy/helpers.py`. - Updated the surrounding comment to explain why propagation is safe to leave on: `proxy.log` (file handler) and stderr/stdout (root's `basicConfig` handler) are two distinct destinations, so nothing is duplicated in either file. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check . All checks passed! $ pytest tests/test_corrupt_golden_bytes_recovery.py -q collected 9 items tests/test_corrupt_golden_bytes_recovery.py ......... [100%] 9 passed in 0.47s ``` `mypy headroom` was not run to a clean result: it fails before reaching this file with a pre-existing, unrelated error in the installed `numpy` stub (`Type statement is only supported in Python 3.12 and greater`) in this sandbox's dependency set. Not caused by this change (a 4-line pure-Python logging tweak). The broader `pytest tests/` collection also hits ~170 pre-existing, unrelated errors in this sandbox: `headroom._core`, the compiled Rust extension, isn't built here (no `cargo`/`maturin` available), so every module importing it fails at collection. This change doesn't touch Rust code or `_core` and isn't implicated. No new tests added: this restores an existing, already-tested code path (see `tests/conftest.py`'s autouse fixture and `tests/test_corrupt_golden_bytes_recovery.py`, both of which already assert/require `propagate=True` for `caplog` to work) to its intended default rather than adding new behavior. ## Real Behavior Proof - Environment: local sandbox, Python 3.12.3, unable to build/import `headroom.proxy.server` directly (missing compiled `headroom._core`). Verified with an isolated script that copies `_setup_file_logging()`'s exact logic byte-for-byte plus the same `logging.basicConfig()` call `proxy/server.py` makes at import time. - Exact command / steps: ran the fixed logic, then emitted `logging.getLogger("headroom.proxy").info("compression timeout: request abc123 exceeded 30s budget")`. - Observed result: the `headroom.proxy` record was written to stderr AND to `proxy.log` simultaneously (full output below); re-running the same script with the old `headroom_logger.propagate = False` line restored reproduces the bug — the record disappears from stderr entirely and only lands in `proxy.log`. ```text propagate flag on 'headroom' logger: True --- emitting a headroom.proxy log record, watch for it below on stderr --- 2026-08-17 21:17:39,545 - headroom.proxy - INFO - compression timeout: request abc123 exceeded 30s budget --- also present in proxy.log (file handler): OK --- ``` - Not tested: end-to-end through the actual compiled proxy binary/server process (blocked by missing `headroom._core` in this sandbox) and real container log collection (Docker/Kubernetes) — the isolated repro exercises the exact same logging configuration code the real process runs, so this is considered sufficient proxy proof for a pure logging-config change. ## Runtime Rollout Safety - Rollout-managed feature(s): None — this is unconditional startup logging configuration, not behind a rollout flag. - Minimum rollout channel: N/A - Stable/default behavior changed: Yes — `headroom.*` log records now also reach stderr/stdout (previously file-only). No change to request/response handling, compression, or any user-facing proxy behavior. - Kill switch / disable path: None needed; this restores default Python logging propagation. If a deployment truly wants file-only logging, `HEADROOM_LOG_TO_STDOUT`-style suppression could be added later, but no such request exists today. - Unsafe override required: No - Qualification impact: None - Rollback path: Revert this commit to restore `propagate = False`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — logging configuration change, no UI impact. ## Additional Notes - "New and existing unit tests pass locally" is left unchecked because the full `pytest tests/` collection can't complete in this sandbox (missing compiled `headroom._core` Rust extension, unrelated to this change) — see Test Output above for what was actually run and passed. - "I have added tests" is left unchecked: existing tests (`tests/conftest.py` autouse fixture, `tests/test_corrupt_golden_bytes_recovery.py`) already assert the code path this restores; happy to add a focused regression test asserting `logging.getLogger("headroom").propagate is True` after `_setup_file_logging()` if maintainers want one. --------- Co-authored-by: = <=> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
=
JerrettDavis
parent
8f3d677354
commit
f78e66f7cc
@@ -1845,11 +1845,10 @@ def _setup_file_logging(
|
||||
return
|
||||
_warn_once_if_owner_only_unsupported(log_path)
|
||||
# Attach to the headroom root logger so all sub-loggers are captured.
|
||||
# Disable propagation to root to avoid duplicate writes when
|
||||
# wrap.py redirects stderr to the same log file.
|
||||
# Keep root propagation enabled for container stdout/stderr while
|
||||
# this handler writes the separate port/worker-specific proxy log.
|
||||
headroom_logger = logging.getLogger("headroom")
|
||||
headroom_logger.setLevel(logging.INFO)
|
||||
headroom_logger.propagate = False
|
||||
# Decide BEFORE constructing the handler: constructing a
|
||||
# RotatingFileHandler opens (creates) the file, so building one only to
|
||||
# discard it would leave an empty stray worker log and leak
|
||||
|
||||
+2
-4
@@ -250,13 +250,11 @@ def _null_binary_pins():
|
||||
def _reset_headroom_logger_propagation():
|
||||
"""Keep `headroom.*` log records flowing to pytest's caplog handler.
|
||||
|
||||
Two sources disable propagation on the headroom logger tree and never
|
||||
restore it, which then makes later `caplog`-based assertions flaky in
|
||||
A benchmark helper disables propagation on the headroom logger tree and
|
||||
never restores it, which then makes later `caplog`-based assertions flaky in
|
||||
full-suite runs (caplog attaches to root, so a `propagate=False` anywhere
|
||||
on the chain silently drops the records):
|
||||
|
||||
- ``headroom.proxy.helpers._setup_file_logging`` sets
|
||||
``getLogger("headroom").propagate = False`` on proxy startup.
|
||||
- ``benchmarks.claude_session_mode_benchmark._disable_headroom_benchmark_logging``
|
||||
(exercised by ``test_claude_session_mode_benchmark``) sets
|
||||
``propagate = False`` + ``CRITICAL`` on ``headroom``, ``headroom.proxy``,
|
||||
|
||||
@@ -61,14 +61,7 @@ def _find_perf_record(records: list[logging.LogRecord]) -> tuple[int, int, int]:
|
||||
|
||||
|
||||
class _ListHandler(logging.Handler):
|
||||
"""Tiny direct handler that survives the proxy disabling propagation.
|
||||
|
||||
``caplog`` attaches to root; ``headroom.proxy.helpers._setup_file_logging``
|
||||
flips ``logging.getLogger("headroom").propagate = False`` once a proxy
|
||||
instance is constructed in the test, after which root-attached handlers
|
||||
stop receiving headroom-namespaced records. Attaching directly to
|
||||
``headroom.proxy`` sidesteps that.
|
||||
"""
|
||||
"""Tiny direct handler immune to unrelated propagation mutations."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__(level=logging.INFO)
|
||||
|
||||
@@ -57,14 +57,7 @@ def _find_perf_record(records: list[logging.LogRecord]) -> tuple[int, int, int]:
|
||||
|
||||
|
||||
class _ListHandler(logging.Handler):
|
||||
"""Tiny direct handler that survives the proxy disabling propagation.
|
||||
|
||||
``caplog`` attaches to root; ``headroom.proxy.helpers._setup_file_logging``
|
||||
flips ``logging.getLogger("headroom").propagate = False`` once a proxy
|
||||
instance is constructed in the test, after which root-attached handlers
|
||||
stop receiving headroom-namespaced records. Attaching directly to
|
||||
``headroom.proxy`` sidesteps that.
|
||||
"""
|
||||
"""Tiny direct handler immune to unrelated propagation mutations."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__(level=logging.INFO)
|
||||
|
||||
@@ -81,6 +81,43 @@ def test_setup_file_logging_targets_per_port_file(workspace: Path) -> None:
|
||||
logger.addHandler(h)
|
||||
|
||||
|
||||
def test_setup_file_logging_keeps_root_propagation_and_writes_once_per_sink(
|
||||
workspace: Path,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
logger = logging.getLogger("headroom")
|
||||
original_handlers = list(logger.handlers)
|
||||
original_propagate = logger.propagate
|
||||
for handler in original_handlers:
|
||||
logger.removeHandler(handler)
|
||||
|
||||
try:
|
||||
logger.propagate = True
|
||||
with caplog.at_level(logging.INFO):
|
||||
_setup_file_logging(8888)
|
||||
logger.info("propagation-regression-record")
|
||||
|
||||
for handler in logger.handlers:
|
||||
handler.flush()
|
||||
|
||||
propagated = [
|
||||
record
|
||||
for record in caplog.records
|
||||
if record.getMessage() == "propagation-regression-record"
|
||||
]
|
||||
file_text = (workspace / "logs" / "proxy-8888.log").read_text()
|
||||
assert len(propagated) == 1
|
||||
assert file_text.count("propagation-regression-record") == 1
|
||||
finally:
|
||||
for handler in list(logger.handlers):
|
||||
if isinstance(handler, RotatingFileHandler):
|
||||
handler.close()
|
||||
logger.removeHandler(handler)
|
||||
logger.propagate = original_propagate
|
||||
for handler in original_handlers:
|
||||
logger.addHandler(handler)
|
||||
|
||||
|
||||
def test_setup_file_logging_targets_worker_file(workspace: Path) -> None:
|
||||
logger = logging.getLogger("headroom")
|
||||
original = list(logger.handlers)
|
||||
|
||||
@@ -141,8 +141,8 @@ async def test_startup_reports_deferred_kompress(caplog):
|
||||
proxy.openai_pipeline = _FakePipeline([])
|
||||
|
||||
try:
|
||||
# Proxy setup disables propagation on the ``headroom`` logger, so
|
||||
# attach caplog's handler directly to the logger that emits this line.
|
||||
# Attach caplog directly so unrelated propagation mutations cannot
|
||||
# affect this assertion.
|
||||
server_mod.logger.addHandler(caplog.handler)
|
||||
try:
|
||||
with caplog.at_level(logging.INFO, logger=server_mod.logger.name):
|
||||
|
||||
@@ -59,9 +59,8 @@ class _CapturingHandler(logging.Handler):
|
||||
def proxy_log_capture():
|
||||
"""Capture ``headroom.proxy`` records.
|
||||
|
||||
``_setup_file_logging`` sets ``propagate = False`` on this logger, so
|
||||
``caplog`` (which hangs off the root) never sees them — the same reason
|
||||
``tests/test_anthropic_stage_timings.py`` attaches its own handler.
|
||||
A direct handler keeps this fixture isolated from unrelated propagation
|
||||
mutations elsewhere in the suite.
|
||||
"""
|
||||
target = logging.getLogger("headroom.proxy")
|
||||
handler = _CapturingHandler()
|
||||
|
||||
@@ -417,8 +417,8 @@ async def test_funnel_emits_perf_log_with_canonical_shape(
|
||||
tok_after, tok_saved, cache_read, cache_write, cache_hit_pct,
|
||||
opt_ms, transforms — in that order, space-separated."""
|
||||
h = _FunnelHarness()
|
||||
# Direct handler attach: caplog otherwise drops propagation-disabled
|
||||
# records (the proxy disables ``headroom.*`` propagation once started).
|
||||
# Direct handler attach keeps capture isolated from unrelated propagation
|
||||
# mutations elsewhere in the suite.
|
||||
target = logging.getLogger("headroom.proxy")
|
||||
captured: list[logging.LogRecord] = []
|
||||
|
||||
|
||||
Reference in New Issue
Block a user