fix(proxy): stop headroom logger from suppressing propagation to stdout/stderr (#3096)

## Description

`_setup_file_logging()` in `headroom/proxy/helpers.py` unconditionally
set `headroom_logger.propagate = False`, on the theory that `wrap.py`
redirects stderr into the same `proxy.log` file the rotating handler
writes to, so propagation would double-write records. That premise is
wrong: `wrap.py` redirects the proxy subprocess's stdout/stderr to a
separate sibling file, `proxy-stdio.log` (see
`_get_proxy_stdio_log_path()`), never `proxy.log`. Severing propagation
bought no deduplication — it only stopped every `headroom.*` log record
from reaching root's `logging.basicConfig()` handler
(`proxy/server.py`), which is what writes to stderr/stdout. In
containerized deployments (Docker/Kubernetes), stdout/stderr is the only
channel the platform collects, so this silently dropped all application
logs.

Closes #3087

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Removed `headroom_logger.propagate = False` from
`_setup_file_logging()` in `headroom/proxy/helpers.py`.
- Updated the surrounding comment to explain why propagation is safe to
leave on: `proxy.log` (file handler) and stderr/stdout (root's
`basicConfig` handler) are two distinct destinations, so nothing is
duplicated in either file.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ ruff check .
All checks passed!

$ pytest tests/test_corrupt_golden_bytes_recovery.py -q
collected 9 items
tests/test_corrupt_golden_bytes_recovery.py .........                    [100%]
9 passed in 0.47s
```

`mypy headroom` was not run to a clean result: it fails before reaching
this file with a pre-existing, unrelated error in the installed `numpy`
stub (`Type statement is only supported in Python 3.12 and greater`) in
this sandbox's dependency set. Not caused by this change (a 4-line
pure-Python logging tweak).

The broader `pytest tests/` collection also hits ~170 pre-existing,
unrelated errors in this sandbox: `headroom._core`, the compiled Rust
extension, isn't built here (no `cargo`/`maturin` available), so every
module importing it fails at collection. This change doesn't touch Rust
code or `_core` and isn't implicated.

No new tests added: this restores an existing, already-tested code path
(see `tests/conftest.py`'s autouse fixture and
`tests/test_corrupt_golden_bytes_recovery.py`, both of which already
assert/require `propagate=True` for `caplog` to work) to its intended
default rather than adding new behavior.

## Real Behavior Proof

- Environment: local sandbox, Python 3.12.3, unable to build/import
`headroom.proxy.server` directly (missing compiled `headroom._core`).
Verified with an isolated script that copies `_setup_file_logging()`'s
exact logic byte-for-byte plus the same `logging.basicConfig()` call
`proxy/server.py` makes at import time.
- Exact command / steps: ran the fixed logic, then emitted
`logging.getLogger("headroom.proxy").info("compression timeout: request
abc123 exceeded 30s budget")`.
- Observed result: the `headroom.proxy` record was written to stderr AND
to `proxy.log` simultaneously (full output below); re-running the same
script with the old `headroom_logger.propagate = False` line restored
reproduces the bug — the record disappears from stderr entirely and only
lands in `proxy.log`.
  ```text
  propagate flag on 'headroom' logger: True
--- emitting a headroom.proxy log record, watch for it below on stderr
---
2026-08-17 21:17:39,545 - headroom.proxy - INFO - compression timeout:
request abc123 exceeded 30s budget
  --- also present in proxy.log (file handler): OK ---
  ```
- Not tested: end-to-end through the actual compiled proxy binary/server
process (blocked by missing `headroom._core` in this sandbox) and real
container log collection (Docker/Kubernetes) — the isolated repro
exercises the exact same logging configuration code the real process
runs, so this is considered sufficient proxy proof for a pure
logging-config change.

## Runtime Rollout Safety

- Rollout-managed feature(s): None — this is unconditional startup
logging configuration, not behind a rollout flag.
- Minimum rollout channel: N/A
- Stable/default behavior changed: Yes — `headroom.*` log records now
also reach stderr/stdout (previously file-only). No change to
request/response handling, compression, or any user-facing proxy
behavior.
- Kill switch / disable path: None needed; this restores default Python
logging propagation. If a deployment truly wants file-only logging,
`HEADROOM_LOG_TO_STDOUT`-style suppression could be added later, but no
such request exists today.
- Unsafe override required: No
- Qualification impact: None
- Rollback path: Revert this commit to restore `propagate = False`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A — logging configuration change, no UI impact.

## Additional Notes

- "New and existing unit tests pass locally" is left unchecked because
the full `pytest tests/` collection can't complete in this sandbox
(missing compiled `headroom._core` Rust extension, unrelated to this
change) — see Test Output above for what was actually run and passed.
- "I have added tests" is left unchecked: existing tests
(`tests/conftest.py` autouse fixture,
`tests/test_corrupt_golden_bytes_recovery.py`) already assert the code
path this restores; happy to add a focused regression test asserting
`logging.getLogger("headroom").propagate is True` after
`_setup_file_logging()` if maintainers want one.

---------

Co-authored-by: = <=>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Sudhindra Desai
2026-10-01 00:03:53 -05:00
committed by GitHub
co-authored by = JerrettDavis
parent 8f3d677354
commit f78e66f7cc
8 changed files with 49 additions and 30 deletions
+2 -3
View File
@@ -1845,11 +1845,10 @@ def _setup_file_logging(
return
_warn_once_if_owner_only_unsupported(log_path)
# Attach to the headroom root logger so all sub-loggers are captured.
# Disable propagation to root to avoid duplicate writes when
# wrap.py redirects stderr to the same log file.
# Keep root propagation enabled for container stdout/stderr while
# this handler writes the separate port/worker-specific proxy log.
headroom_logger = logging.getLogger("headroom")
headroom_logger.setLevel(logging.INFO)
headroom_logger.propagate = False
# Decide BEFORE constructing the handler: constructing a
# RotatingFileHandler opens (creates) the file, so building one only to
# discard it would leave an empty stray worker log and leak
+2 -4
View File
@@ -250,13 +250,11 @@ def _null_binary_pins():
def _reset_headroom_logger_propagation():
"""Keep `headroom.*` log records flowing to pytest's caplog handler.
Two sources disable propagation on the headroom logger tree and never
restore it, which then makes later `caplog`-based assertions flaky in
A benchmark helper disables propagation on the headroom logger tree and
never restores it, which then makes later `caplog`-based assertions flaky in
full-suite runs (caplog attaches to root, so a `propagate=False` anywhere
on the chain silently drops the records):
- ``headroom.proxy.helpers._setup_file_logging`` sets
``getLogger("headroom").propagate = False`` on proxy startup.
- ``benchmarks.claude_session_mode_benchmark._disable_headroom_benchmark_logging``
(exercised by ``test_claude_session_mode_benchmark``) sets
``propagate = False`` + ``CRITICAL`` on ``headroom``, ``headroom.proxy``,
@@ -61,14 +61,7 @@ def _find_perf_record(records: list[logging.LogRecord]) -> tuple[int, int, int]:
class _ListHandler(logging.Handler):
"""Tiny direct handler that survives the proxy disabling propagation.
``caplog`` attaches to root; ``headroom.proxy.helpers._setup_file_logging``
flips ``logging.getLogger("headroom").propagate = False`` once a proxy
instance is constructed in the test, after which root-attached handlers
stop receiving headroom-namespaced records. Attaching directly to
``headroom.proxy`` sidesteps that.
"""
"""Tiny direct handler immune to unrelated propagation mutations."""
def __init__(self) -> None:
super().__init__(level=logging.INFO)
@@ -57,14 +57,7 @@ def _find_perf_record(records: list[logging.LogRecord]) -> tuple[int, int, int]:
class _ListHandler(logging.Handler):
"""Tiny direct handler that survives the proxy disabling propagation.
``caplog`` attaches to root; ``headroom.proxy.helpers._setup_file_logging``
flips ``logging.getLogger("headroom").propagate = False`` once a proxy
instance is constructed in the test, after which root-attached handlers
stop receiving headroom-namespaced records. Attaching directly to
``headroom.proxy`` sidesteps that.
"""
"""Tiny direct handler immune to unrelated propagation mutations."""
def __init__(self) -> None:
super().__init__(level=logging.INFO)
+37
View File
@@ -81,6 +81,43 @@ def test_setup_file_logging_targets_per_port_file(workspace: Path) -> None:
logger.addHandler(h)
def test_setup_file_logging_keeps_root_propagation_and_writes_once_per_sink(
workspace: Path,
caplog: pytest.LogCaptureFixture,
) -> None:
logger = logging.getLogger("headroom")
original_handlers = list(logger.handlers)
original_propagate = logger.propagate
for handler in original_handlers:
logger.removeHandler(handler)
try:
logger.propagate = True
with caplog.at_level(logging.INFO):
_setup_file_logging(8888)
logger.info("propagation-regression-record")
for handler in logger.handlers:
handler.flush()
propagated = [
record
for record in caplog.records
if record.getMessage() == "propagation-regression-record"
]
file_text = (workspace / "logs" / "proxy-8888.log").read_text()
assert len(propagated) == 1
assert file_text.count("propagation-regression-record") == 1
finally:
for handler in list(logger.handlers):
if isinstance(handler, RotatingFileHandler):
handler.close()
logger.removeHandler(handler)
logger.propagate = original_propagate
for handler in original_handlers:
logger.addHandler(handler)
def test_setup_file_logging_targets_worker_file(workspace: Path) -> None:
logger = logging.getLogger("headroom")
original = list(logger.handlers)
+2 -2
View File
@@ -141,8 +141,8 @@ async def test_startup_reports_deferred_kompress(caplog):
proxy.openai_pipeline = _FakePipeline([])
try:
# Proxy setup disables propagation on the ``headroom`` logger, so
# attach caplog's handler directly to the logger that emits this line.
# Attach caplog directly so unrelated propagation mutations cannot
# affect this assertion.
server_mod.logger.addHandler(caplog.handler)
try:
with caplog.at_level(logging.INFO, logger=server_mod.logger.name):
+2 -3
View File
@@ -59,9 +59,8 @@ class _CapturingHandler(logging.Handler):
def proxy_log_capture():
"""Capture ``headroom.proxy`` records.
``_setup_file_logging`` sets ``propagate = False`` on this logger, so
``caplog`` (which hangs off the root) never sees them — the same reason
``tests/test_anthropic_stage_timings.py`` attaches its own handler.
A direct handler keeps this fixture isolated from unrelated propagation
mutations elsewhere in the suite.
"""
target = logging.getLogger("headroom.proxy")
handler = _CapturingHandler()
+2 -2
View File
@@ -417,8 +417,8 @@ async def test_funnel_emits_perf_log_with_canonical_shape(
tok_after, tok_saved, cache_read, cache_write, cache_hit_pct,
opt_ms, transforms — in that order, space-separated."""
h = _FunnelHarness()
# Direct handler attach: caplog otherwise drops propagation-disabled
# records (the proxy disables ``headroom.*`` propagation once started).
# Direct handler attach keeps capture isolated from unrelated propagation
# mutations elsewhere in the suite.
target = logging.getLogger("headroom.proxy")
captured: list[logging.LogRecord] = []