mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): let HEADROOM_LOG_LEVEL=debug turn on Headroom's own debug logs (#4137)
## Description
You couldn't turn on Headroom's own debug logs in a running proxy.
`HEADROOM_LOG_LEVEL` set only uvicorn's level. `_setup_file_logging`
pinned the `headroom` logger and the `proxy.log` handler to INFO, so no
`headroom.*` DEBUG line was ever emitted. Meanwhile
`docs/content/docs/metrics.mdx:320` tells operators to run
`HEADROOM_LOG_LEVEL=debug headroom proxy` to diagnose problems.
Now `debug` and `trace` lower both the `headroom` logger and the
`proxy.log` handler to DEBUG. Every other value keeps INFO. The default,
`warning`, only quiets uvicorn, so Headroom's INFO events (PERF lines
and the like) are still written as before.
**Debug lines stay in the owner-only file.** Some Headroom debug lines
carry request-derived data, such as error messages that quote a request.
With `HEADROOM_DEBUG_DUMP=full`, they also include the tool output the
router compresses. Those lines had never been reachable before. Without
a guard they would also propagate to the root handlers, meaning stdout,
container logs and wrap's stdio capture. A filter on the root handlers
drops `headroom.*` records below INFO, so debug reaches only `proxy.log`
(mode 0600), and stdout shows exactly what it showed before. Turning
debug on logs one startup WARNING that names the file and says what it
now records. A same-port re-setup also lowers the reused handler's
level, so switching from info to debug takes effect.
Refs #4136
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/helpers.py`: `_headroom_log_level()` (DEBUG for
`debug`/`trace`, otherwise INFO) sets the `headroom` logger and the
`proxy.log` handler, including a reused handler. When debug is on, the
`_HeadroomDebugStaysInProxyLog` filter goes on the root handlers, plus
one startup warning.
- `tests/test_perf_per_port_logs.py`: three regression tests. (1)
Parametrized: debug lines and the startup warning reach `proxy.log` only
for `debug`/`TRACE`. (2) A `headroom.*` debug line reaches proxy.log but
not a root stdout handler, while INFO and third-party warnings still do.
(3) Same-port re-setup from info to debug. All four checks fail on the
previous commit.
- `headroom/transforms/content_router.py`: `_log_router_debug`, which
dumps the tool output being routed, now also needs the explicit content
opt-in `HEADROOM_DEBUG_DUMP=full`. `HEADROOM_LOG_LEVEL=debug` alone no
longer writes that content. The startup warning says so.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest -q tests/test_perf_per_port_logs.py -k log_level # on origin/main code
FAILED ...[debug-True] FAILED ...[TRACE-True] 2 failed, 3 passed
$ pytest -q tests/test_perf_per_port_logs.py tests/test_cache_breakpoint_diagnostics.py tests/test_pr208_changes.py tests/test_direct_chat_ccr_resolution.py <every test mentioning HEADROOM_LOG_LEVEL>
125 passed, 1 skipped
$ pytest -q tests/test_perf_per_port_logs.py # 18 passed; on the previous commit: 4 failed
$ pytest -q <40 test files that use caplog on headroom loggers> # 956 passed, 1 skipped
$ ruff check / ruff format --check / mypy headroom/proxy/helpers.py
All checks passed! / 2 files already formatted / Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: macOS, Python 3.12. This branch runs as a real `headroom
proxy` on a random port with a temp HOME and workspace, offline, against
a local fake Anthropic/OpenAI upstream (the shared smoke harness).
- Exact command / steps: `HEADROOM_LOG_LEVEL=debug smoke.sh <worktree>`:
one `POST /v1/messages` and one `POST /v1/chat/completions`, each with a
400-row tool result. Then compared the proxy's stdout with
`proxy-<port>.log`.
- Observed result: at head 2f43785a with `HEADROOM_LOG_LEVEL=debug`,
every run returns HTTP 200 for both requests with 13574 tokens saved
(99.82%), and proxy.log is mode 600.
- **Without** `HEADROOM_DEBUG_DUMP`: proxy.log has 19 Headroom DEBUG
lines and **0** lines with the tool-output marker `item-399`.
- **With** `HEADROOM_DEBUG_DUMP=full`: 25 DEBUG lines and 6 marker
lines.
- stdout has 0 DEBUG lines and 0 marker lines in both runs.
- The startup warning appears in both. With `HEADROOM_LOG_LEVEL` unset,
proxy.log has no DEBUG lines and no warning.
- Not tested: Windows (owner-only perms there follow the existing
`fileperms` path), and multi-worker mode.
## Runtime Rollout Safety
- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A; logging only.
- Stable/default behavior changed: No. The default and every non-debug
value keep today's INFO level; only `debug`/`trace` change, and they
were documented to do this.
- Kill switch / disable path: Unset `HEADROOM_LOG_LEVEL`, or set it to
anything other than `debug`/`trace`.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A
## Additional Notes
Depends on #4138 (shared `headroom.log_safety` helpers). Its files show
in this diff until it merges.
Part of maintainer-requested cleanup #4136. Many Headroom failure paths
log at DEBUG, so before this change nobody running the proxy could see
them. Debug output can include request-derived data, and with
`HEADROOM_DEBUG_DUMP=full` the tool output the router compresses. It
therefore goes only to the owner-only proxy.log, never to stdout. The
`proxy.mdx` knob table will say this in the separate docs PR.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JerrettDavis
parent
7dc20b6fe0
commit
00093ceddc
@@ -1882,6 +1882,54 @@ def _warn_once_if_owner_only_unsupported(log_path: Path) -> None:
|
||||
)
|
||||
|
||||
|
||||
def _headroom_log_level() -> int:
|
||||
"""Level for the ``headroom`` logger and the proxy.log handler.
|
||||
|
||||
``HEADROOM_LOG_LEVEL=debug`` (or ``trace``) turns on Headroom's own debug
|
||||
lines as well as uvicorn's; anything else keeps the INFO default, so the
|
||||
default (``warning``, which only quiets uvicorn) never hides Headroom's
|
||||
INFO events.
|
||||
"""
|
||||
raw = os.environ.get("HEADROOM_LOG_LEVEL", "").strip().lower()
|
||||
return logging.DEBUG if raw in ("debug", "trace") else logging.INFO
|
||||
|
||||
|
||||
class _HeadroomDebugStaysInProxyLog(logging.Filter):
|
||||
"""Drop ``headroom.*`` DEBUG records on every handler except proxy.log.
|
||||
|
||||
Headroom's debug lines can carry request-derived data (error messages that
|
||||
quote a request; with HEADROOM_DEBUG_DUMP=full, the tool output the router
|
||||
compresses). proxy.log is owner-only; stdout, container logs
|
||||
and wrap's stdio capture are not, so debug stays out of them, exactly as
|
||||
when debug was unreachable.
|
||||
"""
|
||||
|
||||
def filter(self, record: logging.LogRecord) -> bool:
|
||||
if record.levelno >= logging.INFO:
|
||||
return True
|
||||
return record.name != "headroom" and not record.name.startswith("headroom.")
|
||||
|
||||
|
||||
def _keep_headroom_debug_out_of_other_handlers() -> None:
|
||||
"""Filter every handler that can see ``headroom.*`` records except proxy.log.
|
||||
|
||||
That is the root handlers, plus any handler already attached to ``headroom``
|
||||
or a child logger (e.g. an extension's). Records stop at proxy.log only.
|
||||
"""
|
||||
loggers: list[logging.Logger] = [logging.getLogger(), logging.getLogger("headroom")]
|
||||
loggers += [
|
||||
candidate
|
||||
for name, candidate in logging.Logger.manager.loggerDict.items()
|
||||
if name.startswith("headroom.") and isinstance(candidate, logging.Logger)
|
||||
]
|
||||
for each in loggers:
|
||||
for handler in each.handlers:
|
||||
if handler.get_name() == _PROXY_LOG_HANDLER_NAME:
|
||||
continue
|
||||
if not any(isinstance(f, _HeadroomDebugStaysInProxyLog) for f in handler.filters):
|
||||
handler.addFilter(_HeadroomDebugStaysInProxyLog())
|
||||
|
||||
|
||||
def _setup_file_logging(
|
||||
port: int | None = None,
|
||||
*,
|
||||
@@ -1928,7 +1976,8 @@ def _setup_file_logging(
|
||||
# Keep root propagation enabled for container stdout/stderr while
|
||||
# this handler writes the separate port/worker-specific proxy log.
|
||||
headroom_logger = logging.getLogger("headroom")
|
||||
headroom_logger.setLevel(logging.INFO)
|
||||
level = _headroom_log_level()
|
||||
headroom_logger.setLevel(level)
|
||||
# Decide BEFORE constructing the handler: constructing a
|
||||
# RotatingFileHandler opens (creates) the file, so building one only to
|
||||
# discard it would leave an empty stray worker log and leak
|
||||
@@ -1940,7 +1989,12 @@ def _setup_file_logging(
|
||||
for h in headroom_logger.handlers
|
||||
if isinstance(h, RotatingFileHandler) and h.name == _PROXY_LOG_HANDLER_NAME
|
||||
]
|
||||
if any(Path(h.baseFilename) == log_path for h in existing):
|
||||
if level == logging.DEBUG:
|
||||
_keep_headroom_debug_out_of_other_handlers()
|
||||
reused = [h for h in existing if Path(h.baseFilename) == log_path]
|
||||
if reused:
|
||||
for h in reused:
|
||||
h.setLevel(level)
|
||||
return
|
||||
handler = handler_cls(
|
||||
log_path,
|
||||
@@ -1949,7 +2003,7 @@ def _setup_file_logging(
|
||||
encoding="utf-8",
|
||||
)
|
||||
handler.set_name(_PROXY_LOG_HANDLER_NAME)
|
||||
handler.setLevel(logging.INFO)
|
||||
handler.setLevel(level)
|
||||
handler.setFormatter(
|
||||
logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
|
||||
)
|
||||
@@ -1957,6 +2011,15 @@ def _setup_file_logging(
|
||||
headroom_logger.removeHandler(stale)
|
||||
stale.close()
|
||||
headroom_logger.addHandler(handler)
|
||||
if level == logging.DEBUG:
|
||||
logger.warning(
|
||||
"Headroom debug logging is on (HEADROOM_LOG_LEVEL). %s now records "
|
||||
"request-derived data such as error messages that can quote requests; the "
|
||||
"file is owner-only and debug lines stay out of stdout. Full tool-output "
|
||||
"dumps also need HEADROOM_DEBUG_DUMP=full. Unset HEADROOM_LOG_LEVEL when "
|
||||
"you finish diagnosing.",
|
||||
log_path,
|
||||
)
|
||||
except OSError as exc:
|
||||
# Non-fatal: can't write logs (read-only fs, permissions, etc.). Say so,
|
||||
# or the missing proxy.log is the only clue. The OSError names the path.
|
||||
|
||||
@@ -63,6 +63,7 @@ from ..config import (
|
||||
is_tool_excluded,
|
||||
unwrap_tool_call,
|
||||
)
|
||||
from ..log_safety import content_logging_enabled
|
||||
from ..offline import OfflineEgressBlocked
|
||||
from ..parser import CCR_RETRIEVAL_MARKER_RE
|
||||
from ..tokenizer import Tokenizer
|
||||
@@ -953,8 +954,14 @@ def _bash_command_is_search(command: str, search_commands: frozenset[str]) -> bo
|
||||
return prog in search_commands
|
||||
|
||||
|
||||
def _router_debug_enabled() -> bool:
|
||||
"""Router debug events carry the tool output being routed, so DEBUG alone is
|
||||
not enough: they also need the explicit content opt-in."""
|
||||
return logger.isEnabledFor(logging.DEBUG) and content_logging_enabled()
|
||||
|
||||
|
||||
def _log_router_debug(event: str, **payload: Any) -> None:
|
||||
if not logger.isEnabledFor(logging.DEBUG):
|
||||
if not _router_debug_enabled():
|
||||
return
|
||||
payload = {"event": event, **payload}
|
||||
logger.debug("event=%s %s", event, _router_debug_dumps(payload))
|
||||
@@ -2886,7 +2893,7 @@ class ContentRouter(Transform):
|
||||
RouterCompressionResult with compressed content and routing metadata.
|
||||
"""
|
||||
context = context or ""
|
||||
debug_enabled = logger.isEnabledFor(logging.DEBUG)
|
||||
debug_enabled = _router_debug_enabled()
|
||||
request_debug = (
|
||||
{
|
||||
"chars": len(content),
|
||||
@@ -3184,7 +3191,7 @@ class ContentRouter(Transform):
|
||||
sections_source,
|
||||
isolate=tuple(placeholder for placeholder, _ in protected),
|
||||
)
|
||||
if logger.isEnabledFor(logging.DEBUG):
|
||||
if _router_debug_enabled():
|
||||
_log_router_debug(
|
||||
"content_router_mixed_sections",
|
||||
section_count=len(sections),
|
||||
@@ -4406,7 +4413,7 @@ class ContentRouter(Transform):
|
||||
compressed, compressed_tokens = dense
|
||||
strategy_chain.append("dense_elide")
|
||||
decision_reason = f"{decision_reason}_dense_elide"
|
||||
if logger.isEnabledFor(logging.DEBUG):
|
||||
if _router_debug_enabled():
|
||||
_log_router_debug(
|
||||
"content_router_strategy_result",
|
||||
requested_strategy=requested_strategy.value,
|
||||
@@ -4466,7 +4473,7 @@ class ContentRouter(Transform):
|
||||
)
|
||||
return elided, elided_tokens, strategy_chain
|
||||
strategy_chain.append(CompressionStrategy.PASSTHROUGH.value)
|
||||
if logger.isEnabledFor(logging.DEBUG):
|
||||
if _router_debug_enabled():
|
||||
_log_router_debug(
|
||||
"content_router_strategy_result",
|
||||
requested_strategy=requested_strategy.value,
|
||||
|
||||
@@ -253,3 +253,151 @@ def test_parse_log_files_aggregates_per_port_and_legacy(workspace: Path) -> None
|
||||
# Non-PERF files never ingested.
|
||||
assert models.isdisjoint({"model-STDIO", "model-STDIO2", "model-ERR"})
|
||||
assert rids.isdisjoint({"hr_stdio", "hr_stdio2", "hr_err"})
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def headroom_log_state(request: pytest.FixtureRequest):
|
||||
"""Restore the headroom logger level and drop the debug filter from root handlers."""
|
||||
from headroom.proxy import helpers
|
||||
|
||||
headroom_logger = logging.getLogger("headroom")
|
||||
level = headroom_logger.level
|
||||
|
||||
def restore() -> None:
|
||||
headroom_logger.setLevel(level)
|
||||
loggers = [logging.getLogger(), headroom_logger] + [
|
||||
lg
|
||||
for name, lg in logging.Logger.manager.loggerDict.items()
|
||||
if name.startswith("headroom.") and isinstance(lg, logging.Logger)
|
||||
]
|
||||
for handler in [h for lg in loggers for h in lg.handlers]:
|
||||
for f in list(handler.filters):
|
||||
if isinstance(f, helpers._HeadroomDebugStaysInProxyLog):
|
||||
handler.removeFilter(f)
|
||||
|
||||
request.addfinalizer(restore)
|
||||
return headroom_logger
|
||||
|
||||
|
||||
def _flush(headroom_logger: logging.Logger) -> None:
|
||||
for handler in headroom_logger.handlers:
|
||||
handler.flush()
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("env", "debug_written"),
|
||||
[("debug", True), ("TRACE", True), ("warning", False), ("info", False), (None, False)],
|
||||
)
|
||||
def test_headroom_log_level_debug_reaches_proxy_log(
|
||||
workspace: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
headroom_log_state: logging.Logger,
|
||||
env: str | None,
|
||||
debug_written: bool,
|
||||
) -> None:
|
||||
"""``HEADROOM_LOG_LEVEL=debug`` must surface Headroom's own debug lines, not only uvicorn's."""
|
||||
if env is None:
|
||||
monkeypatch.delenv("HEADROOM_LOG_LEVEL", raising=False)
|
||||
else:
|
||||
monkeypatch.setenv("HEADROOM_LOG_LEVEL", env)
|
||||
|
||||
_setup_file_logging(port=9911)
|
||||
probe = logging.getLogger("headroom.proxy.log_level_probe")
|
||||
probe.debug("debug-probe")
|
||||
probe.info("info-probe")
|
||||
_flush(headroom_log_state)
|
||||
|
||||
text = _paths.proxy_log_path(9911).read_text()
|
||||
assert "info-probe" in text
|
||||
assert ("debug-probe" in text) is debug_written
|
||||
assert ("Headroom debug logging is on" in text) is debug_written
|
||||
|
||||
|
||||
def test_headroom_debug_stays_out_of_stdout(
|
||||
workspace: Path, monkeypatch: pytest.MonkeyPatch, headroom_log_state: logging.Logger
|
||||
) -> None:
|
||||
"""Debug lines can carry tool output: only the owner-only proxy.log may receive them."""
|
||||
import io
|
||||
|
||||
stdout = io.StringIO()
|
||||
root_handler = logging.StreamHandler(stdout)
|
||||
logging.getLogger().addHandler(root_handler)
|
||||
try:
|
||||
monkeypatch.setenv("HEADROOM_LOG_LEVEL", "debug")
|
||||
_setup_file_logging(port=9912)
|
||||
probe = logging.getLogger("headroom.transforms.log_level_probe")
|
||||
probe.debug("secret-tool-output")
|
||||
probe.info("info-probe")
|
||||
logging.getLogger("thirdparty.log_level_probe").warning("thirdparty-warning")
|
||||
_flush(headroom_log_state)
|
||||
finally:
|
||||
logging.getLogger().removeHandler(root_handler)
|
||||
|
||||
assert "secret-tool-output" in _paths.proxy_log_path(9912).read_text()
|
||||
assert "secret-tool-output" not in stdout.getvalue()
|
||||
assert "info-probe" in stdout.getvalue()
|
||||
assert "thirdparty-warning" in stdout.getvalue()
|
||||
|
||||
|
||||
def test_reused_proxy_log_picks_up_debug(
|
||||
workspace: Path, monkeypatch: pytest.MonkeyPatch, headroom_log_state: logging.Logger
|
||||
) -> None:
|
||||
"""A second same-port setup after switching to debug must not keep the old INFO handler."""
|
||||
monkeypatch.setenv("HEADROOM_LOG_LEVEL", "info")
|
||||
_setup_file_logging(port=9913)
|
||||
monkeypatch.setenv("HEADROOM_LOG_LEVEL", "debug")
|
||||
_setup_file_logging(port=9913)
|
||||
|
||||
logging.getLogger("headroom.proxy.log_level_probe").debug("debug-after-reuse")
|
||||
_flush(headroom_log_state)
|
||||
|
||||
assert "debug-after-reuse" in _paths.proxy_log_path(9913).read_text()
|
||||
|
||||
|
||||
def test_headroom_debug_stays_out_of_child_logger_handlers(
|
||||
workspace: Path, monkeypatch: pytest.MonkeyPatch, headroom_log_state: logging.Logger
|
||||
) -> None:
|
||||
"""A handler on a headroom.* logger (e.g. an extension's) must not receive debug lines either."""
|
||||
import io
|
||||
|
||||
stream = io.StringIO()
|
||||
child = logging.getLogger("headroom.extension_probe")
|
||||
child_handler = logging.StreamHandler(stream)
|
||||
child.addHandler(child_handler)
|
||||
try:
|
||||
monkeypatch.setenv("HEADROOM_LOG_LEVEL", "debug")
|
||||
_setup_file_logging(port=9914)
|
||||
child.debug("secret-tool-output")
|
||||
child.info("info-probe")
|
||||
_flush(headroom_log_state)
|
||||
finally:
|
||||
child.removeHandler(child_handler)
|
||||
|
||||
assert "secret-tool-output" in _paths.proxy_log_path(9914).read_text()
|
||||
assert "secret-tool-output" not in stream.getvalue()
|
||||
assert "info-probe" in stream.getvalue()
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("dump", "logged"), [("", False), ("full", True)])
|
||||
def test_router_content_dump_needs_the_content_opt_in(
|
||||
monkeypatch: pytest.MonkeyPatch, headroom_log_state: logging.Logger, dump: str, logged: bool
|
||||
) -> None:
|
||||
"""HEADROOM_LOG_LEVEL=debug alone must not write the tool output the router compresses."""
|
||||
import io
|
||||
|
||||
from headroom.transforms import content_router
|
||||
|
||||
monkeypatch.setenv("HEADROOM_DEBUG_DUMP", dump)
|
||||
stream = io.StringIO()
|
||||
handler = logging.StreamHandler(stream)
|
||||
router_logger = logging.getLogger(content_router.__name__)
|
||||
previous = router_logger.level
|
||||
router_logger.addHandler(handler)
|
||||
router_logger.setLevel(logging.DEBUG)
|
||||
try:
|
||||
content_router._log_router_debug("routing_decision", content="secret-tool-output")
|
||||
finally:
|
||||
router_logger.removeHandler(handler)
|
||||
router_logger.setLevel(previous)
|
||||
|
||||
assert ("secret-tool-output" in stream.getvalue()) is logged
|
||||
|
||||
Reference in New Issue
Block a user