mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): keep health probes and the Codex attestation out of the proxy log (#4611)
## Description Two things fill and leak through the always-on proxy log (`~/.headroom/logs/proxy-<port>.log`): 1. Health probes. Every `/readyz`, `/livez` and `/health` poll writes an `event=proxy_inbound_request` and an `event=proxy_inbound_response` line at INFO. A desktop wrapper or orchestrator polls these every few seconds: in an Extra Headroom user's proxy log (Codex + Claude Code, 2026-10-27) 23,201 `/readyz` probes were 19% of 54 MB, and on a lighter machine they are 48% of the file. With 10 MB x 5 rotation, that log held only 14 hours; at the busiest hour one 10 MB file lasted about an hour. 2. The Codex attestation. The inbound request line runs headers through `redact_for_wire_debug`, whose key list has no entry for `x-oai-attestation`. Codex sends that ~4 KB signed device attestation on every request, so every Codex request wrote it to the log in clear. That is 3,040 copies (12 MB) in the log above, in a file users are routinely asked to attach to bug reports. #2612 proposed the same key for the opt-in wire-debug capture and was closed for inactivity. The default proxy log uses the same policy, so this fix covers both. Health probes now log at DEBUG (both lines), and `x-oai-attestation` is a redacted key. Every other path still logs at INFO exactly as before. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/server.py`: a module-level `_HEALTH_PROBE_PATHS` (`/health`, `/livez`, `/readyz`). The inbound request/response lines use DEBUG for a GET to one of them, INFO otherwise. `_AUTH_EXEMPT_PATHS` now refers to the same set instead of repeating it. The aborted/error lines are unchanged. - `headroom/proxy/wire_debug_redaction_policy.py`: `x-oai-attestation` added to `WIRE_DEBUG_SECRET_KEYS`. - Tests: `test_health_probes_log_at_debug_and_traffic_at_info` and `test_wire_debug_redacts_codex_attestation_header`. Both fail on `main` and pass here. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest -q tests/test_wire_debug_redaction_policy.py tests/test_proxy/test_inbound_response_size.py 11 passed, 1 warning in 5.36s # every test file that mentions readyz / livez / proxy_token / inbound_request (36 files) $ uv run --frozen --extra dev pytest -q $(git grep -l -E 'readyz|livez|proxy_token|inbound_request' -- 'tests/*.py' 'tests/**/*.py') 4 failed, 1245 passed, 2 skipped # the 4 are tests/test_cli/test_wrap_persistent.py and fail identically on unmodified main on this machine $ uvx ruff check <changed files> -> All checks passed! $ uvx ruff format --check <changed files> -> 4 files already formatted $ uv run --frozen --extra dev mypy headroom/proxy/server.py headroom/proxy/wire_debug_redaction_policy.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: macOS 27.2, this branch via `uv run --frozen --extra proxy headroom proxy --port 18799 --no-optimize`, `HEADROOM_SKIP_UPSTREAM_CHECK=1`. - Exact command / steps: polled `GET /livez` until ready, then `curl /readyz`, then `curl -H 'x-oai-attestation: {"v":1,"s":1,"t":"SIGNED-DEVICE-TOKEN"}' -H 'x-codex-window-id: w:1' /stats`, then read stderr and `~/.headroom/logs/proxy-18799.log`. - Observed result: neither output has a `/livez` or `/readyz` line; the `/stats` request line shows `"x-oai-attestation": "[REDACTED]"` next to the visible `x-codex-window-id`, and `SIGNED-DEVICE-TOKEN` occurs 0 times in either output. - Not tested: a Codex client end to end; the header value is never used by the proxy, only logged. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a, logging only - Stable/default behavior changed: yes, health-probe inbound lines move from INFO to DEBUG and `x-oai-attestation` is redacted in logged headers; forwarding is unchanged - Kill switch / disable path: run with DEBUG logging to see probe lines again - Unsafe override required: no - Qualification impact: none, no request or response bytes change - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Docs: no doc describes these log lines. Credit to #2612 for the redaction key. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4611" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
affd17c5fd
commit
78736a16bc
@@ -487,6 +487,9 @@ logging.basicConfig(
|
||||
)
|
||||
logger = logging.getLogger("headroom.proxy")
|
||||
|
||||
# Liveness/readiness probe routes: auth-exempt, and logged at DEBUG.
|
||||
_HEALTH_PROBE_PATHS = frozenset({"/health", "/livez", "/readyz"})
|
||||
|
||||
|
||||
def _output_reduction_payload(shaper_active: bool, est: Any = None) -> dict[str, Any]:
|
||||
"""/stats payload for the output-shaping layer.
|
||||
@@ -4042,7 +4045,13 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
safe_headers = redact_for_wire_debug(headers)
|
||||
except Exception:
|
||||
safe_headers = {"redaction_error": True}
|
||||
logger.info(
|
||||
# A desktop or orchestrator polls these every few seconds; at INFO they
|
||||
# filled half of an idle proxy log and cut a busy one's history to hours.
|
||||
line_level = (
|
||||
logging.DEBUG if method == "GET" and path in _HEALTH_PROBE_PATHS else logging.INFO
|
||||
)
|
||||
logger.log(
|
||||
line_level,
|
||||
"event=proxy_inbound_request id=%s method=%s path=%s query=%s client=%s "
|
||||
"content_length=%s headers=%s",
|
||||
inbound_id,
|
||||
@@ -4096,7 +4105,8 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
proxy.metrics.record_inbound_response(status_code=response.status_code)
|
||||
except Exception:
|
||||
logger.debug("record_inbound_response failed", exc_info=True)
|
||||
logger.info(
|
||||
logger.log(
|
||||
line_level,
|
||||
"event=proxy_inbound_response id=%s method=%s path=%s status=%s "
|
||||
"duration_ms=%.2f response_content_length=%s",
|
||||
inbound_id,
|
||||
@@ -4140,7 +4150,7 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
# Anything else on these paths falls through to the catch-all passthrough
|
||||
# relay (FastAPI's @app.get does not register HEAD), so exempting it would
|
||||
# let an unauthenticated caller relay arbitrary requests upstream.
|
||||
_AUTH_EXEMPT_PATHS = frozenset({"/health", "/livez", "/readyz"})
|
||||
_AUTH_EXEMPT_PATHS = _HEALTH_PROBE_PATHS
|
||||
|
||||
# A non-loopback bind with no token is the exact shape (``--host 0.0.0.0``
|
||||
# from any launcher) that exposes the unauthenticated /v1/* relay to the
|
||||
|
||||
@@ -16,6 +16,8 @@ WIRE_DEBUG_SECRET_KEYS = (
|
||||
"x-api-key",
|
||||
"openai-api-key",
|
||||
"anthropic-api-key",
|
||||
# Codex's signed device attestation, sent with every request.
|
||||
"x-oai-attestation",
|
||||
"access_token",
|
||||
"refresh_token",
|
||||
"id_token",
|
||||
|
||||
@@ -114,3 +114,22 @@ def test_streaming_response_no_content_length(capture_logs):
|
||||
m = re.search(r"response_content_length=(\S*)", line)
|
||||
assert m, f"expected response_content_length field in: {line}"
|
||||
assert m.group(1) in ("", '""'), f"expected empty for streaming, got: {m.group(1)}"
|
||||
|
||||
|
||||
def test_health_probes_log_at_debug_and_traffic_at_info(capture_logs):
|
||||
"""A desktop polls /readyz every few seconds: at INFO it filled half the log."""
|
||||
app = create_app(_make_config())
|
||||
with TestClient(app) as client:
|
||||
client.app.state.proxy.handle_anthropic_messages = _json_handler
|
||||
client.get("/readyz")
|
||||
client.get("/livez")
|
||||
client.post("/v1/messages", json={"model": "glm-5.2", "messages": []})
|
||||
|
||||
levels = {}
|
||||
for r in capture_logs:
|
||||
m = re.search(r"event=proxy_inbound_(?:request|response) .*?path=(\S+)", r.getMessage())
|
||||
if m:
|
||||
levels.setdefault(m.group(1), set()).add(r.levelno)
|
||||
assert levels["/readyz"] == {logging.DEBUG}
|
||||
assert levels["/livez"] == {logging.DEBUG}
|
||||
assert levels["/v1/messages"] == {logging.INFO}
|
||||
|
||||
@@ -81,3 +81,13 @@ def test_wire_debug_keeps_token_usage_counters_visible() -> None:
|
||||
"token_count",
|
||||
):
|
||||
assert not should_redact_key(key), key
|
||||
|
||||
|
||||
def test_wire_debug_redacts_codex_attestation_header() -> None:
|
||||
# The inbound request log line runs every header through this policy, so
|
||||
# without it each Codex request wrote the ~4 KB signed attestation to proxy.log.
|
||||
redacted = redact_for_wire_debug(
|
||||
{"x-oai-attestation": '{"v":1,"s":1,"t":"signed"}', "x-codex-window-id": "w:1"}
|
||||
)
|
||||
|
||||
assert redacted == {"x-oai-attestation": WIRE_DEBUG_REDACTED, "x-codex-window-id": "w:1"}
|
||||
|
||||
Reference in New Issue
Block a user