fix(proxy): keep health probes and the Codex attestation out of the proxy log (#4611)

## Description

Two things fill and leak through the always-on proxy log
(`~/.headroom/logs/proxy-<port>.log`):

1. Health probes. Every `/readyz`, `/livez` and `/health` poll writes an
`event=proxy_inbound_request` and an `event=proxy_inbound_response` line
at INFO. A desktop wrapper or orchestrator polls these every few
seconds: in an Extra Headroom user's proxy log (Codex + Claude Code,
2026-10-27) 23,201 `/readyz` probes were 19% of 54 MB, and on a lighter
machine they are 48% of the file. With 10 MB x 5 rotation, that log held
only 14 hours; at the busiest hour one 10 MB file lasted about an hour.
2. The Codex attestation. The inbound request line runs headers through
`redact_for_wire_debug`, whose key list has no entry for
`x-oai-attestation`. Codex sends that ~4 KB signed device attestation on
every request, so every Codex request wrote it to the log in clear. That
is 3,040 copies (12 MB) in the log above, in a file users are routinely
asked to attach to bug reports. #2612 proposed the same key for the
opt-in wire-debug capture and was closed for inactivity. The default
proxy log uses the same policy, so this fix covers both.

Health probes now log at DEBUG (both lines), and `x-oai-attestation` is
a redacted key. Every other path still logs at INFO exactly as before.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/server.py`: a module-level `_HEALTH_PROBE_PATHS`
(`/health`, `/livez`, `/readyz`). The inbound request/response lines use
DEBUG for a GET to one of them, INFO otherwise. `_AUTH_EXEMPT_PATHS` now
refers to the same set instead of repeating it. The aborted/error lines
are unchanged.
- `headroom/proxy/wire_debug_redaction_policy.py`: `x-oai-attestation`
added to `WIRE_DEBUG_SECRET_KEYS`.
- Tests: `test_health_probes_log_at_debug_and_traffic_at_info` and
`test_wire_debug_redacts_codex_attestation_header`. Both fail on `main`
and pass here.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest -q tests/test_wire_debug_redaction_policy.py tests/test_proxy/test_inbound_response_size.py
11 passed, 1 warning in 5.36s

# every test file that mentions readyz / livez / proxy_token / inbound_request (36 files)
$ uv run --frozen --extra dev pytest -q $(git grep -l -E 'readyz|livez|proxy_token|inbound_request' -- 'tests/*.py' 'tests/**/*.py')
4 failed, 1245 passed, 2 skipped
# the 4 are tests/test_cli/test_wrap_persistent.py and fail identically on unmodified main on this machine

$ uvx ruff check <changed files>   -> All checks passed!
$ uvx ruff format --check <changed files>   -> 4 files already formatted
$ uv run --frozen --extra dev mypy headroom/proxy/server.py headroom/proxy/wire_debug_redaction_policy.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: macOS 27.2, this branch via `uv run --frozen --extra
proxy headroom proxy --port 18799 --no-optimize`,
`HEADROOM_SKIP_UPSTREAM_CHECK=1`.
- Exact command / steps: polled `GET /livez` until ready, then `curl
/readyz`, then `curl -H 'x-oai-attestation:
{"v":1,"s":1,"t":"SIGNED-DEVICE-TOKEN"}' -H 'x-codex-window-id: w:1'
/stats`, then read stderr and `~/.headroom/logs/proxy-18799.log`.
- Observed result: neither output has a `/livez` or `/readyz` line; the
`/stats` request line shows `"x-oai-attestation": "[REDACTED]"` next to
the visible `x-codex-window-id`, and `SIGNED-DEVICE-TOKEN` occurs 0
times in either output.
- Not tested: a Codex client end to end; the header value is never used
by the proxy, only logged.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a, logging only
- Stable/default behavior changed: yes, health-probe inbound lines move
from INFO to DEBUG and `x-oai-attestation` is redacted in logged
headers; forwarding is unchanged
- Kill switch / disable path: run with DEBUG logging to see probe lines
again
- Unsafe override required: no
- Qualification impact: none, no request or response bytes change
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Docs: no doc describes these log lines. Credit to #2612 for the
redaction key.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4611"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
gglucass
2026-11-03 17:00:02 -06:00
committed by GitHub
co-authored by Claude Opus 5.5
parent affd17c5fd
commit 78736a16bc
4 changed files with 44 additions and 3 deletions
+13 -3
View File
@@ -487,6 +487,9 @@ logging.basicConfig(
)
logger = logging.getLogger("headroom.proxy")
# Liveness/readiness probe routes: auth-exempt, and logged at DEBUG.
_HEALTH_PROBE_PATHS = frozenset({"/health", "/livez", "/readyz"})
def _output_reduction_payload(shaper_active: bool, est: Any = None) -> dict[str, Any]:
"""/stats payload for the output-shaping layer.
@@ -4042,7 +4045,13 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
safe_headers = redact_for_wire_debug(headers)
except Exception:
safe_headers = {"redaction_error": True}
logger.info(
# A desktop or orchestrator polls these every few seconds; at INFO they
# filled half of an idle proxy log and cut a busy one's history to hours.
line_level = (
logging.DEBUG if method == "GET" and path in _HEALTH_PROBE_PATHS else logging.INFO
)
logger.log(
line_level,
"event=proxy_inbound_request id=%s method=%s path=%s query=%s client=%s "
"content_length=%s headers=%s",
inbound_id,
@@ -4096,7 +4105,8 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
proxy.metrics.record_inbound_response(status_code=response.status_code)
except Exception:
logger.debug("record_inbound_response failed", exc_info=True)
logger.info(
logger.log(
line_level,
"event=proxy_inbound_response id=%s method=%s path=%s status=%s "
"duration_ms=%.2f response_content_length=%s",
inbound_id,
@@ -4140,7 +4150,7 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
# Anything else on these paths falls through to the catch-all passthrough
# relay (FastAPI's @app.get does not register HEAD), so exempting it would
# let an unauthenticated caller relay arbitrary requests upstream.
_AUTH_EXEMPT_PATHS = frozenset({"/health", "/livez", "/readyz"})
_AUTH_EXEMPT_PATHS = _HEALTH_PROBE_PATHS
# A non-loopback bind with no token is the exact shape (``--host 0.0.0.0``
# from any launcher) that exposes the unauthenticated /v1/* relay to the
@@ -16,6 +16,8 @@ WIRE_DEBUG_SECRET_KEYS = (
"x-api-key",
"openai-api-key",
"anthropic-api-key",
# Codex's signed device attestation, sent with every request.
"x-oai-attestation",
"access_token",
"refresh_token",
"id_token",
@@ -114,3 +114,22 @@ def test_streaming_response_no_content_length(capture_logs):
m = re.search(r"response_content_length=(\S*)", line)
assert m, f"expected response_content_length field in: {line}"
assert m.group(1) in ("", '""'), f"expected empty for streaming, got: {m.group(1)}"
def test_health_probes_log_at_debug_and_traffic_at_info(capture_logs):
"""A desktop polls /readyz every few seconds: at INFO it filled half the log."""
app = create_app(_make_config())
with TestClient(app) as client:
client.app.state.proxy.handle_anthropic_messages = _json_handler
client.get("/readyz")
client.get("/livez")
client.post("/v1/messages", json={"model": "glm-5.2", "messages": []})
levels = {}
for r in capture_logs:
m = re.search(r"event=proxy_inbound_(?:request|response) .*?path=(\S+)", r.getMessage())
if m:
levels.setdefault(m.group(1), set()).add(r.levelno)
assert levels["/readyz"] == {logging.DEBUG}
assert levels["/livez"] == {logging.DEBUG}
assert levels["/v1/messages"] == {logging.INFO}
+10
View File
@@ -81,3 +81,13 @@ def test_wire_debug_keeps_token_usage_counters_visible() -> None:
"token_count",
):
assert not should_redact_key(key), key
def test_wire_debug_redacts_codex_attestation_header() -> None:
# The inbound request log line runs every header through this policy, so
# without it each Codex request wrote the ~4 KB signed attestation to proxy.log.
redacted = redact_for_wire_debug(
{"x-oai-attestation": '{"v":1,"s":1,"t":"signed"}', "x-codex-window-id": "w:1"}
)
assert redacted == {"x-oai-attestation": WIRE_DEBUG_REDACTED, "x-codex-window-id": "w:1"}