mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description Two things fill and leak through the always-on proxy log (`~/.headroom/logs/proxy-<port>.log`): 1. Health probes. Every `/readyz`, `/livez` and `/health` poll writes an `event=proxy_inbound_request` and an `event=proxy_inbound_response` line at INFO. A desktop wrapper or orchestrator polls these every few seconds: in an Extra Headroom user's proxy log (Codex + Claude Code, 2026-10-27) 23,201 `/readyz` probes were 19% of 54 MB, and on a lighter machine they are 48% of the file. With 10 MB x 5 rotation, that log held only 14 hours; at the busiest hour one 10 MB file lasted about an hour. 2. The Codex attestation. The inbound request line runs headers through `redact_for_wire_debug`, whose key list has no entry for `x-oai-attestation`. Codex sends that ~4 KB signed device attestation on every request, so every Codex request wrote it to the log in clear. That is 3,040 copies (12 MB) in the log above, in a file users are routinely asked to attach to bug reports. #2612 proposed the same key for the opt-in wire-debug capture and was closed for inactivity. The default proxy log uses the same policy, so this fix covers both. Health probes now log at DEBUG (both lines), and `x-oai-attestation` is a redacted key. Every other path still logs at INFO exactly as before. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/server.py`: a module-level `_HEALTH_PROBE_PATHS` (`/health`, `/livez`, `/readyz`). The inbound request/response lines use DEBUG for a GET to one of them, INFO otherwise. `_AUTH_EXEMPT_PATHS` now refers to the same set instead of repeating it. The aborted/error lines are unchanged. - `headroom/proxy/wire_debug_redaction_policy.py`: `x-oai-attestation` added to `WIRE_DEBUG_SECRET_KEYS`. - Tests: `test_health_probes_log_at_debug_and_traffic_at_info` and `test_wire_debug_redacts_codex_attestation_header`. Both fail on `main` and pass here. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest -q tests/test_wire_debug_redaction_policy.py tests/test_proxy/test_inbound_response_size.py 11 passed, 1 warning in 5.36s # every test file that mentions readyz / livez / proxy_token / inbound_request (36 files) $ uv run --frozen --extra dev pytest -q $(git grep -l -E 'readyz|livez|proxy_token|inbound_request' -- 'tests/*.py' 'tests/**/*.py') 4 failed, 1245 passed, 2 skipped # the 4 are tests/test_cli/test_wrap_persistent.py and fail identically on unmodified main on this machine $ uvx ruff check <changed files> -> All checks passed! $ uvx ruff format --check <changed files> -> 4 files already formatted $ uv run --frozen --extra dev mypy headroom/proxy/server.py headroom/proxy/wire_debug_redaction_policy.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: macOS 27.2, this branch via `uv run --frozen --extra proxy headroom proxy --port 18799 --no-optimize`, `HEADROOM_SKIP_UPSTREAM_CHECK=1`. - Exact command / steps: polled `GET /livez` until ready, then `curl /readyz`, then `curl -H 'x-oai-attestation: {"v":1,"s":1,"t":"SIGNED-DEVICE-TOKEN"}' -H 'x-codex-window-id: w:1' /stats`, then read stderr and `~/.headroom/logs/proxy-18799.log`. - Observed result: neither output has a `/livez` or `/readyz` line; the `/stats` request line shows `"x-oai-attestation": "[REDACTED]"` next to the visible `x-codex-window-id`, and `SIGNED-DEVICE-TOKEN` occurs 0 times in either output. - Not tested: a Codex client end to end; the header value is never used by the proxy, only logged. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a, logging only - Stable/default behavior changed: yes, health-probe inbound lines move from INFO to DEBUG and `x-oai-attestation` is redacted in logged headers; forwarding is unchanged - Kill switch / disable path: run with DEBUG logging to see probe lines again - Unsafe override required: no - Qualification impact: none, no request or response bytes change - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Docs: no doc describes these log lines. Credit to #2612 for the redaction key. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4611" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
94 lines
2.9 KiB
Python
94 lines
2.9 KiB
Python
from __future__ import annotations
|
|
|
|
from headroom.proxy.wire_debug_redaction_policy import (
|
|
WIRE_DEBUG_REDACTED,
|
|
redact_for_wire_debug,
|
|
should_redact_key,
|
|
)
|
|
|
|
|
|
def test_wire_debug_redacts_direct_secret_keys() -> None:
|
|
redacted = redact_for_wire_debug(
|
|
{
|
|
"Authorization": "Bearer test-token",
|
|
"x-api-key": "sk-test",
|
|
"safe": "visible",
|
|
}
|
|
)
|
|
|
|
assert redacted == {
|
|
"Authorization": WIRE_DEBUG_REDACTED,
|
|
"x-api-key": WIRE_DEBUG_REDACTED,
|
|
"safe": "visible",
|
|
}
|
|
|
|
|
|
def test_wire_debug_redacts_nested_secret_suffixes() -> None:
|
|
redacted = redact_for_wire_debug(
|
|
{
|
|
"messages": [
|
|
{"content": "visible", "service_access_token": "secret-token"},
|
|
{"metadata": {"database_password": "secret-password", "trace_id": "abc"}},
|
|
]
|
|
}
|
|
)
|
|
|
|
assert redacted["messages"][0]["content"] == "visible"
|
|
assert redacted["messages"][0]["service_access_token"] == WIRE_DEBUG_REDACTED
|
|
assert redacted["messages"][1]["metadata"]["database_password"] == WIRE_DEBUG_REDACTED
|
|
assert redacted["messages"][1]["metadata"]["trace_id"] == "abc"
|
|
|
|
|
|
def test_wire_debug_key_matching_normalizes_dashes_and_case() -> None:
|
|
assert should_redact_key("Anthropic-API-Key")
|
|
assert should_redact_key("custom-refresh-token")
|
|
assert not should_redact_key("token_count")
|
|
|
|
|
|
def test_wire_debug_redacts_proxy_authorization() -> None:
|
|
redacted = redact_for_wire_debug(
|
|
{"Proxy-Authorization": "Basic dXNlcjpwYXNz", "user-agent": "codex/1.0"}
|
|
)
|
|
|
|
assert redacted["Proxy-Authorization"] == WIRE_DEBUG_REDACTED
|
|
assert redacted["user-agent"] == "codex/1.0"
|
|
|
|
|
|
def test_wire_debug_redacts_any_token_suffix() -> None:
|
|
for key in (
|
|
"auth_token",
|
|
"session_token",
|
|
"api_token",
|
|
"github_token",
|
|
"x-amz-security-token",
|
|
"google_id_token",
|
|
):
|
|
assert should_redact_key(key), key
|
|
|
|
|
|
def test_wire_debug_redacts_credentials_and_secret_key() -> None:
|
|
for key in ("credentials", "aws_credentials", "secret_key", "aws_secret_key"):
|
|
assert should_redact_key(key), key
|
|
|
|
|
|
def test_wire_debug_keeps_token_usage_counters_visible() -> None:
|
|
for key in (
|
|
"max_tokens",
|
|
"input_tokens",
|
|
"output_tokens",
|
|
"total_tokens",
|
|
"cache_read_input_tokens",
|
|
"token_count",
|
|
):
|
|
assert not should_redact_key(key), key
|
|
|
|
|
|
def test_wire_debug_redacts_codex_attestation_header() -> None:
|
|
# The inbound request log line runs every header through this policy, so
|
|
# without it each Codex request wrote the ~4 KB signed attestation to proxy.log.
|
|
redacted = redact_for_wire_debug(
|
|
{"x-oai-attestation": '{"v":1,"s":1,"t":"signed"}', "x-codex-window-id": "w:1"}
|
|
)
|
|
|
|
assert redacted == {"x-oai-attestation": WIRE_DEBUG_REDACTED, "x-codex-window-id": "w:1"}
|