fix(security): exempt only GET health probes from the proxy token (#3921)

## Description

When `HEADROOM_PROXY_TOKEN` is set, `_security_gate` exempts `/health`,
`/healthz`, `/livez` and `/readyz` from the token check so orchestrators
can probe a container bound to a non-loopback address. The exemption
matched the path only, ignoring the method.

Only `GET` on `/health`, `/livez` and `/readyz` reaches a health
handler. Everything else on those paths falls through to the catch-all
`/{path:path}` passthrough, which relays the request upstream:
- `/healthz` has no Python route at all; only the Rust `headroom-proxy`
serves it.
- `POST`, `PUT` and `DELETE` on the three health paths do not match the
`GET` routes.
- `HEAD` does not match them either, because FastAPI's `@app.get` does
not register `HEAD`.

So with a token configured, an unauthenticated non-loopback caller could
send any method and body through the passthrough, to the configured
upstream or to a public host chosen with `x-headroom-base-url`. The
token is meant to prevent exactly that.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **`headroom/proxy/server.py`:** the gate exempts a request only when
it is `GET` and its path is in `_AUTH_EXEMPT_PATHS`. `/healthz` is
removed from that set. A comment records why the exemption is limited to
`GET`.
- **`tests/test_proxy_hardening.py`:**
`test_health_exemption_does_not_reach_passthrough` sends `GET /healthz`,
`POST /health`, `PUT /livez`, `DELETE /readyz` and `HEAD /livez` from a
non-loopback peer with a token configured. It asserts a 401, and a spy
on `handle_passthrough` asserts that the passthrough is never reached.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`): one error, already present
on `main`; see below
- [x] New tests added for new functionality
- [ ] Manual testing performed

With the `server.py` change reverted, all 5 new cases fail. With it
applied, they pass.

### Test Output

```text
$ pytest -q tests/test_proxy_hardening.py
36 passed, 1 warning in 4.40s

$ pytest -q tests/test_proxy_hardening.py -k health_exemption_does_not_reach   # server.py change reverted
5 failed, 31 deselected, 1 warning in 2.13s

$ pytest -q tests/test_proxy*.py tests/test_identity_resolution.py
991 passed, 107 skipped, 4 warnings in 78.63s

$ ruff check . && ruff format --check .
All checks passed!
1705 files already formatted

$ mypy headroom
headroom/integrations/langchain/chat_model.py:394: error: Function "RunnableBinding" could always be true in boolean context  [truthy-function]
Found 1 error in 1 file (checked 554 source files)
```

The mypy error is in an unrelated file and appears identically on the
parent commit.

## Real Behavior Proof

- Environment: macOS, Python 3.12, `uv sync --extra proxy --extra dev`.
The app is built with `create_app` and driven with FastAPI `TestClient`
from peer `203.0.113.5` with `proxy_token` set. `handle_passthrough` is
replaced with a recorder, so nothing leaves the machine.
- Exact command / steps: send `GET /healthz`, `POST /health` and `POST
/readyz` with `x-headroom-base-url: https://example.com` and no token,
on the parent commit and on this branch.
- Observed result: on the parent commit, all three return 200 and the
recorder shows each request relayed to `https://example.com`, while a
non-exempt path (`GET /v1/models-x`) returns 401. On this branch, all of
them return 401 and nothing is relayed. `GET /livez` and `GET /readyz`
without a token still succeed
(`test_health_endpoints_exempt_even_nonloopback`).
- Not tested: a live proxy bound to a network interface.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is a fix in the inbound auth
middleware.
- Minimum rollout channel: n/a
- Stable/default behavior changed: only for non-loopback callers when
`HEADROOM_PROXY_TOKEN` is set. Any request to `/healthz`, and any
non-`GET` request to `/health`, `/livez` or `/readyz`, now needs the
token. Loopback callers and token-less setups are unaffected.
- Kill switch / disable path: none (deliberately; an unauthenticated
relay is never correct).
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- **Probes:** every probe in the repo uses `GET /readyz` (Dockerfile
`HEALTHCHECK`, `docker-compose.yml`, `scripts/install.sh`). Nothing
probes the Python proxy's `/healthz`. A probe that uses `HEAD`, or
`/healthz`, against the Python proxy from a non-loopback address with a
token set would now get 401. Before this change such a request was
relayed upstream, not answered by a health handler, so it was never a
real health check.
- **Overlap with #3852:** that PR edits lines next to the gate in
`create_app` but leaves `_AUTH_EXEMPT_PATHS` as it is, so the two are
complementary. Whichever lands second needs a trivial rebase.
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3921"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Satya Bharadwaj
2026-10-05 10:02:07 -05:00
committed by GitHub
co-authored by JerrettDavis
parent d722a6414c
commit d99779dae6
2 changed files with 40 additions and 2 deletions
+8 -2
View File
@@ -3989,7 +3989,12 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
_proxy_token_bytes = _proxy_token.encode("utf-8") if _proxy_token else b""
# Health/readiness probes must stay reachable without a token so
# orchestrators can check a container that binds non-loopback.
_AUTH_EXEMPT_PATHS = frozenset({"/health", "/healthz", "/livez", "/readyz"})
#
# The exemption covers only GET, and only paths with a GET handler below.
# Anything else on these paths falls through to the catch-all passthrough
# relay (FastAPI's @app.get does not register HEAD), so exempting it would
# let an unauthenticated caller relay arbitrary requests upstream.
_AUTH_EXEMPT_PATHS = frozenset({"/health", "/livez", "/readyz"})
# A non-loopback bind with no token is the exact shape (``--host 0.0.0.0``
# from any launcher) that exposes the unauthenticated /v1/* relay to the
@@ -4050,7 +4055,8 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
path = request.url.path
client = getattr(request, "client", None)
client_host = getattr(client, "host", None) if client is not None else None
if path not in _AUTH_EXEMPT_PATHS and not is_loopback_host(client_host):
exempt = request.method == "GET" and path in _AUTH_EXEMPT_PATHS
if not exempt and not is_loopback_host(client_host):
provided = _extract_proxy_token(request.headers)
if provided is None or not hmac.compare_digest(
provided.encode("utf-8", "replace"), _proxy_token_bytes
+32
View File
@@ -16,11 +16,13 @@ import pytest
pytest.importorskip("fastapi")
from fastapi.responses import JSONResponse
from fastapi.testclient import TestClient
from headroom.cache.compression_store import reset_compression_store
from headroom.offline import apply_offline_env, is_offline
from headroom.proxy.audit import is_auditable_path
from headroom.proxy.handlers.openai import OpenAIHandlerMixin
from headroom.proxy.server import (
ProxyConfig,
WebSocketAuthMiddleware,
@@ -127,6 +129,36 @@ class TestInboundAuthToken:
assert c.get("/livez").status_code == 200
assert c.get("/readyz").status_code in (200, 503) # ready/not-ready, never 401
@pytest.mark.parametrize(
("method", "path"),
[
("GET", "/healthz"), # no Python route: lands on the catch-all
("POST", "/health"),
("PUT", "/livez"),
("DELETE", "/readyz"),
("HEAD", "/livez"), # @app.get does not register HEAD
],
)
def test_health_exemption_does_not_reach_passthrough(self, monkeypatch, method, path):
"""Only GET probes are exempt from the token.
Any other request on a health path is not served by the health handler
but by the catch-all passthrough, which relays it upstream. Exempting
those turned the proxy into an unauthenticated relay.
"""
relayed: list[tuple[str, str]] = []
async def _spy_passthrough(self, request, base_url, *args, **kwargs):
relayed.append((request.method, request.url.path))
return JSONResponse({"relayed_to": base_url})
monkeypatch.setattr(OpenAIHandlerMixin, "handle_passthrough", _spy_passthrough)
app = _make_app(proxy_token="s3cr3t-token")
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
resp = c.request(method, path)
assert resp.status_code == 401
assert relayed == []
# ──────────────────── 2.1b inbound auth token over WebSocket ──────────────