mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(security): exempt only GET health probes from the proxy token (#3921)
## Description
When `HEADROOM_PROXY_TOKEN` is set, `_security_gate` exempts `/health`,
`/healthz`, `/livez` and `/readyz` from the token check so orchestrators
can probe a container bound to a non-loopback address. The exemption
matched the path only, ignoring the method.
Only `GET` on `/health`, `/livez` and `/readyz` reaches a health
handler. Everything else on those paths falls through to the catch-all
`/{path:path}` passthrough, which relays the request upstream:
- `/healthz` has no Python route at all; only the Rust `headroom-proxy`
serves it.
- `POST`, `PUT` and `DELETE` on the three health paths do not match the
`GET` routes.
- `HEAD` does not match them either, because FastAPI's `@app.get` does
not register `HEAD`.
So with a token configured, an unauthenticated non-loopback caller could
send any method and body through the passthrough, to the configured
upstream or to a public host chosen with `x-headroom-base-url`. The
token is meant to prevent exactly that.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **`headroom/proxy/server.py`:** the gate exempts a request only when
it is `GET` and its path is in `_AUTH_EXEMPT_PATHS`. `/healthz` is
removed from that set. A comment records why the exemption is limited to
`GET`.
- **`tests/test_proxy_hardening.py`:**
`test_health_exemption_does_not_reach_passthrough` sends `GET /healthz`,
`POST /health`, `PUT /livez`, `DELETE /readyz` and `HEAD /livez` from a
non-loopback peer with a token configured. It asserts a 401, and a spy
on `handle_passthrough` asserts that the passthrough is never reached.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`): one error, already present
on `main`; see below
- [x] New tests added for new functionality
- [ ] Manual testing performed
With the `server.py` change reverted, all 5 new cases fail. With it
applied, they pass.
### Test Output
```text
$ pytest -q tests/test_proxy_hardening.py
36 passed, 1 warning in 4.40s
$ pytest -q tests/test_proxy_hardening.py -k health_exemption_does_not_reach # server.py change reverted
5 failed, 31 deselected, 1 warning in 2.13s
$ pytest -q tests/test_proxy*.py tests/test_identity_resolution.py
991 passed, 107 skipped, 4 warnings in 78.63s
$ ruff check . && ruff format --check .
All checks passed!
1705 files already formatted
$ mypy headroom
headroom/integrations/langchain/chat_model.py:394: error: Function "RunnableBinding" could always be true in boolean context [truthy-function]
Found 1 error in 1 file (checked 554 source files)
```
The mypy error is in an unrelated file and appears identically on the
parent commit.
## Real Behavior Proof
- Environment: macOS, Python 3.12, `uv sync --extra proxy --extra dev`.
The app is built with `create_app` and driven with FastAPI `TestClient`
from peer `203.0.113.5` with `proxy_token` set. `handle_passthrough` is
replaced with a recorder, so nothing leaves the machine.
- Exact command / steps: send `GET /healthz`, `POST /health` and `POST
/readyz` with `x-headroom-base-url: https://example.com` and no token,
on the parent commit and on this branch.
- Observed result: on the parent commit, all three return 200 and the
recorder shows each request relayed to `https://example.com`, while a
non-exempt path (`GET /v1/models-x`) returns 401. On this branch, all of
them return 401 and nothing is relayed. `GET /livez` and `GET /readyz`
without a token still succeed
(`test_health_endpoints_exempt_even_nonloopback`).
- Not tested: a live proxy bound to a network interface.
## Runtime Rollout Safety
- Rollout-managed feature(s): none. This is a fix in the inbound auth
middleware.
- Minimum rollout channel: n/a
- Stable/default behavior changed: only for non-loopback callers when
`HEADROOM_PROXY_TOKEN` is set. Any request to `/healthz`, and any
non-`GET` request to `/health`, `/livez` or `/readyz`, now needs the
token. Loopback callers and token-less setups are unaffected.
- Kill switch / disable path: none (deliberately; an unauthenticated
relay is never correct).
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
- **Probes:** every probe in the repo uses `GET /readyz` (Dockerfile
`HEALTHCHECK`, `docker-compose.yml`, `scripts/install.sh`). Nothing
probes the Python proxy's `/healthz`. A probe that uses `HEAD`, or
`/healthz`, against the Python proxy from a non-loopback address with a
token set would now get 401. Before this change such a request was
relayed upstream, not answered by a health handler, so it was never a
real health check.
- **Overlap with #3852:** that PR edits lines next to the gate in
`create_app` but leaves `_AUTH_EXEMPT_PATHS` as it is, so the two are
complementary. Whichever lands second needs a trivial rebase.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3921"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
JerrettDavis
parent
d722a6414c
commit
d99779dae6
@@ -3989,7 +3989,12 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
_proxy_token_bytes = _proxy_token.encode("utf-8") if _proxy_token else b""
|
||||
# Health/readiness probes must stay reachable without a token so
|
||||
# orchestrators can check a container that binds non-loopback.
|
||||
_AUTH_EXEMPT_PATHS = frozenset({"/health", "/healthz", "/livez", "/readyz"})
|
||||
#
|
||||
# The exemption covers only GET, and only paths with a GET handler below.
|
||||
# Anything else on these paths falls through to the catch-all passthrough
|
||||
# relay (FastAPI's @app.get does not register HEAD), so exempting it would
|
||||
# let an unauthenticated caller relay arbitrary requests upstream.
|
||||
_AUTH_EXEMPT_PATHS = frozenset({"/health", "/livez", "/readyz"})
|
||||
|
||||
# A non-loopback bind with no token is the exact shape (``--host 0.0.0.0``
|
||||
# from any launcher) that exposes the unauthenticated /v1/* relay to the
|
||||
@@ -4050,7 +4055,8 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
path = request.url.path
|
||||
client = getattr(request, "client", None)
|
||||
client_host = getattr(client, "host", None) if client is not None else None
|
||||
if path not in _AUTH_EXEMPT_PATHS and not is_loopback_host(client_host):
|
||||
exempt = request.method == "GET" and path in _AUTH_EXEMPT_PATHS
|
||||
if not exempt and not is_loopback_host(client_host):
|
||||
provided = _extract_proxy_token(request.headers)
|
||||
if provided is None or not hmac.compare_digest(
|
||||
provided.encode("utf-8", "replace"), _proxy_token_bytes
|
||||
|
||||
@@ -16,11 +16,13 @@ import pytest
|
||||
|
||||
pytest.importorskip("fastapi")
|
||||
|
||||
from fastapi.responses import JSONResponse
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from headroom.cache.compression_store import reset_compression_store
|
||||
from headroom.offline import apply_offline_env, is_offline
|
||||
from headroom.proxy.audit import is_auditable_path
|
||||
from headroom.proxy.handlers.openai import OpenAIHandlerMixin
|
||||
from headroom.proxy.server import (
|
||||
ProxyConfig,
|
||||
WebSocketAuthMiddleware,
|
||||
@@ -127,6 +129,36 @@ class TestInboundAuthToken:
|
||||
assert c.get("/livez").status_code == 200
|
||||
assert c.get("/readyz").status_code in (200, 503) # ready/not-ready, never 401
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("method", "path"),
|
||||
[
|
||||
("GET", "/healthz"), # no Python route: lands on the catch-all
|
||||
("POST", "/health"),
|
||||
("PUT", "/livez"),
|
||||
("DELETE", "/readyz"),
|
||||
("HEAD", "/livez"), # @app.get does not register HEAD
|
||||
],
|
||||
)
|
||||
def test_health_exemption_does_not_reach_passthrough(self, monkeypatch, method, path):
|
||||
"""Only GET probes are exempt from the token.
|
||||
|
||||
Any other request on a health path is not served by the health handler
|
||||
but by the catch-all passthrough, which relays it upstream. Exempting
|
||||
those turned the proxy into an unauthenticated relay.
|
||||
"""
|
||||
relayed: list[tuple[str, str]] = []
|
||||
|
||||
async def _spy_passthrough(self, request, base_url, *args, **kwargs):
|
||||
relayed.append((request.method, request.url.path))
|
||||
return JSONResponse({"relayed_to": base_url})
|
||||
|
||||
monkeypatch.setattr(OpenAIHandlerMixin, "handle_passthrough", _spy_passthrough)
|
||||
app = _make_app(proxy_token="s3cr3t-token")
|
||||
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
||||
resp = c.request(method, path)
|
||||
assert resp.status_code == 401
|
||||
assert relayed == []
|
||||
|
||||
|
||||
# ──────────────────── 2.1b inbound auth token over WebSocket ──────────────
|
||||
|
||||
|
||||
Reference in New Issue
Block a user