mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description With `HEADROOM_PROXY_TOKEN` set, every Anthropic SDK client logs a `proxy_auth_rejected` line once per process, Claude Code included and even when it is configured correctly. Claude Code sends an unauthenticated `HEAD /api/hello` reachability probe. That probe never carries `ANTHROPIC_CUSTOM_HEADERS`, so it cannot hold the proxy token: it gets a 401, which the client ignores, and the rejection lands in the log. On a busy deployment these lines bury real rejections. Behind a gateway, the rejection line also showed only the gateway's peer address, so a rejected caller could not be identified. This PR: 1. **Answers the probe locally, behind the token.** The HTTP security gate lets GET/HEAD on exactly `/api/hello` (also under `/p/<project>`) past the token check and marks it. Extension middleware still runs, so an extension can still refuse it. The `/api/hello` passthrough route answers a marked probe with an empty 200 (the gate adds the security headers) and never relays it. This only happens when a token is configured. It is deliberately **not** added to `_AUTH_EXEMPT_PATHS`, which would hand the probe to the passthrough relay: an anonymous upstream request (to Anthropic since #4059, to the OpenAI default target before it). Without a token, the probe keeps its #4059 route unchanged. 2. **Gives the rejection line context.** `proxy_auth_rejected` (HTTP and WebSocket) adds `method=`, `ua=` and `xff=` fields. Both headers are caller-controlled, so neither is logged as received; each maps to a fixed vocabulary: - `ua`: the harness name `CLIENT_UA_MAP` gives the User-Agent, the same name the outcome and cost records use (`claude-code`, `codex`, `cursor`, `opencode`, `aider`, `copilot`, …). Failing that, the product token of a generic HTTP client (`node`, `anthropic`, `openai`, `curl`, `mozilla`, `python-httpx`, …). Otherwise `other`. `X-Client` is not consulted, since it is free text. - `xff`: the leftmost `X-Forwarded-For` hop when it parses as a plain IP address, else `invalid`. An IPv6 zone index (`fe80::1%<zone>`) is free text, so it counts as `invalid`. The hop is unverified, used only for attribution, and drives no decision. - Either is `none` when its header is absent. - No credential header is read for the log. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/server.py`: - **`_security_gate`**: when a token is configured, `GET`/`HEAD` on exactly `/api/hello`, matched on the routed path so `/p/<project>/api/hello` counts too, skips the token check and sets `request.state.headroom_local_hello_probe`. It then calls `call_next` like any request, so extension middleware (installed inside the gate) still runs and can refuse it. `POST /api/hello`, `/api/hello/` and `/api/hello/x` still need the token. - **New helper**: `_auth_rejection_context` returns `(ua, xff)` from the fixed vocabulary above: `CLIENT_UA_MAP` then an allowlist of generic clients (`_AUTH_REJECTION_CLIENTS`) for the User-Agent, and `ipaddress.ip_address` on the first hop (from `forwarded_policy.header_first`). - **Both `proxy_auth_rejected` warnings** gain the new fields: the HTTP gate gets `method=`, and the WebSocket middleware gets the same `ua=`/`xff=` fields, so the two transports stay aligned. - `headroom/providers/proxy_routes.py`: the provider passthrough route answers a marked probe with an empty 200 and never relays it upstream. - `tests/test_proxy_hardening.py`: - **Probe answer** (`test_api_hello_probe_is_answered_locally`): GET/HEAD on `/api/hello` and `/p/demo/api/hello` from a non-loopback peer gets 200 with an empty body and security headers, and the passthrough spy shows nothing relayed. `POST /p/demo/api/hello` and `GET /p/demo/stats` still get 401. `POST /api/hello`, `GET /api/hello/x`, `GET /api/hello/` and `GET /stats` still get 401, with nothing relayed. With no token configured, `/api/hello` still reaches the passthrough. - **Log line**: `claude-cli/2.1.0 (external, cli)` with `X-Forwarded-For: 198.51.100.7, 10.0.0.9` logs `ua=claude-code xff=198.51.100.7`. Absent headers log `ua=none xff=none`. `test_known_clients_keep_their_proxy_wide_name` covers `claude-code`, `codex`, `cursor`, `opencode`, `aider` behind a wrapper prefix, the Anthropic SDK and `node`. - **Caller text never reaches the line** (`test_caller_text_never_reaches_the_line`): a `PRIVATE_CREDENTIAL_CANARY` in the User-Agent (bare, as a product token, behind a fake quoted field) and in X-Forwarded-For (as the first hop, as a zone index, as a port suffix) never appears in the line, which logs `other`/`invalid` instead. CR/LF in either header maps to `other`/`invalid` (tested on the helper directly, since an HTTP client refuses to send CR/LF). - **No credentials**: neither the configured token, a wrong `x-headroom-proxy-token`, `Authorization: Bearer …` nor `x-api-key` ever appears in the line. - **WebSocket**: the rejection line matches the HTTP gate's, and the same canary in both headers never appears in it (`test_rejection_log_keeps_caller_text_out`). - `tests/test_proxy_extension_ordering.py`: - **Extension policy** (`test_api_hello_probe_still_runs_extension_policy`): an extension that answers 403 to everything gets the token-less GET/HEAD probe and refuses it. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_proxy_hardening.py tests/test_anthropic_hello_probe_routing.py \ tests/test_provider_proxy_routes.py tests/test_provider_route_specs.py \ tests/test_proxy_loopback_gating.py tests/test_proxy_extension_ordering.py \ tests/test_proxy_bind_policy.py tests/test_auth_mode.py -q 365 passed in 78.96s $ ruff check headroom/proxy/server.py tests/test_proxy_hardening.py All checks passed! $ ruff format --check headroom/proxy/server.py tests/test_proxy_hardening.py 2 files already formatted $ python -m mypy headroom/proxy/server.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: this branch on Windows 11 with Python 3.13. Ran `headroom proxy --port 18787` from the worktree with a throwaway `HEADROOM_PROXY_TOKEN`. - Exact command / steps: `curl -I http://127.0.0.1:18787/api/hello`, `curl http://127.0.0.1:18787/api/hello`, then checked the proxy log. - Observed result: HEAD and GET returned `200` with 0 bytes and `x-content-type-options: nosniff`, `x-frame-options: DENY`, `referrer-policy: no-referrer` and `strict-transport-security`. The proxy log has no `proxy_inbound_request` and no `outbound_headers` line for either probe, so nothing was forwarded. For contrast, a loopback `POST /api/hello` (loopback is token-exempt, existing behavior) logs `forwarder=openai_passthrough`. - Not tested: the new rejection-line fields behind a real gateway. They were checked against a real non-loopback peer, below. Rejection diagnostics, at `d70ec3c` (contains current `main`): the same worktree proxy, bound to a non-loopback LAN address (`--host 192.168.0.48 --port 18787 --no-optimize`, `HEADROOM_OFFLINE=1`) with a throwaway token, called from that address: ```text GET /stats UA=PRIVATE_CREDENTIAL_CANARY XFF=PRIVATE_CREDENTIAL_CANARY, 192.0.2.1 -> 401 GET /stats UA=claude-cli/2.1.0 (external, cli) XFF=198.51.100.7, 10.0.0.9 -> 401 GET /stats UA=codex-cli/0.46.0 -> 401 GET /stats UA=PRIVATE_CREDENTIAL_CANARY/9 opencode/1.2.3 -> 401 GET /stats no UA, no XFF -> 401 GET /stats UA=PRIVATE_CREDENTIAL_CANARY/1.0 XFF=fe80::1%PRIVATE_CREDENTIAL_CANARY -> 401 GET /stats valid x-headroom-proxy-token -> 200 HEAD /api/hello -> 200 WS upgrade /v1/responses UA=PRIVATE_CREDENTIAL_CANARY XFF=PRIVATE_CREDENTIAL_CANARY -> 403 event=proxy_auth_rejected method=GET path=/stats client=192.168.0.48 reason=missing_token ua=other xff=invalid event=proxy_auth_rejected method=GET path=/stats client=192.168.0.48 reason=missing_token ua=claude-code xff=198.51.100.7 event=proxy_auth_rejected method=GET path=/stats client=192.168.0.48 reason=missing_token ua=codex xff=none event=proxy_auth_rejected method=GET path=/stats client=192.168.0.48 reason=missing_token ua=opencode xff=none event=proxy_auth_rejected method=GET path=/stats client=192.168.0.48 reason=missing_token ua=none xff=none event=proxy_auth_rejected method=GET path=/stats client=192.168.0.48 reason=missing_token ua=other xff=invalid event=proxy_auth_rejected transport=websocket path=/v1/responses client=192.168.0.48 reason=missing_token ua=other xff=invalid $ grep -c PRIVATE_CREDENTIAL_CANARY proxy.out 0 ``` The HEAD `/api/hello` probe from that non-loopback peer logs `proxy_inbound_request` / `proxy_inbound_response` (`status=200`, `response_content_length=0`) and no `outbound_headers` line. Since `d125805` the probe runs through the inbound request logging like any routed request; it is still never forwarded. The loopback run above predates that commit. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: n/a. - Stable/default behavior changed: only when `HEADROOM_PROXY_TOKEN` is set. GET/HEAD `/api/hello` then gets a local 200 instead of a 401, and the rejection line gains three fields. Answered probes appear in the inbound request log (`proxy_inbound_request` / `proxy_inbound_response`) instead of as `proxy_auth_rejected` warnings. With no token, behavior is unchanged. - Kill switch / disable path: none needed. The probe answer carries no data and makes no upstream call. - Unsafe override required: no. - Qualification impact: log parsers that match the exact old `proxy_auth_rejected` line need to accept the extra trailing fields. The `path=`, `client=` and `reason=` fields are unchanged; the HTTP line now starts with `method=` before `path=`. - Rollback path: revert the commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - Documentation: N/A. The docs don't list the token-exempt paths or the `proxy_auth_rejected` format. - Why only when a token is configured: without a token, the #4059 passthrough route is fine. Answering locally only matters when the alternative is a 401 for every client. - `xff` is not resolved through `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS`: the gate runs before forwarded-header resolution, and resolving here would emit untrusted-peer events on every rejection. It is logged only as a validated IP literal (or `invalid`/`none`), unverified, for attribution. - `ua` reuses `CLIENT_UA_MAP`, so a harness added there is named in the rejection line too. Generic clients outside `_AUTH_REJECTION_CLIENTS` log `other`; adding one is a one-line change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4150" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ### Maintainer combined-tree qualification On the integration with main `31dcaf41`, all 375 focused hardening, hello-routing, provider-route/spec, loopback, extension-ordering, bind-policy, and auth-mode tests passed on Windows/Python 3.12 using the existing native extension. Changed-file lint, formatting, and whitespace checks passed. The provider-route insertion conflict retains the PR local-probe early return before main per-request upstream-override processing; AST comparison proves the combined route is the unchanged PR guard followed by the unchanged main route body. The probe test now also supplies a loopback or metadata-address upstream override and confirms the response stays empty/local with nothing relayed. This checks the integration boundary without rewriting contributor authorization or forwarding logic. Hosted qualification for the published sync remains pending. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
1120 lines
44 KiB
Python
1120 lines
44 KiB
Python
"""Tests for the Tier-2 pilot hardening features:
|
|
|
|
- 2.1 optional inbound auth token (HEADROOM_PROXY_TOKEN) on the data plane
|
|
- 3.1 response security headers
|
|
- 2.4 admin/state-mutating audit log
|
|
- 2.2 air-gap master switch (HEADROOM_OFFLINE)
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import contextlib
|
|
import logging
|
|
|
|
import httpx
|
|
import pytest
|
|
|
|
pytest.importorskip("fastapi")
|
|
|
|
from fastapi.responses import JSONResponse
|
|
from fastapi.testclient import TestClient
|
|
|
|
from headroom.cache.compression_store import reset_compression_store
|
|
from headroom.offline import apply_offline_env, is_offline
|
|
from headroom.proxy.audit import is_auditable_path
|
|
from headroom.proxy.handlers.openai import OpenAIHandlerMixin
|
|
from headroom.proxy.server import (
|
|
ProxyConfig,
|
|
WebSocketAuthMiddleware,
|
|
create_app,
|
|
scrub_proxy_token_headers,
|
|
)
|
|
|
|
NONLOOPBACK = ("203.0.113.5", 44444) # TEST-NET-3, never loopback
|
|
LOOPBACK = ("127.0.0.1", 12345)
|
|
|
|
|
|
def _make_app(**overrides):
|
|
reset_compression_store()
|
|
config = ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
**overrides,
|
|
)
|
|
return create_app(config)
|
|
|
|
|
|
def test_create_app_accepts_test_net_host_without_transport_validation(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
) -> None:
|
|
# No transport-level validation of the host string — but a non-loopback
|
|
# bind with no token is an *acknowledged* open bind, never a silent one
|
|
# (see test_proxy_bind_policy.py for the refusal path).
|
|
monkeypatch.setenv("HEADROOM_ALLOW_UNAUTHENTICATED_BIND", "1")
|
|
app = _make_app(host="203.0.113.5", proxy_token=None)
|
|
assert app is not None
|
|
|
|
|
|
# ───────────────────────────── 2.1 inbound auth token ─────────────────────
|
|
|
|
|
|
class TestInboundAuthToken:
|
|
def test_no_token_configured_leaves_data_plane_open(self):
|
|
"""Default (no token): non-loopback callers are not challenged."""
|
|
app = _make_app()
|
|
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
|
assert c.get("/livez").status_code == 200
|
|
|
|
def test_token_set_rejects_nonloopback_without_credential(self):
|
|
app = _make_app(proxy_token="s3cr3t-token")
|
|
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
|
resp = c.get("/stats")
|
|
assert resp.status_code == 401
|
|
|
|
def test_token_set_accepts_correct_bearer(self):
|
|
app = _make_app(proxy_token="s3cr3t-token")
|
|
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
|
resp = c.get("/stats", headers={"Authorization": "Bearer s3cr3t-token"})
|
|
assert resp.status_code != 401
|
|
|
|
def test_token_set_accepts_custom_header(self):
|
|
app = _make_app(proxy_token="s3cr3t-token")
|
|
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
|
resp = c.get("/stats", headers={"X-Headroom-Proxy-Token": "s3cr3t-token"})
|
|
assert resp.status_code != 401
|
|
|
|
def test_token_set_accepts_custom_header_with_upstream_oauth(self):
|
|
"""The upstream OAuth bearer must not override the proxy credential."""
|
|
app = _make_app(proxy_token="s3cr3t-token")
|
|
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
|
resp = c.get(
|
|
"/stats",
|
|
headers={
|
|
"Authorization": "Bearer oauth-subscription-token",
|
|
"X-Headroom-Proxy-Token": "s3cr3t-token",
|
|
},
|
|
)
|
|
assert resp.status_code != 401
|
|
|
|
def test_token_set_rejects_wrong_token(self):
|
|
app = _make_app(proxy_token="s3cr3t-token")
|
|
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
|
resp = c.get("/stats", headers={"Authorization": "Bearer wrong"})
|
|
assert resp.status_code == 401
|
|
|
|
def test_loopback_is_exempt_from_token(self):
|
|
"""Loopback callers (same trust boundary as admin routes) skip the token."""
|
|
app = _make_app(proxy_token="s3cr3t-token")
|
|
with TestClient(app, base_url="http://127.0.0.1", client=LOOPBACK) as c:
|
|
assert c.get("/stats").status_code != 401
|
|
|
|
def test_missing_peer_address_is_not_loopback(self):
|
|
"""No ``scope["client"]`` (UDS, adapters) must fail closed, not open.
|
|
|
|
``is_loopback_host(None)`` used to return True, which silently switched
|
|
the token gate off for any transport that leaves the peer unset.
|
|
"""
|
|
app = _make_app(proxy_token="s3cr3t-token")
|
|
with TestClient(app, base_url="http://127.0.0.1", client=None) as c: # type: ignore[arg-type]
|
|
assert c.get("/stats").status_code == 401
|
|
ok = c.get("/stats", headers={"Authorization": "Bearer s3cr3t-token"})
|
|
assert ok.status_code != 401
|
|
|
|
def test_health_endpoints_exempt_even_nonloopback(self):
|
|
"""Orchestrator health probes must work without the token."""
|
|
app = _make_app(proxy_token="s3cr3t-token")
|
|
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
|
assert c.get("/livez").status_code == 200
|
|
assert c.get("/readyz").status_code in (200, 503) # ready/not-ready, never 401
|
|
|
|
@pytest.mark.parametrize(
|
|
("method", "path"),
|
|
[
|
|
("GET", "/healthz"), # no Python route: lands on the catch-all
|
|
("POST", "/health"),
|
|
("PUT", "/livez"),
|
|
("DELETE", "/readyz"),
|
|
("HEAD", "/livez"), # @app.get does not register HEAD
|
|
],
|
|
)
|
|
def test_health_exemption_does_not_reach_passthrough(self, monkeypatch, method, path):
|
|
"""Only GET probes are exempt from the token.
|
|
|
|
Any other request on a health path is not served by the health handler
|
|
but by the catch-all passthrough, which relays it upstream. Exempting
|
|
those turned the proxy into an unauthenticated relay.
|
|
"""
|
|
relayed: list[tuple[str, str]] = []
|
|
|
|
async def _spy_passthrough(self, request, base_url, *args, **kwargs):
|
|
relayed.append((request.method, request.url.path))
|
|
return JSONResponse({"relayed_to": base_url})
|
|
|
|
monkeypatch.setattr(OpenAIHandlerMixin, "handle_passthrough", _spy_passthrough)
|
|
app = _make_app(proxy_token="s3cr3t-token")
|
|
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
|
resp = c.request(method, path)
|
|
assert resp.status_code == 401
|
|
assert relayed == []
|
|
|
|
@pytest.mark.parametrize("path", ["/api/hello", "/p/demo/api/hello"])
|
|
@pytest.mark.parametrize("method", ["GET", "HEAD"])
|
|
@pytest.mark.parametrize("override", [None, "http://127.0.0.1:1", "http://169.254.169.254"])
|
|
def test_api_hello_probe_is_answered_locally(self, monkeypatch, method, path, override):
|
|
"""The SDK reachability probe never carries custom headers, so it can't
|
|
hold the token. It is answered locally: 200, empty, never relayed, also
|
|
under a /p/<project> base URL."""
|
|
relayed: list[tuple[str, str]] = []
|
|
|
|
async def _spy_passthrough(self, request, base_url, *args, **kwargs):
|
|
relayed.append((request.method, request.url.path))
|
|
return JSONResponse({"relayed_to": base_url})
|
|
|
|
monkeypatch.setattr(OpenAIHandlerMixin, "handle_passthrough", _spy_passthrough)
|
|
app = _make_app(proxy_token="s3cr3t-token")
|
|
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
|
resp = c.request(
|
|
method,
|
|
path,
|
|
headers={"x-headroom-base-url": override} if override else None,
|
|
)
|
|
assert resp.status_code == 200
|
|
assert resp.content == b""
|
|
assert resp.headers["X-Content-Type-Options"] == "nosniff"
|
|
assert relayed == []
|
|
|
|
@pytest.mark.parametrize(
|
|
("method", "path"),
|
|
[
|
|
("POST", "/api/hello"),
|
|
("GET", "/api/hello/x"),
|
|
("GET", "/api/hello/"),
|
|
("POST", "/p/demo/api/hello"),
|
|
("GET", "/p/demo/stats"),
|
|
("GET", "/stats"),
|
|
],
|
|
)
|
|
def test_api_hello_answer_is_exact(self, monkeypatch, method, path):
|
|
"""Only GET/HEAD on exactly /api/hello skips the token."""
|
|
relayed: list[tuple[str, str]] = []
|
|
|
|
async def _spy_passthrough(self, request, base_url, *args, **kwargs):
|
|
relayed.append((request.method, request.url.path))
|
|
return JSONResponse({"relayed_to": base_url})
|
|
|
|
monkeypatch.setattr(OpenAIHandlerMixin, "handle_passthrough", _spy_passthrough)
|
|
app = _make_app(proxy_token="s3cr3t-token")
|
|
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
|
resp = c.request(method, path)
|
|
assert resp.status_code == 401
|
|
assert relayed == []
|
|
|
|
def test_api_hello_without_token_is_left_to_the_passthrough(self, monkeypatch):
|
|
"""No token configured: the probe keeps its normal Anthropic route."""
|
|
relayed: list[tuple[str, str]] = []
|
|
|
|
async def _spy_passthrough(self, request, base_url, *args, **kwargs):
|
|
relayed.append((request.method, request.url.path))
|
|
return JSONResponse({"relayed_to": base_url})
|
|
|
|
monkeypatch.setattr(OpenAIHandlerMixin, "handle_passthrough", _spy_passthrough)
|
|
app = _make_app()
|
|
with TestClient(app, base_url="http://127.0.0.1", client=LOOPBACK) as c:
|
|
c.get("/api/hello")
|
|
assert relayed == [("GET", "/api/hello")]
|
|
|
|
|
|
@contextlib.contextmanager
|
|
def _captured_proxy_warnings():
|
|
"""Collect ``headroom.proxy`` log messages (attached to the logger itself:
|
|
the proxy's logging setup controls propagation, so caplog can miss them)."""
|
|
messages: list[str] = []
|
|
|
|
class _Capture(logging.Handler):
|
|
def emit(self, record: logging.LogRecord) -> None:
|
|
messages.append(record.getMessage())
|
|
|
|
handler = _Capture(level=logging.WARNING)
|
|
proxy_logger = logging.getLogger("headroom.proxy")
|
|
previous_level = proxy_logger.level
|
|
proxy_logger.setLevel(logging.WARNING)
|
|
proxy_logger.addHandler(handler)
|
|
try:
|
|
yield messages
|
|
finally:
|
|
proxy_logger.removeHandler(handler)
|
|
proxy_logger.setLevel(previous_level)
|
|
|
|
|
|
def _rejections(messages: list[str]) -> list[str]:
|
|
return [m for m in messages if "event=proxy_auth_rejected" in m]
|
|
|
|
|
|
# Stands in for anything sensitive a caller may put in a header.
|
|
CANARY = "PRIVATE_CREDENTIAL_CANARY"
|
|
|
|
|
|
class TestAuthRejectionLog:
|
|
"""``proxy_auth_rejected`` carries enough to attribute a caller behind a
|
|
gateway (method, User-Agent, leftmost X-Forwarded-For hop), and never a
|
|
credential."""
|
|
|
|
def test_line_carries_method_client_and_first_forwarded_hop(self):
|
|
app = _make_app(proxy_token="s3cr3t-token")
|
|
with _captured_proxy_warnings() as messages:
|
|
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
|
c.get(
|
|
"/stats",
|
|
headers={
|
|
"User-Agent": "claude-cli/2.1.0 (external, cli)",
|
|
"X-Forwarded-For": "198.51.100.7, 10.0.0.9",
|
|
},
|
|
)
|
|
(line,) = _rejections(messages)
|
|
assert "method=GET path=/stats" in line
|
|
assert "reason=missing_token" in line
|
|
assert line.endswith("ua=claude-code xff=198.51.100.7")
|
|
|
|
def test_absent_headers_log_as_none(self):
|
|
app = _make_app(proxy_token="s3cr3t-token")
|
|
with _captured_proxy_warnings() as messages:
|
|
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
|
c.get("/stats", headers={"User-Agent": ""})
|
|
(line,) = _rejections(messages)
|
|
assert line.endswith("ua=none xff=none")
|
|
|
|
@pytest.mark.parametrize(
|
|
("user_agent", "forwarded_for", "tail"),
|
|
[
|
|
(CANARY, f"{CANARY}, 192.0.2.1", "ua=other xff=invalid"),
|
|
(f"{CANARY}/1.0", "192.0.2.1", "ua=other xff=192.0.2.1"),
|
|
(f'curl" xff="{CANARY}', f"fe80::1%{CANARY}", "ua=other xff=invalid"),
|
|
("node", f"192.0.2.1:{CANARY}", "ua=node xff=invalid"),
|
|
(f"{CANARY}/9 opencode/1.2.3", "192.0.2.1", "ua=opencode xff=192.0.2.1"),
|
|
],
|
|
)
|
|
def test_caller_text_never_reaches_the_line(self, user_agent, forwarded_for, tail):
|
|
app = _make_app(proxy_token="s3cr3t-token")
|
|
with _captured_proxy_warnings() as messages:
|
|
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
|
c.get(
|
|
"/stats", headers={"User-Agent": user_agent, "X-Forwarded-For": forwarded_for}
|
|
)
|
|
(line,) = _rejections(messages)
|
|
assert CANARY not in line
|
|
assert line.endswith(tail)
|
|
|
|
def test_path_cannot_smuggle_control_characters(self):
|
|
# The ASGI server percent-decodes the path. Starlette's URL parsing
|
|
# drops CR/LF/TAB from request.url.path, but every other control
|
|
# character (NUL, ESC for terminal escapes, U+2028) reaches the line.
|
|
app = _make_app(proxy_token="s3cr3t-token")
|
|
with _captured_proxy_warnings() as messages:
|
|
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
|
c.get("/stats%0D%0A%1B%5B1Aevent=forged%00%E2%80%A8")
|
|
(line,) = _rejections(messages)
|
|
assert line.isprintable()
|
|
assert "path=/stats\\x1b[1Aevent=forged\\x00\\u2028 client=" in line
|
|
|
|
def test_context_is_a_fixed_vocabulary(self):
|
|
# Exercised directly: an HTTP client refuses to send CR/LF in a header,
|
|
# but an ASGI server in front of the proxy is not guaranteed to.
|
|
from headroom.proxy.server import _auth_rejection_context
|
|
|
|
assert _auth_rejection_context(
|
|
{"user-agent": "curl\r\nevent=forged", "x-forwarded-for": "192.0.2.1\r\nevent=forged"}
|
|
) == ("other", "invalid")
|
|
assert _auth_rejection_context(
|
|
{"user-agent": "Mozilla/5.0 (Windows NT 10.0)", "x-forwarded-for": "2001:DB8::1"}
|
|
) == ("mozilla", "2001:db8::1")
|
|
assert _auth_rejection_context({}) == ("none", "none")
|
|
# X-Client is free text, so it never names the client here.
|
|
assert _auth_rejection_context({"user-agent": "x", "x-client": "claude-code"}) == (
|
|
"other",
|
|
"none",
|
|
)
|
|
|
|
@pytest.mark.parametrize(
|
|
("user_agent", "client"),
|
|
[
|
|
("claude-cli/2.1.0 (external, cli)", "claude-code"),
|
|
("claude-code/2.0.14", "claude-code"),
|
|
("codex-cli/0.46.0", "codex"),
|
|
("Cursor/1.7.0", "cursor"),
|
|
("opencode/1.2.3", "opencode"),
|
|
("CorpWrapper/3.1 aider/0.86", "aider"),
|
|
("Anthropic/Python 0.69.0", "anthropic"),
|
|
("node", "node"),
|
|
],
|
|
)
|
|
def test_known_clients_keep_their_proxy_wide_name(self, user_agent, client):
|
|
# Harnesses log under the same name the rest of the proxy gives them
|
|
# (CLIENT_UA_MAP); generic HTTP clients under their product token.
|
|
from headroom.proxy.server import _auth_rejection_context
|
|
|
|
assert _auth_rejection_context({"user-agent": user_agent}) == (client, "none")
|
|
|
|
@pytest.mark.parametrize(
|
|
"headers",
|
|
[
|
|
{"Authorization": "Bearer wrong-bearer-value"},
|
|
{
|
|
"x-headroom-proxy-token": "wrong-header-value",
|
|
"Authorization": "Bearer sk-ant-oat01-upstream-oauth",
|
|
},
|
|
{"x-api-key": "sk-ant-api03-provider-key"},
|
|
],
|
|
)
|
|
def test_no_credential_ever_reaches_the_line(self, headers):
|
|
app = _make_app(proxy_token="s3cr3t-token")
|
|
with _captured_proxy_warnings() as messages:
|
|
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
|
c.post("/v1/messages", headers={"User-Agent": "claude-cli/2.1", **headers})
|
|
(line,) = _rejections(messages)
|
|
assert "method=POST" in line
|
|
for secret in ("s3cr3t-token", *headers.values()):
|
|
assert secret.removeprefix("Bearer ") not in line
|
|
assert "authorization" not in line.lower()
|
|
|
|
|
|
# ──────────────────── 2.1b inbound auth token over WebSocket ──────────────
|
|
|
|
|
|
WS_PATHS = ("/v1/responses", "/v1/live")
|
|
|
|
|
|
class _SpyApp:
|
|
"""Downstream ASGI app that records whether it was ever reached."""
|
|
|
|
def __init__(self) -> None:
|
|
self.called = False
|
|
|
|
async def __call__(self, scope, receive, send) -> None:
|
|
self.called = True
|
|
|
|
|
|
def _ws_scope(*, client=NONLOOPBACK, headers=(), path="/v1/responses"):
|
|
return {
|
|
"type": "websocket",
|
|
"path": path,
|
|
"client": client,
|
|
"headers": [(k.lower().encode("latin-1"), v.encode("latin-1")) for k, v in headers],
|
|
}
|
|
|
|
|
|
async def _drive(middleware, scope):
|
|
"""Run one connection through the middleware, returning (sent, downstream)."""
|
|
inbox = [{"type": "websocket.connect"}]
|
|
sent: list[dict] = []
|
|
|
|
async def receive():
|
|
return inbox.pop(0) if inbox else {"type": "websocket.disconnect"}
|
|
|
|
async def send(message):
|
|
sent.append(message)
|
|
|
|
await middleware(scope, receive, send)
|
|
return sent
|
|
|
|
|
|
def _closed_with_policy_violation(sent) -> bool:
|
|
return any(m.get("type") == "websocket.close" and m.get("code") == 1008 for m in sent)
|
|
|
|
|
|
class TestWebSocketAuthMiddleware:
|
|
"""The middleware itself, driven directly over ASGI.
|
|
|
|
Asserted at this layer because a pre-accept close surfaces through
|
|
``TestClient`` as a bare ``AttributeError`` — indistinguishable from any
|
|
other handshake failure — so an exception-shape assertion would pass for
|
|
the wrong reason.
|
|
"""
|
|
|
|
async def test_rejects_missing_credential(self):
|
|
downstream = _SpyApp()
|
|
mw = WebSocketAuthMiddleware(downstream, proxy_token="s3cr3t-token")
|
|
|
|
sent = await _drive(mw, _ws_scope())
|
|
|
|
assert downstream.called is False
|
|
assert _closed_with_policy_violation(sent)
|
|
|
|
async def test_rejects_wrong_credential(self):
|
|
downstream = _SpyApp()
|
|
mw = WebSocketAuthMiddleware(downstream, proxy_token="s3cr3t-token")
|
|
|
|
sent = await _drive(mw, _ws_scope(headers=[("authorization", "Bearer wrong")]))
|
|
|
|
assert downstream.called is False
|
|
assert _closed_with_policy_violation(sent)
|
|
|
|
async def test_rejection_log_matches_the_http_gate(self):
|
|
mw = WebSocketAuthMiddleware(_SpyApp(), proxy_token="s3cr3t-token")
|
|
scope = _ws_scope(
|
|
headers=[
|
|
("user-agent", "node"),
|
|
("x-forwarded-for", "198.51.100.8, 10.0.0.1"),
|
|
("authorization", "Bearer wrong-ws-bearer"),
|
|
]
|
|
)
|
|
|
|
with _captured_proxy_warnings() as messages:
|
|
await _drive(mw, scope)
|
|
|
|
(line,) = _rejections(messages)
|
|
assert "transport=websocket" in line and "reason=bad_token" in line
|
|
assert line.endswith("ua=node xff=198.51.100.8")
|
|
|
|
async def test_rejection_log_keeps_caller_text_out(self):
|
|
mw = WebSocketAuthMiddleware(_SpyApp(), proxy_token="s3cr3t-token")
|
|
scope = _ws_scope(
|
|
headers=[
|
|
("user-agent", CANARY),
|
|
("x-forwarded-for", f"{CANARY}, 192.0.2.1"),
|
|
]
|
|
)
|
|
|
|
with _captured_proxy_warnings() as messages:
|
|
await _drive(mw, scope)
|
|
|
|
(line,) = _rejections(messages)
|
|
assert CANARY not in line
|
|
assert line.endswith("ua=other xff=invalid")
|
|
assert "wrong-ws-bearer" not in line and "s3cr3t-token" not in line
|
|
|
|
async def test_rejection_log_path_cannot_forge_a_line(self):
|
|
mw = WebSocketAuthMiddleware(_SpyApp(), proxy_token="s3cr3t-token")
|
|
scope = _ws_scope(path="/v1/responses\r\nevent=proxy_auth_rejected")
|
|
|
|
with _captured_proxy_warnings() as messages:
|
|
await _drive(mw, scope)
|
|
|
|
(line,) = _rejections(messages)
|
|
assert "\r" not in line and "\n" not in line
|
|
assert "path=/v1/responses\\r\\nevent=proxy_auth_rejected " in line
|
|
|
|
async def test_accepts_correct_bearer(self):
|
|
downstream = _SpyApp()
|
|
mw = WebSocketAuthMiddleware(downstream, proxy_token="s3cr3t-token")
|
|
|
|
sent = await _drive(mw, _ws_scope(headers=[("authorization", "Bearer s3cr3t-token")]))
|
|
|
|
assert downstream.called is True
|
|
assert not _closed_with_policy_violation(sent)
|
|
|
|
async def test_accepts_custom_header(self):
|
|
downstream = _SpyApp()
|
|
mw = WebSocketAuthMiddleware(downstream, proxy_token="s3cr3t-token")
|
|
|
|
sent = await _drive(mw, _ws_scope(headers=[("x-headroom-proxy-token", "s3cr3t-token")]))
|
|
|
|
assert downstream.called is True
|
|
assert not _closed_with_policy_violation(sent)
|
|
|
|
async def test_accepts_custom_header_with_upstream_oauth(self):
|
|
"""The upstream OAuth bearer must not override the proxy credential."""
|
|
downstream = _SpyApp()
|
|
mw = WebSocketAuthMiddleware(downstream, proxy_token="s3cr3t-token")
|
|
|
|
sent = await _drive(
|
|
mw,
|
|
_ws_scope(
|
|
headers=[
|
|
("authorization", "Bearer oauth-subscription-token"),
|
|
("x-headroom-proxy-token", "s3cr3t-token"),
|
|
]
|
|
),
|
|
)
|
|
|
|
assert downstream.called is True
|
|
assert not _closed_with_policy_violation(sent)
|
|
|
|
async def test_loopback_is_exempt(self):
|
|
"""Same trust boundary the HTTP gate already grants loopback.
|
|
|
|
Both gates the HTTP admin guards apply: loopback peer *and* loopback
|
|
``Host`` header (a real handshake always carries one).
|
|
"""
|
|
downstream = _SpyApp()
|
|
mw = WebSocketAuthMiddleware(downstream, proxy_token="s3cr3t-token")
|
|
|
|
sent = await _drive(mw, _ws_scope(client=LOOPBACK, headers=[("host", "127.0.0.1:8787")]))
|
|
|
|
assert downstream.called is True
|
|
assert not _closed_with_policy_violation(sent)
|
|
|
|
async def test_loopback_peer_with_foreign_host_header_is_not_exempt(self):
|
|
"""DNS rebinding: a local browser reaching 127.0.0.1 still says Host: attacker.
|
|
|
|
The HTTP admin routes have checked Host since the loopback guard was
|
|
written; the WebSocket gate checked the peer only. Same rule now.
|
|
"""
|
|
downstream = _SpyApp()
|
|
mw = WebSocketAuthMiddleware(downstream, proxy_token="s3cr3t-token")
|
|
|
|
sent = await _drive(
|
|
mw, _ws_scope(client=LOOPBACK, headers=[("host", "attacker.example:8787")])
|
|
)
|
|
|
|
assert downstream.called is False
|
|
assert _closed_with_policy_violation(sent)
|
|
|
|
async def test_loopback_peer_with_foreign_host_but_valid_token_passes(self):
|
|
"""The Host check only decides the *exemption*; the token still works."""
|
|
downstream = _SpyApp()
|
|
mw = WebSocketAuthMiddleware(downstream, proxy_token="s3cr3t-token")
|
|
|
|
sent = await _drive(
|
|
mw,
|
|
_ws_scope(
|
|
client=LOOPBACK,
|
|
headers=[
|
|
("host", "attacker.example:8787"),
|
|
("authorization", "Bearer s3cr3t-token"),
|
|
],
|
|
),
|
|
)
|
|
|
|
assert downstream.called is True
|
|
assert not _closed_with_policy_violation(sent)
|
|
|
|
async def test_unknown_client_is_not_loopback(self):
|
|
"""Mirrors is_loopback_host(None) -> False: fail closed, like the HTTP gate."""
|
|
downstream = _SpyApp()
|
|
mw = WebSocketAuthMiddleware(downstream, proxy_token="s3cr3t-token")
|
|
|
|
sent = await _drive(mw, _ws_scope(client=None, headers=[("host", "127.0.0.1")]))
|
|
|
|
assert downstream.called is False
|
|
assert _closed_with_policy_violation(sent)
|
|
|
|
async def test_repeated_header_resolves_like_the_http_gate(self):
|
|
"""A duplicated Authorization must mean the same thing on both transports.
|
|
|
|
Starlette's Headers (what the HTTP gate reads) returns the FIRST
|
|
occurrence. A hand-built dict returns the last, which would let the two
|
|
paths disagree about which credential counted.
|
|
"""
|
|
downstream = _SpyApp()
|
|
mw = WebSocketAuthMiddleware(downstream, proxy_token="s3cr3t-token")
|
|
|
|
sent = await _drive(
|
|
mw,
|
|
_ws_scope(
|
|
headers=[
|
|
("authorization", "Bearer s3cr3t-token"),
|
|
("authorization", "Bearer wrong"),
|
|
]
|
|
),
|
|
)
|
|
|
|
# First header wins → authenticated, same as the HTTP gate.
|
|
assert downstream.called is True
|
|
assert not _closed_with_policy_violation(sent)
|
|
|
|
async def test_no_token_configured_is_a_passthrough(self):
|
|
"""Default deployment must gain no new challenge."""
|
|
downstream = _SpyApp()
|
|
mw = WebSocketAuthMiddleware(downstream, proxy_token=None)
|
|
|
|
sent = await _drive(mw, _ws_scope())
|
|
|
|
assert downstream.called is True
|
|
assert not _closed_with_policy_violation(sent)
|
|
|
|
async def test_http_scope_is_left_to_the_http_gate(self):
|
|
downstream = _SpyApp()
|
|
mw = WebSocketAuthMiddleware(downstream, proxy_token="s3cr3t-token")
|
|
|
|
sent = await _drive(mw, {**_ws_scope(), "type": "http"})
|
|
|
|
assert downstream.called is True
|
|
assert not _closed_with_policy_violation(sent)
|
|
|
|
|
|
class TestWebSocketRoutesAreGatedInTheApp:
|
|
"""The middleware is actually wired into ``create_app``.
|
|
|
|
Asserts the security property directly — the route handler must never run
|
|
for an unauthenticated handshake — rather than inspecting the exception the
|
|
client happens to see.
|
|
"""
|
|
|
|
@pytest.mark.parametrize("path", WS_PATHS)
|
|
def test_unauthenticated_handshake_never_reaches_the_handler(self, path, monkeypatch):
|
|
app = _make_app(proxy_token="s3cr3t-token")
|
|
reached = _record_ws_handler_reached(app, monkeypatch)
|
|
|
|
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
|
try:
|
|
with c.websocket_connect(path):
|
|
pass
|
|
except Exception: # noqa: BLE001 - the refusal shape is asserted above
|
|
pass
|
|
|
|
assert reached() is False
|
|
|
|
@pytest.mark.parametrize("path", WS_PATHS)
|
|
def test_authenticated_handshake_reaches_the_handler(self, path, monkeypatch):
|
|
app = _make_app(proxy_token="s3cr3t-token")
|
|
reached = _record_ws_handler_reached(app, monkeypatch)
|
|
|
|
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
|
try:
|
|
with c.websocket_connect(path, headers={"X-Headroom-Proxy-Token": "s3cr3t-token"}):
|
|
pass
|
|
except Exception: # noqa: BLE001 - route may fail with no upstream
|
|
pass
|
|
|
|
assert reached() is True
|
|
|
|
|
|
def _record_ws_handler_reached(app, monkeypatch):
|
|
"""Spy both WebSocket route families; returns a callable reporting arrival."""
|
|
from headroom.providers import proxy_routes
|
|
|
|
seen: list[str] = []
|
|
|
|
# Each spy must terminate the handshake itself: a handler that returns
|
|
# without accepting or closing leaves the client waiting forever.
|
|
async def _responses_spy(websocket):
|
|
seen.append("responses")
|
|
await websocket.close(code=1000)
|
|
|
|
async def _live_spy(websocket, *args, **kwargs):
|
|
seen.append("live")
|
|
await websocket.close(code=1000)
|
|
|
|
monkeypatch.setattr(app.state.proxy, "handle_openai_responses_ws", _responses_spy)
|
|
monkeypatch.setattr(proxy_routes, "handle_codex_live_websocket", _live_spy)
|
|
return lambda: bool(seen)
|
|
|
|
|
|
# ─────────────── 2.1c the proxy token never reaches an upstream ───────────
|
|
|
|
|
|
TOKEN = "s3cr3t-token"
|
|
|
|
|
|
class _CapturingTransport(httpx.AsyncBaseTransport):
|
|
"""Upstream stand-in that records every request the proxy sends it."""
|
|
|
|
def __init__(self) -> None:
|
|
self.requests: list[httpx.Request] = []
|
|
|
|
async def handle_async_request(self, request: httpx.Request) -> httpx.Response:
|
|
self.requests.append(request)
|
|
if "chat/completions" in str(request.url):
|
|
body = {
|
|
"id": "chatcmpl-mock",
|
|
"object": "chat.completion",
|
|
"created": 1,
|
|
"model": "gpt-4o",
|
|
"choices": [
|
|
{
|
|
"index": 0,
|
|
"message": {"role": "assistant", "content": "hi"},
|
|
"finish_reason": "stop",
|
|
}
|
|
],
|
|
"usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2},
|
|
}
|
|
elif "/v1/messages" in str(request.url):
|
|
body = {
|
|
"id": "msg_mock",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"content": [{"type": "text", "text": "hi"}],
|
|
"usage": {"input_tokens": 1, "output_tokens": 1},
|
|
}
|
|
else:
|
|
body = {"object": "list", "data": []}
|
|
return httpx.Response(200, headers={"content-type": "application/json"}, json=body)
|
|
|
|
|
|
@contextlib.contextmanager
|
|
def _upstream_capturing_client(*, client=NONLOOPBACK, base_url="http://testserver"):
|
|
reset_compression_store()
|
|
config = ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
image_optimize=False,
|
|
anthropic_api_url="https://api.anthropic.test",
|
|
openai_api_url="https://api.openai.test",
|
|
proxy_token=TOKEN,
|
|
)
|
|
app = create_app(config)
|
|
transport = _CapturingTransport()
|
|
with TestClient(app, base_url=base_url, client=client) as c:
|
|
# Startup builds the real upstream client, so swap it only after
|
|
# entering the lifespan or requests would leave the test process.
|
|
app.state.proxy.http_client = httpx.AsyncClient(transport=transport)
|
|
yield c, transport
|
|
|
|
|
|
def _anthropic_request(c, headers):
|
|
return c.post(
|
|
"/v1/messages",
|
|
headers={"anthropic-version": "2023-06-01", **headers},
|
|
json={
|
|
"model": "claude-3-5-sonnet-20241022",
|
|
"messages": [{"role": "user", "content": "Hello"}],
|
|
"max_tokens": 16,
|
|
},
|
|
)
|
|
|
|
|
|
def _only_upstream_request(transport) -> httpx.Request:
|
|
assert len(transport.requests) == 1, transport.requests
|
|
return transport.requests[0]
|
|
|
|
|
|
class TestProxyTokenIsNotForwardedUpstream:
|
|
"""The proxy credential authenticates to Headroom only.
|
|
|
|
Handlers build upstream headers from the inbound request and strip only
|
|
``x-headroom-*``, so a token sent as ``Authorization: Bearer`` used to be
|
|
relayed to the provider verbatim. These tests capture what actually leaves
|
|
the proxy.
|
|
"""
|
|
|
|
def test_bearer_token_is_dropped_and_provider_key_kept(self):
|
|
with _upstream_capturing_client() as (c, transport):
|
|
resp = _anthropic_request(
|
|
c, {"x-api-key": "sk-ant-test", "Authorization": f"Bearer {TOKEN}"}
|
|
)
|
|
assert resp.status_code == 200, resp.text
|
|
|
|
upstream = _only_upstream_request(transport)
|
|
assert "authorization" not in upstream.headers
|
|
assert upstream.headers["x-api-key"] == "sk-ant-test"
|
|
assert TOKEN not in str(upstream.headers.raw)
|
|
|
|
def test_lowercase_scheme_is_dropped_too(self):
|
|
"""The gate accepts ``bearer`` in any case, so the scrub must as well."""
|
|
with _upstream_capturing_client() as (c, transport):
|
|
resp = _anthropic_request(
|
|
c, {"x-api-key": "sk-ant-test", "Authorization": f"bearer {TOKEN}"}
|
|
)
|
|
assert resp.status_code == 200, resp.text
|
|
assert "authorization" not in _only_upstream_request(transport).headers
|
|
|
|
def test_custom_header_is_dropped_and_provider_bearer_kept(self):
|
|
with _upstream_capturing_client() as (c, transport):
|
|
resp = c.post(
|
|
"/v1/chat/completions",
|
|
headers={"X-Headroom-Proxy-Token": TOKEN, "Authorization": "Bearer sk-test"},
|
|
json={"model": "gpt-4o", "messages": [{"role": "user", "content": "Hello"}]},
|
|
)
|
|
assert resp.status_code == 200, resp.text
|
|
|
|
upstream = _only_upstream_request(transport)
|
|
assert upstream.headers["authorization"] == "Bearer sk-test"
|
|
assert "x-headroom-proxy-token" not in upstream.headers
|
|
|
|
def test_custom_header_is_dropped_even_with_internal_strip_disabled(self, monkeypatch):
|
|
"""``HEADROOM_STRIP_INTERNAL_HEADERS=disabled`` must not leak the credential."""
|
|
monkeypatch.setenv("HEADROOM_STRIP_INTERNAL_HEADERS", "disabled")
|
|
with _upstream_capturing_client() as (c, transport):
|
|
resp = _anthropic_request(
|
|
c, {"x-api-key": "sk-ant-test", "X-Headroom-Proxy-Token": TOKEN}
|
|
)
|
|
assert resp.status_code == 200, resp.text
|
|
assert "x-headroom-proxy-token" not in _only_upstream_request(transport).headers
|
|
|
|
def test_loopback_caller_token_is_dropped(self):
|
|
"""Loopback skips the check, but its token is still not the provider's."""
|
|
with _upstream_capturing_client(client=LOOPBACK, base_url="http://127.0.0.1") as (
|
|
c,
|
|
transport,
|
|
):
|
|
resp = _anthropic_request(
|
|
c, {"x-api-key": "sk-ant-test", "Authorization": f"Bearer {TOKEN}"}
|
|
)
|
|
assert resp.status_code == 200, resp.text
|
|
assert "authorization" not in _only_upstream_request(transport).headers
|
|
|
|
def test_catch_all_passthrough_drops_bearer_token(self):
|
|
with _upstream_capturing_client() as (c, transport):
|
|
resp = c.get(
|
|
"/v1/models",
|
|
headers={"x-api-key": "sk-ant-test", "Authorization": f"Bearer {TOKEN}"},
|
|
)
|
|
assert resp.status_code == 200, resp.text
|
|
upstream = _only_upstream_request(transport)
|
|
assert "authorization" not in upstream.headers
|
|
assert upstream.headers["x-api-key"] == "sk-ant-test"
|
|
|
|
|
|
class TestScrubProxyTokenHeaders:
|
|
"""The scope-level scrub both gates share."""
|
|
|
|
@staticmethod
|
|
def _scope(*headers):
|
|
return {"headers": [(k.encode("latin-1"), v.encode("latin-1")) for k, v in headers]}
|
|
|
|
def test_drops_matching_bearer_keeps_other_headers(self):
|
|
scope = self._scope(("authorization", f"Bearer {TOKEN}"), ("x-api-key", "sk-ant"))
|
|
scrub_proxy_token_headers(scope, TOKEN.encode())
|
|
assert scope["headers"] == [(b"x-api-key", b"sk-ant")]
|
|
|
|
def test_keeps_non_matching_bearer(self):
|
|
scope = self._scope(("authorization", "Bearer sk-provider"))
|
|
scrub_proxy_token_headers(scope, TOKEN.encode())
|
|
assert scope["headers"] == [(b"authorization", b"Bearer sk-provider")]
|
|
|
|
def test_keeps_non_bearer_authorization(self):
|
|
scope = self._scope(("authorization", f"Basic {TOKEN}"))
|
|
scrub_proxy_token_headers(scope, TOKEN.encode())
|
|
assert scope["headers"] == [(b"authorization", f"Basic {TOKEN}".encode())]
|
|
|
|
def test_duplicate_authorization_drops_only_the_token(self):
|
|
scope = self._scope(
|
|
("authorization", f"Bearer {TOKEN}"),
|
|
("authorization", "Bearer sk-provider"),
|
|
)
|
|
scrub_proxy_token_headers(scope, TOKEN.encode())
|
|
assert scope["headers"] == [(b"authorization", b"Bearer sk-provider")]
|
|
|
|
def test_custom_header_dropped_even_without_configured_token(self):
|
|
scope = self._scope(("x-headroom-proxy-token", "anything"), ("x-api-key", "sk-ant"))
|
|
scrub_proxy_token_headers(scope, b"")
|
|
assert scope["headers"] == [(b"x-api-key", b"sk-ant")]
|
|
|
|
def test_no_configured_token_leaves_authorization_alone(self):
|
|
scope = self._scope(("authorization", f"Bearer {TOKEN}"))
|
|
scrub_proxy_token_headers(scope, b"")
|
|
assert scope["headers"] == [(b"authorization", f"Bearer {TOKEN}".encode())]
|
|
|
|
def test_non_ascii_header_value_does_not_raise(self):
|
|
scope = {"headers": [(b"authorization", "Bearer café".encode("latin-1"))]}
|
|
scrub_proxy_token_headers(scope, TOKEN.encode())
|
|
assert len(scope["headers"]) == 1
|
|
|
|
|
|
class _HeaderSpyApp:
|
|
"""Downstream ASGI app that records the headers it was handed."""
|
|
|
|
def __init__(self) -> None:
|
|
self.headers: list[tuple[bytes, bytes]] | None = None
|
|
|
|
async def __call__(self, scope, receive, send) -> None:
|
|
self.headers = list(scope["headers"])
|
|
|
|
|
|
class TestWebSocketScrubsProxyToken:
|
|
@pytest.mark.parametrize("client", [NONLOOPBACK, LOOPBACK])
|
|
async def test_bearer_token_removed_before_the_app(self, client):
|
|
downstream = _HeaderSpyApp()
|
|
mw = WebSocketAuthMiddleware(downstream, proxy_token=TOKEN)
|
|
|
|
await _drive(
|
|
mw,
|
|
_ws_scope(
|
|
client=client,
|
|
headers=[("authorization", f"Bearer {TOKEN}"), ("x-api-key", "sk-ant")],
|
|
),
|
|
)
|
|
|
|
assert downstream.headers == [(b"x-api-key", b"sk-ant")]
|
|
|
|
@pytest.mark.parametrize("token_header", ["authorization", "x-headroom-proxy-token"])
|
|
async def test_loopback_exempt_handshake_is_still_scrubbed(self, token_header):
|
|
# Loopback peer *and* loopback Host takes the token exemption; the
|
|
# exemption must not skip the scrub, or a local client that sends the
|
|
# token anyway has it forwarded upstream.
|
|
downstream = _HeaderSpyApp()
|
|
mw = WebSocketAuthMiddleware(downstream, proxy_token=TOKEN)
|
|
value = f"Bearer {TOKEN}" if token_header == "authorization" else TOKEN
|
|
|
|
await _drive(
|
|
mw,
|
|
_ws_scope(
|
|
client=LOOPBACK,
|
|
headers=[
|
|
("host", "127.0.0.1:8787"),
|
|
(token_header, value),
|
|
("x-api-key", "sk-ant"),
|
|
],
|
|
),
|
|
)
|
|
|
|
assert downstream.headers == [(b"host", b"127.0.0.1:8787"), (b"x-api-key", b"sk-ant")]
|
|
|
|
async def test_custom_header_removed_provider_bearer_kept(self):
|
|
downstream = _HeaderSpyApp()
|
|
mw = WebSocketAuthMiddleware(downstream, proxy_token=TOKEN)
|
|
|
|
await _drive(
|
|
mw,
|
|
_ws_scope(
|
|
headers=[
|
|
("authorization", "Bearer sk-provider"),
|
|
("x-headroom-proxy-token", TOKEN),
|
|
]
|
|
),
|
|
)
|
|
|
|
assert downstream.headers == [(b"authorization", b"Bearer sk-provider")]
|
|
|
|
def test_responses_route_handler_never_sees_the_token(self, monkeypatch):
|
|
app = _make_app(proxy_token=TOKEN)
|
|
seen: list[dict[str, str]] = []
|
|
|
|
async def _responses_spy(websocket):
|
|
seen.append(dict(websocket.headers))
|
|
await websocket.close(code=1000)
|
|
|
|
monkeypatch.setattr(app.state.proxy, "handle_openai_responses_ws", _responses_spy)
|
|
|
|
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
|
try:
|
|
with c.websocket_connect(
|
|
"/v1/responses", headers={"Authorization": f"Bearer {TOKEN}"}
|
|
):
|
|
pass
|
|
except Exception: # noqa: BLE001 - spy closes the socket immediately
|
|
pass
|
|
|
|
assert len(seen) == 1
|
|
assert "authorization" not in seen[0]
|
|
|
|
|
|
# ───────────────────────────── 3.1 security headers ───────────────────────
|
|
|
|
|
|
class TestSecurityHeaders:
|
|
def test_headers_present_on_responses(self):
|
|
app = _make_app()
|
|
with TestClient(app, base_url="http://127.0.0.1", client=LOOPBACK) as c:
|
|
h = c.get("/livez").headers
|
|
assert h.get("X-Content-Type-Options") == "nosniff"
|
|
assert h.get("X-Frame-Options") == "DENY"
|
|
assert h.get("Referrer-Policy") == "no-referrer"
|
|
assert "max-age=" in h.get("Strict-Transport-Security", "")
|
|
|
|
def test_headers_present_on_401(self):
|
|
app = _make_app(proxy_token="s3cr3t-token")
|
|
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
|
resp = c.get("/stats")
|
|
assert resp.status_code == 401
|
|
assert resp.headers.get("X-Content-Type-Options") == "nosniff"
|
|
|
|
|
|
# ───────────────────────────── 2.4 admin audit log ────────────────────────
|
|
|
|
|
|
class TestAdminAuditLog:
|
|
def test_auditable_path_classification(self):
|
|
assert is_auditable_path("/admin/runtime-env")
|
|
assert is_auditable_path("/cache/clear")
|
|
assert is_auditable_path("/stats/reset")
|
|
assert not is_auditable_path("/v1/messages")
|
|
assert not is_auditable_path("/livez")
|
|
|
|
def test_cache_clear_emits_audit_event(self):
|
|
# Capture the dedicated audit logger directly (the proxy's logging setup
|
|
# configures propagation, so attach to the logger rather than rely on
|
|
# caplog's root handler).
|
|
messages: list[str] = []
|
|
|
|
class _Capture(logging.Handler):
|
|
def emit(self, record: logging.LogRecord) -> None:
|
|
messages.append(record.getMessage())
|
|
|
|
handler = _Capture()
|
|
audit_logger = logging.getLogger("headroom.audit")
|
|
audit_logger.setLevel(logging.INFO)
|
|
audit_logger.addHandler(handler)
|
|
try:
|
|
app = _make_app()
|
|
with TestClient(app, base_url="http://127.0.0.1", client=LOOPBACK) as c:
|
|
assert c.post("/cache/clear").status_code == 200
|
|
finally:
|
|
audit_logger.removeHandler(handler)
|
|
|
|
assert messages, "expected an audit record for /cache/clear"
|
|
assert any("/cache/clear" in m for m in messages)
|
|
assert any("headroom_admin_audit" in m for m in messages)
|
|
assert any('"source_ip": "127.0.0.1"' in m for m in messages)
|
|
|
|
|
|
# ───────────────────────────── 2.2 air-gap switch ─────────────────────────
|
|
|
|
|
|
class TestOfflineSwitch:
|
|
def test_is_offline_reads_env(self, monkeypatch):
|
|
monkeypatch.delenv("HEADROOM_OFFLINE", raising=False)
|
|
assert is_offline() is False
|
|
monkeypatch.setenv("HEADROOM_OFFLINE", "1")
|
|
assert is_offline() is True
|
|
monkeypatch.setenv("HEADROOM_OFFLINE", "off")
|
|
assert is_offline() is False
|
|
|
|
def test_offline_disables_telemetry(self, monkeypatch):
|
|
from headroom.telemetry.beacon import is_telemetry_enabled
|
|
|
|
monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
|
|
monkeypatch.setenv("HEADROOM_OFFLINE", "1")
|
|
assert is_telemetry_enabled() is False # offline overrides the opt-in
|
|
|
|
def test_offline_disables_update_check(self, monkeypatch):
|
|
from headroom.update_check import is_update_check_enabled
|
|
|
|
monkeypatch.delenv("CI", raising=False)
|
|
monkeypatch.delenv("HEADROOM_STATELESS", raising=False)
|
|
monkeypatch.setenv("HEADROOM_OFFLINE", "1")
|
|
assert is_update_check_enabled() is False
|
|
|
|
def test_apply_offline_env_sets_hf_offline(self, monkeypatch):
|
|
monkeypatch.delenv("HF_HUB_OFFLINE", raising=False)
|
|
monkeypatch.delenv("TRANSFORMERS_OFFLINE", raising=False)
|
|
monkeypatch.setenv("HEADROOM_OFFLINE", "1")
|
|
apply_offline_env()
|
|
import os
|
|
|
|
assert os.environ.get("HF_HUB_OFFLINE") == "1"
|
|
assert os.environ.get("TRANSFORMERS_OFFLINE") == "1"
|
|
|
|
|
|
# ─────────────────── missing peer address fails closed everywhere ───────────
|
|
|
|
|
|
def test_is_loopback_host_none_is_not_loopback() -> None:
|
|
from headroom.proxy.loopback_guard import is_loopback_host
|
|
|
|
assert is_loopback_host(None) is False
|
|
# Sanity: the genuine loopback literals are unaffected.
|
|
assert is_loopback_host("127.0.0.1") is True
|
|
assert is_loopback_host("::1") is True
|
|
assert is_loopback_host("localhost") is True
|
|
|
|
|
|
def test_require_loopback_404s_a_request_with_no_peer() -> None:
|
|
"""Admin/debug guards must not open up when ``request.client`` is None."""
|
|
from fastapi import HTTPException
|
|
|
|
from headroom.proxy.loopback_guard import require_loopback
|
|
|
|
class _NoPeerRequest:
|
|
client = None
|
|
headers = {"host": "127.0.0.1:8787"}
|
|
|
|
with pytest.raises(HTTPException) as exc_info:
|
|
require_loopback(_NoPeerRequest()) # type: ignore[arg-type]
|
|
assert exc_info.value.status_code == 404
|
|
|
|
|
|
def test_debug_route_404s_for_request_with_no_peer() -> None:
|
|
app = _make_app()
|
|
with TestClient(app, base_url="http://127.0.0.1", client=None) as c: # type: ignore[arg-type]
|
|
assert c.get("/debug/tasks").status_code == 404
|