mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(oauth2): mint and inject only on requests that go upstream (#3849)
## Why `OAuth2Middleware` rewrote `Authorization` on every HTTP request (findings 03-F4 and 03-F5). Routes the proxy answers itself (`/health`, `/stats*`, `/metrics`, `/v1/compress*`, extension `/ext/*` routes and the rest of the management surface) never go upstream. Yet each one minted a token when the cache was cold, and an unreachable IdP turned health probes and management calls into 502. Separately, the startup log printed the full token URL, which can carry a query string or embedded credentials, and the mint log printed the scope list. ## What changes - The bearer is minted and injected only for requests that go upstream. `LOCAL_ROUTE_PATHS` and `LOCAL_ROUTE_PREFIXES` list what the proxy serves itself; a `/p/<project>/` prefix is stripped first with the core's `split_project_path`. Everything else, including the provider passthrough catch-all, is treated as upstream, so a new provider route needs no change here. Only a new local route does. - Startup log: `scheme://host` of the token URL instead of the full URL. Mint log: the scope count instead of the list. - Version 0.1.1 in both `pyproject.toml` and `__version__`. Plugin `CHANGELOG.md` and README updated. Inbound authentication is not this plugin's job. From headroom-ai 0.40, extension middleware runs inside the `HEADROOM_PROXY_TOKEN` gate (#3847), so an unauthenticated request never reaches it, and a client may authenticate with `Authorization: Bearer <proxy token>` before this plugin replaces that header with the upstream bearer. ## Behaviour change / upgrade notes - Requires headroom-ai >= 0.40. The README says so. - Requests to local routes no longer carry a minted bearer. Nothing used it, but a log line or metric keyed on `Authorization` for those paths will change. ## Tests `plugins/headroom-oauth2/tests/test_scope_and_proxy_token.py`: - Every local route, including a project-prefixed one, passes through with `Authorization` untouched and no mint. - Upstream routes, including a project-prefixed route, the passthrough catch-all and `/`, get the minted bearer. - The install log line carries only `scheme://host`. - End to end through `create_app` from a non-loopback client with an unreachable IdP: `/health` is 200; `/stats` with the proxy-token bearer is 200; `/stats` with no credential is 401; an upstream route with the proxy-token bearer gets 502 `upstream_auth_error` from oauth2, not 401 from the gate. - End to end with a local IdP: the proxy-token bearer passes the gate and is replaced by the minted bearer, `x-headroom-proxy-token` is not added, and one mint happens. A request with no credential is refused with 401 and costs no mint. This plugin has no CI job yet (only `plugins/headroom-agent-hooks` is path-filtered in), so these tests were run locally. ## Merge order Part of a set. Suggested order: #3852 → #3891 → #3849 → #3860 → #3862 → #3863. This PR is independent of the others; the plugin only needs a core >= 0.40. ## Follow-ups - Add a path-filtered CI job for `plugins/headroom-oauth2`. Revised 2026-10-01 after self-review: removed the plugin's own proxy-token handling (now done by the core gate) and the `HEADROOM_OAUTH2_LOCAL_PATHS` setting. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
0147cf0220
commit
f977d525e4
@@ -1,5 +1,16 @@
|
||||
# Changelog
|
||||
|
||||
## 0.1.1
|
||||
|
||||
- Inject the upstream bearer only on requests that go upstream. Local routes
|
||||
(`/health`, `/stats*`, `/metrics`, `/v1/compress*`, `/ext/*`, …; see README)
|
||||
are passed through untouched, so an unreachable IdP no longer 502s health
|
||||
probes and management calls no longer mint tokens. `/p/<project>/` prefixes
|
||||
are stripped first.
|
||||
- Startup log carries `scheme://host` of the token URL and the mint log a scope
|
||||
count, instead of the full URL and scope list.
|
||||
- Requires `headroom-ai` ≥ 0.40 (extension middleware inside the proxy-token gate).
|
||||
|
||||
## 0.1.0
|
||||
|
||||
Initial release — generic OAuth2 client-credentials upstream-auth extension for the Headroom proxy.
|
||||
|
||||
@@ -41,3 +41,19 @@ ignore it, so this extension is a no-op there (it logs a warning at startup).
|
||||
**Transport:** `token_url` must be `https` (loopback `http` is allowed for tests; set
|
||||
`HEADROOM_OAUTH2_ALLOW_INSECURE=1` to override). Tokens are minted with the standard library
|
||||
(`urllib`, system cert store), so a corporate-injected CA is trusted without bundling roots.
|
||||
|
||||
## What gets the upstream bearer
|
||||
|
||||
The minted token is injected only on requests that **go upstream**. Routes the
|
||||
proxy answers itself — `/health`, `/livez`, `/readyz`, `/stats*`, `/metrics`,
|
||||
`/quota`, `/settings*`, `/dashboard*`, `/admin/*`, `/debug/*`, `/v1/compress*`,
|
||||
`/v1/usage`, `/v1/retrieve*`, `/v1/telemetry*`, `/v1/toin*`, `/v1/feedback*`,
|
||||
and any extension route under `/ext/` — are passed through untouched, so an
|
||||
unreachable IdP never turns a health probe into a 502 and management calls never
|
||||
cost a token mint. A `/p/<project>/` base-URL prefix is stripped before the
|
||||
check.
|
||||
|
||||
Requires `headroom-ai` ≥ 0.40, where extension middleware runs inside the
|
||||
proxy's `HEADROOM_PROXY_TOKEN` gate: clients may then authenticate with
|
||||
`Authorization: Bearer <proxy token>` and the gate checks it before this
|
||||
extension replaces it with the upstream bearer.
|
||||
|
||||
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "headroom-oauth2"
|
||||
version = "0.1.0"
|
||||
version = "0.1.1"
|
||||
description = "Generic OAuth2 client-credentials upstream-auth extension for the Headroom proxy"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.10"
|
||||
|
||||
@@ -8,13 +8,14 @@ from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import urllib.parse
|
||||
from typing import Any
|
||||
|
||||
from .middleware import OAuth2Middleware
|
||||
from .provider import OAuth2ClientCredentials, OAuth2Error
|
||||
|
||||
__all__ = ["OAuth2ClientCredentials", "OAuth2Error", "OAuth2Middleware", "install", "parse_headers"]
|
||||
__version__ = "0.1.0"
|
||||
__version__ = "0.1.1"
|
||||
log = logging.getLogger("headroom_oauth2")
|
||||
|
||||
|
||||
@@ -44,6 +45,19 @@ def parse_headers(s: str | None) -> dict[str, str]:
|
||||
return out
|
||||
|
||||
|
||||
def _redact_url(url: str) -> str:
|
||||
"""``scheme://host[:port]`` only -- a tenant token URL can carry a query string or
|
||||
embedded credentials, neither of which belongs in a startup log line."""
|
||||
try:
|
||||
parts = urllib.parse.urlsplit(url)
|
||||
host = parts.hostname or "?"
|
||||
if parts.port:
|
||||
host = f"{host}:{parts.port}"
|
||||
return f"{parts.scheme}://{host}"
|
||||
except ValueError:
|
||||
return "<unparseable>"
|
||||
|
||||
|
||||
def _int(env, key):
|
||||
raw = env.get(key)
|
||||
if raw is None or not str(raw).strip():
|
||||
@@ -137,6 +151,6 @@ def install(app: Any, config: Any) -> None:
|
||||
app.add_middleware(OAuth2Middleware, provider=provider)
|
||||
log.info(
|
||||
"headroom-oauth2: client-credentials auth installed (token_url=%s, style=%s)",
|
||||
provider.token_url,
|
||||
_redact_url(provider.token_url),
|
||||
provider.auth_style,
|
||||
)
|
||||
|
||||
@@ -3,6 +3,16 @@
|
||||
Headroom's litellm backend forwards the request's `Authorization` bearer to the
|
||||
upstream as the API key, so setting it here makes the minted token reach the
|
||||
backend with no core changes.
|
||||
|
||||
Requests the proxy answers itself are left untouched. Health probes, ``/stats``,
|
||||
``/metrics``, the compress/retrieve/telemetry routes and any extension's own
|
||||
``/ext/...`` routes never go upstream, so there is nothing to inject and no
|
||||
reason to mint. Before this rule ``/health`` returned 502 whenever the IdP was
|
||||
unreachable, and every management request cost a mint when the cache was cold.
|
||||
|
||||
Inbound authentication is not this layer's job: on headroom-ai >= 0.40 extension
|
||||
middleware runs inside the ``HEADROOM_PROXY_TOKEN`` gate, so an unauthenticated
|
||||
request never reaches it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -11,10 +21,46 @@ import asyncio
|
||||
import json
|
||||
import logging
|
||||
|
||||
from headroom.proxy.project_policy import split_project_path
|
||||
|
||||
from .provider import OAuth2Error
|
||||
|
||||
log = logging.getLogger("headroom_oauth2")
|
||||
|
||||
# Exact paths the proxy answers itself and that must stay reachable when the
|
||||
# IdP is down (orchestrator probes).
|
||||
LOCAL_ROUTE_PATHS: frozenset[str] = frozenset(
|
||||
{"/health", "/healthz", "/livez", "/readyz", "/favicon.ico", "/subscription-window"}
|
||||
)
|
||||
|
||||
# Prefixes of routes the proxy (or a co-installed extension) serves locally,
|
||||
# never forwarding to a provider. Everything NOT matched here is treated as an
|
||||
# upstream request, including the provider passthrough catch-all, so a new
|
||||
# provider route needs no change here; a new *local* route does.
|
||||
LOCAL_ROUTE_PREFIXES: tuple[str, ...] = (
|
||||
"/stats", # /stats, /stats-history, /stats-lifetime
|
||||
"/metrics",
|
||||
"/quota",
|
||||
"/settings",
|
||||
"/dashboard",
|
||||
"/admin/",
|
||||
"/debug/",
|
||||
"/transformations/",
|
||||
"/v1/compress", # incl. /v1/compress/response
|
||||
"/v1/usage",
|
||||
"/v1/retrieve",
|
||||
"/v1/telemetry",
|
||||
"/v1/toin",
|
||||
"/v1/feedback",
|
||||
"/ext/", # convention for routes registered by other extensions
|
||||
)
|
||||
|
||||
|
||||
def is_local_route(path: str) -> bool:
|
||||
"""True when ``path`` is served by the proxy itself and never goes upstream."""
|
||||
path = split_project_path(path or "/")[1] # drop a /p/<project> base-URL prefix
|
||||
return path in LOCAL_ROUTE_PATHS or path.startswith(LOCAL_ROUTE_PREFIXES)
|
||||
|
||||
|
||||
class OAuth2Middleware:
|
||||
"""ASGI middleware that replaces the request Authorization with a minted bearer."""
|
||||
@@ -27,6 +73,9 @@ class OAuth2Middleware:
|
||||
if scope.get("type") != "http":
|
||||
await self.app(scope, receive, send)
|
||||
return
|
||||
if is_local_route(scope.get("path") or "/"):
|
||||
await self.app(scope, receive, send)
|
||||
return
|
||||
# Hot path: a cached, still-valid token needs no thread hop. Only mint (blocking
|
||||
# urllib) off the event loop when the cache is empty/expired.
|
||||
token = self.provider.cached()
|
||||
|
||||
@@ -145,5 +145,5 @@ class OAuth2ClientCredentials:
|
||||
except (TypeError, ValueError):
|
||||
ttl = 300
|
||||
ttl = max(1, ttl) # 0/negative would cause a stale token or per-request minting
|
||||
log.info("oauth2: minted token (ttl=%ss, scopes=%s)", ttl, self.scopes or "-")
|
||||
log.info("oauth2: minted token (ttl=%ss, scopes=%d)", ttl, len(self.scopes))
|
||||
return token, ttl
|
||||
|
||||
@@ -0,0 +1,276 @@
|
||||
"""The oauth2 layer only acts on upstream requests (VAPT tracker 03-F4).
|
||||
|
||||
The middleware rewrote ``Authorization`` on *every* HTTP request, so local
|
||||
routes such as ``/health`` minted a token and returned 502 whenever the IdP was
|
||||
unreachable. The end-to-end tests also pin that, with extension middleware
|
||||
inside the core's proxy-token gate, a remote client presenting the proxy token
|
||||
as its bearer is authenticated before the bearer is replaced.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
|
||||
import pytest
|
||||
|
||||
from headroom_oauth2 import OAuth2Middleware, install
|
||||
|
||||
try: # only the end-to-end tests need FastAPI; they importorskip the core too
|
||||
from fastapi import Request
|
||||
except ImportError: # pragma: no cover
|
||||
Request = None # type: ignore[assignment,misc]
|
||||
|
||||
NONLOOPBACK = ("203.0.113.5", 44444)
|
||||
|
||||
|
||||
class _Recording:
|
||||
def __init__(self):
|
||||
self.called = False
|
||||
self.scope = None
|
||||
|
||||
async def __call__(self, scope, receive, send):
|
||||
self.called = True
|
||||
self.scope = scope
|
||||
|
||||
|
||||
class _NeverMint:
|
||||
"""Provider double that fails the test if the middleware tries to use it."""
|
||||
|
||||
def cached(self):
|
||||
raise AssertionError("provider consulted for a request that must not be injected")
|
||||
|
||||
def token(self):
|
||||
raise AssertionError("provider minted for a request that must not be injected")
|
||||
|
||||
|
||||
class _Cached:
|
||||
def __init__(self, tok="MINTED"):
|
||||
self.tok = tok
|
||||
self.reads = 0
|
||||
|
||||
def cached(self):
|
||||
self.reads += 1
|
||||
return self.tok
|
||||
|
||||
def token(self):
|
||||
return self.tok
|
||||
|
||||
|
||||
async def _recv():
|
||||
return {"type": "http.request"}
|
||||
|
||||
|
||||
async def _ignore(_msg):
|
||||
pass
|
||||
|
||||
|
||||
def _scope(path="/v1/chat/completions", headers=(), client=NONLOOPBACK):
|
||||
return {
|
||||
"type": "http",
|
||||
"method": "POST",
|
||||
"path": path,
|
||||
"headers": list(headers),
|
||||
"client": client,
|
||||
}
|
||||
|
||||
|
||||
def _run(mw, scope):
|
||||
asyncio.run(mw(scope, _recv, _ignore))
|
||||
|
||||
|
||||
def _auth_of(app) -> bytes | None:
|
||||
return dict(app.scope["headers"]).get(b"authorization")
|
||||
|
||||
|
||||
# --- local routes are never touched -------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"path",
|
||||
[
|
||||
"/health",
|
||||
"/healthz",
|
||||
"/livez",
|
||||
"/readyz",
|
||||
"/stats",
|
||||
"/stats-history",
|
||||
"/metrics",
|
||||
"/quota",
|
||||
"/settings",
|
||||
"/dashboard/",
|
||||
"/admin/upstream",
|
||||
"/debug/tasks",
|
||||
"/v1/compress",
|
||||
"/v1/compress/response",
|
||||
"/v1/usage",
|
||||
"/v1/retrieve/abc",
|
||||
"/v1/telemetry/export",
|
||||
"/v1/toin/stats",
|
||||
"/v1/feedback/tool",
|
||||
"/ext/lossless-guard/record",
|
||||
"/p/myproject/stats", # project-prefixed local route
|
||||
],
|
||||
)
|
||||
def test_local_routes_are_passed_through_without_minting(path):
|
||||
app = _Recording()
|
||||
mw = OAuth2Middleware(app, _NeverMint())
|
||||
_run(mw, _scope(path, headers=[(b"authorization", b"Bearer CLIENT")]))
|
||||
assert app.called
|
||||
assert _auth_of(app) == b"Bearer CLIENT" # untouched
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"path",
|
||||
[
|
||||
"/v1/chat/completions",
|
||||
"/v1/messages",
|
||||
"/v1/responses",
|
||||
"/v1/models",
|
||||
"/p/myproject/v1/messages", # project-prefixed upstream route
|
||||
"/some/vendor/passthrough", # provider catch-all
|
||||
"/",
|
||||
],
|
||||
)
|
||||
def test_upstream_routes_are_injected(path):
|
||||
app = _Recording()
|
||||
mw = OAuth2Middleware(app, _Cached("TOK"))
|
||||
_run(mw, _scope(path, headers=[(b"authorization", b"Bearer CLIENT")]))
|
||||
assert _auth_of(app) == b"Bearer TOK"
|
||||
|
||||
|
||||
def test_install_log_line_does_not_carry_the_token_url_query(monkeypatch, caplog):
|
||||
monkeypatch.setenv("HEADROOM_OAUTH2_TOKEN_URL", "https://idp.example.com/token?client=acme")
|
||||
monkeypatch.setenv("HEADROOM_OAUTH2_CLIENT_ID", "c")
|
||||
monkeypatch.setenv("HEADROOM_OAUTH2_CLIENT_SECRET", "s")
|
||||
monkeypatch.delenv("HEADROOM_OAUTH2_HEADERS", raising=False)
|
||||
caplog.set_level(logging.INFO, logger="headroom_oauth2")
|
||||
|
||||
class App:
|
||||
def add_middleware(self, cls, **kw):
|
||||
pass
|
||||
|
||||
install(App(), type("Cfg", (), {"backend": "litellm-openai"})())
|
||||
[record] = [r for r in caplog.records if "auth installed" in r.getMessage()]
|
||||
# The logged token_url is exactly scheme://host -- no path, no query.
|
||||
assert record.args[0] == "https://idp.example.com"
|
||||
assert "client=acme" not in caplog.text
|
||||
assert "/token" not in caplog.text
|
||||
|
||||
|
||||
# --- end to end through the real proxy app --------------------------------------
|
||||
|
||||
|
||||
def test_remote_client_with_proxy_token_bearer_is_not_refused_when_oauth2_is_enabled(
|
||||
monkeypatch,
|
||||
):
|
||||
"""Through create_app: the 401-on-every-remote-request regression, plus /health with the IdP down."""
|
||||
server = pytest.importorskip("headroom.proxy.server")
|
||||
testclient = pytest.importorskip("fastapi.testclient")
|
||||
from headroom.proxy import extensions
|
||||
|
||||
monkeypatch.setenv("HEADROOM_OAUTH2_TOKEN_URL", "https://idp.invalid/token") # unreachable
|
||||
monkeypatch.setenv("HEADROOM_OAUTH2_CLIENT_ID", "c")
|
||||
monkeypatch.setenv("HEADROOM_OAUTH2_CLIENT_SECRET", "s")
|
||||
monkeypatch.setenv("HEADROOM_OAUTH2_TIMEOUT", "1")
|
||||
monkeypatch.setattr(extensions, "discover", lambda: iter([("oauth2", install)]))
|
||||
|
||||
config = server.ProxyConfig(
|
||||
optimize=False,
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
cost_tracking_enabled=False,
|
||||
proxy_token="s3cr3t",
|
||||
proxy_extensions=["oauth2"],
|
||||
)
|
||||
app = server.create_app(config)
|
||||
with testclient.TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
||||
# Local routes: no mint, so the unreachable IdP is irrelevant.
|
||||
assert c.get("/health").status_code == 200
|
||||
assert c.get("/stats", headers={"Authorization": "Bearer s3cr3t"}).status_code == 200
|
||||
# Still gated: no credential, no access.
|
||||
assert c.get("/stats").status_code == 401
|
||||
# Upstream route with the proxy token: authenticates, then tries to mint -> 502
|
||||
# from oauth2 (IdP unreachable), NOT 401 from the gate.
|
||||
r = c.post(
|
||||
"/v1/chat/completions",
|
||||
json={"model": "x", "messages": []},
|
||||
headers={"Authorization": "Bearer s3cr3t"},
|
||||
)
|
||||
assert r.status_code == 502
|
||||
assert r.json()["error"]["type"] == "upstream_auth_error"
|
||||
|
||||
|
||||
def test_bearer_proxy_token_survives_replacement_when_gate_runs_inside(monkeypatch):
|
||||
"""IdP reachable: the gate authenticates the proxy-token bearer, then it is replaced.
|
||||
|
||||
Registers a probe route under a non-local path so what reaches the handler is
|
||||
exactly what an upstream would have received, without any upstream.
|
||||
"""
|
||||
server = pytest.importorskip("headroom.proxy.server")
|
||||
testclient = pytest.importorskip("fastapi.testclient")
|
||||
import json
|
||||
import threading
|
||||
from http.server import BaseHTTPRequestHandler, HTTPServer
|
||||
|
||||
from headroom.proxy import extensions
|
||||
|
||||
mints = {"n": 0}
|
||||
|
||||
class IdP(BaseHTTPRequestHandler):
|
||||
def do_POST(self):
|
||||
self.rfile.read(int(self.headers.get("content-length", 0) or 0))
|
||||
mints["n"] += 1
|
||||
body = json.dumps({"access_token": "MINTED", "expires_in": 3600}).encode()
|
||||
self.send_response(200)
|
||||
self.send_header("content-type", "application/json")
|
||||
self.send_header("content-length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def log_message(self, *a):
|
||||
pass
|
||||
|
||||
srv = HTTPServer(("127.0.0.1", 0), IdP)
|
||||
threading.Thread(target=srv.serve_forever, daemon=True).start()
|
||||
try:
|
||||
monkeypatch.setenv(
|
||||
"HEADROOM_OAUTH2_TOKEN_URL", f"http://127.0.0.1:{srv.server_address[1]}/token"
|
||||
)
|
||||
monkeypatch.setenv("HEADROOM_OAUTH2_CLIENT_ID", "c")
|
||||
monkeypatch.setenv("HEADROOM_OAUTH2_CLIENT_SECRET", "s")
|
||||
seen = {}
|
||||
|
||||
def spy_install(app, config):
|
||||
install(app, config)
|
||||
|
||||
@app.post("/probe/upstream-like")
|
||||
async def probe(request: Request):
|
||||
seen["authorization"] = request.headers.get("authorization")
|
||||
seen["proxy_header"] = request.headers.get("x-headroom-proxy-token")
|
||||
return {"ok": True}
|
||||
|
||||
monkeypatch.setattr(extensions, "discover", lambda: iter([("oauth2", spy_install)]))
|
||||
config = server.ProxyConfig(
|
||||
optimize=False,
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
cost_tracking_enabled=False,
|
||||
proxy_token="s3cr3t",
|
||||
proxy_extensions=["oauth2"],
|
||||
)
|
||||
app = server.create_app(config)
|
||||
with testclient.TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
||||
r = c.post("/probe/upstream-like", headers={"Authorization": "Bearer s3cr3t"})
|
||||
assert r.status_code == 200, r.text
|
||||
assert seen["authorization"] == "Bearer MINTED" # upstream bearer injected
|
||||
assert seen["proxy_header"] is None # the proxy credential is not carried on
|
||||
assert mints["n"] == 1
|
||||
|
||||
seen.clear()
|
||||
r = c.post("/probe/upstream-like") # no credential at all
|
||||
assert r.status_code == 401
|
||||
assert seen == {} # never reached the handler
|
||||
assert mints["n"] == 1 # and cost no mint
|
||||
finally:
|
||||
srv.shutdown()
|
||||
Reference in New Issue
Block a user