fix(oauth2): mint and inject only on requests that go upstream (#3849)

## Why

`OAuth2Middleware` rewrote `Authorization` on every HTTP request
(findings 03-F4 and 03-F5). Routes the proxy answers itself (`/health`,
`/stats*`, `/metrics`, `/v1/compress*`, extension `/ext/*` routes and
the rest of the management surface) never go upstream. Yet each one
minted a token when the cache was cold, and an unreachable IdP turned
health probes and management calls into 502. Separately, the startup log
printed the full token URL, which can carry a query string or embedded
credentials, and the mint log printed the scope list.

## What changes

- The bearer is minted and injected only for requests that go upstream.
`LOCAL_ROUTE_PATHS` and `LOCAL_ROUTE_PREFIXES` list what the proxy
serves itself; a `/p/<project>/` prefix is stripped first with the
core's `split_project_path`. Everything else, including the provider
passthrough catch-all, is treated as upstream, so a new provider route
needs no change here. Only a new local route does.
- Startup log: `scheme://host` of the token URL instead of the full URL.
Mint log: the scope count instead of the list.
- Version 0.1.1 in both `pyproject.toml` and `__version__`. Plugin
`CHANGELOG.md` and README updated.

Inbound authentication is not this plugin's job. From headroom-ai 0.40,
extension middleware runs inside the `HEADROOM_PROXY_TOKEN` gate
(#3847), so an unauthenticated request never reaches it, and a client
may authenticate with `Authorization: Bearer <proxy token>` before this
plugin replaces that header with the upstream bearer.

## Behaviour change / upgrade notes

- Requires headroom-ai >= 0.40. The README says so.
- Requests to local routes no longer carry a minted bearer. Nothing used
it, but a log line or metric keyed on `Authorization` for those paths
will change.

## Tests

`plugins/headroom-oauth2/tests/test_scope_and_proxy_token.py`:
- Every local route, including a project-prefixed one, passes through
with `Authorization` untouched and no mint.
- Upstream routes, including a project-prefixed route, the passthrough
catch-all and `/`, get the minted bearer.
- The install log line carries only `scheme://host`.
- End to end through `create_app` from a non-loopback client with an
unreachable IdP: `/health` is 200; `/stats` with the proxy-token bearer
is 200; `/stats` with no credential is 401; an upstream route with the
proxy-token bearer gets 502 `upstream_auth_error` from oauth2, not 401
from the gate.
- End to end with a local IdP: the proxy-token bearer passes the gate
and is replaced by the minted bearer, `x-headroom-proxy-token` is not
added, and one mint happens. A request with no credential is refused
with 401 and costs no mint.

This plugin has no CI job yet (only `plugins/headroom-agent-hooks` is
path-filtered in), so these tests were run locally.

## Merge order

Part of a set. Suggested order: #3852 → #3891 → #3849 → #3860 → #3862 →
#3863. This PR is independent of the others; the plugin only needs a
core >= 0.40.

## Follow-ups

- Add a path-filtered CI job for `plugins/headroom-oauth2`.

Revised 2026-10-01 after self-review: removed the plugin's own
proxy-token handling (now done by the core gate) and the
`HEADROOM_OAUTH2_LOCAL_PATHS` setting.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Tejas Chopra
2026-10-02 11:02:12 -05:00
committed by GitHub
co-authored by Claude Fable 5.1
parent 0147cf0220
commit f977d525e4
7 changed files with 370 additions and 4 deletions
+11
View File
@@ -1,5 +1,16 @@
# Changelog
## 0.1.1
- Inject the upstream bearer only on requests that go upstream. Local routes
(`/health`, `/stats*`, `/metrics`, `/v1/compress*`, `/ext/*`, …; see README)
are passed through untouched, so an unreachable IdP no longer 502s health
probes and management calls no longer mint tokens. `/p/<project>/` prefixes
are stripped first.
- Startup log carries `scheme://host` of the token URL and the mint log a scope
count, instead of the full URL and scope list.
- Requires `headroom-ai` ≥ 0.40 (extension middleware inside the proxy-token gate).
## 0.1.0
Initial release — generic OAuth2 client-credentials upstream-auth extension for the Headroom proxy.
+16
View File
@@ -41,3 +41,19 @@ ignore it, so this extension is a no-op there (it logs a warning at startup).
**Transport:** `token_url` must be `https` (loopback `http` is allowed for tests; set
`HEADROOM_OAUTH2_ALLOW_INSECURE=1` to override). Tokens are minted with the standard library
(`urllib`, system cert store), so a corporate-injected CA is trusted without bundling roots.
## What gets the upstream bearer
The minted token is injected only on requests that **go upstream**. Routes the
proxy answers itself — `/health`, `/livez`, `/readyz`, `/stats*`, `/metrics`,
`/quota`, `/settings*`, `/dashboard*`, `/admin/*`, `/debug/*`, `/v1/compress*`,
`/v1/usage`, `/v1/retrieve*`, `/v1/telemetry*`, `/v1/toin*`, `/v1/feedback*`,
and any extension route under `/ext/` — are passed through untouched, so an
unreachable IdP never turns a health probe into a 502 and management calls never
cost a token mint. A `/p/<project>/` base-URL prefix is stripped before the
check.
Requires `headroom-ai` ≥ 0.40, where extension middleware runs inside the
proxy's `HEADROOM_PROXY_TOKEN` gate: clients may then authenticate with
`Authorization: Bearer <proxy token>` and the gate checks it before this
extension replaces it with the upstream bearer.
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "headroom-oauth2"
version = "0.1.0"
version = "0.1.1"
description = "Generic OAuth2 client-credentials upstream-auth extension for the Headroom proxy"
readme = "README.md"
requires-python = ">=3.10"
@@ -8,13 +8,14 @@ from __future__ import annotations
import logging
import os
import urllib.parse
from typing import Any
from .middleware import OAuth2Middleware
from .provider import OAuth2ClientCredentials, OAuth2Error
__all__ = ["OAuth2ClientCredentials", "OAuth2Error", "OAuth2Middleware", "install", "parse_headers"]
__version__ = "0.1.0"
__version__ = "0.1.1"
log = logging.getLogger("headroom_oauth2")
@@ -44,6 +45,19 @@ def parse_headers(s: str | None) -> dict[str, str]:
return out
def _redact_url(url: str) -> str:
"""``scheme://host[:port]`` only -- a tenant token URL can carry a query string or
embedded credentials, neither of which belongs in a startup log line."""
try:
parts = urllib.parse.urlsplit(url)
host = parts.hostname or "?"
if parts.port:
host = f"{host}:{parts.port}"
return f"{parts.scheme}://{host}"
except ValueError:
return "<unparseable>"
def _int(env, key):
raw = env.get(key)
if raw is None or not str(raw).strip():
@@ -137,6 +151,6 @@ def install(app: Any, config: Any) -> None:
app.add_middleware(OAuth2Middleware, provider=provider)
log.info(
"headroom-oauth2: client-credentials auth installed (token_url=%s, style=%s)",
provider.token_url,
_redact_url(provider.token_url),
provider.auth_style,
)
@@ -3,6 +3,16 @@
Headroom's litellm backend forwards the request's `Authorization` bearer to the
upstream as the API key, so setting it here makes the minted token reach the
backend with no core changes.
Requests the proxy answers itself are left untouched. Health probes, ``/stats``,
``/metrics``, the compress/retrieve/telemetry routes and any extension's own
``/ext/...`` routes never go upstream, so there is nothing to inject and no
reason to mint. Before this rule ``/health`` returned 502 whenever the IdP was
unreachable, and every management request cost a mint when the cache was cold.
Inbound authentication is not this layer's job: on headroom-ai >= 0.40 extension
middleware runs inside the ``HEADROOM_PROXY_TOKEN`` gate, so an unauthenticated
request never reaches it.
"""
from __future__ import annotations
@@ -11,10 +21,46 @@ import asyncio
import json
import logging
from headroom.proxy.project_policy import split_project_path
from .provider import OAuth2Error
log = logging.getLogger("headroom_oauth2")
# Exact paths the proxy answers itself and that must stay reachable when the
# IdP is down (orchestrator probes).
LOCAL_ROUTE_PATHS: frozenset[str] = frozenset(
{"/health", "/healthz", "/livez", "/readyz", "/favicon.ico", "/subscription-window"}
)
# Prefixes of routes the proxy (or a co-installed extension) serves locally,
# never forwarding to a provider. Everything NOT matched here is treated as an
# upstream request, including the provider passthrough catch-all, so a new
# provider route needs no change here; a new *local* route does.
LOCAL_ROUTE_PREFIXES: tuple[str, ...] = (
"/stats", # /stats, /stats-history, /stats-lifetime
"/metrics",
"/quota",
"/settings",
"/dashboard",
"/admin/",
"/debug/",
"/transformations/",
"/v1/compress", # incl. /v1/compress/response
"/v1/usage",
"/v1/retrieve",
"/v1/telemetry",
"/v1/toin",
"/v1/feedback",
"/ext/", # convention for routes registered by other extensions
)
def is_local_route(path: str) -> bool:
"""True when ``path`` is served by the proxy itself and never goes upstream."""
path = split_project_path(path or "/")[1] # drop a /p/<project> base-URL prefix
return path in LOCAL_ROUTE_PATHS or path.startswith(LOCAL_ROUTE_PREFIXES)
class OAuth2Middleware:
"""ASGI middleware that replaces the request Authorization with a minted bearer."""
@@ -27,6 +73,9 @@ class OAuth2Middleware:
if scope.get("type") != "http":
await self.app(scope, receive, send)
return
if is_local_route(scope.get("path") or "/"):
await self.app(scope, receive, send)
return
# Hot path: a cached, still-valid token needs no thread hop. Only mint (blocking
# urllib) off the event loop when the cache is empty/expired.
token = self.provider.cached()
@@ -145,5 +145,5 @@ class OAuth2ClientCredentials:
except (TypeError, ValueError):
ttl = 300
ttl = max(1, ttl) # 0/negative would cause a stale token or per-request minting
log.info("oauth2: minted token (ttl=%ss, scopes=%s)", ttl, self.scopes or "-")
log.info("oauth2: minted token (ttl=%ss, scopes=%d)", ttl, len(self.scopes))
return token, ttl
@@ -0,0 +1,276 @@
"""The oauth2 layer only acts on upstream requests (VAPT tracker 03-F4).
The middleware rewrote ``Authorization`` on *every* HTTP request, so local
routes such as ``/health`` minted a token and returned 502 whenever the IdP was
unreachable. The end-to-end tests also pin that, with extension middleware
inside the core's proxy-token gate, a remote client presenting the proxy token
as its bearer is authenticated before the bearer is replaced.
"""
from __future__ import annotations
import asyncio
import logging
import pytest
from headroom_oauth2 import OAuth2Middleware, install
try: # only the end-to-end tests need FastAPI; they importorskip the core too
from fastapi import Request
except ImportError: # pragma: no cover
Request = None # type: ignore[assignment,misc]
NONLOOPBACK = ("203.0.113.5", 44444)
class _Recording:
def __init__(self):
self.called = False
self.scope = None
async def __call__(self, scope, receive, send):
self.called = True
self.scope = scope
class _NeverMint:
"""Provider double that fails the test if the middleware tries to use it."""
def cached(self):
raise AssertionError("provider consulted for a request that must not be injected")
def token(self):
raise AssertionError("provider minted for a request that must not be injected")
class _Cached:
def __init__(self, tok="MINTED"):
self.tok = tok
self.reads = 0
def cached(self):
self.reads += 1
return self.tok
def token(self):
return self.tok
async def _recv():
return {"type": "http.request"}
async def _ignore(_msg):
pass
def _scope(path="/v1/chat/completions", headers=(), client=NONLOOPBACK):
return {
"type": "http",
"method": "POST",
"path": path,
"headers": list(headers),
"client": client,
}
def _run(mw, scope):
asyncio.run(mw(scope, _recv, _ignore))
def _auth_of(app) -> bytes | None:
return dict(app.scope["headers"]).get(b"authorization")
# --- local routes are never touched -------------------------------------------
@pytest.mark.parametrize(
"path",
[
"/health",
"/healthz",
"/livez",
"/readyz",
"/stats",
"/stats-history",
"/metrics",
"/quota",
"/settings",
"/dashboard/",
"/admin/upstream",
"/debug/tasks",
"/v1/compress",
"/v1/compress/response",
"/v1/usage",
"/v1/retrieve/abc",
"/v1/telemetry/export",
"/v1/toin/stats",
"/v1/feedback/tool",
"/ext/lossless-guard/record",
"/p/myproject/stats", # project-prefixed local route
],
)
def test_local_routes_are_passed_through_without_minting(path):
app = _Recording()
mw = OAuth2Middleware(app, _NeverMint())
_run(mw, _scope(path, headers=[(b"authorization", b"Bearer CLIENT")]))
assert app.called
assert _auth_of(app) == b"Bearer CLIENT" # untouched
@pytest.mark.parametrize(
"path",
[
"/v1/chat/completions",
"/v1/messages",
"/v1/responses",
"/v1/models",
"/p/myproject/v1/messages", # project-prefixed upstream route
"/some/vendor/passthrough", # provider catch-all
"/",
],
)
def test_upstream_routes_are_injected(path):
app = _Recording()
mw = OAuth2Middleware(app, _Cached("TOK"))
_run(mw, _scope(path, headers=[(b"authorization", b"Bearer CLIENT")]))
assert _auth_of(app) == b"Bearer TOK"
def test_install_log_line_does_not_carry_the_token_url_query(monkeypatch, caplog):
monkeypatch.setenv("HEADROOM_OAUTH2_TOKEN_URL", "https://idp.example.com/token?client=acme")
monkeypatch.setenv("HEADROOM_OAUTH2_CLIENT_ID", "c")
monkeypatch.setenv("HEADROOM_OAUTH2_CLIENT_SECRET", "s")
monkeypatch.delenv("HEADROOM_OAUTH2_HEADERS", raising=False)
caplog.set_level(logging.INFO, logger="headroom_oauth2")
class App:
def add_middleware(self, cls, **kw):
pass
install(App(), type("Cfg", (), {"backend": "litellm-openai"})())
[record] = [r for r in caplog.records if "auth installed" in r.getMessage()]
# The logged token_url is exactly scheme://host -- no path, no query.
assert record.args[0] == "https://idp.example.com"
assert "client=acme" not in caplog.text
assert "/token" not in caplog.text
# --- end to end through the real proxy app --------------------------------------
def test_remote_client_with_proxy_token_bearer_is_not_refused_when_oauth2_is_enabled(
monkeypatch,
):
"""Through create_app: the 401-on-every-remote-request regression, plus /health with the IdP down."""
server = pytest.importorskip("headroom.proxy.server")
testclient = pytest.importorskip("fastapi.testclient")
from headroom.proxy import extensions
monkeypatch.setenv("HEADROOM_OAUTH2_TOKEN_URL", "https://idp.invalid/token") # unreachable
monkeypatch.setenv("HEADROOM_OAUTH2_CLIENT_ID", "c")
monkeypatch.setenv("HEADROOM_OAUTH2_CLIENT_SECRET", "s")
monkeypatch.setenv("HEADROOM_OAUTH2_TIMEOUT", "1")
monkeypatch.setattr(extensions, "discover", lambda: iter([("oauth2", install)]))
config = server.ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
proxy_token="s3cr3t",
proxy_extensions=["oauth2"],
)
app = server.create_app(config)
with testclient.TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
# Local routes: no mint, so the unreachable IdP is irrelevant.
assert c.get("/health").status_code == 200
assert c.get("/stats", headers={"Authorization": "Bearer s3cr3t"}).status_code == 200
# Still gated: no credential, no access.
assert c.get("/stats").status_code == 401
# Upstream route with the proxy token: authenticates, then tries to mint -> 502
# from oauth2 (IdP unreachable), NOT 401 from the gate.
r = c.post(
"/v1/chat/completions",
json={"model": "x", "messages": []},
headers={"Authorization": "Bearer s3cr3t"},
)
assert r.status_code == 502
assert r.json()["error"]["type"] == "upstream_auth_error"
def test_bearer_proxy_token_survives_replacement_when_gate_runs_inside(monkeypatch):
"""IdP reachable: the gate authenticates the proxy-token bearer, then it is replaced.
Registers a probe route under a non-local path so what reaches the handler is
exactly what an upstream would have received, without any upstream.
"""
server = pytest.importorskip("headroom.proxy.server")
testclient = pytest.importorskip("fastapi.testclient")
import json
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer
from headroom.proxy import extensions
mints = {"n": 0}
class IdP(BaseHTTPRequestHandler):
def do_POST(self):
self.rfile.read(int(self.headers.get("content-length", 0) or 0))
mints["n"] += 1
body = json.dumps({"access_token": "MINTED", "expires_in": 3600}).encode()
self.send_response(200)
self.send_header("content-type", "application/json")
self.send_header("content-length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *a):
pass
srv = HTTPServer(("127.0.0.1", 0), IdP)
threading.Thread(target=srv.serve_forever, daemon=True).start()
try:
monkeypatch.setenv(
"HEADROOM_OAUTH2_TOKEN_URL", f"http://127.0.0.1:{srv.server_address[1]}/token"
)
monkeypatch.setenv("HEADROOM_OAUTH2_CLIENT_ID", "c")
monkeypatch.setenv("HEADROOM_OAUTH2_CLIENT_SECRET", "s")
seen = {}
def spy_install(app, config):
install(app, config)
@app.post("/probe/upstream-like")
async def probe(request: Request):
seen["authorization"] = request.headers.get("authorization")
seen["proxy_header"] = request.headers.get("x-headroom-proxy-token")
return {"ok": True}
monkeypatch.setattr(extensions, "discover", lambda: iter([("oauth2", spy_install)]))
config = server.ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
proxy_token="s3cr3t",
proxy_extensions=["oauth2"],
)
app = server.create_app(config)
with testclient.TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
r = c.post("/probe/upstream-like", headers={"Authorization": "Bearer s3cr3t"})
assert r.status_code == 200, r.text
assert seen["authorization"] == "Bearer MINTED" # upstream bearer injected
assert seen["proxy_header"] is None # the proxy credential is not carried on
assert mints["n"] == 1
seen.clear()
r = c.post("/probe/upstream-like") # no credential at all
assert r.status_code == 401
assert seen == {} # never reached the handler
assert mints["n"] == 1 # and cost no mint
finally:
srv.shutdown()