## Description
`plugins/headroom-oauth2/SPEC.md:67` promised that a missing or invalid
config at
startup makes `install()` raise and the proxy **fail closed** ("won't
start
mis-auth'd"), and its `CHANGELOG.md:25` said "Fails closed on
misconfiguration".
`install()` does raise. But the loader that calls it, `install_all()` in
`headroom/proxy/extensions.py`, wrapped every `install(app, config)` in
`except Exception`, logged a warning, and continued. So with `oauth2`
enabled and
a bad config the proxy started **without** `OAuth2Middleware` — and
since that
middleware is what replaces the client's `Authorization` with the minted
bearer,
the proxy relayed the client's own `Authorization` header upstream
unchanged.
The only signals were a log warning and one stderr line.
Skip-and-continue is the right default and stays the default: an
optional plugin
must not brick the proxy, and a licence-gate failure should leave the
feature
absent rather than fatal. It is the wrong rule for an extension that
*is* the
upstream credential. That is a per-extension property, so the extension
declares
it:
```python
def install(app, config): ...
install.fail_closed = True
```
`install_all` re-raises instead of swallowing when the failing `install`
carries
the marker, so `create_app` raises and the process never serves.
Everything
unmarked keeps today's behaviour exactly, including extensions installed
before
the failing one (none after it run). Reading the marker off the loaded
callable
costs the extension one line and core no per-name allowlist; an older
core that
does not know the marker ignores it and behaves as before — the plugin
documents
that.
Closes #4737
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/extensions.py`: new `fail_closed` marker on the
`install`
callable; `install_all` re-raises a marked extension's failure (chained
from
the original exception) instead of logging and skipping. Module
docstring
gains a "Failing closed" section covering when the marker is
appropriate.
- `headroom/proxy/server.py`: the comment at the `install_all` call site
said the
process never fails on an extension install; it now names the exception.
- `plugins/headroom-oauth2/src/headroom_oauth2/__init__.py`: set
`install.fail_closed = True`, with a comment on why skipping would be
worse
than not starting.
- `plugins/headroom-oauth2`: `SPEC.md` failure-mode row now matches
reality (plus
a paragraph on the marker and old cores), `README.md` gains a
**Startup** note,
`CHANGELOG.md` 0.2.0 entry, version bumped in `pyproject.toml` +
`__version__`.
- `docs/content/docs/oauth2-upstream.mdx`: one line under **Notes**.
- `CHANGELOG.md` is **not** touched — release-please owns it.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```
$ pytest tests/test_proxy_extensions.py -q
6 passed
$ PYTHONPATH=.:plugins/headroom-oauth2/src pytest plugins/headroom-oauth2/tests/test_oauth2.py -q
41 passed in 10.81s
$ ruff check headroom/proxy/extensions.py headroom/proxy/server.py \
tests/test_proxy_extensions.py plugins/headroom-oauth2/
All checks passed!
$ ruff format --check <same files>
11 files already formatted
$ mypy headroom # 19 pre-existing errors in unrelated modules (astgrep, mcp_server,
# model_metadata) from this Windows/mypy version; none in extensions.py
```
Both new tests fail without the fix, for the right reason:
```
$ git stash push -- headroom/proxy/extensions.py
$ pytest tests/test_proxy_extensions.py plugins/headroom-oauth2/tests/test_oauth2.py -q
FAILED tests/test_proxy_extensions.py::test_install_all_aborts_startup_when_fail_closed_extension_fails
FAILED plugins/headroom-oauth2/tests/test_oauth2.py::test_install_all_aborts_startup_on_bad_config
2 failed, 45 passed
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.13.11, this PR's branch checked out,
`PYTHONPATH` pointing at the checkout and `plugins/headroom-oauth2/src`
(no
package install needed). No proxy request was sent; the extension's own
`OAuth2Middleware` behaviour is already covered by its existing ASGI
tests.
- Exact command / steps: the reporter's own script shape — real
`install_all`,
real `headroom_oauth2.install`, config from `os.environ`, `discover()`
replaced
by a one-entry list — plus a second script that goes through the real
`create_app(ProxyConfig(proxy_extensions=["oauth2"]))`.
```python
os.environ["HEADROOM_OAUTH2_TOKEN_URL"] = "http://idp.example.com/token"
os.environ["HEADROOM_OAUTH2_CLIENT_ID"] = "c"
os.environ["HEADROOM_OAUTH2_CLIENT_SECRET"] = "s"
ext.discover = lambda: iter([("oauth2", headroom_oauth2.install)])
create_app(ProxyConfig(optimize=False, cache_enabled=False,
rate_limit_enabled=False, cost_tracking_enabled=False,
proxy_extensions=["oauth2"]))
```
- Observed result: `create_app` raises, and with a valid config installs
the
middleware. After the patch:
```
ok misconfigured: create_app raised RuntimeError: proxy extension
'oauth2' failed
to install and is marked fail_closed, so the proxy will no...
ok valid config: app.user_middleware = ['ActivityMiddleware',
'WebSocketAuthMiddleware',
'BaseHTTPMiddleware', 'RequestBodyLimitMiddleware', 'OAuth2Middleware',
'BaseHTTPMiddleware', 'CORSMiddleware',
'WebSocketProjectPrefixMiddleware']
```
With the plugin's own message preserved as `__cause__`:
```
install_all raised: RuntimeError: proxy extension 'oauth2' failed to
install and is
marked fail_closed, so the proxy will not start: headroom-oauth2
misconfigured:
token_url must be https (loopback http allowed for tests; set
allow_insecure=True /
HEADROOM_OAUTH2_ALLOW_INSECURE=1 to override)
__cause__: RuntimeError headroom-oauth2 misconfigured: token_url must be
https (...)
proxy would NOT start (create_app raises before uvicorn serves)
```
Before the patch (same script, `extensions.py` + plugin stashed):
```
[headroom] proxy extensions SKIPPED: oauth2 (install failed - running
without them; see logs)
FAIL: create_app returned; proxy would start without the middleware
```
which is exactly the issue's reported behaviour.
- Not tested: a real `headroom proxy` process start through uvicorn with
a
live IdP, and the request path at the wire level (the client's
`Authorization`
reaching an upstream). A multi-worker start (`--workers N`) was not
exercised
either; `create_app_from_env` goes through the same `create_app`, so
each
worker that cannot install the extension should exit.
`tests/test_proxy_extension_ordering.py` and two
`test_scope_and_proxy_token.py` cases fail on this machine with or
without the
patch (they need the compiled Rust core, which is not built here) —
verified
pre-existing by stashing the patch.
## Runtime Rollout Safety
- Rollout-managed feature(s): none. No flag, no staged rollout.
- Minimum rollout channel: any; the marker is inert unless an extension
sets it.
- Stable/default behavior changed: **no** for core — an unmarked
extension that
raises is still skipped with the same warning and stderr line. For
`headroom-oauth2` 0.2.0, yes and deliberately: a bad config now stops
startup.
- Kill switch / disable path: drop `--proxy-extension oauth2` /
`HEADROOM_PROXY_EXTENSIONS`, or fix the config. Both need no code
change.
- Unsafe override required: no. `HEADROOM_OAUTH2_ALLOW_INSECURE=1` still
lets an
operator opt into a non-loopback `http` token URL explicitly.
- Qualification impact: none beyond `headroom-oauth2` 0.2.0's own notes.
- Rollback path: revert the commit, or pin `headroom-oauth2<0.2.0` (its
marker is
ignored by any core, so downgrading the plugin alone restores old
behaviour).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
Two decisions a reviewer may want to weigh in on:
1. **Marker vs. core allowlist.** Reading `fail_closed` off the entry
point means
core guarantees nothing about *which* extensions are security-critical,
and a
malicious/broken plugin can also opt into aborting startup. The
alternative was
a hard-coded list of vetted names in core, which makes every new
security-critical extension a core change. Happy to switch if
maintainers
prefer that trade.
2. **Not covered here:** a `fail_closed` extension that is *not
discovered at
all* (typo in the name, package not installed) is still only a warning,
and
that has the same fail-open effect. Closing it needs a way to know the
intended
marker for a name that cannot be loaded — the allowlist in (1), or an
env
opt-in — so I left it out of scope and mention it here rather than
inventing a
mechanism.
Co-authored-by: JD Davis <mxjerrett@gmail.com>
7.0 KiB
Spec — headroom-oauth2 (generic OAuth2 client-credentials upstream auth)
Summary
A vendor-neutral proxy extension (registers on Headroom's headroom.proxy_extension seam) that
mints an OAuth2 client-credentials (RFC 6749 §4.4) bearer from a configured token endpoint and
injects it as the upstream Authorization on every proxied request. Lets Headroom front any
gateway that requires a minted machine token (not a static API key) — with zero core changes
and no vendor specifics (the gateway is entirely config/env).
It complements #510 (env-var auth), which assumes a long-lived static key; this covers the
"mint-then-refresh a short-lived token" case.
API surface (config / CLI / env)
Opt-in only, via Headroom's existing flags — no new CLI flags:
headroom proxy --backend litellm-openai --proxy-extension oauth2
# or HEADROOM_PROXY_EXTENSIONS=oauth2
All configuration is env (12-factor; nothing baked in):
| Env var | Required | Meaning |
|---|---|---|
HEADROOM_OAUTH2_TOKEN_URL |
yes (else no-op) | OAuth2 token endpoint; must be https (loopback http allowed for tests) |
HEADROOM_OAUTH2_CLIENT_ID / _CLIENT_SECRET |
yes | client credentials |
HEADROOM_OAUTH2_SCOPES |
no | space/comma-separated scopes |
HEADROOM_OAUTH2_AUDIENCE |
no | audience form param |
HEADROOM_OAUTH2_RESOURCE |
no | RFC 8707 target resource form param |
HEADROOM_OAUTH2_GRANT_TYPE |
no | default client_credentials |
HEADROOM_OAUTH2_AUTH_STYLE |
no | post (form creds) or basic (HTTP Basic) |
HEADROOM_OAUTH2_HEADERS |
no | static upstream headers, K=V,K2=V2 (control chars rejected) |
HEADROOM_OAUTH2_TIMEOUT / _SKEW |
no | token request timeout / pre-expiry refresh skew (s) |
HEADROOM_OAUTH2_ALLOW_INSECURE |
no | 1 to allow a non-loopback http token_url (discouraged) |
Public Python API: OAuth2ClientCredentials, OAuth2Middleware, OAuth2Error, install,
provider_from_env, parse_headers.
Changes to existing behavior / defaults / compatibility
- None unless explicitly enabled. The entry point is dormant until
--proxy-extension oauth2is passed, and even then a no-op unlessHEADROOM_OAUTH2_TOKEN_URLis set. - When active, it overwrites the request
Authorizationheader with the minted bearer before the backend runs. The client's ownAuthorization/x-api-keyis intentionally replaced (the proxy authenticates to the gateway on the client's behalf). Compatibility note: because the request then carries a bearer, Headroom classifies it as OAuth-mode auth — same as supplying a bearer yourself; no new classification path. - No change to defaults, the request/response body, model routing, or compression.
User stories (Given / When / Then)
- Golden path — Given a proxy started with
--proxy-extension oauth2and validTOKEN_URL/CLIENT_ID/CLIENT_SECRET, When a client sends/v1/messages, Then the extension mints (or reuses a cached) bearer and the upstream receivesAuthorization: Bearer <minted>plus any static headers; the client never sees the secret. - Edge: token endpoint down — Given an unreachable/erroring
TOKEN_URL, When a request arrives, Then the proxy returns502 upstream_auth_error(no upstream call, no secret/body leak) and stays up; the next request retries. - Edge: wrong backend — Given
--backend bedrock(env-auth), When the extension installs, Then it logs a loud warning that the injected bearer will have no effect and to use an OpenAI-compatible/passthrough backend.
Failure modes & recovery
| Failure | Behavior |
|---|---|
| Missing/invalid config at startup | install() raises and is marked fail_closed -> proxy refuses to start (won't start mis-auth'd) |
Token endpoint unreachable / non-2xx / non-JSON / no access_token |
OAuth2Error -> 502, per-request, proxy stays up, retried next request |
expires_in = 0/negative/absent |
clamped to a positive TTL (never stale, never per-request mint) |
| Concurrent first requests | single-flight lock -> exactly one mint per refresh |
Malformed HEADROOM_OAUTH2_HEADERS (CR/LF) |
offending pair dropped with a warning (no header injection) |
Core's default is to log a failed install() and keep serving without that
extension — right for an optional feature, wrong here: skipping this middleware
leaves the proxy relaying the client's own Authorization upstream. The
extension therefore sets install.fail_closed = True, which makes install_all
re-raise so startup aborts. That needs a headroom-ai whose
headroom/proxy/extensions.py honours the marker; on an older core the flag is
ignored and the proxy starts without the middleware, as it did before #4737.
Resilience (Docker / native / wrappers / providers / multi-process)
- Native & Docker: identical; pure env-driven, std-lib only. Token minted via
urllibagainst the system cert store, so a corporate-injected CA is trusted with no bundled roots (works in SSL-inspection networks). - Wrappers (
headroom wrap, agent hooks): the extension lives at the proxy layer, so anything routed through the proxy inherits it transparently. - Providers: effective for OpenAI-compatible / passthrough litellm backends (those that forward
the request bearer upstream).
bedrock/vertex/sagemakerauthenticate from env and ignore the bearer -> the extension warns and is a no-op there. - Multi-process (multiple workers): the token cache is per-process; each worker mints/refreshes independently. Acceptable for client-credentials (idempotent, low rate); no shared state, no cross-process lock needed. Documented so operators can size token-endpoint rate limits.
Security & privacy
- Secrets are env-only; never logged and never returned to the client.
- The IdP error body is drained, not surfaced (may echo sensitive context).
token_urlis https-enforced (loopback exception for tests; explicit opt-out env).- The minted bearer is sent only to the configured upstream; the client's inbound credential is replaced, not forwarded onward.
Observability / logging / telemetry
INFOon install (token_url + auth style, no secrets) and on each mint (ttl, scopes).WARNINGon mint failure, env-auth-backend no-op, and dropped malformed static headers.- No metrics/telemetry emitted; piggybacks on Headroom's existing request logging. (A future counter for mint/refresh/failure could be added if maintainers want it.)
Rollback / migration
- No migration — additive and opt-in; existing deployments are unaffected.
- Instant rollback: drop
--proxy-extension oauth2(or unsetHEADROOM_PROXY_EXTENSIONS), or uninstall the package. No state to clean up, no config format changes.
Dependencies
- Runtime: standard library only (no new core dependency).
litellmis touched only ifHEADROOM_OAUTH2_HEADERSis set, and it is already a Headroom backend dependency — declared here as the optional[litellm]extra, not a hard requirement.