mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description
`plugins/headroom-oauth2/SPEC.md:67` promised that a missing or invalid
config at
startup makes `install()` raise and the proxy **fail closed** ("won't
start
mis-auth'd"), and its `CHANGELOG.md:25` said "Fails closed on
misconfiguration".
`install()` does raise. But the loader that calls it, `install_all()` in
`headroom/proxy/extensions.py`, wrapped every `install(app, config)` in
`except Exception`, logged a warning, and continued. So with `oauth2`
enabled and
a bad config the proxy started **without** `OAuth2Middleware` — and
since that
middleware is what replaces the client's `Authorization` with the minted
bearer,
the proxy relayed the client's own `Authorization` header upstream
unchanged.
The only signals were a log warning and one stderr line.
Skip-and-continue is the right default and stays the default: an
optional plugin
must not brick the proxy, and a licence-gate failure should leave the
feature
absent rather than fatal. It is the wrong rule for an extension that
*is* the
upstream credential. That is a per-extension property, so the extension
declares
it:
```python
def install(app, config): ...
install.fail_closed = True
```
`install_all` re-raises instead of swallowing when the failing `install`
carries
the marker, so `create_app` raises and the process never serves.
Everything
unmarked keeps today's behaviour exactly, including extensions installed
before
the failing one (none after it run). Reading the marker off the loaded
callable
costs the extension one line and core no per-name allowlist; an older
core that
does not know the marker ignores it and behaves as before — the plugin
documents
that.
Closes #4737
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/extensions.py`: new `fail_closed` marker on the
`install`
callable; `install_all` re-raises a marked extension's failure (chained
from
the original exception) instead of logging and skipping. Module
docstring
gains a "Failing closed" section covering when the marker is
appropriate.
- `headroom/proxy/server.py`: the comment at the `install_all` call site
said the
process never fails on an extension install; it now names the exception.
- `plugins/headroom-oauth2/src/headroom_oauth2/__init__.py`: set
`install.fail_closed = True`, with a comment on why skipping would be
worse
than not starting.
- `plugins/headroom-oauth2`: `SPEC.md` failure-mode row now matches
reality (plus
a paragraph on the marker and old cores), `README.md` gains a
**Startup** note,
`CHANGELOG.md` 0.2.0 entry, version bumped in `pyproject.toml` +
`__version__`.
- `docs/content/docs/oauth2-upstream.mdx`: one line under **Notes**.
- `CHANGELOG.md` is **not** touched — release-please owns it.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```
$ pytest tests/test_proxy_extensions.py -q
6 passed
$ PYTHONPATH=.:plugins/headroom-oauth2/src pytest plugins/headroom-oauth2/tests/test_oauth2.py -q
41 passed in 10.81s
$ ruff check headroom/proxy/extensions.py headroom/proxy/server.py \
tests/test_proxy_extensions.py plugins/headroom-oauth2/
All checks passed!
$ ruff format --check <same files>
11 files already formatted
$ mypy headroom # 19 pre-existing errors in unrelated modules (astgrep, mcp_server,
# model_metadata) from this Windows/mypy version; none in extensions.py
```
Both new tests fail without the fix, for the right reason:
```
$ git stash push -- headroom/proxy/extensions.py
$ pytest tests/test_proxy_extensions.py plugins/headroom-oauth2/tests/test_oauth2.py -q
FAILED tests/test_proxy_extensions.py::test_install_all_aborts_startup_when_fail_closed_extension_fails
FAILED plugins/headroom-oauth2/tests/test_oauth2.py::test_install_all_aborts_startup_on_bad_config
2 failed, 45 passed
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.13.11, this PR's branch checked out,
`PYTHONPATH` pointing at the checkout and `plugins/headroom-oauth2/src`
(no
package install needed). No proxy request was sent; the extension's own
`OAuth2Middleware` behaviour is already covered by its existing ASGI
tests.
- Exact command / steps: the reporter's own script shape — real
`install_all`,
real `headroom_oauth2.install`, config from `os.environ`, `discover()`
replaced
by a one-entry list — plus a second script that goes through the real
`create_app(ProxyConfig(proxy_extensions=["oauth2"]))`.
```python
os.environ["HEADROOM_OAUTH2_TOKEN_URL"] = "http://idp.example.com/token"
os.environ["HEADROOM_OAUTH2_CLIENT_ID"] = "c"
os.environ["HEADROOM_OAUTH2_CLIENT_SECRET"] = "s"
ext.discover = lambda: iter([("oauth2", headroom_oauth2.install)])
create_app(ProxyConfig(optimize=False, cache_enabled=False,
rate_limit_enabled=False, cost_tracking_enabled=False,
proxy_extensions=["oauth2"]))
```
- Observed result: `create_app` raises, and with a valid config installs
the
middleware. After the patch:
```
ok misconfigured: create_app raised RuntimeError: proxy extension
'oauth2' failed
to install and is marked fail_closed, so the proxy will no...
ok valid config: app.user_middleware = ['ActivityMiddleware',
'WebSocketAuthMiddleware',
'BaseHTTPMiddleware', 'RequestBodyLimitMiddleware', 'OAuth2Middleware',
'BaseHTTPMiddleware', 'CORSMiddleware',
'WebSocketProjectPrefixMiddleware']
```
With the plugin's own message preserved as `__cause__`:
```
install_all raised: RuntimeError: proxy extension 'oauth2' failed to
install and is
marked fail_closed, so the proxy will not start: headroom-oauth2
misconfigured:
token_url must be https (loopback http allowed for tests; set
allow_insecure=True /
HEADROOM_OAUTH2_ALLOW_INSECURE=1 to override)
__cause__: RuntimeError headroom-oauth2 misconfigured: token_url must be
https (...)
proxy would NOT start (create_app raises before uvicorn serves)
```
Before the patch (same script, `extensions.py` + plugin stashed):
```
[headroom] proxy extensions SKIPPED: oauth2 (install failed - running
without them; see logs)
FAIL: create_app returned; proxy would start without the middleware
```
which is exactly the issue's reported behaviour.
- Not tested: a real `headroom proxy` process start through uvicorn with
a
live IdP, and the request path at the wire level (the client's
`Authorization`
reaching an upstream). A multi-worker start (`--workers N`) was not
exercised
either; `create_app_from_env` goes through the same `create_app`, so
each
worker that cannot install the extension should exit.
`tests/test_proxy_extension_ordering.py` and two
`test_scope_and_proxy_token.py` cases fail on this machine with or
without the
patch (they need the compiled Rust core, which is not built here) —
verified
pre-existing by stashing the patch.
## Runtime Rollout Safety
- Rollout-managed feature(s): none. No flag, no staged rollout.
- Minimum rollout channel: any; the marker is inert unless an extension
sets it.
- Stable/default behavior changed: **no** for core — an unmarked
extension that
raises is still skipped with the same warning and stderr line. For
`headroom-oauth2` 0.2.0, yes and deliberately: a bad config now stops
startup.
- Kill switch / disable path: drop `--proxy-extension oauth2` /
`HEADROOM_PROXY_EXTENSIONS`, or fix the config. Both need no code
change.
- Unsafe override required: no. `HEADROOM_OAUTH2_ALLOW_INSECURE=1` still
lets an
operator opt into a non-loopback `http` token URL explicitly.
- Qualification impact: none beyond `headroom-oauth2` 0.2.0's own notes.
- Rollback path: revert the commit, or pin `headroom-oauth2<0.2.0` (its
marker is
ignored by any core, so downgrading the plugin alone restores old
behaviour).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
Two decisions a reviewer may want to weigh in on:
1. **Marker vs. core allowlist.** Reading `fail_closed` off the entry
point means
core guarantees nothing about *which* extensions are security-critical,
and a
malicious/broken plugin can also opt into aborting startup. The
alternative was
a hard-coded list of vetted names in core, which makes every new
security-critical extension a core change. Happy to switch if
maintainers
prefer that trade.
2. **Not covered here:** a `fail_closed` extension that is *not
discovered at
all* (typo in the name, package not installed) is still only a warning,
and
that has the same fail-open effect. Closing it needs a way to know the
intended
marker for a name that cannot be loaded — the allowlist in (1), or an
env
opt-in — so I left it out of scope and mention it here rather than
inventing a
mechanism.
Co-authored-by: JD Davis <mxjerrett@gmail.com>
137 lines
4.5 KiB
Python
137 lines
4.5 KiB
Python
from __future__ import annotations
|
|
|
|
import logging
|
|
from typing import Any
|
|
|
|
import pytest
|
|
|
|
from headroom.proxy import extensions
|
|
|
|
|
|
def test_install_all_skips_failed_extension_and_continues(
|
|
caplog,
|
|
capsys,
|
|
monkeypatch,
|
|
) -> None:
|
|
calls: list[str] = []
|
|
|
|
def good(app: Any, config: Any) -> None:
|
|
calls.append("good")
|
|
|
|
def bad(app: Any, config: Any) -> None:
|
|
calls.append("bad")
|
|
raise RuntimeError("missing optional dependency")
|
|
|
|
monkeypatch.setattr(
|
|
extensions,
|
|
"discover",
|
|
lambda: iter([("bad_ext", bad), ("good_ext", good)]),
|
|
)
|
|
|
|
with caplog.at_level(logging.WARNING, logger=extensions.log.name):
|
|
installed = extensions.install_all(object(), object(), enabled=["bad_ext", "good_ext"])
|
|
|
|
assert installed == ["good_ext"]
|
|
assert calls == ["bad", "good"]
|
|
assert "bad_ext" in capsys.readouterr().err
|
|
assert "failed to install and was skipped" in caplog.text
|
|
assert "proxy extensions skipped due to install errors: bad_ext" in caplog.text
|
|
|
|
|
|
def test_install_all_warns_for_missing_requested_extension(caplog, monkeypatch) -> None:
|
|
monkeypatch.setattr(extensions, "discover", lambda: iter([]))
|
|
|
|
with caplog.at_level(logging.WARNING, logger=extensions.log.name):
|
|
installed = extensions.install_all(object(), object(), enabled=["missing_ext"])
|
|
|
|
assert installed == []
|
|
assert "proxy extensions requested but not found: missing_ext" in caplog.text
|
|
|
|
|
|
def test_install_all_aborts_startup_when_fail_closed_extension_fails(
|
|
monkeypatch,
|
|
) -> None:
|
|
"""A ``fail_closed`` extension's failure must stop startup, not be skipped.
|
|
|
|
Skipping an extension that mints the upstream credential would leave the
|
|
proxy serving with the client's own ``Authorization`` relayed upstream
|
|
(issue #4737). Every other extension keeps the skip-and-continue default,
|
|
including the ones installed before the failing one.
|
|
"""
|
|
|
|
def auth_ext(app: Any, config: Any) -> None:
|
|
raise RuntimeError("headroom-oauth2 misconfigured: token_url must be https")
|
|
|
|
auth_ext.fail_closed = True # type: ignore[attr-defined]
|
|
|
|
def good(app: Any, config: Any) -> None:
|
|
raise AssertionError("must not install extensions after a fatal one")
|
|
|
|
monkeypatch.setattr(
|
|
extensions,
|
|
"discover",
|
|
lambda: iter([("oauth2", auth_ext), ("good_ext", good)]),
|
|
)
|
|
|
|
with pytest.raises(RuntimeError) as excinfo:
|
|
extensions.install_all(object(), object(), enabled=["oauth2", "good_ext"])
|
|
|
|
assert "'oauth2'" in str(excinfo.value)
|
|
assert "fail_closed" in str(excinfo.value)
|
|
# The plugin's own message survives as the cause, so the operator sees why.
|
|
assert "token_url must be https" in str(excinfo.value.__cause__)
|
|
|
|
|
|
def test_install_all_skips_fail_closed_extension_when_it_succeeds(
|
|
monkeypatch,
|
|
) -> None:
|
|
"""The marker only changes the failure path — a healthy install is unaffected."""
|
|
calls: list[str] = []
|
|
|
|
def auth_ext(app: Any, config: Any) -> None:
|
|
calls.append("auth")
|
|
|
|
auth_ext.fail_closed = True # type: ignore[attr-defined]
|
|
monkeypatch.setattr(extensions, "discover", lambda: iter([("oauth2", auth_ext)]))
|
|
|
|
assert extensions.install_all(object(), object(), enabled=["oauth2"]) == ["oauth2"]
|
|
assert calls == ["auth"]
|
|
|
|
|
|
class _Capture(logging.Handler):
|
|
def __init__(self) -> None:
|
|
super().__init__(logging.DEBUG)
|
|
self.records: list[logging.LogRecord] = []
|
|
|
|
def emit(self, record: logging.LogRecord) -> None:
|
|
self.records.append(record)
|
|
|
|
|
|
def _discover_warnings(monkeypatch, entry_points: Any) -> list[str]:
|
|
monkeypatch.setattr(extensions.importlib.metadata, "entry_points", entry_points)
|
|
capture = _Capture()
|
|
extensions.log.addHandler(capture)
|
|
try:
|
|
assert list(extensions.discover()) == []
|
|
finally:
|
|
extensions.log.removeHandler(capture)
|
|
return [r.getMessage() for r in capture.records if r.levelno == logging.WARNING]
|
|
|
|
|
|
def test_discover_warns_when_entry_point_enumeration_fails(monkeypatch) -> None:
|
|
"""This drops every extension, so it must not hide at DEBUG."""
|
|
|
|
def broken(**_: Any) -> Any:
|
|
raise RuntimeError("corrupt dist-info")
|
|
|
|
warnings = _discover_warnings(monkeypatch, broken)
|
|
|
|
assert len(warnings) == 1
|
|
assert "no proxy extensions will be installed" in warnings[0]
|
|
assert "RuntimeError" in warnings[0]
|
|
assert "corrupt dist-info" not in warnings[0] # detail stays at DEBUG
|
|
|
|
|
|
def test_discover_is_quiet_when_nothing_is_installed(monkeypatch) -> None:
|
|
assert _discover_warnings(monkeypatch, lambda **_: []) == []
|