mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): run extension middleware inside the security gate and body ceiling (#3847)
## Summary
Proxy extensions install *after* the inbound security gate, and
Starlette prepends on `add_middleware`, so every middleware an extension
added was the **outermost** layer of the stack. Three private extensions
inherit the consequence today:
- an extension that answers a request itself (Kiro's `POST /`
interceptor) served **unauthenticated** callers on a token-protected
bind;
- an extension that buffers the request body (tier-router) held an
**unbounded** body before the 401 was ever issued;
- an extension that rewrites `Authorization` (oauth2) did so **before**
the gate compared it, so remote clients using `Authorization: Bearer
<HEADROOM_PROXY_TOKEN>` got 401 the moment oauth2 was enabled.
VAPT-readiness tracker items: **07-F1 (S1), 06-F3 (S2), 03-F4 (S2)**,
theme T2.
This fixes the class in core so every extension inherits it, rather than
patching each plugin:
1. **`install_all` runs before the gate is registered.** Extension
middleware now always sits inside `HEADROOM_PROXY_TOKEN` enforcement on
both transports (HTTP gate + `WebSocketAuthMiddleware`). `_proxy_token`
is still read before extensions run, so an extension cannot change which
credential the gate enforces. `_record_headroom_stack` keeps its
position inside the extensions, so it still does not count requests an
extension rejects.
2. **`RequestBodyLimitMiddleware`** (new,
`headroom/proxy/request_body_limit.py`): a raw ASGI layer between the
gate and the extensions that enforces `MAX_REQUEST_BODY_SIZE` for
everything inside it. A declared oversize `Content-Length` is refused
before any body is read; the stream is metered so chunked/understated
bodies are bounded too; once tripped, the client gets the handlers' own
413 payload (dialect-aware: Anthropic shape on `/v1/messages`, OpenAI
shape elsewhere), the application's late response is discarded, and
`RequestBodyTooLarge` is raised into whatever was reading the stream.
Handlers keep their own check; this layer exists for extension
middleware and for routes that read the body directly.
3. **The order is a contract, not an accident.** `install_all` records
what extensions add on `app.state.extension_middleware`; `extensions.py`
documents what an extension author may rely on (auth first, body
bounded) and the one constraint (an extension cannot be the proxy's auth
layer); `tests/test_proxy_extension_ordering.py` pins the stack
structurally.
Resulting stack, outer → inner:
```
ActivityMiddleware → WebSocketAuthMiddleware → _security_gate → RequestBodyLimitMiddleware
→ [extension middleware] → _record_headroom_stack → CORS → WebSocketProjectPrefix → routes
```
## Reproduction / failing test
New tests run against an `origin/main` snapshot (with only the new
module copied in so they import):
```
FAILED TestExtensionMiddlewareIsInsideTheSecurityGate::test_unauthenticated_request_never_reaches_extension_middleware
FAILED TestExtensionMiddlewareIsInsideTheSecurityGate::test_wrong_token_never_reaches_extension_middleware
FAILED TestExtensionMiddlewareIsInsideTheSecurityGate::test_gate_security_headers_apply_to_extension_responses
FAILED TestExtensionMiddlewareIsInsideTheSecurityGate::test_unauthenticated_websocket_never_reaches_extension_middleware
FAILED TestBodyCeilingAppliesBeforeExtensionMiddleware::test_declared_oversize_body_is_refused_before_extension_buffers_it
FAILED TestBodyCeilingAppliesBeforeExtensionMiddleware::test_chunked_oversize_body_is_cut_off_before_extension_holds_it_all
FAILED TestBodyCeilingAppliesBeforeExtensionMiddleware::test_unauthenticated_oversize_body_is_refused_by_the_gate_first
FAILED TestStackOrderIsPinned::test_extension_middleware_is_inner_to_gate_and_body_limit
FAILED TestStackOrderIsPinned::test_install_all_records_nothing_for_routes_only_extensions
9 failed, 10 passed
```
On this branch: `19 passed`. Also green:
`tests/test_proxy_hardening.py`, `tests/test_proxy_extensions.py`,
`tests/test_proxy/`, `tests/test_provider_proxy_routes.py`,
`tests/test_gateway_*`, `tests/gateway/` — 707 passed, 4 skipped. `ruff
check` / `ruff format --check` clean on 0.16.8 (CI's version).
## Real behavior proof
**Setup:** macOS 15 (Darwin 25.6), Python 3.12.13, starlette 1.3.1 /
fastapi 0.136.3, the real `create_app` served by uvicorn on `0.0.0.0`,
probed from the host's LAN address `10.0.0.39` (non-loopback peer) with
curl. `HEADROOM_PROXY_TOKEN=proof-token`, `HEADROOM_OFFLINE=1`, no
upstream involved. A stand-in extension distribution (entry point
`headroom.proxy_extension`) installs two middlewares that mirror the
problematic shapes: one claims `POST /` and answers it (Kiro's shape),
one buffers the whole body and logs how many bytes it held
(tier-router's shape). Same script served `origin/main` on :8799 and
this branch on :8798.
| Probe (from 10.0.0.39) | origin/main | this branch |
|---|---|---|
| Stack printed at startup | `Activity → ClaimRoot → BufferAll → WSAuth
→ gate → …` | `Activity → WSAuth → gate → RequestBodyLimit → ClaimRoot →
BufferAll → …` |
| `POST /` + vendor header, **no token** | **200**
`{"claimed_by":"hdr_test_ext"}` — extension answered | **401**
`{"error":"unauthorized"}` |
| `POST /` with the proxy token | 200 from extension | 200 from
extension |
| `POST /v1/chat/completions`, token, 105 MB body, Content-Length set |
413 after the extension **buffered 105000000 bytes**
(uploaded=105000000) | **413, uploaded=0**, extension buffered nothing |
| same, `Transfer-Encoding: chunked` | (not run) | 413, extension
buffered nothing |
| 105 MB body, **no token** | 401 after the extension **buffered
105000000 bytes** | 401, uploaded=0 |
| `POST /` from 127.0.0.1, no token | 200 (loopback exempt) | 200
(loopback exempt, unchanged) |
| `GET /health`, no token | 200 | 200 (exempt path, still reaches the
extension after the gate passes it) |
**Not tested:** the real Kiro / oauth2 / tier-router plugins end to end
(they are in separate repos; the stand-in reproduces their middleware
shapes exactly, but their follow-up PRs should re-run against this
core); a WebSocket relay to a live upstream; throughput impact of one
more raw ASGI layer (it is a header scan plus a byte counter per chunk,
no copies).
## Behaviour change / release note
- **Extension middleware is no longer the outermost layer.** Any
extension that relied on running *before* `HEADROOM_PROXY_TOKEN` was
checked — for example to implement its own inbound authentication
*instead of* the proxy token — will now run after the gate. No shipped
extension does this; the documented shape is "apply your own checks in
addition to the token". Extensions that register routes only are
unaffected.
- **The 100 MB request-body ceiling now applies uniformly to every HTTP
route**, including loopback-only admin/telemetry routes that previously
read the body with no bound (`/v1/telemetry/import`, `/settings`,
`/v1/retrieve/tool_call`; tracker 01-F8). A declared oversize body gets
413 before it is read.
- `HEADROOM_PROXY_BODY_TOO_LARGE_STATUS` is honoured by the new layer as
it is by the handlers.
**Decision assumed:** none. The loopback exemption (backlog A-3 /
tracker D1) is unchanged by this PR.
## Follow-ups (other repos)
- `headroom-kiro`: defence-in-depth token re-check in `KiroMiddleware`
for older cores (07-F1).
- `headroom-observability`: open the root span after the gate has passed
the request (07-F1).
- `headroom-tier-router`: keep its own body cap / status-aware latency
recording for older cores (06-F3).
- `plugins/headroom-oauth2`: scope injection to upstream paths and stop
minting for local routes (03-F4, separate PR on this repo).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
c946b6b0b0
commit
1854fd7fd3
@@ -18,6 +18,30 @@ Each ``install`` callable is invoked with the FastAPI ``app`` and the
|
||||
OSS makes no assumptions about what extensions do. The interface is
|
||||
deliberately minimal; extensions own the complexity behind it.
|
||||
|
||||
Where extension middleware sits
|
||||
-------------------------------
|
||||
|
||||
``install_all`` runs **before** the proxy registers its inbound security gate
|
||||
and its request-body ceiling, and Starlette builds the stack so that the last
|
||||
layer registered is the outermost. Middleware an extension adds therefore
|
||||
always runs *inside* both, and an extension author can rely on two things
|
||||
without re-implementing them:
|
||||
|
||||
* a request that reaches extension middleware has already passed
|
||||
``HEADROOM_PROXY_TOKEN`` (when one is configured) — an extension never sees,
|
||||
answers, buffers, or rewrites headers on an unauthenticated request, on
|
||||
either transport (HTTP or WebSocket);
|
||||
* a request body that reaches extension middleware is already bounded by
|
||||
:data:`headroom.proxy.helpers.MAX_REQUEST_BODY_SIZE` — an extension that
|
||||
buffers the body cannot be made to hold more than that.
|
||||
|
||||
The flip side is a constraint: extension middleware cannot act as the
|
||||
proxy's authentication layer, because the proxy's own gate runs first. An
|
||||
extension that needs its own inbound auth applies it in addition to, not
|
||||
instead of, the proxy token. ``install_all`` records every middleware entry an
|
||||
extension adds on ``app.state.extension_middleware`` so the ordering can be
|
||||
asserted by tests rather than assumed.
|
||||
|
||||
Reporting what an extension saved, and what it cost
|
||||
---------------------------------------------------
|
||||
|
||||
@@ -164,9 +188,17 @@ def install_all(
|
||||
wildcard = "*" in enabled_set
|
||||
installed: list[str] = []
|
||||
failed: list[str] = []
|
||||
# Middleware entries added by extensions, in registration order. Starlette
|
||||
# prepends on ``add_middleware``, so the entries an install() adds are
|
||||
# whatever is new at the FRONT of ``app.user_middleware`` afterwards. Kept
|
||||
# on ``app.state`` so the ordering contract in the module docstring is
|
||||
# something the test suite can check against the real app.
|
||||
middleware_list = getattr(app, "user_middleware", None)
|
||||
added_middleware: list[Any] = []
|
||||
for name, install in discovered:
|
||||
if not wildcard and name not in enabled_set:
|
||||
continue
|
||||
before = len(middleware_list) if isinstance(middleware_list, list) else 0
|
||||
try:
|
||||
install(app, config)
|
||||
except Exception as exc: # noqa: BLE001 — one bad extension must not brick the proxy
|
||||
@@ -182,8 +214,17 @@ def install_all(
|
||||
failed.append(name)
|
||||
continue
|
||||
installed.append(name)
|
||||
if isinstance(middleware_list, list):
|
||||
added_middleware.extend(middleware_list[: len(middleware_list) - before])
|
||||
log.info("proxy extension installed: %s", name)
|
||||
|
||||
state = getattr(app, "state", None)
|
||||
if state is not None:
|
||||
try:
|
||||
state.extension_middleware = added_middleware
|
||||
except Exception: # noqa: BLE001 — a bare test double may not accept attributes
|
||||
log.debug("could not record extension middleware on app.state", exc_info=True)
|
||||
|
||||
if failed:
|
||||
skipped = ",".join(sorted(failed))
|
||||
log.warning("proxy extensions skipped due to install errors: %s", skipped)
|
||||
|
||||
@@ -0,0 +1,182 @@
|
||||
"""Request-body ceiling enforced as a raw ASGI layer.
|
||||
|
||||
The proxy already bounds request bodies inside its handlers: every route that
|
||||
reads JSON goes through ``helpers._read_request_body_bytes``, which refuses
|
||||
anything over :data:`headroom.proxy.helpers.MAX_REQUEST_BODY_SIZE` while it is
|
||||
still streaming in. That protects the handlers, and only the handlers. Anything
|
||||
that reads the body *before* a handler runs, which in practice means an ASGI
|
||||
middleware installed by a proxy extension, never sees that check. An extension
|
||||
that buffers the body to inspect or rewrite it (a router that reads
|
||||
``service_tier``, a redactor, a relay) therefore holds an unbounded body for
|
||||
as long as the client cares to send one.
|
||||
|
||||
This module makes the ceiling a property of the ASGI stack instead of a
|
||||
convention every extension has to know about. :class:`RequestBodyLimitMiddleware`
|
||||
is registered by :func:`headroom.proxy.server.create_app` immediately inside
|
||||
the inbound security gate and *outside* every extension, so the guarantee an
|
||||
extension author gets is:
|
||||
|
||||
* an unauthenticated request never reaches extension code (the gate), and
|
||||
* an authenticated request whose body exceeds the ceiling is refused with the
|
||||
same status and payload the handlers use, before any extension buffers it.
|
||||
|
||||
Enforcement is the same two-step the handlers use: a declared
|
||||
``Content-Length`` above the ceiling is refused before the body is read at all,
|
||||
and the body stream itself is metered so an absent or understated
|
||||
``Content-Length`` (chunked transfer) is bounded too. Once the stream trips the
|
||||
ceiling the client gets the refusal, the application's own attempt to respond
|
||||
is discarded, and :class:`~headroom.proxy.helpers.RequestBodyTooLarge` is
|
||||
raised into whatever was reading the stream so it stops.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from collections.abc import Awaitable, Callable, MutableMapping
|
||||
from typing import Any
|
||||
|
||||
from headroom.proxy import helpers as _helpers
|
||||
|
||||
logger = logging.getLogger("headroom.proxy")
|
||||
|
||||
Scope = MutableMapping[str, Any]
|
||||
Receive = Callable[[], Awaitable[MutableMapping[str, Any]]]
|
||||
Send = Callable[[MutableMapping[str, Any]], Awaitable[None]]
|
||||
|
||||
# Paths whose clients speak the Anthropic error dialect. Everything else gets
|
||||
# the OpenAI-style payload, which is what the handlers' own 413s use.
|
||||
_ANTHROPIC_PATH_MARKERS = ("/v1/messages",)
|
||||
|
||||
|
||||
def _too_large_payload(path: str, limit: int) -> bytes:
|
||||
"""Mirror the handlers' own 413 payloads so clients see one shape per dialect."""
|
||||
message = f"Request body too large. Maximum size is {limit // (1024 * 1024)}MB"
|
||||
if any(marker in path for marker in _ANTHROPIC_PATH_MARKERS):
|
||||
payload: dict[str, Any] = {
|
||||
"type": "error",
|
||||
"error": {"type": "request_too_large", "message": message},
|
||||
}
|
||||
else:
|
||||
payload = {
|
||||
"error": {
|
||||
"message": message,
|
||||
"type": "invalid_request_error",
|
||||
"code": "request_too_large",
|
||||
}
|
||||
}
|
||||
return json.dumps(payload).encode("utf-8")
|
||||
|
||||
|
||||
def _declared_content_length(scope: Scope) -> int | None:
|
||||
for name, value in scope.get("headers") or ():
|
||||
if name.lower() == b"content-length":
|
||||
try:
|
||||
return int(value.decode("latin-1").strip())
|
||||
except ValueError:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
class RequestBodyLimitMiddleware:
|
||||
"""Refuse HTTP request bodies above the proxy's ceiling before anything buffers them.
|
||||
|
||||
``max_bytes`` defaults to :data:`headroom.proxy.helpers.MAX_REQUEST_BODY_SIZE`,
|
||||
resolved per request so a test (or a future policy knob) that adjusts the
|
||||
module constant is honoured without rebuilding the app. Non-HTTP scopes
|
||||
(WebSocket, lifespan) pass straight through: WebSocket frames have their own
|
||||
limits in the relay handlers.
|
||||
"""
|
||||
|
||||
def __init__(self, app: Any, *, max_bytes: int | None = None) -> None:
|
||||
self.app = app
|
||||
self._max_bytes = max_bytes
|
||||
|
||||
@property
|
||||
def max_bytes(self) -> int:
|
||||
if self._max_bytes is not None:
|
||||
return self._max_bytes
|
||||
return int(_helpers.MAX_REQUEST_BODY_SIZE)
|
||||
|
||||
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
|
||||
if scope.get("type") != "http":
|
||||
await self.app(scope, receive, send)
|
||||
return
|
||||
|
||||
limit = self.max_bytes
|
||||
path = str(scope.get("path") or "")
|
||||
|
||||
declared = _declared_content_length(scope)
|
||||
if declared is not None and declared > limit:
|
||||
logger.warning(
|
||||
"event=request_body_rejected path=%s reason=content_length declared=%d limit=%d",
|
||||
path,
|
||||
declared,
|
||||
limit,
|
||||
)
|
||||
await self._reject(send, path, limit)
|
||||
return
|
||||
|
||||
# ``received`` meters the stream; ``responded`` records that *we* have
|
||||
# answered, after which anything the application tries to send is
|
||||
# discarded (it is reacting to a body we cut off); ``started`` records
|
||||
# that the application answered first, in which case it is too late
|
||||
# for us to substitute a refusal and we only stop feeding the body.
|
||||
state = {"received": 0, "responded": False, "started": False}
|
||||
|
||||
async def metered_receive() -> MutableMapping[str, Any]:
|
||||
message = await receive()
|
||||
if message.get("type") == "http.request":
|
||||
state["received"] += len(message.get("body") or b"")
|
||||
if state["received"] > limit:
|
||||
logger.warning(
|
||||
"event=request_body_rejected path=%s reason=stream received>%d limit=%d",
|
||||
path,
|
||||
limit,
|
||||
limit,
|
||||
)
|
||||
if not state["started"] and not state["responded"]:
|
||||
state["responded"] = True
|
||||
await self._reject(send, path, limit)
|
||||
raise _helpers.RequestBodyTooLarge(
|
||||
f"Request body exceeds {limit // (1024 * 1024)}MB"
|
||||
)
|
||||
return message
|
||||
|
||||
async def guarded_send(message: MutableMapping[str, Any]) -> None:
|
||||
if state["responded"]:
|
||||
return
|
||||
if message.get("type") == "http.response.start":
|
||||
state["started"] = True
|
||||
await send(message)
|
||||
|
||||
try:
|
||||
await self.app(scope, metered_receive, guarded_send)
|
||||
except _helpers.RequestBodyTooLarge:
|
||||
# Raised by metered_receive above, or by a handler's own bounded
|
||||
# read that nothing caught. Either way the answer is the same 413;
|
||||
# if the application already started a response there is nothing
|
||||
# left to say on the wire.
|
||||
if state["responded"] or state["started"]:
|
||||
return
|
||||
state["responded"] = True
|
||||
await self._reject(send, path, limit)
|
||||
|
||||
@staticmethod
|
||||
async def _reject(send: Send, path: str, limit: int) -> None:
|
||||
body = _too_large_payload(path, limit)
|
||||
await send(
|
||||
{
|
||||
"type": "http.response.start",
|
||||
"status": _helpers.get_body_too_large_status(),
|
||||
"headers": [
|
||||
(b"content-type", b"application/json"),
|
||||
(b"content-length", str(len(body)).encode("ascii")),
|
||||
(b"cache-control", b"no-store"),
|
||||
],
|
||||
}
|
||||
)
|
||||
await send({"type": "http.response.body", "body": body})
|
||||
|
||||
|
||||
__all__ = ["RequestBodyLimitMiddleware"]
|
||||
+42
-14
@@ -179,6 +179,7 @@ from headroom.proxy.project_context import (
|
||||
)
|
||||
from headroom.proxy.prometheus_metrics import PrometheusMetrics # noqa: F401
|
||||
from headroom.proxy.rate_limiter import TokenBucketRateLimiter # noqa: F401
|
||||
from headroom.proxy.request_body_limit import RequestBodyLimitMiddleware
|
||||
from headroom.proxy.request_logger import RequestLogger # noqa: F401
|
||||
from headroom.proxy.savings_tracker import LITELLM_AVAILABLE
|
||||
from headroom.proxy.semantic_cache import SemanticCache # noqa: F401
|
||||
@@ -3816,6 +3817,21 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
# headers, and an audit trail for state-mutating admin endpoints.
|
||||
# WebSocket handshakes are covered separately — see the
|
||||
# WebSocketAuthMiddleware registration just below this block.
|
||||
#
|
||||
# Ordering contract. Starlette's ``add_middleware`` prepends, so the LAST
|
||||
# layer registered is the OUTERMOST at request time. Everything below is
|
||||
# therefore registered inner-to-outer, and the order is load-bearing:
|
||||
#
|
||||
# 1. third-party extensions (``install_all``) — innermost of the three
|
||||
# 2. RequestBodyLimitMiddleware — bounds bodies before
|
||||
# any extension buffers one
|
||||
# 3. _security_gate + WebSocketAuthMiddleware — outermost: auth first
|
||||
#
|
||||
# Extensions used to install *after* the gate, which made their middleware
|
||||
# the outermost layer: an extension that answered a request itself, or
|
||||
# buffered its body, or rewrote ``Authorization``, did so before the proxy
|
||||
# token was ever checked. ``tests/test_proxy_extension_ordering.py`` pins
|
||||
# the order; ``extensions.py`` documents it for extension authors.
|
||||
_proxy_token = config.proxy_token or os.environ.get("HEADROOM_PROXY_TOKEN") or None
|
||||
# Pre-encode once for constant-time comparison (compare_digest on str raises
|
||||
# TypeError for non-ASCII input, which would turn a 401 into a 500).
|
||||
@@ -3845,6 +3861,32 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
"Strict-Transport-Security", "max-age=31536000; includeSubDomains"
|
||||
)
|
||||
|
||||
# Third-party proxy extensions (Enterprise, custom plugins). Discovered via
|
||||
# the `headroom.proxy_extension` entry-point group, but **opt-in only**:
|
||||
# only names listed in config.proxy_extensions (CLI: --proxy-extension,
|
||||
# env: HEADROOM_PROXY_EXTENSIONS) actually get installed. Discovery alone
|
||||
# never runs third-party code. An extension that raises from its install()
|
||||
# disables *itself*: `install_all` logs it, prints a SKIPPED line to stderr
|
||||
# and the proxy keeps serving without that extension (extensions.py).
|
||||
# One bad plugin must not brick the proxy. Note the consequence: a plugin
|
||||
# whose licence gate raises is absent, not fatal — the feature fails closed,
|
||||
# the process does not.
|
||||
#
|
||||
# Installed HERE, before the body limit and the security gate are
|
||||
# registered, so any middleware an extension adds sits INSIDE both (see the
|
||||
# ordering contract above). Routes an extension registers are unaffected
|
||||
# by position. ``_proxy_token`` was read above, before extensions ran, so
|
||||
# an extension cannot change which credential the gate enforces.
|
||||
from headroom.proxy.extensions import install_all as _install_extensions
|
||||
|
||||
_install_extensions(app, config, enabled=getattr(config, "proxy_extensions", None))
|
||||
|
||||
# Body ceiling for everything inside it (extensions and handlers alike).
|
||||
# The handlers keep their own streaming check — this layer exists so that
|
||||
# an extension middleware reading the body is bounded too, and so a
|
||||
# declared oversize Content-Length is refused before any body is read.
|
||||
app.add_middleware(RequestBodyLimitMiddleware)
|
||||
|
||||
# Delegates so the HTTP gate and WebSocketAuthMiddleware read a credential
|
||||
# by exactly one rule; they guard the same token on two transports.
|
||||
_extract_proxy_token = read_proxy_token
|
||||
@@ -3895,20 +3937,6 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
# handshake is refused before any project-prefix or routing work happens.
|
||||
app.add_middleware(WebSocketAuthMiddleware, proxy_token=_proxy_token)
|
||||
|
||||
# Third-party proxy extensions (Enterprise, custom plugins). Discovered via
|
||||
# the `headroom.proxy_extension` entry-point group, but **opt-in only**:
|
||||
# only names listed in config.proxy_extensions (CLI: --proxy-extension,
|
||||
# env: HEADROOM_PROXY_EXTENSIONS) actually get installed. Discovery alone
|
||||
# never runs third-party code. An extension that raises from its install()
|
||||
# disables *itself*: `install_all` logs it, prints a SKIPPED line to stderr
|
||||
# and the proxy keeps serving without that extension (extensions.py:169-183).
|
||||
# One bad plugin must not brick the proxy. Note the consequence: a plugin
|
||||
# whose licence gate raises is absent, not fatal — the feature fails closed,
|
||||
# the process does not.
|
||||
from headroom.proxy.extensions import install_all as _install_extensions
|
||||
|
||||
_install_extensions(app, config, enabled=getattr(config, "proxy_extensions", None))
|
||||
|
||||
# Health & Metrics
|
||||
@app.get("/livez")
|
||||
async def livez():
|
||||
|
||||
@@ -0,0 +1,444 @@
|
||||
"""Extension middleware runs inside the proxy's security gate and body ceiling.
|
||||
|
||||
Regression tests for the ordering contract documented in
|
||||
``headroom/proxy/extensions.py``. Before this contract existed, ``install_all``
|
||||
ran after the gate was registered, and Starlette's prepend-on-register stack
|
||||
made every extension middleware the OUTERMOST layer: an extension that answered
|
||||
``POST /`` itself served unauthenticated callers, one that buffered the body
|
||||
held an unbounded body before the 401 was ever issued, and one that rewrote
|
||||
``Authorization`` did so before the gate compared it.
|
||||
|
||||
The extension used here is a stand-in for the shapes the private extensions
|
||||
actually take (a middleware that claims a request and answers it, and one that
|
||||
buffers the whole body), installed through the real ``install_all`` seam by
|
||||
monkeypatching discovery, so the tests exercise the app exactly as
|
||||
``headroom proxy --proxy-extension <name>`` would.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from headroom.cache.compression_store import reset_compression_store
|
||||
from headroom.proxy import extensions
|
||||
from headroom.proxy import helpers as proxy_helpers
|
||||
from headroom.proxy.request_body_limit import RequestBodyLimitMiddleware
|
||||
from headroom.proxy.server import ProxyConfig, WebSocketAuthMiddleware, create_app
|
||||
|
||||
NONLOOPBACK = ("203.0.113.5", 44444) # TEST-NET-3, never loopback
|
||||
LOOPBACK = ("127.0.0.1", 12345)
|
||||
TOKEN = "s3cr3t-token"
|
||||
|
||||
|
||||
class _RecordingExtension:
|
||||
"""Raw ASGI middleware that records every scope it sees and answers ``POST /``.
|
||||
|
||||
Mirrors the shape of an extension that intercepts a vendor protocol on the
|
||||
root path and relays it upstream itself, never calling the wrapped app.
|
||||
"""
|
||||
|
||||
seen: list[dict[str, Any]] = []
|
||||
|
||||
def __init__(self, app: Any) -> None:
|
||||
self.app = app
|
||||
|
||||
async def __call__(self, scope: Any, receive: Any, send: Any) -> None:
|
||||
if scope["type"] in {"http", "websocket"}:
|
||||
type(self).seen.append({"type": scope["type"], "path": scope.get("path")})
|
||||
if scope["type"] == "http" and scope.get("path") == "/" and scope.get("method") == "POST":
|
||||
body = b'{"claimed_by":"extension"}'
|
||||
await send(
|
||||
{
|
||||
"type": "http.response.start",
|
||||
"status": 200,
|
||||
"headers": [
|
||||
(b"content-type", b"application/json"),
|
||||
(b"content-length", str(len(body)).encode()),
|
||||
],
|
||||
}
|
||||
)
|
||||
await send({"type": "http.response.body", "body": body})
|
||||
return
|
||||
await self.app(scope, receive, send)
|
||||
|
||||
|
||||
class _BufferingExtension:
|
||||
"""Raw ASGI middleware that buffers the full request body before passing it on.
|
||||
|
||||
Mirrors an extension that must read the JSON to decide how to route or
|
||||
rewrite the request. ``max_seen`` records the largest body it ever held.
|
||||
"""
|
||||
|
||||
max_seen = 0
|
||||
|
||||
def __init__(self, app: Any) -> None:
|
||||
self.app = app
|
||||
|
||||
async def __call__(self, scope: Any, receive: Any, send: Any) -> None:
|
||||
if scope["type"] != "http":
|
||||
await self.app(scope, receive, send)
|
||||
return
|
||||
chunks = bytearray()
|
||||
while True:
|
||||
message = await receive()
|
||||
if message["type"] != "http.request":
|
||||
break
|
||||
chunks.extend(message.get("body") or b"")
|
||||
type(self).max_seen = max(type(self).max_seen, len(chunks))
|
||||
if not message.get("more_body"):
|
||||
break
|
||||
replay = {"type": "http.request", "body": bytes(chunks), "more_body": False}
|
||||
sent = False
|
||||
|
||||
async def replay_receive() -> dict[str, Any]:
|
||||
nonlocal sent
|
||||
if not sent:
|
||||
sent = True
|
||||
return replay
|
||||
return {"type": "http.disconnect"}
|
||||
|
||||
await self.app(scope, replay_receive, send)
|
||||
|
||||
|
||||
def _install_recording(app: Any, config: Any) -> None:
|
||||
app.add_middleware(_RecordingExtension)
|
||||
|
||||
|
||||
def _install_buffering(app: Any, config: Any) -> None:
|
||||
app.add_middleware(_BufferingExtension)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _reset_state() -> None:
|
||||
_RecordingExtension.seen = []
|
||||
_BufferingExtension.max_seen = 0
|
||||
|
||||
|
||||
def _make_app(monkeypatch, install, **overrides) -> Any:
|
||||
reset_compression_store()
|
||||
monkeypatch.setattr(extensions, "discover", lambda: iter([("test_ext", install)]))
|
||||
config = ProxyConfig(
|
||||
optimize=False,
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
cost_tracking_enabled=False,
|
||||
proxy_extensions=["test_ext"],
|
||||
**overrides,
|
||||
)
|
||||
return create_app(config)
|
||||
|
||||
|
||||
class TestExtensionMiddlewareIsInsideTheSecurityGate:
|
||||
def test_unauthenticated_request_never_reaches_extension_middleware(self, monkeypatch):
|
||||
app = _make_app(monkeypatch, _install_recording, proxy_token=TOKEN)
|
||||
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
||||
resp = c.post("/", json={}, headers={"x-amz-target": "Vendor.Generate"})
|
||||
assert resp.status_code == 401
|
||||
assert resp.json() == {"error": "unauthorized"}
|
||||
assert _RecordingExtension.seen == [], "extension ran before the proxy token was checked"
|
||||
|
||||
def test_authenticated_request_reaches_extension_middleware(self, monkeypatch):
|
||||
app = _make_app(monkeypatch, _install_recording, proxy_token=TOKEN)
|
||||
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
||||
resp = c.post("/", json={}, headers={"Authorization": f"Bearer {TOKEN}"})
|
||||
assert resp.status_code == 200
|
||||
assert resp.json() == {"claimed_by": "extension"}
|
||||
assert [s["path"] for s in _RecordingExtension.seen] == ["/"]
|
||||
|
||||
def test_wrong_token_never_reaches_extension_middleware(self, monkeypatch):
|
||||
app = _make_app(monkeypatch, _install_recording, proxy_token=TOKEN)
|
||||
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
||||
resp = c.post("/", json={}, headers={"Authorization": "Bearer nope"})
|
||||
assert resp.status_code == 401
|
||||
assert _RecordingExtension.seen == []
|
||||
|
||||
def test_loopback_caller_is_exempt_and_reaches_extension(self, monkeypatch):
|
||||
app = _make_app(monkeypatch, _install_recording, proxy_token=TOKEN)
|
||||
with TestClient(app, base_url="http://127.0.0.1", client=LOOPBACK) as c:
|
||||
resp = c.post("/", json={})
|
||||
assert resp.status_code == 200
|
||||
assert [s["path"] for s in _RecordingExtension.seen] == ["/"]
|
||||
|
||||
def test_health_probe_stays_exempt_with_extension_installed(self, monkeypatch):
|
||||
app = _make_app(monkeypatch, _install_recording, proxy_token=TOKEN)
|
||||
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
||||
resp = c.get("/health")
|
||||
assert resp.status_code == 200
|
||||
# Exempt from auth, so the extension legitimately sees it — the gate
|
||||
# passes it through rather than the extension pre-empting the gate.
|
||||
assert [s["path"] for s in _RecordingExtension.seen] == ["/health"]
|
||||
|
||||
def test_gate_security_headers_apply_to_extension_responses(self, monkeypatch):
|
||||
app = _make_app(monkeypatch, _install_recording, proxy_token=TOKEN)
|
||||
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
||||
resp = c.post("/", json={}, headers={"Authorization": f"Bearer {TOKEN}"})
|
||||
assert resp.headers["X-Content-Type-Options"] == "nosniff"
|
||||
|
||||
def test_unauthenticated_websocket_never_reaches_extension_middleware(self, monkeypatch):
|
||||
app = _make_app(monkeypatch, _install_recording, proxy_token=TOKEN)
|
||||
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
||||
with pytest.raises(Exception): # noqa: B017 — TestClient surfaces the refused upgrade
|
||||
with c.websocket_connect("/v1/responses"):
|
||||
pass
|
||||
assert _RecordingExtension.seen == [], "extension saw a refused WebSocket handshake"
|
||||
|
||||
|
||||
class TestBodyCeilingAppliesBeforeExtensionMiddleware:
|
||||
def test_declared_oversize_body_is_refused_before_extension_buffers_it(self, monkeypatch):
|
||||
monkeypatch.setattr(proxy_helpers, "MAX_REQUEST_BODY_SIZE", 1024)
|
||||
app = _make_app(monkeypatch, _install_buffering, proxy_token=TOKEN)
|
||||
payload = b"x" * 4096
|
||||
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
||||
resp = c.post(
|
||||
"/v1/chat/completions",
|
||||
content=payload,
|
||||
headers={
|
||||
"Authorization": f"Bearer {TOKEN}",
|
||||
"content-type": "application/json",
|
||||
},
|
||||
)
|
||||
assert resp.status_code == 413
|
||||
assert resp.json()["error"]["code"] == "request_too_large"
|
||||
assert _BufferingExtension.max_seen == 0
|
||||
|
||||
def test_chunked_oversize_body_is_cut_off_before_extension_holds_it_all(self, monkeypatch):
|
||||
monkeypatch.setattr(proxy_helpers, "MAX_REQUEST_BODY_SIZE", 1024)
|
||||
app = _make_app(monkeypatch, _install_buffering, proxy_token=TOKEN)
|
||||
chunk = b"x" * 256
|
||||
|
||||
def body():
|
||||
for _ in range(64): # 16 KiB, no Content-Length
|
||||
yield chunk
|
||||
|
||||
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
||||
resp = c.post(
|
||||
"/v1/chat/completions",
|
||||
content=body(),
|
||||
headers={
|
||||
"Authorization": f"Bearer {TOKEN}",
|
||||
"content-type": "application/json",
|
||||
},
|
||||
)
|
||||
assert resp.status_code == 413
|
||||
# The ceiling plus at most one delivered message; never the whole 16 KiB.
|
||||
# (TestClient delivers the generator as one message, so the extension
|
||||
# sees nothing at all here; the multi-message case is covered by
|
||||
# TestRequestBodyLimitMiddlewareDirectly below.)
|
||||
assert _BufferingExtension.max_seen <= 1024 + len(chunk)
|
||||
|
||||
def test_anthropic_dialect_payload_on_messages_path(self, monkeypatch):
|
||||
monkeypatch.setattr(proxy_helpers, "MAX_REQUEST_BODY_SIZE", 1024)
|
||||
app = _make_app(monkeypatch, _install_buffering, proxy_token=TOKEN)
|
||||
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
||||
resp = c.post(
|
||||
"/v1/messages",
|
||||
content=b"x" * 4096,
|
||||
headers={"Authorization": f"Bearer {TOKEN}", "content-type": "application/json"},
|
||||
)
|
||||
assert resp.status_code == 413
|
||||
assert resp.json() == {
|
||||
"type": "error",
|
||||
"error": {
|
||||
"type": "request_too_large",
|
||||
"message": "Request body too large. Maximum size is 0MB",
|
||||
},
|
||||
}
|
||||
|
||||
def test_body_within_ceiling_reaches_extension_intact(self, monkeypatch):
|
||||
monkeypatch.setattr(proxy_helpers, "MAX_REQUEST_BODY_SIZE", 1024)
|
||||
app = _make_app(monkeypatch, _install_buffering, proxy_token=TOKEN)
|
||||
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
||||
# /stats is a local route, so the request completes without an upstream.
|
||||
resp = c.post(
|
||||
"/stats",
|
||||
content=b"y" * 512,
|
||||
headers={"Authorization": f"Bearer {TOKEN}"},
|
||||
)
|
||||
assert resp.status_code != 413
|
||||
assert _BufferingExtension.max_seen == 512
|
||||
|
||||
def test_unauthenticated_oversize_body_is_refused_by_the_gate_first(self, monkeypatch):
|
||||
monkeypatch.setattr(proxy_helpers, "MAX_REQUEST_BODY_SIZE", 1024)
|
||||
app = _make_app(monkeypatch, _install_buffering, proxy_token=TOKEN)
|
||||
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
||||
resp = c.post("/v1/chat/completions", content=b"x" * 4096)
|
||||
assert resp.status_code == 401
|
||||
assert _BufferingExtension.max_seen == 0
|
||||
|
||||
|
||||
class TestStackOrderIsPinned:
|
||||
"""The contract is structural, so check the stack itself, not only its effects."""
|
||||
|
||||
@staticmethod
|
||||
def _index_of(app, predicate) -> int:
|
||||
for index, entry in enumerate(app.user_middleware):
|
||||
if predicate(entry):
|
||||
return index
|
||||
raise AssertionError("middleware not found in stack")
|
||||
|
||||
def test_extension_middleware_is_inner_to_gate_and_body_limit(self, monkeypatch):
|
||||
app = _make_app(monkeypatch, _install_recording, proxy_token=TOKEN)
|
||||
added = app.state.extension_middleware
|
||||
assert len(added) == 1
|
||||
assert added[0].cls is _RecordingExtension
|
||||
|
||||
# Lower index == outer layer (Starlette prepends on add_middleware).
|
||||
ext_index = self._index_of(app, lambda e: e.cls is _RecordingExtension)
|
||||
limit_index = self._index_of(app, lambda e: e.cls is RequestBodyLimitMiddleware)
|
||||
ws_auth_index = self._index_of(app, lambda e: e.cls is WebSocketAuthMiddleware)
|
||||
gate_index = self._index_of(
|
||||
app,
|
||||
lambda e: getattr(e.kwargs.get("dispatch"), "__name__", "") == "_security_gate",
|
||||
)
|
||||
assert ws_auth_index < gate_index < limit_index < ext_index
|
||||
|
||||
def test_install_all_records_nothing_for_routes_only_extensions(self, monkeypatch):
|
||||
def routes_only(app: Any, config: Any) -> None:
|
||||
@app.get("/ext/ping")
|
||||
async def ping():
|
||||
return {"ok": True}
|
||||
|
||||
app = _make_app(monkeypatch, routes_only, proxy_token=TOKEN)
|
||||
assert app.state.extension_middleware == []
|
||||
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
|
||||
assert c.get("/ext/ping").status_code == 401
|
||||
assert c.get("/ext/ping", headers={"Authorization": f"Bearer {TOKEN}"}).json() == {
|
||||
"ok": True
|
||||
}
|
||||
|
||||
|
||||
class TestRequestBodyLimitMiddlewareDirectly:
|
||||
"""Drive the layer with a hand-rolled ASGI conversation to pin the metering."""
|
||||
|
||||
@staticmethod
|
||||
def _http_scope(path="/v1/chat/completions", headers=()):
|
||||
return {"type": "http", "method": "POST", "path": path, "headers": list(headers)}
|
||||
|
||||
@staticmethod
|
||||
def _chunked_receive(chunks):
|
||||
messages = [{"type": "http.request", "body": c, "more_body": True} for c in chunks]
|
||||
messages[-1]["more_body"] = False
|
||||
messages.append({"type": "http.disconnect"})
|
||||
it = iter(messages)
|
||||
|
||||
async def receive():
|
||||
return next(it)
|
||||
|
||||
return receive
|
||||
|
||||
async def test_stream_over_ceiling_answers_413_once_and_stops_the_reader(self):
|
||||
held = bytearray()
|
||||
downstream_sent: list[dict] = []
|
||||
downstream_error: list[BaseException] = []
|
||||
|
||||
async def downstream(scope, receive, send):
|
||||
try:
|
||||
while True:
|
||||
m = await receive()
|
||||
if m["type"] != "http.request":
|
||||
break
|
||||
held.extend(m["body"])
|
||||
if not m.get("more_body"):
|
||||
break
|
||||
except proxy_helpers.RequestBodyTooLarge as exc:
|
||||
downstream_error.append(exc)
|
||||
# A real handler that swallowed this would now try to answer;
|
||||
# that answer must be discarded because the 413 already went out.
|
||||
await send({"type": "http.response.start", "status": 400, "headers": []})
|
||||
await send({"type": "http.response.body", "body": b"{}"})
|
||||
downstream_sent.append({"attempted": True})
|
||||
return
|
||||
raise AssertionError("reader should have been stopped")
|
||||
|
||||
sent: list[dict] = []
|
||||
|
||||
async def send(m):
|
||||
sent.append(m)
|
||||
|
||||
mw = RequestBodyLimitMiddleware(downstream, max_bytes=1000)
|
||||
await mw(self._http_scope(), self._chunked_receive([b"a" * 400] * 5), send)
|
||||
|
||||
assert downstream_error, "the reader was not told to stop"
|
||||
# The message that trips the ceiling is never delivered, so the reader
|
||||
# holds at most the ceiling itself.
|
||||
assert 0 < len(held) <= 1000
|
||||
statuses = [m["status"] for m in sent if m["type"] == "http.response.start"]
|
||||
assert statuses == [413] # exactly one refusal; the downstream 400 was dropped
|
||||
assert downstream_sent == [{"attempted": True}]
|
||||
|
||||
async def test_body_under_ceiling_is_delivered_untouched(self):
|
||||
held = bytearray()
|
||||
|
||||
async def downstream(scope, receive, send):
|
||||
while True:
|
||||
m = await receive()
|
||||
held.extend(m["body"])
|
||||
if not m.get("more_body"):
|
||||
break
|
||||
await send({"type": "http.response.start", "status": 200, "headers": []})
|
||||
await send({"type": "http.response.body", "body": b"ok"})
|
||||
|
||||
sent: list[dict] = []
|
||||
|
||||
async def send(m):
|
||||
sent.append(m)
|
||||
|
||||
mw = RequestBodyLimitMiddleware(downstream, max_bytes=1000)
|
||||
await mw(self._http_scope(), self._chunked_receive([b"a" * 300] * 3), send)
|
||||
assert bytes(held) == b"a" * 900
|
||||
assert [m["status"] for m in sent if m["type"] == "http.response.start"] == [200]
|
||||
|
||||
async def test_response_already_started_is_not_clobbered(self):
|
||||
"""A streaming handler that answered before finishing the body keeps its response."""
|
||||
|
||||
async def downstream(scope, receive, send):
|
||||
await send({"type": "http.response.start", "status": 200, "headers": []})
|
||||
await send({"type": "http.response.body", "body": b"partial", "more_body": True})
|
||||
try:
|
||||
while True:
|
||||
m = await receive()
|
||||
if not m.get("more_body"):
|
||||
break
|
||||
except proxy_helpers.RequestBodyTooLarge:
|
||||
await send({"type": "http.response.body", "body": b"", "more_body": False})
|
||||
|
||||
sent: list[dict] = []
|
||||
|
||||
async def send(m):
|
||||
sent.append(m)
|
||||
|
||||
mw = RequestBodyLimitMiddleware(downstream, max_bytes=1000)
|
||||
await mw(self._http_scope(), self._chunked_receive([b"a" * 600] * 3), send)
|
||||
assert [m["status"] for m in sent if m["type"] == "http.response.start"] == [200]
|
||||
assert sent[-1] == {"type": "http.response.body", "body": b"", "more_body": False}
|
||||
|
||||
async def test_unparseable_content_length_falls_back_to_metering(self):
|
||||
async def downstream(scope, receive, send):
|
||||
while True:
|
||||
m = await receive()
|
||||
if not m.get("more_body"):
|
||||
break
|
||||
|
||||
sent: list[dict] = []
|
||||
|
||||
async def send(m):
|
||||
sent.append(m)
|
||||
|
||||
mw = RequestBodyLimitMiddleware(downstream, max_bytes=100)
|
||||
scope = self._http_scope(headers=[(b"content-length", b"not-a-number")])
|
||||
await mw(scope, self._chunked_receive([b"a" * 80, b"a" * 80]), send)
|
||||
assert [m["status"] for m in sent if m["type"] == "http.response.start"] == [413]
|
||||
|
||||
async def test_non_http_scope_passes_through(self):
|
||||
seen = []
|
||||
|
||||
async def downstream(scope, receive, send):
|
||||
seen.append(scope["type"])
|
||||
|
||||
mw = RequestBodyLimitMiddleware(downstream, max_bytes=1)
|
||||
await mw({"type": "websocket", "path": "/v1/live"}, None, None)
|
||||
await mw({"type": "lifespan"}, None, None)
|
||||
assert seen == ["websocket", "lifespan"]
|
||||
Reference in New Issue
Block a user