mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 18:41:07 +08:00
feat/https-compose-caddy
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
673831a456 |
fix(proxy): answer /api/hello probes behind the token gate, log rejection context (#4150)
## Description With `HEADROOM_PROXY_TOKEN` set, every Anthropic SDK client logs a `proxy_auth_rejected` line once per process, Claude Code included and even when it is configured correctly. Claude Code sends an unauthenticated `HEAD /api/hello` reachability probe. That probe never carries `ANTHROPIC_CUSTOM_HEADERS`, so it cannot hold the proxy token: it gets a 401, which the client ignores, and the rejection lands in the log. On a busy deployment these lines bury real rejections. Behind a gateway, the rejection line also showed only the gateway's peer address, so a rejected caller could not be identified. This PR: 1. **Answers the probe locally, behind the token.** The HTTP security gate lets GET/HEAD on exactly `/api/hello` (also under `/p/<project>`) past the token check and marks it. Extension middleware still runs, so an extension can still refuse it. The `/api/hello` passthrough route answers a marked probe with an empty 200 (the gate adds the security headers) and never relays it. This only happens when a token is configured. It is deliberately **not** added to `_AUTH_EXEMPT_PATHS`, which would hand the probe to the passthrough relay: an anonymous upstream request (to Anthropic since #4059, to the OpenAI default target before it). Without a token, the probe keeps its #4059 route unchanged. 2. **Gives the rejection line context.** `proxy_auth_rejected` (HTTP and WebSocket) adds `method=`, `ua=` and `xff=` fields. Both headers are caller-controlled, so neither is logged as received; each maps to a fixed vocabulary: - `ua`: the harness name `CLIENT_UA_MAP` gives the User-Agent, the same name the outcome and cost records use (`claude-code`, `codex`, `cursor`, `opencode`, `aider`, `copilot`, …). Failing that, the product token of a generic HTTP client (`node`, `anthropic`, `openai`, `curl`, `mozilla`, `python-httpx`, …). Otherwise `other`. `X-Client` is not consulted, since it is free text. - `xff`: the leftmost `X-Forwarded-For` hop when it parses as a plain IP address, else `invalid`. An IPv6 zone index (`fe80::1%<zone>`) is free text, so it counts as `invalid`. The hop is unverified, used only for attribution, and drives no decision. - Either is `none` when its header is absent. - No credential header is read for the log. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/server.py`: - **`_security_gate`**: when a token is configured, `GET`/`HEAD` on exactly `/api/hello`, matched on the routed path so `/p/<project>/api/hello` counts too, skips the token check and sets `request.state.headroom_local_hello_probe`. It then calls `call_next` like any request, so extension middleware (installed inside the gate) still runs and can refuse it. `POST /api/hello`, `/api/hello/` and `/api/hello/x` still need the token. - **New helper**: `_auth_rejection_context` returns `(ua, xff)` from the fixed vocabulary above: `CLIENT_UA_MAP` then an allowlist of generic clients (`_AUTH_REJECTION_CLIENTS`) for the User-Agent, and `ipaddress.ip_address` on the first hop (from `forwarded_policy.header_first`). - **Both `proxy_auth_rejected` warnings** gain the new fields: the HTTP gate gets `method=`, and the WebSocket middleware gets the same `ua=`/`xff=` fields, so the two transports stay aligned. - `headroom/providers/proxy_routes.py`: the provider passthrough route answers a marked probe with an empty 200 and never relays it upstream. - `tests/test_proxy_hardening.py`: - **Probe answer** (`test_api_hello_probe_is_answered_locally`): GET/HEAD on `/api/hello` and `/p/demo/api/hello` from a non-loopback peer gets 200 with an empty body and security headers, and the passthrough spy shows nothing relayed. `POST /p/demo/api/hello` and `GET /p/demo/stats` still get 401. `POST /api/hello`, `GET /api/hello/x`, `GET /api/hello/` and `GET /stats` still get 401, with nothing relayed. With no token configured, `/api/hello` still reaches the passthrough. - **Log line**: `claude-cli/2.1.0 (external, cli)` with `X-Forwarded-For: 198.51.100.7, 10.0.0.9` logs `ua=claude-code xff=198.51.100.7`. Absent headers log `ua=none xff=none`. `test_known_clients_keep_their_proxy_wide_name` covers `claude-code`, `codex`, `cursor`, `opencode`, `aider` behind a wrapper prefix, the Anthropic SDK and `node`. - **Caller text never reaches the line** (`test_caller_text_never_reaches_the_line`): a `PRIVATE_CREDENTIAL_CANARY` in the User-Agent (bare, as a product token, behind a fake quoted field) and in X-Forwarded-For (as the first hop, as a zone index, as a port suffix) never appears in the line, which logs `other`/`invalid` instead. CR/LF in either header maps to `other`/`invalid` (tested on the helper directly, since an HTTP client refuses to send CR/LF). - **No credentials**: neither the configured token, a wrong `x-headroom-proxy-token`, `Authorization: Bearer …` nor `x-api-key` ever appears in the line. - **WebSocket**: the rejection line matches the HTTP gate's, and the same canary in both headers never appears in it (`test_rejection_log_keeps_caller_text_out`). - `tests/test_proxy_extension_ordering.py`: - **Extension policy** (`test_api_hello_probe_still_runs_extension_policy`): an extension that answers 403 to everything gets the token-less GET/HEAD probe and refuses it. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_proxy_hardening.py tests/test_anthropic_hello_probe_routing.py \ tests/test_provider_proxy_routes.py tests/test_provider_route_specs.py \ tests/test_proxy_loopback_gating.py tests/test_proxy_extension_ordering.py \ tests/test_proxy_bind_policy.py tests/test_auth_mode.py -q 365 passed in 78.96s $ ruff check headroom/proxy/server.py tests/test_proxy_hardening.py All checks passed! $ ruff format --check headroom/proxy/server.py tests/test_proxy_hardening.py 2 files already formatted $ python -m mypy headroom/proxy/server.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: this branch on Windows 11 with Python 3.13. Ran `headroom proxy --port 18787` from the worktree with a throwaway `HEADROOM_PROXY_TOKEN`. - Exact command / steps: `curl -I http://127.0.0.1:18787/api/hello`, `curl http://127.0.0.1:18787/api/hello`, then checked the proxy log. - Observed result: HEAD and GET returned `200` with 0 bytes and `x-content-type-options: nosniff`, `x-frame-options: DENY`, `referrer-policy: no-referrer` and `strict-transport-security`. The proxy log has no `proxy_inbound_request` and no `outbound_headers` line for either probe, so nothing was forwarded. For contrast, a loopback `POST /api/hello` (loopback is token-exempt, existing behavior) logs `forwarder=openai_passthrough`. - Not tested: the new rejection-line fields behind a real gateway. They were checked against a real non-loopback peer, below. Rejection diagnostics, at `d70ec3c` (contains current `main`): the same worktree proxy, bound to a non-loopback LAN address (`--host 192.168.0.48 --port 18787 --no-optimize`, `HEADROOM_OFFLINE=1`) with a throwaway token, called from that address: ```text GET /stats UA=PRIVATE_CREDENTIAL_CANARY XFF=PRIVATE_CREDENTIAL_CANARY, 192.0.2.1 -> 401 GET /stats UA=claude-cli/2.1.0 (external, cli) XFF=198.51.100.7, 10.0.0.9 -> 401 GET /stats UA=codex-cli/0.46.0 -> 401 GET /stats UA=PRIVATE_CREDENTIAL_CANARY/9 opencode/1.2.3 -> 401 GET /stats no UA, no XFF -> 401 GET /stats UA=PRIVATE_CREDENTIAL_CANARY/1.0 XFF=fe80::1%PRIVATE_CREDENTIAL_CANARY -> 401 GET /stats valid x-headroom-proxy-token -> 200 HEAD /api/hello -> 200 WS upgrade /v1/responses UA=PRIVATE_CREDENTIAL_CANARY XFF=PRIVATE_CREDENTIAL_CANARY -> 403 event=proxy_auth_rejected method=GET path=/stats client=192.168.0.48 reason=missing_token ua=other xff=invalid event=proxy_auth_rejected method=GET path=/stats client=192.168.0.48 reason=missing_token ua=claude-code xff=198.51.100.7 event=proxy_auth_rejected method=GET path=/stats client=192.168.0.48 reason=missing_token ua=codex xff=none event=proxy_auth_rejected method=GET path=/stats client=192.168.0.48 reason=missing_token ua=opencode xff=none event=proxy_auth_rejected method=GET path=/stats client=192.168.0.48 reason=missing_token ua=none xff=none event=proxy_auth_rejected method=GET path=/stats client=192.168.0.48 reason=missing_token ua=other xff=invalid event=proxy_auth_rejected transport=websocket path=/v1/responses client=192.168.0.48 reason=missing_token ua=other xff=invalid $ grep -c PRIVATE_CREDENTIAL_CANARY proxy.out 0 ``` The HEAD `/api/hello` probe from that non-loopback peer logs `proxy_inbound_request` / `proxy_inbound_response` (`status=200`, `response_content_length=0`) and no `outbound_headers` line. Since `d125805` the probe runs through the inbound request logging like any routed request; it is still never forwarded. The loopback run above predates that commit. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: n/a. - Stable/default behavior changed: only when `HEADROOM_PROXY_TOKEN` is set. GET/HEAD `/api/hello` then gets a local 200 instead of a 401, and the rejection line gains three fields. Answered probes appear in the inbound request log (`proxy_inbound_request` / `proxy_inbound_response`) instead of as `proxy_auth_rejected` warnings. With no token, behavior is unchanged. - Kill switch / disable path: none needed. The probe answer carries no data and makes no upstream call. - Unsafe override required: no. - Qualification impact: log parsers that match the exact old `proxy_auth_rejected` line need to accept the extra trailing fields. The `path=`, `client=` and `reason=` fields are unchanged; the HTTP line now starts with `method=` before `path=`. - Rollback path: revert the commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - Documentation: N/A. The docs don't list the token-exempt paths or the `proxy_auth_rejected` format. - Why only when a token is configured: without a token, the #4059 passthrough route is fine. Answering locally only matters when the alternative is a 401 for every client. - `xff` is not resolved through `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS`: the gate runs before forwarded-header resolution, and resolving here would emit untrusted-peer events on every rejection. It is logged only as a validated IP literal (or `invalid`/`none`), unverified, for attribution. - `ua` reuses `CLIENT_UA_MAP`, so a harness added there is named in the rejection line too. Generic clients outside `_AUTH_REJECTION_CLIENTS` log `other`; adding one is a one-line change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4150" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ### Maintainer combined-tree qualification On the integration with main `31dcaf41`, all 375 focused hardening, hello-routing, provider-route/spec, loopback, extension-ordering, bind-policy, and auth-mode tests passed on Windows/Python 3.12 using the existing native extension. Changed-file lint, formatting, and whitespace checks passed. The provider-route insertion conflict retains the PR local-probe early return before main per-request upstream-override processing; AST comparison proves the combined route is the unchanged PR guard followed by the unchanged main route body. The probe test now also supplies a loopback or metadata-address upstream override and confirms the response stays empty/local with nothing relayed. This checks the integration boundary without rewriting contributor authorization or forwarding logic. Hosted qualification for the published sync remains pending. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
ffc6edb4ba |
fix(proxy): preserve Bedrock body-limit error dialect (#3871)
## Description Bedrock InvokeModel requests that exceed the request-body ceiling were returned with `error.type=invalid_request_error`, even though the Bedrock handler reports the same condition as `request_too_large`. This makes the middleware response inconsistent with the route's wire contract. Closes #3867 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring ## Changes Made - Added explicit dialect handling for `/model/{id}/invoke` and `/model/{id}/invoke-with-response-stream`. - Preserved the existing Anthropic Messages and OpenAI payload formats. - Added regression coverage for both Bedrock routes, the Anthropic route, and the OpenAI fallback, including response-once behavior. ## Testing - [x] Unit tests pass - [x] Linting passes - [x] Type checking passes - [x] New tests added - [ ] Manual testing performed ### Test Output ```text 41 passed: tests/test_proxy/test_bedrock_passthrough.py tests/test_proxy_extension_ordering.py 55 passed, 2 skipped: related body-limit suites ruff check: passed mypy headroom/proxy/request_body_limit.py: passed ``` ## Real Behavior Proof On main, the oversized Bedrock request returned: ```json {"error":{"message":"Request body too large...","type":"invalid_request_error","code":"request_too_large"}} ``` With this change, the same request returns: ```json {"error":{"type":"request_too_large","message":"Request body too large..."}} ``` The request is rejected before any upstream call. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Stable/default behavior changed: only the 413 error payload for Bedrock InvokeModel routes. - Kill switch / disable path: Not applicable. - Unsafe override required: No. - Rollback path: Revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review - [x] I have added tests proving the fix - [x] New and existing tests pass locally - [x] I did not edit `CHANGELOG.md` ## Screenshots (if applicable) Not applicable. |
||
|
|
1854fd7fd3 |
fix(proxy): run extension middleware inside the security gate and body ceiling (#3847)
## Summary
Proxy extensions install *after* the inbound security gate, and
Starlette prepends on `add_middleware`, so every middleware an extension
added was the **outermost** layer of the stack. Three private extensions
inherit the consequence today:
- an extension that answers a request itself (Kiro's `POST /`
interceptor) served **unauthenticated** callers on a token-protected
bind;
- an extension that buffers the request body (tier-router) held an
**unbounded** body before the 401 was ever issued;
- an extension that rewrites `Authorization` (oauth2) did so **before**
the gate compared it, so remote clients using `Authorization: Bearer
<HEADROOM_PROXY_TOKEN>` got 401 the moment oauth2 was enabled.
VAPT-readiness tracker items: **07-F1 (S1), 06-F3 (S2), 03-F4 (S2)**,
theme T2.
This fixes the class in core so every extension inherits it, rather than
patching each plugin:
1. **`install_all` runs before the gate is registered.** Extension
middleware now always sits inside `HEADROOM_PROXY_TOKEN` enforcement on
both transports (HTTP gate + `WebSocketAuthMiddleware`). `_proxy_token`
is still read before extensions run, so an extension cannot change which
credential the gate enforces. `_record_headroom_stack` keeps its
position inside the extensions, so it still does not count requests an
extension rejects.
2. **`RequestBodyLimitMiddleware`** (new,
`headroom/proxy/request_body_limit.py`): a raw ASGI layer between the
gate and the extensions that enforces `MAX_REQUEST_BODY_SIZE` for
everything inside it. A declared oversize `Content-Length` is refused
before any body is read; the stream is metered so chunked/understated
bodies are bounded too; once tripped, the client gets the handlers' own
413 payload (dialect-aware: Anthropic shape on `/v1/messages`, OpenAI
shape elsewhere), the application's late response is discarded, and
`RequestBodyTooLarge` is raised into whatever was reading the stream.
Handlers keep their own check; this layer exists for extension
middleware and for routes that read the body directly.
3. **The order is a contract, not an accident.** `install_all` records
what extensions add on `app.state.extension_middleware`; `extensions.py`
documents what an extension author may rely on (auth first, body
bounded) and the one constraint (an extension cannot be the proxy's auth
layer); `tests/test_proxy_extension_ordering.py` pins the stack
structurally.
Resulting stack, outer → inner:
```
ActivityMiddleware → WebSocketAuthMiddleware → _security_gate → RequestBodyLimitMiddleware
→ [extension middleware] → _record_headroom_stack → CORS → WebSocketProjectPrefix → routes
```
## Reproduction / failing test
New tests run against an `origin/main` snapshot (with only the new
module copied in so they import):
```
FAILED TestExtensionMiddlewareIsInsideTheSecurityGate::test_unauthenticated_request_never_reaches_extension_middleware
FAILED TestExtensionMiddlewareIsInsideTheSecurityGate::test_wrong_token_never_reaches_extension_middleware
FAILED TestExtensionMiddlewareIsInsideTheSecurityGate::test_gate_security_headers_apply_to_extension_responses
FAILED TestExtensionMiddlewareIsInsideTheSecurityGate::test_unauthenticated_websocket_never_reaches_extension_middleware
FAILED TestBodyCeilingAppliesBeforeExtensionMiddleware::test_declared_oversize_body_is_refused_before_extension_buffers_it
FAILED TestBodyCeilingAppliesBeforeExtensionMiddleware::test_chunked_oversize_body_is_cut_off_before_extension_holds_it_all
FAILED TestBodyCeilingAppliesBeforeExtensionMiddleware::test_unauthenticated_oversize_body_is_refused_by_the_gate_first
FAILED TestStackOrderIsPinned::test_extension_middleware_is_inner_to_gate_and_body_limit
FAILED TestStackOrderIsPinned::test_install_all_records_nothing_for_routes_only_extensions
9 failed, 10 passed
```
On this branch: `19 passed`. Also green:
`tests/test_proxy_hardening.py`, `tests/test_proxy_extensions.py`,
`tests/test_proxy/`, `tests/test_provider_proxy_routes.py`,
`tests/test_gateway_*`, `tests/gateway/` — 707 passed, 4 skipped. `ruff
check` / `ruff format --check` clean on 0.16.8 (CI's version).
## Real behavior proof
**Setup:** macOS 15 (Darwin 25.6), Python 3.12.13, starlette 1.3.1 /
fastapi 0.136.3, the real `create_app` served by uvicorn on `0.0.0.0`,
probed from the host's LAN address `10.0.0.39` (non-loopback peer) with
curl. `HEADROOM_PROXY_TOKEN=proof-token`, `HEADROOM_OFFLINE=1`, no
upstream involved. A stand-in extension distribution (entry point
`headroom.proxy_extension`) installs two middlewares that mirror the
problematic shapes: one claims `POST /` and answers it (Kiro's shape),
one buffers the whole body and logs how many bytes it held
(tier-router's shape). Same script served `origin/main` on :8799 and
this branch on :8798.
| Probe (from 10.0.0.39) | origin/main | this branch |
|---|---|---|
| Stack printed at startup | `Activity → ClaimRoot → BufferAll → WSAuth
→ gate → …` | `Activity → WSAuth → gate → RequestBodyLimit → ClaimRoot →
BufferAll → …` |
| `POST /` + vendor header, **no token** | **200**
`{"claimed_by":"hdr_test_ext"}` — extension answered | **401**
`{"error":"unauthorized"}` |
| `POST /` with the proxy token | 200 from extension | 200 from
extension |
| `POST /v1/chat/completions`, token, 105 MB body, Content-Length set |
413 after the extension **buffered 105000000 bytes**
(uploaded=105000000) | **413, uploaded=0**, extension buffered nothing |
| same, `Transfer-Encoding: chunked` | (not run) | 413, extension
buffered nothing |
| 105 MB body, **no token** | 401 after the extension **buffered
105000000 bytes** | 401, uploaded=0 |
| `POST /` from 127.0.0.1, no token | 200 (loopback exempt) | 200
(loopback exempt, unchanged) |
| `GET /health`, no token | 200 | 200 (exempt path, still reaches the
extension after the gate passes it) |
**Not tested:** the real Kiro / oauth2 / tier-router plugins end to end
(they are in separate repos; the stand-in reproduces their middleware
shapes exactly, but their follow-up PRs should re-run against this
core); a WebSocket relay to a live upstream; throughput impact of one
more raw ASGI layer (it is a header scan plus a byte counter per chunk,
no copies).
## Behaviour change / release note
- **Extension middleware is no longer the outermost layer.** Any
extension that relied on running *before* `HEADROOM_PROXY_TOKEN` was
checked — for example to implement its own inbound authentication
*instead of* the proxy token — will now run after the gate. No shipped
extension does this; the documented shape is "apply your own checks in
addition to the token". Extensions that register routes only are
unaffected.
- **The 100 MB request-body ceiling now applies uniformly to every HTTP
route**, including loopback-only admin/telemetry routes that previously
read the body with no bound (`/v1/telemetry/import`, `/settings`,
`/v1/retrieve/tool_call`; tracker 01-F8). A declared oversize body gets
413 before it is read.
- `HEADROOM_PROXY_BODY_TOO_LARGE_STATUS` is honoured by the new layer as
it is by the handlers.
**Decision assumed:** none. The loopback exemption (backlog A-3 /
tracker D1) is unchanged by this PR.
## Follow-ups (other repos)
- `headroom-kiro`: defence-in-depth token re-check in `KiroMiddleware`
for older cores (07-F1).
- `headroom-observability`: open the root span after the gate has passed
the request (07-F1).
- `headroom-tier-router`: keep its own body cap / status-aware latency
recording for older cores (06-F3).
- `plugins/headroom-oauth2`: scope injection to upstream paths and stop
minting for local routes (03-F4, separate PR on this repo).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
|