Commit Graph
3 Commits
Author SHA1 Message Date
673831a456 fix(proxy): answer /api/hello probes behind the token gate, log rejection context (#4150)
## Description

With `HEADROOM_PROXY_TOKEN` set, every Anthropic SDK client logs a
`proxy_auth_rejected` line once per process, Claude Code included and
even when it is configured correctly. Claude Code sends an
unauthenticated `HEAD /api/hello` reachability probe. That probe never
carries `ANTHROPIC_CUSTOM_HEADERS`, so it cannot hold the proxy token:
it gets a 401, which the client ignores, and the rejection lands in the
log. On a busy deployment these lines bury real rejections.

Behind a gateway, the rejection line also showed only the gateway's peer
address, so a rejected caller could not be identified.

This PR:

1. **Answers the probe locally, behind the token.** The HTTP security
gate lets GET/HEAD on exactly `/api/hello` (also under `/p/<project>`)
past the token check and marks it. Extension middleware still runs, so
an extension can still refuse it. The `/api/hello` passthrough route
answers a marked probe with an empty 200 (the gate adds the security
headers) and never relays it. This only happens when a token is
configured. It is deliberately **not** added to `_AUTH_EXEMPT_PATHS`,
which would hand the probe to the passthrough relay: an anonymous
upstream request (to Anthropic since #4059, to the OpenAI default target
before it). Without a token, the probe keeps its #4059 route unchanged.
2. **Gives the rejection line context.** `proxy_auth_rejected` (HTTP and
WebSocket) adds `method=`, `ua=` and `xff=` fields. Both headers are
caller-controlled, so neither is logged as received; each maps to a
fixed vocabulary:
- `ua`: the harness name `CLIENT_UA_MAP` gives the User-Agent, the same
name the outcome and cost records use (`claude-code`, `codex`, `cursor`,
`opencode`, `aider`, `copilot`, …). Failing that, the product token of a
generic HTTP client (`node`, `anthropic`, `openai`, `curl`, `mozilla`,
`python-httpx`, …). Otherwise `other`. `X-Client` is not consulted,
since it is free text.
- `xff`: the leftmost `X-Forwarded-For` hop when it parses as a plain IP
address, else `invalid`. An IPv6 zone index (`fe80::1%<zone>`) is free
text, so it counts as `invalid`. The hop is unverified, used only for
attribution, and drives no decision.
   - Either is `none` when its header is absent.
   - No credential header is read for the log.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/server.py`:
- **`_security_gate`**: when a token is configured, `GET`/`HEAD` on
exactly `/api/hello`, matched on the routed path so
`/p/<project>/api/hello` counts too, skips the token check and sets
`request.state.headroom_local_hello_probe`. It then calls `call_next`
like any request, so extension middleware (installed inside the gate)
still runs and can refuse it. `POST /api/hello`, `/api/hello/` and
`/api/hello/x` still need the token.
- **New helper**: `_auth_rejection_context` returns `(ua, xff)` from the
fixed vocabulary above: `CLIENT_UA_MAP` then an allowlist of generic
clients (`_AUTH_REJECTION_CLIENTS`) for the User-Agent, and
`ipaddress.ip_address` on the first hop (from
`forwarded_policy.header_first`).
- **Both `proxy_auth_rejected` warnings** gain the new fields: the HTTP
gate gets `method=`, and the WebSocket middleware gets the same
`ua=`/`xff=` fields, so the two transports stay aligned.
- `headroom/providers/proxy_routes.py`: the provider passthrough route
answers a marked probe with an empty 200 and never relays it upstream.
- `tests/test_proxy_hardening.py`:
- **Probe answer** (`test_api_hello_probe_is_answered_locally`):
GET/HEAD on `/api/hello` and `/p/demo/api/hello` from a non-loopback
peer gets 200 with an empty body and security headers, and the
passthrough spy shows nothing relayed. `POST /p/demo/api/hello` and `GET
/p/demo/stats` still get 401. `POST /api/hello`, `GET /api/hello/x`,
`GET /api/hello/` and `GET /stats` still get 401, with nothing relayed.
With no token configured, `/api/hello` still reaches the passthrough.
- **Log line**: `claude-cli/2.1.0 (external, cli)` with
`X-Forwarded-For: 198.51.100.7, 10.0.0.9` logs `ua=claude-code
xff=198.51.100.7`. Absent headers log `ua=none xff=none`.
`test_known_clients_keep_their_proxy_wide_name` covers `claude-code`,
`codex`, `cursor`, `opencode`, `aider` behind a wrapper prefix, the
Anthropic SDK and `node`.
- **Caller text never reaches the line**
(`test_caller_text_never_reaches_the_line`): a
`PRIVATE_CREDENTIAL_CANARY` in the User-Agent (bare, as a product token,
behind a fake quoted field) and in X-Forwarded-For (as the first hop, as
a zone index, as a port suffix) never appears in the line, which logs
`other`/`invalid` instead. CR/LF in either header maps to
`other`/`invalid` (tested on the helper directly, since an HTTP client
refuses to send CR/LF).
- **No credentials**: neither the configured token, a wrong
`x-headroom-proxy-token`, `Authorization: Bearer …` nor `x-api-key` ever
appears in the line.
- **WebSocket**: the rejection line matches the HTTP gate's, and the
same canary in both headers never appears in it
(`test_rejection_log_keeps_caller_text_out`).
- `tests/test_proxy_extension_ordering.py`:
- **Extension policy**
(`test_api_hello_probe_still_runs_extension_policy`): an extension that
answers 403 to everything gets the token-less GET/HEAD probe and refuses
it.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_proxy_hardening.py tests/test_anthropic_hello_probe_routing.py \
    tests/test_provider_proxy_routes.py tests/test_provider_route_specs.py \
    tests/test_proxy_loopback_gating.py tests/test_proxy_extension_ordering.py \
    tests/test_proxy_bind_policy.py tests/test_auth_mode.py -q
365 passed in 78.96s

$ ruff check headroom/proxy/server.py tests/test_proxy_hardening.py
All checks passed!
$ ruff format --check headroom/proxy/server.py tests/test_proxy_hardening.py
2 files already formatted

$ python -m mypy headroom/proxy/server.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: this branch on Windows 11 with Python 3.13. Ran `headroom
proxy --port 18787` from the worktree with a throwaway
`HEADROOM_PROXY_TOKEN`.
- Exact command / steps: `curl -I http://127.0.0.1:18787/api/hello`,
`curl http://127.0.0.1:18787/api/hello`, then checked the proxy log.
- Observed result: HEAD and GET returned `200` with 0 bytes and
`x-content-type-options: nosniff`, `x-frame-options: DENY`,
`referrer-policy: no-referrer` and `strict-transport-security`. The
proxy log has no `proxy_inbound_request` and no `outbound_headers` line
for either probe, so nothing was forwarded. For contrast, a loopback
`POST /api/hello` (loopback is token-exempt, existing behavior) logs
`forwarder=openai_passthrough`.
- Not tested: the new rejection-line fields behind a real gateway. They
were checked against a real non-loopback peer, below.

Rejection diagnostics, at `d70ec3c` (contains current `main`): the same
worktree proxy, bound to a non-loopback LAN address (`--host
192.168.0.48 --port 18787 --no-optimize`, `HEADROOM_OFFLINE=1`) with a
throwaway token, called from that address:

```text
GET /stats  UA=PRIVATE_CREDENTIAL_CANARY  XFF=PRIVATE_CREDENTIAL_CANARY, 192.0.2.1          -> 401
GET /stats  UA=claude-cli/2.1.0 (external, cli)  XFF=198.51.100.7, 10.0.0.9                -> 401
GET /stats  UA=codex-cli/0.46.0                                                             -> 401
GET /stats  UA=PRIVATE_CREDENTIAL_CANARY/9 opencode/1.2.3                                   -> 401
GET /stats  no UA, no XFF                                                                   -> 401
GET /stats  UA=PRIVATE_CREDENTIAL_CANARY/1.0  XFF=fe80::1%PRIVATE_CREDENTIAL_CANARY         -> 401
GET /stats  valid x-headroom-proxy-token                                                    -> 200
HEAD /api/hello                                                                             -> 200
WS upgrade /v1/responses  UA=PRIVATE_CREDENTIAL_CANARY  XFF=PRIVATE_CREDENTIAL_CANARY       -> 403

event=proxy_auth_rejected method=GET path=/stats client=192.168.0.48 reason=missing_token ua=other xff=invalid
event=proxy_auth_rejected method=GET path=/stats client=192.168.0.48 reason=missing_token ua=claude-code xff=198.51.100.7
event=proxy_auth_rejected method=GET path=/stats client=192.168.0.48 reason=missing_token ua=codex xff=none
event=proxy_auth_rejected method=GET path=/stats client=192.168.0.48 reason=missing_token ua=opencode xff=none
event=proxy_auth_rejected method=GET path=/stats client=192.168.0.48 reason=missing_token ua=none xff=none
event=proxy_auth_rejected method=GET path=/stats client=192.168.0.48 reason=missing_token ua=other xff=invalid
event=proxy_auth_rejected transport=websocket path=/v1/responses client=192.168.0.48 reason=missing_token ua=other xff=invalid

$ grep -c PRIVATE_CREDENTIAL_CANARY proxy.out
0
```

The HEAD `/api/hello` probe from that non-loopback peer logs
`proxy_inbound_request` / `proxy_inbound_response` (`status=200`,
`response_content_length=0`) and no `outbound_headers` line. Since
`d125805` the probe runs through the inbound request logging like any
routed request; it is still never forwarded. The loopback run above
predates that commit.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: only when `HEADROOM_PROXY_TOKEN` is
set. GET/HEAD `/api/hello` then gets a local 200 instead of a 401, and
the rejection line gains three fields. Answered probes appear in the
inbound request log (`proxy_inbound_request` / `proxy_inbound_response`)
instead of as `proxy_auth_rejected` warnings. With no token, behavior is
unchanged.
- Kill switch / disable path: none needed. The probe answer carries no
data and makes no upstream call.
- Unsafe override required: no.
- Qualification impact: log parsers that match the exact old
`proxy_auth_rejected` line need to accept the extra trailing fields. The
`path=`, `client=` and `reason=` fields are unchanged; the HTTP line now
starts with `method=` before `path=`.
- Rollback path: revert the commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Documentation: N/A. The docs don't list the token-exempt paths or the
`proxy_auth_rejected` format.
- Why only when a token is configured: without a token, the #4059
passthrough route is fine. Answering locally only matters when the
alternative is a 401 for every client.
- `xff` is not resolved through `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS`:
the gate runs before forwarded-header resolution, and resolving here
would emit untrusted-peer events on every rejection. It is logged only
as a validated IP literal (or `invalid`/`none`), unverified, for
attribution.
- `ua` reuses `CLIENT_UA_MAP`, so a harness added there is named in the
rejection line too. Generic clients outside `_AUTH_REJECTION_CLIENTS`
log `other`; adding one is a one-line change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4150"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer combined-tree qualification

On the integration with main `31dcaf41`, all 375 focused hardening,
hello-routing, provider-route/spec, loopback, extension-ordering,
bind-policy, and auth-mode tests passed on Windows/Python 3.12 using the
existing native extension. Changed-file lint, formatting, and whitespace
checks passed. The provider-route insertion conflict retains the PR
local-probe early return before main per-request upstream-override
processing; AST comparison proves the combined route is the unchanged PR
guard followed by the unchanged main route body. The probe test now also
supplies a loopback or metadata-address upstream override and confirms
the response stays empty/local with nothing relayed. This checks the
integration boundary without rewriting contributor authorization or
forwarding logic. Hosted qualification for the published sync remains
pending.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 22:38:51 -06:00
charlieanna ffc6edb4ba fix(proxy): preserve Bedrock body-limit error dialect (#3871)
## Description

Bedrock InvokeModel requests that exceed the request-body ceiling were
returned with `error.type=invalid_request_error`, even though the
Bedrock handler reports the same condition as `request_too_large`. This
makes the middleware response inconsistent with the route's wire
contract.

Closes #3867

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring

## Changes Made

- Added explicit dialect handling for `/model/{id}/invoke` and
`/model/{id}/invoke-with-response-stream`.
- Preserved the existing Anthropic Messages and OpenAI payload formats.
- Added regression coverage for both Bedrock routes, the Anthropic
route, and the OpenAI fallback, including response-once behavior.

## Testing

- [x] Unit tests pass
- [x] Linting passes
- [x] Type checking passes
- [x] New tests added
- [ ] Manual testing performed

### Test Output

```text
41 passed: tests/test_proxy/test_bedrock_passthrough.py tests/test_proxy_extension_ordering.py
55 passed, 2 skipped: related body-limit suites
ruff check: passed
mypy headroom/proxy/request_body_limit.py: passed
```

## Real Behavior Proof

On main, the oversized Bedrock request returned:

```json
{"error":{"message":"Request body too large...","type":"invalid_request_error","code":"request_too_large"}}
```

With this change, the same request returns:

```json
{"error":{"type":"request_too_large","message":"Request body too large..."}}
```

The request is rejected before any upstream call.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Stable/default behavior changed: only the 413 error payload for
Bedrock InvokeModel routes.
- Kill switch / disable path: Not applicable.
- Unsafe override required: No.
- Rollback path: Revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review
- [x] I have added tests proving the fix
- [x] New and existing tests pass locally
- [x] I did not edit `CHANGELOG.md`

## Screenshots (if applicable)

Not applicable.
2026-09-29 22:29:46 -07:00
Tejas ChopraandClaude Fable 5.1 1854fd7fd3 fix(proxy): run extension middleware inside the security gate and body ceiling (#3847)
## Summary

Proxy extensions install *after* the inbound security gate, and
Starlette prepends on `add_middleware`, so every middleware an extension
added was the **outermost** layer of the stack. Three private extensions
inherit the consequence today:

- an extension that answers a request itself (Kiro's `POST /`
interceptor) served **unauthenticated** callers on a token-protected
bind;
- an extension that buffers the request body (tier-router) held an
**unbounded** body before the 401 was ever issued;
- an extension that rewrites `Authorization` (oauth2) did so **before**
the gate compared it, so remote clients using `Authorization: Bearer
<HEADROOM_PROXY_TOKEN>` got 401 the moment oauth2 was enabled.

VAPT-readiness tracker items: **07-F1 (S1), 06-F3 (S2), 03-F4 (S2)**,
theme T2.

This fixes the class in core so every extension inherits it, rather than
patching each plugin:

1. **`install_all` runs before the gate is registered.** Extension
middleware now always sits inside `HEADROOM_PROXY_TOKEN` enforcement on
both transports (HTTP gate + `WebSocketAuthMiddleware`). `_proxy_token`
is still read before extensions run, so an extension cannot change which
credential the gate enforces. `_record_headroom_stack` keeps its
position inside the extensions, so it still does not count requests an
extension rejects.
2. **`RequestBodyLimitMiddleware`** (new,
`headroom/proxy/request_body_limit.py`): a raw ASGI layer between the
gate and the extensions that enforces `MAX_REQUEST_BODY_SIZE` for
everything inside it. A declared oversize `Content-Length` is refused
before any body is read; the stream is metered so chunked/understated
bodies are bounded too; once tripped, the client gets the handlers' own
413 payload (dialect-aware: Anthropic shape on `/v1/messages`, OpenAI
shape elsewhere), the application's late response is discarded, and
`RequestBodyTooLarge` is raised into whatever was reading the stream.
Handlers keep their own check; this layer exists for extension
middleware and for routes that read the body directly.
3. **The order is a contract, not an accident.** `install_all` records
what extensions add on `app.state.extension_middleware`; `extensions.py`
documents what an extension author may rely on (auth first, body
bounded) and the one constraint (an extension cannot be the proxy's auth
layer); `tests/test_proxy_extension_ordering.py` pins the stack
structurally.

Resulting stack, outer → inner:

```
ActivityMiddleware → WebSocketAuthMiddleware → _security_gate → RequestBodyLimitMiddleware
  → [extension middleware] → _record_headroom_stack → CORS → WebSocketProjectPrefix → routes
```

## Reproduction / failing test

New tests run against an `origin/main` snapshot (with only the new
module copied in so they import):

```
FAILED TestExtensionMiddlewareIsInsideTheSecurityGate::test_unauthenticated_request_never_reaches_extension_middleware
FAILED TestExtensionMiddlewareIsInsideTheSecurityGate::test_wrong_token_never_reaches_extension_middleware
FAILED TestExtensionMiddlewareIsInsideTheSecurityGate::test_gate_security_headers_apply_to_extension_responses
FAILED TestExtensionMiddlewareIsInsideTheSecurityGate::test_unauthenticated_websocket_never_reaches_extension_middleware
FAILED TestBodyCeilingAppliesBeforeExtensionMiddleware::test_declared_oversize_body_is_refused_before_extension_buffers_it
FAILED TestBodyCeilingAppliesBeforeExtensionMiddleware::test_chunked_oversize_body_is_cut_off_before_extension_holds_it_all
FAILED TestBodyCeilingAppliesBeforeExtensionMiddleware::test_unauthenticated_oversize_body_is_refused_by_the_gate_first
FAILED TestStackOrderIsPinned::test_extension_middleware_is_inner_to_gate_and_body_limit
FAILED TestStackOrderIsPinned::test_install_all_records_nothing_for_routes_only_extensions
9 failed, 10 passed
```

On this branch: `19 passed`. Also green:
`tests/test_proxy_hardening.py`, `tests/test_proxy_extensions.py`,
`tests/test_proxy/`, `tests/test_provider_proxy_routes.py`,
`tests/test_gateway_*`, `tests/gateway/` — 707 passed, 4 skipped. `ruff
check` / `ruff format --check` clean on 0.16.8 (CI's version).

## Real behavior proof

**Setup:** macOS 15 (Darwin 25.6), Python 3.12.13, starlette 1.3.1 /
fastapi 0.136.3, the real `create_app` served by uvicorn on `0.0.0.0`,
probed from the host's LAN address `10.0.0.39` (non-loopback peer) with
curl. `HEADROOM_PROXY_TOKEN=proof-token`, `HEADROOM_OFFLINE=1`, no
upstream involved. A stand-in extension distribution (entry point
`headroom.proxy_extension`) installs two middlewares that mirror the
problematic shapes: one claims `POST /` and answers it (Kiro's shape),
one buffers the whole body and logs how many bytes it held
(tier-router's shape). Same script served `origin/main` on :8799 and
this branch on :8798.

| Probe (from 10.0.0.39) | origin/main | this branch |
|---|---|---|
| Stack printed at startup | `Activity → ClaimRoot → BufferAll → WSAuth
→ gate → …` | `Activity → WSAuth → gate → RequestBodyLimit → ClaimRoot →
BufferAll → …` |
| `POST /` + vendor header, **no token** | **200**
`{"claimed_by":"hdr_test_ext"}` — extension answered | **401**
`{"error":"unauthorized"}` |
| `POST /` with the proxy token | 200 from extension | 200 from
extension |
| `POST /v1/chat/completions`, token, 105 MB body, Content-Length set |
413 after the extension **buffered 105000000 bytes**
(uploaded=105000000) | **413, uploaded=0**, extension buffered nothing |
| same, `Transfer-Encoding: chunked` | (not run) | 413, extension
buffered nothing |
| 105 MB body, **no token** | 401 after the extension **buffered
105000000 bytes** | 401, uploaded=0 |
| `POST /` from 127.0.0.1, no token | 200 (loopback exempt) | 200
(loopback exempt, unchanged) |
| `GET /health`, no token | 200 | 200 (exempt path, still reaches the
extension after the gate passes it) |

**Not tested:** the real Kiro / oauth2 / tier-router plugins end to end
(they are in separate repos; the stand-in reproduces their middleware
shapes exactly, but their follow-up PRs should re-run against this
core); a WebSocket relay to a live upstream; throughput impact of one
more raw ASGI layer (it is a header scan plus a byte counter per chunk,
no copies).

## Behaviour change / release note

- **Extension middleware is no longer the outermost layer.** Any
extension that relied on running *before* `HEADROOM_PROXY_TOKEN` was
checked — for example to implement its own inbound authentication
*instead of* the proxy token — will now run after the gate. No shipped
extension does this; the documented shape is "apply your own checks in
addition to the token". Extensions that register routes only are
unaffected.
- **The 100 MB request-body ceiling now applies uniformly to every HTTP
route**, including loopback-only admin/telemetry routes that previously
read the body with no bound (`/v1/telemetry/import`, `/settings`,
`/v1/retrieve/tool_call`; tracker 01-F8). A declared oversize body gets
413 before it is read.
- `HEADROOM_PROXY_BODY_TOO_LARGE_STATUS` is honoured by the new layer as
it is by the handlers.

**Decision assumed:** none. The loopback exemption (backlog A-3 /
tracker D1) is unchanged by this PR.

## Follow-ups (other repos)

- `headroom-kiro`: defence-in-depth token re-check in `KiroMiddleware`
for older cores (07-F1).
- `headroom-observability`: open the root span after the gate has passed
the request (07-F1).
- `headroom-tier-router`: keep its own body cap / status-aware latency
recording for older cores (06-F3).
- `plugins/headroom-oauth2`: scope injection to upstream paths and stop
minting for local routes (03-F4, separate PR on this repo).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 16:21:13 -05:00