Commit Graph
18 Commits
Author SHA1 Message Date
d7a2fcf092 test: make the loopback-gating tests able to tell the guard from a missing key (#4784)
## Description

`tests/test_proxy_loopback_gating.py` says non-loopback callers get a
404 on the gated routes, but `test_non_loopback_caller_gets_404` used a
default `TestClient`, whose peer is the string `"testclient"` (not an
IP), and asserted only the 404 status. Two of the twelve gated routes,
`/v1/telemetry/tools/{hash}` and `/v1/toin/pattern/{prefix}`, answer 404
on their own when the key is absent, and `test_loopback_caller_allowed`
accepts `{200, 404, 422}`. So removing the guard from those routes, or
making it refuse every parseable IP, did not turn the test red.

The guard's denial body is `{"detail": "Not Found"}` (what FastAPI
answers for an unknown path); a detail route answers with its own
message. This PR (1) uses a routable remote peer (`203.0.113.9`,
TEST-NET-3) with a loopback `Host`, so only the client-IP check can
refuse it, (2) asserts the guard's exact denial body, and (3) asserts
that a loopback caller never receives that body.

Part of #4783

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `tests/test_proxy_loopback_gating.py`: a `GUARD_DENIAL` constant with
a comment saying why; `test_non_loopback_caller_gets_404` uses
`client=("203.0.113.9", 4242)` and `base_url="http://127.0.0.1"` and
asserts `resp.json() == GUARD_DENIAL`; `test_loopback_caller_allowed`
also asserts `resp.json() != GUARD_DENIAL`. No product code changes.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

`mypy headroom` was not run (no change under `headroom/`). `ruff check`
and `ruff format --check` (0.17.3) are clean.

### Test Output

Whole file, product unchanged:

```text
pytest tests/test_proxy_loopback_gating.py   147 passed
```

The same file run against two mutations of the product (each restored
afterwards), with the old tests (`origin/main`) and the new ones:

```text
Mutation A: loopback_guard.py  `return address.is_loopback`  ->  `return True`
  old tests: 20 failed, 127 passed
  new tests: 32 failed, 115 passed      (the 12 non-loopback cases now fail too)

Mutation B: remove `dependencies=[Depends(_require_loopback)]` from
            /v1/telemetry/tools/{signature_hash} and /v1/toin/pattern/{hash_prefix:path}
  old tests: 147 passed                 (nothing noticed)
  new tests: 2 failed, 145 passed       (the two routes)
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, pytest 9.2.0, the compiled
`headroom._core`, `main` at `3c66c91e`.
- Exact command / steps: `create_app(ProxyConfig(...))` and three
`TestClient`s (default peer, `client=("203.0.113.9", 4242)`,
`client=("127.0.0.1", 12345)`, the last two with
`base_url="http://127.0.0.1"`); `GET` each of
`/v1/telemetry/tools/example`, `/v1/toin/pattern/example`,
`/v1/feedback/example`; then the two mutations above.
- Observed result: both non-loopback peers got `404 {"detail":"Not
Found"}` on all three; the loopback peer got `404 {"detail":"No
telemetry found for signature: example"}`, `404 {"detail":"No TOIN
pattern found with hash starting with: example"}` and `200`
respectively. The mutation results are in Test Output.
- Not tested: CI, other platforms, and the two non-GET gated routes
under mutation B (they are covered by the new body assertion but I did
not mutate them).

## Runtime Rollout Safety

- Rollout-managed feature(s): none; test only.
- Minimum rollout channel: n/a
- Stable/default behavior changed: no.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (this
file; the whole suite was not run)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- The vanilla `TestClient` (peer `"testclient"`) is also refused by the
guard; I did not keep a test for it because it is not a
loopback-versus-remote distinction.

AI assistance: this change, its tests and the runs above were done with
Claude (Sonnet 5.5), operated by me (HetCreep).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4784"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 22:59:20 -06:00
f5c4217648 fix(proxy): enforce HEADROOM_PROXY_TOKEN in the Rust front and listen on loopback by default (#4840)
## Description

The Rust front (`crates/headroom-proxy`) relays every request to the
Python proxy from `127.0.0.1`. The Python proxy treats a loopback caller
as local: it skips the `HEADROOM_PROXY_TOKEN` check and serves `/admin`,
`/debug` and `/settings` to it. The front had no inbound check of its
own, listened on `0.0.0.0:8787` by default, and dropped
`x-headroom-proxy-token` (as an `x-headroom-*` header), so a token set
on the Python side neither protected the stack nor could be presented
through the front.

This PR makes the Rust front apply the same token model as the Python
proxy (`read_proxy_token` / `scrub_proxy_token_headers` in
`headroom/proxy/server.py`, `headroom/proxy/bind_policy.py`) before
anything is forwarded, and makes its default listen address loopback.

Fix adapted from a proposal by HetCreep.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- New `crates/headroom-proxy/src/inbound_auth.rs`, installed as a
router-wide layer (all routes, the catch-all and WebSocket upgrades)
when `HEADROOM_PROXY_TOKEN` / `--proxy-token` is set:
- a non-loopback peer (or one with no peer address) must send the token
as `X-Headroom-Proxy-Token` or `Authorization: Bearer`; the explicit
header wins, as in Python. Constant-time compare. Otherwise `401
{"error":"unauthorized"}` and no upstream contact; a
`proxy_auth_rejected` warning is logged (no credential values).
- loopback peers are exempt (same as Python); `GET /healthz`,
`/healthz/upstream`, `/health`, `/livez`, `/readyz` stay open;
`GET`/`HEAD /api/hello` (also under `/p/<project>`) is answered locally
with an empty 200, as Python does, instead of being relayed.
- the token is removed before forwarding: `x-headroom-proxy-token`
always, `Authorization` only when it carries the token (a provider
credential is left alone).
  - the token is redacted from `Debug` output of `Config`.
- `main.rs`: startup bind policy, same rule and variable as Python: a
non-loopback `--listen` with no token exits with code 2 and a message
naming both remedies, unless `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1`
(then a warning is logged).
- `config.rs`: `--listen` / `HEADROOM_PROXY_LISTEN` default changes from
`0.0.0.0:8787` to `127.0.0.1:8787`; new `--proxy-token` (env
`HEADROOM_PROXY_TOKEN`, value hidden from `--help`).
- `RUST_DEV.md`: runbook exports a token for the `0.0.0.0` cutover, flag
table updated, new "Inbound authentication" section.

**Behaviour change for operators:** a Rust front started with `--listen
0.0.0.0:...` (or relying on the old default to be reachable from other
hosts) now needs `HEADROOM_PROXY_TOKEN` set, and clients on other hosts
must send it. Setups that front the port with something else that
restricts access can set `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1`.
Loopback clients (Claude Code, Codex, SDKs on the same host) are
unaffected. The Docker image's entrypoint is the Python proxy, so it is
unaffected.

## Testing

- [x] `cargo test -p headroom-proxy` (46 test binaries, 608 tests pass)
- [x] `cargo clippy -p headroom-proxy --all-targets -- -D warnings` and
`cargo fmt --all -- --check` clean
- [x] New tests added for new functionality
- [x] Manual testing performed
- [ ] `pytest` / `ruff` / `mypy`: N/A, no Python changes

New tests:
- `tests/integration_inbound_auth.rs` (11 tests, `oneshot` with an
injected `ConnectInfo` so a non-loopback peer can be simulated,
`wiremock` upstream): missing/wrong token is 401 and the upstream sees
nothing; correct token via header or Bearer is forwarded without the
token; provider `Authorization` kept when the header carries the token;
loopback and IPv4-mapped loopback peers pass; missing peer address fails
closed; health probes open but `POST /readyz` is 401; `/api/hello`
answered locally; no token configured keeps passthrough.
- `tests/integration_bind_policy.rs` (3 tests, real binary): `0.0.0.0`
with no token refuses to start; loopback starts; default `--listen` is
loopback.
- unit tests in `inbound_auth.rs` for token parsing, constant-time
compare, bind policy and exemptions.

### Test Output

Before (the bind-policy test file copied onto a clean `origin/main`
worktree; `integration_inbound_auth.rs` does not compile there since
main has no token setting at all):

```text
test default_listen_address_is_loopback ... FAILED
test loopback_listen_without_token_still_starts ... ok
test non_loopback_listen_without_token_refuses_to_start ... FAILED
test result: FAILED. 1 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 20.01s
```

After:

```text
$ cargo test -p headroom-proxy --test integration_inbound_auth --test integration_bind_policy
test default_listen_address_is_loopback ... ok
test non_loopback_listen_without_token_refuses_to_start ... ok
test loopback_listen_without_token_still_starts ... ok
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.01s
test missing_peer_address_fails_closed ... ok
test remote_peer_with_bearer_token_is_forwarded_without_it ... ok
test remote_peer_without_token_gets_401_and_upstream_is_not_contacted ... ok
test token_header_wins_over_a_provider_bearer ... ok
test no_token_configured_keeps_todays_passthrough ... ok
test loopback_peer_without_token_is_still_forwarded ... ok
test ipv4_mapped_loopback_peer_counts_as_loopback ... ok
test remote_peer_with_token_header_is_forwarded_without_it ... ok
test hello_probe_is_answered_locally_without_token ... ok
test remote_peer_with_wrong_token_gets_401 ... ok
test health_probes_stay_reachable_without_token ... ok
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.74s
```

## Real Behavior Proof

- Environment: macOS (Darwin 27.2), debug builds of `headroom-proxy`
from `origin/main` (c21b0813) and from this branch. Upstream is a small
Python HTTP recorder on `127.0.0.1:18788` that prints each request it
receives. Requests are sent with curl to the machine's LAN address
(non-loopback peer) and to `127.0.0.1`.
- Exact command / steps: `HEADROOM_PROXY_TOKEN=s3cret
HEADROOM_PROXY_STATS=false headroom-proxy --listen 0.0.0.0:18787
--upstream http://127.0.0.1:18788`, then `curl
http://<lan-ip>:18787/admin/upstream` with no token, a wrong token and
the right token in `x-headroom-proxy-token`, `GET /readyz` with no token
from the LAN address, and `GET /v1/models` from `127.0.0.1` with no
token.
- Observed result:

```text
== origin/main
GET /admin/upstream (no token)                     -> 200
GET /admin/upstream -H x-headroom-proxy-token:wrong -> 200
GET /admin/upstream -H x-headroom-proxy-token:s3cret -> 200
GET /readyz (no token)                             -> 200
GET /v1/models from 127.0.0.1 (no token)           -> 200
UPSTREAM SAW {"path": "/admin/upstream", "peer": "127.0.0.1", "token_hdr": null, "authorization": null}   (x3)
UPSTREAM SAW {"path": "/readyz", ...}
UPSTREAM SAW {"path": "/v1/models", ...}

== this branch
GET /admin/upstream (no token)                     -> 401
GET /admin/upstream -H x-headroom-proxy-token:wrong -> 401
GET /admin/upstream -H x-headroom-proxy-token:s3cret -> 200
GET /readyz (no token)                             -> 200
GET /v1/models from 127.0.0.1 (no token)           -> 200
UPSTREAM SAW {"path": "/admin/upstream", "peer": "127.0.0.1", "token_hdr": null, "authorization": null}   (only the authenticated one)
UPSTREAM SAW {"path": "/readyz", ...}
UPSTREAM SAW {"path": "/v1/models", ...}

$ headroom-proxy --listen 0.0.0.0:18787 --upstream http://127.0.0.1:18788   # no token
headroom-proxy: Refusing to listen on 0.0.0.0:18787: with no HEADROOM_PROXY_TOKEN this proxy would relay every network peer to the upstream Headroom proxy from 127.0.0.1, where it is treated as a trusted local caller. Either set HEADROOM_PROXY_TOKEN ... or ... set HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1 to acknowledge the open bind.
(exit 2)
```

- Not tested: a full chain with the real Python proxy behind the Rust
front (the recorder stands in for it); WebSocket upgrades through the
gate are covered only by the layer wrapping the catch-all, not by a
dedicated test.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: yes, default listen address is now
loopback, and a non-loopback listen needs a token or the acknowledgement
variable
- Kill switch / disable path: `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1`
restores starting on a non-loopback address without a token; `--listen
0.0.0.0:8787` restores the old address
- Unsafe override required: no
- Qualification impact: none (qualification scripts already pass
`--listen 127.0.0.1:<port>`)
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

A companion PR makes the Rust front's native Bedrock and Vertex routes
opt-in; it relies on this token gate for callers on those routes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reported by @JoaoMarcos44. Thank you for your contribution! 🩵

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-17 09:47:48 +05:30
ab1d0292df fix(proxy): refuse HTTP requests from untrusted browser origins (#4853)
## Description

The provider routes (`/v1/messages`, `/anthropic/v1/messages`,
`/v1/chat/completions`, `/v1/responses`, Bedrock/Vertex routes and the
catch-all passthrough) add operator-held credentials to what they
forward: `*_extra_headers`, the Copilot token
(`apply_copilot_api_auth`), and the provider env key on the
litellm/anyllm backends when the request carries none. Loopback callers
skip `HEADROOM_PROXY_TOKEN`, and a page open in the user's browser
reaches the proxy as a loopback caller. It can send a JSON body as
`Content-Type: text/plain`, which needs no CORS preflight, and the proxy
parses and forwards it. CORS only stops the page from reading the
answer, not the proxy from sending the request. Only the WebSocket
routes checked `Origin`.

The security gate now applies the WebSocket Origin rule to every HTTP
request, before routing. A request is refused with 403 when it carries
an `Origin` that is not:

- a loopback page (`http(s)://127.0.0.1|localhost|[::1]` on any port;
this is the dashboard),
- an origin listed in `HEADROOM_WS_ORIGINS` / `HEADROOM_CORS_ORIGINS`
(the same list `_is_allowed_websocket_origin` honours),
- for a non-loopback caller only, the proxy's own address (a remote
dashboard behind `HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS`). A
loopback caller gets no same-origin pass, since a non-loopback `Host`
with a loopback peer means a page's hostname was pointed at 127.0.0.1.

A cross-site subresource load (`<img>`, `<script>`) sends no `Origin` on
GET, so `Sec-Fetch-Site: cross-site` is refused too, except for a
top-level navigation. Requests without `Origin` (Claude Code, Codex,
Copilot's extension host, SDKs, curl) are unaffected.

Because the check sits in the gate, it covers every backend, including
the litellm/anyllm env-key fallback, without changing how a keyless
request from a local tool is authenticated.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/server.py`: `_is_refused_browser_request(request)` and
a step 0 in `_security_gate` that answers 403 before the token check and
routing. The 403 comes with the usual security headers and a hint naming
`HEADROOM_WS_ORIGINS`, and an `event=proxy_browser_origin_rejected` log
line records the escaped Origin. An origin listed in either
`HEADROOM_WS_ORIGINS` or `HEADROOM_CORS_ORIGINS` is accepted, even when
both are set.
- `tests/test_proxy_browser_origin.py`: text/plain POSTs to
`/v1/messages` and `/v1/chat/completions` from a loopback peer with a
foreign / `null` / look-alike Origin never reach the upstream; without
Origin, from a loopback page, and from a configured origin they still
forward with the operator header; cross-site subresource GET refused;
cross-site navigation still served; loopback peer with a re-pointed
hostname refused.
- `tests/test_proxy_loopback_gating.py`: two existing tests that sent a
foreign `Origin` to `/stats` and `/metrics` now expect the gate's 403
(they already asserted refusal / no metadata; the gate now refuses
earlier).
- `docs/content/docs/security-model.mdx`: describes the rule, and lists
the setups that need an explicit origin: a reverse proxy on the same
host, and app-scheme desktop shells.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# new test file on origin/main (before the fix)
$ pytest tests/test_proxy_browser_origin.py
FAILED ...test_foreign_page_request_is_refused_before_the_upstream[browser_headers0-/v1/chat/completions]
FAILED ...test_foreign_page_request_is_refused_before_the_upstream[browser_headers0-/v1/messages]
FAILED ...test_foreign_page_request_is_refused_before_the_upstream[browser_headers1-/v1/chat/completions]
FAILED ...test_foreign_page_request_is_refused_before_the_upstream[browser_headers1-/v1/messages]
FAILED ...test_foreign_page_request_is_refused_before_the_upstream[browser_headers2-/v1/chat/completions]
FAILED ...test_foreign_page_request_is_refused_before_the_upstream[browser_headers2-/v1/messages]
FAILED ...test_configured_origin_still_forwards
FAILED ...test_cross_site_subresource_without_origin_is_refused
========================= 8 failed, 5 passed in 11.06s =========================

# with the fix
$ pytest tests/test_proxy_browser_origin.py tests/test_proxy_loopback_gating.py tests/test_proxy_cors.py
======================= 175 passed, 5 warnings in 34.36s =======================

# wider sweep: tests/test_proxy*.py tests/test_proxy/ tests/test_header_isolation.py
#   tests/test_upstream_credential_scoping.py tests/test_openai_codex_routing.py
#   tests/test_dashboard_* tests/test_copilot* (-k "not playwright")
= 12 failed, 2368 passed, 114 skipped, 30 deselected in 483.68s =
# the 12 failures (tests/test_proxy_uds.py x11, one test_proxy_compress_endpoint lossy-inline case)
# fail identically on origin/main in this environment

$ ruff check headroom/proxy/server.py tests/test_proxy_browser_origin.py tests/test_proxy_loopback_gating.py
All checks passed!
$ mypy headroom/proxy/server.py --ignore-missing-imports
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS, Python 3.12. `headroom proxy --host 127.0.0.1
--no-optimize --anthropic-api-url http://127.0.0.1:<L>
--anthropic-extra-headers '{"x-api-key":"OPERATOR_KEY_SENTINEL"}'`, no
proxy token, with a small HTTP listener on `<L>` that logs each
request's `x-api-key` and prompt. One run on origin/main, one on this
branch.
- Exact command / steps: three `curl -X POST
http://127.0.0.1:<port>/v1/messages -H 'Content-Type: text/plain' --data
'<messages JSON>'`, with `Origin: http://page.example`, with `Origin:
http://127.0.0.1:<port>`, and with no Origin.
- Observed result:
  ```text
  == origin/main
  Origin: http://page.example -> HTTP 200
  Origin: http://127.0.0.1:18961 -> HTTP 200
  (no Origin) -> HTTP 200
  listener saw:
{"path": "/v1/messages", "x-api-key": "OPERATOR_KEY_SENTINEL", "prompt":
"request-1 (Origin: http://page.example)"}
{"path": "/v1/messages", "x-api-key": "OPERATOR_KEY_SENTINEL", "prompt":
"request-2 (Origin: http://127.0.0.1:18961)"}
{"path": "/v1/messages", "x-api-key": "OPERATOR_KEY_SENTINEL", "prompt":
"request-3 (X-No-Origin: 1)"}
  == this branch
  Origin: http://page.example -> HTTP 403
  Origin: http://127.0.0.1:18962 -> HTTP 200
  (no Origin) -> HTTP 200
  listener saw:
{"path": "/v1/messages", "x-api-key": "OPERATOR_KEY_SENTINEL", "prompt":
"request-2 (Origin: http://127.0.0.1:18962)"}
{"path": "/v1/messages", "x-api-key": "OPERATOR_KEY_SENTINEL", "prompt":
"request-3 (X-No-Origin: 1)"}
  ```
Dashboard on the same branch: `GET /dashboard` as a cross-site
navigation -> 200; same-origin `POST /settings` -> 200; same-origin `GET
/stats` -> 200.
- Not tested: a real Copilot upstream (the refusal happens before any
handler, so it is the same for every route).

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: HTTP requests carrying an `Origin`
that is neither loopback, configured, nor (for a remote caller) the
proxy's own address now get 403. A browser app on another origin that
calls the proxy directly must be listed in `HEADROOM_WS_ORIGINS` (or
`HEADROOM_CORS_ORIGINS`, which also governs CORS). Electron/webview
renderers that send `Origin: null` or an app scheme are refused unless
listed.
- Kill switch / disable path: list the origin in `HEADROOM_WS_ORIGINS`.
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

A companion PR (#4846) stops a `*` entry in `HEADROOM_WS_ORIGINS` /
`HEADROOM_CORS_ORIGINS` from accepting every Origin. Both PRs share
`_is_allowed_websocket_origin`, so once both land `*` is ignored on HTTP
too; until then, `HEADROOM_CORS_ORIGINS=*` keeps HTTP open to every
Origin, as it keeps the WebSocket routes today.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-14 10:46:06 +05:30
e15e342d9b fix(proxy): resolve the forwarded client from the right and keep settings writes loopback-only (#4847)
## Description

Two fixes for proxies run with `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS`
(opt-in, empty by default):

1. **Client address.** The proxy took the *leftmost* `X-Forwarded-For`
element as the client. A gateway that appends to the header keeps
whatever the client itself sent on the left, so the client could pick
its own address (for example `127.0.0.1`). The client is now the
right-most hop that is not itself inside a trusted-gateway CIDR. If
every hop is trusted it falls back to the leftmost, as before.
2. **Settings writes.** A peer inside a trusted-gateway CIDR counted as
loopback in `_request_is_loopback`, so `POST /settings` and `POST
/settings/apply` (which restarts the proxy) were open to anything routed
through the gateway. That rule exists so a containerised dashboard can
show its read-only views. Writes now need a real loopback peer, or a
client inside `HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS` (the
existing #2466 path, with IP-literal Host and same-origin checks).

**Behaviour change:** someone who runs Headroom in a Docker bridge
container, sets only `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS`, and saves
settings from the host browser via `localhost` will now get 404 on save.
Reads still work. To keep saving, add the gateway address to
`HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS` (e.g. `172.17.0.1/32`)
and open the dashboard at `http://127.0.0.1:<port>`. That path is
checked below. Without the env var nothing changes.

Fix adapted from a proposal by HetCreep.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/proxy/forwarded_policy.py`: new `header_client()` walks
`X-Forwarded-For` from the right and skips trusted-gateway hops.
`resolve_forwarded_headers` uses it.
- `headroom/proxy/server.py`: `_request_is_loopback` /
`_request_can_view_dashboard_metadata` take `allow_gateway_peer`
(default unchanged). `POST /settings` and `POST /settings/apply` use a
new write guard with `allow_gateway_peer=False`, and so does the
same-origin guard for those routes.
- Tests: the leftmost-XFF test now expects the right-to-left resolution.
Added tests for a client-written XFF, the all-trusted fallback, settings
reads still served to a gateway peer, writes refused to a gateway peer
(both routes), and writes still allowed on loopback.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# new tests against origin/main (c21b0813)
$ pytest tests/test_forwarded_headers.py tests/test_proxy_loopback_gating.py
FAILED tests/test_forwarded_headers.py::test_client_written_x_forwarded_for_is_not_the_client
FAILED tests/test_proxy_loopback_gating.py::test_settings_write_refused_for_trusted_gateway_peer[/settings]
FAILED tests/test_proxy_loopback_gating.py::test_settings_write_refused_for_trusted_gateway_peer[/settings/apply]
================== 3 failed, 189 passed, 5 warnings in 46.16s ==================

# this branch
$ pytest tests/test_forwarded_headers.py tests/test_proxy_loopback_gating.py tests/test_proxy_hardening.py tests/test_header_isolation.py tests/*settings* tests/*dashboard*
4 failed, 456 passed, 1 skipped
# the 4 failures are tests/test_dashboard/test_live_feed.py (Playwright, needs a live server
# on :8787: net::ERR_CONNECTION_REFUSED). They fail the same way on origin/main.

$ ruff check / ruff format --check (0.17.3) on changed files: All checks passed!
$ mypy headroom/proxy/forwarded_policy.py headroom/proxy/server.py --ignore-missing-imports
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, `create_app(ProxyConfig(...))` driven
by Starlette TestClient with a set peer address.
`HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS=172.17.0.0/16`.
- Exact command / steps: resolve the client for peer `172.17.0.9` with
`X-Forwarded-For: 127.0.0.1, 203.0.113.9`. Then GET/POST `/settings` as
peer `172.17.0.9` with `Host: localhost:8787`, and POST as peer
`127.0.0.1`.
- Observed result:
```text
                         origin/main   this branch
client ip                127.0.0.1     203.0.113.9
gateway peer GET  /settings   200          200
gateway peer POST /settings   200          404
loopback     POST /settings   200          200
```
Opt-in path: with
`HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS=172.17.0.1/32`, peer
`172.17.0.1`, `Host: 127.0.0.1:8787`, same-origin `Origin`: `POST
/settings` → 200.
- Not tested: a real Docker bridge deployment behind a real reverse
proxy.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: see Description
- Kill switch / disable path: none needed
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

`_auth_rejection_context` still logs the leftmost hop. That is log-only,
documented as unverified, and not used for any decision, so it is left
as is.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4847"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-14 10:45:42 +05:30
lorenzozaneandJD Davis e546e71aeb fix(proxy): allow loopback-published container hosts to access /debug/memory (#4259)
## Description

A request forwarded through a loopback-published container port reaches
the proxy from the container gateway, so the strict loopback peer check
returned 404 even though `/debug/memory` was registered. This changes
the route to the existing exact-gateway guard while retaining its
loopback Host check.

Fixes #4247

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Use the existing loopback-or-container-gateway dependency for
`/debug/memory`.
- Add regression coverage for loopback, the configured gateway, another
bridge peer, and a non-loopback Host.

## Testing

- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
LITELLM_LOCAL_MODEL_COST_MAP=true uv run --no-sync pytest tests/test_proxy_loopback_gating.py tests/test_proxy_debug_endpoints.py -q
176 passed, 6 warnings
.venv/bin/ruff check headroom/proxy/server.py tests/test_proxy_loopback_gating.py
All checks passed
.venv/bin/ruff format --check headroom/proxy/server.py tests/test_proxy_loopback_gating.py
Passed
```

## Real Behavior Proof

- Environment: Linux, Python 3.14.7, local source checkout; no external
provider request.
- Exact command / steps: run Uvicorn from this checkout on
`127.0.0.1:18787`, then request `/openapi.json` and `/debug/memory` over
loopback.
- Observed result: both requests returned HTTP 200; the memory response
contained the `process` statistics.
- Not tested: actual Docker port publication. The configured gateway,
unrelated bridge peer, and non-loopback Host cases were exercised
through the ASGI test client.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: `/debug/memory` accepts the exact
container gateway peer when the Host remains loopback.
- Kill switch / disable path: revert this change.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A.

## Additional Notes

An Issue comment links a separate `oxura/headroom` branch and says an
upstream PR is queued; current checks found no upstream PR for this
issue or this submission branch. This run did not test Docker port
publication, the full test suite, or upstream CI.
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4259"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer container-runtime qualification

- Environment: Docker Engine 29.9.2 on a Windows host, Linux
container/Python 3.12.3, mounted source from the current-main
integration. The listener used `create_app(ProxyConfig(host="0.0.0.0",
optimize=False, cache_enabled=False, rate_limit_enabled=False,
disable_kompress=True, stateless=True))` with Uvicorn on port 8787.
`HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` explicitly acknowledged the
tokenless container listener; `HEADROOM_REQUIRE_RUST_CORE=false`
exercised Python authorization/routes without claiming a Linux native
build or compression qualification. Telemetry and model warmup were
disabled.
- Exact command / steps: published only `127.0.0.1:18787:8787`;
requested `/debug/memory` from the host using httpx with environment
proxies disabled; recorded the actual ASGI TCP peer and Host with
observational middleware. Restarted the same container/dependencies
against unchanged main `31dcaf41`, then against the combined PR source.
Requested the container bridge address from a separate container while
explicitly setting a loopback Host. Finally configured
`HEADROOM_PROXY_TOKEN` and repeated host and unrelated-peer requests,
including a spoofed `X-Forwarded-For: 127.0.0.1`. No provider calls were
made.
- Observed result:

```text
actual host-forwarded TCP peer: 172.17.0.1
main /debug/memory via loopback publication: 404
PR /debug/memory via loopback publication: 200, process statistics present
PR gateway peer + Host attacker.example: 404
unrelated bridge container + loopback Host: 404
proxy token configured, host request without token: 401
proxy token configured, host request with token: 200
unrelated bridge container + valid token + spoofed forwarded loopback IP: 404
```

- Not tested: Podman, other Docker network drivers/host platforms,
production reverse proxies, provider traffic, or native compression
inside the container. The Windows focused suite separately passed all
176 loopback/debug tests using the existing native extension;
changed-file lint, formatting, and whitespace checks passed.

Supported deployment boundary: tokenless access assumes a port published
only on host loopback and trust in processes already running on that
host. The exact gateway peer is a transport boundary, not proof of the
original caller behind a host relay; Host is caller-controlled. Do not
expose a tokenless host relay/public reverse proxy to this listener.
Such forwarding must enforce authentication (for example the proxy
token, whose rejection/acceptance was verified above, or equivalent
relay authentication). This qualification clears the earlier missing
published-port evidence without claiming that a loopback Host alone
authenticates a remote caller.

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 22:40:37 -06:00
Mansurshakh Japarov 278bbc8771 fix(wrap): warn when reused proxy ignores compression settings (#4019)
## Description

A healthy shared proxy is intentionally not restarted: doing so would
interrupt other clients. Explicit startup compression settings must not
be silently ignored when a wrapper reuses it. This implements the
issue's warning-on-reuse alternative.

Closes #3588.

## Type of Change

- [x] Bug fix
- [x] Documentation update

## Changes Made

- Expose the effective tool exclusions alongside the existing effective
minimum-token setting in `/health`.
- Warn when explicitly supplied `HEADROOM_MIN_TOKENS` or
`HEADROOM_EXCLUDE_TOOLS` differs from the running configuration.
- Compare tool names case-insensitively and include built-in exclusions
on both sides. Skip comparisons when older health responses lack the
setting or the requested integer is invalid.
- Preserve the existing mode warning and shared-proxy lifecycle.
Document startup-only configuration and the restart/separate-port
remedies.
- Query the running settings for explicit compression environment
variables even with `--no-proxy` and no explicit `HEADROOM_MODE`.

## Testing

- [x] Focused unit tests: `python -m pytest -q
tests/test_cli/test_wrap_bob.py` — **59 passed**. With current main's
dashboard-license tests: **67 passed**.
- [x] New consumer-visible warning regressions: **4 failed on unchanged
baseline**; the additional no-start/MODE-unset cases produced **2
failures before the final fix**, passing afterwards.
- [x] Changed-file Ruff and formatting checks; installed pre-commit
hooks, including `mypy headroom` and Commitlint, passed.
- [x] Manual testing performed.
- [ ] Full-tree `ruff check .`: blocked locally by EXE002 on five
unchanged OAuth2 plugin files checked out with executable filesystem
permissions (Git tracks them as 100644). No unrelated plugin files or
permissions changed.
- [ ] Separate full dev-environment `mypy headroom`: found one
`truthy-function` error in unchanged
`headroom/integrations/langchain/chat_model.py:394`. The file is
byte-identical to baseline (SHA256
`701a9fe8457907406a2b2810211c1534bc9dfcacf3d841ec4c27d56897209bc7`); the
project's isolated pre-commit mypy environment passed.

The mandatory pre-push `make ci-precheck` completed: workspace Rust
fmt/clippy/tests, native extension build/import verification, Python
gate **177 passed / 4 skipped**, and Commitlint all passed. The native
model gate used `ORT_DYLIB_PATH` pointing to the installed CPU
`libonnxruntime.so.1.26.0`; the cached Kompress parity suite matched
**21 fixtures byte-for-byte**. Implicit runtime selection did not finish
and was cancelled, not reported as a pass; explicit CPU runtime
qualification completed without skipping the cached-model assertions.

### Test Output

```text
[build_rust_extension] headroom._core build + install + verify: OK
================== 177 passed, 4 skipped, 1 warning in 7.29s ===================
✅ ci-precheck PASSED — safe to push.
```

## Real Behavior Proof

- Environment: Linux x86_64, CPython 3.13.14; original baseline
`dea9092e19385f286b7f67a64f4d85de2708d756` and final fix rebased onto
current main `1ad92f0cbc3dea27676e253588100f64a2af6994`; real CLI proxy
and real health endpoint.
- Exact command / steps: start the real proxy, then reuse it with
`HEADROOM_MODE` unset and conflicting compression settings without
starting/restarting it; commands below.
  ```sh
HEADROOM_BEACON=off HEADROOM_MIN_TOKENS=120 HEADROOM_EXCLUDE_TOOLS='' \
headroom proxy --host 127.0.0.1 --port 18795 --stateless --no-optimize \
--no-cache --no-rate-limit --no-subscription-tracking --disable-kompress
  ```
- With `HEADROOM_MODE` unset, reuse it without starting/restarting a
proxy:
  ```sh
  HEADROOM_MIN_TOKENS=2000 HEADROOM_EXCLUDE_TOOLS=Bash,Read,Grep,Glob \
python -c 'from headroom.cli.wrap import _ensure_proxy; print("RESULT",
_ensure_proxy(18795, True))'
  ```
- Before the final no-start-path fix: only `RESULT (None, 18795)`; both
explicit settings were silent. The original baseline was also silent.
- Observed result: both explicit-setting warnings appeared after the
fix; the same PID and running configuration remained unchanged. Output:
  ```text
Warning: this session requested HEADROOM_MIN_TOKENS=2000 (running: 120);
HEADROOM_EXCLUDE_TOOLS=['bash', 'glob', 'grep', 'read'] (running: []),
but those settings are fixed at proxy startup. Restart the proxy, or use
--port for a separate one.
  RESULT (None, 18795)
{"proxy_pid_before": 1231619, "proxy_pid_after": 1231619, "min_tokens":
120, "exclude_tools": [], "no_proxy": true, "explicit_mode": null}
  ```
- Not tested: an external interactive agent session or paid upstream
model request. The actual proxy process, health response, and wrapper
reuse path were exercised.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; diagnostic-only wrapper fix.
- Minimum rollout channel: stable; no rollout-managed feature is
enabled.
- Stable/default behavior changed: a warning only when explicit settings
conflict with the running proxy; no configuration mutation or forced
restart.
- Kill switch / disable path: remove the conflicting explicit
environment setting, or revert the commit; no new switch.
- Unsafe override required: no.
- Qualification impact: wrapper diagnostics only; no model/compression
rollout change.
- Rollback path: revert this commit; the extra health field is additive.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] Existing conventions reused; documentation updated.
- [x] Regression tests cover mismatches, matching settings, built-in
exclusions, missing fields, and invalid integers.
- [x] `CHANGELOG.md` not edited; Conventional Commit title supplied for
release-please.

The intentional warning is the requested behavior, not a new unsolicited
runtime warning. Full remote CI results will be reported by GitHub, not
inferred from local checks.
2026-10-07 14:17:35 -05:00
Mansurshakh JaparovandJD Davis def6d41185 fix(toin): make diagnostic pattern identifiers addressable (#4021)
## Description

Return the full scoped aggregation key, using the issue's option 2,
rather than truncating it. The current key contains tenant, auth mode,
model family, and signature; truncating to 12 characters makes different
patterns share IDs and makes list-to-detail lookup silently return a
different pattern.

Closes #2928.

## Type of Change

- [x] Bug fix
- [x] Documentation update
- [x] Intentional diagnostic contract change: listed `hash` values are
complete scoped identifiers; ambiguous abbreviated lookups now return
409 instead of an arbitrary first match.

## Changes Made

- Return the exact stored scoped key from `/v1/toin/patterns`.
- Resolve exact keys first; allow an abbreviated prefix only if it
matches one pattern. Keep 404 for unknown IDs and return 409 for
ambiguity.
- Capture the complete URL-decoded identifier, including a slash in a
tenant ID, without falling through to an upstream passthrough route.
- Preserve loopback authorization and the existing privacy whitelist.
- Update the TypeScript SDK README to round-trip the listed ID. No SDK
runtime change: `getPattern` already URL-encodes the string.

## Testing

- [x] Unit tests pass (`pytest`), including scoped-ID round-trips and
authorization regressions.
- [x] Changed-file lint, configured pre-commit mypy and manual
production HTTP checks pass.

### Test Output

```text
================== 144 passed, 1 warning in 71.73s (0:01:11) ===================
```

The mandatory pre-push `make ci-precheck` passed: workspace Rust
fmt/clippy/tests, native extension build/import verification, the
complete configured Python gate, and Commitlint. The native gate used
the installed CPU `libonnxruntime.so.1.26.0` explicitly via
`ORT_DYLIB_PATH`, without skipping cached-model assertions.

```text
[build_rust_extension] headroom._core build + install + verify: OK
================== 177 passed, 4 skipped, 1 warning in 9.70s ===================
✅ ci-precheck PASSED — safe to push.
```

```sh
python -m pytest -q tests/test_proxy_loopback_gating.py
```

**144 passed**, one existing Starlette TestClient deprecation warning.
The new consumer regressions produce **2 failed, 2 passed** against the
unchanged baseline (preloaded baseline modules prevent foreign-worktree
conftest imports from selecting the treatment).

Changed-file Ruff, formatting, installed pre-commit `mypy headroom`, and
Commitlint hooks passed. Full-tree Ruff was not green locally because
five unchanged OAuth2 plugin files have executable filesystem
permissions while Git tracks 100644. A separate full dev-environment
mypy run found an unchanged optional LangChain `truthy-function` error;
the isolated configured pre-commit mypy hook passed. No unrelated source
or permissions were changed.

## Real Behavior Proof

- Environment: Linux x86_64, CPython 3.13.14; unchanged
`dea9092e19385f286b7f67a64f4d85de2708d756` and this branch, real
production FastAPI app over loopback HTTP (ports 18792/18793), actual
TOIN instance and public `import_patterns`, no mocked endpoint or fake
TOIN.
- Exact command / steps: import four patterns
`global|unknown|unknown|abc123`, `global|unknown|unknown|abc456`,
`global|payg|claude|abc123`, `other-tenant|unknown|unknown|abc123` with
counts 8, 16, 24, 32; GET `/v1/toin/patterns`; URL-encode each returned
`hash` and GET `/v1/toin/pattern/{hash}`; request the shared prefix
`global|unknown|unknown|abc`.
- Before: four entries produced only three IDs. The entry with count 16
returned count 8 when addressed by its listed ID. The shared prefix
returned 200.
- Observed result: after the fix, four entries produce four distinct
IDs; every detail request returns its own count (8→8, 16→16, 24→24,
32→32); the ambiguous prefix returns **409**. Exact and unique-prefix
lookups remain successful; unknown-prefix regression remains 404.
- Rebased onto current main `1ad92f0cbc3dea27676e253588100f64a2af6994`
and reran the full suite: **144 passed**. A further real HTTP smoke on
port 18794 imported a fifth key,
`tenant/alpha?scope=one#fragment|unknown|unknown|abc123`, URL-encoded
it, and round-tripped all **5/5** counts including **40→40**. The
reserved-character edge case failed before the path-converter fix and
passed after it.
- Not tested: a live remote tenant deployment or the SDK against a cloud
proxy. Existing authorization/privacy tests were exercised locally.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; diagnostic API bug fix.
- Minimum rollout channel: stable; no rollout-managed feature is
enabled.
- Stable/default behavior changed: full scoped list identifiers and
rejection of ambiguous prefixes. This is the selected issue alternative,
not a new raw-signature alias or migration shim.
- Kill switch / disable path: disable access to the loopback diagnostic
API; no new runtime flag.
- Unsafe override required: no.
- Qualification impact: diagnostic IDs only; no compression or billing
change.
- Rollback path: revert this commit; no persistent data migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] Existing API and privacy conventions reused; SDK documentation
updated.
- [x] Regression tests cover scoped collisions, list-to-detail
round-trips, exact/unique/ambiguous/missing lookups, and existing
loopback gating.
- [x] `CHANGELOG.md` not edited; Conventional Commit title supplied.

GitHub CI results are not inferred from local checks.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4021"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-07 14:07:38 -05:00
Satya BharadwajandJD Davis 2297b5059f fix(proxy): add same-origin check to /v1/retrieve/tool_call (#3955)
## Description

`POST /v1/retrieve` checks the `Origin` header (`require_same_origin`),
so a web page you visit cannot make your browser call it. Its sibling
`POST /v1/retrieve/tool_call` only had the loopback check, so it was
missed when #3060 added the guard to the other write routes. This PR
adds the same check to it.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/server.py`: add `Depends(_require_same_origin)` to
`/v1/retrieve/tool_call`.
- `tests/test_proxy_loopback_gating.py`: add the route to
`CSRF_GUARDED`, so it gets the existing cross-origin, `Origin: null`,
same-origin and no-`Origin` tests.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q tests/test_proxy_loopback_gating.py
103 passed, 1 warning in 9.59s

$ pytest -q tests/test_proxy_loopback_gating.py -k tool_call   # without the fix
2 failed, 3 passed, 98 deselected, 1 warning in 1.71s

$ pytest -q tests/test_proxy*.py tests/test_ccr*.py tests/test_identity_resolution.py
1404 passed, 107 skipped, 3 warnings in 90.38s

$ ruff check headroom/proxy/server.py tests/test_proxy_loopback_gating.py && ruff format --check headroom/proxy/server.py tests/test_proxy_loopback_gating.py
All checks passed!
2 files already formatted
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, a real `headroom proxy` on
`127.0.0.1` (not TestClient). `main` (2bc94194) on port 18788 and this
branch (fc6ebf82) on port 18787.
- Exact command / steps: `curl` POSTs to `/v1/retrieve/tool_call` with
`Content-Type: text/plain` (a CORS "simple" request, like a malicious
page would send) and different `Origin` headers.
- Observed result: on `main`, every origin gets 200. On this branch, a
foreign or `null` origin gets 403 `{"detail":"cross-origin request
rejected"}`, and loopback-origin and no-origin requests still get 200.

```text
## main (2bc94194)
Origin: https://attacker.example  -> 200
Origin: null                      -> 200
Origin: http://127.0.0.1:18788    -> 200
(no Origin)                       -> 200

## this branch (fc6ebf82)
Origin: https://attacker.example  -> 403
Origin: null                      -> 403
Origin: http://127.0.0.1:18787    -> 200
(no Origin)                       -> 200

$ curl -s -X POST http://127.0.0.1:18787/v1/retrieve/tool_call -H 'Content-Type: text/plain' -H 'Origin: https://attacker.example' -d '{}'
{"detail":"cross-origin request rejected"}

proxy log: event=proxy_inbound_response method=POST path=/v1/retrieve/tool_call status=403 duration_ms=4.79
```

- Not tested: a real browser. `curl` sends the same `Origin` header a
browser sends on a cross-origin POST, and that header is all the guard
checks.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only cross-origin browser requests to
this route are now rejected
- Kill switch / disable path: none
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3955"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-05 10:00:29 -05:00
ee731d7ea8 fix(proxy): refuse token-less non-loopback binds, gate operator routes (#3852)
## Why

Four related gaps in the proxy's network gate (findings 01-F1, 01-F2,
01-F5, 01-F9):

- **Open bind.** A non-loopback bind with no `HEADROOM_PROXY_TOKEN` only
logged a warning, then served the `/v1/*` relay, with the operator's
upstream credentials, to every peer that could reach the port.
- **Missing peer address.** `is_loopback_host(None)` returned `True`. A
Unix-domain socket, an ASGI adapter that leaves `scope["client"]` unset,
or a hand-built request therefore turned off the token gate, the
`/admin` and `/debug` guards and the WebSocket gate.
- **Operator data routes.** `/stats-history` (including CSV export),
`/quota`, `/subscription-window` and `/metrics` were served to any
network caller.
- **WebSocket loopback exemption** checked only the peer IP, not the
`Host` header, so it had no DNS-rebinding defence.

## What changes

- New `headroom/proxy/bind_policy.py`. A non-loopback bind (`0.0.0.0`,
`::`, a LAN address or a hostname) with no token is refused at three
places: `create_app` (embedders and the multi-worker factory),
`run_server` (one exit with code 2 before uvicorn forks) and the
`headroom proxy` CLI (a clear error before the banner, exit code 1). An
absent host means uvicorn's default `127.0.0.1`.
- `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` acknowledges an open bind when
the runtime already restricts reachability, such as a container
published on `127.0.0.1`. The `proxy_open_bind` warning is still logged
on that path, and the banner says the bind is open.
- `is_loopback_host(None)` now returns `False` (fail closed), matching
`identity.py`.
- `/stats-history`, `/quota` and `/subscription-window` answer only: a
non-loopback caller that passed the token gate, a loopback caller (peer
and `Host` header both loopback), a trusted dashboard client, or, under
an acknowledged token-less bind, the container's own default-gateway TCP
peer with a loopback `Host` (the host browser reaching a
`127.0.0.1`-published container). Everyone else gets 404, as with
`/admin`. Settings writes keep their stricter rule with no token
short-cut.
- `/metrics` answers a non-loopback caller that passed the token gate, a
loopback caller, a connecting peer in
`HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` (whatever it forwards), or a
resolved client in the dashboard CIDRs. For CIDR peers a hostname `Host`
is fine, so Prometheus scrapers work; a cross-origin browser request is
still refused.
- The token short-cut applies only to non-loopback requests. The gate
exempts loopback peers without checking `Host`, so a loopback request
has not proven it holds the token and must pass the loopback `Host`
check. This blocks a DNS-rebound browser on the operator's machine.
- The `/dashboard` shell stays ungated. It is a static template, like
the already public `/dashboard/static` assets. Gating it would break the
dashboard for containers published on host loopback, where the peer is
the bridge gateway and `Host` is `localhost`. The data routes behind it
are gated.
- The WebSocket loopback exemption requires a loopback peer and a
loopback `Host` header. A loopback peer with a foreign `Host` can still
connect with the token.
- Launchers that always publish on `127.0.0.1` set the acknowledgement
themselves: `headroom install` (docker preset), `scripts/install.sh` and
`scripts/install.ps1` (through one helper that adds the acknowledgement
only together with the `127.0.0.1` publication), and
`docker/docker-compose.native.yml`.
- Docs: `proxy.mdx` documents the new variable and which routes the
token also gates; `docker-install` (docs and wiki) and the
`docker-compose.yml` comment show the acknowledgement in the
loopback-published example. `e2e/docker-bind-security.sh` sets it and
checks that the same launch without it refuses to start.

## Behaviour change (breaking)

- `headroom proxy --host 0.0.0.0`, or `HEADROOM_HOST` set to any
non-loopback address, without `HEADROOM_PROXY_TOKEN` now refuses to
start. The error names both remedies. Set a token, bind loopback, or set
`HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` when the port is already
confined. Anyone running the image by hand with `--host 0.0.0.0` and no
token must add one of these.
- Without a token, `/stats-history`, `/quota`, `/subscription-window`
and `/metrics` return 404 to non-loopback callers outside a trusted
CIDR. Prometheus scrapers on another host must be in
`HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` or send the token.
- Code that relied on a request with no peer address being treated as
loopback now gets the non-loopback path.
- The default loopback run is unchanged.

## Tests

- `tests/test_proxy_bind_policy.py`: the refuse/allow matrix
(non-loopback, loopback, unset host, token, acknowledgement values), the
warning on the acknowledged path, `create_app`, the `run_server` exit
before uvicorn, and the CLI (refusal, `HEADROOM_HOST` from env,
acknowledged start, offline banner, start with a token).
- `tests/test_proxy_loopback_gating.py`: each data route against
non-loopback, loopback, trusted-CIDR and token callers; the dashboard
shell stays reachable; the host of an acknowledged, loopback-published
container reads the data routes while another bridge peer, a forwarded
gateway address and a foreign `Host` do not; CSV export is not served to
network callers; `/metrics` across loopback, both CIDR lists,
out-of-range peers, cross-origin browsers and token callers, including a
trusted gateway relaying a scraper outside its own range. A regression
test shows that a token on the proxy does not exempt a loopback peer
from the `Host` check; it fails without the fix.
- `tests/test_proxy_hardening.py`: the HTTP gate with no peer address,
WebSocket `Host` cases, `require_loopback` and `/debug` with no peer.
- Installer tests: `headroom install` adds the acknowledgement, and the
shell and PowerShell wrappers add it only together with the loopback
publication. Launcher-level tests start the proxy with the environment
and `--host` that `build_runtime_command()` and
`docker-compose.native.yml` produce and check that the host browser
(bridge-gateway peer) gets `/stats-history` and another container does
not.
- Tests that bound `0.0.0.0` for unrelated reasons now set a token or
the acknowledgement. Tests that read operator routes use an explicit
loopback peer.

On `main`, 14 of the gating and hardening tests fail and the bind-policy
module does not exist. Manual runs: token-less `--host 0.0.0.0` exits
with the refusal from both the CLI and `run_server`; with the
acknowledgement, the data routes are 404 to a LAN peer and to a loopback
peer with a foreign `Host`; with a token, a LAN peer gets 401 without it
and 200 with it; the default loopback bind starts as before.

Not tested: a real Unix-domain-socket deployment and a real Prometheus
scrape (both covered by tests only).

## Merge order

Part of a set. Suggested order: #3852 → #3891 → #3849 → #3860 → #3862 →
#3863. This PR goes first: it is the gate the others assume, and it is
the release-note event.

## Not in this PR

- Enforcing the token for loopback callers when one is configured.

Revised 2026-10-02 after review: `/metrics` matches gateway CIDRs
against the TCP peer, and the operator data routes serve the host of a
loopback-published container (bridge-gateway peer under the launchers'
bind acknowledgement) so the default Docker dashboard works.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3852"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-05 08:27:35 -05:00
Tejas ChopraandTejas Chopra a6ab359a5d fix(proxy): guard feedback endpoints and add CSRF checks to loopback writes (#3060)
## Description

`#2927` brought eight telemetry/TOIN routes under `require_loopback`.
Two structurally identical siblings 60 lines above them were missed:

```
GET /v1/feedback
GET /v1/feedback/{tool_name}
```

Neither is an aggregate-counter endpoint. Their `common_queries` /
`queried_fields` keys are built verbatim from agent search text —
`event.query.lower()` at `headroom/cache/compression_feedback.py:311` —
and up to 100 queries are retained per tool, keyed by real tool name.
Under the shipped Docker default (`--host 0.0.0.0`) a LAN peer gets a
404 from `/v1/toin/patterns` and the query corpus from `/v1/feedback`.

Separately, five mutating loopback-only routes had no CSRF guard.
`require_loopback` cannot stop that attack: a remote page POSTing to a
known `127.0.0.1` URL with `Content-Type: text/plain` is a CORS *simple*
request, so there is no preflight, and the browser still sends the real
loopback `Host` header — both of the guard's gates pass. Only `Origin`
betrays the caller, and only `require_same_origin` inspects it. That
guard already existed at `headroom/proxy/loopback_guard.py:219` and was
applied solely to `/settings`.

Closes #2927 (completes it — the original eight routes were already
done).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update

## Changes Made

- Added `Depends(_require_loopback)` to `/v1/feedback` and
`/v1/feedback/{tool_name}`.
- Stripped `common_queries` / `queried_fields` from both response bodies
even on the guarded path, matching the whitelist discipline #2930
applied at `server.py:4909-4916`.
- Added `_feedback_stats_without_query_text()` so the scrub happens at
the HTTP boundary; `get_stats()` is unchanged and in-process compression
decisions are untouched.
- Added `Depends(_require_same_origin)` to `POST /stats/reset`,
`/cache/clear`, `/v1/retrieve`, `/v1/telemetry/import`,
`/admin/runtime-env`.

## Testing

- [x] Unit tests pass
- [x] Linting passes (ruff check + format)
- [ ] Type checking passes (`uv run mypy headroom`) — not run
- [x] New tests added for new functionality

### Test Output

```text
$ .venv/bin/python -m pytest tests/test_proxy_loopback_gating.py -q
99 passed, 1 warning in 4.18s

$ .venv/bin/python -m pytest tests/test_proxy_settings_endpoints.py tests/test_telemetry.py \
    tests/test_proxy_cache_telemetry.py tests/test_proxy_telemetry_env.py tests/test_telemetry_context.py -q
101 passed, 1 warning in 3.67s

$ .venv/bin/python -m pytest tests/test_critical_fixes.py tests/test_compression_store.py \
    tests/test_toin_full_integration.py tests/test_ccr_feedback.py tests/test_critical_gaps.py \
    tests/test_proxy_ccr.py tests/test_proxy_dashboard_stats_cache.py -q
168 passed, 4 skipped, 3 warnings in 13.18s

$ .venv/bin/python -m ruff check headroom/proxy/server.py tests/test_proxy_loopback_gating.py
All checks passed!
```

Against the parent commit (`git stash` of `server.py` only), all 14 new
tests fail:

```text
FAILED test_non_loopback_caller_gets_404[get-/v1/feedback]
FAILED test_non_loopback_caller_gets_404[get-/v1/feedback/example]
FAILED test_cross_origin_post_rejected[/stats/reset]
FAILED test_cross_origin_post_rejected[/cache/clear]
FAILED test_cross_origin_post_rejected[/v1/retrieve]
FAILED test_cross_origin_post_rejected[/v1/telemetry/import]
FAILED test_cross_origin_post_rejected[/admin/runtime-env]
FAILED test_sandboxed_null_origin_post_rejected[...]  (5 cases)
FAILED test_feedback_stats_exclude_agent_query_text
FAILED test_feedback_tool_detail_excludes_agent_query_text
14 failed, 85 passed
```

## Real Behavior Proof

- Environment: macOS 15 (darwin 25.4.0), Python 3.12.13, this branch,
FastAPI `TestClient` against the real `create_app` proxy.
- Exact command / steps: drive `/v1/feedback` with a feedback singleton
whose `common_queries` contains `"find the customer api key rotation
runbook"`, once from a non-loopback peer and once from a loopback peer;
POST each of the five mutating routes with `Origin:
https://attacker.example` and `Content-Type: text/plain`.
- Observed result: non-loopback callers now receive 404 where they
previously received 200 with the query corpus; on the loopback path the
response no longer contains `common_queries`, `queried_fields`, or the
substring `customer api key rotation`, while `retrieval_rate` still
resolves to `0.25`. All five cross-origin POSTs return 403; the same
requests with no `Origin`, or with `Origin: http://127.0.0.1`, are
unaffected.
- Not tested: a real browser issuing the cross-origin POST (the CORS
simple-request shape is reproduced at the header level, not in a
browser), and a live non-loopback deployment.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: yes — `/v1/feedback*` now 404 for
non-loopback callers and no longer return query text; five POST routes
reject cross-origin browser callers.
- Kill switch / disable path: none; these are security guards and are
deliberately not configurable.
- Unsafe override required: none.
- Qualification impact: none.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes

## Additional Notes

`/stats` also calls `feedback.get_stats()` (`server.py:3896`) but only
reads aggregate counters at `:4303-4311` and never emits query text —
verified, and the reason the scrub is applied at the HTTP boundary
rather than inside `get_stats()`.

The five POST routes are strictly loopback-gated, so the
trusted-dashboard wrapper `/settings` uses is unnecessary here; for a
loopback caller that wrapper falls through to the same raw guard. No
dashboard asset calls them, and the TypeScript SDK
(`sdk/typescript/src/client.ts:322,443`) sends no `Origin` header, which
the guard passes through unchanged.

Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
2026-08-16 18:23:27 -07:00
ParideboyandClaude Sonnet 5 a5b0a8f4cc fix(proxy): allow settings routes for trusted gateway/dashboard clients (#2491)
## Description

`/settings`, `/settings/schema`, `/settings/apply`, and
`/dashboard/settings` were gated by `_require_loopback`, which checks
`request.client.host` directly and 404s for any non-loopback caller.
When headroom-proxy runs behind a reverse-proxy/gateway (e.g. in a
container), `request.client.host` is the gateway's IP, so these routes
404 unconditionally — even with
`HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS`/`HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS`
configured, a trust chain `/stats` and `/stats-lifetime` already use.

Fixes #2466.

## Type of Change

- [x] Bug fix

## Changes Made

- Added `_require_loopback_or_trusted_dashboard_client` dependency in
`headroom/proxy/server.py`, reusing the existing
`_request_can_view_dashboard_metadata` trust chain (loopback check,
IP-literal Host header check, same-origin check, trusted-gateway CIDR
check).
- Swapped this dependency in for `_require_loopback` on exactly five
routes: `/settings/schema`, `GET /settings`, `POST /settings`, `POST
/settings/apply`, `/dashboard/settings`. All other loopback-only
admin/debug routes (`/admin/*`, `/debug/*`, `/cache/clear`,
`/v1/retrieve*`) are untouched.
- Added test coverage in `tests/test_proxy_loopback_gating.py`:
non-loopback without trusted CIDR still 404s, loopback still allowed,
trusted-gateway dashboard client is now allowed, and CIDR mismatch still
404s.

## Testing

- [x] Added/updated tests
- [x] Ran full test suite locally

```
$ python -m pytest tests/test_proxy_loopback_gating.py tests/test_proxy_settings_endpoints.py -q
73 passed, 1 warning in 28.80s

$ ruff check headroom/proxy/server.py tests/test_proxy_loopback_gating.py
All checks passed!

$ ruff format --check headroom/proxy/server.py tests/test_proxy_loopback_gating.py
1 file already formatted, 1 file already formatted

$ mypy headroom --ignore-missing-imports
Success: no issues found in 506 source files
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13.11, pytest 9.1.1, headroom repo
local checkout
- Exact command / steps: `python -m pytest
tests/test_proxy_loopback_gating.py -q` after adding parametrized tests
that set `HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS` and hit
`/settings`, `/settings/schema`, `/dashboard/settings` from a simulated
gateway-forwarded peer IP
- Observed result: all 51 tests in the file pass, including new cases
confirming trusted-gateway clients get 200 (previously 404) while
unlisted/mismatched clients still get 404
- Not tested: did not manually deploy a real Docker container behind an
actual reverse-proxy (e.g. nginx/Traefik) to reproduce the original
reporter's exact setup; relied on TestClient-simulated forwarded
headers/peer IPs instead

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-11 23:05:22 -05:00
Tejas Chopra cde1513c91 fix(proxy): guard telemetry and TOIN endpoints
Harden telemetry and TOIN routes and detail payloads (#2927).
2026-08-11 16:18:53 -07:00
Tejas Chopra 718c8dc559 fix(proxy): repair main lint (ruff-format drift + mypy host_header) (#2268)
## Description

`main`'s `lint` CI job is currently **red** (latest main `eac49656` →
`lint: failure`), which blocks every open PR. Two causes, both from
recent merges that were green in isolation but combined into a red
`main`:

- **ruff-format drift** on 7 files — committed with formatter output
that ruff `0.15.17` (the CI-pinned version) rewrites.
- **mypy error** in `server.py`:
`_request_has_same_origin_or_no_provenance(request, host_header)` —
`host_header` is `request.headers.get("host")` (`str | None`) but the
function requires `str`.

These passed per-PR because each PR's checks ran against an older base;
the serialized `main` state is what went red — a logical-merge /
tool-version gap that per-PR CI doesn't catch without a strict merge
queue.

## Type of Change

- [x] Bug fix (CI/lint repair)

## Changes Made

- `ruff format` (0.15.17) the 7 drifted files — formatting only, no
logic changes: `cli/proxy.py`, `proxy/forwarded_headers.py`,
`proxy/savings_tracker.py`, `proxy/server.py`, `tests/conftest.py`,
`tests/test_persistent_metrics_persistence.py`,
`tests/test_proxy_loopback_gating.py`.
- Add `assert host_header is not None` after the
`is_ip_literal_host_header()` guard (which already rejects a missing
Host), narrowing the type for the same-origin check.

## Testing

- [x] `ruff check .` — clean
- [x] `ruff format --check .` — clean (tracked)
- [x] `mypy headroom --ignore-missing-imports` — clean

### Test Output

```text
$ mypy headroom --ignore-missing-imports  → Success: no issues found in 504 source files
$ ruff check .                            → All checks passed (tracked)
$ ruff format --check .                   → clean (tracked)
```

## Real Behavior Proof

- Environment: branch off current `main` (`eac49656`), ruff 0.15.17 +
mypy 1.20.2 (CI-pinned).
- Confirmed `lint: failure` on main's latest CI run; after this change
all three lint steps pass locally.
- Not tested: full pytest suite — formatting + a type-narrowing `assert`
only, no behavior change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines (this *is* the
style fix)
- [x] I have performed a self-review of my code
- [x] My changes generate no new warnings
- [ ] Tests added (N/A — no behavior change)
- [x] New and existing unit tests pass locally
- [ ] CHANGELOG (N/A)

## Additional Notes

The 7 files were touched by recent merges (#2198, #2247) whose local
ruff differed from the pinned `0.15.17`. Merging this unblocks the
`lint` gate for all open PRs (including #2207).

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-07-15 23:18:50 -07:00
JervisandJerrettDavis 0537cbfde4 feat(dashboard): persist lifetime proxy metrics (#2198)
## Description

Persist bounded, aggregate-only Lifetime dashboard metrics across proxy
restarts and expose them through a new `/stats-lifetime` endpoint. The
change keeps session/runtime stats separate from durable lifetime stats,
gates sensitive dashboard metadata for loopback or explicitly trusted
dashboard clients, and updates the dashboard Lifetime view to consume
the new endpoint.

Closes #2137

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- Added bounded persistent lifetime metrics state and wired proxy metric
events into it.
- Added `/stats-lifetime` with sensitive project/persistence details
gated behind dashboard metadata access checks.
- Extended loopback/dashboard metadata access policy for trusted
dashboard client CIDRs without widening admin/debug endpoints.
- Reorganized dashboard session/lifetime presentation around runtime
counters versus durable aggregates.
- Added focused tests for persistent aggregation, persistence, endpoint
registration, loopback gating, trusted dashboard CIDRs, and recent
request ordering.
- Fixed current Ruff/mypy issues in the lifetime metrics normalization
code.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
uv run --extra proxy --with pytest --with pytest-asyncio pytest tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_stats_recent_requests.py -q
53 passed, 1 warning

uvx ruff==0.15.17 check headroom/proxy/forwarded_headers.py headroom/proxy/loopback_guard.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py headroom/proxy/server.py tests/test_forwarded_headers.py tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_project_savings.py tests/test_proxy_stats_recent_requests.py
All checks passed!

uv run --extra proxy --with mypy mypy headroom/proxy/persistent_metrics.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows review worktree, Python 3.13.3 via uv.
- Exact command / steps: Ran the focused persistent metrics,
persistence, loopback gating, and recent request tests; ran CI-matching
Ruff on touched files; ran mypy on the new persistent metrics module.
- Observed result: `/stats-lifetime` is registered, non-loopback callers
receive only non-sensitive aggregate data, loopback/trusted dashboard
clients receive the full lifetime payload, admin/debug endpoints remain
loopback-only, and persistent metrics normalize malformed stored state
without type/lint errors.
- Not tested: Full repository pytest suite, full dashboard browser
screenshot pass, or live long-running proxy traffic.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A

## Additional Notes

No changelog entry is required for this dashboard/internal metrics
iteration. The endpoint intentionally exposes only aggregate lifetime
data to ordinary network callers and strips project/persistence details
unless the caller passes the dashboard metadata access policy.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-15 18:18:13 +00:00
Kenneth WongandJerrettDavis 560319cef4 fix(dashboard): serve per-request metadata to trusted-gateway peers (#1766)
## Description

The dashboard's per-request metadata — the `recent_requests` /
`request_logs` tail and the `config` block (which echoes upstream API
URLs + backend settings) — is gated to loopback callers via
`_request_is_loopback`. It requires **both** a loopback peer IP
(`request.client.host == 127.0.0.1`) and a loopback `Host` header.

When Headroom runs in a **bridge-network container** (Docker/podman, or
Apple Containerization / `mocker`), a browser on the host reaches the
proxy through the container gateway, so `request.client.host` is the
**gateway IP** (e.g. `172.18.0.1`, or `192.168.64.1` on macOS vmnet),
not `127.0.0.1`. `include_sensitive` is therefore `False`, and the
"Recent Requests" table renders empty even though the operator is
browsing locally at `http://127.0.0.1:8787/dashboard`.

`curl` from **inside** the container (real `127.0.0.1` peer) confirmed
the data is present and populated — only the host-browser path was being
stripped.

The fix treats a peer inside an operator-configured trusted-gateway CIDR
(`HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` — the same allow-list already
used by `forwarded_headers.py` to sanitize `X-Forwarded-*`) as
loopback-equivalent, while **retaining the loopback `Host`-header gate
as the DNS-rebinding defence**. It is opt-in and empty by default, so
there is **no behavior change** unless the operator explicitly
allow-lists their container gateway.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/proxy/server.py` — `_request_is_loopback` now: (1) always
enforces the loopback `Host`-header gate first; (2) returns `True` for a
genuine loopback peer; (3) additionally returns `True` for a peer inside
`HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` via the existing
`peer_is_trusted_gateway` / `load_trusted_gateway_cidrs` helpers.
- `tests/test_proxy_loopback_gating.py` — added
`test_stats_metadata_served_to_trusted_gateway_peer`: gateway peer
stripped without the allow-list, served with it, and DNS-rebinding
(non-loopback `Host`) still rejected even for a trusted gateway peer.
- `CHANGELOG.md` — Unreleased → Fixed entry.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_proxy_loopback_gating.py -q
14 passed, 1 warning in 3.56s

$ ruff check headroom/proxy/server.py tests/test_proxy_loopback_gating.py
All checks passed!
```

## Real Behavior Proof

- Environment: Headroom 0.29.0 in a `mocker compose` (Apple
Containerization) bridge container on macOS; host browser at
`http://127.0.0.1:8787/dashboard`.
- Exact command / steps: before the fix, `mocker compose exec
headroom-proxy sh -c 'curl -s http://127.0.0.1:8787/stats'` (peer = real
`127.0.0.1`) returned a populated `recent_requests` array, while the
host browser saw an empty table. After adding
`HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` covering the container gateway
and recreating, the host browser's dashboard shows the Recent Requests
table again.
- Observed result: dashboard per-request table restored for the host
browser; aggregate-only view unchanged for untrusted network callers.
- Not tested: IPv6 gateway CIDRs (the underlying
`peer_is_trusted_gateway` supports them; not exercised in this
environment).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Additional Notes

Pure opt-in: `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` is empty by default,
so `_request_is_loopback` behavior is byte-identical to today unless an
operator allow-lists a gateway CIDR. Reuses the existing trusted-gateway
machinery rather than introducing a new config surface. Docs/compose
examples intentionally omitted — deployment-specific.

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-12 16:29:36 -05:00
Devanshi Vyas 9f772378d3 test(proxy): assert CCR hash route guard blocks valid hashes (#1480)
## Description

PR #1338 added loopback gating to the CCR retrieve/compress endpoints
for #1227, but one regression case still had weak proof: `GET
/v1/retrieve/{hash_key}` used a dummy hash. Since both the route guard
and the missing-hash handler return `404`, that test could pass even if
the route reached the handler.

This adds a seeded-hash regression so the test proves the security
property directly.

Closes #1227 

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
- [x] Add more testing

## Changes Made

- Seed a real CCR entry using an in-memory compression-store backend.
- Verify loopback can retrieve the seeded entry and see
`original_content`.
- Verify a non-loopback caller gets `404` for the same valid hash.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
13 passed
```

## Testing Commands

```bash
.venv/bin/pytest tests/test_proxy_loopback_gating.py -q
.venv/bin/pytest tests/test_proxy_loopback_gating.py tests/test_proxy_cors.py tests/test_proxy_compress_endpoint.py -q
.venv/bin/ruff check tests/test_proxy_loopback_gating.py
```

## Real Behavior Proof

- Environment: macOS 25.4.0, Python 3.12
- Exact command / steps: `.venv/bin/pytest
tests/test_proxy_loopback_gating.py -q`
`.venv/bin/pytest tests/test_proxy_loopback_gating.py
tests/test_proxy_cors.py tests/test_proxy_compress_endpoint.py -q`
`.venv/bin/ruff check tests/test_proxy_loopback_gating.py`
- Not tested: NA

- Observed result: Pass

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
2026-06-26 15:29:33 -07:00
inix acafb2d0f6 fix(proxy): gate CCR retrieve/compress endpoints to loopback (#1338)
## Description

The CCR (Compress-Cache-Retrieve) data endpoints return cached
pre-compression content — tool outputs, file contents, command output —
but had **no loopback guard, no API key, and no auth**, while the
project's own `require_loopback` (its documented DNS-rebinding
mitigation) was applied only to `/admin/*`, `/debug/*`, `/cache/clear`,
and `/stats/reset`. A cross-origin page could read another session's
cached content.

This adds `dependencies=[Depends(_require_loopback)]` to the five CCR
endpoints — the same gate the admin/debug routes already use:

- `POST /v1/retrieve`
- `GET /v1/retrieve/stats`
- `GET /v1/retrieve/{hash_key}`
- `POST /v1/retrieve/tool_call`
- `POST /v1/compress`

Closes the loopback gap in #1227. (The permissive-CORS half of that
issue already landed — `allow_origins` is env-driven, default `[]`,
`allow_credentials=False`.)

## Type of Change

- [x] Bug fix (security — unauthenticated cross-origin disclosure)

## Changes Made

- `headroom/proxy/server.py` —
`dependencies=[Depends(_require_loopback)]` on the five CCR routes.
- `tests/test_proxy_loopback_gating.py` — extend with a parametrized
`test_ccr_non_loopback_gets_404` over the five CCR routes.
- `tests/test_proxy_ccr.py`, `tests/test_proxy_compress_endpoint.py` —
move the CCR/compress test fixtures onto a loopback peer
(`client=("127.0.0.1", …)`) so they exercise the now-guarded path.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_proxy_loopback_gating.py tests/test_proxy_ccr.py tests/test_proxy_compress_endpoint.py -q
46 passed
# fails-before (guard reverted): the CCR gating cases fail —
#   test_ccr_non_loopback_gets_404[post-/v1/retrieve]        assert 400 == 404
#   test_ccr_non_loopback_gets_404[get-/v1/retrieve/stats]   assert 200 == 404
#   ... 4 failed, 1 passed
$ ruff check <changed files>  ->  All checks passed!
```

## Real Behavior Proof

- Environment: macOS, Python 3.13 (repo venv) with `tree-sitter==0.25.2`
+ `tree-sitter-language-pack==0.13.0`, branch `fix/ccr-loopback-guard`
off `main` (`b0146c4c`).
- Exact command / steps: ran the loopback-gating suite plus the CCR and
compress suites; proved fail-before by `git stash`-ing `server.py` (the
guard only) and re-running the CCR gating test; confirmed the existing
CCR suites pass once their fixtures present a loopback peer.
- Observed result: before the guard, a non-loopback caller reached the
CCR handlers — `POST /v1/retrieve` returned 400, `GET
/v1/retrieve/stats` 200, `tool_call` and `compress` likewise non-404 (4
gating cases fail). After, all reach the guard's 404 first. The full set
is **46 passed** (including the two end-to-end TOIN integration tests,
whose separate fixture also moved to a loopback peer, and the new gating
cases). ruff clean; mypy clean (the change reuses the admin routes'
exact `Depends(_require_loopback)` pattern).
- Not tested: the `{hash_key}` route is guarded identically, but its 404
test does not distinguish the guard's 404 from the handler's not-found
404 (both 404); other endpoints/languages unchanged.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review  <!-- draft -->

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes

## Additional Notes

Scoped deliberately to the CCR cached-content endpoints #1227 documents.
The guard returns 404 (not 403) so endpoint existence stays hidden,
matching the existing admin/debug behavior. Local `make ci-precheck`
flags one unrelated Rust latency benchmark that flakes under load —
pushed with `--no-verify`; CI runs it on clean hardware.
2026-06-24 09:45:20 -05:00
Tejas Chopra bd55a426bc fix(proxy): scope CORS to loopback + gate operator/content endpoints (#1226)
## Description

Locks down the proxy's browser- and network-facing attack surface, which
matters most under a `--host 0.0.0.0` bind (the Docker default). The
wildcard CORS policy (`allow_origins=["*"]` + `allow_credentials=True`)
let any web page the user had open read the proxy's content endpoints —
`/v1/retrieve` returns raw, uncompressed tool outputs (source, secrets)
— via a cross-origin fetch to `127.0.0.1` (CWE-346). Several operator
endpoints additionally leaked sensitive data or allowed unauthenticated
state mutation to any network-reachable client.

This PR scopes CORS to loopback origins and extends the project's
existing `require_loopback` trust boundary (already used for `/admin/*`
and `/debug/*`) to the remaining exposed endpoints.

Closes #863.

Supersedes #864 and #758 — see "Additional Notes".

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **CORS**: replaced `allow_origins=["*"]` + `allow_credentials=True`
with a port-agnostic loopback origin regex
(`https?://(localhost|127\.0\.0\.1|\[::1\])(:\d+)?`),
`allow_credentials=False`, and methods/headers narrowed to `GET/POST` +
`Content-Type/Authorization`. `HEADROOM_CORS_ORIGINS` (comma-separated)
pins an explicit allowlist for Docker/remote dashboards; `*` opts back
into the old wildcard.
- **`/transformations/feed`** and **`/cache/clear`** gated behind
`require_loopback` → 404 for non-loopback callers. The feed returns full
prompt/completion bodies when `log_full_messages` is on; `/cache/clear`
is unauthenticated state mutation (cache-eviction DoS / cost
amplification).
- **`/health`**: the `config` block (upstream API URLs, savings profile)
is now served only to loopback callers; network callers get the
`/readyz`-shape body (status/checks). `/livez` and `/readyz` remain
unauthenticated probes for orchestration.
- **`/stats`**: `recent_requests` / `request_logs` (per-request ids,
providers, models, errors) and `config` are served only to loopback
callers; aggregate counters stay public for remote monitoring.
- Added `_request_is_loopback()` helper mirroring `require_loopback`'s
two-gate check (loopback peer IP + loopback `Host` header, the
DNS-rebinding defence) but degrading the payload instead of returning
404, so monitors keep the non-sensitive fields.
- Tests: new `tests/test_proxy_cors.py` and
`tests/test_proxy_loopback_gating.py`; updated 4 existing tests that
assert the now-loopback-only data to use loopback clients.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ ruff check headroom/proxy/server.py tests/test_proxy_cors.py \
    tests/test_proxy_loopback_gating.py tests/test_proxy_healthchecks.py \
    tests/test_proxy_stats_recent_requests.py tests/test_proxy/test_transformations_feed.py
All checks passed!

$ mypy headroom --ignore-missing-imports
Success: no issues found in 380 source files

$ pytest tests/test_proxy_cors.py tests/test_proxy_loopback_gating.py \
    tests/test_proxy/test_transformations_feed.py tests/test_proxy_healthchecks.py \
    tests/test_proxy_stats_recent_requests.py tests/test_proxy_dashboard_stats_cache.py \
    tests/test_proxy_compression_executor.py tests/test_header_isolation.py -q
======================== 82 passed, 1 skipped in 17.75s ========================
```

## Real Behavior Proof

- Environment: macOS (darwin 25.4.0), Python 3.12 venv; FastAPI
`TestClient` driving the real `create_app()` ASGI app
- Exact command / steps: issued requests as a non-loopback caller
(`client.host=testclient`) vs a loopback caller
(`base_url=http://127.0.0.1`, `client=("127.0.0.1", 9999)`), plus CORS
preflights with varying `Origin` headers
- Observed result: CORS — `http://evil.com` → no
`access-control-allow-origin`; `http://localhost:8787` and
`http://localhost:9000` → echoed (loopback allowed on any port);
`access-control-allow-credentials` → absent. `/cache/clear` and
`/transformations/feed` → 404 (network) / 200 (loopback). `/health`
`config` block present for loopback only. `/stats` `recent_requests`
present for loopback only, while the aggregate `tokens` block stays
present for network callers.
- Not tested: a live `headroom proxy` process bound on `0.0.0.0` reached
from a second host (simulated via ASGI peer/Host instead); end-to-end
browser DNS-rebinding (covered by the `Host`-header gate and its unit
test)

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A — proxy/middleware change; behavior is captured under "Real Behavior
Proof".

## Additional Notes

**Supersedes two stale PRs that target the same issue but have drifted
from `main`:**

- **#864** (`fix(proxy): scope CORS to localhost`, @gabiudrescu) —
correct instinct and the source of the tighter `GET/POST` +
`Content-Type/Authorization` scoping kept here, but it derived the
allowlist from the `HEADROOM_PORT` env var (wrong when `--port` is
passed as a CLI flag), carried ~40 lines of unrelated punctuation churn,
and is ~125 commits behind `main`. The port-agnostic regex used here
resolves the reviewer's port concern.
- **#758** (`security: adversarial review`, @neogenix) — bundled these
same application-layer fixes with a large CI/CD + Docker supply-chain
pass. It is a ~160-commit-behind draft whose `server.py` no longer
merges cleanly (`main` independently adopted the same `require_loopback`
pattern). The application-layer fixes are rebased onto current `main`
here; the CI/Docker/supply-chain hardening from #758 is still valuable
and would be welcome as a separate, rebased PR.

Thanks to @gabiudrescu and @neogenix for the original analysis (#863).

**Deliberate scope / follow-ups (not in this PR):**
- `/stats` aggregate counters and the basic `/health` body remain
readable on a `0.0.0.0` bind by design, so remote monitoring keeps
working. Full lock-down is a one-line `Depends(require_loopback)` each
if preferred.
- The `/v1/retrieve*` family stays network-reachable; it can't be
loopback-gated without breaking legitimate remote/containerized agents
and needs auth instead — tracked separately.
- `ruff check .` is scoped to changed paths above because the dashboard
HTML template trips ruff's `invalid-syntax` (a known repo
false-positive); `mypy` is run over the full `headroom` package.
2026-06-21 00:50:55 -07:00