feat(proxy): show per-request /stats metadata to token-authenticated callers (#4168)

## Description

`/stats` and `/stats-lifetime` withhold the per-request tail
(`recent_requests`, `request_logs`) and project names from every
non-loopback caller, unless the request's `Host` is an IP literal
**and** the peer is inside
`HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS`.

That rule cannot be met by a common deployment: the proxy runs on a
container platform with `HEADROOM_PROXY_TOKEN` set, and the dashboard
sits on its own host behind an authenticating reverse proxy (here Entra
ID sign-in in front of a small Caddy that adds `x-headroom-proxy-token`
after the user signed in). That reverse proxy reaches Headroom with a
domain `Host` (the platform routes internal calls by Host, so it cannot
be an IP) and from a peer address that is not stable enough to list. The
dashboard loads, the aggregate counters work, and the "Recent Requests"
table stays empty with no indication why.

This PR also admits callers whose proxy token the security gate has
already verified (`request.state.proxy_authenticated`, which the gate
sets only after the constant-time token comparison).

**This is a nice-to-have, not a bug fix.** The current behaviour is
documented and deliberate, and the existing trusted-CIDR path keeps
working unchanged. If you would rather not widen who sees this metadata,
closing the PR is fine; we carry the change on our fork either way.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/server.py`: add `_request_can_view_stats_metadata`:
true when the gate verified the proxy token, otherwise the existing
`_request_can_view_dashboard_metadata` result. `/stats` and
`/stats-lifetime` use it. The helper is deliberately not used by
`/settings*` or `/transformations/feed`, which keep their loopback /
trusted-client guards.
- `tests/test_proxy_stats_recent_requests.py`: add a regression test
(domain `Host`, non-loopback peer, token sent) for `/stats` and
`/stats?cached=1`, asserting the tail is served with the token and that
the gate still answers 401 without it or with a wrong one.
- `docs/content/docs/security-model.mdx`: new section **What the proxy
token unlocks**. It lists exactly which fields a token holder now also
sees: `recent_requests`, `request_logs` and `config` on `/stats` (and
`?cached=1`), and `projects` and `persistence.error` on
`/stats-lifetime`. It says the settings routes and the
`require_loopback` routes stay closed to a token. It also warns that the
token is one shared secret, not a per-caller identity: every holder sees
every caller's request metadata, so a deployment that hands one token to
several clients that must not see each other's metadata should run one
proxy per client. The dashboard-CIDR table row now points to the
section.
- `docs/content/docs/proxy.mdx` (the `HEADROOM_PROXY_TOKEN` row) and
`SECURITY.md` (the local-control-surfaces bullet): one line each
pointing to the same consequence.
- `headroom/proxy/server.py` docstrings: the `/stats` handler and
`_request_can_view_stats_metadata` now describe the actual access rule.
The helper no longer claims the change widens nothing.

Who gains access: holders of `HEADROOM_PROXY_TOKEN`. This does widen
what the token grants. Before, it granted inference and the read-only
operator routes. Now it also grants the request metadata of every caller
on the proxy. With no token configured, nothing changes (the flag is
never set).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom/proxy/server.py`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

At `1c6666d` merged with current `main` (`4abbf07`):

```text
$ python -m pytest -q tests/test_proxy_stats_recent_requests.py tests/test_proxy_loopback_gating.py \
    tests/test_dashboard_accessibility.py tests/test_proxy_hardening.py
214 passed, 1 skipped in 44.60s

$ python -m mypy headroom/proxy/server.py
Success: no issues found in 1 source file

$ ruff check headroom/proxy/server.py tests/test_proxy_stats_recent_requests.py
All checks passed!
$ ruff format --check headroom/proxy/server.py tests/test_proxy_stats_recent_requests.py
2 files already formatted
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13, this branch at `4abbf07`
(`1c6666d` merged with current `main`). A real `headroom proxy` with a
throwaway `HEADROOM_PROXY_TOKEN`, bound to a non-loopback LAN address,
with `HEADROOM_OFFLINE=1` and `--anthropic-api-url` pointed at a local
stub that answers `/v1/messages` with a fixed Anthropic message:
  ```text
headroom proxy --host 192.168.0.48 --port 18788 --no-optimize
--anthropic-api-url http://127.0.0.1:18999
  ```
- Exact command / steps: a client on `192.168.0.48` (not loopback, in no
trusted CIDR) sends what the dashboard's reverse proxy sends. That is
`Host: headroom-dashboard.example.test` (a domain, as the platform
routes by Host) plus `x-headroom-proxy-token`. It first makes one
token-authenticated `POST /v1/messages` so there is a request to list,
then reads the stats endpoints with the token, without it, and with a
wrong one. It also tries the settings and loopback-only routes with the
token. Same script against the same build with only
`headroom/proxy/server.py` swapped back to `main`'s, for the before
column.
- Observed result: verified token holders receive shared stats metadata,
while missing or incorrect tokens are rejected; exact before/after
responses follow.

  ```text
request (domain Host, non-loopback peer) before (main) after (this PR)
POST /v1/messages token 200 200
GET /stats token 200, recent_requests/request_logs/ 200,
recent_requests=1 request_logs=1
config absent config present, first={"provider":
"anthropic","model":"claude-sonnet-5-5"}
GET /stats?cached=1 token 200, same fields absent 200, same as /stats
GET /stats-lifetime token 200, projects absent 200, projects present
GET /stats no token 401 401
GET /stats wrong token 401 401
GET /stats-lifetime no token 401 401
GET /settings token 404
GET /settings/schema token 404
GET /dashboard/settings token 404
GET /transformations/feed token 404
  ```

The proxy log shows `event=proxy_auth_rejected path=/stats …
reason=missing_token`, `… reason=bad_token` and `path=/stats-lifetime …
reason=missing_token` for the three rejections.
- Not tested: the full suite (only the four files above). The deployed
dashboard end to end: the run above reproduces its request shape against
a real proxy, without the Entra/Caddy hop in front.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only when `HEADROOM_PROXY_TOKEN` is
set. Token-authenticated remote callers now receive `recent_requests`,
`request_logs` and `config` from `/stats`, and `projects` and
`persistence.error` from `/stats-lifetime`, for every caller of the
proxy. Loopback and trusted-CIDR behaviour is unchanged, and the
settings and loopback-only routes are not opened to the token.
- Kill switch / disable path: unset the token (then nothing changes), or
revert the commit
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Documentation: done; see the security-model section above.
- A possible alternative that avoids widening the rule: an opt-in env
var (e.g. `HEADROOM_STATS_METADATA_FOR_TOKEN=1`). Happy to switch to
that if you prefer the default to stay strict.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_013MpfLG8sVzqPhD34teqdLD

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4168"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer current-main requalification

Exact head `09689b73ddeb8879a2f2c68726f9e885cd9bfdee`, upstream main
`e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`.

Mechanically merged upstream/main e4a60ed7 and re-reviewed
stats-metadata authorization plus the existing security/runtime
discussion. Only callers whose proxy token authentication succeeded gain
the additional per-request stats metadata; settings and loopback-only
routes retain their separate guards. Documentation explicitly states
that token holders share every caller's traffic metadata. No contributor
production logic was rewritten.

All 260 focused stats/recent-request, loopback-gating and hardening
tests pass; Ruff lint/format and whitespace checks pass. Existing
before/after real HTTP evidence remains explicitly scoped; local tests
use the existing native extension, not a fresh native build. Applied the
shared test-only Gemini streamed-request fixture repair for
independently established main failures; all ten cases pass. Formal
approval awaits exact-head hosted qualification.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `fa07187e666ca7895d1cc5708a9d930f38399fe1`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Clean mechanical merge with upstream main
c01cf07d94. Reviewed current author
discussion and Devin runtime/documentation findings. Additional stats
metadata is allowed only after successful proxy-token authentication or
the existing dashboard authorization; settings and loopback-only routes
retain separate guards. The security documentation explicitly states
that one shared token exposes every caller's traffic metadata to every
holder. No contributor authorization policy rewritten.

Fresh focused stats/recent-request, loopback-gating and hardening
verification: 260 tests passed with six existing deprecation/OpenAPI
warnings. Changed-file Ruff lint/format and main-relative whitespace
checks pass; owned tree clean. Existing non-loopback before/after HTTP
evidence remains scoped; no fresh live replay or branch-native build is
claimed. Python tests use the current-main native DLL. No new source
blocker found; approval awaits exact-head hosted checks. Nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `20b62cc35f7a005b91a6c9054c41cfd51ac29c37`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.

Clean main integration against 0ddaa56c21.
315 stats/recent-request, loopback/security gating and incoming-main
tests passed, 11 optional skips, 6 warnings. Exact incoming-main
patches, lint/format, offline lock and whitespace checks pass. Complete
recorded native input-set/hash and artifact checksum verified before
existing DLL reuse; no fresh native build claimed.

Current author discussion and both resolved Devin threads/replies
reviewed. Verified token-authenticated state expands only stats metadata
authorization; settings and require_loopback guards remain separate.
Documentation explicitly lists fields and warns that the shared token
exposes every caller's request metadata to every holder, with separate
proxies required for isolation. This is an intentional documented
shared-trust policy, not a per-user identity. Prior author's
non-loopback real-proxy before/after evidence is retained with its
scope; this refresh adds local tests, not a fresh live replay or
browser/reverse-proxy proof. No new source blocker found.

Source/test/prose delta has no
workflow/setup/dependency/CLI/install/provider/OpenClaw/SDK path
changes. Inherited flagged workflow-validation and Init/Wrap bootstrap
paths are excluded. Docs build uses unchanged-main
workflow/package/lock/config and this PR changes only prose there.
Exact-head Actions and rollup remain pending before approval. Nothing
merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Serge ARADJ
2026-11-07 20:42:27 -06:00
committed by GitHub
co-authored by Claude JD Davis
parent 861ef95620
commit 9b2dec56f9
5 changed files with 106 additions and 6 deletions
+3 -1
View File
@@ -130,7 +130,9 @@ host, and what the local admin surface does and does not allow. The summary:
loopback client address *and* a loopback `Host:` header, plus a same-origin check on
the mutating ones. This is a local-trust model: any process on the same host is
trusted. Setting `HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS` deliberately widens
the settings and stats surface beyond loopback.
the settings and stats surface beyond loopback. Setting `HEADROOM_PROXY_TOKEN` also
shows every token holder the per-request `/stats` metadata of all callers, but opens
none of these routes.
Thank you for helping keep Headroom and its users safe!
+1 -1
View File
@@ -441,7 +441,7 @@ with a privileged job.
| Flag / env | Default | Effect |
|---|---|---|
| `HEADROOM_PROXY_TOKEN` | none | Require a token from non-loopback callers, as `X-Headroom-Proxy-Token: <token>` or `Authorization: Bearer <token>`. The token is removed from the request before it is forwarded upstream; clients that also send a provider credential should use `X-Headroom-Proxy-Token` and keep `Authorization` for the provider. Also gates the read-only operator routes (`/stats-history`, `/quota`, `/subscription-window`, `/metrics`), which otherwise answer loopback and trusted-CIDR callers only. |
| `HEADROOM_PROXY_TOKEN` | none | Require a token from non-loopback callers, as `X-Headroom-Proxy-Token: <token>` or `Authorization: Bearer <token>`. The token is removed from the request before it is forwarded upstream; clients that also send a provider credential should use `X-Headroom-Proxy-Token` and keep `Authorization` for the provider. Also gates the read-only operator routes (`/stats-history`, `/quota`, `/subscription-window`, `/metrics`), which otherwise answer loopback and trusted-CIDR callers only. A token holder also sees the per-request metadata `/stats` and `/stats-lifetime` withhold from other network callers. That metadata covers every caller of the proxy, so only share the token between clients that may see each other's traffic metadata; see [What the proxy token unlocks](/docs/security-model#what-the-proxy-token-unlocks). |
| `HEADROOM_ALLOW_UNAUTHENTICATED_BIND` | `false` | Acknowledge a non-loopback bind (`--host 0.0.0.0`, `::`, a LAN address) with **no** `HEADROOM_PROXY_TOKEN`. Without a token or this flag the proxy refuses to start, because `/v1/*` would relay upstream with the operator's credentials for every peer that can reach the port. Only set it when the runtime already restricts reachability (a container published on `127.0.0.1`). |
| `HEADROOM_COMPRESS_ALLOW_REMOTE` | `false` | Allow non-loopback callers to reach [`POST /v1/compress`](#post-v1compress) and `POST /v1/usage`. Required to run Headroom as a remote gateway/sidecar; without it remote callers get `404`. |
| `HEADROOM_OFFLINE` | `false` | Air-gap mode: refuse every connection Headroom itself initiates — telemetry, update checks, license reporting, model/tokenizer/binary/dataset downloads, litellm's pricing-table refresh, CrewAI's install tracking, remote Kompress, OTLP and Langfuse export, Copilot auth, the subscription pollers, the OpenAI embedders, Headroom Cloud compression and the TLS diagnostics (`headroom doctor --network` reports them as skipped). Four things still work on purpose: forwarding your own requests to the upstream you configured, operator-configured local endpoints (the Ollama embedder), loopback health probes, and paths an `is_offline()` check already disables. Refusals surface as messages, not tracebacks. See the air-gap notes in `docs/metrics-technical-guide.md`. |
+28 -1
View File
@@ -267,7 +267,7 @@ the boundary and should say so in their own threat model.
| Variable | Effect |
|---|---|
| `HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS` | Peers in these CIDRs reach `/settings`, `/settings/schema`, `/dashboard/settings`, and the settings *writes* **without being on loopback** — and additionally see the fields `/stats` and `/stats-lifetime` otherwise withhold from network callers (per-request ids, providers, models, errors, project names, backend config). Added so the settings UI works behind a reverse proxy. |
| `HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS` | Peers in these CIDRs reach `/settings`, `/settings/schema`, `/dashboard/settings`, and the settings *writes* **without being on loopback** — and additionally see the fields `/stats` and `/stats-lifetime` otherwise withhold from network callers (per-request ids, providers, models, errors, project names, backend config). Added so the settings UI works behind a reverse proxy. A caller holding `HEADROOM_PROXY_TOKEN` sees those `/stats` fields too, but none of the settings routes; see [What the proxy token unlocks](#what-the-proxy-token-unlocks). |
| `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` | Peers in these CIDRs are trusted to supply `X-Forwarded-*` headers, which is what makes the client IP the *forwarded* one rather than the gateway's. |
Both parse strictly: a malformed CIDR raises at startup rather than silently emptying the
@@ -275,6 +275,33 @@ allowlist. Note the interaction — trusting a gateway for `X-Forwarded-For` mea
dashboard CIDR check is applied to a client-supplied header, so the two must be configured
together and pointed at infrastructure you control.
### What the proxy token unlocks
With `HEADROOM_PROXY_TOKEN` set, a non-loopback caller that presents the token
(`X-Headroom-Proxy-Token` or `Authorization: Bearer`) can use the provider routes and the
read-only operator routes (`/stats-history`, `/quota`, `/subscription-window`, `/metrics`).
It also sees the per-request metadata that `/stats` and `/stats-lifetime` withhold from other
network callers, the same fields a dashboard CIDR peer sees:
| Endpoint | Fields a token holder also sees |
|---|---|
| `/stats` (and `/stats?cached=1`) | `recent_requests` and `request_logs`: request ids, timestamps, providers, models, token counts, transforms applied, tags, error text. Never prompt or response content. Plus `config`: backend and savings profile. |
| `/stats-lifetime` | `projects` (per-project savings) and `persistence.error` |
That is what a dashboard behind an authenticating reverse proxy needs. The reverse proxy
adds the token after its own sign-in, then reaches `/stats` with a domain `Host:` from a
peer outside any dashboard CIDR, which the CIDR rule never admits.
The token opens nothing else in this section. `/settings`, `/settings/schema`,
`/dashboard/settings` and the settings writes still need loopback or a dashboard CIDR peer.
The `require_loopback` routes (`/admin/*`, `/debug/*`, `/cache/clear`, `/stats/reset`,
`/transformations/feed`, `/v1/retrieve*` and the rest listed above) still need loopback.
The token is one shared secret, not a per-caller identity. Every holder sees the request
metadata of every caller on that proxy: which models they use, their error text, their
project names. If you hand the token to several clients that should not see each other's
traffic metadata, run one proxy per client.
### What `POST /admin/runtime-env` can change
`headroom wrap` hot-syncs settings to a running proxy through this route. It does not
+28 -3
View File
@@ -2955,6 +2955,29 @@ def _request_can_view_dashboard_metadata(
)
def _request_can_view_stats_metadata(
request: Request,
trusted_dashboard_client_cidrs: tuple[ipaddress.IPv4Network | ipaddress.IPv6Network, ...],
) -> bool:
"""Authorize per-request ``/stats`` metadata (ids, providers, models, errors).
Everything :func:`_request_can_view_dashboard_metadata` allows, plus callers
that proved they hold ``HEADROOM_PROXY_TOKEN``: the security gate sets
``request.state.proxy_authenticated`` only after a constant-time token match.
That is what a dashboard fronted by an authenticating reverse proxy sends
(the proxy adds the token after its own sign-in). Such a caller reaches
``/stats`` with a domain ``Host`` and a peer outside any trusted CIDR, so
the Host/CIDR rule above can never admit it, and the dashboard's
"Recent Requests" table stays empty. This widens what the token grants:
every holder sees the request metadata of every caller on this proxy
(docs/content/docs/security-model.mdx, "What the proxy token unlocks").
The settings routes and the loopback-only routes keep their own guards.
"""
if getattr(request.state, "proxy_authenticated", False) is True:
return True
return _request_can_view_dashboard_metadata(request, trusted_dashboard_client_cidrs)
def _request_has_same_origin_or_no_provenance(request: Request, host_header: str) -> bool:
"""Accept no browser provenance, otherwise require same-origin headers."""
@@ -5736,10 +5759,12 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
``recent_requests`` / ``request_logs`` (per-request ids, providers,
models, errors) and ``config`` (backend + savings profile) are embedded
only for loopback callers — the local dashboard. Network callers still
only for callers :func:`_request_can_view_stats_metadata` admits:
loopback (the local dashboard), a trusted dashboard CIDR peer, or a
caller that passed the proxy token check. Other network callers still
get the aggregate counters but never the per-request metadata.
"""
include_sensitive = _request_can_view_dashboard_metadata(
include_sensitive = _request_can_view_stats_metadata(
request,
trusted_dashboard_client_cidrs,
)
@@ -5763,7 +5788,7 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
async def stats_lifetime(request: Request):
"""Return persisted lifetime aggregates with sensitive fields gated."""
payload = dict(proxy.metrics.savings_tracker.lifetime_response())
include_sensitive = _request_can_view_dashboard_metadata(
include_sensitive = _request_can_view_stats_metadata(
request,
trusted_dashboard_client_cidrs,
)
+46
View File
@@ -324,3 +324,49 @@ def test_read_maturation_reports_whether_it_can_fire(
assert (payload["inert_reason"] is None) is effective
banner = [r for r in caplog.records if r.getMessage().startswith("Read maturation:")]
assert [r.levelname for r in banner] == [startup_level]
@pytest.mark.parametrize("path", ["/stats?cached=1", "/stats"])
def test_recent_requests_served_to_token_holder_behind_auth_proxy(
monkeypatch: pytest.MonkeyPatch, path: str
) -> None:
"""A dashboard fronted by an authenticating reverse proxy reaches /stats with
a domain Host from a non-loopback peer, carrying HEADROOM_PROXY_TOKEN. The
Host/CIDR rule can never admit it; the verified token must."""
monkeypatch.setenv("HEADROOM_PROXY_TOKEN", "s3cret-token")
monkeypatch.setenv("HEADROOM_HOST", "0.0.0.0")
app = create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
ccr_inject_tool=False,
ccr_handle_responses=False,
ccr_context_tracking=False,
http2=False,
)
)
logger = FakeRequestLogger()
app.state.proxy.logger = logger
logger.logs = [
FakeLogEntry(
{
"request_id": "req-1",
"timestamp": "2026-10-12T10:00:00Z",
"provider": "anthropic",
"model": "claude-sonnet",
}
)
]
token = {"x-headroom-proxy-token": "s3cret-token"}
with TestClient(app, base_url="http://headroom", client=("10.0.4.7", 5000)) as client:
payload = client.get(path, headers=token).json()
assert [r["request_id"] for r in payload["recent_requests"]] == ["req-1"]
assert payload["request_logs"]
# The gate still refuses callers without (or with a wrong) token.
assert client.get(path).status_code == 401
assert client.get(path, headers={"x-headroom-proxy-token": "nope"}).status_code == 401