Files
headroom/tests/test_proxy_stats_recent_requests.py
9b2dec56f9 feat(proxy): show per-request /stats metadata to token-authenticated callers (#4168)
## Description

`/stats` and `/stats-lifetime` withhold the per-request tail
(`recent_requests`, `request_logs`) and project names from every
non-loopback caller, unless the request's `Host` is an IP literal
**and** the peer is inside
`HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS`.

That rule cannot be met by a common deployment: the proxy runs on a
container platform with `HEADROOM_PROXY_TOKEN` set, and the dashboard
sits on its own host behind an authenticating reverse proxy (here Entra
ID sign-in in front of a small Caddy that adds `x-headroom-proxy-token`
after the user signed in). That reverse proxy reaches Headroom with a
domain `Host` (the platform routes internal calls by Host, so it cannot
be an IP) and from a peer address that is not stable enough to list. The
dashboard loads, the aggregate counters work, and the "Recent Requests"
table stays empty with no indication why.

This PR also admits callers whose proxy token the security gate has
already verified (`request.state.proxy_authenticated`, which the gate
sets only after the constant-time token comparison).

**This is a nice-to-have, not a bug fix.** The current behaviour is
documented and deliberate, and the existing trusted-CIDR path keeps
working unchanged. If you would rather not widen who sees this metadata,
closing the PR is fine; we carry the change on our fork either way.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/server.py`: add `_request_can_view_stats_metadata`:
true when the gate verified the proxy token, otherwise the existing
`_request_can_view_dashboard_metadata` result. `/stats` and
`/stats-lifetime` use it. The helper is deliberately not used by
`/settings*` or `/transformations/feed`, which keep their loopback /
trusted-client guards.
- `tests/test_proxy_stats_recent_requests.py`: add a regression test
(domain `Host`, non-loopback peer, token sent) for `/stats` and
`/stats?cached=1`, asserting the tail is served with the token and that
the gate still answers 401 without it or with a wrong one.
- `docs/content/docs/security-model.mdx`: new section **What the proxy
token unlocks**. It lists exactly which fields a token holder now also
sees: `recent_requests`, `request_logs` and `config` on `/stats` (and
`?cached=1`), and `projects` and `persistence.error` on
`/stats-lifetime`. It says the settings routes and the
`require_loopback` routes stay closed to a token. It also warns that the
token is one shared secret, not a per-caller identity: every holder sees
every caller's request metadata, so a deployment that hands one token to
several clients that must not see each other's metadata should run one
proxy per client. The dashboard-CIDR table row now points to the
section.
- `docs/content/docs/proxy.mdx` (the `HEADROOM_PROXY_TOKEN` row) and
`SECURITY.md` (the local-control-surfaces bullet): one line each
pointing to the same consequence.
- `headroom/proxy/server.py` docstrings: the `/stats` handler and
`_request_can_view_stats_metadata` now describe the actual access rule.
The helper no longer claims the change widens nothing.

Who gains access: holders of `HEADROOM_PROXY_TOKEN`. This does widen
what the token grants. Before, it granted inference and the read-only
operator routes. Now it also grants the request metadata of every caller
on the proxy. With no token configured, nothing changes (the flag is
never set).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom/proxy/server.py`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

At `1c6666d` merged with current `main` (`4abbf07`):

```text
$ python -m pytest -q tests/test_proxy_stats_recent_requests.py tests/test_proxy_loopback_gating.py \
    tests/test_dashboard_accessibility.py tests/test_proxy_hardening.py
214 passed, 1 skipped in 44.60s

$ python -m mypy headroom/proxy/server.py
Success: no issues found in 1 source file

$ ruff check headroom/proxy/server.py tests/test_proxy_stats_recent_requests.py
All checks passed!
$ ruff format --check headroom/proxy/server.py tests/test_proxy_stats_recent_requests.py
2 files already formatted
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13, this branch at `4abbf07`
(`1c6666d` merged with current `main`). A real `headroom proxy` with a
throwaway `HEADROOM_PROXY_TOKEN`, bound to a non-loopback LAN address,
with `HEADROOM_OFFLINE=1` and `--anthropic-api-url` pointed at a local
stub that answers `/v1/messages` with a fixed Anthropic message:
  ```text
headroom proxy --host 192.168.0.48 --port 18788 --no-optimize
--anthropic-api-url http://127.0.0.1:18999
  ```
- Exact command / steps: a client on `192.168.0.48` (not loopback, in no
trusted CIDR) sends what the dashboard's reverse proxy sends. That is
`Host: headroom-dashboard.example.test` (a domain, as the platform
routes by Host) plus `x-headroom-proxy-token`. It first makes one
token-authenticated `POST /v1/messages` so there is a request to list,
then reads the stats endpoints with the token, without it, and with a
wrong one. It also tries the settings and loopback-only routes with the
token. Same script against the same build with only
`headroom/proxy/server.py` swapped back to `main`'s, for the before
column.
- Observed result: verified token holders receive shared stats metadata,
while missing or incorrect tokens are rejected; exact before/after
responses follow.

  ```text
request (domain Host, non-loopback peer) before (main) after (this PR)
POST /v1/messages token 200 200
GET /stats token 200, recent_requests/request_logs/ 200,
recent_requests=1 request_logs=1
config absent config present, first={"provider":
"anthropic","model":"claude-sonnet-5-5"}
GET /stats?cached=1 token 200, same fields absent 200, same as /stats
GET /stats-lifetime token 200, projects absent 200, projects present
GET /stats no token 401 401
GET /stats wrong token 401 401
GET /stats-lifetime no token 401 401
GET /settings token 404
GET /settings/schema token 404
GET /dashboard/settings token 404
GET /transformations/feed token 404
  ```

The proxy log shows `event=proxy_auth_rejected path=/stats …
reason=missing_token`, `… reason=bad_token` and `path=/stats-lifetime …
reason=missing_token` for the three rejections.
- Not tested: the full suite (only the four files above). The deployed
dashboard end to end: the run above reproduces its request shape against
a real proxy, without the Entra/Caddy hop in front.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only when `HEADROOM_PROXY_TOKEN` is
set. Token-authenticated remote callers now receive `recent_requests`,
`request_logs` and `config` from `/stats`, and `projects` and
`persistence.error` from `/stats-lifetime`, for every caller of the
proxy. Loopback and trusted-CIDR behaviour is unchanged, and the
settings and loopback-only routes are not opened to the token.
- Kill switch / disable path: unset the token (then nothing changes), or
revert the commit
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Documentation: done; see the security-model section above.
- A possible alternative that avoids widening the rule: an opt-in env
var (e.g. `HEADROOM_STATS_METADATA_FOR_TOKEN=1`). Happy to switch to
that if you prefer the default to stay strict.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_013MpfLG8sVzqPhD34teqdLD

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4168"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer current-main requalification

Exact head `09689b73ddeb8879a2f2c68726f9e885cd9bfdee`, upstream main
`e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`.

Mechanically merged upstream/main e4a60ed7 and re-reviewed
stats-metadata authorization plus the existing security/runtime
discussion. Only callers whose proxy token authentication succeeded gain
the additional per-request stats metadata; settings and loopback-only
routes retain their separate guards. Documentation explicitly states
that token holders share every caller's traffic metadata. No contributor
production logic was rewritten.

All 260 focused stats/recent-request, loopback-gating and hardening
tests pass; Ruff lint/format and whitespace checks pass. Existing
before/after real HTTP evidence remains explicitly scoped; local tests
use the existing native extension, not a fresh native build. Applied the
shared test-only Gemini streamed-request fixture repair for
independently established main failures; all ten cases pass. Formal
approval awaits exact-head hosted qualification.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `fa07187e666ca7895d1cc5708a9d930f38399fe1`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Clean mechanical merge with upstream main
c01cf07d94. Reviewed current author
discussion and Devin runtime/documentation findings. Additional stats
metadata is allowed only after successful proxy-token authentication or
the existing dashboard authorization; settings and loopback-only routes
retain separate guards. The security documentation explicitly states
that one shared token exposes every caller's traffic metadata to every
holder. No contributor authorization policy rewritten.

Fresh focused stats/recent-request, loopback-gating and hardening
verification: 260 tests passed with six existing deprecation/OpenAPI
warnings. Changed-file Ruff lint/format and main-relative whitespace
checks pass; owned tree clean. Existing non-loopback before/after HTTP
evidence remains scoped; no fresh live replay or branch-native build is
claimed. Python tests use the current-main native DLL. No new source
blocker found; approval awaits exact-head hosted checks. Nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `20b62cc35f7a005b91a6c9054c41cfd51ac29c37`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.

Clean main integration against 0ddaa56c21.
315 stats/recent-request, loopback/security gating and incoming-main
tests passed, 11 optional skips, 6 warnings. Exact incoming-main
patches, lint/format, offline lock and whitespace checks pass. Complete
recorded native input-set/hash and artifact checksum verified before
existing DLL reuse; no fresh native build claimed.

Current author discussion and both resolved Devin threads/replies
reviewed. Verified token-authenticated state expands only stats metadata
authorization; settings and require_loopback guards remain separate.
Documentation explicitly lists fields and warns that the shared token
exposes every caller's request metadata to every holder, with separate
proxies required for isolation. This is an intentional documented
shared-trust policy, not a per-user identity. Prior author's
non-loopback real-proxy before/after evidence is retained with its
scope; this refresh adds local tests, not a fresh live replay or
browser/reverse-proxy proof. No new source blocker found.

Source/test/prose delta has no
workflow/setup/dependency/CLI/install/provider/OpenClaw/SDK path
changes. Inherited flagged workflow-validation and Init/Wrap bootstrap
paths are excluded. Docs build uses unchanged-main
workflow/package/lock/config and this PR changes only prose there.
Exact-head Actions and rollup remain pending before approval. Nothing
merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-07 20:42:27 -06:00

373 lines
12 KiB
Python

from __future__ import annotations
import pytest
from fastapi.testclient import TestClient
from headroom.proxy.models import ProxyConfig
from headroom.proxy.server import create_app
from headroom.rollout import resolve_rollout
class FakeRequestLogger:
def __init__(self) -> None:
self._logs: list[dict[str, object]] = []
@property
def logs(self) -> list[dict[str, object]]:
return self._logs
@logs.setter
def logs(self, value: list[dict[str, object]]) -> None:
self._logs = value
def get_recent(self, limit: int) -> list[dict[str, object]]:
return self._logs[-limit:]
class FakeLogEntry(dict[str, object]):
def __getattr__(self, name: str) -> object:
return self.get(name)
def test_stats_exposes_actual_running_rollout_snapshot() -> None:
rollout = resolve_rollout(
{
"HEADROOM_ROLLOUT_CHANNEL": "canary",
"HEADROOM_FEATURES": "tool_result_interceptors",
}
)
app = create_app(
ProxyConfig(
rollout=rollout,
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
ccr_inject_tool=False,
ccr_handle_responses=False,
ccr_context_tracking=False,
http2=False,
)
)
with TestClient(app, base_url="http://127.0.0.1", client=("127.0.0.1", 12345)) as client:
payload = client.get("/stats").json()["rollout"]
assert payload == rollout.to_dict()
assert payload["qualification_eligible"] is True
def test_stats_refreshes_recent_requests_when_cached() -> None:
app = create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
ccr_inject_tool=False,
ccr_handle_responses=False,
ccr_context_tracking=False,
http2=False,
)
)
logger = FakeRequestLogger()
app.state.proxy.logger = logger
first_log = FakeLogEntry(
{
"timestamp": "2026-06-11T10:00:00Z",
"provider": "openai",
"model": "gpt-4.1",
"input_tokens_original": 100,
"input_tokens_optimized": 60,
"tokens_saved": 40,
"savings_percent": 40.0,
"savings_breakdown": [
{
"source": "tool_search",
"tokens": 40,
"usd": 0.0,
"realized": True,
}
],
}
)
second_log = FakeLogEntry(
{
"timestamp": "2026-06-11T10:01:00Z",
"provider": "anthropic",
"model": "claude-sonnet",
"input_tokens_original": 200,
"input_tokens_optimized": 120,
"tokens_saved": 80,
"savings_percent": 40.0,
}
)
# Loopback client/Host: recent_requests is served only to loopback callers.
with TestClient(app, base_url="http://127.0.0.1", client=("127.0.0.1", 12345)) as client:
logger.logs = [first_log]
first_response = client.get("/stats?cached=1")
assert first_response.status_code == 200
assert first_response.json()["recent_requests"][-1]["model"] == "gpt-4.1"
assert first_response.json()["recent_requests"][-1]["savings_breakdown"] == [
{
"source": "tool_search",
"tokens": 40,
"usd": 0.0,
"realized": True,
}
]
logger.logs = [first_log, second_log]
second_response = client.get("/stats?cached=1")
assert second_response.status_code == 200
second_payload = second_response.json()
assert second_payload["recent_requests"][0]["model"] == "claude-sonnet"
assert second_payload["request_logs"][-1]["model"] == "claude-sonnet"
def test_stats_recent_requests_includes_token_incomplete_requests() -> None:
app = create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
ccr_inject_tool=False,
ccr_handle_responses=False,
ccr_context_tracking=False,
http2=False,
)
)
logger = FakeRequestLogger()
app.state.proxy.logger = logger
logger.logs = [
FakeLogEntry(
{
"request_id": "req-haiku-1",
"timestamp": "2026-07-09T10:00:00Z",
"provider": "anthropic",
"model": "claude-haiku",
"transforms_applied": [],
}
),
FakeLogEntry(
{
"request_id": "req-haiku-2",
"timestamp": "2026-07-09T10:01:00Z",
"provider": "anthropic",
"model": "claude-haiku",
"input_tokens_original": None,
"input_tokens_optimized": None,
"output_tokens": None,
"tokens_saved": 0,
"savings_percent": 0.0,
"transforms_applied": [],
}
),
FakeLogEntry(
{
"request_id": "req-sonnet-1",
"timestamp": "2026-07-09T10:02:00Z",
"provider": "anthropic",
"model": "claude-sonnet",
"input_tokens_original": 200,
"input_tokens_optimized": 120,
"output_tokens": 40,
"tokens_saved": 80,
"savings_percent": 40.0,
"transforms_applied": ["smart_crusher"],
}
),
]
with TestClient(app, base_url="http://127.0.0.1", client=("127.0.0.1", 12345)) as client:
response = client.get("/stats")
assert response.status_code == 200
payload = response.json()
assert [req["model"] for req in payload["recent_requests"]] == [
"claude-sonnet",
"claude-haiku",
"claude-haiku",
]
assert payload["recent_requests"][0]["token_accounting_status"] == "complete"
assert payload["recent_requests"][0]["has_exact_tokens"] is True
assert payload["recent_requests"][1]["output_tokens"] is None
assert payload["recent_requests"][1]["token_accounting_status"] == "partial"
assert payload["recent_requests"][1]["tokens_saved"] == 0
assert payload["recent_requests"][2]["input_tokens_optimized"] is None
assert payload["recent_requests"][2]["token_accounting_status"] == "missing"
assert payload["recent_requests"][2]["has_exact_tokens"] is False
assert payload["summary"]["uncompressed_requests"]["unknown_token_accounting"] == 2
assert payload["request_logs"][-1]["model"] == "claude-sonnet"
def test_agent_usage_totals_use_proxy_only_savings(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("HEADROOM_REQUIRE_RUST_CORE", "false")
app = create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
ccr_inject_tool=False,
ccr_handle_responses=False,
ccr_context_tracking=False,
http2=False,
)
)
logger = FakeRequestLogger()
app.state.proxy.logger = logger
logger.logs = [
FakeLogEntry(
{
"timestamp": "2026-06-11T10:00:00Z",
"provider": "openai",
"model": "gpt-5.2-codex",
"tags": {"client": "codex"},
"input_tokens_original": 1000,
"input_tokens_optimized": 900,
"output_tokens": 50,
"tokens_saved": 100,
"savings_percent": 10.0,
}
)
]
with TestClient(app) as client:
proxy = client.app.state.proxy
proxy.metrics.tokens_input_total = 900
proxy.metrics.tokens_saved_total = 100
proxy.metrics.tokens_output_total = 50
response = client.get("/stats")
assert response.status_code == 200
payload = response.json()
assert payload["tokens"]["saved"] == 100
assert payload["agent_usage"]["totals"]["before_tokens"] == 1000
assert payload["agent_usage"]["totals"]["tokens_saved"] == 100
assert payload["agent_usage"]["totals"]["savings_percent"] == 10.0
assert payload["agent_usage"]["agents"][0]["share_of_saved_percent"] == 100.0
def test_stats_preserves_default_smart_crusher_compaction_state() -> None:
config = ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
)
# Loopback client/Host: the `config` block is served only to loopback callers.
client = TestClient(
create_app(config), base_url="http://127.0.0.1", client=("127.0.0.1", 12345)
)
response = client.get("/stats")
assert response.status_code == 200
assert response.json()["config"]["smart_crusher_with_compaction"] is None
def _read_maturation_app(mode: str, quiesce_turns: int = 5):
return create_app(
ProxyConfig(
mode=mode,
read_maturation=True,
read_maturation_quiesce_turns=quiesce_turns,
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
ccr_inject_tool=False,
ccr_handle_responses=False,
ccr_context_tracking=False,
http2=False,
)
)
@pytest.mark.parametrize(
("mode", "quiesce_turns", "effective", "startup_level"),
[
("cache", 5, False, "WARNING"),
("token", 5, True, "INFO"),
# A zero threshold matures a Read in the still-mutable latest turn.
("cache", 0, True, "INFO"),
],
)
def test_read_maturation_reports_whether_it_can_fire(
mode, quiesce_turns, effective, startup_level, caplog
):
"""Cache mode freezes a Read before it can quiesce, so enabled maturation
never fires there; startup and /stats must say so instead of reporting a
silently inert feature as on."""
app = _read_maturation_app(mode, quiesce_turns)
with caplog.at_level("INFO", logger="headroom.proxy"):
with TestClient(app, base_url="http://127.0.0.1", client=("127.0.0.1", 12345)) as client:
payload = client.get("/stats").json()["read_maturation"]
assert payload["enabled"] is True
assert payload["effective"] is effective
assert (payload["inert_reason"] is None) is effective
banner = [r for r in caplog.records if r.getMessage().startswith("Read maturation:")]
assert [r.levelname for r in banner] == [startup_level]
@pytest.mark.parametrize("path", ["/stats?cached=1", "/stats"])
def test_recent_requests_served_to_token_holder_behind_auth_proxy(
monkeypatch: pytest.MonkeyPatch, path: str
) -> None:
"""A dashboard fronted by an authenticating reverse proxy reaches /stats with
a domain Host from a non-loopback peer, carrying HEADROOM_PROXY_TOKEN. The
Host/CIDR rule can never admit it; the verified token must."""
monkeypatch.setenv("HEADROOM_PROXY_TOKEN", "s3cret-token")
monkeypatch.setenv("HEADROOM_HOST", "0.0.0.0")
app = create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
ccr_inject_tool=False,
ccr_handle_responses=False,
ccr_context_tracking=False,
http2=False,
)
)
logger = FakeRequestLogger()
app.state.proxy.logger = logger
logger.logs = [
FakeLogEntry(
{
"request_id": "req-1",
"timestamp": "2026-10-12T10:00:00Z",
"provider": "anthropic",
"model": "claude-sonnet",
}
)
]
token = {"x-headroom-proxy-token": "s3cret-token"}
with TestClient(app, base_url="http://headroom", client=("10.0.4.7", 5000)) as client:
payload = client.get(path, headers=token).json()
assert [r["request_id"] for r in payload["recent_requests"]] == ["req-1"]
assert payload["request_logs"]
# The gate still refuses callers without (or with a wrong) token.
assert client.get(path).status_code == 401
assert client.get(path, headers={"x-headroom-proxy-token": "nope"}).status_code == 401