Files
headroom/tests/test_proxy_project_savings.py
ee731d7ea8 fix(proxy): refuse token-less non-loopback binds, gate operator routes (#3852)
## Why

Four related gaps in the proxy's network gate (findings 01-F1, 01-F2,
01-F5, 01-F9):

- **Open bind.** A non-loopback bind with no `HEADROOM_PROXY_TOKEN` only
logged a warning, then served the `/v1/*` relay, with the operator's
upstream credentials, to every peer that could reach the port.
- **Missing peer address.** `is_loopback_host(None)` returned `True`. A
Unix-domain socket, an ASGI adapter that leaves `scope["client"]` unset,
or a hand-built request therefore turned off the token gate, the
`/admin` and `/debug` guards and the WebSocket gate.
- **Operator data routes.** `/stats-history` (including CSV export),
`/quota`, `/subscription-window` and `/metrics` were served to any
network caller.
- **WebSocket loopback exemption** checked only the peer IP, not the
`Host` header, so it had no DNS-rebinding defence.

## What changes

- New `headroom/proxy/bind_policy.py`. A non-loopback bind (`0.0.0.0`,
`::`, a LAN address or a hostname) with no token is refused at three
places: `create_app` (embedders and the multi-worker factory),
`run_server` (one exit with code 2 before uvicorn forks) and the
`headroom proxy` CLI (a clear error before the banner, exit code 1). An
absent host means uvicorn's default `127.0.0.1`.
- `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` acknowledges an open bind when
the runtime already restricts reachability, such as a container
published on `127.0.0.1`. The `proxy_open_bind` warning is still logged
on that path, and the banner says the bind is open.
- `is_loopback_host(None)` now returns `False` (fail closed), matching
`identity.py`.
- `/stats-history`, `/quota` and `/subscription-window` answer only: a
non-loopback caller that passed the token gate, a loopback caller (peer
and `Host` header both loopback), a trusted dashboard client, or, under
an acknowledged token-less bind, the container's own default-gateway TCP
peer with a loopback `Host` (the host browser reaching a
`127.0.0.1`-published container). Everyone else gets 404, as with
`/admin`. Settings writes keep their stricter rule with no token
short-cut.
- `/metrics` answers a non-loopback caller that passed the token gate, a
loopback caller, a connecting peer in
`HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` (whatever it forwards), or a
resolved client in the dashboard CIDRs. For CIDR peers a hostname `Host`
is fine, so Prometheus scrapers work; a cross-origin browser request is
still refused.
- The token short-cut applies only to non-loopback requests. The gate
exempts loopback peers without checking `Host`, so a loopback request
has not proven it holds the token and must pass the loopback `Host`
check. This blocks a DNS-rebound browser on the operator's machine.
- The `/dashboard` shell stays ungated. It is a static template, like
the already public `/dashboard/static` assets. Gating it would break the
dashboard for containers published on host loopback, where the peer is
the bridge gateway and `Host` is `localhost`. The data routes behind it
are gated.
- The WebSocket loopback exemption requires a loopback peer and a
loopback `Host` header. A loopback peer with a foreign `Host` can still
connect with the token.
- Launchers that always publish on `127.0.0.1` set the acknowledgement
themselves: `headroom install` (docker preset), `scripts/install.sh` and
`scripts/install.ps1` (through one helper that adds the acknowledgement
only together with the `127.0.0.1` publication), and
`docker/docker-compose.native.yml`.
- Docs: `proxy.mdx` documents the new variable and which routes the
token also gates; `docker-install` (docs and wiki) and the
`docker-compose.yml` comment show the acknowledgement in the
loopback-published example. `e2e/docker-bind-security.sh` sets it and
checks that the same launch without it refuses to start.

## Behaviour change (breaking)

- `headroom proxy --host 0.0.0.0`, or `HEADROOM_HOST` set to any
non-loopback address, without `HEADROOM_PROXY_TOKEN` now refuses to
start. The error names both remedies. Set a token, bind loopback, or set
`HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` when the port is already
confined. Anyone running the image by hand with `--host 0.0.0.0` and no
token must add one of these.
- Without a token, `/stats-history`, `/quota`, `/subscription-window`
and `/metrics` return 404 to non-loopback callers outside a trusted
CIDR. Prometheus scrapers on another host must be in
`HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` or send the token.
- Code that relied on a request with no peer address being treated as
loopback now gets the non-loopback path.
- The default loopback run is unchanged.

## Tests

- `tests/test_proxy_bind_policy.py`: the refuse/allow matrix
(non-loopback, loopback, unset host, token, acknowledgement values), the
warning on the acknowledged path, `create_app`, the `run_server` exit
before uvicorn, and the CLI (refusal, `HEADROOM_HOST` from env,
acknowledged start, offline banner, start with a token).
- `tests/test_proxy_loopback_gating.py`: each data route against
non-loopback, loopback, trusted-CIDR and token callers; the dashboard
shell stays reachable; the host of an acknowledged, loopback-published
container reads the data routes while another bridge peer, a forwarded
gateway address and a foreign `Host` do not; CSV export is not served to
network callers; `/metrics` across loopback, both CIDR lists,
out-of-range peers, cross-origin browsers and token callers, including a
trusted gateway relaying a scraper outside its own range. A regression
test shows that a token on the proxy does not exempt a loopback peer
from the `Host` check; it fails without the fix.
- `tests/test_proxy_hardening.py`: the HTTP gate with no peer address,
WebSocket `Host` cases, `require_loopback` and `/debug` with no peer.
- Installer tests: `headroom install` adds the acknowledgement, and the
shell and PowerShell wrappers add it only together with the loopback
publication. Launcher-level tests start the proxy with the environment
and `--host` that `build_runtime_command()` and
`docker-compose.native.yml` produce and check that the host browser
(bridge-gateway peer) gets `/stats-history` and another container does
not.
- Tests that bound `0.0.0.0` for unrelated reasons now set a token or
the acknowledgement. Tests that read operator routes use an explicit
loopback peer.

On `main`, 14 of the gating and hardening tests fail and the bind-policy
module does not exist. Manual runs: token-less `--host 0.0.0.0` exits
with the refusal from both the CLI and `run_server`; with the
acknowledgement, the data routes are 404 to a LAN peer and to a loopback
peer with a foreign `Host`; with a token, a LAN peer gets 401 without it
and 200 with it; the default loopback bind starts as before.

Not tested: a real Unix-domain-socket deployment and a real Prometheus
scrape (both covered by tests only).

## Merge order

Part of a set. Suggested order: #3852 → #3891 → #3849 → #3860 → #3862 →
#3863. This PR goes first: it is the gate the others assume, and it is
the release-note event.

## Not in this PR

- Enforcing the token for loopback callers when one is configured.

Revised 2026-10-02 after review: `/metrics` matches gateway CIDRs
against the TCP peer, and the operator data routes serve the host of a
loopback-published container (bridge-gateway peer under the launchers'
bind acknowledgement) so the default Docker dashboard works.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3852"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-05 08:27:35 -05:00

367 lines
14 KiB
Python

"""Tests for per-project savings attribution (X-Headroom-Project)."""
import asyncio
import json
import pytest
pytest.importorskip("fastapi")
from fastapi.testclient import TestClient # noqa: E402
from headroom.proxy.outcome import RequestOutcome, emit_request_outcome # noqa: E402
from headroom.proxy.project_context import ( # noqa: E402
classify_project,
get_current_project,
set_current_project,
split_project_path,
with_project_prefix,
)
from headroom.proxy.savings_tracker import ( # noqa: E402
DEFAULT_MAX_PROJECTS,
SavingsTracker,
sanitize_project_name,
)
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
# ---------------------------------------------------------------------------
# sanitize_project_name / classify_project
# ---------------------------------------------------------------------------
def test_sanitize_project_name_normalizes_and_caps():
assert sanitize_project_name(" api-server ") == "api-server"
assert sanitize_project_name("a" * 300) == "a" * 128
assert sanitize_project_name("x\x00\x1by") == "xy"
assert sanitize_project_name("") is None
assert sanitize_project_name(" ") is None
assert sanitize_project_name(None) is None
assert sanitize_project_name(42) is None
def test_sanitize_project_name_decodes_percent_encoded_non_ascii():
"""Percent-encoded non-ASCII cwd names (issue #1069) must decode to Unicode."""
import urllib.parse
chinese = "第二大脑共享"
encoded = urllib.parse.quote(chinese, safe="-_.() ")
assert sanitize_project_name(encoded) == chinese
mixed = "test-中文-项目"
encoded_mixed = urllib.parse.quote(mixed, safe="-_.() ")
assert sanitize_project_name(encoded_mixed) == mixed
# Plain ASCII names must still pass through unchanged.
assert sanitize_project_name("my-project") == "my-project"
def test_classify_project_reads_header_case_insensitively():
assert classify_project({"x-headroom-project": "frontend"}) == "frontend"
assert classify_project({"X-Headroom-Project": " frontend "}) == "frontend"
assert classify_project({"user-agent": "claude-code/1.0"}) is None
assert classify_project(object()) is None
def test_split_project_path_extracts_and_strips():
assert split_project_path("/p/frontend/v1/messages") == ("frontend", "/v1/messages")
assert split_project_path("/p/my%20repo/v1/chat/completions") == (
"my repo",
"/v1/chat/completions",
)
assert split_project_path("/p/frontend") == ("frontend", "/")
# No prefix / unusable name: path passes through untouched.
assert split_project_path("/v1/messages") == (None, "/v1/messages")
assert split_project_path("/p//v1/messages") == (None, "/p//v1/messages")
assert split_project_path("/p/%20%20/v1") == (None, "/p/%20%20/v1")
def test_with_project_prefix_round_trips_through_split():
url = with_project_prefix("http://127.0.0.1:8787/v1", "my repo")
assert url == "http://127.0.0.1:8787/p/my%20repo/v1"
path = url.removeprefix("http://127.0.0.1:8787")
assert split_project_path(path) == ("my repo", "/v1")
# Bare host (anthropic-style base) and unusable names.
assert with_project_prefix("http://127.0.0.1:8787", "api") == "http://127.0.0.1:8787/p/api"
assert with_project_prefix("http://127.0.0.1:8787/v1", " ") == "http://127.0.0.1:8787/v1"
assert with_project_prefix("http://127.0.0.1:8787/v1", None) == "http://127.0.0.1:8787/v1"
def test_project_contextvar_roundtrip():
set_current_project(" demo ")
assert get_current_project() == "demo"
set_current_project(None)
assert get_current_project() is None
# ---------------------------------------------------------------------------
# SavingsTracker per-project aggregation
# ---------------------------------------------------------------------------
def test_tracker_accumulates_per_project_and_persists(tmp_path):
path = tmp_path / "savings.json"
tracker = SavingsTracker(path=str(path))
tracker.record_request(model="gpt-4o", input_tokens=1000, tokens_saved=400, project="api")
tracker.record_request(model="gpt-4o", input_tokens=500, tokens_saved=100, project="api")
tracker.record_request(model="gpt-4o", input_tokens=200, tokens_saved=50, project="web")
tracker.record_request(model="gpt-4o", input_tokens=99, tokens_saved=9) # unattributed
projects = tracker.stats_preview()["projects"]
assert list(projects) == ["api", "web"] # sorted by tokens saved desc
assert projects["api"]["requests"] == 2
assert projects["api"]["tokens_saved"] == 500
assert projects["api"]["total_input_tokens"] == 1500
assert projects["api"]["savings_percent"] == pytest.approx(25.0)
assert projects["web"]["requests"] == 1
assert projects["api"]["last_activity_at"] is not None
# Unattributed traffic still lands in the lifetime totals.
assert tracker.stats_preview()["lifetime"]["requests"] == 4
# Survives a restart via the persisted JSON state.
reloaded = SavingsTracker(path=str(path))
assert reloaded.stats_preview()["projects"]["api"]["tokens_saved"] == 500
assert reloaded.lifetime_response()["projects"]["api"]["tokens_saved"] == 500
def test_tracker_migrates_v2_state_without_projects(tmp_path):
path = tmp_path / "savings.json"
path.write_text(
json.dumps(
{
"schema_version": 2,
"lifetime": {
"requests": 3,
"tokens_saved": 77,
"compression_savings_usd": 0.1,
"total_input_tokens": 500,
"total_input_cost_usd": 0.2,
},
"display_session": None,
"history": [],
}
)
)
tracker = SavingsTracker(path=str(path))
preview = tracker.stats_preview()
assert preview["projects"] == {}
assert preview["lifetime"]["tokens_saved"] == 77
def test_tracker_caps_project_cardinality(tmp_path):
tracker = SavingsTracker(path=str(tmp_path / "savings.json"))
for i in range(DEFAULT_MAX_PROJECTS + 5):
tracker.record_request(
model="gpt-4o",
input_tokens=10,
tokens_saved=i + 1,
project=f"proj-{i:03d}",
)
projects = tracker.stats_preview()["projects"]
assert len(projects) == DEFAULT_MAX_PROJECTS
# The smallest buckets were evicted; the biggest savers survive.
assert "proj-000" not in projects
assert f"proj-{DEFAULT_MAX_PROJECTS + 4:03d}" in projects
def test_tracker_sanitizes_persisted_project_state(tmp_path):
path = tmp_path / "savings.json"
path.write_text(
json.dumps(
{
"schema_version": 3,
"lifetime": {},
"display_session": None,
"history": [],
"projects": {
"ok": {"requests": "2", "tokens_saved": 10},
"": {"requests": 1},
"bad-entry": "not-a-dict",
},
}
)
)
projects = SavingsTracker(path=str(path)).stats_preview()["projects"]
assert set(projects) == {"ok"}
assert projects["ok"]["requests"] == 2
assert projects["ok"]["tokens_saved"] == 10
assert projects["ok"]["compression_savings_usd"] == 0.0
def test_tracker_caps_persisted_projects_on_load(tmp_path):
path = tmp_path / "savings.json"
oversized = {
f"proj-{i:03d}": {"requests": 1, "tokens_saved": i}
for i in range(DEFAULT_MAX_PROJECTS + 10)
}
path.write_text(
json.dumps(
{
"schema_version": 3,
"lifetime": {},
"display_session": None,
"history": [],
"projects": oversized,
}
)
)
projects = SavingsTracker(path=str(path)).stats_preview()["projects"]
assert len(projects) == DEFAULT_MAX_PROJECTS
# Lowest tokens_saved entries are dropped, highest kept.
assert "proj-000" not in projects
assert f"proj-{DEFAULT_MAX_PROJECTS + 9:03d}" in projects
# ---------------------------------------------------------------------------
# End-to-end: outcome funnel -> tracker -> /stats payload
# ---------------------------------------------------------------------------
def _emit_outcome(proxy, *, project_field=None):
outcome = RequestOutcome(
request_id="req-1",
provider="openai",
model="gpt-4o",
original_tokens=1000,
optimized_tokens=600,
output_tokens=20,
tokens_saved=400,
attempted_input_tokens=1000,
project=project_field,
)
asyncio.run(emit_request_outcome(proxy, outcome))
def test_funnel_attributes_savings_from_context_and_stats_exposes_them(tmp_path, monkeypatch):
monkeypatch.setenv("HEADROOM_SAVINGS_PATH", str(tmp_path / "savings.json"))
config = ProxyConfig(cache_enabled=False, rate_limit_enabled=False, log_requests=False)
with TestClient(
create_app(config), base_url="http://127.0.0.1", client=("127.0.0.1", 12345)
) as client:
proxy = client.app.state.proxy
set_current_project("ctx-project")
try:
_emit_outcome(proxy)
finally:
set_current_project(None)
# Explicit outcome.project wins over the bound context.
_emit_outcome(proxy, project_field="field-project")
stats = client.get("/stats").json()
per_project = stats["savings"]["per_project"]
assert per_project["ctx-project"]["tokens_saved"] == 400
assert per_project["field-project"]["tokens_saved"] == 400
assert stats["persistent_savings"]["projects"] == per_project
assert stats["persistent_savings"]["projects_limit"] == DEFAULT_MAX_PROJECTS
history = client.get("/stats-history").json()
assert history["schema_version"] == 6
assert history["projects"]["ctx-project"]["requests"] == 1
# ---------------------------------------------------------------------------
# Regression: pre-feature behavior must be unchanged
# ---------------------------------------------------------------------------
def test_record_request_without_project_matches_legacy_totals(tmp_path):
"""No-header traffic produces exactly the pre-v3 aggregates."""
path = tmp_path / "savings.json"
tracker = SavingsTracker(path=str(path))
tracker.record_request(model="gpt-4o", input_tokens=100, tokens_saved=40)
tracker.record_request(model="gpt-4o", input_tokens=200, tokens_saved=60)
preview = tracker.stats_preview()
assert preview["projects"] == {}
assert preview["lifetime"]["requests"] == 2
assert preview["lifetime"]["tokens_saved"] == 100
assert preview["display_session"]["tokens_saved"] == 100
persisted = json.loads(path.read_text())
# Every legacy top-level key survives alongside the new projects map.
assert set(persisted) >= {"schema_version", "lifetime", "display_session", "history"}
assert persisted["projects"] == {}
def test_stats_payload_keeps_legacy_shape(tmp_path, monkeypatch):
"""Dashboard consumers of the old /stats keys must not break."""
monkeypatch.setenv("HEADROOM_SAVINGS_PATH", str(tmp_path / "savings.json"))
config = ProxyConfig(cache_enabled=False, rate_limit_enabled=False, log_requests=False)
with TestClient(
create_app(config), base_url="http://127.0.0.1", client=("127.0.0.1", 12345)
) as client:
proxy = client.app.state.proxy
_emit_outcome(proxy) # unattributed: no header, no context, no field
stats = client.get("/stats").json()
assert stats["savings"]["per_project"] == {}
for legacy_key in ("requests", "savings", "persistent_savings", "cost"):
assert legacy_key in stats, f"legacy /stats key {legacy_key!r} disappeared"
assert stats["persistent_savings"]["lifetime"]["requests"] == 1
history = client.get("/stats-history").json()
for legacy_key in ("schema_version", "lifetime", "display_session", "retention"):
assert legacy_key in history, f"legacy /stats-history key {legacy_key!r} disappeared"
def test_metrics_record_request_works_without_project_kwarg(tmp_path, monkeypatch):
"""Existing callers that never pass ``project=`` keep working."""
monkeypatch.setenv("HEADROOM_SAVINGS_PATH", str(tmp_path / "savings.json"))
config = ProxyConfig(cache_enabled=False, rate_limit_enabled=False, log_requests=False)
with TestClient(
create_app(config), base_url="http://127.0.0.1", client=("127.0.0.1", 12345)
) as client:
proxy = client.app.state.proxy
asyncio.run(
proxy.metrics.record_request(
provider="openai",
model="gpt-4o",
input_tokens=120,
output_tokens=24,
tokens_saved=30,
latency_ms=15.0,
)
)
preview = proxy.metrics.savings_tracker.stats_preview()
assert preview["lifetime"]["tokens_saved"] == 30
assert preview["projects"] == {}
def test_middleware_binds_project_header_to_context(tmp_path, monkeypatch):
monkeypatch.setenv("HEADROOM_SAVINGS_PATH", str(tmp_path / "savings.json"))
config = ProxyConfig(cache_enabled=False, rate_limit_enabled=False, log_requests=False)
captured: list[str | None] = []
import headroom.proxy.server as server_module
def _capture(project: str | None) -> None:
captured.append(project)
set_current_project(project)
monkeypatch.setattr(server_module, "set_current_project", _capture)
with TestClient(
create_app(config), base_url="http://127.0.0.1", client=("127.0.0.1", 12345)
) as client:
assert client.get("/health", headers={"X-Headroom-Project": " my repo "}).status_code == 200
assert client.get("/health").status_code == 200
# /p/<name> base-URL prefix (aider/copilot/cursor wraps): stripped
# before routing, so the request still reaches /health.
assert client.get("/p/my%20repo/health").status_code == 200
# An explicit header wins over the path prefix.
assert (
client.get(
"/p/prefix-project/health", headers={"X-Headroom-Project": "header-project"}
).status_code
== 200
)
assert captured == ["my repo", None, "my repo", "header-project"]