mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description
Two fixes for proxies run with `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS`
(opt-in, empty by default):
1. **Client address.** The proxy took the *leftmost* `X-Forwarded-For`
element as the client. A gateway that appends to the header keeps
whatever the client itself sent on the left, so the client could pick
its own address (for example `127.0.0.1`). The client is now the
right-most hop that is not itself inside a trusted-gateway CIDR. If
every hop is trusted it falls back to the leftmost, as before.
2. **Settings writes.** A peer inside a trusted-gateway CIDR counted as
loopback in `_request_is_loopback`, so `POST /settings` and `POST
/settings/apply` (which restarts the proxy) were open to anything routed
through the gateway. That rule exists so a containerised dashboard can
show its read-only views. Writes now need a real loopback peer, or a
client inside `HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS` (the
existing #2466 path, with IP-literal Host and same-origin checks).
**Behaviour change:** someone who runs Headroom in a Docker bridge
container, sets only `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS`, and saves
settings from the host browser via `localhost` will now get 404 on save.
Reads still work. To keep saving, add the gateway address to
`HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS` (e.g. `172.17.0.1/32`)
and open the dashboard at `http://127.0.0.1:<port>`. That path is
checked below. Without the env var nothing changes.
Fix adapted from a proposal by HetCreep.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/proxy/forwarded_policy.py`: new `header_client()` walks
`X-Forwarded-For` from the right and skips trusted-gateway hops.
`resolve_forwarded_headers` uses it.
- `headroom/proxy/server.py`: `_request_is_loopback` /
`_request_can_view_dashboard_metadata` take `allow_gateway_peer`
(default unchanged). `POST /settings` and `POST /settings/apply` use a
new write guard with `allow_gateway_peer=False`, and so does the
same-origin guard for those routes.
- Tests: the leftmost-XFF test now expects the right-to-left resolution.
Added tests for a client-written XFF, the all-trusted fallback, settings
reads still served to a gateway peer, writes refused to a gateway peer
(both routes), and writes still allowed on loopback.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
# new tests against origin/main (c21b0813)
$ pytest tests/test_forwarded_headers.py tests/test_proxy_loopback_gating.py
FAILED tests/test_forwarded_headers.py::test_client_written_x_forwarded_for_is_not_the_client
FAILED tests/test_proxy_loopback_gating.py::test_settings_write_refused_for_trusted_gateway_peer[/settings]
FAILED tests/test_proxy_loopback_gating.py::test_settings_write_refused_for_trusted_gateway_peer[/settings/apply]
================== 3 failed, 189 passed, 5 warnings in 46.16s ==================
# this branch
$ pytest tests/test_forwarded_headers.py tests/test_proxy_loopback_gating.py tests/test_proxy_hardening.py tests/test_header_isolation.py tests/*settings* tests/*dashboard*
4 failed, 456 passed, 1 skipped
# the 4 failures are tests/test_dashboard/test_live_feed.py (Playwright, needs a live server
# on :8787: net::ERR_CONNECTION_REFUSED). They fail the same way on origin/main.
$ ruff check / ruff format --check (0.17.3) on changed files: All checks passed!
$ mypy headroom/proxy/forwarded_policy.py headroom/proxy/server.py --ignore-missing-imports
Success: no issues found in 2 source files
```
## Real Behavior Proof
- Environment: macOS, Python 3.12, `create_app(ProxyConfig(...))` driven
by Starlette TestClient with a set peer address.
`HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS=172.17.0.0/16`.
- Exact command / steps: resolve the client for peer `172.17.0.9` with
`X-Forwarded-For: 127.0.0.1, 203.0.113.9`. Then GET/POST `/settings` as
peer `172.17.0.9` with `Host: localhost:8787`, and POST as peer
`127.0.0.1`.
- Observed result:
```text
origin/main this branch
client ip 127.0.0.1 203.0.113.9
gateway peer GET /settings 200 200
gateway peer POST /settings 200 404
loopback POST /settings 200 200
```
Opt-in path: with
`HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS=172.17.0.1/32`, peer
`172.17.0.1`, `Host: 127.0.0.1:8787`, same-origin `Origin`: `POST
/settings` → 200.
- Not tested: a real Docker bridge deployment behind a real reverse
proxy.
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: see Description
- Kill switch / disable path: none needed
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A
## Additional Notes
`_auth_rejection_context` still logs the leftmost hop. That is log-only,
documented as unverified, and not used for any decision, so it is left
as is.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4847"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
277 lines
9.6 KiB
Python
277 lines
9.6 KiB
Python
"""Tests for per-request ``cost_usd`` in ``/stats`` recent_requests (issue #1079).
|
|
|
|
The proxy exposes a per-request billed-cost estimate so tooling (e.g. claude-hud)
|
|
can attribute cost to a Claude Code session by summing over turn_id / time window,
|
|
without an external ``ccusage`` subprocess.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
from types import SimpleNamespace
|
|
|
|
from fastapi.testclient import TestClient
|
|
|
|
from headroom.proxy.cost import CostTracker, request_cost_usd
|
|
from headroom.proxy.models import RequestLog
|
|
from headroom.proxy.outcome import RequestOutcome, emit_request_outcome
|
|
from headroom.proxy.prometheus_metrics import PrometheusMetrics
|
|
from headroom.proxy.request_logger import RequestLogger
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
|
|
|
|
def _tracker(estimate):
|
|
"""A stand-in CostTracker exposing only ``estimate_cost``."""
|
|
return SimpleNamespace(estimate_cost=estimate)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Unit: request_cost_usd helper
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_none_when_cost_tracker_missing() -> None:
|
|
log = {
|
|
"model": "claude-3-5-sonnet-20241022",
|
|
"input_tokens_optimized": 600,
|
|
"output_tokens": 50,
|
|
}
|
|
assert request_cost_usd(None, log) is None
|
|
|
|
|
|
def test_none_when_model_missing() -> None:
|
|
tracker = _tracker(lambda **_: 0.01)
|
|
assert request_cost_usd(tracker, {"input_tokens_optimized": 600, "output_tokens": 50}) is None
|
|
|
|
|
|
def test_none_when_input_tokens_missing() -> None:
|
|
tracker = _tracker(lambda **_: 0.01)
|
|
assert request_cost_usd(tracker, {"model": "claude-3-5-sonnet-20241022"}) is None
|
|
|
|
|
|
def test_passes_through_estimate_and_uses_optimized_tokens() -> None:
|
|
captured: dict = {}
|
|
|
|
def estimate(**kwargs):
|
|
captured.update(kwargs)
|
|
return 0.0135
|
|
|
|
log = {
|
|
"model": "claude-3-5-sonnet-20241022",
|
|
"input_tokens_original": 1000,
|
|
"input_tokens_optimized": 600,
|
|
"output_tokens": 50,
|
|
}
|
|
assert request_cost_usd(_tracker(estimate), log) == 0.0135
|
|
# Bills the post-compression input the provider actually saw, plus output.
|
|
assert captured["input_tokens"] == 600
|
|
assert captured["output_tokens"] == 50
|
|
|
|
|
|
def test_prefers_provider_billed_input_over_local_count() -> None:
|
|
# Local tokenizer counted 600, the provider billed 800: price the 800 the
|
|
# cost tracker records, not the local estimate.
|
|
captured: dict = {}
|
|
|
|
def estimate(**kwargs):
|
|
captured.update(kwargs)
|
|
return 0.02
|
|
|
|
log = {
|
|
"model": "claude-3-5-sonnet-20241022",
|
|
"input_tokens_optimized": 600,
|
|
"provider_input_tokens": 800,
|
|
"output_tokens": 50,
|
|
}
|
|
assert request_cost_usd(_tracker(estimate), log) == 0.02
|
|
assert captured["input_tokens"] == 800
|
|
assert captured["output_tokens"] == 50
|
|
|
|
|
|
def test_falls_back_to_optimized_tokens_when_provider_count_unreported() -> None:
|
|
# 0 means the provider reported no input usage, never "zero tokens billed".
|
|
captured: dict = {}
|
|
|
|
def estimate(**kwargs):
|
|
captured.update(kwargs)
|
|
return 0.01
|
|
|
|
log = {
|
|
"model": "m",
|
|
"input_tokens_optimized": 600,
|
|
"provider_input_tokens": 0,
|
|
"output_tokens": 50,
|
|
}
|
|
assert request_cost_usd(_tracker(estimate), log) == 0.01
|
|
assert captured["input_tokens"] == 600
|
|
|
|
|
|
def test_none_when_output_tokens_unknown() -> None:
|
|
# Pricing a missing output count as zero would publish an input-only figure
|
|
# that looks like a complete request cost.
|
|
calls: list[dict] = []
|
|
|
|
def estimate(**kwargs):
|
|
calls.append(kwargs)
|
|
return 0.002
|
|
|
|
log = {"model": "m", "input_tokens_optimized": 100, "output_tokens": None}
|
|
assert request_cost_usd(_tracker(estimate), log) is None
|
|
assert (
|
|
request_cost_usd(_tracker(estimate), {"model": "m", "input_tokens_optimized": 100}) is None
|
|
)
|
|
assert calls == []
|
|
|
|
|
|
def test_zero_output_tokens_is_a_real_count() -> None:
|
|
captured: dict = {}
|
|
|
|
def estimate(**kwargs):
|
|
captured.update(kwargs)
|
|
return 0.002
|
|
|
|
log = {"model": "m", "input_tokens_optimized": 100, "output_tokens": 0}
|
|
assert request_cost_usd(_tracker(estimate), log) == 0.002
|
|
assert captured["output_tokens"] == 0
|
|
|
|
|
|
def test_none_when_estimate_returns_none() -> None:
|
|
# Unknown model / LiteLLM unavailable → estimate_cost returns None.
|
|
log = {"model": "totally-unknown-model", "input_tokens_optimized": 600, "output_tokens": 50}
|
|
assert request_cost_usd(_tracker(lambda **_: None), log) is None
|
|
|
|
|
|
def test_none_and_isolated_when_estimate_raises() -> None:
|
|
def boom(**_):
|
|
raise RuntimeError("pricing exploded")
|
|
|
|
log = {"model": "m", "input_tokens_optimized": 600, "output_tokens": 50}
|
|
assert request_cost_usd(_tracker(boom), log) is None
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Chain: emit_request_outcome -> RequestLog -> request_cost_usd
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_logged_request_prices_the_input_the_cost_tracker_billed(monkeypatch) -> None:
|
|
cost_tracker = CostTracker()
|
|
billed_inputs: list[int] = []
|
|
|
|
def estimate(**kwargs):
|
|
billed_inputs.append(kwargs["input_tokens"])
|
|
return kwargs["input_tokens"] * 1e-6
|
|
|
|
monkeypatch.setattr(cost_tracker, "estimate_cost", estimate)
|
|
handler = SimpleNamespace(
|
|
metrics=PrometheusMetrics(cost_tracker=cost_tracker, stateless=True),
|
|
cost_tracker=cost_tracker,
|
|
logger=RequestLogger(),
|
|
)
|
|
outcome = RequestOutcome(
|
|
request_id="hr_test_mismatch",
|
|
provider="openai",
|
|
model="gpt-4o-mini",
|
|
original_tokens=1000,
|
|
optimized_tokens=600,
|
|
provider_input_tokens=800,
|
|
output_tokens=50,
|
|
tokens_saved=400,
|
|
attempted_input_tokens=1000,
|
|
)
|
|
|
|
asyncio.run(emit_request_outcome(handler, outcome))
|
|
|
|
(log,) = handler.logger.get_recent(1)
|
|
assert log["input_tokens_optimized"] == 600
|
|
assert log["provider_input_tokens"] == 800
|
|
assert request_cost_usd(cost_tracker, log) == 800 * 1e-6
|
|
# The cost tracker and the per-request field price the same input count.
|
|
assert billed_inputs == [800, 800]
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Integration: /stats exposes cost_usd in recent_requests
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def _sample_log(request_id: str = "hr_test_000001", turn_id: str | None = None) -> RequestLog:
|
|
return RequestLog(
|
|
request_id=request_id,
|
|
timestamp="2026-06-29T00:00:00Z",
|
|
provider="anthropic",
|
|
model="claude-3-5-sonnet-20241022",
|
|
input_tokens_original=1000,
|
|
input_tokens_optimized=600,
|
|
output_tokens=50,
|
|
tokens_saved=400,
|
|
savings_percent=40.0,
|
|
optimization_latency_ms=1.0,
|
|
total_latency_ms=2.0,
|
|
tags={},
|
|
cache_hit=False,
|
|
transforms_applied=[],
|
|
turn_id=turn_id,
|
|
)
|
|
|
|
|
|
def _stats_config() -> ProxyConfig:
|
|
return ProxyConfig(
|
|
optimize=False,
|
|
image_optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=True,
|
|
log_requests=True,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
)
|
|
|
|
|
|
def test_stats_recent_requests_includes_cost_usd(monkeypatch) -> None:
|
|
app = create_app(_stats_config())
|
|
with TestClient(app) as client:
|
|
proxy = client.app.state.proxy
|
|
assert proxy.cost_tracker is not None
|
|
# Deterministic price, independent of the live LiteLLM DB.
|
|
monkeypatch.setattr(proxy.cost_tracker, "estimate_cost", lambda **_: 0.0135)
|
|
# recent_requests is served only to loopback callers; TestClient's peer
|
|
# host is "testclient", so force the loopback branch (not under test).
|
|
monkeypatch.setattr("headroom.proxy.server._request_is_loopback", lambda request, **_: True)
|
|
proxy.logger._logs.append(_sample_log())
|
|
|
|
resp = client.get("/stats")
|
|
assert resp.status_code == 200
|
|
recent = resp.json()["recent_requests"]
|
|
assert recent, "expected at least one recent request"
|
|
entry = next(e for e in recent if e["request_id"] == "hr_test_000001")
|
|
assert entry["cost_usd"] == 0.0135
|
|
|
|
|
|
def test_stats_recent_requests_carry_turn_id_beyond_raw_log_tail(monkeypatch) -> None:
|
|
# recent_requests serves up to 25 priced rows while request_logs keeps only
|
|
# the newest 10, so every priced row must carry its own turn_id for a
|
|
# consumer to group cost by turn.
|
|
app = create_app(_stats_config())
|
|
with TestClient(app) as client:
|
|
proxy = client.app.state.proxy
|
|
monkeypatch.setattr(proxy.cost_tracker, "estimate_cost", lambda **_: 0.001)
|
|
monkeypatch.setattr("headroom.proxy.server._request_is_loopback", lambda request, **_: True)
|
|
for i in range(20):
|
|
proxy.logger._logs.append(_sample_log(f"hr_test_{i:06d}", turn_id=f"turn_{i // 4}"))
|
|
|
|
payload = client.get("/stats").json()
|
|
assert len(payload["request_logs"]) == 10
|
|
recent = [e for e in payload["recent_requests"] if e["request_id"].startswith("hr_test_")]
|
|
assert len(recent) == 20
|
|
|
|
cost_by_turn: dict[str, float] = {}
|
|
for entry in recent:
|
|
assert entry["cost_usd"] == 0.001
|
|
cost_by_turn[entry["turn_id"]] = (
|
|
cost_by_turn.get(entry["turn_id"], 0.0) + entry["cost_usd"]
|
|
)
|
|
assert sorted(cost_by_turn) == [f"turn_{t}" for t in range(5)]
|
|
assert all(abs(total - 0.004) < 1e-12 for total in cost_by_turn.values())
|