mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
feat/https-compose-caddy
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
ee731d7ea8 |
fix(proxy): refuse token-less non-loopback binds, gate operator routes (#3852)
## Why Four related gaps in the proxy's network gate (findings 01-F1, 01-F2, 01-F5, 01-F9): - **Open bind.** A non-loopback bind with no `HEADROOM_PROXY_TOKEN` only logged a warning, then served the `/v1/*` relay, with the operator's upstream credentials, to every peer that could reach the port. - **Missing peer address.** `is_loopback_host(None)` returned `True`. A Unix-domain socket, an ASGI adapter that leaves `scope["client"]` unset, or a hand-built request therefore turned off the token gate, the `/admin` and `/debug` guards and the WebSocket gate. - **Operator data routes.** `/stats-history` (including CSV export), `/quota`, `/subscription-window` and `/metrics` were served to any network caller. - **WebSocket loopback exemption** checked only the peer IP, not the `Host` header, so it had no DNS-rebinding defence. ## What changes - New `headroom/proxy/bind_policy.py`. A non-loopback bind (`0.0.0.0`, `::`, a LAN address or a hostname) with no token is refused at three places: `create_app` (embedders and the multi-worker factory), `run_server` (one exit with code 2 before uvicorn forks) and the `headroom proxy` CLI (a clear error before the banner, exit code 1). An absent host means uvicorn's default `127.0.0.1`. - `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` acknowledges an open bind when the runtime already restricts reachability, such as a container published on `127.0.0.1`. The `proxy_open_bind` warning is still logged on that path, and the banner says the bind is open. - `is_loopback_host(None)` now returns `False` (fail closed), matching `identity.py`. - `/stats-history`, `/quota` and `/subscription-window` answer only: a non-loopback caller that passed the token gate, a loopback caller (peer and `Host` header both loopback), a trusted dashboard client, or, under an acknowledged token-less bind, the container's own default-gateway TCP peer with a loopback `Host` (the host browser reaching a `127.0.0.1`-published container). Everyone else gets 404, as with `/admin`. Settings writes keep their stricter rule with no token short-cut. - `/metrics` answers a non-loopback caller that passed the token gate, a loopback caller, a connecting peer in `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` (whatever it forwards), or a resolved client in the dashboard CIDRs. For CIDR peers a hostname `Host` is fine, so Prometheus scrapers work; a cross-origin browser request is still refused. - The token short-cut applies only to non-loopback requests. The gate exempts loopback peers without checking `Host`, so a loopback request has not proven it holds the token and must pass the loopback `Host` check. This blocks a DNS-rebound browser on the operator's machine. - The `/dashboard` shell stays ungated. It is a static template, like the already public `/dashboard/static` assets. Gating it would break the dashboard for containers published on host loopback, where the peer is the bridge gateway and `Host` is `localhost`. The data routes behind it are gated. - The WebSocket loopback exemption requires a loopback peer and a loopback `Host` header. A loopback peer with a foreign `Host` can still connect with the token. - Launchers that always publish on `127.0.0.1` set the acknowledgement themselves: `headroom install` (docker preset), `scripts/install.sh` and `scripts/install.ps1` (through one helper that adds the acknowledgement only together with the `127.0.0.1` publication), and `docker/docker-compose.native.yml`. - Docs: `proxy.mdx` documents the new variable and which routes the token also gates; `docker-install` (docs and wiki) and the `docker-compose.yml` comment show the acknowledgement in the loopback-published example. `e2e/docker-bind-security.sh` sets it and checks that the same launch without it refuses to start. ## Behaviour change (breaking) - `headroom proxy --host 0.0.0.0`, or `HEADROOM_HOST` set to any non-loopback address, without `HEADROOM_PROXY_TOKEN` now refuses to start. The error names both remedies. Set a token, bind loopback, or set `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` when the port is already confined. Anyone running the image by hand with `--host 0.0.0.0` and no token must add one of these. - Without a token, `/stats-history`, `/quota`, `/subscription-window` and `/metrics` return 404 to non-loopback callers outside a trusted CIDR. Prometheus scrapers on another host must be in `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` or send the token. - Code that relied on a request with no peer address being treated as loopback now gets the non-loopback path. - The default loopback run is unchanged. ## Tests - `tests/test_proxy_bind_policy.py`: the refuse/allow matrix (non-loopback, loopback, unset host, token, acknowledgement values), the warning on the acknowledged path, `create_app`, the `run_server` exit before uvicorn, and the CLI (refusal, `HEADROOM_HOST` from env, acknowledged start, offline banner, start with a token). - `tests/test_proxy_loopback_gating.py`: each data route against non-loopback, loopback, trusted-CIDR and token callers; the dashboard shell stays reachable; the host of an acknowledged, loopback-published container reads the data routes while another bridge peer, a forwarded gateway address and a foreign `Host` do not; CSV export is not served to network callers; `/metrics` across loopback, both CIDR lists, out-of-range peers, cross-origin browsers and token callers, including a trusted gateway relaying a scraper outside its own range. A regression test shows that a token on the proxy does not exempt a loopback peer from the `Host` check; it fails without the fix. - `tests/test_proxy_hardening.py`: the HTTP gate with no peer address, WebSocket `Host` cases, `require_loopback` and `/debug` with no peer. - Installer tests: `headroom install` adds the acknowledgement, and the shell and PowerShell wrappers add it only together with the loopback publication. Launcher-level tests start the proxy with the environment and `--host` that `build_runtime_command()` and `docker-compose.native.yml` produce and check that the host browser (bridge-gateway peer) gets `/stats-history` and another container does not. - Tests that bound `0.0.0.0` for unrelated reasons now set a token or the acknowledgement. Tests that read operator routes use an explicit loopback peer. On `main`, 14 of the gating and hardening tests fail and the bind-policy module does not exist. Manual runs: token-less `--host 0.0.0.0` exits with the refusal from both the CLI and `run_server`; with the acknowledgement, the data routes are 404 to a LAN peer and to a loopback peer with a foreign `Host`; with a token, a LAN peer gets 401 without it and 200 with it; the default loopback bind starts as before. Not tested: a real Unix-domain-socket deployment and a real Prometheus scrape (both covered by tests only). ## Merge order Part of a set. Suggested order: #3852 → #3891 → #3849 → #3860 → #3862 → #3863. This PR goes first: it is the gate the others assume, and it is the release-note event. ## Not in this PR - Enforcing the token for loopback callers when one is configured. Revised 2026-10-02 after review: `/metrics` matches gateway CIDRs against the TCP peer, and the operator data routes serve the host of a loopback-published container (bridge-gateway peer under the launchers' bind acknowledgement) so the default Docker dashboard works. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3852" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
93f2d7a2da |
chore: release main (#2792)
🤖 I have created a release *beep* *boop* --- <details><summary>0.35.0</summary> ## [0.35.0](https://github.com/headroomlabs-ai/headroom/compare/v0.34.0...v0.35.0) (2026-08-12) ### Features * **beacon:** allowlist the routing summary key ([#2818](https://github.com/headroomlabs-ai/headroom/issues/2818)) ([ |
||
|
|
3a27c4dacb |
fix(proxy/debug): reconcile Kompress warmup state in /debug/warmup (#2711)
## Description
`/debug/warmup` serialized the warmup registry verbatim, so a Kompress
slot left at the startup snapshot kept reporting `{"status": "null",
"info": {"source_status": "deferred"}}` forever — even while the ONNX
model was loaded and actively compressing.
`/health` and `/readyz` already fix this: #2402 added
`_reconcile_kompress_health()`, which promotes the slot from live
runtime state. The debug route never called it, so its answer depended
on whether a health probe happened to run first. That is the half of
#2624 still reproducing on `main`.
Second defect: `WarmupSlot.mark_loaded()` only *updates* `info`, so the
startup-planted `source_status: "deferred"` survived promotion and the
slot serialized as the self-contradictory `{"status": "loaded", "info":
{"source_status": "deferred", "backend": "onnx"}}`.
Closes #2624
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/server.py`: call the existing
`_reconcile_kompress_health()` in the `/debug/warmup` route before
serializing the registry. The reconciler never instantiates a compressor
and never calls `preload()` / `ensure_background_load()` / `compress()`
— it only reads `is_ready()` / `ready_backend()` on an already resident
instance, or falls back to the module-level ONNX cache — so the endpoint
stays side-effect free and idempotent.
- `headroom/proxy/server.py`: stamp `source_status="runtime"` at both
`mark_loaded()` promotion sites in `_reconcile_kompress_health()` (the
resident-compressor path and the `_kompress_cache` fallback),
overwriting the stale startup marker.
- `tests/test_proxy_debug_endpoints.py`: three regression tests plus a
read-only compressor stub whose `preload` / `ensure_background_load`
raise, so a future change that makes the debug route trigger a load
fails loudly.
- `tests/test_proxy_health.py`: assert the promoted slot's
`info["source_status"] == "runtime"`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest tests/test_proxy_debug_endpoints.py tests/test_proxy_health.py tests/test_proxy_warmup.py -q
tests\test_proxy_debug_endpoints.py ............................. [ 52%]
tests\test_proxy_health.py ................. [ 83%]
tests\test_proxy_warmup.py ......... [100%]
============================= 55 passed in 36.56s =============================
$ ruff check headroom/proxy/server.py tests/test_proxy_debug_endpoints.py tests/test_proxy_health.py
All checks passed!
$ ruff format --check headroom/proxy/server.py tests/test_proxy_debug_endpoints.py tests/test_proxy_health.py
3 files already formatted
$ mypy headroom --ignore-missing-imports
Success: no issues found in 506 source files
```
The three new tests were confirmed to be genuine regression tests: with
the `server.py` change reverted and the tests kept, all three fail.
```text
$ git stash push -- headroom/proxy/server.py && pytest tests/test_proxy_debug_endpoints.py -q -k kompress
FAILED tests/test_proxy_debug_endpoints.py::test_debug_warmup_promotes_deferred_kompress_after_runtime_load
FAILED tests/test_proxy_debug_endpoints.py::test_debug_warmup_keeps_pending_kompress_null
FAILED tests/test_proxy_debug_endpoints.py::test_debug_warmup_never_starts_kompress_loading
====================== 3 failed, 26 deselected in 3.98s =======================
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.13.11, pytest 9.1.1, ruff 0.15.x,
mypy 1.20.2, branch based on `main` at
|
||
|
|
b4b50253f1 |
fix(security): block DNS-rebinding on /debug/* and /stats/reset via Host-header allowlist (#605)
## Summary `require_loopback` in `headroom/proxy/loopback_guard.py` guards `/debug/tasks`, `/debug/ws-sessions`, `/debug/warmup`, and `/stats/reset` by checking `request.client.host` only. A malicious website can use DNS rebinding to make a victim's browser send requests to `127.0.0.1` while the page origin (and inbound `Host:` header) still reads `attacker.com`. The IP check passes — the browser IS on loopback — and `app.add_middleware(CORSMiddleware, allow_origins=["*"], ...)` (`headroom/proxy/server.py:1678`) lets the attacker's JS read the response. This PR adds the canonical second gate: the `Host:` header must also name loopback (`127.0.0.1[:port]`, `[::1][:port]`, or `localhost[:port]`). ## Impact While the proxy binds to `127.0.0.1` by default (`headroom/proxy/server.py:2956`), any browser the user opens can be used by an unrelated tab to reach the proxy via DNS rebinding. Concretely: - `GET /debug/tasks` → leaks asyncio task names, coroutine qualnames, and (with `?stack=true`) stack depths — reveals which coroutines are mid-flight and which codepaths are warm. Useful for fingerprinting + targeting follow-on attacks. - `GET /debug/ws-sessions` → leaks live WebSocket session metadata including `upstream_url` per session. - `GET /debug/warmup` → leaks warmup registry shape (lower risk). - `POST /stats/reset` → resets in-memory proxy stats (state mutation; minor DoS of dashboards/observability). The fix path matches the OWASP DNS-rebinding mitigation and Starlette's `TrustedHostMiddleware` posture. ## Location - `headroom/proxy/loopback_guard.py:73` — guarded the IP check only, no `Host:` validation - `headroom/proxy/server.py:1819-1845`, `:2312` — endpoints reachable via DNS rebinding under the old gate ## Fix `require_loopback` now runs two gates: (1) the existing `request.client.host` loopback IP check, and (2) a new `Host:` header allowlist via `is_loopback_host_header(...)`. The header helper accepts `127.0.0.1[:port]`, `[::1][:port]`, `localhost[:port]`, and IPv6-mapped IPv4 (`::ffff:127.0.0.1`) — strips brackets and ports, then delegates to the existing `is_loopback_host` logic. Cross-origin browser fetches always carry the attacker's hostname in `Host:`, so rebinding requests now 404 alongside any other external attempt. Same 404 (not 403) semantics, so debug endpoints remain invisible to external scanners. The new helper is in `__all__` for explicit re-use. The existing manual-`Request`-stub unit test path (no `.headers` attribute) is preserved by an `if headers is None: return` fallback so older callers that pass a bare stub still work. Test fixtures in `tests/test_proxy_debug_endpoints.py` were updated to pin `base_url="http://127.0.0.1"` so the loopback-`Host:` invariant holds in the green-path tests, plus a new `app_and_rebinding_client` fixture and `test_debug_endpoints_block_dns_rebinding` exercising the gate at the HTTP level. Six new unit tests cover `is_loopback_host_header` (canonical accept, external reject, malformed reject, and rebinding signature). Net: +219 lines, -4 lines, in two files. **Compatibility note for operators:** a deployment that fronts the proxy behind a reverse proxy with a non-loopback Host (e.g. `headroom.local` mapped to `127.0.0.1`) and still wants `/debug/*` exposed would need to either point that reverse proxy at the upstream API endpoints only, or extend `is_loopback_host_header` with an env-var allowlist in a follow-up. The default `HEADROOM_HOST=127.0.0.1` path is unaffected. ## Detected by Aeon (manual review — no scanner rule for this; pattern matches the threat-model-claims + local-HTTP-server axes from [`skills/vuln-scanner`](https://github.com/aaronjmars/aeon-aaron/blob/main/skills/vuln-scanner/SKILL.md), priors that have surfaced 10+ similar finds across 5 languages over the last 6 weeks). - Severity: medium - CWE-350 (Reliance on Reverse DNS Resolution for a Security-Critical Action) - CWE-352 (Cross-Site Request Forgery) — adjacent - Class match: DNS-rebinding-via-loopback-bind (recurring axis in the tracker) ## Verification - `python3 -m py_compile headroom/proxy/loopback_guard.py` — parses. - The new unit tests in `tests/test_proxy_debug_endpoints.py` cover: canonical loopback Host headers (accept), external Host headers (reject), malformed/empty (reject), an HTTP-level DNS-rebinding simulation (`app_and_rebinding_client` fixture with `base_url="http://attacker.com"` + loopback client tuple), and the unchanged IP-only stub path. - Other test files that use `require_loopback` (`test_proxy_openai_responses_integration.py`, `test_proxy_openai_responses_bypass.py`, `test_proxy_dashboard_stats_cache.py`) all override it via `app.dependency_overrides[require_loopback] = lambda: None`, so they are not affected by the new gate. - I was unable to run `pytest` locally in this sandbox; please confirm the new fixtures slot in cleanly when the CI matrix runs. --- Filed by [Aeon](https://github.com/aaronjmars/aeon-aaron). --------- Co-authored-by: aeonframework <noreply@anthropic.com> |
||
|
|
4e9348dec4 | fix: add anthropic pre-upstream timeouts | ||
|
|
b71b659e1b |
fix(ci): restore repro harness test in dev installs
Add websockets to the dev extra so the repro harness smoke test can import its websocket client dependency in the CI test matrix. Also apply ruff formatting to the files the formatter check was rejecting so the 3.12 lint job passes. |
||
|
|
8bbf3afda0 |
perf(proxy): gate /debug/tasks stack-depth computation behind query param
Task.get_stack(limit=32) walks coroutine frames synchronously and measurably stalls the event loop when called for 50+ relay tasks during a reconnect storm. The /debug/tasks snapshot does not need this by default — the perf-cheap fields (name, coro_qualname, age, done) are enough for the common case of "which tasks are alive". - collect_tasks: add with_stack_depth=False kwarg; default leaves stack_depth=None per entry. - /debug/tasks: read ?stack=true query parameter and forward it. Default response is cheap; opt-in explicitly when human-debugging one snapshot. - Test: verify default entries have stack_depth=None and that ?stack=true produces at least one integer depth. |
||
|
|
55f59fad77 |
chore(proxy): announce pre-upstream concurrency; clean dead RequestLog fields and debug helpers
- Log the resolved Anthropic pre-upstream concurrency at startup so operators can correlate pre_upstream_wait_ms lines with the configured cap. Distinguishes auto-detected vs. explicit vs. unbounded. - Remove the stage_timings, session_id, and stage_timings_path fields from RequestLog. They were declared optional but never populated at any of the three RequestLog construction sites — stage timings flow exclusively through emit_stage_timings_log (structured log line) and Prometheus histograms. Keeping unpopulated fields misled readers. Option (A) from the review. - Inline the one-liner collect_ws_sessions / collect_warmup pass-throughs at their two callers in server.py; keep collect_tasks since it has real logic. Unit tests were rewritten against the registry's own snapshot()/to_dict() serializers. |
||
|
|
0e166c60d4 |
fix(proxy): accept IPv6-mapped IPv4 loopback in debug guard
On Linux dual-stack sockets (IPV6_V6ONLY=0 default), an IPv4 loopback connection arrives as ::ffff:127.0.0.1 and was 404'd by the literal set check — silent outage of /debug/* when the proxy binds to :: or 0.0.0.0. Replace the set-membership check with ipaddress.ip_address(host).is_loopback, special-casing 'localhost' (not an IP literal) and treating ValueError (malformed input) as non-loopback. The None sentinel for TestClient / UDS sockets is preserved. |
||
|
|
a4010aaa1b |
feat(proxy): add loopback-only debug introspection endpoints
Unit 5 of the Codex-proxy resilience plan. Three always-on /debug/* endpoints make "what is this process doing right now?" a single curl away: - GET /debug/tasks serializes asyncio.all_tasks() to name, coro_qualname, stack_depth, done, and age_seconds (derived from the WS session registry for Codex relay tasks, null otherwise). Sorted by age desc. No frame locals, no request bodies, no coroutine args. - GET /debug/ws-sessions returns WebSocketSessionRegistry.snapshot(). - GET /debug/warmup returns WarmupRegistry.to_dict(). A FastAPI Depends(require_loopback) dependency 404s (not 403) any non-loopback caller so debug routes stay invisible to external scanners. Dependency chosen over middleware because the route set is small and explicit Depends(...) makes the guard visible to reviewers and easy to override in tests. |