Files
headroom/e2e/docker-bind-security.sh
ee731d7ea8 fix(proxy): refuse token-less non-loopback binds, gate operator routes (#3852)
## Why

Four related gaps in the proxy's network gate (findings 01-F1, 01-F2,
01-F5, 01-F9):

- **Open bind.** A non-loopback bind with no `HEADROOM_PROXY_TOKEN` only
logged a warning, then served the `/v1/*` relay, with the operator's
upstream credentials, to every peer that could reach the port.
- **Missing peer address.** `is_loopback_host(None)` returned `True`. A
Unix-domain socket, an ASGI adapter that leaves `scope["client"]` unset,
or a hand-built request therefore turned off the token gate, the
`/admin` and `/debug` guards and the WebSocket gate.
- **Operator data routes.** `/stats-history` (including CSV export),
`/quota`, `/subscription-window` and `/metrics` were served to any
network caller.
- **WebSocket loopback exemption** checked only the peer IP, not the
`Host` header, so it had no DNS-rebinding defence.

## What changes

- New `headroom/proxy/bind_policy.py`. A non-loopback bind (`0.0.0.0`,
`::`, a LAN address or a hostname) with no token is refused at three
places: `create_app` (embedders and the multi-worker factory),
`run_server` (one exit with code 2 before uvicorn forks) and the
`headroom proxy` CLI (a clear error before the banner, exit code 1). An
absent host means uvicorn's default `127.0.0.1`.
- `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` acknowledges an open bind when
the runtime already restricts reachability, such as a container
published on `127.0.0.1`. The `proxy_open_bind` warning is still logged
on that path, and the banner says the bind is open.
- `is_loopback_host(None)` now returns `False` (fail closed), matching
`identity.py`.
- `/stats-history`, `/quota` and `/subscription-window` answer only: a
non-loopback caller that passed the token gate, a loopback caller (peer
and `Host` header both loopback), a trusted dashboard client, or, under
an acknowledged token-less bind, the container's own default-gateway TCP
peer with a loopback `Host` (the host browser reaching a
`127.0.0.1`-published container). Everyone else gets 404, as with
`/admin`. Settings writes keep their stricter rule with no token
short-cut.
- `/metrics` answers a non-loopback caller that passed the token gate, a
loopback caller, a connecting peer in
`HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` (whatever it forwards), or a
resolved client in the dashboard CIDRs. For CIDR peers a hostname `Host`
is fine, so Prometheus scrapers work; a cross-origin browser request is
still refused.
- The token short-cut applies only to non-loopback requests. The gate
exempts loopback peers without checking `Host`, so a loopback request
has not proven it holds the token and must pass the loopback `Host`
check. This blocks a DNS-rebound browser on the operator's machine.
- The `/dashboard` shell stays ungated. It is a static template, like
the already public `/dashboard/static` assets. Gating it would break the
dashboard for containers published on host loopback, where the peer is
the bridge gateway and `Host` is `localhost`. The data routes behind it
are gated.
- The WebSocket loopback exemption requires a loopback peer and a
loopback `Host` header. A loopback peer with a foreign `Host` can still
connect with the token.
- Launchers that always publish on `127.0.0.1` set the acknowledgement
themselves: `headroom install` (docker preset), `scripts/install.sh` and
`scripts/install.ps1` (through one helper that adds the acknowledgement
only together with the `127.0.0.1` publication), and
`docker/docker-compose.native.yml`.
- Docs: `proxy.mdx` documents the new variable and which routes the
token also gates; `docker-install` (docs and wiki) and the
`docker-compose.yml` comment show the acknowledgement in the
loopback-published example. `e2e/docker-bind-security.sh` sets it and
checks that the same launch without it refuses to start.

## Behaviour change (breaking)

- `headroom proxy --host 0.0.0.0`, or `HEADROOM_HOST` set to any
non-loopback address, without `HEADROOM_PROXY_TOKEN` now refuses to
start. The error names both remedies. Set a token, bind loopback, or set
`HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` when the port is already
confined. Anyone running the image by hand with `--host 0.0.0.0` and no
token must add one of these.
- Without a token, `/stats-history`, `/quota`, `/subscription-window`
and `/metrics` return 404 to non-loopback callers outside a trusted
CIDR. Prometheus scrapers on another host must be in
`HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` or send the token.
- Code that relied on a request with no peer address being treated as
loopback now gets the non-loopback path.
- The default loopback run is unchanged.

## Tests

- `tests/test_proxy_bind_policy.py`: the refuse/allow matrix
(non-loopback, loopback, unset host, token, acknowledgement values), the
warning on the acknowledged path, `create_app`, the `run_server` exit
before uvicorn, and the CLI (refusal, `HEADROOM_HOST` from env,
acknowledged start, offline banner, start with a token).
- `tests/test_proxy_loopback_gating.py`: each data route against
non-loopback, loopback, trusted-CIDR and token callers; the dashboard
shell stays reachable; the host of an acknowledged, loopback-published
container reads the data routes while another bridge peer, a forwarded
gateway address and a foreign `Host` do not; CSV export is not served to
network callers; `/metrics` across loopback, both CIDR lists,
out-of-range peers, cross-origin browsers and token callers, including a
trusted gateway relaying a scraper outside its own range. A regression
test shows that a token on the proxy does not exempt a loopback peer
from the `Host` check; it fails without the fix.
- `tests/test_proxy_hardening.py`: the HTTP gate with no peer address,
WebSocket `Host` cases, `require_loopback` and `/debug` with no peer.
- Installer tests: `headroom install` adds the acknowledgement, and the
shell and PowerShell wrappers add it only together with the loopback
publication. Launcher-level tests start the proxy with the environment
and `--host` that `build_runtime_command()` and
`docker-compose.native.yml` produce and check that the host browser
(bridge-gateway peer) gets `/stats-history` and another container does
not.
- Tests that bound `0.0.0.0` for unrelated reasons now set a token or
the acknowledgement. Tests that read operator routes use an explicit
loopback peer.

On `main`, 14 of the gating and hardening tests fail and the bind-policy
module does not exist. Manual runs: token-less `--host 0.0.0.0` exits
with the refusal from both the CLI and `run_server`; with the
acknowledgement, the data routes are 404 to a LAN peer and to a loopback
peer with a foreign `Host`; with a token, a LAN peer gets 401 without it
and 200 with it; the default loopback bind starts as before.

Not tested: a real Unix-domain-socket deployment and a real Prometheus
scrape (both covered by tests only).

## Merge order

Part of a set. Suggested order: #3852 → #3891 → #3849 → #3860 → #3862 →
#3863. This PR goes first: it is the gate the others assume, and it is
the release-note event.

## Not in this PR

- Enforcing the token for loopback callers when one is configured.

Revised 2026-10-02 after review: `/metrics` matches gateway CIDRs
against the TCP peer, and the operator data routes serve the host of a
loopback-published container (bridge-gateway peer under the launchers'
bind acknowledgement) so the default Docker dashboard works.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3852"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-05 08:27:35 -05:00

195 lines
9.8 KiB
Bash

#!/usr/bin/env bash
set -euo pipefail
unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY ALL_PROXY all_proxy
base_image=; head_image=; fixture=; base_ref="${HEADROOM_BASE_REF:-origin/main}"
while (($#)); do
case "$1" in
--base-image) base_image="$2"; shift 2 ;;
--head-image) head_image="$2"; shift 2 ;;
--fixture) fixture="$2"; shift 2 ;;
--base-ref) base_ref="$2"; shift 2 ;;
*) echo "unknown argument: $1" >&2; exit 2 ;;
esac
done
[[ -n "$base_image" && -n "$head_image" && -n "$fixture" ]] || exit 2
PYTHON_BIN="${PYTHON_BIN:-python3}"
command -v "$PYTHON_BIN" >/dev/null 2>&1 || PYTHON_BIN=python
command -v "$PYTHON_BIN" >/dev/null 2>&1 || { echo "python3 or python is required" >&2; exit 127; }
base_container="headroom-bind-base-$$"; head_container="headroom-bind-head-$$"
local_container="headroom-bind-local-$$"; public_container="headroom-bind-public-$$"
attacker_container="headroom-bind-attacker-$$"; mock_container="headroom-bind-upstream-$$"
slim_container="headroom-bind-slim-$$"; network="headroom-bind-network-$$"
tmp_dir="$(mktemp -d)"
cleanup() {
docker rm -f "$attacker_container" "$public_container" "$local_container" "$slim_container" "$head_container" "$base_container" "$mock_container" >/dev/null 2>&1 || true
# The CI workflow reuses the head image for installer, compose, and wrap checks.
docker image rm "$base_image" "${head_image}-runtime-slim" >/dev/null 2>&1 || true
docker network rm "$network" >/dev/null 2>&1 || true
rm -rf "$tmp_dir"
}
trap cleanup EXIT
mapfile -t request_paths < <("$PYTHON_BIN" - "$fixture" <<'PY'
import json, sys
for path in json.load(open(sys.argv[1], encoding="utf-8"))["request_paths"]:
print(path)
PY
)
upstream_path="$("$PYTHON_BIN" - "$fixture" <<'PY'
import json, sys
print(json.load(open(sys.argv[1], encoding="utf-8"))["upstream"]["path"])
PY
)"
missing_status="$("$PYTHON_BIN" - "$fixture" <<'PY'
import json, sys
print(json.load(open(sys.argv[1], encoding="utf-8"))["token_control"]["missing"])
PY
)"
wrong_status="$("$PYTHON_BIN" - "$fixture" <<'PY'
import json, sys
print(json.load(open(sys.argv[1], encoding="utf-8"))["token_control"]["wrong"])
PY
)"
correct_status="$("$PYTHON_BIN" - "$fixture" <<'PY'
import json, sys
print(json.load(open(sys.argv[1], encoding="utf-8"))["token_control"]["correct"])
PY
)"
ws_status_expected="$("$PYTHON_BIN" - "$fixture" <<'PY'
import json, sys
print(json.load(open(sys.argv[1], encoding="utf-8"))["websocket_control"]["missing"])
PY
)"
upstream_path="${upstream_path//$'\r'/}"
fixture_description="$("$PYTHON_BIN" - "$fixture" <<'PY'
import json, sys
print(json.load(open(sys.argv[1], encoding="utf-8"))["description"])
PY
)"
upstream_method="$("$PYTHON_BIN" - "$fixture" <<'PY'
import json, sys
print(json.load(open(sys.argv[1], encoding="utf-8"))["upstream"]["method"])
PY
)"
upstream_authorization="$("$PYTHON_BIN" - "$fixture" <<'PY'
import json, sys
value = json.load(open(sys.argv[1], encoding="utf-8"))["upstream"]["authorization"]
print("" if value is None else value)
PY
)"
network_publication="$("$PYTHON_BIN" - "$fixture" <<'PY'
import json, sys
print(json.load(open(sys.argv[1], encoding="utf-8"))["network"]["publication"])
PY
)"
network_peer="$("$PYTHON_BIN" - "$fixture" <<'PY'
import json, sys
print(json.load(open(sys.argv[1], encoding="utf-8"))["network"]["peer"])
PY
)"
test "$fixture_description" = "Unauthenticated public Docker publication"
test "$upstream_method" = POST
test -z "$upstream_authorization"
test "$network_publication" = "0.0.0.0:8787:8787"
test "$network_peer" = non-loopback
git archive "$base_ref" | tar -x -C "$tmp_dir"
docker build -t "$base_image" "$tmp_dir" >/dev/null
docker build -t "$head_image" . >/dev/null
slim_image="${head_image}-runtime-slim"
docker build --target runtime-slim -t "$slim_image" . >/dev/null
docker network create "$network" >/dev/null
mock_script='from http.server import BaseHTTPRequestHandler,HTTPServer
class Handler(BaseHTTPRequestHandler):
def _reply(self):
print(self.command + " " + self.path + " authorization=" + str(self.headers.get("authorization")), flush=True)
self.send_response(200); self.send_header("Content-Type", "application/json"); self.end_headers(); self.wfile.write(b"{}")
do_GET = _reply
do_POST = _reply
def log_message(self, *_): pass
HTTPServer(("0.0.0.0", 8080), Handler).serve_forever()'
docker run -d --rm --name "$mock_container" --network "$network" python:3.12-alpine python -c "$mock_script" >/dev/null
docker run -d --rm --name "$base_container" --network "$network" -p "0.0.0.0:18783:8787" \
-e "OPENAI_TARGET_API_URL=http://${mock_container}:8080" -e OPENAI_API_KEY=upstream-secret \
"$base_image" >/dev/null
docker run -d --rm --name "$head_container" --network "$network" -p "0.0.0.0:18784:8787" "$head_image" >/dev/null
# Loopback-published container that binds 0.0.0.0 inside: the proxy cannot see
# the host publication, so the operator acknowledges the token-less open bind.
docker run -d --rm --name "$local_container" --network "$network" -p "127.0.0.1:18786:8787" \
-e HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1 \
-e "OPENAI_TARGET_API_URL=http://${mock_container}:8080" -e OPENAI_API_KEY=upstream-secret \
"$head_image" --host 0.0.0.0 --port 8787 >/dev/null
# Without the acknowledgement or a token, the same launch must refuse to start
# and name both ways out.
refused_output="$(docker run --rm --network "$network" "$head_image" --host 0.0.0.0 --port 8787 2>&1)" \
&& { echo "token-less 0.0.0.0 bind started without acknowledgement" >&2; exit 1; }
grep -Fq HEADROOM_PROXY_TOKEN <<<"$refused_output"
grep -Fq HEADROOM_ALLOW_UNAUTHENTICATED_BIND <<<"$refused_output"
docker image inspect --format '{{range .Config.Env}}{{println .}}{{end}}' "$slim_image" \
| grep -Fxq 'HEADROOM_HOST=127.0.0.1'
docker run --rm --entrypoint python3 "$slim_image" -c 'import headroom._core; print("rust_core=loaded")'
host_ip="$(docker network inspect -f '{{range .IPAM.Config}}{{.Gateway}}{{end}}' "$network")"
wait_healthy() {
local container="$1"
for _ in $(seq 1 60); do
if docker inspect -f '{{.State.Health.Status}}' "$container" 2>/dev/null | grep -Fxq healthy; then
return 0
fi
sleep 1
done
docker inspect -f '{{.State.Health.Status}}' "$container"
docker logs "$container" 2>&1 || true
return 1
}
wait_healthy "$base_container"
wait_healthy "$head_container"
wait_healthy "$local_container"
docker exec "$base_container" python3 -c 'import urllib.request; urllib.request.urlopen("http://127.0.0.1:8787/readyz", timeout=5)'
docker exec "$head_container" python3 -c 'import urllib.request; urllib.request.urlopen("http://127.0.0.1:8787/readyz", timeout=5)'
docker exec "$local_container" python3 -c 'import urllib.request; urllib.request.urlopen("http://127.0.0.1:8787/readyz", timeout=5)'
request() {
local port="$1" path="$2" token="${3:-}"; local -a args=(--silent --output /dev/null --write-out '%{http_code}' --connect-timeout 3)
[[ -n "$token" ]] && args+=(-H "Authorization: Bearer ${token}")
[[ "$path" == "$upstream_path" ]] && args+=(-H 'Content-Type: application/json' --data '{"model":"fixture","messages":[]}')
docker run --rm --name "$attacker_container" --network "$network" curlimages/curl:8.12.1 \
"${args[@]}" "http://${host_ip}:${port}${path}"
}
first_path="${request_paths[0]//$'\r'/}"
base_first_status="$(request 18783 "$first_path" || true)"
if [[ "$base_first_status" == 200 ]]; then
# The vulnerable base must retain the issue-shaped unauthenticated exposure.
for path in "${request_paths[@]}"; do
path="${path//$'\r'/}"
test "$(request 18783 "$path")" = 200
done
docker logs "$mock_container" 2>&1 | grep -Fq "${upstream_method} ${upstream_path} authorization=${upstream_authorization:-None}"
elif [[ "$base_first_status" != 000 && "$base_first_status" != 401 && "$base_first_status" != 403 ]]; then
echo "unexpected safe-base status: $base_first_status" >&2
exit 1
fi
test "$(request 18784 "$first_path")" = 000
test "$(curl --silent --output /dev/null --write-out '%{http_code}' --connect-timeout 3 "http://127.0.0.1:18786${first_path}")" = 200
test "$(curl --silent --output /dev/null --write-out '%{http_code}' --connect-timeout 3 --header 'Content-Type: application/json' --data '{"model":"fixture","messages":[]}' "http://127.0.0.1:18786${upstream_path}")" = 200
test "$(request 18786 "$first_path" || true)" = 000
token="$(openssl rand -hex 32)"
docker run -d --rm --name "$public_container" --network "$network" -p 0.0.0.0:18785:8787 \
-e "HEADROOM_PROXY_TOKEN=${token}" -e "OPENAI_TARGET_API_URL=http://${mock_container}:8080" \
-e OPENAI_API_KEY=upstream-secret \
"$head_image" --host 0.0.0.0 --port 8787 >/dev/null
for _ in $(seq 1 60); do curl -4 --fail --silent http://127.0.0.1:18785/readyz >/dev/null && break; sleep 1; done
for path in "${request_paths[@]}"; do
path="${path//$'\r'/}"
test "$(request 18785 "$path")" = "$missing_status"
test "$(request 18785 "$path" wrong-token)" = "$wrong_status"
test "$(request 18785 "$path" "$token")" = "$correct_status"
done
ws_key="$(printf '%s' 'the sample nonce' | base64 | tr -d '\n')"
ws_status="$(docker run --rm --name "$attacker_container" --network "$network" curlimages/curl:8.12.1 --silent --output /dev/null --write-out '%{http_code}' --http1.1 -H 'Connection: Upgrade' -H 'Upgrade: websocket' -H 'Sec-WebSocket-Version: 13' -H "Sec-WebSocket-Key: ${ws_key}" "http://${host_ip}:18785/v1/live")"
test "$ws_status" = "$ws_status_expected"
printf 'base_ref=%s base_digest=%s head_digest=%s slim_digest=%s runtime_slim_config=loopback rust_core=loaded local_completion=200 attacker_loopback=000 ws_status=%s\n' "$base_ref" "$(docker image inspect --format='{{.Id}}' "$base_image")" "$(docker image inspect --format='{{.Id}}' "$head_image")" "$(docker image inspect --format='{{.Id}}' "$slim_image")" "$ws_status"
echo "docker bind security: fixture forwarding, loopback isolation, open-bind refusal, built-image health, slim-image loopback config, and token controls passed"