mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): admit container host gateway on compress routes (#3715)
## Description In container deployments (Docker/Podman bridge networks), inbound host traffic forwarded through the bridge arrives with the container's default gateway IP (e.g. `172.17.0.1`) rather than `127.0.0.1`. Under the standard `_require_loopback` guard, `/v1/compress` and `/v1/usage` returned `404 Not Found`. Defaulting `HEADROOM_COMPRESS_ALLOW_REMOTE=1` in the install planner dropped the loopback guard entirely, exposing those routes unauthenticated to peer containers on the shared bridge network. This replaces that default with `require_loopback_or_container_gateway` on `/v1/compress` and `/v1/usage`: - Enforces `is_loopback_host_header`, requiring incoming requests to target a loopback `Host:` (`localhost`, `127.0.0.1`, etc.). This blocks DNS-rebinding attacks and rejects peer containers targeting the container's internal IP or service hostname. - Resolves the container host gateway from `HEADROOM_CONTAINER_HOST_GATEWAY` or `/proc/net/route` when running in a container environment (`/.dockerenv`, `HEADROOM_DEPLOYMENT_RUNTIME=docker`, etc.). - Admits loopback callers and container host gateway traffic, while rejecting arbitrary peer containers on the same bridge network. - Reverts planner defaulting of `HEADROOM_COMPRESS_ALLOW_REMOTE=1`. Closes #3708 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - In `headroom/proxy/loopback_guard.py`: - Added `is_container_environment()` to detect container environments. - Added `get_container_host_gateway()` and `is_container_host_gateway()`. - Added `require_loopback_or_container_gateway()` enforcing loopback Host header and admitting loopback or container host gateway. - In `headroom/proxy/server.py`: - Switched `_compress_dependencies` default from `_require_loopback` to `_require_loopback_or_container_gateway`. - In `headroom/install/planner.py`: - Reverted injection of `HEADROOM_COMPRESS_ALLOW_REMOTE=1` into `base_env`. - In `tests/test_gateway_sidecar_ports.py`: - Added test cases verifying container gateway admission with loopback Host header, peer container rejection (both direct container IP and spoofed loopback Host header), non-loopback Host header rejection from gateway, and unit helper coverage for `/proc/net/route` parsing. - In `docs/`: - Updated documentation in `docs/content/docs/proxy.mdx` and `docs/content/docs/openai-sdk.mdx` to reflect container gateway admission. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text pytest tests/test_gateway_sidecar_ports.py tests/test_proxy_loopback_gating.py tests/test_install/test_planner.py ======================= 135 passed, 2 warnings in 7.33s ======================== ``` ## Real Behavior Proof - Environment: Linux x86_64, Python 3.12 - Verified: Container gateway IP is admitted with loopback Host header, peer containers (`172.17.0.5`) are blocked with 404, non-loopback Host headers are rejected with 404, and non-container environments reject non-loopback IPs with 404. - All 135 tests passed across sidecar ports, loopback gating, and planner suites. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: stable. - Stable/default behavior changed: `/v1/compress` and `/v1/usage` admit host gateway traffic in container environments under loopback Host headers without disabling the loopback guard. - Kill switch / disable path: None needed; `HEADROOM_COMPRESS_ALLOW_REMOTE=1` still available for wide open access if desired. - Unsafe override required: None. - Qualification impact: None. - Rollback path: Revert commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) --------- Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
@@ -46,7 +46,7 @@ That's it. Every call to `client.chat.completions.create()` compresses the messa
|
||||
3. Replaces the original messages with the compressed result
|
||||
4. Forwards the request to OpenAI as normal
|
||||
|
||||
The SDK talks to a **local** proxy, which is why no extra configuration is needed: `/v1/compress` is loopback-only by default. If you move the proxy to another host, set `HEADROOM_COMPRESS_ALLOW_REMOTE=1` on it or requests come back `404`.
|
||||
The SDK talks to a **local** proxy, which is why no extra configuration is needed: `/v1/compress` is loopback-only by default (in container environments, host gateway traffic is admitted automatically). If you move the proxy to another host, set `HEADROOM_COMPRESS_ALLOW_REMOTE=1` on it or requests come back `404`.
|
||||
|
||||
All other client methods are untouched:
|
||||
|
||||
|
||||
@@ -332,7 +332,7 @@ with a privileged job.
|
||||
| Flag / env | Default | Effect |
|
||||
|---|---|---|
|
||||
| `HEADROOM_PROXY_TOKEN` | none | Require a bearer token (`X-Headroom-Proxy-Token`) from non-loopback callers. |
|
||||
| `HEADROOM_COMPRESS_ALLOW_REMOTE` | `false` | Allow non-loopback callers to reach [`POST /v1/compress`](#post-v1compress). Required to run Headroom as a gateway/sidecar; without it remote callers get `404`. |
|
||||
| `HEADROOM_COMPRESS_ALLOW_REMOTE` | `false` | Allow non-loopback callers to reach [`POST /v1/compress`](#post-v1compress) and `POST /v1/usage`. Required to run Headroom as a remote gateway/sidecar; without it remote callers get `404`. |
|
||||
| `--offline` / `HEADROOM_OFFLINE` | `false` | Air-gap mode: hard-disable **all** egress (telemetry, update checks, license reporting, model downloads). |
|
||||
| `--stateless` / `HEADROOM_STATELESS` | `false` | Keep all state in memory; no filesystem writes (disables logs, memory, TOIN). |
|
||||
| `HEADROOM_STRIP_INTERNAL_HEADERS` | `enabled` | Strip internal `x-headroom-*` headers before forwarding upstream. |
|
||||
@@ -460,7 +460,7 @@ If `HEADROOM_KOMPRESS_ENDPOINT` is set, Kompress inference is offloaded over HTT
|
||||
<Callout type="warning" title="Loopback-only by default">
|
||||
This route is restricted to loopback callers and answers everyone else with **`404`**, not `403` — deliberately, so it stays invisible to external scanners. A gateway calling it from another host or pod therefore sees what looks like a missing route.
|
||||
|
||||
Both the client IP and the inbound `Host:` header must name loopback. To allow remote callers, set `HEADROOM_COMPRESS_ALLOW_REMOTE=1`. `HEADROOM_PROXY_TOKEN` still applies if set.
|
||||
Both the client IP and the inbound `Host:` header must name loopback. In container environments (Docker/Podman), requests forwarded from the container host gateway are also admitted under loopback `Host:` headers. To allow arbitrary remote callers, set `HEADROOM_COMPRESS_ALLOW_REMOTE=1`. `HEADROOM_PROXY_TOKEN` still applies if set. The same policy applies to [`POST /v1/usage`](#relaying-provider-usage-post-v1usage).
|
||||
</Callout>
|
||||
|
||||
#### Message format
|
||||
|
||||
@@ -40,6 +40,9 @@ CSRF / DNS-rebinding guidance and the standard Starlette
|
||||
from __future__ import annotations
|
||||
|
||||
import ipaddress
|
||||
import os
|
||||
import socket
|
||||
import struct
|
||||
|
||||
try:
|
||||
from fastapi import HTTPException, Request
|
||||
@@ -50,10 +53,14 @@ except ImportError: # pragma: no cover - fastapi is a hard dep in practice
|
||||
|
||||
__all__ = [
|
||||
"LOOPBACK_HOSTS",
|
||||
"get_container_host_gateway",
|
||||
"is_container_environment",
|
||||
"is_container_host_gateway",
|
||||
"is_ip_literal_host_header",
|
||||
"is_loopback_host",
|
||||
"is_loopback_host_header",
|
||||
"require_loopback",
|
||||
"require_loopback_or_container_gateway",
|
||||
"require_same_origin",
|
||||
]
|
||||
|
||||
@@ -216,6 +223,125 @@ def require_loopback(request: Request) -> None: # type: ignore[valid-type]
|
||||
raise HTTPException(status_code=404)
|
||||
|
||||
|
||||
def is_container_environment() -> bool:
|
||||
"""Return True if running inside a container or containerized deployment."""
|
||||
if os.environ.get("HEADROOM_CONTAINER_HOST_GATEWAY"):
|
||||
return True
|
||||
if os.environ.get("HEADROOM_DEPLOYMENT_RUNTIME") in ("docker", "podman", "container"):
|
||||
return True
|
||||
if os.environ.get("HEADROOM_DEPLOYMENT_PRESET") == "persistent-docker":
|
||||
return True
|
||||
if os.path.exists("/.dockerenv") or os.path.exists("/run/.containerenv"):
|
||||
return True
|
||||
try:
|
||||
with open("/proc/1/cgroup", encoding="utf-8", errors="ignore") as f:
|
||||
content = f.read()
|
||||
if any(marker in content for marker in ("docker", "containerd", "kubepods", "libpod")):
|
||||
return True
|
||||
except OSError:
|
||||
pass
|
||||
return False
|
||||
|
||||
|
||||
def _read_linux_default_gateway() -> str | None:
|
||||
"""Read the default IPv4 gateway address from /proc/net/route."""
|
||||
try:
|
||||
with open("/proc/net/route", encoding="ascii") as f:
|
||||
for line in f:
|
||||
fields = line.strip().split()
|
||||
if len(fields) >= 3 and fields[1] == "00000000":
|
||||
gw_hex = fields[2]
|
||||
if len(gw_hex) == 8:
|
||||
gw_int = int(gw_hex, 16)
|
||||
if gw_int != 0:
|
||||
return socket.inet_ntoa(struct.pack("<L", gw_int))
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def get_container_host_gateway() -> str | None:
|
||||
"""Return the container host gateway IP if running in a container.
|
||||
|
||||
Checks ``HEADROOM_CONTAINER_HOST_GATEWAY`` first, and falls back to
|
||||
resolving the default gateway from ``/proc/net/route`` when running
|
||||
inside a container environment. Returns None when not in a container or
|
||||
if no gateway can be determined.
|
||||
"""
|
||||
env_gw = os.environ.get("HEADROOM_CONTAINER_HOST_GATEWAY")
|
||||
if env_gw:
|
||||
env_gw = env_gw.strip()
|
||||
if env_gw:
|
||||
return env_gw
|
||||
if not is_container_environment():
|
||||
return None
|
||||
return _read_linux_default_gateway()
|
||||
|
||||
|
||||
def _normalize_ip(host: str) -> ipaddress.IPv4Address | ipaddress.IPv6Address | None:
|
||||
try:
|
||||
ip = ipaddress.ip_address(host)
|
||||
except ValueError:
|
||||
return None
|
||||
if isinstance(ip, ipaddress.IPv6Address) and ip.ipv4_mapped is not None:
|
||||
return ip.ipv4_mapped
|
||||
return ip
|
||||
|
||||
|
||||
def is_container_host_gateway(host: str | None) -> bool:
|
||||
"""Return True if ``host`` matches the container host gateway IP.
|
||||
|
||||
Always returns False if not in a container environment or if ``host``
|
||||
is empty. Correctly handles IPv4-mapped IPv6 addresses (e.g.
|
||||
``::ffff:172.17.0.1``).
|
||||
"""
|
||||
if not host or not is_container_environment():
|
||||
return False
|
||||
gateway = get_container_host_gateway()
|
||||
if not gateway:
|
||||
return False
|
||||
parsed_host = _normalize_ip(host)
|
||||
parsed_gw = _normalize_ip(gateway)
|
||||
if parsed_host is not None and parsed_gw is not None:
|
||||
return parsed_host == parsed_gw
|
||||
return host.strip() == gateway.strip()
|
||||
|
||||
|
||||
def require_loopback_or_container_gateway(request: Request) -> None: # type: ignore[valid-type]
|
||||
"""FastAPI dependency: allow loopback callers or container host gateway.
|
||||
|
||||
Used by routes like ``/v1/compress`` and ``/v1/usage``. When running inside
|
||||
a container environment (Docker / Podman), incoming host traffic forwarded
|
||||
over the container bridge arrives with the host gateway IP (e.g. 172.17.0.1)
|
||||
rather than a loopback interface.
|
||||
|
||||
Enforces two gates:
|
||||
1. Client address must be either loopback (:func:`is_loopback_host`) or
|
||||
the container host default gateway (:func:`is_container_host_gateway`).
|
||||
Arbitrary peer containers on the same bridge network are rejected.
|
||||
2. The inbound ``Host:`` header must name a loopback host
|
||||
(:func:`is_loopback_host_header`). This blocks DNS-rebinding attacks and
|
||||
rejects callers addressing the container's bridge IP or hostname directly.
|
||||
"""
|
||||
if HTTPException is None: # pragma: no cover - defensive
|
||||
raise RuntimeError("FastAPI is required for the loopback guard")
|
||||
|
||||
client = getattr(request, "client", None)
|
||||
host = getattr(client, "host", None) if client is not None else None
|
||||
if not (is_loopback_host(host) or is_container_host_gateway(host)):
|
||||
raise HTTPException(status_code=404)
|
||||
|
||||
headers = getattr(request, "headers", None)
|
||||
if headers is None:
|
||||
return
|
||||
try:
|
||||
host_header = headers.get("host")
|
||||
except AttributeError:
|
||||
host_header = None
|
||||
if not is_loopback_host_header(host_header):
|
||||
raise HTTPException(status_code=404)
|
||||
|
||||
|
||||
def require_same_origin(request: Request) -> None: # type: ignore[valid-type]
|
||||
"""FastAPI dependency: reject cross-origin browser requests on mutating routes.
|
||||
|
||||
|
||||
@@ -3844,8 +3844,15 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
from headroom.proxy.debug_introspection import (
|
||||
collect_tasks as _collect_tasks,
|
||||
)
|
||||
from headroom.proxy.loopback_guard import require_loopback as _require_loopback
|
||||
from headroom.proxy.loopback_guard import require_same_origin as _require_same_origin
|
||||
from headroom.proxy.loopback_guard import (
|
||||
require_loopback as _require_loopback,
|
||||
)
|
||||
from headroom.proxy.loopback_guard import (
|
||||
require_loopback_or_container_gateway as _require_loopback_or_container_gateway,
|
||||
)
|
||||
from headroom.proxy.loopback_guard import (
|
||||
require_same_origin as _require_same_origin,
|
||||
)
|
||||
|
||||
def _require_loopback_or_trusted_dashboard_client(request: Request) -> None:
|
||||
"""Allow loopback callers, or gateway-forwarded dashboard clients.
|
||||
@@ -5613,7 +5620,7 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
_compress_dependencies = (
|
||||
[]
|
||||
if _get_env_bool("HEADROOM_COMPRESS_ALLOW_REMOTE", False)
|
||||
else [Depends(_require_loopback)]
|
||||
else [Depends(_require_loopback_or_container_gateway)]
|
||||
)
|
||||
|
||||
@app.post("/v1/compress", dependencies=_compress_dependencies)
|
||||
|
||||
@@ -132,3 +132,98 @@ def test_compress_allows_non_loopback_with_flag(monkeypatch):
|
||||
client = TestClient(_fast_app())
|
||||
resp = client.post("/v1/compress", json=_BODY)
|
||||
assert resp.status_code == 200, resp.text
|
||||
|
||||
|
||||
def test_compress_allows_container_gateway_with_loopback_host(monkeypatch):
|
||||
monkeypatch.delenv("HEADROOM_COMPRESS_ALLOW_REMOTE", raising=False)
|
||||
monkeypatch.setenv("HEADROOM_CONTAINER_HOST_GATEWAY", "172.17.0.1")
|
||||
client = TestClient(
|
||||
_fast_app(),
|
||||
base_url="http://127.0.0.1:8787",
|
||||
client=("172.17.0.1", 12345),
|
||||
)
|
||||
resp = client.post("/v1/compress", json=_BODY)
|
||||
assert resp.status_code == 200, resp.text
|
||||
|
||||
|
||||
def test_compress_blocks_peer_container_on_bridge_network(monkeypatch):
|
||||
monkeypatch.delenv("HEADROOM_COMPRESS_ALLOW_REMOTE", raising=False)
|
||||
monkeypatch.setenv("HEADROOM_CONTAINER_HOST_GATEWAY", "172.17.0.1")
|
||||
# Peer container connecting directly with its own container IP
|
||||
client = TestClient(
|
||||
_fast_app(),
|
||||
base_url="http://172.17.0.2:8787",
|
||||
client=("172.17.0.5", 12345),
|
||||
)
|
||||
assert client.post("/v1/compress", json=_BODY).status_code == 404
|
||||
|
||||
# Peer container trying to spoof a loopback Host header
|
||||
client_spoofed_host = TestClient(
|
||||
_fast_app(),
|
||||
base_url="http://127.0.0.1:8787",
|
||||
client=("172.17.0.5", 12345),
|
||||
)
|
||||
assert client_spoofed_host.post("/v1/compress", json=_BODY).status_code == 404
|
||||
|
||||
|
||||
def test_compress_blocks_non_loopback_host_header_from_gateway(monkeypatch):
|
||||
monkeypatch.delenv("HEADROOM_COMPRESS_ALLOW_REMOTE", raising=False)
|
||||
monkeypatch.setenv("HEADROOM_CONTAINER_HOST_GATEWAY", "172.17.0.1")
|
||||
# Gateway IP caller but with external domain Host header (e.g. DNS rebinding)
|
||||
client = TestClient(
|
||||
_fast_app(),
|
||||
base_url="http://attacker.com",
|
||||
client=("172.17.0.1", 12345),
|
||||
)
|
||||
assert client.post("/v1/compress", json=_BODY).status_code == 404
|
||||
|
||||
|
||||
def test_loopback_guard_container_gateway_helpers(monkeypatch, tmp_path):
|
||||
from headroom.proxy.loopback_guard import (
|
||||
get_container_host_gateway,
|
||||
is_container_environment,
|
||||
is_container_host_gateway,
|
||||
)
|
||||
|
||||
# Clean state - non-container environment
|
||||
monkeypatch.delenv("HEADROOM_CONTAINER_HOST_GATEWAY", raising=False)
|
||||
monkeypatch.delenv("HEADROOM_DEPLOYMENT_RUNTIME", raising=False)
|
||||
monkeypatch.delenv("HEADROOM_DEPLOYMENT_PRESET", raising=False)
|
||||
monkeypatch.setattr("os.path.exists", lambda path: False)
|
||||
assert not is_container_environment()
|
||||
assert get_container_host_gateway() is None
|
||||
assert not is_container_host_gateway("172.17.0.1")
|
||||
|
||||
# Explicit override
|
||||
monkeypatch.setenv("HEADROOM_CONTAINER_HOST_GATEWAY", "172.17.0.1")
|
||||
assert is_container_environment()
|
||||
assert get_container_host_gateway() == "172.17.0.1"
|
||||
assert is_container_host_gateway("172.17.0.1")
|
||||
assert is_container_host_gateway("::ffff:172.17.0.1")
|
||||
assert not is_container_host_gateway("172.17.0.5")
|
||||
|
||||
# Linux route file parsing
|
||||
monkeypatch.delenv("HEADROOM_CONTAINER_HOST_GATEWAY", raising=False)
|
||||
monkeypatch.setenv("HEADROOM_DEPLOYMENT_RUNTIME", "docker")
|
||||
assert is_container_environment()
|
||||
|
||||
route_content = (
|
||||
"Iface\tDestination\tGateway \tFlags\tRefCnt\tUse\tMetric\tMask\t\tMTU\tWindow\tIRTT\n"
|
||||
"eth0\t00000000\t0111A8C0\t0003\t0\t0\t0\t00000000\t0\t0\t0\n"
|
||||
)
|
||||
route_file = tmp_path / "route"
|
||||
route_file.write_text(route_content, encoding="ascii")
|
||||
|
||||
import builtins
|
||||
|
||||
real_open = builtins.open
|
||||
|
||||
def fake_open(file, *args, **kwargs):
|
||||
if file == "/proc/net/route":
|
||||
return real_open(route_file, *args, **kwargs)
|
||||
return real_open(file, *args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(builtins, "open", fake_open)
|
||||
assert get_container_host_gateway() == "192.168.17.1"
|
||||
assert is_container_host_gateway("192.168.17.1")
|
||||
assert not is_container_host_gateway("192.168.17.2")
|
||||
|
||||
Reference in New Issue
Block a user