mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): pin the validated address so DNS rebinding cannot move an upstream (#3731)
Security-remediation item **A-4** (P-2). The agent that wrote this was killed mid-run by a machine sleep, so I reviewed the diff, re-ran the evidence and opened this myself. ## The defect `upstream_guard.is_safe_upstream_url` resolves a caller-supplied hostname, checks every returned address is global, and then hands the **hostname** to httpx — which resolves it *again* when it opens the socket. Two resolutions are two chances to answer, and an attacker who controls the name's authoritative DNS takes both: a public address for the check, `169.254.169.254` or an RFC1918 host for the connection. The verdict was never wrong. It just never reached the socket. ## The fix Two halves: - **`upstream_guard.py`** now records the addresses each accepted destination was *judged on* (60 s TTL, bounded at 512 entries, LRU-evicted). - **`upstream_pinning.py`** (new) wraps the httpcore connection pool's network backend and substitutes a validated address for the hostname at `connect_tcp` time — **and only there**. Substituting at the socket rather than rewriting the URL to an IP literal is the whole design, because the URL still carries the hostname: - httpx still sends `Host: <hostname>`, so gateways and multi-tenant providers still route correctly; - httpcore still passes the hostname as `server_hostname` to `start_tls`, so **SNI and certificate hostname verification are untouched**; - two hostnames sharing an address do not collapse into one pooled origin, so a connection whose certificate was checked for one host can never serve the other. All addresses from the accepted answer are pinned, in resolver order, so an AAAA record on a host with no IPv6 route still falls back the way the OS resolver would have — at no cost in safety, since every address in the set passed the same check. ## Scope — why this cannot regress other harnesses The guard only ever sees **client-supplied** upstreams (`x-headroom-base-url`). Every configured provider upstream — Anthropic, OpenAI, Gemini, Bedrock, Copilot, LiteLLM, a Kong gateway — is operator-configured, never passes through the guard, has no pin, and resolves exactly as before. Redirects and retries re-enter the same path. With an HTTP/SOCKS proxy configured the pool dials the *proxy's* host, which is never pinned, so proxied deployments are unchanged. Allowlist mode pins nothing on purpose: it admits a host by name without resolving it, precisely so split-horizon and on-prem endpoints keep working. `install_upstream_pinning` raises if httpx/httpcore stops exposing the pool it hooks. That is fail-closed by design — the alternative is a proxy that starts happily with the hole quietly reopened. Versions are locked in `uv.lock`, so it can only fire on a deliberate dependency bump, which is exactly when someone should look. ## Verification Scoped to the pinning tests, reverting the source and keeping the tests: ``` BEFORE: 6 failed, 1 passed AFTER: 7 passed ``` The one test passing in both is `test_pinning_does_not_disable_certificate_hostname_verification` — cert verification was already on, and the test exists to prove this change does not weaken it. Full file: **45 passed**. The tests include a stub resolver that returns a different address on successive calls and asserts the connection targets the validated one, plus coverage of pin expiry, store bounding, and the allowlist-pins-nothing rule. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
577336d2b6
commit
a102e05b51
@@ -175,6 +175,28 @@ Matching is on the parsed hostname and is exact — no wildcards — so
|
||||
`api.anthropic.com.evil.example` and `https://api.anthropic.com@evil.example`
|
||||
do not match `api.anthropic.com`.
|
||||
|
||||
#### `x-headroom-base-url` is refused when an outbound proxy is in the way
|
||||
|
||||
Because the destination is chosen by the client, the proxy validates it by
|
||||
resolving the hostname and rejecting non-public addresses — and then dials the
|
||||
address it validated, rather than resolving the name a second time at connect
|
||||
time. Without that second half, a hostname whose authoritative DNS is
|
||||
attacker-controlled can answer the check with a public address and the
|
||||
connection with `169.254.169.254` or an RFC1918 host (DNS rebinding).
|
||||
|
||||
An outbound HTTP or SOCKS proxy (`HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`) is
|
||||
handed the target *by name* — in the request line, in `CONNECT`, in the SOCKS5
|
||||
address — and resolves it itself, from its own network, after the check ran.
|
||||
Headroom cannot enforce the validated address there, so a request carrying
|
||||
`x-headroom-base-url` that would route through the proxy is **refused** rather
|
||||
than forwarded on a verdict nothing enforces.
|
||||
|
||||
This affects only client-supplied upstreams. Every operator-configured provider
|
||||
URL goes through the proxy exactly as before. If you need a specific endpoint
|
||||
reachable via `x-headroom-base-url` through an outbound proxy, name it in
|
||||
`HEADROOM_ALLOWED_BASE_URLS` — an explicit operator decision to admit that
|
||||
destination by name, which also turns off the resolve-and-pin check for it.
|
||||
|
||||
## SmartCrusher Configuration
|
||||
|
||||
Fine-tune JSON compression behavior:
|
||||
|
||||
@@ -179,6 +179,7 @@ from headroom.proxy.savings_tracker import LITELLM_AVAILABLE
|
||||
from headroom.proxy.semantic_cache import SemanticCache # noqa: F401
|
||||
from headroom.proxy.ssl_context import build_httpx_verify
|
||||
from headroom.proxy.tool_schema_savings_policy import tool_schema_saved_from_tags
|
||||
from headroom.proxy.upstream_pinning import install_upstream_pinning
|
||||
from headroom.proxy.warmup import WarmupRegistry
|
||||
from headroom.proxy.ws_session_registry import WebSocketSessionRegistry
|
||||
from headroom.subscription.base import get_quota_registry, reset_quota_registry
|
||||
@@ -1961,11 +1962,23 @@ class HeadroomProxy(
|
||||
# HEADROOM_TLS_STRICT=0, else httpx's default strict verification.
|
||||
_verify = build_httpx_verify()
|
||||
_http2, _client_kwargs = _provider_httpx_client_options(self.config, _verify)
|
||||
self.http_client = httpx.AsyncClient(http2=_http2, **_client_kwargs)
|
||||
# `install_upstream_pinning` is what makes the SSRF guard's verdict
|
||||
# binding: a caller-supplied upstream that passed `is_safe_upstream_url`
|
||||
# is dialled at the address that was checked, instead of being resolved
|
||||
# a second time here (DNS rebinding). It swaps the pool's DNS layer for
|
||||
# direct routes, and refuses guarded upstreams on routes that cannot
|
||||
# honour a pin at all — a proxy resolves the target itself, on its own
|
||||
# network. Operator-configured upstreams have no pin and are untouched,
|
||||
# so trust_env, limits, HTTP/2 and connection reuse are unchanged.
|
||||
self.http_client = install_upstream_pinning(
|
||||
httpx.AsyncClient(http2=_http2, **_client_kwargs)
|
||||
)
|
||||
# Reuse the primary client when HTTP/2 is already off; otherwise keep a
|
||||
# dedicated HTTP/1.1 client for ChatGPT passthrough.
|
||||
self.http_client_h1 = (
|
||||
self.http_client if not _http2 else httpx.AsyncClient(http2=False, **_client_kwargs)
|
||||
self.http_client
|
||||
if not _http2
|
||||
else install_upstream_pinning(httpx.AsyncClient(http2=False, **_client_kwargs))
|
||||
)
|
||||
logger.info("Headroom Proxy started (version %s)", __version__)
|
||||
logger.info(f"Optimization: {'ENABLED' if self.config.optimize else 'DISABLED'}")
|
||||
|
||||
@@ -14,6 +14,22 @@ Policy:
|
||||
only their exact normalized origin. Because that is an explicit operator
|
||||
choice, allowlisted destinations may point at internal/on-prem endpoints.
|
||||
|
||||
Answering "is this safe?" is only half the job: the answer has to survive to the
|
||||
socket. Every accepted destination therefore has the addresses it was judged on
|
||||
recorded against *the request that asked* (see :func:`guarded_pin`), and the
|
||||
proxy's HTTP client dials one of *those* instead of resolving the name a second
|
||||
time — ``headroom/proxy/upstream_pinning.py`` is the connect-time half.
|
||||
|
||||
The record is a :mod:`contextvars` scope, not a process-global cache, and that
|
||||
distinction is the safety property rather than a tidiness one. A hostname-keyed
|
||||
cache can only answer "is there a pin for this name right now?", so an absent
|
||||
one is indistinguishable from a name that was never guarded — which means a
|
||||
connection for a guarded destination whose pin is missing falls back to ordinary
|
||||
DNS, the exact re-resolution this module exists to prevent. A scope answers the
|
||||
question that actually matters, "was *this* request's destination guarded?", so
|
||||
a guarded destination with no usable pin is a denial. It also stops one caller's
|
||||
pin from being visible to another caller's connection.
|
||||
|
||||
This module intentionally depends only on the standard library so it is safe to
|
||||
import from any handler without risking an import cycle.
|
||||
"""
|
||||
@@ -24,8 +40,11 @@ import asyncio
|
||||
import ipaddress
|
||||
import os
|
||||
import socket
|
||||
import time
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from concurrent.futures import TimeoutError as _FutureTimeout
|
||||
from contextvars import ContextVar
|
||||
from dataclasses import dataclass
|
||||
from urllib.parse import urlparse
|
||||
|
||||
ALLOWED_BASE_URLS_ENV = "HEADROOM_ALLOWED_BASE_URLS"
|
||||
@@ -54,6 +73,137 @@ def _resolve_timeout_seconds() -> float:
|
||||
|
||||
_SAFE_SCHEMES = {"http", "https", "ws", "wss"}
|
||||
|
||||
# Pinning the addresses a check was made against.
|
||||
#
|
||||
# `socket.getaddrinfo` below and the HTTP client's own lookup at connect time
|
||||
# are two separate resolutions of the same name, and an attacker who controls
|
||||
# its authoritative DNS answers them differently: a public address for the
|
||||
# check, 169.254.169.254 (or an RFC1918 host) for the connection. That is DNS
|
||||
# rebinding, and it makes a verdict based only on the first answer advisory --
|
||||
# the socket never sees it. So the addresses that were actually judged are
|
||||
# recorded for the connection to dial.
|
||||
#
|
||||
# The record lives in a contextvar, which scopes it to the request that made the
|
||||
# check: a task inherits its creating context, so the pin an `async def` route
|
||||
# writes is the pin its own upstream call reads, and no other caller's. The
|
||||
# entry is also the *marker* that this destination went through the guard --
|
||||
# see `guarded_pin`.
|
||||
#
|
||||
# The TTL bounds how stale a judged address may be by the time a socket opens.
|
||||
# It is short on purpose and deliberately not configurable: a pin only has to
|
||||
# bridge a check and the connection it authorises, while an operator who
|
||||
# stretched it to hours would be authorising a connection on a DNS answer from
|
||||
# hours ago. Running out is NOT a fallback to ordinary resolution -- the whole
|
||||
# point is that the destination is known to be caller-supplied, so the
|
||||
# connect-time layer denies rather than resolving the attacker's name again.
|
||||
_PIN_TTL_SECONDS = 60.0
|
||||
# Hostnames are caller-supplied, so a scope is bounded even though it dies with
|
||||
# the request that owns it. Overflow rejects the destination rather than leaving
|
||||
# it unpinned, since an unpinned entry is one the connect layer would have to
|
||||
# resolve itself. A request guards one or two upstreams; this is not reachable
|
||||
# by any legitimate route.
|
||||
_PIN_MAX_ENTRIES = 64
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class UpstreamPin:
|
||||
"""The addresses one guarded destination was judged on, and when they lapse.
|
||||
|
||||
An instance existing at all is the statement "this request had this host
|
||||
validated by the guard". ``addresses`` is what the check accepted;
|
||||
``expires_at`` is a :func:`time.monotonic` deadline past which the answer is
|
||||
too old to authorise a socket.
|
||||
"""
|
||||
|
||||
addresses: tuple[str, ...]
|
||||
expires_at: float
|
||||
|
||||
def is_expired(self, now: float | None = None) -> bool:
|
||||
return (time.monotonic() if now is None else now) >= self.expires_at
|
||||
|
||||
|
||||
_PIN_SCOPE: ContextVar[dict[str, UpstreamPin] | None] = ContextVar(
|
||||
"headroom_upstream_pin_scope", default=None
|
||||
)
|
||||
|
||||
|
||||
def begin_pin_scope() -> dict[str, UpstreamPin]:
|
||||
"""Create (or return) this context's pin scope. Returns the live mapping.
|
||||
|
||||
Call this on the thread/task that will *make* the request, before handing
|
||||
the check off anywhere else. A contextvar ``set`` inside a worker thread
|
||||
lands in that thread's copy of the context and is lost when it returns,
|
||||
so :func:`is_safe_upstream_url_async` establishes the mapping here and then
|
||||
mutates it from the worker -- the mapping object is shared, the binding is
|
||||
not.
|
||||
"""
|
||||
scope = _PIN_SCOPE.get()
|
||||
if scope is None:
|
||||
scope = {}
|
||||
_PIN_SCOPE.set(scope)
|
||||
return scope
|
||||
|
||||
|
||||
def _record_validated_addresses(
|
||||
host: str,
|
||||
addresses: tuple[str, ...],
|
||||
scope: dict[str, UpstreamPin] | None = None,
|
||||
) -> bool:
|
||||
"""Remember what ``host`` was judged on, for the connection that follows.
|
||||
|
||||
Returns False when the scope is full, which the caller turns into a
|
||||
rejection: a destination we cannot pin is one the connection would resolve
|
||||
for itself.
|
||||
"""
|
||||
if not host or not addresses:
|
||||
return False
|
||||
if scope is None:
|
||||
scope = begin_pin_scope()
|
||||
if host not in scope and len(scope) >= _PIN_MAX_ENTRIES:
|
||||
return False
|
||||
scope[host] = UpstreamPin(addresses, time.monotonic() + _PIN_TTL_SECONDS)
|
||||
return True
|
||||
|
||||
|
||||
def guarded_pin(host: str) -> UpstreamPin | None:
|
||||
"""Return this request's pin for ``host``, expired or not, else ``None``.
|
||||
|
||||
``None`` means "the current request never had this destination checked" --
|
||||
a configured provider upstream, a proxy endpoint, an allowlisted host --
|
||||
and such a destination resolves normally, exactly as it always has.
|
||||
|
||||
A returned pin means the opposite, and it says so even when it has expired.
|
||||
That is the whole reason expiry does not delete the entry: the connect-time
|
||||
layer has to be able to tell "never guarded" from "guarded, but the judged
|
||||
addresses are too old to use", because only the first of those is safe to
|
||||
resolve by name.
|
||||
"""
|
||||
scope = _PIN_SCOPE.get()
|
||||
if not scope:
|
||||
return None
|
||||
key = (host or "").strip().lower()
|
||||
if not key:
|
||||
return None
|
||||
return scope.get(key)
|
||||
|
||||
|
||||
def validated_addresses(host: str) -> tuple[str, ...] | None:
|
||||
"""Return the still-valid addresses this request's check accepted for ``host``.
|
||||
|
||||
``None`` covers both "never guarded here" and "guarded but lapsed", so it is
|
||||
not a safe basis for a connect decision -- use :func:`guarded_pin`, which
|
||||
keeps those apart. Retained for callers that only want the addresses.
|
||||
"""
|
||||
pin = guarded_pin(host)
|
||||
if pin is None or pin.is_expired():
|
||||
return None
|
||||
return pin.addresses
|
||||
|
||||
|
||||
def clear_validated_addresses() -> None:
|
||||
"""Drop this context's pins. For tests, and between reused contexts."""
|
||||
_PIN_SCOPE.set(None)
|
||||
|
||||
|
||||
def _allowlisted_destinations() -> tuple[set[str], set[tuple[str, str, int]]] | None:
|
||||
raw = os.environ.get(ALLOWED_BASE_URLS_ENV)
|
||||
@@ -132,12 +282,23 @@ def _is_internal_address(ip: str) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def is_safe_upstream_url(url: str) -> bool:
|
||||
def is_safe_upstream_url(url: str, *, scope: dict[str, UpstreamPin] | None = None) -> bool:
|
||||
"""Return True if ``url`` is a safe client-chosen upstream destination.
|
||||
|
||||
In allowlist mode only allowlisted hosts pass. Otherwise the host is
|
||||
resolved and rejected if any resolved address is internal/metadata, which
|
||||
also catches DNS names that point at private space.
|
||||
|
||||
A destination accepted on its addresses also has them pinned against the
|
||||
calling request, so the connection that follows cannot be re-pointed by a
|
||||
second DNS answer. Allowlist mode pins nothing: it admits a host by name
|
||||
without resolving it at all, precisely so split-horizon and on-prem
|
||||
endpoints -- whose addresses are the operator's business, and may
|
||||
legitimately move or round-robin -- keep working.
|
||||
|
||||
``scope`` is the pin mapping to write into; it defaults to the calling
|
||||
context's. :func:`is_safe_upstream_url_async` passes its caller's mapping
|
||||
explicitly because the check itself runs on a worker thread.
|
||||
"""
|
||||
parsed = urlparse((url or "").strip())
|
||||
if parsed.scheme.lower() not in _SAFE_SCHEMES:
|
||||
@@ -169,7 +330,20 @@ def is_safe_upstream_url(url: str) -> bool:
|
||||
# A lookup that overruns the budget is treated the same way.
|
||||
# Operators can explicitly allowlist split-horizon/internal endpoints.
|
||||
return False
|
||||
return all(not _is_internal_address(str(info[4][0])) for info in infos)
|
||||
addresses = tuple(dict.fromkeys(str(info[4][0]) for info in infos))
|
||||
if any(_is_internal_address(address) for address in addresses):
|
||||
return False
|
||||
# Every address in this answer passed, so pin them all: the connection may
|
||||
# dial any one of them and still be dialling something this check accepted.
|
||||
# Keeping the whole set (in resolver order) rather than a single winner
|
||||
# leaves the connection somewhere to go when the first address is an AAAA on
|
||||
# a host with no IPv6 route -- the fallback the OS resolver would otherwise
|
||||
# have done for us.
|
||||
#
|
||||
# A destination that cannot be pinned is rejected rather than allowed
|
||||
# through unpinned: an unpinned client-supplied host is one the connection
|
||||
# would have to resolve for itself, which is the hole this closes.
|
||||
return _record_validated_addresses(host.lower(), addresses, scope)
|
||||
|
||||
|
||||
async def is_safe_upstream_url_async(url: str) -> bool:
|
||||
@@ -177,5 +351,12 @@ async def is_safe_upstream_url_async(url: str) -> bool:
|
||||
|
||||
Same policy; the blocking resolution runs off the loop so a hostile or
|
||||
slow-resolving hostname cannot stall unrelated in-flight requests.
|
||||
|
||||
The pin scope is opened *here*, on the caller's task, and handed to the
|
||||
worker to fill in. Letting the worker open it instead would bind the
|
||||
contextvar in the thread's throwaway copy of the context, so the pin would
|
||||
vanish before the request it authorises ever reached a socket -- and a
|
||||
guarded destination with no pin is a denied one.
|
||||
"""
|
||||
return await asyncio.to_thread(is_safe_upstream_url, url)
|
||||
scope = begin_pin_scope()
|
||||
return await asyncio.to_thread(is_safe_upstream_url, url, scope=scope)
|
||||
|
||||
@@ -0,0 +1,296 @@
|
||||
"""Connect-time half of the SSRF guard: dial the address that was validated.
|
||||
|
||||
:mod:`headroom.proxy.upstream_guard` resolves a caller-supplied upstream and
|
||||
judges the answer -- and then hands the *hostname* to httpx, which resolves it
|
||||
again when it opens the socket. Two resolutions mean two chances to answer, and
|
||||
an attacker who controls the name's authoritative DNS takes both: a public
|
||||
address for the check, ``169.254.169.254`` for the connection. The verdict was
|
||||
never wrong; it just never reached the socket (DNS rebinding / TOCTOU).
|
||||
|
||||
This module carries it there. It replaces the connection pool's network backend
|
||||
with one that substitutes a validated address for the hostname at ``connect_tcp``
|
||||
time, and *only* that: the request still carries the hostname in its URL, so
|
||||
|
||||
* httpx still sends ``Host: <hostname>`` -- an upstream that routes by Host
|
||||
(every gateway, every multi-tenant provider) still routes correctly; and
|
||||
* httpcore still passes the hostname to ``start_tls`` as ``server_hostname``,
|
||||
so SNI and certificate hostname verification are untouched.
|
||||
|
||||
That is the whole reason the substitution happens down at the socket instead of
|
||||
by rewriting the URL to an IP literal. A rewritten URL takes the Host header and
|
||||
the certificate identity down with it -- the upstream misroutes, and the
|
||||
handshake either fails or has to be weakened to an unverified one to work at
|
||||
all. It would also collapse two hostnames sharing an address into one pooled
|
||||
origin, letting a connection whose certificate was checked for one host serve
|
||||
requests for the other.
|
||||
|
||||
What "guarded" means here
|
||||
-------------------------
|
||||
The guard records its verdict in a contextvar scope owned by the request that
|
||||
asked (``upstream_guard.guarded_pin``), so this layer can tell three states
|
||||
apart rather than two:
|
||||
|
||||
* **no pin** -- the current request never had this destination checked. Every
|
||||
operator-configured provider upstream (Anthropic, OpenAI, Gemini, Bedrock,
|
||||
Copilot, LiteLLM, a Kong gateway), every allowlisted host, and the proxy
|
||||
endpoint itself land here, and they resolve exactly as they always have.
|
||||
* **a live pin** -- dial one of the addresses the check accepted.
|
||||
* **a pin whose addresses have lapsed** -- *deny*. This is the state a
|
||||
hostname-keyed cache could not express: there, an aged-out pin was simply an
|
||||
absent one, so a guarded request that waited out the TTL (pool saturation,
|
||||
connection limits, a slow upstream ahead of it in the queue) fell back to
|
||||
resolving the attacker's name a second time -- reopening the exact hole.
|
||||
Denying costs a caller-supplied upstream a request it can immediately retry,
|
||||
which re-checks and re-pins; the alternative costs the guarantee.
|
||||
|
||||
Proxies are refused, not silently unpinned
|
||||
------------------------------------------
|
||||
With ``HTTP_PROXY``/``HTTPS_PROXY``/``ALL_PROXY`` or a SOCKS proxy configured,
|
||||
the pool dials the *proxy*, and the target hostname travels onward inside the
|
||||
protocol -- in the request line for a forwarding HTTP proxy, in ``CONNECT`` for
|
||||
a tunnelling one, in the SOCKS5 address for SOCKS. Every one of those addresses
|
||||
the target *by name*, and the proxy resolves it, on the proxy's network, well
|
||||
after the guard ran. Pinning the proxy's own address does nothing about that, so
|
||||
a caller-supplied upstream routed through a proxy is refused outright with
|
||||
:class:`UnpinnableUpstreamError` rather than forwarded on an unenforced verdict.
|
||||
|
||||
Only guarded destinations are refused. Operator-configured upstreams have no pin
|
||||
and are unaffected, so proxied deployments keep working exactly as before unless
|
||||
they also accept ``x-headroom-base-url`` -- in which case the refusal is the
|
||||
honest answer, and ``HEADROOM_ALLOWED_BASE_URLS`` is the supported way to admit
|
||||
specific internal endpoints through a proxy.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterable, Iterator
|
||||
from typing import Any
|
||||
|
||||
import httpcore
|
||||
import httpx
|
||||
|
||||
from headroom.proxy.upstream_guard import guarded_pin
|
||||
|
||||
|
||||
class UnpinnableUpstreamError(RuntimeError):
|
||||
"""A guarded upstream could not be dialled at an address the guard accepted.
|
||||
|
||||
Deliberately not an ``httpcore`` connect error: those are retried, fallen
|
||||
back from, and mapped into httpx's transport errors, and none of that is
|
||||
right for a refusal. httpx's exception mapping re-raises types it does not
|
||||
know, so this surfaces to the caller as itself.
|
||||
"""
|
||||
|
||||
|
||||
def _refusal(host: str, reason: str) -> UnpinnableUpstreamError:
|
||||
return UnpinnableUpstreamError(
|
||||
f"refusing to connect to the client-supplied upstream {host!r}: {reason}. "
|
||||
"The SSRF guard accepted this destination on addresses it resolved, and "
|
||||
"those addresses cannot be enforced on this connection, so the name "
|
||||
"would be resolved a second time and could answer differently (DNS "
|
||||
"rebinding). Retry the request, or allowlist the endpoint with "
|
||||
"HEADROOM_ALLOWED_BASE_URLS — see headroom/proxy/upstream_pinning.py."
|
||||
)
|
||||
|
||||
|
||||
class PinnedAddressBackend(httpcore.AsyncNetworkBackend):
|
||||
"""Network backend that dials a validated address in place of a pinned name.
|
||||
|
||||
Everything else is delegated untouched, including the ``connect_tcp``
|
||||
keyword arguments -- ``local_address`` and ``socket_options`` are how
|
||||
operators bind egress to a chosen interface, so dropping them here would
|
||||
quietly change which source address the proxy connects from.
|
||||
"""
|
||||
|
||||
def __init__(self, inner: httpcore.AsyncNetworkBackend) -> None:
|
||||
self._inner = inner
|
||||
|
||||
async def connect_tcp(
|
||||
self,
|
||||
host: str,
|
||||
port: int,
|
||||
timeout: float | None = None,
|
||||
local_address: str | None = None,
|
||||
socket_options: Iterable[Any] | None = None,
|
||||
) -> httpcore.AsyncNetworkStream:
|
||||
# httpcore hands us the origin host as `str`; be tolerant anyway, since
|
||||
# a bytes host would silently miss every pin and fail open.
|
||||
name = host.decode("ascii", "ignore") if isinstance(host, bytes) else host
|
||||
pin = guarded_pin(name)
|
||||
|
||||
async def dial(target: str) -> httpcore.AsyncNetworkStream:
|
||||
return await self._inner.connect_tcp(
|
||||
target,
|
||||
port,
|
||||
timeout=timeout,
|
||||
local_address=local_address,
|
||||
socket_options=socket_options,
|
||||
)
|
||||
|
||||
if pin is None:
|
||||
# This request never had this destination checked -- an operator
|
||||
# upstream, an allowlisted host, the proxy itself. Resolve as before.
|
||||
return await dial(host)
|
||||
|
||||
if pin.is_expired():
|
||||
# A guarded destination whose judged addresses are too old to
|
||||
# authorise a socket. Resolving the name again is what this module
|
||||
# exists to prevent, so the connection is refused instead.
|
||||
raise _refusal(name, "the addresses it was validated on have expired")
|
||||
|
||||
# Try each validated address in turn, in the resolver's own order. A
|
||||
# later one is reached only when an earlier one could not be connected
|
||||
# to at all -- the answer is unreachable on this host's network, e.g. an
|
||||
# AAAA record where there is no IPv6 route. That is the fallback the OS
|
||||
# resolver would have made had we handed it the name, and it costs
|
||||
# nothing in safety because every address here passed the same check.
|
||||
# Only connect failures fall through: once a socket is open, whatever
|
||||
# happens on it belongs to the caller. The last address is dialled
|
||||
# outside the loop so its failure propagates as itself.
|
||||
addresses = pin.addresses
|
||||
for address in addresses[:-1]:
|
||||
try:
|
||||
return await dial(address)
|
||||
except (httpcore.ConnectError, httpcore.ConnectTimeout):
|
||||
continue
|
||||
return await dial(addresses[-1])
|
||||
|
||||
async def connect_unix_socket(
|
||||
self,
|
||||
path: str,
|
||||
timeout: float | None = None,
|
||||
socket_options: Iterable[Any] | None = None,
|
||||
) -> httpcore.AsyncNetworkStream:
|
||||
return await self._inner.connect_unix_socket(
|
||||
path, timeout=timeout, socket_options=socket_options
|
||||
)
|
||||
|
||||
async def sleep(self, seconds: float) -> None:
|
||||
await self._inner.sleep(seconds)
|
||||
|
||||
|
||||
class GuardedUpstreamRefusingTransport(httpx.AsyncBaseTransport):
|
||||
"""Refuses guarded destinations on a transport that cannot pin them.
|
||||
|
||||
Wraps proxy transports, and any transport whose connection pool this module
|
||||
does not recognise. The check runs here rather than in the network backend
|
||||
because this is the last layer that still knows the *target* -- below it a
|
||||
proxied connection only ever mentions the proxy's own host.
|
||||
|
||||
``handle_async_request`` runs on the task that issued the request, which is
|
||||
the task whose contextvar scope holds the guard's verdict.
|
||||
"""
|
||||
|
||||
def __init__(self, inner: httpx.AsyncBaseTransport, reason: str) -> None:
|
||||
self._inner = inner
|
||||
self._reason = reason
|
||||
|
||||
async def handle_async_request(self, request: httpx.Request) -> httpx.Response:
|
||||
host = request.url.host
|
||||
if guarded_pin(host) is not None:
|
||||
raise _refusal(host, self._reason)
|
||||
return await self._inner.handle_async_request(request)
|
||||
|
||||
async def __aenter__(self) -> GuardedUpstreamRefusingTransport:
|
||||
await self._inner.__aenter__()
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *exc: Any) -> None:
|
||||
await self._inner.__aexit__(*exc)
|
||||
|
||||
async def aclose(self) -> None:
|
||||
await self._inner.aclose()
|
||||
|
||||
|
||||
_PROXY_POOLS: tuple[type, ...] = tuple(
|
||||
pool
|
||||
for pool in (
|
||||
getattr(httpcore, "AsyncHTTPProxy", None),
|
||||
getattr(httpcore, "AsyncSOCKSProxy", None),
|
||||
)
|
||||
if pool is not None
|
||||
)
|
||||
|
||||
|
||||
def _transport_slots(client: httpx.AsyncClient) -> Iterator[tuple[Any, Any]]:
|
||||
"""Yield ``(key, transport)`` for every transport ``client`` may dial through.
|
||||
|
||||
The key is ``None`` for the primary transport and a ``URLPattern`` for a
|
||||
mounted one. Mounted transports matter as much as the primary: httpx builds
|
||||
them for ``HTTPS_PROXY``/``NO_PROXY``-style environments, and a request
|
||||
routed to one of those would otherwise leave through an unhooked path.
|
||||
"""
|
||||
yield None, getattr(client, "_transport", None)
|
||||
yield from getattr(client, "_mounts", {}).items()
|
||||
|
||||
|
||||
def _install(transport: Any) -> tuple[Any, bool]:
|
||||
"""Return ``(transport to use, whether it pins)`` for one transport slot."""
|
||||
if transport is None: # a mount that defers to the primary transport
|
||||
return None, False
|
||||
if isinstance(transport, GuardedUpstreamRefusingTransport): # already installed
|
||||
return transport, False
|
||||
pool = getattr(transport, "_pool", None)
|
||||
if pool is None:
|
||||
return (
|
||||
GuardedUpstreamRefusingTransport(
|
||||
transport, "this transport exposes no connection pool to pin through"
|
||||
),
|
||||
False,
|
||||
)
|
||||
if _PROXY_POOLS and isinstance(pool, _PROXY_POOLS):
|
||||
# The proxy resolves the target itself, from its own network, after the
|
||||
# guard ran. Pinning the proxy's address would look like a fix and be
|
||||
# none, so guarded destinations are refused on this route instead.
|
||||
return (
|
||||
GuardedUpstreamRefusingTransport(
|
||||
transport,
|
||||
"it routes through a proxy, which resolves the target hostname "
|
||||
"itself and cannot be told which address the guard accepted",
|
||||
),
|
||||
False,
|
||||
)
|
||||
backend = getattr(pool, "_network_backend", None)
|
||||
if backend is None:
|
||||
return (
|
||||
GuardedUpstreamRefusingTransport(
|
||||
transport, "its connection pool exposes no network backend to pin through"
|
||||
),
|
||||
False,
|
||||
)
|
||||
if not isinstance(backend, PinnedAddressBackend):
|
||||
pool._network_backend = PinnedAddressBackend(backend)
|
||||
return transport, True
|
||||
|
||||
|
||||
def install_upstream_pinning(client: httpx.AsyncClient) -> httpx.AsyncClient:
|
||||
"""Make ``client`` dial validated addresses for guarded hosts. Returns it.
|
||||
|
||||
Transports that cannot honour a pin -- proxy transports above all -- are
|
||||
wrapped so a guarded destination routed through them is refused rather than
|
||||
forwarded on a verdict nothing enforces.
|
||||
|
||||
Raises ``RuntimeError`` when no direct transport could be pinned at all,
|
||||
which means httpx/httpcore no longer expose the connection pool this hooks.
|
||||
That is fail-closed on purpose: the alternative is a proxy that starts
|
||||
happily and silently re-opens the rebinding hole, and the versions involved
|
||||
are locked in ``uv.lock``, so it can only fire on a deliberate dependency
|
||||
change -- exactly when someone should be looking.
|
||||
"""
|
||||
pinned = 0
|
||||
mounts = getattr(client, "_mounts", None)
|
||||
for key, transport in _transport_slots(client):
|
||||
replacement, did_pin = _install(transport)
|
||||
pinned += did_pin
|
||||
if replacement is not transport:
|
||||
if key is None:
|
||||
client._transport = replacement
|
||||
elif mounts is not None:
|
||||
mounts[key] = replacement
|
||||
if not pinned:
|
||||
raise RuntimeError(
|
||||
"cannot pin validated upstream addresses: this httpx/httpcore "
|
||||
"exposes no direct connection pool to hook. Refusing to run "
|
||||
"unpinned — see headroom/proxy/upstream_pinning.py."
|
||||
)
|
||||
return client
|
||||
+10
-1
@@ -295,7 +295,11 @@ dev = [
|
||||
"litellm>=1.96.2,<2.0", # see core deps note
|
||||
"fastapi>=0.100.0",
|
||||
"uvicorn>=0.23.0,<1.0",
|
||||
"httpx[http2]>=0.24.0",
|
||||
# socks: test_upstream_guard proves a guarded upstream is refused on EVERY
|
||||
# supported proxy transport. Without the extra, httpcore substitutes a stub
|
||||
# AsyncSOCKSProxy that cannot be constructed, and the SOCKS case degrades to
|
||||
# asserting on a type rather than a real pool.
|
||||
"httpx[http2,socks]>=0.24.0",
|
||||
"websockets>=13.0",
|
||||
"opentelemetry-sdk>=1.24.0",
|
||||
"opentelemetry-exporter-otlp-proto-http>=1.24.0",
|
||||
@@ -308,6 +312,11 @@ dev = [
|
||||
"openpyxl>=3.1.0", # exercises spreadsheet_ingest (.xlsx) in the test suite
|
||||
"xlrd>=2.0.1", # exercises spreadsheet_ingest (.xls) in the test suite
|
||||
"xlwt>=1.3.0", # writes the legacy .xls fixtures for spreadsheet_ingest tests
|
||||
# test_upstream_guard builds a self-signed cert to prove the pinned connection
|
||||
# still presents the original hostname for SNI and still verifies the cert.
|
||||
# The floor in the security-pins block below is a *version* constraint on a
|
||||
# transitive dep, not a guarantee it is installed -- CI had neither.
|
||||
"cryptography>=50.0.0",
|
||||
"respx>=0.20.0", # HTTP mock transport for passthrough handler tests
|
||||
]
|
||||
# All optional dependencies (everything you need).
|
||||
|
||||
@@ -5,11 +5,16 @@ All cases use IP literals or ``localhost`` so no external network is required.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import re
|
||||
import socket
|
||||
import ssl
|
||||
import threading
|
||||
from http.server import BaseHTTPRequestHandler, HTTPServer
|
||||
from pathlib import Path
|
||||
|
||||
import httpcore
|
||||
import httpx
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
@@ -152,7 +157,7 @@ class _InternalService:
|
||||
return f"http://127.0.0.1:{self.port}"
|
||||
|
||||
|
||||
def _app(): # noqa: ANN202
|
||||
def _app(**overrides: object): # noqa: ANN202
|
||||
return create_app(
|
||||
ProxyConfig(
|
||||
host="127.0.0.1",
|
||||
@@ -160,6 +165,7 @@ def _app(): # noqa: ANN202
|
||||
optimize=False,
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
**overrides, # type: ignore[arg-type]
|
||||
)
|
||||
)
|
||||
|
||||
@@ -298,3 +304,576 @@ async def test_async_guard_matches_the_sync_policy() -> None:
|
||||
assert await is_safe_upstream_url_async("http://127.0.0.1/") is False
|
||||
assert await is_safe_upstream_url_async("http://169.254.169.254/") is False
|
||||
assert await is_safe_upstream_url_async("https://8.8.8.8/v1") is True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# DNS rebinding (A-4).
|
||||
#
|
||||
# The guard used to resolve the hostname, judge the answer, and then hand the
|
||||
# *name* to httpx -- which resolves it a second time when it opens the socket.
|
||||
# An attacker who controls the authoritative DNS for that name simply answers
|
||||
# the two lookups differently: a public address for the check, an internal one
|
||||
# for the connection. Every check above still passes while the proxy talks to
|
||||
# 127.0.0.1, so the tests here assert on where the socket actually went rather
|
||||
# than on what `is_safe_upstream_url` returned.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
# A path no route claims, so it lands on the catch-all passthrough -- the sink
|
||||
# that forwards to `x-headroom-base-url` verbatim once the guard has cleared it.
|
||||
_PROBE_PATH = "/v1/rebind-probe"
|
||||
|
||||
|
||||
class _RebindingResolver:
|
||||
"""A `getaddrinfo` that answers the first lookup differently from the rest.
|
||||
|
||||
Only the attacker-controlled hostname is intercepted; every other name is
|
||||
delegated to the real resolver, so patching this in globally -- which is
|
||||
what it takes to reach asyncio's own resolution, not just the guard's --
|
||||
leaves the rest of the process alone.
|
||||
"""
|
||||
|
||||
def __init__(self, hostname: str, first: str, then: str) -> None:
|
||||
self.hostname = hostname
|
||||
self.first = first
|
||||
self.then = then
|
||||
self.calls = 0
|
||||
self._real = socket.getaddrinfo
|
||||
self._lock = threading.Lock()
|
||||
|
||||
def __call__(self, host: object, port: object, *args: object, **kwargs: object) -> list:
|
||||
# anyio hands the resolver an ASCII/IDNA-encoded name, the guard a str.
|
||||
name = host.decode("ascii") if isinstance(host, bytes) else host
|
||||
if name != self.hostname:
|
||||
return self._real(host, port, *args, **kwargs) # type: ignore[arg-type]
|
||||
with self._lock:
|
||||
self.calls += 1
|
||||
address = self.first if self.calls == 1 else self.then
|
||||
return [
|
||||
(
|
||||
socket.AF_INET,
|
||||
socket.SOCK_STREAM,
|
||||
socket.IPPROTO_TCP,
|
||||
"",
|
||||
(address, port or 0),
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def test_rebinding_after_the_check_cannot_move_the_forward_to_loopback(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""The validated address is the one dialled, not whatever DNS says later.
|
||||
|
||||
The first lookup (the guard's) answers with a public address so the URL
|
||||
passes validation; every later one answers with the internal service.
|
||||
Before the pin, httpx re-resolved and delivered both the request and the
|
||||
response to loopback.
|
||||
"""
|
||||
resolver = _RebindingResolver("rebind.example", first="8.8.8.8", then="127.0.0.1")
|
||||
monkeypatch.setattr(socket, "getaddrinfo", resolver)
|
||||
|
||||
with (
|
||||
_InternalService() as internal,
|
||||
TestClient(_app(connect_timeout_seconds=1)) as client,
|
||||
):
|
||||
try:
|
||||
response = client.post(
|
||||
_PROBE_PATH,
|
||||
headers={"x-headroom-base-url": f"http://rebind.example:{internal.port}"},
|
||||
json={"query": "x"},
|
||||
)
|
||||
status, text = response.status_code, response.text
|
||||
except Exception as exc: # noqa: BLE001 - an upstream failure is a pass here
|
||||
status, text = 0, repr(exc)
|
||||
|
||||
assert resolver.calls >= 1, "the guard never resolved the hostname"
|
||||
assert status != 400, f"the guard rejected the URL; the pin was never exercised: {text}"
|
||||
assert status != 404, f"the route did not forward at all: {text}"
|
||||
assert internal.hits == [], "the connection followed the rebound answer to loopback"
|
||||
assert "internal-only" not in text
|
||||
|
||||
|
||||
def _self_signed_cert(tmp_path: Path, hostname: str) -> tuple[str, str]:
|
||||
"""Write a self-signed leaf for ``hostname``; return (cert path, key path).
|
||||
|
||||
The certificate is its own issuer, so the same file doubles as the trust
|
||||
bundle the proxy is pointed at through ``SSL_CERT_FILE``.
|
||||
"""
|
||||
import datetime
|
||||
|
||||
from cryptography import x509
|
||||
from cryptography.hazmat.primitives import hashes, serialization
|
||||
from cryptography.hazmat.primitives.asymmetric import ec
|
||||
from cryptography.x509.oid import NameOID
|
||||
|
||||
key = ec.generate_private_key(ec.SECP256R1())
|
||||
subject = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, hostname)])
|
||||
now = datetime.datetime.now(datetime.UTC)
|
||||
certificate = (
|
||||
x509.CertificateBuilder()
|
||||
.subject_name(subject)
|
||||
.issuer_name(subject)
|
||||
.public_key(key.public_key())
|
||||
.serial_number(x509.random_serial_number())
|
||||
.not_valid_before(now - datetime.timedelta(days=1))
|
||||
.not_valid_after(now + datetime.timedelta(days=1))
|
||||
.add_extension(x509.SubjectAlternativeName([x509.DNSName(hostname)]), critical=False)
|
||||
.add_extension(x509.BasicConstraints(ca=True, path_length=None), critical=True)
|
||||
.sign(key, hashes.SHA256())
|
||||
)
|
||||
cert_path = tmp_path / "cert.pem"
|
||||
key_path = tmp_path / "key.pem"
|
||||
cert_path.write_bytes(certificate.public_bytes(serialization.Encoding.PEM))
|
||||
key_path.write_bytes(
|
||||
key.private_bytes(
|
||||
serialization.Encoding.PEM,
|
||||
serialization.PrivateFormat.PKCS8,
|
||||
serialization.NoEncryption(),
|
||||
)
|
||||
)
|
||||
return str(cert_path), str(key_path)
|
||||
|
||||
|
||||
class _TLSUpstream:
|
||||
"""An HTTPS upstream on loopback that records the SNI and Host it was sent."""
|
||||
|
||||
def __init__(self, cert_pem: str, key_pem: str) -> None:
|
||||
self.sni: list[str | None] = []
|
||||
self.host_headers: list[str] = []
|
||||
host_headers = self.host_headers
|
||||
|
||||
class _Handler(BaseHTTPRequestHandler):
|
||||
def do_POST(self) -> None: # noqa: N802
|
||||
host_headers.append(self.headers.get("Host", ""))
|
||||
body = b'{"ok":true}'
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
do_GET = do_POST # noqa: N815
|
||||
|
||||
def log_message(self, *args: object) -> None:
|
||||
return
|
||||
|
||||
context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
|
||||
# Nothing here is about protocol versions -- the assertions are on SNI,
|
||||
# the Host header, and certificate verification -- but a bare
|
||||
# `PROTOCOL_TLS_SERVER` context nominally admits TLS 1.0/1.1, which is a
|
||||
# finding in its own right. Pinning the floor keeps the test's subject
|
||||
# unchanged and says out loud which versions it means.
|
||||
context.minimum_version = ssl.TLSVersion.TLSv1_2
|
||||
context.load_cert_chain(cert_pem, key_pem)
|
||||
context.sni_callback = lambda _sock, name, _ctx: self.sni.append(name)
|
||||
self._server = HTTPServer(("127.0.0.1", 0), _Handler)
|
||||
self._server.socket = context.wrap_socket(self._server.socket, server_side=True)
|
||||
self.port = int(self._server.server_address[1])
|
||||
self._thread = threading.Thread(target=self._server.serve_forever, daemon=True)
|
||||
|
||||
def __enter__(self) -> _TLSUpstream:
|
||||
self._thread.start()
|
||||
return self
|
||||
|
||||
def __exit__(self, *exc: object) -> None:
|
||||
self._server.shutdown()
|
||||
|
||||
|
||||
def _tls_rebinding_app(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
resolver: _RebindingResolver,
|
||||
ca_pem: str,
|
||||
): # noqa: ANN202
|
||||
"""Point the proxy at ``ca_pem`` and at ``resolver``'s answers.
|
||||
|
||||
``_is_internal_address`` is neutered so a loopback answer passes
|
||||
validation: the subject of these two tests is what happens *after* an
|
||||
address is accepted, and faking it this way keeps the upstream on 127.0.0.1
|
||||
instead of requiring real egress to a genuinely public address.
|
||||
"""
|
||||
from headroom.proxy import upstream_guard
|
||||
|
||||
monkeypatch.setattr(socket, "getaddrinfo", resolver)
|
||||
monkeypatch.setattr(upstream_guard, "_is_internal_address", lambda _ip: False)
|
||||
monkeypatch.setenv("SSL_CERT_FILE", ca_pem)
|
||||
return _app(connect_timeout_seconds=1)
|
||||
|
||||
|
||||
def test_pinned_connection_presents_the_original_hostname_to_tls(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
"""Dialling an IP literal must not cost us SNI, the Host header, or the cert.
|
||||
|
||||
The upstream's certificate names `pinned.example` only, so a handshake
|
||||
verified against the pinned address -- or one that relaxed verification to
|
||||
make pinning work at all -- fails here. And because the second DNS answer
|
||||
points off-box, a request arriving at all is proof the first, validated
|
||||
answer is what was dialled.
|
||||
"""
|
||||
cert_pem, key_pem = _self_signed_cert(tmp_path, "pinned.example")
|
||||
resolver = _RebindingResolver("pinned.example", first="127.0.0.1", then="8.8.8.8")
|
||||
|
||||
with _TLSUpstream(cert_pem, key_pem) as upstream:
|
||||
app = _tls_rebinding_app(monkeypatch, resolver, cert_pem)
|
||||
with TestClient(app) as client:
|
||||
try:
|
||||
status = client.post(
|
||||
_PROBE_PATH,
|
||||
headers={"x-headroom-base-url": f"https://pinned.example:{upstream.port}"},
|
||||
json={"query": "x"},
|
||||
).status_code
|
||||
except Exception as exc: # noqa: BLE001
|
||||
pytest.fail(f"pinned TLS request never reached the upstream: {exc!r}")
|
||||
|
||||
assert status == 200, "the pinned upstream was not reached"
|
||||
assert upstream.sni == ["pinned.example"], f"wrong SNI: {upstream.sni}"
|
||||
assert upstream.host_headers, "no request arrived at the pinned upstream"
|
||||
assert upstream.host_headers[0].startswith("pinned.example"), (
|
||||
f"Host header was rewritten to the pinned address: {upstream.host_headers[0]}"
|
||||
)
|
||||
|
||||
|
||||
def test_pinning_does_not_disable_certificate_hostname_verification(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
"""The mirror of the test above: a cert for the wrong name must still fail.
|
||||
|
||||
Identical setup with one difference -- the upstream's certificate names
|
||||
`other.example` while the caller asked for `pinned.example`. Pinning
|
||||
implemented by rewriting the URL to the address without carrying the
|
||||
hostname into the handshake, or by loosening `verify`, would pass this
|
||||
request straight through.
|
||||
"""
|
||||
cert_pem, key_pem = _self_signed_cert(tmp_path, "other.example")
|
||||
resolver = _RebindingResolver("pinned.example", first="127.0.0.1", then="8.8.8.8")
|
||||
|
||||
with _TLSUpstream(cert_pem, key_pem) as upstream:
|
||||
app = _tls_rebinding_app(monkeypatch, resolver, cert_pem)
|
||||
with TestClient(app) as client:
|
||||
try:
|
||||
status = client.post(
|
||||
_PROBE_PATH,
|
||||
headers={"x-headroom-base-url": f"https://pinned.example:{upstream.port}"},
|
||||
json={"query": "x"},
|
||||
).status_code
|
||||
except Exception: # noqa: BLE001 - a refused handshake is the pass
|
||||
status = 0
|
||||
|
||||
assert status != 200, "a certificate for the wrong hostname was accepted"
|
||||
assert upstream.host_headers == [], "the request reached a mis-named upstream"
|
||||
|
||||
|
||||
def test_a_check_pins_every_address_it_accepted(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""All of them, in resolver order -- see the IPv6-fallback note on the pin."""
|
||||
from headroom.proxy import upstream_guard
|
||||
|
||||
def two_answers(*args: object, **kwargs: object) -> list[object]:
|
||||
return [
|
||||
(None, None, None, None, ("8.8.8.8", 443)),
|
||||
(None, None, None, None, ("1.1.1.1", 443)),
|
||||
]
|
||||
|
||||
monkeypatch.setattr(socket, "getaddrinfo", two_answers)
|
||||
upstream_guard.clear_validated_addresses()
|
||||
|
||||
assert is_safe_upstream_url("https://Multi.Homed.Example/v1") is True
|
||||
# Hostnames are case-insensitive; the connection will ask in lower case.
|
||||
assert upstream_guard.validated_addresses("multi.homed.example") == ("8.8.8.8", "1.1.1.1")
|
||||
|
||||
|
||||
def test_a_rejected_or_allowlisted_destination_pins_nothing(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Only an address that was judged may be pinned.
|
||||
|
||||
A rejected destination is never dialled, and an allowlisted one is admitted
|
||||
by name without resolving at all -- pinning either would be recording a
|
||||
verdict that was not reached.
|
||||
"""
|
||||
from headroom.proxy import upstream_guard
|
||||
|
||||
def internal_answer(*args: object, **kwargs: object) -> list[object]:
|
||||
return [(None, None, None, None, ("10.1.2.3", 443))]
|
||||
|
||||
monkeypatch.setattr(socket, "getaddrinfo", internal_answer)
|
||||
upstream_guard.clear_validated_addresses()
|
||||
|
||||
assert is_safe_upstream_url("https://rejected.example/v1") is False
|
||||
assert upstream_guard.validated_addresses("rejected.example") is None
|
||||
|
||||
monkeypatch.setenv("HEADROOM_ALLOWED_BASE_URLS", "gateway.internal")
|
||||
assert is_safe_upstream_url("https://gateway.internal/v1") is True
|
||||
assert upstream_guard.validated_addresses("gateway.internal") is None
|
||||
|
||||
|
||||
def test_an_expired_pin_still_marks_the_destination_as_guarded(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Lapsed addresses, but the record that the guard ran must survive.
|
||||
|
||||
An aged-out pin that simply vanished would be indistinguishable from a host
|
||||
that was never checked, and "never checked" resolves by name. Keeping the
|
||||
record is what lets the connect path deny instead.
|
||||
"""
|
||||
from headroom.proxy import upstream_guard
|
||||
|
||||
def public_answer(*args: object, **kwargs: object) -> list[object]:
|
||||
return [(None, None, None, None, ("8.8.8.8", 443))]
|
||||
|
||||
monkeypatch.setattr(socket, "getaddrinfo", public_answer)
|
||||
monkeypatch.setattr(upstream_guard, "_PIN_TTL_SECONDS", 0.0)
|
||||
upstream_guard.clear_validated_addresses()
|
||||
|
||||
assert is_safe_upstream_url("https://briefly.example/v1") is True
|
||||
# No usable addresses...
|
||||
assert upstream_guard.validated_addresses("briefly.example") is None
|
||||
# ...but still, unmistakably, a destination this request had guarded.
|
||||
pin = upstream_guard.guarded_pin("briefly.example")
|
||||
assert pin is not None, "an expired pin must not decay into 'never checked'"
|
||||
assert pin.is_expired() is True
|
||||
assert pin.addresses == ("8.8.8.8",)
|
||||
|
||||
|
||||
def test_a_pin_is_scoped_to_the_request_that_asked(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""One caller's pin is not visible to another caller's connection.
|
||||
|
||||
The check runs inside a copied context, so the pin it writes stays there.
|
||||
A process-global store keyed by hostname leaked it to every other in-flight
|
||||
request instead, which is both a cross-tenant leak and the reason expiry
|
||||
could not be made to mean anything.
|
||||
"""
|
||||
import contextvars
|
||||
|
||||
from headroom.proxy import upstream_guard
|
||||
|
||||
def public_answer(*args: object, **kwargs: object) -> list[object]:
|
||||
return [(None, None, None, None, ("8.8.8.8", 443))]
|
||||
|
||||
monkeypatch.setattr(socket, "getaddrinfo", public_answer)
|
||||
upstream_guard.clear_validated_addresses()
|
||||
|
||||
other_request = contextvars.copy_context()
|
||||
assert other_request.run(is_safe_upstream_url, "https://tenant-a.example/v1") is True
|
||||
assert other_request.run(upstream_guard.validated_addresses, "tenant-a.example") == ("8.8.8.8",)
|
||||
assert upstream_guard.guarded_pin("tenant-a.example") is None, (
|
||||
"another request's pin was visible here"
|
||||
)
|
||||
|
||||
|
||||
def test_the_pin_scope_is_bounded_and_overflow_rejects(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Hostnames are caller-supplied, so a scope cannot grow without limit.
|
||||
|
||||
Overflow is a rejection rather than an eviction: an evicted host would pass
|
||||
validation while leaving the connection to resolve the name itself, which is
|
||||
the failure mode this whole module exists to remove.
|
||||
"""
|
||||
from headroom.proxy import upstream_guard
|
||||
|
||||
def public_answer(*args: object, **kwargs: object) -> list[object]:
|
||||
return [(None, None, None, None, ("8.8.8.8", 443))]
|
||||
|
||||
monkeypatch.setattr(socket, "getaddrinfo", public_answer)
|
||||
monkeypatch.setattr(upstream_guard, "_PIN_MAX_ENTRIES", 4)
|
||||
upstream_guard.clear_validated_addresses()
|
||||
|
||||
accepted = [index for index in range(20) if is_safe_upstream_url(f"https://f{index}.example/")]
|
||||
|
||||
assert accepted == [0, 1, 2, 3]
|
||||
assert len(upstream_guard.begin_pin_scope()) == 4
|
||||
assert upstream_guard.validated_addresses("f0.example") == ("8.8.8.8",)
|
||||
assert upstream_guard.guarded_pin("f19.example") is None
|
||||
# Re-checking a host already in the scope refreshes it rather than
|
||||
# overflowing, so a retry of the same upstream is never spuriously refused.
|
||||
assert is_safe_upstream_url("https://f0.example/") is True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The connect-time half: what the backend does with each of the three states it
|
||||
# can find a destination in -- unguarded, pinned, guarded-but-lapsed -- and what
|
||||
# happens when the route cannot honour a pin at all.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class _RecordingBackend(httpcore.AsyncNetworkBackend):
|
||||
"""Inner backend that records what it was asked to dial and never dials it."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.targets: list[str] = []
|
||||
|
||||
async def connect_tcp(self, host: str, port: int, *args: object, **kwargs: object) -> object:
|
||||
self.targets.append(host)
|
||||
return object()
|
||||
|
||||
|
||||
async def _connect(host: str) -> tuple[_RecordingBackend, object | None, Exception | None]:
|
||||
from headroom.proxy.upstream_pinning import PinnedAddressBackend
|
||||
|
||||
inner = _RecordingBackend()
|
||||
backend = PinnedAddressBackend(inner)
|
||||
try:
|
||||
return inner, await backend.connect_tcp(host, 443), None
|
||||
except Exception as exc: # noqa: BLE001 - the refusal is the subject
|
||||
return inner, None, exc
|
||||
|
||||
|
||||
async def test_a_pin_expiring_before_the_socket_opens_denies_the_connection(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""The reviewer's case: a guarded request outliving the pin must be refused.
|
||||
|
||||
A request can legitimately sit between the guard's verdict and its socket
|
||||
for longer than the TTL -- pool saturation, connection limits, a slow
|
||||
upstream ahead of it in the queue. When it finally connects, the hostname is
|
||||
still attacker-controlled, so re-resolving it is exactly the rebinding this
|
||||
module prevents. Denial is the only safe answer; the caller can retry, which
|
||||
re-checks and re-pins.
|
||||
"""
|
||||
from headroom.proxy import upstream_guard
|
||||
from headroom.proxy.upstream_pinning import UnpinnableUpstreamError
|
||||
|
||||
def public_answer(*args: object, **kwargs: object) -> list[object]:
|
||||
return [(None, None, None, None, ("8.8.8.8", 443))]
|
||||
|
||||
monkeypatch.setattr(socket, "getaddrinfo", public_answer)
|
||||
upstream_guard.clear_validated_addresses()
|
||||
assert is_safe_upstream_url("https://rebind.example/v1") is True
|
||||
|
||||
# The socket is opened while the request waits; wind the pin past its TTL.
|
||||
scope = upstream_guard.begin_pin_scope()
|
||||
lapsed = scope["rebind.example"]
|
||||
scope["rebind.example"] = upstream_guard.UpstreamPin(
|
||||
lapsed.addresses, lapsed.expires_at - upstream_guard._PIN_TTL_SECONDS - 1.0
|
||||
)
|
||||
|
||||
inner, stream, error = await _connect("rebind.example")
|
||||
|
||||
assert stream is None, "an expired pin was treated as an unchecked destination"
|
||||
assert isinstance(error, UnpinnableUpstreamError), f"wrong failure: {error!r}"
|
||||
assert inner.targets == [], (
|
||||
f"the hostname was handed to the resolver a second time: {inner.targets}"
|
||||
)
|
||||
|
||||
|
||||
async def test_a_live_pin_dials_the_address_and_an_unguarded_host_dials_the_name(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""The two states either side of the denial, so the denial is not blanket."""
|
||||
from headroom.proxy import upstream_guard
|
||||
|
||||
def public_answer(*args: object, **kwargs: object) -> list[object]:
|
||||
return [(None, None, None, None, ("8.8.8.8", 443))]
|
||||
|
||||
monkeypatch.setattr(socket, "getaddrinfo", public_answer)
|
||||
upstream_guard.clear_validated_addresses()
|
||||
assert is_safe_upstream_url("https://pinned.example/v1") is True
|
||||
|
||||
inner, stream, error = await _connect("pinned.example")
|
||||
assert error is None and stream is not None
|
||||
assert inner.targets == ["8.8.8.8"], "a live pin did not reach the socket"
|
||||
|
||||
# An operator-configured upstream never passes through the guard, has no
|
||||
# pin, and must keep resolving by name exactly as it always has.
|
||||
inner, stream, error = await _connect("api.anthropic.com")
|
||||
assert error is None and stream is not None
|
||||
assert inner.targets == ["api.anthropic.com"]
|
||||
|
||||
|
||||
def _client_through_proxy(proxy_url: str) -> httpx.AsyncClient:
|
||||
"""A client routing everything through ``proxy_url``, SOCKS included.
|
||||
|
||||
All three transports build a real pool. SOCKS needs httpx's ``socks`` extra
|
||||
(``socksio``), which is in the dev dependencies for exactly this test: without
|
||||
it httpcore substitutes a stub ``AsyncSOCKSProxy`` that cannot be constructed,
|
||||
and the case degrades into asserting on a type instead of exercising the pool
|
||||
the refusal actually has to classify.
|
||||
"""
|
||||
return httpx.AsyncClient(proxy=proxy_url)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("label", "proxy_url"),
|
||||
[
|
||||
("http", "http://proxy.internal:3128"),
|
||||
("https", "https://proxy.internal:3129"),
|
||||
("socks", "socks5://proxy.internal:1080"),
|
||||
],
|
||||
)
|
||||
async def test_a_guarded_upstream_through_a_proxy_is_refused(
|
||||
monkeypatch: pytest.MonkeyPatch, label: str, proxy_url: str
|
||||
) -> None:
|
||||
"""Every supported proxy transport refuses a guarded upstream, loudly.
|
||||
|
||||
A proxy is handed the target *by name* -- in the request line, in CONNECT,
|
||||
in the SOCKS5 address -- and resolves it itself, on its own network, after
|
||||
the guard ran. Pinning the proxy's address does nothing about that, so the
|
||||
honest answer is a refusal rather than a forward on a verdict nothing
|
||||
enforces.
|
||||
"""
|
||||
from headroom.proxy import upstream_guard
|
||||
from headroom.proxy.upstream_pinning import (
|
||||
GuardedUpstreamRefusingTransport,
|
||||
UnpinnableUpstreamError,
|
||||
install_upstream_pinning,
|
||||
)
|
||||
|
||||
def public_answer(*args: object, **kwargs: object) -> list[object]:
|
||||
return [(None, None, None, None, ("8.8.8.8", 443))]
|
||||
|
||||
monkeypatch.setattr(socket, "getaddrinfo", public_answer)
|
||||
upstream_guard.clear_validated_addresses()
|
||||
|
||||
client = install_upstream_pinning(_client_through_proxy(proxy_url))
|
||||
try:
|
||||
transport = client._transport_for_url(httpx.URL("https://rebind.example/v1"))
|
||||
assert isinstance(transport, GuardedUpstreamRefusingTransport), (
|
||||
f"the {label} proxy transport was left able to resolve the target itself"
|
||||
)
|
||||
|
||||
# Unguarded traffic is untouched: it never reaches the refusal at all.
|
||||
assert upstream_guard.guarded_pin("rebind.example") is None
|
||||
|
||||
assert is_safe_upstream_url("https://rebind.example/v1") is True
|
||||
with pytest.raises(UnpinnableUpstreamError) as refused:
|
||||
await transport.handle_async_request(httpx.Request("POST", "https://rebind.example/v1"))
|
||||
finally:
|
||||
# The stubbed SOCKS pool above was never initialised, so it has no
|
||||
# connection state to close.
|
||||
with contextlib.suppress(AttributeError):
|
||||
await client.aclose()
|
||||
|
||||
assert "rebind.example" in str(refused.value)
|
||||
assert "proxy" in str(refused.value)
|
||||
|
||||
|
||||
async def test_a_proxied_client_still_pins_its_direct_transport(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Configuring a proxy for some traffic must not unpin the rest.
|
||||
|
||||
httpx keeps the direct pool as the primary transport and mounts the proxy
|
||||
per URL pattern, so a `NO_PROXY`-style carve-out still leaves through the
|
||||
pinned path -- and the refusal above applies only to what actually routes
|
||||
via the proxy.
|
||||
"""
|
||||
from headroom.proxy import upstream_guard
|
||||
from headroom.proxy.upstream_pinning import (
|
||||
GuardedUpstreamRefusingTransport,
|
||||
PinnedAddressBackend,
|
||||
install_upstream_pinning,
|
||||
)
|
||||
|
||||
monkeypatch.setenv("HTTPS_PROXY", "http://proxy.internal:3128")
|
||||
monkeypatch.setenv("NO_PROXY", "direct.example")
|
||||
upstream_guard.clear_validated_addresses()
|
||||
|
||||
client = install_upstream_pinning(httpx.AsyncClient())
|
||||
try:
|
||||
direct = client._transport_for_url(httpx.URL("https://direct.example/v1"))
|
||||
assert not isinstance(direct, GuardedUpstreamRefusingTransport)
|
||||
assert isinstance(direct._pool._network_backend, PinnedAddressBackend)
|
||||
|
||||
proxied = client._transport_for_url(httpx.URL("https://elsewhere.example/v1"))
|
||||
assert isinstance(proxied, GuardedUpstreamRefusingTransport)
|
||||
finally:
|
||||
await client.aclose()
|
||||
|
||||
@@ -1924,9 +1924,10 @@ crewai = [
|
||||
]
|
||||
dev = [
|
||||
{ name = "anthropic" },
|
||||
{ name = "cryptography" },
|
||||
{ name = "fastapi" },
|
||||
{ name = "hnswlib" },
|
||||
{ name = "httpx", extra = ["http2"] },
|
||||
{ name = "httpx", extra = ["http2", "socks"] },
|
||||
{ name = "langchain-ollama" },
|
||||
{ name = "litellm" },
|
||||
{ name = "mypy" },
|
||||
@@ -2114,6 +2115,7 @@ requires-dist = [
|
||||
{ name = "botocore", extras = ["crt"], marker = "extra == 'bedrock'", specifier = ">=1.41.0" },
|
||||
{ name = "click", specifier = ">=8.3.3" },
|
||||
{ name = "crewai", marker = "extra == 'crewai'", specifier = ">=1.15.21" },
|
||||
{ name = "cryptography", marker = "extra == 'dev'", specifier = ">=50.0.0" },
|
||||
{ name = "datasets", marker = "extra == 'evals'", specifier = ">=5.0.1" },
|
||||
{ name = "datasets", marker = "extra == 'voice-train'", specifier = ">=5.0.1" },
|
||||
{ name = "fastapi", marker = "extra == 'dev'", specifier = ">=0.100.0" },
|
||||
@@ -2127,8 +2129,8 @@ requires-dist = [
|
||||
{ name = "hnswlib", marker = "extra == 'dev'", specifier = ">=0.8.0" },
|
||||
{ name = "hnswlib", marker = "extra == 'vector'", specifier = ">=0.8.0" },
|
||||
{ name = "httpx", marker = "extra == 'mcp'", specifier = ">=0.24.0" },
|
||||
{ name = "httpx", extras = ["http2"], marker = "extra == 'dev'", specifier = ">=0.24.0" },
|
||||
{ name = "httpx", extras = ["http2"], marker = "extra == 'proxy'", specifier = ">=0.24.0" },
|
||||
{ name = "httpx", extras = ["http2", "socks"], marker = "extra == 'dev'", specifier = ">=0.24.0" },
|
||||
{ name = "huggingface-hub", marker = "extra == 'ml'", specifier = ">=1.5.0,<2.0" },
|
||||
{ name = "jinja2", marker = "extra == 'reports'", specifier = ">=3.0.0" },
|
||||
{ name = "langchain-core", marker = "extra == 'langchain'", specifier = ">=1.3.3,<4.0" },
|
||||
@@ -2363,6 +2365,9 @@ wheels = [
|
||||
http2 = [
|
||||
{ name = "h2" },
|
||||
]
|
||||
socks = [
|
||||
{ name = "socksio" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "httpx-sse"
|
||||
@@ -6737,6 +6742,15 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/e9/44/75a9c9421471a6c4805dbf2356f7c181a29c1879239abab1ea2cc8f38b40/sniffio-1.3.1-py3-none-any.whl", hash = "sha256:2f6da418d1f1e0fddd844478f41680e794e6051915791a034ff65e5f100525a2", size = 10235, upload-time = "2024-02-25T23:20:01.196Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "socksio"
|
||||
version = "1.0.0"
|
||||
source = { registry = "https://pypi.org/simple/" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/f8/5c/48a7d9495be3d1c651198fd99dbb6ce190e2274d0f28b9051307bdec6b85/socksio-1.0.0.tar.gz", hash = "sha256:f88beb3da5b5c38b9890469de67d0cb0f9d494b78b106ca1845f96c10b91c4ac", size = 19055, upload-time = "2020-04-17T15:50:34.664Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/37/c3/6eeb6034408dac0fa653d126c9204ade96b819c936e136c5e8a6897eee9c/socksio-1.0.0-py3-none-any.whl", hash = "sha256:95dc1f15f9b34e8d7b16f06d74b8ccf48f609af32ab33c608d08761c5dcbb1f3", size = 12763, upload-time = "2020-04-17T15:50:31.878Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "sqlalchemy"
|
||||
version = "2.0.49"
|
||||
|
||||
Reference in New Issue
Block a user