fix(proxy): key rate limits by peer unless the caller authenticated (#3860)

## Why

The proxy's request and token rate limits could be walked around by any
caller the proxy cannot authenticate (finding 01-F3). Each provider
handler keyed its buckets on a client-supplied header:

- **OpenAI:** an HMAC of the full `Authorization` / `api-key` value,
else one shared `"default"` bucket. Rotating the header gave a fresh
bucket per request, and all callers without a credential shared one
bucket.
- **Anthropic:** the first 16 characters of the key plus the peer. That
is mostly the common `sk-ant-api03-` prefix.
- **Gemini:** the first 20 characters of `x-goog-api-key`, with no peer
at all.

## What changes

- New `headroom/proxy/rate_limit_identity.py` with one rule for the
Anthropic, OpenAI and Gemini handlers. `rate_limit_identity()` returns a
plain string key:
- Every key starts with the peer from `resolve_client_ip`, which honours
`X-Forwarded-For` only from `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS`
peers. IPv6 peers are grouped by `/64`; an IPv4-mapped IPv6 address
counts as the IPv4 peer.
- **Trusted** requests get one bucket per provider credential:
`peer:<addr>|cred:<hmac>`, an HMAC of the whole credential with a
process-local key, so bucket names hold no credential material. A
request is trusted if it presented a valid `HEADROOM_PROXY_TOKEN` or
came directly from loopback. A trusted-gateway peer is trusted to supply
the client address, not to vouch that the caller authenticated, so a
forwarded request without the token is untrusted and keyed by the
forwarded client address. A request from a gateway address that carries
no `X-Forwarded-For` is judged as a direct request, so a loopback
gateway CIDR does not change how direct localhost callers are keyed.
Distinct principals behind one address keep separate limits, as #3364
intended.
- **Untrusted** requests (no token, remote caller) are charged per peer
(`peer:<addr>`). The credential header is ignored.
- The security gate sets `request.state.proxy_authenticated = True` when
a caller presents a valid proxy token.
- The per-handler key code is removed (`_openai_rate_limit_key` and the
Anthropic and Gemini inline keys).

`rate_limiter.py` is unchanged, and so are token-limit semantics.

## Behaviour change

Requests the proxy cannot authenticate (no `HEADROOM_PROXY_TOKEN`,
remote caller) are now limited per client address. Changing the API-key
header no longer gives a new allowance. Authenticated and
direct-loopback callers keep one allowance per provider credential.
Behind a gateway, set `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` so each
forwarded client address gets its own limit; to keep one allowance per
credential there, callers must also send the proxy token.

The limiter is quota smoothing and abuse control for unauthenticated
callers. It is not a security boundary for authenticated callers, and
this PR does not make it one.

## Tests

- `tests/test_proxy/test_rate_limit_identity.py`: untrusted keys ignore
the credential; trusted keys are per credential and include the peer;
the whole credential is used, not a prefix; bucket names carry no
credential material; loopback is trusted; a request relayed by a
trusted-gateway peer (including one on loopback) is not trusted without
the proxy token; with `127.0.0.0/8` configured as a gateway, direct
loopback callers stay per credential, and a relayed request with an
unusable `X-Forwarded-For` still counts as relayed; IPv6 `/64` grouping;
IPv4-mapped addresses.
- `tests/test_proxy_openai_rate_limit_key.py`, end to end through
`create_app` with `--rpm 1`: authenticated and loopback callers with
distinct keys do not share a bucket; an unauthenticated remote caller
who rotates `api-key` or `Authorization`, or drops the credential, gets
429 on the second request; two different unauthenticated peers do not
share a bucket; through a trusted gateway (`10.0.0.2`, fixed
`X-Forwarded-For: 203.0.113.9`) a caller rotating `api-key` gets 429,
distinct forwarded clients do not share, and token-authenticated callers
stay per credential; with `127.0.0.0/8` as the gateway CIDR, two direct
loopback callers with different keys both pass while a caller relayed
through it with a fixed forwarded address gets 429 on the second key.
The unauthenticated cases fail on `main`.
- Existing rate-limiter and TPM suites pass unchanged.

Manual run: `headroom proxy --host 0.0.0.0 --rpm 2` with six requests
from a LAN address, each with a different `Authorization` key. On `main`
all six reach the upstream. On this branch without a proxy token, the
third and later get 429. With the proxy token sent, all six pass, one
bucket per credential.

Not tested: the Rust `headroom-proxy` binary, which has its own limiter.

## Merge order / conflicts

Part of a set. Suggested order: #3852 → #3891 → #3849 → #3860 → #3862 →
#3863. #3852 makes the untrusted case rare by refusing token-less
non-loopback binds; this PR makes it safe when an operator opts out of
that.

Textual conflict: #3862 adds an import next to the one this PR adds in
`handlers/anthropic.py` and `handlers/openai.py`. Whichever merges
second keeps both imports.

Revised 2026-10-01 after self-review: `rate_limiter.py` is back to
`main`; the separate trusted/untrusted pools, per-owner cap and overflow
bucket were removed.
Revised 2026-10-02 after review: gateway membership only drops the
credential from the key when the gateway actually forwarded a client
address, so direct localhost callers keep per-credential buckets under a
loopback gateway CIDR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3860"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Tejas Chopra
2026-10-05 08:26:56 -05:00
committed by GitHub
co-authored by Claude Opus 5.5 JD Davis
parent 3fdf18eded
commit 6fb7cadfe4
8 changed files with 465 additions and 107 deletions
+5 -13
View File
@@ -51,7 +51,6 @@ from headroom.proxy.buffered_ccr_response import (
buffered_ccr_asgi_call,
)
from headroom.proxy.compression_decision import CompressionDecision
from headroom.proxy.forwarded_headers import resolve_client_ip
from headroom.proxy.handlers._debug_dump import _debug_dump_mode, _redact_debug_value
from headroom.proxy.helpers import (
extract_tags,
@@ -66,6 +65,7 @@ from headroom.proxy.model_router import estimate_input_tokens
from headroom.proxy.nonstream_sse_policy import should_recover_sse_reply
from headroom.proxy.outcome import RequestOutcome
from headroom.proxy.output_shaper import shaper_enabled_for, steering_allowed_for
from headroom.proxy.rate_limit_identity import rate_limit_identity
from headroom.proxy.tenant_key import resolve_tenant_key, set_request_tenant_key
from headroom.proxy.thinking_tokens import ThinkingTokens, extract_thinking_tokens
from headroom.utils import format_exception_message
@@ -1318,18 +1318,10 @@ class AnthropicHandlerMixin:
# Rate limiting
if self.rate_limiter:
api_key = headers.get("x-api-key", "")
if not api_key:
auth = headers.get("authorization", "")
if auth.startswith("Bearer "):
api_key = auth[7:]
# Phase F PR-F4: trust ``X-Forwarded-For`` for the rate-limit
# key only when the connecting peer is in
# ``HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS``; otherwise we use
# the direct peer IP and a malicious client cannot rotate
# rate-limit buckets by forging headers.
client_ip = resolve_client_ip(request) or "unknown"
rate_key = f"{api_key[:16]}:{client_ip}" if api_key else client_ip
# One identity rule for every provider: peer-owned, and
# credential-scoped only for proxy-token / direct loopback
# callers (headroom/proxy/rate_limit_identity.py).
rate_key = rate_limit_identity(request, headers)
allowed, wait_seconds = await self.rate_limiter.check_request(rate_key)
if not allowed:
await self.metrics.record_rate_limited(
+4 -2
View File
@@ -24,6 +24,7 @@ from headroom.proxy.compression_decision import CompressionDecision
from headroom.proxy.helpers import COMPRESSION_TIMEOUT_SECONDS, extract_tags
from headroom.proxy.identity import resolve_memory_identity
from headroom.proxy.outcome import RequestOutcome
from headroom.proxy.rate_limit_identity import rate_limit_identity
from headroom.proxy.token_counting import gemini_output_tokens
logger = logging.getLogger("headroom.proxy")
@@ -441,9 +442,10 @@ class GeminiHandlerMixin:
)
memory_decision.apply_to_tags(tags)
# Rate limiting (use Gemini API key)
# Rate limiting: one identity rule for every provider
# (headroom/proxy/rate_limit_identity.py).
if self.rate_limiter:
rate_key = headers.get("x-goog-api-key", "default")[:20]
rate_key = rate_limit_identity(request, headers)
allowed, wait_seconds = await self.rate_limiter.check_request(rate_key)
if not allowed:
await self.metrics.record_rate_limited(provider=provider_name, source="headroom")
+3 -28
View File
@@ -10,12 +10,10 @@ import contextlib
import contextvars
import copy
import hashlib
import hmac
import json
import logging
import os
import re
import secrets
import ssl
import threading
import time
@@ -38,6 +36,7 @@ from headroom.proxy.helpers import (
from headroom.proxy.identity import resolve_memory_identity
from headroom.proxy.loopback_guard import is_loopback_host
from headroom.proxy.modes import is_cache_mode
from headroom.proxy.rate_limit_identity import rate_limit_identity
from headroom.proxy.stage_timer import StageTimer, emit_stage_timings_log
from headroom.proxy.upstream_guard import is_safe_upstream_url
from headroom.proxy.ws_headers import WS_HOP_BY_HOP_HEADERS
@@ -129,30 +128,6 @@ _CCR_HASH_RE = re.compile(
r"(?:Retrieve (?:more|original): hash=|<<ccr:)([a-fA-F0-9]{12,24})(?=[^a-fA-F0-9]|$)"
)
_BARE_CCR_HASH_RE = re.compile(r"[a-fA-F0-9]{12,24}")
_OPENAI_RATE_KEY_SECRET = secrets.token_bytes(32)
def _openai_rate_limit_key(headers: dict[str, str]) -> str:
"""Return the credential identity used by the OpenAI rate limiter.
OpenAI-compatible gateways may authenticate with either a bearer token or
an ``api-key`` header. Deriving an identity from the complete value keeps common
prefixes distinct without retaining recoverable credential material in
bucket keys. The secret is process-local, like the limiter state.
Requests without either retain the existing shared fallback bucket.
"""
authorization = headers.get("authorization")
api_key = headers.get("api-key")
if authorization:
kind, credential = "authorization", authorization
elif api_key:
kind, credential = "api-key", api_key
else:
return "default"
# API credentials are identifiers, not passwords to verify. A process-keyed
# HMAC keeps bucket keys opaque without a password KDF on the request path.
digest = hmac.digest(_OPENAI_RATE_KEY_SECRET, f"{kind}:{credential}".encode(), "sha256").hex()
return f"{kind}:{digest}"
def _response_ccr_hashes(messages: list[dict[str, Any]], markers: list[str]) -> list[str]:
@@ -3874,7 +3849,7 @@ class OpenAIHandlerMixin:
# Rate limiting
if self.rate_limiter:
rate_key = _openai_rate_limit_key(headers)
rate_key = rate_limit_identity(request, headers)
allowed, wait_seconds = await self.rate_limiter.check_request(rate_key)
if not allowed:
await self.metrics.record_rate_limited(
@@ -6219,7 +6194,7 @@ class OpenAIHandlerMixin:
# Rate limiting
if self.rate_limiter:
rate_key = _openai_rate_limit_key(headers)
rate_key = rate_limit_identity(request, headers)
allowed, wait_seconds = await self.rate_limiter.check_request(rate_key)
if not allowed:
await self.metrics.record_rate_limited(provider="openai", source="headroom")
+123
View File
@@ -0,0 +1,123 @@
"""Who a request is charged to, for the proxy's request/token rate limits.
One rule for every provider handler (Anthropic, OpenAI, Gemini). Before this
module each handler keyed its buckets differently, and every rule could be
walked around (VAPT finding 01-F3):
* OpenAI keyed on an HMAC of the full ``Authorization`` / ``api-key`` value, so
a caller rotating that header got a fresh bucket per request; callers with no
credential all shared one ``"default"`` bucket.
* Anthropic keyed on the first 16 characters of the key plus the peer address —
mostly the common ``sk-ant-api03-`` prefix.
* Gemini keyed on the first 20 characters of ``x-goog-api-key`` with no peer.
The rule now:
1. Every identity is **owned by the peer** — ``resolve_client_ip``, which honours
``X-Forwarded-For`` only from ``HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS`` peers.
IPv6 peers are grouped by ``/64`` so rotating addresses inside one
allocation does not rotate buckets.
2. For a **trusted** request the provider credential is part of the bucket, so
distinct principals behind one address (a team gateway, a NAT) do not share a
limit — decision D2, one proxy is shared by several principals. A request is
trusted when it presented the proxy token (the security gate records that on
``request.state``) or came directly from loopback. A trusted-gateway peer is
trusted to report the caller's address, not to vouch that the caller
authenticated, so forwarded requests without the token are untrusted. A
direct request from a gateway address that forwards no client address is
judged like any other direct request.
3. For an **untrusted** request (no token configured and a remote caller) the
credential is ignored: the bucket is the peer. Rotating a header cannot mint
a new bucket for someone the proxy cannot authenticate.
"""
from __future__ import annotations
import hmac
import ipaddress
import secrets
from typing import Any
from headroom.proxy.forwarded_headers import load_trusted_gateway_cidrs, resolve_client_ip
from headroom.proxy.forwarded_policy import peer_is_trusted_gateway
from headroom.proxy.loopback_guard import is_loopback_host
# Set by the security gate when a caller presented a valid HEADROOM_PROXY_TOKEN.
PROXY_AUTHENTICATED_STATE_ATTR = "proxy_authenticated"
# Checked in order; the first present header is the caller's provider credential.
_CREDENTIAL_HEADERS = ("authorization", "x-api-key", "api-key", "x-goog-api-key")
# Process-local key: bucket names never contain recoverable credential material.
_IDENTITY_SECRET = secrets.token_bytes(32)
def _peer_group(ip: str) -> str:
"""Normalise a peer address; IPv6 is grouped by its /64."""
if not ip:
return "peer:unknown"
try:
addr = ipaddress.ip_address(ip)
except ValueError:
return f"peer:{ip[:64]}"
if isinstance(addr, ipaddress.IPv6Address):
if addr.ipv4_mapped is not None:
return f"peer:{addr.ipv4_mapped}"
network = ipaddress.IPv6Network((int(addr) >> 64 << 64, 64))
return f"peer:{network}"
return f"peer:{addr}"
def _credential(headers: Any) -> str | None:
for name in _CREDENTIAL_HEADERS:
try:
value = headers.get(name)
except AttributeError:
return None
if value:
return f"{name}:{value}"
return None
def _relayed_by_trusted_gateway(request: Any) -> bool:
"""True when a trusted-gateway peer forwarded a client address for this request.
Any ``X-Forwarded-For`` value counts, even one the forwarded-header policy
cannot use, so a relayed request is never mistaken for a direct one.
"""
client = getattr(request, "client", None)
host = getattr(client, "host", None) if client is not None else None
if not host or not peer_is_trusted_gateway(host, load_trusted_gateway_cidrs()):
return False
headers = getattr(request, "headers", None)
return bool(headers is not None and headers.get("x-forwarded-for"))
def is_trusted_request(request: Any) -> bool:
"""True when the caller presented the proxy token, or connected directly from loopback.
A request relayed by a trusted gateway (even one on loopback) is keyed by
the forwarded client address unless it presented the proxy token: the
gateway vouches for that address, not for the caller's credential. A direct
request from a loopback gateway address, with no forwarded client address,
is still a direct loopback caller.
"""
state = getattr(request, "state", None)
if state is not None and getattr(state, PROXY_AUTHENTICATED_STATE_ATTR, False):
return True
if _relayed_by_trusted_gateway(request):
return False
client = getattr(request, "client", None)
host = getattr(client, "host", None) if client is not None else None
return bool(host and is_loopback_host(host))
def rate_limit_identity(request: Any, headers: Any = None) -> str:
"""Return the bucket this request is charged to. See the module docstring."""
headers = headers if headers is not None else getattr(request, "headers", {})
peer = _peer_group(resolve_client_ip(request) or "")
credential = _credential(headers) if is_trusted_request(request) else None
if credential is None:
return peer
digest = hmac.digest(_IDENTITY_SECRET, credential.encode("utf-8", "replace"), "sha256")
return f"{peer}|cred:{digest.hex()[:32]}"
+4
View File
@@ -4058,6 +4058,10 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
rejection = JSONResponse(status_code=401, content={"error": "unauthorized"})
_apply_security_headers(rejection)
return rejection
# The caller proved it holds the proxy token. The rate limiter
# keys authenticated callers per credential and everyone else
# per peer (headroom/proxy/rate_limit_identity.py).
request.state.proxy_authenticated = True
# The credential has done its job; take it off the request so no handler
# can forward it upstream. Runs on the exempt paths too, since a loopback
@@ -0,0 +1,113 @@
"""Rate-limit identity (VAPT 01-F3)."""
from __future__ import annotations
from types import SimpleNamespace
import pytest
from headroom.proxy.rate_limit_identity import rate_limit_identity
def _request(peer: str, *, authenticated: bool = False, headers=None):
return SimpleNamespace(
client=SimpleNamespace(host=peer),
headers=headers or {},
state=SimpleNamespace(proxy_authenticated=authenticated),
)
# ── identity ────────────────────────────────────────────────────────────────
def test_untrusted_identity_ignores_the_credential() -> None:
a = rate_limit_identity(_request("203.0.113.9", headers={"authorization": "Bearer k1"}))
b = rate_limit_identity(_request("203.0.113.9", headers={"x-api-key": "k2"}))
c = rate_limit_identity(_request("203.0.113.9"))
assert a == b == c == "peer:203.0.113.9"
def test_trusted_identity_is_per_credential_and_owned_by_the_peer() -> None:
a = rate_limit_identity(
_request("203.0.113.9", authenticated=True, headers={"authorization": "Bearer k1"})
)
b = rate_limit_identity(
_request("203.0.113.9", authenticated=True, headers={"authorization": "Bearer k2"})
)
assert a != b
assert a.startswith("peer:203.0.113.9|cred:") and b.startswith("peer:203.0.113.9|cred:")
@pytest.mark.parametrize(
"headers",
[
{"authorization": "Bearer sk-ant-api03-SAME-PREFIX-A"},
{"x-api-key": "sk-ant-api03-SAME-PREFIX-A"},
{"x-goog-api-key": "AIzaSyD-SAME-PREFIX-A"},
],
)
def test_trusted_identity_uses_the_whole_credential_not_a_prefix(headers) -> None:
other = {k: v[:-1] + "B" for k, v in headers.items()}
a = rate_limit_identity(_request("127.0.0.1", headers=headers))
b = rate_limit_identity(_request("127.0.0.1", headers=other))
assert a != b
def test_bucket_names_carry_no_credential_material() -> None:
ident = rate_limit_identity(
_request("127.0.0.1", headers={"authorization": "Bearer sk-live-secret-value"})
)
assert "sk-live" not in ident
assert "secret" not in ident
def test_loopback_is_trusted_without_a_token() -> None:
keyed = {"authorization": "Bearer k1"}
assert rate_limit_identity(_request("127.0.0.1", headers=keyed)) != rate_limit_identity(
_request("127.0.0.1")
)
def test_trusted_gateway_peer_alone_does_not_authenticate_the_caller(monkeypatch) -> None:
monkeypatch.setenv("HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS", "10.20.0.0/16")
keyed = {"authorization": "Bearer k1"}
assert rate_limit_identity(_request("10.20.3.4", headers=keyed)) == "peer:10.20.3.4"
assert "|cred:" in rate_limit_identity(_request("10.20.3.4", authenticated=True, headers=keyed))
def test_loopback_gateway_does_not_authenticate_forwarded_callers(monkeypatch) -> None:
monkeypatch.setenv("HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS", "127.0.0.0/8")
keyed = {"authorization": "Bearer k1", "x-forwarded-for": "203.0.113.9"}
assert rate_limit_identity(_request("127.0.0.1", headers=keyed)) == "peer:203.0.113.9"
def test_loopback_gateway_config_keeps_direct_loopback_callers_per_credential(
monkeypatch,
) -> None:
"""A loopback gateway CIDR only matters for requests that forward a client address."""
monkeypatch.setenv("HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS", "127.0.0.0/8")
a = rate_limit_identity(_request("127.0.0.1", headers={"authorization": "Bearer k1"}))
b = rate_limit_identity(_request("127.0.0.1", headers={"authorization": "Bearer k2"}))
assert a != b
assert a.startswith("peer:127.0.0.1|cred:") and b.startswith("peer:127.0.0.1|cred:")
def test_unusable_forwarded_address_still_counts_as_relayed(monkeypatch) -> None:
"""A gateway-relayed request with an empty leftmost hop is not a direct caller."""
monkeypatch.setenv("HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS", "127.0.0.0/8")
keyed = {"authorization": "Bearer k1", "x-forwarded-for": ", 203.0.113.9"}
assert "|cred:" not in rate_limit_identity(_request("127.0.0.1", headers=keyed))
def test_ipv6_peers_are_grouped_by_slash_64() -> None:
a = rate_limit_identity(_request("2001:db8:1:2::1"))
b = rate_limit_identity(_request("2001:db8:1:2:ffff:ffff:ffff:ffff"))
c = rate_limit_identity(_request("2001:db8:1:3::1"))
assert a == b
assert a != c
def test_ipv4_mapped_ipv6_is_the_ipv4_peer() -> None:
assert rate_limit_identity(_request("::ffff:203.0.113.9")) == rate_limit_identity(
_request("203.0.113.9")
)
+7 -2
View File
@@ -20,7 +20,6 @@ from opentelemetry.sdk.metrics.export import InMemoryMetricReader
from headroom.backends.base import BackendResponse
from headroom.observability import HeadroomOtelMetrics, reset_otel_metrics, set_otel_metrics
from headroom.proxy.handlers.openai import _openai_rate_limit_key
from headroom.proxy.server import ProxyConfig, create_app
from tests.test_observability_metrics import _collect_metrics
@@ -1287,6 +1286,12 @@ def test_openai_chat_custom_base_flood_cannot_grow_the_provider_set(
# The flood hosts are unresolvable on purpose; without this stub the SSRF
# guard would drop the override and relabel the flood as "openai".
monkeypatch.setattr("headroom.proxy.handlers.openai.is_safe_upstream_url", lambda url: True)
# Bucket by the raw credential so the stub limiter below can refuse just
# ``limited_auth``; the identity rule itself is covered by its own tests.
monkeypatch.setattr(
"headroom.proxy.handlers.openai.rate_limit_identity",
lambda request, headers: headers.get("authorization", ""),
)
# Isolated sinks: a fresh lifetime store and workspace, so the exact
# counts below are this test's traffic and nothing reaches ~/.headroom.
monkeypatch.setenv("HEADROOM_SAVINGS_PATH", str(tmp_path / "savings.json"))
@@ -1298,7 +1303,7 @@ def test_openai_chat_custom_base_flood_cannot_grow_the_provider_set(
"""Headroom's own limiter, refusing only ``limited_auth``."""
async def check_request(self, key: str = "default") -> tuple[bool, float]:
return key != _openai_rate_limit_key({"authorization": limited_auth}), 1.0
return key != limited_auth, 1.0
async def check_tokens(self, key: str, tokens: int) -> tuple[bool, float]:
return True, 0.0
+206 -62
View File
@@ -1,4 +1,15 @@
"""Regression coverage for OpenAI credential-scoped rate limiting (#3364)."""
"""Credential-scoped rate limiting (#3364) under the shared identity rule (01-F3).
#3364 made distinct OpenAI credentials use distinct buckets. VAPT finding 01-F3
showed the other side of that: a caller the proxy cannot authenticate could
rotate the credential header to mint a fresh bucket per request. Both hold now:
* an **authenticated** caller (proxy token) or a direct loopback caller gets
one bucket per provider credential — #3364's behaviour;
* an **unauthenticated remote** caller is charged per peer, whatever credential
header it sends — including one relayed by a trusted gateway, which is
charged per forwarded client address.
"""
from __future__ import annotations
@@ -11,34 +22,37 @@ pytest.importorskip("fastapi")
from fastapi.testclient import TestClient # noqa: E402
from headroom.proxy.handlers.openai import _openai_rate_limit_key # noqa: E402
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
TOKEN = "hrpt_rate_limit_test_token_0123456789"
def test_openai_rate_limit_key_accepts_api_key_header() -> None:
assert _openai_rate_limit_key({"api-key": "gateway-key-A"}) != _openai_rate_limit_key(
{"api-key": "gateway-key-B"}
)
CHAT = (
"/v1/chat/completions",
{"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "hello"}], "stream": False},
{
"id": "chatcmpl-test",
"object": "chat.completion",
"choices": [
{"index": 0, "message": {"role": "assistant", "content": "ok"}, "finish_reason": "stop"}
],
"usage": {"prompt_tokens": 3, "completion_tokens": 1, "total_tokens": 4},
},
)
RESPONSES = (
"/v1/responses",
{"model": "gpt-4o-mini", "input": "hello", "stream": False},
{
"id": "resp-test",
"object": "response",
"status": "completed",
"output": [],
"usage": {"input_tokens": 3, "output_tokens": 1, "total_tokens": 4},
},
)
def test_openai_rate_limit_key_uses_complete_credential() -> None:
shared_prefix = "a" * 20
assert _openai_rate_limit_key({"api-key": f"{shared_prefix}-A"}) != _openai_rate_limit_key(
{"api-key": f"{shared_prefix}-B"}
)
def test_openai_rate_limit_key_prefers_authorization() -> None:
headers = {"authorization": "Bearer primary", "api-key": "secondary"}
assert _openai_rate_limit_key(headers) == _openai_rate_limit_key(
{"authorization": "Bearer primary"}
)
@pytest.mark.parametrize("endpoint", ["chat", "responses"])
def test_distinct_api_keys_do_not_share_openai_rate_limit_bucket(monkeypatch, endpoint) -> None:
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
config = ProxyConfig(
def _config(**kwargs) -> ProxyConfig:
return ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=True,
@@ -51,47 +65,177 @@ def test_distinct_api_keys_do_not_share_openai_rate_limit_bucket(monkeypatch, en
ccr_context_tracking=False,
image_optimize=False,
retry_enabled=False,
**kwargs,
)
if endpoint == "chat":
path = "/v1/chat/completions"
body = {
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": "hello"}],
"stream": False,
}
upstream_body = {
"id": "chatcmpl-test",
"object": "chat.completion",
"choices": [
{
"index": 0,
"message": {"role": "assistant", "content": "ok"},
"finish_reason": "stop",
}
],
"usage": {"prompt_tokens": 3, "completion_tokens": 1, "total_tokens": 4},
}
else:
path = "/v1/responses"
body = {"model": "gpt-4o-mini", "input": "hello", "stream": False}
upstream_body = {
"id": "resp-test",
"object": "response",
"status": "completed",
"output": [],
"usage": {"input_tokens": 3, "output_tokens": 1, "total_tokens": 4},
}
with TestClient(create_app(config)) as client:
proxy = client.app.state.proxy
proxy._retry_request = AsyncMock(
side_effect=lambda *args, **kwargs: httpx.Response(200, json=upstream_body)
def _two_requests(client, path, body, first_headers, second_headers):
proxy = client.app.state.proxy
return (
proxy,
client.post(path, headers=first_headers, json=body),
client.post(path, headers=second_headers, json=body),
)
@pytest.mark.parametrize("endpoint", [CHAT, RESPONSES], ids=["chat", "responses"])
def test_authenticated_callers_get_one_bucket_per_credential(monkeypatch, endpoint) -> None:
"""#3364: distinct provider credentials from an authenticated caller do not share."""
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
path, body, upstream_body = endpoint
with TestClient(create_app(_config(proxy_token=TOKEN)), client=("203.0.113.9", 1)) as client:
client.app.state.proxy._retry_request = AsyncMock(
side_effect=lambda *a, **k: httpx.Response(200, json=upstream_body)
)
proxy, first, second = _two_requests(
client,
path,
body,
{"x-headroom-proxy-token": TOKEN, "api-key": "gateway-key-A"},
{"x-headroom-proxy-token": TOKEN, "api-key": "gateway-key-B"},
)
assert first.status_code == 200
assert second.status_code == 200
assert proxy._retry_request.await_count == 2
first = client.post(path, headers={"api-key": "gateway-key-A"}, json=body)
second = client.post(path, headers={"api-key": "gateway-key-B"}, json=body)
assert first.status_code == 200
assert second.status_code == 200
assert proxy._retry_request.await_count == 2
@pytest.mark.parametrize("endpoint", [CHAT, RESPONSES], ids=["chat", "responses"])
def test_loopback_callers_get_one_bucket_per_credential(monkeypatch, endpoint) -> None:
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
path, body, upstream_body = endpoint
with TestClient(create_app(_config()), client=("127.0.0.1", 1)) as client:
client.app.state.proxy._retry_request = AsyncMock(
side_effect=lambda *a, **k: httpx.Response(200, json=upstream_body)
)
proxy, first, second = _two_requests(
client, path, body, {"api-key": "key-A"}, {"api-key": "key-B"}
)
assert (first.status_code, second.status_code) == (200, 200)
assert proxy._retry_request.await_count == 2
@pytest.mark.parametrize("endpoint", [CHAT, RESPONSES], ids=["chat", "responses"])
@pytest.mark.parametrize(
"rotating",
[
({"api-key": "rotated-1"}, {"api-key": "rotated-2"}),
({"authorization": "Bearer rotated-1"}, {"authorization": "Bearer rotated-2"}),
({"api-key": "rotated-1"}, {}),
],
ids=["api-key", "authorization", "credential-then-none"],
)
def test_unauthenticated_remote_caller_cannot_rotate_credentials_into_new_buckets(
monkeypatch, endpoint, rotating
) -> None:
"""01-F3: with no proxy token, a remote caller is charged per peer."""
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
path, body, upstream_body = endpoint
with TestClient(create_app(_config()), client=("203.0.113.9", 1)) as client:
client.app.state.proxy._retry_request = AsyncMock(
side_effect=lambda *a, **k: httpx.Response(200, json=upstream_body)
)
proxy, first, second = _two_requests(client, path, body, *rotating)
assert first.status_code == 200
assert second.status_code == 429
assert proxy._retry_request.await_count == 1
def test_unauthenticated_remote_callers_on_different_peers_do_not_share(monkeypatch) -> None:
"""The fix is per peer, not one global bucket for everyone unauthenticated."""
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
path, body, upstream_body = CHAT
app = create_app(_config())
statuses = []
for peer in ("203.0.113.9", "198.51.100.4"):
with TestClient(app, client=(peer, 1)) as client:
client.app.state.proxy._retry_request = AsyncMock(
side_effect=lambda *a, **k: httpx.Response(200, json=upstream_body)
)
statuses.append(client.post(path, json=body).status_code)
assert statuses == [200, 200]
@pytest.mark.parametrize("endpoint", [CHAT, RESPONSES], ids=["chat", "responses"])
def test_unauthenticated_caller_behind_trusted_gateway_cannot_rotate_credentials(
monkeypatch, endpoint
) -> None:
"""A trusted gateway vouches for the client address, not for the caller."""
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
monkeypatch.setenv("HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS", "10.0.0.0/8")
path, body, upstream_body = endpoint
forwarded = {"x-forwarded-for": "203.0.113.9"}
with TestClient(create_app(_config()), client=("10.0.0.2", 1)) as client:
client.app.state.proxy._retry_request = AsyncMock(
side_effect=lambda *a, **k: httpx.Response(200, json=upstream_body)
)
proxy, first, second = _two_requests(
client,
path,
body,
{**forwarded, "api-key": "rotated-A"},
{**forwarded, "api-key": "rotated-B"},
)
assert first.status_code == 200
assert second.status_code == 429
assert proxy._retry_request.await_count == 1
def test_trusted_gateway_keys_unauthenticated_callers_by_forwarded_address(monkeypatch) -> None:
"""Distinct forwarded clients behind one gateway keep distinct buckets."""
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
monkeypatch.setenv("HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS", "10.0.0.0/8")
path, body, upstream_body = CHAT
with TestClient(create_app(_config()), client=("10.0.0.2", 1)) as client:
client.app.state.proxy._retry_request = AsyncMock(
side_effect=lambda *a, **k: httpx.Response(200, json=upstream_body)
)
statuses = [
client.post(path, headers={"x-forwarded-for": ip}, json=body).status_code
for ip in ("203.0.113.9", "198.51.100.4")
]
assert statuses == [200, 200]
def test_authenticated_caller_behind_trusted_gateway_gets_one_bucket_per_credential(
monkeypatch,
) -> None:
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
monkeypatch.setenv("HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS", "10.0.0.0/8")
path, body, upstream_body = CHAT
headers = {"x-forwarded-for": "203.0.113.9", "x-headroom-proxy-token": TOKEN}
with TestClient(create_app(_config(proxy_token=TOKEN)), client=("10.0.0.2", 1)) as client:
client.app.state.proxy._retry_request = AsyncMock(
side_effect=lambda *a, **k: httpx.Response(200, json=upstream_body)
)
_, first, second = _two_requests(
client, path, body, {**headers, "api-key": "key-A"}, {**headers, "api-key": "key-B"}
)
assert (first.status_code, second.status_code) == (200, 200)
@pytest.mark.parametrize("endpoint", [CHAT, RESPONSES], ids=["chat", "responses"])
def test_loopback_gateway_config_direct_and_forwarded_callers(monkeypatch, endpoint) -> None:
"""One loopback gateway CIDR: direct callers stay per credential, relayed ones per client.
Direct loopback callers (no ``X-Forwarded-For``) with distinct keys keep
distinct buckets. A caller relayed through the same loopback gateway with a
fixed forwarded address cannot rotate keys into a fresh bucket.
"""
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
monkeypatch.setenv("HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS", "127.0.0.0/8")
path, body, upstream_body = endpoint
forwarded = {"x-forwarded-for": "203.0.113.9"}
with TestClient(create_app(_config()), client=("127.0.0.1", 1)) as client:
client.app.state.proxy._retry_request = AsyncMock(
side_effect=lambda *a, **k: httpx.Response(200, json=upstream_body)
)
statuses = [
client.post(path, headers=headers, json=body).status_code
for headers in (
{"api-key": "direct-A"},
{"api-key": "direct-B"},
{**forwarded, "api-key": "rotated-A"},
{**forwarded, "api-key": "rotated-B"},
)
]
assert statuses == [200, 200, 200, 429]