mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(perf): make rate limiter bucket checks constant-time (#3373)
## Description The proxy's rate limiter declared a 1,000-identity cap but never enforced it. Above that threshold, every request copied and scanned the entire fresh bucket table while holding the shared asyncio lock. A legitimate high-cardinality gateway could therefore retain an unbounded number of identities and serialize all request checks behind an O(n) scan. This replaces the repeated stale-table scan with a shared O(1) LRU lifecycle for RPM and TPM state. Both tables now remain bounded to `MAX_RATE_LIMITER_BUCKETS`, and the least-recently-used identity is evicted only when a new identity arrives at capacity. Closes #3372 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added a shared `OrderedDict` LRU for request and token bucket identities. - Enforced the existing 1,000-identity cap for both `check_request()` and direct `check_tokens()` callers. - Evicted paired RPM/TPM state together so one identity cannot leave orphaned state in the other table. - Kept request and token debit/refill behavior independent and unchanged below the cap. - Added regression tests for no-scan checks, hard-cap/LRU behavior, token-only callers, paired eviction, and RPM/TPM semantics. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_rate_limiter.py tests/test_rate_limit_policy.py tests/test_proxy_healthchecks.py -q 21 passed, 4 warnings in 18.55s $ ruff check headroom/proxy/rate_limiter.py tests/test_rate_limiter.py tests/test_rate_limit_policy.py All checks passed! $ ruff format headroom/proxy/rate_limiter.py tests/test_rate_limiter.py 2 files left unchanged ``` Targeted mypy with skipped imports was not counted because `headroom.proxy.models` then becomes `Any`; including that module surfaces two unrelated pre-existing `no-any-return` errors in its environment helpers. ## Real Behavior Proof - Environment: Linux 7.1.9 x86_64, Python 3.14.7, Headroom v0.37.0/current `main` at `7b3d226598fec88b5766bcf171cdd7f2a5d1cd3b`, provider-independent proxy rate limiter. - Exact command / steps: called the production `TokenBucketRateLimiter.check_request()` with 10,000 distinct legitimate client identities, timed 100 subsequent checks, then extended the run to 100,000 identities and measured 20 checks alongside a 1 ms asyncio heartbeat. Repeated the same public-API reproduction before and after this patch. - Observed result: before the fix, the declared 1,000-entry cap retained 10,000 entries, creating those identities took 4.681 s, and the next 100 checks took 99.367 ms. At a prepopulated 100,000 fresh entries, median check latency was 14.544 ms and 20 checks stalled the event loop for 290.8 ms. After the fix, 10,000 identities produced exactly 1,000 retained entries in 0.023 s and the next 100 checks took 0.121 ms. After 100,000 identities, the table still contained exactly 1,000 entries; 20 checks took 0.038 ms and the maximum observed heartbeat gap was 1.249 ms. - Not tested: a deployed multi-worker gateway or provider network traffic; the limiter is process-local and provider-independent, so the reproduction exercises the production component directly. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: N/A. - Stable/default behavior changed: only when more than 1,000 distinct identities are active; the least-recently-used identity is now evicted as the existing cap intended. Behavior at or below the cap is unchanged. - Kill switch / disable path: existing rate-limit disable configuration; no new switch. - Unsafe override required: no. - Qualification impact: rate-limit state remains bounded and checks no longer block the event loop with cardinality-dependent scans. - Rollback path: revert this commit to restore the previous stale-scan behavior. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (N/A: internal limiter implementation; no public configuration or API changed) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing focused unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes LRU eviction is the bounded fallback only after the existing 1,000-identity limit is reached. A shared identity order ensures request and token state cannot grow independently, including for callers that invoke token checks directly.
This commit is contained in:
@@ -8,14 +8,11 @@ Extracted from server.py for maintainability.
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import time
|
||||
from collections import defaultdict
|
||||
from collections import OrderedDict
|
||||
|
||||
from headroom.proxy.models import RateLimitState
|
||||
from headroom.proxy.rate_limit_policy import consume_from_bucket, refilled_tokens, stale_bucket_keys
|
||||
|
||||
logger = logging.getLogger("headroom.proxy")
|
||||
from headroom.proxy.rate_limit_policy import consume_from_bucket, refilled_tokens
|
||||
|
||||
# Maximum rate limiter buckets (prevents DoS via spoofed API keys)
|
||||
MAX_RATE_LIMITER_BUCKETS = 1000
|
||||
@@ -32,28 +29,38 @@ class TokenBucketRateLimiter:
|
||||
self.requests_per_minute = requests_per_minute
|
||||
self.tokens_per_minute = tokens_per_minute
|
||||
|
||||
# Per-key buckets (key = API key or IP)
|
||||
self._request_buckets: dict[str, RateLimitState] = defaultdict(
|
||||
lambda: RateLimitState(tokens=requests_per_minute, last_update=time.time())
|
||||
)
|
||||
self._token_buckets: dict[str, RateLimitState] = defaultdict(
|
||||
lambda: RateLimitState(tokens=tokens_per_minute, last_update=time.time())
|
||||
)
|
||||
# Per-key buckets (key = API key or IP). A shared LRU keeps request and
|
||||
# token state on the same bounded lifecycle without scanning all keys.
|
||||
self._request_buckets: dict[str, RateLimitState] = {}
|
||||
self._token_buckets: dict[str, RateLimitState] = {}
|
||||
self._bucket_lru: OrderedDict[str, None] = OrderedDict()
|
||||
self._lock = asyncio.Lock()
|
||||
|
||||
async def _cleanup_stale_buckets(self) -> None:
|
||||
"""Remove buckets that haven't been used in the last 10 minutes."""
|
||||
now = time.time()
|
||||
stale_keys = stale_bucket_keys(
|
||||
{k: v.last_update for k, v in self._request_buckets.items()},
|
||||
now=now,
|
||||
stale_after_seconds=600,
|
||||
)
|
||||
for k in stale_keys:
|
||||
del self._request_buckets[k]
|
||||
self._token_buckets.pop(k, None)
|
||||
if stale_keys:
|
||||
logger.debug(f"Cleaned up {len(stale_keys)} stale rate limiter buckets")
|
||||
def _touch_bucket(self, key: str) -> None:
|
||||
"""Mark a bucket active, evicting the least-recently-used key at capacity."""
|
||||
if key in self._bucket_lru:
|
||||
self._bucket_lru.move_to_end(key)
|
||||
return
|
||||
|
||||
if len(self._bucket_lru) >= MAX_RATE_LIMITER_BUCKETS:
|
||||
evicted_key, _ = self._bucket_lru.popitem(last=False)
|
||||
self._request_buckets.pop(evicted_key, None)
|
||||
self._token_buckets.pop(evicted_key, None)
|
||||
self._bucket_lru[key] = None
|
||||
|
||||
def _request_bucket(self, key: str) -> RateLimitState:
|
||||
state = self._request_buckets.get(key)
|
||||
if state is None:
|
||||
state = RateLimitState(tokens=self.requests_per_minute, last_update=time.time())
|
||||
self._request_buckets[key] = state
|
||||
return state
|
||||
|
||||
def _token_bucket(self, key: str) -> RateLimitState:
|
||||
state = self._token_buckets.get(key)
|
||||
if state is None:
|
||||
state = RateLimitState(tokens=self.tokens_per_minute, last_update=time.time())
|
||||
self._token_buckets[key] = state
|
||||
return state
|
||||
|
||||
def _refill(self, state: RateLimitState, rate_per_minute: float) -> float:
|
||||
"""Refill bucket based on elapsed time."""
|
||||
@@ -70,10 +77,8 @@ class TokenBucketRateLimiter:
|
||||
async def check_request(self, key: str = "default") -> tuple[bool, float]:
|
||||
"""Check if request is allowed. Returns (allowed, wait_seconds)."""
|
||||
async with self._lock:
|
||||
# Prevent unbounded bucket growth from spoofed keys
|
||||
if len(self._request_buckets) > MAX_RATE_LIMITER_BUCKETS:
|
||||
await self._cleanup_stale_buckets()
|
||||
state = self._request_buckets[key]
|
||||
self._touch_bucket(key)
|
||||
state = self._request_bucket(key)
|
||||
available = self._refill(state, self.requests_per_minute)
|
||||
|
||||
allowed, state.tokens, wait_seconds = consume_from_bucket(
|
||||
@@ -86,7 +91,8 @@ class TokenBucketRateLimiter:
|
||||
async def check_tokens(self, key: str, token_count: int) -> tuple[bool, float]:
|
||||
"""Check if token usage is allowed."""
|
||||
async with self._lock:
|
||||
state = self._token_buckets[key]
|
||||
self._touch_bucket(key)
|
||||
state = self._token_bucket(key)
|
||||
available = self._refill(state, self.tokens_per_minute)
|
||||
|
||||
allowed, state.tokens, wait_seconds = consume_from_bucket(
|
||||
@@ -102,5 +108,5 @@ class TokenBucketRateLimiter:
|
||||
return {
|
||||
"requests_per_minute": self.requests_per_minute,
|
||||
"tokens_per_minute": self.tokens_per_minute,
|
||||
"active_keys": len(self._request_buckets),
|
||||
"active_keys": len(self._bucket_lru),
|
||||
}
|
||||
|
||||
@@ -0,0 +1,106 @@
|
||||
"""Regression tests for the async token-bucket rate limiter."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import OrderedDict
|
||||
|
||||
import pytest
|
||||
|
||||
from headroom.proxy.models import RateLimitState
|
||||
from headroom.proxy.rate_limiter import MAX_RATE_LIMITER_BUCKETS, TokenBucketRateLimiter
|
||||
|
||||
|
||||
class _NoIterationDict(dict[str, RateLimitState]):
|
||||
"""Bucket table that fails if a hot-path operation scans every state."""
|
||||
|
||||
def __iter__(self): # type: ignore[override]
|
||||
raise AssertionError("rate-limit hot path scanned all bucket states")
|
||||
|
||||
def items(self): # type: ignore[override]
|
||||
raise AssertionError("rate-limit hot path scanned all bucket states")
|
||||
|
||||
def keys(self): # type: ignore[override]
|
||||
raise AssertionError("rate-limit hot path scanned all bucket states")
|
||||
|
||||
def values(self): # type: ignore[override]
|
||||
raise AssertionError("rate-limit hot path scanned all bucket states")
|
||||
|
||||
|
||||
class _NoIterationOrderedDict(OrderedDict[str, None]):
|
||||
"""OrderedDict that fails if a hot-path operation scans all identities."""
|
||||
|
||||
def __iter__(self): # type: ignore[override]
|
||||
raise AssertionError("rate-limit hot path scanned all bucket identities")
|
||||
|
||||
def items(self): # type: ignore[override]
|
||||
raise AssertionError("rate-limit hot path scanned all bucket identities")
|
||||
|
||||
def keys(self): # type: ignore[override]
|
||||
raise AssertionError("rate-limit hot path scanned all bucket identities")
|
||||
|
||||
def values(self): # type: ignore[override]
|
||||
raise AssertionError("rate-limit hot path scanned all bucket identities")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_existing_bucket_check_does_not_scan_all_identities() -> None:
|
||||
limiter = TokenBucketRateLimiter(requests_per_minute=1_000_000)
|
||||
for index in range(MAX_RATE_LIMITER_BUCKETS):
|
||||
await limiter.check_request(f"client-{index}")
|
||||
|
||||
limiter._request_buckets = _NoIterationDict(limiter._request_buckets)
|
||||
limiter._bucket_lru = _NoIterationOrderedDict(limiter._bucket_lru)
|
||||
|
||||
allowed, wait_seconds = await limiter.check_request("client-0")
|
||||
|
||||
assert allowed is True
|
||||
assert wait_seconds == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_bucket_state_is_hard_capped_and_lru_evicted() -> None:
|
||||
limiter = TokenBucketRateLimiter(
|
||||
requests_per_minute=1_000_000,
|
||||
tokens_per_minute=1_000_000,
|
||||
)
|
||||
for index in range(MAX_RATE_LIMITER_BUCKETS):
|
||||
key = f"client-{index}"
|
||||
await limiter.check_request(key)
|
||||
await limiter.check_tokens(key, 1)
|
||||
|
||||
# Keep client-0 recent so client-1 is now the least-recently-used identity.
|
||||
await limiter.check_request("client-0")
|
||||
await limiter.check_request("overflow-client")
|
||||
|
||||
assert len(limiter._bucket_lru) == MAX_RATE_LIMITER_BUCKETS
|
||||
assert len(limiter._request_buckets) == MAX_RATE_LIMITER_BUCKETS
|
||||
assert len(limiter._token_buckets) == MAX_RATE_LIMITER_BUCKETS - 1
|
||||
assert "client-0" in limiter._request_buckets
|
||||
assert "client-1" not in limiter._request_buckets
|
||||
assert "client-1" not in limiter._token_buckets
|
||||
assert "overflow-client" in limiter._request_buckets
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_token_only_checks_share_the_same_hard_cap() -> None:
|
||||
limiter = TokenBucketRateLimiter(tokens_per_minute=1_000_000)
|
||||
|
||||
for index in range(MAX_RATE_LIMITER_BUCKETS + 50):
|
||||
allowed, wait_seconds = await limiter.check_tokens(f"client-{index}", 1)
|
||||
assert allowed is True
|
||||
assert wait_seconds == 0
|
||||
|
||||
assert len(limiter._bucket_lru) == MAX_RATE_LIMITER_BUCKETS
|
||||
assert len(limiter._token_buckets) == MAX_RATE_LIMITER_BUCKETS
|
||||
assert not limiter._request_buckets
|
||||
assert (await limiter.stats())["active_keys"] == MAX_RATE_LIMITER_BUCKETS
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_request_and_token_limits_still_debit_independently() -> None:
|
||||
limiter = TokenBucketRateLimiter(requests_per_minute=1, tokens_per_minute=5)
|
||||
|
||||
assert (await limiter.check_request("client"))[0] is True
|
||||
assert (await limiter.check_request("client"))[0] is False
|
||||
assert (await limiter.check_tokens("client", 5))[0] is True
|
||||
assert (await limiter.check_tokens("client", 1))[0] is False
|
||||
Reference in New Issue
Block a user