fix(perf): make rate limiter bucket checks constant-time (#3373)

## Description

The proxy's rate limiter declared a 1,000-identity cap but never
enforced it. Above that threshold, every request copied and scanned the
entire fresh bucket table while holding the shared asyncio lock. A
legitimate high-cardinality gateway could therefore retain an unbounded
number of identities and serialize all request checks behind an O(n)
scan.

This replaces the repeated stale-table scan with a shared O(1) LRU
lifecycle for RPM and TPM state. Both tables now remain bounded to
`MAX_RATE_LIMITER_BUCKETS`, and the least-recently-used identity is
evicted only when a new identity arrives at capacity.

Closes #3372

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature would cause existing functionality
to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added a shared `OrderedDict` LRU for request and token bucket
identities.
- Enforced the existing 1,000-identity cap for both `check_request()`
and direct `check_tokens()` callers.
- Evicted paired RPM/TPM state together so one identity cannot leave
orphaned state in the other table.
- Kept request and token debit/refill behavior independent and unchanged
below the cap.
- Added regression tests for no-scan checks, hard-cap/LRU behavior,
token-only callers, paired eviction, and RPM/TPM semantics.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_rate_limiter.py tests/test_rate_limit_policy.py tests/test_proxy_healthchecks.py -q
21 passed, 4 warnings in 18.55s

$ ruff check headroom/proxy/rate_limiter.py tests/test_rate_limiter.py tests/test_rate_limit_policy.py
All checks passed!

$ ruff format headroom/proxy/rate_limiter.py tests/test_rate_limiter.py
2 files left unchanged
```

Targeted mypy with skipped imports was not counted because
`headroom.proxy.models` then becomes `Any`; including that module
surfaces two unrelated pre-existing `no-any-return` errors in its
environment helpers.

## Real Behavior Proof

- Environment: Linux 7.1.9 x86_64, Python 3.14.7, Headroom
v0.37.0/current `main` at `7b3d226598fec88b5766bcf171cdd7f2a5d1cd3b`,
provider-independent proxy rate limiter.
- Exact command / steps: called the production
`TokenBucketRateLimiter.check_request()` with 10,000 distinct legitimate
client identities, timed 100 subsequent checks, then extended the run to
100,000 identities and measured 20 checks alongside a 1 ms asyncio
heartbeat. Repeated the same public-API reproduction before and after
this patch.
- Observed result: before the fix, the declared 1,000-entry cap retained
10,000 entries, creating those identities took 4.681 s, and the next 100
checks took 99.367 ms. At a prepopulated 100,000 fresh entries, median
check latency was 14.544 ms and 20 checks stalled the event loop for
290.8 ms. After the fix, 10,000 identities produced exactly 1,000
retained entries in 0.023 s and the next 100 checks took 0.121 ms. After
100,000 identities, the table still contained exactly 1,000 entries; 20
checks took 0.038 ms and the maximum observed heartbeat gap was 1.249
ms.
- Not tested: a deployed multi-worker gateway or provider network
traffic; the limiter is process-local and provider-independent, so the
reproduction exercises the production component directly.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: only when more than 1,000 distinct
identities are active; the least-recently-used identity is now evicted
as the existing cap intended. Behavior at or below the cap is unchanged.
- Kill switch / disable path: existing rate-limit disable configuration;
no new switch.
- Unsafe override required: no.
- Qualification impact: rate-limit state remains bounded and checks no
longer block the event loop with cardinality-dependent scans.
- Rollback path: revert this commit to restore the previous stale-scan
behavior.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation (N/A:
internal limiter implementation; no public configuration or API changed)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing focused unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

LRU eviction is the bounded fallback only after the existing
1,000-identity limit is reached. A shared identity order ensures request
and token state cannot grow independently, including for callers that
invoke token checks directly.
This commit is contained in:
C0sm0cats
2026-09-24 17:18:35 -05:00
committed by GitHub
parent a28dd4eab7
commit a8bd9becb3
2 changed files with 143 additions and 31 deletions
+37 -31
View File
@@ -8,14 +8,11 @@ Extracted from server.py for maintainability.
from __future__ import annotations
import asyncio
import logging
import time
from collections import defaultdict
from collections import OrderedDict
from headroom.proxy.models import RateLimitState
from headroom.proxy.rate_limit_policy import consume_from_bucket, refilled_tokens, stale_bucket_keys
logger = logging.getLogger("headroom.proxy")
from headroom.proxy.rate_limit_policy import consume_from_bucket, refilled_tokens
# Maximum rate limiter buckets (prevents DoS via spoofed API keys)
MAX_RATE_LIMITER_BUCKETS = 1000
@@ -32,28 +29,38 @@ class TokenBucketRateLimiter:
self.requests_per_minute = requests_per_minute
self.tokens_per_minute = tokens_per_minute
# Per-key buckets (key = API key or IP)
self._request_buckets: dict[str, RateLimitState] = defaultdict(
lambda: RateLimitState(tokens=requests_per_minute, last_update=time.time())
)
self._token_buckets: dict[str, RateLimitState] = defaultdict(
lambda: RateLimitState(tokens=tokens_per_minute, last_update=time.time())
)
# Per-key buckets (key = API key or IP). A shared LRU keeps request and
# token state on the same bounded lifecycle without scanning all keys.
self._request_buckets: dict[str, RateLimitState] = {}
self._token_buckets: dict[str, RateLimitState] = {}
self._bucket_lru: OrderedDict[str, None] = OrderedDict()
self._lock = asyncio.Lock()
async def _cleanup_stale_buckets(self) -> None:
"""Remove buckets that haven't been used in the last 10 minutes."""
now = time.time()
stale_keys = stale_bucket_keys(
{k: v.last_update for k, v in self._request_buckets.items()},
now=now,
stale_after_seconds=600,
)
for k in stale_keys:
del self._request_buckets[k]
self._token_buckets.pop(k, None)
if stale_keys:
logger.debug(f"Cleaned up {len(stale_keys)} stale rate limiter buckets")
def _touch_bucket(self, key: str) -> None:
"""Mark a bucket active, evicting the least-recently-used key at capacity."""
if key in self._bucket_lru:
self._bucket_lru.move_to_end(key)
return
if len(self._bucket_lru) >= MAX_RATE_LIMITER_BUCKETS:
evicted_key, _ = self._bucket_lru.popitem(last=False)
self._request_buckets.pop(evicted_key, None)
self._token_buckets.pop(evicted_key, None)
self._bucket_lru[key] = None
def _request_bucket(self, key: str) -> RateLimitState:
state = self._request_buckets.get(key)
if state is None:
state = RateLimitState(tokens=self.requests_per_minute, last_update=time.time())
self._request_buckets[key] = state
return state
def _token_bucket(self, key: str) -> RateLimitState:
state = self._token_buckets.get(key)
if state is None:
state = RateLimitState(tokens=self.tokens_per_minute, last_update=time.time())
self._token_buckets[key] = state
return state
def _refill(self, state: RateLimitState, rate_per_minute: float) -> float:
"""Refill bucket based on elapsed time."""
@@ -70,10 +77,8 @@ class TokenBucketRateLimiter:
async def check_request(self, key: str = "default") -> tuple[bool, float]:
"""Check if request is allowed. Returns (allowed, wait_seconds)."""
async with self._lock:
# Prevent unbounded bucket growth from spoofed keys
if len(self._request_buckets) > MAX_RATE_LIMITER_BUCKETS:
await self._cleanup_stale_buckets()
state = self._request_buckets[key]
self._touch_bucket(key)
state = self._request_bucket(key)
available = self._refill(state, self.requests_per_minute)
allowed, state.tokens, wait_seconds = consume_from_bucket(
@@ -86,7 +91,8 @@ class TokenBucketRateLimiter:
async def check_tokens(self, key: str, token_count: int) -> tuple[bool, float]:
"""Check if token usage is allowed."""
async with self._lock:
state = self._token_buckets[key]
self._touch_bucket(key)
state = self._token_bucket(key)
available = self._refill(state, self.tokens_per_minute)
allowed, state.tokens, wait_seconds = consume_from_bucket(
@@ -102,5 +108,5 @@ class TokenBucketRateLimiter:
return {
"requests_per_minute": self.requests_per_minute,
"tokens_per_minute": self.tokens_per_minute,
"active_keys": len(self._request_buckets),
"active_keys": len(self._bucket_lru),
}
+106
View File
@@ -0,0 +1,106 @@
"""Regression tests for the async token-bucket rate limiter."""
from __future__ import annotations
from collections import OrderedDict
import pytest
from headroom.proxy.models import RateLimitState
from headroom.proxy.rate_limiter import MAX_RATE_LIMITER_BUCKETS, TokenBucketRateLimiter
class _NoIterationDict(dict[str, RateLimitState]):
"""Bucket table that fails if a hot-path operation scans every state."""
def __iter__(self): # type: ignore[override]
raise AssertionError("rate-limit hot path scanned all bucket states")
def items(self): # type: ignore[override]
raise AssertionError("rate-limit hot path scanned all bucket states")
def keys(self): # type: ignore[override]
raise AssertionError("rate-limit hot path scanned all bucket states")
def values(self): # type: ignore[override]
raise AssertionError("rate-limit hot path scanned all bucket states")
class _NoIterationOrderedDict(OrderedDict[str, None]):
"""OrderedDict that fails if a hot-path operation scans all identities."""
def __iter__(self): # type: ignore[override]
raise AssertionError("rate-limit hot path scanned all bucket identities")
def items(self): # type: ignore[override]
raise AssertionError("rate-limit hot path scanned all bucket identities")
def keys(self): # type: ignore[override]
raise AssertionError("rate-limit hot path scanned all bucket identities")
def values(self): # type: ignore[override]
raise AssertionError("rate-limit hot path scanned all bucket identities")
@pytest.mark.asyncio
async def test_existing_bucket_check_does_not_scan_all_identities() -> None:
limiter = TokenBucketRateLimiter(requests_per_minute=1_000_000)
for index in range(MAX_RATE_LIMITER_BUCKETS):
await limiter.check_request(f"client-{index}")
limiter._request_buckets = _NoIterationDict(limiter._request_buckets)
limiter._bucket_lru = _NoIterationOrderedDict(limiter._bucket_lru)
allowed, wait_seconds = await limiter.check_request("client-0")
assert allowed is True
assert wait_seconds == 0
@pytest.mark.asyncio
async def test_bucket_state_is_hard_capped_and_lru_evicted() -> None:
limiter = TokenBucketRateLimiter(
requests_per_minute=1_000_000,
tokens_per_minute=1_000_000,
)
for index in range(MAX_RATE_LIMITER_BUCKETS):
key = f"client-{index}"
await limiter.check_request(key)
await limiter.check_tokens(key, 1)
# Keep client-0 recent so client-1 is now the least-recently-used identity.
await limiter.check_request("client-0")
await limiter.check_request("overflow-client")
assert len(limiter._bucket_lru) == MAX_RATE_LIMITER_BUCKETS
assert len(limiter._request_buckets) == MAX_RATE_LIMITER_BUCKETS
assert len(limiter._token_buckets) == MAX_RATE_LIMITER_BUCKETS - 1
assert "client-0" in limiter._request_buckets
assert "client-1" not in limiter._request_buckets
assert "client-1" not in limiter._token_buckets
assert "overflow-client" in limiter._request_buckets
@pytest.mark.asyncio
async def test_token_only_checks_share_the_same_hard_cap() -> None:
limiter = TokenBucketRateLimiter(tokens_per_minute=1_000_000)
for index in range(MAX_RATE_LIMITER_BUCKETS + 50):
allowed, wait_seconds = await limiter.check_tokens(f"client-{index}", 1)
assert allowed is True
assert wait_seconds == 0
assert len(limiter._bucket_lru) == MAX_RATE_LIMITER_BUCKETS
assert len(limiter._token_buckets) == MAX_RATE_LIMITER_BUCKETS
assert not limiter._request_buckets
assert (await limiter.stats())["active_keys"] == MAX_RATE_LIMITER_BUCKETS
@pytest.mark.asyncio
async def test_request_and_token_limits_still_debit_independently() -> None:
limiter = TokenBucketRateLimiter(requests_per_minute=1, tokens_per_minute=5)
assert (await limiter.check_request("client"))[0] is True
assert (await limiter.check_request("client"))[0] is False
assert (await limiter.check_tokens("client", 5))[0] is True
assert (await limiter.check_tokens("client", 1))[0] is False