mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:07:29 +08:00
## Description #3350 (shipped in 0.39.0) started enforcing the token-per-minute bucket, which has two problems for large-context coding agents. 1. **A request larger than the bucket is refused forever.** The bucket never holds more than `rate_limit_tokens_per_minute` (`refilled_tokens` caps it there), so a request whose input is over the limit can never be admitted. The 429's `Retry-After` never comes true, and retrying, or `/compact`, which sends the same large conversation, hits the same 429 again. Once a Claude Code session passes 100k input tokens it is stuck until the proxy is restarted. 2. **The implicit 100k default is now enforced on every proxy.** Before #3350 the token bucket was never consumed, so no one ran into the default. Now a stock `headroom proxy` / `headroom wrap claude` is a 100k tokens/minute throttle, which a single 1M-context turn exceeds on its own. Users see `{"detail":"Token rate limited. Retry after 79.2s"}` straight after upgrading. Fixes: 1. `consume_from_bucket` admits an oversized request once the bucket is full, then charges it in full. The balance goes negative and later requests wait it off, so the configured average rate still holds. A request that fits in the bucket behaves exactly as before. 2. The token limit is opt-in again: `rate_limit_tokens_per_minute` defaults to `None` (no token limit). `--tpm` / `HEADROOM_TPM` still enforce one, which is what #3347 asked for. The request-per-minute limit (default 60) is unchanged. Follows up #3347 / #3350. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `rate_limit_policy.consume_from_bucket`: a request larger than the bucket's capacity waits only for a full bucket, then is admitted and charged in full. - `TokenBucketRateLimiter`: `tokens_per_minute=None` never limits and creates no token buckets. - `ProxyConfig.rate_limit_tokens_per_minute`, `headroom proxy --tpm` and the `python -m headroom.proxy.server --tpm` default to `None`. The startup banner prints `unlimited`, and the docs table says so too. - Tests: three policy cases (oversized admitted when full, waits only for a full bucket, debt delays the next request); the four handler integration tests now check that the first oversized request is admitted and charged, and that the second is refused before upstream; the limiter defaults to no token limit; the CLI default is `None`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest tests/test_rate_limit_policy.py tests/test_rate_limiter.py tests/test_proxy_tpm_enforcement.py tests/test_proxy_config_rate_limit.py tests/test_proxy_openai_rate_limit_key.py tests/test_codex_rate_limits.py tests/test_cli_proxy_env.py -q 139 passed, 1 skipped, 1 warning in 4.37s # The same new policy/integration tests on the 0.39.0 consume_from_bucket: $ uv run --frozen --extra dev pytest tests/test_rate_limit_policy.py tests/test_proxy_tpm_enforcement.py -q 6 failed, 6 passed, 1 warning in 2.56s $ uvx ruff check <changed files> All checks passed! $ uvx ruff format --check <changed files> 6 files already formatted $ uv run --frozen --extra dev mypy headroom/proxy/rate_limiter.py headroom/proxy/models.py headroom/cli/proxy.py headroom/proxy/server.py Success: no issues found in 4 source files ``` ## Real Behavior Proof - Environment: macOS 15 (Darwin 24.6.0), Python 3.12. Stock headroom-ai 0.39.0 wheel on port 18902, this branch on port 18901, both `headroom proxy --no-optimize` with default settings. - Exact command / steps: POSTed the same ~435k-input-token `/v1/messages` request (claude-haiku-4-5, invalid API key so the upstream answers 401 instead of running it) twice to each proxy, then repeated against this branch started with `--tpm 100000`. - Observed result: Stock 0.39.0: 429 `Token rate limited. Retry after 200.8s` on both tries, the second in 6 ms, and the same on every retry after that. This branch, default settings: 401 from Anthropic on both tries (forwarded, no token limit). This branch with `--tpm 100000`: the first try was forwarded (401 from Anthropic) and the second got 429 `Retry after 260.6s`, the wait for the 335k-token debt plus a full bucket. That wait now actually ends. - Not tested: A real paid 1M-context session against Anthropic (the upstream 401 is enough to prove the proxy forwarded the request); Gemini and OpenAI Responses live (covered by the handler integration tests only). ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: Stable. - Stable/default behavior changed: Yes. The default token-per-minute limit goes back to unlimited, as it effectively was before 0.39.0; an explicit `--tpm` / `HEADROOM_TPM` still enforces one. - Kill switch / disable path: Set `--tpm` / `HEADROOM_TPM` to restore a token limit; `--no-rate-limit` disables rate limiting entirely. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes The two commits can be taken separately. The first alone fixes the permanent 429 for anyone who sets `--tpm`. The second restores the pre-0.39.0 default. Headroom Desktop is working around this for now by launching the proxy with `--no-rate-limit`. --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
117 lines
4.4 KiB
Python
117 lines
4.4 KiB
Python
"""Regression tests for the async token-bucket rate limiter."""
|
|
|
|
from __future__ import annotations
|
|
|
|
from collections import OrderedDict
|
|
|
|
import pytest
|
|
|
|
from headroom.proxy.models import RateLimitState
|
|
from headroom.proxy.rate_limiter import MAX_RATE_LIMITER_BUCKETS, TokenBucketRateLimiter
|
|
|
|
|
|
class _NoIterationDict(dict[str, RateLimitState]):
|
|
"""Bucket table that fails if a hot-path operation scans every state."""
|
|
|
|
def __iter__(self): # type: ignore[override]
|
|
raise AssertionError("rate-limit hot path scanned all bucket states")
|
|
|
|
def items(self): # type: ignore[override]
|
|
raise AssertionError("rate-limit hot path scanned all bucket states")
|
|
|
|
def keys(self): # type: ignore[override]
|
|
raise AssertionError("rate-limit hot path scanned all bucket states")
|
|
|
|
def values(self): # type: ignore[override]
|
|
raise AssertionError("rate-limit hot path scanned all bucket states")
|
|
|
|
|
|
class _NoIterationOrderedDict(OrderedDict[str, None]):
|
|
"""OrderedDict that fails if a hot-path operation scans all identities."""
|
|
|
|
def __iter__(self): # type: ignore[override]
|
|
raise AssertionError("rate-limit hot path scanned all bucket identities")
|
|
|
|
def items(self): # type: ignore[override]
|
|
raise AssertionError("rate-limit hot path scanned all bucket identities")
|
|
|
|
def keys(self): # type: ignore[override]
|
|
raise AssertionError("rate-limit hot path scanned all bucket identities")
|
|
|
|
def values(self): # type: ignore[override]
|
|
raise AssertionError("rate-limit hot path scanned all bucket identities")
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_existing_bucket_check_does_not_scan_all_identities() -> None:
|
|
limiter = TokenBucketRateLimiter(requests_per_minute=1_000_000)
|
|
for index in range(MAX_RATE_LIMITER_BUCKETS):
|
|
await limiter.check_request(f"client-{index}")
|
|
|
|
limiter._request_buckets = _NoIterationDict(limiter._request_buckets)
|
|
limiter._bucket_lru = _NoIterationOrderedDict(limiter._bucket_lru)
|
|
|
|
allowed, wait_seconds = await limiter.check_request("client-0")
|
|
|
|
assert allowed is True
|
|
assert wait_seconds == 0
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_bucket_state_is_hard_capped_and_lru_evicted() -> None:
|
|
limiter = TokenBucketRateLimiter(
|
|
requests_per_minute=1_000_000,
|
|
tokens_per_minute=1_000_000,
|
|
)
|
|
for index in range(MAX_RATE_LIMITER_BUCKETS):
|
|
key = f"client-{index}"
|
|
await limiter.check_request(key)
|
|
await limiter.check_tokens(key, 1)
|
|
|
|
# Keep client-0 recent so client-1 is now the least-recently-used identity.
|
|
await limiter.check_request("client-0")
|
|
await limiter.check_request("overflow-client")
|
|
|
|
assert len(limiter._bucket_lru) == MAX_RATE_LIMITER_BUCKETS
|
|
assert len(limiter._request_buckets) == MAX_RATE_LIMITER_BUCKETS
|
|
assert len(limiter._token_buckets) == MAX_RATE_LIMITER_BUCKETS - 1
|
|
assert "client-0" in limiter._request_buckets
|
|
assert "client-1" not in limiter._request_buckets
|
|
assert "client-1" not in limiter._token_buckets
|
|
assert "overflow-client" in limiter._request_buckets
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_token_only_checks_share_the_same_hard_cap() -> None:
|
|
limiter = TokenBucketRateLimiter(tokens_per_minute=1_000_000)
|
|
|
|
for index in range(MAX_RATE_LIMITER_BUCKETS + 50):
|
|
allowed, wait_seconds = await limiter.check_tokens(f"client-{index}", 1)
|
|
assert allowed is True
|
|
assert wait_seconds == 0
|
|
|
|
assert len(limiter._bucket_lru) == MAX_RATE_LIMITER_BUCKETS
|
|
assert len(limiter._token_buckets) == MAX_RATE_LIMITER_BUCKETS
|
|
assert not limiter._request_buckets
|
|
assert (await limiter.stats())["active_keys"] == MAX_RATE_LIMITER_BUCKETS
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_request_and_token_limits_still_debit_independently() -> None:
|
|
limiter = TokenBucketRateLimiter(requests_per_minute=1, tokens_per_minute=5)
|
|
|
|
assert (await limiter.check_request("client"))[0] is True
|
|
assert (await limiter.check_request("client"))[0] is False
|
|
assert (await limiter.check_tokens("client", 5))[0] is True
|
|
assert (await limiter.check_tokens("client", 1))[0] is False
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_token_limit_is_off_unless_configured() -> None:
|
|
# 0.39.0 started enforcing an implicit 100k default and 429'd every
|
|
# large-context agent turn; the default is now no token limit.
|
|
limiter = TokenBucketRateLimiter()
|
|
|
|
assert await limiter.check_tokens("client", 900_000) == (True, 0.0)
|
|
assert not limiter._token_buckets
|