fix(proxy): stop the 0.39.0 TPM limiter from refusing large-context requests forever (#3806)

## Description

#3350 (shipped in 0.39.0) started enforcing the token-per-minute bucket,
which has two problems for large-context coding agents.

1. **A request larger than the bucket is refused forever.** The bucket
never holds more than `rate_limit_tokens_per_minute` (`refilled_tokens`
caps it there), so a request whose input is over the limit can never be
admitted. The 429's `Retry-After` never comes true, and retrying, or
`/compact`, which sends the same large conversation, hits the same 429
again. Once a Claude Code session passes 100k input tokens it is stuck
until the proxy is restarted.
2. **The implicit 100k default is now enforced on every proxy.** Before
#3350 the token bucket was never consumed, so no one ran into the
default. Now a stock `headroom proxy` / `headroom wrap claude` is a 100k
tokens/minute throttle, which a single 1M-context turn exceeds on its
own. Users see `{"detail":"Token rate limited. Retry after 79.2s"}`
straight after upgrading.

Fixes:

1. `consume_from_bucket` admits an oversized request once the bucket is
full, then charges it in full. The balance goes negative and later
requests wait it off, so the configured average rate still holds. A
request that fits in the bucket behaves exactly as before.
2. The token limit is opt-in again: `rate_limit_tokens_per_minute`
defaults to `None` (no token limit). `--tpm` / `HEADROOM_TPM` still
enforce one, which is what #3347 asked for. The request-per-minute limit
(default 60) is unchanged.

Follows up #3347 / #3350.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `rate_limit_policy.consume_from_bucket`: a request larger than the
bucket's capacity waits only for a full bucket, then is admitted and
charged in full.
- `TokenBucketRateLimiter`: `tokens_per_minute=None` never limits and
creates no token buckets.
- `ProxyConfig.rate_limit_tokens_per_minute`, `headroom proxy --tpm` and
the `python -m headroom.proxy.server --tpm` default to `None`. The
startup banner prints `unlimited`, and the docs table says so too.
- Tests: three policy cases (oversized admitted when full, waits only
for a full bucket, debt delays the next request); the four handler
integration tests now check that the first oversized request is admitted
and charged, and that the second is refused before upstream; the limiter
defaults to no token limit; the CLI default is `None`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_rate_limit_policy.py tests/test_rate_limiter.py tests/test_proxy_tpm_enforcement.py tests/test_proxy_config_rate_limit.py tests/test_proxy_openai_rate_limit_key.py tests/test_codex_rate_limits.py tests/test_cli_proxy_env.py -q
139 passed, 1 skipped, 1 warning in 4.37s

# The same new policy/integration tests on the 0.39.0 consume_from_bucket:
$ uv run --frozen --extra dev pytest tests/test_rate_limit_policy.py tests/test_proxy_tpm_enforcement.py -q
6 failed, 6 passed, 1 warning in 2.56s

$ uvx ruff check <changed files>
All checks passed!
$ uvx ruff format --check <changed files>
6 files already formatted
$ uv run --frozen --extra dev mypy headroom/proxy/rate_limiter.py headroom/proxy/models.py headroom/cli/proxy.py headroom/proxy/server.py
Success: no issues found in 4 source files
```

## Real Behavior Proof

- Environment: macOS 15 (Darwin 24.6.0), Python 3.12. Stock headroom-ai
0.39.0 wheel on port 18902, this branch on port 18901, both `headroom
proxy --no-optimize` with default settings.
- Exact command / steps: POSTed the same ~435k-input-token
`/v1/messages` request (claude-haiku-4-5, invalid API key so the
upstream answers 401 instead of running it) twice to each proxy, then
repeated against this branch started with `--tpm 100000`.
- Observed result: Stock 0.39.0: 429 `Token rate limited. Retry after
200.8s` on both tries, the second in 6 ms, and the same on every retry
after that. This branch, default settings: 401 from Anthropic on both
tries (forwarded, no token limit). This branch with `--tpm 100000`: the
first try was forwarded (401 from Anthropic) and the second got 429
`Retry after 260.6s`, the wait for the 335k-token debt plus a full
bucket. That wait now actually ends.
- Not tested: A real paid 1M-context session against Anthropic (the
upstream 401 is enough to prove the proxy forwarded the request); Gemini
and OpenAI Responses live (covered by the handler integration tests
only).

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes. The default token-per-minute
limit goes back to unlimited, as it effectively was before 0.39.0; an
explicit `--tpm` / `HEADROOM_TPM` still enforces one.
- Kill switch / disable path: Set `--tpm` / `HEADROOM_TPM` to restore a
token limit; `--no-rate-limit` disables rate limiting entirely.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

The two commits can be taken separately. The first alone fixes the
permanent 429 for anyone who sets `--tpm`. The second restores the
pre-0.39.0 default. Headroom Desktop is working around this for now by
launching the proxy with `--no-rate-limit`.

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
gglucass
2026-09-26 08:37:52 -05:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 66f426171b
commit 7968122658
10 changed files with 114 additions and 34 deletions
+1 -1
View File
@@ -274,7 +274,7 @@ Coding agents re-read the same files repeatedly; these control how stale reads a
| `--connect-timeout-seconds` | `10` | Upstream connect timeout (seconds). |
| `--retry-max-attempts` | `3` | Upstream retries on transient failure. |
| `--limit-concurrency` | `1000` | Concurrent connections before returning 503. |
| `--rpm` / `--tpm` | `60` / `100000` | Requests- and tokens-per-minute rate limits (disable with `--no-rate-limit`). |
| `--rpm` / `--tpm` | `60` / unlimited | Requests- and tokens-per-minute rate limits (disable with `--no-rate-limit`). |
| `--budget` / `--budget-period` | none / `daily` | Spend cap in USD per period; over-budget requests get 429. |
| `--workers` / `HEADROOM_WORKERS` | `1` | Uvicorn worker processes. |
+2 -2
View File
@@ -372,7 +372,7 @@ def dashboard(port: int, no_open: bool) -> None:
default=None,
type=click.IntRange(min=1),
envvar="HEADROOM_TPM",
help="Max tokens per minute. Env: HEADROOM_TPM. Default: 100000.",
help="Max tokens per minute. Env: HEADROOM_TPM. Default: unlimited.",
)
@click.option(
"--no-ccr",
@@ -1324,7 +1324,7 @@ def proxy(
cache_enabled=not no_cache,
rate_limit_enabled=not no_rate_limit,
rate_limit_requests_per_minute=rpm if rpm is not None else 60,
rate_limit_tokens_per_minute=tpm if tpm is not None else 100_000,
rate_limit_tokens_per_minute=tpm,
compress_user_messages=_get_env_bool("HEADROOM_COMPRESS_USER_MESSAGES", False),
periodic_malloc_trim_enabled=_get_env_bool(
"HEADROOM_MALLOC_TRIM", default_periodic_malloc_trim()
+4 -1
View File
@@ -329,7 +329,10 @@ class ProxyConfig:
# Rate limiting
rate_limit_enabled: bool = True
rate_limit_requests_per_minute: int = 60
rate_limit_tokens_per_minute: int = 100000
# None = no token limit. Before 0.39.0 the token bucket was never consumed,
# so an implicit default here throttled every large-context agent session
# the moment #3350 started enforcing it. Opt in with --tpm / HEADROOM_TPM.
rate_limit_tokens_per_minute: int | None = None
# Retry
retry_enabled: bool = True
+11 -3
View File
@@ -22,11 +22,19 @@ def consume_from_bucket(
requested_tokens: float,
rate_per_minute: float,
) -> tuple[bool, float, float]:
"""Return ``(allowed, remaining_tokens, wait_seconds)`` for a token request."""
if available_tokens >= requested_tokens:
"""Return ``(allowed, remaining_tokens, wait_seconds)`` for a token request.
The bucket never holds more than ``rate_per_minute``, so a request larger
than that could never be admitted and its ``wait_seconds`` would never come
true. Such a request waits for a full bucket instead, then is charged in
full: the balance goes negative and later requests wait it off, so the
configured average rate still holds.
"""
required = min(requested_tokens, rate_per_minute)
if available_tokens >= required:
return True, available_tokens - requested_tokens, 0.0
wait_seconds = (requested_tokens - available_tokens) * (60.0 / rate_per_minute)
wait_seconds = (required - available_tokens) * (60.0 / rate_per_minute)
return False, available_tokens, wait_seconds
+10 -7
View File
@@ -24,7 +24,7 @@ class TokenBucketRateLimiter:
def __init__(
self,
requests_per_minute: int = 60,
tokens_per_minute: int = 100000,
tokens_per_minute: int | None = None,
):
self.requests_per_minute = requests_per_minute
self.tokens_per_minute = tokens_per_minute
@@ -55,10 +55,10 @@ class TokenBucketRateLimiter:
self._request_buckets[key] = state
return state
def _token_bucket(self, key: str) -> RateLimitState:
def _token_bucket(self, key: str, capacity: int) -> RateLimitState:
state = self._token_buckets.get(key)
if state is None:
state = RateLimitState(tokens=self.tokens_per_minute, last_update=time.time())
state = RateLimitState(tokens=capacity, last_update=time.time())
self._token_buckets[key] = state
return state
@@ -89,16 +89,19 @@ class TokenBucketRateLimiter:
return allowed, wait_seconds
async def check_tokens(self, key: str, token_count: int) -> tuple[bool, float]:
"""Check if token usage is allowed."""
"""Check if token usage is allowed. ``tokens_per_minute=None`` never limits."""
tpm = self.tokens_per_minute
if tpm is None:
return True, 0.0
async with self._lock:
self._touch_bucket(key)
state = self._token_bucket(key)
available = self._refill(state, self.tokens_per_minute)
state = self._token_bucket(key, tpm)
available = self._refill(state, tpm)
allowed, state.tokens, wait_seconds = consume_from_bucket(
available_tokens=available,
requested_tokens=token_count,
rate_per_minute=self.tokens_per_minute,
rate_per_minute=tpm,
)
return allowed, wait_seconds
+4 -2
View File
@@ -5912,7 +5912,7 @@ def run_server(
║ FEATURES: ║
║ Optimization: {"ENABLED " if config.optimize else "DISABLED"} ║
║ Caching: {"ENABLED " if config.cache_enabled else "DISABLED"} (TTL: {config.cache_ttl_seconds}s) ║
║ Rate Limiting: {"ENABLED " if config.rate_limit_enabled else "DISABLED"} ({config.rate_limit_requests_per_minute} req/min, {config.rate_limit_tokens_per_minute:,} tok/min) ║
║ Rate Limiting: {"ENABLED " if config.rate_limit_enabled else "DISABLED"} ({config.rate_limit_requests_per_minute} req/min, {f"{config.rate_limit_tokens_per_minute:,}" if config.rate_limit_tokens_per_minute else "unlimited"} tok/min) ║
║ Retry: {"ENABLED " if config.retry_enabled else "DISABLED"} (max {config.retry_max_attempts} attempts) ║
║ Cost Tracking: {"ENABLED " if config.cost_tracking_enabled else "DISABLED"} (budget: {"$" + str(config.budget_limit_usd) + "/" + config.budget_period if config.budget_limit_usd else "unlimited"}) ║
║ Code-Aware: {code_aware_status:<52}║
@@ -6398,7 +6398,9 @@ if __name__ == "__main__":
# Rate limiting
parser.add_argument("--no-rate-limit", action="store_true", help="Disable rate limiting")
parser.add_argument("--rpm", type=int, default=60, help="Requests per minute")
parser.add_argument("--tpm", type=int, default=100000, help="Tokens per minute")
parser.add_argument(
"--tpm", type=int, default=None, help="Tokens per minute (default: unlimited)"
)
# Cost
parser.add_argument("--budget", type=float, help="Budget limit in USD")
+2 -2
View File
@@ -1337,7 +1337,7 @@ class TestCLIProxyRpmTpm:
assert captured_config["config"].rate_limit_requests_per_minute == 20
def test_tpm_default(self, runner):
"""Without --tpm, rate_limit_tokens_per_minute defaults to 100000."""
"""Without --tpm, there is no token limit (the pre-0.39.0 behaviour)."""
captured_config = {}
def mock_run_server(config, **kwargs):
@@ -1347,7 +1347,7 @@ class TestCLIProxyRpmTpm:
result = runner.invoke(main, ["proxy"], catch_exceptions=False)
assert result.exit_code == 0, result.output
assert captured_config["config"].rate_limit_tokens_per_minute == 100000
assert captured_config["config"].rate_limit_tokens_per_minute is None
def test_tpm_flag(self, runner):
"""--tpm 50000 should set rate_limit_tokens_per_minute to 50000."""
+30 -16
View File
@@ -14,6 +14,9 @@ from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
def test_anthropic_request_over_tpm_is_rejected_before_upstream(monkeypatch) -> None:
# The first request finds a full bucket and is admitted even though it is
# larger than the whole bucket (otherwise it could never pass); it is
# charged in full, so the second one is refused before upstream.
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
config = ProxyConfig(
optimize=False,
@@ -29,26 +32,32 @@ def test_anthropic_request_over_tpm_is_rejected_before_upstream(monkeypatch) ->
image_optimize=False,
)
with TestClient(create_app(config)) as client:
with TestClient(create_app(config), raise_server_exceptions=False) as client:
proxy = client.app.state.proxy
tokenizer = MagicMock()
proxy._count_tokens_offloaded = AsyncMock(return_value=(tokenizer, 5_000))
proxy._retry_request = AsyncMock(side_effect=AssertionError("upstream must not be called"))
proxy._retry_request = AsyncMock(side_effect=RuntimeError("upstream stub"))
response = client.post(
"/v1/messages",
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
json={
"model": "claude-haiku-4-5",
"max_tokens": 64,
"messages": [{"role": "user", "content": "large request"}],
"stream": False,
},
)
def post():
return client.post(
"/v1/messages",
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
json={
"model": "claude-haiku-4-5",
"max_tokens": 64,
"messages": [{"role": "user", "content": "large request"}],
"stream": False,
},
)
assert post().status_code != 429
assert proxy._retry_request.await_count == 1
response = post()
assert response.status_code == 429
assert response.json()["detail"].startswith("Token rate limited.")
proxy._retry_request.assert_not_awaited()
assert proxy._retry_request.await_count == 1
assert len(proxy.rate_limiter._token_buckets) == 1
@@ -100,15 +109,20 @@ def test_other_handlers_reject_request_over_tpm_before_upstream(
image_optimize=False,
)
with TestClient(create_app(config)) as client:
with TestClient(create_app(config), raise_server_exceptions=False) as client:
proxy = client.app.state.proxy
tokenizer = MagicMock()
proxy._count_tokens_offloaded = AsyncMock(return_value=(tokenizer, 5_000))
proxy._retry_request = AsyncMock(side_effect=AssertionError("upstream must not be called"))
proxy._retry_request = AsyncMock(side_effect=RuntimeError("upstream stub"))
assert client.post(path, headers=headers, json=body).status_code != 429
(bucket,) = proxy.rate_limiter._token_buckets.values()
assert bucket.tokens < 0 # admitted and charged in full
upstream_calls = proxy._retry_request.await_count
response = client.post(path, headers=headers, json=body)
assert response.status_code == 429
assert response.json()["detail"].startswith("Token rate limited.")
proxy._retry_request.assert_not_awaited()
assert proxy._retry_request.await_count == upstream_calls
assert len(proxy.rate_limiter._token_buckets) == 1
+40
View File
@@ -57,6 +57,46 @@ def test_consume_from_bucket_denies_and_reports_wait_time() -> None:
assert wait_seconds == 0.5
def test_consume_from_bucket_admits_request_larger_than_bucket_once_full() -> None:
# 0.39.0 enforced TPM with a 100k default; a 232k-token coding-agent turn
# was refused forever with a Retry-After that never came true.
allowed, remaining, wait_seconds = consume_from_bucket(
available_tokens=100_000,
requested_tokens=232_000,
rate_per_minute=100_000,
)
assert allowed is True
assert remaining == -132_000
assert wait_seconds == 0
def test_consume_from_bucket_oversized_request_waits_only_for_a_full_bucket() -> None:
allowed, remaining, wait_seconds = consume_from_bucket(
available_tokens=40_000,
requested_tokens=232_000,
rate_per_minute=100_000,
)
assert allowed is False
assert remaining == 40_000
assert wait_seconds == 36
def test_consume_from_bucket_debt_delays_the_next_request() -> None:
# Charged in full, so the average rate holds: after the 232k request the
# next small one waits out the 132k debt plus its own tokens.
allowed, remaining, wait_seconds = consume_from_bucket(
available_tokens=-132_000,
requested_tokens=1_000,
rate_per_minute=100_000,
)
assert allowed is False
assert remaining == -132_000
assert wait_seconds == 79.8
def test_stale_bucket_keys_returns_only_old_buckets() -> None:
assert stale_bucket_keys(
{"fresh": 950, "edge": 400, "stale": 399},
+10
View File
@@ -104,3 +104,13 @@ async def test_request_and_token_limits_still_debit_independently() -> None:
assert (await limiter.check_request("client"))[0] is False
assert (await limiter.check_tokens("client", 5))[0] is True
assert (await limiter.check_tokens("client", 1))[0] is False
@pytest.mark.asyncio
async def test_token_limit_is_off_unless_configured() -> None:
# 0.39.0 started enforcing an implicit 100k default and 429'd every
# large-context agent turn; the default is now no token limit.
limiter = TokenBucketRateLimiter()
assert await limiter.check_tokens("client", 900_000) == (True, 0.0)
assert not limiter._token_buckets