mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): don't retry a 429/529 that cannot succeed within the backoff cap (#3992)
## Description
The streaming and non-streaming forwarders retry every 429/529, sleeping
`Retry-After` capped at `retry_max_delay_ms` (30 s). When upstream's
`Retry-After` is longer than the cap, the retry is re-sent before the
reset and can only fail again. An exhausted subscription window resets
hours ahead. The client waits `attempts × cap` (≈ 61 s by default) for
the same 429, and every proxy chained in front multiplies that wait.
Observed on Claude OAuth traffic (Anthropic `rate_limit_error: This
request would exceed your account's rate limit`):
```text
Upstream 429 (attempt 1/3), retrying in 30000ms
Upstream 429 (attempt 2/3), retrying in 30000ms
Forwarding upstream streaming error status=429
event=proxy_inbound_response status=429 duration_ms=61746.83
```
With one more headroom → proxy → headroom hop in front, a single request
took 744 s to return its 429.
```diff
if status in RETRYABLE_OVERLOAD_STATUSES and retry_enabled and attempts left
+ and not overload_retry_is_futile(response, retry_max_delay_ms, retries_left)
sleep(retry_after_ms(...) or jitter)
retry
+overload_retry_is_futile(response, max_ms, retries_left)
+ x-should-retry: false → True
+ uncapped Retry-After > retries_left × max_ms → True (reset beyond every remaining wait)
+ otherwise → False (no hint / reachable reset: retry as before)
```
`retry_after_ms` now reads `Retry-After` through `_retry_after_seconds`,
which returns the uncapped value. The headers forwarded on the final 429
are unchanged.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/helpers.py`: `overload_retry_is_futile(response,
max_ms, retries_left)`; `_retry_after_seconds()` shared with
`retry_after_ms()`. The budget is all remaining retries, not one wait:
with 3 attempts and a 30 s cap, `Retry-After: 31` still retries (30 s,
then the next 429 says 1 s) and recovers.
- `headroom/proxy/handlers/streaming.py`, `headroom/proxy/server.py`
(`_retry_request`): skip the overload retry when it is futile.
- Tests: `tests/test_overload_retry_futile.py` (helper, non-streaming
path, decreasing-Retry-After recovery sequence) and streaming cases in
`tests/test_proxy_streaming_ratelimit_headers.py` (futile hints
forwarded at once; reachable reset recovers on the 3rd attempt).
## Testing
- [x] Unit tests pass (`pytest`): retry/streaming/overload suites listed
below
- [x] Linting passes (`ruff check .`): on the touched files
- [ ] Type checking passes (`mypy headroom`): not run
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
# before the fix: the new tests fail on origin/main
# streaming 429, retry-after: 3600 → send.await_count 3, want 1 FAILED
# streaming 429, x-should-retry: false → send.await_count 3, want 1 FAILED
# first revision of this PR (single-wait budget): reachable-reset sequences fail
# 429 RA 0.031s → 429 RA 0.001s → 200, cap 20 ms → got 429 after 1 call, want 200 after 3 FAILED (both forwarders)
$ pytest -q tests/test_overload_retry_futile.py tests/test_proxy_streaming_ratelimit_headers.py \
tests/test_retry_preserve_upstream_status.py tests/test_5xx_accounting_all_providers.py \
tests/test_h2_stream_reset_retry.py tests/test_proxy_handler_helpers.py tests/test_proxy
505 passed in 38.40s
$ ruff check headroom/proxy <touched tests> → All checks passed!
$ ruff format --check <touched files> → 5 files already formatted
```
## Real Behavior Proof
- Environment: macOS 27, Python 3.11. `headroom proxy` started from a
source checkout of `origin/main` (before) and of this branch (after),
with `--retry-max-delay-ms 2000` (default 3 attempts),
`--anthropic-api-url` pointing at a local stub upstream, and an isolated
`HOME`. The checkouts have no compiled `headroom._core`, so both ran
with `HEADROOM_REQUIRE_RUST_CORE=false` (Python-only mode; the retry
path is pure Python).
- Exact command / steps: `POST /v1/messages` (non-streaming) through the
proxy against two stub scenarios:
- **exhausted**: every call → `429`, `Retry-After: 3600`
- **reachable**: `429 Retry-After: 3` → `429 Retry-After: 1` → `200`
- Observed result: exhausted 429 is forwarded after 1 upstream call in
0.25 s (was 3 calls, 4.29 s); a reachable reset still recovers to 200 in
3 calls (~3.3 s, unchanged):
```text
before exhausted -> status 429, upstream calls 3, 4.29s
after exhausted -> status 429, upstream calls 1, 0.25s
before reachable -> status 200, upstream calls 3, 3.28s
after reachable -> status 200, upstream calls 3, 3.27s
```
The exhausted case is forwarded at once. The reachable case keeps
recovering: `Retry-After: 3` is over one 2 s wait but inside the 2 × 2 s
budget.
- Not tested: live Anthropic traffic through the patched build, and the
streaming path end to end (covered by the streaming unit tests above).
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, narrowly. A 429/529 with
`x-should-retry: false` or a `Retry-After` above `retry_max_delay_ms` is
forwarded after 1 attempt instead of `retry_max_attempts`.
- Kill switch / disable path: raise `--retry-max-delay-ms` above the
upstream hint to restore retrying.
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the commit
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (no doc
covers the retry policy beyond the CLI help)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
Merge danger: two-way door. The blast radius is limited to the overload
retry path, and 5xx and transport retries are untouched. A 429 inside
the cap, such as a short per-minute limit with `Retry-After: 5`, still
retries.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3992"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
JerrettDavis
parent
76cceeb487
commit
a961a6fea0
@@ -18,6 +18,7 @@ from headroom.proxy.handlers._debug_dump import write_upstream_error_dump
|
||||
from headroom.proxy.helpers import (
|
||||
RETRYABLE_OVERLOAD_STATUSES,
|
||||
jitter_delay_ms,
|
||||
overload_retry_is_futile,
|
||||
retry_after_ms,
|
||||
)
|
||||
from headroom.proxy.token_counting import gemini_output_tokens
|
||||
@@ -1307,6 +1308,11 @@ class StreamingMixin:
|
||||
upstream_response.status_code in RETRYABLE_OVERLOAD_STATUSES
|
||||
and self.config.retry_enabled
|
||||
and attempt < retry_attempts - 1
|
||||
and not overload_retry_is_futile(
|
||||
upstream_response,
|
||||
self.config.retry_max_delay_ms,
|
||||
retries_left=retry_attempts - attempt - 1,
|
||||
)
|
||||
):
|
||||
delay_with_jitter = retry_after_ms(
|
||||
upstream_response, self.config.retry_max_delay_ms
|
||||
|
||||
@@ -1621,6 +1621,14 @@ def retry_after_ms(response: httpx.Response, max_ms: int) -> float | None:
|
||||
exponential backoff. Anthropic sends integer seconds; the HTTP-date branch
|
||||
covers other upstreams. Fails open on any parse error.
|
||||
"""
|
||||
seconds = _retry_after_seconds(response)
|
||||
if seconds is None:
|
||||
return None
|
||||
return min(seconds * 1000.0, float(max_ms))
|
||||
|
||||
|
||||
def _retry_after_seconds(response: httpx.Response) -> float | None:
|
||||
"""Uncapped, non-negative ``Retry-After`` in seconds, or ``None`` if absent/unparseable."""
|
||||
value = response.headers.get("retry-after")
|
||||
if not value:
|
||||
return None
|
||||
@@ -1635,7 +1643,25 @@ def retry_after_ms(response: httpx.Response, max_ms: int) -> float | None:
|
||||
seconds = (retry_at - datetime.now(retry_at.tzinfo)).total_seconds()
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
return min(max(seconds, 0.0) * 1000.0, float(max_ms))
|
||||
return max(seconds, 0.0)
|
||||
|
||||
|
||||
def overload_retry_is_futile(response: httpx.Response, max_ms: int, retries_left: int = 1) -> bool:
|
||||
"""True when retrying a 429/529 cannot succeed within the proxy's backoff.
|
||||
|
||||
Upstream says so explicitly with ``x-should-retry: false``, or implicitly with a
|
||||
``Retry-After`` beyond every wait still available: each of the ``retries_left``
|
||||
retries sleeps at most ``max_ms``, so a reset further out than
|
||||
``retries_left * max_ms`` is never reached and the retries only delay the same
|
||||
error (an exhausted subscription window answers with a reset hours away).
|
||||
Forwarding it at once lets the client, or a credential-rotating proxy in front,
|
||||
act on it. A reset within that window stays retryable: later 429s carry a
|
||||
shorter Retry-After as the reset approaches.
|
||||
"""
|
||||
if response.headers.get("x-should-retry", "").strip().lower() == "false":
|
||||
return True
|
||||
seconds = _retry_after_seconds(response)
|
||||
return seconds is not None and seconds * 1000.0 > max(retries_left, 0) * max_ms
|
||||
|
||||
|
||||
# Transient upstream statuses worth retrying with backoff: 429 (rate limit) and
|
||||
|
||||
@@ -150,6 +150,7 @@ from headroom.proxy.helpers import (
|
||||
_setup_file_logging, # noqa: F401
|
||||
is_anthropic_auth, # noqa: F401
|
||||
jitter_delay_ms,
|
||||
overload_retry_is_futile,
|
||||
resolve_display_provider,
|
||||
retry_after_ms,
|
||||
)
|
||||
@@ -2658,6 +2659,11 @@ class HeadroomProxy(
|
||||
if (
|
||||
not self.config.retry_enabled
|
||||
or attempt >= self.config.retry_max_attempts - 1
|
||||
or overload_retry_is_futile(
|
||||
response,
|
||||
self.config.retry_max_delay_ms,
|
||||
retries_left=self.config.retry_max_attempts - attempt - 1,
|
||||
)
|
||||
):
|
||||
return response
|
||||
delay_ms = retry_after_ms(
|
||||
|
||||
@@ -0,0 +1,114 @@
|
||||
"""A 429/529 must not be retried when the retry cannot succeed.
|
||||
|
||||
Anthropic answers an exhausted subscription or a hard rate limit with a 429 whose
|
||||
Retry-After points minutes or hours ahead, and marks non-retryable errors with
|
||||
``x-should-retry: false``. Retrying those after the capped backoff only holds the
|
||||
client for ``retry_max_attempts * retry_max_delay_ms`` before returning the same
|
||||
429, and every proxy layer chained in front multiplies that wait.
|
||||
|
||||
A Retry-After longer than one capped wait can still be reachable across the
|
||||
remaining retries (each later 429 carries a shorter hint), so the budget is
|
||||
``retries_left * retry_max_delay_ms``, not a single wait.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import types
|
||||
|
||||
import httpx
|
||||
|
||||
from headroom.proxy.helpers import overload_retry_is_futile
|
||||
from headroom.proxy.server import HeadroomProxy
|
||||
|
||||
|
||||
def _resp(status, headers=None):
|
||||
req = httpx.Request("POST", "https://api.anthropic.com/v1/messages")
|
||||
return httpx.Response(
|
||||
status_code=status, request=req, headers=headers or {}, content=b'{"type":"error"}'
|
||||
)
|
||||
|
||||
|
||||
class _FakeClient:
|
||||
def __init__(self, response):
|
||||
self._responses = response if isinstance(response, list) else None
|
||||
self._response = response
|
||||
self.calls = 0
|
||||
|
||||
async def post(self, url, content=None, headers=None):
|
||||
self.calls += 1
|
||||
if self._responses is not None:
|
||||
return self._responses[self.calls - 1]
|
||||
return self._response
|
||||
|
||||
|
||||
def _call(response, *, retry_max_delay_ms=30_000):
|
||||
proxy = HeadroomProxy.__new__(HeadroomProxy)
|
||||
client = _FakeClient(response)
|
||||
proxy.http_client = client
|
||||
proxy.config = types.SimpleNamespace(
|
||||
retry_enabled=True,
|
||||
retry_max_attempts=3,
|
||||
retry_base_delay_ms=1,
|
||||
retry_max_delay_ms=retry_max_delay_ms,
|
||||
)
|
||||
out = asyncio.run(
|
||||
proxy._retry_request("POST", "https://api.anthropic.com/v1/messages", {}, {}, stream=False)
|
||||
)
|
||||
return out, client.calls
|
||||
|
||||
|
||||
def test_retry_after_beyond_cap_is_futile():
|
||||
assert overload_retry_is_futile(_resp(429, {"retry-after": "3600"}), 30_000)
|
||||
|
||||
|
||||
def test_retry_after_within_cap_is_not_futile():
|
||||
assert not overload_retry_is_futile(_resp(429, {"retry-after": "5"}), 30_000)
|
||||
|
||||
|
||||
def test_retry_after_within_remaining_retry_window_is_not_futile():
|
||||
# 31 s > one 30 s wait, but two retries left can still reach the reset.
|
||||
response = _resp(429, {"retry-after": "31"})
|
||||
assert not overload_retry_is_futile(response, 30_000, retries_left=2)
|
||||
assert overload_retry_is_futile(response, 30_000, retries_left=1)
|
||||
assert overload_retry_is_futile(_resp(429, {"retry-after": "61"}), 30_000, retries_left=2)
|
||||
|
||||
|
||||
def test_should_retry_false_is_futile():
|
||||
assert overload_retry_is_futile(_resp(429, {"x-should-retry": "false"}), 30_000)
|
||||
|
||||
|
||||
def test_no_hint_is_not_futile():
|
||||
assert not overload_retry_is_futile(_resp(529), 30_000)
|
||||
|
||||
|
||||
def test_long_retry_after_429_returned_without_retry():
|
||||
out, calls = _call(_resp(429, {"retry-after": "3600"}))
|
||||
assert out.status_code == 429
|
||||
assert out.headers["retry-after"] == "3600"
|
||||
assert calls == 1
|
||||
|
||||
|
||||
def test_should_retry_false_429_returned_without_retry():
|
||||
out, calls = _call(_resp(429, {"x-should-retry": "false"}))
|
||||
assert out.status_code == 429
|
||||
assert calls == 1
|
||||
|
||||
|
||||
def test_short_retry_after_429_still_retried():
|
||||
out, calls = _call(_resp(429, {"retry-after": "0"}))
|
||||
assert out.status_code == 429
|
||||
assert calls == 3
|
||||
|
||||
|
||||
def test_retry_after_reachable_across_retries_recovers():
|
||||
# Reset 31 ms away with a 20 ms cap: wait 20 ms, the next 429 says 1 ms,
|
||||
# wait 1 ms, and the final attempt succeeds.
|
||||
out, calls = _call(
|
||||
[
|
||||
_resp(429, {"retry-after": "0.031"}),
|
||||
_resp(429, {"retry-after": "0.001"}),
|
||||
_resp(200),
|
||||
],
|
||||
retry_max_delay_ms=20,
|
||||
)
|
||||
assert out.status_code == 200
|
||||
assert calls == 3
|
||||
@@ -820,6 +820,90 @@ class TestStreamingRatelimitHeaderForwarding:
|
||||
assert snap.primary is not None
|
||||
assert snap.primary.used_percent == 99.5
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
("hint_headers", "expected_sends"),
|
||||
[
|
||||
({"retry-after": "3600"}, 1),
|
||||
({"x-should-retry": "false"}, 1),
|
||||
({}, 3),
|
||||
],
|
||||
)
|
||||
async def test_streaming_429_skips_futile_retries(self, hint_headers, expected_sends):
|
||||
"""A 429 whose reset lies beyond the backoff cap is forwarded at once."""
|
||||
proxy = self._create_mock_proxy()
|
||||
mock_response = self._create_mock_upstream_response()
|
||||
mock_response.status_code = 429
|
||||
mock_response.headers = httpx.Headers({"content-type": "application/json", **hint_headers})
|
||||
mock_response.aread = AsyncMock(
|
||||
return_value=b'{"type":"error","error":{"type":"rate_limit_error"}}'
|
||||
)
|
||||
mock_response.aclose = AsyncMock()
|
||||
|
||||
proxy.http_client.build_request = MagicMock(return_value=MagicMock())
|
||||
proxy.http_client.send = AsyncMock(return_value=mock_response)
|
||||
|
||||
result = await proxy._stream_response(
|
||||
url="https://api.anthropic.com/v1/messages",
|
||||
headers={"x-api-key": "sk-ant-test"},
|
||||
body={"model": "claude-sonnet-4-6", "stream": True, "messages": []},
|
||||
provider="anthropic",
|
||||
model="claude-sonnet-4-6",
|
||||
request_id="test-futile-429",
|
||||
original_tokens=10,
|
||||
optimized_tokens=10,
|
||||
tokens_saved=0,
|
||||
transforms_applied=[],
|
||||
tags={},
|
||||
optimization_latency=0.0,
|
||||
)
|
||||
|
||||
assert result.status_code == 429
|
||||
assert proxy.http_client.send.await_count == expected_sends
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_streaming_429_retry_after_reachable_across_retries_recovers(self):
|
||||
"""A reset beyond one capped wait but within the remaining retries is retried."""
|
||||
proxy = self._create_mock_proxy()
|
||||
proxy._config.retry_max_delay_ms = 20
|
||||
|
||||
def _rate_limited(retry_after):
|
||||
response = self._create_mock_upstream_response()
|
||||
response.status_code = 429
|
||||
response.headers = httpx.Headers(
|
||||
{"content-type": "application/json", "retry-after": retry_after}
|
||||
)
|
||||
response.aread = AsyncMock(return_value=b'{"type":"error"}')
|
||||
response.aclose = AsyncMock()
|
||||
return response
|
||||
|
||||
proxy.http_client.build_request = MagicMock(return_value=MagicMock())
|
||||
proxy.http_client.send = AsyncMock(
|
||||
side_effect=[
|
||||
_rate_limited("0.031"),
|
||||
_rate_limited("0.001"),
|
||||
self._create_mock_upstream_response(),
|
||||
]
|
||||
)
|
||||
|
||||
result = await proxy._stream_response(
|
||||
url="https://api.anthropic.com/v1/messages",
|
||||
headers={"x-api-key": "sk-ant-test"},
|
||||
body={"model": "claude-sonnet-4-6", "stream": True, "messages": []},
|
||||
provider="anthropic",
|
||||
model="claude-sonnet-4-6",
|
||||
request_id="test-reachable-429",
|
||||
original_tokens=10,
|
||||
optimized_tokens=10,
|
||||
tokens_saved=0,
|
||||
transforms_applied=[],
|
||||
tags={},
|
||||
optimization_latency=0.0,
|
||||
)
|
||||
|
||||
assert result.status_code == 200
|
||||
assert proxy.http_client.send.await_count == 3
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_anthropic_stream_leaves_codex_state_untouched(self):
|
||||
"""The now-unconditional capture must be a no-op for non-Codex streams."""
|
||||
|
||||
Reference in New Issue
Block a user