mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 12:05:34 +08:00
## Description Partially addresses #3259. `write` inherited `request_timeout_seconds`, so pushing request bytes got the same 300s budget as waiting for a model to answer: ```python httpx.Timeout( connect=config.connect_timeout_seconds, # 10 read=config.request_timeout_seconds, # 300 write=config.request_timeout_seconds, # 300 <- same knob pool=config.connect_timeout_seconds, # 10 ) ``` Sending a request and waiting for a model to think are different operations, and conflating them left the send **effectively unbounded in practice**. When an upstream stops draining — a pooled socket whose peer went away over a laptop sleep, a network change, a NAT timeout — the send blocks until the OS abandons retransmission, ~180-220s on macOS. That is *under* the inherited 300s, so httpx never timed out and the OS ended the request instead. The retry then hit the same wall and stalled again, which is why one incident blocked the reporter's client for 5-10 minutes. `--connect-timeout-seconds` cannot help, exactly as the reporter says: it only guards establishing a *fresh* connection, and a reused one never connects. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Added `ProxyConfig.write_timeout_seconds` (default **150**), with `--write-timeout-seconds` / `HEADROOM_WRITE_TIMEOUT_SECONDS` and a `SettingField` so it appears in the settings registry. - Used it in `_provider_httpx_client_options` **and** in `_anthropic_buffered_request_timeout` — the buffered path sets its own timeout and is the path most likely to carry a large body, so it needed the split too or it would have kept the unbounded write. Behaviour after: ``` BEFORE: connect=10 read=300 write=300 pool=10 AFTER: connect=10 read=300 write=150 pool=10 ``` ### What `write` actually bounds, and why the default is 150s The first revision of this PR used 60s and claimed httpx applies `write` per write operation, so a large body would be unaffected. **That was wrong, and I measured it before shipping the correction.** httpx hands a bytes body to the transport as a *single* write (`ByteStream.__aiter__` yields the whole body; httpcore's h11 `_send_request_body` wraps each yielded chunk in one `anyio.fail_after(write)`), so on HTTP/1.1 the entire upload runs inside one timer. Against a local peer draining a 16MB body at a steady ~1MB/s: ``` write=3.0: WriteTimeout after 3.0s <- healthy peer, still cut off write=60.0: send completed in 19.1s ``` So this is a bound on total upload time, not a stall detector. 60s would have cut off the 7-15MB bodies the reporter sends over a slow uplink — turning a fix into an outage for exactly the person who filed the issue. 150s is the value that satisfies both real constraints: it carries 15MB (the largest body #3259 reports) over a ~1 Mbps uplink, and still fires well before the ~180s OS retransmit ceiling that made the inherited 300s unreachable. `tests/test_upstream_write_timeout.py` pins both ends of that range so neither can drift. On HTTP/2 the body is split by flow control and the waiting-for-window part is charged to `read` (httpcore `_wait_for_outgoing_flow` → `_receive_events`), so only real socket writes count against the bound there. A genuinely per-chunk bound would separate "dead peer" from "slow peer" perfectly, but it needs the body fed as an iterator without changing the wire framing. That is a larger change than this fix warrants; worth a follow-up if slow-uplink users hit the 150s cap. ### Why the read timeout is deliberately left alone Time-to-first-byte is a legitimately long wait for a model, and the buffered Anthropic path already sets its own 600s read for exactly that reason. Lowering `read` globally would break slow legitimate responses to fix a rare hang — the wrong trade. Operators who want a tighter read bound already have `--request-timeout-seconds`; before this PR that knob also silently shortened the write budget, which is the conflation being removed. ### Failover `_retry_request` catches `httpx.TransportError`, and `WriteTimeout` is a `TimeoutException` under it — so a bounded write drops the bad connection and re-sends on a fresh one instead of stalling behind a socket whose peer is gone. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (CI-pinned `ruff` 0.16.3) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output ```text $ pytest tests/test_proxy/ tests/test_proxy_hardening.py tests/test_upstream_write_timeout.py 302 passed in 40.46s $ pytest tests/test_upstream_write_timeout.py tests/test_proxy/test_anthropic_buffered_timeout.py \ tests/test_proxy/test_settings_store.py tests/test_cli_proxy_env.py tests/test_cli_proxy_improvements.py 181 passed, 1 skipped in 11.65s ``` ## Scope This does **not** explain the reporter's follow-up: consistent ~60s stalls on 7-15MB bodies with `HEADROOM_MAX_KEEPALIVE=0`, where there is no pooled socket to inherit. I would not close #3259 on this PR. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
110 lines
3.9 KiB
Python
110 lines
3.9 KiB
Python
"""The write timeout must be independent of the read timeout.
|
|
|
|
`write` used to inherit `request_timeout_seconds`, so pushing request bytes got
|
|
the same budget as waiting for a model to answer. That left the write phase
|
|
effectively unbounded against a dead peer: when an upstream stops draining, the
|
|
send blocks until the OS abandons retransmission (~180-220s on macOS), which is
|
|
*under* the 300s it inherited — so no timeout fired, the request hung, the retry
|
|
hung again, and a single incident blocked the client for minutes (#3259).
|
|
|
|
Separating them is what lets the existing retry logic fail over to a fresh
|
|
connection instead of stalling behind a socket whose peer is gone.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from headroom.proxy.models import ProxyConfig
|
|
from headroom.proxy.server import _provider_httpx_client_options
|
|
|
|
|
|
def _timeout(config: ProxyConfig):
|
|
_http2, kwargs = _provider_httpx_client_options(config, verify=True)
|
|
return kwargs["timeout"]
|
|
|
|
|
|
def test_write_does_not_inherit_the_read_budget() -> None:
|
|
"""The regression itself: a long read budget must not extend the send."""
|
|
timeout = _timeout(ProxyConfig(request_timeout_seconds=300))
|
|
|
|
assert timeout.read == 300
|
|
assert timeout.write != 300
|
|
assert timeout.write == ProxyConfig().write_timeout_seconds
|
|
|
|
|
|
def test_write_timeout_defaults_below_the_os_retransmit_ceiling() -> None:
|
|
"""A default above ~180s would never fire before the OS gave up anyway.
|
|
|
|
The whole failure mode is that the inherited 300s sat above the point where
|
|
macOS abandons retransmission, so the timeout was unreachable in practice.
|
|
"""
|
|
assert ProxyConfig().write_timeout_seconds < 180
|
|
|
|
|
|
def test_write_timeout_default_can_carry_a_large_body() -> None:
|
|
"""The bound covers the whole upload, so it has to fit a real request.
|
|
|
|
httpx hands a bytes body to the transport as one write, so on HTTP/1.1 the
|
|
entire body is sent inside a single timer -- this is NOT a per-chunk budget.
|
|
Measured against a peer draining a 16MB body at ~1MB/s, WriteTimeout fires
|
|
at exactly the configured bound even though the peer is healthy. #3259's
|
|
reporter sends 7-15MB bodies, so a default that cannot carry 15MB over a
|
|
modest uplink would turn this fix into an outage for them.
|
|
"""
|
|
budget = ProxyConfig().write_timeout_seconds
|
|
largest_reported_body_bytes = 15 * 1024 * 1024
|
|
slow_uplink_bytes_per_second = 125 * 1024 # ~1 Mbps
|
|
|
|
assert largest_reported_body_bytes / slow_uplink_bytes_per_second <= budget
|
|
|
|
|
|
def test_write_timeout_is_configurable() -> None:
|
|
timeout = _timeout(ProxyConfig(write_timeout_seconds=15))
|
|
|
|
assert timeout.write == 15
|
|
|
|
|
|
def test_other_phases_are_unchanged() -> None:
|
|
"""Only `write` moves; connect/read/pool keep the values they always had."""
|
|
config = ProxyConfig(
|
|
request_timeout_seconds=300,
|
|
connect_timeout_seconds=10,
|
|
)
|
|
|
|
timeout = _timeout(config)
|
|
|
|
assert timeout.connect == 10
|
|
assert timeout.read == 300
|
|
assert timeout.pool == 10
|
|
|
|
|
|
@pytest.mark.parametrize("buffered_read", [600, 900])
|
|
def test_buffered_anthropic_turn_keeps_its_long_read_but_bounded_write(
|
|
buffered_read: int,
|
|
) -> None:
|
|
"""A buffered turn waits longer for the answer, not longer to send.
|
|
|
|
This path sets its own timeout, so it needs the split applied too —
|
|
otherwise the one path most likely to carry a large body keeps the
|
|
unbounded write.
|
|
"""
|
|
from headroom.proxy.handlers.anthropic import AnthropicHandlerMixin
|
|
|
|
class _Handler(AnthropicHandlerMixin):
|
|
def __init__(self, config: ProxyConfig) -> None:
|
|
self.config = config
|
|
|
|
handler = _Handler(
|
|
ProxyConfig(
|
|
anthropic_buffered_request_timeout_seconds=buffered_read,
|
|
request_timeout_seconds=300,
|
|
write_timeout_seconds=60,
|
|
)
|
|
)
|
|
|
|
timeout = handler._anthropic_buffered_request_timeout()
|
|
|
|
assert timeout.read == buffered_read
|
|
assert timeout.write == 60
|