Files
headroom/tests/test_upstream_write_timeout.py
Tejas ChopraandClaude Opus 5 a507249b92 fix(proxy): give the upstream write phase its own timeout (#3327)
## Description

Partially addresses #3259.

`write` inherited `request_timeout_seconds`, so pushing request bytes
got the same 300s budget as waiting for a model to answer:

```python
httpx.Timeout(
    connect=config.connect_timeout_seconds,   # 10
    read=config.request_timeout_seconds,      # 300
    write=config.request_timeout_seconds,     # 300  <- same knob
    pool=config.connect_timeout_seconds,      # 10
)
```

Sending a request and waiting for a model to think are different
operations, and conflating them left the send **effectively unbounded in
practice**. When an upstream stops draining — a pooled socket whose peer
went away over a laptop sleep, a network change, a NAT timeout — the
send blocks until the OS abandons retransmission, ~180-220s on macOS.
That is *under* the inherited 300s, so httpx never timed out and the OS
ended the request instead. The retry then hit the same wall and stalled
again, which is why one incident blocked the reporter's client for 5-10
minutes.

`--connect-timeout-seconds` cannot help, exactly as the reporter says:
it only guards establishing a *fresh* connection, and a reused one never
connects.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Added `ProxyConfig.write_timeout_seconds` (default **150**), with
`--write-timeout-seconds` / `HEADROOM_WRITE_TIMEOUT_SECONDS` and a
`SettingField` so it appears in the settings registry.
- Used it in `_provider_httpx_client_options` **and** in
`_anthropic_buffered_request_timeout` — the buffered path sets its own
timeout and is the path most likely to carry a large body, so it needed
the split too or it would have kept the unbounded write.

Behaviour after:

```
BEFORE:  connect=10  read=300  write=300  pool=10
AFTER:   connect=10  read=300  write=150  pool=10
```

### What `write` actually bounds, and why the default is 150s

The first revision of this PR used 60s and claimed httpx applies `write`
per write operation, so a large body would be unaffected. **That was
wrong, and I measured it before shipping the correction.** httpx hands a
bytes body to the transport as a *single* write (`ByteStream.__aiter__`
yields the whole body; httpcore's h11 `_send_request_body` wraps each
yielded chunk in one `anyio.fail_after(write)`), so on HTTP/1.1 the
entire upload runs inside one timer.

Against a local peer draining a 16MB body at a steady ~1MB/s:

```
write=3.0:   WriteTimeout after 3.0s      <- healthy peer, still cut off
write=60.0:  send completed in 19.1s
```

So this is a bound on total upload time, not a stall detector. 60s would
have cut off the 7-15MB bodies the reporter sends over a slow uplink —
turning a fix into an outage for exactly the person who filed the issue.

150s is the value that satisfies both real constraints: it carries 15MB
(the largest body #3259 reports) over a ~1 Mbps uplink, and still fires
well before the ~180s OS retransmit ceiling that made the inherited 300s
unreachable. `tests/test_upstream_write_timeout.py` pins both ends of
that range so neither can drift.

On HTTP/2 the body is split by flow control and the waiting-for-window
part is charged to `read` (httpcore `_wait_for_outgoing_flow` →
`_receive_events`), so only real socket writes count against the bound
there.

A genuinely per-chunk bound would separate "dead peer" from "slow peer"
perfectly, but it needs the body fed as an iterator without changing the
wire framing. That is a larger change than this fix warrants; worth a
follow-up if slow-uplink users hit the 150s cap.

### Why the read timeout is deliberately left alone

Time-to-first-byte is a legitimately long wait for a model, and the
buffered Anthropic path already sets its own 600s read for exactly that
reason. Lowering `read` globally would break slow legitimate responses
to fix a rare hang — the wrong trade. Operators who want a tighter read
bound already have `--request-timeout-seconds`; before this PR that knob
also silently shortened the write budget, which is the conflation being
removed.

### Failover

`_retry_request` catches `httpx.TransportError`, and `WriteTimeout` is a
`TimeoutException` under it — so a bounded write drops the bad
connection and re-sends on a fresh one instead of stalling behind a
socket whose peer is gone.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (CI-pinned `ruff` 0.16.3)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
$ pytest tests/test_proxy/ tests/test_proxy_hardening.py tests/test_upstream_write_timeout.py
302 passed in 40.46s

$ pytest tests/test_upstream_write_timeout.py tests/test_proxy/test_anthropic_buffered_timeout.py \
         tests/test_proxy/test_settings_store.py tests/test_cli_proxy_env.py tests/test_cli_proxy_improvements.py
181 passed, 1 skipped in 11.65s
```

## Scope

This does **not** explain the reporter's follow-up: consistent ~60s
stalls on 7-15MB bodies with `HEADROOM_MAX_KEEPALIVE=0`, where there is
no pooled socket to inherit. I would not close #3259 on this PR.

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-28 18:16:58 +05:30

110 lines
3.9 KiB
Python

"""The write timeout must be independent of the read timeout.
`write` used to inherit `request_timeout_seconds`, so pushing request bytes got
the same budget as waiting for a model to answer. That left the write phase
effectively unbounded against a dead peer: when an upstream stops draining, the
send blocks until the OS abandons retransmission (~180-220s on macOS), which is
*under* the 300s it inherited — so no timeout fired, the request hung, the retry
hung again, and a single incident blocked the client for minutes (#3259).
Separating them is what lets the existing retry logic fail over to a fresh
connection instead of stalling behind a socket whose peer is gone.
"""
from __future__ import annotations
import pytest
from headroom.proxy.models import ProxyConfig
from headroom.proxy.server import _provider_httpx_client_options
def _timeout(config: ProxyConfig):
_http2, kwargs = _provider_httpx_client_options(config, verify=True)
return kwargs["timeout"]
def test_write_does_not_inherit_the_read_budget() -> None:
"""The regression itself: a long read budget must not extend the send."""
timeout = _timeout(ProxyConfig(request_timeout_seconds=300))
assert timeout.read == 300
assert timeout.write != 300
assert timeout.write == ProxyConfig().write_timeout_seconds
def test_write_timeout_defaults_below_the_os_retransmit_ceiling() -> None:
"""A default above ~180s would never fire before the OS gave up anyway.
The whole failure mode is that the inherited 300s sat above the point where
macOS abandons retransmission, so the timeout was unreachable in practice.
"""
assert ProxyConfig().write_timeout_seconds < 180
def test_write_timeout_default_can_carry_a_large_body() -> None:
"""The bound covers the whole upload, so it has to fit a real request.
httpx hands a bytes body to the transport as one write, so on HTTP/1.1 the
entire body is sent inside a single timer -- this is NOT a per-chunk budget.
Measured against a peer draining a 16MB body at ~1MB/s, WriteTimeout fires
at exactly the configured bound even though the peer is healthy. #3259's
reporter sends 7-15MB bodies, so a default that cannot carry 15MB over a
modest uplink would turn this fix into an outage for them.
"""
budget = ProxyConfig().write_timeout_seconds
largest_reported_body_bytes = 15 * 1024 * 1024
slow_uplink_bytes_per_second = 125 * 1024 # ~1 Mbps
assert largest_reported_body_bytes / slow_uplink_bytes_per_second <= budget
def test_write_timeout_is_configurable() -> None:
timeout = _timeout(ProxyConfig(write_timeout_seconds=15))
assert timeout.write == 15
def test_other_phases_are_unchanged() -> None:
"""Only `write` moves; connect/read/pool keep the values they always had."""
config = ProxyConfig(
request_timeout_seconds=300,
connect_timeout_seconds=10,
)
timeout = _timeout(config)
assert timeout.connect == 10
assert timeout.read == 300
assert timeout.pool == 10
@pytest.mark.parametrize("buffered_read", [600, 900])
def test_buffered_anthropic_turn_keeps_its_long_read_but_bounded_write(
buffered_read: int,
) -> None:
"""A buffered turn waits longer for the answer, not longer to send.
This path sets its own timeout, so it needs the split applied too —
otherwise the one path most likely to carry a large body keeps the
unbounded write.
"""
from headroom.proxy.handlers.anthropic import AnthropicHandlerMixin
class _Handler(AnthropicHandlerMixin):
def __init__(self, config: ProxyConfig) -> None:
self.config = config
handler = _Handler(
ProxyConfig(
anthropic_buffered_request_timeout_seconds=buffered_read,
request_timeout_seconds=300,
write_timeout_seconds=60,
)
)
timeout = handler._anthropic_buffered_request_timeout()
assert timeout.read == buffered_read
assert timeout.write == 60