Commit Graph
1 Commits
Author SHA1 Message Date
Tejas ChopraandClaude Opus 5 a507249b92 fix(proxy): give the upstream write phase its own timeout (#3327)
## Description

Partially addresses #3259.

`write` inherited `request_timeout_seconds`, so pushing request bytes
got the same 300s budget as waiting for a model to answer:

```python
httpx.Timeout(
    connect=config.connect_timeout_seconds,   # 10
    read=config.request_timeout_seconds,      # 300
    write=config.request_timeout_seconds,     # 300  <- same knob
    pool=config.connect_timeout_seconds,      # 10
)
```

Sending a request and waiting for a model to think are different
operations, and conflating them left the send **effectively unbounded in
practice**. When an upstream stops draining — a pooled socket whose peer
went away over a laptop sleep, a network change, a NAT timeout — the
send blocks until the OS abandons retransmission, ~180-220s on macOS.
That is *under* the inherited 300s, so httpx never timed out and the OS
ended the request instead. The retry then hit the same wall and stalled
again, which is why one incident blocked the reporter's client for 5-10
minutes.

`--connect-timeout-seconds` cannot help, exactly as the reporter says:
it only guards establishing a *fresh* connection, and a reused one never
connects.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Added `ProxyConfig.write_timeout_seconds` (default **150**), with
`--write-timeout-seconds` / `HEADROOM_WRITE_TIMEOUT_SECONDS` and a
`SettingField` so it appears in the settings registry.
- Used it in `_provider_httpx_client_options` **and** in
`_anthropic_buffered_request_timeout` — the buffered path sets its own
timeout and is the path most likely to carry a large body, so it needed
the split too or it would have kept the unbounded write.

Behaviour after:

```
BEFORE:  connect=10  read=300  write=300  pool=10
AFTER:   connect=10  read=300  write=150  pool=10
```

### What `write` actually bounds, and why the default is 150s

The first revision of this PR used 60s and claimed httpx applies `write`
per write operation, so a large body would be unaffected. **That was
wrong, and I measured it before shipping the correction.** httpx hands a
bytes body to the transport as a *single* write (`ByteStream.__aiter__`
yields the whole body; httpcore's h11 `_send_request_body` wraps each
yielded chunk in one `anyio.fail_after(write)`), so on HTTP/1.1 the
entire upload runs inside one timer.

Against a local peer draining a 16MB body at a steady ~1MB/s:

```
write=3.0:   WriteTimeout after 3.0s      <- healthy peer, still cut off
write=60.0:  send completed in 19.1s
```

So this is a bound on total upload time, not a stall detector. 60s would
have cut off the 7-15MB bodies the reporter sends over a slow uplink —
turning a fix into an outage for exactly the person who filed the issue.

150s is the value that satisfies both real constraints: it carries 15MB
(the largest body #3259 reports) over a ~1 Mbps uplink, and still fires
well before the ~180s OS retransmit ceiling that made the inherited 300s
unreachable. `tests/test_upstream_write_timeout.py` pins both ends of
that range so neither can drift.

On HTTP/2 the body is split by flow control and the waiting-for-window
part is charged to `read` (httpcore `_wait_for_outgoing_flow` →
`_receive_events`), so only real socket writes count against the bound
there.

A genuinely per-chunk bound would separate "dead peer" from "slow peer"
perfectly, but it needs the body fed as an iterator without changing the
wire framing. That is a larger change than this fix warrants; worth a
follow-up if slow-uplink users hit the 150s cap.

### Why the read timeout is deliberately left alone

Time-to-first-byte is a legitimately long wait for a model, and the
buffered Anthropic path already sets its own 600s read for exactly that
reason. Lowering `read` globally would break slow legitimate responses
to fix a rare hang — the wrong trade. Operators who want a tighter read
bound already have `--request-timeout-seconds`; before this PR that knob
also silently shortened the write budget, which is the conflation being
removed.

### Failover

`_retry_request` catches `httpx.TransportError`, and `WriteTimeout` is a
`TimeoutException` under it — so a bounded write drops the bad
connection and re-sends on a fresh one instead of stalling behind a
socket whose peer is gone.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (CI-pinned `ruff` 0.16.3)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
$ pytest tests/test_proxy/ tests/test_proxy_hardening.py tests/test_upstream_write_timeout.py
302 passed in 40.46s

$ pytest tests/test_upstream_write_timeout.py tests/test_proxy/test_anthropic_buffered_timeout.py \
         tests/test_proxy/test_settings_store.py tests/test_cli_proxy_env.py tests/test_cli_proxy_improvements.py
181 passed, 1 skipped in 11.65s
```

## Scope

This does **not** explain the reporter's follow-up: consistent ~60s
stalls on 7-15MB bodies with `HEADROOM_MAX_KEEPALIVE=0`, where there is
no pooled socket to inherit. I would not close #3259 on this PR.

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-28 18:16:58 +05:30