John T. Myers 355d845d3e fix(inference): prevent silent truncation of large streaming responses (#834) ...

* fix(inference): prevent silent truncation of large streaming responses

The L7 inference proxy silently dropped tool_calls from large streaming
responses due to an aggressive 30s per-chunk idle timeout and a reqwest
total-request timeout that capped the entire body stream. Reasoning models
that pause during "thinking" phases triggered these timeouts, producing
valid-looking but truncated HTTP responses with no client-visible error.

- Extract prepare_backend_request() helper and create a streaming variant
  that omits the total request timeout; body stream liveness is now
  enforced solely by the per-chunk idle timeout
- Add 30s connect_timeout to the reqwest Client builder
- Increase CHUNK_IDLE_TIMEOUT from 30s to 120s for reasoning models
- Inject SSE error events (proxy_stream_error) before the HTTP chunked
  terminator on all truncation paths so clients can detect data loss
- Wrap the streaming relay in BufWriter to reduce per-chunk TLS flush
  overhead
- Bump OCSF severity for streaming truncation from Low to Medium

Closes #829

* fix(inference): drop BufWriter to preserve SSE incremental delivery

The BufWriter introduced in the previous commit buffered SSE frames until
the 16KB capacity filled or the stream ended, defeating incremental token
delivery and potentially reintroducing client-visible timeouts on healthy
streams. Revert to per-chunk write_all+flush but keep the single
format_chunk() call that coalesces framing into one write.

Also fix the streaming integration test: add a 3s mock delay that exceeds
the 1s route timeout so the test actually validates that the streaming
path omits the total request timeout. Previously the mock responded
immediately, passing regardless of timeout behavior.

2026-04-14 13:34:39 -07:00

..

backend_integration.rs

fix(inference): prevent silent truncation of large streaming responses (#834)

2026-04-14 13:34:39 -07:00