mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-27 07:03:17 +08:00
John T. Myers
355d845d3e
* fix(inference): prevent silent truncation of large streaming responses The L7 inference proxy silently dropped tool_calls from large streaming responses due to an aggressive 30s per-chunk idle timeout and a reqwest total-request timeout that capped the entire body stream. Reasoning models that pause during "thinking" phases triggered these timeouts, producing valid-looking but truncated HTTP responses with no client-visible error. - Extract prepare_backend_request() helper and create a streaming variant that omits the total request timeout; body stream liveness is now enforced solely by the per-chunk idle timeout - Add 30s connect_timeout to the reqwest Client builder - Increase CHUNK_IDLE_TIMEOUT from 30s to 120s for reasoning models - Inject SSE error events (proxy_stream_error) before the HTTP chunked terminator on all truncation paths so clients can detect data loss - Wrap the streaming relay in BufWriter to reduce per-chunk TLS flush overhead - Bump OCSF severity for streaming truncation from Low to Medium Closes #829 * fix(inference): drop BufWriter to preserve SSE incremental delivery The BufWriter introduced in the previous commit buffered SSE frames until the 16KB capacity filled or the stream ended, defeating incremental token delivery and potentially reintroducing client-visible timeouts on healthy streams. Revert to per-chunk write_all+flush but keep the single format_chunk() call that coalesces framing into one write. Also fix the streaming integration test: add a 3s mock delay that exceeds the 1s route timeout so the test actually validates that the streaming path omits the total request timeout. Previously the mock responded immediately, passing regardless of timeout behavior.
openshell-router
openshell-router is the inference routing and upstream execution engine used by openshell-server.
Responsibilities
- Select an upstream route from a candidate set (based on protocol compatibility).
- Forward raw HTTP requests to the selected upstream backend.
- Normalize upstream failures into router-level errors (
unauthorized,unavailable, protocol/internal errors). - Keep routing decision logic in one place so strategies can evolve (fallbacks, scoring, health-based routing).
Non-responsibilities
- Authentication and sandbox identity.
- Authorization and policy enforcement.
- Persistence of routes/entities.
- Loading sandbox or policy objects.
These are owned by openshell-server.
Integration contract with openshell-server
Current split:
openshell-server:- authenticates request origin
- resolves cluster-managed inference route candidates from providers
- loads enabled route candidates from the entity store
openshell-router:- picks a route from candidates (
proxy_with_candidates) - forwards the HTTP request upstream and returns the raw response
- picks a route from candidates (
Public APIs
Router::proxy_with_candidates(source_protocol, method, path, headers, body, &[ResolvedRoute])- Filters candidates by protocol compatibility, then forwards the request to the first match.
- Preferred path for entity-driven server routing.
Notes
- Route selection matches candidates by
protocolfield (e.g.openai_chat_completions). - Route selection is intentionally simple and will evolve.