mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-28 07:13:04 +08:00
Piotr Mlocek
f869182d8a
> **🏗️ build-from-issue-agent** Closes #79 ## Summary Replaces the gateway-proxied inference model (`ProxyInference` RPC) with sandbox-local execution. The sandbox now resolves routes from either a standalone YAML route file or a cluster bundle fetched via the new `GetSandboxInferenceBundle` RPC, then forwards requests directly to inference backends using `navigator-router`. ### Benefits of removing the gRPC hop Moving inference execution from the gateway to the sandbox eliminates the gRPC round-trip for every inference request: - **No payload size limits**: Direct HTTP forwarding replaces protobuf-over-gRPC serialization. - **Streaming-ready**: Preserves connection semantics for future SSE support (e.g., `stream: true`). - **Lower latency**: Direct sandbox → backend path instead of sandbox → gateway → backend round-trip. - **Reduced gateway load**: Gateway only serves the control-plane bundle delivery RPC. - **Simpler error model**: HTTP status codes flow through directly without gRPC wrapping. ## Changes Made - **Proto**: Removed `ProxyInference` RPC. Added `GetSandboxInferenceBundle` RPC for route bundle delivery. - **Gateway**: Replaced `proxy_inference` with `get_sandbox_inference_bundle`. Removed `Router` and `navigator-router` dependency — gateway is now control-plane only for inference. - **Router**: Switched config file format from TOML to YAML. Added `Debug` impl that redacts API keys. - **Sandbox**: New `InferenceContext` with local `Router` + route cache. Routes loaded from `--inference-routes` file (standalone) or cluster bundle via gRPC, with 30s background refresh. New `--inference-routes` / `NAVIGATOR_INFERENCE_ROUTES` CLI arg. - **Dev sandbox**: Added `inference-routes.yaml` at repo root with default NVIDIA NIM route. `mise run sandbox` now mounts it automatically and supports `-e` flag to forward env vars (e.g., `mise run sandbox`). - **Example**: Added standalone example (`examples/inference/routes.yaml`) and rewrote README to cover both standalone and cluster workflows. - **E2E tests**: Updated existing test names/docs. Added tests for Anthropic messages protocol and multi-route policy filtering. ## Tests Added - **Unit (21 new):** `router_error_to_http` (6 variants), `load_from_file` YAML round-trip (5), `resolve_sandbox_inference_bundle` gRPC handler (6), `build_inference_context` route loading (4) - **E2E (2 new):** Anthropic messages protocol routing, route filtering by `allowed_routes` - **Total:** 188 unit tests passing across 3 crates ## Documentation Updated - `architecture/gateway.md`: Removed router, updated Inference Service - `architecture/sandbox.md`: Updated orchestration flow, InferenceContext, route loading - `architecture/inference-routing.md`: Complete rewrite for sandbox-local architecture ## Verification - [x] All unit tests passing (188 across 3 crates) - [x] Pre-commit Rust checks passing (fmt, clippy, tests) - [x] Architecture documentation updated - [x] E2E tests updated with new test cases - [x] Standalone example and documentation added - [x] Default inference-routes.yaml with env passthrough for mise run sandbox