* fix(sandbox): add GPU device nodes and nvidia-persistenced to landlock baseline
Landlock READ_FILE/WRITE_FILE restricts open(2) on character device files
even when DAC permissions would otherwise allow it. GPU sandboxes need
/dev/nvidiactl, /dev/nvidia-uvm, /dev/nvidia-uvm-tools, /dev/nvidia-modeset,
and per-GPU /dev/nvidiaX nodes in the policy to allow NVML initialization.
Additionally, CDI bind-mounts /run/nvidia-persistenced/socket into the
container. NVML tries to connect to this socket at init time; if the
directory is not in the landlock policy, it receives EACCES (not
ECONNREFUSED), which causes NVML to abort with NVML_ERROR_INSUFFICIENT_PERMISSIONS
even though nvidia-persistenced is optional.
Both classes of paths are auto-added to the baseline when /dev/nvidiactl is
present. Per-GPU device nodes are enumerated at runtime to handle multi-GPU
configurations.
Check for existing sandbox name before persisting, matching the
provider-creation pattern. The CLI now surfaces a clear hint instead
of a raw UNIQUE constraint error.
Closes#691
* fix(l7): reject ambiguous HTTP framing in REST proxy (SEC-009)
Harden the L7 REST proxy HTTP parser against request smuggling:
- Reject requests containing both Content-Length and Transfer-Encoding
headers per RFC 7230 Section 3.3.3 (CL/TE ambiguity)
- Replace String::from_utf8_lossy with strict UTF-8 validation to
prevent interpretation gaps with upstream servers
- Reject bare LF line endings (require CRLF per HTTP spec)
- Validate HTTP version string (HTTP/1.0 or HTTP/1.1 only)
* fix(server): harden shell_escape and command construction (SEC-002)
Harden the gRPC exec handler against command injection via the
structured-to-shell-string conversion:
- Reject null bytes and newlines/carriage returns in shell_escape()
- Add input validation in exec_sandbox: reject control characters in
command args, env values, and workdir
- Enforce size limits: max 1024 args, 32 KiB per arg/value, 4 KiB
workdir, 256 KiB total assembled command string
- Change shell_escape and build_remote_exec_command to return Result
so callers must handle validation failures
* fix(server): add command validation at SSH transport boundary (SEC-003)
Add defense-in-depth validation in run_exec_with_russh before sending
the command to the sandbox SSH server:
- Reject null bytes in command string at transport boundary
- Enforce max command length (256 KiB) at transport boundary
- Enhance stream_exec_over_ssh logging with command length, stdin
length, and truncated command preview for audit trail
* fix(sandbox): validate port range and extend loopback check in SSH (SEC-007)
Harden the SSH direct-tcpip channel handler:
- Validate port_to_connect <= 65535 before u32-to-u16 cast to prevent
port truncation (e.g., 65537 becoming port 1)
- Replace string-literal loopback check with is_loopback_host() that
covers the full 127.0.0.0/8 range, IPv4-mapped IPv6 (::ffff:127.x),
bracketed IPv6, and case-insensitive localhost
- Remove #[allow(clippy::cast_possible_truncation)] since the cast is
now proven safe by the preceding range check
* fix(sandbox): sanitize inference error messages returned to sandbox (SEC-008)
Replace verbatim internal error strings in router_error_to_http with
generic messages to prevent information leakage to sandboxed code.
Upstream URLs, internal hostnames, TLS details, and file paths are no
longer exposed. Full error context is still logged server-side at warn
level by the caller for debugging.
* fix(server): block internal IPs in SSH proxy target validation (SEC-006)
Add IP validation in start_single_use_ssh_proxy to prevent SSRF if a
sandbox status record were poisoned:
- Resolve DNS before connecting and validate the resolved IP
- Block loopback (127.0.0.0/8) and link-local (169.254.0.0/16, covers
cloud metadata endpoint) addresses
- Block IPv4-mapped IPv6 variants of the same ranges
- Connect to the validated SocketAddr directly to prevent TOCTOU
- Add debug logging of resolved target IP for audit
* fix(sandbox): add resource limits to chunked body parser (SEC-010)
Harden parse_chunked_body in the inference interception path:
- Replace all unchecked +2 additions with checked_add for consistent
overflow safety across all target architectures
- Add MAX_CHUNKED_BODY (10 MiB) to cap decoded body size
- Add MAX_CHUNK_COUNT (4096) to prevent CPU exhaustion via tiny chunks
- Early-reject chunk sizes larger than remaining buffer space
* fix(cli): double-escape command for SSH path, validate host and name (SEC-004)
Harden doctor_exec against command injection in the SSH remote path:
- Apply shell_escape to inner_cmd in the SSH path so it survives the
double shell interpretation (SSH remote shell + sh -lc). This also
fixes a correctness bug where multi-word commands were silently
broken in the SSH path.
- Add validate_gateway_name to reject shell metacharacters in gateway
names before use in container_name
- Add validate_ssh_host to reject metacharacters in remote_host loaded
from metadata.json
* fix(sandbox): add CIDR breadth warning and control-plane port blocklist (SEC-005)
Defense-in-depth for the allowed_ips feature:
- Log a warning when a CIDR entry has a prefix length < /16, as overly
broad ranges may unintentionally expose control-plane services
- Block K8s API (6443), etcd (2379/2380), and kubelet (10250/10255)
ports unconditionally in resolve_and_check_allowed_ips, even when
the resolved IP matches an allowed_ips entry
* test(e2e): update assertion for sanitized inference error message (SEC-008)
The SEC-008 fix changed the error message from 'no compatible route
for source protocol ...' to 'no compatible inference route available'.
Update the E2E assertion substring to match.
Closes#101
Add pytest-xdist for parallel e2e test execution with configurable
concurrency. Default to 5 workers; override via E2E_PARALLEL env var
(accepts a number or 'auto' for CPU-count matching). Make session-scoped
mock inference route fixtures worker-safe by incorporating the xdist
worker_id into route names and routing hints.
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
Closes#42
- Change non-CONNECT proxy response from 405 to 403 to align with
how CONNECT denials are surfaced
- Add structured deny logging for non-CONNECT requests with hostname
extraction from absolute-form URIs
- Revise 7 user-facing error messages across proxy.rs and
dev-sandbox-policy.rego to follow consistent principle: generic
policy-deny messages for non-inference requests, descriptive messages
for recognized inference endpoints
- Update E2E test assertion to match new error message
- Update architecture docs to reflect new behavior
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
* fix(router): use protocol-aware auth headers for inference proxying
The router always sent `Authorization: Bearer` when proxying to backends,
which breaks Anthropic endpoints that require `x-api-key`. Thread the
source_protocol through to proxy_to_backend and set the appropriate auth
header based on protocol family.
* fix(server): remove noisy inference bundle log
The "serving inference bundle" info log fired every 30s per sandbox on
each route refresh poll. Remove it since there is no equivalent log for
policy serving.
* fix(sandbox): keep inference routing active when cluster bundle is initially empty
In cluster mode, if GetSandboxInferenceBundle returned zero routes at
sandbox startup, build_inference_context() returned None, disabling
inference routing for the sandbox lifetime and preventing
spawn_route_refresh() from starting. Routes created after sandbox
startup were never picked up.
Keep inference interception active with an empty cache in cluster mode
so background refresh can activate routes created later. File-mode
semantics are unchanged.
> **🏗️ build-from-issue-agent**
Closes#79
## Summary
Replaces the gateway-proxied inference model (`ProxyInference` RPC) with sandbox-local execution. The sandbox now resolves routes from either a standalone YAML route file or a cluster bundle fetched via the new `GetSandboxInferenceBundle` RPC, then forwards requests directly to inference backends using `navigator-router`.
### Benefits of removing the gRPC hop
Moving inference execution from the gateway to the sandbox eliminates the gRPC round-trip for every inference request:
- **No payload size limits**: Direct HTTP forwarding replaces protobuf-over-gRPC serialization.
- **Streaming-ready**: Preserves connection semantics for future SSE support (e.g., `stream: true`).
- **Lower latency**: Direct sandbox → backend path instead of sandbox → gateway → backend round-trip.
- **Reduced gateway load**: Gateway only serves the control-plane bundle delivery RPC.
- **Simpler error model**: HTTP status codes flow through directly without gRPC wrapping.
## Changes Made
- **Proto**: Removed `ProxyInference` RPC. Added `GetSandboxInferenceBundle` RPC for route bundle delivery.
- **Gateway**: Replaced `proxy_inference` with `get_sandbox_inference_bundle`. Removed `Router` and `navigator-router` dependency — gateway is now control-plane only for inference.
- **Router**: Switched config file format from TOML to YAML. Added `Debug` impl that redacts API keys.
- **Sandbox**: New `InferenceContext` with local `Router` + route cache. Routes loaded from `--inference-routes` file (standalone) or cluster bundle via gRPC, with 30s background refresh. New `--inference-routes` / `NAVIGATOR_INFERENCE_ROUTES` CLI arg.
- **Dev sandbox**: Added `inference-routes.yaml` at repo root with default NVIDIA NIM route. `mise run sandbox` now mounts it automatically and supports `-e` flag to forward env vars (e.g., `mise run sandbox`).
- **Example**: Added standalone example (`examples/inference/routes.yaml`) and rewrote README to cover both standalone and cluster workflows.
- **E2E tests**: Updated existing test names/docs. Added tests for Anthropic messages protocol and multi-route policy filtering.
## Tests Added
- **Unit (21 new):** `router_error_to_http` (6 variants), `load_from_file` YAML round-trip (5), `resolve_sandbox_inference_bundle` gRPC handler (6), `build_inference_context` route loading (4)
- **E2E (2 new):** Anthropic messages protocol routing, route filtering by `allowed_routes`
- **Total:** 188 unit tests passing across 3 crates
## Documentation Updated
- `architecture/gateway.md`: Removed router, updated Inference Service
- `architecture/sandbox.md`: Updated orchestration flow, InferenceContext, route loading
- `architecture/inference-routing.md`: Complete rewrite for sandbox-local architecture
## Verification
- [x] All unit tests passing (188 across 3 crates)
- [x] Pre-commit Rust checks passing (fmt, clippy, tests)
- [x] Architecture documentation updated
- [x] E2E tests updated with new test cases
- [x] Standalone example and documentation added
- [x] Default inference-routes.yaml with env passthrough for mise run sandbox
Closes#67
## Summary
Implements transparent inference interception and routing for sandboxed AI agents. The sandbox proxy intercepts outbound AI SDK calls (OpenAI, Anthropic) and reroutes them through the gateway to policy-controlled backends — enabling organizations to redirect inference traffic to local or self-hosted models without modifying agent code.
**Decision model** — a tri-state OPA evaluation for every CONNECT request:
1. Binary + endpoint explicitly allowed in `network_policies` → **allow** (pass through)
2. Not explicitly allowed + `inference.allowed_routes` configured → **inspect for inference** (TLS intercept, detect API patterns, route through gateway)
3. Otherwise → **deny**
No endpoint declarations or binary lists needed for inference routing. Just configure `inference.allowed_routes`.
## Key Changes
### Sandbox (interception)
- **OPA policy**: New `network_action` Rego rule with three outcomes (`allow`, `inspect_for_inference`, `deny`). New `NetworkAction` enum replaces `PolicyDecision.allowed` bool for the proxy's main decision path.
- **Proxy**: New `InspectForInference` path — TLS-terminates client, parses HTTP, detects inference API patterns (`POST /v1/chat/completions`, `/v1/completions`, `/v1/messages`), strips auth headers, forwards via gRPC.
- **New module**: `l7/inference.rs` — `InferenceApiPattern`, `detect_inference_pattern()`, HTTP request/response parsing.
- **gRPC client**: New `proxy_inference()` for sandbox→gateway forwarding.
- **Sandbox init**: Creates OPA engine when inference is configured, even without `network_policies`.
### Gateway (dispatch)
- **InferenceService**: `ProxyInference` RPC loads sandbox policy, resolves allowed routes, dispatches to router. Full CRUD for inference routes.
- **Proto**: `InferenceRoute`, `InferenceRouteSpec`, `ProxyInferenceRequest/Response`, Inference gRPC service.
### Router (backend proxying)
- **New crate**: `navigator-router` with `Router`, `proxy_with_candidates()`, protocol-based route selection, backend HTTP proxying with auth header rewriting.
- **Mock support** for testing (`mock://` scheme).
### CLI
- `nav inference create/update/delete/list` commands for route management.
### Python SDK
- Updated protobuf bindings. Removed old `inference.py` client (replaced by transparent interception).
### Documentation
- New `architecture/inference-routing.md` — end-to-end system documentation.
- Updated `architecture/sandbox.md` — proxy, OPA, and source index sections.
- Updated `architecture/README.md` — new subsystem overview and diagram.
## Addendum: Chunked Transfer Compatibility
This branch now also fixes intercepted SDK requests that send chunked request bodies:
- `inspect_for_inference` now accepts `Transfer-Encoding: chunked` and decodes chunked request bodies before forwarding to the gateway
- Removed the prior `411 Length Required` response for chunked intercepted requests
- Added request/response header sanitization for framing and hop-by-hop headers (`content-length`, `transfer-encoding`, `connection`, etc.) to keep forwarded requests and returned responses valid
- Added unit tests for chunked parsing and header sanitization
Note: this improves compatibility for streaming-style SDK request patterns; true token-by-token passthrough response streaming is still a separate follow-up.
## Minimal Policy for Inference Routing
```yaml
inference:
allowed_routes:
- local
```
Any outgoing connection from a binary not explicitly allowed in `network_policies` will be intercepted and checked for inference API patterns.
## Test Plan
- [x] `cargo test --workspace` — all tests pass
- [x] `mise run pre-commit` — all checks pass
- [x] E2E: OpenAI chat completions routed through gateway
- [x] E2E: Anthropic messages routed through gateway
- [x] E2E: Python OpenAI SDK from sandbox (`examples/inference/inference.py`)
- [x] E2E test: `e2e/python/test_inference_routing.py`
John T. Myers