mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-17 05:31:16 +08:00
Miyoung Choi
b7c763204d
* docs: refresh user-facing docs for recent sandbox and inference changes - architecture: document system CA loading for upstream TLS, `tls: skip` as the opt-out, gateway state persistence across restarts, and OCSF structured logging surface. - inference: document per-provider header allowlist, Authorization stripping, 120s streaming idle tolerance, and extended-thinking timeout guidance. - manage-sandboxes: add "Execute a Command in a Sandbox" section for `openshell sandbox exec` with flag reference. - security best practices: expand seccomp denylist (unconditional and conditional blocks), document two-phase Landlock probe, High-severity `landlock-unavailable` finding, and inference keep-alive closure. - observability logging: document port in HTTP log URLs, `[reason:...]` denial suffixes, proxy 403/502 JSON error bodies, and Landlock CONFIG:ENABLED/CONFIG:OTHER events. Signed-off-by: Miyoung Choi <miyoungc@nvidia.com> * docs(architecture): tighten high-level architecture page Trim implementation detail (CA bundle paths, deprecated TLS keys, SSH handshake secret) from the high-level architecture page, fix accuracy issues surfaced during deep audit, and expand uncommon acronyms on first mention. - Drop unsupported "cost-based routing" claim from Privacy Router row. - Replace "brokers requests across the platform" with auth-boundary description. - Add "inference" to Policy Engine constraint list per AGENTS.md. - Expand Deny rule to include SSRF, blocked control-plane port, and L7 deny paths in addition to deny-by-default. - Switch Allow/Deny labels from hyphen to colon; remove em dashes and a double space. - Expand LLM, SSRF, L7, TLS, CA, PEM, SSH, OCSF, and JSONL on first use. Signed-off-by: Miyoung Choi <miyoungc@nvidia.com> Made-with: Cursor * docs: address audit feedback on refresh PR - observability/logging: rewrite the allowed_ips paragraph after the Denial Reasons table; the previous wording said authors "can use" invalid entries while also stating they were rejected, which was contradictory and conflated load-time validation with the runtime per-CONNECT denial phrases the section documents. - about/architecture: split compound sentences in the new Gateway Lifecycle and Observability sections so each clause stands alone. - inference/about: drop the streaming-tolerance sentence from the prose paragraph since the dedicated Streaming reliability table row already covers it. Signed-off-by: Miyoung Choi <miyoungc@nvidia.com> Made-with: Cursor * docs(architecture): address review feedback on architecture page - Drop the Privacy Router row from the Components table; it is not yet a separately exposed customer-facing component. - Update the page description and intro count to match the remaining three components (gateway, sandbox, policy engine). - Split the policy decision into the three modes that the engine actually implements: Explicit Deny (deny rules and hardening rules, takes precedence), Allow, and Implicit Deny (no rule matched). Signed-off-by: Miyoung Choi <miyoungc@nvidia.com> Made-with: Cursor * docs(architecture): convert policy decisions to a table Promote the three policy decisions (Explicit Deny, Allow, Implicit Deny) to a top-level table with Decision, When it applies, and Outcome columns instead of a nested bulleted list under list item 5. Top-level tables render reliably across markdown renderers, where nested-in-list tables do not. Signed-off-by: Miyoung Choi <miyoungc@nvidia.com> Made-with: Cursor * docs: repharse a bit --------- Signed-off-by: Miyoung Choi <miyoungc@nvidia.com>
69 lines
3.6 KiB
Plaintext
69 lines
3.6 KiB
Plaintext
---
|
|
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
|
# SPDX-License-Identifier: Apache-2.0
|
|
title: "About Inference Routing"
|
|
sidebar-title: "About Inference Routing"
|
|
description: "Understand how OpenShell routes inference traffic through external endpoints and the local privacy router."
|
|
keywords: "Generative AI, Cybersecurity, Inference Routing, Privacy, AI Agents, LLM"
|
|
position: 1
|
|
---
|
|
|
|
NVIDIA OpenShell handles inference traffic through two endpoints: `inference.local` and external endpoints.
|
|
The following table summarizes how OpenShell handles inference traffic.
|
|
|
|
| Path | How It Works |
|
|
|---|---|
|
|
| External endpoints | Traffic to hosts like `api.openai.com` or `api.anthropic.com` is treated like any other outbound request, allowed or denied by `network_policies`. Refer to [Policies](/sandboxes/policies). |
|
|
| `inference.local` | A special endpoint exposed inside every sandbox for routing inference traffic locally, preserving privacy and security. The [privacy router](/about/architecture) strips the original credentials, forwards only approved inference headers, injects the configured backend credentials, and forwards to the managed model endpoint. |
|
|
|
|
## How `inference.local` Works
|
|
|
|
When code inside a sandbox calls `https://inference.local`, the privacy router routes the request to the configured backend for that gateway. The configured model is applied to generation requests, provider credentials are supplied by OpenShell rather than by code inside the sandbox, and only approved inference headers are forwarded upstream.
|
|
|
|
If code calls an external inference host directly, that traffic is evaluated only by `network_policies`.
|
|
|
|
| Property | Detail |
|
|
|---|---|
|
|
| Credentials | No sandbox API keys needed. Credentials come from the configured provider record. The router strips caller-supplied `Authorization` before forwarding the request. |
|
|
| Header forwarding | `inference.local` forwards only a per-provider header allowlist. OpenAI routes allow `openai-organization` and `x-model-id`. Anthropic routes allow `anthropic-version` and `anthropic-beta`. NVIDIA routes allow `x-model-id`. All other caller headers are stripped. |
|
|
| Configuration | One provider and one model define sandbox inference for the active gateway. Every sandbox on that gateway sees the same `inference.local` backend. |
|
|
| Provider support | NVIDIA, any OpenAI-compatible provider, and Anthropic all work through the same endpoint. |
|
|
| Streaming reliability | The router tolerates idle gaps of up to 120 seconds between streamed chunks so long reasoning responses are not cut off mid-stream. |
|
|
| Hot-refresh | OpenShell picks up provider credential changes and inference updates without recreating sandboxes. Changes propagate within about 5 seconds by default. |
|
|
|
|
## Supported API Patterns
|
|
|
|
Supported request patterns depend on the provider configured for `inference.local`.
|
|
|
|
<Tabs>
|
|
<Tab title="OpenAI-compatible">
|
|
|
|
| Pattern | Method | Path |
|
|
|---|---|---|
|
|
| Chat Completions | `POST` | `/v1/chat/completions` |
|
|
| Completions | `POST` | `/v1/completions` |
|
|
| Responses | `POST` | `/v1/responses` |
|
|
| Model Discovery | `GET` | `/v1/models` |
|
|
| Model Discovery | `GET` | `/v1/models/*` |
|
|
|
|
</Tab>
|
|
|
|
<Tab title="Anthropic-compatible">
|
|
|
|
| Pattern | Method | Path |
|
|
|---|---|---|
|
|
| Messages | `POST` | `/v1/messages` |
|
|
|
|
</Tab>
|
|
|
|
</Tabs>
|
|
|
|
Requests to `inference.local` that do not match the configured provider's supported patterns are denied.
|
|
|
|
## Next Steps
|
|
|
|
Continue with one of the following:
|
|
|
|
- To set up the backend behind `inference.local`, refer to [Configure](/inference/configure).
|
|
- To control external endpoints, refer to [Policies](/sandboxes/policies).
|