trending/OpenShell
1
0
Fork 0
mirror of https://github.com/NVIDIA/OpenShell.git synced 2026-10-30 07:33:59 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
gateway-interceptor-impl
OpenShell/docs/sandboxes/inference-routing.mdx
T
Shiju 25abc9e3c7 feat(inference): allow local embeddings route (#1774) 
* feat(inference): allow local embeddings route

Route OpenAI-compatible embeddings through the local inference proxy so
sandboxed vector workloads reach a configured provider via the same
route classification and auth path that chat, completion, and model
discovery already use.

- Add openai_embeddings to the OpenAI-compatible protocol set so
  providers (openai, nvidia) advertise embeddings routing.
- Classify POST /v1/embeddings as the openai_embeddings protocol in the
  sandbox L7 patterns.
- Serve embeddings buffered with an accurate Content-Length, since the
  response is a single JSON object rather than an SSE token stream. The
  streaming path appends an SSE error frame on a size-cap or idle-timeout
  truncation, which would corrupt a one-object body the client parses
  whole. protocol_returns_buffered_body() selects the path.
- Probe an embeddings-only backend against /v1/embeddings during
  validation, after the chat and completion protocols so a multi-protocol
  route still prefers those.
- Extract two shared helpers. http_status_text() backs both response
  formatters and adds 401/422/429/503 for embeddings passthrough and
  router error mapping; write_inference_router_error() backs the streaming
  and buffered routing paths.
- Return an OpenAI-shaped embeddings body from the mock route.

Tests cover profile lookup, L7 pattern detection, the mock body, and
buffered Content-Length framing with no chunked transfer-encoding and no
SSE error frame.

Signed-off-by: Shiju <shiju@nvidia.com>

* fix(inference): cap buffered inference response body

The buffered proxy path read the whole upstream response into memory with
no size bound. The route timeout bounds elapsed time but not memory, so a
misbehaving or oversized upstream could force unbounded allocation in the
sandbox proxy. The streaming path already caps each response at 32 MiB;
the buffered path did not.

Cap the buffered read at the same 32 MiB. An advertised over-cap body is
rejected from its Content-Length before any bytes are read, and chunks
accumulate under the same bound so a chunked or mislabeled body cannot
slip past. An over-cap response fails as an upstream protocol error,
surfaced as HTTP 502 at the proxy boundary, and is never partially
returned.

Tests

- cargo test -p openshell-router \
    proxy_to_backend_rejects_over_cap_response_body

Signed-off-by: Shiju <shiju@nvidia.com>

* fix(inference): validate embeddings models against all advertised protocols

A managed route resolves to its provider profile's full protocol set, so
an embeddings model such as text-embedding-3-small lists
openai_chat_completions alongside openai_embeddings. Route verification
probed only the first writable protocol and stopped on its failure. It
sent a chat probe with the embedding model, the provider rejected it as
wrong-shape, and the route failed validation before the embeddings probe
ran. Embeddings-only configs could not be verified.

Try the advertised protocols in preference order. A request-shape
rejection (HTTP 400, 404, 405, 422) falls through to the next protocol,
so an embeddings model validates against /v1/embeddings even when the
chat probe rejects it. Credential, rate-limit, connectivity, and
upstream-health failures stay terminal and stop validation at the first
probe, so a bad key or a down backend is reported as itself rather than
masked by a later probe.

validation_probe becomes validation_probes, which returns the ordered
list, and the per-probe fallback retry (max_completion_tokens versus
max_tokens) moves into a shared helper.

Tests

- cargo test -p openshell-router \
    verify_embeddings_model_falls_through_chat_probe
- cargo test -p openshell-router verify_stops_on_credentials_failure

Signed-off-by: Shiju <shiju@nvidia.com>

* fix(inference): serve model discovery responses buffered

GET /v1/models returns a single JSON model list, the same response shape
as embeddings. The sandbox inference proxy was routing it through the SSE
streaming path. A streaming size-cap or idle-timeout truncation appends
an SSE error frame to the body, which corrupts a payload the client
parses as one JSON object.

Make response framing a property of the protocol. A new ResponseFraming
field on InferenceApiPattern is set once per pattern in default_patterns.
model_discovery and openai_embeddings are now Buffered, while chat
completions, completions, responses, and Anthropic messages stay
Streaming. The proxy dispatch gates on pattern.is_buffered(), which
replaces the stringly-typed protocol_returns_buffered_body predicate so
the streaming-versus-buffered decision lives in one place and cannot
drift across the sites that read it.

Model discovery now flows through the same buffered path as embeddings,
framed with an accurate Content-Length and bounded by the buffered-read
size cap that path already enforces.

Tests

- cargo test -p openshell-sandbox protocol_framing_classification
- cargo test -p openshell-sandbox \
    inference_model_discovery_served_buffered_with_content_length

Signed-off-by: Shiju <shiju@nvidia.com>

* docs(inference): document embeddings route in supported patterns

Signed-off-by: Shiju <shiju@nvidia.com>

---------

Signed-off-by: Shiju <shiju@nvidia.com>
2026-06-07 10:27:34 -07:00

295 lines
12 KiB
Plaintext
Raw Permalink Blame History

---
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
title: "Inference Routing"
sidebar-title: "Inference Routing"
description: "Understand and configure OpenShell inference routing through inference.local and external endpoints."
keywords: "Generative AI, Cybersecurity, Inference Routing, Configuration, Privacy, LLM, Provider"
position: 7
---
OpenShell handles inference traffic through two paths: external endpoints and `inference.local`.
| Path | How it works |
|---|---|
| External endpoints | Traffic to hosts like `api.openai.com` or `api.anthropic.com` is treated like any other outbound request, allowed or denied by `network_policies`. Refer to [Policies](/sandboxes/policies). |
| `inference.local` | A sandbox-local HTTPS endpoint that routes model requests through the gateway. The privacy router strips sandbox-supplied credentials, forwards only approved inference headers, injects the configured backend credentials, and forwards to the managed model endpoint. |
## How `inference.local` Works
When code inside a sandbox calls `https://inference.local`, the privacy router routes the request to the configured backend for that gateway. The configured model is applied to generation requests, provider credentials come from OpenShell rather than from code inside the sandbox, and only approved inference headers are forwarded upstream.
If code calls an external inference host directly, OpenShell evaluates that traffic only through `network_policies`.
| Property | Detail |
|---|---|
| Credentials | No sandbox API keys needed. Credentials come from the configured provider record. The router strips caller-supplied `Authorization` before forwarding the request. |
| Header forwarding | `inference.local` forwards only a per-provider header allowlist. OpenAI routes allow `openai-organization` and `x-model-id`. Anthropic routes allow `anthropic-version` and `anthropic-beta`. Vertex Claude rawPredict routes strip `anthropic-beta` and do not forward `anthropic-version` as a header because the router injects `anthropic_version` into the Vertex request body. NVIDIA routes allow `x-model-id`. All other caller headers are stripped. |
| Configuration | One provider and one model define sandbox inference for the active gateway. Every sandbox on that gateway sees the same `inference.local` backend. |
| Provider support | NVIDIA, Anthropic, Google Vertex AI, and any OpenAI-compatible provider all work through the same endpoint. Vertex routes Claude models through `/v1/messages` and non-Anthropic models through `/v1/chat/completions`. The gateway resolves the upstream Vertex host from the provider config, including regional, global, and supported multi-region endpoints. |
| Streaming reliability | The router tolerates idle gaps of up to 120 seconds between streamed chunks so long reasoning responses are not cut off mid-stream. |
| Hot refresh | OpenShell picks up provider credential changes and inference updates without recreating sandboxes. Changes propagate within about 5 seconds by default. |
## Supported API Patterns
Supported request patterns depend on the provider configured for `inference.local`.
<Tabs>
<Tab title="OpenAI-compatible">
| Pattern | Method | Path |
|---|---|---|
| Chat Completions | `POST` | `/v1/chat/completions` |
| Completions | `POST` | `/v1/completions` |
| Responses | `POST` | `/v1/responses` |
| Embeddings | `POST` | `/v1/embeddings` |
| Model Discovery | `GET` | `/v1/models` |
| Model Discovery | `GET` | `/v1/models/*` |
</Tab>
<Tab title="Anthropic-compatible">
| Pattern | Method | Path |
|---|---|---|
| Messages | `POST` | `/v1/messages` |
</Tab>
</Tabs>
Requests to `inference.local` that do not match the configured provider's supported patterns are denied.
Google Vertex AI does not expose every OpenAI-compatible path through `inference.local`. Vertex routes for Gemini and other non-Anthropic models currently support Chat Completions. Vertex routes for Claude models use the Anthropic Messages pattern. Base URL overrides are only supported for non-Anthropic Vertex routes.
## Configure Inference Routing
The managed local inference endpoint uses three values:
| Value | Description |
|---|---|
| Provider record | The credential backend OpenShell uses to authenticate with the upstream model host. |
| Model ID | The model to use for generation requests. |
| Timeout | Per-request timeout in seconds for upstream inference calls. Defaults to 60 seconds. |
For tested providers and base URLs, refer to [Supported Inference Providers](/sandboxes/manage-providers#supported-inference-providers).
## Create a Provider
Create a provider that holds the backend credentials you want OpenShell to use.
<Tabs>
<Tab title="NVIDIA API Catalog">
```shell
openshell provider create --name nvidia-prod --type nvidia --from-existing
```
This reads `NVIDIA_API_KEY` from your environment.
</Tab>
<Tab title="OpenAI-compatible Provider">
Any cloud provider that exposes an OpenAI-compatible API works with the `openai` provider type. You need three values from the provider: the base URL, an API key, and a model name.
```shell
openshell provider create \
--name my-cloud-provider \
--type openai \
--credential OPENAI_API_KEY=<your_api_key> \
--config OPENAI_BASE_URL=https://api.example.com/v1
```
Replace the base URL and API key with the values from your provider. For supported providers out of the box, refer to [Supported Inference Providers](/sandboxes/manage-providers#supported-inference-providers). For other providers, refer to your provider's documentation for the correct base URL, available models, and API key setup.
</Tab>
<Tab title="Google Vertex AI">
```shell
openshell provider create \
--name vertex-local \
--type google-vertex-ai \
--from-gcloud-adc \
--config VERTEX_AI_PROJECT_ID=my-gcp-project \
--config VERTEX_AI_REGION=us-central1
```
Use [Google Vertex AI](/providers/google-vertex-ai) for the full auth flows, including the production service-account refresh path, ADC-backed providers that mint `GOOGLE_VERTEX_AI_TOKEN`, and `--from-existing` support.
</Tab>
<Tab title="Local Endpoint">
```shell
openshell provider create \
--name my-local-model \
--type openai \
--credential OPENAI_API_KEY=empty-if-not-required \
--config OPENAI_BASE_URL=http://host.openshell.internal:11434/v1
```
Use `--config OPENAI_BASE_URL` to point to any OpenAI-compatible server running where the gateway runs. For host-backed local inference, use `host.openshell.internal` or the host's LAN IP. Avoid `127.0.0.1` and `localhost`. Set `OPENAI_API_KEY` to a dummy value if the server does not require authentication.
<Tip>
For a self-contained setup, the Ollama sandbox bundles Ollama inside the sandbox itself, so no host-level provider is needed. Refer to [Inference Ollama](/get-started/tutorials/inference-ollama) for details.
</Tip>
Ollama also supports cloud-hosted models using the `:cloud` tag suffix, for example `qwen3.5:cloud`.
</Tab>
<Tab title="Anthropic">
```shell
openshell provider create --name anthropic-prod --type anthropic --from-existing
```
This reads `ANTHROPIC_API_KEY` from your environment.
</Tab>
</Tabs>
## Set Inference Routing
Point `inference.local` at that provider and choose the model to use:
```shell
openshell inference set \
--provider nvidia-prod \
--model nvidia/nemotron-3-nano-30b-a3b
```
To override the default 60-second per-request timeout, add `--timeout`:
```shell
openshell inference set \
--provider nvidia-prod \
--model nvidia/nemotron-3-nano-30b-a3b \
--timeout 300
```
The value is in seconds. When `--timeout` is omitted or set to `0`, the default of 60 seconds applies. Increase `--timeout` when you expect extended thinking phases so the full response completes before the request deadline.
## Inspect and Update the Config
Confirm that the provider and model are set correctly:
```shell
openshell inference get
Gateway inference:
Provider: nvidia-prod
Model: nvidia/nemotron-3-nano-30b-a3b
Timeout: 300s
Version: 1
```
Use `update` when you want to change only one field:
```shell
openshell inference update --model nvidia/nemotron-3-nano-30b-a3b
openshell inference update --provider openai-prod
openshell inference update --timeout 120
```
## Use the Local Endpoint from a Sandbox
After inference is configured, code inside any sandbox can call `https://inference.local` directly. The client-supplied `model` and `api_key` values are not sent upstream — the privacy router injects the real credentials from the configured provider and rewrites the model before forwarding. Some SDKs require a non-empty API key even though `inference.local` does not use the sandbox-provided value; pass any placeholder such as `unused`.
<Tabs>
<Tab title="Claude Code">
```shell
ANTHROPIC_BASE_URL="https://inference.local" ANTHROPIC_API_KEY=unused claude --bare
```
`--bare` skips the OAuth login flow and uses `ANTHROPIC_API_KEY` directly. The key is stripped by the proxy and never reaches the upstream provider.
<Note>
Claude Code appends `/v1/messages` to `ANTHROPIC_BASE_URL`, so omit the `/v1` suffix from the base URL.
</Note>
</Tab>
<Tab title="OpenCode">
```shell
ANTHROPIC_BASE_URL="https://inference.local/v1" ANTHROPIC_API_KEY=unused opencode
```
<Note>
OpenCode appends `/messages` directly to `ANTHROPIC_BASE_URL`. Include the `/v1` suffix so the full path becomes `/v1/messages`, which matches the inference pattern.
</Note>
</Tab>
<Tab title="Python (OpenAI SDK)">
```python
from openai import OpenAI
client = OpenAI(base_url="https://inference.local/v1", api_key="unused")
response = client.chat.completions.create(
model="anything",
messages=[{"role": "user", "content": "Hello"}],
)
```
</Tab>
<Tab title="Python (Anthropic SDK)">
```python
import anthropic
client = anthropic.Anthropic(
base_url="https://inference.local",
api_key="unused",
)
message = client.messages.create(
model="anything",
max_tokens=1024,
messages=[{"role": "user", "content": "Hello"}],
)
```
</Tab>
</Tabs>
Use `inference.local` when inference should stay private and credentials should not be exposed inside the sandbox. External providers reached directly belong in `network_policies` instead.
When the upstream runs on the same machine as the gateway, bind it to `0.0.0.0` and point the provider at `host.openshell.internal` or the host's LAN IP. `127.0.0.1` and `localhost` usually fail because the request originates from the gateway or sandbox runtime, not from your shell.
If the gateway runs on a remote host or behind a cloud deployment, `host.openshell.internal` points to that remote machine, not to your laptop. A locally running Ollama or vLLM process is not reachable from a remote gateway unless you add your own tunnel or shared network path.
## Verify from a Sandbox
`openshell inference set` and `openshell inference update` verify the resolved upstream endpoint by default before saving the configuration. If the endpoint is not live yet, retry with `--no-verify` to persist the route without the probe.
To confirm end-to-end connectivity from a sandbox, run:
```shell
curl https://inference.local/v1/responses \
-H "Content-Type: application/json" \
-d '{
"instructions": "You are a helpful assistant.",
"input": "Hello!"
}'
```
A successful response confirms the privacy router can reach the configured backend and the model is serving requests.
- Gateway-scoped: Every sandbox using the active gateway sees the same `inference.local` backend.
- HTTPS only: `inference.local` is intercepted only for HTTPS traffic.
- Hot reload: Provider, model, and timeout changes are picked up by running sandboxes within about 5 seconds by default. No sandbox recreation is required.
## Next Steps
Explore related topics:
- To follow a complete Ollama-based local setup, refer to [Inference Ollama](/get-started/tutorials/inference-ollama).
- To follow a complete LM Studio-based local setup, refer to [Local Inference LM Studio](/get-started/tutorials/local-inference-lmstudio).
- To control external endpoints, refer to [Policies](/sandboxes/policies).
- To manage provider records, refer to [Providers](/sandboxes/manage-providers).
Reference in New Issue View Git Blame Copy Permalink