OpenShell/docs/how-it-works/inference.mdx

---
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
title: "Inference"
sidebar-title: "Inference"
description: "Grant a sandbox access to model providers through provider profiles, attachments, and native endpoints."
keywords: "Generative AI, Cybersecurity, Inference, Provider Profiles, Credentials, LLM"
position: 8
---

OpenShell grants model-provider access through provider profiles and sandbox
attachments. The workload calls the provider's native API. OpenShell evaluates
the request against the profile-derived network policy and substitutes the real
credential only at an endpoint authorized by that profile.

This keeps the complete access contract in one place:

| Concern | Owner |
|---|---|
| Credential and refresh lifecycle | Provider instance |
| Authorized hosts, ports, paths, and binaries | Provider profile |
| Which workload receives access | Sandbox provider attachment |
| Base URL, model, request shape, and timeout | Native client or workload configuration |

## Define and Attach a Hosted Provider

Start from the [NVIDIA example profile](https://github.com/NVIDIA/OpenShell/blob/main/providers/nvidia.yaml),
review its access, and import an edited copy under a new ID. A new gateway has
no profiles to export. Download the example so you can add the Python paths
used by this workload:

```shell
curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/providers/nvidia.yaml -o nvidia-native.yaml
```

In `nvidia-native.yaml`, change `id` to `nvidia-native`, give the profile a
distinct `display_name`, and set `binaries` to the executables that may call the
API, as described in [Binary
Matching](/how-it-works/policies/network-rules#binary-matching). Keep the
credential and endpoint definitions you intend to grant. For a Python workload,
the edited fields can look like:

```yaml
id: nvidia-native
display_name: NVIDIA Native API
binaries:
  - /usr/bin/python3.*
  - /usr/local/bin/python3.*
  - /sandbox/.venv/**
```

Lint and import the complete edited profile, then create a provider from its
new ID:

```shell
openshell profile lint -f nvidia-native.yaml
openshell profile import -f nvidia-native.yaml

openshell provider create \
  --name nvidia-prod \
  --type nvidia-native \
  --from-existing

openshell sandbox create \
  --name inference-demo \
  --from registry.example.com/team/python-agent:1.0 \
  --provider nvidia-prod \
  -- python app.py
```

The imported provider profile supplies `NVIDIA_API_KEY` as an opaque placeholder and
allows the profile's native endpoint. Configure the client with the real model
identifier:

```python
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://integrate.api.nvidia.com/v1",
    api_key=os.environ["NVIDIA_API_KEY"],
    timeout=300,
)

response = client.responses.create(
    model="nvidia/nemotron-3-super-120b-a12b",
    input="Hello!",
)
print(response.output_text)
```

Use the same export, edit, and import flow for OpenAI, Anthropic, or another
hosted service. Set a new profile ID, retain only the endpoints and credentials
the workload needs, and name the actual client binaries. Provider attachment
does not select or rewrite a model.

The gateway resolves a provider against the profiles you imported, so a provider
whose profile is missing fails at sandbox creation with a message naming the
profile and the import command. Treat the example profiles in
[`providers/`](https://github.com/NVIDIA/OpenShell/tree/main/providers) as
starting templates you copy and adapt, not as defaults.

## Attach a Provider to a Running Sandbox

```shell
openshell sandbox provider attach inference-demo nvidia-prod --wait
openshell sandbox provider list inference-demo
```

Running sandboxes poll for provider and effective-policy changes. `--wait`
confirms the sandbox has applied the attachment. Launch a new process so it
receives the new credential placeholder:

```shell
openshell sandbox exec --name inference-demo -- python app.py
```

An already-running process does not gain new environment variables. Credential
rotation and detach take effect once the sandbox applies the provider change.
Use `--wait` to confirm that change before relying on it.

Detach the provider to revoke its policy and credential access:

```shell
openshell sandbox provider detach inference-demo nvidia-prod --wait
```

## Use a Custom or Self-hosted Endpoint

Do not reuse the `openai` or `anthropic` example profile for an alternate host.
Those profiles bind credentials to the public vendor endpoints, and a provider
that points `OPENAI_BASE_URL` elsewhere is treated as endpointless rather than
silently carrying the credential to the new host. Import a profile that names the
intended endpoint and the binaries that may call it.

For a credentialless Ollama server on the gateway host, save this profile as
`ollama-openai.yaml`:

```yaml
id: ollama-openai
display_name: Ollama
description: Host-local Ollama OpenAI-compatible API
category: inference
inference_capable: true
credentials: []
endpoints:
  - host: host.openshell.internal
    port: 11434
    protocol: rest
    access: read-write
    enforcement: enforce
binaries:
  - /usr/bin/curl
  - /usr/local/bin/curl
  - /usr/bin/python3
  - /usr/local/bin/python
  - /sandbox/.uv/python/**
  - /sandbox/.venv/**
```

Import the profile, create an instance, and attach it:

```shell
openshell profile lint -f ollama-openai.yaml
openshell profile import -f ollama-openai.yaml
openshell provider create --name ollama --type ollama-openai

openshell sandbox create \
  --name ollama-client \
  --from registry.example.com/team/python-agent:1.0 \
  --provider ollama \
  --env OPENAI_BASE_URL=http://host.openshell.internal:11434/v1 \
  -- python app.py
```

Use any non-empty placeholder value if the client library requires an API key
for a server that does not authenticate requests:

```python
import os
from openai import OpenAI

client = OpenAI(
    base_url=os.environ["OPENAI_BASE_URL"],
    api_key="unused",
)
response = client.chat.completions.create(
    model="qwen3.5:0.8b",
    messages=[{"role": "user", "content": "Hello"}],
)
```

For an authenticated alternate endpoint, declare a credential in the custom
profile, bind it to that endpoint, and create the provider from the original
credential source. OpenShell never exports stored credential values.

## Verify Access

Inspect the effective policy, including provider-derived entries:

```shell
openshell policy get inference-demo --full
```

Then run a native request from a new sandbox process. A successful request
confirms endpoint policy, binary attribution, credential substitution, DNS,
and upstream service behavior.

If the request is denied:

- Confirm the provider is attached with `openshell sandbox provider list`.
- Confirm the caller binary and native endpoint appear in
  `openshell policy get <sandbox> --full`.
- If the provider was attached after the process started, launch a new process.
- If the sandbox uses a gateway global policy override, add the native endpoint
  there because a global override suppresses provider-derived policy layers.
- Inspect logs for `credential_endpoint_mismatch`. That error means policy
  admitted the request but the provider profile did not authorize its
  credential at the requested endpoint.

## Migrate from Managed Inference Routes

OpenShell removed the workspace-global managed inference route and the
`openshell inference` commands. Upgrades remove stored route records. Provider
records, provider refresh configuration, and existing sandbox attachments are
preserved. Import the profile each preserved provider refers to before the first
sandbox creation after the upgrade, and give new setups an explicit profile under
their own ID.

The old route cannot be converted automatically. It applied one provider and
model to every sandbox in a workspace, while provider attachments intentionally
grant access to selected sandboxes. OpenShell cannot infer which sandboxes
should receive that authority.

### Now

Earlier releases configured one shared route:

```shell
openshell provider create --name nvidia-prod --type nvidia --from-existing
openshell inference set \
  --provider nvidia-prod \
  --model nvidia/nemotron-3-super-120b-a12b \
  --timeout 300
```

Clients called a managed virtual endpoint, supplied a placeholder key and
model, and relied on OpenShell to rewrite the request.

### After

Export the old provider type's profile, edit its ID and access contract, import
it, and create a replacement provider from the original credential source:

```shell
openshell profile export nvidia -o yaml > nvidia-native.yaml
# Edit id, display_name, endpoints, and binaries in nvidia-native.yaml.
openshell profile lint -f nvidia-native.yaml
openshell profile import -f nvidia-native.yaml
openshell provider create \
  --name nvidia-native-prod \
  --type nvidia-native \
  --from-existing
openshell sandbox provider attach inference-demo nvidia-native-prod --wait
```

OpenShell does not export stored credential values or change a provider's
profile type in place. If the original credential source is unavailable, import
the original profile at its own ID so the preserved provider keeps resolving
while you arrange credential rotation into the replacement provider.

Update each workload to:

1. Call the provider's native endpoint.
2. Read the credential variable declared by its profile.
3. Send the real provider model identifier.
4. Configure request timeouts in the client.
5. Use the provider's native request format.

Launch a new process after attachment and verify a native request before
upgrading production workloads. Code that still calls the retired managed
virtual endpoint fails DNS resolution because OpenShell no longer resolves or
trusts that host.

### Migration Checklist

Before upgrading, record the old provider, model, and timeout with the previous
release's `openshell inference get`. Identify actual consumers, export and edit
the source profile, import it under a new ID, and create the replacement
provider from the original credential source. Attach it only to those
sandboxes, migrate their clients, and test the native path while the old route
is still available. Delete the old route on the previous release to expose
missed consumers. After upgrading the gateway, delete and recreate every
pre-upgrade sandbox so no old supervisor, DNS entry, trust material, or cached
route survives. Reapply only the provider attachments each replacement
sandbox needs.

Special cases require additional work:

- A provider with an alternate `OPENAI_BASE_URL` or `ANTHROPIC_BASE_URL` needs
  an endpoint-bearing custom profile for that host.
- Host-local services need `host.openshell.internal` or a reachable LAN/service
  hostname, not `127.0.0.1` or `localhost`.
- Google Vertex AI clients must use the native Vertex endpoint and
  authentication behavior. See [Google](/how-it-works/providers/google#vertex-ai).
- A bridge-fronted AWS Bedrock deployment needs a custom profile that declares
  the bridge endpoint and allowed client binaries.

## Security Differences

Provider attachment preserves credential non-disclosure: workloads receive
opaque placeholders, and the proxy substitutes a real credential only after
network policy and endpoint binding both pass. Native requests are no longer
filtered or rewritten by a model-specific router. The provider profile's L7
rules therefore define the allowed API surface, and the workload controls
headers, model selection, request shape, streaming, and timeout behavior.

## Next Steps

- [Profiles](/how-it-works/providers/profiles)
- [Providers](/how-it-works/providers/overview)
- [Sandbox Policies](/how-it-works/policies/overview)
- [Google](/how-it-works/providers/google#vertex-ai)