mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-20 06:00:06 +08:00
John T. Myers
f4dc6be4b2
* refactor(inference): remove managed inference routes Closes #3172 Remove the inference route control plane, inference.local data path, built-in router crate, and SDK surface. Move inference workloads to explicitly imported provider profiles and native endpoints, with migration cleanup and updated tests and documentation. Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(policy): preserve alternate upstream isolation Restore the provider policy activation guard so legacy OpenAI and Anthropic providers configured for alternate base URLs do not grant egress to the built-in public vendor endpoints. Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
147 lines
3.9 KiB
Plaintext
147 lines
3.9 KiB
Plaintext
---
|
|
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
|
# SPDX-License-Identifier: Apache-2.0
|
|
title: "Run Local Inference with Ollama"
|
|
sidebar-title: "Inference with Ollama"
|
|
slug: "get-started/tutorials/inference-ollama"
|
|
description: "Run Ollama inside a community sandbox or attach a profile for a host-level Ollama server."
|
|
keywords: "Generative AI, Cybersecurity, Tutorial, Ollama, Local Inference, Sandbox, Provider Profile"
|
|
---
|
|
|
|
This tutorial covers two Ollama deployments:
|
|
|
|
1. The self-contained Ollama community sandbox. This is the recommended path.
|
|
2. A host-level Ollama service shared by explicitly authorized sandboxes.
|
|
|
|
## Prerequisites
|
|
|
|
Complete the [Quickstart](/get-started/quickstart) before proceeding.
|
|
|
|
## Option A: Ollama Community Sandbox
|
|
|
|
The community image bundles Ollama and supported coding agents:
|
|
|
|
```shell
|
|
openshell sandbox create --from ollama
|
|
```
|
|
|
|
Inside the sandbox, run a local or cloud model:
|
|
|
|
```shell
|
|
ollama run qwen3.5
|
|
ollama run kimi-k2.5:cloud
|
|
```
|
|
|
|
Or launch a coding agent against Ollama:
|
|
|
|
```shell
|
|
ollama launch claude
|
|
ollama launch codex
|
|
ollama launch opencode
|
|
```
|
|
|
|
For an automated workflow:
|
|
|
|
```shell
|
|
ollama launch claude --yes --model qwen3.5
|
|
```
|
|
|
|
Useful starter models include `qwen3.5:0.8b` for smoke tests and `qwen3.5`
|
|
for coding and tool use. Check the
|
|
[Ollama model library](https://ollama.com/library) for current model details.
|
|
|
|
Update the bundled Ollama installation with `update-ollama`, or request an
|
|
update at sandbox start:
|
|
|
|
```shell
|
|
openshell sandbox create --from ollama -e OLLAMA_UPDATE=1
|
|
```
|
|
|
|
## Option B: Host-level Ollama
|
|
|
|
Use this path when Ollama runs on the same machine as the OpenShell gateway and
|
|
selected sandboxes should share it.
|
|
|
|
### Start Ollama on a Reachable Address
|
|
|
|
```shell
|
|
curl -fsSL https://ollama.com/install.sh | sh
|
|
OLLAMA_HOST=0.0.0.0:11434 ollama serve
|
|
```
|
|
|
|
In another terminal, pull a model:
|
|
|
|
```shell
|
|
ollama pull qwen3.5:0.8b
|
|
```
|
|
|
|
### Import an Endpoint-bearing Profile
|
|
|
|
Save this as `ollama-openai.yaml`:
|
|
|
|
```yaml
|
|
id: ollama-openai
|
|
display_name: Ollama
|
|
description: Host-level Ollama OpenAI-compatible API
|
|
category: inference
|
|
inference_capable: true
|
|
credentials: []
|
|
endpoints:
|
|
- host: host.openshell.internal
|
|
port: 11434
|
|
protocol: rest
|
|
access: read-write
|
|
enforcement: enforce
|
|
binaries:
|
|
- /usr/bin/curl
|
|
- /usr/local/bin/curl
|
|
- /usr/bin/python3
|
|
- /usr/local/bin/python
|
|
- /sandbox/.uv/python/**
|
|
- /sandbox/.venv/**
|
|
```
|
|
|
|
```shell
|
|
openshell provider profile lint -f ollama-openai.yaml
|
|
openshell provider profile import -f ollama-openai.yaml
|
|
openshell provider create --name ollama --type ollama-openai
|
|
```
|
|
|
|
The provider has no secret because this Ollama server does not authenticate.
|
|
Its attachment still carries the endpoint and binary policy.
|
|
|
|
### Attach and Verify
|
|
|
|
```shell
|
|
openshell sandbox create \
|
|
--name ollama-client \
|
|
--provider ollama \
|
|
-- \
|
|
curl http://host.openshell.internal:11434/v1/chat/completions \
|
|
--json '{
|
|
"model":"qwen3.5:0.8b",
|
|
"messages":[{"role":"user","content":"hello"}],
|
|
"max_tokens":10
|
|
}'
|
|
```
|
|
|
|
For an OpenAI SDK client, set its base URL to
|
|
`http://host.openshell.internal:11434/v1`, use any non-empty API key value the
|
|
SDK accepts, and select the real Ollama model in the request.
|
|
|
|
## Troubleshooting
|
|
|
|
- Bind host-level Ollama to `0.0.0.0`, not `127.0.0.1`.
|
|
- Use `host.openshell.internal`, not `localhost`, from a sandbox.
|
|
- Confirm attachment with `openshell sandbox provider list ollama-client`.
|
|
- Inspect the effective policy with `openshell policy get ollama-client --full`.
|
|
- Run `ollama ps` and `ollama pull <model>` when the model is unavailable.
|
|
- If the gateway is remote, the hostname refers to the remote gateway host,
|
|
not your laptop. Use a shared service address or tunnel instead.
|
|
|
|
## Next Steps
|
|
|
|
- [Provider-backed Inference](/sandboxes/inference-routing)
|
|
- [Profiles](/providers/profiles)
|
|
- [Customize Sandbox Policies](/sandboxes/policies)
|