mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-22 06:19:47 +08:00
Drew Newberry
a649aa42fc
* docs: add 0.1.0 upgrade guide outline Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: add upgrade change provenance Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: update 0.1.0 upgrade guidance Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: move upgrade navigation below security Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: remove release notes page Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: announce OpenShell 0.1.0 Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: reorganize guides and require user-owned images Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: reorganize navigation around core concepts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: simplify navigation and tutorial catalog Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: refine 0.1.0 release highlights Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: align navigation and 0.1.0 guidance Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com>
70 lines
2.4 KiB
Markdown
70 lines
2.4 KiB
Markdown
# Inference Example
|
|
|
|
This example calls the NVIDIA API Catalog through its native OpenAI-compatible
|
|
endpoint. OpenShell supplies endpoint-bound credentials and network policy from
|
|
an explicitly imported provider profile; the Python client owns the endpoint,
|
|
model, request shape, timeout, and streaming behavior.
|
|
|
|
## Files
|
|
|
|
| File | Description |
|
|
|---|---|
|
|
| `nvidia-inference.yaml` | Example profile for the endpoint, credential, and allowed Python binaries |
|
|
| `inference.py` | Native endpoint streaming and non-streaming client |
|
|
| `sandbox-policy.yaml` | Minimal policy that lets Python install the OpenAI client from PyPI |
|
|
|
|
## Run the Example
|
|
|
|
Export the built-in profile as a starting point and compare it with the example
|
|
before import. A custom profile must use a new ID; built-in IDs are reserved.
|
|
|
|
```shell
|
|
openshell provider profile export nvidia -o yaml > /tmp/nvidia-profile.yaml
|
|
diff -u /tmp/nvidia-profile.yaml examples/local-inference/nvidia-inference.yaml
|
|
openshell provider profile lint -f examples/local-inference/nvidia-inference.yaml
|
|
openshell provider profile import -f examples/local-inference/nvidia-inference.yaml
|
|
```
|
|
|
|
Create the provider from the local `NVIDIA_API_KEY`, then attach it to the new
|
|
sandbox:
|
|
|
|
```shell
|
|
openshell provider create \
|
|
--name nvidia-demo \
|
|
--type nvidia-inference \
|
|
--from-existing
|
|
|
|
openshell sandbox create \
|
|
--name inference-demo \
|
|
--provider nvidia-demo \
|
|
--policy examples/local-inference/sandbox-policy.yaml \
|
|
--upload examples/local-inference/inference.py \
|
|
-- python3 /sandbox/inference.py
|
|
```
|
|
|
|
The profile contributes the NVIDIA endpoint to the effective network policy and
|
|
injects an opaque `NVIDIA_API_KEY` placeholder. The proxy substitutes the real
|
|
key only for requests that match the profile endpoint. Inspect the composed
|
|
policy with:
|
|
|
|
```shell
|
|
openshell policy get inference-demo --full
|
|
```
|
|
|
|
To change the endpoint or allowed client binaries, export the custom profile,
|
|
edit it, and submit its `resource_version` with `profile update`. The workload
|
|
still needs a native client configuration that matches the profile.
|
|
|
|
```shell
|
|
openshell provider profile export nvidia-inference -o yaml > nvidia-inference.yaml
|
|
# Edit nvidia-inference.yaml.
|
|
openshell provider profile lint -f nvidia-inference.yaml
|
|
openshell provider profile update nvidia-inference -f nvidia-inference.yaml
|
|
```
|
|
|
|
Delete the sandbox when finished:
|
|
|
|
```shell
|
|
openshell sandbox delete inference-demo
|
|
```
|