John T. Myers
f4dc6be4b2
* refactor(inference): remove managed inference routes Closes #3172 Remove the inference route control plane, inference.local data path, built-in router crate, and SDK surface. Move inference workloads to explicitly imported provider profiles and native endpoints, with migration cleanup and updated tests and documentation. Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(policy): preserve alternate upstream isolation Restore the provider policy activation guard so legacy OpenAI and Anthropic providers configured for alternate base URLs do not grant egress to the built-in public vendor endpoints. Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Provider-backed Inference Example
This example calls the NVIDIA API Catalog through its native OpenAI-compatible endpoint. OpenShell supplies endpoint-bound credentials and network policy from an explicitly imported provider profile; the Python client owns the endpoint, model, request shape, timeout, and streaming behavior.
Files
| File | Description |
|---|---|
nvidia-inference.yaml |
Example profile for the endpoint, credential, and allowed Python binaries |
inference.py |
Native endpoint streaming and non-streaming client |
sandbox-policy.yaml |
Minimal policy that lets Python install the OpenAI client from PyPI |
Run the Example
Export the built-in profile as a starting point and compare it with the example before import. A custom profile must use a new ID; built-in IDs are reserved.
openshell provider profile export nvidia -o yaml > /tmp/nvidia-profile.yaml
diff -u /tmp/nvidia-profile.yaml examples/local-inference/nvidia-inference.yaml
openshell provider profile lint -f examples/local-inference/nvidia-inference.yaml
openshell provider profile import -f examples/local-inference/nvidia-inference.yaml
Create the provider from the local NVIDIA_API_KEY, then attach it to the new
sandbox:
openshell provider create \
--name nvidia-demo \
--type nvidia-inference \
--from-existing
openshell sandbox create \
--name inference-demo \
--provider nvidia-demo \
--policy examples/local-inference/sandbox-policy.yaml \
--upload examples/local-inference/inference.py \
-- python3 /sandbox/inference.py
The profile contributes the NVIDIA endpoint to the effective network policy and
injects an opaque NVIDIA_API_KEY placeholder. The proxy substitutes the real
key only for requests that match the profile endpoint. Inspect the composed
policy with:
openshell policy get inference-demo --full
To change the endpoint or allowed client binaries, export the custom profile,
edit it, and submit its resource_version with profile update. The workload
still needs a native client configuration that matches the profile.
openshell provider profile export nvidia-inference -o yaml > nvidia-inference.yaml
# Edit nvidia-inference.yaml.
openshell provider profile lint -f nvidia-inference.yaml
openshell provider profile update nvidia-inference -f nvidia-inference.yaml
Delete the sandbox when finished:
openshell sandbox delete inference-demo