OpenShell/examples/inference

Inference Routing Example

This example demonstrates Navigator's inference interception and routing. A sandbox process makes standard API calls (OpenAI, Anthropic, etc.) and Navigator transparently intercepts, enforces policy, and reroutes them to a configured backend — without any code changes in the sandboxed application.

How It Works

1. The sandbox proxy intercepts outgoing HTTPS connections.
2. OPA policy determines the action: if the binary has no explicit network policy but inference routing is configured, the connection is inspected.
3. The proxy TLS-terminates, parses the HTTP request, and detects known inference patterns (e.g., POST /v1/chat/completions).
4. Matching requests are forwarded through the gateway to the policy-allowed inference backend. Non-inference requests are denied.

Files

File	Description
inference.py	Python script that calls the OpenAI SDK — works unmodified inside a sandbox
sandbox-policy.yaml	Sandbox policy with inference routing enabled (route hint: local)

Quick Start

1. Start a Navigator cluster

mise run cluster
navigator cluster status

2. Create an inference route

Point the route at any OpenAI-compatible endpoint (local or remote):

# Local model (e.g., LM Studio, Ollama, vLLM)
navigator inference create \
  --routing-hint local \
  --base-url http://<HOST>:<PORT> \
  --model-id <MODEL_NAME>

# Remote provider (e.g., OpenAI, NVIDIA NIM)
navigator inference create \
  --routing-hint local \
  --base-url https://api.openai.com \
  --api-key sk-... \
  --model-id gpt-4o-mini

If --protocol is omitted, Navigator auto-detects supported protocols by probing the endpoint (sends minimal requests with max_tokens: 1).

Verify the route:

navigator inference list

3. Run the example inside a sandbox

navigator sandbox create \
  --policy examples/inference/sandbox-policy.yaml \
  --keep \
  --name inference-demo \
  -- python examples/inference/inference.py

The script targets https://api.openai.com by default, but Navigator intercepts the connection and routes it to whatever backend the local route points at.

Expected output:

model=<backend model name>
content=NAV_OK

4. (Optional) Interactive session

navigator sandbox connect inference-demo
# Inside the sandbox:
python examples/inference/inference.py

5. Cleanup

navigator sandbox delete inference-demo
navigator inference delete <route-name>

Customizing the Policy

Edit sandbox-policy.yaml to control which routes are available:

inference:
  allowed_routes:
    - local          # matches the --routing-hint used in step 2
    - production     # add more route hints as needed

The allowed_routes list determines which inference routes a sandbox can use. Routes are matched by their routing_hint field.

Supported Protocols

Navigator detects and routes the following inference API patterns:

Pattern	Protocol	Kind
POST /v1/chat/completions	openai_chat_completions	Chat completion
POST /v1/completions	openai_completions	Text completion
POST /v1/responses	openai_responses	Responses API
POST /v1/messages	anthropic_messages	Anthropic messages