mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-21 06:11:31 +08:00
John T. Myers
f4dc6be4b2
* refactor(inference): remove managed inference routes Closes #3172 Remove the inference route control plane, inference.local data path, built-in router crate, and SDK surface. Move inference workloads to explicitly imported provider profiles and native endpoints, with migration cleanup and updated tests and documentation. Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(policy): preserve alternate upstream isolation Restore the provider policy activation guard so legacy OpenAI and Anthropic providers configured for alternate base URLs do not grant egress to the built-in public vendor endpoints. Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
140 lines
3.6 KiB
Plaintext
140 lines
3.6 KiB
Plaintext
---
|
|
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
|
# SPDX-License-Identifier: Apache-2.0
|
|
title: "Use Local Inference with LM Studio"
|
|
sidebar-title: "Local Inference with LM Studio"
|
|
slug: "get-started/tutorials/local-inference-lmstudio"
|
|
description: "Attach a provider profile for a local LM Studio server and call its native compatible endpoints."
|
|
keywords: "Generative AI, Cybersecurity, Tutorial, LM Studio, Local Inference, Sandbox, Provider Profile"
|
|
---
|
|
|
|
This tutorial exposes an LM Studio server on the gateway host to selected
|
|
sandboxes. The provider attachment grants access; the workload calls LM
|
|
Studio's OpenAI- or Anthropic-compatible endpoint directly.
|
|
|
|
## Prerequisites
|
|
|
|
- Complete the [Quickstart](/get-started/quickstart).
|
|
- Install [LM Studio](https://lmstudio.ai/download) on the gateway host.
|
|
|
|
For a headless installation:
|
|
|
|
<Tabs>
|
|
<Tab title="Linux/macOS">
|
|
|
|
```shell
|
|
curl -fsSL https://lmstudio.ai/install.sh | bash
|
|
```
|
|
|
|
</Tab>
|
|
<Tab title="Windows">
|
|
|
|
```shell
|
|
irm https://lmstudio.ai/install.ps1 | iex
|
|
```
|
|
|
|
</Tab>
|
|
</Tabs>
|
|
|
|
Start the daemon and server:
|
|
|
|
```shell
|
|
lms daemon up
|
|
lms server start --bind 0.0.0.0
|
|
```
|
|
|
|
Load a model:
|
|
|
|
```shell
|
|
lms get qwen/qwen3.5-2b
|
|
lms load qwen/qwen3.5-2b
|
|
```
|
|
|
|
## Create the Provider Profile
|
|
|
|
Save this as `lmstudio.yaml`:
|
|
|
|
```yaml
|
|
id: lmstudio
|
|
display_name: LM Studio
|
|
description: Host-level LM Studio compatible APIs
|
|
category: inference
|
|
inference_capable: true
|
|
credentials: []
|
|
endpoints:
|
|
- host: host.openshell.internal
|
|
port: 1234
|
|
protocol: rest
|
|
access: read-write
|
|
enforcement: enforce
|
|
binaries:
|
|
- /usr/bin/curl
|
|
- /usr/local/bin/curl
|
|
- /usr/bin/python3
|
|
- /usr/local/bin/python
|
|
- /sandbox/.uv/python/**
|
|
- /sandbox/.venv/**
|
|
```
|
|
|
|
```shell
|
|
openshell provider profile lint -f lmstudio.yaml
|
|
openshell provider profile import -f lmstudio.yaml
|
|
openshell provider create --name lmstudio --type lmstudio
|
|
```
|
|
|
|
The profile intentionally has no credential. LM Studio does not require one by
|
|
default, but the provider attachment still supplies its policy boundary.
|
|
|
|
## Attach and Call the Native Endpoint
|
|
|
|
OpenAI-compatible request:
|
|
|
|
```shell
|
|
openshell sandbox create \
|
|
--name lmstudio-client \
|
|
--provider lmstudio \
|
|
-- \
|
|
curl http://host.openshell.internal:1234/v1/chat/completions \
|
|
--json '{
|
|
"model":"qwen/qwen3.5-2b",
|
|
"messages":[{"role":"user","content":"hello"}],
|
|
"max_tokens":10
|
|
}'
|
|
```
|
|
|
|
Anthropic-compatible request:
|
|
|
|
```shell
|
|
openshell sandbox exec lmstudio-client -- \
|
|
curl http://host.openshell.internal:1234/v1/messages \
|
|
--json '{
|
|
"model":"qwen/qwen3.5-2b",
|
|
"messages":[{"role":"user","content":"hello"}],
|
|
"max_tokens":10
|
|
}'
|
|
```
|
|
|
|
Configure SDKs with the same native base URL. Some SDKs require a non-empty
|
|
API key even when LM Studio ignores it; use a literal non-secret value such as
|
|
`unused`. Keep the actual model ID in the client request.
|
|
|
|
## Troubleshooting
|
|
|
|
- Enable **Serve on Local Network** in the LM Studio Developer tab, or use
|
|
`lms server start --bind 0.0.0.0`.
|
|
- Use `host.openshell.internal`, not `127.0.0.1` or `localhost`, in the
|
|
sandbox.
|
|
- Confirm the model is loaded with `lms ps`.
|
|
- Confirm the attachment with
|
|
`openshell sandbox provider list lmstudio-client`.
|
|
- Inspect endpoint and binary policy with
|
|
`openshell policy get lmstudio-client --full`.
|
|
- A remote gateway cannot reach an LM Studio process on your laptop without a
|
|
tunnel or shared network path.
|
|
|
|
## Next Steps
|
|
|
|
- [Provider-backed Inference](/sandboxes/inference-routing)
|
|
- [Profiles](/providers/profiles)
|
|
- [LM Studio CLI](https://lmstudio.ai/docs/cli)
|