trending/substrate
1
0
Fork 0
mirror of https://github.com/agent-substrate/substrate.git synced 2026-10-02 03:24:42 +08:00
Code Issues Packages Projects Releases Wiki Activity

docs: add guide for running the microVM runtime locally (KVM / Lima) (#743)

Browse Source 
### Why

The README Quickstart covers getting a local Substrate cluster running
with
the default gVisor runtime, but there's no documentation for the
micro-VM
runtime, where the friction can't be scripted away: it requires
`/dev/kvm`
(bare metal, nested virtualization, or Lima on macOS), and on Apple
Silicon
the Lima configuration and asset assembly are non-obvious. New
contributors
  have to reverse-engineer the `hack/` scripts.

  ### What
  
Adds `docs/dev/microvm-local.md` — "Running the microVM runtime locally"
— a
focused guide covering only the microVM delta, based on notes from real
onboarding runs. General setup is deferred to the README Quickstart
(listed
  as the guide's prerequisite) rather than duplicated:

- **Option A: Linux host with KVM** — verifying `/dev/kvm` and CPU virt
support, the rootless-Docker caveat for the KVM probe, cluster creation,
    and the one-shot `run-microvm-demo-kind.sh` bring-up.
- **Option B: Apple Silicon macOS via Lima** — nested virtualization
with the
guest image pinned to Ubuntu 25.10 (until the kernel issue in the
default
image is fixed), `vzNAT` networking, and assembling the arm64 assets
inside
the Lima guest (`assemble.sh` requires a Linux host of the target arch).
- **Trying it out** — defers to the next steps the demo script prints
and
links the counter demo's micro-VM variant, instead of duplicating those
    commands.
- **Troubleshooting** — symptom → root cause → fix entries actually hit
    during onboarding (rootless-Docker KVM probe failures, the `aws` CLI
staging requirement until #804 lands, arm64 `virtiofsd` build deps, M1
    lacking FEAT_NV2).
This commit is contained in:
Lucky Abolorunke
2026-08-12 08:39:41 -04:00
committed by GitHub
parent 4c2e79fb97
commit bce9366b3d
2 changed files with 191 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CONTRIBUTING.md
+8
View File
@@ -25,6 +25,14 @@ sign a new one.
This project follows [Google's Open Source Community
Guidelines](https://opensource.google/conduct/).
### Set up a local development environment
The [Quickstart (Development)](README.md#quickstart-development) in the README
covers bringing up a local cluster with the default (gVisor) runtime. To run
the microVM runtime locally — which needs `/dev/kvm`, or Lima nested
virtualization on Apple Silicon — see
[docs/dev/microvm-local.md](docs/dev/microvm-local.md).
## Contribution process
This is a very new project, so we are still working out exactly how it is going
docs/dev/microvm-local.md
+183
View File
@@ -0,0 +1,183 @@
# Running the microVM runtime locally
The microVM sandbox class (`ateom-microvm`: a Kata guest on Cloud Hypervisor)
needs `/dev/kvm`, which takes some extra setup compared to the default gVisor
path. This guide covers just that delta: getting a KVM-capable Docker
environment — on Linux, or on Apple Silicon macOS via
[Lima](https://lima-vm.io/) — then running the microVM counter demo and
verifying a guest-memory snapshot round-trip.
## Prerequisites
Complete the
[Quickstart (Development)](../../README.md#quickstart-development) in the
README first — it covers the base tooling and the default (gVisor) path this
guide builds on. For background on the runtime, see
[architecture.md](../architecture.md) and
[hack/microvm-assets/README.md](../../hack/microvm-assets/README.md).
## Option A: Linux host with KVM
Works on bare-metal Linux or any cloud VM with nested virtualization enabled
(e.g. GCE N2/N2D instances with nested virt, or equivalent on other clouds).
### 1. Verify KVM
```sh
ls -la /dev/kvm
# Expected: crw-rw---- 1 root kvm 10, 232 ... /dev/kvm
grep -cE '(vmx|svm)' /proc/cpuinfo # >0 means CPU virt support (x86)
```
`hack/create-kind-cluster.sh` probes for KVM by running a root container with
`--device /dev/kvm`, which works out of the box with a standard (rootful)
Docker install. With **rootless Docker** the container's root is remapped to
your user, so the probe fails with `permission denied` — use rootful Docker
instead, or open up the device with `sudo chmod 666 /dev/kvm`.
### 2. Create the cluster
```sh
./hack/create-kind-cluster.sh
# Look for: "/dev/kvm found: micro-VM (kata + cloud-hypervisor) support will be enabled."
kubectl get nodes --show-labels | grep 'ate.dev/sandboxClass=microvm'
```
### 3. Run the microVM demo
```sh
./hack/run-microvm-demo-kind.sh
```
This is a one-shot bring-up: it deploys the control plane, installs the
cluster-wide microVM deps via `hack/install-microvm-deps.sh` — assembling the
guest runtime assets for your architecture (skipped if already present under
`bin/microvm-assets/`), staging them into the in-cluster rustfs bucket, and
applying the `microvm` `SandboxConfig` — then deploys the demo worker pool +
template.
### 4. Verify
```sh
kubectl get pods -n ate-demo-counter-microvm
kubectl get workerpools,actortemplates -A
```
Expected:
```
NAMESPACE NAME DESIRED READY AVAILABLE
ate-demo-counter-microvm workerpool.ate.dev/counter-microvm 1 1 1
NAMESPACE NAME READY AGE
ate-demo-counter-microvm actortemplate.ate.dev/counter-microvm True 1m
```
## Option B: Apple Silicon macOS via Lima
Lima can run a Linux VM with **nested virtualization**, exposing `/dev/kvm` to
Docker (and therefore to the kind node) inside the VM. This is a well-trodden
path — much of Substrate's development happens on macOS via limactl.
> [!IMPORTANT]
> Apple's Virtualization framework
> [supports nested virtualization only on M3 and later](https://developer.apple.com/documentation/virtualization/vzgenericplatformconfiguration/isnestedvirtualizationsupported)
> — on earlier Apple Silicon (M1/M2), Lima fails with
> `[hostagent] Starting VZ ... FATA exiting`. Fall back to Option A on a
> Linux host.
### 1. Install Lima and the Docker CLI
```sh
brew install lima docker
```
### 2. Launch Lima with nested virtualization
The arm64 nested-virtualization kernel regression affects kernels 6.19 and
newer, so use a guest image with an older kernel — pinned here to Ubuntu
24.04 LTS:
```sh
limactl start --name=docker-nested template://docker-rootful --nested-virt --set '.images = [
{"location":"https://cloud-images.ubuntu.com/releases/noble/release/ubuntu-24.04-server-cloudimg-arm64.img","arch":"aarch64"},
{"location":"https://cloud-images.ubuntu.com/releases/noble/release/ubuntu-24.04-server-cloudimg-amd64.img","arch":"x86_64"}
]'
```
When prompted to edit the configuration, set at least:
```yaml
cpus: 8
memory: "16GiB"
nestedVirtualization: true
networks:
- vzNAT: true
mounts:
- location: "~"
writable: true
```
The writable home mount lets the kind/ko workflows write into your checkout,
8 CPUs / 16 GiB is a comfortable floor for the control plane plus a microVM
worker, and `vzNAT` gives the VM outbound networking under the vz VM type.
### 3. Assemble the arm64 assets inside the Lima VM
`hack/microvm-assets/assemble.sh` must run on a Linux host of the target
architecture (on arm64 it builds `virtiofsd` from source with cargo against
Linux-only libraries), so run it in the Lima guest, not on macOS:
```sh
limactl shell docker-nested
# Inside the VM — install build deps once:
sudo apt-get update && sudo apt-get install -y git pkg-config libcap-ng-dev libseccomp-dev zstd
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh # rust via rustup
source "$HOME/.cargo/env"
cd <your substrate checkout> # visible via the writable home mount
./hack/microvm-assets/assemble.sh
exit
```
The assets land in `bin/microvm-assets/arm64/` in your checkout, which is
shared with the host through the home mount — the demo script will find them
there and skip re-assembling.
### 4. Point the Docker CLI at Lima and bring everything up (on macOS)
```sh
export DOCKER_HOST="unix://${HOME}/.lima/docker-nested/sock/docker.sock"
echo 'export DOCKER_HOST="unix://${HOME}/.lima/docker-nested/sock/docker.sock"' >> ~/.zprofile
cd <your substrate checkout>
./hack/create-kind-cluster.sh
./hack/run-microvm-demo-kind.sh
```
Verify as in Option A, step 4.
## Trying it out
On completion, `run-microvm-demo-kind.sh` prints next steps: create an actor
from the `counter-microvm` template, hit the in-RAM counter, then suspend and
resume it and confirm the count continues — proving the guest-memory snapshot
round-tripped. The flow is the same as the
[README Quickstart](../../README.md#quickstart-development), just with the
microVM template; see the
[counter demo's micro-VM variant](../../demos/counter/README.md#micro-vm-variant)
for background. Note that an `actortemplate` reporting `READY=True` in the
verify step already exercises the runtime end-to-end — the golden snapshot
requires a full guest boot and checkpoint.
## Troubleshooting
| Symptom | Root cause | Fix |
|---|---|---|
| `/dev/kvm: permission denied` during the kind KVM probe | Rootless Docker: the probe container's root is remapped to your user, which can't open the device (`660 root:kvm`) | Use rootful Docker, or `sudo chmod 666 /dev/kvm` before `./hack/create-kind-cluster.sh` |
| `cargo not found` from `assemble.sh` | On arm64, `virtiofsd` is built from source | Install the build deps listed in Option B, Step 3 |
| Lima: `[hostagent] Starting VZ ... FATA exiting` on M1/M2 | Apple's Virtualization framework supports nested virtualization only on M3 and later | Use an M3+ Mac, or a Linux/KVM host (Option A) |