substrate/docs/dev/microvm-local.md

Running the microVM runtime locally

The microVM sandbox class (ateom-microvm: a Kata guest on Cloud Hypervisor) needs /dev/kvm, which takes some extra setup compared to the default gVisor path. This guide covers just that delta: getting a KVM-capable Docker environment — on Linux, or on Apple Silicon macOS via Lima — then running the microVM counter demo and verifying a guest-memory snapshot round-trip.

Prerequisites

Complete the Quickstart (Development) in the README first — it covers the base tooling and the default (gVisor) path this guide builds on. For background on the runtime, see architecture.md and hack/microvm-assets/README.md.

Option A: Linux host with KVM

Works on bare-metal Linux or any cloud VM with nested virtualization enabled (e.g. GCE N4/N4D instances with nested virt, or equivalent on other clouds).

1. Verify KVM

ls -la /dev/kvm
# Expected: crw-rw---- 1 root kvm 10, 232 ... /dev/kvm

grep -cE '(vmx|svm)' /proc/cpuinfo   # >0 means CPU virt support (x86)

hack/create-kind-cluster.sh probes for KVM by running a root container with --device /dev/kvm, which works out of the box with a standard (rootful) Docker install. With rootless Docker the container's root is remapped to your user, so the probe fails with permission denied — use rootful Docker instead, or open up the device with sudo chmod 666 /dev/kvm.

2. Create the cluster

./hack/create-kind-cluster.sh
# Look for: "/dev/kvm found: micro-VM (kata + cloud-hypervisor) support will be enabled."

Once the control plane is up, atelet advertises the device on each KVM-capable node, which is what places micro-VM workers:

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.capacity.ate\.dev/kvm}{"\n"}{end}'

3. Run the microVM demo

./hack/run-microvm-demo-kind.sh

This is a one-shot bring-up: it deploys the control plane, installs the cluster-wide microVM deps via hack/install-microvm-deps.sh — assembling the guest runtime assets for your architecture (skipped if already present under bin/microvm-assets/), staging them into the in-cluster rustfs bucket, and applying the microvm SandboxConfig — then deploys the demo worker pool + template.

4. Verify

kubectl get pods -n ate-demo-counter-microvm
kubectl get workerpools -A
kubectl ate get actor-templates -a ate-demo-counter-microvm

Expected:

NAMESPACE                  NAME                                 DESIRED  READY  AVAILABLE
ate-demo-counter-microvm   workerpool.ate.dev/counter-microvm   1        1      1

ATESPACE                   NAME              SANDBOX CLASS           GOLDEN SNAPSHOT                        ERROR   AGE
ate-demo-counter-microvm   counter-microvm   SANDBOX_CLASS_MICROVM   b9f6bd93-3c5a-4b64-9d5e-2f8a1c7d0e42           1m

The template is ready once the GOLDEN SNAPSHOT column is non-empty (the value is a UUID); ERROR means the golden build failed, and -o yaml shows the full error message. An empty GOLDEN SNAPSHOT with no ERROR means the golden build — a full guest boot plus checkpoint — is still running.

Option B: Apple Silicon macOS via Lima

Lima can run a Linux VM with nested virtualization, exposing /dev/kvm to Docker (and therefore to the kind node) inside the VM. This is a well-trodden path — much of Substrate's development happens on macOS via limactl.

Important

Apple's Virtualization framework supports nested virtualization only on M3 and later — on earlier Apple Silicon (M1/M2), Lima fails with [hostagent] Starting VZ ... FATA exiting. Fall back to Option A on a Linux host.

1. Install Lima and the Docker CLI

brew install lima docker

2. Launch Lima with nested virtualization

The arm64 nested-virtualization kernel regression affects kernels 6.19 and newer, so use a guest image with an older kernel — pinned here to Ubuntu 24.04 LTS:

limactl start --name=docker-nested template://docker-rootful --nested-virt --set '.images = [
{"location":"https://cloud-images.ubuntu.com/releases/noble/release/ubuntu-24.04-server-cloudimg-arm64.img","arch":"aarch64"},
{"location":"https://cloud-images.ubuntu.com/releases/noble/release/ubuntu-24.04-server-cloudimg-amd64.img","arch":"x86_64"}
]'

When prompted to edit the configuration, set at least:

cpus: 8
memory: "16GiB"
nestedVirtualization: true
networks:
  - vzNAT: true
mounts:
  - location: "~"
    writable: true

The writable home mount lets the kind/ko workflows write into your checkout, 8 CPUs / 16 GiB is a comfortable floor for the control plane plus a microVM worker, and vzNAT gives the VM outbound networking under the vz VM type.

3. Assemble the arm64 assets inside the Lima VM

Assemble the arm64 assets in the guest because macOS does not ship zstd by default:

limactl shell docker-nested

# Inside the VM — 24.04's git 2.43 can't read a reftable checkout:
sudo add-apt-repository -y ppa:git-core/ppa
sudo apt-get install -y git

cd <your substrate checkout>    # visible via the writable home mount
./hack/microvm-assets/assemble.sh
exit

The script downloads about 590 MB and takes a minute or two, leaving ~285 MB in bin/microvm-assets/arm64/. That directory is shared with the host through the home mount — the demo script will find the assets there and skip re-assembling.

4. Point the Docker CLI at Lima and bring everything up (on macOS)

export DOCKER_HOST="unix://${HOME}/.lima/docker-nested/sock/docker.sock"
echo 'export DOCKER_HOST="unix://${HOME}/.lima/docker-nested/sock/docker.sock"' >> ~/.zprofile

cd <your substrate checkout>

./hack/create-kind-cluster.sh
./hack/run-microvm-demo-kind.sh

Verify as in Option A, step 4.

Trying it out

On completion, run-microvm-demo-kind.sh prints next steps: create an actor from the counter-microvm template, hit the in-RAM counter, then suspend and resume it and confirm the count continues — proving the guest-memory snapshot round-tripped. The flow is the same as the README Quickstart, just with the microVM template; see the counter demo's micro-VM variant for background. Note that an actor template showing a GOLDEN SNAPSHOT in the verify step already exercises the runtime end-to-end — the golden snapshot requires a full guest boot and checkpoint.

Troubleshooting

Symptom	Root cause	Fix
/dev/kvm: permission denied during the kind KVM probe	Rootless Docker: the probe container's root is remapped to your user, which can't open the device (660 root:kvm)	Use rootful Docker, or sudo chmod 666 /dev/kvm before ./hack/create-kind-cluster.sh
Lima: [hostagent] Starting VZ ... FATA exiting on M1/M2	Apple's Virtualization framework supports nested virtualization only on M3 and later	Use an M3+ Mac, or a Linux/KVM host (Option A)