Benjamin Elder
cb72d60bdf
This is the companion to #496 In order to grant access to `/dev/kvm` we have to either: - use a device plugin - use a DRA driver - use an NRI plugin An NRI plugin is highly privileged in it's own right for all pods on the host and is difficult to ship portably at the moment. DRA is promising, but not enough functionality is GA yet at our current 1.35+ target. Device plugin fits reasonably well. We do wind up publishing an ~arbitrarily high limit, which has some cost in kubelet memory, but otherwise is relatively clean. This approach is what kata uses currently. Their device plugin is not available unbundled, and we anyhow have a per-node daemonset. atlet is taught to sniff if /dev/kvm appears on the host at all, so we can also stop using the manually labeled nodes for microVM class and instead schedule to the KVM + TUN devices on nodes that advertise them. Later we can migrate to device plugin by using I implemented that already, but I don't think it's worth merging at the moment. We would want consumable capacity to be on by default. We can migrate later without changing the pod spec by using `extendedResourceName`. https://github.com/agent-substrate/substrate/compare/main...BenTheElder:substrate:ateom-microvm-dra NOTE: I confirmed with upstream that device plugin is not going anywhere despite being "v1beta1", it's GA in all but name. It won't receive new features but we don't really need anyhow. We'll move to DRA down the line. --- By doing this, we can drop `privileged: true` from the uVM ateom pods. We can also drop the `ate.dev/sandboxClass` node label hacks, reducing friction to deploy.
Micro-VM runtime assets + counter demo (kind, fetch-not-bake)
The microvm runtime (cmd/ateom-microvm, kata + cloud-hypervisor) fetches its
toolchain at runtime — nothing kata-specific is baked into the worker image. ateom drives
the kata-agent directly (no kata shim, no containerd). Each actor container's rootfs is an
overlay of a read-only lower (the OCI image, served into the guest over virtio-fs by
virtiofsd) and a writable upper on a guest tmpfs, so virtiofsd is part of the asset
set. The asset set is five files:
cloud-hypervisor— the VMM binary (fetched from its release)virtiofsd— the virtio-fs daemon serving the RO lower (built from source; seeassemble.sh)vmlinux— the guest kernel (from kata-static)rootfs.img— the guest rootfs image (from kata-static)configuration-clh.toml— the base kata config (from kata-static)
These helpers assemble the asset set for your node arch, stage it into the cluster's rustfs
S3 bucket, and the demo manifest's SandboxConfig points at it. When /dev/kvm is
available, hack/create-kind-cluster.sh mounts it into the node; atelet then advertises
it as a device, which is what places micro-VM workers there.
Tip
hack/run-microvm-demo.shautomates the full bring-up below (assets, control plane, demo apply) for kind OR GKE without editing committed files. The steps here are the manual equivalent.
Steps (run on a KVM-capable Linux host matching the node arch)
-
Assemble assets for your arch:
ARCH=arm64 hack/microvm-assets/assemble.shCopy the printed sha256 sums into the
SandboxConfigspec.assetsindemos/counter/counter-microvm.yaml.tmpl(the committed values are arm64; other arches differ). -
Bring up the cluster + control plane:
hack/create-kind-cluster.sh # mounts /dev/kvm into the nodes hack/install-ate-kind.sh # control plane + rustfs (bucket: ate-snapshots) -
Stage assets into rustfs:
OUT="$PWD/microvm-assets-arm64" hack/microvm-assets/stage-to-rustfs.sh -
Apply the demo + drive it:
BUCKET_NAME=ate-snapshots envsubst < demos/counter/counter-microvm.yaml.tmpl | kubectl apply -f -Create an actor from
counter-microvm, hit the in-RAM counter to increment it, suspend (checkpoint), resume on a different worker pod, and confirm the count continues — proving the guest-memory snapshot round-tripped across pods.
Notes
assetsis single-arch (unlike runsc's amd64/arm64): stage assets matching the node arch.