This PR is very large since it updates all existing demos and benchmark
workloads to use the new ActorTemplate substrate proto.
Please use the "Commits" tab to review individual commits.
Verifications done:
* Used this script: gpaste/5143788763348992 to verify that the change
from CRD -> proto are equivalent.
* The e2e tests are using the new susbtrate resources.
* Picked the parking demo to run e2e manually: gpaste/6193361380311040
This is the companion to #496
In order to grant access to `/dev/kvm` we have to either:
- use a device plugin
- use a DRA driver
- use an NRI plugin
An NRI plugin is highly privileged in it's own right for all pods on the
host and is difficult to ship portably at the moment.
DRA is promising, but not enough functionality is GA yet at our current
1.35+ target.
Device plugin fits reasonably well. We do wind up publishing an
~arbitrarily high limit, which has some cost in kubelet memory, but
otherwise is relatively clean.
This approach is what kata uses currently. Their device plugin is not
available unbundled, and we anyhow have a per-node daemonset.
atlet is taught to sniff if /dev/kvm appears on the host at all, so we
can also stop using the manually labeled nodes for microVM class and
instead schedule to the KVM + TUN devices on nodes that advertise them.
Later we can migrate to device plugin by using
I implemented that already, but I don't think it's worth merging at the
moment. We would want consumable capacity to be on by default. We can
migrate later without changing the pod spec by using
`extendedResourceName`.
https://github.com/agent-substrate/substrate/compare/main...BenTheElder:substrate:ateom-microvm-dra
NOTE: I confirmed with upstream that device plugin is not going anywhere
despite being "v1beta1", it's GA in all but name. It won't receive new
features but we don't really need anyhow. We'll move to DRA down the
line.
---
By doing this, we can drop `privileged: true` from the uVM ateom pods.
We can also drop the `ate.dev/sandboxClass` node label hacks, reducing
friction to deploy.
### Why
The README Quickstart covers getting a local Substrate cluster running
with
the default gVisor runtime, but there's no documentation for the
micro-VM
runtime, where the friction can't be scripted away: it requires
`/dev/kvm`
(bare metal, nested virtualization, or Lima on macOS), and on Apple
Silicon
the Lima configuration and asset assembly are non-obvious. New
contributors
have to reverse-engineer the `hack/` scripts.
### What
Adds `docs/dev/microvm-local.md` — "Running the microVM runtime locally"
— a
focused guide covering only the microVM delta, based on notes from real
onboarding runs. General setup is deferred to the README Quickstart
(listed
as the guide's prerequisite) rather than duplicated:
- **Option A: Linux host with KVM** — verifying `/dev/kvm` and CPU virt
support, the rootless-Docker caveat for the KVM probe, cluster creation,
and the one-shot `run-microvm-demo-kind.sh` bring-up.
- **Option B: Apple Silicon macOS via Lima** — nested virtualization
with the
guest image pinned to Ubuntu 25.10 (until the kernel issue in the
default
image is fixed), `vzNAT` networking, and assembling the arm64 assets
inside
the Lima guest (`assemble.sh` requires a Linux host of the target arch).
- **Trying it out** — defers to the next steps the demo script prints
and
links the counter demo's micro-VM variant, instead of duplicating those
commands.
- **Troubleshooting** — symptom → root cause → fix entries actually hit
during onboarding (rootless-Docker KVM probe failures, the `aws` CLI
staging requirement until #804 lands, arm64 `virtiofsd` build deps, M1
lacking FEAT_NV2).
Zoe Zhao