mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-10-02 03:24:42 +08:00
Benjamin Elder
cb72d60bdf
This is the companion to #496 In order to grant access to `/dev/kvm` we have to either: - use a device plugin - use a DRA driver - use an NRI plugin An NRI plugin is highly privileged in it's own right for all pods on the host and is difficult to ship portably at the moment. DRA is promising, but not enough functionality is GA yet at our current 1.35+ target. Device plugin fits reasonably well. We do wind up publishing an ~arbitrarily high limit, which has some cost in kubelet memory, but otherwise is relatively clean. This approach is what kata uses currently. Their device plugin is not available unbundled, and we anyhow have a per-node daemonset. atlet is taught to sniff if /dev/kvm appears on the host at all, so we can also stop using the manually labeled nodes for microVM class and instead schedule to the KVM + TUN devices on nodes that advertise them. Later we can migrate to device plugin by using I implemented that already, but I don't think it's worth merging at the moment. We would want consumable capacity to be on by default. We can migrate later without changing the pod spec by using `extendedResourceName`. https://github.com/agent-substrate/substrate/compare/main...BenTheElder:substrate:ateom-microvm-dra NOTE: I confirmed with upstream that device plugin is not going anywhere despite being "v1beta1", it's GA in all but name. It won't receive new features but we don't really need anyhow. We'll move to DRA down the line. --- By doing this, we can drop `privileged: true` from the uVM ateom pods. We can also drop the `ate.dev/sandboxClass` node label hacks, reducing friction to deploy.