eliranw 934c7418d0 feat: CDI-based NVIDIA GPU passthrough into gVisor actor containers (#502) ...

## Summary

- `atecontroller` propagates a pool's `nvidia.com/gpu` request onto the
`ateom` container and mounts the host NVIDIA toolkit read-only (path
overridable via `ATE_NVIDIA_TOOLKIT_HOST_PATH`)
- `ateom-gvisor` generates a CDI spec with `nvidia-ctk` and injects the
device nodes, driver-library mounts, and env into each actor container's
OCI spec
- runs the CDI `createContainer` hooks except `update-ldcache` which
needs a privileged ateom, staging the SONAME symlinks it would create
from each library's ELF `DT_SONAME`
- enables `runsc --nvproxy` at sandbox creation

Requesting `nvidia.com/gpu` on the pool is the only configuration
needed; a pool that requests N GPUs makes all N usable.

Two details of the CDI spec are worth calling out, because getting
either wrong fails at runtime rather than at parse time. `nvidia-ctk`
leaves `major`/`minor` unset — CDI delegates that to the OCI runtime —
so each device node is resolved by stat-ing the host; without it the
actor gets `0,0` char devices and NVML reports it cannot communicate
with the driver. And it emits per-index, per-UUID, and `all` devices
that repeat the same nodes, so only `all` is applied. The spec is plain
JSON, so `encoding/json` suffices and no CDI library is vendored.

`update-ldcache` is the one hook that cannot run here: its `ldconfig`
unshares a mount namespace and mounts a private `/proc`, which
`mount_too_revealing()` rejects under the pod's masked `/proc`.
Permitting it would need `procMount: Unmasked`, which Kubernetes only
allows with `hostUsers: false`, and that user namespace breaks the
per-actor cgroup delegation from #496. Skipping it avoids the whole
chain, so a GPU worker keeps the same posture as any other unprivileged
gVisor worker. `create-symlinks` and `enable-cuda-compat` still run
unmodified.

`--nvproxy` must be set when the sandbox is created — the `pause`
container, which holds no GPU devices — so runsc's auto-detection never
fires on its own; without the flag the GPU subcontainer crashes the
sentry on start. GPU detection matches any device index rather than
assuming `/dev/nvidia0`, since a worker sharing a multi-GPU node can be
assigned `/dev/nvidia2` and `/dev/nvidia3`.

GPU pools must set `spec.ateomImage` to a glibc build
(`KO_DEFAULTBASEIMAGE=debian:stable-slim ko build ./cmd/ateom-gvisor`)
because the distroless default cannot exec `nvidia-ctk`; the default
base is unchanged for every other pool. `atelet` also has to run on the
GPU nodes to restore actors there, so its DaemonSet needs a toleration
for whatever taint they carry. Both are documented in the API guide
rather than defaulted.

## Testing

- `make test`
- `env -u NO_COLOR make verify`
- Real GPU, Tesla T4 / driver 580.65.06, through the full actor flow:
`nvidia-smi`, `vectorAdd`, `nbody` at 3.77 TFLOP/s, PyTorch matmul via
cuBLAS at 3.5 TFLOP/s (T4 peak FP32 is ~8.1, so no measurable sandbox
penalty)
- Actor whose entrypoint runs the CUDA sample directly exits 0,
confirming the injected env reaches the workload without help from the
test harness
- Two workers holding two GPUs each on one 4-GPU node see disjoint
device sets

Snapshot and restore work when the workload holds no CUDA context. A
live CUDA context cannot be checkpointed - gVisor fails with `can't save
with live nvproxy clients` and the failed checkpoint terminates the
sandbox so a GPU actor can only be suspended between CUDA workloads.
Documented as a known limitation in the API
guide; a follow-up issue will track lifting it via `cuda-checkpoint`.

Fixes #627 

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

---------

Signed-off-by: Eliran Wolff <eliranw@nvidia.com>

2026-08-07 15:08:44 -07:00

..

controllers

feat: CDI-based NVIDIA GPU passthrough into gVisor actor containers (#502)

2026-08-07 15:08:44 -07:00