mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-11-10 10:13:50 +08:00
eliranw
934c7418d0
## Summary - `atecontroller` propagates a pool's `nvidia.com/gpu` request onto the `ateom` container and mounts the host NVIDIA toolkit read-only (path overridable via `ATE_NVIDIA_TOOLKIT_HOST_PATH`) - `ateom-gvisor` generates a CDI spec with `nvidia-ctk` and injects the device nodes, driver-library mounts, and env into each actor container's OCI spec - runs the CDI `createContainer` hooks except `update-ldcache` which needs a privileged ateom, staging the SONAME symlinks it would create from each library's ELF `DT_SONAME` - enables `runsc --nvproxy` at sandbox creation Requesting `nvidia.com/gpu` on the pool is the only configuration needed; a pool that requests N GPUs makes all N usable. Two details of the CDI spec are worth calling out, because getting either wrong fails at runtime rather than at parse time. `nvidia-ctk` leaves `major`/`minor` unset — CDI delegates that to the OCI runtime — so each device node is resolved by stat-ing the host; without it the actor gets `0,0` char devices and NVML reports it cannot communicate with the driver. And it emits per-index, per-UUID, and `all` devices that repeat the same nodes, so only `all` is applied. The spec is plain JSON, so `encoding/json` suffices and no CDI library is vendored. `update-ldcache` is the one hook that cannot run here: its `ldconfig` unshares a mount namespace and mounts a private `/proc`, which `mount_too_revealing()` rejects under the pod's masked `/proc`. Permitting it would need `procMount: Unmasked`, which Kubernetes only allows with `hostUsers: false`, and that user namespace breaks the per-actor cgroup delegation from #496. Skipping it avoids the whole chain, so a GPU worker keeps the same posture as any other unprivileged gVisor worker. `create-symlinks` and `enable-cuda-compat` still run unmodified. `--nvproxy` must be set when the sandbox is created — the `pause` container, which holds no GPU devices — so runsc's auto-detection never fires on its own; without the flag the GPU subcontainer crashes the sentry on start. GPU detection matches any device index rather than assuming `/dev/nvidia0`, since a worker sharing a multi-GPU node can be assigned `/dev/nvidia2` and `/dev/nvidia3`. GPU pools must set `spec.ateomImage` to a glibc build (`KO_DEFAULTBASEIMAGE=debian:stable-slim ko build ./cmd/ateom-gvisor`) because the distroless default cannot exec `nvidia-ctk`; the default base is unchanged for every other pool. `atelet` also has to run on the GPU nodes to restore actors there, so its DaemonSet needs a toleration for whatever taint they carry. Both are documented in the API guide rather than defaulted. ## Testing - `make test` - `env -u NO_COLOR make verify` - Real GPU, Tesla T4 / driver 580.65.06, through the full actor flow: `nvidia-smi`, `vectorAdd`, `nbody` at 3.77 TFLOP/s, PyTorch matmul via cuBLAS at 3.5 TFLOP/s (T4 peak FP32 is ~8.1, so no measurable sandbox penalty) - Actor whose entrypoint runs the CUDA sample directly exits 0, confirming the injected env reaches the workload without help from the test harness - Two workers holding two GPUs each on one 4-GPU node see disjoint device sets Snapshot and restore work when the workload holds no CUDA context. A live CUDA context cannot be checkpointed - gVisor fails with `can't save with live nvproxy clients` and the failed checkpoint terminates the sandbox so a GPU actor can only be suspended between CUDA workloads. Documented as a known limitation in the API guide; a follow-up issue will track lifting it via `cuda-checkpoint`. Fixes #627 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR --------- Signed-off-by: Eliran Wolff <eliranw@nvidia.com>