mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-11-11 10:23:17 +08:00
eliranw
afcd5cd718
Per-container CPU and memory limits for micro-VM actors. First slice of #752. An `ActorTemplate` container can cap its own cpu and memory so it cannot starve or kill its siblings in the same actor. A container that exceeds its memory limit is OOM-killed on its own; the rest of the actor is unaffected. ```yaml sandboxClass: microvm containers: - name: trainer resources: limits: {memory: 1500Mi} - name: sidecar resources: limits: {memory: 256Mi, cpu: "0.2"} ``` #752 asks for per-container device selection as well. This PR does the cpu and memory half; devices are the remaining half, and when they land, GPU assignment should follow the same field rather than staying actor-wide as it is today. ## Why micro-VM only The two sandbox classes support opposite halves of #752. Micro-VM actors have a real guest kernel, so each container gets its own cgroup and the limits bind. gVisor applies cgroup limits at the sandbox level: one sentry backs every container in the actor, so a per-container cgroup is created and then stays empty (google/gvisor#190). Measured on a running actor, the workload container's cgroup reported `memory.current=0` while all 20 sandbox processes sat in the pause leaf. A template that sets `resources` with `sandboxClass: gvisor` is rejected at admission. ## How a limit travels `ActorTemplate` → CEL validation → `ate-api-server` resolves each `resource.Quantity` once → `ateletpb` → atelet writes OCI `linux.resources` → `ateom-microvm` merges kata's defaults and checks the guest envelope → `SpecToAgentPB` → kata agent → guest cgroup. The limit is carried as a standard OCI field rather than a substrate-private concept, so a runtime that gains per-container enforcement picks it up without new plumbing. ## Composition with #679 #679 sizes the sandbox itself from `ActorTemplate.spec.resources`: guest RAM and vCPUs for a micro-VM, the sentry for gVisor. This PR subdivides that sandbox. The two are different fields and complementary layers. They collided in one place. #679 applied the actor-level size to **every** container's OCI spec, overwriting the per-container limits atelet writes into the same field, so a container asking for 64Mi silently received the whole guest. This PR removes that call on the micro-VM path. A container now gets a cgroup limit only when it declares one; an undeclared container is bounded by guest RAM, which is the real ceiling. On micro-VM the stamp never reached the guest at all: `SpecToAgentPB` carried neither `Memory` nor `CPU.Quota`, so it stopped at the bundle's `config.json`. Running the pre-PR pipeline shows `Quota=50000 Memory=2Gi` on disk arriving as `Quota=0 Memory=<nil>` on the wire. gVisor is untouched. There the actor-level size is applied to the `pause` container, and since one sentry backs every container, that is the only cgroup that binds. Two consequences worth calling out for reviewers of #679: - The `sizing.SandboxSize` threaded into `buildActorContainers` and `ensureKataCompatibleSpec` had no remaining reader, so it and the `guestSize` helper are removed. `resolveGuestMemMiB` still sizes the VM and still rejects a declared limit too small to boot. - `internal/sizing`'s package doc claimed both runtimes shared `ApplyToOCISpec` for container cgroups. That is now gVisor-only, and the comments say so. No code in that file changed. `checkResourceEnvelope` runs after `resolveGuestMemMiB`, so it validates the per-container sum against the post-reserve guest rather than the SandboxConfig default. Its error now names `spec.resources.limits.memory` (or `.cpu`) when the actor declared a size, and `SandboxConfig` when it did not. `internal/e2e/suites/sizing` is unaffected: it asserts on `num_cpu` and `mem_total_bytes`, both of which come from VM sizing, and only logs the cgroup files. ## Verified on hardware Same template, run twice, one commit apart on a micro-VM actor: | | before | after | | :--- | :--- | :--- | | `hog_ovl/memory.max` | `max` | `67108864` (the declared 64Mi) | | `bystander_ovl/memory.max` | `max` | `max` | | hog allocates 128MB | survived | OOM-killed | | bystander | alive | alive | The bug in between: `SpecToAgentPB` converted only `Devices` and `CPU.Shares` out of `Linux.Resources`, so a memory limit reached the bundle's `config.json` and was dropped on the way to the agent. Every unit test passed and the on-disk spec was correct while the feature did nothing. ## Notes for review - `ContainerResources` deliberately does not reuse `corev1.ResourceRequirements`, which also carries `requests` and `claims`. There is no scheduler inside an actor to hint at, and for memory a soft request cannot express "must have this much to come back at all". #679 reached the same conclusion from the other direction and now rejects `spec.resources.requests` and `.claims` at admission. - `Limits` is a bounded named type. A `MaxProperties` marker on the field lands at the wrong schema level for a named map, and without a bound the CEL cost estimator rejects the whole schema. This CRD is close to its schema-wide CEL cost ceiling; a rule that iterates containers and parses quantities exceeds it by more than 100x and takes the existing rules down with it. - A cpu limit below `10m` is raised to `10m`: the kernel rejects a CFS quota under 1ms. - `SpecToAgentPB` drops a non-positive quota and a zero period rather than forwarding them. A non-positive quota means unlimited in OCI but was sent as a literal zero, which the guest applies as no CPU at all; a non-nil zero period overwrote the CFS default set for a live quota. Both now read a spec the way `cpuLimitMillis` does, so the envelope check and the conversion agree on the same input. - `mergeKataResources` fills the gaps kata's defaults cover rather than allowlisting known fields, so a field it does not recognise survives the merge. That holds only as far as the merge: `SpecToAgentPB` converts `Devices`, `Memory` and `CPU` only, so `Pids`, `BlockIO`, `HugepageLimits` and `Network` stop at the ttrpc boundary. Nothing sets them today. ## Known gaps - An OOM-killed container is not reported above the guest. The actor stays `STATUS_RUNNING` and nothing records which container died. Raised on #550. Note that #961 settled workload stats at the sandbox level and deliberately does not attribute per container, so surfacing which container died needs a separate signal: the guest cgroup's `memory.events`, not the stats path. - A container's writable rootfs is a guest tmpfs, so filesystem writes are charged to its memory limit and a container writing more than its limit to `/tmp` is OOM-killed. Documented in `api-guide.md`; separating the two budgets needs an `ephemeral-storage` field. - Over-subscription across containers is caught when the actor starts, not at apply time. An admission-time sum check is not implementable within the CEL cost budget. - The atelet hop that attaches resources to the bundle spec has no test: replacing `ctr.GetResources()` with `nil` leaves the whole suite green. Closing it needs an imagecache fixture for `prepareOCIDirectory`. - Restored actors inherit the golden's cgroups rather than applying their own spec. Correct today only because `ActorTemplateSpec` is immutable. - The GPU section of `api-guide.md` still says `ActorTemplate` has no per-container resource fields, which this PR makes false. Correcting it is left to the devices half of #752, which is what will change the GPU behaviour that paragraph describes. --- - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR --------- Signed-off-by: Eliran Wolff <eliranw@nvidia.com>