mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-11-10 10:13:50 +08:00
Per-container CPU and memory limits for micro-VM actors. First slice of #752. An `ActorTemplate` container can cap its own cpu and memory so it cannot starve or kill its siblings in the same actor. A container that exceeds its memory limit is OOM-killed on its own; the rest of the actor is unaffected. ```yaml sandboxClass: microvm containers: - name: trainer resources: limits: {memory: 1500Mi} - name: sidecar resources: limits: {memory: 256Mi, cpu: "0.2"} ``` #752 asks for per-container device selection as well. This PR does the cpu and memory half; devices are the remaining half, and when they land, GPU assignment should follow the same field rather than staying actor-wide as it is today. ## Why micro-VM only The two sandbox classes support opposite halves of #752. Micro-VM actors have a real guest kernel, so each container gets its own cgroup and the limits bind. gVisor applies cgroup limits at the sandbox level: one sentry backs every container in the actor, so a per-container cgroup is created and then stays empty (google/gvisor#190). Measured on a running actor, the workload container's cgroup reported `memory.current=0` while all 20 sandbox processes sat in the pause leaf. A template that sets `resources` with `sandboxClass: gvisor` is rejected at admission. ## How a limit travels `ActorTemplate` → CEL validation → `ate-api-server` resolves each `resource.Quantity` once → `ateletpb` → atelet writes OCI `linux.resources` → `ateom-microvm` merges kata's defaults and checks the guest envelope → `SpecToAgentPB` → kata agent → guest cgroup. The limit is carried as a standard OCI field rather than a substrate-private concept, so a runtime that gains per-container enforcement picks it up without new plumbing. ## Composition with #679 #679 sizes the sandbox itself from `ActorTemplate.spec.resources`: guest RAM and vCPUs for a micro-VM, the sentry for gVisor. This PR subdivides that sandbox. The two are different fields and complementary layers. They collided in one place. #679 applied the actor-level size to **every** container's OCI spec, overwriting the per-container limits atelet writes into the same field, so a container asking for 64Mi silently received the whole guest. This PR removes that call on the micro-VM path. A container now gets a cgroup limit only when it declares one; an undeclared container is bounded by guest RAM, which is the real ceiling. On micro-VM the stamp never reached the guest at all: `SpecToAgentPB` carried neither `Memory` nor `CPU.Quota`, so it stopped at the bundle's `config.json`. Running the pre-PR pipeline shows `Quota=50000 Memory=2Gi` on disk arriving as `Quota=0 Memory=<nil>` on the wire. gVisor is untouched. There the actor-level size is applied to the `pause` container, and since one sentry backs every container, that is the only cgroup that binds. Two consequences worth calling out for reviewers of #679: - The `sizing.SandboxSize` threaded into `buildActorContainers` and `ensureKataCompatibleSpec` had no remaining reader, so it and the `guestSize` helper are removed. `resolveGuestMemMiB` still sizes the VM and still rejects a declared limit too small to boot. - `internal/sizing`'s package doc claimed both runtimes shared `ApplyToOCISpec` for container cgroups. That is now gVisor-only, and the comments say so. No code in that file changed. `checkResourceEnvelope` runs after `resolveGuestMemMiB`, so it validates the per-container sum against the post-reserve guest rather than the SandboxConfig default. Its error now names `spec.resources.limits.memory` (or `.cpu`) when the actor declared a size, and `SandboxConfig` when it did not. `internal/e2e/suites/sizing` is unaffected: it asserts on `num_cpu` and `mem_total_bytes`, both of which come from VM sizing, and only logs the cgroup files. ## Verified on hardware Same template, run twice, one commit apart on a micro-VM actor: | | before | after | | :--- | :--- | :--- | | `hog_ovl/memory.max` | `max` | `67108864` (the declared 64Mi) | | `bystander_ovl/memory.max` | `max` | `max` | | hog allocates 128MB | survived | OOM-killed | | bystander | alive | alive | The bug in between: `SpecToAgentPB` converted only `Devices` and `CPU.Shares` out of `Linux.Resources`, so a memory limit reached the bundle's `config.json` and was dropped on the way to the agent. Every unit test passed and the on-disk spec was correct while the feature did nothing. ## Notes for review - `ContainerResources` deliberately does not reuse `corev1.ResourceRequirements`, which also carries `requests` and `claims`. There is no scheduler inside an actor to hint at, and for memory a soft request cannot express "must have this much to come back at all". #679 reached the same conclusion from the other direction and now rejects `spec.resources.requests` and `.claims` at admission. - `Limits` is a bounded named type. A `MaxProperties` marker on the field lands at the wrong schema level for a named map, and without a bound the CEL cost estimator rejects the whole schema. This CRD is close to its schema-wide CEL cost ceiling; a rule that iterates containers and parses quantities exceeds it by more than 100x and takes the existing rules down with it. - A cpu limit below `10m` is raised to `10m`: the kernel rejects a CFS quota under 1ms. - `SpecToAgentPB` drops a non-positive quota and a zero period rather than forwarding them. A non-positive quota means unlimited in OCI but was sent as a literal zero, which the guest applies as no CPU at all; a non-nil zero period overwrote the CFS default set for a live quota. Both now read a spec the way `cpuLimitMillis` does, so the envelope check and the conversion agree on the same input. - `mergeKataResources` fills the gaps kata's defaults cover rather than allowlisting known fields, so a field it does not recognise survives the merge. That holds only as far as the merge: `SpecToAgentPB` converts `Devices`, `Memory` and `CPU` only, so `Pids`, `BlockIO`, `HugepageLimits` and `Network` stop at the ttrpc boundary. Nothing sets them today. ## Known gaps - An OOM-killed container is not reported above the guest. The actor stays `STATUS_RUNNING` and nothing records which container died. Raised on #550. Note that #961 settled workload stats at the sandbox level and deliberately does not attribute per container, so surfacing which container died needs a separate signal: the guest cgroup's `memory.events`, not the stats path. - A container's writable rootfs is a guest tmpfs, so filesystem writes are charged to its memory limit and a container writing more than its limit to `/tmp` is OOM-killed. Documented in `api-guide.md`; separating the two budgets needs an `ephemeral-storage` field. - Over-subscription across containers is caught when the actor starts, not at apply time. An admission-time sum check is not implementable within the CEL cost budget. - The atelet hop that attaches resources to the bundle spec has no test: replacing `ctr.GetResources()` with `nil` leaves the whole suite green. Closing it needs an imagecache fixture for `prepareOCIDirectory`. - Restored actors inherit the golden's cgroups rather than applying their own spec. Correct today only because `ActorTemplateSpec` is immutable. - The GPU section of `api-guide.md` still says `ActorTemplate` has no per-container resource fields, which this PR makes false. Correcting it is left to the devices half of #752, which is what will change the GPU behaviour that paragraph describes. --- - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR --------- Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
667 lines
19 KiB
Go
667 lines
19 KiB
Go
// Copyright 2026 Google LLC
|
|
//
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
package controlapi
|
|
|
|
import (
|
|
"testing"
|
|
|
|
"github.com/agent-substrate/substrate/internal/proto/ateletpb"
|
|
atev1alpha1 "github.com/agent-substrate/substrate/pkg/api/v1alpha1"
|
|
"github.com/agent-substrate/substrate/pkg/proto/ateapipb"
|
|
"github.com/google/go-cmp/cmp"
|
|
"google.golang.org/protobuf/testing/protocmp"
|
|
corev1 "k8s.io/api/core/v1"
|
|
"k8s.io/apimachinery/pkg/api/resource"
|
|
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
|
)
|
|
|
|
func TestWorkloadSpecFromActorTemplate(t *testing.T) {
|
|
tests := []struct {
|
|
name string
|
|
template *atev1alpha1.ActorTemplate
|
|
want *ateletpb.WorkloadSpec
|
|
}{
|
|
{
|
|
name: "converts DurableDir volume and mounts",
|
|
template: &atev1alpha1.ActorTemplate{
|
|
ObjectMeta: metav1.ObjectMeta{Name: "tmpl1", Namespace: "agent-ns"},
|
|
Spec: atev1alpha1.ActorTemplateSpec{
|
|
Volumes: []atev1alpha1.Volume{
|
|
{Name: "home", VolumeSource: atev1alpha1.VolumeSource{DurableDir: &atev1alpha1.DurableDirVolumeSource{}}},
|
|
},
|
|
Containers: []atev1alpha1.Container{
|
|
{
|
|
Name: "main",
|
|
Image: "main",
|
|
VolumeMounts: []atev1alpha1.VolumeMount{
|
|
{Name: "home", MountPath: "/home/user"},
|
|
{Name: "home", MountPath: "/workspace"},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
want: &ateletpb.WorkloadSpec{
|
|
Volumes: []*ateletpb.Volume{
|
|
{
|
|
Name: "home",
|
|
Source: &ateletpb.Volume_DurableDir{DurableDir: &ateletpb.DurableDirVolume{}},
|
|
},
|
|
},
|
|
Containers: []*ateletpb.Container{
|
|
{
|
|
Name: "main",
|
|
Image: "main",
|
|
VolumeMounts: []*ateletpb.VolumeMount{
|
|
{Name: "home", MountPath: "/home/user"},
|
|
{Name: "home", MountPath: "/workspace"},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
{
|
|
name: "converts SystemInfo volume with actorMetadata items",
|
|
template: &atev1alpha1.ActorTemplate{
|
|
ObjectMeta: metav1.ObjectMeta{Name: "tmpl1", Namespace: "agent-ns"},
|
|
Spec: atev1alpha1.ActorTemplateSpec{
|
|
Volumes: []atev1alpha1.Volume{
|
|
{
|
|
Name: "system-info",
|
|
VolumeSource: atev1alpha1.VolumeSource{
|
|
SystemInfo: &atev1alpha1.SystemInfoVolumeSource{
|
|
DataSources: []atev1alpha1.SystemInfoDataSource{
|
|
{ActorMetadata: &atev1alpha1.ActorMetadataDataSource{
|
|
Items: []atev1alpha1.ActorMetadataItem{
|
|
{Field: atev1alpha1.ActorMetadataFieldName, Path: "actor-name"},
|
|
{Field: atev1alpha1.ActorMetadataFieldAtespace, Path: "atespace"},
|
|
{Field: atev1alpha1.ActorMetadataFieldUID, Path: "identity/actor-uid"},
|
|
},
|
|
}},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
Containers: []atev1alpha1.Container{
|
|
{
|
|
Name: "main",
|
|
Image: "main",
|
|
VolumeMounts: []atev1alpha1.VolumeMount{
|
|
{Name: "system-info", MountPath: "/run/ate"},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
want: &ateletpb.WorkloadSpec{
|
|
Volumes: []*ateletpb.Volume{
|
|
{
|
|
Name: "system-info",
|
|
Source: &ateletpb.Volume_SystemInfo{
|
|
SystemInfo: &ateletpb.SystemInfoVolume{
|
|
DataSources: []*ateletpb.SystemInfoDataSource{
|
|
{DataSource: &ateletpb.SystemInfoDataSource_ActorMetadata{
|
|
ActorMetadata: &ateletpb.ActorMetadataDataSource{
|
|
Items: []*ateletpb.ActorMetadataItem{
|
|
{Field: ateletpb.ActorMetadataField_ACTOR_METADATA_FIELD_NAME, Path: "actor-name"},
|
|
{Field: ateletpb.ActorMetadataField_ACTOR_METADATA_FIELD_ATESPACE, Path: "atespace"},
|
|
{Field: ateletpb.ActorMetadataField_ACTOR_METADATA_FIELD_UID, Path: "identity/actor-uid"},
|
|
},
|
|
},
|
|
}},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
Containers: []*ateletpb.Container{
|
|
{
|
|
Name: "main",
|
|
Image: "main",
|
|
VolumeMounts: []*ateletpb.VolumeMount{
|
|
{Name: "system-info", MountPath: "/run/ate"},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
{
|
|
name: "converts Image volume and mounts",
|
|
template: &atev1alpha1.ActorTemplate{
|
|
ObjectMeta: metav1.ObjectMeta{Name: "tmpl1", Namespace: "agent-ns"},
|
|
Spec: atev1alpha1.ActorTemplateSpec{
|
|
Volumes: []atev1alpha1.Volume{
|
|
{Name: "home", VolumeSource: atev1alpha1.VolumeSource{DurableDir: &atev1alpha1.DurableDirVolumeSource{}}},
|
|
{Name: "agent", VolumeSource: atev1alpha1.VolumeSource{Image: &atev1alpha1.ImageVolumeSource{Reference: "example.com/agent@sha256:abc"}}},
|
|
},
|
|
Containers: []atev1alpha1.Container{
|
|
{
|
|
Name: "main",
|
|
Image: "main",
|
|
VolumeMounts: []atev1alpha1.VolumeMount{
|
|
{Name: "home", MountPath: "/home/user"},
|
|
{Name: "agent", MountPath: "/ate"},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
want: &ateletpb.WorkloadSpec{
|
|
Volumes: []*ateletpb.Volume{
|
|
{
|
|
Name: "home",
|
|
Source: &ateletpb.Volume_DurableDir{DurableDir: &ateletpb.DurableDirVolume{}},
|
|
},
|
|
{
|
|
Name: "agent",
|
|
Source: &ateletpb.Volume_Image{Image: &ateletpb.ImageVolumeSource{Reference: "example.com/agent@sha256:abc"}},
|
|
},
|
|
},
|
|
Containers: []*ateletpb.Container{
|
|
{
|
|
Name: "main",
|
|
Image: "main",
|
|
VolumeMounts: []*ateletpb.VolumeMount{
|
|
{Name: "home", MountPath: "/home/user"},
|
|
{Name: "agent", MountPath: "/ate"},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
{
|
|
name: "converts trustBundle data sources carrying the name for node-side resolution",
|
|
template: &atev1alpha1.ActorTemplate{
|
|
ObjectMeta: metav1.ObjectMeta{Name: "tmpl1", Namespace: "agent-ns"},
|
|
Spec: atev1alpha1.ActorTemplateSpec{
|
|
Volumes: []atev1alpha1.Volume{
|
|
{
|
|
Name: "system-info",
|
|
VolumeSource: atev1alpha1.VolumeSource{
|
|
SystemInfo: &atev1alpha1.SystemInfoVolumeSource{
|
|
DataSources: []atev1alpha1.SystemInfoDataSource{
|
|
{TrustBundle: &atev1alpha1.TrustBundleDataSource{Name: "egress-trust", Path: "trust/ca.pem"}},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
Containers: []atev1alpha1.Container{
|
|
{
|
|
Name: "main",
|
|
Image: "main",
|
|
VolumeMounts: []atev1alpha1.VolumeMount{
|
|
{Name: "system-info", MountPath: "/run/substrate/certs"},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
// The NAME crosses the wire; atelet resolves it against its
|
|
// allowlist and ClusterTrustBundle informer at write time.
|
|
want: &ateletpb.WorkloadSpec{
|
|
Volumes: []*ateletpb.Volume{
|
|
{
|
|
Name: "system-info",
|
|
Source: &ateletpb.Volume_SystemInfo{
|
|
SystemInfo: &ateletpb.SystemInfoVolume{
|
|
DataSources: []*ateletpb.SystemInfoDataSource{
|
|
{DataSource: &ateletpb.SystemInfoDataSource_TrustBundle{
|
|
TrustBundle: &ateletpb.TrustBundleDataSource{Name: "egress-trust", Path: "trust/ca.pem"},
|
|
}},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
Containers: []*ateletpb.Container{
|
|
{
|
|
Name: "main",
|
|
Image: "main",
|
|
VolumeMounts: []*ateletpb.VolumeMount{
|
|
{Name: "system-info", MountPath: "/run/substrate/certs"},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
{
|
|
name: "skips non-DurableDir volumes",
|
|
template: &atev1alpha1.ActorTemplate{
|
|
ObjectMeta: metav1.ObjectMeta{Name: "tmpl1", Namespace: "agent-ns"},
|
|
Spec: atev1alpha1.ActorTemplateSpec{
|
|
Volumes: []atev1alpha1.Volume{
|
|
{Name: "unsupported", VolumeSource: atev1alpha1.VolumeSource{}},
|
|
{Name: "home", VolumeSource: atev1alpha1.VolumeSource{DurableDir: &atev1alpha1.DurableDirVolumeSource{}}},
|
|
},
|
|
Containers: []atev1alpha1.Container{
|
|
{
|
|
Name: "main",
|
|
Image: "main",
|
|
VolumeMounts: []atev1alpha1.VolumeMount{
|
|
{Name: "home", MountPath: "/workspace"},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
want: &ateletpb.WorkloadSpec{
|
|
Volumes: []*ateletpb.Volume{
|
|
{
|
|
Name: "home",
|
|
Source: &ateletpb.Volume_DurableDir{DurableDir: &ateletpb.DurableDirVolume{}},
|
|
},
|
|
},
|
|
Containers: []*ateletpb.Container{
|
|
{
|
|
Name: "main",
|
|
Image: "main",
|
|
VolumeMounts: []*ateletpb.VolumeMount{
|
|
{Name: "home", MountPath: "/workspace"},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
{
|
|
name: "container without volume mounts has none",
|
|
template: &atev1alpha1.ActorTemplate{
|
|
ObjectMeta: metav1.ObjectMeta{Name: "tmpl1", Namespace: "agent-ns"},
|
|
Spec: atev1alpha1.ActorTemplateSpec{
|
|
Volumes: []atev1alpha1.Volume{
|
|
{Name: "home", VolumeSource: atev1alpha1.VolumeSource{DurableDir: &atev1alpha1.DurableDirVolumeSource{}}},
|
|
},
|
|
Containers: []atev1alpha1.Container{
|
|
{Name: "main", Image: "main"},
|
|
},
|
|
},
|
|
},
|
|
want: &ateletpb.WorkloadSpec{
|
|
Volumes: []*ateletpb.Volume{
|
|
{
|
|
Name: "home",
|
|
Source: &ateletpb.Volume_DurableDir{DurableDir: &ateletpb.DurableDirVolume{}},
|
|
},
|
|
},
|
|
Containers: []*ateletpb.Container{{Name: "main", Image: "main"}},
|
|
},
|
|
},
|
|
{
|
|
name: "maps literal env",
|
|
template: &atev1alpha1.ActorTemplate{
|
|
ObjectMeta: metav1.ObjectMeta{Name: "tmpl1", Namespace: "agent-ns"},
|
|
Spec: atev1alpha1.ActorTemplateSpec{
|
|
Containers: []atev1alpha1.Container{
|
|
{
|
|
Name: "main",
|
|
Image: "main",
|
|
Env: []atev1alpha1.EnvVar{
|
|
{Name: "LITERAL", Value: "plain"},
|
|
{Name: "EMPTY", Value: ""},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
want: &ateletpb.WorkloadSpec{
|
|
Containers: []*ateletpb.Container{{
|
|
Name: "main",
|
|
Image: "main",
|
|
Env: []*ateletpb.EnvEntry{
|
|
{Name: "LITERAL", Value: "plain"},
|
|
{Name: "EMPTY", Value: ""},
|
|
},
|
|
}},
|
|
},
|
|
},
|
|
{
|
|
name: "maps command and args",
|
|
template: &atev1alpha1.ActorTemplate{
|
|
ObjectMeta: metav1.ObjectMeta{Name: "tmpl1", Namespace: "agent-ns"},
|
|
Spec: atev1alpha1.ActorTemplateSpec{
|
|
Containers: []atev1alpha1.Container{
|
|
{
|
|
Name: "main",
|
|
Image: "main",
|
|
Command: []string{"/entrypoint"},
|
|
Args: []string{"--foo", "--bar"},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
want: &ateletpb.WorkloadSpec{
|
|
Containers: []*ateletpb.Container{{
|
|
Name: "main",
|
|
Image: "main",
|
|
Command: []string{"/entrypoint"},
|
|
Args: []string{"--foo", "--bar"},
|
|
}},
|
|
},
|
|
},
|
|
}
|
|
|
|
for _, tt := range tests {
|
|
t.Run(tt.name, func(t *testing.T) {
|
|
got, err := workloadSpecFromActorTemplate(tt.template, nil)
|
|
if err != nil {
|
|
t.Fatalf("workloadSpecFromActorTemplate failed: %v", err)
|
|
}
|
|
if diff := cmp.Diff(tt.want, got, protocmp.Transform()); diff != "" {
|
|
t.Errorf("WorkloadSpec mismatch (-want +got):\n%s", diff)
|
|
}
|
|
})
|
|
}
|
|
}
|
|
|
|
func TestWorkloadSpecFromActorTemplatePropagatesReadyz(t *testing.T) {
|
|
got, err := workloadSpecFromActorTemplate(&atev1alpha1.ActorTemplate{
|
|
ObjectMeta: metav1.ObjectMeta{Name: "tmpl-readyz", Namespace: "agent-ns"},
|
|
Spec: atev1alpha1.ActorTemplateSpec{
|
|
Containers: []atev1alpha1.Container{
|
|
{
|
|
Name: "with-probe",
|
|
Image: "main",
|
|
Readyz: &atev1alpha1.ContainerReadyz{
|
|
HTTPGet: &atev1alpha1.HTTPGetAction{Path: "/health", Port: 8080},
|
|
TimeoutSeconds: 45,
|
|
},
|
|
},
|
|
{
|
|
Name: "without-probe",
|
|
Image: "side",
|
|
},
|
|
},
|
|
},
|
|
}, nil)
|
|
if err != nil {
|
|
t.Fatalf("workloadSpecFromActorTemplate failed: %v", err)
|
|
}
|
|
|
|
want := &ateletpb.WorkloadSpec{
|
|
Containers: []*ateletpb.Container{
|
|
{
|
|
Name: "with-probe",
|
|
Image: "main",
|
|
Readyz: &ateletpb.Readyz{
|
|
HttpGet: &ateletpb.HTTPGetAction{Path: "/health", Port: 8080},
|
|
TimeoutSeconds: 45,
|
|
},
|
|
},
|
|
{
|
|
Name: "without-probe",
|
|
Image: "side",
|
|
},
|
|
},
|
|
}
|
|
if diff := cmp.Diff(want, got, protocmp.Transform()); diff != "" {
|
|
t.Errorf("WorkloadSpec mismatch (-want +got):\n%s", diff)
|
|
}
|
|
}
|
|
|
|
func TestAppendExternalVolumes(t *testing.T) {
|
|
template := &atev1alpha1.ActorTemplate{
|
|
Spec: atev1alpha1.ActorTemplateSpec{
|
|
Containers: []atev1alpha1.Container{
|
|
{
|
|
Name: "main",
|
|
VolumeMounts: []atev1alpha1.VolumeMount{
|
|
{Name: "vol-1", MountPath: "/mnt/vol1"},
|
|
},
|
|
},
|
|
},
|
|
Volumes: []atev1alpha1.Volume{
|
|
{
|
|
Name: "vol-1",
|
|
VolumeSource: atev1alpha1.VolumeSource{
|
|
ExternalVolumeTemplate: &atev1alpha1.ExternalVolumeTemplate{
|
|
StorageClassName: "pd-standard",
|
|
},
|
|
},
|
|
},
|
|
{
|
|
Name: "vol-2",
|
|
VolumeSource: atev1alpha1.VolumeSource{
|
|
DurableDir: &atev1alpha1.DurableDirVolumeSource{},
|
|
},
|
|
},
|
|
{
|
|
Name: "unmounted-vol",
|
|
VolumeSource: atev1alpha1.VolumeSource{
|
|
ExternalVolumeTemplate: &atev1alpha1.ExternalVolumeTemplate{
|
|
StorageClassName: "pd-standard",
|
|
},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
}
|
|
|
|
actor := &ateapipb.Actor{
|
|
Metadata: &ateapipb.ResourceMetadata{
|
|
Atespace: "space-abc",
|
|
Name: "actor-123",
|
|
},
|
|
Status: &ateapipb.ActorStatus{
|
|
ActorVolumes: []*ateapipb.ExternalVolume{
|
|
{
|
|
VolumeName: "vol-1",
|
|
StorageVolumeId: "vol-gce-pd-123",
|
|
VolumeType: "pd-standard",
|
|
VolumeContext: map[string]string{"foo": "bar"},
|
|
},
|
|
},
|
|
},
|
|
}
|
|
|
|
workloadSpec := &ateletpb.WorkloadSpec{}
|
|
if err := appendExternalVolumes(workloadSpec, template, actor); err != nil {
|
|
t.Fatalf("appendExternalVolumes unexpected error: %v", err)
|
|
}
|
|
|
|
want := &ateletpb.WorkloadSpec{
|
|
Volumes: []*ateletpb.Volume{
|
|
{
|
|
Name: "vol-1",
|
|
Source: &ateletpb.Volume_External{
|
|
External: &ateletpb.ExternalVolumeSource{
|
|
StorageVolumeId: "vol-gce-pd-123",
|
|
VolumeType: "pd-standard",
|
|
VolumeContext: map[string]string{"foo": "bar"},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
}
|
|
|
|
if diff := cmp.Diff(want, workloadSpec, protocmp.Transform()); diff != "" {
|
|
t.Errorf("appendExternalVolumes mismatch (-want +got):\n%s", diff)
|
|
}
|
|
|
|
// Test missing mounted volume returns an error
|
|
missingActor := &ateapipb.Actor{
|
|
Metadata: &ateapipb.ResourceMetadata{
|
|
Atespace: "space-abc",
|
|
Name: "actor-123",
|
|
},
|
|
Status: &ateapipb.ActorStatus{ActorVolumes: []*ateapipb.ExternalVolume{}},
|
|
}
|
|
if err := appendExternalVolumes(&ateletpb.WorkloadSpec{}, template, missingActor); err == nil {
|
|
t.Errorf("appendExternalVolumes expected error for missing volume, got nil")
|
|
}
|
|
}
|
|
|
|
func TestWorkloadSpecFromActorTemplatePropagatesSecurityContext(t *testing.T) {
|
|
got, err := workloadSpecFromActorTemplate(&atev1alpha1.ActorTemplate{
|
|
ObjectMeta: metav1.ObjectMeta{Name: "tmpl-caps", Namespace: "agent-ns"},
|
|
Spec: atev1alpha1.ActorTemplateSpec{
|
|
Containers: []atev1alpha1.Container{
|
|
{
|
|
Name: "adjusted",
|
|
Image: "main",
|
|
SecurityContext: &atev1alpha1.SecurityContext{
|
|
Capabilities: &atev1alpha1.Capabilities{
|
|
Add: []atev1alpha1.Capability{"NET_ADMIN"},
|
|
Drop: []atev1alpha1.Capability{"ALL"},
|
|
},
|
|
},
|
|
},
|
|
{
|
|
Name: "unset",
|
|
Image: "side",
|
|
},
|
|
{
|
|
// An empty capabilities block asks for no adjustment, so
|
|
// nothing is put on the wire for it.
|
|
Name: "empty",
|
|
Image: "third",
|
|
SecurityContext: &atev1alpha1.SecurityContext{Capabilities: &atev1alpha1.Capabilities{}},
|
|
},
|
|
},
|
|
},
|
|
}, nil)
|
|
if err != nil {
|
|
t.Fatalf("workloadSpecFromActorTemplate failed: %v", err)
|
|
}
|
|
|
|
want := &ateletpb.WorkloadSpec{
|
|
Containers: []*ateletpb.Container{
|
|
{
|
|
Name: "adjusted",
|
|
Image: "main",
|
|
SecurityContext: &ateletpb.SecurityContext{
|
|
Capabilities: &ateletpb.Capabilities{
|
|
Add: []string{"NET_ADMIN"},
|
|
Drop: []string{"ALL"},
|
|
},
|
|
},
|
|
},
|
|
{
|
|
Name: "unset",
|
|
Image: "side",
|
|
},
|
|
{
|
|
Name: "empty",
|
|
Image: "third",
|
|
},
|
|
},
|
|
}
|
|
if diff := cmp.Diff(want, got, protocmp.Transform()); diff != "" {
|
|
t.Errorf("WorkloadSpec mismatch (-want +got):\n%s", diff)
|
|
}
|
|
}
|
|
|
|
func TestToAteletResources(t *testing.T) {
|
|
tests := []struct {
|
|
name string
|
|
in *atev1alpha1.ContainerResources
|
|
want *ateletpb.ResourceLimits
|
|
}{{
|
|
name: "nil resources",
|
|
in: nil,
|
|
want: nil,
|
|
}, {
|
|
name: "empty limits",
|
|
in: &atev1alpha1.ContainerResources{},
|
|
want: nil,
|
|
}, {
|
|
name: "memory only",
|
|
in: &atev1alpha1.ContainerResources{Limits: atev1alpha1.ContainerResourceList{
|
|
corev1.ResourceMemory: resource.MustParse("256Mi"),
|
|
}},
|
|
want: &ateletpb.ResourceLimits{MemoryBytes: 268435456},
|
|
}, {
|
|
name: "cpu only",
|
|
in: &atev1alpha1.ContainerResources{Limits: atev1alpha1.ContainerResourceList{
|
|
corev1.ResourceCPU: resource.MustParse("200m"),
|
|
}},
|
|
want: &ateletpb.ResourceLimits{CpuMillis: 200},
|
|
}, {
|
|
name: "both",
|
|
in: &atev1alpha1.ContainerResources{Limits: atev1alpha1.ContainerResourceList{
|
|
corev1.ResourceMemory: resource.MustParse("1Gi"),
|
|
corev1.ResourceCPU: resource.MustParse("1"),
|
|
}},
|
|
want: &ateletpb.ResourceLimits{MemoryBytes: 1073741824, CpuMillis: 1000},
|
|
}}
|
|
|
|
for _, tc := range tests {
|
|
t.Run(tc.name, func(t *testing.T) {
|
|
got := toAteletResources(tc.in)
|
|
if tc.want == nil {
|
|
if got != nil {
|
|
t.Fatalf("toAteletResources() = %v, want nil", got)
|
|
}
|
|
return
|
|
}
|
|
if got == nil {
|
|
t.Fatalf("toAteletResources() = nil, want %v", tc.want)
|
|
}
|
|
if got.GetMemoryBytes() != tc.want.GetMemoryBytes() {
|
|
t.Errorf("memory_bytes = %d, want %d", got.GetMemoryBytes(), tc.want.GetMemoryBytes())
|
|
}
|
|
if got.GetCpuMillis() != tc.want.GetCpuMillis() {
|
|
t.Errorf("cpu_millis = %d, want %d", got.GetCpuMillis(), tc.want.GetCpuMillis())
|
|
}
|
|
})
|
|
}
|
|
}
|
|
|
|
// The limits a template declares must reach the workload spec atelet receives.
|
|
// Without this, removing the Resources field from the ateletpb.Container
|
|
// literal leaves every test in the repository green while limits never leave
|
|
// ate-api-server.
|
|
func TestWorkloadSpecFromActorTemplatePropagatesResources(t *testing.T) {
|
|
got, err := workloadSpecFromActorTemplate(&atev1alpha1.ActorTemplate{
|
|
ObjectMeta: metav1.ObjectMeta{Name: "tmpl-limits", Namespace: "agent-ns"},
|
|
Spec: atev1alpha1.ActorTemplateSpec{
|
|
SandboxClass: atev1alpha1.SandboxClassMicroVM,
|
|
Containers: []atev1alpha1.Container{
|
|
{
|
|
Name: "limited",
|
|
Image: "main",
|
|
Resources: &atev1alpha1.ContainerResources{
|
|
Limits: atev1alpha1.ContainerResourceList{
|
|
corev1.ResourceMemory: resource.MustParse("256Mi"),
|
|
corev1.ResourceCPU: resource.MustParse("200m"),
|
|
},
|
|
},
|
|
},
|
|
{Name: "unlimited", Image: "main"},
|
|
},
|
|
},
|
|
}, nil)
|
|
if err != nil {
|
|
t.Fatalf("workloadSpecFromActorTemplate failed: %v", err)
|
|
}
|
|
if len(got.GetContainers()) != 2 {
|
|
t.Fatalf("containers = %d, want 2", len(got.GetContainers()))
|
|
}
|
|
|
|
limited := got.GetContainers()[0].GetResources()
|
|
if limited == nil {
|
|
t.Fatal("limited container Resources = nil, want the declared limits")
|
|
}
|
|
if limited.GetMemoryBytes() != 268435456 {
|
|
t.Errorf("memory_bytes = %d, want 268435456", limited.GetMemoryBytes())
|
|
}
|
|
if limited.GetCpuMillis() != 200 {
|
|
t.Errorf("cpu_millis = %d, want 200", limited.GetCpuMillis())
|
|
}
|
|
if r := got.GetContainers()[1].GetResources(); r != nil {
|
|
t.Errorf("unlimited container Resources = %v, want nil", r)
|
|
}
|
|
}
|