Files
substrate/cmd/ateapi/internal/controlapi/workload_spec_test.go
T
Max Thompson 074fd1c12b Add trustBundle as a SystemInfo volume data source (#941)
Part of #932 (PR 1 of 3). Adds the user-declarable trustBundle data
source for SystemInfo volumes (#802) and the end-to-end proof that the
projected anchors work against the MITM egress gateway. Live refresh for
running actors (PR 2) and auto-injection (PR 3) come separately.

What this adds

A SystemInfo volume data source that projects the trust anchors of a
named trust bundle to a PEM file:

volumes:
- name: trust
  systemInfo:
    dataSources:
    - trustBundle:
        name: egress-mitm.ate.dev
        path: egress-ca.pem

Inspired by the Kubernetes clusterTrustBundle projected volume source,
but source-neutral: the template names a bundle; where it's fetched from
is a deployment concern, not part of the API.

Design points

- Resolution lives on the node. The wire carries only {name, path};
atelet resolves the name at write time through an informer-backed lister
on ClusterTrustBundles and writes the sanitized PEM with the temp+rename
discipline from #803 (find-paths safe). Contents refresh on every
Run/Restore. ateapi is not involved, per review discussion — the same
informer is what live refresh (PR 2) will hang off.
- Allowlist in atelet, not the CRD schema. Today only
egress-mitm.ate.dev (the egress gateway CA bundle, #823), mapped to the
ClusterTrustBundle that atecontroller's EgressMITMTrustReconciler (#946)
derives from the egress-mitm-ca-pool Secret. The signer-linked object
name stays a backend detail; the future backend registry (#932) widens
the allowlist without an API change.
- The watch is scoped to the one backing object via a metadata.name
field selector — this informer runs on every node, so an unfiltered
watch would fan every ClusterTrustBundle in the cluster out to every
atelet. RBAC can't express this (resourceNames doesn't apply to
list/watch), so the field selector is the enforcement point.
get/list/watch on clustertrustbundles moves to the atelet ClusterRole.
- No availability probe. The informer registers unconditionally; a
cluster that doesn't serve the feature-gated certificates.k8s.io/v1beta1
blocks atelet startup at cache sync, with the reflector errors naming
the missing API (hack/create-kind-cluster.sh enables the gate).
- Fail-closed. Unknown names, missing bundles, and unusable bundles fail
actor start naming the bundle — an actor that declared a trust bundle
must not start without one.
- Kubelet-parity sanitization (internal/pemutil): CERTIFICATE blocks
only, deduplicated, headers stripped, and anchors deliberately shuffled
so consumers can't grow a dependence on order.
- Schema note: dataSources MaxItems tightened 32→8 while adding the
trustBundle member. Vacuous in practice (the old schema couldn't admit
more than one entry), but flagged since it's ratchet-shaped.

E2E — delivery and consumption

Delivery (identity suite, both sandbox classes): provisions the
egress-mitm-ca-pool Secret and drives the real #946 reconciler (writing
the bundle directly isn't possible — the reconciler reverts hand-edits),
asserts the projected file byte-exact, then rotates the pool across a
suspend/resume to prove refresh-on-restore. Since the probe fixture is
shared and fail-closed, e2e.DeployProbe itself ensures the bundle exists
for whatever suite deploys it.

Consumption (new egressmitm suite, both sandbox classes): deploys the
sdsmint (MITM) egress gateway and proves an actor completes a TLS
handshake with the gateway's per-SNI minted leaf using ONLY the
projected anchors — plus a system-roots negative control that must fail.
The pair is unambiguous in both directions: the positive can't pass
under passthrough (the bundle holds no public CAs), and the negative
can't fail under passthrough.

CI: two steps appended to the existing e2e job after the standard lanes
(the gateway swap is cluster-wide and breaks passthrough assumptions):
--deploy-atenet --experimental-use-sdsmint redeploys only the atenet
components, then the egressmitm suite runs once per sandbox class.

Flake mitigation: the probe fixture pool drops from 3 workers to 2. Each
suite deploys its own copy and drives one actor at a time, so the third
worker per copy was idle memory multiplied across suites on the one-node
CI cluster — pressure that has been killing sandboxes mid-test (runsc:
signal: killed, a vanished ateom socket) on this PR and on main's
identity suite. This reduces the pressure; right-sizing e2e concurrency
or worker-pod QoS cluster-wide is follow-up material.

Not in this PR

- Live refresh for running actors (#932 PR 2) — until then, a running
actor's file is the bundle as of its last Run/Restore, and correctness
rests on overlap rotation by the bundle publisher.
- Auto-injection of the egress trust volume (#932 PR 3).
- Configurable backend registry (#932) — the allowlist is the seam it
will replace.
2026-08-21 19:53:31 -04:00

563 lines
16 KiB
Go

// Copyright 2026 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package controlapi
import (
"testing"
"github.com/agent-substrate/substrate/internal/proto/ateletpb"
atev1alpha1 "github.com/agent-substrate/substrate/pkg/api/v1alpha1"
"github.com/agent-substrate/substrate/pkg/proto/ateapipb"
"github.com/google/go-cmp/cmp"
"google.golang.org/protobuf/testing/protocmp"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
func TestWorkloadSpecFromActorTemplate(t *testing.T) {
tests := []struct {
name string
template *atev1alpha1.ActorTemplate
want *ateletpb.WorkloadSpec
}{
{
name: "converts DurableDir volume and mounts",
template: &atev1alpha1.ActorTemplate{
ObjectMeta: metav1.ObjectMeta{Name: "tmpl1", Namespace: "agent-ns"},
Spec: atev1alpha1.ActorTemplateSpec{
Volumes: []atev1alpha1.Volume{
{Name: "home", VolumeSource: atev1alpha1.VolumeSource{DurableDir: &atev1alpha1.DurableDirVolumeSource{}}},
},
Containers: []atev1alpha1.Container{
{
Name: "main",
Image: "main",
VolumeMounts: []atev1alpha1.VolumeMount{
{Name: "home", MountPath: "/home/user"},
{Name: "home", MountPath: "/workspace"},
},
},
},
},
},
want: &ateletpb.WorkloadSpec{
Volumes: []*ateletpb.Volume{
{
Name: "home",
Source: &ateletpb.Volume_DurableDir{DurableDir: &ateletpb.DurableDirVolume{}},
},
},
Containers: []*ateletpb.Container{
{
Name: "main",
Image: "main",
VolumeMounts: []*ateletpb.VolumeMount{
{Name: "home", MountPath: "/home/user"},
{Name: "home", MountPath: "/workspace"},
},
},
},
},
},
{
name: "converts SystemInfo volume with actorMetadata items",
template: &atev1alpha1.ActorTemplate{
ObjectMeta: metav1.ObjectMeta{Name: "tmpl1", Namespace: "agent-ns"},
Spec: atev1alpha1.ActorTemplateSpec{
Volumes: []atev1alpha1.Volume{
{
Name: "system-info",
VolumeSource: atev1alpha1.VolumeSource{
SystemInfo: &atev1alpha1.SystemInfoVolumeSource{
DataSources: []atev1alpha1.SystemInfoDataSource{
{ActorMetadata: &atev1alpha1.ActorMetadataDataSource{
Items: []atev1alpha1.ActorMetadataItem{
{Field: atev1alpha1.ActorMetadataFieldName, Path: "actor-name"},
{Field: atev1alpha1.ActorMetadataFieldAtespace, Path: "atespace"},
{Field: atev1alpha1.ActorMetadataFieldUID, Path: "identity/actor-uid"},
},
}},
},
},
},
},
},
Containers: []atev1alpha1.Container{
{
Name: "main",
Image: "main",
VolumeMounts: []atev1alpha1.VolumeMount{
{Name: "system-info", MountPath: "/run/ate"},
},
},
},
},
},
want: &ateletpb.WorkloadSpec{
Volumes: []*ateletpb.Volume{
{
Name: "system-info",
Source: &ateletpb.Volume_SystemInfo{
SystemInfo: &ateletpb.SystemInfoVolume{
DataSources: []*ateletpb.SystemInfoDataSource{
{DataSource: &ateletpb.SystemInfoDataSource_ActorMetadata{
ActorMetadata: &ateletpb.ActorMetadataDataSource{
Items: []*ateletpb.ActorMetadataItem{
{Field: ateletpb.ActorMetadataField_ACTOR_METADATA_FIELD_NAME, Path: "actor-name"},
{Field: ateletpb.ActorMetadataField_ACTOR_METADATA_FIELD_ATESPACE, Path: "atespace"},
{Field: ateletpb.ActorMetadataField_ACTOR_METADATA_FIELD_UID, Path: "identity/actor-uid"},
},
},
}},
},
},
},
},
},
Containers: []*ateletpb.Container{
{
Name: "main",
Image: "main",
VolumeMounts: []*ateletpb.VolumeMount{
{Name: "system-info", MountPath: "/run/ate"},
},
},
},
},
},
{
name: "converts Image volume and mounts",
template: &atev1alpha1.ActorTemplate{
ObjectMeta: metav1.ObjectMeta{Name: "tmpl1", Namespace: "agent-ns"},
Spec: atev1alpha1.ActorTemplateSpec{
Volumes: []atev1alpha1.Volume{
{Name: "home", VolumeSource: atev1alpha1.VolumeSource{DurableDir: &atev1alpha1.DurableDirVolumeSource{}}},
{Name: "agent", VolumeSource: atev1alpha1.VolumeSource{Image: &atev1alpha1.ImageVolumeSource{Reference: "example.com/agent@sha256:abc"}}},
},
Containers: []atev1alpha1.Container{
{
Name: "main",
Image: "main",
VolumeMounts: []atev1alpha1.VolumeMount{
{Name: "home", MountPath: "/home/user"},
{Name: "agent", MountPath: "/ate"},
},
},
},
},
},
want: &ateletpb.WorkloadSpec{
Volumes: []*ateletpb.Volume{
{
Name: "home",
Source: &ateletpb.Volume_DurableDir{DurableDir: &ateletpb.DurableDirVolume{}},
},
{
Name: "agent",
Source: &ateletpb.Volume_Image{Image: &ateletpb.ImageVolumeSource{Reference: "example.com/agent@sha256:abc"}},
},
},
Containers: []*ateletpb.Container{
{
Name: "main",
Image: "main",
VolumeMounts: []*ateletpb.VolumeMount{
{Name: "home", MountPath: "/home/user"},
{Name: "agent", MountPath: "/ate"},
},
},
},
},
},
{
name: "converts trustBundle data sources carrying the name for node-side resolution",
template: &atev1alpha1.ActorTemplate{
ObjectMeta: metav1.ObjectMeta{Name: "tmpl1", Namespace: "agent-ns"},
Spec: atev1alpha1.ActorTemplateSpec{
Volumes: []atev1alpha1.Volume{
{
Name: "system-info",
VolumeSource: atev1alpha1.VolumeSource{
SystemInfo: &atev1alpha1.SystemInfoVolumeSource{
DataSources: []atev1alpha1.SystemInfoDataSource{
{TrustBundle: &atev1alpha1.TrustBundleDataSource{Name: "egress-trust", Path: "trust/ca.pem"}},
},
},
},
},
},
Containers: []atev1alpha1.Container{
{
Name: "main",
Image: "main",
VolumeMounts: []atev1alpha1.VolumeMount{
{Name: "system-info", MountPath: "/run/substrate/certs"},
},
},
},
},
},
// The NAME crosses the wire; atelet resolves it against its
// allowlist and ClusterTrustBundle informer at write time.
want: &ateletpb.WorkloadSpec{
Volumes: []*ateletpb.Volume{
{
Name: "system-info",
Source: &ateletpb.Volume_SystemInfo{
SystemInfo: &ateletpb.SystemInfoVolume{
DataSources: []*ateletpb.SystemInfoDataSource{
{DataSource: &ateletpb.SystemInfoDataSource_TrustBundle{
TrustBundle: &ateletpb.TrustBundleDataSource{Name: "egress-trust", Path: "trust/ca.pem"},
}},
},
},
},
},
},
Containers: []*ateletpb.Container{
{
Name: "main",
Image: "main",
VolumeMounts: []*ateletpb.VolumeMount{
{Name: "system-info", MountPath: "/run/substrate/certs"},
},
},
},
},
},
{
name: "skips non-DurableDir volumes",
template: &atev1alpha1.ActorTemplate{
ObjectMeta: metav1.ObjectMeta{Name: "tmpl1", Namespace: "agent-ns"},
Spec: atev1alpha1.ActorTemplateSpec{
Volumes: []atev1alpha1.Volume{
{Name: "unsupported", VolumeSource: atev1alpha1.VolumeSource{}},
{Name: "home", VolumeSource: atev1alpha1.VolumeSource{DurableDir: &atev1alpha1.DurableDirVolumeSource{}}},
},
Containers: []atev1alpha1.Container{
{
Name: "main",
Image: "main",
VolumeMounts: []atev1alpha1.VolumeMount{
{Name: "home", MountPath: "/workspace"},
},
},
},
},
},
want: &ateletpb.WorkloadSpec{
Volumes: []*ateletpb.Volume{
{
Name: "home",
Source: &ateletpb.Volume_DurableDir{DurableDir: &ateletpb.DurableDirVolume{}},
},
},
Containers: []*ateletpb.Container{
{
Name: "main",
Image: "main",
VolumeMounts: []*ateletpb.VolumeMount{
{Name: "home", MountPath: "/workspace"},
},
},
},
},
},
{
name: "container without volume mounts has none",
template: &atev1alpha1.ActorTemplate{
ObjectMeta: metav1.ObjectMeta{Name: "tmpl1", Namespace: "agent-ns"},
Spec: atev1alpha1.ActorTemplateSpec{
Volumes: []atev1alpha1.Volume{
{Name: "home", VolumeSource: atev1alpha1.VolumeSource{DurableDir: &atev1alpha1.DurableDirVolumeSource{}}},
},
Containers: []atev1alpha1.Container{
{Name: "main", Image: "main"},
},
},
},
want: &ateletpb.WorkloadSpec{
Volumes: []*ateletpb.Volume{
{
Name: "home",
Source: &ateletpb.Volume_DurableDir{DurableDir: &ateletpb.DurableDirVolume{}},
},
},
Containers: []*ateletpb.Container{{Name: "main", Image: "main"}},
},
},
{
name: "maps literal env",
template: &atev1alpha1.ActorTemplate{
ObjectMeta: metav1.ObjectMeta{Name: "tmpl1", Namespace: "agent-ns"},
Spec: atev1alpha1.ActorTemplateSpec{
Containers: []atev1alpha1.Container{
{
Name: "main",
Image: "main",
Env: []atev1alpha1.EnvVar{
{Name: "LITERAL", Value: "plain"},
{Name: "EMPTY", Value: ""},
},
},
},
},
},
want: &ateletpb.WorkloadSpec{
Containers: []*ateletpb.Container{{
Name: "main",
Image: "main",
Env: []*ateletpb.EnvEntry{
{Name: "LITERAL", Value: "plain"},
{Name: "EMPTY", Value: ""},
},
}},
},
},
{
name: "maps command and args",
template: &atev1alpha1.ActorTemplate{
ObjectMeta: metav1.ObjectMeta{Name: "tmpl1", Namespace: "agent-ns"},
Spec: atev1alpha1.ActorTemplateSpec{
Containers: []atev1alpha1.Container{
{
Name: "main",
Image: "main",
Command: []string{"/entrypoint"},
Args: []string{"--foo", "--bar"},
},
},
},
},
want: &ateletpb.WorkloadSpec{
Containers: []*ateletpb.Container{{
Name: "main",
Image: "main",
Command: []string{"/entrypoint"},
Args: []string{"--foo", "--bar"},
}},
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := workloadSpecFromActorTemplate(tt.template, nil)
if err != nil {
t.Fatalf("workloadSpecFromActorTemplate failed: %v", err)
}
if diff := cmp.Diff(tt.want, got, protocmp.Transform()); diff != "" {
t.Errorf("WorkloadSpec mismatch (-want +got):\n%s", diff)
}
})
}
}
func TestWorkloadSpecFromActorTemplatePropagatesReadyz(t *testing.T) {
got, err := workloadSpecFromActorTemplate(&atev1alpha1.ActorTemplate{
ObjectMeta: metav1.ObjectMeta{Name: "tmpl-readyz", Namespace: "agent-ns"},
Spec: atev1alpha1.ActorTemplateSpec{
Containers: []atev1alpha1.Container{
{
Name: "with-probe",
Image: "main",
Readyz: &atev1alpha1.ContainerReadyz{
HTTPGet: &atev1alpha1.HTTPGetAction{Path: "/health", Port: 8080},
TimeoutSeconds: 45,
},
},
{
Name: "without-probe",
Image: "side",
},
},
},
}, nil)
if err != nil {
t.Fatalf("workloadSpecFromActorTemplate failed: %v", err)
}
want := &ateletpb.WorkloadSpec{
Containers: []*ateletpb.Container{
{
Name: "with-probe",
Image: "main",
Readyz: &ateletpb.Readyz{
HttpGet: &ateletpb.HTTPGetAction{Path: "/health", Port: 8080},
TimeoutSeconds: 45,
},
},
{
Name: "without-probe",
Image: "side",
},
},
}
if diff := cmp.Diff(want, got, protocmp.Transform()); diff != "" {
t.Errorf("WorkloadSpec mismatch (-want +got):\n%s", diff)
}
}
func TestAppendExternalVolumes(t *testing.T) {
template := &atev1alpha1.ActorTemplate{
Spec: atev1alpha1.ActorTemplateSpec{
Containers: []atev1alpha1.Container{
{
Name: "main",
VolumeMounts: []atev1alpha1.VolumeMount{
{Name: "vol-1", MountPath: "/mnt/vol1"},
},
},
},
Volumes: []atev1alpha1.Volume{
{
Name: "vol-1",
VolumeSource: atev1alpha1.VolumeSource{
ExternalVolumeTemplate: &atev1alpha1.ExternalVolumeTemplate{
StorageClassName: "pd-standard",
},
},
},
{
Name: "vol-2",
VolumeSource: atev1alpha1.VolumeSource{
DurableDir: &atev1alpha1.DurableDirVolumeSource{},
},
},
{
Name: "unmounted-vol",
VolumeSource: atev1alpha1.VolumeSource{
ExternalVolumeTemplate: &atev1alpha1.ExternalVolumeTemplate{
StorageClassName: "pd-standard",
},
},
},
},
},
}
actor := &ateapipb.Actor{
Metadata: &ateapipb.ResourceMetadata{
Atespace: "space-abc",
Name: "actor-123",
},
Status: &ateapipb.ActorStatus{
ActorVolumes: []*ateapipb.ExternalVolume{
{
VolumeName: "vol-1",
StorageVolumeId: "vol-gce-pd-123",
VolumeType: "pd-standard",
VolumeContext: map[string]string{"foo": "bar"},
},
},
},
}
workloadSpec := &ateletpb.WorkloadSpec{}
if err := appendExternalVolumes(workloadSpec, template, actor); err != nil {
t.Fatalf("appendExternalVolumes unexpected error: %v", err)
}
want := &ateletpb.WorkloadSpec{
Volumes: []*ateletpb.Volume{
{
Name: "vol-1",
Source: &ateletpb.Volume_External{
External: &ateletpb.ExternalVolumeSource{
StorageVolumeId: "vol-gce-pd-123",
VolumeType: "pd-standard",
VolumeContext: map[string]string{"foo": "bar"},
},
},
},
},
}
if diff := cmp.Diff(want, workloadSpec, protocmp.Transform()); diff != "" {
t.Errorf("appendExternalVolumes mismatch (-want +got):\n%s", diff)
}
// Test missing mounted volume returns an error
missingActor := &ateapipb.Actor{
Metadata: &ateapipb.ResourceMetadata{
Atespace: "space-abc",
Name: "actor-123",
},
Status: &ateapipb.ActorStatus{ActorVolumes: []*ateapipb.ExternalVolume{}},
}
if err := appendExternalVolumes(&ateletpb.WorkloadSpec{}, template, missingActor); err == nil {
t.Errorf("appendExternalVolumes expected error for missing volume, got nil")
}
}
func TestWorkloadSpecFromActorTemplatePropagatesSecurityContext(t *testing.T) {
got, err := workloadSpecFromActorTemplate(&atev1alpha1.ActorTemplate{
ObjectMeta: metav1.ObjectMeta{Name: "tmpl-caps", Namespace: "agent-ns"},
Spec: atev1alpha1.ActorTemplateSpec{
Containers: []atev1alpha1.Container{
{
Name: "adjusted",
Image: "main",
SecurityContext: &atev1alpha1.SecurityContext{
Capabilities: &atev1alpha1.Capabilities{
Add: []atev1alpha1.Capability{"NET_ADMIN"},
Drop: []atev1alpha1.Capability{"ALL"},
},
},
},
{
Name: "unset",
Image: "side",
},
{
// An empty capabilities block asks for no adjustment, so
// nothing is put on the wire for it.
Name: "empty",
Image: "third",
SecurityContext: &atev1alpha1.SecurityContext{Capabilities: &atev1alpha1.Capabilities{}},
},
},
},
}, nil)
if err != nil {
t.Fatalf("workloadSpecFromActorTemplate failed: %v", err)
}
want := &ateletpb.WorkloadSpec{
Containers: []*ateletpb.Container{
{
Name: "adjusted",
Image: "main",
SecurityContext: &ateletpb.SecurityContext{
Capabilities: &ateletpb.Capabilities{
Add: []string{"NET_ADMIN"},
Drop: []string{"ALL"},
},
},
},
{
Name: "unset",
Image: "side",
},
{
Name: "empty",
Image: "third",
},
},
}
if diff := cmp.Diff(want, got, protocmp.Transform()); diff != "" {
t.Errorf("WorkloadSpec mismatch (-want +got):\n%s", diff)
}
}