mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-11-10 10:13:50 +08:00
Part of #932 (PR 1 of 3). Adds the user-declarable trustBundle data source for SystemInfo volumes (#802) and the end-to-end proof that the projected anchors work against the MITM egress gateway. Live refresh for running actors (PR 2) and auto-injection (PR 3) come separately. What this adds A SystemInfo volume data source that projects the trust anchors of a named trust bundle to a PEM file: volumes: - name: trust systemInfo: dataSources: - trustBundle: name: egress-mitm.ate.dev path: egress-ca.pem Inspired by the Kubernetes clusterTrustBundle projected volume source, but source-neutral: the template names a bundle; where it's fetched from is a deployment concern, not part of the API. Design points - Resolution lives on the node. The wire carries only {name, path}; atelet resolves the name at write time through an informer-backed lister on ClusterTrustBundles and writes the sanitized PEM with the temp+rename discipline from #803 (find-paths safe). Contents refresh on every Run/Restore. ateapi is not involved, per review discussion — the same informer is what live refresh (PR 2) will hang off. - Allowlist in atelet, not the CRD schema. Today only egress-mitm.ate.dev (the egress gateway CA bundle, #823), mapped to the ClusterTrustBundle that atecontroller's EgressMITMTrustReconciler (#946) derives from the egress-mitm-ca-pool Secret. The signer-linked object name stays a backend detail; the future backend registry (#932) widens the allowlist without an API change. - The watch is scoped to the one backing object via a metadata.name field selector — this informer runs on every node, so an unfiltered watch would fan every ClusterTrustBundle in the cluster out to every atelet. RBAC can't express this (resourceNames doesn't apply to list/watch), so the field selector is the enforcement point. get/list/watch on clustertrustbundles moves to the atelet ClusterRole. - No availability probe. The informer registers unconditionally; a cluster that doesn't serve the feature-gated certificates.k8s.io/v1beta1 blocks atelet startup at cache sync, with the reflector errors naming the missing API (hack/create-kind-cluster.sh enables the gate). - Fail-closed. Unknown names, missing bundles, and unusable bundles fail actor start naming the bundle — an actor that declared a trust bundle must not start without one. - Kubelet-parity sanitization (internal/pemutil): CERTIFICATE blocks only, deduplicated, headers stripped, and anchors deliberately shuffled so consumers can't grow a dependence on order. - Schema note: dataSources MaxItems tightened 32→8 while adding the trustBundle member. Vacuous in practice (the old schema couldn't admit more than one entry), but flagged since it's ratchet-shaped. E2E — delivery and consumption Delivery (identity suite, both sandbox classes): provisions the egress-mitm-ca-pool Secret and drives the real #946 reconciler (writing the bundle directly isn't possible — the reconciler reverts hand-edits), asserts the projected file byte-exact, then rotates the pool across a suspend/resume to prove refresh-on-restore. Since the probe fixture is shared and fail-closed, e2e.DeployProbe itself ensures the bundle exists for whatever suite deploys it. Consumption (new egressmitm suite, both sandbox classes): deploys the sdsmint (MITM) egress gateway and proves an actor completes a TLS handshake with the gateway's per-SNI minted leaf using ONLY the projected anchors — plus a system-roots negative control that must fail. The pair is unambiguous in both directions: the positive can't pass under passthrough (the bundle holds no public CAs), and the negative can't fail under passthrough. CI: two steps appended to the existing e2e job after the standard lanes (the gateway swap is cluster-wide and breaks passthrough assumptions): --deploy-atenet --experimental-use-sdsmint redeploys only the atenet components, then the egressmitm suite runs once per sandbox class. Flake mitigation: the probe fixture pool drops from 3 workers to 2. Each suite deploys its own copy and drives one actor at a time, so the third worker per copy was idle memory multiplied across suites on the one-node CI cluster — pressure that has been killing sandboxes mid-test (runsc: signal: killed, a vanished ateom socket) on this PR and on main's identity suite. This reduces the pressure; right-sizing e2e concurrency or worker-pod QoS cluster-wide is follow-up material. Not in this PR - Live refresh for running actors (#932 PR 2) — until then, a running actor's file is the bundle as of its last Run/Restore, and correctness rests on overlap rotation by the bundle publisher. - Auto-injection of the egress trust volume (#932 PR 3). - Configurable backend registry (#932) — the allowlist is the seam it will replace.
246 lines
7.8 KiB
Go
246 lines
7.8 KiB
Go
// Copyright 2026 Google LLC
|
|
//
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
package controlapi
|
|
|
|
import (
|
|
"fmt"
|
|
|
|
"github.com/agent-substrate/substrate/internal/proto/ateletpb"
|
|
atev1alpha1 "github.com/agent-substrate/substrate/pkg/api/v1alpha1"
|
|
"github.com/agent-substrate/substrate/pkg/proto/ateapipb"
|
|
)
|
|
|
|
// workloadSpecFromActorTemplate builds a WorkloadSpec from the template;
|
|
// container env is copied verbatim.
|
|
func workloadSpecFromActorTemplate(actorTemplate *atev1alpha1.ActorTemplate, actor *ateapipb.Actor) (*ateletpb.WorkloadSpec, error) {
|
|
workloadSpec := &ateletpb.WorkloadSpec{}
|
|
|
|
// Convert volumes to atelet's representation. ActorTemplate validation has
|
|
// already ensured that only one source is set.
|
|
for _, vol := range actorTemplate.Spec.Volumes {
|
|
switch {
|
|
case vol.VolumeSource.DurableDir != nil:
|
|
workloadSpec.Volumes = append(workloadSpec.Volumes, &ateletpb.Volume{
|
|
Name: vol.Name,
|
|
Source: &ateletpb.Volume_DurableDir{
|
|
DurableDir: &ateletpb.DurableDirVolume{},
|
|
},
|
|
})
|
|
|
|
case vol.VolumeSource.SystemInfo != nil:
|
|
ateletSystemInfo := &ateletpb.SystemInfoVolume{}
|
|
for _, dataSource := range vol.VolumeSource.SystemInfo.DataSources {
|
|
switch {
|
|
case dataSource.ActorMetadata != nil:
|
|
actorMetadata := &ateletpb.ActorMetadataDataSource{}
|
|
for _, item := range dataSource.ActorMetadata.Items {
|
|
actorMetadata.Items = append(actorMetadata.Items, &ateletpb.ActorMetadataItem{
|
|
Field: toAteletActorMetadataField(item.Field),
|
|
Path: item.Path,
|
|
})
|
|
}
|
|
ateletSystemInfo.DataSources = append(ateletSystemInfo.DataSources, &ateletpb.SystemInfoDataSource{
|
|
DataSource: &ateletpb.SystemInfoDataSource_ActorMetadata{
|
|
ActorMetadata: actorMetadata,
|
|
},
|
|
})
|
|
case dataSource.TrustBundle != nil:
|
|
// atelet resolves named trustBundles against its allowlist
|
|
// and ClusterTrustBundle informer at write time
|
|
ateletSystemInfo.DataSources = append(ateletSystemInfo.DataSources, &ateletpb.SystemInfoDataSource{
|
|
DataSource: &ateletpb.SystemInfoDataSource_TrustBundle{
|
|
TrustBundle: &ateletpb.TrustBundleDataSource{
|
|
Name: dataSource.TrustBundle.Name,
|
|
Path: dataSource.TrustBundle.Path,
|
|
},
|
|
},
|
|
})
|
|
default:
|
|
continue // Drop unrecognized data sources
|
|
}
|
|
}
|
|
workloadSpec.Volumes = append(workloadSpec.Volumes, &ateletpb.Volume{
|
|
Name: vol.Name,
|
|
Source: &ateletpb.Volume_SystemInfo{
|
|
SystemInfo: ateletSystemInfo,
|
|
},
|
|
})
|
|
|
|
case vol.VolumeSource.Image != nil:
|
|
workloadSpec.Volumes = append(workloadSpec.Volumes, &ateletpb.Volume{
|
|
Name: vol.Name,
|
|
Source: &ateletpb.Volume_Image{
|
|
Image: &ateletpb.ImageVolumeSource{
|
|
Reference: vol.VolumeSource.Image.Reference,
|
|
},
|
|
},
|
|
})
|
|
|
|
default:
|
|
continue // Drop unrecognized volumes.
|
|
}
|
|
}
|
|
|
|
// TODO: order may be important for nested mounts. Also need to think about
|
|
// nested mount support in general.
|
|
if err := appendExternalVolumes(workloadSpec, actorTemplate, actor); err != nil {
|
|
return nil, err
|
|
}
|
|
|
|
for _, ctr := range actorTemplate.Spec.Containers {
|
|
ateletCtr := &ateletpb.Container{
|
|
Name: ctr.Name,
|
|
Image: ctr.Image,
|
|
Command: ctr.Command,
|
|
Args: ctr.Args,
|
|
Readyz: toAteletReadyz(ctr.Readyz),
|
|
SecurityContext: toAteletSecurityContext(ctr.SecurityContext),
|
|
}
|
|
for _, env := range ctr.Env {
|
|
ateletCtr.Env = append(ateletCtr.Env, &ateletpb.EnvEntry{
|
|
Name: env.Name,
|
|
Value: env.Value,
|
|
})
|
|
}
|
|
for _, mount := range ctr.VolumeMounts {
|
|
ateletCtr.VolumeMounts = append(ateletCtr.VolumeMounts, &ateletpb.VolumeMount{
|
|
Name: mount.Name,
|
|
MountPath: mount.MountPath,
|
|
})
|
|
}
|
|
workloadSpec.Containers = append(workloadSpec.Containers, ateletCtr)
|
|
}
|
|
|
|
return workloadSpec, nil
|
|
}
|
|
|
|
// appendExternalVolumes maps template external volumes to resolved actor volumes and appends them to workloadSpec
|
|
// if they are referenced in container volumeMounts.
|
|
func appendExternalVolumes(workloadSpec *ateletpb.WorkloadSpec, template *atev1alpha1.ActorTemplate, actor *ateapipb.Actor) error {
|
|
if template == nil {
|
|
return nil
|
|
}
|
|
for _, vol := range template.Spec.Volumes {
|
|
if vol.ExternalVolumeTemplate != nil {
|
|
if !isVolumeMounted(vol.Name, template) {
|
|
continue
|
|
}
|
|
if actor == nil {
|
|
return fmt.Errorf("actor is required when externalVolumeTemplate is present")
|
|
}
|
|
|
|
var storageVolID string
|
|
var volType string
|
|
var volCtx map[string]string
|
|
for _, dbVol := range actor.GetStatus().GetActorVolumes() {
|
|
if dbVol.GetVolumeName() == vol.Name {
|
|
storageVolID = dbVol.GetStorageVolumeId()
|
|
volType = dbVol.GetVolumeType()
|
|
volCtx = dbVol.GetVolumeContext()
|
|
break
|
|
}
|
|
}
|
|
if storageVolID == "" {
|
|
return fmt.Errorf("volume %s not found for actor %s", vol.Name, actor.GetMetadata().GetName())
|
|
}
|
|
workloadSpec.Volumes = append(workloadSpec.Volumes, &ateletpb.Volume{
|
|
Name: vol.Name,
|
|
Source: &ateletpb.Volume_External{
|
|
External: &ateletpb.ExternalVolumeSource{
|
|
StorageVolumeId: storageVolID,
|
|
VolumeType: volType,
|
|
VolumeContext: volCtx,
|
|
},
|
|
},
|
|
})
|
|
}
|
|
}
|
|
return nil
|
|
}
|
|
|
|
func isVolumeMounted(volumeName string, template *atev1alpha1.ActorTemplate) bool {
|
|
for _, ctr := range template.Spec.Containers {
|
|
for _, mount := range ctr.VolumeMounts {
|
|
if mount.Name == volumeName {
|
|
return true
|
|
}
|
|
}
|
|
}
|
|
return false
|
|
}
|
|
|
|
// toAteletReadyz projects the CRD readyz field onto the ateletpb wire type.
|
|
// Returns nil when the source is nil so containers without a probe stay
|
|
// unchanged on the wire.
|
|
// toAteletActorMetadataField projects the CRD field selector onto the atelet
|
|
// wire enum. Unknown values map to UNSPECIFIED, which atelet skips; CRD enum
|
|
// validation makes that unreachable for stored templates.
|
|
func toAteletActorMetadataField(in atev1alpha1.ActorMetadataField) ateletpb.ActorMetadataField {
|
|
switch in {
|
|
case atev1alpha1.ActorMetadataFieldName:
|
|
return ateletpb.ActorMetadataField_ACTOR_METADATA_FIELD_NAME
|
|
case atev1alpha1.ActorMetadataFieldAtespace:
|
|
return ateletpb.ActorMetadataField_ACTOR_METADATA_FIELD_ATESPACE
|
|
case atev1alpha1.ActorMetadataFieldUID:
|
|
return ateletpb.ActorMetadataField_ACTOR_METADATA_FIELD_UID
|
|
default:
|
|
return ateletpb.ActorMetadataField_ACTOR_METADATA_FIELD_UNSPECIFIED
|
|
}
|
|
}
|
|
|
|
func toAteletReadyz(in *atev1alpha1.ContainerReadyz) *ateletpb.Readyz {
|
|
if in == nil {
|
|
return nil
|
|
}
|
|
out := &ateletpb.Readyz{TimeoutSeconds: in.TimeoutSeconds}
|
|
if in.HTTPGet != nil {
|
|
out.HttpGet = &ateletpb.HTTPGetAction{
|
|
Path: in.HTTPGet.Path,
|
|
Port: in.HTTPGet.Port,
|
|
}
|
|
}
|
|
return out
|
|
}
|
|
|
|
// toAteletSecurityContext projects the CRD securityContext onto the ateletpb
|
|
// wire type. Returns nil when the source is nil or carries nothing, so
|
|
// containers that set no security settings stay unchanged on the wire.
|
|
func toAteletSecurityContext(in *atev1alpha1.SecurityContext) *ateletpb.SecurityContext {
|
|
if in == nil || in.Capabilities == nil {
|
|
return nil
|
|
}
|
|
caps := in.Capabilities
|
|
if len(caps.Add) == 0 && len(caps.Drop) == 0 {
|
|
return nil
|
|
}
|
|
return &ateletpb.SecurityContext{
|
|
Capabilities: &ateletpb.Capabilities{
|
|
Add: capabilityNames(caps.Add),
|
|
Drop: capabilityNames(caps.Drop),
|
|
},
|
|
}
|
|
}
|
|
|
|
func capabilityNames(in []atev1alpha1.Capability) []string {
|
|
if len(in) == 0 {
|
|
return nil
|
|
}
|
|
out := make([]string, 0, len(in))
|
|
for _, c := range in {
|
|
out = append(out, string(c))
|
|
}
|
|
return out
|
|
}
|