Files
substrate/cmd/atelet/trustbundle_test.go
Max Thompson f3845f3ce6 atelet: live refresh of projected trust bundles for running actors (#1231)
Part of #932 (PR 2 of 3). PR 1 (#941) added the `trustBundle` SystemInfo
data source, resolved on the node at Run/Restore. This PR keeps those
projections current while the actor runs.

## Live refresh

A `systemInfoVolumeRefresher` in atelet, modeled on kubelet's projected
volumes: `collectData` is the one place that builds a volume's complete
contents from its spec, `write` applies them, and every lifecycle point
uses the pair. Run/Restore registers the actor's volumes, which writes
them fail-closed before the sandbox boots; ClusterTrustBundle events
rewrite them while it runs; Checkpoint, Terminate, and failed starts
deregister. No API, proto, or RBAC changes: the wire still carries only
`{name, path}`.

- Informer events only enqueue bundle names; a single run loop writes.
Failed writes requeue with backoff through a rate-limited workqueue;
resolution failures keep last-good contents and wait for the bundle's
next event. Resync (24h) is only a guard against missed watch events.
- Change detection hashes the raw backing contents, not the projected
output (sanitization shuffles). Files are replaced by temp-and-rename at
stable paths and byte-identical files are left untouched: a needless
rewrite replaces an inode that suspended guests re-bind on resume.
- Nothing is persisted. Registrations are in memory; after an atelet
restart, a running actor's files keep their last-written contents until
its next Run/Restore rewrites them from current cluster state.

## virtiofsd: `--migration-on-error=guest-error`

CI confirmed the hazard (3 of 3 micro-VM runs): a rotation renames over
a file whose inode the guest still references (a dcache reference is
enough, no held fd), the actor suspends before re-reading, and
find-paths serializes an inode with no findable path. Under the default
`abort`, the destination virtiofsd rejects the device state at
vm.restore: suspend succeeds, every restore fails, and the actor is
permanently stuck. Full analysis in the PR comments.

With `guest-error` the restore succeeds and only the stale reference is
faulty: EIO on access until a fresh lookup at the stable path heals it.
That matches the documented reader contract (the platform rewrites the
file, applications re-read it) and also covers guest-created
`O_TMPFILE`s held across suspend, which already fail restores under
`abort`. The flag is backend-process configuration, so existing
snapshots (template goldens included) restore unchanged. Trade-off: a
share-reconstruction bug now surfaces as post-resume EIO and a readyz
failure instead of a loud restore failure; virtiofsd names the faulty
inodes in the worker pod log.

## E2E

The identity suite covers live refresh on both sandbox classes: rotate
the pool under two running actors and wait for both to observe the new
bundle live; rotate/suspend/resume without waiting for propagation, so
the resume must deliver rotated contents whether the rewrite landed
before or after the guest went down (the exact `abort` repro); assert a
sibling that never cycled also got the rotation live. The probe fixture
now keeps its namespace when a test fails so the worker pod logs
survive, and the cloud-hypervisor client includes the vm.restore
response body in errors.

## Not in this PR

Auto-injected egress trust volume (#932 PR 3) and the configurable
backend registry (tracked in #932).
2026-09-10 16:13:55 -07:00

110 lines
3.9 KiB
Go

// Copyright 2026 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"crypto/x509"
"crypto/x509/pkix"
"encoding/pem"
"math/big"
"strings"
"testing"
"time"
certsv1beta1 "k8s.io/api/certificates/v1beta1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
certlisters "k8s.io/client-go/listers/certificates/v1beta1"
"k8s.io/client-go/tools/cache"
)
// testCertPEM mints a throwaway self-signed certificate, PEM-encoded.
func testCertPEM(t *testing.T) []byte {
t.Helper()
key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
if err != nil {
t.Fatal(err)
}
der, err := x509.CreateCertificate(rand.Reader, &x509.Certificate{
SerialNumber: big.NewInt(1),
Subject: pkix.Name{CommonName: "test"},
NotBefore: time.Now(),
NotAfter: time.Now().Add(time.Hour),
}, &x509.Certificate{SerialNumber: big.NewInt(1)}, &key.PublicKey, key)
if err != nil {
t.Fatal(err)
}
return pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
}
func ctbLister(t *testing.T, bundles ...*certsv1beta1.ClusterTrustBundle) certlisters.ClusterTrustBundleLister {
t.Helper()
indexer := cache.NewIndexer(cache.MetaNamespaceKeyFunc, cache.Indexers{})
for _, b := range bundles {
if err := indexer.Add(b); err != nil {
t.Fatal(err)
}
}
return certlisters.NewClusterTrustBundleLister(indexer)
}
// egressTrustBundleObjectName is the backing ClusterTrustBundle the allowlist
// maps EgressTrustBundleName to (named by atecontroller's reconciler).
const egressTrustBundleObjectName = "egress-mitm.ate.dev:mitm:primary-bundle"
func TestRawTrustBundle(t *testing.T) {
certPEM := testCertPEM(t)
t.Run("resolves the allowlisted name through the mapped object, unsanitized", func(t *testing.T) {
raw := "garbage\n" + string(certPEM)
lister := ctbLister(t, &certsv1beta1.ClusterTrustBundle{
ObjectMeta: metav1.ObjectMeta{Name: egressTrustBundleObjectName},
Spec: certsv1beta1.ClusterTrustBundleSpec{TrustBundle: raw},
})
objectName, got, err := rawTrustBundle(lister, EgressTrustBundleName)
if err != nil {
t.Fatalf("rawTrustBundle: %v", err)
}
if objectName != egressTrustBundleObjectName {
t.Errorf("objectName = %q, want %q", objectName, egressTrustBundleObjectName)
}
if got != raw {
t.Errorf("raw = %q, want the backing contents verbatim", got)
}
})
t.Run("unsupported bundle name fails naming it and the allowlist", func(t *testing.T) {
// The lister has the bundle; the allowlist must still reject it —
// supported names are a substrate decision, not a cluster lookup.
lister := ctbLister(t, &certsv1beta1.ClusterTrustBundle{
ObjectMeta: metav1.ObjectMeta{Name: "my-own-bundle"},
Spec: certsv1beta1.ClusterTrustBundleSpec{TrustBundle: string(certPEM)},
})
_, _, err := rawTrustBundle(lister, "my-own-bundle")
if err == nil || !strings.Contains(err.Error(), `"my-own-bundle"`) || !strings.Contains(err.Error(), "not supported") || !strings.Contains(err.Error(), EgressTrustBundleName) {
t.Errorf("error = %v, want unsupported-name error listing the allowlist", err)
}
})
t.Run("missing bundle fails naming it", func(t *testing.T) {
_, _, err := rawTrustBundle(ctbLister(t), EgressTrustBundleName)
if err == nil || !strings.Contains(err.Error(), egressTrustBundleObjectName) || !strings.Contains(err.Error(), "not found") {
t.Errorf("error = %v, want not-found naming the backing object", err)
}
})
}