Files
substrate/cmd/atecontroller/internal/controllers/egressmitmtrust_controller_test.go
T
Jonathan Jamroga 6f45aeb2dd Make substrate's namespace, Service names and ServiceAccount names configurable (#350)
# Description

**Substrate assumes the canonical install layout, and every deviation
fails closed.** The namespace `ate-system`, the Services `api` /
`atenet-router`, and the ServiceAccounts `atelet` / `atenet-router` are
compiled-in constants. An install in a per-developer namespace, or under
a deployment that prefixes resource names, breaks — and no failure
points at the naming.

This series makes the namespace, the Service names, and the
ServiceAccount names configurable. Every option defaults to the
canonical value. A canonical install is byte-for-byte unaffected.

| Hardcoded assumption | Failure | What it looks like instead |
|---|---|---|
| atelet's namespace in ateapi's SPIFFE check | ateapi rejects every
atelet | an mTLS handshake failure, not a naming error |
| atelet's namespace in the worker's broker check | no actor obtains a
certificate | `credential broker is not atelet` |
| NetworkPolicy's ingress namespace | the CNI drops every request to the
pool | a silent network fault |
| ateapi Service name and namespace in the client | the client cannot
authenticate | `services "api" not found`, then `invalid bearer token` |
| Resource names in `hack/install-ate.sh` and the e2e harness |
authorities land in the wrong namespace | suites die in preflight |

**A SPIFFE ID breaks on two axes.** It names a namespace and a
ServiceAccount. Both were constants, and the failure surfaces as a
rejected peer, not a missing object.

## Design notes

- **ate-controller passes the worker-side identities.** ateom already
took `--atunnel-client-identity` as a flag but relied on its default. A
new `--atunnel-broker-identity` flag alone would be inert: correct only
where the constant was already correct. The controller knows the control
plane's namespace, so it supplies both.
- **ateapi takes the whole expected identity, not a namespace.** The
dialer and `ateletauth` used the namespace only to build one string. One
place decides how atelet's identity is spelled.
- **`ateletauth` deduplicates without taking the dependency it was
avoiding.** Its constants are duplicated rather than imported so the
package does not depend on `controlapi` for three strings. `ateletdial`
would make a third copy, so the strings move to
`internal/installdefaults` instead — a leaf package of constants that
imports only the standard library, so consuming it does not reintroduce
the coupling the duplication was there to prevent.
- **The client reads environment variables, not flags.** It runs outside
the cluster. It has no downward API and nothing to discover from.
- **Namespaces come from the downward API, not flags.** Every supported
topology co-locates these components. Only Service and ServiceAccount
names, which a deployment may legitimately rename, get flags.
- **`hack/install-ate.sh` and the e2e harness are in scope.** A
relocated install cannot be bootstrapped or exercised without them. The
script refuses `ATE_NAMESPACE` or `ATE_API_SERVICE_NAME` overrides on
the manifest-applying subcommands, which would half-install:
`manifests/ate-install/` names `ate-system` and `api` literally.

## Why this is one PR

The series is stacked, not parallel. Commit 1 creates
`internal/installdefaults` and seeds seven imports; commit 2 adds
fourteen more; every later commit builds on those constants. Split into
separate PRs, each blocks on the previous merging and none reviews
independently.

The guard test cannot land first. Applied to `main` it fails on eleven
hardcoded literals across `ateletauth`, `controlapi/informer.go`,
`networkpolicy_controller.go`, both `ateom` mains, `ateclient`,
`ateletdial` and three e2e files. It is green only because the preceding
commits removed them.

A partial series still fails closed. Each commit's audit turned up more
places assuming `ate-system`, so landing commits 1-3 without 4-5 leaves
a relocated install broken later and less legibly than before.

`CONTRIBUTING.md` covers this case: when the intermediate steps are not
useful on their own, keep the change as one PR split into commits at
logical break points. Review commit by commit, and preserve the commits
on merge. If you would still rather split, the one defensible cut is by
axis — relocation (commits 1-3) then renaming (commits 4-5) with the
guard test rebased on top.

## Rebase notes (2026-09-15)

The series is rebased onto `main` at `d0d85c39`, ninety-two commits on
from the base the PR first carried:

- `main` moved actor JWT and certificate minting out of
`cmd/ateapi/internal/actoridentity` into `controlapi`, deleting the
package and dropping its inline atelet authorization check in favor of
the unified authorizer. This series previously threaded the expected
atelet identity into that check; the check no longer exists, so that
adaptation is gone. `ateletauth` survives with one caller,
`workerservice.SetWorkerCapacity`, which still takes the configured
identity.
- The same refactor replaced `BrokerConfig.ExpectedActorUID` with an
`ActorAtespace` / `ActorName` / `ActorUID` triple. `AteletSPIFFEID` is
additive and still required by `ateletdial.TLSConfig`.
- `ateletauth` (ateapi's side) and `ateletdial` (the worker's side) each
re-declared the hardcoded SPIFFE ID. Both take the expected identity as
a parameter.
- `main` deleted the atenet DNS subsystem. The series no longer touches
it, and `installdefaults` carries no DNS Service name.
- The controller passes `--atunnel-broker-identity` only when it differs
from the canonical default. An ateom old enough to predate the flag
exits on it, and `docs/upgrade.md` keeps such a pool serving during a
rolling upgrade. A relocated install gets the flag and necessarily runs
an image that accepts it.
- `main` added `internal/e2e/collector_metrics.go` (agentgateway CI
support), which addresses the router through the package-level
`routerNamespace` and `routerService` that this series replaces. Its two
call sites become `SystemNamespace()` and
`ResourceName("atenet-router")`, matching `router_client.go` and
`statusz.go`. A reviewer diffing against the older base sees those call
sites move; nothing else in that file changes.
- `main` added `cmd/credential-provider/kubernetes-secrets` (`#1335`),
whose `injectorSPIFFEID` constant is the mTLS peer check on the only
caller permitted to read Secrets. The comparison is an exact string
match, so a renamed install rejects every fetch at TLS. It becomes
`--injector-identity`, defaulting to
`installdefaults.EgressSPIFFEID(SystemNamespace)`, alongside a new
`EgressServiceAccount` constant and helper. The flag name follows
`--atunnel-client-identity`; the manifest beside it still names
`ate-system`, so a renaming deployment configures both.
- `main` replaced the dialer's worker indexer with
`DialForAteletOnNode`, dropped `WorkerPodInformer` from `controlapi`,
and removed `kataConfig` from ateom-microvm's `NewService`. The series
adapts to each narrower signature and keeps only its own added
parameter.
- The guard test's allowlist entry for the two inert `ate-system`
literals follows the code from `actoridentity.go` to
`controlapi/actor.go`. The refactor re-introduced exactly the class of
constant this series removes, so the guard earns its place.

# Testing

- `go test -race ./...` passes. `make verify` passes boilerplate,
codegen, go-modules, gofmt, golangci-lint, kube-api-linter, licenses,
metrics and postgresql-migrations. `proto-fmt` needs `clang-format`,
which is absent locally; this series touches no `.proto` files.
- Every commit builds and vets individually, via `git rebase -x 'go
build ./... && go vet ./...'`.
- On a fresh kind cluster, all twelve e2e suites pass: `capabilities`,
`combinedvolumes`, `demo`, `egressauthz`, `egressmitm`, `example`,
`identity`, `metrics`, `networking`, `networkpolicy`, `parking`,
`sizing`. The `demo`, `metrics`, `networkpolicy`, `parking` and
`networking` suites need the `--deploy-demo-counter` and
`--deploy-demo-egress` fixtures, and the MITM path needs
`--deploy-demo-egress-mitm`; without them they fail on `actor template
not found`, which reads as a control-plane fault rather than a missing
fixture.
- With the sdsmint egress gateway (`--deploy-atenet
--experimental-use-sdsmint`, `E2E_EGRESS_MITM=1`),
`TestActorEgressMITMTrust` and `TestActorEgressHTTPSByHostnameMITM` both
pass, and the `networking` suite is green at 25 passed. The two modes
are mutually exclusive by design:
`TestActorEgressHTTPSByHostnamePassthrough` covers the plain gateway and
skips under MITM, and the two MITM tests skip without it. Both modes
were run, so every case executed in one of them.
- The new unit tests use a relocated namespace and renamed
ServiceAccounts. Reintroducing each hardcoding makes them fail.

One e2e caveat that predates this series and misleads: the `identity`
suite calls `ReplaceEgressTrustPool`, which takes over the shared
`egress-mitm-ca-pool` Secret, overwrites it and registers no cleanup.
Any MITM test running afterwards fails with `certificate signed by
unknown authority`, which reads as a broken interception path rather
than a mutated fixture. Reinstall the gateway before running
`egressmitm`.

# Additional Notes

**Credential contents stay canonical.** `controlapi/actor.go` still
mints credentials naming `api.ate-system.svc`: the JWT issuer and the
certificate's Issuer CN. Relying parties validate these, so changing
them is a compatibility decision, not a lookup fix. The code carries a
TODO to make the issuer a globally unique, OIDC-compliant name.
2026-09-22 22:28:40 +00:00

304 lines
9.9 KiB
Go

// Copyright 2026 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package controllers
import (
"context"
"encoding/pem"
"strings"
"testing"
"time"
certsv1beta1 "k8s.io/api/certificates/v1beta1"
corev1 "k8s.io/api/core/v1"
k8errors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/types"
clientgoscheme "k8s.io/client-go/kubernetes/scheme"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/client/fake"
"github.com/agent-substrate/substrate/internal/installdefaults"
"github.com/agent-substrate/substrate/internal/localca"
)
func egressMITMScheme(t *testing.T) *runtime.Scheme {
t.Helper()
scheme := runtime.NewScheme()
if err := clientgoscheme.AddToScheme(scheme); err != nil {
t.Fatalf("add scheme: %v", err)
}
return scheme
}
// caPoolSecret marshals ids into a pool Secret shaped the way
// `kubectl-ate admin make-ca-pool` writes it, and returns the roots in pool
// order so a test can assert on exactly what should have been published.
func caPoolSecret(t *testing.T, ids ...string) (*corev1.Secret, *localca.ConcretePool) {
t.Helper()
pool := &localca.ConcretePool{}
for _, id := range ids {
ca, err := localca.GenerateCA(id, localca.KeyTypeED25519, 24*time.Hour)
if err != nil {
t.Fatalf("generate CA %q: %v", id, err)
}
pool.CAs = append(pool.CAs, ca)
}
return secretForPool(t, pool), pool
}
func secretForPool(t *testing.T, pool *localca.ConcretePool) *corev1.Secret {
t.Helper()
wire, err := localca.Marshal(pool)
if err != nil {
t.Fatalf("marshal CA pool: %v", err)
}
ref := EgressMITMCAPoolRef(installdefaults.SystemNamespace)
return &corev1.Secret{
ObjectMeta: metav1.ObjectMeta{Namespace: ref.Namespace, Name: ref.Name},
Data: map[string][]byte{"pool": wire},
}
}
func rootPEM(t *testing.T, pool *localca.ConcretePool) string {
t.Helper()
var b strings.Builder
for _, ca := range pool.CAs {
if err := pem.Encode(&b, &pem.Block{Type: "CERTIFICATE", Bytes: ca.RootCertificate.Raw}); err != nil {
t.Fatalf("encode root: %v", err)
}
}
return b.String()
}
func reconcilePool(t *testing.T, c client.Client) error {
t.Helper()
r := &EgressMITMTrustReconciler{Client: c, SystemNamespace: installdefaults.SystemNamespace}
_, err := r.Reconcile(context.Background(), ctrl.Request{NamespacedName: EgressMITMCAPoolRef(installdefaults.SystemNamespace)})
return err
}
func getTrustBundle(t *testing.T, c client.Client) (*certsv1beta1.ClusterTrustBundle, bool) {
t.Helper()
ctb := &certsv1beta1.ClusterTrustBundle{}
err := c.Get(context.Background(), types.NamespacedName{Name: egressMITMTrustBundleName}, ctb)
if k8errors.IsNotFound(err) {
return nil, false
}
if err != nil {
t.Fatalf("get ClusterTrustBundle: %v", err)
}
return ctb, true
}
func TestEgressMITMTrustPublishesEveryRoot(t *testing.T) {
t.Parallel()
scheme := egressMITMScheme(t)
secret, pool := caPoolSecret(t, "mitm", "mitm-next")
c := fake.NewClientBuilder().WithScheme(scheme).WithObjects(secret).Build()
if err := reconcilePool(t, c); err != nil {
t.Fatalf("Reconcile: %v", err)
}
ctb, ok := getTrustBundle(t, c)
if !ok {
t.Fatal("no ClusterTrustBundle was created")
}
if got, want := ctb.Spec.SignerName, egressMITMSignerName; got != want {
t.Errorf("signerName = %q, want %q", got, want)
}
if got, want := ctb.Labels["podcert.ate.dev/canarying"], "live"; got != want {
t.Errorf("canarying label = %q, want %q", got, want)
}
// Both roots, in pool order. A pool holds more than one CA so the anchor can
// be rotated; dropping the outgoing root breaks leaves it is still signing.
if got, want := ctb.Spec.TrustBundle, rootPEM(t, pool); got != want {
t.Errorf("trustBundle =\n%s\nwant\n%s", got, want)
}
}
// The pool carries each CA's signing key. Publishing it would hand every
// consumer of the bundle the ability to mint leaves for any name.
func TestEgressMITMTrustPublishesNoPrivateKey(t *testing.T) {
t.Parallel()
scheme := egressMITMScheme(t)
secret, _ := caPoolSecret(t, "mitm")
c := fake.NewClientBuilder().WithScheme(scheme).WithObjects(secret).Build()
if err := reconcilePool(t, c); err != nil {
t.Fatalf("Reconcile: %v", err)
}
ctb, ok := getTrustBundle(t, c)
if !ok {
t.Fatal("no ClusterTrustBundle was created")
}
for rest := []byte(ctb.Spec.TrustBundle); len(rest) > 0; {
var block *pem.Block
block, rest = pem.Decode(rest)
if block == nil {
t.Fatalf("trustBundle has trailing non-PEM bytes: %q", rest)
}
if block.Type != "CERTIFICATE" {
t.Errorf("trustBundle contains a %q block; only CERTIFICATE belongs there", block.Type)
}
}
}
func TestEgressMITMTrustFollowsPoolRotation(t *testing.T) {
t.Parallel()
scheme := egressMITMScheme(t)
secret, _ := caPoolSecret(t, "mitm")
c := fake.NewClientBuilder().WithScheme(scheme).WithObjects(secret).Build()
if err := reconcilePool(t, c); err != nil {
t.Fatalf("first Reconcile: %v", err)
}
rotated, rotatedPool := caPoolSecret(t, "mitm", "mitm-next")
current := &corev1.Secret{}
if err := c.Get(context.Background(), EgressMITMCAPoolRef(installdefaults.SystemNamespace), current); err != nil {
t.Fatalf("get pool secret: %v", err)
}
current.Data = rotated.Data
if err := c.Update(context.Background(), current); err != nil {
t.Fatalf("update pool secret: %v", err)
}
if err := reconcilePool(t, c); err != nil {
t.Fatalf("second Reconcile: %v", err)
}
ctb, ok := getTrustBundle(t, c)
if !ok {
t.Fatal("the ClusterTrustBundle disappeared")
}
if got, want := ctb.Spec.TrustBundle, rootPEM(t, rotatedPool); got != want {
t.Errorf("trustBundle after rotation =\n%s\nwant\n%s", got, want)
}
}
func TestEgressMITMTrustRecreatesDeletedBundle(t *testing.T) {
t.Parallel()
scheme := egressMITMScheme(t)
secret, pool := caPoolSecret(t, "mitm")
c := fake.NewClientBuilder().WithScheme(scheme).WithObjects(secret).Build()
if err := reconcilePool(t, c); err != nil {
t.Fatalf("first Reconcile: %v", err)
}
ctb, ok := getTrustBundle(t, c)
if !ok {
t.Fatal("no ClusterTrustBundle was created")
}
if err := c.Delete(context.Background(), ctb); err != nil {
t.Fatalf("delete ClusterTrustBundle: %v", err)
}
if err := reconcilePool(t, c); err != nil {
t.Fatalf("second Reconcile: %v", err)
}
got, ok := getTrustBundle(t, c)
if !ok {
t.Fatal("the ClusterTrustBundle was not recreated")
}
if want := rootPEM(t, pool); got.Spec.TrustBundle != want {
t.Errorf("trustBundle =\n%s\nwant\n%s", got.Spec.TrustBundle, want)
}
}
func TestEgressMITMTrustDeletesBundleWhenPoolIsGone(t *testing.T) {
t.Parallel()
scheme := egressMITMScheme(t)
secret, _ := caPoolSecret(t, "mitm")
c := fake.NewClientBuilder().WithScheme(scheme).WithObjects(secret).Build()
if err := reconcilePool(t, c); err != nil {
t.Fatalf("first Reconcile: %v", err)
}
if err := c.Delete(context.Background(), secret); err != nil {
t.Fatalf("delete pool secret: %v", err)
}
if err := reconcilePool(t, c); err != nil {
t.Fatalf("second Reconcile: %v", err)
}
if _, ok := getTrustBundle(t, c); ok {
t.Error("the ClusterTrustBundle outlived its CA pool")
}
}
func TestEgressMITMTrustLeavesForeignBundleAlone(t *testing.T) {
t.Parallel()
scheme := egressMITMScheme(t)
foreign := &certsv1beta1.ClusterTrustBundle{
ObjectMeta: metav1.ObjectMeta{Name: egressMITMTrustBundleName},
Spec: certsv1beta1.ClusterTrustBundleSpec{SignerName: "someone.else.example/identity"},
}
c := fake.NewClientBuilder().WithScheme(scheme).WithObjects(foreign).Build()
// No pool Secret exists, so this takes the delete path.
if err := reconcilePool(t, c); err == nil {
t.Fatal("Reconcile deleted a ClusterTrustBundle belonging to another signer")
}
if _, ok := getTrustBundle(t, c); !ok {
t.Error("the foreign ClusterTrustBundle was deleted")
}
}
// A pool that cannot be read says nothing about what the anchor should be.
// Truncating the published bundle would break every consumer at once, so the
// last good bundle has to survive an unreadable pool.
func TestEgressMITMTrustKeepsLastGoodBundleOnBadPool(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
data map[string][]byte
}{
{name: "missing key", data: map[string][]byte{"not-pool": []byte("{}")}},
{name: "unparseable", data: map[string][]byte{"pool": []byte("not json")}},
{name: "no CAs", data: map[string][]byte{"pool": []byte(`{"CAs":[]}`)}},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
scheme := egressMITMScheme(t)
secret, pool := caPoolSecret(t, "mitm")
c := fake.NewClientBuilder().WithScheme(scheme).WithObjects(secret).Build()
if err := reconcilePool(t, c); err != nil {
t.Fatalf("first Reconcile: %v", err)
}
current := &corev1.Secret{}
if err := c.Get(context.Background(), EgressMITMCAPoolRef(installdefaults.SystemNamespace), current); err != nil {
t.Fatalf("get pool secret: %v", err)
}
current.Data = tc.data
if err := c.Update(context.Background(), current); err != nil {
t.Fatalf("update pool secret: %v", err)
}
if err := reconcilePool(t, c); err == nil {
t.Error("Reconcile accepted an unreadable pool; it should fail and requeue")
}
ctb, ok := getTrustBundle(t, c)
if !ok {
t.Fatal("the ClusterTrustBundle was removed by an unreadable pool")
}
if want := rootPEM(t, pool); ctb.Spec.TrustBundle != want {
t.Errorf("trustBundle was rewritten from an unreadable pool:\n%s", ctb.Spec.TrustBundle)
}
})
}
}