diff --git a/cmd/ate-setup/README.md b/cmd/ate-setup/README.md index d03123fc8..8ab2f6f8f 100644 --- a/cmd/ate-setup/README.md +++ b/cmd/ate-setup/README.md @@ -10,6 +10,22 @@ make build-ate-setup # builds bin/ate-setup `ate-setup` is a Go port of `hack/install-ate.sh` and the scripts it sources. Both work today and can be used against the same cluster. +## Installing a release + +By default every image is built from the checkout with `ko`, which is what a +developer wants and what the shell installer always did. To install published +images instead, name the registry they were pushed to: + +``` +ate-setup deploy ate-system \ + --image-repo registry.example.com/substrate \ + --image-tag v0.0.0 +``` + +Nothing is built and no registry is pushed to; the manifests still come from the +checkout. Each reference is pinned to the digest its tag names, so the registry +has to be readable from here as well as from the cluster. + - [`commands.md`](commands.md) — every command with its `hack/install-ate.sh` equivalent. - [`differences.md`](differences.md) — where the port deliberately behaves diff --git a/cmd/ate-setup/commands.md b/cmd/ate-setup/commands.md index e69ccb659..b2c565e37 100644 --- a/cmd/ate-setup/commands.md +++ b/cmd/ate-setup/commands.md @@ -31,6 +31,34 @@ a pre-scan pass, so they may appear anywhere on its command line. | `--kubeconfig PATH` | `KUBECONFIG=PATH` | Explicit kubeconfig path | | `--no-dev-env` | `NO_DEV_ENV=1` | Skip `.ate-dev-env.sh` at the repository root | | `--version` / `-v` | — | New; the shell installer had no version | +| `--image-repo REPO` | — | New. Install pre-built images from `REPO` instead of building them with `ko` | +| `--image-tag TAG` | — | New. The tag those images carry. Each of the two requires the other | + +Both have an environment equivalent, read when the flag is absent: +`ATE_IMAGE_REPO` and `ATE_IMAGE_TAG`. + +## Installing a release + +Without `--image-repo`, `ate-setup` builds every image from the checkout with +`ko` and pushes it to `KO_DOCKER_REPO`. That is the developer install and is +unchanged. + +``` +ate-setup deploy ate-system \ + --image-repo registry.example.com/substrate \ + --image-tag v0.0.0 +``` + +installs published images instead, and never invokes `ko`. The manifests still +come from the checkout, so this needs one; what it removes is the build, the Go +toolchain, and write access to a registry. + +`REPO` has to hold every component image the manifests reference, all under the +same tag, which is how a release publishes them. A release that adds a component +has to publish it alongside the others before a pre-built install can use it. +Each reference is then pinned to the digest its tag names, which takes one HEAD +request per image, so the installer needs read access to `REPO` and not only the +cluster does. ## Deploy diff --git a/cmd/ate-setup/differences.md b/cmd/ate-setup/differences.md index a3c49ac7c..6c989eb35 100644 --- a/cmd/ate-setup/differences.md +++ b/cmd/ate-setup/differences.md @@ -73,9 +73,9 @@ Still required, and why: | Binary | Used for | |---|---| -| `ko` | building and publishing images (`ko resolve` only) | -| `go` | locating the pinned `ko` tool, exactly as `hack/run-tool.sh` does | -| `git` | the `git describe` version stamp passed to ko | +| `ko` | building and publishing images (`ko resolve` only); not used with `--image-repo` | +| `go` | locating the pinned `ko` tool, exactly as `hack/run-tool.sh` does; not used with `--image-repo` | +| `git` | the `git describe` version stamp passed to ko; not used with `--image-repo` | | `docker` | the kind CSI setup (`docker exec` into the node) and the claude-code-multiplex workload build | | `gcloud` | GKE `get-credentials`, only when `PROJECT_ID` is set and no context was given | | `bash` | sourcing `.ate-dev-env.sh` | @@ -92,6 +92,56 @@ made ko shell out to kubectl and forced the awkward `-- --context=` special case them). Now ko only resolves, and the resulting manifest is applied through client-go. +## Image sources + +The shell installer had exactly one: build every image from the checkout with +`ko` and push it to `KO_DOCKER_REPO`. That is still the default and is +unchanged. `--image-repo` adds a second, in which nothing is built. + +Manifests carry `ko://` references either way. With `--image-repo` +each is looked up in `images.Components` and rewritten to +`/:`, which is ko's own +`--base-import-paths` naming and so is how the release images are already +published — `ateapi`, `atelet`, `atenet`, `ateom-gvisor`, and the rest. +`--image-tag` supplies the tag. Each of the two requires the other: a tag with +no repository to pull from would be dropped, leaving a build from source that +looks like the release the tag names. + +The rewritten reference then keeps that tag and adds the digest it names, as +`/:@sha256:...`, which is the shape ko itself produces for a +tagged release. Resolving the digest is one HEAD request per image, cached for +the install, so `--image-repo` needs read access to the registry from wherever +`ate-setup` runs rather than only from the cluster. It is not optional: an +ActorTemplate's container image, an image volume's reference, and a +SandboxConfig's `pauseImage` each carry the CEL rule `self.contains('@')`, so an +unpinned reference is rejected by admission and every demo fails to deploy. +Pinning also gives a pre-built install the property a `ko` install had for free, +that a tag moving afterwards cannot change what is running. + +That tag also becomes the substrate version, which names the atelet DaemonSet +and sets the node label partitioning nodes across coexisting versions. It has +to: that label must describe the atelet actually running, which came from the +image, not the checkout `git describe` happens to be sitting on. `VERSION` +overrides it, as it always has. + +The rewrite is textual, one token at a time, matching `SubstituteVersion` above. +A reference is replaced wherever it appears, so the code needs no list of which +fields hold images — `workerImage:` in a CRD spec is rewritten like any other — +and the applied manifest differs from the checked-in YAML only in the image +references. + +Every reference must be in the list. A reference outside the module path, one +carrying an unexpanded `${PLACEHOLDER}`, and one naming a package that is not +installable all fail the same way — an error naming every unmapped reference at +once, rather than an unpullable image reaching the cluster and failing as +`ImagePullBackOff` twenty minutes later. Matching whole references rather than +replacing the listed ones is what makes that hold: a reference extending a +listed package would otherwise inherit its image. `images.Components` is checked +against the real manifests by a test, so adding a component fails a test rather +than an install — though only the list is checked that way. The release also has +to publish the new image under the same repository and tag as the rest, and +until it does, the component can only be installed by building it. + ## Cluster access **Server-side apply.** `kubectl apply -f -` is a client-side apply by default. @@ -219,4 +269,4 @@ outcome. The shell installer had no tests. `cmd/ate-setup` has unit tests for template rendering, overlay selection, config resolution, the authentication config, the apiserver environment ConfigMap, delegated script arguments, manifest deletion, -and per-demo rendering. +per-demo rendering, and image reference rewriting. diff --git a/cmd/ate-setup/internal/cmd/root.go b/cmd/ate-setup/internal/cmd/root.go index 70ecd9dbb..75b93886d 100644 --- a/cmd/ate-setup/internal/cmd/root.go +++ b/cmd/ate-setup/internal/cmd/root.go @@ -95,6 +95,11 @@ func init() { f.StringVar(&opts.AdditionalEgressExtprocService, "experimental-additional-egress-extproc-service", "", "Run an additional ext_proc authorization filter served by NS/SVC:PORT (requires --experimental-use-sdsmint)") f.BoolVar(&opts.NoDevEnv, "no-dev-env", false, "Do not source .ate-dev-env.sh") + f.StringVar(&opts.ImageRepo, "image-repo", "", + "Install pre-built images from this registry path instead of building them from source (e.g. registry.example.com/substrate)") + f.StringVar(&opts.ImageTag, "image-tag", "", + "Tag the pre-built images carry (required with --image-repo)") + // Cobra's default completion command would run PersistentPreRunE and // require a cluster; the tree is not deep enough to justify that. rootCmd.CompletionOptions.DisableDefaultCmd = true diff --git a/cmd/ate-setup/internal/config/config.go b/cmd/ate-setup/internal/config/config.go index ac14bcdca..5f13ac365 100644 --- a/cmd/ate-setup/internal/config/config.go +++ b/cmd/ate-setup/internal/config/config.go @@ -27,6 +27,8 @@ import ( "strconv" "strings" "time" + + "github.com/agent-substrate/substrate/cmd/ate-setup/internal/images" ) // Enumerated values for the install-shaping flags. @@ -85,6 +87,10 @@ type Config struct { // KODefaultPlatforms constrains ko's build platforms. KODefaultPlatforms string + // Images selects where container images come from. Its zero value builds + // them from source with ko, which is what a developer install does. + Images images.Source + // Router selects the atenet router dataplane. Router string // PostgresConnectionString is the apiserver's store connection string. @@ -134,6 +140,10 @@ type Options struct { ExperimentalUseSDSMint bool AdditionalEgressExtprocService string + // Image source selection. + ImageRepo string + ImageTag string + // NoDevEnv skips sourcing .ate-dev-env.sh even when it exists. NoDevEnv bool } @@ -202,6 +212,7 @@ func Load(opts Options) (*Config, error) { BucketName: env["BUCKET_NAME"], KODockerRepo: env["KO_DOCKER_REPO"], KODefaultPlatforms: env["KO_DEFAULTPLATFORMS"], + Images: loadImageSource(opts, env), PostgresConnectionString: env["ATE_API_POSTGRES_CONNECTION_STRING"], RolloutTimeout: rolloutTimeout, rolloutTimeoutSet: timeoutStr != "", @@ -226,6 +237,14 @@ func Load(opts Options) (*Config, error) { return cfg, nil } +// loadImageSource resolves where images come from. +func loadImageSource(opts Options, env map[string]string) images.Source { + return images.Source{ + Repo: strings.TrimSuffix(firstNonEmpty(opts.ImageRepo, env["ATE_IMAGE_REPO"]), "/"), + Tag: firstNonEmpty(opts.ImageTag, env["ATE_IMAGE_TAG"]), + } +} + func applyKindDefaults(cfg *Config) { cfg.ProjectID = "" cfg.ClusterLocation = "" @@ -237,6 +256,9 @@ func applyKindDefaults(cfg *Config) { } func validate(cfg *Config) error { + if err := cfg.Images.Validate(); err != nil { + return err + } switch cfg.Router { case RouterEnvoy, RouterAgentgateway: default: diff --git a/cmd/ate-setup/internal/config/config_test.go b/cmd/ate-setup/internal/config/config_test.go index 6ce8c1886..288fef34f 100644 --- a/cmd/ate-setup/internal/config/config_test.go +++ b/cmd/ate-setup/internal/config/config_test.go @@ -22,6 +22,8 @@ import ( "strings" "testing" "time" + + "github.com/agent-substrate/substrate/cmd/ate-setup/internal/images" ) // loadEnv isolates Load from the ambient environment. @@ -41,6 +43,8 @@ func loadEnv(t *testing.T) { "ATE_API_POSTGRES_CONNECTION_STRING", "ATE_ATENET_ROUTER", "ATE_EXPERIMENTAL_USE_SDSMINT", + "ATE_IMAGE_REPO", + "ATE_IMAGE_TAG", "ATE_INSTALL_PODCERT_WORKERS_PER_SIGNER", "ATE_INSTALL_ROLLOUT_TIMEOUT", "ATE_OTLP_ENDPOINT", @@ -341,3 +345,89 @@ func TestSourceShellEnvReportsFailure(t *testing.T) { t.Fatal("sourceShellEnv() succeeded, want an error") } } + +func TestLoadImageSource(t *testing.T) { + tests := []struct { + name string + env map[string]string + opts Options + want images.Source + wantError string + }{ + { + name: "unset builds from source", + want: images.Source{}, + }, + { + name: "flags", + opts: Options{ImageRepo: "example.com/substrate", ImageTag: "v1.2.3"}, + want: images.Source{Repo: "example.com/substrate", Tag: "v1.2.3"}, + }, + { + name: "environment", + env: map[string]string{"ATE_IMAGE_REPO": "example.com/env", "ATE_IMAGE_TAG": "v9"}, + want: images.Source{Repo: "example.com/env", Tag: "v9"}, + }, + { + name: "flags beat the environment", + env: map[string]string{"ATE_IMAGE_REPO": "example.com/env", "ATE_IMAGE_TAG": "v9"}, + opts: Options{ImageRepo: "example.com/flag", ImageTag: "v1"}, + want: images.Source{Repo: "example.com/flag", Tag: "v1"}, + }, + { + // A repo pasted from a registry UI often carries one, and it would + // otherwise produce a double slash in every reference. + name: "trailing slash is trimmed", + opts: Options{ImageRepo: "example.com/substrate/", ImageTag: "v1"}, + want: images.Source{Repo: "example.com/substrate", Tag: "v1"}, + }, + { + name: "a repo needs a tag", + opts: Options{ImageRepo: "example.com/substrate"}, + wantError: "--image-repo (or ATE_IMAGE_REPO) requires --image-tag", + }, + { + name: "a tag needs a repo", + opts: Options{ImageTag: "v1"}, + wantError: "--image-tag (or ATE_IMAGE_TAG) requires --image-repo", + }, + { + // The environment reaches validation the same way the flags do. + name: "a tag from the environment needs a repo", + env: map[string]string{"ATE_IMAGE_TAG": "v1"}, + wantError: "--image-tag (or ATE_IMAGE_TAG) requires --image-repo", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + loadEnv(t) + for name, value := range tc.env { + t.Setenv(name, value) + } + + cfg, err := Load(tc.opts) + if tc.wantError != "" { + if err == nil { + t.Fatalf("Load() = nil error, want one containing %q", tc.wantError) + } + if !strings.Contains(err.Error(), tc.wantError) { + t.Errorf("Load() error = %v, want it to contain %q", err, tc.wantError) + } + return + } + if err != nil { + t.Fatalf("Load() error = %v", err) + } + if cfg.Images.Repo != tc.want.Repo { + t.Errorf("Images.Repo = %q, want %q", cfg.Images.Repo, tc.want.Repo) + } + if cfg.Images.Tag != tc.want.Tag { + t.Errorf("Images.Tag = %q, want %q", cfg.Images.Tag, tc.want.Tag) + } + if cfg.Images.IsPrebuilt() != (tc.want.Repo != "") { + t.Errorf("Images.IsPrebuilt() = %v, want %v", cfg.Images.IsPrebuilt(), tc.want.Repo != "") + } + }) + } +} diff --git a/cmd/ate-setup/internal/demos/simple.go b/cmd/ate-setup/internal/demos/simple.go index dee0b566d..d4c0bc69f 100644 --- a/cmd/ate-setup/internal/demos/simple.go +++ b/cmd/ate-setup/internal/demos/simple.go @@ -110,7 +110,7 @@ func (d *Simple) DeployWorkload(ctx context.Context, e *steps.Env) error { if err != nil { return err } - return e.KoApplyBytes(ctx, manifest) + return e.ResolveAndApplyBytes(ctx, manifest) } // WaitReady blocks until the demo's Deployments are rolled out. diff --git a/cmd/ate-setup/internal/demos/substrate.go b/cmd/ate-setup/internal/demos/substrate.go index 7fcba7ed4..20091803e 100644 --- a/cmd/ate-setup/internal/demos/substrate.go +++ b/cmd/ate-setup/internal/demos/substrate.go @@ -116,7 +116,7 @@ func (d *Substrate) Deploy(ctx context.Context, e *steps.Env) error { if err != nil { return err } - if err := e.KoApplyBytes(ctx, manifest); err != nil { + if err := e.ResolveAndApplyBytes(ctx, manifest); err != nil { return err } for _, ref := range d.Deployments { @@ -142,10 +142,10 @@ func (d *Substrate) Deploy(ctx context.Context, e *steps.Env) error { if err != nil { return err } - // ko resolve replaces the ko:// image references with pushed digests - // before the manifest is parsed; manifests without ko:// references - // pass through unchanged. - resolved, err := e.KoResolveBytes(ctx, manifest) + // The ko:// image references become digest-pinned ones before the + // manifest is parsed, which is what the ActorTemplate schema requires; + // manifests without ko:// references pass through unchanged. + resolved, err := e.ResolveManifestBytes(ctx, manifest) if err != nil { return err } diff --git a/cmd/ate-setup/internal/images/digest.go b/cmd/ate-setup/internal/images/digest.go new file mode 100644 index 000000000..7cca8979d --- /dev/null +++ b/cmd/ate-setup/internal/images/digest.go @@ -0,0 +1,50 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package images + +import ( + "context" + "fmt" + + "github.com/google/go-containerregistry/pkg/authn" + "github.com/google/go-containerregistry/pkg/name" + "github.com/google/go-containerregistry/pkg/v1/remote" +) + +// Digester resolves a tagged image reference to the digest its tag currently +// names, in "sha256:..." form. Prebuilt takes one so tests can rewrite +// manifests without a registry. +type Digester func(ctx context.Context, ref string) (string, error) + +// RemoteDigest asks ref's registry which manifest the tag names. +// +// One HEAD request per image, needing only read access, and authenticated the +// way docker and ko are: from the docker config file and the credential helpers +// it names. A localhost registry is contacted over plain HTTP, which is what +// the Kind local registry serves. +func RemoteDigest(ctx context.Context, ref string) (string, error) { + parsed, err := name.ParseReference(ref) + if err != nil { + return "", fmt.Errorf("%s is not a valid image reference: %w", ref, err) + } + desc, err := remote.Head(parsed, + remote.WithContext(ctx), + remote.WithAuthFromKeychain(authn.DefaultKeychain), + ) + if err != nil { + return "", fmt.Errorf("resolving %s to a digest: %w", ref, err) + } + return desc.Digest.String(), nil +} diff --git a/cmd/ate-setup/internal/images/images.go b/cmd/ate-setup/internal/images/images.go new file mode 100644 index 000000000..2b1ed1c62 --- /dev/null +++ b/cmd/ate-setup/internal/images/images.go @@ -0,0 +1,107 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package images decides where ate-setup gets container images from. +// +// The manifests name images as ko:// import paths, which ko builds and +// publishes on every install. That needs a checkout, a Go toolchain, and a +// registry to push to. Installing an already-published release instead means +// mapping those same references onto images that exist, which is what Source +// describes and Prebuilt performs. +package images + +import ( + "fmt" + "path" +) + +// ModulePath is this repository's module path. A ko:// reference is it joined +// with a component's package. +const ModulePath = "github.com/agent-substrate/substrate" + +// Components is every package the installer can deploy a published image for, +// as an import path within the module. +// +// The list is closed on purpose. A reference that is not in it names something +// this build has no published image for, and failing beats guessing at a name +// that was never pushed. TestComponentsCoverTheManifests keeps it in step with +// the manifests, which is half of it: a new component also has to be published +// under the same repository and tag as the rest before a pre-built install can +// use it. +var Components = []string{ + "cmd/ateapi", + "cmd/atecontroller", + "cmd/atelet", + "cmd/atenet", + "cmd/ateom-gvisor", + "cmd/ateom-microvm", + "cmd/podcertcontroller", + "demos/counter", + "demos/egress", + "demos/multi-template/fspersist", + "demos/sandbox", +} + +// KoReference is the ko:// reference the manifests name pkg by. +// +// ko resolves a manifest reference as written, so a "./"-relative import path +// works there too. Naming packages in full is this repository's convention +// rather than ko's rule, and only the full form maps to an image here: +// TestComponentsCoverTheManifests holds the convention, and a reference spelled +// any other way fails the install rather than resolving. +func KoReference(pkg string) string { return "ko://" + ModulePath + "/" + pkg } + +// ImageName is the image name pkg is published under: the last element of the +// import path, which is ko's --base-import-paths naming. ko lowercases it as +// well, which changes nothing while every package here already is. +func ImageName(pkg string) string { return path.Base(pkg) } + +// Source describes where images come from. +// +// The zero value builds and publishes them from source with ko. Setting Repo +// switches to images already published in a registry, and ko is never invoked. +type Source struct { + // Repo is the registry path holding the component images, without a + // trailing slash, e.g. "registry.example.com/substrate". + Repo string + // Tag is the tag every component image carries. Releases tag all + // components together, so one tag covers the set. + Tag string +} + +// IsPrebuilt reports whether images should be pulled rather than built. +func (s Source) IsPrebuilt() bool { return s.Repo != "" } + +// Validate checks a Source is usable before anything reaches the cluster. +// +// Both halves are checked. A tag on its own would otherwise be discarded in +// silence, leaving a build from source that looks like the release the tag +// names. +func (s Source) Validate() error { + switch { + case s.IsPrebuilt() && s.Tag == "": + return fmt.Errorf("--image-repo (or ATE_IMAGE_REPO) requires --image-tag (or ATE_IMAGE_TAG)") + case !s.IsPrebuilt() && s.Tag != "": + return fmt.Errorf("--image-tag (or ATE_IMAGE_TAG) requires --image-repo (or ATE_IMAGE_REPO); without one, images are built from source and the tag names nothing") + } + return nil +} + +// Describe renders the source for the install log. +func (s Source) Describe() string { + if !s.IsPrebuilt() { + return "built from source with ko" + } + return fmt.Sprintf("pre-built from %s, tag %s", s.Repo, s.Tag) +} diff --git a/cmd/ate-setup/internal/images/images_test.go b/cmd/ate-setup/internal/images/images_test.go new file mode 100644 index 000000000..30ec2bb00 --- /dev/null +++ b/cmd/ate-setup/internal/images/images_test.go @@ -0,0 +1,165 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// The component list is checked against the real manifests, so this is an +// external test package: config imports images, and only the test needs config. +package images_test + +import ( + "io/fs" + "os" + "path/filepath" + "regexp" + "slices" + "strings" + "testing" + + "github.com/agent-substrate/substrate/cmd/ate-setup/internal/config" + "github.com/agent-substrate/substrate/cmd/ate-setup/internal/images" +) + +// installTrees are the directories the installer reads manifests from. +// +// benchmarking/ and internal/e2e/ are left out deliberately: neither is +// installed through the image resolvers -- the benchmark workloads go through +// a shell script that runs ko itself, and the e2e fixtures are built by the +// test suite. +var installTrees = []string{"manifests", "demos"} + +var koRefPattern = regexp.MustCompile(`ko://[^\s"',\]}]+`) + +// componentsInTree collects the package of every ko:// reference under dir. +// +// A reference that does not carry the module path is recorded whole, so it +// shows up in the failure rather than being silently dropped. +func componentsInTree(t *testing.T, dir string) []string { + t.Helper() + var found []string + err := filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + if d.IsDir() { + return nil + } + switch filepath.Ext(strings.TrimSuffix(p, ".tmpl")) { + case ".yaml", ".yml", ".json": + default: + return nil + } + body, err := os.ReadFile(p) + if err != nil { + return err + } + for _, match := range koRefPattern.FindAllString(string(body), -1) { + pkg, ok := strings.CutPrefix(match, "ko://"+images.ModulePath+"/") + if !ok { + pkg = match + } + if !slices.Contains(found, pkg) { + found = append(found, pkg) + } + } + return nil + }) + if err != nil { + t.Fatalf("walking %s: %v", dir, err) + } + return found +} + +// images.Components is a closed list, so a component added to a manifest but +// not to the list would install fine from source and fail only when someone +// tried to install a release. Catch it here instead. +// +// The same comparison holds the spelling of the references. ko accepts a +// "./"-relative import path as well as a full one, and images.KoReference only +// builds the full form, so a relative reference would resolve to no image. It +// lands here as an entry the list cannot contain. +func TestComponentsCoverTheManifests(t *testing.T) { + root, err := config.RepoRoot() + if err != nil { + t.Fatalf("resolving repo root: %v", err) + } + + var found []string + for _, tree := range installTrees { + for _, pkg := range componentsInTree(t, filepath.Join(root, tree)) { + if !slices.Contains(found, pkg) { + found = append(found, pkg) + } + } + } + slices.Sort(found) + + want := slices.Clone(images.Components) + slices.Sort(want) + + if !slices.Equal(found, want) { + t.Errorf("the components referenced by %s and images.Components disagree:\n"+ + " in the manifests: %v\n"+ + " in images.Components: %v\n"+ + "add the missing entries to images.Components, or remove the stale ones; "+ + "an entry still carrying a ko:// prefix is a reference that does not name "+ + "its package by the full import path %s/...", + strings.Join(installTrees, "/ and "), found, want, images.ModulePath) + } +} + +func TestSourceValidate(t *testing.T) { + tests := []struct { + name string + src images.Source + error string + }{ + { + name: "build from source", + src: images.Source{}, + }, + { + name: "prebuilt", + src: images.Source{Repo: "example.com/substrate", Tag: "v1"}, + }, + { + name: "no tag", + src: images.Source{Repo: "example.com/substrate"}, + error: "--image-repo (or ATE_IMAGE_REPO) requires --image-tag", + }, + { + // A tag with nowhere to pull from would otherwise be dropped, and + // the source build that followed would look like the release. + name: "no repo", + src: images.Source{Tag: "v1"}, + error: "--image-tag (or ATE_IMAGE_TAG) requires --image-repo", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + err := tc.src.Validate() + if tc.error == "" { + if err != nil { + t.Fatalf("Validate() = %v, want nil", err) + } + return + } + if err == nil { + t.Fatalf("Validate() = nil, want an error containing %q", tc.error) + } + if !strings.Contains(err.Error(), tc.error) { + t.Errorf("Validate() error = %v, want it to contain %q", err, tc.error) + } + }) + } +} diff --git a/cmd/ate-setup/internal/images/prebuilt.go b/cmd/ate-setup/internal/images/prebuilt.go new file mode 100644 index 000000000..2631a3bca --- /dev/null +++ b/cmd/ate-setup/internal/images/prebuilt.go @@ -0,0 +1,161 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package images + +import ( + "context" + "errors" + "fmt" + "maps" + "regexp" + "slices" + "strings" + + "github.com/agent-substrate/substrate/cmd/ate-setup/internal/kube" +) + +// koRef matches a whole ko:// reference: the prefix plus everything up to the +// first delimiter that can end a YAML scalar. +// +// Matching the whole reference is what makes the rewrite safe. Replacing the +// table's literals directly would rewrite a prefix of a longer reference, so +// ko:///cmd/atelet-sidecar would silently become the atelet image with +// "-sidecar" left dangling on the tag. Requiring a character after the prefix +// also keeps the "# ko:// image reference" prose in the demo templates from +// matching. +var koRef = regexp.MustCompile(`ko://[^\s"',\]}]+`) + +// imageByRef maps a ko:// reference to the image name it publishes as. +// Rewriting is a lookup in this table. +var imageByRef = func() map[string]string { + m := make(map[string]string, len(Components)) + for _, pkg := range Components { + m[KoReference(pkg)] = ImageName(pkg) + } + return m +}() + +// Prebuilt rewrites ko:// references to point at already-published images. +// +// It is the drop-in counterpart to ko.Runner: same two methods, no builds, no +// registry writes, and no repository checkout beyond the manifests themselves. +// What it does need is read access to the registry, to pin each tag to a +// digest. +type Prebuilt struct { + src Source + digest Digester + + // pinned caches a resolved reference by its tagged form, so a component + // several manifests name costs one registry lookup per install. + pinned map[string]string +} + +// NewPrebuilt returns a resolver for an already-published image set. digest +// turns a tag into the image it names; callers outside tests pass RemoteDigest. +func NewPrebuilt(src Source, digest Digester) *Prebuilt { + return &Prebuilt{src: src, digest: digest, pinned: make(map[string]string)} +} + +// ResolvePath rewrites the manifests a path covers. +func (p *Prebuilt) ResolvePath(ctx context.Context, path string) ([]byte, error) { + manifest, err := kube.ReadPath(path) + if err != nil { + return nil, err + } + out, err := p.rewrite(ctx, manifest) + if err != nil { + return nil, fmt.Errorf("in %s: %w", path, err) + } + return out, nil +} + +// ResolveBytes rewrites an in-memory manifest, such as kustomize output. +func (p *Prebuilt) ResolveBytes(ctx context.Context, manifest []byte) ([]byte, error) { + return p.rewrite(ctx, manifest) +} + +// pin returns the reference to install an image by: its tag, and the digest +// that tag currently names. +// +// The digest is not decoration. An ActorTemplate's container image, an image +// volume's reference, and a SandboxConfig's pauseImage each carry the CEL rule +// self.contains('@'), so a tag on its own is rejected by admission and every +// demo would fail to deploy. Pinning also restores what ko gave the control +// plane deployments for free: an install that cannot shift under a tag someone +// moves later. Keeping the tag alongside the digest is ko's own output shape, +// and leaves the reference legible in `kubectl get`. +func (p *Prebuilt) pin(ctx context.Context, image string) (string, error) { + tagged := p.src.Repo + "/" + image + ":" + p.src.Tag + if ref, ok := p.pinned[tagged]; ok { + return ref, nil + } + digest, err := p.digest(ctx, tagged) + if err != nil { + return "", err + } + ref := tagged + "@" + digest + p.pinned[tagged] = ref + return ref, nil +} + +// rewrite replaces every ko:// reference with the image it maps to. +// +// Substitution is textual, the same way SubstituteVersion fills +// ${SUBSTRATE_VERSION}. References appear in CRD fields as well as pod specs (a +// WorkerPool's spec.workerImage, an ActorTemplate's image), so a schema walk +// would need to know every such field, and rewriting the bytes leaves +// everything else -- the multi-kilobyte Envoy configuration blocks in +// particular -- exactly as committed. +func (p *Prebuilt) rewrite(ctx context.Context, manifest []byte) ([]byte, error) { + // Collect the failures rather than stopping at the first. Diagnosing them + // one install attempt at a time is slow, and they are usually related: a + // registry that is unreachable or a tag that was never pushed fails for + // every component at once. + unknown := make(map[string]bool) + unpinned := make(map[string]error) + + out := koRef.ReplaceAllFunc(manifest, func(match []byte) []byte { + image, ok := imageByRef[string(match)] + if !ok { + unknown[string(match)] = true + return match + } + ref, err := p.pin(ctx, image) + if err != nil { + unpinned[image] = err + return match + } + return []byte(ref) + }) + + var errs []error + if len(unknown) > 0 { + // One line per reference, then the package list once. Repeating the + // list per reference buries the references themselves. + for _, ref := range slices.Sorted(maps.Keys(unknown)) { + errs = append(errs, fmt.Errorf("%s has no published image", ref)) + } + errs = append(errs, fmt.Errorf("installable packages under %s are %s; a reference that is "+ + "templated has to be rendered before it can be resolved", + ModulePath, strings.Join(Components, ", "))) + } + for _, image := range slices.Sorted(maps.Keys(unpinned)) { + errs = append(errs, unpinned[image]) + } + if len(errs) > 0 { + return nil, errors.Join(errs...) + } + return out, nil +} diff --git a/cmd/ate-setup/internal/images/prebuilt_test.go b/cmd/ate-setup/internal/images/prebuilt_test.go new file mode 100644 index 000000000..047ba391d --- /dev/null +++ b/cmd/ate-setup/internal/images/prebuilt_test.go @@ -0,0 +1,340 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package images_test + +import ( + "context" + "errors" + "fmt" + "maps" + "os" + "path/filepath" + "regexp" + "strings" + "testing" + + "github.com/agent-substrate/substrate/cmd/ate-setup/internal/config" + "github.com/agent-substrate/substrate/cmd/ate-setup/internal/images" + "github.com/agent-substrate/substrate/cmd/ate-setup/internal/kube" +) + +// testSource is the repo/tag every rewrite test resolves against. +var testSource = images.Source{Repo: "example.com/substrate", Tag: "v1.2.3"} + +// What a resolved reference carries beyond its tag. Fixed rather than derived +// from the reference so the expected output below reads as a literal. +const ( + testDigest = "sha256:1111111111111111111111111111111111111111111111111111111111111111" + digestSuffix = "@" + testDigest +) + +// stubRegistry stands in for the registry the installer would query: it answers +// every lookup with testDigest, or with err, and records what was asked for. +type stubRegistry struct { + calls map[string]int + err error +} + +func newStubRegistry() *stubRegistry { return &stubRegistry{calls: make(map[string]int)} } + +// digest is the [images.Digester] a test passes to [images.NewPrebuilt]. +func (s *stubRegistry) digest(_ context.Context, ref string) (string, error) { + s.calls[ref]++ + if s.err != nil { + return "", fmt.Errorf("resolving %s to a digest: %w", ref, s.err) + } + return testDigest, nil +} + +func TestPrebuiltResolveBytes(t *testing.T) { + tests := []struct { + name string + src images.Source + in string + want string + error string + }{ + { + name: "bare scalar", + in: " image: ko://github.com/agent-substrate/substrate/cmd/ateapi\n", + want: " image: example.com/substrate/ateapi:v1.2.3" + digestSuffix + "\n", + }, + { + name: "double quoted", + in: `image: "ko://github.com/agent-substrate/substrate/cmd/atelet"`, + want: `image: "example.com/substrate/atelet:v1.2.3` + digestSuffix + `"`, + }, + { + name: "single quoted", + in: `image: 'ko://github.com/agent-substrate/substrate/cmd/atenet'`, + want: `image: 'example.com/substrate/atenet:v1.2.3` + digestSuffix + `'`, + }, + { + name: "flow sequence", + in: `images: [ko://github.com/agent-substrate/substrate/demos/counter, other]`, + want: `images: [example.com/substrate/counter:v1.2.3` + digestSuffix + `, other]`, + }, + { + // The reference is a plain CRD field here, not a pod spec, which is + // why the rewrite cannot be driven off a Kubernetes schema. + name: "CRD field", + in: "spec:\n workerImage: ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor\n", + want: "spec:\n workerImage: example.com/substrate/ateom-gvisor:v1.2.3" + digestSuffix + "\n", + }, + { + name: "nested package maps to its image name", + in: "image: ko://github.com/agent-substrate/substrate/demos/multi-template/fspersist\n", + want: "image: example.com/substrate/fspersist:v1.2.3" + digestSuffix + "\n", + }, + { + name: "two references on one line", + in: "a: ko://github.com/agent-substrate/substrate/cmd/ateapi, b: ko://github.com/agent-substrate/substrate/demos/egress\n", + want: "a: example.com/substrate/ateapi:v1.2.3" + digestSuffix + ", b: example.com/substrate/egress:v1.2.3" + digestSuffix + "\n", + }, + { + // The demo templates carry "# ko:// image reference" as prose. The + // regex requires a character after the prefix so those survive. + name: "prose mention is left alone", + in: " # ko:// image reference for the workload\n", + want: " # ko:// image reference for the workload\n", + }, + { + name: "nothing to rewrite", + in: "apiVersion: v1\nkind: ConfigMap\n", + want: "apiVersion: v1\nkind: ConfigMap\n", + }, + { + // benchmarking/workloads/manifests templates the sandbox class into + // the import path. It has to fail rather than resolve to something. + // + // The reported reference stops at the "}", which ends a YAML flow + // mapping and so terminates the match. That costs a character in the + // message and nothing else: the truncation cannot collide with a + // listed package, because every one of those ends before the "$". + name: "unexpanded placeholder", + in: "image: ko://github.com/agent-substrate/substrate/cmd/ateom-${SANDBOX_CLASS}\n", + error: "cmd/ateom-${SANDBOX_CLASS has no published image", + }, + { + // A reference extending a known one must not inherit its image. A + // literal replacement would leave "-sidecar" dangling on the tag. + name: "reference extending a known one", + in: "image: ko://github.com/agent-substrate/substrate/cmd/atelet-sidecar\n", + error: "cmd/atelet-sidecar has no published image", + }, + { + // ko accepts a "./"-relative import path in a manifest as readily as + // a full one, and would publish this as "atelet". Nothing in the tree + // spells a reference this way, so it is rejected rather than mapped: + // only the packages named in full can be known to have been published. + name: "relative import path", + in: "image: ko://./cmd/atelet\n", + error: "ko://./cmd/atelet has no published image", + }, + { + // ko would map this onto its own "atelet" image, because it names + // images by the last path element alone. Matching the whole reference + // is what keeps another module's package out of this repo's registry + // path. + name: "another module ending in a known name", + in: "image: ko://github.com/example/other/cmd/atelet\n", + error: "ko://github.com/example/other/cmd/atelet has no published image", + }, + { + name: "outside the module", + in: "image: ko://github.com/example/other/cmd/thing\n", + error: "ko://github.com/example/other/cmd/thing has no published image", + }, + { + name: "unknown component", + in: "image: ko://github.com/agent-substrate/substrate/cmd/notacomponent\n", + error: "cmd/notacomponent has no published image", + }, + { + // The e2e fixtures are not part of an install, so a manifest + // reaching the installer with one is a mistake worth reporting. + name: "e2e fixture is not installable", + in: "image: ko://github.com/agent-substrate/substrate/internal/e2e/fixtures/probe\n", + error: "e2e/fixtures/probe has no published image", + }, + { + // Every unmappable reference is named, not just the first. + name: "all failures are reported", + in: "a: ko://github.com/agent-substrate/substrate/cmd/one\nb: ko://github.com/agent-substrate/substrate/cmd/two\n", + error: "cmd/two has no published image", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + src := tc.src + if src.Repo == "" { + src = testSource + } + got, err := images.NewPrebuilt(src, newStubRegistry().digest). + ResolveBytes(context.Background(), []byte(tc.in)) + + if tc.error != "" { + if err == nil { + t.Fatalf("ResolveBytes() = %q, want an error containing %q", got, tc.error) + } + if !strings.Contains(err.Error(), tc.error) { + t.Errorf("ResolveBytes() error = %v, want it to contain %q", err, tc.error) + } + return + } + if err != nil { + t.Fatalf("ResolveBytes() error = %v", err) + } + if string(got) != tc.want { + t.Errorf("ResolveBytes() =\n%q\nwant\n%q", got, tc.want) + } + }) + } +} + +// Every unmappable reference is reported at once, so a manifest with several +// problems takes one install attempt to diagnose rather than several. +func TestPrebuiltReportsEveryFailure(t *testing.T) { + in := "a: ko://github.com/agent-substrate/substrate/cmd/ateom-${SANDBOX_CLASS}\n" + + "b: ko://github.com/example/other/cmd/thing\n" + + "c: ko://github.com/agent-substrate/substrate/cmd/nope\n" + + _, err := images.NewPrebuilt(testSource, newStubRegistry().digest). + ResolveBytes(context.Background(), []byte(in)) + if err == nil { + t.Fatal("ResolveBytes() = nil error, want failures for all three references") + } + for _, want := range []string{"SANDBOX_CLASS", "github.com/example/other", "cmd/nope"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error is missing %q:\n%v", want, err) + } + } +} + +// A repeated bad reference is one problem, not one per occurrence. +func TestPrebuiltDeduplicatesFailures(t *testing.T) { + ref := "image: ko://github.com/agent-substrate/substrate/cmd/nope\n" + _, err := images.NewPrebuilt(testSource, newStubRegistry().digest). + ResolveBytes(context.Background(), []byte(ref+ref+ref)) + if err == nil { + t.Fatal("ResolveBytes() = nil error, want a failure") + } + if n := strings.Count(err.Error(), "cmd/nope has no published image"); n != 1 { + t.Errorf("reported the same reference %d times, want 1:\n%v", n, err) + } +} + +// A tag is resolved once per image, however many manifests name it, and the +// answer is reused across calls: an install applies several manifests through +// one resolver, and a registry round trip each time would be waste. +func TestPrebuiltLooksUpEachImageOnce(t *testing.T) { + registry := newStubRegistry() + resolver := images.NewPrebuilt(testSource, registry.digest) + + in := "a: ko://github.com/agent-substrate/substrate/cmd/ateapi\n" + + "b: ko://github.com/agent-substrate/substrate/cmd/atelet\n" + + "c: ko://github.com/agent-substrate/substrate/cmd/ateapi\n" + for range 2 { + if _, err := resolver.ResolveBytes(context.Background(), []byte(in)); err != nil { + t.Fatalf("ResolveBytes() error = %v", err) + } + } + + want := map[string]int{ + "example.com/substrate/ateapi:v1.2.3": 1, + "example.com/substrate/atelet:v1.2.3": 1, + } + if !maps.Equal(registry.calls, want) { + t.Errorf("registry lookups = %v, want %v", registry.calls, want) + } +} + +// A registry that cannot answer fails the install, and says so for every image +// it was asked about. Falling back to the bare tag would produce manifests that +// admission rejects, and a partially pinned manifest would be worse still. +func TestPrebuiltReportsDigestFailures(t *testing.T) { + registry := newStubRegistry() + registry.err = errors.New("UNAUTHORIZED") + + in := "a: ko://github.com/agent-substrate/substrate/cmd/ateapi\n" + + "b: ko://github.com/agent-substrate/substrate/cmd/atelet\n" + got, err := images.NewPrebuilt(testSource, registry.digest). + ResolveBytes(context.Background(), []byte(in)) + if err == nil { + t.Fatalf("ResolveBytes() = %q, want an error", got) + } + for _, want := range []string{ + "resolving example.com/substrate/ateapi:v1.2.3 to a digest: UNAUTHORIZED", + "resolving example.com/substrate/atelet:v1.2.3 to a digest: UNAUTHORIZED", + } { + if !strings.Contains(err.Error(), want) { + t.Errorf("error is missing %q:\n%v", want, err) + } + } +} + +// resolvedRef matches a rewritten reference in testSource's repository. +var resolvedRef = regexp.MustCompile(regexp.QuoteMeta(testSource.Repo) + `/[^\s"',\]}]+`) + +// Resolving the control plane manifests must produce a manifest that still +// parses and holds no unresolved reference. This is the closest a unit test +// gets to the install itself. +func TestResolveEveryInstallManifest(t *testing.T) { + root, err := config.RepoRoot() + if err != nil { + t.Fatalf("resolving repo root: %v", err) + } + installDir := filepath.Join(root, "manifests", "ate-install") + + entries, err := os.ReadDir(installDir) + if err != nil { + t.Fatalf("listing %s: %v", installDir, err) + } + + resolver := images.NewPrebuilt(testSource, newStubRegistry().digest) + + // The directory as a whole is what a plain GKE install applies. + paths := []string{installDir} + for _, entry := range entries { + if !entry.IsDir() && filepath.Ext(entry.Name()) == ".yaml" { + paths = append(paths, filepath.Join(installDir, entry.Name())) + } + } + + for _, p := range paths { + t.Run(filepath.Base(p), func(t *testing.T) { + out, err := resolver.ResolvePath(context.Background(), p) + if err != nil { + t.Fatalf("ResolvePath() = %v", err) + } + if refs := koRefPattern.FindAllString(string(out), -1); len(refs) > 0 { + t.Errorf("unresolved references survived: %v", refs) + } + // Every reference has to name a digest. ActorTemplate images, + // image volume references, and a SandboxConfig's pauseImage all + // carry the CEL rule self.contains('@'), so an unpinned reference + // is not applied at all, it is rejected by admission. + for _, ref := range resolvedRef.FindAllString(string(out), -1) { + if !strings.Contains(ref, "@sha256:") { + t.Errorf("%s is not pinned to a digest", ref) + } + } + if _, err := kube.DecodeManifestBytes(out); err != nil { + t.Errorf("the resolved manifest no longer parses: %v", err) + } + }) + } +} diff --git a/cmd/ate-setup/internal/kube/manifest.go b/cmd/ate-setup/internal/kube/manifest.go index a97f6adb5..f79b2f70b 100644 --- a/cmd/ate-setup/internal/kube/manifest.go +++ b/cmd/ate-setup/internal/kube/manifest.go @@ -75,12 +75,63 @@ func DecodeManifestBytes(data []byte) ([]*unstructured.Unstructured, error) { // comments there call out specific filename-ordering hazards, so recursing or // reordering here would change install behavior. func LoadPath(path string) ([]*unstructured.Unstructured, error) { + files, err := manifestFiles(path) + if err != nil { + return nil, err + } + + var objs []*unstructured.Unstructured + for _, file := range files { + fileObjs, err := loadFile(file) + if err != nil { + return nil, err + } + objs = append(objs, fileObjs...) + } + return objs, nil +} + +// ReadPath concatenates the same files LoadPath would read into a single +// multi-document stream, leaving them unparsed. +// +// This is the input the pre-built image resolver takes, and sharing +// manifestFiles keeps it reading exactly what an apply applies. ko is handed +// the path and enumerates it itself, accepting only .yaml and .json where this +// also accepts .yml; nothing in the tree uses that extension, and a manifest +// that did would reach an apply with its ko:// references unresolved. +func ReadPath(path string) ([]byte, error) { + files, err := manifestFiles(path) + if err != nil { + return nil, err + } + + var out bytes.Buffer + for _, file := range files { + data, err := os.ReadFile(file) + if err != nil { + return nil, fmt.Errorf("while reading %s: %w", file, err) + } + // A separator before every document, including the first: a leading + // empty document decodes to nothing, whereas a file that does not end + // in a newline would otherwise run into the next one. + out.WriteString("---\n") + out.Write(data) + if !bytes.HasSuffix(data, []byte("\n")) { + out.WriteString("\n") + } + } + return out.Bytes(), nil +} + +// manifestFiles lists the files a manifest path covers: the file itself, or +// every manifest directly inside a directory, in lexical order. +func manifestFiles(path string) ([]string, error) { info, err := os.Stat(path) if err != nil { return nil, fmt.Errorf("while reading %s: %w", path, err) } if !info.IsDir() { - return loadFile(path) + return []string{path}, nil } entries, err := os.ReadDir(path) @@ -95,19 +146,10 @@ func LoadPath(path string) ([]*unstructured.Unstructured, error) { if !manifestExtensions[strings.ToLower(filepath.Ext(entry.Name()))] { continue } - names = append(names, entry.Name()) + names = append(names, filepath.Join(path, entry.Name())) } sort.Strings(names) - - var objs []*unstructured.Unstructured - for _, name := range names { - fileObjs, err := loadFile(filepath.Join(path, name)) - if err != nil { - return nil, err - } - objs = append(objs, fileObjs...) - } - return objs, nil + return names, nil } func loadFile(path string) ([]*unstructured.Unstructured, error) { diff --git a/cmd/ate-setup/internal/kube/manifest_test.go b/cmd/ate-setup/internal/kube/manifest_test.go new file mode 100644 index 000000000..d0866aedd --- /dev/null +++ b/cmd/ate-setup/internal/kube/manifest_test.go @@ -0,0 +1,184 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package kube + +import ( + "errors" + "io/fs" + "os" + "path/filepath" + "testing" +) + +// writeFiles populates a temporary directory and returns its path. +func writeFiles(t *testing.T, files map[string]string) string { + t.Helper() + dir := t.TempDir() + for name, body := range files { + path := filepath.Join(dir, name) + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, []byte(body), 0o600); err != nil { + t.Fatal(err) + } + } + return dir +} + +// manifest returns a minimal object whose name identifies it in assertions. +func manifest(name string) string { + return "apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: " + name + "\n" +} + +func names(t *testing.T, data []byte) []string { + t.Helper() + objs, err := DecodeManifestBytes(data) + if err != nil { + t.Fatalf("DecodeManifestBytes() = %v", err) + } + out := make([]string, 0, len(objs)) + for _, obj := range objs { + out = append(out, obj.GetName()) + } + return out +} + +func equal(a, b []string) bool { + if len(a) != len(b) { + return false + } + for i := range a { + if a[i] != b[i] { + return false + } + } + return true +} + +// ReadPath feeds the pre-built image resolver, and its output is applied in +// place of what LoadPath would have applied. The two must therefore cover the +// same documents in the same order: the install depends on that ordering, and +// a divergence would be a silently different install rather than an error. +func TestReadPathMatchesLoadPath(t *testing.T) { + tests := []struct { + name string + files map[string]string + want []string + }{ + { + name: "single file", + files: map[string]string{"a.yaml": manifest("only")}, + want: []string{"only"}, + }, + { + name: "directory is lexical, not filesystem order", + files: map[string]string{ + "20-second.yaml": manifest("second"), + "10-first.yaml": manifest("first"), + "30-third.yaml": manifest("third"), + }, + want: []string{"first", "second", "third"}, + }, + { + name: "non-manifest extensions are skipped", + files: map[string]string{ + "keep.yaml": manifest("keep"), + "keep2.yml": manifest("keep2"), + "keep3.json": `{"apiVersion":"v1","kind":"ConfigMap","metadata":{"name":"keep3"}}`, + "skip.tmpl": manifest("skip"), + "skip.md": "# not a manifest\n", + "skip.yaml.bak": manifest("skip"), + }, + want: []string{"keep", "keep2", "keep3"}, + }, + { + name: "nested directories are not recursed into", + files: map[string]string{ + "top.yaml": manifest("top"), + "nested/deep.yaml": manifest("deep"), + }, + want: []string{"top"}, + }, + { + name: "a file with no trailing newline does not merge into the next", + files: map[string]string{ + "a.yaml": "apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: a", + "b.yaml": manifest("b"), + }, + want: []string{"a", "b"}, + }, + { + name: "an empty file contributes no document", + files: map[string]string{ + "a.yaml": manifest("a"), + "empty.yaml": "", + }, + want: []string{"a"}, + }, + { + name: "multi-document files keep their internal order", + files: map[string]string{ + "a.yaml": manifest("one") + "---\n" + manifest("two"), + }, + want: []string{"one", "two"}, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + dir := writeFiles(t, tc.files) + + // A single-file case addresses the file directly, which is the + // shape the manifest-path call sites use. + path := dir + if len(tc.files) == 1 { + for name := range tc.files { + path = filepath.Join(dir, name) + } + } + + data, err := ReadPath(path) + if err != nil { + t.Fatalf("ReadPath() = %v", err) + } + got := names(t, data) + if !equal(got, tc.want) { + t.Errorf("ReadPath() objects = %v, want %v", got, tc.want) + } + + objs, err := LoadPath(path) + if err != nil { + t.Fatalf("LoadPath() = %v", err) + } + loaded := make([]string, 0, len(objs)) + for _, obj := range objs { + loaded = append(loaded, obj.GetName()) + } + if !equal(loaded, got) { + t.Errorf("LoadPath() objects = %v, ReadPath() objects = %v; want identical", loaded, got) + } + }) + } +} + +// The manifest-path call sites hand ReadPath a path that may not exist on a +// teardown, and DeletePath distinguishes that from a real failure by checking +// for fs.ErrNotExist. Keep the wrapping transparent to errors.Is. +func TestReadPathMissing(t *testing.T) { + if _, err := ReadPath(filepath.Join(t.TempDir(), "absent.yaml")); !errors.Is(err, fs.ErrNotExist) { + t.Errorf("ReadPath(absent) = %v, want a not-exist error", err) + } +} diff --git a/cmd/ate-setup/internal/steps/csi.go b/cmd/ate-setup/internal/steps/csi.go index f680c9d66..a32a26504 100644 --- a/cmd/ate-setup/internal/steps/csi.go +++ b/cmd/ate-setup/internal/steps/csi.go @@ -75,7 +75,7 @@ func (e *Env) SetupCSI(ctx context.Context) error { if err := e.EnsurePodCertificateCAs(ctx); err != nil { return err } - if err := e.KoApply(ctx, e.Cfg.Manifest("pod-certificate-controller.yaml")); err != nil { + if err := e.ResolveAndApply(ctx, e.Cfg.Manifest("pod-certificate-controller.yaml")); err != nil { return err } if err := e.applyPodcertWorkersOverride(ctx); err != nil { diff --git a/cmd/ate-setup/internal/steps/deploy.go b/cmd/ate-setup/internal/steps/deploy.go index d6c0d9232..fbe5e7013 100644 --- a/cmd/ate-setup/internal/steps/deploy.go +++ b/cmd/ate-setup/internal/steps/deploy.go @@ -85,7 +85,7 @@ func (e *Env) DeployAteSystem(ctx context.Context, opts DeployOptions) error { // The podcertificate controller goes first so it starts signing and // publishing trust bundles immediately. - if err := e.KoApply(ctx, e.Cfg.Manifest("pod-certificate-controller.yaml")); err != nil { + if err := e.ResolveAndApply(ctx, e.Cfg.Manifest("pod-certificate-controller.yaml")); err != nil { return err } if err := e.applyPodcertWorkersOverride(ctx); err != nil { @@ -228,7 +228,7 @@ func (e *Env) DeployAteAPIServer(ctx context.Context) error { if err := e.applyOtelConfig(ctx); err != nil { return err } - if err := e.KoApply(ctx, e.Cfg.Manifest("ate-api-server.yaml")); err != nil { + if err := e.ResolveAndApply(ctx, e.Cfg.Manifest("ate-api-server.yaml")); err != nil { return err } return e.Kube.RolloutStatus(ctx, kube.KindDeployment, NamespaceAteSystem, "ate-api-server", e.Cfg.RolloutTimeout) @@ -247,7 +247,7 @@ func (e *Env) DeployAteController(ctx context.Context) error { if err := e.applyOtelConfig(ctx); err != nil { return err } - if err := e.KoApply(ctx, e.Cfg.Manifest("ate-controller.yaml")); err != nil { + if err := e.ResolveAndApply(ctx, e.Cfg.Manifest("ate-controller.yaml")); err != nil { return err } return e.Kube.RolloutStatus(ctx, kube.KindDeployment, NamespaceAteSystem, "ate-controller", e.Cfg.RolloutTimeout) @@ -276,7 +276,7 @@ func (e *Env) DeployAtelet(ctx context.Context) error { // The kind overlay patches the DaemonSet for the local node layout. manifest, err = e.KustomizeResolve(ctx, installDir+"/kind/atelet") } else { - manifest, err = e.KoResolve(ctx, e.Cfg.Manifest("atelet.yaml")) + manifest, err = e.ResolveManifest(ctx, e.Cfg.Manifest("atelet.yaml")) } if err != nil { return err @@ -322,7 +322,7 @@ func (e *Env) DeployAtenet(ctx context.Context) error { if err := e.applyAtenetEgress(ctx); err != nil { return err } - if err := e.KoApply(ctx, e.Cfg.Manifest("atenet-dns.yaml")); err != nil { + if err := e.ResolveAndApply(ctx, e.Cfg.Manifest("atenet-dns.yaml")); err != nil { return err } @@ -359,7 +359,7 @@ func (e *Env) EnsureCRDs(ctx context.Context) error { // DeployCRDs applies the generated CRDs and RBAC, waiting for them to reach Established condition. func (e *Env) DeployCRDs(ctx context.Context) error { log.Step("deploy_crds") - if err := e.KoApply(ctx, e.Cfg.Manifest("generated")); err != nil { + if err := e.ResolveAndApply(ctx, e.Cfg.Manifest("generated")); err != nil { return err } for _, name := range ateCRDs { diff --git a/cmd/ate-setup/internal/steps/env.go b/cmd/ate-setup/internal/steps/env.go index 2fc03aa5a..d91ba3efe 100644 --- a/cmd/ate-setup/internal/steps/env.go +++ b/cmd/ate-setup/internal/steps/env.go @@ -23,9 +23,11 @@ import ( "time" "github.com/agent-substrate/substrate/cmd/ate-setup/internal/config" + "github.com/agent-substrate/substrate/cmd/ate-setup/internal/images" "github.com/agent-substrate/substrate/cmd/ate-setup/internal/ko" "github.com/agent-substrate/substrate/cmd/ate-setup/internal/kube" "github.com/agent-substrate/substrate/cmd/ate-setup/internal/kustomize" + "github.com/agent-substrate/substrate/cmd/ate-setup/internal/log" ) // Timeouts carried over from the --timeout values in the shell installer. @@ -48,15 +50,24 @@ const ( NamespacePodCert = "podcertificate-controller-system" ) +// imageResolver turns a manifest whose image references are ko:// import paths +// into one whose references are pullable. *ko.Runner builds and publishes them +// from source; *images.Prebuilt maps them onto an already-published release. +type imageResolver interface { + ResolvePath(ctx context.Context, path string) ([]byte, error) + ResolveBytes(ctx context.Context, manifest []byte) ([]byte, error) +} + // Env is the shared execution context for every step: resolved configuration // plus the clients needed to act on the cluster. type Env struct { Cfg *config.Config Kube *kube.Client - // ko is created lazily. Steps that only apply static manifests must work - // on a machine with no container registry credentials configured. - ko *ko.Runner + // resolver is created lazily. Steps that only apply static manifests must + // work on a machine with no container registry credentials configured, and + // the build-from-source resolver fails at construction without them. + resolver imageResolver // substrateVersion caches the derived build version and its object-name // suffix; see SubstrateVersion. @@ -73,51 +84,59 @@ func NewEnv(cfg *config.Config) (*Env, error) { return &Env{Cfg: cfg, Kube: client}, nil } -// Ko returns the ko runner, creating it on first use. -func (e *Env) Ko() (*ko.Runner, error) { - if e.ko != nil { - return e.ko, nil +// imageResolver returns the image resolver, creating it on first use. +func (e *Env) imageResolver() (imageResolver, error) { + if e.resolver != nil { + return e.resolver, nil } + + log.Stepf("images: %s", e.Cfg.Images.Describe()) + if e.Cfg.Images.IsPrebuilt() { + e.resolver = images.NewPrebuilt(e.Cfg.Images, images.RemoteDigest) + return e.resolver, nil + } + runner, err := ko.New(e.Cfg.Root, e.Cfg.KoEnv()) if err != nil { return nil, err } - e.ko = runner - return e.ko, nil + e.resolver = runner + return e.resolver, nil } -// KoApply resolves a manifest path with ko and applies the result. This is the -// run_ko apply of the shell scripts, split into its two real steps. -func (e *Env) KoApply(ctx context.Context, path string) error { - manifest, err := e.KoResolve(ctx, path) +// ResolveAndApply resolves the images in a manifest path and applies the +// result. This is the run_ko apply of the shell scripts, split into its two +// real steps. +func (e *Env) ResolveAndApply(ctx context.Context, path string) error { + manifest, err := e.ResolveManifest(ctx, path) if err != nil { return err } return e.Kube.ApplyBytes(ctx, manifest) } -// KoResolve builds and publishes the images in a manifest path, returning the -// digest-pinned manifest. -func (e *Env) KoResolve(ctx context.Context, path string) ([]byte, error) { - runner, err := e.Ko() +// ResolveManifest turns the ko:// references in a manifest path into pullable +// image references. +func (e *Env) ResolveManifest(ctx context.Context, path string) ([]byte, error) { + resolver, err := e.imageResolver() if err != nil { return nil, err } - return runner.ResolvePath(ctx, path) + return resolver.ResolvePath(ctx, path) } -// KoResolveBytes resolves an in-memory manifest, such as kustomize output. -func (e *Env) KoResolveBytes(ctx context.Context, manifest []byte) ([]byte, error) { - runner, err := e.Ko() +// ResolveManifestBytes resolves an in-memory manifest, such as kustomize output. +func (e *Env) ResolveManifestBytes(ctx context.Context, manifest []byte) ([]byte, error) { + resolver, err := e.imageResolver() if err != nil { return nil, err } - return runner.ResolveBytes(ctx, manifest) + return resolver.ResolveBytes(ctx, manifest) } -// KoApplyBytes resolves an in-memory manifest with ko and applies the result. -func (e *Env) KoApplyBytes(ctx context.Context, manifest []byte) error { - resolved, err := e.KoResolveBytes(ctx, manifest) +// ResolveAndApplyBytes resolves an in-memory manifest and applies the result. +func (e *Env) ResolveAndApplyBytes(ctx context.Context, manifest []byte) error { + resolved, err := e.ResolveManifestBytes(ctx, manifest) if err != nil { return err } @@ -129,14 +148,14 @@ func (e *Env) Kustomize(overlay string) ([]byte, error) { return kustomize.Build(e.Cfg.Path(overlay)) } -// KustomizeResolve renders an overlay and pipes it through ko, the +// KustomizeResolve renders an overlay and resolves its images, the // `kubectl kustomize ... | run_ko resolve -f -` pipeline. func (e *Env) KustomizeResolve(ctx context.Context, overlay string) ([]byte, error) { built, err := e.Kustomize(overlay) if err != nil { return nil, err } - return e.KoResolveBytes(ctx, built) + return e.ResolveManifestBytes(ctx, built) } // EnsureAteSystemNamespace applies the ate-system namespace manifest and waits diff --git a/cmd/ate-setup/internal/steps/overlay.go b/cmd/ate-setup/internal/steps/overlay.go index 8edf9fdfc..166212841 100644 --- a/cmd/ate-setup/internal/steps/overlay.go +++ b/cmd/ate-setup/internal/steps/overlay.go @@ -51,7 +51,7 @@ func (e *Env) renderSystemManifests(ctx context.Context) ([]byte, error) { if overlay := SystemOverlay(e.Cfg); overlay != "" { return e.KustomizeResolve(ctx, overlay) } - return e.KoResolve(ctx, e.Cfg.Manifest()) + return e.ResolveManifest(ctx, e.Cfg.Manifest()) } // renderAtenetRouterManifest produces the atenet router manifest for the @@ -60,7 +60,7 @@ func (e *Env) renderAtenetRouterManifest(ctx context.Context) ([]byte, error) { if e.Cfg.Router == config.RouterAgentgateway { return e.KustomizeResolve(ctx, installDir+"/agentgateway-router") } - return e.KoResolve(ctx, e.Cfg.Manifest("atenet-router.yaml")) + return e.ResolveManifest(ctx, e.Cfg.Manifest("atenet-router.yaml")) } // atenetEgressManifestPath returns the egress manifest path based on configuration. @@ -85,10 +85,10 @@ func (e *Env) renderAtenetEgressManifest(ctx context.Context) ([]byte, error) { if err != nil { return nil, err } - return e.KoResolveBytes(ctx, patched) + return e.ResolveManifestBytes(ctx, patched) } - return e.KoResolve(ctx, e.atenetEgressManifestPath()) + return e.ResolveManifest(ctx, e.atenetEgressManifestPath()) } func (e *Env) patchAtenetEgressManifest() ([]byte, error) { diff --git a/cmd/ate-setup/internal/steps/postgres.go b/cmd/ate-setup/internal/steps/postgres.go index a6c566b86..b5fbf0e93 100644 --- a/cmd/ate-setup/internal/steps/postgres.go +++ b/cmd/ate-setup/internal/steps/postgres.go @@ -37,7 +37,7 @@ func (e *Env) DeployPostgres(ctx context.Context) error { // The StatefulSet's projected serving certificate is issued by this // controller. Applying it here makes `deploy postgres` usable on a fresh // cluster as well as after `deploy ate-system`. - if err := e.KoApply(ctx, e.Cfg.Manifest("pod-certificate-controller.yaml")); err != nil { + if err := e.ResolveAndApply(ctx, e.Cfg.Manifest("pod-certificate-controller.yaml")); err != nil { return err } if err := e.applyPodcertWorkersOverride(ctx); err != nil { diff --git a/cmd/ate-setup/internal/steps/version.go b/cmd/ate-setup/internal/steps/version.go index 8d8ef5e91..00ebbaee2 100644 --- a/cmd/ate-setup/internal/steps/version.go +++ b/cmd/ate-setup/internal/steps/version.go @@ -17,6 +17,7 @@ package steps import ( "context" "fmt" + "os" "regexp" "strings" "time" @@ -31,11 +32,28 @@ import ( // SubstrateVersion returns the build version and the object-name suffix for atelet. // The version is the same one ko stamps into the binaries. +// +// With --image-repo nothing is built, so `git describe` would report the +// checkout rather than the images being installed. The image tag is the version +// in that case. It has to be: the version names the atelet DaemonSet and sets +// the node label that partitions nodes across coexisting versions, so it must +// describe the atelet that is actually running, which came from the image. +// VERSION still wins over both, so a tag that is not a valid label value can be +// worked around the same way. func (e *Env) SubstrateVersion() (version, suffix string, err error) { if e.substrateVersion != "" { return e.substrateVersion, e.substrateVersionSuffix, nil } - raw := ko.BuildVersion(e.Cfg.Root) + var raw string + if e.Cfg.Images.IsPrebuilt() { + raw = os.Getenv("VERSION") + if raw == "" { + raw = e.Cfg.Images.Tag + } + } else { + // ko.BuildVersion consults VERSION itself before `git describe`. + raw = ko.BuildVersion(e.Cfg.Root) + } if v := versionlabel.Value(raw); v != raw { return "", "", fmt.Errorf("build version %q is not a valid label value (it would sanitize to %q); pin a label-safe one with the VERSION env var", raw, v) } diff --git a/cmd/ate-setup/internal/steps/version_test.go b/cmd/ate-setup/internal/steps/version_test.go index f7154d755..3391eb176 100644 --- a/cmd/ate-setup/internal/steps/version_test.go +++ b/cmd/ate-setup/internal/steps/version_test.go @@ -14,7 +14,49 @@ package steps -import "testing" +import ( + "testing" + + "github.com/agent-substrate/substrate/cmd/ate-setup/internal/config" + "github.com/agent-substrate/substrate/cmd/ate-setup/internal/images" +) + +// TestSubstrateVersionPrebuilt covers the version a prebuilt install stamps. +// Root points at a directory that is not a checkout, so `git describe` has +// nothing to say and only the image tag can supply it. +func TestSubstrateVersionPrebuilt(t *testing.T) { + t.Setenv("VERSION", "") + + cases := []struct { + name string + source images.Source + want string + wantSuffix string + }{{ + name: "tag is the version", + source: images.Source{Repo: "example.com/substrate", Tag: "v0.0.0-503-4b3423c0"}, + want: "v0.0.0-503-4b3423c0", + wantSuffix: "v0-0-0-503-4b3423c0", + }, { + name: "built from source falls back to git", + source: images.Source{}, + want: "dev", + wantSuffix: "dev", + }} + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + e := &Env{Cfg: &config.Config{Root: t.TempDir(), Images: tc.source}} + got, suffix, err := e.SubstrateVersion() + if err != nil { + t.Fatalf("SubstrateVersion: %v", err) + } + if got != tc.want || suffix != tc.wantSuffix { + t.Fatalf("SubstrateVersion = %q, %q; want %q, %q", got, suffix, tc.want, tc.wantSuffix) + } + }) + } +} func TestSubstituteVersion(t *testing.T) { e := &Env{substrateVersion: "v1.2.3", substrateVersionSuffix: "v1-2-3"}