substrate/cmd/ate-setup/differences.md

ate-setup vs. the install shell scripts

ate-setup is a Go port of hack/install-ate.sh, the seven hack/install-demo-*.sh scripts it sourced, and hack/setup-csi-*-kind.sh.

The port has replaced them. The demo scripts are deleted, and hack/install-ate.sh is a shim holding no install logic of its own: it translates the flags and environment variables the installer has always accepted onto ate-setup commands, so existing command lines keep working. See commands.md for the flag-by-flag mapping.

This document covers what changed behind that mapping: the design decisions of the port, against the shell it replaced. For what a user of hack/install-ate.sh can observe after the switch, see cli-diff.md. For anything listed in neither, the port is intended to be behavior-preserving.

What is deliberately unchanged

These were treated as contracts and reproduced exactly:

• Step log lines. log.Step prints the same cyan [step]: name and the same step names (deploy_ate_system, create_api_server_env_vars, demo-counter_deploy), so CI log scrapers keep working. Two lines lost a parenthesized qualifier; see cli-diff.md.
• Manifest ordering. Applying a directory is non-recursive and lexical, as kubectl apply -f <dir> was. delete_ate_system depends on that ordering and the shell comments called out specific filename hazards, so kube.LoadPath keeps it.
• Overlay selection. steps.SystemOverlay is the same product of kind × router that render_ate_system_manifests computed with nested ifs, including the plain GKE install rendering the base kustomization rather than the raw directory, which would re-apply the podcertificate-controller and undo its size10 flags and worker count.
• Timeouts. 60s namespace, 60s rollout (--rollout-timeout / ATE_INSTALL_ROLLOUT_TIMEOUT, as in the scripts), 120s for the podcertificate-controller and CSI waits the scripts fixed there, 300s for the trust bundles (one deadline shared across both), 300s demo. --rollout-timeout now reaches the 120s and 300s waits too, which it did not in the shell, but only when it is passed: Config.WaitTimeout leaves each site at its historical value otherwise, so the 60s default cannot shorten the slow bootstrap paths.
• Rendered bytes. authentication.yaml is trimmed of its trailing newline because the shell built it inside $(...), which strips them. Installing over a shell-installed cluster must not rewrite the ConfigMap.
• ko's ldflags. make ldflags emitted -X=<version pkg>.Version=$(git describe --tags --always --dirty); ko.Runner.ldflags computes the identical string without depending on make.

Execution model

	shell	ate-setup
Actions per run	many, one per flag, in command-line order	exactly one subcommand
Argument errors	detected when the dispatch loop reaches the flag, after earlier actions already ran	rejected by cobra before anything runs
Value flags	pre-scanned so they could appear anywhere	positional, per-command, standard flag parsing
Configuration	env vars (ATE_INSTALL_KIND, ATE_ATENET_DATAPLANE, KUBECTL_CONTEXT, …)	flags, with the env vars still honored as defaults
Repository root	git rev-parse --show-toplevel, then cd	walk up for go.mod; no chdir, all paths absolute

The one-action-per-run change is the most visible: a line that passed --deploy-ate-system --deploy-demo-counter becomes two ate-setup calls. That is what the shim does with it, so the combined form keeps working.

Invalid input now fails before any cluster mutation. --atenet-dataplane=nginx used to be caught by a pre-scan validation pass; --worker-count 0 was not caught at all and surfaced from inside deploy_locust.sh after the microvm dependencies had already been installed. --setup-csi is checked the same way: an unknown driver is rejected during argument validation, before the configuration is even loaded, and --setup-csi=hostpath against a cluster that is not Kind is refused before deploy ate-system installs anything.

.ate-dev-env.sh is still sourced through bash — a developer's file can contain arbitrary shell — but its variables are then layered under the process environment and the flags rather than being exported wholesale.

External binaries

Dropped entirely: kubectl, kubectl kustomize, jq, openssl, sed, base64, grep, make, and go run ./cmd/kubectl-ate.

Still required, and why:

Binary	Used for
ko	building and publishing images (ko resolve only); not used with --image-repo
go	locating the pinned ko tool, exactly as hack/run-tool.sh does; not used with --image-repo
git	the git describe version stamp passed to ko; not used with --image-repo
docker	the kind CSI setup (docker exec into the node) and the claude-code-multiplex workload build
gcloud	GKE get-credentials, only when PROJECT_ID is set and no context was given
bash	sourcing .ate-dev-env.sh

Two shell scripts are still invoked rather than reimplemented, because they orchestrate image builds, asset assembly, and object-store staging that are out of scope for an installer: benchmarking/deploy_locust.sh and hack/install-microvm-deps.sh. They receive Config.ScriptEnv(), which reconstructs the environment the shell installer would have exported to them.

ko is no longer asked to apply anything. The scripts ran run_ko apply, which made ko shell out to kubectl and forced the awkward -- --context= special case (only apply/create/delete/run accept args after --; resolve rejects them). Now ko only resolves, and the resulting manifest is applied through client-go.

Image sources

The shell installer had exactly one: build every image from the checkout with ko and push it to KO_DOCKER_REPO. That is still the default and is unchanged. --image-repo adds a second, in which nothing is built.

Manifests carry ko://<import path> references either way. With --image-repo each is looked up in images.Components and rewritten to <repo>/<base of the import path>:<tag>, which is ko's own --base-import-paths naming and so is how the release images are already published — ateapi, atelet, atenet, ateom-gvisor, and the rest. --image-tag supplies the tag. Each of the two requires the other: a tag with no repository to pull from would be dropped, leaving a build from source that looks like the release the tag names.

The rewritten reference then keeps that tag and adds the digest it names, as <repo>/<image>:<tag>@sha256:..., which is the shape ko itself produces for a tagged release. Resolving the digest is one HEAD request per image, cached for the install, so --image-repo needs read access to the registry from wherever ate-setup runs rather than only from the cluster. It is not optional: an ActorTemplate's container image, an image volume's reference, and a SandboxConfig's pauseImage each carry the CEL rule self.contains('@'), so an unpinned reference is rejected by admission and every demo fails to deploy. Pinning also gives a pre-built install the property a ko install had for free, that a tag moving afterwards cannot change what is running.

That tag also becomes the substrate version, which names the atelet DaemonSet and sets the node label partitioning nodes across coexisting versions. It has to: that label must describe the atelet actually running, which came from the image, not the checkout git describe happens to be sitting on. VERSION overrides it, as it always has.

The rewrite is textual, one token at a time, matching SubstituteVersion above. A reference is replaced wherever it appears, so the code needs no list of which fields hold images — workerImage: in a CRD spec is rewritten like any other — and the applied manifest differs from the checked-in YAML only in the image references.

Every reference must be in the list. A reference outside the module path, one carrying an unexpanded ${PLACEHOLDER}, and one naming a package that is not installable all fail the same way — an error naming every unmapped reference at once, rather than an unpullable image reaching the cluster and failing as ImagePullBackOff twenty minutes later. Matching whole references rather than replacing the listed ones is what makes that hold: a reference extending a listed package would otherwise inherit its image. images.Components is checked against the real manifests by a test, so adding a component fails a test rather than an install — though only the list is checked that way. The release also has to publish the new image under the same repository and tag as the rest, and until it does, the component can only be installed by building it.

Cluster access

Server-side apply. kubectl apply -f - is a client-side apply by default. ate-setup uses SSA through the dynamic client with field manager ate-setup and force: true. This is the supported path for repeated reconciliation and avoids the last-applied-configuration annotation growing without bound on the large generated CRDs. Objects previously applied client-side will show both managers in managedFields until the next apply reconciles them.

Discovery caching is a new failure mode. Every kubectl invocation started with a cold discovery cache, so kubectl apply -f generated followed by applying a SandboxConfig just worked. One long-lived process keeps a cached RESTMapper, which would keep serving the pre-CRD discovery document and report no matches for kind. Three mitigations are in place: Apply invalidates after a CRD document in the same stream, DeployCRDs invalidates explicitly, and resourceFor retries once against fresh discovery before giving up.

Rollout status is reimplemented. kubectl rollout status is not a library call, so kube.RolloutStatus polls Deployments, DaemonSets, and StatefulSets with the same conditions kubectl's status viewers use — including the ProgressDeadlineExceeded check and the partitioned-StatefulSet case. Two deliberate differences: a workload that does not exist yet is waited for rather than erroring immediately, and the failure message carries the last observed status (3/5 replicas available) instead of only a timeout.

Deletes are more precise. kubectl delete --ignore-not-found -f was the model, so NotFound is ignored — and so is a kind that no longer resolves, since teardown after the CRDs are gone must not fail. Beyond that, deletes are strict.

Cluster profiles are new. --cluster-size and --cordon-control-plane postdate the shell installer, which the shim forwards them to. size10 renders the podcert-size10 overlay and resizes the bundled PostgreSQL from postgres-size10/postgres-config-patch.yaml; --cordon-control-plane composes the cordon-control-plane component over every control plane apply path. The size10 PostgreSQL changes are made to the decoded objects before the one server-side apply, not patched on afterwards: one rollout, the rollout wait sees the resized pod, and there is no second field manager for a later apply to fight with.

Timing lines are new. ate-setup prints (kubectl apply took 1.234s) after each apply, delete, rollout wait, trust bundle wait, ko invocation, and gcloud call, on stdout with the rest of the progress output.

|| true is gone. The CSI hostpath bundle ships a VolumeSnapshotClass whose CRD is absent on a stock Kind cluster; the shell script handled that by ignoring kubectl's exit code for the whole bundle, which also hid real failures. ApplyTolerant skips exactly the objects whose kind is unmappable, logs each skip, and keeps everything else strict. The best-effort docker exec cleanup on the Kind node stays best-effort, as before.

Certificates and secrets

CA pools are generated in-process. create_*_ca_pool_secret shelled out to go run ./cmd/kubectl-ate admin make-ca-pool, paying a compile per call. steps.createCAPool calls internal/localca and internal/localjwtauthority directly — the same libraries kubectl-ate uses.

Creating an existing pool is no longer an error. make-ca-pool issues a bare Create, so --create-actor-id-ca-pool-secret against a cluster that already had one failed with AlreadyExists. ate-setup create actor-id-ca-pool logs already exists; keeping it and succeeds. It still refuses to overwrite: regenerating would rotate the root out from under every certificate already issued from it. (The --deploy-ate-system path was already guarded by ensure_apiserver_prerequisites, so only the standalone create subcommands change.)

Pool extraction is exact. ca_pool_root_pem piped a Secret through jsonpath | base64 -d | grep -o '"RootCertificateDER":"[^"]*' | sed | base64 -d | openssl x509. That has two problems the shell comments were already worried about: grep -o returns only the first root, silently dropping the rest of a multi-CA pool, and any failure in the pipeline yields an empty string that becomes a trust bundle with no roots. kube.CAPoolRootPEM unmarshals the pool with the library that wrote it, emits every CA's root, and returns an error for a missing key, an undecodable pool, an empty pool, or a CA with no root certificate.

Demos

A registry replaces naming conventions. The scripts appended to an ATE_DEMOS array and discovered ${name}_deploy, ${name}_delete, ${name}_usage, and ${name}_cmdline with declare -F. A demo that misspelled a function silently lost that capability. demos.Demo is an interface, each demo is its own package registering itself from init, and internal/demos/all is the single list of what gets linked in — with a test asserting no demo package is missing from it.

Kind-only demos are visible but refuse. demo-autoscaled-workerpool registered itself only when ATE_INSTALL_KIND=true, so on GKE --deploy-demo-autoscaled-workerpool was rejected as an unknown option. The cobra tree is built before flags are parsed, so --kind is not known yet; the subcommand always exists and fails at run time with demo-autoscaled-workerpool is only supported for Kind installations; re-run with --kind. delete all still skips it on a non-Kind install, via demos.For.

Template expansion is not sed. The demo templates were expanded by a stack of sed -e expressions doing two things: substitute ${PLACEHOLDER}, or delete the whole line (/${NAME}/d) when the placeholder is not in play. internal/render implements exactly those two operations, so the output is byte-identical, but values are no longer interpreted as sed replacement text — an ANTHROPIC_API_KEY or bucket name containing |, &, or a backslash is now substituted literally. A placeholder that is neither substituted nor dropped is left in place, so an unexpected template variable surfaces as invalid YAML instead of vanishing.

Actor cleanup uses the API directly. delete_demo_actors required jq, listed actors with kubectl-ate get actors -A -o json, and filtered with a jq expression. steps.DeleteSubstrateDemo uses internal/ateclient and pages through ListActors. The tolerances are preserved: no ate-api-server, or an apiserver that cannot be reached, skips cleanup rather than failing, because delete all runs this for every demo.

docker buildx imagetools inspect output is parsed as JSON, not piped through jq -r '.manifest.digest'. Build output still goes to stderr so the stdout/stderr split matches under CI log capture.

Errors and diagnostics

• set -o errexit propagated failures but could not see a failing command substitution inside an argument list — the case the shell comments flagged twice around CA extraction. Go returns errors at each call site.
• Errors are wrapped with the object they concern: while applying deployment/ate-api-server -n ate-system: ....
• Warnings go to stderr with a Warning: prefix; step and progress output goes to stdout, as before.
• ate-setup with no subcommand prints the help text and exits 0, without loading config or touching the cluster; the script printed usage and exited 1. Either way nothing is installed implicitly — the subcommand naming the setup to run is always explicit.

Known differences worth flagging

--setup-csi on a non-Kind cluster. nfs is accepted off Kind, as it was — only the hostpath plugin is patched for the single-node Kind layout — but hostpath and both are a hard error there rather than the shell's old warn-and-continue.

The env-hash digest does not agree with the shell's. Both stamp ate.dev/env-hash on the ate-api-server pod template so that a changed environment rolls the Deployment, but the shell computed it with openssl dgst -sha256 over jsonpath output and ate-setup digests the ConfigMap and Secret contents directly. The value is opaque — only changes to it matter — so the only consequence is that the first ate-setup install over a shell-installed cluster rolls ate-api-server once.

Cloud SQL DSNs are synthesized, not adopted verbatim. With ATE_API_POSTGRES_CLOUDSQL_INSTANCE set and no explicit ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING, the DSN is built passwordless against the proxy's loopback listener from the GSA's email (cloud-sql.md), so ATE_API_POSTGRES_CLOUDSQL_IAM_AUTH=false is rejected rather than producing a DSN that cannot authenticate. The owner connection uses the same IAM login unless ATE_API_POSTGRES_OWNER_CONNECTION_STRING is set. An unset instance variable is distinct from an exported empty one: unset adopts whatever the cluster already records in ate-api-server-envvars, including the IP type and the GSA from the iam.gke.io/gcp-service-account annotation, so a redeploy from a shell that never exported the variables leaves a working proxy alone. Exporting it empty removes the sidecar and the annotation and falls back to the bundled StatefulSet.

Testing

The shell installer had no tests. cmd/ate-setup has unit tests for template rendering, overlay selection, config resolution, the authentication config, the apiserver environment ConfigMap and Secret, Cloud SQL resolution, the OTLP endpoint override, delegated script arguments, manifest deletion, per-demo rendering, image reference rewriting, kustomize composition, the control plane pinning on every apply path, and the size10 PostgreSQL resize against the real manifests.