substrate/docs/upgrade.md

# Rolling upgrade runbook

This runbook upgrades a running Agent Substrate install to a new build
version, one node at a time. No actor loses state. On a cluster with
more than one node, at most one node's worth of capacity is out of
service while the rest of the fleet keeps serving; a single-node cluster
stops fully during step 5. It needs `kubectl`, `kubectl ate`, `go run
./cmd/ate-setup`, `jq`, and `grpcurl`. The numbered steps are the same
on every Kubernetes provider; the provider-specific parts sit in their
own sections, [before](#on-gke) and [after](#after-the-roll-on-gke)
the roll. All of the roll's state lives in cluster objects, so you can
stop at any point and pick up again. [Check your
progress](#check-your-progress) tells you where you left off.

The order is `ate-controller` first, then the dataplane, then the rest
of the control plane. The controller goes first because it manages
the worker pools the roll creates. The dataplane goes before the rest
of the control plane so that, by the time ate-api-server and atenet
change, every atelet and worker already understands requests from
either version of the control plane.

## Check your progress

Come back here after a break, and before you touch anything when
something looks wrong. Three commands say where the roll stands:

```bash
# One DaemonSet: only the old dataplane is installed, and its label
# value is the $OLD_VERSION you are upgrading from. Two: step 3 is done,
# and the other label value is the version you are upgrading to.
kubectl get ds -n ate-system -l app=atelet -L ate.dev/substrate-version

# Nodes per version. All on the old one: the per-node roll has not
# started. A mix: it is under way. All on the new one: it is done.
kubectl get nodes -L ate.dev/substrate-version --no-headers \
  | awk '{print $NF}' | sort | uniq -c

# Worker pools. A clone next to each serving pool: step 4 is done.
kubectl get workerpools -A
```

The commands above do not show whether steps 1, 2 and 6 have run.
Those steps are idempotent, so run them again if you are not sure.

## Before you start

### Confirm before you begin

Every item has to hold. Nothing in the roll stops you if one does not.

- [ ] Every node carries the `ate.dev/substrate-version` label, all
  with the same value. That value is `$OLD_VERSION`.

  ```bash
  kubectl get nodes -L ate.dev/substrate-version
  ```

- [ ] The atelet DaemonSet name carries a version suffix. If it does
  not, the cluster was installed from an older build and needs a fresh
  install instead; its DaemonSet selector cannot be changed in place.

  ```bash
  kubectl get ds -n ate-system -l app=atelet
  ```

- [ ] Every serving pool is pinned to `$OLD_VERSION`. An unpinned pool
  cannot take part in the roll.

  ```bash
  kubectl get workerpools -A \
    -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,PIN:.spec.template.nodeSelector.ate\.dev/substrate-version'
  ```

  Pin a pool whose `PIN` column is `<none>` while no actor is assigned
  to its workers, because the edit re-renders the pool's Deployment
  (see [the warnings](#three-things-that-break-an-upgrade)):

  ```bash
  kubectl -n $NS patch workerpool $OLD_WORKERPOOL --type merge \
    -p "spec: {template: {nodeSelector: {ate.dev/substrate-version: '$OLD_VERSION'}}}"
  ```

- [ ] Every serving pool is healthy: `READY` equals `DESIRED`.

  ```bash
  kubectl get workerpools -A
  ```

- [ ] Nothing drains worker nodes on its own: node auto-upgrade is off
  on every pool that runs workers and none of them is spot or
  preemptible, as the
  [Create Cluster warning](../tools/setup-gcp/README.md#2-create-cluster)
  requires.

- [ ] The installed ate-api-server serves `DrainWorker`. Draining a
  worker is one `DrainWorker` RPC, and step 5 calls it with
  [grpcurl](https://github.com/fullstorydev/grpcurl). Open access to
  the Control API now and keep the port-forward running; step 5 uses
  the same `/tmp/ate-ca.pem` and `$TOKEN`.

  ```bash
  kubectl -n ate-system port-forward svc/api 8443:443 >/dev/null 2>&1 &

  kubectl get clustertrustbundles -l podcert.ate.dev/canarying=live \
    -o jsonpath='{range .items[?(@.spec.signerName=="servicedns.podcert.ate.dev/identity")]}{.spec.trustBundle}{end}' \
    > /tmp/ate-ca.pem

  TOKEN=$(kubectl -n ate-system create token ate-client \
    --audience=api.ate-system.svc --duration=48h)

  # Must print ateapi.Control.DrainWorker. If it does not, the cluster
  # predates the versioned builds and needs a fresh install.
  grpcurl -cacert /tmp/ate-ca.pem -authority api.ate-system.svc \
    -H "authorization: Bearer ${TOKEN}" 127.0.0.1:8443 list ateapi.Control \
    | grep DrainWorker
  ```

### Names used throughout

| name | what it is | how to get it |
|---|---|---|
| `$OLD_VERSION` | the version label value you are upgrading from | the one atelet DaemonSet's label, see [Check your progress](#check-your-progress) |
| `$NEW_VERSION` | the version label value you are upgrading to | the second DaemonSet's label once step 3 has run |
| `$NS`, `$OLD_WORKERPOOL` | namespace and name of a serving WorkerPool | `kubectl get workerpools -A` |
| `$NEW_WORKERPOOL` | the clone's name | you pick it in step 4, for example `counter-v2` |
| `$NODE` | the node being rolled | picked per iteration in step 5 from `kubectl get nodes` |

A cluster usually serves more than one WorkerPool, and every serving
pool moves in the same upgrade. Step 4 clones each of them, the
per-node roll in step 5 covers all pools on a node together, and the
retire at the end deletes each old pool. Where the runbook says
`$OLD_WORKERPOOL`, read "each serving pool".

### Checkout and environment

Every `go run ./cmd/ate-setup` command below runs from a checkout of
the new release, with the environment and flags the install used:
`PROJECT_ID`, `CLUSTER_NAME` and `CLUSTER_LOCATION` (or `--context`),
and either `KO_DOCKER_REPO` for a build from source or
`--image-repo`/`--image-tag` for prebuilt images. Keep a checkout of
the old release too as rollback runs the same commands from it.

`ate-setup` takes the version from `$VERSION` when it is set, else
from `git describe` on the checkout for a build from source, or from
`--image-tag` (`ATE_IMAGE_TAG`) for prebuilt images. That value must come out different
from `$OLD_VERSION`, otherwise dataplane upgrade rolls the running atelet in place
instead of adding a second DaemonSet. A tagged checkout differs by
construction; if the install pinned `VERSION`, pin a new value now
and keep it for every command of this upgrade.

Install the new `kubectl ate` with `go install ./cmd/kubectl-ate`.

### On GKE

GKE needs `gcloud` and three more names:

| name | what it is | how to get it |
|---|---|---|
| `$CLUSTER`, `$ZONE` | the cluster and its location | `gcloud container clusters list` |
| `$NODEPOOL` | the node pool | `gcloud container node-pools list --cluster $CLUSTER --zone $ZONE` |

Park the autoscaler. During the roll, the old pool's pods that lost
their node sit Pending on purpose: they are the rollback reserve. The
autoscaler reads Pending pods as demand and would add nodes for pods
that must never schedule. Save its config first; you restore it
[after the roll](#after-the-roll-on-gke).

```bash
gcloud container node-pools describe $NODEPOOL --cluster $CLUSTER --zone $ZONE \
  --format='json(autoscaling)'   # save for after the roll
gcloud container clusters update $CLUSTER --zone $ZONE \
  --node-pool $NODEPOOL --no-enable-autoscaling
```

If a GKE maintenance window falls inside the roll, add a maintenance
exclusion for it too: a node GKE recreates mid-roll comes back at the
pool's label, which is still `$OLD_VERSION`.

Check that the node pool itself carries
`ate.dev/substrate-version=$OLD_VERSION`, or nodes GKE creates later
arrive unlabeled and nothing schedules to them. If it is missing, stamp
it now with the old value:

```bash
gcloud container node-pools describe $NODEPOOL --cluster $CLUSTER --zone $ZONE \
  --format='get(config.labels)' # copy the value
gcloud container node-pools update $NODEPOOL --cluster $CLUSTER --zone $ZONE \
  --node-labels=<every existing user label>,ate.dev/substrate-version=$OLD_VERSION
```

Other providers have their own equivalents: stop whatever adds or
recreates worker nodes during the roll, and make sure a node created
later arrives with the old label.

## Three things that break an upgrade

Each warning comes back at the step where the mistake becomes possible.

> [!WARNING]
> **Do not delete a node's old worker pods before flipping its
> version label.** Otherwise the old pool reschedules replacements onto
> the same node, and old workers end up next to the new atelet: exactly
> the version skew the roll exists to prevent. (Step 5)

> [!WARNING]
> **Do not edit a serving worker pool, and do not scale it down.** The
> controller would roll the pool's Deployment straight through live
> actors. A deleted worker pod does go through the eviction path:
> `SIGTERM` is forwarded into the actor's containers and the control
> plane keeps accepting a suspend for 30 minutes, so an actor suspended
> inside that window saves its state and stays resumable. Handling
> `SIGTERM` by exiting cleanly is not enough on its own; the suspend has
> to reach the control plane and finish. An actor still awake when the
> window closes moves to `ACTOR_STATE_CRASHED`: `resume` and `suspend`
> are both refused, and everything since its last snapshot is lost. Call
> `RevertActor` (`kubectl ate revert`) to discard the crashed run and
> return the actor to `ACTOR_STATE_SUSPENDED` at its last external snapshot
> so it can be resumed. Scaling a serving pool down removes pods the same
> way, without suspending the actors on them. (Step 4 clones the pool; it
> never edits it.)

> [!WARNING]
> **On GKE, do not touch the node pool's label until every node is
> rolled.** A pool label update applies in place to every node in the
> pool, so the whole fleet flips at once, with no drain and no pacing.
> (After the roll)

## Upgrade

### 1. Apply the new CRDs

From the new release's checkout, apply its CRDs. Nothing running
changes; the new schema is in place for the controller that follows.

```bash
# in a checkout of the new release
kubectl apply -f manifests/ate-install/generated
```

### 2. Upgrade ate-controller

```bash
go run ./cmd/ate-setup deploy ate-controller
```

This rolls `ate-controller` (and re-applies the CRDs, which is a
no-op now).

By convention, changes to how the controller renders worker
pods sit behind WorkerPool fields, so the new controller keeps
rendering the serving pools as they are. A release that breaks that
convention says so in its notes. Expect the pools' Deployments to
roll once here in that case. Every actor is suspended through the
worker eviction path, loses no state, and resumes on demand. If they
roll, wait for `READY` to equal `DESIRED` again on every serving pool
(`kubectl get workerpools -A`) before step 4.

The move of the node state root from `/var/lib/ateom-gvisor` to
`/var/lib/ate` is such a release. A worker that lands on a node whose
atelet still uses the old path reaches it only once step 5 moves that node.
The old directory can be deleted afterwards.

### 3. Prepare the new dataplane

The dataplane roll starts with the new atelet DaemonSet, from the
same checkout:

```bash
go run ./cmd/ate-setup deploy atelet
```

It lands next to the old one with zero pods until step 5 flips a
node.

Then read `$NEW_VERSION` off the cluster:

```bash
kubectl get ds -n ate-system -l app=atelet -L ate.dev/substrate-version
```

Exactly two DaemonSets print. The SUBSTRATE-VERSION that is not `$OLD_VERSION` is
`$NEW_VERSION`, and the new DaemonSet shows 0 `DESIRED` because no node carries
its label yet. `kubectl get nodes -L ate.dev/substrate-version` still
shows every node at `$OLD_VERSION`. If only one DaemonSet prints, the
version did not change (see Checkout and environment) and the command
rolled the running atelet in place.

Last, build and push the new worker images. The refs print together
at the end. The one for your pool's `sandboxClass` goes into the clone
in step 4:

```bash
go run ./cmd/ate-setup publish worker-images
```

If you are using prebuilt images there is nothing to publish. The new worker image
is the release's `ateom-<sandboxClass>` image under the same repo and
tag as the control plane, pinned by digest:

```bash
NEW_IMAGE=$IMAGE_REPO/ateom-gvisor:$IMAGE_TAG@$(crane digest $IMAGE_REPO/ateom-gvisor:$IMAGE_TAG)
```

### 4. Create the new pool

**Repeat this step for every serving pool the checklist listed**.

> [!WARNING]
> Clone the old pool. Do not edit it. An edit rolls its Deployment
> straight through the live actors on it (second warning above).

Copy the old pool under a new name (`$NEW_WORKERPOOL`, for example
the old name plus `$NEW_VERSION`) and change only `workerImage` and
the version pin. The command below does exactly that; everything
else, including the `metadata.labels` the scheduler matches actors
by, carries over as is.

```bash
NEW_IMAGE=<the ateom ref from step 3, for this pool's sandboxClass>

kubectl -n $NS get workerpool $OLD_WORKERPOOL -o json \
  | jq --arg name "$NEW_WORKERPOOL" --arg image "$NEW_IMAGE" --arg version "$NEW_VERSION" '
      {apiVersion, kind,
       metadata: {name: $name, namespace: .metadata.namespace,
                  labels: .metadata.labels},
       spec: (.spec + {workerImage: $image})}
      | .spec.template.nodeSelector["ate.dev/substrate-version"] = $version
    ' \
  | kubectl apply -f -
```

Do not shrink `spec.template.resources.limits` while cloning. A
worker only accepts an actor whose limits fit under them.

Verify:

```bash
# $NEW_VERSION, then an image ref that contains @sha256:
kubectl -n $NS get workerpool $NEW_WORKERPOOL \
  -o jsonpath='{.spec.template.nodeSelector.ate\.dev/substrate-version}{"\n"}{.spec.workerImage}{"\n"}'

# One Deployment per pool; new-pool pods all Pending (no node carries
# the new label yet).
kubectl -n $NS get deploy -l ate.dev/worker-pool
kubectl -n $NS get pods -l ate.dev/worker-pool=$NEW_WORKERPOOL
```

While both pools serve, placement between them is random, and that
is fine: a snapshot written on either version restores on either
version. The roll converges because step 5 takes old workers out of
service node by node, not because the scheduler prefers the new pool.

### 5. Roll each node

Repeat for every node, one at a time. On a single-node cluster this
step is a full stop: every actor is suspended at once.

**a. Drain the node's workers.** Bound actors keep running. Draining
only stops new placements.

```bash
# A worker's name is its pod's UID; that is what DrainWorker takes.
for w in $(kubectl ate get workers -o json \
             | jq -r --arg node "$NODE" '.workers[] | select(.nodeName == $node) | .metadata.name'); do
  grpcurl -cacert /tmp/ate-ca.pem -authority api.ate-system.svc \
    -H "authorization: Bearer ${TOKEN}" \
    -d "{\"worker\": {\"name\": \"${w}\"}}" \
    127.0.0.1:8443 ateapi.Control/DrainWorker
done
```

**b. See what is still on the node.** Two lists: the node's workers
with the actor each one hosts, and any paused actor whose local
snapshot lives on this node. A paused actor sits on no worker, so it
shows up only in the second list.

```bash
kubectl ate get workers -o json | jq -r --arg node "$NODE" '
  ["WORKER", "POD", "ASSIGNED ACTOR"],
  (.workers[] | select(.nodeName == $node)
   | [.metadata.name, .workerPod,
      (.status.assignment.actor | if . then .atespace + "/" + .name else "<none>" end)])
  | @tsv' | column -t -s $'\t'

kubectl ate get actors -A -o json | jq -r --arg node "$NODE" '
  ["PAUSED_ACTOR", "STATE"],
  (.actors[]
   | select(.status.localSnapshot.nodeVmsWithLocalSnapshots // [] | index($node))
   | [.metadata.atespace + "/" + .metadata.name, .status.state])
  | @tsv' | column -t -s $'\t'
```

**c. Suspend them at your own pace.** Every actor in either list has to be suspended
before the node moves (`kubectl ate suspend actor <name> -a
<atespace>`). Suspend releases the worker and uploads the durable
snapshot, so the actor resumes on demand onto any free matching worker
afterwards. Repeat step b until the `ASSIGNED ACTOR` column reads
`<none>` throughout and the paused list is empty. Then rerun the drain
in step a once more right before flipping.

> [!WARNING]
> Do not run e before d. Deleting the old-pool pods while the node
> still carries `$OLD_VERSION` lets the old pool put replacements right
> back on this node, next to the new atelet (first warning above).

**d. Flip the label.** The old atelet pod leaves on its own and the
new one starts.

```bash
kubectl label node $NODE ate.dev/substrate-version=$NEW_VERSION --overwrite
```

Wait for the new `atelet` pod on the node to be Ready before going on. Usually that is seconds; the new `atelet` pod stays Pending only while the old one is still exiting, since it holds the node's host ports until then.

```bash
# The pod is named after atelet-<new suffix>.
kubectl get pods -n ate-system -l app=atelet --field-selector spec.nodeName=$NODE
```

**e. Delete the node's old-pool worker pods.** Step c emptied them,
but they are still Ready and hold capacity the new pool needs on this
node. Their Deployment cannot reschedule them here anymore. Repeat
per pool if the node hosts several.

```bash
kubectl -n $NS delete pod -l ate.dev/worker-pool=$OLD_WORKERPOOL \
  --field-selector spec.nodeName=$NODE
```

**f. Confirm the node has moved.** Two checks:

```bash
# New-pool workers came up here.
kubectl -n $NS get pods -l ate.dev/worker-pool=$NEW_WORKERPOOL --field-selector spec.nodeName=$NODE

# No old-pool pod is left here.
kubectl -n $NS get pods -l ate.dev/worker-pool=$OLD_WORKERPOOL --field-selector spec.nodeName=$NODE
```

New-pool pods Pending on other nodes are expected until those nodes
move.

**g. Take the next node.** Start again at step a once `kubectl ate get
workers` shows at least one FREE worker for the suspended actors to
land on.

You are done when `kubectl get nodes -L ate.dev/substrate-version`
shows every node at `$NEW_VERSION` (a node that joined mid-roll still
carries `$OLD_VERSION`; apply step 5 to roll it) and every assigned
actor in `kubectl ate get actors -A` sits on a new-pool pod.

### 6. Upgrade the rest of the control plane

Every actor is now on the new dataplane, running or suspended, and
ate-controller moved in step 2. From the same checkout, move
`ate-api-server` first, then everything else:

```bash
go run ./cmd/ate-setup deploy apiserver
go run ./cmd/ate-setup deploy ate-system
```

The second command rolls atenet and converges the rest of the
install; it re-resolves and re-applies everything, so it could take a
while. The checklist's port-forward dies when the API server rolls.
Restart it and mint a fresh token if you still need to drain.

**NOTE**: Until
this step is done the old ate-api-server is still serving, so do not
start using API fields new in this release before upgrade finishes.

### After the roll, on GKE

> [!WARNING]
> Only after every node shows `$NEW_VERSION` in [Check your
> progress](#check-your-progress). Relabeling the node pool applies to
> every node in it at once, with no drain and no pacing.

Relabel the node pool with `$NEW_VERSION` so nodes created later start
at the new version, then restore the autoscaler from the config you
saved [before the roll](#on-gke):

```bash
# --node-labels REPLACES the pool's full user label set: list the
# current labels first and carry them all over.
gcloud container node-pools describe $NODEPOOL --cluster $CLUSTER --zone $ZONE \
  --format='get(config.labels)' # copy the output

gcloud container node-pools update $NODEPOOL --cluster $CLUSTER --zone $ZONE \
  --node-labels=<every existing user label>,ate.dev/substrate-version=$NEW_VERSION

# restore the autoscaler config
gcloud container clusters update $CLUSTER --zone $ZONE --node-pool $NODEPOOL \
  --enable-autoscaling --min-nodes <min> --max-nodes <max>
```

## Something's wrong, or I need to undo

First run [Check your progress](#check-your-progress). It tells you
which step you reached, and everything below is keyed on that.

If a command failed, rerun it: every step is idempotent. The one
exception is drain. There is no undrain; if you drained the wrong
node, check that its workers are empty (suspending as needed), delete
their pods, and let the Deployment's replacements register as fresh
workers.

To roll back, undo what you did in reverse order: the control plane
first, then the nodes, then ate-controller last. The roll never edits
or deletes the old objects, so the old `WorkerPool` is intact, the pods
that lost their nodes stay Pending, and the old atelet DaemonSet is
still installed. Rolling nodes back is a matter of label flips. Run the
commands from the old release's checkout with the same environment as
the install, `VERSION` included if the install pinned it.

- Relabeled the GKE node pool: park the autoscaler again as
  [before the roll](#on-gke), then move the node pool label back to
  `$OLD_VERSION` with the [after the roll](#after-the-roll-on-gke)
  command.
- Past step 6: `go run ./cmd/ate-setup deploy apiserver` now, and
  `go run ./cmd/ate-setup deploy ate-system` once the nodes are back.
- Past step 5: roll each flipped node back by running step 5 with the
  sides swapped: drain, get every actor off the node as in b and c,
  flip the label back to `$OLD_VERSION`, and delete the node's
  new-pool pods.
- Past step 4: once no actor is assigned to a new-pool worker, delete
  each clone: `kubectl -n $NS delete workerpool $NEW_WORKERPOOL`.
- Past step 3: once no node carries `$NEW_VERSION`, delete the new
  DaemonSet: `kubectl delete daemonset -n ate-system -l app=atelet,ate.dev/substrate-version=$NEW_VERSION`.
- Past step 2: `go run ./cmd/ate-setup deploy ate-controller`.

`kubectl get ds -n ate-system -l app=atelet` must not show a third
DaemonSet afterwards. A third means the old checkout produced a
version other than `$OLD_VERSION`: delete it and check `VERSION`.

Abandoning the upgrade entirely is the whole list, top to bottom.
Retiring the old pool (below) is a separate, deliberate step. As long
as the old objects exist, rollback is one label flip per node.

## Retire the old pool

After the new version has soaked, reclaim the reserve. Save the old
pool's spec first: its `workerImage` ref is by digest and stays
pullable, so the saved file is the last-resort way to recreate the
pool. Then check the guards. Deleting the old objects is what ends
the rollback option.

```bash
kubectl -n $NS get workerpool $OLD_WORKERPOOL -o yaml > old-pool-backup.yaml

# Guards: no node still at $OLD_VERSION; no old-pool pod Running (Pending is
# expected); no actor assigned to an old-pool worker.
kubectl get nodes -l ate.dev/substrate-version=$OLD_VERSION
kubectl -n $NS get pods -l ate.dev/worker-pool=$OLD_WORKERPOOL
kubectl ate get workers

# Retire the old pool (its Deployment and pods go with it) and the
# old atelet DaemonSet.
kubectl -n $NS delete workerpool $OLD_WORKERPOOL
kubectl delete daemonset -n ate-system -l app=atelet,ate.dev/substrate-version=$OLD_VERSION
```

The new pool keeps its name. Names mean nothing to placement, so
`counter-v2` can serve indefinitely, and the next upgrade clones it
to `counter-v3`.