mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-11-12 10:35:06 +08:00
Haven Xia
2cd494384f
The runbook for moving a running substrate to a new build without losing actor state. An user runs the roll by hand with `kubectl`, `kubectl ate`, `ate-setup`, `jq`, and `grpcurl`, and every piece of upgrade state lives in cluster objects, so the roll can stop and resume at any point. The order follows the [upgrade design](https://docs.google.com/document/d/1JduAyGZFyqdNp4UhKiv0EN-is3iW5tf5BwGTWbt2ouI/edit?usp=sharing&resourcekey=0-MXG12QCkleIxhY7cOB7g6A): CRDs, then `ate-controller`, then the dataplane node by node, then `ate-api-server` and `atenet`. The dataplane moves by version label. The new atelet DaemonSet sits next to the old one, each serving WorkerPool is cloned with the new worker image and the new version pin, and each node is drained (workers marked `DRAINING` through the `DrainWorker` RPC), emptied by suspending its actors, relabeled, and cleared of old worker pods. The old objects stay untouched until a separate retire step, so rollback is one label flip per node. Tested end to end locally on a GKE cluster: install at one build, run counter actors, upgrade to a second build following the document, and confirm the actors resume on the new pool with their state intact. Rollback and retire were exercised the same way. Fix #1272, fix #1273.