Part of #232
1. During CreateActor, the volumes to be created get instantiated in the
Actor API in PENDING status. The Actor is still instantiated as
SUSPENDED.
However, the CreateVolume operation is moved to the beginning of
ResumeActor. This is because CreateActor is intentionally designed to
not be idempotent. The downside is that the user won't know dependent
resource creation failed until the first resume.
2. Changes volume id to use actor uid instead of atespace+actorname and
adds "substrate" prefix
3. Adds a new DELETING status that gets persisted before we start to
delete volumes.
This shifts the existing DB DELETE precondition checks to the new
DELETING status. And now you can only delete the actor in DB from
DELETING status.
4. Refactor DeleteActor to use a workflow
In the future, we will also add a workflow that will do mandatory node
cleanup of volumes before starting to delete the volumes.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Sessions are no longer a concept in Substrate; Actor is the glossary
term. This completes the "s/Session/Actor" TODO that sat at the top of
ateapi.proto, and removes the TODO.
API surface:
service SessionIdentity -> ActorIdentity
MintJWTRequest.session_id -> actor_id
MintJWTResponse.session_jwt -> actor_jwt
MintCertRequest.session_id -> actor_id
MintCertResponse.session_certificates -> actor_certificates
Go packages:
cmd/ateapi/internal/sessionidentity -> actoridentity
cmd/ateapi/internal/sessionidjwt -> actoridjwt
Flags and cluster resources:
--session-id-jwt-pool -> --actor-id-jwt-pool
--session-id-ca-pool -> --actor-id-ca-pool
Secrets, volumes and mount paths renamed to match, in both
manifests/ate-install/ate-api-server.yaml and hack/install-ate.sh
(--create-session-id-ca-pool-secret -> --create-actor-id-ca-pool-secret).
Two credential identity values change with the rename:
JWT issuer https://broker.agentic-substrate-session-id-broker.svc
-> https://broker.agentic-substrate-actor-id-broker.svc
SPIFFE ID spiffe://substrate-session.local/app/../session/..
-> spiffe://substrate-actor.local/app/../actor/..
Tokens and certificates issued before this change will not validate
against the new issuer or trust domain.
BREAKING: the gRPC wire path moves from /ateapi.SessionIdentity/* to
/ateapi.ActorIdentity/*, and the Secrets must be recreated under their
new names before the new ate-api-server rolls out.
Part of [#23](https://github.com/agent-substrate/substrate/issues/23)
The overall plan is:
1) Trap SIGTERM in the ateom, and forward SIGTERM to all application
containers.
2) Once all application containers exit, ateom will exit itself.
3) The workerpool syncer will mark workers as 'DRAINING' when a pod
deletion is triggered (pod.DeletionTimestamp) to prevent actors from
being scheduled. On the pod Deleted event, the worker and actor (if it
was assigned) will be cleaned up.
This PR implements 3), there are 2 more PRs coming which will implement
1) and 2) for each ateom (gvisor and uVM).
WorkerPool already served /scale, so `kubectl scale` worked, but a HPA
pointed at one did not: the subresource declared no labelSelectorPath,
and HPA needs a selector to find the pods whose metrics it is averaging
or to compute ambiguous selectors.
https://github.com/agent-substrate/substrate/issues/198
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
DurableDir volumes shipped for gVisor only; the micro-VM runtime dropped
Container.durable_dir_volumes on the floor and the API rejected the
combination outright. Give the guest the volume:
- cmd/ateom-microvm/internal/tarutil: archive/restore a directory tree
preserving modes, ownership, times, symlinks and hardlinks, with
os.Root-confined extraction. Snapshot support (a later commit) ships the
volume as a tar; no reusable helper existed, and atelet's image untar does
not preserve ownership.
- A SECOND virtiofsd serves the actor's durable-dir directory read-write
with cache=auto. The kataShared share stays read-only at cache=always,
which is what its TODO asked for: writable volumes get their own share
rather than weakening the overlay lower's caching.
- The guest mounts the new tag at /run/ateom-durable and each WORKLOAD
container binds <tag>/<volume> at its declared paths. The carrier is
deliberately left alone: its rootfs is the read-only lower, where the
agent could not create a missing mount point.
- Lift the CEL rule banning durableDir on sandboxClass microvm.
ateom learns the volume name from the directory layout atelet prepares: the
wire protocol carries mount paths only, and the API allows at most one
durable-dir volume, so anything but exactly one directory is an error.
Snapshot scopes are unchanged so far, so the contents live only as long as
the actor. Checkpoint and restore follow.
Part of #232
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Initial part of #232.
Extends the ActorTemplate API to add support for per-actor external
volumes and adds control plane and atelet hooks following the Actor
lifecycle.
Callouts to volume operations are abstracted with a Volume interface.
Right now, only a mock volume plugin for testing has been implemented,
but we will add CSI support next.
- [x] Tests pass
- [ ] Appropriate changes to documentation are included in the PR - Not
going to update documentation until we add CSI support.
* Add `args` field in addition to `command`.
* Implement logic in atelet to resolve both argv from the actor template
fields and the OCI image config (`entrypoint` and `cmd` fields).
* Update demos to not hardcode command as now we honor the image
default, which makes it not necessary.
Fixes#456
Replace actor_id and atespace name with `metadata` field which
holds common fields.
This commit only changes the ate-apiserver to support this field, but
intentionally skips upstream / downstream systems (such as kubectl-ate,
atenet, atelet, etc) to keep the scope manageable. This will come in subsequent PRs.
Decouple the actor resumption scheduler from Kubernetes by avoiding
direct queries to WorkerPool CRDs. Instead, cache the worker pool's
labels and sandbox class on the Worker object in Redis during pod
synchronization, and update the scheduler to evaluate worker eligibility
using these cached fields.
Related to #292 and #119
Changes:
1. ResumeActor:
1. During `ResumeActor` we'll put the actor in CRASHED state if the
Resume failed due to any errors related to the snapshot itself, which
includes:
1. Missing or corrupted snapshot when downloading from GCS
2. Runsc command fails.
2. All other types of errors will not put the Actor in CRASHED state.
2. SuspendActor or PauseActor
1. During `SuspendActor` or `PauseActor` we'll put the actor in CRASHED
state if we cannot produce a valid snapshot, which includes:
1. cannot connect to Ateom after retries,
2. runsc command fails.
3. cannot upload the snapshot to GCS (during suspend) ; or cannot save
snapshot locally (during pause).
2. All other types of errors will not put the Actor in CRASHED state.
An actor name is only unique within its atespace, so the two always travel together. Add ActorRef{atespace, name} and use it across the Get/Create/Update/Suspend/Pause/Resume/Delete actor requests in place of separate actor_id + atespace fields. The stored Actor object and Redis key layout are unchanged.
Implement a layer allowing to take durableDir snapshot only, without
taking snapshot of entire memory. This is a part of the #119 feature.
The "durableDir" support is introduced via adding Volumes support to
ActorTemplate.
## Knowns issues / limitations
1. No MicroVM support for homedir yet. Will be implemented in following
PR.
2. Resume from SNAPSHOT_SCOPE_DATA temporary behavior
Current behavior of resume from durableStorage returns immediately after
gVisor process is started, it means container's bootstrap is not
completed by the time resume API returned to the caller. It causes the
issue that first curl request for suspended actor throws an HTTP
exception, since webserver listening on the port inside container is not
running yet.
PR #330 fixes the issue, for a meantime the workaround is to call `
kubectl ate resume actor` and after it call to the curl itself.
- [X ] Unit tests pass
- [X ] E2E tests are updated and pass
Follow-up to https://github.com/agent-substrate/substrate/pull/287 /
#123
- align snapshot behavior: use a tmpfs (for writes) on top of read-only
viritio-fs mount for the container image rootfs instead of ext4 images
- enable multiple container support
- cleanup stale comments
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Extends ActorTemplate.Container spec to allow defining a readyz probe.
* ateoms (both gVisor & microVM) are updated and wait for readyz signal
if it was enabled on container when creating a new container or resume
from a snapshot.
* actorTemplates controller has been updated not to wait 20 sec during
golden snapshot creation, if readyz is enabled on actor. It assumes,
atelet call returns when all containers passed readyz.
Tested manually both counters:
* gVisor
* microvm
Fixes#316
- [x] e2e tests pass
- [X] Appropriate changes to documentation are included in the PR
- workerpool_apply: rename applyMicroVMPodShape -> maybeApplyMicroVMPodShape and
pass wp.Spec.SandboxClass instead of the whole WorkerPool.
- sandboxconfig validation test: add an arm64 micro-VM asset-set case.
- specconv: TODO to forward Seccomp/Sysctl + Apparmor/SELinux for OCI parity.
- run.go: clarify the dialAgentRetry per-attempt cap vs retry-gap comment.
- roadmap: drop the microVM line (shipped).
WorkerPool renders the micro-VM worker pod shape, and the SandboxConfig
validating admission policy enforces the micro-VM asset set (cloud-hypervisor,
guest kernel, guest image, kata config).
Big UX improvement for iterating on #123
Follow-up to https://github.com/agent-substrate/substrate/pull/270
(previously worker pools were directly referenced by actors so you just
made sure to pick an appropriate workerpool, we discussed there)
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This change removes the hard-link from templates to worker pools,
allowing pools to be selected using k8s labels.
Both actors and actor templates are augmented with a label selector
field.
When an actor is resumed, we resolve the set of *eligible* worker pools
by evaluating the intersection of both actor and template selectors.
Then we find a free worker from one of them. The overall design is
described in [1].
[1]:
https://github.com/agent-substrate/substrate/issues/47#issuecomment-4712334620
**Updated Original PR Description**
## Summary
This PR adds a small, controlled set of scheduling fields to WorkerPool
so WorkerPool-managed Pods can be placed onto appropriate Kubernetes
nodes.
Added fields:
- `nodeSelector`
- `tolerations`
- `priorityClassName`
- `nodeAffinity`
## Scope
This intentionally does not expose a full `PodTemplateSpec`, and does
not add support for full `affinity`, `podAffinity`, or
`podAntiAffinity`.
Resource requests/limits are intentionally left out of scope for this CL
while the WorkerPool resource model is still being discussed separately.
The goal is to keep the first version small while addressing the
WorkerPool-to-Kubernetes-Node scheduling gap discussed in
[#212](https://github.com/agent-substrate/substrate/issues/212).
`nodeAffinity` is included to support heterogeneous node pools where
`nodeSelector` is too limited, such as requiring or preferring one of
several equivalent accelerator/local-SSD/cache node pools, or expressing
soft preferences during node pool migration.
`priorityClassName` is included to help preserve warm WorkerPool
capacity under cluster pressure. Since WorkerPool pods are the slots
available for actor resume, higher-priority pools can protect
latency-sensitive or interactive actor workloads from displacement by
lower-priority batch workloads, and allow different WorkerPools to
represent different service classes, e.g. interactive vs. batch.
## Related issue
Part of #212.
Related to #47.
## Problem
`go test -v ./...` (the `run-tests` job) intermittently fails during
envtest **setup** — not in any test logic:
- `fork/exec .../kubebuilder-envtest/k8s/1.36.0-linux-amd64/etcd: text
file busy`
- `setup-envtest: ... unable to create file kubectl from archive for
version-platform pair 1.36.0-linux-amd64`
Three packages use envtest (`cmd/ateapi/internal/controlapi`,
`cmd/atecontroller/internal/controllers`, `pkg/api/v1alpha1`). `go test
./...` runs them as separate test binaries **in parallel**, and each
`TestMain` shells out to `setup-envtest use`. On a cold cache they race
to download, extract, and exec the **shared** kubebuilder binaries —
hence `text file busy` (exec'ing etcd while it's still being written)
and the archive-extraction error.
Seen on `main` (e.g. the #259 push run) and on PRs. It only reproduces
on a **cold** cache, which is why it's intermittent and doesn't show up
locally once the cache is warm.
## Fix
Add `internal/envtestbins.BinaryAssetsDir()`, which runs `setup-envtest
use` under a **cross-process file lock** (`flock` on a fixed file under
`os.TempDir()`, shared by the sibling test binaries). The first caller
populates the shared cache while the others wait, then they hit a warm
no-op path lookup. The duplicated setup in all three `TestMain`s now
calls this helper.
The lock scope (user/machine-global) matches the resource it guards —
the kubebuilder binary cache is itself user-global, shared even across
repo checkouts.
## Testing
Reproduced locally by moving the warm cache aside and running the three
packages concurrently: it fails without the lock and passes with it.
Full `go test ./...` green.
Test-only change; no production code touched.
Implements `PAUSED` state for issue #119.
The snapshot files are kept locally on node VM in a separate folder. At
resume time, scheduler uses a node VM hint and picks up a worker from
the same node where files were stored at suspend time.
The local file management solution is temporary and will be replaced
once @msau42 introduces a new component that is supposed to manage files
on the node VM.
- [X] Tests pass
- [X] Manual tests with counter demo
```
>kubectl ate create actor my-counter-1 --template ate-demo-counter/counter
>curl -X POST -H "Host: my-counter-1.actors.resources.substrate.ate.dev" http://localhost:8000
hello from: 169.254.17.2 | preserved memory count: 1
>curl -X POST -H "Host: my-counter-1.actors.resources.substrate.ate.dev" http://localhost:8000
hello from: 169.254.17.2 | preserved memory count: 2
>kubectl ate pause actor my-counter-1 -o json
> curl -X POST -H "Host: my-counter-1.actors.resources.substrate.ate.dev" http://localhost:8000
hello from: 169.254.17.2 | preserved memory count: 3
```
### Breaking change
This PR introduces a breaking change in the Actor proto. All existing
actor needs to be recreated, prior testing PAUSE functionality.
A couple of TODOs/Discussions which I think are important to have based
on these changes.
1. Do we need a caching client for this to avoid calling the apiserver
every time?
2. Where/how do we want to resolve secrets generally? I think there are
good arguments to be made for the control-plane OR the "node" component.
In my mind this also related to:
https://github.com/agent-substrate/substrate/issues/18Fixes#15
> It's a good idea to open an issue first for discussion.
- [ x ] Tests pass
- [ x ] Appropriate changes to documentation are included in the PR
# Implement Pagination and Pipelining for ListActors
## Overview
**Fixes:** #153
This PR introduces robust, memory-safe pagination to the `ListActors`
operation and optimizes the Valkey retrieval layer.
Previously, `ListActors` performed an unbounded parallel scatter-gather
across all Valkey cluster shards, pulling all actors into memory at
once. At scale (millions/billions of actors), this would cause OOM
errors and severe latency.
This update replaces the concurrent scatter-gather with a deterministic,
sequential shard scanning approach, adhering to
[AIP-158](https://google.aip.dev/158) pagination standards.
## Changes Included
- **gRPC API (`ateapi.proto`)**: Added `page_size`, `page_token`, and
`next_page_token` fields.
- **Client (`kubectl-ate`)**: Updated to seamlessly loop through
`next_page_token` values until all actors are fetched, presenting a
continuous stream to the user without blowing up memory on a single
request.
- **API Handler (`controlapi`)**: Bounded requests with a max
`page_size` limit of `1000`.
- **Persistence (`ateredis`)**:
- Implemented `listActorsPageToken` to safely serialize and encode
cross-shard cursors.
- Replaced `ForEachMaster` concurrent execution with deterministic
sequential iteration.
- Implemented Valkey Pipeline batching (`Pipelined`) for `GET` requests
to eliminate N+1 network latency.
- **Testing**: Added `TestListActors_Pagination` at both the integration
level (`functional_test.go`) and data layer (`ateredis_test.go`).
Refactored sorting to use `cmpopts.SortSlices`.
## Design Decisions
During the design phase, several explicit trade-offs were made to
balance performance with API simplicity:
1. **Opaque Tokens (AIP-158):** The `page_token` is strictly an opaque
string at the `ateapi` layer. The client (`kubectl-ate`) is completely
unaware that it contains base64-encoded JSON mapping to `{"shard",
"cursor"}`. This keeps our API database-agnostic.
2. **Empty Pages & Client-Side Looping:** The persistence layer is kept
simple and "dumb". If a shard scan returns 0 keys but the cursor is not
`0`, we return an empty page. We rely on the client `kubectl-ate` to
`for` loop until `next_page_token` is completely exhausted. Update: I
have server side looping across shards until pagesize is met to limit
round trips due to high shard count or low actor count on distributed
shards.
3. **"Soft" Consistency (Missing/Duplicate Data):** `ListActors` is an
infrequent operations command, not on the critical path. Taking a global
cluster lock for iteration would severely degrade performance. We accept
eventual consistency here: if actors migrate shards during an active
scan, they might be duplicated or missed.
4. **Topology Flux Errors:** If a client passes a valid token pointing
to a shard address that no longer exists (due to a cluster resharding
mid-scan), the API simply returns an `Aborted` error. We do not attempt
complex recovery; the client must just restart the `ListActors` command.
5. **Bounded Page Size:** Requests are currently capped at a maximum of
`1000` to prevent single massive payloads. We can revisit/change the max
later.
NOTE: Actor filtering will be implemented later.
## Architecture Flow (Valkey)
Note: If there are future plugin-db models they will need to handle the
cusor/pages at their own implementation layer per the store.go
interface.
The system now seamlessly iterates across shards, handing off cursors
safely between nodes:
```mermaid
sequenceDiagram
participant CLI as kubectl-ate
participant API as ateapi
participant Valkey as Valkey Cluster (Multiple Shards)
CLI->>API: ListActors(page_size=1000, page_token="")
Note over API: Request 1: Empty Token
API->>API: Decode page_token (empty -> start fresh)
API->>Valkey: Fetch all Master Nodes
Valkey-->>API: [Master C, Master A, Master B]
API->>API: Sort Masters alphabetically by IP/Port
Note over API: Sorted: [Master A, Master B, Master C]
API->>Valkey: SCAN (Master A, cursor=0, count=1000)
Valkey-->>API: keys=[k1..k300], next_cursor=45
Note over API: OPTIMIZATION: Pipelining avoids N+1 problem
API->>Valkey: PIPELINE GET (k1..k300) directly on Master A
Valkey-->>API: values=[v1..v300]
API->>API: Decode Protobufs
API->>API: Encode page_token={"shard": "Master A", "cursor": 45}
API-->>CLI: 300 Actors, next_page_token="base64(eyJzaGFyZ..."
%% SECOND REQUEST
CLI->>API: ListActors(page_size=1000, page_token="base64(...)")
Note over API: Request 2: Resuming Mid-Shard
API->>API: Decode page_token -> Shard="Master A", Cursor=45
API->>API: Fast-forward directly to Master A
API->>Valkey: SCAN (Master A, cursor=45, count=1000)
Valkey-->>API: keys=[k301..k400], next_cursor=0
API->>Valkey: PIPELINE GET (k301..k400) directly on Master A
Valkey-->>API: values=[v301..v400]
Note over API: Shard A is empty. Token points to Shard B!
API->>API: Encode page_token={"shard": "Master B", "cursor": 0}
API-->>CLI: 100 Actors, next_page_token="base64(eyJzaGFyZ..."
%% THIRD REQUEST
CLI->>API: ListActors(page_size=1000, page_token="base64(...)")
Note over API: Request 3: Starting New Shard
API->>API: Decode page_token -> Shard="Master B", Cursor=0
Note over API: Continues scanning Master B...
```
## Testing:
```
> ./bin/kubectl-ate get actors
NAMESPACE TEMPLATE ID STATUS ATEOM POD ATEOM IP VERSION
ate-demo-counter counter 61b478cc-2e44-4ec7-ac6c-1af3f385de86 STATUS_SUSPENDED <none> 5
```
```
~/go/bin/grpcurl -insecure -d '{"page_size": 2}' localhost:9090 ateapi.Control/ListActors
{
"actors": [
{
"actorId": "test-counter-4",
"version": "1",
"actorTemplateNamespace": "ate-demo-counter",
"actorTemplateName": "counter",
"status": "STATUS_SUSPENDED"
}
],
"nextPageToken": "eyJzaGFyZCI6IjEwLjI0NC4wLjE5OjYzNzkiLCJjdXJzb3IiOjI1MjU1fQ=="
}
```
<details>
```
bash -c '
export NEXT_TOKEN=""
while :; do
echo "--- Fetching Page ---"
# Fetch the data and store the response
RESPONSE=$(~/go/bin/grpcurl -insecure -d "{\"page_size\": 2, \"page_token\": \"$NEXT_TOKEN\"}" localhost:9090 ateapi.Control/ListActors)
# Print the response to the terminal
echo "$RESPONSE"
# Extract the token without needing jq
NEXT_TOKEN=$(echo "$RESPONSE" | grep -o "\"nextPageToken\": \"[^\"]*\"" | cut -d"\"" -f4)
# If no token is found, we reached the end
if [ -z "$NEXT_TOKEN" ]; then
echo "--- Reached the end! ---"
break
fi
echo ""
read -p "Press Enter to fetch the next page using the new token..."
done
'
```
--- Fetching Page ---
{
"actors": [
{
"actorId": "test-counter-4",
"version": "1",
"actorTemplateNamespace": "ate-demo-counter",
"actorTemplateName": "counter",
"status": "STATUS_SUSPENDED"
}
],
"nextPageToken":
"eyJzaGFyZCI6IjEwLjI0NC4wLjE5OjYzNzkiLCJjdXJzb3IiOjI1MjU1fQ=="
}
Press Enter to fetch the next page using the new token...
--- Fetching Page ---
{
"nextPageToken": "eyJzaGFyZCI6IjEwLjI0NC4wLjIxOjYzNzkiLCJjdXJzb3IiOjB9"
}
Press Enter to fetch the next page using the new token...
--- Fetching Page ---
{
"actors": [
{
"actorId": "test-counter-1",
"version": "1",
"actorTemplateNamespace": "ate-demo-counter",
"actorTemplateName": "counter",
"status": "STATUS_SUSPENDED"
}
],
"nextPageToken":
"eyJzaGFyZCI6IjEwLjI0NC4wLjIxOjYzNzkiLCJjdXJzb3IiOjQ1NjAwfQ=="
}
Press Enter to fetch the next page using the new token...
--- Fetching Page ---
{
"actors": [
{
"actorId": "test-counter-5",
"version": "1",
"actorTemplateNamespace": "ate-demo-counter",
"actorTemplateName": "counter",
"status": "STATUS_SUSPENDED"
}
],
"nextPageToken":
"eyJzaGFyZCI6IjEwLjI0NC4wLjIxOjYzNzkiLCJjdXJzb3IiOjYyMDg2fQ=="
}
Press Enter to fetch the next page using the new token...
--- Fetching Page ---
{
"nextPageToken": "eyJzaGFyZCI6IjEwLjI0NC4wLjI1OjYzNzkiLCJjdXJzb3IiOjB9"
}
Press Enter to fetch the next page using the new token...
--- Fetching Page ---
{
"actors": [
{
"actorId": "test-counter-2",
"version": "1",
"actorTemplateNamespace": "ate-demo-counter",
"actorTemplateName": "counter",
"status": "STATUS_SUSPENDED"
}
],
"nextPageToken":
"eyJzaGFyZCI6IjEwLjI0NC4wLjI1OjYzNzkiLCJjdXJzb3IiOjE2OTkzfQ=="
}
Press Enter to fetch the next page using the new token...
--- Fetching Page ---
{
"actors": [
{
"actorId": "test-counter-3",
"version": "1",
"actorTemplateNamespace": "ate-demo-counter",
"actorTemplateName": "counter",
"status": "STATUS_SUSPENDED"
},
{
"actorId": "61b478cc-2e44-4ec7-ac6c-1af3f385de86",
"version": "5",
"actorTemplateNamespace": "ate-demo-counter",
"actorTemplateName": "counter",
"status": "STATUS_SUSPENDED",
"lastSnapshot":
"gs://ate-snapshots/ate-demo-counter/61b478cc-2e44-4ec7-ac6c-1af3f385de86/2026-06-03T03:17:34Z-UMYVUZOWW6LDPS5X5FSG3KC4VV"
}
]
}
--- Reached the end! ---
</details>
Pod names alone can easily exceed the max length (253 > 107).
Add a symlink with the full pod name for debugging.
Follow-up #92
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
---
TODO: I'm not sure about the symlink. It's annoying mapping pod =>
hash(pod) otherwise though. We could put a file inside the directory
with the pod name instead, but I think we should probably put the full
pod name _somewhere_ in the structure so it's easier to inspect these.
Now the worker pool can be scaled via regular `kubectl scale` commands.
Use server-side-apply to apply changes.
Propagate the effective number of replicas from the status of the
managed deployment.
Fixes https://github.com/agent-substrate/substrate/issues/48
Eitan Yarmush