Nested virtualization exposes `/dev/kvm`, which microVMs need. The new
`--enable-nested-virtualization` flag turns it on for the node pool
`setup-gcp` creates. It defaults to on, so pass
`--enable-nested-virtualization=false` for a cluster that does not need
it.
Also, rename the env var `GVISOR_NODE_MACHINE_TYPE` to
`NODE_MACHINE_TYPE` to be consistent with other env vars and to avoid
confusion for microVMs. The old env var still works with a warning.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes#1507
Golden snapshots currently remain owned by the temporary golden actor,
so another resume/suspend cycle or actor deletion can collect a snapshot
still referenced by its template. The controller now copies the warmed
snapshot into a published tag, deletes the golden actor, and records the
tag reference on the template. Interrupted tag creation and cleanup
remain retryable; template deletion cleans up both resources.
`CreateActor` resolves an explicit `sourceTag` or the template's golden
tag into the actor's initial snapshot. Actors created before the golden
tag is ready retain their cold-boot behavior. The golden tag uses the
template UID as its name in `ate-golden`. The proto replaces
`golden_snapshot` with `golden_tag` at field 1, without backward
compatibility.
This PR is based directly on `main` and does not depend on #1521.
Follow-up recommendation: move the create → resume → wait → suspend →
tag → delete sequence into a golden-template workflow using the existing
workflow conventions. The reconciler now coordinates multiple
recoverable steps; it could retain scheduling and retries while
delegating that sequence to the workflow. This refactor is outside this
PR.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Validation: full `env -u NO_COLOR make verify` passed after rebasing
onto `main`. After the final proto field-number change, bindings were
regenerated and the control API unit/functional tests plus proto-format
and Go-format checks passed.
---------
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
The lot admitted every request before its resume lookup, so a full lot
shed traffic to already-RUNNING actors — exactly the starvation
docs/request-parking.md rules out (agent-substrate/substrate#1081), and
TestHandleRequestHeaders_ParkingLotFull pinned that behavior as expected.
Replace the resumer's singleflight.Group with a per-actor flight
registry whose flights signal their park transition (the first
retryable error). A caller now waits slot-free while its flight
resolves, acquires a slot only once the flight parks, and is shed with
the existing 503 "router at capacity" only at that transition — after
the single attempt that revealed it would have to wait. Requests
resolved on the first attempt never touch the lot, so a saturated lot
cannot starve running-actor traffic, and parking.active /
parking.wait.duration now count only genuinely parked requests.
The port-forward, CA bundle and token only need the running cluster,
so they can be prepared before the roll starts. Doing it there lets the
checklist confirm the installed ate-api-server serves DrainWorker
before anything changes, instead of finding out in the per-node step.
The intro said at most one node's worth of capacity is out of service
during the roll, which is only true when there is more than one node.
Say so, and point at the single-node note in the per-node step.
Phrase the first warning as the mistake, like the other two. Add
rollback entries for the new DaemonSet and the cloned pools, so an
operator who stopped before flipping any node finds their step in the
list. Say what the progress commands do not show instead of claiming
that steps leave no trace. Wait for a controller-triggered roll to
settle before cloning pools. Note that a single-node cluster makes the
per-node step a full stop. The setup-gcp README's eviction warning
still said 60 seconds; it is 30 minutes.
# Record actor state changes from ateapi
ateapi now writes a log record every time an actor changes state.
## Before
Nothing tells you what state an actor is in. Metrics can't carry actor
identity, and the control plane samples traces at 10%, so most
transitions leave nothing behind at all. If you want to know whether an
agent is running, paused or suspended, there's nowhere to look.
## After
One filter:
```text
ate.actor.uid="8f2a1c4e6b0d47f1" AND ate.actor.state!=""
```
Last record wins. That's the state it's in now, and the timestamp is
when it got there.
The record:
```json
{
"time": "…",
"level": "INFO",
"msg": "Actor state changed",
"ate.atespace": "ate-demo-counter",
"ate.actor.name": "counter-1",
"ate.actor.uid": "8f2a…",
"ate.template.atespace": "ate-demo-counter",
"ate.template.name": "counter",
"ate.actor.operation.name": "suspend",
"ate.actor.state": "suspended"
}
```
## The two new keys
`ate.actor.state` is the `ateapipb.ActorState` values lowercased, so the
log wording and the state machine can't drift apart.
`ate.actor.operation.name` says what caused the change. The state on its
own doesn't tell you: an actor lands in `suspended` from a suspend and
`paused` from a pause, and only one of those gives the worker back.
`Actor crashed` picks up the same state key. A crash is a state change
too, and it's the one an actor can reach without any operation
finishing.
## Notes for review
**Why ateapi.** It owns the state machine. It also sees transitions that
never reach a worker, like deleting an actor that was already suspended,
or a resume that fails in the scheduler.
**It can't log a state the store never had.** The record goes out after
the store commit, and every state commit has a version precondition, so
a losing writer in a concurrent update writes nothing.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes https://github.com/agent-substrate/substrate/issues/1566
Today the Resume workflow resolves its restore source in the following
order
1. First check if the actor has node-local snapshot,
2. then its own durable external snapshot,
3. then the template's golden snapshot.
The boot flag was consulted at exactly one point in that chain, where it
suppressed using the golden-snapshot, which made its behavior much
narrower than "boot from scratch" suggests:
- Actor has its own external snapshot and boot=true: flag ignored,
restores the actor's snapshot.
- Actor has a local snapshot and boot=true: flag ignored, restores the
local snapshot.
- Actor has no snapshot, template has no golden snapshot: cold boot from
the spec regardless of the boot flag.
- Actor has no snapshot, template has a golden snapshot: boot=false
restores the golden, boot=true cold boots from the spec. This is the
only case where the flag is used.
The glutton benchmark was the only caller that set boot=true, on each
actor's first resume, to report true cold-start latency as a separate
ResumeActorColdStart stats row. @maxsmythe let me know if this is
required.
The proto field number and name were reserved.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
## What
Updates the project overview language in the three docs that share it,
and fixes a long-standing typo.
- **`README.md`** — replaces the overview paragraph with the
secure-by-default positioning: density relative to standard container
runtimes, resume latency and activation throughput, and native
kernel/network isolation.
- **`docs/architecture.md`** — adopts the same lead sentence, keeping
the existing control-plane detail; `computer infrastructure` → `compute
infrastructure`.
- **`docs/roadmap.md`** — `computer infrastructure` → `compute
infrastructure`.
## Notes
The performance figures in the README paragraph (density multiple,
sub-500ms resume, activation rate) have been discussed and aligned
separately.
Docs-only change; no code or behavior is affected.
The container image must include the image digest. This validation was
dropped during the migration of the ActorTemplate resource from CRD to
Substrate API.
Part of #932 (PR 2 of 3). PR 1 (#941) added the `trustBundle` SystemInfo
data source, resolved on the node at Run/Restore. This PR keeps those
projections current while the actor runs.
## Live refresh
A `systemInfoVolumeRefresher` in atelet, modeled on kubelet's projected
volumes: `collectData` is the one place that builds a volume's complete
contents from its spec, `write` applies them, and every lifecycle point
uses the pair. Run/Restore registers the actor's volumes, which writes
them fail-closed before the sandbox boots; ClusterTrustBundle events
rewrite them while it runs; Checkpoint, Terminate, and failed starts
deregister. No API, proto, or RBAC changes: the wire still carries only
`{name, path}`.
- Informer events only enqueue bundle names; a single run loop writes.
Failed writes requeue with backoff through a rate-limited workqueue;
resolution failures keep last-good contents and wait for the bundle's
next event. Resync (24h) is only a guard against missed watch events.
- Change detection hashes the raw backing contents, not the projected
output (sanitization shuffles). Files are replaced by temp-and-rename at
stable paths and byte-identical files are left untouched: a needless
rewrite replaces an inode that suspended guests re-bind on resume.
- Nothing is persisted. Registrations are in memory; after an atelet
restart, a running actor's files keep their last-written contents until
its next Run/Restore rewrites them from current cluster state.
## virtiofsd: `--migration-on-error=guest-error`
CI confirmed the hazard (3 of 3 micro-VM runs): a rotation renames over
a file whose inode the guest still references (a dcache reference is
enough, no held fd), the actor suspends before re-reading, and
find-paths serializes an inode with no findable path. Under the default
`abort`, the destination virtiofsd rejects the device state at
vm.restore: suspend succeeds, every restore fails, and the actor is
permanently stuck. Full analysis in the PR comments.
With `guest-error` the restore succeeds and only the stale reference is
faulty: EIO on access until a fresh lookup at the stable path heals it.
That matches the documented reader contract (the platform rewrites the
file, applications re-read it) and also covers guest-created
`O_TMPFILE`s held across suspend, which already fail restores under
`abort`. The flag is backend-process configuration, so existing
snapshots (template goldens included) restore unchanged. Trade-off: a
share-reconstruction bug now surfaces as post-resume EIO and a readyz
failure instead of a loud restore failure; virtiofsd names the faulty
inodes in the worker pod log.
## E2E
The identity suite covers live refresh on both sandbox classes: rotate
the pool under two running actors and wait for both to observe the new
bundle live; rotate/suspend/resume without waiting for propagation, so
the resume must deliver rotated contents whether the rewrite landed
before or after the guest went down (the exact `abort` repro); assert a
sibling that never cycled also got the rotation live. The probe fixture
now keeps its namespace when a test fails so the worker pod logs
survive, and the cloud-hypervisor client includes the vm.restore
response body in errors.
## Not in this PR
Auto-injected egress trust volume (#932 PR 3) and the configurable
backend registry (tracked in #932).
Component authors had the metric registry and the cardinality rules but
no single place that says how to add an instrument: which kind to pick,
how to name it, which labels are allowed, how the Go is shaped, how to
test it with a ManualReader, and how to register it. Each of those
conventions existed only as a pattern in one of the metrics.go files.
The guide sits beside tracing.md and otel-collector.md. It uses the
instruments already in the tree as the examples, ends with a worked
example of a node-local snapshot cache (a counter by outcome, a fill
histogram, a size gauge and an eviction counter, and what is
deliberately not a label). The registry snippets in the guide pass
`weaver registry check` when appended to the real registry.
observability.md and AGENTS.md now point at the guide from the place
that already told contributors to update the registry.
Fixes#1552
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This adds a test that serves as a guardrail to prevent changes to the
atespace-scoped tables that would make it difficult or impossible to
partition it in the future.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
cc @bowei @BenTheElder @juli4n @shrutiyam-glitch @thockin
Documents the per-actor usage events channel that #1206 added, closing
the docs request from that review.
New `Per-Actor Usage Events` section in the logging guide:
- an example record and the consumer contract (filter on `msg` + `kind`;
identity rides the same label group as lifecycle events and container
logs, so the guide's existing query dimensions apply unchanged, and one
`labels."ate.actor.uid"` filter returns an actor's output, transitions,
and usage interleaved);
- per-field semantics, including the part consumers must not get wrong:
`memory_current_bytes`/`memory_working_set_bytes` are point-in-time,
while `memory_peak_bytes`/`cpu_usage_usec` accumulate within an epoch
whose boundary depends on `source` (cgroup restarts on restore,
guest-agent survives it) — so window CPU is the increase between
samples, never a sum;
- the sampling knob (`--actor-stats-poll-interval`) and the delivery
contract (best-effort behind a bounded queue, independent of
`--log-level`);
- the cardinality rule in bold: log-based metrics over these events must
never label by actor identity.
The metrics section now names the `ate.actor.stats.*` instruments in its
registry pointer and cross-links here for per-actor detail.
Every technical claim is checked against the code and the
`WorkloadStatsSample` proto contract (epoch scoping, the
`memory.peak`/Linux 5.19 caveat, trace-context absence, drop-warning
text, flag semantics).
Fixes#1475
`Control.CreateWorker` does not validate `Worker.sandbox_class`, so a
client can register a worker with an empty one (and any record written
before this keeps it). `RegisterWorkerCount` tallied that raw value, so
`ate.workerpool.workers` emitted `ate_sandbox_class=""` — not a member
of the `ate.sandbox.class` registry vocabulary — alongside the pool's
seeded series at 0.
This change normalizes the worker's class with the existing
`ateattr.NormalizeSandboxClass`, so an empty class reports as `unknown`.
New `ateattr.SandboxClassAttribute`, used by every emitter. This closes
the same gap on `ate.actor.crashes` and the `ate.actor.lifecycle.*`.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
The architecture guide still described micro-VM container rootfs writes
as living in guest RAM via a tmpfs overlay.
#846 moved that overlay onto the host, served over the same single
virtio-fs share as the durable-dir volumes.
This updates the one stale sentence to match.
Docs only.
Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
Fixes#1508
Store tag snapshots at `<base>/atespaces/<atespace>/tags/<tag-uid>`.
Replace `in_progress_snapshot_uri` with immutable `storage_location`, so
pending and completed tags share UID-based cleanup independent of the
source actor or template.
- [x] Tests pass: race-enabled control API tests and PostgreSQL tag
contract tests.
- [x] Documentation updated.
Lint and code-generation verification also passed.
---------
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
Documentation half of #1526.
## What
Two additions, no code:
- `tools/setup-gcp/README.md`, Create Cluster: a prerequisite saying
node auto-upgrade must be off on any pool that runs workers, and that
those nodes must not be spot or preemptible. It sits next to the
beta-API warning, which is the block the quickstart already points
bring-your-own-cluster users at.
- `docs/upgrade.md`: the runbook assumptions now state the same thing,
and ground rule 2 says what actually happens to an actor when a serving
worker pool is edited, rather than only that you should not edit one.
Scaling a serving pool down is called out explicitly, since that is not
obviously "editing" it.
## Why
An actor that is awake when its worker pod goes away moves to
`ACTOR_STATE_CRASHED`. That state is terminal: `resume` and `suspend`
are both refused, there is no recover verb, and the `latestSnapshot` the
actor still holds cannot be used to start it. The actor has to be
deleted and recreated.
`buildCreateClusterRequest` sets no `Management` on the node pool, so
GKE's default applies and auto-upgrade is on. Nothing in the install
documentation mentions it. A cluster built exactly as documented can
therefore lose actors on Google's maintenance schedule, and the operator
has no way to know that in advance.
Found during acceptance testing on `release-0.1` at `c48b3a3c`, where a
`WorkerPool` scaled from 12 replicas to 9 destroyed two of ten actors
permanently. That scale-down was our mistake and ground rule 2 already
covered it, which is why this PR is scoped to documentation. What the
docs did not cover is that the consequence is permanent, or that a
trigger exists which fires without an operator doing anything.
## Scope
This does not fix the underlying behaviour and is not a substitute for
#1526. Auto-repair, preemption and OOM kills reach the same path and no
prerequisite can close them. Telling operators to disable a standard
Kubernetes safety feature is a reasonable answer for this release and
not a reasonable one for 1.0, so the precondition relaxation in #1526 is
still owed.
## Testing
Docs only. `make test` passes. `make verify` does not complete in my
environment: `hack/verify/codegen.sh` fails creating the Locust codegen
virtualenv because `python3-venv` is not installed locally, which is
unrelated to this change.
## What
Now that the ActorTemplate CRD has been deleted and its resources moved
to the substrate gRPC API and the control-plane store (created/managed
with `kubectl-ate`, persisted in PostgreSQL), several documents still
describe ActorTemplate as a Kubernetes CRD, or describe namespace/RBAC
relationships that no longer exist. This sweeps the docs for those stale
references.
Fixes#368 (docs side).
> This change was prepared with AI assistance; I have reviewed and
tested it.
- [x] Docs and comment-only change; no functional code changed, no tests
affected
`CreateActorTemplate` accepted `../escape` and `/escaped` in
`ActorMetadataItem.path` and `TrustBundleDataSource.path`, and accepted
several data sources projecting to the same path. atelet only rejects
the traversal at actor start, so the template is created but the golden
actor sits in `RESUMING`. atelet never rejects duplicate paths at all:
it writes the files in order and the last one wins.
This PR moves both checks to template validation.
- The path rule lives in a new `internal/volumepath` package that both
ateapi and atelet call, so the two cannot drift. It splits on `/` and
rejects any segment that is empty, `.`, or `..`, caps depth at 16
segments, and rejects NUL bytes. Any other byte is allowed, so Unicode
paths pass. `ActorMetadataItem.path` and `TrustBundleDataSource.path`
apply it through a `+k8s:customValidation` hook, and atelet's
`writeSystemInfoFile` applies it again before touching the host
filesystem.
- `SystemInfoVolumeSource.data_sources` gets a hook that tracks
projected paths in a set and reports a duplicate on repeats, in the same
shape as the Kubernetes projected-volume validation.
The proto doc comment on `data_sources` also says at most one
`actor_metadata` entry may appear. Nothing enforces that, and this PR
leaves it alone.
New cases in `TestValidateSystemInfoVolumeSource` and
`TestValidateTrustBundleDataSource` cover the rejected shapes, and
`internal/volumepath` has its own table test. `hack/verify/codegen.sh`
and `hack/verify/golangci-lint.sh` pass.
#1231 moves `writeSystemInfoFile` into `systeminfovolume.go` and onto
`os.Root`. Whichever PR lands second points that copy at the shared
rule.
Fixes#1436Fixes#1438
Renames the proto message, its status message and scope enum, the five
RPCs and their request and response messages, the actor_snapshot_tag
request fields, and Actor.source_snapshot_tag to source_tag. The store
interface, its Postgres table, the object-storage prefix segment and the
kubectl-ate verbs follow, so nothing keeps the old spelling.
https://github.com/agent-substrate/substrate/issues/664
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes#664
This PR implements the idea described in
https://github.com/agent-substrate/substrate/issues/664#issuecomment-5499311489
It does more than Garbage Collection of snapshots, because we also got
rid of the Snapshot resource (from the DB/API).
Now, an external snapshot is owned by a single resource:
- An Actor owns the snapshot it writes at suspend
- A tag owns a copy taken at tag creation,
- An actor cloned from a tag borrows the tag's snapshot until its own
first suspend.
Garbage Collection: whoever created/owns the snapshot is the only one
who ever deletes them:
i.e., if an actor is deleted and it owns a snapshot. The underlying
snapshot is deleted with the actor.
this PR:
- Drops table actor_snapshots
- Keeps table actor_snapshot_tags
- Adds an object copy at tag creation, and an owned versus borrowed
distinction on the Actor
- Adds synchronous external snapshot deletion at actor suspend, at actor
delete, and at tag delete
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This PR moves sandbox config selection from the WorkerPool to the
ActorTemplate.
The existing behavior is preserved while we are designing the upgrade:
sandbox config still cannot be updated once set (ActorTemplates are
create-only and `sandbox_config` is immutable).
For now the ActorTemplate still *requires* `sandbox_config.config_name`
— there is no resolution of the cluster default (`spec.default`). This
is temporary while we figure out the defaulting design.
- [ ] Tests pass
- [x] Appropriate changes to documentation are included in the PR
`ate.failure.reason` only ever named infrastructure faults, and the
crash log didn't say which actor crashed.
This PR adds the following:
- **`ate.failure.domain`** (`infrastructure` / `workload` / `unknown`)
is available next to every `ate.failure.reason`. It's a strict function
of the reason so it costs no series, but it's *emitted* rather than
derived downstream, so a component ahead of ateapi can report a reason
this build rejects, which collapses to `UNKNOWN`, and a consumer
matching on reason names would file every one of those as infra.
`UNKNOWN` now reports `unknown`, not `infrastructure`.
- **`Actor crashed`** record from ateapi, with full identity including
the uid, via a helper both crash sites share. It sits under the same
guard as the counter so the two can't disagree on how many crashes
happened. The counter is barred from carrying actor identity, so this
record is the only way to attribute a crash to one agent.
- **`WORKLOAD_NOT_READY`** on the readyz deadline, as the first
workload-domain reason. Deadline branch only: a cancellation is ateom
draining, not the actor failing.
Tested locally:
```
desc = WORKLOAD_NOT_READY: readyz for "counter" never returned 200 within 5s
```
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
### Description
This PR introduces native, secure support for using Cloud SQL as the
PostgreSQL store backend for `ate-api-server`.
To ensure the highest level of security and ease of use in GCP
environments, this integration leverages the Cloud SQL Auth Proxy
sidecar with automatic IAM database authentication. This means transport
security (TLS 1.3 tunnel) is handled automatically, and database
sessions are authenticated using Workload Identity via short-lived OAuth
tokens, completely eliminating the need for database passwords.
### Key Changes
* **Cloud SQL Auth Proxy Sidecar:** Added
`manifests/ate-install/cloudsql-proxy-patch.yaml` to patch the sidecar
into the `ate-api-server` deployment when a Cloud SQL instance is
configured.
* **Automated Provisioning:** Extended `tools/setup-gcp` with a new
`cloudsql` command. This handles the idempotent creation of the Cloud
SQL instance, Google Service Accounts (GSA), IAM bindings, and Workload
Identity bindings.
* **Installation Script Updates:** Updated `hack/install-ate.sh` to
parse new environment variables (e.g.,
`ATE_API_POSTGRES_CLOUDSQL_INSTANCE`, `ATE_API_POSTGRES_CLOUDSQL_GSA`)
and correctly synthesize the passwordless DSN and ConfigMaps for the
proxy.
* **Security & Documentation:**
* Added extensive documentation in `tools/setup-gcp/cloud-sql.md`
covering provisioning, schema privileges, deployment, and database
scaling.
* Updated `docs/threat-model.md` to reflect the new Cloud SQL egress
flows and Auth Proxy tunnel mechanics.
* **Dependencies:** Vendored required Google API clients (`sqladmin/v1`,
`servicenetworking/v1`, `iam/v1`) for the GCP setup tool.
- [X] Tests pass
- [X] Appropriate changes to documentation are included in the PR
The runbook for moving a running substrate to a new build without losing
actor state. An user runs the roll by hand with `kubectl`, `kubectl
ate`, `ate-setup`, `jq`, and `grpcurl`, and every piece of upgrade state
lives in cluster objects, so the roll can stop and resume at any point.
The order follows the [upgrade
design](https://docs.google.com/document/d/1JduAyGZFyqdNp4UhKiv0EN-is3iW5tf5BwGTWbt2ouI/edit?usp=sharing&resourcekey=0-MXG12QCkleIxhY7cOB7g6A):
CRDs, then `ate-controller`, then the dataplane node by node, then
`ate-api-server` and `atenet`.
The dataplane moves by version label. The new atelet DaemonSet sits next
to the old one, each serving WorkerPool is cloned with the new worker
image and the new version pin, and each node is drained (workers marked
`DRAINING` through the `DrainWorker` RPC), emptied by suspending its
actors, relabeled, and cleared of old worker pods. The old objects stay
untouched until a separate retire step, so rollback is one label flip
per node.
Tested end to end locally on a GKE cluster: install at one build, run
counter actors, upgrade to a second build following the document, and
confirm the actors resume on the new pool with their state intact.
Rollback and retire were exercised the same way.
Fix#1272, fix#1273.
An sdsmint egress gateway terminates every TLS connection an actor opens
and re-originates it, so what the actor validates is a per-SNI leaf the
gateway minted rather than the origin's certificate. That leaf chains to
the gateway CA and to no public root, so an actor left on its default
trust store fails every HTTPS request it makes.
Fixes#1005
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
#1398 removed the Volume.Type union discriminator, but #1008's counter
demo test still asserts the rendered manifest contains `type:
ExternalVolumeTemplate`, so run-tests fails on every main push since
both landed. Also drops the same stale line from the csi-volumes.md
example.
## Summary
Improves the cold-node activation performance tracked in #811: the
gVisor release extract on a cold node currently exceeds the router
resume budget, so the first resume after a release bump always fails.
- Point the default gvisor SandboxConfig at the 2026-09-02 nightly,
using the `gvisor.tar.zstd` tarballs the nightlies now publish alongside
`gvisor.tar.bz2`. The win is decode speed, not just download size:
stdlib bzip2 is a pure-Go, single-threaded decoder, and extracting the
~165 MB release costs a node roughly 20 seconds of CPU on the first
actor operation it hosts — the dominant term of a cold-node activation.
With zstd the same extraction takes ~2.9 s, measured on a GKE node
running this change. The tarball is also ~22% smaller to download (129
MB vs 166 MB for x86_64).
- Teach `extractTarArchive` to decompress `.tar.zst` / `.tar.zstd`
archives using `github.com/klauspost/compress/zstd` (already a
dependency via ategcs), with test coverage for both suffixes.
- Update the `gvisor.tar.bz2` mentions in docs, manifests comments, and
the SandboxConfig API comment (generated CRD refreshed via
`hack/update/codegen.sh`).
The bucket only publishes sha512 checksums; the pinned sha256 sums were
computed from downloads verified against the published sha512 sums.
## Test plan
- `go test ./cmd/atelet/` (includes new `.tar.zst` / `.tar.zstd`
extraction cases)
- `go build ./cmd/atelet/...`, `go vet`, `gofmt` clean
- Verified the real x86_64 tarball lists `runsc` plus the `gvisor-bin/`
helpers
- Deployed to a GKE cluster: atelet fetched and extracted the 2026-09-02
x86_64 zstd tarball in ~2.9 s (vs ~20 s for bz2)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
A GPU worker pool injected its devices into every container of every actor on it,
which only held up while a worker ran one actor.
A resource name and a count cannot replace it: that shape cannot express which
container gets a device, sharing one between containers, or resuming onto hardware
compatible with the snapshot. A nvidia.com/gpu pool limit is now an ordinary
extended resource -- it places the pod on a GPU node and nothing else reads it.
internal/cdi stays, unused: whatever the API becomes, an ateom still has to merge
CDI edits into an actor's bundle itself.
> [!WARNING]
> Recreate PostgreSQL databases from earlier development builds.
## Summary
This change replaces startup schema setup with embedded, versioned SQL
migrations.
`ateapi` uses Goose to apply migrations before readiness. Goose stores
one ledger record for each applied migration.
Goose runs each migration and inserts its ledger record in one
PostgreSQL transaction.
Closes#901.
Based on this design:
https://docs.google.com/document/d/13ixDKRoAIFXeLxS-_1nikNcAy76ca8m0eVgobfib93E/edit?usp=sharing
## Migration behavior
`ateapi` gets a session advisory lock for the configured schema before
it applies pending migrations.
One replica applies migrations while other replicas wait. Goose reads
the ledger again after it gets the lock.
If a migration fails, PostgreSQL rolls back its SQL and ledger record.
Earlier successful migrations remain applied and recorded.
Kubernetes restarts the failed replica. The next startup resumes from
the first migration without a ledger record.
## Changes
- Add Goose and a per-migration ledger.
- Replace the initial up and down files with one transactional, up-only
migration.
- Keep migration 1 aligned with the current schema, including actor
egress policy storage.
- Remove existence guards and explicit transaction statements from the
baseline migration.
- Apply all pending migrations before `ateapi` becomes ready.
- Serialize each migration run with a PostgreSQL session advisory lock.
- Start without changes when the database schema is current or ahead.
- Reject application tables that do not have a migration ledger.
- Log the starting, current, and latest versions.
- Log the applied migration count and duration.
- Retry only initial database connection failures.
- Return schema and migration errors without a retry.
- Add `--postgres-schema` and `ATE_API_POSTGRES_SCHEMA`.
- Use `public` as the default PostgreSQL schema.
- Use the configured schema for the main and watch pools.
- Restrict outbox partition maintenance to the configured schema.
- Let the installer use an external PostgreSQL database.
- Add the migration design and recovery policy to the repository.
## Migration file policy
Migration files use sequential versions and contain exactly one Goose
`Up` section.
CI rejects down migrations, nontransactional migrations, environment
substitution, explicit transaction control, and `IF NOT EXISTS` guards.
Before the first stable v1 release, developers can change or squash
migrations. Developers must recreate databases after migration history
changes.
After that release, CI rejects changes or deletions against the latest
stable release tag that contains migrations.
Goose does not store migration checksums. The binary embeds each
migration file, and release-tag checks protect released migration
history.
## Compatibility
No release includes PostgreSQL support. The `v0.0.0` release predates
the PostgreSQL backend.
Users must recreate databases from earlier PostgreSQL development
builds.
Every committed migration prefix must work with the current and previous
`ateapi` releases. This rule supports rolling upgrades and temporary
binary rollback.
A binary rollback does not roll back the database schema.
## Testing
Tests cover:
- Fresh database migration.
- Concurrent startup.
- Advisory lock waits.
- Current and ahead database schemas.
- Rejection of application tables without a migration ledger.
- Atomic rollback of a failed migration.
- Retention of earlier successful migrations.
- Resume from the failed migration after restart.
- Configured schema isolation.
- Outbox partition isolation.
- Migration file policy checks.
- Stable release migration immutability.
Fixes#368 . Deletes ActorTemplate CRD and any references to it.
Note about atenet router: It had a k8sclient controller that monitors
ActorTemplate, but the results were not used. Deleted as well.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Follow-up to the review comments on #1288 (merged before they could land
there).
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Changes:
-
`hack/third_party/csi-driver-host-path/deploy/csi-hostpath-testing.yaml`:
same fix as #1288 — the inline `watched_directory` was a no-op, so the
serving cert and trust bundle move to filesystem SDS (SPIFFE SAN pin
kept inline via `combined_validation_context`). SDS requires a node id
and this sidecar passes no `--service-node`, so the bootstrap gains a
`node:` stanza; the `docs/csi-deployment.md` example had the same gap
and gets one too.
- `overlay_test.go` now asserts `watched_directory` appears nowhere in
the emitted cluster block or the patched `envoy.yaml` (comments
excluded) and is present in each SDS resource file, per review.
- Corrected the test comment claiming the sdsmint e2e suite is skipped
in CI: the sdsmint variant is deployed in the MITM lanes; only the
`--experimental-additional-egress-extproc-service` injection has no e2e
coverage.
Verification: `go test ./cmd/ate-setup/...` passes; `envoy --mode
validate` (v1.34) passes on the reworked csi-hostpath bootstrap with
dummy PEMs mounted at the referenced paths.
This PR is very large since it updates all existing demos and benchmark
workloads to use the new ActorTemplate substrate proto.
Please use the "Commits" tab to review individual commits.
Verifications done:
* Used this script: gpaste/5143788763348992 to verify that the change
from CRD -> proto are equivalent.
* The e2e tests are using the new susbtrate resources.
* Picked the parking demo to run e2e manually: gpaste/6193361380311040
Restore timing breakdown is the only unsampled per-actor latency record
we emit, and since traces are head-sampled at 1% on the data plane, and
actor identity is not available from metric labels on purpose.
This adds this to logs:
Before:
```json
{"msg":"Restore timing breakdown",
"actor":{"type":"*ateapipb.Actor","atespace":"demo","name":"counter-1"},
"download":310000000,"oci_unpack":50000000,"ateom_restore":60000000,"total":420000000}
```
After:
```json
{"msg":"Restore timing breakdown",
"ate.atespace":"ate-demo-counter-substrate","ate.actor.name":"join-probe",
"ate.actor.uid":"3a5b4e86-1a77-4da7-84c4-c0993d9c83ec",
"ate.template.atespace":"ate-demo-counter-substrate","ate.template.name":"counter",
"ate.snapshot.scope":"full","ate.snapshot.kind":"golden","ate.sandbox.class":"gvisor",
"ate.actor.restore.duration.volume_mount":0.000004583,
"ate.actor.restore.duration.manifest_fetch":0.006111636,
"ate.actor.restore.duration.sandbox_assets":0.000098958,
"ate.actor.restore.duration.download":0.025386544,
"ate.actor.restore.duration.oci_unpack":0.000954919,
"ate.actor.restore.duration.ateom_restore":0.147448298,
"ate.actor.restore.duration.total":0.181189689,
"trace_id":"c9d7ea9a5f90a5eb4399bfa0b966d5ac","span_id":"89280ba761137c8e"}
```
The duration keys are the `ate.actor.restore.duration` instrument's name
plus an `ate.snapshot.phase` value, so a log key and the histogram it
mirrors are literally the same identifier in source, which is also why
they're seconds.
Also adds `ateattr.ActorLogAttrs`, the component-log twin of
`ActorLogLabels`, held to the same key set by a test.
Verified on kind:
- the ateom Actor restoring record for the same actor carries the
identical `ate.actor.uid` and `trace_id`, this was impossible before
- `ate_actor_restore_duration_seconds_sum{phase="total"} = 0.181189689`,
matching the log to the nanosecond
- induced a download failure by deleting a snapshot object: record sets
`ate.failure.reason:` `FAILED_GET_EXTERNAL_OBJECT` and correctly omitted
`ateom_restore`; the metric for the same restore shows the same six
phases
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
A rolling upgrade needs two substrate versions running in one cluster,
split by node: `atelet` and `ateom` speak a node-local protocol with no
cross-version guarantees, so everything on a node has to come from one
build. Today nothing records which build a node runs, and the `atelet`
DaemonSet is a single fixed-name object, so a second version can only
replace the first in place, instead of our porposed node by node rolling
([design](https://docs.google.com/document/d/1JduAyGZFyqdNp4UhKiv0EN-is3iW5tf5BwGTWbt2ouI/edit?usp=sharing&resourcekey=0-MXG12QCkleIxhY7cOB7g6A)).
This PR is the basis for upgrade. Nodes and the `atelet` DaemonSet get
keyed by an `ate.dev/substrate-version` label whose value is the build
version stamped into the binaries.
### One derivation for the version label
`internal/versionlabel` turns the build version into its two forms in
one place:
| form | grammar | used for |
|---|---|---|
| label value | k8s label value | node labels, DaemonSet labels,
nodeSelectors |
| name suffix | DNS-1123 | the DaemonSet name `atelet-<suffix>` |
A version that is not a valid label value is rejected, because the label
has to match what the `ldflags` stamp put into the binaries through
MAKEFILE and `ko`. It's also exposed shell, so the install script can
use it.
### A DaemonSet per version
`atelet-<suffix>` as the name, the version label on metadata, selector,
and pod template, and a pod nodeSelector on the same label. Two versions
run side by side on disjoint old/new node sets.
### The install becomes version-aware
- `install-ate.sh` reads the version from the same `make ldflags` output
it stamps the binaries with, then fills the manifest placeholders.
- It labels the nodes that exist at install time, nodes come after it
carry no version label yet (so add notes in README). Re-running the
install is idempotent, this is the basis for upgrade runbook later.
- The `README` documents the invariant, how to read the installed
version off the DaemonSet, and that a node added later hosts no workers
until it carries the label. On GKE the node pool label is the birth
default for new nodes - added in `tools/setup-gcp/README.md`.
### Follow-ups
- `ate-setup` need same modification
- minimize the affect for daily developer (pin the `VERSION` as
`{USER}-dev` instead of from git
- the manual rolling-upgrade runbook.
Fix#1270
Ref [design
doc](https://docs.google.com/document/d/1JduAyGZFyqdNp4UhKiv0EN-is3iW5tf5BwGTWbt2ouI/edit?usp=sharing&resourcekey=0-MXG12QCkleIxhY7cOB7g6A)
This PR is large, I grouped changes to the following commits:
- 1d5d8f06: moves consumers off the CRD path: demos, ate-setup scripts
and the e2e suites address templates by the actor_template ref.
- e502cf60: Ate API changes: removes actor_template_namespace,
actor_template_name from Actor, ActorAssignment and ActorSnapshot in the
public API, drops the CRD conversion fallback.
- 8f6c1adb: atecontroller: removes the ActorTemplate CRD controller.
Once this PR is submitted, existing demos that still uses CRD will stop
working. Created https://github.com/agent-substrate/substrate/pull/1355
to update existing demos.
We want to be able to answer questions like:
- image cache hit ratio per node
- cold starts got slower on one node
but nothing we emit says which node anything is on, this PR fixes this.
Verified on a local kind cluster, where both atelet and ateom-gvisor now
show `k8s_node_name` on `target_info`, including ateom's, which arrives
through the OTLP relay.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
* Renames the`ate.template.namespace` attribute to
`ate.template.atespace` and sources it from the actor's `actor_template`
ObjectRef instead of the legacy CRD namespace/name pair.
* Updated functional tests now to use substrate ActorTemplates and bind
actors through the `actor_template` ref.
We should not make workerImage optional as it provide explicit
information for what image workers use, removing it will need
`atecontroller` to inject workerImage and leads to unnecessary control
plane & dataplane wiring.
Fixes#861
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Huy Pham