mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-10-21 06:40:12 +08:00
main
129
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Dmitry Berkovich
|
11afdf6499 |
atelet: serve shared snapshot restores from the node-local file cache (#2796)
Replaces #1551, which was four commits from September that no longer applied: the restore source message, the download path and the module path all changed underneath it. Same design, ported onto today's main, after #2756 and #2889, in four commits, plus three from review: the sharing derived from the snapshot's owner, the gVisor worker evicting a shared file's pages before unlinking it, and the manifest read through the cache. Milestone M2 of the node-local artifact cache from #690: an actor restoring from a shared snapshot, the one a tag holds and every actor created from it starts on, gets its files from a node-level cache instead of its own download. **Sharing is the control plane's knowledge.** `ObjectStorage` gains `SnapshotSharing`, the property rather than the snapshot's kind, so new shared snapshot kinds need no atelet change. ate-api-server sets it on Restore from the snapshot's recorded owner: SHARED for a tag's snapshot, borrowed at creation and immutable, PRIVATE for the actor's own, which its next suspend replaces, and for an unrecorded owner; the local-copy fallback object carries it too. Neither side infers it from the URI. **The cache.** A `filecache.Store` under `/var/lib/ate/snapshot-cache`, behind `--snapshot-cache-dir` (empty disables), `--snapshot-cache-min-age` and `--snapshot-cache-fetch-timeout`, opened at startup with crash debris swept before atelet serves. The directory must be under the node state dir: the object-store plugin writes only there, so a cache elsewhere could never be filled, and startup refuses it. Every flag is validated whether or not the cache is enabled, and the help text says which are read only while it is. The image cache's base-path check moved to `nodepath.UnderBasePath`, which both caches use. **Eviction.** The cache runs on the eviction loop every file cache on the node shares, `cmd/atelet/internal/filecachegc` from #2889: `--snapshot-cache-gc-period`, `--snapshot-cache-max-bytes` and `--snapshot-cache-gc-dry-run`, a budget with a 90% low water, least recently used first, never an entry a restore still links. Its size, evictions and passes appear under the existing `ate.filecache.*` instruments as a second value of `ate.filecache.name`, `snapshot`; no new instrument and no new loop. Every flag is validated whether or not the cache is enabled, as the other flags in atelet are, and the help text says which ones are read only while it is; a disabled cache starts no loop and reports no size. **The restore path.** A restore whose durable snapshot is declared SHARED stages each file through the cache; concurrent restores of one snapshot share a single download, later ones read disk. How a hit is served depends on the sandbox class the request's runtime resolved to. ateworker-gvisor gets hard links. Once the restore's page loading is done the worker evicts the memory and filesystem images from the page cache and unlinks them from its restore dir, keeping only the small descriptors linked: a file with another link is treated like a preserved restore dir, since unlinking it would free nothing and would leave its pages charged to the actor's cgroup, the case #1917 fixed. So a link protects an image's entry through the restore and the min age, not for the actor's life; an entry evicted later is still read to the end through the descriptors runsc holds open, and the next restore reads the images from local disk. ateworker-microvm gets private hole-preserving copies, not because a link would corrupt the cache, it replaces `config.json` by rename and its delta merge copies when the inode is shared, but because a link would cost more than it saves: an on-demand restore keeps `memory-ranges` open for the VM's life, which would pin the entry that long and turn every suspend's in-place merge into a full copy, and the worker's page-cache eviction would drop the cached pages for every other restore; #2948 tracks what links would take. EXDEV on a link, from a cache bind-mounted apart from the restore dirs, degrades to a copy, warned once per process. The manifest of a shared snapshot is read through the cache too, so a restore that finds the snapshot on the node makes no request to object storage; a private snapshot's manifest is read fresh, as the commit marker it is. The cache key is the object's address without its query, so a signed URL's credential reaches neither the key, the cache's errors, `meta.json` nor the logs; and a restore that fails after staging removes its restore-state dir, so the links do not hold the cache's entries until the next lifecycle call. The fetch goes through the object-store plugin as the direct path does, one file into a scratch dir on the cache's filesystem; because the cache then writes the restore dir by path rather than through the plugin's `os.Root`, a symlink waiting there fails the restore instead of being followed. **Not here.** Private snapshots stay uncached by design. Links for ateworker-microvm are #2948, a worker change. The open-and-sweep code the two file caches share in shape is still duplicated, since the snapshot cache can be disabled and degrades to copies across mounts where the asset cache refuses to start. **Tests.** ate-api functional: a golden restore declares SHARED, the actor's own snapshot PRIVATE. ate-api unit: the detach-and-resume test from #2852 compares the whole Restore snapshot, so its fixture now puts the snapshot under the actor's own UID, and it expects PRIVATE for the downloaded copy and nothing for a local one. atelet: flag validation, enabled or not, including the eviction flags, the fetch timeout, and a directory outside the node state dir; two shared manifest reads download once and a private one reads fresh; open disabled, creating its root, sweeping debris; the sharing gate and the class modes; through the plugin-backed test atelet, two link restores share one download and one inode and the file is read-only, copy mode gives private inodes and survives a consumer's mutation, a nil cache downloads, a missing file's error reaches the caller with nothing left behind, a plugin error through the cached modes keeps its code and leaks no credential from a signed URI, a restore that fails at the worker after staging leaves no file in restore-state and no link on the cache's entry, four concurrent restores download once, a file in the checkpoint subdirectory lands there, and symlinks in place of the file or the subdirectory fail; on Linux, a restore dir on another mount gets a copy with no second download and one warning for two restores. ateworker-gvisor, on Linux: staged images hard-linked elsewhere are evicted after the restore wait and only then unlinked, with the other links intact. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Luomu
|
9e9216573b |
ateapi: record a sync suspend's local snapshot copy and resume from it safely (#2883)
Fixes #2938. Synchronous suspend could retain an uploaded checkpoint while controlapi dropped its local-copy record. Keep a matching copy and its node binding while releasing the worker. - Detach pinned actors before runtime fallback so compatible workers elsewhere can preserve memory. - Allow Resume and Revert to forget an unreachable node after the latest local snapshot is durable. PENDING snapshots stay pinned, version conflicts return `Aborted`, and assigned or crashed executions still require termination. - Preserve snapshot identity through template changes, detach, and restore retries. - Record cross-node restores as `snapshot.kind=latest` and consolidate attachment rules in the API field docs. ## Validation Passed on Linux/amd64 with Go 1.27.2 as an unprivileged user: - Focused controlapi and functional regressions with `-race -count=3 -shuffle=on`: 603 passing test entries, no skips. - `go test -race -count=1 ./...`: 11,078 passing test entries across 181 packages. - `go build ./...` - `bash hack/verify-all.sh` The full race suite skipped 44 entries: 38 root-gated, three requiring IPv6 loopback, two requiring a GCS emulator, and one requiring an S3 integration environment. Functional tests use PostgreSQL, envtest, and a fake atelet. New regressions fail without the fixes. Real gVisor/micro-VM E2E and privileged root-gated tests were not run. AI assistance: Codex helped implement the review fixes, review the diff, and run validation. |
||
|
Benjamin Elder
|
9ee109babd |
resource limits and volume capacity are integers now: cpuMillicores, memoryBytes, capacityBytes (#2769)
This replaces the resource.Quantity strings in the API with plain
integers, so ate-api-server no longer parses quantities and every hop
compares numbers. Removes a performance bottleneck in scheduling and
makes the API ~unambiguous.
This came about after a lot of back and forth with @thockin and @juli4n
about the tradeoffs.
- ActorTemplate and container limits: `resources.limits` is now
`{cpuMillicores, memoryBytes}` instead of a name/quantity list
- Worker capacity/allocated and actor assignments use the same
`ResourceList`
- external volumes: `capacity` is now `capacityBytes`
- the atelet and ateworker RPCs carry the same `Resources` type and
field names, instead of `cpu_milli` / `ResourceLimits.cpu_millis`
it's still `limits`, we'll add `requests` next to it later.
`kubectl ate get workers` still prints memory with binary suffixes.
templates and docs are converted, with the friendly unit as a comment
(`memoryBytes: 536870912 # 512Mi`). benchmarking/workloads/deploy.sh
still takes `--actor-cpu 250m` / `--actor-memory 1536Mi` and converts.
#### Breaking change
ActorTemplate manifests must be updated:
```yaml
resources:
limits:
cpuMillicores: 1000
memoryBytes: 536870912 # 512Mi
```
and external volumes use `capacityBytes: 1073741824` instead of
`capacity: 1Gi`. no compatibility with the old form, per our pre-1.0
policy.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This PR was AI assisted. I have reviewed the changes.
|
||
|
igooch
|
49ebabe007 |
Resuming a suspended actor detaches it when its node has no room (#2881)
A suspended actor with a local snapshot is pinned to its node. While that node was full, its resume failed with `ResourceExhausted` even when other nodes had room, until atelet detached the actor on its own after `--snapshot-detach-delay`. Resume now detaches the actor itself, with `DetachActor` from #2852, and restores it on another node. Part of #589. This covers a full node. A node that is gone, the scale-down case, is still open. #### How the move works Assignment still runs pinned to the node first. When it finds no room there, and a worker on another node has room, resume: 1. Calls `DetachActor` on the pinned node. atelet uploads the latest snapshot and deletes the local copies. 2. Records the detach, which marks the upload `COMPLETED` and clears the locality and `assigned_node`. 3. Assigns a worker on any node and restores from the durable snapshot. `DetachActor` is synchronous. If the snapshot isn't durable yet, the resume waits for atelet to upload it. #### What the restore keeps On a worker of the same runtime, the restore keeps the snapshot's fidelity. On a worker of another runtime, a MEMORY snapshot restores at ROOTFS, as for any durable-only actor since #2880. Process memory is lost, but rootfs writes and volumes are kept. ROOTFS and VOLUMES snapshots keep their fidelity on any worker of the same sandbox class. A template can't yet refuse the downgrade; #2818 would add that. #### When the actor doesn't move - No worker anywhere has room. The resume returns `ResourceExhausted` as before, without calling atelet. - The detach fails. The resume fails, and the actor stays SUSPENDED on its node, not crashed, since it can still resume there. - The node is gone. `DetachActor` can't reach atelet, so the resume returns `Unavailable` and the actor stays pinned. ```mermaid sequenceDiagram participant C as client participant A as ate-api-server participant L1 as atelet on node1 (full) participant O as object storage participant L2 as atelet on node2 C->>A: ResumeActor A->>A: assign on node1: no room A->>A: node2 has room A->>L1: DetachActor L1->>O: upload latest snapshot L1->>L1: delete local copies L1-->>A: state with latest_durable only A->>A: mark COMPLETED, clear locality and assigned_node A->>A: reload restore source, assign on node2 A->>L2: Restore(durable snapshot) L2->>O: download L2-->>A: ok A-->>C: RUNNING on node2 ``` #### Design choices - **Any runtime counts as room.** - Memory is an optimization for GA, so a worker that can't restore the snapshot's process memory still triggers the move, and the actor restores there at ROOTFS. - Assignment still prefers a worker that keeps it. - **No claim before the detach.** - Claiming a worker first would leave the claim and the snapshot record disagreeing if ate-api-server crashed before the detach. - In this order, every step ends in a state resume already handles. - Two actors detached for one free place race for it, and the loser has paid an upload, but it ends durable-only and its next resume schedules it onto any node without another detach. #### Also worth knowing - A concurrent update fails the record write with `Aborted`, and the next resume detaches again; atelet returns the same state. If the node has room by then, the resume restores there from the upload instead, and the record keeps the upload `PENDING`, because the detach cancelled atelet's report. - `DetachActor` errors map as the tag workflow maps `PromoteSnapshot` errors, plus atelet's `Aborted` passed through. - A resume that detaches records two scheduler samples, `no_capacity` then `assigned`. A dashboard that reads `no_capacity` as a failed resume will count these. - No API changes. - `TestResumeActor_RelocatesAfterDetach` checked `ResourceExhausted` at a point where node2 already had room, which now detaches. The check moved to before node2 exists. - The functional fake's `DetachActor` returns the named actor's last checkpointed snapshot, not the last of any actor. |
||
|
shrutiyam-glitch
|
34bde5827a |
ateapi, atelet: remove sandbox_assets from Run and Restore RPCs (#2879)
Completes the removal of `sandbox_assets` from the `ate-api-server` -> `atelet` wire protocol following #2780 and #2781. Since `atelet` now resolves the pause image and sandbox binaries for `Run` and `Restore` directly from `sandbox_runtime` via its advertised-runtime index, `ate-api-server` no longer needs to resolve `SandboxConfig` or send `SandboxAssets` on `RunRequest` and `RestoreRequest`. - **`ateletpb` & validation**: Remove `sandbox_assets` from `RunRequest` and `RestoreRequest`, delete `SandboxAssets`, `ArchAssets`, and `AssetFile`, and make `sandbox_runtime` required on both RPCs. - **`cmd/ateapi` & RBAC**: Delete `sandbox_assets.go`, remove `sandboxConfigLister` and `workerPoolLister` from `ActorWorkflow` and `NewRPCService`, drop `sandboxconfigs` from `ate-api-server` RBAC, and require a non-nil `DefaultRuntime` in `scheduler.Applies`. - **`cmd/atelet`**: Read `op.sandboxClass` from `sandbox_runtime` in `Restore`, delete dead `recordFromRequest`, and replace `poolConfigName` with `WorkerPool.ResolvedSandboxConfig()`. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR |
||
|
Sneha-at
|
6a552d590d |
Add a default cluster snapshot storage location with snapshotConfig.storageLocation override in preview mode (#2544)
> The PR title becomes the release note. Write it for users: what changed for them, not how the code changed. #### Breaking change ate-api-server now stores snapshots under a new, operator-set `--snapshot-location` (for example `gs://BUCKET/` or `gs://BUCKET/PREFIX/`). `ActorTemplate.snapshotConfig.storageLocation` is now a preview field, honored only with `--preview=PerTemplateSnapshotLocation`. `snapshotConfig` itself is now optional; an omitted one takes every default. What you must do when upgrading: - **ate-setup installs:** set `SNAPSHOT_LOCATION` (`--snapshot-location` / `storage.snapshotLocation`). ate-setup passes it to ate-api-server and, unless the `PerTemplateSnapshotLocation` preview gate is enabled, fails before touching the cluster if it is unset. `BUCKET_NAME` alone is not enough. The kind profile defaults it to `gs://$BUCKET_NAME/` (`gs://ate-snapshots/`). ate-setup does not create the bucket or grant access to it on GKE (`go run ./tools/setup-gcp bootstrap` does). - **Custom deployments of ate-api-server:** add `--snapshot-location` (or `ATE_API_SNAPSHOT_LOCATION` with `--snapshot-location=@env`). Without it, the server refuses to start unless the `PerTemplateSnapshotLocation` preview gate is enabled. - **Templates that set `storageLocation`:** without the gate, that value is ignored and new snapshots and tags go under `--snapshot-location`. If you rely on per-template buckets or paths (IAM, retention, billing), enable `--preview=PerTemplateSnapshotLocation` to keep the old behavior. With the gate, the field stays optional: a template that leaves it empty uses `--snapshot-location`, and `CreateActorTemplate` fails with `FailedPrecondition` if neither is set. - **Clients that set `storageLocation` with the gate enabled:** they must opt in to preview features (`kubectl ate --preview`, or the `Ate-Preview-Allowed: true` header). Otherwise `CreateActorTemplate` fails with `InvalidArgument`. - **Existing actors:** snapshots already taken keep working for resume. If an actor's snapshot location changes (by upgrading with the gate off, changing the flag, or toggling the gate), its next suspend writes under the new location and releases the previous snapshot from the old one. Until that suspend, changing the actor's `actor_template` is rejected, and deleting the actor collects its snapshots under the old location. If releasing the previous snapshot fails, its objects stay under the old location and a later delete does not collect them; ate-api-server logs a warning naming the snapshot URI. To keep the old locations, enable the gate before upgrading. ActorTemplate is not the right place to specify snapshot location. Bucket is an implementation detail of snapshots, workloads shouldn’t need to care about it. It is the responsibility of substrate admins to provision the buckets and give access to Substrate. This PR adds a cluster wide storage location as a building step towards the final goal of adding these details in a separate SnapshotStorageConfig. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
hajiler
|
73e4d56216 |
csi: reformat external volumes objects (#2653)
Fixes #2407, #2436. > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR ## What changes One rename in the volume API, split out of #2567 so the breaking part lands before GA and the shared-volume feature can follow as a non-breaking addition. **`Volume.external_volume_template` becomes `Volume.external`**, typed `ExternalVolumeSource`, with the existing `ExternalVolumeTemplate` under its `template` member. The field name now matches its siblings (`durable_dir`, `system_info`, `image`) and the message follows the `*VolumeSource` convention (#2407, #2436). The `ExternalVolumes` preview gate moves onto the new member, so gating behaviour is unchanged. `ExternalVolumeSource` is a union with one member today; the shared-volume reference is added as a second member in #2567. ```yaml volumes: - name: data external: template: capacity: 10Gi storageClassName: standard ``` The per-actor record in ActorStatus gets wrapped in a new ExternalVolumes message. This will make changes for shared volume support easier post GA. Callers, tests, demos, benchmark manifests and docs are updated; generated code is regenerated. ## Testing Unit, store (Postgres), controlapi and functional suites pass; `hack/verify/gofmt.sh`, `hack/verify/boilerplate.sh`, `apitool validate` and its exemption test pass. #### Breaking change `ActorTemplate.volumes[].externalVolumeTemplate` is gone; templates must use `volumes[].external.template`. Field number 3 is retired, not reused, so templates stored under the old field decode with no volume source and must be recreated (they fail validation on their next update). Go consumers of `pkg/proto/ateapipb` see the new `ExternalVolumeSource` type and the moved getter. Requesting the `breaking-change` label. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
shrutiyam-glitch
|
bbf30db148 |
ateapi: Downgrade runtime-incompatible durable resumes to ROOTFS (#2880)
Now that gVisor supports `SNAPSHOT_FIDELITY_ROOTFS` alongside micro-VM (#2673), a snapshot's rootfs upper layer depends only on the container image (`ActorTemplate`) and not on the worker's sandbox runtime version or host CPU features. - **`requiredSnapshotRuntime`**: Only impose a `RequiredRuntime` placement constraint when restoring `SNAPSHOT_FIDELITY_MEMORY` snapshots; `SNAPSHOT_FIDELITY_ROOTFS` and `SNAPSHOT_FIDELITY_VOLUMES` snapshots can schedule onto any eligible worker of the same sandbox class without runtime matching. - **`ensureAteletRestored`**: When a durable `MEMORY` snapshot falls back onto a runtime-incompatible worker, downgrade the restore fidelity to `SNAPSHOT_FIDELITY_ROOTFS` (preserving both rootfs writes and durable volumes) instead of `SNAPSHOT_FIDELITY_VOLUMES`. Snapshots whose `ActorTemplate` was replaced continue to downgrade to `SNAPSHOT_FIDELITY_VOLUMES`. - **Tests & Docs**: Update `workflow_resume_test.go`, `functionaltest/actor_test.go`, and `docs/api-guide.md` to reflect `ROOTFS` fallback semantics. Follow-up to #2673 and #2660. <img width="1364" height="1688" alt="image" src="https://github.com/user-attachments/assets/7a105b64-2c30-440f-8679-5afbeb591f7d" /> - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR |
||
|
Luiz Oliveira
|
5952cfd0c5 |
Remove any_state and allow deleting actors in any state (#2727)
#### Breaking change `DeleteActor` now deletes an actor in any state, and the `any_state` field and the `kubectl ate delete actor --any-state` flag are gone. - Callers that relied on `DeleteActor` returning `FAILED_PRECONDITION` for a running actor no longer get that guard. A running actor is now terminated, its volumes detached, and its worker released. To avoid deleting an actor that changed since you read it, check its state yourself before deleting, or pass `options.uid` and `options.version`. - `DeleteActorRequest.options` moved from field 3 to field 2. Regenerate clients from the new `ateapi.proto`. Old stubs that send `options` will have it silently dropped, and old stubs that set `any_state` will fail to parse. - Remove `any_state` / `AnyState` from `DeleteActorRequest` in your code. - Drop `--any-state` from any scripts that call `kubectl ate delete actor`. The flag is now rejected as unknown. Fixes #1161 > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Luomu
|
da4b60ebf2 |
Preserve actors when stale snapshot requests are rejected (#2819)
Late Checkpoint and PromoteSnapshot requests can conflict with a newer snapshot on atelet. Return `Aborted` for stale generations and same-generation UUID conflicts, and have ateapi preserve the actor and its worker assignment while reporting the conflict to the caller. Other `FailedPrecondition` errors retain their existing crash policy. Checkpoint conflicts are handled at the suspend call site. CreateTag maps PromoteSnapshot conflicts in its own upload error handler. Both ateapi and atelet must include this change to avoid crashing actors on snapshot generation conflicts; mixed versions retain the previous crash behavior. Regression tests cover the atelet error-conversion boundary, in-flight and background-operation conflicts, caller cancellation, and synchronous/asynchronous SuspendActor recovery. The API test keeps the last completed snapshot and its objects through repeated conflicts, then retries the same pending snapshot after the conflict clears. The API guide describes the retry behavior and its limits. Fixes #2808. Validated `13e53825`, including main `ddb46651`, on Linux/amd64 with Go 1.27.2 as a non-root user with `CI=true`. - `go test -race -p=4 -count=1 ./...` passed: 167 packages, 10,695 passing test and subtest results, 43 skips, and no failures. - Builds of `ateapi` and `atelet` passed. - `hack/verify-all.sh` passed all 16 checks. Regeneration and verification left the source tree clean. - Focused PostgreSQL/envtest regressions passed. Removing the conflict mappings made the synchronous and asynchronous Checkpoint cases and the tag-upload conflict case fail as expected. Root-gated tests and real runtime cluster E2E were not run. AI assistance: Codex helped implement and test this change. |
||
|
Zoe Zhao
|
ddb46651d6 |
CreateTag uploads a paused actor's snapshot instead of failing (#2817)
CreateTag on a paused actor whose snapshot is still only on its node now uploads that snapshot first, then tags it, instead of failing with FAILED_PRECONDITION. The call can take as long as the upload. - CreateTag asks the atelet on the snapshot's node to upload it with PromoteSnapshot. atelet keeps its local copy, so the actor still resumes on that node. - CreateTag checks that atelet reports the requested snapshot as durable, then records the upload on the actor. - A failed upload fails the tag with no pending tag left behind, and leaves the actor as it was. - The actor lease is held for the whole tag process, so the actor cannot be resumed or suspended during the upload. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Michelle Au
|
3d3b92921b |
atelet: Require snapshot.object on Restore (#2834)
atelet can upload a paused actor's snapshot and detach the actor before ateapi records the upload, since it reports the upload best-effort. ateapi then restored the actor without snapshot.object, atelet failed with NotFound, and the revert that followed deleted the uploaded snapshot as PENDING. ateapi now names the upload location on every local Restore, whether or not it has recorded the upload, and atelet requires it. atelet reads it only when it holds no usable local copy; the manifest, uploaded last, tells a complete upload from a partial one. Part of https://github.com/agent-substrate/substrate/issues/2069 and https://github.com/agent-substrate/substrate/issues/2571 #### Breaking change atelet requires snapshot.object on Restore - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Luomu
|
d2b9750998 |
Run golden snapshots in the template's atespace (#2543)
Closes #2533. Move Golden actors and tags from the shared `ate-golden` atespace into their ActorTemplate's atespace, named after the template UID. Golden resources use the tenant's ordinary authorization, lifecycle policies, and storage grants. Remove the special handling for the dedicated system atespace. Golden capture follows the template's `preferredFidelity`. MEMORY preserves process state. VOLUMES and ROOTFS restore persisted data and start new processes. Durable snapshot metrics use `ate.snapshot.kind=latest`, including Golden snapshots. Tests cover tenant storage ownership, snapshot inheritance, restore, and cleanup. The VOLUMES Golden fixture matches the template's fidelity. ## Validation Validated `204b8fb3`, rebased onto main `7abc3172`, on Linux with Go 1.27.2. The tested source matches the PR's 13,851 tracked files. - Passed `go test -race -count=1 ./...`: 166 packages passed, including PostgreSQL/envtest functional and authorization tests. - Passed builds of ateapi, atelet, and kubectl-ate, race-enabled compilation of the demo E2E binary, and `tools/apitool` race tests. Local validation and actor-attribute race tests also passed. - Passed `hack/verify-all.sh`, including code generation, lint, modules, licenses, metrics, migrations, and formatting. Regeneration left the source unchanged. - Redeployed the dedicated kind cluster and passed real gVisor `TestGoldenSnapshotLifecycle` and `TestDurableDirLifecycle`: Golden MEMORY/VOLUMES creation, tenant storage ownership, inheritance, resume, template cleanup, and durable-directory lifecycle. JUnit reports zero failures and two ROOTFS skips. All Go compilation and tests ran as a non-root user. The runtime copy uses a host-only Linux 5.14 OverlayFS fallback; source checks use the unmodified PR source. Micro-VM E2E and root-gated tests were not run. ## Operator notes - Follow the pre-1.0 reinstall procedure in the [upgrade guide](https://github.com/agent-substrate/substrate/blob/main/docs/upgrade.md). Existing `ate-golden` resources may still own snapshots borrowed by actors. - Object storage grants must cover actor and tag prefixes in each template's atespace. - Golden actors appear in ordinary Actor lists. Lifecycle and ingress operations can interrupt a Golden build. Deleting a published Golden tag invalidates the template's recorded reference. - Update dashboards that select `ate.snapshot.kind=golden`. The `latest` series now includes Golden and other durable snapshots. AI assistance: Codex helped implement and validate this change. |
||
|
Zoe Zhao
|
b5b3481838 |
ateapi: Remove the PauseActor RPC (#2813)
PauseActor only forwarded to an asynchronous SuspendActor. Since #2805 made async the SuspendActor default, the RPC is redundant, and every client in the repository already calls SuspendActor. - Remove the `PauseActor` RPC and its request/response messages, the handler, request validation, the authorization rule, and the apitool exemptions for its messages. - Drop the functional test cases that called the RPC. - Rename tests named after it to `TestSuspendActorAsync_*`, and `workflow_pause_test.go` to `workflow_suspend_async_test.go`. - Rename the benchmark request names `PauseActor`, `PauseActorRecovery` and `PauseActorWarmup` to `SuspendActorAsync*`, along with the Grafana dashboard queries and the benchmarking README. Dashboards won't line up with runs from before the rename. - docs/authorization.md listed a `can_pause` relation the model never defined; it now lists only SuspendActor and RevertActor. ## Breaking change - The `PauseActor` RPC is gone. Clients calling it get `Unimplemented`. Call `SuspendActor` instead: it suspends asynchronously by default, which is what `PauseActor` did. - Benchmark request names `PauseActor`, `PauseActorRecovery` and `PauseActorWarmup` are now `SuspendActorAsync`, `SuspendActorAsyncRecovery` and `SuspendActorAsyncWarmup`. Saved queries or alerts on the old names need updating. |
||
|
Zoe Zhao
|
b40d34f124 |
SuspendActor defaults to suspend asynchronously (#2805)
> The PR title becomes the release note. Write it for users: what changed for them, not how the code changed. SuspendActor without `options.mode` now runs an asynchronous suspend, as the `SuspendOptions` proto comment already documents: the snapshot stays on the worker's node and atelet uploads it in the background. Callers that need the snapshot in object storage before the call returns pass `DURABILITY_MODE_SYNCHRONOUS`. In-repo callers that relied on the old default now set the mode explicitly: - The template reconciler suspends golden actors synchronously, since it tags the snapshot right after. - `kubectl ate suspend actor` takes `--mode=async|sync` and defaults to `async`, matching the API. Pass `--mode=sync` for a suspend that waits for object storage. - A worker-requested suspend keeps the default, now asynchronous. No worker calls it yet, and a TODO tracks letting the Worker choose the mode. - The boomer and locust "suspend" lifecycle modes are synchronous, so they keep measuring a durable suspend. - Functional and e2e tests keep their existing behavior: they were written expecting SuspendActor to be synchronous, so they ask for a synchronous suspend. Most of them also depend on it, because they tag, revert, repoint the template, or migrate right after suspending. #### Breaking change A SuspendActor call that leaves `options.mode` unset no longer waits for the snapshot to reach object storage. Callers that tag the snapshot (`CreateTag`), move the actor to another node, or otherwise need a durable snapshot right after suspending must set `options.mode` to `DURABILITY_MODE_SYNCHRONOUS`. Until the background upload lands, `CreateTag` fails with `FAILED_PRECONDITION`. `kubectl ate suspend actor` without `--mode` no longer waits for the snapshot to reach object storage either; scripts that tag or migrate right after it need `--mode=sync`. Part of https://github.com/agent-substrate/substrate/issues/2069 and https://github.com/agent-substrate/substrate/issues/2571 - [ ] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Eric Curtin
|
3bff7e791a |
gvisor SandboxConfig no longer accepts a bare runsc asset (#2798)
#### Breaking change A gvisor SandboxConfig must give every architecture the `gvisor` release tarball asset. A bare `runsc` asset is now rejected. Replace it with the tarball. Fixes #2697 Removes the bare `runsc` asset from atelet and the admission policy, and moves the atelet tests onto the gVisor release tarball. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Zoe Zhao
|
905745db2c |
SuspendActor returns a suspended actor as it is, and atelet uploads a paused actor's snapshot in the background (#2804)
SUSPENDED is now a single state. SuspendActor returns a SUSPENDED actor as is. It no longer uploads a paused actor's node-local snapshot through atelet's PromoteSnapshot. atelet uploads the snapshot in the background. - Both suspend modes record a pending durable copy of the snapshot. - Tagging an actor whose latest snapshot is not uploaded yet still fails. The error message no longer suggests suspending it. #### Breaking change A synchronous SuspendActor on a paused actor no longer waits for the snapshot upload. It returns the actor right away. Callers that need the durable copy should wait for the snapshot's durable copy to complete. - [ ] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Zoe Zhao
|
20597e8b3c |
Replace clients of PauseActor to call SuspendActor(ASYNCHRONOUS) instead (#2790)
The goal of this PR is to deprecate `PauseActor`. `SuspendActor` with `DURABILITY_MODE_ASYNCHRONOUS` replaces it. The PR has 4 commits: 1. **ateapi: honor SuspendOptions.mode in SuspendActor**: `SuspendActor` now reads `options.mode`. `DURABILITY_MODE_ASYNCHRONOUS` runs the async suspend that `PauseActor` runs, which keeps the snapshot on the worker's node. Any other value, including an unset mode, still runs the sync suspend. `ActorWorkflow.SuspendActor` now takes the mode directly and replaces the separate `PauseActor` workflow entry point. The `PauseActor` RPC still exists and calls the async mode. 2. **Move PauseActor callers to an async SuspendActor**: `kubectl ate pause actor`, the boomer load generators, and the ateapi functional tests now call `SuspendActor` with `DURABILITY_MODE_ASYNCHRONOUS`. The boomer request names (`PauseActor`, `PauseActorRecovery`, `PauseActorWarmup`) stay the same so existing dashboards and result comparisons still line up. 3. **e2e: pause actors with an async SuspendActor**: the e2e demo suites make the same switch. 4. **ateapi: deprecate PauseActor**: marks the RPC `deprecated` in the proto, so staticcheck flags any new caller before `PauseActor` is removed. The two functional tests that still exercise the `PauseActor` handler are exempt from the check. Part of #798 - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR |
||
|
shrutiyam-glitch
|
0443aec36c |
ateapi: remove SandboxConfig from ActorTemplate (#2780)
Follow-up to #2660 and #2755. Sandbox runtime and version selection live on the `WorkerPool`: its `sandboxClasses` entry resolves a `SandboxConfig`, atelet reports the enabled versions on `WorkerStatus` as `defaultRuntime`/`restorableRuntimes`, and the chosen runtime is stamped onto `WorkerAssignment` and `Snapshot` and sent to atelet on Run and Restore. The `ActorTemplate` no longer has a say in which sandbox an actor runs in, so its `sandbox_config` field goes. **Breaking:** `ActorTemplate.sandbox_config` and the `SandboxConfig` message are removed from the Control API. ate-api-server rejects unknown fields, so manifests or clients still sending `sandboxConfig` fail with `InvalidArgument`. Stored templates are unaffected (proto bytes; the old field is carried as unknown). `kubectl ate get actortemplates` loses its `SANDBOX CLASS` column. **API** - Remove `sandbox_config` and the `SandboxConfig` message from `ActorTemplate` in `ateapi.proto`, and its validation from ate-api-server. - Remove `sandboxConfig` from `kubectl-ate` output, e2e fixtures, demo and benchmark `ActorTemplate` manifests (sweperf included), and the docs. **ateapi** - `sandbox_assets` on Run/Restore is unchanged on the wire; ateapi now resolves it from the assigned worker's pool instead of the template: `WorkerAssignment` → `WorkerPool` → the controller-confirmed `SandboxConfig` (`WorkerPool.ResolvedSandboxConfig()`, new) → default version. `ActorWorkflow` takes a `WorkerPoolLister` for this; the `sandboxconfigs` RBAC stays. Marked TODO: this lookup, both listers and the RBAC go once atelet resolves the version from `sandbox_runtime` and `sandbox_assets` is dropped (the follow-up noted in #2755). - Scheduler: `Constraints.SandboxClass` is no longer taken from the template. It is the class of the snapshot the actor restores from (local, else latest durable, golden tags included), empty on a cold boot — so a worker of another class is never a candidate, not even as the volumes-only fallback when `RequiredRuntime` is relaxed. A cold-booting actor gets the class of the pools its `workerSelector` matches. - Repointing `actor_template` on a SUSPENDED actor no longer requires the same SandboxConfig; the snapshot's class and runtime checks on resume cover compatibility. - Metrics: `ate.sandbox.class` on lifecycle and scheduler operations is derived from the actor's assignment or snapshot runtime rather than the template, and omitted when neither is known yet; registry updated. **Behavior change to flag:** a template with no `workerSelector` whose actor has no snapshot can be placed on a pool of any class. Previously `sandboxConfig.sandboxClass` alone pinned the class; now the selector has to. **Tests** - `TestResolveSandboxAssets` covers pool-based resolution (resolved, unresolved pool, missing pool, missing config, no default version). - Scheduler `Applies` cases for the snapshot-class pin; `assignWorkerAttempt` proves a microvm worker is never the fallback for a gvisor snapshot at MEMORY or VOLUMES fidelity. - Wire-request tests seed the pool lister; the "missing SandboxConfig" cases now go through the pool. - Functional tests write the pool's `status.sandboxClasses[].configRef` via `UpdateStatus` (status subresource) and wait for it in the informer. - `ReleaseWindowWorkflows` makes the worker ineligible via `workerSelector` instead of a sandbox-class mismatch. No atelet changes beyond two comments. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Vu Le
|
95b39f3b7e |
test(volume): verify node payload and mock driver publishContext round-trip. (#2094)
Some external storage backends (such as Cloud Filestore IP export rules) scope volume access to the Kubernetes worker node currently hosting an actor via CSI `ControllerPublishVolume` and `ControllerUnpublishVolume`. This PR verifies end-to-end NodeId and PublishContext forwarding across the CSI plugin, control plane resume workflow, and functional actor lifecycle (including cross-node migration). ### Changes - CSI plugin: `TestPlugin_AttachVolume` / `TestPlugin_DetachVolume` assert the `NodeId` and `VolumeId` sent on publish and unpublish, including the `Unimplemented` path. - Mock drivers: `MockVolumePlugin` and the functional test's `attachFailVolumePlugin` return a publish context naming the target node and volume, so existing lifecycle tests now carry a non-empty `publishContext` through Restore. - Lifecycle test: builds on the #1581 pause lifecycle test, renamed `TestPauseActor_VolumeLifecycle_DetachResumeAndMigrate`. It checks the Restore request's `publishContext` after each resume, adds a migration leg (suspend → worker on node2 → resume → attach to node2), and verifies detach, volume deletion, and actor removal on delete. `lastRestoreVolumePublishContext` replaces the per-test lookup of the Restore request. - Unit test: `TestEnsureVolumesAttached_ReturnsPublishContext` also asserts that `AttachVolume` is called once, for the mounted volume, with the worker's node. - [x] Tests pass |
||
|
shrutiyam-glitch
|
05cb6f247b |
atelet: pass SandboxRuntime on Run and Restore (#2755)
Follow up on #587 1. **Pass `SandboxRuntime` on `ateletpb.RunRequest` and `ateletpb.RestoreRequest`**: - Adds `SandboxRuntime sandbox_runtime` to `ateletpb.RunRequest` (field `12`) and `ateletpb.RestoreRequest` (field `17`), along with declarative validation. - Updates `ensureAteletRestored` in `cmd/ateapi/internal/controlapi` to populate `SandboxRuntime` from `WorkerAssignment.SandboxRuntime` on both `Run` and `Restore` requests alongside `SandboxAssets`. This prepares the `ateapi` -> `atelet` wire contract so `atelet` can resolve the selected `SandboxConfig` version directly from the worker assignment's runtime in a follow-up change. 2. **Rename `sandbox_runtime_hash` to `sandbox_bundle_hash`**: - Renames `hardware.AttrSandboxRuntimeHash` (`"sandbox_runtime_hash"`) to `hardware.AttrSandboxBundleHash` (`"sandbox_bundle_hash"`) across `internal/hardware`, `internal/sandboxruntimes`, `internal/ateworker/register`, `cmd/atelet`, `internal/proto/ateletpb`, and `pkg/proto/ateapipb`. - Clarifies that this attribute is the composite SHA-256 hash of a `SandboxVersionConfig`'s runtime bundle (`pauseImage` + per-architecture asset SHA-256 digests), distinguishing it from the full `SandboxRuntime` proto message (which also includes host hardware attributes like `architecture` and `cpu_features`) and from individual `AssetFile.sha256` digests. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Steven LindeandHoda Mohammadi
|
95d1a916a3 |
atelet: move default port to an IANA-unassigned port (#2665)
Changes atelet's default gRPC port (`atelet.DefaultPort`, also its `hostPort`) from 8085 to 21998, so it is less likely to conflict with other processes on the node. atelet is the only Substrate component that binds a port at the node level. Notes for reviewers: - Only node-level ports are in scope. Pod-network listeners keep their defaults, including atunnel's ingress, CONNECT and egress ports on worker pods (443/8443/15001; the router dials `workerIP:443` directly). So do ateapi, the router, the credential providers, and metrics/health. - ateapi dials `atelet.DefaultPort`, so ateapi and atelet need to roll out together. - If Substrate needs more node-level ports later, we recommend 21992 and 21988, in that order. Like 21998, they are not IANA assigned so rarely used. --------- Co-authored-by: Hoda Mohammadi <173203200+hodaaaaaaaaaa@users.noreply.github.com> |
||
|
Tim Hockin
|
68a082c365 |
Use a Go vanity import in the "main" substrate repo: go.ate.dev/substrate (#2181)
This commit changes everything except HTTP(s) links. github.com/agent-substrate/substrate -> go.ate.dev/substrate I chose "go.ate.dev" for brevity, but "pkg.ate.dev" would be OK, too. This is implemented as "CloudFlare Pages" on my personal account. If we like it, we need to make it more "community operated". This does not yet convert other repos. This breaks a TON of open PRs, so we should consider if/when we want to merge it. /cc @BenTheElder @rakyll @EItanya |
||
|
Zoe Zhao
|
7a51ad1796 |
Remove the pause operation label (#2701)
Pause and suspend are being merged into one workflow, so PauseActor and SuspendActor become one lifecycle operation. Label both as "suspend" in metrics, state-change events and crash records, and drop the "pause" value from the ate.actor.operation.name registry. |
||
|
Benjamin Elder
|
63a17a9203 |
rename ateom -> ateworker (#2254)
> The PR title becomes the release note. Write it for users: what changed for them, not how the code changed. #### Breaking change fixes #2065 renames ateom ahead of GA. this is breaking, atelet and the worker images need to be updated together. - images / binaries: ateom-gvisor, ateom-microvm -> ateworker-gvisor, ateworker-microvm - worker pod container: ateom -> worker - RPCs: ateom.Ateom -> ateworker.Worker, atelet.AteomSupport -> atelet.WorkerSupport, atelet.AteomHerder -> atelet.Atelet, MintAteomActorCertificate -> MintWorkerActorCertificate - SPIFFE path: ateom-for-actor -> worker-for-actor - ate.snapshot.phase values: ateom_restore, ateom_checkpoint -> worker_restore, worker_checkpoint - node paths: /var/lib/ate/ateoms/<uid>/ateom.sock -> workers/<uid>/worker.sock, ateom-support.sock -> worker-support.sock - cgroups: the worker's leaf ateom -> worker, the micro-VM guest's /ateomchv -> /actor - authz relation: can_mint_ateom_actor_credential -> can_mint_worker_actor_credential per review, "worker" where any worker implementation fits and "ateworker" for ours. the internal go packages (internal/ateom*) keep their names for now, that's a follow-up. one commit per rename to make review easier. tested with kind e2e (gVisor + micro-VM) on an earlier rebase, plus CI. This PR was AI assisted. I have reviewed the changes. > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Tim Hockin
|
756bb9e15f |
Add a "preview gate" concept, use it for external volumes (#1994)
I went through a number of models and this seems to be the simplest and
lowest overall impact.
To keep it simple:
* All gates are disabled unless enabled
* Only "preview" and "ga" - no "alpha" or "beta"
* We only have one gate for now, called "Preview" (later we can break
this down, maybe)
* All gates can be activated by passing "*"
* Package internal/preview has an Init() function which gets passed a
list of gates to enable, which is stored as a package-global
* Code can call preview.IsEnabled(gatename)
Fixes #1690
@juli4n @BenTheElder @msau42
|
||
|
Benjamin Elder
|
c58f374ad9 |
require actor cpu and memory limits (#2285)
We can still consider more flexible overcommit in the future, but I think leaving actors uncapped is unsafe, and it's hard to pick a reasonable universal default, it's too workload dependent. Actor authors should really be aware of how much resources their actor requires, and tuning them can have significant performance impacts. In the future we could do something like determining how much memory was in use at snapshot time and consider that for scheduling resumes. #### Breaking change ActorTemplates *must* declare CPU/memory limits. Previously `microvm` actors had defaulting, and gvisor actors were simply uncapped. This made the scheduler unaware of placement for gvisor. Requiring it also ensures actor owners are aware of how much resources they cost. Setting actor size will be important for performance particularly with microvms where higher memory limits have a real cost for e.g. snapshot/resume. |
||
|
Eric Curtin
|
2c9a05f480 |
ExternalVolume no longer has a status field (#2456)
#### Breaking change `ExternalVolume.status` is gone. Use `storage_volume_id` (empty until the volume is created) and the actor state instead. Please add the `breaking-change` label. Fixes #2414 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Luiz Oliveira
|
9ec4975913 |
Changed SandboxClass from enum to string to allow extensibility (#2586)
#### Breaking change An ActorTemplate's `sandboxConfig.sandboxClass` is now a string, `gvisor` or `microvm`, the same values the WorkerPool and SandboxConfig CRDs use. The `SANDBOX_CLASS_GVISOR` and `SANDBOX_CLASS_MICROVM` spellings are rejected. To upgrade: - In ActorTemplate manifests and API requests, replace `SANDBOX_CLASS_GVISOR` with `gvisor` and `SANDBOX_CLASS_MICROVM` with `microvm`. - Go clients of `pkg/proto/ateapipb`: the `SandboxClass` enum is removed and `SandboxConfig.SandboxClass` is a `string`. - Existing ActorTemplates read back with an empty `sandboxClass`, because the stored enum value does not decode into the new string field. Templates are immutable, so delete them and their actors before upgrading and recreate them afterwards. - The `SANDBOX CLASS` column of `kubectl ate get actortemplates` now prints `gvisor` or `microvm`. Fixes #2406 |
||
|
ILLUM1N0X
|
db2f833475 |
WorkerService.MintAteomActorJwt mints ateom-for-actor JWTs for atelet; actor JWT claims move from ate.dev to actor.ate.dev (#2523)
Part of #777 The object-store plugin's control sidecar will exchange an actor JWT for a downscoped object-store credential scoped to that actor. atelet needs a way to obtain that JWT over the channel it already authenticates on, which is `WorkerService`, where it already mints the certificate counterpart with `MintAteomActorCertificate`. This adds `MintAteomActorJwt` beside it. Nothing calls it yet. atelet starts to in a later PR, together with the node plugin exchange. Two commits: 1. **Proto.** `MintAteomActorJwtRequest` mirrors the certificate request (actor, UID guard against a deleted-and-recreated actor) plus `audiences` from `Control.MintActorJwt`, each non-empty. The lifetime is fixed at one hour, so there is no expiration field. Response is the token and its expiry. Generated Go and validation code, and the validation wrapper. 2. **Handler.** The caller must be atelet and the actor must exist. The mint is then refused unless the caller's node hosts the actor, as `RequestActorSuspend` already requires. The actor's state decides which status field is trusted, since both can outlive the hosting they record: the node of the assigned worker while the actor is resuming, running or suspending, and `assignedNode` while it is suspended or suspending with no worker and its latest generation is a local checkpoint with no durable copy, which is how a paused actor looks now that `PAUSED` is no longer a state. The whole rule lives in a new `controlapi.HostingNode`, in `hosting.go`, so the suspend and resume workflows' own notion of a paused actor is what it reads, and so the certificate RPC can reuse it in its follow-up. Anything else resolves to no node and is refused: a worker gone from the store, a reverting actor whose assignment outlives its released worker, a crashed or deleting actor whose `assignedNode` is only a record of its last node, and a suspended actor with a stale `assignedNode`. The node check runs before the UID check and every refusal is `NotFound`, so a node learns nothing about actors it does not host, not even that one exists. A UID mismatch is then answered `Aborted`, as `MintAteomActorCertificate` answers it. Each handler composes its own `actoridjwt.Claims` and signs with the actor-id JWT pool, as `Control.MintActorJwt` does. The two tokens differ only in kind: a token minted here speaks for an ateom acting on the actor's behalf, with subject `ateom-for-actor/<atespace>/<name>` and its claims under `ateom-for-actor.ate.dev`, while `Control.MintActorJwt` mints the actor's own token with subject `actor/<atespace>/<name>` and claims under `actor.ate.dev`. The zero kind is rejected, so a token cannot be minted without choosing. A relying party can therefore tell the actor from a Substrate component acting for it, and the snapshot credential exchange will accept only the `ateom-for-actor` kind, so actor code cannot reach its own snapshots. Follow-up, in its own PR: `MintAteomActorCertificate` gets the same host check. It is an existing RPC on the actor boot and resume path, so the change deserves its own review and e2e run. #### Breaking change Actor JWTs no longer carry the `ate.dev` claim. The actor's own token now carries the same fields under `actor.ate.dev`, and the new ateom-for-actor token carries them under `ateom-for-actor.ate.dev`. The OIDC discovery document's `claims_supported` lists the two new names. Any relying party that reads `ate.dev` from an actor JWT, agentgateway included, has to read `actor.ate.dev` instead, and must not accept an `ateom-for-actor` token as the actor. Relying parties should accept both `ate.dev` and `actor.ate.dev` before the API server is upgraded, and drop `ate.dev` an hour after it, once every token minted before the upgrade has expired. Please add the `breaking-change` label. - [x] Tests pass with `go test -race ./...`. New unit tests cover the handler and the claims package: claim contents per kind, `exp` equal to `expire_time`, the host check for a foreign node, a foreign node with a stale UID, a paused actor on each node, a paused actor being suspended, a crashed actor on its last node, an unhosted actor, a lost worker, and each validation and lookup error. - [x] Appropriate changes to documentation are included in the PR. The RPC, messages and both tokens' claims are documented in the proto, and no guide lists WorkerService RPCs. |
||
|
shrutiyam-glitch
|
fd8304189e |
Stamp snapshots with SandboxRuntime and schedule restores on runtime-compatible workers (#2536)
Restoring a sandboxed actor's memory checkpoint (`MEMORY` snapshot fidelity) requires compatibility between the worker that captured the checkpoint and the worker restoring it — across the sandbox class (`gvisor`, `microvm`), the host hardware (`architecture`, CPU features), and the sandbox runtime assets (`gvisor_asset_hash`, microVM kernel/rootfs digests). Fixes #2249 #2250 Following #2381 (where workers began self-reporting their `default_runtime` and `restorable_runtimes` during registration), this PR closes the control-plane loop across checkpointing, scheduling, and restore: 1. **Assignment-Pinned Runtime Stamping**: When an actor is placed on a worker, the control plane records the specific `SandboxRuntime` that will serve that actor on `WorkerAssignment.sandbox_runtime`. Subsequent checkpoints taken during that assignment (`Pause`, `Suspend`, `Tag`) stamp `Snapshot.sandbox_runtime` from the assignment rather than re-reading the worker's live `default_runtime`. This guarantees that if a worker updates its `default_runtime` while an actor is still running under an older runtime version, checkpoints taken from that actor are still stamped with the runtime that actually produced the snapshot. 2. **Runtime-Aware Scheduling with Graceful Fallback**: When resuming an actor from a `MEMORY` snapshot under its current template, the scheduler prefers workers whose `default_runtime` or `restorable_runtimes` match the snapshot's `sandbox_runtime` (with `hardware.Matches` returning `false` if either runtime is `nil`). If no runtime-compatible worker has capacity (and the actor is not pinned to a local pause checkpoint), the scheduler relaxes the runtime constraint so the actor can still make progress on any otherwise-eligible worker. 3. **Consistent Restore Fidelity Downgrade (`MEMORY` -> `VOLUMES`)**: At restore time, compatibility is evaluated directly against the persisted `WorkerAssignment.sandbox_runtime` — the same field that will stamp the next checkpoint — keeping the restore decision and future snapshot stamps in lockstep across retries and re-entries. When the assigned runtime does not match the snapshot's required runtime, all durable snapshots (actor-owned, explicit `source_tag` clones, and borrowed golden snapshots) uniformly downgrade the `RestoreRequest` fidelity to `SNAPSHOT_FIDELITY_VOLUMES` and log a warning. --- ### Scenarios #### 1. Cold Boot (No Snapshot) or Non-Memory (`VOLUMES` / `ROOTFS`) / Repointed Snapshot - **Scheduling**: No memory checkpoint is being restored (`RequiredRuntime == nil`), so any active worker matching the template's `SandboxClass`, selectors, and resource limits is eligible. - **Assignment**: `WorkerAssignment.sandbox_runtime` is set to the bound worker's `default_runtime`. - **Restore & Next Checkpoint**: The actor cold-boots via `Run` (or restores `VOLUMES`/`ROOTFS`), and any subsequent pause/suspend stamps the new snapshot with `WorkerAssignment.sandbox_runtime`. #### 2. Warm Restore on a Matching Worker (`default_runtime` or `restorable_runtimes`) - **Scheduling**: The actor has a stamped `MEMORY` snapshot (taken under the current template). The scheduler filters for workers where either `default_runtime` or an entry in `restorable_runtimes` matches the snapshot's `sandbox_runtime`. - **Assignment**: Under the worker's row lock, `WorkerAssignment.sandbox_runtime` records the exact runtime entry on that worker that matched the snapshot (`default_runtime` on a steady-state worker, or the matched `restorable_runtimes` entry during a runtime rollout). - **Restore & Next Checkpoint**: Because `WorkerAssignment.sandbox_runtime` matches the snapshot's required runtime, `atelet.Restore` runs at full `SNAPSHOT_FIDELITY_MEMORY`. If the actor was restored via a `restorable_runtimes` entry on a worker whose `default_runtime` has rolled forward, its next pause or suspend is still stamped with the matched `restorable_runtimes` entry. #### 3. Durable Snapshot Fallback When Compatible Workers Are Full (Actor-Owned, `source_tag` Clone, or Borrowed Golden Snapshot) - **Scheduling**: The initial `Schedule` pass with `RequiredRuntime` finds no capacity (`ErrNoCapacity`). Because the snapshot is durable (not a node-local pause checkpoint), the workflow relaxes `RequiredRuntime = nil` and re-runs `Schedule`, picking an eligible worker running a different runtime. - **Assignment**: `WorkerAssignment.sandbox_runtime` records the fallback worker's `default_runtime` (or, if a compatible worker freed up between the two `Schedule` calls, the matched runtime from the unrelaxed snapshot requirement). - **Restore & Next Checkpoint**: `ensureAteletRestored` detects that `WorkerAssignment.sandbox_runtime` does not match the snapshot's required runtime, logs a warning, and downgrades the `RestoreRequest` fidelity to `SNAPSHOT_FIDELITY_VOLUMES` so the actor boots a fresh sandbox under the new runtime while preserving its volume state. Subsequent checkpoints are stamped with the new worker's runtime. #### 4. Local Pause Checkpoint on an Incompatible Node - **Scheduling**: A `PAUSED` actor holds a node-local checkpoint on `AssignedNode`. Because nothing releases the node-local checkpoint and claim on fallback yet, `canRelaxRuntime` returns `false`. If no worker on `AssignedNode` supports the snapshot's `sandbox_runtime`, scheduling returns `ResourceExhausted` rather than silently stranding the local checkpoint. #### 5. Concurrent Worker Re-registration & Workflow Re-entry Races - **Stale cache vs. row-locked worker**: If `atelet` re-registers a worker and drops an older runtime from `restorable_runtimes` before the scheduler's watch cache catches up, `BindActorToWorker` evaluates admission under the worker's DB row lock and `assignWorkerAttempt` computes `WorkerAssignment.sandbox_runtime` from that fresh row-locked worker rather than the cached copy. - **Re-entry in `RESUMING`**: If `ResumeActor` fails after persisting `RESUMING` (e.g., during volume attachment) and re-enters later, `validateAssignedWorker` accepts a worker assigned via runtime relaxation, and `ensureAteletRestored` derives `runtimeIncompatible` from the persisted `WorkerAssignment.sandbox_runtime` so the restore fidelity and the next checkpoint's stamp cannot drift even if the worker's advertised runtimes changed in the interim. ### Testing & Docs - **Functional test** (`TestResumeActor_SandboxRuntimePlacementAndFallback`): Exercises stamping, compatible worker placement, full-worker fallback to `SNAPSHOT_FIDELITY_VOLUMES` (for both actor-owned and borrowed golden snapshots), re-stamping on subsequent suspend, and `restorable_runtimes` warm restore + `PauseActor` -> `SuspendActor`-from-`PAUSED`. - **Cluster E2E test** (`internal/e2e/suites/sandboxruntime` — `TestSnapshotRuntimeStamp`): Verifies live worker `DefaultRuntime` stamping across `ResumeActor`, `PauseActor` (local checkpoint + warm restore), `SuspendActor`-from-`PAUSED`, `SuspendActor`-from-`RUNNING`, and `SourceTag` actor clones. - **API Guide** (`docs/api-guide.md`): Documents `Snapshot.sandbox_runtime` and `WorkerAssignment.sandbox_runtime` placement and fallback semantics. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Scott Funkenhauser
|
7469ab4416 |
Mark workers unavailable if it's pod status changes (#2055)
This prevents actors being scheduled on a worker that is restarting or temporarily unreachable. - [ x ] Tests pass - [ x ] Appropriate changes to documentation are included in the PR |
||
|
Zoe Zhao
|
849b423035 |
Define a paused actor by its snapshots and delete the PAUSED and PAUSING states (#2620)
A paused actor no longer has its own state. "Paused" is now defined by the actor's snapshots alone: an actor is paused when it is `ACTOR_STATE_SUSPENDED` and its latest snapshot is local only, that is, it has a `locality` and no `durable_snapshot`. A suspended actor is `ACTOR_STATE_SUSPENDED` with a durable latest snapshot. `ACTOR_STATE_PAUSED` and `ACTOR_STATE_PAUSING` are deleted. This changes how the state is represented, not what the system does. `PauseActor` and `SuspendActor` stay separate workflows and keep sending the same atelet requests. Resume already chose its restore source and node pinning from the snapshot shape (`findLatestLocalSnapshot`, `status.assigned_node`), so it is unchanged apart from its state gate. atelet is untouched. Three commits, one per step: 1. **Pause leaves the actor SUSPENDED with a node-local snapshot.** `FinalizePaused` commits `SUSPENDED` instead of `PAUSED`; `locality`, `assigned_node` and pruning are as before. The workflows that read `PAUSED` now read the shape through three helpers in `snapshot_helpers.go`, all built on `latestSnapshot`: - `latestSnapshotIsLocalOnly`: the paused shape. - `latestSnapshotIsDurable`: the suspended shape (also true with no snapshot at all, which has nothing to upload). - `latestSnapshotIsPendingLocal` (commit 3): a pause in progress. `SuspendActor` returns at once for `SUSPENDED` + durable and uploads the local snapshot for `SUSPENDED` + local only, as it did for `PAUSED`. `PauseActor` is idempotent on the paused shape. `RevertActor` accepts the paused shape and still refuses a durable `SUSPENDED` actor. Changing an actor's template still refuses a paused actor. `ResumeActor`'s state gate accepts `SUSPENDED` only. 2. **Delete `ACTOR_STATE_PAUSED`.** Enum value, `ate.actor.state` metric label `paused`, and the remaining mentions. 3. **Delete `ACTOR_STATE_PAUSING`.** A pause in progress is `SUSPENDING` like a suspend in progress; the latest snapshot tells them apart (a pause has minted a local snapshot with no locality yet and no durable copy, a suspend has a pending durable copy). The pause workflow re-enters on that shape, `SuspendActor` refuses an actor mid-pause as it refused a `PAUSING` one, and a crash during a pause is still attributed to the pause operation. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR |
||
|
Julian Gutierrez Oschmann
|
6f66b356eb |
Wrap page size and next page token in a ListOptions message. (#2593)
Update the API style guide to reflect the change. Part of #1378 |
||
|
Michelle Au
|
2b9516fa9f |
Update atelet Snapshot API for to support asynchronous uploads (#2592)
Part of Actor Lifecycle v2 https://github.com/agent-substrate/substrate/issues/2069 Restructures the atelet Checkpoint, Resume, and UploadPausedCheckpoint RPCs to organize the snapshot fields into a new Snapshot message, along with a new UploadMode to indicate if the upload should be synchronous or asynchronous. The latest snapshots are also added to the responses for reporting which snapshots have been uploaded (asynchronously). This PR is an API rename, there are no behavior changes. Pause will use the new deferred upload mode (which is unimplemented), and Suspend uses the new immediate mode. #### Breaking change atelet Checkpoint, Resume and UploadPausedCheckpoint APIs use new Snapshot messages. UploadPausedCheckpoint has been renamed to PromoteSnapshot. - [ ] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Ganeshkumar Ashokavardhanan
|
1a3ab489db |
Align timestamp fields with AIP-142 naming conventions (#2438)
## Summary Align the two remaining nonconforming public Timestamp field names with the API's AIP naming conventions: - `MintActorJwtResponse.expires_at` becomes `expire_time`. - `GoldenSnapshotStatus.take_golden_snapshot_at` becomes `snapshot_time`. Update in-repository callers and existing tests, regenerate protobuf and declarative validation code, and document the `_time` naming rule. Retain the existing validation, lifecycle, and JWT functional coverage; the additional protobuf serialization test file has been removed. Fixes #2415. ## Breaking change Regenerate clients and update references to these fields: | Before | After | | --- | --- | | `expires_at` / `expiresAt` | `expire_time` / `expireTime` | | `take_golden_snapshot_at` / `takeGoldenSnapshotAt` | `snapshot_time` / `snapshotTime` | | Go `ExpiresAt` / `GetExpiresAt()` | `ExpireTime` / `GetExpireTime()` | | Go `TakeGoldenSnapshotAt` / `GetTakeGoldenSnapshotAt()` | `SnapshotTime` / `GetSnapshotTime()` | Timestamp semantics, types, and protobuf field numbers are unchanged. Old-name aliases are not retained, following the project's pre-1.0 compatibility policy. Please apply the `breaking-change` label. ## Verification - [x] Race-enabled tests for the retained validation, lifecycle, and egress JWT packages; the public protobuf package builds. - [x] `TestMintActorJWT` through the functional API harness. - [x] Repository Linux-aware Go lint. - [x] Validation coverage check. - [x] `hack/verify-all.sh`: all 15 current repository verifiers passed on the latest conflict refresh. - [ ] Full Linux/root/infrastructure test matrix (not run locally). ## CI follow-up The October 12 E2E failure was two timeouts to public `httpbin.org` in the credential-injection suite. Current `main` includes #2535, which replaces that external origin with an in-cluster echo server and restores the test after the earlier upstream quarantine. This PR adds no test skip of its own. ## Contribution note This is a first contribution, opened as a draft for coordination and feedback. GitHub Copilot CLI was used to help prepare the code, tests, and PR description. |
||
|
Julian Gutierrez Oschmann
|
68895abba0 |
Use TitleCase for initialisms in API proto names. (#2520)
The protobuf style guide treats an abbreviation as a single word in
message, enum, service and RPC names (HttpRule, not HTTPRule).
Our protos mixed that with Go-style all-caps initialisms, so generated
Go code mixed TitleCase field names with all-caps type names. For
example:
```
Https: &ateapipb.HTTPSRule{...}
```
Handwritten Go keeps Go's initialism rule (HTTP, TLS, JWT). Only names
generated from protos use TitleCase. It's not ideal, but it's likely the
least bad option.
Fixes #2405
|
||
|
Julian Gutierrez Oschmann
|
9e7d01ef26 |
Make DrainWorker return DrainWorkerResponse (#2501)
`DrainWorker` was the only custom method that returned its resource
directly; every other custom method returns a `{MethodName}Response`
wrapper. A wrapper keeps the API uniform and leaves room for
method-specific output in the future.
The API style guide now requires custom methods to return a
`{MethodName}Response` and no longer lets them return the resource
itself.
Part of #2411
|
||
|
Eric Curtin
|
14bbd8012b |
Singleton subresource Get and Delete requests take a name (#2452)
#### Breaking change `GetActorEgressPolicy`, `DeleteActorEgressPolicy`, `GetAtespaceAccessPolicy`, `DeleteAtespaceAccessPolicy` and `GetGlobalAccessPolicy` now require `name` set to `default`. Please add the `breaking-change` label. Fixes #2363 Adds `name` to those requests, per the style guide. Any other value is `INVALID_ARGUMENT`. `resources.SingletonName` replaces the repeated `"default"` literals. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Michelle Au
|
c4129c94bc |
Simplify Snapshot API for Actor Lifecycle v2 (#2449)
The initial API developed in #2289 had a repeated SnasphotStorage structure keyed by Durability (RESIDENT, LOCAL and DURABLE). However this was overly descriptive. RESIDENT and LOCAL storage details are local to atelet, and the promotion of snapshots from RESIDENT -> LOCAL -> DURABLE is also driven by atelet. Putting the information in Snapshot API meant that every action triggered by atelet has to invoke ateapi first to record the transition. Instead, this PR simplifies the sturcture by having a single SnapshotStorage to contain the durable storage details. Locality details will not be recorded in the Snapshot API. We only record that a snapshot also has locality on a certain node. #### Breaking change LocalSnapshot is removed and replaced by a Snapshot.locality field. Part of #2069 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
cloudonlyandcloudonly
|
29c049fc86 |
Allow ActorTemplates to disable golden snapshot creation (#2328)
ActorTemplates can opt out of automated golden snapshot creation with `snapshotConfig.goldenSnapshotConfig.mode: GOLDEN_SNAPSHOT_MODE_DISABLED`, avoiding unused golden actors and snapshots. Omitting the configuration or leaving the mode `UNSPECIFIED` defaults to `ENABLED`. The mode is an enum with declarative validation, following the existing snapshot fidelity pattern. Disabled templates are skipped during queue resync and reconciliation. Their regular actors can cold boot and suspend/resume using their own snapshots. `kubectl ate get actor-templates` displays `DISABLED` in the `GOLDEN TAG` column. The API guide, field table, glossary, and CLI documentation describe the opt-out. The functional regression starts a real `ActorTemplateReconciler`. An enabled control template proves it is making progress while the disabled template's golden actor and tag both remain absent. The test also verifies the disabled template's actor cold-boots, creates its own durable snapshot on suspend, and restores that snapshot on resume. The disabled reconciliation case is part of the existing table-driven unit test. Implements the golden snapshot opt-out feature discussed in #2068. AI assistance: Codex helped implement and test these changes. ### Validation - macOS arm64: race and coverage tests passed for defaulting, API validation, control-plane unit tests, and CLI printing. - The CLI table regression failed before adding the disabled state and passed after the change. - Linux amd64, Go 1.27.1: `go test -race ./...` passed for all 147 packages with tests. - With `CI=true`, fresh race tests for control-plane functional tests and PostgreSQL storage passed: 612 tests/subtests, zero failures and zero skips. - Removing both disabled-mode guards made the lifecycle regression fail because it found the disabled template's golden actor. The complete implementation passed. - `hack/verify-all.sh`: all 14 verifiers passed with unmodified settings. The first attempt hit a linter package-loading timeout; a longer standalone lint run reported `0 issues`, and the standard verifier run then passed after the cache warmed. - Linux builds of `ateapi` and `kubectl-ate` passed. Functional tests use real PostgreSQL and an envtest Kubernetes API with a fake atelet. Real gVisor E2E was not run. - [x] Appropriate changes to documentation are included in the PR. - [x] Race tests, repository verifiers, and affected binaries were checked. --------- Co-authored-by: cloudonly <28681487+cloudonly@users.noreply.github.com> |
||
|
Julian Gutierrez Oschmann
|
55b6e00e55 |
Rename MintActorJWTRequest.audience to audiences. (#2421)
The field is repeated (so it should be plural), and it's also not aligned with other messages that call it `audiences`. Fixes #2408 |
||
|
Luiz Oliveira
|
9b2ecb9439 |
Rename target_ateom_uid to worker_pod_uid (#2423)
Fixes #2416 |
||
|
Julian Gutierrez Oschmann
|
75e7f8b50a |
Rename some ExternalVolume fields (#2404)
Naming the field `volume_name` is redundant. Rename to to `name`. Also rename `actor_volumes` to `external_volumes`. Part of #1378 |
||
|
Michelle Au
|
fa3fc5bac3 |
Update Snapshots API to support Actor Lifecycle v2 (#2289)
Partially implements the proposed API changes in: https://docs.google.com/document/d/1lQ6fhrRb2sLCeFrvhsfytaoKVbA8X4w0TEwj84pr8uU/edit?tab=t.0 Depends on (and temporily includes) https://github.com/agent-substrate/substrate/pull/2222, review that one first. Part of #2069 This PR tries not to make major changes to existing semantics. They will be done as a followup. Other things in the proposal that are not implemented in this PR (tracked separately) - hardware information (https://github.com/agent-substrate/substrate/issues/2246) - sandbox information - combining pause/resume and introducing detach (https://github.com/agent-substrate/substrate/issues/798) #### Breaking change The Snapshots API has been completely rewritten and replaces - ActorStatus in_progress_snapshot_uri, external_snapshot, local_snapshot, in_progress_local_snapshot_name fields. - TagStatus snapshot, storage_location fields ---- - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR |
||
|
Zoe Zhao
|
4baf18fb03 |
Add sandboxClasses with an optional SandboxConfig ref to WorkerPool (#2332)
WorkerPool now declares its sandbox class as `spec.sandboxClasses`, with
an optional reference to a SandboxConfig:
```yaml
spec:
sandboxClasses:
- name: gvisor
configRef:
name: gvisor-default
```
- `spec.sandboxClass` is replaced by `spec.sandboxClasses`, a list of
`{name, configRef}`. The list is required and must have exactly one
entry for now. `name` is `gvisor` or `microvm` and has no default.
- `configRef` is optional and names a cluster-scoped SandboxConfig.
Nothing reads it yet: actors still get their sandbox binaries from the
ActorTemplate's SandboxConfig.
- Code that needs the pool's class calls
`WorkerPoolSpec.DefaultSandboxClass()`.
#### Breaking change
Every WorkerPool must now set `spec.sandboxClasses`. Once the CRD is
updated, `spec.sandboxClass` is pruned, so re-apply each existing pool
with `sandboxClasses` set (`- name: gvisor` for pools that used the old
default).
- [ ] Tests pass
- [x] Appropriate changes to documentation are included in the PR
|
||
|
Dmitry Berkovich
|
eeec4eedbd |
Replace SnapshotContentScope with the SnapshotFidelity ladder and preferred_fidelity (#2364)
**This is a rename, not a behavior change.** `SnapshotConfig.onCommit` becomes `preferredFidelity`, and the two-valued `SnapshotContentScope` enum becomes the ordered `SnapshotFidelity` enum, with the same vocabulary carried through atelet, ateom, and the metric label. `MEMORY` is exactly what `FULL` was and `VOLUMES` is exactly what `DATA` was: what gets captured on pause and suspend, what gets restored, the golden-actor override, and the paused-upload narrowing are all unchanged. The point is to put the lifecycle-v2 names in place before step III adds `required_fidelity` and `resumed_fidelity` on top of them. The one addition is `ROOTFS`, defined in the enum so it is complete but rejected everywhere, since no runtime can produce it yet. Part of #2102, step II as tracked in #2351. Step I (#2309) removed `on_pause`. Two commits, meant to be read in order. ## 1. ateapi: `SnapshotFidelity` and `preferred_fidelity` - New ordered enum `SnapshotFidelity { VOLUMES = 1, ROOTFS = 2, MEMORY = 3 }`, each level including the ones below it. `SnapshotContentScope` is deleted. MEMORY is the old FULL and VOLUMES the old DATA, so behavior is unchanged. - `SnapshotConfig.on_commit` becomes `preferred_fidelity`, documented as a best-effort ceiling: the system never packages more than this level and may package less when a layer cannot be captured or is too expensive at the time. A process resumed in place keeps its memory regardless. Defaults to MEMORY. Field numbers rearranged to `storage_location = 1`, `preferred_fidelity = 2`. - `ExternalSnapshot` and `LocalSnapshot` record `fidelity` instead of `content_scope`. - ROOTFS is in the enum so it is complete, but no sandbox runtime captures rootfs without memory yet, so a new field-level validator rejects it at template admission. Actors only take their fidelity from a template, so none can carry it. - kubectl-ate tag output, demos, benchmarking manifests, glossary, API guide, and architecture doc use the new vocabulary. The obsolete apitool exemption is removed. ## 2. atelet, ateom: the same enum on the wire - Both internal protos replace `SnapshotScope` with a `SnapshotFidelity` mirroring the public one; request fields become `fidelity` and `desired_fidelity`. Every hop maps level by level, ROOTFS included, and nothing is promoted: an unexpected value goes out as UNSPECIFIED for the next hop to reject. - Defense in depth for ROOTFS: atelet rejects it in checkpoint, restore, and paused-upload validation, and both ateom runtimes reject it on `CheckpointWorkload` and `RestoreWorkload` through a shared `resources.ValidateSnapshotFidelity`. - A golden snapshot must record MEMORY (UNSPECIFIED is no longer tolerated). - The metric attribute `ate.snapshot.scope` (`full`/`data`/`unknown`) becomes `ate.snapshot.fidelity` (`volumes`/`rootfs`/`memory`/`unknown`) in the OTel registry, `ateattr`, the ateom phase logs, and the snapshot manifest atelet writes and reads back when narrowing a paused upload. Dashboards keyed on the old label need updating. ## Behavior changes - **Breaking API change.** Templates that set `onCommit` or use `SNAPSHOT_CONTENT_SCOPE_*` values are rejected. Per the pre-v1.0 policy there is no compatibility shim, and previously stored templates and snapshots are not expected to exist. - Snapshot manifests written with the old `scope` key read back with no fidelity, which the upload path already refuses with a clear message. - Unchanged: what MEMORY and VOLUMES capture and restore, the golden-actor MEMORY override, and the paused-upload narrowing from MEMORY to VOLUMES. ## Relationship to #2289 #2289 restructures the snapshot status into `Snapshot` and `SnapshotStorage`. If it lands first, the `fidelity` field in this PR moves onto `SnapshotStorage` (per copy), which is where it belongs long-term; the enum, the template field, and the data-plane change are the same either way. ## Follow-ups (step III) `required_fidelity` with `required <= preferred` validation and the CRASHED rule, `ResumeActorResponse.resumed_fidelity`, and ROOTFS support in the micro-VM runtime. ## Verification - `go test` for ateapi (unit and functional), kubectl-ate, atelet, ateom-microvm, ateom-gvisor, ateomphaselog, ateattr, resources, and apitool pass; new validator at 100% coverage. - golangci-lint, kube-api-linter, python-protos, boilerplate verifiers pass; codegen, proto-fmt, licenses, go.mod show no drift. - Not run locally: the Weaver metrics-registry check and shellcheck, which need Docker; relying on CI for those. e2e suites compile but were not run against a cluster. - [x] Breaking change - [x] Not backward compatible 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Zoe Zhao
|
9b76a6988a |
SandboxConfig now holds multiple named versions with a default (#2333)
A SandboxConfig can now carry several sandbox versions, with
`spec.defaultVersion` selecting the one actors boot with:
```yaml
kind: SandboxConfig
metadata:
name: gvisor-default
annotations:
sandboxconfig.ate.dev/is-class-default: "true"
spec:
sandboxClass: gvisor
defaultVersion: v1
versions:
- name: v1
state: Enabled
pauseImage: "registry.k8s.io/pause:3.10.2@sha256:..."
assets: {...}
- name: v0
state: Disabled
pauseImage: "registry.k8s.io/pause:3.10.2@sha256:..."
assets: {...}
```
- `spec.pauseImage` and `spec.assets` move into `spec.versions[]`. Each
entry has `name` (a DNS label, unique in the list), `state` (`Enabled`
by default, or `Disabled`), `pauseImage` and `assets`. A config holds
1–16 versions.
- CEL requires `defaultVersion` to name an existing version that is not
`Disabled`, and applies the per-class pauseImage rule to every version.
- The ValidatingAdmissionPolicy checks the per-class required assets on
every version.
- The shipped `gvisor-default` and `microvm` configs now use
`defaultVersion: v1` and carry the
`sandboxconfig.ate.dev/is-class-default` annotation. More than one
config of a class may carry it. Nothing reads it yet; a later change
resolves it newest-wins.
- ate-api and atelet prewarm read only the default version for now.
#### Breaking change
Every SandboxConfig must use the new shape. The apiVersion stays
`v1alpha1` and there is no conversion, so once the CRD is updated,
`spec.pauseImage` and `spec.assets` are pruned from existing objects,
and ate-api rejects actors that name them (FailedPrecondition) until
they are re-applied with `defaultVersion` and `versions`. The install
re-applies `gvisor-default` and `microvm`; custom SandboxConfigs must be
rewritten by hand. Roll out the CRD together with ate-api: an older
ate-api reading a new-shape config gets an empty pause image.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
|
||
|
ILLUM1N0X
|
fa9875f791 |
Move snapshot transfers behind the object-store snapshot plugin (#2224)
Part of #1827 Move external snapshot transfers in atelet and ate-api-server behind the `objectstoresnapshot.v1` plugin API, and ship a reference plugin for GCS and S3 that runs as a sidecar next to each. ## What changes | Commit | Change | |---|---| | objectstoreplugin: add object-store plugin and snapshot-plugin binary | `internal/objectstoreplugin` implements `NodeProvider` and `ControlProvider` on GCS/S3. `cmd/snapshot-plugin` serves one of them on a Unix socket (`node` or `control` mode). A `healthcheck` mode checks a running plugin's gRPC health service, for the sidecar's startup probe. | | atelet: transfer snapshots through NodeProvider | atelet fetches and uploads external snapshots through the node plugin instead of calling object storage. | | ateapi: clean up and copy snapshots through ControlProvider | ate-api-server deletes and copies external snapshots through the control plugin instead of its own storage client. | | manifests: run snapshot-plugin as atelet and ate-api-server sidecars | Native sidecars next to atelet (node mode) and ate-api-server (control mode), each sharing a pod-local emptyDir for the socket. Each sidecar has an exec `startupProbe` running `snapshot-plugin healthcheck` against its socket, so the main container starts only once the plugin serves. | ## Reviewing - **Moved code.** The node plugin's per-file upload and download are atelet's code moved as is, so it can be found in the old location on `main`. The control plugin wraps the existing `objectstore.DeletePrefix` and `CopyPrefix`. New code is limited to the gRPC glue, request validation, the startup health check, the binary, and the manifests. - **Unchanged object layout.** Files are stored as `<file>.zstd` with an uncompressed `manifest.json`, so snapshots written before and after this change are interchangeable. The manifest still goes last, in its own call, and remains the commit marker. - **Validation added at the plugin boundary.** The plugin now serves a socket API, so it checks every request: the snapshot URI, that the local directory is inside `--root` (`/var/lib/ate`), and that file names are base names with no duplicates. Every directory atelet passes today is under `/var/lib/ate`, and the file-name checks are the ones atelet already applied before transferring. - **A plugin that never comes up fails fast.** Clients dial with `WaitForReady`, so without a check the first transfer would hang. atelet and ate-api- server run `objectstoreplugin.WaitReady` (a gRPC health check) at startup with a one-minute bound and exit if the plugin is not serving. The sidecar's `startupProbe` runs that same health check by executing `snapshot-plugin healthcheck` inside the container, so the kubelet does not start atelet or ate-api-server until the plugin serves. Transfer RPCs get no fixed deadlines, since one would be wrong for large snapshots - **Shared backend selection.** atelet's `ATE_STORAGE_BACKEND` selection moves to `objectstorage.NewFromEnv` in `pkg/objectstorage`, along with `UsesS3` and `NewS3ClientFromEnv`, so atelet and both plugin modes select the backend the same way. These are new exported functions in `pkg/`. - **Package location.** The plugin lives in `internal/objectstoreplugin`, not `internal/plugins/`, because atelet and ate-api-server tests import its in-process clients (`objectstoreplugintest`) and it imports `internal/objectstore`. ## Not in this PR - `actor_jwt` is not set yet. Per-actor authorization is #777. ## Breaking change The object storage settings (`ATE_STORAGE_BACKEND`, `AWS_*`) move from the `ate-api-server` container to its snapshot-plugin` sidecar, which is now the only process in that pod that touches object storage for snapshots. ate-api-server exits at startup if `ATE_STORAGE_BACKEND` is still set on it. An install that patches these settings onto the `ate-api-server` container must patch the sidecar instead; see the kind overlay for an S3 example. ## Testing - `make test` and `make verify`. - New unit tests for the plugins (`internal/objectstoreplugin`), backend selection (`pkg/objectstorage`), and the shared validation helpers (`internal/resources`). - E2E on kind runs in CI (gVisor and micro-VM, S3 via rustfs). - E2E on GKE (GCS) |
||
|
Eren Akbulut
|
890234f686 |
Return the caller's context error when an atelet call fails (#2236)
When the caller's context ends during an atelet call, for example because its deadline passes during a slow restore, the control API keeps the actor assigned but returns `Internal` to the caller. So `ResumeActor` can leave an actor `RESUMING` on a worker whose restore already succeeded while the caller receives a server-failure status. When the workflow context has ended, return its error, so the caller gets `Canceled` or `DeadlineExceeded` (the `apierror` convention for context errors). The actor keeps its worker assignment, so a retry can finish the operation. A `Canceled` or `DeadlineExceeded` from atelet with a live context still returns `Internal`, as on main, so the caller is not told that it cancelled. `Unavailable` keeps the mapping from #2196. Permanent errors keep their crash handling. This follows the shape that @laoj2 suggested in review. The unit tests cover a cancelled and an expired workflow context, atelet `Canceled` and `DeadlineExceeded` with a live context, and the existing permanent failures. The gRPC functional test checks `Unavailable`, atelet `Canceled` and `DeadlineExceeded`, and a caller deadline that passes during a slow restore. In each case the actor stays `RESUMING` with its worker assignment, and a retry reaches `RUNNING` on that worker. A permanent local restore failure still returns `Internal`, records a crash, clears the assignment, and releases worker capacity. ### Validation On main `0b91488d79637052458a29a6ba6c757bcf7a9292`, on 2026-10-08, the `controlapi` (51.4 s) and `functionaltest` (64.9 s, `REQUIRE_DOCKER=true`) race suites pass. Linux-targeted `go vet` and `gofmt` are clean. Without the context-error branch, three `TestHandleAteletError` subtests fail. ### Earlier lab evidence These trials used the first version of this PR, which also mapped atelet's own `Canceled` and `DeadlineExceeded` with a live context. With the current version, the second and third rows return `Internal` again. The actor still stays assigned and the retry still works. A separate two-worker ARM64 lab ran three trials per error code. A diagnostic interceptor replaced the reply only after the real atelet `Restore` handler succeeded. It tested selected error replies, rather than physical packet loss. | Injected reply code | Before: client code | With this mapping: client code | Fresh trials with the fix | | --- | --- | --- | ---: | | `Unavailable` | `Unavailable` | `Unavailable` | 3 passed | | `Canceled` | `Internal` | `Canceled` | 3 passed | | `DeadlineExceeded` | `Internal` | `DeadlineExceeded` | 3 passed | All nine trials passed 72 assertions. The actor retained its worker assignment and one observed live guest; explicit retry reached `RUNNING` with the same actor UID, guest execution ID, worker Pod UID, and checkpoint from reply loss through retry. A local AX outcome-reporting candidate consequently retained an uncertain result instead of reporting a definitive failure. Those trials used the unchanged AX candidate `55602feda4accf71d17705d5e6c48bc2c1a5ff1e` on `be7b1a8`, and Substrate mapping commit `438291462203f253366b1e5f8fc4f5b3fd63af9f` on an earlier lab base containing the availability candidate from #2055. This PR contains only the three-file mapping/test patch, freshly checked on current upstream main; the earlier trials are supporting evidence, not a live run of this rebased branch. The richer retry contract discussed in #1562 and automatic worker recovery remain separate work. |