mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-10-20 06:29:56 +08:00
main
346
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Benjamin Elder
|
d4204fd852 |
report actors whose sandbox dies on its own as crashed (#2695)
Follow-up to #2626 The control plane only found out a sandbox died when the next operation on the actor failed, which for an idle actor could be never. - New WorkerService.ReportActorCrashed (and the atelet AteomSupport mirror), authorized like RequestActorSuspend. A RUNNING actor is crashed under its lease, anything else is refused since the operation in flight owns it. - ateworker-microvm reaps cloud-hypervisor / virtiofsd as soon as they exit, and ateworker-gvisor watches the runsc sandbox. If one dies while ateworker isn't stopping it, ateworker tears the actor down itself (the crash clears the assignment, so nothing else would), writes its final usage record, and reports it - Exits are watched through pidfds on the netpoller, so watching doesn't cost a thread per process - The gVisor sandbox is pinned right after runsc create, which closes the pid reuse gap @sfunkenhauser pointed out on #2626 - Not covered yet: a workload exiting while its sandbox stays up, and a micro-VM guest panic, which reboots the guest without cloud-hypervisor exiting. Both need their own design ... Tested with a new e2e test, TestSandboxCrashIsReported (kills the sandbox from an ephemeral container, expects CRASHED, then revert + resume), plus both full lanes on GKE with no false crash reports. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR (none needed) This PR was AI assisted. I have tested everything on GKE and reviewed the changes. |
||
|
igooch
|
97b871f27d |
ateapi: crash the actor when atelet's returned snapshot state conflicts (#2794)
Finalize applies the snapshot state atelet returns from Checkpoint. When that state conflicts with the actor's record, finalize returned an error, and the comment said a retry could call atelet again. A retry can't recover. Atelet has already destroyed the workload, so the retried Checkpoint fails and the actor crashes with a reason that hides the conflict. This PR crashes the actor at the conflict instead. The crash reason names the conflict, matching finalize's other checks for a corrupt record, and atelet's error is logged. No proto or API changes. Earlier versions of this PR also recorded the async upload URI and named it on resume, released dropped durable snapshots, and tested the snapshot's locality in the template-repoint guard. Main now has all of these from other PRs, so this PR keeps only the crash. |
||
|
ILLUM1N0X
|
e988839346 |
A node's per-actor snapshot credential stops working once the actor leaves that node (#2893)
Part of #777 ate-api-server checks that the caller's node hosts the actor when it mints a worker-for-actor JWT, but the token then lives for an hour and the credential exchange trusted any valid one. A node that kept a JWT after the actor resumed elsewhere could still exchange it for a read-write token on the actor's prefix, and the exchange also answered for crashed, deleting or reverting actors. The JWT now carries a `node` claim, the node it was minted for, set by `MintWorkerActorJwt` from the identity it already verified against `HostingNode`. The exchange refuses the token unless that node is still the actor's `status.assignedNode` and the actor is in a state with a transfer: RESUMING, RUNNING, SUSPENDING or SUSPENDED. `assignedNode` is set by a resume, stays set while the actor is SUSPENDED with its local snapshot on that node, and is cleared when the node detaches the actor, by a revert and by a delete, or replaced by a resume onto another node, so a stale JWT stops working at that status write rather than at expiry. An actor's own token never carries a node. The transfer states are listed once, in `resources.TransferStates`, read by both the host check and the exchange, with a test that fails when a new actor state is added without being classified. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Sneha-at
|
6a552d590d |
Add a default cluster snapshot storage location with snapshotConfig.storageLocation override in preview mode (#2544)
> The PR title becomes the release note. Write it for users: what changed for them, not how the code changed. #### Breaking change ate-api-server now stores snapshots under a new, operator-set `--snapshot-location` (for example `gs://BUCKET/` or `gs://BUCKET/PREFIX/`). `ActorTemplate.snapshotConfig.storageLocation` is now a preview field, honored only with `--preview=PerTemplateSnapshotLocation`. `snapshotConfig` itself is now optional; an omitted one takes every default. What you must do when upgrading: - **ate-setup installs:** set `SNAPSHOT_LOCATION` (`--snapshot-location` / `storage.snapshotLocation`). ate-setup passes it to ate-api-server and, unless the `PerTemplateSnapshotLocation` preview gate is enabled, fails before touching the cluster if it is unset. `BUCKET_NAME` alone is not enough. The kind profile defaults it to `gs://$BUCKET_NAME/` (`gs://ate-snapshots/`). ate-setup does not create the bucket or grant access to it on GKE (`go run ./tools/setup-gcp bootstrap` does). - **Custom deployments of ate-api-server:** add `--snapshot-location` (or `ATE_API_SNAPSHOT_LOCATION` with `--snapshot-location=@env`). Without it, the server refuses to start unless the `PerTemplateSnapshotLocation` preview gate is enabled. - **Templates that set `storageLocation`:** without the gate, that value is ignored and new snapshots and tags go under `--snapshot-location`. If you rely on per-template buckets or paths (IAM, retention, billing), enable `--preview=PerTemplateSnapshotLocation` to keep the old behavior. With the gate, the field stays optional: a template that leaves it empty uses `--snapshot-location`, and `CreateActorTemplate` fails with `FailedPrecondition` if neither is set. - **Clients that set `storageLocation` with the gate enabled:** they must opt in to preview features (`kubectl ate --preview`, or the `Ate-Preview-Allowed: true` header). Otherwise `CreateActorTemplate` fails with `InvalidArgument`. - **Existing actors:** snapshots already taken keep working for resume. If an actor's snapshot location changes (by upgrading with the gate off, changing the flag, or toggling the gate), its next suspend writes under the new location and releases the previous snapshot from the old one. Until that suspend, changing the actor's `actor_template` is rejected, and deleting the actor collects its snapshots under the old location. If releasing the previous snapshot fails, its objects stay under the old location and a later delete does not collect them; ate-api-server logs a warning naming the snapshot URI. To keep the old locations, enable the gate before upgrading. ActorTemplate is not the right place to specify snapshot location. Bucket is an implementation detail of snapshots, workloads shouldn’t need to care about it. It is the responsibility of substrate admins to provision the buckets and give access to Substrate. This PR adds a cluster wide storage location as a building step towards the final goal of adding these details in a separate SnapshotStorageConfig. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
hajiler
|
73e4d56216 |
csi: reformat external volumes objects (#2653)
Fixes #2407, #2436. > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR ## What changes One rename in the volume API, split out of #2567 so the breaking part lands before GA and the shared-volume feature can follow as a non-breaking addition. **`Volume.external_volume_template` becomes `Volume.external`**, typed `ExternalVolumeSource`, with the existing `ExternalVolumeTemplate` under its `template` member. The field name now matches its siblings (`durable_dir`, `system_info`, `image`) and the message follows the `*VolumeSource` convention (#2407, #2436). The `ExternalVolumes` preview gate moves onto the new member, so gating behaviour is unchanged. `ExternalVolumeSource` is a union with one member today; the shared-volume reference is added as a second member in #2567. ```yaml volumes: - name: data external: template: capacity: 10Gi storageClassName: standard ``` The per-actor record in ActorStatus gets wrapped in a new ExternalVolumes message. This will make changes for shared volume support easier post GA. Callers, tests, demos, benchmark manifests and docs are updated; generated code is regenerated. ## Testing Unit, store (Postgres), controlapi and functional suites pass; `hack/verify/gofmt.sh`, `hack/verify/boilerplate.sh`, `apitool validate` and its exemption test pass. #### Breaking change `ActorTemplate.volumes[].externalVolumeTemplate` is gone; templates must use `volumes[].external.template`. Field number 3 is retired, not reused, so templates stored under the old field decode with no volume source and must be recreated (they fail validation on their next update). Go consumers of `pkg/proto/ateapipb` see the new `ExternalVolumeSource` type and the moved getter. Requesting the `breaking-change` label. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
shrutiyam-glitch
|
bbf30db148 |
ateapi: Downgrade runtime-incompatible durable resumes to ROOTFS (#2880)
Now that gVisor supports `SNAPSHOT_FIDELITY_ROOTFS` alongside micro-VM (#2673), a snapshot's rootfs upper layer depends only on the container image (`ActorTemplate`) and not on the worker's sandbox runtime version or host CPU features. - **`requiredSnapshotRuntime`**: Only impose a `RequiredRuntime` placement constraint when restoring `SNAPSHOT_FIDELITY_MEMORY` snapshots; `SNAPSHOT_FIDELITY_ROOTFS` and `SNAPSHOT_FIDELITY_VOLUMES` snapshots can schedule onto any eligible worker of the same sandbox class without runtime matching. - **`ensureAteletRestored`**: When a durable `MEMORY` snapshot falls back onto a runtime-incompatible worker, downgrade the restore fidelity to `SNAPSHOT_FIDELITY_ROOTFS` (preserving both rootfs writes and durable volumes) instead of `SNAPSHOT_FIDELITY_VOLUMES`. Snapshots whose `ActorTemplate` was replaced continue to downgrade to `SNAPSHOT_FIDELITY_VOLUMES`. - **Tests & Docs**: Update `workflow_resume_test.go`, `functionaltest/actor_test.go`, and `docs/api-guide.md` to reflect `ROOTFS` fallback semantics. Follow-up to #2673 and #2660. <img width="1364" height="1688" alt="image" src="https://github.com/user-attachments/assets/7a105b64-2c30-440f-8679-5afbeb591f7d" /> - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR |
||
|
Luiz Oliveira
|
5952cfd0c5 |
Remove any_state and allow deleting actors in any state (#2727)
#### Breaking change `DeleteActor` now deletes an actor in any state, and the `any_state` field and the `kubectl ate delete actor --any-state` flag are gone. - Callers that relied on `DeleteActor` returning `FAILED_PRECONDITION` for a running actor no longer get that guard. A running actor is now terminated, its volumes detached, and its worker released. To avoid deleting an actor that changed since you read it, check its state yourself before deleting, or pass `options.uid` and `options.version`. - `DeleteActorRequest.options` moved from field 3 to field 2. Regenerate clients from the new `ateapi.proto`. Old stubs that send `options` will have it silently dropped, and old stubs that set `any_state` will fail to parse. - Remove `any_state` / `AnyState` from `DeleteActorRequest` in your code. - Drop `--any-state` from any scripts that call `kubectl ate delete actor`. The flag is now rejected as unknown. Fixes #1161 > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Luomu
|
da4b60ebf2 |
Preserve actors when stale snapshot requests are rejected (#2819)
Late Checkpoint and PromoteSnapshot requests can conflict with a newer snapshot on atelet. Return `Aborted` for stale generations and same-generation UUID conflicts, and have ateapi preserve the actor and its worker assignment while reporting the conflict to the caller. Other `FailedPrecondition` errors retain their existing crash policy. Checkpoint conflicts are handled at the suspend call site. CreateTag maps PromoteSnapshot conflicts in its own upload error handler. Both ateapi and atelet must include this change to avoid crashing actors on snapshot generation conflicts; mixed versions retain the previous crash behavior. Regression tests cover the atelet error-conversion boundary, in-flight and background-operation conflicts, caller cancellation, and synchronous/asynchronous SuspendActor recovery. The API test keeps the last completed snapshot and its objects through repeated conflicts, then retries the same pending snapshot after the conflict clears. The API guide describes the retry behavior and its limits. Fixes #2808. Validated `13e53825`, including main `ddb46651`, on Linux/amd64 with Go 1.27.2 as a non-root user with `CI=true`. - `go test -race -p=4 -count=1 ./...` passed: 167 packages, 10,695 passing test and subtest results, 43 skips, and no failures. - Builds of `ateapi` and `atelet` passed. - `hack/verify-all.sh` passed all 16 checks. Regeneration and verification left the source tree clean. - Focused PostgreSQL/envtest regressions passed. Removing the conflict mappings made the synchronous and asynchronous Checkpoint cases and the tag-upload conflict case fail as expected. Root-gated tests and real runtime cluster E2E were not run. AI assistance: Codex helped implement and test this change. |
||
|
Zoe Zhao
|
ddb46651d6 |
CreateTag uploads a paused actor's snapshot instead of failing (#2817)
CreateTag on a paused actor whose snapshot is still only on its node now uploads that snapshot first, then tags it, instead of failing with FAILED_PRECONDITION. The call can take as long as the upload. - CreateTag asks the atelet on the snapshot's node to upload it with PromoteSnapshot. atelet keeps its local copy, so the actor still resumes on that node. - CreateTag checks that atelet reports the requested snapshot as durable, then records the upload on the actor. - A failed upload fails the tag with no pending tag left behind, and leaves the actor as it was. - The actor lease is held for the whole tag process, so the actor cannot be resumed or suspended during the upload. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Michelle Au
|
3d3b92921b |
atelet: Require snapshot.object on Restore (#2834)
atelet can upload a paused actor's snapshot and detach the actor before ateapi records the upload, since it reports the upload best-effort. ateapi then restored the actor without snapshot.object, atelet failed with NotFound, and the revert that followed deleted the uploaded snapshot as PENDING. ateapi now names the upload location on every local Restore, whether or not it has recorded the upload, and atelet requires it. atelet reads it only when it holds no usable local copy; the manifest, uploaded last, tells a complete upload from a partial one. Part of https://github.com/agent-substrate/substrate/issues/2069 and https://github.com/agent-substrate/substrate/issues/2571 #### Breaking change atelet requires snapshot.object on Restore - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Luomu
|
d2b9750998 |
Run golden snapshots in the template's atespace (#2543)
Closes #2533. Move Golden actors and tags from the shared `ate-golden` atespace into their ActorTemplate's atespace, named after the template UID. Golden resources use the tenant's ordinary authorization, lifecycle policies, and storage grants. Remove the special handling for the dedicated system atespace. Golden capture follows the template's `preferredFidelity`. MEMORY preserves process state. VOLUMES and ROOTFS restore persisted data and start new processes. Durable snapshot metrics use `ate.snapshot.kind=latest`, including Golden snapshots. Tests cover tenant storage ownership, snapshot inheritance, restore, and cleanup. The VOLUMES Golden fixture matches the template's fidelity. ## Validation Validated `204b8fb3`, rebased onto main `7abc3172`, on Linux with Go 1.27.2. The tested source matches the PR's 13,851 tracked files. - Passed `go test -race -count=1 ./...`: 166 packages passed, including PostgreSQL/envtest functional and authorization tests. - Passed builds of ateapi, atelet, and kubectl-ate, race-enabled compilation of the demo E2E binary, and `tools/apitool` race tests. Local validation and actor-attribute race tests also passed. - Passed `hack/verify-all.sh`, including code generation, lint, modules, licenses, metrics, migrations, and formatting. Regeneration left the source unchanged. - Redeployed the dedicated kind cluster and passed real gVisor `TestGoldenSnapshotLifecycle` and `TestDurableDirLifecycle`: Golden MEMORY/VOLUMES creation, tenant storage ownership, inheritance, resume, template cleanup, and durable-directory lifecycle. JUnit reports zero failures and two ROOTFS skips. All Go compilation and tests ran as a non-root user. The runtime copy uses a host-only Linux 5.14 OverlayFS fallback; source checks use the unmodified PR source. Micro-VM E2E and root-gated tests were not run. ## Operator notes - Follow the pre-1.0 reinstall procedure in the [upgrade guide](https://github.com/agent-substrate/substrate/blob/main/docs/upgrade.md). Existing `ate-golden` resources may still own snapshots borrowed by actors. - Object storage grants must cover actor and tag prefixes in each template's atespace. - Golden actors appear in ordinary Actor lists. Lifecycle and ingress operations can interrupt a Golden build. Deleting a published Golden tag invalidates the template's recorded reference. - Update dashboards that select `ate.snapshot.kind=golden`. The `latest` series now includes Golden and other durable snapshots. AI assistance: Codex helped implement and validate this change. |
||
|
Zoe Zhao
|
b5b3481838 |
ateapi: Remove the PauseActor RPC (#2813)
PauseActor only forwarded to an asynchronous SuspendActor. Since #2805 made async the SuspendActor default, the RPC is redundant, and every client in the repository already calls SuspendActor. - Remove the `PauseActor` RPC and its request/response messages, the handler, request validation, the authorization rule, and the apitool exemptions for its messages. - Drop the functional test cases that called the RPC. - Rename tests named after it to `TestSuspendActorAsync_*`, and `workflow_pause_test.go` to `workflow_suspend_async_test.go`. - Rename the benchmark request names `PauseActor`, `PauseActorRecovery` and `PauseActorWarmup` to `SuspendActorAsync*`, along with the Grafana dashboard queries and the benchmarking README. Dashboards won't line up with runs from before the rename. - docs/authorization.md listed a `can_pause` relation the model never defined; it now lists only SuspendActor and RevertActor. ## Breaking change - The `PauseActor` RPC is gone. Clients calling it get `Unimplemented`. Call `SuspendActor` instead: it suspends asynchronously by default, which is what `PauseActor` did. - Benchmark request names `PauseActor`, `PauseActorRecovery` and `PauseActorWarmup` are now `SuspendActorAsync`, `SuspendActorAsyncRecovery` and `SuspendActorAsyncWarmup`. Saved queries or alerts on the old names need updating. |
||
|
Zoe Zhao
|
b40d34f124 |
SuspendActor defaults to suspend asynchronously (#2805)
> The PR title becomes the release note. Write it for users: what changed for them, not how the code changed. SuspendActor without `options.mode` now runs an asynchronous suspend, as the `SuspendOptions` proto comment already documents: the snapshot stays on the worker's node and atelet uploads it in the background. Callers that need the snapshot in object storage before the call returns pass `DURABILITY_MODE_SYNCHRONOUS`. In-repo callers that relied on the old default now set the mode explicitly: - The template reconciler suspends golden actors synchronously, since it tags the snapshot right after. - `kubectl ate suspend actor` takes `--mode=async|sync` and defaults to `async`, matching the API. Pass `--mode=sync` for a suspend that waits for object storage. - A worker-requested suspend keeps the default, now asynchronous. No worker calls it yet, and a TODO tracks letting the Worker choose the mode. - The boomer and locust "suspend" lifecycle modes are synchronous, so they keep measuring a durable suspend. - Functional and e2e tests keep their existing behavior: they were written expecting SuspendActor to be synchronous, so they ask for a synchronous suspend. Most of them also depend on it, because they tag, revert, repoint the template, or migrate right after suspending. #### Breaking change A SuspendActor call that leaves `options.mode` unset no longer waits for the snapshot to reach object storage. Callers that tag the snapshot (`CreateTag`), move the actor to another node, or otherwise need a durable snapshot right after suspending must set `options.mode` to `DURABILITY_MODE_SYNCHRONOUS`. Until the background upload lands, `CreateTag` fails with `FAILED_PRECONDITION`. `kubectl ate suspend actor` without `--mode` no longer waits for the snapshot to reach object storage either; scripts that tag or migrate right after it need `--mode=sync`. Part of https://github.com/agent-substrate/substrate/issues/2069 and https://github.com/agent-substrate/substrate/issues/2571 - [ ] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Eric Curtin
|
3bff7e791a |
gvisor SandboxConfig no longer accepts a bare runsc asset (#2798)
#### Breaking change A gvisor SandboxConfig must give every architecture the `gvisor` release tarball asset. A bare `runsc` asset is now rejected. Replace it with the tarball. Fixes #2697 Removes the bare `runsc` asset from atelet and the admission policy, and moves the atelet tests onto the gVisor release tarball. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Zoe Zhao
|
905745db2c |
SuspendActor returns a suspended actor as it is, and atelet uploads a paused actor's snapshot in the background (#2804)
SUSPENDED is now a single state. SuspendActor returns a SUSPENDED actor as is. It no longer uploads a paused actor's node-local snapshot through atelet's PromoteSnapshot. atelet uploads the snapshot in the background. - Both suspend modes record a pending durable copy of the snapshot. - Tagging an actor whose latest snapshot is not uploaded yet still fails. The error message no longer suggests suspending it. #### Breaking change A synchronous SuspendActor on a paused actor no longer waits for the snapshot upload. It returns the actor right away. Callers that need the durable copy should wait for the snapshot's durable copy to complete. - [ ] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Zoe Zhao
|
20597e8b3c |
Replace clients of PauseActor to call SuspendActor(ASYNCHRONOUS) instead (#2790)
The goal of this PR is to deprecate `PauseActor`. `SuspendActor` with `DURABILITY_MODE_ASYNCHRONOUS` replaces it. The PR has 4 commits: 1. **ateapi: honor SuspendOptions.mode in SuspendActor**: `SuspendActor` now reads `options.mode`. `DURABILITY_MODE_ASYNCHRONOUS` runs the async suspend that `PauseActor` runs, which keeps the snapshot on the worker's node. Any other value, including an unset mode, still runs the sync suspend. `ActorWorkflow.SuspendActor` now takes the mode directly and replaces the separate `PauseActor` workflow entry point. The `PauseActor` RPC still exists and calls the async mode. 2. **Move PauseActor callers to an async SuspendActor**: `kubectl ate pause actor`, the boomer load generators, and the ateapi functional tests now call `SuspendActor` with `DURABILITY_MODE_ASYNCHRONOUS`. The boomer request names (`PauseActor`, `PauseActorRecovery`, `PauseActorWarmup`) stay the same so existing dashboards and result comparisons still line up. 3. **e2e: pause actors with an async SuspendActor**: the e2e demo suites make the same switch. 4. **ateapi: deprecate PauseActor**: marks the RPC `deprecated` in the proto, so staticcheck flags any new caller before `PauseActor` is removed. The two functional tests that still exercise the `PauseActor` handler are exempt from the check. Part of #798 - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR |
||
|
Michelle Au
|
fb376fcaf7 |
ateapi: Record the async upload of a local snapshot (#2795)
A pause records no upload location for its snapshot, but atelet uploads it in the background to the location the pause gave it. The next pause's Checkpoint response names that upload, and ate-api-server refused it, leaving the actor SUSPENDING. ate-api-server now records the upload as COMPLETED when its location is the one ate-api-server built for that snapshot. Part of https://github.com/agent-substrate/substrate/issues/2069 and https://github.com/agent-substrate/substrate/issues/2571 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
shrutiyam-glitch
|
0443aec36c |
ateapi: remove SandboxConfig from ActorTemplate (#2780)
Follow-up to #2660 and #2755. Sandbox runtime and version selection live on the `WorkerPool`: its `sandboxClasses` entry resolves a `SandboxConfig`, atelet reports the enabled versions on `WorkerStatus` as `defaultRuntime`/`restorableRuntimes`, and the chosen runtime is stamped onto `WorkerAssignment` and `Snapshot` and sent to atelet on Run and Restore. The `ActorTemplate` no longer has a say in which sandbox an actor runs in, so its `sandbox_config` field goes. **Breaking:** `ActorTemplate.sandbox_config` and the `SandboxConfig` message are removed from the Control API. ate-api-server rejects unknown fields, so manifests or clients still sending `sandboxConfig` fail with `InvalidArgument`. Stored templates are unaffected (proto bytes; the old field is carried as unknown). `kubectl ate get actortemplates` loses its `SANDBOX CLASS` column. **API** - Remove `sandbox_config` and the `SandboxConfig` message from `ActorTemplate` in `ateapi.proto`, and its validation from ate-api-server. - Remove `sandboxConfig` from `kubectl-ate` output, e2e fixtures, demo and benchmark `ActorTemplate` manifests (sweperf included), and the docs. **ateapi** - `sandbox_assets` on Run/Restore is unchanged on the wire; ateapi now resolves it from the assigned worker's pool instead of the template: `WorkerAssignment` → `WorkerPool` → the controller-confirmed `SandboxConfig` (`WorkerPool.ResolvedSandboxConfig()`, new) → default version. `ActorWorkflow` takes a `WorkerPoolLister` for this; the `sandboxconfigs` RBAC stays. Marked TODO: this lookup, both listers and the RBAC go once atelet resolves the version from `sandbox_runtime` and `sandbox_assets` is dropped (the follow-up noted in #2755). - Scheduler: `Constraints.SandboxClass` is no longer taken from the template. It is the class of the snapshot the actor restores from (local, else latest durable, golden tags included), empty on a cold boot — so a worker of another class is never a candidate, not even as the volumes-only fallback when `RequiredRuntime` is relaxed. A cold-booting actor gets the class of the pools its `workerSelector` matches. - Repointing `actor_template` on a SUSPENDED actor no longer requires the same SandboxConfig; the snapshot's class and runtime checks on resume cover compatibility. - Metrics: `ate.sandbox.class` on lifecycle and scheduler operations is derived from the actor's assignment or snapshot runtime rather than the template, and omitted when neither is known yet; registry updated. **Behavior change to flag:** a template with no `workerSelector` whose actor has no snapshot can be placed on a pool of any class. Previously `sandboxConfig.sandboxClass` alone pinned the class; now the selector has to. **Tests** - `TestResolveSandboxAssets` covers pool-based resolution (resolved, unresolved pool, missing pool, missing config, no default version). - Scheduler `Applies` cases for the snapshot-class pin; `assignWorkerAttempt` proves a microvm worker is never the fallback for a gvisor snapshot at MEMORY or VOLUMES fidelity. - Wire-request tests seed the pool lister; the "missing SandboxConfig" cases now go through the pool. - Functional tests write the pool's `status.sandboxClasses[].configRef` via `UpdateStatus` (status subresource) and wait for it in the informer. - `ReleaseWindowWorkflows` makes the worker ineligible via `workerSelector` instead of a sandbox-class mismatch. No atelet changes beyond two comments. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
igooch
|
6275fa01b0 |
ate-api-server applies the snapshot state atelet returns when a suspend finalizes (#2766)
Part of Actor Lifecycle v2 #2069. atelet's `Checkpoint` and `PromoteSnapshot` return its snapshot state for the actor (#2677). The suspend workflow discarded it. This applies it when a suspend finalizes, with the same rules as a `ReportActorSnapshotState` report (#2622, #2698). No proto changes. - `ensureAteletSuspended` and `ensurePausedSnapshotUploaded` return the state atelet reports. - `ensureSuspendedFinalized` applies it with `applyCheckpointState` before it frees the worker, and records the result in the same update that commits `SUSPENDED`. - A durable snapshot records the fidelity atelet reports. For `Checkpoint` and `PromoteSnapshot`, that's the one requested. - An upload atelet reports finished for an older generation is marked `COMPLETED`, so finalize keeps it rather than pruning it as `PENDING`. - `applyLatestDurable` refuses a reported durable snapshot with no fidelity, rather than recording `UNSPECIFIED`. atelet echoes the fidelity it was asked for, and reports already require one. #### Notes for reviewers - **The state is never stale.** Every attempt that reaches finalize has just called atelet, so a retried suspend applies the state from its own call. - **A conflicting state fails the suspend.** Finalize applies atelet's state before it frees the worker. If the state conflicts with the record, for example a different upload URI, finalize returns the error and the actor stays `SUSPENDING` with its worker, so a retry can call atelet again. - **The lifecycle-op metric's fidelity label** stays the requested fidelity. - **With today's atelet, the record doesn't change.** The applied state matches what finalize already records. It starts to matter once atelet uploads in the background: an async suspend's response can report an older snapshot as uploaded before atelet's report arrives, and finalize then keeps that snapshot as `COMPLETED` instead of pruning it. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR |
||
|
ILLUM1N0X
|
81224dd01e |
Suspending a paused actor can mint the per-actor credential for its snapshot upload (#2776)
Part of #777 The suspend workflow marks a paused actor's local snapshot as a PENDING durable copy in the same write that sets SUSPENDING, then asks the node that holds it to upload it through `PromoteSnapshot`. `HostingNode` trusted `assigned_node` only when the latest generation had no durable copy at all, so that node's `MintWorkerActorJwt` for the upload was refused with `NotFound`. The unit test for this case used a fixture with no durable copy, a shape the workflow never produces, so it passed. `HostingNode` now accepts a durable copy that is absent or PENDING, and still refuses a COMPLETED one, after which the node has nothing left to do for the actor. The fixture carries the PENDING copy, and a direct test pins the six no-worker shapes. Nothing changes for users today, since atelet does not yet mint for its transfers. The node plugin PR that follows depends on this. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Vu Le
|
95b39f3b7e |
test(volume): verify node payload and mock driver publishContext round-trip. (#2094)
Some external storage backends (such as Cloud Filestore IP export rules) scope volume access to the Kubernetes worker node currently hosting an actor via CSI `ControllerPublishVolume` and `ControllerUnpublishVolume`. This PR verifies end-to-end NodeId and PublishContext forwarding across the CSI plugin, control plane resume workflow, and functional actor lifecycle (including cross-node migration). ### Changes - CSI plugin: `TestPlugin_AttachVolume` / `TestPlugin_DetachVolume` assert the `NodeId` and `VolumeId` sent on publish and unpublish, including the `Unimplemented` path. - Mock drivers: `MockVolumePlugin` and the functional test's `attachFailVolumePlugin` return a publish context naming the target node and volume, so existing lifecycle tests now carry a non-empty `publishContext` through Restore. - Lifecycle test: builds on the #1581 pause lifecycle test, renamed `TestPauseActor_VolumeLifecycle_DetachResumeAndMigrate`. It checks the Restore request's `publishContext` after each resume, adds a migration leg (suspend → worker on node2 → resume → attach to node2), and verifies detach, volume deletion, and actor removal on delete. `lastRestoreVolumePublishContext` replaces the per-test lookup of the Restore request. - Unit test: `TestEnsureVolumesAttached_ReturnsPublishContext` also asserts that `AttachVolume` is called once, for the mounted volume, with the worker's node. - [x] Tests pass |
||
|
shrutiyam-glitch
|
05cb6f247b |
atelet: pass SandboxRuntime on Run and Restore (#2755)
Follow up on #587 1. **Pass `SandboxRuntime` on `ateletpb.RunRequest` and `ateletpb.RestoreRequest`**: - Adds `SandboxRuntime sandbox_runtime` to `ateletpb.RunRequest` (field `12`) and `ateletpb.RestoreRequest` (field `17`), along with declarative validation. - Updates `ensureAteletRestored` in `cmd/ateapi/internal/controlapi` to populate `SandboxRuntime` from `WorkerAssignment.SandboxRuntime` on both `Run` and `Restore` requests alongside `SandboxAssets`. This prepares the `ateapi` -> `atelet` wire contract so `atelet` can resolve the selected `SandboxConfig` version directly from the worker assignment's runtime in a follow-up change. 2. **Rename `sandbox_runtime_hash` to `sandbox_bundle_hash`**: - Renames `hardware.AttrSandboxRuntimeHash` (`"sandbox_runtime_hash"`) to `hardware.AttrSandboxBundleHash` (`"sandbox_bundle_hash"`) across `internal/hardware`, `internal/sandboxruntimes`, `internal/ateworker/register`, `cmd/atelet`, `internal/proto/ateletpb`, and `pkg/proto/ateapipb`. - Clarifies that this attribute is the composite SHA-256 hash of a `SandboxVersionConfig`'s runtime bundle (`pauseImage` + per-architecture asset SHA-256 digests), distinguishing it from the full `SandboxRuntime` proto message (which also includes host hardware attributes like `architecture` and `cpu_features`) and from individual `AssetFile.sha256` digests. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Steven LindeandHoda Mohammadi
|
95d1a916a3 |
atelet: move default port to an IANA-unassigned port (#2665)
Changes atelet's default gRPC port (`atelet.DefaultPort`, also its `hostPort`) from 8085 to 21998, so it is less likely to conflict with other processes on the node. atelet is the only Substrate component that binds a port at the node level. Notes for reviewers: - Only node-level ports are in scope. Pod-network listeners keep their defaults, including atunnel's ingress, CONNECT and egress ports on worker pods (443/8443/15001; the router dials `workerIP:443` directly). So do ateapi, the router, the credential providers, and metrics/health. - ateapi dials `atelet.DefaultPort`, so ateapi and atelet need to roll out together. - If Substrate needs more node-level ports later, we recommend 21992 and 21988, in that order. Like 21998, they are not IANA assigned so rarely used. --------- Co-authored-by: Hoda Mohammadi <173203200+hodaaaaaaaaaa@users.noreply.github.com> |
||
|
Luiz Oliveira
|
e3d267bcbd |
resources.ToAPIError returns OK on nil errors (#2767)
Before this change, if we ever forgot to check for len(errors) before calling `resources.ToAPIError`, we risked sending an InvalidArgument for a nil error. As a side change, I migrated some callers to use `resources.ToAPIError` instead of using apierror.InvalidArgument directly |
||
|
Luiz Oliveira
|
3a2ca62357 |
Align volume and env message names across ateapi, atelet, and ateom pbs (#2666)
Fixes #2419 1. In `internal/proto/ateletpb/atelet.proto`: - Rename `EnvEntry` -> `EnvVar`. - Rename `DurableDirVolume` -> `DurableDirVolumeSource` and `SystemInfoVolume` -> `SystemInfoVolumeSource`. 2. In `internal/proto/ateompb/ateom.proto`: - Rename `Container.csi_volume_mounts` -> `external_volume_mounts`. - Rename `ActorDirs.volumes_dir` -> `external_volume_mounts_dir`. - Consolidate `VolumeMount`, `DurableDirVolumeMount`, `SystemInfoVolumeMount`, and `ImageVolumeMount` into a single `VolumeMount { string name = 1; string mount_path = 2; }` message (matching `ateapi.VolumeMount` and `atelet.VolumeMount`). |
||
|
Michelle Au
|
e3a75a5205 |
Send and require snapshot object storage URI on all checkpoints (#2752)
Populate CheckpointRequest.Snapshot.Object on both immediate and deferred checkpoints so atelet knows where to upload a paused actor's local snapshot in the background. PauseActor derives the URI from the persisted snapshot UUID rather than recording a pending DurableSnapshot on the actor, so an in-progress pause stays distinguishable from an in-progress suspend. Both workflows validate the template's storage location before marking the actor SUSPENDING. Part of https://github.com/agent-substrate/substrate/issues/2069 and https://github.com/agent-substrate/substrate/issues/2571 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
shrutiyam-glitch
|
36197017a5 |
atelet, ateworker: probe CPU features and watch sandbox-runtimes.json for dynamic worker registration (#2660)
Part of #587. Follows #2521. #2521 added the initial `atelet` syncer that wrote `sandbox_runtime_hash` per enabled `SandboxConfig` version to each local worker pod's directory. This PR completes the runtime and hardware compatibility pipeline from `atelet` through the worker into `WorkerService.RegisterWorker`: 1. **Renames `sandbox-versions.json` to `sandbox-runtimes.json` (`internal/sandboxruntimes`)**: `atelet` now writes the complete set of `VersionedSandboxCompat` attributes for `defaultRuntime` and `restorableRuntimes` (`architecture`, `sandbox_runtime_hash`, `cpu_features` for `gvisor`, and `cpu_vendor` + `cpu_model` for `microvm`). 2. **Probes CPU identity and gVisor `cpu_features` in `atelet`**: `atelet` owns all hardware and runtime probing on the node — reading `/proc/cpuinfo` for microVM CPU vendor/model and running `<runsc> cpu-features` (memoized by asset SHA-256) for each enabled gVisor version, encoded as a canonical lowercase hexadecimal bitmask whose bit numbers are gVisor's own `Feature` constants (`internal/hardware`). 3. **Watches `sandbox-runtimes.json` in the worker**: `ateworker-gvisor` and `ateworker-microvm` watch `/var/lib/ate/workers/<pod-uid>/sandbox-runtimes.json` via `register.WatchFile` and keep `default_runtime` and `restorable_runtimes` registered through `WorkerSupport.RegisterWorker` (`internal/ateworker/register`). ## End-to-End Flow ```mermaid flowchart LR subgraph Node["Kubernetes Node"] direction TB SC["SandboxConfig CRD"] --> Atelet["atelet (workerSandboxSyncer)"] CPU["Host /proc/cpuinfo\n(microVM vendor & model)"] --> Atelet Runsc["Cached runsc binary\n(<runsc> cpu-features)"] --> Atelet Atelet -->|"Atomic rename"| File["/var/lib/ate/workers/<pod-uid>/\nsandbox-runtimes.json"] File -->|"register.WatchFile (fsnotify)"| Worker["ateworker-gvisor / ateworker-microvm"] Worker -->|"WorkerSupport.RegisterWorker"| Atelet end Atelet -->|"WorkerService.RegisterWorker"| AteAPI["ateapi (Control Plane)"] ``` ## How it works ```mermaid sequenceDiagram participant Atelet as atelet participant FS as /var/lib/ate/workers/<pod-uid>/ participant Watcher as register.WatchFile (fsnotify / poll fallback) participant Reconciler as reconciler.run / sync participant Support as WorkerSupport.RegisterWorker Note over Watcher,FS: Watches parent dir for Create/Write on sandbox-runtimes.json Reconciler->>FS: Initial sync(): Read sandbox-runtimes.json alt File not written yet FS-->>Reconciler: ErrNotExist (wait for watch event or 5m resync) end Atelet->>FS: Probe hardware/runsc & atomic rename -> sandbox-runtimes.json FS-->>Watcher: fsnotify Create/Write event Watcher->>Reconciler: signal on changes channel Reconciler->>FS: Read sandbox-runtimes.json Reconciler->>Reconciler: proto.Equal(req, registered)? alt Changed (or first registration) Reconciler->>Support: RegisterWorker(capacity, default_runtime, restorable_runtimes) alt RPC succeeds Support-->>Reconciler: OK -> record registered = req else InvalidArgument (malformed runtime attributes) Support-->>Reconciler: Log Error, do not retry until file changes else Transient RPC error (e.g. Worker record not in ateapi yet) Support-->>Reconciler: Retry with exponential backoff (500ms..30s) end end ``` - **`atelet` owns all probing**: - For **`gvisor`** workers, `atelet` fetches the version's `gvisor` asset via `ensureSandboxAssets`, executes `<runsc> cpu-features` (memoized in-memory by the asset's SHA-256 digest), and encodes the reported names via `hardware.EncodeCPUFeatures` into a compact lowercase hex bitmask alongside `architecture` and `sandbox_runtime_hash`. - For **`microvm`** workers, `atelet` reads `/proc/cpuinfo` once (memoized after the first successful read) to record `cpu_vendor` and `cpu_model` alongside `architecture` and `sandbox_runtime_hash`. - **`cpu_features` bit layout**: - A feature's bit is the value of its gVisor `Feature` constant: on amd64, `32*block + bit` in that block's CPUID register (`X86FeatureAVX2 = 2*32 + 5 = 69`); on arm64, the value gVisor declares. Intel/AMD never reassign a CPUID bit, so a bit never moves when gVisor adds, reorders, or renames a feature, and a bit with no entry is one the hardware reserves. - `internal/hardware/cpufeatures_table.go` is a committed table, produced once from gVisor's `pkg/cpuid` sources at `gvisor.dev/gvisor@v0.0.0-20261014212059-ddc37c698141`: for every `Feature` constant with an entry in `allFeatures`, the constant's value keyed to the display name `runsc cpu-features` prints. Each entry's comment names the constant and, on amd64, its block and bit. The file header records this and how to add features from a newer gVisor by hand; because a bit is the hardware's, existing entries never change. - **This mapping is temporary.** Once `runsc` can print its feature set as this bitmask directly, `atelet` records that value as printed and the table, `EncodeCPUFeatures`, and `DecodeCPUFeatures` go away (TODO in the table header). The mask format and subset matching stay. No gVisor sources or generator are vendored for it. - **Subset matching**: `hardware.MatchesCompat` decodes `cpu_features` into a stack-allocated `[6]uint64` mask (384 bits; amd64 bits reach 261 today, with room for gVisor blocks through 11) and checks bitwise subset inclusion ($S_{\text{snapshot}} \subseteq S_{\text{worker}}$) while requiring exact equality on all other attributes. `DecodeCPUFeatures` rejects a mask that sets a bit with no name. - **Resilient watching & registration in the worker**: - `register.WatchFile` watches the parent directory for `Create`/`Write` events on the target file (reused by both `register.Report` and `ateworker-gvisor`'s `flagStore`), and falls back to periodic polling (`5s`) with a warning if `fsnotify` initialization fails (e.g. when `fs.inotify.max_user_instances` is exhausted on dense nodes). - `reconciler` runs a `5m` periodic resync tick alongside watch events, deduplicates consecutive registrations via `proto.Equal`, retries transient read/RPC failures with capped exponential backoff (`500ms..30s`), and drops permanent `InvalidArgument` rejections until the file changes. ## Compatibility `cpu_features` values recorded before this PR used a different bit numbering and a narrower mask. They do not compare meaningfully against workers built from this PR: any dev-cluster snapshot carrying one should be retaken rather than restored. No compatibility is guaranteed before v1.0.0, so there is no migration. - [x] Tests pass (`go test -race ./internal/hardware/... ./internal/sandboxruntimes/... ./internal/ateworker/register/... ./cmd/atelet/... ./cmd/ateworker-gvisor/... ./cmd/ateworker-microvm/...`) - [x] Appropriate changes to documentation are included in the PR |
||
|
Tim Hockin
|
68a082c365 |
Use a Go vanity import in the "main" substrate repo: go.ate.dev/substrate (#2181)
This commit changes everything except HTTP(s) links. github.com/agent-substrate/substrate -> go.ate.dev/substrate I chose "go.ate.dev" for brevity, but "pkg.ate.dev" would be OK, too. This is implemented as "CloudFlare Pages" on my personal account. If we like it, we need to make it more "community operated". This does not yet convert other repos. This breaks a TON of open PRs, so we should consider if/when we want to merge it. /cc @BenTheElder @rakyll @EItanya |
||
|
Zoe Zhao
|
2b3a40632d |
Merge the pause workflow into the suspend workflow (#2694)
This is the next step toward [async suspend](https://github.com/zoez7/substrate/blob/async-suspend-design/docs/dev/async-suspend-en.md), following #2688. `SuspendActor` and `PauseActor` now run one set of shared steps behind a `suspendMode`: - `suspendSync` (`SuspendActor`): the snapshot is durable before the workflow returns. - `suspendAsync` (`PauseActor`): the snapshot stays on the worker's node. Both APIs stay for now. `workflow_pause.go` is removed. Behavior is unchanged. The tests #2688 moved testing of ensurePaused* and ensureSuspended* functions onto `PauseActor` and `SuspendActor`. The only test edit is in `workflow_test.go`, which called the removed `ensurePausedFinalized`. The fix for #2676 (finalize not checking that the actor is still `SUSPENDING`) will come in a separate PR. - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR |
||
|
Luomu
|
a1471f0628 |
Suspend and pause preserve concurrent actor crashes and report failure (#2707)
Fixes #2676. If worker deletion or another crash writer marks an actor CRASHED after checkpoint succeeds but before finalize re-reads it, SuspendActor currently overwrites that crash with SUSPENDED, while PauseActor reports success for the CRASHED actor. Both finalize steps now require the re-read actor to be SUSPENDING before validating snapshots, releasing the worker, or writing the final state. Otherwise they return FailedPrecondition and preserve the existing state, crash reason, and snapshot records. The existing version precondition still rejects a concurrent crash after the read with Aborted. Completed suspend and pause calls retain their existing success paths at workflow entry. The full-workflow regression now checks FailedPrecondition and preservation of the original crash in both modes, updating the behavior pinned by #2688. The already-crashed pause fixture also clears WorkerAssignment, matching a real crash; it previously reached FailedPrecondition only through the missing-node branch. Validation of the production change on Linux with CI=true and real PostgreSQL test containers: - RED: before the production fix, the corrected already-crashed pause test and both modes of the full-workflow regression fail because the calls incorrectly return OK. - GREEN: `go test -race -count=1 -json ./...` passes: 158 test-bearing packages and 9,670 test entries, including all 763 controlapi entries and 169 API functional entries. - `go build ./cmd/ateapi ./cmd/atelet ./cmd/kubectl-ate`, `hack/verify-all.sh`, and the tools/apitool race tests pass. The verifier leaves the source tree unchanged. - The full suite skips 41 tests requiring root privileges, IPv6 loopback, or configured external object storage. The affected PostgreSQL regressions do not skip. E2E follow-up: The original CI passed both gVisor suites but failed both microVM suites in `TestGracefulWorkerTerminationSuspend`. That test unconditionally expected SuspendActor to succeed after worker deletion, even when deletion had already crashed the actor. The test now ties the RPC result to the stored state. OK requires SUSPENDED with no crash record. FailedPrecondition or Aborted requires CRASHED, a crash reason, and the previous completed durable snapshot. Both outcomes require the worker and its assignment to be removed. All other errors fail the test. Follow-up validation on Linux with CI=true and real PostgreSQL and envtest: - The four affected controlapi regressions pass with `-race -count=3`. - All 763 controlapi test entries and all 169 functionaltest entries pass, with no skips. The functional package was rerun after initializing the isolated source directory's Git metadata, which envtest setup requires. - `go test -race -c ./internal/e2e/suites/demo` and `go vet ./internal/e2e/suites/demo` pass. Native compilation, gofmt, and whitespace checks also pass. - Real runtime E2E has not run at this head. The new [pr-workflow run](https://github.com/agent-substrate/substrate/actions/runs/38817919692) is `action_required` and needs maintainer approval. Draft #2694 refactors these same workflows and currently preserves an out-of-band state while returning success. This focused fix also reports FailedPrecondition, as required by #2676; that behavior should carry forward into the refactor. - [x] Local validation passes; runtime E2E awaits workflow approval. - [x] Documentation reviewed; finalize and regression-test comments describe the corrected behavior. No guide change is needed. |
||
|
Anish Ramasekar
|
ef193d1d83 |
Mint worker-for-actor certificates only for the Worker that the actor is assigned to (#2702)
MintWorkerActorCertificate authenticated the atelet but ignored its node, and atelet did not say which worker was asking. Any node, or any worker on the actor's node, could mint the actor's certificate from its name and UID. This applies the same check MintWorkerActorJwt got in #2668. - ate-api-server mints only when the caller's node hosts the actor and the requesting worker is the one assigned to it. Otherwise it returns NotFound, before the stale-UID check. - `MintWorkerActorCertificateRequest` gains a required `worker` field. atelet fills it from the caller's certificate, as it does for RequestActorSuspend. A paused actor has no worker, so it gets no certificate. - `worker` is required with no fallback; upgrading atelet before ate-api-server, as docs/upgrade.md already orders it, covers the skew. Tested with unit tests for the node, worker, paused, release/rebook and stale-UID cases, plus controlapi tests against a real PostgreSQL store. `go test -race ./cmd/ateapi/internal/... ./cmd/atelet/...` passes. |
||
|
Lior Lieberman
|
088381b632 |
atenet: Router checks the routed worker's SPIFFE ID on atunnel's cert (#2678)
Part of #2078 (Envoy dataplane, @EItanya @howardjohn for agw) The router accepted any podidentity cert in the trust domain as atunnel, so any pod in the cluster could pose as a worker. It now requires the exact SPIFFE ID of the worker each request is routed to. - atecontroller's syncer records the worker pod's SPIFFE ID on the Worker (new `worker_spiffe_id`, built from the pod's namespace and ServiceAccount), and ateapi copies it into the actor's worker assignment. ateapi rejects a value that isn't a valid SPIFFE ID. - ext_proc passes that ID to Envoy as is and refuses to route without it. Envoy requires that exact SAN, on both the :443 and CONNECT paths. `--upstream-spiffe-prefix` is removed; the cluster still checks the trust domain. - The Worker carries the SPIFFE ID rather than the ServiceAccount, so the router doesn't need to know how the ID is built. A different ID scheme only changes atecontroller. - WorkerPools that don't set serviceAccountName now run as an `ateworker` ServiceAccount the controller creates, instead of `default`, so the worker's ID isn't shared with every other pod in the namespace. Remaining gap: another pod running as the same ServiceAccount in the same namespace can still pose as the worker. Closing it needs a signer that puts the pod in the worker's SPIFFE ID; atecontroller would then record that ID, with no router change. #### Breaking change - The router rejects workers registered before this change until their pods are replaced. Pools on the default ServiceAccount roll automatically; pools that set `spec.template.serviceAccountName` need a restart. Upgrade atecontroller and ate-api-server before atenet-router. - Workers now run as an `ateworker` ServiceAccount by default instead of `default`. RBAC, Workload Identity bindings and imagePullSecrets attached to `default` no longer apply to them; set `serviceAccountName: default` or move those to `ateworker`. - atenet-router no longer has `--upstream-spiffe-prefix`. - atecontroller now needs create and patch on serviceaccounts (included in the generated role). > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
igooch
|
a68c7e1d9f |
ate-api-server can apply the snapshot state Checkpoint returns, and logs stale snapshot reports (#2698)
Part of Actor Lifecycle v2 #2069, for #2570. atelet's `Checkpoint` response carries the actor's snapshot state. ate-api-server discards it today. This adds what's needed to apply it, with the same rules `ReportActorSnapshotState` uses. It also logs reports that change nothing. - **Conversion.** atelet's `SnapshotReport` nests an atelet `Snapshot`. The control plane's carries the same fields inline, with its own fidelity enum and object type. `snapshotReportFromAtelet` converts one to the other, and `fidelityFromAtelet` maps the fidelity back. - **`applyCheckpointState`** records an upload the state says finished, through `applyLatestDurable`. atelet echoes the `Snapshot` it was asked for, so the fidelity recorded is the one requested. Unlike a report, a state with no local snapshot is not a detach, since the workflow that called `Checkpoint` records where its snapshot lives. - **Stale reports are logged at Info.** A `latest_durable` older than the latest `COMPLETED` generation, or at a generation the actor no longer records, changes nothing. That's expected for a late report, so each case is logged rather than refused. A repeat of the latest isn't logged, since atelet retries reports freely. Nothing calls `applyCheckpointState` yet. A follow-up applies it in the suspend workflow. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR |
||
|
Zoe Zhao
|
7a51ad1796 |
Remove the pause operation label (#2701)
Pause and suspend are being merged into one workflow, so PauseActor and SuspendActor become one lifecycle operation. Label both as "suspend" in metrics, state-change events and crash records, and drop the "pause" value from the ate.actor.operation.name registry. |
||
|
igooch
|
35c6ab2aa0 |
ate-api-server applies atelet's ReportActorSnapshotState reports (#2622)
Part of Actor Lifecycle v2 #2069. Second PR for #2570, after #2612 added the API. This serves `WorkerService.ReportActorSnapshotState`. With asynchronous suspend, atelet uploads a suspended actor's snapshot and later detaches the actor from its node, on its own timers. The report is how ate-api-server learns about both. Nothing sends reports yet; that comes with atelet's asynchronous upload work. A report lists the snapshots atelet holds for an actor on its node. ate-api-server acts on two things in it: - **`latest_durable` means an upload finished.** If it names a recorded snapshot with the same uuid and upload URI, that snapshot becomes `COMPLETED`, with its manifest's fidelity. - **A report with neither `latest_local` nor `uploading` means atelet deleted its copies (a detach).** It covers this node's snapshots up to the reported `latest_durable` generation. Those localities are cleared, and a suspended actor with no snapshot left on the node is released from it. ate-api-server ignores the rest of the report. <details open> <summary>Flowchart: how a report is applied</summary> ```mermaid flowchart TD A["WorkerService.ReportActorSnapshotState(ReportActorSnapshotStateRequest)<br/>TODO: atelet does not send it yet"] --> B{"Caller presents atelet's certificate?"} B -- no --> X1["Unauthenticated or PermissionDenied"] B -- yes --> C{"Request passes validation?"} C -- no --> X2["InvalidArgument, including a report with none of<br/>latest_local, uploading, and latest_durable"] C -- yes --> E{"Actor exists, actor_uid equals Actor.metadata.uid,<br/>and Actor is on the caller's node?"} E -- no --> X4["NotFound"] E -- yes --> D{"Actor's lease free?"} D -- no --> X3["Aborted, atelet retries"] D -- yes --> E2{"Same check again, under the lease"} E2 -- no --> X4 E2 -- yes --> S{"status.state?"} S -- "CRASHED" --> X9["FailedPrecondition"] S -- "SUSPENDING, RESUMING,<br/>DELETING, REVERTING" --> X10["Aborted, atelet retries"] S -- "any other state" --> X12["Internal"] S -- "SUSPENDED or RUNNING" --> G{"latest_durable set?"} G -- no --> K{"latest_local and uploading both unset?"} G -- yes --> GF{"At or below the latest COMPLETED generation?"} GF -- yes --> K GF -- no --> GR{"Its generation is in status.snapshots?"} GR -- "no, and not newer than<br/>last_assigned_generation" --> K GR -- "no, and newer" --> X5["FailedPrecondition"] GR -- yes --> U{"Same uuid, and an upload URI is recorded?"} U -- no --> X6["FailedPrecondition"] U -- yes --> V{"Reported URI equals the recorded one?"} V -- no --> X7["InvalidArgument"] V -- yes --> H["Mark it COMPLETED"] H --> K K -- yes --> K2{"Latest snapshot on this node, up to<br/>latest_durable's generation, is durable?"} K2 -- no --> X8["FailedPrecondition, nothing recorded"] K2 -- yes --> J["Detach: clear this node's localities up to that generation.<br/>Clear assigned_node if status.state is SUSPENDED, it is this node,<br/>and no snapshot is left on this node"] K -- no --> P{"status changed?"} J --> P P -- no --> Q["Done, nothing written"] P -- yes --> R["Write the actor"] ``` </details> ## Outcomes | Report | Result | | --- | --- | | Any report for a `CRASHED` actor | `FailedPrecondition` | | Any report for a `SUSPENDING`, `RESUMING`, `DELETING`, or `REVERTING` actor | `Aborted`, retried | | `latest_durable` for a recorded snapshot with matching uuid and upload URI | That snapshot becomes `COMPLETED`, with the fidelity its manifest records | | `latest_durable` at or below the latest `COMPLETED` generation | No change | | `latest_durable` for a generation `status.snapshots` no longer has, such as one a revert discarded | No change | | `latest_durable` for an unrecorded generation newer than any assigned | `FailedPrecondition` | | `latest_durable` with another uuid, or for a snapshot with no recorded upload URI | `FailedPrecondition` | | `latest_durable` uploaded to a URI other than the recorded one | `InvalidArgument` | | None of `latest_local`, `uploading`, and `latest_durable` set | `InvalidArgument` | | Neither `latest_local` nor `uploading` set (a detach) | Clears this node's localities up to the `latest_durable` generation. Clears `assigned_node` if the actor is `SUSPENDED` there and no snapshot is left on this node. Refused with `FailedPrecondition` while the latest covered snapshot isn't durable | | A late detach, after a newer snapshot on this node | Leaves the newer snapshot and `assigned_node` alone | | A detach that's refused | Nothing is recorded, including an upload the same report carries | | A report that changes nothing | Nothing is written | ## Notes - **Nothing is pruned or deleted.** The existing workflows prune old snapshots and delete their objects at the next suspend or delete. - **A detach covers only generations up to its `latest_durable`,** so a late or repeated report can't clear a newer snapshot. Repeating any report is safe. - **A detach waits for a durable snapshot.** Clearing the locality of a snapshot that isn't durable would let the actor resume from an older one. - **A report applies as a whole or not at all.** atelet's detach uploads the latest snapshot first, so a refused detach is rare. - **The actor is checked before the lease is taken,** and again under it, so an atelet can't hold the lease of an actor on another node. - **Reports apply only to a `SUSPENDED` or `RUNNING` actor.** A transitioning actor's snapshots belong to its workflow, and a `CRASHED` actor's wait for a revert. - **Every UID or node refusal is `NotFound`,** so an atelet learns nothing about actors on other nodes. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR |
||
|
Zoe Zhao
|
4c3e62308c |
Update suspend and pause tests to use SuspendActor and PauseActor APIs (#2688)
This is a step toward [async suspend](https://github.com/zoez7/substrate/blob/async-suspend-design/docs/dev/async-suspend-en.md). The next PR merges `SuspendActor` and `PauseActor` into one set of shared steps: sync suspend, and async suspend (today's pause). Both APIs stay. The current unit tests call the private step methods (`ensureMarked*`, `ensureAtelet*`, `ensure*Finalized`). But the next PR will rewrite exactly those steps. If the tests stay as they are, they get rewritten in the same PR as the refactor, and nothing shows that the refactor kept the behavior. This PR moves the tests onto the public `SuspendActor` and `PauseActor` entrypoints and checks them against the current implementation. The merge PR then shouldn't need to touch any test file: if this suite still passes unchanged, behavior is preserved. Test-only change. - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR |
||
|
Benjamin Elder
|
cb0e2335ec |
ateworker internal packages cleanup (#2700)
Follow-up to #2254, renames the remaining internal/ateom* go packages. After this change there is no `ateom` left in the repo. 🫡 - Worker only packages move under `internal/ateworker`: register (the registration code, leaving `internal/ateworker` itself for the shared worker code), cgroup, pid1 (was ateominit), actornet (was ateomnet), phaselog, suspend, tunnel - Two packages go flat instead: `internal/workerstats`, which atelet also imports, and `internal/netns`, a generic named netns helper that atunnel's tests also use - The downward API file names atecontroller needs move to `internal/workercapacity`, so it doesn't import the worker registration code - `depguard` (`golangci-lint`) now fails any import of `internal/ateworker` outside `cmd/ateworker-*` and `internal/ateworker` - Drops an unused stopwatch package - Renames cmd/atelet's ateom* files The move commits are only `git mv`, imports and package names are fixed in a separate commit after each round of moves, to make it clear which changes are just file moves and which changes are import rewrites etc. We squash merge, so the intermediate commits don't need to build. This PR was AI assisted. I have reviewed the changes. |
||
|
Michelle Au
|
971236dc8f |
Require snapshot generation to be passed to atelet (#2690)
Part of #2069 ateapi assigns each actor snapshot a generation, but never passed it to atelet: every `ateletpb.Snapshot` it builds left `generation` unset. That covers the pause and suspend `Checkpoint`, the paused-actor `PromoteSnapshot`, and both `Restore` paths. This sets it from the snapshot each request names, and narrows the TODO on `Snapshot.generation` to say only that atelet does not check it yet. Atelet requires generation to be set but doesn't use the value yet. Once #2677 lands, atelet can reject a `Checkpoint` or `PromoteSnapshot` whose generation is stale, as the `Snapshot.generation` doc describes. That check came up in review on #2677. > It's a good idea to open an issue first for discussion. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR |
||
|
Benjamin Elder
|
63a17a9203 |
rename ateom -> ateworker (#2254)
> The PR title becomes the release note. Write it for users: what changed for them, not how the code changed. #### Breaking change fixes #2065 renames ateom ahead of GA. this is breaking, atelet and the worker images need to be updated together. - images / binaries: ateom-gvisor, ateom-microvm -> ateworker-gvisor, ateworker-microvm - worker pod container: ateom -> worker - RPCs: ateom.Ateom -> ateworker.Worker, atelet.AteomSupport -> atelet.WorkerSupport, atelet.AteomHerder -> atelet.Atelet, MintAteomActorCertificate -> MintWorkerActorCertificate - SPIFFE path: ateom-for-actor -> worker-for-actor - ate.snapshot.phase values: ateom_restore, ateom_checkpoint -> worker_restore, worker_checkpoint - node paths: /var/lib/ate/ateoms/<uid>/ateom.sock -> workers/<uid>/worker.sock, ateom-support.sock -> worker-support.sock - cgroups: the worker's leaf ateom -> worker, the micro-VM guest's /ateomchv -> /actor - authz relation: can_mint_ateom_actor_credential -> can_mint_worker_actor_credential per review, "worker" where any worker implementation fits and "ateworker" for ours. the internal go packages (internal/ateom*) keep their names for now, that's a follow-up. one commit per rename to make review easier. tested with kind e2e (gVisor + micro-VM) on an earlier rebase, plus CI. This PR was AI assisted. I have reviewed the changes. > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Tim Hockin
|
756bb9e15f |
Add a "preview gate" concept, use it for external volumes (#1994)
I went through a number of models and this seems to be the simplest and
lowest overall impact.
To keep it simple:
* All gates are disabled unless enabled
* Only "preview" and "ga" - no "alpha" or "beta"
* We only have one gate for now, called "Preview" (later we can break
this down, maybe)
* All gates can be activated by passing "*"
* Package internal/preview has an Init() function which gets passed a
list of gates to enable, which is stored as a package-global
* Code can call preview.IsEnabled(gatename)
Fixes #1690
@juli4n @BenTheElder @msau42
|
||
|
Benjamin Elder
|
c58f374ad9 |
require actor cpu and memory limits (#2285)
We can still consider more flexible overcommit in the future, but I think leaving actors uncapped is unsafe, and it's hard to pick a reasonable universal default, it's too workload dependent. Actor authors should really be aware of how much resources their actor requires, and tuning them can have significant performance impacts. In the future we could do something like determining how much memory was in use at snapshot time and consider that for scheduling resumes. #### Breaking change ActorTemplates *must* declare CPU/memory limits. Previously `microvm` actors had defaulting, and gvisor actors were simply uncapped. This made the scheduler unaware of placement for gvisor. Requiring it also ensures actor owners are aware of how much resources they cost. Setting actor size will be important for performance particularly with microvms where higher memory limits have a real cost for e.g. snapshot/resume. |
||
|
Eric Curtin
|
2c9a05f480 |
ExternalVolume no longer has a status field (#2456)
#### Breaking change `ExternalVolume.status` is gone. Use `storage_volume_id` (empty until the volume is created) and the actor state instead. Please add the `breaking-change` label. Fixes #2414 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Anish Ramasekar
|
b6e6346607 |
HostingNode and MintAteomActorJwt trust a Worker's node only while that Worker holds the actor (#2668)
Follow-up to #2523. Suspend (and pause), crash, and a resume that finds its Worker ineligible release the actor's Worker before they rewrite its status. In that window, or for good if the status write fails, the status still names the Worker, so its node could mint a one-hour ateom-for-actor JWT, even after it took another actor. - `controlapi.HostingNode` returns a Worker's node only while the store's worker assignment binds that Worker to the actor's UID. A failed lookup refuses the mint. - A paused actor, which has no Worker, still resolves to `assigned_node`; that branch is unchanged. - Real-PostgreSQL tests drive each workflow into that window and through a rebook. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR (none needed) |
||
|
Luiz Oliveira
|
e8d40c33f9 |
Consolidate API validation for sandbox class across services (#2587)
Part of https://github.com/agent-substrate/substrate/issues/1709 |
||
|
Shruti Nair
|
45c25eff0b |
ateapi: [Part 4] deny RPCs without an authz rule and authorize tags (#2562)
Working on #1563. Fixes https://github.com/agent-substrate/substrate/issues/1696 ### Summary - **Fail-closed interceptor for unregistered RPCs:** `UnaryServerInterceptor` now returns `PermissionDenied` when an RPC is not in `defaultRPCPermissions` while enforcement is enabled. `WorkerService` RPCs, which authenticate `atelet` in their handlers via `ateletauth`, are explicitly registered with `inHandler(...)`. - **Exhaustive coverage test (`TestDefaultRPCPermissions_CoversEveryServedRPC`):** Walks every method of every service in `ateapi.proto` and fails if any RPC has neither a permission rule nor an `inHandler` reason, if the registry names an RPC that isn't in `ateapi.proto`, or if a service gains a streaming RPC (which the unary interceptor would not check). - **Tags (`Tag` type and rules):** Models tags as atespace children (`can_get`: `viewer`, `can_update`/`can_delete`: `editor` on the parent atespace; `can_create_tag`: `editor` on the source actor's atespace; `can_list_tags`: `viewer` on the atespace or `global:root`). Calling `CreateActor` with an explicit non-golden `source_tag` requires `can_use_tag` (`editor` on the tag's atespace), regardless of whether the tag's scope is `TAG_SCOPE_ATESPACE` or `TAG_SCOPE_PUBLISHED` (the handler continues to enforce that `TAG_SCOPE_ATESPACE` tags are only used within the same atespace). - **Actor identity credentials:** Restricts `MintActorJwt` and `MintActorCertificate` to `egress_gateway` so an atespace `editor` cannot impersonate an actor. - **Model cleanup:** Removes `global.can_get` and `actor_template.can_update`, which no RPC checks. |
||
|
Luiz Oliveira
|
9ec4975913 |
Changed SandboxClass from enum to string to allow extensibility (#2586)
#### Breaking change An ActorTemplate's `sandboxConfig.sandboxClass` is now a string, `gvisor` or `microvm`, the same values the WorkerPool and SandboxConfig CRDs use. The `SANDBOX_CLASS_GVISOR` and `SANDBOX_CLASS_MICROVM` spellings are rejected. To upgrade: - In ActorTemplate manifests and API requests, replace `SANDBOX_CLASS_GVISOR` with `gvisor` and `SANDBOX_CLASS_MICROVM` with `microvm`. - Go clients of `pkg/proto/ateapipb`: the `SandboxClass` enum is removed and `SandboxConfig.SandboxClass` is a `string`. - Existing ActorTemplates read back with an empty `sandboxClass`, because the stored enum value does not decode into the new string field. Templates are immutable, so delete them and their actors before upgrading and recreate them afterwards. - The `SANDBOX CLASS` column of `kubectl ate get actortemplates` now prints `gvisor` or `microvm`. Fixes #2406 |
||
|
ILLUM1N0X
|
db2f833475 |
WorkerService.MintAteomActorJwt mints ateom-for-actor JWTs for atelet; actor JWT claims move from ate.dev to actor.ate.dev (#2523)
Part of #777 The object-store plugin's control sidecar will exchange an actor JWT for a downscoped object-store credential scoped to that actor. atelet needs a way to obtain that JWT over the channel it already authenticates on, which is `WorkerService`, where it already mints the certificate counterpart with `MintAteomActorCertificate`. This adds `MintAteomActorJwt` beside it. Nothing calls it yet. atelet starts to in a later PR, together with the node plugin exchange. Two commits: 1. **Proto.** `MintAteomActorJwtRequest` mirrors the certificate request (actor, UID guard against a deleted-and-recreated actor) plus `audiences` from `Control.MintActorJwt`, each non-empty. The lifetime is fixed at one hour, so there is no expiration field. Response is the token and its expiry. Generated Go and validation code, and the validation wrapper. 2. **Handler.** The caller must be atelet and the actor must exist. The mint is then refused unless the caller's node hosts the actor, as `RequestActorSuspend` already requires. The actor's state decides which status field is trusted, since both can outlive the hosting they record: the node of the assigned worker while the actor is resuming, running or suspending, and `assignedNode` while it is suspended or suspending with no worker and its latest generation is a local checkpoint with no durable copy, which is how a paused actor looks now that `PAUSED` is no longer a state. The whole rule lives in a new `controlapi.HostingNode`, in `hosting.go`, so the suspend and resume workflows' own notion of a paused actor is what it reads, and so the certificate RPC can reuse it in its follow-up. Anything else resolves to no node and is refused: a worker gone from the store, a reverting actor whose assignment outlives its released worker, a crashed or deleting actor whose `assignedNode` is only a record of its last node, and a suspended actor with a stale `assignedNode`. The node check runs before the UID check and every refusal is `NotFound`, so a node learns nothing about actors it does not host, not even that one exists. A UID mismatch is then answered `Aborted`, as `MintAteomActorCertificate` answers it. Each handler composes its own `actoridjwt.Claims` and signs with the actor-id JWT pool, as `Control.MintActorJwt` does. The two tokens differ only in kind: a token minted here speaks for an ateom acting on the actor's behalf, with subject `ateom-for-actor/<atespace>/<name>` and its claims under `ateom-for-actor.ate.dev`, while `Control.MintActorJwt` mints the actor's own token with subject `actor/<atespace>/<name>` and claims under `actor.ate.dev`. The zero kind is rejected, so a token cannot be minted without choosing. A relying party can therefore tell the actor from a Substrate component acting for it, and the snapshot credential exchange will accept only the `ateom-for-actor` kind, so actor code cannot reach its own snapshots. Follow-up, in its own PR: `MintAteomActorCertificate` gets the same host check. It is an existing RPC on the actor boot and resume path, so the change deserves its own review and e2e run. #### Breaking change Actor JWTs no longer carry the `ate.dev` claim. The actor's own token now carries the same fields under `actor.ate.dev`, and the new ateom-for-actor token carries them under `ateom-for-actor.ate.dev`. The OIDC discovery document's `claims_supported` lists the two new names. Any relying party that reads `ate.dev` from an actor JWT, agentgateway included, has to read `actor.ate.dev` instead, and must not accept an `ateom-for-actor` token as the actor. Relying parties should accept both `ate.dev` and `actor.ate.dev` before the API server is upgraded, and drop `ate.dev` an hour after it, once every token minted before the upgrade has expired. Please add the `breaking-change` label. - [x] Tests pass with `go test -race ./...`. New unit tests cover the handler and the claims package: claim contents per kind, `exp` equal to `expire_time`, the host check for a foreign node, a foreign node with a stale UID, a paused actor on each node, a paused actor being suspended, a crashed actor on its last node, an unhosted actor, a lost worker, and each validation and lookup error. - [x] Appropriate changes to documentation are included in the PR. The RPC, messages and both tokens' claims are documented in the proto, and no guide lists WorkerService RPCs. |
||
|
shrutiyam-glitch
|
fd8304189e |
Stamp snapshots with SandboxRuntime and schedule restores on runtime-compatible workers (#2536)
Restoring a sandboxed actor's memory checkpoint (`MEMORY` snapshot fidelity) requires compatibility between the worker that captured the checkpoint and the worker restoring it — across the sandbox class (`gvisor`, `microvm`), the host hardware (`architecture`, CPU features), and the sandbox runtime assets (`gvisor_asset_hash`, microVM kernel/rootfs digests). Fixes #2249 #2250 Following #2381 (where workers began self-reporting their `default_runtime` and `restorable_runtimes` during registration), this PR closes the control-plane loop across checkpointing, scheduling, and restore: 1. **Assignment-Pinned Runtime Stamping**: When an actor is placed on a worker, the control plane records the specific `SandboxRuntime` that will serve that actor on `WorkerAssignment.sandbox_runtime`. Subsequent checkpoints taken during that assignment (`Pause`, `Suspend`, `Tag`) stamp `Snapshot.sandbox_runtime` from the assignment rather than re-reading the worker's live `default_runtime`. This guarantees that if a worker updates its `default_runtime` while an actor is still running under an older runtime version, checkpoints taken from that actor are still stamped with the runtime that actually produced the snapshot. 2. **Runtime-Aware Scheduling with Graceful Fallback**: When resuming an actor from a `MEMORY` snapshot under its current template, the scheduler prefers workers whose `default_runtime` or `restorable_runtimes` match the snapshot's `sandbox_runtime` (with `hardware.Matches` returning `false` if either runtime is `nil`). If no runtime-compatible worker has capacity (and the actor is not pinned to a local pause checkpoint), the scheduler relaxes the runtime constraint so the actor can still make progress on any otherwise-eligible worker. 3. **Consistent Restore Fidelity Downgrade (`MEMORY` -> `VOLUMES`)**: At restore time, compatibility is evaluated directly against the persisted `WorkerAssignment.sandbox_runtime` — the same field that will stamp the next checkpoint — keeping the restore decision and future snapshot stamps in lockstep across retries and re-entries. When the assigned runtime does not match the snapshot's required runtime, all durable snapshots (actor-owned, explicit `source_tag` clones, and borrowed golden snapshots) uniformly downgrade the `RestoreRequest` fidelity to `SNAPSHOT_FIDELITY_VOLUMES` and log a warning. --- ### Scenarios #### 1. Cold Boot (No Snapshot) or Non-Memory (`VOLUMES` / `ROOTFS`) / Repointed Snapshot - **Scheduling**: No memory checkpoint is being restored (`RequiredRuntime == nil`), so any active worker matching the template's `SandboxClass`, selectors, and resource limits is eligible. - **Assignment**: `WorkerAssignment.sandbox_runtime` is set to the bound worker's `default_runtime`. - **Restore & Next Checkpoint**: The actor cold-boots via `Run` (or restores `VOLUMES`/`ROOTFS`), and any subsequent pause/suspend stamps the new snapshot with `WorkerAssignment.sandbox_runtime`. #### 2. Warm Restore on a Matching Worker (`default_runtime` or `restorable_runtimes`) - **Scheduling**: The actor has a stamped `MEMORY` snapshot (taken under the current template). The scheduler filters for workers where either `default_runtime` or an entry in `restorable_runtimes` matches the snapshot's `sandbox_runtime`. - **Assignment**: Under the worker's row lock, `WorkerAssignment.sandbox_runtime` records the exact runtime entry on that worker that matched the snapshot (`default_runtime` on a steady-state worker, or the matched `restorable_runtimes` entry during a runtime rollout). - **Restore & Next Checkpoint**: Because `WorkerAssignment.sandbox_runtime` matches the snapshot's required runtime, `atelet.Restore` runs at full `SNAPSHOT_FIDELITY_MEMORY`. If the actor was restored via a `restorable_runtimes` entry on a worker whose `default_runtime` has rolled forward, its next pause or suspend is still stamped with the matched `restorable_runtimes` entry. #### 3. Durable Snapshot Fallback When Compatible Workers Are Full (Actor-Owned, `source_tag` Clone, or Borrowed Golden Snapshot) - **Scheduling**: The initial `Schedule` pass with `RequiredRuntime` finds no capacity (`ErrNoCapacity`). Because the snapshot is durable (not a node-local pause checkpoint), the workflow relaxes `RequiredRuntime = nil` and re-runs `Schedule`, picking an eligible worker running a different runtime. - **Assignment**: `WorkerAssignment.sandbox_runtime` records the fallback worker's `default_runtime` (or, if a compatible worker freed up between the two `Schedule` calls, the matched runtime from the unrelaxed snapshot requirement). - **Restore & Next Checkpoint**: `ensureAteletRestored` detects that `WorkerAssignment.sandbox_runtime` does not match the snapshot's required runtime, logs a warning, and downgrades the `RestoreRequest` fidelity to `SNAPSHOT_FIDELITY_VOLUMES` so the actor boots a fresh sandbox under the new runtime while preserving its volume state. Subsequent checkpoints are stamped with the new worker's runtime. #### 4. Local Pause Checkpoint on an Incompatible Node - **Scheduling**: A `PAUSED` actor holds a node-local checkpoint on `AssignedNode`. Because nothing releases the node-local checkpoint and claim on fallback yet, `canRelaxRuntime` returns `false`. If no worker on `AssignedNode` supports the snapshot's `sandbox_runtime`, scheduling returns `ResourceExhausted` rather than silently stranding the local checkpoint. #### 5. Concurrent Worker Re-registration & Workflow Re-entry Races - **Stale cache vs. row-locked worker**: If `atelet` re-registers a worker and drops an older runtime from `restorable_runtimes` before the scheduler's watch cache catches up, `BindActorToWorker` evaluates admission under the worker's DB row lock and `assignWorkerAttempt` computes `WorkerAssignment.sandbox_runtime` from that fresh row-locked worker rather than the cached copy. - **Re-entry in `RESUMING`**: If `ResumeActor` fails after persisting `RESUMING` (e.g., during volume attachment) and re-enters later, `validateAssignedWorker` accepts a worker assigned via runtime relaxation, and `ensureAteletRestored` derives `runtimeIncompatible` from the persisted `WorkerAssignment.sandbox_runtime` so the restore fidelity and the next checkpoint's stamp cannot drift even if the worker's advertised runtimes changed in the interim. ### Testing & Docs - **Functional test** (`TestResumeActor_SandboxRuntimePlacementAndFallback`): Exercises stamping, compatible worker placement, full-worker fallback to `SNAPSHOT_FIDELITY_VOLUMES` (for both actor-owned and borrowed golden snapshots), re-stamping on subsequent suspend, and `restorable_runtimes` warm restore + `PauseActor` -> `SuspendActor`-from-`PAUSED`. - **Cluster E2E test** (`internal/e2e/suites/sandboxruntime` — `TestSnapshotRuntimeStamp`): Verifies live worker `DefaultRuntime` stamping across `ResumeActor`, `PauseActor` (local checkpoint + warm restore), `SuspendActor`-from-`PAUSED`, `SuspendActor`-from-`RUNNING`, and `SourceTag` actor clones. - **API Guide** (`docs/api-guide.md`): Documents `Snapshot.sandbox_runtime` and `WorkerAssignment.sandbox_runtime` placement and fallback semantics. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Scott Funkenhauser
|
7469ab4416 |
Mark workers unavailable if it's pod status changes (#2055)
This prevents actors being scheduled on a worker that is restarting or temporarily unreachable. - [ x ] Tests pass - [ x ] Appropriate changes to documentation are included in the PR |
||
|
Zoe Zhao
|
849b423035 |
Define a paused actor by its snapshots and delete the PAUSED and PAUSING states (#2620)
A paused actor no longer has its own state. "Paused" is now defined by the actor's snapshots alone: an actor is paused when it is `ACTOR_STATE_SUSPENDED` and its latest snapshot is local only, that is, it has a `locality` and no `durable_snapshot`. A suspended actor is `ACTOR_STATE_SUSPENDED` with a durable latest snapshot. `ACTOR_STATE_PAUSED` and `ACTOR_STATE_PAUSING` are deleted. This changes how the state is represented, not what the system does. `PauseActor` and `SuspendActor` stay separate workflows and keep sending the same atelet requests. Resume already chose its restore source and node pinning from the snapshot shape (`findLatestLocalSnapshot`, `status.assigned_node`), so it is unchanged apart from its state gate. atelet is untouched. Three commits, one per step: 1. **Pause leaves the actor SUSPENDED with a node-local snapshot.** `FinalizePaused` commits `SUSPENDED` instead of `PAUSED`; `locality`, `assigned_node` and pruning are as before. The workflows that read `PAUSED` now read the shape through three helpers in `snapshot_helpers.go`, all built on `latestSnapshot`: - `latestSnapshotIsLocalOnly`: the paused shape. - `latestSnapshotIsDurable`: the suspended shape (also true with no snapshot at all, which has nothing to upload). - `latestSnapshotIsPendingLocal` (commit 3): a pause in progress. `SuspendActor` returns at once for `SUSPENDED` + durable and uploads the local snapshot for `SUSPENDED` + local only, as it did for `PAUSED`. `PauseActor` is idempotent on the paused shape. `RevertActor` accepts the paused shape and still refuses a durable `SUSPENDED` actor. Changing an actor's template still refuses a paused actor. `ResumeActor`'s state gate accepts `SUSPENDED` only. 2. **Delete `ACTOR_STATE_PAUSED`.** Enum value, `ate.actor.state` metric label `paused`, and the remaining mentions. 3. **Delete `ACTOR_STATE_PAUSING`.** A pause in progress is `SUSPENDING` like a suspend in progress; the latest snapshot tells them apart (a pause has minted a local snapshot with no locality yet and no durable copy, a suspend has a pending durable copy). The pause workflow re-enters on that shape, `SuspendActor` refuses an actor mid-pause as it refused a `PAUSING` one, and a crash during a pause is still attributed to the pause operation. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR |