mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-10-02 03:24:42 +08:00
Dmitry Berkovich
3a2d0c1d86
Part of #791 — the last planned piece of the [implementation plan](https://github.com/agent-substrate/substrate/issues/791#issuecomment-5226674097) (after #810, #812, #813). #791 stays open until #817 (gVisor Full-capture → Data-commit conversion, blocked on #790) is done; this PR covers micro-VM fully and gVisor for scope-matched suspends. Also part of the actor state machine (#119) and a prerequisite for system upgrade flows (#473). `SuspendActor` now accepts a PAUSED actor: instead of checkpointing a running workload, ateapi dials the atelet on the node holding the pause snapshot and has it upload the node-local files to object storage, then finalizes as usual — durable `ActorSnapshot`, `SUSPENDED` status, node pinning cleared. Two commits, reviewable independently: ## Commit 1 — atelet: `UploadPausedCheckpoint` RPC (dead code until commit 2) - New `AteomHerder` RPC: a pure disk→object-storage copy driven by the snapshot's self-describing manifest — no ateom involved (the sandbox is gone). - **Scope conversion** dispatches per sandbox class (`narrowFullCaptureToData`): a micro-VM FULL capture narrows to a DATA upload by carving out `durable-dir.tar` (constant hoisted to `ateompath`, shared with ateom-microvm); gVisor returns `Unimplemented` until split checkpoints land (#790); DATA can never widen to FULL; a scope-less manifest (older atelet) is rejected rather than guessed at. - **Idempotent retry**: local files gone + remote manifest present ⇒ a previous invocation committed, succeed; gone on both sides ⇒ unrecoverable (`LOCAL_SNAPSHOT_GONE`, crashes the actor). Upload failures stay plain retryable errors; the manifest uploads last as the commit marker, never in parallel. - The golden atespace is rejected at validation (fully on the `field.ErrorList` framework): golden actors are never paused. ## Commit 2 — control plane: enable suspend from PAUSED - `FromPaused` discriminator: PAUSED status, or SUSPENDING with no worker assignment and a `LocalSnapshotInfo` — the field alone is stale on resumed-from-pause RUNNING actors, so the nil-assignment conjunct is load-bearing. - `MarkSuspendingStep` accepts PAUSED and rejects a Data-captured pause against a Full commit *before* the actor leaves PAUSED (an upload cannot fabricate memory), using the `content_scope` recorded at pause (#812) with an `onPause` fallback. - `CallAteletSuspendStep` paused branch dials by node (`DialForAteletOnNode`, #813): missing node record ⇒ crash (the snapshot can never be found); unreachable atelet ⇒ retryable; atelet's `LOCAL_SNAPSHOT_GONE` ⇒ crash via `maybeCrashActor`. `FinalizeSuspendedStep` needed no changes thanks to the #813 hoist. - Root-cause guard: `MarkPausingStep` rejects pausing golden-atespace actors. - Docs: pause states + the new `PAUSED → SUSPENDING` edge in the architecture state diagram; glossary Suspend entry covers both origins. ## Tests - **atelet unit**: 10 upload-helper subtests (conversion matrix, idempotency probe, data-loss crash, retryable upload failure) via a recording object-storage fake; validation table. - **control-plane unit**: discriminator table, scope-rejection table (incl. onPause fallback), paused preconditions (no node ⇒ CRASHED, no atelet ⇒ `ErrNoAteletOnNode` + still SUSPENDING), golden-pause rejection, prerequisite matrix updated. - **functional (envtest)**: `TestSuspendActor_FromPaused` (upload called with the pause snapshot name, no Checkpoint RPC, SUSPENDED, pinning cleared, ActorSnapshot at the upload destination) + retry-after-failed-upload (same destination on retry). - **e2e (demo suite)**: the lifecycle driver gains a suspend-from-PAUSED mode; three durable-dir cases — Full/Full, Data/Data, and the micro-VM Full→Data extraction — assert memory/file counters survive the full pause→suspend→resume journey and the node pinning is gone. `go test -race ./...` clean (except the pre-existing macOS-only `internal/atunnel` unix-socket-path failures, untouched by this PR), `gofmt`/`go vet` clean, protos regenerated via `hack/protoc.sh`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)