mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-11-13 10:43:28 +08:00
2eeee044e64761f02df7d2ea8a0924685c5d203d
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Sneha-at
|
b7080602c6 |
Allow deleting an actor from any state. (#788)
Fixes #643 This feature introduces the ability to delete Substrate actors from any lifecycle state (e.g., RUNNING, PAUSED, PENDING), rather than requiring them to be hibernated into SUSPENDED (or CRASHED) beforehand. This is crucial for cleaning up stuck or failed actors. #### 1. Control Plane & gRPC API (pkg/proto/ateapipb, ateapi) • DeleteActorRequest.any_state: Added a new boolean field any_state to DeleteActorRequest. • When false (default): enforces the existing behavior where only actors in ACTOR_STATE_SUSPENDED or ACTOR_STATE_CRASHED (or already ACTOR_STATE_DELETING) can be deleted. • When true: allows deleting an actor in any state (e.g., RUNNING, PAUSED). • Orchestrated Cleanup Workflow (workflow_delete.go): 1. Transitions actor state to ACTOR_STATE_DELETING. 2. Calls atelet.Terminate to stop live workloads on the node. 3. Detaches external volumes from the worker node (with fallback logic using actor status if the ActorTemplate was deleted). 4. Releases the assigned physical worker Pod back to the WorkerPool. 5. Deletes external storage volumes through the storage plugins. 6. Finalizes removal of the actor from the persistence store. #### 2. Node Herder Agent (atelet) • Terminate RPC (main.go): Added a new Terminate method to AteomHerder: • Calls ateom.TerminateWorkload on the target worker pod. • Unmounts external volumes mounted for the actor on the node. • Cleans up and resets actor runtime host directories (OCI bundles, checkpoints, pid files). #### 3. In-Pod Sandboxes (ateom-gvisor & ateom-microvm) • TerminateWorkload RPC: • gVisor (main.go): Stops and deletes active runsc containers, unmounts bundle rootfs overlays, and tears down the actor's interior network namespace. • Micro-VM (checkpoint.go): Shuts down the Cloud Hypervisor VMM, unmounts bundle rootfs overlays, and cleans up the network namespace. • Emits an "Actor terminated" lifecycle log and clears the active actor attribution so the worker can host new workloads. #### 4. CLI (kubectl-ate) • --any-state Flag (delete_actor.go): kubectl-ate delete actor <actor-name> -a <atespace> --any-state ────── ## Verification [x] Changes tested in local with ``` • go build ./...: Passed go test ./...: Passed (all unit and functional tests pass) make lint: Passed (no lint errors) ``` Detailed manual tests ``` ### Scenario 1: Force Delete a Running Actor with --any-state 1. Create the actor: kubectl ate create actor manual-test-1 --template=ate-demo-counter-microvm/counter-microvm -a demo 2. Resume the actor: kubectl ate resume actor manual-test-1 -a demo 3. Force delete the running actor: kubectl ate delete actor manual-test-1 -a demo --any-state actor "manual-test-1" deleted ``` ────── ### Scenario 2: Standard Delete a Suspended Actor ``` 1. Create the actor: kubectl ate create actor manual-test-2 --template=ate-demo-counter-microvm/counter-microvm -a demo 2. Resume the actor: kubectl ate resume actor manual-test-2 -a demo 3. Suspend the actor: kubectl ate suspend actor manual-test-2 -a demo 4. Standard delete the suspended actor: kubectl ate delete actor manual-test-2 -a demo actor "manual-test-2" deleted ``` ────── ### Unit & E2E Tests # Unit & Functional Tests go test ./cmd/ateapi/internal/controlapi -run "TestDeleteActorWorkflow|TestEnsureMarkedDeleting" # E2E Tests E2E_TEMPLATE_NAMESPACE=ate-demo-counter-microvm E2E_TEMPLATE_NAME=counter-microvm ./hack/run-e2e-kind.sh ./internal/e2e/suites/demo -run TestActorLifecycle ./hack/run-e2e-kind.sh ./internal/e2e/suites/demo -run TestForceDeleteActorWithExternalVolume ``` - [ x] Appropriate changes to documentation are included in the PR |
||
|
Julian Gutierrez Oschmann
|
4c1bd9d36b |
Add status fields to Substrate resources (#1025)
Add `status` fields to all Substrate resources that need it. Move server-owned fields under it. Fixes #1006 . |
||
|
Troy Chiu
|
07da819da1 |
controlapi: refactor workflows to typed ensure-steps (#797)
Fixes #554 Each workflow becomes a plain method orchestrating typed, idempotent ensureX steps, e.g.: ```go func (w *ActorWorkflow) ResumeActor(ctx, ref, boot) (*Actor, bool, error) { // acquire actor lock, metrics defer — unchanged actor, tmpl, snapSrc, err := w.loadActorForResume(ctx, ref, boot) if actor.Status == RUNNING { return actor, false, nil } // explicit no-op actor, err = w.ensureVolumesCreated(ctx, actor, tmpl) actor, worker, err = w.ensureWorkerAssigned(ctx, actor, tmpl) err = w.ensureVolumesAttached(ctx, actor, worker, tmpl) tele, err = w.ensureAteletRestored(ctx, actor, worker, tmpl, snapSrc) actor, err = w.finalizeRunning(ctx, ref) return actor, true, nil } ``` Each `ensureX` - early-returns if its postcondition already holds, which is derived from persisted state only, replacing `IsComplete` - validates the state machine edge internally, replacing `CheckPrerequisite` - takes explicit typed inputs and returns what it produced. - persists what it changed before returning, returning the stored copy so the optimistic lock version threads forward. No shared `*ResumeState`. There is one behavioral change: worker ownership/eligibility validation for a re-entered resume now runs before volumes are attached (it used to run in `CallAteletRestoreStep`'s prerequisite, after attach), so a resume that is about to crash on a reassigned/ineligible worker no longer attaches volumes to it first. - [X] Tests pass - [ ] Appropriate changes to documentation are included in the PR |
||
|
Michelle Au
|
ac3f08f7ad |
Handle partial volume creation failures (#527)
Part of #232 1. During CreateActor, the volumes to be created get instantiated in the Actor API in PENDING status. The Actor is still instantiated as SUSPENDED. However, the CreateVolume operation is moved to the beginning of ResumeActor. This is because CreateActor is intentionally designed to not be idempotent. The downside is that the user won't know dependent resource creation failed until the first resume. 2. Changes volume id to use actor uid instead of atespace+actorname and adds "substrate" prefix 3. Adds a new DELETING status that gets persisted before we start to delete volumes. This shifts the existing DB DELETE precondition checks to the new DELETING status. And now you can only delete the actor in DB from DELETING status. 4. Refactor DeleteActor to use a workflow In the future, we will also add a workflow that will do mandatory node cleanup of volumes before starting to delete the volumes. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |