mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-11-10 10:13:50 +08:00
Sneha-at
b7080602c6
Fixes #643 This feature introduces the ability to delete Substrate actors from any lifecycle state (e.g., RUNNING, PAUSED, PENDING), rather than requiring them to be hibernated into SUSPENDED (or CRASHED) beforehand. This is crucial for cleaning up stuck or failed actors. #### 1. Control Plane & gRPC API (pkg/proto/ateapipb, ateapi) • DeleteActorRequest.any_state: Added a new boolean field any_state to DeleteActorRequest. • When false (default): enforces the existing behavior where only actors in ACTOR_STATE_SUSPENDED or ACTOR_STATE_CRASHED (or already ACTOR_STATE_DELETING) can be deleted. • When true: allows deleting an actor in any state (e.g., RUNNING, PAUSED). • Orchestrated Cleanup Workflow (workflow_delete.go): 1. Transitions actor state to ACTOR_STATE_DELETING. 2. Calls atelet.Terminate to stop live workloads on the node. 3. Detaches external volumes from the worker node (with fallback logic using actor status if the ActorTemplate was deleted). 4. Releases the assigned physical worker Pod back to the WorkerPool. 5. Deletes external storage volumes through the storage plugins. 6. Finalizes removal of the actor from the persistence store. #### 2. Node Herder Agent (atelet) • Terminate RPC (main.go): Added a new Terminate method to AteomHerder: • Calls ateom.TerminateWorkload on the target worker pod. • Unmounts external volumes mounted for the actor on the node. • Cleans up and resets actor runtime host directories (OCI bundles, checkpoints, pid files). #### 3. In-Pod Sandboxes (ateom-gvisor & ateom-microvm) • TerminateWorkload RPC: • gVisor (main.go): Stops and deletes active runsc containers, unmounts bundle rootfs overlays, and tears down the actor's interior network namespace. • Micro-VM (checkpoint.go): Shuts down the Cloud Hypervisor VMM, unmounts bundle rootfs overlays, and cleans up the network namespace. • Emits an "Actor terminated" lifecycle log and clears the active actor attribution so the worker can host new workloads. #### 4. CLI (kubectl-ate) • --any-state Flag (delete_actor.go): kubectl-ate delete actor <actor-name> -a <atespace> --any-state ────── ## Verification [x] Changes tested in local with ``` • go build ./...: Passed go test ./...: Passed (all unit and functional tests pass) make lint: Passed (no lint errors) ``` Detailed manual tests ``` ### Scenario 1: Force Delete a Running Actor with --any-state 1. Create the actor: kubectl ate create actor manual-test-1 --template=ate-demo-counter-microvm/counter-microvm -a demo 2. Resume the actor: kubectl ate resume actor manual-test-1 -a demo 3. Force delete the running actor: kubectl ate delete actor manual-test-1 -a demo --any-state actor "manual-test-1" deleted ``` ────── ### Scenario 2: Standard Delete a Suspended Actor ``` 1. Create the actor: kubectl ate create actor manual-test-2 --template=ate-demo-counter-microvm/counter-microvm -a demo 2. Resume the actor: kubectl ate resume actor manual-test-2 -a demo 3. Suspend the actor: kubectl ate suspend actor manual-test-2 -a demo 4. Standard delete the suspended actor: kubectl ate delete actor manual-test-2 -a demo actor "manual-test-2" deleted ``` ────── ### Unit & E2E Tests # Unit & Functional Tests go test ./cmd/ateapi/internal/controlapi -run "TestDeleteActorWorkflow|TestEnsureMarkedDeleting" # E2E Tests E2E_TEMPLATE_NAMESPACE=ate-demo-counter-microvm E2E_TEMPLATE_NAME=counter-microvm ./hack/run-e2e-kind.sh ./internal/e2e/suites/demo -run TestActorLifecycle ./hack/run-e2e-kind.sh ./internal/e2e/suites/demo -run TestForceDeleteActorWithExternalVolume ``` - [ x] Appropriate changes to documentation are included in the PR