mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-31 07:44:57 +08:00
pkhodade-NV
e084e1b09e
* fix(server): keep Error-phase sandbox records through the prune sweep The periodic store-vs-backend reconciliation sweep deleted any persisted sandbox not present in the driver's live backend snapshot, except for Completed and failed-main-process phases. A driver whose registry is in-process-only and never rehydrates after a restart (no persistence of its own) reports every previously-known sandbox as missing on the very first sweep after startup -- including ones already correctly, terminally marked Error by earlier crash detection -- so the sweep silently deleted them shortly after gateway restart, racing any client (GetSandbox/ListSandboxes/DeleteSandbox) working with the same sandbox in that window. Treat Error the same as the existing Completed exemption: it is already a settled, informational terminal state with no live compute resource to reclaim, so keep the durable record instead of deleting it. Signed-off-by: Prashant Khodade <pkhodade@nvidia.com> (cherry picked from commit 721a1659a822a72e76e3c0dffc6847f17129a3fc) * fix(server): narrow the prune sweep's Error-phase exemption The blanket phase == SandboxPhase::Error exemption changed established missing-backend cleanup for every compute driver, not only the MXC restart race the PR intended to fix. It also matched BackendResourceMissing (set by gateway-startup recovery when a previously-known sandbox's backend resource is already gone), StartFailed (startup recovery's driver-error case), and ComputeResourceMissing (this same sweep's own first-pass Error transition for a Stopping/Stopped/Starting sandbox). All three mark exactly the orphaned resources this sweep exists to reclaim across Docker, Podman, VM, Kubernetes, and extension drivers -- exempting them left orphaned names and gateway-owned records in place indefinitely and skipped the idempotent driver cleanup for volumes/secrets until a user explicitly deleted the sandbox. Add is_missing_compute_resource_reason to inspect the sandbox's Ready condition and narrow the exemption to a settled Error record only: one whose reason isn't one of those three. A crashed main process or any other non-resource failure keeps the exemption (no live resource ever expected again); a resource-missing reason keeps flowing through the normal delete-and-cleanup path exactly as before this PR. Adds regression coverage for BackendResourceMissing and ComputeResourceMissing confirming they are still pruned with driver cleanup invoked, and documents the settled-vs-missing-resource retention distinction in architecture/compute-runtimes.md. Signed-off-by: Prashant Khodade <pkhodade@nvidia.com> --------- Signed-off-by: Prashant Khodade <pkhodade@nvidia.com>