Follow-up to #2626
The control plane only found out a sandbox died when the next operation
on the actor failed, which for an idle actor could be never.
- New WorkerService.ReportActorCrashed (and the atelet AteomSupport
mirror), authorized like RequestActorSuspend. A RUNNING actor is crashed
under its lease, anything else is refused since the operation in flight
owns it.
- ateworker-microvm reaps cloud-hypervisor / virtiofsd as soon as they
exit, and ateworker-gvisor watches the runsc sandbox. If one dies while
ateworker isn't stopping it, ateworker tears the actor down itself (the
crash clears the assignment, so nothing else would), writes its final
usage record, and reports it
- Exits are watched through pidfds on the netpoller, so watching doesn't
cost a thread per process
- The gVisor sandbox is pinned right after runsc create, which closes
the pid reuse gap @sfunkenhauser pointed out on #2626
- Not covered yet: a workload exiting while its sandbox stays up, and a
micro-VM guest panic, which reboots the guest without cloud-hypervisor
exiting. Both need their own design ...
Tested with a new e2e test, TestSandboxCrashIsReported (kills the
sandbox from an ephemeral container, expects CRASHED, then revert +
resume), plus both full lanes on GKE with no false crash reports.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR (none
needed)
This PR was AI assisted. I have tested everything on GKE and reviewed
the changes.
Benjamin Elder
d4204fd852