mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-29 07:23:25 +08:00
Emilien Macchi
07b7a74b62
The provisioning-deadline scan takes each provisioning sandbox's local S(global) S(workspace) X(sandbox) keys in ascending key order and waited for them with no deadline. When a sandbox's key sorts before its workspace's key, the scan held X(sandbox) while it waited behind a provider or profile writer, so start, stop and delete for that sandbox waited for the writer too. In the other order, the scan held S(workspace) while it waited for a busy sandbox, and provider writers on that workspace queued behind it. Either way the rest of the pass waited as well. Each pass now has a 500 ms budget for waiting on these keys. A sandbox whose keys stay busy is skipped and picked up on the next pass a second later. Once the budget is spent, the rest of the pass only takes keys that are free when it reaches them, so a busy workspace or global key costs one wait per pass rather than one per sandbox. The local acquisition is cancel-safe, so giving up releases the keys it already took. Skips log at debug and count in the new openshell_server_provisioning_deadline_skips_total counter, labeled by whether the sandbox was already past its deadline. A steady rate of delayed expiries can then be alerted on without a warning every pass. The gateway metrics page and the cluster debugging skill describe it. The test that pinned the old wait is replaced by tests for both key orders: a lifecycle lock or a provider writer proceeds and the scan finishes while the other side holds its key, the skipped sandbox is untouched and expires once the writer releases, and a skip doesn't end the pass for other sandboxes. Another test checks that sandboxes behind one busy workspace share a single wait. Part of #4307 Closes #4547 Signed-off-by: Emilien Macchi <emacchi@redhat.com>