Release run 30385988475 lost three Windows smoke jobs to one phenomenon on two
different surfaces. A Windows file that was just written or just executed can
refuse deletion and rename for a moment while a scanner reads it or while the OS
finishes reaping a child. Both conditions clear on their own within moments.
It reads as nondeterminism because it is timing: windows-2025 passed both amd64
variants while windows-latest -- the SAME image, windows-2025-vs2026 -- failed
both, and windows-11-arm failed too.
PRODUCT: uninstall abandoned a live installation.
With one binary per platform, uninstall must remove the very image it is
executing, and on Windows a running image can only be renamed, never
overwritten. That rename got a single attempt, so a transient sharing violation
surfaced to the user as
error: failed to remove ...; completed configuration/index cleanup may remain
with the install left half-torn-down. install.ps1 already retries this exact
operation ten times for this exact reason; the C path now does the same, and
only for sharing/access/lock violations, so a file that is genuinely held still
fails closed instead of spinning.
Proven with a failure-injection seam rather than by racing a real scanner:
3 injected failures must still succeed, 64 must still fail AND leave the target
in place. Deterministic, so it does not need CI to learn whether it works.
HARNESS: the smoke killed itself silently.
Phases that install a binary then `rm -rf` the directory hit the same lock. Under
`set -euo pipefail` that returns non-zero and ends the run WITHOUT printing
anything -- the logs jump straight from "OK 13h" to job cleanup with no FAIL
line, which is what made this expensive to read.
None of those 16 removals is an assertion; they are fixture cleanup, and
_smoke.yml already states the principle ("a temp dir on an ephemeral runner must
never fail the job") that smoke-test.sh did not honour. They now go through
smoke_rmtree, which retries so the disk is actually reclaimed -- these fixtures
hold ~300 MB binaries and runners are disk-tight -- then warns and continues.
Verified: Windows VM full smoke ALL PASSED (previously died after 13h), macOS
full smoke ALL PASSED, activation_transaction + cli 259 passed on Windows, all
10 shell contracts pass, lint-ci clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel
0a22fb14df