The macos-15-intel leg killed the cli suite at its 900s wall clock while
reporting 7537 passed, 0 failed. Two separate things were wrong.
The suite was misclassified. cli spends 497s of the 900s default on macos-14,
the FASTEST macOS runner, while macos-15-intel in the same matrix is 2.4-3.6x
slower on comparable suites (daemon_runtime 842s vs 349s, stack_overflow_b
277s vs 76s). 497s at that ratio cannot fit. daemon_runtime, at 842s on that
same runner, passes only because it was already in SLOW_SUITES. cli belongs
there too -- it is deterministic and simply large, which is the only thing that
list is allowed to mean.
The drain test was also waiting on a clock instead of on the system. Profiling
the suite per test (315 tests, 300s wall / 166s cpu) put
cli_install_into_host_namespace_still_drains_host_cohort at 42s wall for 19s of
cpu -- 23s idle, the largest single idle block, where the next worst was 11s.
The existing cli_install_force_quiesces_active_cohort_before_replacing_binary
drains a cohort too and idles 0.5s, which is what pointed at the difference.
The cost was the host_serving probe. Its generous budget exists for the
POSITIVE question -- is this daemon still up? -- where a slow reply on a loaded
runner must not be misread as drained; that fixed a real flake and is kept.
Asked in the negative it inverts: no reply is coming, so the full 15s is spent
establishing silence and ASSERT_FALSE is decided by the timeout expiring rather
than by the daemon's behaviour. The drain is already proven positively in that
test -- install returns 0 only after the activation completed, and the host
child exits CLI_SCOPE_HOST_DRAINED -- so the probe only has to confirm it.
Also shortens the drained child's teardown budget, which retried service_free /
lease_release / manager_free against 10s on a path where the activation had
already torn the service down. Behaviour at the deadline is unchanged; it is
reached sooner when nothing is wedged. Measured at only -3s on its own, kept
because it is correct and free.
Local, same machine, ASan+UBSan: the drain test 42.0s -> 25.8s (-38%), the
suite 300s -> 279s, 315 passed before and after -- no test removed, no
assertion weakened. PR #2188, which runs the same leg without these two tests,
passed macos-15-intel in 37m34s, so the suite was not already over budget on
main.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Path.write_text is open-then-write: it creates and truncates the file
first and the content lands afterwards. The parallel harness contract
polls for `<suite>.ready` to appear and then parses its content as the
leader pid, so a reader that wins that window sees an empty file and
fails with `ValueError: invalid literal for int() with base 10: ''` --
a spurious FAIL of the harness contract with a scheduler-side cause.
The same shape applied to `<suite>.leader-exited`.
Publish the barrier files through a helper that writes a same-directory
temp file and os.replace()s it onto the destination, which is atomic on
POSIX and Windows: a poller now sees either no file or complete content.
All three barrier publications go through it.
Pin the contract structurally (no timing): the scheduler may not write a
barrier file in place, and it must rename into place instead. The pin is
RED on the previous scheduler.
Distilled from #1188 with co-author credit.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: Kody <kaidi.shi.1121@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
When a suite leader has already exited, taskkill /T cannot walk its tree,
so run-test-wave.py proves cleanup by asking Windows whether anything is
still parented to the dead PID. That probe was timed with --kill-grace,
which the parallel gate passes as 1s. A cold powershell.exe + CIM start
routinely costs more than that on a runner, so TimeoutExpired -- a fact
about interpreter start-up, not about the process tree -- was read as
"assume the worst" and failed an already-clean shard with exit 2.
Give the probe its own stable-state budget (15s) plus one retry, keeping
the fail-closed semantics: an exhausted or failed probe still counts as
"cannot prove absence". The refusal now names which of the two happened,
an unfinished probe or a counted set of live descendants, instead of
reporting both as a leaked tree.
The contract test pins this structurally -- no sleeps, no timing
thresholds: the probe must be bounded by the module constant rather than
by the kill-grace argument, must retry a timed-out probe, must never read
an unusable probe as absence, and must report the two failures
distinctly. Its Windows leader-exit case now waits on the scheduler's
refusal rather than on a deadline shorter than the new budget.
Refs #1741
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
All three cost real release cycles, and none is fixed by widening a budget.
daemon_application_cancels_physical_job_only_after_final_session waited for the
SUBSCRIBER COUNT to reach 2, then cancelled both sessions and asserted the
physical job had started exactly once. The job starts asynchronously after
subscription, so both cancels could land first and leave starts == 0 --
arguably the correct outcome. It now waits for the state the assertions
actually require. Verified 47/47 on Windows, the only platform it ever failed.
The parallel harness refused outright when the suite leader had already exited,
because taskkill /T cannot walk a tree from a dead PID. But the leader can exit
between the timeout decision and that call, so the harness itself lost a race:
a natural exit at the wrong moment failed the whole wave. It now proves cleanup
the only way still available -- nothing parented to that PID -- and its contract
asserts the PROPERTY rather than the phrase "tree cleanup" it used to grep for.
That string pin is what broke when the guard was reworded while behaving
correctly; the contract now checks rc==2 AND that the descendant really did
survive, which would also catch a guard that claims to fail closed while
leaking.
extract_wide_flat_file_is_linear took ONE sample per size, so the ratio carried
the noise of both. On a loaded Windows VM linear code measured 51x against a 40x
bound (184ms -> 9387ms). Best-of-N instead: timing noise only ever adds time, so
the minimum is the cheapest good estimate of the noise-free cost. The bound is
deliberately unchanged -- it sits where linear (~20x) and quadratic (~128x) are
each >=2x away, so raising it would move the test toward the very signal it
exists to catch. Now measures 19.1x on Windows, 21.4x on macOS.
Also: the smoke's `cli` helper redirected stderr to a file and discarded it, so
any of the 10 bare `VAR=$(cli ...)` assignments could kill the run under
`set -euo pipefail` printing NOTHING. One such abort cost a full Windows cycle
just to locate and still could not be attributed. It now surfaces the command
and its stderr. Neutral wording on purpose: one call site expects a non-zero
exit and must not read as a failure.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel