6 Commits

Include renames

Author	SHA1	Message	Date
Martin Vogel	0d783bcca4	test(harness): hold the scheduler contract's fixtures by construction ... The parallel-scheduler contract failed twice on test-windows CLANG64 while running fake suites only: - PR #2394 (run 36364479687): "Windows timeout race refused without a surviving descendant to refuse over". The refusal came 38s after the forced leader exit (a cold powershell/CIM descendant probe, once in the wave loop and again in the cleanup pass), while the fixture's descendant was a `time.sleep(30)` that had already exited on its own. - PR #2345 (run 36199749518): "suite 'hang_after_summary' taskkill could not prove process-tree cleanup". The scheduler bounded taskkill.exe with --kill-grace, which the contract passes as 1s, so a slow cold start of the tool was reported as a failed cleanup. Both verdicts were a race between a clock and runner latency. An audit of the fixture found more of the same shape: - Every hanging fixture ended on its own after 30s. A starved scheduler could therefore see a "hung" leader exit 0, and a descendant the scheduler failed to kill could vanish before the leak check looked. - The 1s --timeout could kill a slow leader before it had printed its summary (asserted pass=1), spawned its descendant or written the descendant pid. That pid was then read back with int() from a file written create-then-write, so a reader could see no file or an empty one. The empty-file read recorded earlier was the scheduler's ready file, which 3b9052c9 already publishes atomically; the fixture's own writer had the same shape. - The descendant installed its SIGTERM-ignore during its own start-up. A SIGTERM that won that race skipped the SIGKILL escalation the POSIX leg exists to cover, silently and with the verdict unchanged. - On Windows, the stubborn-tree leak check was a single look right after taskkill, but TerminateProcess returns before its target has exited. Fix it by construction, not by budget: - A gate. Every process that must outlive the scheduler's decision blocks reading stdin. That is a pipe whose only write end the contract holds, handed down through the scheduler, and the read returns only at EOF. The contract closes the pipe once the verdict is recorded. If the contract dies, EOF releases everything, so nothing is orphaned and no fixture ends on a clock. - The three hanging scenarios run under the scheduler's existing pre-terminate barrier. They are released only once the fixture has published `<suite>.established`, which happens after the summary is flushed, the descendant has confirmed it is armed, and its pid is known. The marker is written to a temp file and moved into place with os.replace, so a reader sees either no file or the whole file. - The Windows leak check waits on the descendant's process handle. That wait is exact, not a race: the descendant is held by the gate, so the scheduler's kill is the only way it can end. - run-test-wave.py runs taskkill through windows_taskkill_tree() with its own stable-state budget (WINDOWS_TASKKILL_SECONDS = 15), as 132e8fc3 did for the descendant probe. --kill-grace keeps bounding what its name says. Real runs already pass --kill-grace 15, so they are unchanged. The contract pins this structurally, the same way it pins the probe: exactly one taskkill site, no kill-grace there, and a helper bounded by the constant that fails closed on a timeout, a start failure or a non-zero exit. Every assertion keeps its contract: - A leader that hangs after a green summary is recorded as rc=124 pass=1. - A zero-test child is recorded as rc=97. - The wave continues after a bounded failure. - A SIGTERM-resistant tree is killed in full. - A leader that exits between the timeout decision and termination is still cleaned up on POSIX. On Windows the scheduler refuses (rc=2, naming the cleanup failure) over a descendant that is really alive. A new self-check fails loudly if a held leader or descendant is not live while the barrier holds, that is, if the fixture no longer exercises what those assertions claim. Proof, on macOS arm64 and a Linux arm64 container. The Windows branch was reasoned through; the VM is down. - Torture hook (proof only) that delays each hanging leader 2s, past the 1s timeout. The old fixture is RED 3/3 on every scenario (pass=0, FileNotFoundError on descendant.pid, "lost its bounded result"). The new one is green, including at 8s, and 10/10 under the hook plus CPU load. - The #2394 mechanism: after an injected 31s refusal latency the old descendant is gone and the new one is live. It exits 0.02s after the gate closes. - The taskkill pin is RED on the old scheduler and green on the new one. - Stress, 20 runs each under CPU load. The old full contract passed 20/20 on macOS and 20/20 on Linux, so these hosts do not reproduce the CI timing. The rebuilt held scenarios passed 20/20 on macOS and 20/20 on Linux. The full contract after the change passes a single run on macOS and on Linux, and 20/20 on each leg under CPU load (4 load workers; the Linux container is capped at 4 CPUs). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>	2026-09-28 19:45:45 +02:00
Martin Vogel	fa99ff64eb	test(cli): stop a negative probe from spending 15s proving silence ... The macos-15-intel leg killed the cli suite at its 900s wall clock while reporting 7537 passed, 0 failed. Two separate things were wrong. The suite was misclassified. cli spends 497s of the 900s default on macos-14, the FASTEST macOS runner, while macos-15-intel in the same matrix is 2.4-3.6x slower on comparable suites (daemon_runtime 842s vs 349s, stack_overflow_b 277s vs 76s). 497s at that ratio cannot fit. daemon_runtime, at 842s on that same runner, passes only because it was already in SLOW_SUITES. cli belongs there too -- it is deterministic and simply large, which is the only thing that list is allowed to mean. The drain test was also waiting on a clock instead of on the system. Profiling the suite per test (315 tests, 300s wall / 166s cpu) put cli_install_into_host_namespace_still_drains_host_cohort at 42s wall for 19s of cpu -- 23s idle, the largest single idle block, where the next worst was 11s. The existing cli_install_force_quiesces_active_cohort_before_replacing_binary drains a cohort too and idles 0.5s, which is what pointed at the difference. The cost was the host_serving probe. Its generous budget exists for the POSITIVE question -- is this daemon still up? -- where a slow reply on a loaded runner must not be misread as drained; that fixed a real flake and is kept. Asked in the negative it inverts: no reply is coming, so the full 15s is spent establishing silence and ASSERT_FALSE is decided by the timeout expiring rather than by the daemon's behaviour. The drain is already proven positively in that test -- install returns 0 only after the activation completed, and the host child exits CLI_SCOPE_HOST_DRAINED -- so the probe only has to confirm it. Also shortens the drained child's teardown budget, which retried service_free / lease_release / manager_free against 10s on a path where the activation had already torn the service down. Behaviour at the deadline is unchanged; it is reached sooner when nothing is wedged. Measured at only -3s on its own, kept because it is correct and free. Local, same machine, ASan+UBSan: the drain test 42.0s -> 25.8s (-38%), the suite 300s -> 279s, 315 passed before and after -- no test removed, no assertion weakened. PR #2188, which runs the same leg without these two tests, passed macos-15-intel in 37m34s, so the suite was not already over budget on main. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>	2026-09-12 18:01:46 +02:00
Martin VogelandKody	3b9052c955	test: write scheduler barrier files atomically ... Path.write_text is open-then-write: it creates and truncates the file first and the content lands afterwards. The parallel harness contract polls for `<suite>.ready` to appear and then parses its content as the leader pid, so a reader that wins that window sees an empty file and fails with `ValueError: invalid literal for int() with base 10: ''` -- a spurious FAIL of the harness contract with a scheduler-side cause. The same shape applied to `<suite>.leader-exited`. Publish the barrier files through a helper that writes a same-directory temp file and os.replace()s it onto the destination, which is atomic on POSIX and Windows: a poller now sees either no file or complete content. All three barrier publications go through it. Pin the contract structurally (no timing): the scheduler may not write a barrier file in place, and it must rename into place instead. The pin is RED on the previous scheduler. Distilled from #1188 with co-author credit. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> Co-authored-by: Kody <kaidi.shi.1121@gmail.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>	2026-09-05 21:11:48 +02:00
Martin Vogel	132e8fc323	test: give the Windows descendant probe its own budget ... When a suite leader has already exited, taskkill /T cannot walk its tree, so run-test-wave.py proves cleanup by asking Windows whether anything is still parented to the dead PID. That probe was timed with --kill-grace, which the parallel gate passes as 1s. A cold powershell.exe + CIM start routinely costs more than that on a runner, so TimeoutExpired -- a fact about interpreter start-up, not about the process tree -- was read as "assume the worst" and failed an already-clean shard with exit 2. Give the probe its own stable-state budget (15s) plus one retry, keeping the fail-closed semantics: an exhausted or failed probe still counts as "cannot prove absence". The refusal now names which of the two happened, an unfinished probe or a counted set of live descendants, instead of reporting both as a leaked tree. The contract test pins this structurally -- no sleeps, no timing thresholds: the probe must be bounded by the module constant rather than by the kill-grace argument, must retry a timed-out probe, must never read an unusable probe as absence, and must report the two failures distinctly. Its Windows leader-exit case now waits on the scheduler's refusal rather than on a deadline shorter than the new budget. Refs #1741 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>	2026-09-04 17:58:37 +02:00
Martin Vogel	ae52db3147	test: make three Windows nondeterminisms deterministic ... All three cost real release cycles, and none is fixed by widening a budget. daemon_application_cancels_physical_job_only_after_final_session waited for the SUBSCRIBER COUNT to reach 2, then cancelled both sessions and asserted the physical job had started exactly once. The job starts asynchronously after subscription, so both cancels could land first and leave starts == 0 -- arguably the correct outcome. It now waits for the state the assertions actually require. Verified 47/47 on Windows, the only platform it ever failed. The parallel harness refused outright when the suite leader had already exited, because taskkill /T cannot walk a tree from a dead PID. But the leader can exit between the timeout decision and that call, so the harness itself lost a race: a natural exit at the wrong moment failed the whole wave. It now proves cleanup the only way still available -- nothing parented to that PID -- and its contract asserts the PROPERTY rather than the phrase "tree cleanup" it used to grep for. That string pin is what broke when the guard was reworded while behaving correctly; the contract now checks rc==2 AND that the descendant really did survive, which would also catch a guard that claims to fail closed while leaking. extract_wide_flat_file_is_linear took ONE sample per size, so the ratio carried the noise of both. On a loaded Windows VM linear code measured 51x against a 40x bound (184ms -> 9387ms). Best-of-N instead: timing noise only ever adds time, so the minimum is the cheapest good estimate of the noise-free cost. The bound is deliberately unchanged -- it sits where linear (~20x) and quadratic (~128x) are each >=2x away, so raising it would move the test toward the very signal it exists to catch. Now measures 19.1x on Windows, 21.4x on macOS. Also: the smoke's `cli` helper redirected stderr to a file and discarded it, so any of the 10 bare `VAR=$(cli ...)` assignments could kill the run under `set -euo pipefail` printing NOTHING. One such abort cost a full Windows cycle just to locate and still could not be attributed. It now surfaces the command and its stderr. Neutral wording on purpose: one call site expects a non-zero exit and must not read as a failure. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>	2026-07-29 17:05:10 +02:00
Martin Vogel	62821375c4	fix: harden daemon release paths and verification ... Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>	2026-07-24 16:08:42 +02:00