Flake audit: the whole corpus has exactly two wall-clock test deadlines —
the EOF-shutdown alarm(5) and the issue-832 supervisor-spawn alarm(60),
both in the mcp suite (the latter already flaked once under heavy local
load). Serializing mcp costs ~6s of tail and closes the starvable-deadline
class completely; every remaining parallel-wave suite is deadline-free.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
First CI validation round of the parallel runner, three findings:
- Windows failed ALL 104 suites with "0 passed, 1 failed" each: the CRT
emits --list-suites lines as CRLF, so every dispatched name carried a
trailing CR ("arena\r" is an unknown suite). The union guard caught it
loudly, exactly as designed. The driver now strips CR when writing the
suite list.
- The ubuntu-gcc legs had 3 real failures in the cli suite: it spawns
subprocesses with fixed deadlines, and a fully saturated 4-core runner
starves those deadlines into flakes. Deadline-sensitive suites (cli,
subprocess, watcher, incremental, httpd, ui, index_resilience, and the
stack_overflow family) now run SEQUENTIALLY after the parallel wave on a
quiet machine — same suites, same tests, same gates, only the schedule
differs; the union guard checks the combined result set.
- Failing suites now print every FAIL site with context (the tail-30 of a
long suite log hid which tests actually failed on CI).
Local: totals unchanged (6,361 passed / 0 failed / 1 skipped), union guard
clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The runner executed all 104 suites sequentially (~10 min locally, the bulk
of every CI test leg). The suites are process-isolated already (per-process
mkdtemp HOME sentinel; two full parallel runs produced zero cross-suite
failures), so the serialization was pure convention.
- test-runner --list-suites: prints every registered suite, one per line,
emitted by the SAME macro table that executes suites — the list cannot
drift from the run set by construction.
- scripts/run-tests-parallel.sh: runs each suite as its own process
(jobs = CPU count; CBM_TEST_PAR_JOBS overrides) under a ZERO-LOSS
CONTRACT: a union guard fails the gate if the set of suites that
produced a result differs from --list-suites (nothing can be silently
dropped, a newly added suite is picked up automatically); per-suite
pass/fail/skip are summed into the sequential runner's exact summary
format; any suite crash, failure, or omission exits nonzero. Per-suite
wall times are printed for balance tracking.
- make test-par: the parallel target. make test (sequential) is unchanged
and remains the escape hatch (CBM_TEST_SEQUENTIAL=1 in test.sh).
- scripts/test.sh: builds, then routes through test-par — every CI test
leg gets the speedup with zero workflow-topology change (same jobs,
same gates, same billing; the legs just finish sooner).
- The stack_overflow suite is split into a/b/c (7+6+7 of its 20 tests,
pure re-registration): as one suite it was the wall-clock critical path
of any parallel run — every other suite finished underneath its ~4
minutes.
Measured locally (Apple Silicon, ASan runner): sequential ~10 min vs
parallel 232 s, totals identical (6,361 passed / 0 failed / 1 skipped),
union guard clean. The TSan leg keeps its dedicated subset runner
(sequential) — out of scope here.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel