ci: non-product PRs (T0b) run the contract steps too

Under R1, a T0b push (test-infrastructure/, lint config, graph-ui, memwaste
tooling) selected lint and no test leg, so the static contract steps that
police exactly those files -- the VM scripts, the compose and run.sh wiring,
the venue-parity rules -- ran nowhere. Replayed over September, 13 of the 18
T0b pushes ran no contract step at all.

The R1 table now gives T0b lint + contracts, so those PRs get the
contracts job (scripts/test.sh --contracts-only) as T0a PRs do. The
fixture table's six T0b cases expect it, and the history replay asserts
that every push runs the contract steps (the contracts lane or any test
leg). Comments that described the job as docs-only are updated.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This commit is contained in:
Martin Vogel
2026-09-30 23:48:22 +02:00
parent d3efe3baec
commit 16e2732819
7 changed files with 32 additions and 18 deletions
+1 -1
View File
@@ -17,7 +17,7 @@ codes) and rejects unknown flags with exit 2 + `Please consult --help.`
| leg | entry | what a run gives you |
|---|---|---|
| **test** | `test.sh` | DEFAULT = the venue leg: static contracts (Step 0a–0r) + CLEAN ASan+UBSan build + all suites via the parallel harness + prod-binary guards. `--suites a,b` = iteration mode (incremental rebuild, subset, seconds). `--tsan` = the ThreadSanitizer leg. `--contracts-only` = the static contract steps alone (PR CI's docs-only job). CLANGARM64 gets CI's trap-UBSan flags automatically. |
| **test** | `test.sh` | DEFAULT = the venue leg: static contracts (Step 0a–0r) + CLEAN ASan+UBSan build + all suites via the parallel harness + prod-binary guards. `--suites a,b` = iteration mode (incremental rebuild, subset, seconds). `--tsan` = the ThreadSanitizer leg. `--contracts-only` = the static contract steps alone (PR CI's contracts job for docs-only and non-product changes). CLANGARM64 gets CI's trap-UBSan flags automatically. |
| **package wrappers** | `ci/test-package-wrappers.sh` | Go, npm, and PyPI runtime-set publication/lock suites on the current host. CI runs the same entry on Linux and Windows so both platform lock implementations gate. |
| **build** | `build.sh` | CLEAN production runtime set (native executable + authenticated integration asset; `--with-ui` adds one content-addressed UI pack). ccache via `env.sh` makes repeats fast; `CCACHE_COMPILERCHECK=content` guarantees a hit is byte-identical to a cold compile — never stale. `--version`, `STATIC=1`, `BUILD_DIR=`. |
| **lint** | `lint.sh` | clang-tidy + cppcheck + clang-format (+ no-skips policy). `--ci` = the CI gate set (no clang-tidy). Drives the same make targets as `make lint`/`lint-ci`. |
+3 -2
View File
@@ -251,14 +251,15 @@ FULL = {"codeql-gate", "diag", "license-gate", "lint", "lint-mem", "lsan-macos",
"smoke-mac", "smoke-ubuntu", "smoke-win", "tsan-arm", "tsan-mac", "tsan-x86",
"unix-arm64", "unix-macos-intel", "unix-macos14", "unix-x86", "windows",
"windows-guards"}
UNIVERSE = FULL | {"contracts"} # contracts: the docs-only job; every test leg runs them too
UNIVERSE = FULL | {"contracts"} # contracts: the T0a/T0b job; every test leg runs them too
LINT = {"lint", "lint-mem"}
CORE_LEGS = {"unix-x86", "unix-macos14", "windows", "shard-completeness"}
TEST_LEGS = {"unix-x86", "unix-arm64", "unix-macos14", "unix-macos-intel", "windows"}
ALWAYS = {"security-static"}
R1 = {
"T0a-docs": {"contracts"},
"T0b-nonproduct": LINT,
# The contract steps police test-infrastructure/ and lint config too.
"T0b-nonproduct": LINT | {"contracts"},
"T1-tests": LINT | CORE_LEGS,
"T2-extract": LINT | CORE_LEGS | {"codeql-gate", "tsan-x86", "smoke-ubuntu"},
"T3-mid": LINT | CORE_LEGS | {"codeql-gate", "tsan-x86", "smoke-ubuntu"},
+16 -3
View File
@@ -9,7 +9,9 @@
# its documented deviation, and a deviation may only ever raise the tier
# (the fail-safe direction) -- never quietly select less than was measured;
# 2. every real failure of the month (22, fixtures/2026-09-real-failures.tsv)
# had its catching lane selected for its push.
# had its catching lane selected for its push;
# 3. every push runs the static contract steps (the contracts lane or any
# test leg) -- they police docs, test-infrastructure/ and lint config.
# Pure data, one selector process (--batch over the distinct file sets).
#
# Usage: scripts/test-impact/replay-selector.sh [repo-root]
@@ -17,7 +19,7 @@ set -euo pipefail
case "${1:-}" in
-h | --help)
sed -n '2,15p' "$0" | sed 's/^# \{0,1\}//'
sed -n '2,17p' "$0" | sed 's/^# \{0,1\}//'
exit 0
;;
-*)
@@ -111,6 +113,16 @@ for run_id, pr, sha, fileset, analysis, expected, note in rows("2026-09-pushes.t
else:
deviated += 1
# The static contract steps police docs, test-infrastructure/ and lint config,
# so every push runs them: in the contracts job or inside any test leg.
CONTRACT_CARRIERS = {"contracts", "unix-x86", "unix-arm64", "unix-macos14",
"unix-macos-intel", "windows"}
uncovered = collections.Counter(d["tier"] for d in pushes.values()
if not set(d["lanes"]) & CONTRACT_CARRIERS)
for tier, count in sorted(uncovered.items()):
failures.append(f"{count} {tier} push(es) run no contract step (neither the "
f"contracts lane nor a test leg)")
caught = 0
for run_id, attempt, pr, sha, lane, why in rows("2026-09-real-failures.tsv", 6):
decision = pushes.get(run_id)
@@ -129,7 +141,8 @@ if failures:
sys.exit(1)
print(f"selector history replay OK: {len(pushes)} pushes, tier as expected for all "
f"({matched} equal to the analysis, {deviated} documented fail-safe deviations); "
f"{caught} real failures, every catching lane selected")
f"{caught} real failures, every catching lane selected; every push runs the "
f"contract steps")
print(" pushes per tier: " + ", ".join(f"{t} {tiers[t]}" for t in TIERS))
print(f" full runs: {sum(1 for d in pushes.values() if d['full'])}")
PY
+1 -1
View File
@@ -32,7 +32,7 @@ Modes:
--contracts-only
Only the static contract steps (Step 0*) of the default leg:
no compiler, no build, no suites. PR CI runs this for
docs-only changes, where no test leg is selected.
docs-only and non-product changes (tiers T0a, T0b).
Options:
--arch ARCH Force target arch (arm64 | x86_64), e.g. under Rosetta.