mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-10-24 06:23:06 +08:00
A Windows sanitizer job went red on one suite that aborted under AddressSanitizer, and the job log showed an empty "every failure site" section followed by the shadow-byte legend. The summary in scripts/run-tests-parallel.sh searched the suite log for "FAIL" only. A sanitizer report never contains that word, and its useful part (error kind, access size, top frames) is at the start of the report, far above the last 15 lines. The summary now lives in scripts/suite-failure-report.sh, which the harness sources. Between the unchanged failure-site and last-15-lines sections it prints, for a failing suite, the test that was running and the first sanitizer report. The framework flushes each test name before the test body runs, so the last test line ahead of a report is the test that was running; a report that comes after the last test finished is labelled as such instead of blaming a test. The report is shown from its header (AddressSanitizer, LeakSanitizer, ThreadSanitizer, MemorySanitizer, UndefinedBehaviorSanitizer) to its own SUMMARY line, at most 40 lines, without the shadow dump and the legend; a recoverable "runtime error:" line is shown with its stack only. Further sanitizer headers, "runtime error:" lines and SUMMARY lines follow (at most 10), then signal and timeout lines (at most 5). A suite that dies with no report and no completion summary gets the running test named. A trap-mode UBSan build stops on an illegal instruction and prints nothing, so that name is the only evidence its log holds. A log with an assertion failure and no report prints exactly what it printed before. The same file keeps the logs a red run needs. An EXIT trap in the harness copies the log of every suite that did not record rc=0, plus results.txt, into test-logs/failed/. The trap covers the scheduler giving up and the union guard as well as the normal end, and it never changes the exit status. scripts/run-test-wave.py reported pass=0 for a suite that died without its summary line. In that case, and only then, it now counts the PASS markers in the log. A suite with a summary line is never recounted, so the totals of a green run are unchanged. tests/test_failure_report_contract.sh drives the real functions with fabricated suite logs and runs as Step 0i2 of scripts/test.sh. It fails on the previous behaviour in every sanitizer, running-test and result-line check and passes on it in the unchanged-output checks. The awk code uses no interval expressions and no character classes; the test passes with BSD awk and grep on macOS and with mawk and GNU grep on Ubuntu. scripts/README.md names the new internal file. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
554 lines
25 KiB
Bash
Executable File
554 lines
25 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# test.sh — THE canonical test leg. Every venue (local ladder, PR CI, dry run,
|
|
# release) runs tests through this file; iteration happens through its flags,
|
|
# never through a second entry point.
|
|
|
|
set -euo pipefail
|
|
|
|
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
|
cd "$ROOT"
|
|
|
|
usage() {
|
|
cat <<'EOF'
|
|
Usage: scripts/test.sh [--suites LIST | --tsan | --contracts-only] [--arch ARCH] [VAR=VAL ...]
|
|
|
|
The canonical test entry: identical in local CI, PR CI, dry run and release.
|
|
DEFAULT (no --suites) is exactly what CI runs: static contract checks
|
|
(Step 0a-0t), a CLEAN sanitizer build, every suite via the parallel harness,
|
|
then the prod-binary regression guards (Steps 4-6).
|
|
|
|
Modes:
|
|
(default) The venue leg. Clean build (scripts/clean.sh) + all suites +
|
|
all contract steps. This is the shape every gate runs.
|
|
--suites LIST ITERATION mode: comma- or space-separated suite names, e.g.
|
|
--suites daemon,daemon_ipc. Rebuilds the test-runner
|
|
INCREMENTALLY (make dependency tracking, no clean) and runs
|
|
only those suites — seconds, not minutes. Skips the contract
|
|
steps and prod-binary guards; the full default run remains
|
|
the merge gate. Suite names: build/c/test-runner --list-suites.
|
|
--tsan ThreadSanitizer leg (data-race gate): builds and runs the
|
|
widened TSan runner via make test-tsan — the same leg CI's
|
|
tsan jobs and the compose test-tsan service run.
|
|
--contracts-only
|
|
Only the static contract steps (Step 0*) of the default leg:
|
|
no compiler, no build, no suites. PR CI runs this for
|
|
docs-only and non-product changes (tiers T0a, T0b).
|
|
|
|
Options:
|
|
--arch ARCH Force target arch (arm64 | x86_64), e.g. under Rosetta.
|
|
-h, --help This text.
|
|
|
|
Make passthrough (VAR=VAL, forwarded verbatim):
|
|
CC= CXX= Compiler override, e.g. CC=gcc-14 CXX=g++-14.
|
|
BUILD_DIR= Build in an isolated directory (containers/sanitizer variants).
|
|
SANITIZE= Override sanitizer flags. Platform defaults when unset:
|
|
unix/CLANG64 use the Makefile's ASan+UBSan test flags;
|
|
CLANGARM64 (Windows on ARM, no ASan runtime) gets CI's
|
|
trap-UBSan set (-fsanitize=undefined -fsanitize-trap=undefined
|
|
-fstack-protector-strong -fno-omit-frame-pointer) applied HERE
|
|
so local and CI build identical test binaries. Pass SANITIZE=
|
|
(empty) for a plain build when debugging a trap.
|
|
|
|
Environment:
|
|
CBM_TEST_SEQUENTIAL=1 Single-process runner instead of the parallel harness.
|
|
CBM_RUN_HANG_TEST=1 Opt-in C++ index-hang regression (#410, needs prod).
|
|
CBM_NO_CCACHE=1 Disable the content-verified compiler cache.
|
|
CBM_TEST_SHARD/_LEG Set by CI's sharded legs; leave unset locally.
|
|
CBM_TEST_SELECTION_DIR PR CI's test selection (smart CI): a directory with
|
|
test-only.txt (suite / suite:test lines) and
|
|
optional-suites.txt. Narrows the parallel suite step
|
|
only; contract steps and CBM_TEST_SEQUENTIAL=1 runs
|
|
keep running everything.
|
|
|
|
Examples:
|
|
scripts/test.sh # the full venue leg (what CI runs)
|
|
scripts/test.sh --suites daemon_ipc # one suite, incremental, seconds
|
|
scripts/test.sh --suites "arena hash_table" CC=clang CXX=clang++
|
|
scripts/test.sh SANITIZE= --suites daemon_ipc # plain build for trap debugging
|
|
EOF
|
|
}
|
|
|
|
# Parse --help / --suites / --tsan / --arch before sourcing env.sh.
|
|
# STRICT: an unknown flag or a stray word is an immediate usage error, never
|
|
# silently swallowed — agents must know exactly what a run will do.
|
|
SUITES=""
|
|
TSAN=0
|
|
CONTRACTS_ONLY=0
|
|
prev_arg=""
|
|
for arg in "$@"; do
|
|
case "$arg" in
|
|
-h|--help) usage; exit 0 ;;
|
|
--tsan) :;;
|
|
--contracts-only) :;;
|
|
--suites) :;; # next arg is the value, handled below
|
|
--suites=*) SUITES="${arg#--suites=}" ;;
|
|
--arch) :;; # next arg is the value, handled below
|
|
--arch=*) :;; # handled below
|
|
-*)
|
|
echo "test.sh: unknown option '$arg'. Please consult --help." >&2
|
|
exit 2
|
|
;;
|
|
arm64|x86_64)
|
|
if [[ "${prev_arg:-}" != "--arch" && "${prev_arg:-}" != "--suites" ]]; then
|
|
echo "test.sh: unexpected argument '$arg' (did you mean --arch $arg?). Please consult --help." >&2
|
|
exit 2
|
|
fi
|
|
;;
|
|
*=*) :;; # VAR=VAL make passthrough, validated below
|
|
*)
|
|
if [[ "${prev_arg:-}" != "--suites" ]]; then
|
|
echo "test.sh: unexpected argument '$arg'. Please consult --help." >&2
|
|
exit 2
|
|
fi
|
|
;;
|
|
esac
|
|
prev_arg="$arg"
|
|
done
|
|
for arg in "$@"; do
|
|
case "$arg" in
|
|
--tsan) TSAN=1 ;;
|
|
--contracts-only) CONTRACTS_ONLY=1 ;;
|
|
arm64|x86_64)
|
|
if [[ "${prev_arg2:-}" == "--arch" ]]; then
|
|
export CBM_ARCH="$arg"
|
|
fi
|
|
;;
|
|
*)
|
|
if [[ "${prev_arg2:-}" == "--suites" ]]; then
|
|
SUITES="$arg"
|
|
fi
|
|
;;
|
|
esac
|
|
prev_arg2="$arg"
|
|
done
|
|
# Normalize comma separation to the runner's space-separated argv form.
|
|
SUITES="${SUITES//,/ }"
|
|
case "${prev_arg:-}" in
|
|
--suites|--arch)
|
|
echo "test.sh: '$prev_arg' needs a value. Please consult --help." >&2
|
|
exit 2
|
|
;;
|
|
esac
|
|
if [ "$TSAN" -eq 1 ] && [ -n "$SUITES" ]; then
|
|
echo "test.sh: --tsan and --suites are separate modes (the TSan leg has its own suite set). Please consult --help." >&2
|
|
exit 2
|
|
fi
|
|
if [ "$CONTRACTS_ONLY" -eq 1 ] && { [ "$TSAN" -eq 1 ] || [ -n "$SUITES" ]; }; then
|
|
echo "test.sh: --contracts-only is its own mode (no build, no suites). Please consult --help." >&2
|
|
exit 2
|
|
fi
|
|
prev_arg=""
|
|
|
|
# Also support --arch=value
|
|
for arg in "$@"; do
|
|
case "$arg" in
|
|
--arch=*) export CBM_ARCH="${arg#--arch=}" ;;
|
|
esac
|
|
done
|
|
|
|
# shellcheck source=env.sh
|
|
source "$ROOT/scripts/env.sh"
|
|
# shellcheck source=path-safety.sh
|
|
source "$ROOT/scripts/path-safety.sh"
|
|
|
|
# Forward CC/CXX and collect make-passthrough args. BUILD_DIR is honored for
|
|
# the explicit target path below so containerized legs can build in their own
|
|
# directory instead of clobbering the host's native build/c artifacts.
|
|
# MAKE_ARGS is an ARRAY so a VAR=VAL whose value contains spaces (the
|
|
# windows-11-arm leg passes SANITIZE with four flags) survives as ONE make
|
|
# argument. The old string accumulation re-split it at every expansion and
|
|
# make swallowed the second flag's leading -f as its makefile option.
|
|
MAKE_ARGS=()
|
|
BUILD_DIR="build/c"
|
|
SANITIZE_GIVEN=0
|
|
SANITIZE_VALUE=""
|
|
prev_arg=""
|
|
for arg in "$@"; do
|
|
case "$arg" in
|
|
CC=*|CXX=*) export "${arg}" ;;
|
|
--arch|--arch=*) ;; # already handled
|
|
arm64|x86_64) ;; # already handled
|
|
--tsan|--contracts-only) ;; # already handled
|
|
--suites|--suites=*) ;; # already handled (value skipped via prev_arg below)
|
|
BUILD_DIR=*) BUILD_DIR="${arg#BUILD_DIR=}"; MAKE_ARGS+=("$arg") ;;
|
|
SANITIZE=*)
|
|
SANITIZE_GIVEN=1
|
|
SANITIZE_VALUE="${arg#SANITIZE=}"
|
|
MAKE_ARGS+=("$arg")
|
|
;;
|
|
*=*)
|
|
if [[ "${prev_arg:-}" != "--suites" ]]; then
|
|
MAKE_ARGS+=("$arg") # forward any VAR=VAL to make
|
|
fi
|
|
;;
|
|
esac
|
|
prev_arg="$arg"
|
|
done
|
|
|
|
# Platform default absorbed FROM CI (previously inline in _test.yml, so the
|
|
# local arm64 leg silently built without it — that divergence is why the SQLite
|
|
# page-cache misalignment was fatal only on the windows-11-arm runner): native
|
|
# ARM64 Windows has no ASan runtime, so its sanitizer gate is UBSan in trap
|
|
# mode + stack protector. Applied here, once, for every venue; an explicit
|
|
# SANITIZE=... (or SANITIZE=) argument overrides.
|
|
if [ "$SANITIZE_GIVEN" -eq 0 ] && [ "${MSYSTEM:-}" = "CLANGARM64" ]; then
|
|
MAKE_ARGS+=("SANITIZE=-fsanitize=undefined -fsanitize-trap=undefined -fstack-protector-strong -fno-omit-frame-pointer")
|
|
fi
|
|
|
|
# ASan stack-use-after-return runs in the test-diag lane only (_test.yml and
|
|
# `make -f Makefile.cbm diag` set detect_stack_use_after_return=1). gcc's
|
|
# libasan turns it ON by default, and on aarch64 gcc 13/14 never clear a freed
|
|
# fake frame's in-use flag: past 8,192 frames per size class every instrumented
|
|
# call scans that flag array and the check stops detecting anything (gcc arm64
|
|
# ran `extraction` in 688 s, amd64 in 16 s). Off here; an explicit setting wins.
|
|
case "${ASAN_OPTIONS:-}" in
|
|
*detect_stack_use_after_return*) ;;
|
|
*) ASAN_OPTIONS="${ASAN_OPTIONS:+$ASAN_OPTIONS:}detect_stack_use_after_return=0" ;;
|
|
esac
|
|
export ASAN_OPTIONS
|
|
|
|
EXPECTED_SANITIZED=1
|
|
if [ "$SANITIZE_GIVEN" -eq 1 ]; then
|
|
case "$SANITIZE_VALUE" in
|
|
*[![:space:]]*) EXPECTED_SANITIZED=1 ;;
|
|
*) EXPECTED_SANITIZED=0 ;;
|
|
esac
|
|
fi
|
|
|
|
# Refuse to run suites when the built runner disagrees with the lane that
|
|
# produced it. Exact output keeps missing, duplicate, or future unhandled
|
|
# fields fail-closed instead of silently weakening the gate.
|
|
assert_test_runner_build_config() {
|
|
local runner="$1"
|
|
local expected_sanitized="$2"
|
|
local expected="sanitized=$expected_sanitized test_seams=1"
|
|
local marker="__cbm_build_config_end__"
|
|
local captured=""
|
|
local actual=""
|
|
local probe_ok=0
|
|
|
|
if captured="$("$runner" --build-config && printf '%s' "$marker")"; then
|
|
probe_ok=1
|
|
actual="${captured%"$marker"}"
|
|
else
|
|
actual="$captured"
|
|
fi
|
|
if [ "$probe_ok" -ne 1 ] || [ "$actual" != "$expected"$'\n' ]; then
|
|
local reported="${actual//$'\n'/\\n}"
|
|
printf 'ERROR: build config mismatch for %s\n' "$runner" >&2
|
|
printf ' expected: %s\n' "$expected" >&2
|
|
printf ' reported: %s\n' "${reported:-<empty>}" >&2
|
|
printf '%s\n' 'Refusing to run suites; check sanitizer flags and CBM_SANITIZED_BUILD wiring.' >&2
|
|
return 1
|
|
fi
|
|
}
|
|
|
|
print_env "test.sh"
|
|
|
|
# ── TSan mode (--tsan): the data-race gate ──
|
|
# One entry for every venue: CI's tsan jobs and the compose test-tsan service
|
|
# both run this instead of carrying their own make invocation.
|
|
if [ "$TSAN" -eq 1 ]; then
|
|
echo "=== test.sh: TSan leg (make test-tsan) ==="
|
|
make -j"$NPROC" -f Makefile.cbm "$BUILD_DIR/test-runner-tsan" ${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}
|
|
assert_test_runner_build_config "$BUILD_DIR/test-runner-tsan" 1
|
|
make -f Makefile.cbm test-tsan ${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}
|
|
exit "$?"
|
|
fi
|
|
|
|
# ── Iteration mode (--suites): incremental rebuild + subset run ──
|
|
# The documented fast path: no clean, no contract steps, no prod-binary
|
|
# guards — those all still gate every merge through the default full run.
|
|
if [ -n "$SUITES" ]; then
|
|
echo "=== test.sh: ITERATION mode — suites: $SUITES (incremental build) ==="
|
|
make -j"$NPROC" -f Makefile.cbm "$BUILD_DIR/test-runner" ${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}
|
|
assert_test_runner_build_config "$BUILD_DIR/test-runner" "$EXPECTED_SANITIZED"
|
|
# shellcheck disable=SC2086 # suite list is deliberately word-split
|
|
"$BUILD_DIR/test-runner" $SUITES
|
|
exit "$?"
|
|
fi
|
|
|
|
# Step 0: fast build/security harness regressions run before the compiler-heavy
|
|
# suite. The Windows package surface is static here; native launcher behavior is
|
|
# exercised by scripts/test-windows.ps1.
|
|
echo "=== Step 0a: build directory safety contract ==="
|
|
bash "$ROOT/tests/test_build_dir_safety.sh"
|
|
|
|
echo "=== Step 0b: Windows VM worktree sync contract ==="
|
|
bash "$ROOT/tests/test_vm_worktree_manifest.sh"
|
|
|
|
echo "=== Step 0c: UI development proxy security contract ==="
|
|
bash "$ROOT/tests/test_ui_dev_proxy_security.sh"
|
|
|
|
echo "=== Step 0d: daemon soak recovery contract ==="
|
|
bash "$ROOT/tests/test_soak_daemon_recovery_contract.sh"
|
|
|
|
echo "=== Step 0d2: soak harness runtime isolation contract (#1696) ==="
|
|
bash "$ROOT/tests/test_soak_runtime_isolation_contract.sh"
|
|
|
|
echo "=== Step 0e: Windows launcher bundle contract ==="
|
|
bash "$ROOT/tests/test_windows_bundle_contract.sh"
|
|
|
|
echo "=== Step 0e2: activation refusal diagnostic contract ==="
|
|
bash "$ROOT/tests/test_activation_diagnostic_contract.sh"
|
|
|
|
echo "=== Step 0e3: security gate fail-closed contract ==="
|
|
bash "$ROOT/tests/test_security_gate_fail_closed.sh"
|
|
|
|
echo "=== Step 0f: tree-sitter runtime Makefile dependencies ==="
|
|
bash "$ROOT/tests/test_makefile_ts_runtime_dependencies.sh"
|
|
|
|
echo "=== Step 0g: security fuzz harness self-test ==="
|
|
bash "$ROOT/tests/test_security_fuzz_harness.sh"
|
|
|
|
echo "=== Step 0g2: memlab harness runtime isolation contract (#1696) ==="
|
|
bash "$ROOT/tests/test_memlab_runtime_isolation_contract.sh"
|
|
|
|
echo "=== Step 0h: smoke release-fixture contract ==="
|
|
bash "$ROOT/tests/test_smoke_fixture_contract.sh"
|
|
|
|
echo "=== Step 0i: parallel suite scheduler contract ==="
|
|
bash "$ROOT/tests/test_parallel_harness_contract.sh"
|
|
|
|
# Step 0i2: a red suite's summary must show the sanitizer report and the test
|
|
# that was running, not only "FAIL" lines and a shadow-byte legend. Drives the
|
|
# real summary functions with fabricated suite logs, so it runs everywhere.
|
|
echo "=== Step 0i2: suite failure-report contract ==="
|
|
bash "$ROOT/tests/test_failure_report_contract.sh"
|
|
|
|
echo "=== Step 0j: venue parity contract (one harness, every venue) ==="
|
|
bash "$ROOT/tests/test_venue_parity_contract.sh"
|
|
|
|
echo "=== Step 0k: spawn console-window contract (#1427) ==="
|
|
bash "$ROOT/tests/test_spawn_no_window_contract.sh"
|
|
|
|
echo "=== Step 0l: release archive extractor contract ==="
|
|
bash "$ROOT/tests/test_release_archive_extractor_contract.sh"
|
|
|
|
echo "=== Step 0m: VirusTotal release-notes + evidence contract ==="
|
|
bash "$ROOT/tests/test_vt_release_notes_contract.sh"
|
|
|
|
echo "=== Step 0n: VirusTotal gate policy contract ==="
|
|
bash "$ROOT/tests/test_vt_gate_policy_contract.sh"
|
|
|
|
echo "=== Step 0o: MCPB bundle contract (#1246) ==="
|
|
bash "$ROOT/tests/test_mcpb_bundle_contract.sh"
|
|
|
|
echo "=== Step 0p: MCPB registry entries contract (#1246) ==="
|
|
bash "$ROOT/tests/test_mcpb_registry_entries_contract.sh"
|
|
|
|
echo "=== Step 0q: release candidate derivation contract ==="
|
|
bash "$ROOT/tests/test_release_candidate_derivation_contract.sh"
|
|
|
|
echo "=== Step 0r: VirusTotal candidate-selection contract ==="
|
|
bash "$ROOT/tests/test_vt_candidate_selection_contract.sh"
|
|
|
|
echo "=== Step 0s: release gate-chain ordering contract ==="
|
|
bash "$ROOT/tests/test_release_gate_chain_contract.sh"
|
|
|
|
echo "=== Step 0t: test runtime isolation contract (#1691) ==="
|
|
bash "$ROOT/tests/test_runtime_isolation_contract.sh"
|
|
|
|
echo "=== Step 0t2: smoke harness runtime isolation contract (#1696) ==="
|
|
bash "$ROOT/tests/test_smoke_runtime_isolation_contract.sh"
|
|
|
|
echo "=== Step 0u: shell line-ending contract ==="
|
|
bash "$ROOT/tests/test_shell_line_endings.sh"
|
|
|
|
echo "=== Step 0v: nomic blob generator contract ==="
|
|
bash "$ROOT/tests/test_nomic_blob_generator_contract.sh"
|
|
|
|
echo "=== Step 0w: published language-count contract ==="
|
|
bash "$ROOT/tests/test_language_count_contract.sh"
|
|
|
|
echo "=== Step 0x: packaging version-metadata contract ==="
|
|
bash "$ROOT/tests/test_version_metadata_contract.sh"
|
|
|
|
# Step 0y: the Windows leg must not decide its verdict from an exit status that
|
|
# the ssh/msys2_shell chain can mangle — a channel that turns 0 into 1 can turn
|
|
# 1 into 0, and that direction reports a RED Windows leg as green. Runs
|
|
# everywhere (it drives synthetic logs, no VM needed) because the guard it pins
|
|
# is what every Windows verdict rests on.
|
|
echo "=== Step 0y: VM leg verdict contract ==="
|
|
bash "$ROOT/tests/test_vm_verdict_contract.sh"
|
|
|
|
echo "=== Step 0y2: isolated Windows venue contract ==="
|
|
bash "$ROOT/tests/test_vm_isolation_contract.sh"
|
|
|
|
echo "=== Step 0y3: setup scripts install through the installers' verified path ==="
|
|
bash "$ROOT/tests/test_setup_scripts_contract.sh"
|
|
|
|
# Step 0z: PR CI runs only the lanes scripts/ci/select-lanes.sh selects, so a
|
|
# wrong selection is a silent gate loss. The decision table, and the replay of
|
|
# every September 2026 PR push and real failure against it.
|
|
echo "=== Step 0z: PR lane selector contract ==="
|
|
bash "$ROOT/tests/test_select_lanes.sh"
|
|
echo "=== Step 0z2: lane selector history replay ==="
|
|
bash "$ROOT/scripts/test-impact/replay-selector.sh"
|
|
echo "=== Step 0z3: lane-aware aggregate gates (ci-ok, shard union) ==="
|
|
bash "$ROOT/tests/test_lane_gate_contract.sh"
|
|
echo "=== Step 0z4: lane wiring (PRs select, dry run and release run all) ==="
|
|
bash "$ROOT/tests/test_lane_wiring_contract.sh"
|
|
echo "=== Step 0z5: test-impact shadow prediction contract ==="
|
|
bash "$ROOT/tests/test_test_impact_predict.sh"
|
|
# Step 0z6: the per-test coverage map must never report a test as executing
|
|
# less than it did: every case where the builder cannot know (a killed forked
|
|
# child, an unreadable profile, a failed suite) has to come out `incomplete`.
|
|
echo "=== Step 0z6: per-test coverage map builder contract ==="
|
|
bash "$ROOT/tests/test_coverage_map.sh"
|
|
# Step 0z7: a push's incremental coverage map carries the suites it did not
|
|
# re-run onto the new image's function table; a wrong remap would hand the PR
|
|
# selection a map that claims tests ran functions they never touched.
|
|
echo "=== Step 0z7: incremental coverage map merge contract ==="
|
|
bash "$ROOT/tests/test_coverage_merge.sh"
|
|
# Step 0z8: the one place a test-impact answer becomes what PR CI runs. Only a
|
|
# well-formed, self-consistent selection may narrow; everything else runs all.
|
|
echo "=== Step 0z8: test selection (answer -> runner filter) contract ==="
|
|
bash "$ROOT/tests/test_test_impact_selection.sh"
|
|
|
|
if [ "$CONTRACTS_ONLY" -eq 1 ]; then
|
|
echo "=== test.sh: contracts-only — every contract step passed ==="
|
|
exit 0
|
|
fi
|
|
|
|
# Verify compiler supports target arch
|
|
verify_compiler "$CC"
|
|
|
|
# Step 1: Clean (scoped to this leg's build directory)
|
|
BUILD_DIR="$BUILD_DIR" scripts/clean.sh
|
|
|
|
# Step 2 + 3: Build, then run every suite as parallel processes (identical
|
|
# gate quality — see the ZERO-LOSS CONTRACT in scripts/run-tests-parallel.sh:
|
|
# the suite set is enumerated from the runner itself and union-guarded, and
|
|
# pass/fail/skip totals aggregate to the same numbers as the sequential run).
|
|
# CBM_TEST_SEQUENTIAL=1 restores the single-process runner.
|
|
make -j"$NPROC" -f Makefile.cbm "$BUILD_DIR/test-runner" ${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}
|
|
assert_test_runner_build_config "$BUILD_DIR/test-runner" "$EXPECTED_SANITIZED"
|
|
|
|
# Step 2b: per-test selection process regression. CBM_TEST_ONLY must run exactly
|
|
# the tests it names and fail on a token that matches nothing; a suite cannot
|
|
# assert that about the runner executing it. Runs against the runner just built
|
|
# and BEFORE the suites: it takes seconds, and a selection that silently drops
|
|
# tests is a property of the harness every later result depends on.
|
|
echo "=== Step 2b: per-test selection regression (CBM_TEST_ONLY) ==="
|
|
CBM_TEST_RUNNER="$ROOT/$BUILD_DIR/test-runner" bash "$ROOT/tests/test_harness_test_only.sh"
|
|
# Step 2c: PR CI's test selection (smart CI) through the parallel harness — the
|
|
# selected suites only, unknown suites and tests fail, conditional ones drop.
|
|
echo "=== Step 2c: parallel-harness test selection regression ==="
|
|
CBM_TEST_RUNNER="$ROOT/$BUILD_DIR/test-runner" bash "$ROOT/tests/test_harness_selection.sh"
|
|
|
|
# PR CI's test selection (smart CI, the select-tests job) narrows the parallel
|
|
# suite step and nothing else (scripts/run-tests-parallel.sh). A selection that
|
|
# was promised but is missing is a plumbing fault: fail, never run silently.
|
|
SELECTION_ENV=()
|
|
if [ -n "${CBM_TEST_SELECTION_DIR:-}" ]; then
|
|
selection_dir="$CBM_TEST_SELECTION_DIR"
|
|
if command -v cygpath > /dev/null 2>&1; then
|
|
selection_dir="$(cygpath -u "$selection_dir")"
|
|
fi
|
|
if [ ! -s "$selection_dir/test-only.txt" ]; then
|
|
echo "FAIL: CBM_TEST_SELECTION_DIR=$CBM_TEST_SELECTION_DIR holds no test-only.txt" >&2
|
|
exit 1
|
|
fi
|
|
SELECTION_ENV=(CBM_TEST_SELECTION_FILE="$selection_dir/test-only.txt")
|
|
if [ -f "$selection_dir/optional-suites.txt" ]; then
|
|
SELECTION_ENV+=(CBM_TEST_SELECTION_OPTIONAL="$selection_dir/optional-suites.txt")
|
|
fi
|
|
if [ "${CBM_TEST_SEQUENTIAL:-0}" = "1" ]; then
|
|
echo "test.sh: CBM_TEST_SEQUENTIAL=1 runs every test; the selection narrows the parallel harness only"
|
|
fi
|
|
fi
|
|
|
|
if [ "${CBM_TEST_SEQUENTIAL:-0}" = "1" ]; then
|
|
make -f Makefile.cbm test ${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}
|
|
else
|
|
env ${SELECTION_ENV[@]+"${SELECTION_ENV[@]}"} \
|
|
make -f Makefile.cbm test-par ${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}
|
|
fi
|
|
|
|
# Step 3a: the runner must ignore an inherited git repository environment
|
|
# (#2003: the pre-commit hook exports GIT_DIR, which overrides `git -C` and
|
|
# pointed fixture git commands at the committer's real repository).
|
|
echo "=== Step 3a: inherited git environment isolation (#2003) ==="
|
|
bash "$ROOT/tests/test_git_env_isolation_contract.sh" "$ROOT/$BUILD_DIR/test-runner"
|
|
|
|
# Step 4: C++ large-TU index-hang regression guard (#410). Runs the PROD binary
|
|
# in a subprocess with a wall-clock timeout — a hang must fail, not block the run.
|
|
# Opt-in via CBM_RUN_HANG_TEST=1 (it needs the prod binary, which the ASan unit
|
|
# run above does not build). Skipped by default so the fast unit run stays fast.
|
|
if [ "${CBM_RUN_HANG_TEST:-0}" = "1" ]; then
|
|
echo "=== Step 4: C++ index-hang regression (#410) ==="
|
|
bash "$ROOT/tests/test_cpp_index_hang.sh"
|
|
fi
|
|
|
|
# Step 5: Parent-death watchdog regression (#406/#407). Builds the prod stdio
|
|
# binary and verifies it self-exits when its launching parent is killed.
|
|
#
|
|
# TEST_SEAMS=1: the worker-mode leg below needs the crash-orphan probe, which is
|
|
# compiled out of ordinary builds (it forks a SIGTERM-ignoring child — see
|
|
# src/main.c). Requesting it HERE, in the leg that consumes it, is what keeps
|
|
# release artifacts free of it; scripts/ci/check-binary-composition.sh proves
|
|
# they stay that way.
|
|
echo "=== Step 5: parent-death watchdog regression (#406/#407) ==="
|
|
make -j"$NPROC" -f Makefile.cbm cbm TEST_SEAMS=1 ${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}
|
|
WATCHDOG_BINARY="$ROOT/$BUILD_DIR/codebase-memory-mcp"
|
|
CBM_TEST_BINARY="$WATCHDOG_BINARY" bash "$ROOT/tests/test_parent_watchdog.sh"
|
|
|
|
# Step 5a: that watchdog is also the SMALLEST-stack thread in the image, which
|
|
# makes it the first casualty when static TLS grows — glibc takes the TLS block
|
|
# out of each thread's own stack allocation. Checked here, against the binary
|
|
# Step 5 just built, because the failure it prevents surfaces nowhere near its
|
|
# cause (PR #2233: a thread-local cache in an extraction file stopped the index
|
|
# worker from starting, on x86-64 only).
|
|
echo "=== Step 5a: static-TLS budget against the smallest thread stack ==="
|
|
bash "$ROOT/tests/test_thread_stack_tls_contract.sh" "$WATCHDOG_BINARY"
|
|
|
|
# Step 5b: worker-mode parent-death watchdog (#845). A supervised index worker
|
|
# (`cli --index-worker …`) whose supervisor dies must self-exit instead of
|
|
# indexing on as an orphan. Reuses the prod binary built in Step 5.
|
|
echo "=== Step 5b: worker-mode watchdog regression (#845) ==="
|
|
CBM_TEST_BINARY="$WATCHDOG_BINARY" bash "$ROOT/tests/test_worker_watchdog.sh"
|
|
|
|
# Step 5c: a worker-delivered MCP error is transport success. The outer CLI
|
|
# still exits nonzero for the user-facing tool error, but the supervisor must
|
|
# preserve that response instead of misreporting exit_nonzero as a file crash.
|
|
echo "=== Step 5c: worker error-response transport regression ==="
|
|
CBM_TEST_BINARY="$WATCHDOG_BINARY" bash "$ROOT/tests/test_worker_error_response.sh"
|
|
|
|
# Step 5d (#1388) is DELIBERATELY NOT GATING HERE — see
|
|
# tests/test_hook_conflict_notice.sh for the full what-was-tried record.
|
|
# Summary: the test forces a client/daemon build mismatch via the
|
|
# CBM_TEST_HOOK_CLIENT_BUILD seam and asserts the stdout systemMessage. It is
|
|
# reliably green locally against a seam-bearing binary, but on every CI leg the
|
|
# forced mismatch raises no cohort conflict at all: the seam is present (the
|
|
# test asserts that up front), the forced fingerprint is well-formed (64 hex),
|
|
# and `daemon status` reports an active daemon on a DIFFERENT build - yet the
|
|
# client joins silently. Until that local-vs-CI divergence in the cohort
|
|
# admission path is understood, gating on it would make an unexplained red, and
|
|
# skipping it silently would hide the gap. Run it by hand:
|
|
# make -f Makefile.cbm cbm TEST_SEAMS=1 && bash tests/test_hook_conflict_notice.sh
|
|
|
|
# Step 5e: watcher_enabled kill-switch process regression (#335). Reuses the
|
|
# prod binary built in Step 5; drives a real daemon against an isolated cache
|
|
# and proves watcher_enabled=false stops the watcher from being built, started
|
|
# or registered, while auto_index and manual index_repository keep working.
|
|
# Every wait is a bounded poll on an asserted state (a closed daemon lifecycle),
|
|
# never a fixed sleep — see the header of the test for why.
|
|
echo "=== Step 5e: watcher_enabled kill-switch regression (#335) ==="
|
|
CBM_TEST_BINARY="$WATCHDOG_BINARY" bash "$ROOT/tests/test_watcher_disabled.sh"
|
|
|
|
# Step 5f: a supervised worker is scoped to the request the daemon admitted,
|
|
# never to the CBM_ALLOWED_ROOT it inherited from the daemon starter's
|
|
# environment. Reuses the prod binary built in Step 5.
|
|
echo "=== Step 5f: worker request-scope regression ==="
|
|
CBM_TEST_BINARY="$WATCHDOG_BINARY" bash "$ROOT/tests/test_worker_session_scope.sh"
|
|
|
|
# Step 6: security-strings URL allow-list regression. The MSYS2 CLANG64 toolchain
|
|
# bakes its package-tracker URL into the static Windows .exe; the binary string
|
|
# audit must allow-list it (Windows-only — Linux smoke never saw it).
|
|
echo "=== Step 6: security-strings allow-list regression ==="
|
|
bash "$ROOT/tests/test_security_strings_allowlist.sh"
|
|
bash "$ROOT/tests/test_destructive_ordering_contract.sh"
|
|
|
|
echo "=== All tests passed ==="
|