mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-10-20 05:49:37 +08:00
Martin Vogel
e66043bc2b
A Windows sanitizer job went red on one suite that aborted under AddressSanitizer, and the job log showed an empty "every failure site" section followed by the shadow-byte legend. The summary in scripts/run-tests-parallel.sh searched the suite log for "FAIL" only. A sanitizer report never contains that word, and its useful part (error kind, access size, top frames) is at the start of the report, far above the last 15 lines. The summary now lives in scripts/suite-failure-report.sh, which the harness sources. Between the unchanged failure-site and last-15-lines sections it prints, for a failing suite, the test that was running and the first sanitizer report. The framework flushes each test name before the test body runs, so the last test line ahead of a report is the test that was running; a report that comes after the last test finished is labelled as such instead of blaming a test. The report is shown from its header (AddressSanitizer, LeakSanitizer, ThreadSanitizer, MemorySanitizer, UndefinedBehaviorSanitizer) to its own SUMMARY line, at most 40 lines, without the shadow dump and the legend; a recoverable "runtime error:" line is shown with its stack only. Further sanitizer headers, "runtime error:" lines and SUMMARY lines follow (at most 10), then signal and timeout lines (at most 5). A suite that dies with no report and no completion summary gets the running test named. A trap-mode UBSan build stops on an illegal instruction and prints nothing, so that name is the only evidence its log holds. A log with an assertion failure and no report prints exactly what it printed before. The same file keeps the logs a red run needs. An EXIT trap in the harness copies the log of every suite that did not record rc=0, plus results.txt, into test-logs/failed/. The trap covers the scheduler giving up and the union guard as well as the normal end, and it never changes the exit status. scripts/run-test-wave.py reported pass=0 for a suite that died without its summary line. In that case, and only then, it now counts the PASS markers in the log. A suite with a summary line is never recounted, so the totals of a green run are unchanged. tests/test_failure_report_contract.sh drives the real functions with fabricated suite logs and runs as Step 0i2 of scripts/test.sh. It fails on the previous behaviour in every sanitizer, running-test and result-line check and passes on it in the unchanged-output checks. The awk code uses no interval expressions and no character classes; the test passes with BSD awk and grep on macOS and with mawk and GNU grep on Ubuntu. scripts/README.md names the new internal file. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
5.4 KiB
5.4 KiB
scripts/ — the canonical leg entries
The doctrine (enforced, not advisory): a venue may provision a machine;
only a canonical leg script may exercise the product. Local CI, PR CI, the
dry run and the release all call the SAME files below — venues differ only in
host specs, architecture and inputs, never in what a leg does. Platform
differences (arm64 sanitizer flags, native Windows behavior, the Linux portable
binary) live INSIDE these scripts, once, never per-venue.
tests/test_venue_parity_contract.sh (run as Step 0j of every test leg) turns
any violation — inline harness logic in a workflow, a bypassed wrapper, a
missing --help — into a red build.
Every entry answers --help (authoritative, agent-oriented: modes, env, exit
codes) and rejects unknown flags with exit 2 + Please consult --help.
The legs
| leg | entry | what a run gives you |
|---|---|---|
| test | test.sh |
DEFAULT = the venue leg: static contracts (Step 0a–0r) + CLEAN ASan+UBSan build + all suites via the parallel harness + prod-binary guards. --suites a,b = iteration mode (incremental rebuild, subset, seconds). --tsan = the ThreadSanitizer leg. --contracts-only = the static contract steps alone (PR CI's contracts job for docs-only and non-product changes). CLANGARM64 gets CI's trap-UBSan flags automatically. |
| package wrappers | ci/test-package-wrappers.sh |
Go, npm, and PyPI runtime-set publication/lock suites on the current host. CI runs the same entry on Linux and Windows so both platform lock implementations gate. |
| build | build.sh |
CLEAN production runtime set (native executable + authenticated integration asset; --with-ui adds one content-addressed UI pack). ccache via env.sh makes repeats fast; CCACHE_COMPILERCHECK=content guarantees a hit is byte-identical to a cold compile — never stale. --version, STATIC=1, BUILD_DIR=. |
| lint | lint.sh |
clang-tidy + cppcheck + clang-format (+ no-skips policy). --ci = the CI gate set (no clang-tidy). Drives the same make targets as make lint/lint-ci. |
| smoke (unix) | smoke-local.sh |
Stages a full release fixture, serves it on a kernel-assigned port, runs smoke-test.sh (ALL phases incl. download/install/update E2E) inside a disposable HOME/XDG/TMP sandbox. ui variant makes a missing verified UI pack a FAILURE. CBM_SMOKE_ARTIFACT_DIR = smoke an extracted release artifact verbatim (release mode). |
| smoke (windows) | ../test-infrastructure/vm/vm-smoke.sh |
Same verified runtime-set contract on the real Windows VM, plus the user-PATH registry guard (prepare/verify/cleanup). |
| smoke-invariants | smoke-invariants.sh |
Production-path resilience battery (MCP handshake, all tools invocable, malformed-input handling, supervised crash/hang recovery) — no fixture server or install E2E. smoke.yml runs an explicitly seam-enabled build on the WIDEST source matrix; release artifacts remain seam-free and use the release-shaped smoke legs above. |
| soak | soak-legs.sh |
The release-gating soak SEQUENCE: quick then query-leak (the #581 detector — never reindexes, so RSS growth = query-path leak), each guarded by a completion-summary check. --legs quick for the ASan single-leg variant. Duration is per leg. |
Internal harnesses — never called directly by a venue (the contract forbids
it): smoke-test.sh (phases; wrappers provide fixture server + sandbox),
soak-test.sh (one soak run; soak-legs.sh provides the sequence + guards),
run-tests-parallel.sh (reached through test.sh), which sources
suite-failure-report.sh for its end-of-run failure summary (failure sites,
sanitizer report, running test, last lines) and for test-logs/failed/, the
logs of every suite that did not end green.
Conventions
- Exit codes: 0 = pass · 2 = usage error · 90 = guard (a run died without its completion summary — never counts as green) · anything else = the leg's real failure.
- Iteration is a flag, not a side-tool: the fast paths (
--suites,--legs) are modes of the SAME entry the gates run, so a dev loop can never drift from the venue behaviour. - Env sandboxing: the smoke wrappers neutralize every agent-config destination override; a smoke can never scribble on your real config.
Recommended workflows
- Iterating on a change:
scripts/test.sh --suites <suite>(seconds, incremental, same ASan+UBSan flags as the gate). List suites:build/c/test-runner --list-suites. Debugging a Windows-on-ARM trap: re-run withSANITIZE=for a plain build, or use the emulatedwin.sh ubsan-*pair for full diagnostics. - Before any push (the 3-OS ladder):
scripts/test.sh(macOS, full) →./test-infrastructure/run.sh full(Linux + TSan + smoke) →test-infrastructure/vm/win.sh test-par+guards+smoke-install(+soakwhen the change touches memory/daemon paths). - Concurrency-touching change: add
scripts/test.sh --tsanearly — the same leg CI gates on. - Release-shaped verification:
CBM_SMOKE_ARTIFACT_DIR=<extracted artifact> scripts/smoke-local.sh <binary> [ui]smokes exactly what would ship. - A leg is red in CI but green locally: first suspect environment shape,
not code — the preflights (
win.shautomatic;scripts/ci/preflight-docker.sh) andtest-infrastructure/README.md's residuals list cover the knowable differences.
See scripts/ci/README.md for the CI plumbing and
test-infrastructure/README.md for the venue map.