mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-10-26 06:43:15 +08:00
Martin Vogel
f404dfbc7a
The test-msan lane dies whenever the buildx layer cache misses and the
builder's resolver hiccups. PR-CI runs 33816056229 and 33818108315 both
failed in "Build MSan image (cached layers)" with:
#7 26.77 Could not resolve 'apt.llvm.org'
#7 27.34 W: Failed to fetch http://apt.llvm.org/noble/dists/\
llvm-toolchain-noble-22/InRelease Could not resolve 'apt.llvm.org'
ERROR: failed to build: failed to solve: process "/bin/sh -c apt-get \
update && ..." did not complete successfully
The wget of the signing key from that SAME host had succeeded seconds
earlier in the same layer, so this is a transient DNS failure inside
buildkit, not a wrong URL or a dead mirror. 2 of the last 15 PR runs.
Every network-touching step in Dockerfile.msan now runs under a small
POSIX-sh helper written into the image before the first layer that needs
it: /usr/local/bin/retry makes 5 attempts with growing backoff
(5s/10s/20s/40s) and still exits non-zero once they are exhausted, so a
genuine breakage keeps failing the build instead of being swallowed.
Wrapped: both apt-get update/install pairs (also given
-o Acquire::Retries=3), the wget of the LLVM signing key, and the
llvm-project and zlib clones. Two shapes needed care:
- the key is fetched with `wget -O <file>` instead of `-qO- > <file>`,
because a redirect is opened once for all attempts and a partial write
from a failed attempt would be prepended to a later successful one;
- each clone clears its destination first, or a half-finished clone makes
every later attempt fail on "destination path already exists".
The pinned base digest, the pinned llvmorg-22.1.0 / v1.3.1 tags,
--no-install-recommends and the layer structure are unchanged; on a
healthy network the image is byte-for-byte the same work as before.
Verified on Colima:
- `docker buildx build --check -f test-infrastructure/Dockerfile.msan
test-infrastructure/` -> "Check complete, no warnings found."
- scratch image = FROM + helper + the real apt.llvm.org layer, built
--no-cache: layer completes, clang-22/llvm-22 installed.
- scratch image where the wrapped command fails twice then succeeds:
attempts 1 and 2 logged with 5s/10s backoff, attempt 3 succeeds,
build exit 0.
- scratch image where the wrapped command always fails: 5 attempts,
"retry: giving up after 5 attempts", retry_exit=1, build exit 1.
The full MSan image was deliberately not built locally (it compiles
libc++ with MSan); CI builds it on the branch.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
test-infrastructure/ — the local venues
The local half of the 3-OS CI ladder. Every leg here runs the SAME canonical
scripts CI runs (scripts/test.sh, build.sh, smoke-local.sh,
soak-legs.sh, vm-smoke.sh — see scripts/README.md); this directory only
provisions containers and the Windows VM. The venue-parity contract
(tests/test_venue_parity_contract.sh, Step 0j of every test leg) fails the
build if a venue grows its own harness logic — here or in the workflows.
Entry points
| venue | entry | notes |
|---|---|---|
| Linux (arm64 + amd64) + cross-compile | ./run.sh <leg> (--help for the leg list) |
Colima only — Docker Desktop is broken on this machine. A clean-disk preflight (scripts/ci/preflight-docker.sh) runs before every leg: GitHub runners are ephemeral with a known-free 14 GB disk, so the long-lived VM is swept back to that shape first. |
| Real Windows (UTM ARM64 VM) | vm/win.sh <command> (win.sh help) |
Same preflight idea (scripts/ci/clean-test-residue.ps1), plus CI's protected per-user TEMP root. Wine (run.sh windows) is a compile check only — never a substitute for the VM. |
| macOS | run the canonical scripts natively | scripts/test.sh, scripts/build.sh, scripts/smoke-local.sh … |
The ladder (before any push)
scripts/test.sh— macOS native (or--suites …while iterating)./test-infrastructure/run.sh full— Linux arm64 test/build/TSan/smoke/portable + mingw cross-compile (+amd64,soak-linuxas needed)vm/win.sh test-par·guards·smoke-install·soak— the real-Windows legs
Infra unavailable = a run blocker to escalate — never a silent bypass.
Fidelity guarantees (what makes local predictive of CI)
- Same scripts, same sequence, same flags — platform specifics (e.g. the CLANGARM64 trap-UBSan default) live inside the canonical entries, applied identically everywhere.
- Same environment shape — protected TEMP roots, sandboxed smoke, clean builds with the content-verified ccache (a hit is byte-identical to a cold compile; caches only accelerate, never change results).
- Same starting disk — preflights sweep residue and BLOCK below the runner's 14 GB floor (a quietly full disk fails inside install paths and masquerades as a product bug).
- Known, deliberate residuals — runner physics (4 vCPU vs local cores, shared tenancy), Defender ON in the VM vs OFF on GitHub runners, no TSan on Windows anywhere, no ASan runtime on native ARM64 Windows (trap-UBSan + PageHeap stand in).
See vm/README.md for VM provisioning and day-to-day VM mechanics.