MSan: vendored zstd fails to compile. Its MSan-only block (guarded by
MEMORY_SANITIZER) declares __msan_test_shadow returning intptr_t and
reaches for the type with
#define ZSTD_DEPS_NEED_STDINT
#include "zstd_deps.h"
but the amalgamator that produced zstd.c collapsed that second include
into a "skipping file" comment, so the define pulls nothing in and
intptr_t is undeclared. Only this lane compiles that block at all, and
only where <stddef.h> does not drag stdint.h in transitively -- which is
why it built on the local aarch64 container and failed on CI's x86-64.
The lane now forces the header. Patching the vendored amalgamation would
be silently undone by the next re-vendor.
diag: detect_invalid_pointer_pairs comes back out. It fires during static
initialisation inside vendored simplecpp -- a std::string global at
simplecpp.cpp:101 -- with a second "pointer" of 0xfffffffffffffff3, a
sentinel rather than an address: libstdc++ string internals, not
anything this codebase wrote. It is a process-wide runtime flag with no
per-file scoping, so unlike the analyzer's path filter it cannot be
aimed away from vendored code. Keeping it would mean a permanently red
lane reporting a non-defect, which is how a lane gets ignored. The
instrumentation it needed comes out with it.
The other three off-by-default checks stay: stack-use-after-return,
stack-use-after-scope, strict-string-checks. Those are the ones covering
bug classes nothing else in the matrix looks for, and none of them
fired.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The MSan and memory-analyzer jobs drove docker and make directly from
workflow steps, and the diag step used folded `run: >`. All four are
venue-parity violations: a venue may provision, plumb artifacts, or call
a canonical leg script, and nothing else. Anything that actually
exercises the product belongs inside a scripts/ entry so that every
venue runs the same code instead of each workflow growing its own
slightly different invocation.
So the docker work moves to scripts/ci/msan-lane.sh (build | run | all)
and the analyzer gate to scripts/ci/lint-mem.sh, both of which the local
paths already reach through run.sh and the Makefile. The folded step
becomes `run: |`.
Found by the contract itself, running as step 0j of the local Linux leg.
It had never run against these jobs, because they were added and then
exercised only through GitHub CI -- which is exactly the gap the local
ladder exists to close, and the reason the contract runs as step zero of
every leg rather than as a job of its own.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Every lane here was either asserting coverage it did not have, or was
reachable only by bypassing the entry point that is supposed to define
the ladder.
TSan: no exclusions any more. The three suites the Makefile documented
as excluded are back. daemon_ipc and daemon_frontend no longer reproduce
the harness race and thread leaks they were excluded for. daemon_runtime
did not deadlock as the comment claimed -- it reported a real production
data race on the log sink, fixed separately. Excluding a suite from a
sanitizer lane hides exactly the class of bug the lane exists to find,
so the comment block now records what was actually true rather than what
was assumed.
TSAN_OPTIONS gains report_thread_leaks=0. This disables the thread-
HYGIENE check only; race detection is untouched. Several daemon fixtures
fork after the process has gone multi-threaded, and in the forked child
TSan sees the parent's already-finished threads as never-joined even
where the fixture joins them. It fires on macOS and not Linux, i.e. it
tracks fork semantics rather than anything about this code. The
alternative was dropping whole suites, which costs real race coverage;
this costs none.
UBSan: tre.o no longer builds with -fno-sanitize=alignment. Alignment
was switched off for a vendored regex engine that ships in the product,
which is where the check is least redundant, not most.
LSan on macOS: new test-lsan target and test-lsan-macos CI leg. LSan is
on by default under ASan on Linux, so the Linux legs have always had
leak coverage. On macOS it is off by default and Apple's clang refuses
to enable it outright, so that platform had none at all. Apple's refusal
is not a darwin limitation -- upstream LLVM supports LSan on darwin/
arm64. The lane is the ordinary ASan suite built with Homebrew LLVM and
run with detect_leaks=1; it runs the full suite clean and was checked to
still catch a deliberately leaked allocation.
MSan: reachable from the local ladder. The image and compose service
existed but run.sh had no leg, so the only way in was to drive docker
compose by hand -- which means it was not part of the ladder in any
meaningful sense. The image also moves to clang 22, matching the diag
and analyzer lanes instead of sitting four majors behind on noble's
default. The leg documents the aarch64 shadow-mapping failure so a local
arm64 stack overflow in the grammar suites is not mistaken for a code
defect; the GitHub leg runs x86-64, which is the mapping that matters.
Off-by-default ASan checks: the diagnostic lane, and its CI twin, now run
detect_stack_use_after_return, detect_stack_use_after_scope,
detect_invalid_pointer_pairs (with the -fsanitize=pointer-compare,
pointer-subtract instrumentation it requires) and strict_string_checks.
Running ASan is not the same as running all of it, and these four cover
bug classes nothing in the matrix was looking for. They stay on the
diagnostic lane rather than the gating ones until they have a clean
history there; promoting them is a separate deliberate step.
Verified: macOS TSan 940 passed / 3 skipped / 0 races over the full
suite set; the macOS leak lane 7375 passed / 4 skipped / 0 leaks, with
LeakSanitizer confirmed armed under that exact toolchain and option set
by checking it still reports a deliberately leaked allocation.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two corrections.
First, removing the CI job while keeping the compose service and the local
bindings left the ladder carrying a lane CI did not have — the venue
asymmetry the unification work exists to prevent. The lane now lives in both
venues again.
Second, and the reason the removal was wrong: the evidence behind it was
entirely from the LOCAL arm64 container, while the job that got deleted would
have run on x86-64. MSan's shadow and stack handling are materially better
supported on x86-64, so the thread-stack overflows that drove the exclusions
may well be architectural. I never tested the architecture CI uses, and the
local ladder cannot emulate it faithfully — so 'it cannot run in CI' was an
inference from the wrong platform stated as a fact.
The CI leg therefore runs with MSAN_EXCLUDE="" — no exclusions — so the
first run settles the question with real evidence on the real architecture.
The script's exclusion list stays as the LOCAL default, documented as such.
An accepted local/CI divergence for this lane specifically.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Running the lane to completion does not currently work: with three suites
already excluded for thread-stack overflow it now dies with a plain SIGSEGV
elsewhere. Wiring an auto-running job that cannot finish would be exactly the
structurally-red lane O10 forbids, so the test-msan CI job is removed and the
lane is documented as exploratory and local-only.
What it IS worth, and why the infrastructure stays: every suite it does run
is clean under MSan, including the C++ preprocessing path that justified
building the instrumented-libc++ image in the first place, and it correctly
identified one convincing-looking report as a mixed-build artifact rather
than a bug. The image, the script, the compose service, the MSAN_ORIGINS and
CBM_THREAD_STACK_MB knobs, and the full record of what was tried all remain,
so picking this up is a continuation rather than a restart.
The blocker is one problem, stated at the exclusion site: threads whose
stacks are sized outside cbm_thread_create overflow under instrumentation,
and the failure follows cumulative process state rather than any single
suite's depth — which points at per-suite process sharding or finding that
thread creator as the fix.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Stage 2 of the memory-diagnostics program (user decision: go directly to the
instrumented image rather than a C-only probe). MSan detects uninitialized
READS, the one memory-error class no other lane covers dynamically, and it
requires every linked library to be instrumented -- vendored C deps compile
in-tree and instrument for free; the two external links do not:
- test-infrastructure/Dockerfile.msan: pinned-base image building
libc++/libc++abi/libunwind (llvmorg-18.1.8, LLVM_USE_SANITIZER=
MemoryWithOrigins) and static zlib v1.3.1 into /opt/msan, with the
symbolizer and MSan runtime in a separate last layer so tool additions
never invalidate the ~30-min libc++ build.
- scripts/msan.sh: the canonical lane entry. ALWAYS clean-builds its
BUILD_DIR: make does not encode flags into dependencies, and a stage-1
probe's libstdc++ objects surviving into the libc++ lane produced a
convincing-looking uninitialized-value report at preprocessor.cpp:168 --
the uninstrumented .so string constructor wrote the temporary, the
instrumented move constructor read it. The clean rebuild proved it an
artifact: extraction (incl. the C++ preprocessing path) runs 272/272 with
zero reports.
- Makefile.cbm: CXX_STDLIB / CXX_STDLIB_FLAGS hooks so the lane can swap
libstdc++ for the instrumented libc++ (defaults identical; the shipping
build is byte-for-byte unaffected).
- docker-compose test-msan service: same aarch64 seccomp/setarch remedy as
the TSan service (MSan's shadow layout hits the same personality() block).
- CI test-msan job (_test.yml): buildx local-cache via the repo's existing
pinned actions/cache -- no new third-party action pins; a warm run skips
the libc++ build entirely.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The memory-diagnostics report's priority-4 lane (path-sensitive clang-analyzer,
memory checks only) run over all 111 production files. 21 findings triaged;
the real ones, all cold-path (none can explain #581's per-query residual):
LEAKS
- mcp get_architecture: scope_path leaked on the missing-store early return
(REQUIRE_STORE frees only `project`); allocate after the gate.
- pass_definitions: cancellation mid-extraction leaked the pass-owned result
cache including already-extracted entries; mirror the end-of-pass cleanup.
- store package-boundary scan: the row-scan abort path freed the node arrays
but not the boundary accumulators or their duplicated package strings.
- cbm quarantine set: a duplicate path line leaked the replaced value (and a
fresh key copy -- the table borrows key pointers); a partial strdup failure
leaked the surviving half. Reuse the stored key for duplicates.
- pass_githistory: unchecked malloc/strdup -- an OOM dereferenced NULL and a
failed strdup leaked the index cell. Allocate before claiming the slot.
NULL/UB
- cli config subcommand: NULL argv with nonzero argc slipped the guard (the
inner `argv &&` shielded only the help comparison) into argv[0].
- store bfs_multi: a negative max_results broke out before any row was
written, then freed fields of an unwritten negative-index slot. Clamp.
- pass_calls emit_http_async_edge: the service-pattern call sites pass a NULL
target behind a hand-duplicated URL predicate; a drift between the copies
turned target->id into a null deref. The callee is now total.
- sqlite_writer: both leaf-array OOM paths left leaf_count stale with a NULL
array, walking pb_finalize_* into leaves[0]; consistent empty state routes
them to the existing root=0 failure return.
HARDENED (invariants true but invisible to path-sensitive analysis)
- Leiden CSR + aggregate arrays, SCC adjacency: calloc + endpoint guards, so
a future degree/collection miscount degrades benignly instead of UB.
- SCC cycle fill: the ncyc==0 no-slot invariant made local.
RECORDED FALSE POSITIVES (no code change)
- yaml sequence starts (loop bound == alloc bound), cypher agg arrays (same
count both sides), mcp read_message ch (assigned by fgetc each iteration),
pkgmap clean buffer, mcp csize (Tarjan: ncomp>=1 when nverts>=1), vendored
verstable x2.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
GATE + LANES (user decision: runner cost accepted)
- make lint-mem (local triage) and lint-mem-ci (gating: vendored-filtered,
any remaining finding fails). The gate is green because every false
positive above was restructured for provability -- calloc'd fill-cursor
arrays, explicit Tarjan invariant, zeroed buffer tails, min-1-element
allocations -- never suppressed.
- make diag: pinned newest-LLVM ASan/UBSan lane with straighter stacks.
- CI: lint-mem job (_lint.yml) and test-diag job (_test.yml), both on the
pinned LLVM 22 apt toolchain. Cost disclosure: roughly +25-40 min and
+25-60 min (ccache-warm) per push respectively.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The qa/** push trigger auto-ran the board from every qa branch, and the board
is structurally red on GitHub runners: its real-repo corpus tier hardcodes a
local path no runner can satisfy, so every auto-run failed on the skip gate
regardless of the code under test -- red that blocks nobody trains everyone
to ignore red. The board's primary venue is the local multi-leg CI, which can
hold the corpus; workflow_dispatch remains for cross-platform spot checks.
Scope: removes the push trigger only. The workflow was never a required check
(non-gating by its own declaration), so no gating change on any branch.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Run 30504814537 built all eight platforms successfully with skip_tests=true, and
reported:
test: skipped (intended)
smoke: skipped (NOT intended)
soak: skipped (NOT intended)
release-draft: running
GitHub propagates "skipped" TRANSITIVELY down the needs graph. `build` carried
`!cancelled() && !failure()` so it overrode the skipped `test` and ran; `smoke`
had no `if:` and `soak` only checked soak_level, so both inherited the skip.
Nothing failed. Nothing announced it. The run then proceeded toward publishing
binaries that had never been smoke-tested or soaked -- the exact opposite of what
skip_tests was for, which was to skip a phase already proven green, not to
quietly stop testing the artifacts.
The second half of the defect is worse than the first: `release-draft` guarded
itself with a bare `!cancelled() && !failure()`, which is fail-OPEN. A skipped
job is neither cancelled nor failed, so a skipped smoke sails straight through
it. The draft was one gate away from being cut from unverified binaries, and the
only reason it was not is that the run was cancelled by hand.
Fixes, both directions:
- smoke and soak get `!cancelled() && !failure()`, so a deliberately skipped
ancestor no longer skips them;
- release-draft now requires `needs.smoke.result == 'success'` and enumerates
`needs.soak.result` explicitly, so a SKIPPED smoke BLOCKS the draft, and a
legitimately skipped soak (soak_level=none) stays distinguishable from a soak
that never ran.
tests/test_release_gate_chain_contract.sh pins both properties: every job
downstream of the optional phase must carry the override, and the draft must
demand smoke success rather than smoke non-failure. Verified in both directions
-- it passes on this workflow and fails with 2 violations when the previous
smoke/soak conditions are restored.
Nothing was published: the release remained a draft (isDraft=true,
publishedAt=null) and the run is cancelled.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Completes the previous commit, which carried only the two mode changes because
the call-site edits were not staged when it landed (--amend is denied in this
repo, so this is additive rather than a rewrite).
Call sites now go through `bash`, which is what most of this repo already does and
which cannot break if a mode bit is lost to a patch application or a non-POSIX
checkout:
scripts/package-release.sh -> bash scripts/ci/check-binary-composition.sh
.github/workflows/release.yml -> bash scripts/ci/append-vt-notes.sh
With the 100755 modes from the previous commit, both sides are now correct, and
either alone would have been sufficient.
tests/test_script_exec_bit_contract.sh pins the class so it cannot recur: any
tracked .sh whose COMMITTED mode is non-executable must not appear as the first
word of a command in workflows, scripts, test-infrastructure or the Makefiles.
The committed mode is the thing that matters and the thing no local run can
check -- the working copy having the bit is exactly why this shipped.
Verified in both directions: passes on this tree, and fails on the exact defect
when the mode and the call site are reverted together. It joins backslash
continuations before analysing, because its own first draft reported a false
positive on
... && bash \
test-infrastructure/vm/vm-run-tests.sh --soak
where the interpreter sits on the preceding line. A contract that cries wolf
teaches people to ignore contracts, so that had to be right before it could be
useful.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The pre-release ML false-positive tolerance (single-engine Microsoft "!ml"
verdicts downgradable with Defender endpoint evidence, #1340) is reverted by
owner decision: cbm does not ship binaries carrying a VirusTotal detection,
demonstrably false or not. A "trojan" badge on a release asset is a
reputation cost the project is not willing to price in, however good the
accompanying evidence.
The gate returns to its original form: any detection, by any engine, on any
artifact, on any version blocks the release. The endpoint verification tool
and the evidence side-channel are removed with it; the notes renderer keeps
its extracted-script form but only ever states a verified "0 detections".
False positives are resolved upstream instead: verify the bytes on a real
Defender endpoint, submit a Microsoft false-positive report for the exact
hashes, wait for the detection to clear, then RE-RUN the failed verify job --
which does not rebuild, so the cleared hashes are the shipped hashes.
tests/test_vt_gate_zero_tolerance_contract.sh pins the decision: clean
passes; 1 malicious and 1 suspicious each block across stable, -rc., -pre
and -alpha versions; plus a tripwire for the specific reverted evidence
mechanism returning. Loosening this gate again has to consciously delete
that contract.
The skip_tests dispatch input and the script-extracted notes step survive
the revert -- both are orthogonal to gating policy.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Release run 30464288732 was blocked by the VirusTotal gate: three linux-amd64
binaries flagged 1/62 by Microsoft's Wacatac.B!ml -- fully stripped binaries
(0 symbols, verified on the exact artifacts), the state that scanned clean in
the two previous cycles. Meanwhile a real Defender endpoint (engine
1.1.26060.3008, signatures 1.455.410.0 updated the same day, RTP on) scans
the identical bytes clean. Four cycles of evidence now say the same thing:
this verdict is an unstable ML decision boundary, not a property of the code,
and no build-side lever moves it durably -- stripping, downloader removal and
metadata changes each "worked" only until a later build flipped it back.
So stop treating the flag as buildable-away and verify it honestly instead:
check-virustotal.sh may downgrade BLOCKED to TOLERATED only when ALL hold:
- pre-release version (-rc./-pre/-alpha/-beta); stable releases never
- every failing file flagged by exactly ONE engine
- that engine is Microsoft and the verdict ends in "!ml" (never a
signature name)
- hash-pinned Defender ENDPOINT evidence is attached to the draft release
(defender-endpoint-verification.txt) proving Microsoft's shipping
product, signature-updated at scan time, reports the exact bytes clean
av-endpoint-verify.sh (new) produces that evidence: downloads the draft
assets, scans them on the local Windows VM endpoint, refuses to attest if
RTP is off or Defender itself detects, uploads the hash-pinned result.
The gate prints the exact command when evidence is missing; re-running the
failed verify job does not rebuild, so the bytes stay fixed.
append-vt-notes.sh (new, extracted from inline YAML per venue-parity) then
renders the release-notes table honestly: a tolerated file reads "1/62 ML
false positive, endpoint-verified clean", never "0 detections".
tests/test_vt_gate_tolerance_contract.sh pins all nine decision directions
against a stubbed VT API and release store -- clean pass, stable-never,
missing/stale/DETECTED evidence, signature-named verdict, non-Microsoft
engine, multi-engine -- so the tolerance provably fails closed.
Also: release.yml gains skip_tests for re-releases of an already test-green
tree (build/smoke/soak/verify always run; lint failures still gate via
!cancelled() && !failure()).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Windows shipped a PAIR: a small permanent launcher (codebase-memory-mcp.exe)
plus the real product binary (codebase-memory-mcp.payload.exe). The launcher
existed for exactly one reason — a running .exe cannot replace its own image
on Windows, so an in-process self-update needs a second resident binary to do
the swap.
That stub is statically indistinguishable from a dropper: a small, unsigned,
zero-prevalence PE whose whole job is verify-and-execute another binary.
Defender's ML scored it Trojan:Win32/Wacatac.B!ml and blocked the v0.9.1-rc.1
release at the VirusTotal gate. It is not fixable in our code on x64 —
bcrypt-free, stripped, VERSIONINFO'd, minimal-resource and even
resource-FREE builds on CI's own MSYS2 CLANG64 toolchain were all flagged,
while the product binary scans clean on every platform.
So remove the stub and move self-update OUT of the process into install.ps1,
which runs while cbm is NOT running: Windows' image lock only blocks a
process from replacing ITSELF. now prints the exact PowerShell
command (with the Unblock-File hint for Mark-of-the-Web); install.ps1 is
idempotent, so re-running it IS the update — it stops the daemon, renames the
running binary aside (the one mutation Windows permits on a running image),
publishes the new one, and sweeps retired copies.
Windows now matches Linux and macOS: ONE binary per platform.
* packaging, install.ps1, npm and PyPI wrappers all carry a single binary
* the launcher/payload ABI contract and ~2500 lines of stub state machinery
are deleted
* every daemon start, CLI call and hook fire loses a process spawn, a named
pipe handshake and an stdio relay
* test_windows_bundle_contract.sh is rewritten as an INVERTED contract: it
now asserts no shipped surface can reintroduce a launcher/payload pair,
and that install.ps1 retires the running binary before publishing
Verified: VirusTotal 0/67 on the packaged binary and 0/58 on install.ps1 (no
certificate involved); macOS and Linux full suites green; Windows guards all
green including the new update-handoff contract; npm 10/10; PyPI 3/3.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A version containing a prerelease suffix (v0.9.1-rc.1) now publishes as an
RC on every channel instead of masquerading as stable:
- GitHub release is created with prerelease: true (never marked Latest)
- npm publishes under the `next` dist-tag, so plain `npm install` keeps
serving the last stable; testers opt in with codebase-memory-mcp@next
- the MCP Registry sync is skipped (the registry has no channel concept)
- PyPI needs no change: PEP 440 already treats rc versions as pre-releases
that pip only installs with --pre
Stable releases (no dash) are byte-identical in behavior.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
v4's PASSING smoke legs already ran 12-14 minutes against the 15-min
timeout; v5 lost 13 green-path jobs to that wall (GitHub reports job
timeouts as 'cancelled', zero real failures). The unified smoke
legitimately runs more than the pre-unification one — artifact mode,
real download/update phases, a Phase 15 that actually executes.
Budget doctrine (see _soak.yml, violated twice there historically):
the timeout sits above the worst case with headroom.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The version preflight PASSED (both images printed their version) and
then the job died in the cleanup trap: Windows keeps the just-exited
image's file busy for ~100ms (section teardown / first-touch AV), so
'rm -rf' in the EXIT trap raced it — 'Device or resource busy', exit 1
on a job whose actual work was green (observed on windows-11-arm ui in
dry run 30232946513). Temp-dir cleanup on an ephemeral runner is
best-effort with one retry, never a gate.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The manifest content (leg, slice, list hash) is fully determined at
slicing time, so write it up front: a red run's manifest is exactly as
load-bearing for the cross-shard union proof as a green one's.
With that, a missing manifest at upload time can only mean the job
died before the harness started (contract step / build failure) — that
failure is already the job's red, so the upload warns instead of
stacking a second error on top (the recurring 'No files were found
with the provided path: build/c/test-logs/shard-manifest.txt'
annotation). The shard-completeness job still gates the union.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Dry run 30225685743 failed at the ensure-defender step on every
Windows job: the hosted images policy-lock real-time protection OFF
(WinDefend starts, Set-MpPreference accepts, RTP stays off —
untamperable from the job).
Revert to the pre-gate runner behavior: no ensure-defender step in
runner jobs, the release Defender scan is on-demand and fail-soft
again. The local VM preflight KEEPS the fail-closed Defender-ON gate
(same canonical script) — AV-interaction coverage is a deliberate
local superset where the platform allows it.
The venue-parity contract now enforces the INVERSE for runners
(ensure-defender must not gate hosted jobs) and still requires the VM
preflight gate; VM README documents the verified posture delta.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Closes the closable local-vs-remote venue gaps so that, outside arch
legs / YAML glue / release plumbing, every local red is a remote red
and vice versa on shared legs:
- scripts/package-release.sh: THE canonical archive step (names +
five-file Windows bundle layout defined once); _build.yml's eight
inline archive blocks become calls to it.
- scripts/ci/smoke-artifact.sh: the artifact-flow smoke lane — build,
package, extract, then the canonical wrapper in artifact mode; wired
as run.sh smoke-artifact (compose service), win.sh smoke-artifact,
and directly runnable on macOS. Archive-layout bugs now surface
locally instead of in a release dry run.
- glibc-floor leg (Dockerfile.glibc22 + compose + run.sh): portable
binary smokes on ubuntu-22.04/glibc 2.35; the dynamic binary must
refuse there (2.38+ floor by design).
- Defender-ON parity (user directive): scripts/ci/ensure-defender.ps1
enables + VERIFIES real-time protection, fail-closed; runs in every
Windows CI job (_test x2, _soak x3, _smoke, pr.yml) AND in the VM
preflight; _smoke.yml's scan engine-failure soft-skip becomes a red
gate. Expected cost: slower Windows jobs (AV scanning during
install/build/test I/O); the next dry run proves the runner side.
- Contracts: launcher-bundle five-file check retargeted onto
package-release.sh + per-archive canonical-call association;
venue-parity contract requires the new lanes, counts one
ensure-defender step per Windows job, adds --help probes for the new
entries. Extended contract fails on the pre-change tree (verified:
20 violations on HEAD).
- VM README: ephemerality/Defender posture documented — utmctl has no
snapshot verb, so per-run revert stays a manual qcow2 option; the
sweep preflight remains the standing mechanism.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The staged venue-unification bundle: _smoke.yml onto the shared
wrappers with an extracted-artifact input (CBM_SMOKE_ARTIFACT_DIR),
soak-legs.sh as the one canonical soak entry (quick + query-leak legs
across all venues), protected per-user TEMP roots shared across
venues, clean-disk preflights before every VM/Docker run, win.sh and
run.sh routed through the canonical scripts, soak.yml retired, and
the venue-parity contract (whitelist walker: workflows may provision
or call canonical entries, nothing else).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Contributors could not distinguish "blocked" from "ignored". Several PRs sat on
green, mergeable work for over a week with no maintainer comment while
release-critical work absorbed review capacity, and nothing on the PR page said
so. GitHub has no repo-scoped banner for pull requests -- an org announcement
banner renders only for org members, and pinned issues and discussions never
appear on the PR tab -- so a comment on open is the only surface that reaches an
external contributor at the moment they contribute.
Posts one comment when a PR is opened, stating that it is queued, that it will
not be closed for inactivity, and what genuinely speeds up review (rebased
branch, green CI, one claim per PR, signed-off commits).
The message lives in .github/pr-acknowledgement.md rather than in the workflow,
so the status can be edited -- or acknowledgements switched off by blanking the
file -- without a workflow change. A missing, empty, or whitespace-only file is
the documented off switch.
Uses pull_request_target because a pull_request trigger hands fork PRs a
read-only token, which would fail on exactly the contributions most worth
acknowledging. The job never checks out, builds, or executes PR code: checkout
carries no ref and so resolves to the base commit, it is used only to read our
own message file, and the PR number reaches the shell through the environment
rather than string interpolation. Skips bots and owner-authored PRs.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The stale workflow closed `awaiting-reporter` issues as `not_planned` after 35
days. That label is applied by hand and means "the reporter owes the next step",
but it was also being used to mean "we will come back to this" -- and in that
second case the automation closed issues the project itself owed work on.
#56 is the concrete case: a confirmed bug whose reproduction landed in-tree via
#667 (tests/repro/repro_issue56.c), queued to close as `not_planned` purely
because a maintainer had the last word. 13 of the 22 currently labelled issues
are priority/high.
Closing as "not planned" is a maintainer judgement, so the bot no longer makes
it. days-before-issue-close: -1 disables closing entirely; the `stale` label
becomes a review queue that a human sweeps during triage. A mislabel now costs a
stale tag instead of a silently discarded bug.
Also adds exempt-issue-labels for security, task (umbrella/epic issues incl. the
pinned roadmap index #595) and maintainer-notes, so those are never flagged even
if mislabelled. Drops the now-unreachable close-issue-message and
close-issue-reason, and rewrites the stale notice, which promised a closure that
no longer happens.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The windows-11-arm leg's first execution (broad matrix, release dry
run) died before any test ran: the trap-UBSan SANITIZE value passed
unquoted through the workflow ternary, so -fsanitize-trap=undefined
arrived as its own word and make consumed the leading -f as its
makefile flag ('No rule to make target sanitize-trap=undefined').
- The workflow quotes the ternary value so test.sh receives one
argument.
- test.sh forwards make arguments through an ARRAY instead of a
re-split string, so a VAR=VAL whose value contains spaces survives
every boundary (empty-array expansion kept bash-3.2-safe for the
macOS runners under set -u).
Verified: argument-parse proof (argc=1 with the four-flag value) and a
full macOS suite run through the changed plumbing, 6776 passed,
0 failed — identical totals.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Flip shard_suites on for all three callers of the reusable test
workflow. This commit is deliberately the only topology change:
reverting it alone restores the previous single-job-per-leg CI shape
while keeping the (inert without it) sharding machinery and the
cache-warm pipeline.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
PR iteration was bounded by single-job suite runs (~40-50 min critical
path) and by every first-of-ref build starting cold. Same suites, same
sanitizers, same gates — the schedule, the cache reuse, and a new
runtime coverage proof change.
- run-tests-parallel.sh accepts CBM_TEST_SHARD="i/N" and runs a
deterministic slice of --list-suites; unset selects everything, so the
path is inert outside CI. Known-heavy suites are dealt to shards in
weight order first (naive modulo stacked the two slowest, store_arch
and daemon_runtime, onto one job), the parallel wave and the serial
tail are sliced separately so every shard keeps its own quiet tail,
and the per-shard union guard proves each job ran exactly its slice.
- shard-completeness job: per-shard guards cannot see a mis-plumbed
CBM_TEST_SHARD (two jobs running the same slice passes every local
check while a slice runs nowhere). Every leg now uploads a manifest
(leg, i/N, sha of the full suite list, its slice) and one aggregation
job re-proves per leg that the shards agreed on the list, indices are
exactly 1..N, and the union of slices IS the list. Runs on unsharded
topologies too, where each leg's single manifest must cover the list.
- workflow topology (shard_suites input, off by default = byte-identical
to the pre-shard matrix): ubuntu legs run 3 shards, Windows 2 (every
extra Windows shard re-pays ~5 min of MSYS2 setup), macOS stays at 1
(not the critical path; mac runner concurrency ceilings are the
tightest). The Windows test job also pre-creates build\c with a
protected DACL so build products inherit it — workspace drive roots
grant Authenticated Users Modify by inheritance, which the activation
transaction's source-directory policy correctly refuses.
- ccache: a final ref-less restore key lets a fresh PR start from the
newest cache GitHub's scoping permits, and a nightly build-only
cache-warm workflow keeps main-scoped caches at most a day stale —
without it the fallback had no warm source, since nothing built in
main's scope. The strictly-per-ref policy this replaces cost ~8-12
minutes on every first-of-ref build for no safety gain:
CCACHE_COMPILERCHECK=content makes a stale or foreign cache able to
miss but never to return wrong output. Sharded legs restore-all/
save-one (every shard builds the identical objects, so shard 1's
cache carries the full set) — save volume stays flat and warm caches
stop being evicted by per-shard duplicates.
- suite timeouts: daemon_runtime joins the slow tier. It measures ~610s
solo on arm64 under ASan (10-round loop, no hang), so the 900s default
under a loaded 4-job CI runner was a slowness kill masquerading as a
hang detector.
Validated locally: 3-shard union over the real suite list is complete
with no duplicates, one top-heavy suite per shard, and the summed
3-shard totals reproduce the unsharded run exactly (6776 passed,
0 failed, 4 skipped).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Close the cross-platform sanitizer gaps that were leaving real
concurrency and undefined-behavior bugs uncaught, and fix a data race
the first widened run surfaced.
- tsan: the data-race gate ran three suites (mem, slab_alloc, parallel)
over no real threaded production code. It now covers every threaded
surface that runs clean and stable under TSan: the parallel-extraction
worker pool (parallel, worker_pool, pipeline), the filesystem watcher,
the embedded HTTP server (httpd), diagnostics sampling, the MCP server
and mutation guard, subprocess supervision, and the runnable
daemon-coordination paths (daemon, daemon_application). daemon_runtime
(deadlocks under TSan+fork), daemon_ipc and daemon_frontend
(test-harness synchronization, not production) are excluded with the
reasons recorded in the Makefile.
- tsan: fixed a genuine data race the widened gate immediately found —
cbm_lsp_max_walk_depth's lazy cache was read and written by parallel
LSP-extraction workers without synchronization. A data race is
undefined behavior even when every worker computes the same value, so
the cache slot is now a relaxed atomic: a plain load on the hot path,
and a first-touch double-compute simply stores the same value.
- tsan(ci + local): the test-tsan job now runs on Linux amd64, Linux
arm64, AND native ARM64 macOS (the threading code is shared, so a race
is usually caught on all three, but scheduler differences let each
surface one the others miss). The local ladder gained `run.sh tsan`
and `tsan-amd64` plus the matching compose services. TSan's shadow
memory aborts under modern high-entropy ASLR, so the containers run
under `setarch -R` with an unconfined seccomp profile (the personality
syscall is otherwise blocked) and the CI Linux legs lower
vm.mmap_rnd_bits first; amd64 TSan cannot run under x86_64-on-ARM
translation and is a real-hardware/CI gate only (documented in
run.sh).
- ubsan(win/arm64): native ARM64 Windows had no sanitizer at all —
AddressSanitizer ships no aarch64-w64-windows-gnu runtime. UBSan in
trap mode (-fsanitize-trap=undefined) needs no runtime library, so it
instruments natively and turns undefined behavior into an
illegal-instruction trap; -fstack-protector-strong adds stack-smash
coverage the heap tools miss. The GitHub windows-11-arm leg switches
from unsanitized to this, and vm/win.sh gains trap-ubsan-build /
trap-ubsan-test for local iteration (reproduce under the emulated
x86_64 UBSan to see which check fired). The whole codebase builds and
runs clean under it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Daemon lifecycle and Windows correctness, verified on a real Windows 11
ARM64 VM through the maintained test-infrastructure/vm drivers, plus the
macOS and Linux arm64 suites and the container lint gate.
Daemon lifecycle:
- daemon start/stop/status subcommands. `daemon start` launches a
PERMANENT daemon (spawn shape is byte-exact argv; survives idle
periods and session ends) and reports an already-active daemon
instead of failing. Permanence is honored at every stop latch:
last-committed-client disconnect, host initial-client window,
coordinator release, and application final-session close — a
permanent daemon also keeps admitting new sessions after its last
one closes.
- daemon stop refuses while sessions are active and lists the blocking
peers (pid/role) that must finish first; an idle daemon drains
through the activation-shutdown machinery with the ACK ordered after
connection interrupts. A second stop is idempotent. The wire ops are
no-cohort first-frame requests with peer fingerprint authentication,
so stop/status never conflict with an exact-build admission gate.
- One-shot CLI commands now execute through the daemon (index workers
keep their local supervised path). A cold CLI run that had to spawn a
temporary daemon prints a hint that `daemon start` removes the
per-command startup tax; a warm daemon is recycled silently.
- Hooks are connect-only fail-open: with no daemon present the hook
emits a visible, rate-limited notice (Claude-dialect systemMessage
plus stderr for other dialects) and always exits 0 — augmentation is
never allowed to block the caller's tool use.
- Version skew: a newer-build client automatically drains an
older-build permanent daemon (strict semantic-version triples only;
dev builds never auto-drain) and the build-conflict message names
`cbm daemon stop` as the manual escape hatch.
Windows IPC/runtime (real-VM verified):
- ipc(win): persistent pending overlapped ConnectNamedPipe. The accept
path used to destroy its listening pipe instance on every 20 ms poll
timeout; a client attaching in the teardown window was severed or left
on an orphaned pipe object whose HELLO no server handle could ever
read, absorbing the connect until the client's own timeout expired.
The pending connect now survives poll timeouts and nothing is
destroyed while a client could be attaching.
- ipc(win): drain-before-close for final responses. Closing a named-pipe
server handle can discard a just-sent response before the peer reads
it (POSIX stream sockets never lose buffered data on close). A bounded
cbm_daemon_ipc_connection_drain (read-until-peer-EOF; no-op on POSIX,
immediate on interrupted connections) now precedes close in
runtime_worker_finish and runtime_reject_inline, so hello-conflict,
capacity and disconnect acknowledgements reliably reach the peer.
- runtime: CLOSE_INTENT wire frame. A Windows named-pipe client has no
transport half-close, so close_begin now announces departure with an
explicit frame (ordered after APPLICATION_CANCEL, before the local
interrupt); the server releases the client's admission on receipt
instead of waiting for the handle to close. Admission-drop timing is
now identical to POSIX shutdown() semantics on every platform.
- runtime(win): client close cancellation. close_begin serializes with
request publication under the send lock, best-effort sends the active
token's APPLICATION_CANCEL frame, then interrupts local I/O; the
server cancels MCP/subprocess work promptly. Contract tests accept
both correct outcomes (interrupted transport or decoded CANCELLED).
- runtime: activation acknowledgement ordering. The activation ACK is
the requester's license to act on "snapshotted and draining", so every
connection interrupt is now initiated before the ACK is sent; a
session could previously get one more request serviced after the
requester observed the ACK.
- service(win): deadline-bounded private-file prepare. The conflict-log
prepare retry loop (100 x Sleep(2), which rounds up to the ~16 ms
timer granularity) burned ~1.6 s against permanently obstructed paths,
stalling hello rejections past the client's timeout. The retry budget
is now a 250 ms deadline; transient share collisions still retry.
- subprocess(win): cmd.exe /C payload encoder quotes metacharacters
correctly (root cause of the git-on-Windows failure cluster).
- watcher: SHA-256 buffer sizing (CBM_SZ_64 -> CBM_SZ_128) and a native
Windows stop/unwatch cancellation test with exact-image verification.
- httpd: send_all writes in bounded 64 KiB slices. A single giant
nonblocking send() on Windows is absorbed wholesale into AFD kernel
buffering regardless of SO_SNDBUF, so send deadlines and interrupts
could never engage against a slow peer (and the full payload was
pinned in nonpaged pool). Slicing restores a deterministic
backpressure point; a test hook pins SO_SNDBUF for the deadline and
interrupt tests.
- ui/http: shutdown lifecycle — interrupt checks, response-wide send
deadline, explicit connection states, refusal to free a server while
a listener-owned connection is active.
Windows long-path support:
- Central path-aware wide conversion (canonicalize via GetFullPathNameW
and prepend the extended-length prefix for absolute paths >=240) at
the compat chokepoints (cbm_fopen/compat_fs/mkstemp/mkdtemp), sqlite
store opens, and the daemon build-fingerprint/log paths. Deep managed
installs (a 64-hex generation directory routinely exceeds MAX_PATH)
now index, stage and activate correctly.
- activation transaction: its own file APIs and the component-walking
ancestry validators now operate in the extended-length namespace;
the launcher path is canonicalized (and prefixed when deep) once at
entry so every downstream exact-string comparison stays
form-consistent.
- Executable self-resolution uses the wide APIs (GetModuleFileNameW,
GetFileAttributesW) so non-ASCII install paths survive argv[0]
resolution.
Windows launcher install/uninstall transaction:
- FileRenameInfoEx names are NUL-terminated in an over-allocated
buffer. FileNameLength governs per the contract, but filter drivers
read FileName as NUL-terminated and appended adjacent heap bytes to
created names — a flaky, garbage-suffixed rename target. Both the CLI
and the launcher rename helpers are fixed.
- Uninstall retires state via rename-aside (.cbm ->
.cbm-retired-v1-<tag>-<pid>) with the retired tag shortened to 16 hex
chars so the bare rename target stays under the FileRenameInfoEx
NT-conversion ceiling at guard depths; 64 bits still uniquely
identify the generation.
- When the running launcher's mapped generation backings pin .cbm
against rename, the backings are relocated to activation-<pid>-N
.retired tombstones beside the install (a mapped image may be renamed,
never deleted; the launcher's liveness-guarded sweep reclaims stale
tombstones). Every relocation is recorded, and a FAILED uninstall
reverses the moves after restoring .cbm — via MoveFileExW with
extended-length paths on both arguments, since the deep generation
target is beyond the handle-based rename's bare-path reach — so a
restored install keeps its generation backings and stays runnable.
- After a committed uninstall the retired tree's backings are relocated
out so the tree is shallow enough for the detached cleanup's rd, and
the cleanup's working directory strips the extended-length prefix
(CreateProcessW lpCurrentDirectory silently ignores prefixed paths).
- Files created under Administrators-default-owner directories
(CopyFileW destinations, CREATE_NEW tombstones, probe directories)
are explicitly owner-stamped so the exact-owner validators hold on
runner images; guard fixtures stamp hand-built trees the same way.
Diagnostics, tests and infra:
- diagnostics: discovery is now an always-delivered JSON control record
(new cbm_log_control) that survives CBM_LOG_LEVEL suppression and
paths containing spaces; placement honors $TMPDIR with /tmp fallback
via a diagnostics-local helper; the soak parser reads the JSON record;
documented in docs/CONFIGURATION.md. Red-first coverage for suppressed
log levels, TMPDIR-with-spaces, and native Windows output-contract
assertions.
- tests(win): daemon_ipc/daemon_frontend fixtures now build endpoint
parents with production-shaped ancestry (LocalAppData on Windows, via
th_secure_runtime_parent_new) — the runtime ancestry validation
correctly refuses temp roots whose ancestors grant mutation rights to
Authenticated Users (C:/msys64/tmp, GitHub-runner work dirs) — and
drive the documented startup-owner publication flow before reading
generation-bound endpoint addresses. This turns the 26 Windows
failures previously visible in CI's full-test job green without
weakening any validation.
- tests(win): the launcher guard covers the full permanent-launcher
contract including failed-uninstall restore and immediate reinstall
after uninstall; new daemon lifecycle and reworked hook-augment
guards run the start/recycle/stop flow end to end.
- tests: CBM_SKIP_PERF is now actually consumed by the test runner
(it was set by CI but never read, so perf suites ran everywhere);
four throughput/bench suites are classified as perf, the heavy
store_arch suite moved to the slow-timeout tier, and two
wall-clock-sensitive assertions were rewritten as invariant checks
with coarse hang-detector backstops.
- build/test infra: build-dir safety contract, UI dev-proxy security
contract, soak daemon-recovery contract, path-safety helper, the
Windows VM worktree-sync contract wired into scripts/test.sh, and
vm/win.sh guards building its clean embedded-UI product in an
isolated BUILD_DIR so it cannot clobber the incremental test build.
provision-windows.sh now installs Node.js for the guards UI build.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Combines the ccache steps with this branch's Windows hardening: the
protected temp root and profile staging keep their roles, the cache
steps slot alongside them, and the strictly per-ref keys apply
unchanged. The daemon branch's own CI legs warm from the second push
on.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
ccache with CCACHE_COMPILERCHECK=content everywhere: every cache entry
is keyed on the CONTENT of the compiler binary plus the fully
preprocessed translation unit, so a hit is provably the identical
compilation - a stale, foreign, or corrupted cache can only MISS, never
return wrong output. No CCACHE_BASEDIR and no path rewriting: debug
info and sanitizer report paths stay exact.
Locally scripts/env.sh routes compilers through ccache's masquerade
directories when present (opt-out CBM_NO_CCACHE=1) - $CC keeps its
plain name, so verify_compiler, make, and link lines are untouched.
CI caches are strictly per-ref by policy on top of GitHub's own branch
scoping: keys embed github.ref, so no base-branch fallback - a new PR
builds cold once and only its own pushes warm it. Cached: the four
test jobs and the pr-smoke matrix. Release builds in _build.yml stay
deliberately uncached.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Completes the handoff task for the Windows harness: the native test job
ran with the MSYS-shared /tmp and the runner's inherited
LocalAppData\Temp ACLs, which the daemon's strict private-filesystem
checks reject by design. The test-windows job now creates a per-user
root under the profile with an owner-stamped, protected current-SID
DACL and routes TEMP/TMP (native form) and TMPDIR (POSIX form) through
it before scripts/test.sh runs.
The guards script already hardened its root DACL but ownership is never
inherited on Windows: bundle copies created by the admin-group runner
token can come out Administrators-owned, and the launcher's exe policy
requires the exact current user as owner. Stamp the current SID on the
staged bundle dir and both executables explicitly.
Job topology, gates, and triggers are unchanged; the new step is a
sub-second pwsh block in the existing test-windows job.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
github/issue-labeler compiles patterns with JavaScript RegExp, which
rejects PCRE-only inline (?i) groups with SyntaxError: Invalid group, so
the workflow failed before evaluating any rule and no issue ever got an
area label (#764).
Replace the third-party action with first-party actions/github-script:
the same config rules are compiled with the i flag applied centrally
(case-insensitivity preserved), labels stay additive-only via addLabels,
and a pattern that fails to compile now fails the run loudly instead of
silently no-oping. Strip (?i) from all 7 config patterns and document
the line format the workflow parses.
Verified locally: all 7 rules compile and match expected labels across
sample issue texts; the old pattern reproduces the SyntaxError.
Closes#764
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Defender step treated any non-zero MpCmdRun exit as 'flagged binary!'
and failed the job. But MpCmdRun -Scan returns 2 for a real detection and
0 for clean; any other non-zero means the scan engine itself could not run
(e.g. hr=0x800106ba: the Defender antimalware service is unavailable on the
runner), which is a transient runner-side flake, not a detection. This
false-failed the v0.9.0 release on its first attempt. Only exit 2 now hard-
blocks; an engine failure emits a warning and continues.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The persistent windows-11-arm smoke Phase 12a failure was a 404, not a network
error: it requested codebase-memory-mcp-windows-amd64.zip on the arm64 leg. Cause:
DL_ARCH came from `uname -m`, which on windows-11-arm is an emulated x86_64 MSYS2
uname reporting "x86_64" -> wrong (amd64) archive -> 404 (server has arm64). Phase
14 worked because the binary's own detect_arch() is native. Prefer SMOKE_ARCH
(passed from the smoke workflow's matrix.arch) over uname; fall back to uname for
local runs. This is the real cause the earlier curl/proxy/ipv4 attempts masked --
the 404 was swallowed by 2>/dev/null until #905 surfaced it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel