The windows-11-arm leg's first execution (broad matrix, release dry
run) died before any test ran: the trap-UBSan SANITIZE value passed
unquoted through the workflow ternary, so -fsanitize-trap=undefined
arrived as its own word and make consumed the leading -f as its
makefile flag ('No rule to make target sanitize-trap=undefined').
- The workflow quotes the ternary value so test.sh receives one
argument.
- test.sh forwards make arguments through an ARRAY instead of a
re-split string, so a VAR=VAL whose value contains spaces survives
every boundary (empty-array expansion kept bash-3.2-safe for the
macOS runners under set -u).
Verified: argument-parse proof (argc=1 with the four-flag value) and a
full macOS suite run through the changed plumbing, 6776 passed,
0 failed — identical totals.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Flip shard_suites on for all three callers of the reusable test
workflow. This commit is deliberately the only topology change:
reverting it alone restores the previous single-job-per-leg CI shape
while keeping the (inert without it) sharding machinery and the
cache-warm pipeline.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
PR iteration was bounded by single-job suite runs (~40-50 min critical
path) and by every first-of-ref build starting cold. Same suites, same
sanitizers, same gates — the schedule, the cache reuse, and a new
runtime coverage proof change.
- run-tests-parallel.sh accepts CBM_TEST_SHARD="i/N" and runs a
deterministic slice of --list-suites; unset selects everything, so the
path is inert outside CI. Known-heavy suites are dealt to shards in
weight order first (naive modulo stacked the two slowest, store_arch
and daemon_runtime, onto one job), the parallel wave and the serial
tail are sliced separately so every shard keeps its own quiet tail,
and the per-shard union guard proves each job ran exactly its slice.
- shard-completeness job: per-shard guards cannot see a mis-plumbed
CBM_TEST_SHARD (two jobs running the same slice passes every local
check while a slice runs nowhere). Every leg now uploads a manifest
(leg, i/N, sha of the full suite list, its slice) and one aggregation
job re-proves per leg that the shards agreed on the list, indices are
exactly 1..N, and the union of slices IS the list. Runs on unsharded
topologies too, where each leg's single manifest must cover the list.
- workflow topology (shard_suites input, off by default = byte-identical
to the pre-shard matrix): ubuntu legs run 3 shards, Windows 2 (every
extra Windows shard re-pays ~5 min of MSYS2 setup), macOS stays at 1
(not the critical path; mac runner concurrency ceilings are the
tightest). The Windows test job also pre-creates build\c with a
protected DACL so build products inherit it — workspace drive roots
grant Authenticated Users Modify by inheritance, which the activation
transaction's source-directory policy correctly refuses.
- ccache: a final ref-less restore key lets a fresh PR start from the
newest cache GitHub's scoping permits, and a nightly build-only
cache-warm workflow keeps main-scoped caches at most a day stale —
without it the fallback had no warm source, since nothing built in
main's scope. The strictly-per-ref policy this replaces cost ~8-12
minutes on every first-of-ref build for no safety gain:
CCACHE_COMPILERCHECK=content makes a stale or foreign cache able to
miss but never to return wrong output. Sharded legs restore-all/
save-one (every shard builds the identical objects, so shard 1's
cache carries the full set) — save volume stays flat and warm caches
stop being evicted by per-shard duplicates.
- suite timeouts: daemon_runtime joins the slow tier. It measures ~610s
solo on arm64 under ASan (10-round loop, no hang), so the 900s default
under a loaded 4-job CI runner was a slowness kill masquerading as a
hang detector.
Validated locally: 3-shard union over the real suite list is complete
with no duplicates, one top-heavy suite per shard, and the summed
3-shard totals reproduce the unsharded run exactly (6776 passed,
0 failed, 4 skipped).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Close the cross-platform sanitizer gaps that were leaving real
concurrency and undefined-behavior bugs uncaught, and fix a data race
the first widened run surfaced.
- tsan: the data-race gate ran three suites (mem, slab_alloc, parallel)
over no real threaded production code. It now covers every threaded
surface that runs clean and stable under TSan: the parallel-extraction
worker pool (parallel, worker_pool, pipeline), the filesystem watcher,
the embedded HTTP server (httpd), diagnostics sampling, the MCP server
and mutation guard, subprocess supervision, and the runnable
daemon-coordination paths (daemon, daemon_application). daemon_runtime
(deadlocks under TSan+fork), daemon_ipc and daemon_frontend
(test-harness synchronization, not production) are excluded with the
reasons recorded in the Makefile.
- tsan: fixed a genuine data race the widened gate immediately found —
cbm_lsp_max_walk_depth's lazy cache was read and written by parallel
LSP-extraction workers without synchronization. A data race is
undefined behavior even when every worker computes the same value, so
the cache slot is now a relaxed atomic: a plain load on the hot path,
and a first-touch double-compute simply stores the same value.
- tsan(ci + local): the test-tsan job now runs on Linux amd64, Linux
arm64, AND native ARM64 macOS (the threading code is shared, so a race
is usually caught on all three, but scheduler differences let each
surface one the others miss). The local ladder gained `run.sh tsan`
and `tsan-amd64` plus the matching compose services. TSan's shadow
memory aborts under modern high-entropy ASLR, so the containers run
under `setarch -R` with an unconfined seccomp profile (the personality
syscall is otherwise blocked) and the CI Linux legs lower
vm.mmap_rnd_bits first; amd64 TSan cannot run under x86_64-on-ARM
translation and is a real-hardware/CI gate only (documented in
run.sh).
- ubsan(win/arm64): native ARM64 Windows had no sanitizer at all —
AddressSanitizer ships no aarch64-w64-windows-gnu runtime. UBSan in
trap mode (-fsanitize-trap=undefined) needs no runtime library, so it
instruments natively and turns undefined behavior into an
illegal-instruction trap; -fstack-protector-strong adds stack-smash
coverage the heap tools miss. The GitHub windows-11-arm leg switches
from unsanitized to this, and vm/win.sh gains trap-ubsan-build /
trap-ubsan-test for local iteration (reproduce under the emulated
x86_64 UBSan to see which check fired). The whole codebase builds and
runs clean under it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Daemon lifecycle and Windows correctness, verified on a real Windows 11
ARM64 VM through the maintained test-infrastructure/vm drivers, plus the
macOS and Linux arm64 suites and the container lint gate.
Daemon lifecycle:
- daemon start/stop/status subcommands. `daemon start` launches a
PERMANENT daemon (spawn shape is byte-exact argv; survives idle
periods and session ends) and reports an already-active daemon
instead of failing. Permanence is honored at every stop latch:
last-committed-client disconnect, host initial-client window,
coordinator release, and application final-session close — a
permanent daemon also keeps admitting new sessions after its last
one closes.
- daemon stop refuses while sessions are active and lists the blocking
peers (pid/role) that must finish first; an idle daemon drains
through the activation-shutdown machinery with the ACK ordered after
connection interrupts. A second stop is idempotent. The wire ops are
no-cohort first-frame requests with peer fingerprint authentication,
so stop/status never conflict with an exact-build admission gate.
- One-shot CLI commands now execute through the daemon (index workers
keep their local supervised path). A cold CLI run that had to spawn a
temporary daemon prints a hint that `daemon start` removes the
per-command startup tax; a warm daemon is recycled silently.
- Hooks are connect-only fail-open: with no daemon present the hook
emits a visible, rate-limited notice (Claude-dialect systemMessage
plus stderr for other dialects) and always exits 0 — augmentation is
never allowed to block the caller's tool use.
- Version skew: a newer-build client automatically drains an
older-build permanent daemon (strict semantic-version triples only;
dev builds never auto-drain) and the build-conflict message names
`cbm daemon stop` as the manual escape hatch.
Windows IPC/runtime (real-VM verified):
- ipc(win): persistent pending overlapped ConnectNamedPipe. The accept
path used to destroy its listening pipe instance on every 20 ms poll
timeout; a client attaching in the teardown window was severed or left
on an orphaned pipe object whose HELLO no server handle could ever
read, absorbing the connect until the client's own timeout expired.
The pending connect now survives poll timeouts and nothing is
destroyed while a client could be attaching.
- ipc(win): drain-before-close for final responses. Closing a named-pipe
server handle can discard a just-sent response before the peer reads
it (POSIX stream sockets never lose buffered data on close). A bounded
cbm_daemon_ipc_connection_drain (read-until-peer-EOF; no-op on POSIX,
immediate on interrupted connections) now precedes close in
runtime_worker_finish and runtime_reject_inline, so hello-conflict,
capacity and disconnect acknowledgements reliably reach the peer.
- runtime: CLOSE_INTENT wire frame. A Windows named-pipe client has no
transport half-close, so close_begin now announces departure with an
explicit frame (ordered after APPLICATION_CANCEL, before the local
interrupt); the server releases the client's admission on receipt
instead of waiting for the handle to close. Admission-drop timing is
now identical to POSIX shutdown() semantics on every platform.
- runtime(win): client close cancellation. close_begin serializes with
request publication under the send lock, best-effort sends the active
token's APPLICATION_CANCEL frame, then interrupts local I/O; the
server cancels MCP/subprocess work promptly. Contract tests accept
both correct outcomes (interrupted transport or decoded CANCELLED).
- runtime: activation acknowledgement ordering. The activation ACK is
the requester's license to act on "snapshotted and draining", so every
connection interrupt is now initiated before the ACK is sent; a
session could previously get one more request serviced after the
requester observed the ACK.
- service(win): deadline-bounded private-file prepare. The conflict-log
prepare retry loop (100 x Sleep(2), which rounds up to the ~16 ms
timer granularity) burned ~1.6 s against permanently obstructed paths,
stalling hello rejections past the client's timeout. The retry budget
is now a 250 ms deadline; transient share collisions still retry.
- subprocess(win): cmd.exe /C payload encoder quotes metacharacters
correctly (root cause of the git-on-Windows failure cluster).
- watcher: SHA-256 buffer sizing (CBM_SZ_64 -> CBM_SZ_128) and a native
Windows stop/unwatch cancellation test with exact-image verification.
- httpd: send_all writes in bounded 64 KiB slices. A single giant
nonblocking send() on Windows is absorbed wholesale into AFD kernel
buffering regardless of SO_SNDBUF, so send deadlines and interrupts
could never engage against a slow peer (and the full payload was
pinned in nonpaged pool). Slicing restores a deterministic
backpressure point; a test hook pins SO_SNDBUF for the deadline and
interrupt tests.
- ui/http: shutdown lifecycle — interrupt checks, response-wide send
deadline, explicit connection states, refusal to free a server while
a listener-owned connection is active.
Windows long-path support:
- Central path-aware wide conversion (canonicalize via GetFullPathNameW
and prepend the extended-length prefix for absolute paths >=240) at
the compat chokepoints (cbm_fopen/compat_fs/mkstemp/mkdtemp), sqlite
store opens, and the daemon build-fingerprint/log paths. Deep managed
installs (a 64-hex generation directory routinely exceeds MAX_PATH)
now index, stage and activate correctly.
- activation transaction: its own file APIs and the component-walking
ancestry validators now operate in the extended-length namespace;
the launcher path is canonicalized (and prefixed when deep) once at
entry so every downstream exact-string comparison stays
form-consistent.
- Executable self-resolution uses the wide APIs (GetModuleFileNameW,
GetFileAttributesW) so non-ASCII install paths survive argv[0]
resolution.
Windows launcher install/uninstall transaction:
- FileRenameInfoEx names are NUL-terminated in an over-allocated
buffer. FileNameLength governs per the contract, but filter drivers
read FileName as NUL-terminated and appended adjacent heap bytes to
created names — a flaky, garbage-suffixed rename target. Both the CLI
and the launcher rename helpers are fixed.
- Uninstall retires state via rename-aside (.cbm ->
.cbm-retired-v1-<tag>-<pid>) with the retired tag shortened to 16 hex
chars so the bare rename target stays under the FileRenameInfoEx
NT-conversion ceiling at guard depths; 64 bits still uniquely
identify the generation.
- When the running launcher's mapped generation backings pin .cbm
against rename, the backings are relocated to activation-<pid>-N
.retired tombstones beside the install (a mapped image may be renamed,
never deleted; the launcher's liveness-guarded sweep reclaims stale
tombstones). Every relocation is recorded, and a FAILED uninstall
reverses the moves after restoring .cbm — via MoveFileExW with
extended-length paths on both arguments, since the deep generation
target is beyond the handle-based rename's bare-path reach — so a
restored install keeps its generation backings and stays runnable.
- After a committed uninstall the retired tree's backings are relocated
out so the tree is shallow enough for the detached cleanup's rd, and
the cleanup's working directory strips the extended-length prefix
(CreateProcessW lpCurrentDirectory silently ignores prefixed paths).
- Files created under Administrators-default-owner directories
(CopyFileW destinations, CREATE_NEW tombstones, probe directories)
are explicitly owner-stamped so the exact-owner validators hold on
runner images; guard fixtures stamp hand-built trees the same way.
Diagnostics, tests and infra:
- diagnostics: discovery is now an always-delivered JSON control record
(new cbm_log_control) that survives CBM_LOG_LEVEL suppression and
paths containing spaces; placement honors $TMPDIR with /tmp fallback
via a diagnostics-local helper; the soak parser reads the JSON record;
documented in docs/CONFIGURATION.md. Red-first coverage for suppressed
log levels, TMPDIR-with-spaces, and native Windows output-contract
assertions.
- tests(win): daemon_ipc/daemon_frontend fixtures now build endpoint
parents with production-shaped ancestry (LocalAppData on Windows, via
th_secure_runtime_parent_new) — the runtime ancestry validation
correctly refuses temp roots whose ancestors grant mutation rights to
Authenticated Users (C:/msys64/tmp, GitHub-runner work dirs) — and
drive the documented startup-owner publication flow before reading
generation-bound endpoint addresses. This turns the 26 Windows
failures previously visible in CI's full-test job green without
weakening any validation.
- tests(win): the launcher guard covers the full permanent-launcher
contract including failed-uninstall restore and immediate reinstall
after uninstall; new daemon lifecycle and reworked hook-augment
guards run the start/recycle/stop flow end to end.
- tests: CBM_SKIP_PERF is now actually consumed by the test runner
(it was set by CI but never read, so perf suites ran everywhere);
four throughput/bench suites are classified as perf, the heavy
store_arch suite moved to the slow-timeout tier, and two
wall-clock-sensitive assertions were rewritten as invariant checks
with coarse hang-detector backstops.
- build/test infra: build-dir safety contract, UI dev-proxy security
contract, soak daemon-recovery contract, path-safety helper, the
Windows VM worktree-sync contract wired into scripts/test.sh, and
vm/win.sh guards building its clean embedded-UI product in an
isolated BUILD_DIR so it cannot clobber the incremental test build.
provision-windows.sh now installs Node.js for the guards UI build.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Combines the ccache steps with this branch's Windows hardening: the
protected temp root and profile staging keep their roles, the cache
steps slot alongside them, and the strictly per-ref keys apply
unchanged. The daemon branch's own CI legs warm from the second push
on.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
ccache with CCACHE_COMPILERCHECK=content everywhere: every cache entry
is keyed on the CONTENT of the compiler binary plus the fully
preprocessed translation unit, so a hit is provably the identical
compilation - a stale, foreign, or corrupted cache can only MISS, never
return wrong output. No CCACHE_BASEDIR and no path rewriting: debug
info and sanitizer report paths stay exact.
Locally scripts/env.sh routes compilers through ccache's masquerade
directories when present (opt-out CBM_NO_CCACHE=1) - $CC keeps its
plain name, so verify_compiler, make, and link lines are untouched.
CI caches are strictly per-ref by policy on top of GitHub's own branch
scoping: keys embed github.ref, so no base-branch fallback - a new PR
builds cold once and only its own pushes warm it. Cached: the four
test jobs and the pr-smoke matrix. Release builds in _build.yml stay
deliberately uncached.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Completes the handoff task for the Windows harness: the native test job
ran with the MSYS-shared /tmp and the runner's inherited
LocalAppData\Temp ACLs, which the daemon's strict private-filesystem
checks reject by design. The test-windows job now creates a per-user
root under the profile with an owner-stamped, protected current-SID
DACL and routes TEMP/TMP (native form) and TMPDIR (POSIX form) through
it before scripts/test.sh runs.
The guards script already hardened its root DACL but ownership is never
inherited on Windows: bundle copies created by the admin-group runner
token can come out Administrators-owned, and the launcher's exe policy
requires the exact current user as owner. Stamp the current SID on the
staged bundle dir and both executables explicitly.
Job topology, gates, and triggers are unchanged; the new step is a
sub-second pwsh block in the existing test-windows job.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
github/issue-labeler compiles patterns with JavaScript RegExp, which
rejects PCRE-only inline (?i) groups with SyntaxError: Invalid group, so
the workflow failed before evaluating any rule and no issue ever got an
area label (#764).
Replace the third-party action with first-party actions/github-script:
the same config rules are compiled with the i flag applied centrally
(case-insensitivity preserved), labels stay additive-only via addLabels,
and a pattern that fails to compile now fails the run loudly instead of
silently no-oping. Strip (?i) from all 7 config patterns and document
the line format the workflow parses.
Verified locally: all 7 rules compile and match expected labels across
sample issue texts; the old pattern reproduces the SyntaxError.
Closes#764
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Defender step treated any non-zero MpCmdRun exit as 'flagged binary!'
and failed the job. But MpCmdRun -Scan returns 2 for a real detection and
0 for clean; any other non-zero means the scan engine itself could not run
(e.g. hr=0x800106ba: the Defender antimalware service is unavailable on the
runner), which is a transient runner-side flake, not a detection. This
false-failed the v0.9.0 release on its first attempt. Only exit 2 now hard-
blocks; an engine failure emits a warning and continues.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The persistent windows-11-arm smoke Phase 12a failure was a 404, not a network
error: it requested codebase-memory-mcp-windows-amd64.zip on the arm64 leg. Cause:
DL_ARCH came from `uname -m`, which on windows-11-arm is an emulated x86_64 MSYS2
uname reporting "x86_64" -> wrong (amd64) archive -> 404 (server has arm64). Phase
14 worked because the binary's own detect_arch() is native. Prefer SMOKE_ARCH
(passed from the smoke workflow's matrix.arch) over uname; fall back to uname for
local runs. This is the real cause the earlier curl/proxy/ipv4 attempts masked --
the 404 was swallowed by 2>/dev/null until #905 surfaced it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Root cause of the windows-11-arm smoke Phase 12a failure: the smoke `curl` was
curl-8.20.0 from the base MSYS2 (msys) repo, which on an ARM64 host runs as an
x86_64 binary under emulation, and its networking could not reach the local
http.server -- while every NATIVE ARM64 process did (python's http.server bound
and served, and the product binary's own downloader completed Phase 14's update
from the same 127.0.0.1:18080). Install the native mingw-w64-clang-<arch>-curl so
the smoke curl is native (aarch64 on windows-11-arm), matching python and the
product binary. It takes PATH precedence over the base msys curl in the CLANG*
environments. amd64 gets the native x86_64 curl (already worked, now explicit).
Last red legs in the release dry run.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
curl (already installed) failed instantly in smoke Phase 12a on windows-11-arm
while the server was up and the binary's own downloader reached it -- the classic
localhost->::1 (IPv6) resolution: msys2 curl tried IPv6 but python's http.server
was IPv4-only. Bind the server to 127.0.0.1 and use a 127.0.0.1 SMOKE_DOWNLOAD_URL
so both ends are explicit IPv4. amd64 windows is unaffected (already reached via
IPv4). These were the last red legs in the release dry run.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The smoke-windows job's msys2 install list omitted curl, relying on an ambient
curl that resolves on CLANG64 (amd64) but not on CLANGARM64 (windows-11-arm).
Phase 12a's `curl` download failed there even though the server was up -- the
binary's own downloader (Phase 14) succeeded against the same server. Install
curl explicitly so both arches use the msys2 curl in the same network namespace
as the local http.server. These were the last two red legs in the release dry
run (everything else -- test, build incl build-windows-arm64, all smoke, and
soak incl soak-quick-windows-arm64 -- is green).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two release-blockers surfaced once #882 made every smoke leg a required gate
(continue-on-error previously masked them):
- smoke-unix ran the *dynamic* linux binary on ubuntu-22.04 / 22.04-arm, where
it cannot start (it links glibc 2.38+) -> Phase 1 died. Older-glibc coverage
is the -portable (static) binary's job and is green via the smoke-linux-portable
broad legs, so drop the impossible dynamic broad legs (keep macOS).
- the ClamAV scan died on every linux leg: apt auto-starts the clamav-freshclam
daemon which holds the freshclam log lock, so the manual freshclam failed with
"Failed to lock the log file ... Resource temporarily unavailable". Stop the
daemon first.
Also add native ARM64 Windows as a first-class release target -- the one platform
we test (windows-11-arm) but never shipped:
- build-windows-arm64 job (CLANGARM64) producing codebase-memory-mcp[-ui]-windows-arm64.zip
- smoke-windows is now arch-aware: windows-11-arm smokes the NATIVE arm64 binary
instead of the x86_64 binary under emulation
- a native soak-quick-windows-arm64 leg (no sanitizer -- ASan is unavailable on
native ARM64 Windows)
- arm64 detection in install.ps1
release.yml already publishes windows-arm64 (asset filter + merge-multiple) and
build_update_url/detect_arch already resolve it; npm/pip/install.sh compute arm64
already. scoop/winget/chocolatey per-version manifests are updated at release time.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The dry-run smoke Phase 14 runs the binary's real `update` command, which on
linux downloads the fully-static "-portable" asset (build_update_url in
src/cli/cli.c appends -portable on linux; _build.yml's build-linux-portable job
ships codebase-memory-mcp[-ui]-linux-<arch>-portable.tar.gz). The smoke artifact
server only served the non-portable name, so `update` 404'd and the config-
refresh assertion failed on every linux smoke leg. The mismatch (the smoke half
of the -portable delivery change) was masked until every smoke leg became a
required gate.
Mirror the tarball under the -portable name on linux (standard and ui) so the
served names match build_update_url; checksums.txt globs *.tar.gz and picks up
the new files. Verified the fixed logic serves the exact name the binary
requests for both variants; the gating smoke Phase 14 is the regression guard.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Closes#423, #20. A non-ASCII repo path passed to
`codebase-memory-mcp cli index_repository "<json>"` was mangled on Windows, so the
command failed with "repo_path is required" instead of indexing the real directory.
POSIX is unaffected (argv is already UTF-8). Two boundaries had to be fixed:
1. argv read: main() took only the narrow `int main(int argc, char **argv)`, so the
CRT handed it argv in the active ANSI code page. On Windows main() now rebuilds
argv from the wide command line (GetCommandLineW + CommandLineToArgvW) and
converts each element to UTF-8 (cbm_wide_to_utf8).
2. worker spawn: the index supervisor (and the UI index thread) spawned the worker
via CreateProcessA, which re-interpreted the UTF-8 command line through the ANSI
code page -- re-mangling the path at the parent->worker boundary even after (1).
Both spawns now build a wide command line (cbm_utf8_to_wide) and use CreateProcessW.
Without (2) the default (supervisor-enabled) path stayed broken while only the
in-process path worked, so this is the complete end-to-end fix, not just argv.
Promotes tests/windows/test_cli_non_ascii_arg.py from an opt-in known-red to a green
Windows guard (scripts/test-windows.ps1 + the test-windows-guards CI job). The guard
now exercises the DEFAULT supervised path (it drops the suite's CBM_INDEX_SUPERVISOR=0
so it actually crosses the worker-spawn boundary). Verified on native Windows: RED
before the fix, GREEN after (a non-ASCII repo indexes, nodes>0). Also adds -lshell32
explicitly (main.c now uses CommandLineToArgvW) and refreshes RED_TEST_ANALYSIS.md.
Signed-off-by: Flipper <jacobphilipp@ymail.com>
PR CI ran tests but never built the binary or smoke-tested it, so built-binary
regressions only surfaced at the release dry run — e.g. the Windows
CreateProcess argv-quoting bug (worker got mangled JSON) slipped all the way to
the release gate.
Add a path-gated light smoke: on PRs that touch product code (src/, internal/,
Makefile.cbm, build.sh, smoke-test.sh, env.sh), build the PRODUCTION binary and
run scripts/smoke-test.sh on ubuntu-latest + macos-14 + windows-latest (native,
reliable — not the flaky broad/emulated legs). Every leg gates. Docs/CI/test PRs
skip it and stay fast (gh pr diff --name-only filter). The full broad/emulated
smoke stays in the release dry run.
SMOKE_DOWNLOAD_URL is intentionally unset so smoke-test.sh's download/update
phases self-skip; the core index/search/trace phases (which catch the
CreateProcess class) still run. ci-ok now treats skipped as OK so PRs that skip
smoke still pass.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two defects let the release dry run report green despite real smoke failures:
1. Matrix collapse — the smoke matrices used {variant:[standard,ui],
include:$OS_LIST}. GitHub's include-merge overwrites the variant combos
last-wins with each OS entry, so only the LAST OS ran (both variants) and the
earlier ones were silently dropped. In broad mode windows smoke only ran
windows-11-arm (emulated x86_64), never windows-latest (native); unix only
macos-15; portable only ubuntu-22.04-arm. Rebuild each matrix as an explicit
os x variant cartesian so every leg actually runs.
2. Optional escape hatch — the broad legs carried optional:true +
continue-on-error: matrix.optional, so a failing leg didn't fail _smoke.yml
and didn't gate the dry run or a release. Removed all optional flags + the
continue-on-error lines; every product-smoke leg now gates.
smoke-packages stays continue-on-error (external Glama directory image, not the
shipped product). fail-fast:false kept so all legs still run for a full picture.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The release dry run must be all-green on every platform — drop the
optional/continue-on-error flags from the broad matrix legs (ubuntu-22.04,
ubuntu-22.04-arm, macos-15, windows-2025) so a failure on any of them gates
the run. Only affects broad_platforms=true (the dry run); the core PR-CI
matrix is unchanged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
libgit2 is licensed GPLv2-with-linking-exception, and this project is
deliberately GPL-free (see scripts/license-policy.json). libgit2 was only
an OPTIONAL, faster git-history code path guarded by HAVE_LIBGIT2 and
auto-detected via pkg-config; it always shipped with a popen("git log ...")
fallback that release binaries already used (they were built without
libgit2). Making that popen fallback the SOLE git-history implementation
means there is NO change to shipped behavior.
Changes:
- pass_githistory.c: collapse the HAVE_LIBGIT2 #ifdef so only the popen
parse_git_log remains; drop the now-unused <git2.h>/<time.h> includes.
- cbm.c / cbm.h / main.c / subprocess.c / index_supervisor.h: remove the
libgit2 mimalloc allocator bind and its >=1.7.0 version guards; keep the
tree-sitter + sqlite3 binds; correct the comments.
- Makefile.cbm: remove the optional-libgit2 pkg-config detection block,
the REQUIRE_LIBGIT2 error gate, and all LIBGIT2 CFLAGS/LIBS/FLAGS.
- .github/workflows/_test.yml: drop the REQUIRE_LIBGIT2 test-matrix leg
plus its libgit2-dev/pkg-config apt install (lower CI cost, no new
gating, no trigger change).
- flake.nix: drop libgit2/pkg-config from the dev shell (now-dead dep).
HAVE_LIBGIT2 no longer exists anywhere in the tree.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The SBOM hardcoded '158-grammars' for the tree-sitter-grammars aggregate;
the true count is now 159 (Mojo + InterSystems ObjectScript vendored, nim
dropped since the literal was last set), so the shipped SBOM under-counted.
Compute the count from the vendored grammar directories so it cannot drift
again. Also correct the non-MIT summary, which named only clojure CC0-1.0 +
jinja2/just Apache-2.0 while omitting fennel (CC0), the other eight Apache
grammars, fish (Unlicense) and pine/templ (ISC); it now lists the full
non-MIT family set and is explicitly non-exhaustive, deferring to
MANIFEST.md as authoritative. Data-only change to generated sbom.json; no
build or gating behavior changes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Rebased onto current main and reworked in response to review. Three of the four
Windows reds were fixed upstream since the branch was cut at b075f05, so they are
now green regression guards; the fourth stays a genuine known-red.
- test_non_ascii_path.py (#636/#357): green guard - fixed by #700 (per-pass
readers now route through cbm_fopen -> _wfopen). Re-verified green on main.
- test_hook_augment.py (#618): green guard - fixed by #619 (cbm_is_walkable_abs_path
accepts drive-letter X:/ cwd). Re-verified green on main.
- test_ui_drive_listing.py (#548): rewritten. The fix exposes drives via a new
roots[] field, not the dirs[] array the old test asserted (which would stay red
against fixed code). Now asserts every fixed drive is in roots and browsable.
Re-verified green on main (drives C:/D:/E:).
- test_cli_non_ascii_arg.py (#423/#20): unchanged - main() is still narrow-argv
with no wide command line, so this remains genuinely red (the keeper).
- scripts/test-windows.ps1: split green guards (gate CI) from opt-in known-reds;
add -GuardsOnly; run indexing in-process (CBM_INDEX_SUPERVISOR=0) so a guard
reflects the path/hook/drive fix under test, not the index-worker spawn path.
- .github/workflows/_test.yml: new test-windows-guards job builds the product+UI
binary (scripts/build.sh --with-ui) and runs the guards with -GuardsOnly so
#700/#619/#548 stay enforced on Windows CI.
- RED_TEST_ANALYSIS.md: refreshed to record the landed fixes and current status.
Signed-off-by: Flipper <jacobphilipp@ymail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#581 (query-path memory growth) does not reproduce in our soak: a 10-min,
200-file query-only run shows RSS flat-to-shrinking and committed plateauing
(mimalloc already purges on free, purge_delay=0). Rather than ship speculative
mitigations for a leak we cannot measure, this adds the observability to find it
where it actually occurs, plus a continuous guard.
- diag: the diagnostics writer now appends a PERSISTENT NDJSON memory trajectory
(one sample / 5s: rss, committed, peak fields, page_faults, fd, queries), kept
on exit (rotates to a .1 generation past 8 MB) so users can send the time
series post-mortem. The previous latest-snapshot file was overwritten every 5s
and deleted on stop.
- ci: the soak now also runs a read-only query-leak leg (CBM_SOAK_MODE query-leak,
no reindex/mutate) on every platform including Windows, so a regression that
introduces a query-path leak is caught. soak-test.sh RESULTS_DIR is now
env-overridable so both legs keep separate artifacts.
- docs: README Troubleshooting and Diagnostics section + a bug-report field
explain the no-telemetry stance and how to capture and share the trajectory.
Build clean; unit suite 5714/0; trajectory verified to persist post-exit.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The ubuntu-24.04-arm test leg (native ARM64 Linux running the full suite under
AddressSanitizer + the FastAPI incremental index) exceeds the 60-minute job
timeout and was killed, failing ci-ok even though the suite was on track to pass.
ARM + ASan is legitimately 2-3x slower than x86-64.
Raise the substantial compute jobs (test, build, bug-repro board, soak's short
legs, security/codeql, cross-platform smoke, fast-repro) to a generous 240-minute
(4h) cap so a slow-but-correct runner can't false-timeout. Trivial aggregator
jobs (5/10/15m: setup-matrix, ci-ok, dco, lint, license-gate, quick smokes) and
the intentional soak caps (300/320m) are unchanged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The native CLANGARM64 toolchain compiles the project fine, but linking the test
build fails: LLVM ships no AddressSanitizer runtime for aarch64-w64-windows-gnu
(libclang_rt.asan_dynamic.dll.a not found). ASan also cannot intercept under
x86-64 emulation, so Windows ARM64 has no working ASan by any toolchain.
Run the windows-11-arm leg with SANITIZE= (no sanitizer) so it builds and runs
the full suite natively as a real, required functional gate. ASan/UBSan coverage
is provided by the other nine legs, including native-ARM Linux/macOS; x86-64
Windows keeps full sanitizers.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
windows-11-arm is an ARM64 runner but the test matrix built it with the x86-64
msys2 environment (CLANG64), so the binary ran under Windows-on-ARM x86-64
emulation. AddressSanitizer's function interception cannot patch the emulated
system DLLs there (interception_win: unhandled instruction -> asan_malloc_win.cpp
CHECK failed), crashing in ASan init before any test ran. That was being hidden
behind optional:true.
Pin each Windows leg's msys2 environment + package arch to the RUNNER
architecture: x86-64 runners use CLANG64 (mingw-w64-clang-x86_64-*), the ARM64
runner uses CLANGARM64 (mingw-w64-clang-aarch64-*). The windows-11-arm binary is
now native ARM64, so ASan instruments native code and the leg is a real,
non-optional gate (matching smoke.yml, which already uses CLANGARM64 for it).
Every other matrix leg already used a toolchain matching its runner arch.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel