Two defects, one found by the other.
The crash: every fixture-indexing suite (mcp, incremental,
index_resilience) died on the Windows/ARM64 leg with
STATUS_ILLEGAL_INSTRUCTION and no diagnostic. Cause: an earlier commit
in this branch re-armed UBSan's alignment check on the vendored TRE
regex engine, and CLANGARM64 builds with -fsanitize-trap, where a trap
IS an illegal instruction with nothing printed. macOS and Linux stay
clean under the same check because Windows is LLP64 -- 32-bit long --
so TRE's struct layouts and access widths differ there and only there.
TRE is vendored third-party code we do not modify, so suppressing the
check for that one object is the honest scope; every other object keeps
alignment armed. Verified by bisect: origin/main, e2f5b6a and 80afcd6
all pass on Windows; the sanitizer-matrix commit that dropped the
suppression is where it starts failing.
The reason it took a bisect: BUILD_CONFIG_SIG covered TEST_SEAMS and
CFLAGS_EXTRA but not SANITIZE, so changing sanitizer flags did not
trigger a rebuild. `scripts/test.sh SANITIZE= --suites ...` -- the
documented way to get a plain build for exactly this kind of trap
debugging -- silently re-ran the previously instrumented binary. That
produced a "it crashes without sanitizers too" reading that was pure
artifact and cost several probes down the wrong path. SANITIZE now
participates in the signature, so a sanitizer-only change rebuilds.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The closure route stops loading and rewriting the world. Its executor is
now a dedicated subsystem (pipeline_delta.c + orchestration): CLONE the
live generation (copy-on-write where the filesystem offers it), repair
the closure against the clone, PATCH exactly the repaired node/edge set
in one transaction, and publish through the same sealed-staging finalize
leg as the dump path. No full graph load, no full dump, and the general
indexing pipeline is untouched -- the profiled kernel run put those two
at 187s of a 238s one-file repair whose actual resolution work was 0.6s.
Id discipline carries the design. Node ids are AUTOINCREMENT and never
reused; the small in-RAM graph is pre-seeded with PROXY nodes carrying
their real database ids (SELECT ... ORDER BY id with the id watermark
pinned before each insert), and fresh nodes are numbered above the
previous generation's MAX(id) -- so "id > max_db_id" is the complete,
marker-free definition of what the patch inserts, and every edge
endpoint id is database-valid by construction. The inbound-edge snapshot
and its QN-keyed re-link become indexed SQL; a re-link whose endpoint no
longer exists matches no row, which is full-reindex semantics for
deleted symbols.
Fail-closed throughout: an unexpected reference to an unseeded label
surfaces as a UNIQUE-constraint violation that fails the patch
transaction, and EVERY delta failure discards the stage and returns
FORCE_FULL_REINDEX -- the live database is never touched, so a full
rebuild always self-heals whatever the delta could not do.
FTS policy: nodes_fts is contentless, so purged rows cannot be deleted
individually on existing databases; their rowids can never alias a live
node again (AUTOINCREMENT) and dead entries drop out of the rowid join.
The patch inserts rows for exactly the new nodes through the same
cbm_camel_split tokenizer the wholesale rebuild uses.
The legacy gbuf-based tail reverts to serving only the test-only
force_legacy_partial route; the closure orchestration owns its own
coverage merge, publication race gate, surface-row merge and committed
counts, and publishes with fts_wholesale=false.
Gate: the full convergence suite runs against this executor unchanged --
body-edit graph equality with a fresh full index, removed-definition
dropping the dependent's stale edge, tsconfig-alias retargeting, the
decline matrix, and 510 pipeline/incremental/store/integration tests.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Second piece of closure repair. At the collect_all_defs seam -- the only
moment the per-file result cache is alive -- both drivers (parallel and
sequential) now serialize each file's CBMLSPDef slice to canonical JSON,
hash it, and hand the rows to the pipeline; cbm_pipeline_publish_generation
writes them into the staging store next to the manifest, so surface data
and graph always belong to the same generation.
Canonical bytes are the point: every field is written in fixed order with
an explicit null for absent strings (NULL and "" differ in the CBMLSPDef
contract -- receiver_type NULL means "not a method"), so byte equality IS
surface equality and the sha over the bytes is the early-cutoff key.
Registry-only labels that pxc_map_label drops but the name registry serves
(Field) are folded into the hash as a separate "reg" array, or renaming
one would slip past the cutoff.
Behaviour pinned in SUITE(pipeline): a fresh full index persists a
versioned surface row per file; a BODY edit republishes the identical
surface_sha; a SIGNATURE edit changes it. That pair of properties is what
the routing layer will stand on.
cbm_pxc_collect_all_defs gains an optional per-file prefix array -- the
flat all_defs[] otherwise loses the file boundaries the serializer needs.
CORRECTION to 6c22338's scope note: it claimed cbm_pipeline_publish_
generation was reachable only behind CBM_INCREMENTAL_TEST_API. Wrong --
dump_and_persist_hashes calls it on every production full index
(pipeline.c:1863); the grep that "verified" test-only reachability had
excluded pipeline.c itself. The predictable staging name WAS in the
production publish path, which makes that fix a real production hardening,
not a test-path cleanup.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The previous fix put -include stdint.h on the lane's global SANITIZE
line. That traded one vendored compile break for another: force-including
a libc header ahead of every source file freezes glibc's feature-test
macros before sqlite3.c can set _GNU_SOURCE for itself, and its view of
libc loses MREMAP_MAYMOVE and nanosleep (17 errors on the x86-64 CI
leg).
The workaround only ever had one legitimate target -- the zstd
amalgamation whose MEMORY_SANITIZER block lost its stdint re-include --
so it now rides a per-object hook (ZSTD_EXTRA_CFLAGS) that the MSan lane
sets and every other build leaves empty. sqlite3.c compiles exactly as
before in every lane.
Verified locally that zstd compiles with the hook and the default rule
stays untouched; the MSan leg itself is x86-64-only, so CI is its
verification venue.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
MSan: vendored zstd fails to compile. Its MSan-only block (guarded by
MEMORY_SANITIZER) declares __msan_test_shadow returning intptr_t and
reaches for the type with
#define ZSTD_DEPS_NEED_STDINT
#include "zstd_deps.h"
but the amalgamator that produced zstd.c collapsed that second include
into a "skipping file" comment, so the define pulls nothing in and
intptr_t is undeclared. Only this lane compiles that block at all, and
only where <stddef.h> does not drag stdint.h in transitively -- which is
why it built on the local aarch64 container and failed on CI's x86-64.
The lane now forces the header. Patching the vendored amalgamation would
be silently undone by the next re-vendor.
diag: detect_invalid_pointer_pairs comes back out. It fires during static
initialisation inside vendored simplecpp -- a std::string global at
simplecpp.cpp:101 -- with a second "pointer" of 0xfffffffffffffff3, a
sentinel rather than an address: libstdc++ string internals, not
anything this codebase wrote. It is a process-wide runtime flag with no
per-file scoping, so unlike the analyzer's path filter it cannot be
aimed away from vendored code. Keeping it would mean a permanently red
lane reporting a non-defect, which is how a lane gets ignored. The
instrumentation it needed comes out with it.
The other three off-by-default checks stay: stack-use-after-return,
stack-use-after-scope, strict-string-checks. Those are the ones covering
bug classes nothing else in the matrix looks for, and none of them
fired.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The memory gate stays gating, and it now has a way to record a genuine
false positive that cannot quietly outlive its own reasoning.
A whitelist entry names one (file, function, check) and carries two
things: why the analyzer is wrong, argued from the code, and what was
tried before concluding that. The entry is pinned to the sha256 of that
function's text. Edit the function and the entry stops counting -- the
finding comes back and has to be argued again against the code as it now
is. This is the part that matters. A suppression that survives the code
it was written about reads as "reviewed" while being nothing of the
kind, which is worse than no suppression at all.
The gate fails on: a finding with no entry, a finding whose entry has
gone stale, and an entry that asserts rather than argues (there is a
floor on how much reasoning an entry must actually contain -- the
mechanical half of "argued, not asserted"; whether the argument is
correct stays a review question). An entry that matches no finding is
reported but does not fail, because analyzer versions differ across
platforms.
NOLINT is still not honoured here and the gate does not read it.
The whitelist ships empty: the analyzer is currently clean across
LINT_SRCS, so nothing is being suppressed today. The mechanism exists
for the first finding that genuinely warrants it.
Verified by exercising each path rather than only the clean one: an
unaccounted finding fails and names its enclosing function; an argued
entry with a matching hash passes; the same entry fails as STALE once
the hash no longer matches the function; and an entry that says only
"false positive" fails for not arguing its case.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Every lane here was either asserting coverage it did not have, or was
reachable only by bypassing the entry point that is supposed to define
the ladder.
TSan: no exclusions any more. The three suites the Makefile documented
as excluded are back. daemon_ipc and daemon_frontend no longer reproduce
the harness race and thread leaks they were excluded for. daemon_runtime
did not deadlock as the comment claimed -- it reported a real production
data race on the log sink, fixed separately. Excluding a suite from a
sanitizer lane hides exactly the class of bug the lane exists to find,
so the comment block now records what was actually true rather than what
was assumed.
TSAN_OPTIONS gains report_thread_leaks=0. This disables the thread-
HYGIENE check only; race detection is untouched. Several daemon fixtures
fork after the process has gone multi-threaded, and in the forked child
TSan sees the parent's already-finished threads as never-joined even
where the fixture joins them. It fires on macOS and not Linux, i.e. it
tracks fork semantics rather than anything about this code. The
alternative was dropping whole suites, which costs real race coverage;
this costs none.
UBSan: tre.o no longer builds with -fno-sanitize=alignment. Alignment
was switched off for a vendored regex engine that ships in the product,
which is where the check is least redundant, not most.
LSan on macOS: new test-lsan target and test-lsan-macos CI leg. LSan is
on by default under ASan on Linux, so the Linux legs have always had
leak coverage. On macOS it is off by default and Apple's clang refuses
to enable it outright, so that platform had none at all. Apple's refusal
is not a darwin limitation -- upstream LLVM supports LSan on darwin/
arm64. The lane is the ordinary ASan suite built with Homebrew LLVM and
run with detect_leaks=1; it runs the full suite clean and was checked to
still catch a deliberately leaked allocation.
MSan: reachable from the local ladder. The image and compose service
existed but run.sh had no leg, so the only way in was to drive docker
compose by hand -- which means it was not part of the ladder in any
meaningful sense. The image also moves to clang 22, matching the diag
and analyzer lanes instead of sitting four majors behind on noble's
default. The leg documents the aarch64 shadow-mapping failure so a local
arm64 stack overflow in the grammar suites is not mistaken for a code
defect; the GitHub leg runs x86-64, which is the mapping that matters.
Off-by-default ASan checks: the diagnostic lane, and its CI twin, now run
detect_stack_use_after_return, detect_stack_use_after_scope,
detect_invalid_pointer_pairs (with the -fsanitize=pointer-compare,
pointer-subtract instrumentation it requires) and strict_string_checks.
Running ASan is not the same as running all of it, and these four cover
bug classes nothing in the matrix was looking for. They stay on the
diagnostic lane rather than the gating ones until they have a clean
history there; promoting them is a separate deliberate step.
Verified: macOS TSan 940 passed / 3 skipped / 0 races over the full
suite set; the macOS leak lane 7375 passed / 4 skipped / 0 leaks, with
LeakSanitizer confirmed armed under that exact toolchain and option set
by checking it still reports a deliberately leaked allocation.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Stage 2 of the memory-diagnostics program (user decision: go directly to the
instrumented image rather than a C-only probe). MSan detects uninitialized
READS, the one memory-error class no other lane covers dynamically, and it
requires every linked library to be instrumented -- vendored C deps compile
in-tree and instrument for free; the two external links do not:
- test-infrastructure/Dockerfile.msan: pinned-base image building
libc++/libc++abi/libunwind (llvmorg-18.1.8, LLVM_USE_SANITIZER=
MemoryWithOrigins) and static zlib v1.3.1 into /opt/msan, with the
symbolizer and MSan runtime in a separate last layer so tool additions
never invalidate the ~30-min libc++ build.
- scripts/msan.sh: the canonical lane entry. ALWAYS clean-builds its
BUILD_DIR: make does not encode flags into dependencies, and a stage-1
probe's libstdc++ objects surviving into the libc++ lane produced a
convincing-looking uninitialized-value report at preprocessor.cpp:168 --
the uninstrumented .so string constructor wrote the temporary, the
instrumented move constructor read it. The clean rebuild proved it an
artifact: extraction (incl. the C++ preprocessing path) runs 272/272 with
zero reports.
- Makefile.cbm: CXX_STDLIB / CXX_STDLIB_FLAGS hooks so the lane can swap
libstdc++ for the instrumented libc++ (defaults identical; the shipping
build is byte-for-byte unaffected).
- docker-compose test-msan service: same aarch64 seccomp/setarch remedy as
the TSan service (MSan's shadow layout hits the same personality() block).
- CI test-msan job (_test.yml): buildx local-cache via the repo's existing
pinned actions/cache -- no new third-party action pins; a warm run skips
the libc++ build entirely.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The memory-diagnostics report's priority-4 lane (path-sensitive clang-analyzer,
memory checks only) run over all 111 production files. 21 findings triaged;
the real ones, all cold-path (none can explain #581's per-query residual):
LEAKS
- mcp get_architecture: scope_path leaked on the missing-store early return
(REQUIRE_STORE frees only `project`); allocate after the gate.
- pass_definitions: cancellation mid-extraction leaked the pass-owned result
cache including already-extracted entries; mirror the end-of-pass cleanup.
- store package-boundary scan: the row-scan abort path freed the node arrays
but not the boundary accumulators or their duplicated package strings.
- cbm quarantine set: a duplicate path line leaked the replaced value (and a
fresh key copy -- the table borrows key pointers); a partial strdup failure
leaked the surviving half. Reuse the stored key for duplicates.
- pass_githistory: unchecked malloc/strdup -- an OOM dereferenced NULL and a
failed strdup leaked the index cell. Allocate before claiming the slot.
NULL/UB
- cli config subcommand: NULL argv with nonzero argc slipped the guard (the
inner `argv &&` shielded only the help comparison) into argv[0].
- store bfs_multi: a negative max_results broke out before any row was
written, then freed fields of an unwritten negative-index slot. Clamp.
- pass_calls emit_http_async_edge: the service-pattern call sites pass a NULL
target behind a hand-duplicated URL predicate; a drift between the copies
turned target->id into a null deref. The callee is now total.
- sqlite_writer: both leaf-array OOM paths left leaf_count stale with a NULL
array, walking pb_finalize_* into leaves[0]; consistent empty state routes
them to the existing root=0 failure return.
HARDENED (invariants true but invisible to path-sensitive analysis)
- Leiden CSR + aggregate arrays, SCC adjacency: calloc + endpoint guards, so
a future degree/collection miscount degrades benignly instead of UB.
- SCC cycle fill: the ncyc==0 no-slot invariant made local.
RECORDED FALSE POSITIVES (no code change)
- yaml sequence starts (loop bound == alloc bound), cypher agg arrays (same
count both sides), mcp read_message ch (assigned by fgetc each iteration),
pkgmap clean buffer, mcp csize (Tarjan: ncomp>=1 when nverts>=1), vendored
verstable x2.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
GATE + LANES (user decision: runner cost accepted)
- make lint-mem (local triage) and lint-mem-ci (gating: vendored-filtered,
any remaining finding fails). The gate is green because every false
positive above was restructured for provability -- calloc'd fill-cursor
arrays, explicit Tarjan invariant, zeroed buffer tails, min-1-element
allocations -- never suppressed.
- make diag: pinned newest-LLVM ASan/UBSan lane with straighter stacks.
- CI: lint-mem job (_lint.yml) and test-diag job (_test.yml), both on the
pinned LLVM 22 apt toolchain. Cost disclosure: roughly +25-40 min and
+25-60 min (ccache-warm) per push respectively.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The branch adds internal/cbm/lsp/type_registry.h, and PROJECT_HDRS covered
$(CBM_DIR)/*.h but not the lsp/ subdirectory -- so editing an lsp header
rebuilt nothing and an incremental build silently tested stale header values.
The branch's own invariant (repro_make_tracks_headers.sh) caught it on the
repro-unix CI jobs; the same trap class as the lsp_all.o one fixed by #662.
bash tests/repro/repro_make_tracks_headers.sh: green.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
pi has no MCP client, and OpenCode has no declarative hook configuration --
verified against OpenCode's own plugin documentation, which states hooks are
available only through JavaScript/TypeScript plugin modules. For those two
clients a module is the only extension point.
We do not want to ship such a module as a repository asset. #534 proposed one
embedding 380 lines of TypeScript as C string literals and registering 7 of the
15 registry tools; every tool added afterwards would have been silently missing
for that client. Hand-maintained copies of the tool list have already produced
defects here (#1361, and the smoke-invariants count).
So the module is GENERATED from the live registry instead. cbm_client_adapter_pi
walks cbm_mcp_tool_count()/cbm_mcp_tool_name() and registers every tool, which
makes drift structurally impossible rather than merely discouraged: adding a
tool to TOOLS[] adds it to every generated adapter with no second edit. Nothing
in the repository is a .ts file.
Both emitters wrap their output in ownership markers so a caller can rewrite its
own block and leave a user-authored file alone.
Two defects from the prior proposals are closed by construction:
- Path escaping. #616's template rejected the double quote but not the
backslash, so a Windows home like C:\Users\urs\bin produced an invalid
unicode escape and the whole auto-loaded plugin failed to parse, which is
worse than an absent plugin. cbm_client_adapter_escape_js escapes backslash,
quote, newline and CR, and fails closed rather than emitting a truncated
literal; generation aborts if the path cannot be escaped.
- The silent no-op. #616's payload omitted hook_event_name, which hook-augment
requires and without which it accepts nothing, so the plugin emitted zero
bytes for six weeks. The generated payload carries it and a test pins it.
Recorded honestly: the OpenCode emitter hooks tool.execute.after, whose ability
to modify a tool's output is NOT part of OpenCode's documented plugin contract
(only tool.execute.before's argument mutation is). If they change it the
augmentation stops with no error. That risk is accepted deliberately and the
emitter says so in a comment, so a future reader does not have to rediscover it.
This commit adds the generator and its tests only; wiring into the install and
uninstall routines follows separately so the two are reviewable apart.
Tests: every registry tool appears in the pi module (revert-checked by
simulating the 7-of-15 subset, which fails); Windows/quote/newline escaping and
its truncation boundary; the OpenCode payload carries hook_event_name and
registers no tools; NULL/empty binary paths generate nothing.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
test-runner, test-repro-runner, test-runner-tsan, codebase-memory-mcp and
test-foundation are each built in a single compiler invocation from their
sources, and their rules listed only .c files as prerequisites. Editing a
header therefore changed nothing make could see, so the target was not
rebuilt and an incremental build silently kept testing the OLD header value.
Found while revert-checking the label-allowlist change in this branch: the
check passed, which was wrong. Shrinking CBM_SQL_TYPE_LIKE_LABELS should have
turned the pin test red and did not, because nothing had recompiled. Forcing a
rebuild produced the expected failure. A revert-check that cannot fail is worse
than no revert-check, because it reports confidence it did not earn.
This is the same class as the lsp_all.o staleness that #662 fixed with an
explicit dep list; that fix was per-object, this one covers the whole-program
targets.
PROJECT_HDRS globs src/, internal/cbm/ and tests/ headers and is added to the
five affected targets. Vendored trees are deliberately excluded: they are
pinned, they already carry explicit dep lists (LSP_UNITY_DEPS,
TS_RUNTIME_DEPS), and globbing tens of thousands of grammar headers would cost
more than it protects.
Verified: editing only src/foundation/constants.h, with no source file touched,
now rebuilds and turns the pin test red -- the exact edit that was invisible to
make before.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Microsoft's ML flagged the rc.1 release binaries. The decisive evidence is that
the SAME sha256 went from 0/62 clean to Microsoft-detected in about an hour with
no byte change, so the verdict lives partly in scanner-side state and no code
change can promise a clean result. What code CAN do is stop shipping things that
have no business in a release artifact, which is worth doing on its own merits
and incidentally widens the classifier margin. Every claim below is verified
against a built binary by the new gate, not by reading source.
Executable stack (the worst of the findings). vendored/nomic/code_vectors_blob.S
is the only assembly in the build and carried no .note.GNU-stack. An unannotated
object makes ld assume the worst for the whole link, so EVERY Linux release we
have ever shipped had GNU_STACK RWE. Adds the note (cause) plus ELF-only
-Wl,-z,noexecstack (outcome); the gate fails the release if it returns.
Test seams are now opt-in, never opt-out. TEST_SEAMS=1 defines
CBM_ENABLE_TEST_SEAMS; without it the crash-orphan probe -- which forks a child
that ignores SIGTERM and loops forever, then writes its pid to a caller-supplied
path -- and the lease-ownership marker compile to trivial stubs, so call sites are
untouched and the binary holds no fork, no signal handler and no env-var string.
Opt-IN is the point: forgetting the flag yields a clean binary rather than a leaky
one. scripts/test.sh requests it in the leg that consumes it, and
tests/test_worker_watchdog.sh now asserts the capability up front instead of
dying later with an opaque "Killed: 9".
The daemon's background version check is gone. It spawned curl against
api.github.com/repos/.../releases/latest on the first eligible session of every
run to say "a newer version exists" -- a release URL and an outbound request in
every shipped binary, for something the install scripts already report. The
INJECTABLE SEAM survives: update_ops is still honoured, the fakes in
tests/test_daemon_application.c still cover notice/ownership/cancellation/replay,
and with no provider application_update_subscribe_locked returns early so no
generation ever starts. "No network request by default" is now structural.
Dead capability out of release builds. The tar.gz/zip extraction block
(gzip_decompress through cbm_extract_binary_from_zip, plus its cli.h
declarations) moves under CBM_CLI_ENABLE_TEST_API -- verified self-contained, zero
uses of any helper outside it, only callers the excluded updater and
tests/test_cli.c. Downloading an archive, decompressing it, picking an executable
out of it and marking it executable is the canonical dropper composite; it is now
absent rather than merely unreachable. SQLite is built with
-DSQLITE_OMIT_LOAD_EXTENSION (no caller of load_extension anywhere in src/ or
internal/), removing that API surface and part of the dlopen/dlsym surface.
Temp files and environment scanning (S2/S3). Predictable paths in mcp.c,
artifact.c and diagnostics.c are created privately and exclusively and written
through the returned descriptor; pass_envscan.c no longer descends symlinked
directories out of the project root, and its fixed 512-byte path buffers no
longer truncate into pointer arithmetic that could land outside the buffer.
Build-time entropy. mimalloc's version banner baked __DATE__/__TIME__ into every
binary, so two builds of identical source seconds apart could never share a hash
and no release could inherit a false-positive determination made about its
predecessor. Local patch removes it (marked to survive refreshes), -Wdate-time
makes any future use a build error, and -Wl,--no-insert-timestamp stops the PE
header carrying the link clock.
scripts/ci/check-binary-composition.sh is the proof that each removal stays
removed, wired into package-release.sh after strip so the local artifact-flow
smoke enforces exactly what CI does. It asserts absences plus a CANARY string, so
handing it a compressed, stubbed or empty file fails instead of passing
vacuously, and a missing tool is a hard error -- a skipped assertion must never
look like a satisfied one.
Two build-system traps found by that gate, both of which had silently defeated a
fix: the product binary is compiled in one shot from sources, so a flag flip did
not rebuild it (now tracked by a .build-config stamp that also removes the
binary, making it independent of mtime granularity); and prod_sqlite3.o /
prod_mimalloc.o depended on a single named source, so SQLITE_OMIT_LOAD_EXTENSION
and the mimalloc patch BOTH compiled to nothing on the first incremental build.
Source review would have called them done.
Deliberately NOT changed. Three seams stay in release artifacts because
scripts/smoke-test.sh runs against the real artifact and needs them:
CBM_TEST_CRASH_ON and CBM_TEST_HANG_ON inject the faults that prove supervisor
recovery, and CBM_TEST_WINDOWS_USER_PATH_RUN_ID is what keeps the PATH smoke from
writing the tester's actual PATH. The gate treats those as an allowlist, so a
NOVEL seam still fails. The true no-UI standard build is deferred rather than
rushed: src/ui/* is in PROD_SRCS and four files outside src/ui reference UI
symbols, including the daemon that serves the UI, so that assertion reports
instead of failing until the split lands -- a gate everyone knows is red teaches
people to ignore gates.
No grammar is removed. ObjectScript accounts for essentially all binary growth
since the last provably-clean release (+21.4MB rodata, +1.1MB text from two
four-line shims), which made it the obvious ablation candidate, but a dry run
performed twelve real Defender endpoint scans across standard/UI and amd64/arm64
with ObjectScript, the daemon and the expanded hooks all present and every scan
was clean. Nothing there is a deterministic trigger, so cutting a
community-contributed language would spend a real feature on unproven margin.
Lean is not a candidate either: at 99.6MB of source it is by far the largest
grammar, but it shipped in v0.9.0 which scanned 20/20 clean, so removing it would
produce a novel unscanned profile instead of restoring a known-good one.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Windows shipped a PAIR: a small permanent launcher (codebase-memory-mcp.exe)
plus the real product binary (codebase-memory-mcp.payload.exe). The launcher
existed for exactly one reason — a running .exe cannot replace its own image
on Windows, so an in-process self-update needs a second resident binary to do
the swap.
That stub is statically indistinguishable from a dropper: a small, unsigned,
zero-prevalence PE whose whole job is verify-and-execute another binary.
Defender's ML scored it Trojan:Win32/Wacatac.B!ml and blocked the v0.9.1-rc.1
release at the VirusTotal gate. It is not fixable in our code on x64 —
bcrypt-free, stripped, VERSIONINFO'd, minimal-resource and even
resource-FREE builds on CI's own MSYS2 CLANG64 toolchain were all flagged,
while the product binary scans clean on every platform.
So remove the stub and move self-update OUT of the process into install.ps1,
which runs while cbm is NOT running: Windows' image lock only blocks a
process from replacing ITSELF. now prints the exact PowerShell
command (with the Unblock-File hint for Mark-of-the-Web); install.ps1 is
idempotent, so re-running it IS the update — it stops the daemon, renames the
running binary aside (the one mutation Windows permits on a running image),
publishes the new one, and sweeps retired copies.
Windows now matches Linux and macOS: ONE binary per platform.
* packaging, install.ps1, npm and PyPI wrappers all carry a single binary
* the launcher/payload ABI contract and ~2500 lines of stub state machinery
are deleted
* every daemon start, CLI call and hook fire loses a process spawn, a named
pipe handshake and an stdio relay
* test_windows_bundle_contract.sh is rewritten as an INVERTED contract: it
now asserts no shipped surface can reintroduce a launcher/payload pair,
and that install.ps1 retires the running binary before publishing
Verified: VirusTotal 0/67 on the packaged binary and 0/58 on install.ps1 (no
certificate involved); macOS and Linux full suites green; Windows guards all
green including the new update-handoff contract; npm 10/10; PyPI 3/3.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The hosted windows-11-arm shards crash at process load (STATUS_ILLEGAL_INSTRUCTION,
pass=0, secs=0) while main passes them, and removing the TLS detach callback did
not help -- so the remaining candidate is this layer, which touches every
allocation on Windows through the interposer. It declares a _Thread_local guard
read on every malloc, and thread-local storage in a static MinGW aarch64 link is
exactly the fragile case; nothing in it needs to ship.
The profiler did its job: it produced the attribution that identified #581's
mechanism, and that analysis is recorded. Sources stay in the tree, unbuilt, so
a future investigation can re-enable them deliberately.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The shim exists only so the allocation profiler has an observation point on
Linux. Applying its --wrap flags to the test and TSan link redirected malloc
into mimalloc underneath ASan's own interception, putting two allocators on the
same pointers. Production keeps the shim; sanitized builds do not. Windows is
unchanged -- there the wrap flags are what make mimalloc own allocations.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Fixed corpus, fixed request count, one process, no daemon handshake and no
reindex or crash-recovery phases, so two runs differ only where the code does.
Requests go through a request/response driver rather than a batched pipe:
closing stdin makes the server treat it as a client disconnect and shut down
before answering, which yields a census of a process that never worked.
Linux gains a measurement-only --wrap shim so the profiler can observe
allocations the way the Windows interposer already does; macOS has no --wrap
and stays census-only, which the driver reports rather than passing off as
zero.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The census could say which POOL held memory but never which allocation put it
there, so each hypothesis needed a code change plus an eight-minute soak to
test and the investigation could only ask one yes/no question at a time.
Records allocations at or above a threshold (default 32 KiB, since arena
fragmentation is driven by large transient runs) against their captured stack,
into fixed tables whose exhaustion is counted and reported rather than silently
dropped. Emitted alongside every census sample, so pool totals and attribution
always describe the same instant, plus an explicit unattributed remainder so
incomplete coverage announces itself.
scripts/memlab-report.py ranks sites by retained-byte growth using a linear fit
and can diff two platforms' profiles against each other.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The build system now stamps CBM_SANITIZED_BUILD for any non-empty SANITIZE,
so timing budgets key off build-system truth instead of compiler-specific
__SANITIZE_ADDRESS__ probes that miss trap-UBSan and TSan; the two bench
suites use it.
The httpd interrupt test proved 'the interrupt worked' with a wall-clock
bound, which is a lottery under sanitizers. A send-deadline test hook pins
the deadline out of reach so the join itself is the causality proof, and
the stop watchdog becomes a true ten-second hang detector.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Close the cross-platform sanitizer gaps that were leaving real
concurrency and undefined-behavior bugs uncaught, and fix a data race
the first widened run surfaced.
- tsan: the data-race gate ran three suites (mem, slab_alloc, parallel)
over no real threaded production code. It now covers every threaded
surface that runs clean and stable under TSan: the parallel-extraction
worker pool (parallel, worker_pool, pipeline), the filesystem watcher,
the embedded HTTP server (httpd), diagnostics sampling, the MCP server
and mutation guard, subprocess supervision, and the runnable
daemon-coordination paths (daemon, daemon_application). daemon_runtime
(deadlocks under TSan+fork), daemon_ipc and daemon_frontend
(test-harness synchronization, not production) are excluded with the
reasons recorded in the Makefile.
- tsan: fixed a genuine data race the widened gate immediately found —
cbm_lsp_max_walk_depth's lazy cache was read and written by parallel
LSP-extraction workers without synchronization. A data race is
undefined behavior even when every worker computes the same value, so
the cache slot is now a relaxed atomic: a plain load on the hot path,
and a first-touch double-compute simply stores the same value.
- tsan(ci + local): the test-tsan job now runs on Linux amd64, Linux
arm64, AND native ARM64 macOS (the threading code is shared, so a race
is usually caught on all three, but scheduler differences let each
surface one the others miss). The local ladder gained `run.sh tsan`
and `tsan-amd64` plus the matching compose services. TSan's shadow
memory aborts under modern high-entropy ASLR, so the containers run
under `setarch -R` with an unconfined seccomp profile (the personality
syscall is otherwise blocked) and the CI Linux legs lower
vm.mmap_rnd_bits first; amd64 TSan cannot run under x86_64-on-ARM
translation and is a real-hardware/CI gate only (documented in
run.sh).
- ubsan(win/arm64): native ARM64 Windows had no sanitizer at all —
AddressSanitizer ships no aarch64-w64-windows-gnu runtime. UBSan in
trap mode (-fsanitize-trap=undefined) needs no runtime library, so it
instruments natively and turns undefined behavior into an
illegal-instruction trap; -fstack-protector-strong adds stack-smash
coverage the heap tools miss. The GitHub windows-11-arm leg switches
from unsanitized to this, and vm/win.sh gains trap-ubsan-build /
trap-ubsan-test for local iteration (reproduce under the emulated
x86_64 UBSan to see which check fired). The whole codebase builds and
runs clean under it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Daemon lifecycle and Windows correctness, verified on a real Windows 11
ARM64 VM through the maintained test-infrastructure/vm drivers, plus the
macOS and Linux arm64 suites and the container lint gate.
Daemon lifecycle:
- daemon start/stop/status subcommands. `daemon start` launches a
PERMANENT daemon (spawn shape is byte-exact argv; survives idle
periods and session ends) and reports an already-active daemon
instead of failing. Permanence is honored at every stop latch:
last-committed-client disconnect, host initial-client window,
coordinator release, and application final-session close — a
permanent daemon also keeps admitting new sessions after its last
one closes.
- daemon stop refuses while sessions are active and lists the blocking
peers (pid/role) that must finish first; an idle daemon drains
through the activation-shutdown machinery with the ACK ordered after
connection interrupts. A second stop is idempotent. The wire ops are
no-cohort first-frame requests with peer fingerprint authentication,
so stop/status never conflict with an exact-build admission gate.
- One-shot CLI commands now execute through the daemon (index workers
keep their local supervised path). A cold CLI run that had to spawn a
temporary daemon prints a hint that `daemon start` removes the
per-command startup tax; a warm daemon is recycled silently.
- Hooks are connect-only fail-open: with no daemon present the hook
emits a visible, rate-limited notice (Claude-dialect systemMessage
plus stderr for other dialects) and always exits 0 — augmentation is
never allowed to block the caller's tool use.
- Version skew: a newer-build client automatically drains an
older-build permanent daemon (strict semantic-version triples only;
dev builds never auto-drain) and the build-conflict message names
`cbm daemon stop` as the manual escape hatch.
Windows IPC/runtime (real-VM verified):
- ipc(win): persistent pending overlapped ConnectNamedPipe. The accept
path used to destroy its listening pipe instance on every 20 ms poll
timeout; a client attaching in the teardown window was severed or left
on an orphaned pipe object whose HELLO no server handle could ever
read, absorbing the connect until the client's own timeout expired.
The pending connect now survives poll timeouts and nothing is
destroyed while a client could be attaching.
- ipc(win): drain-before-close for final responses. Closing a named-pipe
server handle can discard a just-sent response before the peer reads
it (POSIX stream sockets never lose buffered data on close). A bounded
cbm_daemon_ipc_connection_drain (read-until-peer-EOF; no-op on POSIX,
immediate on interrupted connections) now precedes close in
runtime_worker_finish and runtime_reject_inline, so hello-conflict,
capacity and disconnect acknowledgements reliably reach the peer.
- runtime: CLOSE_INTENT wire frame. A Windows named-pipe client has no
transport half-close, so close_begin now announces departure with an
explicit frame (ordered after APPLICATION_CANCEL, before the local
interrupt); the server releases the client's admission on receipt
instead of waiting for the handle to close. Admission-drop timing is
now identical to POSIX shutdown() semantics on every platform.
- runtime(win): client close cancellation. close_begin serializes with
request publication under the send lock, best-effort sends the active
token's APPLICATION_CANCEL frame, then interrupts local I/O; the
server cancels MCP/subprocess work promptly. Contract tests accept
both correct outcomes (interrupted transport or decoded CANCELLED).
- runtime: activation acknowledgement ordering. The activation ACK is
the requester's license to act on "snapshotted and draining", so every
connection interrupt is now initiated before the ACK is sent; a
session could previously get one more request serviced after the
requester observed the ACK.
- service(win): deadline-bounded private-file prepare. The conflict-log
prepare retry loop (100 x Sleep(2), which rounds up to the ~16 ms
timer granularity) burned ~1.6 s against permanently obstructed paths,
stalling hello rejections past the client's timeout. The retry budget
is now a 250 ms deadline; transient share collisions still retry.
- subprocess(win): cmd.exe /C payload encoder quotes metacharacters
correctly (root cause of the git-on-Windows failure cluster).
- watcher: SHA-256 buffer sizing (CBM_SZ_64 -> CBM_SZ_128) and a native
Windows stop/unwatch cancellation test with exact-image verification.
- httpd: send_all writes in bounded 64 KiB slices. A single giant
nonblocking send() on Windows is absorbed wholesale into AFD kernel
buffering regardless of SO_SNDBUF, so send deadlines and interrupts
could never engage against a slow peer (and the full payload was
pinned in nonpaged pool). Slicing restores a deterministic
backpressure point; a test hook pins SO_SNDBUF for the deadline and
interrupt tests.
- ui/http: shutdown lifecycle — interrupt checks, response-wide send
deadline, explicit connection states, refusal to free a server while
a listener-owned connection is active.
Windows long-path support:
- Central path-aware wide conversion (canonicalize via GetFullPathNameW
and prepend the extended-length prefix for absolute paths >=240) at
the compat chokepoints (cbm_fopen/compat_fs/mkstemp/mkdtemp), sqlite
store opens, and the daemon build-fingerprint/log paths. Deep managed
installs (a 64-hex generation directory routinely exceeds MAX_PATH)
now index, stage and activate correctly.
- activation transaction: its own file APIs and the component-walking
ancestry validators now operate in the extended-length namespace;
the launcher path is canonicalized (and prefixed when deep) once at
entry so every downstream exact-string comparison stays
form-consistent.
- Executable self-resolution uses the wide APIs (GetModuleFileNameW,
GetFileAttributesW) so non-ASCII install paths survive argv[0]
resolution.
Windows launcher install/uninstall transaction:
- FileRenameInfoEx names are NUL-terminated in an over-allocated
buffer. FileNameLength governs per the contract, but filter drivers
read FileName as NUL-terminated and appended adjacent heap bytes to
created names — a flaky, garbage-suffixed rename target. Both the CLI
and the launcher rename helpers are fixed.
- Uninstall retires state via rename-aside (.cbm ->
.cbm-retired-v1-<tag>-<pid>) with the retired tag shortened to 16 hex
chars so the bare rename target stays under the FileRenameInfoEx
NT-conversion ceiling at guard depths; 64 bits still uniquely
identify the generation.
- When the running launcher's mapped generation backings pin .cbm
against rename, the backings are relocated to activation-<pid>-N
.retired tombstones beside the install (a mapped image may be renamed,
never deleted; the launcher's liveness-guarded sweep reclaims stale
tombstones). Every relocation is recorded, and a FAILED uninstall
reverses the moves after restoring .cbm — via MoveFileExW with
extended-length paths on both arguments, since the deep generation
target is beyond the handle-based rename's bare-path reach — so a
restored install keeps its generation backings and stays runnable.
- After a committed uninstall the retired tree's backings are relocated
out so the tree is shallow enough for the detached cleanup's rd, and
the cleanup's working directory strips the extended-length prefix
(CreateProcessW lpCurrentDirectory silently ignores prefixed paths).
- Files created under Administrators-default-owner directories
(CopyFileW destinations, CREATE_NEW tombstones, probe directories)
are explicitly owner-stamped so the exact-owner validators hold on
runner images; guard fixtures stamp hand-built trees the same way.
Diagnostics, tests and infra:
- diagnostics: discovery is now an always-delivered JSON control record
(new cbm_log_control) that survives CBM_LOG_LEVEL suppression and
paths containing spaces; placement honors $TMPDIR with /tmp fallback
via a diagnostics-local helper; the soak parser reads the JSON record;
documented in docs/CONFIGURATION.md. Red-first coverage for suppressed
log levels, TMPDIR-with-spaces, and native Windows output-contract
assertions.
- tests(win): daemon_ipc/daemon_frontend fixtures now build endpoint
parents with production-shaped ancestry (LocalAppData on Windows, via
th_secure_runtime_parent_new) — the runtime ancestry validation
correctly refuses temp roots whose ancestors grant mutation rights to
Authenticated Users (C:/msys64/tmp, GitHub-runner work dirs) — and
drive the documented startup-owner publication flow before reading
generation-bound endpoint addresses. This turns the 26 Windows
failures previously visible in CI's full-test job green without
weakening any validation.
- tests(win): the launcher guard covers the full permanent-launcher
contract including failed-uninstall restore and immediate reinstall
after uninstall; new daemon lifecycle and reworked hook-augment
guards run the start/recycle/stop flow end to end.
- tests: CBM_SKIP_PERF is now actually consumed by the test runner
(it was set by CI but never read, so perf suites ran everywhere);
four throughput/bench suites are classified as perf, the heavy
store_arch suite moved to the slow-timeout tier, and two
wall-clock-sensitive assertions were rewritten as invariant checks
with coarse hang-detector backstops.
- build/test infra: build-dir safety contract, UI dev-proxy security
contract, soak daemon-recovery contract, path-safety helper, the
Windows VM worktree-sync contract wired into scripts/test.sh, and
vm/win.sh guards building its clean embedded-UI product in an
isolated BUILD_DIR so it cannot clobber the incremental test build.
provision-windows.sh now installs Node.js for the guards UI build.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The runner executed all 104 suites sequentially (~10 min locally, the bulk
of every CI test leg). The suites are process-isolated already (per-process
mkdtemp HOME sentinel; two full parallel runs produced zero cross-suite
failures), so the serialization was pure convention.
- test-runner --list-suites: prints every registered suite, one per line,
emitted by the SAME macro table that executes suites — the list cannot
drift from the run set by construction.
- scripts/run-tests-parallel.sh: runs each suite as its own process
(jobs = CPU count; CBM_TEST_PAR_JOBS overrides) under a ZERO-LOSS
CONTRACT: a union guard fails the gate if the set of suites that
produced a result differs from --list-suites (nothing can be silently
dropped, a newly added suite is picked up automatically); per-suite
pass/fail/skip are summed into the sequential runner's exact summary
format; any suite crash, failure, or omission exits nonzero. Per-suite
wall times are printed for balance tracking.
- make test-par: the parallel target. make test (sequential) is unchanged
and remains the escape hatch (CBM_TEST_SEQUENTIAL=1 in test.sh).
- scripts/test.sh: builds, then routes through test-par — every CI test
leg gets the speedup with zero workflow-topology change (same jobs,
same gates, same billing; the legs just finish sooner).
- The stack_overflow suite is split into a/b/c (7+6+7 of its 20 tests,
pure re-registration): as one suite it was the wall-clock critical path
of any parallel run — every other suite finished underneath its ~4
minutes.
Measured locally (Apple Silicon, ASan runner): sequential ~10 min vs
parallel 232 s, totals identical (6,361 passed / 0 failed / 1 skipped),
union guard clean. The TSan leg keeps its dedicated subset runner
(sequential) — out of scope here.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_pipeline_fqn_compute strips the file extension, so a C/C++ header
and its same-stem source collide on both the module QN and the File
QN. Node upserts match by QN, so the header's File node is merged into
the source's - the header has no node of its own and can never receive
an inbound IMPORTS edge; the reporter's zero-inbound query flags 194
of 214 files.
The module-QN unification is (at least partly) load-bearing for C/C++
declaration/definition call joins, and the File-node creation sites
overlap PR #995's pending fix for #994, so the fix is deferred to land
on top of that. This RED reproduction (non-gating repro tier) keeps
the bug tracked until then. Refs #964
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Add ObjectScript (InterSystems IRIS / Caché) as a supported language,
covering the UDL class format (.cls), MAC/INT routines (.mac/.int/.rtn),
include/macro files (.inc), and IRIS Studio Export XML.
Definition extraction (extract_defs.c): Class, Method, ClassMethod,
Property, Parameter, Index, Trigger (with body text), XData, Storage,
and Query members as graph nodes; base classes from the Extends clause.
Call dispatch resolution (extract_calls.c) — four ObjectScript patterns
that are structurally invisible to text search:
1. ##class(Pkg.Class).Method() explicit cross-class call
2. ..Method() relative-dot self-call (the dominant
intra-class form; large impact on
CALLS completeness)
3. $$$Macro macro expansion via a per-project
table built from .inc files
4. type inference from %New/%OpenId + declared return types
Ensemble production topology (pass_ensemble_routing.c): EnsembleItem
nodes per production component and ROUTES_TO edges resolved from
ProductionDefinition XData, plus WorkMgr .Queue("##class(X).method")
dispatch — all parsed statically at index time, no live IRIS required.
Language detection (language.c): .mac/.int/.rtn map to ObjectScript
routine directly; .cls (shared with Apex) and .inc (shared with BitBake)
are disambiguated by content, defaulting to the existing language on any
doubt so neither Apex nor BitBake detection regresses.
The two new per-project tables (macros, return types) are threaded
through a new internal cbm_extract_file_ex() so the public
cbm_extract_file() signature is unchanged.
The tree-sitter grammars for ObjectScript UDL and routine are vendored
in internal/cbm/vendored/grammars/objectscript_{udl,routine}/ from
https://github.com/intersystems/tree-sitter-objectscript (MIT, ABI 15).
Refs #462
Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
Tool responses land in an LLM agent's context window, where the legacy
per-node JSON objects were dominated by data no agent can use: the
similarity/semantic pipeline intermediates fp (~450B minhash hex), bt
(body-token bag) and sp (structural profile) plus 14 complexity metrics
were grafted onto every result by enrich_node_properties — ~64% of every
node emission, ~370 tokens per search hit.
search_graph now defaults to TOON (Token-Oriented Object Notation,
toonformat.dev): scalars as 'key: value' lines and results as a
'results[N]{qn,label,file,lines,in,out}:' header + one row per hit.
Published benchmarks show equal-or-better LLM retrieval accuracy at
40-60% fewer tokens than JSON; rows now also carry line ranges, which
the legacy output lacked.
- fields:[...] opts into per-node property columns (complexity,
signature, docstring, ...); fp/sp/bt are blocklisted even there
- format:"json" restores the legacy verbose objects byte-identically;
include_connected forces it (nested neighbor lists)
- BM25 and semantic modes emit the same table shape (rank/score column)
- semantic-only calls no longer run the unfiltered regex search that
silently prepended up to 200 enriched nodes (102.8KB -> 4.2KB)
- default limit 200 -> 50 (BM25 100 -> 50): cheap first page, page via
has_more/offset as before
- new emitter module src/mcp/compact_out.{c,h} (string builder + TOON
scalar/table emission with spec quoting rules)
Measured on a real index (14-hit regex search): 20,793B -> 1,327B
(-94%); BM25 19,667B -> 5,081B; semantic 102,837B -> 4,181B. A typical
discovery session drops ~50.9K -> ~4.7K tokens combined with the
follow-up slices.
Guards: tool_search_graph_toon_never_leaks_internal_fields (sentinel
fp/sp/bt never emitted, even when requested), rewritten
tool_search_graph_includes_node_properties pins default-lean/fields-
opt-in/json-escape-hatch; suite 5983 passed, smoke green.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
- cbm.c: include lsp/perl_lsp.h and dispatch CBM_LANG_PERL -> cbm_run_perl_lsp
- lsp_all.c: unity-include perl_lsp.c + generated/perl_stdlib_data.c
- Makefile.cbm: register TEST_PERL_LSP_SRCS and append to ALL_TEST_SRCS
- tests/test_perl_lsp.c: placeholder suite so the Makefile var resolves
(full suite + test_main.c registration land in plan 22-04)
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
Follow-up to #787/#828, disclosed during #828's review: the same clobber
class also affects two call sites #828 deliberately left out of scope.
create_env_configures_for_file (pass_definitions.c) and resolve_file_throws
(pass_parallel.c) both resolved their edge source via a direct
cbm_gbuf_find_by_qn(enclosing_func_qn) lookup with no guard against the
lookup landing on the shared Folder/Project node for directory-module
languages (Java/Go) -- the exact #787 root cause, just unguarded at two
more sites. A class-level construct (no enclosing function) has its
enclosing_func_qn fall back to the MODULE QN, which for a directory-based
module collides with the package's Folder node, conflating every
same-package file's CONFIGURES/THROWS edges onto one arbitrary source.
- create_env_configures_for_file: added the same
cbm_pipeline_node_is_dir_container guard already used by the three
#828-fixed finders, falling back to the per-file File node on a
directory-container hit.
- resolve_file_throws: was missing the `rel` parameter its sibling
resolve_file_calls/usages/rw already carry, so it inlined a raw lookup
instead of calling the shared, already-guarded find_source_node. Added
`rel` and switched to find_source_node, matching its siblings exactly.
Regression guard: tests/repro/repro_issue842.c, reproduce-first per the
issue's request for "an #828-style multi-order red fixture":
- CONFIGURES reproduces on a small 5-file sequential fixture (Go, since
Java's System.getenv detection is separately broken -- extract_env_key_from_call
reads a "function" field Java's method_invocation node doesn't have; noted
in the test, out of scope here).
- THROWS only reproduces past MIN_FILES_FOR_PARALLEL=50 (the worker-merge
path) -- confirmed empirically that the small sequential fixture already
attributes correctly even on unfixed HEAD, mirroring #787's own two-faced
(sequential/parallel) bug. Padded with 60 filler classes, following
repro_issue787.c's build_parallel_fixture precedent.
RED on unfixed HEAD (both tests): 0 passed, 2 failed. GREEN with this fix:
2 passed, 0 failed. Full scripts/test.sh (5958 cases) passes clean with
this change; scripts/lint.sh's two pre-existing failures (pass_lsp_cross.h
parameter-name mismatch, extract_defs.c cppcheck AST parse) are unrelated
and present identically on unfixed main.
Fixes#842
Signed-off-by: Dhruvy0804 <dhruvgarg553@gmail.com>
Files whose parse tree contains tree-sitter ERROR/MISSING regions were
silently indexed as if complete — constructs inside those regions are
absent from the graph with no signal anywhere (classic trigger: the
preprocessor-blind #ifdef-split-brace pattern in C).
- cbm_extract_file: set parse_incomplete and record the 1-based line
ranges of the top-most error regions (bounded at 64), serialized as
"start-end,start-end" into the result arena
- pipeline: record such files under the new phase "parse_partial"
(distinct from skips — the file IS indexed) in both the parallel and
sequential extraction paths; stamp the File node with
{parse_incomplete:true, error_ranges} on full and incremental runs
via a new cbm_gbuf_set_node_props (upsert survivor rules bypass)
- mcp: report parse_partial {files, count, truncated, note} separately
from skipped[] (skipped_count semantics unchanged); the per-run
logfile lists parse_partial rows; the index_repository tool
description documents the signal and the query_graph pattern to find
flagged files
- tests: new parse_coverage suite (RED-first, 7 cases incl. false-
positive guards and the region cap) plus an end-to-end resilience
guard asserting response fields, logfile, File-node marker, and the
advertised query_graph query
The signal is explicitly best-effort: a flag means constructs in the
listed ranges were dropped (prefer grep there); the absence of a flag
is NOT a completeness guarantee.
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_cli_sha256_file shelled out to sha256sum/shasum, which is unportable:
the tool may be absent, and under cmd.exe the single-quoted path is not
dequoted so the hash of a downloaded release could never be computed on
Windows. Add a small in-process SHA-256 (FIPS 180-4, src/foundation/sha256.c)
and use it directly — no external tool, no per-OS branching, no popen. The
checksum test now covers three NIST vectors (empty, "abc", and a 56-byte
input that forces padding into a second block).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The semantic pass kept four dense 768-float vectors resident per function
(ri/api/type/deco = 12.3 KB per func, ~9.4 GB transient on the linux kernel)
solely so candidate pairs could be cosine-scored after LSH. Replace the
resident floats with 4-bit quantized codes (~0.5 KB each, ~6x): new
src/semantic/rotsq.{h,c} — rotated scalar quantization from the RaBitQ
family of algorithms (Gao et al., SIGMOD 2024/2025), implemented from the
papers in plain C11 (no Eigen/library dependency): a deterministic
XXH3-seeded +-1 diagonal followed by a Fast Walsh-Hadamard rotation
(768 -> 1024 padded) spreads vector mass so per-vector scalar quantization
is near-optimal; the inner product is recovered from the codes with the
exact SQ expansion (one u8 integer dot + four multiplies) — a deterministic
pure function of the codes. Named rotsq, not rabitq: this is the family
core, not the papers' exact codebook construction.
Producers build each dense vector in a worker-local buffer, encode, and
discard; the int8 qvec export is unchanged. LSH signatures are computed
from DEQUANTIZED codes against hyperplanes in the rotated basis — random
hyperplanes are basis-agnostic, so no dense originals are retained and the
phase structure is untouched.
Estimator quality gated by sem_rotsq_ip_error_bounds (seeded vectors:
mean error < 1%, max < 4% of unit scale, ASan-green). Effect on output:
multi-threaded determinism preserved (xfs: two runs byte-identical);
SEMANTICALLY_RELATED edges shift once as scores move within ~1% of the
0.75 threshold (xfs: ~130 -> 203; sampled new pairs are high quality —
alloc/free, read-verify and collapse/insert families).
Suites: semantic 33 (incl. the new estimator gate), simhash 24,
pipeline 216 — zero failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Multi-threaded indexing loses edges vs single-threaded and is
run-to-run non-deterministic (fs/xfs: ST 58573 edges vs MT 58561 on the
first divergent run; rust subtree flickers 44445-44857 across runs).
Discovered while hardening perf-change equivalence checks; also the
reason raw node/edge counts are not a valid equivalence gate.
The guard indexes a corpus once single-threaded (reference) and K=4x
multi-threaded (fresh store each run) and asserts the SORTED
(source_qn|type|target_qn) edge sets are all identical. RED today;
GREEN if and only if MT output is deterministic and equals ST.
Calibration note (false-guard avoidance): three escalating synthetic C
corpora (300/500/600 files, dense cross-file calls, token-identical
bodies) stayed fully deterministic -- the similarity pass emits zero
edges on synthetic C, and the race needs real-code edge diversity. A
synthetic guard would pass on buggy code, so this repro uses the
smallest real corpus that reproduces (kernel fs/xfs) and SKIPs when the
corpus is absent. Suspected root causes documented in the test header
(per-worker edge-buffer merge, gbuf edge dedup check-then-insert,
symmetric SIMILAR_TO emission, resolve ordering). Fix deferred; this
test is its guard.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Closes#423, #20. A non-ASCII repo path passed to
`codebase-memory-mcp cli index_repository "<json>"` was mangled on Windows, so the
command failed with "repo_path is required" instead of indexing the real directory.
POSIX is unaffected (argv is already UTF-8). Two boundaries had to be fixed:
1. argv read: main() took only the narrow `int main(int argc, char **argv)`, so the
CRT handed it argv in the active ANSI code page. On Windows main() now rebuilds
argv from the wide command line (GetCommandLineW + CommandLineToArgvW) and
converts each element to UTF-8 (cbm_wide_to_utf8).
2. worker spawn: the index supervisor (and the UI index thread) spawned the worker
via CreateProcessA, which re-interpreted the UTF-8 command line through the ANSI
code page -- re-mangling the path at the parent->worker boundary even after (1).
Both spawns now build a wide command line (cbm_utf8_to_wide) and use CreateProcessW.
Without (2) the default (supervisor-enabled) path stayed broken while only the
in-process path worked, so this is the complete end-to-end fix, not just argv.
Promotes tests/windows/test_cli_non_ascii_arg.py from an opt-in known-red to a green
Windows guard (scripts/test-windows.ps1 + the test-windows-guards CI job). The guard
now exercises the DEFAULT supervised path (it drops the suite's CBM_INDEX_SUPERVISOR=0
so it actually crosses the worker-spawn boundary). Verified on native Windows: RED
before the fix, GREEN after (a non-ASCII repo indexes, nodes>0). Also adds -lshell32
explicitly (main.c now uses CommandLineToArgvW) and refreshes RED_TEST_ANALYSIS.md.
Signed-off-by: Flipper <jacobphilipp@ymail.com>
libgit2 is licensed GPLv2-with-linking-exception, and this project is
deliberately GPL-free (see scripts/license-policy.json). libgit2 was only
an OPTIONAL, faster git-history code path guarded by HAVE_LIBGIT2 and
auto-detected via pkg-config; it always shipped with a popen("git log ...")
fallback that release binaries already used (they were built without
libgit2). Making that popen fallback the SOLE git-history implementation
means there is NO change to shipped behavior.
Changes:
- pass_githistory.c: collapse the HAVE_LIBGIT2 #ifdef so only the popen
parse_git_log remains; drop the now-unused <git2.h>/<time.h> includes.
- cbm.c / cbm.h / main.c / subprocess.c / index_supervisor.h: remove the
libgit2 mimalloc allocator bind and its >=1.7.0 version guards; keep the
tree-sitter + sqlite3 binds; correct the comments.
- Makefile.cbm: remove the optional-libgit2 pkg-config detection block,
the REQUIRE_LIBGIT2 error gate, and all LIBGIT2 CFLAGS/LIBS/FLAGS.
- .github/workflows/_test.yml: drop the REQUIRE_LIBGIT2 test-matrix leg
plus its libgit2-dev/pkg-config apt install (lower CI cost, no new
gating, no trigger change).
- flake.nix: drop libgit2/pkg-config from the dev shell (now-dead dep).
HAVE_LIBGIT2 no longer exists anywhere in the tree.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel