Second LSan round from the Linux leg, both verified green in the container
(532 passed, 0 failed under LSan):
- dump_and_persist_hashes' two semantic-manifest abort returns leaked BOTH of
the function's strdups (db_path and db_dir, the latter otherwise freed only
further down). Same ownership rule as the previous fix: every exit releases
what the function allocated.
- test_parallel's sequential harness drives the passes directly and never
destroyed ctx->seq_cross_arena, which the cross pass fills with the shared
per-language registries (stdlib registrations included -- ~20MB per test).
Production's run_sequential_pipeline destroys it after all passes; the
harness now does the same.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Maintainer decision (option B) closing the Kotlin property-reference repro's
platform divergence at its root instead of tie-breaking the fallback.
`holder::handler` (parsed as navigation by the vendored grammar) produced a
USAGE edge whose target was chosen by the name-only registry fallback: with a
same-named property (Holder.handler) and function (Functions.handler) in the
project, the winner was registration order -- readdir order -- so Windows
(lexicographic NTFS) borrowed the FUNCTION while macOS happened to pick the
property. The graph's answer must not depend on directory enumeration.
The occurrence is now claimed by exact, receiver-typed resolution. Five links,
each of which was missing:
1. extract side: a Kotlin navigation member read is a semantic-reference
candidate at the member occurrence, so an LSP row can claim it. With no row
the join finds nothing -- but a candidate no longer falls back to the
name-only registry guess, which is the fail-closed direction this PR is
built on.
2. kotlin cross resolution: a property READ with a proven receiver emits a
CALL_REFERENCE row against the property. The property is not a callable
target, so the join can only produce USAGE, never a fabricated
CALL_REFERENCE. Value reads only; calls stay with the invocation machinery.
3. receiver typing across files: kotlin_resolve_class_name composed
<this module>.<name> for an unimported cross-file type, which can never
name a type defined in another file. A per-call unique-short-name map
(built from the project defs, ambiguous names fail closed) resolves the
annotation to the real registered QN. Hash lookup, no scans.
4. cross registry fields: pxc_map_label dropped Variable defs entirely, so no
cross registry ever saw a property. They now flow through (every language's
registrar filters by explicit label, so only Kotlin consumes them) and the
Kotlin registrar attaches them as fields of their receiver type,
hash-bucketed -- a per-type scan would be the registry-tail-scan quadratic
pattern.
5. def side: class-body variables now record their declaring class
(parent_class) -- previously only methods did. The QN stays module-level,
so this is additive metadata: the only structural parent_class consumer is
Method-gated (DEFINES_METHOD), verified in pass_definitions/pass_parallel/
pipeline_incremental.
Emission targets each field's REAL def QN carried through the field map:
kotlin class properties are minted with module-level QNs (proj.Holder.handler,
not proj.Holder.Holder.handler), so the composed form would name a node that
does not exist and the join would silently drop the edge.
The repro now proves the property edge on every platform for the same reason,
not by racing readdir.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
resolve_db_path returns a strdup the function owns, but neither the
publish-failure return nor the success tail freed it --
cbm_pipeline_refresh_artifact only borrows the pointer. Every pipeline run
leaked one path string; LeakSanitizer on the Linux leg aborted the pipeline,
index_resilience and mcp suites over exactly this pair of exits (every leaked
allocation across the leg traced to this single strdup). macOS stayed green
because this setup has no leak detection there, which is precisely why the
Linux leg exists.
Linux container, same three suites under LSan: green after the fix.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
make lint-ci was red on this branch before any of the rebase work.
cppcheck reported all four halves of two ULARGE_INTEGER values in
cbm_path_info_utf8 as assigned-but-never-read. The code is correct -- it is a
union, and .QuadPart reads exactly what .LowPart/.HighPart wrote -- but cppcheck
does not model that aliasing, so it cannot see the read.
Composing the two 64-bit values arithmetically says the same thing without the
union, so the checker needs no exception. That is the repository's stated
preference: refactor first, adjust the rule second, suppress only as a rare
justified exception -- and a suppression here would have to be re-justified by
every future reader.
Also applies clang-format to the lines this rebase touched in pipeline.c and
extract_usages.c, plus one pre-existing violation in pipeline_incremental.c.
Formatting only; the file set is limited to what LINT_SRCS/LINT_HDRS actually
covers, so no unrelated whole-file reflow rides along.
make -f Makefile.cbm lint-ci: passes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
main and this branch each grew their own atomic publication while the branch was
outstanding, and neither knew about the other:
main cbm_pipeline_run() copies the live database into an mkstemp staging
file, points the whole run at that copy, and renames it over the
destination at the end. That is what stops an INCREMENTAL run from
mutating the live database in place.
branch cbm_pipeline_publish_generation() builds a generation in its own stage
file, validates it (integrity check, FTS rebuild, seal), quarantines a
corrupt destination to a fresh .corrupt name, then renames.
The rebase kept both, nested, so the branch's layer ran against main's staging
file rather than the real database. Two consequences, both silent:
1. The quarantine became a no-op. Its "existing destination" was a staging file
we had just created, so a genuinely corrupt destination reached
cbm_rename_replace and was overwritten -- the one copy of the bytes that
would explain the corruption, destroyed by the recovery path.
2. Every publish failure collapsed to CBM_NOT_FOUND, because the outer wrapper
returned a bare -1 for its own errors, a cancellation, and a failed persist
alike. A caller could not tell "aborted, your data is intact" from "the
persist failed", which is the only distinction that matters at that moment.
Rather than pick one implementation, put each concern at the layer that owns it.
The outer wrapper owns the real destination's lifecycle, so it now owns the
quarantine: prepare_publish_destination() calls the branch's existing
prepare_existing_generation_for_replace() when the destination could not be
copied, which moves it aside only when it is verifiably not a readable SQLite
database. A destination that is valid (the backup failed for some other reason)
is sealed and replaced as before, never renamed away, so a good database is
never mislabelled .corrupt. main's guard that refuses to drop sidecars holding
uncommitted pages is kept ahead of it. A failed rename rolls the quarantine
back, so a caller is never left with no database at all.
The wrapper also stops flattening. Everything it does happens before the
publishing rename, so an abort there is genuinely non-destructive and can say
so: cancellation reports CBM_PIPELINE_ABORT_PRESERVE_DB, a failed seal reports
CBM_PIPELINE_PERSIST_FAILED, and a status from the inner publish propagates
unchanged. Both MCP call sites test only `rc == 0` and are unaffected; the codes
stay in pipeline_internal.h beside the stages that raise them, and pipeline.h no
longer claims a -1 it does not return.
Quarantining is now also correctly refused at the inner layer, which was the
same confusion in the other direction. publish_generation's destination is the
staging file the wrapper created moments earlier, so when that file was not a
readable database it was being parked as
`<db>.stage.<random>.corrupt` -- debris named after a temp file, which nothing
collects and no one can interpret. It only surfaced once the status codes stopped
collapsing, because the assertions that count leftover stage artifacts sat behind
the return-code assertions that failed first. The inner caller now discards an
unreadable staging file instead, and only the wrapper, which owns the user's real
database, ever quarantines.
Two of main's own tests asserted the bare -1. Both are named
cancelled_*_reindex_preserves_committed_db and now assert
CBM_PIPELINE_ABORT_PRESERVE_DB -- the value whose name is their subject. That
tightens the assertion rather than relaxing it: -2 answers their question and
-1 did not.
Closes the last 10 pipeline failures from the rebase.
pipeline, incremental, store_nodes, mcp: 662 passed, 2 skipped.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two fixes on top of the main merge.
1. try_incremental_or_delete_db removed the existing database unconditionally.
On main those two lines are only REACHED on the reindex route, because the
incremental path returns early. This branch restructured the function so there
is no early return, and the merge carried main's cleanup in verbatim -- so the
database was deleted on every route, including the no-op and the successful
incremental publish.
The failure mode was badly misleading: the pipeline genuinely succeeded and
logged pipeline.done nodes=17, and cleanup then destroyed the result, so every
later reader reported dump.verify reason=store_missing. Because the probe
suites all share ~/.cache/codebase-memory-mcp and all have the shape
"index a snippet, open the store, count nodes", one suite that indexed twice
destroyed the shared database and every probe suite after it failed at its
store-open gate.
Full suite went from 6539 passed / 836 failed to 7354 passed / 21 failed. The
narrow arena that isolates it -- test-runner mcp node_creation_probe -- went
from 83 failed to 269 passed, which is exactly main's number for that pairing.
2. pass_parallel.c sized its usage-properties buffer at 256 bytes while the
sequential twin in pass_usages.c uses 512.
esc_ref holds up to 255 bytes and the {"callee":"..."} wrapper adds 13, so a
long callable identifier truncated mid-string, cutting the closing quote-brace
and persisting malformed JSON into edge properties. The two paths must also
agree: the same repository indexed in parallel and sequentially has to produce
byte-identical edge properties. This is RED item 4 from the PR description, and
it was a live defect on main rather than only on this branch.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
process_edges and expand_var_length carried the expansion budget in the LOOP
condition (`ei < edge_count && *new_count < max_new`). Once new_count reached
max_new they stopped iterating entirely, so match_count was never incremented
for the remaining sources -- even though those sources had real neighbours.
expand_pattern_rels' OPTIONAL fallback is ungated (it has a reserved slot per
binding since the #1177 sizing fix), so it saw match_count == 0 and emitted an
unbound row for every such source.
The result is a false negative that reads as a positive assertion:
MATCH (f:Function) OPTIONAL MATCH (f)-[:CALLS]->(c) WHERE c IS NULL
reports a function as having no callers when it has callers, once an earlier
function in the same scan saturated the budget. That is a dead-code query
telling a user that live code is dead.
The budget caps MATERIALISATION, never detection. Both loops now iterate the
full candidate set, increment match_count whenever a real neighbour passes the
filters, and gate only the write into new_bindings. The bound proof from #1177
is unchanged -- writes are still capped at max_new and fallbacks are still at
most one per binding -- so allocation safety does not depend on this change.
Cost: a saturated source still performs its node lookups and filter checks, the
same ones the unsaturated path performs, and materialises nothing.
Reported by @SEPURI-SAI-KRISHNA while working on the sibling bound-terminal path
in #1378; this is the same defect in expand_pattern_rels, which was already live
on main rather than introduced there.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Three items from the #1372 review.
The Windows shard failed in the fixture, not the fix: `system("cd '<dir>' &&
git init …")` used POSIX single quotes, which cmd.exe cannot parse. Switched
to the portable `git -C "<dir>"` shape already used by wt_git in
tests/test_watcher.c, with identity, default branch and signing passed via -c
so the fixture no longer depends on the machine's global git config.
A changed file with hunks but no overlapping definition now keeps whole-file
seeding instead of dropping out. Import-only edits, module-level constants and
anything above the first definition land outside every definition's line range,
so scoping alone would have removed the file from the seed set entirely —
strictly worse recall than the behavior being replaced.
The overlap probe has to apply the same label filter as the seeding loop: a
Module node spans lines 1..EOF, so probing raw nodes reports an overlap for
every hunk and defeats the fallback. Extracting detect_is_seedable_label lets
both sites share one definition. The new regression test caught exactly this —
it failed with seed_symbols 0 until the probe was filtered.
A filled hunk buffer is now treated as a truncated diff: hunks past HUNK_CAP
are gone, so files captured only partially would still look scoped and silently
under-seed. Scoping is dropped for the whole request and the decision logged,
rather than under-reporting a large refactor.
Documented the mixed coordinate systems in the fetch block: base...HEAD hunks
are HEAD-side lines, worktree hunks are worktree-side, node lines come from the
indexed snapshot. They agree while the index is fresh; a stale index plus
earlier-in-file insertions can mis-scope, bounded by the zero-overlap fallback.
Signed-off-by: lishixiang <lishixiang@gmail.com>
cbm.h documents cbm_label_is_type_like() as the single source of truth for
type-like labels, "so adding a new type-like label (e.g. Struct for
Rust/Go/Swift/D structs) updates them all at once instead of scattering
|| strcmp(label,"Struct")==0 across the tree".
A SQL string literal cannot call it. Four queries in store.c and the BM25
ranking in mcp.c carried their own hardcoded label lists, so they silently
opted out of that contract and stopped matching the moment Struct, Interface,
Enum, Type and Trait began being emitted:
store.c:5279 arch_boundaries ('Function','Method','Class')
store.c:5422 arch_packages_from_qn ('Function','Method','Class')
store.c:6689 arch_clusters ('Function','Method','Class')
store.c:7742 cbm_store_vector_search ('Function','Method','Class')
mcp.c:2759 BM25 ranking ('Class','Interface','Type','Enum')
Effect on main today: get_architecture (boundaries, packages, clusters) and
vector search drop every Struct, Interface, Enum, Type and Trait node in the
project, and search_code under-ranks structs -- for Rust, Go, Swift and D,
where struct is the primary type declaration. The BM25 list had drifted
differently again, omitting Struct and Trait but not Interface.
The fix is not a wider literal. CBM_SQL_TYPE_LIKE_LABELS and
CBM_SQL_CALLABLE_OR_TYPE_LABELS live next to each other in constants.h and are
pinned to cbm_label_is_type_like() by a test that checks both directions:
every label the C predicate accepts appears in the SQL fragment, and nothing it
rejects is smuggled in. Adding a type-like label without updating the SQL now
fails that test instead of quietly shrinking query results.
No new labels are introduced. Every label added here is one the extractors
already emit.
Reported by @win4r in #568, whose audit found four of the five sites. Their
patch widened the literals in place and added Actor, a label main does not
emit; this routes through the canonical set instead and also covers Interface,
Type and Trait, which that patch still omitted.
Co-Authored-By: win4r <win4r@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The indexer already records how every CALLS edge was resolved --
pass_calls.c:355 writes {callee, confidence, strategy, candidates} into the
edge properties, and cbm_store_traverse reads it back -- but no tool ever
surfaced it. An agent could see THAT A calls B, never how confident the
resolver was, or whether the edge came from an LSP or from a name heuristic.
trace_path / trace_call_path gain include_evidence (default false). When set,
each row carries two extra columns: the resolution strategy CLASS and the
resolver's confidence.
Two decisions worth recording:
Opt-in, not default. Two extra columns on every row is exactly the inflation
the tree format exists to avoid, so the default response is byte-identical to
before.
A closed class, not the raw strategy. Production emits ~20 internal strategy
names (lsp_trait_dispatch, php_self_static, callee_suffix, ...) and the set
grows with every language. Publishing them verbatim would make each internal
resolver name public API by accident. cbm_mcp_edge_strategy_class maps them to
a fixed vocabulary -- lsp | language_rule | heuristic | unresolved -- so adding
lsp_foo_dispatch maps automatically while a genuinely new KIND of resolution
fails the pinning test and forces a deliberate decision. lsp_unresolved
classifies as unresolved, not lsp: the caller's question is whether the edge is
trustworthy, and "we tried LSP and it did not resolve" answers no.
Evidence is looked up against the PAGINATED view, so it is emitted only for the
rows on the current page -- it does not bypass limit, the cursor, or the
MCP_BFS_LIMIT_MAX guard.
Tests: the vocabulary pin (every strategy production can emit maps to a known
class), and an end-to-end test binding both halves -- absent by default,
present when asked for, and the raw internal name never reaching the client.
Both fail without this change.
Distilled from #559 by @vvenegasv, who found that we already store this and
never show it. The freshness half of that PR is not included: it is
non-functional in production (both pipelines persist through the direct SQLite
writer, which records a hardcoded NULL) and its ALTER TABLE migration never
runs for the read-only opens the MCP server uses.
Co-Authored-By: vvenegasv <vvenegasv@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The bare-node-carry check in with_agg_find_or_create only tested
!property && variable, so labels(n)/id(n)/keys(n)/properties(n) aliases
(variable set, property NULL, func set) were also tagged with the source
node's id. A later alias.property then hit node_prop's stub re-fetch
heuristic and silently returned the source node's real property instead
of empty for the non-node alias.
Adds cypher_issue1111_with_scalar_func_alias_no_node_leak, which fails
on the prior code (returns the source node's file_path) and passes with
this fix.
Addresses DeusData's review on #1221.
Signed-off-by: Amir Fathi <amirfathi.me@gmail.com>
RETURN/WITH aggregation classified each projected column with a bare
`if (item->func)` check to decide group-key vs. aggregate value. That
check is true for any function call, so type()/labels()/id()/keys()/
properties() were routed into the aggregate branch alongside real
aggregates and formatted via format_agg_value's default case, which
emits the row count instead of the function's actual value.
Swap the five call sites (ret_agg_build_key, ret_agg_emit_row,
with_agg_build_key, with_agg_find_or_create, with_agg_accumulate) to
is_aggregate_func(), the predicate already used correctly one level up
to decide whether a query needs aggregation at all. Group-key columns
carrying a non-aggregate function now project through the existing
project_item() helper instead of binding_get_virtual(), since they
need to evaluate the function rather than read a raw property.
Fixes#1111 (the type(r)/count(*) half; the inline-property-map half
of that issue does not reproduce on current HEAD)
Signed-off-by: Amir Fathi <amirfathi.me@gmail.com>
detect_collect_seeds treated every definition in a changed file as a
BFS seed, regardless of which lines actually changed. A one-line edit
inside a single method seeded every other definition in the file too,
producing an impact report an order of magnitude larger than what the
edit actually touched.
Fetch `git diff --unified=0` hunks (cbm_parse_hunks already existed in
pass_gitdiff.c but had no caller) alongside the existing file list, and
scope seeds to definitions whose line range overlaps a hunk. Any
failure fetching hunks (new/untracked files, a transient git error)
falls back to the previous whole-file behavior, so this is a precision
improvement with no new failure mode.
Verified on microsoft/qlib (an unrelated third-party repo): a same-
line-count edit inside one method now seeds 2 symbols (the method +
its containing class) instead of 25 (every definition in the 24-def
file), matching the equivalent output of a different code-graph tool
(GitNexus) on the identical scenario. Reproduced on a second file with
a different definition count (42 defs -> seeds 43 before the fix, 2
after) to rule out coincidence.
cbm_changed_hunk_t / cbm_parse_hunks move from pipeline_internal.h
(explicitly not a public header) to pipeline.h, since detect_changes
now needs them from src/mcp/mcp.c. cbm_detect_node_in_hunks is exposed
non-static via mcp_internal.h, matching this file's existing white-box
test-hook pattern, so the overlap logic has a direct unit test
independent of the git/subprocess/index plumbing around it.
Fixes#1363
Signed-off-by: lishixiang <lishixiang@gmail.com>
CodeQL flagged two high-severity alerts in the previous commit: the help-list
builder accumulated snprintf's return value, which is the length it WOULD have
written rather than what it did. Once len passes cap, the next `cap - len`
underflows to a huge size_t and the following write lands outside the buffer.
That is the same class the Cypher UNWIND list builder was just fixed for in
#1173, where blen advanced past a 2KB buffer the same way.
The capacity computed here is in fact sufficient for the current registry, so
nothing overflows today -- which makes this the more dangerous shape, not the
safer one. The code was correct only by an argument made ten lines away in the
cap loop, so renaming a tool or changing the wrap rule would have turned it into
an out-of-bounds write with nothing in the function to notice.
help_append() returns the bytes ACTUALLY written, clamped to the space left, so
`len <= cap - 1` becomes a local invariant that holds no matter what the
registry or the wrap column later become. Also adds the <stdarg.h> the variadic
helper needs.
Verified: builds clean under -Werror; --help still renders all 15 tools in
registry order with the same wrapping; mcp + cli suites 437 passed, 0 failed;
make -f Makefile.cbm lint-ci clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The top-level --help listed 14 hand-maintained tool names and omitted
check_index_coverage; MCP tools/list correctly advertised all 15. The
help string was a separate copy of the registry, so it drifted silently
when the tool was added.
Render the Tools block from the TOOLS registry itself (new
cbm_mcp_tools_help_list, plus cbm_mcp_tool_count/cbm_mcp_tool_name
accessors) so the two lists share one source of truth and cannot
diverge again. A unit test pins the render against the registry:
every advertised tool appears, cardinality matches, and lines stay
80-column safe.
Fixes#1361
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cppcheck blocked the release run, correctly:
src/main.c:283: Condition '!main_test_worker_project_lock_marker(mutation)' is always false
src/main.c:2049: Condition '!worker_start_watchdog_test_descendant()' is always false
Compiling the seams out left behind stubs that returned true unconditionally, so
negating them was provably constant. Refactored rather than suppressed (repo
rule): the stubs are deleted and the CALL SITES are guarded, so a release build
does not contain the call at all -- which is what "the seam is not in the
artifact" should have meant in the first place.
The worker path needed care. The probe was one term in a five-term `||` chain,
and its POSITION is load-bearing: it must fork AFTER the isolated process group
exists (it inherits that group) and BEFORE the parent-death watchdog thread
starts, because forking a multithreaded process is the bug that ordering avoids.
Hoisting the call out of the chain would compile, pass tests, and silently break
that invariant. So the chain is split into three ordered steps -- process group,
then (test builds only) the probe, then the watchdog -- sharing one
worker_containment_unavailable() failure path so every step still fails
identically, with write()/_exit() rather than stdio and atexit handlers.
Verified in BOTH directions, since a guard that is always off is as wrong as one
always on: a TEST_SEAMS=1 build still contains the probe and
tests/test_worker_watchdog.sh passes against it, while the release build carries
no seam string. `make -f Makefile.cbm lint-ci` now passes locally -- the step
skipped before the last push, and the reason CI found this instead of me.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The update handoff moved out of the product on every platform, but two loose
ends were left behind.
README was actively WRONG, not merely incomplete. It still documented in-place
updates for macOS/Linux, and it promised "The MCP server also checks for updates
on startup and notifies on the first tool call" -- a claim that died with the MCP
update thread and then the daemon's curl check. The section now states the one
contract that holds everywhere (`update` validates flags and prints the install
script command), why it works that way (on Windows a running image cannot
replace itself; on POSIX a deliberate choice, because an in-process updater is
structurally a downloader), and the affirmative property that replaced it: cbm
makes no network request of its own accord and the archives carry no download
URLs at all.
`uninstall` now REPORTS the install script it finds beside the binary, printing
the path and the `rm` command, and says "would leave" under --dry-run. It does
not delete it. install.sh places itself there so `update` has something to hand
off to, but that file may equally be the user's own copy, a symlink into a
checkout, or package-manager managed -- an uninstaller must not delete a file it
cannot prove it owns, and one printed line leaves the decision where it belongs.
Windows checks install.ps1 too and tolerates either path separator.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Microsoft's ML flagged the rc.1 release binaries. The decisive evidence is that
the SAME sha256 went from 0/62 clean to Microsoft-detected in about an hour with
no byte change, so the verdict lives partly in scanner-side state and no code
change can promise a clean result. What code CAN do is stop shipping things that
have no business in a release artifact, which is worth doing on its own merits
and incidentally widens the classifier margin. Every claim below is verified
against a built binary by the new gate, not by reading source.
Executable stack (the worst of the findings). vendored/nomic/code_vectors_blob.S
is the only assembly in the build and carried no .note.GNU-stack. An unannotated
object makes ld assume the worst for the whole link, so EVERY Linux release we
have ever shipped had GNU_STACK RWE. Adds the note (cause) plus ELF-only
-Wl,-z,noexecstack (outcome); the gate fails the release if it returns.
Test seams are now opt-in, never opt-out. TEST_SEAMS=1 defines
CBM_ENABLE_TEST_SEAMS; without it the crash-orphan probe -- which forks a child
that ignores SIGTERM and loops forever, then writes its pid to a caller-supplied
path -- and the lease-ownership marker compile to trivial stubs, so call sites are
untouched and the binary holds no fork, no signal handler and no env-var string.
Opt-IN is the point: forgetting the flag yields a clean binary rather than a leaky
one. scripts/test.sh requests it in the leg that consumes it, and
tests/test_worker_watchdog.sh now asserts the capability up front instead of
dying later with an opaque "Killed: 9".
The daemon's background version check is gone. It spawned curl against
api.github.com/repos/.../releases/latest on the first eligible session of every
run to say "a newer version exists" -- a release URL and an outbound request in
every shipped binary, for something the install scripts already report. The
INJECTABLE SEAM survives: update_ops is still honoured, the fakes in
tests/test_daemon_application.c still cover notice/ownership/cancellation/replay,
and with no provider application_update_subscribe_locked returns early so no
generation ever starts. "No network request by default" is now structural.
Dead capability out of release builds. The tar.gz/zip extraction block
(gzip_decompress through cbm_extract_binary_from_zip, plus its cli.h
declarations) moves under CBM_CLI_ENABLE_TEST_API -- verified self-contained, zero
uses of any helper outside it, only callers the excluded updater and
tests/test_cli.c. Downloading an archive, decompressing it, picking an executable
out of it and marking it executable is the canonical dropper composite; it is now
absent rather than merely unreachable. SQLite is built with
-DSQLITE_OMIT_LOAD_EXTENSION (no caller of load_extension anywhere in src/ or
internal/), removing that API surface and part of the dlopen/dlsym surface.
Temp files and environment scanning (S2/S3). Predictable paths in mcp.c,
artifact.c and diagnostics.c are created privately and exclusively and written
through the returned descriptor; pass_envscan.c no longer descends symlinked
directories out of the project root, and its fixed 512-byte path buffers no
longer truncate into pointer arithmetic that could land outside the buffer.
Build-time entropy. mimalloc's version banner baked __DATE__/__TIME__ into every
binary, so two builds of identical source seconds apart could never share a hash
and no release could inherit a false-positive determination made about its
predecessor. Local patch removes it (marked to survive refreshes), -Wdate-time
makes any future use a build error, and -Wl,--no-insert-timestamp stops the PE
header carrying the link clock.
scripts/ci/check-binary-composition.sh is the proof that each removal stays
removed, wired into package-release.sh after strip so the local artifact-flow
smoke enforces exactly what CI does. It asserts absences plus a CANARY string, so
handing it a compressed, stubbed or empty file fails instead of passing
vacuously, and a missing tool is a hard error -- a skipped assertion must never
look like a satisfied one.
Two build-system traps found by that gate, both of which had silently defeated a
fix: the product binary is compiled in one shot from sources, so a flag flip did
not rebuild it (now tracked by a .build-config stamp that also removes the
binary, making it independent of mtime granularity); and prod_sqlite3.o /
prod_mimalloc.o depended on a single named source, so SQLITE_OMIT_LOAD_EXTENSION
and the mimalloc patch BOTH compiled to nothing on the first incremental build.
Source review would have called them done.
Deliberately NOT changed. Three seams stay in release artifacts because
scripts/smoke-test.sh runs against the real artifact and needs them:
CBM_TEST_CRASH_ON and CBM_TEST_HANG_ON inject the faults that prove supervisor
recovery, and CBM_TEST_WINDOWS_USER_PATH_RUN_ID is what keeps the PATH smoke from
writing the tester's actual PATH. The gate treats those as an allowlist, so a
NOVEL seam still fails. The true no-UI standard build is deferred rather than
rushed: src/ui/* is in PROD_SRCS and four files outside src/ui reference UI
symbols, including the daemon that serves the UI, so that assertion reports
instead of failing until the split lands -- a gate everyone knows is red teaches
people to ignore gates.
No grammar is removed. ObjectScript accounts for essentially all binary growth
since the last provably-clean release (+21.4MB rodata, +1.1MB text from two
four-line shims), which made it the obvious ablation candidate, but a dry run
performed twelve real Defender endpoint scans across standard/UI and amd64/arm64
with ObjectScript, the daemon and the expanded hooks all present and every scan
was clean. Nothing there is a deterministic trigger, so cutting a
community-contributed language would spend a real feature on unproven margin.
Lean is not a candidate either: at 99.6MB of source it is by far the largest
grammar, but it shipped in v0.9.0 which scanned 20/20 clean, so removing it would
produce a novel unscanned profile instead of restoring a known-good one.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Completes the move started on Windows. `update` prints the install script's
command and exits 0 on every platform; the download/extract/chmod/exec sequence
is compiled out of release builds entirely, and the MCP background thread that
ran cbm_popen("curl ... api.github.com ...") on first tool call is gone.
The installers now place themselves beside the binary, which closes the last
gap: `update` referenced a raw.githubusercontent URL purely because an
install.sh-installed user had no local copy to point at. Two details make that
safe.
The source is the install script from the archive we JUST checksum-verified,
never "$0" -- which does not exist under `curl | bash`, and would pin the user
to the OLD installer forever. Since the script ships inside the verified
archive, it inherits that verification instead of needing its own.
And it is published by atomic rename, never written over the live path. Bash
reads a script incrementally by byte offset, so overwriting the file it is
executing continues reading NEW bytes at the OLD offset: silent, bizarre
corruption. cp to a temp name then mv -f swaps the directory entry while the
running shell keeps its original inode. Windows follows the same rule via
Copy-Item + Move-Item even though PowerShell parses up front.
Deliberately NOT done: spawning the new installer to delete and replace its
predecessor. Fetch remote code -> drop to disk -> spawn -> self-delete is a
textbook stager, and self-deletion is among the most heavily weighted
heuristics there is. It also gains nothing -- `"$DLBIN" install` already runs
the NEW release's install logic, because the binary owns the install.
Net: zero install or download URLs remain in the shipped binary, and the
allow-list entry is retired with it. Both scripts scan CLEAN (0/61) with the
self-placement code in them.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The `update` handoff prints a curl one-liner when install.sh is not beside the
binary, and the security audit blocked the release on it:
BLOCKED: src/cli/cli.c: URL not on allow-list: https://raw.githubusercontent.com/DeusData/
Two things were wrong. The URL was not declared, and it was split across string
literal continuations, so the audit extracted only the first fragment -- an
allow-list entry for the real URL could never have matched it. Hoisted to a
single CBM_INSTALL_SH_URL token so the declaration means what it says.
It is the same URL as the documented one-liner install, and it is printed for
the user to paste: the binary no longer downloads anything.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Windows already handed updating to install.ps1, because a running .exe cannot
replace its own image. The remaining platforms now do the same, for a different
and better reason.
An in-process updater is structurally a downloader: it fetches a remote archive,
verifies it, extracts it, marks the result executable and runs it. That composite
lived in every shipped artifact -- along with the release URLs it fetched -- to
serve a command most users run a handful of times. The install scripts already
do all of it, are idempotent (so re-running one IS the update), and run while cbm
is not running.
`update` now prints the exact command for the running platform and exits 0:
unix bash "<dir>/install.sh" (or the curl one-liner when not adjacent)
windows powershell -ExecutionPolicy Bypass -File "<dir>\install.ps1"
Both scripts ship inside their release archive, so the printed path resolves
next to the binary. Flags are still parsed and validated, so `update --dry-run
--ui` keeps rejecting typos rather than silently accepting them.
The updater itself is EXCLUDED from release builds rather than left to dead-code
elimination: build_update_url, download_verify_install, extract_and_install_binary,
checksum fetch, detect_os/detect_arch and the cli_download_* helpers now compile
only under CBM_CLI_ENABLE_TEST_API, which release builds do not define. The C
suite still covers the flow through the activation test seam. Verified against
the release binary: `releases/latest/download` is gone entirely.
Also removes the MCP background update check. It ran on the first tool call and
did:
cbm_popen("curl -sf --max-time 5 -H 'Accept: application/vnd.github+json' "
"'https://api.github.com/repos/.../releases/latest' 2>/dev/null")
An embedded shell command plus a startup network callout, in every session, to
tell users a newer version exists. It shelled out, depended on curl being
installed, and phoned home from every agent session. The install scripts report
versions; this did not need to be in the server.
Smoke follows the same collapse. Phases 6c and 14 asserted a Windows-specific
handoff and a POSIX in-process replacement; they now assert one platform-neutral
contract selected by UPDATE_SCRIPT, and 14a's byte-identity check compares the
driver against ITSELF before and after -- comparing against "$BINARY" passed on
Windows only because Windows has no signing step, while the POSIX fixture
ad-hoc re-signs its copy and so differed before `update` ever ran.
The Linux glibc guard is RELOCATED, not retired. The standard linux asset links
glibc 2.38+ and breaks Debian 11, RHEL 8 and Ubuntu 20.04, so the installer must
fetch the static "-portable" build. That constraint moved from the binary to
install.sh, and the smoke now guards it where the behaviour lives.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Release run 30385988475 lost three Windows smoke jobs to one phenomenon on two
different surfaces. A Windows file that was just written or just executed can
refuse deletion and rename for a moment while a scanner reads it or while the OS
finishes reaping a child. Both conditions clear on their own within moments.
It reads as nondeterminism because it is timing: windows-2025 passed both amd64
variants while windows-latest -- the SAME image, windows-2025-vs2026 -- failed
both, and windows-11-arm failed too.
PRODUCT: uninstall abandoned a live installation.
With one binary per platform, uninstall must remove the very image it is
executing, and on Windows a running image can only be renamed, never
overwritten. That rename got a single attempt, so a transient sharing violation
surfaced to the user as
error: failed to remove ...; completed configuration/index cleanup may remain
with the install left half-torn-down. install.ps1 already retries this exact
operation ten times for this exact reason; the C path now does the same, and
only for sharing/access/lock violations, so a file that is genuinely held still
fails closed instead of spinning.
Proven with a failure-injection seam rather than by racing a real scanner:
3 injected failures must still succeed, 64 must still fail AND leave the target
in place. Deterministic, so it does not need CI to learn whether it works.
HARNESS: the smoke killed itself silently.
Phases that install a binary then `rm -rf` the directory hit the same lock. Under
`set -euo pipefail` that returns non-zero and ends the run WITHOUT printing
anything -- the logs jump straight from "OK 13h" to job cleanup with no FAIL
line, which is what made this expensive to read.
None of those 16 removals is an assertion; they are fixture cleanup, and
_smoke.yml already states the principle ("a temp dir on an ephemeral runner must
never fail the job") that smoke-test.sh did not honour. They now go through
smoke_rmtree, which retries so the disk is actually reclaimed -- these fixtures
hold ~300 MB binaries and runners are disk-tight -- then warns and continues.
Verified: Windows VM full smoke ALL PASSED (previously died after 13h), macOS
full smoke ALL PASSED, activation_transaction + cli 259 passed on Windows, all
10 shell contracts pass, lint-ci clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two Windows tests needed the startup thread to be observed HOLDING the startup
lock, and looked for it by polling the lock for BUSY up to 200 times. That state
has no lower bound: whenever the rendezvous handoff does not block, acquire ->
handoff -> release completes in microseconds, so the window can close between
two polls and startup_observed asserts on a coin flip. This blocked release run
30374135923 on test-windows (windows-2025, CLANG64, x86_64).
Budget-tuning the poll was tried and rejected -- a wider budget only lowers the
failure rate, and a transient window is never a fixed test. Instead the
production path exposes a gate that fires once the startup lock is held and
before the handoff runs, matching the hook pattern already used for POSIX
publication. The test parks the startup thread there and releases it when it is
ready, so the interleaving is pinned by construction. The acquired flag is
monotonic, so the observation cannot be missed; the remaining bound only catches
a thread that never started at all. The gate is inert unless a test installs it.
Also fixes an unrelated use-after-free this exposed. ipc_forever_wait_server
closed the connection and NULLed it from the server thread, while the test was
about to call cbm_daemon_ipc_connection_interrupt on the pointer it had already
loaded -- and receive_frame can return on its own before that call lands. It
crashed roughly one Windows run in five, independent of the change above.
Ownership now stays with the test, which closes after joining, so interrupt can
only ever see a live connection.
Verified on real Windows: 5 consecutive daemon_ipc runs, no failure and no
crash. macOS 46 passed x5 under ASan/UBSan; Linux 6606 passed, 0 failed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Windows smoke guards that an invalid PATH seam must fail closed rather than
silently falling back to the live HKCU\Environment\Path. That guard went green
in a way that meant nothing: the seam still refused, and install still printed
"PATH configuration failed", but the process exited 0.
Windows release builds used to compile a separate managed-install path. With one
binary per platform they compile the shared cbm_cmd_install, which since 30b5f12
downgraded ANY failed dry-run plan check to CLI_OK. That commit was fixing a real
complaint -- the dry-run summary was being skipped entirely on Windows -- but it
bundled the exit status into the fix. Keeping the summary is right; reporting
success is not. --dry-run exists to answer "would this install work?", so a
caller that only sees the exit code could not tell a refused PATH probe from a
clean plan.
The summary and the triage note still print; only the status changes. A real
install already failed non-zero here, so the fail-closed behaviour itself was
never at risk -- just its observability.
The smoke also still encoded the launcher-era portable-vs-managed split, which
does not survive one-binary-per-platform:
* 6b expected uninstall to be REFUSED. That refusal existed because a portable
extracted bundle was a different artifact from the launcher-managed install
it would have torn down. The extracted binary now IS the installed one.
* 6c expected update to be REFUSED, contradicting the contract Phase 14a
asserts against a real update: Windows hands off to install.ps1, exits 0 and
prints the command. 6c now pins that same handoff.
* 14a compared binaries with cmp, which is absent from the Windows MSYS shell,
so it reported "different" for two copies of one file. Hashes instead, the
way Phase 12 already verifies the release archive.
* 14f seeded the MCP entry at the RETIRED binary. The old launcher-managed
update rewrote that entry in-process; the handoff does not, so uninstall
correctly declined to remove an entry owned by a different installation and
the phase demanded the one thing it must never do. It now seeds the
installed binary, which is what install.ps1 leaves a real user holding.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Windows shipped a PAIR: a small permanent launcher (codebase-memory-mcp.exe)
plus the real product binary (codebase-memory-mcp.payload.exe). The launcher
existed for exactly one reason — a running .exe cannot replace its own image
on Windows, so an in-process self-update needs a second resident binary to do
the swap.
That stub is statically indistinguishable from a dropper: a small, unsigned,
zero-prevalence PE whose whole job is verify-and-execute another binary.
Defender's ML scored it Trojan:Win32/Wacatac.B!ml and blocked the v0.9.1-rc.1
release at the VirusTotal gate. It is not fixable in our code on x64 —
bcrypt-free, stripped, VERSIONINFO'd, minimal-resource and even
resource-FREE builds on CI's own MSYS2 CLANG64 toolchain were all flagged,
while the product binary scans clean on every platform.
So remove the stub and move self-update OUT of the process into install.ps1,
which runs while cbm is NOT running: Windows' image lock only blocks a
process from replacing ITSELF. now prints the exact PowerShell
command (with the Unblock-File hint for Mark-of-the-Web); install.ps1 is
idempotent, so re-running it IS the update — it stops the daemon, renames the
running binary aside (the one mutation Windows permits on a running image),
publishes the new one, and sweeps retired copies.
Windows now matches Linux and macOS: ONE binary per platform.
* packaging, install.ps1, npm and PyPI wrappers all carry a single binary
* the launcher/payload ABI contract and ~2500 lines of stub state machinery
are deleted
* every daemon start, CLI call and hook fire loses a process spawn, a named
pipe handshake and an stdio relay
* test_windows_bundle_contract.sh is rewritten as an INVERTED contract: it
now asserts no shipped surface can reintroduce a launcher/payload pair,
and that install.ps1 retires the running binary before publishing
Verified: VirusTotal 0/67 on the packaged binary and 0/58 on install.ps1 (no
certificate involved); macOS and Linux full suites green; Windows guards all
green including the new update-handoff contract; npm 10/10; PyPI 3/3.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Dry run 30206811293: macos-15-intel release smoke failed B3 with a
visibly correct 'semantic: 50' table — BSD grep returns no-match for
the ENTIRE output when any line carries a NUL, control byte, or
invalid UTF-8 (verified: printf poisoned input -> grep -qE fails), so
one bad byte in one row silently unmatches every anchored check.
Close the class at the emission choke point: control bytes and
invalid UTF-8 force the quoted form; quoting escapes controls as
\u00XX and replaces invalid sequences with U+FFFD (RFC 3629
validation, NUL-safe). Tool output is valid UTF-8 by construction.
Unit test proven RED-on-revert (raw bytes emitted) and green with the
fix. The smoke's B3 failure path now also dumps od -c bytes so a
future red names the exact byte.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Dry run 30206811293: windows-11-arm (both variants) and windows-2025
smoke failed in the update/uninstall E2E phases with 'private staged
copy failed secure open (error 32)' — ERROR_SHARING_VIOLATION. On the
newer hosted images a first-touch scanner transiently holds the
just-staged executable without FILE_SHARE_DELETE, so the launcher's
GENERIC_READ|DELETE retirement open collides on cold images.
Absorb the scan window with the same bounded backoff the staged
RENAME path already uses (<=~2.5s, exponential); every retry re-runs
the full secure-open validation, so no TOCTOU surface changes.
No deterministic local repro exists (the VM passes this phase with
Defender active; the collision is image-specific) — the next dry run
on the same three jobs is the arbiter.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This reverts commit f843db8, which removed the #581 fix on a false
attribution. The arm64 shards were crashing because of an unaligned SQLite
page-cache slab (fixed in #1274, arm64 shard 2/2 green since); removing this
callback never changed those crashes, which is what exonerated it. The Windows
soak then failed for the obvious reason: the leak was back.
Static MinGW links have no DllMain and register no TLS callback, so
mi_thread_done never runs and every thread leaks its heap -- 607 heaps after 300
requests holding 170 MiB against a ~300 KiB live set. With the callback the
10-minute Windows soak measured 1.1x against the original 53x.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The slab was a static char array handed to SQLITE_CONFIG_PAGECACHE. A static
char array carries no alignment guarantee and SQLite requires 8-byte-aligned
page-cache memory: x86_64 tolerates the unaligned access, arm64 faults. Every
suite that opens a store died instantly on windows-11-arm with
STATUS_ILLEGAL_INSTRUCTION (pass=0, secs=0) while pure-logic suites in the same
wave passed, and both arm64 shards were green on the commit before the merge.
It was also never the fix it was added for: measurement showed the slab
installed correctly and #581's growth unchanged, because the cause was thread
heaps never being released. Removing it rather than aligning it -- an unverified
optimisation has no claim on a release gate.
Note PR CI does not build the windows-11-arm leg, so this class of defect can
only surface in the dry run.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The refusal notes changed the error text the Windows launcher guards match
against, and test-windows-guards went red on this PR after they landed. They were
investigation instrumentation and they succeeded -- they turned 'staging
transaction open failed (status -3, os 0)' into ERROR_DISK_FULL and named the
exact site -- but a diagnostic that shifts a matched error string does not belong
in a release-blocking change.
Worth redoing deliberately on its own, with the guards' expectations updated in
the same commit.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The hosted windows-11-arm shards crash at process load (STATUS_ILLEGAL_INSTRUCTION,
pass=0, secs=0) while main passes them, and removing the TLS detach callback did
not help -- so the remaining candidate is this layer, which touches every
allocation on Windows through the interposer. It declares a _Thread_local guard
read on every malloc, and thread-local storage in a static MinGW aarch64 link is
exactly the fragile case; nothing in it needs to ship.
The profiler did its job: it produced the attribution that identified #581's
mechanism, and that analysis is recorded. Sources stay in the tree, unbuilt, so
a future investigation can re-enable them deliberately.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reverts the #581 fix mechanism, not the diagnosis.
main's dry run passes both windows-11-arm shards and test-windows-guards; this
branch fails all three. The only change here that affects process LOAD is the
.CRT$XLB detach callback, and the arm64 failures are load-time crashes
(STATUS_ILLEGAL_INSTRUCTION, pass=0, secs=0, five suites in the same second) --
while the guards exercise many launcher and payload spawns. The same binary
loads fine on a UTM arm64 VM, so the placement is tolerated by some loaders and
not others, which is not something to ship on a supported platform.
#581's root cause stands and is documented: a static MinGW link has no DllMain
and registers no TLS callback, so mi_thread_done never runs and every thread
leaks its heap (607 heaps after 300 requests, 170 MiB against ~300 KiB live).
The delivery mechanism needs to be one the loader accepts everywhere -- an FLS
callback registered at runtime is the obvious candidate, with no linker section
placement involved.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The copy loop's failure was reported as 'os 0' because the cleanup and close
calls after the loop clobber the thread's last error. Captures it at the failure
point, and separates a failed read from a failed write -- copying a payload that
another process still holds is a different problem from a full disk.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel