mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-10-21 05:59:35 +08:00
docs/cli-daemon-behavior
219
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Martin Vogel
|
94097f0727 |
Merge pull request #2341 from DeusData/fix/git-spawn-local-env
fix(git): clear git's repository-local env for cbm's own git children (#2003) |
||
|
Martin Vogel
|
c0a06b3d0a |
fix(sql): keep literal INSERT rows out of the SQL parse (#1735)
A mysqldump-style .sql file is a few CREATE TABLEs followed by megabytes of `INSERT ... VALUES (..),(..),...` rows. Tree-sitter built a full tree for every row (~70 bytes of tree per source byte; 0.26 s/MB with MySQL \' escapes, which the grammar does not know and error-recovers from) until the 5 s parse budget ran out, and the whole file, tables included, was skipped as "parse timeout". The reporter's database/sql/world.sql was dropped this way. The rows contribute nothing to the graph. A linear, quote- and comment-aware pre-scan (internal/cbm/sql_values.c) finds INSERT/REPLACE ... VALUES lists and excludes every tuple after the first that consists only of literals: strings (standard '' and MySQL \' escapes, X'..'/B'..'/N'.., _charset introducers), numbers, NULL, TRUE, FALSE, DEFAULT. A tuple holding a subquery, a function call, an identifier or a quoted identifier stays in the parse, and so does one whose string spans a raw newline. The excluded bytes go to ts_parser_set_included_ranges as the complement, so byte offsets and line/column positions of all kept text are unchanged. The ranges are cleared right after the parse because the parser is thread-local and reused. This is a syntax rule, not a work cap: the same file always gives the same ranges, and the parse budget is unchanged. Measured on generated mysqldump-shaped dumps, release build, same machine, index worker's own numbers (base = origin/main): dump extraction pipeline peak RSS graph 8 MB std '' 1310 -> 23 ms 1431 -> 139 ms 625 -> 37 MB identical 8 MB MySQL \' 2712 -> 24 ms 2827 -> 146 ms 474 -> 37 MB identical 32 MB MySQL \' 5145 -> 82 ms 5467 -> 396 ms 1132 -> 62 MB timeout -> indexed 59 MB MySQL \' 5130 -> 138 ms 5667 -> 674 ms 1157 -> 89 MB timeout -> indexed "identical" = every stored node and edge, Module end line included. For the 32/59 MB dumps the base run skipped the file as "parse timeout" (Module + 3 Tables + their DEFINES/USAGE missing). The fixed graph matches the 8 MB graph except for the Module's end line. With standard escapes the extraction and the stored graph are identical to the full parse. With MySQL \' escapes the full parse's error recovery swallows rows next to a misread escape, a subquery row among them, and the cut parse keeps those rows. So it loses nothing the full parse found outside the dropped rows and recovers what the full parse lost. The only things the full parse extracted INSIDE the dropped rows were junk usages (DEFAULT and _binary read as identifiers, fragments of misread strings) that name nothing. Tests (parse_coverage, pipeline; a CBM_TEST_SQL_FULL_PARSE_ON seam turns the exclusion off for one file so the same source can be compared both ways): - scanner cases: comments, strings, dollar quotes, ON DUPLICATE KEY UPDATE, VALUES outside INSERT, unclosed tuples, exact kept positions - extraction equals the full parse on a small dump (standard escapes); nothing lost outside dropped rows (MySQL escapes) - the subquery and function-call rows are still parsed - the tree does not grow when the row count doubles (a count, not a clock) - a 25 MB MySQL dump under the production budget is indexed, not "parse timeout" (asserts the outcome, not the time) - stored nodes and edges of a pipeline index equal the full parse's Refs #1735 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
40dad610ee |
fix(git): clear git's repository-local env for cbm's own git children (#2003)
Git exports repository-local variables (GIT_DIR, GIT_WORK_TREE,
GIT_INDEX_FILE, GIT_COMMON_DIR, ...) into hooks, and some editors export
them into their terminals. They take precedence over `git -C <dir>`, and
every git child cbm spawned inherited them. A cbm started from such an
environment therefore resolved every project against the CALLER's
repository: git_context reported a plain directory as is_git=true with
the caller's worktree root, branch and HEAD, the history pass attributed
the caller's commits to it, and the watcher / detect_changes /
artifact git calls read (and `config merge.ours.driver` could write) the
wrong repository.
Fix: one production helper, src/foundation/git_env.{c,h}, holds the
15-variable list `git rev-parse --local-env-vars` prints and builds the
child environment with exactly those removed -- a POSIX envp array or a
Windows CREATE_UNICODE_ENVIRONMENT block (case-insensitive names). It is
applied per spawn; cbm's own environment is never modified (no unsetenv
in the parent, which would race other threads):
- cbm_popen_git(cmd): read-mode popen for git command lines. POSIX spawns
/bin/sh -c via posix_spawn with the scrubbed envp (stdin /dev/null,
pipe ends close-on-exec, reaped by cbm_pclose); Windows passes the block
to the existing handle-isolated CreateProcessW spawn. Fails closed.
- cbm_proc_opts_t.strip_git_repo_env: cbm_subprocess_spawn passes the
scrubbed envp to posix_spawnp / the fork+exec child, or the block to
CreateProcessW. Fails closed.
- Every git spawn uses them: git_context, pass_githistory, artifact
(4 sites), the watcher's git runner, and detect_changes' git commands.
search_code's grep scan keeps the inherited environment.
Tests (RED before, green after, RED on revert of the helper and of the
call sites):
- git_context::git_context_ignores_inherited_repo_env -- GIT_DIR and the
hook set pointing at a decoy repo: a plain dir is is_git=false, a
fixture repo reports its own HEAD and branch, the decoy is untouched.
- git_context::githistory_ignores_inherited_repo_env -- the history pass
on a plain dir sees 0 commits (was the decoy's 4).
- subprocess::subprocess_strip_git_repo_env_is_per_child and
popen_git_strips_repo_env_and_reports_exit_status -- all 15 variables
are gone in the child, unrelated variables pass through, the parent
environment is unchanged, and cbm_pclose keeps pclose() status
semantics.
Proof (macOS arm64, GIT_DIR/GIT_WORK_TREE/GIT_INDEX_FILE/GIT_PREFIX
exported at a decoy clone, `cli index_status verbose=true`): a plain dir
went from is_git=true, worktree_root=<decoy>, branch=<decoy's> (v0.11.0)
to is_git=false; a fixture repo went from the decoy's root/branch/HEAD
to its own root and branch.
Refs #2003
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Clément DeustandClaude Fable 5
|
f3bdf11a03 |
feat(pipeline): link markdown file references into the graph
Markdown docs reference other repo files constantly (a coding-standards doc links the modules it governs, a README points at entry points) but none of that surfaced as edges, so fan-in queries were blind to documentation hubs. In an A/B retrieval benchmark I ran in August on a docs-heavy repo, the graph's top fan-in answer was off by roughly 17x: the most-referenced file was a standards doc with 173 references and zero inbound edges. New pre-dump pass, pass_doclinks.c, modeled on pass_configlink.c: three strategies emit REFERENCES_FILE edges between existing File nodes only (unresolvable targets are dropped, the pass never invents nodes): inline links, backtick paths, bare path mentions. Targets resolve against the referencing file's directory and the repo root; repeated references collapse into one edge carrying strategy, confidence and count. Registered in the pre-dump sequence after configlink and in the incremental post-passes; REFERENCES_FILE added to the skill's edge-type list and the structural/language contract tests. Tests mirror test_configlink.c (real files in a tmpdir, File nodes in a gbuf, run the pass, assert edges): inline link, backtick, bare mention, http/anchor ignored, anchor-suffixed file link, dedupe with count, relative-vs-root resolution, unresolvable-target guard, NULL repo_path skip. A companion change adds the same linking for shell files (source lines and script invocations); split out to keep each change reviewable. Validation: scripts/build.sh clean; focused serial runner (doclinks, configlink, pipeline, edge_structural, lang_contract) 340 passed under ASan/UBSan; cppcheck clean on the new files with the repo's flags. Signed-off-by: Clément Deust <clement.deust@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Martin Vogel
|
92abefa3f5 |
Merge pull request #1723 from liuchong/feat/index-resource-discovery
feat(index): add opt-in discovery resource limits |
||
|
Martin Vogel
|
6f290e11d0 |
build: make each grammar object depend on the sources it includes
A grammar unity object, grammar_<lang>.c, #includes its own vendored parser.c and, where the grammar has one, scanner.c. Those files were prerequisites of nothing. Editing a vendored scanner therefore left the object untouched and the edit silently absent from the binary. This is not hypothetical: the properties scanner fix in the previous commit appeared not to work at all. An A/B said 3 stalls against 2 - no effect - and the reason was that prod_grammar_properties.o predated the edit by five minutes and `nm` still showed the old `_reached_eof` symbol sitting in it. The measured "no effect" was a stale object, not a bad fix. The build already treats its other two unity objects this way and says why: LSP_UNITY_DEPS and TS_RUNTIME_DEPS both list the sources and headers they pull in, because this Makefile deliberately does not use compiler auto-dependency (-MMD) generation. Grammars were the gap, and MANIFEST.md's "Local source patches" table means patching a vendored scanner is an expected thing to do - so the trap was reachable by design. Secondary expansion is what lets one pattern rule name each grammar's own directory ($$* is the stem, grammar_properties -> properties). Applied to all three grammar object rules: production, test and tsan. Non-grammar objects match no directory and gain no prerequisite. Verified by touching internal/cbm/vendored/grammars/properties/scanner.c and watching exactly one object rebuild. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
81264814f1 |
feat(qa): memory + CPU waste sanitizer over the memory core
Answers a question the existing tooling could not: of everything we
allocate, how much is never used?
Lanes, all driven by scripts/memwaste.sh:
event every allocation observed through the memory core — size, age,
churn, and whether the bytes were ever written
access SanitizerCoverage load/store instrumentation, so "allocated and
never touched" is measured rather than inferred
scaling k vs 2k replicas through clang profile counters, to catch
super-linear work per file
padding allocator slack per size class
ratchet the numbers recorded per venue so a regression is visible
The event lane needed a foreign-free counter: on Windows the override sees
frees for blocks it never allocated (they come from the CRT), and counting
those as untracked tripped the soundness gate. mem_events now separates
them (14 daemon / 5 worker on the Go corpus).
Two portability fixes the lanes surfaced, both real:
- the enable decision must not be cached before `environ` exists. Under
-fsanitize-coverage the UBSan preinit runs dlsym -> malloc -> our
wrapper before glibc sets up environ, so the lane silently disabled
itself and produced no dump at all on Linux.
- the access map needs 48 bits: arm64 Linux hands out addresses above
2^47, which a 47-bit map dropped on the floor.
mem_profile is retired into this: it measured a strict subset and had its
own sampling story.
Local-only by decision — not a PR-CI gate. _memwaste.yml runs the lanes on
all three platforms report-only, and pr.yml/release.yml carry the wiring.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
7b77fd4834 |
feat(mem): memory on the core, waste cut, results spilled under budget
Second increment on the memory core. The first landed the core, the linter
and phase attribution with zero production call sites; this one migrates the
memory that matters, proves where it goes, removes the waste the proof
exposed, and makes the memory budget a promise: over budget, extraction
results go to disk and the run completes at the floor instead of aborting.
Every number below is from a real index on this machine (M5 Pro, 48 GB) with
CBM_MEM_PHASES=1 and the release build.
WHAT THE PROOF SAID (Go corpus: 21,875 files, 345 MB of source)
The graph buffer, blamed for the peak since the 2026-07 research, is 3% of
the memory: 0.55 GB peak. The per-file extraction results are the memory:
14.8 GB written into their arenas, of which 3.4 GB is reachable. The rest
is temporaries -- cbm_node_text copies at 496 call sites, per-node QN
strings, abandoned generations of every growable array -- that an arena
cannot free and that the result kept alive until resolve ended, because
the result owned the arena. The arenas were charged 28.8 GB of capacity
for those 14.8 GB. On the kernel the same shape was 8.9 GB used by 89,731
results, and resolve grew it to 13.3 GB used / 17.2 GB capacity because
the C walk allocated its working set in the result arena and stored arena
pointers into the shared registry. After that, 7 GB of the kernel's
post-resolve memory had no class at all.
WHAT CHANGED
Attribution. The graph buffer (87 raw sites -> 0), arena blocks, the
tree-sitter slab allocator, the bound tree-sitter/SQLite allocators, the
semantic path (pass_semantic_edges, pass_similarity, minhash: 80 sites), the
hash table (Verstable, a class per table via CTX_TY: cbm_ht_create_in) and
CBM_DYN_ARRAY allocate through the core. Classes arena, ts_tree, hash_table
and dyn_array join the table. Two cross-boundary frees of buffer-owned
strings (pass_importance, pass_complexity) go through a new setter,
cbm_gbuf_node_set_properties_json. The linter baseline drops 4086 -> 3916.
Result compaction (internal/cbm/result_compact.c). At the end of a file's
extraction, everything reachable from its result is measured, copied into
one exact-size arena (strings interned by content within the file) and the
working arena is handed back. Later appends restart growth at the default
block (cbm_arena_init_exact, arena grow_size). The stale duplicate
internal/cbm/arena.h, which shared the foundation header's include guard
and won or lost by include order, is a shim now.
Record slimming. CBMCall 320 -> 72 bytes: the eight inline argument slots
(256 bytes, mostly empty) are allocated on first capture. CBMUsage 48 -> 40.
The per-file overlay contract (src/pipeline/pass_lsp_cross.c). One
lifecycle for every language's cross-file resolve: a scratch arena, a
per-file overlay registry chained to the sealed shared base, the walk
against the overlay, resolved calls appended into the result, scratch
destroyed. type_registry gains chain-aware iterators (every yielded index
belongs to it->reg) and copy-on-write refinement
(cbm_registry_func/type_for_update), and the C, C#, Go and Python walks
use them. This is what an ASan heap-use-after-free in the
lsp_resolution_probe suite demanded: the C walk stored per-call arena
objects into the shared registry, and a kernel run with a naive scratch
arena lost 1,045 edges to the resulting garbage reads. It also ends
unsynchronized writes into the shared registry from parallel resolve
workers.
The 7 GB with no class, found and cleaned. gbuf_index tables and keys were
2.5 GB live and 7.1 GB at peak: the 18 worker buffers built the five
secondary indexes they never query (cbm_gbuf_new_worker builds none), and
18M per-key index arrays started at 8 slots (cbm_da_push_min starts them
at 2); edge property JSON is 81% "{}" (interned). The semantic pass
allocated a 512-token stride per function up front (packed tokens with
offsets: cbm_sem_corpus_add_docs_batch). Edge dedup keys are 128-bit
hashes instead of sprintf'd strings (edge_key_map_t). Each worker keeps
one reusable working arena between files (cbm_work_arena_take/give:
rewind, not free) and every phase mark returns freed pages to the OS.
Budget metric. The over-budget check compares the charged footprint --
max(phys_footprint, mimalloc commit) on macOS -- not resident_size: after
extraction the kernel worker sat at 17.4 GB RSS with 5.5 GB charged, because
pages mimalloc has purged stay resident until the OS wants them, and
phys_footprint under-reports after MADV_FREE_REUSABLE cycles.
Spill / admission control (internal/cbm/result_spill.c, the extract gate
in pass_parallel.c). At 15/16 of the budget (or over it, or with
CBM_MEM_SPILL=1) the gate latches spill mode -- early, because the gate
sees the crossing per file pull and the workers' in-flight files carry the
charge past the line before the first sweep lands: every compacted result
is parked on disk as header + block in one append-only file per writer
under <cache>/spill/, the results already cached are swept out by every
worker, and registry build, def collection, surface rows, resolve and the
infra-route passes load a result only for the moment they read it
(relocated by base delta). The abort path stays closed while a sweep runs
or a cached result remains; it fires only at the floor -- graph +
registries + in-flight files. A result that owns sub-results stays in
memory; a retained parse tree is dropped with the parked result and
re-parsed on load. Every consumer of the result cache reads a slot through
cbm_pipeline_result_acquire()/release() (pipeline_internal.h): a parked
slot is NULL in the array, and the passes that indexed it directly lost
every __route__infra__ node and its INFRA_MAPS edges before the contract.
The other things the transient loads found: a def label that was the
result's own pointer (SIGBUS in the Go cross registry builder), two JVM
helpers that passed their input through, a dedup table whose keys borrowed
sr->value (the insert spun on freed memory), and a latch that was visible
before the store was open (a peer found nothing to park and failed the run
while its neighbour wrote 9 GB). The arena rewind reuse branch had
defeated the nblocks = CBM_ARENA_MAX_BLOCKS OOM seam the suites use; it
reuses only blocks that exist. The incremental and probe routes still
index the cache array themselves and therefore never spill
(ctx->spill_allowed stays false there; follow-up).
The semantic pass under a budget (src/pipeline/pass_semantic_edges.c). The
pass held every function's tokens, the corpus's per-document token ids and
the vectors being stored at once: 4.9 GB on the kernel, on top of the
floor, and nothing sized it to the headroom. Now, when the charged
footprint plus that transient would cross the budget, phases 2-4 run per
batch of functions: tokenize -> count into the corpus -> free; finalize
once; tokenize again -> vectorize -> store -> free, with pages returned to
the OS after every batch. Tokenization is deterministic, so both passes
see the same tokens and the graph does not change; the run pays with a
second tokenize. CBM_SEM_BATCH=<n> forces the batch size (the
batched-equals-unbatched test). The corpus frees its per-document token
ids at the end of finalize (the co-occurrence pass is their only reader),
and mem.semantic.step lines (CBM_MEM_PHASES=1) put the class's live/peak
bytes and the charged footprint at every sub-phase.
The semantic corpus and tokenizer allocate through the core (54 raw sites
-> 0). This is not attribution only: cbm_sem_tokenize handed out libc
strdup blocks that pass_semantic_edges freed with cbm_free -- mi_free in
the production build -- since the pass moved onto the core. No libc-backed
test build can see a cross-allocator free, so the core now refuses a block
it does not own (mi_is_in_heap_region) when CBM_MEM_PHASES=1 is set,
fatally: every proof run in this PR ran with it armed, and the Go and
kernel indexes report zero foreign blocks.
Accounting without contention. The core charged every allocation and free
to shared per-class atomics; with 18 workers on 18 cores every block bounced
the same three cache lines, and the bench against the shipped v0.10.8 showed
it as CPU time rising faster than wall time for the same graph: Kotlin 38 ->
121 s CPU (+58% wall), TypeScript 210 -> 596 s (+67%), Go 177 -> 312 s. The
deltas are thread-local now and reach the shared counters once per 256 KB or
512 blocks of change per thread, when a parallel-for worker ends, and
whenever a reader looks (its own thread first). Phase marks read after the
join and are exact; a class peak is a diagnostic and can be low by threads x
256 KB. Result: Kotlin 10.9 s / 43 s CPU (the residual +22% wall against
v0.10.8 tracks its +34% nodes), TypeScript 33.9 s (v0.10.8: 33.4), Go 48.4 s
(v0.10.8: 52.6), Java 81.1 s (v0.10.8: 79.8); graphs unchanged.
SQLite on a heap of its own. The same bench put the kernel's write step at
132 s where v0.10.8 needed 9.7 s, all of it in one coverage publish
statement. A stack sample named the cost: mimalloc's free-page search and
its periodic heap collect, walking the graph's pages -- millions of them
once the graph lives on the core -- for every statement-journal chunk
SQLite allocates. SQLite now allocates from a mimalloc heap of its own per
thread (mi_heap_new; mi_free is heap-agnostic, so nothing else changes),
and the periodic collect is set to the maximum mimalloc allows since the
explicit release at phase marks and spill sweeps is where memory goes
back. Coverage publish 132 -> 9.4 s; profiled kernel 376 -> 248 s
(v0.10.8: 296 s).
Formatting without a process lock. On macOS every vsnprintf takes the
process locale under an unfair lock (localeconv_l inside __vfprintf);
eighteen workers formatting type names on the C# corpus collapsed into
it -- the 10 MB JIT test files took 68 s each instead of under a second
-- and cbm_arena_sprintf paid it twice per string (size pass, write
pass). Each thread now formats with a C locale object of its own
(vsnprintf_l), and the common short string is formatted once into a
stack buffer.
Per-file budgets that cover the whole file. The 5 s per-file budget
covered the parse only; the LSP walks after it had none. C# JIT stress
files (a 23 MB single expression among them) parsed inside the budget --
the old wall-clock parse timeout used to drop them, which hid the rest --
and then held a worker for 346 s each in the usage walk (tree-sitter's
ts_node_parent descends from the root: quadratic on a deep tree), and the
cross-file resolve ran on the same trees. Three rules now, one site for
every language (cbm_extract_file_ex, honoured by cbm_pxc_dispatch_file): a
parse that used more than half the budget disqualifies the file from the
LSP walks (lsp_skipped); the unified walk checks its thread CPU time every
1024 nodes against six budgets and stops there, keeping what it found
(walk_truncated, implies lsp_skipped); a file whose parse plus walk spent
the budget is skipped by the LSP walks as well. Each decision is logged
with the path (extract.lsp.skipped, extract.walk.truncated). C# extraction
355 -> 39 s; the whole index 463 -> 128 s wall and 27 -> 11 GB peak
against v0.10.8 with 299 more nodes. The walk cuts exactly the five JIT
stress files (hugeexpr1, hugeSimpleExpr1, HugeArray1, HugeField1/2); the
15 MB System.Runtime.Intrinsics reference file walks to the end and, with
four generic-nesting JIT regression tests, only skips the LSP walk under
the parse-plus-walk rule.
The crash behind the C# bench. dotnet/runtime is 42,555 files, and its
resolve phase died with SIGBUS on every run of this branch. A libc-backed
ASan build of the server named it: heap-use-after-free in c_adl_resolve,
reading a type name that another worker had allocated in its per-file
scratch arena and freed at the end of its file. The C++ class walk
refines a method's return type when the declaration is more specific
than the pre-registered one (NAMED -> pointer/reference/template), and
it did that by casting the chained lookup result to non-const and
writing a scratch-arena signature into it -- into the sealed shared base
when the entry lived there. The overlay contract had exactly one
bypass, and it was this cast; every lookup returns const, so a grep for
the cast is the audit. The refinement now goes through
cbm_registry_func_for_update: copy-on-write into the overlay, the base
untouched. Test: clsp_method_return_refinement_is_copy_on_write (a
sealed base with a NAMED method return, an in-class pointer declaration;
the base signature pointer is unchanged and the overlay copy carries the
refinement plus a marker only a copy has) -- RED on the cast, GREEN on
the accessor. The fixed ASan build then indexed the whole
corpus clean: 42,555 files, 1,224,981 nodes, 5,794,320 edges, worker
exit 0, no report.
Production build backs the core with mimalloc explicitly (never the global
override, which macOS's two-level namespace forbids); the test build keeps
libc so the sanitizers see every block -- the same split the tree-sitter and
SQLite bindings already use.
PROOF
Go peak RSS 16.7 -> 4.0-5.2 GB; result arenas 14,804/28,817 -> 856/856
MB; wall 55 -> 52-60 s; graph inside the pre-existing run-to-run
jitter (gRPC Route nodes: two identical runs differ by 21).
Go CBM_MEM_SPILL=1: peak RSS 1.7-2.3 GB, 21,875 results parked (950 MB
on disk), graph identical: 39 infra routes, 158 INFRA_MAPS, 21,875
surface rows, nodes/edges inside the Route jitter.
Go CBM_MEM_BUDGET_MB=1200: spill latched at 1,204 MB, the first sweep
took the charge to 768 MB, the run completed at 1.99 GB peak RSS.
Kernel DEFAULT budget (24,576 MB): aborted before, completes now without
spilling. Extraction 35.2 -> 16.5 GB RSS (5.6 GB footprint, 15.3 GB
committed); results 8.9 GB = capacity; resolve tracked 24.6 -> 19.1
GB; peak RSS 27.3 -> 20.1 GB. Nodes identical (8,529,729); edges
inside the +-400 run-to-run band the kernel shows between identical
runs.
Kernel CBM_MEM_BUDGET_MB=16000: aborted before, completes now through the
spill path. Latched at 17.2 GB charged; 89,731 results parked (9.3
GB on disk, 370,188 loads); extraction end 6.2 GB RSS / 5.1 GB
committed; resolve 8.6 GB RSS / 11.5 GB committed; peak RSS 16.6
GB (the semantic pass plateau, 13.3 GB RSS, is graph-derived and
outside what spill can move). Nodes identical; edges 15,773,598.
Kernel CBM_MEM_BUDGET_MB=15000, before the semantic batching: completes,
every phase mark under 15,000 MB by every metric, but the process
RSS high-water mark reached 16.9 GB inside the semantic pass (class
peak 4.9 GB against 0.8 GB live).
Kernel CBM_MEM_BUDGET_MB=15000, with it: completes; the charged footprint
stays under 15,000 MB through the whole semantic pass (14,018 ->
14,381 MB across its sub-phases, 6 batches of 125,644 functions);
every phase mark is under budget by every metric (max footprint
12.6 GB, max commit 11.5 GB); the process RSS high-water mark is
15.65 GB and is set in extraction, at the moment the gate latches
(15,002 MB charged) and before the first sweep lands -- a 4%
overshoot of in-flight work, no longer the semantic pass. Nodes
identical, edges 15,772,913, zero foreign blocks.
Kernel CBM_MEM_BUDGET_MB=15000, with the anticipatory latch: spill mode
enters at 14,062 MB (near_budget); the charged footprint never
exceeds 11.8 GB at any mark; the process RSS high-water mark is
15.1 GB in extraction and 15.3 GB during the final write-out --
1-2% over, the in-flight window and the dump transient. Nodes
identical, edges 15,772,543, zero foreign blocks.
Go CBM_MEM_BUDGET_MB=1200 with it: latched at 1,131 MB, extraction
peak 998 MB RSS, graph identical to the baseline.
BENCH AGAINST THE SHIPPED v0.10.8 (same machine, same driver, 2026-09-14)
corpus wall s peak RSS GB nodes / edges / CALLS
perl 5.7 -> 5.3 0.09 -> 0.05 -3 / -130 / -3
php 6.3 -> 5.8 0.66 -> 0.20 +21.9% / +1.4% / +23.8%
rust 6.8 -> 6.3 1.15 -> 0.67 +1 / +56 / -369
c 5.9 -> 5.7 0.16 -> 0.19 identical
kotlin 8.9 -> 9.4 1.55 -> 0.96 +33.5% / -22.8% / +12.1%
django 10.9 -> 10.2 3.13 -> 1.12 +6 / -14.5% / -28.4%
go 52.6 -> 42.9 17.2 -> 3.9 +14.2% / -10.3% / +2.2%
java 79.8 -> 73.8 27.6 -> 11.0 -121 / -1.9% / -0.6%
csharp 463.3 -> 128.1 27.0 -> 11.2 +299 / -0.9% / -0.6%
typescript 33.4 -> 33.6 12.0 -> 2.5 -186 / -3.5% / +130
kernel 294.8 -> 233.3 27.5 -> 25.1 -171 / -1.3% / -0.1%
No corpus is slower beyond noise; CPU time follows wall (kernel 2,442 ->
1,403 s). Every graph delta against v0.10.8 is main between the release and
the branch base (
|
||
|
Martin Vogel
|
da3258f3c7 |
feat(mem): a central allocation core, a linter that enforces it, and phase attribution in the index pipeline
Memory in this project is now allocated through ONE core. This lands the
core, the gate that keeps new code on it, and the instrumentation that lets
an index run say which class of memory grew in which pass -- the three things
the 2026-09-13 audit found missing.
WHAT THE AUDIT FOUND
There was no centralized allocation layer. foundation/mem.h was policy and
measurement only (budget, RSS, over_budget, memory map, phase marks) with no
alloc/free wrapper, so the budget could be observed after the fact and never
enforced. Raw allocator calls in the tree: 4086 across 92 files. The
extraction engine had already adopted arenas (1301 call sites); the graph
buffer -- which holds the 35 GB peak of a kernel index -- used none,
allocating every 64 B node, 48 B edge and every name/qualified_name/
properties_json individually.
Two further facts made a self-accounting core the only workable design:
1. cbm_mem_map_collect() walks THIS thread's mimalloc heap only. Walking
the process-wide heap is a data race TSan caught on macOS, so it is
deliberately not done, and an 18-worker index attributes almost
nothing -- the rest lands in residual.
2. The mimalloc global override is ON for Linux/MinGW and permanently OFF
for macOS (two-level namespace: this binary's free becomes mi_free while
system libraries keep allocating from the system zone, and a pointer
crossing that boundary aborts). On macOS ordinary malloc is served by
the system allocator; the startup audit reports owned_classes=0/6. Any
accounting that assumes mimalloc owns the pointer is blind on an entire
platform.
THE CORE (src/foundation/mem_core.{h,c})
cbm_alloc / cbm_calloc / cbm_realloc / cbm_mem_strdup / cbm_free, each tagged
with an allocation class (gbuf_node, gbuf_edge, gbuf_string, gbuf_index,
extract, semantic, dump, store, other). Per-class live bytes, live blocks and
peak, kept in atomics -- thread-safe by construction, identical on every
platform, independent of which allocator serves malloc. libc semantics are
preserved exactly (alloc(0) is freeable, free(NULL) is a no-op, realloc(NULL)
allocates, a failed realloc leaves the block intact) so adoption is a
mechanical rename and never a behaviour change.
No per-allocation header: a {class,size} prefix costs 16 bytes on ~100M blocks
at kernel scale, 1.6 GB of overhead to measure a memory problem. Sizes come
from the platform usable-size query (malloc_size / malloc_usable_size /
_msize), which is correct under either allocator and free. Where no query
exists (BSD) the request size is charged, which understates -- the safe
direction for a diagnostic. A mismatched class on free clamps at zero rather
than wrapping, so a small drift can never masquerade as a colossal leak.
Arena-backed allocators report in bulk through cbm_mem_class_add_external /
remove_external so their memory appears in the same table; the extraction
engine must not be rewritten to per-object allocation, that would undo the
batching that keeps its allocation count low.
THE LINTER (scripts/lint-memory-core.py, wired into make lint-ci)
Raw malloc/calloc/realloc/free/strdup/strndup outside the core is a defect.
4086 sites cannot migrate at once, so the gate is a ratchet: a checked-in
baseline (scripts/memory-core-baseline.txt) records each file's count and
the build goes red when any file grows, or a new file appears with any.
Files only ever go down; the baseline line is lowered in the change that
migrates the file, and --strict fails a stale baseline. Comments and string
literals are stripped first, so prose that mentions malloc( cannot trip it --
the security audit already bit on exactly that with fork(. Proven both ways:
one appended malloc turns it red naming the file (83 -> 85), restoring turns
it green. Scans src/ and internal/cbm/: cli, mcp, daemon, store, pipeline.
PHASE ATTRIBUTION (src/pipeline/pipeline.c)
cbm_mem_phase_mark and the new cbm_mem_class_log now fire at pipeline.begin
and at every pass.timing site: parallel_extract, the six sequential passes,
the eight predump passes, tests and dump_and_persist, with peaks reset per
index. Both instruments existed in foundation/ and were wired only into MCP
request handling, never into the index pipeline -- which is where the memory
is. Locating the kernel peak this week required an external RSS sampler
because nothing in-process could say which phase it was in.
PRESSURE PRIMITIVES (system_info.c, mem.{c,h}, platform.h)
cbm_system_available_ram() -- macOS host_statistics64 (free + inactive +
purgeable), Linux MemAvailable, Windows GlobalMemoryStatusEx; 0 when unknown,
never cached -- and cbm_mem_system_under_pressure(), true below 12.5% of RAM
available and false when unknown so nothing ever aborts on a guess. Landed
UNWIRED: an earlier draft used them to let the over-budget latch press on
whenever the system was not under pressure, and that permitted the very
overshoot this work exists to end (Decision A's tests at
test_pipeline.c:13481 and :13583 went red under it, correctly). They are the
backstop for the next step, admission control, which keeps a run under budget
before an allocation rather than measuring it after.
mem suite 10 new tests green; pipeline + mem 339 passed 0 failed; lint-ci
clean including the new gate.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
刘冲
|
21ae02154a |
feat(index): add opt-in discovery resource limits
Indexing accepts whatever a repository contains. A tree carrying a vendored monorepo, a generated dump, or a runaway build directory is discovered in full, and the first sign of trouble is a host under memory pressure with nothing that attributes it to indexing. Add two opt-in limits evaluated during discovery against accepted source files only: index_max_files and index_max_source_mb. Both default to off, so nothing changes until an operator sets one. Crossing a limit fails the whole attempt with a structured resource_limit_exceeded result naming the resource, the observed value and the limit; no partial graph is published, and an existing serving index keeps answering. Limits are read from the CLI-managed _config.db and are not MCP request arguments. A supervised parent replaces any caller-supplied policy before spawning its worker, and the worker rejects a missing or incomplete contract, so the CLI, the daemon and the supervised worker all enforce the same decision. Both keys reach an operator through the existing config get/set/list/reset with no new subcommand. `set` suppresses its own generic message for them because the policy writer names the precise reason -- so that writer speaks on every failure it can return, including a validated value whose write then fails on a database that cannot be written. Exiting non-zero in silence is not an acceptable answer from a CLI. The two shell regressions that hand-roll the supervisor's worker argv carry that contract as well. Without it the worker exits before either guard can observe anything, and the guard would go quietly vacuous. Signed-off-by: 刘冲 <mail@liuchong.dev> |
||
|
Martin Vogel
|
d40ed1a6bb |
Merge remote-tracking branch 'origin/main' into fix/1954-uninstall-symlinked-config
# Conflicts: # tests/test_cli.c |
||
|
Martin VogelandClaude Fable 5.1
|
e5aabdf86e |
fix(cli): follow user-owned symlinked config files in the editors
Every config editor (JSON-like, TOML, YAML, text) opens with O_NOFOLLOW and refuses symlinks, so a link planted at ~/.cursor/mcp.json cannot point a privileged writer at another file. Users keep exactly those files as symlinks into a dotfiles checkout; the blanket refusal made install and uninstall fail on them and, before the previous commit, turned uninstall into data loss (#1954). Decision C: a symlink the INVOKING user owns, resolving to a regular file the same user owns, is followed — as an explicit opt-in of the agent-configuration writers only. cbm_install_agent_configs and the uninstall activation register the user's configuration roots (HOME, XDG_CONFIG_HOME) for the duration of their work; a link whose own directory is outside every registered root, or any link seen by another editor caller, stays refused, so repository paths never follow. One shared helper (src/cli/config_edit_path.c) runs the race-safe sequence: lstat(link) → owner == invoking uid (a root process follows root-owned links only) → realpath → the target's parent directory opened as a dirfd (O_DIRECTORY|O_NOFOLLOW; owned by the invoking user or root, not writable by others unless sticky) → openat(dirfd, base, O_NOFOLLOW) → fstat: regular file, same owner, link count 1, and the same (dev, ino) as stat(realpath). Each editor reads through that validated descriptor, stages its temp file with openat(dirfd, …, O_CREAT|O_EXCL|O_NOFOLLOW), runs its pre-publish snapshot comparison against the resolved path, and commits with renameat on the pinned dirfd, so the link itself is never replaced. Every other link is refused with an observable reason (surfaced in the uninstall diagnostics): another owner, root over a non-root link, dangling, a directory or special target, a hard-linked or swapped target, a foreign or world-writable parent. Whether the target must also stay under a root is a pending user decision behind CBM_CONFIG_EDIT_TARGET_UNDER_ROOT (default: anywhere the user owns). Windows never follows reparse points. Tests: each editor suite gains a same-owner link edited in place under the opt-in (link preserved and still pointing at the same target, temp file staged beside the target, bytes identical to the same edit on a plain file, no stray temp files) and keeps its refusal cases; the JSON-like suite adds no-opt-in, outside-root, dangling, directory, identity-skew and world-writable-parent refusals. The foreign-owner and root-run refusals move the observer through a test seam (a test cannot create a foreign-owned link without root; a real root run demotes the link first). test_cli.c drives `uninstall -y` end to end through a symlinked ~/.cursor/mcp.json. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
fbe9eae11c | Merge branch 'main' into fix/1692-csharp-multi-attribute-decorates | ||
|
Joshua RichterandClaude Opus 5
|
219b348263 |
test(coverage): pin the truncation marker, the partial ceiling and the grammar limit (#963)
Phase 5. Four test groups, each checked RED before it was kept. - The Studio Export range join puts ONE ",+<N>" marker at the end with the summed drop count. A marker left mid-string makes every reader stop there and silently lose the ranges after it. Reaching the join through the pipeline needs an export file with 256+ error regions across two <Class> elements, so it goes through a test seam, following the pattern already in this repo (CBM_COVERAGE_MARKER_TEST_API). - check_index_coverage emits every range in front of a marker, never turns the marker's digits into a range, and reports "truncated" from BOTH caps — the producer's and its own 256 limit. - test_index_resilience now has a ceiling beside its floor: exactly one of the two fixture files is flagged, the clean neighbour is absent, and the range does not cover the whole file. - The three _Thread_local forms are pinned as measured. Only the array form fails today; the plan's Phase 0 also listed the pointer form, and that is wrong on the grammar shipped now. Also fixes 13 clang-format violations the earlier commits on this branch left in cbm.c, mcp.c and pass_definitions.c. `make -f Makefile.cbm lint-format` would have failed CI. The changes are whitespace only — the two reflowed tool descriptions concatenate byte-identically, so no output moved. Full suite: 7735 passed, 28 failed, 7 skipped. The 28 are the pre-existing cli install/uninstall failures, identical at clean HEAD. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Joshua Richter <jrichter5781@gmail.com> |
||
|
Martin Vogel
|
683d4bc312 |
Merge branch 'main' into fix/1692-csharp-multi-attribute-decorates
Brings the branch onto current main (
|
||
|
Martin VogelandPR9000
|
fece72b049 |
fix(build): put the whole GOT inside the RELRO window on ELF targets
Adds -Wl,-z,relro and -Wl,-z,now to ELF_HARDENING_FLAGS, and asserts the
resulting property on the produced artifact rather than on the compiler's
willingness to accept a flag.
The shipped Linux binaries measurably lacked the property. Built the way
.github/workflows/_build.yml builds them (STATIC=1, gcc 13.3 / GNU ld 2.42,
Ubuntu 24.04 aarch64), PT_GNU_RELRO ends at 0x11DA0000 in both cases:
without -z now: .got 0x11d9ec20+0x13c8, .got.plt 0x11d9ffe8+0x40
-> .got.plt ends at 0x11DA0028, 40 bytes PAST the window.
Those GOT slots stayed writable for the whole process
lifetime, in every Linux binary shipped to date.
with -z now: .got.plt folds into .got, the whole GOT sits inside the
window and is re-mapped read-only after startup.
Both flags are free at runtime for a static binary -- everything resolves at
link time, so there is no lazy binding left to pay for. -z relro is already
this toolchain's default and changes nothing here; it is named anyway so the
property stops depending on one distro's spec file, since the musl/portable
and glibc-floor images are different toolchains and a default is not a
guarantee.
A1c/A1d in scripts/ci/check-binary-composition.sh assert the OUTCOME on the
binary via readelf, for the same reason A1 exists beside the .note.GNU-stack
annotation: a flag the linker accepts is not evidence the artifact gained
anything. PR #1138 demonstrated that directly -- its -pie was silently
discarded under -static, and its -D_FORTIFY_SOURCE=2 silently DOWNGRADED
Ubuntu's default of 3, both while every compile probe passed.
The readers use awk rather than a short-circuiting reader on purpose: a
reader that exits at the first match leaves the upstream taking EPIPE, and
under `set -o pipefail` the satisfied case is reported as the failing one.
Deliberately NOT taken from #1138: -D_FORTIFY_SOURCE=2 (a downgrade on the
release platform), -fPIE/-pie (inert under STATIC=1), -fstack-protector-strong
(real but unmeasured cost, and #1138 injected it into SQLite, tree-sitter and
mimalloc -- the indexing hot loop), and the $(shell) probe machinery (~10
compiler spawns per Makefile parse, making the binary a function of the
builder's toolchain).
Co-authored-by: PR9000 <119280965+PR9000@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogelandpetercoxphoto
|
e5f3aab02e |
feat(pipeline): index-time importance scoring (weighted degree)
Score every Function/Method/Class node during indexing and persist the
result as a numeric "importance" key inside the node's existing
properties_json:
importance = sqrt(num_refs) * priv * generic * distinct * test_penalty
num_refs is the incoming CALLS + USAGE degree. The multipliers demote
private (leading-underscore) and generic names (defined in >= 5 distinct
files), promote distinctive identifiers (snake_case or camelCase, len >=
8), and demote test scaffolding -- symbols in a test file per the graph's
own cbm_is_test_path() classifier, or targets of an incoming TESTS edge.
Weighted degree only; a transitive (PageRank) refinement is deliberately
not built here until it can be shown to beat this on a fixed judgment
set.
No schema change and no CBM_INDEX_FORMAT_VERSION bump: the score lives in
the existing properties_json TEXT column, so no index is forced to
rebuild and indexes written by older builds simply lack the key.
Three failure modes this shape invites, all of them silent, are addressed
directly:
* Cost. The generic-name multiplier needs |{files a name is defined in}|.
Computing that per NODE with a pairwise file-path scan is O(k^3) in a
same-name group of size k -- ruinous on real corpora, where a single
Java name group reaches k > 4000. Here the count is computed once per
distinct NAME and memoized, and file paths are deduplicated through a
hash set, so total distinct-file work is linear in the graph. Measured
cold full index of a 666 MB Java corpus, same -O2 binary: 85 s with the
pass disabled, 80 s with this implementation, 481 s with the per-node
shape.
* Registration. PREDUMP_PASS_COUNT is now derived from the pass table
with sizeof instead of being hand-written. A hand-written count that
lags an appended entry silently skips the last-registered pass with
every test still green.
* Incremental. The incremental post-pass sequence runs the pass too, over
a graph rehydrated from the store whose nodes already carry the key.
The write-back overwrites an existing key in place instead of appending
a second one, which would otherwise corrupt properties_json into
{"importance":1.0,...,"importance":2.0}.
tests/test_importance.c covers the formula and binds all three failure
modes; the complexity suite gains a GATED counter asserting the pass's
same-name-group work stays linear (~2x, not ~4x) under a corpus doubling.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: petercoxphoto <20230727+petercoxphoto@users.noreply.github.com>
|
||
|
Martin Vogel
|
feb720809f |
Merge pull request #1104 from LA-10/fix/769-file-qn-keeps-extension
feat(store): index-format rebuild boundary for the File-QN change |
||
|
LA-10
|
4d9da40b36 |
test(769): end-to-end sibling regression + index-format rebuild boundary Refs #769
Signed-off-by: LA-10 <leenshuhail1@gmail.com> |
||
|
Thomas Dyar
|
ee126155ab |
feat(pipeline): add Ensemble production topology routing pass
Parses XData ProductionDefinition blocks to extract production topology
and emits Route/ASYNC_CALLS/HANDLES graph edges for Ensemble
interoperability routing.
Key design points:
- Emit Route nodes (qn: __route__ensemble__<prod>.<item>) with props
{broker, class, enabled, production} — visible in get_architecture
and search_graph at the same priority as HTTP routes
- Emit ASYNC_CALLS edges from source method/item to target Route node
via SendRequestSync literal scan and TargetConfigName(s) settings
- Emit HANDLES edges from each item's entry-point method to its Route
node, completing the code-to-topology link
- Fix scan_source_for_send_targets: scope search to calling method body
using brace-depth tracking to eliminate cross-method contamination
- Handle TargetConfigNames (plural, comma-separated) by emitting one
ASYNC_CALLS edge per token
- Segment-anchored class_name_matches() prevents prefix collisions
("Ens" not matching "Ens.BusinessService")
- Language-gate: skip entirely when no ObjectScript nodes present
Tests: pipeline_ensemble_routing_edges (Route nodes + ASYNC_CALLS +
HANDLES); pipeline_ensemble_routing_method_scoping (two methods, two
targets, no cross-contamination).
Fixes #467
Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
|
||
|
Martin Vogelandalexisperinger-ux
|
43ecc0988c |
feat(dbt): extract dbt model lineage from Jinja-templated SQL
A dbt model is an ordinary .sql file whose dependencies are written
{{ ref('other_model') }} or {{ source('group','table') }}, never as
literal table names. The SQL grammar cannot read those -- FROM {{ ref(
'x') }} is a parse error to it -- so the dependency structure of an
entire dbt project was invisible to the graph.
extract_dbt.c runs as a sub-extractor inside the normal indexing
pipeline. Per qualifying file it emits one Model definition (named by
the file stem, dbt's own model identity) plus one usage per ref()/
source() call; pass_usages resolves those into model -> relation
lineage edges like any other reference.
Model is a relation label alongside Table and View, which is what makes
the rest work without new plumbing: registry seeding, the central
relation veto, the incremental surface hash, search ranking and the
architecture queries all pick it up from cbm_label_is_relation. Sharing
one label class also means a model's source('raw','customers') resolves
onto a Table declared in a plain DDL migration in the same repository,
so dbt lineage and SQL lineage form one graph rather than two, while
the veto keeps model names out of every non-lineage consumer.
The pass is self-gating: SQL files only, and only those carrying a real
ref()/source() call. The dbt builtins are themselves the evidence, which
is cheaper than a dbt_project.yml lookup and more precise -- templated
SQL that is not dbt (an Airflow {{ ds }} parameter) produces no Model
node and no usages even inside a dbt repository.
{% macro %} definitions are deliberately excluded: the vendored
tree-sitter-jinja2 grammar has no node types for {% %} statements, so
they can only be recovered by a hand-written scanner that cannot see
comments or find {% endmacro %} for a correct span. Filed as follow-up
rather than approximated.
Tests cover the lineage, the last-string-argument semantics of both
builtins, the gate (against a plain-SQL control extraction), plain DDL
staying untouched, and an end-to-end pipeline case asserting model ->
model across files, model -> Table onto plain DDL, and cross-language
isolation. Disabling the pass reddens three of them; removing Model from
the relation set breaks lineage outright.
Implements the lineage half of #575.
Co-authored-by: alexisperinger-ux <alexis.peringer@iss-stoxx.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Jumaga2015
|
c4224cf3ef |
test: cover stacked C# attributes for #1692 (repro_issue1692)
A C# method carrying three stacked attributes ([Foo][Bar("x")][Baz]) must
end up with three DECORATES edges, one per bracket group. The fixture is
indexed through the real production pipeline (rh_index) and asserts
rh_count_edges(..., "DECORATES") == 3, so it exercises the MCP path rather
than the extractor in isolation.
Verified to actually reproduce: with find_jvm_modifiers() reverted to its
first-child lookup the suite goes red (decorates=1, expected 3), and green
again with the fix in place.
C#-only by design. Parsing the equivalent fixtures with the vendored
grammars shows `[A] [B] [C]` yields three sibling `attribute_list` nodes in
C#, while PHP's `#[A] #[B] #[C]` yields ONE `attribute_list` holding three
`attribute_group` children — the pre-fix lookup already found all of them,
so PHP could not hit this bug and a stacked-PHP fixture would pass either
way. Java/Kotlin/Swift use the `modifiers` wrapper, a single node that never
repeats. PHP's DECORATES path stays covered by mkc_c3_php8_attribute
(tests/test_matrix_known_classes.c).
Registered in repro_main.c and Makefile.cbm's TEST_REPRO_SRCS.
Signed-off-by: Jumaga2015 <jumaga2015@gmail.com>
|
||
|
Martin Vogel
|
0273250c33 |
Merge pull request #1310 from Yyunozor/fix/1264-hook-diff-aware-clang-tidy
fix(hooks): scope pre-commit clang-tidy to staged changes (#1264) |
||
|
Martin Vogel
|
51770de0b6 |
perf(ts): memoize expression-type evaluation per node
ts_eval_expr_type and ts_signature_for_call are mutually recursive: resolving a call evaluates its argument expressions once per lookup path (method dispatch + namespace fallback), and in tsc-compiled spread files the first argument is itself the next nested Object.assign(...) call — the same subtree re-evaluates once per enclosing level, 2^n total. The TS suite's objectSpreadRepeatedComplexity.js (3.6 KB, 5 nodes) measured 20.4 s; with the memo its eval cost is zero within measurement noise of a one-file control, and the microsoft/TypeScript corpus drops 37.5 -> ~24 s warm (nodes byte-identical, edges within the known scheduler jitter). Expression types are position-pure within a file pass (one node = one scope path; the per-file walk is single-threaded and deterministic), so one eval per node is the correct semantics, not a cache trade-off. The memo is a per-file, arena-backed, linear-probe table keyed on TSNode.id. Results produced under a depth-cap or budget bail are never stored: both bail sites bump a degradation counter, and a store only happens when the subtree completed clean — a degraded UNKNOWN can therefore never shadow a later full evaluation. The regression guard asserts work, not wall-clock: the nested-Object.assign shape must complete without exhausting the deterministic eval budget, read back through a new CBM_ENABLE_TEST_SEAMS accessor pair. The seam lives in the lsp_all unity object, so GRAMMAR_CFLAGS_TEST/TSAN now carry the seams define (test artifacts always have seams; prod never does). Verified RED without the memo (budget exhausted, suite 47.8 s) and green with it (3.2 s). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
e700b6215f |
test(complexity): deterministic O(n^2) guard on work-counter ratios
Finding #1669 took an 11-corpus A/B across two release binaries. This suite makes that bug class fail a unit test in seconds, on every platform, from tiny corpora. Method: build k and 2k REPLICATED module copies, run the full in-process pipeline on both, assert counter RATIOS. Independent copies mean every extensive quantity — nodes, edges, Σ per-file registry defs — must grow linearly (ratio ~2). A files x corpus coupling makes per-file work itself grow with k and lands at ratio ~4. Ratios expose the exponent regardless of absolute scale, so 60-120 files suffice. Verdicts are pure functions of (code, input): gates ride ONLY on deterministic work counters and data-product counts, never on wall time. Throughput (nodes/s, edges/s) is information-only, written to private/benchmarks/complexity-<ts>.json (local, gitignored; skipped under CBM_SKIP_PERF where rates are meaningless). Two corpus shapes, both needed: - Independent modules (java/py/go/ts templates): catches cross-module contamination and dedup breakage. The #1669 bug is GREEN here — fully closed modules filter perfectly, which is exactly why it survived. - The growing shared package (bigpkg): one Java package whose file count scales with k — the real-repo shape (files concentrate in large packages). The JVM namespace filter branch makes per-file work track package size, so this corpus is the honest #1669 reproducer: ratio 4.00 RED on the pre-fix tree, 4.00 RED for a module-scoped overlay, 2.00 GREEN for the own-file overlay. It discriminated the correct fix design before the fix was written. Both legs of every pair exceed MIN_FILES_FOR_PARALLEL(50): below it the sequential path runs, which builds no shared registries and would be the wrong code path to gate (its per-file cost is bounded by the 50-file ceiling). Recorded but deliberately NOT gated, with reasons at the case: tail_candidates and fallback_rows are legitimately superlinear under replication until those scans are bounded, and measured ~1 ns/unit. Every ratio gate carries a non-vacuousness floor on the base counter so broken counter wiring fails loudly instead of green-washing (cbm_pxc_count_perfile_defs feeds the overlay path into the same counter the fallback path already used; wired for Java, extend to the TS overlay when touching ts_lsp). Dynamic coverage: languages iterate CBM_LANG_COUNT; embedded templates cover the LSP-hybrid languages, and tests/fixtures/complexity/<lang>/ dirs are auto-discovered so a new language joins the guard by dropping fixtures. Uncovered languages are listed in the report with the reason. The local report additionally carries per-language node/edge counts with ratios and a per-pass elapsed_ms table per run (captured via a TEE log sink during the in-process pipeline runs) — trend data for humans, still never a gate. Refs #1669. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Mauricio Offermann
|
c059396eb2 |
fix(build): give sanitized-build detection one spelling and a backstop
Four places ask "is this binary instrumented?" and each asked it differently.
One of them, the C# LSP bench, only recognised ASan, so TSan and MSan measured
an instrumented parse against the NATIVE 200ms budget. The other three carried
a hand-copied list of `__SANITIZE_*__` macros that nobody kept in sync.
That drift is what the TSan gap was made of. CFLAGS_TSAN never passed
SANITIZED_DEFINE, and the per-site conditions could not cover for it: they test
`__SANITIZE_THREAD__`, which is GCC's spelling. Clang — the compiler that leg
uses — announces thread instrumentation through `__has_feature(thread_sanitizer)`
only, and no site consulted it. The claim in
|
||
|
Martin Vogel
|
0a163d4f75 |
build(tsan): define CBM_SANITIZED_BUILD on the ThreadSanitizer leg
The previous commit widened the spawn-retry budget for sanitized builds, and it did nothing on TSan — the leg it was written for. `subprocess_run_spawn_failure` failed again on the very PR that was meant to fix it. CBM_SANITIZED_BUILD comes from SANITIZED_DEFINE, which keys off $(SANITIZE). TSan does not use that variable — it has its own TSAN_SANITIZE — and CFLAGS_TSAN never included SANITIZED_DEFINE. So the macro was undefined on that leg and every sanitized-budget branch compiled to its NATIVE value while running an instrumented, several-times-slower binary. The comment above SANITIZED_DEFINE already describes this exact failure for trap-UBSan: "the build system is the single source of truth for is this binary instrumented; compiler-specific probes miss clang's feature-check spelling and every non-ASan sanitizer". That lesson was recorded and the TSan leg was never wired up to it. Nor would compiler probes have saved this: clang spells thread instrumentation __has_feature(thread_sanitizer), not __SANITIZE_THREAD__. CFLAGS_TSAN now defines it unconditionally, which is honest — that flag set exists solely to build an instrumented binary. Checked the neighbours: MSan (scripts/msan.sh passes SANITIZE=) and the diag lane (passes SANITIZE= too) both already get the define. TSan was the only gap. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
93696aeb55 |
docs(security): publish the AV false-positive evidence; harden arm64 W^X
SECURITY.md gains an "Antivirus False Positives" section placed directly after the existing Verification block so it reuses those commands rather than restating them. It states the detection plainly, shows the measured evidence (the verdict inverts across architectures and link modes; identical macOS segment structure splits clean/flagged; entropy is 3.5-4.2 bits/byte, nowhere near packed), lists the cross-project precedent, and — deliberately — records that removing the embedded scripts and externalizing the assets did NOT move the detection. A negative result is still evidence, and publishing it is the point. It also documents the release policy, invites independent audit through the provenance/cosign/checksum path we already ship, and adds an `av-analysis` issue label route for anyone who finds something real. Tone is evidence-first: no vendor blame, and an explicit note that signing helps Windows but that no AV-honoured signing scheme exists for Linux ELF, so it is not a full answer. README carries a short pointer near Quick Start, where someone who just hit a Defender warning will actually look. Unrelated to AV, found while dissecting the artifacts: add -Wl,-z,separate-code. GNU ld enables it by default on x86-64 but not on aarch64, so arm64 emitted ONE R E PT_LOAD covering the whole image and mapped ~259 MB of tree-sitter parse tables executable at runtime. Section flags said A, not AX -- but the kernel applies segment permissions, so the section flags were never the control. The amd64 build of the same source mapped the same data R only. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
d58afe562d |
revert(release): re-embed runtime assets into the single shipped binary
Externalizing the integration templates (#1492/#1493) and the UI bundle
(#1501/#1503) was done to reduce the Microsoft `Wacatac.B!ml` surface. It did
not work: across dry runs the flagged artifact count stayed at ~3 and the
detections merely moved between artifacts.
Dissection of run 31286803592 shows there is no structural cause to fix. The
verdicts split across every axis at once — linux-amd64 (dynamic) flagged while
linux-amd64-portable (static) is clean, but linux-arm64 (dynamic) clean while
linux-arm64-portable (static) is flagged. The two macOS binaries have identical
segment structure and split clean/flagged. Siblings from one build landed in
different variant buckets (.B vs .C). Entropy is low everywhere
(code_vectors.bin 4.166, grammar tables 3.464 bits/byte, against 7.5-8.0 for
packed payloads), so the packed-payload hypothesis is excluded too.
So the complexity bought nothing, and installation goes back to being
self-contained: one binary that carries its own UI and agent integration
templates, with no adjacent data file that has to resolve before `install`
works. Only the UI-capable composition ships from now on, under the historical
unsuffixed archive name.
Removed: src/ui/asset_pack.{c,h}, asset_pack_stub.c, asset_manifest_stub.c,
scripts/pack-ui-assets.mjs, src/cli/integration_assets.{c,h},
assets/cbm-integrations.json, scripts/gen-integrations-hash.sh, the
--verify-runtime-assets probe (nothing adjacent left to verify), and the
composition gates A6/A7 whose property is now deliberately inverted.
Restored: scripts/embed-frontend.sh, src/ui/embedded_{assets.h,stub.c}, the
compiled-in hook/adapter template bodies, and the embed/EMBED_OBJS build path.
Kept from the reverted commits, re-applied by hand where a wholesale file
restore would have dropped them:
- cbm_module_path_utf8() in both self-path sites. GetModuleFileNameA renders
through the ANSI code page and mangles non-ASCII install paths.
- the /__cbm/ui-readiness HMAC proof, secure_random and cbm_hmac_sha256, so
`daemon start --open` still waits for a genuine CBM listener.
- X-Content-Type-Options: nosniff on served assets.
- the MinGW noexecstack gate, -lbcrypt, and the cppcheck/zip CI fixes.
Archives are now codebase-memory-mcp-<os>-<arch>[-portable] with exactly four
members (binary, LICENSE, installer, THIRD_PARTY_NOTICES.md). That restores the
names every static package manifest already points at — aur, chocolatey,
homebrew, scoop, winget and glama were all broken by the -ui- rename.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
54b5ba846b |
fix(lint): generate integration hash before cppcheck
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
00ed56fb77 |
fix(build): restore Windows release compile
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
8018561cfe |
fix(release): externalize runtime assets and harden VT verification
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
497c680aae |
fix(build): make the UI target depend on the generated integrations hash header
The -ui build failed on Windows with "cbm_integrations_hash.h file not found". integration_assets.c includes that generated header, and the subagent added it to PROJECT_HDRS so every single-invocation binary target picks it up — but cbm-with-ui is a phony target with its own direct recipe that does NOT list PROJECT_HDRS, so nothing generated the header before its one-shot compile. Only the UI variant hit this; the standard binary, test-runner, test-repro-runner and tsan targets all carry PROJECT_HDRS and built fine. Added $(INTEGRATIONS_HASH_HDR) as an explicit prerequisite of cbm-with-ui. Confirmed by dry-run: `make cbm-with-ui` now schedules the generator before the compile, and the header regenerates from clean. Audited every target that compiles integration_assets.c (via PROD_SRCS) — all now resolve the header, so this closes the build gap rather than moving it. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
83df500364 |
fix(build,ci): make the integrations-hash generator runnable and quiet a JSON-license false positive
Two CI failures on the integration-asset change, both build/gate plumbing, not the architecture: 1. scripts/gen-integrations-hash.sh was committed 100644 while Makefile.cbm invoked it directly, so a fresh CI checkout died with "Permission denied" building the hash header — which failed EVERY test leg at step 0, since the contract step builds first. It passed locally only because the generated header was already cached, so make never re-ran the generator. Fixed both ways: the recipe now runs it via `sh` (mode-independent, cannot regress from a checkout mode), and the file is committed 100755 to match its siblings. Note: test_script_exec_bit_contract.sh scans shell call sites, not Makefile recipes, so it did not catch this — the `sh` prefix is the durable guard. 2. The license gate (ScanCode) flagged scripts/package-release.sh with the SPDX 'JSON' license. The archive member lists place 'cbm-integrations.json' and 'LICENSE' adjacently, and ScanCode reads the '.json LICENSE' token adjacency as a JSON-license reference. The order is not free to change — the Windows single-binary contract locks the exact member sequence — so this is a genuine false positive on a first-party MIT build script. Added it to the policy's ignored_paths with the same justification the six existing first-party entries carry (the gate scripts, provenance auditor, and discover.c all name licenses for legitimate reasons). No allow-listed SPDX id was added; the JSON license is NOT now permitted anywhere else. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
a6f132bc85 |
harden: ship integration templates as a verified JSON asset, not embedded bytes
The product binary embedded nine complete shebang'd shell scripts, their
PowerShell/.cmd twins, and two node:child_process client-adapter modules as C
string literals, and wrote them into agent config dirs at 0755 on install. A
block of script-shaped bytes inside a large unsigned executable is exactly the
surface Microsoft Defender's ML scored Trojan:Script/Wacatac.B!ml — the
detection that has recurred across the last several release attempts, on the
Linux ELF and darwin-arm64 standard binaries while every UI and Windows variant
stayed clean.
This moves every integration template out of the binary into a single shipped
data file, assets/cbm-integrations.json (compact: one asset, because each
shipped file is itself scanned). The binary now carries only the templates'
identity — one embedded SHA-256, generated at build time from the JSON so it
can never drift — plus the code to load, verify, and render them.
Why a hash and not just a loose file: content compiled into the binary was
tamper-evident with the binary. Moving it to disk must not turn it into
unauthenticated code the installer blindly executes. install verifies the asset
against the embedded SHA-256 and fails closed on mismatch ("integration assets
missing or modified - reinstall from the release archive") before writing
anything. Verified end to end: a one-byte edit to the asset makes install
refuse.
Ownership, for safe upgrade/removal: the verified copy is published to
~/.cbm/assets/<version>/ — a sibling of the DB cache, never inside it, so
clearing the cache cannot strip a user's hooks. A deployed integration file is
ours to remove only if it matches a template rendered from that stored copy;
foreign files and user-edited files at the reserved paths are preserved. The
historical template bodies travel in the JSON's released[] arrays so files
written by older versions are still recognised and cleaned up on uninstall.
Removed from src/: cmm_gate/session/subagent script-prefix and suffix
constants, cbm_build_released_gate_script, the shell/PowerShell/.cmd bodies in
cli.c, and the JS/TS generators in client_adapter.c. Confirmed on the built
standard binary: 0 occurrences of '#!/usr/bin/env bash', '#!/bin/sh',
'node:child_process', and 'ExecutionPolicy' (was non-zero for each).
test_no_embedded_scripts_contract.sh makes the removal a build property, not a
reachability assumption: no production C source may contain a shebang or
node:child_process literal, and the built binary is scanned directly. It strips
comments first (a contract a comment can satisfy is a false guard), and is
revert-checked — reintroducing one shebang literal makes it fail naming the
file:line. check-binary-composition.sh (A6) enforces the same needles on every
release artifact post-strip.
This does not, by itself, prove the detection is cured: the clean UI binary is a
strict superset of the flagged standard one, so no single passage is a
deterministic trigger. It removes the most coherent malware-shaped composite
from every shipped binary, aligns with how gh/starship/zoxide externalise
integration material, and makes the templates maintainable as data rather than
escaped C literals. Causal confirmation still needs a VirusTotal ablation.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
452372eec5 |
fix(mem): route ordinary malloc through mimalloc on Linux (#1360)
The comment above MIMALLOC_OVERRIDE_DEFINE claimed Unix "relies on static-link-order override". That was never true. mimalloc emits strong malloc/free definitions only when MI_MALLOC_OVERRIDE is set — the define its source gates alloc-override.c on — and the build set it for MinGW only. MI_OVERRIDE, which the build does set everywhere, is this project's own prod/test marker that mimalloc never reads. With the override body compiled out there were no strong symbols for link order to prefer, so ordinary malloc went to libc. Measured, not inferred. The shipped v0.9.1-rc.1 artifacts report 0/6 allocator-owned size classes on linux-arm64 glibc AND musl-static, and the same A/B on ubuntu-arm64 here: main warn mem.allocator.not_owned owned_classes=0/6 with this fix info mem.allocator.owned classes=all Every purge/reclaim option cbm_mem_init sets was therefore inert on Linux, applying only to the bound sqlite/tree-sitter populations — the same class of defect as #581, where committed memory ratcheted for months because nothing asserted the wiring on a real artifact. macOS stays off deliberately and permanently: enabling the override there compiles alloc-override.c's forwarding definitions, and under the two-level namespace this binary's free becomes mi_free while system libraries keep allocating from the system allocator, so the first pointer crossing that boundary aborts with "mi_free: invalid pointer". ELF's flat namespace has no such split, which is why Linux can have this and macOS cannot. Three parts: * Makefile.cbm switches MI_MALLOC_OVERRIDE and a new CBM_MEM_GLOBAL_OVERRIDE together in one place, for MinGW and Linux. The latter tells our own sources what to expect and goes into CFLAGS_PROD only, because MIMALLOC_CFLAGS_TEST builds mimalloc with -DMI_OVERRIDE=0 and no override define: a test binary has no global override BY CONSTRUCTION and must never be told to expect one. Deriving that expectation from the platform instead would make every Linux test run warn about a correctly configured build. * mem.c splits the startup audit by what the build actually asked for. Where no override was requested, ordinary malloc reaching libc is the design, so it reports the measured ownership at INFO and names what IS bound. A warning that fires on every run of a correct build is not a tripwire, it is noise — it trains readers to ignore the one line that catches #581. The genuine warning still fires wherever an override was requested and did not take. * smoke-test.sh Phase 1b pins the SHIPPED artifact's wiring per platform and fails in BOTH directions: Windows/Linux must own all classes, macOS must not. This cannot live in a unit test — a from-source test build never has the override — which is exactly why the defect survived so long. Verified on ubuntu-arm64: prod owns all classes; the test build reports bound-populations-only at INFO (0/6) and the mem suite stays green; smoke Phase 1b passes, and fails with the expected message when the same tree is built with the override forced off. On macOS: prod reports bound_populations_only at INFO, smoke Phase 1b passes, mem suite 51/51. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
03e935576b |
feat(foundation): classify indexing roots by breadth and sensitivity
Adds cbm_workspace_classify_root, the breadth half of the workspace
boundary. It answers "is this path obviously too broad or too sensitive to
index as one root", not "is this caller allowed to index it" — that second
question needs the user-level grant store, since it is the only input a
caller cannot write. The header says so, because a reader who mistakes one
for the other would draw the wrong conclusion about what is enforced.
Three rules, each doing what the others cannot:
- Depth below the volume. Two components on POSIX refuses every top-level
tree in one rule with no list to maintain. Windows and UNC count
drive/share-relative and require one, because there the first component is
already user space ("D:/repos") — the system trees that sit at the same
depth ("C:/Windows") are covered by name instead, which is tractable
because that set is small and stable.
- The home directory itself, which is two deep on macOS and Linux and so is
invisible to depth.
- Credential directory names, matched against every component so both
"…/.ssh" and "…/.ssh/keys" are refused. This list is deliberately additive
so extending it needs no design authority.
A path that is or contains the cache directory is refused outright: indexing
such a tree would pull every other project's graph database into this
project's index.
Classification order is load-bearing and commented as such. The home
directory normally contains the cache directory, so testing the cache first
reported every home as "holds the cache" and made it non-overridable, which
contradicts the intent that a person may override it. Depth precedes the
cache test for the same reason: "/Users" satisfies both and "too broad" is
the reason that helps the reader.
Takes home and cache as parameters rather than reading the environment, so
the policy is a pure function and the tests need no filesystem. Twelve table
tests; three of them failed on the first implementation and drove the
ordering fix above.
Not yet wired to any caller — deliberately, so the policy can be reviewed on
its own.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
66e9aec28d |
Merge pull request #1392 from DeusData/feat/generated-client-adapters
feat(install): generate pi and OpenCode extensions from the tool registry |
||
|
Martin Vogel
|
b33a4de9bc |
fix(build): keep alignment checking off vendored tre.o; make SANITIZE rebuild
Two defects, one found by the other. The crash: every fixture-indexing suite (mcp, incremental, index_resilience) died on the Windows/ARM64 leg with STATUS_ILLEGAL_INSTRUCTION and no diagnostic. Cause: an earlier commit in this branch re-armed UBSan's alignment check on the vendored TRE regex engine, and CLANGARM64 builds with -fsanitize-trap, where a trap IS an illegal instruction with nothing printed. macOS and Linux stay clean under the same check because Windows is LLP64 -- 32-bit long -- so TRE's struct layouts and access widths differ there and only there. TRE is vendored third-party code we do not modify, so suppressing the check for that one object is the honest scope; every other object keeps alignment armed. Verified by bisect: origin/main, |
||
|
Martin Vogel
|
bbd13335f5 |
feat(pipeline): delta-merge executor for the closure route
The closure route stops loading and rewriting the world. Its executor is now a dedicated subsystem (pipeline_delta.c + orchestration): CLONE the live generation (copy-on-write where the filesystem offers it), repair the closure against the clone, PATCH exactly the repaired node/edge set in one transaction, and publish through the same sealed-staging finalize leg as the dump path. No full graph load, no full dump, and the general indexing pipeline is untouched -- the profiled kernel run put those two at 187s of a 238s one-file repair whose actual resolution work was 0.6s. Id discipline carries the design. Node ids are AUTOINCREMENT and never reused; the small in-RAM graph is pre-seeded with PROXY nodes carrying their real database ids (SELECT ... ORDER BY id with the id watermark pinned before each insert), and fresh nodes are numbered above the previous generation's MAX(id) -- so "id > max_db_id" is the complete, marker-free definition of what the patch inserts, and every edge endpoint id is database-valid by construction. The inbound-edge snapshot and its QN-keyed re-link become indexed SQL; a re-link whose endpoint no longer exists matches no row, which is full-reindex semantics for deleted symbols. Fail-closed throughout: an unexpected reference to an unseeded label surfaces as a UNIQUE-constraint violation that fails the patch transaction, and EVERY delta failure discards the stage and returns FORCE_FULL_REINDEX -- the live database is never touched, so a full rebuild always self-heals whatever the delta could not do. FTS policy: nodes_fts is contentless, so purged rows cannot be deleted individually on existing databases; their rowids can never alias a live node again (AUTOINCREMENT) and dead entries drop out of the rowid join. The patch inserts rows for exactly the new nodes through the same cbm_camel_split tokenizer the wholesale rebuild uses. The legacy gbuf-based tail reverts to serving only the test-only force_legacy_partial route; the closure orchestration owns its own coverage merge, publication race gate, surface-row merge and committed counts, and publishes with fts_wholesale=false. Gate: the full convergence suite runs against this executor unchanged -- body-edit graph equality with a fresh full index, removed-definition dropping the dependent's stale edge, tsconfig-alias retargeting, the decline matrix, and 510 pipeline/incremental/store/integration tests. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
94189e6990 |
feat(pipeline): serialize and publish per-file LSP surfaces
Second piece of closure repair. At the collect_all_defs seam -- the only moment the per-file result cache is alive -- both drivers (parallel and sequential) now serialize each file's CBMLSPDef slice to canonical JSON, hash it, and hand the rows to the pipeline; cbm_pipeline_publish_generation writes them into the staging store next to the manifest, so surface data and graph always belong to the same generation. Canonical bytes are the point: every field is written in fixed order with an explicit null for absent strings (NULL and "" differ in the CBMLSPDef contract -- receiver_type NULL means "not a method"), so byte equality IS surface equality and the sha over the bytes is the early-cutoff key. Registry-only labels that pxc_map_label drops but the name registry serves (Field) are folded into the hash as a separate "reg" array, or renaming one would slip past the cutoff. Behaviour pinned in SUITE(pipeline): a fresh full index persists a versioned surface row per file; a BODY edit republishes the identical surface_sha; a SIGNATURE edit changes it. That pair of properties is what the routing layer will stand on. cbm_pxc_collect_all_defs gains an optional per-file prefix array -- the flat all_defs[] otherwise loses the file boundaries the serializer needs. CORRECTION to 6c22338's scope note: it claimed cbm_pipeline_publish_ generation was reachable only behind CBM_INCREMENTAL_TEST_API. Wrong -- dump_and_persist_hashes calls it on every production full index (pipeline.c:1863); the grep that "verified" test-only reachability had excluded pipeline.c itself. The predictable staging name WAS in the production publish path, which makes that fix a real production hardening, not a test-path cleanup. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
6a0adb4430 |
fix(msan): scope the zstd stdint workaround to the zstd object
The previous fix put -include stdint.h on the lane's global SANITIZE line. That traded one vendored compile break for another: force-including a libc header ahead of every source file freezes glibc's feature-test macros before sqlite3.c can set _GNU_SOURCE for itself, and its view of libc loses MREMAP_MAYMOVE and nanosleep (17 errors on the x86-64 CI leg). The workaround only ever had one legitimate target -- the zstd amalgamation whose MEMORY_SANITIZER block lost its stdint re-include -- so it now rides a per-object hook (ZSTD_EXTRA_CFLAGS) that the MSan lane sets and every other build leaves empty. sqlite3.c compiles exactly as before in every lane. Verified locally that zstd compiles with the hook and the default rule stays untouched; the MSan leg itself is x86-64-only, so CI is its verification venue. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
7d390c5120 |
fix(ci): unbreak the MSan and diag lanes on x86-64
MSan: vendored zstd fails to compile. Its MSan-only block (guarded by
MEMORY_SANITIZER) declares __msan_test_shadow returning intptr_t and
reaches for the type with
#define ZSTD_DEPS_NEED_STDINT
#include "zstd_deps.h"
but the amalgamator that produced zstd.c collapsed that second include
into a "skipping file" comment, so the define pulls nothing in and
intptr_t is undeclared. Only this lane compiles that block at all, and
only where <stddef.h> does not drag stdint.h in transitively -- which is
why it built on the local aarch64 container and failed on CI's x86-64.
The lane now forces the header. Patching the vendored amalgamation would
be silently undone by the next re-vendor.
diag: detect_invalid_pointer_pairs comes back out. It fires during static
initialisation inside vendored simplecpp -- a std::string global at
simplecpp.cpp:101 -- with a second "pointer" of 0xfffffffffffffff3, a
sentinel rather than an address: libstdc++ string internals, not
anything this codebase wrote. It is a process-wide runtime flag with no
per-file scoping, so unlike the analyzer's path filter it cannot be
aimed away from vendored code. Keeping it would mean a permanently red
lane reporting a non-defect, which is how a lane gets ignored. The
instrumentation it needed comes out with it.
The other three off-by-default checks stay: stack-use-after-return,
stack-use-after-scope, strict-string-checks. Those are the ones covering
bug classes nothing else in the matrix looks for, and none of them
fired.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
0c223d9027 |
build(lint): pin analyzer suppressions to the code they argue about
The memory gate stays gating, and it now has a way to record a genuine false positive that cannot quietly outlive its own reasoning. A whitelist entry names one (file, function, check) and carries two things: why the analyzer is wrong, argued from the code, and what was tried before concluding that. The entry is pinned to the sha256 of that function's text. Edit the function and the entry stops counting -- the finding comes back and has to be argued again against the code as it now is. This is the part that matters. A suppression that survives the code it was written about reads as "reviewed" while being nothing of the kind, which is worse than no suppression at all. The gate fails on: a finding with no entry, a finding whose entry has gone stale, and an entry that asserts rather than argues (there is a floor on how much reasoning an entry must actually contain -- the mechanical half of "argued, not asserted"; whether the argument is correct stays a review question). An entry that matches no finding is reported but does not fail, because analyzer versions differ across platforms. NOLINT is still not honoured here and the gate does not read it. The whitelist ships empty: the analyzer is currently clean across LINT_SRCS, so nothing is being suppressed today. The mechanism exists for the first finding that genuinely warrants it. Verified by exercising each path rather than only the clean one: an unaccounted finding fails and names its enclosing function; an argued entry with a matching hash passes; the same entry fails as STALE once the hash no longer matches the function; and an entry that says only "false positive" fails for not arguing its case. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
0fe0f453be |
build(test): close the gaps in the sanitizer matrix
Every lane here was either asserting coverage it did not have, or was reachable only by bypassing the entry point that is supposed to define the ladder. TSan: no exclusions any more. The three suites the Makefile documented as excluded are back. daemon_ipc and daemon_frontend no longer reproduce the harness race and thread leaks they were excluded for. daemon_runtime did not deadlock as the comment claimed -- it reported a real production data race on the log sink, fixed separately. Excluding a suite from a sanitizer lane hides exactly the class of bug the lane exists to find, so the comment block now records what was actually true rather than what was assumed. TSAN_OPTIONS gains report_thread_leaks=0. This disables the thread- HYGIENE check only; race detection is untouched. Several daemon fixtures fork after the process has gone multi-threaded, and in the forked child TSan sees the parent's already-finished threads as never-joined even where the fixture joins them. It fires on macOS and not Linux, i.e. it tracks fork semantics rather than anything about this code. The alternative was dropping whole suites, which costs real race coverage; this costs none. UBSan: tre.o no longer builds with -fno-sanitize=alignment. Alignment was switched off for a vendored regex engine that ships in the product, which is where the check is least redundant, not most. LSan on macOS: new test-lsan target and test-lsan-macos CI leg. LSan is on by default under ASan on Linux, so the Linux legs have always had leak coverage. On macOS it is off by default and Apple's clang refuses to enable it outright, so that platform had none at all. Apple's refusal is not a darwin limitation -- upstream LLVM supports LSan on darwin/ arm64. The lane is the ordinary ASan suite built with Homebrew LLVM and run with detect_leaks=1; it runs the full suite clean and was checked to still catch a deliberately leaked allocation. MSan: reachable from the local ladder. The image and compose service existed but run.sh had no leg, so the only way in was to drive docker compose by hand -- which means it was not part of the ladder in any meaningful sense. The image also moves to clang 22, matching the diag and analyzer lanes instead of sitting four majors behind on noble's default. The leg documents the aarch64 shadow-mapping failure so a local arm64 stack overflow in the grammar suites is not mistaken for a code defect; the GitHub leg runs x86-64, which is the mapping that matters. Off-by-default ASan checks: the diagnostic lane, and its CI twin, now run detect_stack_use_after_return, detect_stack_use_after_scope, detect_invalid_pointer_pairs (with the -fsanitize=pointer-compare, pointer-subtract instrumentation it requires) and strict_string_checks. Running ASan is not the same as running all of it, and these four cover bug classes nothing in the matrix was looking for. They stay on the diagnostic lane rather than the gating ones until they have a clean history there; promoting them is a separate deliberate step. Verified: macOS TSan 940 passed / 3 skipped / 0 races over the full suite set; the macOS leak lane 7375 passed / 4 skipped / 0 leaks, with LeakSanitizer confirmed armed under that exact toolchain and option set by checking it still reports a deliberately leaked allocation. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
c9886d4f25 |
feat(ci): MemorySanitizer lane — instrumented libc++/zlib image, full C++ coverage
Stage 2 of the memory-diagnostics program (user decision: go directly to the instrumented image rather than a C-only probe). MSan detects uninitialized READS, the one memory-error class no other lane covers dynamically, and it requires every linked library to be instrumented -- vendored C deps compile in-tree and instrument for free; the two external links do not: - test-infrastructure/Dockerfile.msan: pinned-base image building libc++/libc++abi/libunwind (llvmorg-18.1.8, LLVM_USE_SANITIZER= MemoryWithOrigins) and static zlib v1.3.1 into /opt/msan, with the symbolizer and MSan runtime in a separate last layer so tool additions never invalidate the ~30-min libc++ build. - scripts/msan.sh: the canonical lane entry. ALWAYS clean-builds its BUILD_DIR: make does not encode flags into dependencies, and a stage-1 probe's libstdc++ objects surviving into the libc++ lane produced a convincing-looking uninitialized-value report at preprocessor.cpp:168 -- the uninstrumented .so string constructor wrote the temporary, the instrumented move constructor read it. The clean rebuild proved it an artifact: extraction (incl. the C++ preprocessing path) runs 272/272 with zero reports. - Makefile.cbm: CXX_STDLIB / CXX_STDLIB_FLAGS hooks so the lane can swap libstdc++ for the instrumented libc++ (defaults identical; the shipping build is byte-for-byte unaffected). - docker-compose test-msan service: same aarch64 seccomp/setarch remedy as the TSan service (MSan's shadow layout hits the same personality() block). - CI test-msan job (_test.yml): buildx local-cache via the repo's existing pinned actions/cache -- no new third-party action pins; a warm run skips the libc++ build entirely. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
80afcd6bbe |
fix: close the memory-error paths the clang-analyzer lane surfaced
The memory-diagnostics report's priority-4 lane (path-sensitive clang-analyzer, memory checks only) run over all 111 production files. 21 findings triaged; the real ones, all cold-path (none can explain #581's per-query residual): LEAKS - mcp get_architecture: scope_path leaked on the missing-store early return (REQUIRE_STORE frees only `project`); allocate after the gate. - pass_definitions: cancellation mid-extraction leaked the pass-owned result cache including already-extracted entries; mirror the end-of-pass cleanup. - store package-boundary scan: the row-scan abort path freed the node arrays but not the boundary accumulators or their duplicated package strings. - cbm quarantine set: a duplicate path line leaked the replaced value (and a fresh key copy -- the table borrows key pointers); a partial strdup failure leaked the surviving half. Reuse the stored key for duplicates. - pass_githistory: unchecked malloc/strdup -- an OOM dereferenced NULL and a failed strdup leaked the index cell. Allocate before claiming the slot. NULL/UB - cli config subcommand: NULL argv with nonzero argc slipped the guard (the inner `argv &&` shielded only the help comparison) into argv[0]. - store bfs_multi: a negative max_results broke out before any row was written, then freed fields of an unwritten negative-index slot. Clamp. - pass_calls emit_http_async_edge: the service-pattern call sites pass a NULL target behind a hand-duplicated URL predicate; a drift between the copies turned target->id into a null deref. The callee is now total. - sqlite_writer: both leaf-array OOM paths left leaf_count stale with a NULL array, walking pb_finalize_* into leaves[0]; consistent empty state routes them to the existing root=0 failure return. HARDENED (invariants true but invisible to path-sensitive analysis) - Leiden CSR + aggregate arrays, SCC adjacency: calloc + endpoint guards, so a future degree/collection miscount degrades benignly instead of UB. - SCC cycle fill: the ncyc==0 no-slot invariant made local. RECORDED FALSE POSITIVES (no code change) - yaml sequence starts (loop bound == alloc bound), cypher agg arrays (same count both sides), mcp read_message ch (assigned by fgetc each iteration), pkgmap clean buffer, mcp csize (Tarjan: ncomp>=1 when nverts>=1), vendored verstable x2. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> GATE + LANES (user decision: runner cost accepted) - make lint-mem (local triage) and lint-mem-ci (gating: vendored-filtered, any remaining finding fails). The gate is green because every false positive above was restructured for provability -- calloc'd fill-cursor arrays, explicit Tarjan invariant, zeroed buffer tails, min-1-element allocations -- never suppressed. - make diag: pinned newest-LLVM ASan/UBSan lane with straighter stacks. - CI: lint-mem job (_lint.yml) and test-diag job (_test.yml), both on the pinned LLVM 22 apt toolchain. Cost disclosure: roughly +25-40 min and +25-60 min (ccache-warm) per push respectively. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
f056aa938d |
fix(build): track internal/cbm/lsp headers in whole-program targets
The branch adds internal/cbm/lsp/type_registry.h, and PROJECT_HDRS covered $(CBM_DIR)/*.h but not the lsp/ subdirectory -- so editing an lsp header rebuilt nothing and an incremental build silently tested stale header values. The branch's own invariant (repro_make_tracks_headers.sh) caught it on the repro-unix CI jobs; the same trap class as the lsp_all.o one fixed by #662. bash tests/repro/repro_make_tracks_headers.sh: green. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
029416fda6 |
Merge remote-tracking branch 'origin/main' into qa/call-reference-rebase
# Conflicts: # Makefile.cbm # internal/cbm/extract_defs.c # src/foundation/compat_fs.c # src/foundation/compat_fs.h # src/pipeline/pipeline.c # src/pipeline/pipeline_incremental.c # tests/test_main.c # tests/test_pipeline.c # tests/test_store_checkpoint.c |
||
|
Martin Vogel
|
2a353b2a33 |
feat(cli): generate client extension modules from the tool registry
pi has no MCP client, and OpenCode has no declarative hook configuration -- verified against OpenCode's own plugin documentation, which states hooks are available only through JavaScript/TypeScript plugin modules. For those two clients a module is the only extension point. We do not want to ship such a module as a repository asset. #534 proposed one embedding 380 lines of TypeScript as C string literals and registering 7 of the 15 registry tools; every tool added afterwards would have been silently missing for that client. Hand-maintained copies of the tool list have already produced defects here (#1361, and the smoke-invariants count). So the module is GENERATED from the live registry instead. cbm_client_adapter_pi walks cbm_mcp_tool_count()/cbm_mcp_tool_name() and registers every tool, which makes drift structurally impossible rather than merely discouraged: adding a tool to TOOLS[] adds it to every generated adapter with no second edit. Nothing in the repository is a .ts file. Both emitters wrap their output in ownership markers so a caller can rewrite its own block and leave a user-authored file alone. Two defects from the prior proposals are closed by construction: - Path escaping. #616's template rejected the double quote but not the backslash, so a Windows home like C:\Users\urs\bin produced an invalid unicode escape and the whole auto-loaded plugin failed to parse, which is worse than an absent plugin. cbm_client_adapter_escape_js escapes backslash, quote, newline and CR, and fails closed rather than emitting a truncated literal; generation aborts if the path cannot be escaped. - The silent no-op. #616's payload omitted hook_event_name, which hook-augment requires and without which it accepts nothing, so the plugin emitted zero bytes for six weeks. The generated payload carries it and a test pins it. Recorded honestly: the OpenCode emitter hooks tool.execute.after, whose ability to modify a tool's output is NOT part of OpenCode's documented plugin contract (only tool.execute.before's argument mutation is). If they change it the augmentation stops with no error. That risk is accepted deliberately and the emitter says so in a comment, so a future reader does not have to rediscover it. This commit adds the generator and its tests only; wiring into the install and uninstall routines follows separately so the two are reviewable apart. Tests: every registry tool appears in the pi module (revert-checked by simulating the 7-of-15 subset, which fails); Windows/quote/newline escaping and its truncation boundary; the OpenCode payload carries hook_event_name and registers no tools; NULL/empty binary paths generate nothing. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |