mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-10-26 06:43:15 +08:00
d8e116aeee53b081b5eae33693eea917bd002e0c
37
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Martin Vogel
|
d050fcae35 |
Merge main into fix/c-typedefs-macros-enumerators for the release batch
Conflicts: internal/cbm/extract_defs.c includes (union of both sides); src/pipeline/registry.c keeps main's lang-aware index_under_name call, and the fence comment combines both facts (Rust cfg twins are variants of one QN; a C macro's #macro QN never reaches the registry). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
42a9320200 |
graph: one identity per conditional definition, SPAWNS edges, no cross-language guesses
The test-impact engine selects tests by walking the graph, so it can only be as right as the graph. Three gaps made it wrong: One identity for a definition's mutually exclusive variants - A definition written once per #if/#else branch, per #[cfg], per platform file (foo_linux.c / foo_win.c, platform directories) or per Go build tag is ONE node. The graph buffer keeps every variant's span in a `variants` node property (definition labels only), so a change in any variant maps to the definition. - Extraction now sees definitions inside conditional blocks where it skipped them: C++/C#/ObjC class members under #if (one cursor walk, conditional block types per language in lang_specs), Elixir defs under if/unless, Fortran .F90/.F95/.F03/.F08, Rust #[cfg] twins (no more `name#cfg(...)` QNs that lost every call), C platform files and directories (non-static file-scope functions). - Haskell: the vendored scanner reads only an #if group's first branch. The grammar stays upstream; every further branch is parsed as its own same-length view (cpp_branch_views.c) and adds what the raw parse missed. haskell/bytestring: no new partial parses, +27 functions, IMPORTS 322 -> 329, 131 definitions with variant spans. - Incremental closure repair re-parses every file of such a definition when one of them changes (variant partners through DEFINES), so a repair equals a full index. No cross-language name guesses - A name-only resolution (unique_name, suffix_match, qualified_suffix, field_type_hint, and same_module across files that only share a path stem) never binds a symbol another language defines. - Bash imports are the files a script sources: every top-level command was read as an import of its name (`set -e` imported "set", bound to a TSX method in 137 scripts here). - This repository: cross-language CALLS 5,916 -> 5,091; the remainder are C/C++ pairs, builtins and same-stem module collisions (recorded). SPAWNS edges (per language, like the HTTP client patterns) - A call that starts a program (subprocess.run, exec.Command, child_process.spawn, posix_spawn, system, Process.Start, Command::new, ... about 30 languages, through the file's own imports) gets a SPAWNS edge to a Process node named after the program (literal first word, or <dynamic>) instead of a CALLS guess. No target resolution. Both resolvers ask at the same point (parity test). The engine follows the graph - Seeds map changed lines through every variant span; a test written once per #if branch is one test; when a change reaches a non-test `main`, the tests that spawn processes are reached too (the process runs main, and main reaches everything). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
44efa6281b |
fix(extract): C typedefs become nodes; macros and enumerators get their own QNs
In C-family code several different entities competed for one node, and
some had no node at all:
- A typedef name was never a definition: `type_definition` has no `name`
field, so the class path dropped it. With an anonymous
`typedef struct/enum {...} X;` the fields and enumerators went with it.
- A function behind an unknown leading macro (`API RetT name(...)`) was
named after its return type, became a Method `RetT.name`, or was
dropped, depending on the error-recovery shape.
- Every bodyless `struct X` / `enum X` (a parameter, a field type, a
forward declaration) minted a Class/Enum node. In one file, or in a
same-stem .h/.c pair, such a reference displaced the real definition.
- A `#define NAME` shared the QN of the function, type or enumerator
called NAME, and the later line won: a namespace-rename macro replaced
the typedef it renames, an `#else` stub macro replaced the function.
- Enumerators were named `<scope>.<Enum>.<CONST>`, although C and
unscoped C++ enums put them in the enclosing scope.
- A definition repeated in several `#if` branches kept one span; the
others left no trace.
- Braces split across `#if` branches lost whole functions, and a C
keyword could surface as a function name (`if`).
Extraction:
- typedef names are definitions: `Type`, or `Class`/`Enum` with members
for an anonymous `typedef struct/enum {...} X;`; an alias is dropped
when the same file defines the tag of that name
- macro-prefixed functions get their real name in all three recovery
shapes, consistently for defs, call scopes, parameters and the C LSP
- a bodyless tag is a reference and creates no node; a definition head
left as loose ERROR tokens (`struct NAME {`) is recovered
- a C keyword is never a function name
- a first-branch projection (one branch per `#if` group, positions
unchanged) is parsed when the raw parse lost structure wholesale, and
only functions missing from the raw result are adopted from it
- macro-wrapped enumerator lists yield one constant per slot
Identity (semantic index version 4; an existing index is rebuilt once):
- a C-preprocessor macro's QN is `<module>.<NAME>#macro` (C, C++, CUDA,
GLSL, Objective-C, ISPC); name and label are unchanged. Resolvers
prefer a definition and use the macro only when no definition of that
name is visible (LSP target lookup, trace_path, get_code_snippet)
- enumerators of C enums and of unscoped C++/Objective-C enums are
`<enclosing scope>.<CONST>`; `enum class` stays nested; `parent_class`
names the enum
- a node whose file defines its QN more than once under one label
carries `variants`, the line spans of all those definitions (`#if`
twins, per-platform macro redefinitions, C++ overloads)
Full-text search indexes a macro's QN without the fence, so macro rows
score as before.
Measured on redis and curl against the parent commit: nodes 38,625 ->
40,484 and 28,180 -> 29,750; Type nodes 0 -> 573 and 0 -> 193; phantom
Class nodes from references 202 and 821 removed; functions lost to a
same-named macro 45 and 41 restored; curl functions with no node 11 ->
0; nodes of non-C-family files unchanged by QN. Of 174 audited Doxygen
references in xxhash, 174 now reach the right entity (64 before). Index
time is unchanged within noise (+2% pipeline time for 5% more nodes).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
juan Carlos Estrada MontoyaandClaude Opus 5
|
460ebefc87 |
fix(graph-buffer): log why an atomic publish failed instead of a successful dump
cbm_gbuf_dump_to_sqlite() emitted gbuf.dump regardless of the writer's result, so a run that published nothing still logged node and edge counts as if it had worked, and the failure reached the user only as the generic "Pipeline failed. Check repo_path exists and contains source files." The writer's temp -> final rename in publish_writer_output() returned a bare ERR_WRITE_FAILED. #1628 taught cbm_rename_replace() to translate the platform error into errno so callers could report why, and wired up the stage -> final rename in pipeline.c. This is the other rename on that path, and it is the one that runs first. Preserve errno across the cleanup in every publish error branch. That includes cbm_writer_open(), where the cleanup unlinks a file that was never created: without the save it leaves ENOENT behind, so a directory that denied the create is reported as a missing path — the #1620 case, stated wrongly. Read the value in graph_buffer.c before the profiling macro can clobber it, and report gbuf.dump_failed with the return code and that errno. A sticky append failure clears errno rather than attach a reason it does not have, and the field is omitted when there is none. No logging is added inside internal/cbm/, which has none today. Tests cover both halves: the dump reports a failure instead of a summary, the writer names its truncation reason, and the publish rename's errno survives a cleanup that itself fails. Refs #1620, #2001. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012FxkVF773gDGNmBmsLnzN7 Signed-off-by: juan Carlos Estrada Montoya <junk151516@users.noreply.github.com> |
||
|
Martin Vogel
|
c3714f1133 |
fix(mem): size the budget from free memory, and relieve on real scarcity
Indexing the linux kernel with NO ceiling set took the machine down: on a
48 GB host the default budget is 50% of TOTAL ram (24 GB), the run used
essentially all of it (24.5 GB peak RSS), and with a 12 GiB VM already
running there was nothing left. The budget never fired -- it read 22 GB
charged against 24 GB -- while the host was out of memory.
Total RAM is the CEILING; what is free right now is the CONSTRAINT.
- cbm_mem_clamp_to_available() reduces a fraction-derived budget to fit
what is actually free, reserving a quarter of it (held between 1 and
8 GB) for the OS and everything else, never below a 512 MB floor. An
explicit CBM_MEM_BUDGET_MB still wins -- it is the user's deliberate
choice -- but is warned about when it exceeds free memory, because
the failure it buys reads like a product bug rather than a setting.
- cbm_mem_should_relieve() = our budget exceeded OR the machine short.
Deliberately separate from cbm_mem_over_budget(): system pressure
must RELIEVE (spill, reclaim), never abort, since the spike may not
even be ours. The extraction loop now enters spill on real scarcity,
and the dump phase -- where the kernel run actually died, with our
charge comfortably under budget -- keys its free-heavy decision on it
too. cbm_mem_system_under_pressure() existed and had no callers.
Same scenario after: budget clamped to 18,947 MB, peak RSS 19.5 GB,
status indexed in 185.7 s, machine fine.
Spill also gets limits and cleanup, because relief that fills the disk or
leaves gigabytes behind is not relief:
- a free-space preflight (never start spilling below 10 GB free) and a
ceiling at half of what is free; past it results stay in memory and
back-pressure takes over. cbm_fs_free_bytes() is new (statvfs
f_bavail / GetDiskFreeSpaceExW quota).
- POSIX unlinks each spill file while holding it open, so the space
returns on ANY exit including SIGKILL -- a killed run cannot leave
spill behind because there is no name left to leave it under.
Windows keeps close-path removal plus a sweep where an open file
refuses deletion, which is exactly the liveness test.
- the staging-database sweep now reclaims orphans of EVERY project in
the cache directory, not just the one being indexed. A kernel index
killed by the OOM killer used to park ~15 GB until someone re-indexed
the kernel, and indefinitely if nobody did. The per-stage lock probe
still keeps a live writer's stage, whichever project owns it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
a2f29b3fe7 |
fix(mem): the 15 GB budget is met by its own metric, end to end
The 15 GB kernel run completed with every phase mark under budget, but
the budget metric itself -- the charged footprint the extract gate reads
on every file pull -- reached 16.8 GB. Two defects, found with a probe at
the gate (one line per 256 MB of charge after the spill latch, with the
class table; mem.charge.probe, CBM_MEM_PHASES=1, stays in).
Worker graph buffers kept a dense id array over the global id space. The
charge climbed gradually from the 14.06 GB latch to 14.96 GB, then jumped
about 1 GB between two consecutive gate reads, and the only class that
moved in that step was gbuf_index. Worker buffers draw ids from the shared
counter, so each of the 18 kept an id -> node array spanning the whole
global id space -- 8.5M ids x 8 B, next power of two -- almost entirely
empty, and when the global id crossed 2^23 all eighteen doubled in the
same instant. Nothing asks a worker buffer by id before the merge (every
cbm_gbuf_find_by_id caller runs on the main buffer); the worker
constructor now drops the array the way it already drops the five
secondary indexes, and the main buffer answers by id after the merge.
Spill mode did not park what was already cached. The sweeps ran only on an
over-budget observation; a run that latched early (near_budget) and then
stayed under budget through extraction -- which is exactly what the first
fix produced: 14,949 MB at the extract mark -- carried the 44,797 results
cached before the latch (8 GB) into registry build and resolve, which
cannot park, and resolve ran at a charged 21.8 GB against the 15 GB
budget with nothing to stop it. The earlier runs only fitted because the
id-array burst tripped the sweeps by accident. Extraction now ends with a
final sweep whenever spill mode is on: results belong on disk before the
phases that cannot park inherit them.
Kernel, CBM_MEM_BUDGET_MB=15000, before -> after: charged high-water
16,771-16,818 MB -> 14,582 and 14,732 MB in two runs (97-98% of the
budget); the final sweep parks ~45k results and takes the charge from
14.6 to 4.3 GB before resolve, which peaks at 11.5 GB committed; wall
234 -> 224-230 s; nodes identical (8,529,729), edges inside the
run-to-run band. RSS high-water 18.7 GB is
pages already purged and not yet reclaimed: the footprint never exceeds
7.3 GB at a mark.
peak_charged_mb (
|
||
|
Martin Vogel
|
7b77fd4834 |
feat(mem): memory on the core, waste cut, results spilled under budget
Second increment on the memory core. The first landed the core, the linter
and phase attribution with zero production call sites; this one migrates the
memory that matters, proves where it goes, removes the waste the proof
exposed, and makes the memory budget a promise: over budget, extraction
results go to disk and the run completes at the floor instead of aborting.
Every number below is from a real index on this machine (M5 Pro, 48 GB) with
CBM_MEM_PHASES=1 and the release build.
WHAT THE PROOF SAID (Go corpus: 21,875 files, 345 MB of source)
The graph buffer, blamed for the peak since the 2026-07 research, is 3% of
the memory: 0.55 GB peak. The per-file extraction results are the memory:
14.8 GB written into their arenas, of which 3.4 GB is reachable. The rest
is temporaries -- cbm_node_text copies at 496 call sites, per-node QN
strings, abandoned generations of every growable array -- that an arena
cannot free and that the result kept alive until resolve ended, because
the result owned the arena. The arenas were charged 28.8 GB of capacity
for those 14.8 GB. On the kernel the same shape was 8.9 GB used by 89,731
results, and resolve grew it to 13.3 GB used / 17.2 GB capacity because
the C walk allocated its working set in the result arena and stored arena
pointers into the shared registry. After that, 7 GB of the kernel's
post-resolve memory had no class at all.
WHAT CHANGED
Attribution. The graph buffer (87 raw sites -> 0), arena blocks, the
tree-sitter slab allocator, the bound tree-sitter/SQLite allocators, the
semantic path (pass_semantic_edges, pass_similarity, minhash: 80 sites), the
hash table (Verstable, a class per table via CTX_TY: cbm_ht_create_in) and
CBM_DYN_ARRAY allocate through the core. Classes arena, ts_tree, hash_table
and dyn_array join the table. Two cross-boundary frees of buffer-owned
strings (pass_importance, pass_complexity) go through a new setter,
cbm_gbuf_node_set_properties_json. The linter baseline drops 4086 -> 3916.
Result compaction (internal/cbm/result_compact.c). At the end of a file's
extraction, everything reachable from its result is measured, copied into
one exact-size arena (strings interned by content within the file) and the
working arena is handed back. Later appends restart growth at the default
block (cbm_arena_init_exact, arena grow_size). The stale duplicate
internal/cbm/arena.h, which shared the foundation header's include guard
and won or lost by include order, is a shim now.
Record slimming. CBMCall 320 -> 72 bytes: the eight inline argument slots
(256 bytes, mostly empty) are allocated on first capture. CBMUsage 48 -> 40.
The per-file overlay contract (src/pipeline/pass_lsp_cross.c). One
lifecycle for every language's cross-file resolve: a scratch arena, a
per-file overlay registry chained to the sealed shared base, the walk
against the overlay, resolved calls appended into the result, scratch
destroyed. type_registry gains chain-aware iterators (every yielded index
belongs to it->reg) and copy-on-write refinement
(cbm_registry_func/type_for_update), and the C, C#, Go and Python walks
use them. This is what an ASan heap-use-after-free in the
lsp_resolution_probe suite demanded: the C walk stored per-call arena
objects into the shared registry, and a kernel run with a naive scratch
arena lost 1,045 edges to the resulting garbage reads. It also ends
unsynchronized writes into the shared registry from parallel resolve
workers.
The 7 GB with no class, found and cleaned. gbuf_index tables and keys were
2.5 GB live and 7.1 GB at peak: the 18 worker buffers built the five
secondary indexes they never query (cbm_gbuf_new_worker builds none), and
18M per-key index arrays started at 8 slots (cbm_da_push_min starts them
at 2); edge property JSON is 81% "{}" (interned). The semantic pass
allocated a 512-token stride per function up front (packed tokens with
offsets: cbm_sem_corpus_add_docs_batch). Edge dedup keys are 128-bit
hashes instead of sprintf'd strings (edge_key_map_t). Each worker keeps
one reusable working arena between files (cbm_work_arena_take/give:
rewind, not free) and every phase mark returns freed pages to the OS.
Budget metric. The over-budget check compares the charged footprint --
max(phys_footprint, mimalloc commit) on macOS -- not resident_size: after
extraction the kernel worker sat at 17.4 GB RSS with 5.5 GB charged, because
pages mimalloc has purged stay resident until the OS wants them, and
phys_footprint under-reports after MADV_FREE_REUSABLE cycles.
Spill / admission control (internal/cbm/result_spill.c, the extract gate
in pass_parallel.c). At 15/16 of the budget (or over it, or with
CBM_MEM_SPILL=1) the gate latches spill mode -- early, because the gate
sees the crossing per file pull and the workers' in-flight files carry the
charge past the line before the first sweep lands: every compacted result
is parked on disk as header + block in one append-only file per writer
under <cache>/spill/, the results already cached are swept out by every
worker, and registry build, def collection, surface rows, resolve and the
infra-route passes load a result only for the moment they read it
(relocated by base delta). The abort path stays closed while a sweep runs
or a cached result remains; it fires only at the floor -- graph +
registries + in-flight files. A result that owns sub-results stays in
memory; a retained parse tree is dropped with the parked result and
re-parsed on load. Every consumer of the result cache reads a slot through
cbm_pipeline_result_acquire()/release() (pipeline_internal.h): a parked
slot is NULL in the array, and the passes that indexed it directly lost
every __route__infra__ node and its INFRA_MAPS edges before the contract.
The other things the transient loads found: a def label that was the
result's own pointer (SIGBUS in the Go cross registry builder), two JVM
helpers that passed their input through, a dedup table whose keys borrowed
sr->value (the insert spun on freed memory), and a latch that was visible
before the store was open (a peer found nothing to park and failed the run
while its neighbour wrote 9 GB). The arena rewind reuse branch had
defeated the nblocks = CBM_ARENA_MAX_BLOCKS OOM seam the suites use; it
reuses only blocks that exist. The incremental and probe routes still
index the cache array themselves and therefore never spill
(ctx->spill_allowed stays false there; follow-up).
The semantic pass under a budget (src/pipeline/pass_semantic_edges.c). The
pass held every function's tokens, the corpus's per-document token ids and
the vectors being stored at once: 4.9 GB on the kernel, on top of the
floor, and nothing sized it to the headroom. Now, when the charged
footprint plus that transient would cross the budget, phases 2-4 run per
batch of functions: tokenize -> count into the corpus -> free; finalize
once; tokenize again -> vectorize -> store -> free, with pages returned to
the OS after every batch. Tokenization is deterministic, so both passes
see the same tokens and the graph does not change; the run pays with a
second tokenize. CBM_SEM_BATCH=<n> forces the batch size (the
batched-equals-unbatched test). The corpus frees its per-document token
ids at the end of finalize (the co-occurrence pass is their only reader),
and mem.semantic.step lines (CBM_MEM_PHASES=1) put the class's live/peak
bytes and the charged footprint at every sub-phase.
The semantic corpus and tokenizer allocate through the core (54 raw sites
-> 0). This is not attribution only: cbm_sem_tokenize handed out libc
strdup blocks that pass_semantic_edges freed with cbm_free -- mi_free in
the production build -- since the pass moved onto the core. No libc-backed
test build can see a cross-allocator free, so the core now refuses a block
it does not own (mi_is_in_heap_region) when CBM_MEM_PHASES=1 is set,
fatally: every proof run in this PR ran with it armed, and the Go and
kernel indexes report zero foreign blocks.
Accounting without contention. The core charged every allocation and free
to shared per-class atomics; with 18 workers on 18 cores every block bounced
the same three cache lines, and the bench against the shipped v0.10.8 showed
it as CPU time rising faster than wall time for the same graph: Kotlin 38 ->
121 s CPU (+58% wall), TypeScript 210 -> 596 s (+67%), Go 177 -> 312 s. The
deltas are thread-local now and reach the shared counters once per 256 KB or
512 blocks of change per thread, when a parallel-for worker ends, and
whenever a reader looks (its own thread first). Phase marks read after the
join and are exact; a class peak is a diagnostic and can be low by threads x
256 KB. Result: Kotlin 10.9 s / 43 s CPU (the residual +22% wall against
v0.10.8 tracks its +34% nodes), TypeScript 33.9 s (v0.10.8: 33.4), Go 48.4 s
(v0.10.8: 52.6), Java 81.1 s (v0.10.8: 79.8); graphs unchanged.
SQLite on a heap of its own. The same bench put the kernel's write step at
132 s where v0.10.8 needed 9.7 s, all of it in one coverage publish
statement. A stack sample named the cost: mimalloc's free-page search and
its periodic heap collect, walking the graph's pages -- millions of them
once the graph lives on the core -- for every statement-journal chunk
SQLite allocates. SQLite now allocates from a mimalloc heap of its own per
thread (mi_heap_new; mi_free is heap-agnostic, so nothing else changes),
and the periodic collect is set to the maximum mimalloc allows since the
explicit release at phase marks and spill sweeps is where memory goes
back. Coverage publish 132 -> 9.4 s; profiled kernel 376 -> 248 s
(v0.10.8: 296 s).
Formatting without a process lock. On macOS every vsnprintf takes the
process locale under an unfair lock (localeconv_l inside __vfprintf);
eighteen workers formatting type names on the C# corpus collapsed into
it -- the 10 MB JIT test files took 68 s each instead of under a second
-- and cbm_arena_sprintf paid it twice per string (size pass, write
pass). Each thread now formats with a C locale object of its own
(vsnprintf_l), and the common short string is formatted once into a
stack buffer.
Per-file budgets that cover the whole file. The 5 s per-file budget
covered the parse only; the LSP walks after it had none. C# JIT stress
files (a 23 MB single expression among them) parsed inside the budget --
the old wall-clock parse timeout used to drop them, which hid the rest --
and then held a worker for 346 s each in the usage walk (tree-sitter's
ts_node_parent descends from the root: quadratic on a deep tree), and the
cross-file resolve ran on the same trees. Three rules now, one site for
every language (cbm_extract_file_ex, honoured by cbm_pxc_dispatch_file): a
parse that used more than half the budget disqualifies the file from the
LSP walks (lsp_skipped); the unified walk checks its thread CPU time every
1024 nodes against six budgets and stops there, keeping what it found
(walk_truncated, implies lsp_skipped); a file whose parse plus walk spent
the budget is skipped by the LSP walks as well. Each decision is logged
with the path (extract.lsp.skipped, extract.walk.truncated). C# extraction
355 -> 39 s; the whole index 463 -> 128 s wall and 27 -> 11 GB peak
against v0.10.8 with 299 more nodes. The walk cuts exactly the five JIT
stress files (hugeexpr1, hugeSimpleExpr1, HugeArray1, HugeField1/2); the
15 MB System.Runtime.Intrinsics reference file walks to the end and, with
four generic-nesting JIT regression tests, only skips the LSP walk under
the parse-plus-walk rule.
The crash behind the C# bench. dotnet/runtime is 42,555 files, and its
resolve phase died with SIGBUS on every run of this branch. A libc-backed
ASan build of the server named it: heap-use-after-free in c_adl_resolve,
reading a type name that another worker had allocated in its per-file
scratch arena and freed at the end of its file. The C++ class walk
refines a method's return type when the declaration is more specific
than the pre-registered one (NAMED -> pointer/reference/template), and
it did that by casting the chained lookup result to non-const and
writing a scratch-arena signature into it -- into the sealed shared base
when the entry lived there. The overlay contract had exactly one
bypass, and it was this cast; every lookup returns const, so a grep for
the cast is the audit. The refinement now goes through
cbm_registry_func_for_update: copy-on-write into the overlay, the base
untouched. Test: clsp_method_return_refinement_is_copy_on_write (a
sealed base with a NAMED method return, an in-class pointer declaration;
the base signature pointer is unchanged and the overlay copy carries the
refinement plus a marker only a copy has) -- RED on the cast, GREEN on
the accessor. The fixed ASan build then indexed the whole
corpus clean: 42,555 files, 1,224,981 nodes, 5,794,320 edges, worker
exit 0, no report.
Production build backs the core with mimalloc explicitly (never the global
override, which macOS's two-level namespace forbids); the test build keeps
libc so the sanitizers see every block -- the same split the tree-sitter and
SQLite bindings already use.
PROOF
Go peak RSS 16.7 -> 4.0-5.2 GB; result arenas 14,804/28,817 -> 856/856
MB; wall 55 -> 52-60 s; graph inside the pre-existing run-to-run
jitter (gRPC Route nodes: two identical runs differ by 21).
Go CBM_MEM_SPILL=1: peak RSS 1.7-2.3 GB, 21,875 results parked (950 MB
on disk), graph identical: 39 infra routes, 158 INFRA_MAPS, 21,875
surface rows, nodes/edges inside the Route jitter.
Go CBM_MEM_BUDGET_MB=1200: spill latched at 1,204 MB, the first sweep
took the charge to 768 MB, the run completed at 1.99 GB peak RSS.
Kernel DEFAULT budget (24,576 MB): aborted before, completes now without
spilling. Extraction 35.2 -> 16.5 GB RSS (5.6 GB footprint, 15.3 GB
committed); results 8.9 GB = capacity; resolve tracked 24.6 -> 19.1
GB; peak RSS 27.3 -> 20.1 GB. Nodes identical (8,529,729); edges
inside the +-400 run-to-run band the kernel shows between identical
runs.
Kernel CBM_MEM_BUDGET_MB=16000: aborted before, completes now through the
spill path. Latched at 17.2 GB charged; 89,731 results parked (9.3
GB on disk, 370,188 loads); extraction end 6.2 GB RSS / 5.1 GB
committed; resolve 8.6 GB RSS / 11.5 GB committed; peak RSS 16.6
GB (the semantic pass plateau, 13.3 GB RSS, is graph-derived and
outside what spill can move). Nodes identical; edges 15,773,598.
Kernel CBM_MEM_BUDGET_MB=15000, before the semantic batching: completes,
every phase mark under 15,000 MB by every metric, but the process
RSS high-water mark reached 16.9 GB inside the semantic pass (class
peak 4.9 GB against 0.8 GB live).
Kernel CBM_MEM_BUDGET_MB=15000, with it: completes; the charged footprint
stays under 15,000 MB through the whole semantic pass (14,018 ->
14,381 MB across its sub-phases, 6 batches of 125,644 functions);
every phase mark is under budget by every metric (max footprint
12.6 GB, max commit 11.5 GB); the process RSS high-water mark is
15.65 GB and is set in extraction, at the moment the gate latches
(15,002 MB charged) and before the first sweep lands -- a 4%
overshoot of in-flight work, no longer the semantic pass. Nodes
identical, edges 15,772,913, zero foreign blocks.
Kernel CBM_MEM_BUDGET_MB=15000, with the anticipatory latch: spill mode
enters at 14,062 MB (near_budget); the charged footprint never
exceeds 11.8 GB at any mark; the process RSS high-water mark is
15.1 GB in extraction and 15.3 GB during the final write-out --
1-2% over, the in-flight window and the dump transient. Nodes
identical, edges 15,772,543, zero foreign blocks.
Go CBM_MEM_BUDGET_MB=1200 with it: latched at 1,131 MB, extraction
peak 998 MB RSS, graph identical to the baseline.
BENCH AGAINST THE SHIPPED v0.10.8 (same machine, same driver, 2026-09-14)
corpus wall s peak RSS GB nodes / edges / CALLS
perl 5.7 -> 5.3 0.09 -> 0.05 -3 / -130 / -3
php 6.3 -> 5.8 0.66 -> 0.20 +21.9% / +1.4% / +23.8%
rust 6.8 -> 6.3 1.15 -> 0.67 +1 / +56 / -369
c 5.9 -> 5.7 0.16 -> 0.19 identical
kotlin 8.9 -> 9.4 1.55 -> 0.96 +33.5% / -22.8% / +12.1%
django 10.9 -> 10.2 3.13 -> 1.12 +6 / -14.5% / -28.4%
go 52.6 -> 42.9 17.2 -> 3.9 +14.2% / -10.3% / +2.2%
java 79.8 -> 73.8 27.6 -> 11.0 -121 / -1.9% / -0.6%
csharp 463.3 -> 128.1 27.0 -> 11.2 +299 / -0.9% / -0.6%
typescript 33.4 -> 33.6 12.0 -> 2.5 -186 / -3.5% / +130
kernel 294.8 -> 233.3 27.5 -> 25.1 -171 / -1.3% / -0.1%
No corpus is slower beyond noise; CPU time follows wall (kernel 2,442 ->
1,403 s). Every graph delta against v0.10.8 is main between the release and
the branch base (
|
||
|
Joshua RichterandClaude Opus 5
|
62b44519c6 |
fix(confidence): stop an unreadable confidence from reading as 0.0
strtod answers 0.0 for text it cannot read, and 0.0 is a real confidence in both places that call it. So a property blob carrying a malformed value, such as "confidence":null, came back as a recorded confidence of zero. Two things went wrong with that. In src/graph_buffer/graph_buffer.c, edge_props_confidence answers CBM_EDGE_CONF_ABSENT (-1) when an edge carries no confidence, so any real confidence outranks it. A malformed value answered 0.0, which beats -1 in the merge comparison, so the malformed blob displaced a clean stored one. The function's own comment already promised that "absent/unparseable reads as -1". Only the absent half was true. In src/mcp/mcp.c, bfs_edge_evidence_for_hop sets the confidence to -1 and the emitter publishes any value of 0 or more as a recorded number. A malformed value printed as 0.00, which reads as "the resolver was certain this is wrong" rather than "nobody wrote a number here". Both sites now pass an end pointer to strtod and keep the absent sentinel when the pointer never moved, which is the shape src/store/store.c:402 already uses. Two tests come with the change, and both were seen failing before the fix and passing after: gbuf_edge_props_unreadable_confidence_does_not_displace_absent tool_trace_path_unreadable_confidence_reports_not_recorded Red: 258 passed, 2 failed. Green: 260 passed, 0 failed. The full suite reports 7633 passed, 2 failed. Both failures are in tests/test_cli.c (lines 1749 and 6725) and reproduce on a clean tree without this change. They depend on the coding agents installed on the machine, not on this change. make -f Makefile.cbm lint-ci passes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Joshua Richter <jrichter5781@gmail.com> |
||
|
Andrew Studnicky
|
d08fccd912 |
fix(graph_buffer): merge duplicate-edge properties commutatively
confidence, strategy and via are not part of the edge dedup key
(source:target:type), and the same logical edge is legitimately minted by more
than one strategy carrying different values — pass_calls emits CALLS edges from
both LSP resolution and registry-textual matching. cbm_gbuf_insert_edge
replaced the stored blob unconditionally, and per-worker edge buffers merge in
worker-slot order, so which strategy's attributes survived was a function of
thread scheduling: the edge set stayed stable while its attributes flickered
run to run.
Replace the overwrite with a total order over the two candidates, which makes
the result commutative and associative and therefore independent of arrival
order:
1. higher "confidence" wins, which also enforces the intended precedence
that an LSP-resolved call outranks a textual match;
2. on equal confidence, the lexicographically greater blob wins, giving a
deterministic choice between otherwise equal candidates.
An empty or absent incoming blob still never displaces stored properties.
Adds three tests: order independence for the same pair of blobs inserted both
ways, the higher-confidence winner regardless of arrival, and the existing
empty-blob guard. The first two fail before this change.
Signed-off-by: Andrew Studnicky <a.j.studnicky@gmail.com>
|
||
|
Martin Vogel
|
fea266fc53 |
fix(gbuf): canonical collision winner honors the upsert refresh contract
The deterministic same-QN collision rule picked the lexicographically smallest (label, file_path, start_line, name), which silently inverted the documented upsert/merge semantics: re-upserting a QN with updated fields kept the OLD content (gbuf_upsert_updates, gbuf_upsert_same_qn_updates_all_fields, gbuf_merge_overlapping_qns all red on every platform). Replace it with a single mixed-direction total order applied in both cbm_gbuf_upsert_node and merge_update_existing: smallest file_path, then LARGEST start_line, then largest name/label. Still a pure content function of the two candidates (commutative, associative, scheduling-independent — multi-threaded runs stay byte-identical, verified by the xfs MT x4 fingerprint guard), and line-descending within a file restores the classic contract: a later definition in the same file replaces the earlier one, and a full tie is the same entity refreshed in place. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
4ba44a9f07 |
style: clang-format pass over recently touched pipeline and graph-buffer files
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
1d7912d2df |
perf(gbuf): release indexes before dump transients; dense by_id; drop dead field
Three memory quick-wins from the gbuf compaction research (graph output verified identical; all delete/incremental paths suite-covered): 1. Release ALL lookup indexes right after build_dump_nodes instead of after edge building: nothing later in the dump reads them, and at kernel scale they are ~3.8 GB (hash buckets + key strings + pointer arrays) that previously coexisted with every dump-side transient — the dump-phase RSS peak. At 2x-kernel scale this is ~7.5 GB at exactly the OOM point. 2. node_by_id was a hash table keyed on STRDUP'D DECIMAL STRINGS of the int64 id — ~0.44 GB of buckets and key strings, plus a snprintf + strdup + hash on every hot find_by_id call. Ids are dense and sequential; a plain grow-on-demand pointer array replaces all of it with one bounds-checked load. 3. The per-node and per-edge `project` field was write-only (every reader uses the store structs or gb->project directly; -Werror confirms) — 8 bytes x 24M instances ~= 192 MB of dead pointers, deleted. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
d9ea73f9e4 |
fix(index): make parallel indexing deterministic (byte-identical runs)
Multi-threaded indexing produced a different graph on every run (xfs: node labels flipping Class/Function/Macro, ~2k edge lines differing across USAGE/ CALLS/IMPORTS/SIMILAR_TO/SEMANTICALLY_RELATED) and lost edges vs repeated runs. Five root causes, all order- or scheduling-dependence: 1. gbuf QN collisions resolved last-wins in MERGE ORDER: in C a struct, a function and a macro can share one name, and the QN scheme does not encode entity kind, so whichever entity a worker merged last overwrote the node's label/name/lines/props (the flickering node set that fed every pass). Both cbm_gbuf_upsert_node and the parallel-merge path now keep the SAME entity's refresh semantics but resolve cross-entity collisions by a canonical content rule -- lexicographically smallest (label, file_path, start_line, name) -- and keep the label/name secondary indexes consistent when the survivor changes (the old overwrite left nodes listed under their original label, which is how the flicker reached find_by_label consumers). 2. Semantic-edges admission raced per-node max_edges budgets on shared atomic counters under dynamic scheduling: scoring stays parallel and pure; pairs carry (func index, candidate rank); one sequential pass admits in canonical order. funcs[] itself is sorted by qualified name up front (the label-index scan order is merge order). 3. Import resolution's symbol-name fallback returned the FIRST same-named node in registration order; now the lexicographically-smallest-QN candidate. 4. Similarity entries inherited merge order into LSH chains + per-node cap truncation, and pair ownership keyed on run-varying node ids; entries are sorted by QN and ownership compares QNs (lsh entries carry the QN). 5. Call-neighbor context (semantic vectors + tokens) truncated MAX_CALLEES subsets in edge-array order; neighbor names are sorted first. Verified: four consecutive multi-threaded kernel-xfs runs produce byte- identical sorted node AND edge sets (13341/58598, zero diff). The repro guard repro_parallel_edge_determinism (MT x K mutual equality) is GREEN; a new deliberately-RED repro_seq_parallel_equivalence tracks the remaining OPEN bug that the sequential and parallel pipelines systematically disagree (~3459 USAGE / ~1666 WRITES lines on xfs) -- separate code paths, fix deferred. Suites: pipeline 216, c_lsp 750, rust_lsp 506, py_lsp 63, java 94, kotlin 78, php 278, registry 54, store_nodes 56, incremental 161, semantic 32, simhash 24 (one test updated: SIMILAR_TO pair direction is now canonical by QN), mcp 140, subprocess 17 -- zero failures. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin VogelandTommy Corbett
|
f6b9ac9d65 |
fix(graph): stable USAGE-edge ownership for directory-module nodes
Since #667 every file in a Java/Go package shares a directory-based module QN that collides with the pipeline's Folder/Project node. Class-level references resolved their enclosing scope to that shared node, whose file_path was clobbered by each file's always-emitted Module def — last writer wins (sequential upsert) or last-merged worker wins (parallel merge) — so USAGE-edge sources conflated across same-package files and varied run to run. Three coordinated changes: - cbm_gbuf_upsert_node: a Module def colliding with a Project/Folder node no longer updates ANY field (the #667 guard only protected the label). - merge_update_existing: same guard on the parallel worker-gbuf merge path; the ID remap stays outside the guard so worker edges still redirect onto the canonical node. - Source-node finders (pass_usages, pass_parallel, pass_calls) treat a lookup landing on a Folder/Project container as a miss and fall back to the per-file File node (new cbm_pipeline_node_is_dir_container helper). Adds tests/repro/repro_issue787.c: exact-source-set, 5-run stability, and >50-file parallel-path guards (red on the unfixed code in both sequential and parallel flavors). Distilled from #828: the author's commits lacked DCO sign-off trailers and could not be cherry-picked; content applied verbatim plus one clang-format fix in graph_buffer.c. Closes #787. Co-authored-by: Tommy Corbett <spde89@gmail.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin VogelandAlexandros Pappas
|
6012f4626e |
fix(graph): IMPORTS edges keyed by local_name across buffer, store and dump
Distills PR #770 (graph-buffer dedup key) and completes the edge-uniqueness contract it left partial: the contract lives in THREE places, and changing only the buffer ships databases that violate their own UNIQUE constraint (PRAGMA integrity_check: non-unique entry in sqlite_autoindex_edges_1). A single 'import { A, B } from ./lib' produced ONE IMPORTS edge: the graph buffer dedups edges on (source_id, target_id, type) and merge-replaces properties on collision, so the second symbol silently overwrote the first. pass_calls.c, pass_usages.c, pass_semantic.c and pass_lsp_cross.c parse one local_name per IMPORTS edge for cross-file resolution, so the dropped symbol's calls failed to resolve too — not just "who imports X" queries. Changes, kept in sync across all three sites: - graph_buffer.c (from #770): make_edge_key() folds local_name into the dedup key for IMPORTS edges only; all three key call-sites updated. Hardened beyond #770: EDGE_KEY_BUF bumped to 256 and oversized local_names are re-keyed with an FNV-1a hash of the full name instead of being silently truncated (two long names sharing a prefix must not collide back into one edge). - store.c: edges gains local_name_gen, a VIRTUAL generated column (IMPORTS -> coalesce(json_extract(properties,'$.local_name'),''), else '' — NOT NULL because NULLs never conflict in a UNIQUE index); uniqueness widened to UNIQUE(source_id, target_id, type, local_name_gen) and the insert upsert's conflict target matches. init_schema probes pre-#768 DBs (no local_name_gen) and fails the open: SQLite cannot ALTER a table constraint in place, and an unopenable DB already takes the existing repair path — full index deletes + rebuilds, artifact import refuses and falls back to a reindex. Read-only query opens skip init_schema and keep working. - sqlite_writer.c/.h: dump DDL matches the widened schema; the hand-built sqlite_autoindex_edges_1 comparator and entry builder include the local_name column; CBMDumpEdge carries local_name extracted via real JSON parsing (yyjson) so index entries match json_extract's unescaped values exactly, per the idx_edges_url_path precedent. - artifact.h: CBM_ARTIFACT_SCHEMA_VERSION 1 -> 2 so old binaries refuse artifacts carrying the widened schema (their 3-column conflict target can no longer prepare against it). Tests (reproduce-first, all red on the unfixed code): - gbuf tier: multi-symbol import -> 2 IMPORTS edges; long-local_name truncation guard. - store tier: distinct local_name coexists as 2 rows, same local_name still upserts, non-IMPORTS dedup unchanged. - writer tier: dumped DB passes integrity_check with 2 sibling imports and exposes matching local_name_gen values. - end-to-end: TS fixture through the real pipeline -> 2 queryable IMPORTS edges AND integrity_check ok; with only the buffer half of the fix this test still fails on integrity_check, proving the schema half is required. Closes #768. Co-authored-by: Alexandros Pappas <11921291+apappas1129@users.noreply.github.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
e51c93554f |
fix(fqn): keep structural nodes from dir-module QN collision; correct dotfile module QN
A directory-based-module language (Go/Java) derives a file's module QN from its directory: a root file's module == the project name and a file in pkg/ has module proj.pkg -- exactly the QNs of the "Project" root and the "pkg/" Folder structural nodes. The per-file Module def then clobbered those nodes' labels through the gbuf upsert (last-write-wins on label), turning Project/Folder into Module. Guard the upsert so a "Module" never downgrades a "Project" or "Folder" -- the directory node is the package container. One guard covers both the sequential upsert and the parallel local-gbuf merge. Separately, a dotfile such as .env had its leading dot stripped as an "extension", leaving an empty stem whose module QN also collided with the project root. Treat a leading dot as a dotfile marker (not an extension) and drop it from the QN segment, so .env -> proj.env (a distinct module). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
5706bf52f0 |
Extract url_path for the dump index via real JSON parsing
The dump path sliced url_path out of the properties string with strstr/strchr, returning ESCAPED bytes and cutting at embedded escaped quotes, while the index's backing column is GENERATED AS json_extract(properties,'$.url_path') and yields the unescaped value. The mismatch left edges 'missing from index idx_edges_url_path' under PRAGMA integrity_check once the formerly-malformed route props became valid JSON. url_path is now parsed with yyjson so the indexed value matches json_extract exactly. |
||
|
Martin Vogel
|
894932ce39 |
Stream the graph dump to SQLite per partition
cbm_gbuf_dump_to_sqlite now drives the streaming page writer in 64K-node partitions instead of building every node record up front. Under memory pressure (RSS over the configured budget) it frees each partition's heavy properties_json once those rows are persisted — the properties column is write-once and never read again — which bounds the dump/finalize memory peak on large graphs. Output is byte-for-byte unchanged when the pressure path does not engage, so normal indexing and the test suite keep the buffer intact. Verified: full suite green, PRAGMA integrity_check ok, 1x Linux index node-exact and regression-safe. |
||
|
Martin Vogel
|
6ed58bc797 |
Intern repetitive gbuf string fields to cut indexing memory
Node label/file_path and edge type are highly repetitive (~30 labels, ~15 edge types, ~88k file paths across millions of records) yet were strdup'd per node/edge. Intern them into a per-buffer pool so identical content collapses to a single owned copy, freed once at teardown. Cuts ~20M small allocations to ~88k and lowers peak RSS on a full Linux-kernel index by ~1.46 GB (16.45 -> 14.99 GB), with no change to graph output (node count identical; edge dedup is content-keyed). |
||
|
Martin VogelandKoolerx
|
a0715ff1eb |
Add JS/TS IMPORTS resolution, Channel schema, lint cleanup
Features: - Relative import resolution for JS/TS/Python/Ruby (./foo, ../bar, leading dots) via cbm_pipeline_resolve_relative_import in fqn.c - Generic Channel detection: Socket.IO emit/on, EventEmitter patterns produce Channel nodes with EMITS/LISTENS_ON edges and transport property (extract_channels.c) - Constant resolution trick: `const EVENT = "foo"; emit(EVENT)` resolves the channel name through a per-file constant table Lint fixes (337 errors across 16 files): - Named constants replacing all magic numbers (enums/defines) - Cognitive complexity reduced via helper extraction (table-driven sqlite_writer, pipeline predump passes, graph_buffer dump phases) - SQLITE_TRANSIENT int-to-ptr workaround in store.c + mcp.c - clang-format + cppcheck clean Co-Authored-By: Koolerx <tommy@koolerx.com> |
||
|
Martin Vogel
|
8a06d78ac7 |
Parallelize post-passes, fix mode filtering + semantic edge quality
- Parallelize pass_similarity and pass_semantic_edges via worker pool with thread-local edge buffers; sequential final merge since gbuf is not thread-safe. Adds cbm_lsh_query_into() as a thread-safe variant with caller-provided candidate buffer. - Add activatable profiling subsystem (CBM_PROFILE=1 env or --profile flag) for step-level timing of extract, resolve, corpus build, vector phases, and sqlite dump. Zero overhead when disabled. - Fix cbm_index_mode_t enum mismatch between pipeline.h (FULL=0, MODERATE=1, FAST=2) and discover.h (FULL=0, FAST=1). mode=fast silently no-op'd fast-discovery filtering because discover.c compared against the wrong value. Linux kernel fast mode went 1:40 -> 3:11 as a result; now back to 1:40. Broaden the filter guard to mode != CBM_MODE_FULL so MODERATE and FAST both get aggressive discovery. - Clamp cbm_sem_combined_score output to [0, 1]. The proximity multiplier returns up to 1.10 as a same-file boost which could push the final cosine score above 1.0. - Short-circuit semantic scoring when MinHash jaccard >= 0.95. Exact near-clones are already emitted as SIMILAR_TO edges; returning 0 here avoids flooding SEMANTICALLY_RELATED with cross-service copy-paste boilerplate and frees the edge budget for genuine vocabulary-bridged relations. - Validate search_graph semantic_query as an array of strings and return a clear error for a single-string input. Update the tool description to spell out the requirement explicitly with an example. - JSON-escape user-controlled strings (callee names, call arguments, URL paths, import local_name) in call/argument properties. Introduces cbm_json_escape() in foundation/str_util. - Skip SQLite pending_byte_page (file offset 0x40000000) during raw page writes in sqlite_writer to avoid corrupting databases that cross the 1 GiB boundary. - Migrate pretrained vector blob from UniXcoder (51K tokens) to nomic-embed-code (40856 tokens x 768d int8). Includes the extraction script under scripts/extract_nomic_vectors.py. |
||
|
Martin Vogel
|
e81a796fe6 |
Embed Nomic nomic-embed-code pretrained vectors (40K tokens × 768d) for semantic search quality
Major quality improvement: replace random index vectors with pre-trained Nomic nomic-embed-code token embeddings. Tokens like 'error' and 'exception' now start with similar vectors (learned from millions of code repos) instead of arbitrary random projections. Co-occurrence enrichment adds project-specific context on top. Architecture: - vendored/nomic/code_vectors.bin: 37.7MB raw int8 vectors - vendored/nomic/code_vectors_blob.S: assembler .incbin (instant build) - vendored/nomic/code_vectors.h: extern declarations + pretrained_vec_at() - vendored/nomic/code_tokens.h: 40856 token strings (575KB) - semantic.c: cbm_sem_random_index() now looks up pretrained vectors first, falls back to sparse random for unknown tokens - CBM_SEM_DIM raised from 256 to 768 to match Nomic nomic-embed-code Also: RRI (Reflective Random Indexing), code pattern vocabulary injection, 120+ abbreviation expansions, callee/caller/body token enrichment, label filter (Function/Method/Class only) in vector search SQL. Binary size: 136MB → 169MB (+33MB from embedded vectors). Search quality: keyword queries return relevant error-handling functions. Domain-specific keyword queries return the expected functions. |
||
|
Martin Vogel
|
7a9b7db830 |
Add semantic_query to search_graph: vector search via cbm_cosine_i8 SQL function
Vector search pipeline (end-to-end): - store.c: cbm_cosine_i8 custom SQLite function (int8 dot product with sqrt magnitude), registered at store open time - store.c: cbm_store_vector_search() builds merged query vector from keywords via base RI vectors (xxHash), int8-quantizes, runs SQL query with JOIN nodes ON node_id = id for metadata - mcp.c: search_graph gains semantic_query param (array of strings), returns semantic_results alongside regular results - sqlite_writer.c: fixed build_vector_record to include node_id in record body (matches build_node_record pattern) - mcp.c: use yyjson_mut_obj_add_strcpy (not add_str) to avoid use-after-free when vector results are freed before JSON serialization Quality note: base RI vectors at query time don't capture co-occurrence enrichment — results are moderate quality. Storing enriched token vectors would improve this significantly (future work). |
||
|
Martin Vogel
|
1b84943000 |
Separate perf tests from CI, fix cross-platform build issues
- Add CBM_SKIP_PERF=1 env var to skip incremental/perf test suite - CI and Docker test targets skip perf by default (run.sh perf for manual) - Convert all perf assertions to warnings (log timing, never block) - Fix store.h anonymous enum in struct (GCC rejects, clang accepts) - Fix test_store_search.c mkstemp on non-template path - Add ca-certificates to Docker test image for git HTTPS - Add cbm_gmtime_r shim in compat.h (Windows gmtime_s wrapper) - Fix compat.c missing constants.h include (Windows build) - Fix platform.c _environ redeclaration on mingw - Rename trace_call_path -> trace_path in smoke/soak/fuzz scripts |
||
|
Martin Vogel
|
7fa3acd0c6 | WIP: strict linting + RAM-first pipeline (lint fixes pending) | ||
|
Martin Vogel
|
6e4ca93cf0 |
Split 168 functions to cognitive complexity 25, zero lint errors
Lower thresholds to industry defaults: cognitive-complexity: 25 (was 250) statements: 200 (was 400) lines: 400 (was 800) All 168 functions split into smaller helpers across 44 files. Zero NOLINTNEXTLINE suppressions remain. Zero clang-tidy errors. Add clang-tidy to scripts/lint.sh (--ci to skip where unavailable). Fix sqlite_writer B-tree PageRef initialization. Fix Terraform struct parsing, Louvain null guards, const qualifiers, shadow variables. 2741 tests pass. |
||
|
Martin Vogel
|
87c2e144c0 |
Set cognitive complexity to industry default (25), tag 168 functions
Lower thresholds to best-practice values: cognitive-complexity: 250 -> 25 (industry default) statements: 400 -> 200 lines: 800 -> 400 Add NOLINTNEXTLINE to 168 functions that currently exceed these thresholds. Each suppression will be removed as the function is split in subsequent commits. Zero net behavioral changes. Also: extract execute_with_clause from cypher execute_single. |
||
|
Martin Vogel
|
f26acd4ced |
Eliminate NOLINTNEXTLINE suppressions + iterative AST walkers
Remove ~480 dead per-line NOLINTNEXTLINE comments. Update .clang-tidy with documented upstream bug references for disabled checks. Convert key AST walkers to iterative with explicit stacks: walk_defs (with context-aware stack frames for class nesting), walk_throws, walk_readwrites, walk_calls, walk_usages, walk_env_accesses, extract_elixir_call, glob_match, cbm_yaml_free, rmdir_recursive, resolve_func_name, walk_variables_iter, count_branching. Reversed child push for left-to-right traversal. Disable readability-implicit-bool-conversion globally — designed for C++, flags all idiomatic C patterns (bool = x && y) with no config to allow them. Fix clang-format, implicit-bool edge cases, add test_helpers.h. |
||
|
Martin Vogel
|
80680ea367 |
Cross-service communication discovery + RAM-first incremental indexing
AST-based detection of HTTP calls, async dispatch (Pub/Sub, Cloud Tasks, Kafka, SQS, etc.), and config accesses via resolved qualified names. Route nodes as cross-service rendezvous points with infra→handler matching. Constant propagation for module-level string assignments. YAML infrastructure URL extraction from Cloud Scheduler configs. RAM-first incremental pipeline: load DB into graph buffer, purge changed file nodes, extract directly into existing buffer (resolver sees all nodes), dump back to disk. Zero edge gap on kubernetes/django/meilisearch/neovim. - service_patterns.c: ~170 library patterns (90 HTTP, 50 async, 30 config) - pass_route_nodes.c: Route node creation + infra URL matching - extract_unified.c: string constant collection + string ref classification - extract_calls.c: first_string_arg + keyword argument extraction - pipeline_incremental.c: RAM-first load→purge→extract→resolve→dump - graph_buffer.c: load_from_db, delete_by_file, foreach visitors - C++ LSP crash fix: NULL guard in cbm_type_substitute |
||
|
Martin Vogel
|
e6e9c58e80 |
Fix all memory leaks + Docker test infra gaps
Leaks fixed (5506 bytes → 0): - sqlite3_open_v2 allocates handle even on CANTOPEN — must call sqlite3_close before freeing struct (4x 792 bytes, 32 indirect) - graph_buffer node_by_id: reuse existing heap key on hash table replace to avoid leak (2 bytes) Docker test infrastructure: - python3 (full) instead of python3-minimal (missing json module) - git installed (19 tests were silently skipped) - sqlite3_close_v2 + sqlite3_shutdown at test exit |
||
|
Martin Vogel
|
eaf9c83066 |
Eliminate rename() — reindex deletes old DB, index writes directly
Architecture change to prevent concurrent write corruption: Indexing (new project): cbm_write_db writes project.db directly Reindexing (existing): delete project.db, then index as new Incremental: upsert changes into existing project.db (unchanged) Root cause: rename(project.db.tmp, project.db) while watcher/MCP had the old file open via SQLite caused WAL corruption — stale connections wrote to deleted inodes. Changes: - cbm_gbuf_dump_to_sqlite: write to final path, no .tmp + rename - pipeline.c: if DB exists but not eligible for incremental, delete it before proceeding with full index (reindex = delete + index) - Pipeline lock (atomic spinlock) serializes concurrent runs: MCP/autoindex block, watcher skips if busy - delete_project acquires pipeline lock before unlinking - BEGIN IMMEDIATE for all write transactions (respects busy_timeout) - cbm_store_check_integrity: detects corrupt DBs (>5 project rows or invalid root_path), auto-deletes with ERROR log - cbm_store_get_db accessor for test corruption simulation Tests: 5 integrity tests + 4 concurrency lock tests |
||
|
Martin Vogel
|
d6a5f8984e |
Add incremental reindex + ADR hints in index/schema/adr responses
Incremental pipeline (pipeline_incremental.c): - Disk-based delta re-indexing: classifies files by mtime+size vs stored hashes, deletes changed nodes (edges cascade), re-parses only changed files, merges into existing DB via cbm_gbuf_merge_into_store - pipeline.c auto-routes: existing DB with hashes -> incremental, else full - Full pipeline persists file hashes after dump for next incremental run - No-op when nothing changed (watcher polls every 5-60s, 99% skip) - 4 tests: noop, changed file, deleted file, new file added ADR hints (ported from Go): - index_repository: adr_present + adr_hint when no ADR exists - get_graph_schema: adr_present + adr_hint per project - manage_adr GET: adr_hint with creation instructions when no ADR |
||
|
Martin Vogel
|
26a78142d6 |
Fix Windows/MinGW build: TLS, strndup, sys/wait.h
- CBM_TLS: use _Thread_local (C11 standard) instead of __declspec(thread) which only works on MSVC, not MinGW GCC - Replace bare strndup() with cbm_strndup() in 6 files (strndup is POSIX, not available on Windows) - Guard sys/wait.h with #ifndef _WIN32 in http_server.c |
||
|
Martin Vogel
|
a1d849aaaa |
Fix all memory leaks, vendor sqlite3, add Docker test infrastructure
Leak fixes (13 sources, 1.4MB → 0 bytes): - Cypher: free col_names after rb_set_columns, free rejected inline prop nodes, store WITH aliases in qualified_name for auto-cleanup - Store: add cbm_project_free_fields, fix find_node_ids_by_qns leak - Graph buffer: free strdup'd hash keys during delete operations - SQL scanner: free old start_tag before reassignment in deserialize Platform independence: - Vendor sqlite3 amalgamation (3.49.1) — eliminates system libsqlite3 dependency, enables full ASan/LeakSanitizer instrumentation - Add -I for vendored tree-sitter unicode headers — eliminates hidden dependency on system libicu-dev - Add test-infrastructure/ with Docker Compose for local GCC+ASan testing that mirrors CI exactly |
||
|
Martin Vogel
|
18fa9979ee |
Replace vmem with mimalloc global allocator, add extraction-phase prescan, fix __init__.py QN collision
Memory management: - Vendor mimalloc v2.1.9 as global allocator (MI_OVERRIDE=1 in prod) - New mem.h/mem.c: RSS-based budget tracking via mi_process_info() - Remove vmem.c/vmem.h (mmap-based budget tracking) - Remove slab tier2 bump allocator (~300 LOC); >64B goes to mimalloc - Slab tier1 pages from malloc (= mimalloc) instead of vmem - Arena blocks from malloc instead of vmem - Budget raised from 35% to 50% RAM (no more untracked C++ heap) Extraction-phase prescan (eliminates disk re-reads): - HTTP call sites: keyword check + URL extraction during extraction - HTTP routes: decorator + source-based extraction during extraction - Config file refs: regex scan during extraction - httplinks: 41.8s → 13ms on Linux kernel (3,212x faster) - configlink: 41.4s → 0.8s on Linux kernel (54x faster) - Linux kernel fast-mode total: 2m38s → 1m18s Bug fixes: - __init__.py Module QN no longer collides with Folder QN - index.ts same fix for JS/TS packages - 13 regression tests for QN collision at FQN + extraction layers - search_graph/search_code default limit raised from 10 to 500k - Resolve all clang-tidy, cppcheck, and clang-format warnings Repo cleanup: - tree-sitter-form, tree-sitter-magma moved to tools/ - .gitignore: build/, node_modules/, graph-ui/dist/, TEST_PLAN.md |
||
|
Martin Vogel
|
0d216c7936 |
Resolve all clang-tidy, cppcheck, and clang-format warnings across 94 files
Enforce WarningsAsErrors: '*' with zero suppressions of entire check categories. All non-FP warnings fixed at source; true false positives get per-instance NOLINT with reasoning. Fixes include: braces around statements (1679), isolate declarations (75), explicit casts for multilevel pointer conversions, missing includes, function parameter documentation via NOLINT, and proper NOLINTBEGIN/NOLINTEND nesting. |
||
|
Martin Vogel
|
94a9a92f83 |
Add C-native pipeline with blocking clang-tidy, cppcheck, clang-format linting
Complete C implementation of the indexing pipeline (src/), parallel worker pool, graph buffer with merge support, SQLite writer, and 1893-test suite. Linter setup: zero warnings from clang-tidy (all checks enabled, no NOLINT suppressions), cppcheck, and clang-format. All issues fixed at source — proper headers for external linkage, GROW_ARRAY macro restructured to eliminate type parameter, null-deref paths guarded, named intermediates for suspicious-argument checks. |