The deterministic same-QN collision rule picked the lexicographically
smallest (label, file_path, start_line, name), which silently inverted
the documented upsert/merge semantics: re-upserting a QN with updated
fields kept the OLD content (gbuf_upsert_updates,
gbuf_upsert_same_qn_updates_all_fields, gbuf_merge_overlapping_qns all
red on every platform).
Replace it with a single mixed-direction total order applied in both
cbm_gbuf_upsert_node and merge_update_existing: smallest file_path,
then LARGEST start_line, then largest name/label. Still a pure content
function of the two candidates (commutative, associative,
scheduling-independent — multi-threaded runs stay byte-identical,
verified by the xfs MT x4 fingerprint guard), and line-descending
within a file restores the classic contract: a later definition in the
same file replaces the earlier one, and a full tie is the same entity
refreshed in place.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Three memory quick-wins from the gbuf compaction research (graph output
verified identical; all delete/incremental paths suite-covered):
1. Release ALL lookup indexes right after build_dump_nodes instead of after
edge building: nothing later in the dump reads them, and at kernel scale
they are ~3.8 GB (hash buckets + key strings + pointer arrays) that
previously coexisted with every dump-side transient — the dump-phase RSS
peak. At 2x-kernel scale this is ~7.5 GB at exactly the OOM point.
2. node_by_id was a hash table keyed on STRDUP'D DECIMAL STRINGS of the
int64 id — ~0.44 GB of buckets and key strings, plus a snprintf + strdup +
hash on every hot find_by_id call. Ids are dense and sequential; a plain
grow-on-demand pointer array replaces all of it with one bounds-checked
load.
3. The per-node and per-edge `project` field was write-only (every reader
uses the store structs or gb->project directly; -Werror confirms) —
8 bytes x 24M instances ~= 192 MB of dead pointers, deleted.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Multi-threaded indexing produced a different graph on every run (xfs: node
labels flipping Class/Function/Macro, ~2k edge lines differing across USAGE/
CALLS/IMPORTS/SIMILAR_TO/SEMANTICALLY_RELATED) and lost edges vs repeated
runs. Five root causes, all order- or scheduling-dependence:
1. gbuf QN collisions resolved last-wins in MERGE ORDER: in C a struct, a
function and a macro can share one name, and the QN scheme does not encode
entity kind, so whichever entity a worker merged last overwrote the node's
label/name/lines/props (the flickering node set that fed every pass). Both
cbm_gbuf_upsert_node and the parallel-merge path now keep the SAME entity's
refresh semantics but resolve cross-entity collisions by a canonical
content rule -- lexicographically smallest (label, file_path, start_line,
name) -- and keep the label/name secondary indexes consistent when the
survivor changes (the old overwrite left nodes listed under their original
label, which is how the flicker reached find_by_label consumers).
2. Semantic-edges admission raced per-node max_edges budgets on shared atomic
counters under dynamic scheduling: scoring stays parallel and pure; pairs
carry (func index, candidate rank); one sequential pass admits in canonical
order. funcs[] itself is sorted by qualified name up front (the label-index
scan order is merge order).
3. Import resolution's symbol-name fallback returned the FIRST same-named node
in registration order; now the lexicographically-smallest-QN candidate.
4. Similarity entries inherited merge order into LSH chains + per-node cap
truncation, and pair ownership keyed on run-varying node ids; entries are
sorted by QN and ownership compares QNs (lsh entries carry the QN).
5. Call-neighbor context (semantic vectors + tokens) truncated MAX_CALLEES
subsets in edge-array order; neighbor names are sorted first.
Verified: four consecutive multi-threaded kernel-xfs runs produce byte-
identical sorted node AND edge sets (13341/58598, zero diff). The repro guard
repro_parallel_edge_determinism (MT x K mutual equality) is GREEN; a new
deliberately-RED repro_seq_parallel_equivalence tracks the remaining OPEN bug
that the sequential and parallel pipelines systematically disagree (~3459
USAGE / ~1666 WRITES lines on xfs) -- separate code paths, fix deferred.
Suites: pipeline 216, c_lsp 750, rust_lsp 506, py_lsp 63, java 94, kotlin 78,
php 278, registry 54, store_nodes 56, incremental 161, semantic 32, simhash 24
(one test updated: SIMILAR_TO pair direction is now canonical by QN), mcp 140,
subprocess 17 -- zero failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Since #667 every file in a Java/Go package shares a directory-based module
QN that collides with the pipeline's Folder/Project node. Class-level
references resolved their enclosing scope to that shared node, whose
file_path was clobbered by each file's always-emitted Module def — last
writer wins (sequential upsert) or last-merged worker wins (parallel merge)
— so USAGE-edge sources conflated across same-package files and varied run
to run.
Three coordinated changes:
- cbm_gbuf_upsert_node: a Module def colliding with a Project/Folder node
no longer updates ANY field (the #667 guard only protected the label).
- merge_update_existing: same guard on the parallel worker-gbuf merge path;
the ID remap stays outside the guard so worker edges still redirect onto
the canonical node.
- Source-node finders (pass_usages, pass_parallel, pass_calls) treat a
lookup landing on a Folder/Project container as a miss and fall back to
the per-file File node (new cbm_pipeline_node_is_dir_container helper).
Adds tests/repro/repro_issue787.c: exact-source-set, 5-run stability, and
>50-file parallel-path guards (red on the unfixed code in both sequential
and parallel flavors).
Distilled from #828: the author's commits lacked DCO sign-off trailers and
could not be cherry-picked; content applied verbatim plus one clang-format
fix in graph_buffer.c.
Closes#787.
Co-authored-by: Tommy Corbett <spde89@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distills PR #770 (graph-buffer dedup key) and completes the edge-uniqueness
contract it left partial: the contract lives in THREE places, and changing
only the buffer ships databases that violate their own UNIQUE constraint
(PRAGMA integrity_check: non-unique entry in sqlite_autoindex_edges_1).
A single 'import { A, B } from ./lib' produced ONE IMPORTS edge: the graph
buffer dedups edges on (source_id, target_id, type) and merge-replaces
properties on collision, so the second symbol silently overwrote the first.
pass_calls.c, pass_usages.c, pass_semantic.c and pass_lsp_cross.c parse one
local_name per IMPORTS edge for cross-file resolution, so the dropped
symbol's calls failed to resolve too — not just "who imports X" queries.
Changes, kept in sync across all three sites:
- graph_buffer.c (from #770): make_edge_key() folds local_name into the
dedup key for IMPORTS edges only; all three key call-sites updated.
Hardened beyond #770: EDGE_KEY_BUF bumped to 256 and oversized
local_names are re-keyed with an FNV-1a hash of the full name instead
of being silently truncated (two long names sharing a prefix must not
collide back into one edge).
- store.c: edges gains local_name_gen, a VIRTUAL generated column
(IMPORTS -> coalesce(json_extract(properties,'$.local_name'),''),
else '' — NOT NULL because NULLs never conflict in a UNIQUE index);
uniqueness widened to UNIQUE(source_id, target_id, type,
local_name_gen) and the insert upsert's conflict target matches.
init_schema probes pre-#768 DBs (no local_name_gen) and fails the
open: SQLite cannot ALTER a table constraint in place, and an
unopenable DB already takes the existing repair path — full index
deletes + rebuilds, artifact import refuses and falls back to a
reindex. Read-only query opens skip init_schema and keep working.
- sqlite_writer.c/.h: dump DDL matches the widened schema; the hand-built
sqlite_autoindex_edges_1 comparator and entry builder include the
local_name column; CBMDumpEdge carries local_name extracted via real
JSON parsing (yyjson) so index entries match json_extract's unescaped
values exactly, per the idx_edges_url_path precedent.
- artifact.h: CBM_ARTIFACT_SCHEMA_VERSION 1 -> 2 so old binaries refuse
artifacts carrying the widened schema (their 3-column conflict target
can no longer prepare against it).
Tests (reproduce-first, all red on the unfixed code):
- gbuf tier: multi-symbol import -> 2 IMPORTS edges; long-local_name
truncation guard.
- store tier: distinct local_name coexists as 2 rows, same local_name
still upserts, non-IMPORTS dedup unchanged.
- writer tier: dumped DB passes integrity_check with 2 sibling imports
and exposes matching local_name_gen values.
- end-to-end: TS fixture through the real pipeline -> 2 queryable
IMPORTS edges AND integrity_check ok; with only the buffer half of
the fix this test still fails on integrity_check, proving the schema
half is required.
Closes#768.
Co-authored-by: Alexandros Pappas <11921291+apappas1129@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A directory-based-module language (Go/Java) derives a file's module QN
from its directory: a root file's module == the project name and a file
in pkg/ has module proj.pkg -- exactly the QNs of the "Project" root and
the "pkg/" Folder structural nodes. The per-file Module def then clobbered
those nodes' labels through the gbuf upsert (last-write-wins on label),
turning Project/Folder into Module. Guard the upsert so a "Module" never
downgrades a "Project" or "Folder" -- the directory node is the package
container. One guard covers both the sequential upsert and the parallel
local-gbuf merge.
Separately, a dotfile such as .env had its leading dot stripped as an
"extension", leaving an empty stem whose module QN also collided with the
project root. Treat a leading dot as a dotfile marker (not an extension)
and drop it from the QN segment, so .env -> proj.env (a distinct module).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The dump path sliced url_path out of the properties string with strstr/strchr, returning ESCAPED bytes and cutting at embedded escaped quotes, while the index's backing column is GENERATED AS json_extract(properties,'$.url_path') and yields the unescaped value. The mismatch left edges 'missing from index idx_edges_url_path' under PRAGMA integrity_check once the formerly-malformed route props became valid JSON. url_path is now parsed with yyjson so the indexed value matches json_extract exactly.
cbm_gbuf_dump_to_sqlite now drives the streaming page writer in
64K-node partitions instead of building every node record up front.
Under memory pressure (RSS over the configured budget) it frees each
partition's heavy properties_json once those rows are persisted — the
properties column is write-once and never read again — which bounds the
dump/finalize memory peak on large graphs.
Output is byte-for-byte unchanged when the pressure path does not
engage, so normal indexing and the test suite keep the buffer intact.
Verified: full suite green, PRAGMA integrity_check ok, 1x Linux index
node-exact and regression-safe.
Node label/file_path and edge type are highly repetitive (~30 labels,
~15 edge types, ~88k file paths across millions of records) yet were
strdup'd per node/edge. Intern them into a per-buffer pool so identical
content collapses to a single owned copy, freed once at teardown.
Cuts ~20M small allocations to ~88k and lowers peak RSS on a full
Linux-kernel index by ~1.46 GB (16.45 -> 14.99 GB), with no change to
graph output (node count identical; edge dedup is content-keyed).
- Parallelize pass_similarity and pass_semantic_edges via worker pool with
thread-local edge buffers; sequential final merge since gbuf is not
thread-safe. Adds cbm_lsh_query_into() as a thread-safe variant with
caller-provided candidate buffer.
- Add activatable profiling subsystem (CBM_PROFILE=1 env or --profile flag)
for step-level timing of extract, resolve, corpus build, vector phases,
and sqlite dump. Zero overhead when disabled.
- Fix cbm_index_mode_t enum mismatch between pipeline.h (FULL=0, MODERATE=1,
FAST=2) and discover.h (FULL=0, FAST=1). mode=fast silently no-op'd
fast-discovery filtering because discover.c compared against the wrong
value. Linux kernel fast mode went 1:40 -> 3:11 as a result; now back to
1:40. Broaden the filter guard to mode != CBM_MODE_FULL so MODERATE and
FAST both get aggressive discovery.
- Clamp cbm_sem_combined_score output to [0, 1]. The proximity multiplier
returns up to 1.10 as a same-file boost which could push the final
cosine score above 1.0.
- Short-circuit semantic scoring when MinHash jaccard >= 0.95. Exact
near-clones are already emitted as SIMILAR_TO edges; returning 0 here
avoids flooding SEMANTICALLY_RELATED with cross-service copy-paste
boilerplate and frees the edge budget for genuine vocabulary-bridged
relations.
- Validate search_graph semantic_query as an array of strings and return
a clear error for a single-string input. Update the tool description
to spell out the requirement explicitly with an example.
- JSON-escape user-controlled strings (callee names, call arguments,
URL paths, import local_name) in call/argument properties. Introduces
cbm_json_escape() in foundation/str_util.
- Skip SQLite pending_byte_page (file offset 0x40000000) during raw page
writes in sqlite_writer to avoid corrupting databases that cross the
1 GiB boundary.
- Migrate pretrained vector blob from UniXcoder (51K tokens) to
nomic-embed-code (40856 tokens x 768d int8). Includes the extraction
script under scripts/extract_nomic_vectors.py.
Major quality improvement: replace random index vectors with pre-trained
Nomic nomic-embed-code token embeddings. Tokens like 'error' and 'exception' now start
with similar vectors (learned from millions of code repos) instead of
arbitrary random projections. Co-occurrence enrichment adds project-specific
context on top.
Architecture:
- vendored/nomic/code_vectors.bin: 37.7MB raw int8 vectors
- vendored/nomic/code_vectors_blob.S: assembler .incbin (instant build)
- vendored/nomic/code_vectors.h: extern declarations + pretrained_vec_at()
- vendored/nomic/code_tokens.h: 40856 token strings (575KB)
- semantic.c: cbm_sem_random_index() now looks up pretrained vectors first,
falls back to sparse random for unknown tokens
- CBM_SEM_DIM raised from 256 to 768 to match Nomic nomic-embed-code
Also: RRI (Reflective Random Indexing), code pattern vocabulary injection,
120+ abbreviation expansions, callee/caller/body token enrichment,
label filter (Function/Method/Class only) in vector search SQL.
Binary size: 136MB → 169MB (+33MB from embedded vectors).
Search quality: keyword queries return relevant error-handling functions.
Domain-specific keyword queries return the expected functions.
Vector search pipeline (end-to-end):
- store.c: cbm_cosine_i8 custom SQLite function (int8 dot product with
sqrt magnitude), registered at store open time
- store.c: cbm_store_vector_search() builds merged query vector from
keywords via base RI vectors (xxHash), int8-quantizes, runs SQL query
with JOIN nodes ON node_id = id for metadata
- mcp.c: search_graph gains semantic_query param (array of strings),
returns semantic_results alongside regular results
- sqlite_writer.c: fixed build_vector_record to include node_id in
record body (matches build_node_record pattern)
- mcp.c: use yyjson_mut_obj_add_strcpy (not add_str) to avoid
use-after-free when vector results are freed before JSON serialization
Quality note: base RI vectors at query time don't capture co-occurrence
enrichment — results are moderate quality. Storing enriched token vectors
would improve this significantly (future work).
- Add CBM_SKIP_PERF=1 env var to skip incremental/perf test suite
- CI and Docker test targets skip perf by default (run.sh perf for manual)
- Convert all perf assertions to warnings (log timing, never block)
- Fix store.h anonymous enum in struct (GCC rejects, clang accepts)
- Fix test_store_search.c mkstemp on non-template path
- Add ca-certificates to Docker test image for git HTTPS
- Add cbm_gmtime_r shim in compat.h (Windows gmtime_s wrapper)
- Fix compat.c missing constants.h include (Windows build)
- Fix platform.c _environ redeclaration on mingw
- Rename trace_call_path -> trace_path in smoke/soak/fuzz scripts
Lower thresholds to best-practice values:
cognitive-complexity: 250 -> 25 (industry default)
statements: 400 -> 200
lines: 800 -> 400
Add NOLINTNEXTLINE to 168 functions that currently exceed these thresholds.
Each suppression will be removed as the function is split in subsequent
commits. Zero net behavioral changes.
Also: extract execute_with_clause from cypher execute_single.
Architecture change to prevent concurrent write corruption:
Indexing (new project): cbm_write_db writes project.db directly
Reindexing (existing): delete project.db, then index as new
Incremental: upsert changes into existing project.db (unchanged)
Root cause: rename(project.db.tmp, project.db) while watcher/MCP
had the old file open via SQLite caused WAL corruption — stale
connections wrote to deleted inodes.
Changes:
- cbm_gbuf_dump_to_sqlite: write to final path, no .tmp + rename
- pipeline.c: if DB exists but not eligible for incremental, delete
it before proceeding with full index (reindex = delete + index)
- Pipeline lock (atomic spinlock) serializes concurrent runs:
MCP/autoindex block, watcher skips if busy
- delete_project acquires pipeline lock before unlinking
- BEGIN IMMEDIATE for all write transactions (respects busy_timeout)
- cbm_store_check_integrity: detects corrupt DBs (>5 project rows
or invalid root_path), auto-deletes with ERROR log
- cbm_store_get_db accessor for test corruption simulation
Tests: 5 integrity tests + 4 concurrency lock tests
Incremental pipeline (pipeline_incremental.c):
- Disk-based delta re-indexing: classifies files by mtime+size vs stored
hashes, deletes changed nodes (edges cascade), re-parses only changed
files, merges into existing DB via cbm_gbuf_merge_into_store
- pipeline.c auto-routes: existing DB with hashes -> incremental, else full
- Full pipeline persists file hashes after dump for next incremental run
- No-op when nothing changed (watcher polls every 5-60s, 99% skip)
- 4 tests: noop, changed file, deleted file, new file added
ADR hints (ported from Go):
- index_repository: adr_present + adr_hint when no ADR exists
- get_graph_schema: adr_present + adr_hint per project
- manage_adr GET: adr_hint with creation instructions when no ADR
- CBM_TLS: use _Thread_local (C11 standard) instead of __declspec(thread)
which only works on MSVC, not MinGW GCC
- Replace bare strndup() with cbm_strndup() in 6 files (strndup is POSIX,
not available on Windows)
- Guard sys/wait.h with #ifndef _WIN32 in http_server.c
Enforce WarningsAsErrors: '*' with zero suppressions of entire check categories.
All non-FP warnings fixed at source; true false positives get per-instance NOLINT
with reasoning. Fixes include: braces around statements (1679), isolate declarations
(75), explicit casts for multilevel pointer conversions, missing includes, function
parameter documentation via NOLINT, and proper NOLINTBEGIN/NOLINTEND nesting.
Complete C implementation of the indexing pipeline (src/), parallel worker
pool, graph buffer with merge support, SQLite writer, and 1893-test suite.
Linter setup: zero warnings from clang-tidy (all checks enabled, no NOLINT
suppressions), cppcheck, and clang-format. All issues fixed at source —
proper headers for external linkage, GROW_ARRAY macro restructured to
eliminate type parameter, null-deref paths guarded, named intermediates
for suspicious-argument checks.
Martin Vogel