Multi-threaded indexing produced a different graph on every run (xfs: node
labels flipping Class/Function/Macro, ~2k edge lines differing across USAGE/
CALLS/IMPORTS/SIMILAR_TO/SEMANTICALLY_RELATED) and lost edges vs repeated
runs. Five root causes, all order- or scheduling-dependence:
1. gbuf QN collisions resolved last-wins in MERGE ORDER: in C a struct, a
function and a macro can share one name, and the QN scheme does not encode
entity kind, so whichever entity a worker merged last overwrote the node's
label/name/lines/props (the flickering node set that fed every pass). Both
cbm_gbuf_upsert_node and the parallel-merge path now keep the SAME entity's
refresh semantics but resolve cross-entity collisions by a canonical
content rule -- lexicographically smallest (label, file_path, start_line,
name) -- and keep the label/name secondary indexes consistent when the
survivor changes (the old overwrite left nodes listed under their original
label, which is how the flicker reached find_by_label consumers).
2. Semantic-edges admission raced per-node max_edges budgets on shared atomic
counters under dynamic scheduling: scoring stays parallel and pure; pairs
carry (func index, candidate rank); one sequential pass admits in canonical
order. funcs[] itself is sorted by qualified name up front (the label-index
scan order is merge order).
3. Import resolution's symbol-name fallback returned the FIRST same-named node
in registration order; now the lexicographically-smallest-QN candidate.
4. Similarity entries inherited merge order into LSH chains + per-node cap
truncation, and pair ownership keyed on run-varying node ids; entries are
sorted by QN and ownership compares QNs (lsh entries carry the QN).
5. Call-neighbor context (semantic vectors + tokens) truncated MAX_CALLEES
subsets in edge-array order; neighbor names are sorted first.
Verified: four consecutive multi-threaded kernel-xfs runs produce byte-
identical sorted node AND edge sets (13341/58598, zero diff). The repro guard
repro_parallel_edge_determinism (MT x K mutual equality) is GREEN; a new
deliberately-RED repro_seq_parallel_equivalence tracks the remaining OPEN bug
that the sequential and parallel pipelines systematically disagree (~3459
USAGE / ~1666 WRITES lines on xfs) -- separate code paths, fix deferred.
Suites: pipeline 216, c_lsp 750, rust_lsp 506, py_lsp 63, java 94, kotlin 78,
php 278, registry 54, store_nodes 56, incremental 161, semantic 32, simhash 24
(one test updated: SIMILAR_TO pair direction is now canonical by QN), mcp 140,
subprocess 17 -- zero failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
- Parallelize pass_similarity and pass_semantic_edges via worker pool with
thread-local edge buffers; sequential final merge since gbuf is not
thread-safe. Adds cbm_lsh_query_into() as a thread-safe variant with
caller-provided candidate buffer.
- Add activatable profiling subsystem (CBM_PROFILE=1 env or --profile flag)
for step-level timing of extract, resolve, corpus build, vector phases,
and sqlite dump. Zero overhead when disabled.
- Fix cbm_index_mode_t enum mismatch between pipeline.h (FULL=0, MODERATE=1,
FAST=2) and discover.h (FULL=0, FAST=1). mode=fast silently no-op'd
fast-discovery filtering because discover.c compared against the wrong
value. Linux kernel fast mode went 1:40 -> 3:11 as a result; now back to
1:40. Broaden the filter guard to mode != CBM_MODE_FULL so MODERATE and
FAST both get aggressive discovery.
- Clamp cbm_sem_combined_score output to [0, 1]. The proximity multiplier
returns up to 1.10 as a same-file boost which could push the final
cosine score above 1.0.
- Short-circuit semantic scoring when MinHash jaccard >= 0.95. Exact
near-clones are already emitted as SIMILAR_TO edges; returning 0 here
avoids flooding SEMANTICALLY_RELATED with cross-service copy-paste
boilerplate and frees the edge budget for genuine vocabulary-bridged
relations.
- Validate search_graph semantic_query as an array of strings and return
a clear error for a single-string input. Update the tool description
to spell out the requirement explicitly with an example.
- JSON-escape user-controlled strings (callee names, call arguments,
URL paths, import local_name) in call/argument properties. Introduces
cbm_json_escape() in foundation/str_util.
- Skip SQLite pending_byte_page (file offset 0x40000000) during raw page
writes in sqlite_writer to avoid corrupting databases that cross the
1 GiB boundary.
- Migrate pretrained vector blob from UniXcoder (51K tokens) to
nomic-embed-code (40856 tokens x 768d int8). Includes the extraction
script under scripts/extract_nomic_vectors.py.
Four changes to reduce false positives from 382K to 58K SIMILAR_TO edges
on the Linux kernel while maintaining ~98% precision:
1. MIN_NODES raised to 30 (leaf tokens) — aligns with BigCloneBench
standard of 50 raw source tokens / 6 lines minimum clone size
2. collect_ast_tokens now records LEAF nodes only, not internal grammar
nodes. Leaf-only counting is language-agnostic: leaves correspond to
actual source tokens regardless of grammar verbosity (C tree-sitter
produces 2x more internal nodes than Python for the same logic)
3. hash_trigrams tracks unique structural trigram count via hash set.
Functions with < 32 unique structural trigrams are skipped — too few
distinct features for K=64 MinHash to produce reliable estimates
4. Structural-weighted MinHash via repetition: each trigram gets weight
0-3 based on count of non-normalised tokens (I/S/N/T = normalised).
Weight-0 trigrams (all-normalised, e.g. I|I|I) are pure noise and
skipped entirely. Weight 1-3 trigrams use repetition-based weighted
MinHash — higher weight = more hash attempts per seed = more likely
to dominate the signature. Achieves IDF-like effect without needing
corpus-wide statistics.
Also adds O(1) hash-set dedup in LSH query (replacing O(n) linear scan)
and bucket size cap of 200 to skip oversized buckets.
Linux kernel: 1:41, similarity 7.7s, 395K fp, 58K edges, ~98% precision.
Tests updated with larger Go functions to meet new thresholds.
Compute K=64 MinHash signatures from normalized AST node-type trigrams
during function extraction, then generate SIMILAR_TO edges via LSH
(b=32, r=2) for function pairs with Jaccard >= 0.95.
- src/simhash/minhash.{h,c}: MinHash compute, Jaccard, hex encode/decode,
LSH index with band hashing for O(n) candidate generation
- src/pipeline/pass_similarity.c: post-pass reads fingerprints from node
properties, builds LSH index, emits SIMILAR_TO edges with jaccard and
same_file metadata. Same-language only, max 10 edges per node.
- internal/cbm/cbm.h: fingerprint fields on CBMDefinition
- internal/cbm/extract_defs.c: compute_fingerprint() hook at 3 extraction
sites after complexity, skip functions with < 10 AST body nodes
- pass_definitions.c + pass_parallel.c: serialize fingerprint to "fp" hex
in properties_json for both sequential and parallel pipeline paths
- pipeline.c + pipeline_incremental.c: register pass_similarity in both
full and incremental post-pass lists
- tests/test_simhash.c: 28 tests across 4 suites (core, LSH, edge gen,
pipeline integration with generated Go project + incremental)
Martin Vogel