Ports four targeted improvements from #162 (Koolerx):
- FTS5 BM25 ranked search via a new `query` parameter on search_graph.
Uses a contentless nodes_fts virtual table populated at dump time with
a cbm_camel_split() SQL function that splits camelCase identifiers
(updateCloudClient -> update Cloud Client) so tokenized word queries
match identifier internals. BM25 rank is boosted by node label
(Function/Method +10, Route +8, Class/Interface/Type/Enum +5) and
noise labels (File/Folder/Module/Variable) are filtered out. The new
path is opt-in via the query parameter; regex name_pattern and vector
semantic_query paths are untouched.
- Enable SQLITE_ENABLE_FTS5 in the vendored sqlite3 build flags.
- Rebuild nodes_fts after incremental reindex. The btree dump path
bypasses any FTS5 triggers, so the index is wiped and repopulated via
delete-all + INSERT .. SELECT after each incremental merge.
- Add Interface to the label filter on cbm_registry_add in both the
parallel and sequential definition passes, so C#/Java class
declarations like Foo : IBar can resolve IBar to an INHERITS edge
target during enrichment.
- Add an explicit C# base_list handler to extract_base_classes() that
iterates named children directly instead of returning the raw node
text. Without this the fallback path returned ": IExamService" (with
the leading separator) and registry lookup always failed, so C#
INHERITS edges to interfaces never resolved.
Co-Authored-By: Koolerx <Koolerx@users.noreply.github.com>
- Parallelize pass_similarity and pass_semantic_edges via worker pool with
thread-local edge buffers; sequential final merge since gbuf is not
thread-safe. Adds cbm_lsh_query_into() as a thread-safe variant with
caller-provided candidate buffer.
- Add activatable profiling subsystem (CBM_PROFILE=1 env or --profile flag)
for step-level timing of extract, resolve, corpus build, vector phases,
and sqlite dump. Zero overhead when disabled.
- Fix cbm_index_mode_t enum mismatch between pipeline.h (FULL=0, MODERATE=1,
FAST=2) and discover.h (FULL=0, FAST=1). mode=fast silently no-op'd
fast-discovery filtering because discover.c compared against the wrong
value. Linux kernel fast mode went 1:40 -> 3:11 as a result; now back to
1:40. Broaden the filter guard to mode != CBM_MODE_FULL so MODERATE and
FAST both get aggressive discovery.
- Clamp cbm_sem_combined_score output to [0, 1]. The proximity multiplier
returns up to 1.10 as a same-file boost which could push the final
cosine score above 1.0.
- Short-circuit semantic scoring when MinHash jaccard >= 0.95. Exact
near-clones are already emitted as SIMILAR_TO edges; returning 0 here
avoids flooding SEMANTICALLY_RELATED with cross-service copy-paste
boilerplate and frees the edge budget for genuine vocabulary-bridged
relations.
- Validate search_graph semantic_query as an array of strings and return
a clear error for a single-string input. Update the tool description
to spell out the requirement explicitly with an example.
- JSON-escape user-controlled strings (callee names, call arguments,
URL paths, import local_name) in call/argument properties. Introduces
cbm_json_escape() in foundation/str_util.
- Skip SQLite pending_byte_page (file offset 0x40000000) during raw page
writes in sqlite_writer to avoid corrupting databases that cross the
1 GiB boundary.
- Migrate pretrained vector blob from UniXcoder (51K tokens) to
nomic-embed-code (40856 tokens x 768d int8). Includes the extraction
script under scripts/extract_nomic_vectors.py.
Major quality improvement: replace random index vectors with pre-trained
Nomic nomic-embed-code token embeddings. Tokens like 'error' and 'exception' now start
with similar vectors (learned from millions of code repos) instead of
arbitrary random projections. Co-occurrence enrichment adds project-specific
context on top.
Architecture:
- vendored/nomic/code_vectors.bin: 37.7MB raw int8 vectors
- vendored/nomic/code_vectors_blob.S: assembler .incbin (instant build)
- vendored/nomic/code_vectors.h: extern declarations + pretrained_vec_at()
- vendored/nomic/code_tokens.h: 40856 token strings (575KB)
- semantic.c: cbm_sem_random_index() now looks up pretrained vectors first,
falls back to sparse random for unknown tokens
- CBM_SEM_DIM raised from 256 to 768 to match Nomic nomic-embed-code
Also: RRI (Reflective Random Indexing), code pattern vocabulary injection,
120+ abbreviation expansions, callee/caller/body token enrichment,
label filter (Function/Method/Class only) in vector search SQL.
Binary size: 136MB → 169MB (+33MB from embedded vectors).
Search quality: keyword queries return relevant error-handling functions.
Domain-specific keyword queries return the expected functions.
Vector search pipeline (end-to-end):
- store.c: cbm_cosine_i8 custom SQLite function (int8 dot product with
sqrt magnitude), registered at store open time
- store.c: cbm_store_vector_search() builds merged query vector from
keywords via base RI vectors (xxHash), int8-quantizes, runs SQL query
with JOIN nodes ON node_id = id for metadata
- mcp.c: search_graph gains semantic_query param (array of strings),
returns semantic_results alongside regular results
- sqlite_writer.c: fixed build_vector_record to include node_id in
record body (matches build_node_record pattern)
- mcp.c: use yyjson_mut_obj_add_strcpy (not add_str) to avoid
use-after-free when vector results are freed before JSON serialization
Quality note: base RI vectors at query time don't capture co-occurrence
enrichment — results are moderate quality. Storing enriched token vectors
would improve this significantly (future work).
Four changes to reduce false positives from 382K to 58K SIMILAR_TO edges
on the Linux kernel while maintaining ~98% precision:
1. MIN_NODES raised to 30 (leaf tokens) — aligns with BigCloneBench
standard of 50 raw source tokens / 6 lines minimum clone size
2. collect_ast_tokens now records LEAF nodes only, not internal grammar
nodes. Leaf-only counting is language-agnostic: leaves correspond to
actual source tokens regardless of grammar verbosity (C tree-sitter
produces 2x more internal nodes than Python for the same logic)
3. hash_trigrams tracks unique structural trigram count via hash set.
Functions with < 32 unique structural trigrams are skipped — too few
distinct features for K=64 MinHash to produce reliable estimates
4. Structural-weighted MinHash via repetition: each trigram gets weight
0-3 based on count of non-normalised tokens (I/S/N/T = normalised).
Weight-0 trigrams (all-normalised, e.g. I|I|I) are pure noise and
skipped entirely. Weight 1-3 trigrams use repetition-based weighted
MinHash — higher weight = more hash attempts per seed = more likely
to dominate the signature. Achieves IDF-like effect without needing
corpus-wide statistics.
Also adds O(1) hash-set dedup in LSH query (replacing O(n) linear scan)
and bucket size cap of 200 to skip oversized buckets.
Linux kernel: 1:41, similarity 7.7s, 395K fp, 58K edges, ~98% precision.
Tests updated with larger Go functions to meet new thresholds.
Compute K=64 MinHash signatures from normalized AST node-type trigrams
during function extraction, then generate SIMILAR_TO edges via LSH
(b=32, r=2) for function pairs with Jaccard >= 0.95.
- src/simhash/minhash.{h,c}: MinHash compute, Jaccard, hex encode/decode,
LSH index with band hashing for O(n) candidate generation
- src/pipeline/pass_similarity.c: post-pass reads fingerprints from node
properties, builds LSH index, emits SIMILAR_TO edges with jaccard and
same_file metadata. Same-language only, max 10 edges per node.
- internal/cbm/cbm.h: fingerprint fields on CBMDefinition
- internal/cbm/extract_defs.c: compute_fingerprint() hook at 3 extraction
sites after complexity, skip functions with < 10 AST body nodes
- pass_definitions.c + pass_parallel.c: serialize fingerprint to "fp" hex
in properties_json for both sequential and parallel pipeline paths
- pipeline.c + pipeline_incremental.c: register pass_similarity in both
full and incremental post-pass lists
- tests/test_simhash.c: 28 tests across 4 suites (core, LSH, edge gen,
pipeline integration with generated Go project + incremental)
Lower thresholds to best-practice values:
cognitive-complexity: 250 -> 25 (industry default)
statements: 400 -> 200
lines: 800 -> 400
Add NOLINTNEXTLINE to 168 functions that currently exceed these thresholds.
Each suppression will be removed as the function is split in subsequent
commits. Zero net behavioral changes.
Also: extract execute_with_clause from cypher execute_single.
Prefix-to-decorator bridge now matches by file path segments: prefix
"/bq/bulk/ranked" only connects to handlers in paths containing
"bq/bulk/ranked", not all handlers in the service.
HCL infra binding extraction: scan resource blocks for attribute pairs
with source keys (topic, queue_name) and target keys (uri, push_endpoint).
Handles nested blocks (push_config { push_endpoint = "..." }).
Extract route_path and route_method directly from decorator AST nodes
during tree-sitter walk. Route + HANDLES edges created during
definition node creation. ensure_decorator_routes pass recreates
Route+HANDLES for existing nodes in incremental mode.
Prefix Routes bridged to handler Functions by service directory matching.
Route registration detection now works for unresolved local variables
(app.include_router, app.mount) by matching callee name suffix when
QN resolution fails. Scans ALL arguments for URL paths — handles
prefix=, path=, route= keyword args across frameworks.
New route_reg_suffixes: include_router, mount, add_url_rule,
register_blueprint, use, register, add_route, add_api_route.
Infrastructure bindings: YAML/JSON extraction captures topic→URL
pairs from subscription configs (push_endpoint, uri). Creates
INFRA_MAPS edges linking async topic Routes to endpoint URLs.
Improved match_infra_routes: matches prefix Routes from include_router,
supports service-only matching for generic "/" handlers, correct
Cloud Run hostname parsing for new project-number format.
sqlite_writer.c pb_build_interior: final root-level sep_cell
allocations were not freed. Each cbm_write_db call leaked 84 bytes.
Detected by ASan on Linux CI with 155 incremental test iterations.
Also fix CI timing: use MAX_TOOL_MS (15s) for perf assertions,
skip timing on full re-index operations (CI runners are slower).
- Serialize return_type, parent_class, base_classes on definition nodes
- DATA_FLOWS edges link callers through Route nodes to handlers
- YAML nested field extraction with dotted key paths (key_path property)
- Rename trace_call_path → trace_path with mode parameter:
calls (default), data_flow, cross_service
- Backward compatible: trace_call_path alias still accepted
- Updated CLI help text and all tests
CBMCallArg struct captures up to 8 arguments per call with expression
text, resolved constant values, keyword names, and positional index.
Member expressions produce dotted chain expressions automatically.
Serialized as "args" array on CALLS, HTTP_CALLS, ASYNC_CALLS, and
CONFIGURES edge properties. Enables data flow tracing across function
and service boundaries.
Function/Method nodes now include param_names and param_types arrays
in their properties JSON. Data was already extracted by tree-sitter
but not serialized.
- append_json_str_array helper replaces inline decorator serialization
- build_def_props in both pass_parallel.c and pass_definitions.c
- extract_param_names: add Python default_parameter, typed_default_parameter,
bare identifier node types
- Sample project: 114 Python functions now have param_names
- Zero extraction overhead, ~3MB more in properties for large projects
ts_node_child(root, i) and ts_node_next_sibling() both call
ts_node_child_with_descendant() internally, making sibling iteration
O(N²) total on roots with many children (e.g. generated TypeScript
.d.ts files or large Perl files with 5,000–50,000 top-level nodes).
Replace with TSTreeCursor: ts_tree_cursor_goto_next_sibling() maintains
traversal state across calls and is O(1) per step, reducing total
complexity from O(N²) to O(N).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- lang_specs.c: add LangSpec entries for CBM_LANG_KUSTOMIZE and CBM_LANG_K8S
(both reuse tree_sitter_yaml()); add cbm_ts_language() switch cases
- extract_k8s.c: new file implementing cbm_extract_k8s(); kustomize path
walks block_sequence items under resources/bases/patches/components/
patchesStrategicMerge and emits CBMImport per scalar; k8s path extracts
apiVersion/kind/metadata.name and emits CBMDefinition with label "Resource"
and name "Kind/metadata-name"; malformed manifests (missing kind or name)
produce zero definitions
- cbm.h: declare cbm_extract_k8s() alongside other sub-extractor entry points
- cbm.c: call cbm_extract_k8s() after unified extraction for the two new langs
- Makefile.cbm: add extract_k8s.c to EXTRACTION_SRCS
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add CBM_LANG_KUSTOMIZE and CBM_LANG_K8S to CBMLanguage enum (before CBM_LANG_COUNT)
- Add kustomization.yaml/yml to FILENAME_TABLE mapped to CBM_LANG_KUSTOMIZE
- Add Kustomize and Kubernetes entries to LANG_NAMES
- Implement cbm_is_kustomize_file() with to_lower+strcmp pattern
- Implement cbm_is_k8s_manifest() scanning first 4KB for apiVersion: via ci_strstr()
- Declare both helpers in pipeline_internal.h Infrascan helpers section
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Swift tree-sitter grammar uses positional children (not named fields) for
call_expression, so extract_callee_name() returned NULL for every call.
- extract_calls.c: Swift-specific callee extraction via first named child
for call_expression and constructor_expression nodes
- lang_specs.c: add constructor_expression to swift_call_types
- 4 new C tests: simple calls, method calls, constructors, chained calls
Co-Authored-By: Jay Sync <63868998+jay-sync@users.noreply.github.com>
- Arena-allocate pair_field_types in cpp_stdlib_data.c (was stack array
escaping scope via registry pointer — caught by GCC ASan)
- Add sysfs fallback for L2 cache on Linux ARM (sysconf returns 0)
- Expose cbm_node_free_fields() and fix 12+ leak sites in tests where
find_node_by_id/qn results were never freed (caught by LeakSanitizer)
Enforce WarningsAsErrors: '*' with zero suppressions of entire check categories.
All non-FP warnings fixed at source; true false positives get per-instance NOLINT
with reasoning. Fixes include: braces around statements (1679), isolate declarations
(75), explicit casts for multilevel pointer conversions, missing includes, function
parameter documentation via NOLINT, and proper NOLINTBEGIN/NOLINTEND nesting.
Delete Go MCP server, CGo bridges, Go pipeline, store, tools, watcher,
cypher engine, language specs, and all Go tests. Replace Go pre-commit
hook with C-only linting (clang-tidy, cppcheck, clang-format, tests).
Tree-sitter extraction engine (internal/cbm/*.c) retained for C pipeline.
Complete C implementation of the indexing pipeline (src/), parallel worker
pool, graph buffer with merge support, SQLite writer, and 1893-test suite.
Linter setup: zero warnings from clang-tidy (all checks enabled, no NOLINT
suppressions), cppcheck, and clang-format. All issues fixed at source —
proper headers for external linkage, GROW_ARRAY macro restructured to
eliminate type parameter, null-deref paths guarded, named intermediates
for suspicious-argument checks.
Full indexing now runs entirely in RAM:
- One filepath.Walk, one bulk file read (LZ4 HC compressed sourceStore)
- In-memory SQLite for all passes, single Backup API dump at end
- Source access via C LZ4 decompression (~3μs per file, 5.7 GB/s)
Aho-Corasick engine (internal/cbm/ac.c):
- Custom ACISM automaton with bitmask output (≤64 patterns)
- Fused LZ4 decompress + AC scan in C-side thread-local buffer
- Batch API: single CGo call scans all files (ScanLZ4Batch)
- Compact alphabet mode for configlinker (38-char table)
Pipeline optimizations:
- httplinks: AC pre-screens files for HTTP client + route keywords
- configlinker: AC batch scan replaces O(n×m) substring matching
- Infrascan/envscan: iterate sourceStore keys instead of walking disk
- All source reads via getSource() (LZ4 decompress from RAM)
- File hashes pre-computed during bulk load (zero re-reads)
Type system improvements:
- Overload resolution by parameter type (score_overload_match with type compatibility)
- Lambda return type inference (trailing return type + shallow body walk)
- Inline namespace normalization (strip __1, __cxx11, __gnu_cxx)
- Implicit conversions in overload scoring (const char* → string, numeric promotions)
- Virtual dispatch: prefer derived override over base method
- CRTP detection: template param → derived class mapping
- Range-for iterator protocol: map element fix + begin()/operator*() fallback
- Default argument support via min_params range matching
- Free function auto deduction: identifier eval order fix (function before type)
- decltype() return type parsing in c_parse_return_type_text
- Trailing return type syncs both return_type and return_types array
Watcher no longer polls every indexed project via ListProjects(). Instead,
projects are added to an explicit watch list on index/auto-index and removed
on delete. Cross-project tool calls (search_graph, trace_call_path, etc.)
touch the watch list so referenced projects stay fresh.
