Systematic audit of every grammar's parser.c against the current
lang_specs.c entries. For each language, identified all primary
extraction-relevant node types (function/class/call/import) present
in the grammar but missing from the spec.
Changes:
- 44 new node type arrays created for languages that had empty_types
where the grammar provides extractable constructs
- 121 existing arrays extended with additional node types
- Covers all 153 vendored grammars
Examples of additions:
- Go: import, import_declaration added to import_types
- Python: import, import_from_statement, import_statement added
- JavaScript: new_expression, call_expression added to call_types
- PHP: namespace_use_declaration added (was completely missing)
- Rust: macro_invocation added to call_types
- C/C++: struct_specifier, enum_specifier, union_specifier added
All 2797 tests pass with zero regressions.
Fill in function, class, field, call, import, branch, variable,
assignment, throw, and decorator node types for all new languages
based on per-grammar AST analysis. Previously all 89 new languages
had empty_types (module detection only, no extraction).
Programming languages with full specs (31):
Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim,
Crystal, Odin, Ada, Sway, Cairo, Move, Hare, Pony, AWK, Fish,
Zsh, Tcl, Fennel, ReScript, PureScript, Nickel, Teal, Luau,
Squirrel, ISPC, Starlark, Puppet, LLVM IR
IDL/config with type/field specs (12):
Prisma, Thrift, Cap'n Proto, Smithy, WIT, Bicep, WGSL, HLSL,
VHDL, SystemVerilog, Apex, Kconfig
Remaining languages have module_types only (config/data formats
where function/class extraction is not applicable).
Migrate the lang_specs[] array from implicit positional indexing to
C99 designated initializers ([CBM_LANG_X] = {...}), eliminating a
class of silent ordering bugs when adding new languages.
Changes:
- Add ts_factory function pointer to CBMLangSpec struct
- Convert all 66 entries to designated initializer syntax
- Add _Static_assert guard: array size must match CBM_LANG_COUNT
- Replace 140-line cbm_ts_language() switch with 3-line factory dispatch
- Net reduction: 127 lines removed
This is prerequisite infrastructure for the ~90 new language expansion.
Ports four targeted improvements from #162 (Koolerx):
- FTS5 BM25 ranked search via a new `query` parameter on search_graph.
Uses a contentless nodes_fts virtual table populated at dump time with
a cbm_camel_split() SQL function that splits camelCase identifiers
(updateCloudClient -> update Cloud Client) so tokenized word queries
match identifier internals. BM25 rank is boosted by node label
(Function/Method +10, Route +8, Class/Interface/Type/Enum +5) and
noise labels (File/Folder/Module/Variable) are filtered out. The new
path is opt-in via the query parameter; regex name_pattern and vector
semantic_query paths are untouched.
- Enable SQLITE_ENABLE_FTS5 in the vendored sqlite3 build flags.
- Rebuild nodes_fts after incremental reindex. The btree dump path
bypasses any FTS5 triggers, so the index is wiped and repopulated via
delete-all + INSERT .. SELECT after each incremental merge.
- Add Interface to the label filter on cbm_registry_add in both the
parallel and sequential definition passes, so C#/Java class
declarations like Foo : IBar can resolve IBar to an INHERITS edge
target during enrichment.
- Add an explicit C# base_list handler to extract_base_classes() that
iterates named children directly instead of returning the raw node
text. Without this the fallback path returned ": IExamService" (with
the leading separator) and registry lookup always failed, so C#
INHERITS edges to interfaces never resolved.
Co-Authored-By: Koolerx <Koolerx@users.noreply.github.com>
- Parallelize pass_similarity and pass_semantic_edges via worker pool with
thread-local edge buffers; sequential final merge since gbuf is not
thread-safe. Adds cbm_lsh_query_into() as a thread-safe variant with
caller-provided candidate buffer.
- Add activatable profiling subsystem (CBM_PROFILE=1 env or --profile flag)
for step-level timing of extract, resolve, corpus build, vector phases,
and sqlite dump. Zero overhead when disabled.
- Fix cbm_index_mode_t enum mismatch between pipeline.h (FULL=0, MODERATE=1,
FAST=2) and discover.h (FULL=0, FAST=1). mode=fast silently no-op'd
fast-discovery filtering because discover.c compared against the wrong
value. Linux kernel fast mode went 1:40 -> 3:11 as a result; now back to
1:40. Broaden the filter guard to mode != CBM_MODE_FULL so MODERATE and
FAST both get aggressive discovery.
- Clamp cbm_sem_combined_score output to [0, 1]. The proximity multiplier
returns up to 1.10 as a same-file boost which could push the final
cosine score above 1.0.
- Short-circuit semantic scoring when MinHash jaccard >= 0.95. Exact
near-clones are already emitted as SIMILAR_TO edges; returning 0 here
avoids flooding SEMANTICALLY_RELATED with cross-service copy-paste
boilerplate and frees the edge budget for genuine vocabulary-bridged
relations.
- Validate search_graph semantic_query as an array of strings and return
a clear error for a single-string input. Update the tool description
to spell out the requirement explicitly with an example.
- JSON-escape user-controlled strings (callee names, call arguments,
URL paths, import local_name) in call/argument properties. Introduces
cbm_json_escape() in foundation/str_util.
- Skip SQLite pending_byte_page (file offset 0x40000000) during raw page
writes in sqlite_writer to avoid corrupting databases that cross the
1 GiB boundary.
- Migrate pretrained vector blob from UniXcoder (51K tokens) to
nomic-embed-code (40856 tokens x 768d int8). Includes the extraction
script under scripts/extract_nomic_vectors.py.
Major quality improvement: replace random index vectors with pre-trained
Nomic nomic-embed-code token embeddings. Tokens like 'error' and 'exception' now start
with similar vectors (learned from millions of code repos) instead of
arbitrary random projections. Co-occurrence enrichment adds project-specific
context on top.
Architecture:
- vendored/nomic/code_vectors.bin: 37.7MB raw int8 vectors
- vendored/nomic/code_vectors_blob.S: assembler .incbin (instant build)
- vendored/nomic/code_vectors.h: extern declarations + pretrained_vec_at()
- vendored/nomic/code_tokens.h: 40856 token strings (575KB)
- semantic.c: cbm_sem_random_index() now looks up pretrained vectors first,
falls back to sparse random for unknown tokens
- CBM_SEM_DIM raised from 256 to 768 to match Nomic nomic-embed-code
Also: RRI (Reflective Random Indexing), code pattern vocabulary injection,
120+ abbreviation expansions, callee/caller/body token enrichment,
label filter (Function/Method/Class only) in vector search SQL.
Binary size: 136MB → 169MB (+33MB from embedded vectors).
Search quality: keyword queries return relevant error-handling functions.
Domain-specific keyword queries return the expected functions.
Vector search pipeline (end-to-end):
- store.c: cbm_cosine_i8 custom SQLite function (int8 dot product with
sqrt magnitude), registered at store open time
- store.c: cbm_store_vector_search() builds merged query vector from
keywords via base RI vectors (xxHash), int8-quantizes, runs SQL query
with JOIN nodes ON node_id = id for metadata
- mcp.c: search_graph gains semantic_query param (array of strings),
returns semantic_results alongside regular results
- sqlite_writer.c: fixed build_vector_record to include node_id in
record body (matches build_node_record pattern)
- mcp.c: use yyjson_mut_obj_add_strcpy (not add_str) to avoid
use-after-free when vector results are freed before JSON serialization
Quality note: base RI vectors at query time don't capture co-occurrence
enrichment — results are moderate quality. Storing enriched token vectors
would improve this significantly (future work).
Four changes to reduce false positives from 382K to 58K SIMILAR_TO edges
on the Linux kernel while maintaining ~98% precision:
1. MIN_NODES raised to 30 (leaf tokens) — aligns with BigCloneBench
standard of 50 raw source tokens / 6 lines minimum clone size
2. collect_ast_tokens now records LEAF nodes only, not internal grammar
nodes. Leaf-only counting is language-agnostic: leaves correspond to
actual source tokens regardless of grammar verbosity (C tree-sitter
produces 2x more internal nodes than Python for the same logic)
3. hash_trigrams tracks unique structural trigram count via hash set.
Functions with < 32 unique structural trigrams are skipped — too few
distinct features for K=64 MinHash to produce reliable estimates
4. Structural-weighted MinHash via repetition: each trigram gets weight
0-3 based on count of non-normalised tokens (I/S/N/T = normalised).
Weight-0 trigrams (all-normalised, e.g. I|I|I) are pure noise and
skipped entirely. Weight 1-3 trigrams use repetition-based weighted
MinHash — higher weight = more hash attempts per seed = more likely
to dominate the signature. Achieves IDF-like effect without needing
corpus-wide statistics.
Also adds O(1) hash-set dedup in LSH query (replacing O(n) linear scan)
and bucket size cap of 200 to skip oversized buckets.
Linux kernel: 1:41, similarity 7.7s, 395K fp, 58K edges, ~98% precision.
Tests updated with larger Go functions to meet new thresholds.
Compute K=64 MinHash signatures from normalized AST node-type trigrams
during function extraction, then generate SIMILAR_TO edges via LSH
(b=32, r=2) for function pairs with Jaccard >= 0.95.
- src/simhash/minhash.{h,c}: MinHash compute, Jaccard, hex encode/decode,
LSH index with band hashing for O(n) candidate generation
- src/pipeline/pass_similarity.c: post-pass reads fingerprints from node
properties, builds LSH index, emits SIMILAR_TO edges with jaccard and
same_file metadata. Same-language only, max 10 edges per node.
- internal/cbm/cbm.h: fingerprint fields on CBMDefinition
- internal/cbm/extract_defs.c: compute_fingerprint() hook at 3 extraction
sites after complexity, skip functions with < 10 AST body nodes
- pass_definitions.c + pass_parallel.c: serialize fingerprint to "fp" hex
in properties_json for both sequential and parallel pipeline paths
- pipeline.c + pipeline_incremental.c: register pass_similarity in both
full and incremental post-pass lists
- tests/test_simhash.c: 28 tests across 4 suites (core, LSH, edge gen,
pipeline integration with generated Go project + incremental)
Lower thresholds to best-practice values:
cognitive-complexity: 250 -> 25 (industry default)
statements: 400 -> 200
lines: 800 -> 400
Add NOLINTNEXTLINE to 168 functions that currently exceed these thresholds.
Each suppression will be removed as the function is split in subsequent
commits. Zero net behavioral changes.
Also: extract execute_with_clause from cypher execute_single.
Prefix-to-decorator bridge now matches by file path segments: prefix
"/bq/bulk/ranked" only connects to handlers in paths containing
"bq/bulk/ranked", not all handlers in the service.
HCL infra binding extraction: scan resource blocks for attribute pairs
with source keys (topic, queue_name) and target keys (uri, push_endpoint).
Handles nested blocks (push_config { push_endpoint = "..." }).
Extract route_path and route_method directly from decorator AST nodes
during tree-sitter walk. Route + HANDLES edges created during
definition node creation. ensure_decorator_routes pass recreates
Route+HANDLES for existing nodes in incremental mode.
Prefix Routes bridged to handler Functions by service directory matching.
Route registration detection now works for unresolved local variables
(app.include_router, app.mount) by matching callee name suffix when
QN resolution fails. Scans ALL arguments for URL paths — handles
prefix=, path=, route= keyword args across frameworks.
New route_reg_suffixes: include_router, mount, add_url_rule,
register_blueprint, use, register, add_route, add_api_route.
Infrastructure bindings: YAML/JSON extraction captures topic→URL
pairs from subscription configs (push_endpoint, uri). Creates
INFRA_MAPS edges linking async topic Routes to endpoint URLs.
Improved match_infra_routes: matches prefix Routes from include_router,
supports service-only matching for generic "/" handlers, correct
Cloud Run hostname parsing for new project-number format.
sqlite_writer.c pb_build_interior: final root-level sep_cell
allocations were not freed. Each cbm_write_db call leaked 84 bytes.
Detected by ASan on Linux CI with 155 incremental test iterations.
Also fix CI timing: use MAX_TOOL_MS (15s) for perf assertions,
skip timing on full re-index operations (CI runners are slower).
- Serialize return_type, parent_class, base_classes on definition nodes
- DATA_FLOWS edges link callers through Route nodes to handlers
- YAML nested field extraction with dotted key paths (key_path property)
- Rename trace_call_path → trace_path with mode parameter:
calls (default), data_flow, cross_service
- Backward compatible: trace_call_path alias still accepted
- Updated CLI help text and all tests
CBMCallArg struct captures up to 8 arguments per call with expression
text, resolved constant values, keyword names, and positional index.
Member expressions produce dotted chain expressions automatically.
Serialized as "args" array on CALLS, HTTP_CALLS, ASYNC_CALLS, and
CONFIGURES edge properties. Enables data flow tracing across function
and service boundaries.
Function/Method nodes now include param_names and param_types arrays
in their properties JSON. Data was already extracted by tree-sitter
but not serialized.
- append_json_str_array helper replaces inline decorator serialization
- build_def_props in both pass_parallel.c and pass_definitions.c
- extract_param_names: add Python default_parameter, typed_default_parameter,
bare identifier node types
- Sample project: 114 Python functions now have param_names
- Zero extraction overhead, ~3MB more in properties for large projects
ts_node_child(root, i) and ts_node_next_sibling() both call
ts_node_child_with_descendant() internally, making sibling iteration
O(N²) total on roots with many children (e.g. generated TypeScript
.d.ts files or large Perl files with 5,000–50,000 top-level nodes).
Replace with TSTreeCursor: ts_tree_cursor_goto_next_sibling() maintains
traversal state across calls and is O(1) per step, reducing total
complexity from O(N²) to O(N).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- lang_specs.c: add LangSpec entries for CBM_LANG_KUSTOMIZE and CBM_LANG_K8S
(both reuse tree_sitter_yaml()); add cbm_ts_language() switch cases
- extract_k8s.c: new file implementing cbm_extract_k8s(); kustomize path
walks block_sequence items under resources/bases/patches/components/
patchesStrategicMerge and emits CBMImport per scalar; k8s path extracts
apiVersion/kind/metadata.name and emits CBMDefinition with label "Resource"
and name "Kind/metadata-name"; malformed manifests (missing kind or name)
produce zero definitions
- cbm.h: declare cbm_extract_k8s() alongside other sub-extractor entry points
- cbm.c: call cbm_extract_k8s() after unified extraction for the two new langs
- Makefile.cbm: add extract_k8s.c to EXTRACTION_SRCS
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add CBM_LANG_KUSTOMIZE and CBM_LANG_K8S to CBMLanguage enum (before CBM_LANG_COUNT)
- Add kustomization.yaml/yml to FILENAME_TABLE mapped to CBM_LANG_KUSTOMIZE
- Add Kustomize and Kubernetes entries to LANG_NAMES
- Implement cbm_is_kustomize_file() with to_lower+strcmp pattern
- Implement cbm_is_k8s_manifest() scanning first 4KB for apiVersion: via ci_strstr()
- Declare both helpers in pipeline_internal.h Infrascan helpers section
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Swift tree-sitter grammar uses positional children (not named fields) for
call_expression, so extract_callee_name() returned NULL for every call.
- extract_calls.c: Swift-specific callee extraction via first named child
for call_expression and constructor_expression nodes
- lang_specs.c: add constructor_expression to swift_call_types
- 4 new C tests: simple calls, method calls, constructors, chained calls
Co-Authored-By: Jay Sync <63868998+jay-sync@users.noreply.github.com>
- Arena-allocate pair_field_types in cpp_stdlib_data.c (was stack array
escaping scope via registry pointer — caught by GCC ASan)
- Add sysfs fallback for L2 cache on Linux ARM (sysconf returns 0)
- Expose cbm_node_free_fields() and fix 12+ leak sites in tests where
find_node_by_id/qn results were never freed (caught by LeakSanitizer)
Enforce WarningsAsErrors: '*' with zero suppressions of entire check categories.
All non-FP warnings fixed at source; true false positives get per-instance NOLINT
with reasoning. Fixes include: braces around statements (1679), isolate declarations
(75), explicit casts for multilevel pointer conversions, missing includes, function
parameter documentation via NOLINT, and proper NOLINTBEGIN/NOLINTEND nesting.
Delete Go MCP server, CGo bridges, Go pipeline, store, tools, watcher,
cypher engine, language specs, and all Go tests. Replace Go pre-commit
hook with C-only linting (clang-tidy, cppcheck, clang-format, tests).
Tree-sitter extraction engine (internal/cbm/*.c) retained for C pipeline.
Complete C implementation of the indexing pipeline (src/), parallel worker
pool, graph buffer with merge support, SQLite writer, and 1893-test suite.
Linter setup: zero warnings from clang-tidy (all checks enabled, no NOLINT
suppressions), cppcheck, and clang-format. All issues fixed at source —
proper headers for external linkage, GROW_ARRAY macro restructured to
eliminate type parameter, null-deref paths guarded, named intermediates
for suspicious-argument checks.
test