- branch_types: case_clause → switch_case + switch_default (correct
grammar node names)
- branch_types: add do_statement (do-while loops)
- import_types: remove 'require' (not a grammar node type)
Verified: TypeScript, Python, C, C++ specs are all correct — every
node type in the spec exists in the grammar's ts_symbol_names.
- C#: add record_declaration, type_declaration to class types;
add namespace_declaration to import types
- JavaScript: add new_expression to call types; add import, extends,
require to import types (also used by TS/TSX)
- Python: add import to import_types
- TypeScript: add function_signature to func types
- Makefile: add recipe to func, call to calls, include to imports
All 153 grammars now have maximally complete extraction specs.
2797 tests pass with zero regressions.
- pass_parallel.c: detect gRPC calls when resolved QN contains
ServiceClient/ServiceGrpc/Servicer — emits GRPC_CALLS edges for
calls resolved to generated protobuf client interfaces
- pass_parallel.c: emit_grpc_edge falls back to resolved QN when
callee_name alone doesn't contain service.method pattern
- extract_calls.c: add resolve_chained_selector() for iterative
extraction of chained method calls (NewClient(conn).GetBar →
"NewClient.GetBar") without recursion
Known limitation: Go receiver-method chained calls like
pb.NewCartServiceClient(conn).GetCart() in rpc.go are not yet
extracted as CALLS edges — the Go grammar's AST structure for
these patterns needs dedicated investigation.
Systematic audit of every grammar's parser.c against the current
lang_specs.c entries. For each language, identified all primary
extraction-relevant node types (function/class/call/import) present
in the grammar but missing from the spec.
Changes:
- 44 new node type arrays created for languages that had empty_types
where the grammar provides extractable constructs
- 121 existing arrays extended with additional node types
- Covers all 153 vendored grammars
Examples of additions:
- Go: import, import_declaration added to import_types
- Python: import, import_from_statement, import_statement added
- JavaScript: new_expression, call_expression added to call_types
- PHP: namespace_use_declaration added (was completely missing)
- Rust: macro_invocation added to call_types
- C/C++: struct_specifier, enum_specifier, union_specifier added
All 2797 tests pass with zero regressions.
When a single record payload exceeds 65KB (max_local=65501), the writer
now splits it across overflow pages per SQLite's B-tree spec instead of
underflowing the content_offset and corrupting memory.
Closes#139, closes#187.
Based on #175 by dLo999.
Co-Authored-By: Dustin Obrecht <dustin@kurtnoble.com>
Add .codebase-memory/graph.db.zst — a zstd-compressed knowledge graph
artifact that can be committed to the repo. Teammates bootstrap from
the artifact instead of running a full reindex from scratch.
- Vendor zstd 1.5.7 (amalgamated build) for 8-13:1 compression
- Two-tier export: zstd -9 + index stripping for explicit index,
zstd -3 for watcher/incremental auto-updates
- Import: decompress → integrity check → auto-recreate indexes
- Bootstrap in handle_index_repository: when no local DB exists but
artifact is present, import first then run incremental
- Auto-create .gitattributes with merge=ours to prevent conflicts
- Fix: add missing idx_edges_url_path to create_user_indexes and
url_path_gen generated column to init_schema
- 13 new tests (5 zstd wrapper + 8 artifact round-trip/edge cases)
Pipeline: gRPC/GraphQL/tRPC client libs in service_patterns, edge emitters
in pass_parallel, __grpc__ Route nodes from .proto service defs in
pass_route_nodes, cross-repo matchers in pass_cross_repo. Also adds MQTT,
NATS, Dapr async patterns.
Protobuf extraction: service as Class, rpc as Function with rpc_name
fallback in func_name_node. Route nodes created by line-range matching
rpc Functions to enclosing service Classes.
Tested: Online Boutique protos produce 16 gRPC Route nodes.
- Enum member extraction: C#/Java/TS/C++ enum values as Variable nodes
- JS/TS destructured variable splitting: const { A, B, C } = require()
emits individual Variable nodes instead of one blob
- Cypher -- SQL-style single-line comments
- Cypher in_degree/out_degree virtual node properties (thread-safe
via binding store pointer, enables dead code detection via Cypher)
- Cypher regex matching in inline property filters
- Resolution tiebreaking: test/mock QN deprioritization and C# field
type hint resolution for obj.Method() disambiguation
Co-Authored-By: Tommy <Koolerx@users.noreply.github.com>
Fill in function, class, field, call, import, branch, variable,
assignment, throw, and decorator node types for all new languages
based on per-grammar AST analysis. Previously all 89 new languages
had empty_types (module detection only, no extraction).
Programming languages with full specs (31):
Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim,
Crystal, Odin, Ada, Sway, Cairo, Move, Hare, Pony, AWK, Fish,
Zsh, Tcl, Fennel, ReScript, PureScript, Nickel, Teal, Luau,
Squirrel, ISPC, Starlark, Puppet, LLVM IR
IDL/config with type/field specs (12):
Prisma, Thrift, Cap'n Proto, Smithy, WIT, Bicep, WGSL, HLSL,
VHDL, SystemVerilog, Apex, Kconfig
Remaining languages have module_types only (config/data formats
where function/class extraction is not applicable).
Migrate the lang_specs[] array from implicit positional indexing to
C99 designated initializers ([CBM_LANG_X] = {...}), eliminating a
class of silent ordering bugs when adding new languages.
Changes:
- Add ts_factory function pointer to CBMLangSpec struct
- Convert all 66 entries to designated initializer syntax
- Add _Static_assert guard: array size must match CBM_LANG_COUNT
- Replace 140-line cbm_ts_language() switch with 3-line factory dispatch
- Net reduction: 127 lines removed
This is prerequisite infrastructure for the ~90 new language expansion.
Ports four targeted improvements from #162 (Koolerx):
- FTS5 BM25 ranked search via a new `query` parameter on search_graph.
Uses a contentless nodes_fts virtual table populated at dump time with
a cbm_camel_split() SQL function that splits camelCase identifiers
(updateCloudClient -> update Cloud Client) so tokenized word queries
match identifier internals. BM25 rank is boosted by node label
(Function/Method +10, Route +8, Class/Interface/Type/Enum +5) and
noise labels (File/Folder/Module/Variable) are filtered out. The new
path is opt-in via the query parameter; regex name_pattern and vector
semantic_query paths are untouched.
- Enable SQLITE_ENABLE_FTS5 in the vendored sqlite3 build flags.
- Rebuild nodes_fts after incremental reindex. The btree dump path
bypasses any FTS5 triggers, so the index is wiped and repopulated via
delete-all + INSERT .. SELECT after each incremental merge.
- Add Interface to the label filter on cbm_registry_add in both the
parallel and sequential definition passes, so C#/Java class
declarations like Foo : IBar can resolve IBar to an INHERITS edge
target during enrichment.
- Add an explicit C# base_list handler to extract_base_classes() that
iterates named children directly instead of returning the raw node
text. Without this the fallback path returned ": IExamService" (with
the leading separator) and registry lookup always failed, so C#
INHERITS edges to interfaces never resolved.
Co-Authored-By: Koolerx <Koolerx@users.noreply.github.com>
- Parallelize pass_similarity and pass_semantic_edges via worker pool with
thread-local edge buffers; sequential final merge since gbuf is not
thread-safe. Adds cbm_lsh_query_into() as a thread-safe variant with
caller-provided candidate buffer.
- Add activatable profiling subsystem (CBM_PROFILE=1 env or --profile flag)
for step-level timing of extract, resolve, corpus build, vector phases,
and sqlite dump. Zero overhead when disabled.
- Fix cbm_index_mode_t enum mismatch between pipeline.h (FULL=0, MODERATE=1,
FAST=2) and discover.h (FULL=0, FAST=1). mode=fast silently no-op'd
fast-discovery filtering because discover.c compared against the wrong
value. Linux kernel fast mode went 1:40 -> 3:11 as a result; now back to
1:40. Broaden the filter guard to mode != CBM_MODE_FULL so MODERATE and
FAST both get aggressive discovery.
- Clamp cbm_sem_combined_score output to [0, 1]. The proximity multiplier
returns up to 1.10 as a same-file boost which could push the final
cosine score above 1.0.
- Short-circuit semantic scoring when MinHash jaccard >= 0.95. Exact
near-clones are already emitted as SIMILAR_TO edges; returning 0 here
avoids flooding SEMANTICALLY_RELATED with cross-service copy-paste
boilerplate and frees the edge budget for genuine vocabulary-bridged
relations.
- Validate search_graph semantic_query as an array of strings and return
a clear error for a single-string input. Update the tool description
to spell out the requirement explicitly with an example.
- JSON-escape user-controlled strings (callee names, call arguments,
URL paths, import local_name) in call/argument properties. Introduces
cbm_json_escape() in foundation/str_util.
- Skip SQLite pending_byte_page (file offset 0x40000000) during raw page
writes in sqlite_writer to avoid corrupting databases that cross the
1 GiB boundary.
- Migrate pretrained vector blob from UniXcoder (51K tokens) to
nomic-embed-code (40856 tokens x 768d int8). Includes the extraction
script under scripts/extract_nomic_vectors.py.
Major quality improvement: replace random index vectors with pre-trained
Nomic nomic-embed-code token embeddings. Tokens like 'error' and 'exception' now start
with similar vectors (learned from millions of code repos) instead of
arbitrary random projections. Co-occurrence enrichment adds project-specific
context on top.
Architecture:
- vendored/nomic/code_vectors.bin: 37.7MB raw int8 vectors
- vendored/nomic/code_vectors_blob.S: assembler .incbin (instant build)
- vendored/nomic/code_vectors.h: extern declarations + pretrained_vec_at()
- vendored/nomic/code_tokens.h: 40856 token strings (575KB)
- semantic.c: cbm_sem_random_index() now looks up pretrained vectors first,
falls back to sparse random for unknown tokens
- CBM_SEM_DIM raised from 256 to 768 to match Nomic nomic-embed-code
Also: RRI (Reflective Random Indexing), code pattern vocabulary injection,
120+ abbreviation expansions, callee/caller/body token enrichment,
label filter (Function/Method/Class only) in vector search SQL.
Binary size: 136MB → 169MB (+33MB from embedded vectors).
Search quality: keyword queries return relevant error-handling functions.
Domain-specific keyword queries return the expected functions.
Vector search pipeline (end-to-end):
- store.c: cbm_cosine_i8 custom SQLite function (int8 dot product with
sqrt magnitude), registered at store open time
- store.c: cbm_store_vector_search() builds merged query vector from
keywords via base RI vectors (xxHash), int8-quantizes, runs SQL query
with JOIN nodes ON node_id = id for metadata
- mcp.c: search_graph gains semantic_query param (array of strings),
returns semantic_results alongside regular results
- sqlite_writer.c: fixed build_vector_record to include node_id in
record body (matches build_node_record pattern)
- mcp.c: use yyjson_mut_obj_add_strcpy (not add_str) to avoid
use-after-free when vector results are freed before JSON serialization
Quality note: base RI vectors at query time don't capture co-occurrence
enrichment — results are moderate quality. Storing enriched token vectors
would improve this significantly (future work).
Four changes to reduce false positives from 382K to 58K SIMILAR_TO edges
on the Linux kernel while maintaining ~98% precision:
1. MIN_NODES raised to 30 (leaf tokens) — aligns with BigCloneBench
standard of 50 raw source tokens / 6 lines minimum clone size
2. collect_ast_tokens now records LEAF nodes only, not internal grammar
nodes. Leaf-only counting is language-agnostic: leaves correspond to
actual source tokens regardless of grammar verbosity (C tree-sitter
produces 2x more internal nodes than Python for the same logic)
3. hash_trigrams tracks unique structural trigram count via hash set.
Functions with < 32 unique structural trigrams are skipped — too few
distinct features for K=64 MinHash to produce reliable estimates
4. Structural-weighted MinHash via repetition: each trigram gets weight
0-3 based on count of non-normalised tokens (I/S/N/T = normalised).
Weight-0 trigrams (all-normalised, e.g. I|I|I) are pure noise and
skipped entirely. Weight 1-3 trigrams use repetition-based weighted
MinHash — higher weight = more hash attempts per seed = more likely
to dominate the signature. Achieves IDF-like effect without needing
corpus-wide statistics.
Also adds O(1) hash-set dedup in LSH query (replacing O(n) linear scan)
and bucket size cap of 200 to skip oversized buckets.
Linux kernel: 1:41, similarity 7.7s, 395K fp, 58K edges, ~98% precision.
Tests updated with larger Go functions to meet new thresholds.
Compute K=64 MinHash signatures from normalized AST node-type trigrams
during function extraction, then generate SIMILAR_TO edges via LSH
(b=32, r=2) for function pairs with Jaccard >= 0.95.
- src/simhash/minhash.{h,c}: MinHash compute, Jaccard, hex encode/decode,
LSH index with band hashing for O(n) candidate generation
- src/pipeline/pass_similarity.c: post-pass reads fingerprints from node
properties, builds LSH index, emits SIMILAR_TO edges with jaccard and
same_file metadata. Same-language only, max 10 edges per node.
- internal/cbm/cbm.h: fingerprint fields on CBMDefinition
- internal/cbm/extract_defs.c: compute_fingerprint() hook at 3 extraction
sites after complexity, skip functions with < 10 AST body nodes
- pass_definitions.c + pass_parallel.c: serialize fingerprint to "fp" hex
in properties_json for both sequential and parallel pipeline paths
- pipeline.c + pipeline_incremental.c: register pass_similarity in both
full and incremental post-pass lists
- tests/test_simhash.c: 28 tests across 4 suites (core, LSH, edge gen,
pipeline integration with generated Go project + incremental)
Lower thresholds to best-practice values:
cognitive-complexity: 250 -> 25 (industry default)
statements: 400 -> 200
lines: 800 -> 400
Add NOLINTNEXTLINE to 168 functions that currently exceed these thresholds.
Each suppression will be removed as the function is split in subsequent
commits. Zero net behavioral changes.
Also: extract execute_with_clause from cypher execute_single.
Prefix-to-decorator bridge now matches by file path segments: prefix
"/bq/bulk/ranked" only connects to handlers in paths containing
"bq/bulk/ranked", not all handlers in the service.
HCL infra binding extraction: scan resource blocks for attribute pairs
with source keys (topic, queue_name) and target keys (uri, push_endpoint).
Handles nested blocks (push_config { push_endpoint = "..." }).
Extract route_path and route_method directly from decorator AST nodes
during tree-sitter walk. Route + HANDLES edges created during
definition node creation. ensure_decorator_routes pass recreates
Route+HANDLES for existing nodes in incremental mode.
Prefix Routes bridged to handler Functions by service directory matching.
Route registration detection now works for unresolved local variables
(app.include_router, app.mount) by matching callee name suffix when
QN resolution fails. Scans ALL arguments for URL paths — handles
prefix=, path=, route= keyword args across frameworks.
New route_reg_suffixes: include_router, mount, add_url_rule,
register_blueprint, use, register, add_route, add_api_route.
Infrastructure bindings: YAML/JSON extraction captures topic→URL
pairs from subscription configs (push_endpoint, uri). Creates
INFRA_MAPS edges linking async topic Routes to endpoint URLs.
Improved match_infra_routes: matches prefix Routes from include_router,
supports service-only matching for generic "/" handlers, correct
Cloud Run hostname parsing for new project-number format.
sqlite_writer.c pb_build_interior: final root-level sep_cell
allocations were not freed. Each cbm_write_db call leaked 84 bytes.
Detected by ASan on Linux CI with 155 incremental test iterations.
Also fix CI timing: use MAX_TOOL_MS (15s) for perf assertions,
skip timing on full re-index operations (CI runners are slower).
- Serialize return_type, parent_class, base_classes on definition nodes
- DATA_FLOWS edges link callers through Route nodes to handlers
- YAML nested field extraction with dotted key paths (key_path property)
- Rename trace_call_path → trace_path with mode parameter:
calls (default), data_flow, cross_service
- Backward compatible: trace_call_path alias still accepted
- Updated CLI help text and all tests
CBMCallArg struct captures up to 8 arguments per call with expression
text, resolved constant values, keyword names, and positional index.
Member expressions produce dotted chain expressions automatically.
Serialized as "args" array on CALLS, HTTP_CALLS, ASYNC_CALLS, and
CONFIGURES edge properties. Enables data flow tracing across function
and service boundaries.
Function/Method nodes now include param_names and param_types arrays
in their properties JSON. Data was already extracted by tree-sitter
but not serialized.
- append_json_str_array helper replaces inline decorator serialization
- build_def_props in both pass_parallel.c and pass_definitions.c
- extract_param_names: add Python default_parameter, typed_default_parameter,
bare identifier node types
- Sample project: 114 Python functions now have param_names
- Zero extraction overhead, ~3MB more in properties for large projects
ts_node_child(root, i) and ts_node_next_sibling() both call
ts_node_child_with_descendant() internally, making sibling iteration
O(N²) total on roots with many children (e.g. generated TypeScript
.d.ts files or large Perl files with 5,000–50,000 top-level nodes).
Replace with TSTreeCursor: ts_tree_cursor_goto_next_sibling() maintains
traversal state across calls and is O(1) per step, reducing total
complexity from O(N²) to O(N).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- lang_specs.c: add LangSpec entries for CBM_LANG_KUSTOMIZE and CBM_LANG_K8S
(both reuse tree_sitter_yaml()); add cbm_ts_language() switch cases
- extract_k8s.c: new file implementing cbm_extract_k8s(); kustomize path
walks block_sequence items under resources/bases/patches/components/
patchesStrategicMerge and emits CBMImport per scalar; k8s path extracts
apiVersion/kind/metadata.name and emits CBMDefinition with label "Resource"
and name "Kind/metadata-name"; malformed manifests (missing kind or name)
produce zero definitions
- cbm.h: declare cbm_extract_k8s() alongside other sub-extractor entry points
- cbm.c: call cbm_extract_k8s() after unified extraction for the two new langs
- Makefile.cbm: add extract_k8s.c to EXTRACTION_SRCS
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
test