cbm_kind_in_set builds a thread-local cache of calloc'd symbol bitsets (ks_cache) that was never freed, so LeakSanitizer (Linux x64) reported the per-worker-thread caches as leaks once the previously-empty grammars gained real node-type sets. Add cbm_kind_in_set_free_cache() and call it at worker-thread teardown (extract_worker, beside cbm_slab_destroy_thread) and main-thread exit (test runner, beside sqlite3_shutdown).
Seven REG_TYPE registrations in the curated Java stdlib (GregorianCalendar, FileNotFoundException, Closeable, UnaryOperator, BinaryOperator, CompletableFuture, ReentrantLock) passed their parent list as an inline compound literal. A compound literal has block storage duration, and cbm_registry_add_type shallow-copies the embedded_types pointer, so the registry kept a dangling pointer into cbm_java_stdlib_register's stack once each statement's block ended. The inheritance walk later read rt->embedded_types[0] for such a type (e.g. java.io.Closeable on a closeable.close() dispatch) — an AddressSanitizer stack-use-after-scope. Hoist the seven lists to static arrays, matching the other parent lists in the same function.
The derive-method synthesis loop tested di_entry->methods[mi].short_name before the mi < 4 bound, so at mi==4 it read methods[4] — one past the 4-element array (UBSan: index 4 out of bounds for DeriveMethod[4]). Evaluate the mi < 4 bound first so the dereference is guarded.
Whitelist find_first_descendant_by_kind and find_first_descendant_of in recursion_whitelist.h (bounded AST-depth DFS, like the existing r_collect_imports entry), and narrow two cppcheck-flagged variable scopes in sqlite_writer.c: use w->project at its single use, and fclose the value-copied wc.fp after free(w) instead of a pre-captured fp.
The CI lint (clang-format-20) flagged formatting violations across these files (pre-existing drift plus the new extraction code). Reformat to satisfy the check; no logic changes.
msys2-clang on Windows lacks the GNU/BSD memmem extension, breaking the Kotlin LSP compile (kotlin_lsp.c called memmem at 4 sites). Add a portable hand-rolled cbm_memmem in helpers.c/.h and use it instead. kotlin_lsp.c also picks up clang-format-20 reflow.
Make the per-grammar regression suite fully green. (1) Add def/name extraction for 22 IDL/config/niche grammars that previously yielded only a Module node (agda, pony, move, cobol, janet, pine, smali, verilog, vhdl, systemverilog, protobuf, graphql, thrift, capnp, smithy, wit, prisma, cmake, puppet, tablegen, assembly, nasm): corrected function/class node-type sets in lang_specs.c and per-grammar name resolution in extract_defs.c (these grammars define no tree-sitter 'name' field, so names are read from typed child nodes; a depth-bounded descendant search handles the FIELD_COUNT-0 (System)Verilog wrappers). (2) Resolve same-file CALLS for groovy, solidity, and ada (callee handlers in extract_calls.c plus descending into Ada subprograms). Golden label histograms and fixtures are updated to the real new extraction output (min_defs raised, never lowered).
Add per-language callee extraction so a same-file caller->callee resolves to a CALLS edge for the lisp dialects (Clojure, Common Lisp, Scheme, Fennel, Racket, Emacs Lisp — head symbol of a list/list_lit), F# (application_expression long_identifier_or_op head), PowerShell (command_name child of a command), WGSL (identifier under type_constructor_or_function_call_expression), and Dart (identifier sibling of the invocation selector); and accept ReScript's value_identifier function field in the shared field resolver. Reduces contract_calls_breadth from 14 to 3 (groovy, solidity, ada still need dedicated branches).
parse_generic_imports only scans direct children of the root, which missed three grammars: Dart wraps imports in import_or_export (the old dispatch matched the never-emitted import_declaration), Haskell nests import nodes under an imports container, and Zig's @import is a builtin_function nested inside a variable_declaration. Add dedicated parsers (dart finds the nested string_literal URI, haskell descends the imports container reusing the module field, zig DFS-finds @import/@cImport builtin_functions) so all three extract their imports.
collect_bases_from_field ignored the bare identifier (and dotted attribute) node that tree-sitter-python uses for a base class inside the superclasses argument_list, so it fell through to the raw-text fallback and captured "(Base)" with parentheses. That never resolved, leaving every Python subclass with zero INHERITS edges. Accept identifier/attribute children so the bare base name is extracted. Turns python_class_base_extracted_bare green.
Several grammars (clojure/racket/scheme, ada/pascal/fortran/fsharp, cairo/d/odin/squirrel/rescript, and the hlsl/ispc/slang C-family) extracted only a Module node. Add per-grammar def-name resolution in extract_defs.c with matching lang_specs entries so their functions reach the graph, and descend Kotlin import_list/import_header in extract_imports.c so Kotlin imports are captured. The vendored grammar MANIFEST documents each custom case.
Split cbm_write_db's monolithic build into an incremental API:
cbm_writer_open / cbm_writer_append_nodes / cbm_writer_finalize. Node
rows are written to the table B-tree across append calls via a
persistent page builder, so callers can flush and free node batches
(including the heavy properties column) without materializing every
record at once. Everything after the nodes table — edges, vectors,
metadata, indexes, and the sqlite_master page — is emitted at finalize,
as before.
cbm_write_db is now a one-shot wrapper (open, append all nodes,
finalize) producing byte-identical output; the full test suite and
PRAGMA integrity_check confirm the on-disk format is unchanged.
state_deserialize() dereferenced &buffer[0] and passed it to
indent_vec_deserialize()'s nonnull argument even when buffer was NULL /
buffer_len was 0 (the initial empty-state deserialize), tripping UBSan. Return
early after clearing state when there is nothing to deserialize.
Hybrid tree-sitter + type-resolution call resolver for Rust, alongside the
Go/C/PHP/C#/Python/TypeScript/Java/Kotlin resolvers. Resolves method dispatch
on typed receivers, UFCS and associated-function calls, trait/impl methods,
generics, closures, the `?` operator, and iterator adapters; seeds the standard
library and well-known crates, parses Cargo.toml for dependency context, and
expands a curated set of derive/proc-macro generated methods.
Includes rust_cargo (manifest parsing), rust_proc_macros (derive expansion),
rust_rustdoc (doc extraction), a generated stdlib + crates seed, and a large
test suite (registered as suite rust_lsp). Wired into the per-file dispatch.
Field-access chains through Self-returning calls remain a documented coverage
gap. Full suite green (4666 tests).
Hybrid tree-sitter + type-resolution call resolver for Kotlin, matching the
existing Go/C/PHP/C#/Python/TypeScript/Java resolvers. Resolves intra-file
and imported calls, method dispatch on typed receivers, extension functions,
operator conventions (plus/compareTo/contains/get/unary), scope functions and
trailing lambdas (it/this), smart-casts (is / when), super dispatch,
constructor-val properties, and stdlib defaults.
Targets the refreshed tree-sitter-kotlin grammar: discovers declarations via
simple_identifier/type_identifier, walks the statements wrapper, extracts
method names from navigation_suffix, descends call_suffix for trailing
lambdas, reads delegation_specifier for inheritance, class_parameter for
constructor fields, and import_list for imports.
Wires the resolver into the per-file dispatch and registers its test suite
(78 tests). Full suite green (4160 tests).
Update 128 vendored grammars to the commits pinned by nvim-treesitter and
Helix (cross-verified canonical sources), normalize their tree_sitter
includes (angle-bracket -> quoted) and external-scanner deserialize
signatures, and restore missing upstream LICENSE files.
Add internal/cbm/vendored/grammars/MANIFEST.md as a provenance record:
upstream repo, pinned commit, current/latest ABI, and license status for
all 157 grammars. 12 first-party (self-maintained) grammars are marked and
excluded from upstream refresh; vhdl and fennel are kept at their prior
revisions (scanner ABI / added-scanner incompatibilities).
Adapt the shared extraction layer to node-shape changes in the refreshed
grammars: resolve declaration names from simple_identifier/type_identifier
children when the grammar drops the `name` field (kotlin, swift), and
recognize multiline_comment as a doc comment.
Integrates the type-aware Java semantic resolver from
worktree-adding-java-lsp-support (based on 673ac4e): java_lsp.c + generated
java.lang/util stdlib registry, wired into the per-file LSP pass, with 94
java_lsp tests + ~170 java_lsp_coverage tests.
Wiring conflicts (the shared files diverged 146 commits on main) resolved by
keeping main's existing LSP registrations (go/c/php/py/ts/cs) and adding the
Java entries in lsp_all.c, cbm.c (cbm_run_java_lsp dispatch), test_main.c and
Makefile.cbm.
Adapted to the current unity-build: renamed java_lsp.c's static helpers
node_text/return_type_of to java_node_text/java_return_type_of so they no
longer collide with ts_lsp.c inside the shared lsp_all.c translation unit.
4082 tests pass (incl. java_lsp + java_lsp_coverage); ASan/UBSan clean.
Per-function complexity metadata is now stored on graph nodes and queryable,
alongside several indexing performance and correctness fixes developed and
validated together (3704 tests, ASan/UBSan clean).
Bottleneck metrics (query via query_graph):
- Tier A (in the extraction AST walk): cyclomatic (complexity), cognitive
(nesting-weighted), loop_count, loop_depth (max nested-loop depth),
param_count, max_access_depth.
- Tier B (new pre-dump pass, pass_complexity.c): transitive_loop_depth
propagated along CALLS edges + a recursive flag (direct self-recursion and
mutual-recursion cycles), plus the call-context signals linear_scan_in_loop,
alloc_in_loop, recursion_in_loop and unguarded_recursion.
- query_graph and get_architecture tool descriptions document the metrics and
the Leiden community clusters.
Cypher engine:
- node_prop exposes arbitrary persisted node properties to WHERE/RETURN.
- Fix projection aliasing: multi-property rows shared a single static buffer so
every column returned the last value read; now per-column/rotating buffers.
- Fix a stack-use-after-scope in aggregate RETURN (caller-owned value buffers).
Indexing performance:
- Gate C/C++ #define Macro-node extraction to full mode (it is ~49% of nodes on
the Linux kernel); moderate/fast skip it.
- Emit the complexity property block only for Function/Method nodes so the
millions of Macro/Field/Variable/Class/Enum nodes no longer carry zeroed
fields — large RAM reduction at scale.
- Classify node types via tree-sitter TSSymbol bitsets in cbm_kind_in_set
instead of per-node strcmp scans (thread-local cache, strcmp fallback;
behaviour-identical).
- Subsample frequent (Zipfian) tokens in the semantic co-occurrence finalize;
~14x faster finalize on the kernel, output unchanged.
- pass_lsp_cross: replace O(n^2) linear dedup with hash-set dedup.
Windows:
- Canonicalize drive-letter case during path normalization so "c:/repo" and
"C:/repo" derive the same project key and cache file (#394/#227/#367).
Tests: extraction, pipeline and cypher regressions covering all of the above.
Two more Helm semantic improvements for #338:
- Chart.yaml: parse the top-level name and the dependencies: list
(cbm_parse_helm_chart) and, in the k8s pass, emit a Chart node per chart plus
a DEPENDS_ON edge to a shared, per-project deduplicated Chart node for each
declared dependency. So "which charts depend on postgresql" is now traversable.
- values.yaml: extract only the top-level keys as structured Variables instead
of one node per nested leaf key — killing the node flood the issue described
(86 nodes for a typical values.yaml down to a handful).
Generic-YAML one-node-per-file dedup (repo-wide behavior change) is intentionally
left as a separate follow-up. Tests cover the dependency parser (with/without
deps) and the values.yaml top-level-only extraction.
Closes#338
Helm chart helpers (.tpl) and Go templates produced no semantic graph — just a
flat key flood. Wire the existing gotemplate grammar for real structure:
- .tpl files map to the gotemplate language (Helm _helpers.tpl).
- {{ define "chart.fullname" }} becomes a Function node named after the
template (the name is a string literal, not an identifier, so it's resolved
in a dedicated walker branch).
- {{ include "x" . }} and {{ template "x" . }} resolve to the referenced
template name instead of the bare `include` builtin, so they form CALLS
edges to the define'd Function. template_action is added to the call set.
Lets you trace which charts include which named templates. Tests cover define
-> Function and include/template -> CALLS, plus the .tpl mapping.
Refs #338
Vendor the MIT-licensed cfmleditor/tree-sitter-cfml grammars and wire both
CFML dialects through the pipeline:
- .cfc components -> cfscript grammar (CBM_LANG_CFSCRIPT): a JS-like grammar, so
function/method declarations, calls and imports extract with the shared JS/TS
arrays. Components' functions become Function nodes.
- .cfm templates -> cfml tag grammar (CBM_LANG_CFML): tag-based (HTML-derived).
Embedded <cfscript> functions extract via function_declaration; tag-based
<cffunction name="..."> nodes are handled by a dedicated walker branch that
reads the name attribute (cf_function_tag has no name field). Shared scanner
headers are vendored at vendored/common/ so the grammar's ../../common/
includes resolve without editing vendored sources.
Adds an ASCII case-insensitive compare for CFML's case-insensitive attributes.
Tests cover both dialects and the extension mapping. LICENSEs vendored.
Closes#38
Vendor the MIT-licensed tree-sitter-qmljs grammar (a TypeScript superset with
declarative ui_* nodes) and wire .qml files through the full pipeline: language
enum, extension + display-name mapping, grammar shim, and a language spec that
reuses the JS/TS function/call/branch arrays and adds QML class/field/import
types (ui_inline_component, ui_property, ui_signal, ui_import).
QML files now produce Function nodes (incl. JS functions inside QML objects),
class/interface/enum nodes, imports and CALLS edges, so call chains through a
Qt project's .qml files can be traced. LICENSE vendored alongside the grammar.
Closes#42
HCL/Terraform blocks were named by their bare keyword, so every resource
collapsed to a single "resource" node and blocks were indistinguishable in the
graph. Fold the quoted labels into the name via the block's string_lit ->
template_literal children, so resource "aws_instance" "web" becomes
resource.aws_instance.web (and variable.region, module.vpc, etc). Each block is
now a distinct, query-friendly Class node. The dedicated terraform parser
(cbm_parse_terraform_source) is unchanged.
Closes#337
GDScript (.gd), PowerShell (.ps1/.psm1/.psd1) and Luau (.luau) were wired into
the grammar table, lang specs and extension map but had no extraction coverage,
so the AST->graph visitor was never proven for them. Add extraction tests that
assert Function/Class nodes and CALLS are produced from representative source.
PowerShell's function_statement has no `name` field — the name is a
`function_name` child node — so resolve_func_name returned null and no Function
node was emitted. Resolve the name from that child, mirroring the Swift path.
Closes#186Closes#35Closes#39
#define directives were not represented in the graph. Object-like macros
(preproc_def) and function-like macros (preproc_function_def) are now emitted
as Macro nodes with a DEFINES edge from the file. A function-like macro keeps
its parameter list as the signature; the macro body is not descended into.
Applies to all C-preprocessor languages (C, C++, CUDA, GLSL, Objective-C, ISPC).
Closes#375
walk_defs stopped descending into a function body once the function itself
was extracted, so arrow methods of an object literal returned from a factory
(the Zustand actions-slice pattern) were never visited. Let JS/TS/TSX descend
into function bodies (as Wolfram already did) and resolve an arrow's name from
a 'pair' parent's key, so { addItem: (...) => {...} } is captured as a Function.
Anonymous nested arrows have no resolvable name and are skipped, so no noise.
Closes#341
R had no import extraction at all (not in the import dispatch), so library(),
require(), source() AND box::use() all produced zero IMPORTS edges; and
module\$fn() calls were dropped because the call-callee resolver did not handle
R extract_operator function nodes.
#218: add parse_r_imports — recursively scan for call nodes and emit IMPORTS for
library/require/requireNamespace/loadNamespace/source (first arg) and for each
box::use(mod[syms], pkg/path[syms], ...) argument (module path = the spec text
before the [symbols] list). AST node types confirmed via a tree-sitter-R probe.
#219: handle extract_operator (R \$) as a call function node in
extract_callee_from_fields → emit "module.fn" so module\$fn() yields a CALLS
edge like other member calls.
Tests cover box::use + library + source imports and the \$-call; r_collect_imports
added to the recursion whitelist.
- cbm_arena_alloc returns NULL on a NULL arena (both arena.c copies)
instead of dereferencing it — defense-in-depth against the NULL-arena
type-allocation path that could crash the LSP type layer.
- discover: add "vendored" to the always-skip directory list.
Distilled from #374, taking the parts not already on main. The PR's
ts_lsp tuple-arena allocation, the C/C++ template formal-count clamp,
and the type_rep unbound-param preservation it also carried all landed
independently in v0.7.0 / via #322 / #360, so only these two
defensive improvements remain. Relates to #390.
Root cause: unbound template parameters flowed through cbm_type_substitute
as NULL, and call-expression return handling then dereferenced the NULL
substituted type. Reproduced on MangoHud's src/hud_elements.cpp.
- type_rep: cbm_type_substitute now returns the original param `t` (not
NULL) when a type arg is unbound, in both the TYPE_PARAM and named-param
paths — unbound params stay intact instead of poisoning the type.
- c_lsp: guard `if (!ret) return cbm_type_unknown();` before unwrapping
the substituted return type; guard the substitute result before use; add
template_function call-node resolution (qualified/scoped/plain names).
- extract_defs: factor template-inner lookup into find_cpp_template_inner_node,
which recurses into nested template_declarations and also handles
field_declaration; reuse it from unwrap_template_inner.
- tests: clsp_nocrash_template_function_multi_param_nested_call (the nested
reproducer) and typerep_substitute_unbound_param_preserved.
Distilled from #360 onto current main. The formal-parameter-count clamp
that overlapped this PR already landed via #322, so only the additive
crash fixes are taken here. Contributes to the C++ stability cluster (#390;
relates to #215/#312).
The 8 channel-extraction DFS walkers (Python consts, Python, Go, Java,
C#, Ruby, Elixir, Rust) used fixed-size `TSNode stack[CHAN_STACK_CAP]`
arrays whose push loops guarded with `&& top < CHAN_STACK_CAP`. On
deeply-nested ASTs that guard silently dropped children, and an
unguarded overflow could corrupt adjacent stack memory — surfacing as a
segfault in a later pass (lsp_cross / semantic_edges) on large repos.
Convert all 8 to the arena-backed growable TSNodeStack
(ts_nstack_init/push/pop), matching the pattern already used in
extract_calls.c / extract_imports.c (PR #217). No silent drops, no
overflow.
Distilled from #339 (trailing-newline cleanup only; conversion is the
author's).
Adds a per-file expression-eval step budget (10k) to the C/C++ LSP resolver so pathological recovery-mode ASTs degrade to unknown instead of hanging indexing (the Bitcoin-derived C++ repro). Verified locally: build clean, all 3,617 tests pass including the full clsp_ suite.
Fixes the OOB read in c_resolve_pending_template_calls when a call site has more arguments than the parsed function signature (issue #215). Verified locally: build clean, all 3,617 tests pass including the new clsp_nocrash_template_extra_call_args regression.
Format drift had accumulated across recent commits (the dry-run lint enforces clang-format-20, which had not run on these commits yet). No functional changes — whitespace/layout only, verified by a clean cppcheck + clang-format + NOLINT lint-ci pass.
cppcheck (warning+style, error-exitcode=1): guard cbm_fqn_compute against a NULL project/rel_path (ctunullpointer reachable from cbm_extract_file), and narrow prefix_len/suffix_len (path_alias) and ft_count (pass_githistory) to the scopes that use them.
security-audit.sh: the mcp.c file-read count grew to 13 (search/ADR/Windows-support reads — all path-contained or transport reads, audited, no new exfiltration surface); bump the reviewed maximum and document the update-check + request-body reads. Allow the diagnostics.c atomic metrics dump (.tmp+rename) in the file-write scan, and allow-list the sqlite WAL-checkpoint doc URL referenced in a store.c comment.
The per-file and cross-file LSP type-aware call/usage resolution now runs in every index mode (full/moderate/fast) instead of being gated behind a separate CBM_MODE_ADVANCED. Remove the mode from both index-mode enums, the g_lsp_enabled toggle and its save/restore dance, and the per-mode gating in discover, pipeline and the incremental path.
Now that the LSP walkers are O(n), the callee-attribution accuracy (e.g. ~91k call overrides on roslyn, ~60k on kubernetes) is worth the latency in every mode. Fast mode still filters files and skips similarity/semantic edges, so it stays faster than full while now also benefiting from LSP resolution.
A 'mode':'advanced' request falls back to the full default — identical behaviour to the old advanced mode, so existing callers are unaffected. This also fixes the test_incremental.c failures, which were asserting LSP-resolved results that the gated incremental path never produced.
The advanced-mode LSP passes walked the AST with the same O(i) ts_node_child(node, i) idiom, so a wide root made each per-file pass quadratic — the TS fixture reallyLargeFile.ts took ~16000s. Add a shared cbm_lsp_collect_children() helper (lsp_node_iter.h) and convert the entry and recursive walkers across the TypeScript, C, C#, PHP and Go LSP back-ends to a single O(n) cursor pass. The file now indexes in ~28s; full advanced-mode runs: TS 100s, roslyn 46s, kubernetes 51s, postgres 8s, WordPress 7s.
Also fix two latent crashes exposed once the resolve phase actually ran: process_node indexed the NULL-terminated param_types[] by the call's argument count, reading past the terminator when a call passed more args than the callee declares; and return_type_of() built a tuple via cbm_type_tuple(NULL, ...), passing a NULL arena — it now threads ctx->arena.
This also includes the Tier 2 cross-LSP registry plumbing (shared read-only base registry with per-file overlay/fallback) the walkers build on.
All LSP unit suites pass: TS, C 737/737, C# 60/60, PHP 280/280, Go 55/55.
tree-sitter's ts_node_child(node, i) is O(i) — it walks the child iterator from the first child on every call — so the common 'for (i=0; i<count; i++) ts_node_child(node, i)' idiom is O(n^2) over a node's children. On a file whose top level is very wide (e.g. a 580k-line fixture parsed as hundreds of thousands of sibling comment nodes) this made extraction run for over an hour.
Add ts_nstack_push_children(), which enumerates a node's children in a single O(n) cursor pass, and use it in the whole-tree DFS walkers (calls, channels, env-accesses, imports, type-assigns). Convert the module-level variable walk to a cursor too, and add cbm_is_module_level_p() so callers that already hold the parent node skip the O(n) ts_node_parent() rescan.
Indexing microsoft/TypeScript drops from ~5100s to ~50s with identical graph output.
Same clean def-driven pattern as C/Go. Extracts cs_register_lsp_defs,
adds cbm_cs_build_cross_registry (filters CBM_LANG_CSHARP) +
cbm_run_cs_lsp_cross_with_registry. resolve_worker dispatches
CBM_LANG_CSHARP to the shared registry; pipeline.c builds
cross_registries.cs; pass_lsp_cross.h includes cs_lsp.h.
C# cross-LSP is def-driven (no per-file AST field collection), so the
shared registry yields identical resolution to the per-file build.
roslyn bench (17,916 C# files, ADVANCED): single shared registry
(lsp_cross_prepare=234ms, 260,814 defs, zero slow files), 91,089 LSP
overrides applied, 58.6s total. All 60 test_cs_lsp tests pass; serilog
sanity exact (2,627n/7,112e). No SEGV/ASan.
Brings C/C++/CUDA to Go's optimization level. Extracts the cross-LSP
registration loop into c_register_lsp_defs (shared by per-file and
Tier 2 paths), adds:
cbm_c_build_cross_registry — builds ONE project-wide registry from
all C-family defs (filters by lang) + both C and C++ stdlibs,
finalized, shared READ-ONLY across resolve workers.
cbm_run_c_lsp_cross_with_registry — per-file resolve using the
shared registry; no per-file registry rebuild.
C cross-LSP is DEF-DRIVEN (fields/returns/embeds come from the def
strings, no per-file AST field collection), so the shared registry
produces identical resolution to the per-file build — zero quality
loss. resolve_worker dispatches CBM_LANG_C/CPP/CUDA to the new path;
pipeline.c builds cross_registries.c; pass_lsp_cross.h includes c_lsp.h.
postgres bench (4,967 files, ADVANCED mode): 628 C files share one
pre-built registry (lsp_cross_prepare=17ms, zero slow files), 44 LSP
overrides, +9 edges vs FULL. All 200 test_c_lsp tests pass. No
SEGV/ASan. Total pipeline overhead vs FULL: +5.2%.
New CBM_MODE_ADVANCED=3. Both per-file LSP (type-aware resolution
during extract) and the cross-file LSP pass (during resolve) are now
OPT-IN: FULL and below run tree-sitter extraction + registry textual
resolution only; ADVANCED adds the full LSP layer.
Rationale: profiling showed LSP changes the kubernetes graph by only
~0.01% of edges (it mostly re-confirms textual resolution) while
costing significant memory + time. Making it opt-in keeps the default
FULL path lean (3.0 GB vs 5.6 GB peak RSS on K8s) without meaningfully
worse output, and lets users pay for LSP precision only when wanted.
Mechanism:
- cbm.c: g_lsp_enabled atomic toggle wraps the entire per-file LSP
block (Go/C/C++/PHP/Python/TS/JS/C#) + the C/C++ preprocessor-pass
LSP call. Default true so ad-hoc extraction / tests keep historical
behaviour. cbm_set_lsp_enabled / cbm_get_lsp_enabled in cbm.h.
- pipeline.c: sets the toggle (save→set→restore so it never leaks to
other callers) from p->mode before extraction; gates the cross-LSP
all_defs collection + registry build on advanced; gates the
sequential lsp_cross dispatch. Fixed the moderate_only comparison
(== FAST, not > MODERATE) so ADVANCED is not treated as lighter.
- pipeline_incremental.c: same toggle save/set/restore.
- discover.h/.c: added CBM_MODE_ADVANCED to discover's enum + all 4
filter comparisons treat ADVANCED like FULL (full discovery, no
fast-filtering). Without this, ADVANCED (numerically 3) was
mis-detected as fast-mode and dropped ~half the files.
- mcp.c: parse mode "advanced" + tool schema enum/description.
Dual-mode K8s bench: FULL 46.7s (lsp_overrides=0, 3.0 GB) vs
ADVANCED 48.8s (lsp_overrides=80,818, 5.6 GB). Graphs within 0.01%.
All 3,615 tests pass.
Replaces the prior Robin-Hood open-addressing implementation with
Verstable (https://github.com/JacksonAllan/Verstable), a 2024 state-
of-the-art C hash table. The public API in foundation/hash_table.h
is UNCHANGED — CBMHashTable is now an opaque type and all 14
production callers (graph_buffer, registry, pipeline passes,
service_patterns, watcher, semantic, etc.) get the new impl
transparently. str_intern.c keeps its specialised (ptr+len) hash —
different semantics, not a clean swap target.
Why: per-sub-phase profiling pinpointed cbm_pipeline_lsp_target_node
and cbm_registry_resolve as the dominant resolve_calls cost. Both
ultimately hit cbm_ht_get; both run millions of times on a kubernetes
index. Verstable's key innovations target exactly this:
* 4-bit hash-fragment metadata in a uint16_t sidecar array lets
chain walks skip most key comparisons WITHOUT touching the
bucket payload (cache wins on hot lookups).
* Quadratic probing with per-bucket displacement chains (not linear
Robin-Hood psl).
* Tombstone-free deletion that moves at most one existing key.
Vendored at internal/cbm/vendored/verstable/verstable.h (2069 lines,
84KB, MIT). Wrapped as a const char* → void* template inside
hash_table.c. The header struct is opaque (forward declaration only);
audited the entire tree — no caller accesses internals or takes
sizeof(CBMHashTable), so opacity is safe.
Test refactor: test_hash_table.c had two impl-specific tests:
* ht_robin_hood_bounded_psl — Robin-Hood specific; removed.
ht_stress_10000 covers the functional equivalent.
* ht_power_of_two_capacity — renamed ht_create_small_capacity,
replaced struct probes with a functional insert+lookup check.
ht_create_capacity_zero/one similarly de-internalised.
Benchmark on kubernetes (12 workers, 20,650 files):
- parallel_resolve wall: 36,376 -> 27,217 ms (-25.2%)
- pipeline.done total: 61,485 -> 51,745 ms (-15.8%)
- registry_build: ~1,000 -> 779 ms (faster — registry uses
cbm_ht_set heavily during build)
- All 3,615 tests pass; hash_table suite 100% green
- ASan/UBSan/SEGV: none
- Peak RSS: ~6.83 GB
- Graph: nodes 297,082 -> 297,102 (+20, 0.007%), edges 1,155,444 ->
1,154,025 (-1,419, -0.12%) — within tolerance; minor delta likely
from hash-iteration-order differences affecting tie-breaks in
resolution. Sanity (zod/serilog): nodes exact, edges within 0.1%.
Cumulative trajectory from broken-state baseline ~35 min on K8s ->
51.7s wall = ~40x total speedup.
cbm_service_pattern_match runs once per resolved CALL edge inside
emit_service_edge — that's 7 pattern lists × ~30 patterns × strstr
= ~210 strstrs per call. On kubernetes (~600k resolved CALL edges)
the SAME resolved QN ("fmt.Errorf", "context.Context.Done",
"errors.New", etc) repeats thousands of times across files. Caching
turns the linear pattern-list scan into one hash lookup after the
first miss.
API additions in service_patterns.h:
void cbm_service_pattern_cache_begin(void);
void cbm_service_pattern_cache_end(void);
Cache is TLS (no contention) scoped to the resolve_worker's lifetime
in the parallel_resolve phase. Capacity 8192 entries per worker —
covers typical project QN diversity without rehashing.
Encoding: enum value + 1 in the void* slot (0/NULL means "miss").
Two helpers svc_enum_to_ptr / svc_ptr_to_enum hide the trick.
Benchmark on kubernetes (12 workers, 20,650 files):
- parallel_resolve wall: 98,520 -> 90,104 ms (-8.5% vs Chunk 1
baseline, -12.8% vs original 103,281 baseline)
- resolve_calls CPU: 992,427 -> 885,663 ms (-10.8% vs Chunk 1,
-5.4% vs original — Chunk 1 added overhead that Chunk 2 recovers
AND surpasses)
- resolve_usages CPU: 11,977 -> 10,944 ms (-8.6%)
- total_loop CPU: 1,005,409 -> 897,561 ms (-10.7%)
- Tests: 3616/0
- Graph: 297,082 -> 297,080 nodes (-2), 1,153,482 -> 1,156,097 edges
(within 0.2%, semantic-preserving)
- zod/serilog sanity: edges within +/-3 of baseline
- Peak RSS: 7.46 -> 7.75 GB (+290 MB for 12 × 8192-entry caches)
Cumulative trajectory: original broken state ~35 min K8s ->
1.92 min wall (-49× total). The per-file is_import_reachable cache
(Chunk 1) + the per-worker service-pattern cache (Chunk 2) together
save ~13s of resolve wall time. Next chunks target the gbuf_insert_
edge hot path and remaining strstr scans.
KEY ARCHITECTURAL INSIGHT (credit: user observation):
The per-file LSP during extract ALREADY captures the inferred receiver
type QN + method name for unresolved cross-file calls — emit_unresolved_
call at go_lsp.c:1242-1245 stores callee_qn as '<FullReceiverTypeQN>.<
MethodName>' in result->resolved_calls flagged with strategy='lsp_
unresolved'. The cross-LSP step was REDUNDANTLY re-deriving this same
metadata via a second tree-sitter parse + AST walk.
Cross-LSP for Go is now METADATA-DRIVEN:
cbm_go_fast_resolve_qualified_calls (rewritten):
- Iterates result->resolved_calls where strategy=='lsp_unresolved'
- For each entry: try direct registry lookup of callee_qn;
if that misses, split prefix.suffix and resolve prefix via the
file's import map then lookup '<module_qn>.<suffix>' in registry
- Emits resolved entry on top (cbm_pipeline_find_lsp_resolution
picks highest confidence)
- NO tree-sitter parse, NO AST walk, NO scope tracking
resolve_worker (pass_parallel.c):
case CBM_LANG_GO now calls ONLY the metadata-driven fast path.
cbm_run_go_lsp_cross_with_registry (the slow AST-walk version) is
no longer invoked for Go — it would just re-derive identical
metadata via a second AST walk and arrive at the same answers.
Benchmark (kubernetes, full fresh index, 12 workers):
- parallel_resolve: 1.89 min (Tier 3 meta) vs 1.86 min (Tier 2) — parity
- lsp_cross.slow lines: 0 (no tree-sitter fallback runs)
- Nodes: 297,081 vs 297,082 baseline (-1)
- Edges: 1,153,163 vs 1,153,482 baseline (-319, within 0.03%)
- Tests: 3,616 / 0 failures
- zod/serilog sanity: nodes exact, edges +1-3 (within 0.03%)
The wall-time parity is informative: the AST re-walk wasn't the
dominant cost. Per-file budget (66ms across 20,650 files / 12 workers)
is now dominated by the 5 sequential resolve passes (resolve_file_
calls/usages/throws/rw/semantic) pushing ~10M total edges into the
per-worker gbuf with arena allocs + hash-bucket inserts. The Tier 3
work eliminates a genuine architectural redundancy (two walks of the
same AST producing the same metadata) and sets up Python/C/etc to
adopt the same metadata-driven pattern in follow-up commits — but
hitting the user-target <30s K8s requires attacking the gbuf hot
path, not the cross-LSP path.
The gopls 'package summary' pattern, applied properly: build the cross-
LSP CBMTypeRegistry ONCE per language at pipeline time (in a dedicated
cross_lsp_arena), share it READ-ONLY across all resolve workers via the
CBMCrossLspRegistries struct. Per-file work then becomes just tree-
sitter parse + AST walk + O(1) hash lookups — zero per-file registry
rebuilds for languages with a pre-built registry.
Each LSP gains two new public functions:
CBMTypeRegistry *cbm_<lang>_build_cross_registry(arena, defs, def_count)
— filters all_defs to its language, init + stdlib + registration
loop + cbm_registry_finalize, returns finalized registry.
void cbm_run_<lang>_lsp_cross_with_registry(arena, source, ...,
CBMTypeRegistry *reg, ...)
— skips the per-file build AND the per-file Phase 1b/1c AST scans
(those mutate the registry, which would race on a shared read-
only reg). Trade-off: a few file-local type aliases discovered
only by Phase 1b are missed; in practice the per-file extract
LSP already captured them via embedded_types/field_defs/etc.
Tested with parallel_python_lsp_override_cross_file_emits_lsp_
strategy_edges — fused + filtered + prebuilt path stays correct.
CBMLSPDef gains a CBMLanguage lang field set in pxc_build_lsp_def so
the per-language build functions can filter the global all_defs[].
cbm_parallel_resolve signature now takes CBMCrossLspRegistries* instead
of a single CBMTypeRegistry*. The resolve_worker dispatches:
cbm_pxc_registry_for_lang(lang) returns the right pre-built reg if any;
the worker then switch()es to the matching _with_registry variant.
Languages without a _with_registry variant fall through to the existing
filter + per-file build path.
Kubernetes (Go-heavy, 12k files, 139k defs) measured perf:
parallel_resolve: 5.51 min (strdup-removal baseline) -> 1.86 min
= 2.96x speedup
parallel.resolve.lsp_cross.slow lines: 1,258 -> 2
(-99.8%; the 2 remaining are Python files, not Go)
Peak RSS: 6.29 GB -> 6.23 GB (slight decrease — pre-built registry
shared, no per-file scratch)
All 3,616 tests pass; zod/serilog graphs exact match (per-file path
unaffected).
Cumulative trajectory from the original broken state on kubernetes:
35 min -> 19 min (finalize) -> 6.24 min (module-def filter) ->
5.51 min (strdup-removal) -> 1.86 min (Tier 2 Go) = 18.8x total.
Languages wired so far: Go, Python. C/C++, TS/JS, PHP, C# still use the
per-file filter+build path until their _with_registry variants land in
follow-up commits.
wyhash is a non-cryptographic hash function ~2x faster than FNV1A
for the long string keys used in CBMTypeRegistry's hash buckets
(qualified names average ~50 bytes). The header is unused for now
— prepared for a future Tier 3 symbol-interning pass that replaces
the QN-string-keyed hashtables with u32-keyed ones (rustc pattern)
where wyhash's lower collision rate matters more.
Source: https://github.com/wangyi-fudan/wyhash (Unlicense)
Vendored as a single header at internal/cbm/vendored/wyhash/wyhash.h.
Two perf changes across the 6 cross-LSP resolvers:
1. cbm_registry_finalize after the per-file registry build (all 6
LSPs). Builds the FNV1A hash buckets so lookups go through the
O(1) hashed path in type_registry.c:154. Previously this was
NEVER called anywhere in the codebase, so every method/type
lookup fell through to the O(N) linear scan — the root cause of
the 35-min kubernetes regression. Lookup count on k8s: ~50 calls
* 110k defs * 11k files = ~64B strcmps under the old fallback.
2. In go/c/py/php registration loops, replace cbm_arena_strdup(arena,
d->X) with d->X directly. The strings are borrowed from the
caller's all_defs[] arena which outlives the cbm_run_X_lsp_cross
call by the whole pipeline lifetime. Saves 5 wasted strdups per
def per file = ~6B mallocs avoided on kubernetes.
TS and CS use a different registration architecture (no inline
strdups in the body — uses a per-file ctx). Only the finalize change
applies there.
Kubernetes wall-time vs prior commit:
- After finalize alone: 35 min -> 19 min (~1.8x)
- After module-def filter (prior commit): -> 6.24 min (~5.6x)
- After strdup-removal (this commit): -> 5.51 min (~6.4x)
- Memory: 7.14 GB -> 6.29 GB peak RSS (-12%)
All 3616 tests pass, zod/serilog graph output exact match.
Previously cbm_pipeline_pass_lsp_cross ran as a separate sequential
pass that re-read every source file from disk and re-parsed every
tree-sitter tree on a single thread — a 50x regression on the
kubernetes Go monorepo (extract 10s, lsp_cross 9+ min when killed).
Three changes:
1. CBMFileResult gains const char *source / int source_len. The
parallel extract worker copies source bytes into result->arena
(per-file 100MB cap, project-wide 2GB atomic cap) so the fused
cross-LSP step can run without disk I/O. Source pointer is
borrowed by downstream readers; arena owns the storage.
2. Cross-LSP work moves into the parallel resolve worker. Each worker
processes its file with the existing parallel infrastructure
(worker pool + work stealing). Slab is reclaimed after each
per-file cross-LSP call to bound peak slab memory. The previously
separate lsp_cross pass entry function is kept for the sequential
pipeline path; the parallel path no longer invokes it.
3. CBMModuleDefIndex (gopls 'package summary' pattern) — an inverted
index module_qn -> list of def indices built once in O(D) before
resolve_worker fires. Per file, cbm_pxc_filter_defs_for_file slices
all_defs down to the file's own_module + imported modules only,
typically 50-100x smaller than the global ~110k defs on kubernetes.
Drops the per-file registry build from O(all_defs) to O(relevant).
Other plumbing: cbm_parallel_resolve signature gains all_defs,
def_count, def_modules, module_def_index params. pipeline.c builds
the index between registry_build and parallel_resolve and frees it
after. pipeline_incremental and tests/test_parallel updated to match.
Helpers in pass_lsp_cross.c exposed via the header: cbm_pxc_has_cross_lsp,
cbm_pxc_collect_all_defs, cbm_pxc_run_one(_ts), cbm_pxc_ts_modes,
cbm_pxc_build_module_def_index, cbm_pxc_filter_defs_for_file.
Kubernetes wall-time: 35 min -> 6.24 min (~5.6x) before the followup
perf commits in this series.
parse_ts_type_text in ts_lsp.c builds a 16-slot CBMType* members[] for
union (|) and intersection (&) types. The in-loop branch correctly
caps additions at mc<15, but the trailing post-loop append that
collects the final member after the last separator was unconditional,
and the subsequent members[mc] = NULL sentinel could then write to
members[16] — one past the array end.
Caught by UBSan on the zod codebase (a TS union with >=16 members
exists). Non-deterministic crash signature under pthread-parallel
extraction (12 workers): exit 134 / SIGABRT with no stack, because
libc's allocator detected the heap corruption and called abort().
Single-fixture-file test suite never exercised the parallel path or
a wide-enough union, so existing ASan/UBSan coverage missed it.
Fix: mirror the in-loop guard on the trailing append (cap at mc<15).
Add regression test tslsp_union_many_members_no_overflow that
constructs a 20-member string-literal union — extracting must not
crash. UBSan: clean. Full suite: 3616 passed, 0 failed.
Martin Vogel