Split cbm_write_db's monolithic build into an incremental API:
cbm_writer_open / cbm_writer_append_nodes / cbm_writer_finalize. Node
rows are written to the table B-tree across append calls via a
persistent page builder, so callers can flush and free node batches
(including the heavy properties column) without materializing every
record at once. Everything after the nodes table — edges, vectors,
metadata, indexes, and the sqlite_master page — is emitted at finalize,
as before.
cbm_write_db is now a one-shot wrapper (open, append all nodes,
finalize) producing byte-identical output; the full test suite and
PRAGMA integrity_check confirm the on-disk format is unchanged.
state_deserialize() dereferenced &buffer[0] and passed it to
indent_vec_deserialize()'s nonnull argument even when buffer was NULL /
buffer_len was 0 (the initial empty-state deserialize), tripping UBSan. Return
early after clearing state when there is nothing to deserialize.
Hybrid tree-sitter + type-resolution call resolver for Rust, alongside the
Go/C/PHP/C#/Python/TypeScript/Java/Kotlin resolvers. Resolves method dispatch
on typed receivers, UFCS and associated-function calls, trait/impl methods,
generics, closures, the `?` operator, and iterator adapters; seeds the standard
library and well-known crates, parses Cargo.toml for dependency context, and
expands a curated set of derive/proc-macro generated methods.
Includes rust_cargo (manifest parsing), rust_proc_macros (derive expansion),
rust_rustdoc (doc extraction), a generated stdlib + crates seed, and a large
test suite (registered as suite rust_lsp). Wired into the per-file dispatch.
Field-access chains through Self-returning calls remain a documented coverage
gap. Full suite green (4666 tests).
Hybrid tree-sitter + type-resolution call resolver for Kotlin, matching the
existing Go/C/PHP/C#/Python/TypeScript/Java resolvers. Resolves intra-file
and imported calls, method dispatch on typed receivers, extension functions,
operator conventions (plus/compareTo/contains/get/unary), scope functions and
trailing lambdas (it/this), smart-casts (is / when), super dispatch,
constructor-val properties, and stdlib defaults.
Targets the refreshed tree-sitter-kotlin grammar: discovers declarations via
simple_identifier/type_identifier, walks the statements wrapper, extracts
method names from navigation_suffix, descends call_suffix for trailing
lambdas, reads delegation_specifier for inheritance, class_parameter for
constructor fields, and import_list for imports.
Wires the resolver into the per-file dispatch and registers its test suite
(78 tests). Full suite green (4160 tests).
Update 128 vendored grammars to the commits pinned by nvim-treesitter and
Helix (cross-verified canonical sources), normalize their tree_sitter
includes (angle-bracket -> quoted) and external-scanner deserialize
signatures, and restore missing upstream LICENSE files.
Add internal/cbm/vendored/grammars/MANIFEST.md as a provenance record:
upstream repo, pinned commit, current/latest ABI, and license status for
all 157 grammars. 12 first-party (self-maintained) grammars are marked and
excluded from upstream refresh; vhdl and fennel are kept at their prior
revisions (scanner ABI / added-scanner incompatibilities).
Adapt the shared extraction layer to node-shape changes in the refreshed
grammars: resolve declaration names from simple_identifier/type_identifier
children when the grammar drops the `name` field (kotlin, swift), and
recognize multiline_comment as a doc comment.
Integrates the type-aware Java semantic resolver from
worktree-adding-java-lsp-support (based on 673ac4e): java_lsp.c + generated
java.lang/util stdlib registry, wired into the per-file LSP pass, with 94
java_lsp tests + ~170 java_lsp_coverage tests.
Wiring conflicts (the shared files diverged 146 commits on main) resolved by
keeping main's existing LSP registrations (go/c/php/py/ts/cs) and adding the
Java entries in lsp_all.c, cbm.c (cbm_run_java_lsp dispatch), test_main.c and
Makefile.cbm.
Adapted to the current unity-build: renamed java_lsp.c's static helpers
node_text/return_type_of to java_node_text/java_return_type_of so they no
longer collide with ts_lsp.c inside the shared lsp_all.c translation unit.
4082 tests pass (incl. java_lsp + java_lsp_coverage); ASan/UBSan clean.
Per-function complexity metadata is now stored on graph nodes and queryable,
alongside several indexing performance and correctness fixes developed and
validated together (3704 tests, ASan/UBSan clean).
Bottleneck metrics (query via query_graph):
- Tier A (in the extraction AST walk): cyclomatic (complexity), cognitive
(nesting-weighted), loop_count, loop_depth (max nested-loop depth),
param_count, max_access_depth.
- Tier B (new pre-dump pass, pass_complexity.c): transitive_loop_depth
propagated along CALLS edges + a recursive flag (direct self-recursion and
mutual-recursion cycles), plus the call-context signals linear_scan_in_loop,
alloc_in_loop, recursion_in_loop and unguarded_recursion.
- query_graph and get_architecture tool descriptions document the metrics and
the Leiden community clusters.
Cypher engine:
- node_prop exposes arbitrary persisted node properties to WHERE/RETURN.
- Fix projection aliasing: multi-property rows shared a single static buffer so
every column returned the last value read; now per-column/rotating buffers.
- Fix a stack-use-after-scope in aggregate RETURN (caller-owned value buffers).
Indexing performance:
- Gate C/C++ #define Macro-node extraction to full mode (it is ~49% of nodes on
the Linux kernel); moderate/fast skip it.
- Emit the complexity property block only for Function/Method nodes so the
millions of Macro/Field/Variable/Class/Enum nodes no longer carry zeroed
fields — large RAM reduction at scale.
- Classify node types via tree-sitter TSSymbol bitsets in cbm_kind_in_set
instead of per-node strcmp scans (thread-local cache, strcmp fallback;
behaviour-identical).
- Subsample frequent (Zipfian) tokens in the semantic co-occurrence finalize;
~14x faster finalize on the kernel, output unchanged.
- pass_lsp_cross: replace O(n^2) linear dedup with hash-set dedup.
Windows:
- Canonicalize drive-letter case during path normalization so "c:/repo" and
"C:/repo" derive the same project key and cache file (#394/#227/#367).
Tests: extraction, pipeline and cypher regressions covering all of the above.
Two more Helm semantic improvements for #338:
- Chart.yaml: parse the top-level name and the dependencies: list
(cbm_parse_helm_chart) and, in the k8s pass, emit a Chart node per chart plus
a DEPENDS_ON edge to a shared, per-project deduplicated Chart node for each
declared dependency. So "which charts depend on postgresql" is now traversable.
- values.yaml: extract only the top-level keys as structured Variables instead
of one node per nested leaf key — killing the node flood the issue described
(86 nodes for a typical values.yaml down to a handful).
Generic-YAML one-node-per-file dedup (repo-wide behavior change) is intentionally
left as a separate follow-up. Tests cover the dependency parser (with/without
deps) and the values.yaml top-level-only extraction.
Closes#338
Helm chart helpers (.tpl) and Go templates produced no semantic graph — just a
flat key flood. Wire the existing gotemplate grammar for real structure:
- .tpl files map to the gotemplate language (Helm _helpers.tpl).
- {{ define "chart.fullname" }} becomes a Function node named after the
template (the name is a string literal, not an identifier, so it's resolved
in a dedicated walker branch).
- {{ include "x" . }} and {{ template "x" . }} resolve to the referenced
template name instead of the bare `include` builtin, so they form CALLS
edges to the define'd Function. template_action is added to the call set.
Lets you trace which charts include which named templates. Tests cover define
-> Function and include/template -> CALLS, plus the .tpl mapping.
Refs #338
Vendor the MIT-licensed cfmleditor/tree-sitter-cfml grammars and wire both
CFML dialects through the pipeline:
- .cfc components -> cfscript grammar (CBM_LANG_CFSCRIPT): a JS-like grammar, so
function/method declarations, calls and imports extract with the shared JS/TS
arrays. Components' functions become Function nodes.
- .cfm templates -> cfml tag grammar (CBM_LANG_CFML): tag-based (HTML-derived).
Embedded <cfscript> functions extract via function_declaration; tag-based
<cffunction name="..."> nodes are handled by a dedicated walker branch that
reads the name attribute (cf_function_tag has no name field). Shared scanner
headers are vendored at vendored/common/ so the grammar's ../../common/
includes resolve without editing vendored sources.
Adds an ASCII case-insensitive compare for CFML's case-insensitive attributes.
Tests cover both dialects and the extension mapping. LICENSEs vendored.
Closes#38
Vendor the MIT-licensed tree-sitter-qmljs grammar (a TypeScript superset with
declarative ui_* nodes) and wire .qml files through the full pipeline: language
enum, extension + display-name mapping, grammar shim, and a language spec that
reuses the JS/TS function/call/branch arrays and adds QML class/field/import
types (ui_inline_component, ui_property, ui_signal, ui_import).
QML files now produce Function nodes (incl. JS functions inside QML objects),
class/interface/enum nodes, imports and CALLS edges, so call chains through a
Qt project's .qml files can be traced. LICENSE vendored alongside the grammar.
Closes#42
HCL/Terraform blocks were named by their bare keyword, so every resource
collapsed to a single "resource" node and blocks were indistinguishable in the
graph. Fold the quoted labels into the name via the block's string_lit ->
template_literal children, so resource "aws_instance" "web" becomes
resource.aws_instance.web (and variable.region, module.vpc, etc). Each block is
now a distinct, query-friendly Class node. The dedicated terraform parser
(cbm_parse_terraform_source) is unchanged.
Closes#337
GDScript (.gd), PowerShell (.ps1/.psm1/.psd1) and Luau (.luau) were wired into
the grammar table, lang specs and extension map but had no extraction coverage,
so the AST->graph visitor was never proven for them. Add extraction tests that
assert Function/Class nodes and CALLS are produced from representative source.
PowerShell's function_statement has no `name` field — the name is a
`function_name` child node — so resolve_func_name returned null and no Function
node was emitted. Resolve the name from that child, mirroring the Swift path.
Closes#186Closes#35Closes#39
#define directives were not represented in the graph. Object-like macros
(preproc_def) and function-like macros (preproc_function_def) are now emitted
as Macro nodes with a DEFINES edge from the file. A function-like macro keeps
its parameter list as the signature; the macro body is not descended into.
Applies to all C-preprocessor languages (C, C++, CUDA, GLSL, Objective-C, ISPC).
Closes#375
walk_defs stopped descending into a function body once the function itself
was extracted, so arrow methods of an object literal returned from a factory
(the Zustand actions-slice pattern) were never visited. Let JS/TS/TSX descend
into function bodies (as Wolfram already did) and resolve an arrow's name from
a 'pair' parent's key, so { addItem: (...) => {...} } is captured as a Function.
Anonymous nested arrows have no resolvable name and are skipped, so no noise.
Closes#341
R had no import extraction at all (not in the import dispatch), so library(),
require(), source() AND box::use() all produced zero IMPORTS edges; and
module\$fn() calls were dropped because the call-callee resolver did not handle
R extract_operator function nodes.
#218: add parse_r_imports — recursively scan for call nodes and emit IMPORTS for
library/require/requireNamespace/loadNamespace/source (first arg) and for each
box::use(mod[syms], pkg/path[syms], ...) argument (module path = the spec text
before the [symbols] list). AST node types confirmed via a tree-sitter-R probe.
#219: handle extract_operator (R \$) as a call function node in
extract_callee_from_fields → emit "module.fn" so module\$fn() yields a CALLS
edge like other member calls.
Tests cover box::use + library + source imports and the \$-call; r_collect_imports
added to the recursion whitelist.
- cbm_arena_alloc returns NULL on a NULL arena (both arena.c copies)
instead of dereferencing it — defense-in-depth against the NULL-arena
type-allocation path that could crash the LSP type layer.
- discover: add "vendored" to the always-skip directory list.
Distilled from #374, taking the parts not already on main. The PR's
ts_lsp tuple-arena allocation, the C/C++ template formal-count clamp,
and the type_rep unbound-param preservation it also carried all landed
independently in v0.7.0 / via #322 / #360, so only these two
defensive improvements remain. Relates to #390.
Root cause: unbound template parameters flowed through cbm_type_substitute
as NULL, and call-expression return handling then dereferenced the NULL
substituted type. Reproduced on MangoHud's src/hud_elements.cpp.
- type_rep: cbm_type_substitute now returns the original param `t` (not
NULL) when a type arg is unbound, in both the TYPE_PARAM and named-param
paths — unbound params stay intact instead of poisoning the type.
- c_lsp: guard `if (!ret) return cbm_type_unknown();` before unwrapping
the substituted return type; guard the substitute result before use; add
template_function call-node resolution (qualified/scoped/plain names).
- extract_defs: factor template-inner lookup into find_cpp_template_inner_node,
which recurses into nested template_declarations and also handles
field_declaration; reuse it from unwrap_template_inner.
- tests: clsp_nocrash_template_function_multi_param_nested_call (the nested
reproducer) and typerep_substitute_unbound_param_preserved.
Distilled from #360 onto current main. The formal-parameter-count clamp
that overlapped this PR already landed via #322, so only the additive
crash fixes are taken here. Contributes to the C++ stability cluster (#390;
relates to #215/#312).
The 8 channel-extraction DFS walkers (Python consts, Python, Go, Java,
C#, Ruby, Elixir, Rust) used fixed-size `TSNode stack[CHAN_STACK_CAP]`
arrays whose push loops guarded with `&& top < CHAN_STACK_CAP`. On
deeply-nested ASTs that guard silently dropped children, and an
unguarded overflow could corrupt adjacent stack memory — surfacing as a
segfault in a later pass (lsp_cross / semantic_edges) on large repos.
Convert all 8 to the arena-backed growable TSNodeStack
(ts_nstack_init/push/pop), matching the pattern already used in
extract_calls.c / extract_imports.c (PR #217). No silent drops, no
overflow.
Distilled from #339 (trailing-newline cleanup only; conversion is the
author's).
Adds a per-file expression-eval step budget (10k) to the C/C++ LSP resolver so pathological recovery-mode ASTs degrade to unknown instead of hanging indexing (the Bitcoin-derived C++ repro). Verified locally: build clean, all 3,617 tests pass including the full clsp_ suite.
Fixes the OOB read in c_resolve_pending_template_calls when a call site has more arguments than the parsed function signature (issue #215). Verified locally: build clean, all 3,617 tests pass including the new clsp_nocrash_template_extra_call_args regression.
Format drift had accumulated across recent commits (the dry-run lint enforces clang-format-20, which had not run on these commits yet). No functional changes — whitespace/layout only, verified by a clean cppcheck + clang-format + NOLINT lint-ci pass.
cppcheck (warning+style, error-exitcode=1): guard cbm_fqn_compute against a NULL project/rel_path (ctunullpointer reachable from cbm_extract_file), and narrow prefix_len/suffix_len (path_alias) and ft_count (pass_githistory) to the scopes that use them.
security-audit.sh: the mcp.c file-read count grew to 13 (search/ADR/Windows-support reads — all path-contained or transport reads, audited, no new exfiltration surface); bump the reviewed maximum and document the update-check + request-body reads. Allow the diagnostics.c atomic metrics dump (.tmp+rename) in the file-write scan, and allow-list the sqlite WAL-checkpoint doc URL referenced in a store.c comment.
The per-file and cross-file LSP type-aware call/usage resolution now runs in every index mode (full/moderate/fast) instead of being gated behind a separate CBM_MODE_ADVANCED. Remove the mode from both index-mode enums, the g_lsp_enabled toggle and its save/restore dance, and the per-mode gating in discover, pipeline and the incremental path.
Now that the LSP walkers are O(n), the callee-attribution accuracy (e.g. ~91k call overrides on roslyn, ~60k on kubernetes) is worth the latency in every mode. Fast mode still filters files and skips similarity/semantic edges, so it stays faster than full while now also benefiting from LSP resolution.
A 'mode':'advanced' request falls back to the full default — identical behaviour to the old advanced mode, so existing callers are unaffected. This also fixes the test_incremental.c failures, which were asserting LSP-resolved results that the gated incremental path never produced.
The advanced-mode LSP passes walked the AST with the same O(i) ts_node_child(node, i) idiom, so a wide root made each per-file pass quadratic — the TS fixture reallyLargeFile.ts took ~16000s. Add a shared cbm_lsp_collect_children() helper (lsp_node_iter.h) and convert the entry and recursive walkers across the TypeScript, C, C#, PHP and Go LSP back-ends to a single O(n) cursor pass. The file now indexes in ~28s; full advanced-mode runs: TS 100s, roslyn 46s, kubernetes 51s, postgres 8s, WordPress 7s.
Also fix two latent crashes exposed once the resolve phase actually ran: process_node indexed the NULL-terminated param_types[] by the call's argument count, reading past the terminator when a call passed more args than the callee declares; and return_type_of() built a tuple via cbm_type_tuple(NULL, ...), passing a NULL arena — it now threads ctx->arena.
This also includes the Tier 2 cross-LSP registry plumbing (shared read-only base registry with per-file overlay/fallback) the walkers build on.
All LSP unit suites pass: TS, C 737/737, C# 60/60, PHP 280/280, Go 55/55.
tree-sitter's ts_node_child(node, i) is O(i) — it walks the child iterator from the first child on every call — so the common 'for (i=0; i<count; i++) ts_node_child(node, i)' idiom is O(n^2) over a node's children. On a file whose top level is very wide (e.g. a 580k-line fixture parsed as hundreds of thousands of sibling comment nodes) this made extraction run for over an hour.
Add ts_nstack_push_children(), which enumerates a node's children in a single O(n) cursor pass, and use it in the whole-tree DFS walkers (calls, channels, env-accesses, imports, type-assigns). Convert the module-level variable walk to a cursor too, and add cbm_is_module_level_p() so callers that already hold the parent node skip the O(n) ts_node_parent() rescan.
Indexing microsoft/TypeScript drops from ~5100s to ~50s with identical graph output.
Same clean def-driven pattern as C/Go. Extracts cs_register_lsp_defs,
adds cbm_cs_build_cross_registry (filters CBM_LANG_CSHARP) +
cbm_run_cs_lsp_cross_with_registry. resolve_worker dispatches
CBM_LANG_CSHARP to the shared registry; pipeline.c builds
cross_registries.cs; pass_lsp_cross.h includes cs_lsp.h.
C# cross-LSP is def-driven (no per-file AST field collection), so the
shared registry yields identical resolution to the per-file build.
roslyn bench (17,916 C# files, ADVANCED): single shared registry
(lsp_cross_prepare=234ms, 260,814 defs, zero slow files), 91,089 LSP
overrides applied, 58.6s total. All 60 test_cs_lsp tests pass; serilog
sanity exact (2,627n/7,112e). No SEGV/ASan.
Brings C/C++/CUDA to Go's optimization level. Extracts the cross-LSP
registration loop into c_register_lsp_defs (shared by per-file and
Tier 2 paths), adds:
cbm_c_build_cross_registry — builds ONE project-wide registry from
all C-family defs (filters by lang) + both C and C++ stdlibs,
finalized, shared READ-ONLY across resolve workers.
cbm_run_c_lsp_cross_with_registry — per-file resolve using the
shared registry; no per-file registry rebuild.
C cross-LSP is DEF-DRIVEN (fields/returns/embeds come from the def
strings, no per-file AST field collection), so the shared registry
produces identical resolution to the per-file build — zero quality
loss. resolve_worker dispatches CBM_LANG_C/CPP/CUDA to the new path;
pipeline.c builds cross_registries.c; pass_lsp_cross.h includes c_lsp.h.
postgres bench (4,967 files, ADVANCED mode): 628 C files share one
pre-built registry (lsp_cross_prepare=17ms, zero slow files), 44 LSP
overrides, +9 edges vs FULL. All 200 test_c_lsp tests pass. No
SEGV/ASan. Total pipeline overhead vs FULL: +5.2%.
New CBM_MODE_ADVANCED=3. Both per-file LSP (type-aware resolution
during extract) and the cross-file LSP pass (during resolve) are now
OPT-IN: FULL and below run tree-sitter extraction + registry textual
resolution only; ADVANCED adds the full LSP layer.
Rationale: profiling showed LSP changes the kubernetes graph by only
~0.01% of edges (it mostly re-confirms textual resolution) while
costing significant memory + time. Making it opt-in keeps the default
FULL path lean (3.0 GB vs 5.6 GB peak RSS on K8s) without meaningfully
worse output, and lets users pay for LSP precision only when wanted.
Mechanism:
- cbm.c: g_lsp_enabled atomic toggle wraps the entire per-file LSP
block (Go/C/C++/PHP/Python/TS/JS/C#) + the C/C++ preprocessor-pass
LSP call. Default true so ad-hoc extraction / tests keep historical
behaviour. cbm_set_lsp_enabled / cbm_get_lsp_enabled in cbm.h.
- pipeline.c: sets the toggle (save→set→restore so it never leaks to
other callers) from p->mode before extraction; gates the cross-LSP
all_defs collection + registry build on advanced; gates the
sequential lsp_cross dispatch. Fixed the moderate_only comparison
(== FAST, not > MODERATE) so ADVANCED is not treated as lighter.
- pipeline_incremental.c: same toggle save/set/restore.
- discover.h/.c: added CBM_MODE_ADVANCED to discover's enum + all 4
filter comparisons treat ADVANCED like FULL (full discovery, no
fast-filtering). Without this, ADVANCED (numerically 3) was
mis-detected as fast-mode and dropped ~half the files.
- mcp.c: parse mode "advanced" + tool schema enum/description.
Dual-mode K8s bench: FULL 46.7s (lsp_overrides=0, 3.0 GB) vs
ADVANCED 48.8s (lsp_overrides=80,818, 5.6 GB). Graphs within 0.01%.
All 3,615 tests pass.
Replaces the prior Robin-Hood open-addressing implementation with
Verstable (https://github.com/JacksonAllan/Verstable), a 2024 state-
of-the-art C hash table. The public API in foundation/hash_table.h
is UNCHANGED — CBMHashTable is now an opaque type and all 14
production callers (graph_buffer, registry, pipeline passes,
service_patterns, watcher, semantic, etc.) get the new impl
transparently. str_intern.c keeps its specialised (ptr+len) hash —
different semantics, not a clean swap target.
Why: per-sub-phase profiling pinpointed cbm_pipeline_lsp_target_node
and cbm_registry_resolve as the dominant resolve_calls cost. Both
ultimately hit cbm_ht_get; both run millions of times on a kubernetes
index. Verstable's key innovations target exactly this:
* 4-bit hash-fragment metadata in a uint16_t sidecar array lets
chain walks skip most key comparisons WITHOUT touching the
bucket payload (cache wins on hot lookups).
* Quadratic probing with per-bucket displacement chains (not linear
Robin-Hood psl).
* Tombstone-free deletion that moves at most one existing key.
Vendored at internal/cbm/vendored/verstable/verstable.h (2069 lines,
84KB, MIT). Wrapped as a const char* → void* template inside
hash_table.c. The header struct is opaque (forward declaration only);
audited the entire tree — no caller accesses internals or takes
sizeof(CBMHashTable), so opacity is safe.
Test refactor: test_hash_table.c had two impl-specific tests:
* ht_robin_hood_bounded_psl — Robin-Hood specific; removed.
ht_stress_10000 covers the functional equivalent.
* ht_power_of_two_capacity — renamed ht_create_small_capacity,
replaced struct probes with a functional insert+lookup check.
ht_create_capacity_zero/one similarly de-internalised.
Benchmark on kubernetes (12 workers, 20,650 files):
- parallel_resolve wall: 36,376 -> 27,217 ms (-25.2%)
- pipeline.done total: 61,485 -> 51,745 ms (-15.8%)
- registry_build: ~1,000 -> 779 ms (faster — registry uses
cbm_ht_set heavily during build)
- All 3,615 tests pass; hash_table suite 100% green
- ASan/UBSan/SEGV: none
- Peak RSS: ~6.83 GB
- Graph: nodes 297,082 -> 297,102 (+20, 0.007%), edges 1,155,444 ->
1,154,025 (-1,419, -0.12%) — within tolerance; minor delta likely
from hash-iteration-order differences affecting tie-breaks in
resolution. Sanity (zod/serilog): nodes exact, edges within 0.1%.
Cumulative trajectory from broken-state baseline ~35 min on K8s ->
51.7s wall = ~40x total speedup.
cbm_service_pattern_match runs once per resolved CALL edge inside
emit_service_edge — that's 7 pattern lists × ~30 patterns × strstr
= ~210 strstrs per call. On kubernetes (~600k resolved CALL edges)
the SAME resolved QN ("fmt.Errorf", "context.Context.Done",
"errors.New", etc) repeats thousands of times across files. Caching
turns the linear pattern-list scan into one hash lookup after the
first miss.
API additions in service_patterns.h:
void cbm_service_pattern_cache_begin(void);
void cbm_service_pattern_cache_end(void);
Cache is TLS (no contention) scoped to the resolve_worker's lifetime
in the parallel_resolve phase. Capacity 8192 entries per worker —
covers typical project QN diversity without rehashing.
Encoding: enum value + 1 in the void* slot (0/NULL means "miss").
Two helpers svc_enum_to_ptr / svc_ptr_to_enum hide the trick.
Benchmark on kubernetes (12 workers, 20,650 files):
- parallel_resolve wall: 98,520 -> 90,104 ms (-8.5% vs Chunk 1
baseline, -12.8% vs original 103,281 baseline)
- resolve_calls CPU: 992,427 -> 885,663 ms (-10.8% vs Chunk 1,
-5.4% vs original — Chunk 1 added overhead that Chunk 2 recovers
AND surpasses)
- resolve_usages CPU: 11,977 -> 10,944 ms (-8.6%)
- total_loop CPU: 1,005,409 -> 897,561 ms (-10.7%)
- Tests: 3616/0
- Graph: 297,082 -> 297,080 nodes (-2), 1,153,482 -> 1,156,097 edges
(within 0.2%, semantic-preserving)
- zod/serilog sanity: edges within +/-3 of baseline
- Peak RSS: 7.46 -> 7.75 GB (+290 MB for 12 × 8192-entry caches)
Cumulative trajectory: original broken state ~35 min K8s ->
1.92 min wall (-49× total). The per-file is_import_reachable cache
(Chunk 1) + the per-worker service-pattern cache (Chunk 2) together
save ~13s of resolve wall time. Next chunks target the gbuf_insert_
edge hot path and remaining strstr scans.
KEY ARCHITECTURAL INSIGHT (credit: user observation):
The per-file LSP during extract ALREADY captures the inferred receiver
type QN + method name for unresolved cross-file calls — emit_unresolved_
call at go_lsp.c:1242-1245 stores callee_qn as '<FullReceiverTypeQN>.<
MethodName>' in result->resolved_calls flagged with strategy='lsp_
unresolved'. The cross-LSP step was REDUNDANTLY re-deriving this same
metadata via a second tree-sitter parse + AST walk.
Cross-LSP for Go is now METADATA-DRIVEN:
cbm_go_fast_resolve_qualified_calls (rewritten):
- Iterates result->resolved_calls where strategy=='lsp_unresolved'
- For each entry: try direct registry lookup of callee_qn;
if that misses, split prefix.suffix and resolve prefix via the
file's import map then lookup '<module_qn>.<suffix>' in registry
- Emits resolved entry on top (cbm_pipeline_find_lsp_resolution
picks highest confidence)
- NO tree-sitter parse, NO AST walk, NO scope tracking
resolve_worker (pass_parallel.c):
case CBM_LANG_GO now calls ONLY the metadata-driven fast path.
cbm_run_go_lsp_cross_with_registry (the slow AST-walk version) is
no longer invoked for Go — it would just re-derive identical
metadata via a second AST walk and arrive at the same answers.
Benchmark (kubernetes, full fresh index, 12 workers):
- parallel_resolve: 1.89 min (Tier 3 meta) vs 1.86 min (Tier 2) — parity
- lsp_cross.slow lines: 0 (no tree-sitter fallback runs)
- Nodes: 297,081 vs 297,082 baseline (-1)
- Edges: 1,153,163 vs 1,153,482 baseline (-319, within 0.03%)
- Tests: 3,616 / 0 failures
- zod/serilog sanity: nodes exact, edges +1-3 (within 0.03%)
The wall-time parity is informative: the AST re-walk wasn't the
dominant cost. Per-file budget (66ms across 20,650 files / 12 workers)
is now dominated by the 5 sequential resolve passes (resolve_file_
calls/usages/throws/rw/semantic) pushing ~10M total edges into the
per-worker gbuf with arena allocs + hash-bucket inserts. The Tier 3
work eliminates a genuine architectural redundancy (two walks of the
same AST producing the same metadata) and sets up Python/C/etc to
adopt the same metadata-driven pattern in follow-up commits — but
hitting the user-target <30s K8s requires attacking the gbuf hot
path, not the cross-LSP path.
The gopls 'package summary' pattern, applied properly: build the cross-
LSP CBMTypeRegistry ONCE per language at pipeline time (in a dedicated
cross_lsp_arena), share it READ-ONLY across all resolve workers via the
CBMCrossLspRegistries struct. Per-file work then becomes just tree-
sitter parse + AST walk + O(1) hash lookups — zero per-file registry
rebuilds for languages with a pre-built registry.
Each LSP gains two new public functions:
CBMTypeRegistry *cbm_<lang>_build_cross_registry(arena, defs, def_count)
— filters all_defs to its language, init + stdlib + registration
loop + cbm_registry_finalize, returns finalized registry.
void cbm_run_<lang>_lsp_cross_with_registry(arena, source, ...,
CBMTypeRegistry *reg, ...)
— skips the per-file build AND the per-file Phase 1b/1c AST scans
(those mutate the registry, which would race on a shared read-
only reg). Trade-off: a few file-local type aliases discovered
only by Phase 1b are missed; in practice the per-file extract
LSP already captured them via embedded_types/field_defs/etc.
Tested with parallel_python_lsp_override_cross_file_emits_lsp_
strategy_edges — fused + filtered + prebuilt path stays correct.
CBMLSPDef gains a CBMLanguage lang field set in pxc_build_lsp_def so
the per-language build functions can filter the global all_defs[].
cbm_parallel_resolve signature now takes CBMCrossLspRegistries* instead
of a single CBMTypeRegistry*. The resolve_worker dispatches:
cbm_pxc_registry_for_lang(lang) returns the right pre-built reg if any;
the worker then switch()es to the matching _with_registry variant.
Languages without a _with_registry variant fall through to the existing
filter + per-file build path.
Kubernetes (Go-heavy, 12k files, 139k defs) measured perf:
parallel_resolve: 5.51 min (strdup-removal baseline) -> 1.86 min
= 2.96x speedup
parallel.resolve.lsp_cross.slow lines: 1,258 -> 2
(-99.8%; the 2 remaining are Python files, not Go)
Peak RSS: 6.29 GB -> 6.23 GB (slight decrease — pre-built registry
shared, no per-file scratch)
All 3,616 tests pass; zod/serilog graphs exact match (per-file path
unaffected).
Cumulative trajectory from the original broken state on kubernetes:
35 min -> 19 min (finalize) -> 6.24 min (module-def filter) ->
5.51 min (strdup-removal) -> 1.86 min (Tier 2 Go) = 18.8x total.
Languages wired so far: Go, Python. C/C++, TS/JS, PHP, C# still use the
per-file filter+build path until their _with_registry variants land in
follow-up commits.
wyhash is a non-cryptographic hash function ~2x faster than FNV1A
for the long string keys used in CBMTypeRegistry's hash buckets
(qualified names average ~50 bytes). The header is unused for now
— prepared for a future Tier 3 symbol-interning pass that replaces
the QN-string-keyed hashtables with u32-keyed ones (rustc pattern)
where wyhash's lower collision rate matters more.
Source: https://github.com/wangyi-fudan/wyhash (Unlicense)
Vendored as a single header at internal/cbm/vendored/wyhash/wyhash.h.
Two perf changes across the 6 cross-LSP resolvers:
1. cbm_registry_finalize after the per-file registry build (all 6
LSPs). Builds the FNV1A hash buckets so lookups go through the
O(1) hashed path in type_registry.c:154. Previously this was
NEVER called anywhere in the codebase, so every method/type
lookup fell through to the O(N) linear scan — the root cause of
the 35-min kubernetes regression. Lookup count on k8s: ~50 calls
* 110k defs * 11k files = ~64B strcmps under the old fallback.
2. In go/c/py/php registration loops, replace cbm_arena_strdup(arena,
d->X) with d->X directly. The strings are borrowed from the
caller's all_defs[] arena which outlives the cbm_run_X_lsp_cross
call by the whole pipeline lifetime. Saves 5 wasted strdups per
def per file = ~6B mallocs avoided on kubernetes.
TS and CS use a different registration architecture (no inline
strdups in the body — uses a per-file ctx). Only the finalize change
applies there.
Kubernetes wall-time vs prior commit:
- After finalize alone: 35 min -> 19 min (~1.8x)
- After module-def filter (prior commit): -> 6.24 min (~5.6x)
- After strdup-removal (this commit): -> 5.51 min (~6.4x)
- Memory: 7.14 GB -> 6.29 GB peak RSS (-12%)
All 3616 tests pass, zod/serilog graph output exact match.
Previously cbm_pipeline_pass_lsp_cross ran as a separate sequential
pass that re-read every source file from disk and re-parsed every
tree-sitter tree on a single thread — a 50x regression on the
kubernetes Go monorepo (extract 10s, lsp_cross 9+ min when killed).
Three changes:
1. CBMFileResult gains const char *source / int source_len. The
parallel extract worker copies source bytes into result->arena
(per-file 100MB cap, project-wide 2GB atomic cap) so the fused
cross-LSP step can run without disk I/O. Source pointer is
borrowed by downstream readers; arena owns the storage.
2. Cross-LSP work moves into the parallel resolve worker. Each worker
processes its file with the existing parallel infrastructure
(worker pool + work stealing). Slab is reclaimed after each
per-file cross-LSP call to bound peak slab memory. The previously
separate lsp_cross pass entry function is kept for the sequential
pipeline path; the parallel path no longer invokes it.
3. CBMModuleDefIndex (gopls 'package summary' pattern) — an inverted
index module_qn -> list of def indices built once in O(D) before
resolve_worker fires. Per file, cbm_pxc_filter_defs_for_file slices
all_defs down to the file's own_module + imported modules only,
typically 50-100x smaller than the global ~110k defs on kubernetes.
Drops the per-file registry build from O(all_defs) to O(relevant).
Other plumbing: cbm_parallel_resolve signature gains all_defs,
def_count, def_modules, module_def_index params. pipeline.c builds
the index between registry_build and parallel_resolve and frees it
after. pipeline_incremental and tests/test_parallel updated to match.
Helpers in pass_lsp_cross.c exposed via the header: cbm_pxc_has_cross_lsp,
cbm_pxc_collect_all_defs, cbm_pxc_run_one(_ts), cbm_pxc_ts_modes,
cbm_pxc_build_module_def_index, cbm_pxc_filter_defs_for_file.
Kubernetes wall-time: 35 min -> 6.24 min (~5.6x) before the followup
perf commits in this series.
parse_ts_type_text in ts_lsp.c builds a 16-slot CBMType* members[] for
union (|) and intersection (&) types. The in-loop branch correctly
caps additions at mc<15, but the trailing post-loop append that
collects the final member after the last separator was unconditional,
and the subsequent members[mc] = NULL sentinel could then write to
members[16] — one past the array end.
Caught by UBSan on the zod codebase (a TS union with >=16 members
exists). Non-deterministic crash signature under pthread-parallel
extraction (12 workers): exit 134 / SIGABRT with no stack, because
libc's allocator detected the heap corruption and called abort().
Single-fixture-file test suite never exercised the parallel path or
a wide-enough union, so existing ASan/UBSan coverage missed it.
Fix: mirror the in-loop guard on the trailing append (cap at mc<15).
Add regression test tslsp_union_many_members_no_overflow that
constructs a 20-member string-literal union — extracting must not
crash. UBSan: clean. Full suite: 3616 passed, 0 failed.
Add a C# language LSP resolver mirroring the existing Go/C/PHP/Python
modules: type evaluation, using-directive resolution, method lookup,
and a corpus-seeded .NET BCL stdlib data table.
- internal/cbm/lsp/cs_lsp.{c,h}: resolver implementation
- internal/cbm/lsp/generated/cs_stdlib_data.c: BCL type/method tables
- wire CBM_LANG_CSHARP dispatch in cbm.c and lsp_all.c
- register cs_lsp + cs_lsp_bench suites in the test harness
- tests/test_cs_lsp.c, tests/test_cs_lsp_bench.c
(cherry picked from commit 8f677c67a0c9d7829cd6ac27393de2d0aaaac534)
98d5cac silently dropped index cells that exceed a single SQLite page,
which is the correct semantic (the index B-tree has no overflow page
mechanism, unlike the record B-tree fixed in #175). But silent drops
make the failure mode invisible — a caller hitting deeply nested
namespaces or very long file paths gets a partial index with no
indication anything was skipped.
Add an fprintf(stderr) at the skip site reporting the cell length and
its index, matching the existing logging style in this file (lines 311,
1305) so the dropped cells are observable for diagnosis. Consistent
with the cbm_write_db: prefix used by surrounding warnings.
write_index_btree() flushed the page buffer whenever the next cell
didn't fit, on the assumption that a freshly-initialised page would
always accept any cell. That invariant fails for index cells whose
payload (qualified_name, file_path, etc.) is larger than a full
SQLite page — after the flush, pb_cell_fits() is still false but the
code calls pb_add_cell() anyway. The subsequent content_offset
underflow corrupts the page header and the next write triggers
SIGBUS on large repos (observed on a Ruby codebase with ~215K nodes).
Re-check pb_cell_fits() after the flush and continue past cells
that still don't fit. Index entries whose keys exceed a full page
can't be stored and are not expected to survive the writer anyway —
the rest of the index is correctly preserved and the indexer no
longer crashes. Record overflow pages (#175) already covers the
record-btree side of this same shape of bug.
extract_constructor_type only accepted call-form RHS when the function
name's first character was uppercase. Misses two common typed-stub
factory patterns:
Go: client := pb.NewPromoCodeClient(conn)
Java: stub := PromoCodeServiceGrpc.newBlockingStub(channel)
Both have a lower-cased qualifier prefix, so the first character of the
full text is `pb`/`promoCodeServiceGrpc` rather than the factory name.
Match the last `.`-segment instead and accept it when it begins with
`New`/`new` and ends with `Client` or `Stub`. Returns the full
qualified call so downstream passes keep the package context.
Stays scoped — only the new factory shape is added; the uppercase-first-
char path for `Foo()`-style direct calls is unchanged.
Co-authored-by: sponger94 <45746997+sponger94@users.noreply.github.com>
Three C# extractor gaps that left modern .NET classes partially invisible
in the graph:
1. lang_specs.c CSHARP entry had `empty_types` for field_node_types, so
walk_defs never descended into field_declaration / property_declaration
nodes. Add cs_field_types and wire it in. Without this, no C# fields
or properties were ever pushed as Field defs.
2. extract_class_fields assumed the field's "type" is on the immediate
child via ts_node_child_by_field_name. Tree-sitter-c_sharp wraps fields
as field_declaration > variable_declaration(type, variable_declarator),
one level deeper than Java/Go/Rust. Detect the nested shape by walking
named children for variable_declaration and resolve type + name from
inside it. Direct shape continues unchanged.
3. C# 12 primary constructors (`class Foo(IBar bar) : Base { ... }`)
declare implicit captured fields on the class line. Tree-sitter wraps
them inside the hidden _class_declaration_initializer node, so the
`parameters` field on class_declaration may not always resolve
directly; iterate top-level children for parameter_list as a fallback.
Emit each primary-ctor parameter as a Field def with parent_class set.
Effect: dependency-injected fields on .NET 8+/9+ controllers and any C#
auto-property (`public IFoo Foo { get; }`) are now reachable as Field
nodes in the graph.
Co-authored-by: sponger94 <45746997+sponger94@users.noreply.github.com>
Per-file LSP (run from cbm_extract_file) only sees one file's defs, so
callees whose receiver type comes from an imported module stayed
unresolved — even though Go/C/Py/TS already had cross-file entry points
(cbm_run_X_lsp_cross). Those entry points existed but no pipeline pass
ever invoked them. PHP didn't even have a cross-file entry point.
This change lands the missing pieces:
1. cbm_run_php_lsp_cross + cbm_batch_php_lsp_cross — PHP cross-file
resolver mirroring py_lsp_cross / ts_lsp_cross. Reuses PHPLSPContext,
php_lsp_init, php_lsp_add_use, php_lsp_process_file. Handles PHP-
specific labels (Class / Interface / Trait / Enum / Type) and uses
def_module_qn so cross-file return types qualify against the def's
own module rather than the importing module's.
2. pass_lsp_cross.c — new pipeline pass. For every cached file result,
builds a project-wide CBMLSPDef[] (CBMDefinition → CBMLSPDef
converter), per-file import map (from gbuf IMPORTS edges), then
dispatches to the matching cbm_run_X_lsp_cross. Output strings get
copied into the per-file arena (with O(N²) memory bug avoided by
running each file inside a fresh scratch arena that gets destroyed
after the LSP call — the LSP allocates its registry + stdlib +
project defs in scratch, not in the caller's persistent arena).
3. Wired into both pipelines:
- sequential: definitions → k8s → lsp_cross → calls → ...
- parallel: parallel_extract → registry_build → lsp_cross →
parallel_resolve → ...
4. Updated tests/test_parallel.c run_parallel helper to mirror the new
parallel ordering so existing cross-file Python/PHP tests now exercise
pass_lsp_cross.
Languages covered: Go, C/C++/CUDA, Python, TS/JS/JSX/TSX, PHP. PHP cross
file is new. The other four had functions but no pipeline integration
until this change.
Verified on a 1670-file FastAPI index: pass_lsp_cross processes 1050
Python files (525 non-LSP files skipped), collects 4070 project defs in
9.2s, and parallel_resolve subsequently reports lsp_overrides=568 — the
LSP-strategy attribution gain over per-file alone.
Per-LSP emit functions dedup by (caller_qn, callee_qn) within their own
run; pxc_append_results dedups again at append time so cross-file
entries duplicating per-file output get dropped instead of inflating
result->resolved_calls.
Resolved 5 conflicts where the php-lsp branch and main both added LSP
infrastructure but with different APIs:
- Makefile.cbm: kept both TEST_PHP_LSP_SRCS and main's py_lsp/scope/
type_rep/stack_overflow test source variables in ALL_TEST_SRCS.
- internal/cbm/cbm.c: kept both #include lsp/php_lsp.h and lsp/py_lsp.h;
the per-language dispatch already supports stacking PHP after PYTHON.
- internal/cbm/lsp_all.c: kept both lsp/php_lsp.c + php_stdlib_data.c
AND lsp/py_lsp.c + python_stdlib_data.c in the compilation unit.
- src/pipeline/pass_calls.c: took main's version entirely.
find_lsp_resolution + resolve_single_call(ctx, call, lsp_calls, ...)
is language-agnostic — it matches any CBMResolvedCall whose caller_qn
hits the textual call's enclosing function and whose callee_qn
short-name hits the textual callee. PHP LSP populates the same
result->resolved_calls array via cbm_run_php_lsp, so the unified
resolver handles PHP for free. Dropped the php-branch's
qn_short_name + lsp_override_resolution helpers as superseded.
Locked confidence floor at main's CBM_LSP_CONFIDENCE_FLOOR = 0.6f
(was 0.5f on the php branch).
- tests/test_main.c: kept all suite externs and RUN_SUITE invocations
for php_lsp + py_lsp/py_lsp_bench/py_lsp_stress/py_lsp_scale +
scope + type_rep.
Auto-merged: src/pipeline/pass_parallel.c.
Brings in 52 commits from main since the branch was forked at c768b61
(#207 watcher follow-up): #208 through #281, dependabot bumps, Pine
Script, AUR docs, INHERITS edges, get_architecture wiring, codeql
revert, the entire python-lsp subsystem.
Suite: 3249 passed, 0 failed (was 2971 on main; +278 new tests from
php-lsp matches the branch's claimed test count).
Phase 5f, 4ac, 4ad: closing the long-tail gaps toward LSP-server parity.
- @phpstan-type / @psalm-type / @phan-type aliases parsed from class
docblocks and applied during PHPDoc type resolution. References to
the alias name in @var/@param/@return are rewritten to the aliased
type. Aliases live on PHPLSPContext (per-file).
- @phpstan-import-type accepted without crash (cross-file alias
resolution deferred — same root cause as Composer PSR-4).
- @template-covariant T and @template-contravariant T now register T
as a proper type-param (previously skipped along with
@template-extends).
- Closure binding: \\Closure::bind($f, $obj) and $f->bindTo($obj),
when invoked with an inline closure literal as the first arg, walk
the closure body with $this rebound to the second arg's class.
Resolves the Laravel-macroable / VarDumper / encapsulation patterns.
- @return ($x is string ? int : bool) conditional return-type syntax
accepted without crash (parsed leftmost segment as fallback).
- Long PHPDoc blocks with mixed @param/@return/@throws/@internal
parsed without misattributing tags.
Stdlib unchanged.
Tests: 248 → 278 covering:
- phpstan-type alias basic / via @param / union / array-shape
- psalm-type alternate spelling
- phpstan-import-type (no-crash)
- Closure::bind / ::bindTo / static-bind on arrow fn
- conditional return / template-covariant / -contravariant (no-crash)
- class-string<T>, int<0,100>, literal-string (no-crash on phpstan
types we don't fully model)
- 8 realistic patterns: repository, event listener, console command,
form handler, mail sender, cached service, logger w/ context,
Carbon chains in methods, Eloquent model.
Total tests: 3061 → 3091 / 0 failed.
Audit for O(n^2) hot paths after Round 10. The dedup loop in
py_emit_resolved_call scanned ALL prior entries linearly, making
emission O(N) and total file work O(N^2). On a synthetic 2000-class
fixture with 8000 calls, that dominated time at 5083 ms.
Cap the scan window at the last 256 entries. Most duplicate
emissions come from the same expression being walked twice (resolver
+ emitter passes), so they're temporally adjacent — a 256-entry
window catches every common case while keeping the dedup O(1) per
emission.
Scale measurements (test_py_lsp_scale.c added):
| Classes | Calls | Resolved | Before | After |
|--------:|------:|---------:|-------:|------:|
| 100 | 400 | 300 | 55ms | 57ms |
| 500 | 2000 | 1500 | 333ms | 323ms |
| 2000 | 8000 | 6000 | 5083ms | 1804ms|
Ratio for 20x input: 91.8x -> 31.4x (linear would be 20x; quadratic
would be 400x). No remaining quadratic hot path in code added by
this branch.
Other hot paths surveyed:
- py_register_def -> cbm_registry_lookup_type: O(types) per def at
registration, O(D x types) for D defs. Pre-existing linear scan in
type_registry.c; hash-indexing it would be a cross-language
optimization (Go and C++ LSPs benefit too).
- py_lookup_attribute / py_lookup_field: O(types) per call site.
Same pre-existing linear scan. With ~900 stdlib types + per-file
defs, accounts for ~75µs/call at 2000-class scale.
- MODULE submodule prefix scan: O(funcs) per module-attribute access,
early-exits on first match. Bounded by registry size.
- lambda / dict-literal registries: linear, but bounded by per-file
N which is small.
Bench unchanged at 100% / 12ms (52/52 calls, 178 LOC).
Martin Vogel