Deep --no-ff merge of upstream DeusData/main (597 commits, merge-base 988d975)
into fork api-consolidation (HEAD 3018b22), producing a superset that combines
upstream breadth with the fork's surviving depth.
Upstream additions integrated: semantic/embedding search (vendored/nomic),
minhash/SimHash, cross-repo + cross-file LSP + route + pkgmap passes, artifact
import/export, path_alias, git_context, hook_augment, Windows support
(compat_fs/compat_thread), the UI layer, and ~140 expanded tree-sitter grammars.
Fork depth retained: PageRank/LinkRank ranking (src/pagerank/; sort_by=
relevance/linkrank/degree/calls), dependency indexing (src/depindex/;
CBM_INDEX_MODE=DEP), token reduction, and HTTP-endpoint discovery
(src/pipeline/httplink.c, pass_httplinks.c).
Conflict resolution (27 regions across the 39-file intersection):
- src/mcp/mcp.c: kept the fork superset handler pattern (resolve_project_store +
expand_project_param + inject_context_once) as the single path; UNIONed the
tool dispatch table; emit status:"indexed", excluded[], cross_service trace.
- src/store/store.c: upstream base + fork pagerank/linkrank/node_degree tables
and sort_by plumbing; cbm_store_node_degree now excludes INHERITS
(WHERE type != 'INHERITS'); removed the duplicate cbm_store_get_db.
- src/pipeline/pipeline.c: took upstream run_extraction_phase; grafted the fork
passes (httplinks, normalize, flush_store dump branch); call
cbm_pipeline_set_committed_counts before cbm_gbuf_dump_to_sqlite.
- internal/cbm/lang_specs.c: 158 entries combining the fork section_node_types
with upstream ts_factory/embedded_imports; wired 13 orphaned type-arrays.
- src/depindex/depindex.h: renamed CBM_PATH_MAX -> CBM_DEP_PATH_MAX (it collided
with constants.h enum CBM_PATH_MAX=1024) + 11 call sites in depindex.c.
- Makefile.cbm: upstream grammar/nomic units + fork CGO targets; added
httplink.c + pass_httplinks.c to PIPELINE_SRCS; removed mongoose NOSAN
machinery (mongoose was deleted upstream).
- httplinks restored (the merge silently deleted them because the fork had left
them unchanged since the base): httplink.c/.h, pass_httplinks.c,
tests/test_httplink.c; registered RUN_SUITE(httplink) in tests/test_main.c.
- CBM_INDEX_MODE enum unified to FULL=0, MODERATE=1, FAST=2, DEP=3.
Completeness (dual-parent check): 0 upstream files dropped from the merge tree;
6 fork files dropped, all intentional (upstream-superseded): CHANGELOG.md,
vendored/mimalloc/src/{arena-abandon,segment-map,segment}.c,
vendored/mongoose/{mongoose.c,mongoose.h}. Fork features (pagerank, depindex,
token_reduction, httplink) verified present and registered in tests/test_main.c.
src/pipeline/pass_parallel.c has an identical function set to upstream (49=49,
none dropped); the large line-deltas vs upstream are within-function divergence
of the fork extractor, not dropped functions.
Gate (run manually; hooks skip on merge commits): build green
(make -f Makefile.cbm cbm, exit 0); full C test suite 5980 passed / 2 failed /
0 crashes. The fork's pre-existing ASan crash in build_project_list_error_srv is
fixed by upstream's mcp.c rewrite, so the merge is strictly better than the fork
baseline on the crash axis.
Known issue, tracked separately (NOT a merge defect): 2 nondeterministic
failures in tests/test_incremental.c — incr_db_deleted_recovery:772 (diff_pct)
and incr_accuracy_vs_full:803 (full_nodes vs incr_nodes). Cause: the fork's
extractor (internal/cbm/extract_*.c) and parallel-resolve Route-node creation
are not thread-safe under the concurrent use of upstream's parallel pipeline
(run_parallel_pipeline), producing nondeterministic route/infra HANDLES edges
(~234k edge swing between runs). Sequential extraction (CBM_WORKERS=1) is
deterministic and correct. Root cause, bisect, and repro are documented in
notes/2026-06-24-fork-vs-upstream-divergence-assessment.md (section 8f); the fix
is deferred to a dedicated thread-safety audit (the make test-tsan target is
currently a stub and needs wiring).
Vendors the two MIT-licensed grammars from intersystems/tree-sitter-objectscript
@a7ffcdf (the language vendor's official grammars, ABI 15) that PR #467 needs:
internal/cbm/vendored/grammars/objectscript_udl/ (.cls)
internal/cbm/vendored/grammars/objectscript_routine/ (.mac/.inc/.rtn/.int)
Each directory carries the verbatim generated parser.c + scanner.c + tree_sitter/
headers + LICENSE. The one adjustment: each scanner.c's upstream
`#include "../../common/scanner.h"` is repointed to a per-directory
`objectscript_common.h` (a verbatim copy of the upstream common/scanner.h), because
this repo's shared vendored/common/scanner.h belongs to the cfml/fsharp grammars and
differs. Updates MANIFEST.md + THIRD_PARTY.md.
These files are dormant until #467 (the grammar shims + extraction) is rebased on
top; main's build is unaffected.
Refs #462, #467.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
try_route_from_annotation read the annotation name via the "name" field
(ts_node_child_by_field_name). tree-sitter-kotlin annotation nodes have no
"name" field: the name lives in a nested type_identifier under (user_type)
or (constructor_invocation (user_type ...)). The lookup returned null, so
every Kotlin @RestController route was dropped (0 decorator-source routes)
while Java worked. Args are likewise under constructor_invocation as
value_arguments (each wrapped in value_argument), not Java argument_list.
Add annotation_name_node()/annotation_args_node() that fall back to the
Kotlin AST shape (Java path unchanged) and unwrap value_argument when
scanning for the route path. Kotlin @GetMapping/@PostMapping/etc. now yield
Route nodes and HANDLES edges. Adds handles_spring_kotlin regression test.
Signed-off-by: jsw6701 <86887824+jsw6701@users.noreply.github.com>
Python calls of the form iris_obj.classMethodValue('Pkg.Class', 'Method')
emitted no CALLS edge. The detection existed in walk_calls, which is dead
code — the pipeline uses cbm_extract_unified → handle_calls, not walk_calls.
Move the IRIS Native API string-dispatch detection to handle_calls. When a
Python call's callee ends in .classMethodValue, .classMethodVoid,
.classMethodBoolean, or .classMethodObject, and both string-literal arguments
can be extracted, emit a synthetic CALLS edge to Arg1.Arg2.
The pattern generalizes beyond this specific API: any library that takes
class/method names as string literals follows the same shape.
Regression test: python_iris_classMethodValue in test_extraction.c.
Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
* fix(perl): stop extracting config strings as call targets
The Perl branch of extract_scripting_callee blindly returned the text of
child(0) of every call node. In config-heavy Perl (.cgi/.pl with embedded
log4perl-style config), tree-sitter-perl misparses dotted config tokens
(e.g. "log4perl.appender.File.utf8") into call-shaped nodes, and that
dotted string was emitted as a callee_name, later matched by the generic
short-name resolver to unrelated project subs.
Now the Perl branch pulls the real name token (method/function field, else
child(0)) and validates it as a bare Perl sub/method identifier via
perl_is_identifier_callee: must start with a letter or '_' and contain only
[A-Za-z0-9_:] (allowing the '::' package separator). Any '.', whitespace,
quote, or '/' disqualifies it and NULL is returned so no CALLS edge forms.
Gated to CBM_LANG_PERL; other languages are untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
* fix(resolver): don't match Perl builtins to project subs
Perl builtins (push/shift/keys/sprintf/...) carry no language or call-kind
awareness through the generic name-matcher in cbm_registry_resolve. When a
project defines a sub whose name collides with a builtin, an invocation of
the builtin was wired to that sub by same-module / suffix matching - a
false-positive CALLS edge.
Adds cbm_perl_is_builtin (curated, sorted bsearch set of 94 perlfunc core
builtins) and applies it in both call-resolution passes (sequential
resolve_single_call and parallel resolve_file_calls), gated on the file
language == CBM_LANG_PERL and only AFTER LSP resolution has declined, so a
genuine LSP-resolved call is never suppressed. The file language is threaded
into both resolvers via a new trailing CBMLanguage parameter; every other
language reaches cbm_registry_resolve unchanged (byte-identical behavior).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
* fix(resolver): suppress generic short-name matching for Perl method calls
A Perl method call ($obj->m / Class->m) carries no receiver type at the
structural tier, so the generic short-name matcher in cbm_registry_resolve
would wire $dbh->commit / $cgi->param / $logger->log to any project sub
sharing the bare method name - the dominant source of false-positive CALLS
edges in CPAN/framework-heavy Perl. Resolving such a call correctly is the
LSP's job, not the bare-name matcher's.
Adds a CBMCall.is_method flag (zero-init false, so all other languages and
existing call sites are unaffected). method_call_expression is added to the
Perl call node set and handle_calls sets is_method=true only for that node
type when the file language is Perl. Both call-resolution passes then skip
generic resolution for Perl method calls (combined with the builtin guard
from the prior commit). Genuine intra-project function calls (non-method,
non-builtin) still resolve as before. LSP-resolved method calls are
unaffected because the guard runs only after LSP resolution declines.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
* test(perl): cover call-graph noise fixes (builtins, methods, config strings)
Hermetic tests for the three Perl call-graph noise fixes:
test_extraction.c (extraction tier):
- config string is never emitted as a callee; genuine call still extracted
- builtin calls (push/keys) extracted but never flagged is_method
- arrow/method calls ($self->commit / $dbh->commit) set is_method=true,
while the genuine function call (helper) does not
- a JS method call never sets is_method (flag is Perl-only — other
languages unaffected)
test_registry.c (resolver tier):
- cbm_perl_is_builtin recognizes core builtins (incl. first/last of the
sorted set) and rejects project subs, case variants, empty, and NULL
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
* fix(perl): address QA round 1
Round 1 (Claude panel + DO DeepSeek) findings:
- Rework the Perl noise guard so it suppresses only WEAK generic matches
(suffix_match/unique_name) and KEEPS high-confidence same_module/import_map.
The prior guard ran before cbm_registry_resolve and dropped genuine same-file
calls to builtin-named subs (e.g. a project sub log/index/open called as a
bare function) that pre-PR resolved via same_module. Extracted the decision
into pure, unit-tested cbm_perl_suppress_generic_match() shared by the
sequential (pass_calls.c) and parallel (pass_parallel.c) resolvers; corrected
the inaccurate comments (Perl has no LSP/textual stage before the guard).
- Tighten perl_is_identifier_callee to require '::' pairs (reject a lone ':',
':::', or trailing '::').
- Add resolver-contract tests covering weak-match suppression, same_module/
import_map retention, genuine-call survival, non-Perl no-op, and NULL strategy.
Verified on a real Perl monorepo: .cgi builtin/CPAN/config-string noise stays
eliminated while same_module edges to builtin-named subs are recovered.
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
* fix(perl): address QA round 2
Round 2 (Claude panel) caught a regression introduced by the round-1 refactor:
cbm_perl_suppress_generic_match whitelisted only the exact strategies
"same_module" and "import_map", but resolve_import_map can also return
"import_map_suffix" (confidence 0.85 — a genuine import-based resolution, not a
weak short-name guess). A '::'-qualified Perl builtin/method call resolved via
the import-suffix fallback was therefore dropped, contradicting the helper's
documented contract and partially missing acceptance criterion (d).
Add import_map_suffix to the kept (high-confidence) set so only the weak
short-name strategies (suffix_match / unique_name) are suppressed; update the
doc comment and add a unit-test case asserting import_map_suffix is retained.
Deferred as advisory (non-blocking, noted on the PR): a hypothetical leading-'::'
(main:: shorthand) under-extraction in perl_is_identifier_callee, and a
colon-edge-case coverage gap (logic correct by inspection).
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
---------
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
Co-authored-by: Shane McCarron <shane.mccarron@corvexconnect.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
A byte-identity audit of all vendored license files against their
upstream repositories found 113 copies differing only by a missing
trailing newline (vendoring artifact) — replaced with the exact
upstream bytes. Five needed real corrections: fennel carried an
unfilled MIT template although its upstream is CC0-1.0 at the pinned
commit; ron now carries the repo's LICENSE-MIT (dual MIT/Apache
upstream); python, wit, and verilog were refreshed to current upstream
bytes. The manifest's first-party table was also corrected: six of the
twelve grammars are self-maintained forks whose retained upstream
licenses are now byte-verified (assembly's upstream has been deleted
from GitHub; the retained MIT copy is the surviving grant). The audit
tool is kept as scripts/audit-license-provenance.py for future sweeps.
Removes the heaviest vendored grammar (66 MB) along with its language
wiring and test fixtures; the supported-language count moves to 158
across README, site, and manifest. The grammar manifest additionally
records the canonical-source decisions for the five
registry-disagreement grammars and the per-directory license files
restored earlier this week.
The alloc.h/array.h/parser.h headers in
internal/cbm/vendored/common/tree_sitter/ are part of the tree-sitter C
runtime (MIT, (c) 2018 Max Brunsfeld) but had no LICENSE file alongside
them. Copy the upstream MIT license into that directory and clarify in
THIRD_PARTY.md the split between the tree-sitter-html scanner helpers
(common/, (c) 2014) and the core runtime headers (common/tree_sitter/,
(c) 2018).
Restore upstream LICENSE files for every vendored grammar and C library
across both vendored trees (19 grammar dirs, lz4, zstd, simplecpp,
verstable, wyhash, ts_runtime, common, mongoose, xxhash, yyjson; SQLite
public-domain notice). Add the nomic-embed-code Apache-2.0 license plus a
NOTICE describing the embedding derivation. First-party grammars carry
the project MIT license.
THIRD_PARTY.md now defers grammar provenance to the verified
MANIFEST.md instead of a stale hand-written table (which misattributed
clojure as EPL-1.0 — it is CC0-1.0), covers all vendored libraries, and
documents the Hybrid LSP reference servers and stdlib type-data
provenance (typeshed, Go stdlib introspection, hand-curated specs).
Profiling symfony (416 s wall, one core pinned) convicted cbm_registry_lookup_type -> strcmp at 95% of samples: the PHP and Python def-registration loops probe the registry per Method (receiver-stub check) BEFORE the hash indexes exist, i.e. a linear scan over the growing type table - O(methods x types) per file, and the cross resolvers run per file. Java and Kotlin never called cbm_registry_finalize at all, leaving every lookup during their file walks linear over the whole cross registry. Fixes: (1) two-phase registration in php/py (types, finalize, then funcs/methods); (2) finalize added to the java/kotlin cross entry points; (3) lookups now tail-scan entries added after finalize - previously such entries were silently INVISIBLE to the hashed path (reproduce-first: tslsp_hash_registry_post_finalize_adds was red); (4) cbm_registry_finalize_into allocates the index from a per-call scratch arena - putting buckets into the pipeline-lifetime result arena per file accumulated +1.1 GB RSS on the FastAPI incremental test (incr_full_index caught it), and the py tier-2 builder (one def per call) skips mid-build finalize entirely. symfony: 416 s -> 12 s, identical node count; suite 5577/0.
The hand-rolled dump writer stored index cells fully inline regardless of payload size. SQLite's format spills index payloads above the max-local threshold (16,422 bytes at the 64 KB page size) to overflow pages, so a long key written inline makes the reader interpret key bytes as an overflow page number: PRAGMA integrity_check reported 'invalid page number 0x43654C6C' (ASCII from the key text) in idx_nodes_name on elasticsearch (very long Section names), and name lookups on that index silently returned nothing. write_index_btree now rewrites oversized cells (varint + local prefix + overflow chain via write_overflow_pages) before page building; promoted separators carry their chains with them. Reproduce-first: sw_long_index_keys_overflow (20 KB name; integrity_check failed pre-fix, ok post-fix). Also escapes the remaining raw text interpolations in DATA_FLOWS route props and HANDLES handler QNs (the ~51-edge malformed tail after 6c04ab7).
c_process_function's template-param attach wrote rf->type_param_names on the registry — in the Tier-2 cross-LSP phase that registry is shared READ-ONLY across all resolve workers, so the write raced concurrent readers AND stored a pointer into the writing worker's per-file arena; once that arena was recycled the shared registry held dangling memory, crashing other workers intermittently (bitcoin: SIGSEGV in c_process_function strcmp via cbm_run_c_lsp_cross_with_registry, ~1-in-2 runs). A registry_shared flag now marks the cross context and the attach is skipped there (cross-phase template deduction falls back to positional names — graceful degradation); the single-file extract phase keeps full fidelity. Real-repo-tier reproduction: 10x bitcoin index loop, 0 failures post-fix (pre-fix ~50% crash rate); suite 5575/0.
bind_lambda_args and the method-reference arity path indexed the resolved method signature's NULL-terminated param_types array by the CALL-SITE argument index. A call with more arguments than the resolved overload declares (overload mismatch, varargs) read past the terminator and dereferenced whatever followed in the arena as a CBMType* - SIGSEGV while indexing elasticsearch (same out-of-bounds family as #427 and the bitcoin targs fix). Both sites now bound the index by the walked param count. Reproduce-first: 6-line fixture - run(Runnable) invoked with six lambdas - crashed rc=134 pre-fix; fork+exit-signal regression test lsp_java_lambda_args_exceed_params_no_crash added; suite 5574/0.
The C/C++ LSP's explicit-template-argument call path (template_function, e.g. Using<Fmt>(v)) filled a stack array const CBMType *targs[16] without NULL-terminating it before cbm_type_substitute. With more declared type params than explicit args (bitcoin src/serialize.h: Using's Wrapper<Formatter, T&> with 2 params, 1 arg), the bounded args walk read uninitialized stack; any non-NULL garbage got bound to the type param and woven into the REGISTERED return-type graph, then dereferenced at a later call site - SIGSEGV (crash shape: string bytes misread as a CBMType*). All c_lsp targs arrays are now zero-filled, and cbm_type_substitute's args walk treats implausible values (misaligned / below the first page) as the terminator so contract violations can never leak garbage into a type graph. Reproduce-first: deterministic poison-args unit test (typerep_substitute_rejects_garbage_args_entries) + real-repo verification (bitcoin indexes clean under ASan; was a deterministic SIGSEGV).
Indexing large OSS during the perf sweep crashed three LSP resolvers, all by stack overflow in their recursive resolve walks (macOS crash reports): elasticsearch — SIGSEGV in bind_lambda_args under deep recursive java_resolve_calls_in_node frames; bitcoin — SIGSEGV in cbm_type_substitute via c_adl_resolve under deep c_resolve_calls_in_node frames; microsoft/TypeScript — SIGBUS under an unbounded lookup_member_type cycle (cyclic type graph in reallyLargeFile.ts). The expression-eval guards (eval_depth) did not cover these walks. Each resolver now has a depth-guarded entry (walk_depth cap 512 for the Java/C AST walks, member_depth cap 64 for TS member lookup); past the cap the subtree resolves as unknown — graceful degradation instead of a crash. Reproduce-first: prod CLI probes crashed pre-fix (nested-call Java rc=139, nested-call C++ rc=139); fork+exit-signal regression tests added (TS cyclic shape needs a real cross-file registry, so it is verified at the real-repo tier — microsoft/TypeScript now indexes clean). Side-finding documented in the sweep report: Java BLOCK nesting >=3000 is minutes-slow (separate pathology, not this fix).
C++/CUDA out-of-line method definitions (Foo::bar in a .cpp; the class body lives declaration-only in a header) were recorded as free Functions with no link to their class. A new helper cpp_out_of_line_parent_class() detects a qualified_identifier declarator on a function_definition and resolves the immediate enclosing class (descending nested scopes ns::Foo::bar to the direct parent); extract_func_def then promotes the def to label Method with parent_class set to the class QN so DEFINES_METHOD edges resolve. Distilled (reimplemented) from PR #428. Reproduce-first: cpp_out_of_line_method_issue428 (single-level out-of-line defs; the nested-namespace path isn't isolation-testable because tree-sitter-cpp parses a synthetic doubly-qualified def as an ERROR node, so it is exercised by real codebases).
The C++ LSP passes a class's declared template-param names (e.g. template<class T, class Alloc = ...>) alongside an instance's fewer template args (vector<int>). The substitution loop is bounded by type_params, so a param matching beyond the supplied args indexed type_args[i] past the array — reading adjacent arena memory (a type-name string byte) as a CBMType* that c_eval_expr_type later dereferenced -> SEGV (ASan: stack-buffer-overflow in cbm_type_substitute via c_eval_expr_type). Deterministic, all platforms; distinct from the #424 allocator crash. Fix at the single chokepoint: bound the args length by the (NULL-terminated) param count and gate every type_args[i] access by it, so no caller can over-read regardless of language. Reproduce-first: typerep_substitute_short_args_no_oob_issue427.
Defense-in-depth follow-up to the tree-sitter bind: route sqlite3 and libgit2
allocations through mimalloc explicitly so a correct binary never depends on the
fragile MI_OVERRIDE symbol override (where the Windows static-MinGW link can
resolve a library's malloc to mimalloc but its free to the CRT, corrupting the
heap). Prod-only, guarded by the existing CBM_BIND_TS_ALLOCATOR; the CRT+ASan
test build compiles these as no-ops and stays unchanged.
New cbm_alloc_init() consolidates all three binds behind a static idempotency
guard. main() calls it as its very first statement — before cbm_mcp_server_new
opens the in-memory store — because SQLITE_CONFIG_MALLOC must run before the
first sqlite3_open*/sqlite3_initialize, else sqlite3_config returns SQLITE_MISUSE
silently and the bind is ignored (the return code is asserted). cbm_init() also
calls cbm_alloc_init() so pipeline-pass entry points get the binds too; the
ts_set_allocator bind moved out of cbm_init into cbm_alloc_init.
sqlite3: a static sqlite3_mem_methods backed by mi_malloc/mi_free/mi_realloc,
xSize via mi_usable_size, xRoundup to an 8-byte boundary, xInit returning
SQLITE_OK, xShutdown a no-op (int-sized sqlite signatures wrapped with size_t
casts). libgit2: a static git_allocator (gmalloc/grealloc/gfree → mi_*) installed
via git_libgit2_opts(GIT_OPT_SET_ALLOCATOR). The allocator is set BEFORE any
git_libgit2_init: git_allocator_global_init keeps a custom allocator if one is
already set (git__allocator.gmalloc != git_failalloc_malloc), the pre-init global
is the fail-allocator, and there is no allocator reset on git_libgit2_shutdown —
so binding once at startup survives pass_githistory's per-call init/shutdown
pairs. libgit2 code is additionally guarded by HAVE_LIBGIT2.
Test build 5564/0 ASan-clean; prod build clean with CBM_BIND_TS_ALLOCATOR=1; the
HAVE_LIBGIT2 branch type-checked against the libgit2 sys/alloc.h struct shape;
lint-ci clean.
Root cause (corrected from the report): not a tree-sitter null-subtree sentinel — 0xffffffffffffffff is a freed/heap-corrupted pointer. The vendored ts runtime allocates via its overridable ts_current_malloc/free; under the prod build's MI_OVERRIDE=1, the Windows static-MinGW link (--allow-multiple-definition) resolves ts_malloc to mimalloc but ts_free can resolve to the CRT free (or vice-versa), corrupting the heap freelist and crashing mid-parse on large templated C++ headers (scales with parse churn, not syntax). cbm_init now calls ts_set_allocator(mi_malloc, mi_calloc, mi_realloc, mi_free) so ts allocate+free go through one allocator on every platform. Guarded to CFLAGS_PROD (MI_OVERRIDE=1); the test build stays CRT+ASan to avoid an alloc/free mismatch there. Tests in test_stack_overflow.c: allocator-binding mechanism + large-templated-C++ extraction guard (the C++ gap noted in #424).
Remove leftover getenv(CBM_DBG_*) debug fprintf blocks in extract_calls.c/pass_calls.c (cppcheck always-true). Convert parse_lisp_imports to a stack-based walk (no recursion → no NOLINT, avoids deep-nesting overflow). Apply Homebrew clang-format to all touched sources. Local suite 5562/0, lint-ci clean.
fwcd/tree-sitter-kotlin wraps a whole class in an ERROR node when a member uses companion object / inner class combined with a delegation specifier. recover_kotlin_error_classes scans an ERROR node's flat tokens for class/object keyword + identifier, emits the Class def (respecting enclosing class QN) and collects bases from delegation_specifier siblings. Strictly additive (only fires on ERROR regions). Fixes inherit_kotlin (MyClass companion + Node inner).
pass_semantic + pass_parallel: when a decorator name resolves to no local def, upsert a synthetic '<decorator:NAME>' node (label Decorator) and emit DECORATES+USAGE — fixes rust #[derive]/#[allow], swift @discardableResult, scala @deprecated (deduped per project; pipeline-level so extraction histograms unaffected). py_lsp: inject a synthetic CBMCall for resolved subscript/operator dunders on typed receivers so they reach the CALLS pipeline — fixes python s[0]->Seq.__getitem__.
Imports: per-grammar parsers for hare/pascal/powershell/scheme/racket/starlark/tcl/teal/zsh + css/html/cmake/bitbake/kconfig/gn/just/nix/jsonnet/pkl/nickel/thrift/capnp/d/tablegen/crystal/fsharp/ada; node-extraction for D/PowerShell/Pascal/Julia/F#; INHERITS for those. Pipeline edges: register Variable/Field defs for WRITES/READS resolution; CONFIGURES pass consuming env_accesses; DEPENDS_ON from go.mod/requirements.txt; SQS dot-form QN. Golden updates: julia +Class, just +Function (correct new extraction).
Four overload-scoring lookups (cbm_registry_lookup_method_by_types/_by_args, _symbol_by_types/_by_args) linear-scanned the entire project registry on every call site, ignoring the pre-built method_buckets/func_qn_buckets hash index that the by-name lookups already use. On a large C++ header (e.g. 10989 defs) that is O(calls x project_funcs) per file (~34s/file), so big codebases never finish indexing. Add a hashed fast path (O(overloads)) when the registry is finalized; preserve first-match semantics via lowest registration index (equivalence verified over 1606 lookups, 0 divergence); keep the linear fallback for un-finalized single-file registries so normal files are unaffected. Adds tests/test_cpp_index_hang.sh (scale-tier subprocess+timeout repro, opt-in via CBM_RUN_HANG_TEST).
scan_start_tag_name moves scan_tag_name's owned tag_name into a Tag, but the CF_SET/CF_RETURN/CF_ELSEIF/CF_ELSE cases return without pushing the tag to the scanner stack and (unlike CF_VOID) without freeing it — leaking the tag_name buffer under LeakSanitizer. Free the tag in those return paths, matching CF_VOID. Removes the need for the __lsan_default_suppressions(scan_tag_name) workaround in the test runner.
Both vendored generated parsers declared the entrypoint with an empty parameter list, so calling them through the const TSLanguage *(*)(void) factory pointer is an undefined function-type mismatch (UBSan, clang). Add void to match every other grammar and the extern declaration.
tree_sitter_sql_external_scanner_deserialize set state->start_tag = NULL and re-malloc'd it without freeing the previous allocation. tree-sitter calls deserialize repeatedly during a parse, leaking the prior start_tag each time (LeakSanitizer, Linux x64). Free it first — safe since create() NULL-inits it and destroy() frees+NULLs it.
cbm_kind_in_set builds a thread-local cache of calloc'd symbol bitsets (ks_cache) that was never freed, so LeakSanitizer (Linux x64) reported the per-worker-thread caches as leaks once the previously-empty grammars gained real node-type sets. Add cbm_kind_in_set_free_cache() and call it at worker-thread teardown (extract_worker, beside cbm_slab_destroy_thread) and main-thread exit (test runner, beside sqlite3_shutdown).
Seven REG_TYPE registrations in the curated Java stdlib (GregorianCalendar, FileNotFoundException, Closeable, UnaryOperator, BinaryOperator, CompletableFuture, ReentrantLock) passed their parent list as an inline compound literal. A compound literal has block storage duration, and cbm_registry_add_type shallow-copies the embedded_types pointer, so the registry kept a dangling pointer into cbm_java_stdlib_register's stack once each statement's block ended. The inheritance walk later read rt->embedded_types[0] for such a type (e.g. java.io.Closeable on a closeable.close() dispatch) — an AddressSanitizer stack-use-after-scope. Hoist the seven lists to static arrays, matching the other parent lists in the same function.
The derive-method synthesis loop tested di_entry->methods[mi].short_name before the mi < 4 bound, so at mi==4 it read methods[4] — one past the 4-element array (UBSan: index 4 out of bounds for DeriveMethod[4]). Evaluate the mi < 4 bound first so the dereference is guarded.
Whitelist find_first_descendant_by_kind and find_first_descendant_of in recursion_whitelist.h (bounded AST-depth DFS, like the existing r_collect_imports entry), and narrow two cppcheck-flagged variable scopes in sqlite_writer.c: use w->project at its single use, and fclose the value-copied wc.fp after free(w) instead of a pre-captured fp.
The CI lint (clang-format-20) flagged formatting violations across these files (pre-existing drift plus the new extraction code). Reformat to satisfy the check; no logic changes.
msys2-clang on Windows lacks the GNU/BSD memmem extension, breaking the Kotlin LSP compile (kotlin_lsp.c called memmem at 4 sites). Add a portable hand-rolled cbm_memmem in helpers.c/.h and use it instead. kotlin_lsp.c also picks up clang-format-20 reflow.
Make the per-grammar regression suite fully green. (1) Add def/name extraction for 22 IDL/config/niche grammars that previously yielded only a Module node (agda, pony, move, cobol, janet, pine, smali, verilog, vhdl, systemverilog, protobuf, graphql, thrift, capnp, smithy, wit, prisma, cmake, puppet, tablegen, assembly, nasm): corrected function/class node-type sets in lang_specs.c and per-grammar name resolution in extract_defs.c (these grammars define no tree-sitter 'name' field, so names are read from typed child nodes; a depth-bounded descendant search handles the FIELD_COUNT-0 (System)Verilog wrappers). (2) Resolve same-file CALLS for groovy, solidity, and ada (callee handlers in extract_calls.c plus descending into Ada subprograms). Golden label histograms and fixtures are updated to the real new extraction output (min_defs raised, never lowered).
Add per-language callee extraction so a same-file caller->callee resolves to a CALLS edge for the lisp dialects (Clojure, Common Lisp, Scheme, Fennel, Racket, Emacs Lisp — head symbol of a list/list_lit), F# (application_expression long_identifier_or_op head), PowerShell (command_name child of a command), WGSL (identifier under type_constructor_or_function_call_expression), and Dart (identifier sibling of the invocation selector); and accept ReScript's value_identifier function field in the shared field resolver. Reduces contract_calls_breadth from 14 to 3 (groovy, solidity, ada still need dedicated branches).
parse_generic_imports only scans direct children of the root, which missed three grammars: Dart wraps imports in import_or_export (the old dispatch matched the never-emitted import_declaration), Haskell nests import nodes under an imports container, and Zig's @import is a builtin_function nested inside a variable_declaration. Add dedicated parsers (dart finds the nested string_literal URI, haskell descends the imports container reusing the module field, zig DFS-finds @import/@cImport builtin_functions) so all three extract their imports.
collect_bases_from_field ignored the bare identifier (and dotted attribute) node that tree-sitter-python uses for a base class inside the superclasses argument_list, so it fell through to the raw-text fallback and captured "(Base)" with parentheses. That never resolved, leaving every Python subclass with zero INHERITS edges. Accept identifier/attribute children so the bare base name is extracted. Turns python_class_base_extracted_bare green.
Several grammars (clojure/racket/scheme, ada/pascal/fortran/fsharp, cairo/d/odin/squirrel/rescript, and the hlsl/ispc/slang C-family) extracted only a Module node. Add per-grammar def-name resolution in extract_defs.c with matching lang_specs entries so their functions reach the graph, and descend Kotlin import_list/import_header in extract_imports.c so Kotlin imports are captured. The vendored grammar MANIFEST documents each custom case.
Andrew Hundt