tree-sitter's GLR parser merges ambiguous parse-stack heads recursively in
stack_node_add_link (internal/cbm/vendored/ts_runtime/src/stack.c), once per
nesting level. On pathologically nested input whose grammar is ambiguous
(Perl's paren-optional calls, f(f(f(...)))), that recursion overflows the
native stack during the parse — a small ~1 MB Windows thread stack, and even
an 8 MB POSIX stack at extreme depth — before any extraction runs. Grammars
that are unambiguous here (Java, C++) never trigger the recursive merge.
Cap the recursive merge at CBM_TS_STACK_MERGE_MAX_DEPTH (512) via a
depth-tracked inner worker; the public stack_node_add_link is a thin wrapper
seeding depth 0, so every existing call site is unchanged. Past the cap the
ambiguity is left on the GLR stack instead of merged — a valid parse, never a
wrong one. 512 frames is well within a 1 MB stack while far exceeding any
realistic source nesting.
Add perl_glr_deep_parse_recursion_capped: a crash-isolating regression that
parses deep ambiguous Perl directly (a forked child on POSIX, in-process on
Windows), bypassing cbm_extract_file's CBM_PERL_MAX_PARSE_NESTING pre-parse
guard so the vendored cap itself is exercised rather than the guard.
Narrow vendored-runtime change; no other parsing behavior is affected. The
higher-level Perl pre-parse guard (issue #913 context) stays in place and can
be retired in a follow-up once this cap is proven in the field.
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
Two O(n^2) patterns in the Perl LSP, the class the ARM
extract_wide_flat_file_is_linear guard flags:
- Every child-iteration loop used indexed ts_node_child(node, i), which is
O(i) per call and O(n^2) over a wide flat node. Route them all through a new
perl_collect_children() helper mirroring wd_collect_children() in
extract_defs.c: a forward TSTreeCursor collect for wide nodes (>= 64
children), NULL (cheap indexed access) for the small common case.
- perl_attach_methods rebuilt each package's whole NULL-terminated method
array on every sub via perl_type_add_method, O(subs^2) on a single-package
file. Replace with a collect -> qsort-by-package -> build-once batch
(O(S log S)); source order within a package is preserved so first-defined
still wins on redefinition.
No behavior change: all perllsp_* resolution tests and
lsp_perl_deep_expression_no_crash pass; check-lsp-originality.sh reports clean.
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
A deeply nested Perl call chain f(f(f(...))) overflowed the stack while
parsing on small-stack platforms: tree-sitter's GLR parser recurses once
per nesting level in stack_node_add_link (vendored ts_runtime/src/stack.c)
merging the ambiguous parse-stack heads that Perl's f(...) grammar produces.
The overflow happens during ts_parser_parse, before any LSP walk runs, so
the CBM_LSP_PERL_MAX_WALK_DEPTH guards can never fire. This crashed
test-windows (CLANG64) with an AddressSanitizer stack-overflow and hung
test-unix (ubuntu-24.04-arm) to a 4h timeout on lsp_perl_deep_expression_no_crash.
Add cbm_source_nesting_exceeds(): an O(n) early-exit bracket-depth scan, and
skip extraction (clean has_error, zero edges) for Perl input whose nesting
exceeds CBM_PERL_MAX_PARSE_NESTING (128) before it reaches tree-sitter.
Scoped to Perl because only its ambiguous call grammar drives the GLR
recursion to the nesting depth; C/Java/Python parse the same shape with a
single stack head. 128 is ~10x above any real Perl nesting and well below
the depth where the parser overflows a 1MB stack.
Correct the lsp_perl_deep_expression_no_crash comment to describe the actual
mechanism (parser recursion, not the LSP walk).
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
Optional, non-blocking follow-ups from the #461 review:
- perl_eval_function_call_type: split qualified static calls on the LAST
"::" (mirroring perl_resolve_function_call) so return-type chaining on
Foo::Bar::sub() resolves instead of yielding unknown.
- cbm_run_perl_lsp: finalize the type registry into a per-call scratch
arena (as php_lsp/java_lsp do) before resolution, so method/func/symbol
lookups use O(1) hash buckets instead of a per-file linear scan.
- perl_infer_self_type: match the `shift` invocant idiom at word
boundaries and accept `$_[0]`, so `my $self = shifty();` no longer
mis-binds the receiver (tightens the zero-edge guarantee).
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
perl_resolve_function_call split a package-qualified call name on the FIRST
"::" via strstr. For multi-level packages (Foo::Bar::sub) this yielded
pkg "Foo" and a sub name still containing "::" ("Bar::sub"), which never
resolved. The call then fell through to the generic bare-name fallback, which
matches on the bare sub name only — collapsing distinct packages' same-named
subs onto a single winner and orphaning the rest. Single-level calls
(Foo::sub) were unaffected, which is why the existing test missed it.
Split on the LAST "::" instead so the full package name reaches
perl_lookup_method, which resolves to the correct fully-qualified sub.
Adds perllsp_static_multilevel_package_call covering the multi-level form.
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
F1: dot the Exporter import target so seeded CPAN exported subs resolve.
perl_collect_qw_imports built colon-form targets (Scalar::Util::blessed)
but the stdlib registry keys curated CPAN subs in dotted form
(Scalar.Util.blessed) and lookup is exact-match. Wire in perl_pkg_to_dot
to dot the module portion and drop the now-unnecessary (void) cast.
F2: add a recursion-depth guard (CBM_LSP_PERL_MAX_WALK_DEPTH=512) to both
AST walkers (perl_resolve_calls_in_node, perl_pass1_scan) via a depth-
guarded wrapper + inner split, mirroring java_lsp's JAVA_LSP_MAX_WALK_DEPTH.
Past the cap a subtree is skipped (graceful degradation, no wrong edge),
preventing stack overflow on pathologically nested input.
F3: lock the shared last-"::"-segment normalization in lsp_resolve.h with a
direct regression test over cbm_pipeline_find_lsp_resolution: a qualified
static call still resolves AND the cross-namespace mis-attribution edge case
is bounded by caller-QN equality + the confidence floor.
F4: implement SUPER:: dispatch. Populate enclosing_parent_qn from the
enclosing package's first @ISA parent and resolve $self->SUPER::method() to
that parent's method (strategy perl_method_super). No known parent or
unresolved method emits no edge (zero-edge guarantee preserved).
Tests: perllsp_cpan_exported_function, perllsp_super_dispatch,
perllsp_super_no_parent_no_edge, lsp_perl_deep_expression_no_crash,
lsp_resolve_qualified_static_call_normalizes_colons,
lsp_resolve_misattribution_is_bounded.
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
perl_collect_isa_assignment relied on the assignment's `right` field, but
tree-sitter-perl flattens a parenthesized RHS (`our @ISA = ('Base')`) so the
`right` field points at the `(` token while the parent string literals are
sibling children of the assignment. The single form `@ISA = 'Base'` worked but
the common parenthesized form silently collected zero parents, so @ISA
inheritance never populated embedded_types and method dispatch could not walk
the MRO.
Scan every named child after the `=` instead of only the `right` field, covering
both `@ISA = 'Base'` and `@ISA = ('Base', 'Other')`. use parent / use base were
already handled via a separate path and are unaffected.
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
- cbm.c: include lsp/perl_lsp.h and dispatch CBM_LANG_PERL -> cbm_run_perl_lsp
- lsp_all.c: unity-include perl_lsp.c + generated/perl_stdlib_data.c
- Makefile.cbm: register TEST_PERL_LSP_SRCS and append to ALL_TEST_SRCS
- tests/test_perl_lsp.c: placeholder suite so the Makefile var resolves
(full suite + test_main.c registration land in plan 22-04)
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
Four refinements to the coverage signal:
- extraction: subtract DEFINITE recovery before flagging. Tree-sitter
error recovery plus the ERROR-descending def walker often re-extract
constructs inside a failed region (verified: `def broken(:` comes
back as a def); a region whose every line is covered by definitions
that START inside it is not a miss, and a fully recovered file is not
flagged at all. Container defs (Module/Package) don't count as
evidence, and partially covered regions stay flagged — the
#ifdef-split case keeps its flag because the first branch's function
is genuinely lost
- naming: the query_graph option is graph="missed" (the graph shows
ONLY misses — "coverage" was misleading); shadow project renamed to
"<name>::missed"; tool descriptions updated so agents discover both
the option and its semantics
- hook: the CLI-installed PreToolUse augmenter now also matches Read
and injects a coverage note when the file being read is listed as not
fully indexed ("line ranges X-Y could not be parsed — the file
content you are reading is ground truth"). Safe against the old
issue-362 hazard: the augmenter is structurally non-blocking (always
exit 0, additionalContext only), mirroring the Gemini matcher that
already includes read_file; matcher upgrade bookkeeping updated
- ui: "Missed files" toggle in the graph sidebar renders the miss
graph as a second graph option — /api/layout gains graph=missed
(same db file, shadow-project scoping; base project name validated
as before)
Tests: recovery-subtraction cases (recovered def unflagged, garbage
region flagged, trailing recovered defs keep the flag), CLI matcher
tests updated, e2e resilience fixtures switched to an unrecovered
miss; verified end-to-end: hook emits the note for a flagged file and
stays silent otherwise, /api/layout?graph=missed serves the miss graph
and the code graph is unchanged, frontend build + tests green.
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Files whose parse tree contains tree-sitter ERROR/MISSING regions were
silently indexed as if complete — constructs inside those regions are
absent from the graph with no signal anywhere (classic trigger: the
preprocessor-blind #ifdef-split-brace pattern in C).
- cbm_extract_file: set parse_incomplete and record the 1-based line
ranges of the top-most error regions (bounded at 64), serialized as
"start-end,start-end" into the result arena
- pipeline: record such files under the new phase "parse_partial"
(distinct from skips — the file IS indexed) in both the parallel and
sequential extraction paths; stamp the File node with
{parse_incomplete:true, error_ranges} on full and incremental runs
via a new cbm_gbuf_set_node_props (upsert survivor rules bypass)
- mcp: report parse_partial {files, count, truncated, note} separately
from skipped[] (skipped_count semantics unchanged); the per-run
logfile lists parse_partial rows; the index_repository tool
description documents the signal and the query_graph pattern to find
flagged files
- tests: new parse_coverage suite (RED-first, 7 cases incl. false-
positive guards and the region cap) plus an end-to-end resilience
guard asserting response fields, logfile, File-node marker, and the
advertised query_graph query
The signal is explicitly best-effort: a flag means constructs in the
listed ranges were dropped (prefer grep there); the absence of a flag
is NOT a completeness guarantee.
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The import path allocated the destination from the plaintext original_size
field in artifact.json and passed that value to the decoder truncated
through int. A large real database already wraps that cast (a full-kernel
index is ~14 GB), and a doctored field could additionally desync the decoder
capacity from the actual buffer. Read the decompressed size from the zstd
frame's own content-size header, require the metadata field to match it, cap
the total, and run the decoder with a size_t capacity equal to the
allocation. cbm_zstd_decompress now takes size_t lengths and returns int64_t.
Adds a regression test that a mismatched original_size is rejected.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
An 81k-file TypeScript corpus previously never finished indexing. Three
stacked defects, found by stack-sampling the live grind and reading the
supervisor's quarantine records:
1) walk_defs pushed children via index-based ts_node_child(i), which is
O(i) per call in tree-sitter — O(n^2) per wide node. A 3.5 MB file
with ~580k flat comment siblings needed ~1.7e11 iterator steps and
hung extraction past the supervisor's quiet-timeout. All six descent
sites (including the default loop) now collect children linearly with
a TSTreeCursor (wd_collect_children); small nodes keep the direct
index path. Guard: extract_wide_flat_file_is_linear (400k comment
siblings — the monster's exact shape — bounded at 30s).
2) The sequential cross-LSP driver handed the FULL def list to a full
per-file registry build+finalize — O(files x defs); 74% of samples
sat in build_qn_index. The parallel resolve worker's dispatch
(module-def filter -> shared prebuilt registries -> filtered
fallback) is now extracted into cbm_pxc_dispatch_file and drives
BOTH pipelines — one path, one semantics. The shared registries
live in a caller-owned arena that outlives the calls pass
(resolved_calls borrow registry strings; freeing at pass end was a
use-after-free ASan caught in pass_calls). Rust is exempt from the
def filter: Cargo-manifest cross-crate resolution needs defs from
OTHER workspace crates that the own+imports filter drops (this
starvation predated the unification on the parallel path). Guard:
pipeline_seq_ts_cross_uses_shared_registry (full builds 40 -> 0,
cross-file edge preserved).
3) The crash supervisor's recovery re-ran the worker SINGLE-THREADED to
keep one exact marker. At scale that fell into the sequential crawl,
was killed as a hang mid-pass, and the stale extraction marker got
four innocent files quarantined, one 15-minute retry at a time.
Recovery (and the terminal partial run) now re-run PARALLEL; the
marker is an append journal ("S <rel>" / "D <rel>", written by
extraction and both resolve drivers), the suspect set is the open-S
in-flight set, and a file is quarantined only when it recurs across
two consecutive failed runs — one per round, oldest open S first;
disjoint consecutive sets stop the loop rather than blame an
innocent. Guard: index_recovery_parallel_quarantines_crasher
(single-threaded spawn count must stay zero; verified RED against
the old loop).
Also: parse_ts_type_text gains a dynamic work budget proportional to
the input (1M + 64x source bytes, CBM_TS_TYPE_BUDGET override, warn
once then degrade to unknown) replacing unbounded recursion cost.
Measured on microsoft/TypeScript (81,398 files): infinite -> 33s cold
full index, 295k nodes / 779k edges, zero dangling edges, zero files
skipped — the monster file itself now indexes cleanly. Full 9-language
sweep green with no quality flags; kernel unchanged at 390s / 8.5M
nodes / 0 dangling.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Closes#856: native `fetch()` calls (Node 18+ built-in, and the identical
browser/WHATWG API — no static way to tell them apart, and no need to:
both make a real outbound HTTP request) were invisible to cross-repo
intelligence. A bare `fetch(url)` has no import and typically no local
definition, so registry resolution comes back empty and the call was
silently dropped instead of becoming an HTTP_CALLS edge.
The obvious fix — add "fetch" to the http_libraries substring table in
service_patterns.c — is wrong: that table is consulted by an early,
unconditional check in both pass_calls.c and pass_parallel.c that runs
before/regardless of whether the callee resolves to a real local
definition (by design, so `axios.get(url)` still classifies even when the
registry mis-binds bare `get` to an unrelated local method). That's safe
for "axios"/"requests" because nobody names their own function that; it
is not safe for "fetch", which collides with a plausible local identifier
(`function fetch(){}` or `const fetch = () => {}`).
Instead, cbm_service_pattern_is_global_fetch() is a new, narrow, exact-
match check consulted only in the *empty-resolution* fallback in both
files — the existing #523 path for unindexed external libraries. By the
time that branch runs, the registry has already had its chance to
resolve "fetch" to a local/imported definition; only a genuine miss
reaches the new check. This also means a member call like `repo.fetch()`
is naturally excluded (its callee_name is "repo.fetch", not the bare
"fetch" this checks for).
pass_parallel.c's empty-resolution branch calls the low-level
emit_http_async_service_edge() directly rather than going through
emit_service_edge(), which re-derives its own classification from
res->qualified_name via cbm_service_pattern_match() — a call with
"fetch" would come back CBM_SVC_NONE there and silently fall through to
a plain CALLS edge.
Three new tests in test_pipeline.c:
- bare fetch() -> HTTP_CALLS, sequential path (< 50 files)
- bare fetch() -> HTTP_CALLS, forced through the parallel resolver
(>= 50 files)
- a local `function fetch(){}` shadowing the global -> plain CALLS to
the local definition, zero HTTP_CALLS (the false-positive this PR
must not introduce), bundled with a `repo.fetch()` member-call check
in the first test
Verified end-to-end, not just by inspection: prod build clean under
-Wall -Wextra -Werror; test build clean under ASan+UBSan; full suite
5936 passed, 1 skipped (pre-existing, unrelated), 0 failed; clang-format
clean on all touched files; lint-cppcheck's pre-existing failures
(extract_defs.c, compat_regex.c) reproduced identically on unmodified
main via stash/pop, confirming they predate this change.
Refs #592.
Signed-off-by: Alexandros Pappas <11921291+apappas1129@users.noreply.github.com>
The four kernel rust files' wall time was a CONSTANT ~63.5 s that no
resolution-side fix moved -- because it was not resolution. The shared
all_defs registry build had two lookups INSIDE its registration loop
(receiver auto-registration probe via cbm_registry_lookup_type, and the
trait-linkage scan), and before finalize the registry has no buckets: both
probes were full linear scans over the types registered so far. On the
1.4M-def shared build that is ~10^10 strcmps ~= 63 s, paid once by the
building file while the sibling null-filter files waited on the once-guard
for exactly the same wall time. This is the checklist's own
lookup-in-registration-loop pattern -- invisible on small per-file builds.
Fix: a CBMIdxMemo (new in lsp_neg_memo.h: exact-match, strcmp-verified
string-key -> index map, arena-backed, first-put-wins to mirror the linear
scans' first-match semantics) maintained alongside every add_type in
rust_populate_cross_registry; both probes become O(1).
Also rides along: a per-root-invocation macro-expansion memo. Kernel
macro_rules are self-recursive (define_sizes! re-invokes itself) and the
expansion fallback ("no pattern matched -- still expand the first rule")
made recursion non-convergent: unbounded BREADTH under the depth-8 guard
turned 2-44 source invocations into ~200k expansions (each a fresh
tree-sitter parse of the substituted body). Identical (macro, substituted
text) now expands once per chain; the memo resets at each top-level source
site so per-site attribution is untouched, and the arg re-parse dedup only
applies at recursion depth > 0. Side effect: the eval-step cap no longer
exhausts, so a few previously-truncated real calls now resolve (kernel:
+25 edges).
Verified: kernel rust subtree byte-identical before/after (sorted node and
edge sets, exact); rust_lsp suite 506 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Macro-expanded kernel rust asks the same failing (receiver, method) /
(trait, method) questions thousands of times per file; each repeat re-ran the
full registry-pure cascade (embedded-impl walk + trait-default tail) against
the shared 1.4M-entry registry -- ~63 s per trait-heavy file even after the
build-once registry and the trait indexes.
New internal/cbm/lsp/lsp_neg_memo.h: a generic, arena-backed negative-lookup
memo for LSP resolve cascades (open-addressing u64 set, site-tagged FNV pair
keys, lazy alloc, grow-by-rehash). Gated on a SEALED registry (read_only), so
a miss is a pure fact of (registry, query); the cheap direct lookup stays
before the memo check so a hash collision can never hide a real hit. Any
language wires it in a few lines -- rust first (rust_resolve_trait_method and
rust_find_sole_trait_impl, both proven to read nothing but the registry and
their query strings); the C resolver's bespoke memo can migrate, and the
audit's php/c#/java/kotlin cascades are wiring candidates.
Only the full miss is memoized (zero impls and no trait default), preserving
out_impl_count/out_n fidelity for the ambiguous >=2-impl cases.
Verified: kernel rust subtree indexed before/after -- sorted node AND edge
sets byte-identical (0 diff; exact comparison, now that indexing is
deterministic). rust_lsp suite 506 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Profiling the kernel showed 4 tiny trait-heavy rust files at ~90 s EACH in
cross-file resolve. Live call trees put the time not in lookups but in
REGISTRY CONSTRUCTION: cbm_pxc_filter_defs_for_file returns NULL for these
files (their path-derived module QNs never match the def_module_qn index
keys), so each one built its own registry over the ENTIRE project def set
(millions of entries at kernel scale).
Original premise corrected during implementation: only 4/87 rust files (on
the kernel rust subtree) hit the NULL-filter path; 83/87 resolve against
proper import-filtered subsets. A registry shared across ALL rust files
would hand the subset files a superset and could add edges. The fix is
therefore CONDITIONAL, byte-identical for every file by construction:
- filter NULL -> ONE shared all_defs registry (built once in pipeline.c,
finalized, read_only-sealed; module_qn=NULL is safe: an instrumented probe
found 0 NULL def_module_qn across all_defs, so the per-file def_mod
fallback never fired anyway). Kills the amplifier.
- filter SUBSET -> today's per-file subset build, unchanged.
The cross-crate Cargo manifest reaches the shared path through
cbm_pxc_get_rust_manifest() (same worker-thread thread-local the fallback
read), preserving cross-crate resolution exactly.
Byte-identity verified empirically: single-threaded sorted edge-set diff on
the kernel rust subtree, fix off vs on -- zero difference outside
SEMANTICALLY_RELATED edges, which self-differ between two runs of the SAME
unfixed binary (the pre-existing semantic non-determinism guarded by the
repro_parallel_edge_determinism RED test). New suite test
rustlsp_shared_registry_resolves_like_per_file exercises the shared path.
Suites: rust_lsp 506, c_lsp 750, registry 54, pipeline 216, java_lsp 94,
kotlin_lsp 78, php_lsp 278, store_nodes 56 -- zero failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
flatten_trait_into_class() iterated reg->funcs with a live upper bound
(reg->func_count) while cbm_registry_add_func() appended to that same
array inside the loop. Each copied method is tagged with
receiver_type = class_qn, so when a trait adopts itself -- directly
(`trait T { use T; }`) or via an alias that resolves back to the trait
by short name (`use X\T as A; use A;`) -- class_qn equals
canonical_trait_qn and every appended method re-matches the loop filter,
extending the iteration. The loop never terminates: it arena-allocates a
fresh method each pass until the process exhausts all memory (observed:
40 GB+, freezing the host).
Fix:
- Short-circuit self-flattening: a trait cannot meaningfully use itself
(PHP itself rejects it), so return early when class_qn equals the
resolved trait QN.
- Snapshot reg->func_count before the loop so entries appended during
iteration are never revisited (defensive against the mutate-while-
iterating hazard in general).
Add regression test phplsp_trait_self_use_terminates: it hangs/OOMs on
the pre-fix code and passes after. Verified end to end -- the prod binary
now indexes the triggering file in <0.5s at ~6 MB RSS instead of running
away.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Mubariz <mubariz@sportimport.de>
The Rust cross-file resolver's trait-method, sole-trait-impl, and free-func
fallback loops scanned the ENTIRE type/func registry per call site on a miss.
On kernel-scale registries (~10^6 entries, macro-inflated) four tiny trait-heavy
rust files burned ~96 s EACH (~200k eval-capped call sites x full-registry
strcmp scans).
Fix: two hash indexes built at registry finalize, mirroring the existing
bucket+entries idiom -- fnv1a(bare(embedded_type)) -> chain of declaring type
indices (trait -> implementers), and fnv1a(short_name) -> chain of free-func
indices. Chains preserve ascending registry order (first-match tie-break
unchanged); allocation-free iterators fall back to the post-finalize tail and
to a full scan on unfinalized registries, so the candidate set is identical in
every mode. The four rust loops now walk the iterator and re-apply their exact
original predicates -- byte-identical results, loop bodies untouched.
Verified: graph equivalence on the kernel rust/ subtree (nodes 9654, edges
44798 identical with and without the change); registry_short_name_indexes unit
test pins iterator contents/order/dedup; suites c_lsp 750, rust_lsp 505,
registry 54, pipeline 216 -- zero failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
py_register_instance_field mutated the shared Tier-2 registry through direct
struct writes that bypass the read_only seal: it overwrote field-type pointers
in place and reassigned the sealed type entry's field arrays to memory owned by
the per-file arena. In the fused parallel resolve path (one sealed registry
shared by all resolve workers) that is a cross-thread data race, and it leaves
the shared registry pointing into arenas that die with their file -- a
cross-file use-after-free. The count-based seal guard could not see it because
type/func counts never change.
Python extraction emits no instance fields, so resolve-time discovery is the
ONLY source of self.x / PEP-526 fields; a plain read_only gate would silently
drop real attribute-chain CALLS edges. Fix instead routes discovery into a
per-file field overlay on PyLSPContext (arena-backed, dies with the file) and
has py_lookup_field_depth consult shared-base + overlay at the exact precedence
point of the direct field scan. Mutable per-file registries keep the original
in-place write, byte-identical.
Reproduce-first, both halves pinned:
- seal_py_shared_registry_readonly_fields: RED on the bug (field count 0->1 on
a sealed registry), GREEN with the overlay.
- pylsp_fused_self_attr_chain_via_overlay: GREEN with the overlay, RED under a
simulated plain gate (self.b.work() CALLS edge lost) -- proven both ways.
Suites: c_lsp 749, py_lsp 63+45 stress/scale/bench, pipeline 216, registry 54,
scope 10, type_rep 19 -- zero failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Kernel-scale profiling put ~87% of the 361 s cross-file resolve phase inside
c_lookup_member_depth's miss cascade: in macro-heavy C nearly every member
lookup misses, identical (type_qn, member) pairs miss millions of times, and
each miss re-paid an arena_sprintf("%s.%s") (the strlen storm) plus the
O(type_count) short-name registry scan.
Fix, semantics preserved exactly:
- Per-file negative memo (open-addressing uint64 set) on (type_qn, member),
active only at depth 0 under the shared READ-ONLY Tier-2 registry, where the
registry-only cascades are pure functions of (type_qn, registry). The direct
method lookup always runs first (collision/staleness guard) and the
context-dependent scope-alias path is always evaluated; only the provably-
NULL module-prefix/base-class/short-name cascades are skipped on a memo hit.
- Module-prefixed QN retries build into a stack buffer via cached
module_qn_len (arena_sprintf only as truncation fallback).
Soundness: cbm_c_build_cross_registry finalizes then sets read_only=true;
cbm_registry_add_func/add_type hard-return when read_only, so no miss can
flip to a hit during resolve. Suites: c_lsp 748/0 (incl. the tier2 read-only
guards), pipeline+registry+scope+semantic 656/0. Graph equivalence verified
on fs/xfs: nodes 13341 / edges 58559 byte-identical with and without the fix.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
libgit2 is licensed GPLv2-with-linking-exception, and this project is
deliberately GPL-free (see scripts/license-policy.json). libgit2 was only
an OPTIONAL, faster git-history code path guarded by HAVE_LIBGIT2 and
auto-detected via pkg-config; it always shipped with a popen("git log ...")
fallback that release binaries already used (they were built without
libgit2). Making that popen fallback the SOLE git-history implementation
means there is NO change to shipped behavior.
Changes:
- pass_githistory.c: collapse the HAVE_LIBGIT2 #ifdef so only the popen
parse_git_log remains; drop the now-unused <git2.h>/<time.h> includes.
- cbm.c / cbm.h / main.c / subprocess.c / index_supervisor.h: remove the
libgit2 mimalloc allocator bind and its >=1.7.0 version guards; keep the
tree-sitter + sqlite3 binds; correct the comments.
- Makefile.cbm: remove the optional-libgit2 pkg-config detection block,
the REQUIRE_LIBGIT2 error gate, and all LIBGIT2 CFLAGS/LIBS/FLAGS.
- .github/workflows/_test.yml: drop the REQUIRE_LIBGIT2 test-matrix leg
plus its libgit2-dev/pkg-config apt install (lower CI cost, no new
gating, no trigger change).
- flake.nix: drop libgit2/pkg-config from the dev shell (now-dead dep).
HAVE_LIBGIT2 no longer exists anywhere in the tree.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Inline Rust test fns (#[test], #[tokio::test], #[async_std::test],
#[actix_rt::test], #[test_case::case]) inside a regular .rs file were indexed
as ordinary Functions (is_test=false) and leaked past the store.c is_test!=1
filter into graph/agent context — test detection was file-path-based only
(cbm_is_test_file: *_test.rs / test_*). Add rust_def_is_test(decorators)
(mirroring rust_cfg_qualified_name's scan) and set def.is_test in
extract_func_def's Rust branch.
Closes#855.
Signed-off-by: Greg Tiller <tiller@dal.ca>
A JS/TS/TSX member call `x.foo()` whose receiver is not `this`/`super` now
sets CBMCall.is_method, mirroring the existing Perl arrow/method flag. This
lets the call-resolution passes recognise a member call whose receiver type
the TS-LSP could not resolve, so they can decline to bind it to an unrelated
project symbol by a weak short-name guess (#592/#606; precedent PR #477 for
Perl).
`this`/`super` receivers are left unflagged — their target is the enclosing
class, where a namespace-proximity weak match is usually right. Bare calls and
new-expressions have no member receiver, so they stay is_method=false; every
non-TS/JS/Perl language is byte-identical (the struct is zero-init).
The pre-#592 test that asserted a JS member call never sets is_method inverts;
it is replaced by TS and JS flagging tests plus a Go test that keeps the
"flag-exempt languages are unaffected" contract, and the cbm.h field comment
is updated to describe both the Perl and TS/JS semantics.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Hitoshi Okazaki <okazaki.hitoshi@tecnos.co.jp>
- format the include trailing comments and wrap the over-long reap log line
(clang-format-20 violations flagged by lint / lint)
- allowlist subprocess.c's fork with justification (security-static): the
supervisor primitive forks and execs immediately; no code runs in the
forked image
- move bin into the fork-child block alongside argv (cppcheck variableScope
consistency)
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Mixed Gradle/Maven source roots (src/main/java + src/main/kotlin) share
JVM packages that path-derived module QNs cannot join: cross-file LSP
emitted package-shaped names while graph nodes stayed path-derived, and
same-package defs across roots were filtered out, so Java<->Kotlin calls
fell back to weak or wrong textual edges.
The declared package is JVM ground truth. Normalize JVM defs to their
declared package (inferring it from conventional source roots only when
no package was extracted), index defs by declared namespace alongside
the path module so same-package defs survive per-file filtering, restrict
JVM callers to JVM defs, register cross-def return-type signatures for
Kotlin receiver typing, and consume qualified member-call overrides by
leaf name in the parallel lsp_idx fast path.
Distilled from #684 with one design change: the new unique-Class.method
tail-match fallbacks in lsp_resolve.h are gated to JVM callers
(cbm_pipeline_lsp_allow_tail_match, Java/Kotlin only). Tail-matching by
unique leaf is safe where class-per-file package semantics hold; in
other languages a single wrong-module coincidence would fabricate a
CALLS edge. The leaf-key lsp_idx lookup stays global: it aligns the
fast path with the build-side key that already used leaf names.
Tests: carried mixed Java/Kotlin regression (red on main, green here)
plus a gate guard asserting the tail fallbacks stay off for non-JVM
callers and on for JVM.
Co-authored-by: nguyentamdat <nguyentamdat@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The recursion detector flagged any call whose short name matched the
enclosing def as self-recursion, so super().save() inside save,
axios.get() inside get, and console.error() inside error were all
false positives feeding is_recursive, recursion_in_loop and
unguarded_recursion.
Add is_self_receiver() and AND it into the short-name comparison: a
qualified callee only counts when its whole receiver chain (everything
before the last '.') names the same object — self/this/cls/@self, or
the enclosing def's own receiver identifier parsed from
CBMDefinition.receiver (Go: s in 'func (s *Store) save()'). Bare names
keep the prior behavior. Matching the full chain keeps self.obj.recur()
(a field's same-named method) out, and the dynamic receiver whitelist
keeps Go s.save() detected.
Distilled from PR #699 with two corrections: the receiver chain is
matched via the last dot instead of the first segment (self.obj.recur
was still a false positive), and the enclosing-receiver whitelist so
Go method self-recursion is not lost.
Closes#599
Co-authored-by: Gen Li <lg320531124@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distills PR #770 (graph-buffer dedup key) and completes the edge-uniqueness
contract it left partial: the contract lives in THREE places, and changing
only the buffer ships databases that violate their own UNIQUE constraint
(PRAGMA integrity_check: non-unique entry in sqlite_autoindex_edges_1).
A single 'import { A, B } from ./lib' produced ONE IMPORTS edge: the graph
buffer dedups edges on (source_id, target_id, type) and merge-replaces
properties on collision, so the second symbol silently overwrote the first.
pass_calls.c, pass_usages.c, pass_semantic.c and pass_lsp_cross.c parse one
local_name per IMPORTS edge for cross-file resolution, so the dropped
symbol's calls failed to resolve too — not just "who imports X" queries.
Changes, kept in sync across all three sites:
- graph_buffer.c (from #770): make_edge_key() folds local_name into the
dedup key for IMPORTS edges only; all three key call-sites updated.
Hardened beyond #770: EDGE_KEY_BUF bumped to 256 and oversized
local_names are re-keyed with an FNV-1a hash of the full name instead
of being silently truncated (two long names sharing a prefix must not
collide back into one edge).
- store.c: edges gains local_name_gen, a VIRTUAL generated column
(IMPORTS -> coalesce(json_extract(properties,'$.local_name'),''),
else '' — NOT NULL because NULLs never conflict in a UNIQUE index);
uniqueness widened to UNIQUE(source_id, target_id, type,
local_name_gen) and the insert upsert's conflict target matches.
init_schema probes pre-#768 DBs (no local_name_gen) and fails the
open: SQLite cannot ALTER a table constraint in place, and an
unopenable DB already takes the existing repair path — full index
deletes + rebuilds, artifact import refuses and falls back to a
reindex. Read-only query opens skip init_schema and keep working.
- sqlite_writer.c/.h: dump DDL matches the widened schema; the hand-built
sqlite_autoindex_edges_1 comparator and entry builder include the
local_name column; CBMDumpEdge carries local_name extracted via real
JSON parsing (yyjson) so index entries match json_extract's unescaped
values exactly, per the idx_edges_url_path precedent.
- artifact.h: CBM_ARTIFACT_SCHEMA_VERSION 1 -> 2 so old binaries refuse
artifacts carrying the widened schema (their 3-column conflict target
can no longer prepare against it).
Tests (reproduce-first, all red on the unfixed code):
- gbuf tier: multi-symbol import -> 2 IMPORTS edges; long-local_name
truncation guard.
- store tier: distinct local_name coexists as 2 rows, same local_name
still upserts, non-IMPORTS dedup unchanged.
- writer tier: dumped DB passes integrity_check with 2 sibling imports
and exposes matching local_name_gen values.
- end-to-end: TS fixture through the real pipeline -> 2 queryable
IMPORTS edges AND integrity_check ok; with only the buffer half of
the fix this test still fails on integrity_check, proving the schema
half is required.
Closes#768.
Co-authored-by: Alexandros Pappas <11921291+apappas1129@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
py_eval_expr_type evaluated a call node's attribute receiver twice (once
in the container special-case, once in the general attribute path), so an
N-link chained call cost O(2^N) evaluations — real-world ~65-link builder
chains hung indexing for hours (#710). The evaluator also recursed once
per expression nesting level with no depth guard, so a pathologically deep
expression overflowed the native stack (#720).
Three guard layers, mirroring c_eval_expr_type's design in c_lsp.c:
1. Memoization: py_eval_expr_type is now a wrapper around the renamed
py_eval_expr_type_uncached. Results are cached per node in an
open-addressing table on PyLSPContext (arena-allocated, cold per file),
keyed by node identity (TSNode.id) — NOT the start byte, since every
leftmost descendant of a chain shares it and byte-keyed entries alias
distinct nodes into silently wrong resolutions. Entries carry a scope
generation; every scope bind/restore bumps it (O(1) whole-cache flush),
so re-evaluation under changed bindings (lambda re-walks, isinstance
narrowing) behaves exactly as before. Inserts are load-bounded (<75%)
and rejected when growth fails, so a full table can never spin.
2. Depth cap: PY_LSP_MAX_EVAL_DEPTH 256 (same as C_EVAL_DEPTH_LIMIT);
past the cap the evaluator returns unknown instead of overflowing the
stack. Truncated results — including any ancestor of a truncation —
are never cached, so a depth- or budget-limited `unknown` cannot
poison post-truncation reuse.
3. Step budget: PY_EVAL_MAX_STEPS_PER_FILE 10000 (mirrors
C_EVAL_MAX_STEPS_PER_FILE); pathological files degrade to unknown
instead of stalling indexing. Cache hits don't consume budget.
Distills PR #732 (memoization; fixed the start-byte cache key and the
missing staleness handling) and PR #758 (depth cap; raised 64 -> 256 to
match the sibling LSPs, added the never-cache-truncated ordering), and
adds the step budget plus reproduce-first tests: a 40-link chain that
must resolve its final link, a heterogeneous-receiver chain that fails
under byte keying, a 30k-deep expression crash guard, and a budget
degradation guard.
Fixes#710. Fixes#720.
Co-authored-by: LA-10 <lashuha@uwaterloo.com>
Co-authored-by: Dustin Persek <dustin.persek@protonmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Shane McCarron