CodeQL cpp/comparison-with-wider-type (4 high-severity alerts, #69-#72) flagged
two loops in the vendored ObjectScript scanner (objectscript_common.h, in both
the _routine and _udl grammar copies) where a uint8_t counter is compared
against an int length: the reverse_marker scan and the html_marker_buffer
reversal. Both lengths are hard-bounded by MARKER_BUFFER_MAX_LEN (30), so the
uint8_t counter can never wrap and the flagged infinite loop is unreachable in
practice — but widening the counters to int removes the pattern and clears the
gate. Provably behavior-neutral (int holds every uint8_t value; length <= 30).
Recorded as a local modification in the vendored grammar MANIFEST so it is
re-applied on the next re-vendor.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A function-like macro invocation whose argument is a type token — e.g.
ALLOC(int, n) — makes tree-sitter's C/C++ grammar emit an ERROR node (it parses
the type in expression position), which cbm_collect_error_regions recorded as a
parse_partial coverage gap. But the macro is #defined in the same file and the
call sits inside an already-extracted function body, so nothing is actually
missing from the graph — it's a benign call the grammar can't parse without the
preprocessor (#1071, systematic across allocation-macro-style codebases).
Subtract an error region only when it both (a) contains an invocation of a
file-defined function-like macro and (b) is fully enclosed by an extracted
Function/Method body. Condition (b) keeps a TOP-LEVEL macro invocation that
expands to a definition still flagged (#949: the generated def isn't in the
original span), and fails safe — a benign top-level call stays flagged rather
than a real gap being hidden.
Tests: a type-arg macro call inside a function no longer reports parse_partial;
a real in-body syntax error still does; #949/#946 preserved.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Rust cfg-gated definitions append predicate text to qualified names, but cbm_registry_add at src/pipeline/registry.c:495 indexed the suffixed tail instead of the supplied source name. Local twins therefore disappeared from has_permission lookup and an unrelated cross-module symbol became a false unique_name target.
Add cbm_rust_cfg_qualified_name at internal/cbm/extract_defs.c:1912 and use it from both definition extraction and compute_func_qn at internal/cbm/extract_unified.c:442. The helper scans only adjacent Rust attribute spans, allocates no duplicate decorator array, and makes call-source QNs match cfg-qualified Function nodes instead of falling back to File nodes. Index registry candidates by the supplied semantic name while retaining exact QNs as identities.
Guard the resolver collision in tests/test_registry.c:891 and cfg caller identity in tests/repro/repro_issue495.c:212. Verified with ASan/UBSan: registry 60/60, extraction 276/276, parallel 36/36, incremental 162/162 with 15,328-node and 89,030-edge incremental/full parity; scripts/check-source-safety.sh passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Teach parse_kotlin_imports in internal/cbm/extract_imports.c to accept direct named import nodes as well as import_list/import_header layouts.
Handle delegation_specifiers containers and identifier nodes in internal/cbm/extract_defs.c, and recover declarations only inside parser-marked ERROR ranges while skipping comments and literals. The recovery remains linear in the affected source bytes, uses MAX_BASES, and introduces no shared state.
Validated with ASan/UBSan suites: extraction_inheritance 9/9, extraction_imports 22/22, grammar_imports 1/1, lang_contract 35/35, edge_imports 59/59, edge_structural 32/32, matrix_new_constructs 61/61, grammar_regression 1/1, and grammar_labels 2/2.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Two review items applied maintainer-side to save a roundtrip:
1. Restore both vendored ObjectScript scanner.c files to the pristine
upstream bytes (a formatter pass had rewritten indentation/braces).
Vendored grammar files stay byte-for-byte upstream except for the
include repoint documented in MANIFEST.md - style churn breaks
upstream diffability on re-vendor.
2. The Studio Export XML sniff in detect_file_language used a raw
fopen on a repository path; cbm_fopen keeps it working on
non-ASCII Windows paths (project rule, same class as #996).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Index Java and Kotlin definitions with an internal <jvm-default-package> key in src/pipeline/pass_lsp_cross.c so cbm_pxc_filter_defs_for_file() retains sibling-file declarations when no package header exists. The hash table rejects empty keys, which previously removed Algo.kt::maxOf before cross-file resolution.
Initialize cbm_run_kotlin_lsp_cross() with the file module QN so default-package caller identities match extracted graph QNs such as <project>.Main.run. In pxc_append_results(), index resolutions by caller and callee short name and let an equal-confidence project-wide result replace the earlier per-file guess; this changes kotlin.comparisons.maxOf to <project>.Algo.maxOf without adding O(n^2) scans.
ASan/UBSan validation: lsp_resolution_probe passed 83/83, kotlin_lsp passed 79/79, java_lsp passed 379/379, and parallel passed 36/36. The focused lrp_kotlin_s7_generic debug run resolved one CALLS edge from <project>.Main.run to <project>.Algo.maxOf. git diff --check passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Merge upstream/main at 2469ecc3 into integrate-upstream-split-search-tools while retaining branch-specific incremental indexing, MCP API consolidation, and graph extraction behavior.
The staged tree adds upstream's Mojo and Perl grammar support, issue reproduction suites, Graph UI changes, Windows smoke coverage, TOON graph output, subprocess supervision, and Laravel facade route handling. Conflict resolutions in src/pipeline/httplink.c and pass_httplinks.c retain dynamic growth and IMPORTS edge identity; scripts/smoke-test.sh and scripts/soak-test.sh retain hermetic cache isolation.
Validation against the merge plus the follow-up working-tree reconciliation: ASan/UBSan MCP suite 216/216 passed. Follow-up semantic and concurrency fixes remain intentionally unstaged for the next focused commit.
JAX-RS splits a route across two annotations: the verb comes from a bare
@GET/@POST/... and the path from a sibling @Path. The annotation scan
returned on the first mapping annotation, so the @GET matched, defaulted
the path to "/" and the sibling @Path was never read. Class-level @Path
was never recognized as a prefix either, because a lone @Path carries no
verb and the scan reported no route.
scan_route_annotations() now collects the whole annotation set (mapping
verb + optional inline path, and a separate JAX-RS @Path), then:
- method-level: verb required; path = inline mapping path, else @Path,
else "/"
- class-level prefix: inline mapping path, else @Path
Also recognizes bare @HEAD/@OPTIONS verbs.
Spring behavior is unchanged (mapping annotations keep their inline
path); covered by the existing test suite plus a new regression test.
Signed-off-by: Charles Queiroz <fcqueiroz@liquibase.com>
Keep the live database visible until a synced temporary generation can
replace it. Preserve destination sidecars on replacement failure and use
UTF-8-safe Windows file operations.
Signed-off-by: Nikita Bige <wargloom@gmail.com>
Real Laravel apps register routes through the Illuminate facade, whose
class lives in vendor/ and is never indexed - and for that style the
graph minted ZERO Route nodes, flat or grouped. Three stacked defects:
1. The callee of Route::get(...) (scoped_call_expression) was
extracted as bare 'get' via generic field resolution, so the
empty-resolution route fallback ('::get' suffix table) could never
engage. The extractor now emits the qualified 'Route::get' form,
gated to the literal Route scope AND a route-method table match so
Cache::get / Config::get keep bare names and can never mint junk
(guarded by an inverse test; the path gate additionally requires a
leading '/').
2. The sequential pass had no ROUTE_REG empty-resolution fallback at
all (its comment claimed parity with the parallel path, which has
one) - unresolvable route registrations were dropped entirely.
Added, mirroring the parallel callee_suffix fallback.
3. Nothing composed enclosing prefix()->group() chains (same class as
Spring's #734). At extraction - the only place the AST still exists
- each route call inside group closures now walks its ancestors:
every enclosing closure that is the argument of ->group(...)
contributes the prefix('...') found on that group call's receiver
chain, outer-to-inner for nested groups; middleware/name/domain
chain links are skipped.
The existing handles_laravel_php test kept passing throughout because
its fixture defines an in-tree Route class, resolving by QN - which is
exactly how this class stayed unseen.
Closes#952
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_writer_open used a raw ANSI-CP fopen for the hand-rolled SQLite
writer - the one file-creating call on the dump chain without a
UTF-8-to-wide conversion. On a non-ASCII Windows profile
(C:\Users\Kovacs Janos) extract and resolve succeeded but the dump
failed with pipeline.err phase=dump: fopen could not create graph.db
under the profile-derived cache dir while cbm_mkdir_p (already wide)
had happily created the directory around it. Routed through cbm_fopen,
matching #700 (pass readers) and #973 (canonicalization); POSIX is
byte-transparent.
New Windows guard test_non_ascii_cache_dump.py indexes an ASCII repo
into a non-ASCII CBM_CACHE_DIR and requires a query_graph readback
with Function count > 0 - proving the DB was written to and reopened
from the non-ASCII cache, not merely that no error surfaced. The
existing non-ASCII guard only varied the repo path against an ASCII
cache, which is why this class went unseen.
The USERPROFILE-sourced env encoding question (Layer 2 - _environ is
ANSI-CP, not UTF-8) is deliberately not bundled; it needs verification
on a real Windows box and will be filed separately if the guard alone
does not settle it.
Closes#996
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A function whose body braces are split across #ifdef/#else branches
(one open brace per branch, a single shared close) parses with an
ERROR region on the raw source - both branches are present at once, so
the braces are unbalanced - and the defs walk silently dropped it
while its callers stayed as unresolved names (cbm_path_within_root,
handle_process_kill).
The preprocessed second pass (simplecpp) already exists for macro-
hidden CALLS; it evaluates the conditionals, so one branch is chosen
and the expanded tree parses clean, with same-file token lines aligned
to the original. Recover definitions from it, adopting only defs that
(a) intersect a raw-tree ERROR region, (b) have their name visible on
the corresponding original source line (rejects header-inlined defs
whose physical expanded lines alias unrelated raw lines when
compile_commands include paths resolve), and (c) are absent by
qualified name from the raw pass.
Verified on the real instances: handle_process_kill (580-647) and
cbm_path_within_root (5070-5085) both index with correct line ranges.
Closes#961
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
from m import f as g; g() produced no CALLS edge at all. Two stacked
defects: (1) py_lsp_bind_imports keyed from-style detection on the
local name matching the module-path tail - an alias never matches, so
g was bound MODULE(m.f) and the call missed; (2) even with the binding
fixed, the pass join compares the bare tail of the resolved QN (f)
against the textual callee (g) and dropped the resolution.
Fix: (1) a dotted module path whose tail differs from the local name
is an aliased binding - bind NAMED(qn) and let phase 6 upgrade it to
the registered function/class (from-import alias) or module (import
a.b as z); (2) when the identifier-callee NAMED branch resolves a name
whose textual form differs from the QN tail, emit strategy
lsp_import_alias with the textual name stashed in reason, and gate
that strategy in the join's reason list - the exact pattern already
used by lsp_dict_dispatch and lsp_func_ptr. The parallel path's hash
index misses the alias key and falls through to the same gated linear
scan.
The contract test pins all four import forms (aliased from-import,
plain from-import, single-segment module alias, dotted module alias)
so the flip from MODULE to NAMED binding cannot regress the module-
alias attribute chains.
Closes#988
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
const foo = require('./foo') emitted a Variable definition for the
binding, so calls to foo() resolved to that same-module Variable (an
import alias, not a definition) instead of following the recorded
import to the exported function - which stayed at zero inbound CALLS
and surfaced as a disconnected Variable node. ESM import bindings never
materialize as Variables; the identifier-bound require form now behaves
identically: extract_js_vars skips the Variable def exactly when the
declarator value is a require() call with a string-literal path (the
form extract_imports records with local_name = the identifier), letting
resolution fall through to the import map. Destructured requires keep
their Variable defs since the import row only records the module.
Closes#871
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Add ObjectScript (InterSystems IRIS / Caché) as a supported language,
covering the UDL class format (.cls), MAC/INT routines (.mac/.int/.rtn),
include/macro files (.inc), and IRIS Studio Export XML.
Definition extraction (extract_defs.c): Class, Method, ClassMethod,
Property, Parameter, Index, Trigger (with body text), XData, Storage,
and Query members as graph nodes; base classes from the Extends clause.
Call dispatch resolution (extract_calls.c) — four ObjectScript patterns
that are structurally invisible to text search:
1. ##class(Pkg.Class).Method() explicit cross-class call
2. ..Method() relative-dot self-call (the dominant
intra-class form; large impact on
CALLS completeness)
3. $$$Macro macro expansion via a per-project
table built from .inc files
4. type inference from %New/%OpenId + declared return types
Ensemble production topology (pass_ensemble_routing.c): EnsembleItem
nodes per production component and ROUTES_TO edges resolved from
ProductionDefinition XData, plus WorkMgr .Queue("##class(X).method")
dispatch — all parsed statically at index time, no live IRIS required.
Language detection (language.c): .mac/.int/.rtn map to ObjectScript
routine directly; .cls (shared with Apex) and .inc (shared with BitBake)
are disambiguated by content, defaulting to the existing language on any
doubt so neither Apex nor BitBake detection regresses.
The two new per-project tables (macros, return types) are threaded
through a new internal cbm_extract_file_ex() so the public
cbm_extract_file() signature is unchanged.
The tree-sitter grammars for ObjectScript UDL and routine are vendored
in internal/cbm/vendored/grammars/objectscript_{udl,routine}/ from
https://github.com/intersystems/tree-sitter-objectscript (MIT, ABI 15).
Refs #462
Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
Three code paths install a brand-new DB file where a previous
generation lived without removing the destination's -wal/-shm:
the pipeline page writer (cbm_writer_open), artifact import (which
cleaned the TMP file's sidecars but not the destination's), and
cbm_store_dump_to_file (whose comment wrongly asserted stale sidecars
'become stale and get recreated'). SQLite validates a WAL against the
sidecar's own header/checksums — not the main file — so a crashed
session's leftover WAL is replayed ON TOP of the freshly installed
file at the next open, splicing old-generation pages into it.
New shared helper cbm_remove_db_sidecars (compat_fs, wide-path safe on
Windows) called at all three install points.
Reproduce-first: dump_install_ignores_stale_wal_sidecar follows the
issue's recipe (hot-copy a live WAL, clean close, restore the copy,
install generation 2, reopen). RED on the unguarded code — the replay
clobbered the fresh generation entirely (fresh row unqueryable);
GREEN with the cleanup. Non-vacuous: asserts the hot WAL is non-empty
before the copy.
Closes#897
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
tree-sitter's GLR parser merges ambiguous parse-stack heads recursively in
stack_node_add_link (internal/cbm/vendored/ts_runtime/src/stack.c), once per
nesting level. On pathologically nested input whose grammar is ambiguous
(Perl's paren-optional calls, f(f(f(...)))), that recursion overflows the
native stack during the parse — a small ~1 MB Windows thread stack, and even
an 8 MB POSIX stack at extreme depth — before any extraction runs. Grammars
that are unambiguous here (Java, C++) never trigger the recursive merge.
Cap the recursive merge at CBM_TS_STACK_MERGE_MAX_DEPTH (512) via a
depth-tracked inner worker; the public stack_node_add_link is a thin wrapper
seeding depth 0, so every existing call site is unchanged. Past the cap the
ambiguity is left on the GLR stack instead of merged — a valid parse, never a
wrong one. 512 frames is well within a 1 MB stack while far exceeding any
realistic source nesting.
Add perl_glr_deep_parse_recursion_capped: a crash-isolating regression that
parses deep ambiguous Perl directly (a forked child on POSIX, in-process on
Windows), bypassing cbm_extract_file's CBM_PERL_MAX_PARSE_NESTING pre-parse
guard so the vendored cap itself is exercised rather than the guard.
Narrow vendored-runtime change; no other parsing behavior is affected. The
higher-level Perl pre-parse guard (issue #913 context) stays in place and can
be retired in a follow-up once this cap is proven in the field.
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
Two O(n^2) patterns in the Perl LSP, the class the ARM
extract_wide_flat_file_is_linear guard flags:
- Every child-iteration loop used indexed ts_node_child(node, i), which is
O(i) per call and O(n^2) over a wide flat node. Route them all through a new
perl_collect_children() helper mirroring wd_collect_children() in
extract_defs.c: a forward TSTreeCursor collect for wide nodes (>= 64
children), NULL (cheap indexed access) for the small common case.
- perl_attach_methods rebuilt each package's whole NULL-terminated method
array on every sub via perl_type_add_method, O(subs^2) on a single-package
file. Replace with a collect -> qsort-by-package -> build-once batch
(O(S log S)); source order within a package is preserved so first-defined
still wins on redefinition.
No behavior change: all perllsp_* resolution tests and
lsp_perl_deep_expression_no_crash pass; check-lsp-originality.sh reports clean.
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
A deeply nested Perl call chain f(f(f(...))) overflowed the stack while
parsing on small-stack platforms: tree-sitter's GLR parser recurses once
per nesting level in stack_node_add_link (vendored ts_runtime/src/stack.c)
merging the ambiguous parse-stack heads that Perl's f(...) grammar produces.
The overflow happens during ts_parser_parse, before any LSP walk runs, so
the CBM_LSP_PERL_MAX_WALK_DEPTH guards can never fire. This crashed
test-windows (CLANG64) with an AddressSanitizer stack-overflow and hung
test-unix (ubuntu-24.04-arm) to a 4h timeout on lsp_perl_deep_expression_no_crash.
Add cbm_source_nesting_exceeds(): an O(n) early-exit bracket-depth scan, and
skip extraction (clean has_error, zero edges) for Perl input whose nesting
exceeds CBM_PERL_MAX_PARSE_NESTING (128) before it reaches tree-sitter.
Scoped to Perl because only its ambiguous call grammar drives the GLR
recursion to the nesting depth; C/Java/Python parse the same shape with a
single stack head. 128 is ~10x above any real Perl nesting and well below
the depth where the parser overflows a 1MB stack.
Correct the lsp_perl_deep_expression_no_crash comment to describe the actual
mechanism (parser recursion, not the LSP walk).
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
Optional, non-blocking follow-ups from the #461 review:
- perl_eval_function_call_type: split qualified static calls on the LAST
"::" (mirroring perl_resolve_function_call) so return-type chaining on
Foo::Bar::sub() resolves instead of yielding unknown.
- cbm_run_perl_lsp: finalize the type registry into a per-call scratch
arena (as php_lsp/java_lsp do) before resolution, so method/func/symbol
lookups use O(1) hash buckets instead of a per-file linear scan.
- perl_infer_self_type: match the `shift` invocant idiom at word
boundaries and accept `$_[0]`, so `my $self = shifty();` no longer
mis-binds the receiver (tightens the zero-edge guarantee).
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
perl_resolve_function_call split a package-qualified call name on the FIRST
"::" via strstr. For multi-level packages (Foo::Bar::sub) this yielded
pkg "Foo" and a sub name still containing "::" ("Bar::sub"), which never
resolved. The call then fell through to the generic bare-name fallback, which
matches on the bare sub name only — collapsing distinct packages' same-named
subs onto a single winner and orphaning the rest. Single-level calls
(Foo::sub) were unaffected, which is why the existing test missed it.
Split on the LAST "::" instead so the full package name reaches
perl_lookup_method, which resolves to the correct fully-qualified sub.
Adds perllsp_static_multilevel_package_call covering the multi-level form.
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
F1: dot the Exporter import target so seeded CPAN exported subs resolve.
perl_collect_qw_imports built colon-form targets (Scalar::Util::blessed)
but the stdlib registry keys curated CPAN subs in dotted form
(Scalar.Util.blessed) and lookup is exact-match. Wire in perl_pkg_to_dot
to dot the module portion and drop the now-unnecessary (void) cast.
F2: add a recursion-depth guard (CBM_LSP_PERL_MAX_WALK_DEPTH=512) to both
AST walkers (perl_resolve_calls_in_node, perl_pass1_scan) via a depth-
guarded wrapper + inner split, mirroring java_lsp's JAVA_LSP_MAX_WALK_DEPTH.
Past the cap a subtree is skipped (graceful degradation, no wrong edge),
preventing stack overflow on pathologically nested input.
F3: lock the shared last-"::"-segment normalization in lsp_resolve.h with a
direct regression test over cbm_pipeline_find_lsp_resolution: a qualified
static call still resolves AND the cross-namespace mis-attribution edge case
is bounded by caller-QN equality + the confidence floor.
F4: implement SUPER:: dispatch. Populate enclosing_parent_qn from the
enclosing package's first @ISA parent and resolve $self->SUPER::method() to
that parent's method (strategy perl_method_super). No known parent or
unresolved method emits no edge (zero-edge guarantee preserved).
Tests: perllsp_cpan_exported_function, perllsp_super_dispatch,
perllsp_super_no_parent_no_edge, lsp_perl_deep_expression_no_crash,
lsp_resolve_qualified_static_call_normalizes_colons,
lsp_resolve_misattribution_is_bounded.
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
perl_collect_isa_assignment relied on the assignment's `right` field, but
tree-sitter-perl flattens a parenthesized RHS (`our @ISA = ('Base')`) so the
`right` field points at the `(` token while the parent string literals are
sibling children of the assignment. The single form `@ISA = 'Base'` worked but
the common parenthesized form silently collected zero parents, so @ISA
inheritance never populated embedded_types and method dispatch could not walk
the MRO.
Scan every named child after the `=` instead of only the `right` field, covering
both `@ISA = 'Base'` and `@ISA = ('Base', 'Other')`. use parent / use base were
already handled via a separate path and are unaffected.
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
- cbm.c: include lsp/perl_lsp.h and dispatch CBM_LANG_PERL -> cbm_run_perl_lsp
- lsp_all.c: unity-include perl_lsp.c + generated/perl_stdlib_data.c
- Makefile.cbm: register TEST_PERL_LSP_SRCS and append to ALL_TEST_SRCS
- tests/test_perl_lsp.c: placeholder suite so the Makefile var resolves
(full suite + test_main.c registration land in plan 22-04)
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
Andrew Hundt