Reuse scoped constructor typing for direct new field initializers and
collect annotated constructor parameter properties as class fields.
Refs #514
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Copy inherited field-type pointer arrays into the writable registry arena
before Python refines an existing field. Keep shared type objects intact
and avoid retaining per-file scratch types in the sealed fallback.
Add deterministic shared-state and later-overlay coverage, retaining the
sequential Python field regression and adding a forced-parallel variant.
Refs #1277
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Preserve import kinds and resolve class imports through Composer prefixes
to the existing target class file, including autoload-dev mappings.
Refs #1186
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Carry declared instance-field types through extraction and result
compaction into the shared Python type registry so chained calls through
imported classes resolve their receiver types.
Refs #1277
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The vendored tree-sitter-c-sharp (88366631d598) predated upstream's C# 12
collection-expression support (tree-sitter-c-sharp #402, first released in
v0.23.4). A `[...]` literal could only be parsed as an
element_binding_expression, so `c ? [] : x`, wrapped `? []` ternaries and
both-branch collection conditionals had no valid parse. Error recovery then
ate the surrounding statement, the file was reported as parse_partial, and
nearby declarations could fall out of the graph.
Refresh the generated grammar bundle byte-for-byte from upstream tag v0.23.5
(cac6d5fb595f5811a076336682d5d595ac1c9e85). The ABI stays at 15, the MIT
LICENSE is byte-identical, and scanner.c is content-identical. The only new
node types are collection_expression, collection_element,
expression_element and spread_element. No node type is removed or renamed,
and the C# extractor and LSP depend on none of the old or new names.
MANIFEST.md and the vendored checksums are updated.
Add parse_coverage regressions for the reported variants plus a `? null`
control. A second test checks that calls inside a collection expression
(element and spread) are still extracted. A malformed-source guard checks
that broken C# is still flagged.
On dotnet/runtime (60108ba6), 4 files are no longer parse_partial and no
file becomes newly partial. Utf8JsonWriterTests.cs gets its class back, and
216 orphaned Functions are re-attached as Methods. Two files that were
already partial for unrelated unsupported syntax (the preview `closed`
modifier, `async` as an identifier) recover differently and lose 46 and
10 definitions.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Overloads that share a base QN (Java/C#/C++/Kotlin/Swift/Scala/ObjC) merge
into one graph node today, and that node collects the CALLS of every
overload. The agreed cure is one signature-qualified callable identity for
all overloading languages:
callable_qn := base_qn [<tparams>] (params) [cvref]
This change lands the plumbing with every language still at
CBM_CALLABLE_ID_NONE, so the graph is byte-identical; each language is
switched on later in its own change (with its index-format bump).
- internal/cbm/callable_sig.{c,h}: the per-language mode side table
(NONE|TYPED|LABELED_TYPED|LABELED|ARITY; kept out of CBMLangSpec like
cbm_string_dispatch_suffixes, since ~160 positional rows would otherwise
all need the member under -Wmissing-field-initializers), the shared
builder cbm_callable_sig[_mode] for the def and the call-scope QN, and
the inverse cbm_qn_callable_base_len[_named]. Suffix contract: no '.',
'::' or '->'; whitespace-normalized; comments/annotations/attributes
dropped; qualified types keep their last segment; '->' spelled '=>';
variadic '[]' (Java) / '~'; C++ top-level cv dropped, arrays decay,
cvref kept; <= 200 bytes, longer ones keep whole leading entries and end
in '#<fnv1a64 of the full suffix>)'. The builder is iterative (cursor
walk, bounded declarator loop) and self-checks against the inverse.
- The inverse is anchored to the node's bare name wherever the name is
known: real QNs already end in ')' without being callables (a Java field
whose name is its declarator `X = f()`, make `$(VAR)` targets), and only
the anchor makes the inverse exact on them.
- Leaf splitters strip a suffix first (no-op today): registry by-name key
(indexed by the bare name for a suffixed callable), cbm_lsp_bare_segment,
cbm_pipeline_qn_class_method_tail, pxc_last_component, pxc_qn_leaf,
go_index_visit_node, collect_prod_defs, sg_qn_prefix_len.
- New unused fields: CBMDefinition.qn_sig_off, CBMResolvedCall.callee_sig
(relocated by result compaction), CBMRegisteredFunc.sig, CBMLSPDef.sig.
The lsp_surface codec is unchanged (no format bump in this change).
- get_code_snippet and trace_path gain base-QN tiers
(cbm_store_find_nodes_by_qn_base) that find nothing until suffixes exist.
- store: cbm_store_find_nodes_by_qn_suffix built its LIKE pattern in a
512-byte buffer (a long suffix was truncated into a pattern that matched
nothing) and did not escape '_'/'%' (`my_func` also matched `myXfunc`).
The pattern is now heap-built, escaped, and matched with ESCAPE '\'.
Proof: full sorted node+edge+lsp_surface dumps are byte-identical before
and after on 7 corpora (this repo, elasticsearch libs, Exposed, dotnet
System.Linq, System.Collections, coreclr/gc, a C corpus).
Refs #2061
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A method call on an instance of a class defined in another file produced
no CALLS edge, while the identical code in the class's own file resolved
via lsp_ts_method. The type-aware tier typed the receiver wrongly in two
places, so the method lookup in the shared TS registry missed and the
call fell to method_not_in_registry:
1. `new C()` always qualified a bare constructor name against the
CURRENT module. `const t = new ImportedClass(); t.m()` and
`new ImportedClass().m()` were typed as a phantom
`<this module>.ImportedClass`. The constructor type now follows TS
scoping: a class declared in this module, else the class an
unambiguous import binding names, else the old spelling.
2. ts_import_symbol_qn (and the copy of its heuristic in
resolve_type_with_imports) treated an import value ending in
".<name>" as an already-qualified symbol QN. For the dominant TS
convention of a file named after its class (`Foo.ts` exports
`class Foo`, module `x.Foo`, class `x.Foo.Foo`), the receiver was
typed as the MODULE, for `new`, annotations and parameters alike.
The registry now decides the ambiguous case: a registered
`module.name` wins, else the value is the symbol QN as before.
Both are resolved through the existing shared cross-file registry; no
name-based fallback is reintroduced. Cost is one or two hashed lookups
on the finalized overlay / sealed shared registry, per `new` expression
or per ambiguous import spelling during the walk; nothing is looked up
during registration and nothing is written to the shared registry.
typeorm (3,593 .ts files, shallow f279fd13): lsp_ts_method 2,551 ->
3,130, all 579 new edges cross-file (cross-file lsp_ts_method 696 ->
1,275), 0 edges lost, other strategies unchanged, index time
unchanged. 20 sampled new edges checked against source: 20 true
positives.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
NestJS controllers produced zero Route nodes. Two gaps in the decorator
route walk (extract_route_from_decorators):
- It only inspected decorator children of kind `call` (Python). A
TypeScript decorator's call is a `call_expression`, so @Get(':id') was
never examined.
- The path helpers only accept literals that start with "/". Nest writes
segments without one (@Controller('users'), @Get(':id')), and the
class-level prefix was composed only for Java/Kotlin (Spring/JAX-RS).
TS/TSX class methods now read the Nest verb decorators (@Get, @Post, @Put,
@Delete, @Patch, @Head, @Options, @All -> ANY) and compose them with the
class's @Controller / @JsonController prefix. The prefix can be a string,
an array (first entry), or the options object's `path`. Paths are rooted
at "/". A verb decorator outside a controller class forms no route. A
non-literal path argument (a constant) also forms none, so no route is
invented. Route and HANDLES are reused as they are: no new label.
Not covered: the app-level setGlobalPrefix() and enableVersioning() in
main.ts are cross-file and are not applied.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A plain '&' inside JSX -- a URL query in an attribute
(href="https://fonts.googleapis.com/css2?family=Inter&display=swap"),
or text such as <p>Tom & Jerry</p> -- produced a tree-sitter ERROR node
in .js, .jsx and .tsx files. The file was flagged parse_partial in the
coverage view, and the error recovery turned words from the URL
(create-next-app -> create/next/app) into identifiers that resolved to
unrelated symbols as false USAGE edges.
Root cause is upstream tree-sitter-javascript, which the tsx dialect
builds on: unescaped_{double,single}_jsx_string_fragment is
/([^"&]|&[^#A-Za-z])+/, so '&' is only accepted before a space/digit
or as a complete html_character_reference (& / &). jsx_text is
scanned by the external scanner, which stops at every '&', and nothing
in _jsx_child accepted a '&' that is not a reference. So '&' + letter in
a string, and any lone '&' in JSX text, had no valid token.
Fix: javascript and tsx become self-maintained forks of the pins we
already vendored (javascript @ 58404d8cf191; typescript tsx @
75b3874edb2d on its javascript base @ 3a837b6f3658), following the
arkts model. One patch (tools/tree-sitter-javascript/patches/
jsx-lone-ampersand.patch) is applied to both javascript bases:
- a one-character '&' token at lexical precedence -1, aliased to the
existing string_fragment (in strings) and jsx_text (in text); the
lower precedence keeps & / & lexing as html_character_reference
- the string-fragment regexes exclude a '&' directly before the closing
quote, so "a&" no longer runs past the end of the string
Node types are unchanged (same type/named set). The external scanners
stay upstream bytes. Parsers regenerated with tree-sitter-cli 0.25.10
(ABI 15; tsx moves from ABI 14, and its tree_sitter/ headers now match
arkts). Without the patch both fork dirs regenerate upstream exactly.
The upstream corpus passes unchanged: javascript 116/116; tsx gives the
same results as unpatched upstream.
Proof on vercel/next.js examples/ (canary 725705bd, 1,722 js/jsx/tsx
files): parse_partial 92 -> 47 (tsx 33 -> 0, jsx 2 -> 0, js 12 -> 2; the
two left are Babel do-expressions and Flow types), no newly flagged
file, code-graph nodes 14,739 -> 14,739, edges 24,773 -> 24,657: the
115 USAGE edges removed were all false cross-example edges from URL
fragments, plus a 4 SIMILAR_TO / +3 SEMANTICALLY_RELATED reshuffle
from the corrected ASTs.
Registered like arkts: MANIFEST.md fork rows + notes (ABI tally
recounted), audit FORKS map, THIRD_PARTY.md, vendored checksums.
LICENSEs stay upstream's MIT byte-identical.
Tests: parse_coverage gains a .js/.jsx/.tsx matrix (double/single-quoted
attribute, '&' before the quote, '&' + letter/space/digit in text, plus
&/& controls) and a guard that broken JSX is still flagged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A method called on the object a static factory returns got no CALLS
edge when the class lives in another file:
`SecdManager::getInstance()->m()`, `Registry::instance().m()`, and
`auto w = Widget::create(); w->m()` with std::unique_ptr or
std::shared_ptr. The same code in a single file already resolved.
Why: the cross-file C++ LSP resolved the scope of `Class::member` only
as the bare QN or under the caller's own module. A class declared in a
header and defined out of line in a .cpp is registered under that
file's module (`<proj>.app.src.SecdManager.SecdManager`), so the static
call missed and fell to a name heuristic (unique_name 0.38,
qualified_suffix). Its return type stayed unknown, and every receiver
built from it dropped the method call. A name heuristic on the same
site could also bind a same-named function elsewhere: luabridge's
`UserdataSharedHelper<C, false>::push(...)` was bound to the free
function `luabridge::push`.
Fix: once every existing lookup of `Class::member` has missed, resolve
the scope to exactly one registered class (bare, caller module, then a
type whose QN ends in `.<scope>` at a segment boundary) and look the
member up on that class: direct, aliases, bases, never re-anchored onto
a same-named class elsewhere. Two classes with that name name no single
target, so the call stays unresolved. The same resolution types the
call, so the declared return type (pointer, reference, unique_ptr,
shared_ptr) reaches chained and auto receivers. C/C++ LSP only; no
call that resolved before changes.
Proof, qBittorrent e8f7c18b (628 C/C++ files), same machine, fresh
index with a binary built from origin/main and one with this change:
- CALLS 17,401 -> 17,889: +489 edges (487 lsp_type_dispatch, 2
lsp_scoped) and -1 edge, the suffix_match `luabridge::push` false
positive above.
- 295 existing edges now come from the LSP with the same target (268
qualified_suffix, 27 unique_name -> lsp_scoped).
- All 489 added edges checked against their source line: 155 are
`Class::f()->m()` or `Class::m()` on that line, 334 go through a
receiver initialised from `Class::instance()` in the same function
(Preferences, SettingsStorage, UIThemeManager, Logger). None binds a
different class.
- `X::instance()->m()` sites whose enclosing function has the edge:
4/461 -> 304/461. Namespaced scopes (`RSS::Session::instance()`) stay
unresolved: forward declarations mint extra Class nodes, so the
scope does not name one class.
Tests (pipeline suite; each runs the sequential and the parallel path):
one per shape (pointer, reference, unique_ptr, shared_ptr), plus
controls: a class with the same method names is never bound, and an
undefined receiver type and a class name two libraries declare both
stay unresolved. The fixture asserts that every node it names exists,
so a missing node cannot pass as a missing edge. RED 3/3 on main,
green with the fix, RED again on revert; dropping the uniqueness rule
turns the ambiguity control RED.
The Python half of #1153, a method call on a module-level instance
(`manager = StackManager(); manager.status()`), is fixed by the held
Python LSP change 53653c15, not here.
Refs #1153
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
With `import Axios from "axios"`, a common convention in TypeScript
projects, every `Axios.get/post/put/delete(url)` client call was indexed as
a route registration. The result was a `CALLS {via: route_registration}`
edge to a Route node instead of an HTTP_CALLS edge. The client's request
paths therefore posed as server routes, and no HTTP call site existed to
match a handler. microsoft/Game-Control-Puzzle-Event-Administration-Tools
has all 69 of its client calls in this shape.
Root cause: service-pattern matching is case-sensitive by design, so that
"requests" cannot match "Requests" or "myrequests". The HTTP library
table knew only the lowercase id `axios`. `Axios.get` therefore matched no
library, on either resolver. Resolution came back empty, because an
external package is never indexed. The call then fell through to the
callee-suffix fallback, which treats any `<x>.get/post/...(path)` as a route
registration.
Fix: add `Axios` next to `axios` in the HTTP library table. The table
already pairs case variants of one library (`Dio`/`dio`,
`Guzzle`/`guzzle`, `OkHttp`/`okhttp3`). The existing identifier-boundary
rule applies unchanged:
- `Axios.get` and `AxiosInstance.get` (axios's own instance type) match;
- a lowercase continuation such as `Axiosphere` does not.
Express/Koa/Fastify registrations such as `router.get(path, handler)` are
untouched.
Proof (release binary, macOS arm64,
microsoft/Game-Control-Puzzle-Event-Administration-Tools @4fe23196,
before -> after):
- route_registration CALLS 69 -> 0; all 69 were `Axios.*` calls in
client/.
- All 69 reappear as typed HTTP_CALLS from the same caller to the same
Route QN (`__route__PUT__/api/admin/users/{}`), and no typed HTTP_CALLS
edge is lost.
- HTTP_CALLS 4 -> 142. Besides the 69 typed calls, 69 are the method-less
arg_url twins that fix/http-typed-call-no-url-twin removes, and 4 are the
existing `isOnPage` arg_url edges.
- CALLS 868 -> 799.
- SEMANTICALLY_RELATED 83 -> 174, deterministic across two runs of each
binary. The semantic pass tokenizes CALLS-neighbour names, and the fake
registration edges had fed each service function's URL path into its
vector. Without them, the 69 `Axios` wrapper functions read as related.
Five sampled calls were checked against the client source, including
admin/users/service.ts:44 `Axios.put(`/api/admin/users/${participantId}/
${eventInstanceId}`)` -> `__route__PUT__/api/admin/users/{}/{}`.
On the pending ASP.NET route stack (fix/aspnet-route-case-inrepo, 8e734ee5),
together with the arg_url twin fix, the reclassified calls reach their C#
handlers:
- typed HTTP_CALLS 0 -> 69;
- DATA_FLOWS 0 -> 66, and HANDLES 94 -> 106 (+12 case_fold);
- sampled links match their controller source, for example
`Axios.put(`/api/admin/users/${participantId}`)` ->
`[Route("api/admin/users")] [HttpPut("{participantId}")] UpdateUserInfo`;
- the 3 calls left unlinked carry a query string in the URL literal.
Tests: RED 3/3 before the fix, green after, RED again on revert.
- pipeline_capitalized_axios_import_is_http_client (parallel resolver) and
its _sequential twin: `import Axios from "axios"` with Axios.get, .post
and .delete gives 0 route registrations and exactly one typed
HTTP_CALLS per call, while an Express `router.get` registration in the
same project stays a registration.
- infrascan service-pattern tables: `Axios.get`, `Axios.post` and
`AxiosInstance.get` classify as HTTP, and `geo.Axiosphere.map` stays
NONE.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Previous behavior: cbm_scope_lookup(), cbm_scope_contains(), and cbm_scope_lookup_callable() called an out-of-line complete-binding scan after the Python lookup consolidation.
What changed:
- internal/cbm/lsp/scope.h defines cbm_scope_lookup_binding() as the one static-inline scope-chain scan.
- internal/cbm/lsp/scope.c keeps the type, presence, and callable wrappers on that shared implementation without another function boundary.
Why: Python still reads type and callable identity in one traversal, while existing language resolvers retain the direct hot path used before the consolidation.
Verification:
- CBM_ONLY_SUITE=scope build/c/test-runner: 33 passed
- CBM_ONLY_SUITE=cs_lsp build/c/test-runner: 62 passed
- CBM_ONLY_SUITE=py_lsp build/c/test-runner: 125 passed
- neutral 11-pair extraction matrix: output parity; C# latency -2.97% vs first parent and -1.64% vs upstream; retired instructions +0.29% vs first parent and -1.01% vs upstream; Python scale latency -4.25% and instructions -2.81% vs first parent
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
(cherry picked from commit 150c88e1c8)
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Add cbm_scope_lookup_binding so callers can read lexical presence, type, and callable identity in one traversal. Route Python expression, call, and callable-reference resolution through that complete binding and use the existing checked bind APIs for allocation-failure detection.
The neutral 11-pair production matrix kept exact output parity. Python scale latency was 4.33% below merge parent 07255cfe and 61.01% below upstream parent 0c64de93; incremental instructions were 3.09% and 54.07% lower. Focused ASan/UBSan scope and Python suites passed 33 and 125 tests.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
(cherry picked from commit 4d3a103047)
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
extraction::extract_spill_round_trip_keeps_every_field and
parallel::parallel_spill_mode_builds_the_same_graph failed on any machine
with less than 10 GB free: cbm_result_spill_open refuses to spill onto a
nearly full disk (mem.spill.refused reason=low_disk), so the store never
opened. Three pipeline tests took the same refused-store path there and
passed without spilling at all (backpressure_futile_nap_disengages,
over_budget_after_futility_fails_whole_and_preserves_db,
spill_resolves_namespace_imports_like_memory). A verdict must be a pure
function of code, test, platform and seed, not of how full the disk is.
Add a free-space pin to result_spill that exists only in test-seam
builds (CBM_ENABLE_TEST_SEAMS). Production still asks the disk, and its
10 GB floor and half-of-free cap are unchanged. The test runner pins
64 GiB at the top of main, before worker re-exec dispatch, so every
suite and re-exec'd index worker sees the same disk. A new test,
extraction::extract_spill_refuses_below_the_free_disk_floor, pins the
guard itself: one byte under the floor refuses, the floor opens.
Verified with cbm_fs_free_bytes temporarily forced to 9 GB: both tests
RED 3/3 before, green 3/3 after, RED again with the runner pin removed;
the refusal test goes RED with the production guard disabled.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Three iterative AST walkers kept their pending nodes in fixed-size stacks
and silently stopped pushing children once full:
- cbm_compute_complexity and count_branching_iter (helpers.c,
BRANCHING_STACK_CAP 4096) -> cyclomatic, cognitive, loop and access
metrics
- extract_body_ident_tokens (extract_defs.c, BT_STACK 512) -> the
body-token sample that feeds semantic search / SEMANTICALLY_RELATED
Children are pushed last-to-first, so a full stack dropped a wide body's
FIRST children. The metrics undercounted, and the token sampler read a
mid-body window instead of the first identifiers. That let a work cap decide
graph content.
Real input: tree-sitter-llvm's generated ts_lex (a 4654-case switch)
reported cyclomatic 11133 / cognitive 29303. It really has 8776 if + 4654
case + 1 default + 1 switch = 13432 branches, which is what it reports now
(cognitive 35639). On the Linux kernel no function reaches the old 4096 cap,
so complexity is byte-identical for every function. Three functions with
bodies over 512 pending nodes now sample their tokens from the start of the
body (for example rimob_panel_on), which moves 11 SEMANTICALLY_RELATED edges.
Index time is unchanged within noise (main 214.4/207.9 s, fix 197.6/192.2 s).
Fix: cbm_walk_stack_reserve()/cbm_walk_stack_release() in helpers.c. Each
walker keeps a small on-stack first chunk (512 frames) and spills to the heap
through the memory core (cbm_alloc/cbm_realloc/cbm_free, class EXTRACT) only
for deep or wide trees. Traversal order is unchanged. If the stack cannot
grow, the failure is logged ("extract.walk_stack_alloc_failed") and the
walker returns a documented sentinel instead of a truncated result:
CBM_WALK_METRIC_UNAVAILABLE (-1) for the metrics, or no token sample.
The body-token sampler's caps of 128 identifiers and a 2048-byte buffer are
its sampling contract and stay. The 512-entry stack was never part of that
contract, because it broke the "first identifiers in source order" property.
Tests (extraction suite): a C function with 5000 top-level if/while
statements must report exact cyclomatic/cognitive/loop counts through both
the extractor and cbm_count_branching, and a 600-call body must sample from
f0 on. All three are RED on main (4095 / 4095 / sample starting at f91) and
GREEN with the fix.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
When the receiver is iris.cls("<literal>"), the class-qualified name now
replaces the primary call's callee instead of being pushed as a second
call. Keeping the bare name let it weak-match an unrelated same-named
method.
Tests assert exactly one Run call per iris.cls() site, and the pipeline
fixture adds two unrelated Run definitions that must receive no edges.
Refs #1260
Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
walk_defs keeps its pending frames in a growable stack that was bounded by
an 8M-frame ceiling (env CBM_WALK_DEFS_MAX). At the ceiling wd_push logged
"extract.walk_defs_capped" once and then stopped pushing. Children are
pushed last-to-first, so a file wider than the ceiling lost its FIRST
definitions: a work cap decided graph content.
Fix: no ceiling. The stack doubles on demand, heap frames now come from the
memory core (cbm_realloc/cbm_free, class EXTRACT) instead of safe_realloc and
raw free, and the doubling is overflow-guarded (int capacity and size_t
byte count). The only thing that can stop the walk is an allocation failure,
and that is loud: logged at error level as "extract.walk_stack_alloc_failed"
(walker=walk_defs, pending, path) -- the same event the complexity walkers
use -- and the file's result is marked has_error with
"definitions walk: stack allocation failed", which the definitions passes
already report as a per-file extract error. The arena-backed path
(ctx->scratch) gets the same failure handling. Traversal order is unchanged.
CBM_WALK_DEFS_MAX is removed. It was read only in wd_stack_max(); it was not
documented anywhere (docs, --help, config, tests), so nothing else changes.
extract_defs.c loses its raw free() call, so the memory-core baseline for
the file drops from 5 to 4.
Test (extraction suite): walk_defs_wide_file_extracts_every_definition
extracts a C file with 1000 top-level functions with CBM_WALK_DEFS_MAX=256
set, and asserts every wd0..wd999 definition is present. RED on main
(extract.walk_defs_capped limit=256; wd0 missing), GREEN with the fix, RED
again with the production change reverted. The extraction (379), complexity
(5), ast_profile (10) and semantic (35) suites pass.
Real input: Linux kernel, index-only, fresh cache and runtime dirs, same
machine. main and fix both produce 8,530,336 nodes and 15,764,128 edges,
SEMANTICALLY_RELATED 607 each, and the full node set (label, QN, lines) and
edge set (type, source QN, target QN) hash identically -- no kernel file
approaches the old ceiling. Index wall time 210.0 s (main) vs 194.4 s (fix),
within run-to-run noise.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
When a call resolved to a GraphQL client, emit_graphql_edge named the
Route after the whole first string argument, stripping only a leading
"query " or "mutation " (the mutation strip was also one byte short, so
names kept a leading space). An anonymous `query { ... }` therefore became
one long Route, and even a named operation carried its variables and
selection set in the name ("GetUser($id: ID!) { user(id: $id) { name } }").
Every distinct operation text minted its own Route node, and names were
cut off at 256 bytes.
Add cbm_service_pattern_graphql_operation(), a small GraphQL document
scanner. It skips comments, strings, block strings, `${...}` template
interpolations and leading fragment definitions, and reports the first
operation's type and name without copying any operation text. Routes are
now keyed as follows:
named operation __graphql__<Name> name <Name>
anonymous query / mutation / __graphql__<type>__anonymous name (anonymous <type>)
subscription, incl. the `{ ... }` query shorthand
no GraphQL document __graphql__operation__anonymous name (anonymous operation)
(URL, bare word, no string argument)
Keys are deterministic and bounded by the GraphQL Name grammar. The
GRAPHQL_CALLS edge keeps "operation" (now the Route name) and gains
"operation_type". The old fallback to the callee name ("client.query")
is gone: a callee is not an operation.
The sequential path (pass_calls.c) emits no GRAPHQL_CALLS at all, so it
had no copy of this logic to fix.
Proof on graphql-python/gql (374a70df, Python GraphQL client, index-only):
GraphQL Routes 170 -> 16, longest QN 266 -> 34 bytes, 106 -> 0 Routes
carrying operation text. GRAPHQL_CALLS 2643 -> 1158 with the same 984
distinct callers: edges are one per (caller, operation) pair, so calls
that used to hit distinct text-keyed Routes now share their operation's
Route.
Tests (edge_types_probe): graphql_route_keyed_by_operation_name_issue598
asserts the exact GraphQL Route set (QN + name) on the parallel path.
graphql_operation_identity_parser_issue598 covers braces inside comments,
strings and interpolations, fragments before the operation, anonymous
operations with variables, name truncation and non-document arguments.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
In IRIS embedded Python, iris.cls("Pkg.X").Method(...) names its target
class in the receiver's string literal. There was no receiver path for it
next to the classMethod* string dispatch, so after the weak-member-call
suppression in b6a22843 these calls produced no CALLS edge at all.
When a Python call's receiver is iris.cls() with a plain quoted literal as
its first argument, also emit "<class-literal>.<method>" as a callee, the
same way classMethodValue already does. Non-literal class arguments
(variables, f-strings) name nothing and get no class-qualified callee.
Also add invokeClassMethod to the Python dispatch suffixes. It takes the
same (class_name, method_name, *args) arguments as classMethodValue.
Tests: the #1260 repro as a pipeline test (two classes sharing Run, a
nonexistent class, an unrelated receiver, classMethodValue and
invokeClassMethod), plus extraction tests for the synthesized callee,
non-literal and f-string arguments, and invokeClassMethod.
Fixes#1260
Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
The comment on CBM_EXTRACT_BUDGET called it a "maximum byte budget", but
every call site passes it as cbm_extract_file*()'s timeout_micros, which
cbm.c converts to nanoseconds and uses as a per-thread CPU-time deadline
(5 s) with a 12x wall-clock ceiling as a backstop for a stuck parse. No
code path uses the value with the wrong unit; only the documentation was
wrong.
Correct the constant's comment, and bring the timeout_micros description
in cbm.h up to date (CPU time + wall backstop, not a plain timeout) and
back onto cbm_extract_file(), from which later additions had separated it.
No behavior change; the budget value is unchanged.
Reported-by: BobbieBarker
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A generated API client (Orval and similar) passes its request as a config
object to a local fetch wrapper:
http<Customer>({url: `/api/customers/${id}`, method: 'GET', signal})
The template literal itself already canonicalizes to "/api/customers/{}"
(#1006). What was lost is the object wrapper: the call's arguments only
carried a value for bare strings, template literals, constants and URL
builders, so the object argument had no URL. The arg-url heuristic
therefore minted no HTTP_CALLS edge, and cross-repo matching had nothing to
join against the server's GET /api/customers/{id} Route. The same held for
HTTP clients called with a config object (`axios({url})`): no URL, no edge.
The extractor now reads the `url` property of an object argument in
JS/TS/TSX/ArkTS -- both as that argument's value (for the arg-url
heuristic) and as the call's URL (for HTTP-client classification). Only
the `url` key counts. Route registrations (Fastify `route({url, handler})`)
and axios `getUri(config)`, which formats a URL and sends nothing, are
excluded.
orval-labs/orval @ 08d7fcf1 (parallel resolver): HTTP_CALLS 1383 -> 1533
(+150 arg_url edges from the generated mutators such as customInstance,
responseType and customClient; 0 lost), Route 302 -> 308, CALLS unchanged
(19190). Without the getUri exclusion there would have been 546 URL-builder edges.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A 4.7 MB binary Godot resource (piko_walk_mesh.res) never finished
indexing: `.res` maps to ReScript, so the resource was parsed as ReScript
source and the definitions pass spun. Three defects, each fixed:
1. Quadratic usage walk (the hang). The ReScript `let` binding check
(is_first_named_part_of, shared with Julia, Typst, Elm, PureScript
and Nickel) climbed from every named leaf with ts_node_parent, which
descends from the root and scans each level's children. A binary
file parses into a flat error tree (~170k root children per MB), so
each climb cost O(root children) and the file O(n^2): 1 MB spent
46 s in cbm_extract_unified, 4.7 MB never returned. The check now
climbs the unified walk's own cursor (O(1) per hop), like the other
occurrence classifiers; the node-based climb stays only for callers
outside the walk. 1 MB: 46 s -> 0.03 s; 4.7 MB: 2.2 s end to end.
2. Quadratic lexing in error recovery. tree-sitter retries the lexer at
every byte of an unparseable stretch with every external token
valid; the ReScript scanner's template-string loop then ran to the
next backtick/$/backslash/NUL (end of file if none) and discarded
the result. That cost O(n^2) inside lexing, where the parse budget's
progress callback never runs, so such files were dropped by the
5 s CPU budget instead of parsed -- plain text too (100 KB of
non-ReScript text: 15 s -> 0.18 s). Local scanner patch: skip the
template branch when NEWLINE is also valid, which only the ERROR
state allows. Valid files parse to identical trees. Recorded in
vendored/grammars/MANIFEST.md.
3. Binary content parsed as source. `.res` is also Godot's binary
resource format and the Windows compiled-resource format. Following
the .m/.cls/.frm/.inc/.cfc precedent, cbm_disambiguate_res reads the
first 4 KB and reports a file holding a NUL byte as unsupported;
ReScript text never contains one.
Tests (deterministic work counters, no clocks):
- extraction::extract_rescript_let_bindings_use_the_walk_cursor --
zero root-descending parent hops at 128 and 1024 statements
(before: 515 / 4099).
- complexity::complexity_rescript_error_recovery_lexing_is_linear --
lexer bytes pulled through a chunked TSInput, 4 KB vs 8 KB, ratio
must be linear (before 3.99, after 2.00).
- language::lang_res_binary_resource_unsupported and
lang_res_rescript_stays_rescript.
Fixes#2176
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A mysqldump-style .sql file is a few CREATE TABLEs followed by megabytes of
`INSERT ... VALUES (..),(..),...` rows. Tree-sitter built a full tree for
every row (~70 bytes of tree per source byte; 0.26 s/MB with MySQL \'
escapes, which the grammar does not know and error-recovers from) until the
5 s parse budget ran out, and the whole file, tables included, was skipped
as "parse timeout". The reporter's database/sql/world.sql was dropped this
way. The rows contribute nothing to the graph.
A linear, quote- and comment-aware pre-scan (internal/cbm/sql_values.c)
finds INSERT/REPLACE ... VALUES lists and excludes every tuple after the
first that consists only of literals: strings (standard '' and MySQL \'
escapes, X'..'/B'..'/N'.., _charset introducers), numbers, NULL, TRUE,
FALSE, DEFAULT. A tuple holding a subquery, a function call, an identifier
or a quoted identifier stays in the parse, and so does one whose string
spans a raw newline. The excluded bytes go to ts_parser_set_included_ranges
as the complement, so byte offsets and line/column positions of all kept
text are unchanged. The ranges are cleared right after the parse because
the parser is thread-local and reused. This is a syntax rule, not a work
cap: the same file always gives the same ranges, and the parse budget is
unchanged.
Measured on generated mysqldump-shaped dumps, release build, same machine,
index worker's own numbers (base = origin/main):
dump extraction pipeline peak RSS graph
8 MB std '' 1310 -> 23 ms 1431 -> 139 ms 625 -> 37 MB identical
8 MB MySQL \' 2712 -> 24 ms 2827 -> 146 ms 474 -> 37 MB identical
32 MB MySQL \' 5145 -> 82 ms 5467 -> 396 ms 1132 -> 62 MB timeout -> indexed
59 MB MySQL \' 5130 -> 138 ms 5667 -> 674 ms 1157 -> 89 MB timeout -> indexed
"identical" = every stored node and edge, Module end line included. For the
32/59 MB dumps the base run skipped the file as "parse timeout" (Module +
3 Tables + their DEFINES/USAGE missing). The fixed graph matches the 8 MB
graph except for the Module's end line.
With standard escapes the extraction and the stored graph are identical
to the full parse. With MySQL \' escapes the full parse's error recovery
swallows rows next to a misread escape, a subquery row among them, and the
cut parse keeps those rows. So it loses nothing the full parse found
outside the dropped rows and recovers what the full parse lost. The only
things the full parse extracted INSIDE the dropped rows were junk usages
(DEFAULT and _binary read as identifiers, fragments of misread strings)
that name nothing.
Tests (parse_coverage, pipeline; a CBM_TEST_SQL_FULL_PARSE_ON seam turns the
exclusion off for one file so the same source can be compared both ways):
- scanner cases: comments, strings, dollar quotes, ON DUPLICATE KEY
UPDATE, VALUES outside INSERT, unclosed tuples, exact kept positions
- extraction equals the full parse on a small dump (standard escapes);
nothing lost outside dropped rows (MySQL escapes)
- the subquery and function-call rows are still parsed
- the tree does not grow when the row count doubles (a count, not a clock)
- a 25 MB MySQL dump under the production budget is indexed, not
"parse timeout" (asserts the outcome, not the time)
- stored nodes and edges of a pipeline index equal the full parse's
Refs #1735
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_subtract_macro_invocation_regions asked "is this region a call to a
file-defined macro?" before "is it inside a function?". The first question
(cbm_span_is_macro_invocation) walks the source from byte 0 to find the
region's lines, so every file with parse-error regions paid
regions x file bytes, whatever its language and whether or not it has a
single function. On a multi-MB SQL dump that is up to 256 walks over the
whole file.
Both predicates are pure, so asking the cheap one first changes no answer:
a region no function encloses never touches the source, and the regions
that do share one cbm_line_offsets table built on first use. The table
gives exactly the span the walk found; the walk stays as the fallback when
the table cannot be allocated.
#1071 behaviour is unchanged: same regions, same verdicts. Indexing this
repository's src/ before and after gives identical graphs (7,339 nodes,
28,924 edges) and the same 46 parse_partial files with byte-identical
ranges.
Tests count the bytes read, never time them (test-seam counter
cbm_test_macro_line_scan_bytes):
- 40 top-level junk regions: 39,870 bytes read -> 0, still 40 regions
- 60 in-body real errors: 85,725 bytes read -> 2,801 (one table = file
size), still 60 regions
Refs #1735
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Rust LSP types `root: &Path` and dispatches `root.join(..)` to
std.path.Path.join, a registered seed method with no graph node. Both
call resolvers treated "LSP row without a gbuf target" like "no LSP row"
and fell back to the textual registry, which bound the call to the only
project method named `join` (EvidenceTier::join via unique_name). Every
std/seeded-crate call whose leaf name a project method shares was
fabricated this way.
cbm_pipeline_rust_external_target() (lsp_resolve.h) recognises a Rust row
whose strategy emits a registered target (method/trait/deref/bound
dispatch, direct, ufcs, constructor) and whose QN lies outside the
project prefix. Project defs are registered under project-prefixed QNs, so
such a target is a std/core/alloc or seeded-crate symbol by construction.
pass_calls.c and pass_parallel.c skip the registry fallback for those rows
(in lockstep), and the parallel route-method fallback no longer turns a
skipped `map.get(k)` into a source->source self-call. Project-prefixed
rows, unknown-receiver rows and every other language are unchanged.
The call-expression receiver (`root.to_path_buf().join(..)`) needed type
facts: Path::join/to_path_buf/with_extension/with_file_name now return
PathBuf, PathBuf::as_path returns Path, and PathBuf derefs to Path.
Measured on BurntSushi/ripgrep @ 3fce3b5 (fresh index, two runs each,
byte-identical): CALLS 5371 -> 5014; 357 removed (307 suffix_match,
45 unique_name, 5 field_type_hint), 0 added, no lsp_* edge touched.
Removed samples are all std receivers (String::new -> State.new,
msg.contains -> ByteSet.contains, alts.push on a Vec -> TSeq.push).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The bulk writer builds each interior level greedily: it fills a page with
separator cells until the next one no longer fits, uses that child as the
page's right child, and starts the next page after it. When a page filled
up exactly at children[child_count - 2], only the last child was left, so
the next (and last) page of that level was written with ZERO cells and
just a right-child pointer.
SQLite rejects a non-root page with nCell < 1 as SQLITE_CORRUPT the moment
a cursor descends into it (getAndInitPage). Because that page is always the
last one of its level, the failing descent is the rightmost path -- exactly
what the coverage-shadow wipe takes: "DELETE ... WHERE project = ?1" with
"<project>::missed", a key that sorts after every row of "<project>". The
publish then failed with "coverage_shadow_wipe: database disk image is
malformed". It needs an index with more than one interior page on a level
(large graphs or long keys) AND the page-fill boundary landing on the one
bad position, which is why it looked scale-dependent and repo-specific;
the same DBs also fail PRAGMA integrity_check.
When a page fills with exactly one child left, give its last cell back, so
the next page holds one cell plus the last child as its right child. The
removed bytes are zeroed, keeping the output deterministic.
Test: sqlite_writer::sw_interior_level_never_ends_with_empty_page writes
2..120 rows with 12000-byte names (one and two interior levels) and runs
the wipe-shaped DELETE through idx_nodes_name; rows 31-35, 61-65 and 91-95
failed with SQLITE_CORRUPT before the fix. Real input: 41 generated
markdown repos with long headings indexed by v0.11.0 -> 5 databases fail
the "::missed" seek and integrity_check; with the fix 0 of 41.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two deterministic indexing-worker crashes reported on #1743.
C/C++ (the original LLVM report, "segfault at 9"): structured-binding
decomposition read rt->field_types[binding_idx] guarded only by
`rt->field_types[binding_idx] != NULL`. field_types[] is parallel to
field_names[] and NULL-terminated, so a binding list longer than the
struct's field list (ill-formed, but present in compiler test suites,
e.g. llvm clang/test/CXX/dcl.decl/dcl.decomp/p3.cpp) read past the
terminator into the next arena object. That object is a CBMType whose
first word is its kind (9), so the name was bound to the pointer 0x9 and
decltype(name) dereferenced it in c_resolve_name_to_type. All three
decomposition sites now go through c_struct_field_type_at(), which bounds
the index by field_names[] and yields NULL (unknown) past the end.
TypeScript (kilocode report on v0.11.0): lookup_method walked the
extends/implements chain recursively with no depth bound. The Effect idiom
`class Service extends Context.Service<Service, I>()(...)` gets its base
resolved to the class itself, so `Service.of(...)` (or a sibling class
whose base resolves to that Service) recursed until the resolver thread's
stack overflowed. lookup_method now has the same depth-guarded entry as
lookup_member_type (TS_LSP_MAX_MEMBER_DEPTH); past the cap the method is
simply not found.
Proof on real input (macOS arm64, fresh cache each run):
- llvm/llvm-project llvmorg-15.0.7: before, 5 worker SIGSEGVs at 0x9 in
c_resolve_name_to_type and 3 files quarantined; after, 0 crashes,
0 quarantined, 1,375,933 nodes / 4,257,770 edges.
- Kilo-Org/kilocode main: before, the resolve worker died every attempt
(SIGBUS, lookup_method recursion depth ~65k); after, indexed cleanly,
90,745 nodes / 440,654 edges.
Tests: c_lsp::clsp_structured_binding_more_names_than_fields and
ts_lsp::tslsp_nocrash_cyclic_extends_method_call (both crash RED without
the fix).
Fixes#1743
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel