Ten files changed on both sides; every conflict was a union of independent
additions:
- workflows (_lint.yml, _test.yml): keep the branch's lane `if:` and main's
pinned ubuntu-24.04 runner with its reason.
- internal/cbm/cbm.h: main's callable-identity and module-doc fields plus the
branch's configured-test fields.
- Makefile.cbm: both new test files; prod_lsp_all.o keeps main's
.build-config dependency and the branch's coverage flags.
- scripts/test.sh: main's setup-scripts contract becomes Step 0y3, ahead of
the branch's 0z series and the contracts-only exit.
- src/foundation/subprocess.c: main's cbm_posix_close_fds_from (it closes the
error descriptor too).
- src/mcp/mcp.c: main's repository default base for detect_changes, with the
early returns released through the branch's detect_release.
- src/pipeline/lsp_resolve.h, tests/test_main.c: both includes, suites and
start-up steps (main's git environment reset first).
- scripts/memory-core-baseline.txt: recounted on the merged tree and lowered
(mcp.c 773, config_toml_edit.c 73).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
One request pins the change (merge base .. HEAD), builds the graph and the
runnable test registry of exactly the HEAD snapshot, seeds the graph from
the change, walks what depends on it and answers which tests to run
(cbm.test_impact). Missing or doubtful evidence never narrows: it becomes a
run-all answer that names why.
Snapshot (classified ledger -> frozen graph -> registry):
- one shared language classifier for discovery and pinned blobs (raw bytes
on every platform); the classified ledger feeds the frozen build, which
indexes exactly the listed files and records partial parses as per-file
PARSE_GAP instead of refusing the graph;
- the pinned tree skips symlinks and submodule links (counted), as
discovery never indexes them;
- the test model scopes uncertainty to the suites of its file (they run
whole), reads line splices where they occur, treats a mid-file
`#ifndef X` + `#define X` as an ordinary condition, and no longer reads a
plain identifier named like a macro as an invocation. On this repository
the registry is now narrowable and matches a text census exactly
(8,365 cases, 141 runner suites).
Seeding and reach (port of the measured replay, rule B):
- hunk -> overlapping definitions, comment-only hunks dropped, pure
deletions -> enclosing definition, no-definition hunks -> whole file,
deleted names found by text, header changes propagated through the
include closure and the headers' macros (pastes included), file-level
escalation for unclassifiable header lines;
- inbound fixpoint walk over CALLS, ASYNC_CALLS, USAGE, CALL_REFERENCE,
READS (+ routes); cases, suites and the runner's entry points are sinks;
a reached runner main runs everything (RUNNER_REACHED);
- a parse gap whose text names a reached definition reaches the
definition around it (or its file);
- macro-registered tests run their suites whole; cases and suites map to
graph nodes by their start line.
The line patterns reproduce the reference regexes exactly on 1,383,850
lines of this repository and 37,167,926 kernel lines; on 24 replayed
changes the engine selects at least what the reference selected (13
identical, the rest only conservative supersets on the same graph).
Also:
- configured test declarations that cannot be mapped degrade their file
(no configured roles, a test_declarations diagnostic) instead of
aborting the whole index;
- the selector keeps other runners' suites out of the selection, and the
result codec judges the model by narrowability like the selector;
- a change to .codebase-memory.json runs everything (built-in rule);
- .codebase-memory.json with the c-cbm preset and this repository's rules.
Gate fixes (suites, lint-ci, Windows cross-compile):
- the request runs on its own thread with a default-size stack (daemon
tool calls run on 256 KiB threads, where the in-process build died);
- a request-supplied config_path is no longer followed by an unlink of an
uninitialized path buffer;
- the inventory's frame-local reader no longer stays reachable from the
caller's source after the filter call;
- FILE_ID_INFO needs _WIN32_WINNT >= 0x0602; the pinned-tree header pins
it before any system header (toolchains defaulting lower failed);
- memory-core ratchet: the win_utf8 converters gain core-backed
`_in(cls, ...)` forms over one implementation (the legacy names keep
the C library for unmigrated callers), the Git facts module uses them,
and the Windows spawn path releases everything through one failure
exit; pipeline, extraction and config allocations that this work
touched moved to the core in matched pairs; baselines tightened;
- the branch's C files formatted with the project's clang-format.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Preserve the current selector, frozen-input inventory, result codec,
coverage collection, runner filtering, and isolated Windows venue work.
Component-level regression proofs are recorded in the PR handover.
The live classified provider, public execution integration, artifact
admission, fresh oracle/shadow evidence, and full platform gates remain
incomplete. This checkpoint does not enable or claim verified narrowing.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Preserve the original staged graph and LSP work separately from the
smart-CI continuation. This is an unfinished checkpoint; final
integration and cross-platform verification remain outstanding.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reuse scoped constructor typing for direct new field initializers and
collect annotated constructor parameter properties as class fields.
Refs #514
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Copy inherited field-type pointer arrays into the writable registry arena
before Python refines an existing field. Keep shared type objects intact
and avoid retaining per-file scratch types in the sealed fallback.
Add deterministic shared-state and later-overlay coverage, retaining the
sequential Python field regression and adding a forced-parallel variant.
Refs #1277
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Preserve import kinds and resolve class imports through Composer prefixes
to the existing target class file, including autoload-dev mappings.
Refs #1186
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Carry declared instance-field types through extraction and result
compaction into the shared Python type registry so chained calls through
imported classes resolve their receiver types.
Refs #1277
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The previous commit bound a C member access only through the type of its
object. On this repository that removed 9% of the USAGE edges onto Fields; on
the Linux kernel it removed 44% (2,131,043 -> 1,184,323), and 359,236 of the
edges it dropped were onto a member name that exactly one Field in the whole
tree carries. Those were never a guess. The kernel has far more member
accesses the C LSP cannot type: designated initializers, structs reached
through transitively included headers, structs duplicated per architecture.
The defect in bare-name member resolution was the arbitrary pick among
SEVERAL same-named members, not binding by name as such. The rule is now:
1. the object is typed: bind the Field its type names;
2. the object is not typed: bind the member name only when exactly one
Field carries it (cbm_registry_unique_field_qn). Symbols of other labels
are not candidates for a member;
3. otherwise bind nothing.
It applies at all six sites: USAGE and READS/WRITES (sequential and
parallel) and a member call that resolved onto a Field.
In C++ mode the C LSP now also publishes the object's type when that type
declares no such field (`text.size()` on a std::string). The row names no
node, so the resolver sees "typed, no project Field meant" and the name
fallback does not apply. C keeps the fallback there, because a C aggregate's
nested anonymous members are not all registered on it.
Linux kernel, old binary vs this commit (same machine, same run):
wall 239 s -> 240 s; indexer CPU 1235 s -> 1249 s; peak RSS 15.8 -> 15.4 GB
USAGE -> Field 2,131,043 -> 1,571,103
kept 1,248,385; newly typed to the object's struct 322,718
no longer bound 834,502, of which onto a unique member name 1,523
WRITES -> Field 948,187 -> 782,846
CALLS onto a Field 92,440 -> 1,778
nodes: Class +1,658, Enum +782, Field +12,519, Variable +5,646
Redis (from scratch): Class 913 -> 1092, Field 3854 -> 4563, CALLS onto a
Field 455 -> 24.
The 834,502 edges no longer bound are member names several structs share,
on objects the LSP could not type: before, each was attached to one of the
same-named Fields by registry preference. Typing designated initializers and
following transitive includes would bind them correctly; neither is in this
change.
Tests: unique_field_qn_needs_exactly_one_field_of_that_name; the pipeline
member-access tests gain the untyped-object cases (a shared name binds
nothing, a unique one binds).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two defects that fed each other, fixed together because the first alone would
let the new Field nodes attract bare-name calls.
1. An anonymous aggregate under a typedef had no nodes.
`typedef struct { ... } X;` produced no Class node and none of its members a
Field node: the struct_specifier carries no name of its own, and
extract_class_def gave up on it. That is the dominant C spelling of a type
(37 of the 38 structs in internal/cbm/cbm.h). The enclosing type_definition
now names the aggregate through its declarator; only a plain type_identifier
declarator counts, since a pointer, array or function typedef names another
type. Applies to struct, union and enum specifiers in C and C++ (`.h` is
classified C++).
2. A C member access was resolved by its bare member name.
The member half of `a.b` / `a->b` reached the short-name registry as `b`,
which returned whichever Field (or function, or variable) was called `b`: on
this repository 9,150 USAGE and 2,810 WRITES edges onto Fields from C
sources, 3,976 of the USAGE edges onto a name several structs share.
The C LSP already types the object of a field_expression. It now publishes
the owner as a FIELD_REFERENCE row, one per (enclosing function, object type,
field): the pipeline edge is function -> Field, so a row per occurrence would
only add a record per member access. The usage and read/write resolvers
(sequential and parallel) join those rows by function and member name; a C
member name with no typed owner binds nothing.
Four things had to hold for the owner to be found:
- the cross-file registry folded Field defs into their owning type for Go
only, so every C/C++ struct from another file (every header) registered
with zero fields. The fold now covers C and C++;
- c_register_lsp_defs kept the first 63 fields of a type. It is sized from
the list now: a cap there silently dropped edges onto later fields;
- a struct is registered under the header that declares it while the use
site knows its short name. The owner type resolves as written, then
module-qualified, then by short name from this module, a directly included
header, or as the single candidate. Two same-named structs with no
evidence between them resolve to nothing;
- the parallel resolver ran the cross-file pass only for files with
unqualified call sites. A C member access or member call that still has no
owner now requests it too.
Calls landing on a Field follow the same rule (cbm_call_onto_field_policy):
dropped across languages (the #1928 reference rule; a Bash `command` or a
Python `sorted()` is not a C struct member), dropped for a bare C call
(`socket(...)`, a function-pointer parameter), and bound by owner for a C or
C++ member call. Bare C++ calls and every other language are unchanged.
Measured on this repository, old binary vs new:
C/H Class nodes 155 -> 1000
C/H Field nodes 1047 -> 5478
enumerators of formerly anonymous enums +911 Variable nodes
USAGE onto the new types, enums and constants +23k edges
USAGE -> Field from .c, owner type named in the file 58.1% -> 97.6%
USAGE onto a non-Field, name only used as a member 146 -> 0
WRITES -> Field from .c 2981 -> 6534
CALLS 159 removed, 3 added
index wall time unchanged within noise; peak footprint +2.8%
full macOS suite: 8309 passed, 0 failed, 10 skipped
Tests: extract_c_anonymous_typedef_aggregate_is_named_by_its_typedef,
pipeline_c_member_access_binds_field_by_object_type (+ _parallel),
call_onto_field_follows_language_and_shape,
c_member_access_binds_by_owner_only. Each was RED on the unchanged tree; the
pipeline tests are RED again with the call policy neutralized.
Not in this change: a function-pointer member declared inline
(`void (*f)(int);`) has no Field node; a `p->f()` call is not resolved (the
registry splits on `.` and `::` only); inline functions in `.h` files keep
the bare-name member binding; designated initializers are not typed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distilled from #1245 by Andrew Hundt (a slice of 0370d52f).
detect_url_in_args in the parallel resolver runs for every resolved call
with no callee gating. A first argument like "/new/file.txt" clears
normalize_url_arg (it has a second slash and no whitespace) and then
reaches cbm_service_pattern_is_http_route_literal, whose filesystem
extension list knew about config, key and socket files but not about
documents or logs. So
open_file("/new/file.txt")
read_file("/docs/guide.md")
write_file("/data/app.log")
each minted a Route node and an HTTP_CALLS edge -- the graph reported a
web endpoint where the program opens a file. Five extensions join the
list: .log .md .pdf .rst .txt, alphabetical order preserved.
Two things the tests are careful about, because the obvious fixture
would not have been RED. A bare Python builtin open(...) never resolves
to an indexed target, so it never reaches the URL detector at all; the
bug shows only for calls that resolve to a project symbol, which is what
the fixture defines. And /var/... is already rejected by the filesystem
ROOT list, so the fixture paths start with segments that list does not
know, leaving the extension guard as the only thing that can reject them.
RED before the fix:
infrascan_http_route_literal_guard_rejects_document_extensions
FAIL tests/test_infrascan.c:44:
ASSERT(!(cbm_service_pattern_is_http_route_literal("/new/file.txt", "open")))
pipeline_arg_url_rejects_document_file_paths
FAIL tests/test_pipeline.c:6652:
count_nodes_named(s, project, "/new/file.txt") == 1, expected 0 == 0
The pipeline test keeps a positive control above that line -- a genuine
requests.get("/api/items") still yields its Route node and HTTP_CALLS
edge -- so the guard cannot pass by rejecting everything.
GREEN after: infrascan 4 passed; pipeline 282 passed; extraction
edge_types_probe route_canon 420 passed.
Co-authored-by: Andrew Hundt <ATHundt@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The vendored tree-sitter-c-sharp (88366631d598) predated upstream's C# 12
collection-expression support (tree-sitter-c-sharp #402, first released in
v0.23.4). A `[...]` literal could only be parsed as an
element_binding_expression, so `c ? [] : x`, wrapped `? []` ternaries and
both-branch collection conditionals had no valid parse. Error recovery then
ate the surrounding statement, the file was reported as parse_partial, and
nearby declarations could fall out of the graph.
Refresh the generated grammar bundle byte-for-byte from upstream tag v0.23.5
(cac6d5fb595f5811a076336682d5d595ac1c9e85). The ABI stays at 15, the MIT
LICENSE is byte-identical, and scanner.c is content-identical. The only new
node types are collection_expression, collection_element,
expression_element and spread_element. No node type is removed or renamed,
and the C# extractor and LSP depend on none of the old or new names.
MANIFEST.md and the vendored checksums are updated.
Add parse_coverage regressions for the reported variants plus a `? null`
control. A second test checks that calls inside a collection expression
(element and spread) are still extracted. A malformed-source guard checks
that broken C# is still flagged.
On dotnet/runtime (60108ba6), 4 files are no longer parse_partial and no
file becomes newly partial. Utf8JsonWriterTests.cs gets its class back, and
216 orphaned Functions are re-attached as Methods. Two files that were
already partial for unrelated unsupported syntax (the preview `closed`
modifier, `async` as an identifier) recover differently and lose 46 and
10 definitions.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
extract_docstring() took only the single comment node directly before a
definition. Multi-line `//` and `///` docs therefore kept just their last
line (in C# usually `/// </summary>`); docs above attributes, exports and
wrapper declarations were lost; fields, variables and modules never got a
doc; Perl POD was never stored; and every doc was cut at 500 bytes.
Capture the full documentation comment instead:
- contiguous same-style comment runs; Go comment groups with go/ast
directive lines (`//go:`, `//line`, `//export`, ...) stripped
- docs above Rust attributes, JS/TS export statements and wrapper
declarations (arrow and function expressions, class fields,
assignments, ambient declarations) and C# top-level statements
- docs for fields, variables, constants and enum members, found with a
memoised linear walk
- Go package comments and Rust `//!` docs on the File node
- Perl POD sections that name a sub (`=head`/`=item <name>`)
- Rust impl methods, Kotlin KDoc folded into package_header or sibling
gaps, C `typedef struct/enum/union` declarations
- a doc comment wins over an intervening plain `//`; runs detached by a
blank line and runs made only of divider lines are not docs
- no 500-byte cut; the properties buffer grows with the docstring, so a
long doc no longer pushes other properties out
Measured on nine real-world corpora (Go, C#, Java, TypeScript, Rust, C,
PHP, Perl, Kotlin): full-doc capture Go 35% -> 99.999%, C# 10% -> 99.8%,
Java 74% -> 99.99%, Rust 7% -> 99.9%, C 67% -> 99.2%, Kotlin 70% ->
99.96%, TypeScript sources 42% -> 100%, named-sub Perl POD 0% -> 100%.
Node sets and all non-semantic edges are unchanged; index time stays
within run-to-run noise.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Overloads that share a base QN (Java/C#/C++/Kotlin/Swift/Scala/ObjC) merge
into one graph node today, and that node collects the CALLS of every
overload. The agreed cure is one signature-qualified callable identity for
all overloading languages:
callable_qn := base_qn [<tparams>] (params) [cvref]
This change lands the plumbing with every language still at
CBM_CALLABLE_ID_NONE, so the graph is byte-identical; each language is
switched on later in its own change (with its index-format bump).
- internal/cbm/callable_sig.{c,h}: the per-language mode side table
(NONE|TYPED|LABELED_TYPED|LABELED|ARITY; kept out of CBMLangSpec like
cbm_string_dispatch_suffixes, since ~160 positional rows would otherwise
all need the member under -Wmissing-field-initializers), the shared
builder cbm_callable_sig[_mode] for the def and the call-scope QN, and
the inverse cbm_qn_callable_base_len[_named]. Suffix contract: no '.',
'::' or '->'; whitespace-normalized; comments/annotations/attributes
dropped; qualified types keep their last segment; '->' spelled '=>';
variadic '[]' (Java) / '~'; C++ top-level cv dropped, arrays decay,
cvref kept; <= 200 bytes, longer ones keep whole leading entries and end
in '#<fnv1a64 of the full suffix>)'. The builder is iterative (cursor
walk, bounded declarator loop) and self-checks against the inverse.
- The inverse is anchored to the node's bare name wherever the name is
known: real QNs already end in ')' without being callables (a Java field
whose name is its declarator `X = f()`, make `$(VAR)` targets), and only
the anchor makes the inverse exact on them.
- Leaf splitters strip a suffix first (no-op today): registry by-name key
(indexed by the bare name for a suffixed callable), cbm_lsp_bare_segment,
cbm_pipeline_qn_class_method_tail, pxc_last_component, pxc_qn_leaf,
go_index_visit_node, collect_prod_defs, sg_qn_prefix_len.
- New unused fields: CBMDefinition.qn_sig_off, CBMResolvedCall.callee_sig
(relocated by result compaction), CBMRegisteredFunc.sig, CBMLSPDef.sig.
The lsp_surface codec is unchanged (no format bump in this change).
- get_code_snippet and trace_path gain base-QN tiers
(cbm_store_find_nodes_by_qn_base) that find nothing until suffixes exist.
- store: cbm_store_find_nodes_by_qn_suffix built its LIKE pattern in a
512-byte buffer (a long suffix was truncated into a pattern that matched
nothing) and did not escape '_'/'%' (`my_func` also matched `myXfunc`).
The pattern is now heap-built, escaped, and matched with ESCAPE '\'.
Proof: full sorted node+edge+lsp_surface dumps are byte-identical before
and after on 7 corpora (this repo, elasticsearch libs, Exposed, dotnet
System.Linq, System.Collections, coreclr/gc, a C corpus).
Refs #2061
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A method call on an instance of a class defined in another file produced
no CALLS edge, while the identical code in the class's own file resolved
via lsp_ts_method. The type-aware tier typed the receiver wrongly in two
places, so the method lookup in the shared TS registry missed and the
call fell to method_not_in_registry:
1. `new C()` always qualified a bare constructor name against the
CURRENT module. `const t = new ImportedClass(); t.m()` and
`new ImportedClass().m()` were typed as a phantom
`<this module>.ImportedClass`. The constructor type now follows TS
scoping: a class declared in this module, else the class an
unambiguous import binding names, else the old spelling.
2. ts_import_symbol_qn (and the copy of its heuristic in
resolve_type_with_imports) treated an import value ending in
".<name>" as an already-qualified symbol QN. For the dominant TS
convention of a file named after its class (`Foo.ts` exports
`class Foo`, module `x.Foo`, class `x.Foo.Foo`), the receiver was
typed as the MODULE, for `new`, annotations and parameters alike.
The registry now decides the ambiguous case: a registered
`module.name` wins, else the value is the symbol QN as before.
Both are resolved through the existing shared cross-file registry; no
name-based fallback is reintroduced. Cost is one or two hashed lookups
on the finalized overlay / sealed shared registry, per `new` expression
or per ambiguous import spelling during the walk; nothing is looked up
during registration and nothing is written to the shared registry.
typeorm (3,593 .ts files, shallow f279fd13): lsp_ts_method 2,551 ->
3,130, all 579 new edges cross-file (cross-file lsp_ts_method 696 ->
1,275), 0 edges lost, other strategies unchanged, index time
unchanged. 20 sampled new edges checked against source: 20 true
positives.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
NestJS controllers produced zero Route nodes. Two gaps in the decorator
route walk (extract_route_from_decorators):
- It only inspected decorator children of kind `call` (Python). A
TypeScript decorator's call is a `call_expression`, so @Get(':id') was
never examined.
- The path helpers only accept literals that start with "/". Nest writes
segments without one (@Controller('users'), @Get(':id')), and the
class-level prefix was composed only for Java/Kotlin (Spring/JAX-RS).
TS/TSX class methods now read the Nest verb decorators (@Get, @Post, @Put,
@Delete, @Patch, @Head, @Options, @All -> ANY) and compose them with the
class's @Controller / @JsonController prefix. The prefix can be a string,
an array (first entry), or the options object's `path`. Paths are rooted
at "/". A verb decorator outside a controller class forms no route. A
non-literal path argument (a constant) also forms none, so no route is
invented. Route and HANDLES are reused as they are: no new label.
Not covered: the app-level setGlobalPrefix() and enableVersioning() in
main.ts are cross-file and are not applied.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A plain '&' inside JSX -- a URL query in an attribute
(href="https://fonts.googleapis.com/css2?family=Inter&display=swap"),
or text such as <p>Tom & Jerry</p> -- produced a tree-sitter ERROR node
in .js, .jsx and .tsx files. The file was flagged parse_partial in the
coverage view, and the error recovery turned words from the URL
(create-next-app -> create/next/app) into identifiers that resolved to
unrelated symbols as false USAGE edges.
Root cause is upstream tree-sitter-javascript, which the tsx dialect
builds on: unescaped_{double,single}_jsx_string_fragment is
/([^"&]|&[^#A-Za-z])+/, so '&' is only accepted before a space/digit
or as a complete html_character_reference (& / &). jsx_text is
scanned by the external scanner, which stops at every '&', and nothing
in _jsx_child accepted a '&' that is not a reference. So '&' + letter in
a string, and any lone '&' in JSX text, had no valid token.
Fix: javascript and tsx become self-maintained forks of the pins we
already vendored (javascript @ 58404d8cf191; typescript tsx @
75b3874edb2d on its javascript base @ 3a837b6f3658), following the
arkts model. One patch (tools/tree-sitter-javascript/patches/
jsx-lone-ampersand.patch) is applied to both javascript bases:
- a one-character '&' token at lexical precedence -1, aliased to the
existing string_fragment (in strings) and jsx_text (in text); the
lower precedence keeps & / & lexing as html_character_reference
- the string-fragment regexes exclude a '&' directly before the closing
quote, so "a&" no longer runs past the end of the string
Node types are unchanged (same type/named set). The external scanners
stay upstream bytes. Parsers regenerated with tree-sitter-cli 0.25.10
(ABI 15; tsx moves from ABI 14, and its tree_sitter/ headers now match
arkts). Without the patch both fork dirs regenerate upstream exactly.
The upstream corpus passes unchanged: javascript 116/116; tsx gives the
same results as unpatched upstream.
Proof on vercel/next.js examples/ (canary 725705bd, 1,722 js/jsx/tsx
files): parse_partial 92 -> 47 (tsx 33 -> 0, jsx 2 -> 0, js 12 -> 2; the
two left are Babel do-expressions and Flow types), no newly flagged
file, code-graph nodes 14,739 -> 14,739, edges 24,773 -> 24,657: the
115 USAGE edges removed were all false cross-example edges from URL
fragments, plus a 4 SIMILAR_TO / +3 SEMANTICALLY_RELATED reshuffle
from the corrected ASTs.
Registered like arkts: MANIFEST.md fork rows + notes (ABI tally
recounted), audit FORKS map, THIRD_PARTY.md, vendored checksums.
LICENSEs stay upstream's MIT byte-identical.
Tests: parse_coverage gains a .js/.jsx/.tsx matrix (double/single-quoted
attribute, '&' before the quote, '&' + letter/space/digit in text, plus
&/& controls) and a guard that broken JSX is still flagged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A method called on the object a static factory returns got no CALLS
edge when the class lives in another file:
`SecdManager::getInstance()->m()`, `Registry::instance().m()`, and
`auto w = Widget::create(); w->m()` with std::unique_ptr or
std::shared_ptr. The same code in a single file already resolved.
Why: the cross-file C++ LSP resolved the scope of `Class::member` only
as the bare QN or under the caller's own module. A class declared in a
header and defined out of line in a .cpp is registered under that
file's module (`<proj>.app.src.SecdManager.SecdManager`), so the static
call missed and fell to a name heuristic (unique_name 0.38,
qualified_suffix). Its return type stayed unknown, and every receiver
built from it dropped the method call. A name heuristic on the same
site could also bind a same-named function elsewhere: luabridge's
`UserdataSharedHelper<C, false>::push(...)` was bound to the free
function `luabridge::push`.
Fix: once every existing lookup of `Class::member` has missed, resolve
the scope to exactly one registered class (bare, caller module, then a
type whose QN ends in `.<scope>` at a segment boundary) and look the
member up on that class: direct, aliases, bases, never re-anchored onto
a same-named class elsewhere. Two classes with that name name no single
target, so the call stays unresolved. The same resolution types the
call, so the declared return type (pointer, reference, unique_ptr,
shared_ptr) reaches chained and auto receivers. C/C++ LSP only; no
call that resolved before changes.
Proof, qBittorrent e8f7c18b (628 C/C++ files), same machine, fresh
index with a binary built from origin/main and one with this change:
- CALLS 17,401 -> 17,889: +489 edges (487 lsp_type_dispatch, 2
lsp_scoped) and -1 edge, the suffix_match `luabridge::push` false
positive above.
- 295 existing edges now come from the LSP with the same target (268
qualified_suffix, 27 unique_name -> lsp_scoped).
- All 489 added edges checked against their source line: 155 are
`Class::f()->m()` or `Class::m()` on that line, 334 go through a
receiver initialised from `Class::instance()` in the same function
(Preferences, SettingsStorage, UIThemeManager, Logger). None binds a
different class.
- `X::instance()->m()` sites whose enclosing function has the edge:
4/461 -> 304/461. Namespaced scopes (`RSS::Session::instance()`) stay
unresolved: forward declarations mint extra Class nodes, so the
scope does not name one class.
Tests (pipeline suite; each runs the sequential and the parallel path):
one per shape (pointer, reference, unique_ptr, shared_ptr), plus
controls: a class with the same method names is never bound, and an
undefined receiver type and a class name two libraries declare both
stay unresolved. The fixture asserts that every node it names exists,
so a missing node cannot pass as a missing edge. RED 3/3 on main,
green with the fix, RED again on revert; dropping the uniqueness rule
turns the ambiguity control RED.
The Python half of #1153, a method call on a module-level instance
(`manager = StackManager(); manager.status()`), is fixed by the held
Python LSP change 53653c15, not here.
Refs #1153
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
With `import Axios from "axios"`, a common convention in TypeScript
projects, every `Axios.get/post/put/delete(url)` client call was indexed as
a route registration. The result was a `CALLS {via: route_registration}`
edge to a Route node instead of an HTTP_CALLS edge. The client's request
paths therefore posed as server routes, and no HTTP call site existed to
match a handler. microsoft/Game-Control-Puzzle-Event-Administration-Tools
has all 69 of its client calls in this shape.
Root cause: service-pattern matching is case-sensitive by design, so that
"requests" cannot match "Requests" or "myrequests". The HTTP library
table knew only the lowercase id `axios`. `Axios.get` therefore matched no
library, on either resolver. Resolution came back empty, because an
external package is never indexed. The call then fell through to the
callee-suffix fallback, which treats any `<x>.get/post/...(path)` as a route
registration.
Fix: add `Axios` next to `axios` in the HTTP library table. The table
already pairs case variants of one library (`Dio`/`dio`,
`Guzzle`/`guzzle`, `OkHttp`/`okhttp3`). The existing identifier-boundary
rule applies unchanged:
- `Axios.get` and `AxiosInstance.get` (axios's own instance type) match;
- a lowercase continuation such as `Axiosphere` does not.
Express/Koa/Fastify registrations such as `router.get(path, handler)` are
untouched.
Proof (release binary, macOS arm64,
microsoft/Game-Control-Puzzle-Event-Administration-Tools @4fe23196,
before -> after):
- route_registration CALLS 69 -> 0; all 69 were `Axios.*` calls in
client/.
- All 69 reappear as typed HTTP_CALLS from the same caller to the same
Route QN (`__route__PUT__/api/admin/users/{}`), and no typed HTTP_CALLS
edge is lost.
- HTTP_CALLS 4 -> 142. Besides the 69 typed calls, 69 are the method-less
arg_url twins that fix/http-typed-call-no-url-twin removes, and 4 are the
existing `isOnPage` arg_url edges.
- CALLS 868 -> 799.
- SEMANTICALLY_RELATED 83 -> 174, deterministic across two runs of each
binary. The semantic pass tokenizes CALLS-neighbour names, and the fake
registration edges had fed each service function's URL path into its
vector. Without them, the 69 `Axios` wrapper functions read as related.
Five sampled calls were checked against the client source, including
admin/users/service.ts:44 `Axios.put(`/api/admin/users/${participantId}/
${eventInstanceId}`)` -> `__route__PUT__/api/admin/users/{}/{}`.
On the pending ASP.NET route stack (fix/aspnet-route-case-inrepo, 8e734ee5),
together with the arg_url twin fix, the reclassified calls reach their C#
handlers:
- typed HTTP_CALLS 0 -> 69;
- DATA_FLOWS 0 -> 66, and HANDLES 94 -> 106 (+12 case_fold);
- sampled links match their controller source, for example
`Axios.put(`/api/admin/users/${participantId}`)` ->
`[Route("api/admin/users")] [HttpPut("{participantId}")] UpdateUserInfo`;
- the 3 calls left unlinked carry a query string in the URL literal.
Tests: RED 3/3 before the fix, green after, RED again on revert.
- pipeline_capitalized_axios_import_is_http_client (parallel resolver) and
its _sequential twin: `import Axios from "axios"` with Axios.get, .post
and .delete gives 0 route registrations and exactly one typed
HTTP_CALLS per call, while an Express `router.get` registration in the
same project stays a registration.
- infrascan service-pattern tables: `Axios.get`, `Axios.post` and
`AxiosInstance.get` classify as HTTP, and `geo.Axiosphere.map` stays
NONE.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Previous behavior: cbm_scope_lookup(), cbm_scope_contains(), and cbm_scope_lookup_callable() called an out-of-line complete-binding scan after the Python lookup consolidation.
What changed:
- internal/cbm/lsp/scope.h defines cbm_scope_lookup_binding() as the one static-inline scope-chain scan.
- internal/cbm/lsp/scope.c keeps the type, presence, and callable wrappers on that shared implementation without another function boundary.
Why: Python still reads type and callable identity in one traversal, while existing language resolvers retain the direct hot path used before the consolidation.
Verification:
- CBM_ONLY_SUITE=scope build/c/test-runner: 33 passed
- CBM_ONLY_SUITE=cs_lsp build/c/test-runner: 62 passed
- CBM_ONLY_SUITE=py_lsp build/c/test-runner: 125 passed
- neutral 11-pair extraction matrix: output parity; C# latency -2.97% vs first parent and -1.64% vs upstream; retired instructions +0.29% vs first parent and -1.01% vs upstream; Python scale latency -4.25% and instructions -2.81% vs first parent
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
(cherry picked from commit 150c88e1c8)
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Add cbm_scope_lookup_binding so callers can read lexical presence, type, and callable identity in one traversal. Route Python expression, call, and callable-reference resolution through that complete binding and use the existing checked bind APIs for allocation-failure detection.
The neutral 11-pair production matrix kept exact output parity. Python scale latency was 4.33% below merge parent 07255cfe and 61.01% below upstream parent 0c64de93; incremental instructions were 3.09% and 54.07% lower. Focused ASan/UBSan scope and Python suites passed 33 and 125 tests.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
(cherry picked from commit 4d3a103047)
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
extraction::extract_spill_round_trip_keeps_every_field and
parallel::parallel_spill_mode_builds_the_same_graph failed on any machine
with less than 10 GB free: cbm_result_spill_open refuses to spill onto a
nearly full disk (mem.spill.refused reason=low_disk), so the store never
opened. Three pipeline tests took the same refused-store path there and
passed without spilling at all (backpressure_futile_nap_disengages,
over_budget_after_futility_fails_whole_and_preserves_db,
spill_resolves_namespace_imports_like_memory). A verdict must be a pure
function of code, test, platform and seed, not of how full the disk is.
Add a free-space pin to result_spill that exists only in test-seam
builds (CBM_ENABLE_TEST_SEAMS). Production still asks the disk, and its
10 GB floor and half-of-free cap are unchanged. The test runner pins
64 GiB at the top of main, before worker re-exec dispatch, so every
suite and re-exec'd index worker sees the same disk. A new test,
extraction::extract_spill_refuses_below_the_free_disk_floor, pins the
guard itself: one byte under the floor refuses, the floor opens.
Verified with cbm_fs_free_bytes temporarily forced to 9 GB: both tests
RED 3/3 before, green 3/3 after, RED again with the runner pin removed;
the refusal test goes RED with the production guard disabled.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Three iterative AST walkers kept their pending nodes in fixed-size stacks
and silently stopped pushing children once full:
- cbm_compute_complexity and count_branching_iter (helpers.c,
BRANCHING_STACK_CAP 4096) -> cyclomatic, cognitive, loop and access
metrics
- extract_body_ident_tokens (extract_defs.c, BT_STACK 512) -> the
body-token sample that feeds semantic search / SEMANTICALLY_RELATED
Children are pushed last-to-first, so a full stack dropped a wide body's
FIRST children. The metrics undercounted, and the token sampler read a
mid-body window instead of the first identifiers. That let a work cap decide
graph content.
Real input: tree-sitter-llvm's generated ts_lex (a 4654-case switch)
reported cyclomatic 11133 / cognitive 29303. It really has 8776 if + 4654
case + 1 default + 1 switch = 13432 branches, which is what it reports now
(cognitive 35639). On the Linux kernel no function reaches the old 4096 cap,
so complexity is byte-identical for every function. Three functions with
bodies over 512 pending nodes now sample their tokens from the start of the
body (for example rimob_panel_on), which moves 11 SEMANTICALLY_RELATED edges.
Index time is unchanged within noise (main 214.4/207.9 s, fix 197.6/192.2 s).
Fix: cbm_walk_stack_reserve()/cbm_walk_stack_release() in helpers.c. Each
walker keeps a small on-stack first chunk (512 frames) and spills to the heap
through the memory core (cbm_alloc/cbm_realloc/cbm_free, class EXTRACT) only
for deep or wide trees. Traversal order is unchanged. If the stack cannot
grow, the failure is logged ("extract.walk_stack_alloc_failed") and the
walker returns a documented sentinel instead of a truncated result:
CBM_WALK_METRIC_UNAVAILABLE (-1) for the metrics, or no token sample.
The body-token sampler's caps of 128 identifiers and a 2048-byte buffer are
its sampling contract and stay. The 512-entry stack was never part of that
contract, because it broke the "first identifiers in source order" property.
Tests (extraction suite): a C function with 5000 top-level if/while
statements must report exact cyclomatic/cognitive/loop counts through both
the extractor and cbm_count_branching, and a 600-call body must sample from
f0 on. All three are RED on main (4095 / 4095 / sample starting at f91) and
GREEN with the fix.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
When the receiver is iris.cls("<literal>"), the class-qualified name now
replaces the primary call's callee instead of being pushed as a second
call. Keeping the bare name let it weak-match an unrelated same-named
method.
Tests assert exactly one Run call per iris.cls() site, and the pipeline
fixture adds two unrelated Run definitions that must receive no edges.
Refs #1260
Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
walk_defs keeps its pending frames in a growable stack that was bounded by
an 8M-frame ceiling (env CBM_WALK_DEFS_MAX). At the ceiling wd_push logged
"extract.walk_defs_capped" once and then stopped pushing. Children are
pushed last-to-first, so a file wider than the ceiling lost its FIRST
definitions: a work cap decided graph content.
Fix: no ceiling. The stack doubles on demand, heap frames now come from the
memory core (cbm_realloc/cbm_free, class EXTRACT) instead of safe_realloc and
raw free, and the doubling is overflow-guarded (int capacity and size_t
byte count). The only thing that can stop the walk is an allocation failure,
and that is loud: logged at error level as "extract.walk_stack_alloc_failed"
(walker=walk_defs, pending, path) -- the same event the complexity walkers
use -- and the file's result is marked has_error with
"definitions walk: stack allocation failed", which the definitions passes
already report as a per-file extract error. The arena-backed path
(ctx->scratch) gets the same failure handling. Traversal order is unchanged.
CBM_WALK_DEFS_MAX is removed. It was read only in wd_stack_max(); it was not
documented anywhere (docs, --help, config, tests), so nothing else changes.
extract_defs.c loses its raw free() call, so the memory-core baseline for
the file drops from 5 to 4.
Test (extraction suite): walk_defs_wide_file_extracts_every_definition
extracts a C file with 1000 top-level functions with CBM_WALK_DEFS_MAX=256
set, and asserts every wd0..wd999 definition is present. RED on main
(extract.walk_defs_capped limit=256; wd0 missing), GREEN with the fix, RED
again with the production change reverted. The extraction (379), complexity
(5), ast_profile (10) and semantic (35) suites pass.
Real input: Linux kernel, index-only, fresh cache and runtime dirs, same
machine. main and fix both produce 8,530,336 nodes and 15,764,128 edges,
SEMANTICALLY_RELATED 607 each, and the full node set (label, QN, lines) and
edge set (type, source QN, target QN) hash identically -- no kernel file
approaches the old ceiling. Index wall time 210.0 s (main) vs 194.4 s (fix),
within run-to-run noise.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel