Previous behavior: cbm_scope_lookup(), cbm_scope_contains(), and cbm_scope_lookup_callable() called an out-of-line complete-binding scan after the Python lookup consolidation.
What changed:
- internal/cbm/lsp/scope.h defines cbm_scope_lookup_binding() as the one static-inline scope-chain scan.
- internal/cbm/lsp/scope.c keeps the type, presence, and callable wrappers on that shared implementation without another function boundary.
Why: Python still reads type and callable identity in one traversal, while existing language resolvers retain the direct hot path used before the consolidation.
Verification:
- CBM_ONLY_SUITE=scope build/c/test-runner: 33 passed
- CBM_ONLY_SUITE=cs_lsp build/c/test-runner: 62 passed
- CBM_ONLY_SUITE=py_lsp build/c/test-runner: 125 passed
- neutral 11-pair extraction matrix: output parity; C# latency -2.97% vs first parent and -1.64% vs upstream; retired instructions +0.29% vs first parent and -1.01% vs upstream; Python scale latency -4.25% and instructions -2.81% vs first parent
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
(cherry picked from commit 150c88e1c8)
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Add cbm_scope_lookup_binding so callers can read lexical presence, type, and callable identity in one traversal. Route Python expression, call, and callable-reference resolution through that complete binding and use the existing checked bind APIs for allocation-failure detection.
The neutral 11-pair production matrix kept exact output parity. Python scale latency was 4.33% below merge parent 07255cfe and 61.01% below upstream parent 0c64de93; incremental instructions were 3.09% and 54.07% lower. Focused ASan/UBSan scope and Python suites passed 33 and 125 tests.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
(cherry picked from commit 4d3a103047)
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
extraction::extract_spill_round_trip_keeps_every_field and
parallel::parallel_spill_mode_builds_the_same_graph failed on any machine
with less than 10 GB free: cbm_result_spill_open refuses to spill onto a
nearly full disk (mem.spill.refused reason=low_disk), so the store never
opened. Three pipeline tests took the same refused-store path there and
passed without spilling at all (backpressure_futile_nap_disengages,
over_budget_after_futility_fails_whole_and_preserves_db,
spill_resolves_namespace_imports_like_memory). A verdict must be a pure
function of code, test, platform and seed, not of how full the disk is.
Add a free-space pin to result_spill that exists only in test-seam
builds (CBM_ENABLE_TEST_SEAMS). Production still asks the disk, and its
10 GB floor and half-of-free cap are unchanged. The test runner pins
64 GiB at the top of main, before worker re-exec dispatch, so every
suite and re-exec'd index worker sees the same disk. A new test,
extraction::extract_spill_refuses_below_the_free_disk_floor, pins the
guard itself: one byte under the floor refuses, the floor opens.
Verified with cbm_fs_free_bytes temporarily forced to 9 GB: both tests
RED 3/3 before, green 3/3 after, RED again with the runner pin removed;
the refusal test goes RED with the production guard disabled.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Three iterative AST walkers kept their pending nodes in fixed-size stacks
and silently stopped pushing children once full:
- cbm_compute_complexity and count_branching_iter (helpers.c,
BRANCHING_STACK_CAP 4096) -> cyclomatic, cognitive, loop and access
metrics
- extract_body_ident_tokens (extract_defs.c, BT_STACK 512) -> the
body-token sample that feeds semantic search / SEMANTICALLY_RELATED
Children are pushed last-to-first, so a full stack dropped a wide body's
FIRST children. The metrics undercounted, and the token sampler read a
mid-body window instead of the first identifiers. That let a work cap decide
graph content.
Real input: tree-sitter-llvm's generated ts_lex (a 4654-case switch)
reported cyclomatic 11133 / cognitive 29303. It really has 8776 if + 4654
case + 1 default + 1 switch = 13432 branches, which is what it reports now
(cognitive 35639). On the Linux kernel no function reaches the old 4096 cap,
so complexity is byte-identical for every function. Three functions with
bodies over 512 pending nodes now sample their tokens from the start of the
body (for example rimob_panel_on), which moves 11 SEMANTICALLY_RELATED edges.
Index time is unchanged within noise (main 214.4/207.9 s, fix 197.6/192.2 s).
Fix: cbm_walk_stack_reserve()/cbm_walk_stack_release() in helpers.c. Each
walker keeps a small on-stack first chunk (512 frames) and spills to the heap
through the memory core (cbm_alloc/cbm_realloc/cbm_free, class EXTRACT) only
for deep or wide trees. Traversal order is unchanged. If the stack cannot
grow, the failure is logged ("extract.walk_stack_alloc_failed") and the
walker returns a documented sentinel instead of a truncated result:
CBM_WALK_METRIC_UNAVAILABLE (-1) for the metrics, or no token sample.
The body-token sampler's caps of 128 identifiers and a 2048-byte buffer are
its sampling contract and stay. The 512-entry stack was never part of that
contract, because it broke the "first identifiers in source order" property.
Tests (extraction suite): a C function with 5000 top-level if/while
statements must report exact cyclomatic/cognitive/loop counts through both
the extractor and cbm_count_branching, and a 600-call body must sample from
f0 on. All three are RED on main (4095 / 4095 / sample starting at f91) and
GREEN with the fix.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
When the receiver is iris.cls("<literal>"), the class-qualified name now
replaces the primary call's callee instead of being pushed as a second
call. Keeping the bare name let it weak-match an unrelated same-named
method.
Tests assert exactly one Run call per iris.cls() site, and the pipeline
fixture adds two unrelated Run definitions that must receive no edges.
Refs #1260
Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
walk_defs keeps its pending frames in a growable stack that was bounded by
an 8M-frame ceiling (env CBM_WALK_DEFS_MAX). At the ceiling wd_push logged
"extract.walk_defs_capped" once and then stopped pushing. Children are
pushed last-to-first, so a file wider than the ceiling lost its FIRST
definitions: a work cap decided graph content.
Fix: no ceiling. The stack doubles on demand, heap frames now come from the
memory core (cbm_realloc/cbm_free, class EXTRACT) instead of safe_realloc and
raw free, and the doubling is overflow-guarded (int capacity and size_t
byte count). The only thing that can stop the walk is an allocation failure,
and that is loud: logged at error level as "extract.walk_stack_alloc_failed"
(walker=walk_defs, pending, path) -- the same event the complexity walkers
use -- and the file's result is marked has_error with
"definitions walk: stack allocation failed", which the definitions passes
already report as a per-file extract error. The arena-backed path
(ctx->scratch) gets the same failure handling. Traversal order is unchanged.
CBM_WALK_DEFS_MAX is removed. It was read only in wd_stack_max(); it was not
documented anywhere (docs, --help, config, tests), so nothing else changes.
extract_defs.c loses its raw free() call, so the memory-core baseline for
the file drops from 5 to 4.
Test (extraction suite): walk_defs_wide_file_extracts_every_definition
extracts a C file with 1000 top-level functions with CBM_WALK_DEFS_MAX=256
set, and asserts every wd0..wd999 definition is present. RED on main
(extract.walk_defs_capped limit=256; wd0 missing), GREEN with the fix, RED
again with the production change reverted. The extraction (379), complexity
(5), ast_profile (10) and semantic (35) suites pass.
Real input: Linux kernel, index-only, fresh cache and runtime dirs, same
machine. main and fix both produce 8,530,336 nodes and 15,764,128 edges,
SEMANTICALLY_RELATED 607 each, and the full node set (label, QN, lines) and
edge set (type, source QN, target QN) hash identically -- no kernel file
approaches the old ceiling. Index wall time 210.0 s (main) vs 194.4 s (fix),
within run-to-run noise.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
In IRIS embedded Python, iris.cls("Pkg.X").Method(...) names its target
class in the receiver's string literal. There was no receiver path for it
next to the classMethod* string dispatch, so after the weak-member-call
suppression in b6a22843 these calls produced no CALLS edge at all.
When a Python call's receiver is iris.cls() with a plain quoted literal as
its first argument, also emit "<class-literal>.<method>" as a callee, the
same way classMethodValue already does. Non-literal class arguments
(variables, f-strings) name nothing and get no class-qualified callee.
Also add invokeClassMethod to the Python dispatch suffixes. It takes the
same (class_name, method_name, *args) arguments as classMethodValue.
Tests: the #1260 repro as a pipeline test (two classes sharing Run, a
nonexistent class, an unrelated receiver, classMethodValue and
invokeClassMethod), plus extraction tests for the synthesized callee,
non-literal and f-string arguments, and invokeClassMethod.
Fixes#1260
Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
The comment on CBM_EXTRACT_BUDGET called it a "maximum byte budget", but
every call site passes it as cbm_extract_file*()'s timeout_micros, which
cbm.c converts to nanoseconds and uses as a per-thread CPU-time deadline
(5 s) with a 12x wall-clock ceiling as a backstop for a stuck parse. No
code path uses the value with the wrong unit; only the documentation was
wrong.
Correct the constant's comment, and bring the timeout_micros description
in cbm.h up to date (CPU time + wall backstop, not a plain timeout) and
back onto cbm_extract_file(), from which later additions had separated it.
No behavior change; the budget value is unchanged.
Reported-by: BobbieBarker
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A generated API client (Orval and similar) passes its request as a config
object to a local fetch wrapper:
http<Customer>({url: `/api/customers/${id}`, method: 'GET', signal})
The template literal itself already canonicalizes to "/api/customers/{}"
(#1006). What was lost is the object wrapper: the call's arguments only
carried a value for bare strings, template literals, constants and URL
builders, so the object argument had no URL. The arg-url heuristic
therefore minted no HTTP_CALLS edge, and cross-repo matching had nothing to
join against the server's GET /api/customers/{id} Route. The same held for
HTTP clients called with a config object (`axios({url})`): no URL, no edge.
The extractor now reads the `url` property of an object argument in
JS/TS/TSX/ArkTS -- both as that argument's value (for the arg-url
heuristic) and as the call's URL (for HTTP-client classification). Only
the `url` key counts. Route registrations (Fastify `route({url, handler})`)
and axios `getUri(config)`, which formats a URL and sends nothing, are
excluded.
orval-labs/orval @ 08d7fcf1 (parallel resolver): HTTP_CALLS 1383 -> 1533
(+150 arg_url edges from the generated mutators such as customInstance,
responseType and customClient; 0 lost), Route 302 -> 308, CALLS unchanged
(19190). Without the getUri exclusion there would have been 546 URL-builder edges.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A 4.7 MB binary Godot resource (piko_walk_mesh.res) never finished
indexing: `.res` maps to ReScript, so the resource was parsed as ReScript
source and the definitions pass spun. Three defects, each fixed:
1. Quadratic usage walk (the hang). The ReScript `let` binding check
(is_first_named_part_of, shared with Julia, Typst, Elm, PureScript
and Nickel) climbed from every named leaf with ts_node_parent, which
descends from the root and scans each level's children. A binary
file parses into a flat error tree (~170k root children per MB), so
each climb cost O(root children) and the file O(n^2): 1 MB spent
46 s in cbm_extract_unified, 4.7 MB never returned. The check now
climbs the unified walk's own cursor (O(1) per hop), like the other
occurrence classifiers; the node-based climb stays only for callers
outside the walk. 1 MB: 46 s -> 0.03 s; 4.7 MB: 2.2 s end to end.
2. Quadratic lexing in error recovery. tree-sitter retries the lexer at
every byte of an unparseable stretch with every external token
valid; the ReScript scanner's template-string loop then ran to the
next backtick/$/backslash/NUL (end of file if none) and discarded
the result. That cost O(n^2) inside lexing, where the parse budget's
progress callback never runs, so such files were dropped by the
5 s CPU budget instead of parsed -- plain text too (100 KB of
non-ReScript text: 15 s -> 0.18 s). Local scanner patch: skip the
template branch when NEWLINE is also valid, which only the ERROR
state allows. Valid files parse to identical trees. Recorded in
vendored/grammars/MANIFEST.md.
3. Binary content parsed as source. `.res` is also Godot's binary
resource format and the Windows compiled-resource format. Following
the .m/.cls/.frm/.inc/.cfc precedent, cbm_disambiguate_res reads the
first 4 KB and reports a file holding a NUL byte as unsupported;
ReScript text never contains one.
Tests (deterministic work counters, no clocks):
- extraction::extract_rescript_let_bindings_use_the_walk_cursor --
zero root-descending parent hops at 128 and 1024 statements
(before: 515 / 4099).
- complexity::complexity_rescript_error_recovery_lexing_is_linear --
lexer bytes pulled through a chunked TSInput, 4 KB vs 8 KB, ratio
must be linear (before 3.99, after 2.00).
- language::lang_res_binary_resource_unsupported and
lang_res_rescript_stays_rescript.
Fixes#2176
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A mysqldump-style .sql file is a few CREATE TABLEs followed by megabytes of
`INSERT ... VALUES (..),(..),...` rows. Tree-sitter built a full tree for
every row (~70 bytes of tree per source byte; 0.26 s/MB with MySQL \'
escapes, which the grammar does not know and error-recovers from) until the
5 s parse budget ran out, and the whole file, tables included, was skipped
as "parse timeout". The reporter's database/sql/world.sql was dropped this
way. The rows contribute nothing to the graph.
A linear, quote- and comment-aware pre-scan (internal/cbm/sql_values.c)
finds INSERT/REPLACE ... VALUES lists and excludes every tuple after the
first that consists only of literals: strings (standard '' and MySQL \'
escapes, X'..'/B'..'/N'.., _charset introducers), numbers, NULL, TRUE,
FALSE, DEFAULT. A tuple holding a subquery, a function call, an identifier
or a quoted identifier stays in the parse, and so does one whose string
spans a raw newline. The excluded bytes go to ts_parser_set_included_ranges
as the complement, so byte offsets and line/column positions of all kept
text are unchanged. The ranges are cleared right after the parse because
the parser is thread-local and reused. This is a syntax rule, not a work
cap: the same file always gives the same ranges, and the parse budget is
unchanged.
Measured on generated mysqldump-shaped dumps, release build, same machine,
index worker's own numbers (base = origin/main):
dump extraction pipeline peak RSS graph
8 MB std '' 1310 -> 23 ms 1431 -> 139 ms 625 -> 37 MB identical
8 MB MySQL \' 2712 -> 24 ms 2827 -> 146 ms 474 -> 37 MB identical
32 MB MySQL \' 5145 -> 82 ms 5467 -> 396 ms 1132 -> 62 MB timeout -> indexed
59 MB MySQL \' 5130 -> 138 ms 5667 -> 674 ms 1157 -> 89 MB timeout -> indexed
"identical" = every stored node and edge, Module end line included. For the
32/59 MB dumps the base run skipped the file as "parse timeout" (Module +
3 Tables + their DEFINES/USAGE missing). The fixed graph matches the 8 MB
graph except for the Module's end line.
With standard escapes the extraction and the stored graph are identical
to the full parse. With MySQL \' escapes the full parse's error recovery
swallows rows next to a misread escape, a subquery row among them, and the
cut parse keeps those rows. So it loses nothing the full parse found
outside the dropped rows and recovers what the full parse lost. The only
things the full parse extracted INSIDE the dropped rows were junk usages
(DEFAULT and _binary read as identifiers, fragments of misread strings)
that name nothing.
Tests (parse_coverage, pipeline; a CBM_TEST_SQL_FULL_PARSE_ON seam turns the
exclusion off for one file so the same source can be compared both ways):
- scanner cases: comments, strings, dollar quotes, ON DUPLICATE KEY
UPDATE, VALUES outside INSERT, unclosed tuples, exact kept positions
- extraction equals the full parse on a small dump (standard escapes);
nothing lost outside dropped rows (MySQL escapes)
- the subquery and function-call rows are still parsed
- the tree does not grow when the row count doubles (a count, not a clock)
- a 25 MB MySQL dump under the production budget is indexed, not
"parse timeout" (asserts the outcome, not the time)
- stored nodes and edges of a pipeline index equal the full parse's
Refs #1735
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_subtract_macro_invocation_regions asked "is this region a call to a
file-defined macro?" before "is it inside a function?". The first question
(cbm_span_is_macro_invocation) walks the source from byte 0 to find the
region's lines, so every file with parse-error regions paid
regions x file bytes, whatever its language and whether or not it has a
single function. On a multi-MB SQL dump that is up to 256 walks over the
whole file.
Both predicates are pure, so asking the cheap one first changes no answer:
a region no function encloses never touches the source, and the regions
that do share one cbm_line_offsets table built on first use. The table
gives exactly the span the walk found; the walk stays as the fallback when
the table cannot be allocated.
#1071 behaviour is unchanged: same regions, same verdicts. Indexing this
repository's src/ before and after gives identical graphs (7,339 nodes,
28,924 edges) and the same 46 parse_partial files with byte-identical
ranges.
Tests count the bytes read, never time them (test-seam counter
cbm_test_macro_line_scan_bytes):
- 40 top-level junk regions: 39,870 bytes read -> 0, still 40 regions
- 60 in-body real errors: 85,725 bytes read -> 2,801 (one table = file
size), still 60 regions
Refs #1735
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Rust LSP types `root: &Path` and dispatches `root.join(..)` to
std.path.Path.join, a registered seed method with no graph node. Both
call resolvers treated "LSP row without a gbuf target" like "no LSP row"
and fell back to the textual registry, which bound the call to the only
project method named `join` (EvidenceTier::join via unique_name). Every
std/seeded-crate call whose leaf name a project method shares was
fabricated this way.
cbm_pipeline_rust_external_target() (lsp_resolve.h) recognises a Rust row
whose strategy emits a registered target (method/trait/deref/bound
dispatch, direct, ufcs, constructor) and whose QN lies outside the
project prefix. Project defs are registered under project-prefixed QNs, so
such a target is a std/core/alloc or seeded-crate symbol by construction.
pass_calls.c and pass_parallel.c skip the registry fallback for those rows
(in lockstep), and the parallel route-method fallback no longer turns a
skipped `map.get(k)` into a source->source self-call. Project-prefixed
rows, unknown-receiver rows and every other language are unchanged.
The call-expression receiver (`root.to_path_buf().join(..)`) needed type
facts: Path::join/to_path_buf/with_extension/with_file_name now return
PathBuf, PathBuf::as_path returns Path, and PathBuf derefs to Path.
Measured on BurntSushi/ripgrep @ 3fce3b5 (fresh index, two runs each,
byte-identical): CALLS 5371 -> 5014; 357 removed (307 suffix_match,
45 unique_name, 5 field_type_hint), 0 added, no lsp_* edge touched.
Removed samples are all std receivers (String::new -> State.new,
msg.contains -> ByteSet.contains, alts.push on a Vec -> TSeq.push).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The bulk writer builds each interior level greedily: it fills a page with
separator cells until the next one no longer fits, uses that child as the
page's right child, and starts the next page after it. When a page filled
up exactly at children[child_count - 2], only the last child was left, so
the next (and last) page of that level was written with ZERO cells and
just a right-child pointer.
SQLite rejects a non-root page with nCell < 1 as SQLITE_CORRUPT the moment
a cursor descends into it (getAndInitPage). Because that page is always the
last one of its level, the failing descent is the rightmost path -- exactly
what the coverage-shadow wipe takes: "DELETE ... WHERE project = ?1" with
"<project>::missed", a key that sorts after every row of "<project>". The
publish then failed with "coverage_shadow_wipe: database disk image is
malformed". It needs an index with more than one interior page on a level
(large graphs or long keys) AND the page-fill boundary landing on the one
bad position, which is why it looked scale-dependent and repo-specific;
the same DBs also fail PRAGMA integrity_check.
When a page fills with exactly one child left, give its last cell back, so
the next page holds one cell plus the last child as its right child. The
removed bytes are zeroed, keeping the output deterministic.
Test: sqlite_writer::sw_interior_level_never_ends_with_empty_page writes
2..120 rows with 12000-byte names (one and two interior levels) and runs
the wipe-shaped DELETE through idx_nodes_name; rows 31-35, 61-65 and 91-95
failed with SQLITE_CORRUPT before the fix. Real input: 41 generated
markdown repos with long headings indexed by v0.11.0 -> 5 databases fail
the "::missed" seek and integrity_check; with the fix 0 of 41.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two deterministic indexing-worker crashes reported on #1743.
C/C++ (the original LLVM report, "segfault at 9"): structured-binding
decomposition read rt->field_types[binding_idx] guarded only by
`rt->field_types[binding_idx] != NULL`. field_types[] is parallel to
field_names[] and NULL-terminated, so a binding list longer than the
struct's field list (ill-formed, but present in compiler test suites,
e.g. llvm clang/test/CXX/dcl.decl/dcl.decomp/p3.cpp) read past the
terminator into the next arena object. That object is a CBMType whose
first word is its kind (9), so the name was bound to the pointer 0x9 and
decltype(name) dereferenced it in c_resolve_name_to_type. All three
decomposition sites now go through c_struct_field_type_at(), which bounds
the index by field_names[] and yields NULL (unknown) past the end.
TypeScript (kilocode report on v0.11.0): lookup_method walked the
extends/implements chain recursively with no depth bound. The Effect idiom
`class Service extends Context.Service<Service, I>()(...)` gets its base
resolved to the class itself, so `Service.of(...)` (or a sibling class
whose base resolves to that Service) recursed until the resolver thread's
stack overflowed. lookup_method now has the same depth-guarded entry as
lookup_member_type (TS_LSP_MAX_MEMBER_DEPTH); past the cap the method is
simply not found.
Proof on real input (macOS arm64, fresh cache each run):
- llvm/llvm-project llvmorg-15.0.7: before, 5 worker SIGSEGVs at 0x9 in
c_resolve_name_to_type and 3 files quarantined; after, 0 crashes,
0 quarantined, 1,375,933 nodes / 4,257,770 edges.
- Kilo-Org/kilocode main: before, the resolve worker died every attempt
(SIGBUS, lookup_method recursion depth ~65k); after, indexed cleanly,
90,745 nodes / 440,654 edges.
Tests: c_lsp::clsp_structured_binding_more_names_than_fields and
ts_lsp::tslsp_nocrash_cyclic_extends_method_call (both crash RED without
the fix).
Fixes#1743
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Dart language spec listed class_definition, enum_declaration,
mixin_declaration and type_alias as container kinds, but not
extension_declaration. The def walk never treated `extension X on Y { }`
as a container, and extract_func_def could not name the bare
`method_signature` wrappers in the extension body. So the extension and
every member were dropped: only the Module, File, the extended class and
top-level functions made it into the graph.
- lang_specs.c: add extension_declaration to dart_class_types. A named
extension becomes a Class node, and its members become Methods with
DEFINES_METHOD edges from it. In-body calls get the method as caller.
- extract_defs.c: an unnamed `extension on T { }` has no name to attach
its members to. extract_class_def would return before
extract_class_methods and drop them. Walk such an extension's members
as file-level Functions instead, unwrapping method_signature to its
named function_signature.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Vue-admin wrapper pattern -- `const api = axios.create({ baseURL:
'/api' }); export default api` in one module, `api.get('/orders')` in
another -- produced no HTTP_CALLS edge, so cross-repo-intelligence found
nothing to link to the backend's GET /api/orders.
Root cause: nothing recorded that `api` is an axios instance. The call
`api.get('/orders')` fell through to the route-registration suffix
fallback (`.get` + a '/'-leading argument), which treats it as an Express
`app.get` REGISTRATION: a CALLS edge to a phantom server-side Route
`/orders` (via=route_registration), and never an HTTP_CALLS edge. The
base path was unknowable anyway.
Fix, reusing HTTP_CALLS/Route (no new labels):
- extraction (JS/TS/TSX/ArkTS): a module-level binding initialised by
`axios.create(...)` records http_client="axios" and the literal
`baseURL` (NULL when absent or non-literal, never guessed); the Module
def carries the same pair when its default export is such a client
(`export default api` / `export default axios.create(...)`). ES default
imports are flagged (CBMImport.is_default).
- both call resolvers (pass_calls.c, pass_parallel.c), before the #523
callee-spelling check and the route-registration fallback: a
`<recv>.<verb>(path)` whose receiver binds -- same file, named import, or
default import -- to a client emits HTTP_CALLS to base + path (axios
combineURLs: trailing '/' trimmed, absolute URLs kept as-is, path kept
when the base is unknown). Shared helper
cbm_pipeline_http_client_call_url keeps the two paths identical.
- incremental: the client pair joins the file's LSP surface (only for files
that have one, so every other surface sha is unchanged), so a baseURL
edit pulls importers into the closure; delta preseed proxies carry the
two keys so a caller re-resolved alone still sees its unchanged wrapper.
- result_compact relocates the two new strings (ASan caught the dangling
worker-arena pointer).
Look-alikes (`factory.create({ baseURL })`, a plain object's `.get`) are
unchanged.
Test: pipeline_axios_wrapper_baseurl_composes_http_calls_issue1916 --
default/named/same-file/dynamic-base/look-alike cases, sequential vs
parallel resolver equality, and same-DB closure-repair reindex (wrapper
edit, then caller-only edit) converging with fresh indexes. RED on origin/
main (seq.wrapper_http == 0); each of the parallel, surface and preseed
pieces reverted alone turns its own assertion RED.
Fixes#1916
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Accepting relative JAX-RS templates made route_path_from_string_node return
"" for @Path(""), which is non-NULL and so replaced the "/" fallback with an
empty method path. @Path("") is legal JAX-RS and means "the class path
itself"; on main the empty value was rejected and the method fell back to
"/" like a method without @Path. Return NULL for the empty case so that
behaviour is unchanged.
The regression covers both shapes: with a class-level @Path the method
composes to the class path, and without one the route stays "/" (with the
empty string it was dropped entirely, because join_route_paths returns the
NULL prefix for an empty path).
Signed-off-by: Hubert Tarnacki <hubert.tarnacki@gmail.com>
Distilled from #1245 by Andrew Hundt (c51e5cb0, the extract_defs.c
half; the mcp.c and token-reduction parts are not taken).
For C-family functions return_type was the text of the declaration's
`type` node alone. In these grammars the `*`, `&` and `&&` live on the
declarator and cv-qualifiers are sibling nodes, so
const char *get_name(void) published char
robj **lookupKeys(...) published robj
Text &operator=(const Text &) published Text
and no C or C++ function in any index had a pointer return type at all.
The C/C++ resolver's return-type parser already strips a leading
const/volatile and handles trailing `*`, `&`, `&&` -- branches that were
unreachable for functions, so every `Foo *f()` was registered as
returning Foo by value.
c_declared_return_type() renders the declared type at both sites
(functions and methods) for C, C++, CUDA, GLSL, HLSL, ISPC, Slang and
Objective-C -- the list resolve_func_name_c_family uses:
<leading cv-qualifiers, source order> <base type, verbatim>
<one space><markers, outermost first>
so `const char *`, `char **`, `Text &`, `Text &&`, `Text *&`,
`volatile unsigned long *`; a pointer-level qualifier follows its star
(`char *const *`); east-const `char const *` normalises to
`const char *`. With no qualifier and no marker the old text is returned
unchanged. It measures, then writes into one exact-size arena
allocation; the declarator walk is a plain child chain, no recursion.
def.return_types is untouched.
Two deliberate differences from the upstream hunk. Upstream copied every
type_qualifier node, which in these grammars includes constexpr,
_Noreturn, mutable and __extension__ -- `constexpr int f()` would have
become `constexpr int`. Only const, volatile, restrict, __restrict,
__restrict__ and _Atomic are kept, and both cases are asserted unchanged.
And the walk stops at the first declarator that is neither pointer nor
reference, so `int (*f(void))(int)` stays `int` rather than becoming a
wrong `int *`.
RED, production reverted and the final tests kept:
c_function_return_type_preserves_pointer_and_qualifier
FAIL tests/test_extraction.c:1005: "char" != "const char *"
cpp_method_return_type_preserves_pointer_and_qualifier
FAIL tests/test_extraction.c:1072: "char" != "const char *"
351 passed, 2 failed
c_function_return_type_plain_unchanged is the control: it passes before
and after.
GREEN: extraction 353, c_lsp 762, lang_contract 41, pipeline 281.
Effect on real code -- Redis, production binaries, old against new.
39,132 nodes in both. 1,442 Function nodes change, in return_type only:
void -> void * (270), char -> const char * (101), unsigned char ->
unsigned char * (91), char -> char * (81), robj -> robj * (57), ...
Functions with a `*` in return_type: 0 -> 1,440. With one worker the
complete edge set apart from SEMANTICALLY_RELATED is byte-identical old
against new across six runs (three each) -- CALLS, USAGE, WRITES, IMPORTS
and the rest do not move. SEMANTICALLY_RELATED goes 350 -> 325 because
that pass tokenises the return-type string into its type vector.
One thing measured rather than assumed. With several workers, which
`dict.h` Redis' src/dict.c imports (src/ or deps/hiredis/) depends on
worker merge order. That is main's defect, not this change's: unmodified
main picked src/dict.h in 2 of 10 runs and the hiredis header in 8; with
one worker both binaries always pick src/dict.h. This change picked
src/dict.h in 6 of 10. At ten runs a side that gap is not separable from
chance (Fisher exact, two-sided p = 0.17), and a shifted rate would be
unsurprising for a timing-dependent pick; it is recorded here rather
than explained. The include-target fix is a separate change (#2227).
Known limits, all pre-existing and out of scope: struct and class FIELDS
have the same defect (`const char *data;` publishes char); C prototypes
and declaration-only C++ methods are not extracted as definitions, so
they are not reached; where a macro is misparsed as the type
(`LUA_API const char *f()`), the base was already wrong and now reads
`const LUA_API *`; the resolver's parser does not yet understand a
pointer-level qualifier or a leading _Atomic/restrict and falls back to
a named-type lookup for those, as it effectively did for every pointer
return before.
Co-authored-by: Andrew Hundt <ATHundt@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distilled from #1245 by Andrew Hundt (d15071c9, the match_qn half).
match_qn was a raw strstr(qn, library_id). Short ids fired inside
unrelated words, and a service kind REPLACES the plain CALLS edge, so a
false match both invents a service edge and deletes a real call:
"gin." in plugin. -> route registration; any "/"-prefixed
first argument then minted a Route node
"nconf" in encryptionconfig -> CONFIGURES for every call into the
package
"tonic" in Monotonic -> gRPC; unparseable, so NO edge at all
"express" in expression -> route registration
"get_env" in get_envelope, get_environ -> CONFIGURES
"dio" "surf" "curl" "resty" "phin" "hyper" "treq" "rocket" in studio,
surface, curly, restyle, dolphin, hyperbolic, streq, rocketmq
The rule (qn_hit_on_boundary; every occurrence of an id is tried, not
only the first). A separator is any character that is not an ASCII
letter or digit.
before: start of string | separator | the id itself starts with a
separator ("@trpc/server") | the id starts with an uppercase
letter -- a capital opens a new CamelCase word whatever
precedes it (AsyncHttpClient, IHttpClientFactory, NSURLSession)
after: end of string | separator | the id itself ends in a separator
("gin.") | uppercase letter (GuzzleHttp, FeignClient,
KafkaProducer) | digit (urllib2, Mint.HTTP2, amqp091-go)
Rejected: a lowercase- or digit-initial id glued to a preceding letter
or digit, and any id continued by a lowercase letter. Two deliberate
differences from the upstream hunk: upstream's before-rule needed a
GuzzleHttp table entry and would have dropped IHttpClientFactory and
NSURLSession, and without the digit after-boundary urllib2 and
amqp091-go stop matching. All three match on main today.
No recall is traded for this. No boundary rule can tell "grequests"
from "myrequests", so a library whose own name glues a prefix or suffix
onto another id gets an explicit entry with the kind (and broker) it had
on main -- 47 of them, found mechanically by running main's matcher and
the new one over 173 candidate names and listing every one that went
from a kind to NONE:
HTTP grequests txrequests redaxios gaxios libcurl curlpp curlcpp
hyperlocal guzzlehttp
ASYNC aiokafka pykafka rskafka librdkafka rdkafkacpp aioamqp
amqpstorm pamqp pyamqp amqprs amqpcpp pynats jnats aiomqtt
amqtt hbmqtt umqtt mqttools emqtt rumqtt gomqtt libmosquitto
mosquittopp pubsublite gocelery
CONFIG wgetenv getenvb qgetenv dotenvx phpdotenv
ROUTE_REG apiflask honox fasthttprouter
GRAPHQL gqlparser gqlgenc aiogqlc
Left at NONE on purpose: names that are not clients of the id they
contain and were misclassified on main -- hypercorn, hyperlink,
hyperopt, openresty, gqlalchemy, celeryconfig, kafkacat, gnatsd,
pypubsub. One visible reclassification: hyperium/tonic was HTTP through
"hyper" inside "hyperium" and is now gRPC through "tonic".
RED, production reverted and the final tests kept:
infrascan_service_pattern_match_rejects_ids_inside_words
FAIL tests/test_infrascan.c:105:
svc_case_mismatches(cases) == 28, expected 0 == 0
4 passed, 1 failed
All 28 negatives mismatch on main. The 47 glued-library positives are
RED the other way round: against the boundary rule without the entries.
Each positive uses a name containing no second id, so it binds to its
own entry -- "aiokafka.AIOKafkaProducer" would have passed through
"KafkaProducer" and proven nothing.
GREEN: infrascan 5, pipeline 281, parallel 74, extraction 350,
edge_types_probe 59, route_canon 11, lang_contract 41, cross_repo 8,
registry 64, mcp 318 (4 Windows-only skips). 0 failed.
Effect on real code -- production binaries, main 92abefa3 against this
change. main against main differs by 0 edges and 0 nodes on all three
corpora, so every delta below is the change:
django CONFIGURES 76 -> 74; both calls return as CALLS
(geom.get_envelope, WSGIRequestHandler.get_environ)
typescript CONFIGURES 12,026 -> 12,024; one bogus Route "/*type*/"
gone with its CALLS and HANDLES; 3 calls return as CALLS
kubernetes CONFIGURES 7,962 -> 7,609: all 353 are "nconf" inside
encryptionconfig / authorizationconfig /
authenticationconfig, and all 353 return as CALLS.
8 bogus Routes gone -- os.ReadFile of the serviceaccount
namespace file, t.Run("/TwoWay"), "/tmp/test" -- with 26
CALLS, 2 HANDLES and 21 TESTS edges that pointed at them.
374 plain CALLS restored; 8 of them (validate.Monotonic and
friends) had NO edge at all on main. GRPC_CALLS 60 -> 60.
HTTP_CALLS 398 -> 407: nine calls main swallowed as route
registrations now reach the existing arg_url heuristic like
every other package.
88 of the 408 removed service-derived edges were read at the call site
-- all of django and typescript, 82 of 402 in kubernetes: 88 false
positives, 0 true positives lost. None of the three corpora uses a
glued-name library, so the 47 entries are covered by the unit test only.
Known limits: "_" is a separator, so optical_fiber.len still matches
"fiber." exactly as requests_get matches "requests" by design; a
lowercase id followed by a capital matches (kafkaProducer), which also
admits a dioXide-style name; a user wrapper glued without a separator
(mygetenv) is no longer classified; a glued-name library missing from
the list is NONE until someone adds it.
Co-authored-by: Andrew Hundt <ATHundt@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Fallout of rebasing this branch onto main. This branch was written while
the LSP walk could be skipped for a file whose parse had used more than
its share of the budget, and that share was measured in the thread's CPU
time. Main has since removed the rule entirely -- no clock decides what
ends up in the graph -- so cpu_start_ns is gone from the function and the
only line still reading it was computing a value with no remaining
consumer.
The conflict resolution kept main's side in cbm.c, which is what leaves
this line stranded: it is dead by then, and the build says so.
error: use of undeclared identifier 'cpu_start_ns'
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The lint chain stops at its first red target, so a clang-format failure hid the memory-core linter
until PR CI: the coverage line-offset table, the Cypher direction swap and the search label each
added a raw allocator site. The table now comes from cbm_alloc/cbm_free (extract class), the swap
frees through safe_str_free, and the BM25 row array moves to cbm_calloc/cbm_free so mcp.c's raw-site
count goes down (baseline 814 -> 813) rather than up.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Nine findings from the 2026-09-16 graph-quality probe over the eleven bench corpora, each with a
test that fails without its fix and a before/after measurement on the real corpus.
Deterministic resolution (O9). Two indexes of one kernel tree differed in 5,821 IMPORTS and 632
CALLS edges. Three order/load dependencies carried it: the `#include` target was the first by-name
hit whose path ends with the include path — a bucket in worker merge order (include/linux/device.h
vs tools/virtio/linux/device.h); the registry broke suffix_match score ties by bucket position; the
LSP budget-share rule measured wall time. Include targets now prefer the File node, the least nested
path, then the lexicographic path; registry ties prefer the least nested QN, then the smaller QN; the
budget share is measured in thread CPU time like the budget itself. Two kernel indexes are now
identical (8,529,723 nodes, 1,974,940 CALLS, zero churn), as are two elasticsearch indexes.
Findability. search_graph's BM25 path honours the `label` filter and ranks an exact-name definition
above test methods that repeat the token (15/15 probe queries).
Coverage. The parse-coverage gap tree subtracts definitions the walker recovered inside an error
region (multi-line only); kernel/sched/core.c reports 476 flagged lines instead of 7,674.
Precision. Config-language callers (Makefile, CMake, YAML) no longer bind unique_name into code
(kernel Makefile -> C CALLS 195 -> 0). JS-family token names ({}, 1, (x: T) => U) are not
definitions; computed, quoted and escaped member keys stay (TypeScript 204 junk -> 0, 1,782 real
names kept). Names are cut at their first line (elasticsearch YAML 8,306 -> 0); a comment inside an
argument list is not an argument, so no Route node is named by a Java block comment; the wildcard
route `/*` is still a route. HTML/Vue/Svelte/Astro embedded member calls stay unbound.
Recall. A Python call through a self- or cls-rooted attribute keeps a specific unique_name member
target (Django +751, elasticsearch +3,288 CALLS, no new callee names).
Robustness. Edge properties never carry a split UTF-8 sequence (the 120-byte expression cut and the
JSON escaper both trim a partial code point). CLI array flags accept a JSON-array literal. Cypher
seeds a single-hop pattern from its selective end (kernel `MATCH (a)-[:CALLS]->(b) WHERE b.name =
'_printk'`: execution limit -> 7.9 s).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Scala JAX-RS resources declare @Path on the class exactly like Java and
Kotlin, but the class-level prefix was only joined with the method path
for those two languages. Include Scala so a method route is emitted as
`<class path>/<method path>` instead of the bare method template.
Adds a Scala probe test covering both an empty and a non-empty method
template.
Signed-off-by: Hubert Tarnacki <hubert.tarnacki@gmail.com>
JAX-RS allows @Path values without a leading slash, both on the class
and on the method. The route extractor rejected such literals, so
resources such as `@Path("widgets")` + `@Path("{id}")` produced no Route
nodes and no HANDLES edges at all. Accept relative templates and
normalize the class/method composition so the resulting path is always
rooted.
Existing JAX-RS Java probe tests are updated to use relative templates
so they guard the regression.
Signed-off-by: Hubert Tarnacki <hubert.tarnacki@gmail.com>
cbm_gbuf_dump_to_sqlite() emitted gbuf.dump regardless of the writer's
result, so a run that published nothing still logged node and edge counts
as if it had worked, and the failure reached the user only as the generic
"Pipeline failed. Check repo_path exists and contains source files."
The writer's temp -> final rename in publish_writer_output() returned a
bare ERR_WRITE_FAILED. #1628 taught cbm_rename_replace() to translate the
platform error into errno so callers could report why, and wired up the
stage -> final rename in pipeline.c. This is the other rename on that
path, and it is the one that runs first.
Preserve errno across the cleanup in every publish error branch. That
includes cbm_writer_open(), where the cleanup unlinks a file that was never
created: without the save it leaves ENOENT behind, so a directory that
denied the create is reported as a missing path — the #1620 case, stated
wrongly. Read the value in graph_buffer.c before the profiling macro can
clobber it, and report gbuf.dump_failed with the return code and that errno.
A sticky append failure clears errno rather than attach a reason it does not
have, and the field is omitted when there is none. No logging is added
inside internal/cbm/, which has none today.
Tests cover both halves: the dump reports a failure instead of a summary,
the writer names its truncation reason, and the publish rename's errno
survives a cleanup that itself fails.
Refs #1620, #2001.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FxkVF773gDGNmBmsLnzN7
Signed-off-by: juan Carlos Estrada Montoya <junk151516@users.noreply.github.com>
Collect a bounded set of conventional export macro candidates and inject empty definitions into the existing C/C++ preprocessing second pass without overriding explicit caller definitions.
Conservatively reconcile remapped definitions to recover hidden classes, structs, enums, free functions, and inline methods while suppressing matching base-class phantom callables.
Add focused regression coverage for supported suffixes, ordinary all-caps negative controls, candidate limits, comments, strings, raw strings, overlong names, explicit define priority, and C/C++ extraction.
Rebase onto main 92abefa3 and route the temporary superseded-definition bitmap through the extraction memory class to comply with the current allocation contract. Preserve the error-region dropped counter initialization.
Signed-off-by: XIYBHK <xiybhk@163.com>
Martin Vogel