main does not compile. internal/cbm/lsp/type_registry.h declares
type_short_buckets, type_short_entries and type_short_bucket_count twice, and
cbm_registry_types_by_short_name twice with different parameter types.
Two changes implemented the same feature independently:
a4c0ffbc perf(cs): eliminate the C# corpus-proportional scans (2026-08-16, ours)
CBMTypeShortIter, cbm_type_short_iter_next
793716dc perf(lsp): index C++ short-name type lookups (PR #1679, astandrik)
CBMTypeNameIter, cbm_type_name_iter_next
They landed in different hunks of the same struct and the same header section,
so git merged them with no conflict and nothing failed until the merge commit
was built. This is the failure mode where a clean merge is not evidence that
two changes compose.
The two iterators are structurally identical -- same five fields, byte-identical
_next bodies reading the same reg->type_short_entries chain. They differ in one
place, and it matters:
#1679 on aux-index allocation failure, sets tail_i = 0, so the
iterator degrades to a full types[] scan and the candidate
set is unchanged.
a4c0ffbc sets tail_i = reg->type_qn_entry_count unconditionally, so on
that same failure it SKIPS the first type_qn_entry_count types
and silently yields an incomplete candidate set.
So this keeps a4c0ffbc's names -- CBMTypeShortIter reads correctly beside its
type_short_* members and beside CBMTypeEmbedIter and CBMFreeFuncIter -- and
#1679's body, which is the correct one. The surviving declaration carries both
rationales: the C++ short-name lookup and cs_resolve_type_name's step-9
fallback are now stated as two consumers of one index, which is what they are.
The merge also broke two comment/member pairings, repaired here: the duplicated
members had displaced the free-function index members from under their own
comment, and the CBMTypeShortIter block had been interleaved between
CBMFreeFuncIter's typedef and its declarations.
No behaviour change for either consumer beyond the allocation-failure path,
which strictly gains coverage for C#. c_lsp.c and cs_lsp.c now call one
function. #1679's tests, including the fault-injection case that NULLs the
buckets and asserts the full-scan fallback, carry over unchanged apart from the
type name.
Local: c_lsp, cs_lsp and registry -- 881 passed, 0 failed.
Co-Authored-By: astandrik <astandrik@yandex-team.ru>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Go calls written through an aliased struct field (h.svc.Method(ctx,
req)) previously fell through to fuzzy suffix matching because:
- the per-file Go dispatch walk was bypassed, so field selectors were
never descended;
- struct fields extracted as flat "Field" defs were dropped by the LSP
label map, so every Go struct registered with zero fields;
- receiver types written through import aliases (svc.Svc) produced
qualified names that exist nowhere in the project registry.
Changes:
- restore the Go walk in cbm_pxc_dispatch_file and descend field
selectors through the shared prebuilt registry, emitting
lsp_type_dispatch edges;
- fold per-field defs into their owning struct inside
cbm_pxc_collect_all_defs (one site covers both the prebuilt-registry
path and the per-file fallback);
- re-qualify the last alias segment through the calling file's import
map when the direct receiver-type lookup misses
(go_requalify_via_imports).
Tests:
- tests/test_parallel.c: cross-package field chain resolves end to end
with an lsp_type_dispatch edge in the parallel resolver;
- tests/test_go_lsp.c: alias-typed struct field re-qualifies through
the import map and lands on the real receiver type.
Signed-off-by: xkchok <143325001+xkchok@users.noreply.github.com>
Resolves conflicts with the Chialisp language addition (#1920) in three
repro ledgers. Both branches added one call-capable language, so beyond
the textual conflicts, the auto-merged shared totals had to be bumped a
second time:
- repro_language_registry.c: capability partition now 88/27/49,
EXPECTED_CALL_CAPABLE_LANGUAGES 115, EXPECTED_NON_CALL_LANGUAGES 51,
EXPECTED_MATRIX_B_ROWS 49
- repro_call_node_manifest.c: historical total 226, active primary 196,
direct calls 160, constructor calls 22, primary owners 196
- repro_call_argument_matrix_b.c: ROUTINE_ARGUMENT_LANGUAGE_COUNT 39,
matrix static assert 49 (RACKET..CHIALISP)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Refresh the complete generated Perl grammar bundle from upstream v1.2.1 at c3e17b31179bf8f658c9f37c7a3ea6a202212d5a, raising it from ABI 14 to supported ABI 15. Preserve the existing license and record the source and per-file provenance.
Add deterministic coverage for a valid format followed by a named sub, while retaining a malformed-source control.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Python attribute calls were not classified as member calls, so weak
short-name strategies (suffix_match / unique_name / field_type_hint)
could bind `obj.method()` to an arbitrary same-named project symbol —
`accelerator.print()` resolving to MockAccelerator.print.
Flag a Python attribute call is_method when its receiver is unresolved,
and extend the existing TS/JS weak-member guard to Python. Receivers that
ARE known stay exempt: self/cls, super(), and any identifier bound by one
of the file's imports (including the root of an attribute chain), so
Python's canonical `module.function()` cross-file call still resolves.
The guard helper is renamed cbm_tsjs_suppress_weak_method_match ->
cbm_suppress_weak_member_match. Its drop-list is unchanged, and the
per-language gate stays at the two call sites, which enumerate the same
language set in pass_calls.c and pass_parallel.c so the sequential and
parallel resolvers cannot diverge. ArkTS remains in that set.
python/S6/inherited_method in the resolution probe now asserts calls == 0,
recording the same trade TypeScript took when the receiver-aware guard
landed: the edge it used to produce came from a unique_name registry
fallback (measured on main: strategy=unique_name, cands=1, conf=0.7500)
where a weak short-name guess happened to be right in a 2-file fixture.
It flips back to >= 1 once py_lsp_cross resolves inheritance, matching the
flip-back note already carried on the TypeScript S6 case.
Fixes#1276
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: Enferlain <15861396+Enferlain@users.noreply.github.com>
Closes#518 (Section nodes don't index body text) and #519 (META.yaml /
frontmatter description values not indexed). A question asked in words
could not reach the nodes that hold the words: Section carried only its
heading, a config Module only its path, and both labels were filtered out
of BM25 results outright.
nodes_fts gains a fifth column, `body`, backfilled from each node's
EXISTING properties.docstring. No new column on `nodes`, no new table, and
CBM_INDEX_FORMAT_VERSION deliberately does NOT move -- nobody is forced to
reindex. A legacy four-column table survives untouched under CREATE VIRTUAL
TABLE IF NOT EXISTS: the backfill probes for the column before naming it,
and bm25()'s surplus fifth weight is inert on a table that has no fifth
column (FTS5 reads a weight only for a column an instance landed in). Such
a database keeps opening and searching, just without prose.
Two node kinds gain the docstring they lacked:
- Markdown section bodies (#518) -- the blocks following a heading, up to
the next heading or subsection, collapsed to single spaces and capped at
MAX_COMMENT_LEN (500 B) with a UTF-8 backoff that never splits a
multi-byte sequence.
- Top-level YAML/JSON description / summary / purpose (#519), promoted
onto the file's Module node.
Section and Module are no longer excluded from BM25 results. Both the
ranked query and the mirrored count query were changed together; removing
the filter from only one desynchronises results from counts. Column weights
are bm25(nodes_fts, 1.0, 1.0, 1.0, 1.0, 0.3): the four identifier columns
keep parity with each other and prose sits well below them, so a prose-only
hit surfaces but never outranks an identifier match.
The FTS backfill is consolidated into one writer, cbm_store_fts_rebuild(),
used by the wholesale rebuild AND the delta merge. That second site matters:
it had its own hand-written four-column INSERT, which with a fifth column
would leave body NULL for every node arriving by delta -- silently, on the
warm path most users hit, while a full reindex looked perfect. There is now
exactly one place the column list is decided, and a test that fails if the
delta path stops writing prose.
The backfill guards json_extract with json_valid: malformed properties JSON
exists in pre-fix databases and would otherwise abort the whole rebuild.
Tests: markdown section bodies (own body, stops at the next heading, UTF-8
safe cap, empty body stays NULL), YAML/JSON description promotion (block
scalars, synonym fallback, top-level only, config-only), FTS body indexing
and its column isolation, malformed-JSON tolerance, legacy four-column
degrade, incremental watermark, BM25 finding Section and Module by prose,
results/total consistency, identifier-outranks-prose, and the delta-merge
regression.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: ShauryaaSharma <196896785+ShauryaaSharma@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The line read `7x ABI-13, 86x ABI-14, 65x ABI-15`, which sums to 158 against
161 vendored grammars. Counting `#define LANGUAGE_VERSION` in each parser.c
gives 9 / 79 / 73 = 161, so the line was already wrong before Chialisp; bumping
the ABI-14 bucket by one, as the Chialisp change did, would have carried a
four-off error forward under a fresh timestamp.
Recount with Chialisp included: 9 / 80 / 73 = 162, which matches both the
grammar-directory count and the language-count contract. The regeneration
command is now recorded next to the numbers so the next person does not have to
rediscover it:
grep -h '#define LANGUAGE_VERSION' internal/cbm/vendored/grammars/*/parser.c \
| sort | uniq -c
Vendored-integrity gate and the language-count contract both re-run green.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Chialisp (.clsp/.clib/.clinc) is the Chia smart-coin language. The only
public grammar, Quexington/tree-sitter-chialisp, cannot parse it: on the five
reference files from chia-blockchain@main it produced 3-34 ERROR nodes each.
Three defects account for all of them, and each desynchronised the rest of the
file rather than losing one node:
* comments were `/;.*\r\n/` -- CRLF-only, while real files are LF;
* `(include foo.clib)` was rejected because of the dot in the filename;
* `(defconstant NAME <expr>)` accepted only a primitive after the name.
So the grammar is written here instead, in tools/tree-sitter-chialisp/, and
files as first-party: its vendored LICENSE is the project's own, byte-identical
to the repository root, carrying no third-party copyright.
It is deliberately GENERIC -- source_file / list / symbol / string / number /
hex / dot / comment, modelling the clvm_tools reader rather than the Chialisp
form vocabulary. `mod`, `defun`, `defconstant` and `include` are ordinary head
symbols; which lists are definitions is decided in extract_defs.c. A dialect
that adds a form (cl-21 -> cl-23 -> cl-26 each added several) therefore needs
no regenerated parser. Acceptance: all five reference files parse with zero
ERROR and zero MISSING nodes; the three defects above are pinned as tests.
Extraction, and the four decisions in it:
* `Constant` is admitted to the cross-file name registry
(cbm_label_is_registry_symbol). Chialisp's condition codes live as
`(defconstant ...)` in a shared .clib and are referenced by name from every
puzzle; labelled but unregistered they are seeded nowhere, and roughly half
of a Chialisp graph would be present as nodes yet never the target of an
edge. All three registry-seeding sites route through that one predicate, so
they cannot diverge. No other language emits `Constant`, so the blast radius
is exactly Chialisp. Pinned by a two-file graph test that goes red when the
label is removed from the predicate.
* `export` / `namespace` are NOT definition heads. `(export foo)` names a
function already defined in the same file, so treating it as a def head
mints a second node for one symbol. They stay in the not-a-call filter.
* A parameter list is a `list` too, but it binds rather than invokes. Without
that gate `(defun check_conditions (HEIGHTLOCK conditions) ...)` mints a
phantom call to HEIGHTLOCK, and real Chialisp is mostly such definitions.
* `(include *standard-cl-26*)` is a dialect directive, not a file dependency;
recording it would invent a dependency on a file that does not exist.
Three helpers that the defs, calls and unified walks all need -- the quote-data
gate, comment-skipping named-child access, and the Chialisp def-head set -- live
in helpers.c rather than being copied per translation unit. They previously
existed as byte-identical duplicates that had already drifted: extract_defs.c
skipped comments when reading a def head while its mirror in extract_unified.c
did not, so a comment between a head and its name desynchronised defs from call
scope. Both now use the same accessor.
Ledger: 161 -> 162 grammars across the seven published surfaces, ABI-14 86 -> 87,
first-party 13 -> 14, CBM_LANG_COUNT 164 -> 165.
Distilled from PR #919 by darkverbito, whose extraction logic -- the CLVM
not-a-call vocabulary, the quoted-data gate, the filename-derived `mod` name and
the include/embed import handling -- is kept and is the substance of this change.
The grammar that PR vendored is not used.
Co-authored-by: darkverbito <8049985+darkverbito@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Merging main brought in three contracts this branch violated.
repro_call_node_manifest requires a table row for every live
call_node_types entry, so Pkl's unqualifiedAccessExpr/qualifiedAccessExpr
(direct) and newExpr (constructor) each get one, with the partition totals
moved to match. The historical-snapshot column is the audited ledger, not
an archaeology record -- the suite hard-fails on a row that claims it was
absent -- so a newly registered kind joins the snapshot with the change
that registers it.
repro_language_registry then requires every call-capable language to own
exactly one call-argument matrix row, which turned out to be a real
extractor gap rather than a bookkeeping entry. Pkl's grammar labels no
production with a `name` field, so the generic declared-container rule
never recognised a Pkl binding: method names, parameter names, and
property names were each re-emitted as an ordinary read of themselves,
which Go and Nickel do not do. Add a Pkl occurrence policy that binds the
declared identifier of methodHeader, typedIdentifier, classProperty,
objectProperty, clazz, and typeAlias, resolved against the nearest such
container so annotations, defaults, and bodies stay reads. Recording the
binding also supplies the lexical-shadow proof the matrix row asserts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Tony Thayer-Osborne <tony.thayerosborne@conductorone.com>
Pkl was wired end-to-end but shallow: on a real-world 172-file Pkl corpus
the graph had 386 IMPORTS edges and 0 CALLS edges, because pkl_call_types
was empty_types.
Pkl has no dedicated call node. `unqualifiedAccessExpr` and
`qualifiedAccessExpr` are the same node whether they are a call
(`helper(a)`) or a bare property read (`host`) — the only discriminator is
an `argumentList` child. extract_pkl_callee gates on that child, and is
dispatched from extract_callee_name with an unconditional return: falling
through to field-based or generic first-identifier resolution would mint a
CALLS edge for every property read in every Pkl file, since a bare access
expr's first child is an identifier.
Callee resolution:
helper(a) -> "helper"
utils.fallback(a) -> "utils.fallback" (module-qualified; cbm.c
shortens to the last dotted segment)
s.trim().toLowerCase() -> "toLowerCase" (a receiver that is itself a
call has parens in its text and is not prefixed)
new Server { ... } -> "Server" (links to the class def)
Also fills the other empty slots in the Pkl spec, all confirmed against
real parse trees rather than guessed: typeAlias (class), importGlobClause /
importExpr (import), ifExpr / whenGenerator / forGenerator (branch),
throwExpr (throw), annotation (decorator). forGenerator is additionally
registered in cbm_is_loop_node_type so `for (x in xs)` counts toward
loop-nesting depth; the name is Pkl-unique so no other grammar collides.
The Pkl repro test moves from the structural battery (dims 1-5) to the
full callable battery (dims 1-8) and adds an inline negative assertion
that bare property reads do NOT become CALLS edges — the regression the
argumentList gate exists to prevent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Tony Thayer-Osborne <tony.thayerosborne@conductorone.com>
Distilled from #1412 by @allanoepping. Fixes a real recall hole: tag-dialect
.cfc components were routed to the cfscript grammar (extracting nothing),
and <cfscript> bodies inside tag-dialect files are opaque cf_script_content
to the cfml grammar — whole legacy components produced zero functions.
Fix 1 — his .cfc dialect sniffer, nearly verbatim: cbm_disambiguate_cfc()
reads a 16K head via cbm_fopen; <cfcomponent>/<cffunction> anywhere means
tag dialect; a leading <cfscript (after whitespace and <!--- ---> comments)
or no markup at all keeps the old CFSCRIPT routing, so script-dialect files
and read failures behave exactly as before. Wired in detect_file_language
beside the existing _m/_cls/_inc sniffers, with his 8 direct tests.
Fix 2 — his embedded-defs capability, re-based onto the shared machinery
that superseded his mechanism mid-flight (#1852, merged after his last
rebase): a cfml_embedded_imports spec row (cf_script_tag/cf_script_content
-> CFSCRIPT), CFML added to the embedded-imports dispatch, and the
extract_structure gate extended to CFML. Included ranges on the original
buffer give absolute coordinates for free, so his line-number assertions
hold verbatim, and calls inside blocks become extractable as a bonus. His
~90-line slice+shift implementation is deliberately NOT carried — it
mirrored the sanctioned shape of its day, which #1852 replaced.
One correctness guard the port needed: the Vue attribute resolver is now
gated to Vue — cf_script_tag carries no lang=/src= attributes, and running
the resolver would have silently rewritten CFML's embedded language from
CFSCRIPT to the resolver's JavaScript default.
His embedded extraction test (line remap + <cfsetting> ERROR-cascade case)
carries over verbatim and binds: reverting the spec/gate wiring fails it at
ASSERT(has_def(r, "Function", "greet")). Suites: extraction, language,
edge_imports, lang_contract, grammar_labels, grammar_regression — 637/0.
Co-authored-by: Allan Oepping <allan@intellipay.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Vendor AndreasMaierDe/tree-sitter-plsql @ 28aebef209be (MIT, ABI 14, no
external scanner) and wire CBM_LANG_PLSQL end to end:
- extraction onto existing labels only: package spec/body -> Class,
procedure/function (create/definition/declaration) -> Function,
file -> Module; callee extraction for ref_call incl. the
package-qualified pkg.name form; identifier usage vocabulary with the
upstream 'parameter' kind (a ref_call ARGUMENT wrapper) carved out of
the whole-binding rule
- discover: .pks/.pkb/.pck/.pls/.plb/.plsql/.fnc/.trg/.bdy/.tps/.tpb
(.sql stays SQL and .prc stays FORM, both asserted negatively)
- ledgers: CALL_WITHOUT_REFERENCE_VOCAB(PLSQL), call-node manifest
ref_call row (historical total 219 -> 220), call-argument matrix B row
- tests: extraction triple (package body + package-qualified call,
standalone function, honest negative pin of the upstream
CREATE TYPE ... AS OBJECT ERROR-node limitation), grammar-regression
case, label golden Class:1,Function:1,Module:1, MUST_EXTRACT_DEFS,
extension mapping, bug-repro board battery row
- MANIFEST.md (community-niche verdict, local include-quote patch row),
THIRD_PARTY.md section, refreshed vendored checksums
PL/SQL support was contributed by Oguz (@ouzsrcm) in #1033; re-vendored
from upstream per vendoring policy with his language wiring distilled on
top.
Co-authored-by: Oğuz <oguzhan.saricam@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Vue single-file components yielded a Module node and IMPORTS edges but no
Function definitions or CALLS edges from the `<script>` block, so a component's
call graph was empty (#1410).
Adds a control test pinning current behaviour for Svelte, HTML and Astro, which
still extract imports but no functions or calls. That is deliberate scoping,
not an oversight: it documents the remaining gap rather than hiding it, and it
proves this fix does not leak into sibling languages. Svelte is tracked by
#1807 and will be a follow-up.
Re-opened from #1843, which carried no Signed-off-by and so failed DCO. The
tree is byte-identical to that commit; only the sign-off was added.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The vendored LICENSE carried tree-sitter-typescript's MIT text plus an
appended derivative notice. That is not what MIT asks of a derivative -- it
asks that upstream's notice travel with the code -- and the appended text
meant the provenance audit could not byte-verify the file against upstream,
so arkts landed as NO-MANIFEST-ENTRY and failed license-gate.
The vendored LICENSE is now byte-identical to upstream's, and arkts is
registered in the audit's FORKS map, which is defined for exactly this case:
a self-maintained fork whose vendored LICENSE must match its upstream. That
is a routing entry, not an exemption -- check_upstream still byte-verifies
against the real upstream repository, so the fact is checked rather than
asserted. Audit now reports arkts as IDENTICAL.
Our copyright for the ArkTS additions is not lost, it moves to where our
SOURCE is: the grammar.js header (which now states all three copyrights
explicitly) and THIRD_PARTY.md, which ships inside the release archives.
The edit to scripts/audit-license-provenance.py is a maintainer-only
enforcement-surface change, made deliberately and stated here rather than
slipped in; a contribution may never supply its own audit entry.
No generated parser or source changes -- comments, licence text, manifest
prose and one routing entry only.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Routing .ets through the plain TypeScript grammar produced no definition
node for any @Component struct (0/19 across the OpenHarmony sample corpus),
parsed struct members as statement labels and build() as a call expression,
and registered ArkUI builtins (Column, Row, ListItem, ...) as user function
definitions with 28% fabricated cross-file CALLS edges. This adds a
dedicated ArkTS grammar that keeps everything TypeScript already handles
and models the ArkTS syntax TS cannot parse.
Grammar (tools/tree-sitter-arkts/, generated parser vendored to
internal/cbm/vendored/grammars/arkts/, ABI 15 via tree-sitter-cli 0.25.10):
first-party derivative of tree-sitter/tree-sitter-typescript v0.23.2
@ 75b3874edb2d (the commit our vendored typescript/tsx come from) on the
tree-sitter-javascript v0.23.1 base @ 3a837b6f3658 (the version upstream's
package-lock pins). LICENSE carries the upstream MIT text verbatim plus a
derivative notice; the external scanner is a symbol-rename trampoline over
a byte-identical copy of the already-vendored typescript scanner header.
ArkTS additions: decorated struct component declarations with class-shaped
bodies; UI-DSL trailing-closure component calls with attribute chains after
the closing brace; decorators on function declarations (@Builder,
@Extend(T), @Styles); @Extend/@Styles leading-dot attribute-chain bodies;
anonymous stateStyles style blocks; import lazy named imports.
Wiring: CBM_LANG_ARKTS enum + lang spec (struct_declaration as a class
kind labeled Struct, public_field_definition fields), .ets extension,
LANG_NAMES/userconfig entries, grammar shim, and TS-family joins across
the extractors and the name-resolution guards (parse_es_imports, JS
keywords/test-file/func-kind/scope tables, TS heritage/param/receiver/
namespace/type-ref/channel/usage paths, js_ts_family and the weak-method-
match suppression). Field definitions now also carry their decorators, so
@State/@Prop/@Link members are findable with their roles.
Tests: grammar corpus (35 cases), per-language regression + label golden +
import net + extension cases, seven ArkTS extraction tests including the
phantom-builtin regression (ArkUI builtins must never be definitions), and
capability-ledger / call-node-manifest / call-argument-matrix rows. On the
14-file OpenHarmony sample corpus the TS-grammar baseline had 12/14 files
with parse errors (231 ERROR nodes); the arkts grammar parses 14/14 clean,
and an end-to-end index yields all 19 components as Struct nodes, 68
methods, 119 fields, and zero phantom builtin definitions.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Add regression tests verifying index supervisor behavior when a worker exits
with a nonzero status:
- TEST(index_recovery_quarantines_exit_nonzero): proves a single file causing a
nonzero exit is quarantined with phase 'error' while remaining files are indexed.
- TEST(index_recovery_systemic_exit_nonzero_gives_up): proves systemic nonzero
exits cause the supervisor to give_up without false quarantines.
Also document the deterministic first-file progressive quarantine tradeoff in
mcp.c supervisor comment as requested during review.
Signed-off-by: Shashwat Mishra <shashwat78.nits@gmail.com>
handle_url_builders ran for every vendored grammar, so a C helper
returning an absolute pathname became a URL builder and minted a Route
node plus an HTTP_CALLS edge in a language that speaks no HTTP. Gate
recording to JavaScript, TypeScript and TSX.
Builder entries now carry a flag in the per-file constant map: only a
call_expression argument resolves one, so handing a builder to a
callback builds no request. A builder whose returns are not all
route-shaped literals is declined instead of attributing its one literal
to call sites that take the computed branch.
Five negative tests cover the cross-language case, the mixed builder,
the ambiguity tombstone, the builder reference and the non-URL helper.
Signed-off-by: Charles Queiroz <fcqueiroz@liquibase.com>
detect_url_in_args() (the arg_url HTTP_CALLS emitter used when the callee
is a local client wrapper, not a known HTTP library) reads per-arg values,
not first_string_arg. Resolve call_expression args through the builder map
and flatten template_string args to the {} form there as well, so wrappers
like apiFetch(buildPath(id)) emit the edge and join the canonical route.
Signed-off-by: Charles Queiroz <fcqueiroz@liquibase.com>
URL-shaped literals returned from small builder functions never reached
call arguments, so client(buildPath(id)) produced no HTTP_CALLS edge and
no Route node, static or template alike.
handle_url_builders() records builderName -> returned URL in the same
per-file constant map used for module-level consts, covering return
statements and arrow expression bodies; a call_expression branch in
extract_url_or_topic_arg() resolves client(buildPath(id)) through it.
Composed builders inline already-recorded substitutions and truncate the
query string, so `${basePath(id)}?${params}` joins the server route
exactly. Ambiguous builders (two different URLs) are tombstoned.
Signed-off-by: Charles Queiroz <fcqueiroz@liquibase.com>
A Swift protocol requirement — a bodyless `func generate() -> String`
inside a protocol — was absent from the graph entirely. Swift codebases
are heavily protocol-driven, so the requirement is very often the
declaration a reader is looking for, and "who declares generate()" had no
answer.
Three edits deliver it: protocol_body joins the class-body types so a
protocol's members are walked at all; protocol_function_declaration joins
swift_func_types, because extract_class_methods gates on that set and
would otherwise walk the requirement and discard it; and both name
resolvers accept the node, which has the same simple_identifier shape as
function_declaration.
Distilled from #613 by xbsjason. That PR also carried an enum/struct half
whose two assertions fail on CI: tree-sitter-swift has no
struct_declaration or enum_declaration node type (it models both as
class_declaration), so the corresponding swift_class_types entries are
inert and a bare enum is still labeled Class. That is a real pre-existing
modelling gap, left untouched here and now documented at the dead entries
rather than silently deleted.
Also dropped from the original: a find_first_descendant_by_kind rescue in
extract_class_methods. Removing it changes no test outcome once the
function-type entry is present, so it was carrying nothing.
Advances #43.
Co-authored-by: xbsjason <xbsjason@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A dbt model is an ordinary .sql file whose dependencies are written
{{ ref('other_model') }} or {{ source('group','table') }}, never as
literal table names. The SQL grammar cannot read those -- FROM {{ ref(
'x') }} is a parse error to it -- so the dependency structure of an
entire dbt project was invisible to the graph.
extract_dbt.c runs as a sub-extractor inside the normal indexing
pipeline. Per qualifying file it emits one Model definition (named by
the file stem, dbt's own model identity) plus one usage per ref()/
source() call; pass_usages resolves those into model -> relation
lineage edges like any other reference.
Model is a relation label alongside Table and View, which is what makes
the rest work without new plumbing: registry seeding, the central
relation veto, the incremental surface hash, search ranking and the
architecture queries all pick it up from cbm_label_is_relation. Sharing
one label class also means a model's source('raw','customers') resolves
onto a Table declared in a plain DDL migration in the same repository,
so dbt lineage and SQL lineage form one graph rather than two, while
the veto keeps model names out of every non-lineage consumer.
The pass is self-gating: SQL files only, and only those carrying a real
ref()/source() call. The dbt builtins are themselves the evidence, which
is cheaper than a dbt_project.yml lookup and more precise -- templated
SQL that is not dbt (an Airflow {{ ds }} parameter) produces no Model
node and no usages even inside a dbt repository.
{% macro %} definitions are deliberately excluded: the vendored
tree-sitter-jinja2 grammar has no node types for {% %} statements, so
they can only be recovered by a hand-written scanner that cannot see
comments or find {% endmacro %} for a correct span. Filed as follow-up
rather than approximated.
Tests cover the lineage, the last-string-argument semantics of both
builtins, the gate (against a plain-SQL control extraction), plain DDL
staying untouched, and an end-to-end pipeline case asserting model ->
model across files, model -> Table onto plain DDL, and cross-language
isolation. Disabling the pass reddens three of them; removing Model from
the relation set breaks lineage outright.
Implements the lineage half of #575.
Co-authored-by: alexisperinger-ux <alexis.peringer@iss-stoxx.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
CREATE TABLE / CREATE VIEW / CREATE MATERIALIZED VIEW now produce Table
and View nodes (previously generic Variable), CREATE PROCEDURE produces
a Function, and schema-qualified DDL names (schema.table) are named by
the table identifier instead of the schema. A view's FROM/JOIN relations
are emitted as usages and resolve into view -> table USAGE lineage
edges.
Relations join the cross-file name registry so lineage can resolve, with
two structural safeguards:
- Registry membership is defined once by cbm_label_is_registry_symbol
(helpers.c); the full, parallel and incremental seed sites all call
it, ending the KEEP-IN-SYNC copies the old label lists required.
- The default cbm_registry_resolve vetoes relation-labeled results:
common table names (users, orders, config) collide with code
identifiers in every language, and no CALLS/USAGE/READS/WRITES/THROWS/
handler/decorator consumer may bind them. The SQL lineage path opts in
through the new cbm_registry_resolve_lineage.
Table/View also join the registry-only per-file LSP surface labels so a
table rename invalidates dependent SQL files on incremental (no stale
lineage edges), rank with the type tier in BM25 search, and appear in
the architecture boundary/package/cluster queries via the pinned
CBM_SQL_RELATION_LABELS fragment.
Tests: extraction trio (labels, lineage usages, schema-qualified names),
grammar golden + probe updates, relation-label contract pin, and two
pipeline tests: cross-language isolation (binding: fails without the
veto) and incremental table-rename stale-lineage.
Closes#574.
Co-authored-by: alexisperinger-ux <alexis.peringer@iss-stoxx.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
ts_eval_expr_type and ts_signature_for_call are mutually recursive: resolving
a call evaluates its argument expressions once per lookup path (method
dispatch + namespace fallback), and in tsc-compiled spread files the first
argument is itself the next nested Object.assign(...) call — the same subtree
re-evaluates once per enclosing level, 2^n total. The TS suite's
objectSpreadRepeatedComplexity.js (3.6 KB, 5 nodes) measured 20.4 s; with the
memo its eval cost is zero within measurement noise of a one-file control,
and the microsoft/TypeScript corpus drops 37.5 -> ~24 s warm (nodes
byte-identical, edges within the known scheduler jitter).
Expression types are position-pure within a file pass (one node = one scope
path; the per-file walk is single-threaded and deterministic), so one eval
per node is the correct semantics, not a cache trade-off. The memo is a
per-file, arena-backed, linear-probe table keyed on TSNode.id. Results
produced under a depth-cap or budget bail are never stored: both bail sites
bump a degradation counter, and a store only happens when the subtree
completed clean — a degraded UNKNOWN can therefore never shadow a later full
evaluation.
The regression guard asserts work, not wall-clock: the nested-Object.assign
shape must complete without exhausting the deterministic eval budget, read
back through a new CBM_ENABLE_TEST_SEAMS accessor pair. The seam lives in the
lsp_all unity object, so GRAMMAR_CFLAGS_TEST/TSAN now carry the seams define
(test artifacts always have seams; prod never does). Verified RED without the
memo (budget exhausted, suite 47.8 s) and green with it (3.2 s).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_arena_alloc(arena, 0) returns NULL, so the java and cs cross-registry
builders read a def_count of 0 as OOM and returned NULL — a corpus with no
files of that language silently lost its shared registry (and the seal tests
caught exactly that: cbm_cs_build_cross_registry(&arena, NULL, 0) == NULL).
Guard the partition alloc behind def_count > 0; the empty registry is still
built, finalized, and shared.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
ts_eval_expr_type was depth-capped but work-unbounded: crafted expressions
(the TS test suite's repeated object spreads) stay under the depth cap
while fanning out. Charge the same per-file budget the type-text parser
uses, at 16 units per entry (an eval entry does ~two orders of magnitude
more work than a text-parse unit), degrading to UNKNOWN on exhaustion.
Honest status: this hardens the documented budget design, but the known
3.6 KB spread-bomb baseline file still measures ~11 s in-corpus — its
entry path into the evaluator apparently runs unarmed and is recorded as
an open lead (zero budget warnings observed). Suites green (ts_lsp,
extraction, complexity).
Refs #1669.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two measured changes on the TypeScript corpus (81,397 files; v0.9.0 17.9 s,
before 44.1 s, after 37.5 s):
- cov_rebuild_shadow_graph upserted every directory segment for every
failure row — 13,243 parse-partial baseline files under one tests/
subtree meant ~80k redundant node/edge round-trips, 9.1 s of a 9.2 s
coverage_replace. An in-rebuild path->id map creates each directory once;
identical graph (edges deduped by unique key before, absent now).
Coverage block 9,162 -> 2,920 ms. Sub-block timings
(publish.timing.coverage: del/rows/prune/meta/commit + row_count +
detail_bytes) are kept — the caller-level number could not name the
culprit.
- JS/TS export_statement is an import CONTEXT only in its re-export forms
(source field, or a bare specifier list without a declaration). The old
is_export_of_declaration blacklist missed TS-only forms
(ambient_declaration, function_signature, module_declaration), running
declare-heavy subtrees (.d.ts, export namespace) behind inside_import:
suppressed usages + per-identifier ancestor walks. Positive detection
replaces the blacklist; +3,954 restored usage edges on the corpus,
nodes identical. (Measured perf-neutral here — kept for correctness.)
Suites green incl. store_nodes/edges/search, mcp, extraction, ts_lsp,
complexity, and the 53-language calls-breadth contract.
Refs #1669.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Four measured changes on the dotnet/runtime corpus (58,656 files; every
step's numbers below are from full-corpus runs on the same host, baselines
captured this session; v0.9.0 = 444.3 s):
1. Two-phase cs registry build (types -> finalize -> funcs), the same
pre-finalize linear-lookup disease fixed for Java: prepare 280 -> 140 s.
2. C# import-context carve-out: cs_import_types lists namespace_declaration
(for namespace-name mapping) and using_statement (C#'s RAII block — a
grammar-name collision), so EVERY namespaced C# file's whole body ran
with inside_import=true. That both suppressed ordinary usage extraction
under namespaces and sent every identifier through the ancestor-walking
import-binding check (tree-sitter's ts_node_parent re-descends from the
root, so wide files went quadratic: one 147 KB JIT torture file cost
490 s; 6.8 s after). Only using_directive / namespace_use_declaration
open an import scope now. Restores the suppressed usages:
edges 4,291,387 -> 5,869,093 (+37%), nodes identical.
3. The usages walker maintains call/import ancestry as enter/exit counters
on its explicit stack instead of per-node ancestor re-walks
(extract_usages.c had grown from 6 to 100 ts_node_parent calls since
v0.9.0; the two per-node gates are now O(1) with semantics preserved —
strict ancestors only, emit before self-count).
4. Registry short-name indexes replace the two remaining full scans:
cs_lookup_extension walked all 963k funcs per unresolved invocation
(now the existing free-func short-name iterator, first-match order
preserved via min-index selection), and cs_resolve_type_name's step-9
fallback scanned every type per unresolved name IN BOTH the builder and
per-file resolution (new type_short index in finalize, same
reverse-insertion ascending-order pattern, best-score ties keep the
first-in-registration-order winner). Builder 140,095 -> 500 ms; resolve
cross-LSP CPU 5.5M -> 317k ms (us_per_file_per_kdef 191 -> 11).
End state: 449.9 s wall (1.01x of v0.9.0) with +48.6% edges vs v0.9.0 —
per-edge cost 32% BETTER than v0.9.0. Extract's remaining 359 s is the
24 MB hugeexpr1.cs parse floor both versions pay.
Guarded by the complexity suite; cs_lsp/extraction/edge/lang-contract
suites green including the 53-language calls-breadth contract.
Refs #1669.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Cross-file LSP for Java rebuilt a type registry for EVERY file from a def
set the module filter reduces by a constant factor, not to a constant
size: the JVM filter branch includes every def sharing the file's
namespace, so per-file work tracks the corpus (measured defs_per_file
1,292 -> 5,031 in lockstep as defs_total went 179k -> 689k). That makes
the pass O(files x corpus_defs) — 87% of a Java index and the bulk of
the v0.9.0 -> v0.10.x slowdown.
Three changes, one architecture (the pattern Go/Python/C/C#/TS already
use):
1. cbm_java_build_cross_registry — the JVM def universe (Java + Kotlin,
for mixed source roots) built ONCE per run, sealed read-only, shared
across resolve workers. Wired into both pipeline.c and
pipeline_incremental.c.
2. Two-phase registration inside that build: all TYPES first, finalize
(hash buckets exist), then FUNCS. Func registration parses signatures,
and type-name qualification via cbm_registry_lookup_type is a LINEAR
scan until finalize — a single mixed pass measured 0.44 ms/def at
689k defs, a ~300 s sequential build that erased the sharing win.
Two-phase: 306 s -> 2.8 s. Stable partition order because overload
ties resolve to the first registered QN match.
3. cbm_run_java_lsp_cross_with_registry resolves each file against the
shared base through an overlay holding exactly THIS FILE's defs
(register_local_func_or_type_from_file). Own-file scope is the load-
bearing choice: patch_one_method refines signatures from the AST and
must write a private copy (its types live in the per-file arena, the
base is sealed), and any wider scope re-imports the quadratic — a
module/namespace-scoped overlay measured ratio 4.00 on the growing-
package corpus, own-file measures 2.00.
Elasticsearch corpus (46,477 files), same host, CBM_PROFILE=1:
v0.10.5 this change v0.9.0
wall 419.9 s 85.6 s (4.9x) 61.5 s
cross-LSP CPU 6,195,780ms 52,513ms (118x) 110,344 ms
us_per_file 207,448 1,770 3,722
us_per_file_per_kdef 300 2 —
nodes 693,100 693,100 —
edges 5,646,235 5,649,949 —
Cross-LSP CPU now beats v0.9.0. Nodes are identical; edges +0.066%,
consistent with the shared base resolving cross-package targets the old
per-file namespace/import filter could not see, plus source-order
independence from the two-phase build.
Guarded by the complexity suite's shared-package gate (RED at ratio 4.00
on the pre-change tree, 2.00 after — see the suite commit).
Refs #1669.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Avoid repeated full type-registry scans in the namespaced-type fallback. Build the optional index only for C/C++ cross registries while preserving candidate order, post-finalize tails, and allocation fallback.
Refs #1677
Signed-off-by: astandrik <astandrik@yandex-team.ru>
