Reproduce-first probe for the two fixes on this branch: RED without the
descend fix (count_defs_with_label(r, "Field") == 0, expected 3), GREEN
with it. Asserts the three named fields with their declared types in
return_type, the absence of the blank identifier, and that interface
members keep extracting exactly as before.
Signed-off-by: Ilya Brykau <ilya.brykau@orca.security>
Python attribute calls were not classified as member calls, so weak
short-name strategies (suffix_match / unique_name / field_type_hint)
could bind `obj.method()` to an arbitrary same-named project symbol —
`accelerator.print()` resolving to MockAccelerator.print.
Flag a Python attribute call is_method when its receiver is unresolved,
and extend the existing TS/JS weak-member guard to Python. Receivers that
ARE known stay exempt: self/cls, super(), and any identifier bound by one
of the file's imports (including the root of an attribute chain), so
Python's canonical `module.function()` cross-file call still resolves.
The guard helper is renamed cbm_tsjs_suppress_weak_method_match ->
cbm_suppress_weak_member_match. Its drop-list is unchanged, and the
per-language gate stays at the two call sites, which enumerate the same
language set in pass_calls.c and pass_parallel.c so the sequential and
parallel resolvers cannot diverge. ArkTS remains in that set.
python/S6/inherited_method in the resolution probe now asserts calls == 0,
recording the same trade TypeScript took when the receiver-aware guard
landed: the edge it used to produce came from a unique_name registry
fallback (measured on main: strategy=unique_name, cands=1, conf=0.7500)
where a weak short-name guess happened to be right in a 2-file fixture.
It flips back to >= 1 once py_lsp_cross resolves inheritance, matching the
flip-back note already carried on the TypeScript S6 case.
Fixes#1276
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: Enferlain <15861396+Enferlain@users.noreply.github.com>
Closes#518 (Section nodes don't index body text) and #519 (META.yaml /
frontmatter description values not indexed). A question asked in words
could not reach the nodes that hold the words: Section carried only its
heading, a config Module only its path, and both labels were filtered out
of BM25 results outright.
nodes_fts gains a fifth column, `body`, backfilled from each node's
EXISTING properties.docstring. No new column on `nodes`, no new table, and
CBM_INDEX_FORMAT_VERSION deliberately does NOT move -- nobody is forced to
reindex. A legacy four-column table survives untouched under CREATE VIRTUAL
TABLE IF NOT EXISTS: the backfill probes for the column before naming it,
and bm25()'s surplus fifth weight is inert on a table that has no fifth
column (FTS5 reads a weight only for a column an instance landed in). Such
a database keeps opening and searching, just without prose.
Two node kinds gain the docstring they lacked:
- Markdown section bodies (#518) -- the blocks following a heading, up to
the next heading or subsection, collapsed to single spaces and capped at
MAX_COMMENT_LEN (500 B) with a UTF-8 backoff that never splits a
multi-byte sequence.
- Top-level YAML/JSON description / summary / purpose (#519), promoted
onto the file's Module node.
Section and Module are no longer excluded from BM25 results. Both the
ranked query and the mirrored count query were changed together; removing
the filter from only one desynchronises results from counts. Column weights
are bm25(nodes_fts, 1.0, 1.0, 1.0, 1.0, 0.3): the four identifier columns
keep parity with each other and prose sits well below them, so a prose-only
hit surfaces but never outranks an identifier match.
The FTS backfill is consolidated into one writer, cbm_store_fts_rebuild(),
used by the wholesale rebuild AND the delta merge. That second site matters:
it had its own hand-written four-column INSERT, which with a fifth column
would leave body NULL for every node arriving by delta -- silently, on the
warm path most users hit, while a full reindex looked perfect. There is now
exactly one place the column list is decided, and a test that fails if the
delta path stops writing prose.
The backfill guards json_extract with json_valid: malformed properties JSON
exists in pre-fix databases and would otherwise abort the whole rebuild.
Tests: markdown section bodies (own body, stops at the next heading, UTF-8
safe cap, empty body stays NULL), YAML/JSON description promotion (block
scalars, synonym fallback, top-level only, config-only), FTS body indexing
and its column isolation, malformed-JSON tolerance, legacy four-column
degrade, incremental watermark, BM25 finding Section and Module by prose,
results/total consistency, identifier-outranks-prose, and the delta-merge
regression.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: ShauryaaSharma <196896785+ShauryaaSharma@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Chialisp commit was formatted with the pinned Homebrew clang-format, which
is right for the production sources but wrong for the three test files it also
touched: tests/ is not in LINT_SRCS, so those files have never been under the
format gate and were already format-dirty on main. Running the formatter over
them rewrote code that had nothing to do with Chialisp — 478 whitespace-only
lines in repro_invariant_breadth.c alone, plus 50 in test_extraction.c and 14 in
test_lang_contract.c — which is a review burden and hides the actual change.
Restore those three files to main's bytes and re-apply only the Chialisp
additions, in each file's own house style. The result is token-identical to the
version that was tested, apart from one em-dash written as `--` in a comment.
Re-verified after the rewrite: the six chialisp extraction tests, the cross-file
Constant graph test, the matrix-B row, the breadth case
(`[INV-BREADTH] PASS chialisp calls=1 callable=1 module=0`), and `make lint-ci`.
The production sources keep their formatting: they ARE in LINT_SRCS, and the
only whitespace they carry is the trailing-comment realignment in cbm.h that
adding a longer enumerator forces, plus the re-indent in extract_defs.c from
moving the name-resolution block into an else branch.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Chialisp (.clsp/.clib/.clinc) is the Chia smart-coin language. The only
public grammar, Quexington/tree-sitter-chialisp, cannot parse it: on the five
reference files from chia-blockchain@main it produced 3-34 ERROR nodes each.
Three defects account for all of them, and each desynchronised the rest of the
file rather than losing one node:
* comments were `/;.*\r\n/` -- CRLF-only, while real files are LF;
* `(include foo.clib)` was rejected because of the dot in the filename;
* `(defconstant NAME <expr>)` accepted only a primitive after the name.
So the grammar is written here instead, in tools/tree-sitter-chialisp/, and
files as first-party: its vendored LICENSE is the project's own, byte-identical
to the repository root, carrying no third-party copyright.
It is deliberately GENERIC -- source_file / list / symbol / string / number /
hex / dot / comment, modelling the clvm_tools reader rather than the Chialisp
form vocabulary. `mod`, `defun`, `defconstant` and `include` are ordinary head
symbols; which lists are definitions is decided in extract_defs.c. A dialect
that adds a form (cl-21 -> cl-23 -> cl-26 each added several) therefore needs
no regenerated parser. Acceptance: all five reference files parse with zero
ERROR and zero MISSING nodes; the three defects above are pinned as tests.
Extraction, and the four decisions in it:
* `Constant` is admitted to the cross-file name registry
(cbm_label_is_registry_symbol). Chialisp's condition codes live as
`(defconstant ...)` in a shared .clib and are referenced by name from every
puzzle; labelled but unregistered they are seeded nowhere, and roughly half
of a Chialisp graph would be present as nodes yet never the target of an
edge. All three registry-seeding sites route through that one predicate, so
they cannot diverge. No other language emits `Constant`, so the blast radius
is exactly Chialisp. Pinned by a two-file graph test that goes red when the
label is removed from the predicate.
* `export` / `namespace` are NOT definition heads. `(export foo)` names a
function already defined in the same file, so treating it as a def head
mints a second node for one symbol. They stay in the not-a-call filter.
* A parameter list is a `list` too, but it binds rather than invokes. Without
that gate `(defun check_conditions (HEIGHTLOCK conditions) ...)` mints a
phantom call to HEIGHTLOCK, and real Chialisp is mostly such definitions.
* `(include *standard-cl-26*)` is a dialect directive, not a file dependency;
recording it would invent a dependency on a file that does not exist.
Three helpers that the defs, calls and unified walks all need -- the quote-data
gate, comment-skipping named-child access, and the Chialisp def-head set -- live
in helpers.c rather than being copied per translation unit. They previously
existed as byte-identical duplicates that had already drifted: extract_defs.c
skipped comments when reading a def head while its mirror in extract_unified.c
did not, so a comment between a head and its name desynchronised defs from call
scope. Both now use the same accessor.
Ledger: 161 -> 162 grammars across the seven published surfaces, ABI-14 86 -> 87,
first-party 13 -> 14, CBM_LANG_COUNT 164 -> 165.
Distilled from PR #919 by darkverbito, whose extraction logic -- the CLVM
not-a-call vocabulary, the quoted-data gate, the filename-derived `mod` name and
the include/embed import handling -- is kept and is the substance of this change.
The grammar that PR vendored is not used.
Co-authored-by: darkverbito <8049985+darkverbito@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distilled from #1412 by @allanoepping. Fixes a real recall hole: tag-dialect
.cfc components were routed to the cfscript grammar (extracting nothing),
and <cfscript> bodies inside tag-dialect files are opaque cf_script_content
to the cfml grammar — whole legacy components produced zero functions.
Fix 1 — his .cfc dialect sniffer, nearly verbatim: cbm_disambiguate_cfc()
reads a 16K head via cbm_fopen; <cfcomponent>/<cffunction> anywhere means
tag dialect; a leading <cfscript (after whitespace and <!--- ---> comments)
or no markup at all keeps the old CFSCRIPT routing, so script-dialect files
and read failures behave exactly as before. Wired in detect_file_language
beside the existing _m/_cls/_inc sniffers, with his 8 direct tests.
Fix 2 — his embedded-defs capability, re-based onto the shared machinery
that superseded his mechanism mid-flight (#1852, merged after his last
rebase): a cfml_embedded_imports spec row (cf_script_tag/cf_script_content
-> CFSCRIPT), CFML added to the embedded-imports dispatch, and the
extract_structure gate extended to CFML. Included ranges on the original
buffer give absolute coordinates for free, so his line-number assertions
hold verbatim, and calls inside blocks become extractable as a bonus. His
~90-line slice+shift implementation is deliberately NOT carried — it
mirrored the sanctioned shape of its day, which #1852 replaced.
One correctness guard the port needed: the Vue attribute resolver is now
gated to Vue — cf_script_tag carries no lang=/src= attributes, and running
the resolver would have silently rewritten CFML's embedded language from
CFSCRIPT to the resolver's JavaScript default.
His embedded extraction test (line remap + <cfsetting> ERROR-cascade case)
carries over verbatim and binds: reverting the spec/gate wiring fails it at
ASSERT(has_def(r, "Function", "greet")). Suites: extraction, language,
edge_imports, lang_contract, grammar_labels, grammar_regression — 637/0.
Co-authored-by: Allan Oepping <allan@intellipay.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Assert positively that the fixture's tree contains ERROR/MISSING nodes
(parse_incomplete), so a future grammar upgrade that fixes AS OBJECT
turns this RED and prompts claiming the Class def instead of pinning its
absence vacuously.
Co-authored-by: Oğuz <oguzhan.saricam@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Vendor AndreasMaierDe/tree-sitter-plsql @ 28aebef209be (MIT, ABI 14, no
external scanner) and wire CBM_LANG_PLSQL end to end:
- extraction onto existing labels only: package spec/body -> Class,
procedure/function (create/definition/declaration) -> Function,
file -> Module; callee extraction for ref_call incl. the
package-qualified pkg.name form; identifier usage vocabulary with the
upstream 'parameter' kind (a ref_call ARGUMENT wrapper) carved out of
the whole-binding rule
- discover: .pks/.pkb/.pck/.pls/.plb/.plsql/.fnc/.trg/.bdy/.tps/.tpb
(.sql stays SQL and .prc stays FORM, both asserted negatively)
- ledgers: CALL_WITHOUT_REFERENCE_VOCAB(PLSQL), call-node manifest
ref_call row (historical total 219 -> 220), call-argument matrix B row
- tests: extraction triple (package body + package-qualified call,
standalone function, honest negative pin of the upstream
CREATE TYPE ... AS OBJECT ERROR-node limitation), grammar-regression
case, label golden Class:1,Function:1,Module:1, MUST_EXTRACT_DEFS,
extension mapping, bug-repro board battery row
- MANIFEST.md (community-niche verdict, local include-quote patch row),
THIRD_PARTY.md section, refreshed vendored checksums
PL/SQL support was contributed by Oguz (@ouzsrcm) in #1033; re-vendored
from upstream per vendoring policy with his language wiring distilled on
top.
Co-authored-by: Oğuz <oguzhan.saricam@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Vue single-file components yielded a Module node and IMPORTS edges but no
Function definitions or CALLS edges from the `<script>` block, so a component's
call graph was empty (#1410).
Adds a control test pinning current behaviour for Svelte, HTML and Astro, which
still extract imports but no functions or calls. That is deliberate scoping,
not an oversight: it documents the remaining gap rather than hiding it, and it
proves this fix does not leak into sibling languages. Svelte is tracked by
#1807 and will be a follow-up.
Re-opened from #1843, which carried no Signed-off-by and so failed DCO. The
tree is byte-identical to that commit; only the sign-off was added.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The fixture returned "/etc/myapp/conf.d", which is_filesystem_path() rejects
before is_rest_path() is consulted, so the value was never classified as a URL
and never recorded as a builder -- with or without the language gate. The test
passed identically with the gate removed, so it certified nothing.
"/srv/" is not one of the ten filesystem prefixes, so the value is URL-shaped
and only the JS/TS/TSX gate prevents it being recorded for C.
Verified by revert-check: with the gate removed the test now FAILS at
tests/test_extraction.c:3455; with the gate restored it passes.
Follow-up to #1010. The gate implementation is unchanged and is the author's.
Co-authored-by: Charles Queiroz <fcqueiroz@liquibase.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Routing .ets through the plain TypeScript grammar produced no definition
node for any @Component struct (0/19 across the OpenHarmony sample corpus),
parsed struct members as statement labels and build() as a call expression,
and registered ArkUI builtins (Column, Row, ListItem, ...) as user function
definitions with 28% fabricated cross-file CALLS edges. This adds a
dedicated ArkTS grammar that keeps everything TypeScript already handles
and models the ArkTS syntax TS cannot parse.
Grammar (tools/tree-sitter-arkts/, generated parser vendored to
internal/cbm/vendored/grammars/arkts/, ABI 15 via tree-sitter-cli 0.25.10):
first-party derivative of tree-sitter/tree-sitter-typescript v0.23.2
@ 75b3874edb2d (the commit our vendored typescript/tsx come from) on the
tree-sitter-javascript v0.23.1 base @ 3a837b6f3658 (the version upstream's
package-lock pins). LICENSE carries the upstream MIT text verbatim plus a
derivative notice; the external scanner is a symbol-rename trampoline over
a byte-identical copy of the already-vendored typescript scanner header.
ArkTS additions: decorated struct component declarations with class-shaped
bodies; UI-DSL trailing-closure component calls with attribute chains after
the closing brace; decorators on function declarations (@Builder,
@Extend(T), @Styles); @Extend/@Styles leading-dot attribute-chain bodies;
anonymous stateStyles style blocks; import lazy named imports.
Wiring: CBM_LANG_ARKTS enum + lang spec (struct_declaration as a class
kind labeled Struct, public_field_definition fields), .ets extension,
LANG_NAMES/userconfig entries, grammar shim, and TS-family joins across
the extractors and the name-resolution guards (parse_es_imports, JS
keywords/test-file/func-kind/scope tables, TS heritage/param/receiver/
namespace/type-ref/channel/usage paths, js_ts_family and the weak-method-
match suppression). Field definitions now also carry their decorators, so
@State/@Prop/@Link members are findable with their roles.
Tests: grammar corpus (35 cases), per-language regression + label golden +
import net + extension cases, seven ArkTS extraction tests including the
phantom-builtin regression (ArkUI builtins must never be definitions), and
capability-ledger / call-node-manifest / call-argument-matrix rows. On the
14-file OpenHarmony sample corpus the TS-grammar baseline had 12/14 files
with parse errors (231 ERROR nodes); the arkts grammar parses 14/14 clean,
and an end-to-end index yields all 19 components as Struct nodes, 68
methods, 119 fields, and zero phantom builtin definitions.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
handle_url_builders ran for every vendored grammar, so a C helper
returning an absolute pathname became a URL builder and minted a Route
node plus an HTTP_CALLS edge in a language that speaks no HTTP. Gate
recording to JavaScript, TypeScript and TSX.
Builder entries now carry a flag in the per-file constant map: only a
call_expression argument resolves one, so handing a builder to a
callback builds no request. A builder whose returns are not all
route-shaped literals is declined instead of attributing its one literal
to call sites that take the computed branch.
Five negative tests cover the cross-language case, the mixed builder,
the ambiguity tombstone, the builder reference and the non-URL helper.
Signed-off-by: Charles Queiroz <fcqueiroz@liquibase.com>
URL-shaped literals returned from small builder functions never reached
call arguments, so client(buildPath(id)) produced no HTTP_CALLS edge and
no Route node, static or template alike.
handle_url_builders() records builderName -> returned URL in the same
per-file constant map used for module-level consts, covering return
statements and arrow expression bodies; a call_expression branch in
extract_url_or_topic_arg() resolves client(buildPath(id)) through it.
Composed builders inline already-recorded substitutions and truncate the
query string, so `${basePath(id)}?${params}` joins the server route
exactly. Ambiguous builders (two different URLs) are tombstoned.
Signed-off-by: Charles Queiroz <fcqueiroz@liquibase.com>
Java enum methods already resolve on main: find_class_member_body
descends into enum_body_declarations (Java-gated) while find_class_body
keeps returning enum_body, which is what extract_enum_members needs to
reach the constants — they are siblings of enum_body_declarations, not
children.
That distinction was unguarded. java_enum_dedup_preserves_calls_issue1234
asserted the methods and the absence of duplicate Function defs, but
never asserted the constants, so collapsing the two lookups would have
passed the suite while silently dropping every enum constant from the
graph.
PR #984 proposed exactly that collapse — redirecting the shared
find_class_body — which is what surfaced the gap. Verified binding:
pointing extract_enum_members at find_class_member_body reddens the new
Variable assertions and nothing else.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: sahil-mangla <manglasahil2017@gmail.com>
A Swift protocol requirement — a bodyless `func generate() -> String`
inside a protocol — was absent from the graph entirely. Swift codebases
are heavily protocol-driven, so the requirement is very often the
declaration a reader is looking for, and "who declares generate()" had no
answer.
Three edits deliver it: protocol_body joins the class-body types so a
protocol's members are walked at all; protocol_function_declaration joins
swift_func_types, because extract_class_methods gates on that set and
would otherwise walk the requirement and discard it; and both name
resolvers accept the node, which has the same simple_identifier shape as
function_declaration.
Distilled from #613 by xbsjason. That PR also carried an enum/struct half
whose two assertions fail on CI: tree-sitter-swift has no
struct_declaration or enum_declaration node type (it models both as
class_declaration), so the corresponding swift_class_types entries are
inert and a bare enum is still labeled Class. That is a real pre-existing
modelling gap, left untouched here and now documented at the dead entries
rather than silently deleted.
Also dropped from the original: a find_first_descendant_by_kind rescue in
extract_class_methods. Removing it changes no test outcome once the
function-type entry is present, so it was carrying nothing.
Advances #43.
Co-authored-by: xbsjason <xbsjason@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A dbt model is an ordinary .sql file whose dependencies are written
{{ ref('other_model') }} or {{ source('group','table') }}, never as
literal table names. The SQL grammar cannot read those -- FROM {{ ref(
'x') }} is a parse error to it -- so the dependency structure of an
entire dbt project was invisible to the graph.
extract_dbt.c runs as a sub-extractor inside the normal indexing
pipeline. Per qualifying file it emits one Model definition (named by
the file stem, dbt's own model identity) plus one usage per ref()/
source() call; pass_usages resolves those into model -> relation
lineage edges like any other reference.
Model is a relation label alongside Table and View, which is what makes
the rest work without new plumbing: registry seeding, the central
relation veto, the incremental surface hash, search ranking and the
architecture queries all pick it up from cbm_label_is_relation. Sharing
one label class also means a model's source('raw','customers') resolves
onto a Table declared in a plain DDL migration in the same repository,
so dbt lineage and SQL lineage form one graph rather than two, while
the veto keeps model names out of every non-lineage consumer.
The pass is self-gating: SQL files only, and only those carrying a real
ref()/source() call. The dbt builtins are themselves the evidence, which
is cheaper than a dbt_project.yml lookup and more precise -- templated
SQL that is not dbt (an Airflow {{ ds }} parameter) produces no Model
node and no usages even inside a dbt repository.
{% macro %} definitions are deliberately excluded: the vendored
tree-sitter-jinja2 grammar has no node types for {% %} statements, so
they can only be recovered by a hand-written scanner that cannot see
comments or find {% endmacro %} for a correct span. Filed as follow-up
rather than approximated.
Tests cover the lineage, the last-string-argument semantics of both
builtins, the gate (against a plain-SQL control extraction), plain DDL
staying untouched, and an end-to-end pipeline case asserting model ->
model across files, model -> Table onto plain DDL, and cross-language
isolation. Disabling the pass reddens three of them; removing Model from
the relation set breaks lineage outright.
Implements the lineage half of #575.
Co-authored-by: alexisperinger-ux <alexis.peringer@iss-stoxx.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
CREATE TABLE / CREATE VIEW / CREATE MATERIALIZED VIEW now produce Table
and View nodes (previously generic Variable), CREATE PROCEDURE produces
a Function, and schema-qualified DDL names (schema.table) are named by
the table identifier instead of the schema. A view's FROM/JOIN relations
are emitted as usages and resolve into view -> table USAGE lineage
edges.
Relations join the cross-file name registry so lineage can resolve, with
two structural safeguards:
- Registry membership is defined once by cbm_label_is_registry_symbol
(helpers.c); the full, parallel and incremental seed sites all call
it, ending the KEEP-IN-SYNC copies the old label lists required.
- The default cbm_registry_resolve vetoes relation-labeled results:
common table names (users, orders, config) collide with code
identifiers in every language, and no CALLS/USAGE/READS/WRITES/THROWS/
handler/decorator consumer may bind them. The SQL lineage path opts in
through the new cbm_registry_resolve_lineage.
Table/View also join the registry-only per-file LSP surface labels so a
table rename invalidates dependent SQL files on incremental (no stale
lineage edges), rank with the type tier in BM25 search, and appear in
the architecture boundary/package/cluster queries via the pinned
CBM_SQL_RELATION_LABELS fragment.
Tests: extraction trio (labels, lineage usages, schema-qualified names),
grammar golden + probe updates, relation-label contract pin, and two
pipeline tests: cross-language isolation (binding: fails without the
veto) and incremental table-rename stale-lineage.
Closes#574.
Co-authored-by: alexisperinger-ux <alexis.peringer@iss-stoxx.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Regression coverage for #1234: Java interface and enum methods were
emitted as both a Method node and a duplicate top-level Function node.
The production fix landed via 87a0e3f (language-agnostic class-body
routing); this adds the missing test coverage so it cannot silently
regress:
- java_interface_no_duplicate_function_issue1234: interface methods
produce Method nodes only, zero Function nodes
- java_enum_dedup_preserves_calls_issue1234: enum methods dedup'd while
cross-class CALLS edges survive
- cp_interface_method_no_dup_function: convergence probe asserting the
surviving Method still carries its CALLS edge end-to-end
Distilled from PR #1327.
Co-Authored-By: Harshita Joshi <j.harshitaa06@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
r.HandleFunc(base+"/login", handler) fell through extract_positional_url
with no branch for binary_expression, so first_string_arg stayed NULL. A
real BFF with 47 such registrations (all base+"literal") produced only 9
Route nodes. Recover the literal suffix from the right-hand operand of a
"+" concatenation; a right side that is not itself a literal is left
unresolved rather than guessed.
Fixes#1249
Signed-off-by: Amir Fathi <amirfathi.me@gmail.com>
Function/Method nodes only got is_test=true via a Rust #[test] attribute
check; the per-file test-file flag (cbm_is_test_file) was set on the Module
node but never propagated to the definitions inside it, so e.g. a C function
in tests/helpers/fixtures.c stayed is_test=false — invisible to store.c's
is_test!=1 filters — even though trace_path's own independent path check
already treated the file as a test. Propagate is_test_file into
extract_func_def and push_method_def, and teach cbm_is_test_file the same
tests/ (and test/, spec/, __tests__/) directory convention cbm_is_test_path
already uses, so a non-test_-named file anywhere under tests/ is caught too.
Separately, trace_call_path's own include_tests filter (mcp.c is_test_file)
matched a nested ".../tests/..." path but not a project-root-relative one,
so tests/repro/foo.c leaked into results with the default
include_tests=false; add the missing tests/ (and test/, spec/, __tests__/)
prefix check.
Closes#1294.
Signed-off-by: Yyunozor <yyunozor@icloud.com>
nix_module_level_bindings_mint_variables
a `let` binding and an attrset binding both mint Variables, and the QN
carries the attrpath (services.nginx.enable).
nix_nested_bindings_are_not_module_level
a binding two attrsets deep mints nothing, and neither `deep` nor `nested`
becomes a Variable since both are scopes.
nix_lambda_binding_is_function_not_variable
a lambda-valued binding is a Function and NOT a Variable; a scalar binding
is the reverse.
Every absence assertion is paired with a positive one on the same predicate in
the same result — `topLevel` in the nesting test, `val` in the split test — so
none of them can pass by extracting nothing at all. Several false "clean" results
in this area came from predicates that could not have matched.
Signed-off-by: Jason Bowman <jason@json64.dev>
Four extraction cases and one pipeline case, each asserting a distinct claim so a
regression names what it broke:
nix_attrset_scope_disambiguates_leaf_names
two attrsets each holding `dup` produce two definitions with distinct QNs.
Unqualified they shared one QN and the second was discarded.
nix_dotted_attrpath_qualifies_like_nested
`wrap.deep.fn = …` and `wrap = { deep = { fn = …; }; }` produce the SAME
QN. This is the equality that makes attrpath qualification a correctness
fix rather than a preference — the two are the same expression.
nix_quoted_attr_name_strips_quotes
`"kebab-case"` is named kebab-case, and asserts the quoted form is absent
rather than merely that the bare form is present.
nix_interpolated_attr_mints_no_def
`"${dynamic}" = …` mints nothing, and the sibling binding still does — so
the test cannot pass by extracting nothing at all.
pipeline_nix_scoped_binding_calls_resolve (tests/test_pipeline.c)
a call inside a scoped binding reaches its target through the store, for
both routes into a qualified name: an enclosing attrset and a dotted
attrpath.
The pipeline case has to live at that level. Definition QNs and call-scope QNs
come from two separate functions, and if they disagree the edge is dropped at
write with no error — every extraction-level assertion still passes while the
graph quietly loses edges. Only the store shows it.
Adds has_def_qn alongside the existing has_def. find_def_by_name returns the
first match by NAME and so cannot tell two same-named definitions in different
scopes apart, which is exactly what these tests exist to separate.
Extraction and pipeline suites: 491 passed, 0 failed.
Signed-off-by: Jason Bowman <jason@json64.dev>
All three cost real release cycles, and none is fixed by widening a budget.
daemon_application_cancels_physical_job_only_after_final_session waited for the
SUBSCRIBER COUNT to reach 2, then cancelled both sessions and asserted the
physical job had started exactly once. The job starts asynchronously after
subscription, so both cancels could land first and leave starts == 0 --
arguably the correct outcome. It now waits for the state the assertions
actually require. Verified 47/47 on Windows, the only platform it ever failed.
The parallel harness refused outright when the suite leader had already exited,
because taskkill /T cannot walk a tree from a dead PID. But the leader can exit
between the timeout decision and that call, so the harness itself lost a race:
a natural exit at the wrong moment failed the whole wave. It now proves cleanup
the only way still available -- nothing parented to that PID -- and its contract
asserts the PROPERTY rather than the phrase "tree cleanup" it used to grep for.
That string pin is what broke when the guard was reworded while behaving
correctly; the contract now checks rc==2 AND that the descendant really did
survive, which would also catch a guard that claims to fail closed while
leaking.
extract_wide_flat_file_is_linear took ONE sample per size, so the ratio carried
the noise of both. On a loaded Windows VM linear code measured 51x against a 40x
bound (184ms -> 9387ms). Best-of-N instead: timing noise only ever adds time, so
the minimum is the cheapest good estimate of the noise-free cost. The bound is
deliberately unchanged -- it sits where linear (~20x) and quadratic (~128x) are
each >=2x away, so raising it would move the test toward the very signal it
exists to catch. Now measures 19.1x on Windows, 21.4x on macOS.
Also: the smoke's `cli` helper redirected stderr to a file and discarded it, so
any of the 10 bare `VAR=$(cli ...)` assignments could kill the run under
`set -euo pipefail` printing NOTHING. One such abort cost a full Windows cycle
just to locate and still could not be attributed. It now surfaces the command
and its stderr. Neutral wording on purpose: one call site expects a non-zero
exit and must not read as a failure.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
`nix_function` was the only Nix extraction test, and it could not fail for the
bug the previous commit fixes. Three separate reasons:
- its fixture is function-headed — the exact broken shape;
- its binding, `hello = pkgs.writeShellScriptBin "hello" ''...''`, is an
application, not a function, so no Function def is expected from it either
way;
- it asserts only ASSERT_NOT_NULL and ASSERT_FALSE(has_error), omitting the
has_def check every adjacent language test makes.
So the suite reported green while definitions were being dropped for the
dominant file shape in the ecosystem.
This keeps `nix_function` as-is — it usefully pins the "parses, yields no def"
case — and adds one test per root shape, each asserting its own definitions so a
regression names the shape it broke:
nix_defs_in_let_rooted_file let ... in
nix_defs_in_attrset_rooted_file { ... }
nix_defs_in_nested_let let inside let
nix_defs_survive_function_header_let { prelude }: let ... in
nix_defs_survive_function_header_attrset { prelude }: { ... }
nix_defs_survive_curried_header final: prev: { ... }
nix_curried_lambda_mints_one_def guards the other direction
The curried-header case is the nixpkgs overlay signature, the most common
multi-arm header in the ecosystem: two nested function_expressions sit between
the file root and the body. The last test exists because the fix descends past
the header — it pins that `iota = a: b: a + b` mints exactly one Function, so a
change that starts minting the inner arm as a second def is caught.
Verified as a negative control — with the previous commit's one-line change
reverted, the three function-header tests fail on their own assertions (gamma,
eta, iota) while the pre-existing shapes stay green. With it applied the
extraction suite is 260 passed, 0 failed.
Signed-off-by: Jason Bowman <jason@json64.dev>
Client-side URL extraction only recognized static string literals; any
template literal was silently skipped, so parameterized endpoints never
produced HTTP_CALLS edges or Route nodes and cross-repo route matching
missed them (the server side already normalizes path params to {}).
New cbm_template_string_text() flattens a template_string node: string
fragments verbatim, each ${...} substitution becomes {}. Wired into:
- extract_positional_url / extract_string_value (call-arg URLs)
- handle_string_refs (URL-shaped refs from const/return positions)
- handle_string_constants (module-level const lookups)
`/api/v1/things/${id}` now yields __route__ANY__/api/v1/things/{} and
the enclosing function gets the HTTP_CALLS edge, joining the canonical
placeholder shape of server-side routes.
Signed-off-by: Charles Queiroz <fcqueiroz@liquibase.com>
A function-like macro invocation whose argument is a type token — e.g.
ALLOC(int, n) — makes tree-sitter's C/C++ grammar emit an ERROR node (it parses
the type in expression position), which cbm_collect_error_regions recorded as a
parse_partial coverage gap. But the macro is #defined in the same file and the
call sits inside an already-extracted function body, so nothing is actually
missing from the graph — it's a benign call the grammar can't parse without the
preprocessor (#1071, systematic across allocation-macro-style codebases).
Subtract an error region only when it both (a) contains an invocation of a
file-defined function-like macro and (b) is fully enclosed by an extracted
Function/Method body. Condition (b) keeps a TOP-LEVEL macro invocation that
expands to a definition still flagged (#949: the generated def isn't in the
original span), and fails safe — a benign top-level call stays flagged rather
than a real gap being hidden.
Tests: a type-arg macro call inside a function no longer reports parse_partial;
a real in-body syntax error still does; #949/#946 preserved.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Comments are NAMED tree-sitter nodes, so the prev-sibling walk in
extract_decorators() stopped at one — silently dropping every decorator ABOVE
an interleaved comment:
@Post('login') <-- dropped
@HttpCode(HttpStatus.OK) <-- dropped
// throttled per IP and account
@Throttle({ ... }) <-- kept
async login(...)
The route then vanished from decorator/route queries, so documenting a
decorator made the endpoint disappear from the graph. Real-world impact: on a
NestJS backend, 1 of 95 HTTP endpoints was missing — the one whose throttle
policy carried an explanatory comment.
Comments are now transparent to the walk, the same way anonymous tokens
(e.g. TS `export`) already were. Reuses the existing is_comment_node() helper.
Signed-off-by: KolisCode <jhohantma@gmail.com>
Ilya Brykau