A 4.7 MB binary Godot resource (piko_walk_mesh.res) never finished
indexing: `.res` maps to ReScript, so the resource was parsed as ReScript
source and the definitions pass spun. Three defects, each fixed:
1. Quadratic usage walk (the hang). The ReScript `let` binding check
(is_first_named_part_of, shared with Julia, Typst, Elm, PureScript
and Nickel) climbed from every named leaf with ts_node_parent, which
descends from the root and scans each level's children. A binary
file parses into a flat error tree (~170k root children per MB), so
each climb cost O(root children) and the file O(n^2): 1 MB spent
46 s in cbm_extract_unified, 4.7 MB never returned. The check now
climbs the unified walk's own cursor (O(1) per hop), like the other
occurrence classifiers; the node-based climb stays only for callers
outside the walk. 1 MB: 46 s -> 0.03 s; 4.7 MB: 2.2 s end to end.
2. Quadratic lexing in error recovery. tree-sitter retries the lexer at
every byte of an unparseable stretch with every external token
valid; the ReScript scanner's template-string loop then ran to the
next backtick/$/backslash/NUL (end of file if none) and discarded
the result. That cost O(n^2) inside lexing, where the parse budget's
progress callback never runs, so such files were dropped by the
5 s CPU budget instead of parsed -- plain text too (100 KB of
non-ReScript text: 15 s -> 0.18 s). Local scanner patch: skip the
template branch when NEWLINE is also valid, which only the ERROR
state allows. Valid files parse to identical trees. Recorded in
vendored/grammars/MANIFEST.md.
3. Binary content parsed as source. `.res` is also Godot's binary
resource format and the Windows compiled-resource format. Following
the .m/.cls/.frm/.inc/.cfc precedent, cbm_disambiguate_res reads the
first 4 KB and reports a file holding a NUL byte as unsupported;
ReScript text never contains one.
Tests (deterministic work counters, no clocks):
- extraction::extract_rescript_let_bindings_use_the_walk_cursor --
zero root-descending parent hops at 128 and 1024 statements
(before: 515 / 4099).
- complexity::complexity_rescript_error_recovery_lexing_is_linear --
lexer bytes pulled through a chunked TSInput, 4 KB vs 8 KB, ratio
must be linear (before 3.99, after 2.00).
- language::lang_res_binary_resource_unsupported and
lang_res_rescript_stays_rescript.
Fixes#2176
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The XML documents that make up a .NET repository project system were
absent from EXT_TABLE, so .csproj, .props, .targets, .resx, .plist,
.xaml and the app manifests were never indexed and their content was
unreachable through search_code. tree_sitter_xml is already wired up,
so these only needed rows in the table.
Fixes#2229
Signed-off-by: yasmoradi <msynk.dev@gmail.com>
Visual Basic 6 shares .frm with FORM and .cls with Apex/ObjectScript, so VB6
forms were parsed by the FORM grammar (0 defs) and class modules by the Apex
grammar (junk nodes, extract timeouts). Recognise the VB6 export header
(VERSION x.yy / VERSION 1.0 CLASS / Begin VB. / Attribute VB_Name) and report
such files as unsupported (CBM_LANG_COUNT) until a Visual Basic grammar exists:
new cbm_disambiguate_frm(), VB6 check added to cbm_disambiguate_cls() ahead of
its ObjectScript scan. Defaults on any doubt are unchanged (FORM / Apex).
Short-term step agreed in #721; the grammar follows separately.
Refs #721
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Tokarzewski <bartlomiej.tokarzewski@gmail.com>
Follow-up to #1824, which mapped Blazor `.razor` to the C# host and extracted
`@page` routes from it. `.cshtml` is the same Razor syntax with the same C#
host and was equally unmapped: an ASP.NET Core MVC or Razor Pages application
produced no nodes at all for its entire view tree.
`@page` matters more here than it does for components. On a Blazor component
the directive is an optional annotation; in Razor Pages it is the directive
that *defines* a page, so the routable surface of a Razor Pages app lives
exclusively in this file type.
That is why the extension mapping alone is not enough. `cbm_path_is_razor()`
hardcoded a single `.razor` suffix, so a `.cshtml` file would have been
discovered and parsed but skipped by the directive scan -- present in the
graph, with none of its routes. Widened to a suffix table covering both.
Scope is deliberately the Razor family and nothing else:
- `.vbhtml` is excluded because there is no VB.NET language in this repo,
so there is no host to map it to.
- `.aspx` / `.ascx` are excluded because Web Forms is a different templating
syntax (`<%@ %>`, `runat="server"`) with no `@page` directive, so neither
the C# error recovery nor the directive scan applies.
Best-effort on the same terms as `.razor`: the C# grammar recovers the `@{ }`
and `@functions` blocks, and the surrounding markup lands in ERROR regions
reported through parse_partial.
Tests: the extension mapping; a Razor Page asserting the extracted route path
and method; and a negative case for a layout with no `@page`, since most
`.cshtml` files are layouts and partials and a spurious Route node per view
would be worse than the missing ones this fixes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A .razor file had no entry in EXT_TABLE, so discovery skipped it entirely
and a Blazor application produced no graph nodes for any of its
components. The only way in was an undocumented extra_extensions entry in
a per-project .codebase-memory.json.
Map .razor to CBM_LANG_CSHARP. This is best-effort by design: the C#
grammar recovers the @code block, while the surrounding markup lands in
ERROR regions and is reported through parse_partial. That is strictly
more than the Module-and-imports the other markup-hosted languages
(Vue, Svelte, Astro) extract today, and it needs no new grammar.
Covered by tests/test_language.c:lang_ext_razor.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: The1nk <23087600+The1nk@users.noreply.github.com>
Chialisp (.clsp/.clib/.clinc) is the Chia smart-coin language. The only
public grammar, Quexington/tree-sitter-chialisp, cannot parse it: on the five
reference files from chia-blockchain@main it produced 3-34 ERROR nodes each.
Three defects account for all of them, and each desynchronised the rest of the
file rather than losing one node:
* comments were `/;.*\r\n/` -- CRLF-only, while real files are LF;
* `(include foo.clib)` was rejected because of the dot in the filename;
* `(defconstant NAME <expr>)` accepted only a primitive after the name.
So the grammar is written here instead, in tools/tree-sitter-chialisp/, and
files as first-party: its vendored LICENSE is the project's own, byte-identical
to the repository root, carrying no third-party copyright.
It is deliberately GENERIC -- source_file / list / symbol / string / number /
hex / dot / comment, modelling the clvm_tools reader rather than the Chialisp
form vocabulary. `mod`, `defun`, `defconstant` and `include` are ordinary head
symbols; which lists are definitions is decided in extract_defs.c. A dialect
that adds a form (cl-21 -> cl-23 -> cl-26 each added several) therefore needs
no regenerated parser. Acceptance: all five reference files parse with zero
ERROR and zero MISSING nodes; the three defects above are pinned as tests.
Extraction, and the four decisions in it:
* `Constant` is admitted to the cross-file name registry
(cbm_label_is_registry_symbol). Chialisp's condition codes live as
`(defconstant ...)` in a shared .clib and are referenced by name from every
puzzle; labelled but unregistered they are seeded nowhere, and roughly half
of a Chialisp graph would be present as nodes yet never the target of an
edge. All three registry-seeding sites route through that one predicate, so
they cannot diverge. No other language emits `Constant`, so the blast radius
is exactly Chialisp. Pinned by a two-file graph test that goes red when the
label is removed from the predicate.
* `export` / `namespace` are NOT definition heads. `(export foo)` names a
function already defined in the same file, so treating it as a def head
mints a second node for one symbol. They stay in the not-a-call filter.
* A parameter list is a `list` too, but it binds rather than invokes. Without
that gate `(defun check_conditions (HEIGHTLOCK conditions) ...)` mints a
phantom call to HEIGHTLOCK, and real Chialisp is mostly such definitions.
* `(include *standard-cl-26*)` is a dialect directive, not a file dependency;
recording it would invent a dependency on a file that does not exist.
Three helpers that the defs, calls and unified walks all need -- the quote-data
gate, comment-skipping named-child access, and the Chialisp def-head set -- live
in helpers.c rather than being copied per translation unit. They previously
existed as byte-identical duplicates that had already drifted: extract_defs.c
skipped comments when reading a def head while its mirror in extract_unified.c
did not, so a comment between a head and its name desynchronised defs from call
scope. Both now use the same accessor.
Ledger: 161 -> 162 grammars across the seven published surfaces, ABI-14 86 -> 87,
first-party 13 -> 14, CBM_LANG_COUNT 164 -> 165.
Distilled from PR #919 by darkverbito, whose extraction logic -- the CLVM
not-a-call vocabulary, the quoted-data gate, the filename-derived `mod` name and
the include/embed import handling -- is kept and is the substance of this change.
The grammar that PR vendored is not used.
Co-authored-by: darkverbito <8049985+darkverbito@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distilled from #1412 by @allanoepping. Fixes a real recall hole: tag-dialect
.cfc components were routed to the cfscript grammar (extracting nothing),
and <cfscript> bodies inside tag-dialect files are opaque cf_script_content
to the cfml grammar — whole legacy components produced zero functions.
Fix 1 — his .cfc dialect sniffer, nearly verbatim: cbm_disambiguate_cfc()
reads a 16K head via cbm_fopen; <cfcomponent>/<cffunction> anywhere means
tag dialect; a leading <cfscript (after whitespace and <!--- ---> comments)
or no markup at all keeps the old CFSCRIPT routing, so script-dialect files
and read failures behave exactly as before. Wired in detect_file_language
beside the existing _m/_cls/_inc sniffers, with his 8 direct tests.
Fix 2 — his embedded-defs capability, re-based onto the shared machinery
that superseded his mechanism mid-flight (#1852, merged after his last
rebase): a cfml_embedded_imports spec row (cf_script_tag/cf_script_content
-> CFSCRIPT), CFML added to the embedded-imports dispatch, and the
extract_structure gate extended to CFML. Included ranges on the original
buffer give absolute coordinates for free, so his line-number assertions
hold verbatim, and calls inside blocks become extractable as a bonus. His
~90-line slice+shift implementation is deliberately NOT carried — it
mirrored the sanctioned shape of its day, which #1852 replaced.
One correctness guard the port needed: the Vue attribute resolver is now
gated to Vue — cf_script_tag carries no lang=/src= attributes, and running
the resolver would have silently rewritten CFML's embedded language from
CFSCRIPT to the resolver's JavaScript default.
His embedded extraction test (line remap + <cfsetting> ERROR-cascade case)
carries over verbatim and binds: reverting the spec/gate wiring fails it at
ASSERT(has_def(r, "Function", "greet")). Suites: extraction, language,
edge_imports, lang_contract, grammar_labels, grammar_regression — 637/0.
Co-authored-by: Allan Oepping <allan@intellipay.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Vendor AndreasMaierDe/tree-sitter-plsql @ 28aebef209be (MIT, ABI 14, no
external scanner) and wire CBM_LANG_PLSQL end to end:
- extraction onto existing labels only: package spec/body -> Class,
procedure/function (create/definition/declaration) -> Function,
file -> Module; callee extraction for ref_call incl. the
package-qualified pkg.name form; identifier usage vocabulary with the
upstream 'parameter' kind (a ref_call ARGUMENT wrapper) carved out of
the whole-binding rule
- discover: .pks/.pkb/.pck/.pls/.plb/.plsql/.fnc/.trg/.bdy/.tps/.tpb
(.sql stays SQL and .prc stays FORM, both asserted negatively)
- ledgers: CALL_WITHOUT_REFERENCE_VOCAB(PLSQL), call-node manifest
ref_call row (historical total 219 -> 220), call-argument matrix B row
- tests: extraction triple (package body + package-qualified call,
standalone function, honest negative pin of the upstream
CREATE TYPE ... AS OBJECT ERROR-node limitation), grammar-regression
case, label golden Class:1,Function:1,Module:1, MUST_EXTRACT_DEFS,
extension mapping, bug-repro board battery row
- MANIFEST.md (community-niche verdict, local include-quote patch row),
THIRD_PARTY.md section, refreshed vendored checksums
PL/SQL support was contributed by Oguz (@ouzsrcm) in #1033; re-vendored
from upstream per vendoring policy with his language wiring distilled on
top.
Co-authored-by: Oğuz <oguzhan.saricam@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Routing .ets through the plain TypeScript grammar produced no definition
node for any @Component struct (0/19 across the OpenHarmony sample corpus),
parsed struct members as statement labels and build() as a call expression,
and registered ArkUI builtins (Column, Row, ListItem, ...) as user function
definitions with 28% fabricated cross-file CALLS edges. This adds a
dedicated ArkTS grammar that keeps everything TypeScript already handles
and models the ArkTS syntax TS cannot parse.
Grammar (tools/tree-sitter-arkts/, generated parser vendored to
internal/cbm/vendored/grammars/arkts/, ABI 15 via tree-sitter-cli 0.25.10):
first-party derivative of tree-sitter/tree-sitter-typescript v0.23.2
@ 75b3874edb2d (the commit our vendored typescript/tsx come from) on the
tree-sitter-javascript v0.23.1 base @ 3a837b6f3658 (the version upstream's
package-lock pins). LICENSE carries the upstream MIT text verbatim plus a
derivative notice; the external scanner is a symbol-rename trampoline over
a byte-identical copy of the already-vendored typescript scanner header.
ArkTS additions: decorated struct component declarations with class-shaped
bodies; UI-DSL trailing-closure component calls with attribute chains after
the closing brace; decorators on function declarations (@Builder,
@Extend(T), @Styles); @Extend/@Styles leading-dot attribute-chain bodies;
anonymous stateStyles style blocks; import lazy named imports.
Wiring: CBM_LANG_ARKTS enum + lang spec (struct_declaration as a class
kind labeled Struct, public_field_definition fields), .ets extension,
LANG_NAMES/userconfig entries, grammar shim, and TS-family joins across
the extractors and the name-resolution guards (parse_es_imports, JS
keywords/test-file/func-kind/scope tables, TS heritage/param/receiver/
namespace/type-ref/channel/usage paths, js_ts_family and the weak-method-
match suppression). Field definitions now also carry their decorators, so
@State/@Prop/@Link members are findable with their roles.
Tests: grammar corpus (35 cases), per-language regression + label golden +
import net + extension cases, seven ArkTS extraction tests including the
phantom-builtin regression (ArkUI builtins must never be definitions), and
capability-ledger / call-node-manifest / call-argument-matrix rows. On the
14-file OpenHarmony sample corpus the TS-grammar baseline had 12/14 files
with parse errors (231 ERROR nodes); the arkts grammar parses 14/14 clean,
and an end-to-end index yields all 19 components as Struct nodes, 68
methods, 119 fields, and zero phantom builtin definitions.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Removes the heaviest vendored grammar (66 MB) along with its language
wiring and test fixtures; the supported-language count moves to 158
across README, site, and manifest. The grammar manifest additionally
records the canonical-source decisions for the five
registry-disagreement grammars and the per-directory license files
restored earlier this week.
Helm chart helpers (.tpl) and Go templates produced no semantic graph — just a
flat key flood. Wire the existing gotemplate grammar for real structure:
- .tpl files map to the gotemplate language (Helm _helpers.tpl).
- {{ define "chart.fullname" }} becomes a Function node named after the
template (the name is a string literal, not an identifier, so it's resolved
in a dedicated walker branch).
- {{ include "x" . }} and {{ template "x" . }} resolve to the referenced
template name instead of the bare `include` builtin, so they form CALLS
edges to the define'd Function. template_action is added to the call set.
Lets you trace which charts include which named templates. Tests cover define
-> Function and include/template -> CALLS, plus the .tpl mapping.
Refs #338
Vendor the MIT-licensed cfmleditor/tree-sitter-cfml grammars and wire both
CFML dialects through the pipeline:
- .cfc components -> cfscript grammar (CBM_LANG_CFSCRIPT): a JS-like grammar, so
function/method declarations, calls and imports extract with the shared JS/TS
arrays. Components' functions become Function nodes.
- .cfm templates -> cfml tag grammar (CBM_LANG_CFML): tag-based (HTML-derived).
Embedded <cfscript> functions extract via function_declaration; tag-based
<cffunction name="..."> nodes are handled by a dedicated walker branch that
reads the name attribute (cf_function_tag has no name field). Shared scanner
headers are vendored at vendored/common/ so the grammar's ../../common/
includes resolve without editing vendored sources.
Adds an ASCII case-insensitive compare for CFML's case-insensitive attributes.
Tests cover both dialects and the extension mapping. LICENSEs vendored.
Closes#38
Vendor the MIT-licensed tree-sitter-qmljs grammar (a TypeScript superset with
declarative ui_* nodes) and wire .qml files through the full pipeline: language
enum, extension + display-name mapping, grammar shim, and a language spec that
reuses the JS/TS function/call/branch arrays and adds QML class/field/import
types (ui_inline_component, ui_property, ui_signal, ui_import).
QML files now produce Function nodes (incl. JS functions inside QML objects),
class/interface/enum nodes, imports and CALLS edges, so call chains through a
Qt project's .qml files can be traced. LICENSE vendored alongside the grammar.
Closes#42
Laravel Blade templates (*.blade.php) fell through to the single-extension
lookup and were mis-classified as PHP, so the PHP extractor tried to parse
Blade syntax. Add a built-in compound-extension table (checked before the
single-extension fallback and alongside user config) mapping .blade.php to
CBM_LANG_BLADE, so Blade files are discovered and parsed with the Blade
grammar. Combined with the now-non-blocking discovery gate (c29e6d5), this
resolves the "agent stuck on Blade files" report. Plain .php is unaffected.
.mjs (ES modules) and .cjs (CommonJS) were absent from the extension
table, so those files were never assigned a language — they weren't
indexed (no graph nodes) and search_code returned 0 for any pattern in
them. Add .mjs/.cjs → JavaScript and .mts/.cts → TypeScript.
Verified end-to-end: a function in a .mjs file is now extracted and found
by search_code. Closes#197.
Add automated extension-to-language mapping tests for all new grammars
that have file extensions. Removes conflicting extensions:
- .blade.php: compound extension, handled by userconfig system
- .env*: filenames, not extensions (no auto-discovery for dotfiles)
- .sv/.svh: already mapped to Verilog, SystemVerilog uses same grammar
Enforce WarningsAsErrors: '*' with zero suppressions of entire check categories.
All non-FP warnings fixed at source; true false positives get per-instance NOLINT
with reasoning. Fixes include: braces around statements (1679), isolate declarations
(75), explicit casts for multilevel pointer conversions, missing includes, function
parameter documentation via NOLINT, and proper NOLINTBEGIN/NOLINTEND nesting.
Complete C implementation of the indexing pipeline (src/), parallel worker
pool, graph buffer with merge support, SQLite writer, and 1893-test suite.
Linter setup: zero warnings from clang-tidy (all checks enabled, no NOLINT
suppressions), cppcheck, and clang-format. All issues fixed at source —
proper headers for external linkage, GROW_ARRAY macro restructured to
eliminate type parameter, null-deref paths guarded, named intermediates
for suspicious-argument checks.
Martin Vogel