mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-15 12:40:17 +08:00
chore(vendor): vendor the VB.NET tree-sitter grammar
Vendors govindbanura/tree-sitter-vbnet (MIT, (c) 2025 Govind Banura) at 538b7087bf80 (tag v0.1-beta, upstream HEAD) with CodeGraph's real-world VB.NET patch (colbymchenry/codegraph docs/grammars/tree-sitter-vbnet.patch at fe7542b6f6a4, MIT, (c) 2026 Colby Mchenry), as a self-maintained fork in tools/tree-sitter-vbnet/. This is the precondition for VB.NET extraction (#1179, folding in #2885); it registers no language yet. Why this grammar: it is the licensed one. CodeAnt-AI/tree-sitter-vb-dotnet, which #1179 was built on, carries no license and cannot be vendored. Why the patch: upstream checks in grammar.js only and does not parse the code VB.NET projects are made of. Unpatched, the Single literals in every WinForms .Designer.vb (New SizeF(6.0!, 13.0!)) and body-less MustOverride members are ERROR nodes. The 2026-10-06 revision of the patch is pinned because it also keeps names that begin with a keyword whole: the earlier revision (the one #2885 carried) turns `Sub NewItem()` into an ERROR and `Public SharedCache As Integer` into a Shared field named Cache. Provenance by hash: the patch file is byte-identical to CodeGraph's (blob37918d81); upstream grammar.js at the pin is the patch's pre-image (21c8523), and tools/tree-sitter-vbnet/grammar.js minus its 41-line provenance header is the post-image (c9c1d85), as is scanner.c (8334fdb). parser.c is generated here with tree-sitter-cli 0.25.10 (ABI 15); two independent generations are byte-identical. No upstream script, package hook or binary was run or vendored. - internal/cbm/vendored/grammars/vbnet/: the generated parser.c, CodeGraph's external scanner.c (two stateless tokens), the tree_sitter/ headers, upstream's MIT LICENSE byte-identical, and CodeGraph's MIT text as LICENSE-codegraph: the scanner and the patched rules are CodeGraph's work, so its notice has to ship with the binary too. - internal/cbm/grammar_vbnet.c: compiles and links tree_sitter_vbnet. No CBMLanguage, extension or alias routes to it yet; extraction adds them, with `vb` and `visualbasic` meaning VB.NET (VB6 keeps its own name). - MANIFEST.md (fork note, forks table, 163 grammars, ABI recount), THIRD_PARTY.md (fork list and a vbnet section; the stale "160 pre-generated parsers" now reads 163), scripts/vendored-checksums.txt, and the FORKS entry in scripts/audit-license-provenance.py. - README.md and the Chocolatey nuspec say 163 languages: the language-count contract defines the count as the vendored grammar directories. - tests/test_parse_coverage.c: real VB.NET (a WinForms Designer partial class, Handles, Inherits/Implements, MustOverride, #Region, Property, keyword-prefixed names) parses with no ERROR node. The case fails on unpatched upstream (8 ERROR nodes) and on the earlier patch revision (1). Refs #1178, #1179, #2885 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This commit is contained in:
Martin Vogel
@@ -5,7 +5,7 @@
|
||||
[](https://github.com/DeusData/codebase-memory-mcp/actions/workflows/release.yml)
|
||||
[](#tests)
|
||||
[](#mcp-tools)
|
||||
[](https://github.com/DeusData/codebase-memory-mcp)
|
||||
[](https://github.com/DeusData/codebase-memory-mcp)
|
||||
[](#hybrid-lsp)
|
||||
[](https://github.com/DeusData/codebase-memory-mcp)
|
||||
[](https://github.com/DeusData/codebase-memory-mcp)
|
||||
@@ -398,7 +398,7 @@ In this repository, test narrowing is wired to the Linux x86_64 test leg when `T
|
||||
- `SEMANTICALLY_RELATED` (vocabulary-mismatch, same-language, score ≥ 0.80)
|
||||
|
||||
### Indexing pipeline
|
||||
- **162 vendored tree-sitter grammars** compiled into the binary
|
||||
- **163 vendored tree-sitter grammars** compiled into the binary
|
||||
- **Generic package / module resolution**: bare specifiers like `@myorg/pkg`, `github.com/foo/bar`, `use my_crate::foo` resolved via manifest scanning (`package.json`, `go.mod`, `Cargo.toml`, `pyproject.toml`, `composer.json`, `pubspec.yaml`, `pom.xml`, `build.gradle`, `mix.exs`, `*.gemspec`)
|
||||
- **Infrastructure-as-code indexing**: Dockerfiles (`Dockerfile.*`, `*.Dockerfile`, `Containerfile`: bases, what the container runs, `ENV`), container images (`Image`: the normalised repository, tag and digest on the edges; a repository built from several Dockerfiles, one per component tag, gets an `Image` per component), docker compose services (`Resource`), Kubernetes manifests (`Resource`, with `group`, `version`, `kind`, `namespace`, `labels` as `k=v,k2=v2`), Kustomize overlays (`IMPORTS` to their entries and patch files, `OVERRIDE` from their patches), Helm charts (`Chart`, `DEPENDS_ON`), Terraform / OpenTofu (`.tf`, `.tofu`, `.tfvars`) as graph nodes. A Service walks to the code it runs: Service → workload → `Image` → Dockerfile → `main`
|
||||
- **[Hybrid LSP semantic type resolution](#hybrid-lsp)** for Python, TypeScript / JavaScript / JSX / TSX, PHP, C#, Go, C, C++, Java, Kotlin, Rust, and Perl; a lightweight C implementation of language type-resolution algorithms, structurally inspired by and compatible with major language servers including tsserver / typescript-go, pyright, gopls, Roslyn, Eclipse JDT, and rust-analyzer (parameter binding, return-type inference, generic substitution, JSX component dispatch, JSDoc inference for plain JS files, namespace + trait + late-static-binding resolution for PHP, file-scoped namespaces + records + LINQ method syntax for C#, class-hierarchy + overload + lambda resolution for Java, extension-function + scope-function resolution for Kotlin, trait-method + UFCS resolution for Rust)
|
||||
@@ -1124,14 +1124,14 @@ codebase-memory-mcp ships a **lightweight C implementation of language type-reso
|
||||
|
||||
**Two-layer architecture:**
|
||||
|
||||
1. **Tree-sitter pass**: fast, syntactic, runs for every one of the 162 languages. Extracts definitions, calls, imports.
|
||||
1. **Tree-sitter pass**: fast, syntactic, runs for every one of the 163 languages. Extracts definitions, calls, imports.
|
||||
2. **Hybrid LSP pass**: type-aware, runs above the tree-sitter pass per-language. Refines call edges using the import graph plus a per-file or pre-built cross-file definition registry. Languages without a Hybrid LSP pass yet fall back to textual resolution, with coverage varying by language and construct.
|
||||
|
||||
The result is a knowledge graph accurate enough to drive `trace_path` across packages, inheritance hierarchies, and stdlib calls; without paying for a language server process per project.
|
||||
|
||||
## Language Support
|
||||
|
||||
162 languages, counted by the vendored grammar directories in [the parser manifest](internal/cbm/vendored/grammars/MANIFEST.md). Benchmarked against 64 real open-source repositories (78 to 49K nodes):
|
||||
163 languages, counted by the vendored grammar directories in [the parser manifest](internal/cbm/vendored/grammars/MANIFEST.md). Benchmarked against 64 real open-source repositories (78 to 49K nodes):
|
||||
|
||||
| Tier | Score | Languages |
|
||||
|------|-------|-----------|
|
||||
@@ -1158,7 +1158,7 @@ src/
|
||||
ui/ Local HTTP server + embedded CBM Lens frontend
|
||||
foundation/ Platform abstractions (threads, filesystem, logging, memory)
|
||||
graph-ui/ CodeAtlasWeb: React, Monaco, and Three.js browser frontend
|
||||
internal/cbm/ Vendored tree-sitter grammars (162 languages) + AST extraction engine
|
||||
internal/cbm/ Vendored tree-sitter grammars (163 languages) + AST extraction engine
|
||||
```
|
||||
|
||||
## Security
|
||||
|
||||
+24
-4
@@ -37,7 +37,7 @@ The core runtime headers in `internal/cbm/vendored/common/tree_sitter/`
|
||||
|
||||
## Tree-sitter Grammars
|
||||
|
||||
160 pre-generated parsers are vendored in `internal/cbm/vendored/grammars/<lang>/`
|
||||
163 pre-generated parsers are vendored in `internal/cbm/vendored/grammars/<lang>/`
|
||||
(generated `parser.c` plus `scanner.c` where applicable, compiled statically).
|
||||
Each grammar is the work of its upstream authors and each grammar directory
|
||||
contains the upstream `LICENSE` file.
|
||||
@@ -60,8 +60,8 @@ License summary:
|
||||
smart-coin language, written for this project because no usable public
|
||||
grammar exists; its source and corpus tests live in
|
||||
`tools/tree-sitter-chialisp/`.
|
||||
- Ten further grammars (`arkts`, `assembly`, `cfml`,
|
||||
`cfscript`, `dotenv`, `javascript`, `pascal`, `pine`, `qml`, `tsx`) are
|
||||
- Eleven further grammars (`arkts`, `assembly`, `cfml`,
|
||||
`cfscript`, `dotenv`, `javascript`, `pascal`, `pine`, `qml`, `tsx`, `vbnet`) are
|
||||
self-maintained forks that retain their original upstream authors' licenses —
|
||||
see the manifest for per-grammar provenance. `arkts` is a first-party derivative of
|
||||
[tree-sitter/tree-sitter-typescript](https://github.com/tree-sitter/tree-sitter-typescript)
|
||||
@@ -79,7 +79,27 @@ License summary:
|
||||
([Isopod/tree-sitter-pascal](https://github.com/Isopod/tree-sitter-pascal),
|
||||
MIT, (c) 2018 Benjamin Gray) is the upstream grammar with one (c) 2026
|
||||
DeusData patch that accepts non-ASCII letters in identifiers; its grammar
|
||||
source lives in `tools/tree-sitter-pascal/`.
|
||||
source lives in `tools/tree-sitter-pascal/`. `vbnet` is described in its own
|
||||
section below.
|
||||
|
||||
### tree-sitter-vbnet (with CodeGraph's real-world VB.NET patch)
|
||||
|
||||
- **Project:** [govindbanura/tree-sitter-vbnet](https://github.com/govindbanura/tree-sitter-vbnet)
|
||||
- **License:** MIT
|
||||
- **Copyright:** (c) 2025 Govind Banura
|
||||
- **Patch:** `docs/grammars/tree-sitter-vbnet.patch` from
|
||||
[colbymchenry/codegraph](https://github.com/colbymchenry/codegraph)
|
||||
(CodeGraph), MIT, (c) 2026 Colby Mchenry. The patch rewrites much of the
|
||||
grammar and adds the external scanner (`scanner.c`), so CodeGraph's license
|
||||
text ships next to the code as `LICENSE-codegraph`.
|
||||
- **Vendored at:** `internal/cbm/vendored/grammars/vbnet/`
|
||||
- **Pinned commits:** grammar `538b7087bf80` (v0.1-beta); patch `fe7542b6f6a4`
|
||||
- **Notes:** Visual Basic .NET grammar, maintained here as a self-maintained
|
||||
fork: the grammar source, the verbatim patch and the regeneration recipe live
|
||||
in `tools/tree-sitter-vbnet/`, and the parser is generated in this repository
|
||||
with tree-sitter-cli 0.25.10 (upstream checks in no generated sources). The
|
||||
grammar is vendored ahead of VB.NET extraction (#1179); provenance hashes are
|
||||
in `internal/cbm/vendored/grammars/MANIFEST.md`.
|
||||
|
||||
### tree-sitter-plsql
|
||||
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
// Vendored tree-sitter grammar: vbnet
|
||||
// Each grammar compiled as separate unit (conflicting static symbols).
|
||||
#include "vendored/grammars/vbnet/parser.c"
|
||||
#include "vendored/grammars/vbnet/scanner.c"
|
||||
@@ -9,8 +9,9 @@ The grammars were originally vendored as bare `parser.c`+`scanner.c` with **no r
|
||||
|
||||
## Summary
|
||||
|
||||
- Grammars: **162** — vendored-from-upstream: **140**, first-party/self-maintained: **17**, registry-disagreement: **5** (nim removed 2026-06-12; objectscript_udl + objectscript_routine added 2026-06-24; mojo added 2026-07-01; arkts added 2026-08-26; plsql added 2026-08-27; chialisp added 2026-08-28; javascript + tsx became self-maintained forks 2026-09-25; pascal became a self-maintained fork 2026-09-25 — see notes below)
|
||||
- ABI distribution: **9×** ABI-13 **78×** ABI-14 **75×** ABI-15 (runtime ceiling is ABI 15; never vendor ABI 16 without a runtime upgrade)
|
||||
- Grammars: **163** — vendored-from-upstream: **140**, first-party/self-maintained: **18**, registry-disagreement: **5** (nim removed 2026-06-12; objectscript_udl + objectscript_routine added 2026-06-24; mojo added 2026-07-01; arkts added 2026-08-26; plsql added 2026-08-27; chialisp added 2026-08-28; javascript + tsx became self-maintained forks 2026-09-25; pascal became a self-maintained fork 2026-09-25; vbnet added as a self-maintained fork 2026-11-10 — see notes below)
|
||||
- ABI distribution: **9×** ABI-13 **78×** ABI-14 **76×** ABI-15 (runtime ceiling is ABI 15; never vendor ABI 16 without a runtime upgrade)
|
||||
— recounted from the tree 2026-11-10 after adding the ABI-15 `vbnet` grammar (was `9×/78×/75×`).
|
||||
— recounted from the tree 2026-09-25 after the tsx fork was regenerated with tree-sitter-cli 0.25.10, moving `tsx` from ABI 14 to ABI 15 (was `9×/79×/74×`).
|
||||
— recounted from the tree 2026-08-30 after the perl v1.2.1 refresh moved `perl` from ABI 14 to ABI 15 (was `9×/80×/73×`). Neither side of the rebase had this right: main's line was correct for main, and this branch still carried the pre-2026-08-28 `7×/84×/65×`. Regenerate, never increment.
|
||||
— recounted from the tree 2026-08-28. This line had drifted: it read `7×/86×/65×`, which sums to 158 against 161 vendored grammars, so it was wrong before Chialisp was added and incrementing it would have carried the error forward. Regenerate with:
|
||||
@@ -55,6 +56,7 @@ The grammars were originally vendored as bare `parser.c`+`scanner.c` with **no r
|
||||
- **arkts** (added 2026-08-26): **first-party derivative** — a fork of [tree-sitter/tree-sitter-typescript](https://github.com/tree-sitter/tree-sitter-typescript)'s `typescript` dialect pinned @ `75b3874edb2d` (v0.23.2, the same commit our vendored typescript/tsx come from), on the tree-sitter-javascript base @ `3a837b6f3658` (v0.23.1, the version upstream's own package-lock pins), extended with ArkTS/ArkUI syntax (`@Component struct` declarations, UI-DSL trailing-closure calls with post-block attribute chains, decorated function declarations, `import lazy`, `@Extend`/`@Styles` leading-dot attribute chains, anonymous `stateStyles` style blocks). Grammar source + corpus tests live in `tools/tree-sitter-arkts/`; regenerate with `npx tree-sitter-cli@0.25.10 generate` (ABI 15). **External scanner:** `scanner.c` is a symbol-rename trampoline; `_common_scanner.h` is byte-identical (same SHA-256) to the reviewed `typescript/_common_scanner.h` already shipped. **LICENSE:** tree-sitter-typescript's MIT text **verbatim and byte-identical** (© 2017 Max Brunsfeld) — which is what MIT requires of a derivative work, and what lets the provenance audit byte-verify it against upstream rather than take a note on trust. The fork is registered in the audit's `FORKS` map. Our own copyright for the ArkTS additions, and the tree-sitter-javascript attribution (© 2014 Max Brunsfeld), live with the SOURCE in `tools/tree-sitter-arkts/grammar.js` and in `THIRD_PARTY.md`, which is what ships in the release archives.
|
||||
- **javascript + tsx** (forked 2026-09-25, #1736): **self-maintained forks** of the pins we already vendored — [tree-sitter/tree-sitter-javascript](https://github.com/tree-sitter/tree-sitter-javascript) @ `58404d8cf191` and [tree-sitter/tree-sitter-typescript](https://github.com/tree-sitter/tree-sitter-typescript)'s `tsx` dialect @ `75b3874edb2d` (javascript base @ `3a837b6f3658`, the version upstream's package-lock pins, as for arkts). **Why:** upstream accepts a `&` inside a JSX string only before a space/digit or as a complete character reference, and never inside JSX text, so a plain URL query (`href="…?a=1&b=2"`, Google Fonts `&display=swap`) or `<p>Tom & Jerry</p>` produced an ERROR node — a `parse_partial` flag, plus false USAGE edges from identifiers the error recovery pulled out of the URL (on vercel/next.js `examples/`: 45 of 92 flagged files, 115 false cross-example USAGE edges). **The change** is one patch, `tools/tree-sitter-javascript/patches/jsx-lone-ampersand.patch`, applied to both javascript bases: a lone `&` that does not begin a complete `html_character_reference` is a one-character token (lexical precedence -1, so `&`/`&` still lex as references) aliased to the EXISTING `string_fragment` / `jsx_text` nodes, and the string-fragment regexes stop before a `&` that directly precedes the closing quote. Node types are unchanged (same type/named set; `node-types.json` only gains an empty `fields` map on those two entries). Without the patch both directories regenerate upstream exactly (javascript: byte-identical `parser.c` apart from the generator-version comment; tsx: identical state/symbol counts, ABI 14 → 15 from the newer CLI). **External scanners** are unchanged: `javascript/scanner.c` and `tsx/scanner.c` + `_common_scanner.h` stay the reviewed upstream bytes. Regenerate with `npx tree-sitter-cli@0.25.10 generate` in `tools/tree-sitter-{javascript,tsx}/` (ABI 15) and copy `src/parser.c` (tsx: also `src/tree_sitter/*.h`) here. **On an upstream bump:** re-fetch the new pins, re-apply the patch, regenerate, re-run `test/corpus/jsx_ampersand.txt` — and drop the fork if upstream fixes it. **LICENSE:** upstream's MIT text byte-identical (registered in the audit's `FORKS` map); our copyright for the patch lives in the fork `grammar.js` headers and `THIRD_PARTY.md`.
|
||||
- **pascal** (forked 2026-09-25, #2211): **self-maintained fork** of the pin we already vendored — [Isopod/tree-sitter-pascal](https://github.com/Isopod/tree-sitter-pascal) @ `042119eca2e1` (full: `042119eca2e18a60e56317fb06ee3ba5c32cb447`, v0.10.2, upstream `master` HEAD at fork time). **Why:** upstream's `identifier` token is ASCII-only (`/[&]?[a-zA-Z_]+[0-9_a-zA-Z]*/`), so a Delphi identifier with non-ASCII letters — e.g. the Cyrillic member names of 1C's OLE API called through late-bound Variants — produced ERROR nodes and a `parse_partial` flag. **The change** is one patch, `tools/tree-sitter-pascal/patches/unicode-identifiers.patch`, touching only that token: `/[&]?[\p{L}_][\p{L}\p{M}\p{Nd}_]*/` (a Unicode letter or `_` first, then letters, combining marks, decimal digits or `_`). Keywords are still extracted from this `word` token, so `begin`/`end`/`procedure` lex as before (and `begin` glued to Cyrillic letters is one identifier). Node types are unchanged: `node-types.json` is identical to the unpatched regeneration, and versus upstream's 0.24-generated file it only gains the 0.25 generator's `"extra": true` markers. The `\p{}` classes compile to numeric code-point ranges, so the vendored `parser.c` stays **0 non-ASCII bytes**. **Fidelity:** upstream's `parser.c` carries no generator stamp (its package.json pins tree-sitter-cli ^0.24.5). Regenerating the UNPATCHED grammar with 0.25.10 gives the same symbol/token/field/production counts (only state minimisation differs, 4299 vs 2715 states) and parse trees byte-identical to upstream's shipped parser on upstream's 88 corpus cases and its three `examples/*.pas` (37,773 tree lines for `codetoolmanager.pas`); the patched parser (2715 states) gives the same identical trees on all of those inputs. Regenerate with `npx tree-sitter-cli@0.25.10 generate --abi 14` in `tools/tree-sitter-pascal/` (ABI 14, as the upstream pin, so the ABI tally is unchanged) and copy `src/parser.c` + `src/tree_sitter/parser.h` here (`parser.h` is byte-identical to the reviewed chialisp copy); no external scanner. **On an upstream bump:** re-fetch the pinned `grammar.js`, re-apply the patch, regenerate, re-run `test/corpus/unicode_identifiers.txt` plus upstream's corpus — and drop the fork if upstream accepts Unicode identifiers. **LICENSE:** upstream's MIT text byte-identical (registered in the audit's `FORKS` map); our copyright for the patch lives in the fork `grammar.js` header and `THIRD_PARTY.md`.
|
||||
- **vbnet** (added 2026-11-10; refs #1178, #1179): **self-maintained fork** of [govindbanura/tree-sitter-vbnet](https://github.com/govindbanura/tree-sitter-vbnet) @ `538b7087bf80` (full: `538b7087bf80e86004531b392fe1186379c0a2b5`, tag `v0.1-beta`, upstream `main` HEAD; no upstream commit since 2025-07-30) — MIT, (c) 2025 Govind Banura — with CodeGraph's real-world VB.NET patch, `docs/grammars/tree-sitter-vbnet.patch` from [colbymchenry/codegraph](https://github.com/colbymchenry/codegraph) @ `fe7542b6f6a4` (full: `fe7542b6f6a4939b89964affeea74f78a1c78b96`, the latest revision of that file, 2026-10-06) — MIT, (c) 2026 Colby Mchenry. **Why this grammar:** it is the licensed VB.NET grammar; `CodeAnt-AI/tree-sitter-vb-dotnet` carries no license (GitHub reports none; no LICENSE file), so it cannot be vendored. **Why the patch:** upstream checks in only `grammar.js` (no generated `src/`, no scanner) and parses textbook VB.NET but not real code: unpatched, the Single literals every WinForms `.Designer.vb` carries (`New System.Drawing.SizeF(6.0!, 13.0!)`) and body-less `MustOverride` members are ERROR nodes, and names that begin with a keyword are split (`Public SharedCache` lexes as `Shared` + `Cache`). The patch, `tools/tree-sitter-vbnet/patches/tree-sitter-vbnet.patch` (byte-identical to CodeGraph's, git blob `37918d81ed4e`), edits `grammar.js` and adds `src/scanner.c`, an external scanner for two tokens (`EXTERNAL_TOKEN_COUNT 2`: multi-line LINQ clause continuation and opaque XML literals; stateless, no allocation, no I/O; its comments carry typographic non-ASCII characters, its code none). CodeGraph's `docs/grammars/tree-sitter-vbnet.md` at the same commit lists every construct the patch adds and the parse-health census behind it. Its 2026-10-06 revision is the one pinned because it fixes the keyword-prefix split; the earlier revision (`e871c49a`, the one PR #2885 carried) still turns `Sub NewItem()` into an ERROR and `Public SharedCache As Integer` into a `Shared` field named `Cache`. **Provenance by hash:** upstream `grammar.js` at the pin is blob `21c8523b88f3`, the patch's pre-image; `tools/tree-sitter-vbnet/grammar.js` minus its 41-line provenance header is blob `c9c1d85f91ae`, the patch's post-image, and `scanner.c` is blob `8334fdb92fd7`, both exactly as the patch's `index` lines record. **Generated here** with tree-sitter-cli 0.25.10 (`da6fe9beb4f7`) from that `grammar.js`: ABI 15, 14,842 states, 976,623 lines, 0 non-ASCII bytes, no string literal containing a space, one comment (the generator stamp), no call outside the tree-sitter macros and the scanner hooks; two independent generations are byte-identical (`parser.c` SHA-256 `d0c29e9a7741…`, pinned in `scripts/vendored-checksums.txt`). Upstream's other files were not run or vendored (it also checks in a Windows `parser.obj`/`parser.lib`/`parser.exp` and a `build.py`). Upstream's `test/corpus` predates its own `grammar.js` (CodeGraph reports every case failing at the pin), so it is not carried; regression coverage is `vbnet_vendored_grammar_parses_real_vb_clean` in `tests/test_parse_coverage.c`. **Known gaps** (CodeGraph's notes, reproduced here): a GoTo label at column 0 leaves a localized ERROR, and a file whose last line has no trailing newline ends in a MISSING newline (upstream's `_eof` rule); VB.NET extraction must account for both. **Registration:** grammar only. `internal/cbm/grammar_vbnet.c` compiles and links `tree_sitter_vbnet`, but no `CBMLanguage`, file extension or name alias routes to it yet; VB.NET extraction (#1179, with #2885 folded in) adds them, with `vb` and `visualbasic` meaning VB.NET (VB6 keeps its own name). Regenerate with `npx tree-sitter-cli@0.25.10 generate` in `tools/tree-sitter-vbnet/` and copy `src/parser.c`, `src/scanner.c` and `src/tree_sitter/*.h` here. **On an upstream bump:** re-fetch both pins, re-apply the patch, regenerate, re-run the test, and drop the fork if upstream absorbs the patch. **LICENSE:** `LICENSE` is upstream's MIT text byte-identical (SHA-256 `fa7ab39bcf92…`; registered in the audit's `FORKS` map); `LICENSE-codegraph` is CodeGraph's MIT text byte-identical to `colbymchenry/codegraph`'s `LICENSE` at the patch commit (SHA-256 `e6d98f98c666…`), shipped next to the code because the scanner and the patched rules are CodeGraph's work.
|
||||
- **plsql** (added 2026-08-27): vendored from [AndreasMaierDe/tree-sitter-plsql](https://github.com/AndreasMaierDe/tree-sitter-plsql) @ `28aebef209be` (full: `28aebef209be57169600e1aa41ca1431cc6c916f`, upstream tip; repo dormant since 2023-02) — MIT, ABI 14, **no external scanner** (`EXTERNAL_TOKEN_COUNT 0`). Not listed in nvim-treesitter/Helix (`community-niche`, an individual-maintained grammar); provenance verified directly against upstream (pin = `git ls-remote` HEAD; vendored `parser.c` byte-identical to the pinned clone apart from the include-quote local patch below; 0 non-ASCII bytes; 0 dangerous calls). Security review covered only the vendored C surface (`parser.c`, `tree_sitter/parser.h`) plus upstream license/provenance metadata; no package manager hooks, workflow files, prompt/agent instruction files, or generated lockfiles were vendored. Known upstream limitation: `CREATE TYPE ... AS OBJECT` currently yields ERROR nodes (pinned by `plsql_create_type_as_object_limitation` in `tests/test_extraction.c` + `tests/fixtures/plsql/create_type_as_object_limitation.tps`). Originally contributed as PR #1033 by Oğuz (@ouzsrcm); re-vendored from upstream per vendoring policy with the PR's language wiring distilled on top.
|
||||
|
||||
- **chialisp** (added 2026-08-28): **first-party** — authored in this repository, not vendored from anywhere. Grammar source + corpus tests live in `tools/tree-sitter-chialisp/`; regenerate with `npx tree-sitter-cli@0.25.10 generate --abi 14` and copy `src/parser.c` + `src/tree_sitter/*.h` here. ABI 14, **no external scanner** (`EXTERNAL_TOKEN_COUNT 0`), 0 non-ASCII bytes. It is a deliberately GENERIC s-expression grammar (`source_file`/`list`/`symbol`/`string`/`number`/`hex`/`dot`/`comment`) modelling the clvm_tools reader rather than the Chialisp form vocabulary: `mod`/`defun`/`defconstant`/`include` are ordinary head symbols, and which lists are definitions is decided in `internal/cbm/extract_defs.c`, so a dialect that adds a form does not need a regenerated parser. Written because the only public grammar (`Quexington/tree-sitter-chialisp`) cannot parse the language: it required CRLF to terminate a comment (`/;.*\r\n/`) while real files are LF, rejected the `.` in `(include foo.clib)`, and accepted only a primitive after `(defconstant NAME …)` — each of which desynchronised the rest of the file. Acceptance gate: all five `chia-blockchain@main` reference files parse with **zero ERROR and zero MISSING nodes**. **LICENSE:** the project's own LICENSE, byte-identical to the repository root — no third-party copyright is carried, because there is no third party.
|
||||
@@ -291,6 +293,7 @@ upstream author (correctly retained). The table now records the true origin.
|
||||
| javascript | 15 | **self-maintained fork** of tree-sitter/tree-sitter-javascript @ `58404d8cf191` (the previously vendored pin, VERIFIED-BOTH) + the JSX lone-ampersand patch (#1736); grammar source in `tools/tree-sitter-javascript/` | MIT, (c) 2014 Max Brunsfeld — byte-identical to upstream; patch (c) 2026 DeusData, see grammar.js + THIRD_PARTY.md |
|
||||
| tsx | 15 | **self-maintained fork** of tree-sitter/tree-sitter-typescript's `tsx` dialect @ `75b3874edb2d` (v0.23.2, the previously vendored pin, VERIFIED-BOTH; javascript base @ `3a837b6f3658` v0.23.1) + the same JSX lone-ampersand patch (#1736); grammar source in `tools/tree-sitter-tsx/` | MIT, (c) 2017 Max Brunsfeld — byte-identical to upstream; patch (c) 2026 DeusData, see grammar.js + THIRD_PARTY.md |
|
||||
| pascal | 14 | **self-maintained fork** of Isopod/tree-sitter-pascal @ `042119eca2e1` (v0.10.2, the previously vendored pin, VERIFIED-BOTH) + the Unicode-identifier patch (#2211); grammar source in `tools/tree-sitter-pascal/` | MIT, (c) 2018 Benjamin Gray — byte-identical to upstream; patch (c) 2026 DeusData, see grammar.js + THIRD_PARTY.md |
|
||||
| vbnet | 15 | **self-maintained fork** of govindbanura/tree-sitter-vbnet @ `538b7087bf80` (v0.1-beta, upstream HEAD) + CodeGraph's real-world VB.NET patch (colbymchenry/codegraph @ `fe7542b6f6a4`); grammar source in `tools/tree-sitter-vbnet/` | MIT, (c) 2025 Govind Banura — byte-identical to upstream; patch + scanner MIT, (c) 2026 Colby Mchenry — `LICENSE-codegraph`, byte-identical to CodeGraph's |
|
||||
| assembly | 14 | RubixDev/tree-sitter-assembly (**repo deleted from GitHub** — our retained MIT copy, (c) 2023 RubixDev, is the surviving grant) | MIT |
|
||||
| cfml | 15 | cfmleditor/tree-sitter-cfml | MIT, (c) 2014 Gareth Edwards & Gavin Baumanis — byte-identical |
|
||||
| cfscript | 15 | cfmleditor/tree-sitter-cfml | MIT, same — byte-identical |
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2025 Govind Banura
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 Colby Mchenry
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
+976623
File diff suppressed because it is too large
Load Diff
+303
@@ -0,0 +1,303 @@
|
||||
#include "tree_sitter/parser.h"
|
||||
#include <string.h>
|
||||
#include <wctype.h>
|
||||
|
||||
// External tokens for the two constructs that are not LR(1)-parseable with
|
||||
// tree-sitter's newline-as-extra treatment:
|
||||
//
|
||||
// 1. QUERY_CLAUSE_CONTINUATION — a newline run that CONTINUES a LINQ query
|
||||
// (`From x In xs` ↵ `Where …`). At a clause boundary the parser cannot
|
||||
// know from the newline alone whether the query ends (newline = statement
|
||||
// terminator) or continues (newline = insignificant, next line starts with
|
||||
// a query keyword). The scanner looks past the newline at the next word
|
||||
// and only emits the token when it is a query-clause keyword — so the
|
||||
// terminator/continuation choice is made by the lexer, not the LR table.
|
||||
//
|
||||
// 2. XML_LITERAL — a whole VB XML literal (`<Tags><Tag/></Tags>`, including
|
||||
// `<%= expr %>` embedded expressions, comments, CDATA) consumed as one
|
||||
// opaque token. Only valid where a literal can begin an expression, so a
|
||||
// relational `<` (which always FOLLOWS an expression) can never collide.
|
||||
|
||||
enum TokenType {
|
||||
QUERY_CLAUSE_CONTINUATION,
|
||||
XML_LITERAL,
|
||||
};
|
||||
|
||||
void *tree_sitter_vbnet_external_scanner_create(void) { return NULL; }
|
||||
void tree_sitter_vbnet_external_scanner_destroy(void *payload) { (void)payload; }
|
||||
unsigned tree_sitter_vbnet_external_scanner_serialize(void *payload, char *buffer) {
|
||||
(void)payload; (void)buffer;
|
||||
return 0; // stateless
|
||||
}
|
||||
void tree_sitter_vbnet_external_scanner_deserialize(void *payload, const char *buffer, unsigned length) {
|
||||
(void)payload; (void)buffer; (void)length;
|
||||
}
|
||||
|
||||
static inline void advance(TSLexer *lexer) { lexer->advance(lexer, false); }
|
||||
static inline void skip_ws(TSLexer *lexer) { lexer->advance(lexer, true); }
|
||||
|
||||
// Read one alphabetic word (advancing past it), lowercased into buf.
|
||||
// Returns its length (0 if the next char is not a letter).
|
||||
static int read_word(TSLexer *lexer, char *buf, int cap) {
|
||||
int n = 0;
|
||||
while (iswalpha((wint_t)lexer->lookahead)) {
|
||||
if (n < cap - 1) buf[n] = (char)towlower((wint_t)lexer->lookahead);
|
||||
n++;
|
||||
advance(lexer);
|
||||
}
|
||||
buf[n < cap - 1 ? n : cap - 1] = 0;
|
||||
return n;
|
||||
}
|
||||
|
||||
static bool is_query_keyword(const char *w) {
|
||||
static const char *kws[] = {
|
||||
"where", "select", "order", "group", "join", "let", "skip",
|
||||
"take", "distinct", "aggregate", "from", "into", "on", NULL,
|
||||
};
|
||||
for (int i = 0; kws[i]; i++)
|
||||
if (strcmp(w, kws[i]) == 0) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
// [ \t\r]* '\n' [ \t\r\n]* followed by a query-clause keyword.
|
||||
// The token covers only the whitespace/newline run (mark_end before the
|
||||
// keyword lookahead), so the clause keyword itself lexes normally after it.
|
||||
static bool scan_query_continuation(TSLexer *lexer) {
|
||||
bool saw_newline = false;
|
||||
for (;;) {
|
||||
int32_t c = lexer->lookahead;
|
||||
if (c == ' ' || c == '\t' || c == '\r') {
|
||||
advance(lexer);
|
||||
} else if (c == '\n') {
|
||||
saw_newline = true;
|
||||
advance(lexer);
|
||||
} else {
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (!saw_newline) return false;
|
||||
lexer->mark_end(lexer);
|
||||
|
||||
char w[12];
|
||||
int n = read_word(lexer, w, sizeof w);
|
||||
if (n == 0 || n >= (int)(sizeof w)) return false;
|
||||
if (!is_query_keyword(w)) return false;
|
||||
|
||||
// `Select Case` opens a statement, never a select clause.
|
||||
if (strcmp(w, "select") == 0) {
|
||||
while (lexer->lookahead == ' ' || lexer->lookahead == '\t') advance(lexer);
|
||||
char w2[8];
|
||||
int m = read_word(lexer, w2, sizeof w2);
|
||||
if (m > 0 && m < (int)(sizeof w2) && strcmp(w2, "case") == 0) return false;
|
||||
}
|
||||
|
||||
lexer->result_symbol = QUERY_CLAUSE_CONTINUATION;
|
||||
return true;
|
||||
}
|
||||
|
||||
// Consume a VB string inside an embedded `<%= … %>` region ("" escapes).
|
||||
static void consume_vb_string(TSLexer *lexer) {
|
||||
advance(lexer); // opening quote
|
||||
for (;;) {
|
||||
if (lexer->lookahead == 0) return;
|
||||
if (lexer->lookahead == '"') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead != '"') return; // closing (not an escaped "")
|
||||
// escaped quote: fall through, keep consuming
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
|
||||
// Consume `<%= … %>` starting at the '%' (the '<' is already consumed).
|
||||
// Embedded expressions NEST (`<%= From t In ts Select <Tag><%= t.Name %></Tag> %>`,
|
||||
// the staxrip WriteTagfile shape), so track <% / %> depth.
|
||||
static bool consume_embedded_expression(TSLexer *lexer) {
|
||||
advance(lexer); // '%'
|
||||
int depth = 1;
|
||||
for (;;) {
|
||||
int32_t c = lexer->lookahead;
|
||||
if (c == 0) return false;
|
||||
if (c == '"') { consume_vb_string(lexer); continue; }
|
||||
if (c == '<') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == '%') { depth++; advance(lexer); }
|
||||
continue;
|
||||
}
|
||||
if (c == '%') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == '>') {
|
||||
advance(lexer);
|
||||
if (--depth == 0) return true;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
|
||||
// Consume `<!-- … -->` / `<![CDATA[ … ]]>` / `<!DOCTYPE …>` starting at '!'.
|
||||
static bool consume_bang_construct(TSLexer *lexer) {
|
||||
advance(lexer); // '!'
|
||||
if (lexer->lookahead == '-') {
|
||||
// comment: to -->
|
||||
int dashes = 0;
|
||||
for (;;) {
|
||||
int32_t c = lexer->lookahead;
|
||||
if (c == 0) return false;
|
||||
if (c == '-') { dashes++; advance(lexer); continue; }
|
||||
if (c == '>' && dashes >= 2) { advance(lexer); return true; }
|
||||
dashes = 0;
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
if (lexer->lookahead == '[') {
|
||||
// CDATA: to ]]>
|
||||
int brackets = 0;
|
||||
for (;;) {
|
||||
int32_t c = lexer->lookahead;
|
||||
if (c == 0) return false;
|
||||
if (c == ']') { brackets++; advance(lexer); continue; }
|
||||
if (c == '>' && brackets >= 2) { advance(lexer); return true; }
|
||||
brackets = 0;
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
// DOCTYPE-ish: to bare >
|
||||
for (;;) {
|
||||
int32_t c = lexer->lookahead;
|
||||
if (c == 0) return false;
|
||||
if (c == '>') { advance(lexer); return true; }
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
|
||||
// Consume `<? … ?>` starting at the '?'.
|
||||
static bool consume_processing_instruction(TSLexer *lexer) {
|
||||
advance(lexer); // '?'
|
||||
for (;;) {
|
||||
int32_t c = lexer->lookahead;
|
||||
if (c == 0) return false;
|
||||
if (c == '?') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == '>') { advance(lexer); return true; }
|
||||
continue;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
|
||||
static bool is_name_start(int32_t c) {
|
||||
return iswalpha((wint_t)c) || c == '_';
|
||||
}
|
||||
|
||||
static bool scan_xml_literal(TSLexer *lexer) {
|
||||
// Only same-line blanks are skipped: a leading newline must stay unconsumed
|
||||
// so it can serve as a statement/block terminator (a successful scan would
|
||||
// otherwise swallow it as trivia). XML on a continuation line still works:
|
||||
// the newline is consumed as an extra first, then the scanner re-runs.
|
||||
while (lexer->lookahead == ' ' || lexer->lookahead == '\t') skip_ws(lexer);
|
||||
if (lexer->lookahead != '<') return false;
|
||||
advance(lexer);
|
||||
// Only element-start markup opens an XML literal here; comments/PIs as the
|
||||
// outermost construct (`Dim d = <?xml …`) are rare enough to leave alone,
|
||||
// and a bare `<` (comparison) never begins an expression.
|
||||
if (!is_name_start(lexer->lookahead)) return false;
|
||||
|
||||
int depth = 0; // open (unclosed) elements
|
||||
bool in_tag = true; // inside <...> of an element tag
|
||||
bool closing = false;
|
||||
int32_t quote = 0;
|
||||
|
||||
for (;;) {
|
||||
int32_t c = lexer->lookahead;
|
||||
if (c == 0) return false; // unterminated: let the normal parser error
|
||||
|
||||
if (in_tag) {
|
||||
if (quote) {
|
||||
if (c == quote) quote = 0;
|
||||
advance(lexer);
|
||||
continue;
|
||||
}
|
||||
if (c == '"' || c == '\'') { quote = c; advance(lexer); continue; }
|
||||
if (c == '<') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == '%') {
|
||||
// attribute value embedded expression: attr=<%= x %>
|
||||
if (!consume_embedded_expression(lexer)) return false;
|
||||
continue;
|
||||
}
|
||||
continue; // stray < in a tag: malformed, keep consuming
|
||||
}
|
||||
if (c == '/') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == '>') {
|
||||
// self-closing tag
|
||||
advance(lexer);
|
||||
in_tag = false;
|
||||
if (depth == 0) break; // single self-closed root: done
|
||||
continue;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (c == '>') {
|
||||
advance(lexer);
|
||||
if (closing) {
|
||||
depth--;
|
||||
if (depth <= 0) break; // root element closed: done
|
||||
} else {
|
||||
depth++;
|
||||
}
|
||||
in_tag = false;
|
||||
continue;
|
||||
}
|
||||
advance(lexer);
|
||||
continue;
|
||||
}
|
||||
|
||||
// text content between tags
|
||||
if (c == '<') {
|
||||
advance(lexer);
|
||||
int32_t d = lexer->lookahead;
|
||||
if (d == '/') { advance(lexer); closing = true; in_tag = true; continue; }
|
||||
if (d == '%') {
|
||||
if (!consume_embedded_expression(lexer)) return false;
|
||||
continue;
|
||||
}
|
||||
if (d == '!') {
|
||||
if (!consume_bang_construct(lexer)) return false;
|
||||
continue;
|
||||
}
|
||||
if (d == '?') {
|
||||
if (!consume_processing_instruction(lexer)) return false;
|
||||
continue;
|
||||
}
|
||||
closing = false;
|
||||
in_tag = true; // opening tag
|
||||
continue;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
|
||||
lexer->mark_end(lexer);
|
||||
lexer->result_symbol = XML_LITERAL;
|
||||
return true;
|
||||
}
|
||||
|
||||
bool tree_sitter_vbnet_external_scanner_scan(void *payload, TSLexer *lexer, const bool *valid_symbols) {
|
||||
(void)payload;
|
||||
int32_t c = lexer->lookahead;
|
||||
bool ws = (c == ' ' || c == '\t' || c == '\n' || c == '\r');
|
||||
|
||||
// A scan attempt advances the lexer, so exactly ONE attempt runs per
|
||||
// invocation — never chain a second attempt after a failed first (its
|
||||
// reads would start mid-stream). The leading character picks the branch;
|
||||
// a false return discards the attempt entirely.
|
||||
if (valid_symbols[QUERY_CLAUSE_CONTINUATION] && ws) {
|
||||
return scan_query_continuation(lexer);
|
||||
}
|
||||
if (valid_symbols[XML_LITERAL] && (c == '<' || ws)) {
|
||||
return scan_xml_literal(lexer);
|
||||
}
|
||||
return false;
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
#ifndef TREE_SITTER_ALLOC_H_
|
||||
#define TREE_SITTER_ALLOC_H_
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
#include <stdbool.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
|
||||
// Allow clients to override allocation functions
|
||||
#ifdef TREE_SITTER_REUSE_ALLOCATOR
|
||||
|
||||
extern void *(*ts_current_malloc)(size_t size);
|
||||
extern void *(*ts_current_calloc)(size_t count, size_t size);
|
||||
extern void *(*ts_current_realloc)(void *ptr, size_t size);
|
||||
extern void (*ts_current_free)(void *ptr);
|
||||
|
||||
#ifndef ts_malloc
|
||||
#define ts_malloc ts_current_malloc
|
||||
#endif
|
||||
#ifndef ts_calloc
|
||||
#define ts_calloc ts_current_calloc
|
||||
#endif
|
||||
#ifndef ts_realloc
|
||||
#define ts_realloc ts_current_realloc
|
||||
#endif
|
||||
#ifndef ts_free
|
||||
#define ts_free ts_current_free
|
||||
#endif
|
||||
|
||||
#else
|
||||
|
||||
#ifndef ts_malloc
|
||||
#define ts_malloc malloc
|
||||
#endif
|
||||
#ifndef ts_calloc
|
||||
#define ts_calloc calloc
|
||||
#endif
|
||||
#ifndef ts_realloc
|
||||
#define ts_realloc realloc
|
||||
#endif
|
||||
#ifndef ts_free
|
||||
#define ts_free free
|
||||
#endif
|
||||
|
||||
#endif
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif // TREE_SITTER_ALLOC_H_
|
||||
@@ -0,0 +1,291 @@
|
||||
#ifndef TREE_SITTER_ARRAY_H_
|
||||
#define TREE_SITTER_ARRAY_H_
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
#include "./alloc.h"
|
||||
|
||||
#include <assert.h>
|
||||
#include <stdbool.h>
|
||||
#include <stdint.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
|
||||
#ifdef _MSC_VER
|
||||
#pragma warning(push)
|
||||
#pragma warning(disable : 4101)
|
||||
#elif defined(__GNUC__) || defined(__clang__)
|
||||
#pragma GCC diagnostic push
|
||||
#pragma GCC diagnostic ignored "-Wunused-variable"
|
||||
#endif
|
||||
|
||||
#define Array(T) \
|
||||
struct { \
|
||||
T *contents; \
|
||||
uint32_t size; \
|
||||
uint32_t capacity; \
|
||||
}
|
||||
|
||||
/// Initialize an array.
|
||||
#define array_init(self) \
|
||||
((self)->size = 0, (self)->capacity = 0, (self)->contents = NULL)
|
||||
|
||||
/// Create an empty array.
|
||||
#define array_new() \
|
||||
{ NULL, 0, 0 }
|
||||
|
||||
/// Get a pointer to the element at a given `index` in the array.
|
||||
#define array_get(self, _index) \
|
||||
(assert((uint32_t)(_index) < (self)->size), &(self)->contents[_index])
|
||||
|
||||
/// Get a pointer to the first element in the array.
|
||||
#define array_front(self) array_get(self, 0)
|
||||
|
||||
/// Get a pointer to the last element in the array.
|
||||
#define array_back(self) array_get(self, (self)->size - 1)
|
||||
|
||||
/// Clear the array, setting its size to zero. Note that this does not free any
|
||||
/// memory allocated for the array's contents.
|
||||
#define array_clear(self) ((self)->size = 0)
|
||||
|
||||
/// Reserve `new_capacity` elements of space in the array. If `new_capacity` is
|
||||
/// less than the array's current capacity, this function has no effect.
|
||||
#define array_reserve(self, new_capacity) \
|
||||
_array__reserve((Array *)(self), array_elem_size(self), new_capacity)
|
||||
|
||||
/// Free any memory allocated for this array. Note that this does not free any
|
||||
/// memory allocated for the array's contents.
|
||||
#define array_delete(self) _array__delete((Array *)(self))
|
||||
|
||||
/// Push a new `element` onto the end of the array.
|
||||
#define array_push(self, element) \
|
||||
(_array__grow((Array *)(self), 1, array_elem_size(self)), \
|
||||
(self)->contents[(self)->size++] = (element))
|
||||
|
||||
/// Increase the array's size by `count` elements.
|
||||
/// New elements are zero-initialized.
|
||||
#define array_grow_by(self, count) \
|
||||
do { \
|
||||
if ((count) == 0) break; \
|
||||
_array__grow((Array *)(self), count, array_elem_size(self)); \
|
||||
memset((self)->contents + (self)->size, 0, (count) * array_elem_size(self)); \
|
||||
(self)->size += (count); \
|
||||
} while (0)
|
||||
|
||||
/// Append all elements from one array to the end of another.
|
||||
#define array_push_all(self, other) \
|
||||
array_extend((self), (other)->size, (other)->contents)
|
||||
|
||||
/// Append `count` elements to the end of the array, reading their values from the
|
||||
/// `contents` pointer.
|
||||
#define array_extend(self, count, contents) \
|
||||
_array__splice( \
|
||||
(Array *)(self), array_elem_size(self), (self)->size, \
|
||||
0, count, contents \
|
||||
)
|
||||
|
||||
/// Remove `old_count` elements from the array starting at the given `index`. At
|
||||
/// the same index, insert `new_count` new elements, reading their values from the
|
||||
/// `new_contents` pointer.
|
||||
#define array_splice(self, _index, old_count, new_count, new_contents) \
|
||||
_array__splice( \
|
||||
(Array *)(self), array_elem_size(self), _index, \
|
||||
old_count, new_count, new_contents \
|
||||
)
|
||||
|
||||
/// Insert one `element` into the array at the given `index`.
|
||||
#define array_insert(self, _index, element) \
|
||||
_array__splice((Array *)(self), array_elem_size(self), _index, 0, 1, &(element))
|
||||
|
||||
/// Remove one element from the array at the given `index`.
|
||||
#define array_erase(self, _index) \
|
||||
_array__erase((Array *)(self), array_elem_size(self), _index)
|
||||
|
||||
/// Pop the last element off the array, returning the element by value.
|
||||
#define array_pop(self) ((self)->contents[--(self)->size])
|
||||
|
||||
/// Assign the contents of one array to another, reallocating if necessary.
|
||||
#define array_assign(self, other) \
|
||||
_array__assign((Array *)(self), (const Array *)(other), array_elem_size(self))
|
||||
|
||||
/// Swap one array with another
|
||||
#define array_swap(self, other) \
|
||||
_array__swap((Array *)(self), (Array *)(other))
|
||||
|
||||
/// Get the size of the array contents
|
||||
#define array_elem_size(self) (sizeof *(self)->contents)
|
||||
|
||||
/// Search a sorted array for a given `needle` value, using the given `compare`
|
||||
/// callback to determine the order.
|
||||
///
|
||||
/// If an existing element is found to be equal to `needle`, then the `index`
|
||||
/// out-parameter is set to the existing value's index, and the `exists`
|
||||
/// out-parameter is set to true. Otherwise, `index` is set to an index where
|
||||
/// `needle` should be inserted in order to preserve the sorting, and `exists`
|
||||
/// is set to false.
|
||||
#define array_search_sorted_with(self, compare, needle, _index, _exists) \
|
||||
_array__search_sorted(self, 0, compare, , needle, _index, _exists)
|
||||
|
||||
/// Search a sorted array for a given `needle` value, using integer comparisons
|
||||
/// of a given struct field (specified with a leading dot) to determine the order.
|
||||
///
|
||||
/// See also `array_search_sorted_with`.
|
||||
#define array_search_sorted_by(self, field, needle, _index, _exists) \
|
||||
_array__search_sorted(self, 0, _compare_int, field, needle, _index, _exists)
|
||||
|
||||
/// Insert a given `value` into a sorted array, using the given `compare`
|
||||
/// callback to determine the order.
|
||||
#define array_insert_sorted_with(self, compare, value) \
|
||||
do { \
|
||||
unsigned _index, _exists; \
|
||||
array_search_sorted_with(self, compare, &(value), &_index, &_exists); \
|
||||
if (!_exists) array_insert(self, _index, value); \
|
||||
} while (0)
|
||||
|
||||
/// Insert a given `value` into a sorted array, using integer comparisons of
|
||||
/// a given struct field (specified with a leading dot) to determine the order.
|
||||
///
|
||||
/// See also `array_search_sorted_by`.
|
||||
#define array_insert_sorted_by(self, field, value) \
|
||||
do { \
|
||||
unsigned _index, _exists; \
|
||||
array_search_sorted_by(self, field, (value) field, &_index, &_exists); \
|
||||
if (!_exists) array_insert(self, _index, value); \
|
||||
} while (0)
|
||||
|
||||
// Private
|
||||
|
||||
typedef Array(void) Array;
|
||||
|
||||
/// This is not what you're looking for, see `array_delete`.
|
||||
static inline void _array__delete(Array *self) {
|
||||
if (self->contents) {
|
||||
ts_free(self->contents);
|
||||
self->contents = NULL;
|
||||
self->size = 0;
|
||||
self->capacity = 0;
|
||||
}
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_erase`.
|
||||
static inline void _array__erase(Array *self, size_t element_size,
|
||||
uint32_t index) {
|
||||
assert(index < self->size);
|
||||
char *contents = (char *)self->contents;
|
||||
memmove(contents + index * element_size, contents + (index + 1) * element_size,
|
||||
(self->size - index - 1) * element_size);
|
||||
self->size--;
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_reserve`.
|
||||
static inline void _array__reserve(Array *self, size_t element_size, uint32_t new_capacity) {
|
||||
if (new_capacity > self->capacity) {
|
||||
if (self->contents) {
|
||||
self->contents = ts_realloc(self->contents, new_capacity * element_size);
|
||||
} else {
|
||||
self->contents = ts_malloc(new_capacity * element_size);
|
||||
}
|
||||
self->capacity = new_capacity;
|
||||
}
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_assign`.
|
||||
static inline void _array__assign(Array *self, const Array *other, size_t element_size) {
|
||||
_array__reserve(self, element_size, other->size);
|
||||
self->size = other->size;
|
||||
memcpy(self->contents, other->contents, self->size * element_size);
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_swap`.
|
||||
static inline void _array__swap(Array *self, Array *other) {
|
||||
Array swap = *other;
|
||||
*other = *self;
|
||||
*self = swap;
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_push` or `array_grow_by`.
|
||||
static inline void _array__grow(Array *self, uint32_t count, size_t element_size) {
|
||||
uint32_t new_size = self->size + count;
|
||||
if (new_size > self->capacity) {
|
||||
uint32_t new_capacity = self->capacity * 2;
|
||||
if (new_capacity < 8) new_capacity = 8;
|
||||
if (new_capacity < new_size) new_capacity = new_size;
|
||||
_array__reserve(self, element_size, new_capacity);
|
||||
}
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_splice`.
|
||||
static inline void _array__splice(Array *self, size_t element_size,
|
||||
uint32_t index, uint32_t old_count,
|
||||
uint32_t new_count, const void *elements) {
|
||||
uint32_t new_size = self->size + new_count - old_count;
|
||||
uint32_t old_end = index + old_count;
|
||||
uint32_t new_end = index + new_count;
|
||||
assert(old_end <= self->size);
|
||||
|
||||
_array__reserve(self, element_size, new_size);
|
||||
|
||||
char *contents = (char *)self->contents;
|
||||
if (self->size > old_end) {
|
||||
memmove(
|
||||
contents + new_end * element_size,
|
||||
contents + old_end * element_size,
|
||||
(self->size - old_end) * element_size
|
||||
);
|
||||
}
|
||||
if (new_count > 0) {
|
||||
if (elements) {
|
||||
memcpy(
|
||||
(contents + index * element_size),
|
||||
elements,
|
||||
new_count * element_size
|
||||
);
|
||||
} else {
|
||||
memset(
|
||||
(contents + index * element_size),
|
||||
0,
|
||||
new_count * element_size
|
||||
);
|
||||
}
|
||||
}
|
||||
self->size += new_count - old_count;
|
||||
}
|
||||
|
||||
/// A binary search routine, based on Rust's `std::slice::binary_search_by`.
|
||||
/// This is not what you're looking for, see `array_search_sorted_with` or `array_search_sorted_by`.
|
||||
#define _array__search_sorted(self, start, compare, suffix, needle, _index, _exists) \
|
||||
do { \
|
||||
*(_index) = start; \
|
||||
*(_exists) = false; \
|
||||
uint32_t size = (self)->size - *(_index); \
|
||||
if (size == 0) break; \
|
||||
int comparison; \
|
||||
while (size > 1) { \
|
||||
uint32_t half_size = size / 2; \
|
||||
uint32_t mid_index = *(_index) + half_size; \
|
||||
comparison = compare(&((self)->contents[mid_index] suffix), (needle)); \
|
||||
if (comparison <= 0) *(_index) = mid_index; \
|
||||
size -= half_size; \
|
||||
} \
|
||||
comparison = compare(&((self)->contents[*(_index)] suffix), (needle)); \
|
||||
if (comparison == 0) *(_exists) = true; \
|
||||
else if (comparison < 0) *(_index) += 1; \
|
||||
} while (0)
|
||||
|
||||
/// Helper macro for the `_sorted_by` routines below. This takes the left (existing)
|
||||
/// parameter by reference in order to work with the generic sorting function above.
|
||||
#define _compare_int(a, b) ((int)*(a) - (int)(b))
|
||||
|
||||
#ifdef _MSC_VER
|
||||
#pragma warning(pop)
|
||||
#elif defined(__GNUC__) || defined(__clang__)
|
||||
#pragma GCC diagnostic pop
|
||||
#endif
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif // TREE_SITTER_ARRAY_H_
|
||||
@@ -0,0 +1,286 @@
|
||||
#ifndef TREE_SITTER_PARSER_H_
|
||||
#define TREE_SITTER_PARSER_H_
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
#include <stdbool.h>
|
||||
#include <stdint.h>
|
||||
#include <stdlib.h>
|
||||
|
||||
#define ts_builtin_sym_error ((TSSymbol)-1)
|
||||
#define ts_builtin_sym_end 0
|
||||
#define TREE_SITTER_SERIALIZATION_BUFFER_SIZE 1024
|
||||
|
||||
#ifndef TREE_SITTER_API_H_
|
||||
typedef uint16_t TSStateId;
|
||||
typedef uint16_t TSSymbol;
|
||||
typedef uint16_t TSFieldId;
|
||||
typedef struct TSLanguage TSLanguage;
|
||||
typedef struct TSLanguageMetadata {
|
||||
uint8_t major_version;
|
||||
uint8_t minor_version;
|
||||
uint8_t patch_version;
|
||||
} TSLanguageMetadata;
|
||||
#endif
|
||||
|
||||
typedef struct {
|
||||
TSFieldId field_id;
|
||||
uint8_t child_index;
|
||||
bool inherited;
|
||||
} TSFieldMapEntry;
|
||||
|
||||
// Used to index the field and supertype maps.
|
||||
typedef struct {
|
||||
uint16_t index;
|
||||
uint16_t length;
|
||||
} TSMapSlice;
|
||||
|
||||
typedef struct {
|
||||
bool visible;
|
||||
bool named;
|
||||
bool supertype;
|
||||
} TSSymbolMetadata;
|
||||
|
||||
typedef struct TSLexer TSLexer;
|
||||
|
||||
struct TSLexer {
|
||||
int32_t lookahead;
|
||||
TSSymbol result_symbol;
|
||||
void (*advance)(TSLexer *, bool);
|
||||
void (*mark_end)(TSLexer *);
|
||||
uint32_t (*get_column)(TSLexer *);
|
||||
bool (*is_at_included_range_start)(const TSLexer *);
|
||||
bool (*eof)(const TSLexer *);
|
||||
void (*log)(const TSLexer *, const char *, ...);
|
||||
};
|
||||
|
||||
typedef enum {
|
||||
TSParseActionTypeShift,
|
||||
TSParseActionTypeReduce,
|
||||
TSParseActionTypeAccept,
|
||||
TSParseActionTypeRecover,
|
||||
} TSParseActionType;
|
||||
|
||||
typedef union {
|
||||
struct {
|
||||
uint8_t type;
|
||||
TSStateId state;
|
||||
bool extra;
|
||||
bool repetition;
|
||||
} shift;
|
||||
struct {
|
||||
uint8_t type;
|
||||
uint8_t child_count;
|
||||
TSSymbol symbol;
|
||||
int16_t dynamic_precedence;
|
||||
uint16_t production_id;
|
||||
} reduce;
|
||||
uint8_t type;
|
||||
} TSParseAction;
|
||||
|
||||
typedef struct {
|
||||
uint16_t lex_state;
|
||||
uint16_t external_lex_state;
|
||||
} TSLexMode;
|
||||
|
||||
typedef struct {
|
||||
uint16_t lex_state;
|
||||
uint16_t external_lex_state;
|
||||
uint16_t reserved_word_set_id;
|
||||
} TSLexerMode;
|
||||
|
||||
typedef union {
|
||||
TSParseAction action;
|
||||
struct {
|
||||
uint8_t count;
|
||||
bool reusable;
|
||||
} entry;
|
||||
} TSParseActionEntry;
|
||||
|
||||
typedef struct {
|
||||
int32_t start;
|
||||
int32_t end;
|
||||
} TSCharacterRange;
|
||||
|
||||
struct TSLanguage {
|
||||
uint32_t abi_version;
|
||||
uint32_t symbol_count;
|
||||
uint32_t alias_count;
|
||||
uint32_t token_count;
|
||||
uint32_t external_token_count;
|
||||
uint32_t state_count;
|
||||
uint32_t large_state_count;
|
||||
uint32_t production_id_count;
|
||||
uint32_t field_count;
|
||||
uint16_t max_alias_sequence_length;
|
||||
const uint16_t *parse_table;
|
||||
const uint16_t *small_parse_table;
|
||||
const uint32_t *small_parse_table_map;
|
||||
const TSParseActionEntry *parse_actions;
|
||||
const char * const *symbol_names;
|
||||
const char * const *field_names;
|
||||
const TSMapSlice *field_map_slices;
|
||||
const TSFieldMapEntry *field_map_entries;
|
||||
const TSSymbolMetadata *symbol_metadata;
|
||||
const TSSymbol *public_symbol_map;
|
||||
const uint16_t *alias_map;
|
||||
const TSSymbol *alias_sequences;
|
||||
const TSLexerMode *lex_modes;
|
||||
bool (*lex_fn)(TSLexer *, TSStateId);
|
||||
bool (*keyword_lex_fn)(TSLexer *, TSStateId);
|
||||
TSSymbol keyword_capture_token;
|
||||
struct {
|
||||
const bool *states;
|
||||
const TSSymbol *symbol_map;
|
||||
void *(*create)(void);
|
||||
void (*destroy)(void *);
|
||||
bool (*scan)(void *, TSLexer *, const bool *symbol_whitelist);
|
||||
unsigned (*serialize)(void *, char *);
|
||||
void (*deserialize)(void *, const char *, unsigned);
|
||||
} external_scanner;
|
||||
const TSStateId *primary_state_ids;
|
||||
const char *name;
|
||||
const TSSymbol *reserved_words;
|
||||
uint16_t max_reserved_word_set_size;
|
||||
uint32_t supertype_count;
|
||||
const TSSymbol *supertype_symbols;
|
||||
const TSMapSlice *supertype_map_slices;
|
||||
const TSSymbol *supertype_map_entries;
|
||||
TSLanguageMetadata metadata;
|
||||
};
|
||||
|
||||
static inline bool set_contains(const TSCharacterRange *ranges, uint32_t len, int32_t lookahead) {
|
||||
uint32_t index = 0;
|
||||
uint32_t size = len - index;
|
||||
while (size > 1) {
|
||||
uint32_t half_size = size / 2;
|
||||
uint32_t mid_index = index + half_size;
|
||||
const TSCharacterRange *range = &ranges[mid_index];
|
||||
if (lookahead >= range->start && lookahead <= range->end) {
|
||||
return true;
|
||||
} else if (lookahead > range->end) {
|
||||
index = mid_index;
|
||||
}
|
||||
size -= half_size;
|
||||
}
|
||||
const TSCharacterRange *range = &ranges[index];
|
||||
return (lookahead >= range->start && lookahead <= range->end);
|
||||
}
|
||||
|
||||
/*
|
||||
* Lexer Macros
|
||||
*/
|
||||
|
||||
#ifdef _MSC_VER
|
||||
#define UNUSED __pragma(warning(suppress : 4101))
|
||||
#else
|
||||
#define UNUSED __attribute__((unused))
|
||||
#endif
|
||||
|
||||
#define START_LEXER() \
|
||||
bool result = false; \
|
||||
bool skip = false; \
|
||||
UNUSED \
|
||||
bool eof = false; \
|
||||
int32_t lookahead; \
|
||||
goto start; \
|
||||
next_state: \
|
||||
lexer->advance(lexer, skip); \
|
||||
start: \
|
||||
skip = false; \
|
||||
lookahead = lexer->lookahead;
|
||||
|
||||
#define ADVANCE(state_value) \
|
||||
{ \
|
||||
state = state_value; \
|
||||
goto next_state; \
|
||||
}
|
||||
|
||||
#define ADVANCE_MAP(...) \
|
||||
{ \
|
||||
static const uint16_t map[] = { __VA_ARGS__ }; \
|
||||
for (uint32_t i = 0; i < sizeof(map) / sizeof(map[0]); i += 2) { \
|
||||
if (map[i] == lookahead) { \
|
||||
state = map[i + 1]; \
|
||||
goto next_state; \
|
||||
} \
|
||||
} \
|
||||
}
|
||||
|
||||
#define SKIP(state_value) \
|
||||
{ \
|
||||
skip = true; \
|
||||
state = state_value; \
|
||||
goto next_state; \
|
||||
}
|
||||
|
||||
#define ACCEPT_TOKEN(symbol_value) \
|
||||
result = true; \
|
||||
lexer->result_symbol = symbol_value; \
|
||||
lexer->mark_end(lexer);
|
||||
|
||||
#define END_STATE() return result;
|
||||
|
||||
/*
|
||||
* Parse Table Macros
|
||||
*/
|
||||
|
||||
#define SMALL_STATE(id) ((id) - LARGE_STATE_COUNT)
|
||||
|
||||
#define STATE(id) id
|
||||
|
||||
#define ACTIONS(id) id
|
||||
|
||||
#define SHIFT(state_value) \
|
||||
{{ \
|
||||
.shift = { \
|
||||
.type = TSParseActionTypeShift, \
|
||||
.state = (state_value) \
|
||||
} \
|
||||
}}
|
||||
|
||||
#define SHIFT_REPEAT(state_value) \
|
||||
{{ \
|
||||
.shift = { \
|
||||
.type = TSParseActionTypeShift, \
|
||||
.state = (state_value), \
|
||||
.repetition = true \
|
||||
} \
|
||||
}}
|
||||
|
||||
#define SHIFT_EXTRA() \
|
||||
{{ \
|
||||
.shift = { \
|
||||
.type = TSParseActionTypeShift, \
|
||||
.extra = true \
|
||||
} \
|
||||
}}
|
||||
|
||||
#define REDUCE(symbol_name, children, precedence, prod_id) \
|
||||
{{ \
|
||||
.reduce = { \
|
||||
.type = TSParseActionTypeReduce, \
|
||||
.symbol = symbol_name, \
|
||||
.child_count = children, \
|
||||
.dynamic_precedence = precedence, \
|
||||
.production_id = prod_id \
|
||||
}, \
|
||||
}}
|
||||
|
||||
#define RECOVER() \
|
||||
{{ \
|
||||
.type = TSParseActionTypeRecover \
|
||||
}}
|
||||
|
||||
#define ACCEPT_INPUT() \
|
||||
{{ \
|
||||
.type = TSParseActionTypeAccept \
|
||||
}}
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif // TREE_SITTER_PARSER_H_
|
||||
@@ -18,7 +18,7 @@ release-owned runtime assets. Chocolatey retains the complete extracted runtime
|
||||
set and exposes its executable through a PATH shim.
|
||||
|
||||
Full indexes an average repository in milliseconds, the Linux kernel (28M LOC) in
|
||||
3 minutes. Answers structural queries in under 1ms. Supports 162 languages via
|
||||
3 minutes. Answers structural queries in under 1ms. Supports 163 languages via
|
||||
vendored tree-sitter grammars. No hosted service or API key. Plug and play across
|
||||
11 coding agents including Claude Code, Codex CLI, Gemini CLI, and Zed.
|
||||
|
||||
|
||||
@@ -44,6 +44,11 @@ FORKS = { # self-maintained forks: vendored LICENSE must match the original ups
|
||||
"pascal": "Isopod/tree-sitter-pascal",
|
||||
"qml": "yuja/tree-sitter-qmljs",
|
||||
"tsx": "tree-sitter/tree-sitter-typescript",
|
||||
# vbnet: tree-sitter-vbnet + CodeGraph's real-world VB.NET patch; same
|
||||
# routing rule as arkts -- `LICENSE` is upstream's MIT verbatim and is the
|
||||
# file byte-verified here (local_license() takes it before the sibling
|
||||
# LICENSE-codegraph, whose provenance MANIFEST.md records by SHA-256).
|
||||
"vbnet": "govindbanura/tree-sitter-vbnet",
|
||||
}
|
||||
SPECIAL_NOTICE = {
|
||||
"assembly": "RETAINED-MIT (upstream RubixDev/tree-sitter-assembly deleted from GitHub)",
|
||||
|
||||
@@ -5,7 +5,7 @@ c5cfb43042b6b72045f4ba997834d0a7786d2793d91680868b5815b39f14fc78 internal/cbm/v
|
||||
b29c1c9fb7cc82f58c84b376df1297d6e2737a1d655fd356db0859e3c29c2fea internal/cbm/vendored/common/tree_sitter/alloc.h
|
||||
31e60a1bff6f715afacce03b5b70efe42b58371b4f9595dd4af52a577ff9608c internal/cbm/vendored/common/tree_sitter/array.h
|
||||
180b893c8734778fd32f372dfbc27bd6ad1cd2221f26150b31256ff6716320d2 internal/cbm/vendored/common/tree_sitter/parser.h
|
||||
38172147795b4729640a3d9ef61ac94719e68ab5e3f9aaca792aa1c8c36d2940 internal/cbm/vendored/grammars/MANIFEST.md
|
||||
b7fd26f1cb5e720f78ec6e4ab1160b044a4568650fd05d744ba05b230cc182a6 internal/cbm/vendored/grammars/MANIFEST.md
|
||||
ad8425038de519f8c4e3e9339feebf99dfad8a6002dcf79227d91402780a32cc internal/cbm/vendored/grammars/ada/LICENSE
|
||||
02805ec13939b749c891567be36bf024b09034e04c80683a1ae667272458d549 internal/cbm/vendored/grammars/ada/parser.c
|
||||
115a75d000bef9c70c4de7dfbb7f2a80b90cbb14ba265056dc7abb1ef3b9b6db internal/cbm/vendored/grammars/ada/tree_sitter/alloc.h
|
||||
@@ -862,6 +862,13 @@ eeb1ce8085d748ea8ec44b337b179a676dfab4549593b834e9c510362e795b64 internal/cbm/v
|
||||
02d139d44303b0b0b85a899839dbc1346c478e7318d1f2971fad2b38fd57cb73 internal/cbm/vendored/grammars/typst/tree_sitter/array.h
|
||||
2db740ea1be6b71014d2be1385491f01bf2d15b607e61c5b30b4128535efe68a internal/cbm/vendored/grammars/typst/tree_sitter/parser.h
|
||||
6d1a6e017b78433535460ceb3e386d92d8c5dca6e85e6919d5810f1f9aa665a1 internal/cbm/vendored/grammars/typst/unicode.h
|
||||
fa7ab39bcf92c216d429e8aa8be8a5a40db85bfbe4ff14a7bb0ecbbcca937454 internal/cbm/vendored/grammars/vbnet/LICENSE
|
||||
e6d98f98c666bebe065ac2492a0a19232cc318d4d67bac3ca42ffb77bacc8809 internal/cbm/vendored/grammars/vbnet/LICENSE-codegraph
|
||||
d0c29e9a77416fb24e60c8e7f832d60b0cbbf7f40e375f179b119790b0ff0d78 internal/cbm/vendored/grammars/vbnet/parser.c
|
||||
51a81d6616e6499bdce1ee58aacd7798b34623a098a3d7d5a89bd1d61c7842a5 internal/cbm/vendored/grammars/vbnet/scanner.c
|
||||
b29c1c9fb7cc82f58c84b376df1297d6e2737a1d655fd356db0859e3c29c2fea internal/cbm/vendored/grammars/vbnet/tree_sitter/alloc.h
|
||||
5bdf6ed1a78e3409fd443e085ca967a64c188a5d082aaf7f819bccd53a471c94 internal/cbm/vendored/grammars/vbnet/tree_sitter/array.h
|
||||
180b893c8734778fd32f372dfbc27bd6ad1cd2221f26150b31256ff6716320d2 internal/cbm/vendored/grammars/vbnet/tree_sitter/parser.h
|
||||
8d2ff8d1e308828099436c43dc8bfbb8b6f368ad2a012ab9802a80ca22b03b53 internal/cbm/vendored/grammars/verilog/LICENSE
|
||||
bb0f5fe2423ac18a4ab15d8201a2cf155c874b52f3a3022a16e2aa58bb92a4d9 internal/cbm/vendored/grammars/verilog/parser.c
|
||||
b29c1c9fb7cc82f58c84b376df1297d6e2737a1d655fd356db0859e3c29c2fea internal/cbm/vendored/grammars/verilog/tree_sitter/alloc.h
|
||||
|
||||
@@ -2037,6 +2037,147 @@ TEST(pascal_broken_source_still_flagged_issue2211) {
|
||||
PASS();
|
||||
}
|
||||
|
||||
/* ── VB.NET: the vendored grammar parses real VB.NET (#1178, #1179) ────────
|
||||
* tools/tree-sitter-vbnet (tree-sitter-vbnet + CodeGraph's real-world patch)
|
||||
* is vendored ahead of VB.NET extraction: no CBMLanguage routes to it yet, so
|
||||
* these cases parse through tree_sitter_vbnet() directly. The sample is the
|
||||
* shape extraction must index -- Imports, a WinForms Designer partial class,
|
||||
* Handles, Namespace, Inherits/Implements, MustOverride, #Region, Sub New,
|
||||
* Property, Function -- plus names that begin with a keyword. It is RED on the
|
||||
* alternatives: unpatched upstream leaves ERROR nodes at the Designer's Single
|
||||
* literals (8.25!, 6.0!) and at MustOverride, and the patch revision before
|
||||
* 2026-10-06 turns `Sub NewItem()` into an ERROR and `Public SharedCache` into
|
||||
* a Shared field named Cache. */
|
||||
extern const TSLanguage *tree_sitter_vbnet(void);
|
||||
|
||||
static const char VB_REAL_SOURCE[] =
|
||||
"Imports System\n"
|
||||
"Imports System.Collections.Generic\n"
|
||||
"Imports System.Windows.Forms\n"
|
||||
"\n"
|
||||
"<Global.Microsoft.VisualBasic.CompilerServices.DesignerGenerated()> _\n"
|
||||
"Partial Class MainForm\n"
|
||||
" Inherits System.Windows.Forms.Form\n"
|
||||
"\n"
|
||||
" <System.Diagnostics.DebuggerStepThrough()> _\n"
|
||||
" Private Sub InitializeComponent()\n"
|
||||
" Me.btnSave = New System.Windows.Forms.Button()\n"
|
||||
" Me.SuspendLayout()\n"
|
||||
" Me.btnSave.Font = New System.Drawing.Font(\"Segoe UI\", 8.25!)\n"
|
||||
" Me.btnSave.Text = \"Save\"\n"
|
||||
" Me.AutoScaleDimensions = New System.Drawing.SizeF(6.0!, 13.0!)\n"
|
||||
" Me.Controls.Add(Me.btnSave)\n"
|
||||
" Me.ResumeLayout(False)\n"
|
||||
" End Sub\n"
|
||||
"\n"
|
||||
" Friend WithEvents btnSave As System.Windows.Forms.Button\n"
|
||||
"End Class\n"
|
||||
"\n"
|
||||
"Partial Class MainForm\n"
|
||||
" Private Sub btnSave_Click(sender As Object, e As EventArgs) Handles btnSave.Click\n"
|
||||
" MessageBox.Show(\"Saved\")\n"
|
||||
" End Sub\n"
|
||||
"End Class\n"
|
||||
"\n"
|
||||
"Namespace Contoso.Billing\n"
|
||||
" Public MustInherit Class EntityBase\n"
|
||||
" Public MustOverride Function Describe() As String\n"
|
||||
" End Class\n"
|
||||
"\n"
|
||||
" Public Class Customer\n"
|
||||
" Inherits EntityBase\n"
|
||||
" Implements IComparable(Of Customer), IDisposable\n"
|
||||
"\n"
|
||||
"#Region \"Fields\"\n"
|
||||
" Private _name As String\n"
|
||||
" Public SharedCache As Integer\n"
|
||||
" Public Dimension As Integer\n"
|
||||
"#End Region\n"
|
||||
"\n"
|
||||
" Public Sub New(name As String)\n"
|
||||
" _name = name\n"
|
||||
" End Sub\n"
|
||||
"\n"
|
||||
" Public Sub NewItem()\n"
|
||||
" Dimension = 0\n"
|
||||
" End Sub\n"
|
||||
"\n"
|
||||
" Public Property Name As String\n"
|
||||
" Get\n"
|
||||
" Return _name\n"
|
||||
" End Get\n"
|
||||
" Set(value As String)\n"
|
||||
" _name = value\n"
|
||||
" End Set\n"
|
||||
" End Property\n"
|
||||
"\n"
|
||||
" Public Overrides Function Describe() As String\n"
|
||||
" Return _name\n"
|
||||
" End Function\n"
|
||||
"\n"
|
||||
" Public Sub Dispose() Implements IDisposable.Dispose\n"
|
||||
" _name = Nothing\n"
|
||||
" End Sub\n"
|
||||
"\n"
|
||||
" Public Function CompareTo(o As Customer) As Integer _\n"
|
||||
" Implements IComparable(Of Customer).CompareTo\n"
|
||||
" Return String.Compare(_name, o.Name)\n"
|
||||
" End Function\n"
|
||||
" End Class\n"
|
||||
"End Namespace\n";
|
||||
|
||||
/* Counts the `identifier` nodes under `node` whose text is exactly `text`. */
|
||||
static int vb_identifier_count(TSNode node, const char *src, const char *text) {
|
||||
int hits = 0;
|
||||
if (strcmp(ts_node_type(node), "identifier") == 0) {
|
||||
uint32_t start = ts_node_start_byte(node);
|
||||
uint32_t len = ts_node_end_byte(node) - start;
|
||||
if (strlen(text) == len && memcmp(src + start, text, len) == 0) {
|
||||
hits++;
|
||||
}
|
||||
}
|
||||
uint32_t count = ts_node_child_count(node);
|
||||
for (uint32_t i = 0; i < count; i++) {
|
||||
hits += vb_identifier_count(ts_node_child(node, i), src, text);
|
||||
}
|
||||
return hits;
|
||||
}
|
||||
|
||||
TEST(vbnet_vendored_grammar_parses_real_vb_clean) {
|
||||
TSParser *parser = ts_parser_new();
|
||||
ASSERT_NOT_NULL(parser);
|
||||
/* The vendored runtime accepts the generated ABI (15). */
|
||||
ASSERT_TRUE(ts_parser_set_language(parser, tree_sitter_vbnet()));
|
||||
TSTree *tree =
|
||||
ts_parser_parse_string(parser, NULL, VB_REAL_SOURCE, (uint32_t)strlen(VB_REAL_SOURCE));
|
||||
ASSERT_NOT_NULL(tree);
|
||||
TSNode root = ts_tree_root_node(tree);
|
||||
ASSERT_STR_EQ(ts_node_type(root), "source_file");
|
||||
ASSERT_FALSE(ts_node_has_error(root));
|
||||
/* Names that begin with a keyword stay whole identifiers. */
|
||||
ASSERT_GTE(vb_identifier_count(root, VB_REAL_SOURCE, "SharedCache"), 1);
|
||||
ASSERT_GTE(vb_identifier_count(root, VB_REAL_SOURCE, "Dimension"), 1);
|
||||
ASSERT_GTE(vb_identifier_count(root, VB_REAL_SOURCE, "NewItem"), 1);
|
||||
ASSERT_EQ(vb_identifier_count(root, VB_REAL_SOURCE, "Cache"), 0);
|
||||
ASSERT_EQ(vb_identifier_count(root, VB_REAL_SOURCE, "ension"), 0);
|
||||
ts_tree_delete(tree);
|
||||
ts_parser_delete(parser);
|
||||
PASS();
|
||||
}
|
||||
|
||||
TEST(vbnet_vendored_grammar_still_flags_broken_source) {
|
||||
static const char broken[] = "Public Class Broken\n Public Sub (\nEnd Class\n";
|
||||
TSParser *parser = ts_parser_new();
|
||||
ASSERT_NOT_NULL(parser);
|
||||
ASSERT_TRUE(ts_parser_set_language(parser, tree_sitter_vbnet()));
|
||||
TSTree *tree = ts_parser_parse_string(parser, NULL, broken, (uint32_t)strlen(broken));
|
||||
ASSERT_NOT_NULL(tree);
|
||||
ASSERT_TRUE(ts_node_has_error(ts_tree_root_node(tree)));
|
||||
ts_tree_delete(tree);
|
||||
ts_parser_delete(parser);
|
||||
PASS();
|
||||
}
|
||||
|
||||
SUITE(parse_coverage) {
|
||||
RUN_TEST(source_line_count_one_convention);
|
||||
RUN_TEST(c_ifdef_split_brace_sets_parse_incomplete);
|
||||
@@ -2100,4 +2241,6 @@ SUITE(parse_coverage) {
|
||||
RUN_TEST(pascal_cyrillic_procedure_name_extracted_issue2211);
|
||||
RUN_TEST(pascal_ascii_control_still_clean_issue2211);
|
||||
RUN_TEST(pascal_broken_source_still_flagged_issue2211);
|
||||
RUN_TEST(vbnet_vendored_grammar_parses_real_vb_clean);
|
||||
RUN_TEST(vbnet_vendored_grammar_still_flags_broken_source);
|
||||
}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"name": "tree-sitter-vbnet-cbm",
|
||||
"version": "0.1.0-cbm.1",
|
||||
"description": "cbm fork of tree-sitter-vbnet @ 538b708 with CodeGraph's real-world VB.NET patch",
|
||||
"license": "MIT",
|
||||
"devDependencies": {
|
||||
"tree-sitter-cli": "0.25.10"
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,303 @@
|
||||
#include "tree_sitter/parser.h"
|
||||
#include <string.h>
|
||||
#include <wctype.h>
|
||||
|
||||
// External tokens for the two constructs that are not LR(1)-parseable with
|
||||
// tree-sitter's newline-as-extra treatment:
|
||||
//
|
||||
// 1. QUERY_CLAUSE_CONTINUATION — a newline run that CONTINUES a LINQ query
|
||||
// (`From x In xs` ↵ `Where …`). At a clause boundary the parser cannot
|
||||
// know from the newline alone whether the query ends (newline = statement
|
||||
// terminator) or continues (newline = insignificant, next line starts with
|
||||
// a query keyword). The scanner looks past the newline at the next word
|
||||
// and only emits the token when it is a query-clause keyword — so the
|
||||
// terminator/continuation choice is made by the lexer, not the LR table.
|
||||
//
|
||||
// 2. XML_LITERAL — a whole VB XML literal (`<Tags><Tag/></Tags>`, including
|
||||
// `<%= expr %>` embedded expressions, comments, CDATA) consumed as one
|
||||
// opaque token. Only valid where a literal can begin an expression, so a
|
||||
// relational `<` (which always FOLLOWS an expression) can never collide.
|
||||
|
||||
enum TokenType {
|
||||
QUERY_CLAUSE_CONTINUATION,
|
||||
XML_LITERAL,
|
||||
};
|
||||
|
||||
void *tree_sitter_vbnet_external_scanner_create(void) { return NULL; }
|
||||
void tree_sitter_vbnet_external_scanner_destroy(void *payload) { (void)payload; }
|
||||
unsigned tree_sitter_vbnet_external_scanner_serialize(void *payload, char *buffer) {
|
||||
(void)payload; (void)buffer;
|
||||
return 0; // stateless
|
||||
}
|
||||
void tree_sitter_vbnet_external_scanner_deserialize(void *payload, const char *buffer, unsigned length) {
|
||||
(void)payload; (void)buffer; (void)length;
|
||||
}
|
||||
|
||||
static inline void advance(TSLexer *lexer) { lexer->advance(lexer, false); }
|
||||
static inline void skip_ws(TSLexer *lexer) { lexer->advance(lexer, true); }
|
||||
|
||||
// Read one alphabetic word (advancing past it), lowercased into buf.
|
||||
// Returns its length (0 if the next char is not a letter).
|
||||
static int read_word(TSLexer *lexer, char *buf, int cap) {
|
||||
int n = 0;
|
||||
while (iswalpha((wint_t)lexer->lookahead)) {
|
||||
if (n < cap - 1) buf[n] = (char)towlower((wint_t)lexer->lookahead);
|
||||
n++;
|
||||
advance(lexer);
|
||||
}
|
||||
buf[n < cap - 1 ? n : cap - 1] = 0;
|
||||
return n;
|
||||
}
|
||||
|
||||
static bool is_query_keyword(const char *w) {
|
||||
static const char *kws[] = {
|
||||
"where", "select", "order", "group", "join", "let", "skip",
|
||||
"take", "distinct", "aggregate", "from", "into", "on", NULL,
|
||||
};
|
||||
for (int i = 0; kws[i]; i++)
|
||||
if (strcmp(w, kws[i]) == 0) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
// [ \t\r]* '\n' [ \t\r\n]* followed by a query-clause keyword.
|
||||
// The token covers only the whitespace/newline run (mark_end before the
|
||||
// keyword lookahead), so the clause keyword itself lexes normally after it.
|
||||
static bool scan_query_continuation(TSLexer *lexer) {
|
||||
bool saw_newline = false;
|
||||
for (;;) {
|
||||
int32_t c = lexer->lookahead;
|
||||
if (c == ' ' || c == '\t' || c == '\r') {
|
||||
advance(lexer);
|
||||
} else if (c == '\n') {
|
||||
saw_newline = true;
|
||||
advance(lexer);
|
||||
} else {
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (!saw_newline) return false;
|
||||
lexer->mark_end(lexer);
|
||||
|
||||
char w[12];
|
||||
int n = read_word(lexer, w, sizeof w);
|
||||
if (n == 0 || n >= (int)(sizeof w)) return false;
|
||||
if (!is_query_keyword(w)) return false;
|
||||
|
||||
// `Select Case` opens a statement, never a select clause.
|
||||
if (strcmp(w, "select") == 0) {
|
||||
while (lexer->lookahead == ' ' || lexer->lookahead == '\t') advance(lexer);
|
||||
char w2[8];
|
||||
int m = read_word(lexer, w2, sizeof w2);
|
||||
if (m > 0 && m < (int)(sizeof w2) && strcmp(w2, "case") == 0) return false;
|
||||
}
|
||||
|
||||
lexer->result_symbol = QUERY_CLAUSE_CONTINUATION;
|
||||
return true;
|
||||
}
|
||||
|
||||
// Consume a VB string inside an embedded `<%= … %>` region ("" escapes).
|
||||
static void consume_vb_string(TSLexer *lexer) {
|
||||
advance(lexer); // opening quote
|
||||
for (;;) {
|
||||
if (lexer->lookahead == 0) return;
|
||||
if (lexer->lookahead == '"') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead != '"') return; // closing (not an escaped "")
|
||||
// escaped quote: fall through, keep consuming
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
|
||||
// Consume `<%= … %>` starting at the '%' (the '<' is already consumed).
|
||||
// Embedded expressions NEST (`<%= From t In ts Select <Tag><%= t.Name %></Tag> %>`,
|
||||
// the staxrip WriteTagfile shape), so track <% / %> depth.
|
||||
static bool consume_embedded_expression(TSLexer *lexer) {
|
||||
advance(lexer); // '%'
|
||||
int depth = 1;
|
||||
for (;;) {
|
||||
int32_t c = lexer->lookahead;
|
||||
if (c == 0) return false;
|
||||
if (c == '"') { consume_vb_string(lexer); continue; }
|
||||
if (c == '<') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == '%') { depth++; advance(lexer); }
|
||||
continue;
|
||||
}
|
||||
if (c == '%') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == '>') {
|
||||
advance(lexer);
|
||||
if (--depth == 0) return true;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
|
||||
// Consume `<!-- … -->` / `<![CDATA[ … ]]>` / `<!DOCTYPE …>` starting at '!'.
|
||||
static bool consume_bang_construct(TSLexer *lexer) {
|
||||
advance(lexer); // '!'
|
||||
if (lexer->lookahead == '-') {
|
||||
// comment: to -->
|
||||
int dashes = 0;
|
||||
for (;;) {
|
||||
int32_t c = lexer->lookahead;
|
||||
if (c == 0) return false;
|
||||
if (c == '-') { dashes++; advance(lexer); continue; }
|
||||
if (c == '>' && dashes >= 2) { advance(lexer); return true; }
|
||||
dashes = 0;
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
if (lexer->lookahead == '[') {
|
||||
// CDATA: to ]]>
|
||||
int brackets = 0;
|
||||
for (;;) {
|
||||
int32_t c = lexer->lookahead;
|
||||
if (c == 0) return false;
|
||||
if (c == ']') { brackets++; advance(lexer); continue; }
|
||||
if (c == '>' && brackets >= 2) { advance(lexer); return true; }
|
||||
brackets = 0;
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
// DOCTYPE-ish: to bare >
|
||||
for (;;) {
|
||||
int32_t c = lexer->lookahead;
|
||||
if (c == 0) return false;
|
||||
if (c == '>') { advance(lexer); return true; }
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
|
||||
// Consume `<? … ?>` starting at the '?'.
|
||||
static bool consume_processing_instruction(TSLexer *lexer) {
|
||||
advance(lexer); // '?'
|
||||
for (;;) {
|
||||
int32_t c = lexer->lookahead;
|
||||
if (c == 0) return false;
|
||||
if (c == '?') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == '>') { advance(lexer); return true; }
|
||||
continue;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
|
||||
static bool is_name_start(int32_t c) {
|
||||
return iswalpha((wint_t)c) || c == '_';
|
||||
}
|
||||
|
||||
static bool scan_xml_literal(TSLexer *lexer) {
|
||||
// Only same-line blanks are skipped: a leading newline must stay unconsumed
|
||||
// so it can serve as a statement/block terminator (a successful scan would
|
||||
// otherwise swallow it as trivia). XML on a continuation line still works:
|
||||
// the newline is consumed as an extra first, then the scanner re-runs.
|
||||
while (lexer->lookahead == ' ' || lexer->lookahead == '\t') skip_ws(lexer);
|
||||
if (lexer->lookahead != '<') return false;
|
||||
advance(lexer);
|
||||
// Only element-start markup opens an XML literal here; comments/PIs as the
|
||||
// outermost construct (`Dim d = <?xml …`) are rare enough to leave alone,
|
||||
// and a bare `<` (comparison) never begins an expression.
|
||||
if (!is_name_start(lexer->lookahead)) return false;
|
||||
|
||||
int depth = 0; // open (unclosed) elements
|
||||
bool in_tag = true; // inside <...> of an element tag
|
||||
bool closing = false;
|
||||
int32_t quote = 0;
|
||||
|
||||
for (;;) {
|
||||
int32_t c = lexer->lookahead;
|
||||
if (c == 0) return false; // unterminated: let the normal parser error
|
||||
|
||||
if (in_tag) {
|
||||
if (quote) {
|
||||
if (c == quote) quote = 0;
|
||||
advance(lexer);
|
||||
continue;
|
||||
}
|
||||
if (c == '"' || c == '\'') { quote = c; advance(lexer); continue; }
|
||||
if (c == '<') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == '%') {
|
||||
// attribute value embedded expression: attr=<%= x %>
|
||||
if (!consume_embedded_expression(lexer)) return false;
|
||||
continue;
|
||||
}
|
||||
continue; // stray < in a tag: malformed, keep consuming
|
||||
}
|
||||
if (c == '/') {
|
||||
advance(lexer);
|
||||
if (lexer->lookahead == '>') {
|
||||
// self-closing tag
|
||||
advance(lexer);
|
||||
in_tag = false;
|
||||
if (depth == 0) break; // single self-closed root: done
|
||||
continue;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (c == '>') {
|
||||
advance(lexer);
|
||||
if (closing) {
|
||||
depth--;
|
||||
if (depth <= 0) break; // root element closed: done
|
||||
} else {
|
||||
depth++;
|
||||
}
|
||||
in_tag = false;
|
||||
continue;
|
||||
}
|
||||
advance(lexer);
|
||||
continue;
|
||||
}
|
||||
|
||||
// text content between tags
|
||||
if (c == '<') {
|
||||
advance(lexer);
|
||||
int32_t d = lexer->lookahead;
|
||||
if (d == '/') { advance(lexer); closing = true; in_tag = true; continue; }
|
||||
if (d == '%') {
|
||||
if (!consume_embedded_expression(lexer)) return false;
|
||||
continue;
|
||||
}
|
||||
if (d == '!') {
|
||||
if (!consume_bang_construct(lexer)) return false;
|
||||
continue;
|
||||
}
|
||||
if (d == '?') {
|
||||
if (!consume_processing_instruction(lexer)) return false;
|
||||
continue;
|
||||
}
|
||||
closing = false;
|
||||
in_tag = true; // opening tag
|
||||
continue;
|
||||
}
|
||||
advance(lexer);
|
||||
}
|
||||
|
||||
lexer->mark_end(lexer);
|
||||
lexer->result_symbol = XML_LITERAL;
|
||||
return true;
|
||||
}
|
||||
|
||||
bool tree_sitter_vbnet_external_scanner_scan(void *payload, TSLexer *lexer, const bool *valid_symbols) {
|
||||
(void)payload;
|
||||
int32_t c = lexer->lookahead;
|
||||
bool ws = (c == ' ' || c == '\t' || c == '\n' || c == '\r');
|
||||
|
||||
// A scan attempt advances the lexer, so exactly ONE attempt runs per
|
||||
// invocation — never chain a second attempt after a failed first (its
|
||||
// reads would start mid-stream). The leading character picks the branch;
|
||||
// a false return discards the attempt entirely.
|
||||
if (valid_symbols[QUERY_CLAUSE_CONTINUATION] && ws) {
|
||||
return scan_query_continuation(lexer);
|
||||
}
|
||||
if (valid_symbols[XML_LITERAL] && (c == '<' || ws)) {
|
||||
return scan_xml_literal(lexer);
|
||||
}
|
||||
return false;
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
#ifndef TREE_SITTER_ALLOC_H_
|
||||
#define TREE_SITTER_ALLOC_H_
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
#include <stdbool.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
|
||||
// Allow clients to override allocation functions
|
||||
#ifdef TREE_SITTER_REUSE_ALLOCATOR
|
||||
|
||||
extern void *(*ts_current_malloc)(size_t size);
|
||||
extern void *(*ts_current_calloc)(size_t count, size_t size);
|
||||
extern void *(*ts_current_realloc)(void *ptr, size_t size);
|
||||
extern void (*ts_current_free)(void *ptr);
|
||||
|
||||
#ifndef ts_malloc
|
||||
#define ts_malloc ts_current_malloc
|
||||
#endif
|
||||
#ifndef ts_calloc
|
||||
#define ts_calloc ts_current_calloc
|
||||
#endif
|
||||
#ifndef ts_realloc
|
||||
#define ts_realloc ts_current_realloc
|
||||
#endif
|
||||
#ifndef ts_free
|
||||
#define ts_free ts_current_free
|
||||
#endif
|
||||
|
||||
#else
|
||||
|
||||
#ifndef ts_malloc
|
||||
#define ts_malloc malloc
|
||||
#endif
|
||||
#ifndef ts_calloc
|
||||
#define ts_calloc calloc
|
||||
#endif
|
||||
#ifndef ts_realloc
|
||||
#define ts_realloc realloc
|
||||
#endif
|
||||
#ifndef ts_free
|
||||
#define ts_free free
|
||||
#endif
|
||||
|
||||
#endif
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif // TREE_SITTER_ALLOC_H_
|
||||
@@ -0,0 +1,291 @@
|
||||
#ifndef TREE_SITTER_ARRAY_H_
|
||||
#define TREE_SITTER_ARRAY_H_
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
#include "./alloc.h"
|
||||
|
||||
#include <assert.h>
|
||||
#include <stdbool.h>
|
||||
#include <stdint.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
|
||||
#ifdef _MSC_VER
|
||||
#pragma warning(push)
|
||||
#pragma warning(disable : 4101)
|
||||
#elif defined(__GNUC__) || defined(__clang__)
|
||||
#pragma GCC diagnostic push
|
||||
#pragma GCC diagnostic ignored "-Wunused-variable"
|
||||
#endif
|
||||
|
||||
#define Array(T) \
|
||||
struct { \
|
||||
T *contents; \
|
||||
uint32_t size; \
|
||||
uint32_t capacity; \
|
||||
}
|
||||
|
||||
/// Initialize an array.
|
||||
#define array_init(self) \
|
||||
((self)->size = 0, (self)->capacity = 0, (self)->contents = NULL)
|
||||
|
||||
/// Create an empty array.
|
||||
#define array_new() \
|
||||
{ NULL, 0, 0 }
|
||||
|
||||
/// Get a pointer to the element at a given `index` in the array.
|
||||
#define array_get(self, _index) \
|
||||
(assert((uint32_t)(_index) < (self)->size), &(self)->contents[_index])
|
||||
|
||||
/// Get a pointer to the first element in the array.
|
||||
#define array_front(self) array_get(self, 0)
|
||||
|
||||
/// Get a pointer to the last element in the array.
|
||||
#define array_back(self) array_get(self, (self)->size - 1)
|
||||
|
||||
/// Clear the array, setting its size to zero. Note that this does not free any
|
||||
/// memory allocated for the array's contents.
|
||||
#define array_clear(self) ((self)->size = 0)
|
||||
|
||||
/// Reserve `new_capacity` elements of space in the array. If `new_capacity` is
|
||||
/// less than the array's current capacity, this function has no effect.
|
||||
#define array_reserve(self, new_capacity) \
|
||||
_array__reserve((Array *)(self), array_elem_size(self), new_capacity)
|
||||
|
||||
/// Free any memory allocated for this array. Note that this does not free any
|
||||
/// memory allocated for the array's contents.
|
||||
#define array_delete(self) _array__delete((Array *)(self))
|
||||
|
||||
/// Push a new `element` onto the end of the array.
|
||||
#define array_push(self, element) \
|
||||
(_array__grow((Array *)(self), 1, array_elem_size(self)), \
|
||||
(self)->contents[(self)->size++] = (element))
|
||||
|
||||
/// Increase the array's size by `count` elements.
|
||||
/// New elements are zero-initialized.
|
||||
#define array_grow_by(self, count) \
|
||||
do { \
|
||||
if ((count) == 0) break; \
|
||||
_array__grow((Array *)(self), count, array_elem_size(self)); \
|
||||
memset((self)->contents + (self)->size, 0, (count) * array_elem_size(self)); \
|
||||
(self)->size += (count); \
|
||||
} while (0)
|
||||
|
||||
/// Append all elements from one array to the end of another.
|
||||
#define array_push_all(self, other) \
|
||||
array_extend((self), (other)->size, (other)->contents)
|
||||
|
||||
/// Append `count` elements to the end of the array, reading their values from the
|
||||
/// `contents` pointer.
|
||||
#define array_extend(self, count, contents) \
|
||||
_array__splice( \
|
||||
(Array *)(self), array_elem_size(self), (self)->size, \
|
||||
0, count, contents \
|
||||
)
|
||||
|
||||
/// Remove `old_count` elements from the array starting at the given `index`. At
|
||||
/// the same index, insert `new_count` new elements, reading their values from the
|
||||
/// `new_contents` pointer.
|
||||
#define array_splice(self, _index, old_count, new_count, new_contents) \
|
||||
_array__splice( \
|
||||
(Array *)(self), array_elem_size(self), _index, \
|
||||
old_count, new_count, new_contents \
|
||||
)
|
||||
|
||||
/// Insert one `element` into the array at the given `index`.
|
||||
#define array_insert(self, _index, element) \
|
||||
_array__splice((Array *)(self), array_elem_size(self), _index, 0, 1, &(element))
|
||||
|
||||
/// Remove one element from the array at the given `index`.
|
||||
#define array_erase(self, _index) \
|
||||
_array__erase((Array *)(self), array_elem_size(self), _index)
|
||||
|
||||
/// Pop the last element off the array, returning the element by value.
|
||||
#define array_pop(self) ((self)->contents[--(self)->size])
|
||||
|
||||
/// Assign the contents of one array to another, reallocating if necessary.
|
||||
#define array_assign(self, other) \
|
||||
_array__assign((Array *)(self), (const Array *)(other), array_elem_size(self))
|
||||
|
||||
/// Swap one array with another
|
||||
#define array_swap(self, other) \
|
||||
_array__swap((Array *)(self), (Array *)(other))
|
||||
|
||||
/// Get the size of the array contents
|
||||
#define array_elem_size(self) (sizeof *(self)->contents)
|
||||
|
||||
/// Search a sorted array for a given `needle` value, using the given `compare`
|
||||
/// callback to determine the order.
|
||||
///
|
||||
/// If an existing element is found to be equal to `needle`, then the `index`
|
||||
/// out-parameter is set to the existing value's index, and the `exists`
|
||||
/// out-parameter is set to true. Otherwise, `index` is set to an index where
|
||||
/// `needle` should be inserted in order to preserve the sorting, and `exists`
|
||||
/// is set to false.
|
||||
#define array_search_sorted_with(self, compare, needle, _index, _exists) \
|
||||
_array__search_sorted(self, 0, compare, , needle, _index, _exists)
|
||||
|
||||
/// Search a sorted array for a given `needle` value, using integer comparisons
|
||||
/// of a given struct field (specified with a leading dot) to determine the order.
|
||||
///
|
||||
/// See also `array_search_sorted_with`.
|
||||
#define array_search_sorted_by(self, field, needle, _index, _exists) \
|
||||
_array__search_sorted(self, 0, _compare_int, field, needle, _index, _exists)
|
||||
|
||||
/// Insert a given `value` into a sorted array, using the given `compare`
|
||||
/// callback to determine the order.
|
||||
#define array_insert_sorted_with(self, compare, value) \
|
||||
do { \
|
||||
unsigned _index, _exists; \
|
||||
array_search_sorted_with(self, compare, &(value), &_index, &_exists); \
|
||||
if (!_exists) array_insert(self, _index, value); \
|
||||
} while (0)
|
||||
|
||||
/// Insert a given `value` into a sorted array, using integer comparisons of
|
||||
/// a given struct field (specified with a leading dot) to determine the order.
|
||||
///
|
||||
/// See also `array_search_sorted_by`.
|
||||
#define array_insert_sorted_by(self, field, value) \
|
||||
do { \
|
||||
unsigned _index, _exists; \
|
||||
array_search_sorted_by(self, field, (value) field, &_index, &_exists); \
|
||||
if (!_exists) array_insert(self, _index, value); \
|
||||
} while (0)
|
||||
|
||||
// Private
|
||||
|
||||
typedef Array(void) Array;
|
||||
|
||||
/// This is not what you're looking for, see `array_delete`.
|
||||
static inline void _array__delete(Array *self) {
|
||||
if (self->contents) {
|
||||
ts_free(self->contents);
|
||||
self->contents = NULL;
|
||||
self->size = 0;
|
||||
self->capacity = 0;
|
||||
}
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_erase`.
|
||||
static inline void _array__erase(Array *self, size_t element_size,
|
||||
uint32_t index) {
|
||||
assert(index < self->size);
|
||||
char *contents = (char *)self->contents;
|
||||
memmove(contents + index * element_size, contents + (index + 1) * element_size,
|
||||
(self->size - index - 1) * element_size);
|
||||
self->size--;
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_reserve`.
|
||||
static inline void _array__reserve(Array *self, size_t element_size, uint32_t new_capacity) {
|
||||
if (new_capacity > self->capacity) {
|
||||
if (self->contents) {
|
||||
self->contents = ts_realloc(self->contents, new_capacity * element_size);
|
||||
} else {
|
||||
self->contents = ts_malloc(new_capacity * element_size);
|
||||
}
|
||||
self->capacity = new_capacity;
|
||||
}
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_assign`.
|
||||
static inline void _array__assign(Array *self, const Array *other, size_t element_size) {
|
||||
_array__reserve(self, element_size, other->size);
|
||||
self->size = other->size;
|
||||
memcpy(self->contents, other->contents, self->size * element_size);
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_swap`.
|
||||
static inline void _array__swap(Array *self, Array *other) {
|
||||
Array swap = *other;
|
||||
*other = *self;
|
||||
*self = swap;
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_push` or `array_grow_by`.
|
||||
static inline void _array__grow(Array *self, uint32_t count, size_t element_size) {
|
||||
uint32_t new_size = self->size + count;
|
||||
if (new_size > self->capacity) {
|
||||
uint32_t new_capacity = self->capacity * 2;
|
||||
if (new_capacity < 8) new_capacity = 8;
|
||||
if (new_capacity < new_size) new_capacity = new_size;
|
||||
_array__reserve(self, element_size, new_capacity);
|
||||
}
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_splice`.
|
||||
static inline void _array__splice(Array *self, size_t element_size,
|
||||
uint32_t index, uint32_t old_count,
|
||||
uint32_t new_count, const void *elements) {
|
||||
uint32_t new_size = self->size + new_count - old_count;
|
||||
uint32_t old_end = index + old_count;
|
||||
uint32_t new_end = index + new_count;
|
||||
assert(old_end <= self->size);
|
||||
|
||||
_array__reserve(self, element_size, new_size);
|
||||
|
||||
char *contents = (char *)self->contents;
|
||||
if (self->size > old_end) {
|
||||
memmove(
|
||||
contents + new_end * element_size,
|
||||
contents + old_end * element_size,
|
||||
(self->size - old_end) * element_size
|
||||
);
|
||||
}
|
||||
if (new_count > 0) {
|
||||
if (elements) {
|
||||
memcpy(
|
||||
(contents + index * element_size),
|
||||
elements,
|
||||
new_count * element_size
|
||||
);
|
||||
} else {
|
||||
memset(
|
||||
(contents + index * element_size),
|
||||
0,
|
||||
new_count * element_size
|
||||
);
|
||||
}
|
||||
}
|
||||
self->size += new_count - old_count;
|
||||
}
|
||||
|
||||
/// A binary search routine, based on Rust's `std::slice::binary_search_by`.
|
||||
/// This is not what you're looking for, see `array_search_sorted_with` or `array_search_sorted_by`.
|
||||
#define _array__search_sorted(self, start, compare, suffix, needle, _index, _exists) \
|
||||
do { \
|
||||
*(_index) = start; \
|
||||
*(_exists) = false; \
|
||||
uint32_t size = (self)->size - *(_index); \
|
||||
if (size == 0) break; \
|
||||
int comparison; \
|
||||
while (size > 1) { \
|
||||
uint32_t half_size = size / 2; \
|
||||
uint32_t mid_index = *(_index) + half_size; \
|
||||
comparison = compare(&((self)->contents[mid_index] suffix), (needle)); \
|
||||
if (comparison <= 0) *(_index) = mid_index; \
|
||||
size -= half_size; \
|
||||
} \
|
||||
comparison = compare(&((self)->contents[*(_index)] suffix), (needle)); \
|
||||
if (comparison == 0) *(_exists) = true; \
|
||||
else if (comparison < 0) *(_index) += 1; \
|
||||
} while (0)
|
||||
|
||||
/// Helper macro for the `_sorted_by` routines below. This takes the left (existing)
|
||||
/// parameter by reference in order to work with the generic sorting function above.
|
||||
#define _compare_int(a, b) ((int)*(a) - (int)(b))
|
||||
|
||||
#ifdef _MSC_VER
|
||||
#pragma warning(pop)
|
||||
#elif defined(__GNUC__) || defined(__clang__)
|
||||
#pragma GCC diagnostic pop
|
||||
#endif
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif // TREE_SITTER_ARRAY_H_
|
||||
@@ -0,0 +1,286 @@
|
||||
#ifndef TREE_SITTER_PARSER_H_
|
||||
#define TREE_SITTER_PARSER_H_
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
#include <stdbool.h>
|
||||
#include <stdint.h>
|
||||
#include <stdlib.h>
|
||||
|
||||
#define ts_builtin_sym_error ((TSSymbol)-1)
|
||||
#define ts_builtin_sym_end 0
|
||||
#define TREE_SITTER_SERIALIZATION_BUFFER_SIZE 1024
|
||||
|
||||
#ifndef TREE_SITTER_API_H_
|
||||
typedef uint16_t TSStateId;
|
||||
typedef uint16_t TSSymbol;
|
||||
typedef uint16_t TSFieldId;
|
||||
typedef struct TSLanguage TSLanguage;
|
||||
typedef struct TSLanguageMetadata {
|
||||
uint8_t major_version;
|
||||
uint8_t minor_version;
|
||||
uint8_t patch_version;
|
||||
} TSLanguageMetadata;
|
||||
#endif
|
||||
|
||||
typedef struct {
|
||||
TSFieldId field_id;
|
||||
uint8_t child_index;
|
||||
bool inherited;
|
||||
} TSFieldMapEntry;
|
||||
|
||||
// Used to index the field and supertype maps.
|
||||
typedef struct {
|
||||
uint16_t index;
|
||||
uint16_t length;
|
||||
} TSMapSlice;
|
||||
|
||||
typedef struct {
|
||||
bool visible;
|
||||
bool named;
|
||||
bool supertype;
|
||||
} TSSymbolMetadata;
|
||||
|
||||
typedef struct TSLexer TSLexer;
|
||||
|
||||
struct TSLexer {
|
||||
int32_t lookahead;
|
||||
TSSymbol result_symbol;
|
||||
void (*advance)(TSLexer *, bool);
|
||||
void (*mark_end)(TSLexer *);
|
||||
uint32_t (*get_column)(TSLexer *);
|
||||
bool (*is_at_included_range_start)(const TSLexer *);
|
||||
bool (*eof)(const TSLexer *);
|
||||
void (*log)(const TSLexer *, const char *, ...);
|
||||
};
|
||||
|
||||
typedef enum {
|
||||
TSParseActionTypeShift,
|
||||
TSParseActionTypeReduce,
|
||||
TSParseActionTypeAccept,
|
||||
TSParseActionTypeRecover,
|
||||
} TSParseActionType;
|
||||
|
||||
typedef union {
|
||||
struct {
|
||||
uint8_t type;
|
||||
TSStateId state;
|
||||
bool extra;
|
||||
bool repetition;
|
||||
} shift;
|
||||
struct {
|
||||
uint8_t type;
|
||||
uint8_t child_count;
|
||||
TSSymbol symbol;
|
||||
int16_t dynamic_precedence;
|
||||
uint16_t production_id;
|
||||
} reduce;
|
||||
uint8_t type;
|
||||
} TSParseAction;
|
||||
|
||||
typedef struct {
|
||||
uint16_t lex_state;
|
||||
uint16_t external_lex_state;
|
||||
} TSLexMode;
|
||||
|
||||
typedef struct {
|
||||
uint16_t lex_state;
|
||||
uint16_t external_lex_state;
|
||||
uint16_t reserved_word_set_id;
|
||||
} TSLexerMode;
|
||||
|
||||
typedef union {
|
||||
TSParseAction action;
|
||||
struct {
|
||||
uint8_t count;
|
||||
bool reusable;
|
||||
} entry;
|
||||
} TSParseActionEntry;
|
||||
|
||||
typedef struct {
|
||||
int32_t start;
|
||||
int32_t end;
|
||||
} TSCharacterRange;
|
||||
|
||||
struct TSLanguage {
|
||||
uint32_t abi_version;
|
||||
uint32_t symbol_count;
|
||||
uint32_t alias_count;
|
||||
uint32_t token_count;
|
||||
uint32_t external_token_count;
|
||||
uint32_t state_count;
|
||||
uint32_t large_state_count;
|
||||
uint32_t production_id_count;
|
||||
uint32_t field_count;
|
||||
uint16_t max_alias_sequence_length;
|
||||
const uint16_t *parse_table;
|
||||
const uint16_t *small_parse_table;
|
||||
const uint32_t *small_parse_table_map;
|
||||
const TSParseActionEntry *parse_actions;
|
||||
const char * const *symbol_names;
|
||||
const char * const *field_names;
|
||||
const TSMapSlice *field_map_slices;
|
||||
const TSFieldMapEntry *field_map_entries;
|
||||
const TSSymbolMetadata *symbol_metadata;
|
||||
const TSSymbol *public_symbol_map;
|
||||
const uint16_t *alias_map;
|
||||
const TSSymbol *alias_sequences;
|
||||
const TSLexerMode *lex_modes;
|
||||
bool (*lex_fn)(TSLexer *, TSStateId);
|
||||
bool (*keyword_lex_fn)(TSLexer *, TSStateId);
|
||||
TSSymbol keyword_capture_token;
|
||||
struct {
|
||||
const bool *states;
|
||||
const TSSymbol *symbol_map;
|
||||
void *(*create)(void);
|
||||
void (*destroy)(void *);
|
||||
bool (*scan)(void *, TSLexer *, const bool *symbol_whitelist);
|
||||
unsigned (*serialize)(void *, char *);
|
||||
void (*deserialize)(void *, const char *, unsigned);
|
||||
} external_scanner;
|
||||
const TSStateId *primary_state_ids;
|
||||
const char *name;
|
||||
const TSSymbol *reserved_words;
|
||||
uint16_t max_reserved_word_set_size;
|
||||
uint32_t supertype_count;
|
||||
const TSSymbol *supertype_symbols;
|
||||
const TSMapSlice *supertype_map_slices;
|
||||
const TSSymbol *supertype_map_entries;
|
||||
TSLanguageMetadata metadata;
|
||||
};
|
||||
|
||||
static inline bool set_contains(const TSCharacterRange *ranges, uint32_t len, int32_t lookahead) {
|
||||
uint32_t index = 0;
|
||||
uint32_t size = len - index;
|
||||
while (size > 1) {
|
||||
uint32_t half_size = size / 2;
|
||||
uint32_t mid_index = index + half_size;
|
||||
const TSCharacterRange *range = &ranges[mid_index];
|
||||
if (lookahead >= range->start && lookahead <= range->end) {
|
||||
return true;
|
||||
} else if (lookahead > range->end) {
|
||||
index = mid_index;
|
||||
}
|
||||
size -= half_size;
|
||||
}
|
||||
const TSCharacterRange *range = &ranges[index];
|
||||
return (lookahead >= range->start && lookahead <= range->end);
|
||||
}
|
||||
|
||||
/*
|
||||
* Lexer Macros
|
||||
*/
|
||||
|
||||
#ifdef _MSC_VER
|
||||
#define UNUSED __pragma(warning(suppress : 4101))
|
||||
#else
|
||||
#define UNUSED __attribute__((unused))
|
||||
#endif
|
||||
|
||||
#define START_LEXER() \
|
||||
bool result = false; \
|
||||
bool skip = false; \
|
||||
UNUSED \
|
||||
bool eof = false; \
|
||||
int32_t lookahead; \
|
||||
goto start; \
|
||||
next_state: \
|
||||
lexer->advance(lexer, skip); \
|
||||
start: \
|
||||
skip = false; \
|
||||
lookahead = lexer->lookahead;
|
||||
|
||||
#define ADVANCE(state_value) \
|
||||
{ \
|
||||
state = state_value; \
|
||||
goto next_state; \
|
||||
}
|
||||
|
||||
#define ADVANCE_MAP(...) \
|
||||
{ \
|
||||
static const uint16_t map[] = { __VA_ARGS__ }; \
|
||||
for (uint32_t i = 0; i < sizeof(map) / sizeof(map[0]); i += 2) { \
|
||||
if (map[i] == lookahead) { \
|
||||
state = map[i + 1]; \
|
||||
goto next_state; \
|
||||
} \
|
||||
} \
|
||||
}
|
||||
|
||||
#define SKIP(state_value) \
|
||||
{ \
|
||||
skip = true; \
|
||||
state = state_value; \
|
||||
goto next_state; \
|
||||
}
|
||||
|
||||
#define ACCEPT_TOKEN(symbol_value) \
|
||||
result = true; \
|
||||
lexer->result_symbol = symbol_value; \
|
||||
lexer->mark_end(lexer);
|
||||
|
||||
#define END_STATE() return result;
|
||||
|
||||
/*
|
||||
* Parse Table Macros
|
||||
*/
|
||||
|
||||
#define SMALL_STATE(id) ((id) - LARGE_STATE_COUNT)
|
||||
|
||||
#define STATE(id) id
|
||||
|
||||
#define ACTIONS(id) id
|
||||
|
||||
#define SHIFT(state_value) \
|
||||
{{ \
|
||||
.shift = { \
|
||||
.type = TSParseActionTypeShift, \
|
||||
.state = (state_value) \
|
||||
} \
|
||||
}}
|
||||
|
||||
#define SHIFT_REPEAT(state_value) \
|
||||
{{ \
|
||||
.shift = { \
|
||||
.type = TSParseActionTypeShift, \
|
||||
.state = (state_value), \
|
||||
.repetition = true \
|
||||
} \
|
||||
}}
|
||||
|
||||
#define SHIFT_EXTRA() \
|
||||
{{ \
|
||||
.shift = { \
|
||||
.type = TSParseActionTypeShift, \
|
||||
.extra = true \
|
||||
} \
|
||||
}}
|
||||
|
||||
#define REDUCE(symbol_name, children, precedence, prod_id) \
|
||||
{{ \
|
||||
.reduce = { \
|
||||
.type = TSParseActionTypeReduce, \
|
||||
.symbol = symbol_name, \
|
||||
.child_count = children, \
|
||||
.dynamic_precedence = precedence, \
|
||||
.production_id = prod_id \
|
||||
}, \
|
||||
}}
|
||||
|
||||
#define RECOVER() \
|
||||
{{ \
|
||||
.type = TSParseActionTypeRecover \
|
||||
}}
|
||||
|
||||
#define ACCEPT_INPUT() \
|
||||
{{ \
|
||||
.type = TSParseActionTypeAccept \
|
||||
}}
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif // TREE_SITTER_PARSER_H_
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"grammars": [
|
||||
{
|
||||
"name": "vbnet",
|
||||
"camelcase": "Vbnet",
|
||||
"scope": "source.vbnet",
|
||||
"path": ".",
|
||||
"file-types": ["vb"]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"version": "0.1.0-cbm.1",
|
||||
"license": "MIT",
|
||||
"description": "cbm fork of tree-sitter-vbnet @ 538b708 with CodeGraph's real-world VB.NET patch"
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user