Files
codebase-memory-mcp/tools/tree-sitter-sql/patches/sqlite-dialect.patch
T
Martin VogelandKerf d9b22e0b7f fix(grammar): stop flagging valid SQLite migrations as partially parsed
Valid SQLite migration files were reported parse_partial because the
vendored SQL grammar (DerekStride/tree-sitter-sql @ 851e9cb257ba) rejects
SQLite syntax they use: PRAGMA, AUTOINCREMENT, a trigger's BEGIN ... END
body, a column named `key` after the first column, WITHOUT ROWID, an
unparenthesized trigger WHEN, INSERT OR IGNORE, and CREATE VIRTUAL TABLE
... USING fts5(...). Each became an ERROR node; the trigger's error region
in atuin's ai_session schema also swallowed the CREATE TABLE after it.

The `key` case is lexical: after a comma inside a column list the grammar
also accepts MySQL's `KEY name (cols)` index, so `key` lexes as that
keyword there and `key TEXT NOT NULL` became a malformed index.

Fix: sql becomes a self-maintained fork of the pin we already vendored,
following the pascal model. tools/tree-sitter-sql/ holds upstream's
grammar.js + grammar/ (blob-identical to the pin and to upstream main @
3c99273f9b32, where its corpus lives) and one patch,
patches/sqlite-dialect.patch, touching grammar/ only:
- PRAGMA [schema.]name [= value | (value)] (new pragma_statement)
- AUTOINCREMENT as keyword_auto_increment, upstream's own change
  (c042cde1e2a6, 2026-11-08, after our pin)
- `key` accepted as a column name; the type keywords that take no
  arguments are accepted as a MySQL index name so `KEY name (name)` stays
  an index
- WITHOUT ROWID
- a trigger body as the existing block node, after an optional
  unparenthesized WHEN (precedence keeps Postgres `WHEN (...) EXECUTE
  FUNCTION` trees unchanged)
- INSERT OR {ABORT | FAIL | IGNORE | REPLACE | ROLLBACK}
- CREATE VIRTUAL TABLE [IF NOT EXISTS] name USING module(args), aliased
  to create_table (new module_argument node), so the table is extracted
  as a Table like any other
Upstream node types keep their shape: upstream's 511 corpus cases pass
unchanged; test/corpus/sqlite.txt adds 11 (9 fail on the unpatched
grammar, 2 controls pass on both). The vendored scanner.c is kept as is
(it carries our start_tag leak fix 0e61867d9, now listed under the
MANIFEST's "Local source patches"; upstream fixed the same leak after our
pin in ee2f84f9b857).

Provenance: without the patch, tree-sitter-cli 0.25.10 regenerates
upstream's shipped parser.c byte-for-byte except line 1 (upstream's
generator stamp has no version), and grammar.json, node-types.json and
tree_sitter/*.h byte-identical. Patched: three generations are
byte-identical (parser.c SHA-256 3c64e5c4...7887); ABI 15; states 30,622
-> 31,081, symbols 770 -> 782; 0 non-ASCII bytes, no string literal with
a space, one comment, the same call-position identifiers as before. Node
types only gain entries (pragma_statement, module_argument, four
keywords; create_table gains a `module` field).

Proof, indexed from scratch with isolated caches (main 6d34ca15 vs this
change): dani-garcia/vaultwarden @ 58a980f8 migrations/sqlite (116
files) parse_partial 3 -> 0, graph identical (303 nodes, 446 edges);
atuinsh/atuin @ 2535161e *.sql (69 files) 19 -> 7: all 12 valid SQLite
files clean; the 7 left are the invalid `ALTER TABLE ... DROP INDEX`
down migration and the 2 MySQL + 4 Postgres files, same ranges as
before; graph +2 Table nodes (the fts5 table and the table the trigger
error swallowed), +2 DEFINES, +3 USAGE lineage edges. No newly flagged
file. Control: helm @ 9b88efd6 indexes to an identical graph (12,251
nodes, 39,561 edges).

Registered like pascal: MANIFEST.md fork rows + notes (ABI tally
recounted, unchanged), the audit FORKS map, THIRD_PARTY.md, vendored
checksums. LICENSE stays upstream's MIT byte-identical.

Tests: parse_coverage carries the two SQLite tests from #2246 as written
(sqlite_valid_migration_not_flagged, red before this change, and the
sqlite_malformed_still_flagged control) and adds `key` columns, WITHOUT
ROWID, a WHEN trigger with INSERT OR IGNORE plus an fts5 table (asserted
as a Table), PRAGMA forms, and two controls: MySQL's KEY index still
clean, and ALTER TABLE ... DROP INDEX still flagged.

Distilled from #2246.

Co-authored-by: Kerf <kerfern@hotmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-11 19:44:47 +01:00

188 lines
7.0 KiB
Diff

diff --git a/grammar/column-lists.js b/grammar/column-lists.js
index 55d257d70e83c364d875494961944439118fd8b5..2b99c49354039a628fce8b92a7b00119eed80207 100644
--- a/grammar/column-lists.js
+++ b/grammar/column-lists.js
@@ -23,7 +23,10 @@ export default {
),
column_definition: $ => prec.left(seq(
- field('name', $._column),
+ // cbm fork (#2246): SQLite does not reserve `key`, so it is a common column
+ // name, but after the first column it lexes as the keyword of MySQL's
+ // `KEY name (cols)` index. Accept that keyword as a column name too.
+ field('name', choice($._column, alias($.keyword_key, $.identifier))),
field('type', $._type),
repeat($._column_constraint),
)),
@@ -153,7 +156,20 @@ export default {
seq(optional($.keyword_foreign), $.keyword_key, optional($._if_not_exists)),
$.keyword_index,
),
- optional(field('name', $.identifier)),
+ // cbm fork (#2246): with `key` also a column name (column_definition), a
+ // type keyword after KEY now lexes as that type, so accept the types that
+ // take no arguments as index names too (MySQL `KEY name (name)`). A type
+ // that takes arguments (int, varchar, time, ...) is not in the list.
+ optional(field('name', choice(
+ $.identifier,
+ ...[
+ 'boolean', 'image', 'smallserial', 'serial', 'bigserial', 'money',
+ 'smallmoney', 'string', 'text', 'uuid', 'json', 'jsonb', 'xml', 'bytea',
+ 'inet', 'date', 'datetime', 'datetime2', 'smalldatetime', 'timestamptz',
+ 'interval', 'geometry', 'geography', 'box2d', 'box3d', 'oid', 'name',
+ 'regclass', 'regnamespace', 'regproc', 'regtype',
+ ].map((type) => alias($['keyword_' + type], $.identifier)),
+ ))),
$.ordered_columns,
optional(
seq(
diff --git a/grammar/keywords.js b/grammar/keywords.js
index 6be9e8940972d69a3c5804cce56666e59be23fd2..a88aef4e2b85037c56f9eb4d2f27b57846b7fdb4 100644
--- a/grammar/keywords.js
+++ b/grammar/keywords.js
@@ -96,7 +96,9 @@ export default {
keyword_for: _ => make_keyword("for"),
keyword_if: _ => make_keyword("if"),
keyword_exists: _ => make_keyword("exists"),
- keyword_auto_increment: _ => make_keyword("auto_increment"),
+ // cbm fork (#2246): SQLite AUTOINCREMENT. Upstream c042cde1e2a6 (2026-11-08,
+ // after our pin) made exactly this change; drop the hunk on a refresh past it.
+ keyword_auto_increment: _ => choice(make_keyword("auto_increment"),make_keyword("autoincrement")),
keyword_generated: _ => make_keyword("generated"),
keyword_always: _ => make_keyword("always"),
keyword_collate: _ => make_keyword("collate"),
@@ -282,6 +284,11 @@ export default {
keyword_execute: _ => make_keyword("execute"),
keyword_procedure: _ => make_keyword("procedure"),
keyword_object_id: _ => make_keyword("object_id"),
+ // cbm fork (#2246): SQLite PRAGMA, WITHOUT ROWID and INSERT OR <conflict>
+ keyword_pragma: _ => make_keyword("pragma"),
+ keyword_rowid: _ => make_keyword("rowid"),
+ keyword_abort: _ => make_keyword("abort"),
+ keyword_fail: _ => make_keyword("fail"),
// Hive Keywords
keyword_external: _ => make_keyword("external"),
diff --git a/grammar/statements/create.js b/grammar/statements/create.js
index 41f54a244e283c38b7fec7b1f1d6e5e8f595e8a8..388c2e8838fc95e1b34877a3eff47ceb2d0d84ee 100644
--- a/grammar/statements/create.js
+++ b/grammar/statements/create.js
@@ -8,6 +8,7 @@ export default {
_create_statement: $ => seq(
choice(
$.create_table,
+ alias($._create_virtual_table, $.create_table), // cbm fork (#2246)
$.create_view,
$.create_materialized_view,
$.create_index,
@@ -77,10 +78,32 @@ export default {
paren_list($.table_option, true),
),
seq($.keyword_without, $.keyword_oids),
+ seq($.keyword_without, $.keyword_rowid), // cbm fork (#2246): SQLite
$.storage_parameters,
$.table_option,
),
+ // cbm fork (#2246): SQLite CREATE VIRTUAL TABLE [IF NOT EXISTS] name
+ // USING module[(arguments)], e.g. an fts5 full-text index. It is a table,
+ // so it is aliased to create_table and extracted like any other.
+ _create_virtual_table: $ => prec.right(seq(
+ $.keyword_create,
+ $.keyword_virtual,
+ $.keyword_table,
+ optional($._if_not_exists),
+ $.object_reference,
+ $.keyword_using,
+ field('module', $.identifier),
+ optional(paren_list($.module_argument, true)),
+ )),
+
+ // a column (fts5 `body`, `body UNINDEXED`, rtree `minX`) or an option
+ // (fts5 `tokenize = 'unicode61'`, `content = ''`, `contentless_delete = 1`)
+ module_argument: $ => choice(
+ seq(field('name', $.identifier), repeat($.identifier)),
+ seq(field('name', $.identifier), '=', field('value', choice($.literal, $.identifier))),
+ ),
+
stored_as: $ => seq(
$.keyword_stored,
$.keyword_as,
@@ -512,10 +535,20 @@ export default {
seq($.keyword_when, wrapped_in_parenthesis($._expression)),
),
),
- $.keyword_execute,
- choice($.keyword_function, $.keyword_procedure),
- $.object_reference,
- paren_list(field('parameter', $.term)),
+ choice(
+ seq(
+ $.keyword_execute,
+ choice($.keyword_function, $.keyword_procedure),
+ $.object_reference,
+ paren_list(field('parameter', $.term)),
+ ),
+ // cbm fork (#2246): SQLite runs a BEGIN ... END body, after a WHEN
+ // condition SQLite does not parenthesize
+ seq(
+ optional(seq($.keyword_when, $._expression)),
+ $.block,
+ ),
+ ),
),
_create_trigger_event: $ => choice(
diff --git a/grammar/statements/index.js b/grammar/statements/index.js
index 94f0ec628add92d8a258f3441dbc85e551290dec..8f15fbd249086005255a4aee01acbf6f007f4ba4 100644
--- a/grammar/statements/index.js
+++ b/grammar/statements/index.js
@@ -84,8 +84,23 @@ export default {
$.comment_statement,
$.set_statement,
$.reset_statement,
+ $.pragma_statement, // cbm fork (#2246)
),
+ // cbm fork (#2246): SQLite PRAGMA [schema.]name [= value | (value)]
+ pragma_statement: $ => prec.right(seq(
+ $.keyword_pragma,
+ $.object_reference,
+ optional(
+ choice(
+ seq('=', field('value', $._pragma_value)),
+ wrapped_in_parenthesis(field('value', $._pragma_value)),
+ ),
+ ),
+ )),
+
+ _pragma_value: $ => choice($.literal, $.identifier),
+
...create_rules,
...alter_rules,
...drop_rules,
diff --git a/grammar/statements/insert.js b/grammar/statements/insert.js
index f241d126b4a5bfcccaad193ca956ccbefa2ae1a6..d78c104c4aae51a467b63ad6aaeb82d95051b827 100644
--- a/grammar/statements/insert.js
+++ b/grammar/statements/insert.js
@@ -12,6 +12,19 @@ export default {
$.keyword_insert,
$.keyword_replace
),
+ // cbm fork (#2246): SQLite INSERT OR {ABORT | FAIL | IGNORE | REPLACE | ROLLBACK}
+ optional(
+ seq(
+ $.keyword_or,
+ choice(
+ $.keyword_abort,
+ $.keyword_fail,
+ $.keyword_ignore,
+ $.keyword_replace,
+ $.keyword_rollback,
+ ),
+ ),
+ ),
optional(
choice(
$.keyword_low_priority,