mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-15 13:56:34 +08:00
Valid SQLite migration files were reported parse_partial because the
vendored SQL grammar (DerekStride/tree-sitter-sql @ 851e9cb257ba) rejects
SQLite syntax they use: PRAGMA, AUTOINCREMENT, a trigger's BEGIN ... END
body, a column named `key` after the first column, WITHOUT ROWID, an
unparenthesized trigger WHEN, INSERT OR IGNORE, and CREATE VIRTUAL TABLE
... USING fts5(...). Each became an ERROR node; the trigger's error region
in atuin's ai_session schema also swallowed the CREATE TABLE after it.
The `key` case is lexical: after a comma inside a column list the grammar
also accepts MySQL's `KEY name (cols)` index, so `key` lexes as that
keyword there and `key TEXT NOT NULL` became a malformed index.
Fix: sql becomes a self-maintained fork of the pin we already vendored,
following the pascal model. tools/tree-sitter-sql/ holds upstream's
grammar.js + grammar/ (blob-identical to the pin and to upstream main @
3c99273f9b32, where its corpus lives) and one patch,
patches/sqlite-dialect.patch, touching grammar/ only:
- PRAGMA [schema.]name [= value | (value)] (new pragma_statement)
- AUTOINCREMENT as keyword_auto_increment, upstream's own change
(c042cde1e2a6, 2026-11-08, after our pin)
- `key` accepted as a column name; the type keywords that take no
arguments are accepted as a MySQL index name so `KEY name (name)` stays
an index
- WITHOUT ROWID
- a trigger body as the existing block node, after an optional
unparenthesized WHEN (precedence keeps Postgres `WHEN (...) EXECUTE
FUNCTION` trees unchanged)
- INSERT OR {ABORT | FAIL | IGNORE | REPLACE | ROLLBACK}
- CREATE VIRTUAL TABLE [IF NOT EXISTS] name USING module(args), aliased
to create_table (new module_argument node), so the table is extracted
as a Table like any other
Upstream node types keep their shape: upstream's 511 corpus cases pass
unchanged; test/corpus/sqlite.txt adds 11 (9 fail on the unpatched
grammar, 2 controls pass on both). The vendored scanner.c is kept as is
(it carries our start_tag leak fix 0e61867d9, now listed under the
MANIFEST's "Local source patches"; upstream fixed the same leak after our
pin in ee2f84f9b857).
Provenance: without the patch, tree-sitter-cli 0.25.10 regenerates
upstream's shipped parser.c byte-for-byte except line 1 (upstream's
generator stamp has no version), and grammar.json, node-types.json and
tree_sitter/*.h byte-identical. Patched: three generations are
byte-identical (parser.c SHA-256 3c64e5c4...7887); ABI 15; states 30,622
-> 31,081, symbols 770 -> 782; 0 non-ASCII bytes, no string literal with
a space, one comment, the same call-position identifiers as before. Node
types only gain entries (pragma_statement, module_argument, four
keywords; create_table gains a `module` field).
Proof, indexed from scratch with isolated caches (main 6d34ca15 vs this
change): dani-garcia/vaultwarden @ 58a980f8 migrations/sqlite (116
files) parse_partial 3 -> 0, graph identical (303 nodes, 446 edges);
atuinsh/atuin @ 2535161e *.sql (69 files) 19 -> 7: all 12 valid SQLite
files clean; the 7 left are the invalid `ALTER TABLE ... DROP INDEX`
down migration and the 2 MySQL + 4 Postgres files, same ranges as
before; graph +2 Table nodes (the fts5 table and the table the trigger
error swallowed), +2 DEFINES, +3 USAGE lineage edges. No newly flagged
file. Control: helm @ 9b88efd6 indexes to an identical graph (12,251
nodes, 39,561 edges).
Registered like pascal: MANIFEST.md fork rows + notes (ABI tally
recounted, unchanged), the audit FORKS map, THIRD_PARTY.md, vendored
checksums. LICENSE stays upstream's MIT byte-identical.
Tests: parse_coverage carries the two SQLite tests from #2246 as written
(sqlite_valid_migration_not_flagged, red before this change, and the
sqlite_malformed_still_flagged control) and adds `key` columns, WITHOUT
ROWID, a WHEN trigger with INSERT OR IGNORE plus an fts5 table (asserted
as a Table), PRAGMA forms, and two controls: MySQL's KEY index still
clean, and ALTER TABLE ... DROP INDEX still flagged.
Distilled from #2246.
Co-authored-by: Kerf <kerfern@hotmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
188 lines
7.0 KiB
Diff
188 lines
7.0 KiB
Diff
diff --git a/grammar/column-lists.js b/grammar/column-lists.js
|
|
index 55d257d70e83c364d875494961944439118fd8b5..2b99c49354039a628fce8b92a7b00119eed80207 100644
|
|
--- a/grammar/column-lists.js
|
|
+++ b/grammar/column-lists.js
|
|
@@ -23,7 +23,10 @@ export default {
|
|
),
|
|
|
|
column_definition: $ => prec.left(seq(
|
|
- field('name', $._column),
|
|
+ // cbm fork (#2246): SQLite does not reserve `key`, so it is a common column
|
|
+ // name, but after the first column it lexes as the keyword of MySQL's
|
|
+ // `KEY name (cols)` index. Accept that keyword as a column name too.
|
|
+ field('name', choice($._column, alias($.keyword_key, $.identifier))),
|
|
field('type', $._type),
|
|
repeat($._column_constraint),
|
|
)),
|
|
@@ -153,7 +156,20 @@ export default {
|
|
seq(optional($.keyword_foreign), $.keyword_key, optional($._if_not_exists)),
|
|
$.keyword_index,
|
|
),
|
|
- optional(field('name', $.identifier)),
|
|
+ // cbm fork (#2246): with `key` also a column name (column_definition), a
|
|
+ // type keyword after KEY now lexes as that type, so accept the types that
|
|
+ // take no arguments as index names too (MySQL `KEY name (name)`). A type
|
|
+ // that takes arguments (int, varchar, time, ...) is not in the list.
|
|
+ optional(field('name', choice(
|
|
+ $.identifier,
|
|
+ ...[
|
|
+ 'boolean', 'image', 'smallserial', 'serial', 'bigserial', 'money',
|
|
+ 'smallmoney', 'string', 'text', 'uuid', 'json', 'jsonb', 'xml', 'bytea',
|
|
+ 'inet', 'date', 'datetime', 'datetime2', 'smalldatetime', 'timestamptz',
|
|
+ 'interval', 'geometry', 'geography', 'box2d', 'box3d', 'oid', 'name',
|
|
+ 'regclass', 'regnamespace', 'regproc', 'regtype',
|
|
+ ].map((type) => alias($['keyword_' + type], $.identifier)),
|
|
+ ))),
|
|
$.ordered_columns,
|
|
optional(
|
|
seq(
|
|
diff --git a/grammar/keywords.js b/grammar/keywords.js
|
|
index 6be9e8940972d69a3c5804cce56666e59be23fd2..a88aef4e2b85037c56f9eb4d2f27b57846b7fdb4 100644
|
|
--- a/grammar/keywords.js
|
|
+++ b/grammar/keywords.js
|
|
@@ -96,7 +96,9 @@ export default {
|
|
keyword_for: _ => make_keyword("for"),
|
|
keyword_if: _ => make_keyword("if"),
|
|
keyword_exists: _ => make_keyword("exists"),
|
|
- keyword_auto_increment: _ => make_keyword("auto_increment"),
|
|
+ // cbm fork (#2246): SQLite AUTOINCREMENT. Upstream c042cde1e2a6 (2026-11-08,
|
|
+ // after our pin) made exactly this change; drop the hunk on a refresh past it.
|
|
+ keyword_auto_increment: _ => choice(make_keyword("auto_increment"),make_keyword("autoincrement")),
|
|
keyword_generated: _ => make_keyword("generated"),
|
|
keyword_always: _ => make_keyword("always"),
|
|
keyword_collate: _ => make_keyword("collate"),
|
|
@@ -282,6 +284,11 @@ export default {
|
|
keyword_execute: _ => make_keyword("execute"),
|
|
keyword_procedure: _ => make_keyword("procedure"),
|
|
keyword_object_id: _ => make_keyword("object_id"),
|
|
+ // cbm fork (#2246): SQLite PRAGMA, WITHOUT ROWID and INSERT OR <conflict>
|
|
+ keyword_pragma: _ => make_keyword("pragma"),
|
|
+ keyword_rowid: _ => make_keyword("rowid"),
|
|
+ keyword_abort: _ => make_keyword("abort"),
|
|
+ keyword_fail: _ => make_keyword("fail"),
|
|
|
|
// Hive Keywords
|
|
keyword_external: _ => make_keyword("external"),
|
|
diff --git a/grammar/statements/create.js b/grammar/statements/create.js
|
|
index 41f54a244e283c38b7fec7b1f1d6e5e8f595e8a8..388c2e8838fc95e1b34877a3eff47ceb2d0d84ee 100644
|
|
--- a/grammar/statements/create.js
|
|
+++ b/grammar/statements/create.js
|
|
@@ -8,6 +8,7 @@ export default {
|
|
_create_statement: $ => seq(
|
|
choice(
|
|
$.create_table,
|
|
+ alias($._create_virtual_table, $.create_table), // cbm fork (#2246)
|
|
$.create_view,
|
|
$.create_materialized_view,
|
|
$.create_index,
|
|
@@ -77,10 +78,32 @@ export default {
|
|
paren_list($.table_option, true),
|
|
),
|
|
seq($.keyword_without, $.keyword_oids),
|
|
+ seq($.keyword_without, $.keyword_rowid), // cbm fork (#2246): SQLite
|
|
$.storage_parameters,
|
|
$.table_option,
|
|
),
|
|
|
|
+ // cbm fork (#2246): SQLite CREATE VIRTUAL TABLE [IF NOT EXISTS] name
|
|
+ // USING module[(arguments)], e.g. an fts5 full-text index. It is a table,
|
|
+ // so it is aliased to create_table and extracted like any other.
|
|
+ _create_virtual_table: $ => prec.right(seq(
|
|
+ $.keyword_create,
|
|
+ $.keyword_virtual,
|
|
+ $.keyword_table,
|
|
+ optional($._if_not_exists),
|
|
+ $.object_reference,
|
|
+ $.keyword_using,
|
|
+ field('module', $.identifier),
|
|
+ optional(paren_list($.module_argument, true)),
|
|
+ )),
|
|
+
|
|
+ // a column (fts5 `body`, `body UNINDEXED`, rtree `minX`) or an option
|
|
+ // (fts5 `tokenize = 'unicode61'`, `content = ''`, `contentless_delete = 1`)
|
|
+ module_argument: $ => choice(
|
|
+ seq(field('name', $.identifier), repeat($.identifier)),
|
|
+ seq(field('name', $.identifier), '=', field('value', choice($.literal, $.identifier))),
|
|
+ ),
|
|
+
|
|
stored_as: $ => seq(
|
|
$.keyword_stored,
|
|
$.keyword_as,
|
|
@@ -512,10 +535,20 @@ export default {
|
|
seq($.keyword_when, wrapped_in_parenthesis($._expression)),
|
|
),
|
|
),
|
|
- $.keyword_execute,
|
|
- choice($.keyword_function, $.keyword_procedure),
|
|
- $.object_reference,
|
|
- paren_list(field('parameter', $.term)),
|
|
+ choice(
|
|
+ seq(
|
|
+ $.keyword_execute,
|
|
+ choice($.keyword_function, $.keyword_procedure),
|
|
+ $.object_reference,
|
|
+ paren_list(field('parameter', $.term)),
|
|
+ ),
|
|
+ // cbm fork (#2246): SQLite runs a BEGIN ... END body, after a WHEN
|
|
+ // condition SQLite does not parenthesize
|
|
+ seq(
|
|
+ optional(seq($.keyword_when, $._expression)),
|
|
+ $.block,
|
|
+ ),
|
|
+ ),
|
|
),
|
|
|
|
_create_trigger_event: $ => choice(
|
|
diff --git a/grammar/statements/index.js b/grammar/statements/index.js
|
|
index 94f0ec628add92d8a258f3441dbc85e551290dec..8f15fbd249086005255a4aee01acbf6f007f4ba4 100644
|
|
--- a/grammar/statements/index.js
|
|
+++ b/grammar/statements/index.js
|
|
@@ -84,8 +84,23 @@ export default {
|
|
$.comment_statement,
|
|
$.set_statement,
|
|
$.reset_statement,
|
|
+ $.pragma_statement, // cbm fork (#2246)
|
|
),
|
|
|
|
+ // cbm fork (#2246): SQLite PRAGMA [schema.]name [= value | (value)]
|
|
+ pragma_statement: $ => prec.right(seq(
|
|
+ $.keyword_pragma,
|
|
+ $.object_reference,
|
|
+ optional(
|
|
+ choice(
|
|
+ seq('=', field('value', $._pragma_value)),
|
|
+ wrapped_in_parenthesis(field('value', $._pragma_value)),
|
|
+ ),
|
|
+ ),
|
|
+ )),
|
|
+
|
|
+ _pragma_value: $ => choice($.literal, $.identifier),
|
|
+
|
|
...create_rules,
|
|
...alter_rules,
|
|
...drop_rules,
|
|
diff --git a/grammar/statements/insert.js b/grammar/statements/insert.js
|
|
index f241d126b4a5bfcccaad193ca956ccbefa2ae1a6..d78c104c4aae51a467b63ad6aaeb82d95051b827 100644
|
|
--- a/grammar/statements/insert.js
|
|
+++ b/grammar/statements/insert.js
|
|
@@ -12,6 +12,19 @@ export default {
|
|
$.keyword_insert,
|
|
$.keyword_replace
|
|
),
|
|
+ // cbm fork (#2246): SQLite INSERT OR {ABORT | FAIL | IGNORE | REPLACE | ROLLBACK}
|
|
+ optional(
|
|
+ seq(
|
|
+ $.keyword_or,
|
|
+ choice(
|
|
+ $.keyword_abort,
|
|
+ $.keyword_fail,
|
|
+ $.keyword_ignore,
|
|
+ $.keyword_replace,
|
|
+ $.keyword_rollback,
|
|
+ ),
|
|
+ ),
|
|
+ ),
|
|
optional(
|
|
choice(
|
|
$.keyword_low_priority,
|