mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-15 13:56:34 +08:00
Valid SQLite migration files were reported parse_partial because the
vendored SQL grammar (DerekStride/tree-sitter-sql @ 851e9cb257ba) rejects
SQLite syntax they use: PRAGMA, AUTOINCREMENT, a trigger's BEGIN ... END
body, a column named `key` after the first column, WITHOUT ROWID, an
unparenthesized trigger WHEN, INSERT OR IGNORE, and CREATE VIRTUAL TABLE
... USING fts5(...). Each became an ERROR node; the trigger's error region
in atuin's ai_session schema also swallowed the CREATE TABLE after it.
The `key` case is lexical: after a comma inside a column list the grammar
also accepts MySQL's `KEY name (cols)` index, so `key` lexes as that
keyword there and `key TEXT NOT NULL` became a malformed index.
Fix: sql becomes a self-maintained fork of the pin we already vendored,
following the pascal model. tools/tree-sitter-sql/ holds upstream's
grammar.js + grammar/ (blob-identical to the pin and to upstream main @
3c99273f9b32, where its corpus lives) and one patch,
patches/sqlite-dialect.patch, touching grammar/ only:
- PRAGMA [schema.]name [= value | (value)] (new pragma_statement)
- AUTOINCREMENT as keyword_auto_increment, upstream's own change
(c042cde1e2a6, 2026-11-08, after our pin)
- `key` accepted as a column name; the type keywords that take no
arguments are accepted as a MySQL index name so `KEY name (name)` stays
an index
- WITHOUT ROWID
- a trigger body as the existing block node, after an optional
unparenthesized WHEN (precedence keeps Postgres `WHEN (...) EXECUTE
FUNCTION` trees unchanged)
- INSERT OR {ABORT | FAIL | IGNORE | REPLACE | ROLLBACK}
- CREATE VIRTUAL TABLE [IF NOT EXISTS] name USING module(args), aliased
to create_table (new module_argument node), so the table is extracted
as a Table like any other
Upstream node types keep their shape: upstream's 511 corpus cases pass
unchanged; test/corpus/sqlite.txt adds 11 (9 fail on the unpatched
grammar, 2 controls pass on both). The vendored scanner.c is kept as is
(it carries our start_tag leak fix 0e61867d9, now listed under the
MANIFEST's "Local source patches"; upstream fixed the same leak after our
pin in ee2f84f9b857).
Provenance: without the patch, tree-sitter-cli 0.25.10 regenerates
upstream's shipped parser.c byte-for-byte except line 1 (upstream's
generator stamp has no version), and grammar.json, node-types.json and
tree_sitter/*.h byte-identical. Patched: three generations are
byte-identical (parser.c SHA-256 3c64e5c4...7887); ABI 15; states 30,622
-> 31,081, symbols 770 -> 782; 0 non-ASCII bytes, no string literal with
a space, one comment, the same call-position identifiers as before. Node
types only gain entries (pragma_statement, module_argument, four
keywords; create_table gains a `module` field).
Proof, indexed from scratch with isolated caches (main 6d34ca15 vs this
change): dani-garcia/vaultwarden @ 58a980f8 migrations/sqlite (116
files) parse_partial 3 -> 0, graph identical (303 nodes, 446 edges);
atuinsh/atuin @ 2535161e *.sql (69 files) 19 -> 7: all 12 valid SQLite
files clean; the 7 left are the invalid `ALTER TABLE ... DROP INDEX`
down migration and the 2 MySQL + 4 Postgres files, same ranges as
before; graph +2 Table nodes (the fts5 table and the table the trigger
error swallowed), +2 DEFINES, +3 USAGE lineage edges. No newly flagged
file. Control: helm @ 9b88efd6 indexes to an identical graph (12,251
nodes, 39,561 edges).
Registered like pascal: MANIFEST.md fork rows + notes (ABI tally
recounted, unchanged), the audit FORKS map, THIRD_PARTY.md, vendored
checksums. LICENSE stays upstream's MIT byte-identical.
Tests: parse_coverage carries the two SQLite tests from #2246 as written
(sqlite_valid_migration_not_flagged, red before this change, and the
sqlite_malformed_still_flagged control) and adds `key` columns, WITHOUT
ROWID, a WHEN trigger with INSERT OR IGNORE plus an fts5 table (asserted
as a Table), PRAGMA forms, and two controls: MySQL's KEY index still
clean, and ALTER TABLE ... DROP INDEX still flagged.
Distilled from #2246.
Co-authored-by: Kerf <kerfern@hotmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
247 lines
5.1 KiB
JavaScript
247 lines
5.1 KiB
JavaScript
import { paren_list, wrapped_in_parenthesis } from "../helpers.js";
|
|
|
|
export default {
|
|
|
|
// This is only used in create function statement, it is not needed to check
|
|
// the start tag match the end one. The usage of this syntax in other
|
|
// context is done by _dollar_string.
|
|
dollar_quote: () => /\$[^\$]*\$/,
|
|
|
|
create_function: $ => seq(
|
|
$.keyword_create,
|
|
optional($._or_replace),
|
|
$.keyword_function,
|
|
$.object_reference,
|
|
$.function_arguments,
|
|
$.keyword_returns,
|
|
choice(
|
|
$._type,
|
|
seq($.keyword_setof, $._type),
|
|
seq($.keyword_table, $.column_definitions),
|
|
$.keyword_trigger,
|
|
),
|
|
repeat(
|
|
choice(
|
|
$.function_language,
|
|
$.function_volatility,
|
|
$.function_leakproof,
|
|
$.function_security,
|
|
$.function_safety,
|
|
$.function_strictness,
|
|
$.function_cost,
|
|
$.function_rows,
|
|
$.function_support,
|
|
),
|
|
),
|
|
// ensure that there's only one function body -- other specifiers are less
|
|
// variable but the body can have all manner of conflicting stuff
|
|
$.function_body,
|
|
repeat(
|
|
choice(
|
|
$.function_language,
|
|
$.function_volatility,
|
|
$.function_leakproof,
|
|
$.function_security,
|
|
$.function_safety,
|
|
$.function_strictness,
|
|
$.function_cost,
|
|
$.function_rows,
|
|
$.function_support,
|
|
),
|
|
),
|
|
),
|
|
|
|
_argmode: $ => choice(
|
|
$.keyword_in,
|
|
$.keyword_out,
|
|
$.keyword_inout,
|
|
$.keyword_variadic,
|
|
seq($.keyword_in, $.keyword_out),
|
|
),
|
|
|
|
function_argument: $ => seq(
|
|
optional($._argmode),
|
|
optional($.identifier),
|
|
$._type,
|
|
optional(
|
|
seq(
|
|
choice($.keyword_default, '='),
|
|
$.literal,
|
|
),
|
|
),
|
|
),
|
|
|
|
function_arguments: $ => paren_list(
|
|
$.function_argument,
|
|
false,
|
|
),
|
|
|
|
_function_return: $ => seq(
|
|
$.keyword_return,
|
|
$._expression,
|
|
),
|
|
|
|
function_declaration: $ => seq(
|
|
$.identifier,
|
|
$._type,
|
|
optional(
|
|
seq(
|
|
':=',
|
|
choice(
|
|
wrapped_in_parenthesis($.statement),
|
|
// TODO are there more possibilities here? We can't use `_expression` since
|
|
// that includes subqueries
|
|
$.literal,
|
|
),
|
|
),
|
|
),
|
|
';',
|
|
),
|
|
|
|
_function_body_statement: $ => choice(
|
|
$.statement,
|
|
$._function_return,
|
|
),
|
|
|
|
_tsql_function_body_statement: $ => seq(
|
|
optional($.keyword_as),
|
|
$.keyword_begin,
|
|
optional($.var_declarations),
|
|
choice(
|
|
repeat($.statement),
|
|
repeat1(seq(
|
|
$.keyword_begin,
|
|
repeat($.statement),
|
|
$.keyword_end,
|
|
)),
|
|
),
|
|
$._function_return,
|
|
$.keyword_end,
|
|
),
|
|
|
|
function_body: $ => choice(
|
|
seq(
|
|
$._function_return,
|
|
';'
|
|
),
|
|
seq(
|
|
$.keyword_begin,
|
|
$.keyword_atomic,
|
|
repeat1(
|
|
seq(
|
|
$._function_body_statement,
|
|
';',
|
|
),
|
|
),
|
|
$.keyword_end,
|
|
),
|
|
seq(
|
|
$.keyword_as,
|
|
alias($._dollar_quoted_string_start_tag, $.dollar_quote),
|
|
optional(
|
|
seq(
|
|
$.keyword_declare,
|
|
repeat1(
|
|
$.function_declaration,
|
|
),
|
|
),
|
|
),
|
|
$.keyword_begin,
|
|
repeat1(
|
|
seq(
|
|
$._function_body_statement,
|
|
';',
|
|
),
|
|
),
|
|
$.keyword_end,
|
|
optional(';'),
|
|
alias($._dollar_quoted_string_end_tag, $.dollar_quote),
|
|
),
|
|
seq(
|
|
$.keyword_as,
|
|
alias(
|
|
choice(
|
|
$._single_quote_string,
|
|
$._double_quote_string,
|
|
),
|
|
$.literal
|
|
),
|
|
),
|
|
seq(
|
|
$.keyword_as,
|
|
alias($._dollar_quoted_string_start_tag, $.dollar_quote),
|
|
$._function_body_statement,
|
|
optional(';'),
|
|
alias($._dollar_quoted_string_end_tag, $.dollar_quote),
|
|
),
|
|
$._tsql_function_body_statement,
|
|
),
|
|
|
|
function_language: $ => seq(
|
|
$.keyword_language,
|
|
// TODO Maybe we should do different version of function_body_statement in
|
|
// regard to the defined language to match either sql, plsql or
|
|
// plpgsql. Currently the function_body_statement support only sql. And
|
|
// maybe for other language the function_body should be a string.
|
|
$.identifier
|
|
),
|
|
|
|
function_volatility: $ => choice(
|
|
$.keyword_immutable,
|
|
$.keyword_stable,
|
|
$.keyword_volatile,
|
|
),
|
|
|
|
function_leakproof: $ => seq(
|
|
optional($.keyword_not),
|
|
$.keyword_leakproof,
|
|
),
|
|
|
|
function_security: $ => seq(
|
|
optional($.keyword_external),
|
|
$.keyword_security,
|
|
choice($.keyword_invoker, $.keyword_definer),
|
|
),
|
|
|
|
function_safety: $ => seq(
|
|
$.keyword_parallel,
|
|
choice(
|
|
$.keyword_safe,
|
|
$.keyword_unsafe,
|
|
$.keyword_restricted,
|
|
),
|
|
),
|
|
|
|
function_strictness: $ => choice(
|
|
seq(
|
|
choice(
|
|
$.keyword_called,
|
|
seq(
|
|
$.keyword_returns,
|
|
$.keyword_null,
|
|
),
|
|
),
|
|
$.keyword_on,
|
|
$.keyword_null,
|
|
$.keyword_input,
|
|
),
|
|
$.keyword_strict,
|
|
),
|
|
|
|
function_cost: $ => seq(
|
|
$.keyword_cost,
|
|
$._natural_number,
|
|
),
|
|
|
|
function_rows: $ => seq(
|
|
$.keyword_rows,
|
|
$._natural_number,
|
|
),
|
|
|
|
function_support: $ => seq(
|
|
$.keyword_support,
|
|
alias($._literal_string, $.literal),
|
|
),
|
|
|
|
};
|