mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-15 13:56:34 +08:00
Martin VogelandKerf
d9b22e0b7f
Valid SQLite migration files were reported parse_partial because the
vendored SQL grammar (DerekStride/tree-sitter-sql @ 851e9cb257ba) rejects
SQLite syntax they use: PRAGMA, AUTOINCREMENT, a trigger's BEGIN ... END
body, a column named `key` after the first column, WITHOUT ROWID, an
unparenthesized trigger WHEN, INSERT OR IGNORE, and CREATE VIRTUAL TABLE
... USING fts5(...). Each became an ERROR node; the trigger's error region
in atuin's ai_session schema also swallowed the CREATE TABLE after it.
The `key` case is lexical: after a comma inside a column list the grammar
also accepts MySQL's `KEY name (cols)` index, so `key` lexes as that
keyword there and `key TEXT NOT NULL` became a malformed index.
Fix: sql becomes a self-maintained fork of the pin we already vendored,
following the pascal model. tools/tree-sitter-sql/ holds upstream's
grammar.js + grammar/ (blob-identical to the pin and to upstream main @
3c99273f9b32, where its corpus lives) and one patch,
patches/sqlite-dialect.patch, touching grammar/ only:
- PRAGMA [schema.]name [= value | (value)] (new pragma_statement)
- AUTOINCREMENT as keyword_auto_increment, upstream's own change
(c042cde1e2a6, 2026-11-08, after our pin)
- `key` accepted as a column name; the type keywords that take no
arguments are accepted as a MySQL index name so `KEY name (name)` stays
an index
- WITHOUT ROWID
- a trigger body as the existing block node, after an optional
unparenthesized WHEN (precedence keeps Postgres `WHEN (...) EXECUTE
FUNCTION` trees unchanged)
- INSERT OR {ABORT | FAIL | IGNORE | REPLACE | ROLLBACK}
- CREATE VIRTUAL TABLE [IF NOT EXISTS] name USING module(args), aliased
to create_table (new module_argument node), so the table is extracted
as a Table like any other
Upstream node types keep their shape: upstream's 511 corpus cases pass
unchanged; test/corpus/sqlite.txt adds 11 (9 fail on the unpatched
grammar, 2 controls pass on both). The vendored scanner.c is kept as is
(it carries our start_tag leak fix 0e61867d9, now listed under the
MANIFEST's "Local source patches"; upstream fixed the same leak after our
pin in ee2f84f9b857).
Provenance: without the patch, tree-sitter-cli 0.25.10 regenerates
upstream's shipped parser.c byte-for-byte except line 1 (upstream's
generator stamp has no version), and grammar.json, node-types.json and
tree_sitter/*.h byte-identical. Patched: three generations are
byte-identical (parser.c SHA-256 3c64e5c4...7887); ABI 15; states 30,622
-> 31,081, symbols 770 -> 782; 0 non-ASCII bytes, no string literal with
a space, one comment, the same call-position identifiers as before. Node
types only gain entries (pragma_statement, module_argument, four
keywords; create_table gains a `module` field).
Proof, indexed from scratch with isolated caches (main 6d34ca15 vs this
change): dani-garcia/vaultwarden @ 58a980f8 migrations/sqlite (116
files) parse_partial 3 -> 0, graph identical (303 nodes, 446 edges);
atuinsh/atuin @ 2535161e *.sql (69 files) 19 -> 7: all 12 valid SQLite
files clean; the 7 left are the invalid `ALTER TABLE ... DROP INDEX`
down migration and the 2 MySQL + 4 Postgres files, same ranges as
before; graph +2 Table nodes (the fts5 table and the table the trigger
error swallowed), +2 DEFINES, +3 USAGE lineage edges. No newly flagged
file. Control: helm @ 9b88efd6 indexes to an identical graph (12,251
nodes, 39,561 edges).
Registered like pascal: MANIFEST.md fork rows + notes (ABI tally
recounted, unchanged), the audit FORKS map, THIRD_PARTY.md, vendored
checksums. LICENSE stays upstream's MIT byte-identical.
Tests: parse_coverage carries the two SQLite tests from #2246 as written
(sqlite_valid_migration_not_flagged, red before this change, and the
sqlite_malformed_still_flagged control) and adds `key` columns, WITHOUT
ROWID, a WHEN trigger with INSERT OR IGNORE plus an fts5 table (asserted
as a Table), PRAGMA forms, and two controls: MySQL's KEY index still
clean, and ALTER TABLE ... DROP INDEX still flagged.
Distilled from #2246.
Co-authored-by: Kerf <kerfern@hotmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
scripts/ — the canonical leg entries
The doctrine (enforced, not advisory): a venue may provision a machine;
only a canonical leg script may exercise the product. Local CI, PR CI, the
dry run and the release all call the SAME files below — venues differ only in
host specs, architecture and inputs, never in what a leg does. Platform
differences (arm64 sanitizer flags, native Windows behavior, the Linux portable
binary) live INSIDE these scripts, once, never per-venue.
tests/test_venue_parity_contract.sh (run as Step 0j of every test leg) turns
any violation — inline harness logic in a workflow, a bypassed wrapper, a
missing --help — into a red build.
Every entry answers --help (authoritative, agent-oriented: modes, env, exit
codes) and rejects unknown flags with exit 2 + Please consult --help.
The legs
| leg | entry | what a run gives you |
|---|---|---|
| test | test.sh |
DEFAULT = the venue leg: static contracts (Step 0a–0r) + CLEAN ASan+UBSan build + all suites via the parallel harness + prod-binary guards. --suites a,b = iteration mode (incremental rebuild, subset, seconds). --tsan = the ThreadSanitizer leg. --contracts-only = the static contract steps alone (PR CI's contracts job for docs-only and non-product changes). CLANGARM64 gets CI's trap-UBSan flags automatically. |
| package wrappers | ci/test-package-wrappers.sh |
Go, npm, and PyPI runtime-set publication/lock suites on the current host. CI runs the same entry on Linux and Windows so both platform lock implementations gate. |
| build | build.sh |
CLEAN production runtime set (native executable + authenticated integration asset; --with-ui adds one content-addressed UI pack). ccache via env.sh makes repeats fast; CCACHE_COMPILERCHECK=content guarantees a hit is byte-identical to a cold compile — never stale. --version, STATIC=1, BUILD_DIR=. |
| lint | lint.sh |
clang-tidy + cppcheck + clang-format (+ no-skips policy). --ci = the CI gate set (no clang-tidy). Drives the same make targets as make lint/lint-ci. |
| smoke (unix) | smoke-local.sh |
Stages a full release fixture, serves it on a kernel-assigned port, runs smoke-test.sh (ALL phases incl. download/install/update E2E) inside a disposable HOME/XDG/TMP sandbox. ui variant makes a missing verified UI pack a FAILURE. CBM_SMOKE_ARTIFACT_DIR = smoke an extracted release artifact verbatim (release mode). |
| smoke (windows) | ../test-infrastructure/vm/vm-smoke.sh |
Same verified runtime-set contract on the real Windows VM, plus the user-PATH registry guard (prepare/verify/cleanup). |
| smoke-invariants | smoke-invariants.sh |
Production-path resilience battery (MCP handshake, all tools invocable, malformed-input handling, supervised crash/hang recovery) — no fixture server or install E2E. smoke.yml runs an explicitly seam-enabled build on the WIDEST source matrix; release artifacts remain seam-free and use the release-shaped smoke legs above. |
| soak | soak-legs.sh |
The release-gating soak SEQUENCE: quick then query-leak (the #581 detector — never reindexes, so RSS growth = query-path leak), each guarded by a completion-summary check. --legs quick for the ASan single-leg variant. Duration is per leg. |
Internal harnesses — never called directly by a venue (the contract forbids
it): smoke-test.sh (phases; wrappers provide fixture server + sandbox),
soak-test.sh (one soak run; soak-legs.sh provides the sequence + guards),
run-tests-parallel.sh (reached through test.sh), which sources
suite-failure-report.sh for its end-of-run failure summary (failure sites,
sanitizer report, running test, last lines) and for test-logs/failed/, the
logs of every suite that did not end green.
Conventions
- Exit codes: 0 = pass · 2 = usage error · 90 = guard (a run died without its completion summary — never counts as green) · anything else = the leg's real failure.
- Iteration is a flag, not a side-tool: the fast paths (
--suites,--legs) are modes of the SAME entry the gates run, so a dev loop can never drift from the venue behaviour. - Env sandboxing: the smoke wrappers neutralize every agent-config destination override; a smoke can never scribble on your real config.
Recommended workflows
- Iterating on a change:
scripts/test.sh --suites <suite>(seconds, incremental, same ASan+UBSan flags as the gate). List suites:build/c/test-runner --list-suites. Debugging a Windows-on-ARM trap: re-run withSANITIZE=for a plain build, or use the emulatedwin.sh ubsan-*pair for full diagnostics. - Before any push (the 3-OS ladder):
scripts/test.sh(macOS, full) →./test-infrastructure/run.sh full(Linux + TSan + smoke) →test-infrastructure/vm/win.sh test-par+guards+smoke-install(+soakwhen the change touches memory/daemon paths). - Concurrency-touching change: add
scripts/test.sh --tsanearly — the same leg CI gates on. - Release-shaped verification:
CBM_SMOKE_ARTIFACT_DIR=<extracted artifact> scripts/smoke-local.sh <binary> [ui]smokes exactly what would ship. - A leg is red in CI but green locally: first suspect environment shape,
not code — the preflights (
win.shautomatic;scripts/ci/preflight-docker.sh) andtest-infrastructure/README.md's residuals list cover the knowable differences.
See scripts/ci/README.md for the CI plumbing and
test-infrastructure/README.md for the venue map.