Every index is written with 64 KiB pages (sqlite_writer.c), but read-write
connections kept SQLite's default 2000 KiB page cache, which holds about 31
of those pages. A closure-repair delta purges and patches rows across the
node and edge tables and all of their indexes, so the transaction kept
spilling dirty pages into the WAL and rewriting them in place. A one-line
body change on a 294 MB django index wrote 5.2 GiB to the WAL (about 75k
in-place page rewrites) and 5.7 GiB of logical writes in total. Read-write
on-disk connections now use a 64 MiB cache (1024 pages, the budget bulk mode
already used), and end_bulk restores that instead of the 2000 KiB default.
Query connections keep the default.
cbm_pipeline_publish_staged also forced synchronous=FULL on the private
staging database, so every metadata and FTS commit fsynced the WAL. A crash
before the atomic rename discards the stage, and
cbm_store_seal_for_atomic_publish() still raises the connection to FULL for
the TRUNCATE checkpoint and DELETE-mode switch that make the published file
durable, so those per-commit fsyncs protected nothing. The commits now run at
the store's NORMAL level, which SQLite documents as corruption-safe under
WAL. No corruption or atomicity guarantee changes: the live database is
still only replaced by a sealed, integrity-checked file via rename.
Measured on macOS, main vs this change, same machine and corpora:
- django one-line edit (closure repair): WAL writes 5.2 GiB -> 0.52 GiB,
logical writes 5.76 GiB -> 1.08 GiB, WAL fsyncs 13 -> 8
- django full index: WAL writes 58 MiB -> 47.5 MiB, WAL fsyncs 9 -> 2
- php (65 MB index) one-line edit: WAL writes 82.5 MiB -> 32.3 MiB, spill
rewrites 468 -> 0
- kubernetes (1.7 GB index) full index: WAL writes 669 MiB -> 390 MiB, WAL
fsyncs 13 -> 6
The comments that put a healthy WAL at ~4 MiB assumed 4 KiB pages; at 64 KiB
the 1000-page autocheckpoint is ~64 MiB, still well under the 256 MiB
journal_size_limit.
Tests: store_pragmas/store_role_pragmas_issue1419 pins the effective pragmas
of each store role (read-write, bulk, seal, query), and
pipeline_semantic_manifest_repro/
pipeline_staged_publish_commits_below_full_sync_issue1419 reads the effective
synchronous level from a WAL hook on every connection that a full index plus
a closure-repair reindex opens, and requires that no commit runs at FULL.
Refs #1419
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel
eb99608357