The FAST export path (the watcher/incremental auto-update) read the
raw main-file bytes of a live WAL-mode store: committed rows still in
the -wal were missing entirely and mid-checkpoint reads produced torn
snapshots (field artifact: four short indexes + btreeInitPage error 11
across ~12 nodes-table pages). Import then installed the torn file
byte-for-byte — the integrity check only sanity-checked the projects
table.
- Export: both quality levels now snapshot via VACUUM INTO (the
consistent copy); only the index-stripping remains BEST-only, FAST
keeps zstd-3. The fast path pays a full-copy VACUUM INTO now —
correctness over speed; a torn team-shared artifact costs far more.
- Import: new cbm_store_check_integrity_deep (shallow check + PRAGMA
quick_check) — used only on this rare path, hot-path callers keep
the cheap shallow check.
Reproduce-first: artifact_fast_export_snapshots_live_wal_store
exports FAST while the writer connection is open — RED on the old
code with 0 of 60 nodes surviving the round-trip (all rows lived in
the WAL); GREEN now. store_deep_integrity_detects_page_corruption
pins the import guard both directions (healthy passes, zero-banded
pages refused).
Closes#895
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The import path allocated the destination from the plaintext original_size
field in artifact.json and passed that value to the decoder truncated
through int. A large real database already wraps that cast (a full-kernel
index is ~14 GB), and a doctored field could additionally desync the decoder
capacity from the actual buffer. Read the decompressed size from the zstd
frame's own content-size header, require the metadata field to match it, cap
the total, and run the decoder with a size_t capacity equal to the
allocation. cbm_zstd_decompress now takes size_t lengths and returns int64_t.
Adds a regression test that a mismatched original_size is rejected.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
gitattributes apply left to right and the binary macro expands to
-diff -merge -text, so the generated line
graph.db.zst merge=ours binary
leaves the merge attribute UNSET (git check-attr merge reports
"unset"), and concurrent artifact refreshes produce a binary merge
conflict instead of auto-resolving -- the exact failure this file is
generated to prevent. Reordering to
graph.db.zst binary merge=ours
makes check-attr report merge: ours while keeping -diff/-text intact.
Verified on a real repository: before the reorder git check-attr merge
-- .codebase-memory/graph.db.zst printed "unset"; after, "ours".
Extends artifact_gitattributes_created to pin the ordering so the
regression cannot return.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Alex <plosceac.alexandre@gmail.com>
artifact.c shells out to git twice via cbm_popen with the repo path interpolated
straight into the command:
git -C '%s' rev-parse HEAD 2>/dev/null (git_head_hash)
git -C '%s' config merge.ours.driver true 2>/dev/null (ensure_gitattributes)
Both used SINGLE quotes with NO validation of repo_path. cmd.exe does not honor
single quotes, so on Windows a repo path containing a space broke argument grouping
(the artifact commit hash and merge-driver config silently failed for any spaced repo
path), and an embedded quote or shell metacharacter could break out of the intended
argument. This was the only unvalidated git shell interpolation left in the tree.
Different bug class from the CreateProcess argv-quoting fix (which re-parses a single
command STRING back into argv); this is popen/shell command-string quoting — hence a
separate, atomic PR per review.
Fix: validate repo_path with cbm_artifact_repo_path_is_shell_safe() and switch to
double quotes + a per-platform null device, matching git_context.c (the best-hardened
git shell-out): cbm_validate_shell_arg rejects quote / backslash / substitution
metacharacters, and on Windows we also reject the cmd.exe expansion metacharacters
% ! ^. Double quotes are honored by both POSIX sh and cmd.exe, so a legitimate spaced
repo path now works — the concrete regression the single-quote form caused.
The validator is exposed (artifact.h) and guarded by a new test_artifact.c suite:
plain + spaced paths accepted; quote / ; / $() / backtick / pipe injection rejected;
the cmd.exe metacharacters % ! ^ rejected on Windows (allowed on POSIX, where they are
literal inside double quotes).
Refs: #881
Signed-off-by: Flipper <jacobphilipp@ymail.com>
- Declare cbm_artifact_export_last_error() in artifact.h and drop the ad-hoc
forward declarations in pipeline.c and test_artifact.c so there is a single
shared declaration.
- Document that dump_and_persist_hashes now intentionally propagates a failed
artifact export as a pipeline error (previously silently ignored), so the
behavior change is recorded in-tree.
- Use MoveFileExA(MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH) for the
Windows replace in write_file_atomic instead of unlink+rename; it replaces
the destination atomically (credit @Ayush7Ranjan, from #492).
Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Add .codebase-memory/graph.db.zst — a zstd-compressed knowledge graph
artifact that can be committed to the repo. Teammates bootstrap from
the artifact instead of running a full reindex from scratch.
- Vendor zstd 1.5.7 (amalgamated build) for 8-13:1 compression
- Two-tier export: zstd -9 + index stripping for explicit index,
zstd -3 for watcher/incremental auto-updates
- Import: decompress → integrity check → auto-recreate indexes
- Bootstrap in handle_index_repository: when no local DB exists but
artifact is present, import first then run incremental
- Auto-create .gitattributes with merge=ours to prevent conflicts
- Fix: add missing idx_edges_url_path to create_user_indexes and
url_path_gen generated column to init_schema
- 13 new tests (5 zstd wrapper + 8 artifact round-trip/edge cases)
Martin Vogel