Add additive metadata tables and indexes for future staged exact-delta indexing: generations, file state, node/edge owners, symbol exports, import refs, and derived view freshness.
Keep this slice behavior-neutral. Normal store open migrates the schema, query-only open still does not create missing DBs, and writer-produced DBs remain minimal but migrate through cbm_store_open_path().
Validation: make -f Makefile.cbm build/c/test-runner; CBM_ONLY_SUITE=store_nodes ./build/c/test-runner; CBM_ONLY_SUITE=sqlite_writer ./build/c/test-runner; CBM_ONLY_SUITE=tool_consolidation ./build/c/test-runner; CBM_ONLY_SUITE=store_search ./build/c/test-runner; CBM_ONLY_SUITE=store_bulk ./build/c/test-runner; CBM_ONLY_SUITE=incremental ./build/c/test-runner; bash scripts/check-source-safety.sh; git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add a bounded two-thread sqlite_writer regression that starts independent cbm_write_db calls together and verifies both generated databases with SQLite integrity, root_path, node count, and edge count checks.
This guards the reentrant sort-context fix without relying on sleeps, network, or pipeline state. TSan remains the separate race-detection gate.
Validation: make -f Makefile.cbm build/c/test-runner; CBM_ONLY_SUITE=sqlite_writer ./build/c/test-runner; bash scripts/check-source-safety.sh; git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add graph-buffer structural invariant validation before SQLite writer handoff so invalid live node IDs, stale node_by_id mappings, missing edge endpoints, and inconsistent edge indexes fail before temp DB creation instead of being silently remapped or dropped.
Add focused graph-buffer tests for valid graphs and missing endpoint rejection, adjust the pipeline-path isolation fixture to use the same shared ID allocation contract as production worker buffers, and add a direct sqlite_writer replay for node_vectors and token_vectors payloads.
Validation: git diff --check; make -f Makefile.cbm build/c/test-runner; bash scripts/check-source-safety.sh; CBM_ONLY_SUITE=graph_buffer ./build/c/test-runner (58 passed); CBM_ONLY_SUITE=sqlite_writer ./build/c/test-runner (8 passed); CBM_ONLY_SUITE=pipeline ./build/c/test-runner (222 passed); CBM_ONLY_SUITE=incremental ./build/c/test-runner (160 passed).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Two regression probes that isolate the non-deterministic .db corruption
(B1) to the extraction-population hop, each with a decisive negative:
- test_sqlite_writer.c sw_scale_root_path_integrity: variable-length
properties_json (20-1500B, stressing every page boundary) through
cbm_write_db. Clean 10/10 -> rules out the variable-record
page-boundary hypothesis in the writer.
- test_graph_buffer.c gbuf_dump_pipeline_path_integrity: drives
cbm_gbuf_dump_to_sqlite with a populated + merged gbuf (variable heap
properties_json + cbm_gbuf_merge ID remap, 15k nodes / 50k edges).
Clean 10/10 -> rules out the gbuf->dump handoff (build_dump_nodes /
temp_to_final).
Both confirm the writer and the gbuf->dump handoff are correct on
realistic synthetic data; B1 must originate in real tree-sitter
extraction populating the gbuf (needs real source data to reproduce).
test_main.c: add CBM_ONLY_SUITE fast-filters for graph_buffer + pagerank.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Adds sw_scale_root_path_integrity: writes a ~fastapi-scale DB in ISOLATION
(20k nodes / 200k unique edges) via cbm_write_db, then asserts root_path
round-trips exactly, PRAGMA integrity_check == "ok", and node/edge counts.
Purpose: isolate whether the B1 custom-writer corruption reproduces without
the pipeline. Result: PASSES 3/3 — the writer is sound at scale in isolation
(sw_minimal_data covers the tiny case). This rules out the writer's
serialization as the B1 cause and points at the pipeline/parallel handoff
(a data race — to be confirmed with TSan). Also a permanent regression guard
for the custom B-tree writer at scale.
Edges generated with unique (source,target) pairs to respect the table's
UNIQUE(source_id, target_id, type) constraint. Adds sqlite_writer to the
CBM_ONLY_SUITE filter.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
The hand-rolled dump writer stored index cells fully inline regardless of payload size. SQLite's format spills index payloads above the max-local threshold (16,422 bytes at the 64 KB page size) to overflow pages, so a long key written inline makes the reader interpret key bytes as an overflow page number: PRAGMA integrity_check reported 'invalid page number 0x43654C6C' (ASCII from the key text) in idx_nodes_name on elasticsearch (very long Section names), and name lookups on that index silently returned nothing. write_index_btree now rewrites oversized cells (varint + local prefix + overflow chain via write_overflow_pages) before page building; promoted separators carry their chains with them. Reproduce-first: sw_long_index_keys_overflow (20 KB name; integrity_check failed pre-fix, ok post-fix). Also escapes the remaining raw text interpolations in DATA_FLOWS route props and HANDLES handler QNs (the ~51-edge malformed tail after 6c04ab7).
When a single record payload exceeds 65KB (max_local=65501), the writer
now splits it across overflow pages per SQLite's B-tree spec instead of
underflowing the content_offset and corrupting memory.
Closes#139, closes#187.
Based on #175 by dLo999.
Co-Authored-By: Dustin Obrecht <dustin@kurtnoble.com>
Major quality improvement: replace random index vectors with pre-trained
Nomic nomic-embed-code token embeddings. Tokens like 'error' and 'exception' now start
with similar vectors (learned from millions of code repos) instead of
arbitrary random projections. Co-occurrence enrichment adds project-specific
context on top.
Architecture:
- vendored/nomic/code_vectors.bin: 37.7MB raw int8 vectors
- vendored/nomic/code_vectors_blob.S: assembler .incbin (instant build)
- vendored/nomic/code_vectors.h: extern declarations + pretrained_vec_at()
- vendored/nomic/code_tokens.h: 40856 token strings (575KB)
- semantic.c: cbm_sem_random_index() now looks up pretrained vectors first,
falls back to sparse random for unknown tokens
- CBM_SEM_DIM raised from 256 to 768 to match Nomic nomic-embed-code
Also: RRI (Reflective Random Indexing), code pattern vocabulary injection,
120+ abbreviation expansions, callee/caller/body token enrichment,
label filter (Function/Method/Class only) in vector search SQL.
Binary size: 136MB → 169MB (+33MB from embedded vectors).
Search quality: keyword queries return relevant error-handling functions.
Domain-specific keyword queries return the expected functions.
Vector search pipeline (end-to-end):
- store.c: cbm_cosine_i8 custom SQLite function (int8 dot product with
sqrt magnitude), registered at store open time
- store.c: cbm_store_vector_search() builds merged query vector from
keywords via base RI vectors (xxHash), int8-quantizes, runs SQL query
with JOIN nodes ON node_id = id for metadata
- mcp.c: search_graph gains semantic_query param (array of strings),
returns semantic_results alongside regular results
- sqlite_writer.c: fixed build_vector_record to include node_id in
record body (matches build_node_record pattern)
- mcp.c: use yyjson_mut_obj_add_strcpy (not add_str) to avoid
use-after-free when vector results are freed before JSON serialization
Quality note: base RI vectors at query time don't capture co-occurrence
enrichment — results are moderate quality. Storing enriched token vectors
would improve this significantly (future work).
Enforce WarningsAsErrors: '*' with zero suppressions of entire check categories.
All non-FP warnings fixed at source; true false positives get per-instance NOLINT
with reasoning. Fixes include: braces around statements (1679), isolate declarations
(75), explicit casts for multilevel pointer conversions, missing includes, function
parameter documentation via NOLINT, and proper NOLINTBEGIN/NOLINTEND nesting.
Complete C implementation of the indexing pipeline (src/), parallel worker
pool, graph buffer with merge support, SQLite writer, and 1893-test suite.
Linter setup: zero warnings from clang-tidy (all checks enabled, no NOLINT
suppressions), cppcheck, and clang-format. All issues fixed at source —
proper headers for external linkage, GROW_ARRAY macro restructured to
eliminate type parameter, null-deref paths guarded, named intermediates
for suspicious-argument checks.
Andrew Hundt