src/pipeline/pipeline_delta.c now makes delta_batch_contains_edge() compare decoded local_name values when source_qn, target_qn, and type identify an IMPORTS candidate. This matches graph-buffer and SQLite uniqueness, so preserving METHODS_WITH_BODY no longer suppresses REF_PREFIX when both unchanged imports point to a recreated FastAPI.openapi target.
The identity check retains existing behavior for every non-IMPORTS edge. Exact JSON matches avoid parsing; JSON decoding occurs only after an IMPORTS endpoint/type collision, leaving the existing delta scan complexity unchanged and adding only property-sized transient memory.
tests/test_pipeline.c covers cbm_pipeline_file_delta_add_preserved_inbound_edges() with two local_name-distinct IMPORTS edges to one target. tests/test_graph_buffer.c verifies cascade deletion removes both dedup keys before target recreation and reinsertion.
Validated with ASan/UBSan: the focused pipeline and graph-buffer regressions pass, and the complete incremental suite passes 162/162, including incr_formatter_run, incr_db_deleted_recovery, and incr_accuracy_vs_full.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Merge upstream/main at 2469ecc3 into integrate-upstream-split-search-tools while retaining branch-specific incremental indexing, MCP API consolidation, and graph extraction behavior.
The staged tree adds upstream's Mojo and Perl grammar support, issue reproduction suites, Graph UI changes, Windows smoke coverage, TOON graph output, subprocess supervision, and Laravel facade route handling. Conflict resolutions in src/pipeline/httplink.c and pass_httplinks.c retain dynamic growth and IMPORTS edge identity; scripts/smoke-test.sh and scripts/soak-test.sh retain hermetic cache isolation.
Validation against the merge plus the follow-up working-tree reconciliation: ASan/UBSan MCP suite 216/216 passed. Follow-up semantic and concurrency fixes remain intentionally unstaged for the next focused commit.
Replace the database-loss recovery node-count tolerance with the shared canonical graph oracle. The test selects eager derived refresh through the registered incremental configuration policy so its incremental baseline and clean recovery rebuild have equivalent semantics.
Prune nested Folder nodes that lose their last contained file or child folder before graph-wide incremental post-passes. Run the scan only when classification reports deleted files, allocate temporary path storage to the exact folder count, free it on every exit path, and fail closed on graph errors.
Add focused nested-folder lifecycle coverage and validate the complete ordered incremental suite under native and ASan/UBSan runners (161/161 each), with no canonical drift or sanitizer diagnostic. Changed-line formatting, diff hygiene, and source-safety checks also pass.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Replace per-node SQLite upserts during graph-buffer publication with a set-oriented temporary-table batch and deterministic ID remapping. Keep the batch inside the caller-owned replacement transaction so sibling projects remain intact and every failure rolls back the full project replacement.
Finalize all temporary statements, clear connection-scoped staging state before reuse, free node and ID arrays on every success and failure path, and reject invalid temporary IDs before map access. Add transaction rollback coverage and strengthen graph-buffer persistence assertions.
Measured the targeted span at 89 ms for 13,455 nodes in the canonical route-handler self-dogfood scenario. The scenario remained canonically equivalent with cleanup green; synchronous rank refresh remains the dominant end-to-end blocker.
Validation: product build; sanitizer build; source-safety check; store node suite (114 passed); graph-buffer suite (67 passed); pipeline suite (349 passed); focused C source-frontier canonical canaries (2 passed); route-handler canonical self-dogfood.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add a transaction-scoped edge batch helper so replacement graph flushes can reuse store batching without nesting BEGIN/COMMIT inside the active flush transaction.
Use a temp-table + set-oriented INSERT ... SELECT ... ON CONFLICT path for high-fanout direct-ID edges, preserving json_patch merge semantics and reusing the existing ST_DELTA_EDGE_BULK_MIN threshold.
Switch cbm_gbuf_flush_to_store from per-edge INSERT ... RETURNING to the new transaction-scoped batch body after remapping valid edge endpoints. Orphan-edge skipping and flush rollback/bulk-mode boundaries stay unchanged.
Measured on the focused FastAPI single-file canary: gbuf edge insertion dropped from 5232ms to 289ms, replacement flush from 8099ms to 3387ms, and total index_repository from 17356ms to 12393ms.
Validation: git diff --check; scripts/check-source-safety.sh; final ASan/UBSan rebuild; store_edges 26 passed; graph_buffer 67 passed; store_bulk 3 passed; focused incremental canary passed; product build passed. Full incremental suite also passed 161 tests on the pre-final allocation micro-change binary.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Distills PR #770 (graph-buffer dedup key) and completes the edge-uniqueness
contract it left partial: the contract lives in THREE places, and changing
only the buffer ships databases that violate their own UNIQUE constraint
(PRAGMA integrity_check: non-unique entry in sqlite_autoindex_edges_1).
A single 'import { A, B } from ./lib' produced ONE IMPORTS edge: the graph
buffer dedups edges on (source_id, target_id, type) and merge-replaces
properties on collision, so the second symbol silently overwrote the first.
pass_calls.c, pass_usages.c, pass_semantic.c and pass_lsp_cross.c parse one
local_name per IMPORTS edge for cross-file resolution, so the dropped
symbol's calls failed to resolve too — not just "who imports X" queries.
Changes, kept in sync across all three sites:
- graph_buffer.c (from #770): make_edge_key() folds local_name into the
dedup key for IMPORTS edges only; all three key call-sites updated.
Hardened beyond #770: EDGE_KEY_BUF bumped to 256 and oversized
local_names are re-keyed with an FNV-1a hash of the full name instead
of being silently truncated (two long names sharing a prefix must not
collide back into one edge).
- store.c: edges gains local_name_gen, a VIRTUAL generated column
(IMPORTS -> coalesce(json_extract(properties,'$.local_name'),''),
else '' — NOT NULL because NULLs never conflict in a UNIQUE index);
uniqueness widened to UNIQUE(source_id, target_id, type,
local_name_gen) and the insert upsert's conflict target matches.
init_schema probes pre-#768 DBs (no local_name_gen) and fails the
open: SQLite cannot ALTER a table constraint in place, and an
unopenable DB already takes the existing repair path — full index
deletes + rebuilds, artifact import refuses and falls back to a
reindex. Read-only query opens skip init_schema and keep working.
- sqlite_writer.c/.h: dump DDL matches the widened schema; the hand-built
sqlite_autoindex_edges_1 comparator and entry builder include the
local_name column; CBMDumpEdge carries local_name extracted via real
JSON parsing (yyjson) so index entries match json_extract's unescaped
values exactly, per the idx_edges_url_path precedent.
- artifact.h: CBM_ARTIFACT_SCHEMA_VERSION 1 -> 2 so old binaries refuse
artifacts carrying the widened schema (their 3-column conflict target
can no longer prepare against it).
Tests (reproduce-first, all red on the unfixed code):
- gbuf tier: multi-symbol import -> 2 IMPORTS edges; long-local_name
truncation guard.
- store tier: distinct local_name coexists as 2 rows, same local_name
still upserts, non-IMPORTS dedup unchanged.
- writer tier: dumped DB passes integrity_check with 2 sibling imports
and exposes matching local_name_gen values.
- end-to-end: TS fixture through the real pipeline -> 2 queryable
IMPORTS edges AND integrity_check ok; with only the buffer half of
the fix this test still fails on integrity_check, proving the schema
half is required.
Closes#768.
Co-authored-by: Alexandros Pappas <11921291+apappas1129@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Check every store operation in cbm_gbuf_flush_to_store, keep project replacement inside a transaction, create indexes before commit, and roll back on failure instead of silently committing partial work.
Add a regression that exercises a failed nested BEGIN and verifies the existing project graph is preserved. Validated with source-safety, focused graph_buffer and depindex suites, product build, and the MCP self-dogfood one_source_file scenario.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Keep label/name secondary indexes in sync when an existing qualified-name node is promoted by upsert or merge. The complexity pass relies on Function label lookups, so stale secondary buckets could make fresh full builds omit derived recursive and transitive_loop_depth properties for promoted functions.
Add graph_buffer regressions for direct upsert and merge collisions so the old label/name buckets are emptied and the new buckets contain the existing node once.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Make duplicate graph-buffer upserts and worker merges choose richer source spans for definition-like nodes, so full and containment rebuilds no longer depend on worker order for declaration-vs-definition locations.
Pass C/C++ macro extraction mode through extraction contexts instead of relying on process-global pipeline state, preventing MCP runs with mixed modes from leaking macro-node policy across pipelines.
Recompute configlink-owned CONFIGURES edges deterministically by clearing only key_symbol/dependency_import strategy edges, preserving direct CONFIGURES links, and sizing candidate arrays from discovered graph counts instead of fixed caps. Reuse cbm_path_base for manifest basename handling and make it recognize both POSIX and Windows separators.
Validation: git diff --check; ASan/UBSan test-runner rebuild /private/tmp/cbm-pan4-20260702T112332Z-build-test-runner-after-basename.log; str_util 65/65 and configlink 9/9 /private/tmp/cbm-pan4-20260702T112531Z-affected-strutil-configlink.log; product build /private/tmp/cbm-pan4-20260702T112545Z-cbm-build-after-configlink.log; MCP self-dogfood one-source canonical equality /private/tmp/cbm-pan4-20260702T112614Z-one-source-mcp-after-configlink.json; graph_buffer 63/63 and extraction 256/256 /private/tmp/cbm-pan4-20260702T112822Z-affected-graphbuffer-extraction.log; incremental 161/161 /private/tmp/cbm-pan4-20260702T112851Z-incremental-suite.log.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Build deleted-node sets before mutating graph indexes so allocation or invariant failures return before partial deletes.
Reuse existing node_by_id keys as the temporary deleted-set ownership transfer instead of allocating duplicate ID strings, and size path lookup tables to the number of paths instead of over-reserving.
Add batch delete cascade coverage for cbm_gbuf_delete_by_paths and assert delete_by_label return values.
Validation: make -f Makefile.cbm build/c/test-runner; CBM_ONLY_SUITE=graph_buffer ./build/c/test-runner; CBM_ONLY_SUITE=incremental ./build/c/test-runner; bash scripts/check-source-safety.sh; git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add graph-buffer structural invariant validation before SQLite writer handoff so invalid live node IDs, stale node_by_id mappings, missing edge endpoints, and inconsistent edge indexes fail before temp DB creation instead of being silently remapped or dropped.
Add focused graph-buffer tests for valid graphs and missing endpoint rejection, adjust the pipeline-path isolation fixture to use the same shared ID allocation contract as production worker buffers, and add a direct sqlite_writer replay for node_vectors and token_vectors payloads.
Validation: git diff --check; make -f Makefile.cbm build/c/test-runner; bash scripts/check-source-safety.sh; CBM_ONLY_SUITE=graph_buffer ./build/c/test-runner (58 passed); CBM_ONLY_SUITE=sqlite_writer ./build/c/test-runner (8 passed); CBM_ONLY_SUITE=pipeline ./build/c/test-runner (222 passed); CBM_ONLY_SUITE=incremental ./build/c/test-runner (160 passed).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Expose the graph-buffer dump publish fault hook name from graph_buffer.h so tests and implementation share one constant instead of repeating the environment variable string.
Add a small pipeline incremental regression test that forces a dump failure before atomic replacement, asserts the incremental run returns failure, and verifies the previous DB remains unchanged with the new function unpublished.
Validation: git diff --check && make -f Makefile.cbm build/c/test-runner && CBM_ONLY_SUITE=graph_buffer ./build/c/test-runner && CBM_ONLY_SUITE=pipeline ./build/c/test-runner && bash scripts/check-source-safety.sh
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add a narrow test-only fault hook at the graph-buffer dump publish boundary, after temp database verification and before atomic replacement. The hook is enabled only by CBM_TEST_FAIL_GBUF_DUMP_BEFORE_REPLACE and logs through cbm_log_error without stdout output.
Add a regression test that publishes an existing DB, injects a failure before replacement during a second dump, and verifies the original graph remains queryable while the new graph is not published.
Validation: git diff --check && make -f Makefile.cbm build/c/test-runner && CBM_ONLY_SUITE=graph_buffer ./build/c/test-runner && bash scripts/check-source-safety.sh
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Stabilize several graph-producing paths and add stronger regression coverage before the incremental redesign work continues.
Key changes:
- resolve call import maps from resolved graph-buffer IMPORTS edges so re-exported targets are reused consistently
- make semantic, similarity, complexity, route/section upserts, and resolver scoring more deterministic across full/incremental and worker-order differences
- propagate incremental phase/persist failures instead of continuing toward publication after failed phases
- add canonical graph diff helpers and stricter incremental artifacts for full-vs-incremental comparison
- add source-safety lint coverage for MCP stdout, unsafe string APIs, and new raw env/fs API usage
- replace touched unsafe string copies with bounded or project-wrapper-based forms
Validation:
- make -f Makefile.cbm build/c/test-runner
- CBM_ONLY_SUITE=pipeline ./build/c/test-runner: 217 passed
- CBM_ONLY_SUITE=graph_buffer ./build/c/test-runner: 55 passed
- CBM_ONLY_SUITE=mcp ./build/c/test-runner: 117 passed
- CBM_ONLY_SUITE=simhash ./build/c/test-runner: 24 passed
- CBM_ONLY_SUITE=parallel ./build/c/test-runner: 21 passed
- CBM_ONLY_SUITE=watcher ./build/c/test-runner: 57 passed
- bash scripts/check-source-safety.sh: OK
Known remaining:
- CBM_ONLY_SUITE=incremental still fails incr_accuracy_vs_full: 159 passed, 1 failed. Current artifacts show stale IMPORTS/CALLS/USAGE targets in unchanged importers and downstream SEMANTICALLY_RELATED drift; this is tracked in the forensic notes as a fallback-first incremental redesign task.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Rebuild route-derived edges during incremental indexing and clear only derived HANDLES/DATA_FLOWS rows that are owned by the route reconciliation pass.
Make infra route matching order-independent by considering every matching handler route, and make prefix-route bridging inspect every registrar edge while keeping diagnostics on the existing logger.
Reuse the full-index file/folder structure helper for changed files so incremental updates restore File metadata and CONTAINS_FILE/CONTAINS_FOLDER edges consistently.
Add graph-buffer selective edge deletion and regression coverage for route matching, prefix bridging, and full-vs-incremental edge accuracy.
Tests: build/c/codebase-memory-mcp; CBM_ONLY_SUITE=graph_buffer build/c/test-runner; CBM_ONLY_SUITE=infrascan build/c/test-runner; CBM_ONLY_SUITE=pipeline build/c/test-runner; CBM_ONLY_SUITE=incremental build/c/test-runner; CBM_WORKERS=1 CBM_ONLY_SUITE=incremental build/c/test-runner.
Caveat: incremental remains below the >=10x speedup gate for default enablement; full rebuild remains the safe default until the delta design is redesigned and measured.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
The post-dump verify (B1 mitigation, 63de254) used the strict
cbm_store_check_integrity, which treats a non-absolute root_path (e.g. a
relative repo_path like ".") as bad_root_path → the verify DELETED a valid DB.
Indexing with a relative path therefore failed (status=error) — caught by
self-indexing the repo (cbm cli index_repository repo_path="."); the same
index with an absolute path succeeded cleanly (14723 nodes).
Switch the verify to cbm_store_check_integrity_full with the path_only
out-param: a path-only defect (cosmetic root_path) is RETAINED, consistent
with #557 resolve_store — only genuine structural corruption (>5 project
rows) is deleted. Verified: relative-path index now returns status=indexed;
absolute-path index unchanged.
test_graph_buffer gbuf_dump_relative_root_path_retained locks this in.
graph_buffer 50 pass.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Two regression probes that isolate the non-deterministic .db corruption
(B1) to the extraction-population hop, each with a decisive negative:
- test_sqlite_writer.c sw_scale_root_path_integrity: variable-length
properties_json (20-1500B, stressing every page boundary) through
cbm_write_db. Clean 10/10 -> rules out the variable-record
page-boundary hypothesis in the writer.
- test_graph_buffer.c gbuf_dump_pipeline_path_integrity: drives
cbm_gbuf_dump_to_sqlite with a populated + merged gbuf (variable heap
properties_json + cbm_gbuf_merge ID remap, 15k nodes / 50k edges).
Clean 10/10 -> rules out the gbuf->dump handoff (build_dump_nodes /
temp_to_final).
Both confirm the writer and the gbuf->dump handoff are correct on
realistic synthetic data; B1 must originate in real tree-sitter
extraction populating the gbuf (needs real source data to reproduce).
test_main.c: add CBM_ONLY_SUITE fast-filters for graph_buffer + pagerank.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Enforce WarningsAsErrors: '*' with zero suppressions of entire check categories.
All non-FP warnings fixed at source; true false positives get per-instance NOLINT
with reasoning. Fixes include: braces around statements (1679), isolate declarations
(75), explicit casts for multilevel pointer conversions, missing includes, function
parameter documentation via NOLINT, and proper NOLINTBEGIN/NOLINTEND nesting.
Complete C implementation of the indexing pipeline (src/), parallel worker
pool, graph buffer with merge support, SQLite writer, and 1893-test suite.
Linter setup: zero warnings from clang-tidy (all checks enabled, no NOLINT
suppressions), cppcheck, and clang-format. All issues fixed at source —
proper headers for external linkage, GROW_ARRAY macro restructured to
eliminate type parameter, null-deref paths guarded, named intermediates
for suspicious-argument checks.
Andrew Hundt