Route scoped-LSP exact scratch publishes to the existing full-reindex fallback when the scoped planner cannot prove active/full graph equivalence. FastAPI probes showed the no-importer shortcut was faster but produced divergent IMPORTS/CALLS/THROWS edges, so this commit preserves correctness and records the remaining performance blocker.
Also build scratch package maps for overlay/exact delta routes to match full-pipeline resolver context, add opt-in inbound-edge debug logging through cbm_log_debug, and update Python route-decorator tests to expect the guarded full route while retaining fresh-rebuild equality checks.
Validation: focused pipeline suite 335 passed (/private/tmp/cbm-test-pipeline-scoped-gap-kind-fix-20260705T.log); source-safety passed; git diff --check passed; sandbox full suite reached 6391 passed with 13 known HTTP listen failures plus one stale assertion fixed here; escalated HTTP suite 28 passed (/private/tmp/cbm-test-httpd-scoped-gap-kind-fix-20260705T.log). No tag created.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add an internal frontier no-op mask for exact file-delta batches so graph-equal frontier files can prove coverage without recursively expanding or being rewritten. Reuse the existing preserved-inbound-edge helper for scoped changed files and keep the old batch planner/apply APIs as wrappers for non-scoped paths.
Focused validation: ASan/UBSan test-runner build passed, pipeline suite passed with 335 tests, source-safety passed, and product build passed. Synthetic python_reexport exact matrix passed. Real FastAPI routing.py still falls back with inbound_edges_require_full, so default-readiness and the broad FastAPI speed blocker remain open.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Prevent unsafe scoped overlay/containment publication for cross-LSP frontier-too-large edits until scoped parity is proven. FastAPI/Python showed active-overlay and containment mismatches for routing.py; this routes that class through the existing full rebuild path, preserving correctness while leaving performance work open.
Also suppress weak suffix fallback for Python super().__init__ when LSP resolved an external base, and recompute scoped complexity derived fields for changed in-scope nodes instead of preserving stale recursive/transitive loop-depth values.
Validation: git diff --check; source-safety; ASan/UBSan test-runner build; CBM_ONLY_SUITE=pipeline 333 passed; production build/sign; isolated FastAPI oracle passed with full fallback and canonical equality.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Expose when exact incremental frontier telemetry is capped so index_repository callers can distinguish a known affected-path count from a frontier that hit incremental_exact_max_affected_paths. Preserve existing publish routing and response fields while adding affected_paths_limit and affected_paths_truncated only when available.
Also centralize planner fallback reporting and cover oversized inbound-frontier fallback metadata in the pipeline suite.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Expose background overlay compaction lifecycle through index_status so long-lived MCP clients can distinguish idle, running, and finished workers without relying on internal join helpers.
Add focused MCP regression coverage for the finished-worker state after background compaction completes, and tighten local project-name snapshots to use checked snprintf handling.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Allow long-lived MCP servers to start a later overlay compaction pass after a prior background worker has finished but has not been explicitly joined yet. The start path now reaps one finished worker through the existing join/reset logic, preserves false while a worker is active, and keeps failed reaps on the existing log path only.
Add a regression that uses the existing two-overlay fixture to prove a finished unjoined worker does not block a second compaction start.
Validation: build/c/test-runner rebuild /private/tmp/cbm-build-overlay-reap-20260704T1818.log; CBM_ONLY_SUITE=mcp /private/tmp/cbm-test-mcp-overlay-reap-20260704T1825.log (171 passed); source safety /private/tmp/cbm-source-safety-overlay-reap-20260704T1826.log; diff check /private/tmp/cbm-diff-check-overlay-reap-20260704T1826.log; production build/sign /private/tmp/cbm-prod-build-overlay-reap-20260704T1828.log.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Overlay rows intentionally do not expose stable canonical node or edge ids until compaction. Rename the active-read guard around canonical identity, document the id() boundary, and add a Cypher regression proving active overlay rows are used for supported queries but disabled for id() projections.
Validation: CBM_ONLY_SUITE=cypher ./build/c/test-runner; CBM_ONLY_SUITE=mcp ./build/c/test-runner; bash scripts/check-source-safety.sh; git diff --check; make -j8 -f Makefile.cbm cbm.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Fix canonical and active-overlay variable-length Cypher traversal for undirected relationship patterns. The store BFS string API now maps historical directions onto the existing CBM_STORE_EDGE_DIR constants, so canonical and overlay BFS share the same inbound/outbound/any decision instead of duplicating string checks.
query_graph can now answer MATCH (a)-[:TYPE*min..max]-(b) from ready overlay edges, while id() Cypher queries remain explicitly canonical-only until overlay id semantics or compaction exists.
Validation: red canonical TDD /private/tmp/cbm-test-cypher-any-variable-relationship-before-20260704T-current.log; red MCP TDD /private/tmp/cbm-test-mcp-any-variable-relationship-before-20260704T-current.log; green consolidated suites /private/tmp/cbm-test-cypher-any-variable-relationship-consolidated-suite-20260704T-current.log, /private/tmp/cbm-test-mcp-any-variable-relationship-consolidated-suite-20260704T-current.log, /private/tmp/cbm-test-store-nodes-any-variable-relationship-consolidated-20260704T-current.log; source-safety /private/tmp/cbm-source-safety-any-variable-relationship-consolidated-20260704T-current.log; production build /private/tmp/cbm-prod-build-any-variable-relationship-consolidated-20260704T-current.log.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Use the existing qn-keyed overlay BFS helper for directed variable-length Cypher relationship patterns in active query_graph reads.
Keep undirected variable-length traversal and id() canonical-only with explicit warnings until those semantics have a separate tested contract.
TDD: /private/tmp/cbm-test-mcp-active-variable-relationship-before-20260704T-current-2.log failed before the production wiring because FreshVarSource was absent.
Validation: /private/tmp/cbm-build-active-variable-relationship-20260704T-current-2.log; /private/tmp/cbm-test-mcp-active-variable-relationship-focused-20260704T-current-2.log; /private/tmp/cbm-test-mcp-active-variable-relationship-suite-20260704T-current-2.log; /private/tmp/cbm-test-cypher-active-variable-relationship-20260704T-current.log; /private/tmp/cbm-test-store-nodes-active-variable-relationship-20260704T-current.log; /private/tmp/cbm-source-safety-active-variable-relationship-20260704T-current.log; /private/tmp/cbm-prod-build-active-variable-relationship-20260704T-current.log.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Extend active overlay Cypher relationship reads across additional MATCH and OPTIONAL MATCH patterns when every relationship pattern is fixed one-hop.
Reuse the qn-keyed active edge-node store helper for terminal-bound expansion and keep OPTIONAL MATCH no-edge rows on the shared preservation path.
Variable-length relationship traversal and id() projections remain canonical-only with caller-visible warnings.
Validation: /private/tmp/cbm-build-active-multipattern-relationship-20260704T-current-3.log; /private/tmp/cbm-test-mcp-active-multipattern-relationship-focused-20260704T-current-2.log; /private/tmp/cbm-test-mcp-active-multipattern-relationship-suite-20260704T-current-2.log; /private/tmp/cbm-test-cypher-active-multipattern-relationship-20260704T-current-2.log; /private/tmp/cbm-test-store-nodes-active-multipattern-relationship-20260704T-current-2.log; /private/tmp/cbm-source-safety-active-multipattern-relationship-20260704T-current-2.log; /private/tmp/cbm-prod-build-active-multipattern-relationship-20260704T-current-2.log.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add a store-level qn-keyed active edge-node expansion helper and use it in query_graph's active Cypher path for fixed one-hop relationship patterns.
Variable-length, multi-pattern, and id() Cypher remain canonical-only with explicit warnings. This expands AI-facing freshness without widening unsupported relationship semantics.
Validation: build/c/test-runner rebuild; focused MCP active relationship and variable-length canonical tests; full MCP suite 170 passed; full Cypher suite 144 passed; store_nodes 109 passed; source-safety; git diff --check; product build/sign. Logs: /private/tmp/cbm-build-active-cypher-relationship-20260704T-current-2.log, /private/tmp/cbm-test-mcp-active-relationship-query-20260704T-current-2.log, /private/tmp/cbm-test-mcp-variable-relationship-canonical-20260704T-current-2.log, /private/tmp/cbm-test-mcp-active-relationship-suite-20260704T-current-3.log, /private/tmp/cbm-test-cypher-active-relationship-20260704T-current-2.log, /private/tmp/cbm-test-store-nodes-active-relationship-20260704T-current-2.log, /private/tmp/cbm-source-safety-active-relationship-20260704T-current-2.log, /private/tmp/cbm-diff-check-active-relationship-20260704T-current-2.log, /private/tmp/cbm-prod-build-active-relationship-20260704T-current.log.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Teach query_graph's active-overlay Cypher path to answer node-only degree properties and simple EXISTS predicates from ready overlay edges instead of forcing those shapes back to canonical rows.
Relationship-pattern Cypher and id() still remain canonical-only until active relationship binding is available, so the change expands overlay freshness without pretending full active relationship traversal exists.
Validation before commit: build/c/test-runner --suite=mcp --filter active_edges_for_degree_and_exists; build/c/test-runner --suite=mcp --filter id_query_canonical; build/c/test-runner --suite=mcp; build/c/test-runner --suite=cypher; build/c/test-runner --suite=store_nodes; scripts/source_safety_check.sh; git diff --check; make -f Makefile.cbm cbm. Logs: /private/tmp/cbm-test-mcp-active-cypher-edge-derived-20260704T-current-2.log, /private/tmp/cbm-test-mcp-id-canonical-after-active-cypher-20260704T-current-2.log, /private/tmp/cbm-test-mcp-active-cypher-suite-20260704T-current-2.log, /private/tmp/cbm-test-cypher-active-edge-derived-20260704T-current-2.log, /private/tmp/cbm-test-store-nodes-active-edge-derived-20260704T-current-2.log, /private/tmp/cbm-source-safety-active-cypher-edge-derived-20260704T-current-2.log, /private/tmp/cbm-diff-check-active-cypher-edge-derived-20260704T-current-2.log, /private/tmp/cbm-prod-build-active-cypher-edge-derived-20260704T-current.log.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add an end-to-end pipeline regression for a full-mode incremental run that deletes one file and adds another under incremental_derived_refresh=stale_on_exact.
The test verifies exact publish, stale semantic_edges metadata, deleted file cleanup, added file visibility, and absence of synchronous semantic/similarity recomputation. Validation: focused regression, full pipeline suite, and git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add an opt-in incremental_derived_refresh=stale_on_exact policy for full and moderate exact incremental publishes. The default remains eager, while the opt-in path reuses existing derived-view staleness semantics so semantic/similarity queries warn until an eager index run or full reindex rebuilds global edges.
The exact path now skips synchronous SIMILAR_TO/SEMANTICALLY_RELATED recomputation only when file-delta publish marks semantic_edges stale. Broad fallback, overlay, delete, rename, and replacement publish behavior remain conservative.
Validation: CBM_ONLY_SUITE=pipeline build/c/test-runner (330 passed); focused derived-refresh/config tests passed; make -f Makefile.cbm lint-source-safety passed; make -j10 -f Makefile.cbm cbm passed; isolated full-mode benchmark measured 40.153 ms incremental vs 415.058 ms fresh full (10.337x).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Allow all-header C-family incremental batches to publish active overlay rows when a store-level subset proof shows the new delta preserves the file-owned nodes and edges already in the canonical graph. Mixed source/header batches still fall back conservatively with additive_subset_required.
Make MCP read paths treat additive overlay rows as ready even without file tombstones, and fix the BM25 active-overlay query to join active_overlay_files. Keep default non-git search_code scanning on the live root when no file_pattern is provided so active unindexed files remain searchable.
Validation: full ASan/UBSan test runner passed with 6393 passed in /private/tmp/cbm-full-after-live-root-fix-20260704T1340.log; lint-source-safety passed in /private/tmp/cbm-source-safety-after-live-root-fix-20260704T1340.log; production build passed in /private/tmp/cbm-prod-build-after-live-root-fix-20260704T1340.log.
Benchmark: self-dogfood store_pipeline_batch over MCP with overlay_publish=small_deltas and rank_refresh=stale_on_exact passed active overlay graph equality and qualitative oracles; incremental overlay 688 ms vs fresh full rebuild 21676 ms, 31.51x speedup, cleanup true, artifact /private/tmp/cbm-store-pipeline-batch-additive-live-root-fix-20260704T1340.json.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
High-cap diagnostics showed that publishing non-tombstone additive overlay rows for broad C-family header exact frontiers can leave stale canonical edges visible in the active overlay view and can be slower than a full rebuild.
Keep the single-header overlay path separate, but require broad header batches to publish through the exact route until a strict additive-subset proof exists. Update the pipeline canary to require exact publish and fresh-rebuild equivalence for this case.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add a pipeline wrapper for non-tombstone overlay publishes and reuse the store active overlay CTE for file-node reads and summary counts. This lets header/type overlay rows refresh active read views without hiding canonical source-owned rows.
Add focused tests for duplicate-QN additive overlay ranking, single header type/implementation additive overlay behavior, and exact-frontier overlay dirty state.
Validation: bash scripts/check-source-safety.sh; bash scripts/test-source-safety.sh; git diff --check; make -f Makefile.cbm -j8 cbm; CBM_ONLY_SUITE=store_nodes build/c/test-runner; CBM_ONLY_SUITE=pipeline build/c/test-runner.
Scope: this is not the store_pipeline_batch default speed fix. Current benchmark evidence still shows safe full fallback around 2.6x, so narrower dependency-slice or active-read algorithm work remains open.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Split active overlay readability from file replacement tombstones. Replacement overlay publishes still hide canonical file rows, while the new additive publish API layers safe overlay facts without tombstoning the file.
Update the active-overlay CTE, node-view summary, and benchmark mirror so additive overlay rows are visible while canonical rows for the same file remain visible.
Add a store canary proving an additive overlay symbol is readable without creating a file tombstone or hiding existing canonical rows.
Validation: uv py_compile benchmark harness passed; ASan/UBSan test-runner build passed; store_nodes suite passed 108; source-safety passed; diff check clean; production build passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Keep the single-header overlay fast path for safe header-only additions, but fall back when a C-family header delta contains type-like nodes and a same-stem implementation file exists.
This prevents active overlay from selecting header-owned type rows where a fresh full rebuild selects richer source-owned type rows, as seen in the store.h/store.c batch diagnostics.
Adds a pipeline canary for header type/implementation pair fallback while preserving the single-header overlay and multi-file header safety canaries.
Validation: focused header overlay tests passed; pipeline suite passed 326; source safety OK; diff check clean; production build passed; one_source_file benchmark stayed incremental_overlay at 29.677x. Full ASan target was also run and was not clean, with obsolete in-run and environment-sensitive failures recorded in the notes.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Permit the early overlay publish route for a single changed C-family header while keeping multi-file header batches on the conservative exact/full path. The route is still gated by the active overlay graph equality checks instead of relying on an unchecked fast path.
Add a pipeline canary that edits a header, verifies incremental_overlay publication, confirms canonical rows stay unchanged, and confirms overlay rows become ready. The existing multi-file header safety canary remains in place.
Validation: pipeline suite 325 passed; source-safety OK; diff check clean; production build signed; self-dogfood one_source_file benchmark passed via incremental_overlay with 20.7767x speedup and cleanup complete.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Parallel resolve used an unresolved route-registration suffix fallback for callees such as .get and .post. When no route literal was present, that fallback could fall through to a normal CALLS edge using the source node as the target, which fabricated self-calls for ordinary receiver accessors like ec.get(e.type).
Return without emitting when the suffix-only fallback has no route literal, while preserving resolved route API calls and unresolved route registrations that do have a valid route path. Add a parallel resolver canary that first proves extraction sees ec.get and then asserts no self CALLS edge is emitted.
Validation: build/c/test-runner rebuilt; CBM_ONLY_SUITE=parallel CBM_ONLY_TEST=parallel_unresolved_route_suffix_does_not_emit_self_call passed; CBM_ONLY_SUITE=parallel passed 27 tests; lint-source-safety passed; git diff --check passed; make -f Makefile.cbm cbm built and signed; MCP self-dogfood overlay matrix passed 5/5 with cleanup.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Scoped exact and overlay updates run complexity with an incomplete call graph, so they cannot prove that an existing recursive SCC disappeared. Preserve stored recursive=true in scoped complexity mode while full complexity continues to recompute from the current graph.
Adds a focused in-memory pipeline canary for the persisted recursive property. Validated with the focused canary, complexity subset, full pipeline suite, source-safety, diff check, and a production build.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Project active overlay edges as logical graph edges instead of per-file overlay ownership rows. Cross-file edges can be owned by multiple changed files in one overlay generation for cleanup and compaction bookkeeping, but AI-facing active views must not inflate degree or schema counts.
Mirror the production active-edge semantics in the self-dogfood benchmark and add a store regression for duplicate cross-file overlay edge ownership.
Validation: CBM_ONLY_SUITE=store_nodes build/c/test-runner; make -f Makefile.cbm cbm; self-dogfood full matrix at /private/tmp/cbm-overlay-active-edge-dedupe-full-self-dogfood-20260704T.json; source-safety at /private/tmp/cbm-overlay-active-edge-dedupe-source-safety-20260704T.log.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Guard C-family header changes from active overlay publication until active edge parity is proven against fresh rebuilds. Bounded header edits can still use exact base-delta publication, preserving correctness without forcing every safe case to full rebuild.
Add a focused C-header incremental canary that verifies no overlay publish, exact frontier expansion, exact publish stats, and fresh-rebuild equality. Clean up stale direct-frontier helper code from the failed cap64 overlay experiment.
Validation: CBM_ONLY_SUITE=pipeline build/c/test-runner -> 323 passed; make -f Makefile.cbm lint-source-safety -> OK; self-dogfood cap16/cap64 store_pipeline_batch passed canonical equality but still fell back with frontier_too_large at 2.81x/2.74x.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Serialize call args consistently for call-derived edges across sequential and parallel indexing, including CONFIGURES edges used by MCP graph responses and data-flow derivation.
Add a generic argument-list fallback for call extraction so grammars exposing arguments as argument_list still populate CBMCall.args.
Guard exact delta and overlay publication against cross-file node-QN collisions, falling back instead of letting a changed header override a canonical source definition in the active overlay view.
Validation: build/c test runner and binary rebuild passed; CBM_ONLY_SUITE=pipeline passed; lint-source-safety passed; full self-dogfood MCP matrix passed 5/5 in /private/tmp/cbm-cross-file-qn-guard-full-self-dogfood-20260704T.json.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Keep symbol and module FQNs compatible while making File node qualified names injective for same-stem files such as .c/.h pairs.
Add a regression canary for File node FQNs and update pipeline import-map fixtures to use the extension-preserving file identity. Keep the package-map module lookup compatible by trying the legacy exact QN before a deterministic File-node fallback.
Validation: CBM_ONLY_SUITE=fqn build/c/test-runner; CBM_ONLY_SUITE=registry build/c/test-runner; CBM_ONLY_SUITE=pipeline build/c/test-runner; CBM_ONLY_SUITE=mcp build/c/test-runner; make -f Makefile.cbm lint-source-safety.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add an overlay-first pure-upsert route for overlay_publish=small_deltas so bounded changed-file overlays can publish before recursive exact inbound-frontier expansion falls through to containment.
The route reuses existing scratch seed, extract/resolve, postpass, delta metadata, preserved-inbound, and overlay publish helpers. It is gated to pure upserts in FAST-or-higher modes within the existing exact changed/affected path caps, and falls through to the existing exact/containment paths on failure.
Avoid a duplicate failed overlay generation when overlay-first reaches the publish step and fails; exact upsert skips the second overlay publish attempt and continues to canonical exact fallback.
Add a wide-inbound pipeline canary proving a changed Leaf with four unchanged callers now publishes incremental_overlay before frontier_too_large and active overlay BFS still sees CallerA -> Leaf.
Validation: make -f Makefile.cbm -j8 build/c/test-runner; CBM_ONLY_SUITE=pipeline build/c/test-runner (321 passed); CBM_ONLY_SUITE=mcp build/c/test-runner (166 passed); scripts/check-source-safety.sh on touched files; git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Carry inbound edge properties through cbm_store_list_file_delta_inbound_edges so active overlay rows do not lose relationship metadata.
Promote the recomputed-edge skip policy into a shared pipeline predicate with edge-type constants, and reuse it from incremental replay.
Add cbm_pipeline_file_delta_add_preserved_inbound_edges for overlay-producer support: upsert deltas preserve only non-recomputed one-hop inbound edges whose target QN still exists in the new delta, with duplicate suppression.
Validation: make -f Makefile.cbm -j8 build/c/test-runner; CBM_ONLY_SUITE=store_nodes build/c/test-runner (105 passed); CBM_ONLY_SUITE=pipeline build/c/test-runner (320 passed); CBM_ONLY_SUITE=mcp build/c/test-runner (166 passed); scripts/check-source-safety.sh on touched files; git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add a focused MCP canary for active overlay graph reads where an unchanged canonical caller points to a changed overlay target. The test proves the old canonical target is hidden by the overlay tombstone while search_graph relationship reads still return the caller and replacement target with overlay_active_graph metadata.
Validation:
- make -f Makefile.cbm -j8 build/c/test-runner
- CBM_ONLY_SUITE=mcp build/c/test-runner
- bash scripts/check-source-safety.sh tests/test_mcp.c
- git diff --check
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Reuse the existing sequential EnvVar/CONFIGURES materialization helper from the parallel full-index registry phase so full rebuilds remain a valid oracle for incremental slices.
Add a sequential-vs-parallel pipeline canary for C getenv access that verifies the CONFIGURES edge and canonical graph equality. This fixes the route_handler self-dogfood mismatch where incremental containment had EnvVar data that parallel full rebuild omitted.
Validation: make -f Makefile.cbm -j8 build/c/test-runner; CBM_ONLY_SUITE=pipeline build/c/test-runner; bash scripts/check-source-safety.sh src/pipeline/pass_definitions.c src/pipeline/pass_parallel.c src/pipeline/pipeline_internal.h tests/test_pipeline.c; git diff --check; make -f Makefile.cbm -j8 cbm; route_handler self-dogfood rerun passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Create ancestor folder nodes before child folder edges in the full structure pass so first-seen nested directories produce deterministic CONTAINS_FOLDER edges.
Add the matching CBM_PIPELINE_EDGE_CONTAINS_FOLDER constant and reuse it in the exact-delta structural context path. Strengthen the pipeline structure canary to require pkg -> pkg/util containment and keep cleanup ahead of final assertions.
Validation: make -f Makefile.cbm -j8 build/c/test-runner; CBM_ONLY_SUITE=pipeline build/c/test-runner; bash scripts/check-source-safety.sh src/pipeline/pipeline.c src/pipeline/pipeline_delta.c src/pipeline/pipeline_internal.h tests/test_pipeline.c; git diff --check; make -f Makefile.cbm -j8 cbm; refreshed bounded P.A.N8.7 overlay CLI matrix passed 5/5.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add opt-in overlay_compaction_policy=after_publish with a bounded overlay_compaction_max_generations setting. index_repository now starts a one-shot compaction worker only after a successful incremental_overlay publish and reports the trigger status in the response.\n\nValidation:\n- make -f Makefile.cbm -j8 build/c/test-runner\n- CBM_ONLY_SUITE=mcp build/c/test-runner\n- CBM_ONLY_SUITE=pipeline build/c/test-runner\n- make -f Makefile.cbm -j8 cbm\n- git diff --check\n- bash scripts/check-source-safety.sh src/mcp/mcp.c src/cli/cli.c src/pipeline/pipeline.h tests/test_mcp.c tests/test_pipeline.c
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add a one-shot MCP server overlay compaction worker that opens its own project store handle, reuses the bounded store drain helper, and is joined explicitly or during server teardown. The worker is not exposed as an MCP tool, does not poll or sleep, and does not change default behavior.
Reject duplicate starts until the previous worker is joined, validate project names before spawning, use the existing project DB path helper, and open databases with the no-create query path to avoid ghost cache files.
Cover the lifecycle with an isolated-cache MCP canary that compacts one overlay, rejects a duplicate start, joins cleanly, then drains the remaining overlay. Validation: make -f Makefile.cbm -j8 build/c/test-runner; CBM_ONLY_SUITE=mcp build/c/test-runner (161 passed); git diff --check; scripts/check-source-safety.sh src/mcp/mcp.c src/mcp/mcp.h tests/test_mcp.c; make -f Makefile.cbm -j8 cbm.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add a bounded store helper that repeatedly reuses compact_next for ready overlay generations. A named CBM_STORE_COMPACT_ALL_GENERATIONS constant documents the drain-all mode, while positive limits cap the number of generations compacted by one API call.
Cover no-ready drain behavior and a two-overlay canary that proves a one-generation limit leaves later ready overlays visible for a subsequent drain. This prepares the future background worker to reuse one store path instead of duplicating compaction loops.
Validation: make -f Makefile.cbm -j8 build/c/test-runner; CBM_ONLY_SUITE=store_nodes build/c/test-runner (105 passed); git diff --check; scripts/check-source-safety.sh src/store/store.c src/store/store.h tests/test_store_nodes.c; make -f Makefile.cbm -j8 cbm.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add a reusable store primitive that claims the oldest ready overlay generation, reserves a canonical index generation, and reuses the existing overlay compaction path. On reservation or compaction failure, the helper releases the compaction claim and marks the reserved generation failed when applicable.
Keep the behavior inert unless explicitly called by future compaction workers. Cover the no-ready CBM_STORE_NOT_FOUND contract and route an existing tombstone compaction canary through the new helper.
Validation: CBM_ONLY_SUITE=store_nodes build/c/test-runner (104 passed); git diff --check; scripts/check-source-safety.sh src/store/store.c src/store/store.h tests/test_store_nodes.c; make -f Makefile.cbm -j8 cbm.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Replace status-based overlay compaction claims with an explicit overlay_compaction_claims lease table. Claiming now inserts a lease for the oldest unclaimed overlay_ready generation instead of changing the generation status to compacting, so active overlay read paths keep seeing the fresh overlay while a future background compactor owns it.
Recovery now releases abandoned lease rows after process restart or confirmed worker shutdown. The tests cover double-claim prevention, recovery/reclaim, failed-generation preservation, active read visibility while claimed, and FK cleanup of the claim when compaction deletes the overlay generation.
Validation: make -f Makefile.cbm -j8 build/c/test-runner; CBM_ONLY_SUITE=store_nodes build/c/test-runner; git diff --check; bash scripts/check-source-safety.sh src/store/store.c src/store/store.h tests/test_store_nodes.c; make -f Makefile.cbm -j8 cbm.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add a default-inert recovery primitive for overlay generations left in compacting after process restart or confirmed worker shutdown. The API requeues compacting generations to overlay_ready in one existing BEGIN IMMEDIATE transaction and returns the number recovered, with no timeout heuristic or live-worker assumption.
Cover the recovery path in store_nodes by claiming a ready generation, recovering it, proving failed generations are untouched, and proving the recovered generation can be claimed again. The header documents that callers must not run recovery while a compactor is active.
Validation: make -f Makefile.cbm -j8 build/c/test-runner; CBM_ONLY_SUITE=store_nodes build/c/test-runner; git diff --check; bash scripts/check-source-safety.sh src/store/store.c src/store/store.h tests/test_store_nodes.c; make -f Makefile.cbm -j8 cbm.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add cbm_store_claim_ready_overlay_generation() as the narrow store lifecycle primitive for a future background compactor. The claim runs in the existing BEGIN IMMEDIATE transaction path, selects the oldest ready overlay generation, and moves it from overlay_ready to compacting with a guarded status update so two callers cannot claim the same generation.
Cover oldest-ready ordering, non-ready filtering, invalid output handling, and claim-plus-compact composition in store_nodes. This keeps the change default-inert: no background thread, no default-on policy, and no shared store-handle concurrency are introduced.
Validation: make -f Makefile.cbm -j8 build/c/test-runner; CBM_ONLY_SUITE=store_nodes build/c/test-runner; git diff --check; bash scripts/check-source-safety.sh src/store/store.c src/store/store.h tests/test_store_nodes.c; make -f Makefile.cbm -j8 cbm.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Persist full overlay file delta metadata so ready overlay generations can be promoted into canonical rows without re-running extraction. Store file hashes, file state, imports, exports, and derived-view metadata alongside overlay nodes, edges, and tombstones.
Add cbm_store_compact_overlay_generation() as a manual, default-inert store API. It loads persisted overlay deltas, reuses the canonical delete/publish bodies inside one transaction, marks derived graph views stale, finishes the reserved canonical generation, clears dirty file rows, and removes promoted overlay rows.
Cover both upsert and delete-only compaction paths in store_nodes. Validated with ASan/UBSan test-runner build, store_nodes, pipeline, production cbm build, git diff --check, and focused stdout/string-safety scans. Logs are under /private/tmp/cbm-pan85-overlay-compaction-*20260704T.log.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Record changed git paths in the dirty-file ledger before invoking the watcher index callback so freshness surfaces remain truthful if reindexing is slow or fails.
Reuse the existing git snapshot command plumbing, store project lookup, and project field free helper. Skip ledger writes for watched projects that have not been indexed yet so pre-index watcher tests and users do not get noisy warnings.
Add canaries for failing index callbacks and rename status parsing, including both current and previous paths from git porcelain -z output.
Validation: make -f Makefile.cbm -j8 build/c/test-runner; CBM_ONLY_SUITE=watcher build/c/test-runner; CBM_ONLY_SUITE=pipeline build/c/test-runner; CBM_ONLY_SUITE=store_nodes build/c/test-runner; make -f Makefile.cbm -j8 cbm. Logs under /private/tmp/cbm-pan82-watcher-dirty-admission-*20260704T.log.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Prune older ready overlay rows for the same project/path after a newer overlay generation publishes successfully. The cleanup deletes overlay FTS rows before relational rows and removes only ready generations that become empty, so multi-file generations retain unrelated file rows.
Add store canaries for single-file generation pruning, path-scoped multi-file preservation, and FTS cleanup of stale overlay tokens.
Validation:
- git diff --check: /private/tmp/cbm-pan82-overlay-retention-diff-check2-20260704T.log
- make -f Makefile.cbm -j8 build/c/test-runner: /private/tmp/cbm-pan82-overlay-retention-test-build-20260704T.log
- CBM_ONLY_SUITE=store_nodes build/c/test-runner: /private/tmp/cbm-pan82-overlay-retention-store-nodes-20260704T.log (98 passed)
- CBM_ONLY_SUITE=pipeline build/c/test-runner: /private/tmp/cbm-pan82-overlay-retention-pipeline-20260704T.log (317 passed)
- bash scripts/check-source-safety.sh: /private/tmp/cbm-pan82-overlay-retention-source-safety-20260704T.log
- make -f Makefile.cbm -j8 cbm: /private/tmp/cbm-pan82-overlay-retention-product-build-20260704T.log
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Apply the existing incremental extract fault hook to the sequential path so small exact-delta runs can exercise the same failure boundary as the parallel path.
Add an opt-in overlay publish canary that injects extract failure before overlay generation reservation, verifies canonical rows remain unchanged, no overlay generations are ready or failed, and the dirty file remains pending rather than overlay_ready.
Validation:
- make -f Makefile.cbm -j8 build/c/test-runner: /private/tmp/cbm-pan82-overlay-extract-failure-test-build3-20260704T.log
- CBM_ONLY_SUITE=pipeline build/c/test-runner: /private/tmp/cbm-pan82-overlay-extract-failure-pipeline2-20260704T.log (317 passed)
- bash scripts/check-source-safety.sh: /private/tmp/cbm-pan82-overlay-extract-failure-source-safety-20260704T.log
- git diff --check: /private/tmp/cbm-pan82-overlay-extract-failure-diff-check3-20260704T.log
- make -f Makefile.cbm -j8 cbm: /private/tmp/cbm-pan82-overlay-extract-failure-product-build-20260704T.log
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add a CBM_TEST_FAIL_INCREMENTAL_PHASE value for the overlay producer boundary and share the existing phase-failure helper between incremental and delta code. When injected after overlay generation reservation, the producer marks the overlay generation failed and returns an error.
Add a focused pipeline canary proving opt-in overlay publish failure falls back to canonical exact publish: the changed symbol is present canonically, the failed overlay generation is recorded, no ready overlay generation or reserved canonical generation remains, and dirty rows are cleared after canonical repair.
Validation: initial build failed because the helper included the wrong declaration header; corrected to foundation/platform.h. Final gates: make -f Makefile.cbm build/c/test-runner -j8; CBM_ONLY_SUITE=pipeline build/c/test-runner; bash scripts/check-source-safety.sh; git diff --check; make -f Makefile.cbm cbm -j8.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add a focused pipeline canary for repeated opt-in overlay publishes of the same changed file. The test proves active overlay reads stay idempotent from the caller perspective: canonical base remains unchanged, the overlay view exposes one function, dirty_files remains a single overlay_ready row, and no reserved canonical generation is leaked.
This is intentionally test-only. It does not claim overlay generation retention or duplicate-publish suppression is solved; those remain separate compaction/cleanup design items.
Validation: make -f Makefile.cbm build/c/test-runner -j8; CBM_ONLY_SUITE=pipeline build/c/test-runner; bash scripts/check-source-safety.sh; git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Extend the existing default-off overlay_publish=small_deltas path to the single-file exact delete route. The route now reuses the overlay file-delta batch producer to publish a tombstone before canonical generation reservation, leaving canonical state untouched and dirty_files marked overlay_ready for active read views.
Keep overlay_publish=off behavior unchanged: canonical exact delete still applies and existing full-rebuild parity coverage remains in place. Add a pipeline canary proving the opt-in delete overlay keeps canonical base rows visible, hides the deleted file through the overlay view, avoids reserved canonical generation leaks, and preserves overlay_ready freshness state.
Validation: make -f Makefile.cbm build/c/test-runner -j8; CBM_ONLY_SUITE=pipeline build/c/test-runner; bash scripts/check-source-safety.sh; git diff --check; make -f Makefile.cbm cbm -j8.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Validate FastAPI fixtures by required files and the pinned 0.99.1 commit before reuse. A user-provided CBM_TEST_FASTAPI_REPO is treated as read-only; the managed CBM_TEST_FASTAPI_CACHE/default temp cache is refreshed only when invalid.
Each suite still clones an isolated mutable working repo from the validated source, preserving test isolation while avoiding repeated network clones on subsequent runs.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add a default-off overlay_publish policy for bounded exact-delta incremental runs. When enabled, eligible non-noop exact deltas publish ready overlay rows and dirty freshness without mutating canonical graph rows, while default canonical behavior remains unchanged.
Use the latest complete canonical generation as the overlay base, avoid reserving a canonical generation for successful overlay publishes, and report incremental_overlay in pipeline telemetry and benchmark parsing.
Add focused store and pipeline canaries for latest complete generation lookup, opt-in overlay visibility, canonical invisibility, dirty overlay_ready state, and absence of stray reserved canonical generations.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add an atomic store batch API for foreground overlay file deltas and keep the existing single-file API as a compatibility wrapper. The batch path validates one-project inputs, writes overlay rows and tombstones in one transaction, and marks the generation ready only after every file succeeds.
Add an internal pipeline producer helper that reserves an overlay generation, publishes existing file-delta facts, and marks dirty freshness as overlay_ready. If publish or freshness marking fails, the helper marks the overlay generation failed so active read views do not expose a misleading generation.
Cover the contract with focused store and pipeline canaries. Store coverage verifies multi-file rollback leaves no partial overlay rows. Pipeline coverage uses existing exact-delta extraction metadata to prove overlay-visible, canonical-invisible, dirty-ready behavior. Production incremental wiring, compaction, and default-readiness remain separate follow-up work.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Full canonical replacement deletes the project row and relies on FK cascade to remove overlay generations and rows, but contentless nodes_fts_overlay is not FK-backed. Remove overlay FTS entries while overlay_nodes still exists so full reindex/project delete cannot leave stale overlay tokens behind.
Cross-check: upstream/main has only canonical project replacement and no overlay tables, so this preserves upstream replacement semantics and adds fork-only derived-view cleanup.
Validation: CBM_ONLY_SUITE=store_nodes make -j8 -f Makefile.cbm test passed 95 tests after the pre-fix canary failed; bash scripts/check-source-safety.sh passed; make -j8 -f Makefile.cbm cbm built and signed; git diff --check passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Andrew Hundt