Split active overlay readability from file replacement tombstones. Replacement overlay publishes still hide canonical file rows, while the new additive publish API layers safe overlay facts without tombstoning the file.
Update the active-overlay CTE, node-view summary, and benchmark mirror so additive overlay rows are visible while canonical rows for the same file remain visible.
Add a store canary proving an additive overlay symbol is readable without creating a file tombstone or hiding existing canonical rows.
Validation: uv py_compile benchmark harness passed; ASan/UBSan test-runner build passed; store_nodes suite passed 108; source-safety passed; diff check clean; production build passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Project active overlay edges as logical graph edges instead of per-file overlay ownership rows. Cross-file edges can be owned by multiple changed files in one overlay generation for cleanup and compaction bookkeeping, but AI-facing active views must not inflate degree or schema counts.
Mirror the production active-edge semantics in the self-dogfood benchmark and add a store regression for duplicate cross-file overlay edge ownership.
Validation: CBM_ONLY_SUITE=store_nodes build/c/test-runner; make -f Makefile.cbm cbm; self-dogfood full matrix at /private/tmp/cbm-overlay-active-edge-dedupe-full-self-dogfood-20260704T.json; source-safety at /private/tmp/cbm-overlay-active-edge-dedupe-source-safety-20260704T.log.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Guard C-family header changes from active overlay publication until active edge parity is proven against fresh rebuilds. Bounded header edits can still use exact base-delta publication, preserving correctness without forcing every safe case to full rebuild.
Add a focused C-header incremental canary that verifies no overlay publish, exact frontier expansion, exact publish stats, and fresh-rebuild equality. Clean up stale direct-frontier helper code from the failed cap64 overlay experiment.
Validation: CBM_ONLY_SUITE=pipeline build/c/test-runner -> 323 passed; make -f Makefile.cbm lint-source-safety -> OK; self-dogfood cap16/cap64 store_pipeline_batch passed canonical equality but still fell back with frontier_too_large at 2.81x/2.74x.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add an active-overlay graph comparator to the incremental benchmark harness and use it as the structural gate for incremental_overlay rows. Canonical/noop/exact/containment rows continue to use canonical graph equality.
This caught the same-stem File-node collision that answer-level MCP oracles missed, and the follow-up route-handler benchmark now passes with active_overlay_graph.equal=true.
Validation: uv run python -m py_compile scripts/benchmark-incremental-speed.py; /private/tmp/cbm-pan87-active-graph-route-handler-mcp-after-file-qn-20260704T.json passed with 1842 ms incremental vs 22607 ms fresh full, 12.27x speedup.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Keep canonical graph equality as the default graph gate, but let self-dogfood incremental_overlay rows pass on active answer oracles and freshness metadata because overlay publishes intentionally do not mutate canonical rows.
Record the selected graph_gate in benchmark output so maintainers can see whether a row used canonical_graph or overlay_active_oracles policy.
Validation: uv run python -m py_compile scripts/benchmark-incremental-speed.py; route_handler MCP self-dogfood rerun passed with publish_kind=incremental_overlay, graph_gate=overlay_active_oracles, active oracles passed, cleanup removed, and 13.09x speedup.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add a default-off overlay_publish policy for bounded exact-delta incremental runs. When enabled, eligible non-noop exact deltas publish ready overlay rows and dirty freshness without mutating canonical graph rows, while default canonical behavior remains unchanged.
Use the latest complete canonical generation as the overlay base, avoid reserving a canonical generation for successful overlay publishes, and report incremental_overlay in pipeline telemetry and benchmark parsing.
Add focused store and pipeline canaries for latest complete generation lookup, opt-in overlay visibility, canonical invisibility, dirty overlay_ready state, and absence of stray reserved canonical generations.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Use sqlite3's online backup API when preserving benchmark snapshots instead of copying .db files directly. This keeps WAL-mode databases consistent for canonical graph comparisons and removes stale destination sidecars before writing the snapshot.
Validation:
- git diff --check
- uv run python -m py_compile scripts/benchmark-incremental-speed.py
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add optional exact_delta metadata to index_repository responses so benchmark artifacts can distinguish changed, affected, and published exact-delta path counts without scraping logs. Keep the benchmark parser backward-compatible by reading response metadata first and stderr markers for older binaries.
Validation: make -j16 -f Makefile.cbm cbm; make -j16 -f Makefile.cbm build/c/test-runner; CBM_ONLY_SUITE=mcp CBM_ONLY_TEST=tool_index_repository_reports_incremental_containment_reason build/c/test-runner; CBM_ONLY_SUITE=mcp build/c/test-runner; bash scripts/check-source-safety.sh; uv run python -m py_compile scripts/benchmark-incremental-speed.py; MCP telemetry smoke /private/tmp/cbm-pan2-20260703T0353Z-exact-delta-telemetry-smoke.json.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add an opt-in rank_refresh=stale_on_incremental policy while preserving eager as the default and stale_on_exact as exact-only. Replacement publishes now mark rank-derived views stale before refresh, and MCP index, autoindex, and watcher paths share a publish-kind mapper so refresh policy is applied consistently.
Validation: make -f Makefile.cbm -j8 cbm rc=0 (/private/tmp/cbm-rank-refresh-dry-build1.log); make -f Makefile.cbm -j8 build/c/test-runner rc=0 (/private/tmp/cbm-rank-refresh-dry-test-runner-build1.log); CBM_ONLY_SUITE=pagerank build/c/test-runner rc=0, 57/57 (/private/tmp/cbm-rank-refresh-dry-pagerank2.log); CBM_ONLY_SUITE=pipeline build/c/test-runner rc=0, 307/307 (/private/tmp/cbm-rank-refresh-dry-pipeline2.log); make -f Makefile.cbm lint-source-safety rc=0 (/private/tmp/cbm-rank-refresh-dry-source-safety1.log); git diff --check clean.
Benchmark note: stale_on_exact vs stale_on_incremental self-dogfood comparison is still pending because the escalated uv benchmark run was rejected by the environment usage limit. Historical baseline remains /private/tmp/cbm-pan6-no-late-fallback-selfdogfood-cap256.json.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add config-backed exact delta changed/affected frontier limits while preserving the conservative defaults. Exact delete and upsert paths now read the effective limits through shared pipeline getters instead of hardcoded live caps.
Add TDD coverage for default clamping, registry entries, and a configured 5-file inbound frontier that publishes exact and matches a fresh rebuild. Extend the incremental benchmark harness with repeatable --config key=value overrides so cap experiments are reproducible.
Validation: ASan/UBSan test-runner build rc=0; CBM_ONLY_SUITE=pipeline passed 304/304; source-safety rc=0 after harness edit; production cbm build rc=0; benchmark pycompile rc=0. Self-dogfood one_source_file with incremental_exact_max_affected_paths=64 passed canonical/oracle/cleanup checks but still fell back to containment with exact_reason=inbound_edges_require_full and only 3.30x speedup, so incremental remains off by default.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Record subprocess return code, stdout/stderr byte counts and tails, plus raw stdout/stderr/meta files when benchmark CLI subprocesses fail. Surface the structured details in benchmark JSON reports so failed runs can be diagnosed without rerunning expensive cases.
Keep artifacts under each benchmark work root's failures directory, with a temp fallback only when no CBM_CACHE_DIR is present.
Validation: py_compile with cfile in /private/tmp passed, expected-failure wrapper produced error_detail.returncode=37 and raw artifacts, tiny success benchmark returned rc=0 and passed under --min-speedup 0. Temporary work roots and wrapper were cleaned after standalone evidence was saved.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Extend scripts/benchmark-incremental-speed.py with a real-repo self-dogfood mode that reuses the existing CLI/MCP transport, isolated cache setup, canonical graph comparison, publish-kind parsing, freshness capture, and cleanup reporting.
The new mode creates detached temporary git worktrees, applies deterministic CBM-prefixed marker edits, probes search_graph/search_code/query_graph/get_architecture/route freshness, records source git metadata, and resolves the default binary relative to either the caller cwd or the script repo root.
Also make canonical SQLite comparison surrogate-safe for real-repo rows and scope marker search_code probes with file_pattern plus path_filter so oracle latency stays practical.
Validation: no-bytecode compile log /private/tmp/cbm-pan4-20260702T-self-dogfood-compile-filepattern.log; git diff --check; create-path matrix smoke /private/tmp/cbm-pan4-20260702T-self-dogfood-harness-matrix-create-smoke.json passed with publish_kind=incremental_exact.
Current known blocker captured by the harness: /private/tmp/cbm-pan4-20260702T-self-dogfood-one-source-mcp-filepattern.json fails canonical equality for internal.cbm.ac.CBMAutomaton (containment ac.c vs fresh full ac.h), so broad self-dogfood expansion remains blocked pending root-cause work.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add compatibility-preserving freshness metadata to stale derived-view MCP responses while preserving existing warnings text. Responses now include freshness.state=stale_with_warning and freshness.stale_views for the stale derived views already detected by search_graph, query_graph, trace_path, and get_architecture.
Update the existing incremental benchmark parser to retain freshness and freshness_state from tool responses instead of introducing a parallel harness.
Validation: make -f Makefile.cbm cbm; bash scripts/check-source-safety.sh; CBM_ONLY_SUITE=mcp 131/131; CBM_ONLY_SUITE=tool_consolidation 98/98; CBM_ONLY_SUITE=pipeline 298/298; CBM_ONLY_SUITE=pagerank 55/55; uv run python -m py_compile scripts/benchmark-incremental-speed.py. Dogfood: isolated CBM self-index in /private/tmp/cbm-pan2-dogfood-cache.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Detect exact file deltas whose canonical graph rows already match the persisted store, then refresh only file freshness and incremental metadata under a completed generation. This keeps graph-changing exact deltas on the existing planner/apply path while body-only edits publish as incremental_noop with graph_changed=false.
Also update the incremental benchmark harness so incremental_noop is treated as an explicit incremental route while preserving the existing explicit_exact_or_fallback field for compatibility.
Validation: git diff --check; bash scripts/check-source-safety.sh; bash scripts/test-source-safety.sh; python3 -m py_compile scripts/benchmark-incremental-speed.py; make -j8 -f Makefile.cbm build/c/test-runner; CBM_ONLY_SUITE=store_nodes 81 passed; CBM_ONLY_SUITE=pipeline 295 passed; CBM_ONLY_SUITE=incremental 161 passed; make -j8 -f Makefile.cbm cbm; persistent-MCP affected-frontier matrix 8/8 canonical-equal.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add pipeline-owned publish_reason metadata and surface it in index_repository responses when incremental indexing falls back to containment. The reason is set on existing exact-route skip/fallback paths and cleared on exact publish/full run reset, so MCP and CLI JSON callers can distinguish a safe containment rebuild from an opaque incremental result.
Update the affected-frontier benchmark harness to prefer structured publish_reason while keeping stderr parsing for older binaries. Add focused pipeline assertions and an MCP tool test for incremental_containment with missing_existing_ownership.
Validated with git diff --check, bash scripts/check-source-safety.sh, bash scripts/test-source-safety.sh, python3 -m py_compile scripts/benchmark-incremental-speed.py, escalated make -j8 -f Makefile.cbm build/c/test-runner, escalated CBM_ONLY_SUITE=pipeline build/c/test-runner, escalated CBM_ONLY_SUITE=mcp build/c/test-runner, escalated make -j8 -f Makefile.cbm cbm, and escalated persistent MCP affected-frontier matrix 8/8 pass.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Extend the existing incremental benchmark with an opt-in --matrix mode for affected-frontier scenarios. The matrix isolates repos and caches per case, records publish_kind and exact/fallback reasons, compares incremental output to a fresh FAST rebuild with canonical SQLite graph rows, and cleans up generated work roots by default.
This is a measurement and regression harness, not a default-policy change. incremental_reindex remains default-off until the broader matrix, freshness semantics, cleanup, and speedup gates are satisfied.
Validated with python3 -m py_compile scripts/benchmark-incremental-speed.py, git diff --check, bash scripts/check-source-safety.sh, and an escalated two-case matrix smoke for go_modify_1/go_modify_2.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add --transport=cli|mcp to the incremental speed benchmark. The MCP path keeps one JSON-RPC server process alive for each measurement phase and detects full versus exact incremental routes from index_repository publish_kind metadata instead of stderr logs.
The harness still preserves the cold CLI path and cleanup behavior, and reports publish_kind alongside existing wall/logged timing fields for current and future performance comparisons.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add indexed_work_elapsed_ms and unlogged_overhead_ms to the incremental speed benchmark output while preserving the existing elapsed_ms and logged_elapsed_ms fields. This separates cold CLI wall time from the logged indexing path so future speedup work can target the correct latency term.
Validation: python3 -m py_compile scripts/benchmark-incremental-speed.py; git diff --check; scripts/benchmark-incremental-speed.py --files 20 --functions-per-file 4 --changed-files 1 --min-speedup 0.5 --timeout 120 --out /private/tmp/cbm-incr-speed-overhead-fields-smoke-pass.json
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add an opt-in benchmark that compares fast-mode exact incremental indexing against a fresh fast full rebuild in an isolated temp repo and CBM_CACHE_DIR. The gate requires the incremental.exact.done marker, records rank-refresh markers and bounded log tails, and cleans only paths it creates.
This does not change defaults. The current implementation is expected to fail the default 10x threshold until exact-delta post-passes are scoped or marked stale safely.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Andrew Hundt