scripts/benchmark-incremental-speed.py previously wrote rank_refresh=stale_on_exact for every matrix cell, including profiles labeled default. That changed candidate behavior and invalidated comparisons against binaries with different compiled policies.
Add RANK_REFRESH_CANDIDATE_DEFAULT and apply_rank_refresh_override() so default runs make no rank_refresh config write. Record the requested policy and whether an override was applied in synthetic, matrix, self-dogfood, and capability-quality reports; retain explicit eager, stale_on_exact, and stale_on_incremental experiments.
Document the comparison contract in docs/BENCHMARK_CAMPAIGN.md and cover CLI default, no-write, and explicit-write behavior in tests/test_benchmark_incremental_speed.py.
Verification: uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign (90 tests); uv run ruff check scripts/benchmark-incremental-speed.py tests/test_benchmark_incremental_speed.py; bash scripts/check-source-safety.sh; git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add self_dogfood workload expansion in scripts/run-benchmark-campaign.py with exact repository revision/tree identities, candidate-scoped profiles, and result rejection on background mismatches.
Make create_self_dogfood_worktree check out the declared commit, retain pre/post mutation SHA-256 values, and add c_new_leaf for a real indexed create-file route alongside high-fanout fallback scenarios.
Document the JSON contract in docs/BENCHMARK_CAMPAIGN.md. Verify with 125 benchmark/report unit tests, Ruff, source-safety, and an optimized c_new_leaf smoke: incremental_exact 367 ms, fresh rebuild 4456 ms, canonical equality 1/1, cleanup complete.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Capability-quality reports stored canonical equality inside pair_lifecycle, while summarize_group() read only case-level canonical_graph. Eager semantic runs therefore displayed Graph fidelity n/a and Overall quality n/a despite Pair F1=1.0, task success, and an exact incremental-versus-fresh graph.
Include lifecycle canonical equality only when the recorded policy requires immediate freshness; deferred views remain non-applicable instead of becoming false failures. Define result quality as Pair F1 or Retrieval MRR when one is measured, or their arithmetic mean when both are measured, then retain the equal-weight geometric mean across result quality, graph fidelity, and task success.
Verification: focused RED produced graph_fidelity_score=None for a perfect eager lifecycle; GREEN produces Pair F1=1.0, graph fidelity=1.0, task success=1.0, and overall quality=1.0 while the deferred test remains valid. Full benchmark/campaign/report suite: 120 passed; Ruff; scripts/check-source-safety.sh; git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
The stable graph fingerprint added in 5cb6e2eb reused raw canonical qualified names and node properties. Independent benchmark roots therefore produced different hashes even when retained logs showed identical source hashes, node/edge counts, semantic edge counts, similarity edge counts, and judged scores.
Add path-normalized SQL projections that strip only the exact project prefix from qualified names. Hash structural node identity, structural edge identity, source path/content/size, and semantic score/jaccard rows as separate components. Exclude arbitrary node properties and file mtimes from cross-run identity.
Keep compare_canonical_graph() on the original full canonical rows, including properties and mtimes, so incremental-versus-fresh correctness remains strict. Stable experiment identity no longer short-circuits or weakens that gate.
Verification: RED SQLite fixture produced unequal hashes for identical relative graphs under random-root-a and random-root-b; GREEN makes them equal and still detects a semantic score change from 0.873 to 0.811. Full benchmark/campaign/report suite: 119 passed; Ruff; scripts/check-source-safety.sh; git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
scripts/summarize-benchmark-results.py derived pair quality decisions from top-level oracles and ignored initial/incremental/fresh lifecycle failures when those aliases were absent. A BELOW QUALITY TARGET row could therefore have no finding, and Markdown replaced the empty evidence with 'All applicable canonical-graph and task-oracle checks passed.'
Treat initial and fresh pair stages as required and the post-edit stage as required only when the recorded freshness policy expects immediate results. Emit stage names, TP/TN/FP/FN counts, and absent-expected or unexpected-positive counts. Skip failure wording for explicit capability-off controls and preserve deferred post-edit semantics.
Only use the all-checks-passed fallback for PASS decisions. Non-pass rows without a recorded witness now state that limitation and direct readers to retained raw results.
Verification: focused RED returned PASS with empty findings for an initial FN=1; GREEN returns BELOW QUALITY TARGET with 'Initial semantic-pair quality missed' and TP=0, TN=2, FP=0, FN=1. Full benchmark/campaign/report suite: 118 passed; Ruff; scripts/check-source-safety.sh; git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
scripts/benchmark-incremental-speed.py previously loaded complete canonical graph row lists and mismatch sets, requiring O(nodes + edges) Python memory. Supervisor worker logs remained inside disposable work roots, so successful automatic cleanup removed the raw phase and peak-memory evidence.
Hash ordered query rows as length-delimited SHA-256 streams and diagnose mismatches with a two-cursor merge, both using O(1) Python auxiliary memory. Record initial, incremental, and fresh node/edge/source-content fingerprints; exclude volatile mtimes from cross-run identity while retaining the existing mtime-sensitive freshness comparison. Reuse node and edge component hashes during incremental-versus-fresh equality checks to avoid duplicate large-table scans.
Stream discovered worker logs into content-addressed gzip files with mtime=0 under each durable campaign attempt. Record raw and compressed hashes and byte counts in index results, and independently inventory every artifact path, size, and SHA-256 in attempt.json before disposable worktree cleanup.
Verification: 117 benchmark/campaign/report unit tests; Ruff check on all four changed files; scripts/check-source-safety.sh; git diff --check. RED coverage observed missing stream_query_fingerprint/archive_measurement_log symbols and a campaign child without CBM_BENCHMARK_ARTIFACT_DIR before the implementation.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
The generated Historical performance deltas table divided latencies whenever config overrides matched, even when lifecycle outcomes differed. A rejected upstream freshness-policy cell and an accepted deferred-freshness latest cell could therefore display a speedup that was not quality comparable; single observations also lacked a row-level evidence warning.
Require both candidates to have accepted identical lifecycle decisions and equal measured overall, pair-F1, graph-fidelity, and task-success categories before computing ratios. Suppress all ratios for correctness/quality rejects or mismatched freshness decisions. Label quality-matched comparisons with fewer than three observations as descriptive only, and rename the section Quality-constrained cross-version timing.
Tests: uv run python -m unittest tests.test_summarize_benchmark_results tests.test_benchmark_campaign tests.test_benchmark_incremental_speed (112 passed); uv run ruff check scripts/summarize-benchmark-results.py tests/test_summarize_benchmark_results.py; git diff --cached --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
The capability-quality report could label a semantic lifecycle PASS even when pair_lifecycle.incremental_policy.policy_conformance_met was false. This occurred for upstream-main in both SIMILAR_TO and SEMANTICALLY_RELATED smoke cells: the post-edit judged pair was present, but the whole incremental graph was non-canonical and the recorded stale_on_incremental policy did not conform.
Make summarize_group reject non-disabled semantic cells with REJECT: freshness policy and emit a concrete finding that binds the post-edit pair result to the canonical-graph comparison. Preserve expected deferred-freshness and intentional capability-off decisions.
Also copy execution_order, execution_block, and execution_position from the immutable campaign cell into derived report inputs. This prevents paired_interleaved campaigns from being mislabeled grouped after report materialization while leaving raw result JSON immutable.
Tests: uv run python -m unittest tests.test_summarize_benchmark_results tests.test_benchmark_campaign tests.test_benchmark_incremental_speed (111 passed); uv run ruff check scripts/summarize-benchmark-results.py scripts/run-benchmark-campaign.py tests/test_summarize_benchmark_results.py tests/test_benchmark_campaign.py; git diff --cached --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previously, generated summaries treated semantic pair lifecycles as ordinary capability misses: expected deferred views and explicit capability-off controls could look like failures, pair F1 was absent or conflated with retrieval MRR, lifecycle query calls were double counted, dependency phases after mutation were omitted, and a stale fast delta could be shown as a matched-quality speedup.
Render TP/TN/FP/FN and Pair F1 for initial, post-edit, and fresh stages with relationship, task SHA, background commit/tree, freshness policy, and policy conformance. Classify a warned default deferred view as PASS: DEFERRED FRESHNESS while keeping immediate freshness visibly absent. Label individually disabled similarity/semantic controls as expected capability-off contrasts, not successful freshness deferrals. Read lifecycle indexing, dependency, response-size/latency, mutation, canonical, execution-order, and resource fields without duplicating the initial query observation.
Only calculate fresh/incremental speedup when semantic freshness is immediately matched; deferred rows retain raw latency and fresh rebuild cost but report speedup as n/a. Keep retrieval MRR separate from pair classification and explain that canonical comparison uses a matching index mode.
Verification: uv run python -m unittest tests.test_summarize_benchmark_results tests.test_benchmark_campaign tests.test_benchmark_incremental_speed (110 passed); Ruff checks; git diff --check. Retained large and six-cell MCP reports show Pair F1 and confusion witnesses, three query observations, dependency phases, expected deferred warnings, eager canonical equality, explicit disabled controls below target, and no unmatched-quality speedup claim.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previously, semantic pair fixtures ran only in synthetic repositories. A large-background run could copy ambient dirty files or duplicate this repository's tracked benchmark canaries, making cross-version quality and performance results neither isolated nor commit-auditable.
Add quality_background repo/revision/tree matrix identity and pass the exact commit to benchmark-incremental-speed.py. Materialize only tracked files with git archive, exclude dirty and untracked source state plus the benchmark task-definition subtree before overlaying one canary copy, safely validate archive paths, remove the transient archive, and retain source commit, tree, dirty status, exclusions, and copy policy. Campaign result validation rejects missing or mismatched background commit/tree identity.
The copy uses one Git archive process and linear extraction instead of one Git process per file. Runtime is O(tracked archive bytes + files), memory is bounded by tar metadata, and the source checkout/worktree registry is never modified.
Verification: uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed (76 passed); Ruff checks; git diff --check. The Git fixture proves committed VERSION=1 is copied while dirty VERSION=2, an untracked file, and tracked benchmarks/semantic-pairs-v1 canaries are excluded; campaign tests bind resolved path plus full commit/tree hashes and reject a retained revision mismatch.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previously, compact matrix expansion grouped all repetitions for one candidate/profile together, so thermal or ambient load drift could align with a configuration. Attempt records retained commands and logs but not pre/post host load, disk availability, or physical-memory context.
Add opt-in execution_order=paired_interleaved without changing legacy grouped expansion or archived plan hashes. The new order runs every candidate/profile cell once per repetition block and includes block and absolute position in cell identity. Record pre/post load averages, CPU count, physical memory when available, and campaign-filesystem total/used/free bytes in immutable command and attempt records plus the campaign environment snapshot.
The ordering pass is O(cells log cells) time and O(cells) plan memory; execution remains strictly sequential. Resource collection is constant-space and performs no background sampling or production instrumentation.
Verification: uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed (74 passed); Ruff checks; git diff --check. Tests also prove legacy grouped plans remain the default, paired plans execute repetitions as 1,1,1,1,2,2,2,2, and successful attempts retain both resource snapshots.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previously, semantic capability fixtures measured only the initial full index. They could not prove that a changed source file removed an old relationship, added a new relationship, or matched a fresh rebuild, and expected stale-on-incremental warnings appeared as undifferentiated quality misses.
Add content-addressed post-edit Go and Python sources whose single-file mutations swap the judged SIMILAR_TO and SEMANTICALLY_RELATED pairs. Retain pre/post hashes, changed paths, initial/incremental/fresh index results, pair witnesses, edge-score equality, and whole canonical-graph equality. Add incremental_semantic_freshness_eager as a one-setting profile for incremental_derived_refresh=eager.
Classify default stale_on_incremental separately from eager freshness: deferred publication conforms only when it is immediately canonical or returns the structured semantic_edges stale warning; eager publication requires fresh pair and canonical graph equality without that warning. Pair comparison remains O(observed pairs) in runtime and memory.
Verification: uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign (72 passed); Ruff checks; jq manifest parse; gofmt fixture checks; git diff --check. On optimized arm64 binary 36a1297b2571729517b96467965f476f15cd1d60e545454bc0c296f3103b9e17, default similarity/semantic edits conformed with explicit stale warnings at 40/51 ms; eager edits produced TP=1/TN=2/FP=0/FN=0 and canonical fresh equality at 72/71 ms. These are retained single pilot observations, not effect-size claims.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previously, capability-quality campaigns covered only rank and dependency retrieval, so SIMILAR_TO and SEMANTICALLY_RELATED costs had no paired precision/recall evidence. FAST remained the implicit matrix mode even for relationships that require MODERATE or FULL indexing.
Add benchmarks/semantic-pairs-v1/manifest.json with content-addressed Go structural-clone and Python control-flow-variant tasks, explicit lexical negatives, and source hashes. Extend benchmark-incremental-speed.py with unordered pair identities, TP/FP/FN/TN witnesses, precision, recall, F1, false-positive rate, unjudged-background retention, numeric query-score normalization, and response-quality gates. Propagate index_mode through compact campaign cells and document durable task/report semantics.
The bounded scorer is O(judgments + observed pairs) in runtime and memory. Natural repository edges outside the explicit judgment set remain unjudged instead of becoming false positives.
Verification: uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign (68 passed); ruff check on changed Python; jq manifest parse; gofmt fixture check; git diff --check. Optimized arm64 binary 36a1297b2571729517b96467965f476f15cd1d60e545454bc0c296f3103b9e17 produced TP=1/TN=2/FP=0/FN=0 for both enabled fixtures; each individually disabled profile removed its judged positive edge.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previous behavior: load_composition_groups() always re-expanded each matrix spec through execution-time binary validation. Rebuilding build/c/codebase-memory-mcp therefore made a completed campaign fail with candidates[3].binary_sha256 does not match, even though its immutable run records and inputs were intact.
Accept exactly one matrix_spec or plan source per composition campaign. Immutable plans pass through validate_plan() without reopening candidate executables, while matrix_spec retains strict live expansion and SHA validation. Record each source kind, absolute path, and SHA-256 in composition provenance.
Files: scripts/summarize-benchmark-results.py; tests/test_summarize_benchmark_results.py.
Verification: uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign tests.test_summarize_benchmark_results (94 passed); bash scripts/check-source-safety.sh; git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Matrix cases from scripts/benchmark-incremental-speed.py retain changed_paths at the case root, while mutation_reindex_details() only read the nested self-dogfood mutation schema. Generated reports therefore rendered 'not reported' for source mutations whose immutable artifacts contained exact paths.
Read both schemas in scripts/summarize-benchmark-results.py and derive a concrete synthetic <language> inbound-frontier definition description from scenario_metadata. Preserve the nested mutation behavior and union path values without changing raw results.
Add test_markdown_reports_matrix_changed_paths_from_case_root() in tests/test_summarize_benchmark_results.py. Verification: focused red reproduced both missing fields; focused matrix/self-dogfood tests passed; benchmark/campaign/report unittest suite passed 94/94; Ruff passed; audit-only regeneration validated 26/26 supported-language cells with zero missing or corrupt cells and rendered paths including leaf.go and shared.h. Generated reports, manifests, and logs remain ignored and unstaged.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Five-repetition reports showed stable medians alongside occasional system outliers, but the tables exposed only p50/p95 values. That made it difficult to distinguish a typical configuration cost from observed host variability and could invite unsupported confidence claims.
Add query, incremental, and full-index observation counts plus min-max ranges to summarize-benchmark-results.py. The generated section explicitly states that grouped sequential campaigns avoid concurrent contention but do not support paired or randomized effect-size confidence intervals; medians and ratios remain descriptive.
tests/test_summarize_benchmark_results.py verifies exact counts and [min, max] rendering without confidence language. Verified with uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed tests.test_summarize_benchmark_results (93 passed). The ignored 240-input composition report and SHA-256 sidecar were regenerated but not staged.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Separate incremental, rank, and dependency campaigns could not produce categorical overall-quality or Pareto rows because each report lacked the other axes. Hand-merging raw files also risked mixing different config profiles and bypassing completion hashes.
Add --composition-spec to scripts/summarize-benchmark-results.py. Named campaign entries re-expand archived matrix specs, resolve exact cell labels, require every selected completion and binary/result SHA-256 to validate, and materialize derived inputs without changing raw results. Generated Markdown records the composition-spec hash and writes a per-input SHA-256 sidecar.
Preserve execution failures as correctness rejections, but classify a missed rank/dependency target as BELOW QUALITY TARGET when that exact report explicitly disables the corresponding capability. tests/test_summarize_benchmark_results.py covers campaign grouping and composed ablation wording.
Verified with uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed tests.test_summarize_benchmark_results (92 passed). The ignored composition validated 240 retained inputs and identified latest.default.mcp as the only fully passing measured Pareto-front row; generated reports and manifests were not staged.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Rank and dependency fixtures previously required hand-authored campaign cells, which made candidate, profile, transport, and repetition coverage easy to drift and omitted the concise matrix audit trail used by incremental benchmarks.
Add top-level capability_quality expansion in scripts/run-benchmark-campaign.py. Capability cells omit frontier/exact-cap axes, invoke benchmark-incremental-speed.py with --capability-quality and --include-logs, record the fixture in identity parameters, and retain named config profiles for enabled/disabled ablations. Existing incremental expansion remains the default when the field is absent.
tests/test_benchmark_campaign.py verifies a two-transport, two-repetition rank grid has four unique cells, no frontier parameters, the expected command, and accepted exit policy. Verified with uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed tests.test_summarize_benchmark_results (90 passed).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Compact matrix expansion in scripts/run-benchmark-campaign.py hardcoded accepted_exit_codes=[0]. benchmark-incremental-speed.py intentionally exits 1 after writing a valid result when canonical graph or quality gates reject a candidate, so campaigns mislabeled measured upstream failures as missing infrastructure.
Accept a validated top-level matrix-spec exit-code list, copy it into every expanded cell identity, and keep [0] as the default. Result parsing, binary SHA-256 verification, and the structured error check still reject crashes, missing artifacts, hash mismatches, and harness-error documents.
tests/test_benchmark_campaign.py verifies [0,1] propagation. Verified with uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed tests.test_summarize_benchmark_results (89 passed). A retained two-cell upstream probe now reports REJECT: graph correctness with canonical mismatch witnesses instead of missing cells.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Compact matrix specs previously required a positive exact cap and always injected incremental_exact_max_affected_paths. A default-versus-default historical comparison therefore had to overwrite candidate policy with one shared cap.
Allow null in scenarios[].exact_caps to omit the config override, omit the capability identity key, retain exact_cap=null in parameters, and label the cell capdefault. Positive integers keep the existing explicit cap-sweep behavior. Document both forms in docs/BENCHMARK_CAMPAIGN.md.
Tests: uv run python -m unittest tests.test_benchmark_campaign -v (16 passed); uv run ruff check scripts/run-benchmark-campaign.py tests/test_benchmark_campaign.py (passed); git diff --check (passed).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
expand_matrix_spec() previously dropped candidate capability_support even though explicit campaign cells include it in identity and derived reports. Compact matrix runs could therefore revert to mode-based applicability guesses.
Validate candidates[].capability_support as a string-to-boolean object and copy its sorted values into every expanded cell before plan validation. This preserves candidate support in SHA-256 identities and generated report inputs for all matrix repetitions.
Tests: uv run python -m unittest tests.test_benchmark_campaign -v (15 passed); uv run ruff check scripts/run-benchmark-campaign.py tests/test_benchmark_campaign.py (passed); git diff --check (passed).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Capability-quality results store initial_fast_full.elapsed_ms and peak_rss_mb, but summarize_group() previously read only incremental and fresh_fast_full_after_change measurements. Generated cross-version reports therefore showed n/a for full observations and peak RSS despite retaining both values.
Collect initial full wall time and RSS in one pass over cases. Use initial full observations only when no post-mutation fresh rebuild exists, so incremental matrix semantics remain unchanged while rank and dependency fixtures expose their recorded production-build cost.
Tests: uv run python -m unittest tests.test_summarize_benchmark_results -v (28 passed); uv run ruff check scripts/summarize-benchmark-results.py tests/test_summarize_benchmark_results.py (passed); git diff --check (passed).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previous reports inferred every FAST-mode algorithm as applicable, so upstream/main was labeled rank-capable even though git grep at 2469ecc3 finds no cbm_pagerank_compute implementation. Campaign capabilities also affected identity but were not propagated into report inputs.
Add optional string-to-boolean capability_support to cell validation and SHA-256 identity. materialize_report_input() writes a deterministic derived JSON document with candidate support, source-result SHA-256, and cell identity while leaving immutable raw results unchanged. The summarizer now lets explicit unsupported metadata override mode-based applicability and accepts both dependencies and the retained auto_index_deps support key.
Tests: uv run python -m unittest tests.test_benchmark_campaign tests.test_summarize_benchmark_results -v (42 passed); uv run ruff check scripts/run-benchmark-campaign.py scripts/summarize-benchmark-results.py tests/test_benchmark_campaign.py tests/test_summarize_benchmark_results.py (passed); git diff --check (passed).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previous behavior rendered cleanup.requested=false as Cleanup 0/1, making intentionally retained audit evidence look like a failed lifecycle operation. The campaign guide also directed crash-sensitive plans and results to /tmp even though validate_campaign_root() rejects the operating-system temporary tree by default.
Add evidence_lifecycle() in scripts/summarize-benchmark-results.py to label requested disposal, deliberate retention, missing metadata, and requested cleanup failure separately. Rename the generated column to Evidence lifecycle. Point docs/BENCHMARK_CAMPAIGN.md at the ignored durable .worktrees/benchmark-campaign path. Replace the secondary nDCG reference with Järvelin and Kekäläinen's primary DOI 10.1145/582415.582418 and identify exact pairwise Pareto nondominance with Deb et al. DOI 10.1109/4235.996017.
Tests: uv run python -m unittest tests.test_summarize_benchmark_results -v (26 passed); uv run ruff check scripts/summarize-benchmark-results.py tests/test_summarize_benchmark_results.py (passed); git diff --check (passed).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previous behavior: cbm_discover_installed_deps() filtered graph qualified names for dependencies|require, but package.json extraction emits names such as <project>.package.cbmbenchdep. Automatic npm indexing therefore reported zero packages even when node_modules contained the declared source.
Parse dependencies, devDependencies, optionalDependencies, and peerDependencies from a local package.json capped at 1 MiB in src/depindex/depindex.c:364-489. Resolve only installed package directories, deduplicate names with CBMHashTable, grow retained results geometrically to the configured limit, and release files, JSON documents, tables, paths, versions, and partial arrays on every failure path.
Add the exact extracted-QN regression at tests/test_depindex.c:712. Extend scripts/benchmark-incremental-speed.py with a network-free npm quality fixture, same-result source/package/read-only judgments, atomic report publication, worker/process timing attribution, and profile-only CLI lifecycle brackets.
Verification on native Apple Silicon macOS: ASAN+UBSAN depindex 36/36; ASAN+UBSAN CLI 146/146; Python benchmark/campaign/report 84/84; Ruff check passed; production link passed with -O2 -DCBM_BIND_TS_ALLOCATOR=1. Native Windows execution was not performed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
scripts/run-benchmark-campaign.py now resolves --campaign-root through validate_campaign_root before disk checks, plan archival, environment snapshots, or cell execution. Paths equal to or below tempfile.gettempdir() fail with actionable guidance because completed results, failed-attempt logs, and resume markers would be lost after a crash or reboot.
Add --allow-temporary-campaign-root as an explicit disposable-test override. Durable campaign behavior remains unchanged: atomic fsynced writes, immutable cell identities, completion hashes, retained attempts, stale-lock recovery, process-tree termination, disk guards, audit manifests, and resumable cells.
tests/test_benchmark_campaign.py covers default rejection, explicit disposable opt-in, and a repository .worktrees campaign path. Campaign tests passed 14/14; combined benchmark/campaign/report tests passed 80/80; Ruff and git diff --check passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previous behavior: src/mcp/mcp.c:3965 handle_list_projects opened every valid cache database and emitted the complete inventory for every call. On the 046d438 real-cache probe, that path returned 81,646 result bytes and took 6,462.768 ms.
Add MCP_PROJECTS_PAGE_SIZE=50 and MCP_PROJECTS_PAGE_MAX=200, stable filename sorting, lazy valid-database traversal, limit/offset metadata, and an explicit all=true compatibility path. collect_project_db_names frees partial allocations on failure; build_project_json_entry closes lookahead stores without querying counts or Git metadata.
Keep scripts/benchmark-incremental-speed.py:1183 full-inventory scaling semantics explicit with list_projects_arguments={all:true}. Add tests/test_mcp.c:963 coverage for stable pages, next_offset, schema fields, full compatibility, environment restoration, and cache cleanup.
Verification: focused ASAN/UBSan canary 1/1; MCP suite 224/224; tool_consolidation 100/100; benchmark harness unittest 41/41; scripts/check-source-safety.sh passed; git diff --cached --check passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add render_search_projection() at scripts/summarize-benchmark-results.py:795-895 and expose retained projection JSON through --search-projection.
Place ranked-identity parity beside response bytes and token estimates so compact output cannot appear successful by size alone. Show selected property fields, post-call RSS, transport survival, and server cleanup; move long non-compact field inventories below the table for readability.
State that latency has one observation per variant, RSS is post-call rather than peak, and the comparison covers response projection only. Successful runs use Outcome: complete, Equal, Survived, and Reaped instead of visible failure labels.
Add renderer and wrong-document tests at tests/test_summarize_benchmark_results.py:27-85. The retained 30-result pilot renders 6,824 compact bytes, 7,784 selected-field bytes, and 18,374 non-compact bytes with equal identities, zero fp/sp/bt fields, and 62.9% compact byte savings.
Verification:
- uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign tests.test_summarize_benchmark_results: 78/78 passed.
- uv run ruff check scripts/summarize-benchmark-results.py tests/test_summarize_benchmark_results.py: passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add --search-projection to scripts/benchmark-incremental-speed.py. One isolated FAST fixture compares compact default, compact=true, compact=true with fields=[complexity,signature], and compact=false using a fresh MCP server per variant.
Record canonical payload/envelope bytes, deterministic token estimates, descriptive latency, post-call RSS, returned qualified names, property keys, internal fields, transport survival, and server reap state. Gate every variant on identical ranked identities, zero fp/sp/bt keys, its declared projection contract, and compact <= selected <= non-compact bytes.
Extract atomic_write_text() so list-project and projection records both flush, fsync, replace, and remove temporary output files. Auto-created work roots remain isolated and are deleted unless --keep-work-root is explicit.
Add build_search_projection_observation() coverage at tests/test_benchmark_incremental_speed.py:16-38. The four-result integration smoke with optimized binary SHA-256 ec61f92332d46af283c9e8fd0fbaae4a2df5181f70e2eb38d2088166e7515d08 measured 1,240 compact bytes, 1,368 selected-field bytes, and 2,780 non-compact bytes with identity parity, no internal fields, live transports, reaped servers, and removed fixtures.
Verification:
- uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign tests.test_summarize_benchmark_results: 76/76 passed.
- uv run ruff check scripts/benchmark-incremental-speed.py tests/test_benchmark_incremental_speed.py: passed.
- make -f Makefile.cbm cbm: built the -O2 production binary.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add render_list_projects_scaling() at scripts/summarize-benchmark-results.py:795-878 and expose retained JSON through --list-projects-scaling.
Render requested/returned counts, payload bytes, deterministic token estimates, descriptive call latency, post-call RSS, transport survival, server cleanup, and transient fixture disk use in separate columns. Successful cells use Outcome: complete, Survived, and Reaped instead of visible failure labels.
State the one-observation latency limitation, identify post-call RSS as non-peak, retain the experiment claim boundary, and include run ID, binary SHA-256, and fixture cleanup evidence. Generated Markdown remains outside Git.
Add clarity and wrong-document tests at tests/test_summarize_benchmark_results.py:27-83. The retained 1/16/64 pilot renders 284/3,239/12,695 bytes, 71/810/3,174 estimated tokens, and 15.149/25.779/82.118 ms without attributing the external combined multi-tool figure to list_projects alone.
Verification:
- uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign tests.test_summarize_benchmark_results: 75/75 passed.
- uv run ruff check scripts/summarize-benchmark-results.py tests/test_summarize_benchmark_results.py: passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add --list-projects-scaling to scripts/benchmark-incremental-speed.py. The mode indexes one minimal fast-mode seed, snapshots and rekeys valid SQLite project databases, and starts a fresh MCP server for every strictly increasing project count.
Record canonical response bytes, deterministic byte/4 token estimates, MCP envelope bytes, call latency, post-call resident memory, fixture disk bytes, transport survival, and server/thread reap status. Label post-call RSS as non-peak and state that the experiment cannot attribute combined multi-tool response size.
Gate fixture creation at scripts/benchmark-incremental-speed.py:540-573 against --list-project-fixture-max-mb plus a max(2 GiB, 5% free-space) reserve. Write completed or failed JSON atomically and remove the temporary output path in a finally block. Auto-created work roots are removed while --keep-work-root preserves them.
Add count validation, snapshot rekeying, and disk-budget tests in tests/test_benchmark_incremental_speed.py:16-76. The 1→2 integration smoke used optimized binary SHA-256 379585ab9be20eade9edf6aff2e1cea318aceb1659aa3f6cbfa3f51bc8dc5fb4, returned 1/1 and 2/2 projects, kept transport alive, reaped both servers, and removed its work root.
Verification:
- uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign tests.test_summarize_benchmark_results: 73/73 passed.
- uv run ruff check scripts/benchmark-incremental-speed.py tests/test_benchmark_incremental_speed.py: passed.
- make -f Makefile.cbm cbm: built the -O2 production binary.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previous behavior:
- McpClient.__exit__() in scripts/benchmark-incremental-speed.py:807-822 closed stdin and waited for the server but did not join stdout_thread or stderr_thread, close the output streams, or clear retained process/thread references.
Changes:
- Join both reader threads after process exit, close stdout and stderr, clear proc/stdout_thread/stderr_thread, and raise MCP reader thread did not stop after process exit when a normal teardown cannot reap a reader.
- Preserve the existing wait, terminate, then kill escalation sequence.
- Add test_mcp_client_exit_reaps_process_streams_and_reader_threads in tests/test_benchmark_incremental_speed.py:15-60.
Resource lifecycle:
- Repeated benchmark cells now synchronously release the subprocess pipes and reader-thread references before the next cell starts.
- The bounded second join runs only when a reader remains alive after the initial five-second join.
Verification:
- Red: the focused unit test failed because process.stdout.closed was false.
- Green: focused unit test passed; benchmark, campaign, and summarizer suites passed 70/70.
- uv run ruff check passed for scripts/benchmark-incremental-speed.py and tests/test_benchmark_incremental_speed.py.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add --capability-quality rank in scripts/benchmark-incremental-speed.py. create_rank_quality_repo() builds eight lexical decoys and one structurally central Python function with eight callers; run_rank_quality_oracles() records the central symbol's RR, Hit@1, Hit@5, and nDCG@5 under an isolated cache and process lifecycle.
Make score_ranked_relevance() compute reciprocal rank across the full bounded response while retaining the configured DCG/nDCG cutoff, matching the first-correct-result MRR definition at https://trec.nist.gov/data/qa.html. Record execution_passed separately from quality_target_met so a valid ablation below the cutoff exits successfully.
Render BELOW CUTOFF and BELOW QUALITY TARGET instead of reporting rank-nine evidence as missing or broken. The focused benchmark and summarizer suites pass 57 tests, Ruff reports no violations, and git diff --check is clean.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add render_mcp_surface_parity() in scripts/summarize-benchmark-results.py and a --mcp-surface-parity JSON input that can be combined with benchmark inputs. The generated table separates pure classic, streamlined pre-reveal, and the same streamlined process after _hidden_tools, including advertised names, handler recognition, schema parity, tools/list payload size, and notification evidence.
Label empty-argument dispatch probes as addressability evidence rather than behavioral equality, identify the remaining capability-fixture requirement, and reject non-parity documents. tests/test_summarize_benchmark_results.py covers all three rows, alias-schema wording, and invalid input; 52 focused unittest cases pass and Ruff reports no violations.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add score_ranked_relevance() with a bounded cutoff and explicit positive relevance judgments. Compute first-relevant reciprocal rank, Hit@1, Hit@5, DCG, ideal DCG, and nDCG@k in O(k*j) time for cutoff k and judgment count j.
Extend score_quality_oracles() to accept graded judgment objects while preserving existing tuple-based substring expectations. Missing judged evidence contributes zero; probes without judgments remain N/A. Keep strict task and canonical-graph gates independent from descriptive retrieval scores.
Aggregate nDCG@5 and judgment counts in summarize-benchmark-results.py without hiding MRR or Hit@k. Generated methodology links NIST's TREC QA reciprocal-rank definition and the TREC Complex Answer Retrieval graded-ranking evaluation.
Tests: uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed tests.test_summarize_benchmark_results (62 passed)
Static: uv run ruff check scripts/benchmark-incremental-speed.py scripts/summarize-benchmark-results.py tests/test_benchmark_incremental_speed.py tests/test_summarize_benchmark_results.py
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Replace the hardcoded mode=fast index request with --index-mode fast|moderate|full and propagate the selected mode through CLI, MCP, matrix, self-dogfood, incremental, and fresh-reference runs.
Record capability_applicability in every report. FAST cells now mark SIMILAR_TO and SEMANTICALLY_RELATED quality N/A because src/pipeline/pipeline.c skips both passes in CBM_MODE_FAST; retained artifacts without mode metadata remain unknown.
Add similarity_disabled, semantic_edges_disabled, git_history_disabled, and http_links_disabled beside rank_disabled and dependency_disabled. Each controlled profile changes exactly one global key from src/pipeline/pipeline.h or src/pagerank/pagerank.h.
Generated Markdown includes an Algorithm-quality applicability table so an unavailable algorithm cannot be scored as a pass, failure, or zero benefit.
Tests: uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed tests.test_summarize_benchmark_results (58 passed)
Static: uv run ruff check scripts/benchmark-incremental-speed.py scripts/summarize-benchmark-results.py tests/test_benchmark_incremental_speed.py tests/test_summarize_benchmark_results.py
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add --mcp-surface-parity to benchmark-incremental-speed.py. The bounded probe starts isolated classic and streamlined MCP processes with CBM_AUTO_INDEX=false, measures tools/list latency and payload size, hashes every input schema, observes notifications/tools/list_changed, and compares streamlined state before and after _hidden_tools reveal.
Probe hidden classic names with empty argument objects to verify dispatcher recognition without indexing or mutating a repository. Report this separately from advertised name/schema parity so handler recognition is not presented as end-to-end behavioral equivalence. Account for get_code versus get_code_snippet property and required-field contracts explicitly.
Installed binary 692b78434d86 produced 4/15 advertised and 15/15 dispatch-recognized classic names before reveal; after reveal it contained all 15 classic names with zero schema-hash mismatches and observed tools/list_changed. The retained JSON is /private/tmp/cbm-mcp-surface-parity-20260715-v3.json and is not committed.
Tests: uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed tests.test_summarize_benchmark_results (54 passed)
Static: uv run ruff check scripts/benchmark-incremental-speed.py scripts/summarize-benchmark-results.py tests/test_benchmark_incremental_speed.py tests/test_summarize_benchmark_results.py
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add the dependency_disabled profile so benchmark cells can turn off only auto_index_deps without conflating rank, similarity, semantic-edge, git-history, or HTTP-link passes.
Capture prof phase=index_repository sub=dep_auto_index markers and dependencies_indexed counts in build_index_result(), then normalize both current and retained result shapes in summarize-benchmark-results.py. Generated Markdown now distinguishes explicit enabled/disabled, observed, unsupported, and unknown states and reports dependency phase p50 values plus sample counts.
Reference NIST's TREC reciprocal-rank definition and Kalibera/Jones effect-size confidence interval guidance in generated methodology text. One- and three-observation pilots remain explicitly unqualified rather than receiving invented confidence intervals.
Tests: uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed tests.test_summarize_benchmark_results (52 passed)
Static: uv run ruff check scripts/benchmark-incremental-speed.py scripts/summarize-benchmark-results.py tests/test_benchmark_incremental_speed.py tests/test_summarize_benchmark_results.py
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Create a per-run SMOKE_STATE in scripts/smoke-test.sh and export CBM_CACHE_DIR and XDG_CONFIG_HOME before invoking the installed binary. Keep CLI stderr inside that cleanup tree so smoke runs neither inherit nor leave user cache/config artifacts.
Extend mcp_run with an explicit tool-mode argument: streamlined scenarios validate progressive disclosure while index_repository and get_code_snippet scenarios select classic mode. This prevents a persisted tool_mode=classic setting from producing "streamlined tool 'get_code' not found" in Phase 5.
Tests: rtk bash -n scripts/smoke-test.sh; installed-binary smoke completed all phases with 9,039 nodes, 24,079 edges, MCP reveal/classic calls, installer lifecycle, security, and no-orphan checks passing.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add config_signature() and historical_delta_rows() to scripts/summarize-benchmark-results.py. Match baselines to latest candidates by exact config overrides and render baseline/latest ratios for incremental reindex, post-mutation fresh rebuild, and query latency beside both quality scores and the baseline correctness gate.
Add a matching-capability regression fixture in tests/test_summarize_benchmark_results.py. Verified 24 report/campaign tests and Ruff. The generated combined report now calculates 2.87x default, 2.73x rank-off, and 2.36x all-optional-off incremental gains over the 24ba checkpoint without concealing rejected upstream/trajectory quality.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add mutation_reindex_details() to scripts/summarize-benchmark-results.py and render one per-candidate/scenario table row with the source mutation description, changed paths, publication route/reason, end-to-end incremental p50, internal indexing-work p50, post-mutation fresh rebuild p50, speedup, and canonical equality.
Pin the route_handler rendering contract in tests/test_summarize_benchmark_results.py. Verified 23 summarizer/campaign tests and Ruff. Generated final-performance-quality-report-v2.md directly from the six retained July 14 result JSON files while leaving the original report and campaign manifests unchanged.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add frontier_crossover_rows() to scripts/summarize-benchmark-results.py so each frontier pairs the largest configured fallback cap with the smallest exact cap. Report end-to-end p50, indexed-work p50, profiling-only RSS p50, fresh-full p50, the exact/fallback latency ratio, and a documented five-percent tie band.
Extend tests/test_summarize_benchmark_results.py with a three-cap fixture that proves the nearest fallback/exact pair is selected and rendered. Verified 22 benchmark summarizer/campaign tests and Ruff; regenerated the preserved 318-cell campaign with 318 complete and zero missing, corrupt, duplicate, or unplanned cells.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add --matrix-spec to scripts/run-benchmark-campaign.py. Deterministically expand candidate revision/binary/build, capability profile, transport, scenario, frontier size, exact cap, and repetition into immutable cells; archive both the source spec and expanded plan before execution.
Include structured parameters and benchmark-incremental-speed.py SHA-256 in cell identity evidence. Keep exact command arrays, binary SHA-256 validation, atomic completion, retained attempts, disk guards, and report/manifests on the existing campaign path.
Teach frontier_coverage_gate() to accept deliberately configured low-cap cells only when they publish containment/full, report frontier_too_large, include affected_paths_truncated=true, and preserve the canonical graph.
Validated with 43 Python tests plus 15 subtests and Ruff. Real -O2 smoke cells proved one-attempt completion/resume and configured cap=4 versus five affected Go files produced incremental_containment with canonical equality and cleanup.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Define CBM_PIPELINE_DELTA_REASON_SCOPED_LSP_GAP in src/pipeline/pipeline_internal.h and use it when JavaScript, TypeScript, TSX, PHP, C#, Java, Kotlin, or Rust require the existing correctness-preserving full rebuild. Report affected_paths_truncated=false instead of claiming frontier_too_large at 1/64 affected paths.
Extend scripts/benchmark-incremental-speed.py with fixtures for all 13 languages wired through cbm_pxc_has_cross_lsp(). Gate Go, C, C++, CUDA, and Python on observed exact-frontier expansion; gate the other eight on publish_kind=full and exact_reason=scoped_lsp_gap. A fixture that does not exercise its declared contract now fails the matrix.
Validated with the -O2 13-language matrix (13/13 contract and canonical graph gates, cleanup=true), ASan+UBSan pipeline suite (376 passed), language-contract suite (35 passed; breadth checked 159 grammars), benchmark Python tests (39 passed, 15 subtests), and Ruff.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add Go, Python, and C-header inbound-frontier fixtures to scripts/benchmark-incremental-speed.py with an explicit --frontier-files denominator and single-definition mutations.
Teach build_index_result() to accept peak_mb from mem.phase, pipeline.done, or incremental.done. Emit terminal RSS fields from cbm_pipeline_run() and log_incremental_done() only while cbm_profile_active is true, preserving normal-run logging and avoiding memory queries outside experiment mode.
Validated with the -O2 CLI matrix (3/3 canonical graph gates, cleanup=true, peak RSS present on initial/incremental/fresh routes), the ASan+UBSan C runner (375 passed), benchmark Python tests (35 passed, 5 subtests), and Ruff.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Compute retrieval MRR, canonical-graph fidelity, applicable-probe task success, and their equal-weight geometric mean in scripts/summarize-benchmark-results.py. Correctness failures remain hard REJECT decisions and cannot be averaged away.
Render named oracle criteria, expected evidence, rank, Hit@1/Hit@5, raw retrieval/graph/scenario counts, capability profiles, and explicit Pareto eligibility or dominance reasons. Link the generated Markdown to the NIST TREC reciprocal-rank definition at https://trec.nist.gov/pubs/trec14/papers/hummingbird.qa.robust.tera.pdf.
Validation: 32 benchmark/campaign/report unit tests passed; Ruff and scripts/check-source-safety.sh passed; preserved campaign JSON rendered successfully without modifying source artifacts.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add --config-profile choices in scripts/benchmark-incremental-speed.py and record the selected profile beside its expanded config_overrides map.
Define minimal_indexing as auto_index_deps=false plus rank_enabled, similarity_enabled, semantic_edges_enabled, githistory_enabled, and httplinks_enabled set to false. Keep repeated --config values higher priority for controlled ablations.
Document the distinction between optional_graph_disabled and minimal_indexing in docs/BENCHMARK_CAMPAIGN.md. Validate with 29 Python unit tests, Ruff, and scripts/check-source-safety.sh.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add correctness_findings() in scripts/summarize-benchmark-results.py to report canonical count mismatches, bounded left/right witnesses, and failed task-oracle expectations without copying unbounded payloads into Markdown.
Correct the response-size footnote to identify the exact default tool-response payload and disclose the one-case/five-oracle sample basis. Cover the generated evidence and measurement wording in tests/test_summarize_benchmark_results.py; 27 benchmark unit tests and Ruff pass.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Parse log= from the index.supervisor.profile_log line emitted by src/mcp/index_supervisor.c and stream measurement markers from that .worker-<pid>.log before trying the unrelated response logfile path. This matches the retained-log contract observed in the profiled arm64 pilot.
The regression fixture supplies a missing response logfile and a valid supervisor worker log, then verifies peak_rss_mb=320 and pipeline_done=81. All 27 focused pytest cases and Ruff pass.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Set CBM_PROFILE=1 in build_env() for every candidate so index_supervisor.c preserves successful worker logs long enough for build_index_result() to stream mem.phase, pipeline.done, and incremental.done markers. Document that the profiling configuration is consistent across revisions and remains part of the auditable benchmark environment.
Validated with 27 pytest cases across the benchmark, summarizer, and campaign suites; Ruff and git diff --check pass.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Andrew Hundt