Previously scripts/run-benchmark-campaign.py required callers to hand-build binaries, author a plan or matrix spec, and choose a campaign root. Timestamp-only artifact names did not distinguish source time from execution time, repeated preparation rebuilt candidates, and absolute checkout paths prevented portable resume.
Add no-argument quick and --full campaign presets, peeled commit/tree identities, detached candidate worktree reuse, canonical Makefile.cbm CFLAGS_PROD capture, streamed production build logs, binary hashing, clean tracked-tree checks, and tamper-aware candidate cache records. Name new campaign evidence with v0001 definition, runset content ID, explicit commit time, and explicit generation time; identity_version=2 normalizes canonical path mappings while legacy cells keep their prior identity document.
scripts/benchmark-incremental-speed.py now uses explicit sortable UTC filenames for retained failure and scaling evidence. tests/test_benchmark_campaign.py covers tag peeling, worktree reuse, stale binary rebuild, quick/full matrix construction, path remapping, bounded names, clean-tree policy, and legacy identity compatibility.
Verification: uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed tests.test_summarize_benchmark_results tests.test_autotune (159 passed); ruff check on all three changed files; git diff --check; bash scripts/check-source-safety.sh.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
scripts/benchmark-incremental-speed.py validates each candidate cache before setting CBM_CACHE_DIR. It rejects the inherited live cache and directories containing an existing project database, preventing cross-version recovery logic from renaming or mutating user data.
scripts/summarize-benchmark-results.py adds separate bold graph, result/quality, and run/lifecycle error columns to the headline table. Requested cleanup failures now produce REJECT: lifecycle cleanup, so historical speedups and Pareto selection remain unavailable to invalid runs.
tests/test_benchmark_incremental_speed.py and tests/test_summarize_benchmark_results.py reproduce both prior contracts. Verification: 149 benchmark/campaign/report/autotune unittests; Ruff format check on four changed files; scripts/check-source-safety.sh.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previous behavior: each query oracle measured one default-format call and one JSON quality call, then reports treated the first call as query latency. One-shot schema context, watcher startup, and cache warming could land on either call, producing apparent 16 ms versus 261 ms regressions from the same underlying path.
Run three canonical JSON trials after the default-format call for both MCP and CLI. Retain every elapsed value and canonical payload hash, report min/median/max plus byte-equality, and preserve the first JSON timing for backwards compatibility. The summarizer now labels one-shot default latency separately and uses the repeated JSON median for cross-candidate query tables.
Add fake-client timing/hash coverage in tests/test_benchmark_incremental_speed.py and cold-versus-repeated aggregation coverage in tests/test_summarize_benchmark_results.py. Verification: 144 benchmark, campaign, and report tests pass; ruff format reports all four changed files formatted; git diff --check passes.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previous behavior: a matrix profile could set capabilities such as auto_index_deps=false as descriptive metadata while leaving config_overrides empty. The archived plan then claimed an ablation that the benchmark command never applied; retained worker timings exposed dependency and PageRank phases still running.
Make expand_matrix_spec fail closed when config_profile=default claims a disabled capability without the identical false config override. Named profiles remain compatible because their canonical settings are resolved by benchmark-incremental-speed.py.
Add test_default_profile_rejects_disabled_capability_without_config_override in tests/test_benchmark_campaign.py. Verification: all 26 campaign-runner tests pass and git diff --check passes.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previous behavior: self-dogfood graph validation inferred every stale derived view from task-specific MCP warnings. A changed-file query does not mention semantic_edges, so exact incremental runs with intentionally deferred semantic edges were rejected as canonical graph failures despite derived_view_state recording semantic_edges as stale.
Add persisted_stale_views() in scripts/benchmark-incremental-speed.py to read the canonical SQLite freshness ledger through a read-only connection and union it with response-local warnings. Missing legacy derived_view_state tables remain compatible, while core node, edge, and file-hash mismatches are still rejected after excluding only declared SEMANTICALLY_RELATED rows.
Add tests/test_benchmark_incremental_speed.py coverage for project-scoped stale/complete rows and retain tests/test_mcp.c coverage proving moderate incremental_exact publication persists semantic_edges=stale and query_graph reports it for semantic queries.
Verification: 76 benchmark harness tests passed; 262 ASan/UBSan MCP tests passed; focused exact-moderate MCP test passed; git diff --check passed. The repository-wide lint-format target still reports pre-existing formatting violations outside these changes.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add run_mcp_surface_parity() coverage for classic discovery, streamlined discovery before _hidden_tools, and relisting in the same process after reveal. Compare complete tools/list contracts, validation-equivalent get_code aliases, capability outcomes, bounded hidden-handler recognition, and notifications/tools/list_changed without presenting dispatch recognition as behavioral proof.
Make McpClient cleanup record subprocess return codes and require both reader threads and server processes to be reaped. Close SQLite snapshot and cloned-fixture connections explicitly so ResourceWarning-as-error runs detect ownership regressions.
Render capability outcomes before tool counts in scripts/summarize-benchmark-results.py, retain response bytes and single-observation latency with explicit evidence limits, and document the parity mode in docs/BENCHMARK_CAMPAIGN.md.
Verification: 141 warning-clean benchmark/campaign/summarizer tests; optimized build/c/codebase-memory-mcp parity smoke passed for 12 capability groups with full post-reveal classic contract parity and reaped lifecycle; ruff check; source-safety; git diff --cached --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Replace scripts/autotune.py's global config mutation, mutable local-repository lookup, architecture resource request, and source-tree result file with build_matrix_spec() over scripts/run-benchmark-campaign.py.
Bind each plan to the full revision, binary SHA-256, compiler, flags, harness hash, candidate-default/rank-disabled controls, and PageRank parameter profiles. Store content-addressed manifests under the ignored durable campaign root and transfer execution with os.execv so the shared runner owns isolation, resumption, logs, and cleanup.
Add tests/test_autotune.py for shared-runner plan validation, paired profile expansion, and absence of the removed resources/read, atexit, Path.home, and database-deletion paths.
Verification: uv run python -W error::ResourceWarning -m unittest tests.test_autotune (3 tests); ruff check scripts/autotune.py tests/test_autotune.py; git diff --cached --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
summarize_group now derives a Pareto workload identity from scenario, report mode, index mode, corpus metadata, and task-set hashes. mark_pareto_frontier compares candidates only when that identity matches, preventing small semantic canaries from dominating full-repository measurements.
Add test_pareto_does_not_compare_different_workloads in tests/test_summarize_benchmark_results.py. Verified with 140 benchmark campaign/measurement/report tests, ruff check, and regenerated retained composition output.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Flake audit: the whole corpus has exactly two wall-clock test deadlines —
the EOF-shutdown alarm(5) and the issue-832 supervisor-spawn alarm(60),
both in the mcp suite (the latter already flaked once under heavy local
load). Serializing mcp costs ~6s of tail and closes the starvable-deadline
class completely; every remaining parallel-wave suite is deadline-free.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
First CI validation round of the parallel runner, three findings:
- Windows failed ALL 104 suites with "0 passed, 1 failed" each: the CRT
emits --list-suites lines as CRLF, so every dispatched name carried a
trailing CR ("arena\r" is an unknown suite). The union guard caught it
loudly, exactly as designed. The driver now strips CR when writing the
suite list.
- The ubuntu-gcc legs had 3 real failures in the cli suite: it spawns
subprocesses with fixed deadlines, and a fully saturated 4-core runner
starves those deadlines into flakes. Deadline-sensitive suites (cli,
subprocess, watcher, incremental, httpd, ui, index_resilience, and the
stack_overflow family) now run SEQUENTIALLY after the parallel wave on a
quiet machine — same suites, same tests, same gates, only the schedule
differs; the union guard checks the combined result set.
- Failing suites now print every FAIL site with context (the tail-30 of a
long suite log hid which tests actually failed on CI).
Local: totals unchanged (6,361 passed / 0 failed / 1 skipped), union guard
clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The runner executed all 104 suites sequentially (~10 min locally, the bulk
of every CI test leg). The suites are process-isolated already (per-process
mkdtemp HOME sentinel; two full parallel runs produced zero cross-suite
failures), so the serialization was pure convention.
- test-runner --list-suites: prints every registered suite, one per line,
emitted by the SAME macro table that executes suites — the list cannot
drift from the run set by construction.
- scripts/run-tests-parallel.sh: runs each suite as its own process
(jobs = CPU count; CBM_TEST_PAR_JOBS overrides) under a ZERO-LOSS
CONTRACT: a union guard fails the gate if the set of suites that
produced a result differs from --list-suites (nothing can be silently
dropped, a newly added suite is picked up automatically); per-suite
pass/fail/skip are summed into the sequential runner's exact summary
format; any suite crash, failure, or omission exits nonzero. Per-suite
wall times are printed for balance tracking.
- make test-par: the parallel target. make test (sequential) is unchanged
and remains the escape hatch (CBM_TEST_SEQUENTIAL=1 in test.sh).
- scripts/test.sh: builds, then routes through test-par — every CI test
leg gets the speedup with zero workflow-topology change (same jobs,
same gates, same billing; the legs just finish sooner).
- The stack_overflow suite is split into a/b/c (7+6+7 of its 20 tests,
pure re-registration): as one suite it was the wall-clock critical path
of any parallel run — every other suite finished underneath its ~4
minutes.
Measured locally (Apple Silicon, ASan runner): sequential ~10 min vs
parallel 232 s, totals identical (6,361 passed / 0 failed / 1 skipped),
union guard clean. The TSan leg keeps its dedicated subset runner
(sequential) — out of scope here.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The PR CI caught two consumers the format migration missed — both parsed
tool output by its old shape and failed silently or with a wrong count:
- hook-augment built its PreToolUse additionalContext by reading the
legacy 'results' object-array from search_graph format:json; after the
json-tree reshape that key no longer exists, so the hook emitted nothing
(caught end-to-end by tests/windows/test_hook_augment.py). The parser
now walks groups -> qn_prefix/file + column-ordered rows. A new local
test feeds the parser the LIVE server envelope, so any future drift
between response shape and hook parser fails in the normal suite on
every platform — not only in the Windows CI guard. Reproduced RED
(ctx NULL, the exact CI failure) before the fix, GREEN after.
- smoke-test.sh carried six old-shape parsers (callers[N], rows[N]{,
clusters[N], semantic[N]{, and the assert_toon_table helper); the tree
headers never matched, so trace verification read 0 callers. All six
updated to the tree contract; the full smoke script passes end-to-end
locally.
cli suite 222 passed; lint clean; scripted smoke ALL PASSED.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
scripts/benchmark-incremental-speed.py previously measured Git-history and HTTP-link runtime profiles without task-quality oracles, leaving enabled/disabled cost cells unable to demonstrate user-visible graph evidence.
Add deterministic, network-free fixtures that reuse FILE_CHANGES_WITH and HTTP_CALLS from the production pipeline. The Git fixture records four fixed-date co-change commits. The HTTP fixture places a Python client and source-discovered Ktor route in distinct service namespaces so only the optional HTTP linker produces the direct caller-to-handler edge.
Query existing relationship schemas with bounded results and require the declared paths, counterpart, count, caller, handler, and method/confidence evidence in the quality judgments. Register both cases in --capability-quality and cover fixture identity plus oracle queries in tests/test_benchmark_incremental_speed.py.
Optimized binary SHA-256 2f2ceeb052f78bcdfc2b64b6175cf016315f3c3b9e28af4ec1487624e2523806 passed Git and HTTP enabled smokes; the HTTP disabled control missed the direct fetch_order-to-configureRouting edge as required. Every isolated work root was removed.
Verification: 139 benchmark/campaign/report unittests passed; Ruff, scripts/check-source-safety.sh, git diff --check, and optimized MCP enabled/disabled smoke cases passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
mutation_reindex_details() interpreted candidate-default stale-warning metadata without considering an explicit similarity_enabled=false or semantic_edges_enabled=false override. The semantic-pair table correctly said capability disabled while the incremental mutation table contradicted it with deferred with warning.
Pass disabled_pair_capabilities from summarize_group() into mutation_reindex_details() and give the explicit off state precedence over candidate freshness policy metadata. Extend test_disabled_semantic_pair_control_is_not_described_as_freshness_deferral() to cover the mutation table.
Verification: 135 benchmark/campaign/report unittests passed; Ruff, scripts/check-source-safety.sh, and git diff --check passed. Audit-only regeneration preserved 3/3 semantic-off and 3/3 similarity-off large-corpus cells with zero corrupt, missing, duplicate, or unplanned results, and both reports now render capability disabled in the pair and mutation tables.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
scan_campaign() validated completion and result hashes but trusted the artifact inventory in attempt.json. Changed bytes, a missing worker log, or an unlisted extra file therefore remained status=complete even though phase timing and lifecycle evidence no longer matched the recorded attempt.
Add validate_attempt_artifacts() in scripts/run-benchmark-campaign.py. Completed attempts now require matching cell identity and status, then rebuild the recursive artifact inventory and compare every relative path, byte count, SHA-256, file count, and total byte count. Completions without an attempt field retain historical hand-authored-plan compatibility.
Add changed, missing, and unlisted artifact regressions in tests/test_benchmark_campaign.py and document audit-only re-inventory in docs/BENCHMARK_CAMPAIGN.md. Verification: 135 benchmark/campaign/report unittests passed; Ruff, scripts/check-source-safety.sh, and git diff --check passed. Strict audit-only scans rehashed 39 capability cells, 9 semantic cells, and 9 similarity cells with zero corrupt, missing, duplicate, or unplanned results.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Semantic quality cases store incremental-versus-fresh graph comparisons under pair_lifecycle.canonical_graph. correctness_findings() only read the case-level field, so rejected upstream semantic and similarity rows incorrectly reported that no stage-level witness was recorded even though retained results contained exact counts and first differences.
Add canonical_mismatch_finding() in scripts/summarize-benchmark-results.py and use it for both case-level and pair-lifecycle comparisons. Preserve declared-stale-view context and place the intentional deferred-freshness explanation before its expected mismatch witness.
Add test_pair_lifecycle_canonical_rejection_reports_exact_graph_witness() in tests/test_summarize_benchmark_results.py. Verification: 134 benchmark/campaign/report unittests passed; Ruff, scripts/check-source-safety.sh, and git diff --check passed. Audit-only regeneration completed 39/39 capability cells plus 9/9 semantic and 9/9 similarity cells with zero missing, corrupt, duplicate, or unplanned results.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
evaluate_pair_incremental_policy() treated every unconfigured binary as stale_on_incremental. The pinned upstream binary produced immediate similarity pair freshness without a stale warning, so the evaluator rejected correct observed behavior against a policy that upstream does not expose.
Record candidate_default when incremental_derived_refresh is not explicitly configured. Classify the observed lifecycle as immediate_pair_freshness, immediate_full_freshness, deferred_with_warning, or unreported_stale; retain strict contract checks for explicit eager and deferred overrides. Separate pair freshness from whole-canonical-graph equality so each category remains visible.
Document cross-version default handling in docs/BENCHMARK_CAMPAIGN.md and extend tests/test_benchmark_incremental_speed.py with deferred, immediate-pair, unreported-stale, and explicit-eager cases.
Verification: uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign tests.test_summarize_benchmark_results (133 tests); uv run ruff check on benchmark/report files; bash scripts/check-source-safety.sh; git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Large c_new_leaf runs with the default stale_on_incremental policy passed every mutation oracle but differed from fresh rebuilds by 99 SEMANTICALLY_RELATED rows. summarize-benchmark-results.py labeled those disclosed stale views as REJECT: graph correctness, hiding the intended latency/freshness policy.
benchmark-incremental-speed.py now unions structured stale_with_warning views and reruns canonical comparison after excluding only SEMANTICALLY_RELATED rows when semantic_edges is explicitly stale. The declared-stale gate passes only when every remaining node, edge, property, and file hash equals the fresh rebuild; non-semantic or undeclared differences still fail.
summarize-benchmark-results.py reports Core graph separately from Full graph freshness and emits PASS: DECLARED STALE VIEWS for a verified deferred view. docs/BENCHMARK_CAMPAIGN.md documents the gate and tests cover stale-view collection, exact exclusion scope, core mismatches, decision text, and table categories.
Verification: uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign tests.test_summarize_benchmark_results (133 tests); uv run ruff check on changed Python files; bash scripts/check-source-safety.sh; git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
scripts/benchmark-incremental-speed.py previously wrote rank_refresh=stale_on_exact for every matrix cell, including profiles labeled default. That changed candidate behavior and invalidated comparisons against binaries with different compiled policies.
Add RANK_REFRESH_CANDIDATE_DEFAULT and apply_rank_refresh_override() so default runs make no rank_refresh config write. Record the requested policy and whether an override was applied in synthetic, matrix, self-dogfood, and capability-quality reports; retain explicit eager, stale_on_exact, and stale_on_incremental experiments.
Document the comparison contract in docs/BENCHMARK_CAMPAIGN.md and cover CLI default, no-write, and explicit-write behavior in tests/test_benchmark_incremental_speed.py.
Verification: uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign (90 tests); uv run ruff check scripts/benchmark-incremental-speed.py tests/test_benchmark_incremental_speed.py; bash scripts/check-source-safety.sh; git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add self_dogfood workload expansion in scripts/run-benchmark-campaign.py with exact repository revision/tree identities, candidate-scoped profiles, and result rejection on background mismatches.
Make create_self_dogfood_worktree check out the declared commit, retain pre/post mutation SHA-256 values, and add c_new_leaf for a real indexed create-file route alongside high-fanout fallback scenarios.
Document the JSON contract in docs/BENCHMARK_CAMPAIGN.md. Verify with 125 benchmark/report unit tests, Ruff, source-safety, and an optimized c_new_leaf smoke: incremental_exact 367 ms, fresh rebuild 4456 ms, canonical equality 1/1, cleanup complete.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Capability-quality reports stored canonical equality inside pair_lifecycle, while summarize_group() read only case-level canonical_graph. Eager semantic runs therefore displayed Graph fidelity n/a and Overall quality n/a despite Pair F1=1.0, task success, and an exact incremental-versus-fresh graph.
Include lifecycle canonical equality only when the recorded policy requires immediate freshness; deferred views remain non-applicable instead of becoming false failures. Define result quality as Pair F1 or Retrieval MRR when one is measured, or their arithmetic mean when both are measured, then retain the equal-weight geometric mean across result quality, graph fidelity, and task success.
Verification: focused RED produced graph_fidelity_score=None for a perfect eager lifecycle; GREEN produces Pair F1=1.0, graph fidelity=1.0, task success=1.0, and overall quality=1.0 while the deferred test remains valid. Full benchmark/campaign/report suite: 120 passed; Ruff; scripts/check-source-safety.sh; git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
The stable graph fingerprint added in 5cb6e2eb reused raw canonical qualified names and node properties. Independent benchmark roots therefore produced different hashes even when retained logs showed identical source hashes, node/edge counts, semantic edge counts, similarity edge counts, and judged scores.
Add path-normalized SQL projections that strip only the exact project prefix from qualified names. Hash structural node identity, structural edge identity, source path/content/size, and semantic score/jaccard rows as separate components. Exclude arbitrary node properties and file mtimes from cross-run identity.
Keep compare_canonical_graph() on the original full canonical rows, including properties and mtimes, so incremental-versus-fresh correctness remains strict. Stable experiment identity no longer short-circuits or weakens that gate.
Verification: RED SQLite fixture produced unequal hashes for identical relative graphs under random-root-a and random-root-b; GREEN makes them equal and still detects a semantic score change from 0.873 to 0.811. Full benchmark/campaign/report suite: 119 passed; Ruff; scripts/check-source-safety.sh; git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
scripts/summarize-benchmark-results.py derived pair quality decisions from top-level oracles and ignored initial/incremental/fresh lifecycle failures when those aliases were absent. A BELOW QUALITY TARGET row could therefore have no finding, and Markdown replaced the empty evidence with 'All applicable canonical-graph and task-oracle checks passed.'
Treat initial and fresh pair stages as required and the post-edit stage as required only when the recorded freshness policy expects immediate results. Emit stage names, TP/TN/FP/FN counts, and absent-expected or unexpected-positive counts. Skip failure wording for explicit capability-off controls and preserve deferred post-edit semantics.
Only use the all-checks-passed fallback for PASS decisions. Non-pass rows without a recorded witness now state that limitation and direct readers to retained raw results.
Verification: focused RED returned PASS with empty findings for an initial FN=1; GREEN returns BELOW QUALITY TARGET with 'Initial semantic-pair quality missed' and TP=0, TN=2, FP=0, FN=1. Full benchmark/campaign/report suite: 118 passed; Ruff; scripts/check-source-safety.sh; git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
scripts/benchmark-incremental-speed.py previously loaded complete canonical graph row lists and mismatch sets, requiring O(nodes + edges) Python memory. Supervisor worker logs remained inside disposable work roots, so successful automatic cleanup removed the raw phase and peak-memory evidence.
Hash ordered query rows as length-delimited SHA-256 streams and diagnose mismatches with a two-cursor merge, both using O(1) Python auxiliary memory. Record initial, incremental, and fresh node/edge/source-content fingerprints; exclude volatile mtimes from cross-run identity while retaining the existing mtime-sensitive freshness comparison. Reuse node and edge component hashes during incremental-versus-fresh equality checks to avoid duplicate large-table scans.
Stream discovered worker logs into content-addressed gzip files with mtime=0 under each durable campaign attempt. Record raw and compressed hashes and byte counts in index results, and independently inventory every artifact path, size, and SHA-256 in attempt.json before disposable worktree cleanup.
Verification: 117 benchmark/campaign/report unit tests; Ruff check on all four changed files; scripts/check-source-safety.sh; git diff --check. RED coverage observed missing stream_query_fingerprint/archive_measurement_log symbols and a campaign child without CBM_BENCHMARK_ARTIFACT_DIR before the implementation.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
The generated Historical performance deltas table divided latencies whenever config overrides matched, even when lifecycle outcomes differed. A rejected upstream freshness-policy cell and an accepted deferred-freshness latest cell could therefore display a speedup that was not quality comparable; single observations also lacked a row-level evidence warning.
Require both candidates to have accepted identical lifecycle decisions and equal measured overall, pair-F1, graph-fidelity, and task-success categories before computing ratios. Suppress all ratios for correctness/quality rejects or mismatched freshness decisions. Label quality-matched comparisons with fewer than three observations as descriptive only, and rename the section Quality-constrained cross-version timing.
Tests: uv run python -m unittest tests.test_summarize_benchmark_results tests.test_benchmark_campaign tests.test_benchmark_incremental_speed (112 passed); uv run ruff check scripts/summarize-benchmark-results.py tests/test_summarize_benchmark_results.py; git diff --cached --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
The capability-quality report could label a semantic lifecycle PASS even when pair_lifecycle.incremental_policy.policy_conformance_met was false. This occurred for upstream-main in both SIMILAR_TO and SEMANTICALLY_RELATED smoke cells: the post-edit judged pair was present, but the whole incremental graph was non-canonical and the recorded stale_on_incremental policy did not conform.
Make summarize_group reject non-disabled semantic cells with REJECT: freshness policy and emit a concrete finding that binds the post-edit pair result to the canonical-graph comparison. Preserve expected deferred-freshness and intentional capability-off decisions.
Also copy execution_order, execution_block, and execution_position from the immutable campaign cell into derived report inputs. This prevents paired_interleaved campaigns from being mislabeled grouped after report materialization while leaving raw result JSON immutable.
Tests: uv run python -m unittest tests.test_summarize_benchmark_results tests.test_benchmark_campaign tests.test_benchmark_incremental_speed (111 passed); uv run ruff check scripts/summarize-benchmark-results.py scripts/run-benchmark-campaign.py tests/test_summarize_benchmark_results.py tests/test_benchmark_campaign.py; git diff --cached --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previously, generated summaries treated semantic pair lifecycles as ordinary capability misses: expected deferred views and explicit capability-off controls could look like failures, pair F1 was absent or conflated with retrieval MRR, lifecycle query calls were double counted, dependency phases after mutation were omitted, and a stale fast delta could be shown as a matched-quality speedup.
Render TP/TN/FP/FN and Pair F1 for initial, post-edit, and fresh stages with relationship, task SHA, background commit/tree, freshness policy, and policy conformance. Classify a warned default deferred view as PASS: DEFERRED FRESHNESS while keeping immediate freshness visibly absent. Label individually disabled similarity/semantic controls as expected capability-off contrasts, not successful freshness deferrals. Read lifecycle indexing, dependency, response-size/latency, mutation, canonical, execution-order, and resource fields without duplicating the initial query observation.
Only calculate fresh/incremental speedup when semantic freshness is immediately matched; deferred rows retain raw latency and fresh rebuild cost but report speedup as n/a. Keep retrieval MRR separate from pair classification and explain that canonical comparison uses a matching index mode.
Verification: uv run python -m unittest tests.test_summarize_benchmark_results tests.test_benchmark_campaign tests.test_benchmark_incremental_speed (110 passed); Ruff checks; git diff --check. Retained large and six-cell MCP reports show Pair F1 and confusion witnesses, three query observations, dependency phases, expected deferred warnings, eager canonical equality, explicit disabled controls below target, and no unmatched-quality speedup claim.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previously, semantic pair fixtures ran only in synthetic repositories. A large-background run could copy ambient dirty files or duplicate this repository's tracked benchmark canaries, making cross-version quality and performance results neither isolated nor commit-auditable.
Add quality_background repo/revision/tree matrix identity and pass the exact commit to benchmark-incremental-speed.py. Materialize only tracked files with git archive, exclude dirty and untracked source state plus the benchmark task-definition subtree before overlaying one canary copy, safely validate archive paths, remove the transient archive, and retain source commit, tree, dirty status, exclusions, and copy policy. Campaign result validation rejects missing or mismatched background commit/tree identity.
The copy uses one Git archive process and linear extraction instead of one Git process per file. Runtime is O(tracked archive bytes + files), memory is bounded by tar metadata, and the source checkout/worktree registry is never modified.
Verification: uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed (76 passed); Ruff checks; git diff --check. The Git fixture proves committed VERSION=1 is copied while dirty VERSION=2, an untracked file, and tracked benchmarks/semantic-pairs-v1 canaries are excluded; campaign tests bind resolved path plus full commit/tree hashes and reject a retained revision mismatch.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previously, compact matrix expansion grouped all repetitions for one candidate/profile together, so thermal or ambient load drift could align with a configuration. Attempt records retained commands and logs but not pre/post host load, disk availability, or physical-memory context.
Add opt-in execution_order=paired_interleaved without changing legacy grouped expansion or archived plan hashes. The new order runs every candidate/profile cell once per repetition block and includes block and absolute position in cell identity. Record pre/post load averages, CPU count, physical memory when available, and campaign-filesystem total/used/free bytes in immutable command and attempt records plus the campaign environment snapshot.
The ordering pass is O(cells log cells) time and O(cells) plan memory; execution remains strictly sequential. Resource collection is constant-space and performs no background sampling or production instrumentation.
Verification: uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed (74 passed); Ruff checks; git diff --check. Tests also prove legacy grouped plans remain the default, paired plans execute repetitions as 1,1,1,1,2,2,2,2, and successful attempts retain both resource snapshots.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previously, semantic capability fixtures measured only the initial full index. They could not prove that a changed source file removed an old relationship, added a new relationship, or matched a fresh rebuild, and expected stale-on-incremental warnings appeared as undifferentiated quality misses.
Add content-addressed post-edit Go and Python sources whose single-file mutations swap the judged SIMILAR_TO and SEMANTICALLY_RELATED pairs. Retain pre/post hashes, changed paths, initial/incremental/fresh index results, pair witnesses, edge-score equality, and whole canonical-graph equality. Add incremental_semantic_freshness_eager as a one-setting profile for incremental_derived_refresh=eager.
Classify default stale_on_incremental separately from eager freshness: deferred publication conforms only when it is immediately canonical or returns the structured semantic_edges stale warning; eager publication requires fresh pair and canonical graph equality without that warning. Pair comparison remains O(observed pairs) in runtime and memory.
Verification: uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign (72 passed); Ruff checks; jq manifest parse; gofmt fixture checks; git diff --check. On optimized arm64 binary 36a1297b2571729517b96467965f476f15cd1d60e545454bc0c296f3103b9e17, default similarity/semantic edits conformed with explicit stale warnings at 40/51 ms; eager edits produced TP=1/TN=2/FP=0/FN=0 and canonical fresh equality at 72/71 ms. These are retained single pilot observations, not effect-size claims.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previously, capability-quality campaigns covered only rank and dependency retrieval, so SIMILAR_TO and SEMANTICALLY_RELATED costs had no paired precision/recall evidence. FAST remained the implicit matrix mode even for relationships that require MODERATE or FULL indexing.
Add benchmarks/semantic-pairs-v1/manifest.json with content-addressed Go structural-clone and Python control-flow-variant tasks, explicit lexical negatives, and source hashes. Extend benchmark-incremental-speed.py with unordered pair identities, TP/FP/FN/TN witnesses, precision, recall, F1, false-positive rate, unjudged-background retention, numeric query-score normalization, and response-quality gates. Propagate index_mode through compact campaign cells and document durable task/report semantics.
The bounded scorer is O(judgments + observed pairs) in runtime and memory. Natural repository edges outside the explicit judgment set remain unjudged instead of becoming false positives.
Verification: uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign (68 passed); ruff check on changed Python; jq manifest parse; gofmt fixture check; git diff --check. Optimized arm64 binary 36a1297b2571729517b96467965f476f15cd1d60e545454bc0c296f3103b9e17 produced TP=1/TN=2/FP=0/FN=0 for both enabled fixtures; each individually disabled profile removed its judged positive edge.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previous behavior: load_composition_groups() always re-expanded each matrix spec through execution-time binary validation. Rebuilding build/c/codebase-memory-mcp therefore made a completed campaign fail with candidates[3].binary_sha256 does not match, even though its immutable run records and inputs were intact.
Accept exactly one matrix_spec or plan source per composition campaign. Immutable plans pass through validate_plan() without reopening candidate executables, while matrix_spec retains strict live expansion and SHA validation. Record each source kind, absolute path, and SHA-256 in composition provenance.
Files: scripts/summarize-benchmark-results.py; tests/test_summarize_benchmark_results.py.
Verification: uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign tests.test_summarize_benchmark_results (94 passed); bash scripts/check-source-safety.sh; git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Matrix cases from scripts/benchmark-incremental-speed.py retain changed_paths at the case root, while mutation_reindex_details() only read the nested self-dogfood mutation schema. Generated reports therefore rendered 'not reported' for source mutations whose immutable artifacts contained exact paths.
Read both schemas in scripts/summarize-benchmark-results.py and derive a concrete synthetic <language> inbound-frontier definition description from scenario_metadata. Preserve the nested mutation behavior and union path values without changing raw results.
Add test_markdown_reports_matrix_changed_paths_from_case_root() in tests/test_summarize_benchmark_results.py. Verification: focused red reproduced both missing fields; focused matrix/self-dogfood tests passed; benchmark/campaign/report unittest suite passed 94/94; Ruff passed; audit-only regeneration validated 26/26 supported-language cells with zero missing or corrupt cells and rendered paths including leaf.go and shared.h. Generated reports, manifests, and logs remain ignored and unstaged.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Five-repetition reports showed stable medians alongside occasional system outliers, but the tables exposed only p50/p95 values. That made it difficult to distinguish a typical configuration cost from observed host variability and could invite unsupported confidence claims.
Add query, incremental, and full-index observation counts plus min-max ranges to summarize-benchmark-results.py. The generated section explicitly states that grouped sequential campaigns avoid concurrent contention but do not support paired or randomized effect-size confidence intervals; medians and ratios remain descriptive.
tests/test_summarize_benchmark_results.py verifies exact counts and [min, max] rendering without confidence language. Verified with uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed tests.test_summarize_benchmark_results (93 passed). The ignored 240-input composition report and SHA-256 sidecar were regenerated but not staged.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Separate incremental, rank, and dependency campaigns could not produce categorical overall-quality or Pareto rows because each report lacked the other axes. Hand-merging raw files also risked mixing different config profiles and bypassing completion hashes.
Add --composition-spec to scripts/summarize-benchmark-results.py. Named campaign entries re-expand archived matrix specs, resolve exact cell labels, require every selected completion and binary/result SHA-256 to validate, and materialize derived inputs without changing raw results. Generated Markdown records the composition-spec hash and writes a per-input SHA-256 sidecar.
Preserve execution failures as correctness rejections, but classify a missed rank/dependency target as BELOW QUALITY TARGET when that exact report explicitly disables the corresponding capability. tests/test_summarize_benchmark_results.py covers campaign grouping and composed ablation wording.
Verified with uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed tests.test_summarize_benchmark_results (92 passed). The ignored composition validated 240 retained inputs and identified latest.default.mcp as the only fully passing measured Pareto-front row; generated reports and manifests were not staged.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Rank and dependency fixtures previously required hand-authored campaign cells, which made candidate, profile, transport, and repetition coverage easy to drift and omitted the concise matrix audit trail used by incremental benchmarks.
Add top-level capability_quality expansion in scripts/run-benchmark-campaign.py. Capability cells omit frontier/exact-cap axes, invoke benchmark-incremental-speed.py with --capability-quality and --include-logs, record the fixture in identity parameters, and retain named config profiles for enabled/disabled ablations. Existing incremental expansion remains the default when the field is absent.
tests/test_benchmark_campaign.py verifies a two-transport, two-repetition rank grid has four unique cells, no frontier parameters, the expected command, and accepted exit policy. Verified with uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed tests.test_summarize_benchmark_results (90 passed).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Compact matrix expansion in scripts/run-benchmark-campaign.py hardcoded accepted_exit_codes=[0]. benchmark-incremental-speed.py intentionally exits 1 after writing a valid result when canonical graph or quality gates reject a candidate, so campaigns mislabeled measured upstream failures as missing infrastructure.
Accept a validated top-level matrix-spec exit-code list, copy it into every expanded cell identity, and keep [0] as the default. Result parsing, binary SHA-256 verification, and the structured error check still reject crashes, missing artifacts, hash mismatches, and harness-error documents.
tests/test_benchmark_campaign.py verifies [0,1] propagation. Verified with uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed tests.test_summarize_benchmark_results (89 passed). A retained two-cell upstream probe now reports REJECT: graph correctness with canonical mismatch witnesses instead of missing cells.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Compact matrix specs previously required a positive exact cap and always injected incremental_exact_max_affected_paths. A default-versus-default historical comparison therefore had to overwrite candidate policy with one shared cap.
Allow null in scenarios[].exact_caps to omit the config override, omit the capability identity key, retain exact_cap=null in parameters, and label the cell capdefault. Positive integers keep the existing explicit cap-sweep behavior. Document both forms in docs/BENCHMARK_CAMPAIGN.md.
Tests: uv run python -m unittest tests.test_benchmark_campaign -v (16 passed); uv run ruff check scripts/run-benchmark-campaign.py tests/test_benchmark_campaign.py (passed); git diff --check (passed).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
expand_matrix_spec() previously dropped candidate capability_support even though explicit campaign cells include it in identity and derived reports. Compact matrix runs could therefore revert to mode-based applicability guesses.
Validate candidates[].capability_support as a string-to-boolean object and copy its sorted values into every expanded cell before plan validation. This preserves candidate support in SHA-256 identities and generated report inputs for all matrix repetitions.
Tests: uv run python -m unittest tests.test_benchmark_campaign -v (15 passed); uv run ruff check scripts/run-benchmark-campaign.py tests/test_benchmark_campaign.py (passed); git diff --check (passed).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Capability-quality results store initial_fast_full.elapsed_ms and peak_rss_mb, but summarize_group() previously read only incremental and fresh_fast_full_after_change measurements. Generated cross-version reports therefore showed n/a for full observations and peak RSS despite retaining both values.
Collect initial full wall time and RSS in one pass over cases. Use initial full observations only when no post-mutation fresh rebuild exists, so incremental matrix semantics remain unchanged while rank and dependency fixtures expose their recorded production-build cost.
Tests: uv run python -m unittest tests.test_summarize_benchmark_results -v (28 passed); uv run ruff check scripts/summarize-benchmark-results.py tests/test_summarize_benchmark_results.py (passed); git diff --check (passed).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previous reports inferred every FAST-mode algorithm as applicable, so upstream/main was labeled rank-capable even though git grep at 2469ecc3 finds no cbm_pagerank_compute implementation. Campaign capabilities also affected identity but were not propagated into report inputs.
Add optional string-to-boolean capability_support to cell validation and SHA-256 identity. materialize_report_input() writes a deterministic derived JSON document with candidate support, source-result SHA-256, and cell identity while leaving immutable raw results unchanged. The summarizer now lets explicit unsupported metadata override mode-based applicability and accepts both dependencies and the retained auto_index_deps support key.
Tests: uv run python -m unittest tests.test_benchmark_campaign tests.test_summarize_benchmark_results -v (42 passed); uv run ruff check scripts/run-benchmark-campaign.py scripts/summarize-benchmark-results.py tests/test_benchmark_campaign.py tests/test_summarize_benchmark_results.py (passed); git diff --check (passed).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previous behavior rendered cleanup.requested=false as Cleanup 0/1, making intentionally retained audit evidence look like a failed lifecycle operation. The campaign guide also directed crash-sensitive plans and results to /tmp even though validate_campaign_root() rejects the operating-system temporary tree by default.
Add evidence_lifecycle() in scripts/summarize-benchmark-results.py to label requested disposal, deliberate retention, missing metadata, and requested cleanup failure separately. Rename the generated column to Evidence lifecycle. Point docs/BENCHMARK_CAMPAIGN.md at the ignored durable .worktrees/benchmark-campaign path. Replace the secondary nDCG reference with Järvelin and Kekäläinen's primary DOI 10.1145/582415.582418 and identify exact pairwise Pareto nondominance with Deb et al. DOI 10.1109/4235.996017.
Tests: uv run python -m unittest tests.test_summarize_benchmark_results -v (26 passed); uv run ruff check scripts/summarize-benchmark-results.py tests/test_summarize_benchmark_results.py (passed); git diff --check (passed).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previous behavior: cbm_discover_installed_deps() filtered graph qualified names for dependencies|require, but package.json extraction emits names such as <project>.package.cbmbenchdep. Automatic npm indexing therefore reported zero packages even when node_modules contained the declared source.
Parse dependencies, devDependencies, optionalDependencies, and peerDependencies from a local package.json capped at 1 MiB in src/depindex/depindex.c:364-489. Resolve only installed package directories, deduplicate names with CBMHashTable, grow retained results geometrically to the configured limit, and release files, JSON documents, tables, paths, versions, and partial arrays on every failure path.
Add the exact extracted-QN regression at tests/test_depindex.c:712. Extend scripts/benchmark-incremental-speed.py with a network-free npm quality fixture, same-result source/package/read-only judgments, atomic report publication, worker/process timing attribution, and profile-only CLI lifecycle brackets.
Verification on native Apple Silicon macOS: ASAN+UBSAN depindex 36/36; ASAN+UBSAN CLI 146/146; Python benchmark/campaign/report 84/84; Ruff check passed; production link passed with -O2 -DCBM_BIND_TS_ALLOCATOR=1. Native Windows execution was not performed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
scripts/run-benchmark-campaign.py now resolves --campaign-root through validate_campaign_root before disk checks, plan archival, environment snapshots, or cell execution. Paths equal to or below tempfile.gettempdir() fail with actionable guidance because completed results, failed-attempt logs, and resume markers would be lost after a crash or reboot.
Add --allow-temporary-campaign-root as an explicit disposable-test override. Durable campaign behavior remains unchanged: atomic fsynced writes, immutable cell identities, completion hashes, retained attempts, stale-lock recovery, process-tree termination, disk guards, audit manifests, and resumable cells.
tests/test_benchmark_campaign.py covers default rejection, explicit disposable opt-in, and a repository .worktrees campaign path. Campaign tests passed 14/14; combined benchmark/campaign/report tests passed 80/80; Ruff and git diff --check passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previous behavior: src/mcp/mcp.c:3965 handle_list_projects opened every valid cache database and emitted the complete inventory for every call. On the 046d438 real-cache probe, that path returned 81,646 result bytes and took 6,462.768 ms.
Add MCP_PROJECTS_PAGE_SIZE=50 and MCP_PROJECTS_PAGE_MAX=200, stable filename sorting, lazy valid-database traversal, limit/offset metadata, and an explicit all=true compatibility path. collect_project_db_names frees partial allocations on failure; build_project_json_entry closes lookahead stores without querying counts or Git metadata.
Keep scripts/benchmark-incremental-speed.py:1183 full-inventory scaling semantics explicit with list_projects_arguments={all:true}. Add tests/test_mcp.c:963 coverage for stable pages, next_offset, schema fields, full compatibility, environment restoration, and cache cleanup.
Verification: focused ASAN/UBSan canary 1/1; MCP suite 224/224; tool_consolidation 100/100; benchmark harness unittest 41/41; scripts/check-source-safety.sh passed; git diff --cached --check passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add render_search_projection() at scripts/summarize-benchmark-results.py:795-895 and expose retained projection JSON through --search-projection.
Place ranked-identity parity beside response bytes and token estimates so compact output cannot appear successful by size alone. Show selected property fields, post-call RSS, transport survival, and server cleanup; move long non-compact field inventories below the table for readability.
State that latency has one observation per variant, RSS is post-call rather than peak, and the comparison covers response projection only. Successful runs use Outcome: complete, Equal, Survived, and Reaped instead of visible failure labels.
Add renderer and wrong-document tests at tests/test_summarize_benchmark_results.py:27-85. The retained 30-result pilot renders 6,824 compact bytes, 7,784 selected-field bytes, and 18,374 non-compact bytes with equal identities, zero fp/sp/bt fields, and 62.9% compact byte savings.
Verification:
- uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign tests.test_summarize_benchmark_results: 78/78 passed.
- uv run ruff check scripts/summarize-benchmark-results.py tests/test_summarize_benchmark_results.py: passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add --search-projection to scripts/benchmark-incremental-speed.py. One isolated FAST fixture compares compact default, compact=true, compact=true with fields=[complexity,signature], and compact=false using a fresh MCP server per variant.
Record canonical payload/envelope bytes, deterministic token estimates, descriptive latency, post-call RSS, returned qualified names, property keys, internal fields, transport survival, and server reap state. Gate every variant on identical ranked identities, zero fp/sp/bt keys, its declared projection contract, and compact <= selected <= non-compact bytes.
Extract atomic_write_text() so list-project and projection records both flush, fsync, replace, and remove temporary output files. Auto-created work roots remain isolated and are deleted unless --keep-work-root is explicit.
Add build_search_projection_observation() coverage at tests/test_benchmark_incremental_speed.py:16-38. The four-result integration smoke with optimized binary SHA-256 ec61f92332d46af283c9e8fd0fbaae4a2df5181f70e2eb38d2088166e7515d08 measured 1,240 compact bytes, 1,368 selected-field bytes, and 2,780 non-compact bytes with identity parity, no internal fields, live transports, reaped servers, and removed fixtures.
Verification:
- uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign tests.test_summarize_benchmark_results: 76/76 passed.
- uv run ruff check scripts/benchmark-incremental-speed.py tests/test_benchmark_incremental_speed.py: passed.
- make -f Makefile.cbm cbm: built the -O2 production binary.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add render_list_projects_scaling() at scripts/summarize-benchmark-results.py:795-878 and expose retained JSON through --list-projects-scaling.
Render requested/returned counts, payload bytes, deterministic token estimates, descriptive call latency, post-call RSS, transport survival, server cleanup, and transient fixture disk use in separate columns. Successful cells use Outcome: complete, Survived, and Reaped instead of visible failure labels.
State the one-observation latency limitation, identify post-call RSS as non-peak, retain the experiment claim boundary, and include run ID, binary SHA-256, and fixture cleanup evidence. Generated Markdown remains outside Git.
Add clarity and wrong-document tests at tests/test_summarize_benchmark_results.py:27-83. The retained 1/16/64 pilot renders 284/3,239/12,695 bytes, 71/810/3,174 estimated tokens, and 15.149/25.779/82.118 ms without attributing the external combined multi-tool figure to list_projects alone.
Verification:
- uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign tests.test_summarize_benchmark_results: 75/75 passed.
- uv run ruff check scripts/summarize-benchmark-results.py tests/test_summarize_benchmark_results.py: passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Andrew Hundt