Files
codebase-memory-mcp/scripts
Andrew Hundt 4621ed78b2 fix(benchmarks): retain correctness-gate exit results
Compact matrix expansion in scripts/run-benchmark-campaign.py hardcoded accepted_exit_codes=[0]. benchmark-incremental-speed.py intentionally exits 1 after writing a valid result when canonical graph or quality gates reject a candidate, so campaigns mislabeled measured upstream failures as missing infrastructure.

Accept a validated top-level matrix-spec exit-code list, copy it into every expanded cell identity, and keep [0] as the default. Result parsing, binary SHA-256 verification, and the structured error check still reject crashes, missing artifacts, hash mismatches, and harness-error documents.

tests/test_benchmark_campaign.py verifies [0,1] propagation. Verified with uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed tests.test_summarize_benchmark_results (89 passed). A retained two-cell upstream probe now reports REJECT: graph correctness with canonical mismatch witnesses instead of missing cells.

Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-07-16 08:24:10 -04:00
..