mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-10-26 06:43:15 +08:00
Andrew Hundt
8f44fee8f3
Separate incremental, rank, and dependency campaigns could not produce categorical overall-quality or Pareto rows because each report lacked the other axes. Hand-merging raw files also risked mixing different config profiles and bypassing completion hashes. Add --composition-spec to scripts/summarize-benchmark-results.py. Named campaign entries re-expand archived matrix specs, resolve exact cell labels, require every selected completion and binary/result SHA-256 to validate, and materialize derived inputs without changing raw results. Generated Markdown records the composition-spec hash and writes a per-input SHA-256 sidecar. Preserve execution failures as correctness rejections, but classify a missed rank/dependency target as BELOW QUALITY TARGET when that exact report explicitly disables the corresponding capability. tests/test_summarize_benchmark_results.py covers campaign grouping and composed ablation wording. Verified with uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed tests.test_summarize_benchmark_results (92 passed). The ignored composition validated 240 retained inputs and identified latest.default.mcp as the only fully passing measured Pareto-front row; generated reports and manifests were not staged. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>