Files
codebase-memory-mcp/scripts
Andrew Hundt c1e2ba1472 feat(benchmarks): derive contract-bound comparison facts
Add scripts/benchmark_fact_comparisons.py and docs/schema/benchmark-comparisons-v1.schema.json to group repetitions by effective implementation, capability, scope, cache, host, harness, terminology, and correctness contracts.

Emit elapsed ratios only for parity_manifest_and_contract_v1 pairs; emit capability differences without ratios for capability_delta_manifest_v1 pairs; preserve source run and occurrence IDs and outer lifecycle wall-time rows.

Extend fact_result_rows() with graph, freshness, and retrieval-quality outcomes and append generated comparison artifacts from generate_report(). Retained reports without fact bundles remain readable and report comparison status as unavailable.

Define the versioned join and formula IDs in benchmark terminology 1.1.0. Verified 198 passed, 1 skipped, and 34 subtests; Draft 2020-12 schema validation, Ruff, source-safety, and git diff checks pass.

Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-07-23 20:08:05 -04:00
..