Files
codebase-memory-mcp/scripts
Andrew Hundt 975ec70353 feat(benchmarks): isolate rank quality ablation
Add --capability-quality rank in scripts/benchmark-incremental-speed.py. create_rank_quality_repo() builds eight lexical decoys and one structurally central Python function with eight callers; run_rank_quality_oracles() records the central symbol's RR, Hit@1, Hit@5, and nDCG@5 under an isolated cache and process lifecycle.

Make score_ranked_relevance() compute reciprocal rank across the full bounded response while retaining the configured DCG/nDCG cutoff, matching the first-correct-result MRR definition at https://trec.nist.gov/data/qa.html. Record execution_passed separately from quality_target_met so a valid ablation below the cutoff exits successfully.

Render BELOW CUTOFF and BELOW QUALITY TARGET instead of reporting rank-nine evidence as missing or broken. The focused benchmark and summarizer suites pass 57 tests, Ruff reports no violations, and git diff --check is clean.

Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-07-15 20:57:32 -04:00
..