Files
codebase-memory-mcp/scripts
Andrew Hundt 14ed7bf317 feat(benchmarks): report latency observation ranges
Five-repetition reports showed stable medians alongside occasional system outliers, but the tables exposed only p50/p95 values. That made it difficult to distinguish a typical configuration cost from observed host variability and could invite unsupported confidence claims.

Add query, incremental, and full-index observation counts plus min-max ranges to summarize-benchmark-results.py. The generated section explicitly states that grouped sequential campaigns avoid concurrent contention but do not support paired or randomized effect-size confidence intervals; medians and ratios remain descriptive.

tests/test_summarize_benchmark_results.py verifies exact counts and [min, max] rendering without confidence language. Verified with uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed tests.test_summarize_benchmark_results (93 passed). The ignored 240-input composition report and SHA-256 sidecar were regenerated but not staged.

Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-07-16 08:49:52 -04:00
..