mirror of https://github.com/DeusData/codebase-memory-mcp.git synced 2026-11-02 07:53:41 +08:00

Files

T

Andrew Hundt 36698e5f8b fix(benchmarks): omit duplicate minimal-indexing experiment cells

build_automatic_spec emitted optional-graph-disabled and minimal-indexing with identical candidate labels and capability manifests, repeating the same measurements under two names.

Emit only minimal-indexing in new automatic plans. Keep optional_graph_disabled as a retained-plan loader alias backed by the shared MINIMAL_INDEXING_CAPABILITIES map, document that compatibility boundary, and assert candidate/capability signature uniqueness. Correct the auto_dep_limit effective-sentinel comment in src/depindex/depindex.c.

Verified 207 passed, 1 skipped, and 34 subtests across the six benchmark Python suites; Ruff format/check and git diff --check passed.

Signed-off-by: Andrew Hundt <ATHundt@gmail.com>

2026-07-23 23:01:19 -04:00

schema

refactor(benchmarks): consolidate canonical assets under benchmarks

2026-07-23 20:29:05 -04:00

semantic-pairs-v1

feat(benchmarks): verify semantic freshness after source edits

2026-07-16 15:16:33 -04:00

autotune.py

fix(benchmarks): restore upstream scripts and resolve retained plans

2026-07-23 20:44:39 -04:00

config-spellings-v1.json

refactor(benchmarks): consolidate canonical assets under benchmarks

2026-07-23 20:29:05 -04:00

fact_comparisons.py

refactor(benchmarks): hoist report helpers and harden process cleanup

2026-07-23 21:59:17 -04:00

generate_terminology.py

refactor(benchmarks): consolidate canonical assets under benchmarks

2026-07-23 20:29:05 -04:00

README.md

fix(benchmarks): restore upstream scripts and resolve retained plans

2026-07-23 20:44:39 -04:00

run_benchmark.py

fix(benchmarks): omit duplicate minimal-indexing experiment cells

2026-07-23 23:01:19 -04:00

run_experiments.py

fix(benchmarks): omit duplicate minimal-indexing experiment cells

2026-07-23 23:01:19 -04:00

summarize_results.py

refactor(benchmarks): hoist report helpers and harden process cleanup

2026-07-23 21:59:17 -04:00

terminology.json

fix(benchmarks): restore upstream scripts and resolve retained plans

2026-07-23 20:44:39 -04:00

README.md

Benchmark tooling

This directory owns the benchmark implementations, active schemas, machine-readable terminology, configuration-spelling compatibility data, and source fixtures.

Primary entry points:

run_benchmark.py: run one isolated benchmark and emit canonical fact tables.
run_experiments.py: build and execute immutable, resumable benchmark matrices.
summarize_results.py: render quality-gated Markdown from retained result JSON.
fact_comparisons.py: derive parity, capability-delta, and lifecycle tables from canonical facts.
autotune.py: run the isolated PageRank tuning experiment.

Start with the built-in help:

uv run python benchmarks/run_benchmark.py --help
uv run python benchmarks/run_experiments.py --help

run_benchmark.py is the single-run entry point. run_experiments.py is the multi-candidate, repeated-run entry point; automatic modes store durable ignored state under .worktrees/benchmark-campaign/. Explicit runs should use an ignored benchmark-results/ root or another durable path outside the checkout.

schema/ contains schemas for records emitted by current tooling. terminology.json defines every normative fact, step, join, and formula identifier. The generated human view remains in docs/BENCHMARK_TERMINOLOGY.md, and the full workflow is documented in docs/BENCHMARK_EXPERIMENTS.md.

The upstream-owned scripts/benchmark-index.sh, scripts/benchmark-search-graph.sh, and scripts/clone-bench-repos.sh retain their established locations. Branch-created Python benchmark implementations live here without executable compatibility copies. docs/schema/benchmark-facts-v1.schema.json retains its frozen URI because v1 bundles embed that identifier.