Add render_search_projection() at scripts/summarize-benchmark-results.py:795-895 and expose retained projection JSON through --search-projection.
Place ranked-identity parity beside response bytes and token estimates so compact output cannot appear successful by size alone. Show selected property fields, post-call RSS, transport survival, and server cleanup; move long non-compact field inventories below the table for readability.
State that latency has one observation per variant, RSS is post-call rather than peak, and the comparison covers response projection only. Successful runs use Outcome: complete, Equal, Survived, and Reaped instead of visible failure labels.
Add renderer and wrong-document tests at tests/test_summarize_benchmark_results.py:27-85. The retained 30-result pilot renders 6,824 compact bytes, 7,784 selected-field bytes, and 18,374 non-compact bytes with equal identities, zero fp/sp/bt fields, and 62.9% compact byte savings.
Verification:
- uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign tests.test_summarize_benchmark_results: 78/78 passed.
- uv run ruff check scripts/summarize-benchmark-results.py tests/test_summarize_benchmark_results.py: passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add --search-projection to scripts/benchmark-incremental-speed.py. One isolated FAST fixture compares compact default, compact=true, compact=true with fields=[complexity,signature], and compact=false using a fresh MCP server per variant.
Record canonical payload/envelope bytes, deterministic token estimates, descriptive latency, post-call RSS, returned qualified names, property keys, internal fields, transport survival, and server reap state. Gate every variant on identical ranked identities, zero fp/sp/bt keys, its declared projection contract, and compact <= selected <= non-compact bytes.
Extract atomic_write_text() so list-project and projection records both flush, fsync, replace, and remove temporary output files. Auto-created work roots remain isolated and are deleted unless --keep-work-root is explicit.
Add build_search_projection_observation() coverage at tests/test_benchmark_incremental_speed.py:16-38. The four-result integration smoke with optimized binary SHA-256 ec61f92332d46af283c9e8fd0fbaae4a2df5181f70e2eb38d2088166e7515d08 measured 1,240 compact bytes, 1,368 selected-field bytes, and 2,780 non-compact bytes with identity parity, no internal fields, live transports, reaped servers, and removed fixtures.
Verification:
- uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign tests.test_summarize_benchmark_results: 76/76 passed.
- uv run ruff check scripts/benchmark-incremental-speed.py tests/test_benchmark_incremental_speed.py: passed.
- make -f Makefile.cbm cbm: built the -O2 production binary.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add render_list_projects_scaling() at scripts/summarize-benchmark-results.py:795-878 and expose retained JSON through --list-projects-scaling.
Render requested/returned counts, payload bytes, deterministic token estimates, descriptive call latency, post-call RSS, transport survival, server cleanup, and transient fixture disk use in separate columns. Successful cells use Outcome: complete, Survived, and Reaped instead of visible failure labels.
State the one-observation latency limitation, identify post-call RSS as non-peak, retain the experiment claim boundary, and include run ID, binary SHA-256, and fixture cleanup evidence. Generated Markdown remains outside Git.
Add clarity and wrong-document tests at tests/test_summarize_benchmark_results.py:27-83. The retained 1/16/64 pilot renders 284/3,239/12,695 bytes, 71/810/3,174 estimated tokens, and 15.149/25.779/82.118 ms without attributing the external combined multi-tool figure to list_projects alone.
Verification:
- uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign tests.test_summarize_benchmark_results: 75/75 passed.
- uv run ruff check scripts/summarize-benchmark-results.py tests/test_summarize_benchmark_results.py: passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add --list-projects-scaling to scripts/benchmark-incremental-speed.py. The mode indexes one minimal fast-mode seed, snapshots and rekeys valid SQLite project databases, and starts a fresh MCP server for every strictly increasing project count.
Record canonical response bytes, deterministic byte/4 token estimates, MCP envelope bytes, call latency, post-call resident memory, fixture disk bytes, transport survival, and server/thread reap status. Label post-call RSS as non-peak and state that the experiment cannot attribute combined multi-tool response size.
Gate fixture creation at scripts/benchmark-incremental-speed.py:540-573 against --list-project-fixture-max-mb plus a max(2 GiB, 5% free-space) reserve. Write completed or failed JSON atomically and remove the temporary output path in a finally block. Auto-created work roots are removed while --keep-work-root preserves them.
Add count validation, snapshot rekeying, and disk-budget tests in tests/test_benchmark_incremental_speed.py:16-76. The 1→2 integration smoke used optimized binary SHA-256 379585ab9be20eade9edf6aff2e1cea318aceb1659aa3f6cbfa3f51bc8dc5fb4, returned 1/1 and 2/2 projects, kept transport alive, reaped both servers, and removed its work root.
Verification:
- uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign tests.test_summarize_benchmark_results: 73/73 passed.
- uv run ruff check scripts/benchmark-incremental-speed.py tests/test_benchmark_incremental_speed.py: passed.
- make -f Makefile.cbm cbm: built the -O2 production binary.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previous behavior:
- McpClient.__exit__() in scripts/benchmark-incremental-speed.py:807-822 closed stdin and waited for the server but did not join stdout_thread or stderr_thread, close the output streams, or clear retained process/thread references.
Changes:
- Join both reader threads after process exit, close stdout and stderr, clear proc/stdout_thread/stderr_thread, and raise MCP reader thread did not stop after process exit when a normal teardown cannot reap a reader.
- Preserve the existing wait, terminate, then kill escalation sequence.
- Add test_mcp_client_exit_reaps_process_streams_and_reader_threads in tests/test_benchmark_incremental_speed.py:15-60.
Resource lifecycle:
- Repeated benchmark cells now synchronously release the subprocess pipes and reader-thread references before the next cell starts.
- The bounded second join runs only when a reader remains alive after the initial five-second join.
Verification:
- Red: the focused unit test failed because process.stdout.closed was false.
- Green: focused unit test passed; benchmark, campaign, and summarizer suites passed 70/70.
- uv run ruff check passed for scripts/benchmark-incremental-speed.py and tests/test_benchmark_incremental_speed.py.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add --capability-quality rank in scripts/benchmark-incremental-speed.py. create_rank_quality_repo() builds eight lexical decoys and one structurally central Python function with eight callers; run_rank_quality_oracles() records the central symbol's RR, Hit@1, Hit@5, and nDCG@5 under an isolated cache and process lifecycle.
Make score_ranked_relevance() compute reciprocal rank across the full bounded response while retaining the configured DCG/nDCG cutoff, matching the first-correct-result MRR definition at https://trec.nist.gov/data/qa.html. Record execution_passed separately from quality_target_met so a valid ablation below the cutoff exits successfully.
Render BELOW CUTOFF and BELOW QUALITY TARGET instead of reporting rank-nine evidence as missing or broken. The focused benchmark and summarizer suites pass 57 tests, Ruff reports no violations, and git diff --check is clean.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add render_mcp_surface_parity() in scripts/summarize-benchmark-results.py and a --mcp-surface-parity JSON input that can be combined with benchmark inputs. The generated table separates pure classic, streamlined pre-reveal, and the same streamlined process after _hidden_tools, including advertised names, handler recognition, schema parity, tools/list payload size, and notification evidence.
Label empty-argument dispatch probes as addressability evidence rather than behavioral equality, identify the remaining capability-fixture requirement, and reject non-parity documents. tests/test_summarize_benchmark_results.py covers all three rows, alias-schema wording, and invalid input; 52 focused unittest cases pass and Ruff reports no violations.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add score_ranked_relevance() with a bounded cutoff and explicit positive relevance judgments. Compute first-relevant reciprocal rank, Hit@1, Hit@5, DCG, ideal DCG, and nDCG@k in O(k*j) time for cutoff k and judgment count j.
Extend score_quality_oracles() to accept graded judgment objects while preserving existing tuple-based substring expectations. Missing judged evidence contributes zero; probes without judgments remain N/A. Keep strict task and canonical-graph gates independent from descriptive retrieval scores.
Aggregate nDCG@5 and judgment counts in summarize-benchmark-results.py without hiding MRR or Hit@k. Generated methodology links NIST's TREC QA reciprocal-rank definition and the TREC Complex Answer Retrieval graded-ranking evaluation.
Tests: uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed tests.test_summarize_benchmark_results (62 passed)
Static: uv run ruff check scripts/benchmark-incremental-speed.py scripts/summarize-benchmark-results.py tests/test_benchmark_incremental_speed.py tests/test_summarize_benchmark_results.py
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Replace the hardcoded mode=fast index request with --index-mode fast|moderate|full and propagate the selected mode through CLI, MCP, matrix, self-dogfood, incremental, and fresh-reference runs.
Record capability_applicability in every report. FAST cells now mark SIMILAR_TO and SEMANTICALLY_RELATED quality N/A because src/pipeline/pipeline.c skips both passes in CBM_MODE_FAST; retained artifacts without mode metadata remain unknown.
Add similarity_disabled, semantic_edges_disabled, git_history_disabled, and http_links_disabled beside rank_disabled and dependency_disabled. Each controlled profile changes exactly one global key from src/pipeline/pipeline.h or src/pagerank/pagerank.h.
Generated Markdown includes an Algorithm-quality applicability table so an unavailable algorithm cannot be scored as a pass, failure, or zero benefit.
Tests: uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed tests.test_summarize_benchmark_results (58 passed)
Static: uv run ruff check scripts/benchmark-incremental-speed.py scripts/summarize-benchmark-results.py tests/test_benchmark_incremental_speed.py tests/test_summarize_benchmark_results.py
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add --mcp-surface-parity to benchmark-incremental-speed.py. The bounded probe starts isolated classic and streamlined MCP processes with CBM_AUTO_INDEX=false, measures tools/list latency and payload size, hashes every input schema, observes notifications/tools/list_changed, and compares streamlined state before and after _hidden_tools reveal.
Probe hidden classic names with empty argument objects to verify dispatcher recognition without indexing or mutating a repository. Report this separately from advertised name/schema parity so handler recognition is not presented as end-to-end behavioral equivalence. Account for get_code versus get_code_snippet property and required-field contracts explicitly.
Installed binary 692b78434d86 produced 4/15 advertised and 15/15 dispatch-recognized classic names before reveal; after reveal it contained all 15 classic names with zero schema-hash mismatches and observed tools/list_changed. The retained JSON is /private/tmp/cbm-mcp-surface-parity-20260715-v3.json and is not committed.
Tests: uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed tests.test_summarize_benchmark_results (54 passed)
Static: uv run ruff check scripts/benchmark-incremental-speed.py scripts/summarize-benchmark-results.py tests/test_benchmark_incremental_speed.py tests/test_summarize_benchmark_results.py
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add the dependency_disabled profile so benchmark cells can turn off only auto_index_deps without conflating rank, similarity, semantic-edge, git-history, or HTTP-link passes.
Capture prof phase=index_repository sub=dep_auto_index markers and dependencies_indexed counts in build_index_result(), then normalize both current and retained result shapes in summarize-benchmark-results.py. Generated Markdown now distinguishes explicit enabled/disabled, observed, unsupported, and unknown states and reports dependency phase p50 values plus sample counts.
Reference NIST's TREC reciprocal-rank definition and Kalibera/Jones effect-size confidence interval guidance in generated methodology text. One- and three-observation pilots remain explicitly unqualified rather than receiving invented confidence intervals.
Tests: uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed tests.test_summarize_benchmark_results (52 passed)
Static: uv run ruff check scripts/benchmark-incremental-speed.py scripts/summarize-benchmark-results.py tests/test_benchmark_incremental_speed.py tests/test_summarize_benchmark_results.py
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Create a per-run SMOKE_STATE in scripts/smoke-test.sh and export CBM_CACHE_DIR and XDG_CONFIG_HOME before invoking the installed binary. Keep CLI stderr inside that cleanup tree so smoke runs neither inherit nor leave user cache/config artifacts.
Extend mcp_run with an explicit tool-mode argument: streamlined scenarios validate progressive disclosure while index_repository and get_code_snippet scenarios select classic mode. This prevents a persisted tool_mode=classic setting from producing "streamlined tool 'get_code' not found" in Phase 5.
Tests: rtk bash -n scripts/smoke-test.sh; installed-binary smoke completed all phases with 9,039 nodes, 24,079 edges, MCP reveal/classic calls, installer lifecycle, security, and no-orphan checks passing.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add config_signature() and historical_delta_rows() to scripts/summarize-benchmark-results.py. Match baselines to latest candidates by exact config overrides and render baseline/latest ratios for incremental reindex, post-mutation fresh rebuild, and query latency beside both quality scores and the baseline correctness gate.
Add a matching-capability regression fixture in tests/test_summarize_benchmark_results.py. Verified 24 report/campaign tests and Ruff. The generated combined report now calculates 2.87x default, 2.73x rank-off, and 2.36x all-optional-off incremental gains over the 24ba checkpoint without concealing rejected upstream/trajectory quality.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add mutation_reindex_details() to scripts/summarize-benchmark-results.py and render one per-candidate/scenario table row with the source mutation description, changed paths, publication route/reason, end-to-end incremental p50, internal indexing-work p50, post-mutation fresh rebuild p50, speedup, and canonical equality.
Pin the route_handler rendering contract in tests/test_summarize_benchmark_results.py. Verified 23 summarizer/campaign tests and Ruff. Generated final-performance-quality-report-v2.md directly from the six retained July 14 result JSON files while leaving the original report and campaign manifests unchanged.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add frontier_crossover_rows() to scripts/summarize-benchmark-results.py so each frontier pairs the largest configured fallback cap with the smallest exact cap. Report end-to-end p50, indexed-work p50, profiling-only RSS p50, fresh-full p50, the exact/fallback latency ratio, and a documented five-percent tie band.
Extend tests/test_summarize_benchmark_results.py with a three-cap fixture that proves the nearest fallback/exact pair is selected and rendered. Verified 22 benchmark summarizer/campaign tests and Ruff; regenerated the preserved 318-cell campaign with 318 complete and zero missing, corrupt, duplicate, or unplanned cells.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add --matrix-spec to scripts/run-benchmark-campaign.py. Deterministically expand candidate revision/binary/build, capability profile, transport, scenario, frontier size, exact cap, and repetition into immutable cells; archive both the source spec and expanded plan before execution.
Include structured parameters and benchmark-incremental-speed.py SHA-256 in cell identity evidence. Keep exact command arrays, binary SHA-256 validation, atomic completion, retained attempts, disk guards, and report/manifests on the existing campaign path.
Teach frontier_coverage_gate() to accept deliberately configured low-cap cells only when they publish containment/full, report frontier_too_large, include affected_paths_truncated=true, and preserve the canonical graph.
Validated with 43 Python tests plus 15 subtests and Ruff. Real -O2 smoke cells proved one-attempt completion/resume and configured cap=4 versus five affected Go files produced incremental_containment with canonical equality and cleanup.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Define CBM_PIPELINE_DELTA_REASON_SCOPED_LSP_GAP in src/pipeline/pipeline_internal.h and use it when JavaScript, TypeScript, TSX, PHP, C#, Java, Kotlin, or Rust require the existing correctness-preserving full rebuild. Report affected_paths_truncated=false instead of claiming frontier_too_large at 1/64 affected paths.
Extend scripts/benchmark-incremental-speed.py with fixtures for all 13 languages wired through cbm_pxc_has_cross_lsp(). Gate Go, C, C++, CUDA, and Python on observed exact-frontier expansion; gate the other eight on publish_kind=full and exact_reason=scoped_lsp_gap. A fixture that does not exercise its declared contract now fails the matrix.
Validated with the -O2 13-language matrix (13/13 contract and canonical graph gates, cleanup=true), ASan+UBSan pipeline suite (376 passed), language-contract suite (35 passed; breadth checked 159 grammars), benchmark Python tests (39 passed, 15 subtests), and Ruff.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add Go, Python, and C-header inbound-frontier fixtures to scripts/benchmark-incremental-speed.py with an explicit --frontier-files denominator and single-definition mutations.
Teach build_index_result() to accept peak_mb from mem.phase, pipeline.done, or incremental.done. Emit terminal RSS fields from cbm_pipeline_run() and log_incremental_done() only while cbm_profile_active is true, preserving normal-run logging and avoiding memory queries outside experiment mode.
Validated with the -O2 CLI matrix (3/3 canonical graph gates, cleanup=true, peak RSS present on initial/incremental/fresh routes), the ASan+UBSan C runner (375 passed), benchmark Python tests (35 passed, 5 subtests), and Ruff.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Compute retrieval MRR, canonical-graph fidelity, applicable-probe task success, and their equal-weight geometric mean in scripts/summarize-benchmark-results.py. Correctness failures remain hard REJECT decisions and cannot be averaged away.
Render named oracle criteria, expected evidence, rank, Hit@1/Hit@5, raw retrieval/graph/scenario counts, capability profiles, and explicit Pareto eligibility or dominance reasons. Link the generated Markdown to the NIST TREC reciprocal-rank definition at https://trec.nist.gov/pubs/trec14/papers/hummingbird.qa.robust.tera.pdf.
Validation: 32 benchmark/campaign/report unit tests passed; Ruff and scripts/check-source-safety.sh passed; preserved campaign JSON rendered successfully without modifying source artifacts.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add --config-profile choices in scripts/benchmark-incremental-speed.py and record the selected profile beside its expanded config_overrides map.
Define minimal_indexing as auto_index_deps=false plus rank_enabled, similarity_enabled, semantic_edges_enabled, githistory_enabled, and httplinks_enabled set to false. Keep repeated --config values higher priority for controlled ablations.
Document the distinction between optional_graph_disabled and minimal_indexing in docs/BENCHMARK_CAMPAIGN.md. Validate with 29 Python unit tests, Ruff, and scripts/check-source-safety.sh.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add correctness_findings() in scripts/summarize-benchmark-results.py to report canonical count mismatches, bounded left/right witnesses, and failed task-oracle expectations without copying unbounded payloads into Markdown.
Correct the response-size footnote to identify the exact default tool-response payload and disclose the one-case/five-oracle sample basis. Cover the generated evidence and measurement wording in tests/test_summarize_benchmark_results.py; 27 benchmark unit tests and Ruff pass.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Parse log= from the index.supervisor.profile_log line emitted by src/mcp/index_supervisor.c and stream measurement markers from that .worker-<pid>.log before trying the unrelated response logfile path. This matches the retained-log contract observed in the profiled arm64 pilot.
The regression fixture supplies a missing response logfile and a valid supervisor worker log, then verifies peak_rss_mb=320 and pipeline_done=81. All 27 focused pytest cases and Ruff pass.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Set CBM_PROFILE=1 in build_env() for every candidate so index_supervisor.c preserves successful worker logs long enough for build_index_result() to stream mem.phase, pipeline.done, and incremental.done markers. Document that the profiling configuration is consistent across revisions and remains part of the auditable benchmark environment.
Validated with 27 pytest cases across the benchmark, summarizer, and campaign suites; Ruff and git diff --check pass.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Run each cell from scripts/run-benchmark-campaign.py in an isolated process group. On timeout or SIGINT, signal the group, allow 30 seconds for harness cleanup, force-stop remaining descendants, write attempt.json, and release running.lock. This addresses the detached cbm-self-dogfood worktree left by an interrupted pilot.
Read at most 512 mem.phase, pipeline.done, and incremental.done lines from the response logfile in build_index_result() using a streaming scan. Replace the route_handler comment literal with an executable cbm_http_path_match(path, "/api/pan4-oracle") mutation and target the route oracle with name_pattern=pan4-oracle.
Validated with 26 pytest cases, including descendant timeout cleanup, bounded worker-log metrics, and executable route mutation. Ruff and git diff --check pass.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Append EXTRA_CFLAGS, EXTRA_CXXFLAGS, and EXTRA_LDFLAGS in Makefile.cbm after the optimized production defaults. Add EXTRA_CXXFLAGS to both ASan jobs in .github/workflows/_soak.yml so commit 87188913 sanitizer arguments instrument C++ sources as well as C sources and the final link.
Document the scripts/build.sh debug invocation in README.md while retaining -O2 and CBM_BIND_TS_ALLOCATOR=1 for ordinary builds. A forced make dry-run places distinct sentinels on the C compile, C++ compile, and link commands; bash -n, scripts/check-source-safety.sh, and git diff --check pass.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Route run_cli_tool_call() and run_mcp_tool_call() through a default-format request for elapsed_ms, response_bytes, and response_token_estimate, then issue format=json only for quality scoring. Record quality_probe_elapsed_ms and quality_response_bytes so the second request cannot inflate the user-facing latency or replace TOON payload sizing.
This fixes the pilot JSONDecodeError: Expecting value at line 1 column 1 when search_graph returned its default TOON content. Document the two-request instrumentation contract in docs/BENCHMARK_CAMPAIGN.md.
Validated with 23 focused pytest cases across tests/test_benchmark_incremental_speed.py, tests/test_summarize_benchmark_results.py, and tests/test_benchmark_campaign.py; Ruff and git diff --check pass.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add cbm_proc_opts_t.cancel_requested so cbm_mcp_server_free() stops supervised auto-index work before joining. POSIX supervision kills the worker process group; Windows assigns the suspended worker to a kill-on-close Job and falls back to direct process termination when Job assignment is unavailable.
Short-circuit index_run_supervised() crash recovery after intentional cancellation, remove cancellation logs, and expose cbm_mcp_server_join_autoindex() for callers that explicitly need completion instead of shutdown.
scripts/smoke-test.sh now checks the six-tool streamlined surface separately from the _hidden_tools reveal. Validation: 18/18 subprocess ASan/UBSan tests passed, the supervised auto_watch MCP regression passed, source-safety passed, and the full isolated smoke suite ended with smoke-test: ALL PASSED including clean shutdown and no-orphan gates.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Extend score_quality_oracles() in scripts/benchmark-incremental-speed.py with result rank, returned count, reciprocal rank, hit@1, and hit@5 for each applicable marker, changed-path, architecture, and route criterion.
Use mean reciprocal rank as the report quality/Pareto axis while retaining the all-applicable binary gate and passed/applicable counts. Render hit@1 and hit@5 beside exact response bytes, token estimates, query latency, incremental latency, and peak RSS.
Validated by 21 campaign/measurement/report unittest cases, including a rank-2 expected result with MRR 0.5, plus ruff check and git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Require validate_result() in scripts/run-benchmark-campaign.py to reject top-level error reports, missing boolean derived.passed verdicts, and artifacts without non-empty cases or measurement objects before complete.json is written.
Keep accepted exit code 1 available for valid quality/performance gate failures while preventing caught RuntimeError reports from becoming resumable measurements.
Validated by 20 campaign/measurement/report unittest cases, including test_harness_error_report_is_not_marked_complete, and ruff check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add scripts/run-benchmark-campaign.py with SHA-256 cell identities over revisions, binaries, release flags, capabilities, transports, scenarios, repetitions, harnesses, commands, environments, and timeouts.
Retain timestamped command/stdout/stderr/result/attempt artifacts; validate binary and result hashes before atomically writing complete.json. Reject live locks, record stale-lock recovery, enforce disk headroom, archive exact plan bytes and environment snapshots, and classify missing, corrupt, duplicate-attempt, and unplanned run directories in immutable manifests.
Regenerate reports atomically from validated completion records only. docs/BENCHMARK_CAMPAIGN.md documents default, rank_enabled=false, and all-optional-capabilities-off cells plus release-build metadata requirements.
Validated by 19 unittest cases and ruff check across the campaign, measurement, and report scripts; git diff --check passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Record canonical JSON payload bytes separately from CLI/MCP framing in scripts/benchmark-incremental-speed.py and attach the deterministic utf8_bytes_div_4_ceil token estimate to every query oracle.
Require every applicable marker, changed-path, architecture, and route oracle to pass. Render quality checks, query latency, response size, indexing latency, and peak RSS in scripts/summarize-benchmark-results.py; mark Pareto candidates only after correctness passes and all comparison axes are present.
Validated by 10 unittest cases across tests/test_benchmark_incremental_speed.py and tests/test_summarize_benchmark_results.py plus py_compile and git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Parse the maximum peak_mb value from mem.phase records in scripts/benchmark-incremental-speed.py and store the tested binary path, size, and SHA-256 in every synthetic, matrix, and self-dogfood report.
Add scripts/summarize-benchmark-results.py to aggregate repeated JSON artifacts into a quality-first Markdown table. Canonical graph or task-oracle failures force a rejection even when measured speedup is high; accepted rows include p50/p95 latency, median speedup, maximum peak RSS, cleanup status, config overrides, and binary identity.
uv run --no-project python -m unittest tests/test_benchmark_incremental_speed.py tests/test_summarize_benchmark_results.py passed 6/6 contracts. The summarizer --help invocation passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Merge upstream/main at 2469ecc3 into integrate-upstream-split-search-tools while retaining branch-specific incremental indexing, MCP API consolidation, and graph extraction behavior.
The staged tree adds upstream's Mojo and Perl grammar support, issue reproduction suites, Graph UI changes, Windows smoke coverage, TOON graph output, subprocess supervision, and Laravel facade route handling. Conflict resolutions in src/pipeline/httplink.c and pass_httplinks.c retain dynamic growth and IMPORTS edge identity; scripts/smoke-test.sh and scripts/soak-test.sh retain hermetic cache isolation.
Validation against the merge plus the follow-up working-tree reconciliation: ASan/UBSan MCP suite 216/216 passed. Follow-up semantic and concurrency fixes remain intentionally unstaged for the next focused commit.
Measure idle CPU from process CPU-time deltas instead of the lifetime average reported by ps. Expose user and system CPU counters in diagnostics, validate configurable idle and response limits, and fail closed when the server or its protocol pipe exits.
Disable session-CWD auto-indexing inside the explicit-index soak harness so unrelated repository work cannot contend for the pipeline lock, inflate cache/RSS results, or masquerade as idle CPU. Preserve crash/restart coverage and exact cleanup of harness-owned resources.
Validated with the ASan/UBSan MCP suite (171 tests), a bounded 30-second idle soak with crash recovery (0.0% idle CPU, 197 MiB RSS, stable cache/FDs), shell syntax and ShellCheck, source-safety lint, clang-format diff, configuration rejection cases, and diff hygiene.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Restore the upstream query-only leak workflow and timeout budgets so multi-hour jobs are neither dormant nor truncated. Preserve mixed-workload behavior while adding a dedicated read-only mode across Linux, macOS, and Windows.
Run every soak in an isolated project/cache/FIFO root, clean owned servers and diagnostics on normal or abnormal exit, and convert MSYS paths before placing them in JSON or environment variables. Validate initialize and tool responses, retain failure state across crash recovery, inject the crash before waiting for an index response, and require a checked post-restart reindex.
Record cache, database, and WAL sizes alongside RSS and FD metrics. Use post-index peak/final ratios and long-run slope as the portable default memory gates; keep an explicit CBM_SOAK_RSS_MAX_MB override for measured platform budgets instead of the obsolete universal 200 MB ceiling.
Validation:
- bash -n scripts/soak-test.sh
- shellcheck -x scripts/soak-test.sh
- actionlint on soak workflows
- source-safety and diff hygiene
- zero-minute product lifecycle smoke: passed
- asynchronous crash/restart/reindex smoke: passed
- one-minute query-leak run: 106 RPCs, zero failures
- invalid config and failed-server cleanup paths: rejected and cleaned
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Make install.ps1 preserve an existing executable before replacement, treat nonzero native --version status as verification failure, and restore the prior binary when copy or verification fails. Abort safely if the existing executable cannot be preserved instead of risking an unrollbackable overwrite.
Remove the rename-aside backup only after the new binary verifies and place the complete download/extract/install flow under a finally cleanup so temporary cbm-install directories are removed on success, explicit exits, and unexpected errors.
Strengthen the existing Windows smoke path to seed an installed binary, fail on installer errors, and reject an orphaned .old backup after successful replacement.
Validation: bash syntax; source-safety and protocol stdout checks; isolated install security audit; repository PowerShell grammar parse with zero extraction errors; diff hygiene. Native PowerShell replacement execution remains covered by the Windows CI smoke job because PowerShell is unavailable on this host.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
The 8e gate compared uname -s to the bare literal MINGW64_NT, which
never matches the real value (MINGW64_NT-10.0-...), so the POSIX
branch silently ran on Windows and looked for the extensionless shim
that #929 no longer installs. Branch by platform prefix: Windows
expects the .cmd shim and the absence of the legacy extensionless
twin; POSIX unchanged. Content checks (never blocks, delegates to
hook-augment) now run on both.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_writer_open used a raw ANSI-CP fopen for the hand-rolled SQLite
writer - the one file-creating call on the dump chain without a
UTF-8-to-wide conversion. On a non-ASCII Windows profile
(C:\Users\Kovacs Janos) extract and resolve succeeded but the dump
failed with pipeline.err phase=dump: fopen could not create graph.db
under the profile-derived cache dir while cbm_mkdir_p (already wide)
had happily created the directory around it. Routed through cbm_fopen,
matching #700 (pass readers) and #973 (canonicalization); POSIX is
byte-transparent.
New Windows guard test_non_ascii_cache_dump.py indexes an ASCII repo
into a non-ASCII CBM_CACHE_DIR and requires a query_graph readback
with Function count > 0 - proving the DB was written to and reopened
from the non-ASCII cache, not merely that no error surfaced. The
existing non-ASCII guard only varied the repo path against an ASCII
cache, which is why this class went unseen.
The USERPROFILE-sourced env encoding question (Layer 2 - _environ is
ANSI-CP, not UTF-8) is deliberately not bundled; it needs verification
on a real Windows box and will be filed separately if the guard alone
does not settle it.
Closes#996
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
SIGABRT (free of a non-libmalloc pointer in ts_stack_delete) on the
second index_repository in one process, whenever a sequential run is
followed by a parallel one. The sequential path parses on the CALLING
thread and left its TLS parser alive; the next parallel run switches
the global tree-sitter allocator to the slab (cbm_slab_install), the
calling thread participates in extraction, reuses the stale parser,
and its per-file teardown frees mimalloc-epoch memory through
slab_free's foreign-pointer fallback -> plain free() -> libmalloc
abort. The v0.9.0 supervisor masks this on the default MCP path (fresh
worker process per index); the in-process path and every embedded
consumer died. Root-caused with allocator-epoch instrumentation: the
victim parser is created at epoch 0 in cbm_pipeline_pass_definitions
and deleted at epoch 1 in extract_worker.
Two boundary fixes:
- run_sequential_pipeline destroys the calling thread's parser at run
end (also bounds its memory like the parallel workers already do);
- cbm_parallel_extract destroys the calling thread's parser BEFORE
cbm_slab_install, so no parser ever crosses the allocator switch.
Guards: prod-tier smoke invariant inv_second_index_inprocess
(sequential-then-parallel double index over stdio, RED with signal 6
on the unfixed binary — ASan builds mask the allocator seam, so the
unit tier cannot catch this class) + suite-tier fork-isolated
index_second_inprocess_run_survives_issue773 pinning the in-process
double-index contract. Repro 5/5 clean after the fix (was 5/5 abort).
Closes#773
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Completes the compact-output pass across the remaining query tools
(consumers are LLM agents; every response byte is context tokens):
- trace_path: callees[N]{qn,hop}/callers[N]{qn,hop} tables (risk/test/
args columns per flags); the per-hop JSON key envelope was 84% of the
payload. 3,589B -> 1,634B on a representative trace.
- query_graph: rows[N]{cols}: table (columns double as the header);
format:"json" restores columns/rows arrays.
- search_code compact mode: results/raw/dirs tables; drops the
field (duplicate of the qn tail). 2,488B -> 1,135B for 17 hits.
- search_code full mode: per-hit source capped at a 60-line window
anchored on the first match, with source_start/source_truncated
markers — uncapped whole-symbol dumps ran to 142KB per response; the
complete symbol stays one get_code_snippet call away.
- get_architecture: default (no aspects) is now a summary (languages,
packages, entry_points + always-on totals/label/type counts) with an
aspects_hint — the old default rendered EVERYTHING including the full
file_tree (94KB -> 4.5KB -> 2.8KB with TOON tables); all sections
emit as TOON tables, aspects:[...] and ["all"] keep full access
(52KB vs 94KB even for the explicit full dump).
- get_code_snippet: drops the property-blob enrichment (41% of the
response; signature/docstring are literally in the source; fp/sp/bt
never belonged there). Metrics remain reachable via search_graph
fields=[...].
- list_projects: per-project 12-field git block (mostly null) replaced
by the branch name; name/root/nodes/edges/size kept.
- search_graph description: in/out documented as TOTAL all-edge degree,
NOT caller/callee counts (agent evals mis-read it as callers).
- format:"json" escape hatch on every converted tool.
Guards: tool_output_byte_budgets pins absolute byte ceilings on default
search/trace outputs (a property blob sneaking back into row emission
blows them immediately); snippet_enriched_properties inverted to pin
the no-spill contract. Suite 5,984 green; smoke updated to the TOON
contract end-to-end.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Tool responses land in an LLM agent's context window, where the legacy
per-node JSON objects were dominated by data no agent can use: the
similarity/semantic pipeline intermediates fp (~450B minhash hex), bt
(body-token bag) and sp (structural profile) plus 14 complexity metrics
were grafted onto every result by enrich_node_properties — ~64% of every
node emission, ~370 tokens per search hit.
search_graph now defaults to TOON (Token-Oriented Object Notation,
toonformat.dev): scalars as 'key: value' lines and results as a
'results[N]{qn,label,file,lines,in,out}:' header + one row per hit.
Published benchmarks show equal-or-better LLM retrieval accuracy at
40-60% fewer tokens than JSON; rows now also carry line ranges, which
the legacy output lacked.
- fields:[...] opts into per-node property columns (complexity,
signature, docstring, ...); fp/sp/bt are blocklisted even there
- format:"json" restores the legacy verbose objects byte-identically;
include_connected forces it (nested neighbor lists)
- BM25 and semantic modes emit the same table shape (rank/score column)
- semantic-only calls no longer run the unfiltered regex search that
silently prepended up to 200 enriched nodes (102.8KB -> 4.2KB)
- default limit 200 -> 50 (BM25 100 -> 50): cheap first page, page via
has_more/offset as before
- new emitter module src/mcp/compact_out.{c,h} (string builder + TOON
scalar/table emission with spec quoting rules)
Measured on a real index (14-hit regex search): 20,793B -> 1,327B
(-94%); BM25 19,667B -> 5,081B; semantic 102,837B -> 4,181B. A typical
discovery session drops ~50.9K -> ~4.7K tokens combined with the
follow-up slices.
Guards: tool_search_graph_toon_never_leaks_internal_fields (sentinel
fp/sp/bt never emitted, even when requested), rewritten
tool_search_graph_includes_node_properties pins default-lean/fields-
opt-in/json-escape-hatch; suite 5983 passed, smoke green.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Perl LSP landed in #461; scan its implementation against PerlNavigator
(MIT, the leading OSS Perl language server) for defensive
copy-detection, same rationale as the phpactor entry. Verified clean:
no verbatim string/comment overlap.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Check 8d still locked the matcher to the pre-#963 'Grep|Glob' and failed
the pr-smoke leg on all platforms. The matcher now includes Read (the
augmenter injects the coverage note when a not-fully-indexed file is
read; structurally non-blocking, so the issue-#362 gate hazard cannot
recur). The guard now pins the exact new matcher and still rejects
Search/catch-all creep.
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
install.ps1 kept detecting arch=amd64 on windows-11-arm because it runs under x64
emulation, where neither $env:PROCESSOR_ARCHITECTURE nor .NET OSArchitecture (on
Windows PowerShell 5.1) reports the real Arm64. Add a CBM_ARCH env override to
install.ps1 (wins over auto-detect; also a genuine escape hatch for emulated
invocations) and have smoke Phase 13 pass the authoritative DL_ARCH. Deterministic
-- no reliance on in-process detection under emulation. Same emulated-arch class
as #907 (uname) and #908.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The persistent windows-11-arm smoke Phase 12a failure was a 404, not a network
error: it requested codebase-memory-mcp-windows-amd64.zip on the arm64 leg. Cause:
DL_ARCH came from `uname -m`, which on windows-11-arm is an emulated x86_64 MSYS2
uname reporting "x86_64" -> wrong (amd64) archive -> 404 (server has arm64). Phase
14 worked because the binary's own detect_arch() is native. Prefer SMOKE_ARCH
(passed from the smoke workflow's matrix.arch) over uname; fall back to uname for
local runs. This is the real cause the earlier curl/proxy/ipv4 attempts masked --
the 404 was swallowed by 2>/dev/null until #905 surfaced it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Phase 12a swallowed curl's stderr (2>/dev/null), so the persistent windows-arm64
"curl download failed" was undiagnosable. Two changes: (1) add --noproxy '*' so
curl never routes the local 127.0.0.1 test server through a proxy env var -- a
strong candidate since the app's own WinHTTP downloader reaches the server in
Phase 14 while only msys2 curl fails, instantly; (2) print curl's stderr on
failure so the actual reason is visible if it persists. Harmless on all
platforms (the smoke server is always local).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Closes#423, #20. A non-ASCII repo path passed to
`codebase-memory-mcp cli index_repository "<json>"` was mangled on Windows, so the
command failed with "repo_path is required" instead of indexing the real directory.
POSIX is unaffected (argv is already UTF-8). Two boundaries had to be fixed:
1. argv read: main() took only the narrow `int main(int argc, char **argv)`, so the
CRT handed it argv in the active ANSI code page. On Windows main() now rebuilds
argv from the wide command line (GetCommandLineW + CommandLineToArgvW) and
converts each element to UTF-8 (cbm_wide_to_utf8).
2. worker spawn: the index supervisor (and the UI index thread) spawned the worker
via CreateProcessA, which re-interpreted the UTF-8 command line through the ANSI
code page -- re-mangling the path at the parent->worker boundary even after (1).
Both spawns now build a wide command line (cbm_utf8_to_wide) and use CreateProcessW.
Without (2) the default (supervisor-enabled) path stayed broken while only the
in-process path worked, so this is the complete end-to-end fix, not just argv.
Promotes tests/windows/test_cli_non_ascii_arg.py from an opt-in known-red to a green
Windows guard (scripts/test-windows.ps1 + the test-windows-guards CI job). The guard
now exercises the DEFAULT supervised path (it drops the suite's CBM_INDEX_SUPERVISOR=0
so it actually crosses the worker-spawn boundary). Verified on native Windows: RED
before the fix, GREEN after (a non-ASCII repo indexes, nodes>0). Also adds -lshell32
explicitly (main.c now uses CommandLineToArgvW) and refreshes RED_TEST_ANALYSIS.md.
Signed-off-by: Flipper <jacobphilipp@ymail.com>
Insert the FastAPI benchmark probe as an APIRouter method instead of a top-level function before an indented method declaration. The old mutation produced invalid Python and contaminated scoped-LSP overlay diagnostics with parser fallout.
Compile the mutated source before writing it so future FastAPI probe changes fail fast with a precise error instead of producing misleading benchmark data.
Validation: uv run python -m py_compile scripts/benchmark-incremental-speed.py; uv run python -m py_compile /private/tmp/cbm-fastapi-valid-probe-current-20260704T2250/fastapi_insert_probe/repo/fastapi/routing.py; bash scripts/check-source-safety.sh; git diff --check; FastAPI matrix /private/tmp/cbm-fastapi-valid-probe-current-20260704T2250.json passed with canonical_equal=true and publish_kind=full.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Andrew Hundt