Andrew Hundt 33ef7249fe fix(benchmark): separate default response cost from quality JSON ...

Route run_cli_tool_call() and run_mcp_tool_call() through a default-format request for elapsed_ms, response_bytes, and response_token_estimate, then issue format=json only for quality scoring. Record quality_probe_elapsed_ms and quality_response_bytes so the second request cannot inflate the user-facing latency or replace TOON payload sizing.

This fixes the pilot JSONDecodeError: Expecting value at line 1 column 1 when search_graph returned its default TOON content. Document the two-request instrumentation contract in docs/BENCHMARK_CAMPAIGN.md.

Validated with 23 focused pytest cases across tests/test_benchmark_incremental_speed.py, tests/test_summarize_benchmark_results.py, and tests/test_benchmark_campaign.py; Ruff and git diff --check pass.

Signed-off-by: Andrew Hundt <ATHundt@gmail.com>

2026-07-14 13:42:37 -04:00

..

repro

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

windows

fix(writer): open the graph DB wide-safe for non-ASCII cache paths

2026-07-12 19:57:47 +02:00

grammar_cases.h

Add per-grammar regression suite across all 159 grammars

2026-06-05 21:53:25 +02:00

scale_contract.sh

Add per-grammar regression suite across all 159 grammars

2026-06-05 21:53:25 +02:00

smoke_guard.sh

fix: consolidate trace API and rank persistence

2026-06-28 08:36:56 -04:00

test_ac.c

Resolve all clang-tidy, cppcheck, and clang-format warnings across 94 files

2026-03-15 22:37:34 +01:00

test_arena.c

fix(arena): NULL-arena guard in cbm_arena_alloc; skip vendored/ dirs

2026-05-30 18:26:44 +02:00

test_artifact.c

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_ast_profile.c

fix: char buf[1] = "X" → {'X'} for -Wunterminated-string-initialization on Windows/clang

2026-06-29 09:45:28 -06:00

test_benchmark_campaign.py

fix(benchmark): reject harness-error result files

2026-07-14 12:03:56 -04:00

test_benchmark_incremental_speed.py

fix(benchmark): separate default response cost from quality JSON

2026-07-14 13:42:37 -04:00

test_c_lsp.c

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_cli.c

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_configlink.c

fix(pipeline): stabilize incremental graph parity

2026-07-02 07:37:19 -04:00

test_convergence_probe.c

fix: harden index persistence and MCP outputs

2026-06-28 06:34:23 -04:00

test_cpp_index_hang.sh

Fix O(n^2) overload resolution in the C/C++ cross-LSP registry (#410)

2026-06-06 22:16:04 +02:00

test_cs_lsp_bench.c

fix(test): detect thread-sanitized benchmarks

2026-07-13 04:34:16 -04:00

test_cs_lsp.c

feat(lsp): add C# / .NET hybrid LSP resolver

2026-05-19 22:47:53 +02:00

test_cypher.c

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_depindex.c

fix(runtime): preserve TOON contracts and index process ownership

2026-07-14 01:55:02 -04:00

test_discover.c

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_dump_verify_io.c

fix(leak): unhang dump_verify_io_fork_crash under leaks --atExit

2026-06-24 17:03:07 -04:00

test_dump_verify.c

Add post-dump plausibility gate returning status:degraded (#334)

2026-06-23 06:38:45 +09:00

test_dyn_array.c

Fix Windows build: add missing stdint.h include in test_dyn_array

2026-03-25 09:28:11 +01:00

test_edge_imports.c

docs(tests): refresh import edge matrix status

2026-06-30 10:19:32 -04:00

test_edge_structural.c

fix: harden index persistence and MCP outputs

2026-06-28 06:34:23 -04:00

test_edge_types_probe.c

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_extraction_imports.c

Add language-agnostic extraction reproduction suite (inheritance + imports)

2026-06-07 12:35:53 +02:00

test_extraction_inheritance.c

Fix CI lint: drop debug scaffolding, make lisp import walker iterative, apply clang-format

2026-06-08 22:55:23 +02:00

test_extraction.c

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_foundation_main.c

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_fqn.c

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_framework.h

fix(test): detect thread-sanitized benchmarks

2026-07-13 04:34:16 -04:00

test_git_context.c

fix(test): guard git_context git helpers behind #ifndef _WIN32

2026-07-02 00:37:52 +02:00

test_gitignore.c

fix(discover): make cbm_gitignore_merge atomic on allocation failure

2026-06-22 23:51:49 +02:00

test_go_lsp.c

Resolve all clang-tidy, cppcheck, and clang-format warnings across 94 files

2026-03-15 22:37:34 +01:00

test_grammar_imports.c

Add per-grammar regression suite across all 159 grammars

2026-06-05 21:53:25 +02:00

test_grammar_labels.c

Add Mojo language wiring

2026-07-08 00:05:19 +05:30

test_grammar_probe_a.c

fix: harden index persistence and MCP outputs

2026-06-28 06:34:23 -04:00

test_grammar_probe_b.c

fix: harden index persistence and MCP outputs

2026-06-28 06:34:23 -04:00

test_grammar_probe_c.c

fix: harden index persistence and MCP outputs

2026-06-28 06:34:23 -04:00

test_grammar_probe_d.c

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_grammar_probe_e.c

fix: harden index persistence and MCP outputs

2026-06-28 06:34:23 -04:00

test_grammar_probe_f.c

fix: harden index persistence and MCP outputs

2026-06-28 06:34:23 -04:00

test_grammar_probe_g.c

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_grammar_regression.c

Add Mojo language wiring

2026-07-08 00:05:19 +05:30

test_graph_buffer.c

fix(incremental): preserve sibling named imports

2026-07-14 05:06:05 -04:00

test_graph_diff.h

fix(graph): harden determinism and safety canaries

2026-06-30 00:38:07 -04:00

test_hash_table.c

perf(foundation): swap CBMHashTable internals to Verstable v2.2.1

2026-05-21 22:52:16 +02:00

test_helpers.h

Eliminate NOLINTNEXTLINE suppressions + iterative AST walkers

2026-03-31 20:04:11 +02:00

test_httpd.c

fix(ui): close HTTP sockets across exec

2026-07-14 05:33:30 -04:00

test_httplink.c

fix(httplink): harden exclude path ownership

2026-07-01 23:52:19 -04:00

test_incremental.c

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_index_resilience.c

fix(runtime): preserve TOON contracts and index process ownership

2026-07-14 01:55:02 -04:00

test_infrascan.c

fix(pipeline): preserve sveltekit route ownership

2026-07-01 04:04:08 -04:00

test_input_validation.c

fix(runtime): preserve TOON contracts and index process ownership

2026-07-14 01:55:02 -04:00

test_integration.c

fix: harden index persistence and MCP outputs

2026-06-28 06:34:23 -04:00

test_java_lsp_coverage.c

feat(lsp): Java hybrid LSP resolver (type-aware calls, annotations, signatures)

2026-06-02 18:52:37 +02:00

test_java_lsp.c

fix(lsp): restore scoped call resolution

2026-07-01 07:37:23 -04:00

test_kotlin_lsp.c

fix(lsp): restore builtin call targets

2026-07-01 07:22:12 -04:00

test_lang_contract.c

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_language.c

Add Mojo language wiring

2026-07-08 00:05:19 +05:30

test_log.c

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_lsp_resolution_probe.c

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_lz4.c

Resolve all clang-tidy, cppcheck, and clang-format warnings across 94 files

2026-03-15 22:37:34 +01:00

test_main.c

fix(lifecycle): cancel and reap auto-index process trees

2026-07-14 13:14:43 -04:00

test_matrix_known_classes.c

fix: harden index persistence and MCP outputs

2026-06-28 06:34:23 -04:00

test_matrix_new_constructs.c

fix: harden index persistence and MCP outputs

2026-06-28 06:34:23 -04:00

test_mcp.c

fix(lifecycle): cancel and reap auto-index process trees

2026-07-14 13:14:43 -04:00

test_mem.c

feat(mem): pure budget resolver with strict parse + clamp for CBM_MEM_BUDGET_MB

2026-07-08 06:03:54 -04:00

test_node_creation_probe.c

fix: harden index persistence and MCP outputs

2026-06-28 06:34:23 -04:00

test_pagerank.c

feat(config): add auditable graph capability gates

2026-07-14 06:07:39 -04:00

test_parallel.c

fix(indexing): preserve skips and canonical Spring routes

2026-07-14 02:49:30 -04:00

test_parent_watchdog.sh

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_parse_coverage.c

feat(index): missed graph in UI + read-time coverage hook + recovery subtraction

2026-07-08 19:13:25 +02:00

test_path_alias.c

fix(pipeline): honor discovery exclusions in pkgmap/path-alias/envscan walks

2026-07-03 19:57:01 +02:00

test_perl_lsp.c

fix(perl-lsp): split qualified calls on last "::" so multi-level packages resolve

2026-07-09 14:09:39 -05:00

test_php_lsp.c

style: restore original test_php_lsp.c formatting

2026-07-09 22:32:26 +02:00

test_pipeline.c

fix(lifecycle): close incremental arenas and failed SQLite handles

2026-07-14 11:16:05 -04:00

test_platform.c

refactor(platform): centralize env flag parsing

2026-07-02 04:07:36 -04:00

test_py_lsp_bench.c

fix(test): detect thread-sanitized benchmarks

2026-07-13 04:34:16 -04:00

test_py_lsp_scale.c

perf(py_lsp): cap dedup window at 256 entries — 91.8x -> 31.4x scaling

2026-05-09 15:44:09 +02:00

test_py_lsp_stress.c

feat(py_lsp): close all stress gaps — lambda, dict-dispatch, async, callable, next, kwargs

2026-05-09 14:16:15 +02:00

test_py_lsp.c

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_registry.c

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_route_canon.c

refactor(pipeline): consolidate route identity

2026-06-30 06:05:15 -04:00

test_rust_lsp.c

perf(rust-lsp): share the all_defs cross registry across null-filter files

2026-07-06 18:32:50 +02:00

test_scope.c

refactor(lsp): chunked linked-frame CBMScope + MRO depth cap

2026-05-08 22:26:56 +02:00

test_security_strings_allowlist.sh

fix(ci): allow-list MSYS2 toolchain URL in binary string audit

2026-06-24 03:15:36 +02:00

test_security.c

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_semantic.c

perf(semantic): quantize the resident semantic vectors (rotated 4-bit SQ)

2026-07-06 21:52:37 +02:00

test_simhash.c

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_slab_alloc.c

test: add 56 unit tests for semantic, slab_alloc, and ast_profile modules

2026-06-28 21:57:35 -06:00

test_sqlite_helpers.h

feat(store): add exact-delta metadata schema

2026-06-30 11:00:27 -04:00

test_sqlite_writer.c

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_stack_overflow.c

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_store_arch.c

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_store_bulk.c

fix(leak): fix 3 memory leaks and 2 waitpid hangs under leaks --atExit

2026-04-11 09:22:23 -04:00

test_store_checkpoint.c

fix(store): remove destination WAL/SHM sidecars before installing a fresh DB

2026-07-11 02:40:54 +02:00

test_store_edges.c

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_store_nodes.c

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_store_pragmas.c

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_store_search.c

feat(store): gate stale derived rankings

2026-06-30 21:24:07 -04:00

test_str_intern.c

Add 480+ tests, fix 3 bugs discovered by new test coverage

2026-03-24 23:48:23 +01:00

test_str_util.c

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_subprocess.c

fix(lifecycle): cancel and reap auto-index process trees

2026-07-14 13:14:43 -04:00

test_summarize_benchmark_results.py

test(benchmark): rank expected query results

2026-07-14 12:09:22 -04:00

test_token_reduction.c

fix(runtime): preserve TOON contracts and index process ownership

2026-07-14 01:55:02 -04:00

test_tool_consolidation.c

fix(runtime): preserve TOON contracts and index process ownership

2026-07-14 01:55:02 -04:00

test_traces.c

Add 480+ tests, fix 3 bugs discovered by new test coverage

2026-03-24 23:48:23 +01:00

test_ts_lsp.c

refactor(lsp): reuse safe env flag parsing

2026-07-02 03:55:39 -04:00

test_type_rep.c

Fix uninitialized template-arg arrays binding stack garbage into type graphs

2026-06-10 12:11:02 +02:00

test_ui.c

feat(ui): honor the requested node budget up to a 10M ceiling

2026-07-07 23:09:05 +02:00

test_userconfig.c

feat(discover): port user language config module from origin/main (commit 1cdb983)

2026-04-09 23:15:10 -04:00

test_vmem.c

Forbid test skips and convert existing skips to hard failures

2026-06-05 21:53:25 +02:00

test_watcher.c

merge(upstream): integrate main through Laravel route fixes

2026-07-14 01:47:43 -04:00

test_worker_pool.c

perf(parallel): clamp workers to iteration count

2026-07-01 23:42:49 -04:00

test_worker_watchdog.sh

fix(supervisor): wrap only marked host binaries; watchdog in worker mode

2026-07-04 14:38:25 +02:00

test_yaml.c

tests,foundation: add origin/main test files and security functions before merge

2026-04-10 20:09:37 -04:00

test_zstd.c

feat: add persistent artifact storage for team sharing

2026-04-15 23:56:03 +02:00