src/depindex/depindex.c now applies dep_max_files with CBM_MODE_DEP bounded discovery before parsing, skips oversized packages atomically, preserves the existing cbm_dep_auto_index_effective API as a wrapper, and records package/file-cap statistics.
src/mcp/mcp.c returns dependency_auto_index fields and actionable index_dependencies recovery from index_repository. src/cli/hook_augment.c reports active auto_dep_limit and dep_max_files values through the shared normalizers; disabled and unlimited configurations omit inapplicable recovery text.
README.md, docs/CONFIGURATION.md, and the cli.c registry document the enforced behavior. Tests replace the former 1000==1000 assertion with real skip/unlimited fixtures and cover serialized MCP feedback plus config-sensitive hook guidance.
Verification: CLI 243 passed; MCP 274 passed; depindex 42 passed; tool_consolidation 113 passed; input_validation 56 passed; bash scripts/check-source-safety.sh; git diff --cached --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
docs/BENCHMARK_EXPERIMENTS.md presented optional_graph_disabled as a current command even though new experiment plans emit only minimal_indexing. Document minimal_indexing as the active command and retain optional_graph_disabled solely as a historical-plan loader alias.
Align README.md with the 16-tool classic contract asserted by tests/test_tool_consolidation.c, remove a stale source-line reference in tests/test_mcp.c, and use CBM_DEFAULT_AUTO_INDEX_LIMIT for CLI fallback assertions.
Verification: CBM_ONLY_SUITE=cli make -f Makefile.cbm test (243 passed); CBM_ONLY_SUITE=mcp make -f Makefile.cbm test (273 passed); bash scripts/check-source-safety.sh.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
build_automatic_spec emitted optional-graph-disabled and minimal-indexing with identical candidate labels and capability manifests, repeating the same measurements under two names.
Emit only minimal-indexing in new automatic plans. Keep optional_graph_disabled as a retained-plan loader alias backed by the shared MINIMAL_INDEXING_CAPABILITIES map, document that compatibility boundary, and assert candidate/capability signature uniqueness. Correct the auto_dep_limit effective-sentinel comment in src/depindex/depindex.c.
Verified 207 passed, 1 skipped, and 34 subtests across the six benchmark Python suites; Ruff format/check and git diff --check passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
ha_load_guidance_config previously rendered any negative auto_dep_limit from a retained or manually edited _config.db as unlimited, while dependency indexing falls back to CBM_DEFAULT_AUTO_DEP_LIMIT.
Reuse cbm_dep_normalize_configured_limit, document the configured and effective sentinel contracts in src/depindex/depindex.h, and consolidate the raw-config fixture in tests/test_helpers.h. Clarify in docs/BENCHMARK_EXPERIMENTS.md that retired script paths are load-time aliases rather than executable wrappers.
Verified CBM_ONLY_SUITE=cli (243 passed), CBM_ONLY_SUITE=pagerank (60 passed), CBM_ONLY_SUITE=depindex (42 passed), changed-line clang-format checks, and git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Restore scripts/benchmark-index.sh, scripts/benchmark-search-graph.sh, and scripts/clone-bench-repos.sh to their upstream/main paths and exact blob hashes 756bda06, cc94147e, and 883e0788.
Rename the branch-created single-run harness to benchmarks/run_benchmark.py. Remove historical Python frontends and scripts/_benchmark_compat.py so canonical benchmark execution has no shim or sys.path mutation.
Add resolve_benchmark_script_path() and validate_benchmark_script_digest() in benchmarks/run_experiments.py. Retained scripts/benchmark-incremental-speed.py plan entries resolve at execution time without rewriting archived plans; missing cells fail closed when the resolved harness SHA-256 differs from the recorded benchmark_script_sha256.
Ignore benchmark-results/ and *.facts/ while retaining the existing .worktrees/ ignore. Document run_benchmark.py versus run_experiments.py and preserve retained campaign flag and directory spellings.
Verification: 207 passed, 1 skipped, 34 subtests passed; retained 42-cell audit reported 42 complete, 0 missing, 0 corrupt, 0 unplanned; Ruff E4/E7/E9/F, terminology generator --check, comparisons-v1 schema validation, git diff --check, and scripts/check-source-safety.sh passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Move incremental_speed.py, run_experiments.py, summarize_results.py, fact_comparisons.py, autotune.py, active v2 schemas, terminology.json, configuration spelling data, and shell harnesses into benchmarks/.
Keep scripts/_benchmark_compat.py plus historical Python and shell filenames as thin compatibility frontends. Preserve docs/schema/benchmark-facts-v1.schema.json because retained v1 bundles embed that exact URI; accept the former v2 URI only while loading retained results.
Regenerate docs/BENCHMARK_TERMINOLOGY.md and src/foundation/profile_terms_generated.h from benchmarks/terminology.json. Update CONTRIBUTING.md, MAINTAINERS.md, docs/BENCHMARK_EXPERIMENTS.md, docs/EVALUATION_PLAN.md, and tests to use canonical paths.
Verification: 205 passed, 1 skipped, 34 subtests passed; Ruff E4/E7/E9/F passed; comparisons-v1 schema validation passed for 42 retained bundles and 91 comparisons; generator --check, bash -n, git diff --check, and scripts/check-source-safety.sh passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add scripts/benchmark_fact_comparisons.py and docs/schema/benchmark-comparisons-v1.schema.json to group repetitions by effective implementation, capability, scope, cache, host, harness, terminology, and correctness contracts.
Emit elapsed ratios only for parity_manifest_and_contract_v1 pairs; emit capability differences without ratios for capability_delta_manifest_v1 pairs; preserve source run and occurrence IDs and outer lifecycle wall-time rows.
Extend fact_result_rows() with graph, freshness, and retrieval-quality outcomes and append generated comparison artifacts from generate_report(). Retained reports without fact bundles remain readable and report comparison status as unavailable.
Define the versioned join and formula IDs in benchmark terminology 1.1.0. Verified 198 passed, 1 skipped, and 34 subtests; Draft 2020-12 schema validation, Ruff, source-safety, and git diff checks pass.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add docs/benchmark-terminology.json with 86 validated definitions and 17 ordered step IDs, then generate docs/BENCHMARK_TERMINOLOGY.md and src/foundation/profile_terms_generated.h from that single source.
Emit terminology_version, canonical terminology SHA-256, and generator SHA-256 in v2 fact bundles and manifests. Preserve docs/schema/benchmark-facts-v1.schema.json byte-for-byte and load retained v1 bundles without inventing missing terminology metadata.
Verify with 192 passed, 1 skipped, and 34 subtests across the benchmark experiment, shim, normalization, and report suites; Draft 2020-12 validation for both new schemas; scripts/check-source-safety.sh; generator no-drift check; ruff; and git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Read canonical and historical config spellings from scripts/benchmark-config-spellings-v1.json so mixed candidate runs translate only renamed key/value pairs.
Run the once-per-binary probe under a temporary CBM_CACHE_DIR while holding CONFIG_SPELLING_MODES_LOCK. This prevents rank_refresh detection from mutating candidate-default experiment state or racing across workers.
Mark cache state unknown when imported reports did not record it, display-canonicalize retained fact labels without rewriting inputs, and reject conflicting old/new values after canonicalization.
Verification: 176 passed and 34 subtests; ruff format --check and ruff check passed; jq parsed the spelling map; git diff --check passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Set incremental_reindex=always and name its alternatives full_rebuild and fast_mode_indexes_only in src/pipeline/pipeline.h and src/cli/cli.c.
Replace eager/stale_on_* development spellings with at_publish and defer_*_reindexes for derived-result and rank refresh policies. Keep obsolete spellings out of the product registry.
Restrict pass_calls.c same-target registry metadata canonicalization to exact store-backed scratch graphs, preserving LSP target precedence without adding a lookup to normal full or parallel indexing.
Verification: pipeline 385 passed; cli 243 passed; mcp 273 passed; tool_consolidation 113 passed; pagerank 60 passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Make automatic_dependency_source_indexing_disabled the benchmark default and add the paired enabled profile. Scope candidate_native_configuration to older candidates, reject unbacked capability claims, and pin all latest-product graph capabilities so one-factor ablations remain comparable.
Remove inherited CBM_* product variables, retain isolated per-case caches, record native worker selection, and emit requested/effective configuration facts without inventing defaults for candidate-native or legacy reports. Keep retained historical reports readable.
Verification: uv run --with pytest python -m pytest tests/test_benchmark_incremental_speed.py tests/test_benchmark_experiments.py tests/test_benchmark_experiments_shim.py tests/test_autotune.py tests/test_summarize_benchmark_results.py (184 passed, 1 skipped); ruff format --check and ruff check passed for all touched Python files.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Define CBM_DEFAULT_AUTO_INDEX_DEPS and use it in depindex, MCP request handling, hook guidance, and the config registry. Replace the two quality presets with four exact streamlined/classic presets that state whether automatic installed dependency-source indexing is enabled. Keep rank and optional-graph benchmark ablations on the dependency-disabled baseline.
Document that index_dependencies remains available and disabling automation does not delete stored dependency projects.
Verification: CBM_ONLY_SUITE=cli make -f Makefile.cbm test (243 passed); CBM_ONLY_SUITE=depindex make -f Makefile.cbm test (40 passed); CBM_ONLY_SUITE=input_validation make -f Makefile.cbm test (56 passed).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Write runs.json, steps.jsonl, results.json, artifacts.json, facts.json, and hashed manifests from scripts/benchmark-incremental-speed.py. Record candidate revision/build context from scripts/run-benchmark-experiments.py and keep absent legacy metadata as explicit unknown facts.
Use experiment terminology for new runner, composition, autotune, provenance, and documentation records while retaining legacy campaign keys, flags, paths, and the shim at compatibility boundaries.
Verified by 182 Python tests (1 skipped, 25 subtests), a rebuilt-binary smoke with 18 emitted step rows, and Draft 2020-12 validation of the generated facts bundle against docs/schema/benchmark-facts-v1.schema.json.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Teach ha_format_api_guidance and ha_format_freshness_guidance to report effective tool_mode, auto_index, context_injection, auto_watch, auto_index_deps, and cap values from the configuration registry.
Keep streamlined lifecycle output on search_graph, trace_path, get_code, and query_graph without hidden-tool instructions. Preserve the classic ordered search_graph -> trace_path -> get_code_snippet workflow, direct advanced-tool visibility, tracing examples, coverage rules, architecture guidance, and tier selection in installed artifacts.
Replace the Gemini BeforeTool command with mode-neutral graph-tool guidance while recognizing the prior owned JSON command during upgrades. Add artifact-contract and configuration-branch tests; CBM_ONLY_SUITE=cli make -f Makefile.cbm test passes 240 tests.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Read tool_mode, auto_index, auto_index_limit, auto_index_deps, and auto_dep_limit without creating _config.db in ha_load_guidance_config at src/cli/hook_augment.c:1153.
Route Gemini SessionStart through the event-compatible hook-augment command in cbm_upsert_gemini_session_hooks at src/cli/cli.c:4515; retain the released static command only for owned-entry cleanup.
Share CBM_DEFAULT_AUTO_INDEX_LIMIT from src/cli/cli.h:390 and document streamlined reveal, classic source retrieval, and dependency caps in README.md and docs/CONFIGURATION.md.
Verify tests/test_cli.c:4169 plus cli 238, mcp 272, tool_consolidation 113, and input_validation 56 passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Move the full implementation from run-benchmark-campaign.py to
run-benchmark-experiments.py and reduce the old filename to a
backwards-compatible shim that loads the new module by path and
re-exports every public name, so direct imports
(tests/test_benchmark_campaign.py, scripts/autotune.py,
scripts/summarize-benchmark-results.py) and existing invocations keep
working with byte-identical behavior.
- Add --experiment-root and --allow-temporary-experiment-root, keeping
--campaign-root and --allow-temporary-campaign-root as accepted
aliases on both entry points.
- Add --candidate-ref LABEL=REF override and an upstream/main ->
origin/main -> main fallback chain so --quick/--full keep working
after the api-consolidation branch is merged and its pinned refs no
longer resolve; explicitly requested refs remain fail-closed.
- Say experiment instead of campaign in --help text, runtime error
text, and generated-report boilerplate (summarize-benchmark-results.py);
keep the campaign_version manifest key, the campaigns composition key,
and the .worktrees/benchmark-campaign/ directory so every retained
runset stays valid.
- Move docs/BENCHMARK_CAMPAIGN.md content to docs/BENCHMARK_EXPERIMENTS.md
and leave a pointer stub so existing links keep resolving.
- Add tests/test_benchmark_experiments_shim.py (25 subtests) proving the
shim re-exports the implementation and both CLIs accept old and new
flags; update the temporary-root error-text assertion.
Verified: ruff clean on all touched files; 171 passed, 1 skipped across
the four benchmark test files; --help output identical between the two
entry points except the program name.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add run_mcp_surface_parity() coverage for classic discovery, streamlined discovery before _hidden_tools, and relisting in the same process after reveal. Compare complete tools/list contracts, validation-equivalent get_code aliases, capability outcomes, bounded hidden-handler recognition, and notifications/tools/list_changed without presenting dispatch recognition as behavioral proof.
Make McpClient cleanup record subprocess return codes and require both reader threads and server processes to be reaped. Close SQLite snapshot and cloned-fixture connections explicitly so ResourceWarning-as-error runs detect ownership regressions.
Render capability outcomes before tool counts in scripts/summarize-benchmark-results.py, retain response bytes and single-observation latency with explicit evidence limits, and document the parity mode in docs/BENCHMARK_CAMPAIGN.md.
Verification: 141 warning-clean benchmark/campaign/summarizer tests; optimized build/c/codebase-memory-mcp parity smoke passed for 12 capability groups with full post-reveal classic contract parity and reaped lifecycle; ruff check; source-safety; git diff --cached --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Response-format consolidation: add cbm_mcp_response_format/
cbm_mcp_invalid_response_format so every response-generating handler
reads the format argument or default_response_format config through one
function instead of duplicating the enum-mapping logic, and migrate
query_graph's legacy-JSON detection to the canonical
cbm_mcp_output_format_t enum. Fix the regression this surfaced: TOON-path
_context/session_project delivery (toon_append_context_once) was
serializing the one-shot context object as a raw escaped JSON string
inside the TOON stream instead of native TOON fields, silently dropping
it for any TOON-consuming client. Add toon_append_context_model plus its
scalar/table helpers to map the same format-neutral context model
inject_context_once already builds into native TOON output.
Self-healing zero-row hints: query_graph_no_rows_hint reparses the
already-executed query on the zero-row path only, walks MATCH/OPTIONAL
MATCH patterns and EXISTS predicates in WHERE and post-WITH-WHERE
clauses (hint_walk_expr_exists_types/hint_walk_where_exists_types), and
probes each referenced label/edge type against the store via
cbm_store_schema_label_observed/cbm_store_schema_type_observed
(schema_probe_one_row: fail-open on any non-SQLITE_DONE outcome, so a
probe error never produces a false accusation). Unobserved names are
named in the response; a bounded hint_seen_t/hint_seen_add set (capped
at CBM_STORE_SCHEMA_HINT_VOCAB_LIMIT) dedups names repeated across
patterns/UNION branches so "Klass, Klass." can't happen and avoids
re-probing a name already resolved. The vocabulary summary and the
generic zero-row fallback no longer claim WHERE-clause predicates were
verified, since only labels/edge types are ever probed — only the
labels/edge types claim is made.
One invalidation authority: every graph-mutation path now calls
the single cbm_mcp_server_notify_index_published (flags-only,
any-thread-safe) instead of ad hoc local staleness bookkeeping —
sync auto-index, in-process index_repository, in-process
autoindex_thread, overlay compaction (gated on compacted>0),
index_dependencies. handle_delete_project's notify is gated on the
mutation actually succeeding (!is_error): a delete of a nonexistent
project is a no-op and must not stale the tools/list description cache
for nothing. The request-thread-only drain
(mcp_drain_tools_list_changed) is factored once and shared by both
transports (line-delimited and Content-Length-framed), gated so a
notification published before any tools/list has ever been served is
suppressed rather than sent for a client that never asked.
Shared overlay-aware schema-view selector (mcp_get_current_schema/
mcp_overlay_view_ready) ends the schema/overlay-readiness dance
previously triplicated across handle_get_graph_schema,
build_resource_schema, and the query path, so query_graph's advertised
vocabulary can never diverge from the view a query actually runs
against. The selector takes an mcp_cypher_vocabulary_t level instead of
a bool: MCP_CYPHER_MATCH_VOCABULARY (labels/edge types/observed
patterns with counts — enough to compose executable MATCH clauses;
never parses property JSON) vs MCP_CYPHER_FULL_QUERY_VOCABULARY (adds
per-label/type property keys for WHERE/RETURN authorship; O(total
property rows) json_each discovery, so only the docstring cache and
get_graph_schema request it).
Route the two remaining canonical-only cbm_store_get_schema callers
through the selector at MATCH_VOCABULARY level: inject_context_once
(first-response _context previously paid the full property-key scan for
counts it never rendered, and could advertise canonical labels a ready
overlay had tombstoned) and build_resource_architecture (rendered only
rel_patterns yet fetched full property keys, canonical-only). The
architecture resource's mixed-read-model disclosure moves
relationship_patterns into the overlay-aware list accordingly.
build_query_graph_tool_description resolves the project's real
store via resolve_store(srv, project) instead of trusting srv->store
as-is: on the very first tools/list of a fresh session for an
already-indexed project, srv->store was still the empty default
in-memory store (maybe_auto_index detects "already indexed" and skips
reindexing without ever opening it), so the schema section rendered
silently empty until some other tool call happened to resolve it first.
The project name is snapshotted into a local buffer before the
resolve_store call rather than passed as a live alias to
srv->current_project, because resolve_store can call
reap_stale_store internally, which frees srv->current_project when a
deferred invalidation is pending — aliasing it produced a
heap-use-after-free (caught by the ASan full-suite run, confirmed fixed
by rerun).
Docstring clarity for newcomers: the live schema section now embeds its
notation legend in the section header ("Labels name{extra property
keys}[count]:") so the compact schema is self-explanatory in every
tools/list response, and the query_graph/get_architecture/search_code
format parameters gain the brief description their three siblings
already had, all naming default_response_format.
Tests: response-format and TOON-context regressions in test_mcp.c/
test_tool_consolidation.c/test_input_validation.c/test_token_reduction.c;
zero-row hint coverage including no-hidden-tool-name, missed-graph shadow-
project probing, empty-project no-crash, TOON/JSON format parity,
UNION/repeated-name dedup, and the previously-untested EXISTS-predicate
walk; notification coverage for every notify call site including the
delete_project no-op negative direction and the "notify before first
tools/list is suppressed" negative direction; a cold-start regression
test seeding a real on-disk project store and calling tools/list as the
literal first request of a fresh session; overlay-consistency tests for
the two rerouted callers (first_response_context_uses_ready_overlay_schema,
resource_arch_rel_patterns_use_ready_overlay) proving _context and the
architecture resource report the active-overlay vocabulary, not
tombstoned canonical labels/types.
Docs: README.md documents the zero-row hint behavior and the
default_response_format config example; docs/CONFIGURATION.md adds the
default_response_format row to the important-keys table.
Verification: make -f Makefile.cbm test (full ASan/UBSan suite) run on
this exact tree after the final revisions. Earlier iterations of this
content additionally passed make -f Makefile.cbm test-leak (0 leaks for
0 total leaked bytes) and manual checks against the live MCP binary
(zero-row hint names real schema on two repositories; the query_graph
tools/list docstring shows the live schema on the very first request of
a fresh session); revisions since those checks are docstring text, the
vocabulary-enum rename, and the two selector reroutes, all covered by
the full-suite rerun.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
docs/CONFIGURATION.md:82 previously listed auto_index=false even though CBM_CONFIG_REGISTRY in src/cli/cli.c defaults the key to true. This could lead users to assume streamlined first-use indexing was disabled.
tests/test_cli.c:3837 now derives the expected Markdown row from CBM_CONFIG_REGISTRY, so future default changes must keep the configuration table synchronized.
Verification: CBM_ONLY_SUITE=cli build/c/test-runner (151 passed); bash scripts/check-source-safety.sh; git clang-format --diff HEAD -- tests/test_cli.c.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
scan_campaign() validated completion and result hashes but trusted the artifact inventory in attempt.json. Changed bytes, a missing worker log, or an unlisted extra file therefore remained status=complete even though phase timing and lifecycle evidence no longer matched the recorded attempt.
Add validate_attempt_artifacts() in scripts/run-benchmark-campaign.py. Completed attempts now require matching cell identity and status, then rebuild the recursive artifact inventory and compare every relative path, byte count, SHA-256, file count, and total byte count. Completions without an attempt field retain historical hand-authored-plan compatibility.
Add changed, missing, and unlisted artifact regressions in tests/test_benchmark_campaign.py and document audit-only re-inventory in docs/BENCHMARK_CAMPAIGN.md. Verification: 135 benchmark/campaign/report unittests passed; Ruff, scripts/check-source-safety.sh, and git diff --check passed. Strict audit-only scans rehashed 39 capability cells, 9 semantic cells, and 9 similarity cells with zero corrupt, missing, duplicate, or unplanned results.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
evaluate_pair_incremental_policy() treated every unconfigured binary as stale_on_incremental. The pinned upstream binary produced immediate similarity pair freshness without a stale warning, so the evaluator rejected correct observed behavior against a policy that upstream does not expose.
Record candidate_default when incremental_derived_refresh is not explicitly configured. Classify the observed lifecycle as immediate_pair_freshness, immediate_full_freshness, deferred_with_warning, or unreported_stale; retain strict contract checks for explicit eager and deferred overrides. Separate pair freshness from whole-canonical-graph equality so each category remains visible.
Document cross-version default handling in docs/BENCHMARK_CAMPAIGN.md and extend tests/test_benchmark_incremental_speed.py with deferred, immediate-pair, unreported-stale, and explicit-eager cases.
Verification: uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign tests.test_summarize_benchmark_results (133 tests); uv run ruff check on benchmark/report files; bash scripts/check-source-safety.sh; git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Large c_new_leaf runs with the default stale_on_incremental policy passed every mutation oracle but differed from fresh rebuilds by 99 SEMANTICALLY_RELATED rows. summarize-benchmark-results.py labeled those disclosed stale views as REJECT: graph correctness, hiding the intended latency/freshness policy.
benchmark-incremental-speed.py now unions structured stale_with_warning views and reruns canonical comparison after excluding only SEMANTICALLY_RELATED rows when semantic_edges is explicitly stale. The declared-stale gate passes only when every remaining node, edge, property, and file hash equals the fresh rebuild; non-semantic or undeclared differences still fail.
summarize-benchmark-results.py reports Core graph separately from Full graph freshness and emits PASS: DECLARED STALE VIEWS for a verified deferred view. docs/BENCHMARK_CAMPAIGN.md documents the gate and tests cover stale-view collection, exact exclusion scope, core mismatches, decision text, and table categories.
Verification: uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign tests.test_summarize_benchmark_results (133 tests); uv run ruff check on changed Python files; bash scripts/check-source-safety.sh; git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
scripts/benchmark-incremental-speed.py previously wrote rank_refresh=stale_on_exact for every matrix cell, including profiles labeled default. That changed candidate behavior and invalidated comparisons against binaries with different compiled policies.
Add RANK_REFRESH_CANDIDATE_DEFAULT and apply_rank_refresh_override() so default runs make no rank_refresh config write. Record the requested policy and whether an override was applied in synthetic, matrix, self-dogfood, and capability-quality reports; retain explicit eager, stale_on_exact, and stale_on_incremental experiments.
Document the comparison contract in docs/BENCHMARK_CAMPAIGN.md and cover CLI default, no-write, and explicit-write behavior in tests/test_benchmark_incremental_speed.py.
Verification: uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign (90 tests); uv run ruff check scripts/benchmark-incremental-speed.py tests/test_benchmark_incremental_speed.py; bash scripts/check-source-safety.sh; git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add self_dogfood workload expansion in scripts/run-benchmark-campaign.py with exact repository revision/tree identities, candidate-scoped profiles, and result rejection on background mismatches.
Make create_self_dogfood_worktree check out the declared commit, retain pre/post mutation SHA-256 values, and add c_new_leaf for a real indexed create-file route alongside high-fanout fallback scenarios.
Document the JSON contract in docs/BENCHMARK_CAMPAIGN.md. Verify with 125 benchmark/report unit tests, Ruff, source-safety, and an optimized c_new_leaf smoke: incremental_exact 367 ms, fresh rebuild 4456 ms, canonical equality 1/1, cleanup complete.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previously, semantic pair fixtures ran only in synthetic repositories. A large-background run could copy ambient dirty files or duplicate this repository's tracked benchmark canaries, making cross-version quality and performance results neither isolated nor commit-auditable.
Add quality_background repo/revision/tree matrix identity and pass the exact commit to benchmark-incremental-speed.py. Materialize only tracked files with git archive, exclude dirty and untracked source state plus the benchmark task-definition subtree before overlaying one canary copy, safely validate archive paths, remove the transient archive, and retain source commit, tree, dirty status, exclusions, and copy policy. Campaign result validation rejects missing or mismatched background commit/tree identity.
The copy uses one Git archive process and linear extraction instead of one Git process per file. Runtime is O(tracked archive bytes + files), memory is bounded by tar metadata, and the source checkout/worktree registry is never modified.
Verification: uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed (76 passed); Ruff checks; git diff --check. The Git fixture proves committed VERSION=1 is copied while dirty VERSION=2, an untracked file, and tracked benchmarks/semantic-pairs-v1 canaries are excluded; campaign tests bind resolved path plus full commit/tree hashes and reject a retained revision mismatch.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previously, compact matrix expansion grouped all repetitions for one candidate/profile together, so thermal or ambient load drift could align with a configuration. Attempt records retained commands and logs but not pre/post host load, disk availability, or physical-memory context.
Add opt-in execution_order=paired_interleaved without changing legacy grouped expansion or archived plan hashes. The new order runs every candidate/profile cell once per repetition block and includes block and absolute position in cell identity. Record pre/post load averages, CPU count, physical memory when available, and campaign-filesystem total/used/free bytes in immutable command and attempt records plus the campaign environment snapshot.
The ordering pass is O(cells log cells) time and O(cells) plan memory; execution remains strictly sequential. Resource collection is constant-space and performs no background sampling or production instrumentation.
Verification: uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed (74 passed); Ruff checks; git diff --check. Tests also prove legacy grouped plans remain the default, paired plans execute repetitions as 1,1,1,1,2,2,2,2, and successful attempts retain both resource snapshots.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previously, semantic capability fixtures measured only the initial full index. They could not prove that a changed source file removed an old relationship, added a new relationship, or matched a fresh rebuild, and expected stale-on-incremental warnings appeared as undifferentiated quality misses.
Add content-addressed post-edit Go and Python sources whose single-file mutations swap the judged SIMILAR_TO and SEMANTICALLY_RELATED pairs. Retain pre/post hashes, changed paths, initial/incremental/fresh index results, pair witnesses, edge-score equality, and whole canonical-graph equality. Add incremental_semantic_freshness_eager as a one-setting profile for incremental_derived_refresh=eager.
Classify default stale_on_incremental separately from eager freshness: deferred publication conforms only when it is immediately canonical or returns the structured semantic_edges stale warning; eager publication requires fresh pair and canonical graph equality without that warning. Pair comparison remains O(observed pairs) in runtime and memory.
Verification: uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign (72 passed); Ruff checks; jq manifest parse; gofmt fixture checks; git diff --check. On optimized arm64 binary 36a1297b2571729517b96467965f476f15cd1d60e545454bc0c296f3103b9e17, default similarity/semantic edits conformed with explicit stale warnings at 40/51 ms; eager edits produced TP=1/TN=2/FP=0/FN=0 and canonical fresh equality at 72/71 ms. These are retained single pilot observations, not effect-size claims.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previously, capability-quality campaigns covered only rank and dependency retrieval, so SIMILAR_TO and SEMANTICALLY_RELATED costs had no paired precision/recall evidence. FAST remained the implicit matrix mode even for relationships that require MODERATE or FULL indexing.
Add benchmarks/semantic-pairs-v1/manifest.json with content-addressed Go structural-clone and Python control-flow-variant tasks, explicit lexical negatives, and source hashes. Extend benchmark-incremental-speed.py with unordered pair identities, TP/FP/FN/TN witnesses, precision, recall, F1, false-positive rate, unjudged-background retention, numeric query-score normalization, and response-quality gates. Propagate index_mode through compact campaign cells and document durable task/report semantics.
The bounded scorer is O(judgments + observed pairs) in runtime and memory. Natural repository edges outside the explicit judgment set remain unjudged instead of becoming false positives.
Verification: uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign (68 passed); ruff check on changed Python; jq manifest parse; gofmt fixture check; git diff --check. Optimized arm64 binary 36a1297b2571729517b96467965f476f15cd1d60e545454bc0c296f3103b9e17 produced TP=1/TN=2/FP=0/FN=0 for both enabled fixtures; each individually disabled profile removed its judged positive edge.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add cbm_config_apply_preset() in src/cli/cli.c with six named capability sets committed through one SQLite BEGIN IMMEDIATE transaction. Reuse the existing CBM_CONFIG_* keys from cli.h, depindex.h, pagerank.h, and pipeline.h instead of defining parallel settings.
Expose 'config preset list' and 'config preset apply <name>'. Return a nonzero status and identify any environment override whose effective value differs from the stored preset, so comparisons cannot silently run with a mixed configuration.
Document the quality and ablation presets in docs/CONFIGURATION.md. Cover the exact streamlined-quality and minimal-indexing values plus unknown-name rejection in tests/test_cli.c.
Verification: macOS Apple Silicon ASan/UBSan CLI suite 147/147 passed; optimized -O2 binary listed and applied classic-quality in an isolated cache; CBM_TOOL_MODE=streamlined correctly caused classic-quality to exit 1 with the effective-value warning; git diff --cached --check passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Five-repetition reports showed stable medians alongside occasional system outliers, but the tables exposed only p50/p95 values. That made it difficult to distinguish a typical configuration cost from observed host variability and could invite unsupported confidence claims.
Add query, incremental, and full-index observation counts plus min-max ranges to summarize-benchmark-results.py. The generated section explicitly states that grouped sequential campaigns avoid concurrent contention but do not support paired or randomized effect-size confidence intervals; medians and ratios remain descriptive.
tests/test_summarize_benchmark_results.py verifies exact counts and [min, max] rendering without confidence language. Verified with uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed tests.test_summarize_benchmark_results (93 passed). The ignored 240-input composition report and SHA-256 sidecar were regenerated but not staged.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Separate incremental, rank, and dependency campaigns could not produce categorical overall-quality or Pareto rows because each report lacked the other axes. Hand-merging raw files also risked mixing different config profiles and bypassing completion hashes.
Add --composition-spec to scripts/summarize-benchmark-results.py. Named campaign entries re-expand archived matrix specs, resolve exact cell labels, require every selected completion and binary/result SHA-256 to validate, and materialize derived inputs without changing raw results. Generated Markdown records the composition-spec hash and writes a per-input SHA-256 sidecar.
Preserve execution failures as correctness rejections, but classify a missed rank/dependency target as BELOW QUALITY TARGET when that exact report explicitly disables the corresponding capability. tests/test_summarize_benchmark_results.py covers campaign grouping and composed ablation wording.
Verified with uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed tests.test_summarize_benchmark_results (92 passed). The ignored composition validated 240 retained inputs and identified latest.default.mcp as the only fully passing measured Pareto-front row; generated reports and manifests were not staged.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Rank and dependency fixtures previously required hand-authored campaign cells, which made candidate, profile, transport, and repetition coverage easy to drift and omitted the concise matrix audit trail used by incremental benchmarks.
Add top-level capability_quality expansion in scripts/run-benchmark-campaign.py. Capability cells omit frontier/exact-cap axes, invoke benchmark-incremental-speed.py with --capability-quality and --include-logs, record the fixture in identity parameters, and retain named config profiles for enabled/disabled ablations. Existing incremental expansion remains the default when the field is absent.
tests/test_benchmark_campaign.py verifies a two-transport, two-repetition rank grid has four unique cells, no frontier parameters, the expected command, and accepted exit policy. Verified with uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed tests.test_summarize_benchmark_results (90 passed).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Compact matrix expansion in scripts/run-benchmark-campaign.py hardcoded accepted_exit_codes=[0]. benchmark-incremental-speed.py intentionally exits 1 after writing a valid result when canonical graph or quality gates reject a candidate, so campaigns mislabeled measured upstream failures as missing infrastructure.
Accept a validated top-level matrix-spec exit-code list, copy it into every expanded cell identity, and keep [0] as the default. Result parsing, binary SHA-256 verification, and the structured error check still reject crashes, missing artifacts, hash mismatches, and harness-error documents.
tests/test_benchmark_campaign.py verifies [0,1] propagation. Verified with uv run python -m unittest tests.test_benchmark_campaign tests.test_benchmark_incremental_speed tests.test_summarize_benchmark_results (89 passed). A retained two-cell upstream probe now reports REJECT: graph correctness with canonical mismatch witnesses instead of missing cells.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Compact matrix specs previously required a positive exact cap and always injected incremental_exact_max_affected_paths. A default-versus-default historical comparison therefore had to overwrite candidate policy with one shared cap.
Allow null in scenarios[].exact_caps to omit the config override, omit the capability identity key, retain exact_cap=null in parameters, and label the cell capdefault. Positive integers keep the existing explicit cap-sweep behavior. Document both forms in docs/BENCHMARK_CAMPAIGN.md.
Tests: uv run python -m unittest tests.test_benchmark_campaign -v (16 passed); uv run ruff check scripts/run-benchmark-campaign.py tests/test_benchmark_campaign.py (passed); git diff --check (passed).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previous behavior rendered cleanup.requested=false as Cleanup 0/1, making intentionally retained audit evidence look like a failed lifecycle operation. The campaign guide also directed crash-sensitive plans and results to /tmp even though validate_campaign_root() rejects the operating-system temporary tree by default.
Add evidence_lifecycle() in scripts/summarize-benchmark-results.py to label requested disposal, deliberate retention, missing metadata, and requested cleanup failure separately. Rename the generated column to Evidence lifecycle. Point docs/BENCHMARK_CAMPAIGN.md at the ignored durable .worktrees/benchmark-campaign path. Replace the secondary nDCG reference with Järvelin and Kekäläinen's primary DOI 10.1145/582415.582418 and identify exact pairwise Pareto nondominance with Deb et al. DOI 10.1109/4235.996017.
Tests: uv run python -m unittest tests.test_summarize_benchmark_results -v (26 passed); uv run ruff check scripts/summarize-benchmark-results.py tests/test_summarize_benchmark_results.py (passed); git diff --check (passed).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previous behavior: install detection and receipts omitted Claude Desktop even when its platform config directory existed, so claude_desktop_config.json required manual registration.
Add cbm_claude_desktop_config_path() at src/cli/cli.c:1267 and route the detected client through the existing mcpServers editor writer at lines 4504-4509. Route uninstall through cbm_remove_editor_mcp() at lines 5232-5237 so only the codebase-memory-mcp entry is removed.
Add plan/install/uninstall ownership coverage at tests/test_cli.c:2157 and platform-directory detection coverage at line 2479. Document the target in README.md, docs/CONFIGURATION.md, and the CLI help output.
Verification: CBM_ONLY_SUITE=cli build/c/test-runner (ASAN+UBSAN, native Apple Silicon macOS): 146 passed. CBM_ONLY_SUITE=cli build/c/test-runner-nosan: 146 passed. Native Windows execution was not performed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Detect the documented Qwen Code, ForgeCode, and Windsurf config roots and register codebase-memory-mcp using each client's native mcpServers JSON path. Add owned QWEN.md and AGENTS.md instruction blocks where the clients load global guidance.
Use the same code paths for install plans and uninstall. Lifecycle tests preserve foreign MCP servers and user-authored instructions while removing only codebase-memory-mcp entries; plan tests verify no files are written.
Document the complete supported-client matrix and correct the Claude hook description. The focused ASan/UBSan CLI suite passes 144 tests.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add --config-profile choices in scripts/benchmark-incremental-speed.py and record the selected profile beside its expanded config_overrides map.
Define minimal_indexing as auto_index_deps=false plus rank_enabled, similarity_enabled, semantic_edges_enabled, githistory_enabled, and httplinks_enabled set to false. Keep repeated --config values higher priority for controlled ablations.
Document the distinction between optional_graph_disabled and minimal_indexing in docs/BENCHMARK_CAMPAIGN.md. Validate with 29 Python unit tests, Ruff, and scripts/check-source-safety.sh.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Set CBM_PROFILE=1 in build_env() for every candidate so index_supervisor.c preserves successful worker logs long enough for build_index_result() to stream mem.phase, pipeline.done, and incremental.done markers. Document that the profiling configuration is consistent across revisions and remains part of the auditable benchmark environment.
Validated with 27 pytest cases across the benchmark, summarizer, and campaign suites; Ruff and git diff --check pass.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Run each cell from scripts/run-benchmark-campaign.py in an isolated process group. On timeout or SIGINT, signal the group, allow 30 seconds for harness cleanup, force-stop remaining descendants, write attempt.json, and release running.lock. This addresses the detached cbm-self-dogfood worktree left by an interrupted pilot.
Read at most 512 mem.phase, pipeline.done, and incremental.done lines from the response logfile in build_index_result() using a streaming scan. Replace the route_handler comment literal with an executable cbm_http_path_match(path, "/api/pan4-oracle") mutation and target the route oracle with name_pattern=pan4-oracle.
Validated with 26 pytest cases, including descendant timeout cleanup, bounded worker-log metrics, and executable route mutation. Ruff and git diff --check pass.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Route run_cli_tool_call() and run_mcp_tool_call() through a default-format request for elapsed_ms, response_bytes, and response_token_estimate, then issue format=json only for quality scoring. Record quality_probe_elapsed_ms and quality_response_bytes so the second request cannot inflate the user-facing latency or replace TOON payload sizing.
This fixes the pilot JSONDecodeError: Expecting value at line 1 column 1 when search_graph returned its default TOON content. Document the two-request instrumentation contract in docs/BENCHMARK_CAMPAIGN.md.
Validated with 23 focused pytest cases across tests/test_benchmark_incremental_speed.py, tests/test_summarize_benchmark_results.py, and tests/test_benchmark_campaign.py; Ruff and git diff --check pass.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add scripts/run-benchmark-campaign.py with SHA-256 cell identities over revisions, binaries, release flags, capabilities, transports, scenarios, repetitions, harnesses, commands, environments, and timeouts.
Retain timestamped command/stdout/stderr/result/attempt artifacts; validate binary and result hashes before atomically writing complete.json. Reject live locks, record stale-lock recovery, enforce disk headroom, archive exact plan bytes and environment snapshots, and classify missing, corrupt, duplicate-attempt, and unplanned run directories in immutable manifests.
Regenerate reports atomically from validated completion records only. docs/BENCHMARK_CAMPAIGN.md documents default, rank_enabled=false, and all-optional-capabilities-off cells plus release-build metadata requirements.
Validated by 19 unittest cases and ruff check across the campaign, measurement, and report scripts; git diff --check passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Merge upstream/main at 2469ecc3 into integrate-upstream-split-search-tools while retaining branch-specific incremental indexing, MCP API consolidation, and graph extraction behavior.
The staged tree adds upstream's Mojo and Perl grammar support, issue reproduction suites, Graph UI changes, Windows smoke coverage, TOON graph output, subprocess supervision, and Laravel facade route handling. Conflict resolutions in src/pipeline/httplink.c and pass_httplinks.c retain dynamic growth and IMPORTS edge identity; scripts/smoke-test.sh and scripts/soak-test.sh retain hermetic cache isolation.
Validation against the merge plus the follow-up working-tree reconciliation: ASan/UBSan MCP suite 216/216 passed. Follow-up semantic and concurrency fixes remain intentionally unstaged for the next focused commit.
The per-language chapters were LLM-drafted to establish format and
dimension mapping — their symbol targets are not yet ground-truth-derived
and must be regenerated at execution time per §3.1/§12. Make that status
explicit at the top of Appendix A so the drafted symbol names are not
read as validated.
Supersedes #420 (same diff; recreated for DCO author/sign-off).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Add CBM_ALLOWED_ROOT to the environment-variable tables in README.md and
docs/CONFIGURATION.md: when set, index_repository refuses a repo_path that
resolves outside it; unset imposes no restriction.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Documents the current behavior as implemented in src/discover/discover.c
and src/discover/gitignore.c: where .cbmignore is read from, the glob
features the parser supports, the layered precedence against built-in
skip lists / .gitignore hierarchy / git global excludes, and what
negation can and cannot override today. Planned negation unification
(un-skipping built-in dirs, non-negatable safety core, shared predicate
for auxiliary walkers) is listed in an explicitly not-yet-implemented
subsection. Linked from the README "Ignoring Files" section.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Andrew Hundt