Add exact tools/list regression checks for the fork's streamlined and classic MCP surfaces. The tests parse the actual JSON tool array instead of relying on substring matches, so descriptions cannot mask a missing or extra tool entry.
Keep upstream compatibility explicit by verifying the trace_path dispatch alias remains callable while the fork continues to list trace_call_path in its classic surface.
Validation: git diff --check; make -f Makefile.cbm build/c/test-runner; CBM_ONLY_SUITE=tool_consolidation build/c/test-runner (90 passed).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Make index writes and artifact refreshes use verified temp files plus atomic replacement, preserving Windows replace semantics and avoiding fixed temp-path collisions.
Thread the remaining pipeline confidence knobs through config while keeping zero-valued defaults backward compatible with existing upstream behavior.
Tighten route-node gates and architecture route output so infra URL nodes stay queryable in the graph but do not pollute get_architecture route summaries; preserve code-created absolute URL routes for all language extractors.
Harden MCP and CLI path handling around cache paths, grep scratch files, update downloads, checksum validation, stale checks, and project DB discovery without writing protocol noise to stdout.
Add regression coverage for corrupt-DB recovery, concurrent atomic writes, tmp-prefixed project DBs, duplicate search result fields, route output filtering, config registry entries, and focused suite selectors.
Validated with: git diff --check; make -f Makefile.cbm build/c/test-runner; focused suites security, artifact, graph_buffer, sqlite_writer, store_search, store_bulk, store_pragmas, store_checkpoint, watcher, infrascan, pipeline, store_nodes, mcp, cli, parallel, tool_consolidation, incremental, and store_arch; make -f Makefile.cbm cbm; isolated dogfood index/query run.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
README said "158 languages" (7 places incl. the badge) and server.json said
"159" — but the repo actually ships 156 vendored tree-sitter grammars
(internal/cbm/grammar_*.c, verified). README's own wording ties the count to
"vendored tree-sitter grammars", so 156 is the correct value for this tree.
All 3 numbers (156 actual / 158 README / 159 server.json) now agree on 156.
(Caveat: upstream main may have added grammars since this branch's last sync
at aa2796a; if so, a future upstream sync raises the count + these update
again. For this fork's tree, 156 is accurate.)
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Contract test confirming the graph-analytics chain (PageRank + architecture)
runs on a multi-"language" graph (nodes from .py/.ts/.go files with
cross-language call edges). Analytics are language-agnostic (graph-level), so
this complements the language-agnostic pagerank/arch unit tests + the
per-language LSP tests: it asserts the CHAIN runs across the polyglot case
(PageRank ranks all 6 nodes; architecture returns OK). language_count is not
asserted — that depends on language metadata populated at real-index time, not
the analytics. store_arch 53 pass.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
The codebase://architecture resource (key functions = where to start tracing)
is an MCP resource — application-controlled/pull-only by spec, only fetched on
explicit @-mention. So the model never got it automatically. The _context
header already PUSHED schema/status/pagerank (the reliable channel into the
model); now it also pushes a bounded key_functions summary (top by PageRank,
honoring key_functions_exclude), so the model knows where to start without
pulling the resource or enabling the hidden get_architecture tool.
Reuses build_key_functions_sql (the quote-escaped path). Bounded by
CBM_CONTEXT_KEY_FUNCTIONS_LIMIT (10, smaller than get_architecture's 25) to
keep first-response token cost modest. Gated on context_injection (default
true). Verified end-to-end: search_graph's first response now includes
_context.key_functions (10 entries) on the self-indexed repo. mcp 108 pass.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
The post-dump verify (B1 mitigation, 63de254) used the strict
cbm_store_check_integrity, which treats a non-absolute root_path (e.g. a
relative repo_path like ".") as bad_root_path → the verify DELETED a valid DB.
Indexing with a relative path therefore failed (status=error) — caught by
self-indexing the repo (cbm cli index_repository repo_path="."); the same
index with an absolute path succeeded cleanly (14723 nodes).
Switch the verify to cbm_store_check_integrity_full with the path_only
out-param: a path-only defect (cosmetic root_path) is RETAINED, consistent
with #557 resolve_store — only genuine structural corruption (>5 project
rows) is deleted. Verified: relative-path index now returns status=indexed;
absolute-path index unchanged.
test_graph_buffer gbuf_dump_relative_root_path_retained locks this in.
graph_buffer 50 pass.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
The architecture file-tree builder concatenated path components into a stack
char dir[CBM_SZ_512] with strcat, bounded only by ST_MAX_PATH_DEPTH (component
COUNT) — not total length. A file_path with >512 chars across components would
overflow the buffer. Replace the strcat loop with length-checked appends that
stop (truncating that dir key) when the next segment wouldn't fit. Normal
paths are unchanged; store_arch 52 pass (exercises the file tree).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
build_key_functions_sql baked each exclude glob (from the get_architecture
`exclude` MCP tool arg AND the key_functions_exclude config) into the SQL
text as `AND n.file_path NOT LIKE '<glob_to_like output>'`. cbm_glob_to_like
copies single quotes verbatim, so an exclude value containing ' broke out of
the string literal — e.g. exclude=["' OR '1'='1"] became
`NOT LIKE '' OR '1'='1'`, an MCP-input-reachable SQL injection.
The search path is NOT affected (it binds LIKE patterns as ?N parameters via
sqlite3_bind_text, which is safe), so the fix is local to key_functions: add
sql_escape_quotes() (doubles ' -> '', the standard SQL string-literal escape)
and apply it to the glob→like output before interpolation. The bound-parameter
callers of cbm_glob_to_like are intentionally untouched (escaping there would
double-escape).
Found by the quality/security sweep agent; verified independently (reachability
via the `exclude` tool arg at mcp.c:3372; cbm_glob_to_like copies ' verbatim at
store.c:2168). The sweep's other 2 findings (cascade_delete_edges NULL-deref,
parse_fp_from_props off-by-one) were verified as FALSE POSITIVES
(remove_edge_from_ptr_array NULL-checks; the hex check uses HEX_LEN not HEX_BUF).
Green: store_arch 52 / mcp 108.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
- store_prefix_ranks_project_above_dep_all_sort_modes: proves dep-last
ranking holds for EVERY sort_by (relevance/name/degree/calls/linkrank),
not just the default. Dep inserted first (lower id) so a missing dep-last
in any mode would let the dep win the id tiebreak — a real per-mode check.
- pagerank_invalid_inputs_clamp_cleanly: negative/>1 damping, negative/zero
epsilon, negative/zero max_iter all clamp to defaults and produce finite
ranks summing to ~1 (guards the #21 config knobs against bad values, and
complements the #44 NaN fix).
Green: tool_consolidation 88 / pagerank 45.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
pass_similarity hardcoded the SIMILAR-edge Jaccard cutoff to
CBM_MINHASH_JACCARD_THRESHOLD (0.95). Now tunable via config.
- pipeline: add similarity_threshold to cbm_pipeline_t + a setter
cbm_pipeline_set_similarity_threshold(); copy it into the pipeline ctx
(alongside mode). <=0 (unset) keeps the built-in default.
- pass_similarity: the sim_query_ctx_t worker struct carries the threshold
(set from ctx); the per-pair check uses sc->threshold when >0, else the
default. No behavior change unless tuned.
- mcp: CBM_CONFIG_SIMILARITY_THRESHOLD ("similarity_threshold") read via
cbm_config_get_double in handle_index_repository (default 0.0 = built-in)
and applied before pipeline_run.
Together with the Leiden resolution knob (prev commit), all graph-analytics
params are now config-tunable: pagerank damping/epsilon/max_iter + edge
weights, Leiden resolution, similarity threshold. mcp 108 / graph_buffer 49 /
full suite clean through the bulk (only known B1 incremental failures).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
#3 — incremental purge was O(C·(N+E)): cbm_gbuf_delete_by_file was called
once per changed/deleted file, each a full nodes+edges scan. Add
cbm_gbuf_delete_by_paths (one pass over nodes matching a path membership
set + one edge cascade = O(N+E) regardless of path count); refactor
cbm_gbuf_delete_by_file to delegate to it (DRY — single implementation).
pipeline_incremental.c now purges all changed+deleted paths in one call
(OOM fallback keeps the old per-file path). graph_buffer 49 + incremental
purge tests pass; only the known B1 full-reindex failures remain.
#8 — cbm_dep_link_cross_edges was N+1: one cbm_store_search per import
(<=500) to find a matching dep Module. Now ONE bulk fetch of all dep Module
nodes + an in-memory name->id hash (first-wins, preserving limit=1
semantics) -> O(1) per import. depindex 32 pass.
#5 verified NOT a bug (avoided a false fix): node_degree IS populated by
cbm_pagerank_compute (pagerank.c:422 INSERT OR REPLACE INTO node_degree);
the correlated-subquery fallback only fires pre-PageRank.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
cbm_store_search had `pr_col = has_degree_filter ? "pr_rank" : "pr_rank"`
— both branches identical because the pagerank JOIN alias is always
pr_rank (unlike name_col/proj_col which legitimately differ under the
degree-filter subquery wrap). cppcheck flagged the duplicate expression.
Simplify to a direct assignment. No behavior change; tool_consolidation 87 passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
- README: "14 MCP tools" -> 15 (classic), in all 4 places; clarify that a
streamlined subset is the default and CBM_TOOL_MODE=classic exposes all 15.
- mcp.c query_graph description: note that dependency sub-project symbols
(proj.dep.*) are tagged source:dependency, and show the ORDER BY pattern to
rank project symbols above them (Cypher is declarative — user controls
ORDER BY, so this is documentation, not auto-reordering).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
cbm_pagerank_compute's clamp used `damping < 0 || damping > 1` and
`epsilon <= 0`, both of which IEEE-754 NaN bypasses (every NaN
comparison is false). Since damping/epsilon became config-tunable and
cbm_config_get_double uses strtod (which parses "nan"), a
`config set pagerank_damping nan` would let NaN propagate through the
power iteration: every rank becomes NaN, the convergence check
`delta < epsilon` is always false (runs full max_iter), and the pagerank
table is silently corrupted, poisoning the search relevance sort.
Switch to the inverted-range form `!(damping >= 0 && damping <= 1)` and
`!(epsilon > 0)`, which reject NaN because the inner comparison is
false. Found by the silent-failure review agent; verified independently
(strtod parses "nan"; NaN bypassed the old clamp).
test_pagerank.c pagerank_nan_damping_is_clamped: a NaN damping must
yield finite ranks summing to ~1. Green: pagerank 44 passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
cbm_pagerank_compute_with_config already read max_iter and the 12
edge-weight keys from config, but damping (0.85) and epsilon (1e-6) were
passed as hard-coded #defines -- the two core convergence params were
not tunable.
- pagerank.h: CBM_CONFIG_PAGERANK_DAMPING / CBM_CONFIG_PAGERANK_EPSILON.
- pagerank.c: read both via cbm_config_get_double in
cbm_pagerank_compute_with_config (defaults = existing #defines, so no
behavior change unless a user tunes them).
- test_pagerank.c: pagerank_damping_epsilon_config_tunable asserts that
tuning damping (0.5 vs 0.99) materially changes the hub rank value and
ranks still sum to ~1 under a custom config.
Green: pagerank 43 passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
The dep-last tiebreak in cbm_store_search only fired for glob searches
(params->project_pattern), so the common prefix-match path
(params->project, which includes proj.dep.*) let a dependency symbol
outrank the project's own symbol of the same name -- the "Path" concern
(a stdlib Path fronting the user's own Path).
- store.c: hoist dep-last into one clause applied consistently across
ALL sort modes (relevance/degree/calls/linkrank/name) whenever a
search is scoped to a project; gated on the new param.
- store.h: cbm_search_params_t::disable_dep_ranking (default false =
deps rank last; inverted sense so {0}-init preserves the safe default).
- mcp.c: search_disable_dep_ranking config key (bool, default false)
wired through srv->config and documented in the search_graph schema,
following the existing CBM_CONFIG_* pattern.
- test_tool_consolidation.c: store_prefix_ranks_project_above_dep
(default ranks project above dep) and store_prefix_disable_dep_ranking_lets_dep_win
(pure relevance when disabled).
Green: tool_consolidation 87 passed. Full suite unchanged except the 2
known B1 incremental failures.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Two regression probes that isolate the non-deterministic .db corruption
(B1) to the extraction-population hop, each with a decisive negative:
- test_sqlite_writer.c sw_scale_root_path_integrity: variable-length
properties_json (20-1500B, stressing every page boundary) through
cbm_write_db. Clean 10/10 -> rules out the variable-record
page-boundary hypothesis in the writer.
- test_graph_buffer.c gbuf_dump_pipeline_path_integrity: drives
cbm_gbuf_dump_to_sqlite with a populated + merged gbuf (variable heap
properties_json + cbm_gbuf_merge ID remap, 15k nodes / 50k edges).
Clean 10/10 -> rules out the gbuf->dump handoff (build_dump_nodes /
temp_to_final).
Both confirm the writer and the gbuf->dump handoff are correct on
realistic synthetic data; B1 must originate in real tree-sitter
extraction populating the gbuf (needs real source data to reproduce).
test_main.c: add CBM_ONLY_SUITE fast-filters for graph_buffer + pagerank.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
The existing test-leak uses `leaks --atExit`, which finds heap LEAKS — the
wrong tool for non-deterministic corruption (uninit reads, use-after-free,
overruns) that ASan/TSan miss (e.g. the B1 custom-writer bug). Add two
macOS-native targets that run the nosan binary (ASan replaces malloc, which
defeats these libmalloc knobs):
- test-memory: MallocScribble=1 + MallocPreScribble=1 — freed memory -> 0x55
(catches UAF), freshly-allocated -> 0xAA (catches uninitialized reads; the
macOS MSan equivalent, makes uninit deterministic).
- test-gmalloc: DYLD_INSERT_LIBRARIES=/usr/lib/libgmalloc.dylib (Guard Malloc)
— guard page per allocation, crashes at the exact overrun/UAF with a stack
trace. The decisive memory tool.
Both tee to build/c/mem-report.txt; honor CBM_ONLY_SUITE for targeted runs.
Linux falls back to a pointer to test-leak / -fsanitize=memory. Documented in
CLAUDE.md (Memory-Corruption Debugging section). .PHONY updated.
Verified: `make -n test-memory`/`test-gmalloc` expand correctly; default
`test` target unchanged.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
cbm_gbuf_dump_to_sqlite intermittently emits a structurally-corrupt .db
(B1): on the ~16k-node fastapi repo a full index non-deterministically
persists 0/partial node counts and a garbage project root_path, leaving an
unreadable .db that queries then fail on (PRAGMA integrity_check won't even
prepare). The corruption is ASan/TSan/clang--analyze-invisible in the dump
path; root cause still open.
Add a post-write check in cbm_gbuf_dump_to_sqlite: after cbm_writer_finalize,
open the freshly-written .db and run cbm_store_check_integrity (O(1)
projects-table check, not a full integrity_check). If it fails, unlink the
.db and return GB_ERR so the caller re-indexes instead of serving garbage.
Confirmed firing in the incremental suite (logs dump.verify_corrupt, deletes
the .db).
Makes the system fail-safe against the worst (structurally-unreadable) B1
symptom: no silent corrupt .db is left for queries to read. Does not by
itself fix the underlying non-deterministic write corruption — the 2
pre-existing incremental-accuracy failures (incr_db_deleted_recovery,
incr_accuracy_vs_full) remain; that needs the root-cause fix. Complements
the #557 retain-bad-root_path mitigation in 22e438f.
Verified: store_nodes 56/56, sqlite_writer 7/7, mcp 108/108 (no regression).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Adds sw_scale_root_path_integrity: writes a ~fastapi-scale DB in ISOLATION
(20k nodes / 200k unique edges) via cbm_write_db, then asserts root_path
round-trips exactly, PRAGMA integrity_check == "ok", and node/edge counts.
Purpose: isolate whether the B1 custom-writer corruption reproduces without
the pipeline. Result: PASSES 3/3 — the writer is sound at scale in isolation
(sw_minimal_data covers the tiny case). This rules out the writer's
serialization as the B1 cause and points at the pipeline/parallel handoff
(a data race — to be confirmed with TSan). Also a permanent regression guard
for the custom B-tree writer at scale.
Edges generated with unique (source,target) pairs to respect the table's
UNIQUE(source_id, target_id, type) constraint. Adds sqlite_writer to the
CBM_ONLY_SUITE filter.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
When CBM_ONLY_SUITE is set (e.g. "incremental", "mcp", "store_nodes",
"tool_consolidation"), run only that suite then summarize and exit;
otherwise behavior is unchanged. Zero cost when unset. Speeds the
edit/build/test loop for focused work without spinning up the heavy
LSP/stdlib suites.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
resolve_store (mcp.c) auto-cleaned (deleted .db + WAL + SHM) whenever
cbm_store_check_integrity flagged the projects table. A single malformed
project root_path (e.g. a numeric value) is a cosmetic projects-row
defect — node/edge data stays intact and queries key off project name,
not root_path — yet it triggered a wholesale delete of the freshly-indexed
DB, losing all indexed data (upstream #557).
Add cbm_store_check_integrity_full(store, &path_only_failure) (store.h,
store.c) which classifies a bad-root_path-only defect (rows fine) apart
from genuine >5-row corruption. resolve_store now RETAINS path-only-defect
DBs (logs store.integrity_retain) and only auto-deletes genuine
corruption. The watcher registration is guarded against a bogus
(numeric/empty) root_path from a retained DB. cbm_store_check_integrity()
is kept as a thin wrapper so artifact.c, pipeline.c, and existing tests
are unchanged.
Mitigates #557 data loss and stabilizes incremental node-count reads (the
count no longer sporadically drops to 0 after a query). Does NOT fully
close the 2 pre-existing incremental-accuracy failures
(incr_db_deleted_recovery, incr_accuracy_vs_full), which stem from a
separate full-reindex-after-deletion node-count discrepancy — tracked
separately.
Verified: store_nodes 56/56 incl. new store_integrity_full_path_only_classification;
mcp suite green; incremental suite shows only the 2 pre-existing failures
(no regression; diff_pct=96 identical pre/post).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
The full-index hash-persist path (pipeline.c:1127) upserted one row per file under autocommit -- a 10k-file reindex did 10k separate fsyncs under WAL. Wrap the loop in cbm_store_begin/cbm_store_commit (falling back to per-row autocommit if BEGIN fails), matching the incremental path (pipeline_incremental.c:458, persist_hashes). Fork-origin perf finding (#6 from the post-merge latency/complexity audit; git blame = Andrew Hundt). No behavior change: same upserts, single commit.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
persist_hashes (pipeline_incremental.c:452) upserted one file-hash row per file in autocommit -- a 10k-file reindex issued 10k separate COMMITs (each a WAL fsync). Wrap both upsert loops (current discovery + mode-skipped) in cbm_store_begin/commit so N files -> 1 COMMIT. Falls back to per-row autocommit if BEGIN fails (a caller already holds a transaction): 'batched' is gated on BEGIN's return and COMMIT runs only when BEGIN succeeded, so the existing partial-failure policy is unchanged. Fork-origin perf finding #6 (git blame: Andrew Hundt). Compiles clean (make cbm). Other perf findings are upstream-origin (cypher sort/DISTINCT, query_graph JSON, include_connected, properties re-parse -- DeusData #593/#601/#238) or fork-origin but higher-risk (deferred).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
The §4b split (39539ef) deleted the search_code_graph mega-tool but left inconsistent references, found by a systematic tool-name + AI-info-string audit: (1) _hidden_tools pseudo-tool JSON (mcp.c:6365) still listed the 3 now-default tools (14 entries) contradicting its own '11 additional' description -- trimmed to the 11 genuinely-hidden tools (15 TOOLS[] minus 4 default-surface-in-TOOLS[]; get_code is a streamlined-only alias); (2) handle_get_code_snippet error hints (4752, 4923) referenced the deleted tool -- search_graph; (3) search_graph mode-error hints (2663, 2670) pointed at the dead search_in='source' param -- the search_code tool; (4) comments (1566, 4725) updated. String/comment-only; no handler logic changed. The two critical hints previously sent clients/AIs to a non-existent tool from default-surface error paths.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Replace the consolidated search_code_graph mega-tool — which merged graph
search + source grep + cypher behind two nearly disjoint parameter
families — with the three existing focused tools as the DEFAULT surface:
search_graph (graph/structural), search_code (source grep), query_graph
(cypher). These are drawn from TOOLS[] (single schema source, DRY);
trace_call_path + get_code remain on the default surface. The
search_code_graph schema and its thin dispatch router are deleted; the
underlying handlers (handle_search_graph/search_code/query_graph) stay.
The _hidden_tools progressive-disclosure hint now lists 11 hidden tools
(was 14). Repointed three user-facing hint strings (handle_trace_call_path
missing-name and function-not-found; two unknown-tool path hints) from the
deleted tool to search_graph.
Tests updated to the new 5-tool surface:
- tests/test_mcp.c: mcp_tools_list + server_handle_tools_list assert the
5 default tools present and search_code_graph absent.
- tests/test_tool_consolidation.c: renamed streamlined_mode_shows_3_tools
-> ..._5_default_tools; repurposed the two search_code_graph dispatch
tests into search_graph_dispatch / query_graph_dispatch (query_graph
handler accepts both cypher= and query=); revised
schema_compact_documented_as_graph_only (the graph-vs-source compact
collision it warned about no longer exists post-split).
- tests/test_input_validation.c: remapped 16 dispatch calls by mode --
cypher -> query_graph, source-grep -> search_code, graph/structural ->
search_graph.
- tests/test_depindex.c: tool_index_dependencies_listed asserts
search_graph (was search_code_graph).
- src/cli/cli.c: six config-description strings (search_limit, tool_mode,
context_injection, compact, default_sort_by, default_include_dependencies).
Back-compat: blast radius is in-repo only (0 references in Go internal/cmd,
docs, or the CGO wrapper). External callers of search_code_graph must
switch to search_graph / search_code / query_graph, which are strictly
more capable. query_graph keeps its query= backward-compat alias.
Context -- of the three section-4 fixes in the merge plan, only section 4b
was a real change: section 4c (function<->class edges) was already
implemented (commit b950bf2 added the missing MEMBER_OF reverse-edge
coverage), and section 4a (passive resources) was already mitigated by
inject_context_once plus the get_graph_schema/get_architecture/index_status
tools -- both verified against code, not assumed.
Gate: 5981 passed / 2 failed (the 2 are the deferred B1 store-layer
incremental failures, unrelated to section 4b).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
contract_edge_defines_method asserted only the forward DEFINES_METHOD
(Class->Method) edge. The fork's reverse MEMBER_OF (Method->Class) edge —
emitted in pass_definitions.c:322 / pass_parallel.c / pass_normalize.c and
consumed by pagerank.c (member_rank_factor) — had ZERO test coverage
anywhere in the suite. Add the MEMBER_OF assertion so the function<->class
tie is locked in in both directions.
§4c investigation finding (evidence-based, refutes the plan's assumption
that a code fix was needed): the function<->class tie was ALREADY
implemented, not a merge regression. The Method-only gate
(pass_definitions.c:318) is identical in fork (3018b22:270), upstream
(34efbc0:315), and HEAD; DEFINES_METHOD is tested+passing across 9
languages (defines_method_go/rust/java/csharp/php/ruby/kotlin/ts/scala in
test_edge_types_probe.c); the documented "Go parent_class never derived"
bug is already fixed (extract_defs.c:2776-2783); and every class-scoped
callable is labeled Method with parent_class populated (push_method_def /
generic extractor / extract_rust_impl), so there is no class-scoped
Function gap to fix. Broadening the gate would change nothing. The only
real gap was the untested reverse edge, closed here.
Gate: 5981 passed / 2 failed (the 2 are the deferred B1 incremental
store-layer failures, unrelated to §4c).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
test-runner-tsan mirrors test-runner but links vendored objects compiled
with -fsanitize=thread and links the mimalloc object with MI_OVERRIDE=0
(no malloc override, so TSan intercepts the real malloc/free), satisfying
the unconditional mi_* calls in src/foundation/mem.c. TSan cannot combine
with ASan/UBSan and uses the system allocator. Replaces the placeholder
test-tsan stub ("TSan not yet wired for full extraction tests").
make -f Makefile.cbm test-tsan # slow; race reports go to stderr
Also add *.plist to .gitignore: `make test-analyze` (clang --analyze)
drops a .plist report per source file in CWD; these regenerable reports
polluted the tree (183 files after the analyzer run).
Verification: built + ran test-runner-tsan over suite_incremental; 0 data
races. The deferred B1 incremental races (HANDLES-edge nondeterminism
under the concurrent parallel pipeline) are timing-nondeterministic and
did not reproduce under TSan's scheduling, so the B1 fix stays a dedicated
follow-up (tracked in notes sec 8f). TSan targets the incremental +
parallel suites that exercise the racy paths.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
The merged cbm_mcp_tools_list (src/mcp/mcp.c:868) defaults to streamlined mode
when srv is NULL, so tests/test_mcp.c:mcp_tools_list only exercised the 3
consolidated tools. The classic-mode path (CBM_TOOL_MODE=classic, mcp.c:876)
that emits all 15 TOOLS[] split tools had no test — the MERGE-TODO flagged at
tests/test_mcp.c:172.
Add mcp_tools_list_classic_mode: setenv("CBM_TOOL_MODE","classic"), capture the
list, then unsetenv BEFORE any ASSERT so a failed assertion cannot leak the
classic setting into sibling tests (which expect the streamlined default). Assert
the classic split tools (index_repository, search_graph, query_graph) are present
and the streamlined-only consolidated search_code_graph + the _hidden_tools
progressive-disclosure hint are absent. Registered in suite_mcp.
Full suite: 5981 passed / 2 failed (the 2 are the deferred B1 incremental races,
unrelated to this test).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
tests/test_stack_overflow.c and tests/test_lang_contract.c each have a fork()
helper that runs cbm_extract_file on pathological input in a child process (to
verify no-crash), then bare-waitpids with waitpid(pid, &status, 0). Under
`leaks --atExit` (macOS) the forked child is SIGSTOP'd during heap inspection
and the bare waitpid never returns — the identical hang class already fixed in
b336466 (test_store_bulk.c, src/foundation/compat_fs.c) and 148b4d7
(test_dump_verify_io.c).
Replace the bare waitpid with the WUNTRACED+SIGCONT loop used elsewhere in the
suite (kill(child, SIGCONT) on WIFSTOPPED so leaks can finish and the child
proceeds to _exit), and add #include <signal.h> for kill() in both files. The
helpers keep their existing `return WIFSIGNALED(status)` contract.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
dump_verify_io_fork_crash_uncommitted_discarded used bare
waitpid(pid, &status, 0), which never returns when `leaks --atExit` (macOS)
SIGSTOPs the forked child during heap inspection — the identical hang b336466
fixed in test_store_bulk.c (bulk_crash_recovery) and src/foundation/compat_fs.c
(cbm_exec_no_shell). This test was missed by that earlier fix.
Replace the bare waitpid with the WUNTRACED+SIGCONT loop that mirrors
test_store_bulk.c: on WIFSTOPPED, kill(child, SIGCONT) so leaks can finish
inspecting and the child proceeds to _exit(). Add #include <signal.h> for kill().
Unblocks `make test-leak` past this test (the run previously hung here
indefinitely).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
The test-runner-nosan target (used by `make test-leak` on macOS for
`leaks --atExit`) failed to link with Undefined symbols: _LZ4_compressBound,
_LZ4_compress_HC, _LZ4_decompress_safe, _suite_zstd, _cbm_artifact_import,
_PRETRAINED_VECTOR_BLOB.
Root cause: OBJS_VENDORED_NOSAN in Makefile.cbm omitted the LZ4/ZSTD/nomic
objects that OBJS_VENDORED_TEST includes, and the test-runner-nosan target
prerequisites/link line missed $(ZSTD_SRCS). Because the LZ4/ZSTD test objects
are compiled with $(SANITIZE) (ASan) and ASan replaces malloc (incompatible with
the `leaks` tool), add sanitizer-free NOSAN variants ($(NOSAN_DIR)/lz4.o,
lz4hc.o, zstd.o) that mirror the existing NOSAN sqlite3/ts_runtime/lsp_all/
preprocessor objects, plus $(UNIXCODER_OBJ) and $(ZSTD_SRCS).
Result: build/c/test-runner-nosan now links; `make test-leak` builds and runs.
(A separate runtime hang at dump_verify_io_fork_crash_uncommitted_discarded
under `leaks --atExit` — a fork+crash+waitpid/leaks interaction — remains and is
tracked separately; it is unrelated to this link fix.)
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Deep --no-ff merge of upstream DeusData/main (597 commits, merge-base 988d975)
into fork api-consolidation (HEAD 3018b22), producing a superset that combines
upstream breadth with the fork's surviving depth.
Upstream additions integrated: semantic/embedding search (vendored/nomic),
minhash/SimHash, cross-repo + cross-file LSP + route + pkgmap passes, artifact
import/export, path_alias, git_context, hook_augment, Windows support
(compat_fs/compat_thread), the UI layer, and ~140 expanded tree-sitter grammars.
Fork depth retained: PageRank/LinkRank ranking (src/pagerank/; sort_by=
relevance/linkrank/degree/calls), dependency indexing (src/depindex/;
CBM_INDEX_MODE=DEP), token reduction, and HTTP-endpoint discovery
(src/pipeline/httplink.c, pass_httplinks.c).
Conflict resolution (27 regions across the 39-file intersection):
- src/mcp/mcp.c: kept the fork superset handler pattern (resolve_project_store +
expand_project_param + inject_context_once) as the single path; UNIONed the
tool dispatch table; emit status:"indexed", excluded[], cross_service trace.
- src/store/store.c: upstream base + fork pagerank/linkrank/node_degree tables
and sort_by plumbing; cbm_store_node_degree now excludes INHERITS
(WHERE type != 'INHERITS'); removed the duplicate cbm_store_get_db.
- src/pipeline/pipeline.c: took upstream run_extraction_phase; grafted the fork
passes (httplinks, normalize, flush_store dump branch); call
cbm_pipeline_set_committed_counts before cbm_gbuf_dump_to_sqlite.
- internal/cbm/lang_specs.c: 158 entries combining the fork section_node_types
with upstream ts_factory/embedded_imports; wired 13 orphaned type-arrays.
- src/depindex/depindex.h: renamed CBM_PATH_MAX -> CBM_DEP_PATH_MAX (it collided
with constants.h enum CBM_PATH_MAX=1024) + 11 call sites in depindex.c.
- Makefile.cbm: upstream grammar/nomic units + fork CGO targets; added
httplink.c + pass_httplinks.c to PIPELINE_SRCS; removed mongoose NOSAN
machinery (mongoose was deleted upstream).
- httplinks restored (the merge silently deleted them because the fork had left
them unchanged since the base): httplink.c/.h, pass_httplinks.c,
tests/test_httplink.c; registered RUN_SUITE(httplink) in tests/test_main.c.
- CBM_INDEX_MODE enum unified to FULL=0, MODERATE=1, FAST=2, DEP=3.
Completeness (dual-parent check): 0 upstream files dropped from the merge tree;
6 fork files dropped, all intentional (upstream-superseded): CHANGELOG.md,
vendored/mimalloc/src/{arena-abandon,segment-map,segment}.c,
vendored/mongoose/{mongoose.c,mongoose.h}. Fork features (pagerank, depindex,
token_reduction, httplink) verified present and registered in tests/test_main.c.
src/pipeline/pass_parallel.c has an identical function set to upstream (49=49,
none dropped); the large line-deltas vs upstream are within-function divergence
of the fork extractor, not dropped functions.
Gate (run manually; hooks skip on merge commits): build green
(make -f Makefile.cbm cbm, exit 0); full C test suite 5980 passed / 2 failed /
0 crashes. The fork's pre-existing ASan crash in build_project_list_error_srv is
fixed by upstream's mcp.c rewrite, so the merge is strictly better than the fork
baseline on the crash axis.
Known issue, tracked separately (NOT a merge defect): 2 nondeterministic
failures in tests/test_incremental.c — incr_db_deleted_recovery:772 (diff_pct)
and incr_accuracy_vs_full:803 (full_nodes vs incr_nodes). Cause: the fork's
extractor (internal/cbm/extract_*.c) and parallel-resolve Route-node creation
are not thread-safe under the concurrent use of upstream's parallel pipeline
(run_parallel_pipeline), producing nondeterministic route/infra HANDLES edges
(~234k edge swing between runs). Sequential extraction (CBM_WORKERS=1) is
deterministic and correct. Root cause, bisect, and repro are documented in
notes/2026-06-24-fork-vs-upstream-divergence-assessment.md (section 8f); the fix
is deferred to a dedicated thread-safety audit (the make test-tsan target is
currently a stub and needs wiring).
The MSYS2 CLANG64 toolchain bakes its package-tracker URL
(https://github.com/msys2/MINGW-packages) into the static Windows .exe via
libc++/compiler-rt. The Layer 2 binary string audit flagged it as an
unauthorized URL, failing smoke-windows (both the standard and ui variants)
while Linux smoke stayed green. Add it to ALLOWED_URLS alongside the other
compiler/linker toolchain URLs (gcc.gnu.org, sourceware.org, launchpad).
Add tests/test_security_strings_allowlist.sh as a reproduce-first regression
guard: it reproduces the BLOCKED failure with a binary fixture and includes a
negative control proving a genuinely unauthorized URL is still rejected. Wired
into scripts/test.sh as Step 6.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Three native GitHub Actions to automate the mechanical parts of issue
triage:
- stale.yml: actions/stale scoped to the awaiting-reporter label —
warns at 21 days idle, closes at 35, auto-resets when the reporter
replies. Never touches PRs or unlabeled issues.
- label-actions.yml + .github/label-actions.yml: posts a templated
comment when duplicate or awaiting-reporter is applied (duplicates are
linked, not auto-closed).
- issue-labeler.yml + .github/issue-labeler.yml: adds area labels
(windows, stability/performance, parsing/quality, editor/integration,
ux/behavior, cypher, language-request) from title/body keywords.
Additive only; base bug/enhancement labels still come from the forms.
All third-party actions pinned to full commit SHAs with least-privilege
per-job permissions (issues: write only where required).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
- graph-ui: bump vite ^6.4.2 -> ^6.4.3 and add overrides pinning the transitive
form-data >=4.0.6 and @babel/core >=7.29.6 (all dev-scope build/test deps, not
shipped in the binary). Clears the 4 open Dependabot alerts and Scorecard's
VulnerabilitiesID. `npm audit` now reports 0 vulnerabilities.
- codeql.yml: move `security-events: write` from the workflow top level to the
`analyze` job (top level is now `contents: read`), resolving Scorecard's
TokenPermissionsID least-privilege finding. CodeQL still uploads results from the
job-scoped token.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Vendors the two MIT-licensed grammars from intersystems/tree-sitter-objectscript
@a7ffcdf (the language vendor's official grammars, ABI 15) that PR #467 needs:
internal/cbm/vendored/grammars/objectscript_udl/ (.cls)
internal/cbm/vendored/grammars/objectscript_routine/ (.mac/.inc/.rtn/.int)
Each directory carries the verbatim generated parser.c + scanner.c + tree_sitter/
headers + LICENSE. The one adjustment: each scanner.c's upstream
`#include "../../common/scanner.h"` is repointed to a per-directory
`objectscript_common.h` (a verbatim copy of the upstream common/scanner.h), because
this repo's shared vendored/common/scanner.h belongs to the cfml/fsharp grammars and
differs. Updates MANIFEST.md + THIRD_PARTY.md.
These files are dormant until #467 (the grammar shims + extraction) is rebased on
top; main's build is unaffected.
Refs #462, #467.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Andrew Hundt