Commit Graph
34 Commits
Author SHA1 Message Date
Andrew Hundt 4631a483ed fix(graph): reject malformed dump inputs
Add graph-buffer structural invariant validation before SQLite writer handoff so invalid live node IDs, stale node_by_id mappings, missing edge endpoints, and inconsistent edge indexes fail before temp DB creation instead of being silently remapped or dropped.

Add focused graph-buffer tests for valid graphs and missing endpoint rejection, adjust the pipeline-path isolation fixture to use the same shared ID allocation contract as production worker buffers, and add a direct sqlite_writer replay for node_vectors and token_vectors payloads.

Validation: git diff --check; make -f Makefile.cbm build/c/test-runner; bash scripts/check-source-safety.sh; CBM_ONLY_SUITE=graph_buffer ./build/c/test-runner (58 passed); CBM_ONLY_SUITE=sqlite_writer ./build/c/test-runner (8 passed); CBM_ONLY_SUITE=pipeline ./build/c/test-runner (222 passed); CBM_ONLY_SUITE=incremental ./build/c/test-runner (160 passed).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-06-30 04:13:02 -04:00
Andrew Hundt 7f87331e0c test(incremental): cover failed publish propagation
Expose the graph-buffer dump publish fault hook name from graph_buffer.h so tests and implementation share one constant instead of repeating the environment variable string.

Add a small pipeline incremental regression test that forces a dump failure before atomic replacement, asserts the incremental run returns failure, and verifies the previous DB remains unchanged with the new function unpublished.

Validation: git diff --check && make -f Makefile.cbm build/c/test-runner && CBM_ONLY_SUITE=graph_buffer ./build/c/test-runner && CBM_ONLY_SUITE=pipeline ./build/c/test-runner && bash scripts/check-source-safety.sh
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-06-30 01:35:42 -04:00
Andrew Hundt bc6c76dac6 test(graph): cover failed dump publication
Add a narrow test-only fault hook at the graph-buffer dump publish boundary, after temp database verification and before atomic replacement. The hook is enabled only by CBM_TEST_FAIL_GBUF_DUMP_BEFORE_REPLACE and logs through cbm_log_error without stdout output.

Add a regression test that publishes an existing DB, injects a failure before replacement during a second dump, and verifies the original graph remains queryable while the new graph is not published.

Validation: git diff --check && make -f Makefile.cbm build/c/test-runner && CBM_ONLY_SUITE=graph_buffer ./build/c/test-runner && bash scripts/check-source-safety.sh
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-06-30 01:28:23 -04:00
Andrew Hundt ecc8e4d43b fix(graph): harden determinism and safety canaries
Stabilize several graph-producing paths and add stronger regression coverage before the incremental redesign work continues.

Key changes:

- resolve call import maps from resolved graph-buffer IMPORTS edges so re-exported targets are reused consistently

- make semantic, similarity, complexity, route/section upserts, and resolver scoring more deterministic across full/incremental and worker-order differences

- propagate incremental phase/persist failures instead of continuing toward publication after failed phases

- add canonical graph diff helpers and stricter incremental artifacts for full-vs-incremental comparison

- add source-safety lint coverage for MCP stdout, unsafe string APIs, and new raw env/fs API usage

- replace touched unsafe string copies with bounded or project-wrapper-based forms

Validation:

- make -f Makefile.cbm build/c/test-runner

- CBM_ONLY_SUITE=pipeline ./build/c/test-runner: 217 passed

- CBM_ONLY_SUITE=graph_buffer ./build/c/test-runner: 55 passed

- CBM_ONLY_SUITE=mcp ./build/c/test-runner: 117 passed

- CBM_ONLY_SUITE=simhash ./build/c/test-runner: 24 passed

- CBM_ONLY_SUITE=parallel ./build/c/test-runner: 21 passed

- CBM_ONLY_SUITE=watcher ./build/c/test-runner: 57 passed

- bash scripts/check-source-safety.sh: OK

Known remaining:

- CBM_ONLY_SUITE=incremental still fails incr_accuracy_vs_full: 159 passed, 1 failed. Current artifacts show stale IMPORTS/CALLS/USAGE targets in unchanged importers and downstream SEMANTICALLY_RELATED drift; this is tracked in the forensic notes as a fallback-first incremental redesign task.

Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-06-30 00:38:07 -04:00
Andrew Hundt 205e9692e4 fix: stabilize incremental route parity
Rebuild route-derived edges during incremental indexing and clear only derived HANDLES/DATA_FLOWS rows that are owned by the route reconciliation pass.

Make infra route matching order-independent by considering every matching handler route, and make prefix-route bridging inspect every registrar edge while keeping diagnostics on the existing logger.

Reuse the full-index file/folder structure helper for changed files so incremental updates restore File metadata and CONTAINS_FILE/CONTAINS_FOLDER edges consistently.

Add graph-buffer selective edge deletion and regression coverage for route matching, prefix bridging, and full-vs-incremental edge accuracy.

Tests: build/c/codebase-memory-mcp; CBM_ONLY_SUITE=graph_buffer build/c/test-runner; CBM_ONLY_SUITE=infrascan build/c/test-runner; CBM_ONLY_SUITE=pipeline build/c/test-runner; CBM_ONLY_SUITE=incremental build/c/test-runner; CBM_WORKERS=1 CBM_ONLY_SUITE=incremental build/c/test-runner.

Caveat: incremental remains below the >=10x speedup gate for default enablement; full rebuild remains the safe default until the delta design is redesigned and measured.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-06-28 15:39:54 -04:00
Andrew Hundt 0370d52f4d fix: harden index persistence and MCP outputs
Make index writes and artifact refreshes use verified temp files plus atomic replacement, preserving Windows replace semantics and avoiding fixed temp-path collisions.

Thread the remaining pipeline confidence knobs through config while keeping zero-valued defaults backward compatible with existing upstream behavior.

Tighten route-node gates and architecture route output so infra URL nodes stay queryable in the graph but do not pollute get_architecture route summaries; preserve code-created absolute URL routes for all language extractors.

Harden MCP and CLI path handling around cache paths, grep scratch files, update downloads, checksum validation, stale checks, and project DB discovery without writing protocol noise to stdout.

Add regression coverage for corrupt-DB recovery, concurrent atomic writes, tmp-prefixed project DBs, duplicate search result fields, route output filtering, config registry entries, and focused suite selectors.

Validated with: git diff --check; make -f Makefile.cbm build/c/test-runner; focused suites security, artifact, graph_buffer, sqlite_writer, store_search, store_bulk, store_pragmas, store_checkpoint, watcher, infrascan, pipeline, store_nodes, mcp, cli, parallel, tool_consolidation, incremental, and store_arch; make -f Makefile.cbm cbm; isolated dogfood index/query run.

Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-06-28 06:34:23 -04:00
Andrew Hundt 1ad7556fac fix(graph_buffer): retain DB on relative root_path (was false-deleted) (#57)
The post-dump verify (B1 mitigation, 63de254) used the strict
cbm_store_check_integrity, which treats a non-absolute root_path (e.g. a
relative repo_path like ".") as bad_root_path → the verify DELETED a valid DB.
Indexing with a relative path therefore failed (status=error) — caught by
self-indexing the repo (cbm cli index_repository repo_path="."); the same
index with an absolute path succeeded cleanly (14723 nodes).

Switch the verify to cbm_store_check_integrity_full with the path_only
out-param: a path-only defect (cosmetic root_path) is RETAINED, consistent
with #557 resolve_store — only genuine structural corruption (>5 project
rows) is deleted. Verified: relative-path index now returns status=indexed;
absolute-path index unchanged.

test_graph_buffer gbuf_dump_relative_root_path_retained locks this in.
graph_buffer 50 pass.

Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-06-26 22:37:03 -04:00
Andrew Hundt 39de1eb7b5 perf: root-cause fixes for fork-origin #3 (purge) + #8 (dep N+1)
#3 — incremental purge was O(C·(N+E)): cbm_gbuf_delete_by_file was called
once per changed/deleted file, each a full nodes+edges scan. Add
cbm_gbuf_delete_by_paths (one pass over nodes matching a path membership
set + one edge cascade = O(N+E) regardless of path count); refactor
cbm_gbuf_delete_by_file to delegate to it (DRY — single implementation).
pipeline_incremental.c now purges all changed+deleted paths in one call
(OOM fallback keeps the old per-file path). graph_buffer 49 + incremental
purge tests pass; only the known B1 full-reindex failures remain.

#8 — cbm_dep_link_cross_edges was N+1: one cbm_store_search per import
(<=500) to find a matching dep Module. Now ONE bulk fetch of all dep Module
nodes + an in-memory name->id hash (first-wins, preserving limit=1
semantics) -> O(1) per import. depindex 32 pass.

#5 verified NOT a bug (avoided a false fix): node_degree IS populated by
cbm_pagerank_compute (pagerank.c:422 INSERT OR REPLACE INTO node_degree);
the correlated-subquery fallback only fires pre-PageRank.

Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-06-26 20:57:11 -04:00
Andrew Hundt 416e65fc51 fix(graph_buffer): verify + remove corrupt .db after dump (B1 mitigation)
cbm_gbuf_dump_to_sqlite intermittently emits a structurally-corrupt .db
(B1): on the ~16k-node fastapi repo a full index non-deterministically
persists 0/partial node counts and a garbage project root_path, leaving an
unreadable .db that queries then fail on (PRAGMA integrity_check won't even
prepare). The corruption is ASan/TSan/clang--analyze-invisible in the dump
path; root cause still open.

Add a post-write check in cbm_gbuf_dump_to_sqlite: after cbm_writer_finalize,
open the freshly-written .db and run cbm_store_check_integrity (O(1)
projects-table check, not a full integrity_check). If it fails, unlink the
.db and return GB_ERR so the caller re-indexes instead of serving garbage.
Confirmed firing in the incremental suite (logs dump.verify_corrupt, deletes
the .db).

Makes the system fail-safe against the worst (structurally-unreadable) B1
symptom: no silent corrupt .db is left for queries to read. Does not by
itself fix the underlying non-deterministic write corruption — the 2
pre-existing incremental-accuracy failures (incr_db_deleted_recovery,
incr_accuracy_vs_full) remain; that needs the root-cause fix. Complements
the #557 retain-bad-root_path mitigation in 22e438f.

Verified: store_nodes 56/56, sqlite_writer 7/7, mcp 108/108 (no regression).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-06-26 13:52:35 -04:00
Andrew Hundt 92f3b86f7a merge(upstream/main): integrate DeusData main into api-consolidation
Deep --no-ff merge of upstream DeusData/main (597 commits, merge-base 988d975)
into fork api-consolidation (HEAD 3018b22), producing a superset that combines
upstream breadth with the fork's surviving depth.

Upstream additions integrated: semantic/embedding search (vendored/nomic),
minhash/SimHash, cross-repo + cross-file LSP + route + pkgmap passes, artifact
import/export, path_alias, git_context, hook_augment, Windows support
(compat_fs/compat_thread), the UI layer, and ~140 expanded tree-sitter grammars.

Fork depth retained: PageRank/LinkRank ranking (src/pagerank/; sort_by=
relevance/linkrank/degree/calls), dependency indexing (src/depindex/;
CBM_INDEX_MODE=DEP), token reduction, and HTTP-endpoint discovery
(src/pipeline/httplink.c, pass_httplinks.c).

Conflict resolution (27 regions across the 39-file intersection):
- src/mcp/mcp.c: kept the fork superset handler pattern (resolve_project_store +
  expand_project_param + inject_context_once) as the single path; UNIONed the
  tool dispatch table; emit status:"indexed", excluded[], cross_service trace.
- src/store/store.c: upstream base + fork pagerank/linkrank/node_degree tables
  and sort_by plumbing; cbm_store_node_degree now excludes INHERITS
  (WHERE type != 'INHERITS'); removed the duplicate cbm_store_get_db.
- src/pipeline/pipeline.c: took upstream run_extraction_phase; grafted the fork
  passes (httplinks, normalize, flush_store dump branch); call
  cbm_pipeline_set_committed_counts before cbm_gbuf_dump_to_sqlite.
- internal/cbm/lang_specs.c: 158 entries combining the fork section_node_types
  with upstream ts_factory/embedded_imports; wired 13 orphaned type-arrays.
- src/depindex/depindex.h: renamed CBM_PATH_MAX -> CBM_DEP_PATH_MAX (it collided
  with constants.h enum CBM_PATH_MAX=1024) + 11 call sites in depindex.c.
- Makefile.cbm: upstream grammar/nomic units + fork CGO targets; added
  httplink.c + pass_httplinks.c to PIPELINE_SRCS; removed mongoose NOSAN
  machinery (mongoose was deleted upstream).
- httplinks restored (the merge silently deleted them because the fork had left
  them unchanged since the base): httplink.c/.h, pass_httplinks.c,
  tests/test_httplink.c; registered RUN_SUITE(httplink) in tests/test_main.c.
- CBM_INDEX_MODE enum unified to FULL=0, MODERATE=1, FAST=2, DEP=3.

Completeness (dual-parent check): 0 upstream files dropped from the merge tree;
6 fork files dropped, all intentional (upstream-superseded): CHANGELOG.md,
vendored/mimalloc/src/{arena-abandon,segment-map,segment}.c,
vendored/mongoose/{mongoose.c,mongoose.h}. Fork features (pagerank, depindex,
token_reduction, httplink) verified present and registered in tests/test_main.c.
src/pipeline/pass_parallel.c has an identical function set to upstream (49=49,
none dropped); the large line-deltas vs upstream are within-function divergence
of the fork extractor, not dropped functions.

Gate (run manually; hooks skip on merge commits): build green
(make -f Makefile.cbm cbm, exit 0); full C test suite 5980 passed / 2 failed /
0 crashes. The fork's pre-existing ASan crash in build_project_list_error_srv is
fixed by upstream's mcp.c rewrite, so the merge is strictly better than the fork
baseline on the crash axis.

Known issue, tracked separately (NOT a merge defect): 2 nondeterministic
failures in tests/test_incremental.c — incr_db_deleted_recovery:772 (diff_pct)
and incr_accuracy_vs_full:803 (full_nodes vs incr_nodes). Cause: the fork's
extractor (internal/cbm/extract_*.c) and parallel-resolve Route-node creation
are not thread-safe under the concurrent use of upstream's parallel pipeline
(run_parallel_pipeline), producing nondeterministic route/infra HANDLES edges
(~234k edge swing between runs). Sequential extraction (CBM_WORKERS=1) is
deterministic and correct. Root cause, bisect, and repro are documented in
notes/2026-06-24-fork-vs-upstream-divergence-assessment.md (section 8f); the fix
is deferred to a dedicated thread-safety audit (the make test-tsan target is
currently a stub and needs wiring).
2026-06-24 16:12:00 -04:00
Martin Vogel 5706bf52f0 Extract url_path for the dump index via real JSON parsing
The dump path sliced url_path out of the properties string with strstr/strchr, returning ESCAPED bytes and cutting at embedded escaped quotes, while the index's backing column is GENERATED AS json_extract(properties,'$.url_path') and yields the unescaped value. The mismatch left edges 'missing from index idx_edges_url_path' under PRAGMA integrity_check once the formerly-malformed route props became valid JSON. url_path is now parsed with yyjson so the indexed value matches json_extract exactly.
2026-06-10 18:25:08 +02:00
Martin Vogel 894932ce39 Stream the graph dump to SQLite per partition
cbm_gbuf_dump_to_sqlite now drives the streaming page writer in
64K-node partitions instead of building every node record up front.
Under memory pressure (RSS over the configured budget) it frees each
partition's heavy properties_json once those rows are persisted — the
properties column is write-once and never read again — which bounds the
dump/finalize memory peak on large graphs.

Output is byte-for-byte unchanged when the pressure path does not
engage, so normal indexing and the test suite keep the buffer intact.
Verified: full suite green, PRAGMA integrity_check ok, 1x Linux index
node-exact and regression-safe.
2026-06-03 23:03:19 +02:00
Martin Vogel 6ed58bc797 Intern repetitive gbuf string fields to cut indexing memory
Node label/file_path and edge type are highly repetitive (~30 labels,
~15 edge types, ~88k file paths across millions of records) yet were
strdup'd per node/edge. Intern them into a per-buffer pool so identical
content collapses to a single owned copy, freed once at teardown.

Cuts ~20M small allocations to ~88k and lowers peak RSS on a full
Linux-kernel index by ~1.46 GB (16.45 -> 14.99 GB), with no change to
graph output (node count identical; edge dedup is content-keyed).
2026-06-03 20:09:44 +02:00
Andrew Hundt 2ecedbeb80 Merge remote-tracking branch 'origin/main' into api-consolidation
# Conflicts:
#	.gitignore
#	.mcp.json
#	Makefile.cbm
#	cmd/codebase-memory-mcp/assets/skills/codebase-memory-exploring/SKILL.md
#	cmd/codebase-memory-mcp/assets/skills/codebase-memory-quality/SKILL.md
#	cmd/codebase-memory-mcp/assets/skills/codebase-memory-reference/SKILL.md
#	cmd/codebase-memory-mcp/assets/skills/codebase-memory-tracing/SKILL.md
#	internal/cbm/extract_imports.c
#	internal/cbm/lang_specs.c
#	src/discover/language.c
#	src/foundation/compat_fs.c
#	src/foundation/compat_fs.h
#	src/foundation/str_util.h
#	src/graph_buffer/graph_buffer.c
#	src/graph_buffer/graph_buffer.h
#	src/main.c
#	src/mcp/mcp.c
#	src/pipeline/pipeline.c
#	src/pipeline/pipeline.h
#	src/pipeline/pipeline_incremental.c
#	src/pipeline/pipeline_internal.h
#	src/store/store.c
#	src/store/store.h
#	src/watcher/watcher.c
#	tests/test_main.c
#	tests/test_pipeline.c
#	tests/test_store_bulk.c
#	tests/test_store_search.c
2026-04-10 22:13:18 -04:00
Andrew Hundt d00fd4c234 feat(pipeline): port incremental reindex from origin/main to api-consolidation
Ports B1 from the feature matrix (origin/main commits d6a5f89, a90ab5d):

New files:
- src/pipeline/pipeline_incremental.c: disk-based incremental re-indexing
  - classify_files(): compares mtime+size against stored hashes
  - find_deleted_files(): detects files removed since last index
  - persist_hashes(): saves mtime+size for all indexed files
  - cbm_pipeline_run_incremental(): orchestrates classify→delete→reparse→merge→persist
  - Runs passes: definitions, calls, usages, semantic, k8s
  - Merges into live DB via cbm_gbuf_merge_into_store() (no full rewrite)

Modified files:
- src/graph_buffer/graph_buffer.c: add cbm_gbuf_merge_into_store() — upserts nodes
  and edges from a graph buffer into an already-open store without wiping the project;
  used by incremental path to merge changed-file symbols into the live DB
- src/graph_buffer/graph_buffer.h: declare cbm_gbuf_merge_into_store()
- src/pipeline/pipeline.c: add incremental routing after file discovery:
  if DB exists with stored hashes → cbm_pipeline_run_incremental(); otherwise
  delete old DB files and proceed with full reindex; after full-index dump,
  persist file hashes for all files so next run can use incremental path;
  add store/store.h include; add cbm_pipeline_repo_path() and
  cbm_pipeline_cancelled_ptr() accessors needed by pipeline_incremental.c
- src/pipeline/pipeline.h: declare new accessor functions; add <stdatomic.h>
- src/pipeline/pipeline_internal.h: declare cbm_pipeline_run_incremental()
- Makefile.cbm: add pipeline_incremental.c to build
- tests/test_pipeline.c: add 6 incremental tests ported from origin/main:
  incremental_full_then_noop, incremental_detects_changed_file,
  incremental_detects_deleted_file, incremental_new_file_added,
  incremental_k8s_manifest_indexed, incremental_kustomize_module_indexed

Test result: 2298 passed, 1 failed (pre-existing integ_mcp_delete_project failure);
2 previously failing integration tests (integ_mcp_search_graph_by_label,
integ_mcp_search_graph_by_name) now pass with the incremental path active

Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-04-10 00:34:41 -04:00
Martin VogelandKoolerx a0715ff1eb Add JS/TS IMPORTS resolution, Channel schema, lint cleanup
Features:
- Relative import resolution for JS/TS/Python/Ruby (./foo, ../bar,
  leading dots) via cbm_pipeline_resolve_relative_import in fqn.c
- Generic Channel detection: Socket.IO emit/on, EventEmitter patterns
  produce Channel nodes with EMITS/LISTENS_ON edges and transport
  property (extract_channels.c)
- Constant resolution trick: `const EVENT = "foo"; emit(EVENT)` resolves
  the channel name through a per-file constant table

Lint fixes (337 errors across 16 files):
- Named constants replacing all magic numbers (enums/defines)
- Cognitive complexity reduced via helper extraction (table-driven
  sqlite_writer, pipeline predump passes, graph_buffer dump phases)
- SQLITE_TRANSIENT int-to-ptr workaround in store.c + mcp.c
- clang-format + cppcheck clean

Co-Authored-By: Koolerx <tommy@koolerx.com>
2026-04-06 12:41:31 +02:00
Martin Vogel 8a06d78ac7 Parallelize post-passes, fix mode filtering + semantic edge quality
- Parallelize pass_similarity and pass_semantic_edges via worker pool with
  thread-local edge buffers; sequential final merge since gbuf is not
  thread-safe. Adds cbm_lsh_query_into() as a thread-safe variant with
  caller-provided candidate buffer.

- Add activatable profiling subsystem (CBM_PROFILE=1 env or --profile flag)
  for step-level timing of extract, resolve, corpus build, vector phases,
  and sqlite dump. Zero overhead when disabled.

- Fix cbm_index_mode_t enum mismatch between pipeline.h (FULL=0, MODERATE=1,
  FAST=2) and discover.h (FULL=0, FAST=1). mode=fast silently no-op'd
  fast-discovery filtering because discover.c compared against the wrong
  value. Linux kernel fast mode went 1:40 -> 3:11 as a result; now back to
  1:40. Broaden the filter guard to mode != CBM_MODE_FULL so MODERATE and
  FAST both get aggressive discovery.

- Clamp cbm_sem_combined_score output to [0, 1]. The proximity multiplier
  returns up to 1.10 as a same-file boost which could push the final
  cosine score above 1.0.

- Short-circuit semantic scoring when MinHash jaccard >= 0.95. Exact
  near-clones are already emitted as SIMILAR_TO edges; returning 0 here
  avoids flooding SEMANTICALLY_RELATED with cross-service copy-paste
  boilerplate and frees the edge budget for genuine vocabulary-bridged
  relations.

- Validate search_graph semantic_query as an array of strings and return
  a clear error for a single-string input. Update the tool description
  to spell out the requirement explicitly with an example.

- JSON-escape user-controlled strings (callee names, call arguments,
  URL paths, import local_name) in call/argument properties. Introduces
  cbm_json_escape() in foundation/str_util.

- Skip SQLite pending_byte_page (file offset 0x40000000) during raw page
  writes in sqlite_writer to avoid corrupting databases that cross the
  1 GiB boundary.

- Migrate pretrained vector blob from UniXcoder (51K tokens) to
  nomic-embed-code (40856 tokens x 768d int8). Includes the extraction
  script under scripts/extract_nomic_vectors.py.
2026-04-06 12:41:30 +02:00
Martin Vogel e81a796fe6 Embed Nomic nomic-embed-code pretrained vectors (40K tokens × 768d) for semantic search quality
Major quality improvement: replace random index vectors with pre-trained
Nomic nomic-embed-code token embeddings. Tokens like 'error' and 'exception' now start
with similar vectors (learned from millions of code repos) instead of
arbitrary random projections. Co-occurrence enrichment adds project-specific
context on top.

Architecture:
- vendored/nomic/code_vectors.bin: 37.7MB raw int8 vectors
- vendored/nomic/code_vectors_blob.S: assembler .incbin (instant build)
- vendored/nomic/code_vectors.h: extern declarations + pretrained_vec_at()
- vendored/nomic/code_tokens.h: 40856 token strings (575KB)
- semantic.c: cbm_sem_random_index() now looks up pretrained vectors first,
  falls back to sparse random for unknown tokens
- CBM_SEM_DIM raised from 256 to 768 to match Nomic nomic-embed-code

Also: RRI (Reflective Random Indexing), code pattern vocabulary injection,
120+ abbreviation expansions, callee/caller/body token enrichment,
label filter (Function/Method/Class only) in vector search SQL.

Binary size: 136MB → 169MB (+33MB from embedded vectors).
Search quality: keyword queries return relevant error-handling functions.
Domain-specific keyword queries return the expected functions.
2026-04-06 12:41:29 +02:00
Martin Vogel 7a9b7db830 Add semantic_query to search_graph: vector search via cbm_cosine_i8 SQL function
Vector search pipeline (end-to-end):
- store.c: cbm_cosine_i8 custom SQLite function (int8 dot product with
  sqrt magnitude), registered at store open time
- store.c: cbm_store_vector_search() builds merged query vector from
  keywords via base RI vectors (xxHash), int8-quantizes, runs SQL query
  with JOIN nodes ON node_id = id for metadata
- mcp.c: search_graph gains semantic_query param (array of strings),
  returns semantic_results alongside regular results
- sqlite_writer.c: fixed build_vector_record to include node_id in
  record body (matches build_node_record pattern)
- mcp.c: use yyjson_mut_obj_add_strcpy (not add_str) to avoid
  use-after-free when vector results are freed before JSON serialization

Quality note: base RI vectors at query time don't capture co-occurrence
enrichment — results are moderate quality. Storing enriched token vectors
would improve this significantly (future work).
2026-04-04 17:14:47 +02:00
Martin Vogel 1b84943000 Separate perf tests from CI, fix cross-platform build issues
- Add CBM_SKIP_PERF=1 env var to skip incremental/perf test suite
- CI and Docker test targets skip perf by default (run.sh perf for manual)
- Convert all perf assertions to warnings (log timing, never block)
- Fix store.h anonymous enum in struct (GCC rejects, clang accepts)
- Fix test_store_search.c mkstemp on non-template path
- Add ca-certificates to Docker test image for git HTTPS
- Add cbm_gmtime_r shim in compat.h (Windows gmtime_s wrapper)
- Fix compat.c missing constants.h include (Windows build)
- Fix platform.c _environ redeclaration on mingw
- Rename trace_call_path -> trace_path in smoke/soak/fuzz scripts
2026-04-02 14:52:14 +02:00
Martin Vogel 7fa3acd0c6 WIP: strict linting + RAM-first pipeline (lint fixes pending) 2026-04-01 23:22:44 +02:00
Martin Vogel 6e4ca93cf0 Split 168 functions to cognitive complexity 25, zero lint errors
Lower thresholds to industry defaults:
  cognitive-complexity: 25 (was 250)
  statements: 200 (was 400)
  lines: 400 (was 800)

All 168 functions split into smaller helpers across 44 files.
Zero NOLINTNEXTLINE suppressions remain. Zero clang-tidy errors.

Add clang-tidy to scripts/lint.sh (--ci to skip where unavailable).
Fix sqlite_writer B-tree PageRef initialization. Fix Terraform struct
parsing, Louvain null guards, const qualifiers, shadow variables.

2741 tests pass.
2026-03-31 20:04:12 +02:00
Martin Vogel 87c2e144c0 Set cognitive complexity to industry default (25), tag 168 functions
Lower thresholds to best-practice values:
  cognitive-complexity: 250 -> 25 (industry default)
  statements: 400 -> 200
  lines: 800 -> 400

Add NOLINTNEXTLINE to 168 functions that currently exceed these thresholds.
Each suppression will be removed as the function is split in subsequent
commits. Zero net behavioral changes.

Also: extract execute_with_clause from cypher execute_single.
2026-03-31 20:04:12 +02:00
Martin Vogel f26acd4ced Eliminate NOLINTNEXTLINE suppressions + iterative AST walkers
Remove ~480 dead per-line NOLINTNEXTLINE comments. Update .clang-tidy with
documented upstream bug references for disabled checks.

Convert key AST walkers to iterative with explicit stacks: walk_defs (with
context-aware stack frames for class nesting), walk_throws, walk_readwrites,
walk_calls, walk_usages, walk_env_accesses, extract_elixir_call, glob_match,
cbm_yaml_free, rmdir_recursive, resolve_func_name, walk_variables_iter,
count_branching. Reversed child push for left-to-right traversal.

Disable readability-implicit-bool-conversion globally — designed for C++,
flags all idiomatic C patterns (bool = x && y) with no config to allow them.

Fix clang-format, implicit-bool edge cases, add test_helpers.h.
2026-03-31 20:04:11 +02:00
Martin Vogel 80680ea367 Cross-service communication discovery + RAM-first incremental indexing
AST-based detection of HTTP calls, async dispatch (Pub/Sub, Cloud Tasks,
Kafka, SQS, etc.), and config accesses via resolved qualified names.
Route nodes as cross-service rendezvous points with infra→handler matching.
Constant propagation for module-level string assignments. YAML infrastructure
URL extraction from Cloud Scheduler configs.

RAM-first incremental pipeline: load DB into graph buffer, purge changed
file nodes, extract directly into existing buffer (resolver sees all nodes),
dump back to disk. Zero edge gap on kubernetes/django/meilisearch/neovim.

- service_patterns.c: ~170 library patterns (90 HTTP, 50 async, 30 config)
- pass_route_nodes.c: Route node creation + infra URL matching
- extract_unified.c: string constant collection + string ref classification
- extract_calls.c: first_string_arg + keyword argument extraction
- pipeline_incremental.c: RAM-first load→purge→extract→resolve→dump
- graph_buffer.c: load_from_db, delete_by_file, foreach visitors
- C++ LSP crash fix: NULL guard in cbm_type_substitute
2026-03-28 13:59:42 +01:00
Andrew Hundt a775040358 fix(mcp,store,pagerank,pipeline): 18 bugs fixed, DF-1 degree precompute, pass_normalize, 11 TDD tests
Phases 1-8 from comprehensive plan (notes/2026-03-26-0013-plan-*.md):

Phase 1 — Input validation (F1,F4,F6,F7,F9,F10,F15):
  mcp.c: empty label→NULL, limit≤0→default, sort_by/mode enum validation,
  regex pre-validation via cbm_regcomp, depth clamp, direction validation

Phase 2 — B7 Cypher param fix + CQ-2 project expansion:
  mcp.c:handle_query_graph reads "cypher" first with "query" fallback,
  uses resolve_project_store for "self"/"dep"/path shortcuts

Phase 3 — DRY resolve_project_store in 5 handlers:
  handle_get_graph_schema, handle_index_status, handle_get_architecture,
  handle_get_code_snippet, handle_index_dependencies

Phase 4 — DF-1 degree precompute (100× faster queries):
  store.c: node_degree table DDL, search SELECT uses LEFT JOIN with
  HC-6 fallback to edge COUNT, cbm_store_node_degree reads precomputed
  table, arch_hotspots uses nd.calls_in, arch_boundaries adds behavioral types
  pagerank.c: is_calls field, degree accumulation during edge iteration,
  node_degree batch INSERT after LinkRank, OOM-safe allocations

Phase 5 — B2/B5 name-based caller fallback:
  pass_calls.c: 3-step resolution (exact QN → shared helper → Module)
  graph_buffer.c: cbm_gbuf_resolve_by_name_in_file DRY helper (HC-1)

Phase 6 — B17/B13 class-method edge repair:
  NEW pass_normalize.c: enforces I2 (Method→Class) and I3 (Field→Class)
  invariants via QN prefix + name+label+file fallback. O(M+F) runtime.
  pipeline.c: normalize pass before dump. Makefile.cbm updated.

Phase 7 — CBMLangSpec section_node_types field:
  lang_specs.h: added section_node_types (17th field)
  lang_specs.c: all 64 language specs updated with NULL initializer

Phase 8 — IX-1..3 indexing pathway fixes:
  mcp.c: autoindex_failed + just_autoindexed flags in server struct,
  REQUIRE_STORE captures pipeline return code, build_resource_status
  shows "indexing" state + failure detail + action_required hints

Additional fixes:
  G1: summary mode adds results=[] + results_suppressed=true
  CQ-3: Cypher + filter params produces warning

Tests: 2238 pass (11 new in test_input_validation.c covering F1,F6,F9,
F10,F15 edge cases, G1, CQ-3, IX-2). Updated test_store_nodes.c for
total degree. Updated test_token_reduction.c for G1 results key.

Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-26 01:59:37 -04:00
Martin Vogel e6e9c58e80 Fix all memory leaks + Docker test infra gaps
Leaks fixed (5506 bytes → 0):
- sqlite3_open_v2 allocates handle even on CANTOPEN — must call
  sqlite3_close before freeing struct (4x 792 bytes, 32 indirect)
- graph_buffer node_by_id: reuse existing heap key on hash table
  replace to avoid leak (2 bytes)

Docker test infrastructure:
- python3 (full) instead of python3-minimal (missing json module)
- git installed (19 tests were silently skipped)
- sqlite3_close_v2 + sqlite3_shutdown at test exit
2026-03-25 01:59:36 +01:00
Martin Vogel eaf9c83066 Eliminate rename() — reindex deletes old DB, index writes directly
Architecture change to prevent concurrent write corruption:

  Indexing (new project): cbm_write_db writes project.db directly
  Reindexing (existing): delete project.db, then index as new
  Incremental: upsert changes into existing project.db (unchanged)

Root cause: rename(project.db.tmp, project.db) while watcher/MCP
had the old file open via SQLite caused WAL corruption — stale
connections wrote to deleted inodes.

Changes:
- cbm_gbuf_dump_to_sqlite: write to final path, no .tmp + rename
- pipeline.c: if DB exists but not eligible for incremental, delete
  it before proceeding with full index (reindex = delete + index)
- Pipeline lock (atomic spinlock) serializes concurrent runs:
  MCP/autoindex block, watcher skips if busy
- delete_project acquires pipeline lock before unlinking
- BEGIN IMMEDIATE for all write transactions (respects busy_timeout)
- cbm_store_check_integrity: detects corrupt DBs (>5 project rows
  or invalid root_path), auto-deletes with ERROR log
- cbm_store_get_db accessor for test corruption simulation

Tests: 5 integrity tests + 4 concurrency lock tests
2026-03-24 23:14:33 +01:00
Martin Vogel d6a5f8984e Add incremental reindex + ADR hints in index/schema/adr responses
Incremental pipeline (pipeline_incremental.c):
- Disk-based delta re-indexing: classifies files by mtime+size vs stored
  hashes, deletes changed nodes (edges cascade), re-parses only changed
  files, merges into existing DB via cbm_gbuf_merge_into_store
- pipeline.c auto-routes: existing DB with hashes -> incremental, else full
- Full pipeline persists file hashes after dump for next incremental run
- No-op when nothing changed (watcher polls every 5-60s, 99% skip)
- 4 tests: noop, changed file, deleted file, new file added

ADR hints (ported from Go):
- index_repository: adr_present + adr_hint when no ADR exists
- get_graph_schema: adr_present + adr_hint per project
- manage_adr GET: adr_hint with creation instructions when no ADR
2026-03-20 13:15:01 +01:00
Martin Vogel 26a78142d6 Fix Windows/MinGW build: TLS, strndup, sys/wait.h
- CBM_TLS: use _Thread_local (C11 standard) instead of __declspec(thread)
  which only works on MSVC, not MinGW GCC
- Replace bare strndup() with cbm_strndup() in 6 files (strndup is POSIX,
  not available on Windows)
- Guard sys/wait.h with #ifndef _WIN32 in http_server.c
2026-03-18 17:42:44 +01:00
Martin Vogel a1d849aaaa Fix all memory leaks, vendor sqlite3, add Docker test infrastructure
Leak fixes (13 sources, 1.4MB → 0 bytes):
- Cypher: free col_names after rb_set_columns, free rejected inline
  prop nodes, store WITH aliases in qualified_name for auto-cleanup
- Store: add cbm_project_free_fields, fix find_node_ids_by_qns leak
- Graph buffer: free strdup'd hash keys during delete operations
- SQL scanner: free old start_tag before reassignment in deserialize

Platform independence:
- Vendor sqlite3 amalgamation (3.49.1) — eliminates system libsqlite3
  dependency, enables full ASan/LeakSanitizer instrumentation
- Add -I for vendored tree-sitter unicode headers — eliminates hidden
  dependency on system libicu-dev
- Add test-infrastructure/ with Docker Compose for local GCC+ASan
  testing that mirrors CI exactly
2026-03-18 16:28:27 +01:00
Martin Vogel 18fa9979ee Replace vmem with mimalloc global allocator, add extraction-phase prescan, fix __init__.py QN collision
Memory management:
- Vendor mimalloc v2.1.9 as global allocator (MI_OVERRIDE=1 in prod)
- New mem.h/mem.c: RSS-based budget tracking via mi_process_info()
- Remove vmem.c/vmem.h (mmap-based budget tracking)
- Remove slab tier2 bump allocator (~300 LOC); >64B goes to mimalloc
- Slab tier1 pages from malloc (= mimalloc) instead of vmem
- Arena blocks from malloc instead of vmem
- Budget raised from 35% to 50% RAM (no more untracked C++ heap)

Extraction-phase prescan (eliminates disk re-reads):
- HTTP call sites: keyword check + URL extraction during extraction
- HTTP routes: decorator + source-based extraction during extraction
- Config file refs: regex scan during extraction
- httplinks: 41.8s → 13ms on Linux kernel (3,212x faster)
- configlink: 41.4s → 0.8s on Linux kernel (54x faster)
- Linux kernel fast-mode total: 2m38s → 1m18s

Bug fixes:
- __init__.py Module QN no longer collides with Folder QN
- index.ts same fix for JS/TS packages
- 13 regression tests for QN collision at FQN + extraction layers
- search_graph/search_code default limit raised from 10 to 500k
- Resolve all clang-tidy, cppcheck, and clang-format warnings

Repo cleanup:
- tree-sitter-form, tree-sitter-magma moved to tools/
- .gitignore: build/, node_modules/, graph-ui/dist/, TEST_PLAN.md
2026-03-17 22:05:57 +01:00
Martin Vogel 0d216c7936 Resolve all clang-tidy, cppcheck, and clang-format warnings across 94 files
Enforce WarningsAsErrors: '*' with zero suppressions of entire check categories.
All non-FP warnings fixed at source; true false positives get per-instance NOLINT
with reasoning. Fixes include: braces around statements (1679), isolate declarations
(75), explicit casts for multilevel pointer conversions, missing includes, function
parameter documentation via NOLINT, and proper NOLINTBEGIN/NOLINTEND nesting.
2026-03-15 22:37:34 +01:00
Martin Vogel 94a9a92f83 Add C-native pipeline with blocking clang-tidy, cppcheck, clang-format linting
Complete C implementation of the indexing pipeline (src/), parallel worker
pool, graph buffer with merge support, SQLite writer, and 1893-test suite.

Linter setup: zero warnings from clang-tidy (all checks enabled, no NOLINT
suppressions), cppcheck, and clang-format. All issues fixed at source —
proper headers for external linkage, GROW_ARRAY macro restructured to
eliminate type parameter, null-deref paths guarded, named intermediates
for suspicious-argument checks.
2026-03-15 19:26:12 +01:00