Commit Graph
331 Commits
Author SHA1 Message Date
Thomas Dyar c43fc8df0e fix(cypher): label-filtered edge traversal silently truncates at 10 results
MATCH (c:Class)-[:DEFINES_METHOD]->(m:Method) returned at most 10 results
for any class, regardless of how many methods it actually has.

Root cause: bind_cap was set to scan_count (the number of nodes matched in
the initial pattern — typically 1 when querying a single class by name).
max_new = bind_cap * 10 = 10, so the edge expansion loop exited after
collecting 10 results. No error, no warning, no truncation indicator.

This is language-agnostic: any class with more than 10 methods in any
language was silently truncated. The fix is two characters:
  bind_cap = scan_count > max_rows ? scan_count : max_rows

Regression test: a Python class with 15 methods must return all 15 via
MATCH (c:Class)-[:DEFINES_METHOD]->(m:Method) with label filtering.

Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
2026-06-12 14:39:38 -04:00
Martin Vogel 3ee9d5d112 Drop the nim grammar and refresh language counts
Removes the heaviest vendored grammar (66 MB) along with its language
wiring and test fixtures; the supported-language count moves to 158
across README, site, and manifest. The grammar manifest additionally
records the canonical-source decisions for the five
registry-disagreement grammars and the per-directory license files
restored earlier this week.
2026-06-12 02:17:39 +02:00
Martin Vogel 630bd40a90 Replace the graph-UI HTTP server with a first-party implementation
The web server behind the graph UI is now a purpose-built in-house
module (src/ui/httpd.c): localhost-only listener, strict HTTP/1.1
parsing with fixed request caps, a per-connection receive deadline,
and Connection: close semantics. http_server.c keeps the routing and
handlers, rewritten against the new transport API; the public server
API and main.c are unchanged. The previously vendored third-party
server is removed entirely.

Comes with a new 28-test transport + routing suite (tests/test_httpd.c)
covering parsing edge cases (strict CRLF, Content-Length limits, raw
path matching, percent-decode rules) and live-socket behavior (CORS
policy, RPC dispatch, receive deadline, clean shutdown). The security
audit scripts now check the new file layout and treat any network call
in vendored code as a failure.
2026-06-12 02:17:25 +02:00
Martin Vogel 2b67abfd54 Probe the by-name index in import_map_suffix instead of scanning the whole registry
The import_map_suffix fallback ran cbm_ht_foreach over the ENTIRE exact-QN table (with a strlen per entry) for every call the direct candidate missed - on elasticsearch that is a 700k-entry scan per unresolved call site, and sampling showed it at 94% of resolve-phase CPU (cbm_registry_resolve -> cbm_ht_foreach -> ims_scan -> strlen). Any QN matching 'resolved.*.suffix' necessarily has the suffix's last segment as its own last segment, so the existing by_name index already holds the only possible candidates: probe it and prefix/tail-check the handful of entries. elasticsearch full index: 640 s -> 71 s, nodes identical, edges within 0.001% (iteration-order tiebreaks among equal-confidence candidates); strategy semantics covered by resolve_import_map_suffix and registry_confidence_import_map_suffix.
2026-06-11 15:06:50 +02:00
Martin Vogel 5706bf52f0 Extract url_path for the dump index via real JSON parsing
The dump path sliced url_path out of the properties string with strstr/strchr, returning ESCAPED bytes and cutting at embedded escaped quotes, while the index's backing column is GENERATED AS json_extract(properties,'$.url_path') and yields the unescaped value. The mismatch left edges 'missing from index idx_edges_url_path' under PRAGMA integrity_check once the formerly-malformed route props became valid JSON. url_path is now parsed with yyjson so the indexed value matches json_extract exactly.
2026-06-10 18:25:08 +02:00
Martin Vogel 8666a99e29 Close the remaining malformed edge-props producers
Three stragglers after 6c04ab7/7f78f93, found by edge json_valid audits on elasticsearch: (1) the parallel emit_route_registration twin still interpolated callee/route_path raw into CALLS props; (2) HANDLES props buffers were smaller than escaped handler QNs plus the JSON wrapper, so snprintf truncated the closing brace (escaping alone cannot prevent wrapper truncation - value buffers now 512B inside 1KB props); (3) DATA_FLOWS assembles sliced handler_params/args_json fragments that can break in ways escaping does not cover - the assembled props are now validated with yyjson and fall back to a minimal valid envelope. Verified by reindexing elasticsearch: edge json_valid violations 47 -> expected 0.
2026-06-10 17:07:10 +02:00
Martin Vogel 7f78f936ec Spill oversized index keys to overflow pages; escape remaining edge-props text
The hand-rolled dump writer stored index cells fully inline regardless of payload size. SQLite's format spills index payloads above the max-local threshold (16,422 bytes at the 64 KB page size) to overflow pages, so a long key written inline makes the reader interpret key bytes as an overflow page number: PRAGMA integrity_check reported 'invalid page number 0x43654C6C' (ASCII from the key text) in idx_nodes_name on elasticsearch (very long Section names), and name lookups on that index silently returned nothing. write_index_btree now rewrites oversized cells (varint + local prefix + overflow chain via write_overflow_pages) before page building; promoted separators carry their chains with them. Reproduce-first: sw_long_index_keys_overflow (20 KB name; integrity_check failed pre-fix, ok post-fix). Also escapes the remaining raw text interpolations in DATA_FLOWS route props and HANDLES handler QNs (the ~51-edge malformed tail after 6c04ab7).
2026-06-10 16:42:20 +02:00
Martin Vogel 6c04ab7f61 Escape sliced source text in edge properties JSON
Edge properties interpolated raw source-text slices into JSON via snprintf without escaping: decorator text (quotes + raw newlines, e.g. @register.tag("block") or multi-line @override_settings) in DECORATES props - django produced 3826 malformed edges - plus usage ref_name and route-registration callee/url in USAGE/CALLS props. Malformed edge JSON aborts every json_extract consumer, including the url_path_gen generated-column evaluation that runs during PRAGMA integrity_check. All such sites now route through cbm_json_escape (DECORATES twins in pass_parallel.c/pass_semantic.c, usage emit twins, route-registration calls). Regression: pipeline_edge_props_valid_json (register.tag fixture; real-repo red recorded on the django index).
2026-06-10 13:44:09 +02:00
Martin Vogel 359f37cd71 Escape control bytes in properties-JSON array items (raw newlines from multi-line C params)
The array appenders (param_types/param_names/decorators/base_classes) in both props-builder twins escaped only quote and backslash, so items sliced from multi-line C parameter declarations carried raw newline/tab bytes — invalid inside JSON strings (the remaining 118 malformed kernel rows after the truncation fix; e.g. param_types:["struct\n\t\t..."]). Array items now go through the full escape function, and both escapers degrade any other raw control byte (e.g. form feed) to a space. Reproduced via the prod CLI on a multi-line-param fixture (nl_fn json_valid=0 pre-fix); the sweep regression test gained that fixture (61st case).
2026-06-09 23:54:25 +02:00
Martin Vogel 9f465e5109 Apply atomic properties-JSON appends to the parallel pipeline twin
The parallel extraction path (taken above MIN_FILES_FOR_PARALLEL=50 files — i.e. every real repo) has its own copy of the props appenders in pass_parallel.c, so the previous fix in pass_definitions.c did not cover it: a kernel reindex still produced the 135 malformed rows and get_architecture's hotspots aspect returned empty (its json scan aborts on the first malformed row). Same atomic-append change as the serial twin. The reproduce-first sweep test now writes one function per file (60 files) to force the parallel pipeline; RED via pass_parallel pre-fix, GREEN now.
2026-06-09 23:30:51 +02:00
Martin Vogel 4b317c8779 Make definition-properties JSON appends atomic (no mid-value truncation)
build_def_props serialized into a fixed 2KB buffer and cut fields mid-value when full: the closing quote/brace got skipped, leaving malformed properties JSON (Linux kernel: 135 nodes, 50-param functions cut at 2047 bytes inside param_types — '["enum'). Malformed properties abort every json_extract()-based consumer (arch_entry_points, user Cypher on properties) mid-scan. Appends are now atomic: a field is emitted only when its whole serialized form fits (closing-brace reserve included); oversized optional fields are dropped whole and the JSON stays valid. Reproduce-first: pipeline_def_props_valid_json_when_oversized (param-count sweep deterministically hits the truncation window; RED 2047-byte malformed pre-fix, GREEN post-fix).
2026-06-09 22:59:33 +02:00
Martin Vogel 08115fdfb5 get_architecture: skip JSON property-key discovery (counts-only schema)
handle_get_architecture called cbm_store_get_schema, whose per-label/per-type property-key discovery runs json_each() over EVERY node and edge row (~10s of the 23s on a 4.8M-node kernel graph) — yet the handler only renders label/type counts. Added cbm_store_get_schema_counts (same impl, with_props=false) and switched the handler to it. The full get_graph_schema tool is unchanged. A partial expression index on is_entry_point was tried and reverted: json_extract in an index WHERE aborts store open on rows with malformed properties JSON (pre-existing in older DBs; see next commit).
2026-06-09 22:59:32 +02:00
Martin Vogel 66bf33a003 Cap name-resolution candidate lists at 256 (unresolvably ambiguous)
resolve_name_lookup walked the full by_name candidate array (reachability check + score per candidate, re-done per file via the per-file caches) for every cache-miss name. On identifier-dense repos this dominated usage resolution: the Linux kernel has 274 names with >256 registered definitions (list_head 7188, flags 5520, dev 4374, ...) and resolve_usages burned 987s CPU. candidate_count_penalty already floors confidence to ~3/count (<=0.006 at 256), so these matches were noise edges. Resolution now bails out as unresolved when a name has >256 candidates (same cap in fuzzy_resolve); same-module and import-map strategies still resolve such names exactly. Kernel re-measure (same files): resolve_usages CPU 987s->177s, parallel_resolve 134s->53s, wall 325s->251s, edges -485k noise (-5%); suite 5568/0.
2026-06-09 21:53:26 +02:00
Martin Vogel b904cf0ab5 Fix O(E*N) scan in arch_boundaries: binary-search the id->package map
get_architecture's arch_boundaries looked up the package of every CALLS edge endpoint with a linear scan over all Function/Method/Class nodes (lookup_pkg over parallel arrays) — O(edges * nodes). On the Linux kernel graph (~1.4M defs, ~1.4M CALLS edges) the boundaries aspect spun >10 minutes at 100% CPU; latent until C call extraction was fixed (the kernel previously produced almost no CALLS edges). The node query now orders by id and lookup_pkg binary-searches: 80k nodes/160k edges 16.6s -> 207ms in the regression test; kernel-graph get_architecture(all) >10min -> 23s. Reproduce-first: arch_boundaries_no_quadratic_scan (red 16566ms pre-fix vs 8000ms bound, green 207ms post-fix).
2026-06-09 21:32:16 +02:00
Martin Vogel 69c16cada4 fix: deliver the static -portable linux binary on all install/update paths
The standard linux release binary dynamically links glibc 2.38+ and
GLIBCXX_3.4.32, so it fails to start on Debian 11, Ubuntu 20.04/22.04,
RHEL/Rocky 8/9, Amazon Linux 2, etc. — yet install.sh, the npm and PyPI
wrappers, and the binary's own self-update all fetched it by default,
contradicting the "single static binary" promise.

Point every linux install + self-update path at the fully-static
"-portable" asset (gcc -static), which has no glibc floor. macOS/Windows
are unaffected and unchanged.

- install.sh, pkg/npm/install.js, pkg/pypi _cli.py: select -portable on linux
- src/cli/cli.c: self-update download URL AND checksum archive name both use
  -portable on linux (they must match or the update fails checksum verify)
- scripts/smoke-test.sh: assert linux self-update targets the -portable asset
- scripts/ci/check-glibc-compat.sh: new guard — runs the binary inside
  debian:bullseye (glibc 2.31) and asserts it starts
- _smoke.yml: run the guard on the portable binary in smoke-linux-portable

Reproduced: standard binary -> "GLIBC_2.38 not found" on glibc 2.31;
portable binary runs cleanly.
2026-06-09 11:36:00 +02:00
Martin Vogel 332bea6b42 Bind sqlite3 and libgit2 to mimalloc explicitly in prod build (#424 follow-up)
Defense-in-depth follow-up to the tree-sitter bind: route sqlite3 and libgit2
allocations through mimalloc explicitly so a correct binary never depends on the
fragile MI_OVERRIDE symbol override (where the Windows static-MinGW link can
resolve a library's malloc to mimalloc but its free to the CRT, corrupting the
heap). Prod-only, guarded by the existing CBM_BIND_TS_ALLOCATOR; the CRT+ASan
test build compiles these as no-ops and stays unchanged.

New cbm_alloc_init() consolidates all three binds behind a static idempotency
guard. main() calls it as its very first statement — before cbm_mcp_server_new
opens the in-memory store — because SQLITE_CONFIG_MALLOC must run before the
first sqlite3_open*/sqlite3_initialize, else sqlite3_config returns SQLITE_MISUSE
silently and the bind is ignored (the return code is asserted). cbm_init() also
calls cbm_alloc_init() so pipeline-pass entry points get the binds too; the
ts_set_allocator bind moved out of cbm_init into cbm_alloc_init.

sqlite3: a static sqlite3_mem_methods backed by mi_malloc/mi_free/mi_realloc,
xSize via mi_usable_size, xRoundup to an 8-byte boundary, xInit returning
SQLITE_OK, xShutdown a no-op (int-sized sqlite signatures wrapped with size_t
casts). libgit2: a static git_allocator (gmalloc/grealloc/gfree → mi_*) installed
via git_libgit2_opts(GIT_OPT_SET_ALLOCATOR). The allocator is set BEFORE any
git_libgit2_init: git_allocator_global_init keeps a custom allocator if one is
already set (git__allocator.gmalloc != git_failalloc_malloc), the pre-init global
is the fail-allocator, and there is no allocator reset on git_libgit2_shutdown —
so binding once at startup survives pass_githistory's per-call init/shutdown
pairs. libgit2 code is additionally guarded by HAVE_LIBGIT2.

Test build 5564/0 ASan-clean; prod build clean with CBM_BIND_TS_ALLOCATOR=1; the
HAVE_LIBGIT2 branch type-checked against the libgit2 sys/alloc.h struct shape;
lint-ci clean.
2026-06-09 11:25:56 +02:00
Martin Vogel 0314a216c6 Rename k8s enum PATH_MAX to avoid system-macro collision on macOS-15/Windows
PATH_MAX is defined by limits.h on the macOS-15 SDK (1024) and MSYS2 clang64 (260), so 'enum { PATH_MAX = 12 }' expanded to 'enum { 1024 = 12 }' → 'expected identifier' build failure on those platforms (Linux/macOS-14 didn't pull the macro into this TU). Renamed to K8S_PATH_DEPTH.
2026-06-09 00:59:19 +02:00
Martin Vogel 632af52f26 Make namespace import resolution deterministic across platforms
The namespace map kept only the first file per package (by platform-dependent iteration order). A same-package wildcard import (import com.example.* from a file itself in com.example) could resolve to the importer → self-edge suppressed → 0 IMPORTS on amd64 but 1 on arm64 (ei_kotlin_wildcard_import). Now store ALL files per namespace as a delimited list and resolve to the first non-importer sibling — deterministic on amd64/arm64. Re-key via the stored key pointer to avoid a use-after-free.
2026-06-09 00:01:36 +02:00
Martin Vogel 6f3b24df4f Fix CI lint: drop debug scaffolding, make lisp import walker iterative, apply clang-format
Remove leftover getenv(CBM_DBG_*) debug fprintf blocks in extract_calls.c/pass_calls.c (cppcheck always-true). Convert parse_lisp_imports to a stack-based walk (no recursion → no NOLINT, avoids deep-nesting overflow). Apply Homebrew clang-format to all touched sources. Local suite 5562/0, lint-ci clean.
2026-06-08 22:55:23 +02:00
Martin Vogel b30fe13de1 Emit INFRA_MAPS for HCL scheduler + k8s selectors, ASYNC_CALLS for SQS struct args
extract_calls: recover SQS/SNS queue identity from Go composite-literal struct fields (QueueUrl/TopicArn) so emit_http_async_edge forms ASYNC_CALLS. extract_unified: HCL value/body descent + cloud_scheduler_job source synthesis. pipeline: self-create topic Route for standalone-manifest infra bindings + run infra-route passes in the sequential pipeline. pass_k8s: cross-manifest Service-selector -> workload label/name matching -> INFRA_MAPS.
2026-06-08 22:55:23 +02:00
Martin Vogel de2c54ac1f Emit DECORATES to synthetic nodes for external decorators + python subscript-dunder CALLS
pass_semantic + pass_parallel: when a decorator name resolves to no local def, upsert a synthetic '<decorator:NAME>' node (label Decorator) and emit DECORATES+USAGE — fixes rust #[derive]/#[allow], swift @discardableResult, scala @deprecated (deduped per project; pipeline-level so extraction histograms unaffected). py_lsp: inject a synthetic CBMCall for resolved subscript/operator dunders on typed receivers so they reach the CALLS pipeline — fixes python s[0]->Seq.__getitem__.
2026-06-08 22:55:23 +02:00
Martin Vogel 35ac60c544 Resolve C# field writes + Kotlin cross-file calls
lang_specs: C# postfix/prefix_unary_expression as assignments (x++ -> WRITES). extract_semantic: resolve_write_lhs_node for unary inc/dec write targets. helpers: dedicated kotlin_keywords (Kotlin doesn't reserve double/int/... so fun double()/call double() now extract). kotlin_lsp + pass_lsp_cross: cbm_run_kotlin_lsp_cross + sole-definer top-level fallback. java_lsp: sole-definer type fallback for cross-file static calls. Fixes convergence 571, edge_structural 374, lsp_resolution 1210.
2026-06-08 22:55:23 +02:00
Martin Vogel 0d87a86e54 Resolve IMPORTS edges for 12 grammar-only languages
extract_imports: recursive lisp walker (Clojure ns/:require, Common Lisp defpackage/:use, Fennel nested require) + parsers for elm/move/smali(descriptor-demangle)/tlaplus(EXTENDS)/vhdl(use work)/wit/smithy(#->.)/hyprlang(source). lang_specs: astro frontmatter JS re-parse for component imports. pass_pkgmap: basename sibling-file candidate for prefixed include paths. All sound resolution via namespace-map / sibling / symbol-fallback / relative-import.
2026-06-08 22:55:23 +02:00
Martin Vogel 791b768c34 Emit IMPLEMENTS (not INHERITS) when a base resolves to an Interface
pass_semantic process_def: choose edge type by the resolved base node's label — Interface -> IMPLEMENTS (Java implements, C# : IFace, TS implements), Class/Type/Enum -> INHERITS. Fixes es_implements_samefile_java/csharp.
2026-06-08 17:46:32 +02:00
Martin Vogel 65ee5ef7b3 Add HANDLES for Spring/ASP.NET/Laravel routes + Guzzle/RestSharp HTTP_CALLS
extract_defs: Java/Spring + C#/ASP.NET attribute-based route mapping → HANDLES. extract_calls: accept Laravel string/array handler forms; preserve PHP scoped-call namespace QN so Guzzle HTTP_CALLS pattern matches. pass_calls: route C# client calls through service-pattern matching (RestSharp).
2026-06-08 17:24:40 +02:00
Martin Vogel 3c4e7b4eac Extract DECORATES for C#/PHP8/Scala/TS attributes + TS type-position USAGE
find_jvm_modifiers handles attribute_list (C#/PHP8); extract_decorators scans direct children (Scala annotation) and skips anon tokens (TS @Decorator before export); pass_semantic normalizes attribute syntaxes + C# Name->NameAttribute. USAGE: export_statement wrapping a declaration no longer suppresses type-annotation usages. py_lsp: operator/subscript dunder-call resolution for typed receivers.
2026-06-08 16:16:34 +02:00
Martin Vogel 861b9d917e Resolve grammar-only IMPORTS edges for sibling-file imports
pass_pkgmap.c: sibling-file resolution (direct, SCSS _partial, meson subdir/meson.build). language.c: map .just/.inc extensions. extract_imports.c: strip quotes from path imports (pony/func), parse_meson_imports (subdir()), generalized lisp parser for emacslisp/fennel/commonlisp node kinds. Fixes scss/just/bitbake/func/pony/emacslisp/meson import edges.
2026-06-08 14:24:39 +02:00
Martin Vogel 6afb45427c Add grammar-only import parsers + pipeline edges (WRITES/CONFIGURES/DEPENDS_ON)
Imports: per-grammar parsers for hare/pascal/powershell/scheme/racket/starlark/tcl/teal/zsh + css/html/cmake/bitbake/kconfig/gn/just/nix/jsonnet/pkl/nickel/thrift/capnp/d/tablegen/crystal/fsharp/ada; node-extraction for D/PowerShell/Pascal/Julia/F#; INHERITS for those. Pipeline edges: register Variable/Field defs for WRITES/READS resolution; CONFIGURES pass consuming env_accesses; DEPENDS_ON from go.mod/requirements.txt; SQS dot-form QN. Golden updates: julia +Class, just +Function (correct new extraction).
2026-06-08 13:04:12 +02:00
Martin Vogel 7275be3062 Add IMPORTS extraction+edge resolution and Go interface IMPLEMENTS/OVERRIDE
Imports: per-language extraction (PHP namespace_use, C# alias/static, Python wildcard/__future__, ES re-export) + generic import_node_types consumption for grammar-only langs; shared cross-pass import resolver (module-QN, namespace-map, symbol-name fallback, Rust crate-relative) in pass_pkgmap.c creates IMPORTS edges for all 9 hybrid langs + many grammar-only. Go: match interface methods via parent_class DEFINES_METHOD edges so IMPLEMENTS/OVERRIDE resolve. Kotlin throws via jump_expression detection.
2026-06-08 11:07:34 +02:00
Martin Vogel 2d772b01bf Fix extraction/resolution across languages: routing, Go methods, cross-LSP, inheritance, constructors, node kinds
WIP checkpoint (local). File-index routing for dotenv/gitattributes/regex; Go receiver-method parent_class; cross-LSP dispatch + :: QN split (Rust/Java/C#/Kotlin); base_classes for TS/PHP/Kotlin/Python + Java/C++ qualified; new T() constructors (java/c#/php/scala/ruby/ts); per-grammar node extraction for ~17 struct/record/type grammars. Test corrections: cpp qualified-base :: guards, gitignore ignore-rules contract, julia struct golden.
2026-06-07 22:56:12 +02:00
Martin Vogel 586729bdfe Resolve package.json workspaces cross-package IMPORTS on Windows (#408)
cbm_pkgmap_scan_repo was a no-op on Windows and pkgmap_walk_dir was POSIX-only (raw opendir/readdir), so package.json manifests were never read on Windows -> the pkgmap was empty -> bare @org/pkg workspace imports never resolved (cross-package IMPORTS = 0). The walk was disabled on Windows because a naive recursive descent hung on directory-junction cycles. Port it to the portable cbm_opendir/cbm_readdir API (as discover.c), guarantee termination with a hard PKGMAP_WALK_MAX_DEPTH=64 recursion bound (cannot hang regardless of junctions/symlink cycles), skip Windows reparse points and POSIX symlinks before descending, and remove the Windows gates. Verified: full macOS suite 4705 passed / 0 failed (contract_edge_workspaces_imports_issue408 now resolves on the POSIX path; Windows verified by CI).
2026-06-07 00:00:11 +02:00
Martin Vogel 118c327ff7 Fix cppcheck findings in distilled log + excluded summary (#411, #413)
log.c: check the buffer bound before reading raw[i] (arrayIndexThenCheck). mcp.c: compare excluded_count to INDEX_EXCLUDED_DIR_CAP directly for the 'truncated' flag (equivalent to >shown, but avoids cppcheck's knownConditionTrueFalse).
2026-06-06 22:59:48 +02:00
Martin Vogel bd77424648 Apply clang-format to the distilled watchdog in main.c (#406) 2026-06-06 22:53:58 +02:00
Martin Vogel de1d58ab9e Distill PR #407 (#406) and PR #414 (#413): parent-death watchdog + CBM_LOG_LEVEL
Two reviewed external-fork contributions, re-implemented as clean maintainer code.

#406 / PR #407 (thanks @nvt-pankajsharma): a POSIX parent-death watchdog so the stdio MCP server exits when its launching parent dies instead of lingering on stdin. Refactors signal_handler into an idempotent request_shutdown(); a watchdog thread polls getppid() (500ms) and shuts down + exit(0) once the initial ppid (>1) changes; Windows unaffected (#ifndef _WIN32). The fork's getppid()<=1 startup early-exit was dropped (it could wrongly kill a legitimately-launched server during reparent races / in container launchers; the initial_ppid>1 guard already no-ops safely), and thread-create failure is now non-fatal (matches the watcher/HTTP background-thread policy). Adds tests/test_parent_watchdog.sh + scripts/test.sh wiring.

#413 / PR #414 (thanks @santanusinha): runtime log-level control via CBM_LOG_LEVEL (case-insensitive debug|info|warn|error|none, or numeric 0..4; unset/unknown leaves the level unchanged). getenv() before threads start (no race). Adds tests/test_log.c cases + README env-var row.
2026-06-06 22:30:38 +02:00
Martin Vogel ba52d6ba56 Report excluded subtrees in the index_repository result (#411)
index_repository silently dropped entire subtrees (ALWAYS_SKIP dirs in all modes; FAST_SKIP dirs like tools/scripts/bin in moderate/fast) with no record — users couldn't tell what wasn't indexed. The discover walk now records skipped directory rel-paths; cbm_discover_ex returns them; the pipeline holds + exposes them; and the index response carries a compact summary: "excluded":{"dirs":[up to 25],"count":N,"truncated":bool}. Behavior-neutral for indexing. Adds a reproduction in test_integration.c.
2026-06-06 22:16:04 +02:00
Martin Vogel a2c9804397 Make the watcher git helpers work under Windows cmd.exe
The watcher shells out to git via cbm_popen, which runs through cmd.exe on Windows. The commands used POSIX-only constructs: single-quoted paths ('%s'), which cmd.exe does NOT strip — so git received a literal-quoted path, failed with 'cannot find the path', and the watcher fell back to strategy=none (no change detection); plus /dev/null and '| wc -l'. Use double-quoted paths (stripped by cmd.exe and POSIX sh alike), a platform null device (NUL / dev/null), count ls-files lines in C, and gate the submodule-foreach enhancement (whose inner shell command cmd.exe cannot pass) to POSIX. The watcher's git change-detection now works on Windows; this is what the (newly un-skipped) watcher tests exercise.
2026-06-06 19:05:40 +02:00
Martin Vogel c993c8cf84 Normalize cbm_mkdtemp output to forward slashes on Windows
The Windows cbm_mkdtemp built paths with backslashes (%TEMP%\\template). Tests embed that path in the JSON repo_path (where \\t/\\a are invalid escapes, so index_repository fails) and pass it to git -C. Convert to forward slashes — accepted by Windows file APIs, JSON, and git — fixing integration_setup, incremental, and the lang contracts on Windows.
2026-06-06 14:57:29 +02:00
Martin Vogel d8d114747c Free the thread-local node-type cache to satisfy LeakSanitizer
cbm_kind_in_set builds a thread-local cache of calloc'd symbol bitsets (ks_cache) that was never freed, so LeakSanitizer (Linux x64) reported the per-worker-thread caches as leaks once the previously-empty grammars gained real node-type sets. Add cbm_kind_in_set_free_cache() and call it at worker-thread teardown (extract_worker, beside cbm_slab_destroy_thread) and main-thread exit (test runner, beside sqlite3_shutdown).
2026-06-06 12:34:59 +02:00
Martin Vogel 8a5c992bd2 Satisfy lint for the new extraction recursion helpers
Whitelist find_first_descendant_by_kind and find_first_descendant_of in recursion_whitelist.h (bounded AST-depth DFS, like the existing r_collect_imports entry), and narrow two cppcheck-flagged variable scopes in sqlite_writer.c: use w->project at its single use, and fclose the value-copied wc.fp after free(w) instead of a pre-captured fp.
2026-06-06 01:26:06 +02:00
Martin Vogel 863bb70c94 Apply clang-format-20 across flagged sources
The CI lint (clang-format-20) flagged formatting violations across these files (pre-existing drift plus the new extraction code). Reformat to satisfy the check; no logic changes.
2026-06-06 00:20:47 +02:00
Martin Vogel 42dc17ead5 Add a star request to the update-available notice
Append a short "leave a star" prompt with the repo URL to the startup
update notice, so users who are behind a release see it on the first
tool call alongside the upgrade hint. Fits the existing 256-byte notice
buffer.
2026-06-05 18:48:30 +02:00
Martin Vogel 7b67cf9a22 Add memory back-pressure to parallel extraction
The extraction workers parse files concurrently; with the file list
sorted largest-first, they can collectively exceed the RSS budget on
very large repositories. Add a back-pressure gate at the top of the
worker loop: when over budget, a worker reclaims and naps (bounded
spins) before pulling another file so peers can finish and return
memory. Self-disabling when the budget is unset or RSS is under budget,
so small repos and the test suite are unaffected.

Also add per-phase RSS logging (mem.phase) at the registry / cross-LSP /
resolve boundaries for memory profiling.
2026-06-04 10:16:34 +02:00
Martin Vogel 22743b0d40 Reclaim allocator pages at the extract->registry handoff
After parallel extraction the live working set drops to the graph plus
the result cache (~7 GB on a 2x Linux index), but the allocator still
holds ~10+ GB of freed pages from the parallel parse phase. Call
cbm_mem_collect() before registry_build so those pages are returned to
the OS, cutting the process footprint at the handoff (RSS 17.6 -> 6.8 GB
on 2x) and easing system memory pressure. Adds a mem.collect log line
for observability.
2026-06-03 23:28:00 +02:00
Martin Vogel 894932ce39 Stream the graph dump to SQLite per partition
cbm_gbuf_dump_to_sqlite now drives the streaming page writer in
64K-node partitions instead of building every node record up front.
Under memory pressure (RSS over the configured budget) it frees each
partition's heavy properties_json once those rows are persisted — the
properties column is write-once and never read again — which bounds the
dump/finalize memory peak on large graphs.

Output is byte-for-byte unchanged when the pressure path does not
engage, so normal indexing and the test suite keep the buffer intact.
Verified: full suite green, PRAGMA integrity_check ok, 1x Linux index
node-exact and regression-safe.
2026-06-03 23:03:19 +02:00
Martin Vogel 6ed58bc797 Intern repetitive gbuf string fields to cut indexing memory
Node label/file_path and edge type are highly repetitive (~30 labels,
~15 edge types, ~88k file paths across millions of records) yet were
strdup'd per node/edge. Intern them into a per-buffer pool so identical
content collapses to a single owned copy, freed once at teardown.

Cuts ~20M small allocations to ~88k and lowers peak RSS on a full
Linux-kernel index by ~1.46 GB (16.45 -> 14.99 GB), with no change to
graph output (node count identical; edge dedup is content-keyed).
2026-06-03 20:09:44 +02:00
Martin Vogel 12683b84a9 feat(graph): queryable computation-bottleneck metrics + indexing perf & fixes
Per-function complexity metadata is now stored on graph nodes and queryable,
alongside several indexing performance and correctness fixes developed and
validated together (3704 tests, ASan/UBSan clean).

Bottleneck metrics (query via query_graph):
- Tier A (in the extraction AST walk): cyclomatic (complexity), cognitive
  (nesting-weighted), loop_count, loop_depth (max nested-loop depth),
  param_count, max_access_depth.
- Tier B (new pre-dump pass, pass_complexity.c): transitive_loop_depth
  propagated along CALLS edges + a recursive flag (direct self-recursion and
  mutual-recursion cycles), plus the call-context signals linear_scan_in_loop,
  alloc_in_loop, recursion_in_loop and unguarded_recursion.
- query_graph and get_architecture tool descriptions document the metrics and
  the Leiden community clusters.

Cypher engine:
- node_prop exposes arbitrary persisted node properties to WHERE/RETURN.
- Fix projection aliasing: multi-property rows shared a single static buffer so
  every column returned the last value read; now per-column/rotating buffers.
- Fix a stack-use-after-scope in aggregate RETURN (caller-owned value buffers).

Indexing performance:
- Gate C/C++ #define Macro-node extraction to full mode (it is ~49% of nodes on
  the Linux kernel); moderate/fast skip it.
- Emit the complexity property block only for Function/Method nodes so the
  millions of Macro/Field/Variable/Class/Enum nodes no longer carry zeroed
  fields — large RAM reduction at scale.
- Classify node types via tree-sitter TSSymbol bitsets in cbm_kind_in_set
  instead of per-node strcmp scans (thread-local cache, strcmp fallback;
  behaviour-identical).
- Subsample frequent (Zipfian) tokens in the semantic co-occurrence finalize;
  ~14x faster finalize on the kernel, output unchanged.
- pass_lsp_cross: replace O(n^2) linear dedup with hash-set dedup.

Windows:
- Canonicalize drive-letter case during path normalization so "c:/repo" and
  "C:/repo" derive the same project key and cache file (#394/#227/#367).

Tests: extraction, pipeline and cypher regressions covering all of the above.
2026-06-02 18:50:55 +02:00
Martin Vogel 09e3f71bf6 fix(search_code): accept '&' in paths, warn on literal-'|' trap, report timing
#272: '&' is neutralised by the single-/double-quoting of the grep/Select-String/
git commands, so it no longer fails validation. A relaxed validate_search_path_arg
(cbm_validate_shell_arg minus '&') is used for search_code + detect_changes
root_path/file_pattern; all other shell metacharacters stay rejected and
base_branch keeps the strict check.

#282: search_code defaulted regex=false, so 'foo|bar' matched the literal pipe
and silently returned 0 results. The result now carries a warnings[] array that
flags a literal '|' under regex=false (with the regex=true hint), so the trap is
visible instead of looking like a legitimate no-match. (The related invalid-regex
silent-empty case was already fixed in #283.)

Also: every search_code result now reports elapsed_ms, and a perf warning fires
when a search exceeds 5s (also logged as search.slow) so slow calls are visible.

Tests for both issues + the multi-word/invalid-regex no-regression; generous
search_code smoke coverage (basic+elapsed_ms, literal-| warning, '&' acceptance).

Closes #272
Closes #282
2026-06-01 22:53:17 +02:00
Martin Vogel d87cffe168 feat(arch): surface Leiden community clusters in get_architecture
Wire the Leiden detector into get_architecture via a new 'clusters' aspect.
arch_clusters loads Function/Method/Class nodes + CALLS edges (capped for very
large graphs), runs cbm_leiden, and reports the top-N communities compactly —
each with a dominant-package label, member count, cohesion (internal vs boundary
edge ratio), representative top nodes (by degree), and the packages spanned.
Singletons are skipped to avoid noise. The mcp.c serializer already emitted the
clusters array; it is now populated.

Tests: arch_clusters_basic (two cliques + a bridge -> >=2 connected clusters).
Generous smoke coverage added for the full Cypher surface (labels/type/id/keys/
properties/size/reverse/replace/left/coalesce/substring, NOT EXISTS dead-code,
CASE, unsupported-function error) plus the get_architecture clusters aspect.
2026-06-01 22:36:50 +02:00
Martin Vogel 08b62f03b9 feat(cypher): bounded EXISTS { } pattern predicate in WHERE
Add a single-hop, anchored existence predicate:
  WHERE NOT EXISTS { (f)<-[:CALLS]-() }   -- functions with no caller
  WHERE EXISTS { (f)-[:CALLS]->() }       -- functions that call something

Parsed via parse_exists_predicate (reusing parse_node/parse_rel) into a leaf
condition (op=EXISTS, anchor variable, edge type, direction); evaluated against
the bound node with cbm_store_find_edges_by_source_type / _by_target_type. This
is edge-type-specific, so it finds true orphans that in_degree/out_degree miss
(e.g. a node with only a DEFINES edge but no CALLS caller). Multi-hop / nested-
WHERE EXISTS is intentionally unsupported and errors clearly.

Tests cover the dead-code (NOT EXISTS) and has-outgoing (EXISTS) cases; smoke
coverage added. ASan-clean.

Refs: Cypher read suite (EXISTS predicate)
2026-06-01 21:17:56 +02:00
Martin Vogel c160465118 feat(cypher): multi-argument scalar functions (suite tier 2b)
Add an args-list (cbm_func_arg_t) to return items and parse comma-separated
arguments (var.prop or string/number literals), enabling multi-argument scalar
functions in projections:

- coalesce(a, b, ...)        first non-empty value
- substring(s, start[, len]) 0-indexed substring
- replace(s, from, to)       replace all occurrences
- left(s, n) / right(s, n)   leading/trailing n chars

Arguments are freed in free_return_clause (ASan-clean). Unit test
cypher_func_multiarg plus smoke coverage for substring + coalesce.

Refs: Cypher read suite (tier 2b)
2026-06-01 21:04:25 +02:00