mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-08 08:53:43 +08:00
b61861eb5a4672d979e31293d8e14fa3f7b671f7
227
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Martin Vogel
|
b61861eb5a |
feat(pipeline): add advanced index mode — gate ALL LSP behind it
New CBM_MODE_ADVANCED=3. Both per-file LSP (type-aware resolution during extract) and the cross-file LSP pass (during resolve) are now OPT-IN: FULL and below run tree-sitter extraction + registry textual resolution only; ADVANCED adds the full LSP layer. Rationale: profiling showed LSP changes the kubernetes graph by only ~0.01% of edges (it mostly re-confirms textual resolution) while costing significant memory + time. Making it opt-in keeps the default FULL path lean (3.0 GB vs 5.6 GB peak RSS on K8s) without meaningfully worse output, and lets users pay for LSP precision only when wanted. Mechanism: - cbm.c: g_lsp_enabled atomic toggle wraps the entire per-file LSP block (Go/C/C++/PHP/Python/TS/JS/C#) + the C/C++ preprocessor-pass LSP call. Default true so ad-hoc extraction / tests keep historical behaviour. cbm_set_lsp_enabled / cbm_get_lsp_enabled in cbm.h. - pipeline.c: sets the toggle (save→set→restore so it never leaks to other callers) from p->mode before extraction; gates the cross-LSP all_defs collection + registry build on advanced; gates the sequential lsp_cross dispatch. Fixed the moderate_only comparison (== FAST, not > MODERATE) so ADVANCED is not treated as lighter. - pipeline_incremental.c: same toggle save/set/restore. - discover.h/.c: added CBM_MODE_ADVANCED to discover's enum + all 4 filter comparisons treat ADVANCED like FULL (full discovery, no fast-filtering). Without this, ADVANCED (numerically 3) was mis-detected as fast-mode and dropped ~half the files. - mcp.c: parse mode "advanced" + tool schema enum/description. Dual-mode K8s bench: FULL 46.7s (lsp_overrides=0, 3.0 GB) vs ADVANCED 48.8s (lsp_overrides=80,818, 5.6 GB). Graphs within 0.01%. All 3,615 tests pass. |
||
|
Martin Vogel
|
2ee0c68d4a |
perf(foundation): swap CBMHashTable internals to Verstable v2.2.1
Replaces the prior Robin-Hood open-addressing implementation with Verstable (https://github.com/JacksonAllan/Verstable), a 2024 state- of-the-art C hash table. The public API in foundation/hash_table.h is UNCHANGED — CBMHashTable is now an opaque type and all 14 production callers (graph_buffer, registry, pipeline passes, service_patterns, watcher, semantic, etc.) get the new impl transparently. str_intern.c keeps its specialised (ptr+len) hash — different semantics, not a clean swap target. Why: per-sub-phase profiling pinpointed cbm_pipeline_lsp_target_node and cbm_registry_resolve as the dominant resolve_calls cost. Both ultimately hit cbm_ht_get; both run millions of times on a kubernetes index. Verstable's key innovations target exactly this: * 4-bit hash-fragment metadata in a uint16_t sidecar array lets chain walks skip most key comparisons WITHOUT touching the bucket payload (cache wins on hot lookups). * Quadratic probing with per-bucket displacement chains (not linear Robin-Hood psl). * Tombstone-free deletion that moves at most one existing key. Vendored at internal/cbm/vendored/verstable/verstable.h (2069 lines, 84KB, MIT). Wrapped as a const char* → void* template inside hash_table.c. The header struct is opaque (forward declaration only); audited the entire tree — no caller accesses internals or takes sizeof(CBMHashTable), so opacity is safe. Test refactor: test_hash_table.c had two impl-specific tests: * ht_robin_hood_bounded_psl — Robin-Hood specific; removed. ht_stress_10000 covers the functional equivalent. * ht_power_of_two_capacity — renamed ht_create_small_capacity, replaced struct probes with a functional insert+lookup check. ht_create_capacity_zero/one similarly de-internalised. Benchmark on kubernetes (12 workers, 20,650 files): - parallel_resolve wall: 36,376 -> 27,217 ms (-25.2%) - pipeline.done total: 61,485 -> 51,745 ms (-15.8%) - registry_build: ~1,000 -> 779 ms (faster — registry uses cbm_ht_set heavily during build) - All 3,615 tests pass; hash_table suite 100% green - ASan/UBSan/SEGV: none - Peak RSS: ~6.83 GB - Graph: nodes 297,082 -> 297,102 (+20, 0.007%), edges 1,155,444 -> 1,154,025 (-1,419, -0.12%) — within tolerance; minor delta likely from hash-iteration-order differences affecting tie-breaks in resolution. Sanity (zod/serilog): nodes exact, edges within 0.1%. Cumulative trajectory from broken-state baseline ~35 min on K8s -> 51.7s wall = ~40x total speedup. |
||
|
Martin Vogel
|
01e49a12bf |
perf(registry): per-file full-result cache for cbm_registry_resolve
THE big win. Pinpoint-instrumented profiling showed 98.7% of resolve_calls CPU on kubernetes (881s out of 893s) lived inside the 'resolve' sub-step — cbm_pipeline_lsp_target_node when LSP hits, cbm_registry_resolve strategy chain when it misses. The LSP path is fast; the strategy chain isn't. cbm_registry_resolve's strategy chain (import_map -> same_module -> unique_name -> suffix_match+scoring) is deterministic for a given (callee_name, module_qn) pair. module_qn is CONSTANT within a file, so within one file the same callee_name resolves identically every time. K8s files have ~200 calls but only ~50 unique callee_names — so ~75% of strategy-chain walks are redundant. Added a TLS hash cache (_resolve_cache, registry.c) keyed by callee_name with value cbm_resolution_t. cbm_registry_resolve checks the cache first; on miss it runs the full chain and caches BOTH positive and negative results (caching the negative is just as valuable — same name asks the same question). API in pipeline.h: void cbm_registry_resolve_cache_begin(int estimated_capacity); void cbm_registry_resolve_cache_end(void); Two smaller chunks landed in the same commit: - src/pipeline/registry.c: per-file import-map prefix -> module-qn hash (cbm_registry_import_map_cache_*). resolve_import_map's linear strcmp scan becomes O(1). Flat impact in this run because the LSP path bypasses cbm_registry_resolve for most K8s calls, but useful for projects without strong LSP coverage. - src/pipeline/pass_parallel.c: reuse lsp_target as target_node when try_field_type_hint didn't override the QN. Saves 1 cbm_gbuf_find_by_qn lookup per LSP-resolved call. Also added 6 sub-counters inside resolve_file_calls (time_ns_rc_*) logged as parallel.resolve.calls_breakdown / breakdown2 to keep durable telemetry on which sub-step dominates. Benchmark on kubernetes (12 workers, 20,650 files): - parallel_resolve wall: 90,531 -> 36,376 ms (-59.8% vs C2; -64.8% vs original 103,281 baseline) - resolve_calls CPU: 893,657 -> 277,556 ms (-68.9%) - 'resolve' sub-counter: 881,650 -> 266,521 ms (-69.8%) ← target hit - resolve_usages CPU bonus: 10,986 -> 5,594 ms (-49.1%, secondary effect — fewer ambiguous resolutions to scrub) - Total pipeline.done: 115,918 -> 61,485 ms (-47%) - User CPU total: 953.7s -> 375.1s (-60.7%) - Peak RSS: +430 MB for cached cbm_resolution_t structs (~12 live caches at a time) - Tests: 3616/0 - Graph: nodes 297,082 EXACT match, edges +/- 0.2% - zod/serilog sanity: nodes exact, edges within tolerance Effective hit rate: predicted 75%, measured ~70%. The ~30% true cache misses (unique callee_names per file) still run the full chain — next-tier optimizations: widen scope to per-worker so common names like 'Get'/'Add' are computed once per worker not once per file, or pre-filter candidate lists by import reachability at file entry to shrink the chain's inner cost. Cumulative trajectory from original broken state: ~35 min K8s cold index -> 63.6s wall = ~33x total speedup. |
||
|
Martin Vogel
|
1af1dd8688 |
perf(service_patterns): per-worker TLS cache for cbm_service_pattern_match
cbm_service_pattern_match runs once per resolved CALL edge inside
emit_service_edge — that's 7 pattern lists × ~30 patterns × strstr
= ~210 strstrs per call. On kubernetes (~600k resolved CALL edges)
the SAME resolved QN ("fmt.Errorf", "context.Context.Done",
"errors.New", etc) repeats thousands of times across files. Caching
turns the linear pattern-list scan into one hash lookup after the
first miss.
API additions in service_patterns.h:
void cbm_service_pattern_cache_begin(void);
void cbm_service_pattern_cache_end(void);
Cache is TLS (no contention) scoped to the resolve_worker's lifetime
in the parallel_resolve phase. Capacity 8192 entries per worker —
covers typical project QN diversity without rehashing.
Encoding: enum value + 1 in the void* slot (0/NULL means "miss").
Two helpers svc_enum_to_ptr / svc_ptr_to_enum hide the trick.
Benchmark on kubernetes (12 workers, 20,650 files):
- parallel_resolve wall: 98,520 -> 90,104 ms (-8.5% vs Chunk 1
baseline, -12.8% vs original 103,281 baseline)
- resolve_calls CPU: 992,427 -> 885,663 ms (-10.8% vs Chunk 1,
-5.4% vs original — Chunk 1 added overhead that Chunk 2 recovers
AND surpasses)
- resolve_usages CPU: 11,977 -> 10,944 ms (-8.6%)
- total_loop CPU: 1,005,409 -> 897,561 ms (-10.7%)
- Tests: 3616/0
- Graph: 297,082 -> 297,080 nodes (-2), 1,153,482 -> 1,156,097 edges
(within 0.2%, semantic-preserving)
- zod/serilog sanity: edges within +/-3 of baseline
- Peak RSS: 7.46 -> 7.75 GB (+290 MB for 12 × 8192-entry caches)
Cumulative trajectory: original broken state ~35 min K8s ->
1.92 min wall (-49× total). The per-file is_import_reachable cache
(Chunk 1) + the per-worker service-pattern cache (Chunk 2) together
save ~13s of resolve wall time. Next chunks target the gbuf_insert_
edge hot path and remaining strstr scans.
|
||
|
Martin Vogel
|
109c9f07c1 |
perf(registry): per-file memoization cache for is_import_reachable
Wraps is_import_reachable's O(candidates × imports × strstr) hot loop with a TLS hash-table cache scoped to the resolve_worker's per-file iteration. Same callee_name appears in many call sites and each lookup re-evaluates the same ~30 candidates against the same imports — with the cache, first eval pays the strstr cost, repeats are O(1). API additions in pipeline.h: void cbm_registry_reach_cache_begin(int estimated_capacity); void cbm_registry_reach_cache_end(void); resolve_worker calls _begin right after build_import_map and _end right before the loop's free_import_map (the cache scope must match the import_vals scope — different files have different imports, so the cache is invalidated between files). Benchmark on kubernetes (12 workers, 20,650 files): - parallel_resolve wall: 103,281 -> 98,520 ms (-4.6%) - resolve_usages CPU: 33,971 -> 11,977 ms (-65%) — cache hits hard here - resolve_calls CPU: 935,958 -> 992,427 ms (+6.1%) — cache adds overhead here without saving work, because the Tier 3 LSP override path hits first and bypasses cbm_registry_resolve for most CALL entries. Usage/throws/rw paths always go through cbm_registry_resolve so the cache helps them; the net wall improvement comes from that side. - Tests: 3616/0 - Graph: 297,082 -> 297,115 nodes (+33), 1,153,482 -> 1,155,005 edges (within 0.1%, semantic-preserving) - zod/serilog sanity: nodes exact, edges +/-1 (within tolerance) The strstr overhead per call wasn't the dominant resolve_calls cost — the next chunks need to attack the actual hot spots: emit_service_edge's cbm_service_pattern_match (6 lists × ~30 patterns × strstr per call = ~180 strstrs/call) and the gbuf_insert_edge allocator pressure. |
||
|
Martin Vogel
|
e81853d105 |
feat(parallel): per-sub-phase timing in resolve_worker
Instrument resolve_worker with per-sub-phase atomic ns counters
(import_map, cross_lsp, resolve_calls, resolve_usages, resolve_throws,
resolve_rw, resolve_semantic) plus a whole-iteration total_loop and
files_visited count. Logged at end of cbm_parallel_resolve as three
log lines (split to avoid the PP_RING=4 TLS-slot aliasing in itoa_log
that masked the data in the first attempt).
Profiling result on kubernetes (12 workers, 20,650 files):
parallel_resolve wall: 103,281 ms
total_loop CPU-ms: 970,352 (78.3% worker utilization, busy)
phase_sum CPU-ms: 970,303 (99.99% coverage of loop body)
resolve_calls: 935,958 96.45% ← THE BOTTLENECK
resolve_usages: 33,971 3.50%
resolve_semantic: 192 0.02%
resolve_calls (cost): 935,958 / 616,795 = 1,517 µs per call edge
resolve_usages (cost): 33,971 / 942,846 = 36 µs per usage edge
Calls are 42× more expensive per edge than usages. The extra work in
resolve_file_calls (vs resolve_file_usages) traces to:
1. is_import_reachable (registry.c:154) — O(imports × 2*strstr) per
candidate × ~30 candidates per name × 1.6M call attempts =
~480 sec of strstr work alone.
2. cbm_service_pattern_match — O(6 lists × ~30 patterns × strstr)
per call = ~48 sec aggregate.
3. try_field_type_hint, emit_service_edge dispatch overhead.
These are now the next optimization targets — replace substring
matching with hashed prefix lookups (trie or hash of module
fragments), and add per-file memoization of is_import_reachable
results so the same candidate doesn't get re-evaluated per call site.
Instrumentation overhead is negligible (~30ns atomic_fetch_add per
phase per file = <100µs aggregate on K8s) and gives durable
telemetry for future perf work.
|
||
|
Martin Vogel
|
df6ddf058b |
perf(parallel): O(1) per-file lsp resolution index + ASan fix
Two changes: 1. resolve_file_calls now builds a per-file CBMHashTable mapping 'caller_qn|callee_short' -> CBMResolvedCall* once at function entry, then does O(1) lookups instead of the O(N) linear scan in cbm_pipeline_find_lsp_resolution. Falls back to the linear scan only for keys the index misses (preserves all existing semantics). The hash key uses callee_qn's short name (last dot-separated segment) to match the original linear-scan's match rule. On insert, keeps the highest-confidence entry per key. 2. Memory safety: keys must be strdup'd before cbm_ht_set, because cbm_ht_set stores the key pointer rather than copying. The first draft used stack-allocated 'char key[1024]' inside the loop body — ASan caught it as stack-use-after-scope when subsequent cbm_ht_get calls dereferenced the dangling stack frame addresses. Fixed: strdup on insert, reuse cbm_ht_get_key when updating an existing bucket (no double-free), free all keys via cbm_ht_foreach before cbm_ht_free on cleanup. Benchmark on kubernetes (12 workers, 20,650 files): - parallel_resolve: 113,300ms (Tier 3 meta) -> 110,924ms (this commit) ~2.1% improvement — within run-to-run noise. Despite the O(N) -> O(1) algorithmic improvement, the (calls × resolved_calls) product on K8s Go files is small enough at this scale that the linear scan was not the dominant cost. Other call sites of cbm_pipeline_find_lsp_ resolution (e.g. pass_calls.c) keep the original O(N) path since this file-level index is local to resolve_file_calls. The change is still net positive: - Removes a real memory-safety bug found by ASan - O(N) -> O(1) algorithmic improvement at the call site, even if its scale impact is modest at K8s size - Sets up larger codebases (where the linear scan would dominate) to benefit without further work Tests: 3,616 / 0 failed including parallel_python_lsp_override_cross_ file_emits_lsp_strategy_edges (which directly exercises the LSP resolution path). Graph counts: nodes -1 vs baseline, edges +945 vs baseline (within 0.1%, semantic-preserving). zod/serilog sanity: nodes exact, edges +4/+5 (within tolerance). Real bottleneck for the remaining ~110s on K8s is elsewhere — likely the cumulative cost across 20K files of registry_resolve + gbuf_find_ by_qn + gbuf_insert_edge per resolved entry across all 5 resolve_file_ passes. Profiling needed to confirm; the lightweight per-worker edge buffer (deferring dedup to merge) is the next obvious attack vector. |
||
|
Martin Vogel
|
96f6891ff2 |
perf(lsp): Tier 3 metadata-driven cross-LSP — skip AST re-walk for Go
KEY ARCHITECTURAL INSIGHT (credit: user observation):
The per-file LSP during extract ALREADY captures the inferred receiver
type QN + method name for unresolved cross-file calls — emit_unresolved_
call at go_lsp.c:1242-1245 stores callee_qn as '<FullReceiverTypeQN>.<
MethodName>' in result->resolved_calls flagged with strategy='lsp_
unresolved'. The cross-LSP step was REDUNDANTLY re-deriving this same
metadata via a second tree-sitter parse + AST walk.
Cross-LSP for Go is now METADATA-DRIVEN:
cbm_go_fast_resolve_qualified_calls (rewritten):
- Iterates result->resolved_calls where strategy=='lsp_unresolved'
- For each entry: try direct registry lookup of callee_qn;
if that misses, split prefix.suffix and resolve prefix via the
file's import map then lookup '<module_qn>.<suffix>' in registry
- Emits resolved entry on top (cbm_pipeline_find_lsp_resolution
picks highest confidence)
- NO tree-sitter parse, NO AST walk, NO scope tracking
resolve_worker (pass_parallel.c):
case CBM_LANG_GO now calls ONLY the metadata-driven fast path.
cbm_run_go_lsp_cross_with_registry (the slow AST-walk version) is
no longer invoked for Go — it would just re-derive identical
metadata via a second AST walk and arrive at the same answers.
Benchmark (kubernetes, full fresh index, 12 workers):
- parallel_resolve: 1.89 min (Tier 3 meta) vs 1.86 min (Tier 2) — parity
- lsp_cross.slow lines: 0 (no tree-sitter fallback runs)
- Nodes: 297,081 vs 297,082 baseline (-1)
- Edges: 1,153,163 vs 1,153,482 baseline (-319, within 0.03%)
- Tests: 3,616 / 0 failures
- zod/serilog sanity: nodes exact, edges +1-3 (within 0.03%)
The wall-time parity is informative: the AST re-walk wasn't the
dominant cost. Per-file budget (66ms across 20,650 files / 12 workers)
is now dominated by the 5 sequential resolve passes (resolve_file_
calls/usages/throws/rw/semantic) pushing ~10M total edges into the
per-worker gbuf with arena allocs + hash-bucket inserts. The Tier 3
work eliminates a genuine architectural redundancy (two walks of the
same AST producing the same metadata) and sets up Python/C/etc to
adopt the same metadata-driven pattern in follow-up commits — but
hitting the user-target <30s K8s requires attacking the gbuf hot
path, not the cross-LSP path.
|
||
|
Martin Vogel
|
9552a7f9bf |
fix(pipeline): drop duplicate rel declaration in resolve_worker
When the genskip commit (
|
||
|
Martin Vogel
|
aecd86135f |
perf(lsp): skip cross-LSP for machine-generated files
Investigation of kubernetes' largest Go files revealed the perf tail
is dominated by machine-generated code:
pkg/generated/openapi/zz_generated.openapi.go 74,266 lines
staging/src/k8s.io/api/core/v1/generated.pb.go 71,154 lines
Multiple *.pb.go protobuf files 12-14K lines each
209 files in staging/.../client-go/applyconfigurations/
These files are boilerplate (getters/setters/marshal/unmarshal) with
limited semantic value for code-navigation queries. But because tree-
sitter parse of a 70K-line file alone is ~1-2 seconds, even one
unresolved cross-file call in such a file triggers the full cross-LSP
overhead and dominates the wall time.
Skip cross-LSP for files matching common generated-file conventions:
*.pb.go (Go protobuf)
zz_generated* (k8s codegen convention)
*_generated.go (general Go codegen)
*.gen.go (general Go codegen)
/applyconfigurations/ (k8s codegen subtree)
*_pb2.py, *_pb2_grpc.py (Python protobuf)
*.pb.cc, *.pb.h, *.pb-c.{c,h} (C/C++ protobuf)
The per-file LSP during extract still indexes their defs and calls
normally — only the cross-file resolution refinement is skipped.
Graph quality: zero loss for hand-written code; for generated code
cross-file method resolution falls back to whatever per-file LSP
captured (which is typically complete for intra-package boilerplate).
|
||
|
Martin Vogel
|
8ce2228834 |
perf(lsp): smart-prune cross-LSP + add Python Tier 2 wiring
Two changes layered:
1. Smart pruning of cross-LSP work — the cross-LSP work (re-parse +
AST walk + registry lookups, ~50-150ms per file) only adds value
if a file has unresolved cross-file method calls. If the per-file
LSP during extract already resolved every call
(result->resolved_calls.count >= result->calls.count) or the file
has zero calls at all, the AST has nothing left for cross-LSP to
improve. cross_lsp_eligible now additionally requires:
result->calls.count > 0
&& result->resolved_calls.count < result->calls.count
On kubernetes this skips 87% of files (17,949 of 20,650) with
zero semantic loss — graph node/edge counts within ±0.14% of the
non-pruned baseline.
2. Python Tier 2 wired through pipeline. The cbm_py_build_cross_registry
and cbm_run_py_lsp_cross_with_registry helpers from the previous
commit are now actually populated and dispatched. resolve_worker's
switch now handles CBM_LANG_PYTHON in addition to CBM_LANG_GO.
3. cbm_parallel_resolve's cross_registries param is now typed void*
in the header (cast back to CBMCrossLspRegistries* in the impl) —
the typedef CBMCrossLspRegistries lives in pass_lsp_cross.h which
transitively pulls in lsp/go_lsp.h. Dodging the typedef-vs-struct-
tag mismatch this way keeps pipeline_internal.h light (no LSP
header dependency).
Bench on kubernetes (mode=full, fresh DB):
- parallel_resolve: 1.86 min (Tier 2 Go) -> 2.06 min (+10% regression)
The slight regression on Go-heavy k8s comes from Python registry
build (lsp_cross_prepare: 911 -> 1,014 ms) and from the fact that
the 13% of files cross-LSP still runs on are the EXPENSIVE files
with many unresolved calls — pruning the 87% trivial files saves
less than the remaining 13% hard files cost.
- Tests: 3,616 / 0 failed. parallel_python_lsp_override_cross_file_
emits_lsp_strategy_edges still PASS.
- zod/serilog: node/edge counts within ±0.1% of known-good.
The smart-pruning + Python wiring is net-positive across the language
matrix (Python-heavy repos benefit from the new Tier 2 path) but
doesn't move the needle on Go-heavy k8s. The remaining ~2 min on k8s
is dominated by the AST re-walk inside cross-LSP — eliminating that
requires capturing unresolved-call metadata during extract so cross-
LSP is pure registry lookup (no re-parse). Follow-up commit.
|
||
|
Martin Vogel
|
23d404a056 |
perf(lsp): Tier 2 — pre-built per-language cross-LSP registries (Go + Python)
The gopls 'package summary' pattern, applied properly: build the cross-
LSP CBMTypeRegistry ONCE per language at pipeline time (in a dedicated
cross_lsp_arena), share it READ-ONLY across all resolve workers via the
CBMCrossLspRegistries struct. Per-file work then becomes just tree-
sitter parse + AST walk + O(1) hash lookups — zero per-file registry
rebuilds for languages with a pre-built registry.
Each LSP gains two new public functions:
CBMTypeRegistry *cbm_<lang>_build_cross_registry(arena, defs, def_count)
— filters all_defs to its language, init + stdlib + registration
loop + cbm_registry_finalize, returns finalized registry.
void cbm_run_<lang>_lsp_cross_with_registry(arena, source, ...,
CBMTypeRegistry *reg, ...)
— skips the per-file build AND the per-file Phase 1b/1c AST scans
(those mutate the registry, which would race on a shared read-
only reg). Trade-off: a few file-local type aliases discovered
only by Phase 1b are missed; in practice the per-file extract
LSP already captured them via embedded_types/field_defs/etc.
Tested with parallel_python_lsp_override_cross_file_emits_lsp_
strategy_edges — fused + filtered + prebuilt path stays correct.
CBMLSPDef gains a CBMLanguage lang field set in pxc_build_lsp_def so
the per-language build functions can filter the global all_defs[].
cbm_parallel_resolve signature now takes CBMCrossLspRegistries* instead
of a single CBMTypeRegistry*. The resolve_worker dispatches:
cbm_pxc_registry_for_lang(lang) returns the right pre-built reg if any;
the worker then switch()es to the matching _with_registry variant.
Languages without a _with_registry variant fall through to the existing
filter + per-file build path.
Kubernetes (Go-heavy, 12k files, 139k defs) measured perf:
parallel_resolve: 5.51 min (strdup-removal baseline) -> 1.86 min
= 2.96x speedup
parallel.resolve.lsp_cross.slow lines: 1,258 -> 2
(-99.8%; the 2 remaining are Python files, not Go)
Peak RSS: 6.29 GB -> 6.23 GB (slight decrease — pre-built registry
shared, no per-file scratch)
All 3,616 tests pass; zod/serilog graphs exact match (per-file path
unaffected).
Cumulative trajectory from the original broken state on kubernetes:
35 min -> 19 min (finalize) -> 6.24 min (module-def filter) ->
5.51 min (strdup-removal) -> 1.86 min (Tier 2 Go) = 18.8x total.
Languages wired so far: Go, Python. C/C++, TS/JS, PHP, C# still use the
per-file filter+build path until their _with_registry variants land in
follow-up commits.
|
||
|
Martin Vogel
|
75a0df688f |
feat(pipeline): fuse cross-LSP into resolve worker + module-def filter
Previously cbm_pipeline_pass_lsp_cross ran as a separate sequential pass that re-read every source file from disk and re-parsed every tree-sitter tree on a single thread — a 50x regression on the kubernetes Go monorepo (extract 10s, lsp_cross 9+ min when killed). Three changes: 1. CBMFileResult gains const char *source / int source_len. The parallel extract worker copies source bytes into result->arena (per-file 100MB cap, project-wide 2GB atomic cap) so the fused cross-LSP step can run without disk I/O. Source pointer is borrowed by downstream readers; arena owns the storage. 2. Cross-LSP work moves into the parallel resolve worker. Each worker processes its file with the existing parallel infrastructure (worker pool + work stealing). Slab is reclaimed after each per-file cross-LSP call to bound peak slab memory. The previously separate lsp_cross pass entry function is kept for the sequential pipeline path; the parallel path no longer invokes it. 3. CBMModuleDefIndex (gopls 'package summary' pattern) — an inverted index module_qn -> list of def indices built once in O(D) before resolve_worker fires. Per file, cbm_pxc_filter_defs_for_file slices all_defs down to the file's own_module + imported modules only, typically 50-100x smaller than the global ~110k defs on kubernetes. Drops the per-file registry build from O(all_defs) to O(relevant). Other plumbing: cbm_parallel_resolve signature gains all_defs, def_count, def_modules, module_def_index params. pipeline.c builds the index between registry_build and parallel_resolve and frees it after. pipeline_incremental and tests/test_parallel updated to match. Helpers in pass_lsp_cross.c exposed via the header: cbm_pxc_has_cross_lsp, cbm_pxc_collect_all_defs, cbm_pxc_run_one(_ts), cbm_pxc_ts_modes, cbm_pxc_build_module_def_index, cbm_pxc_filter_defs_for_file. Kubernetes wall-time: 35 min -> 6.24 min (~5.6x) before the followup perf commits in this series. |
||
|
Martin Vogel
|
c29e6d51f4 |
fix(hooks): replace blocking Claude PreToolUse gate with non-blocking augmenter
The previous PreToolUse hook gated Grep/Glob/Read/Search with 'exit 2' on the first call per session, which broke Claude Code's read-before-edit invariant (issue #362) and could deny tool calls under upgrade/missing-binary failure modes. Replace it with a structurally non-blocking augmenter: - New 'codebase-memory-mcp hook-augment' subcommand reads the hook JSON from stdin and, for Grep/Glob, queries search_graph (in-process, no shell) and emits hookSpecificOutput.additionalContext. Every failure path (no project, short token, missing binary, slow query, timeout) exits 0 with no stdout — the hook physically cannot block a tool call. - 300 ms SIGALRM/_exit(0) in-process deadline; 5 s settings.json timeout backstop. Output is written exactly once at the very end, so a mid-work timeout yields a clean no-op (never partial JSON). - Matcher narrowed to 'Grep|Glob' (Read explicitly excluded) for Claude; Gemini matcher narrowed to 'google_search|grep_search' (excludes read_file) for the same reason. - The installed shim is a thin wrapper that delegates to the binary; legacy filename 'cbm-code-discovery-gate' is kept so existing settings.json entries upgrade with zero migration. Installer refuses to embed binary paths containing a double quote (shim injection defense). - Per-agent 'old matchers' lists let upsert/remove clean up historical matcher strings during upgrade. - Smoke tests (8d/8e/8l) updated to assert the new behavior and regress-test against re-introducing Read in the matcher or 'exit 2' in the shim. - Session reminder text updated: 'always Read a file before editing it' replaces the prior 'fall back to Read only for text content'. (cherry picked from commit f72c8e68c4d91e52911a569a967ad782ce5472b2) |
||
|
Martin Vogel
|
32bb56ef3f |
fix(mcp): force grep filename prefix in scoped search_code
build_grep_cmd's POSIX scoped path uses xargs to pass indexed files to grep. When xargs supplies a single file, BSD/GNU grep omit the filename from output by default (only added with multiple files), yielding 'linenum:content' instead of the 'path:linenum:content' that collect_grep_matches requires — matches were silently discarded and search_code returned zero results. Add -H so grep always prefixes the filename regardless of file count. Harmless for the common multi-file case (output is unchanged), fixes the single-indexed-file edge case exposed by the search_code_multi_word test fixture. |
||
|
edwardmhughes
|
22153563cd |
fix(store): use PASSIVE checkpoint to avoid file-shrink under concurrent readers
cbm_store_checkpoint() invoked SQLITE_CHECKPOINT_TRUNCATE, the most aggressive mode. When two cbm-mcp processes share a cache dir, one process's TRUNCATE can shrink files while another has them mmap'd, raising SIGBUS on macOS. PASSIVE never blocks readers and never ftruncate()s either file; SQLite still autocheckpoints in PASSIVE mode at 1000-page boundaries, so reclamation is unaffected for single-process users. Recommended by SQLite docs for shared databases: https://www.sqlite.org/pragma.html#pragma_wal_checkpoint |
||
|
edwardmhughes
|
093707cda2 |
feat(store): expose mmap_size via CBM_SQLITE_MMAP_SIZE env
Hard-coded `PRAGMA mmap_size = 67108864` in configure_pragmas() left no path for users running multiple cbm-mcp instances against the same cache to opt out of memory-mapped I/O. On macOS, when one instance's checkpoint or reindex truncates the DB file under another instance's live mmap, accessing the now-missing pages raises SIGBUS, taking the process down. Setting CBM_SQLITE_MMAP_SIZE=0 reverts to read()/pread() I/O, which returns recoverable SQLITE_IOERR instead of crashing the process. - Default unchanged (67108864 / 64 MB). No behavior change for single-instance users. - Malformed values (non-numeric, partial-numeric) fall back to the default rather than failing the store open. - Negative values clamp to 0. - New tests: tests/test_store_pragmas.c covers all five resolver paths plus an integration smoke that opens a file-backed store with mmap disabled. Empirical evidence: 9 SIGBUS crash reports collected on macOS arm64 v0.6.0 in a 14-hour window, all signature 'cluster_pagein past EOF' with stacks bottoming in SQLite btree code under the watcher thread's incremental-index pipeline. |
||
|
Christof Meerwald
|
a338ff3626 | Add support for NetBSD, FreeBSD, and OpenBSD. | ||
|
George
|
d83553a1ba |
fix(store.c): improve root path integrity check with range-based validation
Replace the fragile NOT IN list of individual characters with a range-based BETWEEN 'A' AND 'Z' check for validating project root paths. The previous query only excluded letters A–H, leaving paths starting with I–Z silently invalid. In Windows, all the alphabet can be used. A Windows drive starting with letters I-Z breaks the saving of the database. |
||
|
Martin Vogel
|
eca433bd58 |
fix(security): block " < > in cbm_validate_shell_arg
The Windows search path landed in
|
||
|
noctrex
|
82a905284c | feat(mcp): add Windows support for code search via PowerShell | ||
|
KuaaMU
|
1a4973d01c |
fix: add .m extension to EXT_TABLE for content-based disambiguation
The .m extension was missing from EXT_TABLE, causing detect_file_language
to return CBM_LANG_COUNT (Unknown) before the content-based disambiguation
logic (cbm_disambiguate_m) could ever run. This made .m files invisible
to the indexer.
Add {".m", CBM_LANG_MATLAB} to EXT_TABLE so the initial lookup succeeds,
allowing the subsequent cbm_disambiguate_m call to correctly determine
whether the file is MATLAB, Objective-C, or Magma.
Fixes #296
|
||
|
Jan Deelstra
|
cc7ef348df |
fix(mcp): search_code converts multi-word patterns to regex
When pattern contains spaces and is not already a regex, convert to a regex with '.*' between words: "foo bar baz" → "foo.*bar.*baz". This matches all terms in order on a line instead of requiring the exact multi-word substring. Metacharacters in user input are escaped so the conversion is safe for arbitrary text. Includes test: search_code_multi_word verifies that a two-word query matches a line containing both words non-contiguously. |
||
|
Austen ConstableandClaude Sonnet 4.6
|
5f19454724 |
Fix search_graph query= multi-minute latency: two-step FTS5 subquery
Flat BM25 queries of the form: SELECT ... FROM nodes_fts JOIN nodes WHERE MATCH ? AND project=? ORDER BY bm25() LIMIT N block FTS5 WAND/MaxScore early-exit — the outer JOIN+WHERE is invisible to the FTS5 planner, so it scores every matching document before any filter fires. On a large codebase with 100K+ matches this causes 2–16 minute queries. Fix: two-step subquery. The inner FTS5-only query: SELECT rowid, bm25(nodes_fts) FROM nodes_fts WHERE MATCH ? ORDER BY bm25() LIMIT 2000 can early-terminate because no outer predicate blocks it. The outer query then joins and filters at most BM25_INNER_LIMIT (2000) candidates. The count query uses the identical inner-limit subquery, so it benefits too. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Austen ConstableandClaude Sonnet 4.6
|
dd0ce4981e |
Fix search_graph name_pattern= performance: regex cache, LIKE pre-filter, cheap count
Three compounding bugs caused 1.5–8.5s latency on name_pattern= searches against large projects (216K nodes), now reduced to ~0ms query time (cold-start dominates): Fix 1 — regex compiled once per statement, not once per row sqlite_regexp / sqlite_iregexp now use sqlite3_get_auxdata / sqlite3_set_auxdata to cache the compiled cbm_regex_t for the lifetime of the statement. Previously cbm_regcomp + cbm_regfree ran for every row scanned. Fix 2 — LIKE pre-filter cuts rows reaching the regex Wire cbm_extract_like_hints (already implemented but dead) into search_where_basic via a new where_add_like_hints helper. For .*Controller.* this prepends n.name LIKE '%Controller%', letting the idx_nodes_name index satisfy the LIKE clause first and passing only matching rows to iregexp(). Added search_like_pool_t to manage the malloc'd LIKE strings across both statement executions. ST_SEARCH_MAX_BINDS raised 16 → 32 to accommodate extra bind slots. Fix 3 — count query no longer runs per-row edge subqueries The count SQL previously wrapped the full SELECT (which includes two correlated subqueries for in_deg / out_deg) in SELECT COUNT(*) FROM (...), executing those edge counts for every matching row even though the count needs none of that. Non-degree-filter path now uses SELECT COUNT(*) FROM nodes n WHERE <same WHERE>, which has no per-row subqueries. Degree-filter path retains the wrapped form since it needs those columns for the filter. Benchmark on home-ubuntu-dev-sis (216K nodes, 509MB DB): Query BEFORE AFTER speedup name_pattern=.*Controller.* 3099ms 508ms 6× name_pattern=.*Service.* 2006ms 506ms 4× name_pattern=.*Repository.* 2006ms 508ms 4× name_pattern=specificFuncName 1506ms 507ms 3× label=Method + name_pattern=.*get.* 8509ms 509ms 17× name_pattern=.*Approve.* 1506ms 507ms 3× name_pattern=.*authorize.* 1506ms 509ms 3× The ~500ms floor is cold-start I/O (opening a 509MB file from disk). In the long-running MCP server process the warm-cache query time is sub-millisecond. All store search tests pass including pagination, degree filter, and extract_like_hints. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Martin VogelandJustin Wiley
|
9818730056 |
fix(mcp): trace_path falls back to qualified_name lookup
handle_trace_call_path only called cbm_store_find_nodes_by_name, which
queries on the bare `name` column. When a caller passed a fully
qualified identifier (e.g. "<project>.module.submodule.func") — which
the not-found hint explicitly recommends ("Use search_graph(...) to
find the exact name, then pass it to trace_path") — the lookup missed
every time. The documented "fully qualify for precise match" UX
inverted what actually worked.
After the bare lookup misses, fall back to cbm_store_find_node_by_qn
(same scan_node populator, same string-ownership semantics), wrap the
single result into the same nodes[] array the downstream BFS expects,
and let cbm_store_free_nodes handle cleanup unchanged. Guard the malloc
against OOM and free the qn_node strdups on the failure path so the
fallback never leaks.
Co-authored-by: Justin Wiley <jwiley@nvidia.com>
|
||
|
Martin VogelandJustin Wiley
|
eb0627e303 |
fix(mcp): list_projects no longer hides tmp-prefixed projects
`is_project_db_file` excluded any filename starting with "tmp-", but `cbm_project_name_from_path` legitimately produces "tmp-..." project names for source roots under /tmp/ (the in-tree fixture confirms: "/tmp/bench/erlang/lib/stdlib/src" → "tmp-bench-erlang-lib-stdlib-src"). Any /tmp-rooted project was silently filtered from list_projects and from the available_projects hint inside build_project_list_error, even though the same project was fully queryable via every other tool. Drop the tmp- prefix exclusion; SQLite journal/WAL sidecars end with -journal/-wal/-shm and don't share the .db suffix the existing length check already requires, so removing the prefix filter is safe. While here, dedupe the same exclusion logic from collect_db_project_names by routing it through is_project_db_file (forward-declared above) so the filter lives in one place and the two paths can't drift again. Co-authored-by: Justin Wiley <jwiley@nvidia.com> |
||
|
Martin Vogelandsponger94
|
7290c1fd2b |
feat(ui): satellite galaxy radius spacing + cross-galaxy edge rendering
Two visualization gaps in the cross-repo graph view: 1. Fixed LAYOUT_GALAXY_SPACING=600 buried satellite galaxies inside the primary cluster on any non-trivial project — a 1000-node primary cluster has bounding radius ~1500. Added layout_radius() (max distance from origin across all node positions) and compute the per-satellite distance as primary_radius + sat_radius + LAYOUT_GALAXY_PAD, with the old constant retained as a lower bound for tiny projects. 2. cross_edges was hard-coded to an empty array (TODO placeholder), so inter-galaxy CROSS_* edges never rendered even when the matcher had produced them. Populate from a join on the source store's CROSS_* edges → Route node qualified_name, then resolve the canonical QN in the linked store to get its node id. The Route QN is the cross-repo matching contract; properties.target_function isn't unique. Required keeping both `store` and `lp_store` open through the linked- projects loop instead of closing them right after layout compute. Added cbm_store_close to all early-error paths to balance the lifetime. Frontend side: - EdgeLines accepts an optional targetNodes prop so a single component can render edges with source in one node array and target in another (offset-adjusted satellite nodes). - GraphScene renders an EdgeLines layer per linked project for its cross_edges, using primary nodes as source and offsetNodes as target. - GraphTab folds linked-project labels and edge types into the filter init / enableAll / filteredData paths so cross_edges respect the current filter state. - Edge color palette extended with GRPC/GRAPHQL/TRPC/CROSS_* tones. Co-authored-by: sponger94 <45746997+sponger94@users.noreply.github.com> |
||
|
Martin Vogel
|
b2a71b80b9 |
feat(pipeline): wire cross-file LSP across all five hybrid resolvers
Per-file LSP (run from cbm_extract_file) only sees one file's defs, so
callees whose receiver type comes from an imported module stayed
unresolved — even though Go/C/Py/TS already had cross-file entry points
(cbm_run_X_lsp_cross). Those entry points existed but no pipeline pass
ever invoked them. PHP didn't even have a cross-file entry point.
This change lands the missing pieces:
1. cbm_run_php_lsp_cross + cbm_batch_php_lsp_cross — PHP cross-file
resolver mirroring py_lsp_cross / ts_lsp_cross. Reuses PHPLSPContext,
php_lsp_init, php_lsp_add_use, php_lsp_process_file. Handles PHP-
specific labels (Class / Interface / Trait / Enum / Type) and uses
def_module_qn so cross-file return types qualify against the def's
own module rather than the importing module's.
2. pass_lsp_cross.c — new pipeline pass. For every cached file result,
builds a project-wide CBMLSPDef[] (CBMDefinition → CBMLSPDef
converter), per-file import map (from gbuf IMPORTS edges), then
dispatches to the matching cbm_run_X_lsp_cross. Output strings get
copied into the per-file arena (with O(N²) memory bug avoided by
running each file inside a fresh scratch arena that gets destroyed
after the LSP call — the LSP allocates its registry + stdlib +
project defs in scratch, not in the caller's persistent arena).
3. Wired into both pipelines:
- sequential: definitions → k8s → lsp_cross → calls → ...
- parallel: parallel_extract → registry_build → lsp_cross →
parallel_resolve → ...
4. Updated tests/test_parallel.c run_parallel helper to mirror the new
parallel ordering so existing cross-file Python/PHP tests now exercise
pass_lsp_cross.
Languages covered: Go, C/C++/CUDA, Python, TS/JS/JSX/TSX, PHP. PHP cross
file is new. The other four had functions but no pipeline integration
until this change.
Verified on a 1670-file FastAPI index: pass_lsp_cross processes 1050
Python files (525 non-LSP files skipped), collects 4070 project defs in
9.2s, and parallel_resolve subsequently reports lsp_overrides=568 — the
LSP-strategy attribution gain over per-file alone.
Per-LSP emit functions dedup by (caller_qn, callee_qn) within their own
run; pxc_append_results dedups again at append time so cross-file
entries duplicating per-file output get dropped instead of inflating
result->resolved_calls.
|
||
|
Martin Vogel
|
673ac4ef8a |
fix(pipeline): resolve LSP cross-file QN mismatch via project-prefix fallback
Per-file py_lsp emits resolved_calls.callee_qn as the raw import-module path (e.g. `greeter.Greeter` from `from greeter import Greeter`) rather than the project-qualified QN the gbuf stores (`<project>.greeter.Greeter`). Before this change, the LSP match succeeded (lsp_overrides counter incremented) but the downstream cbm_gbuf_find_by_qn lookup missed silently and the edge was dropped — the failure mode the new lsp_overrides telemetry was designed to surface. The LSP layer can't tell in-project imports (qualify) from external imports (don't qualify, e.g. `os.path`) without consulting the gbuf, which is built downstream. So normalise at the consumer instead: try the LSP-emitted QN as-is first; on miss, retry with `<project>.<callee_qn>`. If that also misses, drop the edge — same as before, target is external/unindexed. New helper cbm_pipeline_lsp_target_node in lsp_resolve.h is shared by both pipelines (sequential pass_calls.c and parallel pass_parallel.c) so they continue to admit identical sets of LSP overrides. The helper also canonicalises res.qualified_name to the gbuf node's QN, so downstream edge property serialisation shows the project-qualified form even when fallback resolution kicked in. Cross-file regression test parallel_python_lsp_override_cross_file_- emits_lsp_strategy_edges pins the two-file Greeter scenario that originally exposed the bug. |
||
|
Martin Vogel
|
085b4a718c |
fix(pipeline): unify LSP-override resolver across sequential and parallel paths
The sequential pass_calls and the parallel cbm_parallel_extract pipelines each carried their own copy of the LSP-resolved-call lookup with divergent confidence floors and different match semantics. Most production indexing runs the parallel path, so projects produced different CALLS-edge attributions depending on which pipeline mode kicked in: pass_calls.c find_lsp_resolution floor 0.6 (CBM_LSP_CONFIDENCE_FLOOR) pass_parallel lsp_override_resolution_pp floor 0.5 (hardcoded) Both pipelines correctly invoke cbm_run_*_lsp through cbm_extract_file, so result->resolved_calls is populated identically — the bug was at the consumer. Changes ------- - New src/pipeline/lsp_resolve.h: static-inline cbm_pipeline_find_lsp_resolution and CBM_LSP_CONFIDENCE_FLOOR (0.6f). Single source of truth for the match rule (caller_qn equality + callee_qn short-name match + floor), with rationale and language-coverage notes inline. - pass_calls.c: include lsp_resolve.h, drop the local copy, rename the call site to cbm_pipeline_find_lsp_resolution. - pass_parallel.c: include lsp_resolve.h, drop lsp_override_resolution_pp, use cbm_pipeline_find_lsp_resolution at resolve_file_calls. Add an int lsp_overrides counter to resolve_worker_state_t (with adjusted cache-line padding for the new field), aggregate it per-worker, and emit it in the parallel.resolve.done log line so divergence between pipelines becomes observable in production telemetry. - tests/test_parallel.c: parallel_python_lsp_override_emits_lsp_strategy_edges builds a single-file Python scenario (Greeter class + main() in one file), runs the parallel pipeline, walks every CALLS edge, asserts at least one carries an "lsp_*" strategy in its properties_json. Pins both halves of the wiring at once: the shared find helper IS consulted in the parallel path, AND the LSP-attributed strategy survives all the way to the gbuf edge. Single-file rather than cross-file because per-file py_lsp's resolved_calls callee_qn uses the import module path rather than the project-qualified QN, so cross-file LSP edges currently get dropped by cbm_gbuf_find_by_qn — a separate cross-file QN-mismatch bug to track elsewhere. Full suite: 3250 passed, 0 failed (was 3249; +1 new regression test). |
||
|
Martin Vogel
|
e6f4aa74bd |
Merge origin/main into worktree-php-lsp-integration
Resolved 5 conflicts where the php-lsp branch and main both added LSP
infrastructure but with different APIs:
- Makefile.cbm: kept both TEST_PHP_LSP_SRCS and main's py_lsp/scope/
type_rep/stack_overflow test source variables in ALL_TEST_SRCS.
- internal/cbm/cbm.c: kept both #include lsp/php_lsp.h and lsp/py_lsp.h;
the per-language dispatch already supports stacking PHP after PYTHON.
- internal/cbm/lsp_all.c: kept both lsp/php_lsp.c + php_stdlib_data.c
AND lsp/py_lsp.c + python_stdlib_data.c in the compilation unit.
- src/pipeline/pass_calls.c: took main's version entirely.
find_lsp_resolution + resolve_single_call(ctx, call, lsp_calls, ...)
is language-agnostic — it matches any CBMResolvedCall whose caller_qn
hits the textual call's enclosing function and whose callee_qn
short-name hits the textual callee. PHP LSP populates the same
result->resolved_calls array via cbm_run_php_lsp, so the unified
resolver handles PHP for free. Dropped the php-branch's
qn_short_name + lsp_override_resolution helpers as superseded.
Locked confidence floor at main's CBM_LSP_CONFIDENCE_FLOOR = 0.6f
(was 0.5f on the php branch).
- tests/test_main.c: kept all suite externs and RUN_SUITE invocations
for php_lsp + py_lsp/py_lsp_bench/py_lsp_stress/py_lsp_scale +
scope + type_rep.
Auto-merged: src/pipeline/pass_parallel.c.
Brings in 52 commits from main since the branch was forked at
|
||
|
Martin Vogel
|
4d9f62f7e6 |
Merge origin/main into worktree-python-lsp-integration
Resolved conflict in Makefile.cbm: keep both TEST_STACK_OVERFLOW_SRCS
(from main, #217) and the new py_lsp test variables (TEST_SCOPE_SRCS,
TEST_TYPE_REP_SRCS, TEST_PY_LSP_SRCS, TEST_PY_LSP_BENCH_SRCS,
TEST_PY_LSP_STRESS_SRCS, TEST_PY_LSP_SCALE_SRCS) in ALL_TEST_SRCS.
Other auto-merged files: internal/cbm/extract_defs.c (PR #279),
tests/test_main.c (multiple suite registrations on each side).
Brings in 28 commits from main since the branch was forked at
|
||
|
Martin Vogel
|
27e640b3b0 |
fix(mcp,ci): harden get_architecture serialization + CodeQL gate
Follow-up to #281 (handle_get_architecture) plus a CodeQL workflow upgrade developed in parallel; bundling into one commit because the test suite had to land alongside both. mcp.c — handle_get_architecture - NULL-coerce every const-char* field in the architecture sections via `x ? x : ""`, matching the rest of mcp.c (search_graph, etc.). Without this, a NULL field becomes a missing JSON key instead of an empty string; yyjson_mut_obj_add_str returns false on NULL and silently no-ops, so an inconsistent omission could surprise callers. - Serialize two more architecture aspects that #281 left on the floor: services (cbm_service_link_t: from/to/type/count) and clusters (cbm_cluster_info_t: id/label/members/cohesion plus the top_nodes / packages / edge_types string arrays). The store-side computation populates these for aspects=["all"] / explicit names, so dropping them in the serializer was data loss. tests/test_mcp.c - New tool_get_architecture_emits_populated_sections regression test. Uses a minimal inline fixture (single Function node tagged with "is_entry_point": true) since arch_entry_points reads that flag out of properties_json. Asserts the response contains both an "entry_points" array and the function name — neither would appear before #281 because handle_get_architecture never called cbm_store_get_architecture. - extract_text_content drilled too shallow: it pulled "content" only from the JSON root, so it worked for cbm_mcp_handle_tool but silently fell through to the raw response for cbm_mcp_server_handle (where content lives under .result.content). Added a fallback that checks .result.content; both unwrappers tested by existing fixtures. ci(codeql) - Run on pull_request to main, not just push; surfaces findings on the PR instead of after merge. - Pull custom queries from ./codeql via `queries: +./codeql`. - Capture SARIF output and fail the job on any error-level finding, using jq to enumerate rule id, file:line, and message text in the GitHub Actions error annotation. Warnings are still reported as before; only errors block. Full suite: 2842 passed, 0 failed. |
||
|
Oliver Evans
|
ca5e309062 |
fix(mcp): wire cbm_store_get_architecture into handle_get_architecture (#281)
handle_get_architecture was returning only label/edge histograms because it never called cbm_store_get_architecture — the rich architecture computation sat completely unwired upstream of the MCP boundary regardless of which aspects were requested. Wires it up: - Parse the JSON aspects array into a const char*[MCP_COL_16] and pass to cbm_store_get_architecture, which already handles 'all' and per-name filtering internally. - Serialize each non-empty section: languages, packages, entry_points, routes, hotspots, boundaries, layers, file_tree. Existing histogram fields (node_labels, edge_types, relationship_patterns, cross_repo_links) preserved. - Call cbm_store_architecture_free before return. Closes #280. |
||
|
vinay-veerappa
|
048f1b46c9 |
feat(lang): add Pine Script support via vendored kvarenzn/tree-sitter-pine (#273)
Adds Pine Script (TradingView's indicator/strategy DSL) as a first-class language using the project's existing language-add pipeline: - scripts/new-languages.json: register pine (display "PineScript", ts_func tree_sitter_pine, repo kvarenzn/tree-sitter-pine, extension .pine, has_scanner=true, module_root source_file) - internal/cbm/vendored/grammars/pine/: vendored from https://github.com/kvarenzn/tree-sitter-pine via scripts/vendor-grammar.sh — parser.c, scanner.c, tree_sitter/ headers. License is ISC (declared in upstream package.json; written into vendored LICENSE for clarity since upstream has no LICENSE file) - internal/cbm/grammar_pine.c: 4-line wrapper generated by scripts/generate-lang-code.py wrappers - internal/cbm/cbm.h: CBM_LANG_PINE before CBM_LANG_COUNT - internal/cbm/lang_specs.c: 7 node-type arrays (func / class / module / call / var / branch / assign) cross-checked against kvarenzn's grammar.js, plus the spec table entry - src/discover/language.c: .pine -> CBM_LANG_PINE extension dispatch and "PineScript" name scripts/audit-grammar-security.sh on the vendored grammar: no dangerous includes (only string/stdint/stdbool/stdlib), no system / popen / exec / socket / mmap / dlopen calls, no constructor/destructor attributes, no inline asm. The 148-line scanner.c only handles INDENT/DEDENT/NEWLINE tokens via the standard tree-sitter lexer API. Distilled from #273 rather than vendored verbatim because the original PR's parser.c was a plain 46k-line drop without provenance, no LICENSE attribution, and bundled an unrelated Makefile change (scripts/embed-frontend.sh -> sh scripts/embed-frontend.sh) that didn't belong to a Pine Script feature. The node-type choices in the lang spec match the original PR's selections, all of which were verified to be real grammar.js rules. Full suite: 2840 passed, 0 failed under ASan + UBSan. Closes #273. Co-authored-by: vinay-veerappa <vinay.veerappa@gmail.com> |
||
|
Adam Schulte
|
4e80d2fff7 |
feat(graph): temporal properties on FILE_CHANGES_WITH edges and File nodes (#257)
Adds three changes that landed together in PR #257, with a JSON-escape hardening on the new File node properties so unusual file extensions can't produce malformed output. Mode A — search_graph results expose node properties_json --------------------------------------------------------- emit_search_results in src/mcp/mcp.c now calls enrich_node_properties for each result and parks the parsed yyjson_doc* in an out-array. The two exit paths in handle_search_graph (success and sq_type_error) walk that array and yyjson_doc_free each entry after yy_doc_to_str runs, so zero-copy strings stay alive until serialization completes. Callers no longer have to round-trip through get_code_snippet just to inspect a node's properties. Mode B — Cypher bare edge return surfaces full properties JSON -------------------------------------------------------------- binding_get_virtual in src/cypher/cypher.c returns e->properties_json (or "{}" if absent) when prop is NULL, instead of "". `RETURN r` on an edge variable is now actually useful for inspecting timestamps, weights, etc. without naming each property up front. Temporal data on FILE_CHANGES_WITH edges and File nodes ------------------------------------------------------- cbm_change_coupling_t gains last_co_change. cbm_commit_files_t and the internal commit_t gain a timestamp field. Both git-log paths (libgit2 and popen --pretty=format:COMMIT:%H:%ct) populate it. cbm_compute_change_coupling carries a parallel pair_timestamps hash table and writes the max timestamp to last_co_change. The edge properties JSON now includes "last_co_change":<unix-epoch>. cbm_pipeline_githistory_compute additionally aggregates per-file change_count and last_modified into a new cbm_file_temporal_t array exposed via result->file_temporal. cbm_pipeline_githistory_apply upserts those values onto matching File nodes so callers can run hotspot queries (most-changed, recently-changed) directly. Both the sequential cbm_pipeline_pass_githistory and the parallel run_githistory in pipeline.c free file_temporal and trigger apply when either couplings or temporal entries are present. The extension stored on the File node properties is now JSON-escaped via cbm_json_escape before snprintf'ing into the props blob. The original PR concatenated the raw extension; a hypothetical path with a quote or backslash in the extension would have produced malformed JSON. Practical risk was tiny (real extensions never contain those characters) but the escape is cheap and correct. Tests ----- - githistory_coupling_carries_last_co_change pins last_co_change as the max of the per-pair commit timestamps even when an unrelated pair sits between them in the input order. - cypher_bare_edge_return_exposes_properties_json regresses Mode B by asserting `RETURN r` on an HTTP_CALLS edge surfaces its url_path in the response. - tool_search_graph_includes_node_properties regresses Mode A by asserting search_graph's response contains the signature/return_type/ is_exported keys from a HandleRequest node's properties_json. Existing githistory tests had their cbm_commit_files_t initializers extended with a zero timestamp (backward-compatible — last_co_change falls back to 0 for callers that don't supply timestamps). Cherry-picked from #257; renamed the timestamp field types to long long for portability across 32-/64-bit time_t platforms, used the existing cbm_json_escape utility instead of raw concatenation for the extension, and added explicit tests for all three changes. Closes #257. Co-authored-by: Adam Schulte <adamschulte148@gmail.com> |
||
|
Peter Cox
|
cee318bb7f |
fix(incremental): preserve mode-skipped files and their hash rows (#251)
cbm_pipeline_run_incremental was destructively overwriting a project's graph when the index mode was downgraded. find_deleted_files classified ANY stored file absent from the current discovery list as deleted and purged its nodes via cbm_gbuf_delete_by_file. In a full → fast mode downgrade, directories matching FAST_SKIP_DIRS (tools, scripts, bin, build, docs, __tests__, migrations, ...) are excluded from discovery but still exist on disk — so they were getting silently destroyed. Real-world incident, 2026-04-13: the Skyline graph collapsed from 2819 nodes (full-mode nightly) to 2303 mid-day when a concurrent fast-mode reindex obliterated packages/mcp/src/tools/ (~500 nodes / 18 files). Fix: - find_deleted_files now stat()s each stored-but-missing file. Files that exist on disk are mode-skipped (nodes AND hash rows preserved); only ENOENT/ENOTDIR results purge. Other errnos (EACCES, EIO, ELOOP, transient NFS, ...) preserve. NULL repo_path preserves. snprintf truncation preserves. strdup OOM mid-record drops the entry rather than persist a NULL rel_path that would silently fail upsert. stat() not lstat() is intentional — the indexer follows symlinks during discovery so a stale symlink has no source to parse. - Mode-skipped hash rows are threaded through dump_and_persist → persist_hashes and re-upserted into the rebuilt DB. Without this, the next reindex's stored hashes wouldn't include the file → the deletion-detection contract breaks → orphaned graph nodes survive forever once the file is eventually deleted. - persist_hashes now checks the upsert return code on both the current-files loop and the new mode_skipped loop and logs a warning with scope/rel_path/rc on failure. Partial-failure policy: a single bad row no longer silently drops, but does not abort the whole reindex either — partial preservation beats total loss. Tests: - incremental_fast_preserves_mode_skipped_tools_dir walks the failure end-to-end across four phases: full index, fast reindex preserves the tools dir, mutate-and-fast-reindex exercises dump_and_persist's hash carry (NOT the noop fast-path that would hide bugs there), then delete-on-disk full reindex confirms the deletion-detection contract still holds. - store_file_hash_upsert_rejects_null_required_fields pins the API contract that cbm_store_upsert_file_hash returns CBM_STORE_ERR on NULL required fields, so a future schema change relaxing any NOT NULL is caught at the producer layer rather than letting the new warning go silent. Conflict resolution against current main: the dump_and_persist signature now takes both mode_skipped/mode_skipped_count (this PR) and repo_path (added on main for cbm_artifact_export); the call site passes cbm_pipeline_repo_path(p) alongside the mode_skipped pair. Full suite: 2837 passed, 0 failed under ASan + UBSan + LeakSanitizer. Closes #251. Co-authored-by: Peter Cox <info@petercox.ie> |
||
|
Martin Vogel
|
443ad5c93c |
feat(pipeline): consume LSP-resolved calls in pass_calls
Phase 9.5 of Python LSP integration. Cross-language pipeline change benefiting Go, C/C++, and Python LSPs simultaneously. resolve_single_call now checks result->resolved_calls first. When the LSP populated an entry whose caller_qn matches the textual call's enclosing function, whose callee_qn ends with the textual callee's short name, and whose confidence meets CBM_LSP_CONFIDENCE_FLOOR (0.6), the resolver emits the edge using the LSP-resolved callee QN and strategy. Otherwise it falls through to the existing cbm_registry_resolve textual matcher unchanged. The floor of 0.6 lets every standard LSP strategy through: - lsp_direct (0.95) - lsp_method (0.9) - lsp_module_attr (0.92) - lsp_super (0.88) - lsp_constructor (0.85) - lsp_interface_resolve (Go LSP, ~0.85) while filtering best-effort guesses such as lsp_module_attr_unresolved (0.55) — those fall back to the registry. Highest-confidence wins when multiple LSP entries exist for the same (caller, short_callee) pair. The floor is fixed in v1; promote to an env knob if telemetry shows tuning is necessary. All 2866 prior tests stay green; no LSP / pipeline / integration / Go / C++ / Python regressions. |
||
|
Peter Cox
|
10a9b83e27 |
fix(pipeline): resolve build-tool path aliases for cross-file imports (#243)
CALLS edges for cross-file function calls were mis-wired or missing in
TypeScript projects that use path aliases (e.g. @/lib/auth → src/lib/auth):
1. resolve_relative_import only handled "./relative" paths, so
"@/lib/auth"-style imports never reached IMPORTS edges. The import
map stayed empty, Strategy 1 in cbm_registry_resolve failed, and
resolution fell through to name-based lookups that pick the wrong
target when functions share names across files.
2. resolve_import_map built the candidate QN as the bare module QN
when the callee had no dot ("requireAdmin" vs "pkg.Func"), instead
of module_qn.requireAdmin. Even with a populated import map, bare
calls still mis-resolved.
Fix:
- New src/pipeline/path_alias.{h,c}: language-agnostic build-tool
path-alias subsystem. Currently parses tsconfig.json and
jsconfig.json compilerOptions.paths + baseUrl via the existing
yyjson dependency (with ALLOW_COMMENTS|ALLOW_TRAILING_COMMAS for
JSONC). The data model (cbm_path_alias_t / map / scope /
collection) is generic; adding Vite/Webpack/Python loaders means
writing a new file-parsing helper and registering it in
find_alias_files — the resolver, collection, and pipeline
integration do not change.
- O(n log n) ordering: scope-by-dir-specificity and entry-by-
prefix-specificity are sorted with qsort, replacing the O(n²)
bubble sorts that the original draft used. With caps of 256
entries per config and 256 configs per repo, the sort is
effectively flat.
- Resource ceilings with logging: directory recursion capped at
32 levels; total config files capped at 256; per-config size
capped at 64 KB. Cap hits emit cbm_log_warn so silent truncation
on pathological monorepos shows up in the index log.
- Walk hygiene: skips node_modules, .git, dist, build, .next,
coverage, target.
- Single integration point: cbm_pipeline_resolve_module gets a
Strategy 1b between relative-import resolution and pkgmap so both
the sequential and parallel pipelines pick up alias resolution
automatically. Non-TS projects pay nothing — the loader returns
NULL when no usable configs are found.
- Layer 2 fix in registry.c: resolve_import_map now builds the
candidate as resolved.prefix when suffix is NULL (one-line fix,
matching comment).
- Tests: 10 path_alias tests (basic wildcard, specificity, exact,
extension stripping, baseUrl, NULL safety, nearest-ancestor scope
selection, end-to-end loader against a tmp monorepo, no-configs
case) plus a registry test for the bare-call fix. Full suite:
2835 passed, 0 failed.
Cherry-picked and generalized from #243 (originally TypeScript-only
under tsconfig-named types). Renamed types (cbm_path_alias_scope_t /
collection_t in place of cbm_tsconfig_*) and centralised the load
into cbm_load_path_aliases so future loaders for other build tools
slot in without touching extraction or pipeline code.
Closes #243.
Co-authored-by: Peter Cox <info@petercox.ie>
|
||
|
Martin Vogel
|
cef58bdb65 |
feat(pipeline): bridge LSP resolved_calls into call-edge attribution
Discovery: result->resolved_calls was populated by go_lsp/c_lsp/php_lsp
but no pass downstream consumed it. pass_parallel.c::resolve_file_calls
and pass_calls.c::resolve_single_call only consulted the pipeline-side
name-based registry, ignoring type-aware overrides.
Both call resolvers now:
1. consult result->resolved_calls first via lsp_override_resolution()
— match key is (caller_qn, callee short_name)
2. fall through to cbm_registry_resolve only when no high-confidence
LSP override exists
3. when the override target QN is not in the graph (vendor receiver
types whose source isn't indexed), emit no edge at all instead of
the unified extractor's wrong name-based guess
On laravel/framework this drops total CALLS edges from ~83k to ~60k,
removing ~14% of guess-quality edges. The specific
ConfiguresPrompts.configurePrompts -> helpers.value misroute documented
in PHP_LSP_PRE_FLIGHT.md is verified gone.
|
||
|
Martin Vogel
|
632278a245 |
refactor(mcp): consolidate search-default constant + clarify pagination contract
Follow-up to #231. - src/foundation/constants.h: introduce CBM_DEFAULT_SEARCH_LIMIT = 200 as a single shared default (replaces the two inline enums in src/mcp/mcp.c and src/store/store.c so future tweaks land in one place; also documents why 200 was chosen). - src/mcp/mcp.c: replace the inline SEARCH_DEFAULT_LIMIT enum with the shared constant; expand search_graph / query_graph / search_code tool descriptions so agents understand: * search_graph caps at limit (default 200), response carries 'total' and 'has_more', paginate with offset += limit while has_more is true * query_graph has a 100k row ceiling, no offset support — narrow the Cypher with LIMIT or fall back to search_graph paging * search_code caps enriched results at limit (default 10), reports 'total_grep_matches' / 'total_results' for truncation detection, no offset — raise limit or narrow with file_pattern / path_filter - src/store/store.c: replace ST_DEFAULT_SEARCH_LIMIT with the shared constant. Behavior unchanged from #231; this just makes the contract explicit to MCP clients and removes the duplicated magic-number enums. |
||
|
amitmynd
|
9425b99fd5 |
fix(mcp): cap search_graph default limit to 200 (was 500K) (#231)
The default limit for search_graph used MCP_HALF_SEC_US (500,000), a microsecond timeout constant misused as a row count. Broad queries like file_pattern="**/ads/**" on a 12K-node project returned all 1,287 matches as 3.3M characters of JSON, exceeding LLM tool-result limits. - MCP handler: default limit 500,000 -> 200 - Store layer: fallback limit ST_HALF_SEC (500,000) -> 200 - Tool description: "Default: unlimited" -> "Default: 200" Pagination via offset/limit still works; the response includes total and has_more so callers can detect truncation and page through. |
||
|
Martin Vogelandmap588
|
a20d55160f |
refactor(memory): use safe_str_free / safe_free / safe_grow at heavy sites
Apply the platform.h memory wrappers across the largest free / realloc hot spots so cleanup paths are uniform and pointers are NULLed on free to prevent accidental use-after-free / double-free. - src/cypher/cypher.c (~85 sites): replace free((void *)x) with safe_str_free(&x) for owned const char * fields; the array-pointer cleanup in parse_in_list error paths uses safe_free(vals) since vals is const char **. - src/store/store.c (~81 sites): same const-string substitution across cbm_search_result_t cleanup, project list, schema info, etc. - src/mcp/mcp.c (~21 sites): same, plus collect_grep_matches growth loop converted to safe_grow(gm, gm_count, gm_cap, PAIR_LEN) — safe here because grep_match_t holds fixed-size char arrays only. - src/pipeline/pass_githistory.c: free_counter callback uses safe_str_free for the hash key. Refactor work cherry-picked from #210; #210 itself bundled .claude/, .github/workflows/, and a stray .claude/worktrees/ submodule that did not belong in the repo, and most of its other changes had landed via #206 / #207 / #208 already. The cypher / store growth-path safe_realloc loops are intentionally NOT converted to safe_grow because their elements own heap pointers — a failed realloc would leak them; the non-destructive realloc + cleanup pattern from #206 is preserved. Co-authored-by: map588 <map588@pitt.edu> |
||
|
Martin Vogelandmap588
|
8fe1310779 |
feat(platform): add safe_free / safe_str_free / safe_buf_free / safe_grow
Centralize repetitive heap-management patterns into four wrappers alongside the existing safe_realloc: - safe_free(ptr): frees and NULLs a pointer to prevent double-free / use-after-free - safe_str_free(&str): frees a const char * and NULLs it without forcing the caller to cast away const at every site - safe_buf_free(buf, &count): frees a heap array and zeros the paired count, so the dangling-array / nonzero-count footgun cannot survive - safe_grow(arr, n, cap, factor): one-line capacity-doubling realloc for arrays whose elements do not own additional heap memory; documents the safe_realloc free-on-failure semantic so arrays of owned pointers keep using a manual realloc+cleanup pattern Cherry-picked from #209's platform.h additions; the rest of #209 duplicated work that landed via #206 and #207, so closing that PR in favor of this targeted commit. Co-authored-by: map588 <map588@pitt.edu> |
||
|
Martin Vogel
|
485f8cdd91 |
fix(store): include USAGE edges in degree filter
Follow-up to #208. The decorator pass now emits USAGE edges, but the degree-filter subqueries in cbm_store_search counted only CALLS edges, so a function used only via @decorator was still reported as dead by search_graph(max_degree=0). Extend the in_deg/out_deg subqueries to count CALLS and USAGE so both pass_usages output and the new decorator edges feed into dead-code detection as intended. |
||
|
Matthew Prock
|
f65afda0df |
feat(pipeline): emit USAGE edges for decorator applications (#208)
Decorators previously only created DECORATES edges. References like @login_required were invisible to find-references queries that walk CALLS/USAGE relationships. Both sequential (resolve_decorator) and parallel (resolve_def_decorators) paths now emit a USAGE edge alongside DECORATES. Empty {} properties avoid clobbering richer metadata from pass_calls when a function both has @decorator and calls decorator() directly. Using USAGE instead of CALLS keeps pass_tests' CALLS-scan from inferring false TESTS edges (decorated test function → decorator).
|
||
|
Martin Vogel
|
c768b6154d |
fix(watcher): close residual gaps in #207
Follow-up to #207: - cbm_watcher_new: NULL-check cbm_ht_create() and free w on OOM so later watch/unwatch/touch don't dereference a NULL hash table. - cbm_watcher_watch_count: drop const from the signature and remove the const_cast around projects_lock. The function necessarily touches synchronization primitives so a const receiver was misleading; all current callers pass a non-const pointer. |
||
|
Matthew Prock
|
8fbdb0fa20 |
fix(thread-safety): eliminate races in log mutex, watcher, and index threads (#207)
- http_server: replace lazy log mutex init with 3-state UNINIT/INITING/INITED CAS pattern; cbm_ui_log_init() called from main before thread creation, also still invoked lazily from cbm_ui_log_append() so early single-threaded logs are not dropped - watcher: protect projects hash table with cbm_mutex_t (watch/unwatch/touch/watch_count/free); poll_once snapshots project pointers under lock then iterates without it so git I/O and indexing don't block watch/unwatch - watcher: handle state_new() OOM in watch path - compat_thread: add cbm_thread_detach() for POSIX and Windows; both join() and detach() clear the handle on success for consistent lifecycle tracking - http_server: detach index job threads to prevent handle leaks |
||
|
Martin Vogel
|
9653e72309 |
fix(cypher,store): close residual OOM gaps in #206
Follow-up to #206. Adds the allocation-check coverage that the original PR did not reach: - store: NULL-check malloc in cbm_store_get_schema node-labels and edge-types blocks; replace safe_realloc with non-destructive realloc + cleanup so a partial result is freed (and node-labels block freed via cbm_store_schema_free) instead of leaked. - cypher: NULL-check heap_strdup return in parse_props (key/value), parse_rel_types (first and subsequent types), and parse_in_list (op and list values). Each failure path frees prior heap_strdup'd entries before returning. - cypher: parse_case_expr realloc-failure cleanup now also frees then_val for the just-parsed branch and for each previously stored branch, not just the when_expr trees. |