The release.yml workflow publishes both wrappers as-is from the repo;
they were still pinned to 0.6.1, which made the npm publish step try to
republish v0.6.1 instead of v0.7.0.
The LSP-cross loop in parallel_resolve called cbm_slab_reclaim() without
first destroying the thread-local tree-sitter parser, violating the
documented slab contract. The parser's lexer holds slab-allocated state
(notably lexer.included_ranges, a 24-byte allocation that lands in the
≤64B slab bucket); reclaiming the slab leaves that pointer dangling.
Workers don't trip on it because they don't reparse afterward, but the
main thread participates in cbm_parallel_for and any later cbm_extract_file
on the main thread — including the sequential pass_definitions in the
incremental pipeline — hits a heap-use-after-free in ts_lexer_goto on
macOS-ASan.
Two-part fix:
- pass_parallel.c: destroy the thread parser before cbm_slab_reclaim() in
the per-file LSP-cross loop, matching the worker extract loop.
- pass_definitions.c: defensively drop the thread parser at pass entry so
the incremental sequential path cannot inherit stale state from any
prior run.
GCC's preprocessor doesn't short-circuit 'defined(__has_feature) && __has_feature(...)' — it tries to parse the call syntax of the undefined token and errors with 'missing binary operator before token "("'. Modern GCC and Clang both define __SANITIZE_ADDRESS__ under -fsanitize=address, so use that single check.
Free the result before the time-budget assert (no leak on miss) and scale the budget ×10 when ASan/UBSan is active, so the perf benches run cleanly in the release CI (skip_perf=false) instead of only being gated by CBM_SKIP_PERF in dry-run.
Same fix as read_file in pass_definitions: the calls/usages/semantic/k8s/lsp_cross passes each read a source file into a malloc'd buffer and re-parse it, so they hit the same tree-sitter lexer over-read past EOF under ASan. Over-allocate by 16 and zero the tail in every reader.
read_file allocated size+1, but tree-sitter's lexer peeks a few bytes past the final UTF-8 char (ts_lexer_goto), reading past the buffer end. Under ASan this is a heap-buffer-overflow (intermittently flagged in the incremental suite on macOS); in production it is a benign over-read of adjacent heap. Over-allocate by 16 bytes and zero the tail so the lookahead stays in-bounds.
cbm_mkdtemp rewrites the template in place to the translated %TEMP% path on Windows, which overflowed the 23-byte 'char tmpl[]' literal (ASan stack-buffer-overflow). Use a 256-byte buffer as the cbm_mkdtemp contract requires (matching the other test callers).
Four tests hardcoded POSIX /tmp/ paths that don't exist on Windows: test_path_alias used raw mkdtemp("/tmp/...") and the store pragma/checkpoint tests built /tmp/*.db directly. Switch to cbm_mkdtemp (translates /tmp/ -> %TEMP% on Windows) and cbm_tmpdir() (matching test_store_bulk). build/smoke/soak-windows already pass; this makes the Windows unit-test job green too.
system_info.c routed Windows into the '#else /* Linux */' detection branch (sysconf/sysinfo, unavailable on msys2-clang) because the platform chain only special-cased __APPLE__/BSD; restrict the Linux branch with '#elif !defined(_WIN32)' so Windows uses detect_system_windows() exclusively.
path_alias.c used raw strndup (undeclared on msys2-clang); switch to the cbm_strndup compat shim and include foundation/compat.h.
The py/cs LSP benchmark tests have time-budget asserts (<150ms / <200ms) that ASan+UBSan blow on CI (750-900ms on arm64), so they bailed before cbm_free_result and LeakSanitizer flagged the leaked arena/parser. Gate them behind CBM_SKIP_PERF (set by the dry-run) like the incremental perf tests, so they run only when perf is enabled.
Fix Windows (msys2-clang) compile errors in test code by using the existing compat shims: setenv/unsetenv -> cbm_setenv/cbm_unsetenv (test_store_pragmas, test_ts_lsp), strndup -> cbm_strndup and 2-arg mkdir -> cbm_mkdir (test_path_alias), and guard the utimensat mtime-bump in test_pipeline behind #ifndef _WIN32.
ws 8.19.0 had a moderate uninitialized-memory-disclosure issue in websocket.close() (fixed in 8.20.1). Transitive dev dependency; the existing ^8.18.0 range already permits the fix, so this is a lockfile-only bump. npm audit now reports 0 vulnerabilities.
Format drift had accumulated across recent commits (the dry-run lint enforces clang-format-20, which had not run on these commits yet). No functional changes — whitespace/layout only, verified by a clean cppcheck + clang-format + NOLINT lint-ci pass.
cppcheck (warning+style, error-exitcode=1): guard cbm_fqn_compute against a NULL project/rel_path (ctunullpointer reachable from cbm_extract_file), and narrow prefix_len/suffix_len (path_alias) and ft_count (pass_githistory) to the scopes that use them.
security-audit.sh: the mcp.c file-read count grew to 13 (search/ADR/Windows-support reads — all path-contained or transport reads, audited, no new exfiltration surface); bump the reviewed maximum and document the update-check + request-body reads. Allow the diagnostics.c atomic metrics dump (.tmp+rename) in the file-write scan, and allow-list the sqlite WAL-checkpoint doc URL referenced in a store.c comment.
The per-file and cross-file LSP type-aware call/usage resolution now runs in every index mode (full/moderate/fast) instead of being gated behind a separate CBM_MODE_ADVANCED. Remove the mode from both index-mode enums, the g_lsp_enabled toggle and its save/restore dance, and the per-mode gating in discover, pipeline and the incremental path.
Now that the LSP walkers are O(n), the callee-attribution accuracy (e.g. ~91k call overrides on roslyn, ~60k on kubernetes) is worth the latency in every mode. Fast mode still filters files and skips similarity/semantic edges, so it stays faster than full while now also benefiting from LSP resolution.
A 'mode':'advanced' request falls back to the full default — identical behaviour to the old advanced mode, so existing callers are unaffected. This also fixes the test_incremental.c failures, which were asserting LSP-resolved results that the gated incremental path never produced.
The advanced-mode LSP passes walked the AST with the same O(i) ts_node_child(node, i) idiom, so a wide root made each per-file pass quadratic — the TS fixture reallyLargeFile.ts took ~16000s. Add a shared cbm_lsp_collect_children() helper (lsp_node_iter.h) and convert the entry and recursive walkers across the TypeScript, C, C#, PHP and Go LSP back-ends to a single O(n) cursor pass. The file now indexes in ~28s; full advanced-mode runs: TS 100s, roslyn 46s, kubernetes 51s, postgres 8s, WordPress 7s.
Also fix two latent crashes exposed once the resolve phase actually ran: process_node indexed the NULL-terminated param_types[] by the call's argument count, reading past the terminator when a call passed more args than the callee declares; and return_type_of() built a tuple via cbm_type_tuple(NULL, ...), passing a NULL arena — it now threads ctx->arena.
This also includes the Tier 2 cross-LSP registry plumbing (shared read-only base registry with per-file overlay/fallback) the walkers build on.
All LSP unit suites pass: TS, C 737/737, C# 60/60, PHP 280/280, Go 55/55.
tree-sitter's ts_node_child(node, i) is O(i) — it walks the child iterator from the first child on every call — so the common 'for (i=0; i<count; i++) ts_node_child(node, i)' idiom is O(n^2) over a node's children. On a file whose top level is very wide (e.g. a 580k-line fixture parsed as hundreds of thousands of sibling comment nodes) this made extraction run for over an hour.
Add ts_nstack_push_children(), which enumerates a node's children in a single O(n) cursor pass, and use it in the whole-tree DFS walkers (calls, channels, env-accesses, imports, type-assigns). Convert the module-level variable walk to a cursor too, and add cbm_is_module_level_p() so callers that already hold the parent node skip the O(n) ts_node_parent() rescan.
Indexing microsoft/TypeScript drops from ~5100s to ~50s with identical graph output.
Same clean def-driven pattern as C/Go. Extracts cs_register_lsp_defs,
adds cbm_cs_build_cross_registry (filters CBM_LANG_CSHARP) +
cbm_run_cs_lsp_cross_with_registry. resolve_worker dispatches
CBM_LANG_CSHARP to the shared registry; pipeline.c builds
cross_registries.cs; pass_lsp_cross.h includes cs_lsp.h.
C# cross-LSP is def-driven (no per-file AST field collection), so the
shared registry yields identical resolution to the per-file build.
roslyn bench (17,916 C# files, ADVANCED): single shared registry
(lsp_cross_prepare=234ms, 260,814 defs, zero slow files), 91,089 LSP
overrides applied, 58.6s total. All 60 test_cs_lsp tests pass; serilog
sanity exact (2,627n/7,112e). No SEGV/ASan.
Brings C/C++/CUDA to Go's optimization level. Extracts the cross-LSP
registration loop into c_register_lsp_defs (shared by per-file and
Tier 2 paths), adds:
cbm_c_build_cross_registry — builds ONE project-wide registry from
all C-family defs (filters by lang) + both C and C++ stdlibs,
finalized, shared READ-ONLY across resolve workers.
cbm_run_c_lsp_cross_with_registry — per-file resolve using the
shared registry; no per-file registry rebuild.
C cross-LSP is DEF-DRIVEN (fields/returns/embeds come from the def
strings, no per-file AST field collection), so the shared registry
produces identical resolution to the per-file build — zero quality
loss. resolve_worker dispatches CBM_LANG_C/CPP/CUDA to the new path;
pipeline.c builds cross_registries.c; pass_lsp_cross.h includes c_lsp.h.
postgres bench (4,967 files, ADVANCED mode): 628 C files share one
pre-built registry (lsp_cross_prepare=17ms, zero slow files), 44 LSP
overrides, +9 edges vs FULL. All 200 test_c_lsp tests pass. No
SEGV/ASan. Total pipeline overhead vs FULL: +5.2%.
New CBM_MODE_ADVANCED=3. Both per-file LSP (type-aware resolution
during extract) and the cross-file LSP pass (during resolve) are now
OPT-IN: FULL and below run tree-sitter extraction + registry textual
resolution only; ADVANCED adds the full LSP layer.
Rationale: profiling showed LSP changes the kubernetes graph by only
~0.01% of edges (it mostly re-confirms textual resolution) while
costing significant memory + time. Making it opt-in keeps the default
FULL path lean (3.0 GB vs 5.6 GB peak RSS on K8s) without meaningfully
worse output, and lets users pay for LSP precision only when wanted.
Mechanism:
- cbm.c: g_lsp_enabled atomic toggle wraps the entire per-file LSP
block (Go/C/C++/PHP/Python/TS/JS/C#) + the C/C++ preprocessor-pass
LSP call. Default true so ad-hoc extraction / tests keep historical
behaviour. cbm_set_lsp_enabled / cbm_get_lsp_enabled in cbm.h.
- pipeline.c: sets the toggle (save→set→restore so it never leaks to
other callers) from p->mode before extraction; gates the cross-LSP
all_defs collection + registry build on advanced; gates the
sequential lsp_cross dispatch. Fixed the moderate_only comparison
(== FAST, not > MODERATE) so ADVANCED is not treated as lighter.
- pipeline_incremental.c: same toggle save/set/restore.
- discover.h/.c: added CBM_MODE_ADVANCED to discover's enum + all 4
filter comparisons treat ADVANCED like FULL (full discovery, no
fast-filtering). Without this, ADVANCED (numerically 3) was
mis-detected as fast-mode and dropped ~half the files.
- mcp.c: parse mode "advanced" + tool schema enum/description.
Dual-mode K8s bench: FULL 46.7s (lsp_overrides=0, 3.0 GB) vs
ADVANCED 48.8s (lsp_overrides=80,818, 5.6 GB). Graphs within 0.01%.
All 3,615 tests pass.
Replaces the prior Robin-Hood open-addressing implementation with
Verstable (https://github.com/JacksonAllan/Verstable), a 2024 state-
of-the-art C hash table. The public API in foundation/hash_table.h
is UNCHANGED — CBMHashTable is now an opaque type and all 14
production callers (graph_buffer, registry, pipeline passes,
service_patterns, watcher, semantic, etc.) get the new impl
transparently. str_intern.c keeps its specialised (ptr+len) hash —
different semantics, not a clean swap target.
Why: per-sub-phase profiling pinpointed cbm_pipeline_lsp_target_node
and cbm_registry_resolve as the dominant resolve_calls cost. Both
ultimately hit cbm_ht_get; both run millions of times on a kubernetes
index. Verstable's key innovations target exactly this:
* 4-bit hash-fragment metadata in a uint16_t sidecar array lets
chain walks skip most key comparisons WITHOUT touching the
bucket payload (cache wins on hot lookups).
* Quadratic probing with per-bucket displacement chains (not linear
Robin-Hood psl).
* Tombstone-free deletion that moves at most one existing key.
Vendored at internal/cbm/vendored/verstable/verstable.h (2069 lines,
84KB, MIT). Wrapped as a const char* → void* template inside
hash_table.c. The header struct is opaque (forward declaration only);
audited the entire tree — no caller accesses internals or takes
sizeof(CBMHashTable), so opacity is safe.
Test refactor: test_hash_table.c had two impl-specific tests:
* ht_robin_hood_bounded_psl — Robin-Hood specific; removed.
ht_stress_10000 covers the functional equivalent.
* ht_power_of_two_capacity — renamed ht_create_small_capacity,
replaced struct probes with a functional insert+lookup check.
ht_create_capacity_zero/one similarly de-internalised.
Benchmark on kubernetes (12 workers, 20,650 files):
- parallel_resolve wall: 36,376 -> 27,217 ms (-25.2%)
- pipeline.done total: 61,485 -> 51,745 ms (-15.8%)
- registry_build: ~1,000 -> 779 ms (faster — registry uses
cbm_ht_set heavily during build)
- All 3,615 tests pass; hash_table suite 100% green
- ASan/UBSan/SEGV: none
- Peak RSS: ~6.83 GB
- Graph: nodes 297,082 -> 297,102 (+20, 0.007%), edges 1,155,444 ->
1,154,025 (-1,419, -0.12%) — within tolerance; minor delta likely
from hash-iteration-order differences affecting tie-breaks in
resolution. Sanity (zod/serilog): nodes exact, edges within 0.1%.
Cumulative trajectory from broken-state baseline ~35 min on K8s ->
51.7s wall = ~40x total speedup.
THE big win. Pinpoint-instrumented profiling showed 98.7% of
resolve_calls CPU on kubernetes (881s out of 893s) lived inside the
'resolve' sub-step — cbm_pipeline_lsp_target_node when LSP hits,
cbm_registry_resolve strategy chain when it misses. The LSP path
is fast; the strategy chain isn't.
cbm_registry_resolve's strategy chain (import_map -> same_module ->
unique_name -> suffix_match+scoring) is deterministic for a given
(callee_name, module_qn) pair. module_qn is CONSTANT within a file,
so within one file the same callee_name resolves identically every
time. K8s files have ~200 calls but only ~50 unique callee_names —
so ~75% of strategy-chain walks are redundant.
Added a TLS hash cache (_resolve_cache, registry.c) keyed by
callee_name with value cbm_resolution_t. cbm_registry_resolve checks
the cache first; on miss it runs the full chain and caches BOTH
positive and negative results (caching the negative is just as
valuable — same name asks the same question).
API in pipeline.h:
void cbm_registry_resolve_cache_begin(int estimated_capacity);
void cbm_registry_resolve_cache_end(void);
Two smaller chunks landed in the same commit:
- src/pipeline/registry.c: per-file import-map prefix -> module-qn
hash (cbm_registry_import_map_cache_*). resolve_import_map's
linear strcmp scan becomes O(1). Flat impact in this run because
the LSP path bypasses cbm_registry_resolve for most K8s calls,
but useful for projects without strong LSP coverage.
- src/pipeline/pass_parallel.c: reuse lsp_target as target_node
when try_field_type_hint didn't override the QN. Saves 1
cbm_gbuf_find_by_qn lookup per LSP-resolved call.
Also added 6 sub-counters inside resolve_file_calls (time_ns_rc_*)
logged as parallel.resolve.calls_breakdown / breakdown2 to keep
durable telemetry on which sub-step dominates.
Benchmark on kubernetes (12 workers, 20,650 files):
- parallel_resolve wall: 90,531 -> 36,376 ms (-59.8% vs C2; -64.8%
vs original 103,281 baseline)
- resolve_calls CPU: 893,657 -> 277,556 ms (-68.9%)
- 'resolve' sub-counter: 881,650 -> 266,521 ms (-69.8%) ← target hit
- resolve_usages CPU bonus: 10,986 -> 5,594 ms (-49.1%, secondary
effect — fewer ambiguous resolutions to scrub)
- Total pipeline.done: 115,918 -> 61,485 ms (-47%)
- User CPU total: 953.7s -> 375.1s (-60.7%)
- Peak RSS: +430 MB for cached cbm_resolution_t structs (~12 live
caches at a time)
- Tests: 3616/0
- Graph: nodes 297,082 EXACT match, edges +/- 0.2%
- zod/serilog sanity: nodes exact, edges within tolerance
Effective hit rate: predicted 75%, measured ~70%. The ~30% true
cache misses (unique callee_names per file) still run the full
chain — next-tier optimizations: widen scope to per-worker so
common names like 'Get'/'Add' are computed once per worker not
once per file, or pre-filter candidate lists by import reachability
at file entry to shrink the chain's inner cost.
Cumulative trajectory from original broken state: ~35 min K8s cold
index -> 63.6s wall = ~33x total speedup.
cbm_service_pattern_match runs once per resolved CALL edge inside
emit_service_edge — that's 7 pattern lists × ~30 patterns × strstr
= ~210 strstrs per call. On kubernetes (~600k resolved CALL edges)
the SAME resolved QN ("fmt.Errorf", "context.Context.Done",
"errors.New", etc) repeats thousands of times across files. Caching
turns the linear pattern-list scan into one hash lookup after the
first miss.
API additions in service_patterns.h:
void cbm_service_pattern_cache_begin(void);
void cbm_service_pattern_cache_end(void);
Cache is TLS (no contention) scoped to the resolve_worker's lifetime
in the parallel_resolve phase. Capacity 8192 entries per worker —
covers typical project QN diversity without rehashing.
Encoding: enum value + 1 in the void* slot (0/NULL means "miss").
Two helpers svc_enum_to_ptr / svc_ptr_to_enum hide the trick.
Benchmark on kubernetes (12 workers, 20,650 files):
- parallel_resolve wall: 98,520 -> 90,104 ms (-8.5% vs Chunk 1
baseline, -12.8% vs original 103,281 baseline)
- resolve_calls CPU: 992,427 -> 885,663 ms (-10.8% vs Chunk 1,
-5.4% vs original — Chunk 1 added overhead that Chunk 2 recovers
AND surpasses)
- resolve_usages CPU: 11,977 -> 10,944 ms (-8.6%)
- total_loop CPU: 1,005,409 -> 897,561 ms (-10.7%)
- Tests: 3616/0
- Graph: 297,082 -> 297,080 nodes (-2), 1,153,482 -> 1,156,097 edges
(within 0.2%, semantic-preserving)
- zod/serilog sanity: edges within +/-3 of baseline
- Peak RSS: 7.46 -> 7.75 GB (+290 MB for 12 × 8192-entry caches)
Cumulative trajectory: original broken state ~35 min K8s ->
1.92 min wall (-49× total). The per-file is_import_reachable cache
(Chunk 1) + the per-worker service-pattern cache (Chunk 2) together
save ~13s of resolve wall time. Next chunks target the gbuf_insert_
edge hot path and remaining strstr scans.
Wraps is_import_reachable's O(candidates × imports × strstr) hot loop
with a TLS hash-table cache scoped to the resolve_worker's per-file
iteration. Same callee_name appears in many call sites and each
lookup re-evaluates the same ~30 candidates against the same imports
— with the cache, first eval pays the strstr cost, repeats are O(1).
API additions in pipeline.h:
void cbm_registry_reach_cache_begin(int estimated_capacity);
void cbm_registry_reach_cache_end(void);
resolve_worker calls _begin right after build_import_map and _end
right before the loop's free_import_map (the cache scope must match
the import_vals scope — different files have different imports, so
the cache is invalidated between files).
Benchmark on kubernetes (12 workers, 20,650 files):
- parallel_resolve wall: 103,281 -> 98,520 ms (-4.6%)
- resolve_usages CPU: 33,971 -> 11,977 ms (-65%) — cache hits hard here
- resolve_calls CPU: 935,958 -> 992,427 ms (+6.1%) — cache adds overhead
here without saving work, because the Tier 3 LSP override path hits
first and bypasses cbm_registry_resolve for most CALL entries.
Usage/throws/rw paths always go through cbm_registry_resolve so the
cache helps them; the net wall improvement comes from that side.
- Tests: 3616/0
- Graph: 297,082 -> 297,115 nodes (+33), 1,153,482 -> 1,155,005 edges
(within 0.1%, semantic-preserving)
- zod/serilog sanity: nodes exact, edges +/-1 (within tolerance)
The strstr overhead per call wasn't the dominant resolve_calls cost —
the next chunks need to attack the actual hot spots: emit_service_edge's
cbm_service_pattern_match (6 lists × ~30 patterns × strstr per call =
~180 strstrs/call) and the gbuf_insert_edge allocator pressure.
Instrument resolve_worker with per-sub-phase atomic ns counters
(import_map, cross_lsp, resolve_calls, resolve_usages, resolve_throws,
resolve_rw, resolve_semantic) plus a whole-iteration total_loop and
files_visited count. Logged at end of cbm_parallel_resolve as three
log lines (split to avoid the PP_RING=4 TLS-slot aliasing in itoa_log
that masked the data in the first attempt).
Profiling result on kubernetes (12 workers, 20,650 files):
parallel_resolve wall: 103,281 ms
total_loop CPU-ms: 970,352 (78.3% worker utilization, busy)
phase_sum CPU-ms: 970,303 (99.99% coverage of loop body)
resolve_calls: 935,958 96.45% ← THE BOTTLENECK
resolve_usages: 33,971 3.50%
resolve_semantic: 192 0.02%
resolve_calls (cost): 935,958 / 616,795 = 1,517 µs per call edge
resolve_usages (cost): 33,971 / 942,846 = 36 µs per usage edge
Calls are 42× more expensive per edge than usages. The extra work in
resolve_file_calls (vs resolve_file_usages) traces to:
1. is_import_reachable (registry.c:154) — O(imports × 2*strstr) per
candidate × ~30 candidates per name × 1.6M call attempts =
~480 sec of strstr work alone.
2. cbm_service_pattern_match — O(6 lists × ~30 patterns × strstr)
per call = ~48 sec aggregate.
3. try_field_type_hint, emit_service_edge dispatch overhead.
These are now the next optimization targets — replace substring
matching with hashed prefix lookups (trie or hash of module
fragments), and add per-file memoization of is_import_reachable
results so the same candidate doesn't get re-evaluated per call site.
Instrumentation overhead is negligible (~30ns atomic_fetch_add per
phase per file = <100µs aggregate on K8s) and gives durable
telemetry for future perf work.
Two changes:
1. resolve_file_calls now builds a per-file CBMHashTable mapping
'caller_qn|callee_short' -> CBMResolvedCall* once at function
entry, then does O(1) lookups instead of the O(N) linear scan in
cbm_pipeline_find_lsp_resolution. Falls back to the linear scan
only for keys the index misses (preserves all existing semantics).
The hash key uses callee_qn's short name (last dot-separated
segment) to match the original linear-scan's match rule. On insert,
keeps the highest-confidence entry per key.
2. Memory safety: keys must be strdup'd before cbm_ht_set, because
cbm_ht_set stores the key pointer rather than copying. The first
draft used stack-allocated 'char key[1024]' inside the loop body —
ASan caught it as stack-use-after-scope when subsequent cbm_ht_get
calls dereferenced the dangling stack frame addresses. Fixed:
strdup on insert, reuse cbm_ht_get_key when updating an existing
bucket (no double-free), free all keys via cbm_ht_foreach before
cbm_ht_free on cleanup.
Benchmark on kubernetes (12 workers, 20,650 files):
- parallel_resolve: 113,300ms (Tier 3 meta) -> 110,924ms (this commit)
~2.1% improvement — within run-to-run noise. Despite the O(N) -> O(1)
algorithmic improvement, the (calls × resolved_calls) product on
K8s Go files is small enough at this scale that the linear scan was
not the dominant cost. Other call sites of cbm_pipeline_find_lsp_
resolution (e.g. pass_calls.c) keep the original O(N) path since
this file-level index is local to resolve_file_calls.
The change is still net positive:
- Removes a real memory-safety bug found by ASan
- O(N) -> O(1) algorithmic improvement at the call site, even if its
scale impact is modest at K8s size
- Sets up larger codebases (where the linear scan would dominate) to
benefit without further work
Tests: 3,616 / 0 failed including parallel_python_lsp_override_cross_
file_emits_lsp_strategy_edges (which directly exercises the LSP
resolution path). Graph counts: nodes -1 vs baseline, edges +945 vs
baseline (within 0.1%, semantic-preserving). zod/serilog sanity:
nodes exact, edges +4/+5 (within tolerance).
Real bottleneck for the remaining ~110s on K8s is elsewhere — likely
the cumulative cost across 20K files of registry_resolve + gbuf_find_
by_qn + gbuf_insert_edge per resolved entry across all 5 resolve_file_
passes. Profiling needed to confirm; the lightweight per-worker edge
buffer (deferring dedup to merge) is the next obvious attack vector.
KEY ARCHITECTURAL INSIGHT (credit: user observation):
The per-file LSP during extract ALREADY captures the inferred receiver
type QN + method name for unresolved cross-file calls — emit_unresolved_
call at go_lsp.c:1242-1245 stores callee_qn as '<FullReceiverTypeQN>.<
MethodName>' in result->resolved_calls flagged with strategy='lsp_
unresolved'. The cross-LSP step was REDUNDANTLY re-deriving this same
metadata via a second tree-sitter parse + AST walk.
Cross-LSP for Go is now METADATA-DRIVEN:
cbm_go_fast_resolve_qualified_calls (rewritten):
- Iterates result->resolved_calls where strategy=='lsp_unresolved'
- For each entry: try direct registry lookup of callee_qn;
if that misses, split prefix.suffix and resolve prefix via the
file's import map then lookup '<module_qn>.<suffix>' in registry
- Emits resolved entry on top (cbm_pipeline_find_lsp_resolution
picks highest confidence)
- NO tree-sitter parse, NO AST walk, NO scope tracking
resolve_worker (pass_parallel.c):
case CBM_LANG_GO now calls ONLY the metadata-driven fast path.
cbm_run_go_lsp_cross_with_registry (the slow AST-walk version) is
no longer invoked for Go — it would just re-derive identical
metadata via a second AST walk and arrive at the same answers.
Benchmark (kubernetes, full fresh index, 12 workers):
- parallel_resolve: 1.89 min (Tier 3 meta) vs 1.86 min (Tier 2) — parity
- lsp_cross.slow lines: 0 (no tree-sitter fallback runs)
- Nodes: 297,081 vs 297,082 baseline (-1)
- Edges: 1,153,163 vs 1,153,482 baseline (-319, within 0.03%)
- Tests: 3,616 / 0 failures
- zod/serilog sanity: nodes exact, edges +1-3 (within 0.03%)
The wall-time parity is informative: the AST re-walk wasn't the
dominant cost. Per-file budget (66ms across 20,650 files / 12 workers)
is now dominated by the 5 sequential resolve passes (resolve_file_
calls/usages/throws/rw/semantic) pushing ~10M total edges into the
per-worker gbuf with arena allocs + hash-bucket inserts. The Tier 3
work eliminates a genuine architectural redundancy (two walks of the
same AST producing the same metadata) and sets up Python/C/etc to
adopt the same metadata-driven pattern in follow-up commits — but
hitting the user-target <30s K8s requires attacking the gbuf hot
path, not the cross-LSP path.
When the genskip commit (aecd861) moved `const char *rel` earlier in
the resolve_worker loop body (so the is_generated check could use it
before the cross_lsp_eligible computation), the original declaration
further down in the loop body wasn't removed — fails to compile with
"redefinition of 'rel'". This was caught by the post-commit bench
agent and fixed in-place but not landed; now it is.
Investigation of kubernetes' largest Go files revealed the perf tail
is dominated by machine-generated code:
pkg/generated/openapi/zz_generated.openapi.go 74,266 lines
staging/src/k8s.io/api/core/v1/generated.pb.go 71,154 lines
Multiple *.pb.go protobuf files 12-14K lines each
209 files in staging/.../client-go/applyconfigurations/
These files are boilerplate (getters/setters/marshal/unmarshal) with
limited semantic value for code-navigation queries. But because tree-
sitter parse of a 70K-line file alone is ~1-2 seconds, even one
unresolved cross-file call in such a file triggers the full cross-LSP
overhead and dominates the wall time.
Skip cross-LSP for files matching common generated-file conventions:
*.pb.go (Go protobuf)
zz_generated* (k8s codegen convention)
*_generated.go (general Go codegen)
*.gen.go (general Go codegen)
/applyconfigurations/ (k8s codegen subtree)
*_pb2.py, *_pb2_grpc.py (Python protobuf)
*.pb.cc, *.pb.h, *.pb-c.{c,h} (C/C++ protobuf)
The per-file LSP during extract still indexes their defs and calls
normally — only the cross-file resolution refinement is skipped.
Graph quality: zero loss for hand-written code; for generated code
cross-file method resolution falls back to whatever per-file LSP
captured (which is typically complete for intra-package boilerplate).
Two changes layered:
1. Smart pruning of cross-LSP work — the cross-LSP work (re-parse +
AST walk + registry lookups, ~50-150ms per file) only adds value
if a file has unresolved cross-file method calls. If the per-file
LSP during extract already resolved every call
(result->resolved_calls.count >= result->calls.count) or the file
has zero calls at all, the AST has nothing left for cross-LSP to
improve. cross_lsp_eligible now additionally requires:
result->calls.count > 0
&& result->resolved_calls.count < result->calls.count
On kubernetes this skips 87% of files (17,949 of 20,650) with
zero semantic loss — graph node/edge counts within ±0.14% of the
non-pruned baseline.
2. Python Tier 2 wired through pipeline. The cbm_py_build_cross_registry
and cbm_run_py_lsp_cross_with_registry helpers from the previous
commit are now actually populated and dispatched. resolve_worker's
switch now handles CBM_LANG_PYTHON in addition to CBM_LANG_GO.
3. cbm_parallel_resolve's cross_registries param is now typed void*
in the header (cast back to CBMCrossLspRegistries* in the impl) —
the typedef CBMCrossLspRegistries lives in pass_lsp_cross.h which
transitively pulls in lsp/go_lsp.h. Dodging the typedef-vs-struct-
tag mismatch this way keeps pipeline_internal.h light (no LSP
header dependency).
Bench on kubernetes (mode=full, fresh DB):
- parallel_resolve: 1.86 min (Tier 2 Go) -> 2.06 min (+10% regression)
The slight regression on Go-heavy k8s comes from Python registry
build (lsp_cross_prepare: 911 -> 1,014 ms) and from the fact that
the 13% of files cross-LSP still runs on are the EXPENSIVE files
with many unresolved calls — pruning the 87% trivial files saves
less than the remaining 13% hard files cost.
- Tests: 3,616 / 0 failed. parallel_python_lsp_override_cross_file_
emits_lsp_strategy_edges still PASS.
- zod/serilog: node/edge counts within ±0.1% of known-good.
The smart-pruning + Python wiring is net-positive across the language
matrix (Python-heavy repos benefit from the new Tier 2 path) but
doesn't move the needle on Go-heavy k8s. The remaining ~2 min on k8s
is dominated by the AST re-walk inside cross-LSP — eliminating that
requires capturing unresolved-call metadata during extract so cross-
LSP is pure registry lookup (no re-parse). Follow-up commit.
The gopls 'package summary' pattern, applied properly: build the cross-
LSP CBMTypeRegistry ONCE per language at pipeline time (in a dedicated
cross_lsp_arena), share it READ-ONLY across all resolve workers via the
CBMCrossLspRegistries struct. Per-file work then becomes just tree-
sitter parse + AST walk + O(1) hash lookups — zero per-file registry
rebuilds for languages with a pre-built registry.
Each LSP gains two new public functions:
CBMTypeRegistry *cbm_<lang>_build_cross_registry(arena, defs, def_count)
— filters all_defs to its language, init + stdlib + registration
loop + cbm_registry_finalize, returns finalized registry.
void cbm_run_<lang>_lsp_cross_with_registry(arena, source, ...,
CBMTypeRegistry *reg, ...)
— skips the per-file build AND the per-file Phase 1b/1c AST scans
(those mutate the registry, which would race on a shared read-
only reg). Trade-off: a few file-local type aliases discovered
only by Phase 1b are missed; in practice the per-file extract
LSP already captured them via embedded_types/field_defs/etc.
Tested with parallel_python_lsp_override_cross_file_emits_lsp_
strategy_edges — fused + filtered + prebuilt path stays correct.
CBMLSPDef gains a CBMLanguage lang field set in pxc_build_lsp_def so
the per-language build functions can filter the global all_defs[].
cbm_parallel_resolve signature now takes CBMCrossLspRegistries* instead
of a single CBMTypeRegistry*. The resolve_worker dispatches:
cbm_pxc_registry_for_lang(lang) returns the right pre-built reg if any;
the worker then switch()es to the matching _with_registry variant.
Languages without a _with_registry variant fall through to the existing
filter + per-file build path.
Kubernetes (Go-heavy, 12k files, 139k defs) measured perf:
parallel_resolve: 5.51 min (strdup-removal baseline) -> 1.86 min
= 2.96x speedup
parallel.resolve.lsp_cross.slow lines: 1,258 -> 2
(-99.8%; the 2 remaining are Python files, not Go)
Peak RSS: 6.29 GB -> 6.23 GB (slight decrease — pre-built registry
shared, no per-file scratch)
All 3,616 tests pass; zod/serilog graphs exact match (per-file path
unaffected).
Cumulative trajectory from the original broken state on kubernetes:
35 min -> 19 min (finalize) -> 6.24 min (module-def filter) ->
5.51 min (strdup-removal) -> 1.86 min (Tier 2 Go) = 18.8x total.
Languages wired so far: Go, Python. C/C++, TS/JS, PHP, C# still use the
per-file filter+build path until their _with_registry variants land in
follow-up commits.
wyhash is a non-cryptographic hash function ~2x faster than FNV1A
for the long string keys used in CBMTypeRegistry's hash buckets
(qualified names average ~50 bytes). The header is unused for now
— prepared for a future Tier 3 symbol-interning pass that replaces
the QN-string-keyed hashtables with u32-keyed ones (rustc pattern)
where wyhash's lower collision rate matters more.
Source: https://github.com/wangyi-fudan/wyhash (Unlicense)
Vendored as a single header at internal/cbm/vendored/wyhash/wyhash.h.
Two perf changes across the 6 cross-LSP resolvers:
1. cbm_registry_finalize after the per-file registry build (all 6
LSPs). Builds the FNV1A hash buckets so lookups go through the
O(1) hashed path in type_registry.c:154. Previously this was
NEVER called anywhere in the codebase, so every method/type
lookup fell through to the O(N) linear scan — the root cause of
the 35-min kubernetes regression. Lookup count on k8s: ~50 calls
* 110k defs * 11k files = ~64B strcmps under the old fallback.
2. In go/c/py/php registration loops, replace cbm_arena_strdup(arena,
d->X) with d->X directly. The strings are borrowed from the
caller's all_defs[] arena which outlives the cbm_run_X_lsp_cross
call by the whole pipeline lifetime. Saves 5 wasted strdups per
def per file = ~6B mallocs avoided on kubernetes.
TS and CS use a different registration architecture (no inline
strdups in the body — uses a per-file ctx). Only the finalize change
applies there.
Kubernetes wall-time vs prior commit:
- After finalize alone: 35 min -> 19 min (~1.8x)
- After module-def filter (prior commit): -> 6.24 min (~5.6x)
- After strdup-removal (this commit): -> 5.51 min (~6.4x)
- Memory: 7.14 GB -> 6.29 GB peak RSS (-12%)
All 3616 tests pass, zod/serilog graph output exact match.
Previously cbm_pipeline_pass_lsp_cross ran as a separate sequential
pass that re-read every source file from disk and re-parsed every
tree-sitter tree on a single thread — a 50x regression on the
kubernetes Go monorepo (extract 10s, lsp_cross 9+ min when killed).
Three changes:
1. CBMFileResult gains const char *source / int source_len. The
parallel extract worker copies source bytes into result->arena
(per-file 100MB cap, project-wide 2GB atomic cap) so the fused
cross-LSP step can run without disk I/O. Source pointer is
borrowed by downstream readers; arena owns the storage.
2. Cross-LSP work moves into the parallel resolve worker. Each worker
processes its file with the existing parallel infrastructure
(worker pool + work stealing). Slab is reclaimed after each
per-file cross-LSP call to bound peak slab memory. The previously
separate lsp_cross pass entry function is kept for the sequential
pipeline path; the parallel path no longer invokes it.
3. CBMModuleDefIndex (gopls 'package summary' pattern) — an inverted
index module_qn -> list of def indices built once in O(D) before
resolve_worker fires. Per file, cbm_pxc_filter_defs_for_file slices
all_defs down to the file's own_module + imported modules only,
typically 50-100x smaller than the global ~110k defs on kubernetes.
Drops the per-file registry build from O(all_defs) to O(relevant).
Other plumbing: cbm_parallel_resolve signature gains all_defs,
def_count, def_modules, module_def_index params. pipeline.c builds
the index between registry_build and parallel_resolve and frees it
after. pipeline_incremental and tests/test_parallel updated to match.
Helpers in pass_lsp_cross.c exposed via the header: cbm_pxc_has_cross_lsp,
cbm_pxc_collect_all_defs, cbm_pxc_run_one(_ts), cbm_pxc_ts_modes,
cbm_pxc_build_module_def_index, cbm_pxc_filter_defs_for_file.
Kubernetes wall-time: 35 min -> 6.24 min (~5.6x) before the followup
perf commits in this series.
parse_ts_type_text in ts_lsp.c builds a 16-slot CBMType* members[] for
union (|) and intersection (&) types. The in-loop branch correctly
caps additions at mc<15, but the trailing post-loop append that
collects the final member after the last separator was unconditional,
and the subsequent members[mc] = NULL sentinel could then write to
members[16] — one past the array end.
Caught by UBSan on the zod codebase (a TS union with >=16 members
exists). Non-deterministic crash signature under pthread-parallel
extraction (12 workers): exit 134 / SIGABRT with no stack, because
libc's allocator detected the heap corruption and called abort().
Single-fixture-file test suite never exercised the parallel path or
a wide-enough union, so existing ASan/UBSan coverage missed it.
Fix: mirror the in-loop guard on the trailing append (cap at mc<15).
Add regression test tslsp_union_many_members_no_overflow that
constructs a 20-member string-literal union — extracting must not
crash. UBSan: clean. Full suite: 3616 passed, 0 failed.
The cli_upsert_claude_hook_fresh / _existing tests still asserted the
old matcher 'Grep|Glob|Read', missed by the previous hook-augment
commit. Update them to mirror what scripts/smoke-test.sh now enforces:
exact matcher value '"Grep|Glob"' present AND 'Glob|Read' absent.
The second assertion regress-protects against re-introducing Read
into the matcher (issue #362).
The previous PreToolUse hook gated Grep/Glob/Read/Search with 'exit 2'
on the first call per session, which broke Claude Code's
read-before-edit invariant (issue #362) and could deny tool calls under
upgrade/missing-binary failure modes.
Replace it with a structurally non-blocking augmenter:
- New 'codebase-memory-mcp hook-augment' subcommand reads the hook JSON
from stdin and, for Grep/Glob, queries search_graph (in-process, no
shell) and emits hookSpecificOutput.additionalContext. Every failure
path (no project, short token, missing binary, slow query, timeout)
exits 0 with no stdout — the hook physically cannot block a tool call.
- 300 ms SIGALRM/_exit(0) in-process deadline; 5 s settings.json timeout
backstop. Output is written exactly once at the very end, so a
mid-work timeout yields a clean no-op (never partial JSON).
- Matcher narrowed to 'Grep|Glob' (Read explicitly excluded) for Claude;
Gemini matcher narrowed to 'google_search|grep_search' (excludes
read_file) for the same reason.
- The installed shim is a thin wrapper that delegates to the binary;
legacy filename 'cbm-code-discovery-gate' is kept so existing
settings.json entries upgrade with zero migration. Installer refuses
to embed binary paths containing a double quote (shim injection
defense).
- Per-agent 'old matchers' lists let upsert/remove clean up historical
matcher strings during upgrade.
- Smoke tests (8d/8e/8l) updated to assert the new behavior and
regress-test against re-introducing Read in the matcher or 'exit 2'
in the shim.
- Session reminder text updated: 'always Read a file before editing it'
replaces the prior 'fall back to Read only for text content'.
(cherry picked from commit f72c8e68c4d91e52911a569a967ad782ce5472b2)
build_grep_cmd's POSIX scoped path uses xargs to pass indexed files
to grep. When xargs supplies a single file, BSD/GNU grep omit the
filename from output by default (only added with multiple files),
yielding 'linenum:content' instead of the 'path:linenum:content'
that collect_grep_matches requires — matches were silently discarded
and search_code returned zero results.
Add -H so grep always prefixes the filename regardless of file count.
Harmless for the common multi-file case (output is unchanged), fixes
the single-indexed-file edge case exposed by the search_code_multi_word
test fixture.
Add a C# language LSP resolver mirroring the existing Go/C/PHP/Python
modules: type evaluation, using-directive resolution, method lookup,
and a corpus-seeded .NET BCL stdlib data table.
- internal/cbm/lsp/cs_lsp.{c,h}: resolver implementation
- internal/cbm/lsp/generated/cs_stdlib_data.c: BCL type/method tables
- wire CBM_LANG_CSHARP dispatch in cbm.c and lsp_all.c
- register cs_lsp + cs_lsp_bench suites in the test harness
- tests/test_cs_lsp.c, tests/test_cs_lsp_bench.c
(cherry picked from commit 8f677c67a0c9d7829cd6ac27393de2d0aaaac534)
cbm_store_checkpoint() invoked SQLITE_CHECKPOINT_TRUNCATE, the most
aggressive mode. When two cbm-mcp processes share a cache dir, one
process's TRUNCATE can shrink files while another has them mmap'd,
raising SIGBUS on macOS. PASSIVE never blocks readers and never
ftruncate()s either file; SQLite still autocheckpoints in PASSIVE
mode at 1000-page boundaries, so reclamation is unaffected for
single-process users.
Recommended by SQLite docs for shared databases:
https://www.sqlite.org/pragma.html#pragma_wal_checkpoint
Hard-coded `PRAGMA mmap_size = 67108864` in configure_pragmas() left
no path for users running multiple cbm-mcp instances against the
same cache to opt out of memory-mapped I/O. On macOS, when one
instance's checkpoint or reindex truncates the DB file under
another instance's live mmap, accessing the now-missing pages
raises SIGBUS, taking the process down.
Setting CBM_SQLITE_MMAP_SIZE=0 reverts to read()/pread() I/O,
which returns recoverable SQLITE_IOERR instead of crashing the
process.
- Default unchanged (67108864 / 64 MB). No behavior change for
single-instance users.
- Malformed values (non-numeric, partial-numeric) fall back to
the default rather than failing the store open.
- Negative values clamp to 0.
- New tests: tests/test_store_pragmas.c covers all five resolver
paths plus an integration smoke that opens a file-backed store
with mmap disabled.
Empirical evidence: 9 SIGBUS crash reports collected on macOS
arm64 v0.6.0 in a 14-hour window, all signature 'cluster_pagein
past EOF' with stacks bottoming in SQLite btree code under the
watcher thread's incremental-index pipeline.
Replace the fragile NOT IN list of individual characters with a range-based BETWEEN 'A' AND 'Z' check for validating project root paths.
The previous query only excluded letters A–H, leaving paths starting with I–Z silently invalid.
In Windows, all the alphabet can be used.
A Windows drive starting with letters I-Z breaks the saving of the database.
The Windows search path landed in 82a9052 wraps shell args in cmd.exe-
level "powershell -Command \"...'%s'...\"". PowerShell's single quotes
hold the inner '%s' interpolation, but a literal " inside the user-
supplied value can close the cmd.exe outer quote. With ' ; | & $ `
already blocked, that wouldn't reach RCE on its own — but < and > were
unblocked, so a quote-break followed by cmd.exe redirection (e.g.
*">C:\evil.txt") would expose a file-write primitive.
Block " < > unconditionally. The validator's contract is "safe inside
single quotes for shell interpolation"; on POSIX these aren't strictly
necessary (single quotes hold), but on Windows the cmd.exe→powershell
wrapping makes them load-bearing, and unconditional blocking keeps the
validator simple — no ifdef-laddered policy and no surprises if a future
caller invokes it from a different shell context.
Three tests added to tests/test_security.c covering each new char.
The .m extension was missing from EXT_TABLE, causing detect_file_language
to return CBM_LANG_COUNT (Unknown) before the content-based disambiguation
logic (cbm_disambiguate_m) could ever run. This made .m files invisible
to the indexer.
Add {".m", CBM_LANG_MATLAB} to EXT_TABLE so the initial lookup succeeds,
allowing the subsequent cbm_disambiguate_m call to correctly determine
whether the file is MATLAB, Objective-C, or Magma.
Fixes#296
When pattern contains spaces and is not already a regex, convert to a
regex with '.*' between words: "foo bar baz" → "foo.*bar.*baz".
This matches all terms in order on a line instead of requiring the
exact multi-word substring.
Metacharacters in user input are escaped so the conversion is safe
for arbitrary text.
Includes test: search_code_multi_word verifies that a two-word query
matches a line containing both words non-contiguously.
98d5cac silently dropped index cells that exceed a single SQLite page,
which is the correct semantic (the index B-tree has no overflow page
mechanism, unlike the record B-tree fixed in #175). But silent drops
make the failure mode invisible — a caller hitting deeply nested
namespaces or very long file paths gets a partial index with no
indication anything was skipped.
Add an fprintf(stderr) at the skip site reporting the cell length and
its index, matching the existing logging style in this file (lines 311,
1305) so the dropped cells are observable for diagnosis. Consistent
with the cbm_write_db: prefix used by surrounding warnings.
write_index_btree() flushed the page buffer whenever the next cell
didn't fit, on the assumption that a freshly-initialised page would
always accept any cell. That invariant fails for index cells whose
payload (qualified_name, file_path, etc.) is larger than a full
SQLite page — after the flush, pb_cell_fits() is still false but the
code calls pb_add_cell() anyway. The subsequent content_offset
underflow corrupts the page header and the next write triggers
SIGBUS on large repos (observed on a Ruby codebase with ~215K nodes).
Re-check pb_cell_fits() after the flush and continue past cells
that still don't fit. Index entries whose keys exceed a full page
can't be stored and are not expected to survive the writer anyway —
the rest of the index is correctly preserved and the indexer no
longer crashes. Record overflow pages (#175) already covers the
record-btree side of this same shape of bug.
Martin Vogel