The full-repo manifest walk added for workspace-import resolution hung
the Windows CI test-runner (pipeline_structure_nodes): pkgmap_walk_dir
uses plain stat() on Windows (no lstat/S_ISLNK), so directory junctions
in the walked tree can be followed into cycles, never terminating.
Make cbm_pkgmap_scan_repo a no-op on Windows and compile pkgmap_walk_dir
only on POSIX. Windows falls back to the files[]-based pkgmap (the
pre-batch baseline behavior). Workspace-import resolution from
discovery-ignored manifests (package.json) is therefore Linux/macOS-only
until the walk is made junction-safe on Windows (follow-up).
The CBM_WORKERS env tests called bare setenv/unsetenv, which are
POSIX-only and undeclared on the MSYS2 Windows test build. Switch to the
cbm_setenv/cbm_unsetenv compat wrappers (a no-op #define to setenv on
POSIX) and include foundation/compat.h. Fixes the test-windows build.
- pass_pkgmap: lstat/S_ISLNK are POSIX-only and undeclared on the MSYS2
Windows build. Guard them under #ifndef _WIN32 (plain stat on Windows,
where symlinks aren't a concern here). Fixes the Windows build break
introduced by the workspace-manifest walker.
- Run clang-format-20 over str_util.c, store.c, mcp.c, pipeline.c,
pass_route_nodes.c, pass_pkgmap.c so the recently-landed pkgmap / route /
security passes satisfy the lint gate.
- cbm_arena_alloc returns NULL on a NULL arena (both arena.c copies)
instead of dereferencing it — defense-in-depth against the NULL-arena
type-allocation path that could crash the LSP type layer.
- discover: add "vendored" to the always-skip directory list.
Distilled from #374, taking the parts not already on main. The PR's
ts_lsp tuple-arena allocation, the C/C++ template formal-count clamp,
and the type_rep unbound-param preservation it also carried all landed
independently in v0.7.0 / via #322 / #360, so only these two
defensive improvements remain. Relates to #390.
Two structural fixes for TS monorepos (pnpm/yarn workspaces) and
SvelteKit projects, measured to take a 885-file SvelteKit+Turborepo repo
from 319 edges to 31802 (100x) with no fixture regressions.
- pkgmap: package.json / composer.json live in IGNORED_JSON_FILES, so the
manifest-driven pkgmap builders never saw them and every
'@my/pkg' workspace import was silently dropped. Add a symlink-safe,
skip-dir-aware filesystem walker (cbm_pkgmap_scan_repo /
cbm_pkgmap_build_from_repo) that harvests manifests directly from the
repo regardless of the discovery filter. Sequential and parallel paths
both feed it now.
- definitions: extract every file's defs (creating Module nodes) BEFORE
resolving imports, so a workspace import in the first file can resolve
against the complete in-memory graph (two-phase, cache-backed).
- routes: synthesise Route nodes + HANDLES edges from SvelteKit's
filesystem layout (+server / +page.server / +layout.server), which has
no app.get(...) call site for pass_calls to pick up.
Distilled from #369 onto current main. The PR's CBM_DISABLE_LSP_CROSS
escape hatch was rebased onto the v0.7.0 fused cross-LSP architecture:
it now gates run_cross_lsp (NULL all_defs makes the fused resolver no-op
cross-file resolution) instead of wrapping the removed standalone
pass_lsp_cross call. Relates to #271 / #56.
Replace ANSI Windows APIs (FindFirstFileA, CreateFileA, etc.) with
wide-char (W) counterparts. Add UTF-8/UTF-16 conversion helpers.
Fix stat() calls in discover pipeline to use _wstat64.
Fixes#357
Add WAL checkpoint on store close and after WAL-mode enable at startup.
This ensures graceful shutdown leaves a clean WAL, and crash recovery
merges stale WAL on next open. Both use PASSIVE mode (non-blocking,
no ftruncate). Best-effort — silently skip if concurrent reader holds
a lock.
Fixes#277
Co-authored-by: HC Cheng <hz.chang@pm.me>
Inside a container, sysconf(_SC_NPROCESSORS_ONLN) and sysinfo() report
the host's CPU count and RAM, not the cgroup's effective quota — so
cbm_default_worker_count over-provisions workers and the mmap budget can
exceed the cgroup memory cap, driving OOMKills (#363).
detect_system_linux now reads the cgroup limits:
- cbm_detect_cgroup_cpus: cgroup v2 cpu.max, then v1 cpu.cfs_quota_us /
cpu.cfs_period_us → ceil(quota/period); -1 when unlimited.
- cbm_detect_cgroup_mem: cgroup v2 memory.max, then v1
memory.limit_in_bytes; 0 when unlimited or the v1 near-ULLONG_MAX
sentinel.
- Effective value is min(cgroup, host), guarding against mis-mounted
cgroups that report more than the host.
Helpers read via a bounded read_small_file (fopen "re", capped fread)
and are exposed through system_info_internal.h for unit tests that drive
them against a fake cgroup tree. The test teardown uses opendir/unlink/
rmdir (no shell spawn).
Distilled from #365 onto current main (unioned the test additions with
the CBM_WORKERS tests from #364, and replaced the test-cleanup
system("rm -rf") with a shell-free recursive remove). Closes#363
together with #364.
Adds a CBM_WORKERS env knob (clamped to [1, 256], invalid values warned
and ignored) that explicitly sets the parallel-indexing worker count.
In containers, sysconf(_SC_NPROCESSORS_ONLN) reports host CPUs rather
than the cgroup's effective quota, so a 1-vCPU pod on a 16-core node
otherwise spawns ~16 workers — the dominant OOMKill driver. When unset,
behaviour is unchanged.
From #364.
Root cause: unbound template parameters flowed through cbm_type_substitute
as NULL, and call-expression return handling then dereferenced the NULL
substituted type. Reproduced on MangoHud's src/hud_elements.cpp.
- type_rep: cbm_type_substitute now returns the original param `t` (not
NULL) when a type arg is unbound, in both the TYPE_PARAM and named-param
paths — unbound params stay intact instead of poisoning the type.
- c_lsp: guard `if (!ret) return cbm_type_unknown();` before unwrapping
the substituted return type; guard the substitute result before use; add
template_function call-node resolution (qualified/scoped/plain names).
- extract_defs: factor template-inner lookup into find_cpp_template_inner_node,
which recurses into nested template_declarations and also handles
field_declaration; reuse it from unwrap_template_inner.
- tests: clsp_nocrash_template_function_multi_param_nested_call (the nested
reproducer) and typerep_substitute_unbound_param_preserved.
Distilled from #360 onto current main. The formal-parameter-count clamp
that overlapped this PR already landed via #322, so only the additive
crash fixes are taken here. Contributes to the C++ stability cluster (#390;
relates to #215/#312).
- str_util: add cbm_validate_project_name() — strict allowlist
([A-Za-z0-9._-], no leading dot, no '..', no path separators) so a
project name can never escape the cache directory.
- store/mcp/http_server: validate project names (fail-closed) before
interpolating them into .db file paths (path-traversal hardening;
defense-in-depth for the wrong-path class in #331).
- store: fix use-after-free in the LIKE pool — like_pool_add() frees its
argument when the pool is full, so capture pool_was_full before the add
and skip the subsequent bind that would touch the freed pointer (two
call sites). Also fix the find_edges_by_url_path LIKE pattern (stray
double-percent) and add NULL guards to cbm_store_checkpoint /
cbm_store_node_degree / cbm_store_restore_from.
- mcp: emit "result": null when a JSON-RPC response would otherwise
carry neither result nor error (2.0 compliance); bounds-check the
project-name accumulator against truncation.
- http_server: JSON-escape directory and parent names in the browse
endpoint to prevent stored-XSS/JSON injection via crafted names.
- cypher: expr_free now recurses on EXPR_FREE_STACK overflow instead of
silently dropping subtrees (memory leak).
Distilled from #352 onto current main (store.c like-pool hunk re-resolved
against the v0.7.0 search refactor; restored cypher.c EOF newline).
The 8 channel-extraction DFS walkers (Python consts, Python, Go, Java,
C#, Ruby, Elixir, Rust) used fixed-size `TSNode stack[CHAN_STACK_CAP]`
arrays whose push loops guarded with `&& top < CHAN_STACK_CAP`. On
deeply-nested ASTs that guard silently dropped children, and an
unguarded overflow could corrupt adjacent stack memory — surfacing as a
segfault in a later pass (lsp_cross / semantic_edges) on large repos.
Convert all 8 to the arena-backed growable TSNodeStack
(ts_nstack_init/push/pop), matching the pattern already used in
extract_calls.c / extract_imports.c (PR #217). No silent drops, no
overflow.
Distilled from #339 (trailing-newline cleanup only; conversion is the
author's).
The git-history popen command used `cd '%s' && git log ... 2>/dev/null`,
which fails on Windows cmd.exe: single quotes are not recognized and
/dev/null is not a valid path, producing "The filename, directory name,
or volume label syntax is incorrect" at the git-history step of a full
index. Switch to `git -C "%s" log ... 2>NUL` (POSIX: 2>/dev/null),
which is valid on both cmd.exe and POSIX shells. Double-quoting the path
is safe because cbm_validate_shell_arg already rejects ", $, backtick,
backslash and the other active metacharacters before the command is built.
Distilled from #325 onto current main (preserves the :%%ct commit-time
field added since the PR branch). Closes#324.
Adds a per-file expression-eval step budget (10k) to the C/C++ LSP resolver so pathological recovery-mode ASTs degrade to unknown instead of hanging indexing (the Bitcoin-derived C++ repro). Verified locally: build clean, all 3,617 tests pass including the full clsp_ suite.
Fixes the OOB read in c_resolve_pending_template_calls when a call site has more arguments than the parsed function signature (issue #215). Verified locally: build clean, all 3,617 tests pass including the new clsp_nocrash_template_extra_call_args regression.
Route Claude Code config paths (skills, .mcp.json, .claude.json,
settings.json, hook scripts) and agent detection through
CLAUDE_CONFIG_DIR-aware helpers, falling back to ~/.claude. Hook command
strings written to settings.json keep the legacy tilde form when the env
var is unset, so existing configs stay portable across HOME values. Prints
a one-line migration nudge when CLAUDE_CONFIG_DIR is set and a legacy
~/.claude tree still exists.
Adds cli_detect_agents_finds_claude_via_env and isolates CLAUDE_CONFIG_DIR
in the existing detection tests so the runner env can't leak in.
Distilled from #321 onto current main (adapts to the v0.7.0 non-blocking
augmenter hook signature, which the original branch predated). Closes#320.
- New top-level "Hybrid LSP" section above "Language Support" with a
short explanation of why tree-sitter alone is insufficient, what the
clean-room LSP layer does, a per-language coverage table (Python,
TS/JS/JSX/TSX, PHP, C#, Go, C/C++) marking what's new in v0.7.0 and
what was sharpened, and the two-layer architecture.
- Hero paragraph + indexing-pipeline bullet now list the full v0.7.0
Hybrid-LSP language set and link to the new section.
- Tests badge bumped 2812 → 3615 (from the ubuntu-latest CI log of the
published v0.7.0 release run).
- New "Hybrid LSP — 6 languages" badge linking to the new section.
- VirusTotal click-through badge + per-binary scans table refreshed to
the v0.7.0 binaries (linux-amd64/arm64, darwin-amd64/arm64,
windows-amd64) using the hashes from the published release body.
The release.yml workflow publishes both wrappers as-is from the repo;
they were still pinned to 0.6.1, which made the npm publish step try to
republish v0.6.1 instead of v0.7.0.
The LSP-cross loop in parallel_resolve called cbm_slab_reclaim() without
first destroying the thread-local tree-sitter parser, violating the
documented slab contract. The parser's lexer holds slab-allocated state
(notably lexer.included_ranges, a 24-byte allocation that lands in the
≤64B slab bucket); reclaiming the slab leaves that pointer dangling.
Workers don't trip on it because they don't reparse afterward, but the
main thread participates in cbm_parallel_for and any later cbm_extract_file
on the main thread — including the sequential pass_definitions in the
incremental pipeline — hits a heap-use-after-free in ts_lexer_goto on
macOS-ASan.
Two-part fix:
- pass_parallel.c: destroy the thread parser before cbm_slab_reclaim() in
the per-file LSP-cross loop, matching the worker extract loop.
- pass_definitions.c: defensively drop the thread parser at pass entry so
the incremental sequential path cannot inherit stale state from any
prior run.
GCC's preprocessor doesn't short-circuit 'defined(__has_feature) && __has_feature(...)' — it tries to parse the call syntax of the undefined token and errors with 'missing binary operator before token "("'. Modern GCC and Clang both define __SANITIZE_ADDRESS__ under -fsanitize=address, so use that single check.
Free the result before the time-budget assert (no leak on miss) and scale the budget ×10 when ASan/UBSan is active, so the perf benches run cleanly in the release CI (skip_perf=false) instead of only being gated by CBM_SKIP_PERF in dry-run.
Same fix as read_file in pass_definitions: the calls/usages/semantic/k8s/lsp_cross passes each read a source file into a malloc'd buffer and re-parse it, so they hit the same tree-sitter lexer over-read past EOF under ASan. Over-allocate by 16 and zero the tail in every reader.
read_file allocated size+1, but tree-sitter's lexer peeks a few bytes past the final UTF-8 char (ts_lexer_goto), reading past the buffer end. Under ASan this is a heap-buffer-overflow (intermittently flagged in the incremental suite on macOS); in production it is a benign over-read of adjacent heap. Over-allocate by 16 bytes and zero the tail so the lookahead stays in-bounds.
cbm_mkdtemp rewrites the template in place to the translated %TEMP% path on Windows, which overflowed the 23-byte 'char tmpl[]' literal (ASan stack-buffer-overflow). Use a 256-byte buffer as the cbm_mkdtemp contract requires (matching the other test callers).
Four tests hardcoded POSIX /tmp/ paths that don't exist on Windows: test_path_alias used raw mkdtemp("/tmp/...") and the store pragma/checkpoint tests built /tmp/*.db directly. Switch to cbm_mkdtemp (translates /tmp/ -> %TEMP% on Windows) and cbm_tmpdir() (matching test_store_bulk). build/smoke/soak-windows already pass; this makes the Windows unit-test job green too.
system_info.c routed Windows into the '#else /* Linux */' detection branch (sysconf/sysinfo, unavailable on msys2-clang) because the platform chain only special-cased __APPLE__/BSD; restrict the Linux branch with '#elif !defined(_WIN32)' so Windows uses detect_system_windows() exclusively.
path_alias.c used raw strndup (undeclared on msys2-clang); switch to the cbm_strndup compat shim and include foundation/compat.h.
The py/cs LSP benchmark tests have time-budget asserts (<150ms / <200ms) that ASan+UBSan blow on CI (750-900ms on arm64), so they bailed before cbm_free_result and LeakSanitizer flagged the leaked arena/parser. Gate them behind CBM_SKIP_PERF (set by the dry-run) like the incremental perf tests, so they run only when perf is enabled.
Fix Windows (msys2-clang) compile errors in test code by using the existing compat shims: setenv/unsetenv -> cbm_setenv/cbm_unsetenv (test_store_pragmas, test_ts_lsp), strndup -> cbm_strndup and 2-arg mkdir -> cbm_mkdir (test_path_alias), and guard the utimensat mtime-bump in test_pipeline behind #ifndef _WIN32.
ws 8.19.0 had a moderate uninitialized-memory-disclosure issue in websocket.close() (fixed in 8.20.1). Transitive dev dependency; the existing ^8.18.0 range already permits the fix, so this is a lockfile-only bump. npm audit now reports 0 vulnerabilities.
Format drift had accumulated across recent commits (the dry-run lint enforces clang-format-20, which had not run on these commits yet). No functional changes — whitespace/layout only, verified by a clean cppcheck + clang-format + NOLINT lint-ci pass.
cppcheck (warning+style, error-exitcode=1): guard cbm_fqn_compute against a NULL project/rel_path (ctunullpointer reachable from cbm_extract_file), and narrow prefix_len/suffix_len (path_alias) and ft_count (pass_githistory) to the scopes that use them.
security-audit.sh: the mcp.c file-read count grew to 13 (search/ADR/Windows-support reads — all path-contained or transport reads, audited, no new exfiltration surface); bump the reviewed maximum and document the update-check + request-body reads. Allow the diagnostics.c atomic metrics dump (.tmp+rename) in the file-write scan, and allow-list the sqlite WAL-checkpoint doc URL referenced in a store.c comment.
The per-file and cross-file LSP type-aware call/usage resolution now runs in every index mode (full/moderate/fast) instead of being gated behind a separate CBM_MODE_ADVANCED. Remove the mode from both index-mode enums, the g_lsp_enabled toggle and its save/restore dance, and the per-mode gating in discover, pipeline and the incremental path.
Now that the LSP walkers are O(n), the callee-attribution accuracy (e.g. ~91k call overrides on roslyn, ~60k on kubernetes) is worth the latency in every mode. Fast mode still filters files and skips similarity/semantic edges, so it stays faster than full while now also benefiting from LSP resolution.
A 'mode':'advanced' request falls back to the full default — identical behaviour to the old advanced mode, so existing callers are unaffected. This also fixes the test_incremental.c failures, which were asserting LSP-resolved results that the gated incremental path never produced.
The advanced-mode LSP passes walked the AST with the same O(i) ts_node_child(node, i) idiom, so a wide root made each per-file pass quadratic — the TS fixture reallyLargeFile.ts took ~16000s. Add a shared cbm_lsp_collect_children() helper (lsp_node_iter.h) and convert the entry and recursive walkers across the TypeScript, C, C#, PHP and Go LSP back-ends to a single O(n) cursor pass. The file now indexes in ~28s; full advanced-mode runs: TS 100s, roslyn 46s, kubernetes 51s, postgres 8s, WordPress 7s.
Also fix two latent crashes exposed once the resolve phase actually ran: process_node indexed the NULL-terminated param_types[] by the call's argument count, reading past the terminator when a call passed more args than the callee declares; and return_type_of() built a tuple via cbm_type_tuple(NULL, ...), passing a NULL arena — it now threads ctx->arena.
This also includes the Tier 2 cross-LSP registry plumbing (shared read-only base registry with per-file overlay/fallback) the walkers build on.
All LSP unit suites pass: TS, C 737/737, C# 60/60, PHP 280/280, Go 55/55.
tree-sitter's ts_node_child(node, i) is O(i) — it walks the child iterator from the first child on every call — so the common 'for (i=0; i<count; i++) ts_node_child(node, i)' idiom is O(n^2) over a node's children. On a file whose top level is very wide (e.g. a 580k-line fixture parsed as hundreds of thousands of sibling comment nodes) this made extraction run for over an hour.
Add ts_nstack_push_children(), which enumerates a node's children in a single O(n) cursor pass, and use it in the whole-tree DFS walkers (calls, channels, env-accesses, imports, type-assigns). Convert the module-level variable walk to a cursor too, and add cbm_is_module_level_p() so callers that already hold the parent node skip the O(n) ts_node_parent() rescan.
Indexing microsoft/TypeScript drops from ~5100s to ~50s with identical graph output.
Same clean def-driven pattern as C/Go. Extracts cs_register_lsp_defs,
adds cbm_cs_build_cross_registry (filters CBM_LANG_CSHARP) +
cbm_run_cs_lsp_cross_with_registry. resolve_worker dispatches
CBM_LANG_CSHARP to the shared registry; pipeline.c builds
cross_registries.cs; pass_lsp_cross.h includes cs_lsp.h.
C# cross-LSP is def-driven (no per-file AST field collection), so the
shared registry yields identical resolution to the per-file build.
roslyn bench (17,916 C# files, ADVANCED): single shared registry
(lsp_cross_prepare=234ms, 260,814 defs, zero slow files), 91,089 LSP
overrides applied, 58.6s total. All 60 test_cs_lsp tests pass; serilog
sanity exact (2,627n/7,112e). No SEGV/ASan.
Brings C/C++/CUDA to Go's optimization level. Extracts the cross-LSP
registration loop into c_register_lsp_defs (shared by per-file and
Tier 2 paths), adds:
cbm_c_build_cross_registry — builds ONE project-wide registry from
all C-family defs (filters by lang) + both C and C++ stdlibs,
finalized, shared READ-ONLY across resolve workers.
cbm_run_c_lsp_cross_with_registry — per-file resolve using the
shared registry; no per-file registry rebuild.
C cross-LSP is DEF-DRIVEN (fields/returns/embeds come from the def
strings, no per-file AST field collection), so the shared registry
produces identical resolution to the per-file build — zero quality
loss. resolve_worker dispatches CBM_LANG_C/CPP/CUDA to the new path;
pipeline.c builds cross_registries.c; pass_lsp_cross.h includes c_lsp.h.
postgres bench (4,967 files, ADVANCED mode): 628 C files share one
pre-built registry (lsp_cross_prepare=17ms, zero slow files), 44 LSP
overrides, +9 edges vs FULL. All 200 test_c_lsp tests pass. No
SEGV/ASan. Total pipeline overhead vs FULL: +5.2%.
New CBM_MODE_ADVANCED=3. Both per-file LSP (type-aware resolution
during extract) and the cross-file LSP pass (during resolve) are now
OPT-IN: FULL and below run tree-sitter extraction + registry textual
resolution only; ADVANCED adds the full LSP layer.
Rationale: profiling showed LSP changes the kubernetes graph by only
~0.01% of edges (it mostly re-confirms textual resolution) while
costing significant memory + time. Making it opt-in keeps the default
FULL path lean (3.0 GB vs 5.6 GB peak RSS on K8s) without meaningfully
worse output, and lets users pay for LSP precision only when wanted.
Mechanism:
- cbm.c: g_lsp_enabled atomic toggle wraps the entire per-file LSP
block (Go/C/C++/PHP/Python/TS/JS/C#) + the C/C++ preprocessor-pass
LSP call. Default true so ad-hoc extraction / tests keep historical
behaviour. cbm_set_lsp_enabled / cbm_get_lsp_enabled in cbm.h.
- pipeline.c: sets the toggle (save→set→restore so it never leaks to
other callers) from p->mode before extraction; gates the cross-LSP
all_defs collection + registry build on advanced; gates the
sequential lsp_cross dispatch. Fixed the moderate_only comparison
(== FAST, not > MODERATE) so ADVANCED is not treated as lighter.
- pipeline_incremental.c: same toggle save/set/restore.
- discover.h/.c: added CBM_MODE_ADVANCED to discover's enum + all 4
filter comparisons treat ADVANCED like FULL (full discovery, no
fast-filtering). Without this, ADVANCED (numerically 3) was
mis-detected as fast-mode and dropped ~half the files.
- mcp.c: parse mode "advanced" + tool schema enum/description.
Dual-mode K8s bench: FULL 46.7s (lsp_overrides=0, 3.0 GB) vs
ADVANCED 48.8s (lsp_overrides=80,818, 5.6 GB). Graphs within 0.01%.
All 3,615 tests pass.
Replaces the prior Robin-Hood open-addressing implementation with
Verstable (https://github.com/JacksonAllan/Verstable), a 2024 state-
of-the-art C hash table. The public API in foundation/hash_table.h
is UNCHANGED — CBMHashTable is now an opaque type and all 14
production callers (graph_buffer, registry, pipeline passes,
service_patterns, watcher, semantic, etc.) get the new impl
transparently. str_intern.c keeps its specialised (ptr+len) hash —
different semantics, not a clean swap target.
Why: per-sub-phase profiling pinpointed cbm_pipeline_lsp_target_node
and cbm_registry_resolve as the dominant resolve_calls cost. Both
ultimately hit cbm_ht_get; both run millions of times on a kubernetes
index. Verstable's key innovations target exactly this:
* 4-bit hash-fragment metadata in a uint16_t sidecar array lets
chain walks skip most key comparisons WITHOUT touching the
bucket payload (cache wins on hot lookups).
* Quadratic probing with per-bucket displacement chains (not linear
Robin-Hood psl).
* Tombstone-free deletion that moves at most one existing key.
Vendored at internal/cbm/vendored/verstable/verstable.h (2069 lines,
84KB, MIT). Wrapped as a const char* → void* template inside
hash_table.c. The header struct is opaque (forward declaration only);
audited the entire tree — no caller accesses internals or takes
sizeof(CBMHashTable), so opacity is safe.
Test refactor: test_hash_table.c had two impl-specific tests:
* ht_robin_hood_bounded_psl — Robin-Hood specific; removed.
ht_stress_10000 covers the functional equivalent.
* ht_power_of_two_capacity — renamed ht_create_small_capacity,
replaced struct probes with a functional insert+lookup check.
ht_create_capacity_zero/one similarly de-internalised.
Benchmark on kubernetes (12 workers, 20,650 files):
- parallel_resolve wall: 36,376 -> 27,217 ms (-25.2%)
- pipeline.done total: 61,485 -> 51,745 ms (-15.8%)
- registry_build: ~1,000 -> 779 ms (faster — registry uses
cbm_ht_set heavily during build)
- All 3,615 tests pass; hash_table suite 100% green
- ASan/UBSan/SEGV: none
- Peak RSS: ~6.83 GB
- Graph: nodes 297,082 -> 297,102 (+20, 0.007%), edges 1,155,444 ->
1,154,025 (-1,419, -0.12%) — within tolerance; minor delta likely
from hash-iteration-order differences affecting tie-breaks in
resolution. Sanity (zod/serilog): nodes exact, edges within 0.1%.
Cumulative trajectory from broken-state baseline ~35 min on K8s ->
51.7s wall = ~40x total speedup.
THE big win. Pinpoint-instrumented profiling showed 98.7% of
resolve_calls CPU on kubernetes (881s out of 893s) lived inside the
'resolve' sub-step — cbm_pipeline_lsp_target_node when LSP hits,
cbm_registry_resolve strategy chain when it misses. The LSP path
is fast; the strategy chain isn't.
cbm_registry_resolve's strategy chain (import_map -> same_module ->
unique_name -> suffix_match+scoring) is deterministic for a given
(callee_name, module_qn) pair. module_qn is CONSTANT within a file,
so within one file the same callee_name resolves identically every
time. K8s files have ~200 calls but only ~50 unique callee_names —
so ~75% of strategy-chain walks are redundant.
Added a TLS hash cache (_resolve_cache, registry.c) keyed by
callee_name with value cbm_resolution_t. cbm_registry_resolve checks
the cache first; on miss it runs the full chain and caches BOTH
positive and negative results (caching the negative is just as
valuable — same name asks the same question).
API in pipeline.h:
void cbm_registry_resolve_cache_begin(int estimated_capacity);
void cbm_registry_resolve_cache_end(void);
Two smaller chunks landed in the same commit:
- src/pipeline/registry.c: per-file import-map prefix -> module-qn
hash (cbm_registry_import_map_cache_*). resolve_import_map's
linear strcmp scan becomes O(1). Flat impact in this run because
the LSP path bypasses cbm_registry_resolve for most K8s calls,
but useful for projects without strong LSP coverage.
- src/pipeline/pass_parallel.c: reuse lsp_target as target_node
when try_field_type_hint didn't override the QN. Saves 1
cbm_gbuf_find_by_qn lookup per LSP-resolved call.
Also added 6 sub-counters inside resolve_file_calls (time_ns_rc_*)
logged as parallel.resolve.calls_breakdown / breakdown2 to keep
durable telemetry on which sub-step dominates.
Benchmark on kubernetes (12 workers, 20,650 files):
- parallel_resolve wall: 90,531 -> 36,376 ms (-59.8% vs C2; -64.8%
vs original 103,281 baseline)
- resolve_calls CPU: 893,657 -> 277,556 ms (-68.9%)
- 'resolve' sub-counter: 881,650 -> 266,521 ms (-69.8%) ← target hit
- resolve_usages CPU bonus: 10,986 -> 5,594 ms (-49.1%, secondary
effect — fewer ambiguous resolutions to scrub)
- Total pipeline.done: 115,918 -> 61,485 ms (-47%)
- User CPU total: 953.7s -> 375.1s (-60.7%)
- Peak RSS: +430 MB for cached cbm_resolution_t structs (~12 live
caches at a time)
- Tests: 3616/0
- Graph: nodes 297,082 EXACT match, edges +/- 0.2%
- zod/serilog sanity: nodes exact, edges within tolerance
Effective hit rate: predicted 75%, measured ~70%. The ~30% true
cache misses (unique callee_names per file) still run the full
chain — next-tier optimizations: widen scope to per-worker so
common names like 'Get'/'Add' are computed once per worker not
once per file, or pre-filter candidate lists by import reachability
at file entry to shrink the chain's inner cost.
Cumulative trajectory from original broken state: ~35 min K8s cold
index -> 63.6s wall = ~33x total speedup.
cbm_service_pattern_match runs once per resolved CALL edge inside
emit_service_edge — that's 7 pattern lists × ~30 patterns × strstr
= ~210 strstrs per call. On kubernetes (~600k resolved CALL edges)
the SAME resolved QN ("fmt.Errorf", "context.Context.Done",
"errors.New", etc) repeats thousands of times across files. Caching
turns the linear pattern-list scan into one hash lookup after the
first miss.
API additions in service_patterns.h:
void cbm_service_pattern_cache_begin(void);
void cbm_service_pattern_cache_end(void);
Cache is TLS (no contention) scoped to the resolve_worker's lifetime
in the parallel_resolve phase. Capacity 8192 entries per worker —
covers typical project QN diversity without rehashing.
Encoding: enum value + 1 in the void* slot (0/NULL means "miss").
Two helpers svc_enum_to_ptr / svc_ptr_to_enum hide the trick.
Benchmark on kubernetes (12 workers, 20,650 files):
- parallel_resolve wall: 98,520 -> 90,104 ms (-8.5% vs Chunk 1
baseline, -12.8% vs original 103,281 baseline)
- resolve_calls CPU: 992,427 -> 885,663 ms (-10.8% vs Chunk 1,
-5.4% vs original — Chunk 1 added overhead that Chunk 2 recovers
AND surpasses)
- resolve_usages CPU: 11,977 -> 10,944 ms (-8.6%)
- total_loop CPU: 1,005,409 -> 897,561 ms (-10.7%)
- Tests: 3616/0
- Graph: 297,082 -> 297,080 nodes (-2), 1,153,482 -> 1,156,097 edges
(within 0.2%, semantic-preserving)
- zod/serilog sanity: edges within +/-3 of baseline
- Peak RSS: 7.46 -> 7.75 GB (+290 MB for 12 × 8192-entry caches)
Cumulative trajectory: original broken state ~35 min K8s ->
1.92 min wall (-49× total). The per-file is_import_reachable cache
(Chunk 1) + the per-worker service-pattern cache (Chunk 2) together
save ~13s of resolve wall time. Next chunks target the gbuf_insert_
edge hot path and remaining strstr scans.
Wraps is_import_reachable's O(candidates × imports × strstr) hot loop
with a TLS hash-table cache scoped to the resolve_worker's per-file
iteration. Same callee_name appears in many call sites and each
lookup re-evaluates the same ~30 candidates against the same imports
— with the cache, first eval pays the strstr cost, repeats are O(1).
API additions in pipeline.h:
void cbm_registry_reach_cache_begin(int estimated_capacity);
void cbm_registry_reach_cache_end(void);
resolve_worker calls _begin right after build_import_map and _end
right before the loop's free_import_map (the cache scope must match
the import_vals scope — different files have different imports, so
the cache is invalidated between files).
Benchmark on kubernetes (12 workers, 20,650 files):
- parallel_resolve wall: 103,281 -> 98,520 ms (-4.6%)
- resolve_usages CPU: 33,971 -> 11,977 ms (-65%) — cache hits hard here
- resolve_calls CPU: 935,958 -> 992,427 ms (+6.1%) — cache adds overhead
here without saving work, because the Tier 3 LSP override path hits
first and bypasses cbm_registry_resolve for most CALL entries.
Usage/throws/rw paths always go through cbm_registry_resolve so the
cache helps them; the net wall improvement comes from that side.
- Tests: 3616/0
- Graph: 297,082 -> 297,115 nodes (+33), 1,153,482 -> 1,155,005 edges
(within 0.1%, semantic-preserving)
- zod/serilog sanity: nodes exact, edges +/-1 (within tolerance)
The strstr overhead per call wasn't the dominant resolve_calls cost —
the next chunks need to attack the actual hot spots: emit_service_edge's
cbm_service_pattern_match (6 lists × ~30 patterns × strstr per call =
~180 strstrs/call) and the gbuf_insert_edge allocator pressure.
Instrument resolve_worker with per-sub-phase atomic ns counters
(import_map, cross_lsp, resolve_calls, resolve_usages, resolve_throws,
resolve_rw, resolve_semantic) plus a whole-iteration total_loop and
files_visited count. Logged at end of cbm_parallel_resolve as three
log lines (split to avoid the PP_RING=4 TLS-slot aliasing in itoa_log
that masked the data in the first attempt).
Profiling result on kubernetes (12 workers, 20,650 files):
parallel_resolve wall: 103,281 ms
total_loop CPU-ms: 970,352 (78.3% worker utilization, busy)
phase_sum CPU-ms: 970,303 (99.99% coverage of loop body)
resolve_calls: 935,958 96.45% ← THE BOTTLENECK
resolve_usages: 33,971 3.50%
resolve_semantic: 192 0.02%
resolve_calls (cost): 935,958 / 616,795 = 1,517 µs per call edge
resolve_usages (cost): 33,971 / 942,846 = 36 µs per usage edge
Calls are 42× more expensive per edge than usages. The extra work in
resolve_file_calls (vs resolve_file_usages) traces to:
1. is_import_reachable (registry.c:154) — O(imports × 2*strstr) per
candidate × ~30 candidates per name × 1.6M call attempts =
~480 sec of strstr work alone.
2. cbm_service_pattern_match — O(6 lists × ~30 patterns × strstr)
per call = ~48 sec aggregate.
3. try_field_type_hint, emit_service_edge dispatch overhead.
These are now the next optimization targets — replace substring
matching with hashed prefix lookups (trie or hash of module
fragments), and add per-file memoization of is_import_reachable
results so the same candidate doesn't get re-evaluated per call site.
Instrumentation overhead is negligible (~30ns atomic_fetch_add per
phase per file = <100µs aggregate on K8s) and gives durable
telemetry for future perf work.
Two changes:
1. resolve_file_calls now builds a per-file CBMHashTable mapping
'caller_qn|callee_short' -> CBMResolvedCall* once at function
entry, then does O(1) lookups instead of the O(N) linear scan in
cbm_pipeline_find_lsp_resolution. Falls back to the linear scan
only for keys the index misses (preserves all existing semantics).
The hash key uses callee_qn's short name (last dot-separated
segment) to match the original linear-scan's match rule. On insert,
keeps the highest-confidence entry per key.
2. Memory safety: keys must be strdup'd before cbm_ht_set, because
cbm_ht_set stores the key pointer rather than copying. The first
draft used stack-allocated 'char key[1024]' inside the loop body —
ASan caught it as stack-use-after-scope when subsequent cbm_ht_get
calls dereferenced the dangling stack frame addresses. Fixed:
strdup on insert, reuse cbm_ht_get_key when updating an existing
bucket (no double-free), free all keys via cbm_ht_foreach before
cbm_ht_free on cleanup.
Benchmark on kubernetes (12 workers, 20,650 files):
- parallel_resolve: 113,300ms (Tier 3 meta) -> 110,924ms (this commit)
~2.1% improvement — within run-to-run noise. Despite the O(N) -> O(1)
algorithmic improvement, the (calls × resolved_calls) product on
K8s Go files is small enough at this scale that the linear scan was
not the dominant cost. Other call sites of cbm_pipeline_find_lsp_
resolution (e.g. pass_calls.c) keep the original O(N) path since
this file-level index is local to resolve_file_calls.
The change is still net positive:
- Removes a real memory-safety bug found by ASan
- O(N) -> O(1) algorithmic improvement at the call site, even if its
scale impact is modest at K8s size
- Sets up larger codebases (where the linear scan would dominate) to
benefit without further work
Tests: 3,616 / 0 failed including parallel_python_lsp_override_cross_
file_emits_lsp_strategy_edges (which directly exercises the LSP
resolution path). Graph counts: nodes -1 vs baseline, edges +945 vs
baseline (within 0.1%, semantic-preserving). zod/serilog sanity:
nodes exact, edges +4/+5 (within tolerance).
Real bottleneck for the remaining ~110s on K8s is elsewhere — likely
the cumulative cost across 20K files of registry_resolve + gbuf_find_
by_qn + gbuf_insert_edge per resolved entry across all 5 resolve_file_
passes. Profiling needed to confirm; the lightweight per-worker edge
buffer (deferring dedup to merge) is the next obvious attack vector.
DeusData