The gopls 'package summary' pattern, applied properly: build the cross-
LSP CBMTypeRegistry ONCE per language at pipeline time (in a dedicated
cross_lsp_arena), share it READ-ONLY across all resolve workers via the
CBMCrossLspRegistries struct. Per-file work then becomes just tree-
sitter parse + AST walk + O(1) hash lookups — zero per-file registry
rebuilds for languages with a pre-built registry.
Each LSP gains two new public functions:
CBMTypeRegistry *cbm_<lang>_build_cross_registry(arena, defs, def_count)
— filters all_defs to its language, init + stdlib + registration
loop + cbm_registry_finalize, returns finalized registry.
void cbm_run_<lang>_lsp_cross_with_registry(arena, source, ...,
CBMTypeRegistry *reg, ...)
— skips the per-file build AND the per-file Phase 1b/1c AST scans
(those mutate the registry, which would race on a shared read-
only reg). Trade-off: a few file-local type aliases discovered
only by Phase 1b are missed; in practice the per-file extract
LSP already captured them via embedded_types/field_defs/etc.
Tested with parallel_python_lsp_override_cross_file_emits_lsp_
strategy_edges — fused + filtered + prebuilt path stays correct.
CBMLSPDef gains a CBMLanguage lang field set in pxc_build_lsp_def so
the per-language build functions can filter the global all_defs[].
cbm_parallel_resolve signature now takes CBMCrossLspRegistries* instead
of a single CBMTypeRegistry*. The resolve_worker dispatches:
cbm_pxc_registry_for_lang(lang) returns the right pre-built reg if any;
the worker then switch()es to the matching _with_registry variant.
Languages without a _with_registry variant fall through to the existing
filter + per-file build path.
Kubernetes (Go-heavy, 12k files, 139k defs) measured perf:
parallel_resolve: 5.51 min (strdup-removal baseline) -> 1.86 min
= 2.96x speedup
parallel.resolve.lsp_cross.slow lines: 1,258 -> 2
(-99.8%; the 2 remaining are Python files, not Go)
Peak RSS: 6.29 GB -> 6.23 GB (slight decrease — pre-built registry
shared, no per-file scratch)
All 3,616 tests pass; zod/serilog graphs exact match (per-file path
unaffected).
Cumulative trajectory from the original broken state on kubernetes:
35 min -> 19 min (finalize) -> 6.24 min (module-def filter) ->
5.51 min (strdup-removal) -> 1.86 min (Tier 2 Go) = 18.8x total.
Languages wired so far: Go, Python. C/C++, TS/JS, PHP, C# still use the
per-file filter+build path until their _with_registry variants land in
follow-up commits.
Previously cbm_pipeline_pass_lsp_cross ran as a separate sequential
pass that re-read every source file from disk and re-parsed every
tree-sitter tree on a single thread — a 50x regression on the
kubernetes Go monorepo (extract 10s, lsp_cross 9+ min when killed).
Three changes:
1. CBMFileResult gains const char *source / int source_len. The
parallel extract worker copies source bytes into result->arena
(per-file 100MB cap, project-wide 2GB atomic cap) so the fused
cross-LSP step can run without disk I/O. Source pointer is
borrowed by downstream readers; arena owns the storage.
2. Cross-LSP work moves into the parallel resolve worker. Each worker
processes its file with the existing parallel infrastructure
(worker pool + work stealing). Slab is reclaimed after each
per-file cross-LSP call to bound peak slab memory. The previously
separate lsp_cross pass entry function is kept for the sequential
pipeline path; the parallel path no longer invokes it.
3. CBMModuleDefIndex (gopls 'package summary' pattern) — an inverted
index module_qn -> list of def indices built once in O(D) before
resolve_worker fires. Per file, cbm_pxc_filter_defs_for_file slices
all_defs down to the file's own_module + imported modules only,
typically 50-100x smaller than the global ~110k defs on kubernetes.
Drops the per-file registry build from O(all_defs) to O(relevant).
Other plumbing: cbm_parallel_resolve signature gains all_defs,
def_count, def_modules, module_def_index params. pipeline.c builds
the index between registry_build and parallel_resolve and frees it
after. pipeline_incremental and tests/test_parallel updated to match.
Helpers in pass_lsp_cross.c exposed via the header: cbm_pxc_has_cross_lsp,
cbm_pxc_collect_all_defs, cbm_pxc_run_one(_ts), cbm_pxc_ts_modes,
cbm_pxc_build_module_def_index, cbm_pxc_filter_defs_for_file.
Kubernetes wall-time: 35 min -> 6.24 min (~5.6x) before the followup
perf commits in this series.
Per-file LSP (run from cbm_extract_file) only sees one file's defs, so
callees whose receiver type comes from an imported module stayed
unresolved — even though Go/C/Py/TS already had cross-file entry points
(cbm_run_X_lsp_cross). Those entry points existed but no pipeline pass
ever invoked them. PHP didn't even have a cross-file entry point.
This change lands the missing pieces:
1. cbm_run_php_lsp_cross + cbm_batch_php_lsp_cross — PHP cross-file
resolver mirroring py_lsp_cross / ts_lsp_cross. Reuses PHPLSPContext,
php_lsp_init, php_lsp_add_use, php_lsp_process_file. Handles PHP-
specific labels (Class / Interface / Trait / Enum / Type) and uses
def_module_qn so cross-file return types qualify against the def's
own module rather than the importing module's.
2. pass_lsp_cross.c — new pipeline pass. For every cached file result,
builds a project-wide CBMLSPDef[] (CBMDefinition → CBMLSPDef
converter), per-file import map (from gbuf IMPORTS edges), then
dispatches to the matching cbm_run_X_lsp_cross. Output strings get
copied into the per-file arena (with O(N²) memory bug avoided by
running each file inside a fresh scratch arena that gets destroyed
after the LSP call — the LSP allocates its registry + stdlib +
project defs in scratch, not in the caller's persistent arena).
3. Wired into both pipelines:
- sequential: definitions → k8s → lsp_cross → calls → ...
- parallel: parallel_extract → registry_build → lsp_cross →
parallel_resolve → ...
4. Updated tests/test_parallel.c run_parallel helper to mirror the new
parallel ordering so existing cross-file Python/PHP tests now exercise
pass_lsp_cross.
Languages covered: Go, C/C++/CUDA, Python, TS/JS/JSX/TSX, PHP. PHP cross
file is new. The other four had functions but no pipeline integration
until this change.
Verified on a 1670-file FastAPI index: pass_lsp_cross processes 1050
Python files (525 non-LSP files skipped), collects 4070 project defs in
9.2s, and parallel_resolve subsequently reports lsp_overrides=568 — the
LSP-strategy attribution gain over per-file alone.
Per-LSP emit functions dedup by (caller_qn, callee_qn) within their own
run; pxc_append_results dedups again at append time so cross-file
entries duplicating per-file output get dropped instead of inflating
result->resolved_calls.
Per-file py_lsp emits resolved_calls.callee_qn as the raw import-module
path (e.g. `greeter.Greeter` from `from greeter import Greeter`) rather
than the project-qualified QN the gbuf stores (`<project>.greeter.Greeter`).
Before this change, the LSP match succeeded (lsp_overrides counter
incremented) but the downstream cbm_gbuf_find_by_qn lookup missed
silently and the edge was dropped — the failure mode the new
lsp_overrides telemetry was designed to surface.
The LSP layer can't tell in-project imports (qualify) from external
imports (don't qualify, e.g. `os.path`) without consulting the gbuf,
which is built downstream. So normalise at the consumer instead: try
the LSP-emitted QN as-is first; on miss, retry with
`<project>.<callee_qn>`. If that also misses, drop the edge — same as
before, target is external/unindexed.
New helper cbm_pipeline_lsp_target_node in lsp_resolve.h is shared by
both pipelines (sequential pass_calls.c and parallel pass_parallel.c)
so they continue to admit identical sets of LSP overrides. The helper
also canonicalises res.qualified_name to the gbuf node's QN, so
downstream edge property serialisation shows the project-qualified
form even when fallback resolution kicked in.
Cross-file regression test parallel_python_lsp_override_cross_file_-
emits_lsp_strategy_edges pins the two-file Greeter scenario that
originally exposed the bug.
The sequential pass_calls and the parallel cbm_parallel_extract pipelines
each carried their own copy of the LSP-resolved-call lookup with
divergent confidence floors and different match semantics. Most production
indexing runs the parallel path, so projects produced different
CALLS-edge attributions depending on which pipeline mode kicked in:
pass_calls.c find_lsp_resolution floor 0.6 (CBM_LSP_CONFIDENCE_FLOOR)
pass_parallel lsp_override_resolution_pp floor 0.5 (hardcoded)
Both pipelines correctly invoke cbm_run_*_lsp through cbm_extract_file,
so result->resolved_calls is populated identically — the bug was at the
consumer.
Changes
-------
- New src/pipeline/lsp_resolve.h: static-inline cbm_pipeline_find_lsp_resolution
and CBM_LSP_CONFIDENCE_FLOOR (0.6f). Single source of truth for the
match rule (caller_qn equality + callee_qn short-name match + floor),
with rationale and language-coverage notes inline.
- pass_calls.c: include lsp_resolve.h, drop the local copy, rename the
call site to cbm_pipeline_find_lsp_resolution.
- pass_parallel.c: include lsp_resolve.h, drop lsp_override_resolution_pp,
use cbm_pipeline_find_lsp_resolution at resolve_file_calls. Add an
int lsp_overrides counter to resolve_worker_state_t (with adjusted
cache-line padding for the new field), aggregate it per-worker,
and emit it in the parallel.resolve.done log line so divergence
between pipelines becomes observable in production telemetry.
- tests/test_parallel.c: parallel_python_lsp_override_emits_lsp_strategy_edges
builds a single-file Python scenario (Greeter class + main() in one
file), runs the parallel pipeline, walks every CALLS edge, asserts
at least one carries an "lsp_*" strategy in its properties_json.
Pins both halves of the wiring at once: the shared find helper IS
consulted in the parallel path, AND the LSP-attributed strategy
survives all the way to the gbuf edge. Single-file rather than
cross-file because per-file py_lsp's resolved_calls callee_qn uses
the import module path rather than the project-qualified QN, so
cross-file LSP edges currently get dropped by cbm_gbuf_find_by_qn —
a separate cross-file QN-mismatch bug to track elsewhere.
Full suite: 3250 passed, 0 failed (was 3249; +1 new regression test).
- cbm_mkdtemp: _mktemp + _mkdir on Windows
- cbm_setenv/cbm_unsetenv: _putenv_s on Windows
- cbm_pipe: _pipe on Windows
- Replace bare POSIX calls in 12 test files with compat wrappers
- All test files now include compat.h
Enforce WarningsAsErrors: '*' with zero suppressions of entire check categories.
All non-FP warnings fixed at source; true false positives get per-instance NOLINT
with reasoning. Fixes include: braces around statements (1679), isolate declarations
(75), explicit casts for multilevel pointer conversions, missing includes, function
parameter documentation via NOLINT, and proper NOLINTBEGIN/NOLINTEND nesting.
Complete C implementation of the indexing pipeline (src/), parallel worker
pool, graph buffer with merge support, SQLite writer, and 1893-test suite.
Linter setup: zero warnings from clang-tidy (all checks enabled, no NOLINT
suppressions), cppcheck, and clang-format. All issues fixed at source —
proper headers for external linkage, GROW_ARRAY macro restructured to
eliminate type parameter, null-deref paths guarded, named intermediates
for suspicious-argument checks.
Martin Vogel