- cbm_arena_alloc returns NULL on a NULL arena (both arena.c copies)
instead of dereferencing it — defense-in-depth against the NULL-arena
type-allocation path that could crash the LSP type layer.
- discover: add "vendored" to the always-skip directory list.
Distilled from #374, taking the parts not already on main. The PR's
ts_lsp tuple-arena allocation, the C/C++ template formal-count clamp,
and the type_rep unbound-param preservation it also carried all landed
independently in v0.7.0 / via #322 / #360, so only these two
defensive improvements remain. Relates to #390.
Root cause: unbound template parameters flowed through cbm_type_substitute
as NULL, and call-expression return handling then dereferenced the NULL
substituted type. Reproduced on MangoHud's src/hud_elements.cpp.
- type_rep: cbm_type_substitute now returns the original param `t` (not
NULL) when a type arg is unbound, in both the TYPE_PARAM and named-param
paths — unbound params stay intact instead of poisoning the type.
- c_lsp: guard `if (!ret) return cbm_type_unknown();` before unwrapping
the substituted return type; guard the substitute result before use; add
template_function call-node resolution (qualified/scoped/plain names).
- extract_defs: factor template-inner lookup into find_cpp_template_inner_node,
which recurses into nested template_declarations and also handles
field_declaration; reuse it from unwrap_template_inner.
- tests: clsp_nocrash_template_function_multi_param_nested_call (the nested
reproducer) and typerep_substitute_unbound_param_preserved.
Distilled from #360 onto current main. The formal-parameter-count clamp
that overlapped this PR already landed via #322, so only the additive
crash fixes are taken here. Contributes to the C++ stability cluster (#390;
relates to #215/#312).
The 8 channel-extraction DFS walkers (Python consts, Python, Go, Java,
C#, Ruby, Elixir, Rust) used fixed-size `TSNode stack[CHAN_STACK_CAP]`
arrays whose push loops guarded with `&& top < CHAN_STACK_CAP`. On
deeply-nested ASTs that guard silently dropped children, and an
unguarded overflow could corrupt adjacent stack memory — surfacing as a
segfault in a later pass (lsp_cross / semantic_edges) on large repos.
Convert all 8 to the arena-backed growable TSNodeStack
(ts_nstack_init/push/pop), matching the pattern already used in
extract_calls.c / extract_imports.c (PR #217). No silent drops, no
overflow.
Distilled from #339 (trailing-newline cleanup only; conversion is the
author's).
Adds a per-file expression-eval step budget (10k) to the C/C++ LSP resolver so pathological recovery-mode ASTs degrade to unknown instead of hanging indexing (the Bitcoin-derived C++ repro). Verified locally: build clean, all 3,617 tests pass including the full clsp_ suite.
Fixes the OOB read in c_resolve_pending_template_calls when a call site has more arguments than the parsed function signature (issue #215). Verified locally: build clean, all 3,617 tests pass including the new clsp_nocrash_template_extra_call_args regression.
Format drift had accumulated across recent commits (the dry-run lint enforces clang-format-20, which had not run on these commits yet). No functional changes — whitespace/layout only, verified by a clean cppcheck + clang-format + NOLINT lint-ci pass.
cppcheck (warning+style, error-exitcode=1): guard cbm_fqn_compute against a NULL project/rel_path (ctunullpointer reachable from cbm_extract_file), and narrow prefix_len/suffix_len (path_alias) and ft_count (pass_githistory) to the scopes that use them.
security-audit.sh: the mcp.c file-read count grew to 13 (search/ADR/Windows-support reads — all path-contained or transport reads, audited, no new exfiltration surface); bump the reviewed maximum and document the update-check + request-body reads. Allow the diagnostics.c atomic metrics dump (.tmp+rename) in the file-write scan, and allow-list the sqlite WAL-checkpoint doc URL referenced in a store.c comment.
The per-file and cross-file LSP type-aware call/usage resolution now runs in every index mode (full/moderate/fast) instead of being gated behind a separate CBM_MODE_ADVANCED. Remove the mode from both index-mode enums, the g_lsp_enabled toggle and its save/restore dance, and the per-mode gating in discover, pipeline and the incremental path.
Now that the LSP walkers are O(n), the callee-attribution accuracy (e.g. ~91k call overrides on roslyn, ~60k on kubernetes) is worth the latency in every mode. Fast mode still filters files and skips similarity/semantic edges, so it stays faster than full while now also benefiting from LSP resolution.
A 'mode':'advanced' request falls back to the full default — identical behaviour to the old advanced mode, so existing callers are unaffected. This also fixes the test_incremental.c failures, which were asserting LSP-resolved results that the gated incremental path never produced.
The advanced-mode LSP passes walked the AST with the same O(i) ts_node_child(node, i) idiom, so a wide root made each per-file pass quadratic — the TS fixture reallyLargeFile.ts took ~16000s. Add a shared cbm_lsp_collect_children() helper (lsp_node_iter.h) and convert the entry and recursive walkers across the TypeScript, C, C#, PHP and Go LSP back-ends to a single O(n) cursor pass. The file now indexes in ~28s; full advanced-mode runs: TS 100s, roslyn 46s, kubernetes 51s, postgres 8s, WordPress 7s.
Also fix two latent crashes exposed once the resolve phase actually ran: process_node indexed the NULL-terminated param_types[] by the call's argument count, reading past the terminator when a call passed more args than the callee declares; and return_type_of() built a tuple via cbm_type_tuple(NULL, ...), passing a NULL arena — it now threads ctx->arena.
This also includes the Tier 2 cross-LSP registry plumbing (shared read-only base registry with per-file overlay/fallback) the walkers build on.
All LSP unit suites pass: TS, C 737/737, C# 60/60, PHP 280/280, Go 55/55.
tree-sitter's ts_node_child(node, i) is O(i) — it walks the child iterator from the first child on every call — so the common 'for (i=0; i<count; i++) ts_node_child(node, i)' idiom is O(n^2) over a node's children. On a file whose top level is very wide (e.g. a 580k-line fixture parsed as hundreds of thousands of sibling comment nodes) this made extraction run for over an hour.
Add ts_nstack_push_children(), which enumerates a node's children in a single O(n) cursor pass, and use it in the whole-tree DFS walkers (calls, channels, env-accesses, imports, type-assigns). Convert the module-level variable walk to a cursor too, and add cbm_is_module_level_p() so callers that already hold the parent node skip the O(n) ts_node_parent() rescan.
Indexing microsoft/TypeScript drops from ~5100s to ~50s with identical graph output.
Same clean def-driven pattern as C/Go. Extracts cs_register_lsp_defs,
adds cbm_cs_build_cross_registry (filters CBM_LANG_CSHARP) +
cbm_run_cs_lsp_cross_with_registry. resolve_worker dispatches
CBM_LANG_CSHARP to the shared registry; pipeline.c builds
cross_registries.cs; pass_lsp_cross.h includes cs_lsp.h.
C# cross-LSP is def-driven (no per-file AST field collection), so the
shared registry yields identical resolution to the per-file build.
roslyn bench (17,916 C# files, ADVANCED): single shared registry
(lsp_cross_prepare=234ms, 260,814 defs, zero slow files), 91,089 LSP
overrides applied, 58.6s total. All 60 test_cs_lsp tests pass; serilog
sanity exact (2,627n/7,112e). No SEGV/ASan.
Brings C/C++/CUDA to Go's optimization level. Extracts the cross-LSP
registration loop into c_register_lsp_defs (shared by per-file and
Tier 2 paths), adds:
cbm_c_build_cross_registry — builds ONE project-wide registry from
all C-family defs (filters by lang) + both C and C++ stdlibs,
finalized, shared READ-ONLY across resolve workers.
cbm_run_c_lsp_cross_with_registry — per-file resolve using the
shared registry; no per-file registry rebuild.
C cross-LSP is DEF-DRIVEN (fields/returns/embeds come from the def
strings, no per-file AST field collection), so the shared registry
produces identical resolution to the per-file build — zero quality
loss. resolve_worker dispatches CBM_LANG_C/CPP/CUDA to the new path;
pipeline.c builds cross_registries.c; pass_lsp_cross.h includes c_lsp.h.
postgres bench (4,967 files, ADVANCED mode): 628 C files share one
pre-built registry (lsp_cross_prepare=17ms, zero slow files), 44 LSP
overrides, +9 edges vs FULL. All 200 test_c_lsp tests pass. No
SEGV/ASan. Total pipeline overhead vs FULL: +5.2%.
New CBM_MODE_ADVANCED=3. Both per-file LSP (type-aware resolution
during extract) and the cross-file LSP pass (during resolve) are now
OPT-IN: FULL and below run tree-sitter extraction + registry textual
resolution only; ADVANCED adds the full LSP layer.
Rationale: profiling showed LSP changes the kubernetes graph by only
~0.01% of edges (it mostly re-confirms textual resolution) while
costing significant memory + time. Making it opt-in keeps the default
FULL path lean (3.0 GB vs 5.6 GB peak RSS on K8s) without meaningfully
worse output, and lets users pay for LSP precision only when wanted.
Mechanism:
- cbm.c: g_lsp_enabled atomic toggle wraps the entire per-file LSP
block (Go/C/C++/PHP/Python/TS/JS/C#) + the C/C++ preprocessor-pass
LSP call. Default true so ad-hoc extraction / tests keep historical
behaviour. cbm_set_lsp_enabled / cbm_get_lsp_enabled in cbm.h.
- pipeline.c: sets the toggle (save→set→restore so it never leaks to
other callers) from p->mode before extraction; gates the cross-LSP
all_defs collection + registry build on advanced; gates the
sequential lsp_cross dispatch. Fixed the moderate_only comparison
(== FAST, not > MODERATE) so ADVANCED is not treated as lighter.
- pipeline_incremental.c: same toggle save/set/restore.
- discover.h/.c: added CBM_MODE_ADVANCED to discover's enum + all 4
filter comparisons treat ADVANCED like FULL (full discovery, no
fast-filtering). Without this, ADVANCED (numerically 3) was
mis-detected as fast-mode and dropped ~half the files.
- mcp.c: parse mode "advanced" + tool schema enum/description.
Dual-mode K8s bench: FULL 46.7s (lsp_overrides=0, 3.0 GB) vs
ADVANCED 48.8s (lsp_overrides=80,818, 5.6 GB). Graphs within 0.01%.
All 3,615 tests pass.
Replaces the prior Robin-Hood open-addressing implementation with
Verstable (https://github.com/JacksonAllan/Verstable), a 2024 state-
of-the-art C hash table. The public API in foundation/hash_table.h
is UNCHANGED — CBMHashTable is now an opaque type and all 14
production callers (graph_buffer, registry, pipeline passes,
service_patterns, watcher, semantic, etc.) get the new impl
transparently. str_intern.c keeps its specialised (ptr+len) hash —
different semantics, not a clean swap target.
Why: per-sub-phase profiling pinpointed cbm_pipeline_lsp_target_node
and cbm_registry_resolve as the dominant resolve_calls cost. Both
ultimately hit cbm_ht_get; both run millions of times on a kubernetes
index. Verstable's key innovations target exactly this:
* 4-bit hash-fragment metadata in a uint16_t sidecar array lets
chain walks skip most key comparisons WITHOUT touching the
bucket payload (cache wins on hot lookups).
* Quadratic probing with per-bucket displacement chains (not linear
Robin-Hood psl).
* Tombstone-free deletion that moves at most one existing key.
Vendored at internal/cbm/vendored/verstable/verstable.h (2069 lines,
84KB, MIT). Wrapped as a const char* → void* template inside
hash_table.c. The header struct is opaque (forward declaration only);
audited the entire tree — no caller accesses internals or takes
sizeof(CBMHashTable), so opacity is safe.
Test refactor: test_hash_table.c had two impl-specific tests:
* ht_robin_hood_bounded_psl — Robin-Hood specific; removed.
ht_stress_10000 covers the functional equivalent.
* ht_power_of_two_capacity — renamed ht_create_small_capacity,
replaced struct probes with a functional insert+lookup check.
ht_create_capacity_zero/one similarly de-internalised.
Benchmark on kubernetes (12 workers, 20,650 files):
- parallel_resolve wall: 36,376 -> 27,217 ms (-25.2%)
- pipeline.done total: 61,485 -> 51,745 ms (-15.8%)
- registry_build: ~1,000 -> 779 ms (faster — registry uses
cbm_ht_set heavily during build)
- All 3,615 tests pass; hash_table suite 100% green
- ASan/UBSan/SEGV: none
- Peak RSS: ~6.83 GB
- Graph: nodes 297,082 -> 297,102 (+20, 0.007%), edges 1,155,444 ->
1,154,025 (-1,419, -0.12%) — within tolerance; minor delta likely
from hash-iteration-order differences affecting tie-breaks in
resolution. Sanity (zod/serilog): nodes exact, edges within 0.1%.
Cumulative trajectory from broken-state baseline ~35 min on K8s ->
51.7s wall = ~40x total speedup.
cbm_service_pattern_match runs once per resolved CALL edge inside
emit_service_edge — that's 7 pattern lists × ~30 patterns × strstr
= ~210 strstrs per call. On kubernetes (~600k resolved CALL edges)
the SAME resolved QN ("fmt.Errorf", "context.Context.Done",
"errors.New", etc) repeats thousands of times across files. Caching
turns the linear pattern-list scan into one hash lookup after the
first miss.
API additions in service_patterns.h:
void cbm_service_pattern_cache_begin(void);
void cbm_service_pattern_cache_end(void);
Cache is TLS (no contention) scoped to the resolve_worker's lifetime
in the parallel_resolve phase. Capacity 8192 entries per worker —
covers typical project QN diversity without rehashing.
Encoding: enum value + 1 in the void* slot (0/NULL means "miss").
Two helpers svc_enum_to_ptr / svc_ptr_to_enum hide the trick.
Benchmark on kubernetes (12 workers, 20,650 files):
- parallel_resolve wall: 98,520 -> 90,104 ms (-8.5% vs Chunk 1
baseline, -12.8% vs original 103,281 baseline)
- resolve_calls CPU: 992,427 -> 885,663 ms (-10.8% vs Chunk 1,
-5.4% vs original — Chunk 1 added overhead that Chunk 2 recovers
AND surpasses)
- resolve_usages CPU: 11,977 -> 10,944 ms (-8.6%)
- total_loop CPU: 1,005,409 -> 897,561 ms (-10.7%)
- Tests: 3616/0
- Graph: 297,082 -> 297,080 nodes (-2), 1,153,482 -> 1,156,097 edges
(within 0.2%, semantic-preserving)
- zod/serilog sanity: edges within +/-3 of baseline
- Peak RSS: 7.46 -> 7.75 GB (+290 MB for 12 × 8192-entry caches)
Cumulative trajectory: original broken state ~35 min K8s ->
1.92 min wall (-49× total). The per-file is_import_reachable cache
(Chunk 1) + the per-worker service-pattern cache (Chunk 2) together
save ~13s of resolve wall time. Next chunks target the gbuf_insert_
edge hot path and remaining strstr scans.
KEY ARCHITECTURAL INSIGHT (credit: user observation):
The per-file LSP during extract ALREADY captures the inferred receiver
type QN + method name for unresolved cross-file calls — emit_unresolved_
call at go_lsp.c:1242-1245 stores callee_qn as '<FullReceiverTypeQN>.<
MethodName>' in result->resolved_calls flagged with strategy='lsp_
unresolved'. The cross-LSP step was REDUNDANTLY re-deriving this same
metadata via a second tree-sitter parse + AST walk.
Cross-LSP for Go is now METADATA-DRIVEN:
cbm_go_fast_resolve_qualified_calls (rewritten):
- Iterates result->resolved_calls where strategy=='lsp_unresolved'
- For each entry: try direct registry lookup of callee_qn;
if that misses, split prefix.suffix and resolve prefix via the
file's import map then lookup '<module_qn>.<suffix>' in registry
- Emits resolved entry on top (cbm_pipeline_find_lsp_resolution
picks highest confidence)
- NO tree-sitter parse, NO AST walk, NO scope tracking
resolve_worker (pass_parallel.c):
case CBM_LANG_GO now calls ONLY the metadata-driven fast path.
cbm_run_go_lsp_cross_with_registry (the slow AST-walk version) is
no longer invoked for Go — it would just re-derive identical
metadata via a second AST walk and arrive at the same answers.
Benchmark (kubernetes, full fresh index, 12 workers):
- parallel_resolve: 1.89 min (Tier 3 meta) vs 1.86 min (Tier 2) — parity
- lsp_cross.slow lines: 0 (no tree-sitter fallback runs)
- Nodes: 297,081 vs 297,082 baseline (-1)
- Edges: 1,153,163 vs 1,153,482 baseline (-319, within 0.03%)
- Tests: 3,616 / 0 failures
- zod/serilog sanity: nodes exact, edges +1-3 (within 0.03%)
The wall-time parity is informative: the AST re-walk wasn't the
dominant cost. Per-file budget (66ms across 20,650 files / 12 workers)
is now dominated by the 5 sequential resolve passes (resolve_file_
calls/usages/throws/rw/semantic) pushing ~10M total edges into the
per-worker gbuf with arena allocs + hash-bucket inserts. The Tier 3
work eliminates a genuine architectural redundancy (two walks of the
same AST producing the same metadata) and sets up Python/C/etc to
adopt the same metadata-driven pattern in follow-up commits — but
hitting the user-target <30s K8s requires attacking the gbuf hot
path, not the cross-LSP path.
The gopls 'package summary' pattern, applied properly: build the cross-
LSP CBMTypeRegistry ONCE per language at pipeline time (in a dedicated
cross_lsp_arena), share it READ-ONLY across all resolve workers via the
CBMCrossLspRegistries struct. Per-file work then becomes just tree-
sitter parse + AST walk + O(1) hash lookups — zero per-file registry
rebuilds for languages with a pre-built registry.
Each LSP gains two new public functions:
CBMTypeRegistry *cbm_<lang>_build_cross_registry(arena, defs, def_count)
— filters all_defs to its language, init + stdlib + registration
loop + cbm_registry_finalize, returns finalized registry.
void cbm_run_<lang>_lsp_cross_with_registry(arena, source, ...,
CBMTypeRegistry *reg, ...)
— skips the per-file build AND the per-file Phase 1b/1c AST scans
(those mutate the registry, which would race on a shared read-
only reg). Trade-off: a few file-local type aliases discovered
only by Phase 1b are missed; in practice the per-file extract
LSP already captured them via embedded_types/field_defs/etc.
Tested with parallel_python_lsp_override_cross_file_emits_lsp_
strategy_edges — fused + filtered + prebuilt path stays correct.
CBMLSPDef gains a CBMLanguage lang field set in pxc_build_lsp_def so
the per-language build functions can filter the global all_defs[].
cbm_parallel_resolve signature now takes CBMCrossLspRegistries* instead
of a single CBMTypeRegistry*. The resolve_worker dispatches:
cbm_pxc_registry_for_lang(lang) returns the right pre-built reg if any;
the worker then switch()es to the matching _with_registry variant.
Languages without a _with_registry variant fall through to the existing
filter + per-file build path.
Kubernetes (Go-heavy, 12k files, 139k defs) measured perf:
parallel_resolve: 5.51 min (strdup-removal baseline) -> 1.86 min
= 2.96x speedup
parallel.resolve.lsp_cross.slow lines: 1,258 -> 2
(-99.8%; the 2 remaining are Python files, not Go)
Peak RSS: 6.29 GB -> 6.23 GB (slight decrease — pre-built registry
shared, no per-file scratch)
All 3,616 tests pass; zod/serilog graphs exact match (per-file path
unaffected).
Cumulative trajectory from the original broken state on kubernetes:
35 min -> 19 min (finalize) -> 6.24 min (module-def filter) ->
5.51 min (strdup-removal) -> 1.86 min (Tier 2 Go) = 18.8x total.
Languages wired so far: Go, Python. C/C++, TS/JS, PHP, C# still use the
per-file filter+build path until their _with_registry variants land in
follow-up commits.
wyhash is a non-cryptographic hash function ~2x faster than FNV1A
for the long string keys used in CBMTypeRegistry's hash buckets
(qualified names average ~50 bytes). The header is unused for now
— prepared for a future Tier 3 symbol-interning pass that replaces
the QN-string-keyed hashtables with u32-keyed ones (rustc pattern)
where wyhash's lower collision rate matters more.
Source: https://github.com/wangyi-fudan/wyhash (Unlicense)
Vendored as a single header at internal/cbm/vendored/wyhash/wyhash.h.
Two perf changes across the 6 cross-LSP resolvers:
1. cbm_registry_finalize after the per-file registry build (all 6
LSPs). Builds the FNV1A hash buckets so lookups go through the
O(1) hashed path in type_registry.c:154. Previously this was
NEVER called anywhere in the codebase, so every method/type
lookup fell through to the O(N) linear scan — the root cause of
the 35-min kubernetes regression. Lookup count on k8s: ~50 calls
* 110k defs * 11k files = ~64B strcmps under the old fallback.
2. In go/c/py/php registration loops, replace cbm_arena_strdup(arena,
d->X) with d->X directly. The strings are borrowed from the
caller's all_defs[] arena which outlives the cbm_run_X_lsp_cross
call by the whole pipeline lifetime. Saves 5 wasted strdups per
def per file = ~6B mallocs avoided on kubernetes.
TS and CS use a different registration architecture (no inline
strdups in the body — uses a per-file ctx). Only the finalize change
applies there.
Kubernetes wall-time vs prior commit:
- After finalize alone: 35 min -> 19 min (~1.8x)
- After module-def filter (prior commit): -> 6.24 min (~5.6x)
- After strdup-removal (this commit): -> 5.51 min (~6.4x)
- Memory: 7.14 GB -> 6.29 GB peak RSS (-12%)
All 3616 tests pass, zod/serilog graph output exact match.
Previously cbm_pipeline_pass_lsp_cross ran as a separate sequential
pass that re-read every source file from disk and re-parsed every
tree-sitter tree on a single thread — a 50x regression on the
kubernetes Go monorepo (extract 10s, lsp_cross 9+ min when killed).
Three changes:
1. CBMFileResult gains const char *source / int source_len. The
parallel extract worker copies source bytes into result->arena
(per-file 100MB cap, project-wide 2GB atomic cap) so the fused
cross-LSP step can run without disk I/O. Source pointer is
borrowed by downstream readers; arena owns the storage.
2. Cross-LSP work moves into the parallel resolve worker. Each worker
processes its file with the existing parallel infrastructure
(worker pool + work stealing). Slab is reclaimed after each
per-file cross-LSP call to bound peak slab memory. The previously
separate lsp_cross pass entry function is kept for the sequential
pipeline path; the parallel path no longer invokes it.
3. CBMModuleDefIndex (gopls 'package summary' pattern) — an inverted
index module_qn -> list of def indices built once in O(D) before
resolve_worker fires. Per file, cbm_pxc_filter_defs_for_file slices
all_defs down to the file's own_module + imported modules only,
typically 50-100x smaller than the global ~110k defs on kubernetes.
Drops the per-file registry build from O(all_defs) to O(relevant).
Other plumbing: cbm_parallel_resolve signature gains all_defs,
def_count, def_modules, module_def_index params. pipeline.c builds
the index between registry_build and parallel_resolve and frees it
after. pipeline_incremental and tests/test_parallel updated to match.
Helpers in pass_lsp_cross.c exposed via the header: cbm_pxc_has_cross_lsp,
cbm_pxc_collect_all_defs, cbm_pxc_run_one(_ts), cbm_pxc_ts_modes,
cbm_pxc_build_module_def_index, cbm_pxc_filter_defs_for_file.
Kubernetes wall-time: 35 min -> 6.24 min (~5.6x) before the followup
perf commits in this series.
parse_ts_type_text in ts_lsp.c builds a 16-slot CBMType* members[] for
union (|) and intersection (&) types. The in-loop branch correctly
caps additions at mc<15, but the trailing post-loop append that
collects the final member after the last separator was unconditional,
and the subsequent members[mc] = NULL sentinel could then write to
members[16] — one past the array end.
Caught by UBSan on the zod codebase (a TS union with >=16 members
exists). Non-deterministic crash signature under pthread-parallel
extraction (12 workers): exit 134 / SIGABRT with no stack, because
libc's allocator detected the heap corruption and called abort().
Single-fixture-file test suite never exercised the parallel path or
a wide-enough union, so existing ASan/UBSan coverage missed it.
Fix: mirror the in-loop guard on the trailing append (cap at mc<15).
Add regression test tslsp_union_many_members_no_overflow that
constructs a 20-member string-literal union — extracting must not
crash. UBSan: clean. Full suite: 3616 passed, 0 failed.
Add a C# language LSP resolver mirroring the existing Go/C/PHP/Python
modules: type evaluation, using-directive resolution, method lookup,
and a corpus-seeded .NET BCL stdlib data table.
- internal/cbm/lsp/cs_lsp.{c,h}: resolver implementation
- internal/cbm/lsp/generated/cs_stdlib_data.c: BCL type/method tables
- wire CBM_LANG_CSHARP dispatch in cbm.c and lsp_all.c
- register cs_lsp + cs_lsp_bench suites in the test harness
- tests/test_cs_lsp.c, tests/test_cs_lsp_bench.c
(cherry picked from commit 8f677c67a0c9d7829cd6ac27393de2d0aaaac534)
98d5cac silently dropped index cells that exceed a single SQLite page,
which is the correct semantic (the index B-tree has no overflow page
mechanism, unlike the record B-tree fixed in #175). But silent drops
make the failure mode invisible — a caller hitting deeply nested
namespaces or very long file paths gets a partial index with no
indication anything was skipped.
Add an fprintf(stderr) at the skip site reporting the cell length and
its index, matching the existing logging style in this file (lines 311,
1305) so the dropped cells are observable for diagnosis. Consistent
with the cbm_write_db: prefix used by surrounding warnings.
write_index_btree() flushed the page buffer whenever the next cell
didn't fit, on the assumption that a freshly-initialised page would
always accept any cell. That invariant fails for index cells whose
payload (qualified_name, file_path, etc.) is larger than a full
SQLite page — after the flush, pb_cell_fits() is still false but the
code calls pb_add_cell() anyway. The subsequent content_offset
underflow corrupts the page header and the next write triggers
SIGBUS on large repos (observed on a Ruby codebase with ~215K nodes).
Re-check pb_cell_fits() after the flush and continue past cells
that still don't fit. Index entries whose keys exceed a full page
can't be stored and are not expected to survive the writer anyway —
the rest of the index is correctly preserved and the indexer no
longer crashes. Record overflow pages (#175) already covers the
record-btree side of this same shape of bug.
extract_constructor_type only accepted call-form RHS when the function
name's first character was uppercase. Misses two common typed-stub
factory patterns:
Go: client := pb.NewPromoCodeClient(conn)
Java: stub := PromoCodeServiceGrpc.newBlockingStub(channel)
Both have a lower-cased qualifier prefix, so the first character of the
full text is `pb`/`promoCodeServiceGrpc` rather than the factory name.
Match the last `.`-segment instead and accept it when it begins with
`New`/`new` and ends with `Client` or `Stub`. Returns the full
qualified call so downstream passes keep the package context.
Stays scoped — only the new factory shape is added; the uppercase-first-
char path for `Foo()`-style direct calls is unchanged.
Co-authored-by: sponger94 <45746997+sponger94@users.noreply.github.com>
Three C# extractor gaps that left modern .NET classes partially invisible
in the graph:
1. lang_specs.c CSHARP entry had `empty_types` for field_node_types, so
walk_defs never descended into field_declaration / property_declaration
nodes. Add cs_field_types and wire it in. Without this, no C# fields
or properties were ever pushed as Field defs.
2. extract_class_fields assumed the field's "type" is on the immediate
child via ts_node_child_by_field_name. Tree-sitter-c_sharp wraps fields
as field_declaration > variable_declaration(type, variable_declarator),
one level deeper than Java/Go/Rust. Detect the nested shape by walking
named children for variable_declaration and resolve type + name from
inside it. Direct shape continues unchanged.
3. C# 12 primary constructors (`class Foo(IBar bar) : Base { ... }`)
declare implicit captured fields on the class line. Tree-sitter wraps
them inside the hidden _class_declaration_initializer node, so the
`parameters` field on class_declaration may not always resolve
directly; iterate top-level children for parameter_list as a fallback.
Emit each primary-ctor parameter as a Field def with parent_class set.
Effect: dependency-injected fields on .NET 8+/9+ controllers and any C#
auto-property (`public IFoo Foo { get; }`) are now reachable as Field
nodes in the graph.
Co-authored-by: sponger94 <45746997+sponger94@users.noreply.github.com>
Per-file LSP (run from cbm_extract_file) only sees one file's defs, so
callees whose receiver type comes from an imported module stayed
unresolved — even though Go/C/Py/TS already had cross-file entry points
(cbm_run_X_lsp_cross). Those entry points existed but no pipeline pass
ever invoked them. PHP didn't even have a cross-file entry point.
This change lands the missing pieces:
1. cbm_run_php_lsp_cross + cbm_batch_php_lsp_cross — PHP cross-file
resolver mirroring py_lsp_cross / ts_lsp_cross. Reuses PHPLSPContext,
php_lsp_init, php_lsp_add_use, php_lsp_process_file. Handles PHP-
specific labels (Class / Interface / Trait / Enum / Type) and uses
def_module_qn so cross-file return types qualify against the def's
own module rather than the importing module's.
2. pass_lsp_cross.c — new pipeline pass. For every cached file result,
builds a project-wide CBMLSPDef[] (CBMDefinition → CBMLSPDef
converter), per-file import map (from gbuf IMPORTS edges), then
dispatches to the matching cbm_run_X_lsp_cross. Output strings get
copied into the per-file arena (with O(N²) memory bug avoided by
running each file inside a fresh scratch arena that gets destroyed
after the LSP call — the LSP allocates its registry + stdlib +
project defs in scratch, not in the caller's persistent arena).
3. Wired into both pipelines:
- sequential: definitions → k8s → lsp_cross → calls → ...
- parallel: parallel_extract → registry_build → lsp_cross →
parallel_resolve → ...
4. Updated tests/test_parallel.c run_parallel helper to mirror the new
parallel ordering so existing cross-file Python/PHP tests now exercise
pass_lsp_cross.
Languages covered: Go, C/C++/CUDA, Python, TS/JS/JSX/TSX, PHP. PHP cross
file is new. The other four had functions but no pipeline integration
until this change.
Verified on a 1670-file FastAPI index: pass_lsp_cross processes 1050
Python files (525 non-LSP files skipped), collects 4070 project defs in
9.2s, and parallel_resolve subsequently reports lsp_overrides=568 — the
LSP-strategy attribution gain over per-file alone.
Per-LSP emit functions dedup by (caller_qn, callee_qn) within their own
run; pxc_append_results dedups again at append time so cross-file
entries duplicating per-file output get dropped instead of inflating
result->resolved_calls.
Resolved 5 conflicts where the php-lsp branch and main both added LSP
infrastructure but with different APIs:
- Makefile.cbm: kept both TEST_PHP_LSP_SRCS and main's py_lsp/scope/
type_rep/stack_overflow test source variables in ALL_TEST_SRCS.
- internal/cbm/cbm.c: kept both #include lsp/php_lsp.h and lsp/py_lsp.h;
the per-language dispatch already supports stacking PHP after PYTHON.
- internal/cbm/lsp_all.c: kept both lsp/php_lsp.c + php_stdlib_data.c
AND lsp/py_lsp.c + python_stdlib_data.c in the compilation unit.
- src/pipeline/pass_calls.c: took main's version entirely.
find_lsp_resolution + resolve_single_call(ctx, call, lsp_calls, ...)
is language-agnostic — it matches any CBMResolvedCall whose caller_qn
hits the textual call's enclosing function and whose callee_qn
short-name hits the textual callee. PHP LSP populates the same
result->resolved_calls array via cbm_run_php_lsp, so the unified
resolver handles PHP for free. Dropped the php-branch's
qn_short_name + lsp_override_resolution helpers as superseded.
Locked confidence floor at main's CBM_LSP_CONFIDENCE_FLOOR = 0.6f
(was 0.5f on the php branch).
- tests/test_main.c: kept all suite externs and RUN_SUITE invocations
for php_lsp + py_lsp/py_lsp_bench/py_lsp_stress/py_lsp_scale +
scope + type_rep.
Auto-merged: src/pipeline/pass_parallel.c.
Brings in 52 commits from main since the branch was forked at c768b61
(#207 watcher follow-up): #208 through #281, dependabot bumps, Pine
Script, AUR docs, INHERITS edges, get_architecture wiring, codeql
revert, the entire python-lsp subsystem.
Suite: 3249 passed, 0 failed (was 2971 on main; +278 new tests from
php-lsp matches the branch's claimed test count).
Phase 5f, 4ac, 4ad: closing the long-tail gaps toward LSP-server parity.
- @phpstan-type / @psalm-type / @phan-type aliases parsed from class
docblocks and applied during PHPDoc type resolution. References to
the alias name in @var/@param/@return are rewritten to the aliased
type. Aliases live on PHPLSPContext (per-file).
- @phpstan-import-type accepted without crash (cross-file alias
resolution deferred — same root cause as Composer PSR-4).
- @template-covariant T and @template-contravariant T now register T
as a proper type-param (previously skipped along with
@template-extends).
- Closure binding: \\Closure::bind($f, $obj) and $f->bindTo($obj),
when invoked with an inline closure literal as the first arg, walk
the closure body with $this rebound to the second arg's class.
Resolves the Laravel-macroable / VarDumper / encapsulation patterns.
- @return ($x is string ? int : bool) conditional return-type syntax
accepted without crash (parsed leftmost segment as fallback).
- Long PHPDoc blocks with mixed @param/@return/@throws/@internal
parsed without misattributing tags.
Stdlib unchanged.
Tests: 248 → 278 covering:
- phpstan-type alias basic / via @param / union / array-shape
- psalm-type alternate spelling
- phpstan-import-type (no-crash)
- Closure::bind / ::bindTo / static-bind on arrow fn
- conditional return / template-covariant / -contravariant (no-crash)
- class-string<T>, int<0,100>, literal-string (no-crash on phpstan
types we don't fully model)
- 8 realistic patterns: repository, event listener, console command,
form handler, mail sender, cached service, logger w/ context,
Carbon chains in methods, Eloquent model.
Total tests: 3061 → 3091 / 0 failed.
Audit for O(n^2) hot paths after Round 10. The dedup loop in
py_emit_resolved_call scanned ALL prior entries linearly, making
emission O(N) and total file work O(N^2). On a synthetic 2000-class
fixture with 8000 calls, that dominated time at 5083 ms.
Cap the scan window at the last 256 entries. Most duplicate
emissions come from the same expression being walked twice (resolver
+ emitter passes), so they're temporally adjacent — a 256-entry
window catches every common case while keeping the dedup O(1) per
emission.
Scale measurements (test_py_lsp_scale.c added):
| Classes | Calls | Resolved | Before | After |
|--------:|------:|---------:|-------:|------:|
| 100 | 400 | 300 | 55ms | 57ms |
| 500 | 2000 | 1500 | 333ms | 323ms |
| 2000 | 8000 | 6000 | 5083ms | 1804ms|
Ratio for 20x input: 91.8x -> 31.4x (linear would be 20x; quadratic
would be 400x). No remaining quadratic hot path in code added by
this branch.
Other hot paths surveyed:
- py_register_def -> cbm_registry_lookup_type: O(types) per def at
registration, O(D x types) for D defs. Pre-existing linear scan in
type_registry.c; hash-indexing it would be a cross-language
optimization (Go and C++ LSPs benefit too).
- py_lookup_attribute / py_lookup_field: O(types) per call site.
Same pre-existing linear scan. With ~900 stdlib types + per-file
defs, accounts for ~75µs/call at 2000-class scale.
- MODULE submodule prefix scan: O(funcs) per module-attribute access,
early-exits on first match. Bounded by registry size.
- lambda / dict-literal registries: linear, but bounded by per-file
N which is small.
Bench unchanged at 100% / 12ms (52/52 calls, 178 LOC).
When a trait method declares `: self` as its return type, flattening
that method into a using class now substitutes the trait's QN with the
using class's QN in the return type. This fixes fluent chains like:
trait T { public function tap(): self { return $this; } }
class C { use T; public function ok(): int { return 1; } }
$c->tap()->ok(); // resolves through to C.ok now
Implementation:
- flatten_trait_into_class scans the source method's signature for
NAMED(trait_qn) in the return type and rewrites it to
NAMED(using_class_qn) when registering the flattened method.
- php_lsp_collect_class_fields now does TWO passes — pass 0 for
trait/interface declarators (so their declared-return-types get
re-parsed before any consumer), pass 1 for class/enum (which may
`use Trait;` and need the trait's signatures already up-to-date).
Total tests: 3013 / 0 failed (200 php_lsp).
Round 10. Closes the last documented gaps from Round 9 plus 21 new
real-world stress tests covering SQLAlchemy / Pydantic / dataclass /
TypedDict / Protocol / generator / async patterns.
Lambda call-site inference:
- PyLSPContext gains a per-context lambda registry. `fn = lambda x:
body` records the lambda's AST node bound to fn. At call site
fn(arg), we push a sub-scope, bind lambda params from arg types,
and walk the body. Edges emitted under a synthetic <lambda> child
of the enclosing function so traces attribute correctly.
Dict-literal dispatch table:
- `funcs = {"a": foo, "b": bar}` records per-literal-key target QNs
in CBMDictLiteralEntry[]. funcs["a"]() emits an edge directly to
foo's QN with strategy=lsp_dict_dispatch.
Async with / async for:
- `async with X() as f` looks up __aenter__ before __enter__.
__aenter__'s declared return type wins; the manager type is the
fallback (async __aenter__s typically return self).
- `async for x in iter` falls back to __anext__'s return type when
the iterable's TEMPLATE form isn't recognized as AsyncIterator /
AsyncGenerator.
Callable / next / iter:
- `f()` where f resolves to CBM_TYPE_CALLABLE returns the callable's
return type. TEMPLATE("Callable", [..., R]) returns R.
- factory()() / (lambda)() etc — when the function expression is
itself a call / parenthesized / lambda, evaluate it and use its
return type.
- `next(it)` returns py_iterable_element_type(type-of-it).
- `iter(x)` is identity.
*args / **kwargs typed-parameter wrapping:
- `def f(*args: T)` -> args is TEMPLATE("tuple", [T]).
- `def f(**kwargs: T)` -> kwargs is TEMPLATE("dict", [str, T]).
- Detected by walking typed_parameter > list_splat_pattern /
dictionary_splat_pattern wrappers in py_bind_parameters.
Tuple / list / set / dict literal expressions:
- py_eval_expr_type now builds structured TUPLE / TEMPLATE types
from element types of the literal. `(Foo(), Bar())` -> TUPLE([Foo,
Bar]); `[Foo()]` -> TEMPLATE("list", [Foo]); `{"k": Foo()}` ->
TEMPLATE("dict", [str, Foo]). Without this, unpacking lost element
precision.
Match sequence pattern:
- list_pattern wraps each element in another case_pattern; capture
appears as dotted_name -> identifier; splat as splat_pattern.
Element-binder walks both wrappers and binds capture/splat to the
subject's iterable element type / list[elem].
Stress test suite expanded from 22 to 43 patterns. All 43 hard-
asserted as PASS:
- typing helpers: NamedTuple, TypedDict (incl. total=False), Protocol
(nominal + structural), ABC + abstractmethod, Mapped[T] (SQLAlchemy
2.0), Pydantic-style BaseModel
- control flow: with-as via __enter__, except-as, post-early-return
narrowing, async with via __aenter__, async for via __anext__
- containers: tuple unpack from function return + literal,
dict.items() comprehension k/v unpack, list slice, dict.get
default, recursive self-referencing types via forward refs
- decorators: factory + functools.wraps, classmethod chained to
method, property setter
- inheritance: Self in chain, diamond MRO with C3, multi-class match
- functional: lambda call-site inference, function-as-dict-value
dispatch, callable return-value, next(iter), generator delegation
(yield from), iterator next, double-star **kwargs, isinstance |
branches, module-level const, dataclass + default_factory
Bench: 178 LOC, 52 calls, **52 resolved (100%)**, 11.25 ms (ASan +
UBSan). Total 2941 / 0. Zero remaining KNOWN GAPs in the stress
suite — every probed pattern resolves.
Two bugs in extract_base_classes() caused most INHERITS edges to be missing for Java (and any other language using similar AST field names):
Bug 1 — Early return: the field loop returned on the first matching field, so class Foo extends Bar implements Baz produced only one INHERITS edge instead of two — the loop never processed 'interfaces' after finding 'superclass'.
Bug 2 — Keyword text: cbm_node_text on the field node returned 'extends Bar' / 'implements Baz, Qux' literally instead of the bare type names. So even the one edge that did get emitted had a malformed target.
Fix: new collect_bases_from_field() walks into child AST nodes and extracts type_identifier / generic_type / qualified_name / scoped_type_identifier / user_type text directly. Handles type_list / interface_type_list children for the multi-interface case. Strips generics at '<'. Falls back to raw node text for languages where the field IS the type name. The outer field loop now collects from all matching fields before returning, and the result is arena-allocated with a NULL terminator like the surrounding extract_csharp_base_list pattern.
Bounds-safe: writes capped at MAX_BASES_MINUS_1 (15) with per-iteration count < out_cap checks; result fits in MAX_BASES (16) including NULL terminator.
Real-world impact: 3 → 116 INHERITS edges on the Modelio Java codebase (253 files).
Round 9 closes 2 of the 3 remaining stress gaps from Round 8:
TypedDict literal-key subscript:
- d["foo"] where d: TD with class-body annotation `foo: Foo` returns
the field's type. Routes through the existing class-body-annotation
field registry (Round 4): TypedDict subclasses already register their
fields the same way. The subscript branch in py_eval_expr_type now
detects NAMED container + literal-string key and looks up the field
via py_lookup_field. Strips both single and double quotes.
Early-return narrowing (post-block flow):
- if not isinstance(x, T): return / raise / break / continue —
applies the *positive* narrow to the enclosing scope after the
if-block when the consequence terminates.
- if x is None: return — narrows x to non-None afterwards.
- py_block_terminates detects return/raise/break/continue as the last
statement of the body block. Subsequent code in the function sees
the narrowed type.
Match sequence pattern (tentative):
- list_pattern / tuple_pattern handler added: bind capture-pattern
elements to the subject's iterable element type; bind splat-pattern
rest to list[elem]. Also walks more wrapper levels (case_pattern ->
pattern -> _simple_pattern). Test still flags as KNOWN GAP because
tree-sitter's pattern-wrapper structure for the bracket-form `[a, b]`
in match/case differs in some grammar versions; documented as
remaining gap pending grammar inspection.
Stress test suite: 22/22 PASS. Hard-asserted patterns: namedtuple,
typeddict, protocol, abc, with-as, except-as, isinstance-early-return,
tuple-unpack, dict.items, slice, decorator-factory, property-setter,
Self chain, diamond MRO, recursive types, nested closure, long method
chains, generator delegation, async-gen for. Documented gaps:
function-as-dict-value, match sequence pattern, lambda inference.
Bench unchanged at 100% / 11.02 ms; total 2920 / 0.
Round 8. Stress test suite (test_py_lsp_stress.c) added with 22 advanced
patterns; bench fixture grew to 178 lines covering the full surface.
with-as binding via __enter__:
- Look up __enter__ on the context manager's NAMED type. Use its
return type when known; fall back to the manager type itself
(matches the practical case where __enter__ returns self). Handles
tree-sitter Python's wrapping of `X as y` in an as_pattern node
inside with_item, plus the flat field-only shape.
except-as binding:
- Bind exception alias to the caught type. Resolves through
except_clause's as_pattern wrapper and the as_pattern_target
identifier wrapper.
Tuple unpacking:
- Both `a, b = f()` and `for k, v in d.items()` destructure the RHS
element-by-element when the rhs is TUPLE or TEMPLATE("tuple", ...).
*rest binds the trailing slice as list[elem]. Shared helper
py_bind_for_target used by for-statement, comprehension for_in_clause,
and assignment LHS.
Slice subscript:
- `lst[1:3]` returns the same TEMPLATE container — list[T][1:3] is
list[T], dict[K, V][1:3] is dict[K, V] (rare but consistent).
Container TypeVar substitution:
- Without a constraint solver we can't generally substitute K, V into
registered method return types. Hand-rolled the most-impactful
cases: dict.items/keys/values/get/pop/copy, list.copy/__iter__/pop,
set.copy. These return properly parameterized ItemsView[K, V] /
KeysView[K] / ValuesView[V] / Optional[V] / etc. so subsequent
iteration and unpacking yield typed elements.
ItemsView / KeysView / ValuesView / dict_items / dict_keys / dict_values
added to py_iterable_element_type. ItemsView[K, V] iterates as
tuple[K, V], unpacked correctly by py_bind_for_target.
Bench: 178 LOC, 52 calls, **52 resolved (100%)**, 11.59 ms (ASan + UBSan).
22 new stress tests, 8 of 10 documented gaps closed in this round.
Remaining gaps documented inline (typeddict_subscript, post-early-return
narrowing, function-as-dict-value, match sequence pattern, lambda
inference). All 2898 prior tests stay green; total 2920 / 0.
Adds Pine Script (TradingView's indicator/strategy DSL) as a first-class
language using the project's existing language-add pipeline:
- scripts/new-languages.json: register pine (display "PineScript",
ts_func tree_sitter_pine, repo kvarenzn/tree-sitter-pine, extension
.pine, has_scanner=true, module_root source_file)
- internal/cbm/vendored/grammars/pine/: vendored from
https://github.com/kvarenzn/tree-sitter-pine via scripts/vendor-grammar.sh
— parser.c, scanner.c, tree_sitter/ headers. License is ISC (declared
in upstream package.json; written into vendored LICENSE for clarity
since upstream has no LICENSE file)
- internal/cbm/grammar_pine.c: 4-line wrapper generated by
scripts/generate-lang-code.py wrappers
- internal/cbm/cbm.h: CBM_LANG_PINE before CBM_LANG_COUNT
- internal/cbm/lang_specs.c: 7 node-type arrays (func / class /
module / call / var / branch / assign) cross-checked against
kvarenzn's grammar.js, plus the spec table entry
- src/discover/language.c: .pine -> CBM_LANG_PINE extension dispatch
and "PineScript" name
scripts/audit-grammar-security.sh on the vendored grammar: no
dangerous includes (only string/stdint/stdbool/stdlib), no system /
popen / exec / socket / mmap / dlopen calls, no constructor/destructor
attributes, no inline asm. The 148-line scanner.c only handles
INDENT/DEDENT/NEWLINE tokens via the standard tree-sitter lexer API.
Distilled from #273 rather than vendored verbatim because the
original PR's parser.c was a plain 46k-line drop without provenance,
no LICENSE attribution, and bundled an unrelated Makefile change
(scripts/embed-frontend.sh -> sh scripts/embed-frontend.sh) that
didn't belong to a Pine Script feature. The node-type choices in
the lang spec match the original PR's selections, all of which were
verified to be real grammar.js rules.
Full suite: 2840 passed, 0 failed under ASan + UBSan.
Closes#273.
Co-authored-by: vinay-veerappa <vinay.veerappa@gmail.com>
Phase 4m–w in one push, plus 25 new tests (100 → 125, all green).
New capabilities:
- Generic @template substitution:
@template T on a class records T as a type-param;
@return T on a method gets parsed as NAMED("T");
when receiver is TEMPLATE("Container", [User]), method-call return
types are substituted T → User so chains like
`$container->get()->name()` resolve through to the element type.
- Closure use(...) capture: typed variables in `use ($a, $b)` clauses
of `function () use (...) {}` are inherited from the parent scope.
- Multi-conjunction narrowing: `if ($x instanceof A && $y instanceof B)`
narrows BOTH $x and $y in the if-body. Up to 8 simultaneous narrowings.
- is_a($x, Foo::class) narrowing — both the ::class and string forms.
- Subscript inference: $arr[$k] on `array<T>` returns T;
on `array<K, V>` returns V;
on a NAMED ArrayAccess class consults registered offsetGet.
- Enum case access: Suit::Hearts evaluates to NAMED(Suit) so
Suit::Hearts->label() resolves to Suit::label.
- @extends ParentClass<X>: appends ParentClass to embedded_types so
the inheritance walk picks up methods from generic ancestors.
- PHPDoc @return on methods with no declared return type updates
the registry signature.
- foreach key=>value: handles tree-sitter-php's `pair` node so
`foreach ($users as $idx => $user)` binds $user to the element type.
- Class-level @template-extends, @template-implements parsed without
crashing.
- Don't override TEMPLATE/NAMED bindings with `null` literal —
preserves @var-typed variables across `$x = null` initialization.
Stdlib expansion (Phase 5d):
- Symfony Console: SymfonyStyle, InputInterface, OutputInterface
- Doctrine ORM: EntityManagerInterface, QueryBuilder chain, Query,
EntityRepository
- Guzzle HTTP: Client and ClientInterface (returning PSR-7 responses)
- Twig: Environment, TemplateWrapper
- Eloquent Model: forwarded static methods (where, whereIn, etc.)
so `User::where()->first()` chains resolve immediately.
Test coverage: 100 -> 125 tests; total 2913 -> 2938 / 0 failed.
Major parity push. Bench 79% -> 98% (51/52 calls resolved) on a 178-line
fixture exercising the full parity surface.
Annotation wrapping:
- Type-wrapper annotations now unwrap to the inner type:
ClassVar[T], Final[T], InitVar[T], ReadOnly[T], Required[T],
NotRequired[T], Annotated[T, ...] (drop metadata),
Mapped[T] / WriteOnlyMapped[T] / DynamicMapped[T] (SQLAlchemy 2.0),
Type[T] / type[T]. Same logic in both py_resolve_annotation
(runtime, scope-aware) and py_parse_type_text_qn (registry-build,
scope-free).
Builtin / template attribute dispatch:
- obj_type=BUILTIN("str"|"int"|"list"|"dict"|...) now resolves
attribute access via "builtins.<typename>" in the typeshed-derived
registry. obj.split(...) on str, list.append(), dict.get() etc all
emit edges with strategy=lsp_builtin_method.
- obj_type=TEMPLATE("dict"|"list"|...) routes the same way; receiver
is the unparameterized container's typeshed entry.
Bare-name builtin fallback:
- Bare-name calls (range, len, list, dict, str, int, print, etc.) fall
through to a "builtins" module symbol lookup, then to a builtins
type lookup for class constructors. Strategies: lsp_builtin /
lsp_builtin_constructor.
Emission deduplication:
- py_emit_resolved_call dedupes by (caller_qn, callee_qn). When two
resolution paths converge on the same edge (expression typing then
call emission, or attribute walk through wrappers), keep one entry
with the higher confidence. Previously the same call could emit
twice — the bench showed >100% ratios as evidence.
Bench fixture expanded from 136 to 178 lines:
- Adds @property access chains, Mapped[Annotated[int, ...]], Final
fields, generator iteration with method dispatch on element type,
multi-class match/case, dict comprehension over .values(), instance
attribute typing through super().__init__, classmethod chained with
Self return.
All 2898 prior tests stay green; bench at 98% in 11.79 ms (ASan +
UBSan).
Round 6:
CBMRegisteredFunc gains a `flags` bitfield (CBMFuncFlags) and a
`decorator_qns` NULL-terminated list. py_register_def populates them
from CBMDefinition.decorators by short-name match — so `@property`,
`@functools.cache`, `@abc.abstractmethod`, `@typing.overload`, and
`@final` all light up regardless of qualified vs bare reference. The
flags aren't consumed by resolution yet (still on roadmap: emit
distinct edge for @property attribute access vs method call); they
are recorded so future overload selection / property modeling can
use them without another schema change.
The struct extension is at-tail and existing call sites that memset
to zero stay correct; non-Python LSPs see flags=0 / decorator_qns=NULL
and behave identically.
Three new verification tests prove that generators, dataclasses, and
@property already work end-to-end through the Round 4-5 infrastructure
without further code changes:
- Generator[Foo, ...] iterables in for-loop bind loop var to Foo via
py_iterable_element_type's existing TEMPLATE handling.
- @dataclass class Point: x: Foo (class-body annotated assignment) is
picked up by Round 4's third-pass field registrar; p.x.method()
resolves through the field type.
- @property def thing(self) -> Foo: c.thing returns Foo because
attribute access on NAMED routes through py_func_return_type which
yields the getter's annotated return.
All 2895 prior tests stay green; 3 new tests pass. Total 2898 / 0.
Bench unchanged at 95% (the small bench fixture doesn't exercise
the new patterns; will expand in Round 7).
Round 5 parity push:
Subscript value typing — py_eval_expr_type's subscript branch resolves
indexing against the container's TEMPLATE type:
- TEMPLATE("dict"|"Mapping"|"defaultdict"|"OrderedDict"|...)[K] -> V
- TEMPLATE("list"|"set"|"frozenset"|"Sequence"|"Iterable"|...)[i] -> elem
- TEMPLATE("tuple", [A, B, ...])[literal_int] -> args[idx]
- Non-literal index returns UNION of all element types.
This unlocks the `self.cache[k].method()` pattern that's everywhere in
real Python. Also fixes `items[0].method()` for typed lists.
super().__init__(...) constructor edge — super().<attr> already worked
for regular methods; __init__ is special-cased because parent classes
typically don't register __init__ with a useful return type. Strategy
"lsp_super_init" with confidence 0.85.
Operator dunders — a + b looks up __add__ on a's type and uses its
return type. Same for *, -, /, //, %, **, <<, >>, &, |, ^, @. Falls
back to left-operand type when registry has no info (correct for the
common int+int / str+str / list+list cases).
3 new tests; all 2892 prior tests stay green. Bench unchanged at 95%
(fixture doesn't exercise these patterns directly — Round 6 will
expand it).
Round 4 / final parity push. Tracks self.x = expr from __init__ and
class-body annotated assignments (`x: Foo`) as registered fields on
the enclosing class type.
py_register_instance_field mutates the registry's CBMRegisteredType
field_names / field_types arrays in place. Re-registering the same
name overwrites; first-seen wins for capture order. Arena-allocated,
not a hot path.
py_lookup_field walks the alias / embedded chain with the same
CBM_LSP_MAX_LOOKUP_DEPTH cap as method lookup, so inherited fields
resolve through MRO without stack-overflow risk on pathological
hierarchies.
Field-fallback wired into py_eval_expr_type's attribute branch:
when method lookup misses, try field lookup. Returns the field's
type so downstream attribute / call resolution chains naturally.
Same fallback applied to UNION (single-match heuristic).
py_process_class now does three passes:
1. Class-body annotated assignments (`x: int = ...`) → fields.
2. __init__ / __post_init__ → self.x = expr fields.
3. All other methods.
This guarantees fields are registered before methods that reference
them, regardless of source order.
Annotated assignment binding fix: py_resolve_annotation is now called
on assignment annotations too, so `self.cache: dict[str, str] = {}`
binds with the proper TEMPLATE type instead of NAMED("dict[str, str]").
Bench result on the 136-line fixture: **95%** (36/38 calls resolved,
8.06 ms). Up from 79%. The 5% gap covers cases that need a full
constraint solver — see Final Stopping Point doc.
2 new tests; all 2890 prior tests stay green. Total 2892 / 0.
Round 3 parity push:
- match/case (PEP 634): when the case pattern is class_pattern (wrapped
in a case_pattern), narrow the match subject identifier to NAMED(class)
inside the case body. Push/pop scope per case. Capture sub-patterns
bind to UNKNOWN (field-type extraction is a v1.1 task — would require
field type info in the registry).
- await expr: identity passthrough. async def f() -> T registers with
return T, so `await f()` returning T is the resolved-call shape we
want without modeling Coroutine[Any, Any, T] explicitly.
- Expanded test_py_lsp_bench.c fixture from 65 lines to 136 lines
exercising every parity feature: dotted-module walks, dataclass +
Self chaining, multi-inheritance with super(), Optional/Union
annotations + narrowing (isinstance / is not None / walrus), generic
list[T] / dict[K, V] / defaultdict comprehensions, classmethod /
staticmethod, match/case dispatch, async/await, common stdlib
(logging.getLogger, json.loads, pathlib.Path, functools.lru_cache).
Bench result on the new fixture: 38 calls, 30 resolved, 79%, 7.50 ms.
Remaining 21% includes patterns where intermediate types lose precision
(e.g. super() constructor, dict subscript), instance attribute typing
(self.x where x isn't annotated at class scope), and decorator
return-type substitution (lru_cache).
All 2888 prior tests stay green; 2 new tests + bench expansion. Total
2890 / 0.
Round 2/3 parity push. Three connected gap fixes that share an annotation-
parser and a generic-aware iterable model.
Annotation parser upgrade:
- py_resolve_annotation now turns subscripted forms into structured types
rather than stripping. `Optional[Foo]` -> UNION(NAMED(Foo), None).
`Union[A, B]` -> UNION. `list[Foo]` / `dict[K, V]` / `Iterable[T]` ->
TEMPLATE with arg list. `X | Y` (PEP 604) handled at depth zero.
- New py_parse_type_text_qn variant: same parser without scope lookup,
used by py_register_def at registry-build time. Bare class names are
qualified to <module_qn>.<name> so registry lookups match.
- py_register_def prefers d->return_type (full subscripted text) over
d->return_types[0] when brackets are present — extract_defs's array
form strips the subscript, losing element types like Optional[Foo].
Narrowing:
- if isinstance(x, T): bind x as NAMED(T) inside the consequence body
(block-scoped via push/pop).
- if x is not None: strip None from x's UNION inside the consequence;
py_match_is_none recognizes "is not" as the single tree-sitter token
it actually is, walks through parens / walrus to find the underlying
identifier.
- Else branch unchanged — narrow's negation (not-T, including-None) is
not modeled in v1.
Walrus + truthy:
- if (x := compute()) is not None: walrus binds x in the enclosing scope
(PEP 572) so the body sees x. py_bind_walrus_in walks the cond AST
looking for named_expression / assignment_expression nodes.
Comprehension + for-loop element typing:
- New py_iterable_element_type maps TEMPLATE(list/set/Iterable/...) to
arg[0]; TEMPLATE(dict) to key arg; TEMPLATE(tuple, ...) to UNION of
args.
- Comprehensions push their own scope and bind for_in_clause loop vars
to the iterable's element type before walking body / filter clauses.
for_statement does the same.
UNION attribute access fallback (used when narrowing can't run):
- obj_type=UNION & exactly one non-None NAMED member has the attr:
emit edge with strategy=lsp_method_union, confidence 0.85.
7 new test_py_lsp.c cases. All 2880 prior tests stay green; 8 new tests
land. Bench unchanged (small fixture, no narrowing exercised there).
Phase 4g+ / 4l / 5b additions toward LSP-server type-resolution parity:
- Generic template parsing in PHPDoc: parse `Collection<User>`,
`array<int, User>`, `iterable<T>`, `list<T>`, etc. Bind variables
to CBM_TYPE_TEMPLATE; method dispatch on a TEMPLATE receiver
resolves against the template's base class.
- Foreach element-type propagation: `foreach ($users as $u)` where
$users: array<User> binds $u to User. Two-arg array<K,V> takes
V. Falls back to Iterator::current() return type for NAMED types
that implement the iterator interface.
- Negative narrowing: `if (!($x instanceof Foo)) return;` (or throw)
narrows $x to Foo for the rest of the enclosing scope.
- Foreach iterable can be an arbitrary expression (variable_name,
member call, function call) — fixed earlier-too-narrow filter.
- PHPDoc @var/@param accept generic syntax: type token now spans
angle brackets so `array<User> $users` parses correctly.
- Stdlib expansion (Laravel + Symfony): full Eloquent Builder
method chain (where/orderBy/limit/get/first/...), Eloquent Model
static factories, Illuminate\Support\Collection chains
(map/filter/sort/...), Symfony HttpFoundation Request/Response,
Carbon date methods, PSR-7 with*() builders.
Test suite: 81 -> 100 tests, all passing. Total: 2913 / 0.
Adds the resolver passes that close most of the gap to a real PHP language
server's type inference, while staying in-process and PHP-runtime-free.
New capabilities:
- Type narrowing: instanceof, is_string/int/float/bool/array/object/...
is_callable, is_iterable, is_numeric. Active inside if/elseif bodies
and after assert(...) (sequential narrowing)
- Property type tracking: typed property declarations, constructor
property promotion (public X $foo), and constructor-body inference
($this->bar = $bar where $bar is typed)
- Late-static-binding chain depth: full ancestor walk with cycle
detection for $obj->method() and Class::method() resolution
- Trait flattening: methods from `use Trait;` inside a class body are
re-registered with the using class as receiver; basic `as` aliasing
is honored
- PHPDoc class-level tags: @property TYPE $name binds a virtual field
on the class; @method TYPE name() registers a virtual method
(Eloquent-style dynamic dispatch)
- Match expression result type: union over arm result expressions,
first concrete type wins
- Ternary result type: prefers the consequent, falls back to alternative
- Variadic parameter binding: $args bound to `array`
- `use Vendor\\Helper as H;` two-name form correctly maps H -> Vendor.Helper
- Static-call unindexed fallback: when class is resolved from `use`
but methods unindexed, emit php_static_unindexed for the bridge
- Stdlib expansion: full Eloquent Builder + Model + Collection method
chains, Symfony HttpFoundation Request/Response, Carbon date/time,
PSR-7 with*() builders
Test suite expanded from 16 to 81 tests (2829 -> 2893 total).
Ivan Porto Carrero