mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-09 09:11:00 +08:00
v0.7.0
177
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Martin Vogel
|
7a6a8cb7a5 |
style: apply clang-format-20 across tree
Format drift had accumulated across recent commits (the dry-run lint enforces clang-format-20, which had not run on these commits yet). No functional changes — whitespace/layout only, verified by a clean cppcheck + clang-format + NOLINT lint-ci pass. |
||
|
Martin Vogel
|
b5f086974d |
fix(ci): resolve cppcheck null-deref + variable-scope and security-audit findings
cppcheck (warning+style, error-exitcode=1): guard cbm_fqn_compute against a NULL project/rel_path (ctunullpointer reachable from cbm_extract_file), and narrow prefix_len/suffix_len (path_alias) and ft_count (pass_githistory) to the scopes that use them. security-audit.sh: the mcp.c file-read count grew to 13 (search/ADR/Windows-support reads — all path-contained or transport reads, audited, no new exfiltration surface); bump the reviewed maximum and document the update-check + request-body reads. Allow the diagnostics.c atomic metrics dump (.tmp+rename) in the file-write scan, and allow-list the sqlite WAL-checkpoint doc URL referenced in a store.c comment. |
||
|
Martin Vogel
|
bb1411620a |
refactor(pipeline): always run LSP; remove opt-in advanced mode
The per-file and cross-file LSP type-aware call/usage resolution now runs in every index mode (full/moderate/fast) instead of being gated behind a separate CBM_MODE_ADVANCED. Remove the mode from both index-mode enums, the g_lsp_enabled toggle and its save/restore dance, and the per-mode gating in discover, pipeline and the incremental path. Now that the LSP walkers are O(n), the callee-attribution accuracy (e.g. ~91k call overrides on roslyn, ~60k on kubernetes) is worth the latency in every mode. Fast mode still filters files and skips similarity/semantic edges, so it stays faster than full while now also benefiting from LSP resolution. A 'mode':'advanced' request falls back to the full default — identical behaviour to the old advanced mode, so existing callers are unaffected. This also fixes the test_incremental.c failures, which were asserting LSP-resolved results that the gated incremental path never produced. |
||
|
Martin Vogel
|
1b9d9c136f |
perf(lsp): make per-file and cross-file LSP walkers O(n); fix two crashes
The advanced-mode LSP passes walked the AST with the same O(i) ts_node_child(node, i) idiom, so a wide root made each per-file pass quadratic — the TS fixture reallyLargeFile.ts took ~16000s. Add a shared cbm_lsp_collect_children() helper (lsp_node_iter.h) and convert the entry and recursive walkers across the TypeScript, C, C#, PHP and Go LSP back-ends to a single O(n) cursor pass. The file now indexes in ~28s; full advanced-mode runs: TS 100s, roslyn 46s, kubernetes 51s, postgres 8s, WordPress 7s. Also fix two latent crashes exposed once the resolve phase actually ran: process_node indexed the NULL-terminated param_types[] by the call's argument count, reading past the terminator when a call passed more args than the callee declares; and return_type_of() built a tuple via cbm_type_tuple(NULL, ...), passing a NULL arena — it now threads ctx->arena. This also includes the Tier 2 cross-LSP registry plumbing (shared read-only base registry with per-file overlay/fallback) the walkers build on. All LSP unit suites pass: TS, C 737/737, C# 60/60, PHP 280/280, Go 55/55. |
||
|
Martin Vogel
|
01bdba2a3f |
perf(extract): eliminate O(n^2) AST traversal on wide-flat files
tree-sitter's ts_node_child(node, i) is O(i) — it walks the child iterator from the first child on every call — so the common 'for (i=0; i<count; i++) ts_node_child(node, i)' idiom is O(n^2) over a node's children. On a file whose top level is very wide (e.g. a 580k-line fixture parsed as hundreds of thousands of sibling comment nodes) this made extraction run for over an hour. Add ts_nstack_push_children(), which enumerates a node's children in a single O(n) cursor pass, and use it in the whole-tree DFS walkers (calls, channels, env-accesses, imports, type-assigns). Convert the module-level variable walk to a cursor too, and add cbm_is_module_level_p() so callers that already hold the parent node skip the O(n) ts_node_parent() rescan. Indexing microsoft/TypeScript drops from ~5100s to ~50s with identical graph output. |
||
|
Martin Vogel
|
7d03632597 |
perf(lsp): C# Tier 2 — pre-built per-language cross-LSP registry
Same clean def-driven pattern as C/Go. Extracts cs_register_lsp_defs, adds cbm_cs_build_cross_registry (filters CBM_LANG_CSHARP) + cbm_run_cs_lsp_cross_with_registry. resolve_worker dispatches CBM_LANG_CSHARP to the shared registry; pipeline.c builds cross_registries.cs; pass_lsp_cross.h includes cs_lsp.h. C# cross-LSP is def-driven (no per-file AST field collection), so the shared registry yields identical resolution to the per-file build. roslyn bench (17,916 C# files, ADVANCED): single shared registry (lsp_cross_prepare=234ms, 260,814 defs, zero slow files), 91,089 LSP overrides applied, 58.6s total. All 60 test_cs_lsp tests pass; serilog sanity exact (2,627n/7,112e). No SEGV/ASan. |
||
|
Martin Vogel
|
ef054ebd4e |
perf(lsp): C/C++ Tier 2 — pre-built per-language cross-LSP registry
Brings C/C++/CUDA to Go's optimization level. Extracts the cross-LSP
registration loop into c_register_lsp_defs (shared by per-file and
Tier 2 paths), adds:
cbm_c_build_cross_registry — builds ONE project-wide registry from
all C-family defs (filters by lang) + both C and C++ stdlibs,
finalized, shared READ-ONLY across resolve workers.
cbm_run_c_lsp_cross_with_registry — per-file resolve using the
shared registry; no per-file registry rebuild.
C cross-LSP is DEF-DRIVEN (fields/returns/embeds come from the def
strings, no per-file AST field collection), so the shared registry
produces identical resolution to the per-file build — zero quality
loss. resolve_worker dispatches CBM_LANG_C/CPP/CUDA to the new path;
pipeline.c builds cross_registries.c; pass_lsp_cross.h includes c_lsp.h.
postgres bench (4,967 files, ADVANCED mode): 628 C files share one
pre-built registry (lsp_cross_prepare=17ms, zero slow files), 44 LSP
overrides, +9 edges vs FULL. All 200 test_c_lsp tests pass. No
SEGV/ASan. Total pipeline overhead vs FULL: +5.2%.
|
||
|
Martin Vogel
|
b61861eb5a |
feat(pipeline): add advanced index mode — gate ALL LSP behind it
New CBM_MODE_ADVANCED=3. Both per-file LSP (type-aware resolution during extract) and the cross-file LSP pass (during resolve) are now OPT-IN: FULL and below run tree-sitter extraction + registry textual resolution only; ADVANCED adds the full LSP layer. Rationale: profiling showed LSP changes the kubernetes graph by only ~0.01% of edges (it mostly re-confirms textual resolution) while costing significant memory + time. Making it opt-in keeps the default FULL path lean (3.0 GB vs 5.6 GB peak RSS on K8s) without meaningfully worse output, and lets users pay for LSP precision only when wanted. Mechanism: - cbm.c: g_lsp_enabled atomic toggle wraps the entire per-file LSP block (Go/C/C++/PHP/Python/TS/JS/C#) + the C/C++ preprocessor-pass LSP call. Default true so ad-hoc extraction / tests keep historical behaviour. cbm_set_lsp_enabled / cbm_get_lsp_enabled in cbm.h. - pipeline.c: sets the toggle (save→set→restore so it never leaks to other callers) from p->mode before extraction; gates the cross-LSP all_defs collection + registry build on advanced; gates the sequential lsp_cross dispatch. Fixed the moderate_only comparison (== FAST, not > MODERATE) so ADVANCED is not treated as lighter. - pipeline_incremental.c: same toggle save/set/restore. - discover.h/.c: added CBM_MODE_ADVANCED to discover's enum + all 4 filter comparisons treat ADVANCED like FULL (full discovery, no fast-filtering). Without this, ADVANCED (numerically 3) was mis-detected as fast-mode and dropped ~half the files. - mcp.c: parse mode "advanced" + tool schema enum/description. Dual-mode K8s bench: FULL 46.7s (lsp_overrides=0, 3.0 GB) vs ADVANCED 48.8s (lsp_overrides=80,818, 5.6 GB). Graphs within 0.01%. All 3,615 tests pass. |
||
|
Martin Vogel
|
2ee0c68d4a |
perf(foundation): swap CBMHashTable internals to Verstable v2.2.1
Replaces the prior Robin-Hood open-addressing implementation with Verstable (https://github.com/JacksonAllan/Verstable), a 2024 state- of-the-art C hash table. The public API in foundation/hash_table.h is UNCHANGED — CBMHashTable is now an opaque type and all 14 production callers (graph_buffer, registry, pipeline passes, service_patterns, watcher, semantic, etc.) get the new impl transparently. str_intern.c keeps its specialised (ptr+len) hash — different semantics, not a clean swap target. Why: per-sub-phase profiling pinpointed cbm_pipeline_lsp_target_node and cbm_registry_resolve as the dominant resolve_calls cost. Both ultimately hit cbm_ht_get; both run millions of times on a kubernetes index. Verstable's key innovations target exactly this: * 4-bit hash-fragment metadata in a uint16_t sidecar array lets chain walks skip most key comparisons WITHOUT touching the bucket payload (cache wins on hot lookups). * Quadratic probing with per-bucket displacement chains (not linear Robin-Hood psl). * Tombstone-free deletion that moves at most one existing key. Vendored at internal/cbm/vendored/verstable/verstable.h (2069 lines, 84KB, MIT). Wrapped as a const char* → void* template inside hash_table.c. The header struct is opaque (forward declaration only); audited the entire tree — no caller accesses internals or takes sizeof(CBMHashTable), so opacity is safe. Test refactor: test_hash_table.c had two impl-specific tests: * ht_robin_hood_bounded_psl — Robin-Hood specific; removed. ht_stress_10000 covers the functional equivalent. * ht_power_of_two_capacity — renamed ht_create_small_capacity, replaced struct probes with a functional insert+lookup check. ht_create_capacity_zero/one similarly de-internalised. Benchmark on kubernetes (12 workers, 20,650 files): - parallel_resolve wall: 36,376 -> 27,217 ms (-25.2%) - pipeline.done total: 61,485 -> 51,745 ms (-15.8%) - registry_build: ~1,000 -> 779 ms (faster — registry uses cbm_ht_set heavily during build) - All 3,615 tests pass; hash_table suite 100% green - ASan/UBSan/SEGV: none - Peak RSS: ~6.83 GB - Graph: nodes 297,082 -> 297,102 (+20, 0.007%), edges 1,155,444 -> 1,154,025 (-1,419, -0.12%) — within tolerance; minor delta likely from hash-iteration-order differences affecting tie-breaks in resolution. Sanity (zod/serilog): nodes exact, edges within 0.1%. Cumulative trajectory from broken-state baseline ~35 min on K8s -> 51.7s wall = ~40x total speedup. |
||
|
Martin Vogel
|
1af1dd8688 |
perf(service_patterns): per-worker TLS cache for cbm_service_pattern_match
cbm_service_pattern_match runs once per resolved CALL edge inside
emit_service_edge — that's 7 pattern lists × ~30 patterns × strstr
= ~210 strstrs per call. On kubernetes (~600k resolved CALL edges)
the SAME resolved QN ("fmt.Errorf", "context.Context.Done",
"errors.New", etc) repeats thousands of times across files. Caching
turns the linear pattern-list scan into one hash lookup after the
first miss.
API additions in service_patterns.h:
void cbm_service_pattern_cache_begin(void);
void cbm_service_pattern_cache_end(void);
Cache is TLS (no contention) scoped to the resolve_worker's lifetime
in the parallel_resolve phase. Capacity 8192 entries per worker —
covers typical project QN diversity without rehashing.
Encoding: enum value + 1 in the void* slot (0/NULL means "miss").
Two helpers svc_enum_to_ptr / svc_ptr_to_enum hide the trick.
Benchmark on kubernetes (12 workers, 20,650 files):
- parallel_resolve wall: 98,520 -> 90,104 ms (-8.5% vs Chunk 1
baseline, -12.8% vs original 103,281 baseline)
- resolve_calls CPU: 992,427 -> 885,663 ms (-10.8% vs Chunk 1,
-5.4% vs original — Chunk 1 added overhead that Chunk 2 recovers
AND surpasses)
- resolve_usages CPU: 11,977 -> 10,944 ms (-8.6%)
- total_loop CPU: 1,005,409 -> 897,561 ms (-10.7%)
- Tests: 3616/0
- Graph: 297,082 -> 297,080 nodes (-2), 1,153,482 -> 1,156,097 edges
(within 0.2%, semantic-preserving)
- zod/serilog sanity: edges within +/-3 of baseline
- Peak RSS: 7.46 -> 7.75 GB (+290 MB for 12 × 8192-entry caches)
Cumulative trajectory: original broken state ~35 min K8s ->
1.92 min wall (-49× total). The per-file is_import_reachable cache
(Chunk 1) + the per-worker service-pattern cache (Chunk 2) together
save ~13s of resolve wall time. Next chunks target the gbuf_insert_
edge hot path and remaining strstr scans.
|
||
|
Martin Vogel
|
96f6891ff2 |
perf(lsp): Tier 3 metadata-driven cross-LSP — skip AST re-walk for Go
KEY ARCHITECTURAL INSIGHT (credit: user observation):
The per-file LSP during extract ALREADY captures the inferred receiver
type QN + method name for unresolved cross-file calls — emit_unresolved_
call at go_lsp.c:1242-1245 stores callee_qn as '<FullReceiverTypeQN>.<
MethodName>' in result->resolved_calls flagged with strategy='lsp_
unresolved'. The cross-LSP step was REDUNDANTLY re-deriving this same
metadata via a second tree-sitter parse + AST walk.
Cross-LSP for Go is now METADATA-DRIVEN:
cbm_go_fast_resolve_qualified_calls (rewritten):
- Iterates result->resolved_calls where strategy=='lsp_unresolved'
- For each entry: try direct registry lookup of callee_qn;
if that misses, split prefix.suffix and resolve prefix via the
file's import map then lookup '<module_qn>.<suffix>' in registry
- Emits resolved entry on top (cbm_pipeline_find_lsp_resolution
picks highest confidence)
- NO tree-sitter parse, NO AST walk, NO scope tracking
resolve_worker (pass_parallel.c):
case CBM_LANG_GO now calls ONLY the metadata-driven fast path.
cbm_run_go_lsp_cross_with_registry (the slow AST-walk version) is
no longer invoked for Go — it would just re-derive identical
metadata via a second AST walk and arrive at the same answers.
Benchmark (kubernetes, full fresh index, 12 workers):
- parallel_resolve: 1.89 min (Tier 3 meta) vs 1.86 min (Tier 2) — parity
- lsp_cross.slow lines: 0 (no tree-sitter fallback runs)
- Nodes: 297,081 vs 297,082 baseline (-1)
- Edges: 1,153,163 vs 1,153,482 baseline (-319, within 0.03%)
- Tests: 3,616 / 0 failures
- zod/serilog sanity: nodes exact, edges +1-3 (within 0.03%)
The wall-time parity is informative: the AST re-walk wasn't the
dominant cost. Per-file budget (66ms across 20,650 files / 12 workers)
is now dominated by the 5 sequential resolve passes (resolve_file_
calls/usages/throws/rw/semantic) pushing ~10M total edges into the
per-worker gbuf with arena allocs + hash-bucket inserts. The Tier 3
work eliminates a genuine architectural redundancy (two walks of the
same AST producing the same metadata) and sets up Python/C/etc to
adopt the same metadata-driven pattern in follow-up commits — but
hitting the user-target <30s K8s requires attacking the gbuf hot
path, not the cross-LSP path.
|
||
|
Martin Vogel
|
23d404a056 |
perf(lsp): Tier 2 — pre-built per-language cross-LSP registries (Go + Python)
The gopls 'package summary' pattern, applied properly: build the cross-
LSP CBMTypeRegistry ONCE per language at pipeline time (in a dedicated
cross_lsp_arena), share it READ-ONLY across all resolve workers via the
CBMCrossLspRegistries struct. Per-file work then becomes just tree-
sitter parse + AST walk + O(1) hash lookups — zero per-file registry
rebuilds for languages with a pre-built registry.
Each LSP gains two new public functions:
CBMTypeRegistry *cbm_<lang>_build_cross_registry(arena, defs, def_count)
— filters all_defs to its language, init + stdlib + registration
loop + cbm_registry_finalize, returns finalized registry.
void cbm_run_<lang>_lsp_cross_with_registry(arena, source, ...,
CBMTypeRegistry *reg, ...)
— skips the per-file build AND the per-file Phase 1b/1c AST scans
(those mutate the registry, which would race on a shared read-
only reg). Trade-off: a few file-local type aliases discovered
only by Phase 1b are missed; in practice the per-file extract
LSP already captured them via embedded_types/field_defs/etc.
Tested with parallel_python_lsp_override_cross_file_emits_lsp_
strategy_edges — fused + filtered + prebuilt path stays correct.
CBMLSPDef gains a CBMLanguage lang field set in pxc_build_lsp_def so
the per-language build functions can filter the global all_defs[].
cbm_parallel_resolve signature now takes CBMCrossLspRegistries* instead
of a single CBMTypeRegistry*. The resolve_worker dispatches:
cbm_pxc_registry_for_lang(lang) returns the right pre-built reg if any;
the worker then switch()es to the matching _with_registry variant.
Languages without a _with_registry variant fall through to the existing
filter + per-file build path.
Kubernetes (Go-heavy, 12k files, 139k defs) measured perf:
parallel_resolve: 5.51 min (strdup-removal baseline) -> 1.86 min
= 2.96x speedup
parallel.resolve.lsp_cross.slow lines: 1,258 -> 2
(-99.8%; the 2 remaining are Python files, not Go)
Peak RSS: 6.29 GB -> 6.23 GB (slight decrease — pre-built registry
shared, no per-file scratch)
All 3,616 tests pass; zod/serilog graphs exact match (per-file path
unaffected).
Cumulative trajectory from the original broken state on kubernetes:
35 min -> 19 min (finalize) -> 6.24 min (module-def filter) ->
5.51 min (strdup-removal) -> 1.86 min (Tier 2 Go) = 18.8x total.
Languages wired so far: Go, Python. C/C++, TS/JS, PHP, C# still use the
per-file filter+build path until their _with_registry variants land in
follow-up commits.
|
||
|
Martin Vogel
|
3b373b725f |
chore(deps): vendor wyhash 4.3 (public domain, single header)
wyhash is a non-cryptographic hash function ~2x faster than FNV1A for the long string keys used in CBMTypeRegistry's hash buckets (qualified names average ~50 bytes). The header is unused for now — prepared for a future Tier 3 symbol-interning pass that replaces the QN-string-keyed hashtables with u32-keyed ones (rustc pattern) where wyhash's lower collision rate matters more. Source: https://github.com/wangyi-fudan/wyhash (Unlicense) Vendored as a single header at internal/cbm/vendored/wyhash/wyhash.h. |
||
|
Martin Vogel
|
770d2731cc |
perf(lsp): finalize cross-LSP registry + borrow def strings
Two perf changes across the 6 cross-LSP resolvers: 1. cbm_registry_finalize after the per-file registry build (all 6 LSPs). Builds the FNV1A hash buckets so lookups go through the O(1) hashed path in type_registry.c:154. Previously this was NEVER called anywhere in the codebase, so every method/type lookup fell through to the O(N) linear scan — the root cause of the 35-min kubernetes regression. Lookup count on k8s: ~50 calls * 110k defs * 11k files = ~64B strcmps under the old fallback. 2. In go/c/py/php registration loops, replace cbm_arena_strdup(arena, d->X) with d->X directly. The strings are borrowed from the caller's all_defs[] arena which outlives the cbm_run_X_lsp_cross call by the whole pipeline lifetime. Saves 5 wasted strdups per def per file = ~6B mallocs avoided on kubernetes. TS and CS use a different registration architecture (no inline strdups in the body — uses a per-file ctx). Only the finalize change applies there. Kubernetes wall-time vs prior commit: - After finalize alone: 35 min -> 19 min (~1.8x) - After module-def filter (prior commit): -> 6.24 min (~5.6x) - After strdup-removal (this commit): -> 5.51 min (~6.4x) - Memory: 7.14 GB -> 6.29 GB peak RSS (-12%) All 3616 tests pass, zod/serilog graph output exact match. |
||
|
Martin Vogel
|
75a0df688f |
feat(pipeline): fuse cross-LSP into resolve worker + module-def filter
Previously cbm_pipeline_pass_lsp_cross ran as a separate sequential pass that re-read every source file from disk and re-parsed every tree-sitter tree on a single thread — a 50x regression on the kubernetes Go monorepo (extract 10s, lsp_cross 9+ min when killed). Three changes: 1. CBMFileResult gains const char *source / int source_len. The parallel extract worker copies source bytes into result->arena (per-file 100MB cap, project-wide 2GB atomic cap) so the fused cross-LSP step can run without disk I/O. Source pointer is borrowed by downstream readers; arena owns the storage. 2. Cross-LSP work moves into the parallel resolve worker. Each worker processes its file with the existing parallel infrastructure (worker pool + work stealing). Slab is reclaimed after each per-file cross-LSP call to bound peak slab memory. The previously separate lsp_cross pass entry function is kept for the sequential pipeline path; the parallel path no longer invokes it. 3. CBMModuleDefIndex (gopls 'package summary' pattern) — an inverted index module_qn -> list of def indices built once in O(D) before resolve_worker fires. Per file, cbm_pxc_filter_defs_for_file slices all_defs down to the file's own_module + imported modules only, typically 50-100x smaller than the global ~110k defs on kubernetes. Drops the per-file registry build from O(all_defs) to O(relevant). Other plumbing: cbm_parallel_resolve signature gains all_defs, def_count, def_modules, module_def_index params. pipeline.c builds the index between registry_build and parallel_resolve and frees it after. pipeline_incremental and tests/test_parallel updated to match. Helpers in pass_lsp_cross.c exposed via the header: cbm_pxc_has_cross_lsp, cbm_pxc_collect_all_defs, cbm_pxc_run_one(_ts), cbm_pxc_ts_modes, cbm_pxc_build_module_def_index, cbm_pxc_filter_defs_for_file. Kubernetes wall-time: 35 min -> 6.24 min (~5.6x) before the followup perf commits in this series. |
||
|
Martin Vogel
|
6226972f13 |
fix(lsp): guard trailing append in TS type-text union/intersection parser
parse_ts_type_text in ts_lsp.c builds a 16-slot CBMType* members[] for union (|) and intersection (&) types. The in-loop branch correctly caps additions at mc<15, but the trailing post-loop append that collects the final member after the last separator was unconditional, and the subsequent members[mc] = NULL sentinel could then write to members[16] — one past the array end. Caught by UBSan on the zod codebase (a TS union with >=16 members exists). Non-deterministic crash signature under pthread-parallel extraction (12 workers): exit 134 / SIGABRT with no stack, because libc's allocator detected the heap corruption and called abort(). Single-fixture-file test suite never exercised the parallel path or a wide-enough union, so existing ASan/UBSan coverage missed it. Fix: mirror the in-loop guard on the trailing append (cap at mc<15). Add regression test tslsp_union_many_members_no_overflow that constructs a 20-member string-literal union — extracting must not crash. UBSan: clean. Full suite: 3616 passed, 0 failed. |
||
|
Martin Vogel
|
03e01dc616 |
feat(lsp): add C# / .NET hybrid LSP resolver
Add a C# language LSP resolver mirroring the existing Go/C/PHP/Python
modules: type evaluation, using-directive resolution, method lookup,
and a corpus-seeded .NET BCL stdlib data table.
- internal/cbm/lsp/cs_lsp.{c,h}: resolver implementation
- internal/cbm/lsp/generated/cs_stdlib_data.c: BCL type/method tables
- wire CBM_LANG_CSHARP dispatch in cbm.c and lsp_all.c
- register cs_lsp + cs_lsp_bench suites in the test harness
- tests/test_cs_lsp.c, tests/test_cs_lsp_bench.c
(cherry picked from commit 8f677c67a0c9d7829cd6ac27393de2d0aaaac534)
|
||
|
Martin Vogel
|
2b6f80ec3c |
fix(sqlite_writer): log skipped oversized index cells
|
||
|
Jos Joosten
|
98d5cacd76 |
fix(sqlite_writer): skip index cells that exceed a single page
write_index_btree() flushed the page buffer whenever the next cell didn't fit, on the assumption that a freshly-initialised page would always accept any cell. That invariant fails for index cells whose payload (qualified_name, file_path, etc.) is larger than a full SQLite page — after the flush, pb_cell_fits() is still false but the code calls pb_add_cell() anyway. The subsequent content_offset underflow corrupts the page header and the next write triggers SIGBUS on large repos (observed on a Ruby codebase with ~215K nodes). Re-check pb_cell_fits() after the flush and continue past cells that still don't fit. Index entries whose keys exceed a full page can't be stored and are not expected to survive the writer anyway — the rest of the index is correctly preserved and the indexer no longer crashes. Record overflow pages (#175) already covers the record-btree side of this same shape of bug. |
||
|
Martin Vogelandsponger94
|
70839551d3 |
feat(extract): detect typed-stub factory constructors
extract_constructor_type only accepted call-form RHS when the function name's first character was uppercase. Misses two common typed-stub factory patterns: Go: client := pb.NewPromoCodeClient(conn) Java: stub := PromoCodeServiceGrpc.newBlockingStub(channel) Both have a lower-cased qualifier prefix, so the first character of the full text is `pb`/`promoCodeServiceGrpc` rather than the factory name. Match the last `.`-segment instead and accept it when it begins with `New`/`new` and ends with `Client` or `Stub`. Returns the full qualified call so downstream passes keep the package context. Stays scoped — only the new factory shape is added; the uppercase-first- char path for `Foo()`-style direct calls is unchanged. Co-authored-by: sponger94 <45746997+sponger94@users.noreply.github.com> |
||
|
Martin Vogelandsponger94
|
beef06ec6d |
feat(extract): C# field/property + C# 12 primary-constructor support
Three C# extractor gaps that left modern .NET classes partially invisible
in the graph:
1. lang_specs.c CSHARP entry had `empty_types` for field_node_types, so
walk_defs never descended into field_declaration / property_declaration
nodes. Add cs_field_types and wire it in. Without this, no C# fields
or properties were ever pushed as Field defs.
2. extract_class_fields assumed the field's "type" is on the immediate
child via ts_node_child_by_field_name. Tree-sitter-c_sharp wraps fields
as field_declaration > variable_declaration(type, variable_declarator),
one level deeper than Java/Go/Rust. Detect the nested shape by walking
named children for variable_declaration and resolve type + name from
inside it. Direct shape continues unchanged.
3. C# 12 primary constructors (`class Foo(IBar bar) : Base { ... }`)
declare implicit captured fields on the class line. Tree-sitter wraps
them inside the hidden _class_declaration_initializer node, so the
`parameters` field on class_declaration may not always resolve
directly; iterate top-level children for parameter_list as a fallback.
Emit each primary-ctor parameter as a Field def with parent_class set.
Effect: dependency-injected fields on .NET 8+/9+ controllers and any C#
auto-property (`public IFoo Foo { get; }`) are now reachable as Field
nodes in the graph.
Co-authored-by: sponger94 <45746997+sponger94@users.noreply.github.com>
|
||
|
Martin Vogel
|
b2a71b80b9 |
feat(pipeline): wire cross-file LSP across all five hybrid resolvers
Per-file LSP (run from cbm_extract_file) only sees one file's defs, so
callees whose receiver type comes from an imported module stayed
unresolved — even though Go/C/Py/TS already had cross-file entry points
(cbm_run_X_lsp_cross). Those entry points existed but no pipeline pass
ever invoked them. PHP didn't even have a cross-file entry point.
This change lands the missing pieces:
1. cbm_run_php_lsp_cross + cbm_batch_php_lsp_cross — PHP cross-file
resolver mirroring py_lsp_cross / ts_lsp_cross. Reuses PHPLSPContext,
php_lsp_init, php_lsp_add_use, php_lsp_process_file. Handles PHP-
specific labels (Class / Interface / Trait / Enum / Type) and uses
def_module_qn so cross-file return types qualify against the def's
own module rather than the importing module's.
2. pass_lsp_cross.c — new pipeline pass. For every cached file result,
builds a project-wide CBMLSPDef[] (CBMDefinition → CBMLSPDef
converter), per-file import map (from gbuf IMPORTS edges), then
dispatches to the matching cbm_run_X_lsp_cross. Output strings get
copied into the per-file arena (with O(N²) memory bug avoided by
running each file inside a fresh scratch arena that gets destroyed
after the LSP call — the LSP allocates its registry + stdlib +
project defs in scratch, not in the caller's persistent arena).
3. Wired into both pipelines:
- sequential: definitions → k8s → lsp_cross → calls → ...
- parallel: parallel_extract → registry_build → lsp_cross →
parallel_resolve → ...
4. Updated tests/test_parallel.c run_parallel helper to mirror the new
parallel ordering so existing cross-file Python/PHP tests now exercise
pass_lsp_cross.
Languages covered: Go, C/C++/CUDA, Python, TS/JS/JSX/TSX, PHP. PHP cross
file is new. The other four had functions but no pipeline integration
until this change.
Verified on a 1670-file FastAPI index: pass_lsp_cross processes 1050
Python files (525 non-LSP files skipped), collects 4070 project defs in
9.2s, and parallel_resolve subsequently reports lsp_overrides=568 — the
LSP-strategy attribution gain over per-file alone.
Per-LSP emit functions dedup by (caller_qn, callee_qn) within their own
run; pxc_append_results dedups again at append time so cross-file
entries duplicating per-file output get dropped instead of inflating
result->resolved_calls.
|
||
|
Martin Vogel
|
d1143fbdce |
feat(lsp): TypeScript / JavaScript / JSX / TSX hybrid LSP resolver
Adds clean-room ts_lsp resolver covering single- and cross-file type resolution for TS, JS, JSX, TSX. Mirrors the structure and entry-point pattern of the existing Go and C/C++ LSP hybrids (single-file cbm_run_ts_lsp + batch cbm_run_ts_lsp_cross), with TS-specific surface: - Hash-table indexed type registry (FNV-1a, lazy via cbm_registry_finalize) - Partial structural relater for NAMED / BUILTIN / LITERAL / TUPLE / UNION / INTERSECTION / FUNC / TEMPLATE / TYPE_PARAM / ALIAS - Conditional types T extends U ? X : Y with union distribution - 'infer X' constraint solver, keyof / typeof / T[K] type-position parsing - Flow-sensitive narrowing (instanceof / typeof / discriminated unions), optional chaining, non-null assertion - TS utility-type passthrough (Partial, Pick, ReturnType, Awaited, etc.) - Function overload resolution by argument types - Async iterables + 'for await ... of' - Curated DOM stdlib subset + comprehensive base stdlib - Mode flags: js_mode (.js/.jsx with JSDoc inference), jsx_mode (JSX expressions), dts_mode (.d.ts ambient declarations, no resolved calls) Touches existing files only with additive changes: - type_rep.h/.c: new TS-specific CBMTypeKind values + constructors - type_registry.h/.c: TS-specific fields, hash indexes, cbm_registry_finalize - cbm.c: dispatch for JS/TS/TSX languages - lsp_all.c, Makefile.cbm, tests/test_main.c: include + suite registration - README.md: feature list mention 291 ts_lsp test cases. CBM_LSP_DISABLED=1 disables the resolver (matches CBM_LSP_DEBUG / CBM_LSP_DISABLED knobs in go_lsp / c_lsp). |
||
|
Martin Vogel
|
e6f4aa74bd |
Merge origin/main into worktree-php-lsp-integration
Resolved 5 conflicts where the php-lsp branch and main both added LSP
infrastructure but with different APIs:
- Makefile.cbm: kept both TEST_PHP_LSP_SRCS and main's py_lsp/scope/
type_rep/stack_overflow test source variables in ALL_TEST_SRCS.
- internal/cbm/cbm.c: kept both #include lsp/php_lsp.h and lsp/py_lsp.h;
the per-language dispatch already supports stacking PHP after PYTHON.
- internal/cbm/lsp_all.c: kept both lsp/php_lsp.c + php_stdlib_data.c
AND lsp/py_lsp.c + python_stdlib_data.c in the compilation unit.
- src/pipeline/pass_calls.c: took main's version entirely.
find_lsp_resolution + resolve_single_call(ctx, call, lsp_calls, ...)
is language-agnostic — it matches any CBMResolvedCall whose caller_qn
hits the textual call's enclosing function and whose callee_qn
short-name hits the textual callee. PHP LSP populates the same
result->resolved_calls array via cbm_run_php_lsp, so the unified
resolver handles PHP for free. Dropped the php-branch's
qn_short_name + lsp_override_resolution helpers as superseded.
Locked confidence floor at main's CBM_LSP_CONFIDENCE_FLOOR = 0.6f
(was 0.5f on the php branch).
- tests/test_main.c: kept all suite externs and RUN_SUITE invocations
for php_lsp + py_lsp/py_lsp_bench/py_lsp_stress/py_lsp_scale +
scope + type_rep.
Auto-merged: src/pipeline/pass_parallel.c.
Brings in 52 commits from main since the branch was forked at
|
||
|
Martin Vogel
|
4d9f62f7e6 |
Merge origin/main into worktree-python-lsp-integration
Resolved conflict in Makefile.cbm: keep both TEST_STACK_OVERFLOW_SRCS
(from main, #217) and the new py_lsp test variables (TEST_SCOPE_SRCS,
TEST_TYPE_REP_SRCS, TEST_PY_LSP_SRCS, TEST_PY_LSP_BENCH_SRCS,
TEST_PY_LSP_STRESS_SRCS, TEST_PY_LSP_SCALE_SRCS) in ALL_TEST_SRCS.
Other auto-merged files: internal/cbm/extract_defs.c (PR #279),
tests/test_main.c (multiple suite registrations on each side).
Brings in 28 commits from main since the branch was forked at
|
||
|
Martin Vogel
|
89531c7501 |
feat(php-lsp): @phpstan-type aliases + Closure::bind + variance + 30 tests
Phase 5f, 4ac, 4ad: closing the long-tail gaps toward LSP-server parity.
- @phpstan-type / @psalm-type / @phan-type aliases parsed from class
docblocks and applied during PHPDoc type resolution. References to
the alias name in @var/@param/@return are rewritten to the aliased
type. Aliases live on PHPLSPContext (per-file).
- @phpstan-import-type accepted without crash (cross-file alias
resolution deferred — same root cause as Composer PSR-4).
- @template-covariant T and @template-contravariant T now register T
as a proper type-param (previously skipped along with
@template-extends).
- Closure binding: \\Closure::bind($f, $obj) and $f->bindTo($obj),
when invoked with an inline closure literal as the first arg, walk
the closure body with $this rebound to the second arg's class.
Resolves the Laravel-macroable / VarDumper / encapsulation patterns.
- @return ($x is string ? int : bool) conditional return-type syntax
accepted without crash (parsed leftmost segment as fallback).
- Long PHPDoc blocks with mixed @param/@return/@throws/@internal
parsed without misattributing tags.
Stdlib unchanged.
Tests: 248 → 278 covering:
- phpstan-type alias basic / via @param / union / array-shape
- psalm-type alternate spelling
- phpstan-import-type (no-crash)
- Closure::bind / ::bindTo / static-bind on arrow fn
- conditional return / template-covariant / -contravariant (no-crash)
- class-string<T>, int<0,100>, literal-string (no-crash on phpstan
types we don't fully model)
- 8 realistic patterns: repository, event listener, console command,
form handler, mail sender, cached service, logger w/ context,
Carbon chains in methods, Eloquent model.
Total tests: 3061 → 3091 / 0 failed.
|
||
|
Martin Vogel
|
ecfbbf48d6 |
perf(py_lsp): cap dedup window at 256 entries — 91.8x -> 31.4x scaling
Audit for O(n^2) hot paths after Round 10. The dedup loop in py_emit_resolved_call scanned ALL prior entries linearly, making emission O(N) and total file work O(N^2). On a synthetic 2000-class fixture with 8000 calls, that dominated time at 5083 ms. Cap the scan window at the last 256 entries. Most duplicate emissions come from the same expression being walked twice (resolver + emitter passes), so they're temporally adjacent — a 256-entry window catches every common case while keeping the dedup O(1) per emission. Scale measurements (test_py_lsp_scale.c added): | Classes | Calls | Resolved | Before | After | |--------:|------:|---------:|-------:|------:| | 100 | 400 | 300 | 55ms | 57ms | | 500 | 2000 | 1500 | 333ms | 323ms | | 2000 | 8000 | 6000 | 5083ms | 1804ms| Ratio for 20x input: 91.8x -> 31.4x (linear would be 20x; quadratic would be 400x). No remaining quadratic hot path in code added by this branch. Other hot paths surveyed: - py_register_def -> cbm_registry_lookup_type: O(types) per def at registration, O(D x types) for D defs. Pre-existing linear scan in type_registry.c; hash-indexing it would be a cross-language optimization (Go and C++ LSPs benefit too). - py_lookup_attribute / py_lookup_field: O(types) per call site. Same pre-existing linear scan. With ~900 stdlib types + per-file defs, accounts for ~75µs/call at 2000-class scale. - MODULE submodule prefix scan: O(funcs) per module-attribute access, early-exits on first match. Bounded by registry size. - lambda / dict-literal registries: linear, but bounded by per-file N which is small. Bench unchanged at 100% / 12ms (52/52 calls, 178 LOC). |
||
|
Martin Vogel
|
d8646dae91 |
feat(php-lsp): trait self substitution + 2-pass class processing
When a trait method declares `: self` as its return type, flattening
that method into a using class now substitutes the trait's QN with the
using class's QN in the return type. This fixes fluent chains like:
trait T { public function tap(): self { return $this; } }
class C { use T; public function ok(): int { return 1; } }
$c->tap()->ok(); // resolves through to C.ok now
Implementation:
- flatten_trait_into_class scans the source method's signature for
NAMED(trait_qn) in the return type and rewrites it to
NAMED(using_class_qn) when registering the flattened method.
- php_lsp_collect_class_fields now does TWO passes — pass 0 for
trait/interface declarators (so their declared-return-types get
re-parsed before any consumer), pass 1 for class/enum (which may
`use Trait;` and need the trait's signatures already up-to-date).
Total tests: 3013 / 0 failed (200 php_lsp).
|
||
|
Martin Vogel
|
20dd651a2e |
feat(php-lsp): @return $this/static, if-condition call walk, +50 tests
Phase 4aa, 4ab partial, 4y/z best-effort, 5e: more stdlib + tests.
- @return $this / @return static / @return self in PHPDoc all bind to
the enclosing class type, enabling fluent-builder chains.
- Declared return-type re-parse uses ctx->enclosing_class_qn so the
`: self` shorthand on a method correctly returns NAMED(class).
- php_resolve_class_name now checks self/static/parent BEFORE the
builtin-name guard, fixing a long-standing miss where these
pseudo-types were rejected as builtins.
- if-condition walk: calls inside `if ($it->valid())` are now
resolved (they previously only got narrowing analysis).
- Removed duplicate Generator type registration in stdlib data.
Stdlib expansion (Phase 5e):
- Symfony Cache (CacheInterface, ItemInterface) + PSR Cache pool/item
- Symfony EventDispatcher (EventDispatcherInterface, Event)
- Symfony Mailer (MailerInterface) + Mime\\Email fluent builder
- Symfony Validator (ValidatorInterface, ConstraintViolationListInterface)
- Laravel Http\\Request, JsonResponse, Response with method chains
- Laravel Auth\\Guard, Authenticatable contracts
- Laravel Session, View
- ReactPHP / GuzzleHttp Promise interfaces
- Monolog Logger (extends PSR LoggerInterface)
- Reflection API (ReflectionClass/Method/Property/Function)
Tests: 150 -> 200 covering @return $this fluent chains, Symfony
Cache/Mailer/Validator, Laravel Request/Auth/View, Promise chains,
Monolog, Reflection, multi-interface impl, deep parent chains, foreach
key-value chains, and a realistic Laravel UserController integration.
Total tests: 2963 -> 3013 / 0 failed.
|
||
|
Martin Vogel
|
39f41c3d47 |
feat(py_lsp): close all stress gaps — lambda, dict-dispatch, async, callable, next, kwargs
Round 10. Closes the last documented gaps from Round 9 plus 21 new
real-world stress tests covering SQLAlchemy / Pydantic / dataclass /
TypedDict / Protocol / generator / async patterns.
Lambda call-site inference:
- PyLSPContext gains a per-context lambda registry. `fn = lambda x:
body` records the lambda's AST node bound to fn. At call site
fn(arg), we push a sub-scope, bind lambda params from arg types,
and walk the body. Edges emitted under a synthetic <lambda> child
of the enclosing function so traces attribute correctly.
Dict-literal dispatch table:
- `funcs = {"a": foo, "b": bar}` records per-literal-key target QNs
in CBMDictLiteralEntry[]. funcs["a"]() emits an edge directly to
foo's QN with strategy=lsp_dict_dispatch.
Async with / async for:
- `async with X() as f` looks up __aenter__ before __enter__.
__aenter__'s declared return type wins; the manager type is the
fallback (async __aenter__s typically return self).
- `async for x in iter` falls back to __anext__'s return type when
the iterable's TEMPLATE form isn't recognized as AsyncIterator /
AsyncGenerator.
Callable / next / iter:
- `f()` where f resolves to CBM_TYPE_CALLABLE returns the callable's
return type. TEMPLATE("Callable", [..., R]) returns R.
- factory()() / (lambda)() etc — when the function expression is
itself a call / parenthesized / lambda, evaluate it and use its
return type.
- `next(it)` returns py_iterable_element_type(type-of-it).
- `iter(x)` is identity.
*args / **kwargs typed-parameter wrapping:
- `def f(*args: T)` -> args is TEMPLATE("tuple", [T]).
- `def f(**kwargs: T)` -> kwargs is TEMPLATE("dict", [str, T]).
- Detected by walking typed_parameter > list_splat_pattern /
dictionary_splat_pattern wrappers in py_bind_parameters.
Tuple / list / set / dict literal expressions:
- py_eval_expr_type now builds structured TUPLE / TEMPLATE types
from element types of the literal. `(Foo(), Bar())` -> TUPLE([Foo,
Bar]); `[Foo()]` -> TEMPLATE("list", [Foo]); `{"k": Foo()}` ->
TEMPLATE("dict", [str, Foo]). Without this, unpacking lost element
precision.
Match sequence pattern:
- list_pattern wraps each element in another case_pattern; capture
appears as dotted_name -> identifier; splat as splat_pattern.
Element-binder walks both wrappers and binds capture/splat to the
subject's iterable element type / list[elem].
Stress test suite expanded from 22 to 43 patterns. All 43 hard-
asserted as PASS:
- typing helpers: NamedTuple, TypedDict (incl. total=False), Protocol
(nominal + structural), ABC + abstractmethod, Mapped[T] (SQLAlchemy
2.0), Pydantic-style BaseModel
- control flow: with-as via __enter__, except-as, post-early-return
narrowing, async with via __aenter__, async for via __anext__
- containers: tuple unpack from function return + literal,
dict.items() comprehension k/v unpack, list slice, dict.get
default, recursive self-referencing types via forward refs
- decorators: factory + functools.wraps, classmethod chained to
method, property setter
- inheritance: Self in chain, diamond MRO with C3, multi-class match
- functional: lambda call-site inference, function-as-dict-value
dispatch, callable return-value, next(iter), generator delegation
(yield from), iterator next, double-star **kwargs, isinstance |
branches, module-level const, dataclass + default_factory
Bench: 178 LOC, 52 calls, **52 resolved (100%)**, 11.25 ms (ASan +
UBSan). Total 2941 / 0. Zero remaining KNOWN GAPs in the stress
suite — every probed pattern resolves.
|
||
|
Loay chlih
|
323c68e4d1 |
fix(extract_defs): emit all INHERITS edges for Java extends+implements (#279)
Two bugs in extract_base_classes() caused most INHERITS edges to be missing for Java (and any other language using similar AST field names): Bug 1 — Early return: the field loop returned on the first matching field, so class Foo extends Bar implements Baz produced only one INHERITS edge instead of two — the loop never processed 'interfaces' after finding 'superclass'. Bug 2 — Keyword text: cbm_node_text on the field node returned 'extends Bar' / 'implements Baz, Qux' literally instead of the bare type names. So even the one edge that did get emitted had a malformed target. Fix: new collect_bases_from_field() walks into child AST nodes and extracts type_identifier / generic_type / qualified_name / scoped_type_identifier / user_type text directly. Handles type_list / interface_type_list children for the multi-interface case. Strips generics at '<'. Falls back to raw node text for languages where the field IS the type name. The outer field loop now collects from all matching fields before returning, and the result is arena-allocated with a NULL terminator like the surrounding extract_csharp_base_list pattern. Bounds-safe: writes capped at MAX_BASES_MINUS_1 (15) with per-iteration count < out_cap checks; result fits in MAX_BASES (16) including NULL terminator. Real-world impact: 3 → 116 INHERITS edges on the Modelio Java codebase (253 files). |
||
|
Martin Vogel
|
4ccd08c6de |
feat(py_lsp): TypedDict subscript + early-return narrowing + match-sequence
Round 9 closes 2 of the 3 remaining stress gaps from Round 8: TypedDict literal-key subscript: - d["foo"] where d: TD with class-body annotation `foo: Foo` returns the field's type. Routes through the existing class-body-annotation field registry (Round 4): TypedDict subclasses already register their fields the same way. The subscript branch in py_eval_expr_type now detects NAMED container + literal-string key and looks up the field via py_lookup_field. Strips both single and double quotes. Early-return narrowing (post-block flow): - if not isinstance(x, T): return / raise / break / continue — applies the *positive* narrow to the enclosing scope after the if-block when the consequence terminates. - if x is None: return — narrows x to non-None afterwards. - py_block_terminates detects return/raise/break/continue as the last statement of the body block. Subsequent code in the function sees the narrowed type. Match sequence pattern (tentative): - list_pattern / tuple_pattern handler added: bind capture-pattern elements to the subject's iterable element type; bind splat-pattern rest to list[elem]. Also walks more wrapper levels (case_pattern -> pattern -> _simple_pattern). Test still flags as KNOWN GAP because tree-sitter's pattern-wrapper structure for the bracket-form `[a, b]` in match/case differs in some grammar versions; documented as remaining gap pending grammar inspection. Stress test suite: 22/22 PASS. Hard-asserted patterns: namedtuple, typeddict, protocol, abc, with-as, except-as, isinstance-early-return, tuple-unpack, dict.items, slice, decorator-factory, property-setter, Self chain, diamond MRO, recursive types, nested closure, long method chains, generator delegation, async-gen for. Documented gaps: function-as-dict-value, match sequence pattern, lambda inference. Bench unchanged at 100% / 11.02 ms; total 2920 / 0. |
||
|
Martin Vogel
|
f35745c79f |
feat(py_lsp): with-as, except-as, tuple unpack, slice, container TypeVar special-cases — bench 100%
Round 8. Stress test suite (test_py_lsp_stress.c) added with 22 advanced
patterns; bench fixture grew to 178 lines covering the full surface.
with-as binding via __enter__:
- Look up __enter__ on the context manager's NAMED type. Use its
return type when known; fall back to the manager type itself
(matches the practical case where __enter__ returns self). Handles
tree-sitter Python's wrapping of `X as y` in an as_pattern node
inside with_item, plus the flat field-only shape.
except-as binding:
- Bind exception alias to the caught type. Resolves through
except_clause's as_pattern wrapper and the as_pattern_target
identifier wrapper.
Tuple unpacking:
- Both `a, b = f()` and `for k, v in d.items()` destructure the RHS
element-by-element when the rhs is TUPLE or TEMPLATE("tuple", ...).
*rest binds the trailing slice as list[elem]. Shared helper
py_bind_for_target used by for-statement, comprehension for_in_clause,
and assignment LHS.
Slice subscript:
- `lst[1:3]` returns the same TEMPLATE container — list[T][1:3] is
list[T], dict[K, V][1:3] is dict[K, V] (rare but consistent).
Container TypeVar substitution:
- Without a constraint solver we can't generally substitute K, V into
registered method return types. Hand-rolled the most-impactful
cases: dict.items/keys/values/get/pop/copy, list.copy/__iter__/pop,
set.copy. These return properly parameterized ItemsView[K, V] /
KeysView[K] / ValuesView[V] / Optional[V] / etc. so subsequent
iteration and unpacking yield typed elements.
ItemsView / KeysView / ValuesView / dict_items / dict_keys / dict_values
added to py_iterable_element_type. ItemsView[K, V] iterates as
tuple[K, V], unpacked correctly by py_bind_for_target.
Bench: 178 LOC, 52 calls, **52 resolved (100%)**, 11.59 ms (ASan + UBSan).
22 new stress tests, 8 of 10 documented gaps closed in this round.
Remaining gaps documented inline (typeddict_subscript, post-early-return
narrowing, function-as-dict-value, match sequence pattern, lambda
inference). All 2898 prior tests stay green; total 2920 / 0.
|
||
|
vinay-veerappa
|
048f1b46c9 |
feat(lang): add Pine Script support via vendored kvarenzn/tree-sitter-pine (#273)
Adds Pine Script (TradingView's indicator/strategy DSL) as a first-class language using the project's existing language-add pipeline: - scripts/new-languages.json: register pine (display "PineScript", ts_func tree_sitter_pine, repo kvarenzn/tree-sitter-pine, extension .pine, has_scanner=true, module_root source_file) - internal/cbm/vendored/grammars/pine/: vendored from https://github.com/kvarenzn/tree-sitter-pine via scripts/vendor-grammar.sh — parser.c, scanner.c, tree_sitter/ headers. License is ISC (declared in upstream package.json; written into vendored LICENSE for clarity since upstream has no LICENSE file) - internal/cbm/grammar_pine.c: 4-line wrapper generated by scripts/generate-lang-code.py wrappers - internal/cbm/cbm.h: CBM_LANG_PINE before CBM_LANG_COUNT - internal/cbm/lang_specs.c: 7 node-type arrays (func / class / module / call / var / branch / assign) cross-checked against kvarenzn's grammar.js, plus the spec table entry - src/discover/language.c: .pine -> CBM_LANG_PINE extension dispatch and "PineScript" name scripts/audit-grammar-security.sh on the vendored grammar: no dangerous includes (only string/stdint/stdbool/stdlib), no system / popen / exec / socket / mmap / dlopen calls, no constructor/destructor attributes, no inline asm. The 148-line scanner.c only handles INDENT/DEDENT/NEWLINE tokens via the standard tree-sitter lexer API. Distilled from #273 rather than vendored verbatim because the original PR's parser.c was a plain 46k-line drop without provenance, no LICENSE attribution, and bundled an unrelated Makefile change (scripts/embed-frontend.sh -> sh scripts/embed-frontend.sh) that didn't belong to a Pine Script feature. The node-type choices in the lang spec match the original PR's selections, all of which were verified to be real grammar.js rules. Full suite: 2840 passed, 0 failed under ASan + UBSan. Closes #273. Co-authored-by: vinay-veerappa <vinay.veerappa@gmail.com> |
||
|
Martin Vogel
|
090fdb8935 |
feat(php-lsp): cover Phases 4t/4v/5d + 25 more tests, 100→150 tests
- Generic substitution end-to-end (T<->concrete-arg) for any class with
@template + @return T, including ancestor walks via belt-and-suspenders
AST-extracted base_clause / class_interface_clause.
- Block-form `namespace App { class Foo {} }` — robust child traversal
through declaration_list / compound_statement variants; also collects
`use` clauses from inside the block.
- Declared method return types re-parsed via php_parse_type_node so
`function f(): \Generator<int, User>` becomes a proper TEMPLATE.
- Stdlib expansion: Symfony Console (SymfonyStyle, In/OutputInterface),
Doctrine ORM (EntityManagerInterface, QueryBuilder, Query, Repository),
Guzzle (Client, ClientInterface), Twig (Environment, TemplateWrapper),
Eloquent Model magic-static methods (where/whereIn/orderBy/...).
- Pipeline bridge unchanged; resolution all happens at LSP level.
Test coverage: 125 -> 150 tests; total 2938 -> 2963 / 0 failed.
|
||
|
Martin Vogel
|
a3a64fc009 |
feat(php-lsp): generic substitution + 25 more features toward LSP parity
Phase 4m–w in one push, plus 25 new tests (100 → 125, all green).
New capabilities:
- Generic @template substitution:
@template T on a class records T as a type-param;
@return T on a method gets parsed as NAMED("T");
when receiver is TEMPLATE("Container", [User]), method-call return
types are substituted T → User so chains like
`$container->get()->name()` resolve through to the element type.
- Closure use(...) capture: typed variables in `use ($a, $b)` clauses
of `function () use (...) {}` are inherited from the parent scope.
- Multi-conjunction narrowing: `if ($x instanceof A && $y instanceof B)`
narrows BOTH $x and $y in the if-body. Up to 8 simultaneous narrowings.
- is_a($x, Foo::class) narrowing — both the ::class and string forms.
- Subscript inference: $arr[$k] on `array<T>` returns T;
on `array<K, V>` returns V;
on a NAMED ArrayAccess class consults registered offsetGet.
- Enum case access: Suit::Hearts evaluates to NAMED(Suit) so
Suit::Hearts->label() resolves to Suit::label.
- @extends ParentClass<X>: appends ParentClass to embedded_types so
the inheritance walk picks up methods from generic ancestors.
- PHPDoc @return on methods with no declared return type updates
the registry signature.
- foreach key=>value: handles tree-sitter-php's `pair` node so
`foreach ($users as $idx => $user)` binds $user to the element type.
- Class-level @template-extends, @template-implements parsed without
crashing.
- Don't override TEMPLATE/NAMED bindings with `null` literal —
preserves @var-typed variables across `$x = null` initialization.
Stdlib expansion (Phase 5d):
- Symfony Console: SymfonyStyle, InputInterface, OutputInterface
- Doctrine ORM: EntityManagerInterface, QueryBuilder chain, Query,
EntityRepository
- Guzzle HTTP: Client and ClientInterface (returning PSR-7 responses)
- Twig: Environment, TemplateWrapper
- Eloquent Model: forwarded static methods (where, whereIn, etc.)
so `User::where()->first()` chains resolve immediately.
Test coverage: 100 -> 125 tests; total 2913 -> 2938 / 0 failed.
|
||
|
Martin Vogel
|
dd3a816163 |
feat(py_lsp): builtin/template attribute dispatch + wrappers + dedup
Major parity push. Bench 79% -> 98% (51/52 calls resolved) on a 178-line
fixture exercising the full parity surface.
Annotation wrapping:
- Type-wrapper annotations now unwrap to the inner type:
ClassVar[T], Final[T], InitVar[T], ReadOnly[T], Required[T],
NotRequired[T], Annotated[T, ...] (drop metadata),
Mapped[T] / WriteOnlyMapped[T] / DynamicMapped[T] (SQLAlchemy 2.0),
Type[T] / type[T]. Same logic in both py_resolve_annotation
(runtime, scope-aware) and py_parse_type_text_qn (registry-build,
scope-free).
Builtin / template attribute dispatch:
- obj_type=BUILTIN("str"|"int"|"list"|"dict"|...) now resolves
attribute access via "builtins.<typename>" in the typeshed-derived
registry. obj.split(...) on str, list.append(), dict.get() etc all
emit edges with strategy=lsp_builtin_method.
- obj_type=TEMPLATE("dict"|"list"|...) routes the same way; receiver
is the unparameterized container's typeshed entry.
Bare-name builtin fallback:
- Bare-name calls (range, len, list, dict, str, int, print, etc.) fall
through to a "builtins" module symbol lookup, then to a builtins
type lookup for class constructors. Strategies: lsp_builtin /
lsp_builtin_constructor.
Emission deduplication:
- py_emit_resolved_call dedupes by (caller_qn, callee_qn). When two
resolution paths converge on the same edge (expression typing then
call emission, or attribute walk through wrappers), keep one entry
with the higher confidence. Previously the same call could emit
twice — the bench showed >100% ratios as evidence.
Bench fixture expanded from 136 to 178 lines:
- Adds @property access chains, Mapped[Annotated[int, ...]], Final
fields, generator iteration with method dispatch on element type,
multi-class match/case, dict comprehension over .values(), instance
attribute typing through super().__init__, classmethod chained with
Self return.
All 2898 prior tests stay green; bench at 98% in 11.79 ms (ASan +
UBSan).
|
||
|
Martin Vogel
|
fd81dffc8f |
feat(py_lsp): decorator flags + verifications for generators / dataclasses / properties
Round 6: CBMRegisteredFunc gains a `flags` bitfield (CBMFuncFlags) and a `decorator_qns` NULL-terminated list. py_register_def populates them from CBMDefinition.decorators by short-name match — so `@property`, `@functools.cache`, `@abc.abstractmethod`, `@typing.overload`, and `@final` all light up regardless of qualified vs bare reference. The flags aren't consumed by resolution yet (still on roadmap: emit distinct edge for @property attribute access vs method call); they are recorded so future overload selection / property modeling can use them without another schema change. The struct extension is at-tail and existing call sites that memset to zero stay correct; non-Python LSPs see flags=0 / decorator_qns=NULL and behave identically. Three new verification tests prove that generators, dataclasses, and @property already work end-to-end through the Round 4-5 infrastructure without further code changes: - Generator[Foo, ...] iterables in for-loop bind loop var to Foo via py_iterable_element_type's existing TEMPLATE handling. - @dataclass class Point: x: Foo (class-body annotated assignment) is picked up by Round 4's third-pass field registrar; p.x.method() resolves through the field type. - @property def thing(self) -> Foo: c.thing returns Foo because attribute access on NAMED routes through py_func_return_type which yields the getter's annotated return. All 2895 prior tests stay green; 3 new tests pass. Total 2898 / 0. Bench unchanged at 95% (the small bench fixture doesn't exercise the new patterns; will expand in Round 7). |
||
|
Martin Vogel
|
ce54e817b7 |
feat(py_lsp): subscript + super().__init__() + operator dunders
Round 5 parity push:
Subscript value typing — py_eval_expr_type's subscript branch resolves
indexing against the container's TEMPLATE type:
- TEMPLATE("dict"|"Mapping"|"defaultdict"|"OrderedDict"|...)[K] -> V
- TEMPLATE("list"|"set"|"frozenset"|"Sequence"|"Iterable"|...)[i] -> elem
- TEMPLATE("tuple", [A, B, ...])[literal_int] -> args[idx]
- Non-literal index returns UNION of all element types.
This unlocks the `self.cache[k].method()` pattern that's everywhere in
real Python. Also fixes `items[0].method()` for typed lists.
super().__init__(...) constructor edge — super().<attr> already worked
for regular methods; __init__ is special-cased because parent classes
typically don't register __init__ with a useful return type. Strategy
"lsp_super_init" with confidence 0.85.
Operator dunders — a + b looks up __add__ on a's type and uses its
return type. Same for *, -, /, //, %, **, <<, >>, &, |, ^, @. Falls
back to left-operand type when registry has no info (correct for the
common int+int / str+str / list+list cases).
3 new tests; all 2892 prior tests stay green. Bench unchanged at 95%
(fixture doesn't exercise these patterns directly — Round 6 will
expand it).
|
||
|
Martin Vogel
|
4b993899a0 |
feat(py_lsp): instance attribute typing — 79% -> 95% resolution
Round 4 / final parity push. Tracks self.x = expr from __init__ and
class-body annotated assignments (`x: Foo`) as registered fields on
the enclosing class type.
py_register_instance_field mutates the registry's CBMRegisteredType
field_names / field_types arrays in place. Re-registering the same
name overwrites; first-seen wins for capture order. Arena-allocated,
not a hot path.
py_lookup_field walks the alias / embedded chain with the same
CBM_LSP_MAX_LOOKUP_DEPTH cap as method lookup, so inherited fields
resolve through MRO without stack-overflow risk on pathological
hierarchies.
Field-fallback wired into py_eval_expr_type's attribute branch:
when method lookup misses, try field lookup. Returns the field's
type so downstream attribute / call resolution chains naturally.
Same fallback applied to UNION (single-match heuristic).
py_process_class now does three passes:
1. Class-body annotated assignments (`x: int = ...`) → fields.
2. __init__ / __post_init__ → self.x = expr fields.
3. All other methods.
This guarantees fields are registered before methods that reference
them, regardless of source order.
Annotated assignment binding fix: py_resolve_annotation is now called
on assignment annotations too, so `self.cache: dict[str, str] = {}`
binds with the proper TEMPLATE type instead of NAMED("dict[str, str]").
Bench result on the 136-line fixture: **95%** (36/38 calls resolved,
8.06 ms). Up from 79%. The 5% gap covers cases that need a full
constraint solver — see Final Stopping Point doc.
2 new tests; all 2890 prior tests stay green. Total 2892 / 0.
|
||
|
Martin Vogel
|
e7dcadc3dd |
feat(py_lsp): match/case narrowing, await passthrough, expanded bench
Round 3 parity push: - match/case (PEP 634): when the case pattern is class_pattern (wrapped in a case_pattern), narrow the match subject identifier to NAMED(class) inside the case body. Push/pop scope per case. Capture sub-patterns bind to UNKNOWN (field-type extraction is a v1.1 task — would require field type info in the registry). - await expr: identity passthrough. async def f() -> T registers with return T, so `await f()` returning T is the resolved-call shape we want without modeling Coroutine[Any, Any, T] explicitly. - Expanded test_py_lsp_bench.c fixture from 65 lines to 136 lines exercising every parity feature: dotted-module walks, dataclass + Self chaining, multi-inheritance with super(), Optional/Union annotations + narrowing (isinstance / is not None / walrus), generic list[T] / dict[K, V] / defaultdict comprehensions, classmethod / staticmethod, match/case dispatch, async/await, common stdlib (logging.getLogger, json.loads, pathlib.Path, functools.lru_cache). Bench result on the new fixture: 38 calls, 30 resolved, 79%, 7.50 ms. Remaining 21% includes patterns where intermediate types lose precision (e.g. super() constructor, dict subscript), instance attribute typing (self.x where x isn't annotated at class scope), and decorator return-type substitution (lru_cache). All 2888 prior tests stay green; 2 new tests + bench expansion. Total 2890 / 0. |
||
|
Martin Vogel
|
fa45e7defc |
feat(py_lsp): narrowing, walrus, comprehension element typing
Round 2/3 parity push. Three connected gap fixes that share an annotation- parser and a generic-aware iterable model. Annotation parser upgrade: - py_resolve_annotation now turns subscripted forms into structured types rather than stripping. `Optional[Foo]` -> UNION(NAMED(Foo), None). `Union[A, B]` -> UNION. `list[Foo]` / `dict[K, V]` / `Iterable[T]` -> TEMPLATE with arg list. `X | Y` (PEP 604) handled at depth zero. - New py_parse_type_text_qn variant: same parser without scope lookup, used by py_register_def at registry-build time. Bare class names are qualified to <module_qn>.<name> so registry lookups match. - py_register_def prefers d->return_type (full subscripted text) over d->return_types[0] when brackets are present — extract_defs's array form strips the subscript, losing element types like Optional[Foo]. Narrowing: - if isinstance(x, T): bind x as NAMED(T) inside the consequence body (block-scoped via push/pop). - if x is not None: strip None from x's UNION inside the consequence; py_match_is_none recognizes "is not" as the single tree-sitter token it actually is, walks through parens / walrus to find the underlying identifier. - Else branch unchanged — narrow's negation (not-T, including-None) is not modeled in v1. Walrus + truthy: - if (x := compute()) is not None: walrus binds x in the enclosing scope (PEP 572) so the body sees x. py_bind_walrus_in walks the cond AST looking for named_expression / assignment_expression nodes. Comprehension + for-loop element typing: - New py_iterable_element_type maps TEMPLATE(list/set/Iterable/...) to arg[0]; TEMPLATE(dict) to key arg; TEMPLATE(tuple, ...) to UNION of args. - Comprehensions push their own scope and bind for_in_clause loop vars to the iterable's element type before walking body / filter clauses. for_statement does the same. UNION attribute access fallback (used when narrowing can't run): - obj_type=UNION & exactly one non-None NAMED member has the attr: emit edge with strategy=lsp_method_union, confidence 0.85. 7 new test_py_lsp.c cases. All 2880 prior tests stay green; 8 new tests land. Bench unchanged (small fixture, no narrowing exercised there). |
||
|
Martin Vogel
|
da9f20d55a |
feat(php-lsp): generic templates, negative narrowing, expanded stdlib
Phase 4g+ / 4l / 5b additions toward LSP-server type-resolution parity:
- Generic template parsing in PHPDoc: parse `Collection<User>`,
`array<int, User>`, `iterable<T>`, `list<T>`, etc. Bind variables
to CBM_TYPE_TEMPLATE; method dispatch on a TEMPLATE receiver
resolves against the template's base class.
- Foreach element-type propagation: `foreach ($users as $u)` where
$users: array<User> binds $u to User. Two-arg array<K,V> takes
V. Falls back to Iterator::current() return type for NAMED types
that implement the iterator interface.
- Negative narrowing: `if (!($x instanceof Foo)) return;` (or throw)
narrows $x to Foo for the rest of the enclosing scope.
- Foreach iterable can be an arbitrary expression (variable_name,
member call, function call) — fixed earlier-too-narrow filter.
- PHPDoc @var/@param accept generic syntax: type token now spans
angle brackets so `array<User> $users` parses correctly.
- Stdlib expansion (Laravel + Symfony): full Eloquent Builder
method chain (where/orderBy/limit/get/first/...), Eloquent Model
static factories, Illuminate\Support\Collection chains
(map/filter/sort/...), Symfony HttpFoundation Request/Response,
Carbon date methods, PSR-7 with*() builders.
Test suite: 81 -> 100 tests, all passing. Total: 2913 / 0.
|
||
|
Martin Vogel
|
82d0935fbd |
feat(php-lsp): Phase 4 — depth toward LSP-server parity
Adds the resolver passes that close most of the gap to a real PHP language
server's type inference, while staying in-process and PHP-runtime-free.
New capabilities:
- Type narrowing: instanceof, is_string/int/float/bool/array/object/...
is_callable, is_iterable, is_numeric. Active inside if/elseif bodies
and after assert(...) (sequential narrowing)
- Property type tracking: typed property declarations, constructor
property promotion (public X $foo), and constructor-body inference
($this->bar = $bar where $bar is typed)
- Late-static-binding chain depth: full ancestor walk with cycle
detection for $obj->method() and Class::method() resolution
- Trait flattening: methods from `use Trait;` inside a class body are
re-registered with the using class as receiver; basic `as` aliasing
is honored
- PHPDoc class-level tags: @property TYPE $name binds a virtual field
on the class; @method TYPE name() registers a virtual method
(Eloquent-style dynamic dispatch)
- Match expression result type: union over arm result expressions,
first concrete type wins
- Ternary result type: prefers the consequent, falls back to alternative
- Variadic parameter binding: $args bound to `array`
- `use Vendor\\Helper as H;` two-name form correctly maps H -> Vendor.Helper
- Static-call unindexed fallback: when class is resolved from `use`
but methods unindexed, emit php_static_unindexed for the bridge
- Stdlib expansion: full Eloquent Builder + Model + Collection method
chains, Symfony HttpFoundation Request/Response, Carbon date/time,
PSR-7 with*() builders
Test suite expanded from 16 to 81 tests (2829 -> 2893 total).
|
||
|
Martin Vogel
|
2e042dab0b |
feat(py_lsp): Round 1 parity push (parent-modules, cast, Self, fwd-refs)
Parity push #1 — six gap fixes on top of the 11-phase plan: - Parent-module bindings: `import a.b.c` now binds `a`, `a.b`, `a.b.c` via py_bind_dotted_prefixes. Walks the dotted prefix chain on every import, regardless of import_is_from_style. Plus an attribute-access path on a MODULE that detects submodules: when the registry has any function whose qn starts with `<mod>.<attr>.`, evaluating `mod.attr` yields MODULE("<mod>.<attr>"). This makes `os.path.join(...)` and similar resolve correctly. - typing.cast(T, x) returns NAMED(T) (re-evaluating T as an annotation, so generic subscripts and forward references both resolve). Detects bare `cast(...)` and qualified `typing.cast(...)` forms. - typing.assert_type(x, T) is a no-op at runtime; we type the result as type-of(x). - Forward references as quoted strings: `def f(x: "Foo")` strips quotes and re-resolves the inner annotation. Also handles double-quoted and single-quoted forms. - Self return type substitution: methods declared returning `Self` / `typing.Self` / `typing_extensions.Self` substitute to NAMED(receiver) at call resolution time. Enables fluent / builder pattern chaining to keep resolving methods after each step. - Generic subscript stripping: annotations like `list[Foo]` strip to `list` for v1; `Optional[Foo]` likewise. Container element-type substitution still deferred (Phase 8+). Generator additions: - Allowlist now includes urllib, http, concurrent (top stdlib usage). - `from X import *` re-export following at generation time. Modules like os.path (a star-import shim of posixpath / ntpath / genericpath) now have all forwarded definitions registered under their own QN. Iterates to a fixed point with an 8-step ceiling. typeshed tree expansion is bounded — only modules transitively reachable from the allowlist are pulled in. Stats: 137 modules, 904 classes (2797 methods), 886 free functions (was 114 / 753 / 794). Generated file grows from 20K to ~22K lines. 6 new test_py_lsp.c cases. All 2874 prior tests stay green. |
||
|
Martin Vogel
|
fa6bfd125f |
feat(py_lsp): stdlib registry from typeshed + generator
Phase 10 of Python LSP integration. Adds:
- scripts/gen-py-stdlib.py: walks a typeshed/stdlib checkout, parses
each .pyi via Python's stdlib `ast` module, and emits one C source
file populating cbm_python_stdlib_register. v1 simplifications:
- overload stacks collapse to first signature
- ParamSpec / TypeVarTuple / Concatenate skipped
- version guards (if sys.version_info >= ...) flattened to union
of branches
- per-symbol min/max version guards not yet emitted (v1.1 follow-up)
Module allowlist matches PYTHON_LSP_PLAN.md Phase 10 list — top
stdlib modules by usage in indexed Python projects, skipping
tkinter / turtle / curses / xml / email for size.
- internal/cbm/lsp/generated/python_stdlib_data.c (auto-generated,
20,021 lines): 114 modules, 753 classes (2,385 methods), 794 free
functions registered. Source pinned to typeshed commit
a7912d521e16ff63caf7a8b64b9072542be36777 (recorded in header).
Compiles cleanly under ASan + UBSan.
- lsp_all.c includes the generated file. The CBM_PYTHON_STDLIB_GENERATED
macro disables py_lsp.c's no-op stub so the real registration runs.
4 new test_py_lsp.c cases verify resolution against the registry:
os.getcwd, collections.defaultdict constructor, pathlib.Path.exists
method via typed parameter, logging.getLogger.
Known v1 limitation captured in test comments: `import os.path` only
binds the leaf `path` in scope, so `os.path.join` style chained
attribute access on the parent module name doesn't yet resolve.
Phase 10.5 will stamp parent-module bindings.
All 2869 prior tests stay green; 4 new stdlib tests pass.
|
||
|
Martin Vogel
|
f4f9ed47f6 |
feat(py_lsp): cross-file resolution + batch entry point
Phase 9 of Python LSP integration. Implements cbm_run_py_lsp_cross and cbm_batch_py_lsp_cross, mirroring the Go LSP pattern: - cbm_run_py_lsp_cross builds a per-call registry from caller-supplied CBMLSPDef[] entries (covering both file-local and cross-file definitions), then re-parses the source via tree_sitter_python (or uses cached_tree when supplied) and runs the standard py_lsp_process_file pipeline. - py_register_lsp_defs translates CBMLSPDef shapes into CBMRegisteredType / CBMRegisteredFunc entries. Python-specific Protocol label is recognized as an interface-like structural type. Pipe-separated embedded_types and method_names_str are split into arena-allocated NULL-terminated arrays. Method receivers auto-create type entries when the type isn't otherwise registered, matching the Go convention. - cbm_batch_py_lsp_cross loops the per-file entry, isolating each file in its own arena and copying string fields into the caller's arena so they outlive the file's scratch space. 3 new test_py_lsp.c cases cover cross-file method dispatch, cross-file inheritance (Child(Base) calling self.shared from another module), and batch processing of two interconnected files. All 2866 prior tests stay green; 3 new cross-file tests pass. |
||
|
Martin Vogel
|
329d266ed3 |
feat(py_lsp): decorators, super() resolution, multi-inheritance
Phases 7-8 of Python LSP integration. Adds: - super().method() resolution: when the call's receiver is a super() proxy inside a class context, walk the enclosing class's embedded_types and resolve the attribute against the first base where it exists. Edge carries strategy="lsp_super" and confidence 0.88. - Multi-inheritance dispatch already worked for the current registry shape (embedded_types is a list, py_lookup_attribute iterates) — added an explicit test to lock in the behavior. Also fixes a Python class-body extraction gap upstream of py_lsp: the generic extract_class_methods walker skipped any class-body child whose node type wasn't a function_definition, which excluded everything wrapped in a decorated_definition (i.e. every @classmethod / @staticmethod / @property method). Patched to peek through decorated_definition for the inner function_definition before calling resolve_method_name. This is the root cause of the C.classmethod() / C.staticmethod() resolution miss and the fix is independent of language-specific decorator semantics — any language whose grammar uses decorated_definition benefits. 7 new test_py_lsp.c cases cover decorated functions, classmethod calls, staticmethod calls, dataclass constructor + method dispatch, super() to the right base, multi-inheritance attribute resolution on both bases, and PEP 695-style class names without subscript handling. All 2859 prior tests stay green; 7 new tests pass. Total: 2866 / 0. |
||
|
Martin Vogel
|
1c83c5df4b |
feat(py_lsp): scope binding, expression typing, method dispatch
Bundles Phases 4-6 of Python LSP integration. py_lsp.c gains a real binder + expression evaluator + call resolver mirroring go_lsp.c's shape: - py_register_def populates the registry from CBMFileResult.defs. Class/Type defs land as registered types; Method/Function defs as registered functions with receiver_type derived from the qualified name parent. Python's extract_defs records a class's bases as a single "(Base, Other)" string, so we split commas at depth 0, strip parens / generic subscripts / keyword args (metaclass=...) to produce usable embedded_types entries. - py_lsp_process_file walks function/class definitions, pushing scope per function and binding self/cls to the enclosing class type (after parameter binding so the receiver wins over an unannotated `self` parameter). - py_bind_parameters resolves type annotations via py_resolve_annotation: scope -> module-qualified registry lookup -> bare NAMED fallback. - py_eval_expr_type covers literals (int/float/str/bool/None plus list/ dict/set/tuple containers), names (via scope, then module symbol), attribute access on MODULE / NAMED, call expressions (constructors, module-local functions, attribute calls), comparison / boolean / ternary expressions, and parenthesized / binary fallthrough. - py_lookup_attribute walks the registry's embedded_types chain with the shared CBM_LSP_MAX_LOOKUP_DEPTH cap so MRO chains terminate cleanly. - py_emit_resolved_call writes (caller_qn, callee_qn, strategy, confidence) entries to result->resolved_calls. Strategies: lsp_direct, lsp_constructor, lsp_method, lsp_module_attr, lsp_module_attr_unresolved. 6 new test_py_lsp.c cases cover direct calls, simple method dispatch, self.method, constructor-then-method, single-inheritance dispatch, and the no-false-positive guarantee on UNKNOWN receivers. All 2853 previous tests stay green; 6 new py_lsp tests pass. |
||
|
Martin Vogel
|
d674b43e0d |
feat(php-lsp): add Light Semantic Pass for PHP
Mirrors go_lsp.c and c_lsp.c. In-process C type resolver, no PHP runtime,
no new dependencies. Uses already-vendored tree-sitter-php.
Coverage (Phase 1+2+3 from PLAN_PHP_LSP_INTEGRATION):
- namespace + use clause resolution (class / function / const)
- typed parameter binding (incl. arrow functions and closures)
- $this and self/parent/static dispatch
- method chaining via return-type propagation
- object_creation_expression typing for new X() chains
- PHPDoc minimal parser (@var, @param) for legacy untyped code
- magic methods: __call (php_method_dynamic), __callStatic (php_dynamic_unresolved)
- synthetic php_method_typed_unindexed for vendor receiver types,
used by the pipeline bridge to suppress name-fallback misroutes
- corpus-seeded stdlib (~200 lines): SPL containers/iterators, PSR
interfaces, DateTime, Throwable hierarchy, Closure, plus the top
Laravel framework facades + base types from the benchmark corpus
|