Three C# extractor gaps that left modern .NET classes partially invisible
in the graph:
1. lang_specs.c CSHARP entry had `empty_types` for field_node_types, so
walk_defs never descended into field_declaration / property_declaration
nodes. Add cs_field_types and wire it in. Without this, no C# fields
or properties were ever pushed as Field defs.
2. extract_class_fields assumed the field's "type" is on the immediate
child via ts_node_child_by_field_name. Tree-sitter-c_sharp wraps fields
as field_declaration > variable_declaration(type, variable_declarator),
one level deeper than Java/Go/Rust. Detect the nested shape by walking
named children for variable_declaration and resolve type + name from
inside it. Direct shape continues unchanged.
3. C# 12 primary constructors (`class Foo(IBar bar) : Base { ... }`)
declare implicit captured fields on the class line. Tree-sitter wraps
them inside the hidden _class_declaration_initializer node, so the
`parameters` field on class_declaration may not always resolve
directly; iterate top-level children for parameter_list as a fallback.
Emit each primary-ctor parameter as a Field def with parent_class set.
Effect: dependency-injected fields on .NET 8+/9+ controllers and any C#
auto-property (`public IFoo Foo { get; }`) are now reachable as Field
nodes in the graph.
Co-authored-by: sponger94 <45746997+sponger94@users.noreply.github.com>
Per-file LSP (run from cbm_extract_file) only sees one file's defs, so
callees whose receiver type comes from an imported module stayed
unresolved — even though Go/C/Py/TS already had cross-file entry points
(cbm_run_X_lsp_cross). Those entry points existed but no pipeline pass
ever invoked them. PHP didn't even have a cross-file entry point.
This change lands the missing pieces:
1. cbm_run_php_lsp_cross + cbm_batch_php_lsp_cross — PHP cross-file
resolver mirroring py_lsp_cross / ts_lsp_cross. Reuses PHPLSPContext,
php_lsp_init, php_lsp_add_use, php_lsp_process_file. Handles PHP-
specific labels (Class / Interface / Trait / Enum / Type) and uses
def_module_qn so cross-file return types qualify against the def's
own module rather than the importing module's.
2. pass_lsp_cross.c — new pipeline pass. For every cached file result,
builds a project-wide CBMLSPDef[] (CBMDefinition → CBMLSPDef
converter), per-file import map (from gbuf IMPORTS edges), then
dispatches to the matching cbm_run_X_lsp_cross. Output strings get
copied into the per-file arena (with O(N²) memory bug avoided by
running each file inside a fresh scratch arena that gets destroyed
after the LSP call — the LSP allocates its registry + stdlib +
project defs in scratch, not in the caller's persistent arena).
3. Wired into both pipelines:
- sequential: definitions → k8s → lsp_cross → calls → ...
- parallel: parallel_extract → registry_build → lsp_cross →
parallel_resolve → ...
4. Updated tests/test_parallel.c run_parallel helper to mirror the new
parallel ordering so existing cross-file Python/PHP tests now exercise
pass_lsp_cross.
Languages covered: Go, C/C++/CUDA, Python, TS/JS/JSX/TSX, PHP. PHP cross
file is new. The other four had functions but no pipeline integration
until this change.
Verified on a 1670-file FastAPI index: pass_lsp_cross processes 1050
Python files (525 non-LSP files skipped), collects 4070 project defs in
9.2s, and parallel_resolve subsequently reports lsp_overrides=568 — the
LSP-strategy attribution gain over per-file alone.
Per-LSP emit functions dedup by (caller_qn, callee_qn) within their own
run; pxc_append_results dedups again at append time so cross-file
entries duplicating per-file output get dropped instead of inflating
result->resolved_calls.
Resolved 5 conflicts where the php-lsp branch and main both added LSP
infrastructure but with different APIs:
- Makefile.cbm: kept both TEST_PHP_LSP_SRCS and main's py_lsp/scope/
type_rep/stack_overflow test source variables in ALL_TEST_SRCS.
- internal/cbm/cbm.c: kept both #include lsp/php_lsp.h and lsp/py_lsp.h;
the per-language dispatch already supports stacking PHP after PYTHON.
- internal/cbm/lsp_all.c: kept both lsp/php_lsp.c + php_stdlib_data.c
AND lsp/py_lsp.c + python_stdlib_data.c in the compilation unit.
- src/pipeline/pass_calls.c: took main's version entirely.
find_lsp_resolution + resolve_single_call(ctx, call, lsp_calls, ...)
is language-agnostic — it matches any CBMResolvedCall whose caller_qn
hits the textual call's enclosing function and whose callee_qn
short-name hits the textual callee. PHP LSP populates the same
result->resolved_calls array via cbm_run_php_lsp, so the unified
resolver handles PHP for free. Dropped the php-branch's
qn_short_name + lsp_override_resolution helpers as superseded.
Locked confidence floor at main's CBM_LSP_CONFIDENCE_FLOOR = 0.6f
(was 0.5f on the php branch).
- tests/test_main.c: kept all suite externs and RUN_SUITE invocations
for php_lsp + py_lsp/py_lsp_bench/py_lsp_stress/py_lsp_scale +
scope + type_rep.
Auto-merged: src/pipeline/pass_parallel.c.
Brings in 52 commits from main since the branch was forked at c768b61
(#207 watcher follow-up): #208 through #281, dependabot bumps, Pine
Script, AUR docs, INHERITS edges, get_architecture wiring, codeql
revert, the entire python-lsp subsystem.
Suite: 3249 passed, 0 failed (was 2971 on main; +278 new tests from
php-lsp matches the branch's claimed test count).
Phase 5f, 4ac, 4ad: closing the long-tail gaps toward LSP-server parity.
- @phpstan-type / @psalm-type / @phan-type aliases parsed from class
docblocks and applied during PHPDoc type resolution. References to
the alias name in @var/@param/@return are rewritten to the aliased
type. Aliases live on PHPLSPContext (per-file).
- @phpstan-import-type accepted without crash (cross-file alias
resolution deferred — same root cause as Composer PSR-4).
- @template-covariant T and @template-contravariant T now register T
as a proper type-param (previously skipped along with
@template-extends).
- Closure binding: \\Closure::bind($f, $obj) and $f->bindTo($obj),
when invoked with an inline closure literal as the first arg, walk
the closure body with $this rebound to the second arg's class.
Resolves the Laravel-macroable / VarDumper / encapsulation patterns.
- @return ($x is string ? int : bool) conditional return-type syntax
accepted without crash (parsed leftmost segment as fallback).
- Long PHPDoc blocks with mixed @param/@return/@throws/@internal
parsed without misattributing tags.
Stdlib unchanged.
Tests: 248 → 278 covering:
- phpstan-type alias basic / via @param / union / array-shape
- psalm-type alternate spelling
- phpstan-import-type (no-crash)
- Closure::bind / ::bindTo / static-bind on arrow fn
- conditional return / template-covariant / -contravariant (no-crash)
- class-string<T>, int<0,100>, literal-string (no-crash on phpstan
types we don't fully model)
- 8 realistic patterns: repository, event listener, console command,
form handler, mail sender, cached service, logger w/ context,
Carbon chains in methods, Eloquent model.
Total tests: 3061 → 3091 / 0 failed.
Audit for O(n^2) hot paths after Round 10. The dedup loop in
py_emit_resolved_call scanned ALL prior entries linearly, making
emission O(N) and total file work O(N^2). On a synthetic 2000-class
fixture with 8000 calls, that dominated time at 5083 ms.
Cap the scan window at the last 256 entries. Most duplicate
emissions come from the same expression being walked twice (resolver
+ emitter passes), so they're temporally adjacent — a 256-entry
window catches every common case while keeping the dedup O(1) per
emission.
Scale measurements (test_py_lsp_scale.c added):
| Classes | Calls | Resolved | Before | After |
|--------:|------:|---------:|-------:|------:|
| 100 | 400 | 300 | 55ms | 57ms |
| 500 | 2000 | 1500 | 333ms | 323ms |
| 2000 | 8000 | 6000 | 5083ms | 1804ms|
Ratio for 20x input: 91.8x -> 31.4x (linear would be 20x; quadratic
would be 400x). No remaining quadratic hot path in code added by
this branch.
Other hot paths surveyed:
- py_register_def -> cbm_registry_lookup_type: O(types) per def at
registration, O(D x types) for D defs. Pre-existing linear scan in
type_registry.c; hash-indexing it would be a cross-language
optimization (Go and C++ LSPs benefit too).
- py_lookup_attribute / py_lookup_field: O(types) per call site.
Same pre-existing linear scan. With ~900 stdlib types + per-file
defs, accounts for ~75µs/call at 2000-class scale.
- MODULE submodule prefix scan: O(funcs) per module-attribute access,
early-exits on first match. Bounded by registry size.
- lambda / dict-literal registries: linear, but bounded by per-file
N which is small.
Bench unchanged at 100% / 12ms (52/52 calls, 178 LOC).
When a trait method declares `: self` as its return type, flattening
that method into a using class now substitutes the trait's QN with the
using class's QN in the return type. This fixes fluent chains like:
trait T { public function tap(): self { return $this; } }
class C { use T; public function ok(): int { return 1; } }
$c->tap()->ok(); // resolves through to C.ok now
Implementation:
- flatten_trait_into_class scans the source method's signature for
NAMED(trait_qn) in the return type and rewrites it to
NAMED(using_class_qn) when registering the flattened method.
- php_lsp_collect_class_fields now does TWO passes — pass 0 for
trait/interface declarators (so their declared-return-types get
re-parsed before any consumer), pass 1 for class/enum (which may
`use Trait;` and need the trait's signatures already up-to-date).
Total tests: 3013 / 0 failed (200 php_lsp).
Round 10. Closes the last documented gaps from Round 9 plus 21 new
real-world stress tests covering SQLAlchemy / Pydantic / dataclass /
TypedDict / Protocol / generator / async patterns.
Lambda call-site inference:
- PyLSPContext gains a per-context lambda registry. `fn = lambda x:
body` records the lambda's AST node bound to fn. At call site
fn(arg), we push a sub-scope, bind lambda params from arg types,
and walk the body. Edges emitted under a synthetic <lambda> child
of the enclosing function so traces attribute correctly.
Dict-literal dispatch table:
- `funcs = {"a": foo, "b": bar}` records per-literal-key target QNs
in CBMDictLiteralEntry[]. funcs["a"]() emits an edge directly to
foo's QN with strategy=lsp_dict_dispatch.
Async with / async for:
- `async with X() as f` looks up __aenter__ before __enter__.
__aenter__'s declared return type wins; the manager type is the
fallback (async __aenter__s typically return self).
- `async for x in iter` falls back to __anext__'s return type when
the iterable's TEMPLATE form isn't recognized as AsyncIterator /
AsyncGenerator.
Callable / next / iter:
- `f()` where f resolves to CBM_TYPE_CALLABLE returns the callable's
return type. TEMPLATE("Callable", [..., R]) returns R.
- factory()() / (lambda)() etc — when the function expression is
itself a call / parenthesized / lambda, evaluate it and use its
return type.
- `next(it)` returns py_iterable_element_type(type-of-it).
- `iter(x)` is identity.
*args / **kwargs typed-parameter wrapping:
- `def f(*args: T)` -> args is TEMPLATE("tuple", [T]).
- `def f(**kwargs: T)` -> kwargs is TEMPLATE("dict", [str, T]).
- Detected by walking typed_parameter > list_splat_pattern /
dictionary_splat_pattern wrappers in py_bind_parameters.
Tuple / list / set / dict literal expressions:
- py_eval_expr_type now builds structured TUPLE / TEMPLATE types
from element types of the literal. `(Foo(), Bar())` -> TUPLE([Foo,
Bar]); `[Foo()]` -> TEMPLATE("list", [Foo]); `{"k": Foo()}` ->
TEMPLATE("dict", [str, Foo]). Without this, unpacking lost element
precision.
Match sequence pattern:
- list_pattern wraps each element in another case_pattern; capture
appears as dotted_name -> identifier; splat as splat_pattern.
Element-binder walks both wrappers and binds capture/splat to the
subject's iterable element type / list[elem].
Stress test suite expanded from 22 to 43 patterns. All 43 hard-
asserted as PASS:
- typing helpers: NamedTuple, TypedDict (incl. total=False), Protocol
(nominal + structural), ABC + abstractmethod, Mapped[T] (SQLAlchemy
2.0), Pydantic-style BaseModel
- control flow: with-as via __enter__, except-as, post-early-return
narrowing, async with via __aenter__, async for via __anext__
- containers: tuple unpack from function return + literal,
dict.items() comprehension k/v unpack, list slice, dict.get
default, recursive self-referencing types via forward refs
- decorators: factory + functools.wraps, classmethod chained to
method, property setter
- inheritance: Self in chain, diamond MRO with C3, multi-class match
- functional: lambda call-site inference, function-as-dict-value
dispatch, callable return-value, next(iter), generator delegation
(yield from), iterator next, double-star **kwargs, isinstance |
branches, module-level const, dataclass + default_factory
Bench: 178 LOC, 52 calls, **52 resolved (100%)**, 11.25 ms (ASan +
UBSan). Total 2941 / 0. Zero remaining KNOWN GAPs in the stress
suite — every probed pattern resolves.
Two bugs in extract_base_classes() caused most INHERITS edges to be missing for Java (and any other language using similar AST field names):
Bug 1 — Early return: the field loop returned on the first matching field, so class Foo extends Bar implements Baz produced only one INHERITS edge instead of two — the loop never processed 'interfaces' after finding 'superclass'.
Bug 2 — Keyword text: cbm_node_text on the field node returned 'extends Bar' / 'implements Baz, Qux' literally instead of the bare type names. So even the one edge that did get emitted had a malformed target.
Fix: new collect_bases_from_field() walks into child AST nodes and extracts type_identifier / generic_type / qualified_name / scoped_type_identifier / user_type text directly. Handles type_list / interface_type_list children for the multi-interface case. Strips generics at '<'. Falls back to raw node text for languages where the field IS the type name. The outer field loop now collects from all matching fields before returning, and the result is arena-allocated with a NULL terminator like the surrounding extract_csharp_base_list pattern.
Bounds-safe: writes capped at MAX_BASES_MINUS_1 (15) with per-iteration count < out_cap checks; result fits in MAX_BASES (16) including NULL terminator.
Real-world impact: 3 → 116 INHERITS edges on the Modelio Java codebase (253 files).
Round 9 closes 2 of the 3 remaining stress gaps from Round 8:
TypedDict literal-key subscript:
- d["foo"] where d: TD with class-body annotation `foo: Foo` returns
the field's type. Routes through the existing class-body-annotation
field registry (Round 4): TypedDict subclasses already register their
fields the same way. The subscript branch in py_eval_expr_type now
detects NAMED container + literal-string key and looks up the field
via py_lookup_field. Strips both single and double quotes.
Early-return narrowing (post-block flow):
- if not isinstance(x, T): return / raise / break / continue —
applies the *positive* narrow to the enclosing scope after the
if-block when the consequence terminates.
- if x is None: return — narrows x to non-None afterwards.
- py_block_terminates detects return/raise/break/continue as the last
statement of the body block. Subsequent code in the function sees
the narrowed type.
Match sequence pattern (tentative):
- list_pattern / tuple_pattern handler added: bind capture-pattern
elements to the subject's iterable element type; bind splat-pattern
rest to list[elem]. Also walks more wrapper levels (case_pattern ->
pattern -> _simple_pattern). Test still flags as KNOWN GAP because
tree-sitter's pattern-wrapper structure for the bracket-form `[a, b]`
in match/case differs in some grammar versions; documented as
remaining gap pending grammar inspection.
Stress test suite: 22/22 PASS. Hard-asserted patterns: namedtuple,
typeddict, protocol, abc, with-as, except-as, isinstance-early-return,
tuple-unpack, dict.items, slice, decorator-factory, property-setter,
Self chain, diamond MRO, recursive types, nested closure, long method
chains, generator delegation, async-gen for. Documented gaps:
function-as-dict-value, match sequence pattern, lambda inference.
Bench unchanged at 100% / 11.02 ms; total 2920 / 0.
Round 8. Stress test suite (test_py_lsp_stress.c) added with 22 advanced
patterns; bench fixture grew to 178 lines covering the full surface.
with-as binding via __enter__:
- Look up __enter__ on the context manager's NAMED type. Use its
return type when known; fall back to the manager type itself
(matches the practical case where __enter__ returns self). Handles
tree-sitter Python's wrapping of `X as y` in an as_pattern node
inside with_item, plus the flat field-only shape.
except-as binding:
- Bind exception alias to the caught type. Resolves through
except_clause's as_pattern wrapper and the as_pattern_target
identifier wrapper.
Tuple unpacking:
- Both `a, b = f()` and `for k, v in d.items()` destructure the RHS
element-by-element when the rhs is TUPLE or TEMPLATE("tuple", ...).
*rest binds the trailing slice as list[elem]. Shared helper
py_bind_for_target used by for-statement, comprehension for_in_clause,
and assignment LHS.
Slice subscript:
- `lst[1:3]` returns the same TEMPLATE container — list[T][1:3] is
list[T], dict[K, V][1:3] is dict[K, V] (rare but consistent).
Container TypeVar substitution:
- Without a constraint solver we can't generally substitute K, V into
registered method return types. Hand-rolled the most-impactful
cases: dict.items/keys/values/get/pop/copy, list.copy/__iter__/pop,
set.copy. These return properly parameterized ItemsView[K, V] /
KeysView[K] / ValuesView[V] / Optional[V] / etc. so subsequent
iteration and unpacking yield typed elements.
ItemsView / KeysView / ValuesView / dict_items / dict_keys / dict_values
added to py_iterable_element_type. ItemsView[K, V] iterates as
tuple[K, V], unpacked correctly by py_bind_for_target.
Bench: 178 LOC, 52 calls, **52 resolved (100%)**, 11.59 ms (ASan + UBSan).
22 new stress tests, 8 of 10 documented gaps closed in this round.
Remaining gaps documented inline (typeddict_subscript, post-early-return
narrowing, function-as-dict-value, match sequence pattern, lambda
inference). All 2898 prior tests stay green; total 2920 / 0.
Adds Pine Script (TradingView's indicator/strategy DSL) as a first-class
language using the project's existing language-add pipeline:
- scripts/new-languages.json: register pine (display "PineScript",
ts_func tree_sitter_pine, repo kvarenzn/tree-sitter-pine, extension
.pine, has_scanner=true, module_root source_file)
- internal/cbm/vendored/grammars/pine/: vendored from
https://github.com/kvarenzn/tree-sitter-pine via scripts/vendor-grammar.sh
— parser.c, scanner.c, tree_sitter/ headers. License is ISC (declared
in upstream package.json; written into vendored LICENSE for clarity
since upstream has no LICENSE file)
- internal/cbm/grammar_pine.c: 4-line wrapper generated by
scripts/generate-lang-code.py wrappers
- internal/cbm/cbm.h: CBM_LANG_PINE before CBM_LANG_COUNT
- internal/cbm/lang_specs.c: 7 node-type arrays (func / class /
module / call / var / branch / assign) cross-checked against
kvarenzn's grammar.js, plus the spec table entry
- src/discover/language.c: .pine -> CBM_LANG_PINE extension dispatch
and "PineScript" name
scripts/audit-grammar-security.sh on the vendored grammar: no
dangerous includes (only string/stdint/stdbool/stdlib), no system /
popen / exec / socket / mmap / dlopen calls, no constructor/destructor
attributes, no inline asm. The 148-line scanner.c only handles
INDENT/DEDENT/NEWLINE tokens via the standard tree-sitter lexer API.
Distilled from #273 rather than vendored verbatim because the
original PR's parser.c was a plain 46k-line drop without provenance,
no LICENSE attribution, and bundled an unrelated Makefile change
(scripts/embed-frontend.sh -> sh scripts/embed-frontend.sh) that
didn't belong to a Pine Script feature. The node-type choices in
the lang spec match the original PR's selections, all of which were
verified to be real grammar.js rules.
Full suite: 2840 passed, 0 failed under ASan + UBSan.
Closes#273.
Co-authored-by: vinay-veerappa <vinay.veerappa@gmail.com>
Phase 4m–w in one push, plus 25 new tests (100 → 125, all green).
New capabilities:
- Generic @template substitution:
@template T on a class records T as a type-param;
@return T on a method gets parsed as NAMED("T");
when receiver is TEMPLATE("Container", [User]), method-call return
types are substituted T → User so chains like
`$container->get()->name()` resolve through to the element type.
- Closure use(...) capture: typed variables in `use ($a, $b)` clauses
of `function () use (...) {}` are inherited from the parent scope.
- Multi-conjunction narrowing: `if ($x instanceof A && $y instanceof B)`
narrows BOTH $x and $y in the if-body. Up to 8 simultaneous narrowings.
- is_a($x, Foo::class) narrowing — both the ::class and string forms.
- Subscript inference: $arr[$k] on `array<T>` returns T;
on `array<K, V>` returns V;
on a NAMED ArrayAccess class consults registered offsetGet.
- Enum case access: Suit::Hearts evaluates to NAMED(Suit) so
Suit::Hearts->label() resolves to Suit::label.
- @extends ParentClass<X>: appends ParentClass to embedded_types so
the inheritance walk picks up methods from generic ancestors.
- PHPDoc @return on methods with no declared return type updates
the registry signature.
- foreach key=>value: handles tree-sitter-php's `pair` node so
`foreach ($users as $idx => $user)` binds $user to the element type.
- Class-level @template-extends, @template-implements parsed without
crashing.
- Don't override TEMPLATE/NAMED bindings with `null` literal —
preserves @var-typed variables across `$x = null` initialization.
Stdlib expansion (Phase 5d):
- Symfony Console: SymfonyStyle, InputInterface, OutputInterface
- Doctrine ORM: EntityManagerInterface, QueryBuilder chain, Query,
EntityRepository
- Guzzle HTTP: Client and ClientInterface (returning PSR-7 responses)
- Twig: Environment, TemplateWrapper
- Eloquent Model: forwarded static methods (where, whereIn, etc.)
so `User::where()->first()` chains resolve immediately.
Test coverage: 100 -> 125 tests; total 2913 -> 2938 / 0 failed.
Major parity push. Bench 79% -> 98% (51/52 calls resolved) on a 178-line
fixture exercising the full parity surface.
Annotation wrapping:
- Type-wrapper annotations now unwrap to the inner type:
ClassVar[T], Final[T], InitVar[T], ReadOnly[T], Required[T],
NotRequired[T], Annotated[T, ...] (drop metadata),
Mapped[T] / WriteOnlyMapped[T] / DynamicMapped[T] (SQLAlchemy 2.0),
Type[T] / type[T]. Same logic in both py_resolve_annotation
(runtime, scope-aware) and py_parse_type_text_qn (registry-build,
scope-free).
Builtin / template attribute dispatch:
- obj_type=BUILTIN("str"|"int"|"list"|"dict"|...) now resolves
attribute access via "builtins.<typename>" in the typeshed-derived
registry. obj.split(...) on str, list.append(), dict.get() etc all
emit edges with strategy=lsp_builtin_method.
- obj_type=TEMPLATE("dict"|"list"|...) routes the same way; receiver
is the unparameterized container's typeshed entry.
Bare-name builtin fallback:
- Bare-name calls (range, len, list, dict, str, int, print, etc.) fall
through to a "builtins" module symbol lookup, then to a builtins
type lookup for class constructors. Strategies: lsp_builtin /
lsp_builtin_constructor.
Emission deduplication:
- py_emit_resolved_call dedupes by (caller_qn, callee_qn). When two
resolution paths converge on the same edge (expression typing then
call emission, or attribute walk through wrappers), keep one entry
with the higher confidence. Previously the same call could emit
twice — the bench showed >100% ratios as evidence.
Bench fixture expanded from 136 to 178 lines:
- Adds @property access chains, Mapped[Annotated[int, ...]], Final
fields, generator iteration with method dispatch on element type,
multi-class match/case, dict comprehension over .values(), instance
attribute typing through super().__init__, classmethod chained with
Self return.
All 2898 prior tests stay green; bench at 98% in 11.79 ms (ASan +
UBSan).
Round 6:
CBMRegisteredFunc gains a `flags` bitfield (CBMFuncFlags) and a
`decorator_qns` NULL-terminated list. py_register_def populates them
from CBMDefinition.decorators by short-name match — so `@property`,
`@functools.cache`, `@abc.abstractmethod`, `@typing.overload`, and
`@final` all light up regardless of qualified vs bare reference. The
flags aren't consumed by resolution yet (still on roadmap: emit
distinct edge for @property attribute access vs method call); they
are recorded so future overload selection / property modeling can
use them without another schema change.
The struct extension is at-tail and existing call sites that memset
to zero stay correct; non-Python LSPs see flags=0 / decorator_qns=NULL
and behave identically.
Three new verification tests prove that generators, dataclasses, and
@property already work end-to-end through the Round 4-5 infrastructure
without further code changes:
- Generator[Foo, ...] iterables in for-loop bind loop var to Foo via
py_iterable_element_type's existing TEMPLATE handling.
- @dataclass class Point: x: Foo (class-body annotated assignment) is
picked up by Round 4's third-pass field registrar; p.x.method()
resolves through the field type.
- @property def thing(self) -> Foo: c.thing returns Foo because
attribute access on NAMED routes through py_func_return_type which
yields the getter's annotated return.
All 2895 prior tests stay green; 3 new tests pass. Total 2898 / 0.
Bench unchanged at 95% (the small bench fixture doesn't exercise
the new patterns; will expand in Round 7).
Round 5 parity push:
Subscript value typing — py_eval_expr_type's subscript branch resolves
indexing against the container's TEMPLATE type:
- TEMPLATE("dict"|"Mapping"|"defaultdict"|"OrderedDict"|...)[K] -> V
- TEMPLATE("list"|"set"|"frozenset"|"Sequence"|"Iterable"|...)[i] -> elem
- TEMPLATE("tuple", [A, B, ...])[literal_int] -> args[idx]
- Non-literal index returns UNION of all element types.
This unlocks the `self.cache[k].method()` pattern that's everywhere in
real Python. Also fixes `items[0].method()` for typed lists.
super().__init__(...) constructor edge — super().<attr> already worked
for regular methods; __init__ is special-cased because parent classes
typically don't register __init__ with a useful return type. Strategy
"lsp_super_init" with confidence 0.85.
Operator dunders — a + b looks up __add__ on a's type and uses its
return type. Same for *, -, /, //, %, **, <<, >>, &, |, ^, @. Falls
back to left-operand type when registry has no info (correct for the
common int+int / str+str / list+list cases).
3 new tests; all 2892 prior tests stay green. Bench unchanged at 95%
(fixture doesn't exercise these patterns directly — Round 6 will
expand it).
Round 4 / final parity push. Tracks self.x = expr from __init__ and
class-body annotated assignments (`x: Foo`) as registered fields on
the enclosing class type.
py_register_instance_field mutates the registry's CBMRegisteredType
field_names / field_types arrays in place. Re-registering the same
name overwrites; first-seen wins for capture order. Arena-allocated,
not a hot path.
py_lookup_field walks the alias / embedded chain with the same
CBM_LSP_MAX_LOOKUP_DEPTH cap as method lookup, so inherited fields
resolve through MRO without stack-overflow risk on pathological
hierarchies.
Field-fallback wired into py_eval_expr_type's attribute branch:
when method lookup misses, try field lookup. Returns the field's
type so downstream attribute / call resolution chains naturally.
Same fallback applied to UNION (single-match heuristic).
py_process_class now does three passes:
1. Class-body annotated assignments (`x: int = ...`) → fields.
2. __init__ / __post_init__ → self.x = expr fields.
3. All other methods.
This guarantees fields are registered before methods that reference
them, regardless of source order.
Annotated assignment binding fix: py_resolve_annotation is now called
on assignment annotations too, so `self.cache: dict[str, str] = {}`
binds with the proper TEMPLATE type instead of NAMED("dict[str, str]").
Bench result on the 136-line fixture: **95%** (36/38 calls resolved,
8.06 ms). Up from 79%. The 5% gap covers cases that need a full
constraint solver — see Final Stopping Point doc.
2 new tests; all 2890 prior tests stay green. Total 2892 / 0.
Round 3 parity push:
- match/case (PEP 634): when the case pattern is class_pattern (wrapped
in a case_pattern), narrow the match subject identifier to NAMED(class)
inside the case body. Push/pop scope per case. Capture sub-patterns
bind to UNKNOWN (field-type extraction is a v1.1 task — would require
field type info in the registry).
- await expr: identity passthrough. async def f() -> T registers with
return T, so `await f()` returning T is the resolved-call shape we
want without modeling Coroutine[Any, Any, T] explicitly.
- Expanded test_py_lsp_bench.c fixture from 65 lines to 136 lines
exercising every parity feature: dotted-module walks, dataclass +
Self chaining, multi-inheritance with super(), Optional/Union
annotations + narrowing (isinstance / is not None / walrus), generic
list[T] / dict[K, V] / defaultdict comprehensions, classmethod /
staticmethod, match/case dispatch, async/await, common stdlib
(logging.getLogger, json.loads, pathlib.Path, functools.lru_cache).
Bench result on the new fixture: 38 calls, 30 resolved, 79%, 7.50 ms.
Remaining 21% includes patterns where intermediate types lose precision
(e.g. super() constructor, dict subscript), instance attribute typing
(self.x where x isn't annotated at class scope), and decorator
return-type substitution (lru_cache).
All 2888 prior tests stay green; 2 new tests + bench expansion. Total
2890 / 0.
Round 2/3 parity push. Three connected gap fixes that share an annotation-
parser and a generic-aware iterable model.
Annotation parser upgrade:
- py_resolve_annotation now turns subscripted forms into structured types
rather than stripping. `Optional[Foo]` -> UNION(NAMED(Foo), None).
`Union[A, B]` -> UNION. `list[Foo]` / `dict[K, V]` / `Iterable[T]` ->
TEMPLATE with arg list. `X | Y` (PEP 604) handled at depth zero.
- New py_parse_type_text_qn variant: same parser without scope lookup,
used by py_register_def at registry-build time. Bare class names are
qualified to <module_qn>.<name> so registry lookups match.
- py_register_def prefers d->return_type (full subscripted text) over
d->return_types[0] when brackets are present — extract_defs's array
form strips the subscript, losing element types like Optional[Foo].
Narrowing:
- if isinstance(x, T): bind x as NAMED(T) inside the consequence body
(block-scoped via push/pop).
- if x is not None: strip None from x's UNION inside the consequence;
py_match_is_none recognizes "is not" as the single tree-sitter token
it actually is, walks through parens / walrus to find the underlying
identifier.
- Else branch unchanged — narrow's negation (not-T, including-None) is
not modeled in v1.
Walrus + truthy:
- if (x := compute()) is not None: walrus binds x in the enclosing scope
(PEP 572) so the body sees x. py_bind_walrus_in walks the cond AST
looking for named_expression / assignment_expression nodes.
Comprehension + for-loop element typing:
- New py_iterable_element_type maps TEMPLATE(list/set/Iterable/...) to
arg[0]; TEMPLATE(dict) to key arg; TEMPLATE(tuple, ...) to UNION of
args.
- Comprehensions push their own scope and bind for_in_clause loop vars
to the iterable's element type before walking body / filter clauses.
for_statement does the same.
UNION attribute access fallback (used when narrowing can't run):
- obj_type=UNION & exactly one non-None NAMED member has the attr:
emit edge with strategy=lsp_method_union, confidence 0.85.
7 new test_py_lsp.c cases. All 2880 prior tests stay green; 8 new tests
land. Bench unchanged (small fixture, no narrowing exercised there).
Phase 4g+ / 4l / 5b additions toward LSP-server type-resolution parity:
- Generic template parsing in PHPDoc: parse `Collection<User>`,
`array<int, User>`, `iterable<T>`, `list<T>`, etc. Bind variables
to CBM_TYPE_TEMPLATE; method dispatch on a TEMPLATE receiver
resolves against the template's base class.
- Foreach element-type propagation: `foreach ($users as $u)` where
$users: array<User> binds $u to User. Two-arg array<K,V> takes
V. Falls back to Iterator::current() return type for NAMED types
that implement the iterator interface.
- Negative narrowing: `if (!($x instanceof Foo)) return;` (or throw)
narrows $x to Foo for the rest of the enclosing scope.
- Foreach iterable can be an arbitrary expression (variable_name,
member call, function call) — fixed earlier-too-narrow filter.
- PHPDoc @var/@param accept generic syntax: type token now spans
angle brackets so `array<User> $users` parses correctly.
- Stdlib expansion (Laravel + Symfony): full Eloquent Builder
method chain (where/orderBy/limit/get/first/...), Eloquent Model
static factories, Illuminate\Support\Collection chains
(map/filter/sort/...), Symfony HttpFoundation Request/Response,
Carbon date methods, PSR-7 with*() builders.
Test suite: 81 -> 100 tests, all passing. Total: 2913 / 0.
Adds the resolver passes that close most of the gap to a real PHP language
server's type inference, while staying in-process and PHP-runtime-free.
New capabilities:
- Type narrowing: instanceof, is_string/int/float/bool/array/object/...
is_callable, is_iterable, is_numeric. Active inside if/elseif bodies
and after assert(...) (sequential narrowing)
- Property type tracking: typed property declarations, constructor
property promotion (public X $foo), and constructor-body inference
($this->bar = $bar where $bar is typed)
- Late-static-binding chain depth: full ancestor walk with cycle
detection for $obj->method() and Class::method() resolution
- Trait flattening: methods from `use Trait;` inside a class body are
re-registered with the using class as receiver; basic `as` aliasing
is honored
- PHPDoc class-level tags: @property TYPE $name binds a virtual field
on the class; @method TYPE name() registers a virtual method
(Eloquent-style dynamic dispatch)
- Match expression result type: union over arm result expressions,
first concrete type wins
- Ternary result type: prefers the consequent, falls back to alternative
- Variadic parameter binding: $args bound to `array`
- `use Vendor\\Helper as H;` two-name form correctly maps H -> Vendor.Helper
- Static-call unindexed fallback: when class is resolved from `use`
but methods unindexed, emit php_static_unindexed for the bridge
- Stdlib expansion: full Eloquent Builder + Model + Collection method
chains, Symfony HttpFoundation Request/Response, Carbon date/time,
PSR-7 with*() builders
Test suite expanded from 16 to 81 tests (2829 -> 2893 total).
Parity push #1 — six gap fixes on top of the 11-phase plan:
- Parent-module bindings: `import a.b.c` now binds `a`, `a.b`, `a.b.c`
via py_bind_dotted_prefixes. Walks the dotted prefix chain on every
import, regardless of import_is_from_style. Plus an attribute-access
path on a MODULE that detects submodules: when the registry has any
function whose qn starts with `<mod>.<attr>.`, evaluating `mod.attr`
yields MODULE("<mod>.<attr>"). This makes `os.path.join(...)` and
similar resolve correctly.
- typing.cast(T, x) returns NAMED(T) (re-evaluating T as an annotation,
so generic subscripts and forward references both resolve). Detects
bare `cast(...)` and qualified `typing.cast(...)` forms.
- typing.assert_type(x, T) is a no-op at runtime; we type the result
as type-of(x).
- Forward references as quoted strings: `def f(x: "Foo")` strips quotes
and re-resolves the inner annotation. Also handles double-quoted and
single-quoted forms.
- Self return type substitution: methods declared returning `Self` /
`typing.Self` / `typing_extensions.Self` substitute to NAMED(receiver)
at call resolution time. Enables fluent / builder pattern chaining
to keep resolving methods after each step.
- Generic subscript stripping: annotations like `list[Foo]` strip to
`list` for v1; `Optional[Foo]` likewise. Container element-type
substitution still deferred (Phase 8+).
Generator additions:
- Allowlist now includes urllib, http, concurrent (top stdlib usage).
- `from X import *` re-export following at generation time. Modules
like os.path (a star-import shim of posixpath / ntpath / genericpath)
now have all forwarded definitions registered under their own QN.
Iterates to a fixed point with an 8-step ceiling. typeshed tree
expansion is bounded — only modules transitively reachable from the
allowlist are pulled in.
Stats: 137 modules, 904 classes (2797 methods), 886 free functions
(was 114 / 753 / 794). Generated file grows from 20K to ~22K lines.
6 new test_py_lsp.c cases. All 2874 prior tests stay green.
Phase 10 of Python LSP integration. Adds:
- scripts/gen-py-stdlib.py: walks a typeshed/stdlib checkout, parses
each .pyi via Python's stdlib `ast` module, and emits one C source
file populating cbm_python_stdlib_register. v1 simplifications:
- overload stacks collapse to first signature
- ParamSpec / TypeVarTuple / Concatenate skipped
- version guards (if sys.version_info >= ...) flattened to union
of branches
- per-symbol min/max version guards not yet emitted (v1.1 follow-up)
Module allowlist matches PYTHON_LSP_PLAN.md Phase 10 list — top
stdlib modules by usage in indexed Python projects, skipping
tkinter / turtle / curses / xml / email for size.
- internal/cbm/lsp/generated/python_stdlib_data.c (auto-generated,
20,021 lines): 114 modules, 753 classes (2,385 methods), 794 free
functions registered. Source pinned to typeshed commit
a7912d521e16ff63caf7a8b64b9072542be36777 (recorded in header).
Compiles cleanly under ASan + UBSan.
- lsp_all.c includes the generated file. The CBM_PYTHON_STDLIB_GENERATED
macro disables py_lsp.c's no-op stub so the real registration runs.
4 new test_py_lsp.c cases verify resolution against the registry:
os.getcwd, collections.defaultdict constructor, pathlib.Path.exists
method via typed parameter, logging.getLogger.
Known v1 limitation captured in test comments: `import os.path` only
binds the leaf `path` in scope, so `os.path.join` style chained
attribute access on the parent module name doesn't yet resolve.
Phase 10.5 will stamp parent-module bindings.
All 2869 prior tests stay green; 4 new stdlib tests pass.
Phase 9 of Python LSP integration. Implements cbm_run_py_lsp_cross and
cbm_batch_py_lsp_cross, mirroring the Go LSP pattern:
- cbm_run_py_lsp_cross builds a per-call registry from caller-supplied
CBMLSPDef[] entries (covering both file-local and cross-file
definitions), then re-parses the source via tree_sitter_python (or
uses cached_tree when supplied) and runs the standard
py_lsp_process_file pipeline.
- py_register_lsp_defs translates CBMLSPDef shapes into
CBMRegisteredType / CBMRegisteredFunc entries. Python-specific
Protocol label is recognized as an interface-like structural type.
Pipe-separated embedded_types and method_names_str are split into
arena-allocated NULL-terminated arrays. Method receivers auto-create
type entries when the type isn't otherwise registered, matching the
Go convention.
- cbm_batch_py_lsp_cross loops the per-file entry, isolating each file
in its own arena and copying string fields into the caller's arena
so they outlive the file's scratch space.
3 new test_py_lsp.c cases cover cross-file method dispatch, cross-file
inheritance (Child(Base) calling self.shared from another module), and
batch processing of two interconnected files. All 2866 prior tests stay
green; 3 new cross-file tests pass.
Phases 7-8 of Python LSP integration. Adds:
- super().method() resolution: when the call's receiver is a super() proxy
inside a class context, walk the enclosing class's embedded_types and
resolve the attribute against the first base where it exists. Edge
carries strategy="lsp_super" and confidence 0.88.
- Multi-inheritance dispatch already worked for the current registry shape
(embedded_types is a list, py_lookup_attribute iterates) — added an
explicit test to lock in the behavior.
Also fixes a Python class-body extraction gap upstream of py_lsp: the
generic extract_class_methods walker skipped any class-body child whose
node type wasn't a function_definition, which excluded everything wrapped
in a decorated_definition (i.e. every @classmethod / @staticmethod /
@property method). Patched to peek through decorated_definition for the
inner function_definition before calling resolve_method_name. This is the
root cause of the C.classmethod() / C.staticmethod() resolution miss and
the fix is independent of language-specific decorator semantics — any
language whose grammar uses decorated_definition benefits.
7 new test_py_lsp.c cases cover decorated functions, classmethod calls,
staticmethod calls, dataclass constructor + method dispatch, super() to
the right base, multi-inheritance attribute resolution on both bases, and
PEP 695-style class names without subscript handling. All 2859 prior
tests stay green; 7 new tests pass. Total: 2866 / 0.
Bundles Phases 4-6 of Python LSP integration. py_lsp.c gains a real
binder + expression evaluator + call resolver mirroring go_lsp.c's
shape:
- py_register_def populates the registry from CBMFileResult.defs.
Class/Type defs land as registered types; Method/Function defs as
registered functions with receiver_type derived from the qualified
name parent. Python's extract_defs records a class's bases as a
single "(Base, Other)" string, so we split commas at depth 0,
strip parens / generic subscripts / keyword args (metaclass=...)
to produce usable embedded_types entries.
- py_lsp_process_file walks function/class definitions, pushing scope
per function and binding self/cls to the enclosing class type
(after parameter binding so the receiver wins over an unannotated
`self` parameter).
- py_bind_parameters resolves type annotations via py_resolve_annotation:
scope -> module-qualified registry lookup -> bare NAMED fallback.
- py_eval_expr_type covers literals (int/float/str/bool/None plus list/
dict/set/tuple containers), names (via scope, then module symbol),
attribute access on MODULE / NAMED, call expressions (constructors,
module-local functions, attribute calls), comparison / boolean /
ternary expressions, and parenthesized / binary fallthrough.
- py_lookup_attribute walks the registry's embedded_types chain with
the shared CBM_LSP_MAX_LOOKUP_DEPTH cap so MRO chains terminate
cleanly.
- py_emit_resolved_call writes (caller_qn, callee_qn, strategy,
confidence) entries to result->resolved_calls. Strategies:
lsp_direct, lsp_constructor, lsp_method, lsp_module_attr,
lsp_module_attr_unresolved.
6 new test_py_lsp.c cases cover direct calls, simple method dispatch,
self.method, constructor-then-method, single-inheritance dispatch,
and the no-false-positive guarantee on UNKNOWN receivers.
All 2853 previous tests stay green; 6 new py_lsp tests pass.
Mirrors go_lsp.c and c_lsp.c. In-process C type resolver, no PHP runtime,
no new dependencies. Uses already-vendored tree-sitter-php.
Coverage (Phase 1+2+3 from PLAN_PHP_LSP_INTEGRATION):
- namespace + use clause resolution (class / function / const)
- typed parameter binding (incl. arrow functions and closures)
- $this and self/parent/static dispatch
- method chaining via return-type propagation
- object_creation_expression typing for new X() chains
- PHPDoc minimal parser (@var, @param) for legacy untyped code
- magic methods: __call (php_method_dynamic), __callStatic (php_dynamic_unresolved)
- synthetic php_method_typed_unindexed for vendor receiver types,
used by the pipeline bridge to suppress name-fallback misroutes
- corpus-seeded stdlib (~200 lines): SPL containers/iterators, PSR
interfaces, DateTime, Throwable hierarchy, Closure, plus the top
Laravel framework facades + base types from the benchmark corpus
Pure rename — no logic change. The header is consumed only by
internal/cbm/extract_*.c, depends on tree_sitter/api.h, and is purpose-
built for AST traversal during extraction. The "extract_" prefix makes
the scope explicit and keeps the file naming consistent with its
neighbours (extract_calls.c, extract_imports.c, ...).
- internal/cbm/ts_node_stack.h -> internal/cbm/extract_node_stack.h
- Include guard CBM_TS_NODE_STACK_H -> CBM_EXTRACT_NODE_STACK_H
- Header banner comment updated
- 9 call sites in internal/cbm/extract_*.c updated to the new path
Full suite: 2824 passed, 0 failed.
Phase 3 of Python LSP integration. py_lsp_bind_imports walks the
import map populated from CBMFileResult.imports and binds each entry
into the root scope:
- `import X` / `import X as Y` -> bind to MODULE(X) (heuristic: when
module_qn does not end with `.<local_name>`).
- `from X import Y` / `from X import Y as Z` -> bind to NAMED(X.Y).
Phase 6 attribute resolution will downgrade to MODULE / class /
function as registry lookups become available.
- `from X import *` -> recorded in the import map but not scope-bound.
Phase 9 cross-file resolution will use the entry to walk re-exports.
- Relative imports (`from . import x`, `from ..pkg import y`) bind via
the same heuristic; module path is preserved so cross-file matching
can still locate the target later.
py_lsp_lookup_in_scope test hook added for unit-level scope inspection.
9 new tests in test_py_lsp.c. All 2844 prior tests stay green.
Phase 2 of Python LSP integration. Adds the py_lsp module mirroring
the go_lsp / c_lsp pattern: PyLSPContext, single-file entry
(cbm_run_py_lsp), cross-file + batch shells, stdlib registration stub.
Bodies are no-ops at this phase. Subsequent phases fill in:
Phase 3 — imports
Phase 4 — scope binding
Phase 5 — expression typing
Phase 6 — attribute resolution + method dispatch
Phase 7 — decorators
Phase 8 — class hierarchy + generics
Phase 9 — cross-file resolution + batch
Phase 10 — cbm_python_stdlib_register from typeshed
Wired into cbm_extract_file alongside the existing Go and C/C++ LSP
calls. lsp_all.c #includes py_lsp.c so the CGo umbrella picks it up
without a separate translation unit.
tests/test_py_lsp.c adds 5 smoke cases: empty source, single function,
single class, syntax error recovery, imports pass-through. The helper
shape (extract_py / find_resolved / require_resolved) matches
test_go_lsp.c so future phases can grow the suite by category.
All 2839 existing tests stay green; 5 new py_lsp tests pass.
Generalizes #224's Svelte/Vue script-block import extraction into a
spec-driven embedded-language pattern that also covers HTML and Astro,
so any future host grammar that keeps sub-language source as raw text
gets coverage with a one-line declaration.
The existing CBMLangSpec already exposes the per-language tree-sitter
grammar via cbm_ts_language(); add a parallel embedded_imports field
that lists which AST node types hold which sub-language. The new
generic walker in extract_imports.c reads that list, locates each
embedded content node in the host AST, re-parses the slice with the
embedded grammar resolved through cbm_ts_language(), and runs the
standard ES import extractor (walk_es_imports) over the inner AST.
No grammar symbols are referenced from the extraction file.
- internal/cbm/lang_specs.h: add CBMEmbeddedLangSpec + embedded_imports
field on CBMLangSpec.
- internal/cbm/lang_specs.c: declare vue/svelte/html/astro embed lists
({"script_element", "raw_text", CBM_LANG_JAVASCRIPT}); set the new
field on those four spec entries; explicit NULL initializer on every
other entry to satisfy -Wmissing-field-initializers.
- internal/cbm/extract_imports.c: add embedded_collect_content_nodes
+ parse_embedded_imports; route SVELTE/VUE/HTML/ASTRO through the
generic helper.
- tests/test_extraction.c: cover Svelte (with and without <script>),
Vue, and HTML.
Full suite: 2824 passed, 0 failed.
Closes#223. Closes#224.
Co-authored-by: James <jmcmacnz@gmail.com>
Phase 1 of Python LSP integration. Five new CBMTypeKind variants needed
for Python type modeling, plus structural-equality helper used by union
deduplication and protocol matching:
- UNION: A | B / Union[A, B] / Optional[T]. cbm_type_union normalizes
by flattening nested UNIONs, deduplicating members via cbm_type_equal,
and collapsing single-member unions to their member. Empty unions
reduce to UNKNOWN.
- LITERAL: Literal["foo", 3] — wraps a base type with literal text;
text-distinct literals compare unequal.
- PROTOCOL: typing.Protocol with method-name set. Structural matching
via cbm_type_protocol_satisfied_by — succeeds when the candidate's
method set is a superset of the protocol's.
- MODULE: bound name from `import X`, distinguishing module-attribute
lookup from class-attribute lookup. Equality is by module_qn.
- CALLABLE: Callable[[A, B], R] including the elliptic Callable[..., R]
encoded as param_count = -1.
cbm_type_equal added as a recursive structural-equality helper covering
all kinds; FUNC/INTERFACE/STRUCT fall back to pointer identity since
their structural members are rarely compared and the cost would be
substantial.
Adds tests/test_type_rep.c with 16 cases covering normalization,
equality, and protocol satisfaction.
All 2823 existing tests stay green; 16 new type-rep tests pass.
AST traversal functions used fixed-size TSNode stack[] arrays. When the
DFS stack filled up, the child-push loop exited silently, dropping
entire subtrees without warning (e.g. 600 TS imports → 511, 130 Express
routes capped at 512). Reported in #199; also addresses #213 (large
TypeScript files producing zero nodes from stack exhaustion) and #215
(SEGV in template calls from stack overflow).
Adds ts_node_stack.h — a growable stack backed by the existing arena
allocator. Initial capacity matches previous fixed caps so small files
allocate no extra memory; doubles on overflow instead of truncating.
Old blocks are abandoned in the arena and freed on arena_destroy at
the end of file extraction (no realloc, no mixed lifetimes).
Applied to all 14 TSNode fixed stacks across 9 extraction files
(extract_calls, extract_channels, extract_defs, extract_env_accesses,
extract_imports, extract_semantic, extract_type_assigns, extract_type_refs,
extract_usages). walk_defs (uses walk_defs_frame_t, different struct
type) deliberately left as-is.
7 regression tests in tests/test_stack_overflow.c cover TS imports >512,
JS/Python calls >512, Go calls >1024, Express routes >150, deeply
nested calls, and YAML vars >256.
Cherry-picked from #217 with merge-conflict resolution against post-#206/#207
main:
- Makefile.cbm: keep TEST_STACK_OVERFLOW_SRCS plus main's new
TEST_ZSTD_SRCS / TEST_ARTIFACT_SRCS.
- extract_channels.c: keep main's scan_string_consts_js name (main also
added scan_string_consts_python after the PR was opened); declare
CHAN_STACK_CAP in the enum block.
- test_stack_overflow.c: replace sprintf with snprintf using the actual
remaining buffer size, since macOS clang rejects sprintf and naive
snprintf with sizeof(p) on a char* fails -Wformat-truncation.
Closes#199, #213, #215. Closes#217.
Phase 0 of Python LSP integration. Two shared-infrastructure fixes that
silently bug Python without affecting Go/C++ regression tests:
- CBMScope replaces the fixed 64-binding array with a growable linked
list of 16-binding chunks. The arena pointer is captured at scope_push
and reused for chunk allocation, so existing callers benefit from
dynamic growth without an API change. Python class bodies and large
dataclass scopes routinely exceed 64 locals; this prevents silent
drops past the legacy cap.
- CBM_LSP_MAX_LOOKUP_DEPTH=16 wraps go_lookup_field_or_method and
c_lookup_member via depth-aware internal helpers. Pathological
multi-inheritance hierarchies (SQLAlchemy / Django / Zope) can exceed
20 levels and previously stack-overflowed; now they degrade to NULL
past depth 16 instead.
Adds tests/test_scope.c with 10 cases covering basic API, parent-chain
walks, child-shadowing, and dynamic growth at 300 bindings + chunk
boundaries. test_main.c and Makefile.cbm wire the new suite.
PYTHON_LSP_PLAN.md added at the worktree root: full phased plan plus
the verbatim challenger review.
All 2823 existing tests stay green; 10 new scope tests pass.
- branch_types: case_clause → switch_case + switch_default (correct
grammar node names)
- branch_types: add do_statement (do-while loops)
- import_types: remove 'require' (not a grammar node type)
Verified: TypeScript, Python, C, C++ specs are all correct — every
node type in the spec exists in the grammar's ts_symbol_names.
- C#: add record_declaration, type_declaration to class types;
add namespace_declaration to import types
- JavaScript: add new_expression to call types; add import, extends,
require to import types (also used by TS/TSX)
- Python: add import to import_types
- TypeScript: add function_signature to func types
- Makefile: add recipe to func, call to calls, include to imports
All 153 grammars now have maximally complete extraction specs.
2797 tests pass with zero regressions.
- pass_parallel.c: detect gRPC calls when resolved QN contains
ServiceClient/ServiceGrpc/Servicer — emits GRPC_CALLS edges for
calls resolved to generated protobuf client interfaces
- pass_parallel.c: emit_grpc_edge falls back to resolved QN when
callee_name alone doesn't contain service.method pattern
- extract_calls.c: add resolve_chained_selector() for iterative
extraction of chained method calls (NewClient(conn).GetBar →
"NewClient.GetBar") without recursion
Known limitation: Go receiver-method chained calls like
pb.NewCartServiceClient(conn).GetCart() in rpc.go are not yet
extracted as CALLS edges — the Go grammar's AST structure for
these patterns needs dedicated investigation.
Systematic audit of every grammar's parser.c against the current
lang_specs.c entries. For each language, identified all primary
extraction-relevant node types (function/class/call/import) present
in the grammar but missing from the spec.
Changes:
- 44 new node type arrays created for languages that had empty_types
where the grammar provides extractable constructs
- 121 existing arrays extended with additional node types
- Covers all 153 vendored grammars
Examples of additions:
- Go: import, import_declaration added to import_types
- Python: import, import_from_statement, import_statement added
- JavaScript: new_expression, call_expression added to call_types
- PHP: namespace_use_declaration added (was completely missing)
- Rust: macro_invocation added to call_types
- C/C++: struct_specifier, enum_specifier, union_specifier added
All 2797 tests pass with zero regressions.
When a single record payload exceeds 65KB (max_local=65501), the writer
now splits it across overflow pages per SQLite's B-tree spec instead of
underflowing the content_offset and corrupting memory.
Closes#139, closes#187.
Based on #175 by dLo999.
Co-Authored-By: Dustin Obrecht <dustin@kurtnoble.com>
Add .codebase-memory/graph.db.zst — a zstd-compressed knowledge graph
artifact that can be committed to the repo. Teammates bootstrap from
the artifact instead of running a full reindex from scratch.
- Vendor zstd 1.5.7 (amalgamated build) for 8-13:1 compression
- Two-tier export: zstd -9 + index stripping for explicit index,
zstd -3 for watcher/incremental auto-updates
- Import: decompress → integrity check → auto-recreate indexes
- Bootstrap in handle_index_repository: when no local DB exists but
artifact is present, import first then run incremental
- Auto-create .gitattributes with merge=ours to prevent conflicts
- Fix: add missing idx_edges_url_path to create_user_indexes and
url_path_gen generated column to init_schema
- 13 new tests (5 zstd wrapper + 8 artifact round-trip/edge cases)
Pipeline: gRPC/GraphQL/tRPC client libs in service_patterns, edge emitters
in pass_parallel, __grpc__ Route nodes from .proto service defs in
pass_route_nodes, cross-repo matchers in pass_cross_repo. Also adds MQTT,
NATS, Dapr async patterns.
Protobuf extraction: service as Class, rpc as Function with rpc_name
fallback in func_name_node. Route nodes created by line-range matching
rpc Functions to enclosing service Classes.
Tested: Online Boutique protos produce 16 gRPC Route nodes.
- Enum member extraction: C#/Java/TS/C++ enum values as Variable nodes
- JS/TS destructured variable splitting: const { A, B, C } = require()
emits individual Variable nodes instead of one blob
- Cypher -- SQL-style single-line comments
- Cypher in_degree/out_degree virtual node properties (thread-safe
via binding store pointer, enables dead code detection via Cypher)
- Cypher regex matching in inline property filters
- Resolution tiebreaking: test/mock QN deprioritization and C# field
type hint resolution for obj.Method() disambiguation
Co-Authored-By: Tommy <Koolerx@users.noreply.github.com>
Fill in function, class, field, call, import, branch, variable,
assignment, throw, and decorator node types for all new languages
based on per-grammar AST analysis. Previously all 89 new languages
had empty_types (module detection only, no extraction).
Programming languages with full specs (31):
Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim,
Crystal, Odin, Ada, Sway, Cairo, Move, Hare, Pony, AWK, Fish,
Zsh, Tcl, Fennel, ReScript, PureScript, Nickel, Teal, Luau,
Squirrel, ISPC, Starlark, Puppet, LLVM IR
IDL/config with type/field specs (12):
Prisma, Thrift, Cap'n Proto, Smithy, WIT, Bicep, WGSL, HLSL,
VHDL, SystemVerilog, Apex, Kconfig
Remaining languages have module_types only (config/data formats
where function/class extraction is not applicable).
