mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-10-28 07:03:20 +08:00
beef06ec6ddb2b745bcab11bc2499fba4cf53d15
568
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Martin Vogelandsponger94
|
beef06ec6d |
feat(extract): C# field/property + C# 12 primary-constructor support
Three C# extractor gaps that left modern .NET classes partially invisible
in the graph:
1. lang_specs.c CSHARP entry had `empty_types` for field_node_types, so
walk_defs never descended into field_declaration / property_declaration
nodes. Add cs_field_types and wire it in. Without this, no C# fields
or properties were ever pushed as Field defs.
2. extract_class_fields assumed the field's "type" is on the immediate
child via ts_node_child_by_field_name. Tree-sitter-c_sharp wraps fields
as field_declaration > variable_declaration(type, variable_declarator),
one level deeper than Java/Go/Rust. Detect the nested shape by walking
named children for variable_declaration and resolve type + name from
inside it. Direct shape continues unchanged.
3. C# 12 primary constructors (`class Foo(IBar bar) : Base { ... }`)
declare implicit captured fields on the class line. Tree-sitter wraps
them inside the hidden _class_declaration_initializer node, so the
`parameters` field on class_declaration may not always resolve
directly; iterate top-level children for parameter_list as a fallback.
Emit each primary-ctor parameter as a Field def with parent_class set.
Effect: dependency-injected fields on .NET 8+/9+ controllers and any C#
auto-property (`public IFoo Foo { get; }`) are now reachable as Field
nodes in the graph.
Co-authored-by: sponger94 <45746997+sponger94@users.noreply.github.com>
|
||
|
Martin Vogel
|
b2a71b80b9 |
feat(pipeline): wire cross-file LSP across all five hybrid resolvers
Per-file LSP (run from cbm_extract_file) only sees one file's defs, so
callees whose receiver type comes from an imported module stayed
unresolved — even though Go/C/Py/TS already had cross-file entry points
(cbm_run_X_lsp_cross). Those entry points existed but no pipeline pass
ever invoked them. PHP didn't even have a cross-file entry point.
This change lands the missing pieces:
1. cbm_run_php_lsp_cross + cbm_batch_php_lsp_cross — PHP cross-file
resolver mirroring py_lsp_cross / ts_lsp_cross. Reuses PHPLSPContext,
php_lsp_init, php_lsp_add_use, php_lsp_process_file. Handles PHP-
specific labels (Class / Interface / Trait / Enum / Type) and uses
def_module_qn so cross-file return types qualify against the def's
own module rather than the importing module's.
2. pass_lsp_cross.c — new pipeline pass. For every cached file result,
builds a project-wide CBMLSPDef[] (CBMDefinition → CBMLSPDef
converter), per-file import map (from gbuf IMPORTS edges), then
dispatches to the matching cbm_run_X_lsp_cross. Output strings get
copied into the per-file arena (with O(N²) memory bug avoided by
running each file inside a fresh scratch arena that gets destroyed
after the LSP call — the LSP allocates its registry + stdlib +
project defs in scratch, not in the caller's persistent arena).
3. Wired into both pipelines:
- sequential: definitions → k8s → lsp_cross → calls → ...
- parallel: parallel_extract → registry_build → lsp_cross →
parallel_resolve → ...
4. Updated tests/test_parallel.c run_parallel helper to mirror the new
parallel ordering so existing cross-file Python/PHP tests now exercise
pass_lsp_cross.
Languages covered: Go, C/C++/CUDA, Python, TS/JS/JSX/TSX, PHP. PHP cross
file is new. The other four had functions but no pipeline integration
until this change.
Verified on a 1670-file FastAPI index: pass_lsp_cross processes 1050
Python files (525 non-LSP files skipped), collects 4070 project defs in
9.2s, and parallel_resolve subsequently reports lsp_overrides=568 — the
LSP-strategy attribution gain over per-file alone.
Per-LSP emit functions dedup by (caller_qn, callee_qn) within their own
run; pxc_append_results dedups again at append time so cross-file
entries duplicating per-file output get dropped instead of inflating
result->resolved_calls.
|
||
|
Martin Vogel
|
d1143fbdce |
feat(lsp): TypeScript / JavaScript / JSX / TSX hybrid LSP resolver
Adds clean-room ts_lsp resolver covering single- and cross-file type resolution for TS, JS, JSX, TSX. Mirrors the structure and entry-point pattern of the existing Go and C/C++ LSP hybrids (single-file cbm_run_ts_lsp + batch cbm_run_ts_lsp_cross), with TS-specific surface: - Hash-table indexed type registry (FNV-1a, lazy via cbm_registry_finalize) - Partial structural relater for NAMED / BUILTIN / LITERAL / TUPLE / UNION / INTERSECTION / FUNC / TEMPLATE / TYPE_PARAM / ALIAS - Conditional types T extends U ? X : Y with union distribution - 'infer X' constraint solver, keyof / typeof / T[K] type-position parsing - Flow-sensitive narrowing (instanceof / typeof / discriminated unions), optional chaining, non-null assertion - TS utility-type passthrough (Partial, Pick, ReturnType, Awaited, etc.) - Function overload resolution by argument types - Async iterables + 'for await ... of' - Curated DOM stdlib subset + comprehensive base stdlib - Mode flags: js_mode (.js/.jsx with JSDoc inference), jsx_mode (JSX expressions), dts_mode (.d.ts ambient declarations, no resolved calls) Touches existing files only with additive changes: - type_rep.h/.c: new TS-specific CBMTypeKind values + constructors - type_registry.h/.c: TS-specific fields, hash indexes, cbm_registry_finalize - cbm.c: dispatch for JS/TS/TSX languages - lsp_all.c, Makefile.cbm, tests/test_main.c: include + suite registration - README.md: feature list mention 291 ts_lsp test cases. CBM_LSP_DISABLED=1 disables the resolver (matches CBM_LSP_DEBUG / CBM_LSP_DISABLED knobs in go_lsp / c_lsp). |
||
|
Martin Vogel
|
673ac4ef8a |
fix(pipeline): resolve LSP cross-file QN mismatch via project-prefix fallback
Per-file py_lsp emits resolved_calls.callee_qn as the raw import-module path (e.g. `greeter.Greeter` from `from greeter import Greeter`) rather than the project-qualified QN the gbuf stores (`<project>.greeter.Greeter`). Before this change, the LSP match succeeded (lsp_overrides counter incremented) but the downstream cbm_gbuf_find_by_qn lookup missed silently and the edge was dropped — the failure mode the new lsp_overrides telemetry was designed to surface. The LSP layer can't tell in-project imports (qualify) from external imports (don't qualify, e.g. `os.path`) without consulting the gbuf, which is built downstream. So normalise at the consumer instead: try the LSP-emitted QN as-is first; on miss, retry with `<project>.<callee_qn>`. If that also misses, drop the edge — same as before, target is external/unindexed. New helper cbm_pipeline_lsp_target_node in lsp_resolve.h is shared by both pipelines (sequential pass_calls.c and parallel pass_parallel.c) so they continue to admit identical sets of LSP overrides. The helper also canonicalises res.qualified_name to the gbuf node's QN, so downstream edge property serialisation shows the project-qualified form even when fallback resolution kicked in. Cross-file regression test parallel_python_lsp_override_cross_file_- emits_lsp_strategy_edges pins the two-file Greeter scenario that originally exposed the bug. |
||
|
Martin Vogel
|
085b4a718c |
fix(pipeline): unify LSP-override resolver across sequential and parallel paths
The sequential pass_calls and the parallel cbm_parallel_extract pipelines each carried their own copy of the LSP-resolved-call lookup with divergent confidence floors and different match semantics. Most production indexing runs the parallel path, so projects produced different CALLS-edge attributions depending on which pipeline mode kicked in: pass_calls.c find_lsp_resolution floor 0.6 (CBM_LSP_CONFIDENCE_FLOOR) pass_parallel lsp_override_resolution_pp floor 0.5 (hardcoded) Both pipelines correctly invoke cbm_run_*_lsp through cbm_extract_file, so result->resolved_calls is populated identically — the bug was at the consumer. Changes ------- - New src/pipeline/lsp_resolve.h: static-inline cbm_pipeline_find_lsp_resolution and CBM_LSP_CONFIDENCE_FLOOR (0.6f). Single source of truth for the match rule (caller_qn equality + callee_qn short-name match + floor), with rationale and language-coverage notes inline. - pass_calls.c: include lsp_resolve.h, drop the local copy, rename the call site to cbm_pipeline_find_lsp_resolution. - pass_parallel.c: include lsp_resolve.h, drop lsp_override_resolution_pp, use cbm_pipeline_find_lsp_resolution at resolve_file_calls. Add an int lsp_overrides counter to resolve_worker_state_t (with adjusted cache-line padding for the new field), aggregate it per-worker, and emit it in the parallel.resolve.done log line so divergence between pipelines becomes observable in production telemetry. - tests/test_parallel.c: parallel_python_lsp_override_emits_lsp_strategy_edges builds a single-file Python scenario (Greeter class + main() in one file), runs the parallel pipeline, walks every CALLS edge, asserts at least one carries an "lsp_*" strategy in its properties_json. Pins both halves of the wiring at once: the shared find helper IS consulted in the parallel path, AND the LSP-attributed strategy survives all the way to the gbuf edge. Single-file rather than cross-file because per-file py_lsp's resolved_calls callee_qn uses the import module path rather than the project-qualified QN, so cross-file LSP edges currently get dropped by cbm_gbuf_find_by_qn — a separate cross-file QN-mismatch bug to track elsewhere. Full suite: 3250 passed, 0 failed (was 3249; +1 new regression test). |
||
|
Martin Vogel
|
e6f4aa74bd |
Merge origin/main into worktree-php-lsp-integration
Resolved 5 conflicts where the php-lsp branch and main both added LSP
infrastructure but with different APIs:
- Makefile.cbm: kept both TEST_PHP_LSP_SRCS and main's py_lsp/scope/
type_rep/stack_overflow test source variables in ALL_TEST_SRCS.
- internal/cbm/cbm.c: kept both #include lsp/php_lsp.h and lsp/py_lsp.h;
the per-language dispatch already supports stacking PHP after PYTHON.
- internal/cbm/lsp_all.c: kept both lsp/php_lsp.c + php_stdlib_data.c
AND lsp/py_lsp.c + python_stdlib_data.c in the compilation unit.
- src/pipeline/pass_calls.c: took main's version entirely.
find_lsp_resolution + resolve_single_call(ctx, call, lsp_calls, ...)
is language-agnostic — it matches any CBMResolvedCall whose caller_qn
hits the textual call's enclosing function and whose callee_qn
short-name hits the textual callee. PHP LSP populates the same
result->resolved_calls array via cbm_run_php_lsp, so the unified
resolver handles PHP for free. Dropped the php-branch's
qn_short_name + lsp_override_resolution helpers as superseded.
Locked confidence floor at main's CBM_LSP_CONFIDENCE_FLOOR = 0.6f
(was 0.5f on the php branch).
- tests/test_main.c: kept all suite externs and RUN_SUITE invocations
for php_lsp + py_lsp/py_lsp_bench/py_lsp_stress/py_lsp_scale +
scope + type_rep.
Auto-merged: src/pipeline/pass_parallel.c.
Brings in 52 commits from main since the branch was forked at
|
||
|
Martin Vogel
|
4d9f62f7e6 |
Merge origin/main into worktree-python-lsp-integration
Resolved conflict in Makefile.cbm: keep both TEST_STACK_OVERFLOW_SRCS
(from main, #217) and the new py_lsp test variables (TEST_SCOPE_SRCS,
TEST_TYPE_REP_SRCS, TEST_PY_LSP_SRCS, TEST_PY_LSP_BENCH_SRCS,
TEST_PY_LSP_STRESS_SRCS, TEST_PY_LSP_SCALE_SRCS) in ALL_TEST_SRCS.
Other auto-merged files: internal/cbm/extract_defs.c (PR #279),
tests/test_main.c (multiple suite registrations on each side).
Brings in 28 commits from main since the branch was forked at
|
||
|
Martin Vogel
|
d5742aab90 |
ci(codeql): revert CodeQL gate hardening from 27e640b
Reverts the CodeQL portion of
|
||
|
Martin Vogel
|
27e640b3b0 |
fix(mcp,ci): harden get_architecture serialization + CodeQL gate
Follow-up to #281 (handle_get_architecture) plus a CodeQL workflow upgrade developed in parallel; bundling into one commit because the test suite had to land alongside both. mcp.c — handle_get_architecture - NULL-coerce every const-char* field in the architecture sections via `x ? x : ""`, matching the rest of mcp.c (search_graph, etc.). Without this, a NULL field becomes a missing JSON key instead of an empty string; yyjson_mut_obj_add_str returns false on NULL and silently no-ops, so an inconsistent omission could surprise callers. - Serialize two more architecture aspects that #281 left on the floor: services (cbm_service_link_t: from/to/type/count) and clusters (cbm_cluster_info_t: id/label/members/cohesion plus the top_nodes / packages / edge_types string arrays). The store-side computation populates these for aspects=["all"] / explicit names, so dropping them in the serializer was data loss. tests/test_mcp.c - New tool_get_architecture_emits_populated_sections regression test. Uses a minimal inline fixture (single Function node tagged with "is_entry_point": true) since arch_entry_points reads that flag out of properties_json. Asserts the response contains both an "entry_points" array and the function name — neither would appear before #281 because handle_get_architecture never called cbm_store_get_architecture. - extract_text_content drilled too shallow: it pulled "content" only from the JSON root, so it worked for cbm_mcp_handle_tool but silently fell through to the raw response for cbm_mcp_server_handle (where content lives under .result.content). Added a fallback that checks .result.content; both unwrappers tested by existing fixtures. ci(codeql) - Run on pull_request to main, not just push; surfaces findings on the PR instead of after merge. - Pull custom queries from ./codeql via `queries: +./codeql`. - Capture SARIF output and fail the job on any error-level finding, using jq to enumerate rule id, file:line, and message text in the GitHub Actions error annotation. Warnings are still reported as before; only errors block. Full suite: 2842 passed, 0 failed. |
||
|
Martin Vogel
|
50b218e5d5 | docs(benchmark): final PHP-LSP numbers — 278 tests, ~10k LoC | ||
|
Martin Vogel
|
89531c7501 |
feat(php-lsp): @phpstan-type aliases + Closure::bind + variance + 30 tests
Phase 5f, 4ac, 4ad: closing the long-tail gaps toward LSP-server parity.
- @phpstan-type / @psalm-type / @phan-type aliases parsed from class
docblocks and applied during PHPDoc type resolution. References to
the alias name in @var/@param/@return are rewritten to the aliased
type. Aliases live on PHPLSPContext (per-file).
- @phpstan-import-type accepted without crash (cross-file alias
resolution deferred — same root cause as Composer PSR-4).
- @template-covariant T and @template-contravariant T now register T
as a proper type-param (previously skipped along with
@template-extends).
- Closure binding: \\Closure::bind($f, $obj) and $f->bindTo($obj),
when invoked with an inline closure literal as the first arg, walk
the closure body with $this rebound to the second arg's class.
Resolves the Laravel-macroable / VarDumper / encapsulation patterns.
- @return ($x is string ? int : bool) conditional return-type syntax
accepted without crash (parsed leftmost segment as fallback).
- Long PHPDoc blocks with mixed @param/@return/@throws/@internal
parsed without misattributing tags.
Stdlib unchanged.
Tests: 248 → 278 covering:
- phpstan-type alias basic / via @param / union / array-shape
- psalm-type alternate spelling
- phpstan-import-type (no-crash)
- Closure::bind / ::bindTo / static-bind on arrow fn
- conditional return / template-covariant / -contravariant (no-crash)
- class-string<T>, int<0,100>, literal-string (no-crash on phpstan
types we don't fully model)
- 8 realistic patterns: repository, event listener, console command,
form handler, mail sender, cached service, logger w/ context,
Carbon chains in methods, Eloquent model.
Total tests: 3061 → 3091 / 0 failed.
|
||
|
Martin Vogel
|
ecfbbf48d6 |
perf(py_lsp): cap dedup window at 256 entries — 91.8x -> 31.4x scaling
Audit for O(n^2) hot paths after Round 10. The dedup loop in py_emit_resolved_call scanned ALL prior entries linearly, making emission O(N) and total file work O(N^2). On a synthetic 2000-class fixture with 8000 calls, that dominated time at 5083 ms. Cap the scan window at the last 256 entries. Most duplicate emissions come from the same expression being walked twice (resolver + emitter passes), so they're temporally adjacent — a 256-entry window catches every common case while keeping the dedup O(1) per emission. Scale measurements (test_py_lsp_scale.c added): | Classes | Calls | Resolved | Before | After | |--------:|------:|---------:|-------:|------:| | 100 | 400 | 300 | 55ms | 57ms | | 500 | 2000 | 1500 | 333ms | 323ms | | 2000 | 8000 | 6000 | 5083ms | 1804ms| Ratio for 20x input: 91.8x -> 31.4x (linear would be 20x; quadratic would be 400x). No remaining quadratic hot path in code added by this branch. Other hot paths surveyed: - py_register_def -> cbm_registry_lookup_type: O(types) per def at registration, O(D x types) for D defs. Pre-existing linear scan in type_registry.c; hash-indexing it would be a cross-language optimization (Go and C++ LSPs benefit too). - py_lookup_attribute / py_lookup_field: O(types) per call site. Same pre-existing linear scan. With ~900 stdlib types + per-file defs, accounts for ~75µs/call at 2000-class scale. - MODULE submodule prefix scan: O(funcs) per module-attribute access, early-exits on first match. Bounded by registry size. - lambda / dict-literal registries: linear, but bounded by per-file N which is small. Bench unchanged at 100% / 12ms (52/52 calls, 178 LOC). |
||
|
Oliver Evans
|
ca5e309062 |
fix(mcp): wire cbm_store_get_architecture into handle_get_architecture (#281)
handle_get_architecture was returning only label/edge histograms because it never called cbm_store_get_architecture — the rich architecture computation sat completely unwired upstream of the MCP boundary regardless of which aspects were requested. Wires it up: - Parse the JSON aspects array into a const char*[MCP_COL_16] and pass to cbm_store_get_architecture, which already handles 'all' and per-name filtering internally. - Serialize each non-empty section: languages, packages, entry_points, routes, hotspots, boundaries, layers, file_tree. Existing histogram fields (node_labels, edge_types, relationship_patterns, cross_repo_links) preserved. - Call cbm_store_architecture_free before return. Closes #280. |
||
|
Martin Vogel
|
65964b4267 | docs(benchmark): refresh PHP numbers — 248 tests, 47k CALLS edges | ||
|
Martin Vogel
|
149d2831b8 |
test(php-lsp): 50 more edge-case tests (200 → 248)
Real-world PHP patterns covered: - Constructor property promotion + chain - Enum method dispatch with backed values - assert + 2-level chain - Negative narrowing through throw - Closure::fromCallable via [\$this, 'method'] - Late assignment chain ($a = null; $a = new A(); $a->go()) - Callable typed parameter - Nullable param chain via if-truthy narrowing - Attribute (#[...]) above class - Method with reference param - Abstract methods (no body) - Interface only (no impl) - Nested match expressions - Long static chain (Carbon) - Iterable typed parameter - array<T> via @var - Two traits in one class - Enum implementing interface - Constructor with readonly + private + promotion - Anonymous class (graceful no-resolve) - Arrow function implicit capture - 5-deep namespace path - PSR-7 full lifecycle - DateTimeImmutable chain - Throwable getMessage/getCode/getFile/getLine - PSR Container has/get - Long nullsafe chain - self/static in constructor - parent:: method dispatch - private/protected/public method visibility - final class - Property with default value - Private static factory + chain Total: 3013 → 3061 / 0 failed (php_lsp 200 → 248). |
||
|
Martin Vogel
|
d8646dae91 |
feat(php-lsp): trait self substitution + 2-pass class processing
When a trait method declares `: self` as its return type, flattening
that method into a using class now substitutes the trait's QN with the
using class's QN in the return type. This fixes fluent chains like:
trait T { public function tap(): self { return $this; } }
class C { use T; public function ok(): int { return 1; } }
$c->tap()->ok(); // resolves through to C.ok now
Implementation:
- flatten_trait_into_class scans the source method's signature for
NAMED(trait_qn) in the return type and rewrites it to
NAMED(using_class_qn) when registering the flattened method.
- php_lsp_collect_class_fields now does TWO passes — pass 0 for
trait/interface declarators (so their declared-return-types get
re-parsed before any consumer), pass 1 for class/enum (which may
`use Trait;` and need the trait's signatures already up-to-date).
Total tests: 3013 / 0 failed (200 php_lsp).
|
||
|
Martin Vogel
|
20dd651a2e |
feat(php-lsp): @return $this/static, if-condition call walk, +50 tests
Phase 4aa, 4ab partial, 4y/z best-effort, 5e: more stdlib + tests.
- @return $this / @return static / @return self in PHPDoc all bind to
the enclosing class type, enabling fluent-builder chains.
- Declared return-type re-parse uses ctx->enclosing_class_qn so the
`: self` shorthand on a method correctly returns NAMED(class).
- php_resolve_class_name now checks self/static/parent BEFORE the
builtin-name guard, fixing a long-standing miss where these
pseudo-types were rejected as builtins.
- if-condition walk: calls inside `if ($it->valid())` are now
resolved (they previously only got narrowing analysis).
- Removed duplicate Generator type registration in stdlib data.
Stdlib expansion (Phase 5e):
- Symfony Cache (CacheInterface, ItemInterface) + PSR Cache pool/item
- Symfony EventDispatcher (EventDispatcherInterface, Event)
- Symfony Mailer (MailerInterface) + Mime\\Email fluent builder
- Symfony Validator (ValidatorInterface, ConstraintViolationListInterface)
- Laravel Http\\Request, JsonResponse, Response with method chains
- Laravel Auth\\Guard, Authenticatable contracts
- Laravel Session, View
- ReactPHP / GuzzleHttp Promise interfaces
- Monolog Logger (extends PSR LoggerInterface)
- Reflection API (ReflectionClass/Method/Property/Function)
Tests: 150 -> 200 covering @return $this fluent chains, Symfony
Cache/Mailer/Validator, Laravel Request/Auth/View, Promise chains,
Monolog, Reflection, multi-interface impl, deep parent chains, foreach
key-value chains, and a realistic Laravel UserController integration.
Total tests: 2963 -> 3013 / 0 failed.
|
||
|
Martin Vogel
|
7a77a178ec |
docs(py_lsp): final results — 43/43 stress patterns PASS, zero gaps
BENCHMARK_PYTHON.md updated through Round 10. The stress-test suite went from 22 patterns (19 hard / 3 KNOWN GAP) to 43 patterns (all hard-asserted PASS). Bench held at 100% (52/52) across the surface expansion. Patterns added in Round 10 cover the framework / real-world cases that distinguish parity-with-LSP from passing-toy-fixtures: SQLAlchemy 2.0 Mapped[T], Pydantic BaseModel, async with / async for, Callable[..., R], next(iter), **kwargs annotation, dict-literal dispatch tables, lambda call-site inference, recursive types, classmethod-to-method chains, dataclass default_factory, nested match patterns. The original Phase 11 plan called this category "compiler-grade". The actual line where compiler-rebuild starts is now narrower: constraint solving for ParamSpec, code-flow narrowing across basic blocks, custom metaclass __call__, and full descriptor protocol. None of those are needed for what's resolved on the bench fixture or the stress suite. |
||
|
Martin Vogel
|
39f41c3d47 |
feat(py_lsp): close all stress gaps — lambda, dict-dispatch, async, callable, next, kwargs
Round 10. Closes the last documented gaps from Round 9 plus 21 new
real-world stress tests covering SQLAlchemy / Pydantic / dataclass /
TypedDict / Protocol / generator / async patterns.
Lambda call-site inference:
- PyLSPContext gains a per-context lambda registry. `fn = lambda x:
body` records the lambda's AST node bound to fn. At call site
fn(arg), we push a sub-scope, bind lambda params from arg types,
and walk the body. Edges emitted under a synthetic <lambda> child
of the enclosing function so traces attribute correctly.
Dict-literal dispatch table:
- `funcs = {"a": foo, "b": bar}` records per-literal-key target QNs
in CBMDictLiteralEntry[]. funcs["a"]() emits an edge directly to
foo's QN with strategy=lsp_dict_dispatch.
Async with / async for:
- `async with X() as f` looks up __aenter__ before __enter__.
__aenter__'s declared return type wins; the manager type is the
fallback (async __aenter__s typically return self).
- `async for x in iter` falls back to __anext__'s return type when
the iterable's TEMPLATE form isn't recognized as AsyncIterator /
AsyncGenerator.
Callable / next / iter:
- `f()` where f resolves to CBM_TYPE_CALLABLE returns the callable's
return type. TEMPLATE("Callable", [..., R]) returns R.
- factory()() / (lambda)() etc — when the function expression is
itself a call / parenthesized / lambda, evaluate it and use its
return type.
- `next(it)` returns py_iterable_element_type(type-of-it).
- `iter(x)` is identity.
*args / **kwargs typed-parameter wrapping:
- `def f(*args: T)` -> args is TEMPLATE("tuple", [T]).
- `def f(**kwargs: T)` -> kwargs is TEMPLATE("dict", [str, T]).
- Detected by walking typed_parameter > list_splat_pattern /
dictionary_splat_pattern wrappers in py_bind_parameters.
Tuple / list / set / dict literal expressions:
- py_eval_expr_type now builds structured TUPLE / TEMPLATE types
from element types of the literal. `(Foo(), Bar())` -> TUPLE([Foo,
Bar]); `[Foo()]` -> TEMPLATE("list", [Foo]); `{"k": Foo()}` ->
TEMPLATE("dict", [str, Foo]). Without this, unpacking lost element
precision.
Match sequence pattern:
- list_pattern wraps each element in another case_pattern; capture
appears as dotted_name -> identifier; splat as splat_pattern.
Element-binder walks both wrappers and binds capture/splat to the
subject's iterable element type / list[elem].
Stress test suite expanded from 22 to 43 patterns. All 43 hard-
asserted as PASS:
- typing helpers: NamedTuple, TypedDict (incl. total=False), Protocol
(nominal + structural), ABC + abstractmethod, Mapped[T] (SQLAlchemy
2.0), Pydantic-style BaseModel
- control flow: with-as via __enter__, except-as, post-early-return
narrowing, async with via __aenter__, async for via __anext__
- containers: tuple unpack from function return + literal,
dict.items() comprehension k/v unpack, list slice, dict.get
default, recursive self-referencing types via forward refs
- decorators: factory + functools.wraps, classmethod chained to
method, property setter
- inheritance: Self in chain, diamond MRO with C3, multi-class match
- functional: lambda call-site inference, function-as-dict-value
dispatch, callable return-value, next(iter), generator delegation
(yield from), iterator next, double-star **kwargs, isinstance |
branches, module-level const, dataclass + default_factory
Bench: 178 LOC, 52 calls, **52 resolved (100%)**, 11.25 ms (ASan +
UBSan). Total 2941 / 0. Zero remaining KNOWN GAPs in the stress
suite — every probed pattern resolves.
|
||
|
Martin Vogel
|
cca345801f |
test(extraction): regression for Java extends+implements INHERITS edges (#279)
Pins the bug-fix from #279 with a Java class declaring both extends and implements. Asserts that base_classes contains: - the superclass name (DefaultDiagramTool) - every implements interface (ILinkTool, Closeable) - bare type names only — no 'extends' / 'implements' keyword text leaking into any entry Without the fix, the field loop returned on the first match (only the extends parent emitted) and cbm_node_text on the field returned the full literal 'extends Bar' / 'implements Baz, Qux' string. |
||
|
Loay chlih
|
323c68e4d1 |
fix(extract_defs): emit all INHERITS edges for Java extends+implements (#279)
Two bugs in extract_base_classes() caused most INHERITS edges to be missing for Java (and any other language using similar AST field names): Bug 1 — Early return: the field loop returned on the first matching field, so class Foo extends Bar implements Baz produced only one INHERITS edge instead of two — the loop never processed 'interfaces' after finding 'superclass'. Bug 2 — Keyword text: cbm_node_text on the field node returned 'extends Bar' / 'implements Baz, Qux' literally instead of the bare type names. So even the one edge that did get emitted had a malformed target. Fix: new collect_bases_from_field() walks into child AST nodes and extracts type_identifier / generic_type / qualified_name / scoped_type_identifier / user_type text directly. Handles type_list / interface_type_list children for the multi-interface case. Strips generics at '<'. Falls back to raw node text for languages where the field IS the type name. The outer field loop now collects from all matching fields before returning, and the result is arena-allocated with a NULL terminator like the surrounding extract_csharp_base_list pattern. Bounds-safe: writes capped at MAX_BASES_MINUS_1 (15) with per-iteration count < out_cap checks; result fits in MAX_BASES (16) including NULL terminator. Real-world impact: 3 → 116 INHERITS edges on the Modelio Java codebase (253 files). |
||
|
Martin Vogel
|
97764d48b4 |
docs(py_lsp): final results table through Round 9
Updates BENCHMARK_PYTHON.md to capture the full trajectory: - Bench: 81% (65 LOC) -> 79% (136 LOC) -> 95% (136 LOC) -> 98% (178 LOC) -> 100% (178 LOC, 52/52) - 22 stress-test patterns probed; 19 hard-asserted, 3 documented gaps - Documented gaps: function-as-dict-value, match sequence pattern, lambda parameter inference The hard-asserted patterns include the full LSP surface a user would expect: NamedTuple, TypedDict, Protocol, ABC, with-as, except-as, post-early-return narrowing, tuple unpacking, dict.items comprehension, slice typing, decorator factories, property setters, Self chains, diamond MRO, recursive types, closures, generator delegation, async-gen for, long fluent chains. |
||
|
Martin Vogel
|
4ccd08c6de |
feat(py_lsp): TypedDict subscript + early-return narrowing + match-sequence
Round 9 closes 2 of the 3 remaining stress gaps from Round 8: TypedDict literal-key subscript: - d["foo"] where d: TD with class-body annotation `foo: Foo` returns the field's type. Routes through the existing class-body-annotation field registry (Round 4): TypedDict subclasses already register their fields the same way. The subscript branch in py_eval_expr_type now detects NAMED container + literal-string key and looks up the field via py_lookup_field. Strips both single and double quotes. Early-return narrowing (post-block flow): - if not isinstance(x, T): return / raise / break / continue — applies the *positive* narrow to the enclosing scope after the if-block when the consequence terminates. - if x is None: return — narrows x to non-None afterwards. - py_block_terminates detects return/raise/break/continue as the last statement of the body block. Subsequent code in the function sees the narrowed type. Match sequence pattern (tentative): - list_pattern / tuple_pattern handler added: bind capture-pattern elements to the subject's iterable element type; bind splat-pattern rest to list[elem]. Also walks more wrapper levels (case_pattern -> pattern -> _simple_pattern). Test still flags as KNOWN GAP because tree-sitter's pattern-wrapper structure for the bracket-form `[a, b]` in match/case differs in some grammar versions; documented as remaining gap pending grammar inspection. Stress test suite: 22/22 PASS. Hard-asserted patterns: namedtuple, typeddict, protocol, abc, with-as, except-as, isinstance-early-return, tuple-unpack, dict.items, slice, decorator-factory, property-setter, Self chain, diamond MRO, recursive types, nested closure, long method chains, generator delegation, async-gen for. Documented gaps: function-as-dict-value, match sequence pattern, lambda inference. Bench unchanged at 100% / 11.02 ms; total 2920 / 0. |
||
|
Martin Vogel
|
f35745c79f |
feat(py_lsp): with-as, except-as, tuple unpack, slice, container TypeVar special-cases — bench 100%
Round 8. Stress test suite (test_py_lsp_stress.c) added with 22 advanced
patterns; bench fixture grew to 178 lines covering the full surface.
with-as binding via __enter__:
- Look up __enter__ on the context manager's NAMED type. Use its
return type when known; fall back to the manager type itself
(matches the practical case where __enter__ returns self). Handles
tree-sitter Python's wrapping of `X as y` in an as_pattern node
inside with_item, plus the flat field-only shape.
except-as binding:
- Bind exception alias to the caught type. Resolves through
except_clause's as_pattern wrapper and the as_pattern_target
identifier wrapper.
Tuple unpacking:
- Both `a, b = f()` and `for k, v in d.items()` destructure the RHS
element-by-element when the rhs is TUPLE or TEMPLATE("tuple", ...).
*rest binds the trailing slice as list[elem]. Shared helper
py_bind_for_target used by for-statement, comprehension for_in_clause,
and assignment LHS.
Slice subscript:
- `lst[1:3]` returns the same TEMPLATE container — list[T][1:3] is
list[T], dict[K, V][1:3] is dict[K, V] (rare but consistent).
Container TypeVar substitution:
- Without a constraint solver we can't generally substitute K, V into
registered method return types. Hand-rolled the most-impactful
cases: dict.items/keys/values/get/pop/copy, list.copy/__iter__/pop,
set.copy. These return properly parameterized ItemsView[K, V] /
KeysView[K] / ValuesView[V] / Optional[V] / etc. so subsequent
iteration and unpacking yield typed elements.
ItemsView / KeysView / ValuesView / dict_items / dict_keys / dict_values
added to py_iterable_element_type. ItemsView[K, V] iterates as
tuple[K, V], unpacked correctly by py_bind_for_target.
Bench: 178 LOC, 52 calls, **52 resolved (100%)**, 11.59 ms (ASan + UBSan).
22 new stress tests, 8 of 10 documented gaps closed in this round.
Remaining gaps documented inline (typeddict_subscript, post-early-return
narrowing, function-as-dict-value, match sequence pattern, lambda
inference). All 2898 prior tests stay green; total 2920 / 0.
|
||
|
vinay-veerappa
|
048f1b46c9 |
feat(lang): add Pine Script support via vendored kvarenzn/tree-sitter-pine (#273)
Adds Pine Script (TradingView's indicator/strategy DSL) as a first-class language using the project's existing language-add pipeline: - scripts/new-languages.json: register pine (display "PineScript", ts_func tree_sitter_pine, repo kvarenzn/tree-sitter-pine, extension .pine, has_scanner=true, module_root source_file) - internal/cbm/vendored/grammars/pine/: vendored from https://github.com/kvarenzn/tree-sitter-pine via scripts/vendor-grammar.sh — parser.c, scanner.c, tree_sitter/ headers. License is ISC (declared in upstream package.json; written into vendored LICENSE for clarity since upstream has no LICENSE file) - internal/cbm/grammar_pine.c: 4-line wrapper generated by scripts/generate-lang-code.py wrappers - internal/cbm/cbm.h: CBM_LANG_PINE before CBM_LANG_COUNT - internal/cbm/lang_specs.c: 7 node-type arrays (func / class / module / call / var / branch / assign) cross-checked against kvarenzn's grammar.js, plus the spec table entry - src/discover/language.c: .pine -> CBM_LANG_PINE extension dispatch and "PineScript" name scripts/audit-grammar-security.sh on the vendored grammar: no dangerous includes (only string/stdint/stdbool/stdlib), no system / popen / exec / socket / mmap / dlopen calls, no constructor/destructor attributes, no inline asm. The 148-line scanner.c only handles INDENT/DEDENT/NEWLINE tokens via the standard tree-sitter lexer API. Distilled from #273 rather than vendored verbatim because the original PR's parser.c was a plain 46k-line drop without provenance, no LICENSE attribution, and bundled an unrelated Makefile change (scripts/embed-frontend.sh -> sh scripts/embed-frontend.sh) that didn't belong to a Pine Script feature. The node-type choices in the lang spec match the original PR's selections, all of which were verified to be real grammar.js rules. Full suite: 2840 passed, 0 failed under ASan + UBSan. Closes #273. Co-authored-by: vinay-veerappa <vinay.veerappa@gmail.com> |
||
|
Martin Vogel
|
3ec56b7bff | docs(benchmark): refresh PHP-LSP numbers post-Phase-4t/v | ||
|
Martin Vogel
|
090fdb8935 |
feat(php-lsp): cover Phases 4t/4v/5d + 25 more tests, 100→150 tests
- Generic substitution end-to-end (T<->concrete-arg) for any class with
@template + @return T, including ancestor walks via belt-and-suspenders
AST-extracted base_clause / class_interface_clause.
- Block-form `namespace App { class Foo {} }` — robust child traversal
through declaration_list / compound_statement variants; also collects
`use` clauses from inside the block.
- Declared method return types re-parsed via php_parse_type_node so
`function f(): \Generator<int, User>` becomes a proper TEMPLATE.
- Stdlib expansion: Symfony Console (SymfonyStyle, In/OutputInterface),
Doctrine ORM (EntityManagerInterface, QueryBuilder, Query, Repository),
Guzzle (Client, ClientInterface), Twig (Environment, TemplateWrapper),
Eloquent Model magic-static methods (where/whereIn/orderBy/...).
- Pipeline bridge unchanged; resolution all happens at LSP level.
Test coverage: 125 -> 150 tests; total 2938 -> 2963 / 0 failed.
|
||
|
Martin Vogel
|
a3a64fc009 |
feat(php-lsp): generic substitution + 25 more features toward LSP parity
Phase 4m–w in one push, plus 25 new tests (100 → 125, all green).
New capabilities:
- Generic @template substitution:
@template T on a class records T as a type-param;
@return T on a method gets parsed as NAMED("T");
when receiver is TEMPLATE("Container", [User]), method-call return
types are substituted T → User so chains like
`$container->get()->name()` resolve through to the element type.
- Closure use(...) capture: typed variables in `use ($a, $b)` clauses
of `function () use (...) {}` are inherited from the parent scope.
- Multi-conjunction narrowing: `if ($x instanceof A && $y instanceof B)`
narrows BOTH $x and $y in the if-body. Up to 8 simultaneous narrowings.
- is_a($x, Foo::class) narrowing — both the ::class and string forms.
- Subscript inference: $arr[$k] on `array<T>` returns T;
on `array<K, V>` returns V;
on a NAMED ArrayAccess class consults registered offsetGet.
- Enum case access: Suit::Hearts evaluates to NAMED(Suit) so
Suit::Hearts->label() resolves to Suit::label.
- @extends ParentClass<X>: appends ParentClass to embedded_types so
the inheritance walk picks up methods from generic ancestors.
- PHPDoc @return on methods with no declared return type updates
the registry signature.
- foreach key=>value: handles tree-sitter-php's `pair` node so
`foreach ($users as $idx => $user)` binds $user to the element type.
- Class-level @template-extends, @template-implements parsed without
crashing.
- Don't override TEMPLATE/NAMED bindings with `null` literal —
preserves @var-typed variables across `$x = null` initialization.
Stdlib expansion (Phase 5d):
- Symfony Console: SymfonyStyle, InputInterface, OutputInterface
- Doctrine ORM: EntityManagerInterface, QueryBuilder chain, Query,
EntityRepository
- Guzzle HTTP: Client and ClientInterface (returning PSR-7 responses)
- Twig: Environment, TemplateWrapper
- Eloquent Model: forwarded static methods (where, whereIn, etc.)
so `User::where()->first()` chains resolve immediately.
Test coverage: 100 -> 125 tests; total 2913 -> 2938 / 0 failed.
|
||
|
Martin Vogel
|
dd3a816163 |
feat(py_lsp): builtin/template attribute dispatch + wrappers + dedup
Major parity push. Bench 79% -> 98% (51/52 calls resolved) on a 178-line
fixture exercising the full parity surface.
Annotation wrapping:
- Type-wrapper annotations now unwrap to the inner type:
ClassVar[T], Final[T], InitVar[T], ReadOnly[T], Required[T],
NotRequired[T], Annotated[T, ...] (drop metadata),
Mapped[T] / WriteOnlyMapped[T] / DynamicMapped[T] (SQLAlchemy 2.0),
Type[T] / type[T]. Same logic in both py_resolve_annotation
(runtime, scope-aware) and py_parse_type_text_qn (registry-build,
scope-free).
Builtin / template attribute dispatch:
- obj_type=BUILTIN("str"|"int"|"list"|"dict"|...) now resolves
attribute access via "builtins.<typename>" in the typeshed-derived
registry. obj.split(...) on str, list.append(), dict.get() etc all
emit edges with strategy=lsp_builtin_method.
- obj_type=TEMPLATE("dict"|"list"|...) routes the same way; receiver
is the unparameterized container's typeshed entry.
Bare-name builtin fallback:
- Bare-name calls (range, len, list, dict, str, int, print, etc.) fall
through to a "builtins" module symbol lookup, then to a builtins
type lookup for class constructors. Strategies: lsp_builtin /
lsp_builtin_constructor.
Emission deduplication:
- py_emit_resolved_call dedupes by (caller_qn, callee_qn). When two
resolution paths converge on the same edge (expression typing then
call emission, or attribute walk through wrappers), keep one entry
with the higher confidence. Previously the same call could emit
twice — the bench showed >100% ratios as evidence.
Bench fixture expanded from 136 to 178 lines:
- Adds @property access chains, Mapped[Annotated[int, ...]], Final
fields, generator iteration with method dispatch on element type,
multi-class match/case, dict comprehension over .values(), instance
attribute typing through super().__init__, classmethod chained with
Self return.
All 2898 prior tests stay green; bench at 98% in 11.79 ms (ASan +
UBSan).
|
||
|
Martin Vogel
|
fd81dffc8f |
feat(py_lsp): decorator flags + verifications for generators / dataclasses / properties
Round 6: CBMRegisteredFunc gains a `flags` bitfield (CBMFuncFlags) and a `decorator_qns` NULL-terminated list. py_register_def populates them from CBMDefinition.decorators by short-name match — so `@property`, `@functools.cache`, `@abc.abstractmethod`, `@typing.overload`, and `@final` all light up regardless of qualified vs bare reference. The flags aren't consumed by resolution yet (still on roadmap: emit distinct edge for @property attribute access vs method call); they are recorded so future overload selection / property modeling can use them without another schema change. The struct extension is at-tail and existing call sites that memset to zero stay correct; non-Python LSPs see flags=0 / decorator_qns=NULL and behave identically. Three new verification tests prove that generators, dataclasses, and @property already work end-to-end through the Round 4-5 infrastructure without further code changes: - Generator[Foo, ...] iterables in for-loop bind loop var to Foo via py_iterable_element_type's existing TEMPLATE handling. - @dataclass class Point: x: Foo (class-body annotated assignment) is picked up by Round 4's third-pass field registrar; p.x.method() resolves through the field type. - @property def thing(self) -> Foo: c.thing returns Foo because attribute access on NAMED routes through py_func_return_type which yields the getter's annotated return. All 2895 prior tests stay green; 3 new tests pass. Total 2898 / 0. Bench unchanged at 95% (the small bench fixture doesn't exercise the new patterns; will expand in Round 7). |
||
|
Martin Vogel
|
ce54e817b7 |
feat(py_lsp): subscript + super().__init__() + operator dunders
Round 5 parity push:
Subscript value typing — py_eval_expr_type's subscript branch resolves
indexing against the container's TEMPLATE type:
- TEMPLATE("dict"|"Mapping"|"defaultdict"|"OrderedDict"|...)[K] -> V
- TEMPLATE("list"|"set"|"frozenset"|"Sequence"|"Iterable"|...)[i] -> elem
- TEMPLATE("tuple", [A, B, ...])[literal_int] -> args[idx]
- Non-literal index returns UNION of all element types.
This unlocks the `self.cache[k].method()` pattern that's everywhere in
real Python. Also fixes `items[0].method()` for typed lists.
super().__init__(...) constructor edge — super().<attr> already worked
for regular methods; __init__ is special-cased because parent classes
typically don't register __init__ with a useful return type. Strategy
"lsp_super_init" with confidence 0.85.
Operator dunders — a + b looks up __add__ on a's type and uses its
return type. Same for *, -, /, //, %, **, <<, >>, &, |, ^, @. Falls
back to left-operand type when registry has no info (correct for the
common int+int / str+str / list+list cases).
3 new tests; all 2892 prior tests stay green. Bench unchanged at 95%
(fixture doesn't exercise these patterns directly — Round 6 will
expand it).
|
||
|
Martin Vogel
|
0be2ac338c |
docs(py_lsp): final results table + stopping-point doc
BENCHMARK_PYTHON.md ends the parity push with the in-process bench results table (81% -> 79% -> 95% across rounds), an explicit stopping-point list of features that would require building a constraint solver / code-flow engine / plugin system, and an "achievable next steps" list for future iterations that stay below the compiler-rebuild line. The 81% -> 79% jump going from a 65-line fixture to 136 lines is deliberate: the smaller fixture was biased toward the resolver's strengths. The 79% -> 95% jump comes from instance-attribute typing (Round 4) which closed the largest single gap on real-world code shapes. Total: 11 LSP commits on this branch (Phases 0-11 plus Rounds 1-4), 2892 / 0 passing, 95% resolution on the parity fixture. |
||
|
Martin Vogel
|
4b993899a0 |
feat(py_lsp): instance attribute typing — 79% -> 95% resolution
Round 4 / final parity push. Tracks self.x = expr from __init__ and
class-body annotated assignments (`x: Foo`) as registered fields on
the enclosing class type.
py_register_instance_field mutates the registry's CBMRegisteredType
field_names / field_types arrays in place. Re-registering the same
name overwrites; first-seen wins for capture order. Arena-allocated,
not a hot path.
py_lookup_field walks the alias / embedded chain with the same
CBM_LSP_MAX_LOOKUP_DEPTH cap as method lookup, so inherited fields
resolve through MRO without stack-overflow risk on pathological
hierarchies.
Field-fallback wired into py_eval_expr_type's attribute branch:
when method lookup misses, try field lookup. Returns the field's
type so downstream attribute / call resolution chains naturally.
Same fallback applied to UNION (single-match heuristic).
py_process_class now does three passes:
1. Class-body annotated assignments (`x: int = ...`) → fields.
2. __init__ / __post_init__ → self.x = expr fields.
3. All other methods.
This guarantees fields are registered before methods that reference
them, regardless of source order.
Annotated assignment binding fix: py_resolve_annotation is now called
on assignment annotations too, so `self.cache: dict[str, str] = {}`
binds with the proper TEMPLATE type instead of NAMED("dict[str, str]").
Bench result on the 136-line fixture: **95%** (36/38 calls resolved,
8.06 ms). Up from 79%. The 5% gap covers cases that need a full
constraint solver — see Final Stopping Point doc.
2 new tests; all 2890 prior tests stay green. Total 2892 / 0.
|
||
|
Martin Vogel
|
e7dcadc3dd |
feat(py_lsp): match/case narrowing, await passthrough, expanded bench
Round 3 parity push: - match/case (PEP 634): when the case pattern is class_pattern (wrapped in a case_pattern), narrow the match subject identifier to NAMED(class) inside the case body. Push/pop scope per case. Capture sub-patterns bind to UNKNOWN (field-type extraction is a v1.1 task — would require field type info in the registry). - await expr: identity passthrough. async def f() -> T registers with return T, so `await f()` returning T is the resolved-call shape we want without modeling Coroutine[Any, Any, T] explicitly. - Expanded test_py_lsp_bench.c fixture from 65 lines to 136 lines exercising every parity feature: dotted-module walks, dataclass + Self chaining, multi-inheritance with super(), Optional/Union annotations + narrowing (isinstance / is not None / walrus), generic list[T] / dict[K, V] / defaultdict comprehensions, classmethod / staticmethod, match/case dispatch, async/await, common stdlib (logging.getLogger, json.loads, pathlib.Path, functools.lru_cache). Bench result on the new fixture: 38 calls, 30 resolved, 79%, 7.50 ms. Remaining 21% includes patterns where intermediate types lose precision (e.g. super() constructor, dict subscript), instance attribute typing (self.x where x isn't annotated at class scope), and decorator return-type substitution (lru_cache). All 2888 prior tests stay green; 2 new tests + bench expansion. Total 2890 / 0. |
||
|
Martin Vogel
|
fa45e7defc |
feat(py_lsp): narrowing, walrus, comprehension element typing
Round 2/3 parity push. Three connected gap fixes that share an annotation- parser and a generic-aware iterable model. Annotation parser upgrade: - py_resolve_annotation now turns subscripted forms into structured types rather than stripping. `Optional[Foo]` -> UNION(NAMED(Foo), None). `Union[A, B]` -> UNION. `list[Foo]` / `dict[K, V]` / `Iterable[T]` -> TEMPLATE with arg list. `X | Y` (PEP 604) handled at depth zero. - New py_parse_type_text_qn variant: same parser without scope lookup, used by py_register_def at registry-build time. Bare class names are qualified to <module_qn>.<name> so registry lookups match. - py_register_def prefers d->return_type (full subscripted text) over d->return_types[0] when brackets are present — extract_defs's array form strips the subscript, losing element types like Optional[Foo]. Narrowing: - if isinstance(x, T): bind x as NAMED(T) inside the consequence body (block-scoped via push/pop). - if x is not None: strip None from x's UNION inside the consequence; py_match_is_none recognizes "is not" as the single tree-sitter token it actually is, walks through parens / walrus to find the underlying identifier. - Else branch unchanged — narrow's negation (not-T, including-None) is not modeled in v1. Walrus + truthy: - if (x := compute()) is not None: walrus binds x in the enclosing scope (PEP 572) so the body sees x. py_bind_walrus_in walks the cond AST looking for named_expression / assignment_expression nodes. Comprehension + for-loop element typing: - New py_iterable_element_type maps TEMPLATE(list/set/Iterable/...) to arg[0]; TEMPLATE(dict) to key arg; TEMPLATE(tuple, ...) to UNION of args. - Comprehensions push their own scope and bind for_in_clause loop vars to the iterable's element type before walking body / filter clauses. for_statement does the same. UNION attribute access fallback (used when narrowing can't run): - obj_type=UNION & exactly one non-None NAMED member has the attr: emit edge with strategy=lsp_method_union, confidence 0.85. 7 new test_py_lsp.c cases. All 2880 prior tests stay green; 8 new tests land. Bench unchanged (small fixture, no narrowing exercised there). |
||
|
Martin Vogel
|
ee380eb551 |
docs(benchmark): record Phase-4 PHP-LSP outcome on laravel/framework
Edge counts (laravel/framework): - Total edges: 196,979 (baseline) -> 152,614 (-22%) - CALLS edges: ~83,000 (baseline) -> 50,150 (-40%) The drop is from suppressing name-fallback misroutes when the receiver is statically typed to a vendor class that isn't indexed. The Phase-4 capabilities (generic templates, narrowing, property tracking, full parent-walk method dispatch, expanded stdlib) cover most of phpactor's type-resolution surface for in-project receivers. 100 unit tests, all passing. Total project tests: 2913 / 0 failed. |
||
|
Martin Vogel
|
da9f20d55a |
feat(php-lsp): generic templates, negative narrowing, expanded stdlib
Phase 4g+ / 4l / 5b additions toward LSP-server type-resolution parity:
- Generic template parsing in PHPDoc: parse `Collection<User>`,
`array<int, User>`, `iterable<T>`, `list<T>`, etc. Bind variables
to CBM_TYPE_TEMPLATE; method dispatch on a TEMPLATE receiver
resolves against the template's base class.
- Foreach element-type propagation: `foreach ($users as $u)` where
$users: array<User> binds $u to User. Two-arg array<K,V> takes
V. Falls back to Iterator::current() return type for NAMED types
that implement the iterator interface.
- Negative narrowing: `if (!($x instanceof Foo)) return;` (or throw)
narrows $x to Foo for the rest of the enclosing scope.
- Foreach iterable can be an arbitrary expression (variable_name,
member call, function call) — fixed earlier-too-narrow filter.
- PHPDoc @var/@param accept generic syntax: type token now spans
angle brackets so `array<User> $users` parses correctly.
- Stdlib expansion (Laravel + Symfony): full Eloquent Builder
method chain (where/orderBy/limit/get/first/...), Eloquent Model
static factories, Illuminate\Support\Collection chains
(map/filter/sort/...), Symfony HttpFoundation Request/Response,
Carbon date methods, PSR-7 with*() builders.
Test suite: 81 -> 100 tests, all passing. Total: 2913 / 0.
|
||
|
Martin Vogel
|
82d0935fbd |
feat(php-lsp): Phase 4 — depth toward LSP-server parity
Adds the resolver passes that close most of the gap to a real PHP language
server's type inference, while staying in-process and PHP-runtime-free.
New capabilities:
- Type narrowing: instanceof, is_string/int/float/bool/array/object/...
is_callable, is_iterable, is_numeric. Active inside if/elseif bodies
and after assert(...) (sequential narrowing)
- Property type tracking: typed property declarations, constructor
property promotion (public X $foo), and constructor-body inference
($this->bar = $bar where $bar is typed)
- Late-static-binding chain depth: full ancestor walk with cycle
detection for $obj->method() and Class::method() resolution
- Trait flattening: methods from `use Trait;` inside a class body are
re-registered with the using class as receiver; basic `as` aliasing
is honored
- PHPDoc class-level tags: @property TYPE $name binds a virtual field
on the class; @method TYPE name() registers a virtual method
(Eloquent-style dynamic dispatch)
- Match expression result type: union over arm result expressions,
first concrete type wins
- Ternary result type: prefers the consequent, falls back to alternative
- Variadic parameter binding: $args bound to `array`
- `use Vendor\\Helper as H;` two-name form correctly maps H -> Vendor.Helper
- Static-call unindexed fallback: when class is resolved from `use`
but methods unindexed, emit php_static_unindexed for the bridge
- Stdlib expansion: full Eloquent Builder + Model + Collection method
chains, Symfony HttpFoundation Request/Response, Carbon date/time,
PSR-7 with*() builders
Test suite expanded from 16 to 81 tests (2829 -> 2893 total).
|
||
|
Martin Vogel
|
2e042dab0b |
feat(py_lsp): Round 1 parity push (parent-modules, cast, Self, fwd-refs)
Parity push #1 — six gap fixes on top of the 11-phase plan: - Parent-module bindings: `import a.b.c` now binds `a`, `a.b`, `a.b.c` via py_bind_dotted_prefixes. Walks the dotted prefix chain on every import, regardless of import_is_from_style. Plus an attribute-access path on a MODULE that detects submodules: when the registry has any function whose qn starts with `<mod>.<attr>.`, evaluating `mod.attr` yields MODULE("<mod>.<attr>"). This makes `os.path.join(...)` and similar resolve correctly. - typing.cast(T, x) returns NAMED(T) (re-evaluating T as an annotation, so generic subscripts and forward references both resolve). Detects bare `cast(...)` and qualified `typing.cast(...)` forms. - typing.assert_type(x, T) is a no-op at runtime; we type the result as type-of(x). - Forward references as quoted strings: `def f(x: "Foo")` strips quotes and re-resolves the inner annotation. Also handles double-quoted and single-quoted forms. - Self return type substitution: methods declared returning `Self` / `typing.Self` / `typing_extensions.Self` substitute to NAMED(receiver) at call resolution time. Enables fluent / builder pattern chaining to keep resolving methods after each step. - Generic subscript stripping: annotations like `list[Foo]` strip to `list` for v1; `Optional[Foo]` likewise. Container element-type substitution still deferred (Phase 8+). Generator additions: - Allowlist now includes urllib, http, concurrent (top stdlib usage). - `from X import *` re-export following at generation time. Modules like os.path (a star-import shim of posixpath / ntpath / genericpath) now have all forwarded definitions registered under their own QN. Iterates to a fixed point with an 8-step ceiling. typeshed tree expansion is bounded — only modules transitively reachable from the allowlist are pulled in. Stats: 137 modules, 904 classes (2797 methods), 886 free functions (was 114 / 753 / 794). Generated file grows from 20K to ~22K lines. 6 new test_py_lsp.c cases. All 2874 prior tests stay green. |
||
|
Chris Werner Rau
|
2b07bc0c3f |
docs: add AUR package install instructions (#278)
Adds an Arch Linux install section to the README pointing at the community-maintained `codebase-memory-mcp-bin` AUR package, with both yay and paru one-liners. README-only change. |
||
|
dependabot[bot]
|
d27f4aa8c6 |
ci(deps): bump actions/setup-node from 6.3.0 to 6.4.0 (#289)
Patch bump (PR title was rebased to 4.2.0->6.4.0 originally; current diff is 6.3.0 -> 6.4.0). |
||
|
dependabot[bot]
|
d70c71776c |
ci(deps): bump msys2/setup-msys2 from 2.31.0 to 2.31.1 (#268)
Patch bump applied across _build.yml, _smoke.yml, _soak.yml, _test.yml. |
||
|
dependabot[bot]
|
3bf38c962d |
ci(deps): bump github/codeql-action from 4.35.1 to 4.35.2 (#270)
Patch bump for codeql-action across codeql.yml and scorecard.yml. |
||
|
dependabot[bot]
|
5da650ff74 |
ci(deps): bump actions/cache from 5.0.4 to 5.0.5 (#269)
Patch bump. |
||
|
dependabot[bot]
|
f81065d3fb |
ci(deps): bump softprops/action-gh-release from 2.6.1 to 3.0.0 (#267)
v3.0.0 moves the action runtime from Node 20 to Node 24. GitHub-hosted runners (which this repo uses) already support the Node 24 Actions runtime, so the upgrade is transparent. |
||
|
Joseph Voss
|
0deb3b70b1 |
feat(nix): add flake definition for reproducible builds (#265)
Adds flake.nix and flake.lock for Nix users. - Inputs: NixOS/nixpkgs nixpkgs-unstable, pinned by commit + narHash for supply-chain integrity - Targets: aarch64/x86_64 × darwin/linux - buildPhase calls Makefile.cbm cbm directly because scripts/build.sh's `file`-based compiler detection fails on Nix (cc is a wrapper script there); Nix stdenv already guarantees the correct compiler + target arch - Dev shell adds pkg-config + libgit2 so the optional libgit2 acceleration in pass_githistory.c is picked up via pkg-config Opt-in: non-Nix users see no behaviour change. flake.nix and flake.lock are inert unless `nix build` or `nix develop` is invoked. |
||
|
Adam Schulte
|
4e80d2fff7 |
feat(graph): temporal properties on FILE_CHANGES_WITH edges and File nodes (#257)
Adds three changes that landed together in PR #257, with a JSON-escape hardening on the new File node properties so unusual file extensions can't produce malformed output. Mode A — search_graph results expose node properties_json --------------------------------------------------------- emit_search_results in src/mcp/mcp.c now calls enrich_node_properties for each result and parks the parsed yyjson_doc* in an out-array. The two exit paths in handle_search_graph (success and sq_type_error) walk that array and yyjson_doc_free each entry after yy_doc_to_str runs, so zero-copy strings stay alive until serialization completes. Callers no longer have to round-trip through get_code_snippet just to inspect a node's properties. Mode B — Cypher bare edge return surfaces full properties JSON -------------------------------------------------------------- binding_get_virtual in src/cypher/cypher.c returns e->properties_json (or "{}" if absent) when prop is NULL, instead of "". `RETURN r` on an edge variable is now actually useful for inspecting timestamps, weights, etc. without naming each property up front. Temporal data on FILE_CHANGES_WITH edges and File nodes ------------------------------------------------------- cbm_change_coupling_t gains last_co_change. cbm_commit_files_t and the internal commit_t gain a timestamp field. Both git-log paths (libgit2 and popen --pretty=format:COMMIT:%H:%ct) populate it. cbm_compute_change_coupling carries a parallel pair_timestamps hash table and writes the max timestamp to last_co_change. The edge properties JSON now includes "last_co_change":<unix-epoch>. cbm_pipeline_githistory_compute additionally aggregates per-file change_count and last_modified into a new cbm_file_temporal_t array exposed via result->file_temporal. cbm_pipeline_githistory_apply upserts those values onto matching File nodes so callers can run hotspot queries (most-changed, recently-changed) directly. Both the sequential cbm_pipeline_pass_githistory and the parallel run_githistory in pipeline.c free file_temporal and trigger apply when either couplings or temporal entries are present. The extension stored on the File node properties is now JSON-escaped via cbm_json_escape before snprintf'ing into the props blob. The original PR concatenated the raw extension; a hypothetical path with a quote or backslash in the extension would have produced malformed JSON. Practical risk was tiny (real extensions never contain those characters) but the escape is cheap and correct. Tests ----- - githistory_coupling_carries_last_co_change pins last_co_change as the max of the per-pair commit timestamps even when an unrelated pair sits between them in the input order. - cypher_bare_edge_return_exposes_properties_json regresses Mode B by asserting `RETURN r` on an HTTP_CALLS edge surfaces its url_path in the response. - tool_search_graph_includes_node_properties regresses Mode A by asserting search_graph's response contains the signature/return_type/ is_exported keys from a HandleRequest node's properties_json. Existing githistory tests had their cbm_commit_files_t initializers extended with a zero timestamp (backward-compatible — last_co_change falls back to 0 for callers that don't supply timestamps). Cherry-picked from #257; renamed the timestamp field types to long long for portability across 32-/64-bit time_t platforms, used the existing cbm_json_escape utility instead of raw concatenation for the extension, and added explicit tests for all three changes. Closes #257. Co-authored-by: Adam Schulte <adamschulte148@gmail.com> |
||
|
Peter Cox
|
cee318bb7f |
fix(incremental): preserve mode-skipped files and their hash rows (#251)
cbm_pipeline_run_incremental was destructively overwriting a project's graph when the index mode was downgraded. find_deleted_files classified ANY stored file absent from the current discovery list as deleted and purged its nodes via cbm_gbuf_delete_by_file. In a full → fast mode downgrade, directories matching FAST_SKIP_DIRS (tools, scripts, bin, build, docs, __tests__, migrations, ...) are excluded from discovery but still exist on disk — so they were getting silently destroyed. Real-world incident, 2026-04-13: the Skyline graph collapsed from 2819 nodes (full-mode nightly) to 2303 mid-day when a concurrent fast-mode reindex obliterated packages/mcp/src/tools/ (~500 nodes / 18 files). Fix: - find_deleted_files now stat()s each stored-but-missing file. Files that exist on disk are mode-skipped (nodes AND hash rows preserved); only ENOENT/ENOTDIR results purge. Other errnos (EACCES, EIO, ELOOP, transient NFS, ...) preserve. NULL repo_path preserves. snprintf truncation preserves. strdup OOM mid-record drops the entry rather than persist a NULL rel_path that would silently fail upsert. stat() not lstat() is intentional — the indexer follows symlinks during discovery so a stale symlink has no source to parse. - Mode-skipped hash rows are threaded through dump_and_persist → persist_hashes and re-upserted into the rebuilt DB. Without this, the next reindex's stored hashes wouldn't include the file → the deletion-detection contract breaks → orphaned graph nodes survive forever once the file is eventually deleted. - persist_hashes now checks the upsert return code on both the current-files loop and the new mode_skipped loop and logs a warning with scope/rel_path/rc on failure. Partial-failure policy: a single bad row no longer silently drops, but does not abort the whole reindex either — partial preservation beats total loss. Tests: - incremental_fast_preserves_mode_skipped_tools_dir walks the failure end-to-end across four phases: full index, fast reindex preserves the tools dir, mutate-and-fast-reindex exercises dump_and_persist's hash carry (NOT the noop fast-path that would hide bugs there), then delete-on-disk full reindex confirms the deletion-detection contract still holds. - store_file_hash_upsert_rejects_null_required_fields pins the API contract that cbm_store_upsert_file_hash returns CBM_STORE_ERR on NULL required fields, so a future schema change relaxing any NOT NULL is caught at the producer layer rather than letting the new warning go silent. Conflict resolution against current main: the dump_and_persist signature now takes both mode_skipped/mode_skipped_count (this PR) and repo_path (added on main for cbm_artifact_export); the call site passes cbm_pipeline_repo_path(p) alongside the mode_skipped pair. Full suite: 2837 passed, 0 failed under ASan + UBSan + LeakSanitizer. Closes #251. Co-authored-by: Peter Cox <info@petercox.ie> |
||
|
Martin Vogel
|
a8c9ace4a5 |
feat(py_lsp): Phase 11 benchmark methodology + integration bench
Final phase of Python LSP integration. Adds:
- docs/BENCHMARK_PYTHON.md — methodology document covering target
metrics (per-file LSP overhead < 10 ms, resolution ratio ≥ 40% on
application code, ≥ 70% with stdlib, < 5% spot-check false positive
rate), the declared accepted-failure-class ceiling (metaclass
__call__, __getattr__/__getattribute__, dynamic setattr, runtime
importlib, monkey-patching, eval/exec, descriptors beyond the
standard property/classmethod/staticmethod, dynamic type() classes),
and instructions for running scripts/benchmark-index.sh against the
largest Python repos in datadice (falkemedia: 232 files, api-hub:
188 files).
- tests/test_py_lsp_bench.c — in-process benchmark wired into
scripts/test.sh. Runs cbm_extract_file on a 65-line Python fixture
covering imports, dataclasses, classmethods, single-inheritance with
super(), typed parameters, attribute chains, and stdlib calls.
Asserts soft floor (resolved >= calls/4) and hard time budget
(< 50 ms under sanitizers). Initial run reports:
bench: 65 lines, 16 calls, 13 resolved (81%), 2.45 ms
81% beats the 40% application-code target and the 70%
application-plus-stdlib target. 2.45 ms under ASan + UBSan is well
inside the 10 ms (no-sanitizer) production budget.
This wraps the 11-phase plan documented in PYTHON_LSP_PLAN.md. Total
test count: 2874 / 0 (100% pass). Repository-level benchmarks via
scripts/benchmark-index.sh remain a manual operation against a fresh
production binary; methodology is now in place for when that runs.
|
||
|
Dustin Obrecht
|
2d00c4a337 |
fix: GitHub Actions shell injection in _build.yml (#249)
Replace direct `${{ ... }}` interpolation in `run:` steps of `.github/workflows/_build.yml` with `env:` blocks per GitHub's hardening guide. Every workflow input (`inputs.version`) and matrix value (`matrix.cc`, `matrix.cxx`, `matrix.goos`, `matrix.goarch`, `matrix.arch`) now flows into the shell through a named environment variable rather than being textually spliced into the script.
Pattern applied uniformly across the 6 flagged steps plus the archive steps that used `${{ matrix.* }}` in tar filenames. The values become opaque shell variables on expansion, so attacker input becomes a literal argv entry rather than interpreted shell code. The if/else form on `$VERSION` avoids word-splitting pitfalls of conditional arg construction.
semgrep `run-shell-injection` findings: 6 → 0. Behaviour preserved — same build.sh invocations, same artefact names, same conditional version handling.
Reference: https://securitylab.github.com/research/github-actions-untrusted-input/
Closes #247.
|