build_grep_cmd's POSIX scoped path uses xargs to pass indexed files
to grep. When xargs supplies a single file, BSD/GNU grep omit the
filename from output by default (only added with multiple files),
yielding 'linenum:content' instead of the 'path:linenum:content'
that collect_grep_matches requires — matches were silently discarded
and search_code returned zero results.
Add -H so grep always prefixes the filename regardless of file count.
Harmless for the common multi-file case (output is unchanged), fixes
the single-indexed-file edge case exposed by the search_code_multi_word
test fixture.
Add a C# language LSP resolver mirroring the existing Go/C/PHP/Python
modules: type evaluation, using-directive resolution, method lookup,
and a corpus-seeded .NET BCL stdlib data table.
- internal/cbm/lsp/cs_lsp.{c,h}: resolver implementation
- internal/cbm/lsp/generated/cs_stdlib_data.c: BCL type/method tables
- wire CBM_LANG_CSHARP dispatch in cbm.c and lsp_all.c
- register cs_lsp + cs_lsp_bench suites in the test harness
- tests/test_cs_lsp.c, tests/test_cs_lsp_bench.c
(cherry picked from commit 8f677c67a0c9d7829cd6ac27393de2d0aaaac534)
cbm_store_checkpoint() invoked SQLITE_CHECKPOINT_TRUNCATE, the most
aggressive mode. When two cbm-mcp processes share a cache dir, one
process's TRUNCATE can shrink files while another has them mmap'd,
raising SIGBUS on macOS. PASSIVE never blocks readers and never
ftruncate()s either file; SQLite still autocheckpoints in PASSIVE
mode at 1000-page boundaries, so reclamation is unaffected for
single-process users.
Recommended by SQLite docs for shared databases:
https://www.sqlite.org/pragma.html#pragma_wal_checkpoint
Hard-coded `PRAGMA mmap_size = 67108864` in configure_pragmas() left
no path for users running multiple cbm-mcp instances against the
same cache to opt out of memory-mapped I/O. On macOS, when one
instance's checkpoint or reindex truncates the DB file under
another instance's live mmap, accessing the now-missing pages
raises SIGBUS, taking the process down.
Setting CBM_SQLITE_MMAP_SIZE=0 reverts to read()/pread() I/O,
which returns recoverable SQLITE_IOERR instead of crashing the
process.
- Default unchanged (67108864 / 64 MB). No behavior change for
single-instance users.
- Malformed values (non-numeric, partial-numeric) fall back to
the default rather than failing the store open.
- Negative values clamp to 0.
- New tests: tests/test_store_pragmas.c covers all five resolver
paths plus an integration smoke that opens a file-backed store
with mmap disabled.
Empirical evidence: 9 SIGBUS crash reports collected on macOS
arm64 v0.6.0 in a 14-hour window, all signature 'cluster_pagein
past EOF' with stacks bottoming in SQLite btree code under the
watcher thread's incremental-index pipeline.
Replace the fragile NOT IN list of individual characters with a range-based BETWEEN 'A' AND 'Z' check for validating project root paths.
The previous query only excluded letters A–H, leaving paths starting with I–Z silently invalid.
In Windows, all the alphabet can be used.
A Windows drive starting with letters I-Z breaks the saving of the database.
The Windows search path landed in 82a9052 wraps shell args in cmd.exe-
level "powershell -Command \"...'%s'...\"". PowerShell's single quotes
hold the inner '%s' interpolation, but a literal " inside the user-
supplied value can close the cmd.exe outer quote. With ' ; | & $ `
already blocked, that wouldn't reach RCE on its own — but < and > were
unblocked, so a quote-break followed by cmd.exe redirection (e.g.
*">C:\evil.txt") would expose a file-write primitive.
Block " < > unconditionally. The validator's contract is "safe inside
single quotes for shell interpolation"; on POSIX these aren't strictly
necessary (single quotes hold), but on Windows the cmd.exe→powershell
wrapping makes them load-bearing, and unconditional blocking keeps the
validator simple — no ifdef-laddered policy and no surprises if a future
caller invokes it from a different shell context.
Three tests added to tests/test_security.c covering each new char.
The .m extension was missing from EXT_TABLE, causing detect_file_language
to return CBM_LANG_COUNT (Unknown) before the content-based disambiguation
logic (cbm_disambiguate_m) could ever run. This made .m files invisible
to the indexer.
Add {".m", CBM_LANG_MATLAB} to EXT_TABLE so the initial lookup succeeds,
allowing the subsequent cbm_disambiguate_m call to correctly determine
whether the file is MATLAB, Objective-C, or Magma.
Fixes#296
When pattern contains spaces and is not already a regex, convert to a
regex with '.*' between words: "foo bar baz" → "foo.*bar.*baz".
This matches all terms in order on a line instead of requiring the
exact multi-word substring.
Metacharacters in user input are escaped so the conversion is safe
for arbitrary text.
Includes test: search_code_multi_word verifies that a two-word query
matches a line containing both words non-contiguously.
98d5cac silently dropped index cells that exceed a single SQLite page,
which is the correct semantic (the index B-tree has no overflow page
mechanism, unlike the record B-tree fixed in #175). But silent drops
make the failure mode invisible — a caller hitting deeply nested
namespaces or very long file paths gets a partial index with no
indication anything was skipped.
Add an fprintf(stderr) at the skip site reporting the cell length and
its index, matching the existing logging style in this file (lines 311,
1305) so the dropped cells are observable for diagnosis. Consistent
with the cbm_write_db: prefix used by surrounding warnings.
write_index_btree() flushed the page buffer whenever the next cell
didn't fit, on the assumption that a freshly-initialised page would
always accept any cell. That invariant fails for index cells whose
payload (qualified_name, file_path, etc.) is larger than a full
SQLite page — after the flush, pb_cell_fits() is still false but the
code calls pb_add_cell() anyway. The subsequent content_offset
underflow corrupts the page header and the next write triggers
SIGBUS on large repos (observed on a Ruby codebase with ~215K nodes).
Re-check pb_cell_fits() after the flush and continue past cells
that still don't fit. Index entries whose keys exceed a full page
can't be stored and are not expected to survive the writer anyway —
the rest of the index is correctly preserved and the indexer no
longer crashes. Record overflow pages (#175) already covers the
record-btree side of this same shape of bug.
Flat BM25 queries of the form:
SELECT ... FROM nodes_fts JOIN nodes WHERE MATCH ? AND project=? ORDER BY bm25() LIMIT N
block FTS5 WAND/MaxScore early-exit — the outer JOIN+WHERE is invisible to
the FTS5 planner, so it scores every matching document before any filter fires.
On a large codebase with 100K+ matches this causes 2–16 minute queries.
Fix: two-step subquery. The inner FTS5-only query:
SELECT rowid, bm25(nodes_fts) FROM nodes_fts WHERE MATCH ? ORDER BY bm25() LIMIT 2000
can early-terminate because no outer predicate blocks it. The outer query
then joins and filters at most BM25_INNER_LIMIT (2000) candidates.
The count query uses the identical inner-limit subquery, so it benefits too.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Make project a required CLI argument instead of a hardcoded name,
and remove internal query strings used during development testing.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three compounding bugs caused 1.5–8.5s latency on name_pattern= searches against
large projects (216K nodes), now reduced to ~0ms query time (cold-start dominates):
Fix 1 — regex compiled once per statement, not once per row
sqlite_regexp / sqlite_iregexp now use sqlite3_get_auxdata / sqlite3_set_auxdata
to cache the compiled cbm_regex_t for the lifetime of the statement. Previously
cbm_regcomp + cbm_regfree ran for every row scanned.
Fix 2 — LIKE pre-filter cuts rows reaching the regex
Wire cbm_extract_like_hints (already implemented but dead) into search_where_basic
via a new where_add_like_hints helper. For .*Controller.* this prepends
n.name LIKE '%Controller%', letting the idx_nodes_name index satisfy the LIKE
clause first and passing only matching rows to iregexp(). Added search_like_pool_t
to manage the malloc'd LIKE strings across both statement executions.
ST_SEARCH_MAX_BINDS raised 16 → 32 to accommodate extra bind slots.
Fix 3 — count query no longer runs per-row edge subqueries
The count SQL previously wrapped the full SELECT (which includes two correlated
subqueries for in_deg / out_deg) in SELECT COUNT(*) FROM (...), executing those
edge counts for every matching row even though the count needs none of that.
Non-degree-filter path now uses SELECT COUNT(*) FROM nodes n WHERE <same WHERE>,
which has no per-row subqueries. Degree-filter path retains the wrapped form
since it needs those columns for the filter.
Benchmark on home-ubuntu-dev-sis (216K nodes, 509MB DB):
Query BEFORE AFTER speedup
name_pattern=.*Controller.* 3099ms 508ms 6×
name_pattern=.*Service.* 2006ms 506ms 4×
name_pattern=.*Repository.* 2006ms 508ms 4×
name_pattern=specificFuncName 1506ms 507ms 3×
label=Method + name_pattern=.*get.* 8509ms 509ms 17×
name_pattern=.*Approve.* 1506ms 507ms 3×
name_pattern=.*authorize.* 1506ms 509ms 3×
The ~500ms floor is cold-start I/O (opening a 509MB file from disk). In the
long-running MCP server process the warm-cache query time is sub-millisecond.
All store search tests pass including pagination, degree filter, and extract_like_hints.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
handle_trace_call_path only called cbm_store_find_nodes_by_name, which
queries on the bare `name` column. When a caller passed a fully
qualified identifier (e.g. "<project>.module.submodule.func") — which
the not-found hint explicitly recommends ("Use search_graph(...) to
find the exact name, then pass it to trace_path") — the lookup missed
every time. The documented "fully qualify for precise match" UX
inverted what actually worked.
After the bare lookup misses, fall back to cbm_store_find_node_by_qn
(same scan_node populator, same string-ownership semantics), wrap the
single result into the same nodes[] array the downstream BFS expects,
and let cbm_store_free_nodes handle cleanup unchanged. Guard the malloc
against OOM and free the qn_node strdups on the failure path so the
fallback never leaks.
Co-authored-by: Justin Wiley <jwiley@nvidia.com>
`is_project_db_file` excluded any filename starting with "tmp-", but
`cbm_project_name_from_path` legitimately produces "tmp-..." project
names for source roots under /tmp/ (the in-tree fixture confirms:
"/tmp/bench/erlang/lib/stdlib/src" → "tmp-bench-erlang-lib-stdlib-src").
Any /tmp-rooted project was silently filtered from list_projects and
from the available_projects hint inside build_project_list_error, even
though the same project was fully queryable via every other tool.
Drop the tmp- prefix exclusion; SQLite journal/WAL sidecars end with
-journal/-wal/-shm and don't share the .db suffix the existing length
check already requires, so removing the prefix filter is safe.
While here, dedupe the same exclusion logic from collect_db_project_names
by routing it through is_project_db_file (forward-declared above) so
the filter lives in one place and the two paths can't drift again.
Co-authored-by: Justin Wiley <jwiley@nvidia.com>
Two visualization gaps in the cross-repo graph view:
1. Fixed LAYOUT_GALAXY_SPACING=600 buried satellite galaxies inside the
primary cluster on any non-trivial project — a 1000-node primary
cluster has bounding radius ~1500. Added layout_radius() (max distance
from origin across all node positions) and compute the per-satellite
distance as primary_radius + sat_radius + LAYOUT_GALAXY_PAD, with the
old constant retained as a lower bound for tiny projects.
2. cross_edges was hard-coded to an empty array (TODO placeholder), so
inter-galaxy CROSS_* edges never rendered even when the matcher had
produced them. Populate from a join on the source store's CROSS_*
edges → Route node qualified_name, then resolve the canonical QN in
the linked store to get its node id. The Route QN is the cross-repo
matching contract; properties.target_function isn't unique.
Required keeping both `store` and `lp_store` open through the linked-
projects loop instead of closing them right after layout compute.
Added cbm_store_close to all early-error paths to balance the lifetime.
Frontend side:
- EdgeLines accepts an optional targetNodes prop so a single component
can render edges with source in one node array and target in another
(offset-adjusted satellite nodes).
- GraphScene renders an EdgeLines layer per linked project for its
cross_edges, using primary nodes as source and offsetNodes as target.
- GraphTab folds linked-project labels and edge types into the filter
init / enableAll / filteredData paths so cross_edges respect the
current filter state.
- Edge color palette extended with GRPC/GRAPHQL/TRPC/CROSS_* tones.
Co-authored-by: sponger94 <45746997+sponger94@users.noreply.github.com>
extract_constructor_type only accepted call-form RHS when the function
name's first character was uppercase. Misses two common typed-stub
factory patterns:
Go: client := pb.NewPromoCodeClient(conn)
Java: stub := PromoCodeServiceGrpc.newBlockingStub(channel)
Both have a lower-cased qualifier prefix, so the first character of the
full text is `pb`/`promoCodeServiceGrpc` rather than the factory name.
Match the last `.`-segment instead and accept it when it begins with
`New`/`new` and ends with `Client` or `Stub`. Returns the full
qualified call so downstream passes keep the package context.
Stays scoped — only the new factory shape is added; the uppercase-first-
char path for `Foo()`-style direct calls is unchanged.
Co-authored-by: sponger94 <45746997+sponger94@users.noreply.github.com>
Three C# extractor gaps that left modern .NET classes partially invisible
in the graph:
1. lang_specs.c CSHARP entry had `empty_types` for field_node_types, so
walk_defs never descended into field_declaration / property_declaration
nodes. Add cs_field_types and wire it in. Without this, no C# fields
or properties were ever pushed as Field defs.
2. extract_class_fields assumed the field's "type" is on the immediate
child via ts_node_child_by_field_name. Tree-sitter-c_sharp wraps fields
as field_declaration > variable_declaration(type, variable_declarator),
one level deeper than Java/Go/Rust. Detect the nested shape by walking
named children for variable_declaration and resolve type + name from
inside it. Direct shape continues unchanged.
3. C# 12 primary constructors (`class Foo(IBar bar) : Base { ... }`)
declare implicit captured fields on the class line. Tree-sitter wraps
them inside the hidden _class_declaration_initializer node, so the
`parameters` field on class_declaration may not always resolve
directly; iterate top-level children for parameter_list as a fallback.
Emit each primary-ctor parameter as a Field def with parent_class set.
Effect: dependency-injected fields on .NET 8+/9+ controllers and any C#
auto-property (`public IFoo Foo { get; }`) are now reachable as Field
nodes in the graph.
Co-authored-by: sponger94 <45746997+sponger94@users.noreply.github.com>
Per-file LSP (run from cbm_extract_file) only sees one file's defs, so
callees whose receiver type comes from an imported module stayed
unresolved — even though Go/C/Py/TS already had cross-file entry points
(cbm_run_X_lsp_cross). Those entry points existed but no pipeline pass
ever invoked them. PHP didn't even have a cross-file entry point.
This change lands the missing pieces:
1. cbm_run_php_lsp_cross + cbm_batch_php_lsp_cross — PHP cross-file
resolver mirroring py_lsp_cross / ts_lsp_cross. Reuses PHPLSPContext,
php_lsp_init, php_lsp_add_use, php_lsp_process_file. Handles PHP-
specific labels (Class / Interface / Trait / Enum / Type) and uses
def_module_qn so cross-file return types qualify against the def's
own module rather than the importing module's.
2. pass_lsp_cross.c — new pipeline pass. For every cached file result,
builds a project-wide CBMLSPDef[] (CBMDefinition → CBMLSPDef
converter), per-file import map (from gbuf IMPORTS edges), then
dispatches to the matching cbm_run_X_lsp_cross. Output strings get
copied into the per-file arena (with O(N²) memory bug avoided by
running each file inside a fresh scratch arena that gets destroyed
after the LSP call — the LSP allocates its registry + stdlib +
project defs in scratch, not in the caller's persistent arena).
3. Wired into both pipelines:
- sequential: definitions → k8s → lsp_cross → calls → ...
- parallel: parallel_extract → registry_build → lsp_cross →
parallel_resolve → ...
4. Updated tests/test_parallel.c run_parallel helper to mirror the new
parallel ordering so existing cross-file Python/PHP tests now exercise
pass_lsp_cross.
Languages covered: Go, C/C++/CUDA, Python, TS/JS/JSX/TSX, PHP. PHP cross
file is new. The other four had functions but no pipeline integration
until this change.
Verified on a 1670-file FastAPI index: pass_lsp_cross processes 1050
Python files (525 non-LSP files skipped), collects 4070 project defs in
9.2s, and parallel_resolve subsequently reports lsp_overrides=568 — the
LSP-strategy attribution gain over per-file alone.
Per-LSP emit functions dedup by (caller_qn, callee_qn) within their own
run; pxc_append_results dedups again at append time so cross-file
entries duplicating per-file output get dropped instead of inflating
result->resolved_calls.
Per-file py_lsp emits resolved_calls.callee_qn as the raw import-module
path (e.g. `greeter.Greeter` from `from greeter import Greeter`) rather
than the project-qualified QN the gbuf stores (`<project>.greeter.Greeter`).
Before this change, the LSP match succeeded (lsp_overrides counter
incremented) but the downstream cbm_gbuf_find_by_qn lookup missed
silently and the edge was dropped — the failure mode the new
lsp_overrides telemetry was designed to surface.
The LSP layer can't tell in-project imports (qualify) from external
imports (don't qualify, e.g. `os.path`) without consulting the gbuf,
which is built downstream. So normalise at the consumer instead: try
the LSP-emitted QN as-is first; on miss, retry with
`<project>.<callee_qn>`. If that also misses, drop the edge — same as
before, target is external/unindexed.
New helper cbm_pipeline_lsp_target_node in lsp_resolve.h is shared by
both pipelines (sequential pass_calls.c and parallel pass_parallel.c)
so they continue to admit identical sets of LSP overrides. The helper
also canonicalises res.qualified_name to the gbuf node's QN, so
downstream edge property serialisation shows the project-qualified
form even when fallback resolution kicked in.
Cross-file regression test parallel_python_lsp_override_cross_file_-
emits_lsp_strategy_edges pins the two-file Greeter scenario that
originally exposed the bug.
The sequential pass_calls and the parallel cbm_parallel_extract pipelines
each carried their own copy of the LSP-resolved-call lookup with
divergent confidence floors and different match semantics. Most production
indexing runs the parallel path, so projects produced different
CALLS-edge attributions depending on which pipeline mode kicked in:
pass_calls.c find_lsp_resolution floor 0.6 (CBM_LSP_CONFIDENCE_FLOOR)
pass_parallel lsp_override_resolution_pp floor 0.5 (hardcoded)
Both pipelines correctly invoke cbm_run_*_lsp through cbm_extract_file,
so result->resolved_calls is populated identically — the bug was at the
consumer.
Changes
-------
- New src/pipeline/lsp_resolve.h: static-inline cbm_pipeline_find_lsp_resolution
and CBM_LSP_CONFIDENCE_FLOOR (0.6f). Single source of truth for the
match rule (caller_qn equality + callee_qn short-name match + floor),
with rationale and language-coverage notes inline.
- pass_calls.c: include lsp_resolve.h, drop the local copy, rename the
call site to cbm_pipeline_find_lsp_resolution.
- pass_parallel.c: include lsp_resolve.h, drop lsp_override_resolution_pp,
use cbm_pipeline_find_lsp_resolution at resolve_file_calls. Add an
int lsp_overrides counter to resolve_worker_state_t (with adjusted
cache-line padding for the new field), aggregate it per-worker,
and emit it in the parallel.resolve.done log line so divergence
between pipelines becomes observable in production telemetry.
- tests/test_parallel.c: parallel_python_lsp_override_emits_lsp_strategy_edges
builds a single-file Python scenario (Greeter class + main() in one
file), runs the parallel pipeline, walks every CALLS edge, asserts
at least one carries an "lsp_*" strategy in its properties_json.
Pins both halves of the wiring at once: the shared find helper IS
consulted in the parallel path, AND the LSP-attributed strategy
survives all the way to the gbuf edge. Single-file rather than
cross-file because per-file py_lsp's resolved_calls callee_qn uses
the import module path rather than the project-qualified QN, so
cross-file LSP edges currently get dropped by cbm_gbuf_find_by_qn —
a separate cross-file QN-mismatch bug to track elsewhere.
Full suite: 3250 passed, 0 failed (was 3249; +1 new regression test).
Resolved 5 conflicts where the php-lsp branch and main both added LSP
infrastructure but with different APIs:
- Makefile.cbm: kept both TEST_PHP_LSP_SRCS and main's py_lsp/scope/
type_rep/stack_overflow test source variables in ALL_TEST_SRCS.
- internal/cbm/cbm.c: kept both #include lsp/php_lsp.h and lsp/py_lsp.h;
the per-language dispatch already supports stacking PHP after PYTHON.
- internal/cbm/lsp_all.c: kept both lsp/php_lsp.c + php_stdlib_data.c
AND lsp/py_lsp.c + python_stdlib_data.c in the compilation unit.
- src/pipeline/pass_calls.c: took main's version entirely.
find_lsp_resolution + resolve_single_call(ctx, call, lsp_calls, ...)
is language-agnostic — it matches any CBMResolvedCall whose caller_qn
hits the textual call's enclosing function and whose callee_qn
short-name hits the textual callee. PHP LSP populates the same
result->resolved_calls array via cbm_run_php_lsp, so the unified
resolver handles PHP for free. Dropped the php-branch's
qn_short_name + lsp_override_resolution helpers as superseded.
Locked confidence floor at main's CBM_LSP_CONFIDENCE_FLOOR = 0.6f
(was 0.5f on the php branch).
- tests/test_main.c: kept all suite externs and RUN_SUITE invocations
for php_lsp + py_lsp/py_lsp_bench/py_lsp_stress/py_lsp_scale +
scope + type_rep.
Auto-merged: src/pipeline/pass_parallel.c.
Brings in 52 commits from main since the branch was forked at c768b61
(#207 watcher follow-up): #208 through #281, dependabot bumps, Pine
Script, AUR docs, INHERITS edges, get_architecture wiring, codeql
revert, the entire python-lsp subsystem.
Suite: 3249 passed, 0 failed (was 2971 on main; +278 new tests from
php-lsp matches the branch's claimed test count).
Reverts the CodeQL portion of 27e640b ('fix(mcp,ci): harden
get_architecture serialization + CodeQL gate'). Leaves the mcp.c +
test_mcp.c changes (NULL-coercion, clusters/services serialization,
extract_text_content fix, regression test) intact.
Restored to the pre-27e640b state:
- drop 'jq' from build-deps install line
- drop 'queries: +./codeql' from codeql-action/init (no ./codeql
custom-query directory exists in this repo)
- drop 'id: analyze' and 'output: codeql-results' on analyze step
- drop the SARIF-parsing 'Fail on CodeQL error-level findings' step
CI workflow changes need their own design discussion per CONTRIBUTING.md
('Project configuration / CI workflows' is in the explicit-approval list);
the gate hardening landed bundled with an unrelated bug fix. Splitting it
out so the upstream MCP fix stays clean and any future CodeQL gate work
can be its own focused PR.
Follow-up to #281 (handle_get_architecture) plus a CodeQL workflow
upgrade developed in parallel; bundling into one commit because the
test suite had to land alongside both.
mcp.c — handle_get_architecture
- NULL-coerce every const-char* field in the architecture sections
via `x ? x : ""`, matching the rest of mcp.c (search_graph, etc.).
Without this, a NULL field becomes a missing JSON key instead of
an empty string; yyjson_mut_obj_add_str returns false on NULL and
silently no-ops, so an inconsistent omission could surprise callers.
- Serialize two more architecture aspects that #281 left on the
floor: services (cbm_service_link_t: from/to/type/count) and
clusters (cbm_cluster_info_t: id/label/members/cohesion plus the
top_nodes / packages / edge_types string arrays). The store-side
computation populates these for aspects=["all"] / explicit names,
so dropping them in the serializer was data loss.
tests/test_mcp.c
- New tool_get_architecture_emits_populated_sections regression test.
Uses a minimal inline fixture (single Function node tagged with
"is_entry_point": true) since arch_entry_points reads that flag
out of properties_json. Asserts the response contains both an
"entry_points" array and the function name — neither would appear
before #281 because handle_get_architecture never called
cbm_store_get_architecture.
- extract_text_content drilled too shallow: it pulled "content" only
from the JSON root, so it worked for cbm_mcp_handle_tool but
silently fell through to the raw response for cbm_mcp_server_handle
(where content lives under .result.content). Added a fallback that
checks .result.content; both unwrappers tested by existing fixtures.
ci(codeql)
- Run on pull_request to main, not just push; surfaces findings on
the PR instead of after merge.
- Pull custom queries from ./codeql via `queries: +./codeql`.
- Capture SARIF output and fail the job on any error-level finding,
using jq to enumerate rule id, file:line, and message text in the
GitHub Actions error annotation. Warnings are still reported as
before; only errors block.
Full suite: 2842 passed, 0 failed.
Phase 5f, 4ac, 4ad: closing the long-tail gaps toward LSP-server parity.
- @phpstan-type / @psalm-type / @phan-type aliases parsed from class
docblocks and applied during PHPDoc type resolution. References to
the alias name in @var/@param/@return are rewritten to the aliased
type. Aliases live on PHPLSPContext (per-file).
- @phpstan-import-type accepted without crash (cross-file alias
resolution deferred — same root cause as Composer PSR-4).
- @template-covariant T and @template-contravariant T now register T
as a proper type-param (previously skipped along with
@template-extends).
- Closure binding: \\Closure::bind($f, $obj) and $f->bindTo($obj),
when invoked with an inline closure literal as the first arg, walk
the closure body with $this rebound to the second arg's class.
Resolves the Laravel-macroable / VarDumper / encapsulation patterns.
- @return ($x is string ? int : bool) conditional return-type syntax
accepted without crash (parsed leftmost segment as fallback).
- Long PHPDoc blocks with mixed @param/@return/@throws/@internal
parsed without misattributing tags.
Stdlib unchanged.
Tests: 248 → 278 covering:
- phpstan-type alias basic / via @param / union / array-shape
- psalm-type alternate spelling
- phpstan-import-type (no-crash)
- Closure::bind / ::bindTo / static-bind on arrow fn
- conditional return / template-covariant / -contravariant (no-crash)
- class-string<T>, int<0,100>, literal-string (no-crash on phpstan
types we don't fully model)
- 8 realistic patterns: repository, event listener, console command,
form handler, mail sender, cached service, logger w/ context,
Carbon chains in methods, Eloquent model.
Total tests: 3061 → 3091 / 0 failed.
Audit for O(n^2) hot paths after Round 10. The dedup loop in
py_emit_resolved_call scanned ALL prior entries linearly, making
emission O(N) and total file work O(N^2). On a synthetic 2000-class
fixture with 8000 calls, that dominated time at 5083 ms.
Cap the scan window at the last 256 entries. Most duplicate
emissions come from the same expression being walked twice (resolver
+ emitter passes), so they're temporally adjacent — a 256-entry
window catches every common case while keeping the dedup O(1) per
emission.
Scale measurements (test_py_lsp_scale.c added):
| Classes | Calls | Resolved | Before | After |
|--------:|------:|---------:|-------:|------:|
| 100 | 400 | 300 | 55ms | 57ms |
| 500 | 2000 | 1500 | 333ms | 323ms |
| 2000 | 8000 | 6000 | 5083ms | 1804ms|
Ratio for 20x input: 91.8x -> 31.4x (linear would be 20x; quadratic
would be 400x). No remaining quadratic hot path in code added by
this branch.
Other hot paths surveyed:
- py_register_def -> cbm_registry_lookup_type: O(types) per def at
registration, O(D x types) for D defs. Pre-existing linear scan in
type_registry.c; hash-indexing it would be a cross-language
optimization (Go and C++ LSPs benefit too).
- py_lookup_attribute / py_lookup_field: O(types) per call site.
Same pre-existing linear scan. With ~900 stdlib types + per-file
defs, accounts for ~75µs/call at 2000-class scale.
- MODULE submodule prefix scan: O(funcs) per module-attribute access,
early-exits on first match. Bounded by registry size.
- lambda / dict-literal registries: linear, but bounded by per-file
N which is small.
Bench unchanged at 100% / 12ms (52/52 calls, 178 LOC).
handle_get_architecture was returning only label/edge histograms because it never called cbm_store_get_architecture — the rich architecture computation sat completely unwired upstream of the MCP boundary regardless of which aspects were requested.
Wires it up:
- Parse the JSON aspects array into a const char*[MCP_COL_16] and pass to cbm_store_get_architecture, which already handles 'all' and per-name filtering internally.
- Serialize each non-empty section: languages, packages, entry_points, routes, hotspots, boundaries, layers, file_tree. Existing histogram fields (node_labels, edge_types, relationship_patterns, cross_repo_links) preserved.
- Call cbm_store_architecture_free before return.
Closes#280.
When a trait method declares `: self` as its return type, flattening
that method into a using class now substitutes the trait's QN with the
using class's QN in the return type. This fixes fluent chains like:
trait T { public function tap(): self { return $this; } }
class C { use T; public function ok(): int { return 1; } }
$c->tap()->ok(); // resolves through to C.ok now
Implementation:
- flatten_trait_into_class scans the source method's signature for
NAMED(trait_qn) in the return type and rewrites it to
NAMED(using_class_qn) when registering the flattened method.
- php_lsp_collect_class_fields now does TWO passes — pass 0 for
trait/interface declarators (so their declared-return-types get
re-parsed before any consumer), pass 1 for class/enum (which may
`use Trait;` and need the trait's signatures already up-to-date).
Total tests: 3013 / 0 failed (200 php_lsp).
BENCHMARK_PYTHON.md updated through Round 10. The stress-test suite
went from 22 patterns (19 hard / 3 KNOWN GAP) to 43 patterns (all
hard-asserted PASS). Bench held at 100% (52/52) across the surface
expansion.
Patterns added in Round 10 cover the framework / real-world cases
that distinguish parity-with-LSP from passing-toy-fixtures:
SQLAlchemy 2.0 Mapped[T], Pydantic BaseModel, async with / async
for, Callable[..., R], next(iter), **kwargs annotation, dict-literal
dispatch tables, lambda call-site inference, recursive types,
classmethod-to-method chains, dataclass default_factory, nested
match patterns.
The original Phase 11 plan called this category "compiler-grade".
The actual line where compiler-rebuild starts is now narrower:
constraint solving for ParamSpec, code-flow narrowing across
basic blocks, custom metaclass __call__, and full descriptor
protocol. None of those are needed for what's resolved on the
bench fixture or the stress suite.
Round 10. Closes the last documented gaps from Round 9 plus 21 new
real-world stress tests covering SQLAlchemy / Pydantic / dataclass /
TypedDict / Protocol / generator / async patterns.
Lambda call-site inference:
- PyLSPContext gains a per-context lambda registry. `fn = lambda x:
body` records the lambda's AST node bound to fn. At call site
fn(arg), we push a sub-scope, bind lambda params from arg types,
and walk the body. Edges emitted under a synthetic <lambda> child
of the enclosing function so traces attribute correctly.
Dict-literal dispatch table:
- `funcs = {"a": foo, "b": bar}` records per-literal-key target QNs
in CBMDictLiteralEntry[]. funcs["a"]() emits an edge directly to
foo's QN with strategy=lsp_dict_dispatch.
Async with / async for:
- `async with X() as f` looks up __aenter__ before __enter__.
__aenter__'s declared return type wins; the manager type is the
fallback (async __aenter__s typically return self).
- `async for x in iter` falls back to __anext__'s return type when
the iterable's TEMPLATE form isn't recognized as AsyncIterator /
AsyncGenerator.
Callable / next / iter:
- `f()` where f resolves to CBM_TYPE_CALLABLE returns the callable's
return type. TEMPLATE("Callable", [..., R]) returns R.
- factory()() / (lambda)() etc — when the function expression is
itself a call / parenthesized / lambda, evaluate it and use its
return type.
- `next(it)` returns py_iterable_element_type(type-of-it).
- `iter(x)` is identity.
*args / **kwargs typed-parameter wrapping:
- `def f(*args: T)` -> args is TEMPLATE("tuple", [T]).
- `def f(**kwargs: T)` -> kwargs is TEMPLATE("dict", [str, T]).
- Detected by walking typed_parameter > list_splat_pattern /
dictionary_splat_pattern wrappers in py_bind_parameters.
Tuple / list / set / dict literal expressions:
- py_eval_expr_type now builds structured TUPLE / TEMPLATE types
from element types of the literal. `(Foo(), Bar())` -> TUPLE([Foo,
Bar]); `[Foo()]` -> TEMPLATE("list", [Foo]); `{"k": Foo()}` ->
TEMPLATE("dict", [str, Foo]). Without this, unpacking lost element
precision.
Match sequence pattern:
- list_pattern wraps each element in another case_pattern; capture
appears as dotted_name -> identifier; splat as splat_pattern.
Element-binder walks both wrappers and binds capture/splat to the
subject's iterable element type / list[elem].
Stress test suite expanded from 22 to 43 patterns. All 43 hard-
asserted as PASS:
- typing helpers: NamedTuple, TypedDict (incl. total=False), Protocol
(nominal + structural), ABC + abstractmethod, Mapped[T] (SQLAlchemy
2.0), Pydantic-style BaseModel
- control flow: with-as via __enter__, except-as, post-early-return
narrowing, async with via __aenter__, async for via __anext__
- containers: tuple unpack from function return + literal,
dict.items() comprehension k/v unpack, list slice, dict.get
default, recursive self-referencing types via forward refs
- decorators: factory + functools.wraps, classmethod chained to
method, property setter
- inheritance: Self in chain, diamond MRO with C3, multi-class match
- functional: lambda call-site inference, function-as-dict-value
dispatch, callable return-value, next(iter), generator delegation
(yield from), iterator next, double-star **kwargs, isinstance |
branches, module-level const, dataclass + default_factory
Bench: 178 LOC, 52 calls, **52 resolved (100%)**, 11.25 ms (ASan +
UBSan). Total 2941 / 0. Zero remaining KNOWN GAPs in the stress
suite — every probed pattern resolves.
Pins the bug-fix from #279 with a Java class declaring both extends and
implements. Asserts that base_classes contains:
- the superclass name (DefaultDiagramTool)
- every implements interface (ILinkTool, Closeable)
- bare type names only — no 'extends' / 'implements' keyword text
leaking into any entry
Without the fix, the field loop returned on the first match (only the
extends parent emitted) and cbm_node_text on the field returned the full
literal 'extends Bar' / 'implements Baz, Qux' string.
Two bugs in extract_base_classes() caused most INHERITS edges to be missing for Java (and any other language using similar AST field names):
Bug 1 — Early return: the field loop returned on the first matching field, so class Foo extends Bar implements Baz produced only one INHERITS edge instead of two — the loop never processed 'interfaces' after finding 'superclass'.
Bug 2 — Keyword text: cbm_node_text on the field node returned 'extends Bar' / 'implements Baz, Qux' literally instead of the bare type names. So even the one edge that did get emitted had a malformed target.
Fix: new collect_bases_from_field() walks into child AST nodes and extracts type_identifier / generic_type / qualified_name / scoped_type_identifier / user_type text directly. Handles type_list / interface_type_list children for the multi-interface case. Strips generics at '<'. Falls back to raw node text for languages where the field IS the type name. The outer field loop now collects from all matching fields before returning, and the result is arena-allocated with a NULL terminator like the surrounding extract_csharp_base_list pattern.
Bounds-safe: writes capped at MAX_BASES_MINUS_1 (15) with per-iteration count < out_cap checks; result fits in MAX_BASES (16) including NULL terminator.
Real-world impact: 3 → 116 INHERITS edges on the Modelio Java codebase (253 files).
Round 9 closes 2 of the 3 remaining stress gaps from Round 8:
TypedDict literal-key subscript:
- d["foo"] where d: TD with class-body annotation `foo: Foo` returns
the field's type. Routes through the existing class-body-annotation
field registry (Round 4): TypedDict subclasses already register their
fields the same way. The subscript branch in py_eval_expr_type now
detects NAMED container + literal-string key and looks up the field
via py_lookup_field. Strips both single and double quotes.
Early-return narrowing (post-block flow):
- if not isinstance(x, T): return / raise / break / continue —
applies the *positive* narrow to the enclosing scope after the
if-block when the consequence terminates.
- if x is None: return — narrows x to non-None afterwards.
- py_block_terminates detects return/raise/break/continue as the last
statement of the body block. Subsequent code in the function sees
the narrowed type.
Match sequence pattern (tentative):
- list_pattern / tuple_pattern handler added: bind capture-pattern
elements to the subject's iterable element type; bind splat-pattern
rest to list[elem]. Also walks more wrapper levels (case_pattern ->
pattern -> _simple_pattern). Test still flags as KNOWN GAP because
tree-sitter's pattern-wrapper structure for the bracket-form `[a, b]`
in match/case differs in some grammar versions; documented as
remaining gap pending grammar inspection.
Stress test suite: 22/22 PASS. Hard-asserted patterns: namedtuple,
typeddict, protocol, abc, with-as, except-as, isinstance-early-return,
tuple-unpack, dict.items, slice, decorator-factory, property-setter,
Self chain, diamond MRO, recursive types, nested closure, long method
chains, generator delegation, async-gen for. Documented gaps:
function-as-dict-value, match sequence pattern, lambda inference.
Bench unchanged at 100% / 11.02 ms; total 2920 / 0.
Round 8. Stress test suite (test_py_lsp_stress.c) added with 22 advanced
patterns; bench fixture grew to 178 lines covering the full surface.
with-as binding via __enter__:
- Look up __enter__ on the context manager's NAMED type. Use its
return type when known; fall back to the manager type itself
(matches the practical case where __enter__ returns self). Handles
tree-sitter Python's wrapping of `X as y` in an as_pattern node
inside with_item, plus the flat field-only shape.
except-as binding:
- Bind exception alias to the caught type. Resolves through
except_clause's as_pattern wrapper and the as_pattern_target
identifier wrapper.
Tuple unpacking:
- Both `a, b = f()` and `for k, v in d.items()` destructure the RHS
element-by-element when the rhs is TUPLE or TEMPLATE("tuple", ...).
*rest binds the trailing slice as list[elem]. Shared helper
py_bind_for_target used by for-statement, comprehension for_in_clause,
and assignment LHS.
Slice subscript:
- `lst[1:3]` returns the same TEMPLATE container — list[T][1:3] is
list[T], dict[K, V][1:3] is dict[K, V] (rare but consistent).
Container TypeVar substitution:
- Without a constraint solver we can't generally substitute K, V into
registered method return types. Hand-rolled the most-impactful
cases: dict.items/keys/values/get/pop/copy, list.copy/__iter__/pop,
set.copy. These return properly parameterized ItemsView[K, V] /
KeysView[K] / ValuesView[V] / Optional[V] / etc. so subsequent
iteration and unpacking yield typed elements.
ItemsView / KeysView / ValuesView / dict_items / dict_keys / dict_values
added to py_iterable_element_type. ItemsView[K, V] iterates as
tuple[K, V], unpacked correctly by py_bind_for_target.
Bench: 178 LOC, 52 calls, **52 resolved (100%)**, 11.59 ms (ASan + UBSan).
22 new stress tests, 8 of 10 documented gaps closed in this round.
Remaining gaps documented inline (typeddict_subscript, post-early-return
narrowing, function-as-dict-value, match sequence pattern, lambda
inference). All 2898 prior tests stay green; total 2920 / 0.
Adds Pine Script (TradingView's indicator/strategy DSL) as a first-class
language using the project's existing language-add pipeline:
- scripts/new-languages.json: register pine (display "PineScript",
ts_func tree_sitter_pine, repo kvarenzn/tree-sitter-pine, extension
.pine, has_scanner=true, module_root source_file)
- internal/cbm/vendored/grammars/pine/: vendored from
https://github.com/kvarenzn/tree-sitter-pine via scripts/vendor-grammar.sh
— parser.c, scanner.c, tree_sitter/ headers. License is ISC (declared
in upstream package.json; written into vendored LICENSE for clarity
since upstream has no LICENSE file)
- internal/cbm/grammar_pine.c: 4-line wrapper generated by
scripts/generate-lang-code.py wrappers
- internal/cbm/cbm.h: CBM_LANG_PINE before CBM_LANG_COUNT
- internal/cbm/lang_specs.c: 7 node-type arrays (func / class /
module / call / var / branch / assign) cross-checked against
kvarenzn's grammar.js, plus the spec table entry
- src/discover/language.c: .pine -> CBM_LANG_PINE extension dispatch
and "PineScript" name
scripts/audit-grammar-security.sh on the vendored grammar: no
dangerous includes (only string/stdint/stdbool/stdlib), no system /
popen / exec / socket / mmap / dlopen calls, no constructor/destructor
attributes, no inline asm. The 148-line scanner.c only handles
INDENT/DEDENT/NEWLINE tokens via the standard tree-sitter lexer API.
Distilled from #273 rather than vendored verbatim because the
original PR's parser.c was a plain 46k-line drop without provenance,
no LICENSE attribution, and bundled an unrelated Makefile change
(scripts/embed-frontend.sh -> sh scripts/embed-frontend.sh) that
didn't belong to a Pine Script feature. The node-type choices in
the lang spec match the original PR's selections, all of which were
verified to be real grammar.js rules.
Full suite: 2840 passed, 0 failed under ASan + UBSan.
Closes#273.
Co-authored-by: vinay-veerappa <vinay.veerappa@gmail.com>
Phase 4m–w in one push, plus 25 new tests (100 → 125, all green).
New capabilities:
- Generic @template substitution:
@template T on a class records T as a type-param;
@return T on a method gets parsed as NAMED("T");
when receiver is TEMPLATE("Container", [User]), method-call return
types are substituted T → User so chains like
`$container->get()->name()` resolve through to the element type.
- Closure use(...) capture: typed variables in `use ($a, $b)` clauses
of `function () use (...) {}` are inherited from the parent scope.
- Multi-conjunction narrowing: `if ($x instanceof A && $y instanceof B)`
narrows BOTH $x and $y in the if-body. Up to 8 simultaneous narrowings.
- is_a($x, Foo::class) narrowing — both the ::class and string forms.
- Subscript inference: $arr[$k] on `array<T>` returns T;
on `array<K, V>` returns V;
on a NAMED ArrayAccess class consults registered offsetGet.
- Enum case access: Suit::Hearts evaluates to NAMED(Suit) so
Suit::Hearts->label() resolves to Suit::label.
- @extends ParentClass<X>: appends ParentClass to embedded_types so
the inheritance walk picks up methods from generic ancestors.
- PHPDoc @return on methods with no declared return type updates
the registry signature.
- foreach key=>value: handles tree-sitter-php's `pair` node so
`foreach ($users as $idx => $user)` binds $user to the element type.
- Class-level @template-extends, @template-implements parsed without
crashing.
- Don't override TEMPLATE/NAMED bindings with `null` literal —
preserves @var-typed variables across `$x = null` initialization.
Stdlib expansion (Phase 5d):
- Symfony Console: SymfonyStyle, InputInterface, OutputInterface
- Doctrine ORM: EntityManagerInterface, QueryBuilder chain, Query,
EntityRepository
- Guzzle HTTP: Client and ClientInterface (returning PSR-7 responses)
- Twig: Environment, TemplateWrapper
- Eloquent Model: forwarded static methods (where, whereIn, etc.)
so `User::where()->first()` chains resolve immediately.
Test coverage: 100 -> 125 tests; total 2913 -> 2938 / 0 failed.
Major parity push. Bench 79% -> 98% (51/52 calls resolved) on a 178-line
fixture exercising the full parity surface.
Annotation wrapping:
- Type-wrapper annotations now unwrap to the inner type:
ClassVar[T], Final[T], InitVar[T], ReadOnly[T], Required[T],
NotRequired[T], Annotated[T, ...] (drop metadata),
Mapped[T] / WriteOnlyMapped[T] / DynamicMapped[T] (SQLAlchemy 2.0),
Type[T] / type[T]. Same logic in both py_resolve_annotation
(runtime, scope-aware) and py_parse_type_text_qn (registry-build,
scope-free).
Builtin / template attribute dispatch:
- obj_type=BUILTIN("str"|"int"|"list"|"dict"|...) now resolves
attribute access via "builtins.<typename>" in the typeshed-derived
registry. obj.split(...) on str, list.append(), dict.get() etc all
emit edges with strategy=lsp_builtin_method.
- obj_type=TEMPLATE("dict"|"list"|...) routes the same way; receiver
is the unparameterized container's typeshed entry.
Bare-name builtin fallback:
- Bare-name calls (range, len, list, dict, str, int, print, etc.) fall
through to a "builtins" module symbol lookup, then to a builtins
type lookup for class constructors. Strategies: lsp_builtin /
lsp_builtin_constructor.
Emission deduplication:
- py_emit_resolved_call dedupes by (caller_qn, callee_qn). When two
resolution paths converge on the same edge (expression typing then
call emission, or attribute walk through wrappers), keep one entry
with the higher confidence. Previously the same call could emit
twice — the bench showed >100% ratios as evidence.
Bench fixture expanded from 136 to 178 lines:
- Adds @property access chains, Mapped[Annotated[int, ...]], Final
fields, generator iteration with method dispatch on element type,
multi-class match/case, dict comprehension over .values(), instance
attribute typing through super().__init__, classmethod chained with
Self return.
All 2898 prior tests stay green; bench at 98% in 11.79 ms (ASan +
UBSan).
Round 6:
CBMRegisteredFunc gains a `flags` bitfield (CBMFuncFlags) and a
`decorator_qns` NULL-terminated list. py_register_def populates them
from CBMDefinition.decorators by short-name match — so `@property`,
`@functools.cache`, `@abc.abstractmethod`, `@typing.overload`, and
`@final` all light up regardless of qualified vs bare reference. The
flags aren't consumed by resolution yet (still on roadmap: emit
distinct edge for @property attribute access vs method call); they
are recorded so future overload selection / property modeling can
use them without another schema change.
The struct extension is at-tail and existing call sites that memset
to zero stay correct; non-Python LSPs see flags=0 / decorator_qns=NULL
and behave identically.
Three new verification tests prove that generators, dataclasses, and
@property already work end-to-end through the Round 4-5 infrastructure
without further code changes:
- Generator[Foo, ...] iterables in for-loop bind loop var to Foo via
py_iterable_element_type's existing TEMPLATE handling.
- @dataclass class Point: x: Foo (class-body annotated assignment) is
picked up by Round 4's third-pass field registrar; p.x.method()
resolves through the field type.
- @property def thing(self) -> Foo: c.thing returns Foo because
attribute access on NAMED routes through py_func_return_type which
yields the getter's annotated return.
All 2895 prior tests stay green; 3 new tests pass. Total 2898 / 0.
Bench unchanged at 95% (the small bench fixture doesn't exercise
the new patterns; will expand in Round 7).
Round 5 parity push:
Subscript value typing — py_eval_expr_type's subscript branch resolves
indexing against the container's TEMPLATE type:
- TEMPLATE("dict"|"Mapping"|"defaultdict"|"OrderedDict"|...)[K] -> V
- TEMPLATE("list"|"set"|"frozenset"|"Sequence"|"Iterable"|...)[i] -> elem
- TEMPLATE("tuple", [A, B, ...])[literal_int] -> args[idx]
- Non-literal index returns UNION of all element types.
This unlocks the `self.cache[k].method()` pattern that's everywhere in
real Python. Also fixes `items[0].method()` for typed lists.
super().__init__(...) constructor edge — super().<attr> already worked
for regular methods; __init__ is special-cased because parent classes
typically don't register __init__ with a useful return type. Strategy
"lsp_super_init" with confidence 0.85.
Operator dunders — a + b looks up __add__ on a's type and uses its
return type. Same for *, -, /, //, %, **, <<, >>, &, |, ^, @. Falls
back to left-operand type when registry has no info (correct for the
common int+int / str+str / list+list cases).
3 new tests; all 2892 prior tests stay green. Bench unchanged at 95%
(fixture doesn't exercise these patterns directly — Round 6 will
expand it).
Martin Vogel