Add a C# language LSP resolver mirroring the existing Go/C/PHP/Python
modules: type evaluation, using-directive resolution, method lookup,
and a corpus-seeded .NET BCL stdlib data table.
- internal/cbm/lsp/cs_lsp.{c,h}: resolver implementation
- internal/cbm/lsp/generated/cs_stdlib_data.c: BCL type/method tables
- wire CBM_LANG_CSHARP dispatch in cbm.c and lsp_all.c
- register cs_lsp + cs_lsp_bench suites in the test harness
- tests/test_cs_lsp.c, tests/test_cs_lsp_bench.c
(cherry picked from commit 8f677c67a0c9d7829cd6ac27393de2d0aaaac534)
cbm_store_checkpoint() invoked SQLITE_CHECKPOINT_TRUNCATE, the most
aggressive mode. When two cbm-mcp processes share a cache dir, one
process's TRUNCATE can shrink files while another has them mmap'd,
raising SIGBUS on macOS. PASSIVE never blocks readers and never
ftruncate()s either file; SQLite still autocheckpoints in PASSIVE
mode at 1000-page boundaries, so reclamation is unaffected for
single-process users.
Recommended by SQLite docs for shared databases:
https://www.sqlite.org/pragma.html#pragma_wal_checkpoint
Hard-coded `PRAGMA mmap_size = 67108864` in configure_pragmas() left
no path for users running multiple cbm-mcp instances against the
same cache to opt out of memory-mapped I/O. On macOS, when one
instance's checkpoint or reindex truncates the DB file under
another instance's live mmap, accessing the now-missing pages
raises SIGBUS, taking the process down.
Setting CBM_SQLITE_MMAP_SIZE=0 reverts to read()/pread() I/O,
which returns recoverable SQLITE_IOERR instead of crashing the
process.
- Default unchanged (67108864 / 64 MB). No behavior change for
single-instance users.
- Malformed values (non-numeric, partial-numeric) fall back to
the default rather than failing the store open.
- Negative values clamp to 0.
- New tests: tests/test_store_pragmas.c covers all five resolver
paths plus an integration smoke that opens a file-backed store
with mmap disabled.
Empirical evidence: 9 SIGBUS crash reports collected on macOS
arm64 v0.6.0 in a 14-hour window, all signature 'cluster_pagein
past EOF' with stacks bottoming in SQLite btree code under the
watcher thread's incremental-index pipeline.
Resolved 5 conflicts where the php-lsp branch and main both added LSP
infrastructure but with different APIs:
- Makefile.cbm: kept both TEST_PHP_LSP_SRCS and main's py_lsp/scope/
type_rep/stack_overflow test source variables in ALL_TEST_SRCS.
- internal/cbm/cbm.c: kept both #include lsp/php_lsp.h and lsp/py_lsp.h;
the per-language dispatch already supports stacking PHP after PYTHON.
- internal/cbm/lsp_all.c: kept both lsp/php_lsp.c + php_stdlib_data.c
AND lsp/py_lsp.c + python_stdlib_data.c in the compilation unit.
- src/pipeline/pass_calls.c: took main's version entirely.
find_lsp_resolution + resolve_single_call(ctx, call, lsp_calls, ...)
is language-agnostic — it matches any CBMResolvedCall whose caller_qn
hits the textual call's enclosing function and whose callee_qn
short-name hits the textual callee. PHP LSP populates the same
result->resolved_calls array via cbm_run_php_lsp, so the unified
resolver handles PHP for free. Dropped the php-branch's
qn_short_name + lsp_override_resolution helpers as superseded.
Locked confidence floor at main's CBM_LSP_CONFIDENCE_FLOOR = 0.6f
(was 0.5f on the php branch).
- tests/test_main.c: kept all suite externs and RUN_SUITE invocations
for php_lsp + py_lsp/py_lsp_bench/py_lsp_stress/py_lsp_scale +
scope + type_rep.
Auto-merged: src/pipeline/pass_parallel.c.
Brings in 52 commits from main since the branch was forked at c768b61
(#207 watcher follow-up): #208 through #281, dependabot bumps, Pine
Script, AUR docs, INHERITS edges, get_architecture wiring, codeql
revert, the entire python-lsp subsystem.
Suite: 3249 passed, 0 failed (was 2971 on main; +278 new tests from
php-lsp matches the branch's claimed test count).
Audit for O(n^2) hot paths after Round 10. The dedup loop in
py_emit_resolved_call scanned ALL prior entries linearly, making
emission O(N) and total file work O(N^2). On a synthetic 2000-class
fixture with 8000 calls, that dominated time at 5083 ms.
Cap the scan window at the last 256 entries. Most duplicate
emissions come from the same expression being walked twice (resolver
+ emitter passes), so they're temporally adjacent — a 256-entry
window catches every common case while keeping the dedup O(1) per
emission.
Scale measurements (test_py_lsp_scale.c added):
| Classes | Calls | Resolved | Before | After |
|--------:|------:|---------:|-------:|------:|
| 100 | 400 | 300 | 55ms | 57ms |
| 500 | 2000 | 1500 | 333ms | 323ms |
| 2000 | 8000 | 6000 | 5083ms | 1804ms|
Ratio for 20x input: 91.8x -> 31.4x (linear would be 20x; quadratic
would be 400x). No remaining quadratic hot path in code added by
this branch.
Other hot paths surveyed:
- py_register_def -> cbm_registry_lookup_type: O(types) per def at
registration, O(D x types) for D defs. Pre-existing linear scan in
type_registry.c; hash-indexing it would be a cross-language
optimization (Go and C++ LSPs benefit too).
- py_lookup_attribute / py_lookup_field: O(types) per call site.
Same pre-existing linear scan. With ~900 stdlib types + per-file
defs, accounts for ~75µs/call at 2000-class scale.
- MODULE submodule prefix scan: O(funcs) per module-attribute access,
early-exits on first match. Bounded by registry size.
- lambda / dict-literal registries: linear, but bounded by per-file
N which is small.
Bench unchanged at 100% / 12ms (52/52 calls, 178 LOC).
Round 8. Stress test suite (test_py_lsp_stress.c) added with 22 advanced
patterns; bench fixture grew to 178 lines covering the full surface.
with-as binding via __enter__:
- Look up __enter__ on the context manager's NAMED type. Use its
return type when known; fall back to the manager type itself
(matches the practical case where __enter__ returns self). Handles
tree-sitter Python's wrapping of `X as y` in an as_pattern node
inside with_item, plus the flat field-only shape.
except-as binding:
- Bind exception alias to the caught type. Resolves through
except_clause's as_pattern wrapper and the as_pattern_target
identifier wrapper.
Tuple unpacking:
- Both `a, b = f()` and `for k, v in d.items()` destructure the RHS
element-by-element when the rhs is TUPLE or TEMPLATE("tuple", ...).
*rest binds the trailing slice as list[elem]. Shared helper
py_bind_for_target used by for-statement, comprehension for_in_clause,
and assignment LHS.
Slice subscript:
- `lst[1:3]` returns the same TEMPLATE container — list[T][1:3] is
list[T], dict[K, V][1:3] is dict[K, V] (rare but consistent).
Container TypeVar substitution:
- Without a constraint solver we can't generally substitute K, V into
registered method return types. Hand-rolled the most-impactful
cases: dict.items/keys/values/get/pop/copy, list.copy/__iter__/pop,
set.copy. These return properly parameterized ItemsView[K, V] /
KeysView[K] / ValuesView[V] / Optional[V] / etc. so subsequent
iteration and unpacking yield typed elements.
ItemsView / KeysView / ValuesView / dict_items / dict_keys / dict_values
added to py_iterable_element_type. ItemsView[K, V] iterates as
tuple[K, V], unpacked correctly by py_bind_for_target.
Bench: 178 LOC, 52 calls, **52 resolved (100%)**, 11.59 ms (ASan + UBSan).
22 new stress tests, 8 of 10 documented gaps closed in this round.
Remaining gaps documented inline (typeddict_subscript, post-early-return
narrowing, function-as-dict-value, match sequence pattern, lambda
inference). All 2898 prior tests stay green; total 2920 / 0.
Final phase of Python LSP integration. Adds:
- docs/BENCHMARK_PYTHON.md — methodology document covering target
metrics (per-file LSP overhead < 10 ms, resolution ratio ≥ 40% on
application code, ≥ 70% with stdlib, < 5% spot-check false positive
rate), the declared accepted-failure-class ceiling (metaclass
__call__, __getattr__/__getattribute__, dynamic setattr, runtime
importlib, monkey-patching, eval/exec, descriptors beyond the
standard property/classmethod/staticmethod, dynamic type() classes),
and instructions for running scripts/benchmark-index.sh against the
largest Python repos in datadice (falkemedia: 232 files, api-hub:
188 files).
- tests/test_py_lsp_bench.c — in-process benchmark wired into
scripts/test.sh. Runs cbm_extract_file on a 65-line Python fixture
covering imports, dataclasses, classmethods, single-inheritance with
super(), typed parameters, attribute chains, and stdlib calls.
Asserts soft floor (resolved >= calls/4) and hard time budget
(< 50 ms under sanitizers). Initial run reports:
bench: 65 lines, 16 calls, 13 resolved (81%), 2.45 ms
81% beats the 40% application-code target and the 70%
application-plus-stdlib target. 2.45 ms under ASan + UBSan is well
inside the 10 ms (no-sanitizer) production budget.
This wraps the 11-phase plan documented in PYTHON_LSP_PLAN.md. Total
test count: 2874 / 0 (100% pass). Repository-level benchmarks via
scripts/benchmark-index.sh remain a manual operation against a fresh
production binary; methodology is now in place for when that runs.
CALLS edges for cross-file function calls were mis-wired or missing in
TypeScript projects that use path aliases (e.g. @/lib/auth → src/lib/auth):
1. resolve_relative_import only handled "./relative" paths, so
"@/lib/auth"-style imports never reached IMPORTS edges. The import
map stayed empty, Strategy 1 in cbm_registry_resolve failed, and
resolution fell through to name-based lookups that pick the wrong
target when functions share names across files.
2. resolve_import_map built the candidate QN as the bare module QN
when the callee had no dot ("requireAdmin" vs "pkg.Func"), instead
of module_qn.requireAdmin. Even with a populated import map, bare
calls still mis-resolved.
Fix:
- New src/pipeline/path_alias.{h,c}: language-agnostic build-tool
path-alias subsystem. Currently parses tsconfig.json and
jsconfig.json compilerOptions.paths + baseUrl via the existing
yyjson dependency (with ALLOW_COMMENTS|ALLOW_TRAILING_COMMAS for
JSONC). The data model (cbm_path_alias_t / map / scope /
collection) is generic; adding Vite/Webpack/Python loaders means
writing a new file-parsing helper and registering it in
find_alias_files — the resolver, collection, and pipeline
integration do not change.
- O(n log n) ordering: scope-by-dir-specificity and entry-by-
prefix-specificity are sorted with qsort, replacing the O(n²)
bubble sorts that the original draft used. With caps of 256
entries per config and 256 configs per repo, the sort is
effectively flat.
- Resource ceilings with logging: directory recursion capped at
32 levels; total config files capped at 256; per-config size
capped at 64 KB. Cap hits emit cbm_log_warn so silent truncation
on pathological monorepos shows up in the index log.
- Walk hygiene: skips node_modules, .git, dist, build, .next,
coverage, target.
- Single integration point: cbm_pipeline_resolve_module gets a
Strategy 1b between relative-import resolution and pkgmap so both
the sequential and parallel pipelines pick up alias resolution
automatically. Non-TS projects pay nothing — the loader returns
NULL when no usable configs are found.
- Layer 2 fix in registry.c: resolve_import_map now builds the
candidate as resolved.prefix when suffix is NULL (one-line fix,
matching comment).
- Tests: 10 path_alias tests (basic wildcard, specificity, exact,
extension stripping, baseUrl, NULL safety, nearest-ancestor scope
selection, end-to-end loader against a tmp monorepo, no-configs
case) plus a registry test for the bare-call fix. Full suite:
2835 passed, 0 failed.
Cherry-picked and generalized from #243 (originally TypeScript-only
under tsconfig-named types). Renamed types (cbm_path_alias_scope_t /
collection_t in place of cbm_tsconfig_*) and centralised the load
into cbm_load_path_aliases so future loaders for other build tools
slot in without touching extraction or pipeline code.
Closes#243.
Co-authored-by: Peter Cox <info@petercox.ie>
Covers parameter binding (typed param, arrow function, $this, catch),
method dispatch (typed receiver, chain, self/parent), static calls,
namespace + use resolution, PHPDoc @var, magic methods, and the
ConfiguresPrompts $prompt->value() regression from PHP_LSP_PRE_FLIGHT.md.
Wired into ALL_TEST_SRCS and suite_php_lsp invocation in test_main.c.
All 16 tests pass; total suite is now 2829/0 passing.
Phase 2 of Python LSP integration. Adds the py_lsp module mirroring
the go_lsp / c_lsp pattern: PyLSPContext, single-file entry
(cbm_run_py_lsp), cross-file + batch shells, stdlib registration stub.
Bodies are no-ops at this phase. Subsequent phases fill in:
Phase 3 — imports
Phase 4 — scope binding
Phase 5 — expression typing
Phase 6 — attribute resolution + method dispatch
Phase 7 — decorators
Phase 8 — class hierarchy + generics
Phase 9 — cross-file resolution + batch
Phase 10 — cbm_python_stdlib_register from typeshed
Wired into cbm_extract_file alongside the existing Go and C/C++ LSP
calls. lsp_all.c #includes py_lsp.c so the CGo umbrella picks it up
without a separate translation unit.
tests/test_py_lsp.c adds 5 smoke cases: empty source, single function,
single class, syntax error recovery, imports pass-through. The helper
shape (extract_py / find_resolved / require_resolved) matches
test_go_lsp.c so future phases can grow the suite by category.
All 2839 existing tests stay green; 5 new py_lsp tests pass.
Phase 1 of Python LSP integration. Five new CBMTypeKind variants needed
for Python type modeling, plus structural-equality helper used by union
deduplication and protocol matching:
- UNION: A | B / Union[A, B] / Optional[T]. cbm_type_union normalizes
by flattening nested UNIONs, deduplicating members via cbm_type_equal,
and collapsing single-member unions to their member. Empty unions
reduce to UNKNOWN.
- LITERAL: Literal["foo", 3] — wraps a base type with literal text;
text-distinct literals compare unequal.
- PROTOCOL: typing.Protocol with method-name set. Structural matching
via cbm_type_protocol_satisfied_by — succeeds when the candidate's
method set is a superset of the protocol's.
- MODULE: bound name from `import X`, distinguishing module-attribute
lookup from class-attribute lookup. Equality is by module_qn.
- CALLABLE: Callable[[A, B], R] including the elliptic Callable[..., R]
encoded as param_count = -1.
cbm_type_equal added as a recursive structural-equality helper covering
all kinds; FUNC/INTERFACE/STRUCT fall back to pointer identity since
their structural members are rarely compared and the cost would be
substantial.
Adds tests/test_type_rep.c with 16 cases covering normalization,
equality, and protocol satisfaction.
All 2823 existing tests stay green; 16 new type-rep tests pass.
AST traversal functions used fixed-size TSNode stack[] arrays. When the
DFS stack filled up, the child-push loop exited silently, dropping
entire subtrees without warning (e.g. 600 TS imports → 511, 130 Express
routes capped at 512). Reported in #199; also addresses #213 (large
TypeScript files producing zero nodes from stack exhaustion) and #215
(SEGV in template calls from stack overflow).
Adds ts_node_stack.h — a growable stack backed by the existing arena
allocator. Initial capacity matches previous fixed caps so small files
allocate no extra memory; doubles on overflow instead of truncating.
Old blocks are abandoned in the arena and freed on arena_destroy at
the end of file extraction (no realloc, no mixed lifetimes).
Applied to all 14 TSNode fixed stacks across 9 extraction files
(extract_calls, extract_channels, extract_defs, extract_env_accesses,
extract_imports, extract_semantic, extract_type_assigns, extract_type_refs,
extract_usages). walk_defs (uses walk_defs_frame_t, different struct
type) deliberately left as-is.
7 regression tests in tests/test_stack_overflow.c cover TS imports >512,
JS/Python calls >512, Go calls >1024, Express routes >150, deeply
nested calls, and YAML vars >256.
Cherry-picked from #217 with merge-conflict resolution against post-#206/#207
main:
- Makefile.cbm: keep TEST_STACK_OVERFLOW_SRCS plus main's new
TEST_ZSTD_SRCS / TEST_ARTIFACT_SRCS.
- extract_channels.c: keep main's scan_string_consts_js name (main also
added scan_string_consts_python after the PR was opened); declare
CHAN_STACK_CAP in the enum block.
- test_stack_overflow.c: replace sprintf with snprintf using the actual
remaining buffer size, since macOS clang rejects sprintf and naive
snprintf with sizeof(p) on a char* fails -Wformat-truncation.
Closes#199, #213, #215. Closes#217.
Phase 0 of Python LSP integration. Two shared-infrastructure fixes that
silently bug Python without affecting Go/C++ regression tests:
- CBMScope replaces the fixed 64-binding array with a growable linked
list of 16-binding chunks. The arena pointer is captured at scope_push
and reused for chunk allocation, so existing callers benefit from
dynamic growth without an API change. Python class bodies and large
dataclass scopes routinely exceed 64 locals; this prevents silent
drops past the legacy cap.
- CBM_LSP_MAX_LOOKUP_DEPTH=16 wraps go_lookup_field_or_method and
c_lookup_member via depth-aware internal helpers. Pathological
multi-inheritance hierarchies (SQLAlchemy / Django / Zope) can exceed
20 levels and previously stack-overflowed; now they degrade to NULL
past depth 16 instead.
Adds tests/test_scope.c with 10 cases covering basic API, parent-chain
walks, child-shadowing, and dynamic growth at 300 bindings + chunk
boundaries. test_main.c and Makefile.cbm wire the new suite.
PYTHON_LSP_PLAN.md added at the worktree root: full phased plan plus
the verbatim challenger review.
All 2823 existing tests stay green; 10 new scope tests pass.
Add .codebase-memory/graph.db.zst — a zstd-compressed knowledge graph
artifact that can be committed to the repo. Teammates bootstrap from
the artifact instead of running a full reindex from scratch.
- Vendor zstd 1.5.7 (amalgamated build) for 8-13:1 compression
- Two-tier export: zstd -9 + index stripping for explicit index,
zstd -3 for watcher/incremental auto-updates
- Import: decompress → integrity check → auto-recreate indexes
- Bootstrap in handle_index_repository: when no local DB exists but
artifact is present, import first then run incremental
- Auto-create .gitattributes with merge=ours to prevent conflicts
- Fix: add missing idx_edges_url_path to create_user_indexes and
url_path_gen generated column to init_schema
- 13 new tests (5 zstd wrapper + 8 artifact round-trip/edge cases)
Compute K=64 MinHash signatures from normalized AST node-type trigrams
during function extraction, then generate SIMILAR_TO edges via LSH
(b=32, r=2) for function pairs with Jaccard >= 0.95.
- src/simhash/minhash.{h,c}: MinHash compute, Jaccard, hex encode/decode,
LSH index with band hashing for O(n) candidate generation
- src/pipeline/pass_similarity.c: post-pass reads fingerprints from node
properties, builds LSH index, emits SIMILAR_TO edges with jaccard and
same_file metadata. Same-language only, max 10 edges per node.
- internal/cbm/cbm.h: fingerprint fields on CBMDefinition
- internal/cbm/extract_defs.c: compute_fingerprint() hook at 3 extraction
sites after complexity, skip functions with < 10 AST body nodes
- pass_definitions.c + pass_parallel.c: serialize fingerprint to "fp" hex
in properties_json for both sequential and parallel pipeline paths
- pipeline.c + pipeline_incremental.c: register pass_similarity in both
full and incremental post-pass lists
- tests/test_simhash.c: 28 tests across 4 suites (core, LSH, edge gen,
pipeline integration with generated Go project + incremental)
CBM_SKIP_PERF=1 now skips only phases 2-7 (incremental delta, stress,
perf regression) inside the suite. Phase 1 (initial index) and phases
8-30 (135 tool integration tests) always run. This keeps ~2672 tests
on CI vs the full ~2692 locally.
Also fix Windows build: guard symlink() with #ifndef _WIN32.
- Add CBM_SKIP_PERF=1 env var to skip incremental/perf test suite
- CI and Docker test targets skip perf by default (run.sh perf for manual)
- Convert all perf assertions to warnings (log timing, never block)
- Fix store.h anonymous enum in struct (GCC rejects, clang accepts)
- Fix test_store_search.c mkstemp on non-template path
- Add ca-certificates to Docker test image for git HTTPS
- Add cbm_gmtime_r shim in compat.h (Windows gmtime_s wrapper)
- Fix compat.c missing constants.h include (Windows build)
- Fix platform.c _environ redeclaration on mingw
- Rename trace_call_path -> trace_path in smoke/soak/fuzz scripts
Comprehensive test suite using real repo (FastAPI 0.99.1, ~1100 files):
- Full index baseline with perf metrics (nodes, edges, RSS, timing)
- Incremental: modify/add/delete/rename files, noop, formatter run
- Adversarial: empty file, syntax errors, 5000-line function, binary
- Stress: rapid reindex, batch 20 files, delete-all + re-add
- Accuracy: incremental vs full within ±2 nodes/edges
- MCP tools: all 14 tools, every parameter, content validation
- Error handling: bad project, missing params for each tool
- Performance: 5s timeout on every tool call
Fix use-after-free in detect_changes and manage_adr:
yyjson_mut_arr_add_str stored pointer to stack buffer without copy.
Changed to yyjson_mut_arr_add_strcpy / yyjson_mut_obj_add_strcpy.
Read extra_extensions from ~/.config/codebase-memory-mcp/config.json
(global) and {repo}/.codebase-memory.json (project), merging them with
project config taking precedence. Unknown language values warn and skip.
Missing files are silently ignored (fail-open).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
cbm_store_begin_bulk() was switching the SQLite journal mode from WAL to
MEMORY for write throughput. If the process crashed mid-bulk-write the
in-memory rollback journal was lost, leaving the database file in a
partially-written, unrecoverable state.
WAL mode is inherently crash-safe: uncommitted WAL entries are discarded
on the next open. The performance benefit of bulk mode is preserved via
synchronous=OFF and an enlarged cache_size, both of which are safe under
WAL.
Remove the PRAGMA journal_mode = MEMORY from cbm_store_begin_bulk and
the matching PRAGMA journal_mode = WAL from cbm_store_end_bulk. Update
the header comments to reflect the new invariant.
Add tests/test_store_bulk.c with three tests:
- bulk_pragma_wal_invariant: asserts journal_mode remains "wal" after
cbm_store_begin_bulk via an independent read-only connection
- bulk_pragma_end_wal_invariant: asserts journal_mode remains "wal" after
cbm_store_end_bulk
- bulk_crash_recovery: forks a child that enters bulk mode, opens an
explicit transaction, writes data, then calls _exit() without committing;
the parent verifies the database opens cleanly and baseline data survives
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Complete C implementation of the indexing pipeline (src/), parallel worker
pool, graph buffer with merge support, SQLite writer, and 1893-test suite.
Linter setup: zero warnings from clang-tidy (all checks enabled, no NOLINT
suppressions), cppcheck, and clang-format. All issues fixed at source —
proper headers for external linkage, GROW_ARRAY macro restructured to
eliminate type parameter, null-deref paths guarded, named intermediates
for suspicious-argument checks.
Martin Vogel