The long-lived MCP server ran the full index pipeline in-process on two
background paths -- the session auto-index (autoindex_thread, mcp.c) and the
watcher re-index (watcher_index_fn, main.c). The parallel pipeline's worker
threads abandon their mimalloc pages at thread exit, and mimalloc v3 defaults
page_reclaim_on_free=0, so those pages are never reclaimed when the main thread
later frees their blocks (mi_collect cannot touch abandoned pages either). RSS
then ratchets across re-index cycles and never comes back down (#832).
The index_repository tool path already avoided this by running the index in a
supervised worker subprocess: on child exit the kernel reclaims 100% of the
child's RSS. Route the two background paths through that same supervised runner:
- supervisor_invalidate_store() is now NULL-safe (the watcher path has no MCP
server / cached store).
- New static index_run_supervised_path() builds {"repo_path": root} and reuses
the existing spawn + skip-and-continue recovery loop (no duplication). New
public cbm_mcp_index_run_supervised_path() is the srv-less entry main.c calls.
- autoindex_thread and watcher_index_fn gate on cbm_index_supervisor_should_wrap():
supervised on a marked host with the kill switch off, else fall through to the
unchanged in-process pipeline (kill switch / spawn-failure degrade). The
watcher keeps its non-blocking try_lock gate + g_shutdown check exactly; the
lock still serialises re-indexes. cbm_mem_collect() stays on the in-process
branch only.
Also fold in the in-process allocator fallback: cbm_mem_init sets
mi_option_page_reclaim_on_free=1, restoring v2 behaviour so any path that stays
in-process (kill switch, degrade, embedders) also reclaims abandoned worker
pages. The subprocess is the primary cure (child exit returns RSS every cycle);
this is the fallback.
Tests: a deterministic gating guard (test_mcp.c) proves the background paths now
spawn a worker child and index the fixture (RED when routed in-process:
spawn_count unchanged; GREEN after). The test-runner acts as a real in-process
worker when spawned as `cli --index-worker` (public APIs only, no production
seam). A non-gating prod-binary RSS reproduction (tests/repro/issue832_rss.py)
shows the supervised server flat at ~10 MB vs ~131 MB in-process.
Scope: JobObject, the #841 spurious-trigger fix, retention caps, budget tiering
and the backpressure busy-spin are follow-ups.
Refs #832, #841.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The index supervisor gate (cbm_index_supervisor_should_wrap) wrapped ANY
embedder of cbm_mcp_handle_tool: it resolved the CURRENT executable and
re-invoked it as '<self> cli --index-worker index_repository ...'. In a
test binary 'self' is the test runner, which ignores those args and
re-runs its suites instead, producing recursive spawn chains (observed
11-minute hangs, and multiplied kernel VM-map/memory load during the
2026-07-04 host panics). The gate is now opt-in: only a process that
called cbm_index_supervisor_mark_host() — the real binary's main(),
before any subcommand dispatch — may spawn workers. Embedders index
in-process by default; the test mains additionally pin
CBM_INDEX_SUPERVISOR=0 as belt-and-suspenders. Guarded by
index_supervisor_gate_requires_marked_host_issue845 (fork+alarm harness,
new spawn-count test hook; red on the old gate, green without the env
kill switch on the new one).
A supervised worker whose supervisor died also kept indexing as an
orphan: the parent-death watchdog only ran in MCP-server mode. run_cli
now starts the same watchdog (detached; on ppid change it only writes
to stderr and _exit(0)s, so it is safe outside server mode) when it
detects --index-worker. Guarded by tests/test_worker_watchdog.sh
(scripts/test.sh Step 5b): a CBM_TEST_HANG_ON busy-spin pins the worker
mid-index, the wrapper parent is killed, and the worker must self-exit
within 15s — red without the wiring, green with it.
Fixes#845.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
New src/foundation/subprocess.{h,c}: spawn a child, tail its output, and
classify how it ended — clean / exit-nonzero / crash / hang / killed — from
POSIX WIFSIGNALED/WTERMSIG and the Windows NTSTATUS exception exit codes
(0xC0000005 access violation, 0xC00000FD stack overflow). Adds an EINTR-safe
reap loop, a quiet-timeout that kills and reports a hung child (no new output
within the window), and partial-line-safe log tailing.
Generalized from the ad-hoc index spawn in src/ui/http_server.c so the
crash/hang supervisor can reuse one primitive across platforms. The exit-code
to outcome mapping lives in a pure cbm_proc_classify() so the Windows
crash-code path is unit-tested on every platform, not just an untested
Windows branch.
tests/test_subprocess.c: 13 tests — the classifier on all platforms plus real
POSIX spawn/reap of clean, non-zero, SIGSEGV-crash and hang children. Green
under ASan/UBSan.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
index_repository silently dropped files that failed to index: CBMFileResult.has_error
was set but never read, oversized files (>100 MB) were dropped with no signal, and
read/extract failures only bumped a logged-only counter — a file that couldn't be
indexed just vanished from the graph.
Collect and surface them:
- New retained {path, reason, phase} error list on struct cbm_pipeline (mirrors the
excluded_dirs pattern) + accessor + a back-pointer on cbm_pipeline_ctx_t so both
extraction paths can append. Wired in BOTH the sequential (pass_definitions) and
parallel (per-worker, merged lock-free) paths — small repos take the sequential
path, so wiring only one would leave the guard vacuous.
- Feeds: read-fail, extract-fail, the newly-CONSUMED has_error (parse timeout /
parse failed, with error_msg), and oversized. The cross_lsp phase is reserved for
the crash supervisor (Track C) and not fed here (the cross-LSP passes are
best-effort with no failure return; feeding the no-source case would be false
positives).
- MCP/CLI response gains top-level "skipped_count" (0 on clean) and, when >0, a
capped "skipped":{files[<=50],count,truncated} + "logfile". Status stays "indexed"
— a reported skip is the expected, handled outcome, not a degradation.
- Per-run logfile (full uncapped list) written ONLY when >=1 file is skipped:
$CBM_INDEX_LOG override else <cache_dir>/logs/<project>-<epoch>.log.
- Generous env-configurable caps (src/foundation/limits.c): CBM_MAX_FILE_BYTES,
default raised 100 MB -> 512 MiB; over-cap files are REPORTED (phase oversized) +
WARNed, never silently dropped.
Reproduce-first: tests/test_index_resilience.c (gating) — an oversized file (cap
lowered via env) must appear in skipped[] with the 2 good files still indexed and a
logfile written; a clean run has skipped_count 0 and no logfile. Genuine guard:
no-op'ing the recorder makes the oversized file silently vanish (RED). Full suite
5750/0, no ASan/UBSan.
Part of the resilient-indexing effort (Track B surfacing layer). Refs #668.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
detect_changes returned an empty impacted_symbols set for projects indexed
inside a git worktree or from a repo subdirectory (#659): canonical_root was
computed wrong. `git rev-parse --git-common-dir` emits a path relative to the
directory passed via -C (input_path), NOT to worktree_root, and the "/.git"
suffix was stripped textually without resolving ".." components. So e.g.
"../.git" joined against worktree_root, or "/ws/scripts/../.git" strip, left an
unresolved ".." and never matched the project's stored root path.
Fix (src/git/git_context.c): join the relative --git-common-dir against
input_path (the -C dir) instead of worktree_root, and realpath()/_fullpath()-
normalize the result BEFORE stripping "/.git" (the .git dir always exists for a
valid repo, so resolution succeeds). This is consistent with the existing
realpath'd root_path used for prefix matching (mcp.c).
Reproduce-first: tests/test_git_context.c. The subdirectory case is the genuine
guard — a repo indexed from a subdir yields a *relative* --git-common-dir, so the
unfixed code leaves an un-normalized "<root>/subdir/.." (verified RED; GREEN after
the realpath fix). The linked-worktree case is a supporting invariant, not the
#659 reproducer: git that emits an *absolute* worktree --git-common-dir (e.g.
2.48.x) doesn't manifest the bug, so it passes either way; it still enforces the
worktree->main-root invariant. All three git-based tests SKIP_PLATFORM on Windows
(the CI shell there cannot init a repo via system()).
Distilled from #672 (thanks @anivaryam); that PR's production fix is taken
verbatim. Its repo_root/subdir tests ran git unconditionally and failed the
Windows CI leg ("failed to init git repo"); here all three git-based tests are
Windows-skipped so ci-ok is green cross-platform. Supersedes #696, which only
realpath'd after the strip without changing the join base and so still yielded
the workspace parent for the subdirectory case.
Closes#659
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-Authored-By: anivaryam <anivaryam.dev@gmail.com>
The CI lint gate (scripts/lint.sh --ci, clang-format-20) flagged 7 files whose
new lines were not run through the project formatter. Reformat only those lines
(line wrapping / continuation alignment) — no behavior change.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Route QNs (__route__METHOD__/path) are the rendezvous key for joining a client
HTTP_CALLS to the server handler that serves it, but they were built from the raw
path. Frameworks write path parameters with different placeholder syntaxes
(":id" Express/clients, "{id}" Axum/Spring/OpenAPI, "<id>" Flask/Rocket), so a
client call and its handler produced different QNs and never joined. Static
routes joined; parameterized routes (most of a REST surface) did not.
Add cbm_route_canon_path() which collapses each parameter token (":name",
"{name}", "<name>", "${...}") to a single "{}" token, and apply it to the path
at every HTTP-route QN construction site (pass_route_nodes, pass_parallel,
pass_calls, pass_cross_repo). Broker/async/infra QNs are left untouched. Only the
QN (identity/dedup key) is canonicalized; the Route node display name keeps the
original path. Parameter names are intentionally discarded so the same logical
endpoint matches across services that name the variable differently.
Adds tests/test_route_canon.c (11 cases). Fixes#505.
Signed-off-by: imateusdev <imateusdev@gmail.com>
Compare persisted SQLite node counts to in-memory dump counts after
index_repository completes so partial WAL/durability loss surfaces as
status:"degraded" instead of silent indexed.
Signed-off-by: Sam Li <yangsec888@gmail.com>
The web server behind the graph UI is now a purpose-built in-house
module (src/ui/httpd.c): localhost-only listener, strict HTTP/1.1
parsing with fixed request caps, a per-connection receive deadline,
and Connection: close semantics. http_server.c keeps the routing and
handlers, rewritten against the new transport API; the public server
API and main.c are unchanged. The previously vendored third-party
server is removed entirely.
Comes with a new 28-test transport + routing suite (tests/test_httpd.c)
covering parsing edge cases (strict CRLF, Content-Length limits, raw
path matching, percent-decode rules) and live-socket behavior (CORS
policy, RPC dispatch, receive deadline, clean shutdown). The security
audit scripts now check the new file layout and treat any network call
in vendored code as a failure.
Final 4 probe suites (grammar_probe_d/e/f/g, 250 cases) bring node/edge-creation coverage to all 159 grammars. Triage fixed 4 fixture mislabels (objc Method label, smithy $version order, json non-ignored filename, sql DDL-only Variables) and confirmed ~46 real reds. Findings map to known classes 2/3 (import/include-edge resolution missing for ~34 grammars: agda/bicep/elm/func/move/nasm/pony/purescript/qml/smali/tlaplus/vhdl/protobuf/thrift/capnp/wit/smithy/astro/nix/nickel/pkl/jsonnet/hyprlang/devicetree/cmake/makefile/meson/gn/just/bitbake/kconfig/html/css/scss), 16 (node-extraction incompleteness: wolfram multi-def/set_top, wit world funcs, bicep resource, just recipe), plus infra gaps (INFRA_MAPS hcl/k8s only from YAML topic bindings; DEPENDS_ON only from Helm Chart.yaml — go.mod/requirements produce none). NEW class 17 = file-index routing gap: .env/.gitignore/.gitattributes/.re have no EXT_TABLE/FILENAME_TABLE entry in discover/language.c so index_repository never indexes them (direct extraction works). Vue/Svelte/Astro component imports: extraction works but JS/TS path resolver doesn't handle those extensions. All 159 grammars now have reproduce-first guards.
Extends the node/edge-creation hunt to ~35 grammar-only code languages (129 graph/extraction cases, 73 green guards / 56 real RED reproductions; triage confirmed 0 fixture errors — all reds are genuine product gaps). Findings map to: class 2 import-EXTRACTION missing (cbm_extract_imports dispatch in extract_imports.c covers only ~38 langs + default:break, so ~25 grammar-only langs' configured *_import_types are never consumed → 0 imports); class 1 INHERITS missing (crystal/julia/pascal/powershell/squirrel/solidity); and NEW class 16 = per-grammar NODE-extraction incompleteness (class_types configured + grammar node present, but extract_defs.c emits 0 type nodes for D struct/class, F# record/union, Gleam type, Julia/Odin struct, Pascal record, PowerShell class, Hare struct, ReScript type, squirrel class, sway struct/abi, tcl namespace, wgsl struct). Affected-grammar matrix in the suite comments. Brings code-grammar coverage to ~61 of 66.
Final hunt batches (convergence_probe 47, matrix_known_classes 44, matrix_new_constructs 63 = 154 graph-level cases) plus triage. Result: ZERO genuinely-new root-cause classes — every confirmed red maps to the already-known taxonomy (classes 1,4,5,6,7,12,13,14). Triage greened 5 fixture mis-builds: recursion self-calls (Go/Python/Rust/Java) produce 0 CALLS because self-loop edges are SUPPRESSED BY DESIGN (pass_calls.c/pass_parallel.c src->id==dst->id) — not a bug, fixtures now add a non-self caller; Ruby bare 'describe' parses as identifier not call (fixed to describe()). 26 reals kept as reproductions with // REAL BUG: root-cause comments. Confirms the bug hunt has converged: aggressive probing across new constructs + more languages surfaces only instances of the ~15 mapped classes.
Reproduce-first probes (215 graph-level cases) hunting node/edge-creation + LSP-pass bugs across 26 languages + web frameworks. node_creation_probe: 80/81 green (node creation is solid; only Go method-on-struct under threshold). lsp_resolution_probe: 72/83 green — RED reproductions for the cross-LSP dispatch gap (Rust/Kotlin S1 cross-file call) and constructor/static/virtual/generic resolution edge cases (C/C++/TS/Java/C#/PHP S3-S7). edge_types_probe: 29/51 green — candidate RED reproductions for HANDLES (7 web frameworks), HTTP_CALLS (5 clients), WRITES (all 5 langs), Kotlin throws/raises, Go DEFINES_METHOD/OVERRIDE (these need real-bug-vs-fixture verification). Green cases retained as regression guards throughout. No unrelated suite regressed.
Reproduce-first coverage of the pipeline step that turns extracted symbols/imports into resolved graph edges, across the 9 hybrid-LSP languages (91 graph-level cases). Live RED reproductions: IMPORTS edges are never created for Rust/Kotlin/Java/C#/PHP (all forms) despite working extraction (+ Python aliased/wildcard, TS re-export edge cases); cross-file CALLS do not resolve for Rust and Kotlin (lsp_cross pass processes 0 files for them) — missed by real-repo checks because same-file calls dominate; same-file IMPLEMENTS edges absent for Java/C# and Go implicit-interface satisfaction; same-file DECORATES absent for TS and C# attributes; cross-file USAGE absent for TS; cross-file INHERITS absent for TS/PHP/Kotlin/Python (extraction-propagated). Green guards retained for every working path (Go/Python/TS imports, cross-file CALLS for 7 langs, INHERITS for Java/C#/C++, etc.). 51 RED reproductions; no other suite affected.
Reproduce-first coverage of class-inheritance (base_classes) and import extraction across the 9 hybrid-LSP languages (~300 table-driven cases). Live RED reproductions: TS/TSX store the extends/implements keyword instead of the base type; PHP never populates base_classes and its use-imports parse the wrong node type; Kotlin does not parse the : supertype list; Python loses Generic[T] (paren/bracket leak); C++ drops qualified bases (std::vector) and leaks ':'; Rust misses several impl-Trait-for-T forms; C# import aliases (using F = X) and Python wildcard/__future__ imports are missed. C# inheritance and Go/TS/JS/Java/Kotlin/Rust import extraction pass as regression guards. Suite is RED until the extractors are fixed; 12 failing reproductions, no other suite affected.
scan_start_tag_name moves scan_tag_name's owned tag_name into a Tag, but the CF_SET/CF_RETURN/CF_ELSEIF/CF_ELSE cases return without pushing the tag to the scanner stack and (unlike CF_VOID) without freeing it — leaking the tag_name buffer under LeakSanitizer. Free the tag in those return paths, matching CF_VOID. Removes the need for the __lsan_default_suppressions(scan_tag_name) workaround in the test runner.
scan_tag_name in the shared vendored HTML-family tree-sitter scanner leaks a small bounded tag-name buffer in an edge path of scan_start_tag_name. tag_for_name's ownership is grammar-specific and not verifiable without HTML-family parse coverage, so the bounded leak is suppressed via __lsan_default_suppressions rather than risking a blind double-free. The larger SQL scanner leak was fixed properly.
cbm_kind_in_set builds a thread-local cache of calloc'd symbol bitsets (ks_cache) that was never freed, so LeakSanitizer (Linux x64) reported the per-worker-thread caches as leaks once the previously-empty grammars gained real node-type sets. Add cbm_kind_in_set_free_cache() and call it at worker-thread teardown (extract_worker, beside cbm_slab_destroy_thread) and main-thread exit (test runner, beside sqlite3_shutdown).
Index a fixture for every grammar through the full pipeline and assert graph-level invariants: golden node-label histograms, def/node breadth, CALLS and IMPORTS breadth, presence of all 26 pipeline edge types (structural, type-relationship, service/dataflow, and similarity), no-crash, and call resolution. Reproduced quality gaps (under-extraction, import-extraction, and CALLS-resolution) are kept as hard failures for the fix phase rather than skipped. Adds a real-repo scale tier (scale_contract.sh) and ignores local-only private/ scratch.
Hybrid tree-sitter + type-resolution call resolver for Rust, alongside the
Go/C/PHP/C#/Python/TypeScript/Java/Kotlin resolvers. Resolves method dispatch
on typed receivers, UFCS and associated-function calls, trait/impl methods,
generics, closures, the `?` operator, and iterator adapters; seeds the standard
library and well-known crates, parses Cargo.toml for dependency context, and
expands a curated set of derive/proc-macro generated methods.
Includes rust_cargo (manifest parsing), rust_proc_macros (derive expansion),
rust_rustdoc (doc extraction), a generated stdlib + crates seed, and a large
test suite (registered as suite rust_lsp). Wired into the per-file dispatch.
Field-access chains through Self-returning calls remain a documented coverage
gap. Full suite green (4666 tests).
Data-driven net (tests/test_grammar_regression.c) that extracts a minimal
sample per language and asserts a catastrophic-break floor (defs >= min) plus
expected definition names across 35 languages. Guards against silent
extraction breakage on future vendored-grammar refreshes — the failure mode
where a renamed/removed node type drops extraction to zero defs (as the
tree-sitter-kotlin name-field change did). Extend by appending a row to CASES.
Hybrid tree-sitter + type-resolution call resolver for Kotlin, matching the
existing Go/C/PHP/C#/Python/TypeScript/Java resolvers. Resolves intra-file
and imported calls, method dispatch on typed receivers, extension functions,
operator conventions (plus/compareTo/contains/get/unary), scope functions and
trailing lambdas (it/this), smart-casts (is / when), super dispatch,
constructor-val properties, and stdlib defaults.
Targets the refreshed tree-sitter-kotlin grammar: discovers declarations via
simple_identifier/type_identifier, walks the statements wrapper, extracts
method names from navigation_suffix, descends call_suffix for trailing
lambdas, reads delegation_specifier for inheritance, class_parameter for
constructor fields, and import_list for imports.
Wires the resolver into the per-file dispatch and registers its test suite
(78 tests). Full suite green (4160 tests).
Integrates the type-aware Java semantic resolver from
worktree-adding-java-lsp-support (based on 673ac4e): java_lsp.c + generated
java.lang/util stdlib registry, wired into the per-file LSP pass, with 94
java_lsp tests + ~170 java_lsp_coverage tests.
Wiring conflicts (the shared files diverged 146 commits on main) resolved by
keeping main's existing LSP registrations (go/c/php/py/ts/cs) and adding the
Java entries in lsp_all.c, cbm.c (cbm_run_java_lsp dispatch), test_main.c and
Makefile.cbm.
Adapted to the current unity-build: renamed java_lsp.c's static helpers
node_text/return_type_of to java_node_text/java_return_type_of so they no
longer collide with ts_lsp.c inside the shared lsp_all.c translation unit.
4082 tests pass (incl. java_lsp + java_lsp_coverage); ASan/UBSan clean.
Add a C# language LSP resolver mirroring the existing Go/C/PHP/Python
modules: type evaluation, using-directive resolution, method lookup,
and a corpus-seeded .NET BCL stdlib data table.
- internal/cbm/lsp/cs_lsp.{c,h}: resolver implementation
- internal/cbm/lsp/generated/cs_stdlib_data.c: BCL type/method tables
- wire CBM_LANG_CSHARP dispatch in cbm.c and lsp_all.c
- register cs_lsp + cs_lsp_bench suites in the test harness
- tests/test_cs_lsp.c, tests/test_cs_lsp_bench.c
(cherry picked from commit 8f677c67a0c9d7829cd6ac27393de2d0aaaac534)
cbm_store_checkpoint() invoked SQLITE_CHECKPOINT_TRUNCATE, the most
aggressive mode. When two cbm-mcp processes share a cache dir, one
process's TRUNCATE can shrink files while another has them mmap'd,
raising SIGBUS on macOS. PASSIVE never blocks readers and never
ftruncate()s either file; SQLite still autocheckpoints in PASSIVE
mode at 1000-page boundaries, so reclamation is unaffected for
single-process users.
Recommended by SQLite docs for shared databases:
https://www.sqlite.org/pragma.html#pragma_wal_checkpoint
Hard-coded `PRAGMA mmap_size = 67108864` in configure_pragmas() left
no path for users running multiple cbm-mcp instances against the
same cache to opt out of memory-mapped I/O. On macOS, when one
instance's checkpoint or reindex truncates the DB file under
another instance's live mmap, accessing the now-missing pages
raises SIGBUS, taking the process down.
Setting CBM_SQLITE_MMAP_SIZE=0 reverts to read()/pread() I/O,
which returns recoverable SQLITE_IOERR instead of crashing the
process.
- Default unchanged (67108864 / 64 MB). No behavior change for
single-instance users.
- Malformed values (non-numeric, partial-numeric) fall back to
the default rather than failing the store open.
- Negative values clamp to 0.
- New tests: tests/test_store_pragmas.c covers all five resolver
paths plus an integration smoke that opens a file-backed store
with mmap disabled.
Empirical evidence: 9 SIGBUS crash reports collected on macOS
arm64 v0.6.0 in a 14-hour window, all signature 'cluster_pagein
past EOF' with stacks bottoming in SQLite btree code under the
watcher thread's incremental-index pipeline.
Resolved 5 conflicts where the php-lsp branch and main both added LSP
infrastructure but with different APIs:
- Makefile.cbm: kept both TEST_PHP_LSP_SRCS and main's py_lsp/scope/
type_rep/stack_overflow test source variables in ALL_TEST_SRCS.
- internal/cbm/cbm.c: kept both #include lsp/php_lsp.h and lsp/py_lsp.h;
the per-language dispatch already supports stacking PHP after PYTHON.
- internal/cbm/lsp_all.c: kept both lsp/php_lsp.c + php_stdlib_data.c
AND lsp/py_lsp.c + python_stdlib_data.c in the compilation unit.
- src/pipeline/pass_calls.c: took main's version entirely.
find_lsp_resolution + resolve_single_call(ctx, call, lsp_calls, ...)
is language-agnostic — it matches any CBMResolvedCall whose caller_qn
hits the textual call's enclosing function and whose callee_qn
short-name hits the textual callee. PHP LSP populates the same
result->resolved_calls array via cbm_run_php_lsp, so the unified
resolver handles PHP for free. Dropped the php-branch's
qn_short_name + lsp_override_resolution helpers as superseded.
Locked confidence floor at main's CBM_LSP_CONFIDENCE_FLOOR = 0.6f
(was 0.5f on the php branch).
- tests/test_main.c: kept all suite externs and RUN_SUITE invocations
for php_lsp + py_lsp/py_lsp_bench/py_lsp_stress/py_lsp_scale +
scope + type_rep.
Auto-merged: src/pipeline/pass_parallel.c.
Brings in 52 commits from main since the branch was forked at c768b61
(#207 watcher follow-up): #208 through #281, dependabot bumps, Pine
Script, AUR docs, INHERITS edges, get_architecture wiring, codeql
revert, the entire python-lsp subsystem.
Suite: 3249 passed, 0 failed (was 2971 on main; +278 new tests from
php-lsp matches the branch's claimed test count).
Audit for O(n^2) hot paths after Round 10. The dedup loop in
py_emit_resolved_call scanned ALL prior entries linearly, making
emission O(N) and total file work O(N^2). On a synthetic 2000-class
fixture with 8000 calls, that dominated time at 5083 ms.
Cap the scan window at the last 256 entries. Most duplicate
emissions come from the same expression being walked twice (resolver
+ emitter passes), so they're temporally adjacent — a 256-entry
window catches every common case while keeping the dedup O(1) per
emission.
Scale measurements (test_py_lsp_scale.c added):
| Classes | Calls | Resolved | Before | After |
|--------:|------:|---------:|-------:|------:|
| 100 | 400 | 300 | 55ms | 57ms |
| 500 | 2000 | 1500 | 333ms | 323ms |
| 2000 | 8000 | 6000 | 5083ms | 1804ms|
Ratio for 20x input: 91.8x -> 31.4x (linear would be 20x; quadratic
would be 400x). No remaining quadratic hot path in code added by
this branch.
Other hot paths surveyed:
- py_register_def -> cbm_registry_lookup_type: O(types) per def at
registration, O(D x types) for D defs. Pre-existing linear scan in
type_registry.c; hash-indexing it would be a cross-language
optimization (Go and C++ LSPs benefit too).
- py_lookup_attribute / py_lookup_field: O(types) per call site.
Same pre-existing linear scan. With ~900 stdlib types + per-file
defs, accounts for ~75µs/call at 2000-class scale.
- MODULE submodule prefix scan: O(funcs) per module-attribute access,
early-exits on first match. Bounded by registry size.
- lambda / dict-literal registries: linear, but bounded by per-file
N which is small.
Bench unchanged at 100% / 12ms (52/52 calls, 178 LOC).
Round 8. Stress test suite (test_py_lsp_stress.c) added with 22 advanced
patterns; bench fixture grew to 178 lines covering the full surface.
with-as binding via __enter__:
- Look up __enter__ on the context manager's NAMED type. Use its
return type when known; fall back to the manager type itself
(matches the practical case where __enter__ returns self). Handles
tree-sitter Python's wrapping of `X as y` in an as_pattern node
inside with_item, plus the flat field-only shape.
except-as binding:
- Bind exception alias to the caught type. Resolves through
except_clause's as_pattern wrapper and the as_pattern_target
identifier wrapper.
Tuple unpacking:
- Both `a, b = f()` and `for k, v in d.items()` destructure the RHS
element-by-element when the rhs is TUPLE or TEMPLATE("tuple", ...).
*rest binds the trailing slice as list[elem]. Shared helper
py_bind_for_target used by for-statement, comprehension for_in_clause,
and assignment LHS.
Slice subscript:
- `lst[1:3]` returns the same TEMPLATE container — list[T][1:3] is
list[T], dict[K, V][1:3] is dict[K, V] (rare but consistent).
Container TypeVar substitution:
- Without a constraint solver we can't generally substitute K, V into
registered method return types. Hand-rolled the most-impactful
cases: dict.items/keys/values/get/pop/copy, list.copy/__iter__/pop,
set.copy. These return properly parameterized ItemsView[K, V] /
KeysView[K] / ValuesView[V] / Optional[V] / etc. so subsequent
iteration and unpacking yield typed elements.
ItemsView / KeysView / ValuesView / dict_items / dict_keys / dict_values
added to py_iterable_element_type. ItemsView[K, V] iterates as
tuple[K, V], unpacked correctly by py_bind_for_target.
Bench: 178 LOC, 52 calls, **52 resolved (100%)**, 11.59 ms (ASan + UBSan).
22 new stress tests, 8 of 10 documented gaps closed in this round.
Remaining gaps documented inline (typeddict_subscript, post-early-return
narrowing, function-as-dict-value, match sequence pattern, lambda
inference). All 2898 prior tests stay green; total 2920 / 0.
Final phase of Python LSP integration. Adds:
- docs/BENCHMARK_PYTHON.md — methodology document covering target
metrics (per-file LSP overhead < 10 ms, resolution ratio ≥ 40% on
application code, ≥ 70% with stdlib, < 5% spot-check false positive
rate), the declared accepted-failure-class ceiling (metaclass
__call__, __getattr__/__getattribute__, dynamic setattr, runtime
importlib, monkey-patching, eval/exec, descriptors beyond the
standard property/classmethod/staticmethod, dynamic type() classes),
and instructions for running scripts/benchmark-index.sh against the
largest Python repos in datadice (falkemedia: 232 files, api-hub:
188 files).
- tests/test_py_lsp_bench.c — in-process benchmark wired into
scripts/test.sh. Runs cbm_extract_file on a 65-line Python fixture
covering imports, dataclasses, classmethods, single-inheritance with
super(), typed parameters, attribute chains, and stdlib calls.
Asserts soft floor (resolved >= calls/4) and hard time budget
(< 50 ms under sanitizers). Initial run reports:
bench: 65 lines, 16 calls, 13 resolved (81%), 2.45 ms
81% beats the 40% application-code target and the 70%
application-plus-stdlib target. 2.45 ms under ASan + UBSan is well
inside the 10 ms (no-sanitizer) production budget.
This wraps the 11-phase plan documented in PYTHON_LSP_PLAN.md. Total
test count: 2874 / 0 (100% pass). Repository-level benchmarks via
scripts/benchmark-index.sh remain a manual operation against a fresh
production binary; methodology is now in place for when that runs.
CALLS edges for cross-file function calls were mis-wired or missing in
TypeScript projects that use path aliases (e.g. @/lib/auth → src/lib/auth):
1. resolve_relative_import only handled "./relative" paths, so
"@/lib/auth"-style imports never reached IMPORTS edges. The import
map stayed empty, Strategy 1 in cbm_registry_resolve failed, and
resolution fell through to name-based lookups that pick the wrong
target when functions share names across files.
2. resolve_import_map built the candidate QN as the bare module QN
when the callee had no dot ("requireAdmin" vs "pkg.Func"), instead
of module_qn.requireAdmin. Even with a populated import map, bare
calls still mis-resolved.
Fix:
- New src/pipeline/path_alias.{h,c}: language-agnostic build-tool
path-alias subsystem. Currently parses tsconfig.json and
jsconfig.json compilerOptions.paths + baseUrl via the existing
yyjson dependency (with ALLOW_COMMENTS|ALLOW_TRAILING_COMMAS for
JSONC). The data model (cbm_path_alias_t / map / scope /
collection) is generic; adding Vite/Webpack/Python loaders means
writing a new file-parsing helper and registering it in
find_alias_files — the resolver, collection, and pipeline
integration do not change.
- O(n log n) ordering: scope-by-dir-specificity and entry-by-
prefix-specificity are sorted with qsort, replacing the O(n²)
bubble sorts that the original draft used. With caps of 256
entries per config and 256 configs per repo, the sort is
effectively flat.
- Resource ceilings with logging: directory recursion capped at
32 levels; total config files capped at 256; per-config size
capped at 64 KB. Cap hits emit cbm_log_warn so silent truncation
on pathological monorepos shows up in the index log.
- Walk hygiene: skips node_modules, .git, dist, build, .next,
coverage, target.
- Single integration point: cbm_pipeline_resolve_module gets a
Strategy 1b between relative-import resolution and pkgmap so both
the sequential and parallel pipelines pick up alias resolution
automatically. Non-TS projects pay nothing — the loader returns
NULL when no usable configs are found.
- Layer 2 fix in registry.c: resolve_import_map now builds the
candidate as resolved.prefix when suffix is NULL (one-line fix,
matching comment).
- Tests: 10 path_alias tests (basic wildcard, specificity, exact,
extension stripping, baseUrl, NULL safety, nearest-ancestor scope
selection, end-to-end loader against a tmp monorepo, no-configs
case) plus a registry test for the bare-call fix. Full suite:
2835 passed, 0 failed.
Cherry-picked and generalized from #243 (originally TypeScript-only
under tsconfig-named types). Renamed types (cbm_path_alias_scope_t /
collection_t in place of cbm_tsconfig_*) and centralised the load
into cbm_load_path_aliases so future loaders for other build tools
slot in without touching extraction or pipeline code.
Closes#243.
Co-authored-by: Peter Cox <info@petercox.ie>
Covers parameter binding (typed param, arrow function, $this, catch),
method dispatch (typed receiver, chain, self/parent), static calls,
namespace + use resolution, PHPDoc @var, magic methods, and the
ConfiguresPrompts $prompt->value() regression from PHP_LSP_PRE_FLIGHT.md.
Wired into ALL_TEST_SRCS and suite_php_lsp invocation in test_main.c.
All 16 tests pass; total suite is now 2829/0 passing.
Phase 2 of Python LSP integration. Adds the py_lsp module mirroring
the go_lsp / c_lsp pattern: PyLSPContext, single-file entry
(cbm_run_py_lsp), cross-file + batch shells, stdlib registration stub.
Bodies are no-ops at this phase. Subsequent phases fill in:
Phase 3 — imports
Phase 4 — scope binding
Phase 5 — expression typing
Phase 6 — attribute resolution + method dispatch
Phase 7 — decorators
Phase 8 — class hierarchy + generics
Phase 9 — cross-file resolution + batch
Phase 10 — cbm_python_stdlib_register from typeshed
Wired into cbm_extract_file alongside the existing Go and C/C++ LSP
calls. lsp_all.c #includes py_lsp.c so the CGo umbrella picks it up
without a separate translation unit.
tests/test_py_lsp.c adds 5 smoke cases: empty source, single function,
single class, syntax error recovery, imports pass-through. The helper
shape (extract_py / find_resolved / require_resolved) matches
test_go_lsp.c so future phases can grow the suite by category.
All 2839 existing tests stay green; 5 new py_lsp tests pass.
Phase 1 of Python LSP integration. Five new CBMTypeKind variants needed
for Python type modeling, plus structural-equality helper used by union
deduplication and protocol matching:
- UNION: A | B / Union[A, B] / Optional[T]. cbm_type_union normalizes
by flattening nested UNIONs, deduplicating members via cbm_type_equal,
and collapsing single-member unions to their member. Empty unions
reduce to UNKNOWN.
- LITERAL: Literal["foo", 3] — wraps a base type with literal text;
text-distinct literals compare unequal.
- PROTOCOL: typing.Protocol with method-name set. Structural matching
via cbm_type_protocol_satisfied_by — succeeds when the candidate's
method set is a superset of the protocol's.
- MODULE: bound name from `import X`, distinguishing module-attribute
lookup from class-attribute lookup. Equality is by module_qn.
- CALLABLE: Callable[[A, B], R] including the elliptic Callable[..., R]
encoded as param_count = -1.
cbm_type_equal added as a recursive structural-equality helper covering
all kinds; FUNC/INTERFACE/STRUCT fall back to pointer identity since
their structural members are rarely compared and the cost would be
substantial.
Adds tests/test_type_rep.c with 16 cases covering normalization,
equality, and protocol satisfaction.
All 2823 existing tests stay green; 16 new type-rep tests pass.
AST traversal functions used fixed-size TSNode stack[] arrays. When the
DFS stack filled up, the child-push loop exited silently, dropping
entire subtrees without warning (e.g. 600 TS imports → 511, 130 Express
routes capped at 512). Reported in #199; also addresses #213 (large
TypeScript files producing zero nodes from stack exhaustion) and #215
(SEGV in template calls from stack overflow).
Adds ts_node_stack.h — a growable stack backed by the existing arena
allocator. Initial capacity matches previous fixed caps so small files
allocate no extra memory; doubles on overflow instead of truncating.
Old blocks are abandoned in the arena and freed on arena_destroy at
the end of file extraction (no realloc, no mixed lifetimes).
Applied to all 14 TSNode fixed stacks across 9 extraction files
(extract_calls, extract_channels, extract_defs, extract_env_accesses,
extract_imports, extract_semantic, extract_type_assigns, extract_type_refs,
extract_usages). walk_defs (uses walk_defs_frame_t, different struct
type) deliberately left as-is.
7 regression tests in tests/test_stack_overflow.c cover TS imports >512,
JS/Python calls >512, Go calls >1024, Express routes >150, deeply
nested calls, and YAML vars >256.
Cherry-picked from #217 with merge-conflict resolution against post-#206/#207
main:
- Makefile.cbm: keep TEST_STACK_OVERFLOW_SRCS plus main's new
TEST_ZSTD_SRCS / TEST_ARTIFACT_SRCS.
- extract_channels.c: keep main's scan_string_consts_js name (main also
added scan_string_consts_python after the PR was opened); declare
CHAN_STACK_CAP in the enum block.
- test_stack_overflow.c: replace sprintf with snprintf using the actual
remaining buffer size, since macOS clang rejects sprintf and naive
snprintf with sizeof(p) on a char* fails -Wformat-truncation.
Closes#199, #213, #215. Closes#217.
Phase 0 of Python LSP integration. Two shared-infrastructure fixes that
silently bug Python without affecting Go/C++ regression tests:
- CBMScope replaces the fixed 64-binding array with a growable linked
list of 16-binding chunks. The arena pointer is captured at scope_push
and reused for chunk allocation, so existing callers benefit from
dynamic growth without an API change. Python class bodies and large
dataclass scopes routinely exceed 64 locals; this prevents silent
drops past the legacy cap.
- CBM_LSP_MAX_LOOKUP_DEPTH=16 wraps go_lookup_field_or_method and
c_lookup_member via depth-aware internal helpers. Pathological
multi-inheritance hierarchies (SQLAlchemy / Django / Zope) can exceed
20 levels and previously stack-overflowed; now they degrade to NULL
past depth 16 instead.
Adds tests/test_scope.c with 10 cases covering basic API, parent-chain
walks, child-shadowing, and dynamic growth at 300 bindings + chunk
boundaries. test_main.c and Makefile.cbm wire the new suite.
PYTHON_LSP_PLAN.md added at the worktree root: full phased plan plus
the verbatim challenger review.
All 2823 existing tests stay green; 10 new scope tests pass.
Add .codebase-memory/graph.db.zst — a zstd-compressed knowledge graph
artifact that can be committed to the repo. Teammates bootstrap from
the artifact instead of running a full reindex from scratch.
- Vendor zstd 1.5.7 (amalgamated build) for 8-13:1 compression
- Two-tier export: zstd -9 + index stripping for explicit index,
zstd -3 for watcher/incremental auto-updates
- Import: decompress → integrity check → auto-recreate indexes
- Bootstrap in handle_index_repository: when no local DB exists but
artifact is present, import first then run incremental
- Auto-create .gitattributes with merge=ours to prevent conflicts
- Fix: add missing idx_edges_url_path to create_user_indexes and
url_path_gen generated column to init_schema
- 13 new tests (5 zstd wrapper + 8 artifact round-trip/edge cases)
Compute K=64 MinHash signatures from normalized AST node-type trigrams
during function extraction, then generate SIMILAR_TO edges via LSH
(b=32, r=2) for function pairs with Jaccard >= 0.95.
- src/simhash/minhash.{h,c}: MinHash compute, Jaccard, hex encode/decode,
LSH index with band hashing for O(n) candidate generation
- src/pipeline/pass_similarity.c: post-pass reads fingerprints from node
properties, builds LSH index, emits SIMILAR_TO edges with jaccard and
same_file metadata. Same-language only, max 10 edges per node.
- internal/cbm/cbm.h: fingerprint fields on CBMDefinition
- internal/cbm/extract_defs.c: compute_fingerprint() hook at 3 extraction
sites after complexity, skip functions with < 10 AST body nodes
- pass_definitions.c + pass_parallel.c: serialize fingerprint to "fp" hex
in properties_json for both sequential and parallel pipeline paths
- pipeline.c + pipeline_incremental.c: register pass_similarity in both
full and incremental post-pass lists
- tests/test_simhash.c: 28 tests across 4 suites (core, LSH, edge gen,
pipeline integration with generated Go project + incremental)
CBM_SKIP_PERF=1 now skips only phases 2-7 (incremental delta, stress,
perf regression) inside the suite. Phase 1 (initial index) and phases
8-30 (135 tool integration tests) always run. This keeps ~2672 tests
on CI vs the full ~2692 locally.
Also fix Windows build: guard symlink() with #ifndef _WIN32.
- Add CBM_SKIP_PERF=1 env var to skip incremental/perf test suite
- CI and Docker test targets skip perf by default (run.sh perf for manual)
- Convert all perf assertions to warnings (log timing, never block)
- Fix store.h anonymous enum in struct (GCC rejects, clang accepts)
- Fix test_store_search.c mkstemp on non-template path
- Add ca-certificates to Docker test image for git HTTPS
- Add cbm_gmtime_r shim in compat.h (Windows gmtime_s wrapper)
- Fix compat.c missing constants.h include (Windows build)
- Fix platform.c _environ redeclaration on mingw
- Rename trace_call_path -> trace_path in smoke/soak/fuzz scripts
Comprehensive test suite using real repo (FastAPI 0.99.1, ~1100 files):
- Full index baseline with perf metrics (nodes, edges, RSS, timing)
- Incremental: modify/add/delete/rename files, noop, formatter run
- Adversarial: empty file, syntax errors, 5000-line function, binary
- Stress: rapid reindex, batch 20 files, delete-all + re-add
- Accuracy: incremental vs full within ±2 nodes/edges
- MCP tools: all 14 tools, every parameter, content validation
- Error handling: bad project, missing params for each tool
- Performance: 5s timeout on every tool call
Fix use-after-free in detect_changes and manage_adr:
yyjson_mut_arr_add_str stored pointer to stack buffer without copy.
Changed to yyjson_mut_arr_add_strcpy / yyjson_mut_obj_add_strcpy.
Martin Vogel