mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-10-19 05:40:03 +08:00
docs/cli-daemon-behavior
33
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Martin Vogel
|
d0ec2313d9 |
Merge pull request #2394 from DeusData/fix/issue-2184-post-extraction-budget
fix(pipeline): spill before the post-extraction phases when they would cross the budget (#2184) |
||
|
Martin Vogel
|
78a2ec399f |
test(parallel): rename the near graph buffer for MinGW
The #2184 parallel test named a local `near`. Windows headers define `near` (and `far`) as empty legacy pointer qualifiers, so under MinGW `cbm_gbuf_t *near = ...` loses its name and test_parallel.c fails to compile (test-windows CLANG64 2/2: "expected identifier or '('"). Rename it to `gb_near`. A MinGW syntax-only check of tests/test_parallel.c reported 4 errors before this change and none after. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
26cbccd998 |
fix(pipeline): spill before the post-extraction phases when they would cross the budget (#2184)
Spill mode was entered only DURING extraction, when the charge crossed
budget - budget/16, and the final sweep at extraction end ran only if that
latch had already fired. A run that ended extraction just under the latch
kept every result in memory, and the phases that cannot spill -- registry
build, cross-LSP prepare (all_defs, per-language registries, surface rows,
module index) and resolve (edges into the graph buffer, cross-LSP appends
into the cached results) -- then grew the process well past the budget.
openclaw at 8 workers with a 4079 MB budget: 3678 MB charged at extraction
end, 4076 after registry build, 4933 after cross-LSP prepare, 5829 in
resolve (+43 %). The same repo at 18 workers latched during extraction and
stayed under budget.
Extraction end now projects the post-extraction growth from the result
counts (defs, calls + usages, files) and enters spill mode when charged +
growth would cross the latch, so the existing final sweep parks every
cached result before registry build. The per-unit costs (1280 B/def,
160 B/ref, 8 KiB/file) are fitted on the measured charge growth from the
parallel_extract to the parallel_resolve mark on five local corpora
(go, django, kotlin, rust, php) and cover openclaw; the model under-reads by
at most 7 % (inside the budget/16 margin) and over-reads by up to 46 %, the
safe side: a spill never changes graph content. The decision is a pure
function of counts (no RSS-window sampling) and is logged as
mem.post_extract.projection.
A charge test seam (cbm_mem_set_charged_for_tests) pins cbm_mem_charged(),
which otherwise reads the process footprint, so the "ended just under the
latch" case is testable deterministically.
Proof (release build, M5 Pro, CBM_WORKERS=8, peak charged = the budget
metric's high-water mark):
go, CBM_MEM_BUDGET_MB=2000 before: extract 1348 -> registry 1497 ->
lsp_prepare 2168 -> resolve 2660 MB (+33 % over budget, no spill)
after: projection charged 1335 + growth 1909 > line 1875 -> spill,
21,882 results parked; extract 413 -> registry 558 -> lsp_prepare 1151
-> resolve 1297, semantic 1746 MB peak: every phase under budget.
Node/edge dumps byte-identical (335,191 nodes / 1,813,857 edges); wall
29.2 s vs 29.0 s.
go, 40000 MB (generous): decision=keep, no spill, identical dumps, wall
27.4 s vs 27.5 s.
django, 400 MB: resolve mark 455 -> 277 MB after the spill, dumps
identical; the remaining peak (476 MB) is the semantic pass's retained
set on top of the graph floor, which this change does not address.
Refs #2184
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
793e0b6127 |
fix(rust): never let a std-typed receiver bind a same-named project method (#2053)
The Rust LSP types `root: &Path` and dispatches `root.join(..)` to std.path.Path.join, a registered seed method with no graph node. Both call resolvers treated "LSP row without a gbuf target" like "no LSP row" and fell back to the textual registry, which bound the call to the only project method named `join` (EvidenceTier::join via unique_name). Every std/seeded-crate call whose leaf name a project method shares was fabricated this way. cbm_pipeline_rust_external_target() (lsp_resolve.h) recognises a Rust row whose strategy emits a registered target (method/trait/deref/bound dispatch, direct, ufcs, constructor) and whose QN lies outside the project prefix. Project defs are registered under project-prefixed QNs, so such a target is a std/core/alloc or seeded-crate symbol by construction. pass_calls.c and pass_parallel.c skip the registry fallback for those rows (in lockstep), and the parallel route-method fallback no longer turns a skipped `map.get(k)` into a source->source self-call. Project-prefixed rows, unknown-receiver rows and every other language are unchanged. The call-expression receiver (`root.to_path_buf().join(..)`) needed type facts: Path::join/to_path_buf/with_extension/with_file_name now return PathBuf, PathBuf::as_path returns Path, and PathBuf derefs to Path. Measured on BurntSushi/ripgrep @ 3fce3b5 (fresh index, two runs each, byte-identical): CALLS 5371 -> 5014; 357 removed (307 suffix_match, 45 unique_name, 5 field_type_hint), 0 added, no lsp_* edge touched. Removed samples are all std receivers (String::new -> State.new, msg.contains -> ByteSet.contains, alts.push on a Vec -> TSeq.push). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
9a2bdc8c1d |
fix(pipeline): spilling must not change the graph
The same repository indexed with results parked on disk produced a DIFFERENT graph than one indexed in memory. Measured on the php corpus, same binary, reproducible 3/3 each way: 16,561 nodes / 57,182 edges in memory against 16,562 / 59,379 while spilling. Not jitter, and not one-directional -- 1,458 CALLS pairs appeared only under spill and 840 only without it. Present on main, so it ships in v0.11.0 today, and it bites exactly the large-repo users who need spilling. Cause: pass_parallel built the namespace map -- what `use` / `using` / package imports resolve against -- from the in-memory result cache, where a PARKED result is NULL. Every spilled file therefore contributed no namespace, and a file missing from that map does not fail to resolve, it resolves through the looser fallback. That is why edge counts moved in both directions instead of simply dropping. The fix keeps each parked file's namespace IN MEMORY while its result is on disk (one short string per file, nothing against the result it replaces), and builds the map from names so the memory and disk paths agree. Non-spill runs take the identical code path and are unchanged. Across ten languages and four memory settings (default, 2 GB, 700 MB, forced spill), php/java/csharp/kotlin now produce identical graphs where they differed by up to 40,341 edges before. The existing spill-parity contract in test_parallel passed throughout, because its fixture is Go+Java and the compact parity harness omits the pass that builds the map -- neither reaches the namespace path. The new test runs the real pipeline over more than MIN_FILES_FOR_PARALLEL namespaced PHP files, with and without spilling, and fails without this fix. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
7b77fd4834 |
feat(mem): memory on the core, waste cut, results spilled under budget
Second increment on the memory core. The first landed the core, the linter
and phase attribution with zero production call sites; this one migrates the
memory that matters, proves where it goes, removes the waste the proof
exposed, and makes the memory budget a promise: over budget, extraction
results go to disk and the run completes at the floor instead of aborting.
Every number below is from a real index on this machine (M5 Pro, 48 GB) with
CBM_MEM_PHASES=1 and the release build.
WHAT THE PROOF SAID (Go corpus: 21,875 files, 345 MB of source)
The graph buffer, blamed for the peak since the 2026-07 research, is 3% of
the memory: 0.55 GB peak. The per-file extraction results are the memory:
14.8 GB written into their arenas, of which 3.4 GB is reachable. The rest
is temporaries -- cbm_node_text copies at 496 call sites, per-node QN
strings, abandoned generations of every growable array -- that an arena
cannot free and that the result kept alive until resolve ended, because
the result owned the arena. The arenas were charged 28.8 GB of capacity
for those 14.8 GB. On the kernel the same shape was 8.9 GB used by 89,731
results, and resolve grew it to 13.3 GB used / 17.2 GB capacity because
the C walk allocated its working set in the result arena and stored arena
pointers into the shared registry. After that, 7 GB of the kernel's
post-resolve memory had no class at all.
WHAT CHANGED
Attribution. The graph buffer (87 raw sites -> 0), arena blocks, the
tree-sitter slab allocator, the bound tree-sitter/SQLite allocators, the
semantic path (pass_semantic_edges, pass_similarity, minhash: 80 sites), the
hash table (Verstable, a class per table via CTX_TY: cbm_ht_create_in) and
CBM_DYN_ARRAY allocate through the core. Classes arena, ts_tree, hash_table
and dyn_array join the table. Two cross-boundary frees of buffer-owned
strings (pass_importance, pass_complexity) go through a new setter,
cbm_gbuf_node_set_properties_json. The linter baseline drops 4086 -> 3916.
Result compaction (internal/cbm/result_compact.c). At the end of a file's
extraction, everything reachable from its result is measured, copied into
one exact-size arena (strings interned by content within the file) and the
working arena is handed back. Later appends restart growth at the default
block (cbm_arena_init_exact, arena grow_size). The stale duplicate
internal/cbm/arena.h, which shared the foundation header's include guard
and won or lost by include order, is a shim now.
Record slimming. CBMCall 320 -> 72 bytes: the eight inline argument slots
(256 bytes, mostly empty) are allocated on first capture. CBMUsage 48 -> 40.
The per-file overlay contract (src/pipeline/pass_lsp_cross.c). One
lifecycle for every language's cross-file resolve: a scratch arena, a
per-file overlay registry chained to the sealed shared base, the walk
against the overlay, resolved calls appended into the result, scratch
destroyed. type_registry gains chain-aware iterators (every yielded index
belongs to it->reg) and copy-on-write refinement
(cbm_registry_func/type_for_update), and the C, C#, Go and Python walks
use them. This is what an ASan heap-use-after-free in the
lsp_resolution_probe suite demanded: the C walk stored per-call arena
objects into the shared registry, and a kernel run with a naive scratch
arena lost 1,045 edges to the resulting garbage reads. It also ends
unsynchronized writes into the shared registry from parallel resolve
workers.
The 7 GB with no class, found and cleaned. gbuf_index tables and keys were
2.5 GB live and 7.1 GB at peak: the 18 worker buffers built the five
secondary indexes they never query (cbm_gbuf_new_worker builds none), and
18M per-key index arrays started at 8 slots (cbm_da_push_min starts them
at 2); edge property JSON is 81% "{}" (interned). The semantic pass
allocated a 512-token stride per function up front (packed tokens with
offsets: cbm_sem_corpus_add_docs_batch). Edge dedup keys are 128-bit
hashes instead of sprintf'd strings (edge_key_map_t). Each worker keeps
one reusable working arena between files (cbm_work_arena_take/give:
rewind, not free) and every phase mark returns freed pages to the OS.
Budget metric. The over-budget check compares the charged footprint --
max(phys_footprint, mimalloc commit) on macOS -- not resident_size: after
extraction the kernel worker sat at 17.4 GB RSS with 5.5 GB charged, because
pages mimalloc has purged stay resident until the OS wants them, and
phys_footprint under-reports after MADV_FREE_REUSABLE cycles.
Spill / admission control (internal/cbm/result_spill.c, the extract gate
in pass_parallel.c). At 15/16 of the budget (or over it, or with
CBM_MEM_SPILL=1) the gate latches spill mode -- early, because the gate
sees the crossing per file pull and the workers' in-flight files carry the
charge past the line before the first sweep lands: every compacted result
is parked on disk as header + block in one append-only file per writer
under <cache>/spill/, the results already cached are swept out by every
worker, and registry build, def collection, surface rows, resolve and the
infra-route passes load a result only for the moment they read it
(relocated by base delta). The abort path stays closed while a sweep runs
or a cached result remains; it fires only at the floor -- graph +
registries + in-flight files. A result that owns sub-results stays in
memory; a retained parse tree is dropped with the parked result and
re-parsed on load. Every consumer of the result cache reads a slot through
cbm_pipeline_result_acquire()/release() (pipeline_internal.h): a parked
slot is NULL in the array, and the passes that indexed it directly lost
every __route__infra__ node and its INFRA_MAPS edges before the contract.
The other things the transient loads found: a def label that was the
result's own pointer (SIGBUS in the Go cross registry builder), two JVM
helpers that passed their input through, a dedup table whose keys borrowed
sr->value (the insert spun on freed memory), and a latch that was visible
before the store was open (a peer found nothing to park and failed the run
while its neighbour wrote 9 GB). The arena rewind reuse branch had
defeated the nblocks = CBM_ARENA_MAX_BLOCKS OOM seam the suites use; it
reuses only blocks that exist. The incremental and probe routes still
index the cache array themselves and therefore never spill
(ctx->spill_allowed stays false there; follow-up).
The semantic pass under a budget (src/pipeline/pass_semantic_edges.c). The
pass held every function's tokens, the corpus's per-document token ids and
the vectors being stored at once: 4.9 GB on the kernel, on top of the
floor, and nothing sized it to the headroom. Now, when the charged
footprint plus that transient would cross the budget, phases 2-4 run per
batch of functions: tokenize -> count into the corpus -> free; finalize
once; tokenize again -> vectorize -> store -> free, with pages returned to
the OS after every batch. Tokenization is deterministic, so both passes
see the same tokens and the graph does not change; the run pays with a
second tokenize. CBM_SEM_BATCH=<n> forces the batch size (the
batched-equals-unbatched test). The corpus frees its per-document token
ids at the end of finalize (the co-occurrence pass is their only reader),
and mem.semantic.step lines (CBM_MEM_PHASES=1) put the class's live/peak
bytes and the charged footprint at every sub-phase.
The semantic corpus and tokenizer allocate through the core (54 raw sites
-> 0). This is not attribution only: cbm_sem_tokenize handed out libc
strdup blocks that pass_semantic_edges freed with cbm_free -- mi_free in
the production build -- since the pass moved onto the core. No libc-backed
test build can see a cross-allocator free, so the core now refuses a block
it does not own (mi_is_in_heap_region) when CBM_MEM_PHASES=1 is set,
fatally: every proof run in this PR ran with it armed, and the Go and
kernel indexes report zero foreign blocks.
Accounting without contention. The core charged every allocation and free
to shared per-class atomics; with 18 workers on 18 cores every block bounced
the same three cache lines, and the bench against the shipped v0.10.8 showed
it as CPU time rising faster than wall time for the same graph: Kotlin 38 ->
121 s CPU (+58% wall), TypeScript 210 -> 596 s (+67%), Go 177 -> 312 s. The
deltas are thread-local now and reach the shared counters once per 256 KB or
512 blocks of change per thread, when a parallel-for worker ends, and
whenever a reader looks (its own thread first). Phase marks read after the
join and are exact; a class peak is a diagnostic and can be low by threads x
256 KB. Result: Kotlin 10.9 s / 43 s CPU (the residual +22% wall against
v0.10.8 tracks its +34% nodes), TypeScript 33.9 s (v0.10.8: 33.4), Go 48.4 s
(v0.10.8: 52.6), Java 81.1 s (v0.10.8: 79.8); graphs unchanged.
SQLite on a heap of its own. The same bench put the kernel's write step at
132 s where v0.10.8 needed 9.7 s, all of it in one coverage publish
statement. A stack sample named the cost: mimalloc's free-page search and
its periodic heap collect, walking the graph's pages -- millions of them
once the graph lives on the core -- for every statement-journal chunk
SQLite allocates. SQLite now allocates from a mimalloc heap of its own per
thread (mi_heap_new; mi_free is heap-agnostic, so nothing else changes),
and the periodic collect is set to the maximum mimalloc allows since the
explicit release at phase marks and spill sweeps is where memory goes
back. Coverage publish 132 -> 9.4 s; profiled kernel 376 -> 248 s
(v0.10.8: 296 s).
Formatting without a process lock. On macOS every vsnprintf takes the
process locale under an unfair lock (localeconv_l inside __vfprintf);
eighteen workers formatting type names on the C# corpus collapsed into
it -- the 10 MB JIT test files took 68 s each instead of under a second
-- and cbm_arena_sprintf paid it twice per string (size pass, write
pass). Each thread now formats with a C locale object of its own
(vsnprintf_l), and the common short string is formatted once into a
stack buffer.
Per-file budgets that cover the whole file. The 5 s per-file budget
covered the parse only; the LSP walks after it had none. C# JIT stress
files (a 23 MB single expression among them) parsed inside the budget --
the old wall-clock parse timeout used to drop them, which hid the rest --
and then held a worker for 346 s each in the usage walk (tree-sitter's
ts_node_parent descends from the root: quadratic on a deep tree), and the
cross-file resolve ran on the same trees. Three rules now, one site for
every language (cbm_extract_file_ex, honoured by cbm_pxc_dispatch_file): a
parse that used more than half the budget disqualifies the file from the
LSP walks (lsp_skipped); the unified walk checks its thread CPU time every
1024 nodes against six budgets and stops there, keeping what it found
(walk_truncated, implies lsp_skipped); a file whose parse plus walk spent
the budget is skipped by the LSP walks as well. Each decision is logged
with the path (extract.lsp.skipped, extract.walk.truncated). C# extraction
355 -> 39 s; the whole index 463 -> 128 s wall and 27 -> 11 GB peak
against v0.10.8 with 299 more nodes. The walk cuts exactly the five JIT
stress files (hugeexpr1, hugeSimpleExpr1, HugeArray1, HugeField1/2); the
15 MB System.Runtime.Intrinsics reference file walks to the end and, with
four generic-nesting JIT regression tests, only skips the LSP walk under
the parse-plus-walk rule.
The crash behind the C# bench. dotnet/runtime is 42,555 files, and its
resolve phase died with SIGBUS on every run of this branch. A libc-backed
ASan build of the server named it: heap-use-after-free in c_adl_resolve,
reading a type name that another worker had allocated in its per-file
scratch arena and freed at the end of its file. The C++ class walk
refines a method's return type when the declaration is more specific
than the pre-registered one (NAMED -> pointer/reference/template), and
it did that by casting the chained lookup result to non-const and
writing a scratch-arena signature into it -- into the sealed shared base
when the entry lived there. The overlay contract had exactly one
bypass, and it was this cast; every lookup returns const, so a grep for
the cast is the audit. The refinement now goes through
cbm_registry_func_for_update: copy-on-write into the overlay, the base
untouched. Test: clsp_method_return_refinement_is_copy_on_write (a
sealed base with a NAMED method return, an in-class pointer declaration;
the base signature pointer is unchanged and the overlay copy carries the
refinement plus a marker only a copy has) -- RED on the cast, GREEN on
the accessor. The fixed ASan build then indexed the whole
corpus clean: 42,555 files, 1,224,981 nodes, 5,794,320 edges, worker
exit 0, no report.
Production build backs the core with mimalloc explicitly (never the global
override, which macOS's two-level namespace forbids); the test build keeps
libc so the sanitizers see every block -- the same split the tree-sitter and
SQLite bindings already use.
PROOF
Go peak RSS 16.7 -> 4.0-5.2 GB; result arenas 14,804/28,817 -> 856/856
MB; wall 55 -> 52-60 s; graph inside the pre-existing run-to-run
jitter (gRPC Route nodes: two identical runs differ by 21).
Go CBM_MEM_SPILL=1: peak RSS 1.7-2.3 GB, 21,875 results parked (950 MB
on disk), graph identical: 39 infra routes, 158 INFRA_MAPS, 21,875
surface rows, nodes/edges inside the Route jitter.
Go CBM_MEM_BUDGET_MB=1200: spill latched at 1,204 MB, the first sweep
took the charge to 768 MB, the run completed at 1.99 GB peak RSS.
Kernel DEFAULT budget (24,576 MB): aborted before, completes now without
spilling. Extraction 35.2 -> 16.5 GB RSS (5.6 GB footprint, 15.3 GB
committed); results 8.9 GB = capacity; resolve tracked 24.6 -> 19.1
GB; peak RSS 27.3 -> 20.1 GB. Nodes identical (8,529,729); edges
inside the +-400 run-to-run band the kernel shows between identical
runs.
Kernel CBM_MEM_BUDGET_MB=16000: aborted before, completes now through the
spill path. Latched at 17.2 GB charged; 89,731 results parked (9.3
GB on disk, 370,188 loads); extraction end 6.2 GB RSS / 5.1 GB
committed; resolve 8.6 GB RSS / 11.5 GB committed; peak RSS 16.6
GB (the semantic pass plateau, 13.3 GB RSS, is graph-derived and
outside what spill can move). Nodes identical; edges 15,773,598.
Kernel CBM_MEM_BUDGET_MB=15000, before the semantic batching: completes,
every phase mark under 15,000 MB by every metric, but the process
RSS high-water mark reached 16.9 GB inside the semantic pass (class
peak 4.9 GB against 0.8 GB live).
Kernel CBM_MEM_BUDGET_MB=15000, with it: completes; the charged footprint
stays under 15,000 MB through the whole semantic pass (14,018 ->
14,381 MB across its sub-phases, 6 batches of 125,644 functions);
every phase mark is under budget by every metric (max footprint
12.6 GB, max commit 11.5 GB); the process RSS high-water mark is
15.65 GB and is set in extraction, at the moment the gate latches
(15,002 MB charged) and before the first sweep lands -- a 4%
overshoot of in-flight work, no longer the semantic pass. Nodes
identical, edges 15,772,913, zero foreign blocks.
Kernel CBM_MEM_BUDGET_MB=15000, with the anticipatory latch: spill mode
enters at 14,062 MB (near_budget); the charged footprint never
exceeds 11.8 GB at any mark; the process RSS high-water mark is
15.1 GB in extraction and 15.3 GB during the final write-out --
1-2% over, the in-flight window and the dump transient. Nodes
identical, edges 15,772,543, zero foreign blocks.
Go CBM_MEM_BUDGET_MB=1200 with it: latched at 1,131 MB, extraction
peak 998 MB RSS, graph identical to the baseline.
BENCH AGAINST THE SHIPPED v0.10.8 (same machine, same driver, 2026-09-14)
corpus wall s peak RSS GB nodes / edges / CALLS
perl 5.7 -> 5.3 0.09 -> 0.05 -3 / -130 / -3
php 6.3 -> 5.8 0.66 -> 0.20 +21.9% / +1.4% / +23.8%
rust 6.8 -> 6.3 1.15 -> 0.67 +1 / +56 / -369
c 5.9 -> 5.7 0.16 -> 0.19 identical
kotlin 8.9 -> 9.4 1.55 -> 0.96 +33.5% / -22.8% / +12.1%
django 10.9 -> 10.2 3.13 -> 1.12 +6 / -14.5% / -28.4%
go 52.6 -> 42.9 17.2 -> 3.9 +14.2% / -10.3% / +2.2%
java 79.8 -> 73.8 27.6 -> 11.0 -121 / -1.9% / -0.6%
csharp 463.3 -> 128.1 27.0 -> 11.2 +299 / -0.9% / -0.6%
typescript 33.4 -> 33.6 12.0 -> 2.5 -186 / -3.5% / +130
kernel 294.8 -> 233.3 27.5 -> 25.1 -171 / -1.3% / -0.1%
No corpus is slower beyond noise; CPU time follows wall (kernel 2,442 ->
1,403 s). Every graph delta against v0.10.8 is main between the release and
the branch base (
|
||
|
xkchok
|
e11d5900cd |
fix(test): free the return-type table in the go field-chain helper
run_go_field_chain_sequential tears down the results, registry and cross arenas but never calls harness_ctx_free_tables, so pass_calls' CBMReturnTypeTable (strdup'd method return types) leaked whenever the helper ran. ASan builds caught it: AddressSanitizer: 32911 byte(s) leaked in 6 allocation(s) from build_return_type_table, failing test-diag, test-lsan-macos and both gcc sanitizer shards. Verify locally: build/c/test-runner parallel -> 73 passed, no leaks. Signed-off-by: xkchok <143325001+xkchok@users.noreply.github.com> |
||
|
xkchok
|
8f50841af4 |
fix(go): resolve cross-package field-chain calls via type dispatch
Go calls written through an aliased struct field (h.svc.Method(ctx, req)) previously fell through to fuzzy suffix matching because: - the per-file Go dispatch walk was bypassed, so field selectors were never descended; - struct fields extracted as flat "Field" defs were dropped by the LSP label map, so every Go struct registered with zero fields; - receiver types written through import aliases (svc.Svc) produced qualified names that exist nowhere in the project registry. Changes: - restore the Go walk in cbm_pxc_dispatch_file and descend field selectors through the shared prebuilt registry, emitting lsp_type_dispatch edges; - fold per-field defs into their owning struct inside cbm_pxc_collect_all_defs (one site covers both the prebuilt-registry path and the per-file fallback); - re-qualify the last alias segment through the calling file's import map when the direct receiver-type lookup misses (go_requalify_via_imports). Tests: - tests/test_parallel.c: cross-package field chain resolves end to end with an lsp_type_dispatch edge in the parallel resolver; - tests/test_go_lsp.c: alias-typed struct field re-qualifies through the import map and lands on the real receiver type. Signed-off-by: xkchok <143325001+xkchok@users.noreply.github.com> |
||
|
Martin Vogel
|
2910e2842f |
feat(lsp): resolve cross-file base classes for the Python/TS cross-LSP
`CBMDefinition.base_classes` carries the SOURCE SPELLING of each base
("Base", "django.db.Model"): extraction strips keywords and generic
arguments, but it cannot know where the name is declared. The Python and
TS cross-file registrars, however, consume `CBMLSPDef.embedded_types` as
fully-qualified names — py_lookup_attribute and ts_lookup_member feed each
entry straight into cbm_registry_lookup_type.
An unqualified spelling therefore matched nothing declared in ANOTHER
file. `class Child(Base)` in child.py never saw Base in base.py, so a
call to an inherited method through a typed receiver found no member and
fell through to the weak textual cascade, where the receiver-aware guard
(#592/#606) correctly kills it. The member-lookup walk over
embedded_types was already there in both languages; only the names it
walked were unusable across files.
cbm_pxc_collect_all_defs now resolves each base spelling to a project QN
using exactly the inputs pass_semantic uses to draw its INHERITS edge:
the project registry, the declaring module, and the file's import map.
Two properties follow. The LSP's inheritance view is the same relation
the graph records, so the two cannot diverge. And the binding is import-
or same-module-backed rather than a short-name guess, so the CALLS edge
it enables is a supported fact that the weak-member guard keeps.
Weak registry strategies (suffix_match / unique_name / field_type_hint /
fuzzy) are rejected for bases via an explicit drop-list: a base bound
because some project type happens to share its name is precisely the
fabricated relation #606 removed, and inheritance multiplies it — every
inherited member of the wrong base would become a callable target. An
unresolved base keeps its raw spelling and the behaviour that predates
this change, so stdlib and third-party bases are unaffected.
Scoped to the languages whose registrars read embedded_types as QNs
(Python, JS/TS/TSX). Go, JVM, C#, C++ and Rust already qualify their own
embedded types and keep the raw join.
Cost is O(defs) hash lookups plus ONE import map per file: no per-call-
site hierarchy walk, no registry scan, no per-file registry rebuild. The
complexity guard stays linear (nodes 1.93, edges 2.07, per-file defs
2.00 against a files x corpus coupling of ~4).
Tests:
- ts/S6 flips to its own documented condition: was `calls == 0` as a
tripwire recording the gap, now asserts calls >= 1 AND an lsp_ts_*
strategy. Measured: strategy=lsp_ts_method, confidence 0.95.
- python/S6b is new and covers what python/S6 structurally cannot: S6's
`def run(c)` parameter is un-annotated, so no receiver type exists to
inherit through and any binding there is a guess about a name (it
resolves via unique_name at 0.75). S6b supplies the receiver three ways
— annotated parameter, `self` in the subclass, constructor result — and
asserts INHERITS >= 1 plus an lsp_* strategy. Revert-check: all three
fall back to unique_name at 0.75 without this change, lsp_method at
0.90 with it.
- Both rows assert the STRATEGY, not just the edge count: a short-name
guess and a resolved inherited member produce the same count in a
two-file fixture, and only the strategy tells them apart.
- Corrects python/S6's stale comment citing a base_classes extraction
bug; extraction_inheritance is green for Python and TS, and
base_classes now holds clean names.
macOS full suite 7690 passed / 0 failed / 8 skipped (140 suites);
make lint-ci clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
5eae752f48 |
test(parallel): record the weak-member trade for the LSP-ambiguity probe
parallel_lsp_index_exact_ambiguity_does_not_fall_through_to_legacy asserted alpha_edges == 1, which the Python weak-member guard suppresses. That assertion is a side effect, not the test's subject. Its subject — that an ambiguous exact LSP match must not fall through to the lower-ranked legacy semantic row — is unchanged: beta_edges == 0, legacy_edges == 0, and all four probe assertions (ambiguity.injected, legacy_injected, carrier_allows_fallback, shared_matcher_failed_closed) still hold. Verified before changing the number, not after. The harness deliberately injects an ambiguous exact match and forces the shared matcher to fail closed, so value.render() drops past the LSP into the registry with three same-named candidates (Alpha/Beta/Legacy.render) and bound by suffix_match -- measured: strategy=suffix_match, cands=3, conf=0.5500. That is a 1-in-3 guess that happened to land on Alpha: exactly the weak member-call class this change suppresses, and the same accepted trade recorded on python/S6 in test_lsp_resolution_probe.c. Not over-suppression: sibling tests in this suite index the same value.render() source and resolve it via lsp_method (cands=1, conf=0.9000), a strategy the guard keeps -- they still assert 1. Only the sabotaged-LSP path degrades to a weak textual guess. Flips back to 1 once py_lsp_cross resolves the receiver on this path. Refs #1276 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
0f1e65d625 |
fix: preserve Cargo context in parallel Rust resolution
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
94189e6990 |
feat(pipeline): serialize and publish per-file LSP surfaces
Second piece of closure repair. At the collect_all_defs seam -- the only moment the per-file result cache is alive -- both drivers (parallel and sequential) now serialize each file's CBMLSPDef slice to canonical JSON, hash it, and hand the rows to the pipeline; cbm_pipeline_publish_generation writes them into the staging store next to the manifest, so surface data and graph always belong to the same generation. Canonical bytes are the point: every field is written in fixed order with an explicit null for absent strings (NULL and "" differ in the CBMLSPDef contract -- receiver_type NULL means "not a method"), so byte equality IS surface equality and the sha over the bytes is the early-cutoff key. Registry-only labels that pxc_map_label drops but the name registry serves (Field) are folded into the hash as a separate "reg" array, or renaming one would slip past the cutoff. Behaviour pinned in SUITE(pipeline): a fresh full index persists a versioned surface row per file; a BODY edit republishes the identical surface_sha; a SIGNATURE edit changes it. That pair of properties is what the routing layer will stand on. cbm_pxc_collect_all_defs gains an optional per-file prefix array -- the flat all_defs[] otherwise loses the file boundaries the serializer needs. CORRECTION to 6c22338's scope note: it claimed cbm_pipeline_publish_ generation was reachable only behind CBM_INCREMENTAL_TEST_API. Wrong -- dump_and_persist_hashes calls it on every production full index (pipeline.c:1863); the grep that "verified" test-only reachability had excluded pipeline.c itself. The predictable staging name WAS in the production publish path, which makes that fix a real production hardening, not a test-path cleanup. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
e2f5b6a798 |
fix(pipeline): module-QN strings must outlive the cross registries they feed
The sequential lsp_cross pass builds its shared per-language cross registries
in ctx->seq_cross_arena, which DELIBERATELY outlives the pass -- resolved_calls
and the registries carry borrowed strings that pass_calls still reads, and the
arena is destroyed only after all passes (the earlier freeing-here bug was a
pass_calls use-after-free, says the comment at the arena's creation).
But the per-file module-QN strings (def_modules[], malloc'd in
cbm_pxc_collect_all_defs and handed to every registrar as def_module_qn) were
freed at the END OF THE PASS -- the exact mistake the arena comment warns
about, one level down. Any registry-reachable structure holding one of those
pointers read freed memory in pass_calls.
AddressSanitizer caught it as a heap-use-after-free (strcmp in
cbm_pipeline_pass_calls on a string freed by the pass-end cleanup) on the
first-ever run of the real-repo determinism tier (linux/fs/xfs, 355 files) --
a tier no CI runner can execute because the corpus is local-only, which is why
it survived: bisect shows it predates today's commits (
|
||
|
Martin Vogel
|
43e861e36a |
fix(pipeline): release db_path/db_dir on the manifest abort paths; free the test harness cross arena
Second LSan round from the Linux leg, both verified green in the container (532 passed, 0 failed under LSan): - dump_and_persist_hashes' two semantic-manifest abort returns leaked BOTH of the function's strdups (db_path and db_dir, the latter otherwise freed only further down). Same ownership rule as the previous fix: every exit releases what the function allocated. - test_parallel's sequential harness drives the passes directly and never destroyed ctx->seq_cross_arena, which the cross pass fills with the shared per-language registries (stdlib registrations included -- ~20MB per test). Production's run_sequential_pipeline destroys it after all passes; the harness now does the same. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
029416fda6 |
Merge remote-tracking branch 'origin/main' into qa/call-reference-rebase
# Conflicts: # Makefile.cbm # internal/cbm/extract_defs.c # src/foundation/compat_fs.c # src/foundation/compat_fs.h # src/pipeline/pipeline.c # src/pipeline/pipeline_incremental.c # tests/test_main.c # tests/test_pipeline.c # tests/test_store_checkpoint.c |
||
|
Martin Vogel
|
50d0cc8c7d |
test: free the harness-built return-type table in the parity runners
The parity harness drives the pipeline passes with a bare ctx and never ran the production teardown, so the return-type table pass_calls builds leaked — latent until the fixture gained typed Java methods, and visible only under Linux LeakSanitizer. Both runners now free it exactly the way pipeline.c does. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
4a3aac6478 |
fix(pipeline): one Interface edge split in both venues, explicit overrides
The parallel per-file resolve hardcoded INHERITS for every base-class relation while the sequential semantic pass made the Interface split, so any corpus large enough for the parallel path had its explicit implements demoted: elasticsearch carried 11,179 INHERITS edges pointing at Interface nodes and 3 IMPLEMENTS. Both venues now share one decision keyed off the target node's label — the graph truth the edge attaches to. OVERRIDE edges existed only for Go's implicit interface satisfaction. A serial full-graph tail run by both pipelines now matches class methods against explicit bases by name and emits Method-to-Method OVERRIDE edges for the explicit languages, with Go excluded to avoid double cover. The venue parity fixture gains a Java interface/implements/extends trio — IMPLEMENTS parity used to compare 0 == 0 and guarded nothing — plus an exact-count test so the parity can never go vacuous again. At scale: elasticsearch IMPLEMENTS 3 to 11,182 and OVERRIDE 0 to 49,695, TypeScript 0 to 4,496 and 594, dotnet/runtime 0 to 12,953 and 45,667, with INHERITS plus IMPLEMENTS sums conserved exactly. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
87a0e3f74f |
fix: preserve semantic graph relationships
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Shane McCarron
|
aa97ec55f2 |
fix(perl-lsp): address QA round 1
F1: dot the Exporter import target so seeded CPAN exported subs resolve. perl_collect_qw_imports built colon-form targets (Scalar::Util::blessed) but the stdlib registry keys curated CPAN subs in dotted form (Scalar.Util.blessed) and lookup is exact-match. Wire in perl_pkg_to_dot to dot the module portion and drop the now-unnecessary (void) cast. F2: add a recursion-depth guard (CBM_LSP_PERL_MAX_WALK_DEPTH=512) to both AST walkers (perl_resolve_calls_in_node, perl_pass1_scan) via a depth- guarded wrapper + inner split, mirroring java_lsp's JAVA_LSP_MAX_WALK_DEPTH. Past the cap a subtree is skipped (graceful degradation, no wrong edge), preventing stack overflow on pathologically nested input. F3: lock the shared last-"::"-segment normalization in lsp_resolve.h with a direct regression test over cbm_pipeline_find_lsp_resolution: a qualified static call still resolves AND the cross-namespace mis-attribution edge case is bounded by caller-QN equality + the confidence floor. F4: implement SUPER:: dispatch. Populate enclosing_parent_qn from the enclosing package's first @ISA parent and resolve $self->SUPER::method() to that parent's method (strategy perl_method_super). No known parent or unresolved method emits no edge (zero-edge guarantee preserved). Tests: perllsp_cpan_exported_function, perllsp_super_dispatch, perllsp_super_no_parent_no_edge, lsp_perl_deep_expression_no_crash, lsp_resolve_qualified_static_call_normalizes_colons, lsp_resolve_misattribution_is_bounded. Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com> |
||
|
Martin Vogelandnguyentamdat
|
e85c7d69f0 |
fix(index): bound + source-cap parallel retention with re-read fallback (low-RAM peak RSS)
Distilled from #685 (nguyentamdat) rebased onto current main, plus two research-driven refinements and a genuine reproduce-first guard. The parallel extract retains each file's source text so the fused cross-file LSP resolve can re-parse it. That retention is transient but a peak-RSS driver. On main the caps are flat (100 MiB/file, 2 GiB total) and a file over the cap is silently unretained AND its cross-file resolution is skipped -- a graph-quality gap. This change bounds retention AND keeps every cross-file edge. - Source-text cap as a FLOOR, not just a ceiling: retention total defaults to min(cbm_mem_budget()/8, 1 GiB), per-file min(32 MiB, total). Following the rust-analyzer memory model, the RAM-derived default is clamped to a small absolute ceiling so a huge-RAM host does not hold tens of GB it would re-read cheaply. Both caps env-overridable via CBM_RETAIN_TOTAL_MB / CBM_RETAIN_PER_FILE_MB (limits.c convention); ceilings bound only the auto-derived default, never an explicit operator/caller choice. A dropped file emits one index.retain_capped WARN per run. - Bounded re-read fallback (the correctness guarantee): resolve_worker re-reads an unretained file's source on demand (bounded, freed immediately) instead of skipping resolution, wired at every cross-LSP site that consumes source. The cap now only trades retained RAM for a bounded re-read, never a lost edge. - cbm_parallel_extract_ex + opts struct (cbm_parallel_extract is now a wrapper passing NULL -> env-derived defaults); malloc/calloc NULL-check hardening. Reproduce-first: parallel_cross_file_reread_preserves_unretained_edges uses a Java<->Kotlin pair whose cross-file lsp edges are genuinely source-dependent; the edges are lost when the caller is unretained and the fallback is absent (RED), present with it (GREEN), with a retained CONTROL scenario proving non-vacuity. #685's original Python red test was a false guard (per-file py_lsp already resolves those calls) and is replaced. Peak-bound guards (retained_bytes <= total_cap; retain_sources=false retains nothing) in test_mem.c. Verify: make -f Makefile.cbm cbm && make -f Makefile.cbm lint-ci; test-runner parallel pipeline incremental py_lsp ts_lsp java_lsp kotlin_lsp c_lsp cs_lsp go_lsp rust_lsp mem -> 2323 passed. Co-authored-by: nguyentamdat <nguyentamdat@gmail.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogelandnguyentamdat
|
5ff29caddc |
fix(jvm): resolve mixed Java/Kotlin callgraph edges via declared packages
Mixed Gradle/Maven source roots (src/main/java + src/main/kotlin) share JVM packages that path-derived module QNs cannot join: cross-file LSP emitted package-shaped names while graph nodes stayed path-derived, and same-package defs across roots were filtered out, so Java<->Kotlin calls fell back to weak or wrong textual edges. The declared package is JVM ground truth. Normalize JVM defs to their declared package (inferring it from conventional source roots only when no package was extracted), index defs by declared namespace alongside the path module so same-package defs survive per-file filtering, restrict JVM callers to JVM defs, register cross-def return-type signatures for Kotlin receiver typing, and consume qualified member-call overrides by leaf name in the parallel lsp_idx fast path. Distilled from #684 with one design change: the new unique-Class.method tail-match fallbacks in lsp_resolve.h are gated to JVM callers (cbm_pipeline_lsp_allow_tail_match, Java/Kotlin only). Tail-matching by unique leaf is safe where class-per-file package semantics hold; in other languages a single wrong-module coincidence would fabricate a CALLS edge. The leaf-key lsp_idx lookup stays global: it aligns the fast path with the build-side key that already used leaf names. Tests: carried mixed Java/Kotlin regression (red on main, green here) plus a gate guard asserting the tail fallbacks stay off for non-JVM callers and on for JVM. Co-authored-by: nguyentamdat <nguyentamdat@gmail.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Andrius SkerlaandClaude Opus 4.8
|
74d15a6064 |
fix(pipeline): prevent stack-buffer-overflow in append_args_json
A call carrying enough long arguments drove append_args_json()'s running position past the fixed CBM_SZ_2K `props` stack buffer in emit_normal_calls_edge(): format_call_arg() returns snprintf's *untruncated* length, so `pos += (size_t)n` could exceed `bufsize`, after which the trailing `buf[pos] = '\0'` (and `buf[pos++] = ']'`) wrote out of bounds. The stack canary caught it as SIGABRT, so full-repo indexing of large TypeScript codebases crashed the server in the parallel resolve pass (emit_service_edge -> emit_normal_calls_edge -> finalize_and_emit -> append_args_json). Confirmed with AddressSanitizer: stack-buffer-overflow WRITE at pass_parallel.c:1124, 'props' (2048 B). Fix: when an argument does not fully fit, roll back to before its separator and stop appending (atomic field, matching append_json_string's behaviour), so `pos` can never advance past the buffer. Add regression test parallel_args_json_no_overflow: indexes a fixture whose single call carries 60 long string args (args JSON well past 2 KB); under the ASan test build it aborts without this fix and passes with it. Signed-off-by: Andrius Skerla <1492322+rainder@users.noreply.github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Martin Vogel
|
eed87fd372 |
Forbid test skips and convert existing skips to hard failures
Add scripts/check-no-test-skips.sh (run from lint) which fails the lint phase on any plain SKIP() or direct tf_skip_count manipulation; only SKIP_PLATFORM() (for genuinely platform-specific tests) is tolerated. Add FAIL() and SKIP_PLATFORM() helpers to the test framework and convert the remaining SKIP()/perf-gated skips across the suite into pass-or-fail assertions, so a suite that cannot meet its preconditions reports a red failure instead of a silent skip. |
||
|
Martin Vogel
|
5f3847c0e5 |
fix(grpc): anchor Route emission on stub suffix; preserve service name (#294)
emit_grpc_edge produced two classes of bad output: 1. Wrong service name: the suffix list stripped "ServiceClient"/"ServiceGrpc" before "Client"/"Grpc", so FooServiceClient collapsed to "Foo" instead of the proto-declared "FooService" — breaking cross-repo Route matching. 2. Phantom Routes: extract_grpc_service_method returned true for any "<recv>.<method>" call, so ordinary receiver vars (_provider.GetGroup, _builder.AddSomeService) became __grpc__provider/... Routes matching no .proto anywhere. Strip only the trailing stub/client token (Client/Stub/Grpc/BlockingStub/…), preserving "Service", and require that a recognized suffix was actually present before emitting — the suffix match is the gRPC stub-type signal. Plain receiver vars no longer yield Routes. extract_grpc_service_method is no longer static (declared in pipeline_internal.h) so it is unit-testable. |
||
|
Martin Vogel
|
23d404a056 |
perf(lsp): Tier 2 — pre-built per-language cross-LSP registries (Go + Python)
The gopls 'package summary' pattern, applied properly: build the cross-
LSP CBMTypeRegistry ONCE per language at pipeline time (in a dedicated
cross_lsp_arena), share it READ-ONLY across all resolve workers via the
CBMCrossLspRegistries struct. Per-file work then becomes just tree-
sitter parse + AST walk + O(1) hash lookups — zero per-file registry
rebuilds for languages with a pre-built registry.
Each LSP gains two new public functions:
CBMTypeRegistry *cbm_<lang>_build_cross_registry(arena, defs, def_count)
— filters all_defs to its language, init + stdlib + registration
loop + cbm_registry_finalize, returns finalized registry.
void cbm_run_<lang>_lsp_cross_with_registry(arena, source, ...,
CBMTypeRegistry *reg, ...)
— skips the per-file build AND the per-file Phase 1b/1c AST scans
(those mutate the registry, which would race on a shared read-
only reg). Trade-off: a few file-local type aliases discovered
only by Phase 1b are missed; in practice the per-file extract
LSP already captured them via embedded_types/field_defs/etc.
Tested with parallel_python_lsp_override_cross_file_emits_lsp_
strategy_edges — fused + filtered + prebuilt path stays correct.
CBMLSPDef gains a CBMLanguage lang field set in pxc_build_lsp_def so
the per-language build functions can filter the global all_defs[].
cbm_parallel_resolve signature now takes CBMCrossLspRegistries* instead
of a single CBMTypeRegistry*. The resolve_worker dispatches:
cbm_pxc_registry_for_lang(lang) returns the right pre-built reg if any;
the worker then switch()es to the matching _with_registry variant.
Languages without a _with_registry variant fall through to the existing
filter + per-file build path.
Kubernetes (Go-heavy, 12k files, 139k defs) measured perf:
parallel_resolve: 5.51 min (strdup-removal baseline) -> 1.86 min
= 2.96x speedup
parallel.resolve.lsp_cross.slow lines: 1,258 -> 2
(-99.8%; the 2 remaining are Python files, not Go)
Peak RSS: 6.29 GB -> 6.23 GB (slight decrease — pre-built registry
shared, no per-file scratch)
All 3,616 tests pass; zod/serilog graphs exact match (per-file path
unaffected).
Cumulative trajectory from the original broken state on kubernetes:
35 min -> 19 min (finalize) -> 6.24 min (module-def filter) ->
5.51 min (strdup-removal) -> 1.86 min (Tier 2 Go) = 18.8x total.
Languages wired so far: Go, Python. C/C++, TS/JS, PHP, C# still use the
per-file filter+build path until their _with_registry variants land in
follow-up commits.
|
||
|
Martin Vogel
|
75a0df688f |
feat(pipeline): fuse cross-LSP into resolve worker + module-def filter
Previously cbm_pipeline_pass_lsp_cross ran as a separate sequential pass that re-read every source file from disk and re-parsed every tree-sitter tree on a single thread — a 50x regression on the kubernetes Go monorepo (extract 10s, lsp_cross 9+ min when killed). Three changes: 1. CBMFileResult gains const char *source / int source_len. The parallel extract worker copies source bytes into result->arena (per-file 100MB cap, project-wide 2GB atomic cap) so the fused cross-LSP step can run without disk I/O. Source pointer is borrowed by downstream readers; arena owns the storage. 2. Cross-LSP work moves into the parallel resolve worker. Each worker processes its file with the existing parallel infrastructure (worker pool + work stealing). Slab is reclaimed after each per-file cross-LSP call to bound peak slab memory. The previously separate lsp_cross pass entry function is kept for the sequential pipeline path; the parallel path no longer invokes it. 3. CBMModuleDefIndex (gopls 'package summary' pattern) — an inverted index module_qn -> list of def indices built once in O(D) before resolve_worker fires. Per file, cbm_pxc_filter_defs_for_file slices all_defs down to the file's own_module + imported modules only, typically 50-100x smaller than the global ~110k defs on kubernetes. Drops the per-file registry build from O(all_defs) to O(relevant). Other plumbing: cbm_parallel_resolve signature gains all_defs, def_count, def_modules, module_def_index params. pipeline.c builds the index between registry_build and parallel_resolve and frees it after. pipeline_incremental and tests/test_parallel updated to match. Helpers in pass_lsp_cross.c exposed via the header: cbm_pxc_has_cross_lsp, cbm_pxc_collect_all_defs, cbm_pxc_run_one(_ts), cbm_pxc_ts_modes, cbm_pxc_build_module_def_index, cbm_pxc_filter_defs_for_file. Kubernetes wall-time: 35 min -> 6.24 min (~5.6x) before the followup perf commits in this series. |
||
|
Martin Vogel
|
b2a71b80b9 |
feat(pipeline): wire cross-file LSP across all five hybrid resolvers
Per-file LSP (run from cbm_extract_file) only sees one file's defs, so
callees whose receiver type comes from an imported module stayed
unresolved — even though Go/C/Py/TS already had cross-file entry points
(cbm_run_X_lsp_cross). Those entry points existed but no pipeline pass
ever invoked them. PHP didn't even have a cross-file entry point.
This change lands the missing pieces:
1. cbm_run_php_lsp_cross + cbm_batch_php_lsp_cross — PHP cross-file
resolver mirroring py_lsp_cross / ts_lsp_cross. Reuses PHPLSPContext,
php_lsp_init, php_lsp_add_use, php_lsp_process_file. Handles PHP-
specific labels (Class / Interface / Trait / Enum / Type) and uses
def_module_qn so cross-file return types qualify against the def's
own module rather than the importing module's.
2. pass_lsp_cross.c — new pipeline pass. For every cached file result,
builds a project-wide CBMLSPDef[] (CBMDefinition → CBMLSPDef
converter), per-file import map (from gbuf IMPORTS edges), then
dispatches to the matching cbm_run_X_lsp_cross. Output strings get
copied into the per-file arena (with O(N²) memory bug avoided by
running each file inside a fresh scratch arena that gets destroyed
after the LSP call — the LSP allocates its registry + stdlib +
project defs in scratch, not in the caller's persistent arena).
3. Wired into both pipelines:
- sequential: definitions → k8s → lsp_cross → calls → ...
- parallel: parallel_extract → registry_build → lsp_cross →
parallel_resolve → ...
4. Updated tests/test_parallel.c run_parallel helper to mirror the new
parallel ordering so existing cross-file Python/PHP tests now exercise
pass_lsp_cross.
Languages covered: Go, C/C++/CUDA, Python, TS/JS/JSX/TSX, PHP. PHP cross
file is new. The other four had functions but no pipeline integration
until this change.
Verified on a 1670-file FastAPI index: pass_lsp_cross processes 1050
Python files (525 non-LSP files skipped), collects 4070 project defs in
9.2s, and parallel_resolve subsequently reports lsp_overrides=568 — the
LSP-strategy attribution gain over per-file alone.
Per-LSP emit functions dedup by (caller_qn, callee_qn) within their own
run; pxc_append_results dedups again at append time so cross-file
entries duplicating per-file output get dropped instead of inflating
result->resolved_calls.
|
||
|
Martin Vogel
|
673ac4ef8a |
fix(pipeline): resolve LSP cross-file QN mismatch via project-prefix fallback
Per-file py_lsp emits resolved_calls.callee_qn as the raw import-module path (e.g. `greeter.Greeter` from `from greeter import Greeter`) rather than the project-qualified QN the gbuf stores (`<project>.greeter.Greeter`). Before this change, the LSP match succeeded (lsp_overrides counter incremented) but the downstream cbm_gbuf_find_by_qn lookup missed silently and the edge was dropped — the failure mode the new lsp_overrides telemetry was designed to surface. The LSP layer can't tell in-project imports (qualify) from external imports (don't qualify, e.g. `os.path`) without consulting the gbuf, which is built downstream. So normalise at the consumer instead: try the LSP-emitted QN as-is first; on miss, retry with `<project>.<callee_qn>`. If that also misses, drop the edge — same as before, target is external/unindexed. New helper cbm_pipeline_lsp_target_node in lsp_resolve.h is shared by both pipelines (sequential pass_calls.c and parallel pass_parallel.c) so they continue to admit identical sets of LSP overrides. The helper also canonicalises res.qualified_name to the gbuf node's QN, so downstream edge property serialisation shows the project-qualified form even when fallback resolution kicked in. Cross-file regression test parallel_python_lsp_override_cross_file_- emits_lsp_strategy_edges pins the two-file Greeter scenario that originally exposed the bug. |
||
|
Martin Vogel
|
085b4a718c |
fix(pipeline): unify LSP-override resolver across sequential and parallel paths
The sequential pass_calls and the parallel cbm_parallel_extract pipelines each carried their own copy of the LSP-resolved-call lookup with divergent confidence floors and different match semantics. Most production indexing runs the parallel path, so projects produced different CALLS-edge attributions depending on which pipeline mode kicked in: pass_calls.c find_lsp_resolution floor 0.6 (CBM_LSP_CONFIDENCE_FLOOR) pass_parallel lsp_override_resolution_pp floor 0.5 (hardcoded) Both pipelines correctly invoke cbm_run_*_lsp through cbm_extract_file, so result->resolved_calls is populated identically — the bug was at the consumer. Changes ------- - New src/pipeline/lsp_resolve.h: static-inline cbm_pipeline_find_lsp_resolution and CBM_LSP_CONFIDENCE_FLOOR (0.6f). Single source of truth for the match rule (caller_qn equality + callee_qn short-name match + floor), with rationale and language-coverage notes inline. - pass_calls.c: include lsp_resolve.h, drop the local copy, rename the call site to cbm_pipeline_find_lsp_resolution. - pass_parallel.c: include lsp_resolve.h, drop lsp_override_resolution_pp, use cbm_pipeline_find_lsp_resolution at resolve_file_calls. Add an int lsp_overrides counter to resolve_worker_state_t (with adjusted cache-line padding for the new field), aggregate it per-worker, and emit it in the parallel.resolve.done log line so divergence between pipelines becomes observable in production telemetry. - tests/test_parallel.c: parallel_python_lsp_override_emits_lsp_strategy_edges builds a single-file Python scenario (Greeter class + main() in one file), runs the parallel pipeline, walks every CALLS edge, asserts at least one carries an "lsp_*" strategy in its properties_json. Pins both halves of the wiring at once: the shared find helper IS consulted in the parallel path, AND the LSP-attributed strategy survives all the way to the gbuf edge. Single-file rather than cross-file because per-file py_lsp's resolved_calls callee_qn uses the import module path rather than the project-qualified QN, so cross-file LSP edges currently get dropped by cbm_gbuf_find_by_qn — a separate cross-file QN-mismatch bug to track elsewhere. Full suite: 3250 passed, 0 failed (was 3249; +1 new regression test). |
||
|
Martin Vogel
|
7fa3acd0c6 | WIP: strict linting + RAM-first pipeline (lint fixes pending) | ||
|
Martin Vogel
|
fdb83f83f7 |
Fix Windows test compilation: mkdtemp, mkdir, setenv, pipe compat
- cbm_mkdtemp: _mktemp + _mkdir on Windows - cbm_setenv/cbm_unsetenv: _putenv_s on Windows - cbm_pipe: _pipe on Windows - Replace bare POSIX calls in 12 test files with compat wrappers - All test files now include compat.h |
||
|
Martin Vogel
|
0d216c7936 |
Resolve all clang-tidy, cppcheck, and clang-format warnings across 94 files
Enforce WarningsAsErrors: '*' with zero suppressions of entire check categories. All non-FP warnings fixed at source; true false positives get per-instance NOLINT with reasoning. Fixes include: braces around statements (1679), isolate declarations (75), explicit casts for multilevel pointer conversions, missing includes, function parameter documentation via NOLINT, and proper NOLINTBEGIN/NOLINTEND nesting. |
||
|
Martin Vogel
|
94a9a92f83 |
Add C-native pipeline with blocking clang-tidy, cppcheck, clang-format linting
Complete C implementation of the indexing pipeline (src/), parallel worker pool, graph buffer with merge support, SQLite writer, and 1893-test suite. Linter setup: zero warnings from clang-tidy (all checks enabled, no NOLINT suppressions), cppcheck, and clang-format. All issues fixed at source — proper headers for external linkage, GROW_ARRAY macro restructured to eliminate type parameter, null-deref paths guarded, named intermediates for suspicious-argument checks. |