mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-07 08:43:35 +08:00
Martin Vogel
7b8b7038f7
An 81k-file TypeScript corpus previously never finished indexing. Three
stacked defects, found by stack-sampling the live grind and reading the
supervisor's quarantine records:
1) walk_defs pushed children via index-based ts_node_child(i), which is
O(i) per call in tree-sitter — O(n^2) per wide node. A 3.5 MB file
with ~580k flat comment siblings needed ~1.7e11 iterator steps and
hung extraction past the supervisor's quiet-timeout. All six descent
sites (including the default loop) now collect children linearly with
a TSTreeCursor (wd_collect_children); small nodes keep the direct
index path. Guard: extract_wide_flat_file_is_linear (400k comment
siblings — the monster's exact shape — bounded at 30s).
2) The sequential cross-LSP driver handed the FULL def list to a full
per-file registry build+finalize — O(files x defs); 74% of samples
sat in build_qn_index. The parallel resolve worker's dispatch
(module-def filter -> shared prebuilt registries -> filtered
fallback) is now extracted into cbm_pxc_dispatch_file and drives
BOTH pipelines — one path, one semantics. The shared registries
live in a caller-owned arena that outlives the calls pass
(resolved_calls borrow registry strings; freeing at pass end was a
use-after-free ASan caught in pass_calls). Rust is exempt from the
def filter: Cargo-manifest cross-crate resolution needs defs from
OTHER workspace crates that the own+imports filter drops (this
starvation predated the unification on the parallel path). Guard:
pipeline_seq_ts_cross_uses_shared_registry (full builds 40 -> 0,
cross-file edge preserved).
3) The crash supervisor's recovery re-ran the worker SINGLE-THREADED to
keep one exact marker. At scale that fell into the sequential crawl,
was killed as a hang mid-pass, and the stale extraction marker got
four innocent files quarantined, one 15-minute retry at a time.
Recovery (and the terminal partial run) now re-run PARALLEL; the
marker is an append journal ("S <rel>" / "D <rel>", written by
extraction and both resolve drivers), the suspect set is the open-S
in-flight set, and a file is quarantined only when it recurs across
two consecutive failed runs — one per round, oldest open S first;
disjoint consecutive sets stop the loop rather than blame an
innocent. Guard: index_recovery_parallel_quarantines_crasher
(single-threaded spawn count must stay zero; verified RED against
the old loop).
Also: parse_ts_type_text gains a dynamic work budget proportional to
the input (1M + 64x source bytes, CBM_TS_TYPE_BUDGET override, warn
once then degrade to unknown) replacing unbounded recursion cost.
Measured on microsoft/TypeScript (81,398 files): infinite -> 33s cold
full index, 295k nodes / 779k edges, zero dangling edges, zero files
skipped — the monster file itself now indexes cleanly. Full 9-language
sweep green with no quality flags; kernel unchanged at 390s / 8.5M
nodes / 0 dangling.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>