py_eval_expr_type evaluated a call node's attribute receiver twice (once
in the container special-case, once in the general attribute path), so an
N-link chained call cost O(2^N) evaluations — real-world ~65-link builder
chains hung indexing for hours (#710). The evaluator also recursed once
per expression nesting level with no depth guard, so a pathologically deep
expression overflowed the native stack (#720).
Three guard layers, mirroring c_eval_expr_type's design in c_lsp.c:
1. Memoization: py_eval_expr_type is now a wrapper around the renamed
py_eval_expr_type_uncached. Results are cached per node in an
open-addressing table on PyLSPContext (arena-allocated, cold per file),
keyed by node identity (TSNode.id) — NOT the start byte, since every
leftmost descendant of a chain shares it and byte-keyed entries alias
distinct nodes into silently wrong resolutions. Entries carry a scope
generation; every scope bind/restore bumps it (O(1) whole-cache flush),
so re-evaluation under changed bindings (lambda re-walks, isinstance
narrowing) behaves exactly as before. Inserts are load-bounded (<75%)
and rejected when growth fails, so a full table can never spin.
2. Depth cap: PY_LSP_MAX_EVAL_DEPTH 256 (same as C_EVAL_DEPTH_LIMIT);
past the cap the evaluator returns unknown instead of overflowing the
stack. Truncated results — including any ancestor of a truncation —
are never cached, so a depth- or budget-limited `unknown` cannot
poison post-truncation reuse.
3. Step budget: PY_EVAL_MAX_STEPS_PER_FILE 10000 (mirrors
C_EVAL_MAX_STEPS_PER_FILE); pathological files degrade to unknown
instead of stalling indexing. Cache hits don't consume budget.
Distills PR #732 (memoization; fixed the start-byte cache key and the
missing staleness handling) and PR #758 (depth cap; raised 64 -> 256 to
match the sibling LSPs, added the never-cache-truncated ordering), and
adds the step budget plus reproduce-first tests: a 40-link chain that
must resolve its final link, a heterogeneous-receiver chain that fails
under byte keying, a 30k-deep expression crash guard, and a budget
degradation guard.
Fixes#710. Fixes#720.
Co-authored-by: LA-10 <lashuha@uwaterloo.com>
Co-authored-by: Dustin Persek <dustin.persek@protonmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The per-language "resolve calls in AST node" walkers recurse once per nesting
level. Four of them — Python (py_resolve_calls_in), Go (resolve_calls_in_node),
PHP (php_resolve_calls_in_node), Kotlin (kt_resolve_calls_in_node) — had no depth
guard, so a deeply-nested or cyclic file could drive them into a native stack
overflow (SIGSEGV) that takes down the whole index run. C and Java already guard
this (C_LSP_MAX_WALK_DEPTH / JAVA_LSP_MAX_WALK_DEPTH).
Add the same wrapper/inner depth guard to all four, backed by a shared,
env-configurable cap: CBM_LSP_MAX_WALK_DEPTH (default 512) in scope.h. Past the
cap the wrapper skips the subtree — those calls stay unresolved (graceful
degradation, not a crash). The walk_depth-- runs after the inner body returns, so
early returns can't leak the counter.
Reproduce-first: lsp_{python,go,php,kotlin}_deep_nesting_no_crash (30000-deep
nested calls, run under a forked child + WIFSIGNALED). RED without the cap (the
child SIGSEGVs — verified by raising CBM_LSP_MAX_WALK_DEPTH to 100M), GREEN with
it. Full suite 5752/0, no ASan/leak.
Part of the resilient-indexing effort (Track B prevention layer). Refs #668.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
bind_lambda_args and the method-reference arity path indexed the resolved method signature's NULL-terminated param_types array by the CALL-SITE argument index. A call with more arguments than the resolved overload declares (overload mismatch, varargs) read past the terminator and dereferenced whatever followed in the arena as a CBMType* - SIGSEGV while indexing elasticsearch (same out-of-bounds family as #427 and the bitcoin targs fix). Both sites now bound the index by the walked param count. Reproduce-first: 6-line fixture - run(Runnable) invoked with six lambdas - crashed rc=134 pre-fix; fork+exit-signal regression test lsp_java_lambda_args_exceed_params_no_crash added; suite 5574/0.
Indexing large OSS during the perf sweep crashed three LSP resolvers, all by stack overflow in their recursive resolve walks (macOS crash reports): elasticsearch — SIGSEGV in bind_lambda_args under deep recursive java_resolve_calls_in_node frames; bitcoin — SIGSEGV in cbm_type_substitute via c_adl_resolve under deep c_resolve_calls_in_node frames; microsoft/TypeScript — SIGBUS under an unbounded lookup_member_type cycle (cyclic type graph in reallyLargeFile.ts). The expression-eval guards (eval_depth) did not cover these walks. Each resolver now has a depth-guarded entry (walk_depth cap 512 for the Java/C AST walks, member_depth cap 64 for TS member lookup); past the cap the subtree resolves as unknown — graceful degradation instead of a crash. Reproduce-first: prod CLI probes crashed pre-fix (nested-call Java rc=139, nested-call C++ rc=139); fork+exit-signal regression tests added (TS cyclic shape needs a real cross-file registry, so it is verified at the real-repo tier — microsoft/TypeScript now indexes clean). Side-finding documented in the sweep report: Java BLOCK nesting >=3000 is minutes-slow (separate pathology, not this fix).
Root cause (corrected from the report): not a tree-sitter null-subtree sentinel — 0xffffffffffffffff is a freed/heap-corrupted pointer. The vendored ts runtime allocates via its overridable ts_current_malloc/free; under the prod build's MI_OVERRIDE=1, the Windows static-MinGW link (--allow-multiple-definition) resolves ts_malloc to mimalloc but ts_free can resolve to the CRT free (or vice-versa), corrupting the heap freelist and crashing mid-parse on large templated C++ headers (scales with parse churn, not syntax). cbm_init now calls ts_set_allocator(mi_malloc, mi_calloc, mi_realloc, mi_free) so ts allocate+free go through one allocator on every platform. Guarded to CFLAGS_PROD (MI_OVERRIDE=1); the test build stays CRT+ASan to avoid an alloc/free mismatch there. Tests in test_stack_overflow.c: allocator-binding mechanism + large-templated-C++ extraction guard (the C++ gap noted in #424).
AST traversal functions used fixed-size TSNode stack[] arrays. When the
DFS stack filled up, the child-push loop exited silently, dropping
entire subtrees without warning (e.g. 600 TS imports → 511, 130 Express
routes capped at 512). Reported in #199; also addresses #213 (large
TypeScript files producing zero nodes from stack exhaustion) and #215
(SEGV in template calls from stack overflow).
Adds ts_node_stack.h — a growable stack backed by the existing arena
allocator. Initial capacity matches previous fixed caps so small files
allocate no extra memory; doubles on overflow instead of truncating.
Old blocks are abandoned in the arena and freed on arena_destroy at
the end of file extraction (no realloc, no mixed lifetimes).
Applied to all 14 TSNode fixed stacks across 9 extraction files
(extract_calls, extract_channels, extract_defs, extract_env_accesses,
extract_imports, extract_semantic, extract_type_assigns, extract_type_refs,
extract_usages). walk_defs (uses walk_defs_frame_t, different struct
type) deliberately left as-is.
7 regression tests in tests/test_stack_overflow.c cover TS imports >512,
JS/Python calls >512, Go calls >1024, Express routes >150, deeply
nested calls, and YAML vars >256.
Cherry-picked from #217 with merge-conflict resolution against post-#206/#207
main:
- Makefile.cbm: keep TEST_STACK_OVERFLOW_SRCS plus main's new
TEST_ZSTD_SRCS / TEST_ARTIFACT_SRCS.
- extract_channels.c: keep main's scan_string_consts_js name (main also
added scan_string_consts_python after the PR was opened); declare
CHAN_STACK_CAP in the enum block.
- test_stack_overflow.c: replace sprintf with snprintf using the actual
remaining buffer size, since macOS clang rejects sprintf and naive
snprintf with sizeof(p) on a char* fails -Wformat-truncation.
Closes#199, #213, #215. Closes#217.
