tree-sitter's GLR parser merges ambiguous parse-stack heads recursively in
stack_node_add_link (internal/cbm/vendored/ts_runtime/src/stack.c), once per
nesting level. On pathologically nested input whose grammar is ambiguous
(Perl's paren-optional calls, f(f(f(...)))), that recursion overflows the
native stack during the parse — a small ~1 MB Windows thread stack, and even
an 8 MB POSIX stack at extreme depth — before any extraction runs. Grammars
that are unambiguous here (Java, C++) never trigger the recursive merge.
Cap the recursive merge at CBM_TS_STACK_MERGE_MAX_DEPTH (512) via a
depth-tracked inner worker; the public stack_node_add_link is a thin wrapper
seeding depth 0, so every existing call site is unchanged. Past the cap the
ambiguity is left on the GLR stack instead of merged — a valid parse, never a
wrong one. 512 frames is well within a 1 MB stack while far exceeding any
realistic source nesting.
Add perl_glr_deep_parse_recursion_capped: a crash-isolating regression that
parses deep ambiguous Perl directly (a forked child on POSIX, in-process on
Windows), bypassing cbm_extract_file's CBM_PERL_MAX_PARSE_NESTING pre-parse
guard so the vendored cap itself is exercised rather than the guard.
Narrow vendored-runtime change; no other parsing behavior is affected. The
higher-level Perl pre-parse guard (issue #913 context) stays in place and can
be retired in a follow-up once this cap is proven in the field.
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
glibc _FORTIFY_SOURCE aborts realpath() into a sub-PATH_MAX buffer
regardless of the actual path length, and Windows has no realpath at
all. Size the buffers at PATH_MAX and skip the guard on Windows (the
#794 canonicalization logic is platform-independent; the four Unix
legs carry it).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
On v0.8.1, index_repository with repo_path "." flowed the literal dot
through everything derived from the path: the project was named
"root", the projects table stored root_path="." verbatim, the
post-dump integrity check flagged bad_root_path and auto-deleted the
entire DB — while the response still reported status "indexed" with
full node counts. Verified fixed on current main (the path is
canonicalized before derivation: correct project name, absolute
root_path, persistent and queryable DB).
The guard indexes with repo_path "." from inside the repo and asserts
the three things the old failure mode broke: path-derived project name
(never "root"), absolute canonical root_path in the projects table,
and a DB that survives the run and answers queries.
Closes#794
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The previous commit's clang-format pass reformatted the whole file
(tests are not in the lint-enforced LINT_SRCS set, so the committed
file was never format-clean under the local formatter). Restore the
original formatting; the branch's net diff against main is now exactly
the two new regression guards.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two reported indexer OOMs reduce to the same self-flattening loop the
trait-uses-itself fix closed (self-flatten short-circuit + snapshotted
loop bound in flatten_trait_into_class):
- a trait whose name collides with an aliased `use ... as` import it
composes (#765's exact single-file reproducer) — the alias
canonicalizes onto the enclosing trait via the short-name fallback
- a class sharing its short name with an aliased trait import (#951's
reproducer, reduced to one file: the trait definition being
unavailable is what forces the short-name fallback onto the class)
Both fixtures were verified RED against the pre-guard code (each hangs
the suite within the 90s alarm when the two guards are reverted, each
independently as the first test run) and GREEN with the guards in
place. Landing them as permanent regression guards; the issues close as
fixed-by-the-#920-pair.
Closes#765Closes#951
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A deeply nested Perl call chain f(f(f(...))) overflowed the stack while
parsing on small-stack platforms: tree-sitter's GLR parser recurses once
per nesting level in stack_node_add_link (vendored ts_runtime/src/stack.c)
merging the ambiguous parse-stack heads that Perl's f(...) grammar produces.
The overflow happens during ts_parser_parse, before any LSP walk runs, so
the CBM_LSP_PERL_MAX_WALK_DEPTH guards can never fire. This crashed
test-windows (CLANG64) with an AddressSanitizer stack-overflow and hung
test-unix (ubuntu-24.04-arm) to a 4h timeout on lsp_perl_deep_expression_no_crash.
Add cbm_source_nesting_exceeds(): an O(n) early-exit bracket-depth scan, and
skip extraction (clean has_error, zero edges) for Perl input whose nesting
exceeds CBM_PERL_MAX_PARSE_NESTING (128) before it reaches tree-sitter.
Scoped to Perl because only its ambiguous call grammar drives the GLR
recursion to the nesting depth; C/Java/Python parse the same shape with a
single stack head. 128 is ~10x above any real Perl nesting and well below
the depth where the parser overflows a 1MB stack.
Correct the lsp_perl_deep_expression_no_crash comment to describe the actual
mechanism (parser recursion, not the LSP walk).
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
perl_resolve_function_call split a package-qualified call name on the FIRST
"::" via strstr. For multi-level packages (Foo::Bar::sub) this yielded
pkg "Foo" and a sub name still containing "::" ("Bar::sub"), which never
resolved. The call then fell through to the generic bare-name fallback, which
matches on the bare sub name only — collapsing distinct packages' same-named
subs onto a single winner and orphaning the rest. Single-level calls
(Foo::sub) were unaffected, which is why the existing test missed it.
Split on the LAST "::" instead so the full package name reaches
perl_lookup_method, which resolves to the correct fully-qualified sub.
Adds perllsp_static_multilevel_package_call covering the multi-level form.
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
F1: dot the Exporter import target so seeded CPAN exported subs resolve.
perl_collect_qw_imports built colon-form targets (Scalar::Util::blessed)
but the stdlib registry keys curated CPAN subs in dotted form
(Scalar.Util.blessed) and lookup is exact-match. Wire in perl_pkg_to_dot
to dot the module portion and drop the now-unnecessary (void) cast.
F2: add a recursion-depth guard (CBM_LSP_PERL_MAX_WALK_DEPTH=512) to both
AST walkers (perl_resolve_calls_in_node, perl_pass1_scan) via a depth-
guarded wrapper + inner split, mirroring java_lsp's JAVA_LSP_MAX_WALK_DEPTH.
Past the cap a subtree is skipped (graceful degradation, no wrong edge),
preventing stack overflow on pathologically nested input.
F3: lock the shared last-"::"-segment normalization in lsp_resolve.h with a
direct regression test over cbm_pipeline_find_lsp_resolution: a qualified
static call still resolves AND the cross-namespace mis-attribution edge case
is bounded by caller-QN equality + the confidence floor.
F4: implement SUPER:: dispatch. Populate enclosing_parent_qn from the
enclosing package's first @ISA parent and resolve $self->SUPER::method() to
that parent's method (strategy perl_method_super). No known parent or
unresolved method emits no edge (zero-edge guarantee preserved).
Tests: perllsp_cpan_exported_function, perllsp_super_dispatch,
perllsp_super_no_parent_no_edge, lsp_perl_deep_expression_no_crash,
lsp_resolve_qualified_static_call_normalizes_colons,
lsp_resolve_misattribution_is_bounded.
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
Replace the plan-01 placeholder with the full Perl LSP test suite covering the
ten foundational resolution scenarios from 22-RESEARCH.md, plus the extract_perl
/ find_resolved / require_resolved / find_resolved_with_strategy helpers cloned
from test_php_lsp.c:
1. method via bless-assignment 6. use parent MRO
2. constructor class-method type 7. use base MRO
3. static package call 8. Exporter import (use Mod qw(f))
4. $self method dispatch 9. require fallback
5. @ISA inheritance 10. unresolvable receiver -> zero edges
Assertions match the resolver's actual QN scheme (module_qn.subname, dotted —
the Perl package governs dispatch, not the emitted QN), and the negative test
confirms the zero-edge guarantee for untyped scalar and unindexed package
receivers. Register suite_perl_lsp in test_main.c so scripts/test.sh runs it.
All 10 perl_lsp tests pass (3563 passed, 1 pre-existing unrelated failure:
search_code_multi_word in tests/test_mcp.c).
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
- cbm.c: include lsp/perl_lsp.h and dispatch CBM_LANG_PERL -> cbm_run_perl_lsp
- lsp_all.c: unity-include perl_lsp.c + generated/perl_stdlib_data.c
- Makefile.cbm: register TEST_PERL_LSP_SRCS and append to ALL_TEST_SRCS
- tests/test_perl_lsp.c: placeholder suite so the Makefile var resolves
(full suite + test_main.c registration land in plan 22-04)
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
Follow-up to #787/#828, disclosed during #828's review: the same clobber
class also affects two call sites #828 deliberately left out of scope.
create_env_configures_for_file (pass_definitions.c) and resolve_file_throws
(pass_parallel.c) both resolved their edge source via a direct
cbm_gbuf_find_by_qn(enclosing_func_qn) lookup with no guard against the
lookup landing on the shared Folder/Project node for directory-module
languages (Java/Go) -- the exact #787 root cause, just unguarded at two
more sites. A class-level construct (no enclosing function) has its
enclosing_func_qn fall back to the MODULE QN, which for a directory-based
module collides with the package's Folder node, conflating every
same-package file's CONFIGURES/THROWS edges onto one arbitrary source.
- create_env_configures_for_file: added the same
cbm_pipeline_node_is_dir_container guard already used by the three
#828-fixed finders, falling back to the per-file File node on a
directory-container hit.
- resolve_file_throws: was missing the `rel` parameter its sibling
resolve_file_calls/usages/rw already carry, so it inlined a raw lookup
instead of calling the shared, already-guarded find_source_node. Added
`rel` and switched to find_source_node, matching its siblings exactly.
Regression guard: tests/repro/repro_issue842.c, reproduce-first per the
issue's request for "an #828-style multi-order red fixture":
- CONFIGURES reproduces on a small 5-file sequential fixture (Go, since
Java's System.getenv detection is separately broken -- extract_env_key_from_call
reads a "function" field Java's method_invocation node doesn't have; noted
in the test, out of scope here).
- THROWS only reproduces past MIN_FILES_FOR_PARALLEL=50 (the worker-merge
path) -- confirmed empirically that the small sequential fixture already
attributes correctly even on unfixed HEAD, mirroring #787's own two-faced
(sequential/parallel) bug. Padded with 60 filler classes, following
repro_issue787.c's build_parallel_fixture precedent.
RED on unfixed HEAD (both tests): 0 passed, 2 failed. GREEN with this fix:
2 passed, 0 failed. Full scripts/test.sh (5958 cases) passes clean with
this change; scripts/lint.sh's two pre-existing failures (pass_lsp_cross.h
parameter-name mismatch, extract_defs.c cppcheck AST parse) are unrelated
and present identically on unfixed main.
Fixes#842
Signed-off-by: Dhruvy0804 <dhruvgarg553@gmail.com>
Completes the #963 coverage taxonomy: alongside the best-effort FAILURE
signals (parse_partial, skipped), the report now lists what was
deliberately NOT indexed — a deterministic, by-design class, never mixed
into the failure surfaces.
- discover: file skips are classified by reason (gitignore / cbmignore /
skip-list / ignored-suffix / fast-pattern / size-cap) and collected via
the new cbm_discover_ex2 (per-file entries capped at 2000 with the
uncapped total reported — truncation is explicit, never silent; whole
excluded subtrees stay exhaustive as before). Unsupported-extension
files are deliberately NOT recorded: no grammar for an extension is
not an ignore decision, and listing every README/asset would drown
the signal
- pipeline: ignored files captured per run (full + incremental) and
persisted into the index_coverage table as not_indexed_file rows,
excluded dirs as not_indexed_dir; incremental runs rewrite the
by-design set fresh from discovery instead of carrying old rows
- store: the deleted-file prune exempts not_indexed_* (deliberately
unindexed paths never have hash rows); the missed shadow graph keeps
showing FAILURES only — the UI's report-an-edge-case callout can
never fire for gitignored paths, and a project whose only coverage
rows are by-design gets no miss graph at all
- mcp: index_repository response gains not_indexed_files {files, count,
truncated, note} (dirs stay under excluded); index_status's coverage
report gains a not_indexed section (dirs + files + note) and no longer
lumps by-design rows into skipped; tool descriptions updated — the
by-design class is presented as deliberate and deterministic (change
the ignore rules and re-index), with no best-effort hedging
- tests: e2e guard (gitignored file + dir reported with reasons on both
surfaces, failure surfaces stay clean, missed graph excludes them,
rows survive an incremental re-index, ignored constructs genuinely
absent from the graph); store unit test covers the prune exemption
and shadow-graph exclusion
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Per review: no dedicated coverage tool.
- index_status now carries the coverage report (parse_partial + skipped
lists from the separate index_coverage table, plus the best-effort
coverage_note); get_index_coverage is removed. Tool description
updated so agents learn the coverage semantics from index_status;
index_repository and the response note now point there
- the read hook resolves coverage through index_status (same envelope
fields, unchanged note wording); verified end-to-end against a real
index (flagged file -> note with exact ranges, clean file -> silence)
- get_code_snippet responses from a parse_partial file now carry a
correlated coverage_note ("line range(s) X could not be parsed ...
the source above is ground truth") — the result names its file, so
the warning is precisely anchored. Entirely-skipped files cannot
appear here (no nodes), so no uncorrelated noise is possible; clean
files carry no note (asserted both ways in the e2e test)
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Four refinements to the coverage signal:
- extraction: subtract DEFINITE recovery before flagging. Tree-sitter
error recovery plus the ERROR-descending def walker often re-extract
constructs inside a failed region (verified: `def broken(:` comes
back as a def); a region whose every line is covered by definitions
that START inside it is not a miss, and a fully recovered file is not
flagged at all. Container defs (Module/Package) don't count as
evidence, and partially covered regions stay flagged — the
#ifdef-split case keeps its flag because the first branch's function
is genuinely lost
- naming: the query_graph option is graph="missed" (the graph shows
ONLY misses — "coverage" was misleading); shadow project renamed to
"<name>::missed"; tool descriptions updated so agents discover both
the option and its semantics
- hook: the CLI-installed PreToolUse augmenter now also matches Read
and injects a coverage note when the file being read is listed as not
fully indexed ("line ranges X-Y could not be parsed — the file
content you are reading is ground truth"). Safe against the old
issue-362 hazard: the augmenter is structurally non-blocking (always
exit 0, additionalContext only), mirroring the Gemini matcher that
already includes read_file; matcher upgrade bookkeeping updated
- ui: "Missed files" toggle in the graph sidebar renders the miss
graph as a second graph option — /api/layout gains graph=missed
(same db file, shadow-project scoping; base project name validated
as before)
Tests: recovery-subtraction cases (recovered def unflagged, garbage
region flagged, trailing recovered defs keep the flag), CLI matcher
tests updated, e2e resilience fixtures switched to an unrecovered
miss; verified end-to-end: hook emits the note for a flagged file and
stays silent otherwise, /api/layout?graph=missed serves the miss graph
and the code graph is unchanged, frontend build + tests green.
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Persist the parse-coverage signal outside the graph and make the misses
queryable both as a list and as a graph.
- store: new index_coverage table (project, rel_path, kind, detail) —
coverage is metadata ABOUT the graph and never mixes into the graph
tables; replace/get/free APIs with a deleted-file prune keyed off
file_hashes (the live-file set)
- store: materialize a derived miss graph under the shadow project
"<project>::coverage" (Project -> Folder chain -> File{kind,detail},
CONTAINS_FOLDER/CONTAINS_FILE edges), rebuilt from the table inside
the same transaction on every replace; invisible to list_projects
(which scans the cache directory, not the projects table)
- pipeline: coverage is written in the same persist step as every other
post-dump SQL write (hashes, ADR, FTS). Full runs replace the whole
set; incremental runs merge rows surviving from files not re-extracted
with this run's fresh entries, so a fixed file's flag clears and
deleted files prune automatically
- mcp: new get_index_coverage tool (per-project parse_partial + skipped
lists with the best-effort note); query_graph gains graph="coverage"
to run the same cypher against the miss graph; the File-node property
marker from the previous commit is dropped in favor of the separate
table — the code graph gains no coverage rows
- wording: constructs in flagged ranges MAY be missing — tree-sitter
error recovery still salvages some (verified: a function inside a
flagged #ifdef region and even a broken def were still extracted)
- tests: store round-trip/prune/shadow-graph unit test; e2e asserts the
response fields, the advertised coverage-graph query, code-graph
purity, and that the flag clears after the file is fixed (incremental
route)
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Files whose parse tree contains tree-sitter ERROR/MISSING regions were
silently indexed as if complete — constructs inside those regions are
absent from the graph with no signal anywhere (classic trigger: the
preprocessor-blind #ifdef-split-brace pattern in C).
- cbm_extract_file: set parse_incomplete and record the 1-based line
ranges of the top-most error regions (bounded at 64), serialized as
"start-end,start-end" into the result arena
- pipeline: record such files under the new phase "parse_partial"
(distinct from skips — the file IS indexed) in both the parallel and
sequential extraction paths; stamp the File node with
{parse_incomplete:true, error_ranges} on full and incremental runs
via a new cbm_gbuf_set_node_props (upsert survivor rules bypass)
- mcp: report parse_partial {files, count, truncated, note} separately
from skipped[] (skipped_count semantics unchanged); the per-run
logfile lists parse_partial rows; the index_repository tool
description documents the signal and the query_graph pattern to find
flagged files
- tests: new parse_coverage suite (RED-first, 7 cases incl. false-
positive guards and the region cap) plus an end-to-end resilience
guard asserting response fields, logfile, File-node marker, and the
advertised query_graph query
The signal is explicitly best-effort: a flag means constructs in the
listed ranges were dropped (prefer grep there); the absence of a flag
is NOT a completeness guarantee.
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The #363 memory axis (CBM_MEM_BUDGET_MB) has shipped, so this reproduce-first
case is now a permanent GREEN guard, not an open bug. Its header claimed the
override "does not exist" and asserted the budget is honoured "regardless of
host RAM" — both stale: the resolver now clamps the override to effective
(cgroup/host) RAM. On a <4 GiB host the old 4096 MiB assertion would clamp and
falsely flip the bug board RED.
Rewrite the header to describe the shipped + clamped behaviour and drop the
override to 128 MiB, which is at/below any realistic host's RAM so it is
honoured exactly and stays stable on every runner. The clamp path itself is
unit-tested in tests/test_mem.c (resolve_budget_override_clamped_to_total).
Signed-off-by: Sam Li <yangsec888@gmail.com>
Fold the inline CBM_MEM_BUDGET_MB override logic in cbm_mem_init into a pure,
testable cbm_mem_resolve_budget() that returns a result struct
(budget/source/clamped/invalid), so the parse + clamp lives in exactly one
place and cbm_mem_init only surfaces the outcome as log lines.
Parsing now matches the strict src/foundation/limits.c convention (errno/ERANGE,
reject trailing garbage, positive-only): a fat-fingered value like "8GB" or a
20-digit typo becomes a warning + safe ram_fraction fallback instead of a
silently wrong budget. A valid-but-huge value clamps to detected total RAM
(logged mem.budget.clamped) rather than overflowing the MiB->bytes multiply and
wrapping to a near-zero budget (which would pin cbm_mem_over_budget() true and
stall indexing). mem.init now logs source= on both the override and fraction
paths, and the README gains the CBM_MEM_BUDGET_MB row it lacked.
Hardening of the existing #363 knob.
Signed-off-by: Sam Li <yangsec888@gmail.com>
cbm_cli_sha256_file shelled out to sha256sum/shasum, which is unportable:
the tool may be absent, and under cmd.exe the single-quoted path is not
dequoted so the hash of a downloaded release could never be computed on
Windows. Add a small in-process SHA-256 (FIPS 180-4, src/foundation/sha256.c)
and use it directly — no external tool, no per-OS branching, no popen. The
checksum test now covers three NIST vectors (empty, "abc", and a 56-byte
input that forces padding into a second block).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The layout endpoint treated the default render cap (2000) as a ceiling:
an explicit max_nodes above it was silently clamped back down, so the UI
could never show more than 2000 nodes regardless of the request. Make the
default a default — an explicit request is now honored up to a hard
ceiling of 10M nodes; CBM_UI_MAX_RENDER_NODES still adjusts the ceiling
for constrained deployments. The default budget rises to 5000, and the
layout refinement pass scales its iteration count down past 100k/500k
bodies to keep very large budgets responsive. Edges need no changes: the
layout already returns every edge between loaded nodes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
index_repository canonicalizes repo_path but imposes no workspace boundary,
so a caller-supplied path can resolve anywhere on disk. Add an opt-in
CBM_ALLOWED_ROOT: when set, a repo_path that resolves outside it is refused
(reusing the same containment check the read sinks use). Unset by default, so
the standard index-the-given-path behaviour is unchanged; deployments where
repo_path may be influenced by an untrusted caller can set a boundary.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The UI handlers accumulate JSON with `pos += snprintf(buf + pos, size - pos,
...)`. snprintf returns the length it WOULD have written, so on truncation
pos ran past the end of the buffer and the next call's `size - pos` wrapped
to a huge value and wrote out of bounds — reachable through the directory
browser, whose readdir entries can fill the buffer before the trailing
parent/roots appends. Route every accumulator through a clamping
http_appendf() helper that pins pos at the buffer size on truncation. Adds a
forked-child regression test that browses a directory wide enough to fill the
buffer.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The macOS digest path invoked `shasum -a CBM_SZ_256` — a literal string
(not a macro; string literals do not expand) naming an invalid algorithm, so
the digest always failed on macOS, and the self-update path then treated a
could-not-verify result as non-fatal and installed the download regardless.
Use the correct algorithm, and install only a positively-verified download:
a mismatch, a missing checksum entry, or an unavailable hash tool now all
abort. Adds a known-vector digest regression test.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
base_branch is spliced into `git diff --name-only "<base>"...HEAD`. The
existing check rejected shell metacharacters but not a leading '-', so a
value like --output=<path> was passed to git as an option and wrote the diff
to an arbitrary file. A real git ref never starts with '-'; reject it
alongside the metacharacter check.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The graph-UI server binds to loopback only and reflected a localhost Origin
for CORS, but did not inspect the Host header — so a request routed to the
local port under a foreign name (a rebinding DNS record, a proxy) was served
normally, reaching the state-changing endpoints. Parse the Host header and
refuse any request whose Host is not a loopback address before dispatch; a
request with no Host (local HTTP/1.0 tooling) still proceeds. Adds a
live-socket regression test.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
get_code_snippet resolved a result path and confirmed it stayed under the
project root before reading, but the search_code source-attach path built
the same kind of path and read it with no such check — so a result whose
indexed path resolves outside the root (a `..` segment, or a symlink /
junction followed during discovery) could be read back into the response.
Extract the existing containment check into a shared cbm_path_within_root()
and route both read sinks through it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Shane McCarron