The per-file tree-sitter parse was bounded by a 5s WALL-CLOCK budget. Under CI
contention a worker thread can be descheduled long enough for 5s of wall time
to elapse against near-zero CPU, so a small, perfectly parseable file was
abandoned with a false "parse timeout" and yielded ZERO defs. When that file
defined a symbol other files resolve against (e.g. celery.app.Celery.send_task
backing an ASYNC_CALLS edge, or a userpb gRPC client backing GRPC_CALLS), the
cross-file service edge then failed to classify -- surfacing as the flaky
convergence_probe / edge_types_probe service-edge assertions (count 0), a
verdict decided by wall-clock scheduling rather than the input (violates O9,
"no timeout decides a test").
Gate the parse on per-thread CPU time instead, with a generous wall ceiling as
a backstop:
- New portable helper cbm_thread_cpu_time_ns() (compat.c/.h): POSIX/macOS
clock_gettime(CLOCK_THREAD_CPUTIME_ID), Windows GetThreadTimes (kernel+user);
returns 0 on clock failure, degrading safely to the wall ceiling. It is NOT
cbm_clock_gettime, which is wall on every platform (QueryPerformanceCounter
on Windows).
- The progress callback's payload becomes {cpu_deadline_ns, wall_ceiling_ns};
it trips when this thread's CPU time passes the budget OR wall time passes
t0 + budget * 12 (~60s for the 5s budget). Descheduling burns wall, not CPU,
so a starved-but-parseable file survives; a CPU-spinning parse still trips
the CPU deadline; a genuinely stuck parse still trips the wall ceiling
(bounded, no hang).
RED-repro (deterministic, CBM_ENABLE_TEST_SEAMS only): a thread-local wall-clock
offset armed by CBM_TEST_WALL_STALL_ON emulates a descheduled worker (wall
reads budget + 1s ahead while CPU is untouched -- above the old 1x wall budget,
below the ceiling). async_calls_celery_wall_stall_seam arms it for the Celery
DEFINITION file and asserts ASYNC_CALLS >= 1. With the fix the parse completes
and the edge resolves; with the budget reverted to wall-only + a 1x ceiling the
def file is dropped and the edge count is 0.
Verified (macOS, sanitized): edge_types_probe + convergence_probe 107 passed /
0 failed with the fix; 58 passed / 1 failed on revert (only the seam test, at
the ASYNC_CALLS assertion); 107 restored. Regression smoke (extraction
parse_coverage pipeline lang_contract grammar_regression) 698 passed / 0
failed. lint-ci clean (cppcheck + clang-format Homebrew LLVM + NOLINT); the
Windows GetThreadTimes path is compile-checked by CI (not runnable on macOS).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
scripts/setup-windows.ps1 has 326 lines and its parse-coverage report read
"113-113,113-113,245-327" — the same line named twice, and an end line that
does not exist. Two separate faults, both in cbm_error_regions_push.
Past-EOF end line. A tree-sitter node that ends at column 0 stopped right
after the previous line's newline, so it holds no text on the row it points
at. Adding 1 to that row named a line past the end of the file whenever the
region ran to EOF. The node here is start=(244,2) end=(326,0). Clamp the end
to the row above when the end column is 0 and the node spans more than one
row.
Duplicate range. Line 113 carries two separate ERROR nodes, at columns 25-29
and 31-32, and each pushed its own range. A line range says nothing new the
second time. Drop a range that exactly repeats the one already open. The
drop runs BEFORE the cap check, so a repeat is never miscounted as a range
the cap threw away.
Only an EXACT repeat is dropped, never a range that merely overlaps. Each
range is judged separately afterwards by cbm_region_is_recovered, which asks
whether definitions starting inside that range cover it. Two ranges holding
the same numbers always get the same verdict, so dropping the repeat changes
nothing. Two different ranges do not. Merging 3-3 into 2-3 hands the wider
range's covering definition to an error that definition does not explain, and
a real parse failure then vanishes from the report.
That is not hypothetical. An earlier version of this commit merged on overlap
and broke perl_malformed_source_remains_partial_issue1838, the test added
with the Perl grammar refresh in 17b5a432. The malformed fixture produces two
ERROR nodes, at lines 2-3 and 3-3. Merged, the 2-3 range looks fully covered
by before_error and is removed, so parse_incomplete comes back false on a file
that plainly does not parse. That test now pins this boundary.
The real file reports "113-113,245-326".
Two tests, both proved RED first with the exact expected text:
coverage_repeated_error_line_reports_one_range_issue963 "2-2,2-2" != "2-2"
coverage_range_never_ends_past_the_last_line_issue963 "1-5" != "1-4"
Suites run on this change: parse_coverage 34, extraction 325, pipeline 264,
mcp 246, index_resilience 7 — all passing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
cbm_refine_regions_with_pp_lines asks cbm_line_is_toplevel_macro_call about
every preprocessor-parsed line in a region. That check walked the source from
byte 0 to find the line, so the refinement cost bytes times lines — on exactly
the whole-file-ERROR C shape the refinement exists to narrow.
Build one line-offset table per file, and hand the byte span to the macro
check. The line-based form stays for the two callers that ask about a whole
region, where a single walk is cheaper than a table. A file with no table (the
allocation failed) still gets the old walk, so the answer never changes.
Measured on one C file whose file-scope brace is split by an #ifndef, padded to
5k / 10k / 20k / 40k lines. Release build, one machine, seconds:
lines main branch this commit
5000 0.98 1.14 0.98
10000 3.30 4.02 3.41
20000 11.64 14.77 12.56
40000 44.04 58.58 46.90
The branch ran 33% over main at 40k lines and now runs 6.5% over. The reported
ranges do not change: 8-8, where main reports the whole file as 5-40000.
A profile of the 40k run no longer reaches the macro check. What is left is
cbm_line_contains and cbm_span_contains_callable_def, which main carries too
and this branch does not touch. The fixture holds 40k definitions, which makes
those two cost more here than on a smaller file.
parse_coverage, extraction, pipeline, mcp and index_resilience: 964 passed,
4 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
Phase 5. Four test groups, each checked RED before it was kept.
- The Studio Export range join puts ONE ",+<N>" marker at the end with the
summed drop count. A marker left mid-string makes every reader stop there
and silently lose the ranges after it. Reaching the join through the
pipeline needs an export file with 256+ error regions across two <Class>
elements, so it goes through a test seam, following the pattern already in
this repo (CBM_COVERAGE_MARKER_TEST_API).
- check_index_coverage emits every range in front of a marker, never turns
the marker's digits into a range, and reports "truncated" from BOTH caps —
the producer's and its own 256 limit.
- test_index_resilience now has a ceiling beside its floor: exactly one of
the two fixture files is flagged, the clean neighbour is absent, and the
range does not cover the whole file.
- The three _Thread_local forms are pinned as measured. Only the array form
fails today; the plan's Phase 0 also listed the pointer form, and that is
wrong on the grammar shipped now.
Also fixes 13 clang-format violations the earlier commits on this branch left
in cbm.c, mcp.c and pass_definitions.c. `make -f Makefile.cbm lint-format`
would have failed CI. The changes are whitespace only — the two reflowed tool
descriptions concatenate byte-identically, so no output moved.
Full suite: 7735 passed, 28 failed, 7 skipped. The 28 are the pre-existing
cli install/uninstall failures, identical at clean HEAD.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
Two silent failures in the parse-coverage report, both made visible by the
Phase 2 range refinement that came before this.
Two caps sat in series and both returned early without saying anything:
CBM_MAX_ERROR_REGIONS = 64 internal/cbm/cbm.c
COVERAGE_RANGE_MAX = 128 src/mcp/mcp.c
Raising only the first would have moved the clip from 64 to 128, so both move
to 256. This was live behaviour, not a theoretical limit: after Phase 2 split
one whole-file range into many small ones, src/cli/cli.c and tests/test_cli.c
both reported exactly 64 ranges — the cap binding, dead-on, twice. Every
coverage figure measured before this change was a floor. With the cap at 256
the true numbers are cli.c 13.9% (not 9.8%) and test_cli.c 3.1%, and the
longest list in the repo is 85 ranges.
A raised cap is still a cap, so the report now says when it clipped:
- cbm_error_regions_t gained a `dropped` counter, and cbm_collect_error_regions
walks to the end instead of stopping at the cap, so the count is exact rather
than a lower bound. That costs little — the walk never descends into an ERROR
subtree.
- cbm_error_ranges_str appends ",+<N>" when N ranges were thrown away.
- coverage_add_ranges reads that marker and sets "truncated": true, and also
sets it when its own limit stops the loop. Before this the marker was
invisible: the parser stopped at the '+' with no error and no leftover, so a
clipped list arrived looking complete.
- objectscript_export_append_error_ranges strips markers off both operands
before joining two Studio Export parts and adds one back at the end. A marker
left mid-string would make every reader stop there and silently lose every
range after it.
"Look at lines 1 to 13047" of a 13046-line file tells a reader nothing. Those
files now carry their own kind rather than being described as partially
covered.
New `parse_unusable` field in CBMFileResult, set when one range covers 80% or
more of the file. Its customers are non-C languages: the Phase 2 refinement
that narrows a whole-file range using the preprocessed parse only runs for C,
C++ and CUDA, so a Python, Java, Ruby or TypeScript file whose root node is
ERROR still reports 1-N. Verified against real files in all four.
The kind is `parse_unusable`, not `parse_failed`. index_coverage.kind already
means one of two things — indexed-but-partial, or a skip phase saying the file
was never indexed at all — and `parse_failed` reads as the second when it is
the first. The store.c schema comment, which is the only written record of this
vocabulary, now describes all three classes and says why.
Two places would have mislabelled the new kind as "skipped", which is exactly
that confusion: coverage_status fell through to its catch-all pass, and
add_coverage_report fell into its else branch. A reader who finds a file under
"skipped" believes it is absent from the graph, when it was indexed. Both now
have explicit branches. index_status gained parse_unusable_count so a CI gate
can read it without parsing anything else, get_code_snippet says "read the
source directly" instead of naming useless ranges, and the three tool
descriptions that listed two coverage kinds now list three.
Seven added. The cap test moved from 64 to 256; a new test asserts the marker
carries a real drop count and that nothing follows it; an inverse test asserts
an under-cap file carries no marker at all. For the new kind: a Python file
whose root is ERROR is unusable, a file with a local parse failure stays
partial, a clean file is neither, and — the one that matters most — the
that last one ever flips, the Phase 2 refinement has stopped working.
Full suite: 7732 passed, 28 failed, 7 skipped. The 28 are pre-existing
agent-client install/uninstall failures in the cli suite, identical in count
and identity at clean HEAD.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
src/cli/cli.c reported an error range of 1-13047 — the whole file. The file
indexed fine; the report was wrong. Three #ifndef _WIN32 blocks split a brace
(two `if` headers, one closing brace), so the raw tree-sitter parse cannot
resync at file scope, the root node becomes ERROR, and cbm.c takes its
whole-file branch.
The pipeline already parses these files a second time after preprocessing, and
that parse is clean. The report just never consulted it.
Build one byte per original line from the preprocessed pass, then cut each raw
error range down to the runs of lines the second parse could not vouch for.
Three rules, all found by running it and all load-bearing:
- An expanded line only vouches for its original line when it HAS TEXT. The
preprocessor emits a blank line where it dropped a branch; treating that
blank as proof suppressed every C range in the suite.
- Preprocessor directive lines (with backslash continuations) never count as
missing code — the preprocessor consumes them, so the second parse can never
vouch for one. Without this every #include block reported as a miss. Known
cost: a #define the raw parse really dropped no longer shows up on its own.
- A TOP-LEVEL macro invocation line never counts as vouched-for even when the
expanded line parses clean. The macro can expand to a whole definition that
the recovery walker deliberately refuses to adopt (#949), so a clean second
parse there proves nothing. An in-body invocation is the benign #1071 case
and is left to the existing macro subtraction.
The order of the three coverage steps is now settled by where each one's
evidence lives:
recovery subtraction -> before the refinement; its evidence is a whole
definition that STARTS inside the range, so it must
be asked while the range still matches the construct
the refinement -> middle
#1071 macro rule -> after the refinement; its evidence is per-line, so a
narrow range points at the call itself
Measured on this repo: src/cli/cli.c goes from one whole-file range to 64
ranges over ~9.8% of the file, tests/test_cli.c from 48.6% to ~2.9%,
src/cli/activation_transaction.c from 38% to 7.5%. What survives is honest —
the biggest remaining ranges in cli.c are genuinely discarded #ifdef _WIN32
and #ifdef CBM_CLI_ENABLE_TEST_API blocks, absent from the graph on this
platform.
Both percentages above are floors, not measurements: cli.c and test_cli.c now
land on exactly 64 ranges, which is CBM_MAX_ERROR_REGIONS. That cap drops
regions with no signal, and a follow-up raises it and adds a truncation marker.
Five tests, all red before the change: the range narrows to the dropped
branch; lines the preprocessor explained are excluded; a range never starts or
ends on a directive; real garbage beside a split brace stays flagged; a clean
file stays unflagged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
Brings the branch onto current main (fe85a6b2) as promised on the PR: the
only conflict was internal/cbm/helpers.h, where both sides added
declarations after cbm_find_child_by_kind; both blocks are kept (the PR's
cbm_find_children_by_kind prototype, then main's Lisp-family gates).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Vue change (#1852) built the shared embedded-script seam — the
included-ranges parse over the original buffer and the lang=/src=
resolver — but gated structural extraction on the host being Vue. A
Svelte component, an HTML page or an Astro file therefore still yielded
only File + Module nodes and IMPORTS edges from its <script> blocks: no
Function definitions, no CALLS, and a false dead-code verdict for
anything used only from a component.
Widen the seam to every host that declares an embedded spec:
- Structural extraction (definitions + calls) runs for every embedded
block; the per-host boolean is gone.
- The Vue attribute resolver becomes script_embedded_language() and
runs for all four markup hosts. It starts from the spec row's language
instead of hard-coding JavaScript, and learns HTML's type= attribute:
absent or empty, module, text/javascript and application/javascript
pass; application/json, importmap, text/x-template and the like bail,
so data blocks cannot leak garbage definitions into the graph. src=
and unsupported lang= values bail as before. CFML's <cfscript> carries
no attributes and keeps its spec-fixed language.
- Astro's frontmatter fence and <script> bodies default to TypeScript:
Astro type-checks both as TS with no lang= attribute to say so, and
the fence cannot carry one. The TypeScript grammar parses untyped
JavaScript unchanged.
Tests: the #1852 control that pinned Svelte/HTML/Astro at zero functions
and zero calls now asserts the real symbols; stay-at-zero controls cover
type="application/json" and <script src> in every host; both Svelte
blocks (context="module", and Svelte 5's bare module) contribute in
host-file coordinates; HTML's accepted type= forms and Astro's TS
frontmatter are covered; an end-to-end CALLS edge inside a .svelte file
is asserted. The Svelte and Astro label goldens gain the Variable their
fixtures always declared.
Verified on a 213-component Vue corpus: node counts per label and per
host language, edge counts and trace_path output are identical before
and after.
Fixes#1807
Signed-off-by: Brian McNaboe <bmcnaboe@gmail.com>
Follow-up to #1824, which mapped Blazor `.razor` to the C# host and extracted
`@page` routes from it. `.cshtml` is the same Razor syntax with the same C#
host and was equally unmapped: an ASP.NET Core MVC or Razor Pages application
produced no nodes at all for its entire view tree.
`@page` matters more here than it does for components. On a Blazor component
the directive is an optional annotation; in Razor Pages it is the directive
that *defines* a page, so the routable surface of a Razor Pages app lives
exclusively in this file type.
That is why the extension mapping alone is not enough. `cbm_path_is_razor()`
hardcoded a single `.razor` suffix, so a `.cshtml` file would have been
discovered and parsed but skipped by the directive scan -- present in the
graph, with none of its routes. Widened to a suffix table covering both.
Scope is deliberately the Razor family and nothing else:
- `.vbhtml` is excluded because there is no VB.NET language in this repo,
so there is no host to map it to.
- `.aspx` / `.ascx` are excluded because Web Forms is a different templating
syntax (`<%@ %>`, `runat="server"`) with no `@page` directive, so neither
the C# error recovery nor the directive scan applies.
Best-effort on the same terms as `.razor`: the C# grammar recovers the `@{ }`
and `@functions` blocks, and the surrounding markup lands in ERROR regions
reported through parse_partial.
Tests: the extension mapping; a Razor Page asserting the extracted route path
and method; and a negative case for a layout with no `@page`, since most
`.cshtml` files are layouts and partials and a spurious Route node per view
would be worse than the missing ones this fixes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
handle_calls reads a call's arguments through one tree-sitter field
lookup, ts_node_child_by_field_name(node, "arguments"). The vendored
Swift grammar declares no "arguments" field at all -- its own
ts_field_names[] table has zero occurrences, against one in Go and two
in TypeScript. Swift models a call as a target expression plus a
call_suffix, and the arguments hang off the suffix as value_arguments.
So args was null for every Swift call ever parsed, first_string_arg was
never populated, and no Swift HTTP call could raise an HTTP_CALLS edge
or a Route node. Alamofire, Moya and URLSession have been in the
service-pattern table the whole time and match the callee text
correctly; the URL simply never arrived.
Three changes, all in extract_calls.c:
- swift_call_args() reaches the argument list through call_suffix, in
the same shape as the existing objectscript_call_args() fallback and
used from the same place. A trailing closure has no value_arguments,
so it returns a null node and that call behaves as before.
- extract_url_or_topic_arg() unwraps Swift's per-argument
value_argument node, stepping past a leading value_argument_label.
Without it, dataTask(with: "/api/v1/widgets") yields the label "with"
rather than the path. PHP and C# already had the same unwrap for
their own "argument" node.
- is_string_like() gains line_string_literal, which is what Swift calls
an ordinary "..." literal. The list already held raw_string_literal,
so only Swift's common case was missing.
This is the layer underneath #1892 rather than the whole of it. A
literal URL argument now arrives; a literal nested inside a
constructor, as in URLSession.shared.data(from: URL(string: "...")!),
still does not, because extract_positional_url reads a literal, a
template string, a concatenation or a named constant and that shape is
none of them.
Reproduce-first: all three tests fail without the fix, two on a null
first_string_arg and one on HTTP_CALLS being 0.
Fixes#1892
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
A file whose final line lacks its newline leaves the grammar's mandatory
terminator MISSING at EOF. #1610 suppressed that phantom parse_partial on the
grounds that the node is ZERO-WIDTH: the parser consumed no source for it, so
by construction nothing was dropped.
That suppression tested `end == source_len` exactly. Trailing blanks are extras
owned by no node, so `ENTRYPOINT ["a"] ` + EOF parks the zero-width terminator
one byte short of source_len and the check missed it. This is why neither a
trailing blank nor an absent final newline flagged on its own -- only the pair
did, exactly as the reporter's byte-exact control matrix showed.
Treat "at EOF" as EOF modulo a trailing blank run. Every blank except newline
qualifies, deliberately not a hand-picked subset: a form feed is no more
content than a space, and #1610 exists precisely because whether a file got
flagged used to hinge on such incidentals. Newline stays excluded because a
terminated final line produces no MISSING terminator at all.
The rule remains ZERO-WIDTH ONLY. A width-bearing MISSING/ERROR at EOF is a
genuine loss and is still flagged (a Makefile's unterminated final recipe
really does vanish), as is any failure earlier in the file; both are pinned by
guard tests alongside the existing #1610 guards.
Verified end to end: the issue's own `cli index_repository` reproduction goes
from parse_partial_count 1 to 0 with node count unchanged, so the phantom flag
is gone without anything dropping out of the graph.
Fixes#1746
Signed-off-by: Rares Popa <2606875+rarepops@users.noreply.github.com>
Fixes#2010. Refs #1997.
Indexing a 14k-file TypeScript repository holds 13.3 GB resident at
post_extract on Linux aarch64, and about 3.1 GB of it is AST traversal
scratch that no CBMFileResult points at.
ts_nstack_init cut its stacks from ctx->arena, which is result->arena. In
the parallel path that arena is stored into ec->result_cache[file_idx]
(pass_parallel.c:963) and freed only after parallel_resolve and the infra
passes (pipeline.c:1324), so a structure written for a one-file lifetime is
in fact held for the whole index, once per file, concurrently. The header
claimed otherwise and was wrong: "freed when the arena is destroyed at end
of file extraction".
The largest instance is channels. CHAN_STACK_CAP is 4096 and sizeof(TSNode)
is 32, so scan_string_consts_js (extract_channels.c:104) and
extract_channels_js (:376) take 128 KB each, per JavaScript or TypeScript
file. Across 12,673 such files that is 3,168 MB, and it produced 23 Channel
nodes out of 164,684.
CBMExtractCtx gains a scratch arena, created and destroyed by the
cbm_extract_file_ex call that builds the context. Nothing else changes about
where anything lives: ctx->arena still owns every string a CBMFileResult
points at, and ctx->scratch owns only traversal stacks, which nothing points
at.
Nothing a CBMFileResult points at moved. A context built without a scratch,
and a file whose scratch arena fails to allocate, both fall back to
ctx->arena, which is exactly the behaviour that shipped before this change,
so no path can be handed a NULL arena and silently lose nodes.
The signature change is what makes the mistake unrepresentable.
ts_nstack_init takes the context rather than an arena, so handing it
ctx->arena is an incompatible pointer type; push and push_children read the
arena the stack recorded, so a stale argument is an arity error. Both fail
under -Werror rather than silently retaining memory.
The 55 init sites are NOT one uniform sed. 29 pass ctx->arena and 26 pass a
local alias, and that alias is shared with result allocations: in
parse_zig_imports (extract_imports.c:1093) the same `a` feeds cbm_node_text,
strip_quotes, path_last and cbm_imports_push (extract_imports.c:1101, 1106,
1108, 1109). Rebinding it would have moved import names and module paths
into memory that dies with the call while the result kept pointing at them.
Each site was edited on its own and no alias was rebound. Re-derive the set
with:
grep -rn 'ts_nstack_init' internal/cbm/extract_*.c
grep -rn 'ts_nstack_push' internal/cbm/extract_*.c
parse_lisp_imports used its alias only for the stack, so the alias goes too.
push_nested_class_nodes and push_class_body_children took an arena purely to
reach the stack, so they take the context now; the one call site passes ctx.
Two measurements, from two different builds, and they are not the same
number.
A macOS census build, which instruments every cbm_arena_alloc and sums the
blocks behind every live CBMFileResult arena, reports the retained per-file
arena set dropping from 11,226 MB mapped to 5,680 MB, and blocks from 48,045
to 34,186. Those are mapped bytes, not resident: that build logs
mem.allocator.bound_populations_only, so untouched block tails never commit.
The resident figures come from Linux aarch64, gcc 13.3, static build, where
the build logs mem.allocator.owned classes=all and mapped equals resident.
Five runs per side on the full 13,999-file tree: mem.collect post_extract
rss_mb 13293 -> 6752 (-49.2%), mem.phase peak_mb 14840 -> 8565 (-42.3%).
nodes 164684, parse_partial 280 and registry entries/defines/imports
787954/804999/100027 identical on all ten runs; edges 835869-835874 against
835867-835881, overlapping ranges, so run to run noise. Wall clock means
35651 ms against 30502 ms, but the ranges overlap and the host had other
load, so the only supportable claim is no regression. ObjectScript Studio
Export on a small fixture: nodes 18, edges 25, both sides.
Scratch block size is 512 KB, chosen from measurement. arena_grow fires on
one file in 12,000 at both 512 KB and 1 MB, and constantly at 256 KB, where
the two channel stacks alone are exactly 262144 bytes. 512 KB therefore
costs half the resident block per worker for the same growth behaviour, and
stays under MI_LARGE_MAX_OBJ_SIZE in the vendored mimalloc, which is
MI_LARGE_PAGE_SIZE/8, exactly 524288 bytes (types.h:426, with
MI_ENABLE_LARGE_PAGES defaulting to 1 at :115 and not overridden by this
build), so a 512 KB block is still bin-allocated from a 4 MiB large page.
Growth is not free at that size for the same reason: arena_grow doubles to 1
MiB, above that bound and therefore a singleton OS allocation. One file in
twelve thousand pays it, which is why it is accepted. The largest single
file's scratch high water mark on this corpus is 33.5 MB, now transient
rather than retained.
Tests. Nothing in the suite asserted on result->arena size, block count or
total_alloc after cbm_extract_file, which is why this went unnoticed for the
life of extract_node_stack.h. traversal_stack_not_in_result_arena_issue2010
pins the byte budget: extracting "export const x = 1;" as TypeScript charged
365984 bytes to the result arena before and charges 87456 after, a
difference of 278528, exactly the two 4096-entry channel walks plus the
512-entry ES import walk at 32 bytes per TSNode. Of the 87456 that remain,
7680 is the defs item array at GROW_ARRAY's starting capacity of 32 times
sizeof(CBMDefinition) 240 and the other 79776 is everything else the
extraction interns, none of it traversal scratch; the 128 KB bound is
therefore above the new figure with room and a factor of four below the old
one. extract_traversal_stacks_come_from_ctx_scratch_issue2010 in test_mem.c
pins the lifetime, since a byte budget alone would also be satisfied by
shrinking CHAN_STACK_CAP: it builds the extraction context directly over two
arenas it owns, runs cbm_extract_channels, and asserts the scratch took at
least the 262144 bytes of the two walks while the result arena did not.
extract_c_macro_hidden_call_survives_preprocessed_pass_issue2010 is not a
scratch test. pp_ctx carries ctx->scratch so every context in the file is
uniform, but nothing reads it there: pp_ctx reaches only cbm_extract_unified
and cbm_run_c_lsp, and neither extract_unified.c nor anything under
internal/cbm/lsp/ includes extract_node_stack.h, so no traversal stack is
built on that path today. The test guards the macro-expansion path itself,
which had no assertion on a call that exists only after expansion.
ObjectScript composite extraction is already covered by
pipeline_objectscript_export_preserves_calls_sequential_parallel and
iris_export_xml_multi_class, so no new test is added there; a per-call arena
is correct on that path by construction, since each call owns and destroys
its own.
Local: scripts/test.sh, 7558 passed, 0 failed, 8 skipped, 139 suites.
Alternatives considered
Thread-local scratch arena, reset per file, with a depth guard for nesting.
Rejected on three counts. This codebase never uses thread-local storage for
owned bulk memory; every existing _Thread_local here is a small cache, a
guard or a log buffer, and every per-worker resource is a field in a context
struct. It would have needed cbm_arena_reset, which has no production caller
and a documented subtlety with grown block sizes (arena.c:213-217). And
extraction threads are created per cbm_parallel_for call and joined
(worker_pool.c:71,88) with no thread-exit hook in this project's own code,
so the arena would have leaked once per thread per dispatch and turned the
leak lane red.
Save and restore mark and release on the arena. Rejected because a per-call
arena already makes nesting correct by construction, and a mark type plus
two functions would have to be added to both arena headers.
Per-walk arenas, one per traversal rather than one per file. Rejected on
cost: 55 sites would each need an explicit span, for the same bound.
Right-sizing the initial capacities alone. CHAN_STACK_CAP at 256 rather than
4096 would cut those two sites from 3,168 MB to about 198 MB, which is real,
but it tunes the constant while the lifetime defect remains at all 55 sites,
and growth is not free: 256 doubling to 4096 abandons 122,880 bytes, and one
file in this corpus took 33.5 MB of stacks. The cap and the scratch block
size are coupled, which is a further reason to change one at a time. Worth
doing separately.
Routing to a scratch inside extract_node_stack.h while keeping the arena
parameters. Rejected because a parameter that is ignored lies about the API,
and the alias hazard above would stay a review obligation instead of a
compile error.
Freeing the per-file result arena earlier. Rejected: the results are read by
parallel_resolve and the infra passes. That is the retention half #1925
addresses, and this change is independent of it.
Signed-off-by: Daniel Ochoa <daniel.ochoa@closedloop.ai>
Replaces the cursor ascent from 0b0d143c. That commit fixed a real hang --
ts_node_parent() is O(depth), so a parent chain is O(depth^2) per call and
O(depth^3) across a file where every level of f(f(f(...))) is itself a bare
call, which hung stack_overflow_b's 30,000-deep fixture at 900s. But it fixed
it by making each hop cheap rather than by removing the per-call walk, so the
cost stayed O(depth) per call and needed a 64-ancestor cap to stay bounded.
That cap FAILS OPEN. Past 64 ancestors the guard stopped suppressing, so
deep-but-ordinary code silently kept the fabricated edges the guard exists to
remove -- and nothing failed when it did.
Carry the answer instead. The unified walk binds a def's or lambda's parameters
when it opens that scope and unwinds them when it closes, so the guard is an
O(1) map lookup. This is the lesson CBMWalkScope already records one struct
away: carry walk state, never recompute it per node.
- CBMParamSlot is a name -> active-count map, not a set. `def outer(run):
def inner(run):` binds one name twice and leaving the inner scope must not
unbind the outer.
- Bindings are pushed at the end of push_boundary_scopes, after every scope
push for that node, so they land in the frame just pushed -- covering
function_definition (push_function_scope) and lambda (push_lexical_boundary)
alike -- and pop_expired_scopes unwinds to the frame's entry height.
- Allocation failure sets py_param_tracking_failed and the lookup answers
false forever after: a lost suppression, never a lost edge.
The repo's other binding table cannot serve this. CBMLexicalBinding is qsort-ed
in cbm_finalize_lexical_usages AFTER the walk and its active_end stays 0 until
then, so its binary search is invalid from handle_calls, which runs mid-walk
and before handle_usages.
extract_python_bare_call_scope_walk_is_bounded pinned the cap's fail-open
contract, which no longer exists, and is replaced by
extract_python_bare_call_flag_is_depth_independent. It keeps the shallow and
200-ancestor cases -- the deep one now asserts the call IS flagged, so
reintroducing a cap fails it -- and adds an unwind case: `handler` as both a
parameter and a module-level def, asserting the two shadowed calls are flagged
and the sibling call after that scope closed is NOT. A binding outliving its
frame would destroy that true edge, the one direction this guard must never
fail in. It passed on the first run.
extraction 327 passed; pipeline registry parallel complexity 407 passed;
stack_overflow_b 6 passed in 38s (46s with the cursor ascent, 900s rc=124
before it), lsp_python_deep_nesting_no_crash PASS.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_go_suppress_bare_field_ref dropped a Field-targeted reference when
strchr(ref_name, '.') == NULL — but the extractor strips the receiver on
every path that reaches the resolver (resolve_lhs_write_name records the
trailing field name of a selector LHS; is_reference_node records the
inner field_identifier), so the dot test could never be false for Go.
The guard was a blanket veto: all ~4588 Go Field nodes recovered by
one 61de19bb were unreachable by USAGE/READS/WRITES, including genuine
selector references (issue #1962, found in maintainer post-merge review
of #1944).
The selector-vs-bare distinction exists at extraction time and was
discarded; record it and let the resolver consume it:
- CBMUsage/CBMReadWrite gain is_member_access. The usage recorder sets
it when the reference node is a field_identifier (the member half of a
selector); resolve_lhs_write_name reports it through an out-param when
it takes the field/member LHS branch.
- cbm_go_suppress_bare_field_ref(is_go, is_member_access, target_label)
refuses a Field bind only for references that were never the member
half of a selector. All four resolver sites (sequential and parallel,
USAGE and READS/WRITES) pass the recorded flag.
Reproduce-first: with the veto restored, the extended #1942 fixtures go
RED on both resolver paths at the new asserts (t.err = nil must WRITE
the field; t.n must produce USAGE); with the fix both twins are GREEN
and the original bare-local negatives still hold. The dead
ASSERT_FALSE(..., "t.err", ...) unit case — an input Go's extractor
cannot produce — is replaced by flag-based cases.
Fixes#1962.
Signed-off-by: Ilya Brykau <ilya.brykau@orca.security>
Express, Fastify, gin and Laravel all put middleware between the route
path and the handler, and the handler comes last:
routerGet("/users", requireAuth, rateLimit, listUsers);
extract_handler_arg returned the FIRST argument that looked like a
function reference, so requireAuth won and the HANDLES edge pointed at
the middleware instead of listUsers. That is a wrong edge, not a
missing one, and it misleads anyone tracing a request through the
graph.
The scan bound hid a second fault. An arrow function matches none of
the accepted node kinds, so three inline middlewares pushed the real
handler past MAX_HANDLER_SCAN and no handler came back at all.
The loop now examines every argument and keeps the last eligible one.
Nothing that follows a handler matches the accepted kinds either, since
an options argument is an object node, so the last match is the
handler. HANDLER_START_IDX stays, because argument 0 really is the
path. MAX_HANDLER_SCAN had no other use and is gone.
A two-argument route has one eligible argument, so first and last agree
and the existing Express, Fastify, gin and Laravel route tests are
unchanged.
Both tests fail without the change, the first reporting "requireAuth"
where "listUsers" belongs and the second reporting no handler at all.
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
Java and Kotlin put no order on annotation attributes, so `path` can
appear after `method`, `produces` and `consumes`:
@RequestMapping(method = RequestMethod.GET,
produces = MediaType.APPLICATION_JSON_VALUE,
consumes = MediaType.APPLICATION_JSON_VALUE,
path = "/orders")
extract_route_path_from_args stopped after the third argument, so the
path was never read and no Route node formed. The endpoint was in the
source and not in the graph.
Four call sites read this function, so the miss covered plain decorator
arguments, Spring mappings, and both JAX-RS paths.
The loop now checks every argument. That matches find_drf_kwarg_in_args
a few lines below, which walks the same kind of node with no cap. Cost
stays small: the loop returns on the first path-shaped string, and each
argument's own subtree walk is still bounded by find_route_path_literal.
The recursive breadth guard inside find_route_path_literal keeps
DECORATOR_SCAN_LIMIT. That loop descends to CBM_DESCENDANT_MAX_DEPTH, so
breadth 3 by depth 6 is 729 visits at worst and unbounded breadth there
is not bounded at all.
The test fails without the change, reporting the class-level "/api"
route present and "/api/orders" missing.
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
get_architecture's routes aspect returned nothing for a Blazor
application: the entry surface of the app was invisible in the graph
even though every @page directive names one.
@page lives in markup above the @code block, which tree-sitter's C#
grammar never parses, so there is no AST node to read it from. Scan the
raw source for it instead. The match is deliberately strict — the
directive must be the first token on its line, followed by whitespace
and a double-quoted path beginning with '/' — so @pageSize and prose
mentions cannot match.
The route is attached to the file's module def. A .razor component's
class is implicit, so there is no class node to carry it, and the module
QN already is the component's identity. insert_def_into_gbuf is
label-agnostic and creates Route + HANDLES from route_path.
pass_route_nodes.c's ensure_decorator_routes gains "Module" for the same
reason. Extraction covers the full-index path on its own; this backstop
is what runs on an incremental re-index, so without it a component's
Route would appear on a full index and vanish the next time that one
file changed. Its loop bound now derives from the labels array rather
than borrowing the unrelated RN_STRIP_PASSES, so adding a label cannot
silently skip it. Verified no double-creation when both paths can fire.
Measured on a real Blazor application (36 .razor files): routes went
from 0 to 11, with 11 matching HANDLES edges, and no change to the
363 Class / 1775 Method / 3805 CALLS already extracted from its C#.
Covered by two new tests in tests/test_extraction.c.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: The1nk <23087600+The1nk@users.noreply.github.com>
The vendoring table claims our copy of a grammar matches the pinned
upstream commit. A local patch makes that claim false unless the patch
is written down, so a future re-vendor would drop the fix without
anyone noticing.
The row covers both changed lines, because both are the same defect in
the same file: OP_SYMBOL_SUPPRESSOR at line 131 and eat_operators at
line 514.
The row also names the two upstream commits that already carry these
changes, and the table intro gains one clause for the case. Upstream
fixed eat_operators in fb63a7004f07 on 2026-04-06 (their issue #558)
and the OP_SYMBOL_SUPPRESSOR entry in 6ab8d1d74ebd on 2026-08-10. Our
pin 8abb3e8b3325 is from 2026-03-20 and predates both, so these are
backports rather than local inventions.
That distinction changes what a re-vendor should do. The table's
instruction is "must re-apply these", which is right for the other four
rows -- upstream never took those. For this row it would mean
hand-applying a change the newer file already has. The row now says to
delete it instead, and the intro allows for that case.
MANIFEST.md is itself listed in scripts/vendored-checksums.txt, so its
own recorded checksum moves with the edit.
Refs #1892
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
scanner.c line 131 shifts a literal 1UL by 32 to build the entry that
suppresses the bang of a try!. unsigned long is 64 bits on Linux and
macOS but 32 bits on Windows, so there the shift count equals the width
of the type, which is undefined.
1ULL is 64 bits on every target this project builds for.
Nothing reports it today. The expression is a compile-time constant, so
no sanitizer sees it run, and Makefile.cbm:719 builds vendored grammars
with -w, which switches off -Wshift-count-overflow.
Upstream tree-sitter-swift made the same change in 6ab8d1d74ebd, after
the commit this grammar is pinned to.
Refs #1892
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
The Swift scanner keeps a 64-bit mask of the symbols that suppress a
match -- the rule that stops `try!` emitting its `!` as a token of its
own. It tests one bit per candidate:
uint64_t suppressing_symbols = OP_SYMBOL_SUPPRESSOR[full_match];
for (uint64_t suppressor = 0; suppressor < TOKEN_COUNT; suppressor++) {
if (!(suppressing_symbols & 1 << suppressor)) {
The mask is uint64_t but the literal `1` is an int, so the shift is an
int shift. TOKEN_COUNT is larger than 32, so once suppressor reaches 31
the shift runs past the width of the type. That is undefined behavior,
and every bit above 31 is tested against a value the standard does not
define.
Nothing caught it because nothing in the tree reached the suppressor
path. Any Swift force-unwrap does: `cached!` is enough.
UBSan reports it as:
scanner.c:514:47: runtime error: left shift of 1 by 31 places
cannot be represented in type 'int'
`1ULL` makes the literal as wide as the mask it is tested against.
The new test in tests/test_extraction.c cannot go red on its own. The
normal test build prints the UBSan message and carries on, which is why
this survived. The Windows CLANGARM64 leg runs UBSan in trap mode, and
there the same shift is an illegal-instruction crash -- so the test
exists to make sure that leg keeps parsing a force-unwrap at all.
scripts/vendored-checksums.txt records the new hash for the one changed
file, as scripts/security-vendored.sh --update writes it. Layer 8 of the
security gate compares vendored content against that manifest, so the
edit and its recorded hash belong in the same commit.
Found while adding Swift URL extraction in #1892 / #1976, and split out
of that PR so the vendored change can be reviewed on its own.
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
main does not compile. internal/cbm/lsp/type_registry.h declares
type_short_buckets, type_short_entries and type_short_bucket_count twice, and
cbm_registry_types_by_short_name twice with different parameter types.
Two changes implemented the same feature independently:
a4c0ffbc perf(cs): eliminate the C# corpus-proportional scans (2026-08-16, ours)
CBMTypeShortIter, cbm_type_short_iter_next
793716dc perf(lsp): index C++ short-name type lookups (PR #1679, astandrik)
CBMTypeNameIter, cbm_type_name_iter_next
They landed in different hunks of the same struct and the same header section,
so git merged them with no conflict and nothing failed until the merge commit
was built. This is the failure mode where a clean merge is not evidence that
two changes compose.
The two iterators are structurally identical -- same five fields, byte-identical
_next bodies reading the same reg->type_short_entries chain. They differ in one
place, and it matters:
#1679 on aux-index allocation failure, sets tail_i = 0, so the
iterator degrades to a full types[] scan and the candidate
set is unchanged.
a4c0ffbc sets tail_i = reg->type_qn_entry_count unconditionally, so on
that same failure it SKIPS the first type_qn_entry_count types
and silently yields an incomplete candidate set.
So this keeps a4c0ffbc's names -- CBMTypeShortIter reads correctly beside its
type_short_* members and beside CBMTypeEmbedIter and CBMFreeFuncIter -- and
#1679's body, which is the correct one. The surviving declaration carries both
rationales: the C++ short-name lookup and cs_resolve_type_name's step-9
fallback are now stated as two consumers of one index, which is what they are.
The merge also broke two comment/member pairings, repaired here: the duplicated
members had displaced the free-function index members from under their own
comment, and the CBMTypeShortIter block had been interleaved between
CBMFreeFuncIter's typedef and its declarations.
No behaviour change for either consumer beyond the allocation-failure path,
which strictly gains coverage for C#. c_lsp.c and cs_lsp.c now call one
function. #1679's tests, including the fault-injection case that NULLs the
buckets and asserts the full-scan fallback, carry over unchanged apart from the
type name.
Local: c_lsp, cs_lsp and registry -- 881 passed, 0 failed.
Co-Authored-By: astandrik <astandrik@yandex-team.ru>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Go calls written through an aliased struct field (h.svc.Method(ctx,
req)) previously fell through to fuzzy suffix matching because:
- the per-file Go dispatch walk was bypassed, so field selectors were
never descended;
- struct fields extracted as flat "Field" defs were dropped by the LSP
label map, so every Go struct registered with zero fields;
- receiver types written through import aliases (svc.Svc) produced
qualified names that exist nowhere in the project registry.
Changes:
- restore the Go walk in cbm_pxc_dispatch_file and descend field
selectors through the shared prebuilt registry, emitting
lsp_type_dispatch edges;
- fold per-field defs into their owning struct inside
cbm_pxc_collect_all_defs (one site covers both the prebuilt-registry
path and the per-file fallback);
- re-qualify the last alias segment through the calling file's import
map when the direct receiver-type lookup misses
(go_requalify_via_imports).
Tests:
- tests/test_parallel.c: cross-package field chain resolves end to end
with an lsp_type_dispatch edge in the parallel resolver;
- tests/test_go_lsp.c: alias-typed struct field re-qualifies through
the import map and lands on the real receiver type.
Signed-off-by: xkchok <143325001+xkchok@users.noreply.github.com>
Resolves conflicts with the Chialisp language addition (#1920) in three
repro ledgers. Both branches added one call-capable language, so beyond
the textual conflicts, the auto-merged shared totals had to be bumped a
second time:
- repro_language_registry.c: capability partition now 88/27/49,
EXPECTED_CALL_CAPABLE_LANGUAGES 115, EXPECTED_NON_CALL_LANGUAGES 51,
EXPECTED_MATRIX_B_ROWS 49
- repro_call_node_manifest.c: historical total 226, active primary 196,
direct calls 160, constructor calls 22, primary owners 196
- repro_call_argument_matrix_b.c: ROUTINE_ARGUMENT_LANGUAGE_COUNT 39,
matrix static assert 49 (RACKET..CHIALISP)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Refresh the complete generated Perl grammar bundle from upstream v1.2.1 at c3e17b31179bf8f658c9f37c7a3ea6a202212d5a, raising it from ABI 14 to supported ABI 15. Preserve the existing license and record the source and per-file provenance.
Add deterministic coverage for a valid format followed by a named sub, while retaining a malformed-source control.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
#1912 hung stack_overflow_b on all seven CI legs: rc=124 at 900s with zero
tests completed, on lsp_python_deep_nesting_no_crash.
The bare-call scope walk used ts_node_parent(), which is NOT O(1) -- it
restarts at the tree root and descends to find the parent (vendored
ts_runtime/src/node.c). Each hop therefore costs O(depth), a parent chain is
O(depth^2) per call, and since every level of f(f(f(...))) is itself a bare
call, O(depth^3) across the file. The fixture nests 30,000 deep, so this is a
hang rather than a slowdown.
A hop-count cap alone is NOT sufficient, and this was measured rather than
assumed: with a 64-hop cap the suite still timed out, because 64 x O(30,000)
x 30,000 calls is still ~5.7e10. When the cost is per hop, bounding the
number of hops cannot fix it -- the walk itself has to be cheap.
Use the unified walk's own cursor. WalkState.current_cursor is already parked
on the current node and ts_tree_cursor_goto_parent() IS O(1), because the
cursor carries its path stack, so ascending costs nothing per hop. Same
current_cursor idiom, including the ts_node_eq identity guard before trusting
the shared cursor, as usage_current_field_name in extract_usages.c. This is
the same lesson CBMWalkScope already records: carry walk state, never
recompute it per node.
The 64-hop cap is retained and now does real work, bounding the remaining
O(depth) per call. Precedent for the shape: CBM_LSP_PERL_MAX_WALK_DEPTH and
LEAN_MAX_PARENT_DEPTH in this file. Both the cap and a missing or mismatched
cursor FAIL OPEN -- they can only ever cost a suppression, never a true edge,
which is the safe direction for a guard whose justification is precision.
lsp_python_deep_nesting_no_crash: 900s timeout, 0 completions -> PASS.
stack_overflow_b overall: 6 passed in 46s.
The regression test pins the cap's contract deterministically rather than by
wall clock: within the cap a shadowed callee is flagged; past 64 ancestors the
guard fails open and leaves it unflagged. Raising the cap flips the deep case
and fails the test, verified.
826 passed / 0 failed across stack_overflow_a stack_overflow_b stack_overflow_c
pipeline registry parallel extraction complexity lsp_resolution_probe.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: Joseph-MingEn <125283161+Joseph-MingEn@users.noreply.github.com>
Martin Vogel