Commit Graph
543 Commits
Author SHA1 Message Date
Martin Vogel 8c1a9d61c2 fix(extract): no clock decides what ends up in the graph
Indexing the same C# corpus twice on the same machine produced 5,794,336 edges
and then 5,794,328. Every other edge type matched exactly, every semantic
checksum matched (447,064 functions, 81,534 tokens, identical vec/idf/toks),
and the difference was ten WRITES edges out of one generated file.

Three places in extraction decided graph content from a clock:

  * the unified AST walk stopped on a thread-CPU deadline;
  * a file whose parse spent more than half its budget was refused the
    per-file and cross-file LSP walks;
  * so was a file whose parse plus walk exceeded the budget.

The first is the one that bit. Any file large enough to reach the deadline is
cut at whatever node the clock lands on, and five generated .NET JIT tests
reached it on every run: hugeSimpleExpr1.cs stopped at node 2,660,352 in one
run and 2,574,336 in the next, out of 2,774,794. That file's tail is field
assignments, so the two runs disagreed by exactly ten WRITES edges. The graph
was a function of how fast the machine happened to be running, and nothing said
so: result->walk_truncated was set and no code anywhere read it.

The deadline existed because those files cost 18-48 us per visited node, and
that turns out to be one fixable thing. csharp_callable_value_site() was the
only language branch in call_reference_candidate_site() still climbing with
ts_node_parent, which tree-sitter answers by descending from the ROOT, scanning
children at every level; every other language already climbs on the walk
cursor. On expressions megabytes deep that was ~85% of the entire unified walk
(sampled: stamp_usage_site -> ts_node_parent -> ts_node_child_with_descendant).
It now takes the cursor and the parent the caller already resolved, and
is_direct_argument_value_cursor() is split so a caller that has stepped the
cursor onto the parent can continue from there rather than paying for it twice.

With that fixed, the budgets are not needed to make indexing finish, so they
are gone rather than made deterministic. A cutoff in nodes would at least be
reproducible, but it still answers "what is in this repo?" with "as much as we
felt like reading". CBM_WALK_MAX_NODES reinstates one for an operator who wants
it; the default is no budget.

Removing the two LSP-skip rules was checked against the file and the path they
were added for (a 23 MB single-expression JIT test that once cost 354 s in the
walk and then crashed the cross-file resolve): the whole 12k-file C# corpus
twice with cross-file resolve, that file alone, and synthetic C# nested 10,000
levels deep — no crash, and past roughly that depth tree-sitter stops producing
a usable tree, so a tree deep enough to endanger a recursive walk never reaches
one.

C# corpus, before -> after:

  edges      5,794,318 / 5,794,328 / 5,794,336   ->  5,794,873 every run
  truncated  5 files, unreported                 ->  0
  time       ~115 s                              ->  97-101 s

Deterministic, about 550 edges more complete (the tails nobody was walking),
and faster than the build that was cutting them off. All ten bench corpora now
return identical nodes and edges across two runs.

A truncated walk and a skipped LSP walk are now reported in the coverage report
with how far the walk got ("1673216/4133946 nodes walked"), like a partial
parse. A file the walk abandoned halfway used to be reported as fully indexed,
which is a graph quietly lying about its coverage.

Tests: extract_csharp_argument_values_use_the_walk_cursor asserts no occurrence
falls back to the root-descending parent lookup — 128 fallbacks without the fix
and 0 with it. Its fixture parenthesises the argument on purpose: a bare
Sink(Target) needs no climb at all, so it cannot tell the two walks apart and
passed either way while the bug was in place.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-09-19 15:50:41 +02:00
Martin Vogel c7ccb479b4 fix(grammar): give the properties scanner per-parser state
Indexing a repo with .properties files in it produced a different graph from
one run to the next. Every few runs one small file - 253 bytes, valid, the same
file that had indexed fine a moment earlier - vanished from the graph, and the
coverage tree put raw-path File/Folder nodes where it had been. On the kotlin
corpus that showed up as node counts wandering between 33,486 and 33,494.

The vendored tree-sitter properties grammar keeps its external scanner's entire
state in a file-scope static:

    static bool reached_eof = false;

    bool ..._external_scanner_scan(void *payload, TSLexer *lexer, const bool *valid) {
      lexer->result_symbol = FAKE_EOL;
      return reached_eof = !reached_eof && valid[FAKE_EOL] && lexer->eof(lexer);
    }

That flag is how the scanner refuses to emit a second end-of-input FAKE_EOL,
and the grammar needs exactly one to close a file. One static is one object for
the whole process, so with several index workers parsing .properties files at
once, whichever thread reached EOF first set the flag and the next thread's
`!reached_eof` was false: its parse never received the FAKE_EOL it was waiting
for and sat at end-of-input asking for a token that would never arrive. The
per-file parse budget stopped it five seconds later and the file was dropped.

Measured on a 253-byte file: 1,060,630 progress callbacks, ~106 M parse
operations, byte offset pinned at EOF, has_error false - the parser was not
recovering from an error, it was waiting.

The state now lives in the payload from ..._external_scanner_create(), which is
what tree-sitter's create/destroy contract is for. The serialize/deserialize
wire format is unchanged: the state is still the returned length.

Proof, interleaved under identical load so both binaries meet the same machine,
40 runs each: unpatched 6 stalls, patched 0. Under load the kotlin corpus now
returns 33494/203579 ten times out of ten and java 693129/5523595 three times
out of three; both used to wander.

All 103 vendored scanners were audited for the same shape. This was the only
one that WRITES a process-wide static. haskell's debug_parse_metadata and
purescript's res_cont/res_fail are read-only - constants missing a const - and
are left alone.

Upstream 6310671b24d4 still carries the static, so MANIFEST.md records this in
the local-patch table and scripts/vendored-checksums.txt is repinned.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-09-19 15:50:31 +02:00
Martin Vogel 1d58385b51 refactor(mem): route this branch's allocations through the memory core
The memory-core linter is a ratchet: raw malloc/calloc/realloc/free/strdup
sites may not grow, because allocations belong on one accounted path
(src/foundation/mem_core.h) rather than spread across hundreds of call
sites. This branch's own work had added 35 raw sites across nine files --
the parallel k8s prep, the configlink key matching, the worker pool's
counted path, the Go shared stdlib registry, the spill namespace copy,
the route service array and the namespace-name arrays.

All of them now use cbm_alloc/cbm_calloc/cbm_realloc/cbm_free/
cbm_mem_strdup. One was a latent bug rather than a style point:
pass_k8s freed a buffer that k8s_read_file had allocated with raw malloc,
so converting only the free would have mismatched the allocator -- the
read function and every free of its result are converted together.

The Windows free-space path widens its argument through the memory core
instead of cbm_utf8_to_wide(), which allocates with raw malloc.

Two files improved enough to tighten their baselines, which the ratchet
requires so the gain cannot be quietly given back: sqlite_writer.c 92->74
(the index-cell arena), pass_k8s.c 13->6 and registry.c 19->17.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-09-18 23:27:23 +02:00
Martin Vogel c3714f1133 fix(mem): size the budget from free memory, and relieve on real scarcity
Indexing the linux kernel with NO ceiling set took the machine down: on a
48 GB host the default budget is 50% of TOTAL ram (24 GB), the run used
essentially all of it (24.5 GB peak RSS), and with a 12 GiB VM already
running there was nothing left. The budget never fired -- it read 22 GB
charged against 24 GB -- while the host was out of memory.

Total RAM is the CEILING; what is free right now is the CONSTRAINT.

  - cbm_mem_clamp_to_available() reduces a fraction-derived budget to fit
    what is actually free, reserving a quarter of it (held between 1 and
    8 GB) for the OS and everything else, never below a 512 MB floor. An
    explicit CBM_MEM_BUDGET_MB still wins -- it is the user's deliberate
    choice -- but is warned about when it exceeds free memory, because
    the failure it buys reads like a product bug rather than a setting.
  - cbm_mem_should_relieve() = our budget exceeded OR the machine short.
    Deliberately separate from cbm_mem_over_budget(): system pressure
    must RELIEVE (spill, reclaim), never abort, since the spike may not
    even be ours. The extraction loop now enters spill on real scarcity,
    and the dump phase -- where the kernel run actually died, with our
    charge comfortably under budget -- keys its free-heavy decision on it
    too. cbm_mem_system_under_pressure() existed and had no callers.

Same scenario after: budget clamped to 18,947 MB, peak RSS 19.5 GB,
status indexed in 185.7 s, machine fine.

Spill also gets limits and cleanup, because relief that fills the disk or
leaves gigabytes behind is not relief:

  - a free-space preflight (never start spilling below 10 GB free) and a
    ceiling at half of what is free; past it results stay in memory and
    back-pressure takes over. cbm_fs_free_bytes() is new (statvfs
    f_bavail / GetDiskFreeSpaceExW quota).
  - POSIX unlinks each spill file while holding it open, so the space
    returns on ANY exit including SIGKILL -- a killed run cannot leave
    spill behind because there is no name left to leave it under.
    Windows keeps close-path removal plus a sweep where an open file
    refuses deletion, which is exactly the liveness test.
  - the staging-database sweep now reclaims orphans of EVERY project in
    the cache directory, not just the one being indexed. A kernel index
    killed by the OOM killer used to park ~15 GB until someone re-indexed
    the kernel, and indefinitely if nobody did. The per-stage lock probe
    still keeps a live writer's stage, whichever project owns it.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-09-18 23:26:28 +02:00
Martin Vogel 92e7e6a913 fix: keep large thread-locals off the static TLS block
Indexing broke on x86-64 Linux: the index worker refused to start and
SIGKILLed its own process group, so every venue reported nothing but
"index worker ended with killed (exit=-1, signal=9)".

glibc places a thread's static TLS block inside the stack allocation it
makes for that thread. This branch's new thread-local caches -- a
512-slot field cache (~24 KB), per-depth tree cursors (~4 KB) and two
parked CBMArenas (~8 KB; an arena carries blocks[256] + block_sizes[256])
-- took static TLS from 47,905 to 93,513 bytes. The 64 KB parent-death
watchdog thread then could not be created at all, and a worker without
process-tree containment correctly refuses to index.

Four layers, because there were four things wrong:

  - The big thread-locals move to the heap behind a thread-local POINTER,
    allocated once per thread and reused, each with a correct fallback if
    allocation fails (the field cache and cursor pool are caches; a miss
    is slower, never wrong). Static TLS is back to 52,233 bytes.
  - PARENT_WATCHDOG_STACK_SIZE 64 KB -> 256 KB. "The watchdog only polls,
    a tiny stack suffices" was true when written and false once TLS
    reached ~50 KB, because the TLS block comes out of that same
    allocation.
  - cbm_thread_create retries with the default stack on EINVAL. A stack
    size is a hint about what a thread needs; the platform refusing the
    hint is not a reason to fail to create the thread.
  - tests/test_thread_stack_tls_contract.sh (Step 5a of scripts/test.sh)
    keeps static TLS a quarter clear of the smallest requested stack,
    reading that smallest size from the sources so a new small stack
    tightens the gate automatically.

Two diagnostics that made this findable at all, and that stay:
smoke-test.sh prints the worker log its failure message names (CI deletes
that sandbox with the job, so the one artifact naming the cause was
unreadable), and worker_containment_unavailable() now reports WHICH of
its conditions failed, with pid/pgrp/ppid.

Verified in the amd64 container, the venue nothing on the local ladder
covers: baseline main indexes 9039 nodes/15067 edges, this branch before
the fix refuses to start the worker, and after it indexes 9039/15067.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-09-18 23:26:05 +02:00
Martin Vogel 959f4de50c perf: cut the allocation churn and repeated work the sanitizer found
Every item here was measured on a real corpus before and after, not
reasoned about. Go corpus, event lane, first run -> now: 245.5 M
allocations -> 134.8 M, 190 GB cumulative -> 29.3 GB, 158 GB never
written -> 9.2 GB, 132 M churn pairs -> 35.7 M, 93 GB of allocator byte
work -> 0.84 GB. Wall clock: Go 40.7 s -> 19.3 s, linux kernel 239.6 s ->
194.6 s, and all ten LSP languages faster with none slower.

Graph output is unchanged, proven rather than assumed: per-class multiset
digests over the full kernel index (8,563,471 nodes identical, 18/20 node
classes and 18/24 edge classes byte-identical, the rest differing only in
the classes a same-binary re-run also perturbs through worker merge order).

The changes, each with its own reason:

  arena      an exact arena that is appended to later restarts growth at
             8 KB instead of the 64 KB default — the cross-file pass adds a
             few resolved calls to a compacted result, so the default block
             was ~60 KB untouched per result, 0.5 GB of the Go worker peak.
             cbm_arena_init_lazy defers the first block for arenas that are
             usually never used at all.
  scratch    per-thread reuse instead of allocate-and-free per unit of
             work: result_compact (8 MB kept), the cross-file pass (4 MB),
             extract_defs' walk stack, and the sqlite index cells (a 1 MB
             arena replacing 12.1 M malloc/free pairs on Go). Walk frames
             moving into scratch is why pass_k8s now opens the keep window
             explicitly: without it every YAML file opened a 512 KB scratch
             on the main thread, +3 GB.
  sqlite     index cells are written in place — building the record and
             copying it into the cell was a second allocation per entry,
             16 M record buffers of pure churn on Go.
  registry   is_test_qn and common_prefix_len in one pass, per-bucket test
             flags computed once instead of per candidate comparison, and
             the reachability keys arena allocated lazily.
  minhash    the unique-trigram set is sized from the trigram count and
             skips repeats, and LSH queries reuse one seen-set per worker
             instead of allocating per query.
  parallel   lsp_surface rows and the k8s manifest prep now run across
             workers (graph writes stay sequential and ordered), the file
             node is hoisted out of register_and_link_def, and configlink
             prepares dependency imports once with a length prefilter.
  go_lsp     one sealed stdlib registry shared by all workers, published
             by CAS, instead of one per worker.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-09-18 03:31:22 +02:00
Martin Vogel d70b794179 fix(mem): the MSan guard in the spill store goes through sanitized.h
cppcheck cannot evaluate `#if __has_feature(memory_sanitizer)` even
behind `#if defined(__has_feature)` ("undefined function-like macro
invocation", the lint lane on d89ccabe), and foundation/sanitized.h
already exists for exactly that: it defines __has_feature away where the
preprocessor lacks it. The spill store's image annotation now includes
that header and tests the feature directly; the annotation itself is
unchanged.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-09-14 22:12:28 +02:00
Martin Vogel d89ccabef0 fix(mem): the parked result image is declared defined for MemorySanitizer
The MSan lane on 9724d903 stopped in the extraction suite's spill
round-trip test: uninitialized bytes in fwrite at offset 4087 of a
6,952-byte region, the parked result's arena block (the stack overflow
that followed is MemorySanitizer's own report path failing on the nested
bug). The park writes a result as an opaque image: the record header, a
struct copy, and the compacted arena block. Both carry padding bytes no
code ever wrote, inside structs and between objects, and MSan tracks that
mark through the compaction's memcpy, so zeroing the destination block
only moved the report (offset 6714 in the local MSan container). The
image is read back whole and never interpreted byte by byte; the padding
content is irrelevant on disk.

Under MSan the header and the block are declared defined right before
the write (__msan_unpoison, compiled in only when the build is
instrumented); every other build compiles the annotation to nothing. The
exact block is also zeroed once when it is created, so the gaps between
objects are deterministic in the spill file.

Verification: extraction, arena and parallel suites under ASan on the
fixed tree; the extraction suite in the local MSan container image
passes 348 / 0 with no sanitizer report (it reproduced the CI report exactly before the fix).

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-09-14 21:26:06 +02:00
Martin Vogel 9724d90304 fix(mem): dedicated SQLite heap only in the worker; guard the committed stat
The daemon leaked 180 KB per query. The Linux soak's quick leg went from
11 to 144 MB of RSS in ten minutes (13.1x, 628 MB/hr), mimalloc's
committed bytes tracking RSS almost byte for byte and the growth linear
in the query count, file descriptors flat -- where the same leg had been
flat at 15 MB in July.

The cause is the previous commit's SQLite heap: one mimalloc heap per
thread, created on first use. The daemon runs a thread per connection,
so every query ran on a fresh thread with a fresh heap, and every block
the shared connection keeps beyond the request -- page cache pages, the
statement cache, schema objects -- was allocated in that heap's pages.
When the thread exited, its heap was deleted and those pages were left
behind holding a few live blocks each: committed for as long as the
connection lives, a few pages per query, never reused by anyone.

The dedicated heap exists for one reason: the index worker's default
heap holds the graph (40M+ blocks), and SQLite churn on that heap paid a
page walk per allocation during the coverage publish (132 s vs 9.7 s on
the kernel). That process has long-lived threads and exits when the
index is done. So the dedicated heap is now a switch, off by default,
that the index worker turns on at its entry (main.c, --index-worker);
everywhere else SQLite allocates from the calling thread's default heap
exactly as before the previous commit. The switch exists in every build
and changes nothing where the allocator binds are compiled out.

Linux soak, quick leg, before -> after: RSS 11 -> 144 MB in ten minutes
(13.1x, 628 MB/hr) -> 10.6 -> 11.5 MB and flat from the first minute
over 564 queries with reindexes every two minutes; query-leak leg:
11.6 MB flat over 1,376 queries; committed heap 6.8 MB flat.

The same soak showed a second thing. From its second sample on, the
query-leak leg reported a committed heap of 2^64 - 121 MB: mimalloc's
committed statistic is a signed counter merged per thread at thread
exit, and a process whose long-lived thread commits what its
short-lived threads free reads it negative -- cast to size_t on the
way out. That figure is one of the two the budget takes the larger of.
cbm_mem_allocator_committed now reports 0 for a reading above
SIZE_MAX/2, so the charge falls back to the OS number instead of a
16 EB budget breach, and the diagnostics report the committed pair
through it (the peak never undercuts the current value and never
carries a wrapped one), the way they already did for RSS.

Detector: the soak's quick and query-leak legs (scripts/soak-legs.sh),
the release-gating sequence in every venue -- RED on the previous tree
(13.1x), GREEN on this one. A unit test cannot see this class: the test
build keeps SQLite on libc so the sanitizers see every block.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-09-14 19:17:40 +02:00
Martin Vogel 7b77fd4834 feat(mem): memory on the core, waste cut, results spilled under budget
Second increment on the memory core. The first landed the core, the linter
and phase attribution with zero production call sites; this one migrates the
memory that matters, proves where it goes, removes the waste the proof
exposed, and makes the memory budget a promise: over budget, extraction
results go to disk and the run completes at the floor instead of aborting.
Every number below is from a real index on this machine (M5 Pro, 48 GB) with
CBM_MEM_PHASES=1 and the release build.

WHAT THE PROOF SAID (Go corpus: 21,875 files, 345 MB of source)

The graph buffer, blamed for the peak since the 2026-07 research, is 3% of
the memory: 0.55 GB peak. The per-file extraction results are the memory:
14.8 GB written into their arenas, of which 3.4 GB is reachable. The rest
is temporaries -- cbm_node_text copies at 496 call sites, per-node QN
strings, abandoned generations of every growable array -- that an arena
cannot free and that the result kept alive until resolve ended, because
the result owned the arena. The arenas were charged 28.8 GB of capacity
for those 14.8 GB. On the kernel the same shape was 8.9 GB used by 89,731
results, and resolve grew it to 13.3 GB used / 17.2 GB capacity because
the C walk allocated its working set in the result arena and stored arena
pointers into the shared registry. After that, 7 GB of the kernel's
post-resolve memory had no class at all.

WHAT CHANGED

Attribution. The graph buffer (87 raw sites -> 0), arena blocks, the
tree-sitter slab allocator, the bound tree-sitter/SQLite allocators, the
semantic path (pass_semantic_edges, pass_similarity, minhash: 80 sites), the
hash table (Verstable, a class per table via CTX_TY: cbm_ht_create_in) and
CBM_DYN_ARRAY allocate through the core. Classes arena, ts_tree, hash_table
and dyn_array join the table. Two cross-boundary frees of buffer-owned
strings (pass_importance, pass_complexity) go through a new setter,
cbm_gbuf_node_set_properties_json. The linter baseline drops 4086 -> 3916.

Result compaction (internal/cbm/result_compact.c). At the end of a file's
extraction, everything reachable from its result is measured, copied into
one exact-size arena (strings interned by content within the file) and the
working arena is handed back. Later appends restart growth at the default
block (cbm_arena_init_exact, arena grow_size). The stale duplicate
internal/cbm/arena.h, which shared the foundation header's include guard
and won or lost by include order, is a shim now.

Record slimming. CBMCall 320 -> 72 bytes: the eight inline argument slots
(256 bytes, mostly empty) are allocated on first capture. CBMUsage 48 -> 40.

The per-file overlay contract (src/pipeline/pass_lsp_cross.c). One
lifecycle for every language's cross-file resolve: a scratch arena, a
per-file overlay registry chained to the sealed shared base, the walk
against the overlay, resolved calls appended into the result, scratch
destroyed. type_registry gains chain-aware iterators (every yielded index
belongs to it->reg) and copy-on-write refinement
(cbm_registry_func/type_for_update), and the C, C#, Go and Python walks
use them. This is what an ASan heap-use-after-free in the
lsp_resolution_probe suite demanded: the C walk stored per-call arena
objects into the shared registry, and a kernel run with a naive scratch
arena lost 1,045 edges to the resulting garbage reads. It also ends
unsynchronized writes into the shared registry from parallel resolve
workers.

The 7 GB with no class, found and cleaned. gbuf_index tables and keys were
2.5 GB live and 7.1 GB at peak: the 18 worker buffers built the five
secondary indexes they never query (cbm_gbuf_new_worker builds none), and
18M per-key index arrays started at 8 slots (cbm_da_push_min starts them
at 2); edge property JSON is 81% "{}" (interned). The semantic pass
allocated a 512-token stride per function up front (packed tokens with
offsets: cbm_sem_corpus_add_docs_batch). Edge dedup keys are 128-bit
hashes instead of sprintf'd strings (edge_key_map_t). Each worker keeps
one reusable working arena between files (cbm_work_arena_take/give:
rewind, not free) and every phase mark returns freed pages to the OS.

Budget metric. The over-budget check compares the charged footprint --
max(phys_footprint, mimalloc commit) on macOS -- not resident_size: after
extraction the kernel worker sat at 17.4 GB RSS with 5.5 GB charged, because
pages mimalloc has purged stay resident until the OS wants them, and
phys_footprint under-reports after MADV_FREE_REUSABLE cycles.

Spill / admission control (internal/cbm/result_spill.c, the extract gate
in pass_parallel.c). At 15/16 of the budget (or over it, or with
CBM_MEM_SPILL=1) the gate latches spill mode -- early, because the gate
sees the crossing per file pull and the workers' in-flight files carry the
charge past the line before the first sweep lands: every compacted result
is parked on disk as header + block in one append-only file per writer
under <cache>/spill/, the results already cached are swept out by every
worker, and registry build, def collection, surface rows, resolve and the
infra-route passes load a result only for the moment they read it
(relocated by base delta). The abort path stays closed while a sweep runs
or a cached result remains; it fires only at the floor -- graph +
registries + in-flight files. A result that owns sub-results stays in
memory; a retained parse tree is dropped with the parked result and
re-parsed on load. Every consumer of the result cache reads a slot through
cbm_pipeline_result_acquire()/release() (pipeline_internal.h): a parked
slot is NULL in the array, and the passes that indexed it directly lost
every __route__infra__ node and its INFRA_MAPS edges before the contract.
The other things the transient loads found: a def label that was the
result's own pointer (SIGBUS in the Go cross registry builder), two JVM
helpers that passed their input through, a dedup table whose keys borrowed
sr->value (the insert spun on freed memory), and a latch that was visible
before the store was open (a peer found nothing to park and failed the run
while its neighbour wrote 9 GB). The arena rewind reuse branch had
defeated the nblocks = CBM_ARENA_MAX_BLOCKS OOM seam the suites use; it
reuses only blocks that exist. The incremental and probe routes still
index the cache array themselves and therefore never spill
(ctx->spill_allowed stays false there; follow-up).

The semantic pass under a budget (src/pipeline/pass_semantic_edges.c). The
pass held every function's tokens, the corpus's per-document token ids and
the vectors being stored at once: 4.9 GB on the kernel, on top of the
floor, and nothing sized it to the headroom. Now, when the charged
footprint plus that transient would cross the budget, phases 2-4 run per
batch of functions: tokenize -> count into the corpus -> free; finalize
once; tokenize again -> vectorize -> store -> free, with pages returned to
the OS after every batch. Tokenization is deterministic, so both passes
see the same tokens and the graph does not change; the run pays with a
second tokenize. CBM_SEM_BATCH=<n> forces the batch size (the
batched-equals-unbatched test). The corpus frees its per-document token
ids at the end of finalize (the co-occurrence pass is their only reader),
and mem.semantic.step lines (CBM_MEM_PHASES=1) put the class's live/peak
bytes and the charged footprint at every sub-phase.

The semantic corpus and tokenizer allocate through the core (54 raw sites
-> 0). This is not attribution only: cbm_sem_tokenize handed out libc
strdup blocks that pass_semantic_edges freed with cbm_free -- mi_free in
the production build -- since the pass moved onto the core. No libc-backed
test build can see a cross-allocator free, so the core now refuses a block
it does not own (mi_is_in_heap_region) when CBM_MEM_PHASES=1 is set,
fatally: every proof run in this PR ran with it armed, and the Go and
kernel indexes report zero foreign blocks.

Accounting without contention. The core charged every allocation and free
to shared per-class atomics; with 18 workers on 18 cores every block bounced
the same three cache lines, and the bench against the shipped v0.10.8 showed
it as CPU time rising faster than wall time for the same graph: Kotlin 38 ->
121 s CPU (+58% wall), TypeScript 210 -> 596 s (+67%), Go 177 -> 312 s. The
deltas are thread-local now and reach the shared counters once per 256 KB or
512 blocks of change per thread, when a parallel-for worker ends, and
whenever a reader looks (its own thread first). Phase marks read after the
join and are exact; a class peak is a diagnostic and can be low by threads x
256 KB. Result: Kotlin 10.9 s / 43 s CPU (the residual +22% wall against
v0.10.8 tracks its +34% nodes), TypeScript 33.9 s (v0.10.8: 33.4), Go 48.4 s
(v0.10.8: 52.6), Java 81.1 s (v0.10.8: 79.8); graphs unchanged.

SQLite on a heap of its own. The same bench put the kernel's write step at
132 s where v0.10.8 needed 9.7 s, all of it in one coverage publish
statement. A stack sample named the cost: mimalloc's free-page search and
its periodic heap collect, walking the graph's pages -- millions of them
once the graph lives on the core -- for every statement-journal chunk
SQLite allocates. SQLite now allocates from a mimalloc heap of its own per
thread (mi_heap_new; mi_free is heap-agnostic, so nothing else changes),
and the periodic collect is set to the maximum mimalloc allows since the
explicit release at phase marks and spill sweeps is where memory goes
back. Coverage publish 132 -> 9.4 s; profiled kernel 376 -> 248 s
(v0.10.8: 296 s).

Formatting without a process lock. On macOS every vsnprintf takes the
process locale under an unfair lock (localeconv_l inside __vfprintf);
eighteen workers formatting type names on the C# corpus collapsed into
it -- the 10 MB JIT test files took 68 s each instead of under a second
-- and cbm_arena_sprintf paid it twice per string (size pass, write
pass). Each thread now formats with a C locale object of its own
(vsnprintf_l), and the common short string is formatted once into a
stack buffer.

Per-file budgets that cover the whole file. The 5 s per-file budget
covered the parse only; the LSP walks after it had none. C# JIT stress
files (a 23 MB single expression among them) parsed inside the budget --
the old wall-clock parse timeout used to drop them, which hid the rest --
and then held a worker for 346 s each in the usage walk (tree-sitter's
ts_node_parent descends from the root: quadratic on a deep tree), and the
cross-file resolve ran on the same trees. Three rules now, one site for
every language (cbm_extract_file_ex, honoured by cbm_pxc_dispatch_file): a
parse that used more than half the budget disqualifies the file from the
LSP walks (lsp_skipped); the unified walk checks its thread CPU time every
1024 nodes against six budgets and stops there, keeping what it found
(walk_truncated, implies lsp_skipped); a file whose parse plus walk spent
the budget is skipped by the LSP walks as well. Each decision is logged
with the path (extract.lsp.skipped, extract.walk.truncated). C# extraction
355 -> 39 s; the whole index 463 -> 128 s wall and 27 -> 11 GB peak
against v0.10.8 with 299 more nodes. The walk cuts exactly the five JIT
stress files (hugeexpr1, hugeSimpleExpr1, HugeArray1, HugeField1/2); the
15 MB System.Runtime.Intrinsics reference file walks to the end and, with
four generic-nesting JIT regression tests, only skips the LSP walk under
the parse-plus-walk rule.

The crash behind the C# bench. dotnet/runtime is 42,555 files, and its
resolve phase died with SIGBUS on every run of this branch. A libc-backed
ASan build of the server named it: heap-use-after-free in c_adl_resolve,
reading a type name that another worker had allocated in its per-file
scratch arena and freed at the end of its file. The C++ class walk
refines a method's return type when the declaration is more specific
than the pre-registered one (NAMED -> pointer/reference/template), and
it did that by casting the chained lookup result to non-const and
writing a scratch-arena signature into it -- into the sealed shared base
when the entry lived there. The overlay contract had exactly one
bypass, and it was this cast; every lookup returns const, so a grep for
the cast is the audit. The refinement now goes through
cbm_registry_func_for_update: copy-on-write into the overlay, the base
untouched. Test: clsp_method_return_refinement_is_copy_on_write (a
sealed base with a NAMED method return, an in-class pointer declaration;
the base signature pointer is unchanged and the overlay copy carries the
refinement plus a marker only a copy has) -- RED on the cast, GREEN on
the accessor. The fixed ASan build then indexed the whole
corpus clean: 42,555 files, 1,224,981 nodes, 5,794,320 edges, worker
exit 0, no report.

Production build backs the core with mimalloc explicitly (never the global
override, which macOS's two-level namespace forbids); the test build keeps
libc so the sanitizers see every block -- the same split the tree-sitter and
SQLite bindings already use.

PROOF

  Go     peak RSS 16.7 -> 4.0-5.2 GB; result arenas 14,804/28,817 -> 856/856
         MB; wall 55 -> 52-60 s; graph inside the pre-existing run-to-run
         jitter (gRPC Route nodes: two identical runs differ by 21).
  Go     CBM_MEM_SPILL=1: peak RSS 1.7-2.3 GB, 21,875 results parked (950 MB
         on disk), graph identical: 39 infra routes, 158 INFRA_MAPS, 21,875
         surface rows, nodes/edges inside the Route jitter.
  Go     CBM_MEM_BUDGET_MB=1200: spill latched at 1,204 MB, the first sweep
         took the charge to 768 MB, the run completed at 1.99 GB peak RSS.
  Kernel DEFAULT budget (24,576 MB): aborted before, completes now without
         spilling. Extraction 35.2 -> 16.5 GB RSS (5.6 GB footprint, 15.3 GB
         committed); results 8.9 GB = capacity; resolve tracked 24.6 -> 19.1
         GB; peak RSS 27.3 -> 20.1 GB. Nodes identical (8,529,729); edges
         inside the +-400 run-to-run band the kernel shows between identical
         runs.
  Kernel CBM_MEM_BUDGET_MB=16000: aborted before, completes now through the
         spill path. Latched at 17.2 GB charged; 89,731 results parked (9.3
         GB on disk, 370,188 loads); extraction end 6.2 GB RSS / 5.1 GB
         committed; resolve 8.6 GB RSS / 11.5 GB committed; peak RSS 16.6
         GB (the semantic pass plateau, 13.3 GB RSS, is graph-derived and
         outside what spill can move). Nodes identical; edges 15,773,598.
  Kernel CBM_MEM_BUDGET_MB=15000, before the semantic batching: completes,
         every phase mark under 15,000 MB by every metric, but the process
         RSS high-water mark reached 16.9 GB inside the semantic pass (class
         peak 4.9 GB against 0.8 GB live).
  Kernel CBM_MEM_BUDGET_MB=15000, with it: completes; the charged footprint
         stays under 15,000 MB through the whole semantic pass (14,018 ->
         14,381 MB across its sub-phases, 6 batches of 125,644 functions);
         every phase mark is under budget by every metric (max footprint
         12.6 GB, max commit 11.5 GB); the process RSS high-water mark is
         15.65 GB and is set in extraction, at the moment the gate latches
         (15,002 MB charged) and before the first sweep lands -- a 4%
         overshoot of in-flight work, no longer the semantic pass. Nodes
         identical, edges 15,772,913, zero foreign blocks.
  Kernel CBM_MEM_BUDGET_MB=15000, with the anticipatory latch: spill mode
         enters at 14,062 MB (near_budget); the charged footprint never
         exceeds 11.8 GB at any mark; the process RSS high-water mark is
         15.1 GB in extraction and 15.3 GB during the final write-out --
         1-2% over, the in-flight window and the dump transient. Nodes
         identical, edges 15,772,543, zero foreign blocks.
  Go     CBM_MEM_BUDGET_MB=1200 with it: latched at 1,131 MB, extraction
         peak 998 MB RSS, graph identical to the baseline.

BENCH AGAINST THE SHIPPED v0.10.8 (same machine, same driver, 2026-09-14)

  corpus      wall s          peak RSS GB     nodes / edges / CALLS
  perl        5.7 -> 5.3      0.09 -> 0.05    -3 / -130 / -3
  php         6.3 -> 5.8      0.66 -> 0.20    +21.9% / +1.4% / +23.8%
  rust        6.8 -> 6.3      1.15 -> 0.67    +1 / +56 / -369
  c           5.9 -> 5.7      0.16 -> 0.19    identical
  kotlin      8.9 -> 9.4      1.55 -> 0.96    +33.5% / -22.8% / +12.1%
  django      10.9 -> 10.2    3.13 -> 1.12    +6 / -14.5% / -28.4%
  go          52.6 -> 42.9    17.2 -> 3.9     +14.2% / -10.3% / +2.2%
  java        79.8 -> 73.8    27.6 -> 11.0    -121 / -1.9% / -0.6%
  csharp      463.3 -> 128.1  27.0 -> 11.2    +299 / -0.9% / -0.6%
  typescript  33.4 -> 33.6    12.0 -> 2.5     -186 / -3.5% / +130
  kernel      294.8 -> 233.3  27.5 -> 25.1    -171 / -1.3% / -0.1%

No corpus is slower beyond noise; CPU time follows wall (kernel 2,442 ->
1,403 s). Every graph delta against v0.10.8 is main between the release and
the branch base (339b3f40), not this PR: a build of the base commit run
through the same driver reproduces the final graphs exactly on django and
kotlin, within the pre-existing gRPC Route jitter on go (30 nodes, 27
edges, CALLS identical), and on C# with the same node count. This PR's
own C# difference against the base is 5 CALLS (the five truncated JIT
stress files) plus one ijwhost swap and 563 edges of 5.79 M; the base
takes 448 s / 29.7 GB for that corpus.

Diagnostics that made this measurable stay in: the extract.arenas and
extract.census lines (behind CBM_MEM_PHASES=1), the CBM_MEM_RELEASE=1
release-to-OS probe, and peak_charged_mb on every mem.phase line -- the
high-water mark of the budget metric itself (cbm_mem_peak_charged), next
to the RSS peak that counts pages already purged but not yet reclaimed.

Tests: extract_compact_* and extract_spill_round_trip_keeps_every_field
(extraction), parallel_spill_mode_builds_the_same_graph (parallel: every
file parked, node count and every edge type equal to the in-memory run),
pipeline_semantic_batched_matches_unbatched (pipeline: one repo indexed
unbatched and with CBM_SEM_BATCH=5 -- byte-identical node vectors, token
vectors and SEMANTICALLY_RELATED edges),
extract_lsp_skipped_when_parse_used_ its_budget_share and
extract_walk_truncated_at_its_cpu_budget (extraction: the seams mark a
file / stop the walk, the LSP walk and the dispatcher skip it, the defs
stay), registry_overlay_chain_iterates_ and_copies_on_write (c_lsp), arena
exact-init/growth, mem charged/footprint/ pressure.

Not in this increment: run-to-run edge jitter (+-400 on the kernel, Route
class on Go) predates this work and has its own item; the incremental route
spilling; a tighter result encoding (def ids instead of QN strings in
resolved calls) is the next lever on the 9 GB the kernel's results still
take.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-09-14 06:11:03 +02:00
Martin Vogel 527016258e fix(extract): budget tree-sitter parses by thread CPU time, not wall clock
The per-file tree-sitter parse was bounded by a 5s WALL-CLOCK budget. Under CI
contention a worker thread can be descheduled long enough for 5s of wall time
to elapse against near-zero CPU, so a small, perfectly parseable file was
abandoned with a false "parse timeout" and yielded ZERO defs. When that file
defined a symbol other files resolve against (e.g. celery.app.Celery.send_task
backing an ASYNC_CALLS edge, or a userpb gRPC client backing GRPC_CALLS), the
cross-file service edge then failed to classify -- surfacing as the flaky
convergence_probe / edge_types_probe service-edge assertions (count 0), a
verdict decided by wall-clock scheduling rather than the input (violates O9,
"no timeout decides a test").

Gate the parse on per-thread CPU time instead, with a generous wall ceiling as
a backstop:
- New portable helper cbm_thread_cpu_time_ns() (compat.c/.h): POSIX/macOS
  clock_gettime(CLOCK_THREAD_CPUTIME_ID), Windows GetThreadTimes (kernel+user);
  returns 0 on clock failure, degrading safely to the wall ceiling. It is NOT
  cbm_clock_gettime, which is wall on every platform (QueryPerformanceCounter
  on Windows).
- The progress callback's payload becomes {cpu_deadline_ns, wall_ceiling_ns};
  it trips when this thread's CPU time passes the budget OR wall time passes
  t0 + budget * 12 (~60s for the 5s budget). Descheduling burns wall, not CPU,
  so a starved-but-parseable file survives; a CPU-spinning parse still trips
  the CPU deadline; a genuinely stuck parse still trips the wall ceiling
  (bounded, no hang).

RED-repro (deterministic, CBM_ENABLE_TEST_SEAMS only): a thread-local wall-clock
offset armed by CBM_TEST_WALL_STALL_ON emulates a descheduled worker (wall
reads budget + 1s ahead while CPU is untouched -- above the old 1x wall budget,
below the ceiling). async_calls_celery_wall_stall_seam arms it for the Celery
DEFINITION file and asserts ASYNC_CALLS >= 1. With the fix the parse completes
and the edge resolves; with the budget reverted to wall-only + a 1x ceiling the
def file is dropped and the edge count is 0.

Verified (macOS, sanitized): edge_types_probe + convergence_probe 107 passed /
0 failed with the fix; 58 passed / 1 failed on revert (only the seam test, at
the ASYNC_CALLS assertion); 107 restored. Regression smoke (extraction
parse_coverage pipeline lang_contract grammar_regression) 698 passed / 0
failed. lint-ci clean (cppcheck + clang-format Homebrew LLVM + NOLINT); the
Windows GetThreadTimes path is compile-checked by CI (not runnable on macOS).

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-09-11 21:40:54 +02:00
Martin Vogel 5b8217d43f Merge pull request #1759 from Jumaga2015/fix/1692-csharp-multi-attribute-decorates
fix(extract): scan every attribute_list sibling, not just the first (#1692)
2026-09-08 01:02:18 +02:00
Joshua RichterandClaude Opus 5 e5c17de621 fix(coverage): stop reporting a duplicate range and a line past EOF (#963)
scripts/setup-windows.ps1 has 326 lines and its parse-coverage report read
"113-113,113-113,245-327" — the same line named twice, and an end line that
does not exist. Two separate faults, both in cbm_error_regions_push.

Past-EOF end line. A tree-sitter node that ends at column 0 stopped right
after the previous line's newline, so it holds no text on the row it points
at. Adding 1 to that row named a line past the end of the file whenever the
region ran to EOF. The node here is start=(244,2) end=(326,0). Clamp the end
to the row above when the end column is 0 and the node spans more than one
row.

Duplicate range. Line 113 carries two separate ERROR nodes, at columns 25-29
and 31-32, and each pushed its own range. A line range says nothing new the
second time. Drop a range that exactly repeats the one already open. The
drop runs BEFORE the cap check, so a repeat is never miscounted as a range
the cap threw away.

Only an EXACT repeat is dropped, never a range that merely overlaps. Each
range is judged separately afterwards by cbm_region_is_recovered, which asks
whether definitions starting inside that range cover it. Two ranges holding
the same numbers always get the same verdict, so dropping the repeat changes
nothing. Two different ranges do not. Merging 3-3 into 2-3 hands the wider
range's covering definition to an error that definition does not explain, and
a real parse failure then vanishes from the report.

That is not hypothetical. An earlier version of this commit merged on overlap
and broke perl_malformed_source_remains_partial_issue1838, the test added
with the Perl grammar refresh in 17b5a432. The malformed fixture produces two
ERROR nodes, at lines 2-3 and 3-3. Merged, the 2-3 range looks fully covered
by before_error and is removed, so parse_incomplete comes back false on a file
that plainly does not parse. That test now pins this boundary.

The real file reports "113-113,245-326".

Two tests, both proved RED first with the exact expected text:
  coverage_repeated_error_line_reports_one_range_issue963  "2-2,2-2" != "2-2"
  coverage_range_never_ends_past_the_last_line_issue963    "1-5"    != "1-4"

Suites run on this change: parse_coverage 34, extraction 325, pipeline 264,
mcp 246, index_resilience 7 — all passing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
2026-09-06 23:57:52 -04:00
Martin Vogel fbe9eae11c Merge branch 'main' into fix/1692-csharp-multi-attribute-decorates 2026-09-07 01:16:11 +02:00
Martin Vogel aa44c28ea5 Merge pull request #1941 from CaptainMittens/fix/parse-coverage-honest-ranges
fix(coverage): narrow parse-error ranges, and stop the report hiding what it dropped (#963)
2026-09-07 01:15:37 +02:00
Joshua RichterandClaude Opus 5 b04f54506a perf(coverage): find a line by table lookup, not by rescanning the file (#963)
cbm_refine_regions_with_pp_lines asks cbm_line_is_toplevel_macro_call about
every preprocessor-parsed line in a region. That check walked the source from
byte 0 to find the line, so the refinement cost bytes times lines — on exactly
the whole-file-ERROR C shape the refinement exists to narrow.

Build one line-offset table per file, and hand the byte span to the macro
check. The line-based form stays for the two callers that ask about a whole
region, where a single walk is cheaper than a table. A file with no table (the
allocation failed) still gets the old walk, so the answer never changes.

Measured on one C file whose file-scope brace is split by an #ifndef, padded to
5k / 10k / 20k / 40k lines. Release build, one machine, seconds:

  lines     main   branch   this commit
   5000     0.98     1.14     0.98
  10000     3.30     4.02     3.41
  20000    11.64    14.77    12.56
  40000    44.04    58.58    46.90

The branch ran 33% over main at 40k lines and now runs 6.5% over. The reported
ranges do not change: 8-8, where main reports the whole file as 5-40000.

A profile of the 40k run no longer reaches the macro check. What is left is
cbm_line_contains and cbm_span_contains_callable_def, which main carries too
and this branch does not touch. The fixture holds 40k definitions, which makes
those two cost more here than on a smaller file.

parse_coverage, extraction, pipeline, mcp and index_resilience: 964 passed,
4 skipped.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
2026-09-05 16:24:25 -04:00
Joshua RichterandClaude Opus 5 219b348263 test(coverage): pin the truncation marker, the partial ceiling and the grammar limit (#963)
Phase 5. Four test groups, each checked RED before it was kept.

- The Studio Export range join puts ONE ",+<N>" marker at the end with the
  summed drop count. A marker left mid-string makes every reader stop there
  and silently lose the ranges after it. Reaching the join through the
  pipeline needs an export file with 256+ error regions across two <Class>
  elements, so it goes through a test seam, following the pattern already in
  this repo (CBM_COVERAGE_MARKER_TEST_API).
- check_index_coverage emits every range in front of a marker, never turns
  the marker's digits into a range, and reports "truncated" from BOTH caps —
  the producer's and its own 256 limit.
- test_index_resilience now has a ceiling beside its floor: exactly one of
  the two fixture files is flagged, the clean neighbour is absent, and the
  range does not cover the whole file.
- The three _Thread_local forms are pinned as measured. Only the array form
  fails today; the plan's Phase 0 also listed the pointer form, and that is
  wrong on the grammar shipped now.

Also fixes 13 clang-format violations the earlier commits on this branch left
in cbm.c, mcp.c and pass_definitions.c. `make -f Makefile.cbm lint-format`
would have failed CI. The changes are whitespace only — the two reflowed tool
descriptions concatenate byte-identically, so no output moved.

Full suite: 7735 passed, 28 failed, 7 skipped. The 28 are the pre-existing
cli install/uninstall failures, identical at clean HEAD.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
2026-09-05 16:02:17 -04:00
Joshua RichterandClaude Opus 5 a953509275 fix(coverage): stop the report hiding what it dropped, and name whole-file failures (#963)
Two silent failures in the parse-coverage report, both made visible by the
Phase 2 range refinement that came before this.

Two caps sat in series and both returned early without saying anything:

    CBM_MAX_ERROR_REGIONS = 64   internal/cbm/cbm.c
    COVERAGE_RANGE_MAX   = 128   src/mcp/mcp.c

Raising only the first would have moved the clip from 64 to 128, so both move
to 256. This was live behaviour, not a theoretical limit: after Phase 2 split
one whole-file range into many small ones, src/cli/cli.c and tests/test_cli.c
both reported exactly 64 ranges — the cap binding, dead-on, twice. Every
coverage figure measured before this change was a floor. With the cap at 256
the true numbers are cli.c 13.9% (not 9.8%) and test_cli.c 3.1%, and the
longest list in the repo is 85 ranges.

A raised cap is still a cap, so the report now says when it clipped:

- cbm_error_regions_t gained a `dropped` counter, and cbm_collect_error_regions
  walks to the end instead of stopping at the cap, so the count is exact rather
  than a lower bound. That costs little — the walk never descends into an ERROR
  subtree.
- cbm_error_ranges_str appends ",+<N>" when N ranges were thrown away.
- coverage_add_ranges reads that marker and sets "truncated": true, and also
  sets it when its own limit stops the loop. Before this the marker was
  invisible: the parser stopped at the '+' with no error and no leftover, so a
  clipped list arrived looking complete.
- objectscript_export_append_error_ranges strips markers off both operands
  before joining two Studio Export parts and adds one back at the end. A marker
  left mid-string would make every reader stop there and silently lose every
  range after it.

"Look at lines 1 to 13047" of a 13046-line file tells a reader nothing. Those
files now carry their own kind rather than being described as partially
covered.

New `parse_unusable` field in CBMFileResult, set when one range covers 80% or
more of the file. Its customers are non-C languages: the Phase 2 refinement
that narrows a whole-file range using the preprocessed parse only runs for C,
C++ and CUDA, so a Python, Java, Ruby or TypeScript file whose root node is
ERROR still reports 1-N. Verified against real files in all four.

The kind is `parse_unusable`, not `parse_failed`. index_coverage.kind already
means one of two things — indexed-but-partial, or a skip phase saying the file
was never indexed at all — and `parse_failed` reads as the second when it is
the first. The store.c schema comment, which is the only written record of this
vocabulary, now describes all three classes and says why.

Two places would have mislabelled the new kind as "skipped", which is exactly
that confusion: coverage_status fell through to its catch-all pass, and
add_coverage_report fell into its else branch. A reader who finds a file under
"skipped" believes it is absent from the graph, when it was indexed. Both now
have explicit branches. index_status gained parse_unusable_count so a CI gate
can read it without parsing anything else, get_code_snippet says "read the
source directly" instead of naming useless ranges, and the three tool
descriptions that listed two coverage kinds now list three.

Seven added. The cap test moved from 64 to 256; a new test asserts the marker
carries a real drop count and that nothing follows it; an inverse test asserts
an under-cap file carries no marker at all. For the new kind: a Python file
whose root is ERROR is unusable, a file with a local parse failure stays
partial, a clean file is neither, and — the one that matters most — the
that last one ever flips, the Phase 2 refinement has stopped working.

Full suite: 7732 passed, 28 failed, 7 skipped. The 28 are pre-existing
agent-client install/uninstall failures in the cli suite, identical in count
and identity at clean HEAD.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
2026-09-05 16:01:05 -04:00
Joshua RichterandClaude Opus 5 cb167b9dcc fix(coverage): narrow parse-error ranges with the preprocessed parse (#963)
src/cli/cli.c reported an error range of 1-13047 — the whole file. The file
indexed fine; the report was wrong. Three #ifndef _WIN32 blocks split a brace
(two `if` headers, one closing brace), so the raw tree-sitter parse cannot
resync at file scope, the root node becomes ERROR, and cbm.c takes its
whole-file branch.

The pipeline already parses these files a second time after preprocessing, and
that parse is clean. The report just never consulted it.

Build one byte per original line from the preprocessed pass, then cut each raw
error range down to the runs of lines the second parse could not vouch for.

Three rules, all found by running it and all load-bearing:

- An expanded line only vouches for its original line when it HAS TEXT. The
  preprocessor emits a blank line where it dropped a branch; treating that
  blank as proof suppressed every C range in the suite.
- Preprocessor directive lines (with backslash continuations) never count as
  missing code — the preprocessor consumes them, so the second parse can never
  vouch for one. Without this every #include block reported as a miss. Known
  cost: a #define the raw parse really dropped no longer shows up on its own.
- A TOP-LEVEL macro invocation line never counts as vouched-for even when the
  expanded line parses clean. The macro can expand to a whole definition that
  the recovery walker deliberately refuses to adopt (#949), so a clean second
  parse there proves nothing. An in-body invocation is the benign #1071 case
  and is left to the existing macro subtraction.

The order of the three coverage steps is now settled by where each one's
evidence lives:

  recovery subtraction  -> before the refinement; its evidence is a whole
                           definition that STARTS inside the range, so it must
                           be asked while the range still matches the construct
  the refinement        -> middle
  #1071 macro rule      -> after the refinement; its evidence is per-line, so a
                           narrow range points at the call itself

Measured on this repo: src/cli/cli.c goes from one whole-file range to 64
ranges over ~9.8% of the file, tests/test_cli.c from 48.6% to ~2.9%,
src/cli/activation_transaction.c from 38% to 7.5%. What survives is honest —
the biggest remaining ranges in cli.c are genuinely discarded #ifdef _WIN32
and #ifdef CBM_CLI_ENABLE_TEST_API blocks, absent from the graph on this
platform.

Both percentages above are floors, not measurements: cli.c and test_cli.c now
land on exactly 64 ranges, which is CBM_MAX_ERROR_REGIONS. That cap drops
regions with no signal, and a follow-up raises it and adds a truncation marker.

Five tests, all red before the change: the range narrows to the dropped
branch; lines the preprocessor explained are excluded; a range never starts or
ends on a directive; real garbage beside a split brace stays flagged; a clean
file stays unflagged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
2026-09-05 15:58:59 -04:00
Martin Vogel 4f82a1e56f Merge branch 'main' into fix/svelte-html-astro-embedded-structure 2026-09-05 19:15:01 +02:00
Martin Vogel 683d4bc312 Merge branch 'main' into fix/1692-csharp-multi-attribute-decorates
Brings the branch onto current main (fe85a6b2) as promised on the PR: the
only conflict was internal/cbm/helpers.h, where both sides added
declarations after cbm_find_child_by_kind; both blocks are kept (the PR's
cbm_find_children_by_kind prototype, then main's Lisp-family gates).

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-09-05 19:05:08 +02:00
Martin Vogel ffb5ebe9e8 Merge pull request #1721 from henry-hz/upstream-pr/elixir-call-arguments
fix(elixir): populate first_string_arg for Elixir calls
2026-09-04 15:38:50 +02:00
Martin Vogel dc703dad4a Merge branch 'main' into upstream-pr/elixir-call-arguments 2026-09-04 09:49:54 +02:00
Martin Vogel 2efb7a6323 Merge branch 'main' into fix/ruby-superclass-base-classes 2026-09-04 09:49:50 +02:00
Brian McNaboe 24f54c0eb6 fix(extract): extract structure from Svelte/HTML/Astro inline scripts
The Vue change (#1852) built the shared embedded-script seam — the
included-ranges parse over the original buffer and the lang=/src=
resolver — but gated structural extraction on the host being Vue. A
Svelte component, an HTML page or an Astro file therefore still yielded
only File + Module nodes and IMPORTS edges from its <script> blocks: no
Function definitions, no CALLS, and a false dead-code verdict for
anything used only from a component.

Widen the seam to every host that declares an embedded spec:

- Structural extraction (definitions + calls) runs for every embedded
  block; the per-host boolean is gone.
- The Vue attribute resolver becomes script_embedded_language() and
  runs for all four markup hosts. It starts from the spec row's language
  instead of hard-coding JavaScript, and learns HTML's type= attribute:
  absent or empty, module, text/javascript and application/javascript
  pass; application/json, importmap, text/x-template and the like bail,
  so data blocks cannot leak garbage definitions into the graph. src=
  and unsupported lang= values bail as before. CFML's <cfscript> carries
  no attributes and keeps its spec-fixed language.
- Astro's frontmatter fence and <script> bodies default to TypeScript:
  Astro type-checks both as TS with no lang= attribute to say so, and
  the fence cannot carry one. The TypeScript grammar parses untyped
  JavaScript unchanged.

Tests: the #1852 control that pinned Svelte/HTML/Astro at zero functions
and zero calls now asserts the real symbols; stay-at-zero controls cover
type="application/json" and <script src> in every host; both Svelte
blocks (context="module", and Svelte 5's bare module) contribute in
host-file coordinates; HTML's accepted type= forms and Astro's TS
frontmatter are covered; an end-to-end CALLS edge inside a .svelte file
is asserted. The Svelte and Astro label goldens gain the Variable their
fixtures always declared.

Verified on a 213-component Vue corpus: node counts per label and per
host language, edge counts and trace_path output are identical before
and after.

Fixes #1807

Signed-off-by: Brian McNaboe <bmcnaboe@gmail.com>
2026-09-03 16:24:29 -04:00
Martin Vogel 380b1c5af1 Merge branch 'main' into feat/cshtml-razor-pages 2026-09-03 21:58:04 +02:00
Martin Vogel 5802ccdd77 Merge pull request #1896 from CaptainMittens/fix/swift-call-arguments
fix(extraction): reach Swift call arguments through call_suffix
2026-09-03 15:12:27 +02:00
Martin Vogel 1770d68a2c feat(extract): discover Razor Pages / MVC views (.cshtml) and route their @page
Follow-up to #1824, which mapped Blazor `.razor` to the C# host and extracted
`@page` routes from it. `.cshtml` is the same Razor syntax with the same C#
host and was equally unmapped: an ASP.NET Core MVC or Razor Pages application
produced no nodes at all for its entire view tree.

`@page` matters more here than it does for components. On a Blazor component
the directive is an optional annotation; in Razor Pages it is the directive
that *defines* a page, so the routable surface of a Razor Pages app lives
exclusively in this file type.

That is why the extension mapping alone is not enough. `cbm_path_is_razor()`
hardcoded a single `.razor` suffix, so a `.cshtml` file would have been
discovered and parsed but skipped by the directive scan -- present in the
graph, with none of its routes. Widened to a suffix table covering both.

Scope is deliberately the Razor family and nothing else:

  - `.vbhtml` is excluded because there is no VB.NET language in this repo,
    so there is no host to map it to.
  - `.aspx` / `.ascx` are excluded because Web Forms is a different templating
    syntax (`<%@ %>`, `runat="server"`) with no `@page` directive, so neither
    the C# error recovery nor the directive scan applies.

Best-effort on the same terms as `.razor`: the C# grammar recovers the `@{ }`
and `@functions` blocks, and the surrounding markup lands in ERROR regions
reported through parse_partial.

Tests: the extension mapping; a Razor Page asserting the extracted route path
and method; and a negative case for a layout with no `@page`, since most
`.cshtml` files are layouts and partials and a spurious Route node per view
would be worse than the missing ones this fixes.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-09-03 11:42:52 +02:00
Martin Vogel 6cca768816 Merge pull request #1773 from rarepops/fix/dockerfile-trailing-blank-eof-1746
fix(extraction): treat trailing blanks at EOF as an absent newline, not a miss (#1746)
2026-09-03 11:26:47 +02:00
Martin Vogel cc3e2638da Merge pull request #1912 from DeusData/feat/python-bare-local-binding
fix(python): suppress bare calls shadowed by an enclosing parameter
2026-09-03 09:39:46 +02:00
Martin Vogel db2f104a1f Merge pull request #2013 from shafty023/fix/1997-extract-scratch-arena
fix(extract): AST traversal stacks no longer outlive the file they walk
2026-09-03 08:14:16 +02:00
Joshua Richter c4e902849f fix(extraction): reach Swift call arguments through call_suffix
handle_calls reads a call's arguments through one tree-sitter field
lookup, ts_node_child_by_field_name(node, "arguments"). The vendored
Swift grammar declares no "arguments" field at all -- its own
ts_field_names[] table has zero occurrences, against one in Go and two
in TypeScript. Swift models a call as a target expression plus a
call_suffix, and the arguments hang off the suffix as value_arguments.

So args was null for every Swift call ever parsed, first_string_arg was
never populated, and no Swift HTTP call could raise an HTTP_CALLS edge
or a Route node. Alamofire, Moya and URLSession have been in the
service-pattern table the whole time and match the callee text
correctly; the URL simply never arrived.

Three changes, all in extract_calls.c:

- swift_call_args() reaches the argument list through call_suffix, in
  the same shape as the existing objectscript_call_args() fallback and
  used from the same place. A trailing closure has no value_arguments,
  so it returns a null node and that call behaves as before.
- extract_url_or_topic_arg() unwraps Swift's per-argument
  value_argument node, stepping past a leading value_argument_label.
  Without it, dataTask(with: "/api/v1/widgets") yields the label "with"
  rather than the path. PHP and C# already had the same unwrap for
  their own "argument" node.
- is_string_like() gains line_string_literal, which is what Swift calls
  an ordinary "..." literal. The list already held raw_string_literal,
  so only Swift's common case was missing.

This is the layer underneath #1892 rather than the whole of it. A
literal URL argument now arrives; a literal nested inside a
constructor, as in URLSession.shared.data(from: URL(string: "...")!),
still does not, because extract_positional_url reads a literal, a
template string, a concatenation or a named constant and that shape is
none of them.

Reproduce-first: all three tests fail without the fix, two on a null
first_string_arg and one on HTTP_CALLS being 0.

Fixes #1892

Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
2026-09-02 20:26:17 -04:00
Martin Vogel 192f29609a Merge pull request #2006 from ilyabrykau-orca/fix/go-field-ref-member-signal
fix(pipeline): key the Go Field guard on the recorded selector shape, not reference text
2026-09-02 23:59:55 +02:00
Martin Vogel 01bf5f23e3 Merge pull request #2009 from CaptainMittens/fix/route-handler-last-argument
fix(routes): take a route's handler from the last argument, not the first
2026-09-02 23:53:32 +02:00
Martin Vogel e6a34fcea0 Merge pull request #1824 from The1nk/feat/blazor-razor-routes
feat(extract): discover Blazor .razor components and emit Route nodes for @page
2026-09-02 23:48:14 +02:00
Martin Vogel 620613ede6 Merge pull request #1986 from CaptainMittens/fix/swift-scanner-try-bang-shift
fix(grammar): stop the Swift try-bang entry shifting past its own type
2026-09-02 22:51:46 +02:00
Rares Popa 572725e61a fix(extraction): treat trailing blanks at EOF as an absent newline, not a miss
A file whose final line lacks its newline leaves the grammar's mandatory
terminator MISSING at EOF. #1610 suppressed that phantom parse_partial on the
grounds that the node is ZERO-WIDTH: the parser consumed no source for it, so
by construction nothing was dropped.

That suppression tested `end == source_len` exactly. Trailing blanks are extras
owned by no node, so `ENTRYPOINT ["a"] ` + EOF parks the zero-width terminator
one byte short of source_len and the check missed it. This is why neither a
trailing blank nor an absent final newline flagged on its own -- only the pair
did, exactly as the reporter's byte-exact control matrix showed.

Treat "at EOF" as EOF modulo a trailing blank run. Every blank except newline
qualifies, deliberately not a hand-picked subset: a form feed is no more
content than a space, and #1610 exists precisely because whether a file got
flagged used to hinge on such incidentals. Newline stays excluded because a
terminated final line produces no MISSING terminator at all.

The rule remains ZERO-WIDTH ONLY. A width-bearing MISSING/ERROR at EOF is a
genuine loss and is still flagged (a Makefile's unterminated final recipe
really does vanish), as is any failure earlier in the file; both are pinned by
guard tests alongside the existing #1610 guards.

Verified end to end: the issue's own `cli index_repository` reproduction goes
from parse_partial_count 1 to 0 with node count unchanged, so the phantom flag
is gone without anything dropping out of the graph.

Fixes #1746

Signed-off-by: Rares Popa <2606875+rarepops@users.noreply.github.com>
2026-09-02 18:42:40 +02:00
Daniel Ochoa 40f2722d61 fix(extract): AST traversal stacks no longer outlive the file they walk
Fixes #2010. Refs #1997.

Indexing a 14k-file TypeScript repository holds 13.3 GB resident at
post_extract on Linux aarch64, and about 3.1 GB of it is AST traversal
scratch that no CBMFileResult points at.

ts_nstack_init cut its stacks from ctx->arena, which is result->arena. In
the parallel path that arena is stored into ec->result_cache[file_idx]
(pass_parallel.c:963) and freed only after parallel_resolve and the infra
passes (pipeline.c:1324), so a structure written for a one-file lifetime is
in fact held for the whole index, once per file, concurrently. The header
claimed otherwise and was wrong: "freed when the arena is destroyed at end
of file extraction".

The largest instance is channels. CHAN_STACK_CAP is 4096 and sizeof(TSNode)
is 32, so scan_string_consts_js (extract_channels.c:104) and
extract_channels_js (:376) take 128 KB each, per JavaScript or TypeScript
file. Across 12,673 such files that is 3,168 MB, and it produced 23 Channel
nodes out of 164,684.

CBMExtractCtx gains a scratch arena, created and destroyed by the
cbm_extract_file_ex call that builds the context. Nothing else changes about
where anything lives: ctx->arena still owns every string a CBMFileResult
points at, and ctx->scratch owns only traversal stacks, which nothing points
at.

Nothing a CBMFileResult points at moved. A context built without a scratch,
and a file whose scratch arena fails to allocate, both fall back to
ctx->arena, which is exactly the behaviour that shipped before this change,
so no path can be handed a NULL arena and silently lose nodes.

The signature change is what makes the mistake unrepresentable.
ts_nstack_init takes the context rather than an arena, so handing it
ctx->arena is an incompatible pointer type; push and push_children read the
arena the stack recorded, so a stale argument is an arity error. Both fail
under -Werror rather than silently retaining memory.

The 55 init sites are NOT one uniform sed. 29 pass ctx->arena and 26 pass a
local alias, and that alias is shared with result allocations: in
parse_zig_imports (extract_imports.c:1093) the same `a` feeds cbm_node_text,
strip_quotes, path_last and cbm_imports_push (extract_imports.c:1101, 1106,
1108, 1109). Rebinding it would have moved import names and module paths
into memory that dies with the call while the result kept pointing at them.
Each site was edited on its own and no alias was rebound. Re-derive the set
with:

  grep -rn 'ts_nstack_init' internal/cbm/extract_*.c
  grep -rn 'ts_nstack_push' internal/cbm/extract_*.c

parse_lisp_imports used its alias only for the stack, so the alias goes too.
push_nested_class_nodes and push_class_body_children took an arena purely to
reach the stack, so they take the context now; the one call site passes ctx.

Two measurements, from two different builds, and they are not the same
number.

A macOS census build, which instruments every cbm_arena_alloc and sums the
blocks behind every live CBMFileResult arena, reports the retained per-file
arena set dropping from 11,226 MB mapped to 5,680 MB, and blocks from 48,045
to 34,186. Those are mapped bytes, not resident: that build logs
mem.allocator.bound_populations_only, so untouched block tails never commit.

The resident figures come from Linux aarch64, gcc 13.3, static build, where
the build logs mem.allocator.owned classes=all and mapped equals resident.
Five runs per side on the full 13,999-file tree: mem.collect post_extract
rss_mb 13293 -> 6752 (-49.2%), mem.phase peak_mb 14840 -> 8565 (-42.3%).
nodes 164684, parse_partial 280 and registry entries/defines/imports
787954/804999/100027 identical on all ten runs; edges 835869-835874 against
835867-835881, overlapping ranges, so run to run noise. Wall clock means
35651 ms against 30502 ms, but the ranges overlap and the host had other
load, so the only supportable claim is no regression. ObjectScript Studio
Export on a small fixture: nodes 18, edges 25, both sides.

Scratch block size is 512 KB, chosen from measurement. arena_grow fires on
one file in 12,000 at both 512 KB and 1 MB, and constantly at 256 KB, where
the two channel stacks alone are exactly 262144 bytes. 512 KB therefore
costs half the resident block per worker for the same growth behaviour, and
stays under MI_LARGE_MAX_OBJ_SIZE in the vendored mimalloc, which is
MI_LARGE_PAGE_SIZE/8, exactly 524288 bytes (types.h:426, with
MI_ENABLE_LARGE_PAGES defaulting to 1 at :115 and not overridden by this
build), so a 512 KB block is still bin-allocated from a 4 MiB large page.
Growth is not free at that size for the same reason: arena_grow doubles to 1
MiB, above that bound and therefore a singleton OS allocation. One file in
twelve thousand pays it, which is why it is accepted. The largest single
file's scratch high water mark on this corpus is 33.5 MB, now transient
rather than retained.

Tests. Nothing in the suite asserted on result->arena size, block count or
total_alloc after cbm_extract_file, which is why this went unnoticed for the
life of extract_node_stack.h. traversal_stack_not_in_result_arena_issue2010
pins the byte budget: extracting "export const x = 1;" as TypeScript charged
365984 bytes to the result arena before and charges 87456 after, a
difference of 278528, exactly the two 4096-entry channel walks plus the
512-entry ES import walk at 32 bytes per TSNode. Of the 87456 that remain,
7680 is the defs item array at GROW_ARRAY's starting capacity of 32 times
sizeof(CBMDefinition) 240 and the other 79776 is everything else the
extraction interns, none of it traversal scratch; the 128 KB bound is
therefore above the new figure with room and a factor of four below the old
one. extract_traversal_stacks_come_from_ctx_scratch_issue2010 in test_mem.c
pins the lifetime, since a byte budget alone would also be satisfied by
shrinking CHAN_STACK_CAP: it builds the extraction context directly over two
arenas it owns, runs cbm_extract_channels, and asserts the scratch took at
least the 262144 bytes of the two walks while the result arena did not.
extract_c_macro_hidden_call_survives_preprocessed_pass_issue2010 is not a
scratch test. pp_ctx carries ctx->scratch so every context in the file is
uniform, but nothing reads it there: pp_ctx reaches only cbm_extract_unified
and cbm_run_c_lsp, and neither extract_unified.c nor anything under
internal/cbm/lsp/ includes extract_node_stack.h, so no traversal stack is
built on that path today. The test guards the macro-expansion path itself,
which had no assertion on a call that exists only after expansion.
ObjectScript composite extraction is already covered by
pipeline_objectscript_export_preserves_calls_sequential_parallel and
iris_export_xml_multi_class, so no new test is added there; a per-call arena
is correct on that path by construction, since each call owns and destroys
its own.

Local: scripts/test.sh, 7558 passed, 0 failed, 8 skipped, 139 suites.

Alternatives considered

Thread-local scratch arena, reset per file, with a depth guard for nesting.
Rejected on three counts. This codebase never uses thread-local storage for
owned bulk memory; every existing _Thread_local here is a small cache, a
guard or a log buffer, and every per-worker resource is a field in a context
struct. It would have needed cbm_arena_reset, which has no production caller
and a documented subtlety with grown block sizes (arena.c:213-217). And
extraction threads are created per cbm_parallel_for call and joined
(worker_pool.c:71,88) with no thread-exit hook in this project's own code,
so the arena would have leaked once per thread per dispatch and turned the
leak lane red.

Save and restore mark and release on the arena. Rejected because a per-call
arena already makes nesting correct by construction, and a mark type plus
two functions would have to be added to both arena headers.

Per-walk arenas, one per traversal rather than one per file. Rejected on
cost: 55 sites would each need an explicit span, for the same bound.

Right-sizing the initial capacities alone. CHAN_STACK_CAP at 256 rather than
4096 would cut those two sites from 3,168 MB to about 198 MB, which is real,
but it tunes the constant while the lifetime defect remains at all 55 sites,
and growth is not free: 256 doubling to 4096 abandons 122,880 bytes, and one
file in this corpus took 33.5 MB of stacks. The cap and the scratch block
size are coupled, which is a further reason to change one at a time. Worth
doing separately.

Routing to a scratch inside extract_node_stack.h while keeping the arena
parameters. Rejected because a parameter that is ignored lies about the API,
and the alias hazard above would stay a review obligation instead of a
compile error.

Freeing the per-file result arena earlier. Rejected: the results are read by
parallel_resolve and the infra passes. That is the retention half #1925
addresses, and this change is independent of it.

Signed-off-by: Daniel Ochoa <daniel.ochoa@closedloop.ai>
2026-09-02 09:22:37 -05:00
Martin Vogel 97517a4652 fix(python): carry bare-call parameter bindings in the walk
Replaces the cursor ascent from 0b0d143c. That commit fixed a real hang --
ts_node_parent() is O(depth), so a parent chain is O(depth^2) per call and
O(depth^3) across a file where every level of f(f(f(...))) is itself a bare
call, which hung stack_overflow_b's 30,000-deep fixture at 900s. But it fixed
it by making each hop cheap rather than by removing the per-call walk, so the
cost stayed O(depth) per call and needed a 64-ancestor cap to stay bounded.

That cap FAILS OPEN. Past 64 ancestors the guard stopped suppressing, so
deep-but-ordinary code silently kept the fabricated edges the guard exists to
remove -- and nothing failed when it did.

Carry the answer instead. The unified walk binds a def's or lambda's parameters
when it opens that scope and unwinds them when it closes, so the guard is an
O(1) map lookup. This is the lesson CBMWalkScope already records one struct
away: carry walk state, never recompute it per node.

  - CBMParamSlot is a name -> active-count map, not a set. `def outer(run):
    def inner(run):` binds one name twice and leaving the inner scope must not
    unbind the outer.
  - Bindings are pushed at the end of push_boundary_scopes, after every scope
    push for that node, so they land in the frame just pushed -- covering
    function_definition (push_function_scope) and lambda (push_lexical_boundary)
    alike -- and pop_expired_scopes unwinds to the frame's entry height.
  - Allocation failure sets py_param_tracking_failed and the lookup answers
    false forever after: a lost suppression, never a lost edge.

The repo's other binding table cannot serve this. CBMLexicalBinding is qsort-ed
in cbm_finalize_lexical_usages AFTER the walk and its active_end stays 0 until
then, so its binary search is invalid from handle_calls, which runs mid-walk
and before handle_usages.

extract_python_bare_call_scope_walk_is_bounded pinned the cap's fail-open
contract, which no longer exists, and is replaced by
extract_python_bare_call_flag_is_depth_independent. It keeps the shallow and
200-ancestor cases -- the deep one now asserts the call IS flagged, so
reintroducing a cap fails it -- and adds an unwind case: `handler` as both a
parameter and a module-level def, asserting the two shadowed calls are flagged
and the sibling call after that scope closed is NOT. A binding outliving its
frame would destroy that true edge, the one direction this guard must never
fail in. It passed on the first run.

extraction 327 passed; pipeline registry parallel complexity 407 passed;
stack_overflow_b 6 passed in 38s (46s with the cursor ascent, 900s rc=124
before it), lsp_python_deep_nesting_no_crash PASS.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-09-02 10:45:37 +02:00
Ilya Brykau 349aecba95 fix(pipeline): key the Go Field guard on the recorded selector shape, not reference text
cbm_go_suppress_bare_field_ref dropped a Field-targeted reference when
strchr(ref_name, '.') == NULL — but the extractor strips the receiver on
every path that reaches the resolver (resolve_lhs_write_name records the
trailing field name of a selector LHS; is_reference_node records the
inner field_identifier), so the dot test could never be false for Go.
The guard was a blanket veto: all ~4588 Go Field nodes recovered by
one 61de19bb were unreachable by USAGE/READS/WRITES, including genuine
selector references (issue #1962, found in maintainer post-merge review
of #1944).

The selector-vs-bare distinction exists at extraction time and was
discarded; record it and let the resolver consume it:

- CBMUsage/CBMReadWrite gain is_member_access. The usage recorder sets
  it when the reference node is a field_identifier (the member half of a
  selector); resolve_lhs_write_name reports it through an out-param when
  it takes the field/member LHS branch.
- cbm_go_suppress_bare_field_ref(is_go, is_member_access, target_label)
  refuses a Field bind only for references that were never the member
  half of a selector. All four resolver sites (sequential and parallel,
  USAGE and READS/WRITES) pass the recorded flag.

Reproduce-first: with the veto restored, the extended #1942 fixtures go
RED on both resolver paths at the new asserts (t.err = nil must WRITE
the field; t.n must produce USAGE); with the fix both twins are GREEN
and the original bare-local negatives still hold. The dead
ASSERT_FALSE(..., "t.err", ...) unit case — an input Go's extractor
cannot produce — is replaced by flag-based cases.

Fixes #1962.

Signed-off-by: Ilya Brykau <ilya.brykau@orca.security>
2026-09-02 10:39:03 +02:00
Joshua Richter 592894a438 fix(routes): take a route's handler from the last argument, not the first
Express, Fastify, gin and Laravel all put middleware between the route
path and the handler, and the handler comes last:

    routerGet("/users", requireAuth, rateLimit, listUsers);

extract_handler_arg returned the FIRST argument that looked like a
function reference, so requireAuth won and the HANDLES edge pointed at
the middleware instead of listUsers. That is a wrong edge, not a
missing one, and it misleads anyone tracing a request through the
graph.

The scan bound hid a second fault. An arrow function matches none of
the accepted node kinds, so three inline middlewares pushed the real
handler past MAX_HANDLER_SCAN and no handler came back at all.

The loop now examines every argument and keeps the last eligible one.
Nothing that follows a handler matches the accepted kinds either, since
an options argument is an object node, so the last match is the
handler. HANDLER_START_IDX stays, because argument 0 really is the
path. MAX_HANDLER_SCAN had no other use and is gone.

A two-argument route has one eligible argument, so first and last agree
and the existing Express, Fastify, gin and Laravel route tests are
unchanged.

Both tests fail without the change, the first reporting "requireAuth"
where "listUsers" belongs and the second reporting no handler at all.

Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
2026-09-01 22:31:03 -04:00
Joshua Richter c36b4fbc44 fix(routes): find a route path wherever it sits in a decorator's arguments
Java and Kotlin put no order on annotation attributes, so `path` can
appear after `method`, `produces` and `consumes`:

    @RequestMapping(method = RequestMethod.GET,
                    produces = MediaType.APPLICATION_JSON_VALUE,
                    consumes = MediaType.APPLICATION_JSON_VALUE,
                    path = "/orders")

extract_route_path_from_args stopped after the third argument, so the
path was never read and no Route node formed. The endpoint was in the
source and not in the graph.

Four call sites read this function, so the miss covered plain decorator
arguments, Spring mappings, and both JAX-RS paths.

The loop now checks every argument. That matches find_drf_kwarg_in_args
a few lines below, which walks the same kind of node with no cap. Cost
stays small: the loop returns on the first path-shaped string, and each
argument's own subtree walk is still bounded by find_route_path_literal.

The recursive breadth guard inside find_route_path_literal keeps
DECORATOR_SCAN_LIMIT. That loop descends to CBM_DESCENDANT_MAX_DEPTH, so
breadth 3 by depth 6 is 729 visits at worst and unbounded breadth there
is not bounded at all.

The test fails without the change, reporting the class-level "/api"
route present and "/api/orders" missing.

Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
2026-09-01 22:30:54 -04:00
The1nkandClaude Opus 5 fd73c347fb feat(extract): emit Route nodes for Blazor @page directives
get_architecture's routes aspect returned nothing for a Blazor
application: the entry surface of the app was invisible in the graph
even though every @page directive names one.

@page lives in markup above the @code block, which tree-sitter's C#
grammar never parses, so there is no AST node to read it from. Scan the
raw source for it instead. The match is deliberately strict — the
directive must be the first token on its line, followed by whitespace
and a double-quoted path beginning with '/' — so @pageSize and prose
mentions cannot match.

The route is attached to the file's module def. A .razor component's
class is implicit, so there is no class node to carry it, and the module
QN already is the component's identity. insert_def_into_gbuf is
label-agnostic and creates Route + HANDLES from route_path.

pass_route_nodes.c's ensure_decorator_routes gains "Module" for the same
reason. Extraction covers the full-index path on its own; this backstop
is what runs on an incremental re-index, so without it a component's
Route would appear on a full index and vanish the next time that one
file changed. Its loop bound now derives from the labels array rather
than borrowing the unrelated RN_STRIP_PASSES, so adding a label cannot
silently skip it. Verified no double-creation when both paths can fire.

Measured on a real Blazor application (36 .razor files): routes went
from 0 to 11, with 11 matching HANDLES edges, and no change to the
363 Class / 1775 Method / 3805 CALLS already extracted from its C#.

Covered by two new tests in tests/test_extraction.c.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: The1nk <23087600+The1nk@users.noreply.github.com>
2026-09-01 22:15:52 -04:00
Joshua Richter e820409233 docs(vendored): record the Swift scanner shift patch in MANIFEST.md
The vendoring table claims our copy of a grammar matches the pinned
upstream commit. A local patch makes that claim false unless the patch
is written down, so a future re-vendor would drop the fix without
anyone noticing.

The row covers both changed lines, because both are the same defect in
the same file: OP_SYMBOL_SUPPRESSOR at line 131 and eat_operators at
line 514.

The row also names the two upstream commits that already carry these
changes, and the table intro gains one clause for the case. Upstream
fixed eat_operators in fb63a7004f07 on 2026-04-06 (their issue #558)
and the OP_SYMBOL_SUPPRESSOR entry in 6ab8d1d74ebd on 2026-08-10. Our
pin 8abb3e8b3325 is from 2026-03-20 and predates both, so these are
backports rather than local inventions.

That distinction changes what a re-vendor should do. The table's
instruction is "must re-apply these", which is right for the other four
rows -- upstream never took those. For this row it would mean
hand-applying a change the newer file already has. The row now says to
delete it instead, and the intro allows for that case.

MANIFEST.md is itself listed in scripts/vendored-checksums.txt, so its
own recorded checksum moves with the edit.

Refs #1892

Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
2026-09-01 20:44:13 -04:00
Martin Vogel 63f79365e3 Merge branch 'main' into feat/python-bare-local-binding 2026-09-02 01:50:50 +02:00
Joshua Richter a2ea711c29 fix(grammar): stop the Swift try-bang entry shifting past its own type
scanner.c line 131 shifts a literal 1UL by 32 to build the entry that
suppresses the bang of a try!. unsigned long is 64 bits on Linux and
macOS but 32 bits on Windows, so there the shift count equals the width
of the type, which is undefined.

1ULL is 64 bits on every target this project builds for.

Nothing reports it today. The expression is a compile-time constant, so
no sanitizer sees it run, and Makefile.cbm:719 builds vendored grammars
with -w, which switches off -Wshift-count-overflow.

Upstream tree-sitter-swift made the same change in 6ab8d1d74ebd, after
the commit this grammar is pinned to.

Refs #1892

Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
2026-09-01 17:52:38 -04:00
Joshua Richter 3f831936b5 fix(grammar): stop the Swift scanner shifting past the width of an int
The Swift scanner keeps a 64-bit mask of the symbols that suppress a
match -- the rule that stops `try!` emitting its `!` as a token of its
own. It tests one bit per candidate:

    uint64_t suppressing_symbols = OP_SYMBOL_SUPPRESSOR[full_match];
    for (uint64_t suppressor = 0; suppressor < TOKEN_COUNT; suppressor++) {
        if (!(suppressing_symbols & 1 << suppressor)) {

The mask is uint64_t but the literal `1` is an int, so the shift is an
int shift. TOKEN_COUNT is larger than 32, so once suppressor reaches 31
the shift runs past the width of the type. That is undefined behavior,
and every bit above 31 is tested against a value the standard does not
define.

Nothing caught it because nothing in the tree reached the suppressor
path. Any Swift force-unwrap does: `cached!` is enough.

UBSan reports it as:

    scanner.c:514:47: runtime error: left shift of 1 by 31 places
    cannot be represented in type 'int'

`1ULL` makes the literal as wide as the mask it is tested against.

The new test in tests/test_extraction.c cannot go red on its own. The
normal test build prints the UBSan message and carries on, which is why
this survived. The Windows CLANGARM64 leg runs UBSan in trap mode, and
there the same shift is an illegal-instruction crash -- so the test
exists to make sure that leg keeps parsing a force-unwrap at all.

scripts/vendored-checksums.txt records the new hash for the one changed
file, as scripts/security-vendored.sh --update writes it. Layer 8 of the
security gate compares vendored content against that manifest, so the
edit and its recorded hash belong in the same commit.

Found while adding Swift URL extraction in #1892 / #1976, and split out
of that PR so the vendored change can be reviewed on its own.

Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
2026-09-01 17:52:38 -04:00
Martin Vogelandastandrik da61c81af2 fix(lsp): unify the two type short-name indexes that broke the main build
main does not compile. internal/cbm/lsp/type_registry.h declares
type_short_buckets, type_short_entries and type_short_bucket_count twice, and
cbm_registry_types_by_short_name twice with different parameter types.

Two changes implemented the same feature independently:

  a4c0ffbc  perf(cs): eliminate the C# corpus-proportional scans  (2026-08-16, ours)
            CBMTypeShortIter, cbm_type_short_iter_next
  793716dc  perf(lsp): index C++ short-name type lookups     (PR #1679, astandrik)
            CBMTypeNameIter,  cbm_type_name_iter_next

They landed in different hunks of the same struct and the same header section,
so git merged them with no conflict and nothing failed until the merge commit
was built. This is the failure mode where a clean merge is not evidence that
two changes compose.

The two iterators are structurally identical -- same five fields, byte-identical
_next bodies reading the same reg->type_short_entries chain. They differ in one
place, and it matters:

  #1679          on aux-index allocation failure, sets tail_i = 0, so the
                 iterator degrades to a full types[] scan and the candidate
                 set is unchanged.
  a4c0ffbc       sets tail_i = reg->type_qn_entry_count unconditionally, so on
                 that same failure it SKIPS the first type_qn_entry_count types
                 and silently yields an incomplete candidate set.

So this keeps a4c0ffbc's names -- CBMTypeShortIter reads correctly beside its
type_short_* members and beside CBMTypeEmbedIter and CBMFreeFuncIter -- and
#1679's body, which is the correct one. The surviving declaration carries both
rationales: the C++ short-name lookup and cs_resolve_type_name's step-9
fallback are now stated as two consumers of one index, which is what they are.

The merge also broke two comment/member pairings, repaired here: the duplicated
members had displaced the free-function index members from under their own
comment, and the CBMTypeShortIter block had been interleaved between
CBMFreeFuncIter's typedef and its declarations.

No behaviour change for either consumer beyond the allocation-failure path,
which strictly gains coverage for C#. c_lsp.c and cs_lsp.c now call one
function. #1679's tests, including the fault-injection case that NULLs the
buckets and asserts the full-scan fallback, carry over unchanged apart from the
type name.

Local: c_lsp, cs_lsp and registry -- 881 passed, 0 failed.

Co-Authored-By: astandrik <astandrik@yandex-team.ru>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-09-01 18:43:37 +02:00
Martin Vogel 7808686cee Merge pull request #1679 from astandrik/codex/perf-cpp-short-name-lookups
perf(lsp): index C++ short-name type lookups
2026-09-01 17:08:48 +02:00
Martin Vogel 3627eff580 Merge pull request #1389 from Yyunozor/fix/arg-url-false-positives
fix(pipeline): reject non-HTTP slash arguments
2026-09-01 17:00:12 +02:00