mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-09 09:11:00 +08:00
9db483402ea2fd4e0a5ff3bdd2b1fc50a7518ec9
543
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
8c1a9d61c2 |
fix(extract): no clock decides what ends up in the graph
Indexing the same C# corpus twice on the same machine produced 5,794,336 edges
and then 5,794,328. Every other edge type matched exactly, every semantic
checksum matched (447,064 functions, 81,534 tokens, identical vec/idf/toks),
and the difference was ten WRITES edges out of one generated file.
Three places in extraction decided graph content from a clock:
* the unified AST walk stopped on a thread-CPU deadline;
* a file whose parse spent more than half its budget was refused the
per-file and cross-file LSP walks;
* so was a file whose parse plus walk exceeded the budget.
The first is the one that bit. Any file large enough to reach the deadline is
cut at whatever node the clock lands on, and five generated .NET JIT tests
reached it on every run: hugeSimpleExpr1.cs stopped at node 2,660,352 in one
run and 2,574,336 in the next, out of 2,774,794. That file's tail is field
assignments, so the two runs disagreed by exactly ten WRITES edges. The graph
was a function of how fast the machine happened to be running, and nothing said
so: result->walk_truncated was set and no code anywhere read it.
The deadline existed because those files cost 18-48 us per visited node, and
that turns out to be one fixable thing. csharp_callable_value_site() was the
only language branch in call_reference_candidate_site() still climbing with
ts_node_parent, which tree-sitter answers by descending from the ROOT, scanning
children at every level; every other language already climbs on the walk
cursor. On expressions megabytes deep that was ~85% of the entire unified walk
(sampled: stamp_usage_site -> ts_node_parent -> ts_node_child_with_descendant).
It now takes the cursor and the parent the caller already resolved, and
is_direct_argument_value_cursor() is split so a caller that has stepped the
cursor onto the parent can continue from there rather than paying for it twice.
With that fixed, the budgets are not needed to make indexing finish, so they
are gone rather than made deterministic. A cutoff in nodes would at least be
reproducible, but it still answers "what is in this repo?" with "as much as we
felt like reading". CBM_WALK_MAX_NODES reinstates one for an operator who wants
it; the default is no budget.
Removing the two LSP-skip rules was checked against the file and the path they
were added for (a 23 MB single-expression JIT test that once cost 354 s in the
walk and then crashed the cross-file resolve): the whole 12k-file C# corpus
twice with cross-file resolve, that file alone, and synthetic C# nested 10,000
levels deep — no crash, and past roughly that depth tree-sitter stops producing
a usable tree, so a tree deep enough to endanger a recursive walk never reaches
one.
C# corpus, before -> after:
edges 5,794,318 / 5,794,328 / 5,794,336 -> 5,794,873 every run
truncated 5 files, unreported -> 0
time ~115 s -> 97-101 s
Deterministic, about 550 edges more complete (the tails nobody was walking),
and faster than the build that was cutting them off. All ten bench corpora now
return identical nodes and edges across two runs.
A truncated walk and a skipped LSP walk are now reported in the coverage report
with how far the walk got ("1673216/4133946 nodes walked"), like a partial
parse. A file the walk abandoned halfway used to be reported as fully indexed,
which is a graph quietly lying about its coverage.
Tests: extract_csharp_argument_values_use_the_walk_cursor asserts no occurrence
falls back to the root-descending parent lookup — 128 fallbacks without the fix
and 0 with it. Its fixture parenthesises the argument on purpose: a bare
Sink(Target) needs no climb at all, so it cannot tell the two walks apart and
passed either way while the bug was in place.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
|
c7ccb479b4 |
fix(grammar): give the properties scanner per-parser state
Indexing a repo with .properties files in it produced a different graph from
one run to the next. Every few runs one small file - 253 bytes, valid, the same
file that had indexed fine a moment earlier - vanished from the graph, and the
coverage tree put raw-path File/Folder nodes where it had been. On the kotlin
corpus that showed up as node counts wandering between 33,486 and 33,494.
The vendored tree-sitter properties grammar keeps its external scanner's entire
state in a file-scope static:
static bool reached_eof = false;
bool ..._external_scanner_scan(void *payload, TSLexer *lexer, const bool *valid) {
lexer->result_symbol = FAKE_EOL;
return reached_eof = !reached_eof && valid[FAKE_EOL] && lexer->eof(lexer);
}
That flag is how the scanner refuses to emit a second end-of-input FAKE_EOL,
and the grammar needs exactly one to close a file. One static is one object for
the whole process, so with several index workers parsing .properties files at
once, whichever thread reached EOF first set the flag and the next thread's
`!reached_eof` was false: its parse never received the FAKE_EOL it was waiting
for and sat at end-of-input asking for a token that would never arrive. The
per-file parse budget stopped it five seconds later and the file was dropped.
Measured on a 253-byte file: 1,060,630 progress callbacks, ~106 M parse
operations, byte offset pinned at EOF, has_error false - the parser was not
recovering from an error, it was waiting.
The state now lives in the payload from ..._external_scanner_create(), which is
what tree-sitter's create/destroy contract is for. The serialize/deserialize
wire format is unchanged: the state is still the returned length.
Proof, interleaved under identical load so both binaries meet the same machine,
40 runs each: unpatched 6 stalls, patched 0. Under load the kotlin corpus now
returns 33494/203579 ten times out of ten and java 693129/5523595 three times
out of three; both used to wander.
All 103 vendored scanners were audited for the same shape. This was the only
one that WRITES a process-wide static. haskell's debug_parse_metadata and
purescript's res_cont/res_fail are read-only - constants missing a const - and
are left alone.
Upstream 6310671b24d4 still carries the static, so MANIFEST.md records this in
the local-patch table and scripts/vendored-checksums.txt is repinned.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
|
1d58385b51 |
refactor(mem): route this branch's allocations through the memory core
The memory-core linter is a ratchet: raw malloc/calloc/realloc/free/strdup sites may not grow, because allocations belong on one accounted path (src/foundation/mem_core.h) rather than spread across hundreds of call sites. This branch's own work had added 35 raw sites across nine files -- the parallel k8s prep, the configlink key matching, the worker pool's counted path, the Go shared stdlib registry, the spill namespace copy, the route service array and the namespace-name arrays. All of them now use cbm_alloc/cbm_calloc/cbm_realloc/cbm_free/ cbm_mem_strdup. One was a latent bug rather than a style point: pass_k8s freed a buffer that k8s_read_file had allocated with raw malloc, so converting only the free would have mismatched the allocator -- the read function and every free of its result are converted together. The Windows free-space path widens its argument through the memory core instead of cbm_utf8_to_wide(), which allocates with raw malloc. Two files improved enough to tighten their baselines, which the ratchet requires so the gain cannot be quietly given back: sqlite_writer.c 92->74 (the index-cell arena), pass_k8s.c 13->6 and registry.c 19->17. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
|
c3714f1133 |
fix(mem): size the budget from free memory, and relieve on real scarcity
Indexing the linux kernel with NO ceiling set took the machine down: on a
48 GB host the default budget is 50% of TOTAL ram (24 GB), the run used
essentially all of it (24.5 GB peak RSS), and with a 12 GiB VM already
running there was nothing left. The budget never fired -- it read 22 GB
charged against 24 GB -- while the host was out of memory.
Total RAM is the CEILING; what is free right now is the CONSTRAINT.
- cbm_mem_clamp_to_available() reduces a fraction-derived budget to fit
what is actually free, reserving a quarter of it (held between 1 and
8 GB) for the OS and everything else, never below a 512 MB floor. An
explicit CBM_MEM_BUDGET_MB still wins -- it is the user's deliberate
choice -- but is warned about when it exceeds free memory, because
the failure it buys reads like a product bug rather than a setting.
- cbm_mem_should_relieve() = our budget exceeded OR the machine short.
Deliberately separate from cbm_mem_over_budget(): system pressure
must RELIEVE (spill, reclaim), never abort, since the spike may not
even be ours. The extraction loop now enters spill on real scarcity,
and the dump phase -- where the kernel run actually died, with our
charge comfortably under budget -- keys its free-heavy decision on it
too. cbm_mem_system_under_pressure() existed and had no callers.
Same scenario after: budget clamped to 18,947 MB, peak RSS 19.5 GB,
status indexed in 185.7 s, machine fine.
Spill also gets limits and cleanup, because relief that fills the disk or
leaves gigabytes behind is not relief:
- a free-space preflight (never start spilling below 10 GB free) and a
ceiling at half of what is free; past it results stay in memory and
back-pressure takes over. cbm_fs_free_bytes() is new (statvfs
f_bavail / GetDiskFreeSpaceExW quota).
- POSIX unlinks each spill file while holding it open, so the space
returns on ANY exit including SIGKILL -- a killed run cannot leave
spill behind because there is no name left to leave it under.
Windows keeps close-path removal plus a sweep where an open file
refuses deletion, which is exactly the liveness test.
- the staging-database sweep now reclaims orphans of EVERY project in
the cache directory, not just the one being indexed. A kernel index
killed by the OOM killer used to park ~15 GB until someone re-indexed
the kernel, and indefinitely if nobody did. The per-stage lock probe
still keeps a live writer's stage, whichever project owns it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
|
92e7e6a913 |
fix: keep large thread-locals off the static TLS block
Indexing broke on x86-64 Linux: the index worker refused to start and
SIGKILLed its own process group, so every venue reported nothing but
"index worker ended with killed (exit=-1, signal=9)".
glibc places a thread's static TLS block inside the stack allocation it
makes for that thread. This branch's new thread-local caches -- a
512-slot field cache (~24 KB), per-depth tree cursors (~4 KB) and two
parked CBMArenas (~8 KB; an arena carries blocks[256] + block_sizes[256])
-- took static TLS from 47,905 to 93,513 bytes. The 64 KB parent-death
watchdog thread then could not be created at all, and a worker without
process-tree containment correctly refuses to index.
Four layers, because there were four things wrong:
- The big thread-locals move to the heap behind a thread-local POINTER,
allocated once per thread and reused, each with a correct fallback if
allocation fails (the field cache and cursor pool are caches; a miss
is slower, never wrong). Static TLS is back to 52,233 bytes.
- PARENT_WATCHDOG_STACK_SIZE 64 KB -> 256 KB. "The watchdog only polls,
a tiny stack suffices" was true when written and false once TLS
reached ~50 KB, because the TLS block comes out of that same
allocation.
- cbm_thread_create retries with the default stack on EINVAL. A stack
size is a hint about what a thread needs; the platform refusing the
hint is not a reason to fail to create the thread.
- tests/test_thread_stack_tls_contract.sh (Step 5a of scripts/test.sh)
keeps static TLS a quarter clear of the smallest requested stack,
reading that smallest size from the sources so a new small stack
tightens the gate automatically.
Two diagnostics that made this findable at all, and that stay:
smoke-test.sh prints the worker log its failure message names (CI deletes
that sandbox with the job, so the one artifact naming the cause was
unreadable), and worker_containment_unavailable() now reports WHICH of
its conditions failed, with pid/pgrp/ppid.
Verified in the amd64 container, the venue nothing on the local ladder
covers: baseline main indexes 9039 nodes/15067 edges, this branch before
the fix refuses to start the worker, and after it indexes 9039/15067.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
|
959f4de50c |
perf: cut the allocation churn and repeated work the sanitizer found
Every item here was measured on a real corpus before and after, not
reasoned about. Go corpus, event lane, first run -> now: 245.5 M
allocations -> 134.8 M, 190 GB cumulative -> 29.3 GB, 158 GB never
written -> 9.2 GB, 132 M churn pairs -> 35.7 M, 93 GB of allocator byte
work -> 0.84 GB. Wall clock: Go 40.7 s -> 19.3 s, linux kernel 239.6 s ->
194.6 s, and all ten LSP languages faster with none slower.
Graph output is unchanged, proven rather than assumed: per-class multiset
digests over the full kernel index (8,563,471 nodes identical, 18/20 node
classes and 18/24 edge classes byte-identical, the rest differing only in
the classes a same-binary re-run also perturbs through worker merge order).
The changes, each with its own reason:
arena an exact arena that is appended to later restarts growth at
8 KB instead of the 64 KB default — the cross-file pass adds a
few resolved calls to a compacted result, so the default block
was ~60 KB untouched per result, 0.5 GB of the Go worker peak.
cbm_arena_init_lazy defers the first block for arenas that are
usually never used at all.
scratch per-thread reuse instead of allocate-and-free per unit of
work: result_compact (8 MB kept), the cross-file pass (4 MB),
extract_defs' walk stack, and the sqlite index cells (a 1 MB
arena replacing 12.1 M malloc/free pairs on Go). Walk frames
moving into scratch is why pass_k8s now opens the keep window
explicitly: without it every YAML file opened a 512 KB scratch
on the main thread, +3 GB.
sqlite index cells are written in place — building the record and
copying it into the cell was a second allocation per entry,
16 M record buffers of pure churn on Go.
registry is_test_qn and common_prefix_len in one pass, per-bucket test
flags computed once instead of per candidate comparison, and
the reachability keys arena allocated lazily.
minhash the unique-trigram set is sized from the trigram count and
skips repeats, and LSH queries reuse one seen-set per worker
instead of allocating per query.
parallel lsp_surface rows and the k8s manifest prep now run across
workers (graph writes stay sequential and ordered), the file
node is hoisted out of register_and_link_def, and configlink
prepares dependency imports once with a length prefilter.
go_lsp one sealed stdlib registry shared by all workers, published
by CAS, instead of one per worker.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
|
d70b794179 |
fix(mem): the MSan guard in the spill store goes through sanitized.h
cppcheck cannot evaluate `#if __has_feature(memory_sanitizer)` even
behind `#if defined(__has_feature)` ("undefined function-like macro
invocation", the lint lane on
|
||
|
|
d89ccabef0 |
fix(mem): the parked result image is declared defined for MemorySanitizer
The MSan lane on
|
||
|
|
9724d90304 |
fix(mem): dedicated SQLite heap only in the worker; guard the committed stat
The daemon leaked 180 KB per query. The Linux soak's quick leg went from 11 to 144 MB of RSS in ten minutes (13.1x, 628 MB/hr), mimalloc's committed bytes tracking RSS almost byte for byte and the growth linear in the query count, file descriptors flat -- where the same leg had been flat at 15 MB in July. The cause is the previous commit's SQLite heap: one mimalloc heap per thread, created on first use. The daemon runs a thread per connection, so every query ran on a fresh thread with a fresh heap, and every block the shared connection keeps beyond the request -- page cache pages, the statement cache, schema objects -- was allocated in that heap's pages. When the thread exited, its heap was deleted and those pages were left behind holding a few live blocks each: committed for as long as the connection lives, a few pages per query, never reused by anyone. The dedicated heap exists for one reason: the index worker's default heap holds the graph (40M+ blocks), and SQLite churn on that heap paid a page walk per allocation during the coverage publish (132 s vs 9.7 s on the kernel). That process has long-lived threads and exits when the index is done. So the dedicated heap is now a switch, off by default, that the index worker turns on at its entry (main.c, --index-worker); everywhere else SQLite allocates from the calling thread's default heap exactly as before the previous commit. The switch exists in every build and changes nothing where the allocator binds are compiled out. Linux soak, quick leg, before -> after: RSS 11 -> 144 MB in ten minutes (13.1x, 628 MB/hr) -> 10.6 -> 11.5 MB and flat from the first minute over 564 queries with reindexes every two minutes; query-leak leg: 11.6 MB flat over 1,376 queries; committed heap 6.8 MB flat. The same soak showed a second thing. From its second sample on, the query-leak leg reported a committed heap of 2^64 - 121 MB: mimalloc's committed statistic is a signed counter merged per thread at thread exit, and a process whose long-lived thread commits what its short-lived threads free reads it negative -- cast to size_t on the way out. That figure is one of the two the budget takes the larger of. cbm_mem_allocator_committed now reports 0 for a reading above SIZE_MAX/2, so the charge falls back to the OS number instead of a 16 EB budget breach, and the diagnostics report the committed pair through it (the peak never undercuts the current value and never carries a wrapped one), the way they already did for RSS. Detector: the soak's quick and query-leak legs (scripts/soak-legs.sh), the release-gating sequence in every venue -- RED on the previous tree (13.1x), GREEN on this one. A unit test cannot see this class: the test build keeps SQLite on libc so the sanitizers see every block. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
|
7b77fd4834 |
feat(mem): memory on the core, waste cut, results spilled under budget
Second increment on the memory core. The first landed the core, the linter
and phase attribution with zero production call sites; this one migrates the
memory that matters, proves where it goes, removes the waste the proof
exposed, and makes the memory budget a promise: over budget, extraction
results go to disk and the run completes at the floor instead of aborting.
Every number below is from a real index on this machine (M5 Pro, 48 GB) with
CBM_MEM_PHASES=1 and the release build.
WHAT THE PROOF SAID (Go corpus: 21,875 files, 345 MB of source)
The graph buffer, blamed for the peak since the 2026-07 research, is 3% of
the memory: 0.55 GB peak. The per-file extraction results are the memory:
14.8 GB written into their arenas, of which 3.4 GB is reachable. The rest
is temporaries -- cbm_node_text copies at 496 call sites, per-node QN
strings, abandoned generations of every growable array -- that an arena
cannot free and that the result kept alive until resolve ended, because
the result owned the arena. The arenas were charged 28.8 GB of capacity
for those 14.8 GB. On the kernel the same shape was 8.9 GB used by 89,731
results, and resolve grew it to 13.3 GB used / 17.2 GB capacity because
the C walk allocated its working set in the result arena and stored arena
pointers into the shared registry. After that, 7 GB of the kernel's
post-resolve memory had no class at all.
WHAT CHANGED
Attribution. The graph buffer (87 raw sites -> 0), arena blocks, the
tree-sitter slab allocator, the bound tree-sitter/SQLite allocators, the
semantic path (pass_semantic_edges, pass_similarity, minhash: 80 sites), the
hash table (Verstable, a class per table via CTX_TY: cbm_ht_create_in) and
CBM_DYN_ARRAY allocate through the core. Classes arena, ts_tree, hash_table
and dyn_array join the table. Two cross-boundary frees of buffer-owned
strings (pass_importance, pass_complexity) go through a new setter,
cbm_gbuf_node_set_properties_json. The linter baseline drops 4086 -> 3916.
Result compaction (internal/cbm/result_compact.c). At the end of a file's
extraction, everything reachable from its result is measured, copied into
one exact-size arena (strings interned by content within the file) and the
working arena is handed back. Later appends restart growth at the default
block (cbm_arena_init_exact, arena grow_size). The stale duplicate
internal/cbm/arena.h, which shared the foundation header's include guard
and won or lost by include order, is a shim now.
Record slimming. CBMCall 320 -> 72 bytes: the eight inline argument slots
(256 bytes, mostly empty) are allocated on first capture. CBMUsage 48 -> 40.
The per-file overlay contract (src/pipeline/pass_lsp_cross.c). One
lifecycle for every language's cross-file resolve: a scratch arena, a
per-file overlay registry chained to the sealed shared base, the walk
against the overlay, resolved calls appended into the result, scratch
destroyed. type_registry gains chain-aware iterators (every yielded index
belongs to it->reg) and copy-on-write refinement
(cbm_registry_func/type_for_update), and the C, C#, Go and Python walks
use them. This is what an ASan heap-use-after-free in the
lsp_resolution_probe suite demanded: the C walk stored per-call arena
objects into the shared registry, and a kernel run with a naive scratch
arena lost 1,045 edges to the resulting garbage reads. It also ends
unsynchronized writes into the shared registry from parallel resolve
workers.
The 7 GB with no class, found and cleaned. gbuf_index tables and keys were
2.5 GB live and 7.1 GB at peak: the 18 worker buffers built the five
secondary indexes they never query (cbm_gbuf_new_worker builds none), and
18M per-key index arrays started at 8 slots (cbm_da_push_min starts them
at 2); edge property JSON is 81% "{}" (interned). The semantic pass
allocated a 512-token stride per function up front (packed tokens with
offsets: cbm_sem_corpus_add_docs_batch). Edge dedup keys are 128-bit
hashes instead of sprintf'd strings (edge_key_map_t). Each worker keeps
one reusable working arena between files (cbm_work_arena_take/give:
rewind, not free) and every phase mark returns freed pages to the OS.
Budget metric. The over-budget check compares the charged footprint --
max(phys_footprint, mimalloc commit) on macOS -- not resident_size: after
extraction the kernel worker sat at 17.4 GB RSS with 5.5 GB charged, because
pages mimalloc has purged stay resident until the OS wants them, and
phys_footprint under-reports after MADV_FREE_REUSABLE cycles.
Spill / admission control (internal/cbm/result_spill.c, the extract gate
in pass_parallel.c). At 15/16 of the budget (or over it, or with
CBM_MEM_SPILL=1) the gate latches spill mode -- early, because the gate
sees the crossing per file pull and the workers' in-flight files carry the
charge past the line before the first sweep lands: every compacted result
is parked on disk as header + block in one append-only file per writer
under <cache>/spill/, the results already cached are swept out by every
worker, and registry build, def collection, surface rows, resolve and the
infra-route passes load a result only for the moment they read it
(relocated by base delta). The abort path stays closed while a sweep runs
or a cached result remains; it fires only at the floor -- graph +
registries + in-flight files. A result that owns sub-results stays in
memory; a retained parse tree is dropped with the parked result and
re-parsed on load. Every consumer of the result cache reads a slot through
cbm_pipeline_result_acquire()/release() (pipeline_internal.h): a parked
slot is NULL in the array, and the passes that indexed it directly lost
every __route__infra__ node and its INFRA_MAPS edges before the contract.
The other things the transient loads found: a def label that was the
result's own pointer (SIGBUS in the Go cross registry builder), two JVM
helpers that passed their input through, a dedup table whose keys borrowed
sr->value (the insert spun on freed memory), and a latch that was visible
before the store was open (a peer found nothing to park and failed the run
while its neighbour wrote 9 GB). The arena rewind reuse branch had
defeated the nblocks = CBM_ARENA_MAX_BLOCKS OOM seam the suites use; it
reuses only blocks that exist. The incremental and probe routes still
index the cache array themselves and therefore never spill
(ctx->spill_allowed stays false there; follow-up).
The semantic pass under a budget (src/pipeline/pass_semantic_edges.c). The
pass held every function's tokens, the corpus's per-document token ids and
the vectors being stored at once: 4.9 GB on the kernel, on top of the
floor, and nothing sized it to the headroom. Now, when the charged
footprint plus that transient would cross the budget, phases 2-4 run per
batch of functions: tokenize -> count into the corpus -> free; finalize
once; tokenize again -> vectorize -> store -> free, with pages returned to
the OS after every batch. Tokenization is deterministic, so both passes
see the same tokens and the graph does not change; the run pays with a
second tokenize. CBM_SEM_BATCH=<n> forces the batch size (the
batched-equals-unbatched test). The corpus frees its per-document token
ids at the end of finalize (the co-occurrence pass is their only reader),
and mem.semantic.step lines (CBM_MEM_PHASES=1) put the class's live/peak
bytes and the charged footprint at every sub-phase.
The semantic corpus and tokenizer allocate through the core (54 raw sites
-> 0). This is not attribution only: cbm_sem_tokenize handed out libc
strdup blocks that pass_semantic_edges freed with cbm_free -- mi_free in
the production build -- since the pass moved onto the core. No libc-backed
test build can see a cross-allocator free, so the core now refuses a block
it does not own (mi_is_in_heap_region) when CBM_MEM_PHASES=1 is set,
fatally: every proof run in this PR ran with it armed, and the Go and
kernel indexes report zero foreign blocks.
Accounting without contention. The core charged every allocation and free
to shared per-class atomics; with 18 workers on 18 cores every block bounced
the same three cache lines, and the bench against the shipped v0.10.8 showed
it as CPU time rising faster than wall time for the same graph: Kotlin 38 ->
121 s CPU (+58% wall), TypeScript 210 -> 596 s (+67%), Go 177 -> 312 s. The
deltas are thread-local now and reach the shared counters once per 256 KB or
512 blocks of change per thread, when a parallel-for worker ends, and
whenever a reader looks (its own thread first). Phase marks read after the
join and are exact; a class peak is a diagnostic and can be low by threads x
256 KB. Result: Kotlin 10.9 s / 43 s CPU (the residual +22% wall against
v0.10.8 tracks its +34% nodes), TypeScript 33.9 s (v0.10.8: 33.4), Go 48.4 s
(v0.10.8: 52.6), Java 81.1 s (v0.10.8: 79.8); graphs unchanged.
SQLite on a heap of its own. The same bench put the kernel's write step at
132 s where v0.10.8 needed 9.7 s, all of it in one coverage publish
statement. A stack sample named the cost: mimalloc's free-page search and
its periodic heap collect, walking the graph's pages -- millions of them
once the graph lives on the core -- for every statement-journal chunk
SQLite allocates. SQLite now allocates from a mimalloc heap of its own per
thread (mi_heap_new; mi_free is heap-agnostic, so nothing else changes),
and the periodic collect is set to the maximum mimalloc allows since the
explicit release at phase marks and spill sweeps is where memory goes
back. Coverage publish 132 -> 9.4 s; profiled kernel 376 -> 248 s
(v0.10.8: 296 s).
Formatting without a process lock. On macOS every vsnprintf takes the
process locale under an unfair lock (localeconv_l inside __vfprintf);
eighteen workers formatting type names on the C# corpus collapsed into
it -- the 10 MB JIT test files took 68 s each instead of under a second
-- and cbm_arena_sprintf paid it twice per string (size pass, write
pass). Each thread now formats with a C locale object of its own
(vsnprintf_l), and the common short string is formatted once into a
stack buffer.
Per-file budgets that cover the whole file. The 5 s per-file budget
covered the parse only; the LSP walks after it had none. C# JIT stress
files (a 23 MB single expression among them) parsed inside the budget --
the old wall-clock parse timeout used to drop them, which hid the rest --
and then held a worker for 346 s each in the usage walk (tree-sitter's
ts_node_parent descends from the root: quadratic on a deep tree), and the
cross-file resolve ran on the same trees. Three rules now, one site for
every language (cbm_extract_file_ex, honoured by cbm_pxc_dispatch_file): a
parse that used more than half the budget disqualifies the file from the
LSP walks (lsp_skipped); the unified walk checks its thread CPU time every
1024 nodes against six budgets and stops there, keeping what it found
(walk_truncated, implies lsp_skipped); a file whose parse plus walk spent
the budget is skipped by the LSP walks as well. Each decision is logged
with the path (extract.lsp.skipped, extract.walk.truncated). C# extraction
355 -> 39 s; the whole index 463 -> 128 s wall and 27 -> 11 GB peak
against v0.10.8 with 299 more nodes. The walk cuts exactly the five JIT
stress files (hugeexpr1, hugeSimpleExpr1, HugeArray1, HugeField1/2); the
15 MB System.Runtime.Intrinsics reference file walks to the end and, with
four generic-nesting JIT regression tests, only skips the LSP walk under
the parse-plus-walk rule.
The crash behind the C# bench. dotnet/runtime is 42,555 files, and its
resolve phase died with SIGBUS on every run of this branch. A libc-backed
ASan build of the server named it: heap-use-after-free in c_adl_resolve,
reading a type name that another worker had allocated in its per-file
scratch arena and freed at the end of its file. The C++ class walk
refines a method's return type when the declaration is more specific
than the pre-registered one (NAMED -> pointer/reference/template), and
it did that by casting the chained lookup result to non-const and
writing a scratch-arena signature into it -- into the sealed shared base
when the entry lived there. The overlay contract had exactly one
bypass, and it was this cast; every lookup returns const, so a grep for
the cast is the audit. The refinement now goes through
cbm_registry_func_for_update: copy-on-write into the overlay, the base
untouched. Test: clsp_method_return_refinement_is_copy_on_write (a
sealed base with a NAMED method return, an in-class pointer declaration;
the base signature pointer is unchanged and the overlay copy carries the
refinement plus a marker only a copy has) -- RED on the cast, GREEN on
the accessor. The fixed ASan build then indexed the whole
corpus clean: 42,555 files, 1,224,981 nodes, 5,794,320 edges, worker
exit 0, no report.
Production build backs the core with mimalloc explicitly (never the global
override, which macOS's two-level namespace forbids); the test build keeps
libc so the sanitizers see every block -- the same split the tree-sitter and
SQLite bindings already use.
PROOF
Go peak RSS 16.7 -> 4.0-5.2 GB; result arenas 14,804/28,817 -> 856/856
MB; wall 55 -> 52-60 s; graph inside the pre-existing run-to-run
jitter (gRPC Route nodes: two identical runs differ by 21).
Go CBM_MEM_SPILL=1: peak RSS 1.7-2.3 GB, 21,875 results parked (950 MB
on disk), graph identical: 39 infra routes, 158 INFRA_MAPS, 21,875
surface rows, nodes/edges inside the Route jitter.
Go CBM_MEM_BUDGET_MB=1200: spill latched at 1,204 MB, the first sweep
took the charge to 768 MB, the run completed at 1.99 GB peak RSS.
Kernel DEFAULT budget (24,576 MB): aborted before, completes now without
spilling. Extraction 35.2 -> 16.5 GB RSS (5.6 GB footprint, 15.3 GB
committed); results 8.9 GB = capacity; resolve tracked 24.6 -> 19.1
GB; peak RSS 27.3 -> 20.1 GB. Nodes identical (8,529,729); edges
inside the +-400 run-to-run band the kernel shows between identical
runs.
Kernel CBM_MEM_BUDGET_MB=16000: aborted before, completes now through the
spill path. Latched at 17.2 GB charged; 89,731 results parked (9.3
GB on disk, 370,188 loads); extraction end 6.2 GB RSS / 5.1 GB
committed; resolve 8.6 GB RSS / 11.5 GB committed; peak RSS 16.6
GB (the semantic pass plateau, 13.3 GB RSS, is graph-derived and
outside what spill can move). Nodes identical; edges 15,773,598.
Kernel CBM_MEM_BUDGET_MB=15000, before the semantic batching: completes,
every phase mark under 15,000 MB by every metric, but the process
RSS high-water mark reached 16.9 GB inside the semantic pass (class
peak 4.9 GB against 0.8 GB live).
Kernel CBM_MEM_BUDGET_MB=15000, with it: completes; the charged footprint
stays under 15,000 MB through the whole semantic pass (14,018 ->
14,381 MB across its sub-phases, 6 batches of 125,644 functions);
every phase mark is under budget by every metric (max footprint
12.6 GB, max commit 11.5 GB); the process RSS high-water mark is
15.65 GB and is set in extraction, at the moment the gate latches
(15,002 MB charged) and before the first sweep lands -- a 4%
overshoot of in-flight work, no longer the semantic pass. Nodes
identical, edges 15,772,913, zero foreign blocks.
Kernel CBM_MEM_BUDGET_MB=15000, with the anticipatory latch: spill mode
enters at 14,062 MB (near_budget); the charged footprint never
exceeds 11.8 GB at any mark; the process RSS high-water mark is
15.1 GB in extraction and 15.3 GB during the final write-out --
1-2% over, the in-flight window and the dump transient. Nodes
identical, edges 15,772,543, zero foreign blocks.
Go CBM_MEM_BUDGET_MB=1200 with it: latched at 1,131 MB, extraction
peak 998 MB RSS, graph identical to the baseline.
BENCH AGAINST THE SHIPPED v0.10.8 (same machine, same driver, 2026-09-14)
corpus wall s peak RSS GB nodes / edges / CALLS
perl 5.7 -> 5.3 0.09 -> 0.05 -3 / -130 / -3
php 6.3 -> 5.8 0.66 -> 0.20 +21.9% / +1.4% / +23.8%
rust 6.8 -> 6.3 1.15 -> 0.67 +1 / +56 / -369
c 5.9 -> 5.7 0.16 -> 0.19 identical
kotlin 8.9 -> 9.4 1.55 -> 0.96 +33.5% / -22.8% / +12.1%
django 10.9 -> 10.2 3.13 -> 1.12 +6 / -14.5% / -28.4%
go 52.6 -> 42.9 17.2 -> 3.9 +14.2% / -10.3% / +2.2%
java 79.8 -> 73.8 27.6 -> 11.0 -121 / -1.9% / -0.6%
csharp 463.3 -> 128.1 27.0 -> 11.2 +299 / -0.9% / -0.6%
typescript 33.4 -> 33.6 12.0 -> 2.5 -186 / -3.5% / +130
kernel 294.8 -> 233.3 27.5 -> 25.1 -171 / -1.3% / -0.1%
No corpus is slower beyond noise; CPU time follows wall (kernel 2,442 ->
1,403 s). Every graph delta against v0.10.8 is main between the release and
the branch base (
|
||
|
|
527016258e |
fix(extract): budget tree-sitter parses by thread CPU time, not wall clock
The per-file tree-sitter parse was bounded by a 5s WALL-CLOCK budget. Under CI
contention a worker thread can be descheduled long enough for 5s of wall time
to elapse against near-zero CPU, so a small, perfectly parseable file was
abandoned with a false "parse timeout" and yielded ZERO defs. When that file
defined a symbol other files resolve against (e.g. celery.app.Celery.send_task
backing an ASYNC_CALLS edge, or a userpb gRPC client backing GRPC_CALLS), the
cross-file service edge then failed to classify -- surfacing as the flaky
convergence_probe / edge_types_probe service-edge assertions (count 0), a
verdict decided by wall-clock scheduling rather than the input (violates O9,
"no timeout decides a test").
Gate the parse on per-thread CPU time instead, with a generous wall ceiling as
a backstop:
- New portable helper cbm_thread_cpu_time_ns() (compat.c/.h): POSIX/macOS
clock_gettime(CLOCK_THREAD_CPUTIME_ID), Windows GetThreadTimes (kernel+user);
returns 0 on clock failure, degrading safely to the wall ceiling. It is NOT
cbm_clock_gettime, which is wall on every platform (QueryPerformanceCounter
on Windows).
- The progress callback's payload becomes {cpu_deadline_ns, wall_ceiling_ns};
it trips when this thread's CPU time passes the budget OR wall time passes
t0 + budget * 12 (~60s for the 5s budget). Descheduling burns wall, not CPU,
so a starved-but-parseable file survives; a CPU-spinning parse still trips
the CPU deadline; a genuinely stuck parse still trips the wall ceiling
(bounded, no hang).
RED-repro (deterministic, CBM_ENABLE_TEST_SEAMS only): a thread-local wall-clock
offset armed by CBM_TEST_WALL_STALL_ON emulates a descheduled worker (wall
reads budget + 1s ahead while CPU is untouched -- above the old 1x wall budget,
below the ceiling). async_calls_celery_wall_stall_seam arms it for the Celery
DEFINITION file and asserts ASYNC_CALLS >= 1. With the fix the parse completes
and the edge resolves; with the budget reverted to wall-only + a 1x ceiling the
def file is dropped and the edge count is 0.
Verified (macOS, sanitized): edge_types_probe + convergence_probe 107 passed /
0 failed with the fix; 58 passed / 1 failed on revert (only the seam test, at
the ASYNC_CALLS assertion); 107 restored. Regression smoke (extraction
parse_coverage pipeline lang_contract grammar_regression) 698 passed / 0
failed. lint-ci clean (cppcheck + clang-format Homebrew LLVM + NOLINT); the
Windows GetThreadTimes path is compile-checked by CI (not runnable on macOS).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
|
5b8217d43f |
Merge pull request #1759 from Jumaga2015/fix/1692-csharp-multi-attribute-decorates
fix(extract): scan every attribute_list sibling, not just the first (#1692) |
||
|
|
e5c17de621 |
fix(coverage): stop reporting a duplicate range and a line past EOF (#963)
scripts/setup-windows.ps1 has 326 lines and its parse-coverage report read
"113-113,113-113,245-327" — the same line named twice, and an end line that
does not exist. Two separate faults, both in cbm_error_regions_push.
Past-EOF end line. A tree-sitter node that ends at column 0 stopped right
after the previous line's newline, so it holds no text on the row it points
at. Adding 1 to that row named a line past the end of the file whenever the
region ran to EOF. The node here is start=(244,2) end=(326,0). Clamp the end
to the row above when the end column is 0 and the node spans more than one
row.
Duplicate range. Line 113 carries two separate ERROR nodes, at columns 25-29
and 31-32, and each pushed its own range. A line range says nothing new the
second time. Drop a range that exactly repeats the one already open. The
drop runs BEFORE the cap check, so a repeat is never miscounted as a range
the cap threw away.
Only an EXACT repeat is dropped, never a range that merely overlaps. Each
range is judged separately afterwards by cbm_region_is_recovered, which asks
whether definitions starting inside that range cover it. Two ranges holding
the same numbers always get the same verdict, so dropping the repeat changes
nothing. Two different ranges do not. Merging 3-3 into 2-3 hands the wider
range's covering definition to an error that definition does not explain, and
a real parse failure then vanishes from the report.
That is not hypothetical. An earlier version of this commit merged on overlap
and broke perl_malformed_source_remains_partial_issue1838, the test added
with the Perl grammar refresh in
|
||
|
|
fbe9eae11c | Merge branch 'main' into fix/1692-csharp-multi-attribute-decorates | ||
|
|
aa44c28ea5 |
Merge pull request #1941 from CaptainMittens/fix/parse-coverage-honest-ranges
fix(coverage): narrow parse-error ranges, and stop the report hiding what it dropped (#963) |
||
|
|
b04f54506a |
perf(coverage): find a line by table lookup, not by rescanning the file (#963)
cbm_refine_regions_with_pp_lines asks cbm_line_is_toplevel_macro_call about every preprocessor-parsed line in a region. That check walked the source from byte 0 to find the line, so the refinement cost bytes times lines — on exactly the whole-file-ERROR C shape the refinement exists to narrow. Build one line-offset table per file, and hand the byte span to the macro check. The line-based form stays for the two callers that ask about a whole region, where a single walk is cheaper than a table. A file with no table (the allocation failed) still gets the old walk, so the answer never changes. Measured on one C file whose file-scope brace is split by an #ifndef, padded to 5k / 10k / 20k / 40k lines. Release build, one machine, seconds: lines main branch this commit 5000 0.98 1.14 0.98 10000 3.30 4.02 3.41 20000 11.64 14.77 12.56 40000 44.04 58.58 46.90 The branch ran 33% over main at 40k lines and now runs 6.5% over. The reported ranges do not change: 8-8, where main reports the whole file as 5-40000. A profile of the 40k run no longer reaches the macro check. What is left is cbm_line_contains and cbm_span_contains_callable_def, which main carries too and this branch does not touch. The fixture holds 40k definitions, which makes those two cost more here than on a smaller file. parse_coverage, extraction, pipeline, mcp and index_resilience: 964 passed, 4 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Joshua Richter <jrichter5781@gmail.com> |
||
|
|
219b348263 |
test(coverage): pin the truncation marker, the partial ceiling and the grammar limit (#963)
Phase 5. Four test groups, each checked RED before it was kept. - The Studio Export range join puts ONE ",+<N>" marker at the end with the summed drop count. A marker left mid-string makes every reader stop there and silently lose the ranges after it. Reaching the join through the pipeline needs an export file with 256+ error regions across two <Class> elements, so it goes through a test seam, following the pattern already in this repo (CBM_COVERAGE_MARKER_TEST_API). - check_index_coverage emits every range in front of a marker, never turns the marker's digits into a range, and reports "truncated" from BOTH caps — the producer's and its own 256 limit. - test_index_resilience now has a ceiling beside its floor: exactly one of the two fixture files is flagged, the clean neighbour is absent, and the range does not cover the whole file. - The three _Thread_local forms are pinned as measured. Only the array form fails today; the plan's Phase 0 also listed the pointer form, and that is wrong on the grammar shipped now. Also fixes 13 clang-format violations the earlier commits on this branch left in cbm.c, mcp.c and pass_definitions.c. `make -f Makefile.cbm lint-format` would have failed CI. The changes are whitespace only — the two reflowed tool descriptions concatenate byte-identically, so no output moved. Full suite: 7735 passed, 28 failed, 7 skipped. The 28 are the pre-existing cli install/uninstall failures, identical at clean HEAD. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Joshua Richter <jrichter5781@gmail.com> |
||
|
|
a953509275 |
fix(coverage): stop the report hiding what it dropped, and name whole-file failures (#963)
Two silent failures in the parse-coverage report, both made visible by the
Phase 2 range refinement that came before this.
Two caps sat in series and both returned early without saying anything:
CBM_MAX_ERROR_REGIONS = 64 internal/cbm/cbm.c
COVERAGE_RANGE_MAX = 128 src/mcp/mcp.c
Raising only the first would have moved the clip from 64 to 128, so both move
to 256. This was live behaviour, not a theoretical limit: after Phase 2 split
one whole-file range into many small ones, src/cli/cli.c and tests/test_cli.c
both reported exactly 64 ranges — the cap binding, dead-on, twice. Every
coverage figure measured before this change was a floor. With the cap at 256
the true numbers are cli.c 13.9% (not 9.8%) and test_cli.c 3.1%, and the
longest list in the repo is 85 ranges.
A raised cap is still a cap, so the report now says when it clipped:
- cbm_error_regions_t gained a `dropped` counter, and cbm_collect_error_regions
walks to the end instead of stopping at the cap, so the count is exact rather
than a lower bound. That costs little — the walk never descends into an ERROR
subtree.
- cbm_error_ranges_str appends ",+<N>" when N ranges were thrown away.
- coverage_add_ranges reads that marker and sets "truncated": true, and also
sets it when its own limit stops the loop. Before this the marker was
invisible: the parser stopped at the '+' with no error and no leftover, so a
clipped list arrived looking complete.
- objectscript_export_append_error_ranges strips markers off both operands
before joining two Studio Export parts and adds one back at the end. A marker
left mid-string would make every reader stop there and silently lose every
range after it.
"Look at lines 1 to 13047" of a 13046-line file tells a reader nothing. Those
files now carry their own kind rather than being described as partially
covered.
New `parse_unusable` field in CBMFileResult, set when one range covers 80% or
more of the file. Its customers are non-C languages: the Phase 2 refinement
that narrows a whole-file range using the preprocessed parse only runs for C,
C++ and CUDA, so a Python, Java, Ruby or TypeScript file whose root node is
ERROR still reports 1-N. Verified against real files in all four.
The kind is `parse_unusable`, not `parse_failed`. index_coverage.kind already
means one of two things — indexed-but-partial, or a skip phase saying the file
was never indexed at all — and `parse_failed` reads as the second when it is
the first. The store.c schema comment, which is the only written record of this
vocabulary, now describes all three classes and says why.
Two places would have mislabelled the new kind as "skipped", which is exactly
that confusion: coverage_status fell through to its catch-all pass, and
add_coverage_report fell into its else branch. A reader who finds a file under
"skipped" believes it is absent from the graph, when it was indexed. Both now
have explicit branches. index_status gained parse_unusable_count so a CI gate
can read it without parsing anything else, get_code_snippet says "read the
source directly" instead of naming useless ranges, and the three tool
descriptions that listed two coverage kinds now list three.
Seven added. The cap test moved from 64 to 256; a new test asserts the marker
carries a real drop count and that nothing follows it; an inverse test asserts
an under-cap file carries no marker at all. For the new kind: a Python file
whose root is ERROR is unusable, a file with a local parse failure stays
partial, a clean file is neither, and — the one that matters most — the
that last one ever flips, the Phase 2 refinement has stopped working.
Full suite: 7732 passed, 28 failed, 7 skipped. The 28 are pre-existing
agent-client install/uninstall failures in the cli suite, identical in count
and identity at clean HEAD.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
|
||
|
|
cb167b9dcc |
fix(coverage): narrow parse-error ranges with the preprocessed parse (#963)
src/cli/cli.c reported an error range of 1-13047 — the whole file. The file indexed fine; the report was wrong. Three #ifndef _WIN32 blocks split a brace (two `if` headers, one closing brace), so the raw tree-sitter parse cannot resync at file scope, the root node becomes ERROR, and cbm.c takes its whole-file branch. The pipeline already parses these files a second time after preprocessing, and that parse is clean. The report just never consulted it. Build one byte per original line from the preprocessed pass, then cut each raw error range down to the runs of lines the second parse could not vouch for. Three rules, all found by running it and all load-bearing: - An expanded line only vouches for its original line when it HAS TEXT. The preprocessor emits a blank line where it dropped a branch; treating that blank as proof suppressed every C range in the suite. - Preprocessor directive lines (with backslash continuations) never count as missing code — the preprocessor consumes them, so the second parse can never vouch for one. Without this every #include block reported as a miss. Known cost: a #define the raw parse really dropped no longer shows up on its own. - A TOP-LEVEL macro invocation line never counts as vouched-for even when the expanded line parses clean. The macro can expand to a whole definition that the recovery walker deliberately refuses to adopt (#949), so a clean second parse there proves nothing. An in-body invocation is the benign #1071 case and is left to the existing macro subtraction. The order of the three coverage steps is now settled by where each one's evidence lives: recovery subtraction -> before the refinement; its evidence is a whole definition that STARTS inside the range, so it must be asked while the range still matches the construct the refinement -> middle #1071 macro rule -> after the refinement; its evidence is per-line, so a narrow range points at the call itself Measured on this repo: src/cli/cli.c goes from one whole-file range to 64 ranges over ~9.8% of the file, tests/test_cli.c from 48.6% to ~2.9%, src/cli/activation_transaction.c from 38% to 7.5%. What survives is honest — the biggest remaining ranges in cli.c are genuinely discarded #ifdef _WIN32 and #ifdef CBM_CLI_ENABLE_TEST_API blocks, absent from the graph on this platform. Both percentages above are floors, not measurements: cli.c and test_cli.c now land on exactly 64 ranges, which is CBM_MAX_ERROR_REGIONS. That cap drops regions with no signal, and a follow-up raises it and adds a truncation marker. Five tests, all red before the change: the range narrows to the dropped branch; lines the preprocessor explained are excluded; a range never starts or ends on a directive; real garbage beside a split brace stays flagged; a clean file stays unflagged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Joshua Richter <jrichter5781@gmail.com> |
||
|
|
4f82a1e56f | Merge branch 'main' into fix/svelte-html-astro-embedded-structure | ||
|
|
683d4bc312 |
Merge branch 'main' into fix/1692-csharp-multi-attribute-decorates
Brings the branch onto current main (
|
||
|
|
ffb5ebe9e8 |
Merge pull request #1721 from henry-hz/upstream-pr/elixir-call-arguments
fix(elixir): populate first_string_arg for Elixir calls |
||
|
|
dc703dad4a | Merge branch 'main' into upstream-pr/elixir-call-arguments | ||
|
|
2efb7a6323 | Merge branch 'main' into fix/ruby-superclass-base-classes | ||
|
|
24f54c0eb6 |
fix(extract): extract structure from Svelte/HTML/Astro inline scripts
The Vue change (#1852) built the shared embedded-script seam — the included-ranges parse over the original buffer and the lang=/src= resolver — but gated structural extraction on the host being Vue. A Svelte component, an HTML page or an Astro file therefore still yielded only File + Module nodes and IMPORTS edges from its <script> blocks: no Function definitions, no CALLS, and a false dead-code verdict for anything used only from a component. Widen the seam to every host that declares an embedded spec: - Structural extraction (definitions + calls) runs for every embedded block; the per-host boolean is gone. - The Vue attribute resolver becomes script_embedded_language() and runs for all four markup hosts. It starts from the spec row's language instead of hard-coding JavaScript, and learns HTML's type= attribute: absent or empty, module, text/javascript and application/javascript pass; application/json, importmap, text/x-template and the like bail, so data blocks cannot leak garbage definitions into the graph. src= and unsupported lang= values bail as before. CFML's <cfscript> carries no attributes and keeps its spec-fixed language. - Astro's frontmatter fence and <script> bodies default to TypeScript: Astro type-checks both as TS with no lang= attribute to say so, and the fence cannot carry one. The TypeScript grammar parses untyped JavaScript unchanged. Tests: the #1852 control that pinned Svelte/HTML/Astro at zero functions and zero calls now asserts the real symbols; stay-at-zero controls cover type="application/json" and <script src> in every host; both Svelte blocks (context="module", and Svelte 5's bare module) contribute in host-file coordinates; HTML's accepted type= forms and Astro's TS frontmatter are covered; an end-to-end CALLS edge inside a .svelte file is asserted. The Svelte and Astro label goldens gain the Variable their fixtures always declared. Verified on a 213-component Vue corpus: node counts per label and per host language, edge counts and trace_path output are identical before and after. Fixes #1807 Signed-off-by: Brian McNaboe <bmcnaboe@gmail.com> |
||
|
|
380b1c5af1 | Merge branch 'main' into feat/cshtml-razor-pages | ||
|
|
5802ccdd77 |
Merge pull request #1896 from CaptainMittens/fix/swift-call-arguments
fix(extraction): reach Swift call arguments through call_suffix |
||
|
|
1770d68a2c |
feat(extract): discover Razor Pages / MVC views (.cshtml) and route their @page
Follow-up to #1824, which mapped Blazor `.razor` to the C# host and extracted `@page` routes from it. `.cshtml` is the same Razor syntax with the same C# host and was equally unmapped: an ASP.NET Core MVC or Razor Pages application produced no nodes at all for its entire view tree. `@page` matters more here than it does for components. On a Blazor component the directive is an optional annotation; in Razor Pages it is the directive that *defines* a page, so the routable surface of a Razor Pages app lives exclusively in this file type. That is why the extension mapping alone is not enough. `cbm_path_is_razor()` hardcoded a single `.razor` suffix, so a `.cshtml` file would have been discovered and parsed but skipped by the directive scan -- present in the graph, with none of its routes. Widened to a suffix table covering both. Scope is deliberately the Razor family and nothing else: - `.vbhtml` is excluded because there is no VB.NET language in this repo, so there is no host to map it to. - `.aspx` / `.ascx` are excluded because Web Forms is a different templating syntax (`<%@ %>`, `runat="server"`) with no `@page` directive, so neither the C# error recovery nor the directive scan applies. Best-effort on the same terms as `.razor`: the C# grammar recovers the `@{ }` and `@functions` blocks, and the surrounding markup lands in ERROR regions reported through parse_partial. Tests: the extension mapping; a Razor Page asserting the extracted route path and method; and a negative case for a layout with no `@page`, since most `.cshtml` files are layouts and partials and a spurious Route node per view would be worse than the missing ones this fixes. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
|
6cca768816 |
Merge pull request #1773 from rarepops/fix/dockerfile-trailing-blank-eof-1746
fix(extraction): treat trailing blanks at EOF as an absent newline, not a miss (#1746) |
||
|
|
cc3e2638da |
Merge pull request #1912 from DeusData/feat/python-bare-local-binding
fix(python): suppress bare calls shadowed by an enclosing parameter |
||
|
|
db2f104a1f |
Merge pull request #2013 from shafty023/fix/1997-extract-scratch-arena
fix(extract): AST traversal stacks no longer outlive the file they walk |
||
|
|
c4e902849f |
fix(extraction): reach Swift call arguments through call_suffix
handle_calls reads a call's arguments through one tree-sitter field lookup, ts_node_child_by_field_name(node, "arguments"). The vendored Swift grammar declares no "arguments" field at all -- its own ts_field_names[] table has zero occurrences, against one in Go and two in TypeScript. Swift models a call as a target expression plus a call_suffix, and the arguments hang off the suffix as value_arguments. So args was null for every Swift call ever parsed, first_string_arg was never populated, and no Swift HTTP call could raise an HTTP_CALLS edge or a Route node. Alamofire, Moya and URLSession have been in the service-pattern table the whole time and match the callee text correctly; the URL simply never arrived. Three changes, all in extract_calls.c: - swift_call_args() reaches the argument list through call_suffix, in the same shape as the existing objectscript_call_args() fallback and used from the same place. A trailing closure has no value_arguments, so it returns a null node and that call behaves as before. - extract_url_or_topic_arg() unwraps Swift's per-argument value_argument node, stepping past a leading value_argument_label. Without it, dataTask(with: "/api/v1/widgets") yields the label "with" rather than the path. PHP and C# already had the same unwrap for their own "argument" node. - is_string_like() gains line_string_literal, which is what Swift calls an ordinary "..." literal. The list already held raw_string_literal, so only Swift's common case was missing. This is the layer underneath #1892 rather than the whole of it. A literal URL argument now arrives; a literal nested inside a constructor, as in URLSession.shared.data(from: URL(string: "...")!), still does not, because extract_positional_url reads a literal, a template string, a concatenation or a named constant and that shape is none of them. Reproduce-first: all three tests fail without the fix, two on a null first_string_arg and one on HTTP_CALLS being 0. Fixes #1892 Signed-off-by: Joshua Richter <jrichter5781@gmail.com> |
||
|
|
192f29609a |
Merge pull request #2006 from ilyabrykau-orca/fix/go-field-ref-member-signal
fix(pipeline): key the Go Field guard on the recorded selector shape, not reference text |
||
|
|
01bf5f23e3 |
Merge pull request #2009 from CaptainMittens/fix/route-handler-last-argument
fix(routes): take a route's handler from the last argument, not the first |
||
|
|
e6a34fcea0 |
Merge pull request #1824 from The1nk/feat/blazor-razor-routes
feat(extract): discover Blazor .razor components and emit Route nodes for @page |
||
|
|
620613ede6 |
Merge pull request #1986 from CaptainMittens/fix/swift-scanner-try-bang-shift
fix(grammar): stop the Swift try-bang entry shifting past its own type |
||
|
|
572725e61a |
fix(extraction): treat trailing blanks at EOF as an absent newline, not a miss
A file whose final line lacks its newline leaves the grammar's mandatory terminator MISSING at EOF. #1610 suppressed that phantom parse_partial on the grounds that the node is ZERO-WIDTH: the parser consumed no source for it, so by construction nothing was dropped. That suppression tested `end == source_len` exactly. Trailing blanks are extras owned by no node, so `ENTRYPOINT ["a"] ` + EOF parks the zero-width terminator one byte short of source_len and the check missed it. This is why neither a trailing blank nor an absent final newline flagged on its own -- only the pair did, exactly as the reporter's byte-exact control matrix showed. Treat "at EOF" as EOF modulo a trailing blank run. Every blank except newline qualifies, deliberately not a hand-picked subset: a form feed is no more content than a space, and #1610 exists precisely because whether a file got flagged used to hinge on such incidentals. Newline stays excluded because a terminated final line produces no MISSING terminator at all. The rule remains ZERO-WIDTH ONLY. A width-bearing MISSING/ERROR at EOF is a genuine loss and is still flagged (a Makefile's unterminated final recipe really does vanish), as is any failure earlier in the file; both are pinned by guard tests alongside the existing #1610 guards. Verified end to end: the issue's own `cli index_repository` reproduction goes from parse_partial_count 1 to 0 with node count unchanged, so the phantom flag is gone without anything dropping out of the graph. Fixes #1746 Signed-off-by: Rares Popa <2606875+rarepops@users.noreply.github.com> |
||
|
|
40f2722d61 |
fix(extract): AST traversal stacks no longer outlive the file they walk
Fixes #2010. Refs #1997. Indexing a 14k-file TypeScript repository holds 13.3 GB resident at post_extract on Linux aarch64, and about 3.1 GB of it is AST traversal scratch that no CBMFileResult points at. ts_nstack_init cut its stacks from ctx->arena, which is result->arena. In the parallel path that arena is stored into ec->result_cache[file_idx] (pass_parallel.c:963) and freed only after parallel_resolve and the infra passes (pipeline.c:1324), so a structure written for a one-file lifetime is in fact held for the whole index, once per file, concurrently. The header claimed otherwise and was wrong: "freed when the arena is destroyed at end of file extraction". The largest instance is channels. CHAN_STACK_CAP is 4096 and sizeof(TSNode) is 32, so scan_string_consts_js (extract_channels.c:104) and extract_channels_js (:376) take 128 KB each, per JavaScript or TypeScript file. Across 12,673 such files that is 3,168 MB, and it produced 23 Channel nodes out of 164,684. CBMExtractCtx gains a scratch arena, created and destroyed by the cbm_extract_file_ex call that builds the context. Nothing else changes about where anything lives: ctx->arena still owns every string a CBMFileResult points at, and ctx->scratch owns only traversal stacks, which nothing points at. Nothing a CBMFileResult points at moved. A context built without a scratch, and a file whose scratch arena fails to allocate, both fall back to ctx->arena, which is exactly the behaviour that shipped before this change, so no path can be handed a NULL arena and silently lose nodes. The signature change is what makes the mistake unrepresentable. ts_nstack_init takes the context rather than an arena, so handing it ctx->arena is an incompatible pointer type; push and push_children read the arena the stack recorded, so a stale argument is an arity error. Both fail under -Werror rather than silently retaining memory. The 55 init sites are NOT one uniform sed. 29 pass ctx->arena and 26 pass a local alias, and that alias is shared with result allocations: in parse_zig_imports (extract_imports.c:1093) the same `a` feeds cbm_node_text, strip_quotes, path_last and cbm_imports_push (extract_imports.c:1101, 1106, 1108, 1109). Rebinding it would have moved import names and module paths into memory that dies with the call while the result kept pointing at them. Each site was edited on its own and no alias was rebound. Re-derive the set with: grep -rn 'ts_nstack_init' internal/cbm/extract_*.c grep -rn 'ts_nstack_push' internal/cbm/extract_*.c parse_lisp_imports used its alias only for the stack, so the alias goes too. push_nested_class_nodes and push_class_body_children took an arena purely to reach the stack, so they take the context now; the one call site passes ctx. Two measurements, from two different builds, and they are not the same number. A macOS census build, which instruments every cbm_arena_alloc and sums the blocks behind every live CBMFileResult arena, reports the retained per-file arena set dropping from 11,226 MB mapped to 5,680 MB, and blocks from 48,045 to 34,186. Those are mapped bytes, not resident: that build logs mem.allocator.bound_populations_only, so untouched block tails never commit. The resident figures come from Linux aarch64, gcc 13.3, static build, where the build logs mem.allocator.owned classes=all and mapped equals resident. Five runs per side on the full 13,999-file tree: mem.collect post_extract rss_mb 13293 -> 6752 (-49.2%), mem.phase peak_mb 14840 -> 8565 (-42.3%). nodes 164684, parse_partial 280 and registry entries/defines/imports 787954/804999/100027 identical on all ten runs; edges 835869-835874 against 835867-835881, overlapping ranges, so run to run noise. Wall clock means 35651 ms against 30502 ms, but the ranges overlap and the host had other load, so the only supportable claim is no regression. ObjectScript Studio Export on a small fixture: nodes 18, edges 25, both sides. Scratch block size is 512 KB, chosen from measurement. arena_grow fires on one file in 12,000 at both 512 KB and 1 MB, and constantly at 256 KB, where the two channel stacks alone are exactly 262144 bytes. 512 KB therefore costs half the resident block per worker for the same growth behaviour, and stays under MI_LARGE_MAX_OBJ_SIZE in the vendored mimalloc, which is MI_LARGE_PAGE_SIZE/8, exactly 524288 bytes (types.h:426, with MI_ENABLE_LARGE_PAGES defaulting to 1 at :115 and not overridden by this build), so a 512 KB block is still bin-allocated from a 4 MiB large page. Growth is not free at that size for the same reason: arena_grow doubles to 1 MiB, above that bound and therefore a singleton OS allocation. One file in twelve thousand pays it, which is why it is accepted. The largest single file's scratch high water mark on this corpus is 33.5 MB, now transient rather than retained. Tests. Nothing in the suite asserted on result->arena size, block count or total_alloc after cbm_extract_file, which is why this went unnoticed for the life of extract_node_stack.h. traversal_stack_not_in_result_arena_issue2010 pins the byte budget: extracting "export const x = 1;" as TypeScript charged 365984 bytes to the result arena before and charges 87456 after, a difference of 278528, exactly the two 4096-entry channel walks plus the 512-entry ES import walk at 32 bytes per TSNode. Of the 87456 that remain, 7680 is the defs item array at GROW_ARRAY's starting capacity of 32 times sizeof(CBMDefinition) 240 and the other 79776 is everything else the extraction interns, none of it traversal scratch; the 128 KB bound is therefore above the new figure with room and a factor of four below the old one. extract_traversal_stacks_come_from_ctx_scratch_issue2010 in test_mem.c pins the lifetime, since a byte budget alone would also be satisfied by shrinking CHAN_STACK_CAP: it builds the extraction context directly over two arenas it owns, runs cbm_extract_channels, and asserts the scratch took at least the 262144 bytes of the two walks while the result arena did not. extract_c_macro_hidden_call_survives_preprocessed_pass_issue2010 is not a scratch test. pp_ctx carries ctx->scratch so every context in the file is uniform, but nothing reads it there: pp_ctx reaches only cbm_extract_unified and cbm_run_c_lsp, and neither extract_unified.c nor anything under internal/cbm/lsp/ includes extract_node_stack.h, so no traversal stack is built on that path today. The test guards the macro-expansion path itself, which had no assertion on a call that exists only after expansion. ObjectScript composite extraction is already covered by pipeline_objectscript_export_preserves_calls_sequential_parallel and iris_export_xml_multi_class, so no new test is added there; a per-call arena is correct on that path by construction, since each call owns and destroys its own. Local: scripts/test.sh, 7558 passed, 0 failed, 8 skipped, 139 suites. Alternatives considered Thread-local scratch arena, reset per file, with a depth guard for nesting. Rejected on three counts. This codebase never uses thread-local storage for owned bulk memory; every existing _Thread_local here is a small cache, a guard or a log buffer, and every per-worker resource is a field in a context struct. It would have needed cbm_arena_reset, which has no production caller and a documented subtlety with grown block sizes (arena.c:213-217). And extraction threads are created per cbm_parallel_for call and joined (worker_pool.c:71,88) with no thread-exit hook in this project's own code, so the arena would have leaked once per thread per dispatch and turned the leak lane red. Save and restore mark and release on the arena. Rejected because a per-call arena already makes nesting correct by construction, and a mark type plus two functions would have to be added to both arena headers. Per-walk arenas, one per traversal rather than one per file. Rejected on cost: 55 sites would each need an explicit span, for the same bound. Right-sizing the initial capacities alone. CHAN_STACK_CAP at 256 rather than 4096 would cut those two sites from 3,168 MB to about 198 MB, which is real, but it tunes the constant while the lifetime defect remains at all 55 sites, and growth is not free: 256 doubling to 4096 abandons 122,880 bytes, and one file in this corpus took 33.5 MB of stacks. The cap and the scratch block size are coupled, which is a further reason to change one at a time. Worth doing separately. Routing to a scratch inside extract_node_stack.h while keeping the arena parameters. Rejected because a parameter that is ignored lies about the API, and the alias hazard above would stay a review obligation instead of a compile error. Freeing the per-file result arena earlier. Rejected: the results are read by parallel_resolve and the infra passes. That is the retention half #1925 addresses, and this change is independent of it. Signed-off-by: Daniel Ochoa <daniel.ochoa@closedloop.ai> |
||
|
|
97517a4652 |
fix(python): carry bare-call parameter bindings in the walk
Replaces the cursor ascent from
|
||
|
|
349aecba95 |
fix(pipeline): key the Go Field guard on the recorded selector shape, not reference text
cbm_go_suppress_bare_field_ref dropped a Field-targeted reference when
strchr(ref_name, '.') == NULL — but the extractor strips the receiver on
every path that reaches the resolver (resolve_lhs_write_name records the
trailing field name of a selector LHS; is_reference_node records the
inner field_identifier), so the dot test could never be false for Go.
The guard was a blanket veto: all ~4588 Go Field nodes recovered by
one
|
||
|
|
592894a438 |
fix(routes): take a route's handler from the last argument, not the first
Express, Fastify, gin and Laravel all put middleware between the route
path and the handler, and the handler comes last:
routerGet("/users", requireAuth, rateLimit, listUsers);
extract_handler_arg returned the FIRST argument that looked like a
function reference, so requireAuth won and the HANDLES edge pointed at
the middleware instead of listUsers. That is a wrong edge, not a
missing one, and it misleads anyone tracing a request through the
graph.
The scan bound hid a second fault. An arrow function matches none of
the accepted node kinds, so three inline middlewares pushed the real
handler past MAX_HANDLER_SCAN and no handler came back at all.
The loop now examines every argument and keeps the last eligible one.
Nothing that follows a handler matches the accepted kinds either, since
an options argument is an object node, so the last match is the
handler. HANDLER_START_IDX stays, because argument 0 really is the
path. MAX_HANDLER_SCAN had no other use and is gone.
A two-argument route has one eligible argument, so first and last agree
and the existing Express, Fastify, gin and Laravel route tests are
unchanged.
Both tests fail without the change, the first reporting "requireAuth"
where "listUsers" belongs and the second reporting no handler at all.
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
|
||
|
|
c36b4fbc44 |
fix(routes): find a route path wherever it sits in a decorator's arguments
Java and Kotlin put no order on annotation attributes, so `path` can
appear after `method`, `produces` and `consumes`:
@RequestMapping(method = RequestMethod.GET,
produces = MediaType.APPLICATION_JSON_VALUE,
consumes = MediaType.APPLICATION_JSON_VALUE,
path = "/orders")
extract_route_path_from_args stopped after the third argument, so the
path was never read and no Route node formed. The endpoint was in the
source and not in the graph.
Four call sites read this function, so the miss covered plain decorator
arguments, Spring mappings, and both JAX-RS paths.
The loop now checks every argument. That matches find_drf_kwarg_in_args
a few lines below, which walks the same kind of node with no cap. Cost
stays small: the loop returns on the first path-shaped string, and each
argument's own subtree walk is still bounded by find_route_path_literal.
The recursive breadth guard inside find_route_path_literal keeps
DECORATOR_SCAN_LIMIT. That loop descends to CBM_DESCENDANT_MAX_DEPTH, so
breadth 3 by depth 6 is 729 visits at worst and unbounded breadth there
is not bounded at all.
The test fails without the change, reporting the class-level "/api"
route present and "/api/orders" missing.
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
|
||
|
|
fd73c347fb |
feat(extract): emit Route nodes for Blazor @page directives
get_architecture's routes aspect returned nothing for a Blazor application: the entry surface of the app was invisible in the graph even though every @page directive names one. @page lives in markup above the @code block, which tree-sitter's C# grammar never parses, so there is no AST node to read it from. Scan the raw source for it instead. The match is deliberately strict — the directive must be the first token on its line, followed by whitespace and a double-quoted path beginning with '/' — so @pageSize and prose mentions cannot match. The route is attached to the file's module def. A .razor component's class is implicit, so there is no class node to carry it, and the module QN already is the component's identity. insert_def_into_gbuf is label-agnostic and creates Route + HANDLES from route_path. pass_route_nodes.c's ensure_decorator_routes gains "Module" for the same reason. Extraction covers the full-index path on its own; this backstop is what runs on an incremental re-index, so without it a component's Route would appear on a full index and vanish the next time that one file changed. Its loop bound now derives from the labels array rather than borrowing the unrelated RN_STRIP_PASSES, so adding a label cannot silently skip it. Verified no double-creation when both paths can fire. Measured on a real Blazor application (36 .razor files): routes went from 0 to 11, with 11 matching HANDLES edges, and no change to the 363 Class / 1775 Method / 3805 CALLS already extracted from its C#. Covered by two new tests in tests/test_extraction.c. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: The1nk <23087600+The1nk@users.noreply.github.com> |
||
|
|
e820409233 |
docs(vendored): record the Swift scanner shift patch in MANIFEST.md
The vendoring table claims our copy of a grammar matches the pinned upstream commit. A local patch makes that claim false unless the patch is written down, so a future re-vendor would drop the fix without anyone noticing. The row covers both changed lines, because both are the same defect in the same file: OP_SYMBOL_SUPPRESSOR at line 131 and eat_operators at line 514. The row also names the two upstream commits that already carry these changes, and the table intro gains one clause for the case. Upstream fixed eat_operators in fb63a7004f07 on 2026-04-06 (their issue #558) and the OP_SYMBOL_SUPPRESSOR entry in 6ab8d1d74ebd on 2026-08-10. Our pin 8abb3e8b3325 is from 2026-03-20 and predates both, so these are backports rather than local inventions. That distinction changes what a re-vendor should do. The table's instruction is "must re-apply these", which is right for the other four rows -- upstream never took those. For this row it would mean hand-applying a change the newer file already has. The row now says to delete it instead, and the intro allows for that case. MANIFEST.md is itself listed in scripts/vendored-checksums.txt, so its own recorded checksum moves with the edit. Refs #1892 Signed-off-by: Joshua Richter <jrichter5781@gmail.com> |
||
|
|
63f79365e3 | Merge branch 'main' into feat/python-bare-local-binding | ||
|
|
a2ea711c29 |
fix(grammar): stop the Swift try-bang entry shifting past its own type
scanner.c line 131 shifts a literal 1UL by 32 to build the entry that suppresses the bang of a try!. unsigned long is 64 bits on Linux and macOS but 32 bits on Windows, so there the shift count equals the width of the type, which is undefined. 1ULL is 64 bits on every target this project builds for. Nothing reports it today. The expression is a compile-time constant, so no sanitizer sees it run, and Makefile.cbm:719 builds vendored grammars with -w, which switches off -Wshift-count-overflow. Upstream tree-sitter-swift made the same change in 6ab8d1d74ebd, after the commit this grammar is pinned to. Refs #1892 Signed-off-by: Joshua Richter <jrichter5781@gmail.com> |
||
|
|
3f831936b5 |
fix(grammar): stop the Swift scanner shifting past the width of an int
The Swift scanner keeps a 64-bit mask of the symbols that suppress a
match -- the rule that stops `try!` emitting its `!` as a token of its
own. It tests one bit per candidate:
uint64_t suppressing_symbols = OP_SYMBOL_SUPPRESSOR[full_match];
for (uint64_t suppressor = 0; suppressor < TOKEN_COUNT; suppressor++) {
if (!(suppressing_symbols & 1 << suppressor)) {
The mask is uint64_t but the literal `1` is an int, so the shift is an
int shift. TOKEN_COUNT is larger than 32, so once suppressor reaches 31
the shift runs past the width of the type. That is undefined behavior,
and every bit above 31 is tested against a value the standard does not
define.
Nothing caught it because nothing in the tree reached the suppressor
path. Any Swift force-unwrap does: `cached!` is enough.
UBSan reports it as:
scanner.c:514:47: runtime error: left shift of 1 by 31 places
cannot be represented in type 'int'
`1ULL` makes the literal as wide as the mask it is tested against.
The new test in tests/test_extraction.c cannot go red on its own. The
normal test build prints the UBSan message and carries on, which is why
this survived. The Windows CLANGARM64 leg runs UBSan in trap mode, and
there the same shift is an illegal-instruction crash -- so the test
exists to make sure that leg keeps parsing a force-unwrap at all.
scripts/vendored-checksums.txt records the new hash for the one changed
file, as scripts/security-vendored.sh --update writes it. Layer 8 of the
security gate compares vendored content against that manifest, so the
edit and its recorded hash belong in the same commit.
Found while adding Swift URL extraction in #1892 / #1976, and split out
of that PR so the vendored change can be reviewed on its own.
Signed-off-by: Joshua Richter <jrichter5781@gmail.com>
|
||
|
|
da61c81af2 |
fix(lsp): unify the two type short-name indexes that broke the main build
main does not compile. internal/cbm/lsp/type_registry.h declares type_short_buckets, type_short_entries and type_short_bucket_count twice, and cbm_registry_types_by_short_name twice with different parameter types. Two changes implemented the same feature independently: |
||
|
|
7808686cee |
Merge pull request #1679 from astandrik/codex/perf-cpp-short-name-lookups
perf(lsp): index C++ short-name type lookups |
||
|
|
3627eff580 |
Merge pull request #1389 from Yyunozor/fix/arg-url-false-positives
fix(pipeline): reject non-HTTP slash arguments |