Replace raw malloc/free with cbm_alloc/cbm_free (CBM_MEM_CLASS_EXTRACT)
and a growable CBM_DYN_ARRAY for matches. Remove the 4KiB line-copy
cap; lines are split in place instead of copied. Remove the
DOCLINK_MAX_REFS(256) cap; the ref vector now grows, is sorted by
target id, then collapsed. Track reference kind (bare, relative,
rooted) so /x never attempts a directory join and ./x never falls
back to root. Report per-file skips via cbm_pipeline_add_file_error
and cbm_log_warn, and check cbm_pipeline_check_cancel in the per-file
loop.
Add negative-control tests for all of the above, plus CRLF handling,
fenced-code-block pinning, bare URLs, backtick-basename exactness,
and ".." escaping.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Clément Deust <clement.deust@gmail.com>
Markdown docs reference other repo files constantly (a coding-standards
doc links the modules it governs, a README points at entry points) but
none of that surfaced as edges, so fan-in queries were blind to
documentation hubs. In an A/B retrieval benchmark I ran in August on a
docs-heavy repo, the graph's top fan-in answer was off by roughly 17x:
the most-referenced file was a standards doc with 173 references and
zero inbound edges.
New pre-dump pass, pass_doclinks.c, modeled on pass_configlink.c: three
strategies emit REFERENCES_FILE edges between existing File nodes only
(unresolvable targets are dropped, the pass never invents nodes):
inline links, backtick paths, bare path mentions. Targets resolve
against the referencing file's directory and the repo root; repeated
references collapse into one edge carrying strategy, confidence and
count. Registered in the pre-dump sequence after configlink and in the
incremental post-passes; REFERENCES_FILE added to the skill's edge-type
list and the structural/language contract tests.
Tests mirror test_configlink.c (real files in a tmpdir, File nodes in a
gbuf, run the pass, assert edges): inline link, backtick, bare mention,
http/anchor ignored, anchor-suffixed file link, dedupe with count,
relative-vs-root resolution, unresolvable-target guard, NULL repo_path
skip.
A companion change adds the same linking for shell files (source lines
and script invocations); split out to keep each change reviewable.
Validation: scripts/build.sh clean; focused serial runner (doclinks,
configlink, pipeline, edge_structural, lang_contract) 340 passed under
ASan/UBSan; cppcheck clean on the new files with the repo's flags.
Signed-off-by: Clément Deust <clement.deust@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CLI_USAGE described outer --json but did not mention tool-level
--format json. Add a help-only pointer that tools which accept format
support --format tree|json, and distinguish payload JSON from the full
MCP envelope. No new flags, env vars, or output-default changes.
Fixes#2102
Signed-off-by: Lowpower <3133734+Lowpower@users.noreply.github.com>
Several processes first-starting together against a runtime directory
that does not exist yet -- test-windows-guards' section_cold_storm, or a
host that launches more than one MCP server on its very first run -- all
observe the final path component absent. One CreateDirectoryW wins; the
others get ERROR_ALREADY_EXISTS. win_private_directory_tree_secure()
treated that as a failure and recorded no validation detail, so every
loser exited with a bare
secure CLI coordination could not be created (endpoint)
The function called right after the walk, win_runtime_directory_secure(),
has always tolerated ERROR_ALREADY_EXISTS for the same directory, and the
POSIX walk tolerates EEXIST at the same point; only the Windows walk in
front of them did not.
Seen four times on 2026-09-21 on unrelated PRs (#1768 three attempts in
a row, #2140, #1999, #808). The racy lines date from July; the guard
began exercising them on 2026-09-03, when each guard section was given
its own empty CBM_RUNTIME_DIR, so the final component is now absent at
storm time on every run.
The walk now treats ERROR_ALREADY_EXISTS from its own CreateDirectoryW as
"the directory I wanted exists". Nothing is trusted because of that: an
ancestor still goes through win_directory_component_secure(), and the
final component through win_runtime_directory_secure(), which refuses a
non-directory or reparse point and enforces owner and DACL.
Every refusal on this path now names its component and its rule. The
walk reports the Windows error when it can neither create nor inspect a
component, and win_runtime_directory_secure() says whether the path
could not be created, cannot be inspected, exists but is not a directory,
or is a reparse point. One helper owns the wide-to-UTF-8 conversion for
these messages and the existing ancestor message now uses it too, so
src/daemon/ipc.c stays at its memory-core baseline.
Deterministic reproduction, no threads and no timing: a test seam fires
in the walk between "component observed absent" and CreateDirectoryW,
and the test plays the process that wins the creation.
daemon_ipc_windows_private_directory_survives_lost_creation_race
daemon_ipc_windows_private_directory_refuses_and_names_a_planted_file
Verification. macOS arm64 (ASan+UBSan): build clean, daemon_ipc 53 passed
(2 Windows-only skips), daemon_bootstrap 28 passed, lint-memory-core
unchanged (ipc.c stays at 146 raw sites). The two new tests are Windows-only
and have NOT been run locally: the Windows VM was down when this was written,
so the RED run on the seam-only tree and the GREEN run with this fix are
delegated to CI's windows-latest legs (test-windows shards, test-windows-guards)
by an explicit maintainer decision. If the guard suite's cold storm still
fails with this in place, the attribution above is wrong and this reverts.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The lint chain stops at its first red target, so a clang-format failure hid the memory-core linter
until PR CI: the coverage line-offset table, the Cypher direction swap and the search label each
added a raw allocator site. The table now comes from cbm_alloc/cbm_free (extract class), the swap
frees through safe_str_free, and the BM25 row array moves to cbm_calloc/cbm_free so mcp.c's raw-site
count goes down (baseline 814 -> 813) rather than up.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Nine findings from the 2026-09-16 graph-quality probe over the eleven bench corpora, each with a
test that fails without its fix and a before/after measurement on the real corpus.
Deterministic resolution (O9). Two indexes of one kernel tree differed in 5,821 IMPORTS and 632
CALLS edges. Three order/load dependencies carried it: the `#include` target was the first by-name
hit whose path ends with the include path — a bucket in worker merge order (include/linux/device.h
vs tools/virtio/linux/device.h); the registry broke suffix_match score ties by bucket position; the
LSP budget-share rule measured wall time. Include targets now prefer the File node, the least nested
path, then the lexicographic path; registry ties prefer the least nested QN, then the smaller QN; the
budget share is measured in thread CPU time like the budget itself. Two kernel indexes are now
identical (8,529,723 nodes, 1,974,940 CALLS, zero churn), as are two elasticsearch indexes.
Findability. search_graph's BM25 path honours the `label` filter and ranks an exact-name definition
above test methods that repeat the token (15/15 probe queries).
Coverage. The parse-coverage gap tree subtracts definitions the walker recovered inside an error
region (multi-line only); kernel/sched/core.c reports 476 flagged lines instead of 7,674.
Precision. Config-language callers (Makefile, CMake, YAML) no longer bind unique_name into code
(kernel Makefile -> C CALLS 195 -> 0). JS-family token names ({}, 1, (x: T) => U) are not
definitions; computed, quoted and escaped member keys stay (TypeScript 204 junk -> 0, 1,782 real
names kept). Names are cut at their first line (elasticsearch YAML 8,306 -> 0); a comment inside an
argument list is not an argument, so no Route node is named by a Java block comment; the wildcard
route `/*` is still a route. HTML/Vue/Svelte/Astro embedded member calls stay unbound.
Recall. A Python call through a self- or cls-rooted attribute keeps a specific unique_name member
target (Django +751, elasticsearch +3,288 CALLS, no new callee names).
Robustness. Edge properties never carry a split UTF-8 sequence (the 120-byte expression cut and the
JSON escaper both trim a partial code point). CLI array flags accept a JSON-array literal. Cypher
seeds a single-hop pattern from its selective end (kernel `MATCH (a)-[:CALLS]->(b) WHERE b.name =
'_printk'`: execution limit -> 7.9 s).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
main fails the memory-core linter:
src/mcp/mcp.c: grew by 3 (814 -> 817)
PR #1723 (merged as 92abefa3) added a new early-return path to
handle_index_repository and, following the function's convention, freed
its three argument strings there with raw free(). Its CI lint was green
on 2026-09-09. On 2026-09-19 the memwaste merge cut this file's raw sites
and ratcheted the baseline down to 814, so by the time #1723 landed its
green was measured against a rule that no longer existed. The linter is
right and the merge-result check that let it through was incomplete: it
built and ran the suites and did not run the linter. It does now.
The fix is not cbm_free. The three strings come from
cbm_mcp_get_string_arg and resolved_repo_path_from_project_arg and are
raw heap memory, released raw by every other path in the function;
routing three of them through the core would free raw memory through the
accounting layer. The actual defect is that the function had 41 raw
free() calls of the same three arguments across a dozen early returns,
and #1723 added one more block in the same shape.
One helper, index_args_free(repo_path, mode_str, name_override), now
replaces twelve of those blocks. free(NULL) is a no-op, so the one path
where repo_path is still NULL passes NULL. On the cross-repo branch
mode_str used to be freed before handle_cross_repo_mode and the other
two after it; mode_str is neither passed to nor read by that call, so
releasing all three after it is not observable. Five sites on the
normal-completion path are left as they were: they free the strings at
different points as ownership is handed to the pipeline, and collapsing
them would change that order for no gain.
Raw sites in this file: 817 -> 785. The baseline follows the improvement,
814 -> 785, as the linter asks; it can only tighten.
Build: clean, -Werror, ASan/UBSan. mcp cli daemon_application:
692 passed, 4 skipped, 0 failed -- index_repository_honors_allowed_root,
which runs through a replaced early return, included.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_gbuf_dump_to_sqlite() emitted gbuf.dump regardless of the writer's
result, so a run that published nothing still logged node and edge counts
as if it had worked, and the failure reached the user only as the generic
"Pipeline failed. Check repo_path exists and contains source files."
The writer's temp -> final rename in publish_writer_output() returned a
bare ERR_WRITE_FAILED. #1628 taught cbm_rename_replace() to translate the
platform error into errno so callers could report why, and wired up the
stage -> final rename in pipeline.c. This is the other rename on that
path, and it is the one that runs first.
Preserve errno across the cleanup in every publish error branch. That
includes cbm_writer_open(), where the cleanup unlinks a file that was never
created: without the save it leaves ENOENT behind, so a directory that
denied the create is reported as a missing path — the #1620 case, stated
wrongly. Read the value in graph_buffer.c before the profiling macro can
clobber it, and report gbuf.dump_failed with the return code and that errno.
A sticky append failure clears errno rather than attach a reason it does not
have, and the field is omitted when there is none. No logging is added
inside internal/cbm/, which has none today.
Tests cover both halves: the dump reports a failure instead of a summary,
the writer names its truncation reason, and the publish rename's errno
survives a cleanup that itself fails.
Refs #1620, #2001.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012FxkVF773gDGNmBmsLnzN7
Signed-off-by: juan Carlos Estrada Montoya <junk151516@users.noreply.github.com>
TSan failed on all three platforms in PR CI. Both races are in code this branch
added, and both were invisible to the local ladder because it has no TSan leg —
that lane is CI-only, so "macOS and Linux green" said nothing about either.
1. system_info.c: the cached system info
cbm_system_info() published its cache with a plain flag:
if (!info_cached) { cached_info = detect_...(); info_cached = 1; }
Fine while only the main thread asked. The memory-relief work started calling
cbm_mem_system_under_pressure() from the per-file extract loop, so several
workers now race on the flag AND on the struct — and a struct assignment is
not atomic, so a reader could see half of one. Reported at
system_info.c:284 against :274, from extract_worker, on macOS, ubuntu-arm and
ubuntu-latest alike.
Now a three-state atomic: whichever thread wins the compare-exchange is the
only writer of cached_info, a loser returns the copy it detected itself (the
same answer — detection only reads the OS), and later readers see the
published struct through the acquire/release pair. No mutex, so nothing has
to be initialised before first use.
2. pass_parallel.c: the spill pointer
pp_spill_active() tested ec->pctx->spill BEFORE its atomic spill_mode gate,
so it read an 8-byte pointer with no synchronisation while pp_spill_enter was
publishing that pointer under spill_mu. pp_spill_enter assigns the pointer
and only then release-stores spill_mode, so gating on the atomic first makes
the pointer read ordered by that release. Short-circuit order is load bearing
in that expression now, and the comment says so.
Found by running the TSan lane locally with halt_on_error=0 rather than CI's
halt_on_error=1: the first setting stops at the first race, so each fix would
have revealed the next one a CI round trip later. Enumerating instead showed
the second race was the last one in that suite.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
PR CI's lint job failed on src/foundation/mem_override_libc.c:143-144 while the
local lint lane passed on the same file. The two run different formatters: CI
installs LLVM 20, the local lane uses the Homebrew build, which is now 22. A
wrapped attribute is one of the few things they lay out differently, so the
wrapped form was stable locally and a violation in CI.
__attribute__((no_builtin("memcpy", "memmove",
"memset"))) static void *slow_move(...)
Written on one line the declaration is 117 columns, over the 100-column limit,
so it has to wrap and the two versions disagree about how. Spelling the
attribute as a macro brings the declaration to 73 columns, which leaves no
wrapping decision for either version to have an opinion about. The three
siblings below it already fit inline and are left alone.
Also repins scripts/vendored-checksums.txt for MANIFEST.md itself: the security
job's vendored-integrity layer checksums the manifest too, so editing it to
record the properties scanner patch invalidated its own entry. Produced by
scripts/security-vendored.sh --update; the one changed line is the manifest
hash, which is also the check that the earlier manual scanner.c repin was right.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Indexing the same C# corpus twice on the same machine produced 5,794,336 edges
and then 5,794,328. Every other edge type matched exactly, every semantic
checksum matched (447,064 functions, 81,534 tokens, identical vec/idf/toks),
and the difference was ten WRITES edges out of one generated file.
Three places in extraction decided graph content from a clock:
* the unified AST walk stopped on a thread-CPU deadline;
* a file whose parse spent more than half its budget was refused the
per-file and cross-file LSP walks;
* so was a file whose parse plus walk exceeded the budget.
The first is the one that bit. Any file large enough to reach the deadline is
cut at whatever node the clock lands on, and five generated .NET JIT tests
reached it on every run: hugeSimpleExpr1.cs stopped at node 2,660,352 in one
run and 2,574,336 in the next, out of 2,774,794. That file's tail is field
assignments, so the two runs disagreed by exactly ten WRITES edges. The graph
was a function of how fast the machine happened to be running, and nothing said
so: result->walk_truncated was set and no code anywhere read it.
The deadline existed because those files cost 18-48 us per visited node, and
that turns out to be one fixable thing. csharp_callable_value_site() was the
only language branch in call_reference_candidate_site() still climbing with
ts_node_parent, which tree-sitter answers by descending from the ROOT, scanning
children at every level; every other language already climbs on the walk
cursor. On expressions megabytes deep that was ~85% of the entire unified walk
(sampled: stamp_usage_site -> ts_node_parent -> ts_node_child_with_descendant).
It now takes the cursor and the parent the caller already resolved, and
is_direct_argument_value_cursor() is split so a caller that has stepped the
cursor onto the parent can continue from there rather than paying for it twice.
With that fixed, the budgets are not needed to make indexing finish, so they
are gone rather than made deterministic. A cutoff in nodes would at least be
reproducible, but it still answers "what is in this repo?" with "as much as we
felt like reading". CBM_WALK_MAX_NODES reinstates one for an operator who wants
it; the default is no budget.
Removing the two LSP-skip rules was checked against the file and the path they
were added for (a 23 MB single-expression JIT test that once cost 354 s in the
walk and then crashed the cross-file resolve): the whole 12k-file C# corpus
twice with cross-file resolve, that file alone, and synthetic C# nested 10,000
levels deep — no crash, and past roughly that depth tree-sitter stops producing
a usable tree, so a tree deep enough to endanger a recursive walk never reaches
one.
C# corpus, before -> after:
edges 5,794,318 / 5,794,328 / 5,794,336 -> 5,794,873 every run
truncated 5 files, unreported -> 0
time ~115 s -> 97-101 s
Deterministic, about 550 edges more complete (the tails nobody was walking),
and faster than the build that was cutting them off. All ten bench corpora now
return identical nodes and edges across two runs.
A truncated walk and a skipped LSP walk are now reported in the coverage report
with how far the walk got ("1673216/4133946 nodes walked"), like a partial
parse. A file the walk abandoned halfway used to be reported as fully indexed,
which is a graph quietly lying about its coverage.
Tests: extract_csharp_argument_values_use_the_walk_cursor asserts no occurrence
falls back to the root-descending parent lookup — 128 fallbacks without the fix
and 0 with it. Its fixture parenthesises the argument on purpose: a bare
Sink(Target) needs no climb at all, so it cannot tell the two walks apart and
passed either way while the bug was in place.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
inject_calls_pattern_tokens read a node's CALLS edges straight out of the
graph buffer, which holds them in insertion order — and insertion order is the
order parallel workers happened to merge. It then injected the callee names in
that order and truncated at max_tokens, so which callees survived depended on
worker scheduling. Co-occurrence enrichment reads a window around each token
position, so the surviving vectors differed between runs while the document
frequencies (and therefore idf) stayed identical, which is why this showed up
as a handful of SEMANTICALLY_RELATED edges flickering and nothing else.
It now collects the neighbours through collect_sorted_call_neighbors(), the
same canonical ordering the rest of the pass already uses.
Three runs of the Go corpus now agree exactly — 334535 nodes / 1795409 edges
with vec=0e45744ff1042b51 idf=7e994a1d864a7b07 toks=2518a08a0b6af0c7 — and so
do php, kotlin and rust. The edge counts move slightly against the previous
build (php 57182 -> 57198, kotlin +17, rust +3): sorting changes which tokens
survive the truncation, so the semantic edges change with it. That is a
deliberate output change, not a regression.
Also adds the checksums that found it, all off unless their env var is set:
CBM_SEM_STATE_DUMP writes one hash per scoring input (codes, tf-idf, struct
profile, minhash) and one per corpus (ri vectors, idf, and the token SEQUENCE,
which is what differed here), and CBM_SEM_PAIR_DUMP writes every candidate pair
that passed the threshold with its score, in canonical order. This subsystem
has now produced three order-dependence bugs; a checksum at the seam is what
turns "the graph differs" into a localised answer in minutes, and it costs
nothing when the variable is unset.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The memory-core linter is a ratchet: raw malloc/calloc/realloc/free/strdup
sites may not grow, because allocations belong on one accounted path
(src/foundation/mem_core.h) rather than spread across hundreds of call
sites. This branch's own work had added 35 raw sites across nine files --
the parallel k8s prep, the configlink key matching, the worker pool's
counted path, the Go shared stdlib registry, the spill namespace copy,
the route service array and the namespace-name arrays.
All of them now use cbm_alloc/cbm_calloc/cbm_realloc/cbm_free/
cbm_mem_strdup. One was a latent bug rather than a style point:
pass_k8s freed a buffer that k8s_read_file had allocated with raw malloc,
so converting only the free would have mismatched the allocator -- the
read function and every free of its result are converted together.
The Windows free-space path widens its argument through the memory core
instead of cbm_utf8_to_wide(), which allocates with raw malloc.
Two files improved enough to tighten their baselines, which the ratchet
requires so the gain cannot be quietly given back: sqlite_writer.c 92->74
(the index-cell arena), pass_k8s.c 13->6 and registry.c 19->17.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two indexes of the same corpus with the same binary produced different
graphs. On the go corpus the difference was 9 nodes and 22 edges, all of
it Route nodes and their HANDLES edges.
Two order-dependent choices, both reading arrays that are in graph-buffer
insertion order -- i.e. in whatever order parallel workers happened to
merge:
- create_grpc_routes() collected proto services into a fixed
services[64] and stopped there, so WHICH services got Routes depended
on scheduling. It was also a silent quality cut: every service past
the 64th got no Routes at all, and kubernetes has far more than 64.
All services are collected now, which is why the go corpus gains ~48
Route nodes that were previously dropped.
- match_one_infra_route() linked an infra Route to the FIRST matching
handler in array order. It now picks the best candidate under a total
order on qualified name, so the choice cannot depend on scheduling.
Two runs of the go corpus after this: node sets identical, edge sets
differing by a single SEMANTICALLY_RELATED edge -- down from 9 nodes and
22 edges. Node counts are identical across four consecutive runs.
The remaining flicker is one near-threshold semantic edge. Ruled out by
reading rather than guessing: funcs[] is already canonicalised by
qualified name, the CALLS inputs are byte-identical between runs, LSH
bucket fill is sequential, candidate truncation follows from that,
admission is applied sequentially in canonical order, corpus batches are
sequential and co-occurrence accumulates per target in fixed order. It
needs instrumented tracing of the pair that flips, and is recorded rather
than claimed fixed.
The new test indexes 70 proto services -- more than the old cap -- and
asserts every one gets its Route, so the cap cannot come back whatever
order the workers produce.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The same repository indexed with results parked on disk produced a
DIFFERENT graph than one indexed in memory. Measured on the php corpus,
same binary, reproducible 3/3 each way: 16,561 nodes / 57,182 edges in
memory against 16,562 / 59,379 while spilling. Not jitter, and not
one-directional -- 1,458 CALLS pairs appeared only under spill and 840
only without it. Present on main, so it ships in v0.11.0 today, and it
bites exactly the large-repo users who need spilling.
Cause: pass_parallel built the namespace map -- what `use` / `using` /
package imports resolve against -- from the in-memory result cache, where
a PARKED result is NULL. Every spilled file therefore contributed no
namespace, and a file missing from that map does not fail to resolve, it
resolves through the looser fallback. That is why edge counts moved in
both directions instead of simply dropping.
The fix keeps each parked file's namespace IN MEMORY while its result is
on disk (one short string per file, nothing against the result it
replaces), and builds the map from names so the memory and disk paths
agree. Non-spill runs take the identical code path and are unchanged.
Across ten languages and four memory settings (default, 2 GB, 700 MB,
forced spill), php/java/csharp/kotlin now produce identical graphs where
they differed by up to 40,341 edges before.
The existing spill-parity contract in test_parallel passed throughout,
because its fixture is Go+Java and the compact parity harness omits the
pass that builds the map -- neither reaches the namespace path. The new
test runs the real pipeline over more than MIN_FILES_FOR_PARALLEL
namespaced PHP files, with and without spilling, and fails without this
fix.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Indexing the linux kernel with NO ceiling set took the machine down: on a
48 GB host the default budget is 50% of TOTAL ram (24 GB), the run used
essentially all of it (24.5 GB peak RSS), and with a 12 GiB VM already
running there was nothing left. The budget never fired -- it read 22 GB
charged against 24 GB -- while the host was out of memory.
Total RAM is the CEILING; what is free right now is the CONSTRAINT.
- cbm_mem_clamp_to_available() reduces a fraction-derived budget to fit
what is actually free, reserving a quarter of it (held between 1 and
8 GB) for the OS and everything else, never below a 512 MB floor. An
explicit CBM_MEM_BUDGET_MB still wins -- it is the user's deliberate
choice -- but is warned about when it exceeds free memory, because
the failure it buys reads like a product bug rather than a setting.
- cbm_mem_should_relieve() = our budget exceeded OR the machine short.
Deliberately separate from cbm_mem_over_budget(): system pressure
must RELIEVE (spill, reclaim), never abort, since the spike may not
even be ours. The extraction loop now enters spill on real scarcity,
and the dump phase -- where the kernel run actually died, with our
charge comfortably under budget -- keys its free-heavy decision on it
too. cbm_mem_system_under_pressure() existed and had no callers.
Same scenario after: budget clamped to 18,947 MB, peak RSS 19.5 GB,
status indexed in 185.7 s, machine fine.
Spill also gets limits and cleanup, because relief that fills the disk or
leaves gigabytes behind is not relief:
- a free-space preflight (never start spilling below 10 GB free) and a
ceiling at half of what is free; past it results stay in memory and
back-pressure takes over. cbm_fs_free_bytes() is new (statvfs
f_bavail / GetDiskFreeSpaceExW quota).
- POSIX unlinks each spill file while holding it open, so the space
returns on ANY exit including SIGKILL -- a killed run cannot leave
spill behind because there is no name left to leave it under.
Windows keeps close-path removal plus a sweep where an open file
refuses deletion, which is exactly the liveness test.
- the staging-database sweep now reclaims orphans of EVERY project in
the cache directory, not just the one being indexed. A kernel index
killed by the OOM killer used to park ~15 GB until someone re-indexed
the kernel, and indefinitely if nobody did. The per-stage lock probe
still keeps a live writer's stage, whichever project owns it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Indexing broke on x86-64 Linux: the index worker refused to start and
SIGKILLed its own process group, so every venue reported nothing but
"index worker ended with killed (exit=-1, signal=9)".
glibc places a thread's static TLS block inside the stack allocation it
makes for that thread. This branch's new thread-local caches -- a
512-slot field cache (~24 KB), per-depth tree cursors (~4 KB) and two
parked CBMArenas (~8 KB; an arena carries blocks[256] + block_sizes[256])
-- took static TLS from 47,905 to 93,513 bytes. The 64 KB parent-death
watchdog thread then could not be created at all, and a worker without
process-tree containment correctly refuses to index.
Four layers, because there were four things wrong:
- The big thread-locals move to the heap behind a thread-local POINTER,
allocated once per thread and reused, each with a correct fallback if
allocation fails (the field cache and cursor pool are caches; a miss
is slower, never wrong). Static TLS is back to 52,233 bytes.
- PARENT_WATCHDOG_STACK_SIZE 64 KB -> 256 KB. "The watchdog only polls,
a tiny stack suffices" was true when written and false once TLS
reached ~50 KB, because the TLS block comes out of that same
allocation.
- cbm_thread_create retries with the default stack on EINVAL. A stack
size is a hint about what a thread needs; the platform refusing the
hint is not a reason to fail to create the thread.
- tests/test_thread_stack_tls_contract.sh (Step 5a of scripts/test.sh)
keeps static TLS a quarter clear of the smallest requested stack,
reading that smallest size from the sources so a new small stack
tightens the gate automatically.
Two diagnostics that made this findable at all, and that stay:
smoke-test.sh prints the worker log its failure message names (CI deletes
that sandbox with the job, so the one artifact naming the cause was
unreadable), and worker_containment_unavailable() now reports WHICH of
its conditions failed, with pid/pgrp/ppid.
Verified in the amd64 container, the venue nothing on the local ladder
covers: baseline main indexes 9039 nodes/15067 edges, this branch before
the fix refuses to start the worker, and after it indexes 9039/15067.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Every item here was measured on a real corpus before and after, not
reasoned about. Go corpus, event lane, first run -> now: 245.5 M
allocations -> 134.8 M, 190 GB cumulative -> 29.3 GB, 158 GB never
written -> 9.2 GB, 132 M churn pairs -> 35.7 M, 93 GB of allocator byte
work -> 0.84 GB. Wall clock: Go 40.7 s -> 19.3 s, linux kernel 239.6 s ->
194.6 s, and all ten LSP languages faster with none slower.
Graph output is unchanged, proven rather than assumed: per-class multiset
digests over the full kernel index (8,563,471 nodes identical, 18/20 node
classes and 18/24 edge classes byte-identical, the rest differing only in
the classes a same-binary re-run also perturbs through worker merge order).
The changes, each with its own reason:
arena an exact arena that is appended to later restarts growth at
8 KB instead of the 64 KB default — the cross-file pass adds a
few resolved calls to a compacted result, so the default block
was ~60 KB untouched per result, 0.5 GB of the Go worker peak.
cbm_arena_init_lazy defers the first block for arenas that are
usually never used at all.
scratch per-thread reuse instead of allocate-and-free per unit of
work: result_compact (8 MB kept), the cross-file pass (4 MB),
extract_defs' walk stack, and the sqlite index cells (a 1 MB
arena replacing 12.1 M malloc/free pairs on Go). Walk frames
moving into scratch is why pass_k8s now opens the keep window
explicitly: without it every YAML file opened a 512 KB scratch
on the main thread, +3 GB.
sqlite index cells are written in place — building the record and
copying it into the cell was a second allocation per entry,
16 M record buffers of pure churn on Go.
registry is_test_qn and common_prefix_len in one pass, per-bucket test
flags computed once instead of per candidate comparison, and
the reachability keys arena allocated lazily.
minhash the unique-trigram set is sized from the trigram count and
skips repeats, and LSH queries reuse one seen-set per worker
instead of allocating per query.
parallel lsp_surface rows and the k8s manifest prep now run across
workers (graph writes stay sequential and ordered), the file
node is hoisted out of register_and_link_def, and configlink
prepares dependency imports once with a length prefilter.
go_lsp one sealed stdlib registry shared by all workers, published
by CAS, instead of one per worker.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_mem_charged() trusted mimalloc's committed counter. In a release build
that counter never decreases on purge: prim.c only tracks decommit under
`#if !MI_DEBUG && MI_SECURE<=2`, so after a large spill the budget still
believed memory it had already returned was held.
On the linux kernel corpus with a 15 GB budget that read 19,787 MB charged
against 15,000 allowed — the run spilled and swept repeatedly while the OS
had the memory back. Charging max(OS charge, tracked live bytes) gives
14,379 MB peak charged, sweep down from 14,990 MB to 5,067 MB, and no
budget-pressure warnings.
The reasoning and the measured numbers are at the call site, because the
next reader will otherwise re-derive "the allocator says X" from scratch.
Also adds cbm_mem_allocator_stats_log() behind CBM_MEM_ALLOCATOR_STATS:
mimalloc's own MIMALLOC_SHOW_STATS never prints for the worker, which
exits through _Exit, and mi_stats_merge does not exist in the vendored v3.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Answers a question the existing tooling could not: of everything we
allocate, how much is never used?
Lanes, all driven by scripts/memwaste.sh:
event every allocation observed through the memory core — size, age,
churn, and whether the bytes were ever written
access SanitizerCoverage load/store instrumentation, so "allocated and
never touched" is measured rather than inferred
scaling k vs 2k replicas through clang profile counters, to catch
super-linear work per file
padding allocator slack per size class
ratchet the numbers recorded per venue so a regression is visible
The event lane needed a foreign-free counter: on Windows the override sees
frees for blocks it never allocated (they come from the CRT), and counting
those as untracked tripped the soundness gate. mem_events now separates
them (14 daemon / 5 worker on the Go corpus).
Two portability fixes the lanes surfaced, both real:
- the enable decision must not be cached before `environ` exists. Under
-fsanitize-coverage the UBSan preinit runs dlsym -> malloc -> our
wrapper before glibc sets up environ, so the lane silently disabled
itself and produced no dump at all on Linux.
- the access map needs 48 bits: arm64 Linux hands out addresses above
2^47, which a 47-bit map dropped on the floor.
mem_profile is retired into this: it measured a strict subset and had its
own sampling story.
Local-only by decision — not a PR-CI gate. _memwaste.yml runs the lanes on
all three platforms report-only, and pr.yml/release.yml carry the wiring.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The XML documents that make up a .NET repository project system were
absent from EXT_TABLE, so .csproj, .props, .targets, .resx, .plist,
.xaml and the app manifests were never indexed and their content was
unreachable through search_code. tree_sitter_xml is already wired up,
so these only needed rows in the table.
Fixes#2229
Signed-off-by: yasmoradi <msynk.dev@gmail.com>
The v0.10.9 release run (34948714902) failed in both Linux portable
builds: the static link found malloc, free, realloc, calloc, memalign,
valloc, pvalloc and posix_memalign defined twice, once by mimalloc's
override and once by libc.a(malloc.o). Nothing in the build flags had
changed since v0.10.8; what changed was one call. cbm_mem_release_to_os
called glibc's malloc_trim(0), and that symbol lives only in malloc.o, so
the reference pulled the whole object into the link next to mimalloc's
definitions. Dynamic builds never noticed: interposition resolves the
duplicates at load time, which is why the whole test matrix was green.
The call was pointless where it broke things: on Linux and Windows the
override routes every malloc to mimalloc, so glibc's heap holds nothing
to trim and mi_collect(true) is the release. The branch and its include
are gone; macOS keeps its pressure-relief call, since the system heap
still serves everything outside the core there.
Verification: the static build (scripts/build.sh CC=gcc CXX=g++
STATIC=1) in the local arm64 container reproduced the CI failure exactly
before the change and links after it, the binary reporting its version;
mem and parallel suites green on the host.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The MSan lane on 9724d903 stopped in the extraction suite's spill
round-trip test: uninitialized bytes in fwrite at offset 4087 of a
6,952-byte region, the parked result's arena block (the stack overflow
that followed is MemorySanitizer's own report path failing on the nested
bug). The park writes a result as an opaque image: the record header, a
struct copy, and the compacted arena block. Both carry padding bytes no
code ever wrote, inside structs and between objects, and MSan tracks that
mark through the compaction's memcpy, so zeroing the destination block
only moved the report (offset 6714 in the local MSan container). The
image is read back whole and never interpreted byte by byte; the padding
content is irrelevant on disk.
Under MSan the header and the block are declared defined right before
the write (__msan_unpoison, compiled in only when the build is
instrumented); every other build compiles the annotation to nothing. The
exact block is also zeroed once when it is created, so the gaps between
objects are deterministic in the spill file.
Verification: extraction, arena and parallel suites under ASan on the
fixed tree; the extraction suite in the local MSan container image
passes 348 / 0 with no sanitizer report (it reproduced the CI report exactly before the fix).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two CI-only lanes went red on 9724d903, both this branch's own doing.
The macOS leak lane: every test passed and LeakSanitizer reported 16 to
101 objects of 1,472 bytes per test process, allocated from
cbm_arena_sprintf. The formatter creates one C locale per thread on macOS
(the vsnprintf locale lock fix) and kept it in a thread-local pointer
only, so every worker thread that ever formatted a name leaked its locale
at exit. A pthread key destructor now frees it when the thread exits; the
main thread keeps its locale until process exit.
The TSan lane on Ubuntu arm: pipeline_backpressure_futile_nap_disengages
counted 8 nap cycles against a bound of 7, the "gate re-paid per pull"
shape the test exists for. The spill shortcut reports "not over" while a
sweep is still bringing memory down, and that reading re-armed the
futility latch, so the next over-budget pull paid a full nap cycle again.
The latch now re-arms only on a genuine under-budget reading; a reading
the shortcut produced is not one.
Verification: the local macOS leak lane (make -f Makefile.cbm test-lsan)
reports zero leaks, 8050 passed / 10 skipped (the CI lane had 16-101
leaked objects per process); the local TSan lane (scripts/test.sh --tsan)
passes 1157 / 9 skipped with the nap-cycle test green (the CI lane had it
at 8 cycles); pipeline, parallel and arena suites twice and lint-ci on the
fixed tree.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The daemon leaked 180 KB per query. The Linux soak's quick leg went from
11 to 144 MB of RSS in ten minutes (13.1x, 628 MB/hr), mimalloc's
committed bytes tracking RSS almost byte for byte and the growth linear
in the query count, file descriptors flat -- where the same leg had been
flat at 15 MB in July.
The cause is the previous commit's SQLite heap: one mimalloc heap per
thread, created on first use. The daemon runs a thread per connection,
so every query ran on a fresh thread with a fresh heap, and every block
the shared connection keeps beyond the request -- page cache pages, the
statement cache, schema objects -- was allocated in that heap's pages.
When the thread exited, its heap was deleted and those pages were left
behind holding a few live blocks each: committed for as long as the
connection lives, a few pages per query, never reused by anyone.
The dedicated heap exists for one reason: the index worker's default
heap holds the graph (40M+ blocks), and SQLite churn on that heap paid a
page walk per allocation during the coverage publish (132 s vs 9.7 s on
the kernel). That process has long-lived threads and exits when the
index is done. So the dedicated heap is now a switch, off by default,
that the index worker turns on at its entry (main.c, --index-worker);
everywhere else SQLite allocates from the calling thread's default heap
exactly as before the previous commit. The switch exists in every build
and changes nothing where the allocator binds are compiled out.
Linux soak, quick leg, before -> after: RSS 11 -> 144 MB in ten minutes
(13.1x, 628 MB/hr) -> 10.6 -> 11.5 MB and flat from the first minute
over 564 queries with reindexes every two minutes; query-leak leg:
11.6 MB flat over 1,376 queries; committed heap 6.8 MB flat.
The same soak showed a second thing. From its second sample on, the
query-leak leg reported a committed heap of 2^64 - 121 MB: mimalloc's
committed statistic is a signed counter merged per thread at thread
exit, and a process whose long-lived thread commits what its
short-lived threads free reads it negative -- cast to size_t on the
way out. That figure is one of the two the budget takes the larger of.
cbm_mem_allocator_committed now reports 0 for a reading above
SIZE_MAX/2, so the charge falls back to the OS number instead of a
16 EB budget breach, and the diagnostics report the committed pair
through it (the peak never undercuts the current value and never
carries a wrapped one), the way they already did for RSS.
Detector: the soak's quick and query-leak legs (scripts/soak-legs.sh),
the release-gating sequence in every venue -- RED on the previous tree
(13.1x), GREEN on this one. A unit test cannot see this class: the test
build keeps SQLite on libc so the sanitizers see every block.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The 15 GB kernel run completed with every phase mark under budget, but
the budget metric itself -- the charged footprint the extract gate reads
on every file pull -- reached 16.8 GB. Two defects, found with a probe at
the gate (one line per 256 MB of charge after the spill latch, with the
class table; mem.charge.probe, CBM_MEM_PHASES=1, stays in).
Worker graph buffers kept a dense id array over the global id space. The
charge climbed gradually from the 14.06 GB latch to 14.96 GB, then jumped
about 1 GB between two consecutive gate reads, and the only class that
moved in that step was gbuf_index. Worker buffers draw ids from the shared
counter, so each of the 18 kept an id -> node array spanning the whole
global id space -- 8.5M ids x 8 B, next power of two -- almost entirely
empty, and when the global id crossed 2^23 all eighteen doubled in the
same instant. Nothing asks a worker buffer by id before the merge (every
cbm_gbuf_find_by_id caller runs on the main buffer); the worker
constructor now drops the array the way it already drops the five
secondary indexes, and the main buffer answers by id after the merge.
Spill mode did not park what was already cached. The sweeps ran only on an
over-budget observation; a run that latched early (near_budget) and then
stayed under budget through extraction -- which is exactly what the first
fix produced: 14,949 MB at the extract mark -- carried the 44,797 results
cached before the latch (8 GB) into registry build and resolve, which
cannot park, and resolve ran at a charged 21.8 GB against the 15 GB
budget with nothing to stop it. The earlier runs only fitted because the
id-array burst tripped the sweeps by accident. Extraction now ends with a
final sweep whenever spill mode is on: results belong on disk before the
phases that cannot park inherit them.
Kernel, CBM_MEM_BUDGET_MB=15000, before -> after: charged high-water
16,771-16,818 MB -> 14,582 and 14,732 MB in two runs (97-98% of the
budget); the final sweep parks ~45k results and takes the charge from
14.6 to 4.3 GB before resolve, which peaks at 11.5 GB committed; wall
234 -> 224-230 s; nodes identical (8,529,729), edges inside the
run-to-run band. RSS high-water 18.7 GB is
pages already purged and not yet reclaimed: the footprint never exceeds
7.3 GB at a mark.
peak_charged_mb (7b77fd48) is what made this measurable: the kernel at
the default budget reads charged 22.3 GB against 24.6 GB while ps reports
29 GB RSS, and the 15 GB run read 16.8 GB where every mark was under.
Tests: gbuf_worker_buffer_keeps_no_by_id_array (graph_buffer: one node at
id 2M in a worker buffer must not grow the index class by even 1 MB where
the dense array costs 16 MB; the worker answers by QN, the main buffer by
id after the merge) -- RED on the dense array, GREEN without it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Second increment on the memory core. The first landed the core, the linter
and phase attribution with zero production call sites; this one migrates the
memory that matters, proves where it goes, removes the waste the proof
exposed, and makes the memory budget a promise: over budget, extraction
results go to disk and the run completes at the floor instead of aborting.
Every number below is from a real index on this machine (M5 Pro, 48 GB) with
CBM_MEM_PHASES=1 and the release build.
WHAT THE PROOF SAID (Go corpus: 21,875 files, 345 MB of source)
The graph buffer, blamed for the peak since the 2026-07 research, is 3% of
the memory: 0.55 GB peak. The per-file extraction results are the memory:
14.8 GB written into their arenas, of which 3.4 GB is reachable. The rest
is temporaries -- cbm_node_text copies at 496 call sites, per-node QN
strings, abandoned generations of every growable array -- that an arena
cannot free and that the result kept alive until resolve ended, because
the result owned the arena. The arenas were charged 28.8 GB of capacity
for those 14.8 GB. On the kernel the same shape was 8.9 GB used by 89,731
results, and resolve grew it to 13.3 GB used / 17.2 GB capacity because
the C walk allocated its working set in the result arena and stored arena
pointers into the shared registry. After that, 7 GB of the kernel's
post-resolve memory had no class at all.
WHAT CHANGED
Attribution. The graph buffer (87 raw sites -> 0), arena blocks, the
tree-sitter slab allocator, the bound tree-sitter/SQLite allocators, the
semantic path (pass_semantic_edges, pass_similarity, minhash: 80 sites), the
hash table (Verstable, a class per table via CTX_TY: cbm_ht_create_in) and
CBM_DYN_ARRAY allocate through the core. Classes arena, ts_tree, hash_table
and dyn_array join the table. Two cross-boundary frees of buffer-owned
strings (pass_importance, pass_complexity) go through a new setter,
cbm_gbuf_node_set_properties_json. The linter baseline drops 4086 -> 3916.
Result compaction (internal/cbm/result_compact.c). At the end of a file's
extraction, everything reachable from its result is measured, copied into
one exact-size arena (strings interned by content within the file) and the
working arena is handed back. Later appends restart growth at the default
block (cbm_arena_init_exact, arena grow_size). The stale duplicate
internal/cbm/arena.h, which shared the foundation header's include guard
and won or lost by include order, is a shim now.
Record slimming. CBMCall 320 -> 72 bytes: the eight inline argument slots
(256 bytes, mostly empty) are allocated on first capture. CBMUsage 48 -> 40.
The per-file overlay contract (src/pipeline/pass_lsp_cross.c). One
lifecycle for every language's cross-file resolve: a scratch arena, a
per-file overlay registry chained to the sealed shared base, the walk
against the overlay, resolved calls appended into the result, scratch
destroyed. type_registry gains chain-aware iterators (every yielded index
belongs to it->reg) and copy-on-write refinement
(cbm_registry_func/type_for_update), and the C, C#, Go and Python walks
use them. This is what an ASan heap-use-after-free in the
lsp_resolution_probe suite demanded: the C walk stored per-call arena
objects into the shared registry, and a kernel run with a naive scratch
arena lost 1,045 edges to the resulting garbage reads. It also ends
unsynchronized writes into the shared registry from parallel resolve
workers.
The 7 GB with no class, found and cleaned. gbuf_index tables and keys were
2.5 GB live and 7.1 GB at peak: the 18 worker buffers built the five
secondary indexes they never query (cbm_gbuf_new_worker builds none), and
18M per-key index arrays started at 8 slots (cbm_da_push_min starts them
at 2); edge property JSON is 81% "{}" (interned). The semantic pass
allocated a 512-token stride per function up front (packed tokens with
offsets: cbm_sem_corpus_add_docs_batch). Edge dedup keys are 128-bit
hashes instead of sprintf'd strings (edge_key_map_t). Each worker keeps
one reusable working arena between files (cbm_work_arena_take/give:
rewind, not free) and every phase mark returns freed pages to the OS.
Budget metric. The over-budget check compares the charged footprint --
max(phys_footprint, mimalloc commit) on macOS -- not resident_size: after
extraction the kernel worker sat at 17.4 GB RSS with 5.5 GB charged, because
pages mimalloc has purged stay resident until the OS wants them, and
phys_footprint under-reports after MADV_FREE_REUSABLE cycles.
Spill / admission control (internal/cbm/result_spill.c, the extract gate
in pass_parallel.c). At 15/16 of the budget (or over it, or with
CBM_MEM_SPILL=1) the gate latches spill mode -- early, because the gate
sees the crossing per file pull and the workers' in-flight files carry the
charge past the line before the first sweep lands: every compacted result
is parked on disk as header + block in one append-only file per writer
under <cache>/spill/, the results already cached are swept out by every
worker, and registry build, def collection, surface rows, resolve and the
infra-route passes load a result only for the moment they read it
(relocated by base delta). The abort path stays closed while a sweep runs
or a cached result remains; it fires only at the floor -- graph +
registries + in-flight files. A result that owns sub-results stays in
memory; a retained parse tree is dropped with the parked result and
re-parsed on load. Every consumer of the result cache reads a slot through
cbm_pipeline_result_acquire()/release() (pipeline_internal.h): a parked
slot is NULL in the array, and the passes that indexed it directly lost
every __route__infra__ node and its INFRA_MAPS edges before the contract.
The other things the transient loads found: a def label that was the
result's own pointer (SIGBUS in the Go cross registry builder), two JVM
helpers that passed their input through, a dedup table whose keys borrowed
sr->value (the insert spun on freed memory), and a latch that was visible
before the store was open (a peer found nothing to park and failed the run
while its neighbour wrote 9 GB). The arena rewind reuse branch had
defeated the nblocks = CBM_ARENA_MAX_BLOCKS OOM seam the suites use; it
reuses only blocks that exist. The incremental and probe routes still
index the cache array themselves and therefore never spill
(ctx->spill_allowed stays false there; follow-up).
The semantic pass under a budget (src/pipeline/pass_semantic_edges.c). The
pass held every function's tokens, the corpus's per-document token ids and
the vectors being stored at once: 4.9 GB on the kernel, on top of the
floor, and nothing sized it to the headroom. Now, when the charged
footprint plus that transient would cross the budget, phases 2-4 run per
batch of functions: tokenize -> count into the corpus -> free; finalize
once; tokenize again -> vectorize -> store -> free, with pages returned to
the OS after every batch. Tokenization is deterministic, so both passes
see the same tokens and the graph does not change; the run pays with a
second tokenize. CBM_SEM_BATCH=<n> forces the batch size (the
batched-equals-unbatched test). The corpus frees its per-document token
ids at the end of finalize (the co-occurrence pass is their only reader),
and mem.semantic.step lines (CBM_MEM_PHASES=1) put the class's live/peak
bytes and the charged footprint at every sub-phase.
The semantic corpus and tokenizer allocate through the core (54 raw sites
-> 0). This is not attribution only: cbm_sem_tokenize handed out libc
strdup blocks that pass_semantic_edges freed with cbm_free -- mi_free in
the production build -- since the pass moved onto the core. No libc-backed
test build can see a cross-allocator free, so the core now refuses a block
it does not own (mi_is_in_heap_region) when CBM_MEM_PHASES=1 is set,
fatally: every proof run in this PR ran with it armed, and the Go and
kernel indexes report zero foreign blocks.
Accounting without contention. The core charged every allocation and free
to shared per-class atomics; with 18 workers on 18 cores every block bounced
the same three cache lines, and the bench against the shipped v0.10.8 showed
it as CPU time rising faster than wall time for the same graph: Kotlin 38 ->
121 s CPU (+58% wall), TypeScript 210 -> 596 s (+67%), Go 177 -> 312 s. The
deltas are thread-local now and reach the shared counters once per 256 KB or
512 blocks of change per thread, when a parallel-for worker ends, and
whenever a reader looks (its own thread first). Phase marks read after the
join and are exact; a class peak is a diagnostic and can be low by threads x
256 KB. Result: Kotlin 10.9 s / 43 s CPU (the residual +22% wall against
v0.10.8 tracks its +34% nodes), TypeScript 33.9 s (v0.10.8: 33.4), Go 48.4 s
(v0.10.8: 52.6), Java 81.1 s (v0.10.8: 79.8); graphs unchanged.
SQLite on a heap of its own. The same bench put the kernel's write step at
132 s where v0.10.8 needed 9.7 s, all of it in one coverage publish
statement. A stack sample named the cost: mimalloc's free-page search and
its periodic heap collect, walking the graph's pages -- millions of them
once the graph lives on the core -- for every statement-journal chunk
SQLite allocates. SQLite now allocates from a mimalloc heap of its own per
thread (mi_heap_new; mi_free is heap-agnostic, so nothing else changes),
and the periodic collect is set to the maximum mimalloc allows since the
explicit release at phase marks and spill sweeps is where memory goes
back. Coverage publish 132 -> 9.4 s; profiled kernel 376 -> 248 s
(v0.10.8: 296 s).
Formatting without a process lock. On macOS every vsnprintf takes the
process locale under an unfair lock (localeconv_l inside __vfprintf);
eighteen workers formatting type names on the C# corpus collapsed into
it -- the 10 MB JIT test files took 68 s each instead of under a second
-- and cbm_arena_sprintf paid it twice per string (size pass, write
pass). Each thread now formats with a C locale object of its own
(vsnprintf_l), and the common short string is formatted once into a
stack buffer.
Per-file budgets that cover the whole file. The 5 s per-file budget
covered the parse only; the LSP walks after it had none. C# JIT stress
files (a 23 MB single expression among them) parsed inside the budget --
the old wall-clock parse timeout used to drop them, which hid the rest --
and then held a worker for 346 s each in the usage walk (tree-sitter's
ts_node_parent descends from the root: quadratic on a deep tree), and the
cross-file resolve ran on the same trees. Three rules now, one site for
every language (cbm_extract_file_ex, honoured by cbm_pxc_dispatch_file): a
parse that used more than half the budget disqualifies the file from the
LSP walks (lsp_skipped); the unified walk checks its thread CPU time every
1024 nodes against six budgets and stops there, keeping what it found
(walk_truncated, implies lsp_skipped); a file whose parse plus walk spent
the budget is skipped by the LSP walks as well. Each decision is logged
with the path (extract.lsp.skipped, extract.walk.truncated). C# extraction
355 -> 39 s; the whole index 463 -> 128 s wall and 27 -> 11 GB peak
against v0.10.8 with 299 more nodes. The walk cuts exactly the five JIT
stress files (hugeexpr1, hugeSimpleExpr1, HugeArray1, HugeField1/2); the
15 MB System.Runtime.Intrinsics reference file walks to the end and, with
four generic-nesting JIT regression tests, only skips the LSP walk under
the parse-plus-walk rule.
The crash behind the C# bench. dotnet/runtime is 42,555 files, and its
resolve phase died with SIGBUS on every run of this branch. A libc-backed
ASan build of the server named it: heap-use-after-free in c_adl_resolve,
reading a type name that another worker had allocated in its per-file
scratch arena and freed at the end of its file. The C++ class walk
refines a method's return type when the declaration is more specific
than the pre-registered one (NAMED -> pointer/reference/template), and
it did that by casting the chained lookup result to non-const and
writing a scratch-arena signature into it -- into the sealed shared base
when the entry lived there. The overlay contract had exactly one
bypass, and it was this cast; every lookup returns const, so a grep for
the cast is the audit. The refinement now goes through
cbm_registry_func_for_update: copy-on-write into the overlay, the base
untouched. Test: clsp_method_return_refinement_is_copy_on_write (a
sealed base with a NAMED method return, an in-class pointer declaration;
the base signature pointer is unchanged and the overlay copy carries the
refinement plus a marker only a copy has) -- RED on the cast, GREEN on
the accessor. The fixed ASan build then indexed the whole
corpus clean: 42,555 files, 1,224,981 nodes, 5,794,320 edges, worker
exit 0, no report.
Production build backs the core with mimalloc explicitly (never the global
override, which macOS's two-level namespace forbids); the test build keeps
libc so the sanitizers see every block -- the same split the tree-sitter and
SQLite bindings already use.
PROOF
Go peak RSS 16.7 -> 4.0-5.2 GB; result arenas 14,804/28,817 -> 856/856
MB; wall 55 -> 52-60 s; graph inside the pre-existing run-to-run
jitter (gRPC Route nodes: two identical runs differ by 21).
Go CBM_MEM_SPILL=1: peak RSS 1.7-2.3 GB, 21,875 results parked (950 MB
on disk), graph identical: 39 infra routes, 158 INFRA_MAPS, 21,875
surface rows, nodes/edges inside the Route jitter.
Go CBM_MEM_BUDGET_MB=1200: spill latched at 1,204 MB, the first sweep
took the charge to 768 MB, the run completed at 1.99 GB peak RSS.
Kernel DEFAULT budget (24,576 MB): aborted before, completes now without
spilling. Extraction 35.2 -> 16.5 GB RSS (5.6 GB footprint, 15.3 GB
committed); results 8.9 GB = capacity; resolve tracked 24.6 -> 19.1
GB; peak RSS 27.3 -> 20.1 GB. Nodes identical (8,529,729); edges
inside the +-400 run-to-run band the kernel shows between identical
runs.
Kernel CBM_MEM_BUDGET_MB=16000: aborted before, completes now through the
spill path. Latched at 17.2 GB charged; 89,731 results parked (9.3
GB on disk, 370,188 loads); extraction end 6.2 GB RSS / 5.1 GB
committed; resolve 8.6 GB RSS / 11.5 GB committed; peak RSS 16.6
GB (the semantic pass plateau, 13.3 GB RSS, is graph-derived and
outside what spill can move). Nodes identical; edges 15,773,598.
Kernel CBM_MEM_BUDGET_MB=15000, before the semantic batching: completes,
every phase mark under 15,000 MB by every metric, but the process
RSS high-water mark reached 16.9 GB inside the semantic pass (class
peak 4.9 GB against 0.8 GB live).
Kernel CBM_MEM_BUDGET_MB=15000, with it: completes; the charged footprint
stays under 15,000 MB through the whole semantic pass (14,018 ->
14,381 MB across its sub-phases, 6 batches of 125,644 functions);
every phase mark is under budget by every metric (max footprint
12.6 GB, max commit 11.5 GB); the process RSS high-water mark is
15.65 GB and is set in extraction, at the moment the gate latches
(15,002 MB charged) and before the first sweep lands -- a 4%
overshoot of in-flight work, no longer the semantic pass. Nodes
identical, edges 15,772,913, zero foreign blocks.
Kernel CBM_MEM_BUDGET_MB=15000, with the anticipatory latch: spill mode
enters at 14,062 MB (near_budget); the charged footprint never
exceeds 11.8 GB at any mark; the process RSS high-water mark is
15.1 GB in extraction and 15.3 GB during the final write-out --
1-2% over, the in-flight window and the dump transient. Nodes
identical, edges 15,772,543, zero foreign blocks.
Go CBM_MEM_BUDGET_MB=1200 with it: latched at 1,131 MB, extraction
peak 998 MB RSS, graph identical to the baseline.
BENCH AGAINST THE SHIPPED v0.10.8 (same machine, same driver, 2026-09-14)
corpus wall s peak RSS GB nodes / edges / CALLS
perl 5.7 -> 5.3 0.09 -> 0.05 -3 / -130 / -3
php 6.3 -> 5.8 0.66 -> 0.20 +21.9% / +1.4% / +23.8%
rust 6.8 -> 6.3 1.15 -> 0.67 +1 / +56 / -369
c 5.9 -> 5.7 0.16 -> 0.19 identical
kotlin 8.9 -> 9.4 1.55 -> 0.96 +33.5% / -22.8% / +12.1%
django 10.9 -> 10.2 3.13 -> 1.12 +6 / -14.5% / -28.4%
go 52.6 -> 42.9 17.2 -> 3.9 +14.2% / -10.3% / +2.2%
java 79.8 -> 73.8 27.6 -> 11.0 -121 / -1.9% / -0.6%
csharp 463.3 -> 128.1 27.0 -> 11.2 +299 / -0.9% / -0.6%
typescript 33.4 -> 33.6 12.0 -> 2.5 -186 / -3.5% / +130
kernel 294.8 -> 233.3 27.5 -> 25.1 -171 / -1.3% / -0.1%
No corpus is slower beyond noise; CPU time follows wall (kernel 2,442 ->
1,403 s). Every graph delta against v0.10.8 is main between the release and
the branch base (339b3f40), not this PR: a build of the base commit run
through the same driver reproduces the final graphs exactly on django and
kotlin, within the pre-existing gRPC Route jitter on go (30 nodes, 27
edges, CALLS identical), and on C# with the same node count. This PR's
own C# difference against the base is 5 CALLS (the five truncated JIT
stress files) plus one ijwhost swap and 563 edges of 5.79 M; the base
takes 448 s / 29.7 GB for that corpus.
Diagnostics that made this measurable stay in: the extract.arenas and
extract.census lines (behind CBM_MEM_PHASES=1), the CBM_MEM_RELEASE=1
release-to-OS probe, and peak_charged_mb on every mem.phase line -- the
high-water mark of the budget metric itself (cbm_mem_peak_charged), next
to the RSS peak that counts pages already purged but not yet reclaimed.
Tests: extract_compact_* and extract_spill_round_trip_keeps_every_field
(extraction), parallel_spill_mode_builds_the_same_graph (parallel: every
file parked, node count and every edge type equal to the in-memory run),
pipeline_semantic_batched_matches_unbatched (pipeline: one repo indexed
unbatched and with CBM_SEM_BATCH=5 -- byte-identical node vectors, token
vectors and SEMANTICALLY_RELATED edges),
extract_lsp_skipped_when_parse_used_ its_budget_share and
extract_walk_truncated_at_its_cpu_budget (extraction: the seams mark a
file / stop the walk, the LSP walk and the dispatcher skip it, the defs
stay), registry_overlay_chain_iterates_ and_copies_on_write (c_lsp), arena
exact-init/growth, mem charged/footprint/ pressure.
Not in this increment: run-to-run edge jitter (+-400 on the kernel, Route
class on Go) predates this work and has its own item; the incremental route
spilling; a tighter result encoding (def ids instead of QN strings in
resolved calls) is the next lever on the 9 GB the kernel's results still
take.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel