The memory-core linter is a ratchet: raw malloc/calloc/realloc/free/strdup
sites may not grow, because allocations belong on one accounted path
(src/foundation/mem_core.h) rather than spread across hundreds of call
sites. This branch's own work had added 35 raw sites across nine files --
the parallel k8s prep, the configlink key matching, the worker pool's
counted path, the Go shared stdlib registry, the spill namespace copy,
the route service array and the namespace-name arrays.
All of them now use cbm_alloc/cbm_calloc/cbm_realloc/cbm_free/
cbm_mem_strdup. One was a latent bug rather than a style point:
pass_k8s freed a buffer that k8s_read_file had allocated with raw malloc,
so converting only the free would have mismatched the allocator -- the
read function and every free of its result are converted together.
The Windows free-space path widens its argument through the memory core
instead of cbm_utf8_to_wide(), which allocates with raw malloc.
Two files improved enough to tighten their baselines, which the ratchet
requires so the gain cannot be quietly given back: sqlite_writer.c 92->74
(the index-cell arena), pass_k8s.c 13->6 and registry.c 19->17.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Every item here was measured on a real corpus before and after, not
reasoned about. Go corpus, event lane, first run -> now: 245.5 M
allocations -> 134.8 M, 190 GB cumulative -> 29.3 GB, 158 GB never
written -> 9.2 GB, 132 M churn pairs -> 35.7 M, 93 GB of allocator byte
work -> 0.84 GB. Wall clock: Go 40.7 s -> 19.3 s, linux kernel 239.6 s ->
194.6 s, and all ten LSP languages faster with none slower.
Graph output is unchanged, proven rather than assumed: per-class multiset
digests over the full kernel index (8,563,471 nodes identical, 18/20 node
classes and 18/24 edge classes byte-identical, the rest differing only in
the classes a same-binary re-run also perturbs through worker merge order).
The changes, each with its own reason:
arena an exact arena that is appended to later restarts growth at
8 KB instead of the 64 KB default — the cross-file pass adds a
few resolved calls to a compacted result, so the default block
was ~60 KB untouched per result, 0.5 GB of the Go worker peak.
cbm_arena_init_lazy defers the first block for arenas that are
usually never used at all.
scratch per-thread reuse instead of allocate-and-free per unit of
work: result_compact (8 MB kept), the cross-file pass (4 MB),
extract_defs' walk stack, and the sqlite index cells (a 1 MB
arena replacing 12.1 M malloc/free pairs on Go). Walk frames
moving into scratch is why pass_k8s now opens the keep window
explicitly: without it every YAML file opened a 512 KB scratch
on the main thread, +3 GB.
sqlite index cells are written in place — building the record and
copying it into the cell was a second allocation per entry,
16 M record buffers of pure churn on Go.
registry is_test_qn and common_prefix_len in one pass, per-bucket test
flags computed once instead of per candidate comparison, and
the reachability keys arena allocated lazily.
minhash the unique-trigram set is sized from the trigram count and
skips repeats, and LSH queries reuse one seen-set per worker
instead of allocating per query.
parallel lsp_surface rows and the k8s manifest prep now run across
workers (graph writes stay sequential and ordered), the file
node is hoisted out of register_and_link_def, and configlink
prepares dependency imports once with a length prefilter.
go_lsp one sealed stdlib registry shared by all workers, published
by CAS, instead of one per worker.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Replaces the prior Robin-Hood open-addressing implementation with
Verstable (https://github.com/JacksonAllan/Verstable), a 2024 state-
of-the-art C hash table. The public API in foundation/hash_table.h
is UNCHANGED — CBMHashTable is now an opaque type and all 14
production callers (graph_buffer, registry, pipeline passes,
service_patterns, watcher, semantic, etc.) get the new impl
transparently. str_intern.c keeps its specialised (ptr+len) hash —
different semantics, not a clean swap target.
Why: per-sub-phase profiling pinpointed cbm_pipeline_lsp_target_node
and cbm_registry_resolve as the dominant resolve_calls cost. Both
ultimately hit cbm_ht_get; both run millions of times on a kubernetes
index. Verstable's key innovations target exactly this:
* 4-bit hash-fragment metadata in a uint16_t sidecar array lets
chain walks skip most key comparisons WITHOUT touching the
bucket payload (cache wins on hot lookups).
* Quadratic probing with per-bucket displacement chains (not linear
Robin-Hood psl).
* Tombstone-free deletion that moves at most one existing key.
Vendored at internal/cbm/vendored/verstable/verstable.h (2069 lines,
84KB, MIT). Wrapped as a const char* → void* template inside
hash_table.c. The header struct is opaque (forward declaration only);
audited the entire tree — no caller accesses internals or takes
sizeof(CBMHashTable), so opacity is safe.
Test refactor: test_hash_table.c had two impl-specific tests:
* ht_robin_hood_bounded_psl — Robin-Hood specific; removed.
ht_stress_10000 covers the functional equivalent.
* ht_power_of_two_capacity — renamed ht_create_small_capacity,
replaced struct probes with a functional insert+lookup check.
ht_create_capacity_zero/one similarly de-internalised.
Benchmark on kubernetes (12 workers, 20,650 files):
- parallel_resolve wall: 36,376 -> 27,217 ms (-25.2%)
- pipeline.done total: 61,485 -> 51,745 ms (-15.8%)
- registry_build: ~1,000 -> 779 ms (faster — registry uses
cbm_ht_set heavily during build)
- All 3,615 tests pass; hash_table suite 100% green
- ASan/UBSan/SEGV: none
- Peak RSS: ~6.83 GB
- Graph: nodes 297,082 -> 297,102 (+20, 0.007%), edges 1,155,444 ->
1,154,025 (-1,419, -0.12%) — within tolerance; minor delta likely
from hash-iteration-order differences affecting tie-breaks in
resolution. Sanity (zod/serilog): nodes exact, edges within 0.1%.
Cumulative trajectory from broken-state baseline ~35 min on K8s ->
51.7s wall = ~40x total speedup.
Enforce WarningsAsErrors: '*' with zero suppressions of entire check categories.
All non-FP warnings fixed at source; true false positives get per-instance NOLINT
with reasoning. Fixes include: braces around statements (1679), isolate declarations
(75), explicit casts for multilevel pointer conversions, missing includes, function
parameter documentation via NOLINT, and proper NOLINTBEGIN/NOLINTEND nesting.
Complete C implementation of the indexing pipeline (src/), parallel worker
pool, graph buffer with merge support, SQLite writer, and 1893-test suite.
Linter setup: zero warnings from clang-tidy (all checks enabled, no NOLINT
suppressions), cppcheck, and clang-format. All issues fixed at source —
proper headers for external linkage, GROW_ARRAY macro restructured to
eliminate type parameter, null-deref paths guarded, named intermediates
for suspicious-argument checks.
Martin Vogel