Martin Vogel 959f4de50c perf: cut the allocation churn and repeated work the sanitizer found ...

Every item here was measured on a real corpus before and after, not
reasoned about. Go corpus, event lane, first run -> now: 245.5 M
allocations -> 134.8 M, 190 GB cumulative -> 29.3 GB, 158 GB never
written -> 9.2 GB, 132 M churn pairs -> 35.7 M, 93 GB of allocator byte
work -> 0.84 GB. Wall clock: Go 40.7 s -> 19.3 s, linux kernel 239.6 s ->
194.6 s, and all ten LSP languages faster with none slower.

Graph output is unchanged, proven rather than assumed: per-class multiset
digests over the full kernel index (8,563,471 nodes identical, 18/20 node
classes and 18/24 edge classes byte-identical, the rest differing only in
the classes a same-binary re-run also perturbs through worker merge order).

The changes, each with its own reason:

  arena      an exact arena that is appended to later restarts growth at
             8 KB instead of the 64 KB default — the cross-file pass adds a
             few resolved calls to a compacted result, so the default block
             was ~60 KB untouched per result, 0.5 GB of the Go worker peak.
             cbm_arena_init_lazy defers the first block for arenas that are
             usually never used at all.
  scratch    per-thread reuse instead of allocate-and-free per unit of
             work: result_compact (8 MB kept), the cross-file pass (4 MB),
             extract_defs' walk stack, and the sqlite index cells (a 1 MB
             arena replacing 12.1 M malloc/free pairs on Go). Walk frames
             moving into scratch is why pass_k8s now opens the keep window
             explicitly: without it every YAML file opened a 512 KB scratch
             on the main thread, +3 GB.
  sqlite     index cells are written in place — building the record and
             copying it into the cell was a second allocation per entry,
             16 M record buffers of pure churn on Go.
  registry   is_test_qn and common_prefix_len in one pass, per-bucket test
             flags computed once instead of per candidate comparison, and
             the reachability keys arena allocated lazily.
  minhash    the unique-trigram set is sized from the trigram count and
             skips repeats, and LSH queries reuse one seen-set per worker
             instead of allocating per query.
  parallel   lsp_surface rows and the k8s manifest prep now run across
             workers (graph writes stay sequential and ordered), the file
             node is hoisted out of register_and_link_def, and configlink
             prepares dependency imports once with a length prefilter.
  go_lsp     one sealed stdlib registry shared by all workers, published
             by CAS, instead of one per worker.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>

2026-09-18 03:31:22 +02:00

..

minhash.c

perf: cut the allocation churn and repeated work the sanitizer found

2026-09-18 03:31:22 +02:00

minhash.h

perf: cut the allocation churn and repeated work the sanitizer found

2026-09-18 03:31:22 +02:00