fork() copies only the calling thread, so a child forked while the
allocator's prefault helper held scr_sa_pf_lock waited on that lock forever
at its first fault-in request, and a child that released the range the helper
was faulting in waited forever for the helper to finish it. Native builds'
emit workers hung this way, with the parent waiting in waitpid.
The helper's first start now registers pthread_atfork handlers. The prepare
handler takes the lock across the fork and the parent handler releases it.
The child handler resets the ring, the busy range, the waiter state and the
heap polling state, reinitializes the lock and both conditions, and marks the
helper not started, so the child's first request starts a helper of its own.
check_fork_while_prefaulting in scr_alloc.test.c forks while the helper
faults in 64 MiB, over 64 rounds; each child releases that range, faults in a
range of its own and must exit within five seconds. ThreadSanitizer builds
skip it.
Blocks above 512 bytes come from glibc, which serves each worker thread
from a 64 MiB heap of its own and makes it accessible with mprotect a
page-rounded step at a time. Every 2 MiB range of a heap is first touched
while part of it is still inaccessible, so the heaps never get transparent
huge pages: on a 16-checker type check of VS Code they hold about 0.9 GB on
small pages, close to half of the process's page faults.
Once a worker thread has allocated 16 MiB of large blocks, the heap of each
large block it gets is made accessible whole and advised MADV_HUGEPAGE,
once per heap. glibc's calloc still clears only what lies below its own
high-water mark, so no page is touched early. glibc's mmap and trim
thresholds are set to 4 MiB and 8 MiB, so that the largest blocks get
mappings of their own and a heap's free top goes back to the system: with
huge pages, a heap otherwise keeps resident memory it never writes. The brk
heap policy's own thresholds take precedence once it switches on.
The policy is off unless THP is in madvise mode and malloc is glibc's, and
when the glibc.malloc.hugetlb tunable is set or the system forbids
overcommit. A heap is advised only when /proc/self/maps shows anonymous
mappings covering exactly its range. Library archives leave their
embedder's heaps alone.
- process.memoryUsage(): heapTotal counts the region's chunks that
threads hold, minus the chunks discarded workers returned, and
heapUsed counts them less the tails threads left unused when they
moved to a new chunk.
- The region builds only where it can be used: Linux runtimes with
the size-class allocator, which excludes the sanitizer runtimes
(scriptc keeps 8-byte references there). The standalone allocator,
cycle and young-nursery tests link again: the allocator test
defines the scr_trap its region needs, and checks the region (below
32 GiB, aligned, zeroed, chunks larger than one, heap statistics,
threads).
- scr_plr_chunk has an initializer, like the other runtime globals.
When a script thread parks its allocator arena, each class slice still
holds the pages past its bump pointer that were faulted in but never
carved: the rest of the huge page the bump pointer is in, and the pages
the pre-fault helper faulted in ahead of it. The same applies to region
chunks when the thread exits. Until another thread claims the arena,
nothing carves there, so a program that continues on threads that never
claim it (parse threads ahead of a check on the main thread) kept them.
- scr_sa_park_arena returns those pages with MADV_DONTNEED. The next
thread to claim the arena faults them in again as it carves, and they
read as zero, as fresh pages do.
- scr_rgn_thread_exit returns each chunk's pages past its bump on worker
threads; nothing carves in an exited thread's regions.
- scr_mem_release_ahead drops pre-fault requests that overlap the range
and waits for one in progress before returning it, so the helper does
not fault the pages in again.
- scr_alloc.test.c checks that a worker that shares and carves past the
huge-page switch leaves no resident pages past its frontier, keeps its
blocks, and that the next thread carves on from the frontier, zeroed.
Each class slice a thread opens took one madvise(MADV_DODUMP) call,
which takes the mmap lock for writing, and the threads of a worker
executable open hundreds of slices as they start. The slices are now
made dumpable only when the kernel would write a core of the process:
when kernel.core_pattern starts with '|' or '@', or when RLIMIT_CORE is
above 0. The reservation stays MADV_DONTDUMP in every case.
The trim helper returned parked workers' memory once no worker had run for
100 ms. A compiler in watch mode starts its workers again for each rebuild
and faulted the returned pages back in, and a rebuild that runs on the main
thread alone had the discards run alongside it.
- The delay starts at 1 s (SCR_SA_DISCARD_DELAY_MS) and doubles each time
workers start within ten delays of a discard, up to
SCR_SA_DISCARD_DELAY_MAX_MS (10 s by default).
- Once the event loop of the thread that is not a worker runs, the helper
discards one arena at a time, and only while that loop has been in one
wait with nothing ready for a step (scr_sa_loop_waiting, called by the
event loop around its sleep). A loop that does not wait gets the
discards SCR_SA_DISCARD_DELAY_MAX_MS after the delay.
- Programs without a running event loop discard after the delay, as before.
When the last running worker parks, the trim helper now waits
SCR_SA_DISCARD_DELAY_MS (100 by default) before it returns the parked
workers' pages, and stops if a worker starts meanwhile. A program that
exits within the delay, such as a one-shot type check whose checkers finish
just before it reports and exits, no longer returns pages during its last
milliseconds. A thread that finds every arena claimed discards one waiting
arena and claims it, instead of running every waiting discard.
On Linux, a worker thread that exits without having made any object it
allocated reachable from another thread gives back all of its memory: the
pages it carved, including ProgramLifetime instances and other objects it
never freed, and the system blocks it still holds. The memory goes back
once no other worker thread is running, so that returning it does not slow
the workers that are still running.
A worker carves past each slice's frontier while the arena's free lists and
returned regions are set aside, takes from those free lists only once a
slice has no bump room left, and records the system blocks it allocates.
Messages and intern tables that share published graphs by pointer, and the
process-wide key arrays of typed records, mark the thread as sharing; so
does a remote free of one of its blocks. A worker that did not share parks
with its arena claimed and its discard queued: returning pages makes the
kernel interrupt every running thread of the process to flush its TLB, and
glibc's trim locks the arenas other threads allocate from. The last worker
to park starts one helper that returns each queued worker's pages, frees
its system blocks, and restores its arena to its state at the claim, with
the free blocks the worker took free again. A thread that finds every arena
claimed runs the queued discards itself. A worker that shared keeps its
memory as before.
SCR_SA_DISCARD=0 keeps every worker's memory; SCR_SA_DISCARD=poison makes
the discarded pages inaccessible instead, for tests.
A cycle pass tears down its dead set in the order the walk found it, so
the LIFO free lists handed the blocks of a large dead structure to the
next allocations scattered over every page it used, and the structures
built from them missed the cache on most objects. Sets of at least 16384
objects are now torn down in descending address order
(scr_mem_order_for_free, a bitmap of block indices per size class), so the
free lists hand the blocks back in ascending order, the order the bump
pointers carve fresh memory in.
A full pass takes every candidate buffer, so markGray now clears each
reached object's buffered mark in the header write it makes anyway,
instead of a separate loop over the candidates.
check_discard ran trim_allocate again for its returned-regions case, which
writes the blocks into trim_blocks, the array check_trim had freed. Where
glibc maps an 8 MB block by itself (its adaptive mmap threshold, which the
allocator's heap policy leaves alone unless transparent huge pages are in
madvise mode), the free unmaps it and the worker thread's first write
faults. The case now allocates the array again and frees it.
The check that a worker takes the free blocks set aside compared the
highest block it carved with that of the worker before it, which took its
blocks from free lists below the slice's frontier, and it passed when a
worker never took them. Each run now counts the blocks it allocates past
the highest block of the class so far: at most the bump room the slice had
left, against all of them when a worker does not take the free blocks. The
residency bound after a discard allows for pre-fault requests still in
flight.
On Linux, a worker thread that exits without having made any object it
allocated reachable from another thread gives back all of its memory: the
pages it carved, including ProgramLifetime instances and other objects it
never freed, and the system blocks it still holds.
A worker carves past each slice's frontier while the arena's free lists and
returned regions are set aside, takes from those free lists only once a
slice has no bump room left, and records the system blocks it allocates.
Messages and intern tables that share published graphs by pointer, and the
process-wide key arrays of typed records, mark the thread as sharing; so
does a remote free of one of its blocks. When a worker that did not share
parks, the trim helper returns its pages, frees its system blocks, and
restores the arena to its state at the claim, with the free blocks the
worker took free again. A worker that shared keeps its memory as before.
SCR_SA_DISCARD=0 keeps every worker's memory; SCR_SA_DISCARD=poison makes
the discarded pages inaccessible instead, for tests.
Blocks above the small-object classes (array storage, hash tables, link pages, large strings) come from glibc, whose brk heap and per-block mappings stay on 4 KiB pages where THP is in madvise mode; glibc advises its own memory only under the glibc.malloc.hugetlb tunable, which a program cannot set for itself. Once the brk heap passes 32 MiB, every large allocation now advises the heap grown since the previous one (and the aligned interior of big blocks mapped outside it), and glibc's top pad is raised to 32 MiB so heap growth no longer leaves each 2 MiB range straddling the heap end when it is first touched; the mmap and trim thresholds are pinned to their adaptive maxima, which setting the pad would otherwise freeze at their defaults. Linux and glibc only.
First touch of fresh memory costs the allocating thread a page fault and
a page clear. Linux executables now hand that to a helper thread:
- Size-class slices raise their bump limits in steps (256 KiB on the
small-page prefix, half a huge page after the switch), and each step
asks the helper to fault in the next step ahead with
MADV_POPULATE_WRITE, which commits zeroed pages without storing, so it
cannot race with carving. A slice holds at most about one step of
untouched committed memory; a slice's first chunk is not faulted ahead.
- glibc's heap gets a 16 MiB top pad and the helper faults in each brk
extension as soon as it sees the break move (checked after every
request, then polled every 1 ms while it keeps growing; an idle program
does not wake it).
The fast paths are unchanged; pages the helper has not reached are still
faulted by the owner. The helper masks all signals. SCR_SA_PREFAULT=0
turns it off, SCR_SA_PREFAULT_HEAP=0 keeps glibc's heap defaults; kernels
without MADV_POPULATE_WRITE (< 5.14) fall back to the old policy, and
library archives keep it.
The slice-growth allocator check now carves past several huge-page steps
and checks every fresh block reads as zero (UBSan and TSan builds run
with the helper).
These PRs were merged into their stack base branches instead of main. This merges perf/alloc-hugepages (with the CI fixes below) into main. Conflicts: order-parity.json (union of entries); scr_alloc.c/scr_alloc.test.ts, where main already has the worker exit trim (#858, adapted to the pre-hugepages allocator): resolved by taking the hugepages allocator and re-applying the original trim commits written on top of it (ea88b87c + 4816696d, clean apply on an identical base).
A script thread that exits parks its allocator arena with every freed
block still resident, so the memory of a finished worker stays with the
process until another thread claims the arena. On Linux, a parking thread
that carved at least 16 MiB now hands back every page of its arena that
holds only free blocks (MADV_DONTNEED), relinks the remaining free blocks
in address order, and keeps the returned page runs as zeroed regions that
the next owner carves before the slice frontier. It also trims glibc
malloc. SCR_SA_STATS=1 reports what each park returned; SCR_SA_TRIM=0
turns it off.
Every class slice used to opt out of transparent huge pages, so each
4 KiB the bump pointers crossed cost a page fault. Those faults (fault
entry, memcg charge, LRU insertion and the page clear) were 6-14% of
tsc-ts samples in the kernel. Backing the whole reservation with huge
pages removes most of them, but makes the 2 MiB page around every bump
pointer resident at once. A worker executable has up to 32 class slices
per script thread, so tRPC at 8 checkers grew from 558 to 955 MB.
A slice now starts on small pages. Once it has carved 8 MiB, the rest
of its range switches to huge pages (MADV_HUGEPAGE). Only slices that
already hold 8 MiB pay for a partial last huge page. The reservation is
huge-page aligned, so the switch point and every slice start on a 2 MiB
boundary. The slow path raises a slice's bump limit in two steps (to the
switch point, then to the slice end), so the inline fast paths are
unchanged. The policy is Linux-only; other targets keep one limit per
slice, as before. The allocator checks now build with _GNU_SOURCE on
Linux like the runtime packs, so the madvise advice is compiled in and
checked (alignment, contiguous carving across the switch, VmFlags nh/hg).
Measured with tsc-ts best-v1 (parallel-check + 6247795 e24057a 64d9af6
1c42c55), built with this runtime vs origin/main, on two Linux Vercel
Sandboxes (8 vCPU; 5-7 interleaved runs, medians). Columns: wall, max
RSS, minor faults; "all" is MADV_HUGEPAGE on the whole reservation:
single-threaded wall RSS faults all: wall / RSS
Excalidraw -6.1% +0.8% 331k -> 112k -10.0% / +3.5%
Playwright -1.8% +1.0% 371k -> 111k -8.4% / +3.2%
TypeORM -7.2% +0.9% 259k -> 75k -11.8% / +4.1%
tRPC -5.0% +2.0% 101k -> 47k -9.0% / +11.3%
tsc-ts self-check -2.9% +2.6% 158k -> 64k -3.1% / +8.4%
--checkers 8
Excalidraw -2.9% +2.6% 1075k -> 508k -6.3% / +10.6%
Playwright -10.0% +2.6% 739k -> 381k -10.3% / +12.2%
TypeORM -5.9% +2.2% 372k -> 232k -10.1% / +27.7%
tRPC -2.5% +2.5% 159k -> 125k -4.3% / +71.2%
tsc-ts self-check -4.0% +3.9% 213k -> 172k -12.1% / +46.3%
Switching at 2 or 4 MiB was up to about 1.5 points faster single-threaded
but cost up to +11% RSS at 8 checkers. 16 MiB gave back most of the gain.
Prefaulting small pages with MADV_POPULATE_WRITE did not reduce sys time.
Collapsing the small-page prefix with MADV_COLLAPSE at the switch did not
help either. The runtime benchmark suite (--layouts=4 --runs=16) shows a
geomean change of -1.2%: inventory-report -9.9%, validate-errors -5.3%,
log-summary -2.9%, and regex-logs +2.1%. regex-logs peaks at 6.8 MB and
never reaches the switch point; with one binary the policy measures -0.9%
there, so its delta comes from code layout.
Worker executables compiled the small-object allocator out and sent
every allocation and free through the system allocator, because
several script threads would race on one set of free lists. Each script
thread now owns an arena: scr_sa is thread-local in worker builds and
describes one of 32 equal slices of a single process-wide reservation,
so the owning arena of any block is address arithmetic. A thread claims
an arena on its first allocation and parks it, free lists and bump
pointers intact, when it exits; the next thread continues from there.
Memory is never unmapped, so blocks a worker leaves behind (published
immortal objects) stay valid after it exits. A block freed by another
thread goes to its owner's lock-free remote list, drained on the owner's
slow path; realloc moves foreign blocks into the caller's arena.
The emitted constructor and release fast paths are inlined for worker
programs too, addressing the thread-local allocator state, live count
and dispose hook through llvm.threadlocal.address.
The allocator unit test gains a worker build that hands blocks across
threads and reclaims a parked arena, also under ThreadSanitizer.
Move the allocator into scr_alloc.c with inline fast paths in scr_runtime.h;
strings, array headers/slot buffers, and emitted acyclic records/classes
(scr_rt_calloc/scr_rt_free) now use it. Adds scr_alloc.test.{c,ts}.
Chris Tate