Comments only: scr_trap is the runtime's failure path and the
standalone allocator test defines its own, and the region's chunks
count in process.memoryUsage() through scr_heap_stats.
- process.memoryUsage(): heapTotal counts the region's chunks that
threads hold, minus the chunks discarded workers returned, and
heapUsed counts them less the tails threads left unused when they
moved to a new chunk.
- The region builds only where it can be used: Linux runtimes with
the size-class allocator, which excludes the sanitizer runtimes
(scriptc keeps 8-byte references there). The standalone allocator,
cycle and young-nursery tests link again: the allocator test
defines the scr_trap its region needs, and checks the region (below
32 GiB, aligned, zeroed, chunks larger than one, heap statistics,
threads).
- scr_plr_chunk has an initializer, like the other runtime globals.
When a script thread parks its allocator arena, each class slice still
holds the pages past its bump pointer that were faulted in but never
carved: the rest of the huge page the bump pointer is in, and the pages
the pre-fault helper faulted in ahead of it. The same applies to region
chunks when the thread exits. Until another thread claims the arena,
nothing carves there, so a program that continues on threads that never
claim it (parse threads ahead of a check on the main thread) kept them.
- scr_sa_park_arena returns those pages with MADV_DONTNEED. The next
thread to claim the arena faults them in again as it carves, and they
read as zero, as fresh pages do.
- scr_rgn_thread_exit returns each chunk's pages past its bump on worker
threads; nothing carves in an exited thread's regions.
- scr_mem_release_ahead drops pre-fault requests that overlap the range
and waits for one in progress before returning it, so the helper does
not fault the pages in again.
- scr_alloc.test.c checks that a worker that shares and carves past the
huge-page switch leaves no resident pages past its frontier, keeps its
blocks, and that the next thread carves on from the frontier, zeroed.
Each class slice a thread opens took one madvise(MADV_DODUMP) call,
which takes the mmap lock for writing, and the threads of a worker
executable open hundreds of slices as they start. The slices are now
made dumpable only when the kernel would write a core of the process:
when kernel.core_pattern starts with '|' or '@', or when RLIMIT_CORE is
above 0. The reservation stays MADV_DONTDUMP in every case.
The trim helper returned parked workers' memory once no worker had run for
100 ms. A compiler in watch mode starts its workers again for each rebuild
and faulted the returned pages back in, and a rebuild that runs on the main
thread alone had the discards run alongside it.
- The delay starts at 1 s (SCR_SA_DISCARD_DELAY_MS) and doubles each time
workers start within ten delays of a discard, up to
SCR_SA_DISCARD_DELAY_MAX_MS (10 s by default).
- Once the event loop of the thread that is not a worker runs, the helper
discards one arena at a time, and only while that loop has been in one
wait with nothing ready for a step (scr_sa_loop_waiting, called by the
event loop around its sleep). A loop that does not wait gets the
discards SCR_SA_DISCARD_DELAY_MAX_MS after the delay.
- Programs without a running event loop discard after the delay, as before.
When the last running worker parks, the trim helper now waits
SCR_SA_DISCARD_DELAY_MS (100 by default) before it returns the parked
workers' pages, and stops if a worker starts meanwhile. A program that
exits within the delay, such as a one-shot type check whose checkers finish
just before it reports and exits, no longer returns pages during its last
milliseconds. A thread that finds every arena claimed discards one waiting
arena and claims it, instead of running every waiting discard.
On Linux, a worker thread that exits without having made any object it
allocated reachable from another thread gives back all of its memory: the
pages it carved, including ProgramLifetime instances and other objects it
never freed, and the system blocks it still holds. The memory goes back
once no other worker thread is running, so that returning it does not slow
the workers that are still running.
A worker carves past each slice's frontier while the arena's free lists and
returned regions are set aside, takes from those free lists only once a
slice has no bump room left, and records the system blocks it allocates.
Messages and intern tables that share published graphs by pointer, and the
process-wide key arrays of typed records, mark the thread as sharing; so
does a remote free of one of its blocks. A worker that did not share parks
with its arena claimed and its discard queued: returning pages makes the
kernel interrupt every running thread of the process to flush its TLB, and
glibc's trim locks the arenas other threads allocate from. The last worker
to park starts one helper that returns each queued worker's pages, frees
its system blocks, and restores its arena to its state at the claim, with
the free blocks the worker took free again. A thread that finds every arena
claimed runs the queued discards itself. A worker that shared keeps its
memory as before.
SCR_SA_DISCARD=0 keeps every worker's memory; SCR_SA_DISCARD=poison makes
the discarded pages inaccessible instead, for tests.
A cycle pass tears down its dead set in the order the walk found it, so
the LIFO free lists handed the blocks of a large dead structure to the
next allocations scattered over every page it used, and the structures
built from them missed the cache on most objects. Sets of at least 16384
objects are now torn down in descending address order
(scr_mem_order_for_free, a bitmap of block indices per size class), so the
free lists hand the blocks back in ascending order, the order the bump
pointers carve fresh memory in.
A full pass takes every candidate buffer, so markGray now clears each
reached object's buffered mark in the header write it makes anyway,
instead of a separate loop over the candidates.
On Linux, a worker thread that exits without having made any object it
allocated reachable from another thread gives back all of its memory: the
pages it carved, including ProgramLifetime instances and other objects it
never freed, and the system blocks it still holds.
A worker carves past each slice's frontier while the arena's free lists and
returned regions are set aside, takes from those free lists only once a
slice has no bump room left, and records the system blocks it allocates.
Messages and intern tables that share published graphs by pointer, and the
process-wide key arrays of typed records, mark the thread as sharing; so
does a remote free of one of its blocks. When a worker that did not share
parks, the trim helper returns its pages, frees its system blocks, and
restores the arena to its state at the claim, with the free blocks the
worker took free again. A worker that shared keeps its memory as before.
SCR_SA_DISCARD=0 keeps every worker's memory; SCR_SA_DISCARD=poison makes
the discarded pages inaccessible instead, for tests.
The allocator's helper thread faults in memory just ahead of each
class slice's bump limit: one 256 KiB step on small pages and the next
2 MiB huge page once a slice is on huge pages. Every thread got that
policy. Programs with many worker threads (parse and check pools)
committed faulted-in ranges, up to a huge page per hot class per
thread, that the workers never carved, and all workers queued on the
one helper.
Worker threads now fault ahead only after their slice limits have
advanced 4 MiB, and the ranges faulted ahead of all workers together
stay within 8 MiB. A range counts against that budget until its
class's limit reaches it or its thread parks its arena, and each class
has one range in flight. Steps outside the budget are faulted by the
owner, as with SCR_SA_PREFAULT=0. A program whose allocation runs on
one worker keeps the pre-faulting. The main thread keeps the previous
policy, and executables built without worker support compile only that
policy, so single-threaded programs are unchanged.
The previous commit said the helper and the owner cleared the same huge
pages twice. Clean per-run page counters don't show that: with and without
the helper's heap mode a Sentry type check allocates the same number of
pages (1.07M). The cost is the page size: heap ranges the helper populates
before the owner advises them, and the partial huge pages at the ends of
each extension, stay on 4 KiB pages (Sentry 1816 instead of 1930 huge
pages, VS Code 2740 instead of 3051), next to the mmap-lock waits and the
smaller top pad. Also splits the THP-mode ternary to fit the line width.
Where THP is in madvise mode the heap policy (#873) moves glibc's brk heap
to huge pages once it has grown 32 MiB, the same point at which the prefault
helper (#871) starts faulting in every heap extension. Together they cost
more than they save: the helper populates an extension from its start, where
the owner is already carving, so both threads clear the same 2 MiB pages and
the kernel drops one copy; slab requests queue behind the multi-megabyte
heap populates; the owner's brk and madvise wait for the mmap lock those
populates hold; and the helper's 16 MiB top pad replaces the policy's 32 MiB.
In madvise mode the helper now leaves the heap alone (no polling, no top
pad, no heap populate) and keeps faulting ahead in the allocator's slices.
Heaps on small pages (THP never or absent, or always) keep the heap mode.
The first version committed memory ahead for every program: the heap's
16 MiB top pad was faulted in as soon as the helper started, and every
slice past its first 256 KiB faulted the next chunk ahead. A small server
(runtime suite http-api) peaked at 20.7 MiB instead of 4.4 MiB.
- The helper now only watches the break until the heap has grown by
32 MiB; only then does it set the top pad and fault growth in ahead.
Smaller heaps keep glibc's defaults, including its dynamic mmap
threshold, which any top-pad mallopt fixes.
- A slice faults ahead only once it has carved 1 MiB, so lightly used
classes commit nothing extra (and a program that never fills one does
not start the helper at all).
Blocks above the small-object classes (array storage, hash tables, link pages, large strings) come from glibc, whose brk heap and per-block mappings stay on 4 KiB pages where THP is in madvise mode; glibc advises its own memory only under the glibc.malloc.hugetlb tunable, which a program cannot set for itself. Once the brk heap passes 32 MiB, every large allocation now advises the heap grown since the previous one (and the aligned interior of big blocks mapped outside it), and glibc's top pad is raised to 32 MiB so heap growth no longer leaves each 2 MiB range straddling the heap end when it is first touched; the mmap and trim thresholds are pinned to their adaptive maxima, which setting the pad would otherwise freeze at their defaults. Linux and glibc only.
First touch of fresh memory costs the allocating thread a page fault and
a page clear. Linux executables now hand that to a helper thread:
- Size-class slices raise their bump limits in steps (256 KiB on the
small-page prefix, half a huge page after the switch), and each step
asks the helper to fault in the next step ahead with
MADV_POPULATE_WRITE, which commits zeroed pages without storing, so it
cannot race with carving. A slice holds at most about one step of
untouched committed memory; a slice's first chunk is not faulted ahead.
- glibc's heap gets a 16 MiB top pad and the helper faults in each brk
extension as soon as it sees the break move (checked after every
request, then polled every 1 ms while it keeps growing; an idle program
does not wake it).
The fast paths are unchanged; pages the helper has not reached are still
faulted by the owner. The helper masks all signals. SCR_SA_PREFAULT=0
turns it off, SCR_SA_PREFAULT_HEAP=0 keeps glibc's heap defaults; kernels
without MADV_POPULATE_WRITE (< 5.14) fall back to the old policy, and
library archives keep it.
The slice-growth allocator check now carves past several huge-page steps
and checks every fresh block reads as zero (UBSan and TSan builds run
with the helper).
These PRs were merged into their stack base branches instead of main. This merges perf/alloc-hugepages (with the CI fixes below) into main. Conflicts: order-parity.json (union of entries); scr_alloc.c/scr_alloc.test.ts, where main already has the worker exit trim (#858, adapted to the pre-hugepages allocator): resolved by taking the hugepages allocator and re-applying the original trim commits written on top of it (ea88b87c + 4816696d, clean apply on an identical base).
A script thread that exits parks its allocator arena with every freed
block still resident, so the memory of a finished worker stays with the
process until another thread claims the arena. On Linux, a parking thread
that carved at least 16 MiB now hands back every page of its arena that
holds only free blocks (MADV_DONTNEED), relinks the remaining free blocks
in address order, and keeps the returned page runs as zeroed regions that
the next owner carves before the slice frontier. It also trims glibc
malloc. SCR_SA_STATS=1 reports what each park returned; SCR_SA_TRIM=0
turns it off.
Every class slice used to opt out of transparent huge pages, so each
4 KiB the bump pointers crossed cost a page fault. Those faults (fault
entry, memcg charge, LRU insertion and the page clear) were 6-14% of
tsc-ts samples in the kernel. Backing the whole reservation with huge
pages removes most of them, but makes the 2 MiB page around every bump
pointer resident at once. A worker executable has up to 32 class slices
per script thread, so tRPC at 8 checkers grew from 558 to 955 MB.
A slice now starts on small pages. Once it has carved 8 MiB, the rest
of its range switches to huge pages (MADV_HUGEPAGE). Only slices that
already hold 8 MiB pay for a partial last huge page. The reservation is
huge-page aligned, so the switch point and every slice start on a 2 MiB
boundary. The slow path raises a slice's bump limit in two steps (to the
switch point, then to the slice end), so the inline fast paths are
unchanged. The policy is Linux-only; other targets keep one limit per
slice, as before. The allocator checks now build with _GNU_SOURCE on
Linux like the runtime packs, so the madvise advice is compiled in and
checked (alignment, contiguous carving across the switch, VmFlags nh/hg).
Measured with tsc-ts best-v1 (parallel-check + 6247795 e24057a 64d9af6
1c42c55), built with this runtime vs origin/main, on two Linux Vercel
Sandboxes (8 vCPU; 5-7 interleaved runs, medians). Columns: wall, max
RSS, minor faults; "all" is MADV_HUGEPAGE on the whole reservation:
single-threaded wall RSS faults all: wall / RSS
Excalidraw -6.1% +0.8% 331k -> 112k -10.0% / +3.5%
Playwright -1.8% +1.0% 371k -> 111k -8.4% / +3.2%
TypeORM -7.2% +0.9% 259k -> 75k -11.8% / +4.1%
tRPC -5.0% +2.0% 101k -> 47k -9.0% / +11.3%
tsc-ts self-check -2.9% +2.6% 158k -> 64k -3.1% / +8.4%
--checkers 8
Excalidraw -2.9% +2.6% 1075k -> 508k -6.3% / +10.6%
Playwright -10.0% +2.6% 739k -> 381k -10.3% / +12.2%
TypeORM -5.9% +2.2% 372k -> 232k -10.1% / +27.7%
tRPC -2.5% +2.5% 159k -> 125k -4.3% / +71.2%
tsc-ts self-check -4.0% +3.9% 213k -> 172k -12.1% / +46.3%
Switching at 2 or 4 MiB was up to about 1.5 points faster single-threaded
but cost up to +11% RSS at 8 checkers. 16 MiB gave back most of the gain.
Prefaulting small pages with MADV_POPULATE_WRITE did not reduce sys time.
Collapsing the small-page prefix with MADV_COLLAPSE at the switch did not
help either. The runtime benchmark suite (--layouts=4 --runs=16) shows a
geomean change of -1.2%: inventory-report -9.9%, validate-errors -5.3%,
log-summary -2.9%, and regex-logs +2.1%. regex-logs peaks at 6.8 MB and
never reaches the switch point; with one binary the policy measures -0.9%
there, so its delta comes from code layout.
In worker executables the small-object allocator state, the collector's
live and old-freed counts and the weak dispose hook were four separate
thread-locals, and every allocation or free touched them all. On
Darwin each distinct thread-local costs a _tlv_get_addr call per
function, so a runtime free made four such calls.
They now share one ScrThreadHot block (macros keep the runtime sources
unchanged; ordinary and library builds keep separate globals). The
emitted inline paths address the fields at fixed offsets from a single
llvm.threadlocal.address, which LLVM shares across a function.
tsc-ts self-check, macOS, Worker build vs non-worker build: 1.12x ->
1.10x.
Worker executables compiled the small-object allocator out and sent
every allocation and free through the system allocator, because
several script threads would race on one set of free lists. Each script
thread now owns an arena: scr_sa is thread-local in worker builds and
describes one of 32 equal slices of a single process-wide reservation,
so the owning arena of any block is address arithmetic. A thread claims
an arena on its first allocation and parks it, free lists and bump
pointers intact, when it exits; the next thread continues from there.
Memory is never unmapped, so blocks a worker leaves behind (published
immortal objects) stay valid after it exits. A block freed by another
thread goes to its owner's lock-free remote list, drained on the owner's
slow path; realloc moves foreign blocks into the caller's arena.
The emitted constructor and release fast paths are inlined for worker
programs too, addressing the thread-local allocator state, live count
and dispose hook through llvm.threadlocal.address.
The allocator unit test gains a worker build that hands blocks across
threads and reclaims a parked arena, also under ThreadSanitizer.
Emitted record and class constructors called scr_rt_calloc/scr_cyc_alloc
with a constant size, so the size class, the zeroing length and the
collector header setup were recomputed per object behind a call (plus an
empty scr_obj_alloc_note call and a variable-length memset). llvm/alloc.ts
now emits the fast paths inline: pop the class free list (clearing only
the bytes the helper does not store itself) or bump, with the old calls as
the cold slow path. Releases inline the weak-dispose hook check, the live
count, and the free-list push; blocks outside the reservation still go to
scr_rt_free.
The emitted IR does not know which runtime it links against (the sanitized
lane only differs at link time), so the runtime now defines scr_sa in
every build. Where the allocator is compiled out (ASan, SCR_RC_AUDIT,
wasm32, Windows, 32-bit, thread-instanced libraries) it stays all-zero and
the inline paths always take the slow path, which keeps the RC-audit
object notes. ScrSaState's layout is pinned with static asserts. Inline
emission is additionally gated off for 32-bit, WASI, Windows triples and
thread-instanced libraries.
Runtime callers with hot constant-size allocations (union boxes, arrays,
boxes, closures, dyn nodes) use new static inline twins
scr_cyc_alloc_inline/scr_cyc_free_inline; scr_cyc_live is exported.
The ABI guard now recognizes function-pointer extern data
(scr_weak_dispose_hook). New corpus program 4410 pins recycled-block
clearing, size classes, objects above the largest class, cycles and weak
observers.
`ScrSaState scr_sa;` was a tentative definition, which the runtime build
emits as a COMMON symbol. Library localization (abi.localize_runtime)
deliberately keeps COMMON symbols global (the ASan image-registration guard
relies on that), so every runtime-localized library archive exported
scr_sa next to its declared API, failing tests/harness/library-multi.test.ts
M1 ("external definitions equal the declared set exactly").
Initialize it explicitly so it is a regular zero-initialized definition that
localization demotes like every other runtime global. No other global added
by the stack is a tentative definition. The library-multi suite passes
(it fails M1 without this change).
Move the allocator into scr_alloc.c with inline fast paths in scr_runtime.h;
strings, array headers/slot buffers, and emitted acyclic records/classes
(scr_rt_calloc/scr_rt_free) now use it. Adds scr_alloc.test.{c,ts}.
Chris Tate