Counting array appends as edge-preserving makes `keep(item, items) { items.push(item); return item; }` a borrowed-return function: it preserves every heap edge, borrows every reference parameter and returns one of them. Its borrowing body now retains only the appended element and returns the parameter +0, and the owned adapter retains the result before releasing its parameters. An append never releases an element (the slow path stores past the old length), so the caller's owner keeps the result alive.
These PRs were merged into their stack base branches instead of main. This merges perf/alloc-hugepages (with the CI fixes below) into main. Conflicts: order-parity.json (union of entries); scr_alloc.c/scr_alloc.test.ts, where main already has the worker exit trim (#858, adapted to the pre-hugepages allocator): resolved by taking the hugepages allocator and re-applying the original trim commits written on top of it (ea88b87c + 4816696d, clean apply on an identical base).
Brings in cc7c6cb7 (inline truncation decided for the host when no triple is given), which this branch predates; without it the native bootstrap's LLVM comparison with the Node seed fails on x86-64 Linux.
Conflicts, all resolved to this branch's side:
- emitter.ts (numeric-table get, map index probe), expr-bytes.ts, local-array-reads.ts, string-reads.ts: the base only respelled the size-to-double conversions in the old range checks (uitofp nneg, now sitofp); exactIndexLines replaces those checks entirely. The base's other hunks in these files (host inline truncation, the remaining sitofp conversions) merge as they are.
- int32-slots.ts: #837 and #843 both extracted dynamicFieldStores with identical bodies; keep this branch's signature, whose optional records/unions maps also serve #837's single-argument callers.
The `nneg` flag on `uitofp` is LLVM 19 syntax. Sanitized builds hand the textual module to the host clang, and Ubuntu 24.04's clang 18 rejects it ("expected type"), which failed every Linux sanitized Effect shard. These operands are size_t values below 2^63, so `sitofp` converts them identically, is accepted by every supported LLVM, and still lowers to one cvtsi2sd on x86-64 (LLVM 19+ canonicalizes it to `uitofp nneg` where it can prove the sign).
Since #837 a type is traced only if it lies in a cycle-capable component or reaches an intrinsic reference; a type that merely holds a cycle cannot be reached back from it. In both backend-analysis cases `record:outer` only holds `record:cycle`, so the Node oracle and the native compiler agree on ["record:cycle"] and the expectations still listed `record:outer`.
Brings in cc7c6cb7 (inline truncation decided for the host when no triple is given), which this branch predates; without it the native bootstrap's LLVM comparison with the Node seed fails on x86-64 Linux.
The custom Dockerfile.sandbox image builds the LLVM helper against the same LLVM 22 CMake exports as the managed bootstrap, which reference the Polly libraries; without libpolly-22-dev CMake configure fails.
Three workers each hold and drop about 80 MB of small objects and exit;
three more then allocate as much again in the arenas the first ones
returned (on Linux the first ones hand those pages back when they exit).
Records the fixture's preflight/order baseline.
A script thread that exits parks its allocator arena with every freed
block still resident, so the memory of a finished worker stays with the
process until another thread claims the arena. On Linux, a parking thread
that carved at least 16 MiB now hands back every page of its arena that
holds only free blocks (MADV_DONTNEED), relinks the remaining free blocks
in address order, and keeps the returned page runs as zeroed regions that
the next owner carves before the slice frontier. It also trims glibc
malloc. SCR_SA_STATS=1 reports what each park returned; SCR_SA_TRIM=0
turns it off.
Map.get of a reference value under a string key called scr_map_get_str_ref,
which retained the hit through the collection's value adapter: an indirect
tail call into an out-of-line retain function on every hit (in tsc-ts the
symbol tables' Symbol and Type retains, 0.4% of samples, were only reached
that way). Number keys already read the entry and retain with the value
type's own retain. String keys now do the same: scr_map_entry_str returns the
entry, and the generated code loads the value slot and retains it inline.
Both the ordinary and the stack-boxed (local) reads take the entry path.
Corpus 4651 keeps string-keyed results alive past delete, overwrite, clear
and the map itself for instances, records, arrays, nested maps, cycles and
union values; dropping the inline retain makes it (and 523) fail under ASan.
Every worker mailbox is a pipe. Once a process passes 64 descriptors the kernel grows its descriptor table, and while other threads share the table expand_fdtable first waits a full RCU grace period. tsc-ts on a 16-vCPU host stalled 11-25 ms on the thread starting its 14th checker (16 parse threads plus 13 checkers), on the critical path. Worker builds now grow the table to 256 entries at load, while the process has one thread, which never waits; the table never shrinks.
Publication (@scriptc/threads publish() and the zero-copy delivery of
deeply frozen plain-data graphs) makes objects immortal: nothing frees
them, and the RC audit already stops counting them. LeakSanitizer did not
know, so a program that drops its last reference to a published graph
reported every such object as a direct leak whenever leak detection was
on (SCRIPTC_SAN=1 runs on Linux without ASAN_OPTIONS=detect_leaks=0, such
as the documented local fallback). Both publication paths now mark each
object they make immortal with the weak __lsan_ignore_object hook in ASan
builds; ignored chunks are leak-scan roots, so the verdict on the rest of
the heap is unchanged. Other builds compile the hook away.
The sanitized lanes run with detect_leaks=0, so a harness test pins the
contract with LeakSanitizer on (Linux, SCRIPTC_SAN=1 lane).
An empty map's tables are already empty, so clear() returns at once
instead of freeing oversized storage that the next fill would regrow.
Only a clear that removes entries frees storage far larger than them.
The compaction test in delete is marked unlikely.
new Map(m) and new Set(s) copied the source's direct-index size, which
follows the source's peak. The copy now covers its largest live key, and
keys that are sparse for their count hash instead.
Number-keyed maps index small integer keys through a direct table sized
by the largest key ever stored, and bucket tables are sized by the most
entries ever held; neither shrank. scr_map_clear memset both on every
call, even on an empty map, so a collection that was large once paid its
peak size on every later clear. tsc-ts' Relater.reset clears a
Set<number> on every relation check: on Sentry, 36% of the CPU was that
memset.
- clear() resets only the direct-index slots of the entries it removes
(the table is zero at every absent key), resets the bucket table only
when entries pointed into it and it is proportional to them, and frees
storage far larger than the entries removed, returning the map to its
initial state as V8 does. Clearing an empty map does no work.
- delete compacts once fewer than a quarter of at least 64 entries are
live (V8's shrink point), so iteration after mass deletion is O(live).
- Compaction shrinks oversized bucket tables, and after a delete or an
iteration it shrinks the direct index to the surviving keys or moves
keys that became sparse back to hashing. The append path keeps the
current index: it holds a slot of it.
Insertion order and live iteration are unchanged: nothing compacts while
an iteration is active.
DynHost.liveDynRefAdapter was declared to return { snapshot; commit }
while LlEmitter returns its cached LlStreamTypedRefAdapter. Under Node
LlDyn received that same object and bindLiveArrayOps found arrayOps; the
native compiler width-copied the record into the narrower shape, dropped
arrayOps and emitted no scr_dyn_typed_ref_bind_array call, so native and
Node builds of the compiler emitted different LLVM. Declare the real
return type so both pass the shared record.
bindLiveArrayOps took an anonymous `{ declare(line: string): void }` and
four callers passed the LlEmitter class instance. The self-hosted
compiler projects a class instance into a record only from plain fields,
not methods, so the self-hosting IR check rejected each call with SC2002.
Type the parameter as ShapeHost and pass `host.shapeHost`, as the
neighbouring vAdapters/typedRefConstructor calls already do. The emitted
LLVM is unchanged.
Full passes are scheduled by heap growth, and the fraction they wait for
widens while growth-triggered passes keep finding nothing. Widening went one
doubling per pass after two passes in a row had asked for it, so a heap that
grows for a whole run never reached the cap of 16 that #833 introduced: on
tsc-ts, runs with seven full passes widened to 9x at the sixth pass, and the
seventh still walked 2.7-3.1M objects (294-393 ms) late in the run.
The old level now widens two doublings per qualifying pass, never past what
the pass's yield asks for, so passes that keep finding nothing reach the cap
on the fourth pass instead of the sixth, while the heap is still small.
Narrowing is unchanged: a pass that finds old garbage moves the fraction
straight to what its yield asks for, and widening still waits for two passes
in a row. The mature level keeps a doubling per pass: a mature pass walks only
what was promoted since the last one, so its total walking hardly depends on
how passes are spaced, and narrower spacing lets less young garbage float.
The memory bound is #833's: once passes stop finding garbage, the heap may
grow to 17 times its size after the last full pass before the next. It is
now reached after four such passes instead of six. Jumping straight to the
cap after two was also measured; it reaches the bound at heaps of a few
thousand objects and is not better on tsc-ts at --checkers 8.
test_cycle pins the schedule from where an old-garbage ring narrows the
fraction: the pass after it does not widen, the second pass in a row that
finds nothing reaches the cap (one doubling per pass fails here at caps 1
and 16), and a capped pass comes exactly at the cap. cycle.test.ts also runs
SCR_CYCLE_GROWTH_CAP=8.
Measured with tsc-ts origin/parallel-check e469e33 (--optimization=speed)
against the parent commit, interleaved in Linux Vercel Sandboxes (8 vCPU),
medians; diagnostics and exit codes identical. The build is the next
commit's with its parallel walk off (SCR_CYCLE_PARALLEL=0), which runs this
commit's collector. RSS in MB:
single-threaded (10 rounds) wall RSS
Excalidraw 7.94 -> 7.32 -7.8% 1263 -> 1280
Playwright 7.09 -> 6.90 -2.7% 1457 -> 1470
TypeORM 6.67 -> 6.39 -4.1% 1014 -> 1046
tsc-ts self-check 4.33 -> 4.03 -6.8% 626 -> 622
--checkers 8 (10 rounds, second host)
Excalidraw 2.56 -> 2.55 -0.6% 2120 -> 2200
Playwright 2.46 -> 2.39 -3.0% 2649 -> 2686
TypeORM 1.96 -> 1.85 -5.4% 1270 -> 1315
tsc-ts self-check 1.04 -> 1.04 +0.5% 742 -> 744
Pass log (instrumented build, one single-threaded run each): full-pass time
337 -> 89 ms (self-check), 426 -> 130 ms (TypeORM), 323 -> 104 ms
(Excalidraw); the largest full pass now walks 1.1-1.3M objects early in the
run, and total collector time drops by 27-35%.
Scriptc-compiled strings stop at 2^31 - 1 UTF-8 bytes and exceeding the
cap traps ("scriptc: out of memory") instead of throwing. In the native
compiler the self-hosted IR was one 515 MB string, and the frontend
cache entry JSON.stringify'd LLVM and IR together into one ~920 MB
string, about 2.3x below the trap.
- NativeExecutableInput.ir is compact IR as ~16M-char parts; every IR
output is written with writeTextPartsSync.
- Frontend cache entries move to scriptc.native-frontend.v3: the input
metadata carries placeholders and the LLVM and IR follow as separate
members, so the entry is written as parts with an incremental digest
(NativeCache.writeParts). Entries a native reader could not decode as
one string are neither published nor read (NativeCache.read maxBytes),
turning the old trap into a cache miss.
- SCRIPTC_BOOTSTRAP_IR_INPUT decodes the seed IR per function with
deserializeModuleBytes.
The native LLVM text remains one string (344 MB for the compiler,
~6x below the native cap).
The self-hosted compiler's compact IR is ~515-528M characters, a few
million below V8's maximum string length (2^29 - 24). serializeModule
joined it into one string, deserializeModule parsed one string, and the
native bootstrap comparison read cli.ir.json and cli.ll as utf8 strings,
so any compiler growth crashed --emit=ir and the bootstrap tests with
"RangeError: Invalid string length".
- serializeModuleParts returns the compact encoding as ordered parts
(header, one part per function, remainder). serializeModule(mod, true)
is their join, so the on-disk format is byte-identical.
- deserializeModuleBytes splits a UTF-8 Buffer at the root object's
"functions" array with a byte-level scanner and parses the header and
each function separately. Malformed input falls back to the
whole-document parse for its exact error.
- The Node API writes --emit=ir, --emit-ir sidecars and --emit=llvm
through writeTextParts (the former writeLlvmSource batching writer).
- The native bootstrap harness decodes IR per function and compares
LLVM as Buffers, removing ABI-check lines at the byte level.
- The self-hosting serialization fixture gains a "bytes" mode so the
scriptc-compiled serializer exercises both new entry points.
Self-hosted compiler IR (514.6 MB, macOS arm64, Node 24.12):
read old 5.85 s / 4.26 GB max RSS -> new 5.84 s / 1.36 GB
write old 2.05-2.53 s / 4.48 GB -> new 2.30-2.37 s / 2.10 GB
2x module (1.03 GB): old serialize and read throw; new round-trips.
The remaining per-string bound is the largest function (16.3M chars,
~33x headroom) and the header (2.3M chars).
Malte Ubl