The native compiler's frontend-dispose sweep frees about 450 MB on a cold
tsc-ts build, but the size-class allocator keeps the freed blocks on their
classes' free lists, which only later blocks of the same size reuse. Code
generation allocates other sizes, so its pages stacked on top of the freed
heap and set the build's peak resident memory.
- scr_heap_trim (scr_alloc.c) returns the pages of the calling thread's
size-class slices that hold only free blocks (MADV_DONTNEED) and the
system allocator's free pages (malloc_trim). In the single-threaded
allocator the blocks of a returned page leave the free list and the
class carves past its frontier instead; the remaining free blocks are
relinked in address order. Worker runtimes return the calling thread's
arena pages as a parking thread does.
- scr_collect_cycles_phase, the full sweep after a phase's data died,
calls it; the native compiler calls that sweep when it disposes the
frontend.
- test_heap_trim.c checks that the pages go back, that the blocks still
allocated keep their contents, and that new blocks overlap none of
them, with and without worker arenas (another thread allocating and
freeing remotely throughout).
Cold tsc-ts build, 12 interleaved rounds: peak RSS 1407 -> 1247 MB
(-160 MB, -11.4%, every round); wall 87.1 -> 85.9 s (median).
A warm rebuild ran about 3.5 MB of the native compiler's code, but the code
was spread over its 69 MB text, so 64 KB fault-around mapped about 20 MB of
it. The emitter can now take a function layout: a list of functions in
tiers. It writes the listed definitions, generated helpers included, first
and by tier, each in a `layout-<tier>` partition group. The LLVM helper
compiles each partition's definitions in text order, so the listed ones
come first in their partitions, and a helper that places functions by
group keeps a tier's definitions in the same partitions. The rest keep
their order and their source-file groups.
- compile() takes a `functionLayout` file (`<tier> <symbol>` lines,
gzip-compressed when the name ends in .gz). Keys leave out module
numbers.
- build-native-cli.mts uses the committed packages/compiler/native/cli.layout.gz,
or the file SCRIPTC_NATIVE_LAYOUT names; an empty value and instrumented
builds use none.
- scripts/refresh-cli-layout.mjs builds an instrumented compiler, builds
the runtime workloads and a corpus slice cold and then unchanged, and
reads the functions each run executed from the raw profiles. Tier 1 is
`--version` and rebuilds that hit both caches; tier 2 is the other builds.
- The release workflow's cli-profile job also collects a layout on the
release commit, and the CLI builds use it. Names of closures and
generated helpers change with nearby edits, so a committed layout
matches fewer functions as main moves on.
- Forked lowering and emission workers reset the profile counters after
fork and write their profile before _exit, so instrumented builds also
count the work done in workers. This applies to refresh-cli-profile.mjs
too.
Executables the compiler produces are unchanged.
The native compiler now starts with the cycle collector's nursery trigger
at 1024 new candidates instead of 256. scr_cyc_set_nursery_threshold sets
the trigger that SCR_CYCLE_THRESHOLD sets, on the pattern of
scr_cyc_set_growth_cap; the environment variable still takes precedence.
The compiler sets it from a constructor in its host object, before its
modules initialize. Set from main, the trigger leaves the build's
collection work where it was: the cost depends on the collector's first
passes. Programs keep the default trigger.
A cold tsc-ts build takes 4% less wall time (12 paired runs) and its peak
RSS goes from 1.41 GB to 1.31 GB. The program's IR is unchanged.
host.c defines _POSIX_C_SOURCE, under which Darwin's <sys/file.h> hides
flock and LOCK_EX/LOCK_UN, so the native compiler build failed on macOS
with an implicit declaration error. Define _DARWIN_C_SOURCE on Apple
targets before the system headers. Linux preprocessing is unchanged.
The cycle collector widens the growth fraction that schedules full passes
up to 16 times the heap left by the last pass while passes find little.
Lowering a large program leaves hundreds of megabytes of cyclic garbage,
and whether a scheduled pass reclaims it before the build's peak depends
on where the growth rule fires: small changes to the compiler moved the
peak of a tsc-ts build between 1.66 and 2.05 GB.
The native compiler now sets the growth cap to 2 at startup, as
SCR_CYCLE_GROWTH_CAP=2 does, so a full pass runs once the heap has grown
to three times its size after the last one. scr_cyc_set_growth_cap sets
the cap for a program from its own code; the environment variable still
takes precedence, and child processes are not affected.
On hosts that can fork, the native compiler now emits a large module's
function bodies on worker processes (SCRIPTC_EMIT_WORKERS, 4 by default).
The function list splits into contiguous ranges, each emitted by a forked
child of the emitter, and the ranges merge in function order into the text
one emitter gives. A range whose worker fails is emitted in the parent.
Modules under 2,000 functions, Node and Windows keep the serial path.
A range's emitter records an ordered journal (emit-journal.ts):
- every helper it numbers from a counter (the dyn conversions and
matchers, message listeners and readers, JSON writers, publication
walkers, stream adapters and resolve thunks), recorded when its number
is taken, with the entry point and arguments that create it;
- the first request of each key of a module-level table (declarations,
literals, C strings, unit and absent instances, retype guards, regex and
template-strings instances, map descriptors, alias tags) made outside a
helper's definition;
- unkeyed sites (inline caches, speculative loop sites) taken outside a
definition. A speculative loop site's global is now `@sc_spec_bails_N`,
named like the other numbered sites.
The merging emitter replays each range's journal in order. Replay calls
the same entry points, so every helper gets the number and definition the
serial emission gives it, and table entries land in the serial order. The
range's text is then renamed to the merged numbers in one pass. Sets and
flags the ranges set (await callees, function values, RC helper requests)
merge after the replay. LlvmTargetOptions.functionChunks runs the same
merge in one process, for tests.
The fork, exit, wait and lock primitives come from the native host
(host.c, process-fork.ts); the native compiler passes them to the emitter
as LlvmTargetOptions.forkPrimitives.
The frontend's syntax trees and program state refer to each other, so
reference counting frees almost none of them when a build disposes the
frontend. Whether a cycle pass reclaimed them before the build ended
depended on where the collector's schedule happened to be. Passes during
the frontend found its data live and widened the growth fractions, so the
next pass waited until the heap had grown well past its size at disposal,
and the dead frontend and the emitter's garbage stayed until the build
ended.
The native compiler now runs scr_collect_cycles_phase right after it
disposes the frontend, through a host function. The runtime entry point
sweeps, then restarts the growth fractions and the death rule's backoff
from the heap the sweep leaves, as after a productive pass. Under Node
the hook is unset.
* feat: ship the self-hosted native scriptc CLI
- Run the shared compiler and CLI natively, including library builds and compile-time evaluation.
- Install native platform packages and ship relocatable standalone distributions.
- Preserve program semantics and reuse validated build artifacts for fast rebuilds.
* fix: complete native CLI integration and validation
- Preserve library diagnostics and run recursive compiler work on the main stack.
- Restore packaged helper permissions and exercise native npm and Wasm library builds.
- Recover interrupted Sandbox status probes within the original command deadline.
Chris Tate