The implicit-toolchain probe includes every <...> header the runtime and
vendored sources name, each behind __has_include. clang ships its x86
intrinsic headers for every target, so on arm64 macOS __has_include finds
<emmintrin.h> (included by scr_bytes.c and scr_string.c), and including it
there is an #error. The probe guarded only <cpuid.h> and <immintrin.h> by
target, so on arm64 macOS it failed, native cache warming reported that it
could not validate the compiler toolchain, and builds ran without the
persistent cache. The probe now guards every x86 intrinsic header family
(*mmintrin.h, <x86intrin.h>, <x86gprintrin.h>) and <cpuid.h> by target.
ts7-process.test.c defines _POSIX_C_SOURCE, which hides mkdtemp on macOS;
it now also defines _DARWIN_C_SOURCE there, as host.c does.
The native compiler's frontend-dispose sweep frees about 450 MB on a cold
tsc-ts build, but the size-class allocator keeps the freed blocks on their
classes' free lists, which only later blocks of the same size reuse. Code
generation allocates other sizes, so its pages stacked on top of the freed
heap and set the build's peak resident memory.
- scr_heap_trim (scr_alloc.c) returns the pages of the calling thread's
size-class slices that hold only free blocks (MADV_DONTNEED) and the
system allocator's free pages (malloc_trim). In the single-threaded
allocator the blocks of a returned page leave the free list and the
class carves past its frontier instead; the remaining free blocks are
relinked in address order. Worker runtimes return the calling thread's
arena pages as a parking thread does.
- scr_collect_cycles_phase, the full sweep after a phase's data died,
calls it; the native compiler calls that sweep when it disposes the
frontend.
- test_heap_trim.c checks that the pages go back, that the blocks still
allocated keep their contents, and that new blocks overlap none of
them, with and without worker arenas (another thread allocating and
freeing remotely throughout).
Cold tsc-ts build, 12 interleaved rounds: peak RSS 1407 -> 1247 MB
(-160 MB, -11.4%, every round); wall 87.1 -> 85.9 s (median).
A warm rebuild ran about 3.5 MB of the native compiler's code, but the code
was spread over its 69 MB text, so 64 KB fault-around mapped about 20 MB of
it. The emitter can now take a function layout: a list of functions in
tiers. It writes the listed definitions, generated helpers included, first
and by tier, each in a `layout-<tier>` partition group. The LLVM helper
compiles each partition's definitions in text order, so the listed ones
come first in their partitions, and a helper that places functions by
group keeps a tier's definitions in the same partitions. The rest keep
their order and their source-file groups.
- compile() takes a `functionLayout` file (`<tier> <symbol>` lines,
gzip-compressed when the name ends in .gz). Keys leave out module
numbers.
- build-native-cli.mts uses the committed packages/compiler/native/cli.layout.gz,
or the file SCRIPTC_NATIVE_LAYOUT names; an empty value and instrumented
builds use none.
- scripts/refresh-cli-layout.mjs builds an instrumented compiler, builds
the runtime workloads and a corpus slice cold and then unchanged, and
reads the functions each run executed from the raw profiles. Tier 1 is
`--version` and rebuilds that hit both caches; tier 2 is the other builds.
- The release workflow's cli-profile job also collects a layout on the
release commit, and the CLI builds use it. Names of closures and
generated helpers change with nearby edits, so a committed layout
matches fewer functions as main moves on.
- Forked lowering and emission workers reset the profile counters after
fork and write their profile before _exit, so instrumented builds also
count the work done in workers. This applies to refresh-cli-profile.mjs
too.
Executables the compiler produces are unchanged.
Every file tsc-ts reads goes through scriptc's file system callbacks,
including the ~80 files of tsc-ts's own default library directory: one
round trip from the checker to the compiler and back for each. Those
files belong to the installed tsc-ts (the build caches key on its
executable's identity), and no build serves them differently.
When tsc-ts takes --default-libraries-from-disk, the native compiler
passes it to the checker it starts and to the shared checker server, so
tsc-ts reads those files itself; every other path keeps the callbacks.
Released tsc-ts versions reject the flag, so checker-features.ts asks
each tsc-ts executable once (tsc-ts --api --default-libraries-from-disk
with an empty input exits with 0 only when it knows the flag) and keeps
the answer in the native cache, keyed by the executable's identity. With
a tsc-ts that does not know the flag, builds run as before.
Edit builds of small programs through the shared checker server are
about 18 ms faster, and 6-9 ms faster without it.
The native compiler now starts with the cycle collector's nursery trigger
at 1024 new candidates instead of 256. scr_cyc_set_nursery_threshold sets
the trigger that SCR_CYCLE_THRESHOLD sets, on the pattern of
scr_cyc_set_growth_cap; the environment variable still takes precedence.
The compiler sets it from a constructor in its host object, before its
modules initialize. Set from main, the trigger leaves the build's
collection work where it was: the cost depends on the collector's first
passes. Programs keep the default trigger.
A cold tsc-ts build takes 4% less wall time (12 paired runs) and its peak
RSS goes from 1.41 GB to 1.31 GB. The program's IR is unchanged.
Closing a checker process checked for its exit once and then slept 10 ms
before checking again, although the process exits a fraction of a
millisecond after its input closes. The checks now start 0.1 ms apart and
double, so the process is reaped as soon as it exits. The total wait before
the kill stays at about 50 ms (51.1 ms).
With SCRIPTC_CHECKER_SERVER=1 on Linux and macOS, the native compiler's
checker connections go to a shared `tsc-ts --api --listen` server, which
keeps the files it parsed for the next build. When the variable is unset,
nothing changes.
checkerServerName names the server by a digest of the tsc-ts executable's
identity, the working directory, the file system callbacks, the timing
flag and the TSC_TS_* environment. scriptc_ts7_attach takes the server's
lock file with a non-blocking flock, connects to its socket in this
user's private server directory ($XDG_RUNTIME_DIR/scriptc or
/tmp/scriptc-<uid>), and starts the server when none answers: in its own
process group, with its standard streams on /dev/null and no other
descriptor of the build, with --idle-exit 300 and TSC_TS_LISTEN_MEMORY=2048
unless the environment sets it. A build that finds the server in use,
cannot start or reach it, or gets no answer to a ping checks with its own
tsc-ts process, as without the variable; so does a tsc-ts without
--listen, which exits as it starts. The syntax-only source parser, which
reads no libraries, always uses its own process and leaves the server to
the program's checker.
scriptc_ts7_json_value_end searched for the next quote again after each
backslash escape in a string, so a string with many escapes was scanned
once per escape. The quote it found now stays the candidate end until an
escape skips it, and each byte of a string is scanned once by each
search.
A test scans strings of 2,000,000 newline, backslash and quote escapes,
alone and inside an object, and compares the ends with jsonValueEnd's;
before the change it does not finish within 60 s.
The committed profile was collected before about 200 commits that changed
12,800 lines of the compiler. Functions whose control flow changed since are
optimized as without a profile, so its gain had mostly gone: a cold tsc-ts
speed build was 2.7% faster with it than with no profile, against 5% to 6%
with a fresh one.
Regenerated with scripts/refresh-cli-profile.mjs on this commit.
The LLVM emitter's constructor runs many analyses that each walk every
function body to collect facts from a few node kinds. IrNodeIndex
(ir/node-index.ts) walks the bodies once, before those analyses, and
records each body's nodes in everyStmtList's order with a kind code per
node and the kinds present per function. An analysis names the kinds it
reads and receives exactly those nodes, in the same order, without
recursing through the tree; a function the index does not hold (a
rewritten copy) is walked as before. The index also lists the distinct
type objects of the module, which the boxed-union and runtime-feature
scans read instead of walking every expression's type tree.
These read the index: walk borrows, appended array literals, constant
callbacks and the rebinding census, int32 slots (dynamic field stores,
call facts, the box census, candidate writes and the range walks),
initializer bindings, stack observations, the may-throw call-graph
census, published types and construction facts, final fields, stable
globals, reference writes (private arrays, call targets, definitions,
fresh constructors), boxed unions and runtime features. Analyses whose
walks track their position (statement lists, nesting, ordered
dataflow) keep their own walks.
Callers outside the emitter pass no index and walk as before. The
emitted LLVM IR is unchanged.
packages/compiler/native/cli.profdata.gz is refreshed with
scripts/refresh-cli-profile.mjs on this change, so the native
compiler's optimization profile covers the rewritten analyses.
host.c defines _POSIX_C_SOURCE, under which Darwin's <sys/file.h> hides
flock and LOCK_EX/LOCK_UN, so the native compiler build failed on macOS
with an implicit declaration error. Define _DARWIN_C_SOURCE on Apple
targets before the system headers. Linux preprocessing is unchanged.
The cycle collector widens the growth fraction that schedules full passes
up to 16 times the heap left by the last pass while passes find little.
Lowering a large program leaves hundreds of megabytes of cyclic garbage,
and whether a scheduled pass reclaims it before the build's peak depends
on where the growth rule fires: small changes to the compiler moved the
peak of a tsc-ts build between 1.66 and 2.05 GB.
The native compiler now sets the growth cap to 2 at startup, as
SCR_CYCLE_GROWTH_CAP=2 does, so a full pass runs once the heap has grown
to three times its size after the last one. scr_cyc_set_growth_cap sets
the cap for a program from its own code; the environment variable still
takes precedence, and child processes are not affected.
On hosts that can fork, the native compiler now emits a large module's
function bodies on worker processes (SCRIPTC_EMIT_WORKERS, 4 by default).
The function list splits into contiguous ranges, each emitted by a forked
child of the emitter, and the ranges merge in function order into the text
one emitter gives. A range whose worker fails is emitted in the parent.
Modules under 2,000 functions, Node and Windows keep the serial path.
A range's emitter records an ordered journal (emit-journal.ts):
- every helper it numbers from a counter (the dyn conversions and
matchers, message listeners and readers, JSON writers, publication
walkers, stream adapters and resolve thunks), recorded when its number
is taken, with the entry point and arguments that create it;
- the first request of each key of a module-level table (declarations,
literals, C strings, unit and absent instances, retype guards, regex and
template-strings instances, map descriptors, alias tags) made outside a
helper's definition;
- unkeyed sites (inline caches, speculative loop sites) taken outside a
definition. A speculative loop site's global is now `@sc_spec_bails_N`,
named like the other numbered sites.
The merging emitter replays each range's journal in order. Replay calls
the same entry points, so every helper gets the number and definition the
serial emission gives it, and table entries land in the serial order. The
range's text is then renamed to the merged numbers in one pass. Sets and
flags the ranges set (await callees, function values, RC helper requests)
merge after the replay. LlvmTargetOptions.functionChunks runs the same
merge in one process, for tests.
The fork, exit, wait and lock primitives come from the native host
(host.c, process-fork.ts); the native compiler passes them to the emitter
as LlvmTargetOptions.forkPrimitives.
The frontend's syntax trees and program state refer to each other, so
reference counting frees almost none of them when a build disposes the
frontend. Whether a cycle pass reclaimed them before the build ended
depended on where the collector's schedule happened to be. Passes during
the frontend found its data live and widened the growth fractions, so the
next pass waited until the heap had grown well past its size at disposal,
and the dead frontend and the emitter's garbage stayed until the build
ended.
The native compiler now runs scr_collect_cycles_phase right after it
disposes the frontend, through a host function. The runtime entry point
sweeps, then restarts the growth fractions and the death rule's backoff
from the heap the sweep leaves, as after a productive pass. Under Node
the hook is unset.
* feat: ship the self-hosted native scriptc CLI
- Run the shared compiler and CLI natively, including library builds and compile-time evaluation.
- Install native platform packages and ship relocatable standalone distributions.
- Preserve program semantics and reuse validated build artifacts for fast rebuilds.
* fix: complete native CLI integration and validation
- Preserve library diagnostics and run recursive compiler work on the main stack.
- Restore packaged helper permissions and exercise native npm and Wasm library builds.
- Recover interrupted Sandbox status probes within the original command deadline.
- Launch and manage TypeScript server processes through a portable native FFI boundary.
- Share the production session API between Node and statically compiled clients.
- Preserve Unicode command-line arguments and release server resources on every exit path.
Chris Tate