`scripts/refresh-cli-profile.mjs --context-sensitive` runs the training twice. The second run uses a compiler that the first profile optimizes and that is instrumented again after inlining, and it merges those counts into the first profile. Builds with the result also order each function's blocks by the counts and split cold blocks off (context-sensitive profiles). `--base-profile` runs only the second phase. `scripts/build-native-cli.mts` builds such a compiler when `SCRIPTC_NATIVE_PROFILE_GENERATE` and `SCRIPTC_NATIVE_PROFILE_USE` are both set. The release workflow's cli-profile job collects the profile this way, with a 120-minute timeout.
Under ASan, self-hosting-llvm-emitter, three-library, npm-static and npm
each take 20 to 40 minutes as one Vitest file, so a sharded run cannot
finish before them.
- self-hosting-llvm-emitter splits into its emitter-cases-and-programs
and fixed-point halves; a shard builds the seed once for the halves it
owns. three-library skips the optimization its shard does not own.
Both put their tests in a describe block, which skipUnownedTest
partitions.
- CI runs both in every shard's case-sharded side queue
(scripts/ci-test-plan.mjs); pnpm test:sandbox case-shards them with npm
and npm-static.
- npm.test.ts keeps a skipped placeholder in a shard that owns no
program, so an empty shard does not fail the file.
The sanitized native-driver bootstrap ran its instrumented compiler with
the default 8 MiB stack. Its frontend phase overflows it (SC3004) on main
as well, outside CI's ulimit for the emitter phase. The test now raises
the soft stack limit to the hard one for every sanitized compiler process
it starts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A unit's record holds its IR in the compact encoding, the entries it added
to the helper, lifted-function and other appendable lowerer tables, and the
reachability, generic demand, statement and flag events it made. A unit has
no record when it grew a table no record carries, changed a field, replaced
an entry it did not add, or emitted a diagnostic. Every lowerer collection
and scalar field is classified, and a test fails on one that is not.
SCRIPTC_LOWER_JOURNAL=serial records every unit and replaces its results
with the ones its record carries. scripts/recorded-lowering.mjs
(pnpm test:recorded-lowering) compares corpus programs' LLVM text with and
without the mode. SCRIPTC_LOWER_JOURNAL_DUMP=<dir> also turns the mode on
and appends each unit's record to <dir>/records.jsonl. Without either,
lowering is unchanged.
Linux links use ld.lld when clang finds it and the system linker
otherwise, and the two produce different executables. A baseline record
made with one must not settle candidates linked with the other.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI's pinned LLVM ships ld.lld, and the compiler links Linux executables
with it whenever clang finds it. The Sandbox image and the managed-image
bootstrap installed clang-22 without lld-22, so Sandbox builds linked
with GNU ld: static hello came out at 99 KB instead of 65 KB, and
runtime-tree-shaking failed only in Sandboxes. Install lld-22 in both
and check that clang finds it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The generator listed a field as a scalar when its walk read nothing, so
a nested record holding only numbers and booleans got a ValuesNotRead
check on the record itself, which fails because the record is an object,
even after regenerating. A field is now listed as a scalar only when its
type holds no string and no object; such a record is checked through its
own fields, so a field added to it or a scalar in it that starts to hold a
string still fails type checking with its name.
The generator exports generateIrFields(irText) for the new test, which
generates the readers for an ir.ts with such a record and type-checks
them. The generated file for the current ir.ts is unchanged.
fields.generated.ts reads IR nodes' fields by name, so a field ir.ts gains
after the last `pnpm generate:ir-fields` is skipped by the walkers until
someone regenerates the file; `--check` in CI reports it, but only on the
run that looks.
The generator now ends the file with type-level checks, one per node kind
and per nested record the walkers enumerate: a field outside the list the
file was generated from fails type checking with the field's name, and so
does a field read as a scalar (no string, no object) that can now hold a
string or an object. Fields the file reads already fail type checking when
they are removed or change type. The checks are types only and emit no
code.
- forin-keys.test.ts matches the record-ops table with its eighth field, the
store hook, which is null for a record no untyped code writes.
- bench/record-untyped-writes writes typed records through `unknown` and
`Object.assign` views, so it links dynamic-tier symbols; DYNAMIC_WORKLOADS
lists it.
- The sanitized self-hosting emission test gets a 600 s timeout. It takes
about half of the default 300 s on an idle 16-core host, with or without
these changes, and exceeded the default on a loaded CI shard.
A recorded baseline entry now holds a hash of its program's inputs: the
entry file and its relative imports (paths and contents) and the
optimization level. --baseline-cache uses an entry only when the program
still hashes the same, so a program or an import that changed since the
record builds its baseline, instead of a recorded build error or artifact
hash of the old sources settling the comparison as both-fail or identical.
Records take schema 2; older records are ignored.
The full gate takes 30 to 75 minutes on one Sandbox, which made it the
longest step of a validation round. It can now be split and merged:
- --shard=i/n splits the bench and corpus programs (the self-hosted
compiler runs in shard 1, or alone with --set=self), and
scripts/zero-cost-merge.mjs merges the shards' --json reports into one
verdict, recomputing the compile-time intervals over every shard.
- --record-baseline-cache records the baseline's artifact hashes once;
--baseline-cache skips the baseline build of every program whose
candidate artifacts hash the same. The record is tied to the built
baseline checkout, toolchain and paths.
- The candidate-source self build overlaps the comparison builds on
hosts with room for it.
pnpm test:sandbox can reuse a prepared snapshot's LLVM helper and runtime
pack (SCRIPTC_SANDBOX_NATIVE_REFERENCE) when their inputs are identical,
and retries rate-limited Sandbox creation with backoff.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The self program compiles the compiler source of the checkout the programs
come from (the baseline by default), so a change adding compiler code the
self-hosted frontend cannot lower passed the gate and failed CI's
self-hosting tests. With the self set, the gate now also builds the
candidate's packages/compiler/src/native/cli.ts with the candidate (LLVM
only, not compared) and fails when that build fails.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each @scriptc/cli-<platform> package bundled its platform's LLVM helper
and runtime pack, which are also published as @scriptc/llvm-<platform> and
@scriptc/runtime-<platform>. The CLI packages now depend on those packages
at the same version (workspace:*, which pnpm pack writes as the exact
version) and leave them out: cli-linux-x64-gnu goes from 115 MB to 60 MB
packed.
- build-native-cli.mts --platform-packages, which each CLI package's
build:native passes, writes a manifest without llvm_package, runtime_pack
and runtime_packs.
- loadNativeToolchain resolves a missing helper or host runtime pack as
Node resolves the package from the compiler's directory: nested in the
platform package, beside it (hoisted, or pnpm's virtual store) or higher
up. It requires the compiler's version, and an absent or mismatched
package fails with an error that names it.
- installNativeCli records where the package manager put both packages in
the installed manifest, with the same checks.
- verify-native-cli.mjs requires the exact dependencies of a platform
package and checks the resolved packages' versions.
- package-native-cli.mjs bundles the helper and runtime pack into the
standalone archives, which stay self-contained.
- smoke-native-install.mjs installs the helper and runtime pack tarballs
with the platform package; the bootstrap's exported package drops its
bundled copies first.
- release.yml publishes the CLI packages only once every helper and
runtime pack package resolves on npm, and scriptc only once the CLI
packages resolve, each wait at most 25 minutes. A rerun skips the
uploads npm already accepted and waits again.
The native sanitizer build chose the variants to compile by artifact path.
Since the runtime pack stores identical objects once, two variants of a
unit can share a path, so it compiled such a unit for both variants and
linked it twice (duplicate symbols in the sanitized native bootstrap).
selectedRuntimeUnits now returns each selected unit with its variant;
selectRuntimePackArtifacts and the sanitizer both use it.
Staging the native CLI's assets ran llvm-strip unconditionally and failed
on hosts without it. Without llvm-strip on PATH, tsc-ts now keeps its
symbols and the build prints a warning; the release hosts install LLVM.
The bootstrap test builds its own seed when CI shares none, and that build
still applied the committed function layout, so its LLVM listed functions in
a different order from the native compiler's own emission. Both seed builds
now share one environment, which leaves the layout out.
The native bootstrap compares the seed's LLVM, which the Node-hosted
compiler emits, with the LLVM the native compiler emits for itself. The
seed's build applied the committed function layout and the native
compiler has none, so the two listed the same functions in different
orders. The layout only orders functions, so the seed is built without it.
Runtime packs: variants of a unit that do not read their defines, and
flavors that compile a unit the same way, produce identical objects and
bitcode. The pack build now points those manifest entries at one file
(shareIdenticalArtifacts) and deletes the copies. Each link selects one
flavor and one variant per unit, so no link names a shared file twice. The
linux-x64-gnu pack keeps 1,191 of its 1,448 artifact files.
Native CLI packages:
- The comptime evaluator builds with -g0 and the executable strip flags,
and the Windows build no longer ships its .pdb.
- The copied tsc-ts executable is stripped with llvm-strip, which rewrites
the linker-signed code signature of Mach-O executables.
- On arm64 macOS, verify-native-cli.mjs --run checks the code signatures
of the compiler, tsc-ts and the comptime evaluator.
- SCRIPTC_NATIVE_KEEP_SYMBOLS=1 keeps the symbols of all three.
- The copied runtime sources and declarations get LF line endings. A
Windows checkout converts them to CRLF, so the Windows CLI shipped files
that differed from @scriptc/runtime's.
With --cpu-dispatch, an x86-64 executable carries two copies of each
program function: the baseline code and a copy compiled for x86-64-v3.
main reads CPUID and XGETBV once. Calls between the program's functions
go to the caller's copy. A function that is also named outside the
copies (a function value, a call from main, a runtime callback or a
coroutine body) keeps a dispatcher under its own name, which tests the
level and tail-calls one copy. Functions without such a use get no
dispatcher, so their copies have the callers the original had and LLVM
inlines them the same way.
The executable still runs on any x86-64 CPU. SCRIPTC_CPU_LEVEL caps the
level at run time. The option is off by default, applies to executables
without a --target-cpu level, and is part of the executable cache key.
bench:runtime takes --baseline-cpu-dispatch and --candidate-cpu-dispatch,
and the LLVM differential suite runs each program at levels 0 and 3 with
SCRIPTC_CPU_DISPATCH=1.
A build with an empty cache type-checked scriptc's own declaration files,
the TypeScript libraries and scriptc's declarations, before reusing the
answer in later builds. That check was 18 ms of a 105 ms first build of
hello world.
- build-native-cli.mts has the built compiler build a module without
imports and a module with a type error, each in an empty cache, and
records the content keys of the declaration file sets the checker
reported no diagnostics in: the checker release, the compiler options,
and each file's name and sha256. They ship in
lib/declarations/lib-diagnostics.json, named by the toolchain manifest's
lib_diagnostics.
- A build whose cache has no answer computes the files' content key and,
when the shipped answers hold it, uses no diagnostics for those files
and stores that answer in the cache. The shipped answers apply only
while the checker is the distribution's own tsc-ts and its package
version is the release they were recorded with; a changed file has
another key.
- A compiler the build host cannot run ships without them.
The Linux helpers linked LLVM 22.1.8's release archive. The helper's code
generation is LLVM's code, so building that LLVM from its release sources
with ThinLTO and with a profile of the helper's own work makes code
generation 5-9% faster than the helper as released, with identical objects.
- native/llvm-codegen/llvm.profdata.gz is the profile, collected by
scripts/refresh-llvm-profile.mjs: it builds LLVM with IR instrumentation
and an instrumented helper, and runs that helper on tsc-ts, five Effect
fixtures, the runtime benchmark workloads and every 50th corpus program
(scripts/llvm-helper-inputs.mjs). It follows LLVM's sources, so it needs
a refresh when LLVM is bumped.
- scripts/build-llvm-dist.mjs builds LLVM with that profile and ThinLTO,
and packs it with a checksum. The llvm-dist workflow, run by hand, builds
it for x64 and arm64, requires a helper linked against it to give the
same objects as one linked against the release archive
(scripts/check-llvm-dist.mjs), prints both helpers' times and the
profile's mismatch count, and attaches the archive to the LLVM_DIST_TAG
release of this repository.
- The release workflow's Linux jobs, and the wasm32-wasi job, which also
uploads the x64 GNU helper, download that archive and build the helper
with SCRIPTC_LLVM_THINLTO=1: clang, lld and ThinLTO, and the profile for
the helper's own sources. Without the archive, or if that build fails,
the helper links the release archive as before.
- The four Linux helper packages share their build in
native/llvm-codegen/linux-helper.mjs. Without SCRIPTC_LLVM_THINLTO it
builds the same executable as before.
Chris Tate