Commit Graph
141 Commits
Author SHA1 Message Date
Chris Tate 2b48451cf4 Build the native compiler with profile-guided optimization 2026-10-21 15:33:37 -05:00
Chris Tate cf1e79b5f9 Keep the write-target AST getters out of the generated region 2026-10-21 11:49:05 -05:00
Chris Tate cb75c51eda Fail the zero-cost gate when it compares nothing
A benchmark workload or the self-hosted compiler that fails to build
with both checkouts now fails the gate, and so does a run in which no
program built with both. Before this, a run whose builds all failed the
same way, such as with a helper of another version, reported PASS. A
build error now prints in full under its program; its first line is
often a warning.
2026-10-21 03:45:36 -05:00
Chris Tate 79f98f4684 Regenerate the compatibility files offline, and after merges (#1609) 2026-10-20 21:25:47 -05:00
Chris Tate 100e7738a0 Keep generated-file conflicts until regeneration succeeds, and take the side for the merged pin
regen-generated.mjs resolved each conflicted generated file with
`git checkout <ref> --`, which also staged it, so a failing generator left
the conflict resolved with the branch's stale file. It now writes the
chosen side's file to the working tree only and stages it after both
generators succeed.

It also always took MERGE_HEAD's ledger, so merging a branch with an older
Node pin into one with a newer pin regenerated from a ledger for the older
pin and failed. It now takes the side (index stage 2 or 3) whose ledger is
for the merged pin in node-v24.json, and when neither is, stops and says
to run pnpm node-compat, which fetches the pinned docs.
2026-10-20 21:21:14 -05:00
Chris Tate bde7afa980 Compile undici from a built-in implementation (#1425) 2026-10-20 21:14:37 -05:00
Chris Tate 9c24e64ee2 Regenerate the compatibility files offline, and after merges
pnpm node-compat fetched Node's docs on every run, even though only the
classification changes between pins. pnpm node-compat --local now
reclassifies the committed ledger rows from the current sources and
rewrites the ledger and the public artifact, the same files a fetching
run writes while the pin is unchanged. scripts/regen-generated.mjs uses
it after a merge: for each generated file still in conflict it takes the
merged-in branch's version, regenerates the surface manifest and the
compatibility files from the merged sources, and stages them, so these
files never need a hand merge.
2026-10-20 20:50:04 -05:00
Chris Tate 9eae4af46c Drop the undici shim type test from the TypeScript 5 import allowlist
The test checks the shims with the TypeScript 7 checker and no longer
imports typescript5, so test-ts7 reported its allowlist entry as stale.
2026-10-20 20:08:11 -05:00
Chris Tate 50f93d0be0 Merge main 2026-10-20 19:05:22 -05:00
Chris Tate f3da708774 Compile polyfill packages that are Node builtins as the builtins (#1458) 2026-10-20 18:28:42 -05:00
Chris Tate 74633db92f Merge main 2026-10-20 18:21:35 -05:00
Chris Tate fd34d72e00 Keep each runtime benchmark workload in its own file
workloads.json listed every workload in one array, and its README table
listed them in one hand order, so changes that each added a workload
appended at the same place and conflicted. Each workload is now a JSON
file in benchmarks/runtime/workloads/ named after it, read through
benchmarks/runtime/workloads.mjs by the bench runner, the zero-cost
gate, the strace script and the dynamic-tier test, ordered by category
and then name. The loader still reads workloads.json in a checkout from
before this change, so the gate can compare against one. The README
table follows the same order.
2026-10-20 18:08:10 -05:00
Chris Tate 3306c78914 Check the frontend's types against tsc-ts/api (#1538) 2026-10-20 13:49:45 -05:00
Chris Tate f3c4ba0c20 Run macOS CI jobs nightly and for pull requests that change darwin code
Pushes to main now run every lane except macos, so a merged commit no
longer queues the macOS native build, the darwin test slices and the
macOS LLVM artifact jobs. The nightly sweep and manual runs still run
every lane. Pull requests keep running the macOS lane when
scripts/ci-lanes.mjs selects it.

- The iOS runtime pack builds move from mobile_runtime_packs (linux lane)
  to mobile_runtime_packs_ios (macos lane), since they run on macOS
  runners. The Android pack stays in the linux lane.
- The macos_full dispatch input is replaced by macos_suite (full or
  darwin, default full). Manual runs run the complete macOS suite unless
  macos_suite is darwin.
- ci-lanes.mjs also selects the macos lane for paths naming kqueue, the
  runtime's platform header, and every file of the darwin suite in
  scripts/ci-test-plan.mjs (adding tests/harness/differential.test.ts
  and the darwin runtime pack's archive test).
- The required test check counts a skipped macOS job as passing when the
  macos lane is off, as it already did for pull requests.
2026-10-20 13:43:22 -05:00
Chris Tate 5765d89227 Check the frontend's types against tsc-ts/api
The frontend calls tokenStartOfNode, getTextOfNode, getSourceFileOfNode and
forEachChild instead of AstNode's position and traversal methods. Node lists
answer hasTrailingComma(), and a declaration's modifiers are a modifier list
with their flags.

Symbol, Type, Signature, IndexInfo, TypePredicate, Program, TypeChecker and
Diagnostic have tsc-ts/api's members and types. Compiler code reads symbol
names with symbolName(symbol).

tsc-ts goes to 0.0.5. api-conformance.ts asserts that AstNode, AstNodeList,
SourceFile and each node interface are assignable to tsc-ts/api's interface of
the same name, and names the members where the wire format differs.
2026-10-20 13:41:39 -05:00
Chris Tate 8c40f92019 Merge main 2026-10-20 11:00:03 -05:00
Chris Tate 5632decbb9 Skip untaken global-detection arms, and fix the shim test checks
- An if statement whose condition is a global-detection test
  (static-typeof.ts) lowers only its taken arm, unless the untaken arm
  declares a var or function binding. Code Node never runs no longer
  counts as a deferred site, so global-detection-typeof.cjs is fully
  static.
- The resolver parity sweep skips specifiers the package-shim registry
  serves: they resolve to the built-in module by design.
- undici-shim-types.test.ts joins the TypeScript 5 import allowlist.
- node-types-timers uses a 200 ms timer, so the immediate runs first on
  a loaded machine as well.
2026-10-20 08:52:39 -05:00
Chris Tate 5e0ad5bbaa Compare checker panics without their text in the checker identity check
A statement whose lowering meets a checker panic keeps the panic's text in its SC0004 fence. tsgo and tsc-ts reach the same panic for `s.match(re) || []` (6187-js-match-or-empty-array.js) and describe it in their own words, so the check now replaces the text after `panic:` before it compares the IR and the diagnostics.
2026-10-20 08:51:47 -05:00
Chris Tate ce5e4294ad Merge main 2026-10-20 08:49:25 -05:00
Chris Tate 9d80738297 Run LeakSanitizer over the sanitized corpus on Linux CI (#1492) 2026-10-20 00:47:53 -05:00
Chris Tate a495dc35a6 Run LeakSanitizer over the sanitized corpus on Linux CI
The Linux test jobs set ASAN_OPTIONS=detect_leaks=0 for both flavors and
left leak detection to the RC audit, which does not see runtime tables
allocated with calloc. A worker thread's record-source table leaked that
way, and only local runs reported it. The san shards' case-sharded corpus
task now sets detect_leaks=1 and an LSAN_OPTIONS suppressions file,
tests/harness/lsan.supp, which starts empty. Plain shards, the side queue,
the Effect jobs and macOS, whose ASan has no LeakSanitizer, keep leak
detection off.
2026-10-20 00:38:58 -05:00
Chris Tate 3d00b5b837 Size encoded messages to their nodes and wake a channel's owner once per batch
A message encoder started its node array at 16 nodes of 184 bytes and
kept the array, with its unused capacity, until the message was
received. A port that queued many messages held about 3 KB of zeroed
memory per message: 200,000 queued numbers took about 590 MB, and the
page faults made postMessage cost about 1.3 µs. The node array now
starts at 4 nodes and is trimmed to the message's node count when
encoding ends.

scr_channel_post posted a wake event, and with it a pipe write and a
condition broadcast, to the receiving thread's mailbox for every
message. The owner's loop counts a port with queued messages as ready,
so only a message that finds the queue empty needs to wake it.

bench/channel-messages, added here, posts 200,000 messages (numbers and
small objects) on a MessageChannel before the port delivers any.
2026-10-19 23:24:56 -05:00
Malte UblandClaude Opus 5.5 c59457b0dc Map npm packages to built-in implementations and mark optional dependencies unloaded
package-shims.ts registers implementation modules shipped with the
compiler per package and major version (chosen by the importer's
installed version) and reads the project's never-loaded packages from
"scriptc": { "packages": { "<name>": "unloaded" } }. Module edges go
through resolveProjectImport; the npm resolution, the npm-edge walk and
the --npm-static set skip such packages; the tsgo host reads a shimmed
package's index.d.ts as a re-export of its implementation; an unloaded
package's require() throws Node's not-installed MODULE_NOT_FOUND.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 17:31:38 -07:00
Malte UblandClaude Opus 5.5 7ea35785ee Check the bench workloads' dynamic-tier symbols in CI
The zero-cost gate's I3 check (a fully typed bench workload defines no
dynamic-tier runtime symbol) runs only when someone runs the gate, and on
Darwin it never matched: Mach-O names carry a leading underscore, so no
symbol had a DYNAMIC_TIER_PREFIXES prefix.

DYNAMIC_WORKLOADS and the nm function-name reader move to
zero-cost-compare.mjs, which now drops the Mach-O underscore. A harness
test builds every bench program on Linux and fails when a workload outside
DYNAMIC_WORKLOADS links a dynamic-tier symbol, or when a listed workload
no longer links any, so the list shrinks as leaks close.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 06:44:29 -07:00
Chris Tate b9fbe4b50b Merge main 2026-10-19 08:42:36 -05:00
Chris Tate c80f9e62a1 Record new corpus program costs in per-program files (#1361) 2026-10-19 07:48:44 -05:00
Chris Tate 8320cc2289 Spin briefly before a mailbox wait blocks, and signal after the unlock
A worker message round trip cost about 12 µs, almost all of it in two
blocked waits: the poster signaled the waiting thread while it still held
the mailbox lock, so the woken thread went back to sleep on the lock, and
every wait paid a sleep and a wakeup. scr_mailbox_post and close now
signal after the unlock, and scr_mailbox_wait spins for up to 20 µs (or
the wait's timeout, if shorter) on a flag the queue keeps before it
blocks. A process that may run on only one CPU never spins.

A new bench:runtime workload, worker-messages, posts 100,000 round trips
between the main thread and a worker: 1296 -> 360 ms.
2026-10-19 07:46:06 -05:00
Chris Tate 3c749a2895 Read AST lists through their nodes
The frontend's node lists become node lists, { pos, end, nodes }, the
shape of tsc-ts's NodeList, so that the frontend can later read tsc-ts's
own nodes. AstNode's list getters return an AstNodeList, ts.NodeArray<T>
is that class, and the frontend reads a list's elements through nodes.
The lists already carried their positions on the wire.

The class is the type, not a structural interface, because native code
represents a class and a record differently.

A decoded file keeps each list's nodes array, as before, and each read
of a list field returns a new AstNodeList over it, so the file holds no
more objects than before. nodes keeps its identity; a list object is not
the same object on each read.

The change is mechanical: a type-aware rewrite of each list field read,
plus hand edits where lists are read through casts, generic accessors
or test mocks. The generated getters and types follow from
generate-ts7-ast-schema.mjs. AstFile and Ts7Session drop their unused
listMetadata hook.
2026-10-19 06:33:33 -05:00
Chris Tate 20a989e405 Format Dates with Node's local time zone names (#1355) 2026-10-19 04:12:27 -05:00
Chris Tate 15636ecf90 List bench/thread-slots among the zero-cost gate's dynamic workloads
The workload's release binary links dynamic-tier symbols through its
worker messages: node:worker_threads values (Worker, parentPort,
workerData) are dyn objects, and worker.on and postMessage lower to
dynamic method calls. ThreadSlots and InternTable link none on their
own. intern-table and typed-array-scan-workers are listed for the same
reason.
2026-10-18 22:18:10 -05:00
Chris Tate ec2f2a341f Record new corpus program costs in per-program files
Every change that adds a corpus program also added its CI costs to
tests/harness/corpus-costs.json. The table is sorted by program name and
programs are numbered in sequence, so concurrent changes inserted lines at
the same place and conflicted. A change now writes
tests/harness/corpus-costs/<program>.json ({ "plain": ms, "san": ms }).
The shard loader applies these files over the table, and
scripts/ci-corpus-costs.mjs folds them into the table and deletes them when
it refreshes the costs from CI logs. With no per-program files the table and
the shard assignment are unchanged.
2026-10-18 22:04:17 -05:00
Chris Tate 705c197a71 Format Dates with Node's local time zone names
Date.prototype.toString, toDateString and toTimeString, String(date),
template literals and string concatenation print local time with the GMT
offset and the zone's long name, as Node does. The runtime identifies the
host zone from TZ or /etc/localtime, again after each write to
process.env.TZ, and looks its name up in a table generated from Node's
ICU data. Dates held in untyped values convert to strings with toString()
and to numbers with their time value.
2026-10-18 20:20:05 -05:00
Chris Tate 6c07441c24 Check programs with tsc-ts by default
scriptc's frontend talks to a TypeScript 7 API server. tsc-ts serves the
same API as tsgo, and it is now the default checker; SCRIPTC_CHECKER=tsgo
selects tsgo, which scriptc still includes.

- @scriptc/compiler depends on tsc-ts 0.0.3. The native distribution
  ships its platform package as lib/tsc-ts, named by a new tsc_ts field
  of the toolchain manifest; a manifest without it takes the executable
  of an installed tsc-ts package.
- The executable cache key includes the tsc-ts executable's identity, in
  the compiler and in the CLI's startup route.
- tsc-ts keeps the syntax trees it builds for the life of its process,
  so the source parser starts a new server after 256 KB of parsed
  source instead of letting one server grow with every file.
- A CI job compiles every corpus program with both checkers, in release
  and dev, and requires the same LLVM IR and diagnostics
  (scripts/checker-identity.mjs).
- Tests that compare with the TypeScript 7 SDK, the checker-panic
  diagnostics, and the protocol tests run against tsgo.
2026-10-18 19:55:52 -05:00
Chris Tate 29c8ad237c Throw when nm cannot list an executable's symbols in the zero-cost gate
The gate read an empty symbol list when nm could not start, was killed by
a signal, or exited with a non-zero status, so the dynamic-tier symbol
check and the executable comparison passed without any symbols. nmOutput
now throws with the binary, the reason and nm's stderr.
2026-10-18 11:36:13 -05:00
Malte UblandClaude Opus 5.5 4128e6f6f5 Count instructions for differing corpus executables in the zero-cost gate
--cachegrind now also runs differing corpus executables (without
arguments, accepting intentional non-zero exits), so emitter changes that
only touch corpus programs get per-function instruction deltas. AGENTS.md
mentions the dynamic-tier symbol rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 08:57:34 -07:00
Malte UblandClaude Opus 5.5 8e6986c561 Add the zero-cost gate
scripts/zero-cost-gate.mjs compiles a fixed good-code set (the runtime
benchmark workloads, the self-hosted compiler, and the TypeScript corpus)
with a baseline and a candidate checkout. It requires byte-identical IR,
LLVM and release executables, and names the first differing IR function,
LLVM function or executable symbol when something changes. Programs that
match the change's pattern signature (--allow) may differ. It also checks
compile time (paired builds, bootstrap interval), binary size, optional
per-function cachegrind instruction counts, and the two-tier I3 symbol
rule: no executable gains dynamic-tier runtime symbols, and fully typed
benchmark workloads define none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 08:17:51 -07:00
Chris Tate de4e10d542 Shift the runtime in bench:runtime layouts
The --layouts padding object was linked directly before the program
object. The runtime is linked before the program object, so every layout
moved only the program's code and left each runtime function at the same
address. A change to the runtime's code size then moved its functions in
all layouts at once, and the verdicts could not tell that placement
change from a real one.

The padding object now goes before every other input on the link line,
so each layout shifts the runtime and the program code.
2026-10-17 21:34:52 -05:00
Chris Tate f61e5dc938 Add readline's terminal mode, cursor helpers and emitKeypressEvents 2026-10-17 02:10:27 -05:00
Chris Tate e5dc49c0cb Refresh the Effect fixture costs from CI measurements (#1097) 2026-10-16 03:38:04 -05:00
Chris Tate bf98a562d8 Run each darwin corpus slice as two concurrent halves
The macOS lane of pull requests and pushes to main runs the darwin suite in
three slices. Each slice runs differential.test.ts over a third of the
corpus while the darwin contracts run beside it. vitest runs one file's
cases in sequence, so a slice's corpus time is the sum of its case times:
in three green main runs, 587-891 s (plain) and 800-1,731 s (sanitized),
on 3-core runners whose contracts task ends after 46-180 s.

ciTestPlan now runs slice i of n as case shards 2i-1 and 2i of 2n in two
vitest processes: the first beside the side queue, the second in the side
queue after the darwin contracts. The halves of all slices are the case
shards of 2n slices, so every program still runs once.

The comment above darwinFiles now says which macOS suite each CI event
runs: pull requests and pushes to main run the darwin suite, and the
nightly schedule and manual runs with macos_full run the full suite.
2026-10-16 01:55:40 -05:00
Chris Tate c8999078ca Refresh the Effect fixture costs from CI measurements
tests/harness/effect-costs.json and effect4-costs.json were recorded on
2026-10-01 and give the fixtures' relative cost for balancedShardSelect
across the 16 Effect shards of each flavor. Against the fixture times of
three green main runs (38851650148, 38851726594, 38852266290), the old
tables put 1,171 s of plain and 1,295 s of sanitized fixture time on the
longest shard, against a mean of 1,027 and 1,111 s.

scripts/ci-effect-costs.mjs refreshes both tables from Effect job logs:
each fixture's plain and sanitized medians, summed in seconds, as the
tables recorded before. Fixtures without a measurement keep their value,
and fixtures that no longer exist are removed. With the refreshed tables,
the longest shard of the same measurements carries 1,082 s (plain) and
1,144 s (sanitized).
2026-10-16 00:50:37 -05:00
Chris Tate 9acf71cfe3 Split the native frontend test across three files
self-hosting-native-frontend.test.ts was one test of 577 s (plain) and 607 s
(sanitized) in the cost table, and 748 s on san 1/20 in run 38835202089; once
the other long files were spread, it alone bounded the Linux test lane. It
builds the compiler frontend into a native executable, then checks 17 corpus
programs and four other cases with it. Locally the build took about 189 s and
the checks 297 s (14-20 s per program).

The build and the checks move to tests/harness/native-frontend.ts without
changes to what they check; useNativeFrontend builds the executable in a
beforeAll of the calling suite. self-hosting-native-frontend.test.ts,
-programs.test.ts and -checks.test.ts each build once and run programs 1-6,
7-12, and 13-17 plus the four other checks, each check as its own test. The
diagnostics, library contract and FFI checks use project directories of their
own instead of rewriting one shared main.ts.

scripts/ci-test-costs.json replaces the old file's entry with estimates for
the three files (the build share of its measured cost plus each file's share of
the checks); the next refresh measures them, since vitest's file time includes
beforeAll. The largest side-queue share per shard at 20 shards goes from 580 to
367 s (plain) and from 608 to 432 s (sanitized), for two more builds per flavor.
2026-10-15 23:46:14 -05:00
Chris Tate bbd24beba3 Split npm-static.test.ts across the Linux shards by measured test cost
npm-static.test.ts ran whole on one shard as an isolated suite: 406 s plain and
519 s sanitized in the cost table, and 672 s on san 2/20 in run 38835202089,
the second-longest job of that run.

skipUnownedTest (tests/harness/shard.ts), called from the suite's beforeEach,
balances the suite's tests across the shards of SCRIPTC_TEST_SHARD by their
measured milliseconds (tests/harness/npm-static-costs.json; an unmeasured test
weighs as much as the median) and skips the tests this shard does not own.
Repeated test.each titles get an occurrence suffix as their key. Without a
shard spec every test runs.

The CI plan runs the suite on every shard in its own vitest process
(caseShardedSideFiles) instead of on one owner. At 20 shards the largest
shares are 113 and 106 s plain and 181 and 137 s sanitized (the two published
Effect generator tests), and about 10 s elsewhere, plus about 11 s per shard to
collect the file. scripts/ci-test-costs.mjs also refreshes the per-test table
and drops side-queue files that no longer run whole.
2026-10-15 22:43:42 -05:00
Chris Tate fc7aa5d63a Keep the long three.js case and the longest side-queue files apart 2026-10-15 21:26:41 -05:00
Chris Tate 97eb372fe1 CI: run the macOS, Effect and bootstrap lanes on pull requests only when their areas change (#1071) 2026-10-15 21:23:10 -05:00
Chris Tate 5f954cfbb0 CI: run the macOS, Effect and bootstrap lanes on pull requests only when their areas change 2026-10-15 20:27:04 -05:00
Chris Tate a08abf4abc Balance the corpus shards by each program's measured cost
The differential and llvm-differential corpus split their programs across the ten Linux test shards by a hash of
the program's name, so a shard's share of the measured work varied from 565 to 913 s (plain) and 868 to 1358 s
(sanitized). corpusSelect assigns programs longest first to the lightest shard by their measured cost per flavor
(tests/harness/corpus-costs.json: the differential plus llvm-differential milliseconds on a CI runner, medians of
four green main runs), giving 740-741 s and 1098-1099 s. Both harnesses still select identically, so a shard's
compile cache serves both. Programs without a measurement weigh as much as the median one.
scripts/ci-corpus-costs.mjs refreshes the table from downloaded CI shard logs.
2026-10-15 19:49:01 -05:00
Chris Tate d0cff5e6f3 Run the npm case fixtures natively and fix what they found
The npm case fixtures lost their only native test with the native-only
compilation change. tests/harness/npm.test.ts runs every case again with the
default static npm compile and compares it with Node. A KNOWN table records
the cases the compile refuses (with the diagnostic code), the cases whose
run reaches a deferred refusal (the code in the program's output), and the
two that still diverge, which run as expected failures. The suite joins the
case-sharded CI corpus files.

Fixes from the run:

- A package imported only by subpath (`pkg/env.js` from a package with no
  "main", "exports" or index file) was refused as having no runtime entry.
  Eligibility now resolves the imported specifier when the root has none.
- diagnostics_channel: a throwing subscriber stopped publish, and a
  throwing bindStore transform stopped runStores. Node runs the remaining
  subscribers, skips that store, and reports each error as uncaught on a
  next tick; the runtime now does the same.
- A '#' import that the package's "imports" field maps to another package
  failed at run time with Node's "not defined" error, which Node does not
  raise. It is now a compile-time SC1010 refusal.
- An uncaught runtime refusal whose code is an SC code now names the code.
- Mapping a type in the optional array-element analysis could lower an
  unsupported local class and let its PoisonError escape as a compiler
  crash; the analysis now treats that element ABI as unknown, so the
  program fails with the recorded diagnostic.
- A lexical module binding initialized from an unchecked array element
  holding a function failed validation (SC9001): its widened storage was a
  union, which a TDZ guard cannot hold. Such bindings now keep checked-value
  storage and their guard.
- A dynamically reached ArrayBuffer had no constructor property.
2026-10-15 14:15:33 -05:00
Chris Tate b249cc6bae Regenerate the runtime header split on current main 2026-10-15 03:01:21 -05:00
Chris Tate 72cd278f17 Run the Mach-O library and arm64 string oracle tests in the pull request macOS suite (#994) 2026-10-15 02:34:39 -05:00