Commit Graph
168 Commits
Author SHA1 Message Date
Malte Ubl 56abb3d77a Merge ni-2-node-island-report (#1691) into the Node islands base 2026-10-22 14:45:33 -07:00
Malte Ubl 5cb2329780 Merge remote-tracking branch 'origin/main' into ni-2-node-island-report 2026-10-22 14:31:26 -07:00
Malte Ubl 13b00dd78b Merge remote-tracking branch 'origin/main' into ni/1-libnode-loader
# Conflicts:
#	packages/compiler/src/frontend/lowering/lower-modules.ts
#	tests/harness/differential.test.ts
#	tests/harness/linux-differential.test.ts
#	tests/harness/llvm-differential.test.ts
#	tests/harness/windows-differential.test.ts
2026-10-22 14:10:07 -07:00
Chris Tate 9aaebbab25 Prepare v0.4.2 release 2026-10-22 11:14:27 -05:00
Chris Tate f87a04fa06 Build typed 'message' listener parameters from the message
A 'message' listener with a typed parameter received each message as a
checked-dynamic value: the port decoded the message into dyn objects, the
listener's thunk matched the value against each union arm, read every
field again to build the record, and the dyn objects were then released.
For bench/channel-messages that was about a third of the delivery time.

- A closure registered with on, once, addListener, prependListener or
  prependOnceListener for 'message' gets a typed message entry when its
  parameter is a number, boolean, string, a record of those without union
  fields, or a boxed union of those arms with null or undefined. The
  entry builds the parameter from the message's nodes and calls the
  closure.
- The entry handles a message only when it has the parameter's exact
  layout: the union arm the checked conversion picks, and a record's
  declared keys in declaration order with no others. Any other message
  runs through the thunk on the decoded value, as before, so undeclared
  keys, other orders and mismatches behave as they did.
- Ports, workers and parentPort deliver a message through the entries
  when no value in it is reached twice and it holds no ports or native
  handles. Listeners without an entry share one decoded value, made when
  the first of them runs.
- The message readers are runtime entry points, so the runtime ABI is now
  27.

Corpus 7130 and 7131 check typed listeners on MessageChannel ports, a
worker and parentPort against Node: exact layouts, extra keys, shared
values, nested records, untyped listeners beside typed ones, once
listeners and a transferred port.
2026-10-22 07:08:26 -05:00
Chris Tate daac2a4a15 Support objectMode streams with any value as a chunk (#1785) 2026-10-22 05:55:45 -05:00
Chris Tate c086610b94 Keep each build's translation unit private until the build ends (#1766) 2026-10-22 05:50:07 -05:00
Chris Tate 6bf3b7d9df Install driver-built translation units before the native build
The private copy of the LLVM text had no extension. Builds that compile it
with the C driver (sanitize builds, targets without the LLVM helper) passed
it to clang directly when the native cache was off, and clang read it as a
linker input, so the build failed. The copy is now named with `.ll`.

Those builds also record the output directory in their native cache proof.
Renaming the private copies to the shared names after the native build
changed that directory, so every unchanged repeat missed the proof and
relinked the executable. The shared names are now installed before the
native build, which compiles and caches copies in a private directory
outside the output directory.
2026-10-22 05:44:28 -05:00
Chris Tate 67247dd26b Support objectMode streams with any value as a chunk
Readable, Writable, Duplex, Transform and PassThrough take the
objectMode option, and a Duplex, Transform or PassThrough the
readableObjectMode and writableObjectMode options, as true or false
literals in an options object literal or in an options value. An
object-mode side holds any value that converts to a checked-dynamic
value: push, unshift, write, end, pipe, pipeline, for await, 'data'
listeners, Readable.from and the transform callback carry values in
TypeScript and JavaScript, and constructor-function streams and options
held in variables do the same. The object-mode high-water mark counts
values and defaults to 16. A readable buffer that mixes strings, Buffers
and other values holds them all, and push(undefined) and other values on
a byte-mode stream report ERR_INVALID_ARG_TYPE through 'error', as in
Node.

In typed code a write or transform callback, or a _write or _transform
method, may declare its chunk with any such type, which converts when
the callback runs. Writes pass the encoding argument to _write and
_transform as Node does: undefined when absent, null when a callback
took its place, otherwise the value unchanged. A parameter declared
string | null | undefined receives that value and one declared string
receives "" for undefined and null. write(chunk, encoding) on a
byte-mode stream encodes a string chunk with any known encoding and
throws ERR_UNKNOWN_ENCODING for an unknown one. Readable.from takes
typed Maps and Sets of any such value, and arrays that mix strings and
Buffers.

for await over a readable-only stream that will emit 'close' now ends
after 'close', as Node's eos does. Buffered writes that complete inside
clearBuffer run their callbacks in order, and 'finish' waits for a
deferred afterWrite that carries a callback, as Node's pendingcb does,
unless a _final callback schedules it.

The 'data' listener adapters take three slots and write and transform
callbacks read their chunk and encoding through runtime accessors, so
RUNTIME_ABI_VERSION is now 26.
2026-10-22 04:21:55 -05:00
Chris Tate d426960f45 Make Cipheriv and Decipheriv Transforms and accept function stages anywhere in pipeline()
Cipheriv and Decipheriv values are Transforms, as in Node: they can be
written and ended, piped into, used as pipeline() and stream/promises
pipeline stages, composed, read, iterated with for await, read by
stream/consumers, passed to finished(), and listened to, and their stream
properties read as in Node. Each chunk goes through the cipher's
update(), and the end pushes final() or reports its error through
'error'; writing after final() throws out of write(). The stream unit
builds a cipher's Transform on its first stream use from operations the
cipher's handle provides (ScrDynHandleOps.transform), so programs that
use ciphers without streams link as before. The cipher is a
collector-headered object behind a traced handle: its Transform's steps
hold it, and the pair is reclaimed as a cycle. Through untyped values, in
a program that uses streams, a cipher's stream properties and its on,
once, off, pipe, unpipe, read, pause, resume, setEncoding, write, end,
destroy, cork and uncork members run on its Transform; other stream
members throw a coded SC2020 error.

A Transform holds a write's callback while its readable side is at the
high-water mark and runs it at the next read, and runs it on a tick once
the readable side has ended, as Transform._write does. write() on a
PassThrough, a Transform, a Hash or a cipher used to return true however
much the readable side buffered. zlib streams applied the rule already
and keep their timing.

pipeline() accepts function stages anywhere among its streams, so runs of
async generator stages can sit on both sides of a stream. A stage can be
named by a function declaration, or a const bound to a function
expression, that nothing reassigns; its body lowers again at the
pipeline with its source typed by the pipeline. The source can also be
an array, a string, a synchronous generator object, or a synchronous
generator function without parameters: an async generator steps the
source's iterator and yields each element, and the next stage receives
each element on the same tick as with Node's for await over the source.
The runtime's function-stage entry points take a bitmask of the function
positions and one (next, read, return) helper triple for each generator
pumped into a stream, so RUNTIME_ABI_VERSION is now 25. In JavaScript,
the chunks a stage receives from the previous stage's generator read as
the strings or Buffers it yields, so their string and Buffer methods
compile.

Corpus 7055-7059 cover cipher streams in TypeScript and JavaScript, their
errors, close order and backpressure, the pipeline stages and sources
with their tick order, and Transform backpressure.
2026-10-22 01:55:38 -05:00
Chris Tate 63001cbc13 Await a suspending stackless callee without its promise 2026-10-21 23:55:37 -05:00
Malte Ubl 680af076cf Merge remote-tracking branch 'origin/main' into ni/1-libnode-loader
# Conflicts:
#	packages/compiler/src/diagnostics/diagnostic.ts
#	packages/compiler/src/index.ts
2026-10-21 20:07:52 -07:00
Malte UblandClaude Opus 5.5 e61fbf8c0b Merge origin/main into ni-2-node-island-report
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 20:07:41 -07:00
Chris Tate fa7bc7c7a0 Warn about likely-hot functions that use untyped values
A build now names the functions of the program's own files that compile
to the dynamic tier and likely run often, so a slow program with a little
untyped code says where the time goes:

  warning SC6101: 3 functions that likely run often use untyped values, which runs slower than typed code:
    advance (nbody.js:22): untyped JavaScript file, in a loop
    ...

- The functions come from the tier assignment lowering already makes
  (Lowerer.dynamicFunctions). Without a profile, a function counts as
  likely hot when a dynamic operation runs inside one of its loops, when
  a loop calls it, or when it calls itself (lowering/dynamic-tier.ts).
- A function is named only when its source says why its values are
  untyped: an `any` (for a for...of variable, the value it iterates), an
  untyped JSON.parse result, or an untyped JavaScript file. Generated
  %dyn.* helpers, module top levels and functions that are dynamic for an
  operation without a static lowering are left out.
- At most five functions are named, most likely hot first, with the
  total; dynamic-tier code in npm packages is counted per package on one
  line. All of the warning's text is in diagnostics/dynamic-tier.ts.
- SC6101 opens the SC61xx band, performance notes, with a new
  "performance" category in the diagnostics envelope; SC60xx stay the
  divergences.
- scriptc build prints it after the divergence warnings, and the
  executable cache stores it with the entry, so a cached build (including
  the CLI's routed fast path) repeats it. scriptc coverage reports it, and
  a comment-only edit no longer replays a verdict that carries it, since
  its text holds line numbers.
- --no-dynamic-tier-warning leaves it out of build and coverage;
  coverage --fail-on=dynamic fails on it.
- Docs: coverage.mdx (Dynamic tier, Gating builds and CI, the envelope
  categories) and cli.mdx.
2026-10-21 18:53:43 -05:00
Chris Tate 692569a804 Collect the compiler's heap as lowering and LLVM emission grow it 2026-10-21 15:50:32 -05:00
Chris Tate 6d19198eed Merge main 2026-10-21 12:43:37 -05:00
Malte UblandClaude Opus 5.5 091b8f1934 Report which packages and modules would run in a Node island
Adds the report-only Node island policy, configuration and detector of
the scope-B plan (milestone NI-2). Coverage now lists the npm packages a
build would run in a Node island and why, gap islands first, without
changing code generation.

- project-config.ts reads the `scriptc` key of the entry's package.json
  with a schema; an unknown value in `scriptc.packages` or
  `scriptc.nodeIslands` is SC8004 instead of being ignored. Package shims
  and the WebAssembly scanner read through it.
- node-islands/: origin (user, npm, shim, builtin; package instances),
  rules R1.1-R2.5 and the four classes, a syntactic pre-scan for R1/R2
  markers, classification of refusals and runtime fences (reason ids at
  the refusal sites that need them), the policy (auto, explicit, off;
  forced node and native packages; marked user modules) and the report
  with island closure.
- coverage: a "Node islands" section, `--explain-island=<pkg>`, a
  `nodeIslands` field in the diagnostics envelope, and `--fail-on`
  `node-islands`, `eager-node-islands` and `island-gaps`.
- CLI: `--node-islands`, `--node-island`, `--native`, `--node` (validated,
  report-only) and `--strict-user-js` (SC8101-SC8105 for R2 constructs in
  user JavaScript).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 10:22:35 -07:00
Chris Tate c9d206ee3e Run the frontend of a first executable build in a child process (#1649) 2026-10-21 11:10:14 -05:00
Malte UblandClaude Opus 5.5 9a0a3f1562 Load Node.js in-process on first use: libnode bridge and runtime loader
Scope-B milestone NI-1, runtime side. Programs whose IR has a
`nodeIsland.*` operation link scr_node_island.c and the SCR_NODE_ISLANDS
build of scr_async.c; every other program links neither, byte for byte.

- scr_node_island.c dlopens the bridge of a libnode pack on the first call
  (RTLD_NOW | RTLD_GLOBAL), checks its bridge ABI and that it links exactly
  the Node version recorded at build time, starts one Node environment and
  runs the embedded bootstrap (scr_node_island_host.cjs). A run that never
  calls into Node never maps libnode. The pack directory is SCRIPTC_LIBNODE
  at run time, else the recorded path; a `$ORIGIN/` path resolves against
  the executable's directory (the beside packaging, and the hook for
  shipping Node with the program).
- scr_node_bridge.h is the C ABI between the runtime and the bridge; the
  runtime includes no Node or V8 header. The bridge
  (packages/libnode-common/bridge/scr_node_bridge.cc) initializes Node once
  with no default signal handling, no debug-signal handler or inspector,
  no stdio reinitialization and one platform worker, creates the
  environment from the built-in snapshot, routes Node's process exit to
  the runtime's exit sequence, promotes libnode to the global scope for
  native addons, and runs calls in a CallbackScope with a TryCatch.
  build-pack.mjs assembles a pack (libnode, the bridge, manifest.json) from
  a Node --shared build and the matching headers.
- The SCR_NODE_ISLANDS build of scr_async.c adds a loop slot through which
  the runtime's loop drives Node's until NI-3 hosts the runtime's queues on
  Node's libuv: it keeps the program alive while Node has ref'd work,
  sleeps on Node's backend fd up to its next timer, and runs one
  non-blocking turn per loop turn.
- The `nodeIsland` runtime feature (rule, link features, unit table, pack
  matrix row and scr_async.c variant), the nodeIsland.call/callSync
  libCalls, the install call in main, and the build-time record of the
  pack (node-islands/libnode.ts: --libnode=<pack>|beside or
  SCRIPTC_LIBNODE, SC8005 when none is usable, Node >=24.13.1).
- The zero-cost gate and the bench nm test require that no good-code
  executable defines a Node island symbol.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 08:54:47 -07:00
Chris Tate 3cc72ac96c Prepare v0.4.1 release 2026-10-21 09:51:40 -05:00
Chris Tate 8fc36e61f2 Merge main 2026-10-21 09:17:28 -05:00
Chris Tate 0cc0674308 Prepare the first build of an executable in a frontend process
The frontend, lowering and LLVM emission of an optimized executable build run in a child
process when no earlier build of the same invocation emitted its native program object.
The parent passes the LLVM text to the native helper as the file the child wrote, so the
helper runs after the pages the frontend used are released. Later builds of the
invocation find most native partitions cached and peak in the frontend, so they, along
with development builds and builds with SCRIPTC_FRONTEND_PROCESS=0, prepare their input
in-process. The child runs with the parent's Node options without --eval, --print and
their code, or inspector, watch and test runner options. With SCRIPTC_TIMING=1, it
reports its phases on the parent's clock.
2026-10-21 08:23:18 -05:00
Chris Tate 8065be3ccd Add profile-guided optimization 2026-10-21 03:13:45 -05:00
Chris Tate dc5e0cb5bb Merge main 2026-10-20 17:46:26 -05:00
Chris Tate 282e4820d2 v0.4.0 2026-10-20 16:10:37 -05:00
Chris Tate 7b510efacd Merge main 2026-10-20 14:13:21 -05:00
Chris Tate 1d6bd1ea04 Shrink the array header to 48 bytes
ScrArr stores len and cap as 32-bit fields and no longer stores the
state pointer. The dense states share the values' allocation and run
backward from its end: the state of slot i is at
data + cap * (slot width + 1) - 1 - i. Removing slots from the front
(shift, splice(0, n)) advances data and lowers cap, which leaves every
remaining state at its address; the head room size, needed only to free
or reallocate the block, is kept in the vacated slot before data. On
64-bit targets the header goes from 64 to 48 bytes, and from 36 to 32
bytes on wasm32. Array lengths were already limited to 2^32 - 1.

The emitted code loads len and cap as i32 and zero-extends them, and
computes state addresses from data and cap. Runtime loops compute the
end of the states once per call. Local array reads load data once for
both the state and value paths. A local or strict element read whose
index is a loop's induction variable reads slots above the hole-free
prefix through the runtime: an inline state check there makes LLVM keep
-1 - i as a second induction variable, which costs a register in the
loop.

Bulk operations do not read back states they just wrote: an append to
an array whose hole-free prefix reaches its length moves the prefix bound
to the new length, a copy from below the source's prefix writes value
states without reading the source's, and releasing a string array skips
the state checks below the prefix. Copies of up to eight slots and of
short state runs use overlapping word moves instead of a call. Emitted
element access computes the state address only on the state-checked
path. Without these, programs that build many small arrays with split or
slice ran 2-4% slower than with the separate state pointer.

Integer loops over an array's length compare the counter with the i32
length field in 32 bits; the counter never exceeds 2^32 - 1 there. The
array-length stability proof accepts Map and Set reads (get, has, size)
and checked union extractions (`x!`), which run no user code, so loops
that look up maps keep the dense fast copy, which tests the hole-free
prefix once before the loop.

Small constructions with the small allocator request 6 slots, so their
blocks stay 112 bytes, now with 7 inline slots (6 after a collector
header). With 96-byte blocks of 5 slots, programs that free and reuse
many small arrays ran 15-20% slower.

RUNTIME_ABI_VERSION goes from 22 to 23.
2026-10-20 00:21:45 -05:00
Chris Tate 6c07441c24 Check programs with tsc-ts by default
scriptc's frontend talks to a TypeScript 7 API server. tsc-ts serves the
same API as tsgo, and it is now the default checker; SCRIPTC_CHECKER=tsgo
selects tsgo, which scriptc still includes.

- @scriptc/compiler depends on tsc-ts 0.0.3. The native distribution
  ships its platform package as lib/tsc-ts, named by a new tsc_ts field
  of the toolchain manifest; a manifest without it takes the executable
  of an installed tsc-ts package.
- The executable cache key includes the tsc-ts executable's identity, in
  the compiler and in the CLI's startup route.
- tsc-ts keeps the syntax trees it builds for the life of its process,
  so the source parser starts a new server after 256 KB of parsed
  source instead of letting one server grow with every file.
- A CI job compiles every corpus program with both checkers, in release
  and dev, and requires the same LLVM IR and diagnostics
  (scripts/checker-identity.mjs).
- Tests that compare with the TypeScript 7 SDK, the checker-panic
  diagnostics, and the protocol tests run against tsgo.
2026-10-18 19:55:52 -05:00
Chris Tate 3032248c11 v0.3.6 2026-10-17 13:47:41 -05:00
Chris Tate de08ab1920 v0.3.5 2026-10-16 10:33:14 -05:00
Chris Tate bf050d1077 v0.3.4 2026-10-16 01:29:54 -05:00
Chris Tate 2f8aea3279 v0.3.3 2026-10-15 11:23:16 -05:00
Chris Tate 97a9257fa5 Inline string retains and dynamic value reference counting in release builds
Release builds now emit the string retain and the dyn retain and release
fast paths inline. The dyn release decides the cases scr_dyn_release
decides by kind and calls scr_dyn_release_survivor for capsules and
handles. String releases and the other families stay runtime calls, as
string-heavy programs have many release sites. The speed posture keeps
every fast path and gains the dyn release. The runtime ABI version moves
to 21 because program objects call the new runtime entry point.
2026-10-15 05:15:33 -05:00
Chris Tate 6cc88fbf11 v0.3.2 2026-10-14 09:22:56 -05:00
Chris Tate e932714bfa Shrink the cycle-collector object header to 8 bytes 2026-10-14 04:53:07 -05:00
Chris Tate ea6428bad6 Keep the first elements of small appended arrays in the array's block 2026-10-13 23:36:22 -05:00
Chris Tate 947f665896 Store boolean arrays as one byte per element 2026-10-13 18:45:47 -05:00
Chris Tate 733ab38a0c v0.3.1 2026-10-13 13:26:10 -05:00
Chris Tate bf40cf4255 Compile regex literals to native matchers 2026-10-13 11:30:19 -05:00
Chris Tate 1334a8acd2 Lend arguments to closure bodies and comparators 2026-10-13 10:35:18 -05:00
Chris Tate 2c8dccd52b Merge main into the docs rewrite 2026-10-13 07:46:52 -05:00
Chris Tate 5df589706b Skip array state bytes below a hole-free prefix 2026-10-13 02:13:46 -05:00
Chris Tate e5ed06771f Use plain wording in the introduction and READMEs 2026-10-13 00:40:11 -05:00
Chris Tate d316552918 Rewrite docs for a coherent new-user flow 2026-10-13 00:34:50 -05:00
Chris Tate ef25cf119d Prepare v0.3.0 2026-10-12 15:57:06 -05:00
Chris Tate 7e6294ca0f Make scriptc compilation native-only 2026-10-12 12:25:30 -05:00
Chris Tate d75d7a22f0 Prepare v0.2.10 2026-10-12 10:31:02 -05:00
Malte Ubl fa55ce6b07 Keep the RC entry points of array elements in one shared descriptor
Every SCR_ELEM_REF array stored its element type's retain, release and trace entry points in its own header (24 bytes), so ScrArr was 80 bytes: an 80-byte block for arrays of acyclic elements and a 96-byte block for cycle-capable ones. The three pointers now live in one constant ScrArrOps descriptor per element type, which the compiler emits once per module (@sc_arrops.*) and the runtime keeps static, and the header holds a pointer to it plus a 4-byte `cyc` flag beside `elem` that the inline RC fast paths test. ScrArr is 64 bytes: arrays take 64- and 80-byte blocks.

scr_arr_new_ref(retain, release, trace, cap) becomes scr_arr_new_ops(ops, cap); scr_set_to_arr_ref takes the result's descriptor from its caller instead of building an array from the set's key adapters. The backend names the %ScrArr fields in SCR_ARR_FIELD. The layout change bumps the runtime ABI marker to scr_runtime_abi_v13.
2026-10-11 19:46:48 -07:00
Chris Tate fa11424e5f Prepare v0.2.9 2026-10-11 17:25:14 -05:00
Malte Ubl fc93d10871 Merge remote-tracking branch 'origin/perf/array-layout-main' into HEAD 2026-10-11 12:23:05 -07:00