Commit Graph
29 Commits
Author SHA1 Message Date
Malte Ubl f9419b5d47 Merge remote-tracking branch 'origin/ni/1-libnode-loader' into HEAD
# Conflicts:
#	packages/compiler/src/index.ts
2026-10-27 05:40:18 -07:00
Chris Tate df77671e33 Merge main 2026-10-26 23:13:34 -05:00
Chris Tate 2b8873bb92 Keep the native compiler's profile matching its functions (#2357) 2026-10-26 18:07:49 -05:00
Chris Tate 5ab76a4d54 Keep the native compiler's profile matching its functions 2026-10-26 15:31:14 -05:00
Malte UblandClaude Opus 5.5 5f3e97998d Merge the macOS island fixes of ni/1-libnode-loader
The libnode-linux action becomes .github/actions/libnode, which serves
the Linux x64 and macOS lanes: the libnode-packs cache for the runner's
platform, else the development pack.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 12:50:39 -07:00
Malte Ubl 141279b74c Merge remote-tracking branch 'origin/ni/1-libnode-loader' into ni/13-libnode-packs 2026-10-26 11:15:43 -07:00
Malte Ubl 5817be4d0f Merge remote-tracking branch 'origin/main' into ni/1-libnode-loader
# Conflicts:
#	packages/compiler/src/diagnostics/diagnostic.ts
#	packages/compiler/src/ir/function-references.test.ts
2026-10-26 11:08:44 -07:00
Chris Tate d1a0bea84e Have the native CLI packages depend on their LLVM helper and runtime pack
Each @scriptc/cli-<platform> package bundled its platform's LLVM helper
and runtime pack, which are also published as @scriptc/llvm-<platform> and
@scriptc/runtime-<platform>. The CLI packages now depend on those packages
at the same version (workspace:*, which pnpm pack writes as the exact
version) and leave them out: cli-linux-x64-gnu goes from 115 MB to 60 MB
packed.

- build-native-cli.mts --platform-packages, which each CLI package's
  build:native passes, writes a manifest without llvm_package, runtime_pack
  and runtime_packs.
- loadNativeToolchain resolves a missing helper or host runtime pack as
  Node resolves the package from the compiler's directory: nested in the
  platform package, beside it (hoisted, or pnpm's virtual store) or higher
  up. It requires the compiler's version, and an absent or mismatched
  package fails with an error that names it.
- installNativeCli records where the package manager put both packages in
  the installed manifest, with the same checks.
- verify-native-cli.mjs requires the exact dependencies of a platform
  package and checks the resolved packages' versions.
- package-native-cli.mjs bundles the helper and runtime pack into the
  standalone archives, which stay self-contained.
- smoke-native-install.mjs installs the helper and runtime pack tarballs
  with the platform package; the bootstrap's exported package drops its
  bundled copies first.
- release.yml publishes the CLI packages only once every helper and
  runtime pack package resolves on npm, and scriptc only once the CLI
  packages resolve, each wait at most 25 minutes. A rerun skips the
  uploads npm already accepted and waits again.
2026-10-26 07:48:11 -05:00
Chris Tate c3b265f68c Merge main 2026-10-26 04:39:20 -05:00
Chris Tate 71de76b813 Order the native compiler's cold functions by how many builds run them 2026-10-26 02:02:18 -05:00
Chris Tate 3c057b501f Lay out the native compiler's functions by the code each kind of run executes
A warm rebuild ran about 3.5 MB of the native compiler's code, but the code
was spread over its 69 MB text, so 64 KB fault-around mapped about 20 MB of
it. The emitter can now take a function layout: a list of functions in
tiers. It writes the listed definitions, generated helpers included, first
and by tier, each in a `layout-<tier>` partition group. The LLVM helper
compiles each partition's definitions in text order, so the listed ones
come first in their partitions, and a helper that places functions by
group keeps a tier's definitions in the same partitions. The rest keep
their order and their source-file groups.

- compile() takes a `functionLayout` file (`<tier> <symbol>` lines,
  gzip-compressed when the name ends in .gz). Keys leave out module
  numbers.
- build-native-cli.mts uses the committed packages/compiler/native/cli.layout.gz,
  or the file SCRIPTC_NATIVE_LAYOUT names; an empty value and instrumented
  builds use none.
- scripts/refresh-cli-layout.mjs builds an instrumented compiler, builds
  the runtime workloads and a corpus slice cold and then unchanged, and
  reads the functions each run executed from the raw profiles. Tier 1 is
  `--version` and rebuilds that hit both caches; tier 2 is the other builds.
- The release workflow's cli-profile job also collects a layout on the
  release commit, and the CLI builds use it. Names of closures and
  generated helpers change with nearby edits, so a committed layout
  matches fewer functions as main moves on.
- Forked lowering and emission workers reset the profile counters after
  fork and write their profile before _exit, so instrumented builds also
  count the work done in workers. This applies to refresh-cli-profile.mjs
  too.

Executables the compiler produces are unchanged.
2026-10-25 16:44:46 -05:00
Chris Tate e0f1de0a2b Merge main 2026-10-24 20:00:43 -05:00
Chris Tate 40faf6a9f1 Collect the native compiler profile on the release commit
The release workflow's new cli-profile job runs
scripts/refresh-cli-profile.mjs on Linux x64 and uploads the profile.
Every platform's native CLI build uses it through
SCRIPTC_NATIVE_PROFILE_USE, and the GitHub release attaches it as
scriptc-<version>-cli.profdata.gz. When the job fails, the CLI builds
use the committed profile.

build-native-cli.mts accepts a gzip-compressed profile in
SCRIPTC_NATIVE_PROFILE_USE when the name ends in .gz.
refresh-cli-profile.mjs creates the directory of --output before it
starts.
2026-10-24 14:16:32 -05:00
Malte Ubl 43cd8dea21 Merge ni/1-libnode-loader into ni/13-libnode-packs 2026-10-21 20:12:41 -07:00
Malte Ubl 680af076cf Merge remote-tracking branch 'origin/main' into ni/1-libnode-loader
# Conflicts:
#	packages/compiler/src/diagnostics/diagnostic.ts
#	packages/compiler/src/index.ts
2026-10-21 20:07:52 -07:00
Malte UblandClaude Opus 5.5 75aa8509f6 Assemble libnode packs with NI-1's build-pack.mjs and run island tests in CI
Stack the libnode packs on the in-process loader (#1685) and use its
pack assembler and manifest format instead of a second one:
build-pack.mjs keeps its CLI and fields (schema, nodeVersion,
moduleVersion, bridgeAbi, platform, arch, files), adds the libnode and
bridge names, strips and (on macOS) re-signs the bridge and links it with
-z defs, and exports buildPack(), which the platform packs call with their
package, target and build inputs as extra manifest fields.

libnodeRecord falls back to the installed @scriptc/libnode-<platform>
pack, found from the entry's directory and then next to the compiler,
when neither --libnode nor SCRIPTC_LIBNODE names one.

The Linux test lanes restore the libnode build that the libnode-packs
workflow cached, assemble the linux-x64 pack from it in seconds
(build.mjs --from-cache never builds Node) and export SCRIPTC_LIBNODE, so
the Node island corpus and harness tests run instead of skipping.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 17:52:22 -07:00
Malte UblandClaude Opus 5.5 e9c6b68c07 Build prebuilt libnode packs for Node islands
Add the @scriptc/libnode-<platform> packs (linux-x64-gnu, linux-arm64-gnu,
darwin-arm64, darwin-x64): the pinned Node.js 24.22.0 built as a stripped
shared library with full ICU and its startup snapshot, the scriptc bridge
compiled against that release's headers, Node's LICENSE and a manifest
with the Node version, NODE_MODULE_VERSION, bridge ABI and file hashes.

packages/libnode-common holds the pin, the build (fetch and verify the
source tarball, configure --shared, build only the libnode target, strip,
stage with headers in a cache keyed by release, flags, platform and
toolchain; the bridge rebuilds in seconds against the cached headers),
a verifier, and a smoke test: a C program that loads a pack the way the
runtime does, starts Node from the snapshot and evaluates expressions
synchronously, asynchronously and throwing.

The libnode-packs workflow builds the packs on native runners, Linux in
the manylinux_2_28 images for a glibc 2.28 floor, caches each libnode
build, smoke-tests and uploads every pack. It never runs in the ordinary
build or test path. The compiler gains the pack lookup (SCRIPTC_LIBNODE,
then the installed package), validation, the nodeIslands.libnode
placements and the license notices for the Node island loader to use.

The bridge source and its C ABI header are the NI-1 bridge agent's,
included here so the packs build; the packs stay private until the
in-process island runtime ships.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 17:52:22 -07:00
Chris Tate 2b48451cf4 Build the native compiler with profile-guided optimization 2026-10-21 15:33:37 -05:00
Malte UblandClaude Opus 5.5 51e9812493 Call into Node.js from compiled programs with @scriptc/node
Scope-B milestone NI-1, API side: the explicit call API that replaces the
build container's hand-written node subprocess (api patch 09).

- The new package @scriptc/node exports callSync(specifier, name, ...args)
  and call(...). The module loads with require() from the calling module;
  arguments and results cross as JSON copies; a thrown Error crosses as its
  nearest intrinsic class with its message and string code, other thrown
  values as JSON copies; callSync refuses a function that returns a
  promise. Its Node implementation runs the calls in-process with the same
  copies, so a program prints the same under Node and compiled.
- The compiler lowers calls to the named or namespace imports
  (node-api-import.ts, lower-node-island.ts) to the nodeIsland.callSync
  and nodeIsland.call libCalls: the calling module's real path, the
  specifier, the export name and the boxed argument array, with the result
  checked against the declared type like `JSON.parse(s) as T`. Arguments
  and results whose types are not JSON data are refused at build time, and
  the functions and the namespace refuse as values.
- Corpus programs 6800-6803 (sync and async results, errors, process.exit
  from Node with the program's exit listeners, Node's pending timers
  keeping the program alive) run under Node and as binaries that load
  libnode; `// @requires-libnode` skips them where SCRIPTC_LIBNODE names no
  pack (until the pack CI matrix publishes packs). The node-island harness
  covers loading Node only on the first call, the beside packaging, a
  missing pack, a pack of another Node version or bridge ABI, and the
  executable's symbols.
- Release tooling publishes @scriptc/node with the other packages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 09:59:55 -07:00
Chris Tate 090ccabb72 Add LifetimeRegion to @scriptc/experimental 2026-10-13 21:12:20 -05:00
Malte Ubl 272aef3247 Release @scriptc/threads with the other packages
sync-versions stamps packages/threads with the CLI version (0.2.7 now),
the release workflow checks that version and publishes the package
before scriptc, and RELEASING.md lists it with the packages that always
release together. npm trusted publishing must be configured for
@scriptc/threads before the next release: a package without it fails
before upload.
2026-10-10 01:00:36 -05:00
Chris Tate ee9f0d64eb feat: ship the self-hosted native scriptc CLI (#585)
* feat: ship the self-hosted native scriptc CLI

- Run the shared compiler and CLI natively, including library builds and compile-time evaluation.
- Install native platform packages and ship relocatable standalone distributions.
- Preserve program semantics and reuse validated build artifacts for fast rebuilds.

* fix: complete native CLI integration and validation

- Preserve library diagnostics and run recursive compiler work on the main stack.
- Restore packaged helper permissions and exercise native npm and Wasm library builds.
- Recover interrupted Sandbox status probes within the original command deadline.
2026-09-30 02:17:56 -05:00
Chris Tate c41f1a8be9 Expand LLVM native output targets (#280)
* Expand LLVM native output targets

- Add owned helper and runtime-pack contracts for macOS x64, Linux glibc/musl, Windows x64, and WASI.
- Generalize LLVM target selection, package validation, linker plans, and release/CI matrices.
- Document linker boundaries and add focused native-output and runtime-pack coverage.

* Pin Windows LLVM helper CI to VS 2022

The official LLVM archive needs the Visual Studio 2022 CMake generator, which is not guaranteed by windows-latest.

* Align bootstrap cache with LLVM runtime-pack builds

Use the same target-specific helper and linker identity before and after compiler loading so routed executable cache hits remain lightweight.

* Install Windows LLVM CI tools with Chocolatey

windows-2022 includes Visual Studio but not winget; use its available Chocolatey bootstrap for CMake and Ninja.

* Prevent Zig version probes from leaving runtime-pack objects

Run runtime-pack compiler version probes in a private temporary directory and remove the accidental tracked WASI a.o file.

* Use 7-Zip for Windows LLVM setup

Extract the official LLVM development archive with two-stage 7-Zip instead of Windows tar.exe, which timed out while materializing the toolchain tree.

* Define ssize_t for the MSVC runtime pack

Clang's MSVC target does not expose POSIX ssize_t through sys/types.h; define the pointer-sized runtime type without affecting MinGW.

* Build the Windows runtime pack with Zig

Use the MinGW-compatible Windows sysroot required by the runtime while retaining COFF helper output and installing Zig in the Windows native CI lane.

* Give macOS LLVM differential shard time to finish

Shard 1 passed its setup and focused contracts but was cancelled during its cold LLVM differential slice at the 20-minute job limit.
2026-09-01 13:41:34 -05:00
Chris Tate 64b2a285d5 feat(runtime): ship precompiled macOS artifacts (#248)
* feat(runtime): ship precompiled macOS artifacts

- Add a versioned macOS arm64 runtime pack with deterministic feature selection and artifact verification.
- Link LLVM helper objects against release-built runtime and vendor inputs without compiling user-machine C.
- Wire package publishing, cache validation, documentation, and full-gate coverage for the new path.

* fix(runtime): harden precompiled artifact handling

* fix(runtime): make precompiled artifacts reproducible

* fix(runtime): reject opaque linkers from executable cache

* fix(runtime): normalize precompiled archive metadata

* fix(runtime): harden precompiled artifact caching

* fix(runtime): close executable cache link race

* fix(runtime): stage verified pack artifacts

* fix(runtime): bracket helper cache inputs

* fix(runtime): preserve executable link identity

* fix(runtime): trace selected linker dependencies
2026-08-28 09:29:21 -05:00
Chris Tate 401338f437 feat(compiler): add native LLVM object emission (#242)
* feat(compiler): add native LLVM object emission

- ship the pinned macOS arm64 LLVM helper and platform package
- add assembly/object CLI outputs, caching, diagnostics, and runtime ABI checks
- validate helper artifacts through CI, packaging, docs, and differential coverage

* fix(compiler): harden native LLVM emission

* fix(compiler): harden LLVM helper packaging

* fix(compiler): stabilize LLVM helper package size

* fix(compiler): harden native helper validation

* fix(compiler): harden LLVM helper verification

* fix(compiler): harden native object emission
2026-08-27 10:30:40 -05:00
Chris Tate af333016d4 perf: preseed native build caches (#192)
* perf: preseed native build caches

- Warm runtime, TLS, and dynamic-engine artifacts during npm installation
- Add an explicit toolchain-aware cache warm command for images and CI
- Move vendor artifacts into the user cache and build QuickJS directly

* fix: harden native cache warming

* fix: promote staged vendor cache entries

* fix: isolate shared native cache entries

* fix: validate shared vendor cache artifacts
2026-08-21 19:52:25 -05:00
Chris Tate d9b0e4fb30 Require cross-library gate before releases
- Add the cross-target library conformance lane to the release checklist.
- Require its artifact-portability verdict before the publishing commit.
2026-07-30 15:52:53 -05:00
Chris Tate d66c36b020 Ship a per-release surface manifest with stable ids and refusal codes
- packages/compiler/surface-manifest.json is generated (pnpm manifest), committed, and shipped in @scriptc/compiler; entries project mechanically from the diagnostics registry, the unsupported-syntax dispatch tables, the stdlib/builtin lowering tables, and the supported-builtin-module list
- every non-static entry carries the SC code the compiler raises for it, and the version spine is the exact published release version
- FENCE_CODES joins the diagnostics registry so factory-minted construct fences are enumerable
- the sampling harness compiles listed-static probes and asserts each sampled non-static entry refuses with exactly its listed code, beside a byte-identical staleness guard
- the release workflow regenerates the manifest, fails on drift, and attaches it to the GitHub release
2026-07-22 22:01:45 -05:00
Chris Tate 9deedfd074 Add the agent-driven release process: docs, sync script, tag job
- RELEASING.md documents the lockstep three-package prepare flow and the marked-changelog convention
- CHANGELOG.md starts with an Unreleased section and the marked 0.0.1 entry that becomes the GitHub release body
- AGENTS.md states repo-wide build/test conventions and defers docs-site specifics to docs/AGENTS.md
- scripts/sync-versions.mjs stamps runtime and compiler from the CLI version; the workflow's sync check now hints at it
- release.yml gains a github-release job that tags v<version> after a successful publish, never gating npm
2026-07-22 21:03:10 -05:00