scriptc/benchmarks/runtime/README.md

Runtime benchmarks

Application-shaped workloads for measuring the speed of compiled executables. Each workload is deterministic, generates its own input (or reads a generated file), and prints a summary that must match Node.js byte-for-byte before it is timed.

Workload	Category	Exercises
json-records	cpu	JSON.stringify, typed JSON.parse, Map aggregation
regex-logs	cpu	regex exec with groups, test, global replace
ast-interp	cpu	tokenizer, recursive-descent parser, class dispatch
records-sort	cpu	comparator sorts, group-by, reduce/filter
template-render	cpu	template literals, escaping, map/join
functional-pipeline	cpu	filter/map/reduce/flatMap chains, closures, small records
async-pipeline	cpu	async/await, Promise.all fan-out
word-graph	cpu	string-keyed Map/Set, BFS
alloc-trees	cpu	allocation-heavy trees, immutable object-spread updates
cyclic-graphs	cpu	object graphs with back references built and dropped: trees with parent and sibling links, rings of records sharing a long-lived registry, closures capturing their owner
numeric-kernels	cpu	sieve, hashing, number[][] matmul, Float64Array
typed-array-scan, typed-array-scan-workers	cpu	element-by-element scan loops over Uint8Array, Uint16Array, Int32Array and Float64Array; the second entry is a worker-enabled executable (it can start a worker), so each access also checks for shared storage
intern-table, intern-table-map, intern-table-1, intern-table-map-1	cpu	worker threads interning the same type-like objects under string keys and records under number keys, then hits and misses: one shared InternTable per kind of key, or a module-level Map per thread (the -map entries), with 8 threads or 1 (the -1 entries); under Node.js every thread has its own tables
thread-slots, thread-slots-cache	cpu	worker threads reading one published type graph from an InternTable and keeping per-thread state for each type they visit: a ThreadSlots family, or a direct-mapped cache by id in front of a Map by id (the -cache entry); under Node.js both are per thread
atomics-contended	cpu	four worker threads updating SharedArrayBuffer counters with Atomics: one shared counter, one counter per worker, a compareExchange loop, and Atomics.wait/notify handoffs
value-objects	cpu	small immutable records and class instances: vector helpers, method chains, particle updates
value-records	cpu	records that hold strings, spread state updates and destructured tuples: token helpers, an immutable reducer, tuple returns
region-builds	cpu	a watch-mode build loop over LifetimeRegion: each build binds a program in a new region and releases the previous one, which a cache keeps referenced for two builds
optional-fields	cpu	records and class instances with number | undefined and other scalar-union fields
record-boundaries	cpu	records built from unchecked array reads passed to and returned from annotated functions
number-maps	cpu	number-keyed Map/Set: a dense-id table, sparse ids in many small maps, get/has/set mixes
published-graphs	cpu	syntax trees with parent links and a symbol table, published with @scriptc/threads and then read
validate-errors	cpu	throw/catch on a fraction of inputs, small call chains
csv-numbers	cpu	split, Number, parseInt, toFixed
log-lines	cpu	200k immediate console.log/process.stdout.write lines into the runner's pipe
log-summary, inventory-report	cpu	the multi-module applications from benchmarks/builds, on 300k-line inputs
cli-config	startup	a deploy-plan CLI: read a ~110 KB JSON manifest, validate, interpolate env values, order 80 services into dependency waves, print a summary
http-api	server	a node:http JSON API: routing, query parsing, a seeded catalog, per-cart state, typed JSON bodies, ETag/If-None-Match, auth and request-id headers

Compare two checkouts (each with a built CLI and native artifacts for the host):

$ pnpm bench:runtime --baseline=../scriptc-main --candidate=. --runs=15 --json=result.json

Without --baseline, the runner reports absolute medians. Baseline and candidate launches are interleaved, and each workload reports the median ratio with a bootstrap 95% confidence interval. faster/slower verdicts require the whole interval on one side of 1.0. A per-host lock serializes the timed phase so concurrent runs on one machine do not distort each other. Executables that time out or diverge from Node are reported, never timed. Use --workloads=cli-config,http-api to run a subset.

Node is timed in the same interleaved loop and every row reports candidate/Node; the summary prints the overall geomean and, when several categories ran, the geomean per category (cpu, startup, server). Node runs each .ts entry directly, so its times include type stripping and ESM detection, as a user running the TypeScript source would see.

Workload kinds

Entries in workloads.json are process workloads unless they set kind.

Process workloads time whole launches (wall time, startup included). An optional launches count makes each sample the median of that many sequential launches; cli-config uses 20 so that a startup-dominated command (a few milliseconds compiled, tens of milliseconds under Node) is measured with low noise. Its table row is labeled (per launch). On a heavily loaded macOS host, whole batches of launches can run several times slower (samples become bimodal), so judge startup changes on the Linux lane or a quiet machine.

Server workloads ("kind": "server") run a listening program and a load client:

• The server binds 127.0.0.1 on port 0 and reports PORT <n> on stderr, the same protocol as tests/harness/server.test.ts.
• The runner then starts the workload's client script under the runner's Node. clients/http-load.mjs is the client for every server, Node and compiled alike, so the numbers differ only by server. It drives keep-alive HTTP/1.1 connections over loopback with one request in flight per connection (closed loop, no pipelining), spread across worker threads so that the client has CPU headroom.
• Each connection issues a fixed, seeded request sequence; mutable server state is partitioned per connection, so every response is a pure function of the connection and request index regardless of interleaving. A session runs an untimed warmup phase (which also opens the connections), a barrier, then the measured phase, and finally POST /shutdown, after which the server prints its totals and exits.
• Correctness compares four legs against Node byte-for-byte: server stdout, server stderr without the PORT line, server exit status, and client stdout (request and status counts plus a SHA-256 digest over every response's status, content-type, cache-control, etag, x-request-id, and body). Framing (content-length or chunked) and date are not compared.
• The timed sample is the measured-phase wall time for the fixed request count, so the main table, ratios, and confidence intervals read exactly like other rows (lower is better); the row is labeled (server). A second table reports median requests per second, p50/p90/p99 latency, client CPU cores used during the measured phase, and peak RSS for Node and each contender. Client cores near the client's thread count mean the client, not the server, limits throughput.
• Server sessions already warm up in-process, so the runner uses at most one extra warmup session per contender.

http-api uses 64 connections, 2 client threads, 4,000 warmup and 40,000 measured requests (about one second per session).