Commit Graph
4 Commits
Author SHA1 Message Date
Malte Ubl 1c5dfda937 Add a scaling probe suite and strace tools for pathology hunting
benchmarks/scaling: 128 probes of common operations (arrays, strings,
Map/Set/records, JSON, closures, exceptions, regex, dates, promises,
timers, console, fs) run at 3-4 sizes under scriptc and Node with
identical-stdout checks; run.mjs flags superlinear growth and >3x Node,
--strace-only reports syscalls per operation, strace-workloads.mjs
straces the runtime benchmark suite. Results and the full probe table:
results/h40-pathology-hunt/REPORT.md.

Pathologies fixed on this branch (probe ms at 1e3/1e4/1e5/1e6, Linux):
  i in number[]        base 43/5291/timeout   -> 0.03/0.20/1.5/15   (Node 0.19/0.73/6.8/39)
  k in Record<string>  base 4.8/436/timeout   -> 0.15/1.5/20/319    (Node 0.96/8.9/100/1452)
  re.exec /g loop      base 1.1/70/7910/t.o.  -> 0.49/4.1/40/416    (Node 0.47/4.6/43/398)
  clearTimeout n live  base 1.3/10/1014/t.o.  -> 1.3/3.9/38/740     (Node 3.8/9.4/46/527)
  new RegExp repeat    base 3.9/38/368/3811   -> 0.13/1.1/11/106    (Node 0.49/2.9/23/174)
  console.error        2 writes/line          -> 1 write/line

Suite A/B, Linux x86_64 sandbox, candidate vs perf/stack/06 (--layouts=4
--runs=16), all workloads ok:

| workload | baseline ms | candidate ms | change | 95% CI | verdict | size | node ms | candidate/node |
| --- | ---: | ---: | ---: | --- | --- | ---: | ---: | ---: |
| json-records | 202.3 | 201.9 | -0.2% | -7.0% .. +3.8% | neutral | +0.0% | 458.29 | 0.44x |
| regex-logs | 166.8 | 166.4 | -0.2% | -1.6% .. +1.1% | neutral | +0.1% | 239.97 | 0.69x |
| ast-interp | 268.6 | 269.9 | +0.5% | -4.6% .. +3.8% | neutral | +0.0% | 242.7 | 1.11x |
| records-sort | 648.7 | 670.4 | +3.3% | -4.7% .. +10.0% | neutral | +0.0% | 222.69 | 3.01x |
| template-render | 69.6 | 70.7 | +1.7% | -0.2% .. +2.6% | neutral | +0.0% | 187.3 | 0.38x |
| functional-pipeline | 221.8 | 214.9 | -3.1% | -7.3% .. +0.7% | neutral | +0.0% | 249.06 | 0.86x |
| async-pipeline | 197.3 | 198.2 | +0.4% | -1.2% .. +3.0% | neutral | +0.2% | 89.25 | 2.22x |
| word-graph | 259.2 | 256.0 | -1.3% | -6.3% .. +3.0% | neutral | +0.0% | 469.48 | 0.55x |
| alloc-trees | 318.8 | 316.8 | -0.6% | -3.0% .. +1.2% | neutral | +0.0% | 127.68 | 2.48x |
| numeric-kernels | 46.4 | 46.5 | +0.1% | -1.6% .. +3.9% | neutral | +0.0% | 133.42 | 0.35x |
| validate-errors | 176.5 | 178.1 | +0.9% | -3.1% .. +4.7% | neutral | +0.0% | 702.45 | 0.25x |
| csv-numbers | 336.2 | 336.8 | +0.2% | -2.0% .. +1.2% | neutral | +0.0% | 500.14 | 0.67x |
| log-summary | 165.7 | 164.6 | -0.6% | -5.9% .. +1.9% | neutral | +0.0% | 290.81 | 0.57x |
| inventory-report | 321.9 | 320.1 | -0.6% | -6.7% .. +9.5% | neutral | +0.0% | 431.4 | 0.74x |

geomean change: +0.0%
geomean candidate/node: 0.768x (lower is better; goal: well below 1)

macOS lane not run (all measurement in the sandbox per task instructions).
2026-10-08 15:48:28 -05:00
Malte Ubl 2b131ddec3 Add server and startup workloads to the runtime benchmark suite
The suite only had single-process CPU workloads. Add the two largest
real-world categories:

- http-api (kind "server"): a node:http JSON API in the static subset
  (routing, query parsing, seeded catalog, per-cart state, typed JSON
  bodies, ETag/If-None-Match, auth and request-id headers), driven by
  clients/http-load.mjs, a raw-socket keep-alive HTTP/1.1 load client
  that is identical for every server. 64 connections, 2 client threads,
  4,000 untimed warmup and 40,000 measured requests on loopback. The
  per-connection request schedule is seeded and server state is
  partitioned per connection, so the response digest is deterministic.
  Correctness compares server stdout, server stderr (minus the PORT
  line), exit status, and client stdout with Node byte-for-byte. The
  timed sample is the measured-phase wall time; a second table reports
  req/s, p50/p90/p99 latency, client CPU, and peak RSS.
- cli-config (process, launches: 20): a deploy-plan CLI that reads a
  ~110 KB JSON manifest, validates, interpolates env values, and orders
  80 services into dependency waves. Each sample is the median of 20
  launches, so startup dominates.

The runner stays backward compatible: process workloads behave as
before, the JSON report keeps its fields and adds kind, launches,
category, server, node_peak_rss_bytes, and geomean_vs_node_by_category,
and the summary prints per-category geomeans (cpu, startup, server).
2026-10-08 15:33:00 -05:00
Malte Ubl 10b246d604 Add a runtime benchmark suite with Node-verified A/B timing
Fourteen application-shaped workloads (JSON, regex, parsing and class
dispatch, sorting, rendering, collection pipelines, async fan-out,
Map/Set graphs, allocation, numeric kernels, exceptions, CSV, and the
two multi-module build benchmark apps on large inputs).

bench:runtime builds each workload with one or two checkouts, refuses to
time any executable whose stdout, stderr, or exit status differs from
Node, interleaves baseline and candidate launches, and reports medians
with bootstrap 95% confidence intervals, binary size, and peak RSS.
2026-10-08 09:38:23 -07:00
Chris Tate 0845c7ff56 Make native installs portable and smaller (#599)
* Make native installs portable and smaller

- Support glibc 2.34 with portable compiler binaries and static LLVM helpers.
- Bundle the host runtime and discover optional cross-target packs from project dependencies.
- Add application build benchmarks covering latency, executable size, and memory.

* Keep bootstrap tools available in Sandbox validation

- Isolate Zig for native compiler bootstrap and glibc entropy contracts.
- Preserve the existing toolchain scope for cache and corpus tests.

* Fix portable compiler bootstrap linking

- Keep native support objects compatible with the destination host linker.
- Resolve installed targets using the host declared by the toolchain manifest.
2026-10-01 10:46:16 -05:00