Commit Graph
7 Commits
Author SHA1 Message Date
Malte Ubl cd5e46843c bench-runtime: separate the workload and posture help paragraphs 2026-10-08 15:46:34 -05:00
Malte Ubl 2372b75ec2 Add an opt-in --optimization=speed posture
Speed is a third native optimization posture next to release (default,
-O2) and dev (-O0). It is optimized exactly like release at the C/LLVM
level and is the switch for optimizations that trade executable size and
build time for run-time speed, so the default release build stays inside
the binary-size budget.

- backend/optimization.ts: the NativeOptimization type, its -O class, and
  cache-key helpers. Release keeps its historical key shapes (absent).
- CLI: build/run/cache warm accept speed; usage text names the trade.
  Bootstrap's routed executable cache mirrors the posture.
- Early executable cache keys and native-feature validation include speed,
  so release, dev, and speed entries never satisfy one another.
- Runtime packs: an optional executable `speed` flavor (-O2, only beside
  release). Speed links it when present and the release objects otherwise;
  library modes link their release flavor. The sanitizer lane follows the
  same flavor key.
- Library profiles accept "speed".
- WASI release-only linker flags apply to speed too.
- bench-runtime: --baseline-optimization/--candidate-optimization and a
  per-contender cold_build_ms.

Release executables are byte-identical to the stack tip for all 14
benchmark workloads (macOS arm64, sha256 of executables built from both
trees).
2026-10-08 15:46:34 -05:00
Malte Ubl 2b131ddec3 Add server and startup workloads to the runtime benchmark suite
The suite only had single-process CPU workloads. Add the two largest
real-world categories:

- http-api (kind "server"): a node:http JSON API in the static subset
  (routing, query parsing, seeded catalog, per-cart state, typed JSON
  bodies, ETag/If-None-Match, auth and request-id headers), driven by
  clients/http-load.mjs, a raw-socket keep-alive HTTP/1.1 load client
  that is identical for every server. 64 connections, 2 client threads,
  4,000 untimed warmup and 40,000 measured requests on loopback. The
  per-connection request schedule is seeded and server state is
  partitioned per connection, so the response digest is deterministic.
  Correctness compares server stdout, server stderr (minus the PORT
  line), exit status, and client stdout with Node byte-for-byte. The
  timed sample is the measured-phase wall time; a second table reports
  req/s, p50/p90/p99 latency, client CPU, and peak RSS.
- cli-config (process, launches: 20): a deploy-plan CLI that reads a
  ~110 KB JSON manifest, validates, interpolates env values, and orders
  80 services into dependency waves. Each sample is the median of 20
  launches, so startup dominates.

The runner stays backward compatible: process workloads behave as
before, the JSON report keeps its fields and adds kind, launches,
category, server, node_peak_rss_bytes, and geomean_vs_node_by_category,
and the summary prints per-category geomeans (cpu, startup, server).
2026-10-08 15:33:00 -05:00
Malte Ubl 2a2e84c3fc bench-runtime: add --layouts to pool runs over perturbed code layouts
--layouts=N links every executable N times with 0, 16, 32, ... bytes of
retained padding just before the program object (a SCRIPTC_LINKER
wrapper that forwards to clang) and rotates the timed runs through
those layouts, so medians and CIs cover code-placement luck. Each
layout's binary is checked against Node. The default of 1 keeps the
existing behaviour and output; per-layout medians go into the JSON.
Works with any checkout's compiler, including an unmodified baseline.

Linux x86-64 (Vercel Sandbox), H23 vs perf/harness, --layouts=4 --runs=16:

| workload | baseline ms | candidate ms | change | 95% CI | verdict | size |
| --- | ---: | ---: | ---: | --- | --- | ---: |
| json-records | 754.4 | 771.2 | +2.2% | -1.9% .. +5.6% | neutral | +1.9% |
| regex-logs | 380.6 | 384.9 | +1.1% | -9.8% .. +7.3% | neutral | +2.7% |
| ast-interp | 385.9 | 395.6 | +2.5% | -5.0% .. +10.6% | neutral | +3.1% |
| records-sort | 1257.0 | 1259.6 | +0.2% | -1.9% .. +2.8% | neutral | +3.2% |
| template-render | 250.7 | 246.1 | -1.8% | -9.9% .. +2.3% | neutral | +3.7% |
| functional-pipeline | 428.8 | 396.5 | -7.5% | -9.5% .. -4.0% | faster | +4.0% |
| async-pipeline | 204.2 | 203.6 | -0.3% | -1.9% .. +1.0% | neutral | +2.5% |
| word-graph | 325.5 | 327.3 | +0.6% | -2.5% .. +3.2% | neutral | +6.5% |
| alloc-trees | 510.1 | 509.4 | -0.1% | -2.3% .. +2.2% | neutral | +4.9% |
| numeric-kernels | 119.5 | 117.6 | -1.6% | -3.1% .. -0.1% | faster | +4.4% |
| validate-errors | 201.4 | 200.4 | -0.5% | -3.2% .. +2.1% | neutral | +3.0% |
| csv-numbers | 456.3 | 443.0 | -2.9% | -3.8% .. -0.3% | faster | +4.0% |
| log-summary | 252.9 | 255.9 | +1.2% | -0.8% .. +2.3% | neutral | +3.9% |
| inventory-report | 461.3 | 452.2 | -2.0% | -6.5% .. +0.2% | neutral | +2.8% |

geomean change: -0.7%

macOS arm64, H23 vs perf/harness, 15 runs:

| workload | baseline ms | candidate ms | change | 95% CI | verdict | size |
| --- | ---: | ---: | ---: | --- | --- | ---: |
| json-records | 265.1 | 263.8 | -0.5% | -1.1% .. -0.2% | faster | -0.0% |
| regex-logs | 297.9 | 290.9 | -2.4% | -3.2% .. -0.8% | faster | +6.3% |
| ast-interp | 228.8 | 229.5 | +0.3% | -1.0% .. +1.4% | neutral | -0.0% |
| records-sort | 595.5 | 602.9 | +1.2% | +0.6% .. +2.0% | slower | -0.0% |
| template-render | 194.7 | 193.0 | -0.9% | -2.0% .. +0.4% | neutral | +8.8% |
| functional-pipeline | 219.9 | 223.6 | +1.7% | +0.8% .. +2.3% | slower | -0.0% |
| async-pipeline | 251.3 | 250.4 | -0.3% | -1.2% .. +0.7% | neutral | -0.0% |
| word-graph | 161.6 | 161.4 | -0.1% | -0.6% .. +1.2% | neutral | -0.0% |
| alloc-trees | 275.6 | 273.7 | -0.7% | -1.5% .. +0.0% | neutral | +0.0% |
| numeric-kernels | 67.5 | 67.3 | -0.2% | -1.8% .. +0.9% | neutral | -0.0% |
| validate-errors | 144.4 | 143.3 | -0.8% | -1.4% .. +0.5% | neutral | +0.0% |
| csv-numbers | 276.3 | 276.3 | +0.0% | -0.7% .. +0.6% | neutral | +0.0% |
| log-summary | 149.9 | 151.3 | +1.0% | -0.2% .. +1.7% | neutral | +0.0% |
| inventory-report | 197.2 | 198.1 | +0.4% | -0.2% .. +1.1% | neutral | -0.0% |

geomean change: -0.1%

No-op change (+32 B program object), Linux, 15 runs: baseline
template-render +10.8% slower (CI +9.3..+12.9%); with H23 every
workload is neutral (regex-logs +1.4%, template-render +0.5%,
records-sort -0.4%). Details: results/h23-layout-noise/REPORT.md.
2026-10-08 15:22:27 -05:00
Malte Ubl 1dc9e04fc5 Time Node in the interleaved loop and report candidate/Node ratios
Node is the reference the project aims to beat. A single cold oracle
launch overstated its time; it is now warmed and sampled in the same
interleaved loop as the compiled executables, and the report adds a
per-workload candidate/Node ratio and its geomean. --no-node skips it.
2026-10-08 09:38:24 -07:00
Malte Ubl 523a8271f8 Skip peak RSS when /usr/bin/time is unavailable 2026-10-08 09:38:24 -07:00
Malte Ubl 10b246d604 Add a runtime benchmark suite with Node-verified A/B timing
Fourteen application-shaped workloads (JSON, regex, parsing and class
dispatch, sorting, rendering, collection pipelines, async fan-out,
Map/Set graphs, allocation, numeric kernels, exceptions, CSV, and the
two multi-module build benchmark apps on large inputs).

bench:runtime builds each workload with one or two checkouts, refuses to
time any executable whose stdout, stderr, or exit status differs from
Node, interleaves baseline and candidate launches, and reports medians
with bootstrap 95% confidence intervals, binary size, and peak RSS.
2026-10-08 09:38:23 -07:00