Malte Ubl
2a2e84c3fc
bench-runtime: add --layouts to pool runs over perturbed code layouts
...
--layouts=N links every executable N times with 0, 16, 32, ... bytes of
retained padding just before the program object (a SCRIPTC_LINKER
wrapper that forwards to clang) and rotates the timed runs through
those layouts, so medians and CIs cover code-placement luck. Each
layout's binary is checked against Node. The default of 1 keeps the
existing behaviour and output; per-layout medians go into the JSON.
Works with any checkout's compiler, including an unmodified baseline.
Linux x86-64 (Vercel Sandbox), H23 vs perf/harness, --layouts=4 --runs=16:
| workload | baseline ms | candidate ms | change | 95% CI | verdict | size |
| --- | ---: | ---: | ---: | --- | --- | ---: |
| json-records | 754.4 | 771.2 | +2.2% | -1.9% .. +5.6% | neutral | +1.9% |
| regex-logs | 380.6 | 384.9 | +1.1% | -9.8% .. +7.3% | neutral | +2.7% |
| ast-interp | 385.9 | 395.6 | +2.5% | -5.0% .. +10.6% | neutral | +3.1% |
| records-sort | 1257.0 | 1259.6 | +0.2% | -1.9% .. +2.8% | neutral | +3.2% |
| template-render | 250.7 | 246.1 | -1.8% | -9.9% .. +2.3% | neutral | +3.7% |
| functional-pipeline | 428.8 | 396.5 | -7.5% | -9.5% .. -4.0% | faster | +4.0% |
| async-pipeline | 204.2 | 203.6 | -0.3% | -1.9% .. +1.0% | neutral | +2.5% |
| word-graph | 325.5 | 327.3 | +0.6% | -2.5% .. +3.2% | neutral | +6.5% |
| alloc-trees | 510.1 | 509.4 | -0.1% | -2.3% .. +2.2% | neutral | +4.9% |
| numeric-kernels | 119.5 | 117.6 | -1.6% | -3.1% .. -0.1% | faster | +4.4% |
| validate-errors | 201.4 | 200.4 | -0.5% | -3.2% .. +2.1% | neutral | +3.0% |
| csv-numbers | 456.3 | 443.0 | -2.9% | -3.8% .. -0.3% | faster | +4.0% |
| log-summary | 252.9 | 255.9 | +1.2% | -0.8% .. +2.3% | neutral | +3.9% |
| inventory-report | 461.3 | 452.2 | -2.0% | -6.5% .. +0.2% | neutral | +2.8% |
geomean change: -0.7%
macOS arm64, H23 vs perf/harness, 15 runs:
| workload | baseline ms | candidate ms | change | 95% CI | verdict | size |
| --- | ---: | ---: | ---: | --- | --- | ---: |
| json-records | 265.1 | 263.8 | -0.5% | -1.1% .. -0.2% | faster | -0.0% |
| regex-logs | 297.9 | 290.9 | -2.4% | -3.2% .. -0.8% | faster | +6.3% |
| ast-interp | 228.8 | 229.5 | +0.3% | -1.0% .. +1.4% | neutral | -0.0% |
| records-sort | 595.5 | 602.9 | +1.2% | +0.6% .. +2.0% | slower | -0.0% |
| template-render | 194.7 | 193.0 | -0.9% | -2.0% .. +0.4% | neutral | +8.8% |
| functional-pipeline | 219.9 | 223.6 | +1.7% | +0.8% .. +2.3% | slower | -0.0% |
| async-pipeline | 251.3 | 250.4 | -0.3% | -1.2% .. +0.7% | neutral | -0.0% |
| word-graph | 161.6 | 161.4 | -0.1% | -0.6% .. +1.2% | neutral | -0.0% |
| alloc-trees | 275.6 | 273.7 | -0.7% | -1.5% .. +0.0% | neutral | +0.0% |
| numeric-kernels | 67.5 | 67.3 | -0.2% | -1.8% .. +0.9% | neutral | -0.0% |
| validate-errors | 144.4 | 143.3 | -0.8% | -1.4% .. +0.5% | neutral | +0.0% |
| csv-numbers | 276.3 | 276.3 | +0.0% | -0.7% .. +0.6% | neutral | +0.0% |
| log-summary | 149.9 | 151.3 | +1.0% | -0.2% .. +1.7% | neutral | +0.0% |
| inventory-report | 197.2 | 198.1 | +0.4% | -0.2% .. +1.1% | neutral | -0.0% |
geomean change: -0.1%
No-op change (+32 B program object), Linux, 15 runs: baseline
template-render +10.8% slower (CI +9.3..+12.9%); with H23 every
workload is neutral (regex-logs +1.4%, template-render +0.5%,
records-sort -0.4%). Details: results/h23-layout-noise/REPORT.md.
2026-10-08 15:22:27 -05:00
Malte Ubl
10b246d604
Add a runtime benchmark suite with Node-verified A/B timing
...
Fourteen application-shaped workloads (JSON, regex, parsing and class
dispatch, sorting, rendering, collection pipelines, async fan-out,
Map/Set graphs, allocation, numeric kernels, exceptions, CSV, and the
two multi-module build benchmark apps on large inputs).
bench:runtime builds each workload with one or two checkouts, refuses to
time any executable whose stdout, stderr, or exit status differs from
Node, interleaves baseline and candidate launches, and reports medians
with bootstrap 95% confidence intervals, binary size, and peak RSS.
2026-10-08 09:38:23 -07:00
Malte Ubl