mirror of
https://github.com/vercel-labs/scriptc.git
synced 2026-10-27 06:43:19 +08:00
Merge main
This commit is contained in:
Chris Tate
@@ -48,6 +48,12 @@ Corpus programs are differential tests against Node: every program runs under No
|
||||
|
||||
Before opening a PR that adds or changes corpus, diagnostic, or frontend fixtures, run `pnpm test:ts7 --baselines-only`. This fast completeness check uses the same fixture discovery as the parity suite; it does not type-check or replace behavioral parity. Generate missing preflight/order baselines, review their diagnostics and module order, and run the affected `order-parity.test.ts` batches before pushing. Keep existing baseline entries unchanged unless the corresponding behavior intentionally changes. The full `pnpm test:ts7` command also checks completeness before starting its sweep.
|
||||
|
||||
A change that must not cost good code anything, such as a compatibility fix that replaces a refusal, proves it with the zero-cost gate. The gate compiles the benchmark workloads, the self-hosted compiler, and the TypeScript corpus with two built checkouts. It requires byte-identical IR, LLVM, and release executables, and compile time within tolerance. `--allow=<regex>` names the change's pattern signature: programs whose sources match may differ, and the PR lists them. Any other difference fails the gate, which names the first differing function. The gate also enforces the two-tier runtime separation: no executable may gain symbols from `DYNAMIC_TIER_PREFIXES` in `scripts/zero-cost-compare.mjs`, and fully typed benchmarks may link none; extend that list as dynamic-tier units land. `--quick` checks only the benchmarks; `--cachegrind` adds per-function instruction counts for executables that differ. Run it in a Sandbox with both checkouts prepared:
|
||||
|
||||
```bash
|
||||
node scripts/zero-cost-gate.mjs --baseline=/vercel/sandbox/prev --candidate=. --allow='\barguments\b'
|
||||
```
|
||||
|
||||
Test location follows scope:
|
||||
|
||||
- Co-locate white-box unit tests with implementation files under
|
||||
|
||||
@@ -2,3 +2,4 @@
|
||||
- **Snapshots:** `t.assert.snapshot` and `t.assert.fileSnapshot` compile, with Node's default serializer, `JSON.stringify(value, null, 2)`, or an `options.serializers` array literal. A compiled test reads the snapshot file Node writes beside the entry file, `<entry file>.snapshot`, at the entry file's path when the program was compiled. Running the binary with Node's flag `--test-update-snapshots` as its first argument writes the snapshot files instead; as in Node, the flag is kept out of `process.argv` and appears in `process.execArgv`. Snapshot files are read in the form Node writes them, `` exports[`key`] = `value`; `` statements. The `snapshot.setDefaultSnapshotSerializers` and `snapshot.setResolveSnapshotPath` module functions throw `SC2020`.
|
||||
- **TestContext values:** `t.filePath` is the entry file's path when the program was compiled. `t.attempt` is `0`, since a test runs once. `t.passed` is `false` while the body runs and the test's outcome in its `afterEach` and `t.after` hooks. `t.workerId` is `Number(process.env.NODE_TEST_WORKER_ID) || undefined`, as in Node. `t.tags` is `[]`, since the `tags` test option throws `SC2020`; the array is not frozen as Node's is.
|
||||
- **Suite and hook contexts:** a `describe` body may take its `SuiteContext`, which has `name`, `fullName`, `filePath`, `attempt`, `passed` and `diagnostic`. Hooks may take their context, as in Node: the suite's for `before` and `after`, the test's for `beforeEach`, `afterEach`, `t.before` and `t.after`. The `SuiteContext` `signal` and `log` members, and hook `done` callbacks, throw `SC2020`.
|
||||
- **Module-level hooks:** `before` hooks outside any suite run at their `before()` call, and the first top-level test waits for the async ones; `after` hooks outside any suite run after the last top-level test. Both get the root's context, named `<root>`. As in Node, a throwing root `before` fails each top-level test with its error, without running its body, and cancels the tests and suites inside each top-level suite. A throwing root `after` reports the entry file as a failed result, and the exit code is unchanged.
|
||||
|
||||
@@ -473,7 +473,7 @@ function lowerHook(
|
||||
return {
|
||||
kind: "libCall",
|
||||
fn: "test.hook",
|
||||
args: [numLit(which, loc), cb, numLit(flags, loc)],
|
||||
args: [numLit(which, loc), cb, numLit(flags, loc), strLit(atStringOf(expr), loc)],
|
||||
type: VOID,
|
||||
loc,
|
||||
};
|
||||
|
||||
@@ -1438,7 +1438,8 @@ export const LIB_FN_SIGS = defineLibFnSignatures({
|
||||
},
|
||||
// The suite body takes no parameters or its SuiteContext (flag 2).
|
||||
"test.suite": { argTypes: [STRING, F64, STRING, null, STRING, F64, F64], result: VOID },
|
||||
"test.hook": { argTypes: [F64, null, F64], result: VOID },
|
||||
// The trailing string is the hook call's "file:line:col".
|
||||
"test.hook": { argTypes: [F64, null, F64, STRING], result: VOID },
|
||||
"test.sub": {
|
||||
argTypes: [TESTCTX_T, STRING, F64, STRING, null, F64, STRING, F64, F64, F64],
|
||||
result: { kind: "promise", inner: VOID },
|
||||
|
||||
@@ -80,6 +80,7 @@ enum {
|
||||
typedef struct {
|
||||
ScrClosure *cb;
|
||||
int flags;
|
||||
ScrStr *at; /* "file:line:col" of the hook call, owned; NULL for t.* hooks */
|
||||
} ScrTestHook;
|
||||
|
||||
typedef struct {
|
||||
@@ -123,6 +124,7 @@ struct ScrTestCtx {
|
||||
ScrPromise *slot; /* owned; the parent waiting for its next subtest to finish */
|
||||
struct ScrMockTimers *mock_timers; /* t.mock.timers, made on first use; freed at exit */
|
||||
bool passed; /* t.passed: set when the body passes, as Node's pass() does */
|
||||
bool listed; /* a suite listed in the failing section (a failing root before) */
|
||||
};
|
||||
|
||||
static SCR_TL ScrTestCtx *scr_test_root = NULL;
|
||||
@@ -136,6 +138,18 @@ static SCR_TL long scr_test_ct_tests = 0, scr_test_ct_suites = 0, scr_test_ct_pa
|
||||
scr_test_ct_todo = 0;
|
||||
static SCR_TL double scr_test_t0 = 0;
|
||||
static SCR_TL bool scr_test_suite_hook_failed = false;
|
||||
/* The promises of async module-level before hooks, which the runner waits
|
||||
* for before the first top-level test. */
|
||||
static SCR_TL ScrPromise **scr_test_root_pending = NULL;
|
||||
static SCR_TL size_t scr_test_root_npending = 0, scr_test_root_pending_cap = 0;
|
||||
|
||||
/* A module-level before hook threw or rejected: the first failure is kept
|
||||
* for the top-level tests and suites, later ones are dropped. */
|
||||
static void scr_test_root_before_failed(ScrTestCtx *root) {
|
||||
if (!scr_exc_pending()) return;
|
||||
if (!root->before_err) root->before_err = scr_exc_take();
|
||||
else scr_exc_clear();
|
||||
}
|
||||
/* Above 0 while subtests run concurrently: result lines are held, and the
|
||||
* outermost concurrent parent prints its subtrees in declaration order,
|
||||
* as Node's reporter does, once they all finish. */
|
||||
@@ -164,7 +178,10 @@ ScrTestCtx *scr_testctx_retain(ScrTestCtx *t) {
|
||||
}
|
||||
|
||||
static void scr_test_hooks_free(ScrTestHooks *h) {
|
||||
for (size_t i = 0; i < h->n; i++) scr_closure_release(h->v[i].cb);
|
||||
for (size_t i = 0; i < h->n; i++) {
|
||||
scr_closure_release(h->v[i].cb);
|
||||
scr_str_release(h->v[i].at);
|
||||
}
|
||||
free(h->v);
|
||||
h->v = NULL;
|
||||
h->n = h->cap = 0;
|
||||
@@ -277,7 +294,9 @@ static void scr_test_indent(int depth) {
|
||||
|
||||
/* "(1.234567ms)" — durations print microsecond-rounded (Node's hrtime
|
||||
* granularity), not with double noise. */
|
||||
/* NaN prints nothing: a test cancelled before it started has no duration. */
|
||||
static void scr_test_print_dur(double ms) {
|
||||
if (isnan(ms)) return;
|
||||
char num[32];
|
||||
fputs(" (", stdout);
|
||||
size_t n = scr_f64_to_str(round(ms * 1e6) / 1e6, num);
|
||||
@@ -409,10 +428,14 @@ static void scr_test_mocks_restore(ScrTestHooks *mocks, bool forget) {
|
||||
/* `ctx` is the context a hook that takes one receives: the suite or test
|
||||
* for its own before/after hooks, the test for the beforeEach/afterEach
|
||||
* chain. */
|
||||
static void scr_test_run_hooks_of(ScrTestCtx *owner, int which, ScrTestCtx *ctx) {
|
||||
for (size_t i = 0; i < owner->hooks[which].n && !scr_exc_pending(); i++) {
|
||||
scr_test_call(owner->hooks[which].v[i].cb, owner->hooks[which].v[i].flags, ctx, NULL);
|
||||
static ScrTestHook *scr_test_run_hooks_of(ScrTestCtx *owner, int which, ScrTestCtx *ctx) {
|
||||
if (scr_exc_pending()) return NULL; /* an earlier hook in the chain threw */
|
||||
for (size_t i = 0; i < owner->hooks[which].n; i++) {
|
||||
ScrTestHook *h = &owner->hooks[which].v[i];
|
||||
scr_test_call(h->cb, h->flags, ctx, NULL);
|
||||
if (scr_exc_pending()) return h; /* the hook that threw */
|
||||
}
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/* Returns whether the chain has any `which` hooks. */
|
||||
@@ -559,6 +582,36 @@ static void scr_test_finish_concurrent(ScrTestCtx *t) {
|
||||
for (size_t i = 0; i < t->nchildren; i++) scr_test_report_tree(t->children[i]);
|
||||
}
|
||||
|
||||
/* A failing root before hook (Node): each top-level test fails with its
|
||||
* error without running its body or beforeEach hooks, though its
|
||||
* afterEach hooks run; a top-level suite fails with it, and the tests and
|
||||
* suites in it are cancelled before they start. */
|
||||
static void scr_test_cancel_unrun(ScrTestCtx *c) {
|
||||
for (size_t i = 0; i < c->nchildren; i++) scr_test_cancel_unrun(c->children[i]);
|
||||
static const char msg[] = "test did not finish before its parent and was cancelled";
|
||||
scr_throw_str(scr_str_new(msg, sizeof msg - 1));
|
||||
c->err = scr_exc_take();
|
||||
c->listed = true;
|
||||
c->dur_ms = NAN;
|
||||
c->state = SCR_TS_CANCELLED;
|
||||
scr_test_report_result(c);
|
||||
}
|
||||
|
||||
static void scr_test_fail_unrun(ScrTestCtx *c, ScrCaught *err) {
|
||||
double start = scr_now_ms();
|
||||
c->err = scr_caught_retain(err);
|
||||
c->listed = true;
|
||||
if (c->is_suite) {
|
||||
for (size_t i = 0; i < c->nchildren; i++) scr_test_cancel_unrun(c->children[i]);
|
||||
} else {
|
||||
scr_test_run_each_hooks(c, SCR_HOOK_AFTER_EACH, false);
|
||||
if (scr_exc_pending()) scr_exc_clear();
|
||||
}
|
||||
c->dur_ms = scr_now_ms() - start;
|
||||
c->state = SCR_TS_FAIL;
|
||||
scr_test_report_result(c);
|
||||
}
|
||||
|
||||
static void scr_test_run_children(ScrTestCtx *t) {
|
||||
/* `only` at this level: when a child is marked, or contains a marked
|
||||
* test or suite, only those children run. The others are dropped entirely
|
||||
@@ -572,6 +625,10 @@ static void scr_test_run_children(ScrTestCtx *t) {
|
||||
/* Subtests t.test already ran INLINE at their call site, or started on
|
||||
* their own fiber. */
|
||||
if (c->state != SCR_TS_PENDING || c->done) continue;
|
||||
if (t == scr_test_root && t->before_err) {
|
||||
scr_test_fail_unrun(c, t->before_err);
|
||||
continue;
|
||||
}
|
||||
if (t->concurrency > 1) scr_test_start(c);
|
||||
else scr_test_run_node(c);
|
||||
}
|
||||
@@ -813,7 +870,7 @@ static void scr_test_print_failures(ScrTestCtx *t, bool *any) {
|
||||
for (size_t i = 0; i < t->nchildren; i++) {
|
||||
ScrTestCtx *c = t->children[i];
|
||||
scr_test_print_failures(c, any);
|
||||
if (!c->is_suite &&
|
||||
if ((!c->is_suite || c->listed) &&
|
||||
(c->state == SCR_TS_FAIL || c->state == SCR_TS_TODO_FAIL ||
|
||||
c->state == SCR_TS_CANCELLED) &&
|
||||
c->err) {
|
||||
@@ -1404,7 +1461,41 @@ static void scr_test_runner_entry(ScrFiber *self, void *argpack) {
|
||||
* the first test only after the file's synchronous body completes. */
|
||||
scr_await_hop();
|
||||
ScrTestCtx *root = scr_test_root;
|
||||
/* The root's before hooks ran at their before() calls; the first test
|
||||
* waits for the async ones. */
|
||||
for (size_t i = 0; i < scr_test_root_npending; i++) {
|
||||
scr_await_void(scr_test_root_pending[i]);
|
||||
scr_test_root_before_failed(root);
|
||||
scr_promise_release(scr_test_root_pending[i]);
|
||||
}
|
||||
free(scr_test_root_pending);
|
||||
scr_test_root_pending = NULL;
|
||||
scr_test_root_npending = scr_test_root_pending_cap = 0;
|
||||
scr_test_run_children(root);
|
||||
/* Its after hooks: a throw reports the root as "✖ <file>" with the hook's
|
||||
* location, after the top-level results, without counting it or changing
|
||||
* the exit code, as Node does. */
|
||||
double after_start = scr_now_ms();
|
||||
ScrTestHook *after_failed = scr_test_run_hooks_of(root, SCR_HOOK_AFTER, root);
|
||||
if (scr_exc_pending()) {
|
||||
root->err = scr_exc_take();
|
||||
root->dur_ms = scr_now_ms() - after_start;
|
||||
root->state = SCR_TS_FAIL;
|
||||
if (after_failed && after_failed->at) {
|
||||
root->at = scr_str_retain(after_failed->at);
|
||||
/* The name is the file: the location without its ":line:col". */
|
||||
size_t end = root->at->len;
|
||||
for (int colons = 0; end > 0 && colons < 2; end--) {
|
||||
if (root->at->data[end - 1] == ':') colons++;
|
||||
}
|
||||
scr_str_release(root->name);
|
||||
root->name = scr_str_new(root->at->data, end);
|
||||
}
|
||||
fputs("\xE2\x9C\x96 ", stdout);
|
||||
scr_test_print_str(root->name);
|
||||
scr_test_print_dur(root->dur_ms);
|
||||
fputc('\n', stdout);
|
||||
}
|
||||
scr_test_run_done = true;
|
||||
scr_snap_write_all();
|
||||
scr_test_count(root);
|
||||
@@ -1418,6 +1509,10 @@ static void scr_test_runner_entry(ScrFiber *self, void *argpack) {
|
||||
scr_test_info_line("duration_ms", round((scr_now_ms() - scr_test_t0) * 1e6) / 1e6);
|
||||
bool any = false;
|
||||
scr_test_print_failures(root, &any);
|
||||
if (root->err) {
|
||||
if (!any) fputs("\n\xE2\x9C\x96 failing tests:\n", stdout);
|
||||
scr_test_print_failure(root);
|
||||
}
|
||||
/* The tree is done reporting: release it (test bodies may still hold
|
||||
* their TestContext refs — the boxes stay alive through those). */
|
||||
scr_test_reg_parent = NULL;
|
||||
@@ -1534,7 +1629,41 @@ void scr_test_suite(ScrStr *name /*borrowed*/, double mode, ScrStr *msg /*borrow
|
||||
scr_closure_release(cb);
|
||||
}
|
||||
|
||||
void scr_test_hook(double which, ScrClosure *cb /*moves*/, double flags) {
|
||||
/* A module-level before hook runs at its before() call, as Node runs the
|
||||
* hooks of its already-started root at once: the synchronous part runs
|
||||
* now, on the caller's stack, and the runner waits for an async hook's
|
||||
* promise before the first top-level test. The first failure fails the
|
||||
* top-level tests and suites (scr_test_fail_unrun). */
|
||||
static void scr_test_root_before_now(ScrTestCtx *root, ScrClosure *cb, int flags) {
|
||||
bool takes_ctx = (flags & SCR_TEST_F_CTX) != 0;
|
||||
if (flags & SCR_TEST_F_ASYNC) {
|
||||
ScrPromise *p =
|
||||
takes_ctx
|
||||
? ((ScrPromise * (*)(ScrClosure *, ScrTestCtx *)) cb->fn)(cb, scr_testctx_retain(root))
|
||||
: ((ScrPromise * (*)(ScrClosure *)) cb->fn)(cb);
|
||||
if (p && !scr_exc_pending()) {
|
||||
if (scr_test_root_npending == scr_test_root_pending_cap) {
|
||||
scr_test_root_pending_cap = scr_test_root_pending_cap ? scr_test_root_pending_cap * 2 : 2;
|
||||
scr_test_root_pending = realloc(scr_test_root_pending,
|
||||
scr_test_root_pending_cap * sizeof *scr_test_root_pending);
|
||||
if (!scr_test_root_pending) scr_test_oom();
|
||||
}
|
||||
/* The runner observes it: a rejection is the hook's failure, not an
|
||||
* unhandled rejection. */
|
||||
scr_promise_mark_handled(p);
|
||||
scr_test_root_pending[scr_test_root_npending++] = p;
|
||||
return;
|
||||
}
|
||||
scr_promise_release(p);
|
||||
} else if (takes_ctx) {
|
||||
((void (*)(ScrClosure *, ScrTestCtx *))cb->fn)(cb, scr_testctx_retain(root));
|
||||
} else {
|
||||
((void (*)(ScrClosure *))cb->fn)(cb);
|
||||
}
|
||||
scr_test_root_before_failed(root);
|
||||
}
|
||||
|
||||
void scr_test_hook(double which, ScrClosure *cb /*moves*/, double flags, ScrStr *at) {
|
||||
scr_test_ensure_root();
|
||||
ScrTestCtx *owner = scr_test_parent_get();
|
||||
ScrTestHooks *h = &owner->hooks[(int)which];
|
||||
@@ -1545,7 +1674,10 @@ void scr_test_hook(double which, ScrClosure *cb /*moves*/, double flags) {
|
||||
}
|
||||
h->v[h->n].cb = cb;
|
||||
h->v[h->n].flags = (int)flags;
|
||||
h->v[h->n].at = at && at->len > 0 ? scr_str_retain(at) : NULL;
|
||||
h->n++;
|
||||
if ((int)which == SCR_HOOK_BEFORE && owner == scr_test_root)
|
||||
scr_test_root_before_now(owner, cb, (int)flags);
|
||||
}
|
||||
|
||||
/* t.test(...): registers under `t` and — because subtests run inline on
|
||||
@@ -1620,6 +1752,7 @@ void scr_test_ctx_hook(ScrTestCtx *t, double which, ScrClosure *cb /*moves*/, do
|
||||
}
|
||||
h->v[h->n].cb = cb;
|
||||
h->v[h->n].flags = (int)flags;
|
||||
h->v[h->n].at = NULL;
|
||||
h->n++;
|
||||
}
|
||||
|
||||
@@ -1631,6 +1764,7 @@ static void scr_test_mocks_add(ScrTestHooks *h, ScrClosure *restore /*moves*/) {
|
||||
}
|
||||
h->v[h->n].cb = restore;
|
||||
h->v[h->n].flags = 0;
|
||||
h->v[h->n].at = NULL;
|
||||
h->n++;
|
||||
}
|
||||
|
||||
@@ -2172,6 +2306,7 @@ void scr_test_ctx_diagnostic(ScrTestCtx *t, ScrStr *msg /*borrowed*/) {
|
||||
}
|
||||
|
||||
ScrStr *scr_test_ctx_name(ScrTestCtx *t) {
|
||||
if (!t->name && !t->parent) return scr_str_new("<root>", 6); /* Node's root test */
|
||||
return t->name ? scr_str_retain(t->name) : scr_str_new("", 0);
|
||||
}
|
||||
|
||||
|
||||
@@ -33,8 +33,10 @@ void scr_test_register(ScrStr *name, double mode, ScrStr *msg, ScrClosure *cb, d
|
||||
void scr_test_suite(ScrStr *name, double mode, ScrStr *msg, ScrClosure *cb, ScrStr *at,
|
||||
double flags, double concurrency);
|
||||
/* before/after/beforeEach/afterEach on the CURRENT registration parent
|
||||
* (`which`: 0 before, 1 after, 2 beforeEach, 3 afterEach). */
|
||||
void scr_test_hook(double which, ScrClosure *cb, double flags);
|
||||
* (`which`: 0 before, 1 after, 2 beforeEach, 3 afterEach); `at` is the
|
||||
* hook call's "file:line:col". The root's before and after hooks run
|
||||
* around the top-level tests. */
|
||||
void scr_test_hook(double which, ScrClosure *cb, double flags, ScrStr *at);
|
||||
/* t.test(...): registers under `t`, runs the subtest INLINE on the
|
||||
* runner fiber, and returns the settled promise the `await` consumes (+1). */
|
||||
ScrPromise *scr_test_sub(ScrTestCtx *t, ScrStr *name, double mode, ScrStr *msg, ScrClosure *cb, double flags, ScrStr *at,
|
||||
|
||||
@@ -0,0 +1,490 @@
|
||||
/* Comparison logic for scripts/zero-cost-gate.mjs, kept free of process
|
||||
* orchestration so tests/harness/zero-cost-gate.test.ts can pin it.
|
||||
*
|
||||
* Artifacts are compared as bytes first. Only when they differ are they split
|
||||
* into named units (IR functions, LLVM functions, executable symbols) to name
|
||||
* the first unit that differs. The self-hosted compiler's IR and LLVM exceed
|
||||
* V8's maximum string length, so splitting works on Buffers and hashes each
|
||||
* unit instead of decoding the artifact. */
|
||||
import { createHash } from "node:crypto";
|
||||
|
||||
const hash = (bytes) => createHash("sha1").update(bytes).digest("hex");
|
||||
|
||||
/** Named units of an artifact: `order` lists unit keys in artifact order,
|
||||
* `hashes` maps each key to a content hash, and `module` holds the
|
||||
* module-level remainder (IR header, LLVM globals and declarations) as a list
|
||||
* of comparable pieces with their labels. Repeated names get `#n` suffixes. */
|
||||
function unitsBuilder() {
|
||||
const order = [];
|
||||
const hashes = new Map();
|
||||
const seen = new Map();
|
||||
return {
|
||||
order,
|
||||
hashes,
|
||||
module: [],
|
||||
add(name, bytes) {
|
||||
const count = seen.get(name) ?? 0;
|
||||
seen.set(name, count + 1);
|
||||
const key = count === 0 ? name : `${name}#${count + 1}`;
|
||||
order.push(key);
|
||||
hashes.set(key, hash(bytes));
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/* ── IR (compact or readable JSON) ─────────────────────────────────────── */
|
||||
|
||||
function skipString(bytes, index) {
|
||||
for (index++; index < bytes.length; index++) {
|
||||
if (bytes[index] === 0x5c) index++;
|
||||
else if (bytes[index] === 0x22) return index + 1;
|
||||
}
|
||||
return bytes.length;
|
||||
}
|
||||
|
||||
/** Byte ranges of the root object's `"functions"` array elements, or null. */
|
||||
export function irFunctionSpans(bytes) {
|
||||
const key = Buffer.from('"functions"');
|
||||
let depth = 0;
|
||||
for (let index = 0; index < bytes.length;) {
|
||||
const byte = bytes[index];
|
||||
if (byte === 0x22) {
|
||||
const end = skipString(bytes, index);
|
||||
if (depth === 1 && end - index === key.length && bytes.subarray(index, end).equals(key)) {
|
||||
let next = end;
|
||||
while (next < bytes.length && bytes[next] !== 0x3a) next++;
|
||||
next++;
|
||||
while (next < bytes.length && bytes[next] !== 0x5b && bytes[next] <= 0x20) next++;
|
||||
if (bytes[next] === 0x5b) return elementSpans(bytes, next);
|
||||
}
|
||||
index = end;
|
||||
continue;
|
||||
}
|
||||
if (byte === 0x7b || byte === 0x5b) depth++;
|
||||
else if (byte === 0x7d || byte === 0x5d) depth--;
|
||||
index++;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function elementSpans(bytes, open) {
|
||||
const spans = [];
|
||||
let depth = 0;
|
||||
let start = -1;
|
||||
for (let index = open + 1; index < bytes.length;) {
|
||||
const byte = bytes[index];
|
||||
if (byte === 0x22) {
|
||||
if (start < 0) start = index;
|
||||
index = skipString(bytes, index);
|
||||
continue;
|
||||
}
|
||||
if (byte === 0x7b || byte === 0x5b) {
|
||||
if (start < 0) start = index;
|
||||
depth++;
|
||||
} else if (byte === 0x7d || byte === 0x5d) {
|
||||
if (depth === 0) {
|
||||
if (start >= 0) spans.push([start, trimEnd(bytes, index)]);
|
||||
return { open, close: index, spans };
|
||||
}
|
||||
depth--;
|
||||
} else if (byte === 0x2c && depth === 0) {
|
||||
spans.push([start, trimEnd(bytes, index)]);
|
||||
start = -1;
|
||||
} else if (start < 0 && byte > 0x20) start = index;
|
||||
index++;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function trimEnd(bytes, end) {
|
||||
while (end > 0 && bytes[end - 1] <= 0x20) end--;
|
||||
return end;
|
||||
}
|
||||
|
||||
/** The `name` of one serialized IR function. Functions serialize `name`
|
||||
* first; anything else is parsed (a single function is small). */
|
||||
function irFunctionName(bytes, [start, end]) {
|
||||
const head = bytes.toString("utf8", start, Math.min(end, start + 4096));
|
||||
const match = /^\{\s*"name"\s*:\s*("(?:[^"\\]|\\.)*")/.exec(head);
|
||||
if (match) return JSON.parse(match[1]);
|
||||
const fn = JSON.parse(bytes.toString("utf8", start, end));
|
||||
return typeof fn?.name === "string" ? fn.name : "<unnamed>";
|
||||
}
|
||||
|
||||
/** IR units: one per function, plus the module header split by top-level
|
||||
* key (and, for arrays, by element) so header differences are named too. */
|
||||
export function irUnits(bytes) {
|
||||
const units = unitsBuilder();
|
||||
const located = irFunctionSpans(bytes);
|
||||
let header;
|
||||
if (located === null) header = JSON.parse(bytes.toString("utf8"));
|
||||
else {
|
||||
for (const span of located.spans)
|
||||
units.add(irFunctionName(bytes, span), bytes.subarray(span[0], span[1]));
|
||||
header = JSON.parse(
|
||||
bytes.toString("utf8", 0, located.open + 1) + bytes.toString("utf8", located.close),
|
||||
);
|
||||
}
|
||||
for (const [key, value] of Object.entries(header)) {
|
||||
if (key === "functions" && located !== null) continue;
|
||||
if (Array.isArray(value))
|
||||
value.forEach((item, index) => {
|
||||
const name = item && typeof item === "object" && "name" in item ? `:${item.name}` : "";
|
||||
units.module.push({ label: `${key}[${index}${name}]`, text: JSON.stringify(item) });
|
||||
});
|
||||
else units.module.push({ label: key, text: JSON.stringify(value) });
|
||||
}
|
||||
return units;
|
||||
}
|
||||
|
||||
/* ── LLVM text ─────────────────────────────────────────────────────────── */
|
||||
|
||||
const DEFINE_NAME = /@("(?:[^"\\]|\\.)*"|[-\w$.]+)\s*\(/;
|
||||
|
||||
/** LLVM units: one per `define … { … }` body; every other line is
|
||||
* module-level text compared line by line. */
|
||||
export function llvmUnits(bytes) {
|
||||
const units = unitsBuilder();
|
||||
let fnStart = -1;
|
||||
let fnName = "";
|
||||
let lineNumber = 0;
|
||||
for (let start = 0; start < bytes.length;) {
|
||||
let end = bytes.indexOf(10, start);
|
||||
if (end < 0) end = bytes.length;
|
||||
lineNumber++;
|
||||
if (fnStart >= 0) {
|
||||
if (end - start === 1 && bytes[start] === 0x7d) {
|
||||
units.add(fnName, bytes.subarray(fnStart, end));
|
||||
fnStart = -1;
|
||||
}
|
||||
} else if (bytes[start] === 0x64 && bytes.toString("latin1", start, start + 7) === "define ") {
|
||||
const line = bytes.toString("utf8", start, end);
|
||||
fnName = DEFINE_NAME.exec(line)?.[1] ?? `<line ${lineNumber}>`;
|
||||
fnStart = start;
|
||||
if (line.endsWith("}")) {
|
||||
units.add(fnName, bytes.subarray(fnStart, end));
|
||||
fnStart = -1;
|
||||
}
|
||||
} else if (end > start) {
|
||||
units.module.push({ label: `line ${lineNumber}`, text: bytes.toString("utf8", start, end) });
|
||||
}
|
||||
start = end + 1;
|
||||
}
|
||||
return units;
|
||||
}
|
||||
|
||||
/* ── unit comparison ───────────────────────────────────────────────────── */
|
||||
|
||||
const clip = (text, width = 160) => (text.length > width ? `${text.slice(0, width)}…` : text);
|
||||
|
||||
/** Compare two unit sets. `first` names the first difference in baseline
|
||||
* order: a changed or removed function, then an added one, then the first
|
||||
* differing module-level piece. */
|
||||
export function compareUnits(baseline, candidate) {
|
||||
const changed = [];
|
||||
const removed = [];
|
||||
for (const key of baseline.order) {
|
||||
const other = candidate.hashes.get(key);
|
||||
if (other === undefined) removed.push(key);
|
||||
else if (other !== baseline.hashes.get(key)) changed.push(key);
|
||||
}
|
||||
const added = candidate.order.filter((key) => !baseline.hashes.has(key));
|
||||
let module = null;
|
||||
const count = Math.max(baseline.module.length, candidate.module.length);
|
||||
for (let index = 0; index < count && module === null; index++) {
|
||||
const a = baseline.module[index];
|
||||
const b = candidate.module[index];
|
||||
if (a?.text !== b?.text)
|
||||
module = {
|
||||
label: (a ?? b).label,
|
||||
baseline: a === undefined ? null : clip(a.text),
|
||||
candidate: b === undefined ? null : clip(b.text),
|
||||
};
|
||||
}
|
||||
const firstFunction = baseline.order.find(
|
||||
(key) => !candidate.hashes.has(key) || candidate.hashes.get(key) !== baseline.hashes.get(key),
|
||||
);
|
||||
let first = null;
|
||||
if (firstFunction !== undefined)
|
||||
first = `${changed.includes(firstFunction) ? "changed" : "removed"} ${firstFunction}`;
|
||||
else if (added.length > 0) first = `added ${added[0]}`;
|
||||
else if (module !== null) first = `module ${module.label}`;
|
||||
else if (baseline.order.join("\n") !== candidate.order.join("\n")) first = "function order";
|
||||
return {
|
||||
identical: first === null,
|
||||
first,
|
||||
changed,
|
||||
added,
|
||||
removed,
|
||||
module,
|
||||
functions: { baseline: baseline.order.length, candidate: candidate.order.length },
|
||||
};
|
||||
}
|
||||
|
||||
/** Byte-compare an artifact and, when it differs, name what changed. */
|
||||
export function compareArtifact(kind, baseline, candidate) {
|
||||
if (baseline.equals(candidate)) return { identical: true };
|
||||
const split = kind === "ir" ? irUnits : llvmUnits;
|
||||
const result = compareUnits(split(baseline), split(candidate));
|
||||
// Equal units with unequal bytes means formatting outside the units.
|
||||
return { ...result, identical: false, first: result.first ?? "bytes outside named units" };
|
||||
}
|
||||
|
||||
/* ── executables ───────────────────────────────────────────────────────── */
|
||||
|
||||
/** The output of `nm` for `binary`, from its spawnSync result. Throws when
|
||||
* nm could not start, was killed by a signal, or exited with a non-zero
|
||||
* status, because the symbol checks would otherwise read an empty list. */
|
||||
export function nmOutput(binary, result) {
|
||||
if (result.error) throw new Error(`nm ${binary}: ${result.error.message}`);
|
||||
if (result.signal) throw new Error(`nm ${binary}: killed by ${result.signal}`);
|
||||
if (result.status !== 0) {
|
||||
const detail = typeof result.stderr === "string" ? result.stderr.trim() : "";
|
||||
throw new Error(
|
||||
`nm ${binary}: exited with status ${result.status}${detail ? `: ${detail}` : ""}`,
|
||||
);
|
||||
}
|
||||
return result.stdout;
|
||||
}
|
||||
|
||||
/** Defined symbols from `nm -S` output (GNU/LLVM nm, address size type
|
||||
* name) or `nm` output without sizes (Apple, address type name). */
|
||||
export function parseNm(text) {
|
||||
const symbols = [];
|
||||
for (const line of text.split("\n")) {
|
||||
const fields = line.trim().split(/\s+/);
|
||||
if (fields.length === 4 && /^[0-9a-f]+$/i.test(fields[1]))
|
||||
symbols.push({
|
||||
address: BigInt(`0x${fields[0]}`),
|
||||
size: BigInt(`0x${fields[1]}`),
|
||||
type: fields[2],
|
||||
name: fields[3],
|
||||
});
|
||||
else if (fields.length === 3 && /^[0-9a-f]+$/i.test(fields[0]))
|
||||
symbols.push({
|
||||
address: BigInt(`0x${fields[0]}`),
|
||||
size: null,
|
||||
type: fields[1],
|
||||
name: fields[2],
|
||||
});
|
||||
}
|
||||
return symbols;
|
||||
}
|
||||
|
||||
/** The virtual address of file offset `offset` in an ELF64 little-endian
|
||||
* executable, from its PT_LOAD program headers, or null. */
|
||||
export function elfOffsetToAddress(bytes, offset) {
|
||||
if (bytes.length < 64 || bytes.readUInt32BE(0) !== 0x7f454c46 || bytes[4] !== 2) return null;
|
||||
const phoff = Number(bytes.readBigUInt64LE(0x20));
|
||||
const phentsize = bytes.readUInt16LE(0x36);
|
||||
const phnum = bytes.readUInt16LE(0x38);
|
||||
for (let index = 0; index < phnum; index++) {
|
||||
const at = phoff + index * phentsize;
|
||||
if (bytes.readUInt32LE(at) !== 1) continue;
|
||||
const fileOffset = bytes.readBigUInt64LE(at + 8);
|
||||
const address = bytes.readBigUInt64LE(at + 16);
|
||||
const fileSize = bytes.readBigUInt64LE(at + 32);
|
||||
const target = BigInt(offset);
|
||||
if (target >= fileOffset && target < fileOffset + fileSize)
|
||||
return address + (target - fileOffset);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/** Name what differs between two executables: symbols added, removed or
|
||||
* resized, then the symbol holding the first differing byte. */
|
||||
export function compareExecutables(baseline, candidate, baselineNm, candidateNm) {
|
||||
if (baseline.equals(candidate)) return { identical: true };
|
||||
const before = new Map(parseNm(baselineNm).map((s) => [s.name, s]));
|
||||
const after = new Map(parseNm(candidateNm).map((s) => [s.name, s]));
|
||||
const removed = [...before.keys()].filter((name) => !after.has(name));
|
||||
const added = [...after.keys()].filter((name) => !before.has(name));
|
||||
const resized = [...before.values()]
|
||||
.filter((s) => after.has(s.name) && s.size !== null && after.get(s.name).size !== s.size)
|
||||
.map((s) => ({
|
||||
name: s.name,
|
||||
baseline: Number(s.size),
|
||||
candidate: Number(after.get(s.name).size),
|
||||
}));
|
||||
let offset = 0;
|
||||
const limit = Math.min(baseline.length, candidate.length);
|
||||
while (offset < limit && baseline[offset] === candidate[offset]) offset++;
|
||||
const address = elfOffsetToAddress(baseline, offset);
|
||||
let holder = null;
|
||||
if (address !== null)
|
||||
for (const s of before.values())
|
||||
if (s.size !== null && address >= s.address && address < s.address + s.size) holder = s.name;
|
||||
const first =
|
||||
resized[0] !== undefined
|
||||
? `resized ${resized[0].name}`
|
||||
: removed[0] !== undefined
|
||||
? `removed ${removed[0]}`
|
||||
: added[0] !== undefined
|
||||
? `added ${added[0]}`
|
||||
: holder !== null
|
||||
? `bytes in ${holder}`
|
||||
: `byte offset ${offset}`;
|
||||
return { identical: false, first, offset, holder, added, removed, resized };
|
||||
}
|
||||
|
||||
/* ── dynamic-tier symbols (two-tier design, invariant I3) ────────────────── */
|
||||
|
||||
/** Runtime symbol prefixes that belong to the dynamic tier: code a fully
|
||||
* typed program has no reason to reach. A typed program's release binary
|
||||
* must define none of them. Extend this list as dynamic-tier units land
|
||||
* (S3/S4 add their scr_js_* families) and as legacy ScrDyn families move.
|
||||
* The ScrDyn prefixes are the object-model entry points the two-tier audit
|
||||
* (results/design-two-tier/DESIGN.md §3.2-3.3) lists as dynamic-only. */
|
||||
export const DYNAMIC_TIER_PREFIXES = [
|
||||
// Dynamic-tier runtime units (scr_js_value, scr_js_object, ...), from S3.
|
||||
"scr_js_",
|
||||
// ScrDyn property bags: reads, writes, lookup and key indexing.
|
||||
"scr_dyn_obj_",
|
||||
"scr_dyn_find_",
|
||||
"scr_dyn_index_",
|
||||
"scr_dyn_key_",
|
||||
"sc_dyn_key_",
|
||||
// Descriptors, own-property reflection, prototypes and proxies.
|
||||
"scr_dyn_property_",
|
||||
"scr_dyn_define_",
|
||||
"scr_dyn_descriptor_",
|
||||
"scr_dyn_own_",
|
||||
"scr_dyn_get_own_",
|
||||
"scr_dyn_prototype",
|
||||
"scr_dyn_proxy_",
|
||||
// Dynamic calls and the dynamic receiver read.
|
||||
"scr_dyn_call",
|
||||
"scr_dyn_apply",
|
||||
"scr_dyn_invoke",
|
||||
"scr_dyn_construct",
|
||||
"scr_dyn_function_apply",
|
||||
"scr_dyn_fn_get",
|
||||
"scr_dyn_this_get",
|
||||
// Dynamic operators and coercions.
|
||||
"scr_dyn_add_",
|
||||
"scr_dyn_arithmetic",
|
||||
"scr_dyn_to_numeric",
|
||||
"scr_dyn_number_coerce",
|
||||
"scr_dyn_string_coerce",
|
||||
];
|
||||
|
||||
/** Symbols under those prefixes that typed binaries reach today through a
|
||||
* leak the two-tier design schedules for removal (S1). Remove an entry when
|
||||
* its leak closes. */
|
||||
export const DYNAMIC_TIER_EXEMPT = new Map([
|
||||
// §3.3 L10: scr_dyn_release's dispose switch, linked whenever any dyn
|
||||
// value can be released.
|
||||
["scr_dyn_obj_drop_keys", "L10"],
|
||||
]);
|
||||
|
||||
/** The dynamic-tier symbols among `names`, sorted. */
|
||||
export function dynamicTierSymbols(names) {
|
||||
return [...new Set(names)]
|
||||
.filter(
|
||||
(name) =>
|
||||
DYNAMIC_TIER_PREFIXES.some((prefix) => name.startsWith(prefix)) &&
|
||||
!DYNAMIC_TIER_EXEMPT.has(name),
|
||||
)
|
||||
.sort();
|
||||
}
|
||||
|
||||
/* ── cachegrind ────────────────────────────────────────────────────────── */
|
||||
|
||||
/** Background threads whose instruction counts depend on scheduling: the
|
||||
* allocator's prefault thread. */
|
||||
export const CACHEGRIND_EXCLUDED = new Set(["scr_sa_pf_main", "scr_sa_prefault"]);
|
||||
|
||||
/** Per-function instruction counts (Ir) from a cachegrind output file,
|
||||
* restricted to `keep` (the executable's own defined symbols). Dynamic
|
||||
* linker and libc/pthread functions are outside `keep`; their counts vary
|
||||
* run to run with lazy binding and thread wakeups. */
|
||||
export function cachegrindCounts(text, keep) {
|
||||
const counts = new Map();
|
||||
let events = null;
|
||||
let current = null;
|
||||
for (const line of text.split("\n")) {
|
||||
if (line.startsWith("events:")) events = line.slice(7).trim().split(/\s+/);
|
||||
else if (line.startsWith("fn=")) {
|
||||
const name = line.slice(3).trim();
|
||||
current = keep.has(name) && !CACHEGRIND_EXCLUDED.has(name) ? name : null;
|
||||
} else if (line.startsWith("fl=") || line.startsWith("fi=") || line.startsWith("fe=")) continue;
|
||||
else if (current !== null && /^\d/.test(line)) {
|
||||
const fields = line.trim().split(/\s+/);
|
||||
const column = events === null ? 0 : Math.max(0, events.indexOf("Ir"));
|
||||
const ir = Number(fields[column + 1] ?? 0);
|
||||
counts.set(current, (counts.get(current) ?? 0) + ir);
|
||||
}
|
||||
}
|
||||
return counts;
|
||||
}
|
||||
|
||||
/** Per-function instruction-count deltas, largest absolute first. */
|
||||
export function compareCounts(baseline, candidate) {
|
||||
const names = new Set([...baseline.keys(), ...candidate.keys()]);
|
||||
const deltas = [];
|
||||
let totalBaseline = 0;
|
||||
let totalCandidate = 0;
|
||||
for (const name of names) {
|
||||
const a = baseline.get(name) ?? 0;
|
||||
const b = candidate.get(name) ?? 0;
|
||||
totalBaseline += a;
|
||||
totalCandidate += b;
|
||||
if (a !== b) deltas.push({ name, baseline: a, candidate: b, delta: b - a });
|
||||
}
|
||||
deltas.sort((x, y) => Math.abs(y.delta) - Math.abs(x.delta) || (x.name < y.name ? -1 : 1));
|
||||
return {
|
||||
identical: deltas.length === 0,
|
||||
total: { baseline: totalBaseline, candidate: totalCandidate },
|
||||
deltas,
|
||||
};
|
||||
}
|
||||
|
||||
/* ── verdicts ──────────────────────────────────────────────────────────── */
|
||||
|
||||
/** One program's verdict from its build results and artifact comparisons.
|
||||
* `allowed` marks programs that match the change's pattern signature. */
|
||||
export function programVerdict({ baseline, candidate, artifacts, allowed }) {
|
||||
if (!baseline.ok || !candidate.ok) {
|
||||
if (!baseline.ok && !candidate.ok && baseline.error === candidate.error)
|
||||
return { verdict: "both-fail", fails: false };
|
||||
return { verdict: "build-error", fails: true };
|
||||
}
|
||||
const differing = Object.entries(artifacts).filter(([, r]) => !r.identical);
|
||||
if (differing.length === 0) return { verdict: "identical", fails: false };
|
||||
return allowed ? { verdict: "allowed", fails: false } : { verdict: "differs", fails: true };
|
||||
}
|
||||
|
||||
export function median(values) {
|
||||
const sorted = [...values].sort((a, b) => a - b);
|
||||
const middle = sorted.length >> 1;
|
||||
return sorted.length % 2 ? sorted[middle] : (sorted[middle - 1] + sorted[middle]) / 2;
|
||||
}
|
||||
|
||||
export function geomean(values) {
|
||||
return values.length === 0
|
||||
? 1
|
||||
: Math.exp(values.reduce((sum, value) => sum + Math.log(value), 0) / values.length);
|
||||
}
|
||||
|
||||
/** Compile-time ratio (candidate / baseline) for a set of programs. Each
|
||||
* row holds paired samples: one baseline and one candidate build that ran
|
||||
* side by side, so both saw the same machine load. The estimate is the
|
||||
* geometric mean of each program's median pair ratio, with a 95% bootstrap
|
||||
* interval over the pairs. Builds on a shared VM vary by 10-20% run to run,
|
||||
* so the gate judges the interval, not the point estimate. */
|
||||
export function timeRatio(rows, resamples = 2000) {
|
||||
let state = 0x5eed;
|
||||
const random = () => {
|
||||
state = (Math.imul(state, 1664525) + 1013904223) >>> 0;
|
||||
return state / 4294967296;
|
||||
};
|
||||
const pick = (xs) =>
|
||||
Array.from({ length: xs.length }, () => xs[Math.floor(random() * xs.length)]);
|
||||
const estimate = (draw) => geomean(rows.map((r) => median(draw(r.ratios))));
|
||||
const ratios = Array.from({ length: resamples }, () => estimate(pick)).sort((a, b) => a - b);
|
||||
return {
|
||||
ratio: estimate((xs) => xs),
|
||||
low: ratios[Math.floor(resamples * 0.025)],
|
||||
high: ratios[Math.floor(resamples * 0.975)],
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,652 @@
|
||||
/* Zero-cost gate: prove that a compiler change leaves good code untouched.
|
||||
*
|
||||
* Compiles a fixed good-code set with a baseline and a candidate checkout and
|
||||
* compares IR, LLVM and release executables byte for byte, plus compile time
|
||||
* and binary size, and optionally per-function cachegrind instruction counts.
|
||||
* Both checkouts compile the same program files (from --programs, default the
|
||||
* baseline), so program paths embedded in artifacts match; checkout paths and
|
||||
* timestamps do not reach the artifacts. Builds bypass the native cache.
|
||||
*
|
||||
* The good-code set: the runtime benchmark workloads (and benchmarks/builds
|
||||
* applications), the self-hosted compiler (IR and LLVM), and the TypeScript
|
||||
* corpus programs. A program whose sources match --allow (the change's
|
||||
* pattern signature) may differ; any other difference fails the gate. */
|
||||
import assert from "node:assert/strict";
|
||||
import { spawn, spawnSync } from "node:child_process";
|
||||
import {
|
||||
existsSync,
|
||||
mkdirSync,
|
||||
mkdtempSync,
|
||||
readFileSync,
|
||||
readdirSync,
|
||||
rmSync,
|
||||
statSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { availableParallelism } from "node:os";
|
||||
import { dirname, join, relative, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { parseArgs } from "node:util";
|
||||
import {
|
||||
cachegrindCounts,
|
||||
compareArtifact,
|
||||
compareCounts,
|
||||
compareExecutables,
|
||||
dynamicTierSymbols,
|
||||
median,
|
||||
nmOutput,
|
||||
parseNm,
|
||||
programVerdict,
|
||||
timeRatio,
|
||||
} from "./zero-cost-compare.mjs";
|
||||
|
||||
const repoRoot = fileURLToPath(new URL("..", import.meta.url));
|
||||
const { values } = parseArgs({
|
||||
options: {
|
||||
baseline: { type: "string" },
|
||||
candidate: { type: "string", default: repoRoot },
|
||||
programs: { type: "string", default: "baseline" },
|
||||
set: { type: "string", default: "bench,self,corpus" },
|
||||
quick: { type: "boolean", default: false },
|
||||
corpus: { type: "string", default: "ts" },
|
||||
shard: { type: "string" },
|
||||
only: { type: "string" },
|
||||
allow: { type: "string" },
|
||||
"timing-runs": { type: "string" },
|
||||
"self-timing-runs": { type: "string", default: "0" },
|
||||
"max-time-growth": { type: "string", default: "2" },
|
||||
"max-size-growth": { type: "string", default: "0" },
|
||||
cachegrind: { type: "boolean", default: false },
|
||||
jobs: { type: "string", default: String(Math.max(1, availableParallelism())) },
|
||||
json: { type: "string" },
|
||||
keep: { type: "boolean", default: false },
|
||||
help: { type: "boolean", default: false },
|
||||
},
|
||||
});
|
||||
if (values.help || values.baseline === undefined) {
|
||||
console.log(`Usage: node scripts/zero-cost-gate.mjs --baseline=<checkout> [--candidate=<checkout>]
|
||||
[--programs=baseline|candidate|<dir>] [--set=bench,self,corpus] [--quick]
|
||||
[--corpus=ts|all] [--shard=i/n] [--only=<regex>] [--allow=<regex>]
|
||||
[--timing-runs=N] [--self-timing-runs=N] [--max-time-growth=2]
|
||||
[--max-size-growth=0]
|
||||
[--cachegrind] [--jobs=N] [--json=<file>] [--keep]
|
||||
|
||||
Each checkout must be built (packages/cli/dist/bootstrap.js and this host's
|
||||
runtime pack). Both compile the same program files, taken from --programs
|
||||
(default: the baseline checkout).
|
||||
|
||||
--set picks the good-code set: bench (benchmarks/runtime workloads and the
|
||||
benchmarks/builds applications), self (packages/compiler/src/native/cli.ts,
|
||||
LLVM only), corpus (tests/corpus TypeScript programs; --corpus=all adds
|
||||
JavaScript entries). --quick is --set=bench --timing-runs=0, for CI.
|
||||
--shard=i/n keeps every n-th corpus program; --only filters program ids.
|
||||
|
||||
--allow=<regex> is the change's pattern signature: programs whose source
|
||||
files match it may differ (reported as "allowed"). Any other difference in
|
||||
IR, LLVM or the release executable fails the gate.
|
||||
|
||||
--timing-runs=N (default 5) times N paired --emit=llvm builds per bench
|
||||
program, baseline and candidate side by side. The self-hosted compiler's
|
||||
comparison build is its first pair; --self-timing-runs adds more. The gate
|
||||
fails when the 95% bootstrap interval of the bench geomean ratio, or of the
|
||||
self-hosted ratio, lies entirely above --max-time-growth percent; an interval
|
||||
that straddles it is reported as inconclusive.
|
||||
--max-size-growth bounds executable growth (percent) of allowed programs.
|
||||
|
||||
Dynamic-tier symbols (two-tier invariant I3): no executable may gain
|
||||
symbols under DYNAMIC_TIER_PREFIXES (scripts/zero-cost-compare.mjs) unless
|
||||
it matches --allow, and fully typed bench workloads (all but
|
||||
DYNAMIC_WORKLOADS in this script) may define none.
|
||||
|
||||
--cachegrind (Linux, valgrind on PATH) runs differing executables under
|
||||
cachegrind (bench workloads with their arguments, corpus programs without)
|
||||
and compares instruction counts per function of the executable's own
|
||||
symbols, excluding the allocator's prefault thread.
|
||||
|
||||
Exit status: 0 when the gate passes, 1 when it fails.`);
|
||||
process.exit(values.help ? 0 : 2);
|
||||
}
|
||||
|
||||
const trees = { baseline: resolve(values.baseline), candidate: resolve(values.candidate) };
|
||||
for (const [label, root] of Object.entries(trees))
|
||||
assert.ok(
|
||||
existsSync(join(root, "packages/cli/dist/bootstrap.js")),
|
||||
`${label} checkout ${root} has no built CLI (packages/cli/dist/bootstrap.js)`,
|
||||
);
|
||||
const programsRoot =
|
||||
values.programs === "baseline" || values.programs === "candidate"
|
||||
? trees[values.programs]
|
||||
: resolve(values.programs);
|
||||
const sets = new Set((values.quick ? "bench" : values.set).split(","));
|
||||
for (const set of sets) assert.ok(["bench", "self", "corpus"].includes(set), `unknown set ${set}`);
|
||||
const timingRuns = Number(values["timing-runs"] ?? (values.quick ? 0 : 5));
|
||||
const selfTimingRuns = Number(values["self-timing-runs"]);
|
||||
const maxTimeGrowth = Number(values["max-time-growth"]) / 100;
|
||||
const maxSizeGrowth = Number(values["max-size-growth"]) / 100;
|
||||
const jobs = Number(values.jobs);
|
||||
assert.ok(Number.isInteger(timingRuns) && timingRuns >= 0, "--timing-runs must be >= 0");
|
||||
assert.ok(Number.isInteger(jobs) && jobs >= 1, "--jobs must be >= 1");
|
||||
assert.ok(["ts", "all"].includes(values.corpus), "--corpus must be ts or all");
|
||||
const only = values.only ? new RegExp(values.only) : null;
|
||||
const allow = values.allow ? new RegExp(values.allow, "m") : null;
|
||||
if (values.cachegrind)
|
||||
assert.equal(
|
||||
spawnSync("valgrind", ["--version"]).status,
|
||||
0,
|
||||
"--cachegrind needs valgrind on PATH",
|
||||
);
|
||||
|
||||
const work = mkdtempSync("/tmp/scriptc-zero-cost-");
|
||||
|
||||
/* ── the good-code set ─────────────────────────────────────────────────── */
|
||||
|
||||
/** Every file a program reads: the entry and its relative imports. */
|
||||
function programInputs(entry) {
|
||||
const files = new Set();
|
||||
const pending = [entry];
|
||||
while (pending.length > 0) {
|
||||
const file = pending.pop();
|
||||
if (files.has(file)) continue;
|
||||
files.add(file);
|
||||
const text = readFileSync(file, "utf8");
|
||||
for (const match of text.matchAll(
|
||||
/(?:from|import|require)\s*\(?\s*["'](\.{1,2}\/[^"']+)["']/g,
|
||||
)) {
|
||||
const base = resolve(dirname(file), match[1]);
|
||||
const stem = base.replace(/\.(m|c)?js$/, "");
|
||||
const found = [
|
||||
base,
|
||||
`${stem}.ts`,
|
||||
`${stem}.mts`,
|
||||
`${stem}.cts`,
|
||||
`${base}.ts`,
|
||||
join(base, "index.ts"),
|
||||
].find((candidate) => existsSync(candidate) && statSync(candidate).isFile());
|
||||
if (found !== undefined) pending.push(found);
|
||||
}
|
||||
}
|
||||
return [...files];
|
||||
}
|
||||
|
||||
function goodCodeSet() {
|
||||
const programs = [];
|
||||
if (sets.has("bench")) {
|
||||
const suite = join(programsRoot, "benchmarks/runtime");
|
||||
const manifest = JSON.parse(readFileSync(join(suite, "workloads.json"), "utf8"));
|
||||
const seen = new Set();
|
||||
for (const w of manifest.workloads) {
|
||||
const entry = join(suite, w.entry);
|
||||
if (seen.has(entry)) continue;
|
||||
seen.add(entry);
|
||||
programs.push({ id: `bench/${w.name}`, entry, kind: "exe", workload: w });
|
||||
}
|
||||
}
|
||||
if (sets.has("self"))
|
||||
programs.push({
|
||||
id: "self/cli",
|
||||
entry: join(programsRoot, "packages/compiler/src/native/cli.ts"),
|
||||
kind: "source",
|
||||
});
|
||||
if (sets.has("corpus")) {
|
||||
const corpus = join(programsRoot, "tests/corpus");
|
||||
const exts = values.corpus === "all" ? ["ts", "js", "mjs", "cjs"] : ["ts"];
|
||||
const entries = [];
|
||||
for (const name of readdirSync(corpus).sort()) {
|
||||
const path = join(corpus, name);
|
||||
if (statSync(path).isDirectory()) {
|
||||
const main = exts.map((ext) => join(path, `main.${ext}`)).find((p) => existsSync(p));
|
||||
if (main !== undefined) entries.push(main);
|
||||
} else if (exts.some((ext) => name.endsWith(`.${ext}`))) entries.push(path);
|
||||
}
|
||||
let selected = entries;
|
||||
if (values.shard) {
|
||||
const [index, count] = values.shard.split("/").map(Number);
|
||||
assert.ok(index >= 1 && index <= count, "--shard must be i/n with 1 <= i <= n");
|
||||
selected = entries.filter((_, k) => k % count === index - 1);
|
||||
}
|
||||
for (const entry of selected) {
|
||||
const head = readFileSync(entry, "utf8").split("\n", 2);
|
||||
const speed = head.some((line) => /^\/\/ @optimization:\s*speed\s*$/.test(line));
|
||||
programs.push({
|
||||
id: `corpus/${relative(corpus, entry)}`,
|
||||
entry,
|
||||
kind: "exe",
|
||||
optimization: speed ? "speed" : "release",
|
||||
});
|
||||
}
|
||||
}
|
||||
return programs.filter((p) => only === null || only.test(p.id));
|
||||
}
|
||||
|
||||
/* ── builds ────────────────────────────────────────────────────────────── */
|
||||
|
||||
function exec(command, args, options = {}) {
|
||||
return new Promise((resolvePromise) => {
|
||||
const started = process.hrtime.bigint();
|
||||
const child = spawn(command, args, { stdio: ["ignore", "pipe", "pipe"], ...options });
|
||||
let stdout = "";
|
||||
let stderr = "";
|
||||
child.stdout.setEncoding("utf8").on("data", (c) => (stdout += c));
|
||||
child.stderr.setEncoding("utf8").on("data", (c) => (stderr += c));
|
||||
child.on("close", (status, signal) =>
|
||||
resolvePromise({
|
||||
status,
|
||||
signal,
|
||||
stdout,
|
||||
stderr,
|
||||
ms: Number(process.hrtime.bigint() - started) / 1e6,
|
||||
}),
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
function buildEnv() {
|
||||
const env = { ...process.env, SCRIPTC_NO_CACHE: "1" };
|
||||
delete env.SCRIPTC_TIMING;
|
||||
delete env.SCRIPTC_CACHE_DIR;
|
||||
return env;
|
||||
}
|
||||
|
||||
function compile(tree, args) {
|
||||
return exec(
|
||||
process.execPath,
|
||||
["--max-old-space-size=8192", join(tree, "packages/cli/dist/bootstrap.js"), "build", ...args],
|
||||
{ cwd: programsRoot, env: buildEnv() },
|
||||
);
|
||||
}
|
||||
|
||||
/** Build one program with one checkout. Executable programs build once,
|
||||
* keeping the LLVM and IR beside the release executable. The self-hosted
|
||||
* compiler emits LLVM only: each of its frontend runs takes minutes, and its
|
||||
* LLVM names the first differing function as well as its IR would. */
|
||||
async function buildProgram(program, label) {
|
||||
const dir = join(work, label, program.id.replace(/[^\w.-]+/g, "_"));
|
||||
mkdirSync(dir, { recursive: true });
|
||||
const tree = trees[label];
|
||||
const failure = (result) => ({
|
||||
ok: false,
|
||||
error: (result.stderr || result.stdout || `exit ${result.status ?? result.signal}`)
|
||||
.replaceAll(dir, "<out>")
|
||||
.trim()
|
||||
.slice(0, 4000),
|
||||
});
|
||||
if (program.kind === "source") {
|
||||
const llvm = join(dir, "program.ll");
|
||||
const result = await compile(tree, [program.entry, "--emit=llvm", "-o", llvm]);
|
||||
if (result.status !== 0) return failure(result);
|
||||
return { ok: true, dir, llvm, ms: result.ms };
|
||||
}
|
||||
const exe = join(dir, "program");
|
||||
const result = await compile(tree, [
|
||||
program.entry,
|
||||
`--optimization=${program.optimization ?? "release"}`,
|
||||
"--keep-llvm",
|
||||
"--emit-ir",
|
||||
"-o",
|
||||
exe,
|
||||
]);
|
||||
if (result.status !== 0) return failure(result);
|
||||
const files = readdirSync(dir);
|
||||
const pick = (suffix) => {
|
||||
const name = files.find((f) => f.endsWith(suffix));
|
||||
return name === undefined ? null : join(dir, name);
|
||||
};
|
||||
return { ok: true, dir, exe, ir: pick(".ir.json"), llvm: pick(".ll"), ms: result.ms };
|
||||
}
|
||||
|
||||
const nmCache = new Map();
|
||||
function nm(binary) {
|
||||
if (nmCache.has(binary)) return nmCache.get(binary);
|
||||
const result = spawnSync("nm", process.platform === "darwin" ? [binary] : ["-S", binary], {
|
||||
encoding: "utf8",
|
||||
maxBuffer: 256 * 1024 * 1024,
|
||||
});
|
||||
const text = nmOutput(binary, result);
|
||||
nmCache.set(binary, text);
|
||||
return text;
|
||||
}
|
||||
|
||||
/** Names of the functions an executable defines. */
|
||||
const definedFunctions = (binary) =>
|
||||
parseNm(nm(binary))
|
||||
.filter((s) => /^[tTwW]$/.test(s.type))
|
||||
.map((s) => s.name);
|
||||
|
||||
/* Bench workloads whose release binaries link dynamic-tier symbols on
|
||||
* b14377864 (DYNAMIC_TIER_PREFIXES in zero-cost-compare.mjs): JSON.parse
|
||||
* trees, worker messages, regex exec metadata (two-tier §3.3 L5), and a
|
||||
* shared scr_dyn_call/scr_dyn_define_property core not yet traced to its
|
||||
* typed root. Every other bench workload is fully typed and must link none.
|
||||
* Remove a workload here when a leak fix makes its binary clean. */
|
||||
const DYNAMIC_WORKLOADS = new Set([
|
||||
"bench/json-records",
|
||||
"bench/cli-config",
|
||||
"bench/http-api",
|
||||
"bench/validate-errors",
|
||||
"bench/regex-logs",
|
||||
"bench/atomics-contended",
|
||||
"bench/intern-table",
|
||||
"bench/typed-array-scan-workers",
|
||||
"bench/log-summary",
|
||||
"bench/inventory-report",
|
||||
]);
|
||||
|
||||
/** The I3 symbol check. Every executable may not gain dynamic-tier symbols
|
||||
* over the baseline (unless the program matches --allow); a fully typed
|
||||
* bench workload may contain none at all. */
|
||||
function dynamicTierCheck(program, baseline, candidate, artifacts, allowed) {
|
||||
if (!baseline.exe || !artifacts.exe) return null;
|
||||
const typed = program.id.startsWith("bench/") && !DYNAMIC_WORKLOADS.has(program.id);
|
||||
if (artifacts.exe.identical && !typed) return null;
|
||||
const after = dynamicTierSymbols(definedFunctions(candidate.exe));
|
||||
const before = artifacts.exe.identical
|
||||
? after
|
||||
: dynamicTierSymbols(definedFunctions(baseline.exe));
|
||||
const added = after.filter((name) => !before.includes(name));
|
||||
const failures = [];
|
||||
if (typed && after.length > 0)
|
||||
failures.push(
|
||||
`fully typed program links dynamic-tier symbols: ${after.slice(0, 5).join(", ")}`,
|
||||
);
|
||||
if (added.length > 0 && !allowed)
|
||||
failures.push(`gained dynamic-tier symbols: ${added.slice(0, 5).join(", ")}`);
|
||||
return { typed, count: after.length, added, failures };
|
||||
}
|
||||
|
||||
function compareBuilds(baseline, candidate) {
|
||||
const artifacts = {};
|
||||
for (const kind of ["ir", "llvm"]) {
|
||||
if (!(kind in baseline)) continue;
|
||||
if (baseline[kind] === null || candidate[kind] === null) {
|
||||
artifacts[kind] = { identical: false, first: `missing ${kind} artifact` };
|
||||
continue;
|
||||
}
|
||||
artifacts[kind] = compareArtifact(
|
||||
kind,
|
||||
readFileSync(baseline[kind]),
|
||||
readFileSync(candidate[kind]),
|
||||
);
|
||||
}
|
||||
if (baseline.exe) {
|
||||
const a = readFileSync(baseline.exe);
|
||||
const b = readFileSync(candidate.exe);
|
||||
artifacts.exe = a.equals(b)
|
||||
? { identical: true }
|
||||
: compareExecutables(a, b, nm(baseline.exe), nm(candidate.exe));
|
||||
artifacts.exe.bytes = { baseline: a.length, candidate: b.length };
|
||||
}
|
||||
return artifacts;
|
||||
}
|
||||
|
||||
/** Run tasks with at most `jobs` at a time; heavy tasks run alone. */
|
||||
async function pool(tasks) {
|
||||
const results = new Array(tasks.length);
|
||||
let next = 0;
|
||||
let running = 0;
|
||||
let heavy = false;
|
||||
await new Promise((done) => {
|
||||
const pump = () => {
|
||||
if (next === tasks.length && running === 0) return done();
|
||||
while (next < tasks.length && running < jobs && !heavy) {
|
||||
const task = tasks[next];
|
||||
if (task.heavy && running > 0) break;
|
||||
const index = next++;
|
||||
running++;
|
||||
if (task.heavy) heavy = true;
|
||||
task.run().then((value) => {
|
||||
results[index] = value;
|
||||
running--;
|
||||
if (task.heavy) heavy = false;
|
||||
pump();
|
||||
});
|
||||
}
|
||||
};
|
||||
pump();
|
||||
});
|
||||
return results;
|
||||
}
|
||||
|
||||
/* ── cachegrind ────────────────────────────────────────────────────────── */
|
||||
|
||||
async function instructionCounts(binary, args, out) {
|
||||
const result = await exec("valgrind", [
|
||||
"--tool=cachegrind",
|
||||
"--cache-sim=no",
|
||||
`--cachegrind-out-file=${out}`,
|
||||
binary,
|
||||
...args,
|
||||
]);
|
||||
// Corpus programs may exit non-zero on purpose (`// @exit: 1`).
|
||||
if (result.signal !== null || !existsSync(out)) return null;
|
||||
return cachegrindCounts(readFileSync(out, "utf8"), new Set(definedFunctions(binary)));
|
||||
}
|
||||
|
||||
/* ── main ──────────────────────────────────────────────────────────────── */
|
||||
|
||||
function revision(root) {
|
||||
const r = spawnSync("git", ["-C", root, "rev-parse", "--short", "HEAD"], { encoding: "utf8" });
|
||||
if (r.status !== 0) return null;
|
||||
const dirty = spawnSync("git", ["-C", root, "status", "--porcelain", "--untracked-files=no"], {
|
||||
encoding: "utf8",
|
||||
});
|
||||
return r.stdout.trim() + (dirty.stdout.trim() ? "+dirty" : "");
|
||||
}
|
||||
|
||||
const fmtMs = (ms) => `${(ms / 1000).toFixed(2)}s`;
|
||||
const pct = (ratio) => `${ratio >= 1 ? "+" : ""}${((ratio - 1) * 100).toFixed(2)}%`;
|
||||
|
||||
const programs = goodCodeSet();
|
||||
assert.ok(programs.length > 0, "the selected good-code set is empty");
|
||||
const report = {
|
||||
schema: 1,
|
||||
date: new Date().toISOString(),
|
||||
host: { platform: process.platform, arch: process.arch, node: process.version },
|
||||
trees: Object.fromEntries(
|
||||
Object.entries(trees).map(([label, root]) => [label, { root, revision: revision(root) }]),
|
||||
),
|
||||
programsRoot,
|
||||
allow: values.allow ?? null,
|
||||
programs: [],
|
||||
timing: null,
|
||||
failures: [],
|
||||
};
|
||||
console.log(
|
||||
`zero-cost gate: ${programs.length} programs (${[...sets].join(", ")}), ` +
|
||||
`baseline ${trees.baseline}, candidate ${trees.candidate}, programs from ${programsRoot}`,
|
||||
);
|
||||
|
||||
let failed = false;
|
||||
const started = Date.now();
|
||||
await pool(
|
||||
programs.map((program) => ({
|
||||
heavy: program.kind === "source",
|
||||
run: async () => {
|
||||
const [baseline, candidate] = await Promise.all([
|
||||
buildProgram(program, "baseline"),
|
||||
buildProgram(program, "candidate"),
|
||||
]);
|
||||
const allowed =
|
||||
allow !== null &&
|
||||
programInputs(program.entry).some((f) => allow.test(readFileSync(f, "utf8")));
|
||||
const artifacts = baseline.ok && candidate.ok ? compareBuilds(baseline, candidate) : {};
|
||||
const { verdict, fails } = programVerdict({ baseline, candidate, artifacts, allowed });
|
||||
const entry = {
|
||||
id: program.id,
|
||||
verdict,
|
||||
allowed,
|
||||
buildMs: { baseline: baseline.ms ?? null, candidate: candidate.ms ?? null },
|
||||
artifacts,
|
||||
};
|
||||
if (!baseline.ok) entry.baselineError = baseline.error;
|
||||
if (!candidate.ok) entry.candidateError = candidate.error;
|
||||
const size = artifacts.exe?.bytes;
|
||||
if (verdict === "allowed" && size && size.candidate > size.baseline * (1 + maxSizeGrowth)) {
|
||||
entry.sizeFailure = true;
|
||||
report.failures.push(
|
||||
`${program.id}: executable grew ${size.baseline} -> ${size.candidate} bytes`,
|
||||
);
|
||||
}
|
||||
if (fails) report.failures.push(`${program.id}: ${verdict}`);
|
||||
const symbols =
|
||||
baseline.ok && candidate.ok
|
||||
? dynamicTierCheck(program, baseline, candidate, artifacts, allowed)
|
||||
: null;
|
||||
if (symbols !== null) entry.dynamicTier = symbols;
|
||||
for (const failure of symbols?.failures ?? [])
|
||||
report.failures.push(`${program.id}: ${failure}`);
|
||||
if (fails || entry.sizeFailure || symbols?.failures.length) failed = true;
|
||||
if (values.cachegrind && artifacts.exe && !artifacts.exe.identical) {
|
||||
// Bench workloads run with their benchmark arguments; corpus
|
||||
// programs take none.
|
||||
const w = program.workload ?? { args: [] };
|
||||
if (w.kind === "server" || w.args.includes("{input}"))
|
||||
entry.cachegrind = { skipped: "needs a server client or generated input" };
|
||||
else {
|
||||
const counts = await Promise.all([
|
||||
instructionCounts(baseline.exe, w.args, join(baseline.dir, "cg.out")),
|
||||
instructionCounts(candidate.exe, w.args, join(candidate.dir, "cg.out")),
|
||||
]);
|
||||
entry.cachegrind = counts.includes(null)
|
||||
? { skipped: "valgrind run failed" }
|
||||
: (({ deltas, ...rest }) => ({ ...rest, deltas: deltas.slice(0, 20) }))(
|
||||
compareCounts(counts[0], counts[1]),
|
||||
);
|
||||
}
|
||||
}
|
||||
printProgram(entry);
|
||||
report.programs.push(entry);
|
||||
nmCache.delete(baseline.exe);
|
||||
nmCache.delete(candidate.exe);
|
||||
if (!values.keep) {
|
||||
if (baseline.dir) rmSync(baseline.dir, { recursive: true, force: true });
|
||||
if (candidate.dir) rmSync(candidate.dir, { recursive: true, force: true });
|
||||
}
|
||||
},
|
||||
})),
|
||||
);
|
||||
|
||||
function printProgram(entry) {
|
||||
const label = {
|
||||
identical: "identical",
|
||||
allowed: "allowed ",
|
||||
differs: "DIFFERS ",
|
||||
"build-error": "ERROR ",
|
||||
"both-fail": "both-fail",
|
||||
}[entry.verdict];
|
||||
const notes = [];
|
||||
for (const [kind, result] of Object.entries(entry.artifacts))
|
||||
if (!result.identical) {
|
||||
const counts = result.changed
|
||||
? ` (${result.changed.length} changed, ${result.added.length} added, ${result.removed.length} removed)`
|
||||
: "";
|
||||
notes.push(`${kind}: ${result.first}${counts}`);
|
||||
}
|
||||
const size = entry.artifacts.exe?.bytes;
|
||||
if (size && size.baseline !== size.candidate)
|
||||
notes.push(
|
||||
`size ${size.baseline} -> ${size.candidate} B (${pct(size.candidate / size.baseline)})`,
|
||||
);
|
||||
if (entry.cachegrind?.total) {
|
||||
const { total, deltas } = entry.cachegrind;
|
||||
notes.push(
|
||||
`Ir ${total.baseline} -> ${total.candidate} (${pct(total.candidate / Math.max(1, total.baseline))})` +
|
||||
(deltas[0]
|
||||
? `, largest ${deltas[0].name} ${deltas[0].delta > 0 ? "+" : ""}${deltas[0].delta}`
|
||||
: ""),
|
||||
);
|
||||
} else if (entry.cachegrind?.skipped)
|
||||
notes.push(`cachegrind skipped: ${entry.cachegrind.skipped}`);
|
||||
for (const failure of entry.dynamicTier?.failures ?? []) notes.push(`SYMBOLS: ${failure}`);
|
||||
if (entry.baselineError) notes.push(`baseline: ${entry.baselineError.split("\n")[0]}`);
|
||||
if (entry.candidateError) notes.push(`candidate: ${entry.candidateError.split("\n")[0]}`);
|
||||
console.log(`${label} ${entry.id}${notes.length ? ` ${notes.join("; ")}` : ""}`);
|
||||
}
|
||||
|
||||
/* Compile time: paired --emit=llvm builds, the baseline and the candidate
|
||||
* side by side, one program at a time. Identical LLVM means everything after
|
||||
* emission (LLVM optimization, linking) costs the same, so the timed part is
|
||||
* the compiler's own work. The self-hosted compiler takes minutes per build;
|
||||
* its main comparison build is its first pair. */
|
||||
if (timingRuns > 0) {
|
||||
const rows = [];
|
||||
const pair = async (program) => {
|
||||
const [baseline, candidate] = await Promise.all(
|
||||
["baseline", "candidate"].map((label) =>
|
||||
compile(trees[label], [
|
||||
program.entry,
|
||||
"--emit=llvm",
|
||||
"-o",
|
||||
join(work, `timing-${label}.ll`),
|
||||
]),
|
||||
),
|
||||
);
|
||||
return baseline.status === 0 && candidate.status === 0 ? [baseline.ms, candidate.ms] : null;
|
||||
};
|
||||
for (const program of programs) {
|
||||
if (program.id.startsWith("corpus/")) continue;
|
||||
const row = { id: program.id, baselineMs: [], candidateMs: [], ratios: [] };
|
||||
const add = (sample) => {
|
||||
if (sample === null) return;
|
||||
row.baselineMs.push(Math.round(sample[0]));
|
||||
row.candidateMs.push(Math.round(sample[1]));
|
||||
row.ratios.push(sample[1] / sample[0]);
|
||||
};
|
||||
const runs = program.kind === "source" ? selfTimingRuns : timingRuns;
|
||||
if (program.kind === "source") {
|
||||
const built = report.programs.find((p) => p.id === program.id)?.buildMs;
|
||||
if (built?.baseline && built?.candidate) add([built.baseline, built.candidate]);
|
||||
}
|
||||
for (let run = 0; run < runs; run++) add(await pair(program));
|
||||
if (row.ratios.length === 0) continue;
|
||||
rows.push(row);
|
||||
console.log(
|
||||
`time ${program.id} ${fmtMs(median(row.baselineMs))} -> ${fmtMs(median(row.candidateMs))} ` +
|
||||
`(${pct(median(row.ratios))}, ${row.ratios.length} pairs)`,
|
||||
);
|
||||
}
|
||||
report.timing = { runs: timingRuns, maxGrowth: maxTimeGrowth, rows, groups: {} };
|
||||
const groups = {
|
||||
bench: rows.filter((r) => r.id.startsWith("bench/")),
|
||||
self: rows.filter((r) => r.id === "self/cli"),
|
||||
};
|
||||
for (const [name, group] of Object.entries(groups)) {
|
||||
if (group.length === 0) continue;
|
||||
const result = timeRatio(group);
|
||||
report.timing.groups[name] = result;
|
||||
const verdict =
|
||||
result.low > 1 + maxTimeGrowth
|
||||
? "SLOWER"
|
||||
: result.high > 1 + maxTimeGrowth
|
||||
? "inconclusive (rerun with more runs)"
|
||||
: "within tolerance";
|
||||
console.log(
|
||||
`time ${name} ${group.length > 1 ? "geomean " : ""}${pct(result.ratio)} ` +
|
||||
`[95% CI ${pct(result.low)}, ${pct(result.high)}]: ${verdict}`,
|
||||
);
|
||||
if (result.low > 1 + maxTimeGrowth) {
|
||||
failed = true;
|
||||
report.failures.push(
|
||||
`${name} compile time grew ${pct(result.ratio)} (CI low ${pct(result.low)})`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const counts = {};
|
||||
for (const p of report.programs) counts[p.verdict] = (counts[p.verdict] ?? 0) + 1;
|
||||
report.counts = counts;
|
||||
report.verdict = failed ? "fail" : "pass";
|
||||
report.elapsedMs = Date.now() - started;
|
||||
if (values.json) {
|
||||
mkdirSync(dirname(resolve(values.json)), { recursive: true });
|
||||
writeFileSync(values.json, JSON.stringify(report, null, 2) + "\n");
|
||||
}
|
||||
if (!values.keep) rmSync(work, { recursive: true, force: true });
|
||||
else console.log(`artifacts kept in ${work}`);
|
||||
console.log(
|
||||
`zero-cost gate: ${failed ? "FAIL" : "PASS"} — ${Object.entries(counts)
|
||||
.map(([k, v]) => `${v} ${k}`)
|
||||
.join(", ")} in ${fmtMs(report.elapsedMs)}`,
|
||||
);
|
||||
for (const failure of report.failures) console.log(` ${failure}`);
|
||||
process.exit(failed ? 1 : 0);
|
||||
@@ -0,0 +1,10 @@
|
||||
// A throwing module-level after hook: the run reports the entry file as a
|
||||
// failed result with the hook's location, after the top-level results,
|
||||
// without counting it, and the exit code stays 0, as in Node.
|
||||
import { after, test } from "node:test";
|
||||
|
||||
after(() => {
|
||||
throw new Error("root after failed");
|
||||
});
|
||||
|
||||
test("t", () => {});
|
||||
@@ -0,0 +1,30 @@
|
||||
// @exit: 1
|
||||
// A throwing module-level before hook: each top-level test fails with its
|
||||
// error without running its body or beforeEach hooks, and each top-level
|
||||
// suite fails with it while the tests and suites in it are cancelled
|
||||
// before they start. Later module-level before hooks still run at their
|
||||
// before() calls, and the first failure is the one reported.
|
||||
import { before, beforeEach, describe, it, test } from "node:test";
|
||||
|
||||
before(() => {
|
||||
throw new Error("root before failed");
|
||||
});
|
||||
before(async () => {
|
||||
await Promise.resolve();
|
||||
throw new Error("a later failure");
|
||||
});
|
||||
beforeEach(() => {
|
||||
throw new Error("beforeEach never runs");
|
||||
});
|
||||
|
||||
describe("s", () => {
|
||||
it("x", () => {});
|
||||
describe("inner", () => {
|
||||
it("y", () => {});
|
||||
});
|
||||
it("z", () => {});
|
||||
});
|
||||
|
||||
test("t", () => {
|
||||
throw new Error("the body never runs");
|
||||
});
|
||||
+44
@@ -0,0 +1,44 @@
|
||||
// Module-level before hooks run at their before() calls, with the root's
|
||||
// context (named "<root>"), and the first top-level test waits for the
|
||||
// async ones; module-level after hooks run after the last top-level test
|
||||
// or suite, and module-level beforeEach hooks run before each test, with
|
||||
// that test's context.
|
||||
import { after, before, beforeEach, describe, it, test } from "node:test";
|
||||
import * as assert from "node:assert";
|
||||
|
||||
const seen: string[] = [];
|
||||
|
||||
before((c) => {
|
||||
seen.push(`before: ${c.name}`);
|
||||
});
|
||||
seen.push("after the before() call");
|
||||
before(async () => {
|
||||
seen.push("async before starts");
|
||||
await new Promise<void>((resolve) => setTimeout(resolve, 5));
|
||||
seen.push("async before ends");
|
||||
});
|
||||
beforeEach((c) => {
|
||||
seen.push(`beforeEach: ${c.name}`);
|
||||
});
|
||||
after(() => {
|
||||
// Throws, and so fails the run visibly, unless every hook ran in order.
|
||||
assert.deepStrictEqual(seen, [
|
||||
"before: <root>",
|
||||
"after the before() call",
|
||||
"async before starts",
|
||||
"suite body",
|
||||
"async before ends",
|
||||
"beforeEach: x",
|
||||
"beforeEach: t",
|
||||
"report",
|
||||
]);
|
||||
});
|
||||
|
||||
describe("s", () => {
|
||||
seen.push("suite body");
|
||||
it("x", () => {});
|
||||
});
|
||||
|
||||
test("t", () => {
|
||||
seen.push("report");
|
||||
});
|
||||
+6015
-5419
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,160 @@
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { expect, test } from "vitest";
|
||||
import {
|
||||
cachegrindCounts,
|
||||
compareArtifact,
|
||||
compareCounts,
|
||||
compareExecutables,
|
||||
dynamicTierSymbols,
|
||||
nmOutput,
|
||||
programVerdict,
|
||||
} from "../../scripts/zero-cost-compare.mjs";
|
||||
|
||||
const ir = (functions: object[], globals: object[]) =>
|
||||
Buffer.from(JSON.stringify({ irVersion: 16, functions, globals, sourceFile: "/p/main.ts" }));
|
||||
|
||||
const llvm = (body: string, extra = "") =>
|
||||
Buffer.from(
|
||||
[
|
||||
"; Generated by scriptc (LLVM backend) from /p/main.ts. Do not edit.",
|
||||
'@.str = private constant [3 x i8] c"hi\\00"',
|
||||
"define internal i64 @add(i64 %a, i64 %b) {",
|
||||
` ${body}`,
|
||||
"}",
|
||||
'define void @"main.$1"() { ret void }',
|
||||
extra,
|
||||
"declare void @scr_print(ptr)",
|
||||
"",
|
||||
].join("\n"),
|
||||
);
|
||||
|
||||
test("the zero-cost gate names the first differing IR and LLVM unit", () => {
|
||||
const f = { name: "f", params: [], body: [{ kind: "return" }] };
|
||||
const g = { name: "g", params: [], body: [] };
|
||||
const base = ir([f, g, f], [{ name: "a", type: "i64" }]);
|
||||
expect(compareArtifact("ir", base, Buffer.from(base))).toEqual({ identical: true });
|
||||
|
||||
const changed = compareArtifact(
|
||||
"ir",
|
||||
base,
|
||||
ir([f, { ...g, params: [1] }, f], [{ name: "a", type: "i64" }]),
|
||||
);
|
||||
expect(changed).toMatchObject({
|
||||
identical: false,
|
||||
first: "changed g",
|
||||
changed: ["g"],
|
||||
added: [],
|
||||
module: null,
|
||||
});
|
||||
|
||||
const header = compareArtifact("ir", base, ir([f, g, f], [{ name: "a", type: "f64" }]));
|
||||
expect(header).toMatchObject({ first: "module globals[0:a]", changed: [] });
|
||||
// Repeated names are matched by occurrence, so an insertion shows as added.
|
||||
expect(compareArtifact("ir", base, ir([f, g, f, f], [{ name: "a", type: "i64" }]))).toMatchObject(
|
||||
{
|
||||
first: "added f#3",
|
||||
},
|
||||
);
|
||||
|
||||
const ll = llvm("%r = add i64 %a, %b\n ret i64 %r");
|
||||
expect(compareArtifact("llvm", ll, Buffer.from(ll))).toEqual({ identical: true });
|
||||
expect(compareArtifact("llvm", ll, llvm("%r = sub i64 %a, %b\n ret i64 %r"))).toMatchObject({
|
||||
first: "changed add",
|
||||
changed: ["add"],
|
||||
functions: { baseline: 2, candidate: 2 },
|
||||
});
|
||||
expect(
|
||||
compareArtifact(
|
||||
"llvm",
|
||||
ll,
|
||||
llvm("%r = add i64 %a, %b\n ret i64 %r", "define void @new() { ret void }"),
|
||||
),
|
||||
).toMatchObject({ first: "added new", added: ["new"] });
|
||||
const global = compareArtifact("llvm", ll, Buffer.from(ll.toString().replace("hi", "ho")));
|
||||
expect(global).toMatchObject({ first: "module line 2", changed: [] });
|
||||
expect(global.module.candidate).toContain("ho");
|
||||
});
|
||||
|
||||
test("the zero-cost gate compares executables and instruction counts per function", () => {
|
||||
const nm =
|
||||
"0000000000001000 0000000000000010 T main\n0000000000001010 0000000000000020 t scr_add\n";
|
||||
const resized = compareExecutables(
|
||||
Buffer.from("abcd"),
|
||||
Buffer.from("abXd"),
|
||||
nm,
|
||||
nm.replace("0000000000000020 t", "0000000000000030 t"),
|
||||
);
|
||||
expect(resized).toMatchObject({ identical: false, first: "resized scr_add", offset: 2 });
|
||||
|
||||
const output = (main: number, prefault: number, loader: number) =>
|
||||
[
|
||||
"events: Ir",
|
||||
"fl=???",
|
||||
"fn=main",
|
||||
`0 ${main}`,
|
||||
"fn=scr_sa_prefault",
|
||||
`0 ${prefault}`,
|
||||
"fn=_dl_lookup_symbol_x",
|
||||
`0 ${loader}`,
|
||||
"fn=scr_add",
|
||||
"0 7",
|
||||
"0 3",
|
||||
].join("\n");
|
||||
const own = new Set(["main", "scr_add", "scr_sa_prefault"]);
|
||||
const base = cachegrindCounts(output(100, 50, 9000), own);
|
||||
expect([...base]).toEqual([
|
||||
["main", 100],
|
||||
["scr_add", 10],
|
||||
]);
|
||||
expect(compareCounts(base, cachegrindCounts(output(100, 75, 9300), own)).identical).toBe(true);
|
||||
expect(compareCounts(base, cachegrindCounts(output(104, 50, 9000), own))).toMatchObject({
|
||||
identical: false,
|
||||
total: { baseline: 110, candidate: 114 },
|
||||
deltas: [{ name: "main", delta: 4 }],
|
||||
});
|
||||
});
|
||||
|
||||
test("the zero-cost gate throws when nm does not list a binary's symbols", () => {
|
||||
const ok = { status: 0, signal: null, stdout: "0000000000001000 T main\n", stderr: "" };
|
||||
expect(nmOutput("prog", ok)).toBe(ok.stdout);
|
||||
const missing = spawnSync("scriptc-missing-nm", ["prog"], { encoding: "utf8" });
|
||||
expect(() => nmOutput("prog", missing)).toThrow(/^nm prog: .*ENOENT/);
|
||||
expect(() =>
|
||||
nmOutput("prog", { status: null, signal: "SIGKILL", stdout: "", stderr: "" }),
|
||||
).toThrow("nm prog: killed by SIGKILL");
|
||||
expect(() =>
|
||||
nmOutput("prog", { status: 1, signal: null, stdout: "", stderr: "nm: prog: no symbols\n" }),
|
||||
).toThrow("nm prog: exited with status 1: nm: prog: no symbols");
|
||||
});
|
||||
|
||||
test("the zero-cost gate recognizes dynamic-tier runtime symbols", () => {
|
||||
expect(
|
||||
dynamicTierSymbols([
|
||||
"main",
|
||||
"scr_dyn_release",
|
||||
"scr_dyn_obj_drop_keys",
|
||||
"scr_dyn_this_push_dyn",
|
||||
"scr_js_load_ic",
|
||||
"scr_dyn_obj_read",
|
||||
"scr_dyn_obj_read",
|
||||
]),
|
||||
).toEqual(["scr_dyn_obj_read", "scr_js_load_ic"]);
|
||||
});
|
||||
|
||||
test("the zero-cost gate fails differing programs unless they match the signature", () => {
|
||||
const ok = { ok: true };
|
||||
const same = { ir: { identical: true }, exe: { identical: true } };
|
||||
const differs = { ir: { identical: true }, exe: { identical: false } };
|
||||
const verdict = (
|
||||
artifacts: object,
|
||||
allowed = false,
|
||||
baseline: object = ok,
|
||||
candidate: object = ok,
|
||||
) => programVerdict({ baseline, candidate, artifacts, allowed });
|
||||
expect(verdict(same)).toEqual({ verdict: "identical", fails: false });
|
||||
expect(verdict(differs)).toEqual({ verdict: "differs", fails: true });
|
||||
expect(verdict(differs, true)).toEqual({ verdict: "allowed", fails: false });
|
||||
const failed = { ok: false, error: "SC1090" };
|
||||
expect(verdict({}, false, failed, failed)).toEqual({ verdict: "both-fail", fails: false });
|
||||
expect(verdict({}, true, ok, failed)).toEqual({ verdict: "build-error", fails: true });
|
||||
});
|
||||
Reference in New Issue
Block a user