Name new corpus programs instead of numbering them

Parallel changes kept taking the same next corpus number (6992 twice, the
Node island programs renumbered several times). The harness never needed
the numbers: it discovers every entry by glob, and 818 programs already
have descriptive names. AGENTS.md now makes a name, without a number, the
convention for new programs and keeps the numbered ones as they are.

The two scripts that sampled programs by number skipped named programs.
They now key a named program by a stable hash of its name, so they sample
it at the same rate; the numbered programs they pick are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Malte Ubl
2026-10-25 15:17:09 -07:00
co-authored by Claude Opus 5.5
parent 8be991aeca
commit 1733a4c84b
3 changed files with 44 additions and 11 deletions
+2
View File
@@ -46,6 +46,8 @@ SCRIPTC_TEST_WORKERS=4 SCRIPTC_SAN=1 pnpm test # sanitized lane
Corpus programs are differential tests against Node: every program runs under Node and as a compiled native binary, and stdout, stderr, and exit codes must match byte-for-byte. A new feature lands with corpus programs that pin its behavior both ways.
Name a new corpus program after what it tests, without a number: lowercase words joined by hyphens, such as `tests/corpus/intersection-restated-members.ts`, or a directory such as `tests/corpus/<name>/main.ts` for a multi-file program. Parallel changes kept taking the same next number; with names, two changes collide only when they add the same name, which git reports as a conflict. Leave existing numbered programs as they are: never rename or renumber them, and name a new program even when it extends a numbered family.
Record a new or remeasured program's CI cost (used to balance the test shards) in `tests/harness/corpus-costs/<program>.json`, not in `tests/harness/corpus-costs.json`; see the README there.
Before opening a PR that adds or changes corpus, diagnostic, or frontend fixtures, run `pnpm test:ts7 --baselines-only`. This fast completeness check uses the same fixture discovery as the parity suite; it does not type-check or replace behavioral parity. Generate missing preflight/order baselines, review their diagnostics and module order, and run the affected `order-parity.test.ts` batches before pushing. Keep existing baseline entries unchanged unless the corresponding behavior intentionally changes. The full `pnpm test:ts7` command also checks completeness before starting its sweep.
+20 -4
View File
@@ -16,13 +16,16 @@
* [--seeds 1,2,3] [--jobs 8] [--entry <file.ts>]... [--out <dir>]
* [--fail-on names|content]
*
* --stride and --offset sample the top-level corpus programs by number, or by
* a hash of the name of a program without one.
*
* Needs a built compiler (pnpm build). Writes <out>/results.json and prints
* a ranking of fields by the programs whose IR they change. With --fail-on,
* it exits with status 1 when some program's IR differs between orders:
* in any way (names), or beyond names and ids (content). */
import { spawn } from "node:child_process";
import { createHash } from "node:crypto";
import { mkdirSync, readFileSync, readdirSync, writeFileSync } from "node:fs";
import { mkdirSync, readFileSync, readdirSync, statSync, writeFileSync } from "node:fs";
import { cpus, tmpdir } from "node:os";
import { basename, join, resolve } from "node:path";
import { fileURLToPath, pathToFileURL } from "node:url";
@@ -306,12 +309,25 @@ if (values["fail-on"] !== undefined && !["names", "content"].includes(values["fa
const out = resolve(values.out);
mkdirSync(out, { recursive: true });
const seeds = values.seeds.split(",").filter(Boolean);
/** A top-level corpus program's sampling key: its number, or, for a program
* named without one, a stable hash of its name (FNV-1a). */
const programKey = (name) =>
/^\d+-/.test(name)
? Number.parseInt(name, 10)
: [...name].reduce((h, c) => Math.imul(h ^ c.charCodeAt(0), 16777619) >>> 0, 2166136261);
/** Numbered programs by number, then the named ones by name. */
const programOrder = (a, b) =>
/^\d+-/.test(a) !== /^\d+-/.test(b)
? /^\d+-/.test(a)
? -1
: 1
: programKey(a) - programKey(b) || (a < b ? -1 : a > b ? 1 : 0);
const corpus = join(root, "tests/corpus");
const programs = [
...readdirSync(corpus)
.filter((name) => /^\d+-.*\.ts$/.test(name))
.filter((name) => Number.parseInt(name, 10) % Number(values.stride) === Number(values.offset))
.sort((a, b) => Number.parseInt(a, 10) - Number.parseInt(b, 10))
.filter((name) => /^[\w-]+\.ts$/.test(name) && statSync(join(corpus, name)).isFile())
.filter((name) => programKey(name) % Number(values.stride) === Number(values.offset))
.sort(programOrder)
.slice(0, Number(values.limit))
.map((name) => join(corpus, name)),
...values.entry.map((entry) => resolve(entry)),
+22 -7
View File
@@ -7,8 +7,9 @@
* Builds an instrumented native compiler for this Linux host, runs it on the
* training builds below, and merges the raw profiles it writes into one
* sparse indexed profile. The training builds are tsc-ts (a cold `speed`
* build, then a rebuild after an edit), every seventh corpus program at the
* default optimization and a further slice in `dev`, and the runtime
* build, then a rebuild after an edit), every seventh top-level corpus
* program (by number, or by a hash of the name of a program without one) at
* the default optimization and a further slice in `dev`, and the runtime
* benchmark workloads. Run it on a quiet Linux x64 host after changes that
* move much compiler code; a stale profile only loses part of its gain. A
* refresh takes about 11 minutes on 16 cores.
@@ -126,16 +127,30 @@ async function pool(tasks) {
return passed;
}
const programNumber = (name) => Number.parseInt(name, 10);
/** A top-level corpus program's sampling key: its number, or, for a program
* named without one, a stable hash of its name (FNV-1a). */
const programKey = (name) =>
/^\d+-/.test(name)
? Number.parseInt(name, 10)
: [...name].reduce((h, c) => Math.imul(h ^ c.charCodeAt(0), 16777619) >>> 0, 2166136261);
/** Numbered programs by number, then the named ones by name. */
const programOrder = (a, b) =>
/^\d+-/.test(a) !== /^\d+-/.test(b)
? /^\d+-/.test(a)
? -1
: 1
: programKey(a) - programKey(b) || (a < b ? -1 : a > b ? 1 : 0);
const corpus = readdirSync(join(root, "tests/corpus"))
.filter((name) => /^\d+-.*\.ts$/.test(name))
.sort((a, b) => programNumber(a) - programNumber(b));
.filter(
(name) => /^[\w-]+\.ts$/.test(name) && statSync(join(root, "tests/corpus", name)).isFile(),
)
.sort(programOrder);
const training = [
...corpus
.filter((name) => programNumber(name) % 7 === 0)
.filter((name) => programKey(name) % 7 === 0)
.map((name) => ({ file: join(root, "tests/corpus", name), optimization: "release" })),
...corpus
.filter((name) => programNumber(name) % 14 === 7)
.filter((name) => programKey(name) % 14 === 7)
.map((name) => ({ file: join(root, "tests/corpus", name), optimization: "dev" })),
...readdirSync(join(root, "benchmarks/runtime/workloads"))
.filter((name) => name.endsWith(".ts"))