Merge main

This commit is contained in:
Chris Tate
2026-10-27 20:40:14 -05:00
80 changed files with 5514 additions and 362 deletions
+1 -1
View File
@@ -54,7 +54,7 @@ Record a new or remeasured program's CI cost (used to balance the test shards) i
Before opening a PR that adds or changes corpus, diagnostic, or frontend fixtures, run `pnpm test:ts7 --baselines-only`. This fast completeness check uses the same fixture discovery as the parity suite; it does not type-check or replace behavioral parity. Generate missing preflight/order baselines, review their diagnostics and module order, and run the affected `order-parity.test.ts` batches before pushing. Keep existing baseline entries unchanged unless the corresponding behavior intentionally changes. The full `pnpm test:ts7` command also checks completeness before starting its sweep.
A change that must not cost good code anything, such as a compatibility fix that replaces a refusal, proves it with the zero-cost gate. The gate compiles the benchmark workloads, the self-hosted compiler, and the TypeScript corpus with two built checkouts. It requires byte-identical IR, LLVM, and release executables, and compile time within tolerance. `--allow=<regex>` names the change's pattern signature: programs whose sources match may differ, and the PR lists them. Any other difference fails the gate, which names the first differing function. The gate also enforces the two-tier runtime separation: no executable may gain symbols from `DYNAMIC_TIER_PREFIXES` in `scripts/zero-cost-compare.mjs`, and fully typed benchmarks may link none; extend that list as dynamic-tier units land. `--quick` checks only the benchmarks; `--cachegrind` adds per-function instruction counts for executables that differ. Run it in a Sandbox with both checkouts prepared:
A change that must not cost good code anything, such as a compatibility fix that replaces a refusal, proves it with the zero-cost gate. The gate compiles the benchmark workloads, the self-hosted compiler, and the TypeScript corpus with two built checkouts. It requires byte-identical IR, LLVM, and release executables, and compile time within tolerance. `--allow=<regex>` names the change's pattern signature: programs whose sources match may differ, and the PR lists them. The gate classifies every other difference. A program is `runtime-only` when its IR and LLVM are identical and its executable differs only through runtime objects that differ between the two checkouts' runtime packs; the gate lists the changed runtime functions each such program links and the size change, and passes with a note, so a runtime C change does not fail the gate when no program's code changes. A program is `nondeterministic` when the baseline, building it again, differs from its own first build in the same artifacts; it also passes with a note. Any other difference is `differs` and fails the gate, which names the first differing function. `--strict` fails runtime-only and nondeterministic programs too, for a change that must leave every executable byte-identical. The gate also enforces the two-tier runtime separation: no executable may gain symbols from `DYNAMIC_TIER_PREFIXES` in `scripts/zero-cost-compare.mjs`, and fully typed benchmarks may link none; extend that list as dynamic-tier units land. `--quick` checks only the benchmarks; `--cachegrind` adds per-function instruction counts for executables that differ. Run it in a Sandbox with both checkouts prepared:
```bash
node scripts/zero-cost-gate.mjs --baseline=/vercel/sandbox/prev --candidate=. --allow='\barguments\b'
+1 -1
View File
@@ -42,7 +42,7 @@ Parsing, type checking, lowering, and LLVM IR generation run one after another o
### Partitions
The LLVM helper divides an executable's LLVM IR into partitions by its size. Up to 1 MiB of LLVM text is one partition, and up to 2 MiB is four. Larger programs get one partition per 4 MiB of LLVM text, rounded up to a power of two, and at least 8 and at most 64. `--emit=llvm` writes this text, so its file size shows how many partitions a program gets.
The LLVM helper divides an executable's LLVM IR into partitions by its size. Up to 1 MiB of LLVM text is one partition, and up to 2 MiB is four. Larger programs get one partition per 4 MiB of LLVM text, rounded up to a power of two, and at least 8 and at most 64. `--emit=llvm` writes this text, so its file size shows how many partitions a program gets. `dev` executables, whose partitions are compiled on their own with nothing imported, are divided from 64 KiB: up to 128 KiB is four partitions, and larger `dev` programs get at least 8.
<table>
<thead>
+2
View File
@@ -53,6 +53,8 @@ fib-linux: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically
Linux executables use the native Linux implementations of the event loop (epoll), servers, TLS (with the system's CA bundle), and `fs.watch`, and they are tested against Node.js on Linux, including Alpine.
An executable with at least 4 MiB of program code is linked with 2 MiB segment alignment when lld or zig links it, so the kernel can map its code with 2 MiB pages instead of 4 KiB pages. This takes effect when transparent huge pages are set to `madvise` or `always` and the file system keeps files in the page cache as 2 MiB folios (XFS, for example). scriptc writes such an executable with one write so that the page cache holds it that way; a copy written in small chunks maps with 4 KiB pages until it is read from disk again. The alignment also aligns the executable's load address to 2 MiB, which leaves 19 bits of address randomization for it instead of 28. Smaller executables, and links by GNU ld, keep 4 KiB alignment.
### Windows (x86_64)
```console
@@ -34,8 +34,8 @@
A `Duplex.from` body function may take the `{ signal }` argument, which is aborted with the error the Duplex is destroyed with.
`stream.compose` accepts streams and async generator functions in any position, an async function as the last stage, and an array, a string, or a generator as the first stage; one stage is `Duplex.from` of it.
A web `ReadableStream` is read as an async iterable by `Readable.from` and as the first stage of `compose` or `pipeline`, and through its reader by `Readable.fromWeb` (with `objectMode`, `highWaterMark`, `encoding`, and `signal` in an options object literal) and `Duplex.from`.
`new TransformStream()` without arguments builds the identity transform: its readable side is a web `ReadableStream`, its writable side supports `getWriter`, `abort`, `close`, and `locked`, and the writer supports `write`, `close`, `abort`, `releaseLock`, `ready`, `closed`, and `desiredSize`, with Node's backpressure (a write completes when the readable side reads its chunk).
A web stream in a later compose stage, `Readable.toWeb`, a `TransformStream` with a transformer or queuing strategies, the `WritableStream` constructor, `pipeTo`, and `pipeThrough` are unsupported.
In TypeScript, `new TransformStream()` without arguments builds the identity transform: its readable side is a web `ReadableStream`, its writable side supports `getWriter`, `abort`, `close`, and `locked`, and the writer supports `write`, `close`, `abort`, `releaseLock`, `ready`, `closed`, and `desiredSize`, with Node's backpressure (a write completes when the readable side reads its chunk).
A web stream in a later compose stage and `Readable.toWeb` are unsupported, and in TypeScript, so are a `TransformStream` with a transformer or queuing strategies, the `WritableStream` constructor, `pipeTo`, and `pipeThrough`.
The `ERR_INVALID_ARG_VALUE` message for a stream that is no longer readable or writable names the stream's class instead of printing the stream.
`readable.compose(stage, { signal })` is `stream.compose(readable, stage)` with the signal added; the options are an object literal whose `signal` is a variable or a property read.
`pipeline` from `node:stream/promises` accepts a trailing `{ signal, end }` object, with or without function stages, and `finished` accepts `readable`, `writable`, `error`, `cleanup`, and `signal` options, with boolean literals for the flags.
@@ -60,8 +60,9 @@
The `readline/promises` `Readline` class writes to the same outputs.
`readline.emitKeypressEvents` emits `'keypress'` events on `process.stdin` or a readable stream, shared with terminal interfaces on that stream as in Node; a listener's first parameter must admit `undefined`, which it receives for escape sequences.
Chained listener registration is unsupported.
`node:stream/web` exports the same `ReadableStream` class as the global; in TypeScript, its other classes are unsupported.
In JavaScript, `ReadableStream`, `ReadableStreamDefaultReader`, `ReadableStreamDefaultController`, `CountQueuingStrategy`, and `ByteLengthQueuingStrategy`, as globals and as `node:stream/web` exports, are constructor values with Node's prototypes, brand checks, and errors.
Their underlying sources (`start`, `pull`, `cancel`), queuing strategies with a `size` function, readers, `tee`, `ReadableStream.from`, and async iteration settle promises in Node's order.
`node:stream/web` exports the same classes as the globals; in TypeScript, only `ReadableStream` and the identity `TransformStream` have lowerings.
In JavaScript, `ReadableStream`, `WritableStream`, and `TransformStream` with their default readers, writers, and controllers, `CountQueuingStrategy`, `ByteLengthQueuingStrategy`, `TextEncoderStream`, and `TextDecoderStream`, as globals and as `node:stream/web` exports, are constructor values with Node's prototypes, brand checks, and errors.
Their underlying sources, sinks, and transformers, queuing strategies with a `size` function, readers and writers, `tee`, `pipeTo` and `pipeThrough` with their options and an abort signal, `ReadableStream.from`, and async iteration settle promises in Node's order, and the streams take own properties.
`TextDecoderStream` decodes UTF-8; its other encodings are unsupported.
A JavaScript `ReadableStream` given as a `Response` or `Request` body, or as `fetch`'s body, is read through a stream of its own: `response.body` is that stream rather than the one passed in, and the stream passed in is locked from the start.
In JavaScript, a byte stream (`type: 'bytes'`), `pipeTo`, `pipeThrough`, `Object.getPrototypeOf` on these streams, readers, controllers, and strategies, their use as `WeakMap` and `WeakSet` keys, `ReadableStream.from` of a function that is not iterable, and the other `node:stream/web` classes are unsupported.
In JavaScript, a byte stream (`type: 'bytes'`), a BYOB reader, a class that extends one of these classes, `Object.getPrototypeOf` on these values, their use as `WeakMap` and `WeakSet` keys, and `ReadableStream.from` of a function that is not iterable are unsupported.
@@ -81289,9 +81289,9 @@
"inherited": true
},
"static": {
"status": "not-implemented",
"detail": "No static compiler lowering is registered for this API yet.",
"verification": "registry-gap"
"status": "partial",
"detail": "Implemented for the documented static async-iteration subset.",
"verification": "test-backed"
}
},
{
@@ -111929,6 +111929,8 @@
"tests/corpus/stream-web-readable.ts",
"tests/corpus/stream-web-require.cjs",
"tests/corpus/web-stream-readable-untyped.mjs",
"tests/corpus/web-stream-transform-pipes-untyped.mjs",
"tests/corpus/web-stream-writable-untyped.mjs",
"tests/harness/fetch-conformance.test.ts"
]
},
@@ -113011,8 +113013,11 @@
"inherited": true
},
"static": {
"status": "not-implemented",
"evidence": "compiler-unmatched:webstreams"
"status": "partial",
"evidence": "compiler-feature:webstreams.WritableStream",
"tests": [
"tests/corpus/web-stream-writable-untyped.mjs"
]
},
"anchorSource": "exact"
},
@@ -113405,7 +113410,8 @@
"status": "partial",
"evidence": "compiler-feature:webstreams.TransformStream",
"tests": [
"tests/corpus/7743-transform-stream-identity.ts"
"tests/corpus/7743-transform-stream-identity.ts",
"tests/corpus/web-stream-transform-pipes-untyped.mjs"
]
},
"anchorSource": "exact"
+2 -1
View File
@@ -434,7 +434,8 @@
{ "chapter": "dns", "symbols": ["dns.lookup", "dns.lookupService", "dns.getServers", "dns.getDefaultResultOrder", "dns.setDefaultResultOrder"], "status": "partial", "evidence": ["tests/corpus/5648-dns-lookup-forms.ts", "tests/corpus/5649-dns-service-servers.ts", "tests/corpus/5650-dns-lookup-errors-untyped.cjs"] },
{ "chapter": "dns", "symbols": ["dnsPromises.lookup", "dnsPromises.lookupService", "dnsPromises.getDefaultResultOrder"], "status": "partial", "evidence": ["tests/corpus/dns-promises-lookup.ts", "tests/corpus/dns-promises-untyped.cjs"] },
{ "chapter": "webstreams", "symbols": ["ReadableStream"], "status": "partial", "evidence": ["tests/corpus/stream-web-readable.ts", "tests/corpus/web-stream-readable-untyped.mjs", "tests/corpus/stream-web-require.cjs", "tests/corpus/2862-web-readable-async-iteration.ts"] },
{ "chapter": "webstreams", "symbols": ["TransformStream"], "status": "partial", "evidence": ["tests/corpus/7743-transform-stream-identity.ts"] },
{ "chapter": "webstreams", "symbols": ["TransformStream"], "status": "partial", "evidence": ["tests/corpus/7743-transform-stream-identity.ts", "tests/corpus/web-stream-transform-pipes-untyped.mjs"] },
{ "chapter": "webstreams", "symbols": ["WritableStream"], "status": "partial", "evidence": ["tests/corpus/web-stream-writable-untyped.mjs"] },
{ "chapter": "globals", "signatures": ["Class: `TransformStream`"], "status": "partial", "evidence": ["tests/corpus/7743-transform-stream-identity.ts"] }
],
"dedicated": [
@@ -0,0 +1,89 @@
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { afterEach, expect, test } from "vitest";
import {
HUGE_TEXT_MIN_CODE_BYTES,
elfObjectCodeBytes,
hugeTextLinkerArgs,
hugeTextProgram,
} from "./huge-text.js";
const dirs: string[] = [];
afterEach(() => {
for (const dir of dirs.splice(0)) rmSync(dir, { recursive: true, force: true });
});
/** An ELF64 little-endian relocatable object's headers: one section per
* entry, as [type, flags, size]. Section contents are left out, since only
* the headers are read. */
function elfObject(sections: readonly [number, number, number][]): Buffer {
const shoff = 64;
const bytes = Buffer.alloc(shoff + (sections.length + 1) * 64);
bytes.writeUInt32BE(0x7f454c46, 0);
bytes[4] = 2; // ELFCLASS64
bytes[5] = 1; // ELFDATA2LSB
bytes[6] = 1;
bytes.writeUInt16LE(1, 16); // ET_REL
bytes.writeUInt16LE(62, 18); // EM_X86_64
bytes.writeBigUInt64LE(BigInt(shoff), 40);
bytes.writeUInt16LE(64, 52);
bytes.writeUInt16LE(64, 58);
bytes.writeUInt16LE(sections.length + 1, 60);
sections.forEach(([type, flags, size], index) => {
const at = shoff + (index + 1) * 64;
bytes.writeUInt32LE(type, at + 4);
bytes.writeBigUInt64LE(BigInt(flags), at + 8);
bytes.writeBigUInt64LE(BigInt(size), at + 32);
});
return bytes;
}
function file(name: string, bytes: Buffer): string {
const dir = mkdtempSync(join(tmpdir(), "scriptc-huge-text-"));
dirs.push(dir);
const path = join(dir, name);
writeFileSync(path, bytes);
return path;
}
const PROGBITS = 1;
const NOBITS = 8;
const ALLOC = 0x2;
const EXECINSTR = 0x4;
test("an object's code is the size of its executable PROGBITS sections", () => {
const object = file(
"program.o",
elfObject([
[PROGBITS, ALLOC | EXECINSTR, 3 << 20],
[PROGBITS, ALLOC | EXECINSTR, 1000],
[PROGBITS, ALLOC, 5 << 20],
[NOBITS, ALLOC, 7 << 20],
]),
);
expect(elfObjectCodeBytes(object)).toBe((3 << 20) + 1000);
expect(elfObjectCodeBytes(file("not-elf.o", Buffer.from("not an object")))).toBe(0);
expect(elfObjectCodeBytes(join(object, "..", "missing.o"))).toBe(0);
});
test("only Linux programs with an interior 2 MiB window of code link for huge pages", () => {
const linux = { platform: "linux" } as const;
const half = file(
"half.o",
elfObject([[PROGBITS, ALLOC | EXECINSTR, HUGE_TEXT_MIN_CODE_BYTES / 2]]),
);
const small = file("small.o", elfObject([[PROGBITS, ALLOC | EXECINSTR, 1 << 20]]));
expect(hugeTextProgram(linux, [half, small])).toBe(false);
// Partitions of one program count together.
expect(hugeTextProgram(linux, [half, half])).toBe(true);
expect(hugeTextProgram({ platform: "darwin" } as const, [half, half])).toBe(false);
expect(hugeTextProgram(linux, [])).toBe(false);
});
test("the 2 MiB alignment applies to lld and zig links of huge-text programs", () => {
expect(hugeTextLinkerArgs(true, true)).toEqual(["-Wl,-z,max-page-size=0x200000"]);
// GNU ld pads the file to the alignment.
expect(hugeTextLinkerArgs(true, false)).toEqual([]);
expect(hugeTextLinkerArgs(false, true)).toEqual([]);
});
+110
View File
@@ -0,0 +1,110 @@
/**
* Text on 2 MiB pages for large Linux executables.
*
* The kernel maps a 2 MiB window of an executable's text with one huge page
* when the window's virtual address and file offset are both 2 MiB aligned
* and the page cache holds the window as one 2 MiB folio. A link with
* `-z max-page-size=0x200000` keeps every segment's address congruent to its
* file offset modulo 2 MiB, so every 2 MiB window inside the text segment
* qualifies. lld and zig add no file padding for it; GNU ld pads the file to
* the alignment, so links that fall back to it leave the flag out.
*
* The alignment also aligns a position-independent executable's load
* address to 2 MiB, which leaves 19 bits of address randomization for it
* instead of 28. Only executables whose program code fills an interior 2 MiB
* window can gain from huge pages, so smaller executables link as before.
*
* lld writes its output through a memory mapping, which leaves the file in
* the page cache as 4 KiB folios. A large executable is published by writing
* its bytes with one write instead, which leaves 2 MiB folios where the file
* system supports them. The runtime advises the interior windows huge pages
* at startup (scr_lib.c), so a file that is read back from disk is read as
* 2 MiB folios too.
*/
import {
chmodSync,
closeSync,
openSync,
readFileSync,
readSync,
renameSync,
statSync,
writeFileSync,
} from "node:fs";
import type { NativeTargetSpec } from "./targets.js";
/** Program code from which an executable is linked for huge pages: text of
* this size has at least one 2 MiB window inside it wherever it starts. */
export const HUGE_TEXT_MIN_CODE_BYTES = 4 << 20;
const SHF_EXECINSTR = 0x4n;
/** The bytes of executable sections in an ELF64 little-endian object, or 0
* when the file is not one or cannot be read. Reads only the file and
* section headers. */
export function elfObjectCodeBytes(path: string): number {
let fd: number;
try {
fd = openSync(path, "r");
} catch {
return 0;
}
try {
const header = Buffer.alloc(64);
if (readSync(fd, header, 0, 64, 0) !== 64) return 0;
const view = new DataView(header.buffer, header.byteOffset, header.byteLength);
if (view.getUint32(0, false) !== 0x7f454c46 || header[4] !== 2 || header[5] !== 1) return 0;
const shoff = Number(view.getBigUint64(40, true));
const shentsize = view.getUint16(58, true);
let shnum = view.getUint16(60, true);
if (shentsize !== 64 || shoff === 0) return 0;
if (shnum === 0) {
const first = Buffer.alloc(64);
if (readSync(fd, first, 0, 64, shoff) !== 64) return 0;
shnum = Number(new DataView(first.buffer, first.byteOffset, 64).getBigUint64(32, true));
}
const table = Buffer.alloc(shnum * 64);
if (readSync(fd, table, 0, table.length, shoff) !== table.length) return 0;
const sections = new DataView(table.buffer, table.byteOffset, table.byteLength);
let code = 0;
for (let i = 0; i < shnum; i++) {
const at = i * 64;
// SHT_PROGBITS sections with SHF_EXECINSTR.
if (sections.getUint32(at + 4, true) !== 1) continue;
if ((sections.getBigUint64(at + 8, true) & SHF_EXECINSTR) === 0n) continue;
code += Number(sections.getBigUint64(at + 32, true));
}
return code;
} finally {
closeSync(fd);
}
}
/** Whether a link of these program objects is laid out for huge pages. */
export function hugeTextProgram(
target: Pick<NativeTargetSpec, "platform">,
programObjects: readonly string[],
): boolean {
if (target.platform !== "linux") return false;
let code = 0;
for (const object of programObjects) {
code += elfObjectCodeBytes(object);
if (code >= HUGE_TEXT_MIN_CODE_BYTES) return true;
}
return false;
}
/** The link arguments that align a huge-text executable's segments to
* 2 MiB, for links by lld or zig. */
export function hugeTextLinkerArgs(hugeText: boolean, linksWithLldOrZig: boolean): string[] {
return hugeText && linksWithLldOrZig ? ["-Wl,-z,max-page-size=0x200000"] : [];
}
/** Rewrites a linked huge-text executable with one write, so the page cache
* holds it in 2 MiB folios; its caller then publishes it as usual. */
export function rewriteHugeTextExecutable(linked: string): void {
const written = `${linked}.written`;
writeFileSync(written, readFileSync(linked));
chmodSync(written, statSync(linked).mode & 0o7777);
renameSync(written, linked);
}
@@ -13,6 +13,7 @@ import type { NativeArtifactDependency } from "./native/contracts.js";
import { needsDarwinDebugSymbols } from "./debug-symbols.js";
import { loadRuntimePack, type RuntimePackSelection } from "./runtime-pack.js";
import { executableLinkInputs } from "./link-plan-core.js";
import { hugeTextProgram } from "./huge-text.js";
import type { NativeTargetSpec, WindowsSubsystem } from "./targets.js";
export interface NativeLinkPlan {
@@ -30,6 +31,8 @@ export interface NativeLinkPlan {
darwinDebugSymbols?: boolean;
/** Linked with `--static`: the linked executable is checked before use. */
staticLink?: boolean;
/** Program code large enough for text on huge pages (huge-text.ts). */
hugeText?: boolean;
}
export async function createNativeLinkPlan(options: {
@@ -85,5 +88,11 @@ export async function createNativeLinkPlan(options: {
programObjectDependencies: [...(options.programObjectDependencies ?? [])],
runtimePack,
...(options.staticLink === true ? { staticLink: true } : {}),
...(hugeTextProgram(options.target, [
options.programObject,
...(options.programPartitions ?? []),
])
? { hugeText: true }
: {}),
};
}
+4
View File
@@ -24,6 +24,7 @@ import { type NativeArtifactDependency } from "./native/contracts.js";
import { RuntimePackError, stageRuntimePackArtifacts } from "./runtime-pack.js";
import type { NativeLinkPlan } from "./link-plan.js";
import { inputsWithoutLld, lldLinkerArgs } from "./lld.js";
import { hugeTextLinkerArgs, rewriteHugeTextExecutable } from "./huge-text.js";
import { systemLibraryLinkerArgs } from "./link-plan-core.js";
import {
createDarwinDebugSymbols,
@@ -248,11 +249,13 @@ export async function linkNativeExecutable(
stagedRoot = staged.root;
// Linux links with lld when the default driver finds it.
const lld = explicitLinker === undefined ? lldLinkerArgs(plan.target.platform, linker) : [];
const zig = explicitLinker === undefined && plan.target.defaultLinker === "zig";
const linkArgs = (withLld: boolean): string[] => {
const inputs = plan.inputs.map((input) => staged.replacements.get(input) ?? input);
return [
...(explicitLinker === undefined ? plan.target.defaultLinkerArgs : []),
...(withLld ? lld : []),
...hugeTextLinkerArgs(plan.hugeText === true, (withLld && lld.length !== 0) || zig),
...plan.driverFlags,
...mapArgs,
...(withLld ? inputs : inputsWithoutLld(plan.target.platform, inputs)),
@@ -298,6 +301,7 @@ export async function linkNativeExecutable(
});
const output = await stat(privateOut);
if (!output.isFile() || output.size === 0) throw new Error("linker produced no executable");
if (plan.hugeText === true) rewriteHugeTextExecutable(privateOut);
if (mapArgs.length !== 0) {
const refusal = staticExecutableRefusal(plan.target, true, await readFile(linkMap, "utf8"));
if (refusal !== null) throw new NativeCodegenError("SC3002", refusal);
@@ -9,7 +9,8 @@ import { everyExprChild, everyStmtChild } from "../../ir/traverse.js";
import { borrowsStringInputs } from "./string-lifetimes.js";
import { borrowsMapReadInputs } from "./map-read-lifetimes.js";
import { LazyCaptures } from "./lazy-captures.js";
import { storedLastUseRead } from "./local-extents.js";
import { BodyLastUses, storedLastUseRead } from "./local-extents.js";
import { InternalCompilerError } from "../../errors.js";
interface ForwardedUse {
callee: string;
@@ -497,6 +498,9 @@ export function analyzeCallLifetimes(
// environment box takes a reference of its own when it is created.
const unchangedCaptures = lazyCaptures.parameters(fn);
const walks = walkParameters.get(fn.name);
// One walk of the body answers every parameter's last use (consumedParameter).
let lastUses: BodyLastUses | undefined;
const bodyLastUses = (): BodyLastUses => (lastUses ??= new BodyLastUses(fn.body));
fn.params.forEach((param, index) => {
const local = locals.get(param.localId);
if (
@@ -511,7 +515,7 @@ export function analyzeCallLifetimes(
!walks?.has(index) &&
!unchangedCaptures.has(local.id) &&
param.name !== "this" &&
consumedParameter(fn, param.localId)
consumedParameter(fn, param.localId, bodyLastUses)
)
consumed.push({ fn: fn.name, index });
}
@@ -626,6 +630,9 @@ export function analyzeCallLifetimes(
return result;
}
/** SCRIPTC_EMIT_FACTS_VERIFY=1: every BodyLastUses answer is checked against storedLastUseRead. */
const verifiesLastUses = process.env["SCRIPTC_EMIT_FACTS_VERIFY"] === "1";
/** A parameter whose last use in the body stores it (a field, record or
* element store, a record literal field, a push) and moves its reference
* there (lastUseRead). Its owned convention lets a caller hand over a
@@ -633,8 +640,12 @@ export function analyzeCallLifetimes(
* copy while the caller releases its temporary right after the call. The
* read must be the only node of its kind in the function, like every
* moved read. */
function consumedParameter(fn: IrFunction, localId: string): boolean {
const read = storedLastUseRead(fn.body, -1, localId, true);
function consumedParameter(fn: IrFunction, localId: string, lastUses: () => BodyLastUses): boolean {
const read = lastUses().storedRead(localId);
if (verifiesLastUses && read !== storedLastUseRead(fn.body, -1, localId, true))
throw new InternalCompilerError(
`call lifetimes: the body's last use of ${localId} in ${fn.name} differs`,
);
if (read === null) return false;
let count = 0;
const expr = (e: IrExpr): boolean => {
@@ -0,0 +1,46 @@
import { expect, test } from "vitest";
import type { IntegerCountedForLoop } from "../../ir/integer-loops.js";
import { F64, type IrExpr } from "../../ir/ir.js";
import type { IntegerRange } from "../../ir/integer-ranges.js";
import { BlockBuilder } from "./blocks.js";
import { emitCountedLoopLimit } from "./counted-loops.js";
import type { LlValue, LlvmEmitterContext } from "./expr-context.js";
const loc = { file: "test.ts", start: 0, end: 0 };
/** The endpoint emitCountedLoopLimit answers for a limit that emits as
* `value` and has the proven `range`, and the lines it writes. */
function endpoint(
value: LlValue,
range: IntegerRange,
step: number,
inclusive: boolean,
): { endpoint: string; lines: string } {
const limit: IrExpr = { loc, kind: "varRef", localId: "n", type: F64 };
const B = new BlockBuilder();
const host = {
B,
integerRanges: new Map([[limit, range]]),
emitExpr: () => value,
declare: () => {},
} as unknown as LlvmEmitterContext;
const loop = { step, inclusive, limit } as unknown as IntegerCountedForLoop;
const result = emitCountedLoopLimit(host, loop);
return { endpoint: result, lines: B.render() };
}
test("a negative int32 limit whose ToUint32 view is a literal keeps its sign", () => {
// A number literal's value carries its ToUint32 bits as a literal, so -1
// has the view 4294967295; the endpoint is -1, as in `i > -1`.
const minusOne: LlValue = { name: "-1.0", type: F64, uint32: "4294967295" };
expect(endpoint(minusOne, { min: -1, max: -1 }, -1, false).endpoint).toBe("-1");
const inclusive = endpoint(minusOne, { min: -1, max: -1 }, -1, true);
expect(inclusive.lines).toContain("add nsw i64 -1, -1");
const minusTwo: LlValue = { name: "-2.0", type: F64, uint32: "4294967294" };
expect(endpoint(minusTwo, { min: -2, max: -2 }, 1, false).endpoint).toBe("-2");
// A uint32 past int32 stays positive.
const wide: LlValue = { name: "4294967295.0", type: F64, uint32: "4294967295" };
expect(endpoint(wide, { min: 4294967295, max: 4294967295 }, 1, false).endpoint).toBe(
"4294967295",
);
});
@@ -13,7 +13,10 @@ import { integerOrNaNRange } from "./wrapping-sums.js";
/** The caller has proved or guarded the direction's finite endpoint. Clamp
* the opposite infinity before integer conversion. Fractional limits become
* exclusive integer endpoints for signed comparisons in either direction. */
* exclusive integer endpoints for signed comparisons in either direction. A
* limit that is an exact integer (an int32 local, an integer view, or a
* proven range) is its own floor and ceiling and within ±2^53, so it is the
* endpoint without the double conversion and rounding. */
export function emitCountedLoopLimit(
host: LlvmEmitterContext,
loop: IntegerCountedForLoop,
@@ -35,6 +38,24 @@ export function emitCountedLoopLimit(
}
const B = host.B;
const value = host.emitExpr(loop.limit);
const exact = exactInteger(host, value, loop.limit);
if (exact) {
// An i32 literal may be the unsigned bit pattern of a negative int32
// (a ToUint32 view): read it back with the integer's signedness.
const integer = /^-?\d+$/.test(exact.name)
? String(
exact.type === "i64"
? Number(exact.name)
: exact.signed
? Number(exact.name) | 0
: Number(exact.name) >>> 0,
)
: widenInteger(host, exact);
if (!loop.inclusive) return integer;
const limit = B.tmp();
B.line(`${limit} = add nsw i64 ${integer}, ${ascending ? 1 : -1}`);
return limit;
}
const within = B.tmp(),
bounded = B.tmp(),
rounded = B.tmp(),
@@ -670,6 +670,9 @@ export function emitFunction(emitter: LlEmitter, fn: IrFunction): string {
// checked dynamic call. Keeping that fallback out of line leaves the
// tests small enough to inline at each call site.
if (fn.name.startsWith("%dyn.class.protocol.fallback.")) attrs = `noinline ${attrs}`;
// A keyed record read's inherited-method miss path (record-inherited.ts)
// stays out of line, so the hit path keeps its own small frame.
else if (fn.name.startsWith("%rec.inherited.")) attrs = `noinline cold ${attrs}`;
else if (fn.name.startsWith("%dyn.class.protocol.")) attrs = `inlinehint ${attrs}`;
if (fn.name.startsWith(RESULT_ENTRY_PREFIX)) attrs = `inlinehint ${attrs}`;
if (unoptimized && attrs === FN_ATTRS) attrs = `noinline optnone ${attrs}`;
+47 -4
View File
@@ -1,6 +1,7 @@
import type { TargetCpu } from "../target-cpu.js";
import { loopEntryInlines } from "./loop-entry-versions.js";
import type { FunctionLayout } from "./function-layout.js";
import type { CompilationTiming } from "../../timing.js";
import { relieveHeapPressure } from "../../heap-pressure.js";
import { analyzeFrameEscapes, type FrameEscapes } from "./frame-escapes.js";
import { wasmBoundsChecks } from "./wasm-modules.js";
@@ -299,8 +300,10 @@ import {
emitBytesIntrinsic,
} from "./expr-bytes.js";
import {
borrowedRecordKeyGetOr,
emitRecordKeyGet,
emitRegexIntrinsic,
immortalDefault,
keyedRecordReadInto,
pointerRecordKeyGet,
} from "./expr-records.js";
@@ -522,6 +525,11 @@ export interface LlvmTargetOptions {
forkPrimitives?: ForkPrimitives | undefined;
/** Functions to write first, by tier (function-layout.ts). */
functionLayout?: FunctionLayout;
/** Marks emission's serial steps: `llvm-emit-rewrites` after the module
* passes, `llvm-emit-int32` after int32 selection, and
* `llvm-emit-analyses` after the emitter's whole-program analyses. The
* caller's next phase is the function bodies. */
timing?: CompilationTiming;
/** Exact frontend sources for dev-build locations and variable metadata. */
debugSources?: ReadonlyMap<string, string>;
/** Pointer width of the target C ABI. Native targets are 64-bit today. */
@@ -706,6 +714,7 @@ function moduleEmitter(input: IrModule, options: LlvmTargetOptions): LlEmitter {
};
// Keep source storage intact for debugger inspection in dev builds.
if (options.debugSources !== undefined) return new LlEmitter(mod, options, null, null);
const timing = options.timing ?? (() => {});
const passed = versionParameters(
addRecordResultEntries(
scalarizeLiteralResults(
@@ -737,6 +746,7 @@ function moduleEmitter(input: IrModule, options: LlvmTargetOptions): LlEmitter {
),
),
);
timing("llvm-emit-rewrites");
let emitted = selectInt32Arrays(passed, proven, resultEntryTargets);
let prepared = rounds.last;
// A proof that revised functions analyzed the module of the functions it
@@ -758,7 +768,15 @@ function moduleEmitter(input: IrModule, options: LlvmTargetOptions): LlEmitter {
}
if (process.env["SCRIPTC_INT32_VERIFY"] === "1")
verifyInt32Selection(passed, emitted, prepared?.mod === emitted ? prepared : null);
return new LlEmitter(emitted, options, null, prepared?.mod === emitted ? prepared : null);
timing("llvm-emit-int32");
const emitter = new LlEmitter(
emitted,
options,
null,
prepared?.mod === emitted ? prepared : null,
);
timing("llvm-emit-analyses");
return emitter;
}
/** SCRIPTC_INT32_VERIFY=1: throws unless the selection with revisions
@@ -2128,7 +2146,7 @@ export class LlEmitter {
(options.targetTriple
? /-linux-(?:gnu|musl)/.test(options.targetTriple)
: process.platform === "linux")
? unwindAbiFunctions(mod, this.mayThrow, options.unwindExcluded)
? unwindAbiFunctions(mod, this.mayThrow, options.unwindExcluded, this.nodeIndex)
: new Set();
this.workerEntryPolls = mt.workerEntryPolls ?? null;
this.stackChecks = mt.stackChecks ?? null;
@@ -2221,6 +2239,7 @@ export class LlEmitter {
(type.kind !== "union" ||
this.nullableUnions.single(type.unionId)?.arm.kind !== "string"),
this.walkBorrowsByFunction.get(fn.name) ?? new Set(),
this.nodeIndex,
);
if (rebound.size === 0) continue;
this.reboundByFunction.set(fn.name, rebound);
@@ -3576,8 +3595,14 @@ export class LlEmitter {
/** Registers a SLOT whose current contents the frame owns (conditional
* results: optional chains, branch joins that park ownership). */
ownSlot(slot: string, type: IrType): void {
if (isRefCounted(type)) this.currentFrame().push({ name: slot, type, slot: true });
ownSlot(slot: string, type: IrType, rarelyFilled = false): void {
if (isRefCounted(type))
this.currentFrame().push({
name: slot,
type,
slot: true,
...(rarelyFilled ? { rarelyFilled: true } : {}),
});
}
/** The innermost lexical scope releases a frame object's reference
@@ -3689,6 +3714,17 @@ export class LlEmitter {
this.B.line(`${t} = load ptr, ptr ${v.name}`);
if (v.boxed) {
this.B.line(`call void ${boxReleaseSym(this.shapeHost)}(ptr ${t})`);
} else if (v.rarelyFilled && !this.B.isTerminated()) {
// Skip the release call on the common empty path.
const filled = this.B.tmp();
this.B.line(`${filled} = icmp ne ptr ${t}, null`);
const release = this.B.newLabel("slot.release"),
join = this.B.newLabel("slot.release.j");
this.B.condBr(filled, release, join);
this.B.startBlock(release);
this.releaseFrame(t, v.type);
this.B.br(join);
this.B.startBlock(join);
} else {
this.releaseFrame(t, v.type);
}
@@ -6298,6 +6334,11 @@ export class LlEmitter {
case "recordGet":
case "recordKeyGet":
return this.borrowsWithoutOwning(e) && this.unownedWalk(e.obj);
// `r[k] ?? inherited(r, k)` whose default is immortal.
case "nullish": {
const read = borrowedRecordKeyGetOr(this, e);
return read !== null && immortalDefault(this, e.right) && this.unownedWalk(read);
}
case "mapIntrinsic": {
// A Map binding's own pointer (maps are not plainReference types).
if (!this.borrowsWithoutOwning(e) || e.receiver.kind !== "varRef") return false;
@@ -7574,6 +7615,8 @@ export class LlEmitter {
return pointerRecordKeyGet(this, e);
case "mapIntrinsic":
return pointerMapGet(this, e);
case "nullish":
return borrowedRecordKeyGetOr(this, e) !== null;
// emitReadReceiver lends a required element of a borrowed array when
// computing the index removes no edge.
case "arrayGet":
@@ -35,6 +35,8 @@ export interface LlValue {
* convert it instead of the sum (expr-bytes.ts). */
offsetIndex?: OffsetIndex;
slot?: boolean;
/** A frame slot that usually stays NULL: its release tests it first. */
rarelyFilled?: boolean;
/** A sequence local held in a capture box; release the box itself. */
boxed?: boolean;
/** A group arena list slot (arenas.ts): the frame closes the newest
@@ -1,7 +1,14 @@
/* Focused LLVM expression emission extracted from emitter.ts. */
import { InternalCompilerError } from "../../errors.js";
import { undefinedArmTag, unionWideningTags } from "../../ir/analysis.js";
import { type IrExpr, type IrType, isRefCounted, type SrcLoc, typeEquals } from "../../ir/ir.js";
import {
type IrExpr,
type IrType,
isRefCounted,
isUnitType,
type SrcLoc,
typeEquals,
} from "../../ir/ir.js";
import { LlvmUnsupportedError } from "./unsupported.js";
import type { LlvmEmitterContext, LlValue } from "./expr-context.js";
import { f64Lit, llvmCommentText } from "./common.js";
@@ -11,6 +18,7 @@ import { canReplaceWith, regexLiteralCompiles } from "./regex-matchers.js";
import { isScalarUnionDef, loadInlineUnion, presentBits } from "./inline-unions.js";
import { preservesRegexInputs } from "./reference-effects.js";
import { emitNumberMapSlot, emitStringMapEntry } from "./map-number-lookup.js";
import { everyStmtList } from "../../ir/traverse.js";
/** Widen one tagged read. The source is borrowed unless the overflow map
* supplied an owned value. Each selected reference payload is retained
@@ -490,6 +498,7 @@ export function pointerRecordKeyGet(host: LlvmEmitterContext, e: IrExpr): boolea
export function emitBorrowedRecordKeyGet(
host: LlvmEmitterContext,
e: IrExpr & { kind: "recordKeyGet" },
fallback: IrExpr | null = null,
): LlValue {
const B = host.B;
if (e.type.kind !== "union")
@@ -499,6 +508,15 @@ export function emitBorrowedRecordKeyGet(
const key = host.canBorrowReceiver(e.key) ? host.emitReadReceiver(e.key) : host.emitExpr(e.key);
const slot = B.slot();
B.entryAllocas.push(`${slot} = alloca ptr`);
// A default that needs an owner gets a frame slot, empty on every path
// until a miss fills it.
let owner: string | null = null;
if (fallback !== null && !immortalDefault(host, fallback)) {
owner = B.slot();
B.entryAllocas.push(`${owner} = alloca ptr`);
B.line(`store ptr null, ptr ${owner}`);
host.ownSlot(owner, resultType, true);
}
const join = B.newLabel("rkb.j");
const shape = host.recordShape(e.shapeId);
for (const f of e.overflowOnly === true ? [] : shape.fields) {
@@ -528,8 +546,23 @@ export function emitBorrowedRecordKeyGet(
B.br(join);
B.startBlock(ln);
}
const undefTag = undefinedArmTag(resultType, host.unionsById);
B.line(`store ptr ${host.unitInstanceRef(resultType.unionId, undefTag)}, ptr ${slot}`);
if (fallback !== null && immortalDefault(host, fallback)) {
// An immortal default (an interned function value or undefined) needs
// no owner at all.
const value = host.emitExpr(fallback);
host.toHeap(value);
B.line(`store ptr ${value.name}, ptr ${slot}`);
} else if (fallback !== null && owner !== null) {
// `r[k] ?? d` (borrowedRecordKeyGetOr): the default is owned by a
// statement-frame slot, which the consumer may borrow like a hit.
const value = host.emitExpr(fallback);
host.toHeap(value);
B.line(`store ptr ${value.name}, ptr ${owner}`);
B.line(`store ptr ${value.name}, ptr ${slot}`);
} else {
const undefTag = undefinedArmTag(resultType, host.unionsById);
B.line(`store ptr ${host.unitInstanceRef(resultType.unionId, undefTag)}, ptr ${slot}`);
}
B.br(join);
B.startBlock(join);
const t = B.tmp();
@@ -537,6 +570,54 @@ export function emitBorrowedRecordKeyGet(
return { name: t, type: e.type };
}
/** `r[k] ?? d` in its pass-through shape over a pointerRecordKeyGet whose
* stored values are never unit (so only a miss evaluates `d`): the hit
* lends the stored pointer and the default is frame-owned, so the result
* borrows for the rest of the statement. */
export function borrowedRecordKeyGetOr(
host: LlvmEmitterContext,
e: IrExpr,
): (IrExpr & { kind: "recordKeyGet" }) | null {
if (e.kind !== "nullish" || e.left.kind !== "recordKeyGet") return null;
if (!typeEquals(e.type, e.left.type) || !pointerRecordKeyGet(host, e.left)) return null;
const shape = host.recordShape(e.left.shapeId);
const holdsUnit = (t: IrType): boolean =>
isUnitType(t) ||
t.kind === "dyn" ||
(t.kind === "union" && (host.unionsById.get(t.unionId)?.arms ?? []).some(isUnitType));
if (shape.indexValue !== undefined && holdsUnit(shape.indexValue)) return null;
if (e.left.overflowOnly !== true && shape.fields.some((f) => holdsUnit(f.type))) return null;
return e.left;
}
/** The default is a call to a helper whose every result is immortal: the
* unit arm or a closure over no captures (an interned function value), as
* the unbound inherited-method reads return. */
export function immortalDefault(host: LlvmEmitterContext, fallback: IrExpr): boolean {
if (fallback.kind !== "call") return false;
const fn = host.fnByName.get(fallback.callee);
if (!fn || fn.async || fn.generator) return false;
const immortal = (v: IrExpr | null): boolean => {
if (v === null) return false;
if (v.kind === "unitLit") return true;
if (v.kind === "unionWrap") return immortal(v.value);
if (v.kind === "closure") {
const target = host.fnByName.get(v.fnName);
return v.captures.length === 0 && target !== undefined && target.captures === undefined;
}
return false;
};
let ok = true;
everyStmtList(fn.body, {
stmt: (st) => {
if (st.kind === "return" && !immortal(st.value)) ok = false;
return ok;
},
expr: () => ok,
});
return ok;
}
/** A keyed read whose result is a scalar union (`r[k]` on a
* `Record<string, number>`, at `number | undefined`) and whose every hit
* is a number or boolean arm of it, or an inline field of the same union.
@@ -45,6 +45,15 @@ function lastUse(
const { extent, derived } = extentOf(stmts, index, localId, rebound);
const last = extent[extent.length - 1];
if (last === undefined) return null;
return lastUseIn(last, localId, derived);
}
/** The binding's read in `last`, the last statement of its extent, whose derived bindings are `derived`. */
function lastUseIn(
last: IrStmt,
localId: string,
derived: ReadonlySet<string>,
): { read: IrExpr; stored: boolean } | null {
const state: { read: IrExpr | null; stored: boolean; ok: boolean } = {
read: null,
stored: false,
@@ -183,6 +192,109 @@ function collectAssignedExpr(e: IrExpr, assigned: Set<string>): boolean {
);
}
/**
* `storedLastUseRead(stmts, -1, localId, true)` for every parameter of one function body, from one walk of the
* body instead of a walk per binding. With `index` -1 and `rebound`, no binding is excluded as assigned, so a
* binding is derived when one of its definitions mentions a derived binding (reachability over "definition
* mentions") and the extent ends at the last statement that mentions one (`absentStmt`). The walk records, for
* each top-level statement, the bindings it mentions, and for each definition, the bindings its value mentions.
*/
export class BodyLastUses {
private readonly stmts: readonly IrStmt[];
/** The bindings each top-level statement mentions, as `absentStmt` reads them. */
private readonly mentions: Set<string>[] = [];
/** For each binding, the bindings whose definitions mention it. */
private readonly definedFrom = new Map<string, string[]>();
constructor(stmts: readonly IrStmt[]) {
this.stmts = stmts;
for (const s of stmts) {
const mentioned = new Set<string>();
this.stmt(s, mentioned);
this.mentions.push(mentioned);
}
}
/** What `storedLastUseRead(stmts, -1, localId, true)` answers. */
storedRead(localId: string): IrExpr | null {
const derived = new Set<string>([localId]);
const queue = [localId];
while (queue.length > 0) {
for (const target of this.definedFrom.get(queue.pop()!) ?? [])
if (!derived.has(target)) {
derived.add(target);
queue.push(target);
}
}
let last = -1;
for (let i = this.mentions.length - 1; i >= 0 && last < 0; i--)
for (const id of derived)
if (this.mentions[i]!.has(id)) {
last = i;
break;
}
if (last < 0) return null;
const use = lastUseIn(this.stmts[last]!, localId, derived);
return use !== null && use.stored ? use.read : null;
}
/** Adds the bindings `s` mentions to `into`, and records the definitions in it. */
private stmt(s: IrStmt, into: Set<string>): boolean {
if (s.kind === "varDecl" || s.kind === "assign") into.add(s.localId);
if (s.kind === "varDecl" || s.kind === "forOf" || s.kind === "assign") {
const value = s.kind === "varDecl" ? s.init : s.kind === "assign" ? s.value : s.iterable;
if (value !== null) this.define(s.localId, value);
}
everyStmtChild(
s,
(e) => this.expr(e, into),
(c) => this.stmt(c, into),
);
return true;
}
private expr(e: IrExpr, into: Set<string>): boolean {
if (e.kind === "varRef" || e.kind === "assignExpr" || e.kind === "incDec") into.add(e.localId);
else if (e.kind === "closure" || e.kind === "classRef")
for (const id of e.captures ?? []) into.add(id);
if (e.kind === "assignExpr") this.define(e.localId, e.value);
everyExprChild(
e,
(c) => this.expr(c, into),
(c) => this.stmt(c, into),
);
return true;
}
/** A definition of `target` whose value is `value`: every binding the value mentions derives `target`. */
private define(target: string, value: IrExpr): void {
const mentioned = new Set<string>();
mentionsOf(value, mentioned);
for (const id of mentioned) {
let targets = this.definedFrom.get(id);
if (targets === undefined) this.definedFrom.set(id, (targets = []));
targets.push(target);
}
}
}
/** The bindings an expression mentions, as `absentExpr` reads them. */
function mentionsOf(e: IrExpr, into: Set<string>): void {
const expr = (x: IrExpr): boolean => {
if (x.kind === "varRef" || x.kind === "assignExpr" || x.kind === "incDec") into.add(x.localId);
else if (x.kind === "closure" || x.kind === "classRef")
for (const id of x.captures ?? []) into.add(id);
everyExprChild(x, expr, stmt);
return true;
};
const stmt = (s: IrStmt): boolean => {
if (s.kind === "varDecl" || s.kind === "assign") into.add(s.localId);
everyStmtChild(s, expr, stmt);
return true;
};
expr(e);
}
/** True while no node of the tree mentions a derived binding. */
function absentExpr(e: IrExpr, derived: ReadonlySet<string>): boolean {
if (e.kind === "varRef" || e.kind === "assignExpr" || e.kind === "incDec") {
@@ -1,5 +1,9 @@
import type { IrExpr, IrFunction, IrStmt, IrType } from "../../ir/ir.js";
import { everyExprChild, everyStmtChild } from "../../ir/traverse.js";
import { eachNodeOfKinds, type IrNodeIndex } from "../../ir/node-index.js";
/** Assignment statements: only these rebind a parameter. */
const ASSIGN_KINDS = ["assign"] as const;
/** Parameters that keep the borrowed calling convention although the body
* rebinds them (`source = (source as LiteralType).regularType`). The
@@ -15,6 +19,8 @@ export function findReboundParameters(
fn: IrFunction,
plainReference: (type: IrType) => boolean,
excluded: ReadonlySet<string>,
/** The module's node index: a body that assigns no candidate parameter is not walked. */
nodes?: IrNodeIndex,
): ReadonlySet<string> {
const result = new Set<string>();
if (fn.async || fn.generator || fn.captures || fn.classCaptures) return result;
@@ -34,6 +40,15 @@ export function findReboundParameters(
.map((param) => param.localId),
);
if (candidates.size === 0) return result;
// A parameter is rebound only by an assignment statement to it.
if (nodes !== undefined) {
let assigned = false;
eachNodeOfKinds(fn, nodes.scan(ASSIGN_KINDS), (node) => {
if (node.kind === "assign" && candidates.has(node.localId)) assigned = true;
return !assigned;
});
if (!assigned) return result;
}
const rebound = new Set<string>();
const invalid = new Set<string>();
const expr = (e: IrExpr): boolean => {
@@ -24,6 +24,7 @@ import { emitDowncastGuard, emitUnionNarrowGuard } from "./narrow-guards.js";
import { emitScalarOptional, isScalarOptionalSource } from "./local-union-storage.js";
import { borrowableInputs } from "./borrowed-inputs.js";
import {
borrowedRecordKeyGetOr,
emitBorrowedRecordKeyGet,
emitStackRecordKeyGet,
pointerRecordKeyGet,
@@ -104,6 +105,10 @@ export function emitReadReceiver(emitter: LlEmitter, e: IrExpr): LlValue {
if (arrayRead) return emitProjectedArrayRead(emitter, arrayRead);
if (e.kind === "recordKeyGet" && emitter.canBorrowReceiver(e))
return emitBorrowedRecordKeyGet(emitter, e);
if (e.kind === "nullish" && emitter.canBorrowReceiver(e)) {
const read = borrowedRecordKeyGetOr(emitter, e);
if (read) return emitBorrowedRecordKeyGet(emitter, read, e.right);
}
if (e.kind === "mapIntrinsic" && emitter.canBorrowReceiver(e))
return emitBorrowedMapGet(emitter, e);
const narrow = emitter.checkedNarrows.get(e);
@@ -160,6 +165,12 @@ export function emitReadReceiver(emitter: LlEmitter, e: IrExpr): LlValue {
no = B.newLabel("read.else"),
join = B.newLabel("read.join");
B.condBr(cond.name, yes, no);
// Slots an arm's borrowed read fills (the owner of a `r[k] ?? d`
// default) may hold what the result points to: the enclosing frame
// releases them after the use, and the other arm's path leaves them
// empty so that release skips them.
const armSlots: LlValue[][] = [];
const ends: string[] = [];
for (const [label, value] of [
[yes, e.then],
[no, e.else_],
@@ -168,9 +179,21 @@ export function emitReadReceiver(emitter: LlEmitter, e: IrExpr): LlValue {
emitter.frames.push([]);
const result = emitter.emitReadReceiver(value);
B.line(`store ptr ${result.name}, ptr ${slot}`);
emitter.releaseTemps(emitter.frames.pop()!);
const frame = emitter.frames.pop()!;
armSlots.push(frame.filter((entry) => entry.slot === true && entry.boxed !== true));
emitter.releaseTemps(frame.filter((entry) => entry.slot !== true || entry.boxed === true));
const end = B.newLabel("read.end");
ends.push(end);
B.br(end);
}
for (const [i, end] of ends.entries()) {
B.startBlock(end);
for (const entry of armSlots[1 - i]!) B.line(`store ptr null, ptr ${entry.name}`);
B.br(join);
}
for (const entries of armSlots)
for (const entry of entries)
emitter.ownSlot(entry.name, entry.type, entry.rarelyFilled === true);
B.startBlock(join);
const value = B.tmp();
B.line(`${value} = load ptr, ptr ${slot}`);
@@ -315,6 +338,12 @@ export function canBorrowReceiver(emitter: LlEmitter, e: IrExpr): boolean {
emitter.canBorrowReceiver(e.obj) &&
emitter.referenceEffects.preserves(e.key)
);
// `r[k] ?? inherited(r, k)` lends a hit the same way; its default is
// owned by the statement frame (so never by a longer-lived alias).
case "nullish": {
const read = borrowedRecordKeyGetOr(emitter, e);
return read !== null && emitter.canBorrowReceiver(read);
}
// So does a Map.get while the map keeps it.
case "mapIntrinsic":
return (
@@ -930,6 +930,10 @@ export class ReferenceWrites {
out.push(ANY_RECORD_KEY);
for (const f of this.records.get(node.shapeId)?.fields ?? []) out.push(`r:${f.name}`);
return this.expr(node.key) === null && walk(node.obj);
case "nullish":
// `r[k] ?? d`: a hit is the read's own value; the default runs only
// on a miss and is owned apart from any location.
return walk(node.left);
case "mapIntrinsic":
// A typed Map.get: any entry of any map.
if (node.method !== "get" || node.args.length !== 1 || !borrowsMapReadInputs(node))
@@ -1,8 +1,14 @@
/* The functions that propagate exceptions by unwinding instead of the
* pending flag (see unwindAbiFunctions). */
import type { IrExpr, IrModule } from "../../ir/ir.js";
import { everyStmtList } from "../../ir/traverse.js";
import type { IrExpr, IrModule, IrStmt } from "../../ir/ir.js";
import { eachStringField } from "../../ir/fields.generated.js";
import { eachNodeOfKinds, type IrNodeIndex } from "../../ir/node-index.js";
import { InternalCompilerError } from "../../errors.js";
/** The kinds of every recorded node, for eachNodeOfKinds over whole bodies. */
const EVERY_KIND: ReadonlySet<string> = new Set();
/** Virtual calls, which name the methods a vtable must reach. */
const VIRTUAL_CALL_KINDS = ["virtualCall"] as const;
/** How many calls above a `throw` functions keep the pending flag. */
const THROW_LEVELS = 3;
@@ -80,6 +86,8 @@ export function unwindAbiFunctions(
mod: IrModule,
mayThrow: ReadonlySet<string>,
excluded: ReadonlySet<string> = new Set(),
/** The module's node index: the bodies' nodes are read from it, in walk order, without walking. */
nodes?: IrNodeIndex,
): Set<string> {
const names = new Set(mod.functions.map((fn) => fn.name));
const blocked = new Set<string>(excluded);
@@ -97,16 +105,16 @@ export function unwindAbiFunctions(
// class's method of that name.
const classes = new Map((mod.classes ?? []).map((cls) => [cls.name, cls]));
const anyClass = new Set<string>();
const virtualCalls = nodes?.scan(VIRTUAL_CALL_KINDS);
for (const fn of mod.functions)
everyStmtList(fn.body, {
expr: (e: IrExpr) => {
if (e.kind !== "virtualCall") return true;
if (e.targets !== undefined) for (const target of e.targets) blocked.add(target);
else if (classes.has(e.className)) blocked.add(`%${e.className}.${e.method}`);
else anyClass.add(e.method);
return true;
},
stmt: () => true,
// Without the index (or for a function it didn't record) every node
// arrives, statements included: the kind test narrows, not a cast.
eachNodeOfKinds(fn, virtualCalls, (node) => {
if (node.kind !== "virtualCall") return true;
if (node.targets !== undefined) for (const target of node.targets) blocked.add(target);
else if (classes.has(node.className)) blocked.add(`%${node.className}.${node.method}`);
else anyClass.add(node.method);
return true;
});
// A hierarchy's vtables hold the methods declared by more than one of its
// classes (an override and what it overrides).
@@ -161,9 +169,10 @@ export function unwindAbiFunctions(
if (list === undefined) methodsNamed.set(m, (list = []));
list.push(`%${cls.name}.${m}`);
}
const everyNode = nodes?.scanExcept(EVERY_KIND);
for (const fn of mod.functions) {
const coroutine = fn.async === true || fn.generator !== undefined;
everyStmtList(fn.body, {
const visit = {
expr: (e: IrExpr) => {
const kind = e.kind as string;
if (kind === "throw" || kind === "rethrow") throws.add(fn.name);
@@ -192,13 +201,16 @@ export function unwindAbiFunctions(
}
return true;
},
stmt: (s) => {
stmt: (s: IrStmt) => {
const kind = s.kind as string;
if (kind === "throw" || kind === "rethrow") throws.add(fn.name);
eachStringField(s, noteName);
return true;
},
});
};
eachNodeOfKinds(fn, everyNode, (node, statement) =>
statement ? visit.stmt(node as IrStmt) : visit.expr(node as IrExpr),
);
}
for (const name of referenced) {
blocked.add(name);
@@ -228,6 +240,13 @@ export function unwindAbiFunctions(
if (fn.async === true || fn.generator !== undefined) continue;
result.add(fn.name);
}
if (nodes !== undefined && process.env["SCRIPTC_EMIT_FACTS_VERIFY"] === "1") {
const walked = unwindAbiFunctions(mod, mayThrow, excluded);
if (walked.size !== result.size || [...walked].some((name) => !result.has(name)))
throw new InternalCompilerError(
"unwind ABI: the functions read from the node index differ from a walk's",
);
}
return result;
}
@@ -14,20 +14,32 @@ const PROGRAM_PARTITION_BYTES = 4 * 1024 * 1024;
const MIN_PARTITION_BYTES = 512 * 1024;
const MIN_PROGRAM_PARTITIONS = 8;
const MAX_PROGRAM_PARTITIONS = 64;
/** SINGLE_PARTITION_BYTES and MIN_PARTITION_BYTES of unoptimized (dev)
* executables. */
const UNOPTIMIZED_SINGLE_PARTITION_BYTES = 64 * 1024;
const UNOPTIMIZED_MIN_PARTITION_BYTES = 32 * 1024;
/** Executable objects larger than SINGLE_PARTITION_BYTES are divided into
* partitions that the helper simplifies, optimizes and compiles concurrently,
* importing small functions across partitions for inlining, and reuses
* individually across builds. A divided module gets at least
* MIN_PROGRAM_PARTITIONS so that mid-size programs use several cores. The
* count is a power of two that depends only on the module's size, on every
* target, so artifacts never vary with the host and ordinary edits keep every
* function in its partition. */
export function nativeProgramPartitions(llvmBytes: number): number {
if (llvmBytes <= SINGLE_PARTITION_BYTES) return 1;
* count is a power of two that depends only on the module's size and
* whether it is optimized, on every target, so artifacts never vary with the
* host and ordinary edits keep every function in its partition.
*
* Unoptimized (dev) partitions are compiled on their own with nothing
* imported, so they cost no run time and an edit compiles only the partitions
* whose functions it changes: dev executables are divided from
* UNOPTIMIZED_SINGLE_PARTITION_BYTES, at least
* UNOPTIMIZED_MIN_PARTITION_BYTES each. */
export function nativeProgramPartitions(llvmBytes: number, unoptimized = false): number {
const single = unoptimized ? UNOPTIMIZED_SINGLE_PARTITION_BYTES : SINGLE_PARTITION_BYTES;
const least = unoptimized ? UNOPTIMIZED_MIN_PARTITION_BYTES : MIN_PARTITION_BYTES;
if (llvmBytes <= single) return 1;
const required = Math.max(
Math.ceil(llvmBytes / PROGRAM_PARTITION_BYTES),
Math.min(MIN_PROGRAM_PARTITIONS, Math.ceil(llvmBytes / MIN_PARTITION_BYTES)),
Math.min(MIN_PROGRAM_PARTITIONS, Math.ceil(llvmBytes / least)),
);
return Math.min(MAX_PROGRAM_PARTITIONS, 2 ** Math.ceil(Math.log2(required)));
}
@@ -408,6 +408,14 @@ test("program partitions depend only on module size", () => {
expect(nativeProgramPartitions(32 * megabyte + 1)).toBe(16);
expect(nativeProgramPartitions(100 * megabyte)).toBe(32);
expect(nativeProgramPartitions(1024 * megabyte)).toBe(64);
// Unoptimized (dev) programs are divided from 64 KiB.
const kilobyte = 1024;
expect(nativeProgramPartitions(64 * kilobyte, true)).toBe(1);
expect(nativeProgramPartitions(64 * kilobyte + 1, true)).toBe(4);
expect(nativeProgramPartitions(128 * kilobyte, true)).toBe(4);
expect(nativeProgramPartitions(128 * kilobyte + 1, true)).toBe(8);
expect(nativeProgramPartitions(megabyte, true)).toBe(8);
expect(nativeProgramPartitions(32 * megabyte + 1, true)).toBe(16);
expect(nativePartitionPaths("/build.d/program.o", 3)).toEqual([
"/build.d/program.o",
"/build.d/program.part1.o",
@@ -21,6 +21,13 @@ export const WEB_STREAM_CLASSES = [
"ReadableStreamDefaultController",
"CountQueuingStrategy",
"ByteLengthQueuingStrategy",
"WritableStream",
"WritableStreamDefaultWriter",
"WritableStreamDefaultController",
"TransformStream",
"TransformStreamDefaultController",
"TextEncoderStream",
"TextDecoderStream",
] as const;
const CLASS_NAMES: readonly string[] = WEB_STREAM_CLASSES;
@@ -48,6 +55,33 @@ export function webStreamClassKindOf(lowerer: Lowerer, node: ts.Expression): num
return index < 0 ? null : index;
}
/** Whether a JavaScript source of the program names one of the classes.
* The global object answers them by a computed name only then
* (global-values.ts), so a program without web streams does not link
* them for its other computed global reads. */
export function programNamesWebStreamClass(lowerer: Lowerer): boolean {
if (lowerer.namesWebStreamClass !== undefined) return lowerer.namesWebStreamClass;
const names: ReadonlySet<string> = new Set(WEB_STREAM_CLASSES);
let found = false;
for (const file of lowerer.program.getSourceFiles()) {
if (file.isDeclarationFile || !isJsSourceFile(file)) continue;
ts.walkPreorder(file, (node) => {
if (
ts.isIdentifier(node) &&
names.has(node.text) &&
webStreamClassKindOf(lowerer, node) !== null
) {
found = true;
return "stop";
}
return undefined;
});
if (found) break;
}
lowerer.namesWebStreamClass = found;
return found;
}
export function webStreamClassValue(kind: number, loc: SrcLoc): IrExpr {
return { kind: "libCall", fn: "webstream.class", args: [numLit(kind, loc)], type: DYN, loc };
}
@@ -130,6 +164,8 @@ const WEB_STREAM_INTERFACES: ReadonlySet<string> = new Set([
"ReadableStreamReadValueResult",
"ReadableStreamReadDoneResult",
"UnderlyingSource",
"UnderlyingSink",
"Transformer",
"QueuingStrategy",
"QueuingStrategySize",
]);
@@ -32,6 +32,11 @@ import { builtinNamespaceValue } from "./builtin-namespaces.js";
import { uriFunctionValue } from "./expressions/identifiers.js";
import { navigatorValue } from "./builtins/navigator.js";
import { primitiveCtorClosure } from "./lower-exprs.js";
import {
WEB_STREAM_CLASSES,
programNamesWebStreamClass,
webStreamClassValue,
} from "./builtins/web-streams-dyn.js";
const LOOKUP_FN = "%global.builtinValue";
@@ -177,9 +182,14 @@ export function globalValuesInstall(
const name: IrExpr = { kind: "varRef", localId: "name.0", type: STRING, loc };
const body: IrStmt[] = [];
for (const global of [...new Set(names)].sort()) {
const webStream = (WEB_STREAM_CLASSES as readonly string[]).indexOf(global);
const value: IrExpr | null = REFUSED.has(global)
? { kind: "unitLit", unit: "null", type: { kind: "nullT" }, loc }
: jsGlobalValueByName(lowerer, global, loc);
: webStream >= 0
? programNamesWebStreamClass(lowerer)
? webStreamClassValue(webStream, loc)
: null
: jsGlobalValueByName(lowerer, global, loc);
if (value === null) continue;
body.push({
kind: "if",
@@ -3659,6 +3659,9 @@ export class Lowerer {
/** global-values.ts's answer per JavaScript source: the receiver of its
* first read of the global object by a computed name, or null. */
readonly computedGlobalReads = new Map<ts.SourceFile, ts.Expression | null>();
/** web-streams-dyn.ts's answer: whether a JavaScript source names a web
* stream class; undefined until asked. */
namesWebStreamClass: boolean | undefined = undefined;
/** The first lifted function of each name (liftedFnNamed), for the
* liftedFns indexed so far. liftedFns only grows. */
private readonly liftedFnsByName = new Map<string, IrFunction>();
@@ -4275,6 +4278,7 @@ export class Lowerer {
"createRequireBindingTargets",
"createRequireElementTargets",
"computedGlobalReads",
"namesWebStreamClass",
"timeValueDateDecisions",
"timeValueDateWrites",
"dynamicOverloadAnswers",
@@ -499,7 +499,8 @@ function builtinAnswer(
type: BOOL,
loc,
});
if (arg === undefined || !mayBeSymbol(arg.type)) return has(propertyKeyOf(lowerer, arg, loc));
if (arg === undefined || !mayBeSymbol(lowerer, arg.type))
return has(propertyKeyOf(lowerer, arg, loc));
const id = declare("%protoKey", DYN);
const held = varRef(id, DYN, loc);
return {
@@ -520,13 +521,16 @@ function builtinAnswer(
}
}
/** Whether a value of `type` may be a symbol at run time. */
function mayBeSymbol(type: IrType): boolean {
return !(
isUnitType(type) ||
type.kind === "string" ||
type.kind === "f64" ||
type.kind === "bool"
/** Whether a value of `type` may be a symbol at run time: a checked value
* or a union with a symbol or checked arm. Records, class instances,
* arrays and functions never are, so their keys skip the symbol test (a
* key that converts without running code leaves the method's impl no heap
* writer to ReferenceWrites). */
function mayBeSymbol(lowerer: Lowerer, type: IrType): boolean {
if (type.kind === "dyn" || type.kind === "symbol") return true;
if (type.kind !== "union") return false;
return (lowerer.unions.get(type.unionId)?.arms ?? []).some(
(arm) => arm.kind === "dyn" || arm.kind === "symbol",
);
}
@@ -540,9 +544,29 @@ function propertyKeyOf(lowerer: Lowerer, arg: IrExpr | undefined, loc: SrcLoc):
type: STRING,
loc,
};
if (arg.type.kind === "string") return arg;
if (arg.type.kind === "f64" || arg.type.kind === "bool")
const type = arg.type;
if (type.kind === "string") return arg;
if (type.kind === "f64" || type.kind === "bool")
return { kind: "toString", operand: arg, type: STRING, loc };
// Unions of scalars convert through the union's own ToString.
if (
type.kind === "union" &&
(lowerer.unions.get(type.unionId)?.arms ?? []).every(
(arm) =>
isUnitType(arm) || arm.kind === "string" || arm.kind === "f64" || arm.kind === "bool",
)
)
return { kind: "toString", operand: arg, type: STRING, loc };
// A record or class instance that answers Object.prototype.toString
// spells "[object Object]" without running any code.
if (
(type.kind === "record" && defaultRecordToString(lowerer, type.shapeId)) ||
(type.kind === "object" && defaultToString(lowerer, type.className))
)
return { kind: "strLit", value: "[object Object]", type: STRING, loc };
// Any other key (an array, a function, a record or class instance with
// its own toString, a checked value) converts as ToPropertyKey does,
// through the dynamic tier: the conversion may run the key's code.
return {
kind: "libCall",
fn: "dyn.toStringCoerce",
@@ -552,6 +576,29 @@ function propertyKeyOf(lowerer: Lowerer, arg: IrExpr | undefined, loc: SrcLoc):
};
}
/** A record shape whose values answer Object.prototype.toString: no
* `toString` field, and no index signature that may hold one at run time
* (or give the record a null prototype). */
function defaultRecordToString(lowerer: Lowerer, shapeId: string): boolean {
const shape = lowerer.shapes.get(shapeId);
return (
shape !== undefined &&
shape.indexValue === undefined &&
!shape.fields.some((field) => field.name === "toString")
);
}
/** A source class whose instances answer Object.prototype.toString. */
function defaultToString(lowerer: Lowerer, className: string): boolean {
const info = lowerer.classes.get(className);
return (
info !== undefined &&
info.decl !== undefined &&
lowerer.findMethodOn(info, "toString") === null &&
!lowerer.overrideBelow(info, "toString")
);
}
/** `value` returned as `ret`: as is, wrapped into a union arm or a checked
* value, discarded for void, or the checked-conversion TypeError. */
function returnAs(lowerer: Lowerer, value: IrExpr, ret: IrType, loc: SrcLoc): IrStmt[] {
@@ -147,20 +147,27 @@ test("a map table hands over the entries added after a size", () => {
expect([...map]).toEqual([["a", "%h.1"]]);
});
test("a held table notices a replaced entry", () => {
test("a table's content snapshot notices changed, reordered and removed entries", () => {
const map = new Map<string, string>([["a", "%h.1"]]);
const held = mapTable("helpers", map).hold!();
const held = mapTable("helpers", map).contents!();
map.set("b", "%h.2");
expect(held.same()).toBe(true);
expect(held.changed()).toBeNull();
map.set("a", "%h.9");
expect(held.same()).toBe(false);
const set = new Set(["x"]);
const heldSet = setTable("names", set).hold!();
expect(held.changed()).toBe("a");
// A unit names the helper it got, not the key it was looked up by.
expect(held.changedAmong(new Set(["%h.1"]))).toBe("a");
expect(held.changedAmong(new Set(["%h.2"]))).toBeNull();
held.refresh(new Set(["%h.1"]));
expect(held.changed()).toBeNull();
const set = new Set(["x", "y"]);
const heldSet = setTable("names", set).contents!();
set.delete("x");
set.add("x");
expect(heldSet.same()).toBe(true);
expect(heldSet.changed()).toBe("x");
set.delete("x");
expect(heldSet.same()).toBe(false);
expect(heldSet.changedAmong(new Set(["x"]))).toBe("x");
heldSet.refresh(new Set(["x"]));
expect(heldSet.changed()).toBeNull();
});
test("a function table keeps the function an earlier unit made", () => {
@@ -280,7 +287,7 @@ test("a unit that changes an entry it did not add has no record", () => {
state.helpers.set("shared", "%h.first");
const unit = recordUnit(subject, "a");
expect(unit.record).toBeNull();
expect(unit.reasons).toEqual(["helpers:replaced"]);
expect(unit.reasons).toEqual(["helpers:changed"]);
});
test("a recorded wave throws when a unit changes a held function in place", () => {
@@ -96,8 +96,6 @@ export interface JournalTable {
readonly digest?: (key: string, before: number) => string | null;
/** Removes the entries from position `from` on. */
readonly truncate?: (from: number) => void;
/** The entries the table holds now. */
readonly hold?: () => HeldEntries;
/** The content of the entries the table holds now, for tables whose
* values are objects a unit could change in place. */
readonly contents?: () => HeldContents;
@@ -127,10 +125,13 @@ export function contentDigest(text: string): string {
return `${text.length}:${(a >>> 0).toString(16)}${(b >>> 0).toString(16)}`;
}
/** The strings quoted in JSON `texts`, unescaped only as far as their
* delimiting quotes: the names a unit's encoded IR and entries refer to. */
/** The names quoted in JSON `texts`, unescaped only as far as their
* delimiting quotes: the strings that start with `%` (helper, lifted and
* IR data names) and union ids (`u` and 12 hex digits), which the tables
* a unit reads and changes are keyed by. */
export function quotedStrings(texts: readonly string[]): Set<string> {
const out = new Set<string>();
const hex = (char: number): boolean => (char >= 48 && char <= 57) || (char >= 97 && char <= 102);
for (const text of texts) {
let start = -1;
for (let index = 0; index < text.length; index++) {
@@ -139,7 +140,13 @@ export function quotedStrings(texts: readonly string[]): Set<string> {
else if (char === 34 /* " */) {
if (start < 0) start = index + 1;
else {
out.add(text.slice(start, index));
const first = text.charCodeAt(start);
let name = first === 37; /* % */
if (!name && first === 117 /* u */ && index - start === 13) {
name = true;
for (let at = start + 1; at < index && name; at++) name = hex(text.charCodeAt(at));
}
if (name) out.add(text.slice(start, index));
start = -1;
}
}
@@ -148,12 +155,6 @@ export function quotedStrings(texts: readonly string[]): Set<string> {
return out;
}
/** A table's entries at one point (JournalTable.hold). */
export interface HeldEntries {
/** Whether the table's first entries are still these keys and values. */
same(): boolean;
}
/** A field a unit's lowering must leave as it found it (fixedScalar), or a
* flag it may set, which applying its record sets too (flagScalar). */
export interface JournalScalar {
@@ -288,7 +289,11 @@ export function recordUnit(
): RecordedUnit {
const tables = host.tables;
const before = tables.map((table) => table.size());
const held = tables.map((table) => (table.hold === undefined ? null : table.hold()));
// Outside a recorded wave, the unit's own snapshot of the tables' content.
const own =
contents === null
? tables.map((table) => (table.contents === undefined ? null : table.contents()))
: null;
const marks = host.scalars.map((scalar) => scalar.mark());
const counts = host.statementCounts();
const recorder = new UnitEvents();
@@ -296,8 +301,6 @@ export function recordUnit(
const reasons: string[] = [];
const grown: number[] = [];
tables.forEach((table, index) => {
const check = held[index] ?? null;
if (check !== null && !check.same()) reasons.push(`${table.name}:replaced`);
if (table.size() === before[index]) return;
if (table.entries === undefined) reasons.push(table.name);
else grown.push(index);
@@ -339,7 +342,12 @@ export function recordUnit(
if (entries !== null) for (const entry of entries) texts.push(entry);
}
const names = quotedStrings(texts);
if (contents !== null) {
if (own !== null) {
if (reasons.length === 0)
tables.forEach((table, index) => {
if ((own[index] ?? null)?.changed() != null) reasons.push(`${table.name}:changed`);
});
} else if (contents !== null) {
// A unit with a record must leave the entries it names as they were; a
// unit without one keeps its changes.
if (reasons.length === 0)
@@ -419,6 +427,69 @@ export function lowerWaveRecorded(
});
}
/** The content of a keyed table's entries (JournalTable.contents): each
* entry's encoded value, and the order of the keys, since a table only adds
* entries at the end. An entry is found by its key, or by the name its value
* holds (`valueName`): a unit names the helper it got, not the key it was
* looked up by. `text` answers undefined for a key the table does not
* hold. */
function keyedContents(
size: () => number,
keys: () => Iterable<string>,
text: (key: string) => string | null | undefined,
valueName: (key: string) => string | null,
): HeldContents {
const texts = new Map<string, string | null>();
let order: string[] = [];
const byName = new Map<string, string>();
const differs = (key: string): boolean => text(key) !== texts.get(key);
const keyOf = (name: string): string | undefined => (texts.has(name) ? name : byName.get(name));
const held: HeldContents = {
extend: () => {
if (texts.size >= size()) return;
for (const key of keys()) {
if (texts.has(key)) continue;
texts.set(key, text(key) ?? null);
order.push(key);
const named = valueName(key);
if (named !== null) byName.set(named, key);
}
},
changed: () => {
let index = 0;
for (const key of keys()) {
if (index >= order.length) break;
if (key !== order[index]) return order[index]!;
index++;
}
if (index < order.length) return order[index]!;
for (const key of order) if (differs(key)) return key;
return null;
},
changedAmong: (names) => {
for (const name of names) {
const key = keyOf(name);
if (key !== undefined && differs(key)) return key;
}
return null;
},
refresh: (names) => {
for (const name of names) {
const key = keyOf(name);
if (key === undefined) continue;
const now = text(key);
if (now !== undefined) texts.set(key, now);
else {
texts.delete(key);
order = order.filter((held) => held !== key);
}
}
},
};
held.extend();
return held;
}
/** An appendable table over a Map from strings to strings, such as helper
* names by key. */
export function mapTable(name: string, map: Map<string, string>): JournalTable {
@@ -427,6 +498,7 @@ export function mapTable(name: string, map: Map<string, string>): JournalTable {
map,
(value) => JSON.stringify(value),
(text) => JSON.parse(text) as string,
(value) => value,
);
}
@@ -444,42 +516,15 @@ export function countTable(name: string, map: Map<string, number>): JournalTable
* JSON. JSON writes a number that is not finite as null, so a value whose
* encoding holds a null does not encode. */
export function irDataTable<V>(name: string, map: Map<string, V>): JournalTable {
const encode = (value: V): string | null => {
const text = JSON.stringify(value);
return /[:,[]null\b/.test(text) ? null : text;
};
return {
...codecTable(name, map, encode, (text) => JSON.parse(text) as V),
contents: () => {
const texts = new Map<string, string | null>();
const differs = (key: string): boolean => {
const value = map.get(key);
return value === undefined || encode(value) !== texts.get(key);
};
const held: HeldContents = {
extend: () => {
if (texts.size === map.size) return;
for (const [key, value] of map) if (!texts.has(key)) texts.set(key, encode(value));
},
changed: () => {
for (const key of texts.keys()) if (differs(key)) return key;
return null;
},
changedAmong: (keys) => {
for (const key of keys) if (texts.has(key) && differs(key)) return key;
return null;
},
refresh: (keys) => {
for (const key of keys) {
const value = map.get(key);
if (texts.has(key) && value !== undefined) texts.set(key, encode(value));
}
},
};
held.extend();
return held;
return codecTable(
name,
map,
(value) => {
const text = JSON.stringify(value);
return /[:,[]null\b/.test(text) ? null : text;
},
};
(text) => JSON.parse(text) as V,
);
}
/** An appendable table over a Map from strings to values that `encode`
@@ -491,6 +536,7 @@ export function codecTable<V>(
map: Map<string, V>,
encode: (value: V) => string | null,
decode: (text: string) => V,
nameOf: ((value: V) => string) | null = null,
): JournalTable {
return {
name,
@@ -517,21 +563,19 @@ export function codecTable<V>(
const keys = [...map.keys()].slice(from);
for (const key of keys) map.delete(key);
},
hold: () => {
const keys = [...map.keys()];
const values = [...map.values()];
return {
same: () => {
let index = 0;
for (const [key, value] of map) {
if (index >= keys.length) break;
if (key !== keys[index] || value !== values[index]) return false;
index++;
}
return index === keys.length;
contents: () =>
keyedContents(
() => map.size,
() => map.keys(),
(key) => {
const value = map.get(key);
return value === undefined ? undefined : encode(value);
},
};
},
(key) => {
const value = map.get(key);
return value === undefined || nameOf === null ? null : nameOf(value);
},
),
};
}
@@ -548,20 +592,13 @@ export function setTable(name: string, set: Set<string>): JournalTable {
const entries = [...set].slice(from);
for (const entry of entries) set.delete(entry);
},
hold: () => {
const entries = [...set];
return {
same: () => {
let index = 0;
for (const entry of set) {
if (index >= entries.length) break;
if (entry !== entries[index]) return false;
index++;
}
return index === entries.length;
},
};
},
contents: () =>
keyedContents(
() => set.size,
() => set.keys(),
(key) => (set.has(key) ? "" : undefined),
() => null,
),
};
}
@@ -626,10 +663,6 @@ export function functionTable(
while (functions.length > from) functions.pop();
changed();
},
hold: () => {
const held = [...functions];
return { same: () => held.every((fn, index) => functions[index] === fn) };
},
contents: () => {
const texts: (string | null)[] = [];
const indexes = new Map<string, number>();
+9 -10
View File
@@ -323,16 +323,16 @@ export class ShapeRegistry {
* writer after all priority candidates have closed. */
private readonly derivedDeclaredOrderFinalizers: (() => void)[] = [];
private currentDeclaredOrderPriority: DeclaredOrderPriority = [0, 0];
/** The declared-order candidates and the attached names the registry
* holds, counted as they are added (journalSizes). */
private candidateCount = 0;
private extendedCount = 0;
/** All interned shapes in first-seen order. */
readonly shapes: IrRecordShape[] = [];
/** The sizes of the registry's tables, which only grow while shapes are
* interned (unit-journal.ts). */
journalSizes(): number[] {
let candidates = 0;
for (const list of this.declaredOrderCandidates.values()) candidates += list.length;
let extended = 0;
for (const names of this.extendedNames.values()) extended += names.length;
return [
this.revision,
this.shapes.length,
@@ -341,12 +341,12 @@ export class ShapeRegistry {
this.keyedNameKeys.size,
this.declaredOrderPriority.size,
this.declaredOrderCandidates.size,
candidates,
this.candidateCount,
this.derivedDeclaredOrderFinalizers.length,
this.recIds.size,
this.pendingRec.size,
this.extendedNames.size,
extended,
this.extendedCount,
];
}
/** ts.Types currently being mapped — a BACK-REFERENCE to one of these is
@@ -401,6 +401,7 @@ export class ShapeRegistry {
private adoptDeclaredOrder(shape: IrRecordShape, declaredOrder: string[] | undefined): void {
if (declaredOrder === undefined) return;
this.candidateCount++;
const candidates = this.declaredOrderCandidates.get(shape.id);
const candidate = { order: declaredOrder, priority: this.currentDeclaredOrderPriority };
if (candidates) candidates.push(candidate);
@@ -556,6 +557,7 @@ export class ShapeRegistry {
this.byId.set(id, shape);
this.shapes.push(shape);
if (declaredOrder !== undefined) {
this.candidateCount++;
this.declaredOrderCandidates.set(id, [
{ order: declaredOrder, priority: this.currentDeclaredOrderPriority },
]);
@@ -618,6 +620,7 @@ export class ShapeRegistry {
shape.fields = [...shape.fields, ...fresh].sort((a, b) =>
a.name < b.name ? -1 : a.name > b.name ? 1 : 0,
);
this.extendedCount += fresh.length;
this.extendedNames.set(shapeId, [
...(this.extendedNames.get(shapeId) ?? []),
...fresh.map((f) => f.name),
@@ -745,10 +748,6 @@ export class UnionRegistry {
this.keyedNameKeys.delete(def.id);
}
},
hold: () => {
const held = [...this.unions];
return { same: () => held.every((def, index) => this.unions[index] === def) };
},
contents: () => {
// A recursive placeholder still pending fills in place when its
// outer frame completes, which changes journalSizes.
+5 -1
View File
@@ -789,7 +789,9 @@ async function emitNativeProgramObject(
// An optimized program that stays one partition is optimized whole and
// generates code for its parts concurrently.
const codegenParts = single || opts.optimization === "dev" ? 1 : nativeCodegenParts(llvmBytes);
const partitions = single ? 1 : Math.max(nativeProgramPartitions(llvmBytes), codegenParts);
const partitions = single
? 1
: Math.max(nativeProgramPartitions(llvmBytes, opts.optimization === "dev"), codegenParts);
try {
const artifact = await emitNativeArtifact({
outputPath: linkPath,
@@ -1079,6 +1081,7 @@ async function prepareExecutableInput(
objectAudit: opts.sanitize === true,
sanitizeAttributes: opts.sanitize === true,
lowRegion: lowRegionTarget(buildPlatform, opts.sanitize === true),
timing,
...(opts.sanitize === true ? { wasmBoundsChecks: true } : {}),
...inlineRcOption(opts.optimization, opts.sanitize === true),
...(opts.detachedExit === true ? { detachedExit: true } : {}),
@@ -2600,6 +2603,7 @@ async function compileLibraryTracked(
wasi: buildPlatform === "wasi",
objectAudit: opts.sanitize === true,
sanitizeAttributes: opts.sanitize === true,
timing,
...inlineRcOption(profile.optimization, opts.sanitize === true),
});
timing("llvm-emit", {
+3 -2
View File
@@ -1044,8 +1044,9 @@ const RUNTIME_FEATURE_RULES = {
mime: rule({ prefixes: ["mime."] }),
/** Web streams in checked-dynamic code (scr_webstreams.c). */
webStreams: rule({ prefixes: ["webstream."] }),
/** AbortController and AbortSignal (scr_abort.c, or the fetch unit). */
abortSignal: rule({ prefixes: ["abort."] }),
/** AbortController and AbortSignal (scr_abort.c, or the fetch unit).
* A WritableStream controller's signal is one (scr_webstreams.c). */
abortSignal: rule({ prefixes: ["abort.", "webstream."] }),
/** util.debounce and util.throttle (scr_util_timing.c). */
utilTiming: rule({ fns: ["util.debounce", "util.throttle"] }),
/** events.once(emitter, name, { signal }) (scr_events_once_signal.c). */
+13 -1
View File
@@ -75,6 +75,11 @@ import { splitLlvmLibraryProgram } from "../backend/llvm/split.js";
import { prepareNativeExecutable, prepareNativeLibrary } from "./prepare.js";
import { buildSanitizedRuntime, sanitizerDriver, sanitizerFlags } from "./sanitizer.js";
import { inputsWithoutLld, lldLinkerArgs } from "../backend/lld.js";
import {
hugeTextLinkerArgs,
hugeTextProgram,
rewriteHugeTextExecutable,
} from "../backend/huge-text.js";
import { openNativeExecutableCache } from "./executable-cache.js";
import { LinkerJobs } from "./linker-job.js";
import { nativeFileIdentity } from "./file-identity.js";
@@ -381,7 +386,9 @@ export class NativeCompiler {
// An optimized program that stays one partition is optimized whole and
// generates code for its parts concurrently.
const codegenParts = library || optimization === "dev" ? 1 : nativeCodegenParts(size);
const partitions = library ? 1 : Math.max(nativeProgramPartitions(size), codegenParts);
const partitions = library
? 1
: Math.max(nativeProgramPartitions(size, optimization === "dev"), codegenParts);
return this.emitObject(
input,
output,
@@ -637,11 +644,15 @@ export class NativeCompiler {
});
// Linux links with lld when the driver finds it.
const lld = options.sanitize ? [] : lldLinkerArgs(target.platform, toolchain.linker);
const hugeText =
!options.sanitize && hugeTextProgram(target, [object, ...programObjects.slice(1)]);
const zig = process.env["SCRIPTC_LINKER"] === undefined && target.defaultLinker === "zig";
const linkMap = join(stage, "link.map");
const mapArgs = staticLinkMapArgs(target, options.staticLink === true, linkMap);
const linkArgsWith = (withLld: boolean): string[] => [
...(options.sanitize ? ["-fsanitize=address"] : toolchain.linkerArgs),
...(withLld ? lld : []),
...hugeTextLinkerArgs(hugeText, (withLld && lld.length !== 0) || zig),
...plan.driverFlags,
...mapArgs,
...(withLld ? plan.inputs : inputsWithoutLld(target.platform, plan.inputs)),
@@ -679,6 +690,7 @@ export class NativeCompiler {
}
timing("native-link");
requireNativeArtifact(stagedOutput);
if (hugeText) rewriteHugeTextExecutable(stagedOutput);
if (mapArgs.length !== 0) {
const refusal = staticExecutableRefusal(target, true, readFileSync(linkMap, "utf8"));
if (refusal !== null) throw new NativeCodegenError("SC3002", refusal);
+2
View File
@@ -440,6 +440,7 @@ export function prepareNativeExecutable(
lowRegion: lowRegionTarget(toolchain.target.platform, options.sanitize === true),
functionWorkers: emitWorkers(),
forkPrimitives: emitForkPrimitives(),
timing,
...inlineRcOption(options.optimization, options.sanitize === true),
...(options.detachedExit === true && outputKind === "exe" ? { detachedExit: true } : {}),
...targetCpuField(options.targetCpu),
@@ -556,6 +557,7 @@ export function prepareNativeLibrary(
wasi: toolchain.target.platform === "wasi",
objectAudit: options.sanitize === true,
sanitizeAttributes: options.sanitize === true,
timing,
...inlineRcOption(profile.optimization, options.sanitize === true),
});
timing("llvm-emit");
@@ -438,3 +438,36 @@ test("a loop that starts from a typed-array read guards the read's integer and o
await rm(dir, { recursive: true, force: true });
}
});
test("an exact integer loop limit becomes the endpoint without rounding", async () => {
const dir = await mkdtemp(join(tmpdir(), "scriptc-counted-limit-"));
try {
const entry = join(import.meta.dirname, "../../../tests/corpus/counted-loop-integer-limits.ts");
const outPath = join(dir, "limits.ir.json");
const result = await compile(entry, { outPath, outDir: dir, outputKind: "ir" });
if (!result.ok) throw new Error(result.diagnostics.map((d) => d.message).join("\n"));
const ll = emitLlvmModule(deserializeModule(await readFile(outPath, "utf8")), {});
const bodies = (method: string): string[] =>
[
...ll.matchAll(
new RegExp(
`define internal [^\\n]+ @\\w*Page_${method}(?:\\.i32)?\\([^\\n]*\\) #0 \\{ ; [^\\n]*([\\s\\S]*?)\\n\\}`,
"g",
),
),
].map((m) => m[1]!);
// The int32 rank is the endpoint of the loops in both directions, with
// exclusive and inclusive bounds: no floor, ceil or 2^53 clamp.
for (const method of ["set", "suffix"]) {
const found = bodies(method);
expect(found.length).toBeGreaterThan(0);
for (const body of found) {
expect(body).toContain("integer induction i");
expect(body).not.toMatch(/@llvm\.(floor|ceil)\.f64/);
expect(body).not.toContain("0x4340000000000000");
}
}
} finally {
await rm(dir, { recursive: true, force: true });
}
});
@@ -0,0 +1,140 @@
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { afterEach, expect, test } from "vitest";
import { compile, deserializeModule } from "../src/index.js";
import { emitLlvmModule } from "../src/backend/llvm/emitter.js";
import type { IrExpr, IrModule } from "../src/ir/ir.js";
const directories: string[] = [];
afterEach(async () => {
await Promise.all(directories.splice(0).map((dir) => rm(dir, { recursive: true, force: true })));
});
async function emit(source: string): Promise<string> {
const outDir = await mkdtemp(join(tmpdir(), "scriptc-function-table-"));
directories.push(outDir);
const entry = join(outDir, "main.ts");
await writeFile(entry, source);
const outPath = join(outDir, "program.ll");
const result = await compile(entry, { entry, outDir, outPath, outputKind: "llvm" });
expect(result.ok, JSON.stringify(result)).toBe(true);
return readFile(outPath, "utf8");
}
async function lowerIr(source: string): Promise<IrModule> {
const outDir = await mkdtemp(join(tmpdir(), "scriptc-function-table-"));
directories.push(outDir);
const entry = join(outDir, "main.ts");
await writeFile(entry, source);
const outPath = join(outDir, "main.ir.json");
const result = await compile(entry, { outDir, outPath, outputKind: "ir" });
expect(result.ok, JSON.stringify(result)).toBe(true);
return deserializeModule(await readFile(outPath, "utf8"));
}
function definition(llvm: string, symbol: string): string {
const match = new RegExp(`^define internal [^\\n]*@${symbol}(?:\\.i32)?\\([^]*?^}`, "m").exec(
llvm,
);
expect(match, symbol).not.toBeNull();
return match![0];
}
// A record of functions dispatched by key: the hit lends the stored closure
// to the call (no union box, no lookup through the runtime, no closure
// retain or release), and only a miss tests the key against
// Object.prototype's method names.
const source = `
type Op = (acc: number, arg: number) => number;
const ops: Record<string, Op> = {
add: (acc, arg) => acc + arg,
neg: (acc) => -acc,
};
function asserted(name: string, acc: number): number {
return ops[name]!(acc, 1);
}
function optional(name: string, acc: number): number {
return ops[name]?.(acc, 1) ?? acc;
}
// Object arguments: the inherited methods' key conversion must not make
// every indirect call a heap writer (which would end all borrowing).
class Point {
constructor(readonly x: number) {}
}
type Measure = (p: Point, q: Point) => number;
const measures: Record<string, Measure> = { sum: (p, q) => p.x + q.x };
function objects(name: string, p: Point): number {
return measures[name]!(p, p);
}
function checked(name: string, acc: number): number {
const handler = ops[name];
return handler ? handler(acc, 1) : acc;
}
let acc = 1;
for (const name of ["add", "neg", "nope", "toString"]) {
acc = asserted(name === "nope" ? "add" : name === "toString" ? "neg" : name, acc);
acc = optional(name, acc);
acc = checked(name, acc);
acc += objects(name === "nope" ? "sum" : "sum", new Point(acc));
}
console.log(acc);
`;
test("function-table dispatch borrows the stored closure on a hit", async () => {
const llvm = await emit(source);
for (const fn of ["asserted", "optional", "checked", "objects"]) {
const body = definition(llvm, `sc_bf_${fn}`);
expect(body, fn).not.toMatch(/scr_union_new_ref|scr_map_get_str_ref/);
expect(body, fn).not.toMatch(/scr_closure_retain|sc_nretain_func/);
expect(body, fn).toMatch(/sc_map_entry_str/);
}
});
// A borrowed ternary whose arms are fallback reads: a miss binds the
// inherited method into a frame-owned slot, which must outlive the arm
// until the call has used the lent closure. The lowering builds no such
// ternary today (a bound read is a call's own callee), so the test wraps
// the callee read of `pick` in one.
test("a borrowed ternary of fallback reads keeps a bound default past its arm", async () => {
const mod = await lowerIr(`
type Op = (x: string) => string;
const table: Record<string, Op> = { echo: (x) => x };
function pick(name: string, flag: boolean): string {
return table[name]!("echo");
}
console.log(pick(process.argv[2] ?? "hasOwnProperty", process.argv.length > 3));
`);
const pick = mod.functions.find((fn) => fn.name === "pick")!;
const flag = pick.params[1]!;
let wrapped = 0;
const visit = (node: unknown): void => {
if (node === null || typeof node !== "object") return;
for (const [key, value] of Object.entries(node)) {
const e = value as IrExpr | null;
if (e !== null && typeof e === "object" && e.kind === "nullish" && e.right.kind === "call") {
const cond: IrExpr = { kind: "varRef", localId: flag.localId, type: flag.type, loc: e.loc };
(node as Record<string, unknown>)[key] = {
kind: "ternary",
cond,
then: e,
else_: structuredClone(e),
type: e.type,
loc: e.loc,
};
wrapped++;
} else visit(value);
}
};
visit(pick.body);
expect(wrapped).toBe(1);
const body = definition(emitLlvmModule(mod), "sc_bf_pick");
// Each arm lends its read and leaves: nothing it filled is released
// before the join.
const arms = [...body.matchAll(/^read\.(?:then|else)\d+:\n([^]*?)(?=^read\.)/gm)].map(
(m) => m[1]!,
);
expect(arms).toHaveLength(2);
for (const arm of arms)
expect(arm).toMatch(/store ptr %t\d+, ptr %s\d+\n {2}br label %read\.end\d+\n$/);
});
@@ -0,0 +1,6 @@
{
"order": [
"<repo>/tests/corpus/8363-record-inherited-keys.ts"
],
"diags": []
}
@@ -0,0 +1,6 @@
{
"order": [
"<repo>/tests/corpus/8364-record-nullish-receiver-arm/main.ts"
],
"diags": []
}
@@ -0,0 +1,6 @@
{
"order": [
"<repo>/tests/corpus/8408-fetch-connection-reuse.ts"
],
"diags": []
}
@@ -0,0 +1,6 @@
{
"order": [
"<repo>/tests/corpus/8409-fetch-compressed-read-after-end.ts"
],
"diags": []
}
@@ -0,0 +1,6 @@
{
"order": [
"<repo>/tests/corpus/counted-loop-integer-limits.ts"
],
"diags": []
}
@@ -0,0 +1,6 @@
{
"order": [
"<repo>/tests/corpus/pad-case-converted-strings.ts"
],
"diags": []
}
@@ -0,0 +1,6 @@
{
"order": [
"<repo>/tests/corpus/web-stream-transform-pipes-untyped.mjs"
],
"diags": []
}
@@ -0,0 +1,6 @@
{
"order": [
"<repo>/tests/corpus/web-stream-writable-untyped.mjs"
],
"diags": []
}
+9 -2
View File
@@ -1551,8 +1551,11 @@ static ScrDyn *scr_dyn_then_of(const ScrDyn *value, const ScrDyn *reader) {
return NULL;
}
/* Resolves `target` with `value` (borrowed). */
static void scr_dyn_resolve_value(ScrPromise *target, ScrDyn *value, const ScrDyn *reader) {
/* Resolves `target` with `value` (borrowed). Inlined into both callers, so
* the code of scr_dyn_then_resolve_call does not depend on whether a
* program links scr_dyn_promise_resolve_with. */
__attribute__((always_inline)) static inline void scr_dyn_resolve_value(ScrPromise *target, ScrDyn *value,
const ScrDyn *reader) {
if (value->kind == SCR_DYN_PROMISE) {
scr_promise_resolve_dyn(target, scr_dyn_retain(value));
return;
@@ -1669,6 +1672,10 @@ ScrPromise *scr_dyn_promise_resolve(const ScrDyn *value, const ScrDyn *reader) {
return scr_dyn_promise_entry(NULL, value, reader);
}
void scr_dyn_promise_resolve_with(ScrPromise *target, ScrDyn *value) {
scr_dyn_resolve_value(target, value, NULL);
}
static void scr_dyn_settled_store(ScrArr *a, double i, ScrPromise *src) {
ScrDyn *record = scr_dyn_new_obj();
bool fulfilled = !scr_promise_is_rejected(src);
+24 -1
View File
@@ -244,7 +244,14 @@ typedef enum {
Y(WS_COUNT_STRATEGY) /* CountQueuingStrategy (scr_webstreams.c) */ \
Y(WS_RCONTROLLER) /* ReadableStreamDefaultController (scr_webstreams.c) */ \
Y(WS_READABLE) /* ReadableStream for checked-dynamic code (scr_webstreams.c) */ \
Y(WS_READER) /* ReadableStreamDefaultReader (scr_webstreams.c) */
Y(WS_READER) /* ReadableStreamDefaultReader (scr_webstreams.c) */ \
Y(WS_TCONTROLLER) /* TransformStreamDefaultController (scr_webstreams.c) */ \
Y(WS_TEXT_DECODER) /* TextDecoderStream (scr_webstreams.c) */ \
Y(WS_TEXT_ENCODER) /* TextEncoderStream (scr_webstreams.c) */ \
Y(WS_TRANSFORM) /* TransformStream for checked-dynamic code (scr_webstreams.c) */ \
Y(WS_WCONTROLLER) /* WritableStreamDefaultController (scr_webstreams.c) */ \
Y(WS_WRITABLE) /* WritableStream for checked-dynamic code (scr_webstreams.c) */ \
Y(WS_WRITER) /* WritableStreamDefaultWriter (scr_webstreams.c) */
/* Values below the largest pinned one that no tag holds, each with the
* reason. Never reuse one without removing it here. */
@@ -2054,6 +2061,12 @@ typedef struct ScrDynHandleClass {
/* The prototype's methods and getters in definition order (the order
* Object.keys reports); NULL = the methods, then the getters. */
const char *const *order;
/* A call of one of the class's method values
* (`WritableStream.prototype.close.call(stream)`): the class's method,
* which an own property of the same name on the handle does not
* replace. NULL = ScrDynHandleOps.invoke. */
ScrDyn *(*method_invoke)(void *h, ScrDyn *self, const char *method, ScrDyn *const *args, size_t argc,
const char *what);
} ScrDynHandleClass;
/* Installs a tag's ops with the class its handles belong to; both outlive
@@ -2080,6 +2093,11 @@ ScrDyn *scr_dyn_handle_class_prototype_of(const ScrDyn *handle);
* receiver (+1); NULL when the chain has none or with the exception
* pending. */
ScrDyn *scr_dyn_handle_class_member(ScrDynHandleTag tag, void *h, const char *key, size_t len);
/* A method value's call on a handle of its tag (`self`, borrowed): the
* class's method_invoke, or the ops' invoke for a tag without one. +1, or
* NULL with the exception pending. */
ScrDyn *scr_dyn_handle_class_method_call(const ScrDynHandleOps *ops, ScrDyn *self, const char *method,
ScrDyn *const *args, size_t argc);
/* The operations a Transform handle gives the stream unit (scr_stream.c):
* the slot where the handle keeps its stream (+1) with the stream's
@@ -2156,6 +2174,11 @@ ScrPromise *scr_dyn_promise_any(const ScrDyn *entries, const ScrDyn *reader);
* itself, a thenable adopts through its resolve job, a throwing `then`
* read rejects, and any other value fulfills. +1. */
ScrPromise *scr_dyn_promise_resolve(const ScrDyn *value, const ScrDyn *reader);
/* Resolves `target` with a dyn value (borrowed), as a promise's resolve
* function does: a native promise adopts, a thenable adopts through its
* resolve job, a throwing `then` read rejects, and any other value
* fulfills. */
void scr_dyn_promise_resolve_with(ScrPromise *target, ScrDyn *value);
/* Native predicates for typed reference capsules and boxed errors. */
bool scr_dyn_ref_typeof_is(const ScrDyn *d, const char *name);
@@ -183,6 +183,14 @@ ScrDyn *scr_dyn_handle_class_member(ScrDynHandleTag tag, void *h, const char *ke
return value;
}
ScrDyn *scr_dyn_handle_class_method_call(const ScrDynHandleOps *ops, ScrDyn *self, const char *method,
ScrDyn *const *args, size_t argc) {
const ScrDynHandleClass *klass = hc_classes[self->v.handle.tag];
if (klass && klass->method_invoke)
return klass->method_invoke(self->v.handle.ptr, self, method, args, argc, method);
return ops->invoke(self->v.handle.ptr, self, method, args, argc, method);
}
static void hc_define_method(ScrDyn *prototype, ScrDynHandleTag tag, const char *name) {
ScrDyn *fn = scr_dyn_handle_class_method(tag, name);
if (fn) hc_define(prototype, name, fn, true, true, true);
+30 -7
View File
@@ -1140,15 +1140,19 @@ static void sf_response_pause_if_full(SfStream *s) {
static void sf_response_resume_if_ready(SfStream *s) {
SfTransfer *t = s->response_owner;
if (!t || t->done || !t->response_paused || !t->response_socket ||
s->queued >= 1 || s->close_requested || s->closed || s->error) {
if (!t || t->done || s->queued >= 1 || s->close_requested || s->closed ||
s->error || t->inflate_draining) {
return;
}
if (t->inflate_draining) return;
if (t->inflating && (t->inflate_pending || t->inflate_needs_drain)) {
/* Compressed input or decoder output still buffered drains on demand:
* while the socket is paused, and after 'end', when the transfer no
* longer holds the socket and no more data arrives. */
if (t->inflating && (t->inflate_pending || t->inflate_needs_drain) &&
(t->response_paused || t->response_ended)) {
sf_inflate_resume(t);
return;
}
if (!t->response_paused || !t->response_socket) return;
t->response_paused = false;
scr_net_sock_release(scr_net_sock_resume(t->response_socket));
}
@@ -5538,9 +5542,21 @@ static void sf_on_data(ScrClosure *cb, ScrBytes *chunk) {
sf_release(t);
}
/* After 'end' the keep-alive agent can hand the socket to the next request
* on the connection, so the transfer leaves it reading and stops pausing
* and resuming it. */
static void sf_drop_response_socket(SfTransfer *t) {
if (!t->response_socket) return;
if (t->response_paused) scr_net_sock_release(scr_net_sock_resume(t->response_socket));
t->response_paused = false;
scr_net_sock_release(t->response_socket);
t->response_socket = NULL;
}
static void sf_on_end(ScrClosure *cb) {
SfTransfer *t = sf_from(cb);
if (!t) return;
sf_drop_response_socket(t);
if (!t->done) {
if (t->inflating && t->response_stream &&
!t->response_stream->discarded) {
@@ -6081,9 +6097,16 @@ static bool sf_start_hop(SfTransfer *t) {
wrap_ctx = scr_tls_fetch_client_ctx(sni, true);
wrap = &scr_tls_fetch_client_wrap;
}
ScrHttpClientReq *client = scr_http_request_ex(
dial, dial_port, path, t->method, 0, headers, false, NULL, NULL,
default_port, wrap, wrap_ctx);
/* A direct connection goes through fetch's keep-alive agent, which hands
* an idle connection to the next request for the same origin, as
* undici's dispatcher does; the bare host keys the pool, so origins that
* share an address keep their own connections. A proxied request dials
* its own connection. */
ScrHttpClientReq *client =
proxy ? scr_http_request_ex(dial, dial_port, path, t->method, 0, headers, false, NULL,
NULL, default_port, wrap, wrap_ctx)
: scr_http_fetch_request(https, dial, dial_port, path, t->method, headers,
default_port, wrap, wrap_ctx, bare);
scr_arr_release(headers);
scr_str_release(path);
scr_str_release(dial);
+105 -53
View File
@@ -5286,6 +5286,9 @@ typedef struct ScrHttpAgent {
double max_sockets; /* INFINITY = Node's default */
double max_free;
double timeout_ms; /* < 0 = unset */
/* without a timeout, how long an idle pooled socket stays open (0 = until
* the server closes it); fetch's agent keeps undici's 4 s */
double idle_ms;
double default_port; /* settable (agent.defaultPort = p) */
bool destroyed;
ScrHttpAgentEnt *ents; /* append order — Node's FIFO queue */
@@ -5519,7 +5522,7 @@ static bool scr_http_client_pool(struct ScrHttpConn *conn, struct ScrHttpClientR
scr_str_release(name);
return false;
}
double timeout = ag->timeout_ms > 0 ? ag->timeout_ms : 0;
double timeout = ag->timeout_ms > 0 ? ag->timeout_ms : ag->idle_ms;
if (hint >= 0) {
double server = hint - 1000;
if (server < 0) {
@@ -5606,20 +5609,12 @@ static ScrStr *scr_http_agent_name(const char *host, size_t host_len, const char
return scr_jb_finish(&b);
}
ScrDyn *scr_http_agent_new(bool secure, bool keep_alive, double ka_msecs,
double max_sockets, double max_free, double timeout_ms,
double port /* < 0 = unset */, const ScrDyn *scheduling) {
/* A registered agent (+1), the half of new http.Agent(opts) after option
* validation. */
static ScrHttpAgent *scr_http_agent_alloc(bool secure, bool keep_alive, double ka_msecs,
double max_sockets, double max_free, double timeout_ms,
double port, bool fifo) {
scr_http_install();
/* Node: options.scheduling || 'lifo', then validateOneOf */
bool fifo = false;
if (scheduling != NULL && scr_dyn_truthy(scheduling)) {
const ScrStr *s = scheduling->kind == SCR_DYN_STR ? scheduling->v.str : NULL;
if (s != NULL && s->len == 4 && memcmp(s->data, "fifo", 4) == 0) fifo = true;
else if (s == NULL || s->len != 4 || memcmp(s->data, "lifo", 4) != 0) {
scr_dyn_arg_value_fail("scheduling", "must be one of: 'fifo', 'lifo'", scheduling);
return NULL;
}
}
ScrHttpAgent *a = calloc(1, sizeof *a);
if (!a) scr_http_oom();
a->rc = 1;
@@ -5642,6 +5637,25 @@ ScrDyn *scr_http_agent_new(bool secure, bool keep_alive, double ka_msecs,
a->in_registry = true;
a->next = scr_http_agents;
scr_http_agents = scr_http_agent_retain(a);
return a;
}
ScrDyn *scr_http_agent_new(bool secure, bool keep_alive, double ka_msecs,
double max_sockets, double max_free, double timeout_ms,
double port /* < 0 = unset */, const ScrDyn *scheduling) {
scr_http_install();
/* Node: options.scheduling || 'lifo', then validateOneOf */
bool fifo = false;
if (scheduling != NULL && scr_dyn_truthy(scheduling)) {
const ScrStr *s = scheduling->kind == SCR_DYN_STR ? scheduling->v.str : NULL;
if (s != NULL && s->len == 4 && memcmp(s->data, "fifo", 4) == 0) fifo = true;
else if (s == NULL || s->len != 4 || memcmp(s->data, "lifo", 4) != 0) {
scr_dyn_arg_value_fail("scheduling", "must be one of: 'fifo', 'lifo'", scheduling);
return NULL;
}
}
ScrHttpAgent *a = scr_http_agent_alloc(secure, keep_alive, ka_msecs, max_sockets, max_free,
timeout_ms, port, fifo);
ScrDyn *d = scr_dyn_new_handle(a, SCR_DYNH_HTTP_AGENT);
scr_http_agent_release(a); /* the handle's retain holds it */
return d;
@@ -5703,6 +5717,59 @@ static SCR_TL ScrStr *scr_http_next_socket_path = NULL;
extern SCR_TL ScrNetSocket *(*scr_http_socket_dialer)(ScrStr *path, bool deferred);
SCR_TL ScrNetSocket *(*scr_http_socket_dialer)(ScrStr *path, bool deferred) = NULL;
/* A request through agent `ag`: an idle pooled socket for the request's
* name (and pool_key) serves it, a free slot dials, and otherwise it
* queues. socket_path (borrowed, nullable) dials a Unix socket. */
static ScrHttpClientReq *scr_http_request_pooled(ScrHttpAgent *ag, ScrStr *host /*borrowed*/,
double port, ScrStr *path /*borrowed*/,
ScrStr *method /*borrowed*/, double timeout_ms,
ScrArr *header_pairs /*borrowed*/, bool auto_end,
ScrClosure *cb /*moves, nullable*/,
ScrHttpRespFn fn, int default_port,
void (*wrap)(ScrNetSocket *, void *),
void *wrap_ctx, ScrStr *pool_key /*borrowed*/,
ScrStr *socket_path /*borrowed*/) {
double p = port >= 0 ? port
: ag->default_port > 0 ? ag->default_port
: (double)default_port;
char portbuf[16];
int portn = snprintf(portbuf, sizeof portbuf, "%d", (int)p);
ScrStr *name = scr_http_agent_name(host->data, host->len, portbuf, (size_t)portn,
NULL, 0, 0, socket_path ? socket_path->data : NULL,
socket_path ? socket_path->len : 0);
/* a free pooled socket for the name serves the request right away */
ScrHttpAgentFree *reuse = ag->keep_alive ? scr_http_agent_free_take(ag, name, pool_key) : NULL;
bool queue = reuse == NULL && scr_http_agent_active(ag, name) >= ag->max_sockets;
ScrNetSocket *presock = socket_path != NULL ? scr_http_socket_dialer(socket_path, queue || reuse)
: queue || reuse ? scr_net_connect_deferred(p, host)
: NULL;
ScrHttpClientReq *c = scr_http_request_impl(host, p, path, method, timeout_ms, header_pairs,
auto_end, cb, fn, default_port, wrap, wrap_ctx,
presock);
if (c == NULL) { /* the method-token throw: nothing registered */
if (reuse != NULL) scr_http_agent_free_untake(ag, reuse); /* back to the list it came from */
scr_str_release(name);
return NULL;
}
c->pool_key = pool_key ? scr_str_retain(pool_key) : NULL;
if (ag->timeout_ms >= 0 && timeout_ms <= 0) {
c->timeout_ms = ag->timeout_ms;
scr_net_sock_set_timeout(c->sock, ag->timeout_ms);
}
if (reuse != NULL) scr_http_client_rebind(c, reuse, true);
else if (!queue) scr_http_conn_count(c->conn, ag->sock_count); /* dialed now */
c->agent = scr_http_agent_retain(ag);
ScrHttpAgentEnt *e = calloc(1, sizeof *e);
if (!e) scr_http_oom();
e->name = name; /* ownership moves */
e->client = scr_http_client_retain(c);
e->queued = queue;
ScrHttpAgentEnt **link = &ag->ents;
while (*link) link = &(*link)->next;
*link = e;
return c;
}
ScrHttpClientReq *scr_http_request_agent_ex(ScrStr *host /*borrowed*/, double port,
ScrStr *path /*borrowed*/, ScrStr *method /*borrowed*/,
double timeout_ms, ScrArr *header_pairs /*borrowed*/,
@@ -5754,46 +5821,29 @@ ScrHttpClientReq *scr_http_request_agent_ex(ScrStr *host /*borrowed*/, double po
scr_dyn_arg_type_fail("options.agent", "an instance of http.Agent, false, null, or undefined", agent);
return NULL;
}
ScrHttpAgent *ag = (ScrHttpAgent *)agent->v.handle.ptr;
double p = port >= 0 ? port
: ag->default_port > 0 ? ag->default_port
: (double)default_port;
char portbuf[16];
int portn = snprintf(portbuf, sizeof portbuf, "%d", (int)p);
ScrStr *name = scr_http_agent_name(host->data, host->len, portbuf, (size_t)portn,
NULL, 0, 0, socket_path ? socket_path->data : NULL,
socket_path ? socket_path->len : 0);
/* a free pooled socket for the name serves the request right away */
ScrHttpAgentFree *reuse = ag->keep_alive ? scr_http_agent_free_take(ag, name, pool_key) : NULL;
bool queue = reuse == NULL && scr_http_agent_active(ag, name) >= ag->max_sockets;
ScrNetSocket *presock = socket_path != NULL ? scr_http_socket_dialer(socket_path, queue || reuse)
: queue || reuse ? scr_net_connect_deferred(p, host)
: NULL;
ScrHttpClientReq *c = scr_http_request_impl(host, p, path, method, timeout_ms, header_pairs,
auto_end, cb, fn, default_port, wrap, wrap_ctx,
presock);
if (c == NULL) { /* the method-token throw: nothing registered */
if (reuse != NULL) scr_http_agent_free_untake(ag, reuse); /* back to the list it came from */
scr_str_release(name);
return NULL;
return scr_http_request_pooled((ScrHttpAgent *)agent->v.handle.ptr, host, port, path, method,
timeout_ms, header_pairs, auto_end, cb, fn, default_port, wrap,
wrap_ctx, pool_key, socket_path);
}
/* The agents fetch() dispatches through, like undici's global dispatcher:
* keep-alive connections per origin with no socket limit and no timeout on
* a request in flight; an idle connection closes after 4 s, or the server's
* Keep-Alive hint less one second when that is shorter. */
static SCR_TL ScrHttpAgent *scr_http_fetch_agents[2];
ScrHttpClientReq *scr_http_fetch_request(bool secure, ScrStr *host /*borrowed*/, double port,
ScrStr *path /*borrowed*/, ScrStr *method /*borrowed*/,
ScrArr *header_pairs /*borrowed*/, int default_port,
void (*wrap)(ScrNetSocket *, void *), void *wrap_ctx,
ScrStr *pool_key /*borrowed*/) {
ScrHttpAgent **slot = &scr_http_fetch_agents[secure ? 1 : 0];
if (*slot == NULL) {
*slot = scr_http_agent_alloc(secure, true, -1, -1, -1, -1, -1, false);
(*slot)->idle_ms = 4000;
}
c->pool_key = pool_key ? scr_str_retain(pool_key) : NULL;
if (ag->timeout_ms >= 0 && timeout_ms <= 0) {
c->timeout_ms = ag->timeout_ms;
scr_net_sock_set_timeout(c->sock, ag->timeout_ms);
}
if (reuse != NULL) scr_http_client_rebind(c, reuse, true);
else if (!queue) scr_http_conn_count(c->conn, ag->sock_count); /* dialed now */
c->agent = scr_http_agent_retain(ag);
ScrHttpAgentEnt *e = calloc(1, sizeof *e);
if (!e) scr_http_oom();
e->name = name; /* ownership moves */
e->client = scr_http_client_retain(c);
e->queued = queue;
ScrHttpAgentEnt **link = &ag->ents;
while (*link) link = &(*link)->next;
*link = e;
return c;
return scr_http_request_pooled(*slot, host, port, path, method, 0, header_pairs, false, NULL, NULL,
default_port, wrap, wrap_ctx, pool_key, NULL);
}
ScrHttpClientReq *scr_http_request_agent(ScrStr *host /*borrowed*/, double port,
@@ -5811,6 +5861,8 @@ static void scr_http_agents_cleanup(void) {
for (size_t i = 0; i < 2; i++) {
scr_dyn_release(scr_http_global_agents[i]);
scr_http_global_agents[i] = NULL;
if (scr_http_fetch_agents[i]) scr_http_agent_release(scr_http_fetch_agents[i]);
scr_http_fetch_agents[i] = NULL;
}
while (scr_http_agents) {
ScrHttpAgent *a = scr_http_agents;
+8
View File
@@ -365,6 +365,14 @@ ScrHttpClientReq *scr_http_request_agent_ex(ScrStr *host /*borrowed*/, double po
ScrStr *pool_key /*borrowed, nullable: https TLS options*/); /* +1 */
/* http.globalAgent / https.globalAgent: borrowed, and as a +1 value. */
const ScrDyn *scr_http_global_agent(bool secure);
/* fetch()'s client request for a direct connection: through fetch's own
* keep-alive agent, which reuses an idle connection with the same host,
* port and pool_key (+1). */
ScrHttpClientReq *scr_http_fetch_request(bool secure, ScrStr *host /*borrowed*/, double port,
ScrStr *path /*borrowed*/, ScrStr *method /*borrowed*/,
ScrArr *header_pairs /*borrowed*/, int default_port,
void (*wrap)(ScrNetSocket *, void *), void *wrap_ctx,
ScrStr *pool_key /*borrowed*/);
ScrDyn *scr_http_global_agent_value(bool secure);
ScrHttpClientReq *scr_http_request_agent(ScrStr *host /*borrowed*/, double port,
ScrStr *path /*borrowed*/, ScrStr *method /*borrowed*/,
+7
View File
@@ -7004,7 +7004,14 @@ static ScrDyn *scr_dynh_method_call(ScrClosure *closure, ScrDyn *const *args, si
ScrDyn *self = scr_dyn_this_get();
ScrDyn *result;
if (self->kind == SCR_DYN_HANDLE && self->v.handle.tag == tag) {
#ifdef SCR_HANDLE_CLASSES
/* A class's method value reaches the class's method past the handle's
* own properties; this build is the one programs with such classes
* link. */
result = scr_dyn_handle_class_method_call(ops, self, method->name, args, argc);
#else
result = ops->invoke(self->v.handle.ptr, self, method->name, args, argc, method->name);
#endif
} else {
const char *message = method->invalid_this ? method->invalid_this
: ops->invalid_this ? ops->invalid_this
+42
View File
@@ -2066,6 +2066,48 @@ bool scr_process_has_exit_code(void) {
return scr_exit_code_assigned || scr_exit_code_hint_get() != 0;
}
#if defined(__linux__) && !defined(SCR_LIB)
#include <link.h>
#include <sys/mman.h>
/* Text on huge pages (huge-text.ts). An executable with enough program code
* is linked with 2 MiB segment alignment, so each 2 MiB window inside its
* text segment starts at a 2 MiB file offset, and the kernel maps a window
* with one huge page when the page cache holds it as one 2 MiB folio.
* MADV_HUGEPAGE on those windows makes the kernel read a window that is not
* cached as a 2 MiB folio. glibc runs it from .preinit_array, before _init
* and the constructors fault in any of the text; musl, which has no
* .preinit_array, runs it as a constructor. Executables with 4 KiB alignment
* make no call. */
#define SCR_TEXT_HUGE_PAGE ((uintptr_t)2 << 20)
static int scr_text_segment(struct dl_phdr_info *info, size_t size, void *data) {
(void)size;
uintptr_t *range = data;
for (size_t i = 0; i < info->dlpi_phnum; i++) {
const ElfW(Phdr) *ph = &info->dlpi_phdr[i];
if (ph->p_type != PT_LOAD || !(ph->p_flags & PF_X) || ph->p_align < SCR_TEXT_HUGE_PAGE) continue;
range[0] = (uintptr_t)info->dlpi_addr + (uintptr_t)ph->p_vaddr;
range[1] = range[0] + (uintptr_t)ph->p_memsz;
break;
}
return 1; /* the first object is the executable */
}
static bool scr_text_huge_pages_done;
__attribute__((constructor)) static void scr_text_huge_pages(void) {
if (scr_text_huge_pages_done) return;
scr_text_huge_pages_done = true;
uintptr_t range[2] = {0, 0};
(void)dl_iterate_phdr(scr_text_segment, range);
uintptr_t lo = (range[0] + SCR_TEXT_HUGE_PAGE - 1) & ~(SCR_TEXT_HUGE_PAGE - 1);
uintptr_t hi = range[1] & ~(SCR_TEXT_HUGE_PAGE - 1);
if (hi > lo) (void)madvise((void *)lo, hi - lo, MADV_HUGEPAGE);
}
__attribute__((section(".preinit_array"), used)) static void (*const scr_text_huge_pages_entry)(void) =
scr_text_huge_pages;
#endif
#if defined(__linux__)
#include <sched.h>
#include <signal.h>
+8 -1
View File
@@ -14,7 +14,14 @@ test("number-key maps keep Map semantics across linear, direct and hashed indexe
try {
await exec("clang", [
"-std=c11",
"-O1",
// The runtime link set includes scr_json.c, scr_lib.c and scr_hash.c.
// With ASan/UBSan their optimized builds dominate the test's cost: -O1
// takes about 90s (scr_hash.c alone 29s, its unrolled digest rounds
// instrumented) and regularly starved past this test's timeout when the
// lane shared the box. The test checks Map semantics under sanitizers,
// not codegen, so -O0 keeps every assertion and sanitizer check and
// builds in about 15s.
"-O0",
"-Wall",
"-Wextra",
"-fsanitize=address,undefined",
+1
View File
@@ -2161,6 +2161,7 @@ static ScrStr *scr_str_case_conv(const ScrStr *s, int to_lower) {
out->data[i] = (char)(c >= first && c <= last ? c ^ 0x20 : c);
}
out->data[s->len] = '\0';
scr_str_note_ascii(out);
return out;
}
ScrJsonBuf b;
+22 -6
View File
@@ -2118,6 +2118,14 @@ double scr_str_utf16_len(ScrStr *s) {
return (double)(int64_t)units;
}
/* scr_str_utf16_len as a size: proven-ASCII and short strings are counted
* here, without the call. */
static inline size_t scr_str_len16(ScrStr *s) {
if (s->ascii) return s->len;
if (s->len <= 64) return scr_utf16_units_short(s->data, s->len);
return (size_t)scr_str_utf16_len(s);
}
/* Records the UTF-16 length of a string whose producer counted it (latin1
* text has one unit per byte), so scr_str_utf16_len does not walk it. Only
* strings that scr_str_utf16_len would otherwise record are kept. */
@@ -3137,15 +3145,17 @@ ScrStr *scr_str_replace_literal(ScrStr *s, ScrStr *search, ScrStr *rep, bool all
static ScrStr *scr_pad_impl(ScrStr *s, double maxLength, ScrStr *fill,
bool at_start) {
double target = scr_to_integer_or_infinity(maxLength);
size_t len16 = scr_sidx_len(s, scr_sidx_for(s));
size_t len16 = scr_str_len16(s);
if (!(target > (double)len16) || fill->len == 0) return scr_str_retain(s);
/* Reject pad sizes malloc could not satisfy before the double→size_t
* conversion can overflow (each unit is at most 3 bytes here: BMP chars
* and lone surrogates; astral chars are 4 bytes for 2 units). */
if (target > (double)((SIZE_MAX - sizeof(ScrStr) - 1) / 4)) scr_oom();
size_t pad16 = (size_t)target - len16;
size_t fill16 = scr_sidx_len(fill, scr_sidx_for(fill));
size_t reps = pad16 / fill16, rem16 = pad16 % fill16;
size_t fill16 = scr_str_len16(fill);
/* A one-unit fill, the common case, needs no division. */
size_t reps = fill16 == 1 ? pad16 : pad16 / fill16;
size_t rem16 = fill16 == 1 ? 0 : pad16 % fill16;
/* The truncated prefix of fill: whole chars while they fit in rem16
* units; a final astral char that doesn't fit contributes its high
* surrogate. */
@@ -3167,9 +3177,14 @@ static ScrStr *scr_pad_impl(ScrStr *s, double maxLength, ScrStr *fill,
size_t total = s->len + pad_b;
ScrStr *r = scr_str_alloc(total, total);
char *w = r->data + (at_start ? 0 : s->len);
for (size_t i = 0; i < reps; i++) {
memcpy(w, fill->data, fill->len);
w += fill->len;
if (fill->len == 1) {
memset(w, fill->data[0], reps);
w += reps;
} else {
for (size_t i = 0; i < reps; i++) {
memcpy(w, fill->data, fill->len);
w += fill->len;
}
}
memcpy(w, fill->data, prefix_b);
w += prefix_b;
@@ -3179,6 +3194,7 @@ static ScrStr *scr_pad_impl(ScrStr *s, double maxLength, ScrStr *fill,
}
memcpy(at_start ? w : r->data, s->data, s->len);
r->data[total] = '\0';
if (s->ascii && fill->ascii) scr_str_note_ascii(r);
/* Only a piece that starts with a lone low surrogate can complete a
* pair across a seam: the fill (after a fill or the receiver) or the
* receiver (after a fill ending in a high surrogate). */
File diff suppressed because it is too large Load Diff
+118 -3
View File
@@ -502,6 +502,86 @@ export function compareCounts(baseline, candidate) {
};
}
/* ── runtime packs ─────────────────────────────────────────────────────── */
/** Every file a runtime-pack.json lists (each flavor's runtime unit
* variants, their bitcode, and the vendor archives), as path → sha256. */
export function runtimePackFiles(manifest) {
const files = new Map();
const visit = (value) => {
if (Array.isArray(value)) value.forEach(visit);
else if (value !== null && typeof value === "object") {
if (typeof value.path === "string" && typeof value.sha256 === "string")
files.set(value.path, value.sha256);
Object.values(value).forEach(visit);
}
};
visit(manifest.flavors ?? {});
visit(manifest.archives ?? {});
return files;
}
/** The runtime pack files that differ between two runtime-pack.json
* manifests: changed, added and removed paths, each sorted. */
export function runtimePackDiff(baseline, candidate) {
const before = runtimePackFiles(baseline);
const after = runtimePackFiles(candidate);
const changed = [...before].filter(([path, sha]) => after.has(path) && after.get(path) !== sha);
return {
changed: changed.map(([path]) => path).sort(),
added: [...after.keys()].filter((path) => !before.has(path)).sort(),
removed: [...before.keys()].filter((path) => !after.has(path)).sort(),
};
}
/** The changed runtime pack files whose symbols executables link: runtime
* objects and vendor archives. */
export function linkedRuntimeFiles(diff) {
return [...diff.changed, ...diff.added, ...diff.removed].filter(
(path) => path.endsWith(".o") || path.endsWith(".a"),
);
}
/** Defined functions of textual LLVM IR (llvm-dis output) as name → hash of
* the body. Attribute group and metadata numbers are dropped first, since
* an unrelated change elsewhere in the unit renumbers them. */
export function llvmFunctionHashes(text) {
const functions = new Map();
const lines = text.split("\n");
for (let index = 0; index < lines.length; index++) {
const head = /^define\b[^@]*@("(?:[^"\\]|\\.)*"|[\w.$-]+)\(/.exec(lines[index]);
if (head === null) continue;
const body = [];
for (; index < lines.length && lines[index] !== "}"; index++) body.push(lines[index]);
const normalized = body.join("\n").replace(/ #\d+/g, "").replace(/!\d+/g, "!");
functions.set(head[1].replace(/^"|"$/g, ""), hash(normalized));
}
return functions;
}
/** Names whose hashes differ between two function maps, and the names only
* one side defines. */
export function changedFunctions(before, after) {
return {
changed: [...before]
.filter(([name, h]) => after.has(name) && after.get(name) !== h)
.map(([name]) => name)
.sort(),
added: [...after.keys()].filter((name) => !before.has(name)).sort(),
removed: [...before.keys()].filter((name) => !after.has(name)).sort(),
};
}
/** Whether every symbol an executable comparison added, removed or resized
* is one of `runtimeSymbols`, the symbols the changed runtime objects
* define. Mach-O nm prints no sizes, so there only additions and removals
* are checked. */
export function confinedToRuntime(exe, runtimeSymbols) {
return [...exe.added, ...exe.removed, ...exe.resized.map((s) => s.name)].every((name) =>
runtimeSymbols.has(name),
);
}
/** The report lines the gate prints for one program. */
export function programLines(entry) {
const lines = [];
@@ -510,6 +590,8 @@ export function programLines(entry) {
identical: "identical",
allowed: "allowed ",
differs: "DIFFERS ",
"runtime-only": "runtime ",
nondeterministic: "nondet ",
"build-error": "ERROR ",
"both-fail": "both-fail",
}[entry.verdict];
@@ -536,6 +618,17 @@ export function programLines(entry) {
);
} else if (entry.cachegrind?.skipped)
notes.push(`cachegrind skipped: ${entry.cachegrind.skipped}`);
if (entry.runtime) {
const linked = [...entry.runtime.changed, ...entry.runtime.added];
notes.push(
`runtime-only: links ${linked.length} changed runtime function${linked.length === 1 ? "" : "s"}` +
(linked.length
? ` (${linked.slice(0, 8).join(", ")}${linked.length > 8 ? ", ..." : ""})`
: ""),
);
}
if (entry.nondeterministic)
notes.push(`the baseline's own rebuild differs in ${entry.nondeterministic.join(", ")}`);
for (const failure of entry.dynamicTier?.failures ?? []) notes.push(`SYMBOLS: ${failure}`);
lines.push(`${label} ${entry.id}${notes.length ? ` ${notes.join("; ")}` : ""}`);
// A build error's first line is often a warning; print all of it.
@@ -560,7 +653,23 @@ export function programLines(entry) {
/** `required` programs must build with both checkouts: a failure both share
* means the gate compared nothing for them. Other programs may fail the same
* way on both sides, as a corpus program the compiler refuses does. */
export function programVerdict({ baseline, candidate, artifacts, allowed, required = false }) {
/** A difference in the release executable alone, with identical IR and
* LLVM, is "runtime-only" when `runtimeOnly` says the changed runtime
* objects explain it (the gate checks the runtime packs and the symbols).
* A difference the baseline also shows when it builds the program again is
* "nondeterministic": `nondeterministic` lists the artifact kinds whose
* baseline rebuild differs, and every differing artifact must be among
* them. Neither fails the gate unless `strict` is set. */
export function programVerdict({
baseline,
candidate,
artifacts,
allowed,
required = false,
runtimeOnly = false,
nondeterministic = [],
strict = false,
}) {
if (!baseline.ok || !candidate.ok) {
if (!baseline.ok && !candidate.ok && baseline.error === candidate.error)
return { verdict: "both-fail", fails: required };
@@ -568,7 +677,12 @@ export function programVerdict({ baseline, candidate, artifacts, allowed, requir
}
const differing = Object.entries(artifacts).filter(([, r]) => !r.identical);
if (differing.length === 0) return { verdict: "identical", fails: false };
return allowed ? { verdict: "allowed", fails: false } : { verdict: "differs", fails: true };
if (allowed) return { verdict: "allowed", fails: false };
if (runtimeOnly && differing.every(([kind]) => kind === "exe"))
return { verdict: "runtime-only", fails: strict };
if (differing.every(([kind]) => nondeterministic.includes(kind)))
return { verdict: "nondeterministic", fails: strict };
return { verdict: "differs", fails: true };
}
export function median(values) {
@@ -737,7 +851,8 @@ export function mergeReports(reports) {
}
const counts = {};
for (const p of merged.programs) counts[p.verdict] = (counts[p.verdict] ?? 0) + 1;
if (!merged.programs.some((p) => ["identical", "allowed", "differs"].includes(p.verdict)))
const compared = ["identical", "allowed", "runtime-only", "nondeterministic", "differs"];
if (!merged.programs.some((p) => compared.includes(p.verdict)))
merged.globalFailures.push("no program built with both checkouts, so nothing was compared");
merged.failures.push(...merged.globalFailures);
merged.counts = counts;
+227 -15
View File
@@ -10,7 +10,11 @@
* The good-code set: the runtime benchmark workloads (and benchmarks/builds
* applications), the self-hosted compiler (IR and LLVM), and the TypeScript
* corpus programs. A program whose sources match --allow (the change's
* pattern signature) may differ; any other difference fails the gate. */
* pattern signature) may differ. A program whose IR and LLVM are identical
* and whose executable differs only through changed runtime objects is
* "runtime-only", and a difference the baseline reproduces against itself is
* "nondeterministic": both pass with a note unless --strict. Any other
* difference fails the gate. */
import assert from "node:assert/strict";
import { spawn, spawnSync } from "node:child_process";
import { createHash } from "node:crypto";
@@ -33,19 +37,25 @@ import {
artifactHash,
cachedComparison,
cachegrindCounts,
changedFunctions,
compareArtifact,
compareCounts,
compareExecutables,
confinedToRuntime,
DYNAMIC_WORKLOADS,
definedFunctionNames,
dynamicTierSymbols,
linkedRuntimeFiles,
llvmFunctionHashes,
median,
nodeIslandSymbols,
nmOutput,
parseNm,
programInputsHash,
programLines,
programVerdict,
recordedBaseline,
runtimePackDiff,
timingGroups,
timingVerdict,
} from "./zero-cost-compare.mjs";
@@ -62,6 +72,7 @@ const { values } = parseArgs({
shard: { type: "string" },
only: { type: "string" },
allow: { type: "string" },
strict: { type: "boolean", default: false },
"timing-runs": { type: "string" },
"self-timing-runs": { type: "string", default: "0" },
"max-time-growth": { type: "string", default: "2" },
@@ -79,7 +90,7 @@ if (values.help || values.baseline === undefined) {
console.log(`Usage: node scripts/zero-cost-gate.mjs --baseline=<checkout> [--candidate=<checkout>]
[--programs=baseline|candidate|<dir>] [--set=bench,self,corpus] [--quick]
[--corpus=ts|all] [--shard=i/n] [--only=<regex>] [--allow=<regex>]
[--timing-runs=N] [--self-timing-runs=N] [--max-time-growth=2]
[--strict] [--timing-runs=N] [--self-timing-runs=N] [--max-time-growth=2]
[--max-size-growth=0]
[--cachegrind] [--jobs=N] [--json=<file>] [--keep]
[--baseline-cache=<file>] [--record-baseline-cache=<file>]
@@ -98,8 +109,20 @@ in shard 1. scripts/zero-cost-merge.mjs merges the shards' --json reports
into one verdict. --only filters program ids.
--allow=<regex> is the change's pattern signature: programs whose source
files match it may differ (reported as "allowed"). Any other difference in
IR, LLVM or the release executable fails the gate.
files match it may differ (reported as "allowed").
Other differences are classified:
runtime-only IR and LLVM identical; the executable differs only
through runtime objects that differ between the two
checkouts' runtime packs (every symbol the executable
gains, loses or resizes is defined by one of them), with
the same LLVM helper. Reported with the changed runtime
functions it links and its size change.
nondeterministic the baseline, building the program a second time,
differs from its own first build in the same artifacts.
differs anything else, in IR, LLVM or the executable.
"differs" fails the gate. runtime-only and nondeterministic pass with a
note; --strict fails them too, so any difference fails as before.
--timing-runs=N (default 5) times N paired --emit=llvm builds per bench
program, baseline and candidate side by side. The self-hosted compiler's
@@ -290,10 +313,9 @@ function compile(tree, args) {
* keeping the LLVM and IR beside the release executable. The self-hosted
* compiler emits LLVM only: each of its frontend runs takes minutes, and its
* LLVM names the first differing function as well as its IR would. */
async function buildProgram(program, label) {
async function buildProgram(program, label, tree = trees[label]) {
const dir = join(work, label, program.id.replace(/[^\w.-]+/g, "_"));
mkdirSync(dir, { recursive: true });
const tree = trees[label];
const failure = (result) => ({
ok: false,
error: (result.stderr || result.stdout || `exit ${result.status ?? result.signal}`)
@@ -434,6 +456,111 @@ async function instructionCounts(binary, args, out) {
return cachegrindCounts(readFileSync(out, "utf8"), new Set(definedFunctions(binary)));
}
/* ── runtime changes ───────────────────────────────────────────────────── */
const sha256 = (file) => createHash("sha256").update(readFileSync(file)).digest("hex");
/** llvm-dis from PATH or the pinned LLVM install, or null. */
function llvmDis() {
for (const candidate of ["llvm-dis", "llvm-dis-22", "/usr/lib/llvm-22/bin/llvm-dis"])
if (spawnSync(candidate, ["--version"]).status === 0) return candidate;
const config = spawnSync("llvm-config", ["--bindir"], { encoding: "utf8" });
const fromConfig = config.status === 0 ? join(config.stdout.trim(), "llvm-dis") : null;
return fromConfig !== null && existsSync(fromConfig) ? fromConfig : null;
}
/** What differs between the two checkouts' runtime packs for this host, or
* null when no object or archive does: the changed pack files, whether the
* LLVM helper is the same, the symbols the changed objects define (on either
* side), and the functions that changed in the bitcode of the changed
* release and speed objects. Bitcode alone is not a change: it records the
* checkout's source path, so it differs between any two checkouts. */
function runtimeChanges() {
const pack = (root) =>
readdirSync(join(root, "packages"))
.filter((name) => name.startsWith("runtime-"))
.filter((name) => existsSync(join(root, "packages", name, "runtime-pack.json")));
const shared = pack(trees.candidate).filter((name) => pack(trees.baseline).includes(name));
if (shared.length !== 1) return null;
const packDir = (root) => join(root, "packages", shared[0]);
const manifest = (root) =>
JSON.parse(readFileSync(join(packDir(root), "runtime-pack.json"), "utf8"));
const linked = (paths) => paths.filter((path) => !path.endsWith(".bc"));
const all = runtimePackDiff(manifest(trees.baseline), manifest(trees.candidate));
const diff = {
changed: linked(all.changed),
added: linked(all.added),
removed: linked(all.removed),
};
if (diff.changed.length + diff.added.length + diff.removed.length === 0) return null;
const helper = (root) => {
const file = join(
root,
"packages",
shared[0].replace(/^runtime-/, "llvm-"),
"bin/scriptc-llvm-codegen",
);
return existsSync(file) ? sha256(file) : null;
};
const helperSame =
helper(trees.baseline) !== null && helper(trees.baseline) === helper(trees.candidate);
const symbols = new Set();
const objects = [...diff.changed, ...diff.added, ...diff.removed].filter((path) =>
path.endsWith(".o"),
);
for (const path of linkedRuntimeFiles(diff))
for (const root of [trees.baseline, trees.candidate]) {
const file = join(packDir(root), path);
if (!existsSync(file)) continue;
const listed = spawnSync("nm", ["--defined-only", file], {
encoding: "utf8",
maxBuffer: 1 << 28,
});
for (const symbol of parseNm(listed.stdout ?? "")) symbols.add(symbol.name);
}
// Changed functions, by unit, from the bitcode of the release and speed
// flavors (the ones the gate builds with).
const dis = llvmDis();
const functions = { changed: new Set(), added: new Set(), removed: new Set(), units: new Set() };
for (const object of objects)
functions.units.add(object.replace(/^.*\//, "").replace(/\.o$/, ".c"));
const bitcode = objects
.filter((path) => /^artifacts\/(release|speed)\//.test(path))
.map((path) => path.replace(/\.o$/, ".bc"));
for (const path of bitcode) {
if (dis === null) continue;
const text = (root) => {
const file = join(packDir(root), path);
if (!existsSync(file)) return new Map();
const out = spawnSync(dis, [file, "-o", "-"], { encoding: "utf8", maxBuffer: 1 << 30 });
return llvmFunctionHashes(out.stdout ?? "");
};
const delta = changedFunctions(text(trees.baseline), text(trees.candidate));
for (const kind of ["changed", "added", "removed"])
for (const name of delta[kind]) functions[kind].add(name);
}
return {
pack: shared[0],
diff,
helperSame,
symbols,
disassembler: dis,
functions: Object.fromEntries(
Object.entries(functions).map(([kind, names]) => [kind, [...names].sort()]),
),
};
}
/** The changed runtime functions an executable defines. */
function linkedRuntimeChanges(runtime, binary) {
const defined = new Set(definedFunctions(binary));
const pick = (names) => names.filter((name) => defined.has(name));
return {
changed: pick(runtime.functions.changed),
added: pick(runtime.functions.added),
};
}
/* ── main ──────────────────────────────────────────────────────────────── */
function revision(root) {
@@ -523,6 +650,7 @@ if (recordPath === undefined && values["baseline-cache"] !== undefined) {
const programs = goodCodeSet().filter((p) => recordPath === undefined || p.kind !== "source");
assert.ok(programs.length > 0, "the selected good-code set is empty");
const runtime = recordPath === undefined ? runtimeChanges() : null;
const report = {
schema: 1,
date: new Date().toISOString(),
@@ -532,6 +660,16 @@ const report = {
),
programsRoot,
allow: values.allow ?? null,
strict: values.strict,
runtime:
runtime === null
? null
: {
pack: runtime.pack,
...runtime.diff,
helperSame: runtime.helperSame,
functions: runtime.functions,
},
programs: [],
timing: null,
failures: [],
@@ -547,6 +685,24 @@ console.log(
`zero-cost gate: ${programs.length} programs (${[...sets].join(", ")}), ` +
`baseline ${trees.baseline}, candidate ${trees.candidate}, programs from ${programsRoot}`,
);
if (runtime !== null) {
const { changed, added, removed } = runtime.diff;
console.log(
`runtime pack ${runtime.pack} differs: ${changed.length} changed, ${added.length} added, ` +
`${removed.length} removed files; units ${runtime.functions.units.join(", ") || "none"}` +
(runtime.helperSame ? "" : "; the LLVM helper differs too, so nothing is runtime-only"),
);
if (runtime.disassembler === null)
console.log("runtime functions: llvm-dis not found, so changed functions are not listed");
else
console.log(
`runtime functions: ${runtime.functions.changed.length} changed` +
(runtime.functions.changed.length
? ` (${runtime.functions.changed.slice(0, 20).join(", ")})`
: "") +
`, ${runtime.functions.added.length} added, ${runtime.functions.removed.length} removed`,
);
}
let failed = false;
const started = Date.now();
@@ -641,13 +797,39 @@ await pool(
(baseline.ok && candidate.ok ? compareBuilds(baseline, candidate) : {});
// The benchmark workloads and the self-hosted compiler always build.
const required = !program.id.startsWith("corpus/");
const { verdict, fails } = programVerdict({
baseline,
candidate,
artifacts,
allowed,
required,
});
const runtimeOnly =
runtime !== null &&
runtime.helperSame &&
artifacts.exe !== undefined &&
!artifacts.exe.identical &&
confinedToRuntime(artifacts.exe, runtime.symbols);
const judge = (nondeterministic) =>
programVerdict({
baseline,
candidate,
artifacts,
allowed,
required,
runtimeOnly,
nondeterministic,
strict: values.strict,
});
let { verdict, fails } = judge([]);
// Build a difference the change cannot explain once more with the
// baseline: when the baseline's two builds differ in the same
// artifacts, the difference is the build's, not the change's.
let recheck = null;
if (verdict === "differs") {
const again = await buildProgram(program, "recheck", trees.baseline);
if (again.ok) {
const self = compareBuilds(baseline, again);
recheck = Object.keys(artifacts).filter(
(kind) => !artifacts[kind].identical && !self[kind]?.identical,
);
if (again.dir && !values.keep) rmSync(again.dir, { recursive: true, force: true });
}
if (recheck?.length) ({ verdict, fails } = judge(recheck));
}
const entry = {
id: program.id,
verdict,
@@ -655,6 +837,8 @@ await pool(
buildMs: { baseline: baseline.ms ?? null, candidate: candidate.ms ?? null },
artifacts,
};
if (verdict === "runtime-only") entry.runtime = linkedRuntimeChanges(runtime, candidate.exe);
if (verdict === "nondeterministic") entry.nondeterministic = recheck;
if (settled !== null) entry.cachedBaseline = true;
if (!baseline.ok) entry.baselineError = baseline.error;
if (!candidate.ok) entry.candidateError = candidate.error;
@@ -787,7 +971,8 @@ if (checkCandidateSelf) {
const counts = {};
for (const p of report.programs) counts[p.verdict] = (counts[p.verdict] ?? 0) + 1;
if (!report.programs.some((p) => ["identical", "allowed", "differs"].includes(p.verdict))) {
const compared = ["identical", "allowed", "runtime-only", "nondeterministic", "differs"];
if (!report.programs.some((p) => compared.includes(p.verdict))) {
failed = true;
const failure = "no program built with both checkouts, so nothing was compared";
report.failures.push(failure);
@@ -802,11 +987,38 @@ if (values.json) {
}
if (!values.keep) rmSync(work, { recursive: true, force: true });
else console.log(`artifacts kept in ${work}`);
const noted = report.programs.filter((p) =>
["runtime-only", "nondeterministic"].includes(p.verdict),
);
console.log(
`zero-cost gate: ${failed ? "FAIL" : "PASS"} — ${Object.entries(counts)
`zero-cost gate: ${failed ? "FAIL" : noted.length ? "PASS with notes" : "PASS"} — ${Object.entries(
counts,
)
.map(([k, v]) => `${v} ${k}`)
.join(", ")} in ${fmtMs(report.elapsedMs)}` +
(report.baselineCache ? ` (${report.baselineCache.used} baselines from the cache)` : ""),
);
const runtimeOnlyPrograms = report.programs.filter((p) => p.verdict === "runtime-only");
if (runtimeOnlyPrograms.length > 0) {
const deltas = runtimeOnlyPrograms.map(
(p) => p.artifacts.exe.bytes.candidate - p.artifacts.exe.bytes.baseline,
);
const signed = (n) => `${n > 0 ? "+" : ""}${n}`;
const linked = new Set(
runtimeOnlyPrograms.flatMap((p) => [...p.runtime.changed, ...p.runtime.added]),
);
console.log(
` note: ${runtimeOnlyPrograms.length} runtime-only programs (IR and LLVM identical); ` +
`executable size ${signed(Math.min(...deltas))} to ${signed(Math.max(...deltas))} bytes; ` +
`changed runtime functions linked: ${[...linked].sort().join(", ") || "none listed"}` +
(values.strict ? "" : " (--strict fails these)"),
);
}
const nondeterministicPrograms = report.programs.filter((p) => p.verdict === "nondeterministic");
if (nondeterministicPrograms.length > 0)
console.log(
` note: ${nondeterministicPrograms.length} nondeterministic programs, whose baseline rebuild differs too: ` +
nondeterministicPrograms.map((p) => p.id).join(", "),
);
for (const failure of report.failures) console.log(` ${failure}`);
process.exit(failed ? 1 : 0);
@@ -0,0 +1,51 @@
// Object.prototype's own-key tests on records of functions convert their
// argument to a property key as Node does: an array joins its elements, and
// a record or class instance with its own toString calls it. A keyed read
// that falls back to an inherited method keeps the bound method alive
// through the checked projection that lends it.
type ListTest = (key: string[]) => unknown;
const lists: Record<string, ListTest> = { a: () => "a", "a,b": () => "a,b" };
console.log(
lists["hasOwnProperty"](["a"]),
lists["hasOwnProperty"](["a", "b"]),
lists["propertyIsEnumerable"](["b"]),
);
type Named = { toString(): string };
type NamedTest = (key: Named) => unknown;
const named: Record<string, NamedTest> = { echo: () => "echo" };
let calls = 0;
const echoKey: Named = {
toString() {
calls++;
return "echo";
},
};
console.log(
named["hasOwnProperty"](echoKey),
named["propertyIsEnumerable"]({ toString: () => "nope" }),
calls,
);
class Label {
readonly text: string;
constructor(text: string) {
this.text = text;
}
toString(): string {
return this.text;
}
}
type LabelTest = (key: Label) => unknown;
const labels: Record<string, LabelTest> = { x: () => "x" };
console.log(labels["hasOwnProperty"](new Label("x")), labels["hasOwnProperty"](new Label("y")));
type Op = (x: string) => unknown;
const table: Record<string, Op> = { echo: (x) => x };
// `table[name]!` is a checked projection: its successful arm lends the
// read, whose miss binds the inherited method to the record.
function run(name: string): unknown {
return table[name]!("echo");
}
for (const name of ["echo", "hasOwnProperty", "toString", "propertyIsEnumerable"])
console.log(name, String(run(name)));
@@ -0,0 +1,29 @@
// A conditional receiver whose arm reads a record with a fallback for a
// missing key, `(c ? names[k] ?? fallback(k) : other)`: the fallback's
// fresh string lives until the read of its length.
const names: Record<string, string> = { a: "alpha" };
let made = 0;
function fallback(k: string): string | undefined {
made++;
return k === "none" ? undefined : `fallback for ${k} #${made}`.repeat(2);
}
function asserted(useNames: boolean, k: string, other: string): number {
return (useNames ? (names[k] ?? fallback(k)) : other)!.length;
}
function optional(useNames: boolean, k: string, other: string): number | undefined {
return (useNames ? (names[k] ?? fallback(k)) : other)?.length;
}
function upper(useNames: boolean, k: string, other: string): string | undefined {
return (useNames ? (names[k] ?? fallback(k)) : other)?.toUpperCase();
}
for (const k of ["a", "b", "none"])
for (const useNames of [true, false]) {
let length: number | string;
try {
length = asserted(useNames, k, "other");
} catch (e) {
length = (e as Error).name;
}
console.log(k, useNames, length, optional(useNames, k, "other"), upper(useNames, k, "x"));
}
console.log("made", made);
@@ -0,0 +1,6 @@
{
"compilerOptions": {
"strict": true,
"noUncheckedIndexedAccess": true
}
}
@@ -0,0 +1,95 @@
// fetch() reuses keep-alive connections as Node's fetch does: sequential
// requests share connections, and every response is the same with reuse:
// concurrent requests, `connection: close` responses and requests,
// redirects, request bodies, an aborted response followed by another
// request, and a request after the server closed its idle connections. The
// program exits while idle connections remain open.
import { createServer, type IncomingMessage, type ServerResponse } from "node:http";
import type { AddressInfo } from "node:net";
let connections = 0;
const server = createServer((req: IncomingMessage, res: ServerResponse) => {
const url = req.url ?? "/";
if (url === "/close") {
res.writeHead(200, { connection: "close" });
res.end("closing");
return;
}
if (url === "/redirect") {
res.writeHead(302, { location: "/hello" });
res.end();
return;
}
if (url === "/echo") {
let body = "";
req.setEncoding("utf8");
req.on("data", (chunk: string) => {
body += chunk;
});
req.on("end", () => res.end(`echo:${body}`));
return;
}
if (url === "/slow") {
res.writeHead(200, { "content-type": "text/plain" });
res.write("first ");
setTimeout(() => res.end("rest"), 200);
return;
}
res.end(`hello ${url}`);
});
server.on("connection", () => {
connections++;
});
async function text(port: number, path: string, init?: RequestInit): Promise<string> {
const res = await fetch(`http://127.0.0.1:${port}${path}`, init);
return `${res.status} ${await res.text()}`;
}
async function main(port: number): Promise<void> {
const seen: string[] = [];
for (let i = 0; i < 5; i++) seen.push(await text(port, `/seq${i}`));
console.log("sequential", connections < 5, seen.join(" | "));
const parallel = await Promise.all([1, 2, 3].map((i) => text(port, `/par${i}`)));
for (let i = 0; i < 3; i++) parallel.push(await text(port, `/after${i}`));
console.log("concurrent", parallel.join(" | "));
console.log(await text(port, "/close"), await text(port, "/close"));
console.log(await text(port, "/a", { headers: { connection: "close" } }), await text(port, "/b"));
const res = await fetch(`http://127.0.0.1:${port}/redirect`);
console.log("redirect", res.redirected, res.url.endsWith("/hello"), await res.text());
const bodies = [];
for (let i = 0; i < 3; i++) bodies.push(await text(port, "/echo", { method: "POST", body: `n${i}` }));
console.log("post", bodies.join(" | "));
const controller = new AbortController();
const slow = await fetch(`http://127.0.0.1:${port}/slow`, { signal: controller.signal });
controller.abort();
try {
await slow.text();
} catch (e) {
console.log("aborted", (e as Error).name);
}
console.log(await text(port, "/after-abort"));
// A server that closes idle connections after 50 ms: the next request
// after a pause dials again.
const brief = createServer((req: IncomingMessage, res: ServerResponse) => res.end(`brief ${req.url}`));
brief.keepAliveTimeout = 50;
await new Promise<void>((resolve) => brief.listen(0, "127.0.0.1", () => resolve()));
const briefPort = (brief.address() as AddressInfo).port;
console.log(await text(briefPort, "/1"), await text(briefPort, "/2"));
await new Promise<void>((resolve) => setTimeout(resolve, 200));
console.log(await text(briefPort, "/3"));
brief.close();
server.close();
console.log("done");
}
server.listen(0, "127.0.0.1", () => {
void main((server.address() as AddressInfo).port);
});
@@ -0,0 +1,66 @@
// A compressed fetch() response whose whole body arrived before it was read:
// the decoded body is larger than one read, so decoded output is still
// buffered when the response ends, and every later read drains it. Reads
// after a pause (the end was processed meanwhile) return the rest of the
// body, as in Node, for gzip and deflate, through a reader and through
// text(), and the next request on the same connection works.
import { createServer, type IncomingMessage, type ServerResponse } from "node:http";
import type { AddressInfo } from "node:net";
import { deflateSync, gzipSync } from "node:zlib";
const body = Array.from({ length: 20000 }, (_, i) => `line ${i % 97} of the body\n`).join("");
const gzipped = gzipSync(body);
const deflated = deflateSync(body);
const server = createServer((req: IncomingMessage, res: ServerResponse) => {
const url = req.url ?? "/";
if (url === "/gzip" || url === "/gzip-text") {
res.writeHead(200, { "content-encoding": "gzip", "content-length": gzipped.length });
res.end(gzipped);
return;
}
if (url === "/deflate") {
res.writeHead(200, { "content-encoding": "deflate", "content-length": deflated.length });
res.end(deflated);
return;
}
res.end(`plain ${url}`);
});
const pause = (ms: number): Promise<void> => new Promise((resolve) => setTimeout(resolve, ms));
async function readSlowly(port: number, path: string): Promise<string> {
const res = await fetch(`http://127.0.0.1:${port}${path}`);
const reader = res.body!.getReader();
const decoder = new TextDecoder();
let text = "";
let reads = 0;
for (;;) {
// Let the response end before most reads.
await pause(reads < 3 ? 30 : 0);
const { done, value } = await reader.read();
if (done) break;
reads++;
text += decoder.decode(value, { stream: true });
}
text += decoder.decode();
return `${path} ${res.status} ${text.length} ${text === body}`;
}
async function main(port: number): Promise<void> {
console.log(await readSlowly(port, "/gzip"));
console.log(await readSlowly(port, "/deflate"));
const res = await fetch(`http://127.0.0.1:${port}/gzip-text`);
await pause(50);
const text = await res.text();
console.log("text after end", text.length, text === body);
const plain = await fetch(`http://127.0.0.1:${port}/after`);
console.log(await plain.text());
}
server.listen(0, "127.0.0.1", () => {
const { port } = server.address() as AddressInfo;
main(port)
.catch((error: unknown) => console.log("failed", String(error)))
.finally(() => server.close());
});
+152
View File
@@ -0,0 +1,152 @@
// Counted for loops whose limit is an exact integer: an int32 local, an
// int32 field, the result of a function returning an int32, a value in a
// proven range (a masked byte, an array length), counting up and down,
// with exclusive and inclusive bounds, empty ranges and the int32 edges.
// Fractional, NaN and infinite limits take the double path and give the
// same trip counts as Node.
function bitCount(x: number): number {
x = x - ((x >>> 1) & 0x55555555);
x = (x & 0x33333333) + ((x >>> 2) & 0x33333333);
return Math.imul((x + (x >>> 4)) & 0x0f0f0f0f, 0x01010101) >>> 24;
}
class Window {
low = 0;
high = 0;
constructor(low: number, high: number) {
this.low = low | 0;
this.high = high | 0;
}
}
function countUp(from: number, to: number): number {
let trips = 0;
for (let i = from; i < to; i++) trips++;
return trips;
}
function countUpInclusive(from: number, to: number): number {
let trips = 0;
for (let i = from; i <= to; i++) trips++;
return trips;
}
function countDown(from: number, to: number): number {
let trips = 0;
for (let i = from; i > to; i--) trips++;
return trips;
}
function countDownInclusive(from: number, to: number): number {
let trips = 0;
for (let i = from; i >= to; i--) trips++;
return trips;
}
// An int32 local limit, as in an insertion that shifts the elements after a rank.
function insertAt(values: number[], at: number, value: number): void {
values.push(value);
for (let i = values.length - 1; i > at; i--) values[i] = values[i - 1]!;
values[at] = value;
}
function removeAt(values: number[], at: number): void {
for (let i = at; i < values.length - 1; i++) values[i] = values[i + 1]!;
values.pop();
}
const values = [10, 20, 30, 40];
const rank = bitCount(0b1011 & 0b0011);
insertAt(values, rank, 25);
insertAt(values, 0, 5);
insertAt(values, values.length - 1, 45);
console.log("after inserts", values.join(","));
removeAt(values, bitCount(1));
removeAt(values, values.length - 1);
console.log("after removes", values.join(","));
// A rank from a method as the limit: a page that keeps its values in id
// order and shifts the later ones on insert and remove.
class Page {
readonly values: number[] = [];
bits = 0;
rank(slot: number): number {
return bitCount(this.bits & ((1 << (slot & 31)) - 1));
}
set(slot: number, value: number): void {
const bit = 1 << (slot & 31);
const at = this.rank(slot);
const values = this.values;
if ((this.bits & bit) !== 0) {
for (let i = at; i < values.length - 1; i++) values[i] = values[i + 1]!;
values.pop();
this.bits = (this.bits & ~bit) | 0;
return;
}
values.push(value);
for (let i = values.length - 1; i > at; i--) values[i] = values[i - 1]!;
values[at] = value;
this.bits = (this.bits | bit) | 0;
}
// Inclusive bounds at a rank, in both directions.
suffix(slot: number): number {
const at = this.rank(slot);
const values = this.values;
let total = 0;
for (let i = values.length - 1; i >= at; i--) total += values[i]!;
for (let i = 0; i <= at - 1; i++) total -= values[i]!;
return total;
}
}
const page = new Page();
for (const slot of [5, 1, 9, 3, 31, 0, 9, 1, 17]) page.set(slot, slot * 10);
console.log("page", page.values.join(","), page.bits >>> 0);
console.log("suffix", page.suffix(0), page.suffix(4), page.suffix(17), page.suffix(31));
// Limits from fields, calls and proven ranges.
const window = new Window(-3, 7);
let sum = 0;
for (let i = window.low; i < window.high; i++) sum += i;
console.log("field limits", sum);
sum = 0;
for (let i = window.high; i >= window.low; i--) sum += i;
console.log("field limits down", sum);
let byte = 0;
for (let k = 0; k < 6; k++) {
const limit = (k * 77) & 0xff;
let trips = 0;
for (let i = 0; i <= limit; i += 37) trips++;
byte += trips;
}
console.log("masked limits", byte);
let fromCalls = 0;
for (let i = bitCount(0xff); i > bitCount(0x3); i--) fromCalls++;
console.log("call limits", fromCalls);
// Edges and empty ranges.
console.log("up", countUp(0, 5), countUp(5, 0), countUp(-5, -2), countUp(2147483645, 2147483647));
console.log(
"up inclusive",
countUpInclusive(0, 5),
countUpInclusive(3, 3),
countUpInclusive(2147483645, 2147483647),
);
console.log("down", countDown(5, 0), countDown(0, 5), countDown(-2147483646, -2147483648));
console.log(
"down inclusive",
countDownInclusive(5, 0),
countDownInclusive(-2147483646, -2147483648),
countDownInclusive(0, 0),
);
// Limits that are not exact integers.
console.log("fractional", countUp(0, 2.5), countUpInclusive(0, 2.5), countDown(3, 0.5), countDownInclusive(3, 0.5));
console.log("negative fractional", countUp(-3, -0.5), countDownInclusive(0, -2.5));
console.log("NaN", countUp(0, Number.NaN), countDown(3, Number.NaN));
let capped = 0;
for (let i = 0; i < Number.POSITIVE_INFINITY; i++) {
if (++capped === 4) break;
}
console.log("infinite", capped);
let floored = 0;
for (let i = 3; i > Number.NEGATIVE_INFINITY; i--) {
if (++floored === 5) break;
}
console.log("negative infinite", floored);
@@ -0,0 +1,39 @@
// padStart and padEnd on the results of case conversion and slicing, with
// ASCII and non-ASCII receivers and fills, and index reads of the results.
const words = ["alpha", "beta", "gamma", "Delta", "épée", "straße", "日本語", "a😀b"];
const fills = ["-", "ab", "é", "😀", "", " "];
function describe(s: string): string {
return [
JSON.stringify(s),
s.length,
s.charAt(1),
s.at(-1),
s.indexOf("A"),
s.slice(1, 4),
s.charCodeAt(s.length - 1),
s.codePointAt(0),
].join(" ");
}
for (const word of words) {
const upper = word.slice(0, 3).toUpperCase();
const lower = word.toLowerCase();
for (const fill of fills) {
console.log(describe(upper.padStart(6, fill)));
console.log(describe(lower.padEnd(9, fill)));
}
console.log(describe(upper.padStart(2)), describe(upper.padEnd(5)));
}
// The same string padded again, compared and used as a key.
const counts = new Map<string, number>();
for (let i = 0; i < 40; i++) {
const key = words[i % words.length]!.slice(0, 3).toUpperCase().padStart(4, "-");
counts.set(key, (counts.get(key) ?? 0) + 1);
}
console.log([...counts].map(([k, v]) => `${k}=${v}`).join(","));
const padded = "abc".toUpperCase().padStart(5, "😀");
console.log(padded.length, [...padded].length, padded === "\ud83dABC".padStart(5, "😀"));
console.log("x".toUpperCase().padEnd(4, "é").normalize("NFD").length);
@@ -0,0 +1,212 @@
// Web TransformStream and pipes from untyped JavaScript: transformers with
// start, transform, flush and cancel, backpressure between the sides,
// controller errors and terminate, pipeTo and pipeThrough with their
// options, abort signals, own properties on streams and method wrappers
// over them, a transformer start that returns a thenable, and encoder and
// decoder streams, in Node's promise order.
import { TransformStream as WebTransformStream, TransformStreamDefaultController } from "node:stream/web";
const log = [];
const out = (...args) => log.push(args.map(String).join(" "));
const caught = (fn) => {
try {
fn();
return "ok";
} catch (e) {
return `${e.name}:${e.code ?? ""}:${e.message}`;
}
};
const settled = (p) => p.then((v) => `ok:${v}`, (e) => `${e?.name}:${e?.code ?? ""}:${e?.message ?? e}`);
const collect = async (readable) => {
const got = [];
for await (const chunk of readable) got.push(chunk);
return got.join(",");
};
out("classes", TransformStream === WebTransformStream, TransformStream.name, TransformStream.length, caught(() => new TransformStreamDefaultController()));
out("prototype", Object.keys(TransformStream.prototype).join(","), Object.keys(TransformStreamDefaultController.prototype).join(","));
{
const order = [];
const t = new TransformStream({
start(c) {
order.push(`start ${c.desiredSize}`);
c.enqueue("S");
},
transform(chunk, c) {
order.push(`transform ${chunk}`);
c.enqueue(chunk.toUpperCase());
},
flush(c) {
order.push("flush");
c.enqueue("F");
},
});
const writer = t.writable.getWriter();
writer.write("a");
writer.write("b");
writer.close();
out("transform", await collect(t.readable), order.join(" | "), await settled(writer.closed));
}
{
const t = new TransformStream({}, undefined, { highWaterMark: 1 });
const writer = t.writable.getWriter();
out("backpressure", writer.desiredSize, t.readable.locked);
const first = writer.write("x");
const second = writer.write("y");
out("queued", writer.desiredSize);
const reader = t.readable.getReader();
out("read", (await reader.read()).value, (await reader.read()).value, await settled(first), await settled(second));
}
{
const failure = new Error("transform-failure");
const t = new TransformStream({ transform() { throw failure; } });
const writer = t.writable.getWriter();
const reader = t.readable.getReader();
const read = settled(reader.read());
const write = settled(writer.write("z"));
out("transform-error", await write, await read, await settled(writer.closed));
}
{
/** @type {any} */
let controller;
const t = new TransformStream({ start(c) { controller = c; } });
controller.enqueue("before");
controller.terminate();
const reader = t.readable.getReader();
out("terminate", (await reader.read()).value, (await reader.read()).done, await settled(t.writable.getWriter().write("after")), caught(() => controller.enqueue("late")));
}
{
const cancels = [];
const t = new TransformStream({ cancel(reason) { cancels.push(`cancel ${reason}`); } });
out("cancel", await settled(t.readable.cancel("stop")), cancels.join(","), await settled(t.writable.getWriter().closed));
}
{
const source = ReadableStream.from(["p", "q", "r"]);
const got = [];
const sink = new WritableStream({ write(chunk) { got.push(chunk); } });
const piped = source.pipeThrough(new TransformStream({ transform(chunk, c) { c.enqueue(chunk + chunk); } })).pipeTo(sink);
out("pipe", await settled(piped), got.join(","), source.locked);
}
{
const source = new ReadableStream({ start(c) { c.enqueue(1); c.close(); } });
const sink = new WritableStream();
out("prevent-close", await settled(source.pipeTo(sink, { preventClose: true })), sink.locked, await settled(sink.getWriter().write(2)));
}
{
const failure = new Error("source-failure");
const source = new ReadableStream({ start(c) { c.error(failure); } });
const aborts = [];
const sink = new WritableStream({ abort(reason) { aborts.push(reason === failure); } });
out("source-error", await settled(source.pipeTo(sink)), aborts.join(","));
}
{
const controller = new AbortController();
const reason = new Error("signal");
const source = new ReadableStream({ pull() {}, cancel(e) { out("signal-cancel", e === reason); } }, { highWaterMark: 0 });
const sink = new WritableStream({ abort(e) { out("signal-abort", e === reason); } });
const piped = source.pipeTo(sink, { signal: controller.signal });
await Promise.resolve();
controller.abort(reason);
out("signal", await settled(piped), source.locked, sink.locked);
}
{
const source = ReadableStream.from(["own"]);
const sink = new WritableStream();
/** @type {any} */ (source).getReader = () => { throw new Error("public reader"); };
/** @type {any} */ (sink).tag = "kept";
await source.pipeTo(sink);
out("own-properties", /** @type {any} */ (sink).tag, typeof /** @type {any} */ (source).getReader, caught(() => { /** @type {any} */ (source).locked = true; }));
}
{
// An own property that wraps the saved prototype method reaches the
// class's method.
const calls = [];
const wrap = (target, name, method) => {
target[name] = function (...args) {
calls.push(name);
return method.apply(this, args);
};
};
const writable = new WritableStream({ write(chunk) { calls.push(`sink:${chunk}`); } });
wrap(writable, "getWriter", WritableStream.prototype.getWriter);
const writer = writable.getWriter();
wrap(writer, "write", WritableStreamDefaultWriter.prototype.write);
wrap(writer, "close", WritableStreamDefaultWriter.prototype.close);
await writer.write("a");
await writer.close();
const readable = ReadableStream.from(["b"]);
wrap(readable, "getReader", ReadableStream.prototype.getReader);
const reader = readable.getReader();
wrap(reader, "read", ReadableStreamDefaultReader.prototype.read);
const first = await reader.read();
const t = new TransformStream({
transform(chunk, controller) {
wrap(controller, "enqueue", TransformStreamDefaultController.prototype.enqueue);
controller.enqueue(`${chunk}!`);
},
});
t.writable.getWriter().write("c");
const transformed = await t.readable.getReader().read();
out("wrappers", calls.join(","), first.value, transformed.value, writable.locked);
}
{
// A transformer start that returns a thenable: both sides start when it
// settles, and its rejection errors the stream.
const order = [];
const later = { then(resolve) { order.push("then"); setTimeout(() => { order.push("resolve"); resolve(); }, 0); } };
const t = new TransformStream({ start() { return later; }, transform(chunk, c) { order.push(`transform:${chunk}`); c.enqueue(chunk); } });
Promise.resolve().then(() => order.push("tick1")).then(() => order.push("tick2")).then(() => order.push("tick3"));
const written = t.writable.getWriter().write("x").then(() => order.push("written"));
const read = await t.readable.getReader().read();
await written;
out("start-thenable", order.join(","), read.value);
const ticks = [];
const sync = new TransformStream({ start() { return { then(resolve) { ticks.push("then"); resolve(); } }; }, transform(chunk, c) { ticks.push("transform"); c.enqueue(chunk); } });
Promise.resolve().then(() => ticks.push("tick1")).then(() => ticks.push("tick2")).then(() => ticks.push("tick3")).then(() => ticks.push("tick4"));
sync.writable.getWriter().write("z").then(() => ticks.push("written"));
sync.readable.getReader().read().then(() => ticks.push("read"));
await new Promise((r) => setTimeout(r, 0));
out("start-thenable-ticks", ticks.join(","));
const failing = new TransformStream({ start() { return { then(_, reject) { reject(new Error("start failed")); } }; } });
out("start-thenable-reject", await settled(failing.writable.getWriter().write("y")), await settled(failing.readable.getReader().read()));
const throwing = new TransformStream({ start() { return { get then() { throw new Error("then getter"); } }; } });
out("start-then-getter", await settled(throwing.writable.getWriter().write("y")));
const plain = new TransformStream({ start() { return { then: 1 }; } });
const pending = plain.readable.getReader().read();
out("start-object", await settled(plain.writable.getWriter().write("p")), (await pending).value);
}
out(
"invalid",
await settled(ReadableStream.from([]).pipeTo(/** @type {any} */ ({}))),
caught(() => ReadableStream.from([]).pipeThrough(/** @type {any} */ ({}))),
caught(() => new TransformStream({ readableType: "bytes" })),
caught(() => new TransformStream({ transform: 1 })),
);
{
const bytes = [];
await ReadableStream.from(["hé", "\ud83d", "\ude00", "\ud800"])
.pipeThrough(new TextEncoderStream())
.pipeTo(new WritableStream({ write(chunk) { bytes.push(Array.from(chunk).join(".")); } }));
out("encoder", bytes.join(" | "), new TextEncoderStream().encoding, Object.prototype.toString.call(new TextEncoderStream()));
const text = [];
await ReadableStream.from([new Uint8Array([0xe2, 0x82]), new Uint8Array([0xac, 0x21]), new Uint8Array([0xf0])])
.pipeThrough(new TextDecoderStream())
.pipeTo(new WritableStream({ write(chunk) { text.push(chunk); } }));
const decoder = new TextDecoderStream("utf-8", { fatal: true });
out("decoder", text.join("|"), decoder.encoding, decoder.fatal, decoder.ignoreBOM, caught(() => new TextDecoderStream("nope")));
}
console.log(log.join("\n"));
@@ -0,0 +1,144 @@
// Web WritableStream from untyped JavaScript: the classes as constructor
// values, prototypes and brand checks, writers with backpressure and their
// ready and closed promises, queuing strategies, sink errors, close and
// abort ordering, the controller's signal, and argument errors, in Node's
// promise order.
import { WritableStream as WebWritableStream, WritableStreamDefaultWriter, WritableStreamDefaultController } from "node:stream/web";
const log = [];
const out = (...args) => log.push(args.map(String).join(" "));
const caught = (fn) => {
try {
fn();
return "ok";
} catch (e) {
return `${e.name}:${e.code ?? ""}:${e.message}`;
}
};
const settled = (p) => p.then((v) => `ok:${v}`, (e) => `${e?.name}:${e?.code ?? ""}:${e?.message ?? e}`);
out("classes", typeof WritableStream, WritableStream === WebWritableStream, WritableStream.name, WritableStream.length, WritableStreamDefaultWriter.length);
out("illegal", caught(() => new WritableStreamDefaultController()), caught(() => /** @type {any} */ (WritableStream)()));
out("prototype", Object.keys(WritableStream.prototype).join(","), Object.keys(WritableStreamDefaultWriter.prototype).join(","), Object.keys(WritableStreamDefaultController.prototype).join(","));
{
const order = [];
const chunks = [];
const stream = new WritableStream(
{
start() { order.push("start"); },
write(chunk, controller) {
order.push(`write ${chunk}`);
chunks.push(chunk);
return new Promise((resolve) => setTimeout(resolve, 1));
},
close() { order.push("close"); },
},
new CountQueuingStrategy({ highWaterMark: 2 }),
);
const writer = stream.getWriter();
out("writer", writer instanceof WritableStreamDefaultWriter, stream.locked, writer.desiredSize, caught(() => stream.getWriter()));
const writes = [writer.write("a"), writer.write("b"), writer.write("c")];
out("queued", writer.desiredSize, Object.prototype.toString.call(stream), Object.prototype.toString.call(writer));
let readySettled = false;
writer.ready.then(() => { readySettled = true; });
await Promise.resolve();
out("backpressure", readySettled, order.join(" | "));
for (const w of writes) await w;
await writer.ready;
out("drained", writer.desiredSize, readySettled, chunks.join(""));
await writer.close();
out("closed", await settled(writer.closed), writer.desiredSize, order.join(" | "), await settled(writer.write("late")));
writer.releaseLock();
out("released", stream.locked, await settled(writer.closed), caught(() => writer.desiredSize));
}
{
const sizes = [];
const stream = new WritableStream(
{ write() {} },
{ highWaterMark: 10, size(chunk) { sizes.push(chunk.length); return chunk.length; } },
);
const writer = stream.getWriter();
const good = writer.write("abcd");
out("sizes", writer.desiredSize, sizes.join(","));
const bad = writer.write({ length: -1 });
out("bad-size", await settled(good), await settled(bad), await settled(writer.closed), writer.desiredSize);
}
{
const failure = new Error("sink-write");
const stream = new WritableStream({ write() { throw failure; } });
const writer = stream.getWriter();
const first = writer.write("x");
const second = writer.write("y");
out("write-error", await settled(first), await settled(second), await settled(writer.closed), await settled(writer.ready));
}
{
const log2 = [];
/** @type {any} */
let controller;
const reason = new Error("stop");
const stream = new WritableStream({
start(c) {
controller = c;
c.signal.addEventListener("abort", () => log2.push(`signal ${c.signal.reason === reason}`));
},
write(chunk) { log2.push(`write ${chunk}`); },
abort(e) { log2.push(`abort ${e === reason}`); },
});
const writer = stream.getWriter();
await writer.write("one");
const abort = writer.abort(reason);
log2.push("aborting");
out("abort", await settled(abort), log2.join(" | "), controller.signal.aborted, await settled(writer.closed), writer.desiredSize);
}
{
/** @type {any} */
let controller;
const stream = new WritableStream({ start(c) { controller = c; } });
const writer = stream.getWriter();
controller.error(new Error("controller-error"));
out("controller-error", await settled(writer.write("z")), await settled(writer.closed), writer.desiredSize, await settled(stream.abort()));
}
{
const stream = new WritableStream();
const close = stream.close();
out("direct-close", await settled(stream.close()), await settled(close), await settled(stream.abort()));
}
out(
"brands",
caught(() => WritableStream.prototype.getWriter.call({})),
await settled(WritableStream.prototype.abort.call({})),
await settled(/** @type {any} */ (Object.getOwnPropertyDescriptor(WritableStreamDefaultWriter.prototype, "ready")).get.call({})),
caught(() => /** @type {any} */ (Object.getOwnPropertyDescriptor(WritableStreamDefaultWriter.prototype, "desiredSize")).get.call({})),
caught(() => WritableStreamDefaultController.prototype.error.call({})),
);
out(
"invalid",
caught(() => new WritableStream(null)),
caught(() => new WritableStream({ type: "bytes" })),
caught(() => new WritableStream({ write: 1 })),
caught(() => new WritableStream({ start: 1 })),
caught(() => new WritableStream({}, { highWaterMark: -1 })),
caught(() => new WritableStreamDefaultWriter(/** @type {any} */ ({}))),
);
{
const classes = { WritableStream, WritableStreamDefaultWriter, TransformStream };
const isInstance = (value, C) => value instanceof C;
const stream = new WritableStream();
const transform = new TransformStream();
out(
"by-value",
isInstance(stream, classes.WritableStream),
isInstance(stream.getWriter(), classes.WritableStreamDefaultWriter),
isInstance(transform, classes.TransformStream),
isInstance(transform.writable, classes.WritableStream),
isInstance(stream, classes.TransformStream),
);
}
console.log(log.join("\n"));
@@ -0,0 +1 @@
{ "plain": 2000, "san": 2000 }
@@ -0,0 +1 @@
{ "plain": 2000, "san": 2000 }
@@ -0,0 +1 @@
{ "plain": 9400, "san": 30000 }
@@ -0,0 +1 @@
{ "plain": 2800, "san": 42400 }
@@ -0,0 +1 @@
{ "plain": 3100, "san": 2000 }
@@ -0,0 +1 @@
{ "plain": 3600, "san": 3700 }
@@ -0,0 +1 @@
{ "plain": 4300, "san": 55000 }
@@ -0,0 +1 @@
{ "plain": 3900, "san": 6700 }
+2 -9
View File
@@ -57,6 +57,7 @@ const KNOWN: Record<string, Known> = {
divergence:
"Timeout handles are numbers (typeof) and lose their ref flag once fired or cleared",
},
// require() inside a function.
"island-web-plumbing": { run: "SC1090" },
"lazy-traps": { build: "SC2013" },
"misc-shims": { run: "SC1090" },
@@ -87,15 +88,6 @@ const KNOWN: Record<string, Known> = {
"node24-url-mutation": { run: "SC1090" },
"node24-util-adapters": { run: "Object.create with this value is not supported yet" },
"node24-util-types": { run: "SC1090" },
"node24-web-stream-encoding": { run: "SC2020" },
"node24-web-stream-errors": { run: "SC1090" },
"node24-web-stream-lifecycle": { run: "SC2020" },
"node24-web-stream-pipe-abort": { run: "SC2020" },
"node24-web-stream-pipe-errors": { run: "SC2020" },
"node24-web-stream-pipes": { run: "SC2020" },
"node24-web-stream-queues": { run: "SC2020" },
"node24-web-stream-tee": { run: "SC1090" },
"node24-web-stream-transforms": { run: "SC1090" },
"package-imports": { build: "SC2013" },
"promise-bridge": { build: "SC1090" },
"self-name": { build: "SC2013" },
@@ -103,6 +95,7 @@ const KNOWN: Record<string, Known> = {
"util-shims": { run: "SC2020" },
// The package redeclares the var it exported: Node copies the first value.
"var-export-redeclared": { run: "SC1090" },
// `class Upper extends TransformStream`: extending a builtin class from JavaScript.
"web-streams": { run: "SC1090" },
"workspace-copied": { build: "SC2013" },
"workspace-unclean": { build: "SC2013" },
+74 -64
View File
@@ -7,59 +7,68 @@ import { skipUnownedTest } from "./shard.js";
const fixture = join(import.meta.dirname, "../library-mode/wasm");
const hasClang = spawnSync("clang", ["--version"]).status === 0;
// Release compiles the whole three.js library into one LLVM module that a
// single process optimizes and codegens on one core, which takes 5-10
// minutes; sanitized dev builds are slower still. Lanes measured 331-581s for
// release against the old 600s budget, so both get 1_200_000ms. The plain dev
// case runs at about a fifth of release and keeps 600_000ms.
function timeoutFor(optimization: "dev" | "release"): number {
return optimization === "release" || process.env["SCRIPTC_SAN"] === "1" ? 1_200_000 : 600_000;
}
// The two optimizations take similar time (about 10 minutes each under ASan):
// SCRIPTC_TEST_SHARD runs them on different shards (tests/harness/shard.ts).
// skipUnownedTest partitions a describe block's tests.
describe("three.js native library", () => {
beforeEach((context) => skipUnownedTest(context, {}));
test.skipIf(!hasClang).each(["dev", "release"] as const)(
"three.js sends animated geometry, materials and resource lifecycle to a native host in %s",
async (optimization) => {
const directory = await mkdtemp("/tmp/scriptc-three-library-");
const sanitize = process.env["SCRIPTC_SAN"] === "1";
try {
const profile = JSON.parse(await readFile(join(fixture, "three.json"), "utf8"));
const profilePath = join(directory, "profile.json");
await writeFile(
profilePath,
JSON.stringify({
...profile,
name: "three-native-embedding",
entry: join(fixture, "three.mjs"),
optimization,
}),
);
const built = await compileLibrary({ profilePath, outDir: directory, sanitize });
if (!built.ok)
throw new Error(
built.diagnostics
.map((diagnostic) => `${diagnostic.code}: ${diagnostic.message}`)
.join("\n"),
for (const optimization of ["dev", "release"] as const) {
test.skipIf(!hasClang)(
`three.js sends animated geometry, materials and resource lifecycle to a native host in ${optimization}`,
async () => {
const directory = await mkdtemp("/tmp/scriptc-three-library-");
const sanitize = process.env["SCRIPTC_SAN"] === "1";
try {
const profile = JSON.parse(await readFile(join(fixture, "three.json"), "utf8"));
const profilePath = join(directory, "profile.json");
await writeFile(
profilePath,
JSON.stringify({
...profile,
name: "three-native-embedding",
entry: join(fixture, "three.mjs"),
optimization,
}),
);
const binary = join(directory, "host");
const linked = spawnSync(
"clang",
[
"-std=c11",
...(sanitize ? ["-fsanitize=address,undefined"] : []),
join(fixture, "three-native.c"),
built.archivePath,
"-lm",
"-o",
binary,
],
{ encoding: "utf8" },
);
expect(linked.status, linked.stderr).toBe(0);
const actual = spawnSync(binary, [], { encoding: "utf8", timeout: 30_000 });
expect(actual.status, actual.stderr).toBe(0);
const reference = spawnSync(
process.execPath,
[
"--no-warnings",
"--input-type=module",
"-e",
`
const built = await compileLibrary({ profilePath, outDir: directory, sanitize });
if (!built.ok)
throw new Error(
built.diagnostics
.map((diagnostic) => `${diagnostic.code}: ${diagnostic.message}`)
.join("\n"),
);
const binary = join(directory, "host");
const linked = spawnSync(
"clang",
[
"-std=c11",
...(sanitize ? ["-fsanitize=address,undefined"] : []),
join(fixture, "three-native.c"),
built.archivePath,
"-lm",
"-o",
binary,
],
{ encoding: "utf8" },
);
expect(linked.status, linked.stderr).toBe(0);
const actual = spawnSync(binary, [], { encoding: "utf8", timeout: 30_000 });
expect(actual.status, actual.stderr).toBe(0);
const reference = spawnSync(
process.execPath,
[
"--no-warnings",
"--input-type=module",
"-e",
`
const events=[];
for (const name of ["vertex","surface","resource"]) globalThis[name]=(...values)=>events.push([name,...values.map(value=>Number(value.toFixed(9)))]);
const {frame,dispose}=await import(${JSON.stringify(join(fixture, "three.mjs"))});
@@ -67,21 +76,22 @@ describe("three.js native library", () => {
dispose();
console.log(JSON.stringify(events));
`,
],
{ encoding: "utf8" },
);
expect(reference.status, reference.stderr).toBe(0);
const events = JSON.parse(reference.stdout);
expect(
actual.stdout
.trim()
.split("\n")
.map((line) => JSON.parse(line)),
).toEqual([...events, ...events]);
} finally {
await rm(directory, { recursive: true, force: true });
}
},
600_000,
);
],
{ encoding: "utf8" },
);
expect(reference.status, reference.stderr).toBe(0);
const events = JSON.parse(reference.stdout);
expect(
actual.stdout
.trim()
.split("\n")
.map((line) => JSON.parse(line)),
).toEqual([...events, ...events]);
} finally {
await rm(directory, { recursive: true, force: true });
}
},
timeoutFor(optimization),
);
}
});
+126
View File
@@ -3,17 +3,22 @@ import { expect, test } from "vitest";
import {
cachedComparison,
cachegrindCounts,
changedFunctions,
compareArtifact,
compareCounts,
compareExecutables,
confinedToRuntime,
definedFunctionNames,
dynamicTierSymbols,
linkedRuntimeFiles,
llvmFunctionHashes,
mergeReports,
nmOutput,
programInputsHash,
programLines,
programVerdict,
recordedBaseline,
runtimePackDiff,
timingVerdict,
} from "../../scripts/zero-cost-compare.mjs";
@@ -198,6 +203,116 @@ test("the zero-cost gate fails differing programs unless they match the signatur
expect(verdict({}, true, ok, failed)).toEqual({ verdict: "build-error", fails: true });
});
test("the zero-cost gate classifies runtime-only and nondeterministic differences", () => {
const ok = { ok: true };
const exeOnly = { ir: { identical: true }, llvm: { identical: true }, exe: { identical: false } };
const llvmToo = {
ir: { identical: true },
llvm: { identical: false },
exe: { identical: false },
};
const verdict = (artifacts: object, flags: object) =>
programVerdict({ baseline: ok, candidate: ok, artifacts, allowed: false, ...flags });
expect(verdict(exeOnly, { runtimeOnly: true })).toEqual({
verdict: "runtime-only",
fails: false,
});
expect(verdict(exeOnly, { runtimeOnly: true, strict: true })).toEqual({
verdict: "runtime-only",
fails: true,
});
// A changed runtime cannot explain different LLVM.
expect(verdict(llvmToo, { runtimeOnly: true })).toEqual({ verdict: "differs", fails: true });
expect(verdict(llvmToo, { nondeterministic: ["llvm", "exe"] })).toEqual({
verdict: "nondeterministic",
fails: false,
});
expect(verdict(llvmToo, { nondeterministic: ["llvm", "exe"], strict: true })).toEqual({
verdict: "nondeterministic",
fails: true,
});
// A nondeterministic executable does not excuse a deterministic LLVM difference.
expect(verdict(llvmToo, { nondeterministic: ["exe"] })).toEqual({
verdict: "differs",
fails: true,
});
});
test("the zero-cost gate reads the symbols of changed runtime objects and vendor archives", () => {
expect(
linkedRuntimeFiles({
changed: [
"artifacts/release/runtime/default/scr_a.o",
"artifacts/release/runtime/default/scr_a.bc",
],
added: ["artifacts/vendor/libz/libscriptc-zlib.a"],
removed: [],
}),
).toEqual([
"artifacts/release/runtime/default/scr_a.o",
"artifacts/vendor/libz/libscriptc-zlib.a",
]);
});
test("the zero-cost gate names the runtime pack files and functions that changed", () => {
const unit = (sha: string, bc: string) => ({
path: "artifacts/release/runtime/default/scr_a.o",
sha256: sha,
bitcode: { path: "artifacts/release/runtime/default/scr_a.bc", sha256: bc },
});
const pack = (a: object, extra: object[] = []) => ({
flavors: { release: { runtime_units: [{ source: "scr_a.c", variants: [a, ...extra] }] } },
archives: [{ path: "artifacts/vendor/libz.a", sha256: "z" }],
});
const other = { path: "artifacts/release/runtime/default/scr_b.o", sha256: "b" };
expect(runtimePackDiff(pack(unit("1", "2")), pack(unit("1", "2")))).toEqual({
changed: [],
added: [],
removed: [],
});
expect(runtimePackDiff(pack(unit("1", "2")), pack(unit("3", "4"), [other]))).toEqual({
changed: [
"artifacts/release/runtime/default/scr_a.bc",
"artifacts/release/runtime/default/scr_a.o",
],
added: ["artifacts/release/runtime/default/scr_b.o"],
removed: [],
});
const ll = (body: string, attrs: number) =>
[
`define i32 @scr_same(i32 %x) #${attrs} {`,
" ret i32 %x, !dbg !" + attrs,
"}",
"define internal void @scr_changed() {",
` ${body}`,
"}",
"",
].join("\n");
const before = llvmFunctionHashes(ll("ret void", 0));
const after = llvmFunctionHashes(
ll("call void @f()\n ret void", 7) + 'define void @"scr new"() {\n}\n',
);
// Renumbered attribute groups and metadata are not changes.
expect(changedFunctions(before, after)).toEqual({
changed: ["scr_changed"],
added: ["scr new"],
removed: [],
});
});
test("the zero-cost gate confines runtime-only differences to runtime symbols", () => {
const exe = (added: string[], removed: string[], resized: string[]) => ({
added,
removed,
resized: resized.map((name) => ({ name, baseline: 1, candidate: 2 })),
});
const runtime = new Set(["scr_a", "scr_b"]);
expect(confinedToRuntime(exe([], [], ["scr_a"]), runtime)).toBe(true);
expect(confinedToRuntime(exe(["scr_b"], ["scr_a"], []), runtime)).toBe(true);
expect(confinedToRuntime(exe([], [], ["main.$1"]), runtime)).toBe(false);
});
test("a recorded baseline settles identical programs and shared build errors", () => {
const hashes = { ir: "a", llvm: "b", exe: "c" };
const entry = { ok: true, ms: 12, hashes, exeBytes: 100 };
@@ -298,4 +413,15 @@ test("merged gate shards judge compile time and comparisons over every shard", (
"no program built with both checkouts, so nothing was compared",
]);
expect(() => mergeReports([fast, fast])).toThrow(/more than one gate report/);
// A runtime-only program was compared, and its line names the runtime change.
const runtimeOnly = {
id: "bench/r",
verdict: "runtime-only",
artifacts: { exe: { identical: false, first: "resized scr_a" } },
runtime: { changed: ["scr_a"], added: [] },
};
expect(mergeReports([shard([runtimeOnly], [])]).report.verdict).toBe("pass");
expect(programLines(runtimeOnly)).toEqual([
"runtime bench/r exe: resized scr_a; runtime-only: links 1 changed runtime function (scr_a)",
]);
});