mirror of
https://github.com/vercel-labs/scriptc.git
synced 2026-11-02 07:43:42 +08:00
Merge main
This commit is contained in:
@@ -54,7 +54,7 @@ Record a new or remeasured program's CI cost (used to balance the test shards) i
|
||||
|
||||
Before opening a PR that adds or changes corpus, diagnostic, or frontend fixtures, run `pnpm test:ts7 --baselines-only`. This fast completeness check uses the same fixture discovery as the parity suite; it does not type-check or replace behavioral parity. Generate missing preflight/order baselines, review their diagnostics and module order, and run the affected `order-parity.test.ts` batches before pushing. Keep existing baseline entries unchanged unless the corresponding behavior intentionally changes. The full `pnpm test:ts7` command also checks completeness before starting its sweep.
|
||||
|
||||
A change that must not cost good code anything, such as a compatibility fix that replaces a refusal, proves it with the zero-cost gate. The gate compiles the benchmark workloads, the self-hosted compiler, and the TypeScript corpus with two built checkouts. It requires byte-identical IR, LLVM, and release executables, and compile time within tolerance. `--allow=<regex>` names the change's pattern signature: programs whose sources match may differ, and the PR lists them. Any other difference fails the gate, which names the first differing function. The gate also enforces the two-tier runtime separation: no executable may gain symbols from `DYNAMIC_TIER_PREFIXES` in `scripts/zero-cost-compare.mjs`, and fully typed benchmarks may link none; extend that list as dynamic-tier units land. `--quick` checks only the benchmarks; `--cachegrind` adds per-function instruction counts for executables that differ. Run it in a Sandbox with both checkouts prepared:
|
||||
A change that must not cost good code anything, such as a compatibility fix that replaces a refusal, proves it with the zero-cost gate. The gate compiles the benchmark workloads, the self-hosted compiler, and the TypeScript corpus with two built checkouts. It requires byte-identical IR, LLVM, and release executables, and compile time within tolerance. `--allow=<regex>` names the change's pattern signature: programs whose sources match may differ, and the PR lists them. The gate classifies every other difference. A program is `runtime-only` when its IR and LLVM are identical and its executable differs only through runtime objects that differ between the two checkouts' runtime packs; the gate lists the changed runtime functions each such program links and the size change, and passes with a note, so a runtime C change does not fail the gate when no program's code changes. A program is `nondeterministic` when the baseline, building it again, differs from its own first build in the same artifacts; it also passes with a note. Any other difference is `differs` and fails the gate, which names the first differing function. `--strict` fails runtime-only and nondeterministic programs too, for a change that must leave every executable byte-identical. The gate also enforces the two-tier runtime separation: no executable may gain symbols from `DYNAMIC_TIER_PREFIXES` in `scripts/zero-cost-compare.mjs`, and fully typed benchmarks may link none; extend that list as dynamic-tier units land. `--quick` checks only the benchmarks; `--cachegrind` adds per-function instruction counts for executables that differ. Run it in a Sandbox with both checkouts prepared:
|
||||
|
||||
```bash
|
||||
node scripts/zero-cost-gate.mjs --baseline=/vercel/sandbox/prev --candidate=. --allow='\barguments\b'
|
||||
|
||||
@@ -42,7 +42,7 @@ Parsing, type checking, lowering, and LLVM IR generation run one after another o
|
||||
|
||||
### Partitions
|
||||
|
||||
The LLVM helper divides an executable's LLVM IR into partitions by its size. Up to 1 MiB of LLVM text is one partition, and up to 2 MiB is four. Larger programs get one partition per 4 MiB of LLVM text, rounded up to a power of two, and at least 8 and at most 64. `--emit=llvm` writes this text, so its file size shows how many partitions a program gets.
|
||||
The LLVM helper divides an executable's LLVM IR into partitions by its size. Up to 1 MiB of LLVM text is one partition, and up to 2 MiB is four. Larger programs get one partition per 4 MiB of LLVM text, rounded up to a power of two, and at least 8 and at most 64. `--emit=llvm` writes this text, so its file size shows how many partitions a program gets. `dev` executables, whose partitions are compiled on their own with nothing imported, are divided from 64 KiB: up to 128 KiB is four partitions, and larger `dev` programs get at least 8.
|
||||
|
||||
<table>
|
||||
<thead>
|
||||
|
||||
@@ -53,6 +53,8 @@ fib-linux: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically
|
||||
|
||||
Linux executables use the native Linux implementations of the event loop (epoll), servers, TLS (with the system's CA bundle), and `fs.watch`, and they are tested against Node.js on Linux, including Alpine.
|
||||
|
||||
An executable with at least 4 MiB of program code is linked with 2 MiB segment alignment when lld or zig links it, so the kernel can map its code with 2 MiB pages instead of 4 KiB pages. This takes effect when transparent huge pages are set to `madvise` or `always` and the file system keeps files in the page cache as 2 MiB folios (XFS, for example). scriptc writes such an executable with one write so that the page cache holds it that way; a copy written in small chunks maps with 4 KiB pages until it is read from disk again. The alignment also aligns the executable's load address to 2 MiB, which leaves 19 bits of address randomization for it instead of 28. Smaller executables, and links by GNU ld, keep 4 KiB alignment.
|
||||
|
||||
### Windows (x86_64)
|
||||
|
||||
```console
|
||||
|
||||
@@ -34,8 +34,8 @@
|
||||
A `Duplex.from` body function may take the `{ signal }` argument, which is aborted with the error the Duplex is destroyed with.
|
||||
`stream.compose` accepts streams and async generator functions in any position, an async function as the last stage, and an array, a string, or a generator as the first stage; one stage is `Duplex.from` of it.
|
||||
A web `ReadableStream` is read as an async iterable by `Readable.from` and as the first stage of `compose` or `pipeline`, and through its reader by `Readable.fromWeb` (with `objectMode`, `highWaterMark`, `encoding`, and `signal` in an options object literal) and `Duplex.from`.
|
||||
`new TransformStream()` without arguments builds the identity transform: its readable side is a web `ReadableStream`, its writable side supports `getWriter`, `abort`, `close`, and `locked`, and the writer supports `write`, `close`, `abort`, `releaseLock`, `ready`, `closed`, and `desiredSize`, with Node's backpressure (a write completes when the readable side reads its chunk).
|
||||
A web stream in a later compose stage, `Readable.toWeb`, a `TransformStream` with a transformer or queuing strategies, the `WritableStream` constructor, `pipeTo`, and `pipeThrough` are unsupported.
|
||||
In TypeScript, `new TransformStream()` without arguments builds the identity transform: its readable side is a web `ReadableStream`, its writable side supports `getWriter`, `abort`, `close`, and `locked`, and the writer supports `write`, `close`, `abort`, `releaseLock`, `ready`, `closed`, and `desiredSize`, with Node's backpressure (a write completes when the readable side reads its chunk).
|
||||
A web stream in a later compose stage and `Readable.toWeb` are unsupported, and in TypeScript, so are a `TransformStream` with a transformer or queuing strategies, the `WritableStream` constructor, `pipeTo`, and `pipeThrough`.
|
||||
The `ERR_INVALID_ARG_VALUE` message for a stream that is no longer readable or writable names the stream's class instead of printing the stream.
|
||||
`readable.compose(stage, { signal })` is `stream.compose(readable, stage)` with the signal added; the options are an object literal whose `signal` is a variable or a property read.
|
||||
`pipeline` from `node:stream/promises` accepts a trailing `{ signal, end }` object, with or without function stages, and `finished` accepts `readable`, `writable`, `error`, `cleanup`, and `signal` options, with boolean literals for the flags.
|
||||
@@ -60,8 +60,9 @@
|
||||
The `readline/promises` `Readline` class writes to the same outputs.
|
||||
`readline.emitKeypressEvents` emits `'keypress'` events on `process.stdin` or a readable stream, shared with terminal interfaces on that stream as in Node; a listener's first parameter must admit `undefined`, which it receives for escape sequences.
|
||||
Chained listener registration is unsupported.
|
||||
`node:stream/web` exports the same `ReadableStream` class as the global; in TypeScript, its other classes are unsupported.
|
||||
In JavaScript, `ReadableStream`, `ReadableStreamDefaultReader`, `ReadableStreamDefaultController`, `CountQueuingStrategy`, and `ByteLengthQueuingStrategy`, as globals and as `node:stream/web` exports, are constructor values with Node's prototypes, brand checks, and errors.
|
||||
Their underlying sources (`start`, `pull`, `cancel`), queuing strategies with a `size` function, readers, `tee`, `ReadableStream.from`, and async iteration settle promises in Node's order.
|
||||
`node:stream/web` exports the same classes as the globals; in TypeScript, only `ReadableStream` and the identity `TransformStream` have lowerings.
|
||||
In JavaScript, `ReadableStream`, `WritableStream`, and `TransformStream` with their default readers, writers, and controllers, `CountQueuingStrategy`, `ByteLengthQueuingStrategy`, `TextEncoderStream`, and `TextDecoderStream`, as globals and as `node:stream/web` exports, are constructor values with Node's prototypes, brand checks, and errors.
|
||||
Their underlying sources, sinks, and transformers, queuing strategies with a `size` function, readers and writers, `tee`, `pipeTo` and `pipeThrough` with their options and an abort signal, `ReadableStream.from`, and async iteration settle promises in Node's order, and the streams take own properties.
|
||||
`TextDecoderStream` decodes UTF-8; its other encodings are unsupported.
|
||||
A JavaScript `ReadableStream` given as a `Response` or `Request` body, or as `fetch`'s body, is read through a stream of its own: `response.body` is that stream rather than the one passed in, and the stream passed in is locked from the start.
|
||||
In JavaScript, a byte stream (`type: 'bytes'`), `pipeTo`, `pipeThrough`, `Object.getPrototypeOf` on these streams, readers, controllers, and strategies, their use as `WeakMap` and `WeakSet` keys, `ReadableStream.from` of a function that is not iterable, and the other `node:stream/web` classes are unsupported.
|
||||
In JavaScript, a byte stream (`type: 'bytes'`), a BYOB reader, a class that extends one of these classes, `Object.getPrototypeOf` on these values, their use as `WeakMap` and `WeakSet` keys, and `ReadableStream.from` of a function that is not iterable are unsupported.
|
||||
|
||||
@@ -81289,9 +81289,9 @@
|
||||
"inherited": true
|
||||
},
|
||||
"static": {
|
||||
"status": "not-implemented",
|
||||
"detail": "No static compiler lowering is registered for this API yet.",
|
||||
"verification": "registry-gap"
|
||||
"status": "partial",
|
||||
"detail": "Implemented for the documented static async-iteration subset.",
|
||||
"verification": "test-backed"
|
||||
}
|
||||
},
|
||||
{
|
||||
|
||||
@@ -111929,6 +111929,8 @@
|
||||
"tests/corpus/stream-web-readable.ts",
|
||||
"tests/corpus/stream-web-require.cjs",
|
||||
"tests/corpus/web-stream-readable-untyped.mjs",
|
||||
"tests/corpus/web-stream-transform-pipes-untyped.mjs",
|
||||
"tests/corpus/web-stream-writable-untyped.mjs",
|
||||
"tests/harness/fetch-conformance.test.ts"
|
||||
]
|
||||
},
|
||||
@@ -113011,8 +113013,11 @@
|
||||
"inherited": true
|
||||
},
|
||||
"static": {
|
||||
"status": "not-implemented",
|
||||
"evidence": "compiler-unmatched:webstreams"
|
||||
"status": "partial",
|
||||
"evidence": "compiler-feature:webstreams.WritableStream",
|
||||
"tests": [
|
||||
"tests/corpus/web-stream-writable-untyped.mjs"
|
||||
]
|
||||
},
|
||||
"anchorSource": "exact"
|
||||
},
|
||||
@@ -113405,7 +113410,8 @@
|
||||
"status": "partial",
|
||||
"evidence": "compiler-feature:webstreams.TransformStream",
|
||||
"tests": [
|
||||
"tests/corpus/7743-transform-stream-identity.ts"
|
||||
"tests/corpus/7743-transform-stream-identity.ts",
|
||||
"tests/corpus/web-stream-transform-pipes-untyped.mjs"
|
||||
]
|
||||
},
|
||||
"anchorSource": "exact"
|
||||
|
||||
@@ -434,7 +434,8 @@
|
||||
{ "chapter": "dns", "symbols": ["dns.lookup", "dns.lookupService", "dns.getServers", "dns.getDefaultResultOrder", "dns.setDefaultResultOrder"], "status": "partial", "evidence": ["tests/corpus/5648-dns-lookup-forms.ts", "tests/corpus/5649-dns-service-servers.ts", "tests/corpus/5650-dns-lookup-errors-untyped.cjs"] },
|
||||
{ "chapter": "dns", "symbols": ["dnsPromises.lookup", "dnsPromises.lookupService", "dnsPromises.getDefaultResultOrder"], "status": "partial", "evidence": ["tests/corpus/dns-promises-lookup.ts", "tests/corpus/dns-promises-untyped.cjs"] },
|
||||
{ "chapter": "webstreams", "symbols": ["ReadableStream"], "status": "partial", "evidence": ["tests/corpus/stream-web-readable.ts", "tests/corpus/web-stream-readable-untyped.mjs", "tests/corpus/stream-web-require.cjs", "tests/corpus/2862-web-readable-async-iteration.ts"] },
|
||||
{ "chapter": "webstreams", "symbols": ["TransformStream"], "status": "partial", "evidence": ["tests/corpus/7743-transform-stream-identity.ts"] },
|
||||
{ "chapter": "webstreams", "symbols": ["TransformStream"], "status": "partial", "evidence": ["tests/corpus/7743-transform-stream-identity.ts", "tests/corpus/web-stream-transform-pipes-untyped.mjs"] },
|
||||
{ "chapter": "webstreams", "symbols": ["WritableStream"], "status": "partial", "evidence": ["tests/corpus/web-stream-writable-untyped.mjs"] },
|
||||
{ "chapter": "globals", "signatures": ["Class: `TransformStream`"], "status": "partial", "evidence": ["tests/corpus/7743-transform-stream-identity.ts"] }
|
||||
],
|
||||
"dedicated": [
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, expect, test } from "vitest";
|
||||
import {
|
||||
HUGE_TEXT_MIN_CODE_BYTES,
|
||||
elfObjectCodeBytes,
|
||||
hugeTextLinkerArgs,
|
||||
hugeTextProgram,
|
||||
} from "./huge-text.js";
|
||||
|
||||
const dirs: string[] = [];
|
||||
afterEach(() => {
|
||||
for (const dir of dirs.splice(0)) rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
/** An ELF64 little-endian relocatable object's headers: one section per
|
||||
* entry, as [type, flags, size]. Section contents are left out, since only
|
||||
* the headers are read. */
|
||||
function elfObject(sections: readonly [number, number, number][]): Buffer {
|
||||
const shoff = 64;
|
||||
const bytes = Buffer.alloc(shoff + (sections.length + 1) * 64);
|
||||
bytes.writeUInt32BE(0x7f454c46, 0);
|
||||
bytes[4] = 2; // ELFCLASS64
|
||||
bytes[5] = 1; // ELFDATA2LSB
|
||||
bytes[6] = 1;
|
||||
bytes.writeUInt16LE(1, 16); // ET_REL
|
||||
bytes.writeUInt16LE(62, 18); // EM_X86_64
|
||||
bytes.writeBigUInt64LE(BigInt(shoff), 40);
|
||||
bytes.writeUInt16LE(64, 52);
|
||||
bytes.writeUInt16LE(64, 58);
|
||||
bytes.writeUInt16LE(sections.length + 1, 60);
|
||||
sections.forEach(([type, flags, size], index) => {
|
||||
const at = shoff + (index + 1) * 64;
|
||||
bytes.writeUInt32LE(type, at + 4);
|
||||
bytes.writeBigUInt64LE(BigInt(flags), at + 8);
|
||||
bytes.writeBigUInt64LE(BigInt(size), at + 32);
|
||||
});
|
||||
return bytes;
|
||||
}
|
||||
|
||||
function file(name: string, bytes: Buffer): string {
|
||||
const dir = mkdtempSync(join(tmpdir(), "scriptc-huge-text-"));
|
||||
dirs.push(dir);
|
||||
const path = join(dir, name);
|
||||
writeFileSync(path, bytes);
|
||||
return path;
|
||||
}
|
||||
|
||||
const PROGBITS = 1;
|
||||
const NOBITS = 8;
|
||||
const ALLOC = 0x2;
|
||||
const EXECINSTR = 0x4;
|
||||
|
||||
test("an object's code is the size of its executable PROGBITS sections", () => {
|
||||
const object = file(
|
||||
"program.o",
|
||||
elfObject([
|
||||
[PROGBITS, ALLOC | EXECINSTR, 3 << 20],
|
||||
[PROGBITS, ALLOC | EXECINSTR, 1000],
|
||||
[PROGBITS, ALLOC, 5 << 20],
|
||||
[NOBITS, ALLOC, 7 << 20],
|
||||
]),
|
||||
);
|
||||
expect(elfObjectCodeBytes(object)).toBe((3 << 20) + 1000);
|
||||
expect(elfObjectCodeBytes(file("not-elf.o", Buffer.from("not an object")))).toBe(0);
|
||||
expect(elfObjectCodeBytes(join(object, "..", "missing.o"))).toBe(0);
|
||||
});
|
||||
|
||||
test("only Linux programs with an interior 2 MiB window of code link for huge pages", () => {
|
||||
const linux = { platform: "linux" } as const;
|
||||
const half = file(
|
||||
"half.o",
|
||||
elfObject([[PROGBITS, ALLOC | EXECINSTR, HUGE_TEXT_MIN_CODE_BYTES / 2]]),
|
||||
);
|
||||
const small = file("small.o", elfObject([[PROGBITS, ALLOC | EXECINSTR, 1 << 20]]));
|
||||
expect(hugeTextProgram(linux, [half, small])).toBe(false);
|
||||
// Partitions of one program count together.
|
||||
expect(hugeTextProgram(linux, [half, half])).toBe(true);
|
||||
expect(hugeTextProgram({ platform: "darwin" } as const, [half, half])).toBe(false);
|
||||
expect(hugeTextProgram(linux, [])).toBe(false);
|
||||
});
|
||||
|
||||
test("the 2 MiB alignment applies to lld and zig links of huge-text programs", () => {
|
||||
expect(hugeTextLinkerArgs(true, true)).toEqual(["-Wl,-z,max-page-size=0x200000"]);
|
||||
// GNU ld pads the file to the alignment.
|
||||
expect(hugeTextLinkerArgs(true, false)).toEqual([]);
|
||||
expect(hugeTextLinkerArgs(false, true)).toEqual([]);
|
||||
});
|
||||
@@ -0,0 +1,110 @@
|
||||
/**
|
||||
* Text on 2 MiB pages for large Linux executables.
|
||||
*
|
||||
* The kernel maps a 2 MiB window of an executable's text with one huge page
|
||||
* when the window's virtual address and file offset are both 2 MiB aligned
|
||||
* and the page cache holds the window as one 2 MiB folio. A link with
|
||||
* `-z max-page-size=0x200000` keeps every segment's address congruent to its
|
||||
* file offset modulo 2 MiB, so every 2 MiB window inside the text segment
|
||||
* qualifies. lld and zig add no file padding for it; GNU ld pads the file to
|
||||
* the alignment, so links that fall back to it leave the flag out.
|
||||
*
|
||||
* The alignment also aligns a position-independent executable's load
|
||||
* address to 2 MiB, which leaves 19 bits of address randomization for it
|
||||
* instead of 28. Only executables whose program code fills an interior 2 MiB
|
||||
* window can gain from huge pages, so smaller executables link as before.
|
||||
*
|
||||
* lld writes its output through a memory mapping, which leaves the file in
|
||||
* the page cache as 4 KiB folios. A large executable is published by writing
|
||||
* its bytes with one write instead, which leaves 2 MiB folios where the file
|
||||
* system supports them. The runtime advises the interior windows huge pages
|
||||
* at startup (scr_lib.c), so a file that is read back from disk is read as
|
||||
* 2 MiB folios too.
|
||||
*/
|
||||
import {
|
||||
chmodSync,
|
||||
closeSync,
|
||||
openSync,
|
||||
readFileSync,
|
||||
readSync,
|
||||
renameSync,
|
||||
statSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import type { NativeTargetSpec } from "./targets.js";
|
||||
|
||||
/** Program code from which an executable is linked for huge pages: text of
|
||||
* this size has at least one 2 MiB window inside it wherever it starts. */
|
||||
export const HUGE_TEXT_MIN_CODE_BYTES = 4 << 20;
|
||||
|
||||
const SHF_EXECINSTR = 0x4n;
|
||||
|
||||
/** The bytes of executable sections in an ELF64 little-endian object, or 0
|
||||
* when the file is not one or cannot be read. Reads only the file and
|
||||
* section headers. */
|
||||
export function elfObjectCodeBytes(path: string): number {
|
||||
let fd: number;
|
||||
try {
|
||||
fd = openSync(path, "r");
|
||||
} catch {
|
||||
return 0;
|
||||
}
|
||||
try {
|
||||
const header = Buffer.alloc(64);
|
||||
if (readSync(fd, header, 0, 64, 0) !== 64) return 0;
|
||||
const view = new DataView(header.buffer, header.byteOffset, header.byteLength);
|
||||
if (view.getUint32(0, false) !== 0x7f454c46 || header[4] !== 2 || header[5] !== 1) return 0;
|
||||
const shoff = Number(view.getBigUint64(40, true));
|
||||
const shentsize = view.getUint16(58, true);
|
||||
let shnum = view.getUint16(60, true);
|
||||
if (shentsize !== 64 || shoff === 0) return 0;
|
||||
if (shnum === 0) {
|
||||
const first = Buffer.alloc(64);
|
||||
if (readSync(fd, first, 0, 64, shoff) !== 64) return 0;
|
||||
shnum = Number(new DataView(first.buffer, first.byteOffset, 64).getBigUint64(32, true));
|
||||
}
|
||||
const table = Buffer.alloc(shnum * 64);
|
||||
if (readSync(fd, table, 0, table.length, shoff) !== table.length) return 0;
|
||||
const sections = new DataView(table.buffer, table.byteOffset, table.byteLength);
|
||||
let code = 0;
|
||||
for (let i = 0; i < shnum; i++) {
|
||||
const at = i * 64;
|
||||
// SHT_PROGBITS sections with SHF_EXECINSTR.
|
||||
if (sections.getUint32(at + 4, true) !== 1) continue;
|
||||
if ((sections.getBigUint64(at + 8, true) & SHF_EXECINSTR) === 0n) continue;
|
||||
code += Number(sections.getBigUint64(at + 32, true));
|
||||
}
|
||||
return code;
|
||||
} finally {
|
||||
closeSync(fd);
|
||||
}
|
||||
}
|
||||
|
||||
/** Whether a link of these program objects is laid out for huge pages. */
|
||||
export function hugeTextProgram(
|
||||
target: Pick<NativeTargetSpec, "platform">,
|
||||
programObjects: readonly string[],
|
||||
): boolean {
|
||||
if (target.platform !== "linux") return false;
|
||||
let code = 0;
|
||||
for (const object of programObjects) {
|
||||
code += elfObjectCodeBytes(object);
|
||||
if (code >= HUGE_TEXT_MIN_CODE_BYTES) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/** The link arguments that align a huge-text executable's segments to
|
||||
* 2 MiB, for links by lld or zig. */
|
||||
export function hugeTextLinkerArgs(hugeText: boolean, linksWithLldOrZig: boolean): string[] {
|
||||
return hugeText && linksWithLldOrZig ? ["-Wl,-z,max-page-size=0x200000"] : [];
|
||||
}
|
||||
|
||||
/** Rewrites a linked huge-text executable with one write, so the page cache
|
||||
* holds it in 2 MiB folios; its caller then publishes it as usual. */
|
||||
export function rewriteHugeTextExecutable(linked: string): void {
|
||||
const written = `${linked}.written`;
|
||||
writeFileSync(written, readFileSync(linked));
|
||||
chmodSync(written, statSync(linked).mode & 0o7777);
|
||||
renameSync(written, linked);
|
||||
}
|
||||
@@ -13,6 +13,7 @@ import type { NativeArtifactDependency } from "./native/contracts.js";
|
||||
import { needsDarwinDebugSymbols } from "./debug-symbols.js";
|
||||
import { loadRuntimePack, type RuntimePackSelection } from "./runtime-pack.js";
|
||||
import { executableLinkInputs } from "./link-plan-core.js";
|
||||
import { hugeTextProgram } from "./huge-text.js";
|
||||
import type { NativeTargetSpec, WindowsSubsystem } from "./targets.js";
|
||||
|
||||
export interface NativeLinkPlan {
|
||||
@@ -30,6 +31,8 @@ export interface NativeLinkPlan {
|
||||
darwinDebugSymbols?: boolean;
|
||||
/** Linked with `--static`: the linked executable is checked before use. */
|
||||
staticLink?: boolean;
|
||||
/** Program code large enough for text on huge pages (huge-text.ts). */
|
||||
hugeText?: boolean;
|
||||
}
|
||||
|
||||
export async function createNativeLinkPlan(options: {
|
||||
@@ -85,5 +88,11 @@ export async function createNativeLinkPlan(options: {
|
||||
programObjectDependencies: [...(options.programObjectDependencies ?? [])],
|
||||
runtimePack,
|
||||
...(options.staticLink === true ? { staticLink: true } : {}),
|
||||
...(hugeTextProgram(options.target, [
|
||||
options.programObject,
|
||||
...(options.programPartitions ?? []),
|
||||
])
|
||||
? { hugeText: true }
|
||||
: {}),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -24,6 +24,7 @@ import { type NativeArtifactDependency } from "./native/contracts.js";
|
||||
import { RuntimePackError, stageRuntimePackArtifacts } from "./runtime-pack.js";
|
||||
import type { NativeLinkPlan } from "./link-plan.js";
|
||||
import { inputsWithoutLld, lldLinkerArgs } from "./lld.js";
|
||||
import { hugeTextLinkerArgs, rewriteHugeTextExecutable } from "./huge-text.js";
|
||||
import { systemLibraryLinkerArgs } from "./link-plan-core.js";
|
||||
import {
|
||||
createDarwinDebugSymbols,
|
||||
@@ -248,11 +249,13 @@ export async function linkNativeExecutable(
|
||||
stagedRoot = staged.root;
|
||||
// Linux links with lld when the default driver finds it.
|
||||
const lld = explicitLinker === undefined ? lldLinkerArgs(plan.target.platform, linker) : [];
|
||||
const zig = explicitLinker === undefined && plan.target.defaultLinker === "zig";
|
||||
const linkArgs = (withLld: boolean): string[] => {
|
||||
const inputs = plan.inputs.map((input) => staged.replacements.get(input) ?? input);
|
||||
return [
|
||||
...(explicitLinker === undefined ? plan.target.defaultLinkerArgs : []),
|
||||
...(withLld ? lld : []),
|
||||
...hugeTextLinkerArgs(plan.hugeText === true, (withLld && lld.length !== 0) || zig),
|
||||
...plan.driverFlags,
|
||||
...mapArgs,
|
||||
...(withLld ? inputs : inputsWithoutLld(plan.target.platform, inputs)),
|
||||
@@ -298,6 +301,7 @@ export async function linkNativeExecutable(
|
||||
});
|
||||
const output = await stat(privateOut);
|
||||
if (!output.isFile() || output.size === 0) throw new Error("linker produced no executable");
|
||||
if (plan.hugeText === true) rewriteHugeTextExecutable(privateOut);
|
||||
if (mapArgs.length !== 0) {
|
||||
const refusal = staticExecutableRefusal(plan.target, true, await readFile(linkMap, "utf8"));
|
||||
if (refusal !== null) throw new NativeCodegenError("SC3002", refusal);
|
||||
|
||||
@@ -9,7 +9,8 @@ import { everyExprChild, everyStmtChild } from "../../ir/traverse.js";
|
||||
import { borrowsStringInputs } from "./string-lifetimes.js";
|
||||
import { borrowsMapReadInputs } from "./map-read-lifetimes.js";
|
||||
import { LazyCaptures } from "./lazy-captures.js";
|
||||
import { storedLastUseRead } from "./local-extents.js";
|
||||
import { BodyLastUses, storedLastUseRead } from "./local-extents.js";
|
||||
import { InternalCompilerError } from "../../errors.js";
|
||||
|
||||
interface ForwardedUse {
|
||||
callee: string;
|
||||
@@ -497,6 +498,9 @@ export function analyzeCallLifetimes(
|
||||
// environment box takes a reference of its own when it is created.
|
||||
const unchangedCaptures = lazyCaptures.parameters(fn);
|
||||
const walks = walkParameters.get(fn.name);
|
||||
// One walk of the body answers every parameter's last use (consumedParameter).
|
||||
let lastUses: BodyLastUses | undefined;
|
||||
const bodyLastUses = (): BodyLastUses => (lastUses ??= new BodyLastUses(fn.body));
|
||||
fn.params.forEach((param, index) => {
|
||||
const local = locals.get(param.localId);
|
||||
if (
|
||||
@@ -511,7 +515,7 @@ export function analyzeCallLifetimes(
|
||||
!walks?.has(index) &&
|
||||
!unchangedCaptures.has(local.id) &&
|
||||
param.name !== "this" &&
|
||||
consumedParameter(fn, param.localId)
|
||||
consumedParameter(fn, param.localId, bodyLastUses)
|
||||
)
|
||||
consumed.push({ fn: fn.name, index });
|
||||
}
|
||||
@@ -626,6 +630,9 @@ export function analyzeCallLifetimes(
|
||||
return result;
|
||||
}
|
||||
|
||||
/** SCRIPTC_EMIT_FACTS_VERIFY=1: every BodyLastUses answer is checked against storedLastUseRead. */
|
||||
const verifiesLastUses = process.env["SCRIPTC_EMIT_FACTS_VERIFY"] === "1";
|
||||
|
||||
/** A parameter whose last use in the body stores it (a field, record or
|
||||
* element store, a record literal field, a push) and moves its reference
|
||||
* there (lastUseRead). Its owned convention lets a caller hand over a
|
||||
@@ -633,8 +640,12 @@ export function analyzeCallLifetimes(
|
||||
* copy while the caller releases its temporary right after the call. The
|
||||
* read must be the only node of its kind in the function, like every
|
||||
* moved read. */
|
||||
function consumedParameter(fn: IrFunction, localId: string): boolean {
|
||||
const read = storedLastUseRead(fn.body, -1, localId, true);
|
||||
function consumedParameter(fn: IrFunction, localId: string, lastUses: () => BodyLastUses): boolean {
|
||||
const read = lastUses().storedRead(localId);
|
||||
if (verifiesLastUses && read !== storedLastUseRead(fn.body, -1, localId, true))
|
||||
throw new InternalCompilerError(
|
||||
`call lifetimes: the body's last use of ${localId} in ${fn.name} differs`,
|
||||
);
|
||||
if (read === null) return false;
|
||||
let count = 0;
|
||||
const expr = (e: IrExpr): boolean => {
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
import { expect, test } from "vitest";
|
||||
import type { IntegerCountedForLoop } from "../../ir/integer-loops.js";
|
||||
import { F64, type IrExpr } from "../../ir/ir.js";
|
||||
import type { IntegerRange } from "../../ir/integer-ranges.js";
|
||||
import { BlockBuilder } from "./blocks.js";
|
||||
import { emitCountedLoopLimit } from "./counted-loops.js";
|
||||
import type { LlValue, LlvmEmitterContext } from "./expr-context.js";
|
||||
|
||||
const loc = { file: "test.ts", start: 0, end: 0 };
|
||||
|
||||
/** The endpoint emitCountedLoopLimit answers for a limit that emits as
|
||||
* `value` and has the proven `range`, and the lines it writes. */
|
||||
function endpoint(
|
||||
value: LlValue,
|
||||
range: IntegerRange,
|
||||
step: number,
|
||||
inclusive: boolean,
|
||||
): { endpoint: string; lines: string } {
|
||||
const limit: IrExpr = { loc, kind: "varRef", localId: "n", type: F64 };
|
||||
const B = new BlockBuilder();
|
||||
const host = {
|
||||
B,
|
||||
integerRanges: new Map([[limit, range]]),
|
||||
emitExpr: () => value,
|
||||
declare: () => {},
|
||||
} as unknown as LlvmEmitterContext;
|
||||
const loop = { step, inclusive, limit } as unknown as IntegerCountedForLoop;
|
||||
const result = emitCountedLoopLimit(host, loop);
|
||||
return { endpoint: result, lines: B.render() };
|
||||
}
|
||||
|
||||
test("a negative int32 limit whose ToUint32 view is a literal keeps its sign", () => {
|
||||
// A number literal's value carries its ToUint32 bits as a literal, so -1
|
||||
// has the view 4294967295; the endpoint is -1, as in `i > -1`.
|
||||
const minusOne: LlValue = { name: "-1.0", type: F64, uint32: "4294967295" };
|
||||
expect(endpoint(minusOne, { min: -1, max: -1 }, -1, false).endpoint).toBe("-1");
|
||||
const inclusive = endpoint(minusOne, { min: -1, max: -1 }, -1, true);
|
||||
expect(inclusive.lines).toContain("add nsw i64 -1, -1");
|
||||
const minusTwo: LlValue = { name: "-2.0", type: F64, uint32: "4294967294" };
|
||||
expect(endpoint(minusTwo, { min: -2, max: -2 }, 1, false).endpoint).toBe("-2");
|
||||
// A uint32 past int32 stays positive.
|
||||
const wide: LlValue = { name: "4294967295.0", type: F64, uint32: "4294967295" };
|
||||
expect(endpoint(wide, { min: 4294967295, max: 4294967295 }, 1, false).endpoint).toBe(
|
||||
"4294967295",
|
||||
);
|
||||
});
|
||||
@@ -13,7 +13,10 @@ import { integerOrNaNRange } from "./wrapping-sums.js";
|
||||
|
||||
/** The caller has proved or guarded the direction's finite endpoint. Clamp
|
||||
* the opposite infinity before integer conversion. Fractional limits become
|
||||
* exclusive integer endpoints for signed comparisons in either direction. */
|
||||
* exclusive integer endpoints for signed comparisons in either direction. A
|
||||
* limit that is an exact integer (an int32 local, an integer view, or a
|
||||
* proven range) is its own floor and ceiling and within ±2^53, so it is the
|
||||
* endpoint without the double conversion and rounding. */
|
||||
export function emitCountedLoopLimit(
|
||||
host: LlvmEmitterContext,
|
||||
loop: IntegerCountedForLoop,
|
||||
@@ -35,6 +38,24 @@ export function emitCountedLoopLimit(
|
||||
}
|
||||
const B = host.B;
|
||||
const value = host.emitExpr(loop.limit);
|
||||
const exact = exactInteger(host, value, loop.limit);
|
||||
if (exact) {
|
||||
// An i32 literal may be the unsigned bit pattern of a negative int32
|
||||
// (a ToUint32 view): read it back with the integer's signedness.
|
||||
const integer = /^-?\d+$/.test(exact.name)
|
||||
? String(
|
||||
exact.type === "i64"
|
||||
? Number(exact.name)
|
||||
: exact.signed
|
||||
? Number(exact.name) | 0
|
||||
: Number(exact.name) >>> 0,
|
||||
)
|
||||
: widenInteger(host, exact);
|
||||
if (!loop.inclusive) return integer;
|
||||
const limit = B.tmp();
|
||||
B.line(`${limit} = add nsw i64 ${integer}, ${ascending ? 1 : -1}`);
|
||||
return limit;
|
||||
}
|
||||
const within = B.tmp(),
|
||||
bounded = B.tmp(),
|
||||
rounded = B.tmp(),
|
||||
|
||||
@@ -670,6 +670,9 @@ export function emitFunction(emitter: LlEmitter, fn: IrFunction): string {
|
||||
// checked dynamic call. Keeping that fallback out of line leaves the
|
||||
// tests small enough to inline at each call site.
|
||||
if (fn.name.startsWith("%dyn.class.protocol.fallback.")) attrs = `noinline ${attrs}`;
|
||||
// A keyed record read's inherited-method miss path (record-inherited.ts)
|
||||
// stays out of line, so the hit path keeps its own small frame.
|
||||
else if (fn.name.startsWith("%rec.inherited.")) attrs = `noinline cold ${attrs}`;
|
||||
else if (fn.name.startsWith("%dyn.class.protocol.")) attrs = `inlinehint ${attrs}`;
|
||||
if (fn.name.startsWith(RESULT_ENTRY_PREFIX)) attrs = `inlinehint ${attrs}`;
|
||||
if (unoptimized && attrs === FN_ATTRS) attrs = `noinline optnone ${attrs}`;
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import type { TargetCpu } from "../target-cpu.js";
|
||||
import { loopEntryInlines } from "./loop-entry-versions.js";
|
||||
import type { FunctionLayout } from "./function-layout.js";
|
||||
import type { CompilationTiming } from "../../timing.js";
|
||||
import { relieveHeapPressure } from "../../heap-pressure.js";
|
||||
import { analyzeFrameEscapes, type FrameEscapes } from "./frame-escapes.js";
|
||||
import { wasmBoundsChecks } from "./wasm-modules.js";
|
||||
@@ -299,8 +300,10 @@ import {
|
||||
emitBytesIntrinsic,
|
||||
} from "./expr-bytes.js";
|
||||
import {
|
||||
borrowedRecordKeyGetOr,
|
||||
emitRecordKeyGet,
|
||||
emitRegexIntrinsic,
|
||||
immortalDefault,
|
||||
keyedRecordReadInto,
|
||||
pointerRecordKeyGet,
|
||||
} from "./expr-records.js";
|
||||
@@ -522,6 +525,11 @@ export interface LlvmTargetOptions {
|
||||
forkPrimitives?: ForkPrimitives | undefined;
|
||||
/** Functions to write first, by tier (function-layout.ts). */
|
||||
functionLayout?: FunctionLayout;
|
||||
/** Marks emission's serial steps: `llvm-emit-rewrites` after the module
|
||||
* passes, `llvm-emit-int32` after int32 selection, and
|
||||
* `llvm-emit-analyses` after the emitter's whole-program analyses. The
|
||||
* caller's next phase is the function bodies. */
|
||||
timing?: CompilationTiming;
|
||||
/** Exact frontend sources for dev-build locations and variable metadata. */
|
||||
debugSources?: ReadonlyMap<string, string>;
|
||||
/** Pointer width of the target C ABI. Native targets are 64-bit today. */
|
||||
@@ -706,6 +714,7 @@ function moduleEmitter(input: IrModule, options: LlvmTargetOptions): LlEmitter {
|
||||
};
|
||||
// Keep source storage intact for debugger inspection in dev builds.
|
||||
if (options.debugSources !== undefined) return new LlEmitter(mod, options, null, null);
|
||||
const timing = options.timing ?? (() => {});
|
||||
const passed = versionParameters(
|
||||
addRecordResultEntries(
|
||||
scalarizeLiteralResults(
|
||||
@@ -737,6 +746,7 @@ function moduleEmitter(input: IrModule, options: LlvmTargetOptions): LlEmitter {
|
||||
),
|
||||
),
|
||||
);
|
||||
timing("llvm-emit-rewrites");
|
||||
let emitted = selectInt32Arrays(passed, proven, resultEntryTargets);
|
||||
let prepared = rounds.last;
|
||||
// A proof that revised functions analyzed the module of the functions it
|
||||
@@ -758,7 +768,15 @@ function moduleEmitter(input: IrModule, options: LlvmTargetOptions): LlEmitter {
|
||||
}
|
||||
if (process.env["SCRIPTC_INT32_VERIFY"] === "1")
|
||||
verifyInt32Selection(passed, emitted, prepared?.mod === emitted ? prepared : null);
|
||||
return new LlEmitter(emitted, options, null, prepared?.mod === emitted ? prepared : null);
|
||||
timing("llvm-emit-int32");
|
||||
const emitter = new LlEmitter(
|
||||
emitted,
|
||||
options,
|
||||
null,
|
||||
prepared?.mod === emitted ? prepared : null,
|
||||
);
|
||||
timing("llvm-emit-analyses");
|
||||
return emitter;
|
||||
}
|
||||
|
||||
/** SCRIPTC_INT32_VERIFY=1: throws unless the selection with revisions
|
||||
@@ -2128,7 +2146,7 @@ export class LlEmitter {
|
||||
(options.targetTriple
|
||||
? /-linux-(?:gnu|musl)/.test(options.targetTriple)
|
||||
: process.platform === "linux")
|
||||
? unwindAbiFunctions(mod, this.mayThrow, options.unwindExcluded)
|
||||
? unwindAbiFunctions(mod, this.mayThrow, options.unwindExcluded, this.nodeIndex)
|
||||
: new Set();
|
||||
this.workerEntryPolls = mt.workerEntryPolls ?? null;
|
||||
this.stackChecks = mt.stackChecks ?? null;
|
||||
@@ -2221,6 +2239,7 @@ export class LlEmitter {
|
||||
(type.kind !== "union" ||
|
||||
this.nullableUnions.single(type.unionId)?.arm.kind !== "string"),
|
||||
this.walkBorrowsByFunction.get(fn.name) ?? new Set(),
|
||||
this.nodeIndex,
|
||||
);
|
||||
if (rebound.size === 0) continue;
|
||||
this.reboundByFunction.set(fn.name, rebound);
|
||||
@@ -3576,8 +3595,14 @@ export class LlEmitter {
|
||||
|
||||
/** Registers a SLOT whose current contents the frame owns (conditional
|
||||
* results: optional chains, branch joins that park ownership). */
|
||||
ownSlot(slot: string, type: IrType): void {
|
||||
if (isRefCounted(type)) this.currentFrame().push({ name: slot, type, slot: true });
|
||||
ownSlot(slot: string, type: IrType, rarelyFilled = false): void {
|
||||
if (isRefCounted(type))
|
||||
this.currentFrame().push({
|
||||
name: slot,
|
||||
type,
|
||||
slot: true,
|
||||
...(rarelyFilled ? { rarelyFilled: true } : {}),
|
||||
});
|
||||
}
|
||||
|
||||
/** The innermost lexical scope releases a frame object's reference
|
||||
@@ -3689,6 +3714,17 @@ export class LlEmitter {
|
||||
this.B.line(`${t} = load ptr, ptr ${v.name}`);
|
||||
if (v.boxed) {
|
||||
this.B.line(`call void ${boxReleaseSym(this.shapeHost)}(ptr ${t})`);
|
||||
} else if (v.rarelyFilled && !this.B.isTerminated()) {
|
||||
// Skip the release call on the common empty path.
|
||||
const filled = this.B.tmp();
|
||||
this.B.line(`${filled} = icmp ne ptr ${t}, null`);
|
||||
const release = this.B.newLabel("slot.release"),
|
||||
join = this.B.newLabel("slot.release.j");
|
||||
this.B.condBr(filled, release, join);
|
||||
this.B.startBlock(release);
|
||||
this.releaseFrame(t, v.type);
|
||||
this.B.br(join);
|
||||
this.B.startBlock(join);
|
||||
} else {
|
||||
this.releaseFrame(t, v.type);
|
||||
}
|
||||
@@ -6298,6 +6334,11 @@ export class LlEmitter {
|
||||
case "recordGet":
|
||||
case "recordKeyGet":
|
||||
return this.borrowsWithoutOwning(e) && this.unownedWalk(e.obj);
|
||||
// `r[k] ?? inherited(r, k)` whose default is immortal.
|
||||
case "nullish": {
|
||||
const read = borrowedRecordKeyGetOr(this, e);
|
||||
return read !== null && immortalDefault(this, e.right) && this.unownedWalk(read);
|
||||
}
|
||||
case "mapIntrinsic": {
|
||||
// A Map binding's own pointer (maps are not plainReference types).
|
||||
if (!this.borrowsWithoutOwning(e) || e.receiver.kind !== "varRef") return false;
|
||||
@@ -7574,6 +7615,8 @@ export class LlEmitter {
|
||||
return pointerRecordKeyGet(this, e);
|
||||
case "mapIntrinsic":
|
||||
return pointerMapGet(this, e);
|
||||
case "nullish":
|
||||
return borrowedRecordKeyGetOr(this, e) !== null;
|
||||
// emitReadReceiver lends a required element of a borrowed array when
|
||||
// computing the index removes no edge.
|
||||
case "arrayGet":
|
||||
|
||||
@@ -35,6 +35,8 @@ export interface LlValue {
|
||||
* convert it instead of the sum (expr-bytes.ts). */
|
||||
offsetIndex?: OffsetIndex;
|
||||
slot?: boolean;
|
||||
/** A frame slot that usually stays NULL: its release tests it first. */
|
||||
rarelyFilled?: boolean;
|
||||
/** A sequence local held in a capture box; release the box itself. */
|
||||
boxed?: boolean;
|
||||
/** A group arena list slot (arenas.ts): the frame closes the newest
|
||||
|
||||
@@ -1,7 +1,14 @@
|
||||
/* Focused LLVM expression emission extracted from emitter.ts. */
|
||||
import { InternalCompilerError } from "../../errors.js";
|
||||
import { undefinedArmTag, unionWideningTags } from "../../ir/analysis.js";
|
||||
import { type IrExpr, type IrType, isRefCounted, type SrcLoc, typeEquals } from "../../ir/ir.js";
|
||||
import {
|
||||
type IrExpr,
|
||||
type IrType,
|
||||
isRefCounted,
|
||||
isUnitType,
|
||||
type SrcLoc,
|
||||
typeEquals,
|
||||
} from "../../ir/ir.js";
|
||||
import { LlvmUnsupportedError } from "./unsupported.js";
|
||||
import type { LlvmEmitterContext, LlValue } from "./expr-context.js";
|
||||
import { f64Lit, llvmCommentText } from "./common.js";
|
||||
@@ -11,6 +18,7 @@ import { canReplaceWith, regexLiteralCompiles } from "./regex-matchers.js";
|
||||
import { isScalarUnionDef, loadInlineUnion, presentBits } from "./inline-unions.js";
|
||||
import { preservesRegexInputs } from "./reference-effects.js";
|
||||
import { emitNumberMapSlot, emitStringMapEntry } from "./map-number-lookup.js";
|
||||
import { everyStmtList } from "../../ir/traverse.js";
|
||||
|
||||
/** Widen one tagged read. The source is borrowed unless the overflow map
|
||||
* supplied an owned value. Each selected reference payload is retained
|
||||
@@ -490,6 +498,7 @@ export function pointerRecordKeyGet(host: LlvmEmitterContext, e: IrExpr): boolea
|
||||
export function emitBorrowedRecordKeyGet(
|
||||
host: LlvmEmitterContext,
|
||||
e: IrExpr & { kind: "recordKeyGet" },
|
||||
fallback: IrExpr | null = null,
|
||||
): LlValue {
|
||||
const B = host.B;
|
||||
if (e.type.kind !== "union")
|
||||
@@ -499,6 +508,15 @@ export function emitBorrowedRecordKeyGet(
|
||||
const key = host.canBorrowReceiver(e.key) ? host.emitReadReceiver(e.key) : host.emitExpr(e.key);
|
||||
const slot = B.slot();
|
||||
B.entryAllocas.push(`${slot} = alloca ptr`);
|
||||
// A default that needs an owner gets a frame slot, empty on every path
|
||||
// until a miss fills it.
|
||||
let owner: string | null = null;
|
||||
if (fallback !== null && !immortalDefault(host, fallback)) {
|
||||
owner = B.slot();
|
||||
B.entryAllocas.push(`${owner} = alloca ptr`);
|
||||
B.line(`store ptr null, ptr ${owner}`);
|
||||
host.ownSlot(owner, resultType, true);
|
||||
}
|
||||
const join = B.newLabel("rkb.j");
|
||||
const shape = host.recordShape(e.shapeId);
|
||||
for (const f of e.overflowOnly === true ? [] : shape.fields) {
|
||||
@@ -528,8 +546,23 @@ export function emitBorrowedRecordKeyGet(
|
||||
B.br(join);
|
||||
B.startBlock(ln);
|
||||
}
|
||||
const undefTag = undefinedArmTag(resultType, host.unionsById);
|
||||
B.line(`store ptr ${host.unitInstanceRef(resultType.unionId, undefTag)}, ptr ${slot}`);
|
||||
if (fallback !== null && immortalDefault(host, fallback)) {
|
||||
// An immortal default (an interned function value or undefined) needs
|
||||
// no owner at all.
|
||||
const value = host.emitExpr(fallback);
|
||||
host.toHeap(value);
|
||||
B.line(`store ptr ${value.name}, ptr ${slot}`);
|
||||
} else if (fallback !== null && owner !== null) {
|
||||
// `r[k] ?? d` (borrowedRecordKeyGetOr): the default is owned by a
|
||||
// statement-frame slot, which the consumer may borrow like a hit.
|
||||
const value = host.emitExpr(fallback);
|
||||
host.toHeap(value);
|
||||
B.line(`store ptr ${value.name}, ptr ${owner}`);
|
||||
B.line(`store ptr ${value.name}, ptr ${slot}`);
|
||||
} else {
|
||||
const undefTag = undefinedArmTag(resultType, host.unionsById);
|
||||
B.line(`store ptr ${host.unitInstanceRef(resultType.unionId, undefTag)}, ptr ${slot}`);
|
||||
}
|
||||
B.br(join);
|
||||
B.startBlock(join);
|
||||
const t = B.tmp();
|
||||
@@ -537,6 +570,54 @@ export function emitBorrowedRecordKeyGet(
|
||||
return { name: t, type: e.type };
|
||||
}
|
||||
|
||||
/** `r[k] ?? d` in its pass-through shape over a pointerRecordKeyGet whose
|
||||
* stored values are never unit (so only a miss evaluates `d`): the hit
|
||||
* lends the stored pointer and the default is frame-owned, so the result
|
||||
* borrows for the rest of the statement. */
|
||||
export function borrowedRecordKeyGetOr(
|
||||
host: LlvmEmitterContext,
|
||||
e: IrExpr,
|
||||
): (IrExpr & { kind: "recordKeyGet" }) | null {
|
||||
if (e.kind !== "nullish" || e.left.kind !== "recordKeyGet") return null;
|
||||
if (!typeEquals(e.type, e.left.type) || !pointerRecordKeyGet(host, e.left)) return null;
|
||||
const shape = host.recordShape(e.left.shapeId);
|
||||
const holdsUnit = (t: IrType): boolean =>
|
||||
isUnitType(t) ||
|
||||
t.kind === "dyn" ||
|
||||
(t.kind === "union" && (host.unionsById.get(t.unionId)?.arms ?? []).some(isUnitType));
|
||||
if (shape.indexValue !== undefined && holdsUnit(shape.indexValue)) return null;
|
||||
if (e.left.overflowOnly !== true && shape.fields.some((f) => holdsUnit(f.type))) return null;
|
||||
return e.left;
|
||||
}
|
||||
|
||||
/** The default is a call to a helper whose every result is immortal: the
|
||||
* unit arm or a closure over no captures (an interned function value), as
|
||||
* the unbound inherited-method reads return. */
|
||||
export function immortalDefault(host: LlvmEmitterContext, fallback: IrExpr): boolean {
|
||||
if (fallback.kind !== "call") return false;
|
||||
const fn = host.fnByName.get(fallback.callee);
|
||||
if (!fn || fn.async || fn.generator) return false;
|
||||
const immortal = (v: IrExpr | null): boolean => {
|
||||
if (v === null) return false;
|
||||
if (v.kind === "unitLit") return true;
|
||||
if (v.kind === "unionWrap") return immortal(v.value);
|
||||
if (v.kind === "closure") {
|
||||
const target = host.fnByName.get(v.fnName);
|
||||
return v.captures.length === 0 && target !== undefined && target.captures === undefined;
|
||||
}
|
||||
return false;
|
||||
};
|
||||
let ok = true;
|
||||
everyStmtList(fn.body, {
|
||||
stmt: (st) => {
|
||||
if (st.kind === "return" && !immortal(st.value)) ok = false;
|
||||
return ok;
|
||||
},
|
||||
expr: () => ok,
|
||||
});
|
||||
return ok;
|
||||
}
|
||||
|
||||
/** A keyed read whose result is a scalar union (`r[k]` on a
|
||||
* `Record<string, number>`, at `number | undefined`) and whose every hit
|
||||
* is a number or boolean arm of it, or an inline field of the same union.
|
||||
|
||||
@@ -45,6 +45,15 @@ function lastUse(
|
||||
const { extent, derived } = extentOf(stmts, index, localId, rebound);
|
||||
const last = extent[extent.length - 1];
|
||||
if (last === undefined) return null;
|
||||
return lastUseIn(last, localId, derived);
|
||||
}
|
||||
|
||||
/** The binding's read in `last`, the last statement of its extent, whose derived bindings are `derived`. */
|
||||
function lastUseIn(
|
||||
last: IrStmt,
|
||||
localId: string,
|
||||
derived: ReadonlySet<string>,
|
||||
): { read: IrExpr; stored: boolean } | null {
|
||||
const state: { read: IrExpr | null; stored: boolean; ok: boolean } = {
|
||||
read: null,
|
||||
stored: false,
|
||||
@@ -183,6 +192,109 @@ function collectAssignedExpr(e: IrExpr, assigned: Set<string>): boolean {
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* `storedLastUseRead(stmts, -1, localId, true)` for every parameter of one function body, from one walk of the
|
||||
* body instead of a walk per binding. With `index` -1 and `rebound`, no binding is excluded as assigned, so a
|
||||
* binding is derived when one of its definitions mentions a derived binding (reachability over "definition
|
||||
* mentions") and the extent ends at the last statement that mentions one (`absentStmt`). The walk records, for
|
||||
* each top-level statement, the bindings it mentions, and for each definition, the bindings its value mentions.
|
||||
*/
|
||||
export class BodyLastUses {
|
||||
private readonly stmts: readonly IrStmt[];
|
||||
/** The bindings each top-level statement mentions, as `absentStmt` reads them. */
|
||||
private readonly mentions: Set<string>[] = [];
|
||||
/** For each binding, the bindings whose definitions mention it. */
|
||||
private readonly definedFrom = new Map<string, string[]>();
|
||||
|
||||
constructor(stmts: readonly IrStmt[]) {
|
||||
this.stmts = stmts;
|
||||
for (const s of stmts) {
|
||||
const mentioned = new Set<string>();
|
||||
this.stmt(s, mentioned);
|
||||
this.mentions.push(mentioned);
|
||||
}
|
||||
}
|
||||
|
||||
/** What `storedLastUseRead(stmts, -1, localId, true)` answers. */
|
||||
storedRead(localId: string): IrExpr | null {
|
||||
const derived = new Set<string>([localId]);
|
||||
const queue = [localId];
|
||||
while (queue.length > 0) {
|
||||
for (const target of this.definedFrom.get(queue.pop()!) ?? [])
|
||||
if (!derived.has(target)) {
|
||||
derived.add(target);
|
||||
queue.push(target);
|
||||
}
|
||||
}
|
||||
let last = -1;
|
||||
for (let i = this.mentions.length - 1; i >= 0 && last < 0; i--)
|
||||
for (const id of derived)
|
||||
if (this.mentions[i]!.has(id)) {
|
||||
last = i;
|
||||
break;
|
||||
}
|
||||
if (last < 0) return null;
|
||||
const use = lastUseIn(this.stmts[last]!, localId, derived);
|
||||
return use !== null && use.stored ? use.read : null;
|
||||
}
|
||||
|
||||
/** Adds the bindings `s` mentions to `into`, and records the definitions in it. */
|
||||
private stmt(s: IrStmt, into: Set<string>): boolean {
|
||||
if (s.kind === "varDecl" || s.kind === "assign") into.add(s.localId);
|
||||
if (s.kind === "varDecl" || s.kind === "forOf" || s.kind === "assign") {
|
||||
const value = s.kind === "varDecl" ? s.init : s.kind === "assign" ? s.value : s.iterable;
|
||||
if (value !== null) this.define(s.localId, value);
|
||||
}
|
||||
everyStmtChild(
|
||||
s,
|
||||
(e) => this.expr(e, into),
|
||||
(c) => this.stmt(c, into),
|
||||
);
|
||||
return true;
|
||||
}
|
||||
|
||||
private expr(e: IrExpr, into: Set<string>): boolean {
|
||||
if (e.kind === "varRef" || e.kind === "assignExpr" || e.kind === "incDec") into.add(e.localId);
|
||||
else if (e.kind === "closure" || e.kind === "classRef")
|
||||
for (const id of e.captures ?? []) into.add(id);
|
||||
if (e.kind === "assignExpr") this.define(e.localId, e.value);
|
||||
everyExprChild(
|
||||
e,
|
||||
(c) => this.expr(c, into),
|
||||
(c) => this.stmt(c, into),
|
||||
);
|
||||
return true;
|
||||
}
|
||||
|
||||
/** A definition of `target` whose value is `value`: every binding the value mentions derives `target`. */
|
||||
private define(target: string, value: IrExpr): void {
|
||||
const mentioned = new Set<string>();
|
||||
mentionsOf(value, mentioned);
|
||||
for (const id of mentioned) {
|
||||
let targets = this.definedFrom.get(id);
|
||||
if (targets === undefined) this.definedFrom.set(id, (targets = []));
|
||||
targets.push(target);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** The bindings an expression mentions, as `absentExpr` reads them. */
|
||||
function mentionsOf(e: IrExpr, into: Set<string>): void {
|
||||
const expr = (x: IrExpr): boolean => {
|
||||
if (x.kind === "varRef" || x.kind === "assignExpr" || x.kind === "incDec") into.add(x.localId);
|
||||
else if (x.kind === "closure" || x.kind === "classRef")
|
||||
for (const id of x.captures ?? []) into.add(id);
|
||||
everyExprChild(x, expr, stmt);
|
||||
return true;
|
||||
};
|
||||
const stmt = (s: IrStmt): boolean => {
|
||||
if (s.kind === "varDecl" || s.kind === "assign") into.add(s.localId);
|
||||
everyStmtChild(s, expr, stmt);
|
||||
return true;
|
||||
};
|
||||
expr(e);
|
||||
}
|
||||
|
||||
/** True while no node of the tree mentions a derived binding. */
|
||||
function absentExpr(e: IrExpr, derived: ReadonlySet<string>): boolean {
|
||||
if (e.kind === "varRef" || e.kind === "assignExpr" || e.kind === "incDec") {
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
import type { IrExpr, IrFunction, IrStmt, IrType } from "../../ir/ir.js";
|
||||
import { everyExprChild, everyStmtChild } from "../../ir/traverse.js";
|
||||
import { eachNodeOfKinds, type IrNodeIndex } from "../../ir/node-index.js";
|
||||
|
||||
/** Assignment statements: only these rebind a parameter. */
|
||||
const ASSIGN_KINDS = ["assign"] as const;
|
||||
|
||||
/** Parameters that keep the borrowed calling convention although the body
|
||||
* rebinds them (`source = (source as LiteralType).regularType`). The
|
||||
@@ -15,6 +19,8 @@ export function findReboundParameters(
|
||||
fn: IrFunction,
|
||||
plainReference: (type: IrType) => boolean,
|
||||
excluded: ReadonlySet<string>,
|
||||
/** The module's node index: a body that assigns no candidate parameter is not walked. */
|
||||
nodes?: IrNodeIndex,
|
||||
): ReadonlySet<string> {
|
||||
const result = new Set<string>();
|
||||
if (fn.async || fn.generator || fn.captures || fn.classCaptures) return result;
|
||||
@@ -34,6 +40,15 @@ export function findReboundParameters(
|
||||
.map((param) => param.localId),
|
||||
);
|
||||
if (candidates.size === 0) return result;
|
||||
// A parameter is rebound only by an assignment statement to it.
|
||||
if (nodes !== undefined) {
|
||||
let assigned = false;
|
||||
eachNodeOfKinds(fn, nodes.scan(ASSIGN_KINDS), (node) => {
|
||||
if (node.kind === "assign" && candidates.has(node.localId)) assigned = true;
|
||||
return !assigned;
|
||||
});
|
||||
if (!assigned) return result;
|
||||
}
|
||||
const rebound = new Set<string>();
|
||||
const invalid = new Set<string>();
|
||||
const expr = (e: IrExpr): boolean => {
|
||||
|
||||
@@ -24,6 +24,7 @@ import { emitDowncastGuard, emitUnionNarrowGuard } from "./narrow-guards.js";
|
||||
import { emitScalarOptional, isScalarOptionalSource } from "./local-union-storage.js";
|
||||
import { borrowableInputs } from "./borrowed-inputs.js";
|
||||
import {
|
||||
borrowedRecordKeyGetOr,
|
||||
emitBorrowedRecordKeyGet,
|
||||
emitStackRecordKeyGet,
|
||||
pointerRecordKeyGet,
|
||||
@@ -104,6 +105,10 @@ export function emitReadReceiver(emitter: LlEmitter, e: IrExpr): LlValue {
|
||||
if (arrayRead) return emitProjectedArrayRead(emitter, arrayRead);
|
||||
if (e.kind === "recordKeyGet" && emitter.canBorrowReceiver(e))
|
||||
return emitBorrowedRecordKeyGet(emitter, e);
|
||||
if (e.kind === "nullish" && emitter.canBorrowReceiver(e)) {
|
||||
const read = borrowedRecordKeyGetOr(emitter, e);
|
||||
if (read) return emitBorrowedRecordKeyGet(emitter, read, e.right);
|
||||
}
|
||||
if (e.kind === "mapIntrinsic" && emitter.canBorrowReceiver(e))
|
||||
return emitBorrowedMapGet(emitter, e);
|
||||
const narrow = emitter.checkedNarrows.get(e);
|
||||
@@ -160,6 +165,12 @@ export function emitReadReceiver(emitter: LlEmitter, e: IrExpr): LlValue {
|
||||
no = B.newLabel("read.else"),
|
||||
join = B.newLabel("read.join");
|
||||
B.condBr(cond.name, yes, no);
|
||||
// Slots an arm's borrowed read fills (the owner of a `r[k] ?? d`
|
||||
// default) may hold what the result points to: the enclosing frame
|
||||
// releases them after the use, and the other arm's path leaves them
|
||||
// empty so that release skips them.
|
||||
const armSlots: LlValue[][] = [];
|
||||
const ends: string[] = [];
|
||||
for (const [label, value] of [
|
||||
[yes, e.then],
|
||||
[no, e.else_],
|
||||
@@ -168,9 +179,21 @@ export function emitReadReceiver(emitter: LlEmitter, e: IrExpr): LlValue {
|
||||
emitter.frames.push([]);
|
||||
const result = emitter.emitReadReceiver(value);
|
||||
B.line(`store ptr ${result.name}, ptr ${slot}`);
|
||||
emitter.releaseTemps(emitter.frames.pop()!);
|
||||
const frame = emitter.frames.pop()!;
|
||||
armSlots.push(frame.filter((entry) => entry.slot === true && entry.boxed !== true));
|
||||
emitter.releaseTemps(frame.filter((entry) => entry.slot !== true || entry.boxed === true));
|
||||
const end = B.newLabel("read.end");
|
||||
ends.push(end);
|
||||
B.br(end);
|
||||
}
|
||||
for (const [i, end] of ends.entries()) {
|
||||
B.startBlock(end);
|
||||
for (const entry of armSlots[1 - i]!) B.line(`store ptr null, ptr ${entry.name}`);
|
||||
B.br(join);
|
||||
}
|
||||
for (const entries of armSlots)
|
||||
for (const entry of entries)
|
||||
emitter.ownSlot(entry.name, entry.type, entry.rarelyFilled === true);
|
||||
B.startBlock(join);
|
||||
const value = B.tmp();
|
||||
B.line(`${value} = load ptr, ptr ${slot}`);
|
||||
@@ -315,6 +338,12 @@ export function canBorrowReceiver(emitter: LlEmitter, e: IrExpr): boolean {
|
||||
emitter.canBorrowReceiver(e.obj) &&
|
||||
emitter.referenceEffects.preserves(e.key)
|
||||
);
|
||||
// `r[k] ?? inherited(r, k)` lends a hit the same way; its default is
|
||||
// owned by the statement frame (so never by a longer-lived alias).
|
||||
case "nullish": {
|
||||
const read = borrowedRecordKeyGetOr(emitter, e);
|
||||
return read !== null && emitter.canBorrowReceiver(read);
|
||||
}
|
||||
// So does a Map.get while the map keeps it.
|
||||
case "mapIntrinsic":
|
||||
return (
|
||||
|
||||
@@ -930,6 +930,10 @@ export class ReferenceWrites {
|
||||
out.push(ANY_RECORD_KEY);
|
||||
for (const f of this.records.get(node.shapeId)?.fields ?? []) out.push(`r:${f.name}`);
|
||||
return this.expr(node.key) === null && walk(node.obj);
|
||||
case "nullish":
|
||||
// `r[k] ?? d`: a hit is the read's own value; the default runs only
|
||||
// on a miss and is owned apart from any location.
|
||||
return walk(node.left);
|
||||
case "mapIntrinsic":
|
||||
// A typed Map.get: any entry of any map.
|
||||
if (node.method !== "get" || node.args.length !== 1 || !borrowsMapReadInputs(node))
|
||||
|
||||
@@ -1,8 +1,14 @@
|
||||
/* The functions that propagate exceptions by unwinding instead of the
|
||||
* pending flag (see unwindAbiFunctions). */
|
||||
import type { IrExpr, IrModule } from "../../ir/ir.js";
|
||||
import { everyStmtList } from "../../ir/traverse.js";
|
||||
import type { IrExpr, IrModule, IrStmt } from "../../ir/ir.js";
|
||||
import { eachStringField } from "../../ir/fields.generated.js";
|
||||
import { eachNodeOfKinds, type IrNodeIndex } from "../../ir/node-index.js";
|
||||
import { InternalCompilerError } from "../../errors.js";
|
||||
|
||||
/** The kinds of every recorded node, for eachNodeOfKinds over whole bodies. */
|
||||
const EVERY_KIND: ReadonlySet<string> = new Set();
|
||||
/** Virtual calls, which name the methods a vtable must reach. */
|
||||
const VIRTUAL_CALL_KINDS = ["virtualCall"] as const;
|
||||
|
||||
/** How many calls above a `throw` functions keep the pending flag. */
|
||||
const THROW_LEVELS = 3;
|
||||
@@ -80,6 +86,8 @@ export function unwindAbiFunctions(
|
||||
mod: IrModule,
|
||||
mayThrow: ReadonlySet<string>,
|
||||
excluded: ReadonlySet<string> = new Set(),
|
||||
/** The module's node index: the bodies' nodes are read from it, in walk order, without walking. */
|
||||
nodes?: IrNodeIndex,
|
||||
): Set<string> {
|
||||
const names = new Set(mod.functions.map((fn) => fn.name));
|
||||
const blocked = new Set<string>(excluded);
|
||||
@@ -97,16 +105,16 @@ export function unwindAbiFunctions(
|
||||
// class's method of that name.
|
||||
const classes = new Map((mod.classes ?? []).map((cls) => [cls.name, cls]));
|
||||
const anyClass = new Set<string>();
|
||||
const virtualCalls = nodes?.scan(VIRTUAL_CALL_KINDS);
|
||||
for (const fn of mod.functions)
|
||||
everyStmtList(fn.body, {
|
||||
expr: (e: IrExpr) => {
|
||||
if (e.kind !== "virtualCall") return true;
|
||||
if (e.targets !== undefined) for (const target of e.targets) blocked.add(target);
|
||||
else if (classes.has(e.className)) blocked.add(`%${e.className}.${e.method}`);
|
||||
else anyClass.add(e.method);
|
||||
return true;
|
||||
},
|
||||
stmt: () => true,
|
||||
// Without the index (or for a function it didn't record) every node
|
||||
// arrives, statements included: the kind test narrows, not a cast.
|
||||
eachNodeOfKinds(fn, virtualCalls, (node) => {
|
||||
if (node.kind !== "virtualCall") return true;
|
||||
if (node.targets !== undefined) for (const target of node.targets) blocked.add(target);
|
||||
else if (classes.has(node.className)) blocked.add(`%${node.className}.${node.method}`);
|
||||
else anyClass.add(node.method);
|
||||
return true;
|
||||
});
|
||||
// A hierarchy's vtables hold the methods declared by more than one of its
|
||||
// classes (an override and what it overrides).
|
||||
@@ -161,9 +169,10 @@ export function unwindAbiFunctions(
|
||||
if (list === undefined) methodsNamed.set(m, (list = []));
|
||||
list.push(`%${cls.name}.${m}`);
|
||||
}
|
||||
const everyNode = nodes?.scanExcept(EVERY_KIND);
|
||||
for (const fn of mod.functions) {
|
||||
const coroutine = fn.async === true || fn.generator !== undefined;
|
||||
everyStmtList(fn.body, {
|
||||
const visit = {
|
||||
expr: (e: IrExpr) => {
|
||||
const kind = e.kind as string;
|
||||
if (kind === "throw" || kind === "rethrow") throws.add(fn.name);
|
||||
@@ -192,13 +201,16 @@ export function unwindAbiFunctions(
|
||||
}
|
||||
return true;
|
||||
},
|
||||
stmt: (s) => {
|
||||
stmt: (s: IrStmt) => {
|
||||
const kind = s.kind as string;
|
||||
if (kind === "throw" || kind === "rethrow") throws.add(fn.name);
|
||||
eachStringField(s, noteName);
|
||||
return true;
|
||||
},
|
||||
});
|
||||
};
|
||||
eachNodeOfKinds(fn, everyNode, (node, statement) =>
|
||||
statement ? visit.stmt(node as IrStmt) : visit.expr(node as IrExpr),
|
||||
);
|
||||
}
|
||||
for (const name of referenced) {
|
||||
blocked.add(name);
|
||||
@@ -228,6 +240,13 @@ export function unwindAbiFunctions(
|
||||
if (fn.async === true || fn.generator !== undefined) continue;
|
||||
result.add(fn.name);
|
||||
}
|
||||
if (nodes !== undefined && process.env["SCRIPTC_EMIT_FACTS_VERIFY"] === "1") {
|
||||
const walked = unwindAbiFunctions(mod, mayThrow, excluded);
|
||||
if (walked.size !== result.size || [...walked].some((name) => !result.has(name)))
|
||||
throw new InternalCompilerError(
|
||||
"unwind ABI: the functions read from the node index differ from a walk's",
|
||||
);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
|
||||
@@ -14,20 +14,32 @@ const PROGRAM_PARTITION_BYTES = 4 * 1024 * 1024;
|
||||
const MIN_PARTITION_BYTES = 512 * 1024;
|
||||
const MIN_PROGRAM_PARTITIONS = 8;
|
||||
const MAX_PROGRAM_PARTITIONS = 64;
|
||||
/** SINGLE_PARTITION_BYTES and MIN_PARTITION_BYTES of unoptimized (dev)
|
||||
* executables. */
|
||||
const UNOPTIMIZED_SINGLE_PARTITION_BYTES = 64 * 1024;
|
||||
const UNOPTIMIZED_MIN_PARTITION_BYTES = 32 * 1024;
|
||||
|
||||
/** Executable objects larger than SINGLE_PARTITION_BYTES are divided into
|
||||
* partitions that the helper simplifies, optimizes and compiles concurrently,
|
||||
* importing small functions across partitions for inlining, and reuses
|
||||
* individually across builds. A divided module gets at least
|
||||
* MIN_PROGRAM_PARTITIONS so that mid-size programs use several cores. The
|
||||
* count is a power of two that depends only on the module's size, on every
|
||||
* target, so artifacts never vary with the host and ordinary edits keep every
|
||||
* function in its partition. */
|
||||
export function nativeProgramPartitions(llvmBytes: number): number {
|
||||
if (llvmBytes <= SINGLE_PARTITION_BYTES) return 1;
|
||||
* count is a power of two that depends only on the module's size and
|
||||
* whether it is optimized, on every target, so artifacts never vary with the
|
||||
* host and ordinary edits keep every function in its partition.
|
||||
*
|
||||
* Unoptimized (dev) partitions are compiled on their own with nothing
|
||||
* imported, so they cost no run time and an edit compiles only the partitions
|
||||
* whose functions it changes: dev executables are divided from
|
||||
* UNOPTIMIZED_SINGLE_PARTITION_BYTES, at least
|
||||
* UNOPTIMIZED_MIN_PARTITION_BYTES each. */
|
||||
export function nativeProgramPartitions(llvmBytes: number, unoptimized = false): number {
|
||||
const single = unoptimized ? UNOPTIMIZED_SINGLE_PARTITION_BYTES : SINGLE_PARTITION_BYTES;
|
||||
const least = unoptimized ? UNOPTIMIZED_MIN_PARTITION_BYTES : MIN_PARTITION_BYTES;
|
||||
if (llvmBytes <= single) return 1;
|
||||
const required = Math.max(
|
||||
Math.ceil(llvmBytes / PROGRAM_PARTITION_BYTES),
|
||||
Math.min(MIN_PROGRAM_PARTITIONS, Math.ceil(llvmBytes / MIN_PARTITION_BYTES)),
|
||||
Math.min(MIN_PROGRAM_PARTITIONS, Math.ceil(llvmBytes / least)),
|
||||
);
|
||||
return Math.min(MAX_PROGRAM_PARTITIONS, 2 ** Math.ceil(Math.log2(required)));
|
||||
}
|
||||
|
||||
@@ -408,6 +408,14 @@ test("program partitions depend only on module size", () => {
|
||||
expect(nativeProgramPartitions(32 * megabyte + 1)).toBe(16);
|
||||
expect(nativeProgramPartitions(100 * megabyte)).toBe(32);
|
||||
expect(nativeProgramPartitions(1024 * megabyte)).toBe(64);
|
||||
// Unoptimized (dev) programs are divided from 64 KiB.
|
||||
const kilobyte = 1024;
|
||||
expect(nativeProgramPartitions(64 * kilobyte, true)).toBe(1);
|
||||
expect(nativeProgramPartitions(64 * kilobyte + 1, true)).toBe(4);
|
||||
expect(nativeProgramPartitions(128 * kilobyte, true)).toBe(4);
|
||||
expect(nativeProgramPartitions(128 * kilobyte + 1, true)).toBe(8);
|
||||
expect(nativeProgramPartitions(megabyte, true)).toBe(8);
|
||||
expect(nativeProgramPartitions(32 * megabyte + 1, true)).toBe(16);
|
||||
expect(nativePartitionPaths("/build.d/program.o", 3)).toEqual([
|
||||
"/build.d/program.o",
|
||||
"/build.d/program.part1.o",
|
||||
|
||||
@@ -21,6 +21,13 @@ export const WEB_STREAM_CLASSES = [
|
||||
"ReadableStreamDefaultController",
|
||||
"CountQueuingStrategy",
|
||||
"ByteLengthQueuingStrategy",
|
||||
"WritableStream",
|
||||
"WritableStreamDefaultWriter",
|
||||
"WritableStreamDefaultController",
|
||||
"TransformStream",
|
||||
"TransformStreamDefaultController",
|
||||
"TextEncoderStream",
|
||||
"TextDecoderStream",
|
||||
] as const;
|
||||
|
||||
const CLASS_NAMES: readonly string[] = WEB_STREAM_CLASSES;
|
||||
@@ -48,6 +55,33 @@ export function webStreamClassKindOf(lowerer: Lowerer, node: ts.Expression): num
|
||||
return index < 0 ? null : index;
|
||||
}
|
||||
|
||||
/** Whether a JavaScript source of the program names one of the classes.
|
||||
* The global object answers them by a computed name only then
|
||||
* (global-values.ts), so a program without web streams does not link
|
||||
* them for its other computed global reads. */
|
||||
export function programNamesWebStreamClass(lowerer: Lowerer): boolean {
|
||||
if (lowerer.namesWebStreamClass !== undefined) return lowerer.namesWebStreamClass;
|
||||
const names: ReadonlySet<string> = new Set(WEB_STREAM_CLASSES);
|
||||
let found = false;
|
||||
for (const file of lowerer.program.getSourceFiles()) {
|
||||
if (file.isDeclarationFile || !isJsSourceFile(file)) continue;
|
||||
ts.walkPreorder(file, (node) => {
|
||||
if (
|
||||
ts.isIdentifier(node) &&
|
||||
names.has(node.text) &&
|
||||
webStreamClassKindOf(lowerer, node) !== null
|
||||
) {
|
||||
found = true;
|
||||
return "stop";
|
||||
}
|
||||
return undefined;
|
||||
});
|
||||
if (found) break;
|
||||
}
|
||||
lowerer.namesWebStreamClass = found;
|
||||
return found;
|
||||
}
|
||||
|
||||
export function webStreamClassValue(kind: number, loc: SrcLoc): IrExpr {
|
||||
return { kind: "libCall", fn: "webstream.class", args: [numLit(kind, loc)], type: DYN, loc };
|
||||
}
|
||||
@@ -130,6 +164,8 @@ const WEB_STREAM_INTERFACES: ReadonlySet<string> = new Set([
|
||||
"ReadableStreamReadValueResult",
|
||||
"ReadableStreamReadDoneResult",
|
||||
"UnderlyingSource",
|
||||
"UnderlyingSink",
|
||||
"Transformer",
|
||||
"QueuingStrategy",
|
||||
"QueuingStrategySize",
|
||||
]);
|
||||
|
||||
@@ -32,6 +32,11 @@ import { builtinNamespaceValue } from "./builtin-namespaces.js";
|
||||
import { uriFunctionValue } from "./expressions/identifiers.js";
|
||||
import { navigatorValue } from "./builtins/navigator.js";
|
||||
import { primitiveCtorClosure } from "./lower-exprs.js";
|
||||
import {
|
||||
WEB_STREAM_CLASSES,
|
||||
programNamesWebStreamClass,
|
||||
webStreamClassValue,
|
||||
} from "./builtins/web-streams-dyn.js";
|
||||
|
||||
const LOOKUP_FN = "%global.builtinValue";
|
||||
|
||||
@@ -177,9 +182,14 @@ export function globalValuesInstall(
|
||||
const name: IrExpr = { kind: "varRef", localId: "name.0", type: STRING, loc };
|
||||
const body: IrStmt[] = [];
|
||||
for (const global of [...new Set(names)].sort()) {
|
||||
const webStream = (WEB_STREAM_CLASSES as readonly string[]).indexOf(global);
|
||||
const value: IrExpr | null = REFUSED.has(global)
|
||||
? { kind: "unitLit", unit: "null", type: { kind: "nullT" }, loc }
|
||||
: jsGlobalValueByName(lowerer, global, loc);
|
||||
: webStream >= 0
|
||||
? programNamesWebStreamClass(lowerer)
|
||||
? webStreamClassValue(webStream, loc)
|
||||
: null
|
||||
: jsGlobalValueByName(lowerer, global, loc);
|
||||
if (value === null) continue;
|
||||
body.push({
|
||||
kind: "if",
|
||||
|
||||
@@ -3659,6 +3659,9 @@ export class Lowerer {
|
||||
/** global-values.ts's answer per JavaScript source: the receiver of its
|
||||
* first read of the global object by a computed name, or null. */
|
||||
readonly computedGlobalReads = new Map<ts.SourceFile, ts.Expression | null>();
|
||||
/** web-streams-dyn.ts's answer: whether a JavaScript source names a web
|
||||
* stream class; undefined until asked. */
|
||||
namesWebStreamClass: boolean | undefined = undefined;
|
||||
/** The first lifted function of each name (liftedFnNamed), for the
|
||||
* liftedFns indexed so far. liftedFns only grows. */
|
||||
private readonly liftedFnsByName = new Map<string, IrFunction>();
|
||||
@@ -4275,6 +4278,7 @@ export class Lowerer {
|
||||
"createRequireBindingTargets",
|
||||
"createRequireElementTargets",
|
||||
"computedGlobalReads",
|
||||
"namesWebStreamClass",
|
||||
"timeValueDateDecisions",
|
||||
"timeValueDateWrites",
|
||||
"dynamicOverloadAnswers",
|
||||
|
||||
@@ -499,7 +499,8 @@ function builtinAnswer(
|
||||
type: BOOL,
|
||||
loc,
|
||||
});
|
||||
if (arg === undefined || !mayBeSymbol(arg.type)) return has(propertyKeyOf(lowerer, arg, loc));
|
||||
if (arg === undefined || !mayBeSymbol(lowerer, arg.type))
|
||||
return has(propertyKeyOf(lowerer, arg, loc));
|
||||
const id = declare("%protoKey", DYN);
|
||||
const held = varRef(id, DYN, loc);
|
||||
return {
|
||||
@@ -520,13 +521,16 @@ function builtinAnswer(
|
||||
}
|
||||
}
|
||||
|
||||
/** Whether a value of `type` may be a symbol at run time. */
|
||||
function mayBeSymbol(type: IrType): boolean {
|
||||
return !(
|
||||
isUnitType(type) ||
|
||||
type.kind === "string" ||
|
||||
type.kind === "f64" ||
|
||||
type.kind === "bool"
|
||||
/** Whether a value of `type` may be a symbol at run time: a checked value
|
||||
* or a union with a symbol or checked arm. Records, class instances,
|
||||
* arrays and functions never are, so their keys skip the symbol test (a
|
||||
* key that converts without running code leaves the method's impl no heap
|
||||
* writer to ReferenceWrites). */
|
||||
function mayBeSymbol(lowerer: Lowerer, type: IrType): boolean {
|
||||
if (type.kind === "dyn" || type.kind === "symbol") return true;
|
||||
if (type.kind !== "union") return false;
|
||||
return (lowerer.unions.get(type.unionId)?.arms ?? []).some(
|
||||
(arm) => arm.kind === "dyn" || arm.kind === "symbol",
|
||||
);
|
||||
}
|
||||
|
||||
@@ -540,9 +544,29 @@ function propertyKeyOf(lowerer: Lowerer, arg: IrExpr | undefined, loc: SrcLoc):
|
||||
type: STRING,
|
||||
loc,
|
||||
};
|
||||
if (arg.type.kind === "string") return arg;
|
||||
if (arg.type.kind === "f64" || arg.type.kind === "bool")
|
||||
const type = arg.type;
|
||||
if (type.kind === "string") return arg;
|
||||
if (type.kind === "f64" || type.kind === "bool")
|
||||
return { kind: "toString", operand: arg, type: STRING, loc };
|
||||
// Unions of scalars convert through the union's own ToString.
|
||||
if (
|
||||
type.kind === "union" &&
|
||||
(lowerer.unions.get(type.unionId)?.arms ?? []).every(
|
||||
(arm) =>
|
||||
isUnitType(arm) || arm.kind === "string" || arm.kind === "f64" || arm.kind === "bool",
|
||||
)
|
||||
)
|
||||
return { kind: "toString", operand: arg, type: STRING, loc };
|
||||
// A record or class instance that answers Object.prototype.toString
|
||||
// spells "[object Object]" without running any code.
|
||||
if (
|
||||
(type.kind === "record" && defaultRecordToString(lowerer, type.shapeId)) ||
|
||||
(type.kind === "object" && defaultToString(lowerer, type.className))
|
||||
)
|
||||
return { kind: "strLit", value: "[object Object]", type: STRING, loc };
|
||||
// Any other key (an array, a function, a record or class instance with
|
||||
// its own toString, a checked value) converts as ToPropertyKey does,
|
||||
// through the dynamic tier: the conversion may run the key's code.
|
||||
return {
|
||||
kind: "libCall",
|
||||
fn: "dyn.toStringCoerce",
|
||||
@@ -552,6 +576,29 @@ function propertyKeyOf(lowerer: Lowerer, arg: IrExpr | undefined, loc: SrcLoc):
|
||||
};
|
||||
}
|
||||
|
||||
/** A record shape whose values answer Object.prototype.toString: no
|
||||
* `toString` field, and no index signature that may hold one at run time
|
||||
* (or give the record a null prototype). */
|
||||
function defaultRecordToString(lowerer: Lowerer, shapeId: string): boolean {
|
||||
const shape = lowerer.shapes.get(shapeId);
|
||||
return (
|
||||
shape !== undefined &&
|
||||
shape.indexValue === undefined &&
|
||||
!shape.fields.some((field) => field.name === "toString")
|
||||
);
|
||||
}
|
||||
|
||||
/** A source class whose instances answer Object.prototype.toString. */
|
||||
function defaultToString(lowerer: Lowerer, className: string): boolean {
|
||||
const info = lowerer.classes.get(className);
|
||||
return (
|
||||
info !== undefined &&
|
||||
info.decl !== undefined &&
|
||||
lowerer.findMethodOn(info, "toString") === null &&
|
||||
!lowerer.overrideBelow(info, "toString")
|
||||
);
|
||||
}
|
||||
|
||||
/** `value` returned as `ret`: as is, wrapped into a union arm or a checked
|
||||
* value, discarded for void, or the checked-conversion TypeError. */
|
||||
function returnAs(lowerer: Lowerer, value: IrExpr, ret: IrType, loc: SrcLoc): IrStmt[] {
|
||||
|
||||
@@ -147,20 +147,27 @@ test("a map table hands over the entries added after a size", () => {
|
||||
expect([...map]).toEqual([["a", "%h.1"]]);
|
||||
});
|
||||
|
||||
test("a held table notices a replaced entry", () => {
|
||||
test("a table's content snapshot notices changed, reordered and removed entries", () => {
|
||||
const map = new Map<string, string>([["a", "%h.1"]]);
|
||||
const held = mapTable("helpers", map).hold!();
|
||||
const held = mapTable("helpers", map).contents!();
|
||||
map.set("b", "%h.2");
|
||||
expect(held.same()).toBe(true);
|
||||
expect(held.changed()).toBeNull();
|
||||
map.set("a", "%h.9");
|
||||
expect(held.same()).toBe(false);
|
||||
const set = new Set(["x"]);
|
||||
const heldSet = setTable("names", set).hold!();
|
||||
expect(held.changed()).toBe("a");
|
||||
// A unit names the helper it got, not the key it was looked up by.
|
||||
expect(held.changedAmong(new Set(["%h.1"]))).toBe("a");
|
||||
expect(held.changedAmong(new Set(["%h.2"]))).toBeNull();
|
||||
held.refresh(new Set(["%h.1"]));
|
||||
expect(held.changed()).toBeNull();
|
||||
const set = new Set(["x", "y"]);
|
||||
const heldSet = setTable("names", set).contents!();
|
||||
set.delete("x");
|
||||
set.add("x");
|
||||
expect(heldSet.same()).toBe(true);
|
||||
expect(heldSet.changed()).toBe("x");
|
||||
set.delete("x");
|
||||
expect(heldSet.same()).toBe(false);
|
||||
expect(heldSet.changedAmong(new Set(["x"]))).toBe("x");
|
||||
heldSet.refresh(new Set(["x"]));
|
||||
expect(heldSet.changed()).toBeNull();
|
||||
});
|
||||
|
||||
test("a function table keeps the function an earlier unit made", () => {
|
||||
@@ -280,7 +287,7 @@ test("a unit that changes an entry it did not add has no record", () => {
|
||||
state.helpers.set("shared", "%h.first");
|
||||
const unit = recordUnit(subject, "a");
|
||||
expect(unit.record).toBeNull();
|
||||
expect(unit.reasons).toEqual(["helpers:replaced"]);
|
||||
expect(unit.reasons).toEqual(["helpers:changed"]);
|
||||
});
|
||||
|
||||
test("a recorded wave throws when a unit changes a held function in place", () => {
|
||||
|
||||
@@ -96,8 +96,6 @@ export interface JournalTable {
|
||||
readonly digest?: (key: string, before: number) => string | null;
|
||||
/** Removes the entries from position `from` on. */
|
||||
readonly truncate?: (from: number) => void;
|
||||
/** The entries the table holds now. */
|
||||
readonly hold?: () => HeldEntries;
|
||||
/** The content of the entries the table holds now, for tables whose
|
||||
* values are objects a unit could change in place. */
|
||||
readonly contents?: () => HeldContents;
|
||||
@@ -127,10 +125,13 @@ export function contentDigest(text: string): string {
|
||||
return `${text.length}:${(a >>> 0).toString(16)}${(b >>> 0).toString(16)}`;
|
||||
}
|
||||
|
||||
/** The strings quoted in JSON `texts`, unescaped only as far as their
|
||||
* delimiting quotes: the names a unit's encoded IR and entries refer to. */
|
||||
/** The names quoted in JSON `texts`, unescaped only as far as their
|
||||
* delimiting quotes: the strings that start with `%` (helper, lifted and
|
||||
* IR data names) and union ids (`u` and 12 hex digits), which the tables
|
||||
* a unit reads and changes are keyed by. */
|
||||
export function quotedStrings(texts: readonly string[]): Set<string> {
|
||||
const out = new Set<string>();
|
||||
const hex = (char: number): boolean => (char >= 48 && char <= 57) || (char >= 97 && char <= 102);
|
||||
for (const text of texts) {
|
||||
let start = -1;
|
||||
for (let index = 0; index < text.length; index++) {
|
||||
@@ -139,7 +140,13 @@ export function quotedStrings(texts: readonly string[]): Set<string> {
|
||||
else if (char === 34 /* " */) {
|
||||
if (start < 0) start = index + 1;
|
||||
else {
|
||||
out.add(text.slice(start, index));
|
||||
const first = text.charCodeAt(start);
|
||||
let name = first === 37; /* % */
|
||||
if (!name && first === 117 /* u */ && index - start === 13) {
|
||||
name = true;
|
||||
for (let at = start + 1; at < index && name; at++) name = hex(text.charCodeAt(at));
|
||||
}
|
||||
if (name) out.add(text.slice(start, index));
|
||||
start = -1;
|
||||
}
|
||||
}
|
||||
@@ -148,12 +155,6 @@ export function quotedStrings(texts: readonly string[]): Set<string> {
|
||||
return out;
|
||||
}
|
||||
|
||||
/** A table's entries at one point (JournalTable.hold). */
|
||||
export interface HeldEntries {
|
||||
/** Whether the table's first entries are still these keys and values. */
|
||||
same(): boolean;
|
||||
}
|
||||
|
||||
/** A field a unit's lowering must leave as it found it (fixedScalar), or a
|
||||
* flag it may set, which applying its record sets too (flagScalar). */
|
||||
export interface JournalScalar {
|
||||
@@ -288,7 +289,11 @@ export function recordUnit(
|
||||
): RecordedUnit {
|
||||
const tables = host.tables;
|
||||
const before = tables.map((table) => table.size());
|
||||
const held = tables.map((table) => (table.hold === undefined ? null : table.hold()));
|
||||
// Outside a recorded wave, the unit's own snapshot of the tables' content.
|
||||
const own =
|
||||
contents === null
|
||||
? tables.map((table) => (table.contents === undefined ? null : table.contents()))
|
||||
: null;
|
||||
const marks = host.scalars.map((scalar) => scalar.mark());
|
||||
const counts = host.statementCounts();
|
||||
const recorder = new UnitEvents();
|
||||
@@ -296,8 +301,6 @@ export function recordUnit(
|
||||
const reasons: string[] = [];
|
||||
const grown: number[] = [];
|
||||
tables.forEach((table, index) => {
|
||||
const check = held[index] ?? null;
|
||||
if (check !== null && !check.same()) reasons.push(`${table.name}:replaced`);
|
||||
if (table.size() === before[index]) return;
|
||||
if (table.entries === undefined) reasons.push(table.name);
|
||||
else grown.push(index);
|
||||
@@ -339,7 +342,12 @@ export function recordUnit(
|
||||
if (entries !== null) for (const entry of entries) texts.push(entry);
|
||||
}
|
||||
const names = quotedStrings(texts);
|
||||
if (contents !== null) {
|
||||
if (own !== null) {
|
||||
if (reasons.length === 0)
|
||||
tables.forEach((table, index) => {
|
||||
if ((own[index] ?? null)?.changed() != null) reasons.push(`${table.name}:changed`);
|
||||
});
|
||||
} else if (contents !== null) {
|
||||
// A unit with a record must leave the entries it names as they were; a
|
||||
// unit without one keeps its changes.
|
||||
if (reasons.length === 0)
|
||||
@@ -419,6 +427,69 @@ export function lowerWaveRecorded(
|
||||
});
|
||||
}
|
||||
|
||||
/** The content of a keyed table's entries (JournalTable.contents): each
|
||||
* entry's encoded value, and the order of the keys, since a table only adds
|
||||
* entries at the end. An entry is found by its key, or by the name its value
|
||||
* holds (`valueName`): a unit names the helper it got, not the key it was
|
||||
* looked up by. `text` answers undefined for a key the table does not
|
||||
* hold. */
|
||||
function keyedContents(
|
||||
size: () => number,
|
||||
keys: () => Iterable<string>,
|
||||
text: (key: string) => string | null | undefined,
|
||||
valueName: (key: string) => string | null,
|
||||
): HeldContents {
|
||||
const texts = new Map<string, string | null>();
|
||||
let order: string[] = [];
|
||||
const byName = new Map<string, string>();
|
||||
const differs = (key: string): boolean => text(key) !== texts.get(key);
|
||||
const keyOf = (name: string): string | undefined => (texts.has(name) ? name : byName.get(name));
|
||||
const held: HeldContents = {
|
||||
extend: () => {
|
||||
if (texts.size >= size()) return;
|
||||
for (const key of keys()) {
|
||||
if (texts.has(key)) continue;
|
||||
texts.set(key, text(key) ?? null);
|
||||
order.push(key);
|
||||
const named = valueName(key);
|
||||
if (named !== null) byName.set(named, key);
|
||||
}
|
||||
},
|
||||
changed: () => {
|
||||
let index = 0;
|
||||
for (const key of keys()) {
|
||||
if (index >= order.length) break;
|
||||
if (key !== order[index]) return order[index]!;
|
||||
index++;
|
||||
}
|
||||
if (index < order.length) return order[index]!;
|
||||
for (const key of order) if (differs(key)) return key;
|
||||
return null;
|
||||
},
|
||||
changedAmong: (names) => {
|
||||
for (const name of names) {
|
||||
const key = keyOf(name);
|
||||
if (key !== undefined && differs(key)) return key;
|
||||
}
|
||||
return null;
|
||||
},
|
||||
refresh: (names) => {
|
||||
for (const name of names) {
|
||||
const key = keyOf(name);
|
||||
if (key === undefined) continue;
|
||||
const now = text(key);
|
||||
if (now !== undefined) texts.set(key, now);
|
||||
else {
|
||||
texts.delete(key);
|
||||
order = order.filter((held) => held !== key);
|
||||
}
|
||||
}
|
||||
},
|
||||
};
|
||||
held.extend();
|
||||
return held;
|
||||
}
|
||||
|
||||
/** An appendable table over a Map from strings to strings, such as helper
|
||||
* names by key. */
|
||||
export function mapTable(name: string, map: Map<string, string>): JournalTable {
|
||||
@@ -427,6 +498,7 @@ export function mapTable(name: string, map: Map<string, string>): JournalTable {
|
||||
map,
|
||||
(value) => JSON.stringify(value),
|
||||
(text) => JSON.parse(text) as string,
|
||||
(value) => value,
|
||||
);
|
||||
}
|
||||
|
||||
@@ -444,42 +516,15 @@ export function countTable(name: string, map: Map<string, number>): JournalTable
|
||||
* JSON. JSON writes a number that is not finite as null, so a value whose
|
||||
* encoding holds a null does not encode. */
|
||||
export function irDataTable<V>(name: string, map: Map<string, V>): JournalTable {
|
||||
const encode = (value: V): string | null => {
|
||||
const text = JSON.stringify(value);
|
||||
return /[:,[]null\b/.test(text) ? null : text;
|
||||
};
|
||||
return {
|
||||
...codecTable(name, map, encode, (text) => JSON.parse(text) as V),
|
||||
contents: () => {
|
||||
const texts = new Map<string, string | null>();
|
||||
const differs = (key: string): boolean => {
|
||||
const value = map.get(key);
|
||||
return value === undefined || encode(value) !== texts.get(key);
|
||||
};
|
||||
const held: HeldContents = {
|
||||
extend: () => {
|
||||
if (texts.size === map.size) return;
|
||||
for (const [key, value] of map) if (!texts.has(key)) texts.set(key, encode(value));
|
||||
},
|
||||
changed: () => {
|
||||
for (const key of texts.keys()) if (differs(key)) return key;
|
||||
return null;
|
||||
},
|
||||
changedAmong: (keys) => {
|
||||
for (const key of keys) if (texts.has(key) && differs(key)) return key;
|
||||
return null;
|
||||
},
|
||||
refresh: (keys) => {
|
||||
for (const key of keys) {
|
||||
const value = map.get(key);
|
||||
if (texts.has(key) && value !== undefined) texts.set(key, encode(value));
|
||||
}
|
||||
},
|
||||
};
|
||||
held.extend();
|
||||
return held;
|
||||
return codecTable(
|
||||
name,
|
||||
map,
|
||||
(value) => {
|
||||
const text = JSON.stringify(value);
|
||||
return /[:,[]null\b/.test(text) ? null : text;
|
||||
},
|
||||
};
|
||||
(text) => JSON.parse(text) as V,
|
||||
);
|
||||
}
|
||||
|
||||
/** An appendable table over a Map from strings to values that `encode`
|
||||
@@ -491,6 +536,7 @@ export function codecTable<V>(
|
||||
map: Map<string, V>,
|
||||
encode: (value: V) => string | null,
|
||||
decode: (text: string) => V,
|
||||
nameOf: ((value: V) => string) | null = null,
|
||||
): JournalTable {
|
||||
return {
|
||||
name,
|
||||
@@ -517,21 +563,19 @@ export function codecTable<V>(
|
||||
const keys = [...map.keys()].slice(from);
|
||||
for (const key of keys) map.delete(key);
|
||||
},
|
||||
hold: () => {
|
||||
const keys = [...map.keys()];
|
||||
const values = [...map.values()];
|
||||
return {
|
||||
same: () => {
|
||||
let index = 0;
|
||||
for (const [key, value] of map) {
|
||||
if (index >= keys.length) break;
|
||||
if (key !== keys[index] || value !== values[index]) return false;
|
||||
index++;
|
||||
}
|
||||
return index === keys.length;
|
||||
contents: () =>
|
||||
keyedContents(
|
||||
() => map.size,
|
||||
() => map.keys(),
|
||||
(key) => {
|
||||
const value = map.get(key);
|
||||
return value === undefined ? undefined : encode(value);
|
||||
},
|
||||
};
|
||||
},
|
||||
(key) => {
|
||||
const value = map.get(key);
|
||||
return value === undefined || nameOf === null ? null : nameOf(value);
|
||||
},
|
||||
),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -548,20 +592,13 @@ export function setTable(name: string, set: Set<string>): JournalTable {
|
||||
const entries = [...set].slice(from);
|
||||
for (const entry of entries) set.delete(entry);
|
||||
},
|
||||
hold: () => {
|
||||
const entries = [...set];
|
||||
return {
|
||||
same: () => {
|
||||
let index = 0;
|
||||
for (const entry of set) {
|
||||
if (index >= entries.length) break;
|
||||
if (entry !== entries[index]) return false;
|
||||
index++;
|
||||
}
|
||||
return index === entries.length;
|
||||
},
|
||||
};
|
||||
},
|
||||
contents: () =>
|
||||
keyedContents(
|
||||
() => set.size,
|
||||
() => set.keys(),
|
||||
(key) => (set.has(key) ? "" : undefined),
|
||||
() => null,
|
||||
),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -626,10 +663,6 @@ export function functionTable(
|
||||
while (functions.length > from) functions.pop();
|
||||
changed();
|
||||
},
|
||||
hold: () => {
|
||||
const held = [...functions];
|
||||
return { same: () => held.every((fn, index) => functions[index] === fn) };
|
||||
},
|
||||
contents: () => {
|
||||
const texts: (string | null)[] = [];
|
||||
const indexes = new Map<string, number>();
|
||||
|
||||
@@ -323,16 +323,16 @@ export class ShapeRegistry {
|
||||
* writer after all priority candidates have closed. */
|
||||
private readonly derivedDeclaredOrderFinalizers: (() => void)[] = [];
|
||||
private currentDeclaredOrderPriority: DeclaredOrderPriority = [0, 0];
|
||||
/** The declared-order candidates and the attached names the registry
|
||||
* holds, counted as they are added (journalSizes). */
|
||||
private candidateCount = 0;
|
||||
private extendedCount = 0;
|
||||
/** All interned shapes in first-seen order. */
|
||||
readonly shapes: IrRecordShape[] = [];
|
||||
|
||||
/** The sizes of the registry's tables, which only grow while shapes are
|
||||
* interned (unit-journal.ts). */
|
||||
journalSizes(): number[] {
|
||||
let candidates = 0;
|
||||
for (const list of this.declaredOrderCandidates.values()) candidates += list.length;
|
||||
let extended = 0;
|
||||
for (const names of this.extendedNames.values()) extended += names.length;
|
||||
return [
|
||||
this.revision,
|
||||
this.shapes.length,
|
||||
@@ -341,12 +341,12 @@ export class ShapeRegistry {
|
||||
this.keyedNameKeys.size,
|
||||
this.declaredOrderPriority.size,
|
||||
this.declaredOrderCandidates.size,
|
||||
candidates,
|
||||
this.candidateCount,
|
||||
this.derivedDeclaredOrderFinalizers.length,
|
||||
this.recIds.size,
|
||||
this.pendingRec.size,
|
||||
this.extendedNames.size,
|
||||
extended,
|
||||
this.extendedCount,
|
||||
];
|
||||
}
|
||||
/** ts.Types currently being mapped — a BACK-REFERENCE to one of these is
|
||||
@@ -401,6 +401,7 @@ export class ShapeRegistry {
|
||||
|
||||
private adoptDeclaredOrder(shape: IrRecordShape, declaredOrder: string[] | undefined): void {
|
||||
if (declaredOrder === undefined) return;
|
||||
this.candidateCount++;
|
||||
const candidates = this.declaredOrderCandidates.get(shape.id);
|
||||
const candidate = { order: declaredOrder, priority: this.currentDeclaredOrderPriority };
|
||||
if (candidates) candidates.push(candidate);
|
||||
@@ -556,6 +557,7 @@ export class ShapeRegistry {
|
||||
this.byId.set(id, shape);
|
||||
this.shapes.push(shape);
|
||||
if (declaredOrder !== undefined) {
|
||||
this.candidateCount++;
|
||||
this.declaredOrderCandidates.set(id, [
|
||||
{ order: declaredOrder, priority: this.currentDeclaredOrderPriority },
|
||||
]);
|
||||
@@ -618,6 +620,7 @@ export class ShapeRegistry {
|
||||
shape.fields = [...shape.fields, ...fresh].sort((a, b) =>
|
||||
a.name < b.name ? -1 : a.name > b.name ? 1 : 0,
|
||||
);
|
||||
this.extendedCount += fresh.length;
|
||||
this.extendedNames.set(shapeId, [
|
||||
...(this.extendedNames.get(shapeId) ?? []),
|
||||
...fresh.map((f) => f.name),
|
||||
@@ -745,10 +748,6 @@ export class UnionRegistry {
|
||||
this.keyedNameKeys.delete(def.id);
|
||||
}
|
||||
},
|
||||
hold: () => {
|
||||
const held = [...this.unions];
|
||||
return { same: () => held.every((def, index) => this.unions[index] === def) };
|
||||
},
|
||||
contents: () => {
|
||||
// A recursive placeholder still pending fills in place when its
|
||||
// outer frame completes, which changes journalSizes.
|
||||
|
||||
@@ -789,7 +789,9 @@ async function emitNativeProgramObject(
|
||||
// An optimized program that stays one partition is optimized whole and
|
||||
// generates code for its parts concurrently.
|
||||
const codegenParts = single || opts.optimization === "dev" ? 1 : nativeCodegenParts(llvmBytes);
|
||||
const partitions = single ? 1 : Math.max(nativeProgramPartitions(llvmBytes), codegenParts);
|
||||
const partitions = single
|
||||
? 1
|
||||
: Math.max(nativeProgramPartitions(llvmBytes, opts.optimization === "dev"), codegenParts);
|
||||
try {
|
||||
const artifact = await emitNativeArtifact({
|
||||
outputPath: linkPath,
|
||||
@@ -1079,6 +1081,7 @@ async function prepareExecutableInput(
|
||||
objectAudit: opts.sanitize === true,
|
||||
sanitizeAttributes: opts.sanitize === true,
|
||||
lowRegion: lowRegionTarget(buildPlatform, opts.sanitize === true),
|
||||
timing,
|
||||
...(opts.sanitize === true ? { wasmBoundsChecks: true } : {}),
|
||||
...inlineRcOption(opts.optimization, opts.sanitize === true),
|
||||
...(opts.detachedExit === true ? { detachedExit: true } : {}),
|
||||
@@ -2600,6 +2603,7 @@ async function compileLibraryTracked(
|
||||
wasi: buildPlatform === "wasi",
|
||||
objectAudit: opts.sanitize === true,
|
||||
sanitizeAttributes: opts.sanitize === true,
|
||||
timing,
|
||||
...inlineRcOption(profile.optimization, opts.sanitize === true),
|
||||
});
|
||||
timing("llvm-emit", {
|
||||
|
||||
@@ -1044,8 +1044,9 @@ const RUNTIME_FEATURE_RULES = {
|
||||
mime: rule({ prefixes: ["mime."] }),
|
||||
/** Web streams in checked-dynamic code (scr_webstreams.c). */
|
||||
webStreams: rule({ prefixes: ["webstream."] }),
|
||||
/** AbortController and AbortSignal (scr_abort.c, or the fetch unit). */
|
||||
abortSignal: rule({ prefixes: ["abort."] }),
|
||||
/** AbortController and AbortSignal (scr_abort.c, or the fetch unit).
|
||||
* A WritableStream controller's signal is one (scr_webstreams.c). */
|
||||
abortSignal: rule({ prefixes: ["abort.", "webstream."] }),
|
||||
/** util.debounce and util.throttle (scr_util_timing.c). */
|
||||
utilTiming: rule({ fns: ["util.debounce", "util.throttle"] }),
|
||||
/** events.once(emitter, name, { signal }) (scr_events_once_signal.c). */
|
||||
|
||||
@@ -75,6 +75,11 @@ import { splitLlvmLibraryProgram } from "../backend/llvm/split.js";
|
||||
import { prepareNativeExecutable, prepareNativeLibrary } from "./prepare.js";
|
||||
import { buildSanitizedRuntime, sanitizerDriver, sanitizerFlags } from "./sanitizer.js";
|
||||
import { inputsWithoutLld, lldLinkerArgs } from "../backend/lld.js";
|
||||
import {
|
||||
hugeTextLinkerArgs,
|
||||
hugeTextProgram,
|
||||
rewriteHugeTextExecutable,
|
||||
} from "../backend/huge-text.js";
|
||||
import { openNativeExecutableCache } from "./executable-cache.js";
|
||||
import { LinkerJobs } from "./linker-job.js";
|
||||
import { nativeFileIdentity } from "./file-identity.js";
|
||||
@@ -381,7 +386,9 @@ export class NativeCompiler {
|
||||
// An optimized program that stays one partition is optimized whole and
|
||||
// generates code for its parts concurrently.
|
||||
const codegenParts = library || optimization === "dev" ? 1 : nativeCodegenParts(size);
|
||||
const partitions = library ? 1 : Math.max(nativeProgramPartitions(size), codegenParts);
|
||||
const partitions = library
|
||||
? 1
|
||||
: Math.max(nativeProgramPartitions(size, optimization === "dev"), codegenParts);
|
||||
return this.emitObject(
|
||||
input,
|
||||
output,
|
||||
@@ -637,11 +644,15 @@ export class NativeCompiler {
|
||||
});
|
||||
// Linux links with lld when the driver finds it.
|
||||
const lld = options.sanitize ? [] : lldLinkerArgs(target.platform, toolchain.linker);
|
||||
const hugeText =
|
||||
!options.sanitize && hugeTextProgram(target, [object, ...programObjects.slice(1)]);
|
||||
const zig = process.env["SCRIPTC_LINKER"] === undefined && target.defaultLinker === "zig";
|
||||
const linkMap = join(stage, "link.map");
|
||||
const mapArgs = staticLinkMapArgs(target, options.staticLink === true, linkMap);
|
||||
const linkArgsWith = (withLld: boolean): string[] => [
|
||||
...(options.sanitize ? ["-fsanitize=address"] : toolchain.linkerArgs),
|
||||
...(withLld ? lld : []),
|
||||
...hugeTextLinkerArgs(hugeText, (withLld && lld.length !== 0) || zig),
|
||||
...plan.driverFlags,
|
||||
...mapArgs,
|
||||
...(withLld ? plan.inputs : inputsWithoutLld(target.platform, plan.inputs)),
|
||||
@@ -679,6 +690,7 @@ export class NativeCompiler {
|
||||
}
|
||||
timing("native-link");
|
||||
requireNativeArtifact(stagedOutput);
|
||||
if (hugeText) rewriteHugeTextExecutable(stagedOutput);
|
||||
if (mapArgs.length !== 0) {
|
||||
const refusal = staticExecutableRefusal(target, true, readFileSync(linkMap, "utf8"));
|
||||
if (refusal !== null) throw new NativeCodegenError("SC3002", refusal);
|
||||
|
||||
@@ -440,6 +440,7 @@ export function prepareNativeExecutable(
|
||||
lowRegion: lowRegionTarget(toolchain.target.platform, options.sanitize === true),
|
||||
functionWorkers: emitWorkers(),
|
||||
forkPrimitives: emitForkPrimitives(),
|
||||
timing,
|
||||
...inlineRcOption(options.optimization, options.sanitize === true),
|
||||
...(options.detachedExit === true && outputKind === "exe" ? { detachedExit: true } : {}),
|
||||
...targetCpuField(options.targetCpu),
|
||||
@@ -556,6 +557,7 @@ export function prepareNativeLibrary(
|
||||
wasi: toolchain.target.platform === "wasi",
|
||||
objectAudit: options.sanitize === true,
|
||||
sanitizeAttributes: options.sanitize === true,
|
||||
timing,
|
||||
...inlineRcOption(profile.optimization, options.sanitize === true),
|
||||
});
|
||||
timing("llvm-emit");
|
||||
|
||||
@@ -438,3 +438,36 @@ test("a loop that starts from a typed-array read guards the read's integer and o
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test("an exact integer loop limit becomes the endpoint without rounding", async () => {
|
||||
const dir = await mkdtemp(join(tmpdir(), "scriptc-counted-limit-"));
|
||||
try {
|
||||
const entry = join(import.meta.dirname, "../../../tests/corpus/counted-loop-integer-limits.ts");
|
||||
const outPath = join(dir, "limits.ir.json");
|
||||
const result = await compile(entry, { outPath, outDir: dir, outputKind: "ir" });
|
||||
if (!result.ok) throw new Error(result.diagnostics.map((d) => d.message).join("\n"));
|
||||
const ll = emitLlvmModule(deserializeModule(await readFile(outPath, "utf8")), {});
|
||||
const bodies = (method: string): string[] =>
|
||||
[
|
||||
...ll.matchAll(
|
||||
new RegExp(
|
||||
`define internal [^\\n]+ @\\w*Page_${method}(?:\\.i32)?\\([^\\n]*\\) #0 \\{ ; [^\\n]*([\\s\\S]*?)\\n\\}`,
|
||||
"g",
|
||||
),
|
||||
),
|
||||
].map((m) => m[1]!);
|
||||
// The int32 rank is the endpoint of the loops in both directions, with
|
||||
// exclusive and inclusive bounds: no floor, ceil or 2^53 clamp.
|
||||
for (const method of ["set", "suffix"]) {
|
||||
const found = bodies(method);
|
||||
expect(found.length).toBeGreaterThan(0);
|
||||
for (const body of found) {
|
||||
expect(body).toContain("integer induction i");
|
||||
expect(body).not.toMatch(/@llvm\.(floor|ceil)\.f64/);
|
||||
expect(body).not.toContain("0x4340000000000000");
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
@@ -0,0 +1,140 @@
|
||||
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, expect, test } from "vitest";
|
||||
import { compile, deserializeModule } from "../src/index.js";
|
||||
import { emitLlvmModule } from "../src/backend/llvm/emitter.js";
|
||||
import type { IrExpr, IrModule } from "../src/ir/ir.js";
|
||||
|
||||
const directories: string[] = [];
|
||||
afterEach(async () => {
|
||||
await Promise.all(directories.splice(0).map((dir) => rm(dir, { recursive: true, force: true })));
|
||||
});
|
||||
|
||||
async function emit(source: string): Promise<string> {
|
||||
const outDir = await mkdtemp(join(tmpdir(), "scriptc-function-table-"));
|
||||
directories.push(outDir);
|
||||
const entry = join(outDir, "main.ts");
|
||||
await writeFile(entry, source);
|
||||
const outPath = join(outDir, "program.ll");
|
||||
const result = await compile(entry, { entry, outDir, outPath, outputKind: "llvm" });
|
||||
expect(result.ok, JSON.stringify(result)).toBe(true);
|
||||
return readFile(outPath, "utf8");
|
||||
}
|
||||
|
||||
async function lowerIr(source: string): Promise<IrModule> {
|
||||
const outDir = await mkdtemp(join(tmpdir(), "scriptc-function-table-"));
|
||||
directories.push(outDir);
|
||||
const entry = join(outDir, "main.ts");
|
||||
await writeFile(entry, source);
|
||||
const outPath = join(outDir, "main.ir.json");
|
||||
const result = await compile(entry, { outDir, outPath, outputKind: "ir" });
|
||||
expect(result.ok, JSON.stringify(result)).toBe(true);
|
||||
return deserializeModule(await readFile(outPath, "utf8"));
|
||||
}
|
||||
|
||||
function definition(llvm: string, symbol: string): string {
|
||||
const match = new RegExp(`^define internal [^\\n]*@${symbol}(?:\\.i32)?\\([^]*?^}`, "m").exec(
|
||||
llvm,
|
||||
);
|
||||
expect(match, symbol).not.toBeNull();
|
||||
return match![0];
|
||||
}
|
||||
|
||||
// A record of functions dispatched by key: the hit lends the stored closure
|
||||
// to the call (no union box, no lookup through the runtime, no closure
|
||||
// retain or release), and only a miss tests the key against
|
||||
// Object.prototype's method names.
|
||||
const source = `
|
||||
type Op = (acc: number, arg: number) => number;
|
||||
const ops: Record<string, Op> = {
|
||||
add: (acc, arg) => acc + arg,
|
||||
neg: (acc) => -acc,
|
||||
};
|
||||
function asserted(name: string, acc: number): number {
|
||||
return ops[name]!(acc, 1);
|
||||
}
|
||||
function optional(name: string, acc: number): number {
|
||||
return ops[name]?.(acc, 1) ?? acc;
|
||||
}
|
||||
// Object arguments: the inherited methods' key conversion must not make
|
||||
// every indirect call a heap writer (which would end all borrowing).
|
||||
class Point {
|
||||
constructor(readonly x: number) {}
|
||||
}
|
||||
type Measure = (p: Point, q: Point) => number;
|
||||
const measures: Record<string, Measure> = { sum: (p, q) => p.x + q.x };
|
||||
function objects(name: string, p: Point): number {
|
||||
return measures[name]!(p, p);
|
||||
}
|
||||
function checked(name: string, acc: number): number {
|
||||
const handler = ops[name];
|
||||
return handler ? handler(acc, 1) : acc;
|
||||
}
|
||||
let acc = 1;
|
||||
for (const name of ["add", "neg", "nope", "toString"]) {
|
||||
acc = asserted(name === "nope" ? "add" : name === "toString" ? "neg" : name, acc);
|
||||
acc = optional(name, acc);
|
||||
acc = checked(name, acc);
|
||||
acc += objects(name === "nope" ? "sum" : "sum", new Point(acc));
|
||||
}
|
||||
console.log(acc);
|
||||
`;
|
||||
|
||||
test("function-table dispatch borrows the stored closure on a hit", async () => {
|
||||
const llvm = await emit(source);
|
||||
for (const fn of ["asserted", "optional", "checked", "objects"]) {
|
||||
const body = definition(llvm, `sc_bf_${fn}`);
|
||||
expect(body, fn).not.toMatch(/scr_union_new_ref|scr_map_get_str_ref/);
|
||||
expect(body, fn).not.toMatch(/scr_closure_retain|sc_nretain_func/);
|
||||
expect(body, fn).toMatch(/sc_map_entry_str/);
|
||||
}
|
||||
});
|
||||
|
||||
// A borrowed ternary whose arms are fallback reads: a miss binds the
|
||||
// inherited method into a frame-owned slot, which must outlive the arm
|
||||
// until the call has used the lent closure. The lowering builds no such
|
||||
// ternary today (a bound read is a call's own callee), so the test wraps
|
||||
// the callee read of `pick` in one.
|
||||
test("a borrowed ternary of fallback reads keeps a bound default past its arm", async () => {
|
||||
const mod = await lowerIr(`
|
||||
type Op = (x: string) => string;
|
||||
const table: Record<string, Op> = { echo: (x) => x };
|
||||
function pick(name: string, flag: boolean): string {
|
||||
return table[name]!("echo");
|
||||
}
|
||||
console.log(pick(process.argv[2] ?? "hasOwnProperty", process.argv.length > 3));
|
||||
`);
|
||||
const pick = mod.functions.find((fn) => fn.name === "pick")!;
|
||||
const flag = pick.params[1]!;
|
||||
let wrapped = 0;
|
||||
const visit = (node: unknown): void => {
|
||||
if (node === null || typeof node !== "object") return;
|
||||
for (const [key, value] of Object.entries(node)) {
|
||||
const e = value as IrExpr | null;
|
||||
if (e !== null && typeof e === "object" && e.kind === "nullish" && e.right.kind === "call") {
|
||||
const cond: IrExpr = { kind: "varRef", localId: flag.localId, type: flag.type, loc: e.loc };
|
||||
(node as Record<string, unknown>)[key] = {
|
||||
kind: "ternary",
|
||||
cond,
|
||||
then: e,
|
||||
else_: structuredClone(e),
|
||||
type: e.type,
|
||||
loc: e.loc,
|
||||
};
|
||||
wrapped++;
|
||||
} else visit(value);
|
||||
}
|
||||
};
|
||||
visit(pick.body);
|
||||
expect(wrapped).toBe(1);
|
||||
const body = definition(emitLlvmModule(mod), "sc_bf_pick");
|
||||
// Each arm lends its read and leaves: nothing it filled is released
|
||||
// before the join.
|
||||
const arms = [...body.matchAll(/^read\.(?:then|else)\d+:\n([^]*?)(?=^read\.)/gm)].map(
|
||||
(m) => m[1]!,
|
||||
);
|
||||
expect(arms).toHaveLength(2);
|
||||
for (const arm of arms)
|
||||
expect(arm).toMatch(/store ptr %t\d+, ptr %s\d+\n {2}br label %read\.end\d+\n$/);
|
||||
});
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"order": [
|
||||
"<repo>/tests/corpus/8363-record-inherited-keys.ts"
|
||||
],
|
||||
"diags": []
|
||||
}
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"order": [
|
||||
"<repo>/tests/corpus/8364-record-nullish-receiver-arm/main.ts"
|
||||
],
|
||||
"diags": []
|
||||
}
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"order": [
|
||||
"<repo>/tests/corpus/8408-fetch-connection-reuse.ts"
|
||||
],
|
||||
"diags": []
|
||||
}
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"order": [
|
||||
"<repo>/tests/corpus/8409-fetch-compressed-read-after-end.ts"
|
||||
],
|
||||
"diags": []
|
||||
}
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"order": [
|
||||
"<repo>/tests/corpus/counted-loop-integer-limits.ts"
|
||||
],
|
||||
"diags": []
|
||||
}
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"order": [
|
||||
"<repo>/tests/corpus/pad-case-converted-strings.ts"
|
||||
],
|
||||
"diags": []
|
||||
}
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"order": [
|
||||
"<repo>/tests/corpus/web-stream-transform-pipes-untyped.mjs"
|
||||
],
|
||||
"diags": []
|
||||
}
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"order": [
|
||||
"<repo>/tests/corpus/web-stream-writable-untyped.mjs"
|
||||
],
|
||||
"diags": []
|
||||
}
|
||||
@@ -1551,8 +1551,11 @@ static ScrDyn *scr_dyn_then_of(const ScrDyn *value, const ScrDyn *reader) {
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/* Resolves `target` with `value` (borrowed). */
|
||||
static void scr_dyn_resolve_value(ScrPromise *target, ScrDyn *value, const ScrDyn *reader) {
|
||||
/* Resolves `target` with `value` (borrowed). Inlined into both callers, so
|
||||
* the code of scr_dyn_then_resolve_call does not depend on whether a
|
||||
* program links scr_dyn_promise_resolve_with. */
|
||||
__attribute__((always_inline)) static inline void scr_dyn_resolve_value(ScrPromise *target, ScrDyn *value,
|
||||
const ScrDyn *reader) {
|
||||
if (value->kind == SCR_DYN_PROMISE) {
|
||||
scr_promise_resolve_dyn(target, scr_dyn_retain(value));
|
||||
return;
|
||||
@@ -1669,6 +1672,10 @@ ScrPromise *scr_dyn_promise_resolve(const ScrDyn *value, const ScrDyn *reader) {
|
||||
return scr_dyn_promise_entry(NULL, value, reader);
|
||||
}
|
||||
|
||||
void scr_dyn_promise_resolve_with(ScrPromise *target, ScrDyn *value) {
|
||||
scr_dyn_resolve_value(target, value, NULL);
|
||||
}
|
||||
|
||||
static void scr_dyn_settled_store(ScrArr *a, double i, ScrPromise *src) {
|
||||
ScrDyn *record = scr_dyn_new_obj();
|
||||
bool fulfilled = !scr_promise_is_rejected(src);
|
||||
|
||||
@@ -244,7 +244,14 @@ typedef enum {
|
||||
Y(WS_COUNT_STRATEGY) /* CountQueuingStrategy (scr_webstreams.c) */ \
|
||||
Y(WS_RCONTROLLER) /* ReadableStreamDefaultController (scr_webstreams.c) */ \
|
||||
Y(WS_READABLE) /* ReadableStream for checked-dynamic code (scr_webstreams.c) */ \
|
||||
Y(WS_READER) /* ReadableStreamDefaultReader (scr_webstreams.c) */
|
||||
Y(WS_READER) /* ReadableStreamDefaultReader (scr_webstreams.c) */ \
|
||||
Y(WS_TCONTROLLER) /* TransformStreamDefaultController (scr_webstreams.c) */ \
|
||||
Y(WS_TEXT_DECODER) /* TextDecoderStream (scr_webstreams.c) */ \
|
||||
Y(WS_TEXT_ENCODER) /* TextEncoderStream (scr_webstreams.c) */ \
|
||||
Y(WS_TRANSFORM) /* TransformStream for checked-dynamic code (scr_webstreams.c) */ \
|
||||
Y(WS_WCONTROLLER) /* WritableStreamDefaultController (scr_webstreams.c) */ \
|
||||
Y(WS_WRITABLE) /* WritableStream for checked-dynamic code (scr_webstreams.c) */ \
|
||||
Y(WS_WRITER) /* WritableStreamDefaultWriter (scr_webstreams.c) */
|
||||
|
||||
/* Values below the largest pinned one that no tag holds, each with the
|
||||
* reason. Never reuse one without removing it here. */
|
||||
@@ -2054,6 +2061,12 @@ typedef struct ScrDynHandleClass {
|
||||
/* The prototype's methods and getters in definition order (the order
|
||||
* Object.keys reports); NULL = the methods, then the getters. */
|
||||
const char *const *order;
|
||||
/* A call of one of the class's method values
|
||||
* (`WritableStream.prototype.close.call(stream)`): the class's method,
|
||||
* which an own property of the same name on the handle does not
|
||||
* replace. NULL = ScrDynHandleOps.invoke. */
|
||||
ScrDyn *(*method_invoke)(void *h, ScrDyn *self, const char *method, ScrDyn *const *args, size_t argc,
|
||||
const char *what);
|
||||
} ScrDynHandleClass;
|
||||
|
||||
/* Installs a tag's ops with the class its handles belong to; both outlive
|
||||
@@ -2080,6 +2093,11 @@ ScrDyn *scr_dyn_handle_class_prototype_of(const ScrDyn *handle);
|
||||
* receiver (+1); NULL when the chain has none or with the exception
|
||||
* pending. */
|
||||
ScrDyn *scr_dyn_handle_class_member(ScrDynHandleTag tag, void *h, const char *key, size_t len);
|
||||
/* A method value's call on a handle of its tag (`self`, borrowed): the
|
||||
* class's method_invoke, or the ops' invoke for a tag without one. +1, or
|
||||
* NULL with the exception pending. */
|
||||
ScrDyn *scr_dyn_handle_class_method_call(const ScrDynHandleOps *ops, ScrDyn *self, const char *method,
|
||||
ScrDyn *const *args, size_t argc);
|
||||
|
||||
/* The operations a Transform handle gives the stream unit (scr_stream.c):
|
||||
* the slot where the handle keeps its stream (+1) with the stream's
|
||||
@@ -2156,6 +2174,11 @@ ScrPromise *scr_dyn_promise_any(const ScrDyn *entries, const ScrDyn *reader);
|
||||
* itself, a thenable adopts through its resolve job, a throwing `then`
|
||||
* read rejects, and any other value fulfills. +1. */
|
||||
ScrPromise *scr_dyn_promise_resolve(const ScrDyn *value, const ScrDyn *reader);
|
||||
/* Resolves `target` with a dyn value (borrowed), as a promise's resolve
|
||||
* function does: a native promise adopts, a thenable adopts through its
|
||||
* resolve job, a throwing `then` read rejects, and any other value
|
||||
* fulfills. */
|
||||
void scr_dyn_promise_resolve_with(ScrPromise *target, ScrDyn *value);
|
||||
|
||||
/* Native predicates for typed reference capsules and boxed errors. */
|
||||
bool scr_dyn_ref_typeof_is(const ScrDyn *d, const char *name);
|
||||
|
||||
@@ -183,6 +183,14 @@ ScrDyn *scr_dyn_handle_class_member(ScrDynHandleTag tag, void *h, const char *ke
|
||||
return value;
|
||||
}
|
||||
|
||||
ScrDyn *scr_dyn_handle_class_method_call(const ScrDynHandleOps *ops, ScrDyn *self, const char *method,
|
||||
ScrDyn *const *args, size_t argc) {
|
||||
const ScrDynHandleClass *klass = hc_classes[self->v.handle.tag];
|
||||
if (klass && klass->method_invoke)
|
||||
return klass->method_invoke(self->v.handle.ptr, self, method, args, argc, method);
|
||||
return ops->invoke(self->v.handle.ptr, self, method, args, argc, method);
|
||||
}
|
||||
|
||||
static void hc_define_method(ScrDyn *prototype, ScrDynHandleTag tag, const char *name) {
|
||||
ScrDyn *fn = scr_dyn_handle_class_method(tag, name);
|
||||
if (fn) hc_define(prototype, name, fn, true, true, true);
|
||||
|
||||
@@ -1140,15 +1140,19 @@ static void sf_response_pause_if_full(SfStream *s) {
|
||||
|
||||
static void sf_response_resume_if_ready(SfStream *s) {
|
||||
SfTransfer *t = s->response_owner;
|
||||
if (!t || t->done || !t->response_paused || !t->response_socket ||
|
||||
s->queued >= 1 || s->close_requested || s->closed || s->error) {
|
||||
if (!t || t->done || s->queued >= 1 || s->close_requested || s->closed ||
|
||||
s->error || t->inflate_draining) {
|
||||
return;
|
||||
}
|
||||
if (t->inflate_draining) return;
|
||||
if (t->inflating && (t->inflate_pending || t->inflate_needs_drain)) {
|
||||
/* Compressed input or decoder output still buffered drains on demand:
|
||||
* while the socket is paused, and after 'end', when the transfer no
|
||||
* longer holds the socket and no more data arrives. */
|
||||
if (t->inflating && (t->inflate_pending || t->inflate_needs_drain) &&
|
||||
(t->response_paused || t->response_ended)) {
|
||||
sf_inflate_resume(t);
|
||||
return;
|
||||
}
|
||||
if (!t->response_paused || !t->response_socket) return;
|
||||
t->response_paused = false;
|
||||
scr_net_sock_release(scr_net_sock_resume(t->response_socket));
|
||||
}
|
||||
@@ -5538,9 +5542,21 @@ static void sf_on_data(ScrClosure *cb, ScrBytes *chunk) {
|
||||
sf_release(t);
|
||||
}
|
||||
|
||||
/* After 'end' the keep-alive agent can hand the socket to the next request
|
||||
* on the connection, so the transfer leaves it reading and stops pausing
|
||||
* and resuming it. */
|
||||
static void sf_drop_response_socket(SfTransfer *t) {
|
||||
if (!t->response_socket) return;
|
||||
if (t->response_paused) scr_net_sock_release(scr_net_sock_resume(t->response_socket));
|
||||
t->response_paused = false;
|
||||
scr_net_sock_release(t->response_socket);
|
||||
t->response_socket = NULL;
|
||||
}
|
||||
|
||||
static void sf_on_end(ScrClosure *cb) {
|
||||
SfTransfer *t = sf_from(cb);
|
||||
if (!t) return;
|
||||
sf_drop_response_socket(t);
|
||||
if (!t->done) {
|
||||
if (t->inflating && t->response_stream &&
|
||||
!t->response_stream->discarded) {
|
||||
@@ -6081,9 +6097,16 @@ static bool sf_start_hop(SfTransfer *t) {
|
||||
wrap_ctx = scr_tls_fetch_client_ctx(sni, true);
|
||||
wrap = &scr_tls_fetch_client_wrap;
|
||||
}
|
||||
ScrHttpClientReq *client = scr_http_request_ex(
|
||||
dial, dial_port, path, t->method, 0, headers, false, NULL, NULL,
|
||||
default_port, wrap, wrap_ctx);
|
||||
/* A direct connection goes through fetch's keep-alive agent, which hands
|
||||
* an idle connection to the next request for the same origin, as
|
||||
* undici's dispatcher does; the bare host keys the pool, so origins that
|
||||
* share an address keep their own connections. A proxied request dials
|
||||
* its own connection. */
|
||||
ScrHttpClientReq *client =
|
||||
proxy ? scr_http_request_ex(dial, dial_port, path, t->method, 0, headers, false, NULL,
|
||||
NULL, default_port, wrap, wrap_ctx)
|
||||
: scr_http_fetch_request(https, dial, dial_port, path, t->method, headers,
|
||||
default_port, wrap, wrap_ctx, bare);
|
||||
scr_arr_release(headers);
|
||||
scr_str_release(path);
|
||||
scr_str_release(dial);
|
||||
|
||||
+105
-53
@@ -5286,6 +5286,9 @@ typedef struct ScrHttpAgent {
|
||||
double max_sockets; /* INFINITY = Node's default */
|
||||
double max_free;
|
||||
double timeout_ms; /* < 0 = unset */
|
||||
/* without a timeout, how long an idle pooled socket stays open (0 = until
|
||||
* the server closes it); fetch's agent keeps undici's 4 s */
|
||||
double idle_ms;
|
||||
double default_port; /* settable (agent.defaultPort = p) */
|
||||
bool destroyed;
|
||||
ScrHttpAgentEnt *ents; /* append order — Node's FIFO queue */
|
||||
@@ -5519,7 +5522,7 @@ static bool scr_http_client_pool(struct ScrHttpConn *conn, struct ScrHttpClientR
|
||||
scr_str_release(name);
|
||||
return false;
|
||||
}
|
||||
double timeout = ag->timeout_ms > 0 ? ag->timeout_ms : 0;
|
||||
double timeout = ag->timeout_ms > 0 ? ag->timeout_ms : ag->idle_ms;
|
||||
if (hint >= 0) {
|
||||
double server = hint - 1000;
|
||||
if (server < 0) {
|
||||
@@ -5606,20 +5609,12 @@ static ScrStr *scr_http_agent_name(const char *host, size_t host_len, const char
|
||||
return scr_jb_finish(&b);
|
||||
}
|
||||
|
||||
ScrDyn *scr_http_agent_new(bool secure, bool keep_alive, double ka_msecs,
|
||||
double max_sockets, double max_free, double timeout_ms,
|
||||
double port /* < 0 = unset */, const ScrDyn *scheduling) {
|
||||
/* A registered agent (+1), the half of new http.Agent(opts) after option
|
||||
* validation. */
|
||||
static ScrHttpAgent *scr_http_agent_alloc(bool secure, bool keep_alive, double ka_msecs,
|
||||
double max_sockets, double max_free, double timeout_ms,
|
||||
double port, bool fifo) {
|
||||
scr_http_install();
|
||||
/* Node: options.scheduling || 'lifo', then validateOneOf */
|
||||
bool fifo = false;
|
||||
if (scheduling != NULL && scr_dyn_truthy(scheduling)) {
|
||||
const ScrStr *s = scheduling->kind == SCR_DYN_STR ? scheduling->v.str : NULL;
|
||||
if (s != NULL && s->len == 4 && memcmp(s->data, "fifo", 4) == 0) fifo = true;
|
||||
else if (s == NULL || s->len != 4 || memcmp(s->data, "lifo", 4) != 0) {
|
||||
scr_dyn_arg_value_fail("scheduling", "must be one of: 'fifo', 'lifo'", scheduling);
|
||||
return NULL;
|
||||
}
|
||||
}
|
||||
ScrHttpAgent *a = calloc(1, sizeof *a);
|
||||
if (!a) scr_http_oom();
|
||||
a->rc = 1;
|
||||
@@ -5642,6 +5637,25 @@ ScrDyn *scr_http_agent_new(bool secure, bool keep_alive, double ka_msecs,
|
||||
a->in_registry = true;
|
||||
a->next = scr_http_agents;
|
||||
scr_http_agents = scr_http_agent_retain(a);
|
||||
return a;
|
||||
}
|
||||
|
||||
ScrDyn *scr_http_agent_new(bool secure, bool keep_alive, double ka_msecs,
|
||||
double max_sockets, double max_free, double timeout_ms,
|
||||
double port /* < 0 = unset */, const ScrDyn *scheduling) {
|
||||
scr_http_install();
|
||||
/* Node: options.scheduling || 'lifo', then validateOneOf */
|
||||
bool fifo = false;
|
||||
if (scheduling != NULL && scr_dyn_truthy(scheduling)) {
|
||||
const ScrStr *s = scheduling->kind == SCR_DYN_STR ? scheduling->v.str : NULL;
|
||||
if (s != NULL && s->len == 4 && memcmp(s->data, "fifo", 4) == 0) fifo = true;
|
||||
else if (s == NULL || s->len != 4 || memcmp(s->data, "lifo", 4) != 0) {
|
||||
scr_dyn_arg_value_fail("scheduling", "must be one of: 'fifo', 'lifo'", scheduling);
|
||||
return NULL;
|
||||
}
|
||||
}
|
||||
ScrHttpAgent *a = scr_http_agent_alloc(secure, keep_alive, ka_msecs, max_sockets, max_free,
|
||||
timeout_ms, port, fifo);
|
||||
ScrDyn *d = scr_dyn_new_handle(a, SCR_DYNH_HTTP_AGENT);
|
||||
scr_http_agent_release(a); /* the handle's retain holds it */
|
||||
return d;
|
||||
@@ -5703,6 +5717,59 @@ static SCR_TL ScrStr *scr_http_next_socket_path = NULL;
|
||||
extern SCR_TL ScrNetSocket *(*scr_http_socket_dialer)(ScrStr *path, bool deferred);
|
||||
SCR_TL ScrNetSocket *(*scr_http_socket_dialer)(ScrStr *path, bool deferred) = NULL;
|
||||
|
||||
/* A request through agent `ag`: an idle pooled socket for the request's
|
||||
* name (and pool_key) serves it, a free slot dials, and otherwise it
|
||||
* queues. socket_path (borrowed, nullable) dials a Unix socket. */
|
||||
static ScrHttpClientReq *scr_http_request_pooled(ScrHttpAgent *ag, ScrStr *host /*borrowed*/,
|
||||
double port, ScrStr *path /*borrowed*/,
|
||||
ScrStr *method /*borrowed*/, double timeout_ms,
|
||||
ScrArr *header_pairs /*borrowed*/, bool auto_end,
|
||||
ScrClosure *cb /*moves, nullable*/,
|
||||
ScrHttpRespFn fn, int default_port,
|
||||
void (*wrap)(ScrNetSocket *, void *),
|
||||
void *wrap_ctx, ScrStr *pool_key /*borrowed*/,
|
||||
ScrStr *socket_path /*borrowed*/) {
|
||||
double p = port >= 0 ? port
|
||||
: ag->default_port > 0 ? ag->default_port
|
||||
: (double)default_port;
|
||||
char portbuf[16];
|
||||
int portn = snprintf(portbuf, sizeof portbuf, "%d", (int)p);
|
||||
ScrStr *name = scr_http_agent_name(host->data, host->len, portbuf, (size_t)portn,
|
||||
NULL, 0, 0, socket_path ? socket_path->data : NULL,
|
||||
socket_path ? socket_path->len : 0);
|
||||
/* a free pooled socket for the name serves the request right away */
|
||||
ScrHttpAgentFree *reuse = ag->keep_alive ? scr_http_agent_free_take(ag, name, pool_key) : NULL;
|
||||
bool queue = reuse == NULL && scr_http_agent_active(ag, name) >= ag->max_sockets;
|
||||
ScrNetSocket *presock = socket_path != NULL ? scr_http_socket_dialer(socket_path, queue || reuse)
|
||||
: queue || reuse ? scr_net_connect_deferred(p, host)
|
||||
: NULL;
|
||||
ScrHttpClientReq *c = scr_http_request_impl(host, p, path, method, timeout_ms, header_pairs,
|
||||
auto_end, cb, fn, default_port, wrap, wrap_ctx,
|
||||
presock);
|
||||
if (c == NULL) { /* the method-token throw: nothing registered */
|
||||
if (reuse != NULL) scr_http_agent_free_untake(ag, reuse); /* back to the list it came from */
|
||||
scr_str_release(name);
|
||||
return NULL;
|
||||
}
|
||||
c->pool_key = pool_key ? scr_str_retain(pool_key) : NULL;
|
||||
if (ag->timeout_ms >= 0 && timeout_ms <= 0) {
|
||||
c->timeout_ms = ag->timeout_ms;
|
||||
scr_net_sock_set_timeout(c->sock, ag->timeout_ms);
|
||||
}
|
||||
if (reuse != NULL) scr_http_client_rebind(c, reuse, true);
|
||||
else if (!queue) scr_http_conn_count(c->conn, ag->sock_count); /* dialed now */
|
||||
c->agent = scr_http_agent_retain(ag);
|
||||
ScrHttpAgentEnt *e = calloc(1, sizeof *e);
|
||||
if (!e) scr_http_oom();
|
||||
e->name = name; /* ownership moves */
|
||||
e->client = scr_http_client_retain(c);
|
||||
e->queued = queue;
|
||||
ScrHttpAgentEnt **link = &ag->ents;
|
||||
while (*link) link = &(*link)->next;
|
||||
*link = e;
|
||||
return c;
|
||||
}
|
||||
|
||||
ScrHttpClientReq *scr_http_request_agent_ex(ScrStr *host /*borrowed*/, double port,
|
||||
ScrStr *path /*borrowed*/, ScrStr *method /*borrowed*/,
|
||||
double timeout_ms, ScrArr *header_pairs /*borrowed*/,
|
||||
@@ -5754,46 +5821,29 @@ ScrHttpClientReq *scr_http_request_agent_ex(ScrStr *host /*borrowed*/, double po
|
||||
scr_dyn_arg_type_fail("options.agent", "an instance of http.Agent, false, null, or undefined", agent);
|
||||
return NULL;
|
||||
}
|
||||
ScrHttpAgent *ag = (ScrHttpAgent *)agent->v.handle.ptr;
|
||||
double p = port >= 0 ? port
|
||||
: ag->default_port > 0 ? ag->default_port
|
||||
: (double)default_port;
|
||||
char portbuf[16];
|
||||
int portn = snprintf(portbuf, sizeof portbuf, "%d", (int)p);
|
||||
ScrStr *name = scr_http_agent_name(host->data, host->len, portbuf, (size_t)portn,
|
||||
NULL, 0, 0, socket_path ? socket_path->data : NULL,
|
||||
socket_path ? socket_path->len : 0);
|
||||
/* a free pooled socket for the name serves the request right away */
|
||||
ScrHttpAgentFree *reuse = ag->keep_alive ? scr_http_agent_free_take(ag, name, pool_key) : NULL;
|
||||
bool queue = reuse == NULL && scr_http_agent_active(ag, name) >= ag->max_sockets;
|
||||
ScrNetSocket *presock = socket_path != NULL ? scr_http_socket_dialer(socket_path, queue || reuse)
|
||||
: queue || reuse ? scr_net_connect_deferred(p, host)
|
||||
: NULL;
|
||||
ScrHttpClientReq *c = scr_http_request_impl(host, p, path, method, timeout_ms, header_pairs,
|
||||
auto_end, cb, fn, default_port, wrap, wrap_ctx,
|
||||
presock);
|
||||
if (c == NULL) { /* the method-token throw: nothing registered */
|
||||
if (reuse != NULL) scr_http_agent_free_untake(ag, reuse); /* back to the list it came from */
|
||||
scr_str_release(name);
|
||||
return NULL;
|
||||
return scr_http_request_pooled((ScrHttpAgent *)agent->v.handle.ptr, host, port, path, method,
|
||||
timeout_ms, header_pairs, auto_end, cb, fn, default_port, wrap,
|
||||
wrap_ctx, pool_key, socket_path);
|
||||
}
|
||||
|
||||
/* The agents fetch() dispatches through, like undici's global dispatcher:
|
||||
* keep-alive connections per origin with no socket limit and no timeout on
|
||||
* a request in flight; an idle connection closes after 4 s, or the server's
|
||||
* Keep-Alive hint less one second when that is shorter. */
|
||||
static SCR_TL ScrHttpAgent *scr_http_fetch_agents[2];
|
||||
|
||||
ScrHttpClientReq *scr_http_fetch_request(bool secure, ScrStr *host /*borrowed*/, double port,
|
||||
ScrStr *path /*borrowed*/, ScrStr *method /*borrowed*/,
|
||||
ScrArr *header_pairs /*borrowed*/, int default_port,
|
||||
void (*wrap)(ScrNetSocket *, void *), void *wrap_ctx,
|
||||
ScrStr *pool_key /*borrowed*/) {
|
||||
ScrHttpAgent **slot = &scr_http_fetch_agents[secure ? 1 : 0];
|
||||
if (*slot == NULL) {
|
||||
*slot = scr_http_agent_alloc(secure, true, -1, -1, -1, -1, -1, false);
|
||||
(*slot)->idle_ms = 4000;
|
||||
}
|
||||
c->pool_key = pool_key ? scr_str_retain(pool_key) : NULL;
|
||||
if (ag->timeout_ms >= 0 && timeout_ms <= 0) {
|
||||
c->timeout_ms = ag->timeout_ms;
|
||||
scr_net_sock_set_timeout(c->sock, ag->timeout_ms);
|
||||
}
|
||||
if (reuse != NULL) scr_http_client_rebind(c, reuse, true);
|
||||
else if (!queue) scr_http_conn_count(c->conn, ag->sock_count); /* dialed now */
|
||||
c->agent = scr_http_agent_retain(ag);
|
||||
ScrHttpAgentEnt *e = calloc(1, sizeof *e);
|
||||
if (!e) scr_http_oom();
|
||||
e->name = name; /* ownership moves */
|
||||
e->client = scr_http_client_retain(c);
|
||||
e->queued = queue;
|
||||
ScrHttpAgentEnt **link = &ag->ents;
|
||||
while (*link) link = &(*link)->next;
|
||||
*link = e;
|
||||
return c;
|
||||
return scr_http_request_pooled(*slot, host, port, path, method, 0, header_pairs, false, NULL, NULL,
|
||||
default_port, wrap, wrap_ctx, pool_key, NULL);
|
||||
}
|
||||
|
||||
ScrHttpClientReq *scr_http_request_agent(ScrStr *host /*borrowed*/, double port,
|
||||
@@ -5811,6 +5861,8 @@ static void scr_http_agents_cleanup(void) {
|
||||
for (size_t i = 0; i < 2; i++) {
|
||||
scr_dyn_release(scr_http_global_agents[i]);
|
||||
scr_http_global_agents[i] = NULL;
|
||||
if (scr_http_fetch_agents[i]) scr_http_agent_release(scr_http_fetch_agents[i]);
|
||||
scr_http_fetch_agents[i] = NULL;
|
||||
}
|
||||
while (scr_http_agents) {
|
||||
ScrHttpAgent *a = scr_http_agents;
|
||||
|
||||
@@ -365,6 +365,14 @@ ScrHttpClientReq *scr_http_request_agent_ex(ScrStr *host /*borrowed*/, double po
|
||||
ScrStr *pool_key /*borrowed, nullable: https TLS options*/); /* +1 */
|
||||
/* http.globalAgent / https.globalAgent: borrowed, and as a +1 value. */
|
||||
const ScrDyn *scr_http_global_agent(bool secure);
|
||||
/* fetch()'s client request for a direct connection: through fetch's own
|
||||
* keep-alive agent, which reuses an idle connection with the same host,
|
||||
* port and pool_key (+1). */
|
||||
ScrHttpClientReq *scr_http_fetch_request(bool secure, ScrStr *host /*borrowed*/, double port,
|
||||
ScrStr *path /*borrowed*/, ScrStr *method /*borrowed*/,
|
||||
ScrArr *header_pairs /*borrowed*/, int default_port,
|
||||
void (*wrap)(ScrNetSocket *, void *), void *wrap_ctx,
|
||||
ScrStr *pool_key /*borrowed*/);
|
||||
ScrDyn *scr_http_global_agent_value(bool secure);
|
||||
ScrHttpClientReq *scr_http_request_agent(ScrStr *host /*borrowed*/, double port,
|
||||
ScrStr *path /*borrowed*/, ScrStr *method /*borrowed*/,
|
||||
|
||||
@@ -7004,7 +7004,14 @@ static ScrDyn *scr_dynh_method_call(ScrClosure *closure, ScrDyn *const *args, si
|
||||
ScrDyn *self = scr_dyn_this_get();
|
||||
ScrDyn *result;
|
||||
if (self->kind == SCR_DYN_HANDLE && self->v.handle.tag == tag) {
|
||||
#ifdef SCR_HANDLE_CLASSES
|
||||
/* A class's method value reaches the class's method past the handle's
|
||||
* own properties; this build is the one programs with such classes
|
||||
* link. */
|
||||
result = scr_dyn_handle_class_method_call(ops, self, method->name, args, argc);
|
||||
#else
|
||||
result = ops->invoke(self->v.handle.ptr, self, method->name, args, argc, method->name);
|
||||
#endif
|
||||
} else {
|
||||
const char *message = method->invalid_this ? method->invalid_this
|
||||
: ops->invalid_this ? ops->invalid_this
|
||||
|
||||
@@ -2066,6 +2066,48 @@ bool scr_process_has_exit_code(void) {
|
||||
return scr_exit_code_assigned || scr_exit_code_hint_get() != 0;
|
||||
}
|
||||
|
||||
#if defined(__linux__) && !defined(SCR_LIB)
|
||||
#include <link.h>
|
||||
#include <sys/mman.h>
|
||||
/* Text on huge pages (huge-text.ts). An executable with enough program code
|
||||
* is linked with 2 MiB segment alignment, so each 2 MiB window inside its
|
||||
* text segment starts at a 2 MiB file offset, and the kernel maps a window
|
||||
* with one huge page when the page cache holds it as one 2 MiB folio.
|
||||
* MADV_HUGEPAGE on those windows makes the kernel read a window that is not
|
||||
* cached as a 2 MiB folio. glibc runs it from .preinit_array, before _init
|
||||
* and the constructors fault in any of the text; musl, which has no
|
||||
* .preinit_array, runs it as a constructor. Executables with 4 KiB alignment
|
||||
* make no call. */
|
||||
#define SCR_TEXT_HUGE_PAGE ((uintptr_t)2 << 20)
|
||||
|
||||
static int scr_text_segment(struct dl_phdr_info *info, size_t size, void *data) {
|
||||
(void)size;
|
||||
uintptr_t *range = data;
|
||||
for (size_t i = 0; i < info->dlpi_phnum; i++) {
|
||||
const ElfW(Phdr) *ph = &info->dlpi_phdr[i];
|
||||
if (ph->p_type != PT_LOAD || !(ph->p_flags & PF_X) || ph->p_align < SCR_TEXT_HUGE_PAGE) continue;
|
||||
range[0] = (uintptr_t)info->dlpi_addr + (uintptr_t)ph->p_vaddr;
|
||||
range[1] = range[0] + (uintptr_t)ph->p_memsz;
|
||||
break;
|
||||
}
|
||||
return 1; /* the first object is the executable */
|
||||
}
|
||||
|
||||
static bool scr_text_huge_pages_done;
|
||||
|
||||
__attribute__((constructor)) static void scr_text_huge_pages(void) {
|
||||
if (scr_text_huge_pages_done) return;
|
||||
scr_text_huge_pages_done = true;
|
||||
uintptr_t range[2] = {0, 0};
|
||||
(void)dl_iterate_phdr(scr_text_segment, range);
|
||||
uintptr_t lo = (range[0] + SCR_TEXT_HUGE_PAGE - 1) & ~(SCR_TEXT_HUGE_PAGE - 1);
|
||||
uintptr_t hi = range[1] & ~(SCR_TEXT_HUGE_PAGE - 1);
|
||||
if (hi > lo) (void)madvise((void *)lo, hi - lo, MADV_HUGEPAGE);
|
||||
}
|
||||
__attribute__((section(".preinit_array"), used)) static void (*const scr_text_huge_pages_entry)(void) =
|
||||
scr_text_huge_pages;
|
||||
#endif
|
||||
|
||||
#if defined(__linux__)
|
||||
#include <sched.h>
|
||||
#include <signal.h>
|
||||
|
||||
@@ -14,7 +14,14 @@ test("number-key maps keep Map semantics across linear, direct and hashed indexe
|
||||
try {
|
||||
await exec("clang", [
|
||||
"-std=c11",
|
||||
"-O1",
|
||||
// The runtime link set includes scr_json.c, scr_lib.c and scr_hash.c.
|
||||
// With ASan/UBSan their optimized builds dominate the test's cost: -O1
|
||||
// takes about 90s (scr_hash.c alone 29s, its unrolled digest rounds
|
||||
// instrumented) and regularly starved past this test's timeout when the
|
||||
// lane shared the box. The test checks Map semantics under sanitizers,
|
||||
// not codegen, so -O0 keeps every assertion and sanitizer check and
|
||||
// builds in about 15s.
|
||||
"-O0",
|
||||
"-Wall",
|
||||
"-Wextra",
|
||||
"-fsanitize=address,undefined",
|
||||
|
||||
@@ -2161,6 +2161,7 @@ static ScrStr *scr_str_case_conv(const ScrStr *s, int to_lower) {
|
||||
out->data[i] = (char)(c >= first && c <= last ? c ^ 0x20 : c);
|
||||
}
|
||||
out->data[s->len] = '\0';
|
||||
scr_str_note_ascii(out);
|
||||
return out;
|
||||
}
|
||||
ScrJsonBuf b;
|
||||
|
||||
@@ -2118,6 +2118,14 @@ double scr_str_utf16_len(ScrStr *s) {
|
||||
return (double)(int64_t)units;
|
||||
}
|
||||
|
||||
/* scr_str_utf16_len as a size: proven-ASCII and short strings are counted
|
||||
* here, without the call. */
|
||||
static inline size_t scr_str_len16(ScrStr *s) {
|
||||
if (s->ascii) return s->len;
|
||||
if (s->len <= 64) return scr_utf16_units_short(s->data, s->len);
|
||||
return (size_t)scr_str_utf16_len(s);
|
||||
}
|
||||
|
||||
/* Records the UTF-16 length of a string whose producer counted it (latin1
|
||||
* text has one unit per byte), so scr_str_utf16_len does not walk it. Only
|
||||
* strings that scr_str_utf16_len would otherwise record are kept. */
|
||||
@@ -3137,15 +3145,17 @@ ScrStr *scr_str_replace_literal(ScrStr *s, ScrStr *search, ScrStr *rep, bool all
|
||||
static ScrStr *scr_pad_impl(ScrStr *s, double maxLength, ScrStr *fill,
|
||||
bool at_start) {
|
||||
double target = scr_to_integer_or_infinity(maxLength);
|
||||
size_t len16 = scr_sidx_len(s, scr_sidx_for(s));
|
||||
size_t len16 = scr_str_len16(s);
|
||||
if (!(target > (double)len16) || fill->len == 0) return scr_str_retain(s);
|
||||
/* Reject pad sizes malloc could not satisfy before the double→size_t
|
||||
* conversion can overflow (each unit is at most 3 bytes here: BMP chars
|
||||
* and lone surrogates; astral chars are 4 bytes for 2 units). */
|
||||
if (target > (double)((SIZE_MAX - sizeof(ScrStr) - 1) / 4)) scr_oom();
|
||||
size_t pad16 = (size_t)target - len16;
|
||||
size_t fill16 = scr_sidx_len(fill, scr_sidx_for(fill));
|
||||
size_t reps = pad16 / fill16, rem16 = pad16 % fill16;
|
||||
size_t fill16 = scr_str_len16(fill);
|
||||
/* A one-unit fill, the common case, needs no division. */
|
||||
size_t reps = fill16 == 1 ? pad16 : pad16 / fill16;
|
||||
size_t rem16 = fill16 == 1 ? 0 : pad16 % fill16;
|
||||
/* The truncated prefix of fill: whole chars while they fit in rem16
|
||||
* units; a final astral char that doesn't fit contributes its high
|
||||
* surrogate. */
|
||||
@@ -3167,9 +3177,14 @@ static ScrStr *scr_pad_impl(ScrStr *s, double maxLength, ScrStr *fill,
|
||||
size_t total = s->len + pad_b;
|
||||
ScrStr *r = scr_str_alloc(total, total);
|
||||
char *w = r->data + (at_start ? 0 : s->len);
|
||||
for (size_t i = 0; i < reps; i++) {
|
||||
memcpy(w, fill->data, fill->len);
|
||||
w += fill->len;
|
||||
if (fill->len == 1) {
|
||||
memset(w, fill->data[0], reps);
|
||||
w += reps;
|
||||
} else {
|
||||
for (size_t i = 0; i < reps; i++) {
|
||||
memcpy(w, fill->data, fill->len);
|
||||
w += fill->len;
|
||||
}
|
||||
}
|
||||
memcpy(w, fill->data, prefix_b);
|
||||
w += prefix_b;
|
||||
@@ -3179,6 +3194,7 @@ static ScrStr *scr_pad_impl(ScrStr *s, double maxLength, ScrStr *fill,
|
||||
}
|
||||
memcpy(at_start ? w : r->data, s->data, s->len);
|
||||
r->data[total] = '\0';
|
||||
if (s->ascii && fill->ascii) scr_str_note_ascii(r);
|
||||
/* Only a piece that starts with a lone low surrogate can complete a
|
||||
* pair across a seam: the fill (after a fill or the receiver) or the
|
||||
* receiver (after a fill ending in a high surrogate). */
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -502,6 +502,86 @@ export function compareCounts(baseline, candidate) {
|
||||
};
|
||||
}
|
||||
|
||||
/* ── runtime packs ─────────────────────────────────────────────────────── */
|
||||
|
||||
/** Every file a runtime-pack.json lists (each flavor's runtime unit
|
||||
* variants, their bitcode, and the vendor archives), as path → sha256. */
|
||||
export function runtimePackFiles(manifest) {
|
||||
const files = new Map();
|
||||
const visit = (value) => {
|
||||
if (Array.isArray(value)) value.forEach(visit);
|
||||
else if (value !== null && typeof value === "object") {
|
||||
if (typeof value.path === "string" && typeof value.sha256 === "string")
|
||||
files.set(value.path, value.sha256);
|
||||
Object.values(value).forEach(visit);
|
||||
}
|
||||
};
|
||||
visit(manifest.flavors ?? {});
|
||||
visit(manifest.archives ?? {});
|
||||
return files;
|
||||
}
|
||||
|
||||
/** The runtime pack files that differ between two runtime-pack.json
|
||||
* manifests: changed, added and removed paths, each sorted. */
|
||||
export function runtimePackDiff(baseline, candidate) {
|
||||
const before = runtimePackFiles(baseline);
|
||||
const after = runtimePackFiles(candidate);
|
||||
const changed = [...before].filter(([path, sha]) => after.has(path) && after.get(path) !== sha);
|
||||
return {
|
||||
changed: changed.map(([path]) => path).sort(),
|
||||
added: [...after.keys()].filter((path) => !before.has(path)).sort(),
|
||||
removed: [...before.keys()].filter((path) => !after.has(path)).sort(),
|
||||
};
|
||||
}
|
||||
|
||||
/** The changed runtime pack files whose symbols executables link: runtime
|
||||
* objects and vendor archives. */
|
||||
export function linkedRuntimeFiles(diff) {
|
||||
return [...diff.changed, ...diff.added, ...diff.removed].filter(
|
||||
(path) => path.endsWith(".o") || path.endsWith(".a"),
|
||||
);
|
||||
}
|
||||
|
||||
/** Defined functions of textual LLVM IR (llvm-dis output) as name → hash of
|
||||
* the body. Attribute group and metadata numbers are dropped first, since
|
||||
* an unrelated change elsewhere in the unit renumbers them. */
|
||||
export function llvmFunctionHashes(text) {
|
||||
const functions = new Map();
|
||||
const lines = text.split("\n");
|
||||
for (let index = 0; index < lines.length; index++) {
|
||||
const head = /^define\b[^@]*@("(?:[^"\\]|\\.)*"|[\w.$-]+)\(/.exec(lines[index]);
|
||||
if (head === null) continue;
|
||||
const body = [];
|
||||
for (; index < lines.length && lines[index] !== "}"; index++) body.push(lines[index]);
|
||||
const normalized = body.join("\n").replace(/ #\d+/g, "").replace(/!\d+/g, "!");
|
||||
functions.set(head[1].replace(/^"|"$/g, ""), hash(normalized));
|
||||
}
|
||||
return functions;
|
||||
}
|
||||
|
||||
/** Names whose hashes differ between two function maps, and the names only
|
||||
* one side defines. */
|
||||
export function changedFunctions(before, after) {
|
||||
return {
|
||||
changed: [...before]
|
||||
.filter(([name, h]) => after.has(name) && after.get(name) !== h)
|
||||
.map(([name]) => name)
|
||||
.sort(),
|
||||
added: [...after.keys()].filter((name) => !before.has(name)).sort(),
|
||||
removed: [...before.keys()].filter((name) => !after.has(name)).sort(),
|
||||
};
|
||||
}
|
||||
|
||||
/** Whether every symbol an executable comparison added, removed or resized
|
||||
* is one of `runtimeSymbols`, the symbols the changed runtime objects
|
||||
* define. Mach-O nm prints no sizes, so there only additions and removals
|
||||
* are checked. */
|
||||
export function confinedToRuntime(exe, runtimeSymbols) {
|
||||
return [...exe.added, ...exe.removed, ...exe.resized.map((s) => s.name)].every((name) =>
|
||||
runtimeSymbols.has(name),
|
||||
);
|
||||
}
|
||||
|
||||
/** The report lines the gate prints for one program. */
|
||||
export function programLines(entry) {
|
||||
const lines = [];
|
||||
@@ -510,6 +590,8 @@ export function programLines(entry) {
|
||||
identical: "identical",
|
||||
allowed: "allowed ",
|
||||
differs: "DIFFERS ",
|
||||
"runtime-only": "runtime ",
|
||||
nondeterministic: "nondet ",
|
||||
"build-error": "ERROR ",
|
||||
"both-fail": "both-fail",
|
||||
}[entry.verdict];
|
||||
@@ -536,6 +618,17 @@ export function programLines(entry) {
|
||||
);
|
||||
} else if (entry.cachegrind?.skipped)
|
||||
notes.push(`cachegrind skipped: ${entry.cachegrind.skipped}`);
|
||||
if (entry.runtime) {
|
||||
const linked = [...entry.runtime.changed, ...entry.runtime.added];
|
||||
notes.push(
|
||||
`runtime-only: links ${linked.length} changed runtime function${linked.length === 1 ? "" : "s"}` +
|
||||
(linked.length
|
||||
? ` (${linked.slice(0, 8).join(", ")}${linked.length > 8 ? ", ..." : ""})`
|
||||
: ""),
|
||||
);
|
||||
}
|
||||
if (entry.nondeterministic)
|
||||
notes.push(`the baseline's own rebuild differs in ${entry.nondeterministic.join(", ")}`);
|
||||
for (const failure of entry.dynamicTier?.failures ?? []) notes.push(`SYMBOLS: ${failure}`);
|
||||
lines.push(`${label} ${entry.id}${notes.length ? ` ${notes.join("; ")}` : ""}`);
|
||||
// A build error's first line is often a warning; print all of it.
|
||||
@@ -560,7 +653,23 @@ export function programLines(entry) {
|
||||
/** `required` programs must build with both checkouts: a failure both share
|
||||
* means the gate compared nothing for them. Other programs may fail the same
|
||||
* way on both sides, as a corpus program the compiler refuses does. */
|
||||
export function programVerdict({ baseline, candidate, artifacts, allowed, required = false }) {
|
||||
/** A difference in the release executable alone, with identical IR and
|
||||
* LLVM, is "runtime-only" when `runtimeOnly` says the changed runtime
|
||||
* objects explain it (the gate checks the runtime packs and the symbols).
|
||||
* A difference the baseline also shows when it builds the program again is
|
||||
* "nondeterministic": `nondeterministic` lists the artifact kinds whose
|
||||
* baseline rebuild differs, and every differing artifact must be among
|
||||
* them. Neither fails the gate unless `strict` is set. */
|
||||
export function programVerdict({
|
||||
baseline,
|
||||
candidate,
|
||||
artifacts,
|
||||
allowed,
|
||||
required = false,
|
||||
runtimeOnly = false,
|
||||
nondeterministic = [],
|
||||
strict = false,
|
||||
}) {
|
||||
if (!baseline.ok || !candidate.ok) {
|
||||
if (!baseline.ok && !candidate.ok && baseline.error === candidate.error)
|
||||
return { verdict: "both-fail", fails: required };
|
||||
@@ -568,7 +677,12 @@ export function programVerdict({ baseline, candidate, artifacts, allowed, requir
|
||||
}
|
||||
const differing = Object.entries(artifacts).filter(([, r]) => !r.identical);
|
||||
if (differing.length === 0) return { verdict: "identical", fails: false };
|
||||
return allowed ? { verdict: "allowed", fails: false } : { verdict: "differs", fails: true };
|
||||
if (allowed) return { verdict: "allowed", fails: false };
|
||||
if (runtimeOnly && differing.every(([kind]) => kind === "exe"))
|
||||
return { verdict: "runtime-only", fails: strict };
|
||||
if (differing.every(([kind]) => nondeterministic.includes(kind)))
|
||||
return { verdict: "nondeterministic", fails: strict };
|
||||
return { verdict: "differs", fails: true };
|
||||
}
|
||||
|
||||
export function median(values) {
|
||||
@@ -737,7 +851,8 @@ export function mergeReports(reports) {
|
||||
}
|
||||
const counts = {};
|
||||
for (const p of merged.programs) counts[p.verdict] = (counts[p.verdict] ?? 0) + 1;
|
||||
if (!merged.programs.some((p) => ["identical", "allowed", "differs"].includes(p.verdict)))
|
||||
const compared = ["identical", "allowed", "runtime-only", "nondeterministic", "differs"];
|
||||
if (!merged.programs.some((p) => compared.includes(p.verdict)))
|
||||
merged.globalFailures.push("no program built with both checkouts, so nothing was compared");
|
||||
merged.failures.push(...merged.globalFailures);
|
||||
merged.counts = counts;
|
||||
|
||||
+227
-15
@@ -10,7 +10,11 @@
|
||||
* The good-code set: the runtime benchmark workloads (and benchmarks/builds
|
||||
* applications), the self-hosted compiler (IR and LLVM), and the TypeScript
|
||||
* corpus programs. A program whose sources match --allow (the change's
|
||||
* pattern signature) may differ; any other difference fails the gate. */
|
||||
* pattern signature) may differ. A program whose IR and LLVM are identical
|
||||
* and whose executable differs only through changed runtime objects is
|
||||
* "runtime-only", and a difference the baseline reproduces against itself is
|
||||
* "nondeterministic": both pass with a note unless --strict. Any other
|
||||
* difference fails the gate. */
|
||||
import assert from "node:assert/strict";
|
||||
import { spawn, spawnSync } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
@@ -33,19 +37,25 @@ import {
|
||||
artifactHash,
|
||||
cachedComparison,
|
||||
cachegrindCounts,
|
||||
changedFunctions,
|
||||
compareArtifact,
|
||||
compareCounts,
|
||||
compareExecutables,
|
||||
confinedToRuntime,
|
||||
DYNAMIC_WORKLOADS,
|
||||
definedFunctionNames,
|
||||
dynamicTierSymbols,
|
||||
linkedRuntimeFiles,
|
||||
llvmFunctionHashes,
|
||||
median,
|
||||
nodeIslandSymbols,
|
||||
nmOutput,
|
||||
parseNm,
|
||||
programInputsHash,
|
||||
programLines,
|
||||
programVerdict,
|
||||
recordedBaseline,
|
||||
runtimePackDiff,
|
||||
timingGroups,
|
||||
timingVerdict,
|
||||
} from "./zero-cost-compare.mjs";
|
||||
@@ -62,6 +72,7 @@ const { values } = parseArgs({
|
||||
shard: { type: "string" },
|
||||
only: { type: "string" },
|
||||
allow: { type: "string" },
|
||||
strict: { type: "boolean", default: false },
|
||||
"timing-runs": { type: "string" },
|
||||
"self-timing-runs": { type: "string", default: "0" },
|
||||
"max-time-growth": { type: "string", default: "2" },
|
||||
@@ -79,7 +90,7 @@ if (values.help || values.baseline === undefined) {
|
||||
console.log(`Usage: node scripts/zero-cost-gate.mjs --baseline=<checkout> [--candidate=<checkout>]
|
||||
[--programs=baseline|candidate|<dir>] [--set=bench,self,corpus] [--quick]
|
||||
[--corpus=ts|all] [--shard=i/n] [--only=<regex>] [--allow=<regex>]
|
||||
[--timing-runs=N] [--self-timing-runs=N] [--max-time-growth=2]
|
||||
[--strict] [--timing-runs=N] [--self-timing-runs=N] [--max-time-growth=2]
|
||||
[--max-size-growth=0]
|
||||
[--cachegrind] [--jobs=N] [--json=<file>] [--keep]
|
||||
[--baseline-cache=<file>] [--record-baseline-cache=<file>]
|
||||
@@ -98,8 +109,20 @@ in shard 1. scripts/zero-cost-merge.mjs merges the shards' --json reports
|
||||
into one verdict. --only filters program ids.
|
||||
|
||||
--allow=<regex> is the change's pattern signature: programs whose source
|
||||
files match it may differ (reported as "allowed"). Any other difference in
|
||||
IR, LLVM or the release executable fails the gate.
|
||||
files match it may differ (reported as "allowed").
|
||||
|
||||
Other differences are classified:
|
||||
runtime-only IR and LLVM identical; the executable differs only
|
||||
through runtime objects that differ between the two
|
||||
checkouts' runtime packs (every symbol the executable
|
||||
gains, loses or resizes is defined by one of them), with
|
||||
the same LLVM helper. Reported with the changed runtime
|
||||
functions it links and its size change.
|
||||
nondeterministic the baseline, building the program a second time,
|
||||
differs from its own first build in the same artifacts.
|
||||
differs anything else, in IR, LLVM or the executable.
|
||||
"differs" fails the gate. runtime-only and nondeterministic pass with a
|
||||
note; --strict fails them too, so any difference fails as before.
|
||||
|
||||
--timing-runs=N (default 5) times N paired --emit=llvm builds per bench
|
||||
program, baseline and candidate side by side. The self-hosted compiler's
|
||||
@@ -290,10 +313,9 @@ function compile(tree, args) {
|
||||
* keeping the LLVM and IR beside the release executable. The self-hosted
|
||||
* compiler emits LLVM only: each of its frontend runs takes minutes, and its
|
||||
* LLVM names the first differing function as well as its IR would. */
|
||||
async function buildProgram(program, label) {
|
||||
async function buildProgram(program, label, tree = trees[label]) {
|
||||
const dir = join(work, label, program.id.replace(/[^\w.-]+/g, "_"));
|
||||
mkdirSync(dir, { recursive: true });
|
||||
const tree = trees[label];
|
||||
const failure = (result) => ({
|
||||
ok: false,
|
||||
error: (result.stderr || result.stdout || `exit ${result.status ?? result.signal}`)
|
||||
@@ -434,6 +456,111 @@ async function instructionCounts(binary, args, out) {
|
||||
return cachegrindCounts(readFileSync(out, "utf8"), new Set(definedFunctions(binary)));
|
||||
}
|
||||
|
||||
/* ── runtime changes ───────────────────────────────────────────────────── */
|
||||
|
||||
const sha256 = (file) => createHash("sha256").update(readFileSync(file)).digest("hex");
|
||||
|
||||
/** llvm-dis from PATH or the pinned LLVM install, or null. */
|
||||
function llvmDis() {
|
||||
for (const candidate of ["llvm-dis", "llvm-dis-22", "/usr/lib/llvm-22/bin/llvm-dis"])
|
||||
if (spawnSync(candidate, ["--version"]).status === 0) return candidate;
|
||||
const config = spawnSync("llvm-config", ["--bindir"], { encoding: "utf8" });
|
||||
const fromConfig = config.status === 0 ? join(config.stdout.trim(), "llvm-dis") : null;
|
||||
return fromConfig !== null && existsSync(fromConfig) ? fromConfig : null;
|
||||
}
|
||||
|
||||
/** What differs between the two checkouts' runtime packs for this host, or
|
||||
* null when no object or archive does: the changed pack files, whether the
|
||||
* LLVM helper is the same, the symbols the changed objects define (on either
|
||||
* side), and the functions that changed in the bitcode of the changed
|
||||
* release and speed objects. Bitcode alone is not a change: it records the
|
||||
* checkout's source path, so it differs between any two checkouts. */
|
||||
function runtimeChanges() {
|
||||
const pack = (root) =>
|
||||
readdirSync(join(root, "packages"))
|
||||
.filter((name) => name.startsWith("runtime-"))
|
||||
.filter((name) => existsSync(join(root, "packages", name, "runtime-pack.json")));
|
||||
const shared = pack(trees.candidate).filter((name) => pack(trees.baseline).includes(name));
|
||||
if (shared.length !== 1) return null;
|
||||
const packDir = (root) => join(root, "packages", shared[0]);
|
||||
const manifest = (root) =>
|
||||
JSON.parse(readFileSync(join(packDir(root), "runtime-pack.json"), "utf8"));
|
||||
const linked = (paths) => paths.filter((path) => !path.endsWith(".bc"));
|
||||
const all = runtimePackDiff(manifest(trees.baseline), manifest(trees.candidate));
|
||||
const diff = {
|
||||
changed: linked(all.changed),
|
||||
added: linked(all.added),
|
||||
removed: linked(all.removed),
|
||||
};
|
||||
if (diff.changed.length + diff.added.length + diff.removed.length === 0) return null;
|
||||
const helper = (root) => {
|
||||
const file = join(
|
||||
root,
|
||||
"packages",
|
||||
shared[0].replace(/^runtime-/, "llvm-"),
|
||||
"bin/scriptc-llvm-codegen",
|
||||
);
|
||||
return existsSync(file) ? sha256(file) : null;
|
||||
};
|
||||
const helperSame =
|
||||
helper(trees.baseline) !== null && helper(trees.baseline) === helper(trees.candidate);
|
||||
const symbols = new Set();
|
||||
const objects = [...diff.changed, ...diff.added, ...diff.removed].filter((path) =>
|
||||
path.endsWith(".o"),
|
||||
);
|
||||
for (const path of linkedRuntimeFiles(diff))
|
||||
for (const root of [trees.baseline, trees.candidate]) {
|
||||
const file = join(packDir(root), path);
|
||||
if (!existsSync(file)) continue;
|
||||
const listed = spawnSync("nm", ["--defined-only", file], {
|
||||
encoding: "utf8",
|
||||
maxBuffer: 1 << 28,
|
||||
});
|
||||
for (const symbol of parseNm(listed.stdout ?? "")) symbols.add(symbol.name);
|
||||
}
|
||||
// Changed functions, by unit, from the bitcode of the release and speed
|
||||
// flavors (the ones the gate builds with).
|
||||
const dis = llvmDis();
|
||||
const functions = { changed: new Set(), added: new Set(), removed: new Set(), units: new Set() };
|
||||
for (const object of objects)
|
||||
functions.units.add(object.replace(/^.*\//, "").replace(/\.o$/, ".c"));
|
||||
const bitcode = objects
|
||||
.filter((path) => /^artifacts\/(release|speed)\//.test(path))
|
||||
.map((path) => path.replace(/\.o$/, ".bc"));
|
||||
for (const path of bitcode) {
|
||||
if (dis === null) continue;
|
||||
const text = (root) => {
|
||||
const file = join(packDir(root), path);
|
||||
if (!existsSync(file)) return new Map();
|
||||
const out = spawnSync(dis, [file, "-o", "-"], { encoding: "utf8", maxBuffer: 1 << 30 });
|
||||
return llvmFunctionHashes(out.stdout ?? "");
|
||||
};
|
||||
const delta = changedFunctions(text(trees.baseline), text(trees.candidate));
|
||||
for (const kind of ["changed", "added", "removed"])
|
||||
for (const name of delta[kind]) functions[kind].add(name);
|
||||
}
|
||||
return {
|
||||
pack: shared[0],
|
||||
diff,
|
||||
helperSame,
|
||||
symbols,
|
||||
disassembler: dis,
|
||||
functions: Object.fromEntries(
|
||||
Object.entries(functions).map(([kind, names]) => [kind, [...names].sort()]),
|
||||
),
|
||||
};
|
||||
}
|
||||
|
||||
/** The changed runtime functions an executable defines. */
|
||||
function linkedRuntimeChanges(runtime, binary) {
|
||||
const defined = new Set(definedFunctions(binary));
|
||||
const pick = (names) => names.filter((name) => defined.has(name));
|
||||
return {
|
||||
changed: pick(runtime.functions.changed),
|
||||
added: pick(runtime.functions.added),
|
||||
};
|
||||
}
|
||||
|
||||
/* ── main ──────────────────────────────────────────────────────────────── */
|
||||
|
||||
function revision(root) {
|
||||
@@ -523,6 +650,7 @@ if (recordPath === undefined && values["baseline-cache"] !== undefined) {
|
||||
|
||||
const programs = goodCodeSet().filter((p) => recordPath === undefined || p.kind !== "source");
|
||||
assert.ok(programs.length > 0, "the selected good-code set is empty");
|
||||
const runtime = recordPath === undefined ? runtimeChanges() : null;
|
||||
const report = {
|
||||
schema: 1,
|
||||
date: new Date().toISOString(),
|
||||
@@ -532,6 +660,16 @@ const report = {
|
||||
),
|
||||
programsRoot,
|
||||
allow: values.allow ?? null,
|
||||
strict: values.strict,
|
||||
runtime:
|
||||
runtime === null
|
||||
? null
|
||||
: {
|
||||
pack: runtime.pack,
|
||||
...runtime.diff,
|
||||
helperSame: runtime.helperSame,
|
||||
functions: runtime.functions,
|
||||
},
|
||||
programs: [],
|
||||
timing: null,
|
||||
failures: [],
|
||||
@@ -547,6 +685,24 @@ console.log(
|
||||
`zero-cost gate: ${programs.length} programs (${[...sets].join(", ")}), ` +
|
||||
`baseline ${trees.baseline}, candidate ${trees.candidate}, programs from ${programsRoot}`,
|
||||
);
|
||||
if (runtime !== null) {
|
||||
const { changed, added, removed } = runtime.diff;
|
||||
console.log(
|
||||
`runtime pack ${runtime.pack} differs: ${changed.length} changed, ${added.length} added, ` +
|
||||
`${removed.length} removed files; units ${runtime.functions.units.join(", ") || "none"}` +
|
||||
(runtime.helperSame ? "" : "; the LLVM helper differs too, so nothing is runtime-only"),
|
||||
);
|
||||
if (runtime.disassembler === null)
|
||||
console.log("runtime functions: llvm-dis not found, so changed functions are not listed");
|
||||
else
|
||||
console.log(
|
||||
`runtime functions: ${runtime.functions.changed.length} changed` +
|
||||
(runtime.functions.changed.length
|
||||
? ` (${runtime.functions.changed.slice(0, 20).join(", ")})`
|
||||
: "") +
|
||||
`, ${runtime.functions.added.length} added, ${runtime.functions.removed.length} removed`,
|
||||
);
|
||||
}
|
||||
|
||||
let failed = false;
|
||||
const started = Date.now();
|
||||
@@ -641,13 +797,39 @@ await pool(
|
||||
(baseline.ok && candidate.ok ? compareBuilds(baseline, candidate) : {});
|
||||
// The benchmark workloads and the self-hosted compiler always build.
|
||||
const required = !program.id.startsWith("corpus/");
|
||||
const { verdict, fails } = programVerdict({
|
||||
baseline,
|
||||
candidate,
|
||||
artifacts,
|
||||
allowed,
|
||||
required,
|
||||
});
|
||||
const runtimeOnly =
|
||||
runtime !== null &&
|
||||
runtime.helperSame &&
|
||||
artifacts.exe !== undefined &&
|
||||
!artifacts.exe.identical &&
|
||||
confinedToRuntime(artifacts.exe, runtime.symbols);
|
||||
const judge = (nondeterministic) =>
|
||||
programVerdict({
|
||||
baseline,
|
||||
candidate,
|
||||
artifacts,
|
||||
allowed,
|
||||
required,
|
||||
runtimeOnly,
|
||||
nondeterministic,
|
||||
strict: values.strict,
|
||||
});
|
||||
let { verdict, fails } = judge([]);
|
||||
// Build a difference the change cannot explain once more with the
|
||||
// baseline: when the baseline's two builds differ in the same
|
||||
// artifacts, the difference is the build's, not the change's.
|
||||
let recheck = null;
|
||||
if (verdict === "differs") {
|
||||
const again = await buildProgram(program, "recheck", trees.baseline);
|
||||
if (again.ok) {
|
||||
const self = compareBuilds(baseline, again);
|
||||
recheck = Object.keys(artifacts).filter(
|
||||
(kind) => !artifacts[kind].identical && !self[kind]?.identical,
|
||||
);
|
||||
if (again.dir && !values.keep) rmSync(again.dir, { recursive: true, force: true });
|
||||
}
|
||||
if (recheck?.length) ({ verdict, fails } = judge(recheck));
|
||||
}
|
||||
const entry = {
|
||||
id: program.id,
|
||||
verdict,
|
||||
@@ -655,6 +837,8 @@ await pool(
|
||||
buildMs: { baseline: baseline.ms ?? null, candidate: candidate.ms ?? null },
|
||||
artifacts,
|
||||
};
|
||||
if (verdict === "runtime-only") entry.runtime = linkedRuntimeChanges(runtime, candidate.exe);
|
||||
if (verdict === "nondeterministic") entry.nondeterministic = recheck;
|
||||
if (settled !== null) entry.cachedBaseline = true;
|
||||
if (!baseline.ok) entry.baselineError = baseline.error;
|
||||
if (!candidate.ok) entry.candidateError = candidate.error;
|
||||
@@ -787,7 +971,8 @@ if (checkCandidateSelf) {
|
||||
|
||||
const counts = {};
|
||||
for (const p of report.programs) counts[p.verdict] = (counts[p.verdict] ?? 0) + 1;
|
||||
if (!report.programs.some((p) => ["identical", "allowed", "differs"].includes(p.verdict))) {
|
||||
const compared = ["identical", "allowed", "runtime-only", "nondeterministic", "differs"];
|
||||
if (!report.programs.some((p) => compared.includes(p.verdict))) {
|
||||
failed = true;
|
||||
const failure = "no program built with both checkouts, so nothing was compared";
|
||||
report.failures.push(failure);
|
||||
@@ -802,11 +987,38 @@ if (values.json) {
|
||||
}
|
||||
if (!values.keep) rmSync(work, { recursive: true, force: true });
|
||||
else console.log(`artifacts kept in ${work}`);
|
||||
const noted = report.programs.filter((p) =>
|
||||
["runtime-only", "nondeterministic"].includes(p.verdict),
|
||||
);
|
||||
console.log(
|
||||
`zero-cost gate: ${failed ? "FAIL" : "PASS"} — ${Object.entries(counts)
|
||||
`zero-cost gate: ${failed ? "FAIL" : noted.length ? "PASS with notes" : "PASS"} — ${Object.entries(
|
||||
counts,
|
||||
)
|
||||
.map(([k, v]) => `${v} ${k}`)
|
||||
.join(", ")} in ${fmtMs(report.elapsedMs)}` +
|
||||
(report.baselineCache ? ` (${report.baselineCache.used} baselines from the cache)` : ""),
|
||||
);
|
||||
const runtimeOnlyPrograms = report.programs.filter((p) => p.verdict === "runtime-only");
|
||||
if (runtimeOnlyPrograms.length > 0) {
|
||||
const deltas = runtimeOnlyPrograms.map(
|
||||
(p) => p.artifacts.exe.bytes.candidate - p.artifacts.exe.bytes.baseline,
|
||||
);
|
||||
const signed = (n) => `${n > 0 ? "+" : ""}${n}`;
|
||||
const linked = new Set(
|
||||
runtimeOnlyPrograms.flatMap((p) => [...p.runtime.changed, ...p.runtime.added]),
|
||||
);
|
||||
console.log(
|
||||
` note: ${runtimeOnlyPrograms.length} runtime-only programs (IR and LLVM identical); ` +
|
||||
`executable size ${signed(Math.min(...deltas))} to ${signed(Math.max(...deltas))} bytes; ` +
|
||||
`changed runtime functions linked: ${[...linked].sort().join(", ") || "none listed"}` +
|
||||
(values.strict ? "" : " (--strict fails these)"),
|
||||
);
|
||||
}
|
||||
const nondeterministicPrograms = report.programs.filter((p) => p.verdict === "nondeterministic");
|
||||
if (nondeterministicPrograms.length > 0)
|
||||
console.log(
|
||||
` note: ${nondeterministicPrograms.length} nondeterministic programs, whose baseline rebuild differs too: ` +
|
||||
nondeterministicPrograms.map((p) => p.id).join(", "),
|
||||
);
|
||||
for (const failure of report.failures) console.log(` ${failure}`);
|
||||
process.exit(failed ? 1 : 0);
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
// Object.prototype's own-key tests on records of functions convert their
|
||||
// argument to a property key as Node does: an array joins its elements, and
|
||||
// a record or class instance with its own toString calls it. A keyed read
|
||||
// that falls back to an inherited method keeps the bound method alive
|
||||
// through the checked projection that lends it.
|
||||
type ListTest = (key: string[]) => unknown;
|
||||
const lists: Record<string, ListTest> = { a: () => "a", "a,b": () => "a,b" };
|
||||
console.log(
|
||||
lists["hasOwnProperty"](["a"]),
|
||||
lists["hasOwnProperty"](["a", "b"]),
|
||||
lists["propertyIsEnumerable"](["b"]),
|
||||
);
|
||||
|
||||
type Named = { toString(): string };
|
||||
type NamedTest = (key: Named) => unknown;
|
||||
const named: Record<string, NamedTest> = { echo: () => "echo" };
|
||||
let calls = 0;
|
||||
const echoKey: Named = {
|
||||
toString() {
|
||||
calls++;
|
||||
return "echo";
|
||||
},
|
||||
};
|
||||
console.log(
|
||||
named["hasOwnProperty"](echoKey),
|
||||
named["propertyIsEnumerable"]({ toString: () => "nope" }),
|
||||
calls,
|
||||
);
|
||||
|
||||
class Label {
|
||||
readonly text: string;
|
||||
constructor(text: string) {
|
||||
this.text = text;
|
||||
}
|
||||
toString(): string {
|
||||
return this.text;
|
||||
}
|
||||
}
|
||||
type LabelTest = (key: Label) => unknown;
|
||||
const labels: Record<string, LabelTest> = { x: () => "x" };
|
||||
console.log(labels["hasOwnProperty"](new Label("x")), labels["hasOwnProperty"](new Label("y")));
|
||||
|
||||
type Op = (x: string) => unknown;
|
||||
const table: Record<string, Op> = { echo: (x) => x };
|
||||
// `table[name]!` is a checked projection: its successful arm lends the
|
||||
// read, whose miss binds the inherited method to the record.
|
||||
function run(name: string): unknown {
|
||||
return table[name]!("echo");
|
||||
}
|
||||
for (const name of ["echo", "hasOwnProperty", "toString", "propertyIsEnumerable"])
|
||||
console.log(name, String(run(name)));
|
||||
@@ -0,0 +1,29 @@
|
||||
// A conditional receiver whose arm reads a record with a fallback for a
|
||||
// missing key, `(c ? names[k] ?? fallback(k) : other)`: the fallback's
|
||||
// fresh string lives until the read of its length.
|
||||
const names: Record<string, string> = { a: "alpha" };
|
||||
let made = 0;
|
||||
function fallback(k: string): string | undefined {
|
||||
made++;
|
||||
return k === "none" ? undefined : `fallback for ${k} #${made}`.repeat(2);
|
||||
}
|
||||
function asserted(useNames: boolean, k: string, other: string): number {
|
||||
return (useNames ? (names[k] ?? fallback(k)) : other)!.length;
|
||||
}
|
||||
function optional(useNames: boolean, k: string, other: string): number | undefined {
|
||||
return (useNames ? (names[k] ?? fallback(k)) : other)?.length;
|
||||
}
|
||||
function upper(useNames: boolean, k: string, other: string): string | undefined {
|
||||
return (useNames ? (names[k] ?? fallback(k)) : other)?.toUpperCase();
|
||||
}
|
||||
for (const k of ["a", "b", "none"])
|
||||
for (const useNames of [true, false]) {
|
||||
let length: number | string;
|
||||
try {
|
||||
length = asserted(useNames, k, "other");
|
||||
} catch (e) {
|
||||
length = (e as Error).name;
|
||||
}
|
||||
console.log(k, useNames, length, optional(useNames, k, "other"), upper(useNames, k, "x"));
|
||||
}
|
||||
console.log("made", made);
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"strict": true,
|
||||
"noUncheckedIndexedAccess": true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
// fetch() reuses keep-alive connections as Node's fetch does: sequential
|
||||
// requests share connections, and every response is the same with reuse:
|
||||
// concurrent requests, `connection: close` responses and requests,
|
||||
// redirects, request bodies, an aborted response followed by another
|
||||
// request, and a request after the server closed its idle connections. The
|
||||
// program exits while idle connections remain open.
|
||||
import { createServer, type IncomingMessage, type ServerResponse } from "node:http";
|
||||
import type { AddressInfo } from "node:net";
|
||||
|
||||
let connections = 0;
|
||||
const server = createServer((req: IncomingMessage, res: ServerResponse) => {
|
||||
const url = req.url ?? "/";
|
||||
if (url === "/close") {
|
||||
res.writeHead(200, { connection: "close" });
|
||||
res.end("closing");
|
||||
return;
|
||||
}
|
||||
if (url === "/redirect") {
|
||||
res.writeHead(302, { location: "/hello" });
|
||||
res.end();
|
||||
return;
|
||||
}
|
||||
if (url === "/echo") {
|
||||
let body = "";
|
||||
req.setEncoding("utf8");
|
||||
req.on("data", (chunk: string) => {
|
||||
body += chunk;
|
||||
});
|
||||
req.on("end", () => res.end(`echo:${body}`));
|
||||
return;
|
||||
}
|
||||
if (url === "/slow") {
|
||||
res.writeHead(200, { "content-type": "text/plain" });
|
||||
res.write("first ");
|
||||
setTimeout(() => res.end("rest"), 200);
|
||||
return;
|
||||
}
|
||||
res.end(`hello ${url}`);
|
||||
});
|
||||
server.on("connection", () => {
|
||||
connections++;
|
||||
});
|
||||
|
||||
async function text(port: number, path: string, init?: RequestInit): Promise<string> {
|
||||
const res = await fetch(`http://127.0.0.1:${port}${path}`, init);
|
||||
return `${res.status} ${await res.text()}`;
|
||||
}
|
||||
|
||||
async function main(port: number): Promise<void> {
|
||||
const seen: string[] = [];
|
||||
for (let i = 0; i < 5; i++) seen.push(await text(port, `/seq${i}`));
|
||||
console.log("sequential", connections < 5, seen.join(" | "));
|
||||
|
||||
const parallel = await Promise.all([1, 2, 3].map((i) => text(port, `/par${i}`)));
|
||||
for (let i = 0; i < 3; i++) parallel.push(await text(port, `/after${i}`));
|
||||
console.log("concurrent", parallel.join(" | "));
|
||||
|
||||
console.log(await text(port, "/close"), await text(port, "/close"));
|
||||
console.log(await text(port, "/a", { headers: { connection: "close" } }), await text(port, "/b"));
|
||||
|
||||
const res = await fetch(`http://127.0.0.1:${port}/redirect`);
|
||||
console.log("redirect", res.redirected, res.url.endsWith("/hello"), await res.text());
|
||||
|
||||
const bodies = [];
|
||||
for (let i = 0; i < 3; i++) bodies.push(await text(port, "/echo", { method: "POST", body: `n${i}` }));
|
||||
console.log("post", bodies.join(" | "));
|
||||
|
||||
const controller = new AbortController();
|
||||
const slow = await fetch(`http://127.0.0.1:${port}/slow`, { signal: controller.signal });
|
||||
controller.abort();
|
||||
try {
|
||||
await slow.text();
|
||||
} catch (e) {
|
||||
console.log("aborted", (e as Error).name);
|
||||
}
|
||||
console.log(await text(port, "/after-abort"));
|
||||
|
||||
// A server that closes idle connections after 50 ms: the next request
|
||||
// after a pause dials again.
|
||||
const brief = createServer((req: IncomingMessage, res: ServerResponse) => res.end(`brief ${req.url}`));
|
||||
brief.keepAliveTimeout = 50;
|
||||
await new Promise<void>((resolve) => brief.listen(0, "127.0.0.1", () => resolve()));
|
||||
const briefPort = (brief.address() as AddressInfo).port;
|
||||
console.log(await text(briefPort, "/1"), await text(briefPort, "/2"));
|
||||
await new Promise<void>((resolve) => setTimeout(resolve, 200));
|
||||
console.log(await text(briefPort, "/3"));
|
||||
|
||||
brief.close();
|
||||
server.close();
|
||||
console.log("done");
|
||||
}
|
||||
|
||||
server.listen(0, "127.0.0.1", () => {
|
||||
void main((server.address() as AddressInfo).port);
|
||||
});
|
||||
@@ -0,0 +1,66 @@
|
||||
// A compressed fetch() response whose whole body arrived before it was read:
|
||||
// the decoded body is larger than one read, so decoded output is still
|
||||
// buffered when the response ends, and every later read drains it. Reads
|
||||
// after a pause (the end was processed meanwhile) return the rest of the
|
||||
// body, as in Node, for gzip and deflate, through a reader and through
|
||||
// text(), and the next request on the same connection works.
|
||||
import { createServer, type IncomingMessage, type ServerResponse } from "node:http";
|
||||
import type { AddressInfo } from "node:net";
|
||||
import { deflateSync, gzipSync } from "node:zlib";
|
||||
|
||||
const body = Array.from({ length: 20000 }, (_, i) => `line ${i % 97} of the body\n`).join("");
|
||||
const gzipped = gzipSync(body);
|
||||
const deflated = deflateSync(body);
|
||||
|
||||
const server = createServer((req: IncomingMessage, res: ServerResponse) => {
|
||||
const url = req.url ?? "/";
|
||||
if (url === "/gzip" || url === "/gzip-text") {
|
||||
res.writeHead(200, { "content-encoding": "gzip", "content-length": gzipped.length });
|
||||
res.end(gzipped);
|
||||
return;
|
||||
}
|
||||
if (url === "/deflate") {
|
||||
res.writeHead(200, { "content-encoding": "deflate", "content-length": deflated.length });
|
||||
res.end(deflated);
|
||||
return;
|
||||
}
|
||||
res.end(`plain ${url}`);
|
||||
});
|
||||
|
||||
const pause = (ms: number): Promise<void> => new Promise((resolve) => setTimeout(resolve, ms));
|
||||
|
||||
async function readSlowly(port: number, path: string): Promise<string> {
|
||||
const res = await fetch(`http://127.0.0.1:${port}${path}`);
|
||||
const reader = res.body!.getReader();
|
||||
const decoder = new TextDecoder();
|
||||
let text = "";
|
||||
let reads = 0;
|
||||
for (;;) {
|
||||
// Let the response end before most reads.
|
||||
await pause(reads < 3 ? 30 : 0);
|
||||
const { done, value } = await reader.read();
|
||||
if (done) break;
|
||||
reads++;
|
||||
text += decoder.decode(value, { stream: true });
|
||||
}
|
||||
text += decoder.decode();
|
||||
return `${path} ${res.status} ${text.length} ${text === body}`;
|
||||
}
|
||||
|
||||
async function main(port: number): Promise<void> {
|
||||
console.log(await readSlowly(port, "/gzip"));
|
||||
console.log(await readSlowly(port, "/deflate"));
|
||||
const res = await fetch(`http://127.0.0.1:${port}/gzip-text`);
|
||||
await pause(50);
|
||||
const text = await res.text();
|
||||
console.log("text after end", text.length, text === body);
|
||||
const plain = await fetch(`http://127.0.0.1:${port}/after`);
|
||||
console.log(await plain.text());
|
||||
}
|
||||
|
||||
server.listen(0, "127.0.0.1", () => {
|
||||
const { port } = server.address() as AddressInfo;
|
||||
main(port)
|
||||
.catch((error: unknown) => console.log("failed", String(error)))
|
||||
.finally(() => server.close());
|
||||
});
|
||||
@@ -0,0 +1,152 @@
|
||||
// Counted for loops whose limit is an exact integer: an int32 local, an
|
||||
// int32 field, the result of a function returning an int32, a value in a
|
||||
// proven range (a masked byte, an array length), counting up and down,
|
||||
// with exclusive and inclusive bounds, empty ranges and the int32 edges.
|
||||
// Fractional, NaN and infinite limits take the double path and give the
|
||||
// same trip counts as Node.
|
||||
|
||||
function bitCount(x: number): number {
|
||||
x = x - ((x >>> 1) & 0x55555555);
|
||||
x = (x & 0x33333333) + ((x >>> 2) & 0x33333333);
|
||||
return Math.imul((x + (x >>> 4)) & 0x0f0f0f0f, 0x01010101) >>> 24;
|
||||
}
|
||||
|
||||
class Window {
|
||||
low = 0;
|
||||
high = 0;
|
||||
constructor(low: number, high: number) {
|
||||
this.low = low | 0;
|
||||
this.high = high | 0;
|
||||
}
|
||||
}
|
||||
|
||||
function countUp(from: number, to: number): number {
|
||||
let trips = 0;
|
||||
for (let i = from; i < to; i++) trips++;
|
||||
return trips;
|
||||
}
|
||||
function countUpInclusive(from: number, to: number): number {
|
||||
let trips = 0;
|
||||
for (let i = from; i <= to; i++) trips++;
|
||||
return trips;
|
||||
}
|
||||
function countDown(from: number, to: number): number {
|
||||
let trips = 0;
|
||||
for (let i = from; i > to; i--) trips++;
|
||||
return trips;
|
||||
}
|
||||
function countDownInclusive(from: number, to: number): number {
|
||||
let trips = 0;
|
||||
for (let i = from; i >= to; i--) trips++;
|
||||
return trips;
|
||||
}
|
||||
|
||||
// An int32 local limit, as in an insertion that shifts the elements after a rank.
|
||||
function insertAt(values: number[], at: number, value: number): void {
|
||||
values.push(value);
|
||||
for (let i = values.length - 1; i > at; i--) values[i] = values[i - 1]!;
|
||||
values[at] = value;
|
||||
}
|
||||
function removeAt(values: number[], at: number): void {
|
||||
for (let i = at; i < values.length - 1; i++) values[i] = values[i + 1]!;
|
||||
values.pop();
|
||||
}
|
||||
|
||||
const values = [10, 20, 30, 40];
|
||||
const rank = bitCount(0b1011 & 0b0011);
|
||||
insertAt(values, rank, 25);
|
||||
insertAt(values, 0, 5);
|
||||
insertAt(values, values.length - 1, 45);
|
||||
console.log("after inserts", values.join(","));
|
||||
removeAt(values, bitCount(1));
|
||||
removeAt(values, values.length - 1);
|
||||
console.log("after removes", values.join(","));
|
||||
|
||||
// A rank from a method as the limit: a page that keeps its values in id
|
||||
// order and shifts the later ones on insert and remove.
|
||||
class Page {
|
||||
readonly values: number[] = [];
|
||||
bits = 0;
|
||||
rank(slot: number): number {
|
||||
return bitCount(this.bits & ((1 << (slot & 31)) - 1));
|
||||
}
|
||||
set(slot: number, value: number): void {
|
||||
const bit = 1 << (slot & 31);
|
||||
const at = this.rank(slot);
|
||||
const values = this.values;
|
||||
if ((this.bits & bit) !== 0) {
|
||||
for (let i = at; i < values.length - 1; i++) values[i] = values[i + 1]!;
|
||||
values.pop();
|
||||
this.bits = (this.bits & ~bit) | 0;
|
||||
return;
|
||||
}
|
||||
values.push(value);
|
||||
for (let i = values.length - 1; i > at; i--) values[i] = values[i - 1]!;
|
||||
values[at] = value;
|
||||
this.bits = (this.bits | bit) | 0;
|
||||
}
|
||||
// Inclusive bounds at a rank, in both directions.
|
||||
suffix(slot: number): number {
|
||||
const at = this.rank(slot);
|
||||
const values = this.values;
|
||||
let total = 0;
|
||||
for (let i = values.length - 1; i >= at; i--) total += values[i]!;
|
||||
for (let i = 0; i <= at - 1; i++) total -= values[i]!;
|
||||
return total;
|
||||
}
|
||||
}
|
||||
const page = new Page();
|
||||
for (const slot of [5, 1, 9, 3, 31, 0, 9, 1, 17]) page.set(slot, slot * 10);
|
||||
console.log("page", page.values.join(","), page.bits >>> 0);
|
||||
console.log("suffix", page.suffix(0), page.suffix(4), page.suffix(17), page.suffix(31));
|
||||
|
||||
// Limits from fields, calls and proven ranges.
|
||||
const window = new Window(-3, 7);
|
||||
let sum = 0;
|
||||
for (let i = window.low; i < window.high; i++) sum += i;
|
||||
console.log("field limits", sum);
|
||||
sum = 0;
|
||||
for (let i = window.high; i >= window.low; i--) sum += i;
|
||||
console.log("field limits down", sum);
|
||||
let byte = 0;
|
||||
for (let k = 0; k < 6; k++) {
|
||||
const limit = (k * 77) & 0xff;
|
||||
let trips = 0;
|
||||
for (let i = 0; i <= limit; i += 37) trips++;
|
||||
byte += trips;
|
||||
}
|
||||
console.log("masked limits", byte);
|
||||
let fromCalls = 0;
|
||||
for (let i = bitCount(0xff); i > bitCount(0x3); i--) fromCalls++;
|
||||
console.log("call limits", fromCalls);
|
||||
|
||||
// Edges and empty ranges.
|
||||
console.log("up", countUp(0, 5), countUp(5, 0), countUp(-5, -2), countUp(2147483645, 2147483647));
|
||||
console.log(
|
||||
"up inclusive",
|
||||
countUpInclusive(0, 5),
|
||||
countUpInclusive(3, 3),
|
||||
countUpInclusive(2147483645, 2147483647),
|
||||
);
|
||||
console.log("down", countDown(5, 0), countDown(0, 5), countDown(-2147483646, -2147483648));
|
||||
console.log(
|
||||
"down inclusive",
|
||||
countDownInclusive(5, 0),
|
||||
countDownInclusive(-2147483646, -2147483648),
|
||||
countDownInclusive(0, 0),
|
||||
);
|
||||
|
||||
// Limits that are not exact integers.
|
||||
console.log("fractional", countUp(0, 2.5), countUpInclusive(0, 2.5), countDown(3, 0.5), countDownInclusive(3, 0.5));
|
||||
console.log("negative fractional", countUp(-3, -0.5), countDownInclusive(0, -2.5));
|
||||
console.log("NaN", countUp(0, Number.NaN), countDown(3, Number.NaN));
|
||||
let capped = 0;
|
||||
for (let i = 0; i < Number.POSITIVE_INFINITY; i++) {
|
||||
if (++capped === 4) break;
|
||||
}
|
||||
console.log("infinite", capped);
|
||||
let floored = 0;
|
||||
for (let i = 3; i > Number.NEGATIVE_INFINITY; i--) {
|
||||
if (++floored === 5) break;
|
||||
}
|
||||
console.log("negative infinite", floored);
|
||||
@@ -0,0 +1,39 @@
|
||||
// padStart and padEnd on the results of case conversion and slicing, with
|
||||
// ASCII and non-ASCII receivers and fills, and index reads of the results.
|
||||
|
||||
const words = ["alpha", "beta", "gamma", "Delta", "épée", "straße", "日本語", "a😀b"];
|
||||
const fills = ["-", "ab", "é", "😀", "", " "];
|
||||
|
||||
function describe(s: string): string {
|
||||
return [
|
||||
JSON.stringify(s),
|
||||
s.length,
|
||||
s.charAt(1),
|
||||
s.at(-1),
|
||||
s.indexOf("A"),
|
||||
s.slice(1, 4),
|
||||
s.charCodeAt(s.length - 1),
|
||||
s.codePointAt(0),
|
||||
].join(" ");
|
||||
}
|
||||
|
||||
for (const word of words) {
|
||||
const upper = word.slice(0, 3).toUpperCase();
|
||||
const lower = word.toLowerCase();
|
||||
for (const fill of fills) {
|
||||
console.log(describe(upper.padStart(6, fill)));
|
||||
console.log(describe(lower.padEnd(9, fill)));
|
||||
}
|
||||
console.log(describe(upper.padStart(2)), describe(upper.padEnd(5)));
|
||||
}
|
||||
|
||||
// The same string padded again, compared and used as a key.
|
||||
const counts = new Map<string, number>();
|
||||
for (let i = 0; i < 40; i++) {
|
||||
const key = words[i % words.length]!.slice(0, 3).toUpperCase().padStart(4, "-");
|
||||
counts.set(key, (counts.get(key) ?? 0) + 1);
|
||||
}
|
||||
console.log([...counts].map(([k, v]) => `${k}=${v}`).join(","));
|
||||
const padded = "abc".toUpperCase().padStart(5, "😀");
|
||||
console.log(padded.length, [...padded].length, padded === "\ud83dABC".padStart(5, "😀"));
|
||||
console.log("x".toUpperCase().padEnd(4, "é").normalize("NFD").length);
|
||||
@@ -0,0 +1,212 @@
|
||||
// Web TransformStream and pipes from untyped JavaScript: transformers with
|
||||
// start, transform, flush and cancel, backpressure between the sides,
|
||||
// controller errors and terminate, pipeTo and pipeThrough with their
|
||||
// options, abort signals, own properties on streams and method wrappers
|
||||
// over them, a transformer start that returns a thenable, and encoder and
|
||||
// decoder streams, in Node's promise order.
|
||||
import { TransformStream as WebTransformStream, TransformStreamDefaultController } from "node:stream/web";
|
||||
|
||||
const log = [];
|
||||
const out = (...args) => log.push(args.map(String).join(" "));
|
||||
const caught = (fn) => {
|
||||
try {
|
||||
fn();
|
||||
return "ok";
|
||||
} catch (e) {
|
||||
return `${e.name}:${e.code ?? ""}:${e.message}`;
|
||||
}
|
||||
};
|
||||
const settled = (p) => p.then((v) => `ok:${v}`, (e) => `${e?.name}:${e?.code ?? ""}:${e?.message ?? e}`);
|
||||
const collect = async (readable) => {
|
||||
const got = [];
|
||||
for await (const chunk of readable) got.push(chunk);
|
||||
return got.join(",");
|
||||
};
|
||||
|
||||
out("classes", TransformStream === WebTransformStream, TransformStream.name, TransformStream.length, caught(() => new TransformStreamDefaultController()));
|
||||
out("prototype", Object.keys(TransformStream.prototype).join(","), Object.keys(TransformStreamDefaultController.prototype).join(","));
|
||||
|
||||
{
|
||||
const order = [];
|
||||
const t = new TransformStream({
|
||||
start(c) {
|
||||
order.push(`start ${c.desiredSize}`);
|
||||
c.enqueue("S");
|
||||
},
|
||||
transform(chunk, c) {
|
||||
order.push(`transform ${chunk}`);
|
||||
c.enqueue(chunk.toUpperCase());
|
||||
},
|
||||
flush(c) {
|
||||
order.push("flush");
|
||||
c.enqueue("F");
|
||||
},
|
||||
});
|
||||
const writer = t.writable.getWriter();
|
||||
writer.write("a");
|
||||
writer.write("b");
|
||||
writer.close();
|
||||
out("transform", await collect(t.readable), order.join(" | "), await settled(writer.closed));
|
||||
}
|
||||
|
||||
{
|
||||
const t = new TransformStream({}, undefined, { highWaterMark: 1 });
|
||||
const writer = t.writable.getWriter();
|
||||
out("backpressure", writer.desiredSize, t.readable.locked);
|
||||
const first = writer.write("x");
|
||||
const second = writer.write("y");
|
||||
out("queued", writer.desiredSize);
|
||||
const reader = t.readable.getReader();
|
||||
out("read", (await reader.read()).value, (await reader.read()).value, await settled(first), await settled(second));
|
||||
}
|
||||
|
||||
{
|
||||
const failure = new Error("transform-failure");
|
||||
const t = new TransformStream({ transform() { throw failure; } });
|
||||
const writer = t.writable.getWriter();
|
||||
const reader = t.readable.getReader();
|
||||
const read = settled(reader.read());
|
||||
const write = settled(writer.write("z"));
|
||||
out("transform-error", await write, await read, await settled(writer.closed));
|
||||
}
|
||||
|
||||
{
|
||||
/** @type {any} */
|
||||
let controller;
|
||||
const t = new TransformStream({ start(c) { controller = c; } });
|
||||
controller.enqueue("before");
|
||||
controller.terminate();
|
||||
const reader = t.readable.getReader();
|
||||
out("terminate", (await reader.read()).value, (await reader.read()).done, await settled(t.writable.getWriter().write("after")), caught(() => controller.enqueue("late")));
|
||||
}
|
||||
|
||||
{
|
||||
const cancels = [];
|
||||
const t = new TransformStream({ cancel(reason) { cancels.push(`cancel ${reason}`); } });
|
||||
out("cancel", await settled(t.readable.cancel("stop")), cancels.join(","), await settled(t.writable.getWriter().closed));
|
||||
}
|
||||
|
||||
{
|
||||
const source = ReadableStream.from(["p", "q", "r"]);
|
||||
const got = [];
|
||||
const sink = new WritableStream({ write(chunk) { got.push(chunk); } });
|
||||
const piped = source.pipeThrough(new TransformStream({ transform(chunk, c) { c.enqueue(chunk + chunk); } })).pipeTo(sink);
|
||||
out("pipe", await settled(piped), got.join(","), source.locked);
|
||||
}
|
||||
|
||||
{
|
||||
const source = new ReadableStream({ start(c) { c.enqueue(1); c.close(); } });
|
||||
const sink = new WritableStream();
|
||||
out("prevent-close", await settled(source.pipeTo(sink, { preventClose: true })), sink.locked, await settled(sink.getWriter().write(2)));
|
||||
}
|
||||
|
||||
{
|
||||
const failure = new Error("source-failure");
|
||||
const source = new ReadableStream({ start(c) { c.error(failure); } });
|
||||
const aborts = [];
|
||||
const sink = new WritableStream({ abort(reason) { aborts.push(reason === failure); } });
|
||||
out("source-error", await settled(source.pipeTo(sink)), aborts.join(","));
|
||||
}
|
||||
|
||||
{
|
||||
const controller = new AbortController();
|
||||
const reason = new Error("signal");
|
||||
const source = new ReadableStream({ pull() {}, cancel(e) { out("signal-cancel", e === reason); } }, { highWaterMark: 0 });
|
||||
const sink = new WritableStream({ abort(e) { out("signal-abort", e === reason); } });
|
||||
const piped = source.pipeTo(sink, { signal: controller.signal });
|
||||
await Promise.resolve();
|
||||
controller.abort(reason);
|
||||
out("signal", await settled(piped), source.locked, sink.locked);
|
||||
}
|
||||
|
||||
{
|
||||
const source = ReadableStream.from(["own"]);
|
||||
const sink = new WritableStream();
|
||||
/** @type {any} */ (source).getReader = () => { throw new Error("public reader"); };
|
||||
/** @type {any} */ (sink).tag = "kept";
|
||||
await source.pipeTo(sink);
|
||||
out("own-properties", /** @type {any} */ (sink).tag, typeof /** @type {any} */ (source).getReader, caught(() => { /** @type {any} */ (source).locked = true; }));
|
||||
}
|
||||
|
||||
{
|
||||
// An own property that wraps the saved prototype method reaches the
|
||||
// class's method.
|
||||
const calls = [];
|
||||
const wrap = (target, name, method) => {
|
||||
target[name] = function (...args) {
|
||||
calls.push(name);
|
||||
return method.apply(this, args);
|
||||
};
|
||||
};
|
||||
const writable = new WritableStream({ write(chunk) { calls.push(`sink:${chunk}`); } });
|
||||
wrap(writable, "getWriter", WritableStream.prototype.getWriter);
|
||||
const writer = writable.getWriter();
|
||||
wrap(writer, "write", WritableStreamDefaultWriter.prototype.write);
|
||||
wrap(writer, "close", WritableStreamDefaultWriter.prototype.close);
|
||||
await writer.write("a");
|
||||
await writer.close();
|
||||
const readable = ReadableStream.from(["b"]);
|
||||
wrap(readable, "getReader", ReadableStream.prototype.getReader);
|
||||
const reader = readable.getReader();
|
||||
wrap(reader, "read", ReadableStreamDefaultReader.prototype.read);
|
||||
const first = await reader.read();
|
||||
const t = new TransformStream({
|
||||
transform(chunk, controller) {
|
||||
wrap(controller, "enqueue", TransformStreamDefaultController.prototype.enqueue);
|
||||
controller.enqueue(`${chunk}!`);
|
||||
},
|
||||
});
|
||||
t.writable.getWriter().write("c");
|
||||
const transformed = await t.readable.getReader().read();
|
||||
out("wrappers", calls.join(","), first.value, transformed.value, writable.locked);
|
||||
}
|
||||
|
||||
{
|
||||
// A transformer start that returns a thenable: both sides start when it
|
||||
// settles, and its rejection errors the stream.
|
||||
const order = [];
|
||||
const later = { then(resolve) { order.push("then"); setTimeout(() => { order.push("resolve"); resolve(); }, 0); } };
|
||||
const t = new TransformStream({ start() { return later; }, transform(chunk, c) { order.push(`transform:${chunk}`); c.enqueue(chunk); } });
|
||||
Promise.resolve().then(() => order.push("tick1")).then(() => order.push("tick2")).then(() => order.push("tick3"));
|
||||
const written = t.writable.getWriter().write("x").then(() => order.push("written"));
|
||||
const read = await t.readable.getReader().read();
|
||||
await written;
|
||||
out("start-thenable", order.join(","), read.value);
|
||||
const ticks = [];
|
||||
const sync = new TransformStream({ start() { return { then(resolve) { ticks.push("then"); resolve(); } }; }, transform(chunk, c) { ticks.push("transform"); c.enqueue(chunk); } });
|
||||
Promise.resolve().then(() => ticks.push("tick1")).then(() => ticks.push("tick2")).then(() => ticks.push("tick3")).then(() => ticks.push("tick4"));
|
||||
sync.writable.getWriter().write("z").then(() => ticks.push("written"));
|
||||
sync.readable.getReader().read().then(() => ticks.push("read"));
|
||||
await new Promise((r) => setTimeout(r, 0));
|
||||
out("start-thenable-ticks", ticks.join(","));
|
||||
const failing = new TransformStream({ start() { return { then(_, reject) { reject(new Error("start failed")); } }; } });
|
||||
out("start-thenable-reject", await settled(failing.writable.getWriter().write("y")), await settled(failing.readable.getReader().read()));
|
||||
const throwing = new TransformStream({ start() { return { get then() { throw new Error("then getter"); } }; } });
|
||||
out("start-then-getter", await settled(throwing.writable.getWriter().write("y")));
|
||||
const plain = new TransformStream({ start() { return { then: 1 }; } });
|
||||
const pending = plain.readable.getReader().read();
|
||||
out("start-object", await settled(plain.writable.getWriter().write("p")), (await pending).value);
|
||||
}
|
||||
|
||||
out(
|
||||
"invalid",
|
||||
await settled(ReadableStream.from([]).pipeTo(/** @type {any} */ ({}))),
|
||||
caught(() => ReadableStream.from([]).pipeThrough(/** @type {any} */ ({}))),
|
||||
caught(() => new TransformStream({ readableType: "bytes" })),
|
||||
caught(() => new TransformStream({ transform: 1 })),
|
||||
);
|
||||
|
||||
{
|
||||
const bytes = [];
|
||||
await ReadableStream.from(["hé", "\ud83d", "\ude00", "\ud800"])
|
||||
.pipeThrough(new TextEncoderStream())
|
||||
.pipeTo(new WritableStream({ write(chunk) { bytes.push(Array.from(chunk).join(".")); } }));
|
||||
out("encoder", bytes.join(" | "), new TextEncoderStream().encoding, Object.prototype.toString.call(new TextEncoderStream()));
|
||||
const text = [];
|
||||
await ReadableStream.from([new Uint8Array([0xe2, 0x82]), new Uint8Array([0xac, 0x21]), new Uint8Array([0xf0])])
|
||||
.pipeThrough(new TextDecoderStream())
|
||||
.pipeTo(new WritableStream({ write(chunk) { text.push(chunk); } }));
|
||||
const decoder = new TextDecoderStream("utf-8", { fatal: true });
|
||||
out("decoder", text.join("|"), decoder.encoding, decoder.fatal, decoder.ignoreBOM, caught(() => new TextDecoderStream("nope")));
|
||||
}
|
||||
console.log(log.join("\n"));
|
||||
@@ -0,0 +1,144 @@
|
||||
// Web WritableStream from untyped JavaScript: the classes as constructor
|
||||
// values, prototypes and brand checks, writers with backpressure and their
|
||||
// ready and closed promises, queuing strategies, sink errors, close and
|
||||
// abort ordering, the controller's signal, and argument errors, in Node's
|
||||
// promise order.
|
||||
import { WritableStream as WebWritableStream, WritableStreamDefaultWriter, WritableStreamDefaultController } from "node:stream/web";
|
||||
|
||||
const log = [];
|
||||
const out = (...args) => log.push(args.map(String).join(" "));
|
||||
const caught = (fn) => {
|
||||
try {
|
||||
fn();
|
||||
return "ok";
|
||||
} catch (e) {
|
||||
return `${e.name}:${e.code ?? ""}:${e.message}`;
|
||||
}
|
||||
};
|
||||
const settled = (p) => p.then((v) => `ok:${v}`, (e) => `${e?.name}:${e?.code ?? ""}:${e?.message ?? e}`);
|
||||
|
||||
out("classes", typeof WritableStream, WritableStream === WebWritableStream, WritableStream.name, WritableStream.length, WritableStreamDefaultWriter.length);
|
||||
out("illegal", caught(() => new WritableStreamDefaultController()), caught(() => /** @type {any} */ (WritableStream)()));
|
||||
out("prototype", Object.keys(WritableStream.prototype).join(","), Object.keys(WritableStreamDefaultWriter.prototype).join(","), Object.keys(WritableStreamDefaultController.prototype).join(","));
|
||||
|
||||
{
|
||||
const order = [];
|
||||
const chunks = [];
|
||||
const stream = new WritableStream(
|
||||
{
|
||||
start() { order.push("start"); },
|
||||
write(chunk, controller) {
|
||||
order.push(`write ${chunk}`);
|
||||
chunks.push(chunk);
|
||||
return new Promise((resolve) => setTimeout(resolve, 1));
|
||||
},
|
||||
close() { order.push("close"); },
|
||||
},
|
||||
new CountQueuingStrategy({ highWaterMark: 2 }),
|
||||
);
|
||||
const writer = stream.getWriter();
|
||||
out("writer", writer instanceof WritableStreamDefaultWriter, stream.locked, writer.desiredSize, caught(() => stream.getWriter()));
|
||||
const writes = [writer.write("a"), writer.write("b"), writer.write("c")];
|
||||
out("queued", writer.desiredSize, Object.prototype.toString.call(stream), Object.prototype.toString.call(writer));
|
||||
let readySettled = false;
|
||||
writer.ready.then(() => { readySettled = true; });
|
||||
await Promise.resolve();
|
||||
out("backpressure", readySettled, order.join(" | "));
|
||||
for (const w of writes) await w;
|
||||
await writer.ready;
|
||||
out("drained", writer.desiredSize, readySettled, chunks.join(""));
|
||||
await writer.close();
|
||||
out("closed", await settled(writer.closed), writer.desiredSize, order.join(" | "), await settled(writer.write("late")));
|
||||
writer.releaseLock();
|
||||
out("released", stream.locked, await settled(writer.closed), caught(() => writer.desiredSize));
|
||||
}
|
||||
|
||||
{
|
||||
const sizes = [];
|
||||
const stream = new WritableStream(
|
||||
{ write() {} },
|
||||
{ highWaterMark: 10, size(chunk) { sizes.push(chunk.length); return chunk.length; } },
|
||||
);
|
||||
const writer = stream.getWriter();
|
||||
const good = writer.write("abcd");
|
||||
out("sizes", writer.desiredSize, sizes.join(","));
|
||||
const bad = writer.write({ length: -1 });
|
||||
out("bad-size", await settled(good), await settled(bad), await settled(writer.closed), writer.desiredSize);
|
||||
}
|
||||
|
||||
{
|
||||
const failure = new Error("sink-write");
|
||||
const stream = new WritableStream({ write() { throw failure; } });
|
||||
const writer = stream.getWriter();
|
||||
const first = writer.write("x");
|
||||
const second = writer.write("y");
|
||||
out("write-error", await settled(first), await settled(second), await settled(writer.closed), await settled(writer.ready));
|
||||
}
|
||||
|
||||
{
|
||||
const log2 = [];
|
||||
/** @type {any} */
|
||||
let controller;
|
||||
const reason = new Error("stop");
|
||||
const stream = new WritableStream({
|
||||
start(c) {
|
||||
controller = c;
|
||||
c.signal.addEventListener("abort", () => log2.push(`signal ${c.signal.reason === reason}`));
|
||||
},
|
||||
write(chunk) { log2.push(`write ${chunk}`); },
|
||||
abort(e) { log2.push(`abort ${e === reason}`); },
|
||||
});
|
||||
const writer = stream.getWriter();
|
||||
await writer.write("one");
|
||||
const abort = writer.abort(reason);
|
||||
log2.push("aborting");
|
||||
out("abort", await settled(abort), log2.join(" | "), controller.signal.aborted, await settled(writer.closed), writer.desiredSize);
|
||||
}
|
||||
|
||||
{
|
||||
/** @type {any} */
|
||||
let controller;
|
||||
const stream = new WritableStream({ start(c) { controller = c; } });
|
||||
const writer = stream.getWriter();
|
||||
controller.error(new Error("controller-error"));
|
||||
out("controller-error", await settled(writer.write("z")), await settled(writer.closed), writer.desiredSize, await settled(stream.abort()));
|
||||
}
|
||||
|
||||
{
|
||||
const stream = new WritableStream();
|
||||
const close = stream.close();
|
||||
out("direct-close", await settled(stream.close()), await settled(close), await settled(stream.abort()));
|
||||
}
|
||||
|
||||
out(
|
||||
"brands",
|
||||
caught(() => WritableStream.prototype.getWriter.call({})),
|
||||
await settled(WritableStream.prototype.abort.call({})),
|
||||
await settled(/** @type {any} */ (Object.getOwnPropertyDescriptor(WritableStreamDefaultWriter.prototype, "ready")).get.call({})),
|
||||
caught(() => /** @type {any} */ (Object.getOwnPropertyDescriptor(WritableStreamDefaultWriter.prototype, "desiredSize")).get.call({})),
|
||||
caught(() => WritableStreamDefaultController.prototype.error.call({})),
|
||||
);
|
||||
out(
|
||||
"invalid",
|
||||
caught(() => new WritableStream(null)),
|
||||
caught(() => new WritableStream({ type: "bytes" })),
|
||||
caught(() => new WritableStream({ write: 1 })),
|
||||
caught(() => new WritableStream({ start: 1 })),
|
||||
caught(() => new WritableStream({}, { highWaterMark: -1 })),
|
||||
caught(() => new WritableStreamDefaultWriter(/** @type {any} */ ({}))),
|
||||
);
|
||||
{
|
||||
const classes = { WritableStream, WritableStreamDefaultWriter, TransformStream };
|
||||
const isInstance = (value, C) => value instanceof C;
|
||||
const stream = new WritableStream();
|
||||
const transform = new TransformStream();
|
||||
out(
|
||||
"by-value",
|
||||
isInstance(stream, classes.WritableStream),
|
||||
isInstance(stream.getWriter(), classes.WritableStreamDefaultWriter),
|
||||
isInstance(transform, classes.TransformStream),
|
||||
isInstance(transform.writable, classes.WritableStream),
|
||||
isInstance(stream, classes.TransformStream),
|
||||
);
|
||||
}
|
||||
console.log(log.join("\n"));
|
||||
@@ -0,0 +1 @@
|
||||
{ "plain": 2000, "san": 2000 }
|
||||
@@ -0,0 +1 @@
|
||||
{ "plain": 2000, "san": 2000 }
|
||||
@@ -0,0 +1 @@
|
||||
{ "plain": 9400, "san": 30000 }
|
||||
@@ -0,0 +1 @@
|
||||
{ "plain": 2800, "san": 42400 }
|
||||
@@ -0,0 +1 @@
|
||||
{ "plain": 3100, "san": 2000 }
|
||||
@@ -0,0 +1 @@
|
||||
{ "plain": 3600, "san": 3700 }
|
||||
@@ -0,0 +1 @@
|
||||
{ "plain": 4300, "san": 55000 }
|
||||
@@ -0,0 +1 @@
|
||||
{ "plain": 3900, "san": 6700 }
|
||||
@@ -57,6 +57,7 @@ const KNOWN: Record<string, Known> = {
|
||||
divergence:
|
||||
"Timeout handles are numbers (typeof) and lose their ref flag once fired or cleared",
|
||||
},
|
||||
// require() inside a function.
|
||||
"island-web-plumbing": { run: "SC1090" },
|
||||
"lazy-traps": { build: "SC2013" },
|
||||
"misc-shims": { run: "SC1090" },
|
||||
@@ -87,15 +88,6 @@ const KNOWN: Record<string, Known> = {
|
||||
"node24-url-mutation": { run: "SC1090" },
|
||||
"node24-util-adapters": { run: "Object.create with this value is not supported yet" },
|
||||
"node24-util-types": { run: "SC1090" },
|
||||
"node24-web-stream-encoding": { run: "SC2020" },
|
||||
"node24-web-stream-errors": { run: "SC1090" },
|
||||
"node24-web-stream-lifecycle": { run: "SC2020" },
|
||||
"node24-web-stream-pipe-abort": { run: "SC2020" },
|
||||
"node24-web-stream-pipe-errors": { run: "SC2020" },
|
||||
"node24-web-stream-pipes": { run: "SC2020" },
|
||||
"node24-web-stream-queues": { run: "SC2020" },
|
||||
"node24-web-stream-tee": { run: "SC1090" },
|
||||
"node24-web-stream-transforms": { run: "SC1090" },
|
||||
"package-imports": { build: "SC2013" },
|
||||
"promise-bridge": { build: "SC1090" },
|
||||
"self-name": { build: "SC2013" },
|
||||
@@ -103,6 +95,7 @@ const KNOWN: Record<string, Known> = {
|
||||
"util-shims": { run: "SC2020" },
|
||||
// The package redeclares the var it exported: Node copies the first value.
|
||||
"var-export-redeclared": { run: "SC1090" },
|
||||
// `class Upper extends TransformStream`: extending a builtin class from JavaScript.
|
||||
"web-streams": { run: "SC1090" },
|
||||
"workspace-copied": { build: "SC2013" },
|
||||
"workspace-unclean": { build: "SC2013" },
|
||||
|
||||
@@ -7,59 +7,68 @@ import { skipUnownedTest } from "./shard.js";
|
||||
|
||||
const fixture = join(import.meta.dirname, "../library-mode/wasm");
|
||||
const hasClang = spawnSync("clang", ["--version"]).status === 0;
|
||||
// Release compiles the whole three.js library into one LLVM module that a
|
||||
// single process optimizes and codegens on one core, which takes 5-10
|
||||
// minutes; sanitized dev builds are slower still. Lanes measured 331-581s for
|
||||
// release against the old 600s budget, so both get 1_200_000ms. The plain dev
|
||||
// case runs at about a fifth of release and keeps 600_000ms.
|
||||
function timeoutFor(optimization: "dev" | "release"): number {
|
||||
return optimization === "release" || process.env["SCRIPTC_SAN"] === "1" ? 1_200_000 : 600_000;
|
||||
}
|
||||
// The two optimizations take similar time (about 10 minutes each under ASan):
|
||||
// SCRIPTC_TEST_SHARD runs them on different shards (tests/harness/shard.ts).
|
||||
// skipUnownedTest partitions a describe block's tests.
|
||||
describe("three.js native library", () => {
|
||||
beforeEach((context) => skipUnownedTest(context, {}));
|
||||
test.skipIf(!hasClang).each(["dev", "release"] as const)(
|
||||
"three.js sends animated geometry, materials and resource lifecycle to a native host in %s",
|
||||
async (optimization) => {
|
||||
const directory = await mkdtemp("/tmp/scriptc-three-library-");
|
||||
const sanitize = process.env["SCRIPTC_SAN"] === "1";
|
||||
try {
|
||||
const profile = JSON.parse(await readFile(join(fixture, "three.json"), "utf8"));
|
||||
const profilePath = join(directory, "profile.json");
|
||||
await writeFile(
|
||||
profilePath,
|
||||
JSON.stringify({
|
||||
...profile,
|
||||
name: "three-native-embedding",
|
||||
entry: join(fixture, "three.mjs"),
|
||||
optimization,
|
||||
}),
|
||||
);
|
||||
const built = await compileLibrary({ profilePath, outDir: directory, sanitize });
|
||||
if (!built.ok)
|
||||
throw new Error(
|
||||
built.diagnostics
|
||||
.map((diagnostic) => `${diagnostic.code}: ${diagnostic.message}`)
|
||||
.join("\n"),
|
||||
for (const optimization of ["dev", "release"] as const) {
|
||||
test.skipIf(!hasClang)(
|
||||
`three.js sends animated geometry, materials and resource lifecycle to a native host in ${optimization}`,
|
||||
async () => {
|
||||
const directory = await mkdtemp("/tmp/scriptc-three-library-");
|
||||
const sanitize = process.env["SCRIPTC_SAN"] === "1";
|
||||
try {
|
||||
const profile = JSON.parse(await readFile(join(fixture, "three.json"), "utf8"));
|
||||
const profilePath = join(directory, "profile.json");
|
||||
await writeFile(
|
||||
profilePath,
|
||||
JSON.stringify({
|
||||
...profile,
|
||||
name: "three-native-embedding",
|
||||
entry: join(fixture, "three.mjs"),
|
||||
optimization,
|
||||
}),
|
||||
);
|
||||
const binary = join(directory, "host");
|
||||
const linked = spawnSync(
|
||||
"clang",
|
||||
[
|
||||
"-std=c11",
|
||||
...(sanitize ? ["-fsanitize=address,undefined"] : []),
|
||||
join(fixture, "three-native.c"),
|
||||
built.archivePath,
|
||||
"-lm",
|
||||
"-o",
|
||||
binary,
|
||||
],
|
||||
{ encoding: "utf8" },
|
||||
);
|
||||
expect(linked.status, linked.stderr).toBe(0);
|
||||
const actual = spawnSync(binary, [], { encoding: "utf8", timeout: 30_000 });
|
||||
expect(actual.status, actual.stderr).toBe(0);
|
||||
const reference = spawnSync(
|
||||
process.execPath,
|
||||
[
|
||||
"--no-warnings",
|
||||
"--input-type=module",
|
||||
"-e",
|
||||
`
|
||||
const built = await compileLibrary({ profilePath, outDir: directory, sanitize });
|
||||
if (!built.ok)
|
||||
throw new Error(
|
||||
built.diagnostics
|
||||
.map((diagnostic) => `${diagnostic.code}: ${diagnostic.message}`)
|
||||
.join("\n"),
|
||||
);
|
||||
const binary = join(directory, "host");
|
||||
const linked = spawnSync(
|
||||
"clang",
|
||||
[
|
||||
"-std=c11",
|
||||
...(sanitize ? ["-fsanitize=address,undefined"] : []),
|
||||
join(fixture, "three-native.c"),
|
||||
built.archivePath,
|
||||
"-lm",
|
||||
"-o",
|
||||
binary,
|
||||
],
|
||||
{ encoding: "utf8" },
|
||||
);
|
||||
expect(linked.status, linked.stderr).toBe(0);
|
||||
const actual = spawnSync(binary, [], { encoding: "utf8", timeout: 30_000 });
|
||||
expect(actual.status, actual.stderr).toBe(0);
|
||||
const reference = spawnSync(
|
||||
process.execPath,
|
||||
[
|
||||
"--no-warnings",
|
||||
"--input-type=module",
|
||||
"-e",
|
||||
`
|
||||
const events=[];
|
||||
for (const name of ["vertex","surface","resource"]) globalThis[name]=(...values)=>events.push([name,...values.map(value=>Number(value.toFixed(9)))]);
|
||||
const {frame,dispose}=await import(${JSON.stringify(join(fixture, "three.mjs"))});
|
||||
@@ -67,21 +76,22 @@ describe("three.js native library", () => {
|
||||
dispose();
|
||||
console.log(JSON.stringify(events));
|
||||
`,
|
||||
],
|
||||
{ encoding: "utf8" },
|
||||
);
|
||||
expect(reference.status, reference.stderr).toBe(0);
|
||||
const events = JSON.parse(reference.stdout);
|
||||
expect(
|
||||
actual.stdout
|
||||
.trim()
|
||||
.split("\n")
|
||||
.map((line) => JSON.parse(line)),
|
||||
).toEqual([...events, ...events]);
|
||||
} finally {
|
||||
await rm(directory, { recursive: true, force: true });
|
||||
}
|
||||
},
|
||||
600_000,
|
||||
);
|
||||
],
|
||||
{ encoding: "utf8" },
|
||||
);
|
||||
expect(reference.status, reference.stderr).toBe(0);
|
||||
const events = JSON.parse(reference.stdout);
|
||||
expect(
|
||||
actual.stdout
|
||||
.trim()
|
||||
.split("\n")
|
||||
.map((line) => JSON.parse(line)),
|
||||
).toEqual([...events, ...events]);
|
||||
} finally {
|
||||
await rm(directory, { recursive: true, force: true });
|
||||
}
|
||||
},
|
||||
timeoutFor(optimization),
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
@@ -3,17 +3,22 @@ import { expect, test } from "vitest";
|
||||
import {
|
||||
cachedComparison,
|
||||
cachegrindCounts,
|
||||
changedFunctions,
|
||||
compareArtifact,
|
||||
compareCounts,
|
||||
compareExecutables,
|
||||
confinedToRuntime,
|
||||
definedFunctionNames,
|
||||
dynamicTierSymbols,
|
||||
linkedRuntimeFiles,
|
||||
llvmFunctionHashes,
|
||||
mergeReports,
|
||||
nmOutput,
|
||||
programInputsHash,
|
||||
programLines,
|
||||
programVerdict,
|
||||
recordedBaseline,
|
||||
runtimePackDiff,
|
||||
timingVerdict,
|
||||
} from "../../scripts/zero-cost-compare.mjs";
|
||||
|
||||
@@ -198,6 +203,116 @@ test("the zero-cost gate fails differing programs unless they match the signatur
|
||||
expect(verdict({}, true, ok, failed)).toEqual({ verdict: "build-error", fails: true });
|
||||
});
|
||||
|
||||
test("the zero-cost gate classifies runtime-only and nondeterministic differences", () => {
|
||||
const ok = { ok: true };
|
||||
const exeOnly = { ir: { identical: true }, llvm: { identical: true }, exe: { identical: false } };
|
||||
const llvmToo = {
|
||||
ir: { identical: true },
|
||||
llvm: { identical: false },
|
||||
exe: { identical: false },
|
||||
};
|
||||
const verdict = (artifacts: object, flags: object) =>
|
||||
programVerdict({ baseline: ok, candidate: ok, artifacts, allowed: false, ...flags });
|
||||
expect(verdict(exeOnly, { runtimeOnly: true })).toEqual({
|
||||
verdict: "runtime-only",
|
||||
fails: false,
|
||||
});
|
||||
expect(verdict(exeOnly, { runtimeOnly: true, strict: true })).toEqual({
|
||||
verdict: "runtime-only",
|
||||
fails: true,
|
||||
});
|
||||
// A changed runtime cannot explain different LLVM.
|
||||
expect(verdict(llvmToo, { runtimeOnly: true })).toEqual({ verdict: "differs", fails: true });
|
||||
expect(verdict(llvmToo, { nondeterministic: ["llvm", "exe"] })).toEqual({
|
||||
verdict: "nondeterministic",
|
||||
fails: false,
|
||||
});
|
||||
expect(verdict(llvmToo, { nondeterministic: ["llvm", "exe"], strict: true })).toEqual({
|
||||
verdict: "nondeterministic",
|
||||
fails: true,
|
||||
});
|
||||
// A nondeterministic executable does not excuse a deterministic LLVM difference.
|
||||
expect(verdict(llvmToo, { nondeterministic: ["exe"] })).toEqual({
|
||||
verdict: "differs",
|
||||
fails: true,
|
||||
});
|
||||
});
|
||||
|
||||
test("the zero-cost gate reads the symbols of changed runtime objects and vendor archives", () => {
|
||||
expect(
|
||||
linkedRuntimeFiles({
|
||||
changed: [
|
||||
"artifacts/release/runtime/default/scr_a.o",
|
||||
"artifacts/release/runtime/default/scr_a.bc",
|
||||
],
|
||||
added: ["artifacts/vendor/libz/libscriptc-zlib.a"],
|
||||
removed: [],
|
||||
}),
|
||||
).toEqual([
|
||||
"artifacts/release/runtime/default/scr_a.o",
|
||||
"artifacts/vendor/libz/libscriptc-zlib.a",
|
||||
]);
|
||||
});
|
||||
|
||||
test("the zero-cost gate names the runtime pack files and functions that changed", () => {
|
||||
const unit = (sha: string, bc: string) => ({
|
||||
path: "artifacts/release/runtime/default/scr_a.o",
|
||||
sha256: sha,
|
||||
bitcode: { path: "artifacts/release/runtime/default/scr_a.bc", sha256: bc },
|
||||
});
|
||||
const pack = (a: object, extra: object[] = []) => ({
|
||||
flavors: { release: { runtime_units: [{ source: "scr_a.c", variants: [a, ...extra] }] } },
|
||||
archives: [{ path: "artifacts/vendor/libz.a", sha256: "z" }],
|
||||
});
|
||||
const other = { path: "artifacts/release/runtime/default/scr_b.o", sha256: "b" };
|
||||
expect(runtimePackDiff(pack(unit("1", "2")), pack(unit("1", "2")))).toEqual({
|
||||
changed: [],
|
||||
added: [],
|
||||
removed: [],
|
||||
});
|
||||
expect(runtimePackDiff(pack(unit("1", "2")), pack(unit("3", "4"), [other]))).toEqual({
|
||||
changed: [
|
||||
"artifacts/release/runtime/default/scr_a.bc",
|
||||
"artifacts/release/runtime/default/scr_a.o",
|
||||
],
|
||||
added: ["artifacts/release/runtime/default/scr_b.o"],
|
||||
removed: [],
|
||||
});
|
||||
|
||||
const ll = (body: string, attrs: number) =>
|
||||
[
|
||||
`define i32 @scr_same(i32 %x) #${attrs} {`,
|
||||
" ret i32 %x, !dbg !" + attrs,
|
||||
"}",
|
||||
"define internal void @scr_changed() {",
|
||||
` ${body}`,
|
||||
"}",
|
||||
"",
|
||||
].join("\n");
|
||||
const before = llvmFunctionHashes(ll("ret void", 0));
|
||||
const after = llvmFunctionHashes(
|
||||
ll("call void @f()\n ret void", 7) + 'define void @"scr new"() {\n}\n',
|
||||
);
|
||||
// Renumbered attribute groups and metadata are not changes.
|
||||
expect(changedFunctions(before, after)).toEqual({
|
||||
changed: ["scr_changed"],
|
||||
added: ["scr new"],
|
||||
removed: [],
|
||||
});
|
||||
});
|
||||
|
||||
test("the zero-cost gate confines runtime-only differences to runtime symbols", () => {
|
||||
const exe = (added: string[], removed: string[], resized: string[]) => ({
|
||||
added,
|
||||
removed,
|
||||
resized: resized.map((name) => ({ name, baseline: 1, candidate: 2 })),
|
||||
});
|
||||
const runtime = new Set(["scr_a", "scr_b"]);
|
||||
expect(confinedToRuntime(exe([], [], ["scr_a"]), runtime)).toBe(true);
|
||||
expect(confinedToRuntime(exe(["scr_b"], ["scr_a"], []), runtime)).toBe(true);
|
||||
expect(confinedToRuntime(exe([], [], ["main.$1"]), runtime)).toBe(false);
|
||||
});
|
||||
|
||||
test("a recorded baseline settles identical programs and shared build errors", () => {
|
||||
const hashes = { ir: "a", llvm: "b", exe: "c" };
|
||||
const entry = { ok: true, ms: 12, hashes, exeBytes: 100 };
|
||||
@@ -298,4 +413,15 @@ test("merged gate shards judge compile time and comparisons over every shard", (
|
||||
"no program built with both checkouts, so nothing was compared",
|
||||
]);
|
||||
expect(() => mergeReports([fast, fast])).toThrow(/more than one gate report/);
|
||||
// A runtime-only program was compared, and its line names the runtime change.
|
||||
const runtimeOnly = {
|
||||
id: "bench/r",
|
||||
verdict: "runtime-only",
|
||||
artifacts: { exe: { identical: false, first: "resized scr_a" } },
|
||||
runtime: { changed: ["scr_a"], added: [] },
|
||||
};
|
||||
expect(mergeReports([shard([runtimeOnly], [])]).report.verdict).toBe("pass");
|
||||
expect(programLines(runtimeOnly)).toEqual([
|
||||
"runtime bench/r exe: resized scr_a; runtime-only: links 1 changed runtime function (scr_a)",
|
||||
]);
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user