diff --git a/CHANGELOG.md b/CHANGELOG.md index 612df47b5d..e8ab725f0c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,42 @@ All notable changes to scriptc will be documented in this file. +## 0.4.5 + +### Highlights + +- **Fixes:** + - Release builds read a typed array's length without a header load that could read a freed view. + - `net.Socket` `write()` returns Node's backpressure boolean and fires `'drain'`. + - Super calls in JavaScript method values call the declared base method. + - Calling a non-function through an asserted callee throws Node's TypeError. +- **Faster programs:** + - Exceptions propagate by unwinding from directly called functions on Linux, instead of status checks after every call. + - Typed-array loops prove their bounds and keep their views disjoint in more cases, including views over shared memory in worker programs; integer loops stay in integers across nested counted loops, pure helper calls, powers of two and byte reads. + - Lone array callbacks run as loops with their bodies inlined, and row sums run two rows per pass. + - Number-key maps look keys up on their integer view and write updates through the entry the read found; small number-key maps use a direct index. + - Strings: base64 decodes eight characters at a time; Buffers fill, swap byte order and decode `utf16le` a word at a time; readline splits input in one scan; string scans over non-ASCII strings use their UTF-16 units. + - Untyped code: property caches also cache absent properties, object literal members go through the store caches, calls on plain arrays skip the method ladder, and dynamic values dispose without key-cache walks. + - JSON parses into discriminated unions, and dynamic values convert into nested unions, in one pass. + - BigInt division settles quotients from the top limbs, and limb shifts are vectorized. +- **Faster builds:** + - Function bodies are emitted on up to eight workers, including in dev builds, and repeated work in LLVM module emission is cut. + - Linux executables link with lld when the driver finds it, and unchanged executables are reused without selecting the runtime pack. + - Dev builds reuse their frontend output after edits that keep token positions, and unchanged frontend inputs are checked by file identity before hashing. + - Lowering fetches only the syntax trees it reads, and the large compiler tables are built on first use. + - Release builds of the native compiler use a profile collected on the release commit. + - Type checking can use a tsc-ts server shared by later builds (opt-in). +- **Node.js parity:** + - Crypto: RSA key generation lengths, RSA-PSS keys, encrypted private keys, `X509Certificate` fields and methods, and TLS certificates and `tls.Server` contexts. + - zlib: Brotli compression and decompression. + - Dates are mutable objects with Node's setters and conversions; URL hosts and paths parse as the URL Standard does, including non-ASCII hosts with UTS #46. + - `Math`, `JSON` and `Reflect` are real objects in JavaScript; objects can inherit from JavaScript class prototypes; typed arrays held as untyped values support named properties and `Symbol.species`. + - Error messages: `JSON.parse` syntax errors use V8's messages and positions; `Object.getPrototypeOf` of an Error answers `Error.prototype`. + - `util.inspect` of dynamic classes, functions and `%o` arguments, and `Symbol.toStringTag` fields and getters, match Node. +- **npm packages:** more npm and CommonJS dependencies compile statically, including namespace-style packages with guarded declarations and `var` exports, and untyped Maps, Sets and arrays are typed from their uses. + + + ## 0.4.4 ### Highlights @@ -29,8 +65,6 @@ All notable changes to scriptc will be documented in this file. - Split CommonJS `require` bindings, `__`-prefixed exports, string-key destructuring of `any` values, and record key order for values JavaScript built. - **npm packages:** zod 4 and `@hono/zod-validator` values are projected at the declaration boundary. - - ## 0.4.3 ### Highlights diff --git a/benchmarks/runtime/README.md b/benchmarks/runtime/README.md index af3d3d2396..0157503fdf 100644 --- a/benchmarks/runtime/README.md +++ b/benchmarks/runtime/README.md @@ -32,6 +32,7 @@ Application-shaped workloads for measuring the speed of compiled executables. Ea | `regex-logs` | cpu | regex `exec` with groups, `test`, global `replace` | | `regex-replacers` | cpu | regex `replace`/`replaceAll` with inline callbacks whose parameters have no annotations: dictionary placeholders, snake_case to camelCase, date reordering, optional captures, capitalization, and a string search that reads the offset | | `region-builds` | cpu | a watch-mode build loop over `LifetimeRegion`: each build binds a program in a new region and releases the previous one, which a cache keeps referenced for two builds | +| `string-iteration` | cpu | code point iteration over text that mixes ASCII, Latin-1 letters, CJK and emoji: a for-of word tokenizer, spread reversal, and `Array.from` over words | | `template-render` | cpu | template literals, escaping, `map`/`join` | | `throw-chains` | cpu | exceptions caught 2, 5 and 20 recursive calls above the throw, through direct calls and through a callback, and a backtracking parser that throws on a failed alternative | | `thread-slots`, `thread-slots-cache` | cpu | worker threads reading one published type graph from an `InternTable` and keeping per-thread state for each type they visit: a `ThreadSlots` family, or a direct-mapped cache by id in front of a `Map` by id (the `-cache` entry); under Node.js both are per thread | diff --git a/benchmarks/runtime/workloads/string-iteration.json b/benchmarks/runtime/workloads/string-iteration.json new file mode 100644 index 0000000000..21456bd235 --- /dev/null +++ b/benchmarks/runtime/workloads/string-iteration.json @@ -0,0 +1,6 @@ +{ + "entry": "workloads/string-iteration.ts", + "args": [ + "1" + ] +} diff --git a/benchmarks/runtime/workloads/string-iteration.ts b/benchmarks/runtime/workloads/string-iteration.ts new file mode 100644 index 0000000000..2abebd6350 --- /dev/null +++ b/benchmarks/runtime/workloads/string-iteration.ts @@ -0,0 +1,59 @@ +// Code point iteration over mixed text: a word tokenizer over for-of, a +// character histogram, reversal through spread, and Array.from over +// words. The text mixes ASCII, Latin-1 letters, CJK and emoji. +let seed = 11; +function random(): number { + seed = (Math.imul(seed, 1103515245) + 12345) >>> 0; + return seed / 4294967296; +} +const pieces = [ + "the", "quick", "brown", "fox", "jumps", "over", "lazy", "dog", + "café", "naïve", "über", "señor", "straße", "déjà", "vu", + "日本", "語", "テキスト", "中文", "한국어", + "😀", "🚀", "👍", "🎉", +]; +const punctuation = [" ", " ", " ", ", ", ". ", "! ", "\n"]; +const scale = Number(process.argv[2] ?? "1"); +const lines: string[] = []; +for (let i = 0; i < Math.floor(4000 * scale); i++) { + let line = ""; + const words = 6 + Math.floor(random() * 10); + for (let w = 0; w < words; w++) { + line += pieces[Math.floor(random() * pieces.length)]!; + line += punctuation[Math.floor(random() * punctuation.length)]!; + } + lines.push(line); +} + +function isWordChar(c: string): boolean { + const code = c.codePointAt(0)!; + return (code >= 0x61 && code <= 0x7a) || (code >= 0x41 && code <= 0x5a) || code >= 0xc0; +} + +let tokens = 0; +let tokenUnits = 0; +let reversedLength = 0; +let fromLength = 0; +const histogram = new Map(); +for (let round = 0; round < 30; round++) { + for (const line of lines) { + let word = ""; + for (const c of line) { + if (isWordChar(c)) { + word += c; + continue; + } + if (word !== "") { + tokens++; + tokenUnits += word.length; + if (round === 0) histogram.set(word, (histogram.get(word) ?? 0) + 1); + fromLength += Array.from(word).length; + word = ""; + } + } + reversedLength += [...line].reverse().join("").length; + } +} +const top = [...histogram].sort((a, b) => b[1] - a[1] || (a[0] < b[0] ? -1 : 1)).slice(0, 5); +console.log(top.map(([word, count]) => `${word}:${count}`).join(" ")); +console.log("tokens", tokens, "units", tokenUnits, "reversed", reversedLength, "from", fromLength); diff --git a/packages/cli-darwin-arm64/package.json b/packages/cli-darwin-arm64/package.json index bac01cc442..1a8ee5d2c7 100644 --- a/packages/cli-darwin-arm64/package.json +++ b/packages/cli-darwin-arm64/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/cli-darwin-arm64", - "version": "0.4.4", + "version": "0.4.5", "description": "Native scriptc compiler for darwin-arm64", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/cli-darwin-x64/package.json b/packages/cli-darwin-x64/package.json index 6c2fb3123a..0cb24b9084 100644 --- a/packages/cli-darwin-x64/package.json +++ b/packages/cli-darwin-x64/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/cli-darwin-x64", - "version": "0.4.4", + "version": "0.4.5", "description": "Native scriptc compiler for darwin-x64", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/cli-linux-arm64-gnu/package.json b/packages/cli-linux-arm64-gnu/package.json index 289d8d6bd5..3a1c7488de 100644 --- a/packages/cli-linux-arm64-gnu/package.json +++ b/packages/cli-linux-arm64-gnu/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/cli-linux-arm64-gnu", - "version": "0.4.4", + "version": "0.4.5", "description": "Native scriptc compiler for linux-arm64-gnu", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/cli-linux-arm64-musl/package.json b/packages/cli-linux-arm64-musl/package.json index 3a601ca337..e671e926bd 100644 --- a/packages/cli-linux-arm64-musl/package.json +++ b/packages/cli-linux-arm64-musl/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/cli-linux-arm64-musl", - "version": "0.4.4", + "version": "0.4.5", "description": "Native scriptc compiler for linux-arm64-musl", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/cli-linux-x64-gnu/package.json b/packages/cli-linux-x64-gnu/package.json index 8f6b1aa5ee..2b56308fd7 100644 --- a/packages/cli-linux-x64-gnu/package.json +++ b/packages/cli-linux-x64-gnu/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/cli-linux-x64-gnu", - "version": "0.4.4", + "version": "0.4.5", "description": "Native scriptc compiler for linux-x64-gnu", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/cli-linux-x64-musl/package.json b/packages/cli-linux-x64-musl/package.json index ff0d1d6863..4ca4c28376 100644 --- a/packages/cli-linux-x64-musl/package.json +++ b/packages/cli-linux-x64-musl/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/cli-linux-x64-musl", - "version": "0.4.4", + "version": "0.4.5", "description": "Native scriptc compiler for linux-x64-musl", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/cli-win32-x64-msvc/package.json b/packages/cli-win32-x64-msvc/package.json index e94ba31df3..d734c0807b 100644 --- a/packages/cli-win32-x64-msvc/package.json +++ b/packages/cli-win32-x64-msvc/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/cli-win32-x64-msvc", - "version": "0.4.4", + "version": "0.4.5", "description": "Native scriptc compiler for win32-x64-msvc", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/cli/package.json b/packages/cli/package.json index 26cf10f076..59f9db90a7 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -1,6 +1,6 @@ { "name": "scriptc", - "version": "0.4.4", + "version": "0.4.5", "description": "Compile ordinary TypeScript and JavaScript to small, fast native executables and WebAssembly modules that run without Node.js", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/cli/test/native-link-info.test.ts b/packages/cli/test/native-link-info.test.ts index ef07c838d0..f63cc790cb 100644 --- a/packages/cli/test/native-link-info.test.ts +++ b/packages/cli/test/native-link-info.test.ts @@ -72,7 +72,7 @@ describe.runIf(supported)("macOS arm64 native link info", () => { minimum_os: "14.0", }, program: { object, entry_symbol: "main" }, - runtime_abi: { version: 44, marker: "scr_runtime_abi_v44" }, + runtime_abi: { version: 45, marker: "scr_runtime_abi_v45" }, runtime_pack: { kind: "precompiled", package: "@scriptc/runtime-darwin-arm64", @@ -165,6 +165,6 @@ describe.runIf(supported)("macOS arm64 native link info", () => { () => null, (failure: { stderr?: string }) => failure, ); - expect(error?.stderr).toContain("scr_runtime_abi_v44"); + expect(error?.stderr).toContain("scr_runtime_abi_v45"); }); }); diff --git a/packages/compiler/ambient/scriptc-node-fallback.d.ts b/packages/compiler/ambient/scriptc-node-fallback.d.ts index b0fce1d6f4..058b262ead 100644 --- a/packages/compiler/ambient/scriptc-node-fallback.d.ts +++ b/packages/compiler/ambient/scriptc-node-fallback.d.ts @@ -5345,10 +5345,10 @@ declare module "node:zlib" { * with the real port discoverable as address().port (the composed read * is the lowered form — bare address() has no lowering); close([cb]) * firing once connections drain; socket write/end/destroy/pipe and the - * data/end/close/error/connect events (on and once — listeners take at - * most the declared parameter). write/end return void here (Node returns - * the backpressure boolean / the socket); on/once return void (chaining - * is fenced). connect's host is a numeric IP or "localhost" (pinned to + * data/end/close/error/connect/drain events (on and once — listeners + * take at most the declared parameter). write returns Node's + * backpressure boolean; end returns void here (Node returns the socket); + * on/once return void (chaining is fenced). connect's host is a numeric IP or "localhost" (pinned to * 127.0.0.1 — SEMANTICS.md); every failure is the async 'error' event, * and an 'error' with no listener exits 1 like an unhandled EventEmitter * 'error'. */ @@ -5380,7 +5380,13 @@ declare module "net" { readonly bytesWritten: number; /* write's callback runs once the data is flushed, with null; end's * once the socket finishes. */ - write(data: string | Uint8Array, callback?: (err?: Error | null) => void): void; + /* false when the bytes this write left unsent reach the 65536-byte + * high-water mark, as in Node; 'drain' fires when they are all sent. */ + write(data: string | Uint8Array, callback?: (err?: Error | null) => void): boolean; + /* the bytes written and not yet sent */ + readonly writableLength: number; + /* a write answered false and 'drain' has not fired yet */ + readonly writableNeedDrain: boolean; end(callback?: () => void): void; end(data: string | Uint8Array, callback?: () => void): void; destroy(): void; @@ -5428,6 +5434,7 @@ declare module "net" { | "connect" | "ready" | "timeout" + | "drain" | "readable" | "finish" | "secureConnect" @@ -5452,6 +5459,7 @@ declare module "net" { | "connect" | "ready" | "timeout" + | "drain" | "readable" | "finish" | "secureConnect" @@ -5475,6 +5483,7 @@ declare module "net" { | "connect" | "ready" | "timeout" + | "drain" | "readable" | "finish" | "secureConnect" diff --git a/packages/compiler/package.json b/packages/compiler/package.json index 570c0a1a5b..03af418a5d 100644 --- a/packages/compiler/package.json +++ b/packages/compiler/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/compiler", - "version": "0.4.4", + "version": "0.4.5", "description": "The scriptc compiler — TypeScript/JavaScript frontend, typed IR, LLVM backend", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/compiler/src/backend/llvm/dense-array-access.ts b/packages/compiler/src/backend/llvm/dense-array-access.ts index df37d9c58a..e002ecd8a0 100644 --- a/packages/compiler/src/backend/llvm/dense-array-access.ts +++ b/packages/compiler/src/backend/llvm/dense-array-access.ts @@ -453,7 +453,11 @@ export function emitDenseArrayRead( B.br(join); B.startBlock(slow); const slowValue = B.tmp(); - const attrs = acc === "ref" && retain ? " cold" : READ_FALLBACK; + // ToNumber of a foreign element (SCR_ARR_FOREIGN) can run user code. + const attrs = + (acc === "ref" && retain) || (getter === "scr_arr_get_number" && host.mayHoldForeign(element)) + ? " cold" + : READ_FALLBACK; B.line(`${slowValue} = call ${ty} @${getter}(ptr ${array}, double ${index.name})${attrs}`); let slowInt: string | null = null; if (fastInt !== null) { @@ -610,6 +614,28 @@ export function emitDenseArraySet( B.line(`store ptr ${value}, ptr ${valuePtr}${host.fieldAliasAttachment(valuePtr)}`); } }; + if ((acc === "f64" || kind === "ref") && host.mayHoldForeign(element)) { + // Past the hole-free prefix a slot may hold a foreign element, whose + // value the runtime setter releases; below it every slot is a value. + B.br(slow); + B.startBlock(filledLabel); + const filledValuePtr = valuePointer(host, guard, kind); + let filledOld: string | null = null; + if (kind === "ref") { + filledOld = B.tmp(); + B.line( + `${filledOld} = load ptr, ptr ${filledValuePtr}${host.fieldAliasAttachment(filledValuePtr)}`, + ); + } + storeValue(filledValuePtr); + if (filledOld) host.releaseHeap(filledOld, element); + B.br(join); + B.startBlock(slow); + call(" cold"); + B.br(join); + B.startBlock(join); + return; + } const stated = B.newLabel("arr.set.states"), plain = B.newLabel("arr.set.plain"), plainStore = B.newLabel("arr.set.values"), diff --git a/packages/compiler/src/backend/llvm/dyn.ts b/packages/compiler/src/backend/llvm/dyn.ts index 9ba8c3c874..12ea996882 100644 --- a/packages/compiler/src/backend/llvm/dyn.ts +++ b/packages/compiler/src/backend/llvm/dyn.ts @@ -1980,6 +1980,87 @@ export class LlDyn { /** `sc_dc_(ptr d, ptr path) -> T` — validate the checked-dynamic tree against T and * BUILD the typed value (+1), or throw the catchable path-annotated * TypeError and return a dummy with the pending flag set. */ + /** The array a live view's snapshot commits back into, for element types + * whose arrays can hold foreign elements (LlEmitter.mayHoldForeign): + * like dynCheckHelper's array case on a checked array `%d`, except that + * an element the element type cannot hold is stored as a foreign element + * (undefined as a present undefined) instead of failing the conversion. + * Boolean arrays, whose slots cannot hold one, keep the failing check. */ + dynViewArrayHelper(t: IrType & { kind: "array" }): string { + const builderKey = `view:${typeKey(t)}`; + const existing = this.dynBuilders.get(builderKey); + if (existing) return existing; + const name = `sc_dva_${this.dynBuilders.size}`; + this.dynBuilders.set(builderKey, name); + const host = this.host; + const elem = t.elem; + const check = this.dynCheckHelper(elem); + const match = this.dynMatchHelper(elem); + const B = new BlockBuilder(); + const n = this.lenOf(B, "%d"); + const a = B.tmp(); + B.line(`${a} = ${arrNewCall(host, elem, n)}`); + const items = this.itemsOf(B, "%d"); + const acc = elemAccess(elem); + const accTy = acc === "f64" ? "double" : acc === "bool" ? "i1" : "ptr"; + host.declare( + `declare double @scr_arr_push_${acc}(ptr, ${acc === "bool" ? "i1 zeroext" : accTy})`, + ); + host.declare(`declare zeroext i1 @scr_dyn_arr_has_index(ptr, ${this.S})`); + host.declare(`declare void @scr_arr_set_len(ptr, double)`); + host.declare(`declare zeroext i1 @scr_arr_set_foreign(ptr, double, ptr)`); + this.i64Loop(B, "dva", n, (i) => { + const index = B.tmp(); + B.line(`${index} = ${sizeToDouble(this.S)} ${this.S} ${i} to double`); + const present = B.tmp(); + B.line(`${present} = call zeroext i1 @scr_dyn_arr_has_index(ptr %d, ${this.S} ${i})`); + const valueLabel = B.newLabel("dva.present"); + const holeLabel = B.newLabel("dva.hole"); + const typedLabel = B.newLabel("dva.typed"); + const foreignLabel = B.newLabel("dva.foreign"); + const doneLabel = B.newLabel("dva.done"); + B.condBr(present, valueLabel, holeLabel); + B.startBlock(holeLabel); + const length = B.tmp(); + B.line(`${length} = fadd double ${index}, 1.0`); + B.line(`call void @scr_arr_set_len(ptr ${a}, double ${length})`); + B.br(doneLabel); + B.startBlock(valueLabel); + const e = this.itemAt(B, items, i); + const fits = B.tmp(); + B.line(`${fits} = call zeroext i1 @${match}(ptr ${e})`); + B.condBr(fits, typedLabel, foreignLabel); + B.startBlock(foreignLabel); + const kept = B.tmp(); + B.line(`${kept} = call zeroext i1 @scr_arr_set_foreign(ptr ${a}, double ${index}, ptr ${e})`); + B.condBr(kept, doneLabel, typedLabel); + B.startBlock(typedLabel); + const v = B.tmp(); + B.line(`${v} = call ${this.valTy(elem)} @${check}(ptr ${e}, ptr null)`); + this.pendingBail( + B, + "dva", + () => { + host.declare(`declare void @scr_arr_release(ptr)`); + B.line(`call void @scr_arr_release(ptr ${a})`); + }, + "ptr null", + ); + const pushed = B.tmp(); + B.line(`${pushed} = call double @scr_arr_push_${acc}(ptr ${a}, ${accTy} ${v})`); + B.br(doneLabel); + B.startBlock(doneLabel); + }); + B.terminate(`ret ptr ${a}`); + this.defs.push( + `define internal ptr @${name}(ptr %d, ptr %path) ${FN_ATTRS} { ; view commit ${typeKey(t)}`, + B.render(), + `}`, + ``, + ); + return name; + } + dynCheckHelper(t: IrType, preserveRefs = false): string { // A `(T | undefined)[]` stored as T elements also takes undefined ones. const key = checkedConversionKey(t); @@ -3686,7 +3767,25 @@ export class LlDyn { B.br(doneLabel); B.startBlock(presentLabel); B.line(`${undefinedState} = fcmp oeq double ${state}, 2.0`); // SCR_ARR_UNDEFINED - B.terminate(`br i1 ${undefinedState}, label %${undefinedLabel}, label %${valueLabel}`); + if (host.mayHoldForeign(t.elem)) { + // A foreign element (SCR_ARR_FOREIGN) holds its own value. + const typedLabel = B.newLabel("tda.typed"); + const foreignLabel = B.newLabel("tda.foreign"); + B.terminate(`br i1 ${undefinedState}, label %${undefinedLabel}, label %${typedLabel}`); + B.startBlock(typedLabel); + const foreign = B.tmp(); + B.line(`${foreign} = fcmp oeq double ${state}, 3.0`); + B.condBr(foreign, foreignLabel, valueLabel); + B.startBlock(foreignLabel); + host.declare(`declare ptr @scr_arr_get_foreign(ptr, double)`); + host.declare(`declare void @scr_dyn_arr_push(ptr, ptr)`); + const payload = B.tmp(); + B.line(`${payload} = call ptr @scr_arr_get_foreign(ptr %v, double ${i}) ; +1`); + B.line(`call void @scr_dyn_arr_push(ptr ${d}, ptr ${payload})`); + B.br(doneLabel); + } else { + B.terminate(`br i1 ${undefinedState}, label %${undefinedLabel}, label %${valueLabel}`); + } B.startBlock(undefinedLabel); const retained = this.retainDyn(B, this.undef(B)); B.line(`call void @scr_dyn_arr_push(ptr ${d}, ptr ${retained})`); diff --git a/packages/compiler/src/backend/llvm/emit-module.ts b/packages/compiler/src/backend/llvm/emit-module.ts index ee90890710..af00ebc2ce 100644 --- a/packages/compiler/src/backend/llvm/emit-module.ts +++ b/packages/compiler/src/backend/llvm/emit-module.ts @@ -600,6 +600,23 @@ export function emitModuleParts(emitter: LlEmitter): string[] { regexMatchers.set(key, code); for (const decl of code.decls) emitter.declare(decl); } + // Replacers of global substitutions with a literal replacement; the call + // sites register only patterns inside the compiled subset. + const regexReplacers: RegexMatcherCode[] = []; + for (const rr of emitter.regexReplacers.values()) { + const code = compileRegexMatcher( + rr.sym, + rr.pattern, + rr.flags, + emitter.sizeType, + !emitter.wasi, + rr.replacement, + ); + if (code === null) + throw new InternalCompilerError(`llvm emitter bug: replacer ${rr.sym} did not compile`); + regexReplacers.push(code); + for (const decl of code.decls) emitter.declare(decl); + } const out: string[] = [ `; Generated by scriptc (LLVM backend) from ${emitter.mod.sourceFile}. Do not edit.`, @@ -882,6 +899,7 @@ export function emitModuleParts(emitter: LlEmitter): string[] { } if (emitter.regexInstances.size > 0) out.push(``); for (const code of regexMatchers.values()) out.push(...code.lines, ``); + for (const code of regexReplacers) out.push(...code.lines, ``); for (const [, inst] of emitter.templateStringsInstances) { // One immortal ScrArr per tagged-template site: a data global of // interned cooked-string literals, and the ScrArr header over it diff --git a/packages/compiler/src/backend/llvm/emitter.ts b/packages/compiler/src/backend/llvm/emitter.ts index 1a2c8bef84..5569599ee3 100644 --- a/packages/compiler/src/backend/llvm/emitter.ts +++ b/packages/compiler/src/backend/llvm/emitter.ts @@ -1303,6 +1303,12 @@ export class LlEmitter { string, { sym: string; src: string; fl: string; pattern: string; flags: string } >(); + /** Build-time replacers: one per (pattern, flags, replacement) of a + * global substitution with a literal replacement (regex-matchers.ts). */ + readonly regexReplacers = new Map< + string, + { sym: string; pattern: string; flags: string; replacement: string } + >(); /** Interned tagged-template strings objects: per-site key → { symbol, * interned cooked-literal refs }. One immortal ScrArr of string slots * per template SITE (the spec's per-occurrence identity — the C @@ -1633,6 +1639,9 @@ export class LlEmitter { * commit back into their objects. Library builds keep their commits: * their values cross a native boundary. */ readonly liveViewsReadOnly: boolean; + /** IrModule.foreignElementKeys: element keys whose arrays can hold + * SCR_ARR_FOREIGN elements. */ + readonly foreignElementKeys: ReadonlySet; /** The program freezes, seals or prevents extensions of untyped values * (IrModule.untypedIntegrity): the commits of class views store the * view's integrity state in the instance's integrity word. */ @@ -1925,6 +1934,7 @@ export class LlEmitter { !this.untypedIntegrity && !dynamicStores.all && dynamicStores.names.size === 0; + this.foreignElementKeys = new Set(mod.foreignElementKeys ?? []); this.int32Abis = this.debug === null ? findInt32Abis(mod, this.int32Slots) : new Map(); this.immortalGlobals = new Set(mod.contracts?.immortalGlobals ?? []); this.programLifetime = new Set(mod.contracts?.programLifetime ?? []); @@ -2219,6 +2229,7 @@ export class LlEmitter { declare: (decl) => this.declare(decl), needOom: () => this.needOom(), needBadTag: () => this.needBadTag(), + mayHoldForeign: (elem) => this.mayHoldForeign(elem), internLiteral: (text) => this.internLiteral(text), cstr: (text) => this.cstr(text), mapOps: (operands) => this.mapOps(operands), @@ -5439,6 +5450,8 @@ export class LlEmitter { return addNumbered(this.threadSlotsBlocks, key, value as string, "@sc_slots_", names); case "regex": return addNumberedSymbol(this.regexInstances, key, value as never, "sc_re_", names); + case "regexReplacer": + return addNumberedSymbol(this.regexReplacers, key, value as never, "sc_rr_", names); case "templateStrings": return addNumberedSymbol( this.templateStringsInstances, @@ -6716,6 +6729,12 @@ export class LlEmitter { } } + /** Arrays of this element type can hold SCR_ARR_FOREIGN elements. */ + mayHoldForeign(elem: IrType): boolean { + if (this.foreignElementKeys.size === 0) return false; + return this.foreignElementKeys.has("*") || this.foreignElementKeys.has(typeKey(elem)); + } + /** Lifetime regions in modules with lifetimeRegions (llvm/regions.ts). * Created with the class graph; null elsewhere. */ regions: LifetimeRegions | null = null; diff --git a/packages/compiler/src/backend/llvm/expr-containers.ts b/packages/compiler/src/backend/llvm/expr-containers.ts index 3074817370..696dad3abb 100644 --- a/packages/compiler/src/backend/llvm/expr-containers.ts +++ b/packages/compiler/src/backend/llvm/expr-containers.ts @@ -1,3 +1,4 @@ +import { dynDesc } from "../../ir/analysis.js"; /* Focused LLVM expression emission extracted from emitter.ts. */ import { arraySearchHelperName, arraySearchKind } from "./array-search.js"; import { InternalCompilerError } from "../../errors.js"; @@ -49,11 +50,13 @@ import { emitFusedMapSet, rememberMapEntry } from "./map-entry-fusion.js"; import { emitScratchStringSlice, emitStringSliceRead } from "./string-slices.js"; import { STRING_CHAR_AT_DECL, + STRING_CP_AT_BYTE_DECL, STRING_CHAR_AT_INT_MARKER, STRING_CHAR_CODE_AT_DECL, STRING_CHAR_CODE_AT_INT_MARKER, emitAsciiChar, emitAsciiCharCode, + emitStringByteLength, emitUnitCharCode, STRING_UTF16_LEN_DECL, } from "./string-reads.js"; @@ -564,16 +567,30 @@ export function emitStrIntrinsic( return call("scr_str_is_well_formed", "zeroext i1 (ptr)", `ptr ${r.name}`, "i1", false); case "toWellFormed": return call("scr_str_to_well_formed", "ptr (ptr)", `ptr ${r.name}`, "ptr", true); - case "cpAt": - // The code point AT an index as a one-code-point string (+1) — - // the string-for-of desugar's read. - return call( - "scr_str_cp_at", - "ptr (ptr, double)", - `ptr ${r.name}, double ${args[0]!.name}`, - "ptr", - true, - ); + case "byteLength": { + const t = B.tmp(); + // A byte count fits in a signed size, whose conversion is one instruction. + B.line(`${t} = sitofp ${host.sizeType} ${emitStringByteLength(host, r.name)} to double`); + return { name: t, type: e.type }; + } + case "cpAtByte": { + // sc_str_cp_at_byte (string-reads.ts) answers ASCII bytes from the + // charAt table inline. The cursor is a character start, an exact + // integer below the byte count. + host.declare(STRING_CHAR_AT_DECL); + host.declare(STRING_CP_AT_BYTE_DECL); + const index = e.args[0] ? exactInteger(host, args[0]!, e.args[0]) : null; + let wide: string; + if (index) { + wide = widenInteger(host, index); + } else { + wide = B.tmp(); + B.line(`${wide} = fptosi double ${args[0]!.name} to i64`); + } + const t = B.tmp(); + B.line(`${t} = call ptr @sc_str_cp_at_byte(ptr ${r.name}, i64 ${wide})`); + return host.own({ name: t, type: e.type }); + } default: { const _exhaustive: never = method; void _exhaustive; @@ -790,6 +807,9 @@ export function emitArrIntrinsic( case "shift": { host.emitPublishedArrayGuard(r.name, e.receiver.type, method === "pop" ? 2 : 6); const checkedValue = elem.kind === "dyn" && e.type.kind === "dyn"; + // A foreign element (SCR_ARR_FOREIGN) cannot be the typed result: + // removing one throws the checked conversion's TypeError. + const foreignElements = !checkedValue && host.mayHoldForeign(elem); if (!checkedValue && e.type.kind !== "union") throw new InternalCompilerError("llvm emitter bug: array removal result is not a union"); const def = e.type.kind === "union" ? host.unionsById.get(e.type.unionId) : undefined; @@ -832,7 +852,25 @@ export function emitArrIntrinsic( ); B.line(`${slowRaw} = load i64, ptr ${rawSlot}`); B.line(`${has} = icmp eq i8 ${state}, 1`); - B.condBr(has, lp, la); + const lf = foreignElements ? B.newLabel("remove.f") : la; + B.condBr(has, lp, lf); + if (foreignElements) { + // Only the runtime call removes a foreign element; its value moves + // out with the slot, and the throw releases it. + B.startBlock(lf); + const foreign = B.tmp(); + const throwLabel = B.newLabel("remove.foreign"); + B.line(`${foreign} = icmp eq i8 ${state}, 3`); + B.condBr(foreign, throwLabel, la); + B.startBlock(throwLabel); + const payload = B.tmp(); + B.line(`${payload} = inttoptr i64 ${slowRaw} to ptr`); + host.declare(`declare void @scr_dyn_foreign_throw(ptr, ptr)`); + const want = host.cstr(dynDesc(e.type, host.recordsById, host.unionsById)); + B.line(`call void @scr_dyn_foreign_throw(ptr ${payload}, ptr ${want})`); + host.emitAlwaysPendingUnwind(); + B.br(la); + } B.startBlock(lp); let raw = slowRaw; const value = B.tmp(); diff --git a/packages/compiler/src/backend/llvm/expr-primitives.ts b/packages/compiler/src/backend/llvm/expr-primitives.ts index d3b72897e8..eed138fe2a 100644 --- a/packages/compiler/src/backend/llvm/expr-primitives.ts +++ b/packages/compiler/src/backend/llvm/expr-primitives.ts @@ -661,6 +661,29 @@ export function regexLiteralTemplate( return `@${re.sym}`; } +/** The build-time replacer of a global substitution of regex literal `e` + * with the literal `replacement` (regex-matchers.ts), registered on first + * use. */ +export function regexLiteralReplacer( + host: LlvmEmitterContext, + e: IrExpr & { kind: "regexLit" }, + replacement: string, +): string { + const key = `${e.flags}/${e.pattern}\0${replacement}`; + let rr = host.regexReplacers.get(key); + if (!rr) { + rr = { + sym: `sc_rr_${host.regexReplacers.size}`, + pattern: e.pattern, + flags: e.flags, + replacement, + }; + host.regexReplacers.set(key, rr); + } + host.journal?.entry("regexReplacer", key, rr); + return `@${rr.sym}`; +} + /** A string equality or comparison between `s.charAt(i)` and a literal of * exactly one UTF-16 code unit: the charAt call, the unit, and whether the * charAt is the left operand. */ diff --git a/packages/compiler/src/backend/llvm/expr-records.ts b/packages/compiler/src/backend/llvm/expr-records.ts index 59ab625e5a..51688eafab 100644 --- a/packages/compiler/src/backend/llvm/expr-records.ts +++ b/packages/compiler/src/backend/llvm/expr-records.ts @@ -6,7 +6,8 @@ import { LlvmUnsupportedError } from "./unsupported.js"; import type { LlvmEmitterContext, LlValue } from "./expr-context.js"; import { f64Lit, llvmCommentText } from "./common.js"; import { borrowableInputs, emitBorrowedInputs, emitBorrowedInput } from "./borrowed-inputs.js"; -import { regexLiteralTemplate } from "./expr-primitives.js"; +import { regexLiteralReplacer, regexLiteralTemplate } from "./expr-primitives.js"; +import { canReplaceWith, regexLiteralCompiles } from "./regex-matchers.js"; import { isScalarUnionDef, loadInlineUnion, presentBits } from "./inline-unions.js"; import { preservesRegexInputs } from "./reference-effects.js"; @@ -160,6 +161,39 @@ function directRegexOperandMethod(method: string): boolean { } } +/** The build-time replacer of a substitution: a regex literal running on + * its template, with a literal replacement the replacer can write as is. + * Null when the substitution takes the runtime's loop. */ +function literalReplacer( + host: LlvmEmitterContext, + literal: IrExpr | null, + template: string | null, + replacement: IrExpr, +): string | null { + if (literal === null || template === null || literal.kind !== "regexLit") return null; + if (replacement.kind !== "strLit") return null; + if (!canReplaceWith(literal.pattern, literal.flags, replacement.value)) return null; + if (!regexLiteralCompiles(literal.pattern, literal.flags)) return null; + return regexLiteralReplacer(host, literal, replacement.value); +} + +/** scr_regex_replace_with: the runtime runs `replacer` on an ASCII subject + * and its own loop on any other. */ +function emitReplaceWith( + host: LlvmEmitterContext, + e: IrExpr, + r: LlValue, + args: LlValue[], + replacer: string, +): LlValue { + host.declare(`declare ptr @scr_regex_replace_with(ptr, ptr, ptr, ptr)`); + const t = host.B.tmp(); + host.B.line( + `${t} = call ptr @scr_regex_replace_with(ptr ${r.name}, ptr ${args[0]!.name}, ptr ${args[1]!.name}, ptr ${replacer})`, + ); + return host.own({ name: t, type: e.type }); +} + export function emitRegexIntrinsic( host: LlvmEmitterContext, e: IrExpr & { kind: "regexIntrinsic" }, @@ -234,11 +268,16 @@ export function emitRegexIntrinsic( "scr_regex_match_all_captures_into", `ptr ${r.name}, ptr ${args[0]!.name}, ptr ${args[1]!.name}`, ); - case "replaceAll": + case "replaceAll": { + // A global literal (the only kind a replacer is built for) never takes + // replaceAll's non-global TypeError. + const replacer = literalReplacer(host, literal, template, e.args[1]!); + if (replacer !== null) return emitReplaceWith(host, e, r, args, replacer); return fallible( "scr_regex_replace_all", `ptr ${r.name}, ptr ${args[0]!.name}, ptr ${args[1]!.name}`, ); + } case "split": host.declare(`declare ptr @scr_regex_split_limit(ptr, ptr, double)`); { @@ -323,6 +362,8 @@ export function emitRegexIntrinsic( return host.own({ name: t, type: e.type }); } case "replace": { + const replacer = literalReplacer(host, literal, template, e.args[1]!); + if (replacer !== null) return emitReplaceWith(host, e, r, args, replacer); host.declare(`declare ptr @scr_regex_replace(ptr, ptr, ptr)`); const t = B.tmp(); B.line( diff --git a/packages/compiler/src/backend/llvm/expr-stream-bridges.ts b/packages/compiler/src/backend/llvm/expr-stream-bridges.ts index 57172068f0..f711795d36 100644 --- a/packages/compiler/src/backend/llvm/expr-stream-bridges.ts +++ b/packages/compiler/src/backend/llvm/expr-stream-bridges.ts @@ -184,7 +184,11 @@ export function streamTypedRefCommitAdapter( } if (t.kind === "array") { const commit = `${snapshot}_commit`; - const check = host.dyn.dynCheckHelper(t); + // A value the element type cannot hold stays in the array as a foreign + // element where the program can store one (LlEmitter.mayHoldForeign). + const check = host.mayHoldForeign(t.elem) + ? host.dyn.dynViewArrayHelper(t) + : host.dyn.dynCheckHelper(t); host.declare(`declare void @scr_arr_release(ptr)`); host.declare(`declare void @scr_arr_outline(ptr)`); host.declare(`declare void @scr_arr_adopt_view_properties(ptr, ptr, ptr)`); @@ -1235,6 +1239,21 @@ export function streamTypedRefMaterializeAdapter( B.line(`call void @scr_dyn_arr_push_hole(ptr ${out})`); B.br(doneLabel); B.startBlock(undefinedLabel); + if (host.mayHoldForeign(elem)) { + // A foreign element (SCR_ARR_FOREIGN) holds its own value. + const foreign = B.tmp(); + const foreignLabel = B.newLabel("live.array.foreign"); + const presentUndefined = B.newLabel("live.array.present.undefined"); + B.line(`${foreign} = fcmp oeq double ${state}, 3.0`); + B.condBr(foreign, foreignLabel, presentUndefined); + B.startBlock(foreignLabel); + host.declare(`declare ptr @scr_arr_get_foreign(ptr, double)`); + const payload = B.tmp(); + B.line(`${payload} = call ptr @scr_arr_get_foreign(ptr %p, double ${index}) ; +1`); + B.line(`call void @scr_dyn_arr_push(ptr ${out}, ptr ${payload})`); + B.br(doneLabel); + B.startBlock(presentUndefined); + } const undefinedValue = B.tmp(); const retained = B.tmp(); B.line(`${undefinedValue} = call ptr @scr_dyn_undefined()`); @@ -1333,7 +1352,16 @@ function liveArrayOps( const refuse = S.newLabel("live.array.refuse"); S.condBr(fits, store, refuse); S.startBlock(refuse); - S.terminate(`ret i1 false`); + if (host.mayHoldForeign(elem)) { + // The array keeps a value its element type cannot hold as a foreign + // element, and undefined as a present undefined. + host.declare(`declare zeroext i1 @scr_arr_set_foreign(ptr, double, ptr)`); + const kept = S.tmp(); + S.line(`${kept} = call zeroext i1 @scr_arr_set_foreign(ptr %p, double %i, ptr %v)`); + S.terminate(`ret i1 ${kept}`); + } else { + S.terminate(`ret i1 false`); + } S.startBlock(store); const next = S.tmp(); S.line( diff --git a/packages/compiler/src/backend/llvm/js-ic.ts b/packages/compiler/src/backend/llvm/js-ic.ts index 11b894a603..d51b838bbf 100644 --- a/packages/compiler/src/backend/llvm/js-ic.ts +++ b/packages/compiler/src/backend/llvm/js-ic.ts @@ -406,9 +406,13 @@ function emitGet(host: Host, e: ExprOf<"call">, callee: IrFunction, key: string) inlineLoad(host, receiver.name, ic, INLINE_WAYS, INLINE_PROTO, (cell) => cell, hits, hit, probe); B.startBlock(probe); // The read's probe also answers keys the receiver's plain chain lacks. - host.declare(`declare ptr @scr_js_get_probe(ptr, ptr, ptr)`); + // Only `length` and `next` reads can be answered for a native array or + // iterator, so only those sites take the probe that tests them. + const probeFn = + key === "length" || key === "next" ? "scr_js_get_probe_native" : "scr_js_get_probe"; + host.declare(`declare ptr @${probeFn}(ptr, ptr, ptr)`); const probed = B.tmp(); - B.line(`${probed} = call ptr @scr_js_get_probe(ptr ${receiver.name}, ptr ${name}, ptr ${ic})`); + B.line(`${probed} = call ptr @${probeFn}(ptr ${receiver.name}, ptr ${name}, ptr ${ic})`); const found = B.tmp(); B.line(`${found} = icmp ne ptr ${probed}, null`); const probeEnd = B.currentLabel(); diff --git a/packages/compiler/src/backend/llvm/lib-dispatch.ts b/packages/compiler/src/backend/llvm/lib-dispatch.ts index 4af8182a7d..07356cdb95 100644 --- a/packages/compiler/src/backend/llvm/lib-dispatch.ts +++ b/packages/compiler/src/backend/llvm/lib-dispatch.ts @@ -1143,6 +1143,8 @@ export function emitLibCall(host: LlvmEmitterContext, e: LibCallExpr): LlValue { case "blob": case "wasm": case "array": + if (e.fn === "array.foreignThrow") return emitArrayForeignThrow(host, e); + return host.emitGenericLibCall(e); case "collection": case "record": case "disposable": @@ -1164,6 +1166,29 @@ export function emitLibCall(host: LlvmEmitterContext, e: LibCallExpr): LlValue { } } +/** `array.foreignThrow`: the checked conversion's TypeError for a foreign + * element read into a typed result. Always throws; the result is a dummy + * of the replaced read's type (the error.argTypeThrow pattern). */ +function emitArrayForeignThrow(host: LlvmEmitterContext, e: LibCallExpr): LlValue { + const B = host.B; + const arr = host.emitExpr(e.args[0]!); + const index = host.emitExpr(e.args[1]!); + const want = host.emitExpr(e.args[2]!); + host.declare(`declare void @scr_arr_foreign_throw(ptr, double, ptr)`); + B.line( + `call void @scr_arr_foreign_throw(ptr ${arr.name}, double ${index.name}, ptr ${want.name})`, + ); + const ty = host.llType(e.type); + if (ty === "void") { + host.emitAlwaysPendingUnwind(); + return { name: "", type: e.type }; + } + const dummy = ty === "double" ? "0.0" : ty === "i1" ? "false" : "null"; + const out = host.own({ name: dummy, type: e.type }); + host.emitAlwaysPendingUnwind(); + return out; +} + /** The settlement adapter of Promise.all's tuple form for the builder * `fnName`, which fulfills the result with the builder's tuple. */ function allTupleAdapterFor(host: LlvmEmitterContext, fnName: string, tuple: IrType): string { diff --git a/packages/compiler/src/backend/llvm/lib-network.ts b/packages/compiler/src/backend/llvm/lib-network.ts index d49bbde141..35eb885cdd 100644 --- a/packages/compiler/src/backend/llvm/lib-network.ts +++ b/packages/compiler/src/backend/llvm/lib-network.ts @@ -309,6 +309,7 @@ export function emitNetworkHttpLibCall(host: LlvmEmitterContext, e: LibCallExpr) e.fn === "net.sockOnClose" || e.fn === "net.sockOnConnect" || e.fn === "net.sockOnTimeout" || + e.fn === "net.sockOnDrain" || e.fn === "net.sockOnReadable" || e.fn === "net.sockOnReady" || e.fn === "http.reqOnEnd" || @@ -329,6 +330,7 @@ export function emitNetworkHttpLibCall(host: LlvmEmitterContext, e: LibCallExpr) "net.sockOnClose": "scr_net_sock_on_close", "net.sockOnConnect": "scr_net_sock_on_connect", "net.sockOnTimeout": "scr_net_sock_on_timeout", + "net.sockOnDrain": "scr_net_sock_on_drain", "net.sockOnReadable": "scr_net_sock_on_readable", "net.sockOnReady": "scr_net_sock_on_ready", "http.reqOnEnd": "scr_http_req_on_end", diff --git a/packages/compiler/src/backend/llvm/lib-shared.ts b/packages/compiler/src/backend/llvm/lib-shared.ts index def2dd6bf1..fb294aa300 100644 --- a/packages/compiler/src/backend/llvm/lib-shared.ts +++ b/packages/compiler/src/backend/llvm/lib-shared.ts @@ -104,6 +104,7 @@ export const LIB_FN_SYMS: Record = { "str.localeCase": "scr_str_locale_case", "str.replaceLiteral": "scr_str_replace_literal", "str.replaceChars": "scr_str_replace_chars", + "str.codePoints": "scr_str_code_points", "num.fromString": "scr_string_to_number", "math.round": "scr_math_round", "math.sin": "sin", @@ -1337,6 +1338,8 @@ export const LIB_FN_SYMS: Record = { "net.sockResetAndDestroy": "scr_net_sock_reset_and_destroy", "net.sockAddress": "scr_net_sock_address", "net.sockBytesWritten": "scr_net_sock_bytes_written", + "net.sockWritableLength": "scr_net_sock_writable_length", + "net.sockWritableNeedDrain": "scr_net_sock_writable_need_drain", "net.sockAddrField": "scr_net_sock_addr_field", "net.sockBytesRead": "scr_net_sock_bytes_read", "net.sockConnecting": "scr_net_sock_connecting", @@ -1661,6 +1664,7 @@ export const LIB_FN_SYMS: Record = { "regex.groupParseInt": "scr_regex_group_parse_int", "regex.resultProp": "scr_regex_result_prop", "array.namedKeys": "scr_arr_named_keys", + "array.foreign": "scr_arr_get_foreign", "array.hasNamedKey": "scr_arr_has_named_key", "array.hasProperties": "scr_arr_has_properties", "array.undefinedKey": "scr_arr_undefined_key", diff --git a/packages/compiler/src/backend/llvm/reference-writes.ts b/packages/compiler/src/backend/llvm/reference-writes.ts index 89132a715d..c37af81d40 100644 --- a/packages/compiler/src/backend/llvm/reference-writes.ts +++ b/packages/compiler/src/backend/llvm/reference-writes.ts @@ -107,6 +107,7 @@ const INERT_LIB_CALLS = new Set([ "regex.groupParseInt", "regex.resultProp", "array.namedKeys", + "array.foreign", "array.hasNamedKey", "array.hasProperties", "process.cwd", diff --git a/packages/compiler/src/backend/llvm/regex-matchers.test.ts b/packages/compiler/src/backend/llvm/regex-matchers.test.ts index d009de7c9d..5f93f72959 100644 --- a/packages/compiler/src/backend/llvm/regex-matchers.test.ts +++ b/packages/compiler/src/backend/llvm/regex-matchers.test.ts @@ -1,5 +1,6 @@ import { describe, expect, test } from "vitest"; import { + canReplaceWith, compileRegexMatcher, regexLiteralBailReason, regexLiteralCaptures, @@ -152,3 +153,46 @@ describe("regex literal matcher code", () => { expect(text).not.toContain("constant [128 x i8]"); }); }); + +describe("regex literal replacers", () => { + test("take global, non-sticky, non-plain-text patterns and ASCII replacements without $", () => { + expect(canReplaceWith("a[bc]", "g", "<2>")).toBe(true); + expect(canReplaceWith("a[bc]", "gi", "")).toBe(true); + expect(canReplaceWith("a[bc]", "", "x")).toBe(false); + expect(canReplaceWith("a[bc]", "gy", "x")).toBe(false); + expect(canReplaceWith("a[bc]", "g", "$1")).toBe(false); + expect(canReplaceWith("a[bc]", "g", "é")).toBe(false); + // Plain text is a string search at run time. + expect(canReplaceWith("&", "g", "&")).toBe(false); + expect(canReplaceWith("a\\.b", "g", "x")).toBe(false); + // Letters under i are not plain text. + expect(canReplaceWith("ab", "gi", "x")).toBe(true); + }); + + test("append the gap and the replacement and continue the search", () => { + const code = compileRegexMatcher("r", "<[^>]*>", "g", "i64", true, "|"); + const text = code!.lines.join("\n"); + expect(text).toContain('@r.rep = private unnamed_addr constant [1 x i8] c"|"'); + expect(text).toContain("call void @scr_re_replace_out_reserve(ptr %out, i64 %m.need)"); + expect(text).toContain("load <16 x i8>, ptr %m.src, align 1"); + expect(text).toMatch(/br i1 %m\.past, label %nomatch, label %scan/); + expect(text).not.toContain("m.ret:"); + expect(text).toContain("ret i32 %n.found"); + // The search after a match is a 16-byte scan, not a memchr call. + expect(text).not.toContain("@memchr"); + expect(code!.decls).toContain("declare void @scr_re_replace_out_reserve(ptr, i64)"); + // Without vectors the gap is a memcpy. + const scalar = compileRegexMatcher("r", "<[^>]*>", "g", "i64", false, "|")!.lines.join("\n"); + expect(scalar).not.toContain("<16 x i8>"); + expect(scalar).toContain( + "call void @llvm.memcpy.p0.p0.i64(ptr %m.dst, ptr %m.src, i64 %m.gapw, i1 false)", + ); + }); + + test("escape the replacement bytes and skip the copy of an empty one", () => { + const quoted = compileRegexMatcher("r", "a[bc]", "g", "i64", true, '"\\\n')!.lines.join("\n"); + expect(quoted).toContain('c"\\22\\5C\\0A"'); + const empty = compileRegexMatcher("r", "a[bc]", "g", "i64", true, "")!.lines.join("\n"); + expect(empty).not.toContain("@r.rep"); + }); +}); diff --git a/packages/compiler/src/backend/llvm/regex-matchers.ts b/packages/compiler/src/backend/llvm/regex-matchers.ts index d83d6bae54..0491a64469 100644 --- a/packages/compiler/src/backend/llvm/regex-matchers.ts +++ b/packages/compiler/src/backend/llvm/regex-matchers.ts @@ -24,6 +24,8 @@ * `len`, it returns the capture count (whole match included) so the runtime * can check it against the compiled bytecode. */ +import { InternalCompilerError } from "../../errors.js"; + const ASCII = 128; const INF = -1; /** Larger patterns keep the interpreter (binary size). */ @@ -1853,10 +1855,13 @@ export function compileRegexMatcher( flags: string, sizeType: string, vector: boolean, + replacement: string | null = null, ): RegexMatcherCode | null { const parsed = parseSubset(pattern, flags); if (parsed === null) return null; const sticky = flags.includes("y"); + if (replacement !== null && !canReplaceWith(pattern, flags, replacement)) + throw new InternalCompilerError(`llvm emitter bug: no replacer for /${pattern}/${flags}`); const { root, ncap } = parsed; const gen = new RxGen(sym, ncap, vector); const end: FirstInfo = { set: emptySet(), nullable: true, bol: false, uncond: true }; @@ -1873,6 +1878,12 @@ export function compileRegexMatcher( fn.push(` %stk = alloca ptr`); fn.push(` %sbuf = alloca [64 x i32]`); for (let r = 0; r < gen.regs; r++) fn.push(` %r${r} = alloca i32`); + if (replacement !== null) { + fn.push(` %next = alloca i32`); + fn.push(` %nfound = alloca i32`); + fn.push(` store i32 0, ptr %next`); + fn.push(` store i32 0, ptr %nfound`); + } fn.push(` store ptr %sbuf, ptr %stk`); fn.push(` store i32 64, ptr %scap`); fn.push(` store i32 %start, ptr %at`); @@ -1900,7 +1911,9 @@ export function compileRegexMatcher( } else { fn.push(` br label %seek`); } - if (firstCount === 1) { + // A replacer seeks again after every match, often a few bytes on: the + // 16-byte scan avoids a memchr call per match. + if (firstCount === 1 && replacement === null) { fn.push(`seek:`); const byte = intervals(first.set)[0]![0]!; fn.push(` %rest = sub i32 %len, %a`); @@ -1944,6 +1957,98 @@ export function compileRegexMatcher( fn.push(` br i1 %over, label %nomatch, label %scan`); } + if (replacement !== null) { + emitReplaceMatch(fn, sym, replacement, vector); + } else { + emitReturnMatch(fn, ncap); + } + + const result = replacement !== null ? "%n.found" : "0"; + fn.push(`nomatch:`); + if (replacement !== null) { + fn.push(` %n.next = load i32, ptr %next`); + fn.push(` %n.nextp = getelementptr inbounds i8, ptr %out, i64 16`); + fn.push(` %n.nextw = sext i32 %n.next to i64`); + fn.push(` store i64 %n.nextw, ptr %n.nextp`); + fn.push(` %n.found = load i32, ptr %nfound`); + } + fn.push(` %n.stk = load ptr, ptr %stk`); + fn.push(` %n.heap = icmp ne ptr %n.stk, %sbuf`); + fn.push(` br i1 %n.heap, label %n.free, label %n.ret`); + fn.push(`n.free:`); + fn.push(` call void @scr_regex_stack_free(ptr %n.stk)`); + fn.push(` br label %n.ret`); + fn.push(`n.ret:`); + fn.push(` ret i32 ${result}`); + + gen.emitTail(); + for (const l of gen.body) fn.push(l); + fn.push(`}`); + + const decls = REGEX_MATCHER_DECLS.slice(); + if (gen.body.some((l) => l.includes("@llvm.cttz.i16"))) + decls.push(`declare i16 @llvm.cttz.i16(i16, i1 immarg)`); + if (!sticky && !first.nullable && firstCount === 1 && replacement === null) + decls.push(`declare ptr @memchr(ptr, i32, ${sizeType})`); + if (replacement !== null) { + decls.push(`declare void @scr_re_replace_out_reserve(ptr, i64)`); + decls.push(`declare void @llvm.memcpy.p0.p0.i64(ptr, ptr, i64, i1 immarg)`); + } + const globals = [...gen.globals]; + if (replacement !== null && replacement.length > 0) + globals.push( + `@${sym}.rep = private unnamed_addr constant [${replacement.length} x i8] c"${llvmBytes(replacement)}"`, + ); + return { lines: [...globals, ...fn], decls }; +} + +/** Whether a substitution can run through a build-time replacer: a global, + * non-sticky pattern that is not plain text (the runtime searches those as + * strings) and an ASCII replacement without `$` patterns. */ +export function canReplaceWith(pattern: string, flags: string, replacement: string): boolean { + if (!flags.includes("g") || flags.includes("y") || isPlainTextPattern(pattern, flags)) + return false; + for (let i = 0; i < replacement.length; i++) { + const c = replacement.charCodeAt(i); + if (c >= ASCII || c === 0x24) return false; + } + return true; +} + +/** scr_regex_plain_text's rule: a pattern of literal characters and escaped + * metacharacters (no letters under `i`) that the runtime searches as text. */ +function isPlainTextPattern(pattern: string, flags: string): boolean { + const fold = flags.includes("i"); + if (pattern.length === 0 || pattern.length > 256) return false; + for (let i = 0; i < pattern.length; i++) { + let c = pattern[i]!; + if (c === "\\") { + if (i + 1 >= pattern.length) return false; + c = pattern[++i]!; + if (!"^$\\.*+?()[]{}|/".includes(c)) return false; + } else { + const code = c.charCodeAt(0); + if (code < 0x20 || code >= 0x7f || "^$.*+?()[]{}|".includes(c)) return false; + } + if (fold && /[a-z]/i.test(c)) return false; + } + return true; +} + +function llvmBytes(text: string): string { + let out = ""; + for (let i = 0; i < text.length; i++) { + const c = text.charCodeAt(i); + out += + c >= 0x20 && c < 0x7f && c !== 0x22 && c !== 0x5c + ? text[i] + : `\\${c.toString(16).toUpperCase().padStart(2, "0")}`; + } + return out; +} + +/** The match block of a matcher: store the capture pointers, return 1. */ +function emitReturnMatch(fn: string[], ncap: number): void { fn.push(`match:`); fn.push(` %m.end = load i32, ptr %pos`); fn.push(` %m.start = load i32, ptr %at`); @@ -1969,25 +2074,72 @@ export function compileRegexMatcher( fn.push(` br label %m.ret`); fn.push(`m.ret:`); fn.push(` ret i32 1`); +} - fn.push(`nomatch:`); - fn.push(` %n.stk = load ptr, ptr %stk`); - fn.push(` %n.heap = icmp ne ptr %n.stk, %sbuf`); - fn.push(` br i1 %n.heap, label %n.free, label %n.ret`); - fn.push(`n.free:`); - fn.push(` call void @scr_regex_stack_free(ptr %n.stk)`); - fn.push(` br label %n.ret`); - fn.push(`n.ret:`); - fn.push(` ret i32 0`); - - gen.emitTail(); - for (const l of gen.body) fn.push(l); - fn.push(`}`); - - const decls = REGEX_MATCHER_DECLS.slice(); - if (gen.body.some((l) => l.includes("@llvm.cttz.i16"))) - decls.push(`declare i16 @llvm.cttz.i16(i16, i1 immarg)`); - if (!sticky && !first.nullable && firstCount === 1) - decls.push(`declare ptr @memchr(ptr, i32, ${sizeType})`); - return { lines: [...gen.globals, ...fn], decls }; +/** The match block of a replacer: append the gap since the previous match + * and the replacement to the output (ScrReReplaceOut: len, cap, next at + * 0, 8 and 16, then data), and continue the search at the match end, or one past it for an + * empty match, as the runtime's replace loop does. */ +function emitReplaceMatch(fn: string[], sym: string, replacement: string, vector: boolean): void { + const n = replacement.length; + fn.push(`match:`); + fn.push(` %m.end = load i32, ptr %pos`); + fn.push(` %m.start = load i32, ptr %at`); + fn.push(` %m.next = load i32, ptr %next`); + fn.push(` %m.gap = sub i32 %m.start, %m.next`); + fn.push(` %m.gapw = zext i32 %m.gap to i64`); + fn.push(` %m.lenp = getelementptr inbounds i8, ptr %out, i64 0`); + fn.push(` %m.capp = getelementptr inbounds i8, ptr %out, i64 8`); + fn.push(` %m.datap = getelementptr inbounds i8, ptr %out, i64 24`); + fn.push(` %m.len0 = load i64, ptr %m.lenp`); + fn.push(` %m.cap = load i64, ptr %m.capp`); + fn.push(` %m.room = sub i64 %m.cap, %m.len0`); + // 16 bytes of slack past the gap and the replacement for the 16-byte copy. + fn.push(` %m.need = add i64 %m.gapw, ${n + 32}`); + fn.push(` %m.fits = icmp uge i64 %m.room, %m.need`); + fn.push(` br i1 %m.fits, label %m.put, label %m.grow`); + fn.push(`m.grow:`); + fn.push(` call void @scr_re_replace_out_reserve(ptr %out, i64 %m.need)`); + fn.push(` br label %m.put`); + fn.push(`m.put:`); + fn.push(` %m.data = load ptr, ptr %m.datap`); + fn.push(` %m.len = load i64, ptr %m.lenp`); + fn.push(` %m.dst = getelementptr inbounds i8, ptr %m.data, i64 %m.len`); + fn.push(` %m.nextw = sext i32 %m.next to i64`); + fn.push(` %m.src = getelementptr inbounds i8, ptr %s, i64 %m.nextw`); + if (vector) { + // A gap of at most 16 bytes with 16 readable subject bytes is one + // unaligned 16-byte copy. + fn.push(` %m.small = icmp ule i32 %m.gap, 16`); + fn.push(` %m.rest = sub i32 %len, %m.next`); + fn.push(` %m.readable = icmp sge i32 %m.rest, 16`); + fn.push(` %m.fast = and i1 %m.small, %m.readable`); + fn.push(` br i1 %m.fast, label %m.copy16, label %m.copy`); + fn.push(`m.copy16:`); + fn.push(` %m.v = load <16 x i8>, ptr %m.src, align 1`); + fn.push(` store <16 x i8> %m.v, ptr %m.dst, align 1`); + fn.push(` br label %m.rep`); + } else { + fn.push(` br label %m.copy`); + } + fn.push(`m.copy:`); + fn.push(` call void @llvm.memcpy.p0.p0.i64(ptr %m.dst, ptr %m.src, i64 %m.gapw, i1 false)`); + fn.push(` br label %m.rep`); + fn.push(`m.rep:`); + fn.push(` %m.rdst = getelementptr inbounds i8, ptr %m.dst, i64 %m.gapw`); + if (n > 0) + fn.push(` call void @llvm.memcpy.p0.p0.i64(ptr %m.rdst, ptr @${sym}.rep, i64 ${n}, i1 false)`); + fn.push(` %m.len1 = add i64 %m.len, %m.gapw`); + fn.push(` %m.len2 = add i64 %m.len1, ${n}`); + fn.push(` store i64 %m.len2, ptr %m.lenp`); + fn.push(` store i32 %m.end, ptr %next`); + fn.push(` %m.found0 = load i32, ptr %nfound`); + fn.push(` %m.found1 = add i32 %m.found0, 1`); + fn.push(` store i32 %m.found1, ptr %nfound`); + fn.push(` %m.empty = icmp eq i32 %m.end, %m.start`); + fn.push(` %m.after = add i32 %m.end, 1`); + fn.push(` %m.at = select i1 %m.empty, i32 %m.after, i32 %m.end`); + fn.push(` store i32 %m.at, ptr %at`); + fn.push(` %m.past = icmp sgt i32 %m.at, %len`); + fn.push(` br i1 %m.past, label %nomatch, label %scan`); } diff --git a/packages/compiler/src/backend/llvm/shapes.test.ts b/packages/compiler/src/backend/llvm/shapes.test.ts index 61c1a074d3..d9e5fa6416 100644 --- a/packages/compiler/src/backend/llvm/shapes.test.ts +++ b/packages/compiler/src/backend/llvm/shapes.test.ts @@ -146,11 +146,19 @@ describe("LLVM runtime RC symbols", () => { // The dyn fast path buffers surviving values by kind, as scr_dyn_release does. const ir = emitInlineRcHelpers(host).join("\n"); expect(ir).toContain("call void @scr_dyn_release_survivor(ptr %o)"); - expect(ir).toMatch( - /switch i32 %kind, label %done \[ i32 4, label %container i32 5, label %container i32 8, label %root/, + // Every kind that can be a candidate takes one path; scalars stay inline. + expect(ir).toContain( + "switch i32 %kind, label %done [ i32 4, label %candidate i32 5, label %candidate i32 8, label %candidate i32 9, label %candidate i32 12, label %candidate i32 13, label %candidate ]", ); - // Arrays and objects not yet buffered ask the runtime, which skips intact documents. + // Values not yet buffered ask the runtime, which decides per value. expect(ir).toMatch(/br i1 %cqueued, label %done, label %survivor/); + // The runtime buffers dyn candidates, so the dyn helper has no inline append. + const dynHelper = ir.slice( + ir.indexOf("@sc_rc_release_dyn("), + ir.indexOf("\n}", ir.indexOf("@sc_rc_release_dyn(")), + ); + expect(dynHelper).not.toContain("sc_cyc_append"); + expect(dynHelper).not.toContain("scr_cyc_on_release"); }); test("sanitized builds call the fast-path helpers out of line", () => { @@ -208,36 +216,42 @@ describe("LLVM runtime RC symbols", () => { }); test("optimized builds buffer candidates through the module's append helper", () => { - const emit = (inlineRc: true | "release", extra: Partial) => { + const emit = (inlineRc: true | "release", extra: Partial, kind: "union" | "dyn") => { const { host } = declarationHost(64, inlineRc); const configured: ShapeHost = { ...host, inlineAlloc: true, ...extra }; - releaseSym(configured, { kind: "dyn" }); + releaseSym(configured, kind === "union" ? ({ kind, unionId: "u0" } as never) : { kind }); return { ir: emitInlineRcHelpers(configured).join("\n"), decls: inlineRcDecls(configured), }; }; + const { ir, decls } = emit(true, {}, "union"); + expect(ir).toContain("call preserve_mostcc void @sc_cyc_append(ptr %o)"); + expect(ir).toContain("define internal preserve_mostcc void @sc_cyc_append(ptr %o) noinline"); + // The young test reads the generation byte at obj-5; the append fills + // scr_cyc_buffers' nursery buffer and the header's position at obj-4. + expect(ir).toContain("getelementptr i8, ptr %o, i64 -5"); + expect(ir).toContain("getelementptr inbounds i8, ptr @scr_cyc_buffers, i64 96"); + expect(ir).toContain("getelementptr inbounds i8, ptr @scr_cyc_buffers, i64 104"); + expect(ir).toContain("load i64, ptr @scr_cyc_live"); + expect(ir).toContain("or i32 %i, -2147483648"); + expect(ir).toContain("call void @scr_cyc_on_release(ptr %o)"); + expect(decls).toContain( + "@scr_cyc_buffers = external global { [3 x { ptr, i64, i64, i64 }], i64, i64 }", + ); + expect(decls).toContain("@scr_cyc_live = external global i64"); + // The dyn release buffers through scr_dyn_release_survivor, so a module + // whose only headered release is dyn's has no append helper. for (const mode of [true, "release"] as const) { - const { ir, decls } = emit(mode, {}); - expect(ir).toContain("call preserve_mostcc void @sc_cyc_append(ptr %o)"); - expect(ir).toContain("define internal preserve_mostcc void @sc_cyc_append(ptr %o) noinline"); - // The young test reads the generation byte at obj-5; the append fills - // scr_cyc_buffers' nursery buffer and the header's position at obj-4. - expect(ir).toContain("getelementptr i8, ptr %o, i64 -5"); - expect(ir).toContain("getelementptr inbounds i8, ptr @scr_cyc_buffers, i64 96"); - expect(ir).toContain("getelementptr inbounds i8, ptr @scr_cyc_buffers, i64 104"); - expect(ir).toContain("load i64, ptr @scr_cyc_live"); - expect(ir).toContain("or i32 %i, -2147483648"); - expect(ir).toContain("call void @scr_cyc_on_release(ptr %o)"); - expect(decls).toContain( - "@scr_cyc_buffers = external global { [3 x { ptr, i64, i64, i64 }], i64, i64 }", - ); - expect(decls).toContain("@scr_cyc_live = external global i64"); + const dyn = emit(mode, {}, "dyn"); + expect(dyn.ir).not.toContain("sc_cyc_append"); + expect(dyn.ir).not.toContain("scr_cyc_on_release"); + expect(dyn.decls.join("\n")).not.toContain("scr_cyc_buffers"); } // Worker executables and targets without inline allocation call // scr_cyc_on_release, which tries the same append first. for (const extra of [{ threadLocalAlloc: true }, { inlineAlloc: false }]) { - const { ir, decls } = emit(true, extra); + const { ir, decls } = emit(true, extra, "union"); expect(ir).not.toContain("sc_cyc_append"); expect(ir).toContain("call void @scr_cyc_on_release(ptr %o)"); expect(decls.join("\n")).not.toContain("scr_cyc_buffers"); diff --git a/packages/compiler/src/backend/llvm/shapes.ts b/packages/compiler/src/backend/llvm/shapes.ts index ab623f9341..7498235b04 100644 --- a/packages/compiler/src/backend/llvm/shapes.ts +++ b/packages/compiler/src/backend/llvm/shapes.ts @@ -589,9 +589,10 @@ interface InlineRcFamily { * repeats its own checks, so the fast path never duplicates it. * * The dyn release buffers a surviving value by kind (scr_dyn_release): - * the fast path decides the always and never cases and calls - * scr_dyn_release_survivor for kinds whose candidacy is per value, - * including arrays and objects of intact documents (ScrDynRegion). + * the fast path skips scalars and values already in a candidate buffer or + * never candidates, and calls scr_dyn_release_survivor for the rest, which + * decides per value (intact documents, ScrDynRegion; capsules and handles + * without edges). * * The array release inlines the decrement of an array that stays alive * and is not headered (`a->cyc`): those are most array releases in @@ -698,9 +699,10 @@ function requestRcHelper(host: ShapeHost, family: string, op: RcOp): string { for (const decl of rcHelperDecls(family, op)) host.declare(decl); // Its body is written last (emitInlineRcHelpers), after the requests are // collected: a release that can buffer a candidate requests the append - // helper now. The jsval row (JS_RC_ROW) has no collector header; its - // cells release through dyn's helper, which makes its own request. - if (op === "release" && row.cycle === "always") requestCycleAppend(host); + // helper now. The dyn release buffers through scr_dyn_release_survivor + // instead. The jsval row (JS_RC_ROW) has no collector header; its cells + // release through dyn's helper. + if (op === "release" && row.cycle === "always" && family !== "dyn") requestCycleAppend(host); return `@${inlineRcName(family, op)}`; } @@ -820,8 +822,11 @@ function rcHelperDecls(family: string, op: RcOp): string[] { if (op === "retain" || row.release === null) return []; return [ `declare void @${row.release}(ptr)`, - ...(row.cycle === "always" ? [`declare void @scr_cyc_on_release(ptr)`] : []), - ...(family === "dyn" ? [`declare void @scr_dyn_release_survivor(ptr)`] : []), + ...(family === "dyn" + ? [`declare void @scr_dyn_release_survivor(ptr)`] + : row.cycle === "always" + ? [`declare void @scr_cyc_on_release(ptr)`] + : []), ]; } @@ -931,21 +936,21 @@ function releaseHelper(host: ShapeHost, family: string, row: InlineRcFamily): st // scr_dyn_release buffers a surviving value by kind: always for // functions and proxies, per value for arrays and objects (not // while their document is intact, ScrDynRegion), capsules and - // handles (scr_dyn_release_survivor), never for scalars. An - // array or object already in a candidate buffer stays inline. + // handles, never for scalars. Every kind that can be buffered + // takes one path, which keeps each inlined site small: a value + // already in a candidate buffer (or never one) stays inline, and + // scr_dyn_release_survivor decides the rest. ` %kp = getelementptr i8, ptr %o, ${S} ${S === "i64" ? 8 : 4} ; ScrDyn.kind`, ` %kind = load i32, ptr %kp`, - // SCR_DYN_ARR 4, OBJ 5, FUNC 8, PROXY 13; HANDLE 9, TYPED_REF 12. - ` switch i32 %kind, label %done [ i32 4, label %container i32 5, label %container i32 8, label %root i32 13, label %root i32 9, label %survivor i32 12, label %survivor ]`, - `container:`, + // SCR_DYN_ARR 4, OBJ 5, FUNC 8, HANDLE 9, TYPED_REF 12, PROXY 13. + ` switch i32 %kind, label %done [ i32 4, label %candidate i32 5, label %candidate i32 8, label %candidate i32 9, label %candidate i32 12, label %candidate i32 13, label %candidate ]`, + `candidate:`, ` %cbufp = getelementptr i8, ptr %o, ${S} -${host.cycleBufOffset}`, ` %cbuf = load i32, ptr %cbufp`, ` %cqueued = icmp slt i32 %cbuf, 0`, ` br i1 %cqueued, label %done, label %survivor`, `survivor:`, ` call void @scr_dyn_release_survivor(ptr %o)`, - ` br label %done`, - `root:`, ] : family === "box" ? [ @@ -964,7 +969,9 @@ function releaseHelper(host: ShapeHost, family: string, row: InlineRcFamily): st `root:`, ] : []), - ...(cycle === "always" ? cycleRootLines(host, "done") : [` br label %done`]), + ...(cycle === "always" && family !== "dyn" + ? cycleRootLines(host, "done") + : [` br label %done`]), `done:`, ` ret void`, `}`, diff --git a/packages/compiler/src/backend/llvm/string-lifetimes.test.ts b/packages/compiler/src/backend/llvm/string-lifetimes.test.ts index 07ad311533..72fc577523 100644 --- a/packages/compiler/src/backend/llvm/string-lifetimes.test.ts +++ b/packages/compiler/src/backend/llvm/string-lifetimes.test.ts @@ -341,7 +341,7 @@ const methodCases: { { method: "normalize", args: [ref("argument")], result: STRING, target: "normalize" }, { method: "isWellFormed", args: [], result: BOOL, target: "is_well_formed" }, { method: "toWellFormed", args: [], result: STRING, target: "to_well_formed" }, - { method: "cpAt", args: [num(1)], result: STRING, target: "cp_at" }, + { method: "cpAtByte", args: [num(1)], result: STRING, target: "cp_at_byte", inline: true }, ]; test.each(methodCases)( diff --git a/packages/compiler/src/backend/llvm/string-lifetimes.ts b/packages/compiler/src/backend/llvm/string-lifetimes.ts index 6c3079cc16..5a45048134 100644 --- a/packages/compiler/src/backend/llvm/string-lifetimes.ts +++ b/packages/compiler/src/backend/llvm/string-lifetimes.ts @@ -30,7 +30,8 @@ export function borrowsStringInputs(method: IrStrIntrinsicMethod): boolean { case "normalize": case "isWellFormed": case "toWellFormed": - case "cpAt": + case "byteLength": + case "cpAtByte": return true; default: return false; diff --git a/packages/compiler/src/backend/llvm/string-reads.ts b/packages/compiler/src/backend/llvm/string-reads.ts index 5b7a26c6c4..1f77de2c29 100644 --- a/packages/compiler/src/backend/llvm/string-reads.ts +++ b/packages/compiler/src/backend/llvm/string-reads.ts @@ -43,6 +43,10 @@ export const STRING_CHAR_CODE_AT_INT_MARKER = "; uses @sc_str_char_code_at_int"; export const STRING_CHAR_AT_DECL = "declare ptr @scr_str_char_at(ptr, double)"; /** Requests the integer-index charAt helper, as the marker above does. */ export const STRING_CHAR_AT_INT_MARKER = "; uses @sc_str_char_at_int"; +/** The code-point character starting at a byte offset (the string + * iterator's step). Requested together with STRING_CHAR_AT_DECL, which + * defines the ASCII table. */ +export const STRING_CP_AT_BYTE_DECL = "declare ptr @scr_str_cp_at_byte(ptr, i64)"; /** The proven-ASCII flag of string `s`, as an i1. */ export function emitStringAsciiFlag(host: LlvmEmitterContext, s: string): string { @@ -351,6 +355,29 @@ export const STRING_READ_HELPERS: readonly StringReadHelper[] = [ ]; }, }, + { + // The string iterator's step at byte `%b`, a character start below the + // byte count: an ASCII byte is the table entry for it; any other lead + // byte takes the runtime, which copies the character's bytes. + decl: STRING_CP_AT_BYTE_DECL, + define: (sz, attrs) => [ + `define internal ptr @sc_str_cp_at_byte(ptr %s, i64 %b) ${attrs} {`, + `entry:`, + ` %data = getelementptr inbounds %ScrStr, ptr %s, i32 1`, + ` %p = getelementptr inbounds i8, ptr %data, i64 %b`, + ` %byte = load i8, ptr %p`, + ` %ascii = icmp sge i8 %byte, 0`, + ` br i1 %ascii, label %read, label %slow`, + `read:`, + ` %code = zext i8 %byte to i64`, + ` %char = getelementptr inbounds [128 x { ${sz}, ${sz}, ${sz}, [2 x i8] }], ptr @sc_ascii_chars, i64 0, i64 %code`, + ` ret ptr %char`, + `slow:`, + ` %r = call ptr @scr_str_cp_at_byte(ptr %s, i64 %b) nounwind willreturn`, + ` ret ptr %r`, + `}`, + ], + }, { // An exact integer index, as for sc_str_char_code_at_int. Requested // together with STRING_CHAR_AT_DECL, which defines the table. diff --git a/packages/compiler/src/backend/llvm/walkers.ts b/packages/compiler/src/backend/llvm/walkers.ts index 58df3e845c..7c67e98ae1 100644 --- a/packages/compiler/src/backend/llvm/walkers.ts +++ b/packages/compiler/src/backend/llvm/walkers.ts @@ -26,6 +26,7 @@ import { llFieldType, loadStructField, releaseSym, + retainSym, traceAdapter, type ShapeHost, } from "./shapes.js"; @@ -41,6 +42,8 @@ import { undefinedArmTag } from "../../ir/analysis.js"; * scr_error-style label texts). */ export interface WalkerHost extends ShapeHost { readonly unionsById: Map; + /** Arrays of this element type can hold SCR_ARR_FOREIGN elements. */ + mayHoldForeign(elem: IrType): boolean; /** `@`-ref of an interned immortal ScrStr literal. */ internLiteral(text: string): string; /** `@`-ref of an interned NUL-terminated byte-array constant. */ @@ -325,9 +328,7 @@ export class LlWalkers { if (i > 0) this.putc(B, "%b", "44"); // ',' if (edgeable(f.type)) this.jbEdgeIdx(B, String(i)); const v = this.loadField(B, "%v", shapeId, fieldIndex.get(f.name)!, f.type); - B.line( - `call void @${this.jsonWriteHelper(f.type)}(ptr %b, ${this.valTy(f.type)} ${v}) ; [${llvmCommentText(f.name)}]`, - ); + this.writeField(B, v, f.type, `[${llvmCommentText(f.name)}]`); }); this.putc(B, "%b", "93"); // ']' if (cyclic) this.jbLeave(B); @@ -359,9 +360,7 @@ export class LlWalkers { this.puts(B, "%b", `${i > 0 ? "," : ""}${jsonObjectKeyLabel(f.name)}`); if (edgeable(f.type)) this.jbEdgeProp(B, f.name); const v = this.loadField(B, "%v", shapeId, fieldIndex.get(f.name)!, f.type); - B.line( - `call void @${this.jsonWriteHelper(f.type)}(ptr %b, ${this.valTy(f.type)} ${v}) ; ${llvmCommentText(f.name)}`, - ); + this.writeField(B, v, f.type, `${llvmCommentText(f.name)}`); }); } else { const first = B.slot(); @@ -396,9 +395,7 @@ export class LlWalkers { comma(); this.puts(B, "%b", jsonObjectKeyLabel(f.name)); if (edgeable(f.type)) this.jbEdgeProp(B, f.name); - B.line( - `call void @${this.jsonWriteHelper(f.type)}(ptr %b, ${this.valTy(f.type)} ${v}) ; ${llvmCommentText(f.name)}`, - ); + this.writeField(B, v, f.type, `${llvmCommentText(f.name)}`); if (skip !== null) { B.br(skip); B.startBlock(skip); @@ -520,9 +517,7 @@ export class LlWalkers { B.startBlock(lKey); this.puts(B, "%b", jsonObjectKeyLabel(fieldName)); if (cyclic && traceAdapter(this.host, f.type) !== null) this.jbEdgeProp(B, fieldName); - B.line( - `call void @${this.jsonWriteHelper(f.type)}(ptr %b, ${this.valTy(f.type)} ${v}) ; ${llvmCommentText(fieldName)}`, - ); + this.writeField(B, v, f.type, `${llvmCommentText(fieldName)}`); B.terminate("ret i1 true"); }); B.startBlock(lSkip); @@ -539,14 +534,15 @@ export class LlWalkers { /** Whether the walker for `t` can leave an exception pending: cyclic * values, records serialized through a kept checked source, and - * checked-dynamic values (toJSON methods, getters, BigInts). */ + * checked-dynamic values (toJSON methods, getters, BigInts), including + * the foreign elements of arrays that can hold them. */ jsonWriteMayThrow(t: IrType, seen = new Set()): boolean { if (traceAdapter(this.host, t) !== null) return true; switch (t.kind) { case "dyn": return true; case "array": - return this.jsonWriteMayThrow(t.elem, seen); + return this.host.mayHoldForeign(t.elem) || this.jsonWriteMayThrow(t.elem, seen); case "union": return (this.host.unionsById.get(t.unionId)?.arms ?? []).some((arm) => this.jsonWriteMayThrow(arm, seen), @@ -678,6 +674,47 @@ export class LlWalkers { B.line(`call void @scr_arr_release(ptr ${ks})`); } + /** Serializing a value of `t` can call user code that changes the + * arrays or records holding it: it reaches an array that can hold + * foreign elements (their toJSON and getters), or a value whose own + * serialization may (a class instance, a checked-dynamic value), in a + * program that has foreign elements at all. */ + private serializationMayCallUserCode(t: IrType, seen = new Set()): boolean { + const host = this.host; + if (!host.mayHoldForeign({ kind: "f64" }) && !host.mayHoldForeign({ kind: "string" })) + return false; + const key = typeKey(t); + if (seen.has(key)) return false; + seen.add(key); + switch (t.kind) { + case "dyn": + case "object": + return true; + case "array": + return host.mayHoldForeign(t.elem) || this.serializationMayCallUserCode(t.elem, seen); + case "record": + return (host.recordsById.get(t.shapeId)?.fields ?? []).some((field) => + this.serializationMayCallUserCode(field.type, seen), + ); + case "union": + return (host.unionsById.get(t.unionId)?.arms ?? []).some((arm) => + this.serializationMayCallUserCode(arm, seen), + ); + default: + return false; + } + } + + /** Write a field value loaded borrowed from its record. When its + * serialization can call user code that replaces the field, the value + * is held across the write. */ + private writeField(B: BlockBuilder, v: string, t: IrType, comment: string): void { + const held = isRefCounted(t) && this.serializationMayCallUserCode(t); + if (held) B.line(`call ptr ${retainSym(this.host, t)}(ptr ${v})`); + B.line(`call void @${this.jsonWriteHelper(t)}(ptr %b, ${this.valTy(t)} ${v}) ; ${comment}`); + if (held) B.line(`call void ${releaseSym(this.host, t)}(ptr ${v})`); + } + private emitArrayWriter(B: BlockBuilder, elem: IrType, cyclic: boolean): void { const host = this.host; const w = this.jsonWriteHelper(elem); @@ -687,7 +724,12 @@ export class LlWalkers { if (cyclic) this.jbEnter(B, true); this.putc(B, "%b", "91"); // '[' const finish = B.newLabel("jwa.end"); - if (elem.kind !== "bool") this.emitDenseArrayElements(B, elem, w, cyclic, finish); + // The dense walk borrows the storage and each element. When an + // element's serialization can call user code, which can grow or + // shrink the array or replace the element, the general walk below + // reads each index afresh and holds its element instead. + if (elem.kind !== "bool" && !this.serializationMayCallUserCode(elem)) + this.emitDenseArrayElements(B, elem, w, cyclic, finish); const len = B.tmp(); B.line(`${len} = call double @scr_arr_len(ptr %v)`); B.countedLoop(len, (i) => { @@ -710,6 +752,25 @@ export class LlWalkers { B.line(`${present} = fcmp oeq double ${state}, ${f64Lit(1)}`); B.condBr(present, valueBlock, missingBlock); B.startBlock(missingBlock); + if (host.mayHoldForeign(elem)) { + // A foreign element (SCR_ARR_FOREIGN) serializes its own value, + // after its toJSON runs with the index as its key. + const foreign = B.tmp(); + const foreignBlock = B.newLabel("jwa.foreign"); + const nullBlock = B.newLabel("jwa.null"); + B.line(`${foreign} = fcmp oeq double ${state}, ${f64Lit(3)}`); + B.condBr(foreign, foreignBlock, nullBlock); + B.startBlock(foreignBlock); + host.declare(`declare ptr @scr_arr_get_foreign(ptr, double)`); + host.declare(`declare void @scr_dyn_release(ptr)`); + const payload = B.tmp(); + B.line(`${payload} = call ptr @scr_arr_get_foreign(ptr %v, double ${i}) ; +1`); + host.declare(`declare void @scr_jb_put_dyn_element(ptr, ptr, double)`); + B.line(`call void @scr_jb_put_dyn_element(ptr %b, ptr ${payload}, double ${i})`); + B.line(`call void @scr_dyn_release(ptr ${payload})`); + B.br(doneBlock); + B.startBlock(nullBlock); + } host.declare(`declare void @scr_jb_puts(ptr, ptr)`); B.line(`call void @scr_jb_puts(ptr %b, ptr ${host.cstr("null")})`); B.br(doneBlock); @@ -911,7 +972,28 @@ export class LlWalkers { const lvalue = B.newLabel("uj.p"); B.line(`${state} = call double @scr_arr_state(ptr %a, double ${i})`); B.line(`${isValue} = fcmp oeq double ${state}, ${f64Lit(1)}`); - B.condBr(isValue, lvalue, next); + if (host.mayHoldForeign({ kind: "union", unionId })) { + // A foreign element (SCR_ARR_FOREIGN) joins as its ToString, like + // every element but undefined and null. + const notValue = B.newLabel("uj.f"); + const foreignLabel = B.newLabel("uj.foreign"); + B.condBr(isValue, lvalue, notValue); + B.startBlock(notValue); + const foreign = B.tmp(); + B.line(`${foreign} = fcmp oeq double ${state}, ${f64Lit(3)}`); + B.condBr(foreign, foreignLabel, next); + B.startBlock(foreignLabel); + host.declare(`declare ptr @scr_arr_get_foreign(ptr, double)`); + host.declare(`declare void @scr_dyn_release(ptr)`); + const payload = B.tmp(); + B.line(`${payload} = call ptr @scr_arr_get_foreign(ptr %a, double ${i}) ; +1`); + host.declare(`declare void @scr_jb_join_dyn(ptr, ptr)`); + B.line(`call void @scr_jb_join_dyn(ptr ${buf}, ptr ${payload})`); + B.line(`call void @scr_dyn_release(ptr ${payload})`); + B.br(next); + } else { + B.condBr(isValue, lvalue, next); + } B.startBlock(lvalue); const u = B.tmp(); B.line(`${u} = call ptr @scr_arr_peek_ref(ptr %a, double ${i}) ; borrowed scalar union`); diff --git a/packages/compiler/src/backend/runtime-abi.ts b/packages/compiler/src/backend/runtime-abi.ts index 7ac0877eb8..fe13879a26 100644 --- a/packages/compiler/src/backend/runtime-abi.ts +++ b/packages/compiler/src/backend/runtime-abi.ts @@ -2,8 +2,8 @@ * of the link contract: a program object keeps this symbol undefined and the * matching runtime defines it, so incompatible manual links fail before the * program can start. */ -export const RUNTIME_ABI_VERSION = 44 as const; -export const RUNTIME_ABI_MARKER = "scr_runtime_abi_v44" as const; +export const RUNTIME_ABI_VERSION = 45 as const; +export const RUNTIME_ABI_MARKER = "scr_runtime_abi_v45" as const; /** Object consumption is useful today, but the complete scr_* surface may * still change before 1.0. The versioned marker prevents accidental mixing; diff --git a/packages/compiler/src/backend/runtime-pack-targets.test.ts b/packages/compiler/src/backend/runtime-pack-targets.test.ts index 650e7c5431..492b71217f 100644 --- a/packages/compiler/src/backend/runtime-pack-targets.test.ts +++ b/packages/compiler/src/backend/runtime-pack-targets.test.ts @@ -52,7 +52,7 @@ async function fixture(target: NativeTargetSpec, workers = false) { object_format: target.objectFormat, minimum_os: target.minimumOs, }, - runtime_abi: { version: 44, marker: "scr_runtime_abi_v44" }, + runtime_abi: { version: 45, marker: "scr_runtime_abi_v45" }, compiler: { command: "fixture", identity: "fixture", target: target.llvmTriple }, macros: { executable: [], excluded: ["SCR_LIB"], sanitizer: "external-toolchain-required" }, flavors: { diff --git a/packages/compiler/src/backend/runtime-pack.test.ts b/packages/compiler/src/backend/runtime-pack.test.ts index 443c22d9e9..fdb0ba29e7 100644 --- a/packages/compiler/src/backend/runtime-pack.test.ts +++ b/packages/compiler/src/backend/runtime-pack.test.ts @@ -129,7 +129,7 @@ async function fixture(target: NativeTargetSpec = MACOS_ARM64_TARGET) { object_format: target.objectFormat, minimum_os: target.minimumOs, }, - runtime_abi: { version: 44, marker: "scr_runtime_abi_v44" }, + runtime_abi: { version: 45, marker: "scr_runtime_abi_v45" }, compiler: { command: "clang", identity: "fixture clang", diff --git a/packages/compiler/src/frontend/lowering/array-values.ts b/packages/compiler/src/frontend/lowering/array-values.ts index 6e4a45f865..ed52b8da5c 100644 --- a/packages/compiler/src/frontend/lowering/array-values.ts +++ b/packages/compiler/src/frontend/lowering/array-values.ts @@ -58,7 +58,33 @@ export function arrayNumberRead(arr: IrExpr, index: IrExpr, loc: SrcLoc): IrExpr }; } -/** Read one stabilized receiver/index pair. Both a hole and present undefined yield undefined. */ +/** A foreign element read into a typed result: it never fits, so the read + * throws the checked conversion's TypeError (array.foreignThrow). */ +export function foreignElementThrow( + lowerer: Lowerer, + arr: IrExpr, + index: IrExpr, + type: IrType, + loc: SrcLoc, +): IrExpr { + return { + kind: "libCall", + fn: "array.foreignThrow", + args: [arr, index, { kind: "strLit", value: lowerer.fmt(type), type: STRING, loc }], + type, + loc, + }; +} + +/** Arrays of this element type can hold elements outside it, stored + * through dynamic views (Lowerer.foreignElementKeys). */ +export function elementMayBeForeign(lowerer: Lowerer, elem: IrType): boolean { + const keys = lowerer.foreignElementKeys; + return keys.size > 0 && elem.kind !== "dyn" && (keys.has("*") || keys.has(typeKey(elem))); +} + +/** Read one stabilized receiver/index pair. Both a hole and present undefined yield undefined; + * a foreign element (state 3) takes the checked conversion into the typed result. */ export function arrayValueRead( lowerer: Lowerer, arr: IrExpr, @@ -68,8 +94,25 @@ export function arrayValueRead( ): IrExpr { const type = arrayValueType(lowerer, elem); const read: IrExpr = { kind: "arrayGet", arr, index, type: elem, loc }; - const missing: IrExpr = + let missing: IrExpr = type.kind === "dyn" ? dynUndefinedExpr(loc) : lowerer.wrappedUndefined(type, loc)!; + if (elementMayBeForeign(lowerer, elem)) { + missing = { + kind: "ternary", + cond: { + kind: "bin", + op: "===", + left: { kind: "arrayState", arr, index, type: F64, loc }, + right: { kind: "numLit", value: 3, type: F64, loc }, + type: BOOL, + loc, + }, + then: foreignElementThrow(lowerer, arr, index, type, loc), + else_: missing, + type, + loc, + }; + } return { kind: "ternary", cond: { @@ -684,6 +727,9 @@ export function tryLowerNumericIndexRead( operand.callee !== lowerer.arrHofHelpers.get(`idxOr:${typeKey(F64)}`) ) return null; + // ToNumber of a foreign element differs from its own value under `+`, + // comparisons and ToBoolean. + if (elementMayBeForeign(lowerer, F64)) return null; const [arr, index] = operand.args; if ( operand.args.length !== 2 || @@ -777,6 +823,8 @@ export function tryFuseIndexRead( if (arr?.type.kind !== "array" || index?.type.kind !== "f64") return null; const elem = arr.type.elem; if (elem.kind === "union" || elem.kind === "void") return null; + // A foreign element is neither the present value nor the missing one. + if (elementMayBeForeign(lowerer, elem)) return null; if (operand.callee !== lowerer.arrHofHelpers.get(`idxOr:${typeKey(elem)}`)) return null; if (arr.kind !== "varRef" || (index.kind !== "varRef" && index.kind !== "numLit")) return null; const loc = operand.loc; diff --git a/packages/compiler/src/frontend/lowering/containers/array-callbacks.ts b/packages/compiler/src/frontend/lowering/containers/array-callbacks.ts index 46459fadb1..2c4d5d9b41 100644 --- a/packages/compiler/src/frontend/lowering/containers/array-callbacks.ts +++ b/packages/compiler/src/frontend/lowering/containers/array-callbacks.ts @@ -42,6 +42,8 @@ import { arrayValueStore, arrayValueType, currentArrayIndexPresent, + elementMayBeForeign, + foreignElementThrow, primitiveElement, } from "../array-values.js"; import { typeKey } from "../../type-mapper.js"; @@ -2531,6 +2533,9 @@ export function lowerArraySortCall( loc, native, keyed && { order, key: reads !== null, keys: keyed.length }, + elementMayBeForeign(lowerer, elem) + ? (arr, index) => foreignElementThrow(lowerer, arr, index, elem, loc) + : null, ), ); } diff --git a/packages/compiler/src/frontend/lowering/containers/array-concat.ts b/packages/compiler/src/frontend/lowering/containers/array-concat.ts index a7ce0a1064..440cbe9e24 100644 --- a/packages/compiler/src/frontend/lowering/containers/array-concat.ts +++ b/packages/compiler/src/frontend/lowering/containers/array-concat.ts @@ -1,5 +1,6 @@ import { keyedHelperName } from "../stable-names.js"; import { identityPreservingWidening } from "../coercions/identity.js"; +import { elementMayBeForeign, foreignElementThrow } from "../array-values.js"; import { countedFor, numLit, varRef } from "../../../ir/build.js"; import { BOOL, @@ -131,7 +132,33 @@ export function arrayConcatHelper( loc, }, then: [{ kind: "arraySetUndefined", arr: out, index: destination, loc }], - else_: null, + else_: elementMayBeForeign(lowerer, sourceElem) + ? [ + { + // A foreign element takes the checked conversion's TypeError. + kind: "if", + cond: { + kind: "bin", + op: "===", + left: { kind: "arrayState", arr: source, index, type: F64, loc }, + right: numLit(3, loc), + type: BOOL, + loc, + }, + then: [ + { + kind: "arraySet", + arr: out, + index: destination, + value: foreignElementThrow(lowerer, source, index, elem, loc), + loc, + }, + ], + else_: null, + loc, + }, + ] + : null, loc, }, ], diff --git a/packages/compiler/src/frontend/lowering/containers/array-construction.ts b/packages/compiler/src/frontend/lowering/containers/array-construction.ts index 5029abc36a..702837e40b 100644 --- a/packages/compiler/src/frontend/lowering/containers/array-construction.ts +++ b/packages/compiler/src/frontend/lowering/containers/array-construction.ts @@ -1081,9 +1081,7 @@ export function lowerArrayFromCall( if ((args.length === 1 || args.length === 2) && !ts.isObjectLiteralExpression(args[0]!)) { const source = lowerer.lowerExpr(args[0]!); let src = - source.type.kind === "string" && args.length === 2 - ? strCharsCall(lowerer, source, loc) - : source; + source.type.kind === "string" && args.length === 2 ? strCharsCall(source, loc) : source; if (src.type.kind === "array") { const arrT = src.type; if (args.length === 1) { @@ -1171,7 +1169,7 @@ export function lowerArrayFromCall( } // String iteration advances by code point; the same helper serves // `[...s]` and both Array.from(s) forms. - if (src.type.kind === "string" && args.length === 1) return strCharsCall(lowerer, src, loc); + if (src.type.kind === "string" && args.length === 1) return strCharsCall(src, loc); if (src.type.kind === "dyn" && args.length === 1) { return lowerCheckedArrayFrom(lowerer, src, loc); } @@ -1318,112 +1316,9 @@ function buildArrayFromArrayFn( } /** `Array.from(s)` / `[...s]` on a STRING: the code-point split into a - * fresh string[], through one interned helper per module. */ -export function strCharsCall(lowerer: Lowerer, src: IrExpr, loc: SrcLoc): IrExpr { - const key = "strChars"; - let helper = lowerer.arrHofHelpers.get(key); - if (!helper) { - helper = "%str.chars"; - lowerer.arrHofHelpers.set(key, helper); - lowerer.liftedFns.push(buildStrCharsFn(helper, loc)); - } - return { kind: "call", callee: helper, args: [src], type: arrayOf(STRING), loc }; -} - -/** The code-point split, from existing IR nodes — the string for-of - * desugar's UTF-16 cursor as a function: - * - * out = []; - * i = 0; - * while (i < s.length) { ch = cpAt(s, i); i += ch.length; out.push(ch); } - * return out; - */ -function buildStrCharsFn(name: string, loc: SrcLoc): IrFunction { - const outT = arrayOf(STRING); - - const sLen = (recv: IrExpr): IrExpr => ({ - kind: "strIntrinsic", - method: "length", - receiver: recv, - args: [], - type: F64, - loc, - }); - const body: IrStmt[] = [ - { - kind: "varDecl", - localId: "out.0", - init: { kind: "arrayLit", elems: [], type: outT, loc }, - loc, - }, - { kind: "varDecl", localId: "i.0", init: { kind: "numLit", value: 0, type: F64, loc }, loc }, - { - kind: "while", - cond: { - kind: "bin", - op: "<", - left: varRef("i.0", F64, loc), - right: sLen(varRef("s.0", STRING, loc)), - type: BOOL, - loc, - }, - body: [ - { - kind: "varDecl", - localId: "ch.0", - init: { - kind: "strIntrinsic", - method: "cpAt", - receiver: varRef("s.0", STRING, loc), - args: [varRef("i.0", F64, loc)], - type: STRING, - loc, - }, - loc, - }, - { - kind: "assign", - localId: "i.0", - value: { - kind: "bin", - op: "+", - left: varRef("i.0", F64, loc), - right: sLen(varRef("ch.0", STRING, loc)), - type: F64, - loc, - }, - loc, - }, - { - kind: "exprStmt", - expr: { - kind: "arrIntrinsic", - method: "push", - receiver: varRef("out.0", outT, loc), - args: [varRef("ch.0", STRING, loc)], - type: F64, - loc, - }, - loc, - }, - ], - loc, - }, - { kind: "return", value: varRef("out.0", outT, loc), loc }, - ]; - return { - name, - params: [{ localId: "s.0", name: "s", type: STRING }], - returnType: outT, - locals: [ - { id: "s.0", name: "s", type: STRING, mutable: true }, - { id: "out.0", name: "out", type: outT, mutable: false }, - { id: "i.0", name: "i", type: F64, mutable: true }, - { id: "ch.0", name: "ch", type: STRING, mutable: false }, - ], - body, - loc, - }; + * fresh string[] (scr_str_code_points). */ +export function strCharsCall(src: IrExpr, loc: SrcLoc): IrExpr { + return { kind: "libCall", fn: "str.codePoints", args: [src], type: arrayOf(STRING), loc }; } /** The lowered `length` value of the one-property source literal, or null diff --git a/packages/compiler/src/frontend/lowering/containers/array-methods.ts b/packages/compiler/src/frontend/lowering/containers/array-methods.ts index f406e57ebc..3b16e714f2 100644 --- a/packages/compiler/src/frontend/lowering/containers/array-methods.ts +++ b/packages/compiler/src/frontend/lowering/containers/array-methods.ts @@ -1886,17 +1886,27 @@ function arraySearchHelper( }, loc, }; - const visitPresent: IrStmt[] = [ - { kind: "varDecl", localId: "value.0", init: value, loc }, - { - kind: "if", - cond: match, - then: [{ kind: "return", value: method === "includes" ? boolLit(true, loc) : indexRef, loc }], - else_: null, - loc, - }, - advance, - ]; + const found: IrStmt = { + kind: "if", + cond: directNeedle ? equal : match, + then: [{ kind: "return", value: method === "includes" ? boolLit(true, loc) : indexRef, loc }], + else_: null, + loc, + }; + // A direct needle reads only value slots: an undefined or foreign + // element written through a dynamic view has no element-typed payload. + const visitPresent: IrStmt[] = directNeedle + ? [ + { + kind: "if", + cond: { kind: "bin", op: "===", left: stateRef, right: numLit(1, loc), type: BOOL, loc }, + then: [{ kind: "varDecl", localId: "value.0", init: value, loc }, found], + else_: null, + loc, + }, + advance, + ] + : [{ kind: "varDecl", localId: "value.0", init: value, loc }, found, advance]; const skipHole: IrStmt[] = method === "lastIndexOf" ? [advance] diff --git a/packages/compiler/src/frontend/lowering/containers/array-pipelines.ts b/packages/compiler/src/frontend/lowering/containers/array-pipelines.ts index 197af2a9ed..6307d871f4 100644 --- a/packages/compiler/src/frontend/lowering/containers/array-pipelines.ts +++ b/packages/compiler/src/frontend/lowering/containers/array-pipelines.ts @@ -34,6 +34,7 @@ import { arrayValueRead, arrayValueStore, arrayValueType, + elementMayBeForeign, } from "../array-values.js"; import type { Lowerer } from "../lowerer.js"; @@ -156,7 +157,8 @@ const PURE_STRING = new Set([ "toUpperCase", "isWellFormed", "toWellFormed", - "cpAt", + "byteLength", + "cpAtByte", ]); function primitive(t: IrType): boolean { @@ -1643,7 +1645,9 @@ class FusedLoop { ]; } const valueT = arrayValueType(this.lowerer, arrT.elem); - if (stage.fast) { + // A foreign element is neither the plain element nor undefined: the + // ordinary read below converts it. + if (stage.fast && !elementMayBeForeign(this.lowerer, arrT.elem)) { // The split visit: a value goes on as the plain element, a present // undefined as the undefined arm. const missing = this.lowerer.wrappedUndefined(valueT, loc); diff --git a/packages/compiler/src/frontend/lowering/containers/collection-methods.ts b/packages/compiler/src/frontend/lowering/containers/collection-methods.ts index f18291a635..bfaca6f96b 100644 --- a/packages/compiler/src/frontend/lowering/containers/collection-methods.ts +++ b/packages/compiler/src/frontend/lowering/containers/collection-methods.ts @@ -1135,7 +1135,7 @@ function setFromSeedValue( if (source.type.kind === "set" && typeEquals(source.type, setT)) { return { kind: "setIntrinsic", method: "clone", receiver: source, args: [], type: setT, loc }; } else if (source.type.kind === "string" && setT.elem.kind === "string") { - seed = strCharsCall(lowerer, source, loc); + seed = strCharsCall(source, loc); } else if (source.type.kind === "array" && typeEquals(source.type.elem, setT.elem)) { seed = source; } else if ( diff --git a/packages/compiler/src/frontend/lowering/expressions/array-literals.ts b/packages/compiler/src/frontend/lowering/expressions/array-literals.ts index 903f0884d5..77c16594f2 100644 --- a/packages/compiler/src/frontend/lowering/expressions/array-literals.ts +++ b/packages/compiler/src/frontend/lowering/expressions/array-literals.ts @@ -504,7 +504,7 @@ export function lowerArrayLiteral( // interned helper as Array.from(s); the result rides the array // machinery below like any string[] source. if (src.type.kind === "string") { - src = strCharsCall(lowerer, src, locOf(el)); + src = strCharsCall(src, locOf(el)); } // `[...typedArray]`: represented typed arrays are dense numeric // iterables, so drain their elements into the fresh number[] that diff --git a/packages/compiler/src/frontend/lowering/expressions/element-access.ts b/packages/compiler/src/frontend/lowering/expressions/element-access.ts index a2f2ebdb62..59b6574e67 100644 --- a/packages/compiler/src/frontend/lowering/expressions/element-access.ts +++ b/packages/compiler/src/frontend/lowering/expressions/element-access.ts @@ -59,6 +59,7 @@ import { arrayValueRead, arrayValueStore, arrayValueType, + elementMayBeForeign, lowerSafeBytesRead, lowerSafeIndexRead, primitiveElement, @@ -1623,11 +1624,14 @@ export function lowerElementCompoundParts( const indexRef = (): IrExpr => varRef(indexLocal.id, F64, loc); const oldValue: IrExpr = receiver.type.kind === "array" - ? receiver.type.elem.kind === "f64" + ? receiver.type.elem.kind === "f64" && + !(op === "+" && elementMayBeForeign(lowerer, receiver.type.elem)) ? // Every numeric operator applies ToNumber to the old value, so a // missing element is NaN without materializing `number | undefined`. arrayNumberRead(receiverRef(), indexRef(), locOf(target)) - : receiver.type.elem.kind === "string" && op === "+" + : receiver.type.elem.kind === "string" && + op === "+" && + !elementMayBeForeign(lowerer, receiver.type.elem) ? arrayStringOperand(receiverRef(), indexRef(), locOf(target)) : arrayValueRead(lowerer, receiverRef(), indexRef(), receiver.type.elem, locOf(target)) : { diff --git a/packages/compiler/src/frontend/lowering/expressions/nullish-assignment.ts b/packages/compiler/src/frontend/lowering/expressions/nullish-assignment.ts index 9664f591d6..8a25eb1240 100644 --- a/packages/compiler/src/frontend/lowering/expressions/nullish-assignment.ts +++ b/packages/compiler/src/frontend/lowering/expressions/nullish-assignment.ts @@ -14,7 +14,12 @@ import { import { varRef } from "../../../ir/build.js"; import { locOf } from "../../program.js"; import type { Lowerer } from "../lowerer.js"; -import { arrayNumberRead, arrayValueRead, arrayValueStore } from "../array-values.js"; +import { + arrayNumberRead, + arrayValueRead, + arrayValueStore, + elementMayBeForeign, +} from "../array-values.js"; import { fenceNodeModuleMutation } from "../lower-node-module.js"; import { globalSymbolKey } from "./global-symbols.js"; import { tryLowerExpression } from "./try-lower-expression.js"; @@ -185,7 +190,12 @@ export function lowerShortCircuitAssignment(lowerer: Lowerer, expr: ts.BinaryExp const element = storage; write = (value) => arrayValueStore(lowerer, receiver, key, value, element, loc); const elementLoc = locOf(target); - if (nullish && NON_NULLISH_ELEMENTS.has(element.kind)) { + // A foreign element can be any value, nullish or not. + if ( + nullish && + NON_NULLISH_ELEMENTS.has(element.kind) && + !elementMayBeForeign(lowerer, element) + ) { // Only a missing element is nullish: keep the present value or // store the right side, without the optional element value. rhs = lowerer.lowerExprExpecting(expr.right, element); diff --git a/packages/compiler/src/frontend/lowering/generator-delegation.ts b/packages/compiler/src/frontend/lowering/generator-delegation.ts index a606c177e7..81ecc83535 100644 --- a/packages/compiler/src/frontend/lowering/generator-delegation.ts +++ b/packages/compiler/src/frontend/lowering/generator-delegation.ts @@ -577,10 +577,11 @@ export function lowerBuiltinDelegation( loc, }); } else if (sourceT.kind === "string") { + // A byte cursor over the WTF-8 data, as for-of over a string walks. const ch = lowerer.declareHiddenLocal("%delegateElement", STRING); const length = (receiver: IrExpr): IrExpr => ({ kind: "strIntrinsic", - method: "length", + method: "byteLength", receiver, args: [], type: F64, @@ -595,7 +596,7 @@ export function lowerBuiltinDelegation( localId: ch.id, init: { kind: "strIntrinsic", - method: "cpAt", + method: "cpAtByte", receiver: src(), args: [at()], type: STRING, diff --git a/packages/compiler/src/frontend/lowering/iterator-adapters.ts b/packages/compiler/src/frontend/lowering/iterator-adapters.ts index 4909e7f453..78fcff84d5 100644 --- a/packages/compiler/src/frontend/lowering/iterator-adapters.ts +++ b/packages/compiler/src/frontend/lowering/iterator-adapters.ts @@ -743,7 +743,7 @@ export function lowerStringIteratorValue( left: index, right: { kind: "strIntrinsic", - method: "length", + method: "byteLength", receiver: text, args: [], type: F64, @@ -758,7 +758,7 @@ export function lowerStringIteratorValue( localId: "point.0", init: { kind: "strIntrinsic", - method: "cpAt", + method: "cpAtByte", receiver: text, args: [index], type: STRING, @@ -775,7 +775,7 @@ export function lowerStringIteratorValue( left: index, right: { kind: "strIntrinsic", - method: "length", + method: "byteLength", receiver: point, args: [], type: F64, diff --git a/packages/compiler/src/frontend/lowering/lower-array-sort.ts b/packages/compiler/src/frontend/lowering/lower-array-sort.ts index 5521d30ef5..62507ddcd1 100644 --- a/packages/compiler/src/frontend/lowering/lower-array-sort.ts +++ b/packages/compiler/src/frontend/lowering/lower-array-sort.ts @@ -926,6 +926,10 @@ export function buildArraySortFn( loc: SrcLoc, native = false, keyed: { order: number; key: boolean; keys: number } | null = null, + /** The throw for a foreign element (SCR_ARR_FOREIGN) at `index` of + * `arr`, which the comparator's element type cannot hold; for element + * types whose arrays can hold one. */ + checkForeign: ((arr: IrExpr, index: IrExpr) => IrExpr) | null = null, ): IrFunction { const b = new SortIr(loc); const arrT = arrayOf(elem), @@ -1035,7 +1039,21 @@ export function buildArraySortFn( b.branch( b.eq(state, b.num(1)), [compactValue()], - [b.branch(b.eq(state, b.num(2)), [b.step(undefinedCount)])], + [ + b.branch( + b.eq(state, b.num(2)), + [b.step(undefinedCount)], + checkForeign + ? [ + // A foreign element does not fit the comparator's element + // type: the checked conversion's TypeError. + b.branch(b.eq(state, b.num(3)), [ + { kind: "exprStmt", expr: checkForeign(src, index), loc }, + ]), + ] + : null, + ), + ], ), ]), // Sort only the compacted prefix. The tail is overwritten with undefined diff --git a/packages/compiler/src/frontend/lowering/lower-inspect.ts b/packages/compiler/src/frontend/lowering/lower-inspect.ts index f1ccaebc8d..1ff19782ce 100644 --- a/packages/compiler/src/frontend/lowering/lower-inspect.ts +++ b/packages/compiler/src/frontend/lowering/lower-inspect.ts @@ -1,5 +1,6 @@ import { keyedHelperName } from "./stable-names.js"; import { perKeyOrder } from "./key-order-witness.js"; +import { elementMayBeForeign } from "./array-values.js"; import { classPrototypeView } from "./class-prototypes.js"; import { dynUndefinedExpr, @@ -1421,11 +1422,29 @@ function inspectHelper(lowerer: Lowerer, t: IrType, loc: SrcLoc): string { const i = (): IrExpr => varRef("i.0", F64, loc); const next = (): IrExpr => varRef("next.0", F64, loc); const hasMore = (): IrExpr => lt(i(), n()); + // A foreign element (state 3) holds a value outside the element type. + const foreignElements = elementMayBeForeign(lowerer, t.elem); + const foreign = (): IrExpr => ({ + kind: "libCall", + fn: "array.foreign", + args: [v(), i()], + type: DYN, + loc, + }); const isNumber = (): IrExpr => ({ kind: "ternary", cond: eq(state(i()), numLit(1, loc)), then: isNumberFlag(lowerer, t.elem, () => at(i()), loc), - else_: boolLit(false, loc), + else_: foreignElements + ? { + kind: "ternary", + cond: eq(state(i()), numLit(3, loc)), + then: isNumberFlag(lowerer, DYN, foreign, loc), + else_: boolLit(false, loc), + type: BOOL, + loc, + } + : boolLit(false, loc), type: BOOL, loc, }); @@ -1504,7 +1523,16 @@ function inspectHelper(lowerer: Lowerer, t: IrType, loc: SrcLoc): string { kind: "ternary", cond: eq(state(i()), numLit(1, loc)), then: child(t.elem, at(i())), - else_: strLit("undefined", loc), + else_: foreignElements + ? { + kind: "ternary", + cond: eq(state(i()), numLit(3, loc)), + then: child(DYN, foreign()), + else_: strLit("undefined", loc), + type: STRING, + loc, + } + : strLit("undefined", loc), type: STRING, loc, }, diff --git a/packages/compiler/src/frontend/lowering/lower-server.ts b/packages/compiler/src/frontend/lowering/lower-server.ts index 7359b56a8d..614d6a2067 100644 --- a/packages/compiler/src/frontend/lowering/lower-server.ts +++ b/packages/compiler/src/frontend/lowering/lower-server.ts @@ -2425,7 +2425,9 @@ function lowerNetSocketMethodCall( }; } if (name === "write" || name === "end") { - requireStatementPosition(lowerer, call, `socket.${name}(...)`); + // write() answers Node's backpressure boolean; end() stays a statement + if (name === "end") requireStatementPosition(lowerer, call, `socket.${name}(...)`); + const resultT = name === "write" ? BOOL : VOID; // write(chunk, encoding) — the two-argument encoding form: 'buffer' // beside a string chunk is Writable's encoding check (a synchronous // ERR_UNKNOWN_ENCODING TypeError, "Unknown encoding: buffer" — the @@ -2451,7 +2453,7 @@ function lowerNetSocketMethodCall( const socket = coerceToHandle(lowerer, access.expression, NETSOCKET_T); const chunk = lowerer.lowerExpr(args[0]!); const fn: IrLibFn = chunk.type.kind === "string" ? "net.sockWrite" : "net.sockWriteBytes"; - return { kind: "libCall", fn, args: [socket, chunk], type: VOID, loc }; + return { kind: "libCall", fn, args: [socket, chunk], type: BOOL, loc }; } } } @@ -2510,7 +2512,7 @@ function lowerNetSocketMethodCall( NARROW_DATA_HINT, ); } - main = { kind: "libCall", fn, args: [receiver(), data], type: VOID, loc }; + main = { kind: "libCall", fn, args: [receiver(), data], type: resultT, loc }; } const cb = lowerer.lowerExpr(last); let register: IrExpr; @@ -2531,8 +2533,24 @@ function lowerNetSocketMethodCall( loc, }; } - stmts.push({ kind: "exprStmt", expr: main, loc }); - return { kind: "seqExpr", stmts, result: register, type: VOID, loc }; + if (resultT.kind === "void") { + stmts.push({ kind: "exprStmt", expr: main, loc }); + return { kind: "seqExpr", stmts, result: register, type: VOID, loc }; + } + // write(data, cb): the data is written, the callback registered, and + // the write's answer is the value + const okLocal = lowerer.declareHiddenLocal("%sockWriteOk", BOOL); + stmts.push( + { kind: "varDecl", localId: okLocal.id, init: main, loc }, + { kind: "exprStmt", expr: register, loc }, + ); + return { + kind: "seqExpr", + stmts, + result: { kind: "varRef", localId: okLocal.id, type: BOOL, loc }, + type: BOOL, + loc, + }; } const maxArgs = name === "write" ? 1 : 1; const minArgs = name === "write" ? 1 : 0; @@ -2552,17 +2570,17 @@ function lowerNetSocketMethodCall( const data = lowerer.lowerExpr(args[0]!); if (data.type.kind === "string") { const fn: IrLibFn = name === "write" ? "net.sockWrite" : "net.sockEndStr"; - return { kind: "libCall", fn, args: [receiver, data], type: VOID, loc }; + return { kind: "libCall", fn, args: [receiver, data], type: resultT, loc }; } if (data.type.kind === "bytes" && data.type.elem === "u8") { const fn: IrLibFn = name === "write" ? "net.sockWriteBytes" : "net.sockEndBytes"; - return { kind: "libCall", fn, args: [receiver, data], type: VOID, loc }; + return { kind: "libCall", fn, args: [receiver, data], type: resultT, loc }; } if (data.type.kind === "dyn") { // An untyped JS payload (a helper's parameter): the runtime // dispatches STR/BYTES and throws Node's chunk TypeError otherwise. const fn: IrLibFn = name === "write" ? "net.sockWriteDyn" : "net.sockEndDyn"; - return { kind: "libCall", fn, args: [receiver, data], type: VOID, loc }; + return { kind: "libCall", fn, args: [receiver, data], type: resultT, loc }; } lowerer.noLowering( `${name} of '${lowerer.fmt(data.type)}' data`, @@ -2944,6 +2962,7 @@ function lowerNetSocketMethodCall( event === "connect" || event === "ready" || event === "timeout" || + event === "drain" || event === "readable" ) { const { cb } = lowerCallbackArg( @@ -2962,11 +2981,13 @@ function lowerNetSocketMethodCall( ? "net.sockOnClose" : event === "timeout" ? "net.sockOnTimeout" - : event === "readable" - ? "net.sockOnReadable" - : event === "ready" - ? "net.sockOnReady" - : "net.sockOnConnect"; + : event === "drain" + ? "net.sockOnDrain" + : event === "readable" + ? "net.sockOnReadable" + : event === "ready" + ? "net.sockOnReady" + : "net.sockOnConnect"; return { kind: "libCall", fn, args: [receiver, cb, once], type: VOID, loc }; } if (event === "lookup") { @@ -3957,6 +3978,18 @@ export function lowerServerProperty( const receiver = coerceToHandle(lowerer, expr.expression, NETSOCKET_T); return { kind: "libCall", fn: "net.sockBytesWritten", args: [receiver], type: F64, loc }; } + if ( + recvKind === "netSocket" && + lowerer.isStdlibMember(expr) && + (expr.name.text === "writableLength" || expr.name.text === "writableNeedDrain") + ) { + // The write buffer's unsent bytes, and whether a write answered false + // and 'drain' has not fired yet. + const receiver = coerceToHandle(lowerer, expr.expression, NETSOCKET_T); + return expr.name.text === "writableLength" + ? { kind: "libCall", fn: "net.sockWritableLength", args: [receiver], type: F64, loc } + : { kind: "libCall", fn: "net.sockWritableNeedDrain", args: [receiver], type: BOOL, loc }; + } if (recvKind === "netSocket" && lowerer.isStdlibMember(expr) && expr.name.text === "readable") { // true until the read half is done (peer FIN / destroy). const receiver = coerceToHandle(lowerer, expr.expression, NETSOCKET_T); diff --git a/packages/compiler/src/frontend/lowering/lower-stmts.ts b/packages/compiler/src/frontend/lowering/lower-stmts.ts index d6d8c1cc7a..9586727b3b 100644 --- a/packages/compiler/src/frontend/lowering/lower-stmts.ts +++ b/packages/compiler/src/frontend/lowering/lower-stmts.ts @@ -2668,7 +2668,7 @@ export function lowerBindingPattern( lowerer.unsupported("SC1031", el, "rest elements over a union of array or tuple layouts"); const loc = locOf(el); let value = unionArrayValueRead(lowerer, srcRef(), numLit(i, loc), loc, (text) => - strCharsCall(lowerer, text, loc), + strCharsCall(text, loc), ); if (!value) lowerer.unsupported( @@ -2798,7 +2798,7 @@ export function lowerBindingPattern( } // STRING sources (`const [a, b] = str`): array destructuring walks // the STRING ITERATOR — code points, astral characters whole — - // which is Array.from(string)'s own machinery (%str.chars), so the + // which is Array.from(string)'s own machinery (str.codePoints), so the // source splits once into a hidden chars array and the pattern // lowers as an array pattern over string[]. Positions, holes, // defaults (with the bounds test), rest (the remaining code points @@ -2817,7 +2817,7 @@ export function lowerBindingPattern( out.push({ kind: "varDecl", localId: chars.id, - init: strCharsCall(lowerer, srcRef(), loc), + init: strCharsCall(srcRef(), loc), loc, }); lowerer.lowerBindingPattern( @@ -9826,7 +9826,7 @@ function lowerArrayAssignInto( let srcType = tmpRef().type; // A STRING source (`[a, b] = s`): the declaration path's rule in // assignment position — the string iterator's code-point split - // (%str.chars, astral characters whole) into a hidden chars array, + // (str.codePoints, astral characters whole) into a hidden chars array, // then the pattern proceeds as an array pattern over string[] // (defaults with the bounds test, rest packing the remaining code // points, elisions; positions past the last code point inherit the @@ -9839,7 +9839,7 @@ function lowerArrayAssignInto( out.push({ kind: "varDecl", localId: chars.id, - init: strCharsCall(lowerer, tmpRef(), loc), + init: strCharsCall(tmpRef(), loc), loc, }); tmpRef = () => ({ kind: "varRef", localId: chars.id, type: charsT, loc }); @@ -11383,16 +11383,18 @@ function lowerForOfMatchAll( /** `for (const ch of s)` over a STRING: JS's string iterator, which walks * code POINTS — each pass binds the whole character (a two-unit string * for astral chars, where charAt/slice would return one surrogate half). - * Desugars over the UTF-16 cursor machinery: a hidden unit - * index reads the character with the cpAt intrinsic and advances by ITS - * length before the body runs (so `continue` never skips the advance): + * Desugars over a byte cursor into the WTF-8 data: a hidden byte offset + * reads the character that starts there with the cpAtByte intrinsic and + * advances by ITS byte count before the body runs (so `continue` never + * skips the advance): * * { const %s = ; let %i = 0; - * while (%i < %s.length) { const ch = cpAt(%s, %i); %i += ch.length; } } + * while (%i < byteLength(%s)) { const ch = cpAtByte(%s, %i); %i += byteLength(ch); } } * - * Strings are immutable, so the length is stable and there is no + * Strings are immutable, so the byte count is stable and there is no * iterator state to unwind on throw; a lone surrogate is a one-unit - * character, so every step lands on a character start. */ + * character with its own three-byte encoding, so every step lands on a + * character start. */ /** for-of over a CLASS ITERABLE (classIteratorOf's shape) — the * iterator protocol as ordinary method calls: * @@ -12087,7 +12089,7 @@ function lowerForOfString( try { const s = lowerer.declareHiddenLocal("%strof", STRING); const i = lowerer.declareHiddenLocal("%iterof", F64); - i.mutable = true; // the unit index reassigns (hidden locals default const) + i.mutable = true; // the byte cursor reassigns (hidden locals default const) const sRef = (): IrExpr => ({ kind: "varRef", localId: s.id, type: STRING, loc }); const iRef = (): IrExpr => ({ kind: "varRef", localId: i.id, type: F64, loc }); // `for (var c of str)`: the character mechanics stay on a hidden @@ -12118,7 +12120,7 @@ function lowerForOfString( localId: ch.id, init: { kind: "strIntrinsic", - method: "cpAt", + method: "cpAtByte", receiver: sRef(), args: [iRef()], type: STRING, @@ -12145,7 +12147,7 @@ function lowerForOfString( left: iRef(), right: { kind: "strIntrinsic", - method: "length", + method: "byteLength", receiver: chRef, args: [], type: F64, @@ -12177,7 +12179,7 @@ function lowerForOfString( left: iRef(), right: { kind: "strIntrinsic", - method: "length", + method: "byteLength", receiver: sRef(), args: [], type: F64, diff --git a/packages/compiler/src/frontend/lowering/lowerer.ts b/packages/compiler/src/frontend/lowering/lowerer.ts index ee7477e431..2a04b33c3e 100644 --- a/packages/compiler/src/frontend/lowering/lowerer.ts +++ b/packages/compiler/src/frontend/lowering/lowerer.ts @@ -2399,6 +2399,11 @@ export class Lowerer { /** for-of loops over arrays that can hold neither holes nor undefined: * each pass reads the element payload directly. */ readonly presentElementLoops = new Set(); + /** Element keys (typeKey of the element type) of native arrays that can + * hold elements outside their element type, stored through dynamic views + * (SCR_ARR_FOREIGN); "*" for every array. Empty unless the program stores + * values through a dynamic value that can view a native array. */ + foreignElementKeys: ReadonlySet = new Set(); /** The fs/promises glob() call lowerForOf is lowering as a for await * head; other glob() calls have no lowering. */ fspGlobHead: ts.Node | null = null; @@ -6472,6 +6477,20 @@ export class Lowerer { let dynamicViews: Set | null = null; // Element ABIs of the native arrays that an `any` or `unknown` value is // converted to, including arrays nested in the converted value. + // Assignments, including the logical ones (`view ??= numbers`). + const assignmentOperators = new Set([ + ts.SyntaxKind.EqualsToken, + ts.SyntaxKind.QuestionQuestionEqualsToken, + ts.SyntaxKind.BarBarEqualsToken, + ts.SyntaxKind.AmpersandAmpersandEqualsToken, + ]); + // Initializers of class properties, parameter defaults and + // destructuring defaults, which convert to the binding's type. + const initializedBinding = ( + node: ts.Node, + ): node is ts.PropertyDeclaration | ts.ParameterDeclaration | ts.BindingElement => + (ts.isPropertyDeclaration(node) || ts.isParameter(node) || ts.isBindingElement(node)) && + node.initializer !== undefined; const collectDynamicViews = (): Set => { const keys = new Set(); const dynamicValue = (node: ts.Expression): boolean => { @@ -6522,14 +6541,18 @@ export class Lowerer { dynamicValue(node.initializer) ) target(typeOrNull(node.name)); + else if (initializedBinding(node) && dynamicValue(node.initializer!)) + target(typeOrNull(node.name ?? node)); else if ( ts.isBinaryExpression(node) && - node.operatorToken.kind === ts.SyntaxKind.EqualsToken && + assignmentOperators.has(node.operatorToken.kind) && dynamicValue(node.right) ) target(typeOrNull(node.left)); else if (ts.isReturnStatement(node) && node.expression && dynamicValue(node.expression)) target(contextualOrNull(node.expression)); + else if (ts.isYieldExpression(node) && node.expression && dynamicValue(node.expression)) + target(contextualOrNull(node.expression)); else if (ts.isCallExpression(node) || ts.isNewExpression(node)) { for (const arg of node.arguments?.nodes ?? []) if (!ts.isSpreadElement(arg) && dynamicValue(arg)) target(contextualOrNull(arg)); @@ -6558,6 +6581,237 @@ export class Lowerer { }); return keys; }; + // Element ABIs of the native arrays a value converted to a dynamic type + // can reach: the conversion keeps each such array as a live view, so + // stores through the dynamic value write the native storage. + // Standard library functions that only read the arguments they take as + // `any` or `unknown`: the console methods, String, Number, Boolean, + // Array.isArray, Object.is, JSON.stringify without a replacer function, + // and util's inspect and format. Any other function can keep an + // argument (Promise.resolve, Map.prototype.set, push onto an + // `unknown[]`) or pass it to a callback (a JSON.stringify replacer, an + // emitted event), which can write through it. + const readerGlobals = new Map | null>([ + ["console", null], + ["JSON", new Set(["stringify"])], + ["Array", new Set(["isArray"])], + ["Object", new Set(["is"])], + ]); + const readerFunctions = new Set(["String", "Number", "Boolean"]); + const utilReaders = new Set(["inspect", "format"]); + const standardLibraryReader = (node: ts.CallExpression | ts.NewExpression): boolean => + panicSafe(() => { + if (!ts.isCallExpression(node)) return false; + const callee = peel(node.expression); + if (ts.isIdentifier(callee)) + return [...readerFunctions].some((name) => this.isStdlibGlobal(callee, name)); + if (!ts.isPropertyAccessExpression(callee)) return false; + const member = callee.name.text; + for (const [global, members] of readerGlobals) { + if (!this.isStdlibGlobal(callee.expression, global)) continue; + if (members !== null && !members.has(member)) return false; + if (global !== "JSON") return true; + const replacer = node.arguments.nodes[1]; + return ( + replacer === undefined || + ts.isArrayLiteralExpression(peel(replacer)) || + peel(replacer).kind === ts.SyntaxKind.NullKeyword + ); + } + if (!utilReaders.has(member)) return false; + const declaration = this.checker.getResolvedSignature(node)?.declaration; + if (declaration === undefined) return false; + const file = ts.getSourceFileOfNode(declaration); + return this.isStdlibFile(file) && /[\\/]util\.d\.ts$/.test(file.fileName); + }, false); + const collectNativeViews = (): Set => { + const keys = new Set(); + const seen = new Set(); + const collect = (type: IrType | null): void => { + if (!type) return; + const id = typeKey(type); + if (seen.has(id)) return; + seen.add(id); + if (type.kind === "array") { + if (type.elem.kind !== "dyn") keys.add(typeKey(type.elem)); + collect(type.elem); + } else if (type.kind === "union") { + for (const arm of this.unions.get(type.unionId)?.arms ?? []) collect(arm); + } else if (type.kind === "record") { + for (const field of this.shapes.get(type.shapeId)?.fields ?? []) collect(field.type); + } else if (type.kind === "object") { + for (const field of this.classes.get(type.className)?.fields.values() ?? []) + collect(field); + } else if (type.kind === "map") { + collect(type.key); + collect(type.value); + } else if (type.kind === "set") collect(type.elem); + else if (type.kind === "promise") collect(type.inner); + }; + const holdsDynamic = (type: IrType | null, depth: number): boolean => { + if (!type || depth > 4) return false; + if (type.kind === "dyn") return true; + if (type.kind === "array" || type.kind === "set") return holdsDynamic(type.elem, depth + 1); + if (type.kind === "map") return holdsDynamic(type.value, depth + 1); + if (type.kind === "promise") return holdsDynamic(type.inner, depth + 1); + if (type.kind === "union") + return (this.unions.get(type.unionId)?.arms ?? []).some((arm) => + holdsDynamic(arm, depth + 1), + ); + if (type.kind === "record") + return (this.shapes.get(type.shapeId)?.fields ?? []).some((field) => + holdsDynamic(field.type, depth + 1), + ); + return false; + }; + const mapped = (type: ts.Type | null | undefined): IrType | null => + type ? panicSafe(() => this.mapTypeOf(type), null) : null; + const flow = (source: ts.Expression, target: ts.Type | null | undefined): void => { + if (target === undefined) return; + const into = target === null ? null : mapped(target); + // An unanswerable target may be dynamic. + if (target !== null && into !== null && !holdsDynamic(into, 0)) return; + const from = typeOrNull(source); + if (from === null) keys.add("*"); + else collect(mapped(from)); + }; + const visit = (node: ts.Node): void => { + if (ts.isAsExpression(node) || ts.isTypeAssertion(node)) + flow(node.expression, typeOrNull(node)); + else if (ts.isVariableDeclaration(node) && node.type && node.initializer) + flow(node.initializer, typeOrNull(node.name)); + else if (initializedBinding(node)) flow(node.initializer!, typeOrNull(node.name ?? node)); + else if (ts.isBinaryExpression(node) && assignmentOperators.has(node.operatorToken.kind)) + flow(node.right, typeOrNull(node.left)); + else if (ts.isReturnStatement(node) && node.expression) + flow(node.expression, contextualOrNull(node.expression)); + else if (ts.isYieldExpression(node) && node.expression) + flow(node.expression, contextualOrNull(node.expression)); + // A thrown value reaches a catch binding, which is dynamic. + else if (ts.isThrowStatement(node)) flow(node.expression, null); + else if (ts.isCallExpression(node) || ts.isNewExpression(node)) { + if (!standardLibraryReader(node)) + for (const arg of node.arguments?.nodes ?? []) + flow(ts.isSpreadElement(arg) ? arg.expression : arg, contextualOrNull(arg)); + } else if (ts.isPropertyAssignment(node)) + flow(node.initializer, contextualOrNull(node.initializer)); + else if (ts.isShorthandPropertyAssignment(node)) { + const name = node.name as ts.Expression; + flow(name, contextualOrNull(name)); + } else if (ts.isArrayLiteralExpression(node)) { + for (const element of node.elements.nodes) + if (!ts.isSpreadElement(element) && !ts.isOmittedExpression(element)) + flow(element, contextualOrNull(element)); + } else if (ts.isArrowFunction(node) && !ts.isBlock(node.body)) + flow(node.body as ts.Expression, contextualOrNull(node.body as ts.Expression)); + }; + for (const sf of sourceFiles) + ts.walkPreorder(sf, (node) => { + // A checker panic hides this conversion: any array. + if (!panicSafe(() => (visit(node), true), false)) keys.add("*"); + return undefined; + }); + return keys; + }; + // A receiver typed `any`, `unknown` or an array of either can be a live + // view of a native array. A receiver of another type (`Worker[]`, + // `Record`) is one only after a lying assertion, even + // when its values are dynamic. + const dynamicType = (type: ts.Type): boolean => { + if ((type.flags & (ts.TypeFlags.Any | ts.TypeFlags.Unknown)) !== 0) return true; + if (!this.checker.isArrayType(type) || this.checker.isTupleType(type)) return false; + const element = this.checker.getTypeArguments(type as ts.TypeReference)[0]; + return ( + element !== undefined && (element.flags & (ts.TypeFlags.Any | ts.TypeFlags.Unknown)) !== 0 + ); + }; + // A binding that only ever holds the array its declaration creates (a + // const variable, or a readonly property with an initializer) is never + // a view of a native array, even when its elements are dynamic. + const freshDynamicStorage = (node: ts.Expression, depth: number): boolean => { + const e = peel(node); + if (ts.isArrayLiteralExpression(e)) return true; + if (ts.isNewExpression(e) || ts.isCallExpression(e)) { + const callee = peel(e.expression); + if (ts.isIdentifier(callee)) return this.isStdlibGlobal(callee, "Array"); + return ( + ts.isPropertyAccessExpression(callee) && + (callee.name.text === "of" || callee.name.text === "from") && + this.isStdlibGlobal(callee.expression, "Array") + ); + } + if (depth > 4 || (!ts.isIdentifier(e) && !ts.isPropertyAccessExpression(e))) return false; + // A typed array along the way is the array a view shows. + const type = typeOrNull(e); + if (type === null || !dynamicType(type)) return false; + const symbol = panicSafe( + () => this.checker.getSymbolAtLocation(ts.isIdentifier(e) ? e : e.name), + undefined, + ); + const declaration = symbol?.valueDeclaration; + if (!symbol || !declaration) return false; + // Variables and class properties only: parameters take any argument, + // and object literals fill interface properties. + if ( + !(ts.isVariableDeclaration(declaration) && ts.isIdentifier(declaration.name)) && + !ts.isPropertyDeclaration(declaration) + ) + return false; + const assigned = assignedValues(symbol); + if (assigned === null) return false; + const values = [ + ...(declaration.initializer ? [declaration.initializer] : []), + ...(assigned ?? []), + ]; + return values.length > 0 && values.every((value) => freshDynamicStorage(value, depth + 1)); + }; + // Every value `=` assigns to each variable or property of the program, + // by symbol; null for one also written another way (compound or + // destructuring assignment). Built on first use. + let assignments: Map | null = null; + const assignedValues = (symbol: ts.Symbol): ts.Expression[] | null | undefined => { + if (assignments === null) { + const found = new Map(); + const add = (target: ts.Expression, value: ts.Expression | null): void => { + const e = peel(target); + if (ts.isArrayLiteralExpression(e) || ts.isObjectLiteralExpression(e)) { + // Destructuring: every target is written with a part of the value. + ts.forEachChild(e, (child) => { + if (ts.isIdentifier(child) || ts.isPropertyAccessExpression(child)) add(child, null); + else if (ts.isSpreadElement(child)) add(child.expression, null); + else if (ts.isPropertyAssignment(child)) add(child.initializer, null); + else if (ts.isShorthandPropertyAssignment(child)) + add(child.name as ts.Expression, null); + else if (ts.isBinaryExpression(child)) add(child.left, null); + return undefined; + }); + return; + } + const name = ts.isIdentifier(e) ? e : ts.isPropertyAccessExpression(e) ? e.name : null; + const written = name + ? panicSafe(() => this.checker.getSymbolAtLocation(name), undefined) + : undefined; + if (!written) return; + const list = found.get(written); + if (list === null) return; + if (value === null) found.set(written, null); + else if (list) list.push(value); + else found.set(written, [value]); + }; + for (const sf of sourceFiles) + ts.walkPreorder(sf, (node) => { + if (ts.isBinaryExpression(node)) { + const op = node.operatorToken.kind; + if (op === ts.SyntaxKind.EqualsToken) add(node.left, node.right); + else if (op >= ts.SyntaxKind.FirstAssignment && op <= ts.SyntaxKind.LastAssignment) + add(node.left, null); + } + return undefined; + }); + assignments = found; + } + return assignments.get(symbol); + }; // `t[i]!` on a typed array, stored into an array, keeps an invalid // index's undefined (assertedReadObservesUndefined): `!` performs no // runtime check. A read whose consumers treat undefined like NaN stores @@ -6575,6 +6829,11 @@ export class Lowerer { ); }; const elementStates = new ArrayElementStates({ + isDynamicArray: (node) => { + const type = typeOrNull(node); + return (type === null || dynamicType(type)) && !freshDynamicStorage(node, 0); + }, + nativeViewKeys: collectNativeViews, elementAdmitsUndefined: (key) => { siteDependencies?.read(key); return optionalElementKeys.has(key); @@ -7495,6 +7754,7 @@ export class Lowerer { } for (const [site, present] of presentSites) if (present) this.presentElementCalls.add(site); for (const [loop, present] of presentLoops) if (present) this.presentElementLoops.add(loop); + this.foreignElementKeys = elementStates.foreignKeys(); const globalsById = new Map(this.globalsList.map((global) => [global.id, global])); for (const [symbol, field] of staticFieldsBySymbol) { if (!optionalSymbols.has(symbol)) continue; @@ -8047,6 +8307,9 @@ export class Lowerer { ...(this.usesWorkers ? { workers: true } : {}), ...(this.usesLifetimeRegions ? { lifetimeRegions: true as const } : {}), ...(this.declBoundary ? { declBoundary: true as const } : {}), + ...(this.foreignElementKeys.size > 0 + ? { foreignElementKeys: [...this.foreignElementKeys].sort() } + : {}), ...(this.threadSlotKeys.size > 0 ? { threadSlotKeys: [...this.threadSlotKeys].sort() } : {}), diff --git a/packages/compiler/src/frontend/lowering/runtime-optional-elements.ts b/packages/compiler/src/frontend/lowering/runtime-optional-elements.ts index 7561f4a65e..0840cef91a 100644 --- a/packages/compiler/src/frontend/lowering/runtime-optional-elements.ts +++ b/packages/compiler/src/frontend/lowering/runtime-optional-elements.ts @@ -43,8 +43,17 @@ export interface ElementStateHost { ownerOf(node: ts.Expression): ts.Symbol | null; /** The owning slot a newly built array is stored into. */ allocationOwner(node: ts.Expression): ts.Symbol | null; + /** `node` is a dynamic value (`any`, `unknown`, `unknown[]`): when it is + * a live view of a native array, a store through it writes that array's + * storage. */ + isDynamicArray(node: ts.Expression): boolean; + /** Element keys of the native arrays a value converted to a dynamic type + * can reach, as live views ("*" for any array). */ + nativeViewKeys(): ReadonlySet; /** Debug provenance for a new fact. */ - note: ((fact: "undefined" | "holes", key: string, site: ts.Node | undefined) => void) | null; + note: + | ((fact: "undefined" | "holes" | "foreign", key: string, site: ts.Node | undefined) => void) + | null; /** The scan read, or added to, the facts of an element key, of an owning * slot, or of every array (ANY_ARRAY), for dependency tracking * (SiteDependencies). */ @@ -54,6 +63,16 @@ export interface ElementStateHost { /** The fact name of the facts shared by every array. */ const ANY_ARRAY = "*"; +/** The fact name of holes stored through dynamic views. */ +const VIEW_HOLES = "%view"; +/** The fact name of values stored through dynamic views. */ +const VIEW_VALUES = "%view-values"; + +/** Element keys whose arrays store a value written through a dynamic view + * that the element type cannot hold, as a foreign element: `number[]` and + * `string[]`. Arrays of other element types reject such a write with the + * checked conversion's TypeError. */ +const FOREIGN_ELEMENT_KEYS: ReadonlySet = new Set(["f64", "string"]); /** What an array value can contain besides present values. */ interface Contents { @@ -87,6 +106,13 @@ export class ArrayElementStates { * whose element ABI is decided per generic instantiation. */ private anyUndefined = false; private anyHoley = false; + /** A store through a dynamic value can leave holes; they land in every + * native array such a value views. */ + private viewHoles = false; + /** A store through a dynamic value can write any value, which the + * viewed native arrays then hold as undefined or foreign elements. */ + private viewValues = false; + private viewKeys: ReadonlySet | null = null; /** Facts of arrays owned by one storage slot. */ private readonly undefinedOwners = new Set(); private readonly holeyOwners = new Set(); @@ -336,14 +362,65 @@ export class ArrayElementStates { this.host.read?.(ANY_ARRAY); if (keys === null) return null; for (const key of keys) this.host.read?.(key); + this.host.read?.(VIEW_HOLES); + this.host.read?.(VIEW_VALUES); return { undefined: this.anyUndefined || - keys.some((key) => this.undefinedKeys.has(key) || this.host.elementAdmitsUndefined(key)), - holes: this.anyHoley || keys.some((key) => this.holeyKeys.has(key)), + keys.some( + (key) => + this.undefinedKeys.has(key) || + this.host.elementAdmitsUndefined(key) || + this.viewedValues(key), + ), + holes: this.anyHoley || keys.some((key) => this.holeyKeys.has(key) || this.viewed(key)), }; } + /** Holes stored through a dynamic view can be in arrays of this key. */ + private viewed(key: string): boolean { + return this.viewHoles && this.isViewed(key); + } + + /** Values of any type stored through a dynamic view can be in arrays of + * this key. */ + private viewedValues(key: string): boolean { + return this.viewValues && FOREIGN_ELEMENT_KEYS.has(key) && this.isViewed(key); + } + + private isViewed(key: string): boolean { + this.viewKeys ??= this.host.nativeViewKeys(); + return this.viewKeys.has(key) || this.viewKeys.has(ANY_ARRAY); + } + + /** Element keys whose arrays can hold elements outside their element + * type, stored through dynamic views (of FOREIGN_ELEMENT_KEYS). Final + * once the fixed point is reached. */ + foreignKeys(): ReadonlySet { + if (!this.viewValues) return new Set(); + return new Set([...FOREIGN_ELEMENT_KEYS].filter((key) => this.isViewed(key))); + } + + /** A store through `receiver`, which has no native element key, can + * write a value of any type into a viewed native array. */ + private addViewValues(receiver: ts.Expression): boolean { + if (this.viewValues || !this.host.isDynamicArray(receiver)) return false; + this.viewValues = true; + this.host.changed?.(VIEW_VALUES); + this.host.note?.("foreign", VIEW_VALUES, this.site); + return true; + } + + /** A store through `receiver`, which has no native element key, can + * leave holes. Only a dynamic value can be a view of a native array. */ + private addViewHoles(receiver: ts.Expression): boolean { + if (this.viewHoles || !this.host.isDynamicArray(receiver)) return false; + this.viewHoles = true; + this.host.changed?.(VIEW_HOLES); + this.host.note?.("holes", VIEW_HOLES, this.site); + return true; + } + /** Reading every index of `node` (a spread, an iteration, `Array.from`) * can produce `undefined`. Only native arrays can: other iterables reject * an undefined value when it is stored. */ @@ -437,7 +514,11 @@ export class ArrayElementStates { /** A store through `receiver` can leave holes. */ private addHolesTo(receiver: ts.Expression): boolean { const owner = this.host.ownerOf(receiver); - if (owner === null) return this.addHoles(this.host.arrayKeys(receiver)); + if (owner === null) { + const keys = this.host.arrayKeys(receiver); + if (keys !== null && keys.length === 0) return this.addViewHoles(receiver); + return this.addHoles(keys); + } if (this.holeyOwners.has(owner)) return false; this.holeyOwners.add(owner); this.host.changed?.(owner); @@ -497,10 +578,18 @@ export class ArrayElementStates { op === ts.SyntaxKind.QuestionQuestionEqualsToken || op === ts.SyntaxKind.BarBarEqualsToken || op === ts.SyntaxKind.AmpersandAmpersandEqualsToken; + const keys = this.host.arrayKeys(target.expression); + if (keys !== null && keys.length === 0) { + // Through a dynamic value: any element write can store a value the + // viewed native array's element type does not hold. + if (namesProperty(target)) return false; + let changed = false; + if (holes && this.addViewHoles(target.expression)) changed = true; + if (this.addViewValues(target.expression)) changed = true; + return changed; + } const undefinedValue = storesValue && this.host.mayBeUndefined(node.right); if (!holes && !undefinedValue) return false; - const keys = this.host.arrayKeys(target.expression); - if (keys !== null && keys.length === 0) return false; let changed = false; if (holes && this.addHolesTo(target.expression)) changed = true; if (undefinedValue && this.addUndefinedTo(target.expression)) changed = true; @@ -511,8 +600,6 @@ export class ArrayElementStates { target.name.text === "length" && !isShrinkingLength(node, target) ) { - const keys = this.host.arrayKeys(target.expression); - if (keys !== null && keys.length === 0) return false; return this.addHolesTo(target.expression); } return false; @@ -563,7 +650,14 @@ export class ArrayElementStates { return changed; } const receiver = this.host.arrayKeys(callee.expression); - if (receiver !== null && receiver.length === 0) return changed; + if (receiver !== null && receiver.length === 0) { + if ( + (stored.length > 0 || callee.name.text === "fill") && + this.addViewValues(callee.expression) + ) + changed = true; + return changed; + } for (const value of stored) if (this.valueMayBeUndefined(value) && this.addUndefinedTo(callee.expression)) changed = true; return changed; @@ -596,10 +690,17 @@ export class ArrayElementStates { )); if (!reflective) return false; let changed = false; - for (const arg of args) { + // The written object: `Reflect.apply`'s receiver argument, otherwise + // the first argument. + const written = this.host.isGlobal(owner, "Reflect") && method === "apply" ? 1 : 0; + for (const [index, arg] of args.entries()) { const target = ts.isSpreadElement(arg) ? arg.expression : arg; const keys = this.host.arrayKeys(target); - if (keys !== null && keys.length === 0) continue; + if (keys !== null && keys.length === 0) { + if (index === written && this.addViewHoles(target)) changed = true; + if (index === written && this.addViewValues(target)) changed = true; + continue; + } if (this.addUndefinedTo(target)) changed = true; if (this.addHolesTo(target)) changed = true; } @@ -783,6 +884,14 @@ function isAppendIndex(target: ts.ElementAccessExpression): boolean { ); } +/** `o["name"]`: a string literal key that is not an array index writes a + * named property, never an element. */ +function namesProperty(target: ts.ElementAccessExpression): boolean { + const key = peel(target.argumentExpression); + if (!ts.isStringLiteral(key) && !ts.isNoSubstitutionTemplateLiteral(key)) return false; + return !/^(0|[1-9][0-9]*)$/.test(key.text) || Number(key.text) >= 2 ** 32 - 1; +} + /** `xs.length = 0`, `xs.length = xs.length - k` and `xs.length -= k` * truncate; truncation never leaves holes. */ function isShrinkingLength( diff --git a/packages/compiler/src/frontend/type-mapper.ts b/packages/compiler/src/frontend/type-mapper.ts index 17f9d0d110..edb3cf1440 100644 --- a/packages/compiler/src/frontend/type-mapper.ts +++ b/packages/compiler/src/frontend/type-mapper.ts @@ -4948,6 +4948,19 @@ function mapTypeInner(type: ts.Type, ctx: TypeMapperCtx): IrType | null { const record = mapRecordType(widened, ctx); if (record) return record; } + // An intersection whose constituents all map to ONE representation is + // that representation. An intersection that restates an optional member + // as required (`Opts & { files: Array<{ path: string }> }` over + // `files?: Array<{ path: string }>`) gives the member the type `A[] & B[]` + // with two separately written but structurally equal element literals; + // the checker keeps both, while their arrays (tuples, maps, ...) intern + // to one IR type. Reached only after every other intersection rule + // declined, so intersections that already mapped keep their mapping. A + // dynamic result stays unmapped: two opaque types are not one type. + if (widened.isIntersectionType()) { + const same = mapEqualConstituentsIntersection(widened, ctx); + if (same) return same; + } // Unions. `boolean` is internally `true | false`, and narrowing produces // literal unions like `"a" | "b"` — parts that map to the SAME IR type // collapse (deduplicated by typeKey), so a single surviving arm is just @@ -6813,6 +6826,19 @@ export function withUnitArm( * assigned keys, which is also Node's answer for Object.assign(fn, obj) * (a function's own length/name are non-enumerable). Null whenever the * shape doesn't fit — the caller falls through to the plain-func mapping. */ +/** The IR type every constituent of an intersection maps to, or null when a + * constituent is unmappable or dynamic, or two constituents differ. */ +function mapEqualConstituentsIntersection(widened: ts.Type, ctx: TypeMapperCtx): IrType | null { + let same: IrType | null = null; + for (const part of ts.constituentTypes(widened)) { + const mapped = mapType(part, ctx); + if (mapped === null || mapped.kind === "dyn") return null; + if (same === null) same = mapped; + else if (!typeEquals(same, mapped)) return null; + } + return same; +} + function mapHybridCallableIntersection(widened: ts.Type, ctx: TypeMapperCtx): IrType | null { const { checker, shapes } = ctx; if (!widened.isIntersectionType()) return null; diff --git a/packages/compiler/src/frontend/type-patches/layer.ts b/packages/compiler/src/frontend/type-patches/layer.ts index 9c156b68c8..bb3b3627aa 100644 --- a/packages/compiler/src/frontend/type-patches/layer.ts +++ b/packages/compiler/src/frontend/type-patches/layer.ts @@ -64,6 +64,8 @@ interface MatchRecord extends PatchMatch { interface State { manifest: string; + /** The directory of the project's package.json. */ + projectDir: string; manifestText: string; parse: TypePatchParse; byPackage: Map; @@ -112,6 +114,7 @@ export function configureTypePatches(entryPath: string, parse: TypePatchParse): } state = { manifest: config.manifest, + projectDir: dirname(config.manifest), manifestText: config.manifestText ?? "", parse, byPackage, @@ -151,6 +154,22 @@ function versionAt(dir: string): string | null { } } +/** The version of the runtime package `pkg` an `@types` twin under + * `nodeModules` describes: the copy installed beside it, else the one the + * project resolves (pnpm keeps each package in its own directory, so a + * twin has no runtime package beside it), else null. */ +function twinRuntimeVersion(s: State, nodeModules: string, pkg: string): string | null { + const beside = versionAt(nodeModules + pkg); + if (beside !== null) return beside; + for (let dir = s.projectDir; ;) { + const version = versionAt(`${dir}/node_modules/${pkg}`); + if (version !== null) return version; + const parent = dirname(dir); + if (parent === dir) return null; + dir = parent; + } +} + /** The installed package copy a file belongs to, with the requests that * apply to it; null for files outside node_modules or of unpatched * packages. */ @@ -172,9 +191,9 @@ function copyOf(s: State, path: string, touch = true): { copy: CopyInfo; rel: st const pkg = twin ? unmangleTypesName(name) : name; const patches = s.byPackage.get(pkg); if (patches !== undefined) { - // A twin describes the runtime package installed beside it. + // A twin is matched by the runtime package's version, never its own. const version = twin - ? (versionAt(path.slice(0, at + marker.length) + pkg) ?? versionAt(root)) + ? twinRuntimeVersion(s, path.slice(0, at + marker.length), pkg) : versionAt(root); const applicable = patches.entries.filter( (e) => version !== null && satisfies(version, e.range), diff --git a/packages/compiler/src/frontend/type-patches/locate.ts b/packages/compiler/src/frontend/type-patches/locate.ts index fc85a16344..6c762f5c3e 100644 --- a/packages/compiler/src/frontend/type-patches/locate.ts +++ b/packages/compiler/src/frontend/type-patches/locate.ts @@ -596,6 +596,26 @@ function parseJsDoc(text: string, start: number, end: number): JsDocComment { return { start, end, tags }; } +/** The spans of a file's string, template and regular expression literal + * tokens. */ +function literalSpans(sf: ts.SourceFile): { start: number; end: number }[] { + const out: { start: number; end: number }[] = []; + const kinds = new Set([ + ts.SyntaxKind.StringLiteral, + ts.SyntaxKind.NoSubstitutionTemplateLiteral, + ts.SyntaxKind.TemplateHead, + ts.SyntaxKind.TemplateMiddle, + ts.SyntaxKind.TemplateTail, + ts.SyntaxKind.RegularExpressionLiteral, + ]); + const visit = (n: ts.Node): void => { + if (kinds.has(n.kind)) out.push({ start: tokenStartOfNode(n), end: n.end }); + ts.forEachChild(n, visit); + }; + visit(sf); + return out.sort((a, b) => a.start - b.start); +} + const RETURN_TAGS = new Set(["returns", "return"]); const PROPERTY_TAGS = new Set(["property", "prop"]); @@ -615,15 +635,27 @@ class JsResolver { private readonly text: string, sf: ts.SourceFile, ) { + // JSDoc comments, not text inside string, template or regular + // expression literals that looks like one. + const literals = literalSpans(sf); const re = /\/\*\*(?!\/)[\s\S]*?\*\//g; - for (let m = re.exec(text); m !== null; m = re.exec(text)) + for (let m = re.exec(text); m !== null; m = re.exec(text)) { + const inside = literals.find((l) => l.start <= m!.index && m!.index < l.end); + if (inside !== undefined) { + re.lastIndex = inside.end; + continue; + } this.comments.push(parseJsDoc(text, m.index, m.index + m[0].length)); + } for (const c of this.comments) c.tags.forEach((t, i) => { if ((t.tag === "typedef" || t.tag === "callback") && t.name !== null) this.add(t.name, { kind: "typedef", comment: c, tagIndex: i, prefix: "" }); }); - for (const s of sf.statements.nodes) this.collect(s); + // Declarations first: `exports.name = name` may come before the + // function declaration it exports, which is hoisted. + for (const s of sf.statements.nodes) if (!ts.isExpressionStatement(s)) this.collect(s); + for (const s of sf.statements.nodes) if (ts.isExpressionStatement(s)) this.collect(s); } private add(name: string, pos: JsPos): void { @@ -681,9 +713,11 @@ class JsResolver { if (named !== undefined) this.exportEquals.push(...named); } } else if (ts.isIdentifier(e.right) && this.roots.has(e.right.text)) { - // `exports.name = name`: the named declaration is the export. - for (const pos of this.roots.get(e.right.text)!) - if (pos.kind === "decl") this.add(target, pos); + // `exports.name = local`: the named declaration is the export (and + // already is under its own name). + if (target !== e.right.text) + for (const pos of [...this.roots.get(e.right.text)!]) + if (pos.kind === "decl") this.add(target, pos); } else this.add(target, this.declPos(s, s, e.right)); } diff --git a/packages/compiler/src/ir/builtin-effects.ts b/packages/compiler/src/ir/builtin-effects.ts index 76f570e328..72621f10c8 100644 --- a/packages/compiler/src/ir/builtin-effects.ts +++ b/packages/compiler/src/ir/builtin-effects.ts @@ -880,6 +880,7 @@ export const MAY_THROW_LIB_FNS: ReadonlySet = new Set([ "net.setAutoSelTimeout", "error.argTypeThrow", "error.propTypeThrow", + "array.foreignThrow", "emitter.setMaxChk", "emitter.setDefaultMaxChk", "fs.readFileSyncBytes", @@ -1081,6 +1082,7 @@ export const ALWAYS_THROW_LIB_FNS: ReadonlySet = new Set([ "error.nodeThrow", "error.argTypeThrow", "error.propTypeThrow", + "array.foreignThrow", "fs.mkdtempChk", "fs.readFileChk", "fs.opendirChk", diff --git a/packages/compiler/src/ir/builtin-signatures.ts b/packages/compiler/src/ir/builtin-signatures.ts index 916c645837..091504e656 100644 --- a/packages/compiler/src/ir/builtin-signatures.ts +++ b/packages/compiler/src/ir/builtin-signatures.ts @@ -848,6 +848,7 @@ function buildLibFnSignatures() { // (subject, characters, NUL-separated replacement texts): the fused // chain of global one-character replacements. "str.replaceChars": { argTypes: [STRING, STRING, STRING], result: STRING }, + "str.codePoints": { argTypes: [STRING], result: arrayOf(STRING) }, /* ToNumber(string) — ECMA-262 7.1.4.1 StringToNumber (scr_string.c): * trim the JS StrWhiteSpace set, empty/whitespace-only → +0, then the * whole span must be one StrNumericLiteral — signed decimal (Infinity @@ -1547,15 +1548,18 @@ function buildLibFnSignatures() { * true } and a callback taking an address string or list, then dials * the answers in order. MAY THROW (host validation, a lookup throw). */ "net.connectLookupDyn": { argTypes: [F64, STRING, DYN], result: NETSOCKET_T }, - "net.sockWrite": { argTypes: [NETSOCKET_T, STRING], result: VOID }, - "net.sockWriteBytes": { argTypes: [NETSOCKET_T, BYTES_U8], result: VOID }, + /** socket.write: Node's Writable answer — false when the writes not yet + * completely sent, each counted in full, reach the 65536-byte + * high-water mark ('drain' follows), and on an ended or closed socket. */ + "net.sockWrite": { argTypes: [NETSOCKET_T, STRING], result: BOOL }, + "net.sockWriteBytes": { argTypes: [NETSOCKET_T, BYTES_U8], result: BOOL }, "net.sockEnd": { argTypes: [NETSOCKET_T], result: VOID }, "net.sockEndStr": { argTypes: [NETSOCKET_T, STRING], result: VOID }, "net.sockEndBytes": { argTypes: [NETSOCKET_T, BYTES_U8], result: VOID }, /** write/end with a CHECKED-DYNAMIC chunk (an untyped JS payload into a * typed socket): the runtime dispatches STR/BYTES and throws Node's * ERR_INVALID_ARG_TYPE chunk TypeError on any other kind. */ - "net.sockWriteDyn": { argTypes: [NETSOCKET_T, DYN], result: VOID }, + "net.sockWriteDyn": { argTypes: [NETSOCKET_T, DYN], result: BOOL }, "net.sockEndDyn": { argTypes: [NETSOCKET_T, DYN], result: VOID }, "net.sockDestroy": { argTypes: [NETSOCKET_T], result: VOID }, "net.sockPipe": { argTypes: [NETSOCKET_T, NETSOCKET_T], result: VOID }, @@ -2013,6 +2017,12 @@ function buildLibFnSignatures() { argTypes: [NETSOCKET_T, { kind: "func", params: [], ret: VOID }, BOOL], result: VOID, }, + /** on('drain') / once('drain'): fires after a write answered false, once + * the write buffer empties. */ + "net.sockOnDrain": { + argTypes: [NETSOCKET_T, { kind: "func", params: [], ret: VOID }, BOOL], + result: VOID, + }, "net.sockRemoteAddress": { argTypes: [NETSOCKET_T], result: VOID }, /** socket.encrypted — `boolean | undefined`: the true arm iff the * socket carries a TLS transport (Node types `encrypted: true` on @@ -2139,6 +2149,12 @@ function buildLibFnSignatures() { /** socket.address(): { address, family, port } or {} as a dyn. */ "net.sockAddress": { argTypes: [NETSOCKET_T], result: DYN }, "net.sockBytesWritten": { argTypes: [NETSOCKET_T], result: F64 }, + /** socket.writableLength: Node's state.length, the length of every + * write not yet completely sent. */ + "net.sockWritableLength": { argTypes: [NETSOCKET_T], result: F64 }, + /** socket.writableNeedDrain: a write answered false and 'drain' has not + * fired, with the write half open. */ + "net.sockWritableNeedDrain": { argTypes: [NETSOCKET_T], result: BOOL }, /** remotePort (0), remoteFamily (1), localAddress (2), localPort (3), * localFamily (4): dyn values, undefined when unknown. */ "net.sockAddrField": { argTypes: [NETSOCKET_T, F64], result: DYN }, @@ -3974,6 +3990,15 @@ function buildLibFnSignatures() { // The string keys `for...in` visits on an array (args[0], any element // type) after its indices. Borrowed; never throws. "array.namedKeys": { argTypes: [null], result: arrayOf(STRING) }, + // The value of the foreign element (SCR_ARR_FOREIGN) at args[1] of an + // array (args[0], any element type); callers test the element state + // first. Borrowed; never throws. + "array.foreign": { argTypes: [null, F64], result: DYN }, + /** The checked conversion's TypeError for the foreign element at args[1] + * of an array (args[0]) read where a value described by args[2] is + * expected. ALWAYS THROWS; polymorphic result (the error.nodeThrow + * pattern). May-throw seed. */ + "array.foreignThrow": { argTypes: [null, F64, STRING], result: VOID }, // Whether the array (args[0]) still has the key (args[1]) from its // namedKeys list. Borrowed; never throws. "array.hasNamedKey": { argTypes: [null, STRING], result: BOOL }, diff --git a/packages/compiler/src/ir/function-references.test.ts b/packages/compiler/src/ir/function-references.test.ts index db872a525b..e97025e828 100644 --- a/packages/compiler/src/ir/function-references.test.ts +++ b/packages/compiler/src/ir/function-references.test.ts @@ -221,6 +221,7 @@ const MODULE_FIELDS: Record>, Role> = { "wasmUntranslated[].sha256": "other", "wasmUntranslated[].reason": "other", "threadSlotKeys[]": "other", + "foreignElementKeys[]": "other", "freezableArrays[]": "other", }; diff --git a/packages/compiler/src/ir/integer-ranges.ts b/packages/compiler/src/ir/integer-ranges.ts index 1909040c42..51c0a929b4 100644 --- a/packages/compiler/src/ir/integer-ranges.ts +++ b/packages/compiler/src/ir/integer-ranges.ts @@ -867,6 +867,8 @@ export function analyzeIntegerRangeFacts( ); if (e.type.kind === "f64") { if (e.method === "length") range = { min: 0, max: STRING_MAX_UNITS }; + // A string holds at most SCR_STR_MAX_CAP (0x7fffffff) bytes. + else if (e.method === "byteLength") range = { min: 0, max: STRING_MAX_UNITS }; // An empty needle is found at the clamped position, up to the length. else if (e.method === "indexOf") range = { min: -1, max: STRING_MAX_UNITS }; } diff --git a/packages/compiler/src/ir/intrinsic-signatures.ts b/packages/compiler/src/ir/intrinsic-signatures.ts index eda5698835..3e26e77f2a 100644 --- a/packages/compiler/src/ir/intrinsic-signatures.ts +++ b/packages/compiler/src/ir/intrinsic-signatures.ts @@ -31,7 +31,8 @@ export const STR_INTRINSIC_SIGS: Record< normalize: { argTypes: [STRING], minArgs: 1, result: STRING }, isWellFormed: { argTypes: [], minArgs: 0, result: BOOL }, toWellFormed: { argTypes: [], minArgs: 0, result: STRING }, - cpAt: { argTypes: [F64], minArgs: 1, result: STRING }, + byteLength: { argTypes: [], minArgs: 0, result: F64 }, + cpAtByte: { argTypes: [F64], minArgs: 1, result: STRING }, }; /** Per-method signature for regexIntrinsic. test/source/flags take the diff --git a/packages/compiler/src/ir/ir.ts b/packages/compiler/src/ir/ir.ts index 19a5be71df..c946d799f0 100644 --- a/packages/compiler/src/ir/ir.ts +++ b/packages/compiler/src/ir/ir.ts @@ -1067,6 +1067,10 @@ export interface IrModule { * ProgramLifetime instance lives in a counted region (the region.* * intrinsics; backend/llvm/regions.ts). */ lifetimeRegions?: true; + /** Element keys (typeKey of the element type) of native arrays that can + * hold elements outside their element type (SCR_ARR_FOREIGN), stored + * through dynamic views; "*" for every array. Absent when none can. */ + foreignElementKeys?: string[]; /** Classes used as ThreadSlots keys (@scriptc/experimental): the root of * each one's hierarchy carries a slot word (the slots.* intrinsics; * backend/llvm/thread-slots.ts). */ @@ -2447,10 +2451,13 @@ export type IrStrIntrinsicMethod = // surrogate, and the copy with each one replaced by U+FFFD. | "isWellFormed" | "toWellFormed" - // The string iterator's step: the full code-POINT character at a UTF-16 - // index (astral chars come back whole where charAt truncates) — for-of - // over strings desugars onto it, advancing by the result's length. - | "cpAt"; + // The string iterator over the WTF-8 bytes: the byte count of a string, + // and the code-point character that starts at a byte offset (a lone + // surrogate is a one-unit character). for-of, spread and Array.from of a + // string walk a byte cursor with them, advancing by the character's + // byte count; the cursor is always a character start below the count. + | "byteLength" + | "cpAtByte"; /** The typed-array/Buffer method surface (bytesIntrinsic). Receiver/arg * conventions (validated): `length`/`byteLength` are property reads → f64; diff --git a/packages/compiler/src/ir/validate.ts b/packages/compiler/src/ir/validate.ts index bbb79438d3..45ebf5c790 100644 --- a/packages/compiler/src/ir/validate.ts +++ b/packages/compiler/src/ir/validate.ts @@ -4313,7 +4313,9 @@ function validateFunction( if ( e.fn === "array.namedKeys" || e.fn === "array.hasNamedKey" || - e.fn === "array.hasProperties" + e.fn === "array.hasProperties" || + e.fn === "array.foreign" || + e.fn === "array.foreignThrow" ) { if (e.args[0]?.type.kind !== "array") { err(`libCall ${e.fn} arg 0: expected an array, got ${e.args[0]?.type.kind}`, e.loc); diff --git a/packages/compiler/surface-manifest.json b/packages/compiler/surface-manifest.json index 58b3da7e29..5f5d8f7e03 100644 --- a/packages/compiler/surface-manifest.json +++ b/packages/compiler/surface-manifest.json @@ -1,6 +1,6 @@ { "schemaVersion": 2, - "compilerVersion": "0.4.4", + "compilerVersion": "0.4.5", "coverage": [ "Entries are projected mechanically from the compiler's own decision tables: the diagnostics registry, the unsupported-syntax dispatch tables, the stdlib and node-builtin lowering tables, and the supported-builtin-module list. Nothing is hand-maintained; the manifest regenerates byte-identically from the source tree at this version.", "Absence from this manifest means 'not projected', never 'unsupported'. Surfaces lowered through dedicated code paths rather than tables are not yet projected: console, JSON, Promise/async and the timer surface, the net/http/tls/https/dgram/dns/assert/test/stream/readline module member surfaces, template literals, the regex slice, global functions (parseInt, parseFloat, isNaN, isFinite), and the process surface outside its ambient slice.", diff --git a/packages/compiler/test/js-ic-emission.test.ts b/packages/compiler/test/js-ic-emission.test.ts index 79342e99d1..b350c743d5 100644 --- a/packages/compiler/test/js-ic-emission.test.ts +++ b/packages/compiler/test/js-ic-emission.test.ts @@ -201,7 +201,7 @@ console.log(lib.total([1, 2, 3])); expect(fn).toContain("@scr_js_keyed_probe("); // The length read answers from the native array's length. expect(fn).toContain("icg.rest"); - expect(fn).toContain("@scr_js_get_probe("); + expect(fn).toContain("@scr_js_get_probe_native("); } finally { await rm(dir, { recursive: true, force: true }); } diff --git a/packages/compiler/test/native-codegen-integration.test.ts b/packages/compiler/test/native-codegen-integration.test.ts index dc3f94e6c2..a57d96f013 100644 --- a/packages/compiler/test/native-codegen-integration.test.ts +++ b/packages/compiler/test/native-codegen-integration.test.ts @@ -378,7 +378,7 @@ done: expect(await symbols(helperObject, ["-u"])).toEqual(await symbols(clangObject, ["-u"])); expect(await symbols(helperObject, ["-gU"])).toEqual(await symbols(clangObject, ["-gU"])); expect(await symbols(helperObject, ["-gU"])).toEqual(["0000000000000000 T _main"]); - expect(await symbols(helperObject, ["-u"])).toContain("_scr_runtime_abi_v44"); + expect(await symbols(helperObject, ["-u"])).toContain("_scr_runtime_abi_v45"); const clangExe = join(dir, "clang-program"); const linkDriver = join(dir, "clang-driver.c"); @@ -712,7 +712,7 @@ done: .split("\n"); expect(undefinedSymbols).toContain("_sf_scale"); expect(undefinedSymbols).toContain("_sf_callback_mix"); - expect(undefinedSymbols).toContain("_scr_runtime_abi_v44"); + expect(undefinedSymbols).toContain("_scr_runtime_abi_v45"); }); test("helper and clang object paths retain the same outbound FFI C ABI", async () => { diff --git a/packages/compiler/test/regex-replacers.test.ts b/packages/compiler/test/regex-replacers.test.ts new file mode 100644 index 0000000000..746279eb4c --- /dev/null +++ b/packages/compiler/test/regex-replacers.test.ts @@ -0,0 +1,42 @@ +import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { expect, test } from "vitest"; +import { compile } from "../src/index.js"; + +async function emitLlvm(source: string): Promise { + const dir = await mkdtemp(join(tmpdir(), "scriptc-regex-replacers-")); + try { + const entry = join(dir, "main.ts"); + const outPath = join(dir, "main.ll"); + await writeFile(entry, source); + const result = await compile(entry, { outDir: dir, outPath, outputKind: "llvm" }); + expect(result.ok, JSON.stringify(result.diagnostics)).toBe(true); + return await readFile(outPath, "utf8"); + } finally { + await rm(dir, { recursive: true, force: true }); + } +} + +test("global substitutions of regex literals with literal replacements run a replacer", async () => { + const llvm = await emitLlvm(` +const s = process.argv[2] ?? "ac"; +const rep = process.argv[3] ?? "-"; +console.log( + s.replace(/<[^>]*>/g, "|"), + s.replaceAll(/a[bc]/g, "<2>"), + s.replace(/<[^>]*>/g, "|"), + s.replace(/<[^>]*>/g, "$&$&"), + s.replace(/<[^>]*>/, "|"), + s.replace(/<[^>]*>/gy, "|"), + s.replace(/<[^>]*>/g, rep), +); +`); + const replacers = llvm.match(/^define internal i32 @sc_rr_\d+\(/gm) ?? []; + // One replacer per (pattern, flags, replacement): the repeated site shares it. + expect(replacers).toHaveLength(2); + expect(llvm.match(/call ptr @scr_regex_replace_with\(/g)).toHaveLength(3); + // $ patterns, non-global and sticky patterns, and computed replacements + // keep the runtime's loop. + expect(llvm.match(/call ptr @scr_regex_replace\(/g)!.length).toBeGreaterThanOrEqual(4); +}); diff --git a/packages/compiler/test/runtime-optional-precision.test.ts b/packages/compiler/test/runtime-optional-precision.test.ts index b47cb828dc..e8b794b441 100644 --- a/packages/compiler/test/runtime-optional-precision.test.ts +++ b/packages/compiler/test/runtime-optional-precision.test.ts @@ -102,6 +102,82 @@ console.log(copy.map((item) => weigh(item)).join(",")); expect(signature(spread, "weigh")).toBe("weigh(Item|undefined): f64"); }); +test("holes stored through a dynamic view widen callbacks over the arrays it views", async () => { + const viewed = await lower(`${ITEM} +const items: Item[] = [new Item(1)]; +const view: unknown[] = items; +view[3] = new Item(2); +console.log(items.findIndex((item) => weigh(item) > 1)); +`); + expect(signature(viewed, "weigh")).toBe("weigh(Item|undefined): f64"); + // A dynamic store cannot reach an element type no dynamic value views. + const other = await lower(`${ITEM} +const items: Item[] = [new Item(1)]; +const values: unknown[] = [1]; +values[3] = 2; +console.log(items.findIndex((item) => weigh(item) > 1)); +`); + expect(signature(other, "weigh")).toBe("weigh(Item): f64"); + // A view without stores that leave holes keeps the parameters. + const read = await lower(`${ITEM} +const items: Item[] = [new Item(1)]; +const view: unknown[] = items; +console.log(view.length, items.findIndex((item) => weigh(item) > 1)); +`); + expect(signature(read, "weigh")).toBe("weigh(Item): f64"); + // Standard library functions that take arrays as `any` only read them. + const readers = await lower(`${ITEM} +const items: Item[] = [new Item(1)]; +console.log(items.length, JSON.stringify(items)); +const values: unknown[] = [1]; +values[3] = 2; +console.log(items.findIndex((item) => weigh(item) > 1)); +`); + expect(signature(readers, "weigh")).toBe("weigh(Item): f64"); + // A store through a Record type is not a store through a view. + const record = await lower(`${ITEM} +const items: Item[] = [new Item(1)]; +const view: unknown = items; +const bag: Record = {}; +bag["3"] = view; +console.log(items.findIndex((item) => weigh(item) > 1)); +`); + expect(signature(record, "weigh")).toBe("weigh(Item): f64"); + // An unknown[] that only holds the array its declaration creates is never + // a view, so stores past its end leave the viewed arrays alone. + const fresh = await lower(`${ITEM} +class Slots { + readonly states: unknown[] = []; +} +const slots = new Slots(); +const items: Item[] = [new Item(1)]; +const view: unknown = items; +const states = slots.states; +states[3] = view; +console.log(items.findIndex((item) => weigh(item) > 1)); +`); + expect(signature(fresh, "weigh")).toBe("weigh(Item): f64"); + // A mutable property every assignment of which is a fresh array is fresh + // storage too. + const reassigned = await lower(`${ITEM} +class Facts { + private facts: unknown[] = []; + begin(): void { + this.facts = []; + } + add(fact: unknown): void { + this.facts[3] = fact; + } +} +const items: Item[] = [new Item(1)]; +const facts = new Facts(); +facts.begin(); +facts.add(items); +console.log(items.findIndex((item) => weigh(item) > 1)); +`); + expect(signature(reassigned, "weigh")).toBe("weigh(Item): f64"); +}); + test("holes that only their owning slot can observe stay out of callbacks", async () => { const module = await lower(`${ITEM} class Cache { diff --git a/packages/compiler/test/string-iteration-emission.test.ts b/packages/compiler/test/string-iteration-emission.test.ts new file mode 100644 index 0000000000..7cda3da55a --- /dev/null +++ b/packages/compiler/test/string-iteration-emission.test.ts @@ -0,0 +1,47 @@ +import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, expect, test } from "vitest"; +import { compile } from "../src/index.js"; + +const directories: string[] = []; +afterEach(async () => { + await Promise.all(directories.splice(0).map((dir) => rm(dir, { recursive: true, force: true }))); +}); + +async function emit(source: string): Promise { + const outDir = await mkdtemp(join(tmpdir(), "scriptc-string-iteration-")); + directories.push(outDir); + const entry = join(outDir, "main.ts"); + await writeFile(entry, source); + const outPath = join(outDir, "program.ll"); + const result = await compile(entry, { entry, outDir, outPath, outputKind: "llvm" }); + expect(result.ok, JSON.stringify(result)).toBe(true); + return readFile(outPath, "utf8"); +} + +const calls = (llvm: string, symbol: string): string[] => + llvm.split("\n").filter((line) => line.includes("call ") && line.includes(`@${symbol}(`)); + +test("for-of over a string walks a byte cursor", async () => { + const llvm = await emit(` +function count(s: string): number { + let n = 0; + for (const c of s) n += c.length; + return n; +} +console.log(count(process.argv[1] ?? "")); +`); + expect(calls(llvm, "sc_str_cp_at_byte")).toHaveLength(1); + expect(calls(llvm, "scr_str_cp_at")).toHaveLength(0); + // The helper answers ASCII bytes from the one-character table. + expect(llvm).toContain("define internal ptr @sc_str_cp_at_byte(ptr %s, i64 %b)"); +}); + +test("spread and Array.from of a string build the array in one call", async () => { + const llvm = await emit(` +const s = process.argv[1] ?? ""; +console.log([...s].length, Array.from(s).length); +`); + expect(calls(llvm, "scr_str_code_points")).toHaveLength(2); +}); diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6067-view-holes-in-typed-arrays.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6067-view-holes-in-typed-arrays.ts.json new file mode 100644 index 0000000000..9d257fd491 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6067-view-holes-in-typed-arrays.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6067-view-holes-in-typed-arrays.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6068-foreign-elements-through-views.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6068-foreign-elements-through-views.ts.json new file mode 100644 index 0000000000..7da41a80fa --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6068-foreign-elements-through-views.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6068-foreign-elements-through-views.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6069-foreign-element-writers.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6069-foreign-element-writers.ts.json new file mode 100644 index 0000000000..973bdb93e5 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6069-foreign-element-writers.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6069-foreign-element-writers.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/7515-net-socket-backpressure.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/7515-net-socket-backpressure.ts.json new file mode 100644 index 0000000000..16f6edb8d5 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/7515-net-socket-backpressure.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/7515-net-socket-backpressure.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/7519-net-socket-drain-listener-close.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/7519-net-socket-drain-listener-close.ts.json new file mode 100644 index 0000000000..a3c10da992 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/7519-net-socket-drain-listener-close.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/7519-net-socket-drain-listener-close.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8010-http-header-raw-bytes.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8010-http-header-raw-bytes.ts.json new file mode 100644 index 0000000000..78240d3108 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8010-http-header-raw-bytes.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/8010-http-header-raw-bytes.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8011-foreign-element-tojson.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8011-foreign-element-tojson.ts.json new file mode 100644 index 0000000000..e11c1b240e --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8011-foreign-element-tojson.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/8011-foreign-element-tojson.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8347-string-iteration-code-points.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8347-string-iteration-code-points.ts.json new file mode 100644 index 0000000000..1e83f7b6fa --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8347-string-iteration-code-points.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/8347-string-iteration-code-points.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8348-string-iteration-worker-characters.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8348-string-iteration-worker-characters.ts.json new file mode 100644 index 0000000000..e767eaf65a --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8348-string-iteration-worker-characters.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/8348-string-iteration-worker-characters.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8351-regex-literal-replacers.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8351-regex-literal-replacers.ts.json new file mode 100644 index 0000000000..be01492872 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8351-regex-literal-replacers.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/8351-regex-literal-replacers.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/intersection-restated-members.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/intersection-restated-members.ts.json new file mode 100644 index 0000000000..4560cb4ae2 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/intersection-restated-members.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/intersection-restated-members.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/native-array-dynamic-reads/main.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/native-array-dynamic-reads/main.ts.json new file mode 100644 index 0000000000..c3b3013ea9 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/native-array-dynamic-reads/main.ts.json @@ -0,0 +1,7 @@ +{ + "order": [ + "/tests/corpus/native-array-dynamic-reads/reads.js", + "/tests/corpus/native-array-dynamic-reads/main.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/type-patches.test.ts b/packages/compiler/test/type-patches.test.ts index ddcd5b88db..5b586fb013 100644 --- a/packages/compiler/test/type-patches.test.ts +++ b/packages/compiler/test/type-patches.test.ts @@ -2,7 +2,7 @@ * packages: commander's HelpContext.helpWidth @typedef property and ms's * @return tags, patched from the fixtures' package.json; installed-version * selection; and the warnings for patches that do not apply as written. */ -import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { mkdirSync, mkdtempSync, readFileSync, rmSync, symlinkSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { afterEach, expect, test } from "vitest"; @@ -125,6 +125,101 @@ test("an @types twin is matched by the runtime package's installed version", () } }); +test("an @types twin in pnpm's layout is matched by the runtime package the project resolves", () => { + // pnpm keeps each package in its own directory under node_modules/.pnpm + // and links the project's dependencies from node_modules: the twin has no + // runtime package beside it. + const root = mkdtempSync(join(tmpdir(), "scriptc-type-patches-pnpm-")); + try { + const store = join(root, "node_modules/.pnpm"); + const runtime = join(store, "tallyjs@0.5.0/node_modules/tallyjs"); + const types = join(store, "@types+tallyjs@0.4.0/node_modules/@types/tallyjs"); + mkdirSync(runtime, { recursive: true }); + mkdirSync(types, { recursive: true }); + writeFileSync(join(runtime, "package.json"), '{ "name": "tallyjs", "version": "0.5.0" }'); + writeFileSync(join(runtime, "index.js"), "exports.totals = () => ({});\n"); + writeFileSync(join(types, "package.json"), '{ "name": "@types/tallyjs", "version": "0.4.0" }'); + writeFileSync( + join(types, "index.d.ts"), + "export type Totals = {\n max: number;\n};\nexport declare function totals(): Totals;\n", + ); + mkdirSync(join(root, "node_modules/@types"), { recursive: true }); + symlinkSync(runtime, join(root, "node_modules/tallyjs")); + symlinkSync(types, join(root, "node_modules/@types/tallyjs")); + const patchFor = (range: string) => + writeFileSync( + join(root, "package.json"), + JSON.stringify({ + scriptc: { + typePatches: { + [`tallyjs@${range}`]: { declarations: { "Totals.max": "| undefined" } }, + }, + }, + }), + ); + // A range the runtime (0.5.0) satisfies applies, though the twin's own + // version (0.4.0) does not satisfy it. + patchFor("0.5.0"); + configureTypePatches(join(root, "main.ts"), parse); + expect(typePatchedText(join(types, "index.d.ts"))).toContain("max: number | undefined;"); + // A range only the twin's own version satisfies does not. + patchFor("0.4.0"); + configureTypePatches(join(root, "main.ts"), parse); + expect(typePatchedText(join(types, "index.d.ts"))).toBeUndefined(); + } finally { + rmSync(root, { recursive: true, force: true }); + } +}); + +test("an export before its hoisted function is patched, and string text that looks like JSDoc is not", () => { + const root = mkdtempSync(join(tmpdir(), "scriptc-type-patches-js-")); + try { + const pkg = join(root, "node_modules/hoistkit"); + mkdirSync(pkg, { recursive: true }); + writeFileSync(join(pkg, "package.json"), '{ "name": "hoistkit", "version": "1.0.0" }'); + const source = [ + '"use strict";', + "exports.measure = size;", + 'exports.doc = "/** @typedef {object} Opts\\n * @property {number} width */";', + "", + "/**", + " * @returns {number}", + " */", + "function size(s) {", + " return s.length || undefined;", + "}", + "", + ].join("\n"); + writeFileSync(join(pkg, "index.js"), source); + writeFileSync( + join(root, "package.json"), + JSON.stringify({ + scriptc: { + typePatches: { + "hoistkit@^1.0.0": { + declarations: { "measure()": "| undefined", "Opts.width": "| undefined" }, + }, + }, + }, + }), + ); + configureTypePatches(join(root, "main.ts"), parse); + // The export, under another name, comes before the function it exports: + // the function's @returns tag widens. + expect(typePatchedText(join(pkg, "index.js"))).toBe( + source.replace(" * @returns {number}\n", " * @returns {number | undefined}\n"), + ); + // The string's text is a value, not a typedef: the patch finds nothing. + expect( + typePatchDiagnostics()! + .diagnostics.map((d) => d.message) + .join("\n"), + ).toContain("Opts.width"); + } finally { + rmSync(root, { recursive: true, force: true }); + } +}); + test("files replaces a declaration file and declarations still apply on top", () => { const root = join(fixtures, "npm/cases/type-patch-files"); configureTypePatches(join(root, "main.ts"), parse); diff --git a/packages/experimental/package.json b/packages/experimental/package.json index 583a2bc58a..4636be674f 100644 --- a/packages/experimental/package.json +++ b/packages/experimental/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/experimental", - "version": "0.4.4", + "version": "0.4.5", "description": "Experimental scriptc APIs, with Node.js implementations: lifetime regions for ProgramLifetime objects, intern tables shared between threads, and per-thread slots on shared objects", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/llvm-darwin-arm64/package.json b/packages/llvm-darwin-arm64/package.json index 76d5fdb72d..d1ac54f3ee 100644 --- a/packages/llvm-darwin-arm64/package.json +++ b/packages/llvm-darwin-arm64/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/llvm-darwin-arm64", - "version": "0.4.4", + "version": "0.4.5", "description": "Pinned LLVM code-generation helper for scriptc on macOS arm64", "license": "Apache-2.0 WITH LLVM-exception", "homepage": "https://scriptc.dev", diff --git a/packages/llvm-darwin-x64/package.json b/packages/llvm-darwin-x64/package.json index fbb755f4ae..b96a90cae1 100644 --- a/packages/llvm-darwin-x64/package.json +++ b/packages/llvm-darwin-x64/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/llvm-darwin-x64", - "version": "0.4.4", + "version": "0.4.5", "description": "Pinned LLVM code-generation helper for scriptc on macOS x64", "license": "Apache-2.0 WITH LLVM-exception", "homepage": "https://scriptc.dev", diff --git a/packages/llvm-linux-arm64-gnu/package.json b/packages/llvm-linux-arm64-gnu/package.json index 81fdc7d5be..49796f5229 100644 --- a/packages/llvm-linux-arm64-gnu/package.json +++ b/packages/llvm-linux-arm64-gnu/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/llvm-linux-arm64-gnu", - "version": "0.4.4", + "version": "0.4.5", "description": "Pinned LLVM code-generation helper for scriptc on Linux arm64", "license": "Apache-2.0 WITH LLVM-exception", "homepage": "https://scriptc.dev", diff --git a/packages/llvm-linux-arm64-musl/package.json b/packages/llvm-linux-arm64-musl/package.json index 8b3b2f1853..3001949c6b 100644 --- a/packages/llvm-linux-arm64-musl/package.json +++ b/packages/llvm-linux-arm64-musl/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/llvm-linux-arm64-musl", - "version": "0.4.4", + "version": "0.4.5", "description": "Statically linked LLVM code-generation helper for scriptc on Linux arm64 musl", "license": "Apache-2.0 WITH LLVM-exception", "homepage": "https://scriptc.dev", diff --git a/packages/llvm-linux-x64-gnu/package.json b/packages/llvm-linux-x64-gnu/package.json index 79b7eba3f8..11b375ba4f 100644 --- a/packages/llvm-linux-x64-gnu/package.json +++ b/packages/llvm-linux-x64-gnu/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/llvm-linux-x64-gnu", - "version": "0.4.4", + "version": "0.4.5", "description": "Pinned LLVM code-generation helper for scriptc on Linux x64", "license": "Apache-2.0 WITH LLVM-exception", "homepage": "https://scriptc.dev", diff --git a/packages/llvm-linux-x64-musl/package.json b/packages/llvm-linux-x64-musl/package.json index 5bffa5bef2..ffe1ca7ae2 100644 --- a/packages/llvm-linux-x64-musl/package.json +++ b/packages/llvm-linux-x64-musl/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/llvm-linux-x64-musl", - "version": "0.4.4", + "version": "0.4.5", "description": "Statically linked LLVM code-generation helper for scriptc on Linux x64 musl", "license": "Apache-2.0 WITH LLVM-exception", "homepage": "https://scriptc.dev", diff --git a/packages/llvm-win32-x64-msvc/package.json b/packages/llvm-win32-x64-msvc/package.json index 14c747a4a6..77e6fe6879 100644 --- a/packages/llvm-win32-x64-msvc/package.json +++ b/packages/llvm-win32-x64-msvc/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/llvm-win32-x64-msvc", - "version": "0.4.4", + "version": "0.4.5", "description": "Pinned LLVM code-generation helper for scriptc on Windows x64", "license": "Apache-2.0 WITH LLVM-exception", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-android-arm64/package.json b/packages/runtime-android-arm64/package.json index 1dfa1b177a..7aea1da1e2 100644 --- a/packages/runtime-android-arm64/package.json +++ b/packages/runtime-android-arm64/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-android-arm64", - "version": "0.4.4", + "version": "0.4.5", "description": "Precompiled scriptc library runtime pack for Android arm64", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-darwin-arm64/package.json b/packages/runtime-darwin-arm64/package.json index 32463357f5..ae068a00a6 100644 --- a/packages/runtime-darwin-arm64/package.json +++ b/packages/runtime-darwin-arm64/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-darwin-arm64", - "version": "0.4.4", + "version": "0.4.5", "description": "Precompiled scriptc runtime pack for macOS arm64", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-darwin-arm64/scripts/build.mjs b/packages/runtime-darwin-arm64/scripts/build.mjs index 7858c9782a..3305d69de8 100644 --- a/packages/runtime-darwin-arm64/scripts/build.mjs +++ b/packages/runtime-darwin-arm64/scripts/build.mjs @@ -366,7 +366,7 @@ async function build() { package: packageManifest.name, version: packageManifest.version, target: RUNTIME_PACK_MATRIX.target, - runtime_abi: { version: 44, marker: "scr_runtime_abi_v44" }, + runtime_abi: { version: 45, marker: "scr_runtime_abi_v45" }, compiler: { command: compiler, identity: compilerVersion, diff --git a/packages/runtime-darwin-x64/package.json b/packages/runtime-darwin-x64/package.json index c0577b04c7..7dbdaf6b9a 100644 --- a/packages/runtime-darwin-x64/package.json +++ b/packages/runtime-darwin-x64/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-darwin-x64", - "version": "0.4.4", + "version": "0.4.5", "description": "Precompiled scriptc runtime pack for macOS x64", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-ios-arm64/package.json b/packages/runtime-ios-arm64/package.json index 9c21342803..8a868a6063 100644 --- a/packages/runtime-ios-arm64/package.json +++ b/packages/runtime-ios-arm64/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-ios-arm64", - "version": "0.4.4", + "version": "0.4.5", "description": "Precompiled scriptc library runtime pack for iOS arm64", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-ios-simulator-arm64/package.json b/packages/runtime-ios-simulator-arm64/package.json index 6b7563a6af..011d41b39e 100644 --- a/packages/runtime-ios-simulator-arm64/package.json +++ b/packages/runtime-ios-simulator-arm64/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-ios-simulator-arm64", - "version": "0.4.4", + "version": "0.4.5", "description": "Precompiled scriptc library runtime pack for iOS simulator arm64", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-linux-arm64-gnu/package.json b/packages/runtime-linux-arm64-gnu/package.json index f0ae56929a..92a3192e7d 100644 --- a/packages/runtime-linux-arm64-gnu/package.json +++ b/packages/runtime-linux-arm64-gnu/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-linux-arm64-gnu", - "version": "0.4.4", + "version": "0.4.5", "description": "Precompiled scriptc runtime pack for Linux arm64 glibc", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-linux-arm64-musl/package.json b/packages/runtime-linux-arm64-musl/package.json index 79fa8e7dd6..5e751d8293 100644 --- a/packages/runtime-linux-arm64-musl/package.json +++ b/packages/runtime-linux-arm64-musl/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-linux-arm64-musl", - "version": "0.4.4", + "version": "0.4.5", "description": "Precompiled scriptc runtime pack for Linux arm64 musl", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-linux-x64-gnu/package.json b/packages/runtime-linux-x64-gnu/package.json index 45af8d2b74..63c045385f 100644 --- a/packages/runtime-linux-x64-gnu/package.json +++ b/packages/runtime-linux-x64-gnu/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-linux-x64-gnu", - "version": "0.4.4", + "version": "0.4.5", "description": "Precompiled scriptc runtime pack for Linux x64 glibc", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-linux-x64-musl/package.json b/packages/runtime-linux-x64-musl/package.json index fe9d11bcdd..a8602b3952 100644 --- a/packages/runtime-linux-x64-musl/package.json +++ b/packages/runtime-linux-x64-musl/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-linux-x64-musl", - "version": "0.4.4", + "version": "0.4.5", "description": "Precompiled scriptc runtime pack for Linux x64 musl", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-pack-common/scripts/build.mjs b/packages/runtime-pack-common/scripts/build.mjs index b770d81b65..8c0863b045 100644 --- a/packages/runtime-pack-common/scripts/build.mjs +++ b/packages/runtime-pack-common/scripts/build.mjs @@ -469,7 +469,7 @@ async function build() { package: packageManifest.name, version: packageManifest.version, target: matrix.target, - runtime_abi: { version: 44, marker: "scr_runtime_abi_v44" }, + runtime_abi: { version: 45, marker: "scr_runtime_abi_v45" }, compiler: { command: compiler, identity: compilerVersion, diff --git a/packages/runtime-wasm32-wasi/package.json b/packages/runtime-wasm32-wasi/package.json index ac2f4a5e99..799863cf06 100644 --- a/packages/runtime-wasm32-wasi/package.json +++ b/packages/runtime-wasm32-wasi/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-wasm32-wasi", - "version": "0.4.4", + "version": "0.4.5", "description": "Precompiled scriptc runtime pack for WASI Preview 1", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-win32-x64-msvc/package.json b/packages/runtime-win32-x64-msvc/package.json index 9a5731da28..421f1f71fb 100644 --- a/packages/runtime-win32-x64-msvc/package.json +++ b/packages/runtime-win32-x64-msvc/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-win32-x64-msvc", - "version": "0.4.4", + "version": "0.4.5", "description": "Precompiled scriptc runtime pack for Windows x64 MSVC", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime/package.json b/packages/runtime/package.json index 1d58b00e59..90140ab15a 100644 --- a/packages/runtime/package.json +++ b/packages/runtime/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime", - "version": "0.4.4", + "version": "0.4.5", "description": "scriptc native runtime sources and vendored dependencies", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime/src/scr_array.c b/packages/runtime/src/scr_array.c index 39998b8ffb..9d4884390b 100644 --- a/packages/runtime/src/scr_array.c +++ b/packages/runtime/src/scr_array.c @@ -269,11 +269,57 @@ static uint64_t scr_elem_retain_slot(const ScrArr *a, uint64_t slot) { return scr_elem_retain_with(a->elem, scr_arr_ops(a), slot); } +/* ToNumber and ToString of a foreign element. Both can run user code and + * pull in the dynamic tier, so they are reached through this table, which + * scr_arr_set_foreign installs: every foreign element is stored by it, and + * a program that never stores one links neither conversion. */ +typedef struct ScrArrForeignConversions { + double (*to_number)(const ScrDyn *); + ScrStr *(*to_string)(const ScrDyn *); +} ScrArrForeignConversions; + +static const ScrArrForeignConversions scr_arr_dyn_conversions = { + scr_dyn_number_coerce, + scr_dyn_string_coerce_js, +}; +static const ScrArrForeignConversions *scr_arr_conversions; + +static const ScrArrForeignConversions *scr_arr_foreign_conversions(void) { + return __atomic_load_n(&scr_arr_conversions, __ATOMIC_RELAXED); +} + +/* The array may hold SCR_ARR_FOREIGN elements (ScrArrExt.foreign). Such + * an array always has states. */ +static inline bool scr_arr_may_hold_foreign(const ScrArr *a) { + return SCR_AX(a)->foreign; +} + +/* `dst` takes slots of `src` (moved or retained): it may hold foreign + * elements too. Marking allocates the out-of-line storage, which can move + * the dense slots: mark before taking pointers into them. */ +static inline void scr_arr_inherit_foreign(ScrArr *dst, const ScrArr *src) { + if (scr_arr_may_hold_foreign(src)) scr_arr_ext(dst)->foreign = true; +} + static uint64_t scr_arr_retain_state_slot(const ScrArr *a, uint64_t slot, uint8_t state) { + if (state == SCR_ARR_FOREIGN) { + scr_dyn_retain(scr_slot_to_ptr(slot)); + return slot; + } return state == SCR_ARR_VALUE ? scr_elem_retain_slot(a, slot) : 0; } +/* Releases what a slot in `state` owns: a reference element, or the value + * of a foreign element. */ +static void scr_arr_release_state_slot(const ScrArr *a, uint64_t slot, uint8_t state) { + if (state == SCR_ARR_VALUE) { + if (scr_elem_is_ref(a->elem)) scr_elem_release(a, slot); + } else if (state == SCR_ARR_FOREIGN) { + scr_dyn_release(scr_slot_to_ptr(slot)); + } +} + /* ── storage and lifecycle ─────────────────────────────────────────────── */ typedef struct { @@ -981,6 +1027,7 @@ static inline bool scr_arr_slot_at(const ScrArr *a, size_t index, uint64_t *out) static void scr_arr_store_state_owned(ScrArr *a, size_t index, uint64_t slot, uint8_t state) { if (state != SCR_ARR_VALUE) scr_arr_cut(a, index); + if (state == SCR_ARR_FOREIGN) scr_arr_ext(a)->foreign = true; if (!scr_arr_has_states(a) && (state != SCR_ARR_VALUE || index != a->len)) scr_arr_add_states(a); scr_arr_grow_for_index(a, index); if (index < a->cap) { @@ -1034,6 +1081,7 @@ static void scr_arr_replace_state_owned(ScrArr *a, size_t index, uint64_t slot, uint8_t old_state; uint64_t old = 0; if (state != SCR_ARR_VALUE) scr_arr_cut(a, index); + if (state == SCR_ARR_FOREIGN) scr_arr_ext(a)->foreign = true; /* Without states, a value below the length replaces a value. */ if (!scr_arr_has_states(a) && (state != SCR_ARR_VALUE || index >= a->len)) { if (state == SCR_ARR_VALUE && index == a->len) { @@ -1044,7 +1092,7 @@ static void scr_arr_replace_state_owned(ScrArr *a, size_t index, uint64_t slot, } if (index < a->cap) { old_state = scr_arr_dense_state(a, index); - if (old_state == SCR_ARR_VALUE) old = scr_arr_dense_get(a, index); + if (old_state == SCR_ARR_VALUE || old_state == SCR_ARR_FOREIGN) old = scr_arr_dense_get(a, index); scr_arr_dense_put(a, index, slot); if (scr_arr_has_states(a)) *scr_arr_state_ptr(a, index) = state; } else { @@ -1058,7 +1106,7 @@ static void scr_arr_replace_state_owned(ScrArr *a, size_t index, uint64_t slot, SCR_AX(a)->sparse[pos].slot = slot; SCR_AX(a)->sparse[pos].state = state; } - if (old_state == SCR_ARR_VALUE && scr_elem_is_ref(a->elem)) scr_elem_release(a, old); + scr_arr_release_state_slot(a, old, old_state); } static void scr_arr_replace_owned(ScrArr *a, size_t index, uint64_t slot) { @@ -1300,6 +1348,7 @@ static void scr_arr_copy_dense(ScrArr *dst, size_t to, const ScrArr *src, size_t from, size_t count, bool reverse, bool materialize_holes) { if (count == 0) return; + scr_arr_inherit_foreign(dst, src); scr_arr_cut(dst, to); /* 8-byte slots: a constant slot size, and constant-width state ends. */ size_t width = scr_arr_slot_width(dst); @@ -1350,12 +1399,15 @@ static void scr_arr_copy_dense(ScrArr *dst, size_t to, const ScrArr *src, *scr_arr_state_in(dst_states, to + i) = values ? SCR_ARR_VALUE : *scr_arr_state_in(src_states, source); } } - if (!refs && (!materialize_holes || values)) return; + bool foreign = !values && scr_arr_may_hold_foreign(src); + if (!refs && !foreign && (!materialize_holes || values)) return; uint8_t elem = src->elem; const ScrArrOps *ops = scr_arr_ops(src); for (size_t i = to; i < to + count; i++) { if (*scr_arr_state_in(dst_states, i) == SCR_ARR_VALUE) { if (refs) dst->data[i] = scr_elem_retain_with(elem, ops, dst->data[i]); + } else if (*scr_arr_state_in(dst_states, i) == SCR_ARR_FOREIGN) { + scr_dyn_retain(scr_slot_to_ptr(dst->data[i])); } else if (materialize_holes && *scr_arr_state_in(dst_states, i) == SCR_ARR_HOLE) { scr_arr_dense_put(dst, i, 0); *scr_arr_state_in(dst_states, i) = SCR_ARR_UNDEFINED; @@ -1477,11 +1529,13 @@ void scr_arr_trace_v(void *a0, ScrTraceVisit visit, void *ctx) { } else { const uint8_t *states = scr_arr_states_end(a); for (size_t i = 0; i < end; i++) { - if (*scr_arr_state_in(states, i) == SCR_ARR_VALUE) visit(scr_slot_to_ptr(a->data[i]), ctx); + uint8_t state = *scr_arr_state_in(states, i); + if (state == SCR_ARR_VALUE || state == SCR_ARR_FOREIGN) visit(scr_slot_to_ptr(a->data[i]), ctx); } } for (size_t i = 0; i < SCR_AX(a)->sparse_len; i++) { - if (SCR_AX(a)->sparse[i].state == SCR_ARR_VALUE) { + uint8_t state = SCR_AX(a)->sparse[i].state; + if (state == SCR_ARR_VALUE || state == SCR_ARR_FOREIGN) { visit(scr_slot_to_ptr(SCR_AX(a)->sparse[i].slot), ctx); } } @@ -1590,11 +1644,37 @@ static inline __attribute__((always_inline)) void scr_arr_free(ScrArr *a) { scr_mem_free(a); } +/* Releases the values of foreign elements and leaves holes in their place, + * so the release below sees only typed elements. Arrays that hold foreign + * elements have states; a copy carries the mark of its source whatever + * range it takes, so an array with the mark and no states holds only + * values in its dense slots. */ +static void scr_arr_release_foreign(ScrArr *a) { + uint8_t *states = scr_arr_has_states(a) ? scr_arr_states_end(a) : NULL; + for (size_t i = 0, end = states ? (a->len < a->cap ? a->len : a->cap) : 0; i < end; i++) { + if (*scr_arr_state_in(states, i) != SCR_ARR_FOREIGN) continue; + uint64_t old = a->data[i]; + *scr_arr_state_in(states, i) = SCR_ARR_HOLE; + a->data[i] = 0; + scr_dyn_release(scr_slot_to_ptr(old)); + } + ScrArrExt *x = scr_arr_ext_of(a); + for (size_t i = 0; i < x->sparse_len; i++) { + if (x->sparse[i].state != SCR_ARR_FOREIGN) continue; + uint64_t old = x->sparse[i].slot; + x->sparse[i].state = SCR_ARR_HOLE; + x->sparse[i].slot = 0; + scr_dyn_release(scr_slot_to_ptr(old)); + } + x->foreign = false; +} + static void scr_arr_destroy(void *object) { ScrArr *a = object; if (scr_arr_ext_of(a)) { scr_dyn_release(scr_arr_ext_of(a)->metadata); scr_arr_ext_of(a)->metadata = NULL; + if (scr_arr_ext_of(a)->foreign) scr_arr_release_foreign(a); } if (scr_elem_is_ref(a->elem)) { size_t end = a->len < a->cap ? a->len : a->cap; @@ -1898,14 +1978,14 @@ void scr_arr_set_len(ScrArr *a, double length) { } return; } - if (scr_elem_is_ref(a->elem)) { + if (scr_elem_is_ref(a->elem) || scr_arr_may_hold_foreign(a)) { for (size_t i = dense_stop; i < dense_end; i++) { if (*scr_arr_state_ptr(a, i) != SCR_ARR_HOLE) { uint64_t old = a->data[i]; uint8_t state = *scr_arr_state_ptr(a, i); *scr_arr_state_ptr(a, i) = 0; a->data[i] = 0; - if (state == SCR_ARR_VALUE) scr_elem_release(a, old); + scr_arr_release_state_slot(a, old, state); } } if (scr_arr_ext_of(a)) { @@ -1915,7 +1995,7 @@ void scr_arr_set_len(ScrArr *a, double length) { uint64_t old = scr_arr_ext_of(a)->sparse[i].slot; uint8_t state = scr_arr_ext_of(a)->sparse[i].state; scr_arr_ext_of(a)->sparse_len = i; - if (state == SCR_ARR_VALUE) scr_elem_release(a, old); + scr_arr_release_state_slot(a, old, state); } } } else { @@ -1949,7 +2029,7 @@ void scr_arr_copy_index(ScrArr *dst, size_t dst_index, const ScrArr *src, uint64_t slot; uint8_t state = scr_arr_state_at(src, src_index, &slot); if (state == SCR_ARR_HOLE) return; - if (state == SCR_ARR_VALUE) slot = scr_elem_retain_slot(src, slot); + slot = scr_arr_retain_state_slot(src, slot, state); scr_arr_replace_state_owned(dst, dst_index, slot, state); } @@ -2056,7 +2136,7 @@ ScrArr *scr_arr_copy_within(ScrArr *a, double target, double start, double end) if (state == SCR_ARR_HOLE) { scr_arr_delete(a, (double)(to + offset)); } else { - if (state == SCR_ARR_VALUE) slot = scr_elem_retain_slot(a, slot); + slot = scr_arr_retain_state_slot(a, slot, state); /* Each replacement publishes its edge before releasing the previous * owner; cycle collection can run between successive positions. */ scr_arr_replace_state_owned(a, to + offset, slot, state); @@ -2074,7 +2154,7 @@ static ScrArr *scr_arr_fill_slot(ScrArr *a, uint64_t slot, uint8_t state, bool refs = scr_elem_is_ref(a->elem); if (scr_arr_dense_ok(a, until, until - from)) scr_arr_grow_dense(a, until); if (state != SCR_ARR_VALUE) scr_arr_need_states(a); - if (!refs && until <= a->cap) { + if (!refs && until <= a->cap && !scr_arr_may_hold_foreign(a)) { if (state != SCR_ARR_VALUE) scr_arr_cut(a, from); if (a->elem == SCR_ELEM_BOOL) memset(scr_arr_dense_at(a, from), (int)slot, until - from); else if (a->elem == SCR_ELEM_CREF) for (size_t i = from; i < until; i++) scr_arr_dense_put(a, i, slot); @@ -2114,7 +2194,63 @@ ScrArr *scr_arr_fill_undefined(ScrArr *a, double start, double end) { * The JS fold exactly (ECMA Math.max/min applied to the elements): any * NaN poisons the result, +0 beats -0 for max (the reverse for min), and * the empty array yields the zero-argument constants. Borrows the array. */ +/* ToNumber of every element in order, as Math.max/min/hypot apply it to + * their spread arguments before comparing; for arrays that may hold + * foreign elements, whose conversion can call user code. As in the spread, + * the elements are taken first: the foreign values are retained before + * any conversion runs, so a conversion that changes the array changes + * neither the count (*count) nor the values. NULL with the exception + * pending when a conversion throws. +1 malloc'd buffer. */ +static double *scr_arr_coerced_numbers(const ScrArr *a, size_t *count) { + size_t n = a->len; + /* A sparse array's length can exceed what the buffers can count. */ + if (n > SIZE_MAX / sizeof(double)) scr_arr_oom(); + double *out = malloc((n ? n : 1) * sizeof(*out)); + ScrDyn **held = calloc(n ? n : 1, sizeof(*held)); + if (!out || !held) scr_arr_oom(); + for (size_t i = 0; i < n; i++) { + uint64_t slot; + uint8_t state = scr_arr_f64_state_at(a, i, &slot); + if (state == SCR_ARR_FOREIGN) held[i] = scr_dyn_retain(scr_slot_to_ptr(slot)); + out[i] = state == SCR_ARR_VALUE ? scr_slot_to_f64(slot) : NAN; + } + bool failed = false; + for (size_t i = 0; i < n; i++) { + if (!held[i]) continue; + if (!failed) { + out[i] = scr_arr_foreign_conversions()->to_number(held[i]); + failed = scr_exc_pending(); + } + scr_dyn_release(held[i]); + } + free(held); + if (failed) { + free(out); + return NULL; + } + *count = n; + return out; +} + +static double scr_math_extreme_foreign(ScrArr *a, bool max) { + size_t n; + double *values = scr_arr_coerced_numbers(a, &n); + if (!values) return NAN; + double best = max ? -INFINITY : INFINITY; + bool nan = false; + for (size_t i = 0; i < n; i++) { + double v = values[i]; + if (isnan(v)) nan = true; + else if (max ? (v > best || (v == 0.0 && best == 0.0 && !signbit(v))) + : (v < best || (v == 0.0 && best == 0.0 && signbit(v)))) + best = v; + } + free(values); + return nan ? NAN : best; +} + double scr_math_max_arr(ScrArr *a) { + if (scr_arr_may_hold_foreign(a)) return scr_math_extreme_foreign(a, true); double best = -INFINITY; for (size_t i = 0; i < a->len; i++) { uint64_t slot; @@ -2127,6 +2263,7 @@ double scr_math_max_arr(ScrArr *a) { } double scr_math_min_arr(ScrArr *a) { + if (scr_arr_may_hold_foreign(a)) return scr_math_extreme_foreign(a, false); double best = INFINITY; for (size_t i = 0; i < a->len; i++) { uint64_t slot; @@ -2143,30 +2280,43 @@ double scr_math_min_arr(ScrArr *a) { double scr_math_hypot_arr(ScrArr *a) { double scale = 0.0; bool has_nan = false; - for (size_t i = 0; i < a->len; i++) { + double *coerced = NULL; + size_t n = a->len; + if (scr_arr_may_hold_foreign(a) && !(coerced = scr_arr_coerced_numbers(a, &n))) return NAN; + for (size_t i = 0; i < n; i++) { uint64_t slot; - if (scr_arr_f64_state_at(a, i, &slot) != SCR_ARR_VALUE) { + double v; + if (coerced) { + v = fabs(coerced[i]); + } else if (scr_arr_f64_state_at(a, i, &slot) != SCR_ARR_VALUE) { has_nan = true; continue; + } else { + v = fabs(scr_slot_to_f64(slot)); + } + if (isinf(v)) { + free(coerced); + return INFINITY; } - double v = fabs(scr_slot_to_f64(slot)); - if (isinf(v)) return INFINITY; if (isnan(v)) has_nan = true; else if (v > scale) scale = v; } - if (has_nan) return NAN; - if (scale == 0.0) return 0.0; + if (has_nan || scale == 0.0) { + free(coerced); + return has_nan ? NAN : 0.0; + } double sum = 0.0; double correction = 0.0; - for (size_t i = 0; i < a->len; i++) { - uint64_t slot; - (void)scr_arr_f64_state_at(a, i, &slot); - double v = scr_slot_to_f64(slot) / scale; + for (size_t i = 0; i < n; i++) { + uint64_t slot = 0; + if (!coerced) (void)scr_arr_f64_state_at(a, i, &slot); + double v = (coerced ? coerced[i] : scr_slot_to_f64(slot)) / scale; double term = v * v - correction; double next = sum + term; correction = (next - sum) - term; sum = next; } + free(coerced); return scale * sqrt(sum); } @@ -2211,6 +2361,13 @@ static uint64_t scr_arr_require_index(const ScrArr *a, size_t idx, double i) { scr_trap_fmt("scriptc: RangeError: array index %s is undefined (length %zu)\n", buf, a->len); } + if (state == SCR_ARR_FOREIGN) { + char buf[32]; + scr_f64_to_str(i, buf); + scr_trap_fmt("scriptc: RangeError: array index %s holds a value outside its element type " + "(length %zu)\n", + buf, a->len); + } return slot; } @@ -2218,16 +2375,20 @@ double scr_arr_get_f64(ScrArr *a, double i) { return scr_slot_to_f64(scr_arr_require_slot(a, i)); } -static bool scr_arr_read_value(const ScrArr *a, double i, uint64_t *slot) { +/* The state of a[i] for any numeric key (a hole when missing), with its + * slot. */ +static uint8_t scr_arr_read_state(const ScrArr *a, double i, uint64_t *slot) { size_t idx; uint8_t state; if (scr_arr_valid_index(i, &idx)) { - if (idx >= a->len) return false; - state = scr_arr_state_at(a, idx, slot); - } else if (!scr_arr_prop_get_state(a, i, slot, &state)) { - return false; + if (idx >= a->len) return SCR_ARR_HOLE; + return scr_arr_state_at(a, idx, slot); } - return state == SCR_ARR_VALUE; + return scr_arr_prop_get_state(a, i, slot, &state) ? state : SCR_ARR_HOLE; +} + +static bool scr_arr_read_value(const ScrArr *a, double i, uint64_t *slot) { + return scr_arr_read_state(a, i, slot) == SCR_ARR_VALUE; } void *scr_arr_peek_ref(const ScrArr *a, double i) { @@ -2257,6 +2418,13 @@ double scr_arr_get_number(const ScrArr *a, double i) { } else if (!scr_arr_prop_get_state(a, i, &slot, &state)) { return NAN; } + if (state == SCR_ARR_FOREIGN) { + /* The conversion can call user code that removes the element. */ + ScrDyn *value = scr_dyn_retain(scr_slot_to_ptr(slot)); + double number = scr_arr_foreign_conversions()->to_number(value); + scr_dyn_release(value); + return number; + } return state == SCR_ARR_VALUE ? scr_slot_to_f64(slot) : NAN; } @@ -2267,14 +2435,26 @@ bool scr_arr_index_eq(const ScrArr *a, double i, const ScrArr *b, double j) { if (i >= 0 && i < (double)a->cap && i == trunc(i) && j >= 0 && j < (double)b->cap && j == trunc(j)) { size_t ai = (size_t)i, bi = (size_t)j; - bool ap = ai < a->len && scr_arr_dense_state(a, ai) == SCR_ARR_VALUE; - bool bp = bi < b->len && scr_arr_dense_state(b, bi) == SCR_ARR_VALUE; + uint8_t as = ai < a->len ? scr_arr_dense_state(a, ai) : SCR_ARR_HOLE; + uint8_t bs = bi < b->len ? scr_arr_dense_state(b, bi) : SCR_ARR_HOLE; + if (as == SCR_ARR_FOREIGN || bs == SCR_ARR_FOREIGN) { + /* A foreign element is never a value of the element type, undefined + * or a hole. */ + return as == bs && scr_dyn_strict_eq(scr_slot_to_ptr(scr_arr_dense_get(a, ai)), + scr_slot_to_ptr(scr_arr_dense_get(b, bi))); + } + bool ap = as == SCR_ARR_VALUE; + bool bp = bs == SCR_ARR_VALUE; if (!ap || !bp) return ap == bp; av = scr_arr_dense_get(a, ai); bv = scr_arr_dense_get(b, bi); } else { - bool ap = scr_arr_read_value(a, i, &av); - bool bp = scr_arr_read_value(b, j, &bv); + uint8_t as = scr_arr_read_state(a, i, &av); + uint8_t bs = scr_arr_read_state(b, j, &bv); + if (as == SCR_ARR_FOREIGN || bs == SCR_ARR_FOREIGN) + return as == bs && scr_dyn_strict_eq(scr_slot_to_ptr(av), scr_slot_to_ptr(bv)); + bool ap = as == SCR_ARR_VALUE; + bool bp = bs == SCR_ARR_VALUE; if (!ap || !bp) return ap == bp; } if (a->elem == SCR_ELEM_F64) return scr_slot_to_f64(av) == scr_slot_to_f64(bv); @@ -2349,6 +2529,44 @@ void scr_arr_set_undefined(ScrArr *a, double i) { scr_arr_replace_state_owned(a, idx, 0, SCR_ARR_UNDEFINED); } +bool scr_arr_set_foreign(ScrArr *a, double i, ScrDyn *v) { + size_t idx; + if (!scr_arr_valid_index(i, &idx)) return false; + if (v->kind == SCR_DYN_UNDEF) { + scr_arr_set_undefined(a, i); + return true; + } + /* A foreign element keeps a pointer in its slot: int32 slots widen to + * doubles first, and 1-byte and compact slots cannot hold one. */ + if (a->elem == SCR_ELEM_I32) scr_arr_widen_i32(a); + if (scr_arr_slot_width(a) != sizeof(uint64_t)) return false; + __atomic_store_n(&scr_arr_conversions, &scr_arr_dyn_conversions, __ATOMIC_RELAXED); + scr_dyn_retain(v); + scr_arr_need_states(a); + if (idx >= a->len) a->len = idx + 1; + scr_arr_replace_state_owned(a, idx, scr_slot_from_ptr(v), SCR_ARR_FOREIGN); + return true; +} + +ScrDyn *scr_arr_get_foreign(const ScrArr *a, double i) { + size_t idx; + uint64_t slot; + if (!scr_arr_valid_index(i, &idx) || idx >= a->len) return NULL; + if (scr_arr_state_at(a, idx, &slot) != SCR_ARR_FOREIGN) return NULL; + return scr_dyn_retain(scr_slot_to_ptr(slot)); +} + +void scr_arr_foreign_throw(const ScrArr *a, double i, const ScrStr *want) { + ScrDyn *value = scr_arr_get_foreign(a, i); + scr_dyn_check_fail(NULL, want->data, value ? value : scr_dyn_undefined()); + scr_dyn_release(value); +} + +void scr_dyn_foreign_throw(ScrDyn *value, const char *want) { + scr_dyn_check_fail(NULL, want, value); + scr_dyn_release(value); +} + bool scr_arr_delete(ScrArr *a, double i) { size_t idx; if (!scr_arr_valid_index(i, &idx)) return scr_arr_prop_delete(a, i); @@ -2356,9 +2574,7 @@ bool scr_arr_delete(ScrArr *a, double i) { uint64_t old; uint8_t state; if (!scr_arr_take_state(a, idx, &old, &state)) return false; - if (state == SCR_ARR_VALUE && scr_elem_is_ref(a->elem)) { - scr_elem_release(a, old); - } + scr_arr_release_state_slot(a, old, state); return true; } @@ -2429,8 +2645,8 @@ double scr_arr_push_spread(ScrArr *a, const ScrArr *src) { uint64_t slot; uint8_t state = scr_arr_state_at_storage(&old, i, &slot); if (state != SCR_ARR_HOLE) scr_arr_store_state_owned(a, i, slot, state); - if (state == SCR_ARR_VALUE) { - scr_arr_store_owned(a, old_len + i, scr_elem_retain_slot(a, slot)); + if (state == SCR_ARR_VALUE || state == SCR_ARR_FOREIGN) { + scr_arr_store_state_owned(a, old_len + i, scr_arr_retain_state_slot(a, slot, state), state); } else { scr_arr_store_state_owned(a, old_len + i, 0, SCR_ARR_UNDEFINED); } @@ -2445,8 +2661,8 @@ double scr_arr_push_spread(ScrArr *a, const ScrArr *src) { for (size_t i = 0; i < add; i++) { uint64_t slot; uint8_t state = scr_arr_state_at(src, i, &slot); - if (state == SCR_ARR_VALUE) { - scr_arr_store_owned(a, old_len + i, scr_elem_retain_slot(src, slot)); + if (state == SCR_ARR_VALUE || state == SCR_ARR_FOREIGN) { + scr_arr_store_state_owned(a, old_len + i, scr_arr_retain_state_slot(src, slot, state), state); } else { scr_arr_store_state_owned(a, old_len + i, 0, SCR_ARR_UNDEFINED); } @@ -2474,11 +2690,8 @@ double scr_arr_concat_copy(ScrArr *a, const ScrArr *src) { uint64_t slot; uint8_t state = scr_arr_state_at_storage(&old, i, &slot); if (state != SCR_ARR_HOLE) scr_arr_store_state_owned(a, i, slot, state); - if (state == SCR_ARR_VALUE) { - scr_arr_store_owned(a, old_len + i, scr_elem_retain_slot(a, slot)); - } else if (state == SCR_ARR_UNDEFINED) { - scr_arr_store_state_owned(a, old_len + i, 0, state); - } + if (state != SCR_ARR_HOLE) + scr_arr_store_state_owned(a, old_len + i, scr_arr_retain_state_slot(a, slot, state), state); } a->len = old_len + add; scr_arr_free_storage(&old); @@ -2659,8 +2872,9 @@ double scr_arr_unshift_spread(ScrArr *a, const ScrArr *src) { uint8_t state = src == a ? scr_arr_state_at_storage(&old, i, &slot) : scr_arr_state_at(src, i, &slot); - if (state == SCR_ARR_VALUE) { - scr_arr_store_owned(a, i, scr_arr_retain_state_slot(src == a ? a : src, slot, state)); + if (state == SCR_ARR_VALUE || state == SCR_ARR_FOREIGN) { + scr_arr_store_state_owned(a, i, scr_arr_retain_state_slot(src == a ? a : src, slot, state), + state); } else { scr_arr_store_state_owned(a, i, 0, SCR_ARR_UNDEFINED); } @@ -2769,7 +2983,8 @@ uint8_t scr_arr_shift_state(ScrArr *a, uint64_t *slot_out) { size_t old_len = a->len; if (scr_arr_is_dense(a)) { uint8_t state = scr_arr_dense_state(a, 0); - if (slot_out && state == SCR_ARR_VALUE) *slot_out = scr_arr_dense_get(a, 0); + if (slot_out && (state == SCR_ARR_VALUE || state == SCR_ARR_FOREIGN)) + *slot_out = scr_arr_dense_get(a, 0); /* The first slot becomes head room; the tail stays where it is. */ if (scr_arr_drop_front(a, 1)) return state; /* The prefix slides down with the tail. */ @@ -2842,6 +3057,7 @@ ScrArr *scr_arr_splice(ScrArr *a, double start, double deleteCount) { size_t old_len = a->len; if (scr_arr_is_dense(a)) { if (n) { + scr_arr_inherit_foreign(out, a); /* The removed elements keep their states unless all are values. */ if (!scr_arr_dense_run_values(a, from, n)) scr_arr_add_states(out); memcpy(out->data, scr_arr_dense_at(a, from), n * scr_arr_slot_width(a)); @@ -2927,6 +3143,7 @@ ScrArr *scr_arr_splice_insert(ScrArr *a, double start, double deleteCount, ? scr_arr_new_ops(scr_arr_ops(a), n) : scr_arr_new(a->elem, n); if (n) { + scr_arr_inherit_foreign(removed, a); if (!scr_arr_dense_run_values(a, from, n)) scr_arr_add_states(removed); memcpy(removed->data, scr_arr_dense_at(a, from), n * scr_arr_slot_width(a)); if (scr_arr_has_states(removed)) @@ -2981,22 +3198,30 @@ void scr_arr_splice_drop(ScrArr *a, double start, double deleteCount, size_t cou } /* Removed slots, kept until the receiver is rewritten. */ uint64_t inline_removed[8]; + uint8_t inline_states[8]; uint64_t *removed = inline_removed; + uint8_t *removed_states = inline_states; size_t kept = 0; - if (refs && n > 0) { + if ((refs || scr_arr_may_hold_foreign(a)) && n > 0) { if (n > 8) { removed = scr_mem_alloc(n * sizeof(uint64_t)); - if (!removed) scr_arr_oom(); + removed_states = scr_mem_alloc(n); + if (!removed || !removed_states) scr_arr_oom(); } for (size_t i = from; i < from + n; i++) { - if (scr_arr_dense_state(a, i) == SCR_ARR_VALUE) removed[kept++] = a->data[i]; + uint8_t state = scr_arr_dense_state(a, i); + if ((refs && state == SCR_ARR_VALUE) || state == SCR_ARR_FOREIGN) { + removed_states[kept] = state; + removed[kept++] = a->data[i]; + } } } size_t old_len = a->len; if (from == 0 && count == 0 && scr_arr_drop_front(a, n)) { /* A removed prefix becomes head room. */ - for (size_t i = 0; i < kept; i++) scr_elem_release(a, removed[i]); + for (size_t i = 0; i < kept; i++) scr_arr_release_state_slot(a, removed[i], removed_states[i]); if (removed != inline_removed) scr_mem_free(removed); + if (removed_states != inline_states) scr_mem_free(removed_states); return; } scr_arr_grow_dense(a, next_len); @@ -3011,8 +3236,9 @@ void scr_arr_splice_drop(ScrArr *a, double start, double deleteCount, size_t cou /* Values inserted into a hole-free array keep it hole-free. */ if (packed) a->filled = (uint32_t)next_len; else scr_arr_refill(a); - for (size_t i = 0; i < kept; i++) scr_elem_release(a, removed[i]); + for (size_t i = 0; i < kept; i++) scr_arr_release_state_slot(a, removed[i], removed_states[i]); if (removed != inline_removed) scr_mem_free(removed); + if (removed_states != inline_states) scr_mem_free(removed_states); } /* FlattenIntoArray for static arrays: a dense copy with depth zero, or one @@ -3236,7 +3462,10 @@ typedef struct { typedef struct { ScrElemKind kind; const ScrNumberSortKey *numbers; - ScrStr *const *texts; /* reference elements: each element's String() text */ + /* When set, values are indices into these string keys: each element's + * String() text (reference elements such as Dates, and the elements of an + * array that may hold foreign elements). */ + ScrStr *const *texts; } ScrPrimitiveOrder; static int scr_arr_primitive_compare(uint64_t left, uint64_t right, @@ -3329,6 +3558,110 @@ static void scr_arr_primitive_order(uint64_t *values, size_t count, if (runs != local_runs) free(runs); } +/* The default ordering of an array that may hold foreign elements: present + * elements sort by their string conversions, then present undefined + * values, then holes (toSorted reads them as undefined). The conversions + * may call user code, so the elements are taken first, each retained, as + * the sort's element list is; a throw leaves the array unchanged. */ +static ScrArr *scr_arr_sort_foreign(ScrArr *a, bool copy) { + size_t length = a->len; + size_t dense = a->len < a->cap ? a->len : a->cap; + size_t cap = dense + SCR_AX(a)->sparse_len; + if (cap > SIZE_MAX / sizeof(uint64_t)) scr_arr_oom(); + uint64_t *slots = cap ? malloc(cap * sizeof(*slots)) : NULL; + uint8_t *states = cap ? malloc(cap) : NULL; + ScrStr **keys = cap ? malloc(cap * sizeof(*keys)) : NULL; + uint64_t *order_of = cap ? malloc(cap * sizeof(*order_of)) : NULL; + if (cap && (!slots || !states || !keys || !order_of)) scr_arr_oom(); + size_t count = 0, undefined_count = 0; + for (size_t at = 0; at < cap; at++) { + uint64_t slot; + uint8_t state; + if (at < dense) { + state = scr_arr_dense_state(a, at); + slot = scr_arr_dense_get(a, at); + } else { + state = SCR_AX(a)->sparse[at - dense].state; + slot = SCR_AX(a)->sparse[at - dense].slot; + } + if (state == SCR_ARR_UNDEFINED) undefined_count++; + if (state != SCR_ARR_VALUE && state != SCR_ARR_FOREIGN) continue; + slots[count] = scr_arr_retain_state_slot(a, slot, state); + states[count] = state; + count++; + } + bool failed = false; + size_t keyed = 0; + for (; keyed < count; keyed++) { + uint64_t slot = slots[keyed]; + ScrStr *key; + if (states[keyed] == SCR_ARR_FOREIGN) { + key = scr_arr_foreign_conversions()->to_string(scr_slot_to_ptr(slot)); + } else if (a->elem == SCR_ELEM_STR) { + key = scr_str_retain(scr_slot_to_ptr(slot)); + } else { + char text[32]; + size_t n = scr_f64_to_str(scr_slot_to_f64(slot), text); + key = scr_str_new(text, n); + } + if (!key) { + failed = true; + break; + } + keys[keyed] = key; + order_of[keyed] = keyed; + } + ScrArr *out = a; + if (!failed) { + ScrPrimitiveOrder order = {a->elem, NULL, keys}; + scr_arr_primitive_order(order_of, count, &order); + if (copy) { + /* The values, then present undefined for every other index. */ + out = scr_arr_has_ops(a) ? scr_arr_new_ops(scr_arr_ops(a), length) + : scr_arr_new(a->elem, length); + for (size_t i = 0; i < length; i++) { + if (i < count) scr_arr_store_state_owned(out, i, slots[order_of[i]], states[order_of[i]]); + else scr_arr_store_state_owned(out, i, 0, SCR_ARR_UNDEFINED); + out->len = (uint32_t)(i + 1); + } + } else { + /* The values, then the present undefined values; holes after them + * up to the sorted length. Elements a conversion added at or past + * that length stay, and the array keeps the longer of its current + * length and the sorted count, as in Node. The taken elements own + * their references; release the ones the array holds now below the + * sorted length, which a conversion may have changed. */ + size_t now = a->len; + size_t sorted = count + undefined_count; + ScrArrStorage old = scr_arr_take_storage(a); + a->len = (uint32_t)(now > sorted ? now : sorted); + for (size_t i = 0; i < count; i++) + scr_arr_store_state_owned(a, i, slots[order_of[i]], states[order_of[i]]); + for (size_t i = count; i < sorted; i++) scr_arr_store_state_owned(a, i, 0, SCR_ARR_UNDEFINED); + size_t old_dense = now < old.cap ? now : old.cap; + for (size_t i = 0; i < old_dense + old.sparse_len; i++) { + size_t index = i < old_dense ? i : old.sparse[i - old_dense].index; + uint64_t slot = 0; + uint8_t state = scr_arr_state_at_storage(&old, index, &slot); + if (state == SCR_ARR_HOLE) continue; + if (index < length) scr_arr_release_state_slot(a, slot, state); + else scr_arr_store_state_owned(a, index, slot, state); + } + scr_arr_free_storage(&old); + } + scr_arr_refill(out); + } else { + for (size_t i = 0; i < count; i++) scr_arr_release_state_slot(a, slots[i], states[i]); + } + for (size_t i = 0; i < keyed; i++) scr_str_release(keys[i]); + free(slots); + free(states); + free(keys); + free(order_of); + if (failed) return copy ? scr_arr_new(a->elem, 0) : scr_arr_retain(a); + return copy ? out : scr_arr_retain(a); +} + static ScrArr *scr_arr_sort_default(ScrArr *a, bool copy, ScrStr *(*text)(void *)); static ScrArr *scr_arr_new_like(const ScrArr *a, size_t cap) { @@ -3338,6 +3671,7 @@ static ScrArr *scr_arr_new_like(const ScrArr *a, size_t cap) { ScrArr *scr_arr_sort_primitive(ScrArr *a, bool copy) { if (a->elem != SCR_ELEM_STR && a->elem != SCR_ELEM_F64 && a->elem != SCR_ELEM_BOOL) scr_trap("scriptc: invalid primitive array ordering\n"); + if (a->len > 0 && scr_arr_may_hold_foreign(a)) return scr_arr_sort_foreign(a, copy); return scr_arr_sort_default(a, copy, NULL); } @@ -5190,7 +5524,8 @@ bool scr_arr_sort_dense(ScrArr *a, ScrClosure *f, uint32_t arity, bool owned, in for (size_t i = 0; i < n; i++) scr_elem_release_with(elem, ops, src[i]); scr_array_frozen_error(0, a->len, 0, 0, 0); } else if (a->len >= n && a->cap >= n) { - if (!retained) { + /* The comparator may have stored foreign elements through a view. */ + if (!retained && !scr_arr_may_hold_foreign(a)) { memcpy(a->data, src, n * sizeof(*src)); if (scr_arr_has_states(a)) memset(scr_arr_state_run(a, 0, n), SCR_ARR_VALUE, n); } else { @@ -5198,11 +5533,11 @@ bool scr_arr_sort_dense(ScrArr *a, ScrClosure *f, uint32_t arity, bool owned, in * release can collect cycles. The buffer still holds the slots not * yet written. */ for (size_t i = 0; i < n; i++) { - bool held = scr_arr_dense_state(a, i) == SCR_ARR_VALUE; - uint64_t old = held ? a->data[i] : 0; + uint8_t old_state = scr_arr_dense_state(a, i); + uint64_t old = old_state == SCR_ARR_VALUE || old_state == SCR_ARR_FOREIGN ? a->data[i] : 0; a->data[i] = src[i]; if (scr_arr_has_states(a)) *scr_arr_state_ptr(a, i) = SCR_ARR_VALUE; - if (held) scr_elem_release(a, old); + scr_arr_release_state_slot(a, old, old_state); } } } else { @@ -5270,10 +5605,28 @@ ScrStr *scr_arr_join(ScrArr *a, ScrStr *sep) { * across its seam; the finished string then needs fusing. */ bool lone = scr_wtf8_any(); bool lead_low = lone && a->len > 1 && scr_wtf8_low_at(sep->data); - for (size_t i = 0; i < a->len; i++) { + /* The length is read once: a conversion that shortens the array leaves + * the indices past its new end empty. */ + size_t length = a->len; + for (size_t i = 0; i < length; i++) { if (i > 0) scr_join_append(&out, sep->data, sep->len); + if (i >= a->len) continue; uint64_t slot; uint8_t state = scr_arr_state_at(a, i, &slot); + if (state == SCR_ARR_FOREIGN) { + /* join converts every element but undefined and null with ToString, + * which can call user code that removes the element; a throw ends + * the join with the exception pending. */ + ScrDyn *value = scr_slot_to_ptr(slot); + if (value->kind == SCR_DYN_UNDEF || value->kind == SCR_DYN_NULL) continue; + scr_dyn_retain(value); + ScrStr *text = scr_arr_foreign_conversions()->to_string(value); + scr_dyn_release(value); + if (!text) break; + scr_join_append(&out, text->data, text->len); + scr_str_release(text); + continue; + } if (state != SCR_ARR_VALUE) continue; switch (a->elem) { case SCR_ELEM_F64: { diff --git a/packages/runtime/src/scr_array.h b/packages/runtime/src/scr_array.h index 48fa4f26f0..fab85fff1b 100644 --- a/packages/runtime/src/scr_array.h +++ b/packages/runtime/src/scr_array.h @@ -60,6 +60,10 @@ typedef enum { SCR_ARR_HOLE = 0, SCR_ARR_VALUE = 1, SCR_ARR_UNDEFINED = 2, + /* A present element outside the element type, stored through a dynamic + * view: the slot holds a retained ScrDyn * (scr_arr_set_foreign). Arrays + * that hold one mark their out-of-line storage (ScrArrExt.foreign). */ + SCR_ARR_FOREIGN = 3, } ScrArrState; /* The RC entry points of an SCR_ELEM_REF array's element type: one constant @@ -171,6 +175,9 @@ typedef struct ScrArrExt { /* The element ops of an array whose ops have no registry id * (SCR_ARR_OPS_OVERFLOW). */ const ScrArrOps *ops; + /* Some slot may hold SCR_ARR_FOREIGN; release and copies walk the states + * for their payloads only then. */ + bool foreign; } ScrArrExt; /* The LLVM backend mirrors ScrArr as %ScrArr: its inline retain/release read * `cyc` (field 4, i8) to decide mark-live / candidate buffering exactly like @@ -595,6 +602,20 @@ void scr_arr_set_f64(ScrArr *a, double i, double v); void scr_arr_set_bool(ScrArr *a, double i, bool v); void scr_arr_set_ref(ScrArr *a, double i, void *v); void scr_arr_set_undefined(ScrArr *a, double i); +/* Stores `v` (borrowed, retained) at a canonical index as a value outside + * the element type (SCR_ARR_FOREIGN), or as a present undefined when it is + * undefined, extending the length like the typed setters. False, with + * nothing stored, for an index that is not canonical and for a foreign + * value in a boolean array, whose 1-byte dense slots cannot hold it. */ +bool scr_arr_set_foreign(ScrArr *a, double i, struct ScrDyn *v); +/* The value of a foreign element (+1), or NULL when a[i] is not one. */ +struct ScrDyn *scr_arr_get_foreign(const ScrArr *a, double i); +/* Throws the checked conversion's TypeError for the foreign element a[i] + * read where a value of type `want` is expected. */ +void scr_arr_foreign_throw(const ScrArr *a, double i, const ScrStr *want); +/* The same TypeError for a foreign element's value removed from an array + * (pop/shift into a typed result); takes and releases `value`. */ +void scr_dyn_foreign_throw(struct ScrDyn *value, const char *want); bool scr_arr_delete(ScrArr *a, double i); /* slice(start?, end?): a fresh +1 shallow copy of the index range — diff --git a/packages/runtime/src/scr_bytes.c b/packages/runtime/src/scr_bytes.c index df64b2ab2b..5fb8a58d43 100644 --- a/packages/runtime/src/scr_bytes.c +++ b/packages/runtime/src/scr_bytes.c @@ -2240,29 +2240,26 @@ ScrStr *scr_text_decode_stream(ScrDyn *state, const ScrBytes *input, return result; } -/* Decode the next code point from an ScrStr's storage — ALWAYS valid - * UTF-8 (the string runtime never stores ill-formed sequences), so no - * validation re-runs here. */ -static uint32_t scr_bytes_next_cp(const uint8_t *in, size_t *ip) { - uint8_t b = in[*ip]; - if (b < 0x80) { - *ip += 1; - return b; - } - if ((b & 0xe0) == 0xc0) { - uint32_t cp = ((uint32_t)(b & 0x1f) << 6) | (in[*ip + 1] & 0x3f); - *ip += 2; - return cp; - } - if ((b & 0xf0) == 0xe0) { - uint32_t cp = ((uint32_t)(b & 0xf) << 12) | ((uint32_t)(in[*ip + 1] & 0x3f) << 6) | - (in[*ip + 2] & 0x3f); - *ip += 3; - return cp; - } - uint32_t cp = ((uint32_t)(b & 0x7) << 18) | ((uint32_t)(in[*ip + 1] & 0x3f) << 12) | - ((uint32_t)(in[*ip + 2] & 0x3f) << 6) | (in[*ip + 3] & 0x3f); - *ip += 4; +/* Decode the next code point from the n bytes of an ScrStr's storage. + * Storage is WTF-8, except where raw outside bytes become a string + * unchecked (HTTP header values). A byte that does not start a complete, + * well-formed sequence stands for U+0000..U+00FF, as each byte of such a + * header does in Node, and advances by one; nothing is read past n. */ +static uint32_t scr_bytes_next_cp(const uint8_t *in, size_t n, size_t *ip) { + size_t i = *ip; + uint8_t b = in[i]; + *ip = i + 1; + if (b < 0x80) return b; + size_t len = b >= 0xc2 && b <= 0xdf ? 2 : b >= 0xe0 && b <= 0xef ? 3 : b >= 0xf0 && b <= 0xf4 ? 4 : 0; + if (len == 0 || n - i < len) return b; + uint8_t lo = b == 0xe0 ? 0xa0 : b == 0xf0 ? 0x90 : 0x80; + uint8_t hi = b == 0xf4 ? 0x8f : 0xbf; + if (in[i + 1] < lo || in[i + 1] > hi) return b; + for (size_t k = 2; k < len; k++) + if ((in[i + k] & 0xc0) != 0x80) return b; + uint32_t cp = b & (0x7f >> len); + for (size_t k = 1; k < len; k++) cp = (cp << 6) | (in[i + k] & 0x3f); + *ip = i + len; return cp; } @@ -2287,7 +2284,7 @@ ScrDyn *scr_text_encode_into(const ScrDyn *source, const ScrDyn *destination) { read += ascii; if (written == limit) break; size_t end = written; - uint32_t cp = scr_bytes_next_cp((const uint8_t *)text->data, &end); + uint32_t cp = scr_bytes_next_cp((const uint8_t *)text->data, text->len, &end); if (end > dest->len) break; read += cp > 0xffff ? 2 : 1; written = end; @@ -2440,7 +2437,7 @@ ScrBytes *scr_buffer_transcode(const ScrDyn *source, const ScrDyn *from, const S } ScrStr *text = scr_str_from_utf8_lossy(input->data, len); for (size_t i = 0; i < text->len;) { - uint32_t cp = scr_bytes_next_cp((const uint8_t *)text->data, &i); + uint32_t cp = scr_bytes_next_cp((const uint8_t *)text->data, text->len, &i); written = scr_transcode_put(result->data, written, cp, to_encoding); } scr_str_release(text); @@ -2774,6 +2771,7 @@ static ScrBytes *scr_bytes_decode_b64(ScrBytesSink *sink, const uint8_t *in, siz o += 6; i += 8; } + if (i == n) break; /* the groups ended the input */ } /* Common complete groups need neither per-character branches nor state. */ if (have == 0 && n - i >= 4) { @@ -2824,7 +2822,7 @@ static ScrBytes *scr_bytes_encode_str(ScrBytesSink *sink, const ScrStr *s, const size_t o = s->ascii ? n : scr_bytes_ascii_prefix(in, n), i = o; memcpy(b->data, in, o); while (i < n) { - uint32_t cp = scr_bytes_next_cp(in, &i); + uint32_t cp = scr_bytes_next_cp(in, n, &i); if (cp > 0xffff) { cp -= 0x10000; b->data[o++] = (uint8_t)(0xd800 + (cp >> 10)); @@ -2841,7 +2839,7 @@ static ScrBytes *scr_bytes_encode_str(ScrBytesSink *sink, const ScrStr *s, const ScrBytes *b = scr_bytes_conversion_storage_in(sink, n * 2); size_t o = 0, i = 0; while (i < n) { - uint32_t cp = scr_bytes_next_cp(in, &i); + uint32_t cp = scr_bytes_next_cp(in, n, &i); if (cp > 0xffff) { cp -= 0x10000; uint32_t hi = 0xd800 + (cp >> 10), lo = 0xdc00 + (cp & 0x3ff); @@ -3745,7 +3743,7 @@ double scr_bytes_write_str(ScrBytes *b, const ScrStr *s, const ScrStr *enc, continue; } } - uint32_t cp = scr_bytes_next_cp((const uint8_t *)s->data, &i); + uint32_t cp = scr_bytes_next_cp((const uint8_t *)s->data, s->len, &i); uint32_t first = cp, second = 0; if (cp > 0xffff) { cp -= 0x10000; diff --git a/packages/runtime/src/scr_console.c b/packages/runtime/src/scr_console.c index ad49e4f6f9..67d4aca154 100644 --- a/packages/runtime/src/scr_console.c +++ b/packages/runtime/src/scr_console.c @@ -148,7 +148,7 @@ static void scr_enable_utf8_console_output(void) { } #endif -void scr_runtime_abi_v44(void) {} +void scr_runtime_abi_v45(void) {} void scr_init(void) { #ifdef SCR_WORKERS diff --git a/packages/runtime/src/scr_core.h b/packages/runtime/src/scr_core.h index 8e50ec0ca4..206463e184 100644 --- a/packages/runtime/src/scr_core.h +++ b/packages/runtime/src/scr_core.h @@ -81,7 +81,7 @@ void scr_init(void); /* Program objects emitted by the bundled LLVM helper reference this symbol. * Its versioned spelling makes a mismatched manual runtime link fail before * the program can start. */ -void scr_runtime_abi_v44(void); +void scr_runtime_abi_v45(void); void scr_runtime_workers_v8(void); /* ── the trap funnel (scr_console.c; scr_library.c under -DSCR_LIB) ────── diff --git a/packages/runtime/src/scr_cycle.c b/packages/runtime/src/scr_cycle.c index d08f54a212..0d9df7106f 100644 --- a/packages/runtime/src/scr_cycle.c +++ b/packages/runtime/src/scr_cycle.c @@ -603,7 +603,8 @@ static SCR_TL unsigned scr_cyc_backlog_shift[SCR_CYC_NGENS]; static SCR_TL bool scr_cyc_backlog_judged[SCR_CYC_NGENS]; #endif -/* Mature objects in the last full pass's dead set (scr_cyc_adapt_growth). */ +/* Mature objects in the last pass's dead set (scr_cyc_adapt_growth, + * scr_cyc_adapt_nursery). */ #ifndef SCR_WORKERS static SCR_TL size_t scr_cyc_mature_dead = 0; #endif @@ -639,23 +640,74 @@ static SCR_TL size_t scr_cyc_death_due_live = 0; /* The nursery trigger a program sets for itself * (scr_cyc_set_nursery_threshold), 0 for the default; SCR_CYCLE_THRESHOLD - * takes precedence over it. */ + * takes precedence over it. Either pins the trigger. */ static long scr_cyc_nursery_threshold_set = 0; static SCR_TL size_t scr_cyc_nursery_threshold_cached = 0; +#define SCR_CYC_NURSERY_ADAPTIVE SIZE_MAX -static size_t scr_cyc_nursery_threshold(void) { +/* An unpinned trigger follows where cyclic garbage dies. Garbage that + * outlives the nursery window survives its nursery pass and is promoted, + * and then only a mature or full pass finds it, after walking that level's + * live candidates. So after each scheduled mature or full pass whose dead + * set held more mature and old objects than 1/SCR_CYC_NURSERY_RATIO of what + * the nursery passes since the previous one freed, the window doubles, up + * to SCR_CYC_NURSERY_MAX_SHIFT doublings (4096 candidates); after one that + * held less than a quarter of that share, it halves, down to + * SCR_CYC_NURSERY_CANDIDATES. + * `nursery_freed`: what the nursery passes freed since the last scheduled + * mature or full pass. */ +#define SCR_CYC_NURSERY_RATIO 8 +#define SCR_CYC_NURSERY_MAX_SHIFT 4 +static SCR_TL unsigned char scr_cyc_nursery_shift = 0; +static SCR_TL size_t scr_cyc_nursery_freed = 0; + +static size_t scr_cyc_nursery_pinned(void) { if (scr_cyc_nursery_threshold_cached == 0) { const char *env = scr_getenv("SCR_CYCLE_THRESHOLD"); long v = env ? strtol(env, NULL, 10) : 0; if (v <= 0) v = scr_cyc_nursery_threshold_set; - scr_cyc_nursery_threshold_cached = v > 0 ? (size_t)v : SCR_CYC_NURSERY_CANDIDATES; + scr_cyc_nursery_threshold_cached = v > 0 ? (size_t)v : SCR_CYC_NURSERY_ADAPTIVE; } return scr_cyc_nursery_threshold_cached; } +/* The pinned trigger, or the default one. The scheduled-age rule counts + * collection points in it, so the adaptation does not lengthen how long an + * older backlog waits at them. */ +static size_t scr_cyc_nursery_base(void) { + size_t pinned = scr_cyc_nursery_pinned(); + return pinned != SCR_CYC_NURSERY_ADAPTIVE ? pinned : SCR_CYC_NURSERY_CANDIDATES; +} + +static size_t scr_cyc_nursery_threshold(void) { + size_t pinned = scr_cyc_nursery_pinned(); + if (pinned != SCR_CYC_NURSERY_ADAPTIVE) return pinned; + return (size_t)SCR_CYC_NURSERY_CANDIDATES << scr_cyc_nursery_shift; +} + +/* Adapts an unpinned trigger to the mature and old objects a scheduled + * mature or full pass reclaimed; returns whether the trigger changed. */ +static SCR_CYC_SCHEDULING bool scr_cyc_adapt_nursery(size_t older_freed) { + size_t nursery_freed = scr_cyc_nursery_freed; + scr_cyc_nursery_freed = 0; + if (scr_cyc_nursery_pinned() != SCR_CYC_NURSERY_ADAPTIVE) return false; + if (older_freed * SCR_CYC_NURSERY_RATIO > nursery_freed) { + if (scr_cyc_nursery_shift == SCR_CYC_NURSERY_MAX_SHIFT) return false; + scr_cyc_nursery_shift++; + return true; + } + if (older_freed * SCR_CYC_NURSERY_RATIO * 4 < nursery_freed && scr_cyc_nursery_shift > 0) { + scr_cyc_nursery_shift--; + return true; + } + return false; +} + void scr_cyc_set_nursery_threshold(long threshold) { scr_cyc_nursery_threshold_set = threshold; scr_cyc_nursery_threshold_cached = 0; + scr_cyc_nursery_shift = 0; + scr_cyc_nursery_freed = 0; } /* The number of buffered candidates (across every generation) at which the @@ -881,7 +933,14 @@ static SCR_CYC_SCHEDULING void scr_cyc_scheduled_pass(unsigned gen, bool by_grow size_t freed = scr_cyc_pass(gen); scr_cyc_keep_base = false; size_t old_reclaimed = scr_cyc_old_freed - old_freed_before + scr_cyc_mature_dead; - if (gen == SCR_CYC_NURSERY) return; + if (gen == SCR_CYC_NURSERY) { + scr_cyc_nursery_freed += freed; + return; + } + if (scr_cyc_adapt_nursery(old_reclaimed)) { + scr_cyc_trigger = scr_cyc_buffered() + scr_cyc_nursery_threshold(); + scr_cyc_window_update(); + } if (scr_cands.n != 0) { scr_cyc_backlog_judged[gen] = true; if (freed * SCR_CYC_GROWTH_DIV >= scr_cands.n) @@ -940,7 +999,7 @@ void scr_cyc_collect_scheduled(void) { if (!scr_cyc_older_backlog()) { scr_cyc_scheduled_mature_age = 0; } else if (++scr_cyc_scheduled_mature_age - >= scr_cyc_nursery_threshold()) { + >= scr_cyc_nursery_base()) { scr_cyc_scheduled_pass(SCR_CYC_OLD, false, false); return; } @@ -1981,7 +2040,7 @@ static SCR_CYC_COMPACT size_t scr_cyc_settle(unsigned gen_limit, bool doomed) { ScrCycHdr *h = scr_cyc_hdr(obj); const ScrCycType *type = scr_cyc_type(h); if (gen_limit < SCR_CYC_OLD) scr_cyc_freed(type, gen_limit); - else scr_cyc_mature_dead += scr_cyc_gen(h) == SCR_CYC_MATURE; + scr_cyc_mature_dead += scr_cyc_gen(h) == SCR_CYC_MATURE; #ifdef SCR_YIN if (scr_cyc_gen(h) == SCR_CYC_YOUNG) scr_yin_leave(h, obj); #endif diff --git a/packages/runtime/src/scr_dyn.h b/packages/runtime/src/scr_dyn.h index 656f2b9943..1004a93c22 100644 --- a/packages/runtime/src/scr_dyn.h +++ b/packages/runtime/src/scr_dyn.h @@ -1941,7 +1941,8 @@ extern void scr_dyn_listener_fire_data(ScrClosure *cb, ScrBytes *chunk); extern void scr_dyn_listener_fire_err(ScrClosure *cb, ScrStr *msg); void *scr_dyn_retain_v(void *d); -/* After a compiled fast path decremented a surviving capsule or handle. */ +/* After a compiled fast path decremented a surviving value of a kind that + * can be a cycle candidate and is not one yet (scr_json.c). */ void scr_dyn_release_survivor(ScrDyn *d); void scr_dyn_release_v(void *d); void scr_dyn_trace_v(void *d, ScrTraceVisit visit, void *ctx); @@ -1955,6 +1956,9 @@ long scr_dyn_live_count(void); * scr_json.c, which programs with dynamic-tier code link. */ /* The builtin Object.prototype node, or NULL before its first use. */ const ScrDyn *scr_dyn_tier_object_prototype(void); +/* A read site's answer for a native array's length or a native iterator's + * next (scr_js_get_probe_native): +1, or NULL. */ +ScrDyn *scr_dyn_tier_unshaped_get(const ScrDyn *receiver, const ScrStr *key); /* Shared key bytes for `key` (+1, NUL-terminated), and their release. */ char *scr_dyn_tier_key_copy(const char *key, size_t key_len); void scr_dyn_tier_key_release(char *key, size_t key_len); diff --git a/packages/runtime/src/scr_dyn_invoke.c b/packages/runtime/src/scr_dyn_invoke.c index 2fd3aa406a..e31359efd6 100644 --- a/packages/runtime/src/scr_dyn_invoke.c +++ b/packages/runtime/src/scr_dyn_invoke.c @@ -159,6 +159,10 @@ static void scr_dyn_display_buf(ScrJsonBuf *b, const ScrDyn *d, bool to_string) } } +void scr_jb_join_dyn(ScrJsonBuf *b, const ScrDyn *d) { + if (d->kind != SCR_DYN_UNDEF && d->kind != SCR_DYN_NULL) scr_dyn_display_buf(b, d, true); +} + ScrStr *scr_dyn_display(const ScrDyn *d) { ScrJsonBuf b; scr_jb_init(&b); diff --git a/packages/runtime/src/scr_js.h b/packages/runtime/src/scr_js.h index 3ff9becd3a..5abab2053e 100644 --- a/packages/runtime/src/scr_js.h +++ b/packages/runtime/src/scr_js.h @@ -235,6 +235,10 @@ ScrDyn *scr_js_load_probe(ScrDyn *receiver, const ScrStr *key, ScrJsIc *ic); * a key that the receiver and its plain prototype chain lack, and caches * that answer. A method read uses scr_js_load_probe. */ ScrDyn *scr_js_get_probe(ScrDyn *receiver, const ScrStr *key, ScrJsIc *ic); +/* scr_js_get_probe for a site whose key is `length` or `next`, which also + * answers a native array's length and a native iterator's next + * (scr_dyn_tier_unshaped_get). */ +ScrDyn *scr_js_get_probe_native(ScrDyn *receiver, const ScrStr *key, ScrJsIc *ic); /* `receiver.key = value` (value borrowed) when it replaces a writable own * data property or adds one the prototype chain lacks: true when done. */ bool scr_js_store_probe(ScrDyn *receiver, const ScrStr *key, ScrDyn *value, ScrJsIc *ic); diff --git a/packages/runtime/src/scr_js_object.c b/packages/runtime/src/scr_js_object.c index e3d0b3a542..4cee9e7e88 100644 --- a/packages/runtime/src/scr_js_object.c +++ b/packages/runtime/src/scr_js_object.c @@ -407,6 +407,11 @@ ScrDyn *scr_js_get_probe(ScrDyn *receiver, const ScrStr *key, ScrJsIc *ic) { return scr_js_probe_load(receiver, key, ic, true); } +ScrDyn *scr_js_get_probe_native(ScrDyn *receiver, const ScrStr *key, ScrJsIc *ic) { + if (receiver->kind != SCR_DYN_OBJ) return scr_dyn_tier_unshaped_get(receiver, key); + return scr_js_probe_load(receiver, key, ic, true); +} + /* ── stores ───────────────────────────────────────────────────────────── */ /* The generic store keeps a native Error's fields in sync with these keys diff --git a/packages/runtime/src/scr_json.c b/packages/runtime/src/scr_json.c index cd9eeecde6..fb62bc7a42 100644 --- a/packages/runtime/src/scr_json.c +++ b/packages/runtime/src/scr_json.c @@ -96,9 +96,13 @@ static SCR_TL ScrDyn *scr_iterator_symbol; static SCR_TL ScrDyn *scr_async_iterator_symbol; static SCR_TL ScrDyn *scr_to_primitive_symbol; static bool scr_dyn_generator(const ScrDyn *value) { - return value && value->kind == SCR_DYN_TYPED_REF && - (value->v.typed_ref.type_key[0] == 'g' || value->v.typed_ref.type_key[0] == 'a') && - (!strncmp(value->v.typed_ref.type_key, "generator<", 10) || !strncmp(value->v.typed_ref.type_key, "async-generator<", 16)); + if (!value || value->kind != SCR_DYN_TYPED_REF) return false; + /* Array keys also start with 'a': the second byte sets them apart before + * any comparison. */ + const char *key = value->v.typed_ref.type_key; + size_t len = value->v.typed_ref.type_key_len; + if (key[0] == 'g') return len >= 10 && !memcmp(key, "generator<", 10); + return key[0] == 'a' && key[1] == 's' && len >= 16 && !memcmp(key, "async-generator<", 16); } static bool scr_dyn_generator_symbol(const ScrDyn *value, const ScrDyn *key) { const ScrDyn *symbol = value->v.typed_ref.type_key[0] == 'a' ? scr_async_iterator_symbol : scr_iterator_symbol; @@ -1317,9 +1321,9 @@ static void scr_dyn_gcfree(void *ptr) { scr_dyn_dispose(ptr, true); } static void scr_dyn_destroy(void *ptr) { scr_dyn_dispose(ptr, false); } /* The cycle bookkeeping of a value that survives a release. Compiled - * release fast paths decrement first and call this for the kinds whose - * candidacy depends on per-value state, matching scr_dyn_release: arrays - * and objects not yet buffered (skipped while their document is intact, + * release fast paths decrement first and call this for every kind that can + * be a candidate and is not one yet, matching scr_dyn_release: functions + * and proxies, arrays and objects (skipped while their document is intact, * ScrDynRegion), capsules and handles. */ void scr_dyn_release_survivor(ScrDyn *d) { if (d->kind == SCR_DYN_ARR || d->kind == SCR_DYN_OBJ) { @@ -1327,6 +1331,10 @@ void scr_dyn_release_survivor(ScrDyn *d) { if (!region || !region->intact) scr_cyc_released(d); return; } + if (d->kind == SCR_DYN_FUNC || d->kind == SCR_DYN_PROXY) { + scr_cyc_released(d); + return; + } if ((d->kind == SCR_DYN_TYPED_REF && (d->v.typed_ref.materialized || d->v.typed_ref.casts || d->v.typed_ref.traced || d->prototype || d->symbol_properties || d->symbol_keys)) || @@ -4344,6 +4352,7 @@ ScrDyn *scr_dyn_typed_ref_element(const ScrDyn *d, size_t index) { double state = scr_arr_state(arr, (double)index); if (state == (double)SCR_ARR_VALUE) return scr_dyn_typed_ref_array_ops(d)->get(arr, (double)index); if (state == (double)SCR_ARR_UNDEFINED) return scr_dyn_retain(scr_dyn_undefined()); + if (state == (double)SCR_ARR_FOREIGN) return scr_arr_get_foreign(arr, (double)index); return NULL; } @@ -7494,6 +7503,23 @@ static ScrDyn *scr_native_iterator_new(ScrDyn *source) { return result; } +#ifdef SCR_JS_TIER +/* A read site's answer for a receiver without a JS shape whose property is + * plain data the runtime holds directly: a native array's length (the + * generic read would build its checked view first) and a native iterator's + * `next` (the builtin the generic read returns). +1, or NULL for the slow + * path. */ +ScrDyn *scr_dyn_tier_unshaped_get(const ScrDyn *receiver, const ScrStr *key) { + if (key->len == 6 && receiver->kind == SCR_DYN_TYPED_REF && !memcmp(key->data, "length", 6) && + scr_dyn_typed_ref_array_ops(receiver) && scr_dyn_ref_is_array(receiver)) + return scr_dyn_new_num((double)((ScrArr *)receiver->v.typed_ref.ptr)->len); + if (key->len == 4 && receiver->kind == SCR_DYN_HANDLE && + receiver->v.handle.tag == SCR_DYNH_ITERATOR && !memcmp(key->data, "next", 4)) + return scr_native_iterator_get(receiver->v.handle.ptr, key->data, key->len); + return NULL; +} +#endif + static ScrDyn *scr_native_collection_iterator(ScrDyn *source, unsigned selection) { ScrDyn *result = scr_native_iterator_new(source); ((ScrNativeIterator *)result->v.handle.ptr)->selection = selection; @@ -8384,7 +8410,9 @@ ScrDyn *scr_dyn_iterator(const ScrDyn *source, const ScrStr *spell) { } /* An array whose iterator method is the builtin values(): what calling * it with the array as `this` returns, without the call. */ - if (source->kind == SCR_DYN_ARR && method == scr_iterator_functions[1]) { + if (method == scr_iterator_functions[1] && + (source->kind == SCR_DYN_ARR || + (source->kind == SCR_DYN_TYPED_REF && scr_dyn_ref_is_array(source)))) { scr_dyn_release(method); return scr_native_iterator_new((ScrDyn *)source); } @@ -14398,6 +14426,13 @@ void scr_jb_put_dyn(ScrJsonBuf *b, const ScrDyn *d) { else scr_json_replaced_write(b, d, scr_dyn_undefined(), NULL, &scr_json_compact_gap, b->seen_len); } +void scr_jb_put_dyn_element(ScrJsonBuf *b, const ScrDyn *d, double index) { + ScrDyn *value = scr_json_to_json_at(scr_dyn_retain((ScrDyn *)d), NULL, (size_t)index); + if (!value) return; + scr_jb_put_dyn(b, value); + scr_dyn_release(value); +} + /* Untyped RC adapters (box/promise/exception-cell currency). */ void *scr_dyn_retain_v(void *d) { return scr_dyn_retain((ScrDyn *)d); } void scr_dyn_release_v(void *d) { scr_dyn_release((ScrDyn *)d); } diff --git a/packages/runtime/src/scr_json.h b/packages/runtime/src/scr_json.h index a8a59b60c6..c1aa5dc103 100644 --- a/packages/runtime/src/scr_json.h +++ b/packages/runtime/src/scr_json.h @@ -180,6 +180,9 @@ void scr_dyn_from_enter(const void *v); void scr_dyn_from_leave(void); void scr_jb_putc(ScrJsonBuf *b, char c); void scr_jb_puts(ScrJsonBuf *b, const char *s); +/* Array.prototype.join's conversion of one element, appended to `b`: + * nothing for undefined and null, ToString of anything else. */ +void scr_jb_join_dyn(ScrJsonBuf *b, const ScrDyn *d); /* JS JSON.stringify number rules: NaN/±Infinity → null, -0 → 0, else * shortest-roundtrip via scr_f64_to_str. */ void scr_jb_put_f64(ScrJsonBuf *b, double v); @@ -199,6 +202,10 @@ void scr_jb_put_date(ScrJsonBuf *b, const ScrDate *date); * slot's rule); containers follow JSON.stringify (toJSON, accessors, * circular structures). May leave an exception pending. */ void scr_jb_put_dyn(ScrJsonBuf *b, const ScrDyn *d); +/* scr_jb_put_dyn for array element `index` (a foreign element of a typed + * array): its toJSON runs first, with the index as its key. May leave an + * exception pending. */ +void scr_jb_put_dyn_element(ScrJsonBuf *b, const ScrDyn *d, double index); /* The entries of record `rec`'s `unknown` index signature (its overflow * map) after the record's earlier members: separators, keys and values in * JS own-key order with JSON.stringify's member rules (toJSON called with diff --git a/packages/runtime/src/scr_mailbox.c b/packages/runtime/src/scr_mailbox.c index b2af227747..6e56783783 100644 --- a/packages/runtime/src/scr_mailbox.c +++ b/packages/runtime/src/scr_mailbox.c @@ -21,6 +21,7 @@ #include #include #ifdef __linux__ +#include #include #endif #endif @@ -321,7 +322,32 @@ void scr_mailbox_wait(ScrMailbox *mailbox, double milliseconds) { if (!(milliseconds > 0)) return; if (scr_mailbox_spin(mailbox, milliseconds)) return; scr_mailbox_lock(mailbox); - if (!mailbox->head && !mailbox->closed) { + bool waiting = !mailbox->head && !mailbox->closed; +#ifdef __linux__ + /* Linux sleeps in ppoll(2) on the wake pipe rather than on the + * condition: a pipe write wakes its reader as a synchronous wakeup, + * which the scheduler can run on the writer's CPU when the writer is + * about to block. A worker that posts its result and then waits frees + * that CPU, so the thread it woke runs there at once instead of queueing + * behind another busy thread until the scheduler preempts it. Bytes left + * by signals whose events were already taken are drained first, under + * the lock: an event posted after the unlock writes a new byte. */ + if (waiting) scr_mailbox_drain(mailbox); + scr_mailbox_unlock(mailbox); + if (!waiting) return; + struct pollfd wake = {mailbox->read_fd, POLLIN, 0}; + if (isinf(milliseconds)) (void)ppoll(&wake, 1, NULL, NULL); + else { + /* The event loop rechecks its monotonic deadlines after every wake, + * including spurious ones. Bound conversion before casting time_t. */ + if (milliseconds > 86400000) milliseconds = 86400000; + struct timespec timeout = {(time_t)(milliseconds / 1000), + (long)ceil(fmod(milliseconds, 1000) * 1000000)}; + if (timeout.tv_nsec == 1000000000) { timeout.tv_sec++; timeout.tv_nsec = 0; } + (void)ppoll(&wake, 1, &timeout, NULL); + } +#else + if (waiting) { #ifdef _WIN32 DWORD timeout = milliseconds >= INFINITE ? INFINITE : (DWORD)ceil(milliseconds); (void)SleepConditionVariableSRW(&mailbox->condition, &mailbox->lock, timeout, 0); @@ -348,6 +374,7 @@ void scr_mailbox_wait(ScrMailbox *mailbox, double milliseconds) { #endif } scr_mailbox_unlock(mailbox); +#endif } int scr_mailbox_pollfd(const ScrMailbox *mailbox) { diff --git a/packages/runtime/src/scr_net.c b/packages/runtime/src/scr_net.c index bd12339781..a940fb7f14 100644 --- a/packages/runtime/src/scr_net.c +++ b/packages/runtime/src/scr_net.c @@ -516,6 +516,16 @@ struct ScrNetSocket { /* write buffer: [whead, wlen) of wbuf is unsent */ char *wbuf; size_t whead, wlen, wcap; + /* Node's writable state.length: each write with bytes in the buffer + * counts in full until its last byte is sent (a partly sent write + * included). wq[wq_head, wq_n) holds those writes in order, each with + * the offset where its bytes end, counted from wbase, the offset of + * wbuf[0] in everything buffered since the buffer last emptied. */ + struct ScrNetPendingWrite { + size_t end, len; + } *wq; + size_t wq_head, wq_n, wq_cap; + size_t wbase, wpending; /* receive buffer: [rhead, rlen) of rbuf is arrived-but-unconsumed — * fed by paused-mode reads ('readable' consumers) and unshift; drained * FIRST by every consumer (data listeners, the parser attach, the TLS @@ -565,6 +575,11 @@ struct ScrNetSocket { ScrNetLs lookup_ls; struct ScrNetLookupEv *lookup_evs; ScrNetLs data_ls, end_ls, close_ls, err_ls, conn_ls, timeout_ls, readable_ls; + /* Writable backpressure: a user write answered false (the bytes it left + * unsent reached SCR_NET_WRITE_HWM), so 'drain' fires once the write + * buffer empties */ + ScrNetLs drain_ls; + bool need_drain; /* Flow control (pause()/resume()): user_paused holds reads OFF even * with consumers (kernel/TCP backpressure is the buffer); flowing * counts as a consumer even with no 'data' listener @@ -747,6 +762,7 @@ void scr_net_sock_release(ScrNetSocket *s) { scr_net_ls_drop(&s->conn_ls); scr_net_ls_drop(&s->timeout_ls); scr_net_ls_drop(&s->readable_ls); + scr_net_ls_drop(&s->drain_ls); scr_net_ls_drop(&s->finish_ls); scr_net_ls_drop(&s->wcb_ls); scr_net_ls_drop(&s->ready_ls); @@ -764,6 +780,7 @@ void scr_net_sock_release(ScrNetSocket *s) { if (s->lookup_evs != NULL) scr_net_lookup_ops->free(s); scr_net_tunnel_free(s->tunnel); free(s->wbuf); + free(s->wq); free(s->rbuf); if (s->pipe_dst) scr_net_sock_release(s->pipe_dst); if (s->conn_pending) scr_net_server_release(s->conn_pending); @@ -1082,13 +1099,34 @@ static void scr_net_sock_maybe_finish_write(ScrNetSocket *s) { else scr_net_sock_update_read(s); /* the drain-to-EOF mode arms here */ } +/* The write buffer emptied (or its bytes moved elsewhere): every pending + * write is complete. */ +static void scr_net_sock_wq_clear(ScrNetSocket *s) { + s->whead = s->wlen = 0; + s->wq_head = s->wq_n = 0; + s->wbase = 0; + s->wpending = 0; +} + +/* Completes the pending writes whose last byte the flush has sent. */ +static void scr_net_sock_wq_settle(ScrNetSocket *s) { + size_t sent_to = s->wbase + s->whead; + while (s->wq_head < s->wq_n && s->wq[s->wq_head].end <= sent_to) { + s->wpending -= s->wq[s->wq_head].len; + s->wq_head++; + } + if (s->wq_head == s->wq_n) s->wq_head = s->wq_n = 0; +} + /* Append bytes to the write buffer (post-immediate-write remainder, or - * everything while connecting/buffered). */ -static void scr_net_sock_buffer(ScrNetSocket *s, const char *data, size_t len) { + * everything while connecting/buffered). `counted` is the length of the + * write they belong to, which counts in full until they are sent. */ +static void scr_net_sock_buffer(ScrNetSocket *s, const char *data, size_t len, size_t counted) { if (len == 0) return; /* compact the consumed head first */ if (s->whead > 0) { memmove(s->wbuf, s->wbuf + s->whead, s->wlen - s->whead); + s->wbase += s->whead; s->wlen -= s->whead; s->whead = 0; } @@ -1101,6 +1139,63 @@ static void scr_net_sock_buffer(ScrNetSocket *s, const char *data, size_t len) { } memcpy(s->wbuf + s->wlen, data, len); s->wlen += len; + if (s->wq_n == s->wq_cap && s->wq_head > 0) { + /* reclaim the completed writes before growing: a buffer that never + * empties completes old writes while new ones arrive */ + memmove(s->wq, s->wq + s->wq_head, (s->wq_n - s->wq_head) * sizeof *s->wq); + s->wq_n -= s->wq_head; + s->wq_head = 0; + } + if (s->wq_n == s->wq_cap) { + s->wq_cap = s->wq_cap ? s->wq_cap * 2 : 8; + s->wq = realloc(s->wq, s->wq_cap * sizeof *s->wq); + if (!s->wq) scr_net_oom(); + } + s->wq[s->wq_n].end = s->wbase + s->wlen; + s->wq[s->wq_n].len = counted; + s->wq_n++; + s->wpending += counted; +} + +/* Node's net.Socket writableHighWaterMark (the Writable default, + * getDefaultHighWaterMark()). */ +#define SCR_NET_WRITE_HWM 65536 + +/* 'drain' once a false write's bytes are all out: Node's afterWrite emits + * it when needDrain is set and nothing is pending, unless the stream is + * ending or destroyed. */ +static void scr_net_sock_maybe_drain(ScrNetSocket *s) { + if (!s->need_drain || s->wlen > s->whead) return; + if (s->wr_ending || s->close_emitted || s->fd < 0) return; + s->need_drain = false; + scr_net_fire0_this(&s->drain_ls, s, SCR_DYNH_NET_SOCKET); +} + +static void scr_net_sock_write_raw(ScrNetSocket *s, const char *data, size_t len); + +/* The write half takes writes: not ending or closed, and connected, + * connecting, or waiting to dial (the agent's deferred dial, the caller + * lookup), which buffers them until the connection is established. */ +static bool scr_net_sock_write_open(ScrNetSocket *w) { + if (w->wr_ending || w->close_emitted) return false; + return w->fd >= 0 || w->dial_deferred || w->lookup_wait; +} + +/* A user write: Writable.write's answer, as Node 24's writeOrBuffer + * computes it after the socket writes. Each write with bytes still + * unsent counts in full (wpending, Node's state.length), so the answer is + * false when the pending writes reach the high-water mark, and needDrain + * is set. A write the kernel takes at once answers true whatever its + * size. A write to an ended or closed socket answers false, and so does + * one whose send fails and closes the socket. */ +static bool scr_net_sock_user_write(ScrNetSocket *s, const char *data, size_t len) { + ScrNetSocket *w = s->adopted_by != NULL ? s->adopted_by : s; + bool open = scr_net_sock_write_open(w); + scr_net_sock_write_raw(s, data, len); + if (!open || !scr_net_sock_write_open(w)) return false; + bool ok = w->wpending < SCR_NET_WRITE_HWM; + if (!ok) w->need_drain = true; + return ok; } /* The write path shared by write()/end(data)/pipe delivery. Ignored on a @@ -1136,7 +1231,7 @@ static void scr_net_sock_write_raw(ScrNetSocket *s, const char *data, size_t len (s->tops && !s->t_est)) { /* mid-connect and mid-handshake bytes buffer; the flush after * 'connect'/establishment sends them */ - scr_net_sock_buffer(s, data, len); + scr_net_sock_buffer(s, data, len, len); scr_net_sock_update_write(s); return; } @@ -1155,7 +1250,7 @@ static void scr_net_sock_write_raw(ScrNetSocket *s, const char *data, size_t len } } if ((size_t)sent < len) { - scr_net_sock_buffer(s, data + sent, len - (size_t)sent); + scr_net_sock_buffer(s, data + sent, len - (size_t)sent, len); scr_net_sock_update_write(s); } } @@ -1220,7 +1315,7 @@ void scr_net_sock_writev_native(ScrNetSocket *s, const ScrNetSlice *slices, size skip -= len; continue; } - scr_net_sock_buffer(s, (const char *)iov[i].iov_base + skip, len - skip); + scr_net_sock_buffer(s, (const char *)iov[i].iov_base + skip, len - skip, len); skip = 0; } scr_net_sock_update_write(s); @@ -1247,11 +1342,15 @@ static void scr_net_sock_flush(ScrNetSocket *s) { } if (sent > 0) scr_net_sock_touch(s); s->whead += (size_t)sent; + scr_net_sock_wq_settle(s); } if (s->wlen == s->whead) { - s->whead = s->wlen = 0; + scr_net_sock_wq_clear(s); scr_net_sock_update_write(s); scr_net_sock_maybe_finish_write(s); + /* Node's afterWrite for a write the socket completed later: 'drain' + * now, unless the write half is ending or the socket is gone */ + scr_net_sock_maybe_drain(s); } } @@ -2735,7 +2834,7 @@ ScrNetSocket *scr_net_connect_deferred(double port, ScrStr *host /*borrowed, nul * for a slot takes a pooled socket instead of dialing. */ void scr_net_sock_move_writes(ScrNetSocket *from, ScrNetSocket *to) { if (from->wlen > from->whead) scr_net_sock_write_native(to, from->wbuf + from->whead, from->wlen - from->whead); - from->whead = from->wlen = 0; + scr_net_sock_wq_clear(from); } void scr_net_sock_dial_start(ScrNetSocket *s) { @@ -3150,16 +3249,42 @@ void scr_net_lookup_answer(ScrClosure *self, bool has_err, ScrStr *msg /*borrowe /* ── the socket surface ──────────────────────────────────────────────── */ -void scr_net_sock_write_str(ScrNetSocket *s, ScrStr *data /*borrowed*/) { +bool scr_net_sock_write_str(ScrNetSocket *s, ScrStr *data /*borrowed*/) { char *owned; const char *bytes = scr_str_utf8(data, &owned); - scr_net_sock_write_raw(s, bytes, data->len); + bool ok = scr_net_sock_user_write(s, bytes, data->len); scr_str_utf8_end(owned); + return ok; } -void scr_net_sock_write_bytes(ScrNetSocket *s, ScrBytes *data /*borrowed*/) { +bool scr_net_sock_write_bytes(ScrNetSocket *s, ScrBytes *data /*borrowed*/) { SCR_BYTES_SNAPSHOT(data); - scr_net_sock_write_raw(s, (const char *)data->data, (data->len * scr_bytes_elem_size(data->elem))); + return scr_net_sock_user_write(s, (const char *)data->data, + (data->len * scr_bytes_elem_size(data->elem))); +} + +/* socket.writableLength: Node's state.length, the length of every write + * not yet completely sent. */ +double scr_net_sock_writable_length(ScrNetSocket *s) { + ScrNetSocket *w = s->adopted_by != NULL ? s->adopted_by : s; + return (double)w->wpending; +} + +/* socket.writableNeedDrain: a write answered false and 'drain' has not + * fired, on a socket whose write half is open (including one still + * waiting to dial). */ +bool scr_net_sock_writable_need_drain(ScrNetSocket *s) { + ScrNetSocket *w = s->adopted_by != NULL ? s->adopted_by : s; + return w->need_drain && scr_net_sock_write_open(w); +} + +/* on('drain') / once('drain'). */ +void scr_net_sock_on_drain(ScrNetSocket *s, ScrClosure *cb /*moves*/, bool once) { + if (s->close_emitted) { + scr_closure_release(cb); + return; + } + scr_net_ls_add(&s->drain_ls, cb, NULL, once); } void scr_net_sock_end(ScrNetSocket *s) { @@ -5264,7 +5389,7 @@ ScrNetSocket *scr_net_sock_adopt(ScrNetSocket *raw) { tun->send_only = true; t->tunnel = tun; } - raw->whead = raw->wlen = 0; + scr_net_sock_wq_clear(raw); raw->adopted_by = scr_net_sock_retain(t); t->adopted_from = scr_net_sock_retain(raw); scr_net_sock_register(t); @@ -5523,6 +5648,7 @@ static bool scr_net_sock_settle(ScrNetSocket *sock) { scr_net_ls_drop(&sock->conn_ls); scr_net_ls_drop(&sock->timeout_ls); scr_net_ls_drop(&sock->readable_ls); + scr_net_ls_drop(&sock->drain_ls); scr_net_ls_drop(&sock->finish_ls); scr_net_ls_drop(&sock->wcb_ls); scr_net_ls_drop(&sock->ready_ls); @@ -5860,6 +5986,7 @@ static void scr_net_cleanup_atexit(void) { scr_net_ls_drop(&s->conn_ls); scr_net_ls_drop(&s->timeout_ls); scr_net_ls_drop(&s->readable_ls); + scr_net_ls_drop(&s->drain_ls); scr_net_ls_drop(&s->lookup_ls); /* Tear down the protocol layer's parser NOW: an in-flight http * exchange (a handler that threw before res.end()) holds req/res, @@ -5984,8 +6111,7 @@ static ScrDyn *scr_net_dynh_sock_invoke(void *h, ScrDyn *self, const char *metho /* fires once when the FIN goes out — once either way */ scr_net_sock_on_finish(s, scr_dyn_listener_closure0(cb)); } else if (scr_net_dynh_name_is(name, "drain")) { - /* This surface never backpressures (write answers true) — an - * accepted, never-fired registration is the consistent answer. */ + scr_net_sock_on_drain(s, scr_dyn_listener_closure0(cb), once); } else if (scr_net_dynh_tls != NULL && name->kind == SCR_DYN_STR && scr_net_dynh_tls->on(s, name->v.str->data, cb, once)) { /* a TLS event name ('secureConnect'), handled by the TLS unit */ @@ -6008,13 +6134,13 @@ static ScrDyn *scr_net_dynh_sock_invoke(void *h, ScrDyn *self, const char *metho scr_dyn_arg_type_fail("chunk", "of type string or an instance of Buffer or Uint8Array", chunk); return NULL; } - if (chunk->kind == SCR_DYN_STR) scr_net_sock_write_str(s, chunk->v.str); - else scr_net_sock_write_bytes(s, chunk->v.bytes); + bool ok = chunk->kind == SCR_DYN_STR ? scr_net_sock_write_str(s, chunk->v.str) + : scr_net_sock_write_bytes(s, chunk->v.bytes); if (argc > 1 && args[1]->kind == SCR_DYN_FUNC) { /* write(chunk, cb): fires when the buffer drains (sweep-deferred) */ scr_net_sock_on_write_flush(s, scr_dyn_listener_closure0(args[1])); } - return scr_dyn_new_bool(true); /* backpressure is not modeled (SEMANTICS.md) */ + return scr_dyn_new_bool(ok); } if (strcmp(method, "end") == 0) { const ScrDyn *chunk = argc > 0 ? args[0] : scr_dyn_undefined(); @@ -6193,10 +6319,12 @@ static ScrDyn *scr_net_dynh_sock_get(void *h, const char *key, size_t key_len) { } } if (strcmp(key, "readable") == 0) return scr_dyn_new_bool(scr_net_sock_readable(s)); + if (strcmp(key, "writableLength") == 0) return scr_dyn_new_num(scr_net_sock_writable_length(s)); + if (strcmp(key, "writableNeedDrain") == 0) + return scr_dyn_new_bool(scr_net_sock_writable_need_drain(s)); if (strcmp(key, "writableHighWaterMark") == 0 || strcmp(key, "readableHighWaterMark") == 0) { - /* Node's default stream highWaterMark — a constant here (backpressure - * is not modeled; SEMANTICS.md) */ - return scr_dyn_new_num(16384); + /* Node's default stream highWaterMark (getDefaultHighWaterMark()) */ + return scr_dyn_new_num(SCR_NET_WRITE_HWM); } if (scr_net_dynh_tls != NULL) { ScrDyn *out = NULL; @@ -6500,10 +6628,11 @@ void scr_net_dyn_install(void) { * STR/BYTES dispatch onto the typed entries; undefined/null end()s plain * (Node accepts them); anything else throws Node's ERR_INVALID_ARG_TYPE * chunk TypeError. Both borrowed. */ -void scr_net_sock_write_dynv(ScrNetSocket *s, const ScrDyn *d) { - if (d->kind == SCR_DYN_STR) scr_net_sock_write_str(s, d->v.str); - else if (d->kind == SCR_DYN_BYTES) scr_net_sock_write_bytes(s, d->v.bytes); - else scr_dyn_arg_type_fail("chunk", "of type string or an instance of Buffer or Uint8Array", d); +bool scr_net_sock_write_dynv(ScrNetSocket *s, const ScrDyn *d) { + if (d->kind == SCR_DYN_STR) return scr_net_sock_write_str(s, d->v.str); + if (d->kind == SCR_DYN_BYTES) return scr_net_sock_write_bytes(s, d->v.bytes); + scr_dyn_arg_type_fail("chunk", "of type string or an instance of Buffer or Uint8Array", d); + return false; } void scr_net_sock_end_dynv(ScrNetSocket *s, const ScrDyn *d) { diff --git a/packages/runtime/src/scr_net.h b/packages/runtime/src/scr_net.h index 95490aaa5d..6a41e26d86 100644 --- a/packages/runtime/src/scr_net.h +++ b/packages/runtime/src/scr_net.h @@ -227,15 +227,21 @@ ScrNetSocket *scr_net_connect_lookup(double port, ScrStr *host /*borrowed*/, ScrClosure *lookup /*moves*/, void *answer_fn); /* +1 */ void scr_net_lookup_answer(ScrClosure *self, bool has_err, ScrStr *msg /*borrowed, nullable*/, ScrArr *ips /*borrowed, nullable*/); -void scr_net_sock_write_str(ScrNetSocket *s, ScrStr *data /*borrowed*/); -void scr_net_sock_write_bytes(ScrNetSocket *s, ScrBytes *data /*borrowed*/); +/* write(): Node's Writable answer — false when the bytes the write left + * unsent reach the 65536-byte high-water mark, after which 'drain' fires + * when the write buffer empties; false on an ended or closed socket. */ +bool scr_net_sock_write_str(ScrNetSocket *s, ScrStr *data /*borrowed*/); +bool scr_net_sock_write_bytes(ScrNetSocket *s, ScrBytes *data /*borrowed*/); +double scr_net_sock_writable_length(ScrNetSocket *s); +bool scr_net_sock_writable_need_drain(ScrNetSocket *s); +void scr_net_sock_on_drain(ScrNetSocket *s, ScrClosure *cb /*moves*/, bool once); void scr_net_sock_end(ScrNetSocket *s); void scr_net_sock_end_str(ScrNetSocket *s, ScrStr *data /*borrowed*/); void scr_net_sock_end_bytes(ScrNetSocket *s, ScrBytes *data /*borrowed*/); /* Checked-dynamic chunk forms (an untyped JS payload into a typed * socket): STR/BYTES dispatch; end() takes undefined/null as no chunk; * anything else throws Node's ERR_INVALID_ARG_TYPE chunk TypeError. */ -void scr_net_sock_write_dynv(ScrNetSocket *s, const ScrDyn *d /*borrowed*/); +bool scr_net_sock_write_dynv(ScrNetSocket *s, const ScrDyn *d /*borrowed*/); void scr_net_sock_end_dynv(ScrNetSocket *s, const ScrDyn *d /*borrowed*/); void scr_net_sock_destroy(ScrNetSocket *s); /* Flow control (pause/resume — see the struct's flag comments), the diff --git a/packages/runtime/src/scr_rc.h b/packages/runtime/src/scr_rc.h index de9f21f443..171e420c20 100644 --- a/packages/runtime/src/scr_rc.h +++ b/packages/runtime/src/scr_rc.h @@ -703,8 +703,9 @@ void scr_cyc_set_growth_cap(long cap); /* Sets the nursery trigger SCR_CYCLE_THRESHOLD sets (scr_cycle.c): the * number of new candidates at which the release path runs a pass, for a * program that knows its own collection profile. The environment variable - * takes precedence; 0 restores the default. The trigger re-arms with it at - * the end of the next pass. Call it before the program allocates much. */ + * takes precedence; 0 restores the default, which starts at 256 and adapts + * to the program. The trigger re-arms with it at the end of the next pass. + * Call it before the program allocates much. */ void scr_cyc_set_nursery_threshold(long threshold); void scr_cyc_context_cleanup(void); diff --git a/packages/runtime/src/scr_regex.c b/packages/runtime/src/scr_regex.c index ab323c2880..2a98e92c6a 100644 --- a/packages/runtime/src/scr_regex.c +++ b/packages/runtime/src/scr_regex.c @@ -1416,6 +1416,51 @@ ScrStr *scr_regex_replace(ScrStr *s, ScrRegex *re, ScrStr *rep) { return scr_replace_impl(s, re, rep); } +void scr_re_replace_out_reserve(ScrReReplaceOut *o, int64_t need) { + if (need < 0 || (uint64_t)need > SCR_STR_MAX_CAP - (uint64_t)o->len) scr_regex_oom(); + size_t cap = (size_t)o->cap * 2; + if (cap < (size_t)(o->len + need)) cap = (size_t)(o->len + need); + if (cap > SCR_STR_MAX_CAP) cap = SCR_STR_MAX_CAP; + o->str->len = (size_t)o->len; + o->str = scr_str_regrow(o->str, cap); + o->data = o->str->data; + o->cap = (int64_t)o->str->cap; +} + +/* A global substitution with a literal ASCII replacement through the + * build-time replacer of its (pattern, replacement) pair: on an ASCII + * subject the replacer writes each gap and the replacement into the output + * and continues the search after every match without returning. Every + * other subject, and a plain-text pattern (string search), takes + * scr_replace_impl. */ +ScrStr *scr_regex_replace_with(ScrStr *s, ScrRegex *re, ScrStr *rep, + ScrRegexMatcherFn replacer) { + (void)scr_regex_bc(re); + if (scr_regex_lit(re) || !scr_text_ascii(s)) return scr_replace_impl(s, re, rep); + if (s->len > (size_t)INT32_MAX - 1) scr_regex_oom(); + re->last_index = 0; + ScrReReplaceOut o; + o.str = scr_str_alloc_raw(0, s->len + 64); + o.data = o.str->data; + o.len = 0; + o.cap = (int64_t)o.str->cap; + o.next = 0; + int32_t n = replacer((const uint8_t *)s->data, (int32_t)s->len, 0, (uint8_t **)(void *)&o); + if (n == 0) { + o.str->len = 0; + scr_str_release(o.str); + return scr_str_retain(s); + } + size_t tail = s->len - (size_t)o.next; + if ((int64_t)(tail + 1) > o.cap - o.len) scr_re_replace_out_reserve(&o, (int64_t)tail + 1); + scr_copy_bytes(o.data + o.len, s->data + o.next, tail); + o.len += (int64_t)tail; + o.str->len = (size_t)o.len; + o.data[o.len] = '\0'; + scr_str_note_ascii(o.str); + return o.str; +} + typedef struct { ScrDyn *arguments; int start, end; diff --git a/packages/runtime/src/scr_regex.h b/packages/runtime/src/scr_regex.h index cb61e49f35..502a29266e 100644 --- a/packages/runtime/src/scr_regex.h +++ b/packages/runtime/src/scr_regex.h @@ -80,6 +80,22 @@ ScrStr *scr_regex_flags(ScrRegex *re); /* +1 */ ScrStr *scr_regex_to_string(ScrRegex *re); /* +1 */ /* replace: first match without /g, every match with it. */ ScrStr *scr_regex_replace(ScrStr *s, ScrRegex *re, ScrStr *rep); +/* The output of a build-time replacer (regex-matchers.ts): the replacer + * appends each gap and the replacement at data[len] after reserving room, + * and leaves in `next` the subject index the unmatched tail starts at. */ +typedef struct { + int64_t len; + int64_t cap; + int64_t next; + char *data; /* at offset 24 on every target */ + ScrStr *str; +} ScrReReplaceOut; +/* Grows o->str so that `need` more bytes fit at o->data[o->len]. */ +void scr_re_replace_out_reserve(ScrReReplaceOut *o, int64_t need); +/* A global substitution with a literal ASCII replacement, through the + * replacer built for that pattern and replacement. Borrows all. */ +ScrStr *scr_regex_replace_with(ScrStr *s, ScrRegex *re, ScrStr *rep, + ScrRegexMatcherFn replacer); /* replace/replaceAll with a replacer closure of `arity` (at most * SCR_REPLACER_MAX_ARITY) parameters. Parameter i receives what Node passes * at position i: the match and captures below the capture count n, the diff --git a/packages/runtime/src/scr_string.c b/packages/runtime/src/scr_string.c index 1e7802fbbf..64a4772e5c 100644 --- a/packages/runtime/src/scr_string.c +++ b/packages/runtime/src/scr_string.c @@ -3514,6 +3514,8 @@ ScrStr *scr_bool_to_scrstr(bool b) { * decode is a whole char; an i addressing the LOW half of an astral pair * (unreachable from the iterator, which only lands on char starts) answers * the containing char. Out of range → empty string. Borrows s; +1. */ +static ScrStr *scr_str_char_of(const char *bytes, size_t adv); + ScrStr *scr_str_cp_at(ScrStr *s, double i) { double idx = scr_to_integer_or_infinity(i); if (!(idx >= 0)) return scr_str_empty(); @@ -3528,7 +3530,80 @@ ScrStr *scr_str_cp_at(ScrStr *s, double i) { if (off >= s->len) return scr_str_empty(); /* out of range */ size_t adv; (void)scr_utf8_decode(s->data + off, &adv); - return scr_str_from_span(s->data + off, adv); + return scr_str_char_of(s->data + off, adv); +} + +/* The one-character strings of U+0080..U+07FF (two-byte sequences, lead + * bytes 0xC2..0xDF), made immortal on first use and shared by every + * thread, as the ASCII ones are static: the string iterator returns them + * instead of a new string per character. A string is immortal before its + * slot publishes it; a thread that loses the race to fill a slot keeps its + * own immortal copy, so at most one extra string per slot and thread. The + * strings come from the system allocator, as other state that outlives a + * thread does: a worker's own memory goes back when it exits without + * sharing (scr_alloc.c), and the table outlives the worker that filled a + * slot. */ +static ScrStr *scr_two_byte_chars[0x800 - 0x80]; + +static ScrStr *scr_two_byte_char(const char *bytes) { + uint32_t cp = ((uint32_t)((unsigned char)bytes[0] & 0x1F) << 6) | ((unsigned char)bytes[1] & 0x3F); + ScrStr **slot = &scr_two_byte_chars[cp - 0x80]; + ScrStr *known = __atomic_load_n(slot, __ATOMIC_ACQUIRE); + if (known) return known; + ScrStr *made = calloc(1, sizeof(ScrStr) + 3); + if (!made) scr_oom(); + made->len = 2; + made->cap = 2; + memcpy(made->data, bytes, 2); + scr_str_hash_prime(made); + made->rc = SIZE_MAX; + scr_lsan_immortal(made); + ScrStr *expected = NULL; + if (__atomic_compare_exchange_n(slot, &expected, made, false, __ATOMIC_ACQ_REL, + __ATOMIC_ACQUIRE)) + return made; + return expected; +} + +/* The character whose encoding starts at bytes and is adv bytes long (+1, + * or immortal). */ +static ScrStr *scr_str_char_of(const char *bytes, size_t adv) { + unsigned char lead = (unsigned char)bytes[0]; + if (adv == 1 && lead < 0x80) return (ScrStr *)&scr_ascii1[lead]; + if (adv == 2) return scr_two_byte_char(bytes); + return scr_str_from_span(bytes, adv); +} + +ScrStr *scr_str_cp_at_byte(ScrStr *s, uint64_t b) { + if (b >= s->len) return scr_str_empty(); + size_t adv; + (void)scr_utf8_decode(s->data + b, &adv); + return scr_str_char_of(s->data + b, adv); +} + +ScrArr *scr_str_code_points(ScrStr *s) { + /* Every byte but a continuation byte starts one character; a lone + * surrogate's three bytes are one character. */ + size_t count = s->len; + if (!s->ascii) + for (size_t b = 0; b < s->len; b++) count -= ((unsigned char)s->data[b] & 0xC0) == 0x80; + ScrArr *out = scr_arr_new_inline(SCR_ELEM_STR, count, false); + size_t k = 0; + for (size_t b = 0; b < s->len;) { + unsigned char c = (unsigned char)s->data[b]; + if (c < 0x80) { + out->data[k++] = (uint64_t)(uintptr_t)&scr_ascii1[c]; + b++; + continue; + } + size_t adv; + (void)scr_utf8_decode(s->data + b, &adv); + out->data[k++] = (uint64_t)(uintptr_t)scr_str_char_of(s->data + b, adv); + b += adv; + } + out->len = (uint32_t)count; + out->filled = (uint32_t)count; + return out; } /* ── the URI component codecs (encodeURIComponent/decodeURIComponent) ── diff --git a/packages/runtime/src/scr_string_methods.h b/packages/runtime/src/scr_string_methods.h index 01ccdd8ae6..2e69ee6ac9 100644 --- a/packages/runtime/src/scr_string_methods.h +++ b/packages/runtime/src/scr_string_methods.h @@ -150,6 +150,15 @@ ScrStr *scr_str_char_at(ScrStr *s, double i); * (two units); the consuming loop advances by the result's length. * Out of range → empty string. Returns +1. */ ScrStr *scr_str_cp_at(ScrStr *s, double i); +/* The string iterator's step over the WTF-8 bytes: the code-point + * character whose encoding starts at byte offset b, a character start + * below s->len (a lone surrogate is its own three-byte character). The + * compiler answers ASCII bytes inline and calls this for the rest. + * Returns +1. */ +ScrStr *scr_str_cp_at_byte(ScrStr *s, uint64_t b); +/* [...s] and Array.from(s): the code-point characters of s in order, as a + * fresh string array of exactly that length. Returns +1. */ +struct ScrArr *scr_str_code_points(ScrStr *s); /* trimStart()/trimEnd(): the one-sided halves of trim. Returns +1. */ ScrStr *scr_str_trim_start(ScrStr *s); diff --git a/packages/runtime/test/test_array.c b/packages/runtime/test/test_array.c index 7ca0ebd1a1..9126c154fd 100644 --- a/packages/runtime/test/test_array.c +++ b/packages/runtime/test/test_array.c @@ -2513,6 +2513,84 @@ static void test_states_on_demand(void) { #endif } +/* Elements outside the element type (SCR_ARR_FOREIGN): stores, reads, + * copies, sorting and removal keep the stored value's count balanced. */ +static void test_foreign_elements(void) { + ScrStr *text = scr_str_new("x", 1); + ScrDyn *x = scr_dyn_new_str(text); + scr_str_release(text); + ScrArr *a = scr_arr_new(SCR_ELEM_F64, 0); + scr_arr_push_f64(a, 1); + scr_arr_push_f64(a, 2); + check(scr_arr_set_foreign(a, 2, x), "foreign store appends"); + check_f64(scr_arr_len(a), 3, "foreign store extends the length"); + check_f64(scr_arr_state(a, 2), SCR_ARR_FOREIGN, "foreign element state"); + ScrDyn *got = scr_arr_get_foreign(a, 2); + check(got == x, "get_foreign answers the stored value"); + scr_dyn_release(got); + check(scr_arr_get_foreign(a, 0) == NULL, "a number is not a foreign element"); + check(isnan(scr_arr_get_number(a, 2)), "numeric read of a foreign 'x' is NaN"); + check(scr_arr_set_foreign(a, 3, scr_dyn_undefined()), "undefined store"); + check_f64(scr_arr_state(a, 3), SCR_ARR_UNDEFINED, "undefined is a present undefined"); + ScrStr *sep = scr_str_new(",", 1); + ScrStr *joined = scr_arr_join(a, sep); + check(joined->len == 6 && memcmp(joined->data, "1,2,x,", 6) == 0, "join converts foreign"); + scr_str_release(joined); + ScrArr *copy = scr_arr_slice(a, 0, INFINITY); + check_f64(scr_arr_state(copy, 2), SCR_ARR_FOREIGN, "slice keeps a foreign element"); + scr_arr_release(copy); + ScrArr *head = scr_arr_new(SCR_ELEM_F64, 0); + scr_arr_set_foreign(head, 0, x); + scr_arr_push_f64(head, 2); + scr_arr_push_f64(head, 3); + copy = scr_arr_slice(head, 1, INFINITY); + check_f64(scr_arr_get_f64(copy, 1), 3, "slice past a foreign element"); + scr_arr_release(copy); + scr_arr_release(head); + scr_arr_set_f64(a, 2, 5); + check_f64(scr_arr_get_f64(a, 2), 5, "a number replaces a foreign element"); + scr_arr_set_foreign(a, 2, x); + scr_arr_set_len(a, 1); + check_f64(scr_arr_len(a), 1, "truncation drops foreign elements"); + scr_arr_set_foreign(a, 1, x); + check(scr_arr_delete(a, 1), "delete removes a foreign element"); + scr_arr_set_foreign(a, 1, x); + uint64_t slot = 0; + check(scr_arr_pop_state(a, &slot) == SCR_ARR_FOREIGN, "pop answers the foreign state"); + check((ScrDyn *)(uintptr_t)slot == x, "pop moves the value out"); + scr_dyn_release((ScrDyn *)(uintptr_t)slot); + for (int i = 0; i < 12; i++) scr_arr_set_foreign(a, (double)(1 + i), x); + double seven = 7; + uint64_t inserted[1]; + memcpy(&inserted[0], &seven, sizeof seven); + scr_arr_splice_drop(a, 1, 10, 1, inserted); + check_f64(scr_arr_len(a), 4, "splice removes foreign elements"); + check_f64(scr_arr_get_f64(a, 1), 7, "splice inserts after them"); + scr_arr_release(a); + + ScrStr *letter = scr_str_new("a", 1); + ScrDyn *a_value = scr_dyn_new_str(letter); + scr_str_release(letter); + ScrArr *s = scr_arr_new(SCR_ELEM_F64, 0); + scr_arr_push_f64(s, 10); + scr_arr_push_f64(s, 9); + scr_arr_set_foreign(s, 2, a_value); + scr_arr_push_f64(s, 1); + scr_arr_release(scr_arr_sort_primitive(s, false)); + check(scr_arr_get_f64(s, 0) == 1 && scr_arr_get_f64(s, 1) == 10 && scr_arr_get_f64(s, 2) == 9, + "default sort orders numbers by their strings"); + check_f64(scr_arr_state(s, 3), SCR_ARR_FOREIGN, "default sort orders 'a' after the digits"); + scr_arr_release(s); + scr_dyn_release(a_value); + + ScrArr *flags = scr_arr_new(SCR_ELEM_BOOL, 0); + check(!scr_arr_set_foreign(flags, 0, x), "boolean arrays hold no foreign elements"); + scr_arr_release(flags); + scr_str_release(sep); + check(x->rc == 1, "every foreign store released its count"); + scr_dyn_release(x); +} + int main(int argc, char **argv) { if (argc > 1) { ScrArr *a = scr_arr_new(SCR_ELEM_F64, 0); @@ -2573,6 +2651,8 @@ int main(int argc, char **argv) { test_head_room(); test_states_on_demand(); + test_foreign_elements(); + fprintf(stderr, "%ld/%ld cases passed\n", total - failed, total); return failed == 0 ? 0 : 1; } diff --git a/packages/runtime/test/test_bytes.c b/packages/runtime/test/test_bytes.c index 8c66029489..1aea7a2d04 100644 --- a/packages/runtime/test/test_bytes.c +++ b/packages/runtime/test/test_bytes.c @@ -216,6 +216,135 @@ static void test_encodings(void) { scr_str_release(b64); } +/* The length of the well-formed WTF-8 sequence at in[i], or 0. */ +static size_t ref_seq_len(const uint8_t *in, size_t n, size_t i) { + uint8_t b = in[i]; + if (b < 0x80) return 1; + size_t len; + uint8_t lo = 0x80, hi = 0xbf; + if (b >= 0xc2 && b <= 0xdf) len = 2; + else if (b >= 0xe0 && b <= 0xef) len = 3; + else if (b >= 0xf0 && b <= 0xf4) len = 4; + else return 0; + if (b == 0xe0) lo = 0xa0; + if (b == 0xf0) lo = 0x90; + if (b == 0xf4) hi = 0x8f; + if (n - i < len || in[i + 1] < lo || in[i + 1] > hi) return 0; + for (size_t k = 2; k < len; k++) + if ((in[i + k] & 0xc0) != 0x80) return 0; + return len; +} + +/* The UTF-16 units of string storage, one at a time: a well-formed + * sequence gives its code point's units and any other byte stands for + * U+0000..U+00FF (a raw HTTP header byte). */ +static size_t ref_units(const uint8_t *in, size_t n, uint16_t *out) { + size_t k = 0; + for (size_t i = 0; i < n;) { + size_t len = ref_seq_len(in, n, i); + if (len == 0) { + out[k++] = in[i++]; + continue; + } + uint32_t cp = len == 1 ? in[i] : (uint32_t)(in[i] & (0x7f >> len)); + for (size_t j = 1; j < len; j++) cp = (cp << 6) | (in[i + j] & 0x3f); + i += len; + if (cp > 0xffff) { + out[k++] = (uint16_t)(0xd800 + ((cp - 0x10000) >> 10)); + out[k++] = (uint16_t)(0xdc00 + ((cp - 0x10000) & 0x3ff)); + } else { + out[k++] = (uint16_t)cp; + } + } + return k; +} + +/* Encodes storage that is not well-formed (raw header bytes become string + * storage unchecked) and compares latin1, ascii and utf16le with the + * reference; hex and base64 decode the latin1 bytes. ASan checks that no + * path reads or writes outside either buffer. */ +/* encs: latin1, ascii, utf16le, hex, base64. */ +static void check_raw_encodings(ScrStr *const *encs, const uint8_t *in, size_t n, const char *what) { + ScrStr *latin1 = encs[0], *ascii = encs[1], *utf16le = encs[2], *hex = encs[3], *b64 = encs[4]; + ScrStr *s = scr_str_new((const char *)in, n); + uint16_t units[64]; + size_t k = ref_units(in, n, units); + ScrStr *narrow[] = {latin1, ascii}; + for (size_t e = 0; e < 2; e++) { + ScrBytes *b = scr_bytes_from_str(s, narrow[e]); + bool ok = b->len == k; + for (size_t j = 0; ok && j < k; j++) ok = b->data[j] == (uint8_t)units[j]; + check(ok, what); + scr_bytes_release(b); + } + ScrBytes *wide = scr_bytes_from_str(s, utf16le); + bool ok = wide->len == 2 * k; + for (size_t j = 0; ok && j < k; j++) + ok = wide->data[2 * j] == (uint8_t)units[j] && wide->data[2 * j + 1] == units[j] >> 8; + check(ok, what); + scr_bytes_release(wide); + scr_bytes_release(scr_bytes_from_str(s, hex)); + scr_bytes_release(scr_bytes_from_str(s, b64)); + scr_str_release(s); +} + +static void test_raw_storage_encodings(void) { + /* Each pattern at every offset in an eight-byte word, alone and followed + * by ASCII, two-byte characters or more of the pattern. */ + static const struct { const char *bytes; size_t n; } patterns[] = { + {"\x80", 1}, {"\x80\x81\xbf\x90", 4}, {"\xc3", 1}, {"\xc3\xa9", 2}, + {"\xc3\xa9\xc3", 3}, {"\xc0\x80\xc1\xbf", 4}, {"\xe9\x41\xe9", 3}, + {"\xf0\x9f\x41", 3}, {"\xe2\x82", 2}, {"\xe2\x82\xac", 3}, + {"\xf0\x9f\x98\x80", 4}, {"\xed\xa0\x80", 3}, {"\xff\xfe\xf5", 3}, + {"\xdf\xbf\x80\x80", 4}, {"\xc2\x80\xc2", 3}, {"x\xe9y\x80\x80z\xdf", 7}, + }; + static const struct { const char *bytes; size_t n; } tails[] = { + {"", 0}, {"bcdefghij", 9}, {"\xc3\xa9\xc3\xa9\xc3\xa9\xc3\xa9", 8}, {"\x80\xc3\xa9", 3}, + }; + ScrStr *encs[] = {S("latin1"), S("ascii"), S("utf16le"), S("hex"), S("base64")}; + uint8_t buf[64]; + char what[96]; + for (size_t p = 0; p < sizeof patterns / sizeof patterns[0]; p++) { + for (size_t offset = 0; offset < 17; offset++) { + for (size_t t = 0; t < sizeof tails / sizeof tails[0]; t++) { + for (size_t lead = 0; lead < 2; lead++) { + /* lead: the offset is two-byte characters instead of ASCII. */ + size_t n = 0; + for (size_t j = 0; j < offset; j++) { + if (lead && j + 1 < offset) { + buf[n++] = 0xc3; + buf[n++] = 0xa9; + j++; + } else { + buf[n++] = 'a'; + } + } + memcpy(buf + n, patterns[p].bytes, patterns[p].n); + n += patterns[p].n; + memcpy(buf + n, tails[t].bytes, tails[t].n); + n += tails[t].n; + snprintf(what, sizeof what, "raw storage encodings: pattern %zu offset %zu tail %zu lead %zu", + p, offset, t, lead); + check_raw_encodings(encs, buf, n, what); + } + } + } + } + /* Random bytes, mostly high, of every length up to 40. */ + uint32_t seed = 12345; + for (int round = 0; round < 4000; round++) { + size_t n = (size_t)round % 41; + for (size_t j = 0; j < n; j++) { + seed = seed * 1103515245u + 12345u; + uint8_t r = (uint8_t)(seed >> 16); + buf[j] = (seed >> 27) & 1 ? (uint8_t)(0x80 | r) : (uint8_t)(r & 0x7f); + } + snprintf(what, sizeof what, "raw storage encodings: random round %d", round); + check_raw_encodings(encs, buf, n, what); + } + for (size_t e = 0; e < sizeof encs / sizeof encs[0]; e++) scr_str_release(encs[e]); +} + static void test_concat_and_u32be(void) { ScrArr *list = scr_arr_new(SCR_ELEM_BYTES, 0); ScrBytes *a = scr_bytes_new(SCR_BYTES_U8, 2); @@ -445,6 +574,7 @@ int main(int argc, char **argv) { scr_bytes_test_bulk_storage(); scr_bytes_test_buffer_pool(); test_encodings(); + test_raw_storage_encodings(); test_concat_and_u32be(); test_from_arr_and_random(); test_zlib_roundtrip(); diff --git a/packages/runtime/test/test_cycle.c b/packages/runtime/test/test_cycle.c index 0ab827e476..dc48abd263 100644 --- a/packages/runtime/test/test_cycle.c +++ b/packages/runtime/test/test_cycle.c @@ -575,6 +575,60 @@ static void check_program_nursery_threshold(void) { scr_collect_cycles(); } +/* How many dead rings the release path takes to run its next pass, from + * the end of the previous one: the nursery trigger in effect. */ +static size_t nursery_window(void) { + size_t before = freed, n = 0; + while (freed == before && n < 65536) { + release_live(make_ring(1)); + n++; + } + check(freed - before == n, "a nursery pass left dead rings behind"); + return n; +} + +/* A scheduled full pass that reclaims nothing: its only candidate is an + * older live leaf. */ +static void scheduled_pass_finding_nothing(Node *leaf) { + size_t before = freed; + read_live(leaf); + for (size_t i = 0; i < configured_nursery_threshold(); i++) + scr_cyc_collect_scheduled(); + check(freed == before, "a scheduled pass over a live leaf freed something"); +} + +/* Unpinned, the nursery trigger starts at 256 candidates. Each scheduled + * full pass that reclaims promoted rings doubles it, up to 4096; each one + * that reclaims nothing after the nursery passes freed rings halves it, + * down to 256. */ +static void check_adaptive_nursery_threshold(void) { + if (getenv("SCR_CYCLE_THRESHOLD") != NULL) return; /* it pins the trigger */ + scr_cyc_set_nursery_threshold(0); + scr_collect_cycles(); + check(nursery_window() == 256, "the unpinned nursery trigger did not start at 256"); + static const size_t widened[] = { 512, 1024, 2048, 4096, 4096 }; + for (size_t i = 0; i < sizeof(widened) / sizeof(widened[0]); i++) { + collect_dead_mature_rings(); + check(nursery_window() == widened[i], + "a full pass reclaiming promoted rings did not double the nursery trigger"); + } + + static const size_t narrowed[] = { 2048, 1024, 512, 256, 256 }; + Node *leaf = make_leaf(); + read_live(leaf); + scr_collect_cycles(); /* promote */ + for (size_t i = 0; i < sizeof(narrowed) / sizeof(narrowed[0]); i++) { + scheduled_pass_finding_nothing(leaf); + check(nursery_window() == narrowed[i], + "a full pass reclaiming nothing did not halve the nursery trigger"); + } + leaf->rc--; + scr_cyc_on_dead(leaf); + node_free(leaf); + scr_cyc_set_nursery_threshold(0); + scr_collect_cycles(); +} + /* The phase sweep (scr_collect_cycles_phase) is for a caller that knows a * phase's data has just died: it sweeps, and the growth schedule starts * over. After passes that find nothing have widened the old level's @@ -1506,6 +1560,9 @@ static void check_acyclic_objects(void) { } int main(void) { + /* The scheduling checks count candidates against a fixed nursery trigger; + * check_adaptive_nursery_threshold covers the unpinned one. */ + scr_cyc_set_nursery_threshold((long)configured_nursery_threshold()); check_inline_buffering(); check_old_garbage_reclaimed_while_growing(); check_older_rings_into_old(); @@ -1541,6 +1598,7 @@ int main(void) { check_acyclic_objects(); check_program_growth_cap(); check_program_nursery_threshold(); + check_adaptive_nursery_threshold(); printf("cycle collection checks passed: threshold=%zu growth_cap=%zu\n", configured_nursery_threshold(), configured_growth_cap()); return 0; diff --git a/packages/threads/package.json b/packages/threads/package.json index c29967cc67..6ebcf5b8bf 100644 --- a/packages/threads/package.json +++ b/packages/threads/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/threads", - "version": "0.4.4", + "version": "0.4.5", "description": "Zero-copy publication of immutable object graphs to worker threads (scriptc), with a deep-freezing Node.js implementation", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/types/package.json b/packages/types/package.json index db88b4b2fc..b2aa805bee 100644 --- a/packages/types/package.json +++ b/packages/types/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/types", - "version": "0.4.4", + "version": "0.4.5", "description": "Contract types for scriptc: type-only annotations (Int32, NoThrow, Immutable) that scriptc checks and optimizes, with no runtime semantics", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/tests/corpus/1561-forof-strings.ts b/tests/corpus/1561-forof-strings.ts index 9d16efd9a7..0ea09e755a 100644 --- a/tests/corpus/1561-forof-strings.ts +++ b/tests/corpus/1561-forof-strings.ts @@ -1,7 +1,7 @@ // for-of over STRINGS: JS's string iterator walks code POINTS — each pass // binds the whole character, so astral chars arrive as one two-unit string -// (where charAt/slice would truncate the halves). Desugars over the UTF-16 -// cursor with the cpAt step intrinsic. +// (where charAt/slice would truncate the halves). Desugars over a byte +// cursor with the cpAtByte step intrinsic. for (const ch of "héllo 🎉x") console.log(ch, ch.length); // The empty string iterates zero times. diff --git a/tests/corpus/6067-view-holes-in-typed-arrays.ts b/tests/corpus/6067-view-holes-in-typed-arrays.ts new file mode 100644 index 0000000000..831b3be7a1 --- /dev/null +++ b/tests/corpus/6067-view-holes-in-typed-arrays.ts @@ -0,0 +1,115 @@ +// Holes written through an `unknown[]` or `any` view of a typed array are +// holes of the typed array: for...of visits them as undefined and the find +// family passes them to its callback, as in Node. The views reach the array +// through a binding, an `any` parameter, a generic instantiated with +// `unknown`, a record field, a class field and a nested array. + +const numbers: number[] = [1, 2]; +const numbersView: unknown[] = numbers; +numbersView[3] = 5; +for (const value of numbers) console.log("numbers", value, typeof value); +console.log(numbers.findIndex((value) => value === undefined)); +console.log(numbers.findLastIndex((value) => value === undefined)); +console.log(numbers.find((value) => value !== undefined && value > 1)); + +const grown: string[] = ["a"]; +const grownView: unknown[] = grown; +grownView.length = 3; +for (const value of grown) console.log("grown", value); + +function growAny(values: any): void { + values[2] = 9; +} +const passed: number[] = [7]; +growAny(passed); +for (const value of passed) console.log("passed", value); + +function keep(value: T): T { + return value; +} +const kept: number[] = [3]; +(keep(kept) as unknown[])[2] = 4; +for (const value of kept) console.log("kept", value); + +const boxed: number[] = [1]; +const box: unknown = { values: boxed }; +(box as { values: unknown[] }).values[2] = 3; +for (const value of boxed) console.log("boxed", value); + +class Holder { + values: number[] = [1, 2]; +} +const holder = new Holder(); +const holderView: unknown = holder; +(holderView as { values: unknown[] }).values[3] = 4; +for (const value of holder.values) console.log("holder", value); + +const nested: number[][] = [[1]]; +const nestedView: unknown[][] = nested; +nestedView[0]![2] = 3; +for (const value of nested[0]!) console.log("nested", value); + +// A named property written through a view is not an element. +const named: number[] = [1, 2]; +const bag = named as unknown as Record; +bag["label"] = "x"; +for (const value of named) console.log("named", value); +console.log(named.length, bag["label"]); + +// Views that reach dynamic code through class field initializers, +// parameter defaults, logical assignments, yield and throw, and through +// standard library calls that keep their argument or pass it to a +// callback: Promise.resolve, push onto an `unknown[]`, Map.prototype.set +// and a JSON.stringify replacer. +const fieldNumbers: number[] = [1, 2]; +class FieldView { + view: unknown[] = fieldNumbers; +} +new FieldView().view[3] = 4; +for (const value of fieldNumbers) console.log("field", value); +const defaultNumbers: number[] = [1]; +function extend(view: unknown[] = defaultNumbers): void { + view.length = 3; +} +extend(); +console.log(defaultNumbers.findIndex((value) => value === undefined), defaultNumbers.length); +let assigned: unknown[] | undefined; +const assignedNumbers: number[] = [5]; +assigned ??= assignedNumbers; +assigned[2] = 6; +for (const value of assignedNumbers) console.log("assigned", value); +const yieldedNumbers: number[] = [7]; +function* yieldViews(): Generator { + yield yieldedNumbers; +} +for (const view of yieldViews()) view[2] = 8; +console.log(yieldedNumbers.findLastIndex((value) => value === undefined), yieldedNumbers.length); +const thrownNumbers: number[] = [9]; +try { + throw thrownNumbers; +} catch (error) { + (error as unknown[])[2] = 10; +} +for (const value of thrownNumbers) console.log("thrown", value); +const retainedNumbers: number[] = [12]; +const retained: unknown[] = []; +retained.push(retainedNumbers); +(retained[0] as unknown[])[2] = 13; +for (const value of retainedNumbers) console.log("kept", value); +const mappedNumbers: number[] = [14]; +const byName = new Map(); +byName.set("a", mappedNumbers); +(byName.get("a") as unknown[])[2] = 15; +console.log(mappedNumbers.findIndex((value) => value === undefined), mappedNumbers.length); +const replacedNumbers: number[] = [16]; +JSON.stringify({ replacedNumbers: replacedNumbers }, (key, value: unknown) => { + if (key === "replacedNumbers") (value as unknown[])[2] = 17; + return value; +}); +for (const value of replacedNumbers) console.log("replaced", value); +const resolvedNumbers: number[] = [11]; +void Promise.resolve(resolvedNumbers).then((view) => { + (view as unknown[])[2] = 18; + for (const value of resolvedNumbers) console.log("resolved", value); +}); + diff --git a/tests/corpus/6068-foreign-elements-through-views.ts b/tests/corpus/6068-foreign-elements-through-views.ts new file mode 100644 index 0000000000..1b23f6fa7e --- /dev/null +++ b/tests/corpus/6068-foreign-elements-through-views.ts @@ -0,0 +1,202 @@ +// A number[] or string[] seen as `unknown[]` holds the values written +// through the view that its element type cannot hold, as in Node. The view +// and the typed array are the same array: length, the view's reads and +// iteration, printing, JSON, join, searches, copies, Math over a spread, +// sorting and truncation see the written values. + +const numbers: number[] = [1, 2]; +const numbersView: unknown[] = numbers; +numbersView.push("x"); +numbersView[0] = true; +console.log(numbers.length, numbersView.length, numbers === numbersView); +console.log(numbersView[0], numbersView[2], typeof numbersView[2]); +console.log(numbers, String(numbers), numbers.join("|"), JSON.stringify(numbers)); +for (const value of numbersView) console.log("value", value, typeof value); +console.log(numbers.indexOf(2), numbers.includes(2)); +console.log([...numbers], numbers.slice(1), numbers.slice(1, 2), numbers.concat([9])); + +// Math over a spread applies ToNumber to every element, as in Node. +const digits: number[] = [5]; +const digitsView: unknown[] = digits; +digitsView.push("4", null); +console.log(Math.max(...digits), Math.min(...digits), Math.hypot(...digits)); + +// Present undefined, null and objects keep their own values. +const mixed: number[] = [1]; +const mixedView: unknown[] = mixed; +mixedView.push(undefined, null, { a: 1 }, [2, 3]); +console.log(mixed, mixed.length, 1 in mixed, mixedView[2] === null); +console.log(JSON.stringify(mixed), mixed.join("-")); + +// Writing a number over a foreign element makes it a number again. +const replaced: number[] = [1, 2]; +const replacedView: unknown[] = replaced; +replacedView.push("x"); +replaced[2] = 5; +console.log(replaced, replacedView, replaced[2] + 1); + +// Default sort orders every element by its string, as in Node. +const sorted: number[] = [10, 9, 1]; +const sortedView: unknown[] = sorted; +sortedView.push("b", "a"); +console.log(sorted.sort(), sorted.length, sortedView[4]); + +// Truncation drops written values; growing again leaves holes. +const truncated: number[] = [1]; +const truncatedView: unknown[] = truncated; +truncatedView.push("x", "y"); +truncated.length = 1; +truncated.length = 3; +console.log(truncated, truncatedView.length); + +// string[] holds values of other types the same way. +const words: string[] = ["a"]; +const wordsView: unknown[] = words; +wordsView.push(1, false); +console.log(words, words.join("-"), words.length, typeof wordsView[1]); + +// A conversion that changes the array does not change the spread's values, +// and toSorted reads holes as undefined. +const grown: number[] = [3, 1]; +const grownView: unknown[] = grown; +grownView.push({ + valueOf: () => { + for (let i = 0; i < 64; i++) grown.push(i); + grown.length = 1; + return 2; + }, +}); +console.log(Math.max(...grown), Math.min(...grown), grown.length); +const holey: string[] = ["c", "a"]; +const holeyView: unknown[] = holey; +holeyView[4] = 7; +holeyView[6] = undefined; +console.log(holey.toSorted(), holey.length); +holey.sort(); +console.log(holey, holey.length); + +// Conversions during an in-place sort that change the array's length: +// elements added past the sorted length stay. +const appended: number[] = [3, 1, 2]; +const appendedView: unknown[] = appended; +let appendedCalls = 0; +appendedView[1] = { + toString(): string { + if (appendedCalls++ === 0) appended.push(7, 8); + return "1"; + }, +}; +appended.sort(); +console.log(appended.length, appended.slice(3), String(appendedView[0]), appended[2]); +const reduced: number[] = [3, 1, 2]; +const reducedView: unknown[] = reduced; +reducedView[4] = 4; +let reducedCalls = 0; +reducedView[1] = { + toString(): string { + if (reducedCalls++ === 0) reduced.length = 1; + return "1"; + }, +}; +reduced.sort(); +console.log(reduced.length, 3 in reduced, 4 in reduced, reduced[1], reduced[2]); +const padded: number[] = [3, 1, 2]; +const paddedView: unknown[] = padded; +paddedView.length = 5; +let paddedCalls = 0; +paddedView[1] = { + toString(): string { + if (paddedCalls++ === 0) padded.push(9); + return "1"; + }, +}; +padded.sort(); +console.log(padded.length, 3 in padded, 4 in padded, padded[5]); +const spread: number[] = [4, 3]; +const spreadView: unknown[] = spread; +let spreadCalls = 0; +spreadView.push({ + toString(): string { + if (spreadCalls++ === 0) { + spreadView[1000] = 1; + for (let i = 0; i < 64; i++) spread.push(i); + } + return "2"; + }, +}); +console.log(spread.toSorted().length, spread.length); +spread.sort(); +console.log(spread.length, spread[0], spread[1], spread[3], 999 in spread, spread[1000]); +const grownMath: number[] = [1]; +const grownMathView: unknown[] = grownMath; +grownMathView[10] = { + valueOf(): number { + grownMathView[5000] = 7; + for (let i = 0; i < 256; i++) grownMath.push(i); + return 3; + }, +}; +console.log(Math.max(...grownMath), grownMath.length); + +// A conversion that removes the element it converts. +const shrinking: number[] = [1]; +const shrinkingView: unknown[] = shrinking; +shrinkingView.push({ + toString: () => { + shrinking.length = 0; + return "gone"; + }, +}); +console.log(shrinking.join("+"), shrinking.length); +const cut: string[] = ["a", "b"]; +const cutView: unknown[] = cut; +cutView.splice(1, 0, { toString: () => ((cut.length = 0), "x") }); +console.log(JSON.stringify(cut.join("-")), cut.length); + +// JSON of elements whose getters change the array or record holding them. +const fillers: number[][] = []; +const rows: number[][] = [[1], [2]]; +const rowView: unknown[] = rows[0]!; +rowView.push({ + get grown() { + for (let i = 0; i < 64; i++) rows.push([i]); + rows.length = 3; + // Reuse the memory the old storage held. + for (let i = 0; i < 64; i++) fillers.push([i, i]); + return true; + }, +}); +console.log(JSON.stringify(rows), rows.length); +const holder = { values: [5] as number[], name: "h" }; +const holderView: unknown[] = holder.values; +holderView.push({ + get replaced() { + holder.values = [9]; + for (let i = 0; i < 64; i++) fillers.push([i]); + return true; + }, +}); +console.log(JSON.stringify(holder), JSON.stringify(holder)); + +// toSorted keeps the full length and reads holes as undefined; indexOf, +// lastIndexOf and includes skip the undefined and foreign elements. +const searched: string[] = ["b", "a"]; +const searchedView: unknown[] = searched; +searchedView[9] = 4; +searchedView[5] = undefined; +console.log(searched.indexOf("b"), searched.lastIndexOf("b"), searched.lastIndexOf("a", 8)); +console.log(searched.includes("a"), searched.lastIndexOf("z")); +const searchedSorted = searched.toSorted(); +console.log(searchedSorted.length, searchedSorted.slice(0, 4), searchedSorted.lastIndexOf("b")); +console.log(searchedSorted.indexOf("a"), searchedSorted.includes("b")); +const counts: number[] = [3, 1]; +const countsView: unknown[] = counts; +countsView[6] = "x"; +countsView[4] = 2; +const countsSorted = counts.toSorted(); +console.log(countsSorted.length, countsSorted, countsSorted.lastIndexOf(3), counts.lastIndexOf(2)); +const far: number[] = [5]; +const farView: unknown[] = far; +farView[70000] = "z"; +const farSorted = far.toSorted(); +console.log(farSorted.length, farSorted.slice(0, 3), farSorted.slice(69999), farSorted.lastIndexOf(5)); diff --git a/tests/corpus/6069-foreign-element-writers.ts b/tests/corpus/6069-foreign-element-writers.ts new file mode 100644 index 0000000000..c3accaa09f --- /dev/null +++ b/tests/corpus/6069-foreign-element-writers.ts @@ -0,0 +1,54 @@ +// Values of other types written into typed arrays by code that sees them as +// `any` or `unknown[]`: an `any` parameter, a record field, methods that +// reorder the whole array through the view, and an `unknown[]` parameter. The typed +// array keeps every written value, and the view reads them back. + +function label(values: any): void { + values.push("label"); + values[0] = "first"; +} +const labelled: number[] = [1, 2]; +label(labelled); +console.log(labelled, labelled.length, JSON.stringify(labelled)); + +interface Box { + values: unknown[]; +} +const boxed: number[] = [1, 2, 3]; +const box: Box = { values: boxed }; +box.values[1] = "two"; +box.values.push(null); +console.log(boxed, box.values === boxed); + +// reverse and sort through the view rewrite the typed array. +const filled: number[] = [1, 2, 3, 4]; +const filledView: unknown[] = filled; +filledView[1] = "z"; +filledView[2] = "z"; +console.log(filled); +filledView.reverse(); +console.log(filled, filled[0], filled[3]); +filledView.sort(); +console.log(filled); + +function setAt(values: unknown[], index: number, value: unknown): void { + values[index] = value; +} +const assigned: number[] = [1, 2]; +setAt(assigned, 1, "b"); +setAt(assigned, 3, "d"); +console.log(assigned, assigned.length); + +// The view's own methods and iteration see the same elements. +const shared: number[] = [1, 2]; +const sharedView: unknown[] = shared; +sharedView.push("x"); +console.log(sharedView.map((value) => typeof value).join(",")); +console.log(sharedView.filter((value) => typeof value === "number").length); +for (const value of sharedView) console.log("view", value); +for (const key in shared) console.log("key", key); +console.log(Object.keys(sharedView), sharedView.indexOf("x"), sharedView.includes("x")); + +// Writing a number through the view stores an ordinary element. +sharedView[2] = 3; +console.log(shared, shared[2] + 1); diff --git a/tests/corpus/7515-net-socket-backpressure.ts b/tests/corpus/7515-net-socket-backpressure.ts new file mode 100644 index 0000000000..260726a153 --- /dev/null +++ b/tests/corpus/7515-net-socket-backpressure.ts @@ -0,0 +1,136 @@ +// net.Socket write() backpressure, as in Node 24: write answers false when +// the writes not yet completely sent reach the 65536-byte high-water mark, +// each counting in full, which a write made before the connection does; +// writableNeedDrain stays true until 'drain', and 'drain' fires once the +// buffered bytes are sent. The same holds while a custom lookup is pending, +// writableLength counts a large write the kernel took in part in full, +// untyped code reads the same properties, and a write whose send fails +// after the peer reset the connection answers false. An echo client that +// waits for 'drain' when write answers false moves every byte. +import { connect, createServer, Socket } from "node:net"; +import type { LookupAddress, LookupAllOptions, LookupOneOptions } from "node:dns"; + +const CHUNK = 65536; +const TOTAL = 64 * CHUNK; +const chunk = Buffer.alloc(CHUNK, 120); + +function untyped(socket: any): string { + return `${socket.writableLength} ${socket.writableNeedDrain} ${socket.writableHighWaterMark}`; +} + +function echo(port: number): Promise { + return new Promise((resolve) => { + const client = connect(port, "127.0.0.1"); + console.log("small write before connect", client.write("hi"), client.writableLength); + console.log("large write before connect", client.write(Buffer.alloc(70000, 121)), client.writableLength); + console.log("need drain", client.writableNeedDrain, "untyped", untyped(client)); + let sent = 70002; + let received = 0; + const pump = (): void => { + while (sent < TOTAL) { + sent += CHUNK; + if (!client.write(chunk)) { + client.once("drain", pump); + return; + } + } + client.end(); + }; + client.once("drain", () => { + console.log("drain fired, need drain", client.writableNeedDrain, "pending", client.writableLength); + pump(); + }); + client.on("data", (data: Buffer) => { + received += data.length; + }); + client.on("end", () => { + console.log("received", received, "of", sent); + resolve(); + }); + }); +} + +function lookupPending(port: number): Promise { + return new Promise((resolve) => { + const client = connect({ + port, + host: "localhost", + lookup: ( + _host: string, + _options: LookupOneOptions | LookupAllOptions | number | undefined, + callback: (err: NodeJS.ErrnoException | null, address: string | LookupAddress[], family?: number) => void, + ) => { + const answer: LookupAddress[] = [{ address: "127.0.0.1", family: 4 }]; + setTimeout(() => callback(null, answer), 20); + }, + }); + console.log("lookup write", client.write(Buffer.alloc(70000, 97))); + console.log("lookup state", client.writableLength, client.writableNeedDrain, "untyped", untyped(client)); + client.once("drain", () => { + console.log("lookup drain", client.writableLength, client.writableNeedDrain); + client.destroy(); + resolve(); + }); + }); +} + +function largeWrite(port: number): Promise { + return new Promise((resolve) => { + const client = connect(port, "127.0.0.1", () => { + const size = 16 * 1024 * 1024; + console.log("large write", client.write(Buffer.alloc(size, 98))); + console.log("large state", client.writableLength === size, client.writableNeedDrain); + client.once("drain", () => { + console.log("large drain", client.writableLength, client.writableNeedDrain); + client.destroy(); + resolve(); + }); + setTimeout(() => { + if (paused !== null) paused.resume(); + }, 30); + }); + }); +} + +function afterReset(port: number): Promise { + return new Promise((resolve) => { + const client = connect(port, "127.0.0.1", () => { + client.pause(); + setTimeout(() => { + console.log("first write after the peer closed", client.write("a")); + setTimeout(() => { + console.log("write after reset", client.write("b")); + }, 50); + }, 50); + }); + client.on("error", () => {}); + client.on("close", () => { + console.log("closed, destroyed", client.destroyed); + resolve(); + }); + }); +} + +let paused: Socket | null = null; +let mode = "echo"; +const server = createServer((socket: Socket) => { + if (mode === "echo") socket.pipe(socket); + else if (mode === "drain") socket.resume(); + else if (mode === "pause") { + socket.pause(); + paused = socket; + } else socket.destroy(); +}); + +server.listen(0, "127.0.0.1", async () => { + const addr = server.address(); + const port = typeof addr === "object" && addr !== null ? addr.port : 0; + await echo(port); + mode = "drain"; + await lookupPending(port); + mode = "pause"; + await largeWrite(port); + mode = "end"; + await afterReset(port); + server.close(); +}); diff --git a/tests/corpus/7519-net-socket-drain-listener-close.ts b/tests/corpus/7519-net-socket-drain-listener-close.ts new file mode 100644 index 0000000000..e1b51e7b4e --- /dev/null +++ b/tests/corpus/7519-net-socket-drain-listener-close.ts @@ -0,0 +1,62 @@ +// 'drain' listeners that capture their socket do not keep it after it +// closes: a once('drain') listener still pending when the peer disconnects +// before the buffer drains, and an on('drain') listener on a socket that +// drained and then closed. The sanitized lane's exit audit checks that +// both sockets are freed. +import { connect, createServer, Socket } from "node:net"; + +function disconnectBeforeDrain(): Promise { + return new Promise((resolve) => { + const server = createServer((socket: Socket) => { + socket.on("error", () => {}); + socket.once("drain", () => console.log("unexpected drain", socket.writableLength)); + console.log("server write", socket.write(Buffer.alloc(16 * 1024 * 1024, 97))); + socket.on("close", () => { + console.log("server socket closed, need drain", socket.writableNeedDrain); + server.close(() => resolve()); + }); + }); + server.listen(0, "127.0.0.1", () => { + const addr = server.address(); + const port = typeof addr === "object" && addr !== null ? addr.port : 0; + const client = connect(port, "127.0.0.1", () => { + client.pause(); + setTimeout(() => client.destroy(), 30); + }); + }); + }); +} + +function drainThenClose(): Promise { + return new Promise((resolve) => { + const server = createServer((socket: Socket) => { + socket.pause(); + setTimeout(() => socket.resume(), 30); + }); + server.listen(0, "127.0.0.1", () => { + const addr = server.address(); + const port = typeof addr === "object" && addr !== null ? addr.port : 0; + const client = connect(port, "127.0.0.1", () => { + let drains = 0; + client.on("drain", () => { + drains++; + console.log("drain", drains, client.writableLength); + client.end(); + }); + console.log("client write", client.write(Buffer.alloc(16 * 1024 * 1024, 98))); + }); + client.on("close", () => { + console.log("client closed"); + server.close(() => resolve()); + }); + }); + }); +} + +async function main(): Promise { + await disconnectBeforeDrain(); + await drainThenClose(); + console.log("done"); +} + +main(); diff --git a/tests/corpus/8010-http-header-raw-bytes.ts b/tests/corpus/8010-http-header-raw-bytes.ts new file mode 100644 index 0000000000..d8408f3356 --- /dev/null +++ b/tests/corpus/8010-http-header-raw-bytes.ts @@ -0,0 +1,72 @@ +// Header values hold the raw bytes a client sent, one character per byte +// (Node decodes them as latin1). Bytes that are not well-formed UTF-8 (lone +// continuation bytes, runs of them, truncated leads, overlong leads and +// bytes no sequence uses) at every offset within an eight-byte word encode +// to the same bytes as in Node with latin1, ascii, binary, hex, base64 and +// utf16le. Strings built in the program take the same encodings with two-, +// three- and four-byte characters at every offset. +import { createServer } from "node:http"; +import { connect } from "node:net"; + +const patterns: number[][] = [ + [0x80], + [0x80, 0x81, 0xbf, 0x90], + [0xc3], + [0xc0, 0x80, 0xc1, 0xbf], + [0xe9, 0x41, 0xe9], + [0xf0, 0x9f, 0x41], + [0xe2, 0x82], + [0xff, 0xfe, 0xf5], + [0x78, 0xe9, 0x79, 0x80, 0x80, 0x7a, 0xdf], +]; +const encodings = ["latin1", "ascii", "binary", "hex", "base64", "utf16le"] as const; + +function hex(bytes: Buffer): string { + return bytes.toString("hex"); +} + +function describe(label: string, value: string): void { + console.log(label, encodings.map((encoding) => `${encoding}=${hex(Buffer.from(value, encoding))}`).join(" ")); +} + +const names: string[] = []; +const head: Buffer[] = [Buffer.from("GET / HTTP/1.1\r\nHost: x\r\nConnection: close\r\n")]; +for (let p = 0; p < patterns.length; p++) { + for (let offset = 0; offset < 10; offset++) { + for (const tail of ["", "bcdefghij"]) { + const name = `x-${p}-${offset}-${tail.length}`; + names.push(name); + head.push(Buffer.from(`${name}: ${"a".repeat(offset)}`)); + head.push(Buffer.from(patterns[p]!)); + head.push(Buffer.from(`${tail}\r\n`)); + } + } +} +head.push(Buffer.from("\r\n")); + +const server = createServer((req, res) => { + for (const name of names) { + const value = req.headers[name]; + if (typeof value === "string") describe(name, value); + else console.log(name, "missing"); + } + res.end("ok"); +}); + +server.listen(0, "127.0.0.1", () => { + const address = server.address(); + const port = typeof address === "object" && address !== null ? address.port : 0; + const socket = connect({ port, host: "127.0.0.1" }, () => { + socket.write(Buffer.concat(head)); + }); + socket.on("data", () => {}); + socket.on("close", () => { + server.close(); + for (const text of ["é", "éü", "߀", "😀", "é😀ü", "߿\u0080"]) { + for (let offset = 0; offset < 17; offset++) { + const value = "a".repeat(offset) + text + "b".repeat(offset % 9) + text; + describe(`text ${offset} ${JSON.stringify(text)}`, value); + } + } + }); +}); diff --git a/tests/corpus/8011-foreign-element-tojson.ts b/tests/corpus/8011-foreign-element-tojson.ts new file mode 100644 index 0000000000..60db9c9bdc --- /dev/null +++ b/tests/corpus/8011-foreign-element-tojson.ts @@ -0,0 +1,72 @@ +// JSON.stringify of a typed array runs the toJSON of a foreign element +// (an object written through an unknown[] view) with the element's index +// as its key, as Node does, including when the hook grows, truncates or +// replaces the arrays holding it. A hook or getter that throws stops the +// stringify with its exception. +const nums: number[] = [1, 2, 3]; +const view: unknown[] = nums; +view[1] = { + toJSON(key: string) { + return `v${key}`; + }, +}; +view[2] = { toJSON: 5 }; +console.log(JSON.stringify(nums), JSON.stringify(view)); + +const words: string[] = ["a", "b"]; +const wordsView: unknown[] = words; +wordsView[0] = { toJSON: () => undefined }; +wordsView[1] = { toJSON: () => ({ nested: [1, 2] }) }; +console.log(JSON.stringify(words), JSON.stringify({ words })); + +// The hook grows the parent past its storage and then truncates it. +const outer: number[][] = [[1, 2], [3], [4, 5, 6]]; +const innerView: unknown[] = outer[1]!; +let calls = 0; +innerView[1] = { + toJSON() { + calls++; + for (let i = 0; i < 300; i++) outer.push([i, i + 1]); + if (calls === 1) outer.length = 2; + return "hooked"; + }, +}; +console.log(JSON.stringify(outer), outer.length, calls); + +// The hook replaces the parent's elements two levels up. +const deep: number[][][] = [[[1], [2]], [[3]]]; +const deepView: unknown[] = deep[0]![1]!; +deepView[0] = { + toJSON() { + for (let i = 0; i < 200; i++) deep.push([[i]]); + deep[0] = []; + return 9; + }, +}; +console.log(JSON.stringify(deep).length, deep.length); + +// A throwing hook or getter stops the stringify. +const thrown: number[] = [1, 2]; +const thrownView: unknown[] = thrown; +thrownView[1] = { + toJSON() { + throw new Error("from toJSON"); + }, +}; +try { + console.log(JSON.stringify(thrown)); +} catch (error) { + console.log("caught", (error as Error).message); +} +thrownView[1] = { + get a() { + throw new Error("from getter"); + }, +}; +try { + console.log(JSON.stringify(thrown)); +} catch (error) { + console.log("caught", (error as Error).message); +} +const after: unknown[] = [1, { b: 2 }]; +console.log(JSON.stringify(after)); diff --git a/tests/corpus/8167-encrypted-private-keys.cjs b/tests/corpus/8167-encrypted-private-keys.cjs index 561a0aec31..7b274e500a 100644 --- a/tests/corpus/8167-encrypted-private-keys.cjs +++ b/tests/corpus/8167-encrypted-private-keys.cjs @@ -51,11 +51,40 @@ tryIt("empty passphrase", () => { return same(c.createPrivateKey({ key: out, passphrase: "" }), rsa); }); -// Reading encrypted keys. -const p8pem = ec.privateKey.export({ type: "pkcs8", format: "pem", cipher: "aes-256-cbc", passphrase: "pw" }); -const p8der = ec.privateKey.export({ type: "pkcs8", format: "der", cipher: "aes-256-cbc", passphrase: "pw" }); -const sec1 = String(ec.privateKey.export({ type: "sec1", format: "pem", cipher: "aes-128-cbc", passphrase: "pw" })); -tryIt("public from encrypted", () => c.createPublicKey({ key: p8pem, passphrase: "pw" }).equals(ec.publicKey)); +// Reading encrypted keys. These are fixed encryptions of one P-256 key +// with the passphrase "pw": decrypting with a wrong passphrase gives +// random-looking bytes, which a fresh encryption would make fail at +// different steps from run to run. +const fixedKey = [ + "-----BEGIN PRIVATE KEY-----", + "MIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgExcc5/xZ3vYL2Dpg", + "XnB7E8gQhcW1bkUcIGJWBGvrrGChRANCAAR0wxNUH1yo/U9nqFY4Gt5SVzPjvWq6", + "NzlStUC6Njk9BdR7SvBlNIGDAFlOfMwhfZbvvvMP+rLMR3RtBg3Ketxe", + "-----END PRIVATE KEY-----", +].join("\n") + "\n"; +const p8pem = [ + "-----BEGIN ENCRYPTED PRIVATE KEY-----", + "MIH0MF8GCSqGSIb3DQEFDTBSMDEGCSqGSIb3DQEFDDAkBBBkpqs2611cRjGDZTd4", + "FmNeAgIIADAMBggqhkiG9w0CCQUAMB0GCWCGSAFlAwQBKgQQRhL9yxS4i0ngtke+", + "frhURASBkN0dk+GQNCGU5cFC+MMswOr9XwGaQnLbJo7VnhCje3//JaVJDjexrd69", + "YZliX7pKGNghfFf+eBAeOAkwbAK39T8KQPwTJVJ+Z5O485N2fl5dPlr3gaEA3+cm", + "LQgasmL7Zv9UOH2iwvWllFYL3bnwr7VsqTCW2xe53qTB+dWVurP0cCiZNGEnUzrf", + "4bvWNrW0nA==", + "-----END ENCRYPTED PRIVATE KEY-----", +].join("\n") + "\n"; +const p8der = Buffer.from("MIH0MF8GCSqGSIb3DQEFDTBSMDEGCSqGSIb3DQEFDDAkBBAOrVavYYEvy3KUHwK0hbQnAgIIADAMBggqhkiG9w0CCQUAMB0GCWCGSAFlAwQBKgQQmEnqHOLegUirb49LD8oNhASBkGUwe3SZciScH/tWuoK5JWujckNHRKROwu81a5AGjIKxkvqCm+Qp9Gr4rPsb2vbv3bgfojbEKvSGvtuqB/X/jpN5UMZwWI3D9Ocd1+jEq1FqDb3rb3qxcoO9jmwuFFzNeYy4sW29XFuKTxu4aJHT1sEtMzIVlRLv53aXzpp9eQ9j47EOTWFqIpkt02TRFuazCQ==", "base64"); +const sec1 = [ + "-----BEGIN EC PRIVATE KEY-----", + "Proc-Type: 4,ENCRYPTED", + "DEK-Info: AES-128-CBC,E73A95A6783C45F914B397FE018521D2", + "", + "nsuhQFMPr+QFITcpmr2JWs+Zep4wxoCWMEk1jzyyXSJddfebIXOFg4xmtK8YJXIy", + "vqa6pvAIiak0iEZjVSn20+JAg2wqQ0sLaFPhBmhcZk+xuQoLD8CcYvROO5VtVELm", + "DkgOUcJ1oEfZJIRM6U2oDNFmj2NB/SnSQUf1niy7ba8=", + "-----END EC PRIVATE KEY-----", +].join("\n") + "\n"; +const fixedPublic = c.createPublicKey(fixedKey); +tryIt("public from encrypted", () => c.createPublicKey({ key: p8pem, passphrase: "pw" }).equals(fixedPublic)); tryIt("public from traditional", () => c.createPublicKey({ key: sec1, passphrase: "pw" }).type); tryIt("pkcs8 pem no passphrase", () => c.createPrivateKey(p8pem)); tryIt("pkcs8 der no passphrase", () => c.createPrivateKey({ key: p8der, format: "der", type: "pkcs8" })); diff --git a/tests/corpus/8347-string-iteration-code-points.ts b/tests/corpus/8347-string-iteration-code-points.ts new file mode 100644 index 0000000000..3a1286140b --- /dev/null +++ b/tests/corpus/8347-string-iteration-code-points.ts @@ -0,0 +1,96 @@ +// The string iterator: for-of, spread, Array.from, yield* and +// [Symbol.iterator]() walk code points. Astral characters come back whole, +// a lone surrogate is a one-unit character, and halves joined by +// concatenation form one character, as in Node. +function unitCodes(c: string): string { + const codes: string[] = []; + for (let k = 0; k < c.length; k++) codes.push(c.charCodeAt(k).toString(16)); + return `${c.length}:${codes.join(".")}`; +} +const show = (chars: string[]): string => chars.map(unitCodes).join(" "); + +const samples = [ + "", + "abc", + "aé😀b", + "\u0000\u007f\u0080߿ࠀ￿", + "\u{10000}\u{10ffff}", + "é ñ", + "\ud800", + "\udc00", + "x\ud83d", + "\ude00x", + "\udc00\ud800", + "\ud83d" + "\ude00", + "a\ud83d" + "\ude00b\ud83d", + "日本語テキスト", +]; +for (const s of samples) { + const looped: string[] = []; + for (const c of s) looped.push(c); + console.log(JSON.stringify(s), show(looped), show([...s]), show(Array.from(s)), [...s].length, s.length); +} + +// Loop control, heads and bindings. +const text = "a😀b\ud800c"; +let out = ""; +outer: for (const c of text) { + if (c === "b") continue; + for (const d of "xy") { + if (d === "y" && c === "c") continue outer; + out += c + d; + } + if (c === "\ud800") out += "|"; +} +console.log(out); +for (var v of "hé") out += v; +console.log(out); +let last = ""; +for (last of "q😀") out += last.length; +console.log(out, last); +for (const [first] of ["😀z", "ab"].join("")) out += first; +console.log(out); +let source = "abc"; +for (const c of source) { + source += c; + out += source.length; +} +console.log(out, source); +let n = 0; +for (const c of "stop😀here") { + if (c === "😀") break; + n += c.length; +} +console.log(n); + +// Characters as values: lengths, Map keys and code points. +const counts = new Map(); +for (const c of "abracadabra😀😀é") counts.set(c, (counts.get(c) ?? 0) + 1); +console.log([...counts].map(([k, v]) => `${k}=${v}`).join(",")); +let points = 0; +for (const c of "😀é\ud800a") points += c.codePointAt(0)!; +console.log(points); + +// Generators and explicit iterators. +function* letters(s: string): Generator { + yield* s; +} +console.log(show([...letters("x😀\udc00y")])); +const it = "o😀"[Symbol.iterator](); +let step = it.next(); +while (!step.done) { + out += step.value.length; + step = it.next(); +} +console.log(out); + +// A long string built from mixed pieces. +let long = ""; +for (let i = 0; i < 2000; i++) long += i % 3 === 0 ? "a" : i % 3 === 1 ? "é" : "😀"; +let units = 0; +let chars = 0; +for (const c of long) { + units += c.length; + chars++; +} +console.log(units, chars, long.length, [...long].length); diff --git a/tests/corpus/8348-string-iteration-worker-characters.ts b/tests/corpus/8348-string-iteration-worker-characters.ts new file mode 100644 index 0000000000..52a38fb78e --- /dev/null +++ b/tests/corpus/8348-string-iteration-worker-characters.ts @@ -0,0 +1,44 @@ +import { Worker, isMainThread, parentPort } from "node:worker_threads"; + +// The one-character strings of U+0080..U+07FF that string iteration returns +// are shared by every thread. A worker that is the first to iterate them +// and exits without sharing anything else leaves them in use: the main +// thread iterates the same characters after the worker's memory goes back. + +const text = "aé ñ ß ü Ω ж א ع ߿"; + +function codes(s: string): string { + const out: string[] = []; + for (const c of s) out.push(`${c.length}:${c.charCodeAt(0).toString(16)}`); + return out.join(" "); +} + +function runWorker(): Promise { + return new Promise((resolve, reject) => { + const worker = new Worker(new URL(import.meta.url)); + let count = -1; + worker.on("message", (value: number) => { + count = value; + }); + worker.on("error", reject); + worker.on("exit", () => resolve(count)); + }); +} + +async function main(): Promise { + console.log("worker", await runWorker()); + // The worker's memory goes back shortly after it exits. + await new Promise((resolve) => setTimeout(resolve, 300)); + console.log("main", codes(text)); + console.log("spread", [...text].join("|")); + console.log("from", Array.from("ñandú").map((c) => c.length).join(",")); +} + +if (isMainThread) { + void main(); +} else { + let count = 0; + for (const c of text) count += c.length; + count += [...text].length + Array.from("ñandú").length; + parentPort!.postMessage(count); +} diff --git a/tests/corpus/8351-regex-literal-replacers.ts b/tests/corpus/8351-regex-literal-replacers.ts new file mode 100644 index 0000000000..12968d4d97 --- /dev/null +++ b/tests/corpus/8351-regex-literal-replacers.ts @@ -0,0 +1,72 @@ +// Global substitutions of regex literals with literal replacements: matches +// at every offset around 16-character blocks and at the end of the +// subject, empty matches, alternation order, case-insensitive and +// multiline flags, word boundaries, replacements that are empty, longer +// than the match or hold quotes and backslashes, outputs that grow far past +// the subject, subjects without a match, and subjects outside ASCII (which +// take the general loop). replace and replaceAll. +const substitutions: [string, (s: string) => string][] = [ + ["<[^>]*>", (s) => s.replace(/<[^>]*>/g, "|")], + ["a[NSt]|BY", (s) => s.replace(/a[NSt]|BY/g, "<2>")], + ["tHa[Nt]", (s) => s.replace(/tHa[Nt]/g, "<4>")], + ["\\|[^|][^|]*\\|", (s) => s.replace(/\|[^|][^|]*\|/g, "-")], + ["x*", (s) => s.replace(/x*/g, "-")], + ["\\b", (s) => s.replace(/\b/g, "^")], + ["^ab", (s) => s.replace(/^ab/gm, "Z")], + ["ab|abc|b", (s) => s.replace(/ab|abc|b/g, "#")], + ["AB (i)", (s) => s.replace(/AB/gi, "")], + ["(a)(b)?", (s) => s.replace(/(a)(b)?/g, "q")], + ["\\d+", (s) => s.replace(/\d+/g, '"\\\n')], + ["[^a-z]", (s) => s.replace(/[^a-z]/g, "")], + ["a", (s) => s.replace(/a/g, "<<<>>>")], + ["cat (replaceAll)", (s) => s.replaceAll(/cat/g, "dog")], + ["[aeiou]{2} (replaceAll)", (s) => s.replaceAll(/[aeiou]{2}/g, "~")], +]; + +function hash(text: string, h: number): number { + for (let i = 0; i < text.length; i++) h = (h * 31 + text.charCodeAt(i)) >>> 0; + return h; +} + +const filler = "qwertyuiop asdfghjkl zxcvbnm"; +const needles = ["<2>", "", "aN", "BY", "tHat", "|ab|", "||", "xx", "ab", "abc", "AB", "123", "cat", "aeio", "a"]; +let checksum = 0; +let lines = 0; +for (let length = 0; length <= 70; length += 7) { + for (const needle of needles) { + for (const offset of [0, 1, 14, 15, 16, 17, 31, 32, 33]) { + if (offset + needle.length > length) continue; + let subject = ""; + while (subject.length < length) subject += filler; + subject = subject.slice(0, offset) + needle + subject.slice(offset + needle.length, length); + for (const [name, sub] of substitutions) { + const out = sub(subject); + checksum = hash(out, checksum); + lines++; + if (length === 35 && (offset === 15 || offset === 16)) console.log(name, offset, JSON.stringify(out)); + } + } + } +} +console.log("lines", lines, "checksum", checksum); + +// Dense matches, matches that end the subject, and an output many times +// larger than the subject. +const dense = "<1><22><333>".repeat(40) + "tail"; +for (const [name, sub] of substitutions) console.log(name, JSON.stringify(sub(dense).slice(-40)), sub(dense).length); +const many = "a".repeat(5000); +console.log("grow", many.replace(/a/g, "<<<>>>").length, many.replace(/a/g, "<<<>>>").slice(4990)); +console.log("empty", JSON.stringify("".replace(/x*/g, "-")), JSON.stringify("".replace(/a/g, "b"))); +console.log("none", "zzzz".replace(/a[bc]/g, "!"), "zzzz".replaceAll(/a[bc]/g, "!")); + +// Subjects outside ASCII run the general loop with the same results. +for (const subject of ["héllo wörld ", "日本<語>", "a\u{1F600}b", "cat café cat"]) { + console.log(JSON.stringify(subject.replace(/<[^>]*>/g, "|")), JSON.stringify(subject.replaceAll(/cat/g, "dog")), JSON.stringify(subject.replace(/x*/g, "-"))); +} + +// A long DNA-like subject, as in regex-redux. +let dna = ""; +for (let i = 0; i < 4000; i++) dna += "acgt"[(i * 7 + (i >> 2)) % 4] + (i % 61 === 0 ? "tHaN" : i % 37 === 0 ? "aNBY" : ""); +const step1 = dna.replace(/tHa[Nt]/g, "<4>").replace(/a[NSt]|BY/g, "<2>"); +const step2 = step1.replace(/<[^>]*>/g, "|").replace(/\|[^|][^|]*\|/g, "-"); +console.log("dna", dna.length, step1.length, step2.length, hash(step2, 0)); diff --git a/tests/corpus/intersection-restated-members.ts b/tests/corpus/intersection-restated-members.ts new file mode 100644 index 0000000000..249b9ddfbe --- /dev/null +++ b/tests/corpus/intersection-restated-members.ts @@ -0,0 +1,82 @@ +// An intersection that restates a member with a separately written but +// structurally equal type maps as that one type. Restating an optional member +// as required makes it required; the checker keeps both element literals +// (`A[] & B[]`), and their arrays, tuples, and maps share one representation. + +interface Options { + root: string; + files?: Array<{ path: string; size: number }>; +} +type Normalized = Options & { files: Array<{ path: string; size: number }> }; + +function normalize(options: Options): Normalized { + return { ...options, files: options.files ?? [] }; +} + +function totalSize(n: Normalized): number { + let total = 0; + for (const f of n.files) total += f.size; + return total; +} + +const plain = normalize({ root: "/a" }); +console.log(plain.root, plain.files.length, totalSize(plain)); +const full = normalize({ + root: "/b", + files: [ + { path: "x.txt", size: 3 }, + { path: "y.txt", size: 4 }, + ], +}); +full.files.push({ path: "z.txt", size: 5 }); +console.log(full.files.map((f) => f.path).join(","), totalSize(full)); +console.log(JSON.stringify(full)); + +let maybe: Normalized | undefined; +if (process.argv.length < 99) maybe = { root: "/", files: [{ path: "a", size: 1 }] }; +console.log(maybe?.root, maybe?.files[0]?.path); + +// Nested records whose inner arrays are restated. +interface Settings { + name: string; + meta?: { tags: Array<{ key: string; value: string }> }; +} +type Resolved = Settings & { meta: { tags: Array<{ key: string; value: string }> } }; +const resolved: Resolved = { name: "s", meta: { tags: [{ key: "k", value: "v" }] } }; +resolved.meta.tags.push({ key: "k2", value: "v2" }); +for (const t of resolved.meta.tags) console.log(`${t.key}=${t.value}`); + +// Tuples, nested arrays, and maps. +interface Layout { + pair?: [{ x: number }, string]; + grid?: Array>; + index?: Map; +} +type Filled = Layout & { + pair: [{ x: number }, string]; + grid: Array>; + index: Map; +}; +const filled: Filled = { + pair: [{ x: 7 }, "seven"], + grid: [[{ cell: 1 }, { cell: 2 }], [{ cell: 3 }]], + index: new Map([["home", { hits: 2 }]]), +}; +filled.index.set("about", { hits: 1 }); +console.log(filled.pair[0].x, filled.pair[1]); +console.log(filled.grid.map((row) => row.map((c) => c.cell).join("+")).join(" | ")); +for (const [k, v] of filled.index) console.log(k, v.hits); + +// Both sides required, and both sides optional (the member stays optional). +type Twice = { items: Array<{ n: number }> } & { items: Array<{ n: number }> }; +const twice: Twice = { items: [{ n: 1 }, { n: 2 }] }; +console.log(twice.items.reduce((s, i) => s + i.n, 0)); +type BothOptional = { extra?: Array<{ n: number }> } & { extra?: Array<{ n: number }> }; +const none: BothOptional = {}; +const some: BothOptional = { extra: [{ n: 9 }] }; +console.log(none.extra === undefined, some.extra?.[0]?.n); + +// A restated member read through a function value. +type Handler = (event: Normalized) => string; +const describe: Handler = (event) => `${event.root}:${event.files.length}`; +console.log(describe(full), describe(plain)); diff --git a/tests/corpus/native-array-dynamic-reads/main.ts b/tests/corpus/native-array-dynamic-reads/main.ts new file mode 100644 index 0000000000..0641114bfd --- /dev/null +++ b/tests/corpus/native-array-dynamic-reads/main.ts @@ -0,0 +1,32 @@ +// Native arrays and iterators from typed code, read by untyped code: for-of +// over them, their length, an iterator's next read as a value and called, +// generators and async generators beside them, and an array whose +// iteration a Symbol.iterator use elsewhere in the program does not change. +import { asyncKinds, describe, lengths, sumAll, stepper } from "./reads.js"; + +interface Item { + sku: string; + qty: number; +} +const numbers: number[] = [1, 2, 3, 4]; +const items: Item[] = [ + { sku: "a", qty: 2 }, + { sku: "b", qty: 5 }, +]; +const empty: string[] = []; +function* counting(): Generator { + yield 10; + yield 20; +} +console.log(sumAll(numbers), sumAll(items.map((i) => i.qty)), sumAll(empty.map(Number))); +console.log(lengths([numbers, items, empty]).join()); +numbers.push(5); +console.log(lengths([numbers]).join(), sumAll(numbers)); +console.log(describe(items), describe(counting())); +console.log(stepper(numbers).join(), stepper(counting()).join()); +const iterable = { *[Symbol.iterator]() { yield "own"; } }; +console.log([...iterable].join(), sumAll(numbers)); +async function* later(): AsyncGenerator { + yield 1; +} +asyncKinds(later(), numbers).then((kinds) => console.log(kinds.join())); diff --git a/tests/corpus/native-array-dynamic-reads/reads.js b/tests/corpus/native-array-dynamic-reads/reads.js new file mode 100644 index 0000000000..67e1a824af --- /dev/null +++ b/tests/corpus/native-array-dynamic-reads/reads.js @@ -0,0 +1,29 @@ +export function sumAll(values) { + let total = 0; + for (const value of values) total += value; + return total; +} +export function lengths(arrays) { + const out = []; + for (const array of arrays) out.push(array.length); + return out; +} +export function describe(values) { + const parts = []; + for (const value of values) parts.push(typeof value === "object" ? value.sku : value); + return parts.join("+"); +} +export function stepper(values) { + const iterator = values[Symbol.iterator](); + const next = iterator.next; + const out = []; + for (let step = next.call(iterator); !step.done; step = next.call(iterator)) out.push(step.value); + out.push(typeof iterator.next, iterator.next === next); + return out; +} +export async function asyncKinds(generator, array) { + const out = []; + for await (const value of generator) out.push(value); + for (const value of array) out.push(value); + return out; +} diff --git a/tests/fixtures/self-hosting/llvm-layouts.ts b/tests/fixtures/self-hosting/llvm-layouts.ts index 36d55bddf2..8e19896d23 100644 --- a/tests/fixtures/self-hosting/llvm-layouts.ts +++ b/tests/fixtures/self-hosting/llvm-layouts.ts @@ -51,6 +51,7 @@ try { declare: (decl) => { declarations.add(decl); }, needOom: () => { needsOom = true; }, needBadTag: () => { needsBadTag = true; }, + mayHoldForeign: () => false, internLiteral: (text) => { let index = literals.indexOf(text); if (index === -1) { index = literals.length; literals.push(text); } diff --git a/tests/harness/corpus-costs/6067-view-holes-in-typed-arrays.ts.json b/tests/harness/corpus-costs/6067-view-holes-in-typed-arrays.ts.json new file mode 100644 index 0000000000..b8b740bf8f --- /dev/null +++ b/tests/harness/corpus-costs/6067-view-holes-in-typed-arrays.ts.json @@ -0,0 +1 @@ +{ "plain": 3967, "san": 4623 } diff --git a/tests/harness/corpus-costs/6068-foreign-elements-through-views.ts.json b/tests/harness/corpus-costs/6068-foreign-elements-through-views.ts.json new file mode 100644 index 0000000000..eea1236acb --- /dev/null +++ b/tests/harness/corpus-costs/6068-foreign-elements-through-views.ts.json @@ -0,0 +1 @@ +{ "plain": 6595, "san": 5482 } diff --git a/tests/harness/corpus-costs/6069-foreign-element-writers.ts.json b/tests/harness/corpus-costs/6069-foreign-element-writers.ts.json new file mode 100644 index 0000000000..48bc4a979e --- /dev/null +++ b/tests/harness/corpus-costs/6069-foreign-element-writers.ts.json @@ -0,0 +1 @@ +{ "plain": 3978, "san": 3898 } diff --git a/tests/harness/corpus-costs/7515-net-socket-backpressure.ts.json b/tests/harness/corpus-costs/7515-net-socket-backpressure.ts.json new file mode 100644 index 0000000000..feabbe8021 --- /dev/null +++ b/tests/harness/corpus-costs/7515-net-socket-backpressure.ts.json @@ -0,0 +1 @@ +{ "plain": 2800, "san": 4200 } diff --git a/tests/harness/corpus-costs/7519-net-socket-drain-listener-close.ts.json b/tests/harness/corpus-costs/7519-net-socket-drain-listener-close.ts.json new file mode 100644 index 0000000000..78ee9421a0 --- /dev/null +++ b/tests/harness/corpus-costs/7519-net-socket-drain-listener-close.ts.json @@ -0,0 +1 @@ +{ "plain": 2300, "san": 3100 } diff --git a/tests/harness/corpus-costs/8010-http-header-raw-bytes.ts.json b/tests/harness/corpus-costs/8010-http-header-raw-bytes.ts.json new file mode 100644 index 0000000000..532e3bd742 --- /dev/null +++ b/tests/harness/corpus-costs/8010-http-header-raw-bytes.ts.json @@ -0,0 +1 @@ +{ "plain": 3468, "san": 7463 } diff --git a/tests/harness/corpus-costs/8347-string-iteration-code-points.ts.json b/tests/harness/corpus-costs/8347-string-iteration-code-points.ts.json new file mode 100644 index 0000000000..3e2ae0706e --- /dev/null +++ b/tests/harness/corpus-costs/8347-string-iteration-code-points.ts.json @@ -0,0 +1 @@ +{ "plain": 2770, "san": 3750 } diff --git a/tests/harness/corpus-costs/8348-string-iteration-worker-characters.ts.json b/tests/harness/corpus-costs/8348-string-iteration-worker-characters.ts.json new file mode 100644 index 0000000000..4c96af583a --- /dev/null +++ b/tests/harness/corpus-costs/8348-string-iteration-worker-characters.ts.json @@ -0,0 +1 @@ +{ "plain": 3960, "san": 3900 } diff --git a/tests/harness/corpus-costs/8351-regex-literal-replacers.ts.json b/tests/harness/corpus-costs/8351-regex-literal-replacers.ts.json new file mode 100644 index 0000000000..56546d44c8 --- /dev/null +++ b/tests/harness/corpus-costs/8351-regex-literal-replacers.ts.json @@ -0,0 +1 @@ +{ "plain": 3074, "san": 44176 } diff --git a/tests/harness/corpus-costs/intersection-restated-members.ts.json b/tests/harness/corpus-costs/intersection-restated-members.ts.json new file mode 100644 index 0000000000..1254aa6e43 --- /dev/null +++ b/tests/harness/corpus-costs/intersection-restated-members.ts.json @@ -0,0 +1 @@ +{ "plain": 4000 } diff --git a/tests/harness/corpus-costs/native-array-dynamic-reads/main.ts.json b/tests/harness/corpus-costs/native-array-dynamic-reads/main.ts.json new file mode 100644 index 0000000000..ac2ef5ae87 --- /dev/null +++ b/tests/harness/corpus-costs/native-array-dynamic-reads/main.ts.json @@ -0,0 +1 @@ +{ "plain": 3567, "san": 6402 } diff --git a/tests/harness/string-iteration-workers.test.ts b/tests/harness/string-iteration-workers.test.ts new file mode 100644 index 0000000000..bc1feca95c --- /dev/null +++ b/tests/harness/string-iteration-workers.test.ts @@ -0,0 +1,44 @@ +/* The one-character strings that string iteration shares between threads + * outlive the worker that first made them. On Linux a worker that exits + * without sharing anything returns its memory; with SCR_SA_DISCARD=poison + * that memory becomes inaccessible instead, so a cached character + * allocated from the worker's memory would fault when the main thread + * reads it. The corpus program pins Node.js parity; this suite runs it + * with the discard poisoned and immediate. Under SCRIPTC_SAN=1 the runtime + * keeps every worker's memory, so the check does not apply. */ +import { execFile } from "node:child_process"; +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { promisify } from "node:util"; +import { expect, test } from "vitest"; +import { compile } from "@scriptc/compiler"; + +const exec = promisify(execFile); +const program = join(import.meta.dirname, "../corpus/8348-string-iteration-worker-characters.ts"); +const sanitize = process.env["SCRIPTC_SAN"] === "1"; + +test.skipIf(process.platform !== "linux" || sanitize)( + "characters a discarded worker cached stay readable", + { timeout: 240_000 }, + async () => { + const outDir = await mkdtemp(join(tmpdir(), "scriptc-string-iteration-workers-")); + try { + const result = await compile(program, { + backend: "llvm", + outDir, + outPath: join(outDir, "program"), + }); + expect(result.ok, JSON.stringify(result.diagnostics)).toBe(true); + if (!result.ok) return; + const { stdout } = await exec(result.binaryPath, [], { + env: { ...process.env, SCR_SA_DISCARD: "poison", SCR_SA_DISCARD_DELAY_MS: "0" }, + timeout: 120_000, + }); + expect(stdout).toContain("main 1:61 1:e9 1:20 1:f1"); + expect(stdout).toContain("spread a|é| |ñ"); + } finally { + await rm(outDir, { recursive: true, force: true }); + } + }, +);